2 Parameters

Mathematically, we use the same linear algebra to estimate the all the parameters in a regression model.
However, we have a number of different types of parameters with somewhat different interpretations.

  • Intercept (constant). One and only one, this exists in every regression model.
  • Relative intercept(s). One per each additional level introduced by each categorical variable. Interaction terms among categorical variables introduce more relative intercepts.
  • Slope(s). One per continuous variable.
  • Relative slopes. Interaction terms between categorical variables and a single continuous variable.
  • Relative curvature. Interaction terms between two or more continuous variables, including a continuous variable with itself.

2.1 “Main” Effects - Intercepts and Slopes

“Main” effects would be better called base or baseline effects.

Every regression model has an intercept. This is generally the predicted value of \(y\) where everything else is 0. If there are no other variables, this is the mean of \(y\). This is a point in the data space. Points have no dimension, so we think of this as a zero-order parameter in the model.

The simple regression model with one continuous \(x\) also has a slope. Taken together with an intercept, this defines a unique line in the data space. Lines have one dimension, so we think of this as a first-order parameter.

In an additive regression model (where the effects of variables are independent and each variable appears in only one term) adding more continuous variables adds new slopes, with each variable adding a new dimension to our data space. Each of these is a first-order parameter, and they all are taken with the same intercept. Our regression model is now a surface or a hyperplane in higher dimensional models.

In an additive regression model adding categorical variables (indicator variables) adds new intercepts, whether this is additional levels within a set of categories or additional sets of categories.
These are specified relative to the main intercept, offsets to the predicted value where everything except the category indictor is zero. These again are zero-order parameters.

2.2 Interaction Terms

Interaction terms are multiplicative combinations.

The interpretation and the order of interaction parameters depend upon what types of variables are used in combination.

  • Categorical by categorical: relative intercepts, 0-order
  • Categorical by continuous: relative slopes, 1st-order
  • Continous by continuous: curvature, 2nd-order
    • Two different variables
    • The same variable twice: polynomial curvature