Chapter 26 Conditions of the Least Squares Method
In the previous chapters we mentioned different starting points and assumptions for regression analysis with the least squares method. This chapter goes through these assumptions in somewhat more detail and discusses their implications.
26.1 Gauss-Markov
The theoretical assumptions for the least squares method are described in the Gauss-Markov theorem, named after Carl Friedrich Gauss and Andrey Markov. The theorem describes the conditions for the least squares method estimator to have the smallest sampling variance among all linear unbiased estimators. The word “unbiased” refers to the model estimating the population value correctly. The conditions can be presented in slightly different ways and the description depends partly on what type of data we are studying.
If one or more of these conditions are not fulfilled, the results we estimate using the least squares method risk becoming misleading. When we analyze real data, it is common that one or more conditions are not fulfilled. This may sound discouraging, but fortunately there are also several established methods for handling these challenges. This chapter provides occasional simple examples of solutions, but for those interested there is a large amount of literature to delve deeper into.
26.2 Assumption 1: Linearity
We start from the following regression model for the population, described with matrices:
\[\begin{equation} Y = XB + V \tag{26.1} \end{equation}\]
where we have \(n\) number of observations and \(k\) number of explanatory variables \(x_1, \ldots, x_k\) and \(k+1\) number of coefficients \(b_0, b_1, \ldots, b_k\). The letters in the regression model symbolize the same matrices as described in previous examples.
The regression model describes a linear relationship that is found in the population. Linearity means that all coefficients in a regression model have exponent = 1. The following models are not linear:
\[ \begin{aligned} Y &= a + \frac{1}{b}X + e \\ \log Z &= \alpha + \beta D + \epsilon \end{aligned} \]
In the first regression model we have \(\frac{1}{b} = b^{-1}\). In the second model we have that \(\log(Z_1 + Z_2) \neq \log(Z_1) + \log(Z_2)\), where \(Z_1\) and \(Z_2\) are two arbitrary values in variable \(Z\).
It can be difficult to test whether a regression model is a good linear approximation of the data we are studying. We may for instance inspect the data in a scatter graph or histogram, and compare different measures of dispersion. We can both compare how the explained variable covaries with our explanatory variables, and compare how the residuals are distributed around a regression line.
Let us illustrate this by comparing US GDP every tenth year during the period 1800–2000 using data from Measuring Worth,1 see Table 26.1, with years, GDP and logarithmic GDP. In Figure 26.1 the linear trend over time for GDP and ln(GDP) respectively is illustrated in separate graphs. In the left graph the trend is estimated based on the regression model:
\[Y_t = a + b\,\text{YEAR}_t + V_t\]
where \(Y_t\) is GDP in year \(t\) and \(\text{YEAR}_t\) is the year variable. In the right graph we have estimated the regression model:
\[\begin{equation} \ln Y_t = c + d\,\text{YEAR}_t + U_t \tag{26.2} \end{equation}\]
where \(\ln Y_t\) is logarithmic GDP in year \(t\).
| Year | GDP (billions USD) | ln GDP |
|---|---|---|
| 1800 | 0.486 | -0.7 |
| 1810 | 0.714 | -0.3 |
| 1820 | 0.718 | -0.3 |
| 1830 | 1.032 | 0.0 |
| 1840 | 1.590 | 0.5 |
| 1850 | 2.656 | 1.0 |
| 1860 | 4.410 | 1.5 |
| 1870 | 7.899 | 2.1 |
| 1880 | 10.592 | 2.4 |
| 1890 | 15.607 | 2.7 |
| 1900 | 21.197 | 3.1 |
| 1910 | 33.746 | 3.5 |
| 1920 | 89.246 | 4.5 |
| 1930 | 92.160 | 4.5 |
| 1940 | 102.899 | 4.6 |
| 1950 | 299.827 | 5.7 |
| 1960 | 542.382 | 6.3 |
| 1970 | 1073.303 | 7.0 |
| 1980 | 2857.307 | 8.0 |
| 1990 | 5963.144 | 8.7 |
| 2000 | 10250.952 | 9.2 |
A linear model fits the data relatively poorly when we use GDP, which we see in the left graph. The data points in the graph are placed approximately in the shape of a lying L. The development is exponential and not linear. The regression line does capture the positive development but does not give a representative picture of the long-term trend in GDP. During the years 1800–1850, to the left in the graph, the regression line is below 0. This means that the regression model predicts that GDP was negative all years before 1850. Since GDP is a measure of everything that is produced, bought and sold in a society, this is not possible. The linear model does however capture the long-term trend in logarithmic GDP relatively well, which is seen in the right graph where the points follow the regression line for all the years 1800–2000.
Figure 26.1: Linear trend for GDP and ln GDP.
Another way to compare how well the linear regression model captures the covariation in the data is to compare the residuals. In the two graphs vertical lines connect each point and the regression line. In the left graph the distance between the points and the regression line varies considerably — in a way that they do not in the right graph.
Logarithms can often be used to create a more linear relationship between variables. This often works well for exponential series. Unfortunately there is no general solution that works for all regression models. Note however how the data points in the right graph do not describe an exact linear relationship.
In practice, we often encounter populations with variables with nonlinear relationships. But even then we benefit from a linear regression model as a linear approximation, rather than an exact description of the population. When this is more or less suitable depends on what the purpose of the analysis is.
26.3 Assumption 2: Independent and Identically Distributed Observations
In the regression model the explained variable \(Y\) is a random variable drawn from the population. The explanatory variables in matrix \(X\) can either be random (as in an observational study) or created through a controlled process, for example in a controlled experiment where we ourselves design the treatment. Regardless, none of the variables are correlated with the error term.
In the description of the least squares method we assume that each observation \((y_i, x_{1,i}, \ldots, x_{k,i})\), \(i = 1, \ldots, n\), is independent and identically distributed (i.i.d.). That is, each observation is drawn in a random sample from a population with the same distribution. That the observations are independent means that the observations do not covary with each other in a systematic way.
Say for example that we want to study the covariation between some characteristics that people have. The characteristics are our variables and each person has their own observation. We draw a random sample from the population, which fulfills the assumption of i.i.d.
But say now that we want to follow people over time, so that each observation is information about a person at a point in time. One of the variables is earnings. Even though people’s earnings vary over time, we will be able to see patterns where different observations can be followed over time. The observations for a person’s earnings are therefore not independent of each other and therefore also do not fulfill the assumption of i.i.d.
26.4 Assumption 3: Expected Value and Variance of the Error Terms
The expected value of the error terms with respect to the explanatory variables \(X\) is \(E(V|X) = 0\). This also means that the error terms \(V\) do not covary with any of the explanatory variables \(X\). This can be described as us having exogenous explanatory variables \(X\). If any explanatory variable \(x_1, \ldots, x_k\) is correlated with the error term \(v\), this is called the variable being endogenous. In addition we assume that the error terms \(V\) have the same variance, regardless of observation and variable:
\[var(V|X) = \sigma^2 I_n\]
where the symbol \(\sigma^2\) symbolizes the population’s constant variance and \(I_n\) is an \(n \times n\) identity matrix. That the error term exhibits constant variance is usually called homoscedasticity. Its opposite is called heteroscedasticity, which means that the variance in the error term varies across the observations in one or more explanatory variables \(X\).
Figure 26.2 illustrates heteroscedasticity by comparing the residuals’ covariation against the explanatory variable \(X\). The points’ distance to the regression line varies across \(X\). Further to the right in the graph the distance between the line and the points is larger compared to the points further to the left. The residuals’ variance is thus not constant (homoscedasticity) across different observations but varies (heteroscedasticity).
Figure 26.2: Heteroscedasticity: the distance between points and the regression line increases with X.
A common assumption for the error terms is that these follow a normal distribution (see Section 26.7 below). Combined with the assumption of homoscedasticity this means that the error terms’ dispersion around the regression line follows a normal distribution with the same shape, regardless of which values for the explanatory variables \(X\) we compare against. With heteroscedasticity we may for example have normally distributed error terms but with different variance. Figure 26.3 illustrates an example where we have three normal distributions with different variance along a regression line.
Figure 26.3: Normal distributions with different variance along a regression line.
If the error term’s variance is not constant across different observations (heteroscedasticity), this can lead to the statistical tests (for example t-test and F-test) giving incorrect results. It is therefore often important to check for heteroscedasticity, which can be done in different ways. One way is to compare the dispersion of the residuals in graphs.
Above we described how we in many situations can use a linear regression model to describe a linear approximation for a relationship between two variables that is actually nonlinear. In such cases it is also common that the error terms do not have constant variance, that is, we find heteroscedasticity.
26.5 Assumption 4: Multicollinearity
The explanatory variables in the regression model must not correlate too strongly with each other. If a variable is a linear function of another variable in the regression model this is called them exhibiting collinearity. If several variables exhibit high correlation this is called multicollinearity. Suppose we have the explanatory variable \(x_1\)= earnings in USD, and \(x_2\)= earnings in thousands of USD. Variable \(x_1\) can be described as a linear function of \(x_2\): \(x_1 = 1{,}000 \cdot x_2\). The relationship between \(x_1\) and \(x_2\) is called perfect collinearity. If \(x_1\) is included in the regression model then it does not add any information to our analysis to also add \(x_2\).
Multicollinearity can affect the results in the regression analysis and disturb the calculation of the slope coefficients. In addition it can cause the variance in slope coefficients to become larger than what would otherwise have been the case. This can in turn reduce the probability that we in a t-test discover a statistically significant covariation between the variables concerned and the explained variable. Instead we risk accepting a false null hypothesis, a type 2 error.
The assumption about the absence of multicollinearity does not mean that two or more variables in a regression model may not covary at all. In regression analysis it is common that many variables have some form of relationship and that they therefore often covary more or less. The problem arises when they exhibit too high correlation. Perfect collinearity is usually easy to remedy by removing one of the variables from the regression model. Sometimes however it happens that two variables happen to have high correlation. In such cases it is difficult to give general recommendations.
To detect collinearity we can study covariation between variables in graphs. A common relatively simple check is to estimate the correlation coefficient \(r_{xy}\) for all pairs of explanatory variables. There is no exact value for \(r_{xy}\) that guarantees high collinearity, but a common recommendation is that if two explanatory variables exhibit \(r_{xy} > 0.8\), this should be regarded as an indication of problematically high collinearity.
Another common method for checking the occurrence of multicollinearity is to estimate the Variance Inflation Factor (VIF). Suppose we have the following regression model:
\[\begin{equation} Y = a + bX + cZ + V \tag{26.3} \end{equation}\]
To estimate VIF we take the auxiliary regression:
\[X = \beta_1 + \beta_2 Z + U\]
We estimate \(R^2\) for this regression and thereafter the VIF factor for \(\hat{b}\):
\[\begin{equation} VIF(\hat{b}) = \frac{1}{1 - R^2} \tag{26.4} \end{equation}\]
A rule of thumb is that if the VIF factor for an explanatory variable is over 5 or 10, this indicates high multicollinearity.
26.6 Assumption 5: Correlation Between Residuals
The error terms in the regression model are assumed to be uncorrelated, which can be described as the covariance \(cov(v_i, v_j) = 0\) for observation \(i\) and \(j\)(where \(i\) and \(j\) refer to any observations). Another way to describe this is that the error term should not covary with itself. This phenomenon is called autocorrelation, or serial correlation. We usually do not know the error terms in the population but we may examine the occurrence of autocorrelation among our estimated residuals.
Autocorrelation can for example arise if our regression model estimates the covariation between variables over time. We illustrate this by again comparing the example with the linear trend for the natural log of GDP (Section 26.2):
\[\ln Y_t = c + d\,\text{YEAR}_t + U_t\]
Figure 26.4 illustrates the covariation between the estimated residuals \(\hat{U}_t\) from this model. In the left graph only the residuals are shown, where the marked horizontal line at \(Y = 0\) is the regression line. The points in the left graph (the residuals) form the shape of a U. Residuals that are positive, above the 0-line, are often followed by another positive residual. And the residuals that are negative, below the 0-line, are often followed by another negative residual.
The pattern in the left graph is confirmed in the right graph where we instead see the covariation between residual \(\hat{U}_t\) and the residual for the previous year \(\hat{U}_{t-1}\). The straight diagonal line in the graph is the regression line from the regression model \(\hat{U}_t = \beta\hat{U}_{t-1} + \epsilon\). The correlation is positive, which indicates autocorrelation.
Figure 26.4: Residuals from the ln GDP regression model.
Autocorrelation does not affect the coefficients in our original regression model. But autocorrelation can lead to the standard error becoming smaller than it actually is, which in turn can cause the t-tests for the slope coefficients to give a higher t-value than otherwise and we may reject a true null hypothesis (type 1 error).
That is, in the t-test for slope coefficient \(b\) we estimate the t-value \(t = \frac{\hat{b} - b_0}{se(\hat{b})}\), where \(se(\hat{b})\) is the standard error for estimated \(\hat{b}\). If we test the hypothesis that \(b = 0\) then \(b_0 = 0\) and \(t = \frac{\hat{b}}{se(\hat{b})}\). If the standard error in the denominator of the t-value decreases then our estimated t-value increases and we will then get a value further out in the tails of the t-distribution.
26.7 Assumption 6: Normally Distributed Error Terms
In addition to the above conditions for the least squares method, an additional condition is often mentioned in this context: that the error term is normally distributed across all observations and explanatory variables in \(X\). We describe this as \(V|X \sim N(0, \sigma^2 I_n)\) where \(\sigma^2\) is the variance and \(I_n\) is an \(n \times n\) identity matrix. Given that the error terms and the residuals are approximately normally distributed we use t-tests to calculate the probability that our estimated coefficients are a result of a random process.
Even if we have good reasons to expect that the error term follows a normal distribution, in practice error terms can follow any distribution. This condition too we therefore check by studying the residuals. A first step can be to compare how the residuals are distributed in graphs. Let us illustrate this using the same data for logarithmic GDP from Figure 26.1. The regression model is \(\ln Y_t = c + d\,\text{YEAR}_t + U_t\). If we take the estimated residuals from that model and compare how they are distributed around their own mean, we get the distribution shown in Figure 26.5.
Figure 26.5: Residuals from the ln GDP regression model.
The bars show the number of residuals. In the graph we have also drawn a curve for a normal distribution with the same mean and standard deviation as the estimated residuals. The bars do not really follow the normal distribution curve. The residuals show a U-shaped pattern over time (as seen in Figure 26.4), which means their distribution is also non-normal — with fewer observations near zero than a normal distribution would imply.
Let us look at another example. Consider a regression model:
\[\begin{equation} L_i = a + bM_i + e_i \tag{26.5} \end{equation}\]
where \(L_i\) is mean life expectancy for the inhabitants in country \(i\) and \(M_i\) is GDP per capita. Figure 26.6 illustrates estimated results from this regression model in two graphs. In the graph to the left we see a positive covariation between life expectancy and earnings. In the graph to the right the residuals are shown, where the bars show the proportional distribution of the residuals and a normal distribution density curve with the same mean and variance as the residuals.
Figure 26.6: Life expectancy and income in Sweden’s 290 municipalities.
We compare the bars to the density curve to judge if the residuals’ distribution resembles a normal distribution. Whether we accept the residuals’ distribution as an indication that the population error terms follow a normal distribution depends, at least partly, on how big deviations from a normal distribution we wish to tolerate.
Sometimes a transformation of the variables in the regression model, for example taking logarithms, can lead to the residuals following a normal distribution. There are also specific methods for how to perform a regression analysis and instead assume that the error terms follow other types of probability distribution, which there is no room to go through here.
26.8 Adjusted Standard Errors
In this chapter we have gone through the conditions of the least squares method as well as examples of phenomena that violate these. Some of these deviating phenomena can be remedied by thinking through the model’s design. Some deviating phenomena can be handled by adjusting the data we study.
There are also many well-known alternative methods for estimating regression models and using statistical tests. A well-known such example is what is called robust standard errors, also called heteroscedasticity-robust standard errors or heteroscedasticity-consistent standard errors. Sometimes the same thing is called White’s standard errors after an article by Halbert White (1980) that contributed to the spread of this method within economics. Here follows a brief description, largely based on Hansen (2022) chapter 4. Many computer programs for statistical analysis have simple shortcut commands for all such calculations.
Robust standard errors is a summary term for a type of adjustment that we can perform on the residuals so that these can be used for regression analysis based on the least squares method, even in the presence of heteroscedasticity. Suppose we have the following regression model for the population:
\[Y = XB + V\]
where \(Y\) and \(X\) are matrices with explained and explanatory variables respectively, \(B\) is the coefficients and \(V\) is the error terms. We have previously defined the estimator \(\hat{B} = (X^TX)^{-1}X^TY\) and the variance-covariance matrix:
\[\begin{equation} var(\hat{B}|X) = (X^TX)^{-1} X^T E[VV^T|X] X (X^TX)^{-1} \tag{26.6} \end{equation}\]
Given that the error terms exhibit constant variance \(\sigma^2\)(homoscedasticity), \(E(VV^T|X) = \sigma^2 I\) and \(var(\hat{B}|X) = \sigma^2(X^TX)^{-1}\). If we work with sample data it is common that the residuals exhibit irregular variance (heteroscedasticity). What we now seek is an alternative version of the variance-covariance matrix that takes into account whether we have heteroscedasticity in our data.
It is possible to estimate the residuals’ variance as \(\hat{s}^2 = \frac{\sum(y_i - \hat{y}_i)^2}{(n-p)}\). The letters \(V\) in the expression \(E(VV^T|X)\) are the error terms in the population, which we do not know. Instead we use the estimated residuals \(\hat{V}\) and get \(E(\hat{V}\hat{V}^T|X) = \text{diag}(\hat{v}_1^2, \hat{v}_2^2, \ldots, \hat{v}_n^2)\), which is a diagonal matrix where the diagonal elements consist of the squared estimated residuals. We insert this into the variance-covariance matrix in equation (26.6) and rewrite:
\[\begin{equation} var(\hat{B}|X)^{\text{HC0}} = (X^TX)^{-1} \!\left(\sum_{i=1}^n X_i X_i^T \hat{v}_i^2\right)\! (X^TX)^{-1} \tag{26.7} \end{equation}\]
This is a variant of the variance-covariance matrix, robust for heteroscedasticity. We mark this version of the variance-covariance matrix with “HC0”: \(var(\hat{B}|X)^{\text{HC0}}\). Estimated results in \(var(\hat{B}|X)^{\text{HC0}}\) we use to for example estimate t-tests for the coefficients in regression models where the data exhibit heteroscedasticity.
In the chapter on multiple regression we saw how the variance-covariance matrix \(var(\hat{B}|X)\) is symmetric, the elements in the diagonal in \(var(\hat{B}|X)\) are the variance for each coefficient’s estimator, while the other elements are covariance for the coefficients’ estimators. The difference with \(var(\hat{B}|X)^{\text{HC0}}\) consists in how the estimates are estimated.
There are also additional variants of \(var(\hat{B}|X)\). The following version is often denoted \(var(\hat{B})^{\text{HC1}}\) and aims to adjust for \(\hat{v}_i^2\) risking becoming too low:
\[\begin{equation} var(\hat{B})^{\text{HC1}} = \frac{n}{n-k}\,(X^TX)^{-1} \!\left(\sum_{i=1}^n X_i X_i^T \hat{v}_i^2\right)\! (X^TX)^{-1} \tag{26.8} \end{equation}\]
| \(y\) | \(x\) | \(z\) |
|---|---|---|
| 3 | 3 | 1 |
| 2 | 4 | 4 |
| 5 | 6 | 0 |
| 4 | 7 | 1 |
where \(n\) is the number of observations and \(k\) is the number of coefficients in the regression model. The factor \(\frac{n}{n-k}\) means that HC1 is often recommended over HC0. We illustrate the differences between the three versions of the variance-covariance matrix \(var(\hat{B})\), \(var(\hat{B})^{\text{HC0}}\) and \(var(\hat{B})^{\text{HC1}}\) with the three variables \(y\), \(x\) and \(z\), the four observations in Table 26.2 and the following regression model:
\[ Y = XB + V, \quad \begin{bmatrix}3\\2\\5\\4\end{bmatrix} = \begin{bmatrix}1&3&1\\1&4&4\\1&6&0\\1&7&1\end{bmatrix} \begin{bmatrix}a\\b\\c\end{bmatrix} + \begin{bmatrix}v_1\\v_2\\v_3\\v_4\end{bmatrix} \]
where \(V\) is the error terms. We estimated this regression model in a previous chapter, found the residuals \(\hat{V} \approx \{-0.2, 0.14, 0.41, -0.34\}\) and the variance \(var(\hat{V}) = \hat{s}^2 \approx 0.338\). We do not go through the calculations in detail here but primarily describe the results:
\[\begin{align} var(\hat{B}|X) &= \hat{s}^2(X^TX)^{-1} \nonumber \\ &\approx \begin{bmatrix}1.49 & -0.23 & -0.16\\ -0.23 & 0.04 & 0.02\\ -0.16 & 0.02 & 0.05\end{bmatrix} \end{align}\]
For \(var(\hat{B}|X)^{\text{HC0}}\) we get:
\[\begin{align} var(\hat{B}|X)^{\text{HC0}} &= (X^TX)^{-1}\!\left(\sum_i^n X_i X_i^T \hat{e}_i^2\right)\!(X^TX)^{-1} \nonumber \\ &\approx \begin{bmatrix}0.247 & -0.041 & -0.025\\ -0.041 & 0.0086 & 0.002\\ -0.025 & 0.002 & 0.0066\end{bmatrix} \end{align}\]
For \(var(\hat{B}|X)^{\text{HC1}}\):
\[\begin{align} var(\hat{B}|X)^{\text{HC1}} &= \left(\frac{n}{n-k}\right)(X^TX)^{-1}\!\left(\sum_i^n X_i X_i^T \hat{e}_i^2\right)\!(X^TX)^{-1} \nonumber \\ &\approx \begin{bmatrix}0.987 & -0.164 & -0.102\\ -0.164 & 0.034 & 0.008\\ -0.102 & 0.008 & 0.026\end{bmatrix} \end{align}\]
where \(\frac{n}{n-k} = \frac{4}{4-3} = 4\). Note how the elements in the diagonal for each variance-covariance matrix represent the variance for the coefficients’ estimators. Of these three matrices we get the highest values in \(var(\hat{B}|X)\). Second highest values are given by \(var(\hat{B}|X)^{\text{HC1}}\), while the lowest variance for the coefficients’ estimators is found in \(var(\hat{B}|X)^{\text{HC0}}\). These are three variants of the variance-covariance matrix among many conceivable ones.
26.9 Chapter Summary
The Gauss-Markov theorem describes the theoretical assumptions that must hold for the least squares method’s estimator to have the smallest sampling variance among all linear unbiased estimators. A common, but not necessary, assumption in regression analysis with the least squares method is that the error terms follow a normal distribution.
It is common that one or several of the assumptions in the Gauss-Markov theorem are not fulfilled. There are different methods and tests to check if the conditions are fulfilled as well as different methods to handle deviations from the conditions.
As an example, in the presence of heteroscedasticity one can use adjusted standard errors (robust standard errors).
Data available at www.measuringworth.org/usgdp/.↩︎