Chapter 25 Statistical Tests in Regression Analysis with Multiple Variables
This chapter describes how we can use statistical tests with regression analysis when we have two or more explanatory variables. For this we use matrices and linear algebra, introduced in chapter 14.
25.1 Starting Points and Assumptions
We start from the regression model written with matrices for a population with unknown characteristics — the population model:
\[\begin{equation} Y = XB + V \tag{25.1} \end{equation}\]
where \(Y\) is an \(n \times 1\) column matrix with the explained variable, \(B\) is a \(k \times 1\) column matrix for all coefficients, and \(V\) is an \(n \times 1\) column matrix with \(n\) error terms. \(X\) is an \(n \times k\) matrix with the explanatory variables in separate columns and the observations in separate rows. The regression model includes the intercept \(b_0\) as the first coefficient, which is why all elements in the first column of matrix \(X\) have the value 1.
Theoretical assumptions:
The population’s regression model describes a linear relationship between \(X\) and \(Y\) that exists in the population.
\(X\) and \(Y\) are random variables from a population whose characteristics are unknown. Explanatory variables can also be the result of a controlled process. Observation \((Y_i, X_{1,i}, \ldots, X_{k,i})\) is independent and identically distributed (i.i.d.).
The explanatory variables in \(X\) have no strong correlation among themselves. None of the variables \(X_j\) can be described as a linear combination of any of the other variables \(X_k\).
The error terms are not correlated with each other: \(cov(v_i v_j \mid X) = 0\) for \(i \neq j\).
The conditional expected value of the error terms is zero: \(E(V \mid X) = 0\), and \(V\) and \(X\) do not covary: \(cov(V, X) = 0\).
The error terms have constant variance \(\sigma_V^2\) across all values of all explanatory variables \(X\), described as:
\[E(VV^T \mid X) = \sigma_V^2 I = \begin{bmatrix} \sigma_V^2 & 0 & \cdots & 0 \\ 0 & \sigma_V^2 & \cdots & 0 \\ \vdots & \vdots & \ddots & 0 \\ 0 & 0 & 0 & \sigma_V^2 \end{bmatrix}\]
where \(V^T\) is the transposed column matrix and \(\sigma_V^2 I\) is an \(n \times n\) diagonal matrix.
- The error terms follow the normal distribution: \(V \mid X \sim N(0,\, \sigma_V^2 I)\).
25.2 Estimators of the Coefficients
Based on the least squares method we estimate the coefficients as:
\[\begin{equation} \hat{B} = (X^T X)^{-1} X^T Y \tag{25.2} \end{equation}\]
The expected value of the estimator \(\hat{B}\) is the population model’s coefficients: \(E(\hat{B}) = B\), making the estimator unbiased. We can show this by substituting \(Y = XB + V\):
\[\begin{equation} \begin{aligned} \hat{B} &= (X^T X)^{-1} X^T Y \\ &= (X^T X)^{-1} X^T (XB + V) \\ &= (X^T X)^{-1}(X^T X)B + (X^T X)^{-1} X^T V \\ &= B + (X^T X)^{-1} X^T V \end{aligned} \tag{25.3} \end{equation}\]
where in the fourth line we use \(AA^{-1} = I\) and \(IA = A\). Taking the expected value conditional on \(X\) and using \(E(V \mid X) = 0\):
\[E(\hat{B} \mid X) = B + (X^T X)^{-1} X^T E(V \mid X) = B\]
Given that the assumptions hold, the least squares method gives unbiased estimates: if we were to take an infinite number of samples from the same population, the mean of the estimated coefficients would equal the population values.
To test whether each coefficient in \(B\) is statistically significantly different from zero we set up null and alternative hypotheses \(H_0: b_j = 0\) and \(H_1: b_j \neq 0\), where \(b_j\) is one of the coefficients in \(B\). Given assumption 7 (normal errors), the estimator \(\hat{B}\) follows the t-distribution, and the t-statistic for each coefficient is:
\[t = \frac{\hat{b}_j}{se(\hat{b}_j)}\]
25.3 The Variance-Covariance Matrix
We describe the variance and standard errors for all coefficient estimators using the variance-covariance matrix:
\[var(\hat{B} \mid X) = E\left[(\hat{B} - B)(\hat{B} - B)^T \mid X\right]\]
From equation (25.3) we have \(\hat{B} - B = (X^T X)^{-1} X^T V\). Substituting:
\[\begin{equation} \begin{aligned} var(\hat{B} \mid X) &= E\left[(X^T X)^{-1} X^T V V^T X (X^T X)^{-1} \mid X\right] \\ &= (X^T X)^{-1} X^T E(V V^T \mid X)\, X (X^T X)^{-1} \end{aligned} \tag{25.4} \end{equation}\]
where we factor out \(X\) since the expectation is conditional on \(X\). By assumption 6, \(E(VV^T \mid X) = \sigma_V^2 I\), giving:
\[\begin{equation} var(\hat{B} \mid X) = \sigma_V^2 (X^T X)^{-1} \tag{25.5} \end{equation}\]
Since \(\sigma_V^2\) is unknown we estimate it with \(\hat{s}_V^2 = \sum(y_i - \hat{y}_i)^2 / (n-p)\), yielding the estimated variance-covariance matrix:
\[\begin{equation} var(\hat{B} \mid X) = \hat{s}_V^2 (X^T X)^{-1} = \frac{\sum(y_i - \hat{y}_i)^2}{n - p}\, (X^T X)^{-1} \tag{25.6} \end{equation}\]
Structure for simple regression. For the model \(Y = a + bX + V\) with two coefficients, the estimated variance-covariance matrix is a \(2 \times 2\) matrix:
\[var(\hat{B} \mid X) = \hat{s}_V^2 (X^T X)^{-1} = \begin{bmatrix} var(\hat{a}) & cov(\hat{a}, \hat{b}) \\ cov(\hat{b}, \hat{a}) & var(\hat{b}) \end{bmatrix}\]
The diagonal elements are the variances of the estimators. Taking their square roots gives the standard errors needed for the t-tests. The off-diagonal elements are the covariances between estimators. For simple regression, the explicit formulas reduce to those derived in chapter 24:
\[var(\hat{b}) = \frac{\sum(y_i - \hat{y}_i)^2}{(n-p)\,\sum(x_i - \bar{x})^2}, \qquad var(\hat{a}) = var(\hat{b}) \cdot \frac{\sum x_i^2}{n}\]
The variance-covariance matrix illustrates how each explanatory variable affects the results for the statistical tests for all coefficients in the model. Adding or removing a variable from the regression model can change the estimated coefficients and their standard errors.
25.4 T-tests for Multiple Regression
We now have all the tools needed to use statistical tests when we have more than two coefficients. We illustrate with the regression model from chapter 14:
\[\begin{equation} y_i = a + b x_i + c z_i + v_i \tag{25.7} \end{equation}\]
with the four observations in the table below.
| \(i\) | \(y_i\) | \(x_i\) | \(z_i\) |
|---|---|---|---|
| 1 | 3 | 3 | 1 |
| 2 | 2 | 4 | 4 |
| 3 | 5 | 6 | 0 |
| 4 | 4 | 7 | 1 |
The OLS estimates are \(\hat{a} \approx 2.89\), \(\hat{b} \approx 0.28\), \(\hat{c} \approx -0.54\).
We test three null hypotheses: \(H_0: a = 0\), \(H_0: b = 0\), \(H_0: c = 0\)(each against the corresponding two-sided alternative). The t-statistics are:
\[t_{\hat{a}} = \frac{\hat{a}}{se(\hat{a})}, \qquad t_{\hat{b}} = \frac{\hat{b}}{se(\hat{b})}, \qquad t_{\hat{c}} = \frac{\hat{c}}{se(\hat{c})}\]
From the calculations above: \(\sum(y_i - \hat{y}_i)^2 \approx 0.338\), \(p = 3\) parameters, \(n - p = 1\) degree of freedom, so \(\hat{s}_V^2 \approx 0.338\). With \(\bar{x} = 5\), \(\bar{z} = 1.5\):
\[\sum(x_i - \bar{x})^2 = 10, \qquad \sum(z_i - \bar{z})^2 = 9, \qquad \sum(x_i - \bar{x})(z_i - \bar{z}) = -4\]
The standard errors are:
\[se(\hat{b}) = \left(0.338 \cdot \frac{9}{10 \cdot 9 - (-4)^2}\right)^{1/2} = \left(0.338 \cdot \frac{9}{74}\right)^{1/2} \approx 0.203\]
\[se(\hat{c}) = \left(0.338 \cdot \frac{10}{74}\right)^{1/2} \approx 0.214, \qquad se(\hat{a}) \approx 1.22\]
Giving t-statistics:
\[t_{\hat{a}} = \frac{2.89}{1.22} \approx 2.37, \qquad t_{\hat{b}} = \frac{0.28}{0.203} \approx 1.38, \qquad t_{\hat{c}} = \frac{-0.54}{0.214} \approx -2.52\]
With only \(k = n - p = 1\) degree of freedom, the critical value at \(\alpha = 0.05\)(two-sided) is \(t^* \approx 12.7\). None of the t-statistics exceed this critical value, so we reject none of the three null hypotheses. The estimated coefficients could have arisen through a random process.
The R output below verifies these calculations:
## Estimate Std. Error t value Pr(>|t|)
## (Intercept) 2.892 1.220 2.37 0.254
## x 0.284 0.203 1.40 0.395
## z -0.541 0.214 -2.53 0.240## (Intercept) x z
## (Intercept) 1.4883 -0.2328 -0.1598
## x -0.2328 0.0411 0.0183
## z -0.1598 0.0183 0.0457The variance-covariance matrix returned by vcov() shows the variances (diagonal) and covariances (off-diagonal) for the three coefficient estimators. The square root of the diagonal gives the standard errors shown in the coefficient table.
25.5 F-test for the Regression Model
Another commonly used statistical test in connection with regression analysis is the F-test, which tests whether a regression model as a whole can explain variation in \(Y\) better than a simpler model. Consider two models:
\[\begin{align} \text{Model 1 (unrestricted):} & \quad y = b_0 + b_1 x_1 + \cdots + b_k x_k + u \\ \text{Model 2 (restricted):} & \quad y = a + v \tag{25.8} \end{align}\]
We formulate the null and alternative hypotheses:
\[H_0: b_1 = b_2 = \cdots = b_k = 0, \qquad H_1: \text{at least one } b_i \neq 0\]
Let \(q = df_{m2} - df_{m1} = k\) be the number of restrictions (the number of slopes set to zero). The F-test statistic is:
\[\begin{equation} F = \frac{SSR_{m2} - SSR_{m1}}{SSR_{m1}} \cdot \frac{n - k - 1}{q} \tag{25.9} \end{equation}\]
where \(SSR = \sum(y_i - \hat{y}_i)^2\) is the sum of squared residuals. An equivalent formula using \(R^2\) is:
\[\begin{equation} F = \frac{R^2}{1 - R^2} \cdot \frac{n - k - 1}{k} \tag{25.10} \end{equation}\]
The F-statistic follows an \(F(q,\; n-k-1)\) distribution under \(H_0\). We reject \(H_0\) if \(F > F^*_{q,\; n-k-1}\) at the chosen significance level \(\alpha\).
Partial F-test. We can also compare two nested models where the restricted model is not the intercept-only model. If we drop \(q\) variables from the unrestricted model (\(k\) variables, \(R^2_U\)) to get the restricted model (\(R^2_R\)):
\[\begin{equation} F = \frac{(R^2_U - R^2_R)/q}{(1 - R^2_U)/(n - k - 1)} \tag{25.11} \end{equation}\]
Numerical example. Using the same four observations and the model \(y_i = a + bx_i + cz_i + v_i\):
- Unrestricted m1: \(SSR_{m1} \approx 0.338\)
- Restricted m2 (\(y = a + v\), i.e., \(\bar{y} = 3.5\)): \(SSR_{m2} = \sum(y_i - \bar{y})^2 = 5\)
With \(n = 4\), \(k_{m1} = 2\), \(q = 2\):
\[F = \frac{5 - 0.338}{0.338} \cdot \frac{4 - 2 - 1}{2} = \frac{4.662}{0.338} \cdot \frac{1}{2} \approx 6.9\]
The critical value \(F^*(2, 1)\) at \(\alpha = 0.05\) is approximately 199. Our \(F = 6.9 < 199\), so we do not reject \(H_0\). With only 1 degree of freedom in the denominator, the critical value is extremely large and almost nothing is significant — a consequence of having so few observations.
25.6 Chapter Summary
The population model written with matrices \(Y = XB + V\) describes a linear relationship between the explained \(Y\) and the explanatory variables in \(X\). Given that the theoretical assumptions hold, the OLS estimator \(\hat{B} = (X^T X)^{-1} X^T Y\) is unbiased: \(E(\hat{B}) = B\).
For each coefficient in \(B\) we can perform a two-sided t-test: \(H_0: b_j = 0\), \(H_1: b_j \neq 0\). The t-statistic is \(t = \hat{b}_j / se(\hat{b}_j)\) where the standard errors come from the diagonal of the variance-covariance matrix.
The estimated variance-covariance matrix is \(var(\hat{B} \mid X) = \hat{s}_V^2 (X^T X)^{-1}\). The diagonal elements are the variances of the individual coefficient estimators; the off-diagonal elements are covariances between estimators. Including or excluding variables from the model affects both the coefficient estimates and the statistical tests for all coefficients.
The F-test tests whether the regression model as a whole — all slope coefficients jointly — is statistically significantly different from zero. Under \(H_0: b_1 = \cdots = b_k = 0\) the F-statistic follows an \(F(k, n-k-1)\) distribution.
Statistical power applies to multiple regression as well. With a small sample, a larger estimated deviation is required to reject the null hypothesis.