Gauss–Markov theorem
Gauss–Markov theorem
In statistics, the Gauss–Markov theorem states that in a linear regression model in which the errors are uncorrelated, have equal variances and expectation value of zero, the best linear unbiased estimator (BLUE) of the coefficients is given by the ordinary least squares (OLS) estimator, provided it exists. Here "best" means giving the lowest variance of the estimate, as compared to other unbiased, linear estimators. The errors do not need to be normal, nor do they need to be independent and identically distributed (only uncorrelated with mean zero and homoscedastic with finite variance). The requirement that the estimator be unbiased cannot be dropped, since biased estimators exist with lower variance. See, for example, the James–Stein estimator (which also drops linearity) or ridge regression.
The theorem was named after Carl Friedrich Gauss and Andrey Markov.
Statement
Suppose we have in matrix notation,
expanding to,
They have mean zero:
They are homoscedastic, that is all have the same finite variance: for all and
Distinct error terms are uncorrelated:
![{\displaystyle \operatorname {E} \left[{\widehat {\beta }}_{j}\right]=\beta _{j}}](https://wikimedia.org/api/rest_v1/media/math/render/svg/51967081e42dea692d45a7331ec58e7a29acf3d5)
![{\displaystyle \operatorname {E} \left[\left(\sum _{j=1}^{K}\lambda _{j}\left({\widehat {\beta }}_{j}-\beta _{j}\right)\right)^{2}\right],}](https://wikimedia.org/api/rest_v1/media/math/render/svg/fae4692be86723675447fb34706028914ec2cddb)
The ordinary least squares estimator (OLS) is the function
Proof
![{\displaystyle {\begin{aligned}\operatorname {E} \left[{\tilde {\beta }}\right]&=\operatorname {E} [Cy]\\&=\operatorname {E} \left[\left((X'X)^{-1}X'+D\right)(X\beta +\varepsilon )\right]\\&=\left((X'X)^{-1}X'+D\right)X\beta +\left((X'X)^{-1}X'+D\right)\operatorname {E} [\varepsilon ]\\&=\left((X'X)^{-1}X'+D\right)X\beta &&\operatorname {E} [\varepsilon ]=0\\&=(X'X)^{-1}X'X\beta +DX\beta \\&=(I_{K}+DX)\beta .\\\end{aligned}}}](https://wikimedia.org/api/rest_v1/media/math/render/svg/7339087b0805075d1bbb86e6277fdf53c35d6825)
Remarks on the proof
Generalized least squares estimator
Gauss–Markov theorem as stated in econometrics
Linearity
Data transformations are often used to convert an equation into a linear form. For example, the Cobb–Douglas function—often used in economics—is nonlinear:
But it can be expressed in linear form by taking the natural logarithm of both sides:[4]
This assumption also covers specification issues: assuming that the proper functional form has been selected and there are no omitted variables.
One should be aware, however, that the parameters that minimize the residuals of the transformed equation not necessarily minimize the residuals of the original equation.
Strict exogeneity
![{\displaystyle \operatorname {E} [\,\varepsilon _{i}\mid \mathbf {X} ]=\operatorname {E} [\,\varepsilon _{i}\mid \mathbf {x_{1}} ,\dots ,\mathbf {x_{n}} ]=0.}](https://wikimedia.org/api/rest_v1/media/math/render/svg/8f97023f52a55c0257fc8373f0d4c49db863fe76)
![{\displaystyle \operatorname {E} [\,\mathbf {x} _{j}\cdot \varepsilon _{i}\,]={\begin{bmatrix}\operatorname {E} [\,{x}_{j1}\cdot \varepsilon _{i}\,]\\\operatorname {E} [\,{x}_{j2}\cdot \varepsilon _{i}\,]\\\vdots \\\operatorname {E} [\,{x}_{jk}\cdot \varepsilon _{i}\,]\end{bmatrix}}=\mathbf {0} \quad {\text{for all }}i,j\in n}](https://wikimedia.org/api/rest_v1/media/math/render/svg/09401fc11433c3ecccdda992a223aa3769103861)
This assumption is violated if the explanatory variables are stochastic, for instance when they are measured with error, or are endogenous.[6] Endogeneity can be the result of simultaneity, where causality flows back and forth between both the dependent and independent variable. Instrumental variable techniques are commonly used to address this problem.
Full rank
A violation of this assumption is perfect multicollinearity, i.e. some explanatory variables are linearly dependent. One scenario in which this will occur is called "dummy variable trap," when a base dummy variable is not omitted resulting in perfect correlation between the dummy variables and the constant term.[7]
Multicollinearity (as long as it is not "perfect") can be present resulting in a less efficient, but still unbiased estimate. The estimates will be less precise and highly sensitive to particular sets of data.[8] Multicollinearity can be detected from condition number or the variance inflation factor, among other tests.
Spherical errors
The outer product of the error vector must be spherical.
![{\displaystyle \operatorname {E} [\,{\boldsymbol {\varepsilon }}{\boldsymbol {\varepsilon ^{\mathsf {T}}}}\mid \mathbf {X} ]=\operatorname {Var} [\,{\boldsymbol {\varepsilon }}\mid \mathbf {X} ]={\begin{bmatrix}\sigma ^{2}&0&\dots &0\\0&\sigma ^{2}&\dots &0\\\vdots &\vdots &\ddots &\vdots \\0&0&\dots &\sigma ^{2}\end{bmatrix}}=\sigma ^{2}\mathbf {I} \quad {\text{with }}\sigma ^{2}>0}](https://wikimedia.org/api/rest_v1/media/math/render/svg/a9308256c29e9781180504255baf90222d0fc08e)
![{\displaystyle \operatorname {Var} [\,{\boldsymbol {\varepsilon }}\mid \mathbf {X} ]=\sigma ^{2}\mathbf {I} }](https://wikimedia.org/api/rest_v1/media/math/render/svg/5d2080a7e02e8a8047535900767aa813fb9f1a90)
Heteroskedasticity occurs when the amount of error is correlated with an independent variable. For example, in a regression on food expenditure and income, the error is correlated with income. Low income people generally spend a similar amount on food, while high income people may spend a very large amount or as little as low income people spend. Heteroskedastic can also be caused by changes in measurement practices. For example, as statistical offices improve their data, measurement error decreases, so the error term declines over time.
This assumption is violated when there is autocorrelation. Autocorrelation can be visualized on a data plot when a given observation is more likely to lie above a fitted line if adjacent observations also lie above the fitted regression line. Autocorrelation is common in time series data where a data series may experience "inertia."[11] If a dependent variable takes a while to fully absorb a shock. Spatial autocorrelation can also occur geographic areas are likely to have similar errors. Autocorrelation may be the result of misspecification such as choosing the wrong functional form. In these cases, correcting the specification is one possible way to deal with autocorrelation.
In the presence of spherical errors, the generalized least squares estimator can be shown to be BLUE.[12]
See also
Linear regression
Measurement uncertainty
Other unbiased statistics
Best linear unbiased prediction (BLUP)
Minimum-variance unbiased estimator (MVUE)