> [!tldr] Variance Inflation Factor > In the OLS model $\mathbf{y} \sim \mathbf{X}$ where $\mathbf{X}$ is the design matrix (including the intercept), the **variance inflation factor (VIF)** of the $j$th covariate $\mathbf{x}_{j}$ is $\mathrm{VIF}_{j}:= \frac{1}{1-R_{j}^{2}},$where $R_{j}^{2}$ is the [[Pearson's R2|Pearson's $R^{2}$]] score of regressing $\mathbf{x}_{j} \sim \mathbf{X}_{-j}$ (the design matrix without the $j$th covariate). Therefore, if the $\mathrm{VIF}$ is large if and only if $\mathbf{x}_{j}$ does not carry much of information new to $\mathbf{X}_{-j}$. ### Motivation of the Name VIF The $\mathrm{VIF}$ appears in the following identity under the assumption that $Y=f(X)+\epsilon$ for some general $f$ (not necessarily linear) and independent error $\epsilon \sim [0, \sigma^{2}]$: > [!theorem|*] Variance Inflation > If $\tilde{\beta}_{j}$ is the coefficient of $\mathbf{x}_{j}$ in the OLS $\mathbf{y} \sim 1+\mathbf{x}_j$, and $\hat{\beta}_{j}$ that of the full OLS $\mathbf{y} \sim \mathbf{X}$, $\mathrm{Var}(\hat{\beta}_{j})=\mathrm{VIF}_{j}\cdot \mathrm{Var}(\tilde{\beta}_{j}).$In particular, $\mathrm{Var}(\hat{\beta}_{j})=\underbrace{\frac{1}{1-R_{j}^{2}}}_{\mathrm{VIF}}\cdot \underbrace{\frac{\sigma^{2}}{\| \mathbf{x}_{j}-\bar{x}_{j} \| ^{2}}}_{\mathrm{Var}(\tilde{\beta}_{j})}.$ > That is, *introducing the covariates $\mathbf{X}_{-j}$ inflates the variance of $\mathbf{x}_{j}s coefficient by $\mathrm{VIF}_{j}$.* > > > [!proof] > > By the [[Orthogonal Projection, Confounding, and Missing Variable in OLS#^4760b7|FWL theorem]], $\hat{\beta}_{j}$ equals the coefficient from the OLS $\mathbf{y} \sim \mathbf{\tilde{x}}_{j}$, where $\mathbf{\tilde{x}}_{j}$ are the residuals of OLS models $\mathbf{x}_{j}\sim \mathbf{X}_{-j}$. Therefore, it equals $\hat{\beta}_{j}=\frac{\mathbf{y} \cdot \tilde{\mathbf{x}}_{j}}{\| \mathbf{\tilde{x}} _{j}\|^{2} },$with variance $\mathrm{Var}(\hat{\beta}_{j})= \frac{1}{\| \mathbf{\tilde{x}}_{j} \|^{4} }\tilde{\mathbf{x}}_{j}^{T}\mathrm{Cov}(\mathbf{y})\tilde{\mathbf{x}}_{j}= \frac{\sigma^{2} }{\|\mathbf{\tilde{x}}_{j} \|^{2} }.$ > > But the denominator is just $\mathrm{RSS}$ from $\mathbf{x}_{j} \sim \mathbf{X}_{-j}$, which equals $(1-R^{2}_{j})\mathrm{TSS}$, where $\mathrm{TSS}=\| \mathbf{x}_{j}-\bar{x}_{j} \|^{2}$ is from the same regression. Now plug this into the denominator gives the theorem.