## Fitting AR Processes ### Fitting Parameters Given the order $p$ of an $\mathrm{AR}(p)$ process, it remains to determine the coefficients $\mu,\alpha_{1},\dots,\alpha_{p}$ in the model $X_{t}-\mu=Z_{t} + \sum_{i=1}^{p}\alpha_{i}B^{i}(X_{t}-\mu)$ One method of estimating $\alpha_{i}$ is minimizing the RSS in the time series: $\hat{\mu},\underline{\hat{\alpha}}=\mathrm{arg}\min_{\mu,\underline{\alpha}}\sum_{t=p+1}^{N} \left[x_{t} - \mu - \sum_{i=1}^{p}\alpha_{i}B^{i}(x_{t}-\mu) \right]$which also gives the MLE if the error $Z_{t}$ is normal. By substituting $\hat{\mu}=\bar{x}$ and some other approximations, an low-order AR process has $\begin{align*} \mathrm{AR}(1): &\hphantom{\begin{cases}1\end{cases}}\hat{\alpha}_{1}\approx r_{1}\\[0.4em] \mathrm{AR}(2): &\begin{cases} \hat{\alpha}_{1} \approx \frac{r_{1}(1-r_{2})}{1-r_{1}^{2}}\\[0.2em] \hat{\alpha}_{2} \approx \frac{r_{2}-r_{1}^{2}}{1-r_{1}^{2}} \end{cases} \end{align*}$where in $\mathrm{AR}(2)$, the $\hat{\alpha}_{2}$ is the amount of autocorrelation not accounted for by applying the 1-step correlation twice. Alternatively, starting by approximating $\hat{\mu}=\bar{x}$ allows standard least squares algorithms to find $\underline{\hat{\alpha}}=\mathrm{arg}\min_{\underline{\alpha}}\sum_{t=p+1}^{N} \left[x_{t} -\bar{x} - \sum_{i=1}^{p}\alpha_{i}B^{i}(x_{t}-\bar{x}) \right]$ Another method is to use the [[Autoregressive Processes#Yule-Walker Equations|Yule-Walker equations]] to solve for $\underline{\hat{\alpha}}$, substituting $\rho(k)$ with $r_{k}$: $\begin{bmatrix} r_{1} \\ \vdots \\ r_{p} \end{bmatrix} = \begin{bmatrix} r_{0} & \cdots & r_{p-1} \\ \vdots & \ddots & \vdots \\ r_{p-1} & \cdots & r_{0} \end{bmatrix}\begin{bmatrix} \alpha_{1} \\ \vdots \\ \alpha_{p} \end{bmatrix}$ ### Fitting the Order $p$ The order of an AR process is a lot harder to fit than its parameters, as the ac.f. of high-order AR processes are hard to identify. One approach is to fit models $p=1,\dots$, and choose the $p$ where higher orders do not offer much improvement in the RSS. Another approach uses the **partial autocorrelation function** $\pi_{p}$, the $p$th coefficient of the order-$p$ AR process. It is the amount of autocorrelations not accounted for by lower-order models. The order of the AR process is then the minimal $p$ for which all $\pi_{k > p}$ are non-significant. ### Implementation in R ```R fold # simulate data ar_params <- list(order = c(2,0,0), ar = c(0.7,-0.4)) ar_data <- arima.sim(n = 500, ar_params) # plotting dev.off() par(mfrow=c(2,1), mai=c(1,4,1,1)/4, oma=c(2,0,0,0), bg=NA) plot(ar_data, type="l", ylab="Series Value", main="AR(2) Process") acf(ar_data, ylab="Partial ACF.", main="", type="partial") ``` ![[AR2PACF.png#invert]] ```R file:"Fitting AR processes (given order)" ar_fit <- arima(ar_data, order=c(2,0,0)) print(ar_fit) ``` ## Fitting MA Processes ### Fitting Parameters The parameters of MA processes are much harder to fit, as the RSS cannot be written in closed form as a function of parameters. One alternative is to numerically compute the RSS for different choices of $\hat{\mu}$ and $\underline{\hat{\beta}}$ (usually a grid), then finding the optimal choice that minimize the RSS. - Modern algorithms use iterative optimization procedures (e.g. hill-climbing) instead of covering the whole grid. Because of these difficulties, its common to use AR models, even when they need more parameters than MA models to describe the process. ### Fitting the Order $p$ Since the ac.f. of an $\mathrm{MA}(p)$ process "cuts off" for lags $k>p$, straightforward examinations suffices for determining the order. ### Implementation in R ```R fold # simulate data ma_params <- list(order = c(0,0,2), ma = c(0.7,-0.4)) ma_data <- arima.sim(n = 500, ma_params) # plotting dev.off() par(mfrow=c(2,1), mai=c(1,4,1,1)/4, oma=c(2,0,0,0), bg=NA) plot(ma_data, type="l", ylab="Series Value", main="MA(2) Process") acf(ma_data, ylab="Autocorr.", main="") ``` ![[MA2ACF2.png#invert]] And by inspection we see that the cut-off is at lag $k=2$, suggesting an $\mathrm{MA}(2)$ process. ```R file:"Fitting AR processes (given order)" ma_fit <- arima(ma_data, order=c(0,0,2)) print(ma_fit) ```