## Fitting AR Processes
### Fitting Parameters
Given the order $p$ of an $\mathrm{AR}(p)$ process, it remains to determine the coefficients $\mu,\alpha_{1},\dots,\alpha_{p}$ in the model $X_{t}-\mu=Z_{t} + \sum_{i=1}^{p}\alpha_{i}B^{i}(X_{t}-\mu)$
One method of estimating $\alpha_{i}$ is minimizing the RSS in the time series: $\hat{\mu},\underline{\hat{\alpha}}=\mathrm{arg}\min_{\mu,\underline{\alpha}}\sum_{t=p+1}^{N} \left[x_{t} - \mu - \sum_{i=1}^{p}\alpha_{i}B^{i}(x_{t}-\mu) \right]$which also gives the MLE if the error $Z_{t}$ is normal. By substituting $\hat{\mu}=\bar{x}$ and some other approximations, an low-order AR process has $\begin{align*}
\mathrm{AR}(1): &\hphantom{\begin{cases}1\end{cases}}\hat{\alpha}_{1}\approx r_{1}\\[0.4em]
\mathrm{AR}(2): &\begin{cases}
\hat{\alpha}_{1} \approx \frac{r_{1}(1-r_{2})}{1-r_{1}^{2}}\\[0.2em]
\hat{\alpha}_{2} \approx \frac{r_{2}-r_{1}^{2}}{1-r_{1}^{2}}
\end{cases}
\end{align*}$where in $\mathrm{AR}(2)$, the $\hat{\alpha}_{2}$ is the amount of autocorrelation not accounted for by applying the 1-step correlation twice.
Alternatively, starting by approximating $\hat{\mu}=\bar{x}$ allows standard least squares algorithms to find $\underline{\hat{\alpha}}=\mathrm{arg}\min_{\underline{\alpha}}\sum_{t=p+1}^{N} \left[x_{t} -\bar{x} - \sum_{i=1}^{p}\alpha_{i}B^{i}(x_{t}-\bar{x}) \right]$
Another method is to use the [[Autoregressive Processes#Yule-Walker Equations|Yule-Walker equations]] to solve for $\underline{\hat{\alpha}}$, substituting $\rho(k)$ with $r_{k}$: $\begin{bmatrix}
r_{1} \\ \vdots \\ r_{p}
\end{bmatrix} = \begin{bmatrix}
r_{0} & \cdots & r_{p-1} \\
\vdots & \ddots & \vdots \\
r_{p-1} & \cdots & r_{0}
\end{bmatrix}\begin{bmatrix}
\alpha_{1} \\ \vdots \\ \alpha_{p}
\end{bmatrix}$
### Fitting the Order $p$
The order of an AR process is a lot harder to fit than its parameters, as the ac.f. of high-order AR processes are hard to identify.
One approach is to fit models $p=1,\dots$, and choose the $p$ where higher orders do not offer much improvement in the RSS.
Another approach uses the **partial autocorrelation function** $\pi_{p}$, the $p$th coefficient of the order-$p$ AR process. It is the amount of autocorrelations not accounted for by lower-order models. The order of the AR process is then the minimal $p$ for which all $\pi_{k > p}$ are non-significant.
### Implementation in R
```R fold
# simulate data
ar_params <- list(order = c(2,0,0),
ar = c(0.7,-0.4))
ar_data <- arima.sim(n = 500, ar_params)
# plotting
dev.off()
par(mfrow=c(2,1),
mai=c(1,4,1,1)/4,
oma=c(2,0,0,0), bg=NA)
plot(ar_data, type="l",
ylab="Series Value",
main="AR(2) Process")
acf(ar_data, ylab="Partial ACF.",
main="", type="partial")
```
![[AR2PACF.png#invert]]
```R file:"Fitting AR processes (given order)"
ar_fit <- arima(ar_data, order=c(2,0,0))
print(ar_fit)
```
## Fitting MA Processes
### Fitting Parameters
The parameters of MA processes are much harder to fit, as the RSS cannot be written in closed form as a function of parameters.
One alternative is to numerically compute the RSS for different choices of $\hat{\mu}$ and $\underline{\hat{\beta}}$ (usually a grid), then finding the optimal choice that minimize the RSS.
- Modern algorithms use iterative optimization procedures (e.g. hill-climbing) instead of covering the whole grid.
Because of these difficulties, its common to use AR models, even when they need more parameters than MA models to describe the process.
### Fitting the Order $p$
Since the ac.f. of an $\mathrm{MA}(p)$ process "cuts off" for lags $k>p$, straightforward examinations suffices for determining the order.
### Implementation in R
```R fold
# simulate data
ma_params <- list(order = c(0,0,2),
ma = c(0.7,-0.4))
ma_data <- arima.sim(n = 500, ma_params)
# plotting
dev.off()
par(mfrow=c(2,1),
mai=c(1,4,1,1)/4,
oma=c(2,0,0,0), bg=NA)
plot(ma_data, type="l",
ylab="Series Value",
main="MA(2) Process")
acf(ma_data, ylab="Autocorr.", main="")
```
![[MA2ACF2.png#invert]]
And by inspection we see that the cut-off is at lag $k=2$, suggesting an $\mathrm{MA}(2)$ process.
```R file:"Fitting AR processes (given order)"
ma_fit <- arima(ma_data, order=c(0,0,2))
print(ma_fit)
```