> [!definition|*] Time Invariant Linear System > A map $\mathcal{L}: x(t) \mapsto y(t)$ is a **time-invariant linear system** if it is time invariant (i.e. $t$ only indirectly affects $x(t) \mapsto y(t)$ via $x(t)$) and linear, i.e. $\forall x_{1},x_{2}:\mathcal{L}(\lambda x_{1}+(1-\lambda)x_{2})=\lambda \mathcal{L}x_{1}+(1-\lambda)\mathcal{L}x_{2}.$ - For example, $y_{t}=2x_{t}$ and $y_{t}=y_{t-1}+x_{t}$ are linear, but $y_{t}=x_{t}^{2}$ is not. - Note that $y_{t}=2x_{t}+1$ is not linear but can be made into one with an **affine transformation** $z_{t}:= y_{t}-1=2x_{t}$. A more general example arises from the **difference equations** (in discrete time): $\left( \sum_{k=0}^{\infty} \alpha_{k}\nabla^{k} \right) y_{t}=\left( \sum_{k=0}^{\infty} \beta_{k}\nabla^{k} \right) x_{t},$where $\nabla$ is the differencing operator (so $\nabla y_{t}=y_{t}-y_{t-1}$). It can be transformed into the form $y_{t}=a_{1}y_{t-1}+a_{2}y_{t-2}+\dots+b_{0}x_{t}+b_{1}x_{t-1}+\cdots,$either be recursively solving for $y$ or by inverting the polynomial of $B=1-\nabla$ on the $\mathrm{LHS}$. ## Impulse Response Function > [!definition|*] Impulse Response Function > A linear system can in general be written as $y(t)=\int h(u)x(t-u) ~ du=h *x,$where $*$ is the convolution. Here $h(u)$ is the contribution of $x$ at lag $u$, called the **impulse response function**. A system is **physically realizable** or **causable** if the impulse response function $h$ is $0$ for any negative input (i.e. future inputs $x(t-u)$ cannot affect the present $y(t)$). It is **stable** if $\int | h(u) | ~ du < \infty$. Stable systems have good properties like the existence of Fourier transformations. Common examples include: - The identity map $y(t)=x(t)$ has the IRF $h(u)=\delta_{0}(u)$. In general, a lagged response $y(t)=x(t-\tau)$ has $h(u)=\delta_{\tau}(u)$. - Discrete time analogues are the indicator functions $\mathbf{1}_{u=\tau}$. - One important class is the **delayed exponentials**, given by $h(u)=\frac{g}{T}\cdot\exp(-(u-\tau) / T) \cdot\mathbf{1}_{u>\tau},$where the input at $t-\tau$ has the largest contribution, and the effect decays exponentially for earlier inputs. Here $g$ controls the gain, and $T$ controls the speed of decay. ## Frequency Domain As do all convolutions, the impulse response function can be studied with its [[Continuous Fourier Transform|Fourier transformation]]: > [!definition|*] Frequency Response Function > The **frequency response function** is $H(\omega)= \mathcal{F}h:=\int h(u)\exp(-i\omega u) ~ du ,$where $\mathcal{F}$ is the Fourier transformation operator. Then a [[Continuous Fourier Transform#^64adae|the convolution theorem]] gives $\mathcal{F}y=\mathcal{F}(h*x)=\mathcal{F}h\cdot \mathcal{F}x=H\cdot \mathcal{F}x,$where $\cdot$ is pointwise multiplication. - Therefore, if the input $x(t)$ is a sinusoid $x(t)=\exp(i\omega t)$ in steady state (i.e. applied at $t=-\infty$), then $\mathcal{F}x=x$, and output has the neat form of $y=\exp(i\omega t+\phi(\omega)),$where $\phi(\omega)$ is a phase-shift. ### Gain and Phase > [!definition|*] Gain and Phase > The FRF $H(\omega)$ can be written in the **gain-phase form** as $H(\omega)=G(\omega)e^{i\phi(\omega)},$where $G \in \mathbb{R}$ is the **gain**, and $\phi(\omega)\in \mathbb{R}$ is the **phase**. - Usually we assume $G>0$, but that may cause $\phi$ to be undetermined up to $\pm {2}k\pi$. We may impose $\phi \in (0, 2\pi)$ or $(-\pi, \pi)$, but that would cause discontinuities. which corresponds to the gain and phase-spectrum decomposition of [[Cross-Spectral Analysis#The Cross-Spectrum|the cross-spectrum]] between the input $X=(x(t))$ and output $Y=(y(t))$. Plotting $G,\phi$ against $\omega$ gives the **gain diagram** and **phase diagram**. ![[GainDiagrams.png#invert]] > [!theorem|*] Auto-covariance of the Response > Furthermore, *in the absence of noise*, we can compute the [[Spectral Distribution Function|spectral distribution function]] of the response as $f_{Y}(\omega)=G(\omega)^{2}f_{X}(\omega).$ ^587dfa ## System Identification In practice, difficultie[](Spectral%20Distribution%20Function.md)ht be noise $N(t)$ injected into the system, so $Y(t)=N(t)+\mathcal{L}X(t),$where the noise has mean $0$ and is independent of $X$. - The system might have feedback (i.e. $Y$ affects $X$, like how ad spending increases sales, which in turn encourages more spending). - We might have access to observed data only, so we cannot tweak the input (e.g. by passing a step function to test the response). This leads to two approaches of modeling the system: - Finding the [[Cross-Spectral Analysis#^5c77e3|cross-spectrum]] $f_{XY}$ and [[Spectral Distribution Function|spectral density function]] $f_{X},f_{Y}$ in the frequency domain, solve for the frequency response function $H$, then using inverse Fourier transform to recover the impulse response function $h$ if necessary. - Alternatively, model the system in the time domain and identify $h$ directly. ## Cross-Covariance and Frequency-Domain Modeling For simplicity, assume $\mathbb{E}[X(t)]=0$, so $\mathbb{E}[Y(t)]=0$ as well. Writing $Y$ in terms of the impulse response function $h$, we have $Y(t)=\int h(u)X(t-u) ~ du + n(t)= h * X +n(t).$The integral is over $\mathbb{R}^{+}$ for physically realizable systems, and $\mathbb{R}$ in general. Now multiply by $X(t-\tau)$ and take expectation to get $\begin{align*} \gamma_{XY}(\tau)&= \mathbb{E}[X(t-\tau)\cdot(h * X)(t)] + \mathbb{E}[N(t)X(t-\tau)]\\ &= h * \mathbb{E}[X(t-\tau)\cdot X(t)] + 0\\ &= h * \gamma_{X}(\tau). \end{align*}$ This is difficult to solve for $h$ in the time domain, so the Fourier transformation gives the simple equality in frequency domain: $f_{XY}(\omega)=H(\omega) \cdot f_{X}(\omega).$Since this holds even with the presence of noise, we can estimate $H$ by plugging in estimates $\hat{f}_{XY},\hat{f}_{X}$. In practice, we can directly estimate the gain and phase as $\begin{align*} \hat{G}&= | \hat{f}_{XY} | / \hat{f}_{X},\\ \hat{\phi}&= \arctan (-\hat{q} / \hat{c})=\arctan\left(- \frac{\mathrm{Im}~f_{XY}}{\mathrm{Re}~f_{XY}} \right). \end{align*}$ ### Covariance of the Noise In presence of noise, the [[#^587dfa|equaltion of auto-covariance of the resposne]] becomes $\begin{align*} f_{Y}(\omega)&= G(\omega)^{2}f_{X}(\omega)+f_{N}(\omega)\\ &= Cf_{Y} + f_{N}. \end{align*}$Therefore, we can solve for $f_{N}$ to get $f_{N}=\left( 1-C\right)f_{Y},$where $C$ is the [[Cross-Spectral Analysis#^232d53|coherence]]. From this, we see that when there is no noise, so $f_{N}=0$, we must have $C=1$. So in some sense, *the coherence measures the amount of variance in $Y$ explained by a linear relationship with $X$*. ## Time-Domain Modeling That directly estimating each $h(t)$ is not realistic even for discrete series, since the variance of the estimates are inflated by the autocovariance within each series. Alternatively, Box and Jenkins proposed an approach to model the system in time domain directly: - [[Prewhitening of Time Series|Prewhiten]] the input $X$ with some appropriate ARIMA model: $\phi(B)X_{t}=\theta(B)\alpha_{t},$where $\alpha$ is a [[Purely Random Processes|PRP]]. Therefore $\theta(B)^{-1}\phi(B)X_{t}=\alpha_{t}$ - Apply the same transformation to $Y_{t}$ to get $\beta_{t}:= \theta(B)^{-1}\phi(B)Y_{t},$another discrete time series. - Now study the cross-covariance between $\alpha$ and $\beta$, which gives better estimates of $h$ because $\alpha$ has less autocovariance. Note that assuming a linear system with added error, we have $\begin{align*} \beta_{t}&= \theta^{-1}(B)\phi(B)Y_{t}\\ &= \theta^{-1}(B)\phi(B)[h(B)X_{t}+N_{t}]\\ &= h(B)\alpha_{t}+\theta^{-1}(B)\phi(B)N_{t}. \end{align*}$Therefore, assuming $N$ is independent of $\alpha$, multiply both sides with $\alpha_{t-k}$ and take expectation to get $\gamma_{\alpha \beta}(k)=h_{k}\gamma_{\alpha}(k)=h_{k}\mathrm{Var}(\alpha),$since $\alpha$ is a [[Purely Random Processes|PRP]]. A simple substitution gives the estimator $\hat{h}_{k}=c_{\alpha \beta}(k) / s^{2}_{\alpha}.$where $c_{\alpha \beta}$ is the sample cross variance observed.