STA457: Time Series Analysis
Lecture 6
Lijia Wang
Department of Statistical Sciences
University of Toronto
Lijia Wang (UofT) STA457: Time Series Analysis 1 / 47
Overview
Last Time:
1 Definitions of stationary
2 Estimation of correlation
3 Large sample properties of sample statistics
Today:
1 Autoregressive (AR) process
2 Moving average (MA) process
3 Autoregressive moving average (ARMA)
Lijia Wang (UofT) STA457: Time Series Analysis 2 / 47
Outline
1 Autoregressive (AR) process
AR(p) Model
Backshift Operator
Explosiveness and causality
2 Moving Average Models
MA(q) model
Invertibility and causality
3 Autoregressive Moving Average Models
ARMA(p,q)
Causality of a ARMA(p,q) process
Lijia Wang (UofT) STA457: Time Series Analysis 3 / 47
Contents
Over the next few weeks, we will learn the following models:
1 Autoregressive (AR)
2 Moving average (MA)
3 Autoregressive moving average (ARMA)
4 Autoregressive integrated moving average (ARIMA)
Lijia Wang (UofT) STA457: Time Series Analysis 4 / 47
Introduction to Autoregressive Models
Autoregressive models are based on the idea that the current value of the
series, xt , can be explained as a function of p past values,
xt→1 , xt→2 , · · · , xt→p , where p determines the number of steps into the
past needed to forecast the current value.
For an example,
xt = xt→1 → 0.9xt→2 + wt
where wt ↑ N(0, 1). This is the autoregressive model with order 2.
Lijia Wang (UofT) STA457: Time Series Analysis 5 / 47
AR(p) Model
Definition: An autoregressive model of order p, abbreviated AR(p), is of
the form
xt = ω1 xt→1 + ω2 xt→2 + · · · + ωp xt→p + wt
! 2
"
where xt is stationary, wt ↑ wn 0, εw and ω1 , ω2 , · · · , ωp are constants
(ωp ↓= 0). We have E (xt ) = 0.
Lijia Wang (UofT) STA457: Time Series Analysis 6 / 47
AR(p) Model with nonzero mean
Definition: If the mean, µ, of xt is not zero, replace xt by xt → µ. We
can get
xt → µ = ω1 (xt→1 → µ) + ω2 (xt→2 → µ) + · · · + ωp (xt→p → µ) + wt ,
or
xt = ϑ + ω1 xt→1 + ω2 xt→2 + · · · + ωp xt→p + wt ,
where
ϑ = µ (1 → ω1 → ω2 → · · · → ωp ) .
Lijia Wang (UofT) STA457: Time Series Analysis 7 / 47
Backshift Operator
We define the backshift operator by
Bxt = xt→1
and extend it to powers
B 2 xt = B (Bxt ) = Bxt→1 = xt→2 ,
and so on. Thus,
B k xt = xt→k
Lijia Wang (UofT) STA457: Time Series Analysis 8 / 47
AR(p) Model Backshift Operator
Using the backshift operator, we can write AR (p) Model as
! 2 p
"
1 → ω 1 B → ω 2 B → · · · → ω p B xt = w t
↔ ω(B)xt = wt ,
where ! "
2 p
ω(B) = 1 → ω1 B → ω2 B → · · · → ωp B .
Lijia Wang (UofT) STA457: Time Series Analysis 9 / 47
Example 1: AR(1)
Example 1: Let xt = ωxt→1 + wt is AR(1) process, where |ω| < 1. Show
that
#↑ j
1 x =
t j=0 ω wt→j . That is, xt is a linear process.
2 the autocovariance function
εw2 ωh
ϖ(h) = 2
;h ↗ 0
1→ω
3 the autocorrelation function
ϱ(h) = ωh ; h ↗ 0.
4 xt is stationary.
Lijia Wang (UofT) STA457: Time Series Analysis 10 / 47
More on the AR(1) Example
For any AR(1) process, xt = ωxt→1 + wt is , where |ω| < 1, one can show
that,
↑
$
xt = ωj wt→j = ς(B)wt .
j=0
1 This is called the stationary solution of the AR(1) model
2 It expresses xt as convergent infinite summations of shocks (wi ’s),
more specifically past shocks.
3 It also express xt as a linear process.
Lijia Wang (UofT) STA457: Time Series Analysis 11 / 47
Linear Process
Definition: A linear process, xt , is defined to be a linear combination of
white noise variates wt , and is given by
↑
$ ↑
$
xt = µ + ςj wt→j , |ςj | < ↘
j=→↑ j=→↑
For the linear process, we may show that the autocovariance function is
given by
↑
$
ϖx (h) = εw2 ςj+h ςj ,
j=→↑
for h > 0.
Lijia Wang (UofT) STA457: Time Series Analysis 12 / 47
Stationarity
The stationary solution of a time series is the steady-state (equilibrium) of
the original process {xt } that:
is consistent with the original stochastic equation governing the
process;
exhibits time-invariant statistical properties (mean, variance, and
autocovariance);
is independent of initial conditions (e.g., x0 ).
An intuitive understanding of “stationarity”: After a su!ciently long time,
the influence of the initial state of the series becomes negligible, and the
series reaches a “stable” status, where the statistical properties of the
process (like mean, variance, and autocovariance) do not change over time.
Lijia Wang (UofT) STA457: Time Series Analysis 13 / 47
Example 2: AR simulation
Example 2: Generate AR (1) process for ω = 0.9 and ω = →0.9.
1 Plot the generated observations.
2 Draw ACF and partial ACF graphs for the generated series
Lijia Wang (UofT) STA457: Time Series Analysis 14 / 47
Example 2: AR simulation
Lijia Wang (UofT) STA457: Time Series Analysis 15 / 47
Example 2: AR simulation
Lijia Wang (UofT) STA457: Time Series Analysis 16 / 47
Explosive AR Models and Causality
! "
Consider the random walk model xt = xt→1 + wt , where wt ↑ wn 0, εw2 .
1 Show that the autocovariance function
ϖx (s, t) = min{s, t}εw2 .
2 xt is not stationary.
3 Consider AR (1) process xt = ωxt→1 + wt with |ω| > 1. Such
processes are called explosive because the values of the time series
quickly become large in magnitude.
Lijia Wang (UofT) STA457: Time Series Analysis 17 / 47
AR Models: Explosive and Causality
For the explosive AR (1) process xt = ωxt→1 + wt with |ω| > 1, we may
re-write it as
1 1
xt = xt+1 → wt+1 .
ω ω
Because |ω|→1 < 1, this result suggests the stationary future-dependent
AR(1) model
$↑
xt = → ω→1 wt+j
j=1
When a process does not depend on the future, such as the AR(1)
when |ω| < 1, we will say the process is causal.
In the explosive case of this example, the process is stationary, but it
is also future dependent, therefore not causal.
Lijia Wang (UofT) STA457: Time Series Analysis 18 / 47
Causality
Definition: A time series xt is causal if it can be written as a convergent
infinite series of past shocks (or innovations) in the following form:
↑
$
xt = ςj wt→j = ς(B)wt ,
j=0
#↑
where j=0 |ςj | < ↘.
An intuitive understanding of “causality”: The steady status of the series
can be reached from the past values (shocks).
Lijia Wang (UofT) STA457: Time Series Analysis 19 / 47
Summary on stationarity and causality of AR(1) model
For an AR(1) series xt = ωxt→1 + wt , we have:
If |ω| < 1, the series is stationary and causal;
If |ω| = 1, the series becomes a random-walk series and is not
stationary;
If |ω| > 1, the series is stationary but not causal.
Lijia Wang (UofT) STA457: Time Series Analysis 20 / 47
Stationary Solution of AR(p)
The technique of iterating backward works well for AR(1), but not for
larger p. A general technique is matching coe!cients. Since we write
AR(p) of the form
ω(B)xt = wt ,
the stationary solution usually has the form
↑
$
xt = ςj wt→j = ς(B)wt ,
j=0
therefore,
ω(B)ς(B)wt = wt ,
matching the coe!cients yields the solution.
Lijia Wang (UofT) STA457: Time Series Analysis 21 / 47
Stationary Solution of AR(p)
Another way to think about the operations:
xt = ς(B)wt = ω→1 (B)wt .
Since
ς(B) = ω→1 (B) = 1 + ωB + ω2 B 2 + · · · + ωj B j + . . . ,
treating the backshift operator as a complex number z and solving the
polynomial yields
→1 1
ω (z) = = 1 + ωz + ω2 z 2 + · · · + ωj z j + . . . , |z| ≃ 1.
(1 → ωz)
Lijia Wang (UofT) STA457: Time Series Analysis 22 / 47
Outline
1 Autoregressive (AR) process
AR(p) Model
Backshift Operator
Explosiveness and causality
2 Moving Average Models
MA(q) model
Invertibility and causality
3 Autoregressive Moving Average Models
ARMA(p,q)
Causality of a ARMA(p,q) process
Lijia Wang (UofT) STA457: Time Series Analysis 23 / 47
Moving Average Models
Moving average models are based on the idea that the current value of the
series, xt , is the moving average of q past steps of white noise,
wt→1 , wt→2 , . . . , wt→q , where q determines the number of steps into the
past.
AR(p) model: {xt } on the left-hand side of the defining equation are
assumed to be combined linearly;
MA(q) model: {wt } on the right-hand side of the defining equation
are combined linearly.
Lijia Wang (UofT) STA457: Time Series Analysis 24 / 47
MA(q) model
Definition: The moving average model of order q, or MA(q) model, is
defined to be
xt = wt + φ1 wt→1 + φ2 wt→2 + · · · + φq wt→q ,
! 2
"
where wt ↑ wn 0, εw , and φ1 , φ2 , · · · , φq (φq ↓= 0) are parameters.
Lijia Wang (UofT) STA457: Time Series Analysis 25 / 47
MA(q) Model Backshift Operator
The MA(q) model can be written by
xt = φ(B)wt
where φ(B) = 1 + φ1 B + φ2 B 2 + · · · + φq B q that is called the moving
average operator.
Lijia Wang (UofT) STA457: Time Series Analysis 26 / 47
Example 3: MA(1)
Example 3: Consider the MA(1) model xt = wt + φwt→1 . Find
E (xt )
the autocovariance function
the autocorrelation function
Lijia Wang (UofT) STA457: Time Series Analysis 27 / 47
Example 4: MA simulation
Generate MA(1) model for φ = 0.5 and φ = →0.5.
1 Plot the generated observations
2 Draw ACF and partial ACF graphs for the generated series
Lijia Wang (UofT) STA457: Time Series Analysis 28 / 47
Example 4: MA simulation
Lijia Wang (UofT) STA457: Time Series Analysis 29 / 47
MA models: non-uniqueness
For the MA(1) model, xt = wt + φwt→1 , notice that the following models
share the same ACFs:
1
φ= 5 and εw = 5, i.e.
1 i.i.d
xt = wt + wt→1 , wt ↑ N(0, 25)
5
φ = 5 and εw = 1, i.e.
i.i.d
yt = vt + 5vt→1 , vt ↑ N(0, 1)
By mimicking the criterion of causality for AR models, we will choose the
model with an infinite AR representation. Such a process is called an
invertible process.
Lijia Wang (UofT) STA457: Time Series Analysis 30 / 47
MA models: invertibility
To discover which model is the invertible model, we can reverse the roles
of xt and wt (mimicking the AR case),
wt = →φwt→1 + xt .
If |φ| < 1, the MA(1) as wt = →φwt→1 + xt can be written by
↑
$
wt = (→φ)j xt→j = ↼(B)xt ,
j=0
which is the infinite AR representation of the model. Such a process is
called an invertible process.
Lijia Wang (UofT) STA457: Time Series Analysis 31 / 47
Polynomials of MA(q)
Similar to the AR(p) process, we write MA(q) of the form
xt = φ(B)wt ,
the equivalent expression has the form
↼(B)xt = wt ,
we may treat the backshift operator as a complex number z and solving
for the polynomial ↼(z) = φ→1 (z).
Lijia Wang (UofT) STA457: Time Series Analysis 32 / 47
Outline
1 Autoregressive (AR) process
AR(p) Model
Backshift Operator
Explosiveness and causality
2 Moving Average Models
MA(q) model
Invertibility and causality
3 Autoregressive Moving Average Models
ARMA(p,q)
Causality of a ARMA(p,q) process
Lijia Wang (UofT) STA457: Time Series Analysis 33 / 47
ARMA model
We now proceed with the general development of autoregressive, moving
average, and mixed autoregressive moving average (ARMA), models
for stationary time series.
Lijia Wang (UofT) STA457: Time Series Analysis 34 / 47
ARMA(p,q)
Definition: A time series {xt ; t = 0, ±1, ±2, · · · } is ARMA(p, q) if it is
stationary and
xt = ω1 xt→1 + ω2 xt→2 + · · · + ωp xt→p + wt + φ1 wt→1 + φ2 wt→2 + · · · + φq wt→q
with ωp ↓= 0, φq ↓= 0, εw2 > 0. The parameters p and q are called the
autoregressive and the moving average orders, respectively.
The ARMA(p, q) model can then be written in concise form as
ω(B)xt = φ(B)wt .
Lijia Wang (UofT) STA457: Time Series Analysis 35 / 47
ARMA(p,q) nonzero mean
If xt has a nonzero mean µ, we set ϑ = µ (1 → ω1 → ω2 → · · · → ωp ) and
write the model as
xt = ϑ+ω1 xt→1 +ω2 xt→2 +· · ·+ωp xt→p +wt +φ1 wt→1 +φ2 wt→2 +· · ·+φq wt→q ,
! 2
"
where wt ↑ wn 0, εw .
Lijia Wang (UofT) STA457: Time Series Analysis 36 / 47
Parameter Redundancy
Consider a white noise process xt = wt . If we multiply both sides of the
equation by ↽(B) = (1 → 0.5B), then the model becomes
(1 → 0.5B)xt = (1 → 0.5B)wt
or
xt = 0.5xt→1 → 0.5wt→1 + wt
.
This model looks like an ARMA(1,1) model.
In fact, xt is white noise because of the parameter redundancy or
over-parameterization.
Lijia Wang (UofT) STA457: Time Series Analysis 37 / 47
Problems encountered
To summarize, we have seen the following problems:
(i) parameter redundant models,
(ii) stationary AR models that depend on the future, and
(iii) MA models that are not unique.
Lijia Wang (UofT) STA457: Time Series Analysis 38 / 47
polynomials
Definitions: The AR and MA polynomials are defined as
! 2 p
"
AR: ω(z) = 1 → ω1 z → ω2 z → · · · → ωp z , ωp ↓= 0, and
MA: φ(z) = 1 + φ1 z + φ2 z 2 + · · · + φq z q , φq ↓= 0,
respectively, where z is a complex number.
Lijia Wang (UofT) STA457: Time Series Analysis 39 / 47
Problems encountered - Solutions
To summarize, we have seen the following problems:
(i) parameter redundant models,
Solution: we require that ω(z) and φ(z) have no common factors.
(ii) stationary AR models that depend on the future, and
(iii) MA models that are not unique.
Lijia Wang (UofT) STA457: Time Series Analysis 40 / 47
Problems encountered - Solutions
To summarize, we have seen the following problems:
(i) parameter redundant models,
Solution: we require that ω(z) and φ(z) have no common factors.
(ii) stationary AR models that depend on the future, and
Solution: A formal definition of causality for ARMA models.
(iii) MA models that are not unique.
Lijia Wang (UofT) STA457: Time Series Analysis 41 / 47
Definition for causal
Definition: An ARMA(p, q) model is said to be causal, if the time series
{xt ; t = 0, ±1, ±2, · · · } can be written as a one-sided linear process:
↑
$
xt = ςj wt→j = ς(B)wt ,
j=0
where
↑
$ ↑
$
ς(B) = ςj B j , and |ςj | < ↘; we set ς0 = 1.
j=0 j=0
Lijia Wang (UofT) STA457: Time Series Analysis 42 / 47
Causality of ARMA(p,q) process
An ARMA(p, q) model is causal if and only if ω(z) ↓= 0 for |z| ≃ 1. The
coe!cients ςj ’s of the linear process can be determined by solving
↑
$
j φ(z)
ς(z) = ςj z = , |z| ≃ 1
ω(z)
j=0
Remark: An ARMA process is causal only when the roots of ω(z) lie
outside the unit circle; that is, ω(z) = 0 only when |z| > 1.
Lijia Wang (UofT) STA457: Time Series Analysis 43 / 47
Problems encountered - Solutions
To summarize, we have seen the following problems:
(i) parameter redundant models,
Solution: we require that ω(z) and φ(z) have no common factors.
(ii) stationary AR models that depend on the future, and
Solution: A formal definition of causality for ARMA models.
(iii) MA models that are not unique. Solution: The formal definition of
invertible property allows an infinite autoregressive representation.
Lijia Wang (UofT) STA457: Time Series Analysis 44 / 47
Definition for invertible
Definition: An ARMA(p, q) model is said to be invertible, if the time
series {xt ; t = 0, ±1, ±2, · · · } can be written as
↑
$
↼(B)xt = ↼j xt→j = wt
j=0
#↑ #↑
where ↼(B) = j=0 ↼j Bj , and j=0 |↼j | < ↘; we set ↼0 = 1. ↼(B)
Lijia Wang (UofT) STA457: Time Series Analysis 45 / 47
ARMA(p,q): Invertible
An ARMA(p, q) model is invertible if and only φ(z) ↓= 0 for |z| ≃ 1. The
coe!cients ↼j S of ↼(B) can be determined by solving
↑
$
j ω(z)
↼(z) = ↼j z = , |z| ≃ 1
φ(z)
j=0
Remark: An ARMA process is invertible only when the roots of φ(z) lie
outside the unit circle; that is, φ(z) = 0 only when |z| > 1.
Lijia Wang (UofT) STA457: Time Series Analysis 46 / 47
Example: ARMA(p,q)
Example 5: Consider the following time series model
xt = 0.4xt→1 + 0.45xt→2 + wt + wt→1 + 0.25wt→2 (1)
1 Identify the above model as ARMA(p, q) model (watch out parameter
redundancy)
2 Determine whether the model is causal and/or invertible
3 If the model is causal, write the model as a linear process
Lijia Wang (UofT) STA457: Time Series Analysis 47 / 47
