TA457: Time Series Analysis

Lecture 4

Lijia Wang

Department of Statistical Sciences

University of Toronto

Lijia Wang (UofT) TA457: Time Series Analysis 1 / 33

Overview

Last Time:
1 ETS Models
2 White noise
Today:
1 Time series statistical models
2 Autocorrelation
3 Stationarity

Lijia Wang (UofT) TA457: Time Series Analysis 2 / 33

Outline

1 Time series statistical models

Measure of dependence
Examples of time series models

2 Autocorrelation
Autocovariance
Autocorrelation function
Other autocorrelation functions

3 Stationary
Weak stationary
Strong stationary
Other forms of stationary

Lijia Wang (UofT) TA457: Time Series Analysis 3 / 33

Review: Time series Data

Univariate Time Series:

A univariate time series is a sequence of measurements of the same

variable collected over time. Most often, the measurements are made at
regular time intervals.

Data are not necessarily independent and not necessarily identically

distributed.
Dependence is important
The ordering matters

Lijia Wang (UofT) TA457: Time Series Analysis 4 / 33

Measures of Dependence

A complete description of a time series, observed as a collection of n

random variables at arbitrary time points t1 , t2 , . . . , tn , for any positive
integer n, is provided by the joint distribution function, evaluated as the
probability that the values of the series are jointly less than the n
constants, c1 , c2 , . . . , cn ; i.e.,
Definition:

Ft1 ,t2 ,...,tn (c1 , c2 , . . . , cn ) = P (xt1 → c1 , xt2 → c2 , . . . , xtn → cn )

Unfortunately, these multidimensional distribution functions cannot

investigated easily unless the random variables are jointly normal

Lijia Wang (UofT) TA457: Time Series Analysis 5 / 33

The marginal distributions

Definition: Let Ft (x) be the marginal distribution functions of xt and is

defined by
Ft (x) = P {xt → x} .
The corresponding marginal density function is then defined as

ωFt (x)
ft (x) =
ωx
provided that the density exists.

Lijia Wang (UofT) TA457: Time Series Analysis 6 / 33

Examples of time series models

Some typical examples of time series models:

White noises
Moving Averages
Autoregressions
Random Walk

Lijia Wang (UofT) TA457: Time Series Analysis 7 / 33

White Noise (Review)

Definition: Suppose that {wt , t ↑ Z} is a sequence of independent and

identically distributed random variables with mean zero, and variance
E [wt2 ] = εw2 < ↓.
xt = w t
is called a strong white noise series in L2 (abbreviated xt ↔ iid(0, εw2 )).

If in addition, we assume wt ↔ N(0, εw2 ), then we say xt is a Gaussian

iid
white noise, abbreviated xt ↔ N(0, εw2 ).

Lijia Wang (UofT) TA457: Time Series Analysis 8 / 33

Random Walk (Review)

Definition: Consider a time series model where the present state xt equals
xt→1 plus some constant and an error term:

xt = ϑ + xt→1 + ϖt

This is sometimes called a random walk with drift model. Simple

arithmetic yields that
t
!
xt = ϑt + x0 + ϖi
i=1

Lijia Wang (UofT) TA457: Time Series Analysis 9 / 33

Example: Moving Average

A moving average is a tool to smooth the series. A moving average of

order 3 for the white noise process wt can be written as
1
vt = (wt→1 + wt + wt+1 )
3

A more rigorous definition will be given next week.

Lijia Wang (UofT) TA457: Time Series Analysis 10 / 33

Example: Autoregressions

The output variable xt of the autoregressive model depends on the past

values of the series. An example of the autoregressive model (AR of order
2) is
xt = xt→1 ↗ 0.9xt→2 + wt for t = 1, 2, · · · , 500.

A more rigorous definition will be given next week.

Lijia Wang (UofT) TA457: Time Series Analysis 11 / 33

Mean function

We further define the mean of a time series random variable.

Definition: The mean function is defined as

" ↑
µxt = E (xt ) = xft (x) dx,
→↑

provided it exists, where E denotes the usual expected value operator.

When no confusion exists about which time series we are referring to, we
will drop a subscript and write µxt as µt .

Lijia Wang (UofT) TA457: Time Series Analysis 12 / 33

Example: Moving average series

Example: A moving average is a tool to smooth the series. A moving

average of order 3 for the white noise process wt can be written as
1
ϱt = (wt→1 + wt + wt+1 ),
3
i.i.d
where wt ↔ N(0, εw2 ).

Question: Derive the mean of ϱt .

1
µωt = E (ϱt ) = [E (wt→1 ) + E (wt ) + E (wt+1 )] = 0.
3

Lijia Wang (UofT) TA457: Time Series Analysis 13 / 33

Example: Random Walk series

Example: Consider the following random walk series:

xt = xt→1 + wt ,
i.i.d
with initial condition x0 = 0 and where wt ↔ N(0, εw2 ).

Question: Derive the mean of xt .

t
!
µxt = E (xt ) = E (xt→1 ) + E (wt ) = E (x0 ) + E (wi ) = 0.
i=1

Lijia Wang (UofT) TA457: Time Series Analysis 14 / 33

Outline

1 Time series statistical models

Measure of dependence
Examples of time series models

2 Autocorrelation
Autocovariance
Autocorrelation function
Other autocorrelation functions

3 Stationary
Weak stationary
Strong stationary
Other forms of stationary

Lijia Wang (UofT) TA457: Time Series Analysis 15 / 33

Autocovariance

Definition: For a given time series xt , the autocovariance function is

defined as the second-moment product

ςx (s, t) = Cov (xs , xt ) = E [(xs ↗ µs ) (xt ↗ µt )] ,

for all s and t.

We may write ςx (s, t) as ς(s, t) if no confusion.

Note that ςx (s, t) = ςx (t, s) for all time points s and t.

Lijia Wang (UofT) TA457: Time Series Analysis 16 / 33

Autocovariance: properties

The Autocovariance have the following properties:

The autocovariance measures the linear dependence between two
points on the same series observed at di!erent times.
Very smooth series exhibit autocovariance functions that stay large
even when the t and s are far apart, whereas choppy series tend to
have autocovariance functions that are nearly zero for large
separations.
if ςx (s, t) = 0, xs and xt are not linearly related, but there still may be
some dependence structure between them. If, however, xs and xt are
bivariate normal, ςx (s, t) = 0 ensures their independence.

Lijia Wang (UofT) TA457: Time Series Analysis 17 / 33

Variance

Definition: The variance function is defined as

# $
var (xt ) = E (xt ↗ µt )2 ,

for all t.

Obviously, for s = t, the autocovariance reduces to the (assumed finite)

variance, because
# $
2
ςx (t, t) = E (xt ↗ µt ) = Var (xt )

Lijia Wang (UofT) TA457: Time Series Analysis 18 / 33

Examples

Example: Compute the mean and autocovariance function of a:

1 White noise sequence wt
2 “Moving average sequence” xt = wt + φwt→1
3 The time series xt = cos(2↼t) + sin(2↼t)wt

Lijia Wang (UofT) TA457: Time Series Analysis 19 / 33

Autocorrelation function (ACF)

Definition: The autocorrelation function (ACF) is defined as

ς(s, t)
↽(s, t) = % .
ς(s, s)ς(t, t)

The ACF measures the linear predictability of the series at time t, say xt ,
using only the value xs .

Lijia Wang (UofT) TA457: Time Series Analysis 20 / 33

Autocorrelation function (ACF): properties

The ACF have the following properties:

1 ↗1 → ↽(s, t) → 1
2 Suppose xt = ⇀0 + ⇀1 xs .
1 If ⇀1 > 0, then ↽(s, t) = 1;
2 If ⇀1 < 0, then ↽(s, t) = ↗1.

Lijia Wang (UofT) TA457: Time Series Analysis 21 / 33

The cross-covariance function

Definition: The cross-covariance function between two series, xt and yt , is

ςxy (s, t) = cov (xs , yt ) = E [(xs ↗ µxs ) (yt ↗ µyt )] .

There is also a scaled version of the cross-covariance function, called the

cross-correlation function (CCF), and is given by

ςxy (s, t)
↽xy (s, t) = % .
ςx (s, s)ςy (t, t)

Lijia Wang (UofT) TA457: Time Series Analysis 22 / 33

Outline

1 Time series statistical models

Measure of dependence
Examples of time series models

2 Autocorrelation
Autocovariance
Autocorrelation function
Other autocorrelation functions

3 Stationary
Weak stationary
Strong stationary
Other forms of stationary

Lijia Wang (UofT) TA457: Time Series Analysis 23 / 33

Weak Stationarity

Definition:
A weakly stationary time series, xt , is a finite variance process such that
(i) the mean value function, µt , is constant and does not depend on time
t, and
(ii) the autocovariance function, ς(s, t), depends on s and t only through
their di!erence |s ↗ t|.
We will use the term stationary to mean weakly stationary.

Lijia Wang (UofT) TA457: Time Series Analysis 24 / 33

Weak Stationarity: Notations

Note that for a stationary process:

The mean function, E (xt ) = µt , of a stationary time series is
independent of time t, we will write µt = µ.
The autocovariance function, ς(s, t), of a stationary time series, xt ,
depends on s and t only through their di!erence |s ↗ t|, we may
simplify the notation. Let s = t + h, where h represents the time shift
or lag. We have

ς(t + h, t) = Cov (xt+h , xt ) = Cov (xh , x0 ) = ς(h, 0) = ς(h)

Lijia Wang (UofT) TA457: Time Series Analysis 25 / 33

Weak Stationarity: Cont.

Definition: The autocovariance function of a stationary time series will be

written as

ς(h) = cov (xt+h , xt ) = E [(xt+h ↗ µ) (xt ↗ µ)] .

Definition: The autocorrelation function (ACF) of a stationary time series

will be written as
ς(t + h, t) ς(h)
↽(h) = % = .
ς(t + h, t + h)ς(t, t) ς(0)

Lijia Wang (UofT) TA457: Time Series Analysis 26 / 33

Weak Stationarity: properties

For a stationary series, we have

&n &n
0 → var (a1 x1 + · · · + an xn ) = j=1 k=1 aj ak ς(j ↗ k)
|ς(h)| → ς(0)
ς(h) = ς(↗h).

Lijia Wang (UofT) TA457: Time Series Analysis 27 / 33

Additional properties of Stationarity

Properties and additional terminology:

The property that the mean is time-invariant is also referred to as
first order stationarity or mean stationarity.
First order stationarity along with the property that the
autocovariance is a function of only the lag is also referred to as
second order stationarity.
In this course, unless stated specifically, we use the word “stationary”
to refer to “weak stationary” or “second order stationarity”

Lijia Wang (UofT) TA457: Time Series Analysis 28 / 33

Stationary Examples

Examples:
1 The white noise wt is stationary
2 The moving average series of white noise series
xt = wt + φwt→1 , φ ↑ R is stationary.
3 A random walk series is NOT stationary
1 Though a random walk with mean zero increments is first-order
stationary, it is not second-order stationary.
2 A random walk with drift is neither first-order nor second-order
stationary.

Lijia Wang (UofT) TA457: Time Series Analysis 29 / 33

Strong Stationarity

Definition: We say that the time series xt is strictly stationary or

strongly stationary if its finite dimensional distributions are
shift-invariant. Namely, the probability distribution of

{xt1 , xt2 , . . . , xtk }

is identical to that of the time shifted set

{xt1 +h , xt2 +h , . . . , xtk +h } ,

That is,

P {xt1 → c1 , . . . , xtk → ck } = P {xt1 +h → c1 , . . . , xtk +h → ck }

for all k = 1, 2, . . ., all time points t1 , t2 , . . . , tk , all numbers c1 , c2 , . . . , ck ,

and all time shifts h = 0, ±1, ±2, . . ..

Lijia Wang (UofT) TA457: Time Series Analysis 30 / 33

Weak vs. Strong Stationarity

Relationships between weak and strong stationarity:

1 xt strictly stationary ↘≃ xt weakly stationary.
2 xt strictly stationary and E[xt2 ] < ↓ ≃ xt weakly stationary.
3 xt weakly stationary and xt a Gaussian process ≃ xt strictly
stationary.
4 If xt ↑ L2 for all t ↑ Z, then xt not weakly stationary ≃ xt is not
strictly stationary.

Lijia Wang (UofT) TA457: Time Series Analysis 31 / 33

Joint stationary

Definition: Two time series, say, xt and yt , are said to be jointly

stationary if they are each stationary, and the cross-covariance function

ςxy (h) = cov (xt+h , yt ) = E [(xt+h ↗ µx ) (yt ↗ µy )] .

is a function only of lag h.

Definition: The cross-correlation function (CCF) of jointly stationary

time series xt and yt is defined as

ςxy (h)
↽xy (h) = %
ςx (0)ςy (0)

Lijia Wang (UofT) TA457: Time Series Analysis 32 / 33

Trend Stationarity

For time series xt = ⇁ + ⇀t + yt , where yt is stationary:

The mean function is µx,t = E (xt ) = ⇁ + ⇀t + µy , which is not
independent of time.
Therefore, the process is not stationary.
The autocovariance function, however, is independent of time,
because
ςx (h) = cov (xt+h , xt ) = E [(xt+h ↗ µx,t+h ) (xt ↗ µx,t )] = E [(yt+h ↗
µy ) (yt ↗ µy )] = ςy (h).
Thus, the model may be considered as having stationary behavior around a
linear trend; this behavior is sometimes called trend stationarity.

Lijia Wang (UofT) TA457: Time Series Analysis 33 / 33

 TA457: Time Series Analysis
Lecture 4

Lijia Wang

Department of Statistical Sciences

University of Toronto

Lijia Wang (UofT) TA457: Time Series Analysis 1 / 14

Overview

Last Time:
1 Time series statistical models
2 Autocorrelation
3 Stationarity
Today:
1 Estimation of correlation
2 Large sample properties of sample statistics

Lijia Wang (UofT) TA457: Time Series Analysis 2 / 14

Outline

1 Estimation of correlation
Sample mean
Sample auto-covariance

2 Large sample properties of sample statistics

Large sample properties of sample autocovariance
Sample cross-covariance function

Lijia Wang (UofT) TA457: Time Series Analysis 3 / 14

Features of time series data

Classical statistics vs. Time seriese

From the point of view of classical statistics, we have i.i.d copies of
xt for calculating sample statistics.
For data series data, we don’t have i.i.d copies
In the usual situation with only one realization, the assumption of
stationarity becomes critical.

Lijia Wang (UofT) TA457: Time Series Analysis 4 / 14

Sample mean

Definition: Under the assumption that a time series is stationary,

the mean function µt = µ is constant so that we can estimate it by the
sample mean,
n
1!
x̄ = xt .
n
t=1

x̄ is an unbiased estimator, i.e. E (x̄) = µ.

Lijia Wang (UofT) TA457: Time Series Analysis 5 / 14

Variance of sample mean

Definition: Var (x̄) can be estimated by

" n # " n n
#
1! 1 ! !
Var(x̄) = var xt = 2 cov xt , xs
n n
t=1 t=1 s=1
! n $ %
1 |h|
= 1→ ωx (h)
n n
h=→n

Lijia Wang (UofT) TA457: Time Series Analysis 6 / 14

Sample (Empirical) Autocorrelation

Definition: The sample/empirical autocovariance function of a time series

(presumed to be second-order stationary) is
T
! →h
1
ω̂(h) = (xt → x̄)(xt+h → x̄),
T
t=1

with ω̂(→h) = ω̂(h).

Definition: The sample/empirical autocorrelation function (ACF) of a

time series is
ω̂(h)
ε̂(h) = .
ω̂(0)

Lijia Wang (UofT) TA457: Time Series Analysis 7 / 14

ACF Plot

Figure: (a) White noise Figure: (b) ACF of white noise

Figure: White noise of length 150 and (b) its ACF. The ACF plot is sometimes
called a correlogram.

Lijia Wang (UofT) TA457: Time Series Analysis 8 / 14

Global Temperatures and ACF

Figure: (b) ACF of global

Figure: (a) Global temperature
temperature

Figure: Deviation of global mean yearly temperature from mean computed from
1850-2013.
Lijia Wang (UofT) TA457: Time Series Analysis 9 / 14
Global Temperatures First Di!erence and ACF

Figure: (a) Global temperature first Figure: (b) ACF of global

di!erences temperature first di!erences

Figure: Deviation of global mean yearly temperature from mean computed from
1850-2013 (First di!erences).
Lijia Wang (UofT) TA457: Time Series Analysis 10 / 14
Outline

1 Estimation of correlation
Sample mean
Sample auto-covariance

2 Large sample properties of sample statistics

Large sample properties of sample autocovariance
Sample cross-covariance function

Lijia Wang (UofT) TA457: Time Series Analysis 11 / 14

Large sample properties of sample statistics

Theorem: Under general conditions, if xt is white noise, then for n large,

the sample ACF, ε̂x (h), for h = 1, 2, . . . , H, where H is fixed but arbitrary,
is approximately normally distributed with zero mean and standard
deviation given by
1
ϑω̂x (h) = ↑
n

A rough method of assessing whether peaks in ε̂x (h) are significant is by

↑
determining whether the observed peak is out- side the interval ±2/ n.

Lijia Wang (UofT) TA457: Time Series Analysis 12 / 14

Frame Title

The applications of the above property develop because many

statistical modeling procedures depend on reducing a time series to a
white noise series using various kinds of transformations. After such a
procedure is applied, the plotted ACFs of the residuals should then lie
roughly within the limits given above.
For a white noise sequence, approximately 95% of the sample ACFs
should be within these ± ↑2n .

Lijia Wang (UofT) TA457: Time Series Analysis 13 / 14

Sample cross-covariance function

Definition: The sample cross-covariance function is defined by

n→h
1!
ω̂xy (h) = [(xt+h → x̄) (yt → ȳ )]
n
t=1

with ω̂xy (→h) = ω̂xy (h).

Definition: The sample cross-correlation function is defined by

ω̂xy (h)
ε̂xy (h) = & .
ω̂x (0)ω̂y (0)

Lijia Wang (UofT) TA457: Time Series Analysis 14 / 14