Chapter 2
Fundamental Concepts
2.1 Stochastic Processes
Family of time indexed random variables.
( ) belongs to a sample space ; to an index set.
As a function of ( ) is called a sample function.
A time series is a sample function from a stochastic process.
Consider { } from * ( ) +.
The dimensional distribution function
( ) ( )
The process is 1-st order stationary in distribution if
( ) ( )
for any integers and
The process is 2-nd order stationary in distribution if
( ) ( )
for any integers and .
The process is -order stationary in distribution if
( ) ( )
for any -tuple ( ) and of integers.
Consider the real-valued process
* +.
The mean function of the process
( ).
The variance function of the process
( ) .
The covariance function between and
( ) ( )( ).
The correlation function between and
( )
( )
For a strictly stationary process
if (| |) .
, if ( ) .
( ) ( )
( ) ( )
Set and From the previous two expressions
( ) ( ) ( )
( ) ( ) ( )
The covariance and correlation between and depends
only on the time difference .
Example of a strictly stationary process i.i.d. random
variables.
A process is th order weakly stationary if all its joint moments
up to order exist and are time invariant
A second order weakly stationary process will have constant
mean and variance. The covariance and correlation will be
functions of the time difference alone.
A second order weakly stationary process is sometimes called
covariance stationary.
A strictly stationary process can also be covariance stationary.
Examples 2.1 to 2.3 in Wei.
Gaussian process: the joint probabilty distribution of the
process is normal.
2.2 The Autocovariance and Autocorrelation Functions
For a stationary process with ( ) and ( )
( ) the covariance and correlation functions
can be written as
( ) ( )( )
and
( )
( )
is the autocovariance function ; the autocorrelation
function.
Properties:
1. ( )
2. | | | | .
3. and
4. and must be positive semidefinite, i.e.,
| |
| |
for any time points and real numbers .
2.3 The Partial Autocorrelation Function
The PACF is defined as ( | ).
Consider the regression equation
is from a zero mean stationary process.
denotes the th regression parameter.
is an error term with zero mean and uncorrelated
with the independent variables.
Multiplying on both sides by and taking expected
value, gives
For we have the system of equations
Solving successively for , we have
| |
| |
etc.
2.4 White Noise Processes
A process * + is called a white noise process if
it is a sequence of uncorrelated random variables
( ) can be zero ; ( )
( )
A white noise process * + is stationary with autocovariance
function
{
and autocorrelation function
and PACF
A white noise process is Gaussian if its joint distribution is
normal.
The basic phenomenon of the process is that the ACF and PACF
are identically equal to zero.
2.5 Estimation
Mean square convergence: * + converge in mean square to
if
( ) as
The sample mean
is an unbiased estimator for ( ) .
The variance is
| |
( ) ( )
( )
If
| |
[ ( ) ]
( )
then ( ) as and is a consistent
estimator for , i.e,
The process is ergodic for the mean if the last result holds.
Sample Autocovariance Function
The following estimators are employed for :
( )( )
or
( )( )
The expected values can be approximated by
( ) ( ) ( )
( ) ( )
Both estimators are biased.
becomes unbiased if ( ) is ignored.
has a larger bias than when is large w.r.t.
If ( ) then both and are asymptotically
unbiased.
is positive semidefinite ; not necessarily so for .
Sample Autocorrelation Function
For a given observed time series the sample ACF is
defined as
( )( )
( )
A plot of vs is called a correlogram.
For stationary Gaussian processes we have for
( ) ( )
which can be estimated by
( )
To test a white noise process, use
The sample ACF is symmetric about the origin, i.e.,
Sample Partial Autocorrelation Function
The sample PACF can be calculated by substituting parameters
by sample estimates in the expressions for .
A recursive formule is available for calculating the sample PACF.
See Wei p.22.
To test for a white noise process, the variance of can be
approximated by
( )
The quantity can be used as limits on to test the
hypothesis of a white noise process.
2.6 Two representations of a time series process
Moving average representation
The process is written as a linear combination of a sequence
of uncorrelated random variables, i.e.,
where * + is a zero mean white noise process, and
An infinite sum is defined as the limit in quadratic mean of the
finite partial sums, i.e.,
[( ) ]
where .
Using the backshift operator , the process can also
be written as
( )
where ( ) . For this process
( )
and
( )
The autocovariance function is given by
( )
( )
The ACF is
For the process to be stationary must be finite for each
Thus
| | | ( )| , ( ) ( )-
A required condition for stationarity is
The process is also called a linear process.
Let be a sequence of autocovariances. The
autocovariance generating function is defined as
( )
The variance, is the coefficient of and the
autocovariance of lag is the coefficient of both and
. Using the expression for , we have
( )
( ) ( )
Provides a way for calculating the autocovariances of some
linear processes. The autocorrelation generating function is
( )
( )
Autoregressive (AR) representation
or
( )
where ( ) ; | |
A process is called invertible if it can be written in this form.
Not every stationary process is invertible.
The linear process ( ) , will be invertible if the roots
of ( ) lie outside the unit circle.
If is a real root then | | If the the root is complex then
and | | .
An invertible process will be stationary if it can be written in
the linear form
( )
( )
so that is satisfied. The condition is that the roots
of ( ) lie outside the unit circle.
Different time series models
Autoregressive process of order
Moving average process of order .
Mixed autoregressive moving average model of order ( ).
