Modeling and Forecasting
Forecasting Methods
Measuring the Accuracy of Forecast
FORECASTING AND MODEL SELECTION
Anurag Prasad
Department of Mathematics and Statistics
Indian Institute of Technology Kanpur, India
REACH Symposium, March 15-18, 2008
1 Forecasting and Model Selection
Modeling and Forecasting
Forecasting Methods
Measuring the Accuracy of Forecast
Outline
1 Modeling and Forecasting
2 Forecasting Methods
3 Measuring the Accuracy of Forecast
2 Forecasting and Model Selection
Modeling and Forecasting
Forecasting Methods
Measuring the Accuracy of Forecast
Assumptions of Forecasting
1 Element of Uncertainty
2 Blind Spots
3 Change in Forecast Accuracy
3 Forecasting and Model Selection
Modeling and Forecasting
Forecasting Methods
Measuring the Accuracy of Forecast
Framework of a Forecast System
Model−Building Phase Forecasting Phase
Theory and/or
Historical Data
No No
Model Model Diagnostic Forecast Stability Forecast
Specification Estimation Checking Generation Checking Updation
Yes Yes
New
Data
Observations
4 Forecasting and Model Selection
Modeling and Forecasting
Forecasting Methods
Measuring the Accuracy of Forecast
Choice of a Particular Forecast Model
1 Degree of Accuracy Required
2 Cost of Producing Forecasts
3 Forecast Horizon
4 Degree of Complexity Required
5 Available Data
5 Forecasting and Model Selection
Modeling and Forecasting
Forecasting Methods
Measuring the Accuracy of Forecast
Classification of Estimation Methods
1 Time Series Methods
2 Causal Methods
3 Judgemental Methods
6 Forecasting and Model Selection
Modeling and Forecasting
Forecasting Methods
Measuring the Accuracy of Forecast
Time Series Methods
• Use historical data as a basis
• Underlying patterns are fairly stable
1 Autoregressive Moving Average (ARMA)
2 Exponential Smoothing
3 Extrapolation
4 Linear Prediction
5 Trend Estimation
6 Growth Curve
7 Box-Jenkins Approach
7 Forecasting and Model Selection
Modeling and Forecasting
Forecasting Methods
Measuring the Accuracy of Forecast
Causal Methods
• Belief that some other time series can be useful
• Assumption that it is possible to identify the underlying factors
1 Regression Analysis
* Linear Regression
* Non-Linear Regression
2 Econometrics
8 Forecasting and Model Selection
Modeling and Forecasting
Forecasting Methods
Measuring the Accuracy of Forecast
Judgemental Methods
• Incorporate intuitive judgements, opinions and probability
estimates
1 Composite Forecasts
2 Surveys
3 Delphi Method
4 Scenario Building
5 Technology Forecasting
6 Forecast by Analogy
9 Forecasting and Model Selection
Modeling and Forecasting
Forecasting Methods
Measuring the Accuracy of Forecast
Forecast Error
For t = 1, . . . , N,
y (t) : Actual value at period t,
yb(t) : Forecast value at period t;
e(t) : Forecast error at period t; e(t) = y (t) − yb(t)
^y(t )
y(t) i
y(t )
i
t
t i
10 Forecasting and Model Selection
Modeling and Forecasting
Forecasting Methods
Measuring the Accuracy of Forecast
Graphical Measures of Forecast Accuracy
Plot of y (t) versus yb(t)
• Keep the same scale for both the axes.
• Departure of points from the 450 line through origin indicates
imperfect forecasts.
11 Forecasting and Model Selection
Modeling and Forecasting
Forecasting Methods
Measuring the Accuracy of Forecast
y(t) y(t)
^
y(t) ^
y(t)
Correct Model Form Incorrect Model Form
12 Forecasting and Model Selection
Modeling and Forecasting
Forecasting Methods
Measuring the Accuracy of Forecast
Plot of e(t) versus t
• Reveals patterns of variability which the model has failed to
explain.
• For a good model, the forecast errors should vary in a
horizontal band around zero.
13 Forecasting and Model Selection
Modeling and Forecasting
Forecasting Methods
Measuring the Accuracy of Forecast
e(t) e(t)
0 0
t t
Correct Model Form Incorrect Model Form
e(t) e(t)
0 0
t t
Incorrect Model Form Incorrect Model Form
14 Forecasting and Model Selection
Modeling and Forecasting
Forecasting Methods
Measuring the Accuracy of Forecast
Descriptive Measures of Forecast Accuracy
Descriptive Measures of Forecast Accuracy are used to...
1 Provide a single, easily interpreted measure of model’s
reliability
2 Compare the accuracy of two different models
3 Search for an optimal model
4 Monitor a model’s performance
15 Forecasting and Model Selection
Modeling and Forecasting
Forecasting Methods
Measuring the Accuracy of Forecast
Descriptive Measures of Forecast Accuracy
Mean Absolute Deviation (MAD)
N
|forecast error| |e(t)|
MAD = number of forecasts = t=1
N
Mean Square Error (MSE)
N
(forecast error)2 e(t)2
MSE = number of forecasts = t=1
N
Root Mean Square Error (RMSE)
√
RMSE = MSE
16 Forecasting and Model Selection
Modeling and Forecasting
Forecasting Methods
Measuring the Accuracy of Forecast
Descriptive Measures of Forecast Accuracy
Mean Absolute Percent Error (MAPE)
|forecast error/actual value|
MAPE = number of forecasts .100%
N
|e(t)/y (t)|
= t=1
N .100%
Pearson’s Correlation Coefficient (r )
between y (t) and yb(t)
N ¯
t=1 (y (t)−ȳ)(y (t)−y)
r=
N 2 N ¯ 2
t=1 (y (t)−ȳ) t=1 (y (t)−y)
17 Forecasting and Model Selection
Modeling and Forecasting
Forecasting Methods
Measuring the Accuracy of Forecast
Descriptive Measures of Forecast Accuracy
"No Change" model is : yb(t + 1) = y (t)
Theil’s Inequality Coefficient (U)
RMSE("new" model)
U= RMSE("no change" model)
U>1 ⇒ worse than "no change" model
U=1 ⇒ as good as "no change" model
U<1 ⇒ better than "no change" model
18 Forecasting and Model Selection
Modeling and Forecasting
Forecasting Methods
Measuring the Accuracy of Forecast
Further Readings
Quantitative Forecasting Methods, N.R. Farnum and L.W.
Stanton, 1989, PWS-KENT Publishing Co.
Statistical Methods for Forecasting, B. Abraham and J.
Ledolter, 1983, John Wiley & Sons
Introduction to Time Series and Forecasting, P.J. Brockwell
and R.A. Davis, 2002 (Second Edition) , Springer-Verlag
Time Series Analysis and Forecasting, R. Yaffee, 2000,
Academic Press
19 Forecasting and Model Selection
Modeling and Forecasting
Forecasting Methods
Measuring the Accuracy of Forecast
Thank You
20 Forecasting and Model Selection
