Reprinted with the permission of AACE International

Page 1 CIMFP Exhibit P-00959 209 Prairie Ave., Suite 100, Morgantown, WV 25601 USA
2007 AACE International Transactions
Phone 800-858-COST/304-296-8444
Fax: 304-291-5728
RISK.03 Internet: http://www.aacei.org
E-mail: [email protected]
Copyright © by AACE International; all rights reserved

The Monte-Carlo Challenge:

A Better Approach

John K. Hollmann, PE CCE

he definition of contingency and how to estimate it ment, empirically-based knowledge, and risk analysis methods

T are among the most controversial topics in cost engi-

neering. While there is consensus among cost engi-
neers on what contingency is, there is much less con-
sensus on how to estimate it. This lack of consensus and the
unfortunate political nature of contingency issues partly
such as Monte-Carlo.
This paper outlines a practical approach for estimating con-
tingency that addresses the findings of the IPA research, and, in
my opinion, better represent best-practice. However, before out-
lining the improved methods, more explanation is in order as to
explains why AACE International has never established a rec- why line-by-line Monte-Carlo often does not work and what the
ommended practice for how to estimate contingency. attributes of a best practice should be.
In general, Industry can agree that there are four general
classes of methods used to estimate contingency. These include
the following. MONTE CARLO (AS COMMONLY MISPRACTICED)

• Expert judgment. The most common method of Monte Carlo based contin-
• Predetermined guidelines (with varying degrees of judg- gency estimating used by industry is “line-by-line” estimating of
ment and empiricism used). ranges with Monte Carlo simulation applied. In this approach,
• Monte Carlo or other simulation analysis (primarily risk as commonly applied, the estimate line-items (e.g., install steel
analysis judgment incorporated in a simulation). And, structure, mechanical engineering, etc.), or estimate subtotals
• Parametric Modeling (empirically-based algorithm, usually by work breakdown or other estimate categories are entered in
derived through regression analysis, with varying degrees of an Excel spreadsheet which serves as the starting basis of a
judgment used). Monte Carlo model. The more detailed the estimate, the more
lines that are usually modeled. Using @Risk® or a similar
I know of only one published study of the efficacy of these spreadsheet add-on program, the analyst/estimator then
methods. In 2004, Independent Project Analysis (IPA) present- replaces each fixed line-item or subtotal cost entry with a statis-
ed a paper that for the first time quantitatively explored the his- tical distribution of cost outcomes for the line item. These line
torical performance of the various techniques [2]. The IPA item distributions are the simulation model inputs. For simplic-
authors found that, despite decades of discussion and develop- ity, the distribution used is almost always “triangular” with the
ment, “…contingency estimates are, on average, getting further line-item point estimate being the peak value, and the high and
from the actual contingency required.” They further state that, low “range” points of the triangle being assigned by the analyst
“This result is especially surprising considering that the percent- or the project team during a “risk analysis” meeting. The high-
age of projects using more sophisticated approaches to contin- low range is usually skewed to the high side (e.g., +50 percent/-
gency setting has been increasing.” In particular when they 30 percent). The analyst then runs the Monte Carlo model sim-
looked at projects for which the scope was poorly defined, they ulation to obtain a distribution of bottom line cost outcomes.
found that the more sophisticated techniques were “a disaster”. Users like the simplicity of the line-by-line range estimating
The sophisticated techniques they referred to were predomi- method. Management likes the graphical outputs.
nately Monte Carlo analysis of line-item ranges. Given how Unfortunately, the method as generally practiced is highly
popular Monte Carlo has become, these are sobering findings flawed. First, the outcomes are unreliable because few practi-
that cost engineers must not ignore. tioners define the “dependencies” or correlation between the
The IPA paper offered a partial remedy; namely that empiri- model inputs (i.e., between the estimate line-items). Valid
cal, regression-based models “…can be a viable alternative or an Monte Carlo modeling requires the analyst to quantify the
excellent supplement to the traditionally used methods for con- degree to which each line item is related to the others. @Risk
tingency setting.” This is particularly true when project scope is incorporates correlation matrices to facilitate this task. As an
poorly defined. In summary, the lesson learned from the IPA example of cost dependency, most estimators would agree that
study is that Monte Carlo, as practiced, is failing and we need construction management costs are somewhat dependent on
to find better methods that incorporate the best of expert judg-
RISK.03.1
 CIMFP Exhibit P-00959 Page 2
2007 AACE International Transactions
field labor costs; if field labor costs come in high, it is likely that The definition uses the words “in aggregate” for a reason. The
construction management will also come in high. reason is that systemic (i.e., non-project or cost item specific)
With independent inputs, each Monte Carlo simulation iter- risk drivers such as the level of project scope definition affect
ation will pick high values for some items and low values for individual, disaggregated estimate line-items in ways that are
others. The highs and lows tend to cancel each other out. The hard to see and predict. For example, no team member in a risk
result is too low of a contingency (i.e., too tight of an outcome analysis can really judge how “poor scope definition” will affect
distribution). Furthermore, analysts can easily bias the simula- a line-item such as civil engineering, steel structure, and so on.
tion outcome without changing any of the risk analysis ranges; The relationship of systemic risk drivers to cost impacts at a dis-
all they need to do is change the number of line items represent- aggregated level is highly obscure—only empirical, statistical
ed by distributions in the model (e.g., look only at subtotals). research shows a clear relationship to cost growth, and then only
These quirks, intentional or otherwise, mean that results are not to bottom-line or highly aggregated costs.
replicable between analysts. Project teams that evaluate risks line-by-line are also tempted
If Monte Carlo is used (in any kind of model) a best practice to then assign contingency to each line, subtotal or WBS ele-
is to define dependencies between model variables. However, a ment and manage it that way. One research study indicated that
possibly more serious shortcoming of the line-by-line Monte this method (and the temptation to spend contingency once so
Carlo method is that it is inherently inconsistent with basic risk assigned) contributes to project failure [7].
management principles. In best practice then, a contingency estimation method
should address systemic risk drivers using empirical knowledge
(actual drivers and project cost history) to produce stochastic
RISK MANAGEMENT models that link known risk drivers (e.g., level of scope defini-
AND CONTINGENCY ESTIMATING tion, level of technology, etc.) to bottom-line project cost
growth.
Contingency estimating is one step the risk management
process. As defined by AACE International, the risk manage-
ment process includes identifying and analyzing risk factors or CONFUSING COST DRIVERS WITH RISK DRIVERS
drivers, mitigating the risk drivers where appropriate, estimating
their impact on plans (e.g., including setting contingency after Risks are things that drive uncertainty of future outcomes.
mitigation) and then monitoring and controlling risk during Risks should not be confused with things that are simply higher
execution [4]. A key concept in risk management is that the in cost. For example, some people will say that revamp work in
contingency estimate must reflect the quantified impacts of risk a process plant is “risky” because it costs more (or takes more
“drivers” or causes; the process seeks to mitigate and manage hours) than new work. However, revamp work is an attribute of
these drivers. In other words, contingency estimating is not an a project scope that only increases the risk significantly if the
end in itself; it is part of a driver-focused process. scope development and project planning practices that define
In line-by-line Monte Carlo, users do not model how risk driv- and mitigate the potential cost impacts of revamp work are not
ers affect cost outcomes. Sometimes the project team will go done well. If the process plant as-built and physical condition
through the effort to identify and discuss risk drivers in the risk has been well examined, the range of possible cost outcomes (or
analysis meeting, but when it comes time to quantify the risks risk) for revamp work will not be significantly wider than new
and estimate contingency, they revert to applying high-low work in percentage terms. In this case, the level of scope defini-
ranges to line-items with only the vaguest idea of how any par- tion and planning is the risk driver or cause, not the fact that the
ticular risk driver affects the cost of a given line item. work is revamp (which may be a cost driver).
In best practice, the contingency estimating method should This relates to our discussion of line-by-line Monte Carlo
explicitly model and document how the risk drivers affect the because, lacking a focus on risk drivers, teams using this method
cost outcomes. Such as model would support risk management tend to focus on why line item costs are high. The exercise
and contingency drawdown during project execution (i.e., as becomes focused on cost reduction or value improvement
teams monitor and assess risk drivers during project execution, rather than risk mitigation. While total cost management recog-
they can determine if the risk drivers have or have not hap- nizes that value and risk management are closely related con-
pened, and the associated contingency can be rationally man- cepts and should be practiced in an integrated way, users must
aged). be careful not to confuse them. Once again, the confusion
comes because systemic risk drivers cannot be effectively dis-
cussed or dealt with at a line item level.
THE EFFECTS OF SYSTEMIC RISK DRIVERS In best practice, a combined risk analysis/contingency esti-
CAN’T BE CONSIDERED LINE-BY-LINE mating method should start with identifying the risk drivers and
events. The cost impacts of the risk drivers and events are then
The AACE International definition of Contingency is “an considered specifically for each driver. For systemic risk drivers,
amount added to an estimate to allow for items, conditions, or stochastic estimating methods are best. However, for project or
events for which the state, occurrence, and/or effect is uncertain item specific risks, more deterministic cost estimates of the
and that experience shows will likely result, in aggregate, in effects of risk drivers are generally appropriate.
additional cost.”

RISK.03.2
 CIMFP Exhibit P-00959 Page 3
2007 AACE International Transactions

Figure 1—Probability Concepts Typically Applied In Contingency Estimating

PROBABILITIES, RANGES AND lighted that best estimating practice for contingency should
CONTINGENCY ESTIMATING include these features:

There is industry consensus that probabilistic contingency • Start with identifying and understanding the risk drivers.
estimating, that addresses the predictive nature of cost estimat- • Recognize the differences between systemic and project-
ing, is a best practice. A cost estimate is not a single value, but a specific risk drivers.
distribution of probable outcomes. As shown in Figure 1, using • Address systemic risk drivers using empirically-based sto-
a probabilistic method, contingency is simply an amount of chastic models.
money that must be added to the point estimate (i.e., best esti- • Address project-specific risk drivers using methods that
mate of all known items) to obtain a cost value that provides explicitly link risk drivers and cost outcomes. And,
management with an acceptable level of confidence (e.g., 50 • If the method uses Monte Carlo, address dependencies.
percent) that the final cost will be less.
Distributions and ranges are one area where Monte Carlo The good news is that contingency estimating methods that
methods always shine. However, there is often a misunderstand- apply best practices are not overly complex and the technology
ing that only Monte Carlo can produce probabilistic outcomes. is well-documented. The author, in conjunction with the
Parametric modeling methods can provide probabilistic infor- Center for Cost Engineering (C4CE; an alliance of Conquest
mation as well. Consulting Group and Validation Estimating LLC) have devel-
oped tools that successfully apply these best practices. The
remainder of this paper summarizes industry information about
DRIVER-BASED METHODS: A BETTER APPROACH empirically-based stochastic models, discusses project-specific
“driver-based” cost models using Monte Carlo, and reviews
In summary, line-by-line Monte Carlo range estimating for C4CE’s integrated application of these practices.
contingency is not working. In part, this is because the method
is inconsistent with best risk management practice. The preced-
ing assessment of line-by-line Monte Carlo’s shortcomings high-

RISK.03.3
 CIMFP Exhibit P-00959 Page 4
2007 AACE International Transactions
EMPIRICAL, DRIVER-BASED STOCHASTIC developed a way to score an early estimate in order to “assess the
CONTINGENCY MODELS IN INDUSTRY thoroughness, quality, and accuracy and thus provide an objec-
tive method for assigning contingency” [12]. The team collect-
IPA’s 2004 research suggested empirical, regression-based ed and analyzed the data on 67 completed projects. Again, their
contingency estimating models as one approach for improved findings generally confirmed the findings of the earlier models.
contingency estimating. This approach is conceptually simple; In a related development, CII has also introduced its project
just collect quantitative historical data about project cost development rating index (PDRI). While the CII validated that
growth, practices and attributes. Then, using regression analysis, the PDRI was correlated with cost growth, no PDRI-driven con-
look for correlations between the cost growth and the practices tingency model has been published.
and attributes (i.e., risk drivers), keeping in mind that you are Table 1 summarizes the primary types of risk drivers included
looking for causal relationships. Unfortunately, most companies in the published cost growth models. Because these are empiri-
do not have the historical data available for analysis. However, cal models, the results are influenced by the project types
there are publicly available industry sources that provide the included in the study datasets. The Rand study was focused on
basic relationships. The primary sources include the work of the pioneer process plants so it was better able to quantify the signif-
late John Hackney, the Rand Institute, and the Construction icance of process technology and complexity drivers. The CII
Industry Institute (CII). dataset included more conventional projects and highlighted
Hackney: John W. Hackney (sometimes referred to as the more risk drivers related to the estimating process itself. Each
father of cost engineering) first described the relationship study defined and measured the risk drivers somewhat different-
between the level of project scope definition and project cost ly making direct comparison difficult. However, all studies have
growth in his 1965 book “Control and Management of Capital found that the level of process and project definition is the most
Projects” (given the books long term importance to industry, significant systemic risk driver. The impacts of estimating
AACE International acquired the publication rights; see the process drivers (e.g., quality of estimating data available) are rel-
www.aacei.org book store) [3]. Mr. Hackney developed a defini- atively less and only become significant when the project is oth-
tion checklist and rating system, and using data from 30 actual erwise well defined.
projects, showed how the definition rating was related to cost In 2002, IPA published further empirical industry research
overruns and could be used as a basis of contingency estimating. that showed that project control practices were also a systemic
Rand: In 1981, Mr. Edward Merrow of the Rand Institute risk driver [5]. Poor control practices can negate the benefits of
(Mr. Merrow later founded IPA) led a study for the US good project scope definition by allowing costs to grow unfet-
Department of Energy on cost growth and performance short- tered during execution (i.e., good project definition practices
falls in pioneer process plant projects [8]. The Rand study exam- before authorization do not guarantee well disciplined practices
ined detail data from 44 projects from 34 major process indus- after).
try companies, confirming and expanding on Mr. Hackney’s This industry research is reflected in AACE International’s
findings, and providing a basic parametric cost growth model Recommended Practice for cost estimate classification [1].
applicable to the process industries. That document outlines the level of scope definition that is rec-
CII: In 1998, an industry research team formed by the CII ommended for each class of estimate (e.g., Classes 5 through 1).
(lead researchers were Garold Oberlender and Steven Trost) It also provides typical contingency and accuracy range “bands”

Table 1—Systemic Risk Drivers Included In Published Cost Growth Models

RISK.03.4
 CIMFP Exhibit P-00959 Page 5
2007 AACE International Transactions

Figure 2—Expected Value In a Standard Risk Model

(i.e., a range of ranges) for process industry projects. These account. Project contingency is then the sum of the expected
range bands represent the consensus of industry experts and are impacts from all significant risk drivers.
generally consistent with the outcomes of the studies discussed Terms such as “cause-risk-effect” have been used instead of
here. “driver-event-impact,” but the concept is the same. A key advan-
Lacking in-house data, a company can use the information in tage of this method is that it unambiguously ties the risk drivers
these studies and standards to create a contingency estimating to the cost impact and therefore allows for effective risk manage-
model based on systemic drivers. While not the most elegant ment. A drawback is that the method can become complex if
approach, the tool can be developed through trial and error. the analyst does not screen the risk drivers/events and focus only
First, substitute best and worst case ratings for each driver in on those that have significant probability and impact.
each published model and assess the sensitivity of the outcomes The ETA/EV approach provides point-estimates of the most-
to the drivers. After deciding how you are going to rate the risk likely cost impacts of each risk driver. Without further analysis,
drivers for your company projects (e.g., you can use the AACE the sum of the expected cost impacts for each risk event can be
International estimate classification attributes, PDRI, Lickert used as the contingency. However, the method supports proba-
scale ratings such as used by CII, etc.), create a first-pass trial bilistic outcomes through Monte Carlo simulation. In that case,
model of factors and parameters along the lines of those pub- distributions are used to express the risk event probabilities and
lished. You may also incorporate some obvious cost growth cost impacts. To obtain range information (i.e., cost outcome
inhibitors such as how much of the estimate is fixed price or distributions), the user can enter the risk event model in a
major equipment. Then, iteratively adjust your model until it spreadsheet and apply Monte Carlo simulation to it (making
reasonably replicates the results of the published models and sure to address dependencies). I call this approach driver-based
standards. The last and most important step is to use your com- Monte-Carlo (DBM) to differentiate it from traditional line-
pany’s actual risk driver and cost outcome data to validate, cali- item approaches.
brate and improve the model over time.

PUTTING THE METHODS INTO PRACTICE

A PROJECT-SPECIFIC, DRIVER-BASED AN INTEGRATED APPROACH
CONTINGENCY ESTIMATING MODEL
Using the approaches discussed above, C4CE has developed a
While there are a number contingency modeling approaches basic parametric contingency estimating model for systemic
possible for non-systemic, project-specific risks (i.e., event-driv- risks, and an expected value template for modeling project spe-
en) the method that is most accessible to the average cost engi- cific risk drivers using Monte Carlo. For early estimates (i.e.,
neer is event or probability tree analysis (ETA). ETA uses the AACE International Class 5 or 4), the parametric model can be
concept of expected value (EV) to quantify the likely cost out- used alone. For authorization and control estimates (i.e., AACE
come of a risk event. The event tree/expected value approach is International Class 3), the tools are integrated by incorporating
used in what some call the “standard risk model” [6,11]. It is the parametric model output as the first “risk driver” (i.e., sys-
also used in decision analysis [10]. Figure 2 provides a simple temic risks) in the expected value model. C4CE refers to the
example how the standard risk model, using the concept of EV, combined approach as DBM. As indicated in figure 3, the
can be used to estimate the expected impact on a single cost DBM output is a single probabilistic cost distribution consider-
ing all risk drivers. Contingency is then determined based on
RISK.03.5
 CIMFP Exhibit P-00959 Page 6
2007 AACE International Transactions

Figure 3—The C4CE’s DBM Method Integrates Best Practices

management’s desired level of confidence that the project will experienced with the approach so the team will surface the crit-
underrun the cost. ical risks without going overboard and getting lost in tangents
The reason that the parametric model can be used alone for and details.
Class 5 and most Class 4 estimates is that for early estimates, the A unique element of the C4CE approach is the practical inte-
cost impacts of project-specific risks are relatively insignificant gration of best practices. The practices themselves are docu-
in comparison to systemic risk drivers. Also, given that the proj- mented in the industry literature (re: this paper’s references)
ect scope is poorly defined for early estimates, project-specific although most companies need some help putting it together.
risk drivers are not readily definable. On the other hand, for C4CE does not sell software; its mission is to help owner clients
well-defined Class 3 or better estimates, the systemic risks (other build and implement their own core cost engineering capabili-
than the use of new technology) tend to become less significant ties in-house. Therefore, C4CE starts with basic contingency
than the project-specific risk. estimating tool templates, customizes them to work with a com-
In practice, the DBM method requires more explicit risk pany’s estimating process (e.g., does the company use AACE
analysis (i.e., risk identification, screening, and quantification) International’s estimate classification matrix?, CII’s PDRI
than the line-by-line approach; this is simply the price of using checklists?, etc.), and develops risk analysis guidelines that
a valid risk management method. A good practical reference on address the company’s typical project risks. After some training
how to do risk analysis is the recent text by Mulcahy [9]. First, in how to use the tool and conduct risk analysis, the owner com-
after getting all the risk out on the table, the team must be selec- pany has everything it needs in-house that it needs to put best
tive and explicit in defining the most probable and costly risk practices for risk analysis and contingency estimating into
drivers and events. Second, the team must quickly prepare con- action.
ceptual (i.e., AACE International Class 5 quality) range esti-
mates of each risk event’s impact. The method requires that the
risk analysis team include some participants with expertise in onte-Carlo techniques for estimating contingency, as
the key project execution roles (engineering, construction, etc),
and some with conceptual cost estimating skills. Another
requirement is that the risk analysis be facilitated by someone
M typically applied, are not working. They fail for three
basic reasons: users are not addressing dependencies
between model variables; they are not modeling the relation-

RISK.03.6
 CIMFP Exhibit P-00959 Page 7
2007 AACE International Transactions
ships of risk drivers to cost outcomes (i.e., their methods are
“line-item” driven); and they fail to recognize the differences
between systemic and project-specific risks. This paper provided
references for and described a practical “driver-based” approach
that combines best practices for parametric modeling of sys-
temic risk drivers and Monte-Carlo analysis of project-specific
drivers to produce reliable contingency estimates at all project John K. Hollmann, PE CCE
estimate phases. Hopefully, future research of the outcome of Owner/Consultant
industry’s contingency estimates will show improving results as Validation Estimating, LLC
methods such as these are incorporated. 47633 Weatherburn Terrace
Sterling, VA 20165-4741, US
REFERENCES Phone: +1.703.945.5483
1. AACE International, Recommended Practice No. 18R-97, Email: [email protected]
Cost Estimate Classification System—As Applied in
Engineering, Procurement, and Construction for the
Process Industries.
2. Burroughs Scott E. and Gob Juntima, “Exploring
Techniques for Contingency Setting”, AACE International
Transactions, 2004.
3. Hackney, John W. (Kenneth H. Humprhies, Editor),
Control and Management of Capital Projects, 2nd Edition,
AACE International, 1997.
4. Hollmann, John K., Editor, Total Cost Management
Framework (Chapter 7.6 Risk Management), AACE
International, Morgantown WV, 2006.
5. Hollmann, John K., Best Owner Practices for Project
Control, AACE International Transactions, 2002.
6. Kaliprasad, Minnesh, Proactive Risk Management, Cost
Engineering, Vol 49, No 12, December 2006.
7. Kujawski, Edouard, “Why Projects Often Fail Even With
High Cost-Contingencies”, US Dept. Of Energy, Office of
Scientific & Technical Information, Report# LBNL—
51349, 2002.
8. Merrow, Edward W., Kenneth E. Phillips, and
Christopher W. Meyers, Understanding Cost Growth and
Performance Shortfalls in Pioneer Process Plants, (R-2569-
DOE), Rand Corporation, 1981.
9. Mulcahy, Rita, Risk Management, RMC Publications
Inc., 2003
10. Schuyler, John, Risk and Decision Analysis in Projects,
Project Management Institute, 2001.
11. Smith, Preston G. and Guy M. Merritt, Proactive Risk
Management: Controlling Uncertainty in Product
Development, Productivity Press, 2002.
12. Trost, Steven M. and Garold D. Oberlender, Predicting
Accuracy of Early Cost Estimates Using Factor Analysis
and Multivariate Regression, Journal of Construction
Engineering and Management, Volume 129, Issue 2, pp.
198-204 (March/April 2003)

RISK.03.7