RAPID PROTOTYPING:

LESSONS LEARNED
Opinions on rapid
prototyping as a practical
development tool vary
widely, with conventional
wisdom seeing it more as
a research topic than a
workable method. The
authors counter this
notion with results from
39 case studies, most of
which have used this
approach successfully.

V. SCOll GORDON
Sonoma State University
JAMES M. BIEMAN
S electing an appropriate develop
ment approach is crucial to build-
ing a successful software system. Al-
and costing. T h e study is continuing,
and we hope to publish additional
results in the near future.
though the waterfall model remains T h e box on pp. 86-88 describes our
Colorado State University the most common life-cycle paradigm, evaluation approach. Finding rapid
interest in evolutionary methods such prototyping case studies was not easy.
as rapid prototyping is growing. But is There are many books and research
this approach actually being used? If papers on the subject, but few report
so, how effective is it? actual experience. W e found only 2 2
In an attempt to answer these ques- sources of published case studies that
tions, we launched a study four years include information on the technique's
ago that involved collecting data on as effectiveness. W e supplemented these
many published case studies of rapid with solicited first-hand accounts, as
prototyping as we could find and the box describes. In total, we analyzed
determining commonalities. the results of 39 studies.
In another publication, we docu- Our overall objective is to use the
mented the effects of rapid prototyp- studies to develop guidelines on how
ing on software quality.' H e r e we to use rapid prototyping effectively.
extend that work to include not only An important goal is to compare the
its effects on the product, but also its use of the two main p r o t o t y p i n g
influence on the software process methodologies: throwaway, in which
itself, in areas like development effort the prototype is discarded and not

IEEE SOFTWARE 07407459/95/$M W 0 1994 IEEE 85

Authorized licensed use limited to: NWFP UNIV OF ENGINEERING AND TECHNOLOGY. Downloaded on January 05,2021 at 17:56:16 UTC from IEEE Xplore. Restrictions apply.
 used in the delivered product, and WO- because prototyping itself is a process, able information for modifying the
lutionaly, in which all or part of the pro- and it may not have occurred to au- design.
totype is retained. In presenting our thors to report other process effects. One commercial developer found
results, we pay special attention to fac- + Problems. These include perfor- that prototyping helped reveal misun-
tors that contribute to the selection of mance, end-user misunderstandings, derstandings between developers and
one prototyping method over another. maintenance, delivery of a throwaway users. Sometimes users are not sure
Our results show that developers prototype, budgeting, and prototype they want certain functions imple-
considered rapid prototyping a success completion and conversion. Authors mented until they can actually try
in 33 of the 39 cases. Of the remaining rarely described actual occurrences of them, or they may not know they need
six, three were described as failures these problems, but many detailed the certain features until actual use exposes
and three did not claim success or fail- steps they took to avoid them. As part an omission or inconvenience. A com-
ure. W e suspect some bias in this of our data analysis, we matched stat- mercial study found that prototyping
number, however, because people are ed problems with solutions from other helped ensure that the nucleus of the
often reluctant to report failures, but sources. system is right. One academic devel-
even with this bias, the results clearly oper found prototypes useful because
show that rapid prototyping is a viable PRODUCT ATTRIBUTES “omissions of functions are ... difficult
development tool. Some of the suc- ... to recognize in formal specifica-
cessful sources address intermediate Figure 1 shows the six product tions.” [Case study 201
difficulties encountered and perceived attributes commonly affected by proto- I n working with the prototype,
disadvantages. typing. For each attribute, the figure users may also find certain features or
Another surprise, in light of most indicates case studies that experienced a terminology confusing. Thus proto-
attitudes about evolutionary prototyp- positive or negative impact. The box on typing helps ensure that the first
ing,*was that of the 39 studies, 22 used pp. 86-88 gives the corresponding stud- implementation (after the prototype)
evolutionary prototyping, ies. W e also indicate the will meet user needs, especially when
while only -eight use2 relative number of studies the prototype includes the user inter-
throwaway (the rest did DEVELOPERS in which the particular face. One academic developer states
not state a choice). effect was not observed or “The traditional model of software
T h e studies reflect REPORTED discussed. T h e relatively development relied on the assumption
military (12), commercial
(17), and academic (10)
THAT RAPID high number of unreport-
ed effects reflects the di-
that designers could stabilize and fi-eeze
the requirements. In practice, however,
applications. After evalu- PROTOIMNG versim of reporting- meth- the design of accurate and stable reguire-
ating the data for com-
mon experiences and
WaAS(J((Ess IN o d s ‘amon-g t h e c a s e
studies.
ments cannot be completed until users
gain some experience with the proposed
opinions, we tallied the 33 OF 39 CASES. T h e first three Droduct sojhvare vstem. ” [Case study 171
commonalities to identify
recurring themes. W e
MOST USED attributes - ease of use,
user needs, and number
These findings are consistent with
Fred Brooks’ famous maxim,3 “plan to
were able to group these EVOLUTIONARY. of features - can be con- throw one away; you will, anyhow.”
into three areas: sidered usability factors. T h e effect of prototyping on the
+ Product attributes. The remaining three (per- number of features in a final system is
These include ease of use, match with formance, design quality, and main- less clear. Only eight studies reported
user needs, performance, design quali- tainability) are structural issues. any data in this area. Thus, the evi-
ty, maintainability, and number of fea- dence neither supports nor refutes the
tures. Although design quality and Usability factors. Users have an op- intuitive notion that prototyping gives
maintainability are closely related, portunity to interact with the proto- the end user a license to demand more
many studies reported the two sepa- type, and give direct feedback t o and more functionality.
rately. In some instances, the authors designers. Consequently, difficulties As Figure 1 shows, five cases report
were able to observe maintenance and with the interface are quickly revealed. that the prototype increased the num-
so report specifically about it. In other Seventeen of our case studies reported ber of features (in one case, features
cases, there were maintenance tools. improvements in ease of use as a direct increased b u t the author failed to
+ Process atwibutes. These include result of rapid prototyping; n o n e observe the effect). Authors saw the
effort, degree of end-user participa- reported negative effects. After work- increase as the result of
tion, cost estimation, and expertise ing with a prototype, one user in an special-purpose prototyping lan-
requirements. There were fewer com- academic study soon tired of retyping guages, which made it easy to add new
monalities i n this area, perhaps definitions for each run.This was valu- features,

88 JANUARY 1995

II

Authorized licensed use limited to: NWFP UNIV OF ENGINEERING AND TECHNOLOGY. Downloaded on January 05,2021 at 17:56:16 UTC from IEEE Xplore. Restrictions apply.
 Improved (1 7) Not stated (22)

Ease of use 1,4,7,1l,12,15,16~,20,22,A,C,D,F,I,J,M,N

I n j i q I

Not 5

User needs
x

Increased (5) Decreased(3) Not stated (31)

Performance

Designquality 3, Sa, 6,8, 13, IS, 160, l6c, H 4, Sb, 21, J, 1

Mainiainabiliiy 3, 4, Sa, 13, 18, 0, H, N 5b, 11, 15, I, J, 1

Figure 1. How case-study devehpen perceive the efect of rapid prototyping on the product. Referenced case studies appear iri
the box o n pp. 86-88.

+ internal software development, phase separately from the rest of O n e commercial developer cited
which tends to require less time for development. “design baggage” as a reason for per-
baselining requirements and thus formance problems. Another academic
more time for considering additional Structural factors. In this area, proto- developer suggested considering per-
features, and typing has more opportunity to pro- formance “as early as possible if [the
+ users’ demand for m o r e and duce negative effects. The effect on sys- prototype] is to evolve into the final
more functionality, because the pro- tem performance, for example, depends system.” [Case study 111 T h e case
totype lets them visualize increased partly on the prototype’s scope. When studies contain more evidence of per-
functionality without considering the the prototype focuses solely on the user formance problems for evolutionary
cost of adding features. interface, system performance is likely prototypes than for throwaways.
These authors viewed the increase to be unaffected. When it is used to Design-quality effects are alsc
as a negative effect. O n e military examine various design alternatives, the mixed. M o r e sources indicated a n
developer commented “The customer outcome can vary. improvement in design quality for
goes crazy adding features and mak- Sometimes prototyping can lead to both kinds of prototyping, but others
ing changes.” better system performance because it report that evolutionary prototyping
In contrast, three studies report a is easier t o test several design ap- can result in a less coherent design and
decrease in features as a result of pro- proaches. As one military developer more difficult integration. Still others
totyping. In these studies, prototyp- put it, “...with large and complex sys- state that the iterative design-modify-
ing caused critical components to be tems, one’s intuition is often a poor review process can yield a better over-
emphasized and noncritical ones to be guide for identifylng the performance- all design. T h r e e sources reportec
suppressed, thus reducing the total critical regions that will require opti- more lines of code, while four reportec
number of features. O n e academic mization.” [Case study 81 Prototyping fewer lines. However, neither side
study reports that prototyping “fos- can also provide insights into alterna- interpreted this as good or bad.
tered a higher threshold for incorpo- tive design approaches. One academic Quality also suffers when, during
rating marginally useful features ....” developer found that the prototype let evolutionary prototyping, design stan-
[Case study 41 As a result only the the team assess early on the techniques dards are not enforced in the proto-
most important features were includ- needed to implement specific features. type system. Even when producec
ed in the final product. Evolutionary prototyping, however, with good tools, a design can suffex
One military developer suggested can lead to problems when perfor- when developers fail to remove rem-
that contractors consider incorporat- mance is not adequately measured and nants of discarded design alternatives
ing some mechanism for increasing either inefficient code is retained in One commercial developer cited docu-
the cost of the final product when the the final product o r the prototype mentation with no design of systerr
user requests extra functionality. This demonstrates functionality that is procedures and control flows. Tc
might mean costing the prototype unrealizable under normal usage loads. avoid these problems, the same devel-

IEEE SOFTWARE a7

Authorized licensed use limited to: NWFP UNIV OF ENGINEERING AND TECHNOLOGY. Downloaded on January 05,2021 at 17:56:16 UTC from IEEE Xplore. Restrictions apply.
 SURVEY SCOPE AND RATIONALE
The 39 case studies included in our analysis describe we tried to standardize on the definitions suggested by
particular software projects and discuss how prototyping Bob Patton’ and B. Ratcliff because we found their terms
helped or hindered development. In an effort to obtain a to be the ones most consistent with the terms used in the
good cross-section of practical use, we combined published case studies. Some terminology remains general because
case studies with more focused surveys of individuals who the authors concentrate on different details, and because
claimed to have been involved in software-development the software systems described in the case studies are them-
projects that used rapid prototyping. Consequently, the case selves so diverse. “Design quality,” for example, can mean
studies come from a variety of sources - each with their many things, depending on the nature of the system or the
own motivations and goals - and few addressed all the designer’s point of view. One source may illustrate im-
effects worth tracking. provement in design quality by specifically listing improve-
T o obtain some degree of corroboration among the stud- ments in code structure, reducing patches, and increasing
ies, we have tried to locate the effects that were ob-served in flexibility,while other sources list different items or none
a t least three case studies. Thus, when reviewing the results at all.
by effect, we incurred a preponderance of “not stated” Although we could have included additional attributes,
responses. Perhaps we could decrease this number in the or subdivided attributes into more specific categories, we
future by doing a controlled study, where we h e w in did not because the results would have been less clear and
advance what effects would be tracked. there would have been fewer commonalities among the
The case studies are described below. Each is identified by studies. We did not include attributes that the case studies
a number (published) or letter (anonymous), which is used as did not mention, even if we thought they might be useful.
a reference in the figures in the main text. Also, we omitted some attributes that would have been
interesting to observe, such as cohesion, coupling, cyclo-
Case-study analysis. Case studies varied in degree of rigor. matic complexity, and product lifespan, because of insuffi-
Four sources observed multiple teams (or projects) and pre- cient data in the case studies.
sented one set of conclusions based on careful quantitative
measurements of the results. Six sources described projects
PUBLISHED CASE STUDIES
that involved no customer; the goal was simply to create a
system for the developers. We avoided drawing strong con- 1. Commercial study. The author investigates 12 informa-
clusions about the clarity of requirements or successful tion-systems development projects using the prototyp-
analysis of user needs when a project did not involve a sepa- ing approach in six organizations. H e also presents an
rate user. experiment involving nine student groups, in which
More often, the cases offer subjective conclusions and some groups used prototyping and some used specifica-
suggestions in a less specific, more qualitative manner, tion. The experiment confirmed the findings of the
acquired from personal experience in a single project. The investigation. -M. Alavi, “An Assessment of the
case studies have varied objectives and intended audiences. Prototyping Approach to Information Systems
One study may report difficulty with a particular rapid-pro- Development,” Comm. ACM, June 1984, pp. 556-563.
totyping activity, while another may suggest a remedy for
2. Commercial m d y . The authors describe a methodology
the same problem. Some of the studies include a minimal
for applying object-oriented technology to switching
amount of quantitative measurement interspersed with
judgments. systems and discuss the advantages of using an object-
The published sources listed below represent a variety oriented approach in conjunction with prototyping. -
of organizations in academic, military, and commercial E. Arnold and D. Brown, “Object-Oriented Software
Technologies Applied to Switching System
environments: AT&T,General Electric, Rand, Mitre,
Martin Marietta, Los Alamos National Laboratory, Architectures and Software-Development Processes,”
Tektronix, Rome Air Force Base, Hughes, government Proc. Int’l Switching Symp., 1990, pp. 97-106.
divisions, and others. T h e anonymous sources also repre- 3. Milita? study. Evolutionary prototyping using object-
sent a sampling from these three environments. oriented programming is shown to be an effective way
Some cases involve separate prototyping and develop- to develop a medium-to-large (50,000 LOC) signal-
ment teams. Ten cases are projects conducted at universi- processing system. -B. Barry, “Prototyping a Real-
ties, with only three being student projects. Twelve describe Time Embedded System in Smalltalk,” Proc. OOPSLA,
military projects, and the remaining 17 describe other pro- A C M Press, New York, 1989, pp. 255-265.
fessional software development. (Note that two sources 4. Academic study. Seven student teams developed the same
describe multiple projects.) product: three used prototyping, and four used specifi-
Attribute selection. The lack of commonalities, the diverse cation. The authors compare the two techniques. - B.
reporting methods, and the varying terminology made Boehm, T. Gray, and T. Seewaldt, “Prototyping Versus
amibute selection difficult. We began by identifymg attrib- Specifymg: A Multiproject Experiment,” IEEE Trans.
utes (effects) that appeared in three or more sources and SofcwareEng., May 1984, pp. 290-302.
used these attributes in our analysis. We then tallied the Sa,b. Commercial studies. Both cases involve the use of the
attributes, along with relevant opinions, observations, and Ingres database system. Project Sa involved rewriting a
suggestions. We emphasized conclusions that were reached resource-control system for tracking the cost of soft-
by multiple sources independently. ware projects. It was completed and the system imple-
Although the terminology in the case studies differs, mented to the apparent satisfaction of all parties.

88 JANUARY 1995

Authorized licensed use limited to: NWFP UNIV OF ENGINEERING AND TECHNOLOGY. Downloaded on January 05,2021 at 17:56:16 UTC from IEEE Xplore. Restrictions apply.
 Project Sb involved developing a requirements docu- 13. Academic study. The author describes how rapid-proto-
ment generator. It was criticized by various parties and typing methodologies were used to develop a prototyp-
eventually abandoned. -J. Connell and L. Brice, “The ing language, EPROI,, and a prototyping system,
Impact of Implementing a Rapid Prototype on System EPROS. - S. Hekmatpour, “Experience with
Maintenance,” Proc. AFIPS, TEEE CS Press, Los Evolutionary Prototyping in a Large Software Project,”
Alamitos, Calif., 1985, pp. 515-524. Sofrware Eng. Notes.Jan. 1987, pp. 38-41.
6. Academic study. The authors describe their experiences 14. Military study. The authors propose a new methodology
applying prototyping techniques to the development of for rapid prototyping, called software storming, which
programming languages with advanced features. - R. involves an intense week of videotaped interaction
Ford and C. Marlin, “Implementation Prototypes in the between software designers and users. - P. Jordan et
Development of Programming Language Features,” al., “Software Storming: Combining Rapid Prototyping
Sofhuare Eng. Notes, Dec. 1982, pp. 61-66. and Knowledge Engineering,” Computer, may 1989, pp.
39-48.
7. Commercialstudy. The author describes a case study
15. Academic study. The authors compare specification and
using Promis, a large process-management and infor-
prototyping in a controlled study using student teams.
mation system developed a t General Electric to fabri-
Effort curves were smoother for the prototyping teams,
cate integrated circuits, and tracks the progress of pro-
reducing the “deadline effect” common in software
totyping and subsequent development. - H. Gomaa,
development. - W. Junk, P. Oman, and G. Spencer,
“The Impact of Rapid Prototyping on Specifylng User
“Comparing the Effectiveness of Software Development
Requirements,” Sofhuare Eng. Notes, Apr. 1983, pp. 17-
Paradigms: Spiral-Prototyping vs. Specifying,” Proc.
28.
Pacijc Northwest Sofhuare Qualiq Con$, PNSQC,
8. Military study. The author describes a large multiphase Portland, 1991, pp. 3-18.
software- and hardware-development effort in Ada that 16a-c. Commercial studies. Describes five cases, three of
used incremental development techniques (with rapid which are used in the study reported in this article.
prototyping) and object-oriented design. - M. Goyden, Project 16a involved an expert system to support a cus-
“The Software Lifecycle with Ada: A Command and tomer-advice service. Project I6b involved a distributed
Control Application,” Proc. Tri-Ada, ACM Press, New database accessed via a wide-area network. Project 16c
York, 1989, pp. 40-55. involved a real-time control system for quality assurance
9. Commercial study. The authors describe how small clini- of a chemical process. The authors attempt to highlight
cal research programs were developed and evaluated for commonalities between the cases, and in the process
users who have difficulty clearly expressing their com- show that prototyping has advantages over specification.
puting needs. Rapid prototyping provided a positive Project 16b can be characterized as large. - H. Lichter
environment, in which users were more willing to par- et al., “Prototyping in Industrial Software Projects:
ticipate and better able to express their views. - G. Bridging the Gap Between Theory and Practice,” Proc.
Groner et al., “Requirements Analysis in Clinical ICSE, IEEE CS Press, Los Alamitos, Calif., 1993, pp.
Research Information Processing: A Case Study,” 221-229.
Camputer, Sept. 1979, pp. 100-108. 17. Academic study. The author introduces CAPS, a com-
puter-aided prototyping system that is essentially an
10. Commercialstudy. The author describes 48 Fortune 1000 integrated set of computer-aided software tools, and
companies and evaluates the feasibility of discarding describes its application to the development of a
prototypes. Although most of the companies use throw- medical system. - Luqi, “Software Evolution
away prototyping, the author concludes that evolution- through Rapid Prototyping,” Computer, May 1989,
ary prototyping is preferable. - T. Guimaraes, pp. 13-25.
“Prototyping: Orchestrating for Success,” Datamation,
Dec. 1987, pp. 101-106. 18. Military study. Evolutionary prototyping is shown
to have advantages over specifying, even on large
11.Academic s d y . The author demonstrates the suitability (100,000 LOC) projects. The authors point out that a
of rapid prototyping for the development of CAD sys- significant portion of delivered military systems do not
tems and for object-oriented development. The focus is meet original needs and that prototyping can greatly
on the application rather than the merits of prototyping improve this situation. - C. Martin et al., “Team-
in general. - R. Gupta et al., “An Object-Oriented Based Incremental Acquisition of Large-scale
VLSI CAD Framework: A Case Study in Rapid Unprecedented Systems,” Poliq Sciences, Feb. 1992, pp.
Prototyping,” Computer, May 1989, pp. 28-36. 57-75.
12. Military study. The authors track the development of a 19. Military study. The author describes a prototyping envi-
small prototype messaging system that uses Lisp. Focus ronment at the Rome Air Development Center, con-
is on the role of the prototype, prototyping efforts in centrating on a heavily VO intensive application. There
general, and reuse of prototype code. - C. Heitmeyer, was a significant decrease in development time as a
C. Landwehr, and M. Cornwell. “The Use of Quick direct result of prototyping. - W. Rzepka, “A
Prototypes in the Secure Military Message Systems Requirements Engineering Testbed: Concept, Status
Project,” Sojhare Eng. Notes, Dec. 1982, pp. 85-87. and First Results,” in Proc. Hawaii Con$ System Sciences:

IEEE SOFTWARE 89

Authorized licensed use limited to: NWFP UNIV OF ENGINEERING AND TECHNOLOGY. Downloaded on January 05,2021 at 17:56:16 UTC from IEEE Xplore. Restrictions apply.
 Vol.II, IEEE CS Press, Los Alamitos, Calif., 1989, pp. with user needs. Some people became upset when their
339-347. ideas were quickly discredited by experiences with the
prototype.
20. Academic study. The authors describe how retaining pro-
totype code and using a special-purpose prototyping G. rzlilitaiy study. An engineer a t a sinal1 software company
language can be useful for small software projects. - E. reports prototyping worked well in a small project for
Strand and W. Jones, “Prototyping and Small Software developing low-level system software. One observation
Projects,” Sofnuare Eng. Notes, Dec. 1982, pp. 169-170. was that you can bid lines-of-code-per-day at a higher
rate. Another is that prototyping works only if experi-
21. Military study. The author describes how a major ($100 enced engineers are available.
million) military project was successfully implemented
using rapid prototyping in a CICS development envi- H. Military study. An engineer a t a small military contractor
ronment. The study provides insights into how proto- that develops all products using prototyping reports cm
typing affects management techniques and the develop- the use of special prototyping tools that require fewer
ment process in general. - D. Tamanaha, “An programming skills to master than typical programming
Integrated Rapid Prototyping Methodology for environments.
Command and Control Systems: Experience and I. Comnze~ialstudy. An engineer at a large manufacturer
Insight,” Software Ezg. Notes, Dec. 1982, pp. 387-396. reports on the use of evolutionary prototyping to devel-
22. Academic study. The author describes experiences imple- op very large software systems for workstations. The
menting Backus’s FFP System using rapid ability to overlap some of the requirements, design, and
prototyping. The need for certain functionality coding tasks has improved productivity.
became apparent while exercising the prototype. - M. J. Commercialstudy. An engineer at a large communica-
Zelkowitz, “A Case Study in Rapid Prototyping,” SOB- tions-and-control contractor describes how proof-of-
ware Practice and Expel-ience, Dec. 1980, concept prototypes are developed in Smalltalk, and then
pp. 1037-1042. the final products are developed using evolutionary pro-
totyping with C. Although total effort has decreased and
ANONYMOUS SOURCES the product is easier to use, design quality and perfor-
mance have sometimes suffered because design stan-
A. Academic study. Employees a t a major university describe dards are not rigorous enough.
the development of an online registration system using K. MiZitary study. A consultant for a defense contractor
a special-purpose prototyping environment. They reports how rapid prototyping was used for internal
report improved customer satisfaction, ease of use, and support software, but misunderstandings between engi-
ease of training as a direct result of rapid prototyping. neers and marketing staff, along with difficulties in
B. Academic study. A researcher at a major university reports scheduling and costing, have led the company to return
satisfaction with using Scheme as a prototyping lan- to specification.
guage to produce a working campus-support system in L. Commercial study. An engineer at a large electronics firm
C. describes how operating systems are developed using
C. Commer-cialstudy.An engineer a t a large telecommunica- the Rapid prototyping language. The interface between
tions firm describes problems with rapid prototyping Rapid and C was the “messiest” part of the final system,
that stemmed from management not understanding the but overall the project was successful.
limits of a prototype. The problems resulted in develop- M. Commercial study. An engineer at a small software com-
ment hardships and premature announcements of pany reports that the organization successfully used
delivery dates. rapid prototyping on several nonmilitary internal devel-
D. Military study. An engineer a t a large military con- opment projects and that users are much happier with
tractor reports substantially improved product quali- products developed with prototyping methods.
ty, reduced effort, lower maintenance costs, and N. Commercialstudy. An engineer at a large communica-
faster delivery as a direct result of using rapid proto- tions firm reports that rapid prototyping improved qual-
typing. Leveraging with off-the-shelf products ity, increased user participation, and made final prod-
helped greatly. ucts easier to use. However, developers are often pres-
E. Commercial study. An engineer a t a large data- sured into reusing a throwaway prototype. The engi-
processing firm reports that prototyping has been quite neer recommends carefully defining the prototype’s
effective. Recommendations include using an object- scope and definition.
oriented approach and throwaway prototyping and
carefully selecting an appropriate prototyping
language. REFERENCES
F. Military study. An engineer a t a large military division 1. B. Patton, “Prototyping -A Nomenclature Problem,”
describes the successful development of small govern- Sofiware Eng. Notes, Apr. 1983, pp. 14-16.
ment systems through the use of rapid prototyping. 2. B. Ratcliff, “Early and Not-So-Early Prototyping -
Some results include reuse of typically half the proto- Rationale and Tool Support,” Proc. Compsac, IEEE CS
type, improved product quality, and better matching Press, Los Alamitos, Calif., 1988, pp. 127-134.

90 JANUARY 1995

Authorized licensed use limited to: NWFP UNIV OF ENGINEERING AND TECHNOLOGY. Downloaded on January 05,2021 at 17:56:16 UTC from IEEE Xplore. Restrictions apply.
 oper recommended using a design quality attributes such as performance, ment effort increased. T h e reasons
checklist for each section of incorpo- design quality, and maintainability can cited were frequent changes in user
rated code. suffer during evolutionary prototyping requirements, which tended to frus-
Quality can also be improved by if developers fail to take steps to avoid trate the designers, and a failure to
limiting the scope of the prototype to potential problems, as we describe later. establish explicit procedures for nian-
a particular subset (often the user Unfortunately, the sample size for aging and controlling the prototyping
interface), and by including a design throwaway prototyping is too small to effort. Another commercial developer
phase with each iteration of the proto- be statistically useful in evaluating its also suggested the lack of an organized
type. Another option is to completely specific effect on structural factors. methodology as a possible cause of
discard the prototype. (Only two authors of the eight ad- wasted effort, although it is not clear
Maintainability effects are similar dressed quetions on in- that this was actually ever
to those observed for design quality. dividual attributes.) observed. I n both these
More studies cite better maintainabili- OVERALL, THE cases, prototyping might
ty than worse. Four described success- PROCESS ATTRIBUTES have decreased develop-
ful maintenance of prototyped sys- EFFECT ON THE ment effort if the tech-
terns, even for large projects. (It is Figure 2 lists the four PRODUCT APPEARS niques to manage it had
with some trepidation that we use the process attributes. Again, been more effective.
term “large” a t all, since it means dif- the figure indicates which TO BE POSITIVE, Another coni m c r c i a 1
ferent things to different people. case studies observed ei- study reported an in-
However, in this article, “large” refers ther a positive or negative EVEN FOR URGE crease in development ef-
to systems of 100,000-plus lines of effect. PROJECTS. fort because prototyping
code.) T h e high degree of modularity Although the studies revealed that the initial
required for successful evolutionary discuss other process at- set of requirements was
prototyping can generate easily main- tributes, such as interface design and inadequate, and they had to be extend-
tainable code, because such a system is language selection, there is less com- ed. W e think it is unfair to call this a
more likely to contain reusable and monality than for product attributes. case of increased effort, however, be-
replaceable functional modules. O n We do, however, briefly cover these cause prototyping actually prevented sig-
t h e flip side, including hastily attributes apart from the four in Figure nificant wasted effort in the long run.
designed modules in the final system 2 because our overall goal is to describe I n viewing cost estimation in a
could cause problems. Indeed, for the effective use of rapid prototyping. rapid-prototyping environment, the
evolutionary prototyping specifically general mood is one of skepticism.
( n o t shown i n t h e figure), m o r e Effort and estimation. One of the most One commercia1 developer observed
sources observe worse maintainability. commonly cited benefits of rapid pro- that the early availability of visible out-
Maintenance costs can also go down totyping is that it can decrease develop- puts can cause users and managers to
simply because the final system more ment effort. Most of the case studies be “easily seduced into believing that
accurately reflects user needs. L2/lain- support this notion. In some instances, [subsequent phases] can be skimped
tenance often involves correcting the decrease is dramatic. One military on” [Case study Sb] and will be easy to
invalid requirements or responding to study reports a reduction of 70 percent. complete. As a result, projects can be
changing requirements. Rapid proto- An academic study reports a reduction underbid. O n e military consultant
typing can help reduce this type of of 45 percent. O n e military project cited a case in which a two-year pro-
maintenance because it is likely that team observed a productivity of 34 ject was sold to management as a six-
user needs will have been better met. lines of code per day, more than six week project! Most case studies, how-
times the estimate for typical military ever, did not address cost estimation in
Summary. Overall, we find that sofi- software systems. As one military de- much detail. For that reason, it is diffi-
ware product effects are generally posi- veloper put it, “A buck buys more soft- cult to draw any conclusions about the
tive, with certain problems related pri- ware now than before.” [Case study HI effects of prototyping in this area.
marily to evolutionary prototyping. There are various reasons for this Three sources (two commercial and
W e found, however, that developers positive effect. Faster design is possible one military) did offer some sugges-
can use evolutionary prototyping effec- when requirements are clearer or more tions, though. Managers can use rapid
tively, even for large projects. One aca- streamlined. Also, in evolutionary pro- prototyping in a contract environment
demic developer even stated that, for totyping, part (or all) of the prototype as a separately costed proof-of-concept
small contracts, throwaway prototyp- can be leveraged, so the requirements item. The prototype can be used to test
ing was economically infeasible com- and development efforts tend to overlap. feasibility and/or cost-effectiveness,
pared with evolutionary. Nonetheless, In one commercial case, develop- allowing a customer to abandon a pro-

IEEE SOFTWARE 91

Authorized licensed use limited to: NWFP UNIV OF ENGINEERING AND TECHNOLOGY. Downloaded on January 05,2021 at 17:56:16 UTC from IEEE Xplore. Restrictions apply.
 ject at a greatly reduced expense. It political difficulty that can surface aids in clarifying requirements, match-
also gives the developer a chance to when using rapid prototyping: “Some- ing user needs, and creating an easier-
better estimate (or abandon) a project times our prototype proved that [a to-use product.
that might have been underbid with particular person’s] idea wasn’t so On the other hand, there may be a
specification. Developers using proof- great. Result: one upset persodorga- tendency to design the entire system
of-concept prototyping may have to nization.” [Case study F] from the user interface. This can be
bid actual development of a complete Eight case studies considered an dangerous because the user interface
system separately. Bernard BoarZgives experienced, well-trained team essen- may not characterize the best overall
a number of suggestions for keeping tial for successful prototyping, because system structure. Thus, a user-interface
cost estimates under control. prototyping often involves high-level prototype should be considered part of
design decisions about complex pro- a requirements specification, not a basis
Human factors and staffing, As Figure 2 gramming tasks. Problems can result for system design. Again, discarding
shows, most studies reported greater when inexperienced team members are the prototype is an option, but is only
end-user participation in requirements forced to make such decisions. One helpful when the prototype’s perfor-
definition. This is not surprising, since case specifically described a project that mance is unimportant - not the case
users are likely to be more comfortable failed in part because temporary stu- when the goal is to evaluate the perfor-
with a prototype than a specification - dent programmers were thrown into a mance of a particular design.
which, as one commercial developer rapid-prototyping environment. Other Another attribute that affects proto-
noted, can be dull reading. A military case studies indicated that prototyping typing is language selection. Most case
source also noted that documentation would not have been successful without studies agreed on the importance of
can be “open to interpretation; [where- highly experienced engineers. choosing a language suitable for proto-
as] sample display output is more d e h - Two cases did use entry-level pro- typing, but there was much less agree-
tive.” [Case study 81 Thus the prototype grammers effectively, attributing their ment on what language to choose; in
makes it easier for users to make well- success to the availability of good pro- 38 cases, there were 26 languages! T h e
informed suggestions. totyping tools. However, the failed most popular language was Lisp, and it
As we described earlier, increased project that had inexperienced teams was used in only four cases. Object-
user participation has a positive effect was using a fourth-generation-Ian- oriented methods are receiving in-
on the product by increasing the like guage environment, contradicting the creased attention for rapid prototyp-
lihood that user needs will suggestion t h a t having ing;4 six sources identify object-orient-
be met. In fact, lack of suf- good tools is sufficient; ed methods (three used Small-talk) as
ficient user participation being particularly well-suited for pro-
can negate some prototyp- totyping. One commercial study even
ing benefits. One source cited the reverse relationship: “...the

I
deicribes a case in which
DECREASE IN use
Other effects. One common 00 life-cycle almost always in-cludes
the customer’s manage- of rapid prototyping is prototyping.” [Case study E]
ment purposely excluded DEVELOPMENT to develop a user inter- Two sources cited problems with
end users from interacting face. Various tools exist ex- special-purpose prototyping languages,
with the prototype. T h e OF clusively for quick devel- especially when the language is inter-
intent was to hide the inap- 70 PERCENT. opment of user-friendly preted rather than compiled. Here,
propriate allocation of per- environments, for example the potential for evolutionary proto-
sonnel (in a particular divi- Next’s Interface Builder typing to result in performance prob-
sion’s favor) for as long as possible. and Emultek Corp.’s Rapid. These lems is more clear.
Another source observed the same tools naturally fit into a rapid-prototyp-
technique, calling it “staff rational- ing methodology. When special-pur- PROBLEMS
ization.” pose prototyping tools are used, prod-
This dangerous political maneuver uct maintainability may depend on Most of the studies described suc-
can be avoided simply by making sure these tools remaining available. cessful projects, so there is less direct
that end users remain actively involv- Early emphasis on the user inter- data on prototyping problems. How-
ed. Because a main advantage of rapid face can produce either positive or ever, many authors described antici-
prototyping is in revealing actual re- negative effects, depending on the sys- pating and avoiding particular situa-
quirements, developers should insist tem being developed. tions. In a few cases, the project could
o n prototype interaction with end O n the one hand, early develop- have avoided the reported problem by
users, not just with middle management. ment of a prototype gives users the using a suggestion described in anoth-
A military developer noted another chance to test-drive the software. This er source. Thus, we were able to see

92 JANUARY 1995

11’

Authorized licensed use limited to: NWFP UNIV OF ENGINEERING AND TECHNOLOGY. Downloaded on January 05,2021 at 17:56:16 UTC from IEEE Xplore. Restrictions apply.
 Decreased (1 6) Increased (1 1 Not stated (22)

Effort 2,3,4,8,15,17,18,19,20,21,22,A,G,H,l,M 1

Better (2) Worse (3) Not stated (34)

Cost estimation 161,16!~ s(l%K

Less (2) More (8) Not stated (29)

Expedise
required ~~~-
2,50,5b, 6,
H, M~
14, 165 19, J

Figure 2. How case-study developers perceive the effect of rapid prototyping on the development process. Referenced case studies
appear in the box on pp. 86-88.

common problems and match them user expectations were typically fueled maintainability when the use of a spe-
with possible solutions. by too much or too little access to the cial-purpose prototyping language
prototype. results in maintenance engineers hav-
Performance. When prototyping is By limiting user interaction to a ing to deal with the prototyping lan-
used to evaluate design alternatives, more controlled setting, developers guage, the target language, and the
early measurement of performance is can keep user expectations at reason- interface between them. Complexity
important, and delays in addressing able levels. Users should be clearly can increase, even when system design
problems can result in design prob- told that they are interacting with a is good. A prototyped system can
lems that may be costly to repair later. mock-up, not a working system, and become impossible to maintain if it
A prototype can also demonstrate that the purpose is to clarify require- was developed using tools that are not
functionality that is not possible under ments. Sometimes developers might available to maintenance engineers.
real-time constraints, and this problem want to limit interaction to specific
may not be discovered until long after sequences and administer them per- Delivering a throwaway. One of the
the prototype phase is complete. One sonally. Further, developers should not perils of throwaway prototyping is that
academic source suggested avoiding oversell the prototype in an effort to the prototype may n o t get thrown
this problem by using an open-systems impress the customer. away. This surprisingly common prob-
environment, which would make it Finally, organizations must train lem occurs when managers who agree
easier to integrate faster routines when sales and managerial staff to properly to throwaway prototyping later decide
necessary. understand (and convey) the nature it costs too much to “redo” the system
and purpose of the prototyping phase and try to massage the prototype into
End-user mkundeatandings. Given too and the prototype itself. the product. T h e resulting system
much access t o the prototype, end often lacks robustness, is poorly de-
users may equate the incompleteness Code maintainability. Certainly, a pro- signed, and is not maintainable.
and imperfections in a prototype with totype developed quickly, massaged Managers can avoid this problem
shoddy design. In two cases, this effect into the final product, and then hur- by maintaining a firm commitment to
contributed to the project’s ultimate riedly documented can be very difficult whatever prototyping paradigm was
failure. In another case, rapid proto- to maintain or enhance. Adding to that initially chosen. When a prototype is
typing was abandoned as a suitable is the failure to reevaluate a prototype slated for disposal, then it is best to do
development method because it gave design before starting to implement so because it was probably not design-
users the unrealistic expectation that the final system. T h e result is a prod- ed with retention in mind. Developers
there would be a complete and work- uct that inherits patches from the pro- can do their part by fully defining the
ing system in a short time. totype phase. prototype’s scope and purpose.
Insufficient knowledge of rapid- T o avoid these pitfalls, developers
prototyping techniques is not limited should include documentation criteria Budgeting. T h r e e cases described
to users. Sales staff may pass along in a design checklist, which will help scenarios in which projects were un-
inappropriate expectations to cus- ensure that the prototype is completely derbid. Because visible outputs are
tomers after seeing “working” proto- documented. Other sources suggested quickly available, managers and sales-
types. Users then understandably be- conducting frequent reviews and using persons may believe that development
come skeptical or upset when told that object-oriented technology. Discard- is all downhill from here. This breeds
development will take longer than they ing the prototype is also an option. overconfidence, which in turn can
were led to believe. In the studies, high Prototyping can decrease system result in underbidding.

IEEE SOFTWARE 93

Authorized licensed use limited to: NWFP UNIV OF ENGINEERING AND TECHNOLOGY. Downloaded on January 05,2021 at 17:56:16 UTC from IEEE Xplore. Restrictions apply.
 the ultimate target language does not ur study identified several sur-
21 prising prototyping trends and
have the same simple 110 handling.
Another example is the use of an ob- debunked several myths about the
ject-oriented language such as Small- approach. For example, we found that
talk with a target language that does rapid prototyping is indeed appropri-
not have inheritance. ate for large systems, and there seems
A few sources observed cost or time to be more successful use of evolution-
overruns. Carefully defining the pro- ary prototyping than throwaway. How-
totype’s scope and systematically com- ever, with evolutionary prototyping,
paring the features of both prototyp- developers must be careful to address
Small Medium large
ing and target languages can help a- performance issues early in the process,
void this problem. particularly if parts of the prototype are
F i p r e 3. Distribution of case studies to be included in the final system.
by system size. large systems. As we noted earlier,
opinions vary widely on what consti- T h e study also underlined proto-
tutes a large software system. In one typing’s potential problems. T h e most
In the case in which the two-yea case study, the author described a 200- serious are poor design quality and
project was presented as a six-wee1 line system as large! Although some maintainability (especially when using
project, management believed tha researchers might claim that our defi- evolutionary prototyping), underbid-
prototyping would achieve fast, work nition of large (100,000 LOC) is more ding, and misunderstandings between
ing results. Again, organizations mus suited t o t h e medium range, few developers and users. All of these can
train sales and managerial staff to un would argue that it is small. T o distin- be remedied by carefully defining the
derstand the nature and purpose o guish between medium and small pro- purpose and scope of the prototype
the prototyping phase and the proto jects, we used whatever description and acting in accordance with its limi-
type itself and to make the distinctioi the case study reports used and avoid- tations. 91 organizations can further
between a prototype and a completl ed selecting a specific boundary. reduce design problems by using expe-
system. Figure 3 shows a breakdown of rienced, rather than entry-level, pro-
projects by size. W e find no support grammers whenever prototyping activ-
Completion and conversion. Prototyp for the common notion that evolu- ities involve design decisions. Finally,
development can be time consuming tionary prototyping is dangerous for to avoid creating unrealistic expecta-
especially when the purpose and scop8 large projects. In fact, all seven case tions about prototyping, any end-user
of the prototype is not initially well studies that used evolutionary proto- interaction with the prototype should
defined. BoarZ describes how inade typing with systems of more than be carefully monitored.
quately narrowing scope can lead tc 100,000 L O C reported success!
“thrashing” or “aimless wandering Many case-study authors suggested We recognize that our data is some-
through tasks. Six of the studies sup that problems with evolutionary pro- what incomplete. W e hope in future
port this. Suggested solutions includ totyping will grow in proportion to work to conduct follow-up surveys of
using a disciplined approach to sched system size, although the data they both our published case studies and
ule prototyping activities, careful1 report does n o t provide any direct anonymous sources. W e encourage
defining the prototype’s scope, ani confirmation of this. authors of case studies to contact us
keeping entry-level programmers ou Nonetheless, evolutionary proto- with updated information. W e are
of a rapid-prototyping environment. typing on large projects can pose prob- also seeking additional case studies,
Prototyping languages are oftei lems. It can yield a system filled with especially descriptions of rapid proto-
used to ease the implementation of patches - hastily designed prototype typing in failed software projects.
particular system aspect. For example modules that become the root of later
if the prototype is developed to tes problems. T h e same problems that Our results show that rapid proto-
user-interface options, the develope challenge performance and mainte- typing has had a number of positive
will want a language that provides con nance will become more pronounced effects on both the software product
venient VO. as the system grows. T h e same solu- and development process and that it
However, converting the prototyp tions apply, such as using an object- can be used successfully in a variety of
into the final system may require sig oriented approach or limiting proto- situations. W e hope that the successes
nificant effort and time. This problen typing to user-interface modules, and failures reported here will help
is exacerbated when a separate proto which are less likely to involve impor- those who are now considering this
typing language is used, especially i tant structural design decisions. technique. +
94 JANUARY 1995

Authorized licensed use limited to: NWFP UNIV OF ENGINEERING AND TECHNOLOGY. Downloaded on January 05,2021 at 17:56:16 UTC from IEEE Xplore. Restrictions apply.
 ‘I
‘I
REFERENCES
I \-. ( h d i i i i and J. Hieman, “Reported
,
V. Scott Gordon is
iiiciiiher of the computcr
and information science
Effects of Rapid Prototyping on Industrial faculty at Sonoma State
Soft\i are Quality.” SI+RIY Q I I N ~J., June
I ~
University, where he
1993, pp 93-110. teaches database s)-stems,
programming languages,
2. H. Hoar, .-ipplrcntio7~Pmtotspiiig - ‘1P~n~ei.t and computer literacy fur

’
i
.111111qe711ti7tPeiivprrtrre, ?mierican Manage-
iiieiit ~\ssoc.\lemtiership Publications
Division, Kcv I-orL, 1985.
educators. ,A\ a softMare
developer for TRII‘
Systems, he was iiivohed
in the prototyping of an expert system for coiitrol-
jects involved the practical applicatim of spechcd-
tioiis to sofnt ,Ire teiting and the application of niea-
\ i l ~ l ~ s o i i - ~ ~ ’ e sReading,
ley, Mass., 1975. ling ground-ha\ed mohilc emitters. I le participated sureiiieiit theor! to sriftuare nietrics. IIe i s dso
4. K. .\uer et al., “From Prototvpe to Product?” iii the nork reported in this article while puriuing interested in the develriplnent of iiietrics for softuare
Pi.oc OOPSLI, .AC%lPress, New York, 1989, his I’hll at Colorado State Uiiiversitv. rruie, cohesion, a n d ohicct-oriented sriftv :ire.
p p 4x2-484. Chrclon received a 13s and an from Hieman recen ed a BS in chemical engineering
California State Unibersity, Sacrameiitii, and a PhD friim LVayiic St‘ite University, an MPP in puhhc
froin Colorado State University, all in computer policy from the Lnivcrsih of \Iichipn, a n d an \1S
scicncc. l l c is a meinher o f L‘psiliin Pi Epsilon. and a PhIl, hoth in computer science, from the Uni-
versity of Southn estern Imuislana. He is a senior
I meinher of the :\(;.$I, chair
o f the IEEE-(:S ‘l‘cchnical Council on Software
.\ddress questions a b o u t this article to C~ordiinat CIS Dept., S i i n i i m a Statc Uni\ersity, 1801 E. Cotati .\\e., Engineering’s Chiiinittee on Quantitative Alcthiids,
Riihnert Park. (:.I Y402X; [email protected] or Uieman a t C:S Dept., Coliiradii State University, Fort and chair of the Steering C:ommittce fur the IEEE-
Collins, CO 8052 3 ; [email protected]~iii C S Intern,itiiind S! mporium on SoftLvare Aletrics.

The Tenth Annual Knowledge-Based

Software Engineering Conference
Boston, MA
November 12-15,1995 (tenative)
Call For Papers/Panels/Tutorials
Conference Chair
Howard Reubenstein The Knowledge-Based Software Engineering Conference provides a forum for
The MITRE Corporation researchers and practitioners to discuss application of automated reasoning,
202 Burlington Road knowledge representation, and artificial intelligence techniques to software
Bedford MA 01730- 1420 engineering problems. This conference focuses on specific knowledge-based
[email protected] techniques for constructing, representing, reasoning with, and understanding
Program Chair software artifacts and processes. These techniques may automate human tasks
Dorothy Setliff or support and/or cooperate with humans.
341 Benedum Hall KBSE-95 encourages contributions describing basic research, novel applica-
Department of Electrical Engineering tions, and experience reports. Recent successful contribution topics include:
University of Pittsburgh applications, automating software design and synthesis, education, maintenance
Pittsburgh PA 15261 and evolution, process management, program understanding, requirements,
[email protected] reuse, security, user interfaces and human interaction, as well as validation and
FAX: (412) 624-8003
verification.
PaneU’htorial Chair
Further conference information can be obtained by sending an electronic mail
Chris Welty, Vassar College
[email protected] message to [email protected], or through the World-Wide Web at URL:
http://sigart.acm.org/Conferenceskbse
Demo Chair
KBSE-95 is sponsored by Rome Lab,in cooperation with AAAI, the IEEE
Richard Piazza, The MITRE Corp.
rich @mitre.org Computer Society, and ACM SIGART.

Authorized licensed use limited to: NWFP UNIV OF ENGINEERING AND TECHNOLOGY. Downloaded on January 05,2021 at 17:56:16 UTC from IEEE Xplore. Restrictions apply.