Fast & Flexible

Software Development

Michael A. Cusumano
MIT Sloan School of Management
[email protected]
© 2006
1
2
Different Cultural Orientations
EUROPE: Software as a Science
–Formal Methods, Object-Oriented Design
JAPAN: Software as Production
–Software Factories, Zero-Defects
INDIA: Software as a Service
–Infosys, Tata, Wipro, Satym, Cognizant, Patni
The USA: Software as a Business
–Windows, Office, Navigator, $$$$
3
Problems in Software Development
• Similar problems recurring since the 1960s
• 1969 NATO Report on Software Engineering:
Documented problems in
– requirements, design vs. coding separation
– estimates, monitoring progress, communication
– productivity (26:1), metrics, reliability (bugs)
– hardware dependencies, reuse
– maintenance costs
• Sound familiar??
4
 Proposed Solutions
• Many attempts
– IBM-style software engineering (1960s, 1970s)
– Japanese “software factories” (1970s, 1980s – stable
teams, standard process & tools, reuse)
– SEI Capabilities Maturity Model (1980s to present)
– “Iterative” and “Agile” methods (US, Europe)
• No one process perfect for all projects
– Variations: business models, customer requirements,
application domain, competition, pace of change, local
culture??, etc.
• Common Theme: How balance quality,
features & design flexibility, with cost & speed 5
 Different Process Philosophies
• Waterfall-style (sequential, “Stage-gate”)
versus
• Iterative-style (agile, incremental)
– Spiral
– Rapid Prototyping
– Synch-and-Stabilize (Microsoft, PC makers)
– IBM Rational’s Unified Process & Toolkit
– HP’s Evo Process (short cycles of mini-waterfalls)
– Extreme Programming (XP), SCRUM, AGILE
– Many other variations at companies
6
 Traditional Waterfall Model
(One development cycle)
Requirements 
Functional design 
Detailed module design
Module construction 
Module construction 
Module construction 
Integration/system test 
Module rework (debug) 
Re-test, debugging 
Product release 
7
Frequent Waterfall Process Result
Requirements 
Functional design 
Detailed module design
If modules
 change a lot,
 Module construction  integration fails
and you can
 Module construction 
experience an
 Module construction  infinite defect
  loop.
Integration/system test

8
 Reality: Spectrum of Approaches
Process Choices for Different Projects

Early/Often
High XP
Rapid
Prototyping 
User Feedback Agile or
During Project
Iterative
Uncertainty in Methods
Requirements 
Traditional Waterfall,
Older “Factory-like” Incremental
Approaches
Late/Occasional
Low
1 # of Cycles, Releases Many
9
Adapted from Bill Crandall (HP)
SIterative
YNCH-&-S TABILIZE
Style WATERFALL
(“Synch & Stabilize”) Waterfall-Style
(or other Iterative styles)

Spec, Coding, Testing in Parallel Separate Phases in Waterfall Sequence

Vision Statement & Evolving Spec “Complete” Spec Document

(specification = output, not input) & Detailed Design Before Coding

Prioritized Features Built in 3 or 4 All Pieces of a Product Built

Milestones Simultaneously

Frequent Synchs (Daily Builds) & One “Late & Large” Integration &
Intermediate Stabilizations (Milestones) Test Phase at Project End

“Fixed” Ship Dates & Multiple Attempt to Achieve Feature &

Release Cycles Product “Perfection”

Customer Feedback During Development Feedback as Input for Future Projects

10
 International Comparisons
(2003 IEEE Software, “Software Dev. Worldwide”)
• Survey: Completed in 2002-2003, with Alan MacCormack (HBS),
Chris Kemerer (Pittsburgh), and Bill Crandall (HP)
• Objective: Determine usage of iterative (Synch-&-Stabilize)
versus Waterfall-ish techniques, with performance comparisons
– 118 projects plus 30 from HP-Agilent for pilot survey
• Participants
– India: Motorola MEI, Infosys, Tata, Patni
– Japan: Hitachi, NEC, IBM Japan, NTT Data, SRA, Matsushita,
Omron, Fuji Xerox, Olympus
– US: IBM, HP, Agilent, Microsoft, Siebel, AT&T, Fidelity,
Merrill Lynch, Lockheed Martin, TRW, Micron Tech
– Europe: Siemens, Nokia, Business Objects 11
 “Conventional” Good Practices
India Japan USA Europe etc Total

Number of Projects 24 27 31 22 104

Architectural Specs % 83% 70% 55% 73% 69%

Functional Specs % 96% 93% 74% 82% 86%

Detailed Design % 100% 85% 32% 68% 69%

Code Generators -- Yes 63% 41% 52% 55% 52%

Design Reviews -- Yes 100% 100% 77% 77% 88%

Code Reviews -- Yes 96% 74% 71% 82% 80%12

 “Newer” Iterative Practices
India Japan USA Europe etc Total
No. of Projects 24 27 31 22 104
Subcycles -- Yes 79% 44% 55% 86% 64%
Beta tests -- Yes 67% 67% 77% 82% 73%

Pair Testing -- Yes 54% 44% 35% 32% 41%

Pair Programmer -- Yes 58% 22% 36% 27% 35%

Daily Builds at project start 17% 22% 36% 9% 22%

In the middle 13% 26% 29% 27% 24%
At the end 29% 37% 36% 41% 36%

Regression test each build 92% 96% 71% 77% 84%

13
 “Crude” Output Comparisons
India Japan USA Europe TOTAL
etc.

Projects 24 27 31 22 104

LOC/ median 209 469 270 436 374

Month cf. 389 cf. 245
in 1990 in 1990

Bugs/ median .263 .020 .400 .225 .150

1000 cf. .20 cf. .80
LOC in 1990 in 1990
14
 Observations from Global Survey
• Most projects (64%) not pure waterfall; 36% were!
• Mix of “conventional” and “iterative” common -- use of
functional specs, design & code reviews, but with subcycles,
regression tests on frequent builds
• Customer-reported defects improved -- over past decade
in US and Japan; LOC “productivity” may have improved a little

• Japanese still report best quality & productivity -- but

what does this mean? Preoccupation with “zero defects”? Need
more lines of code to write same functionality per day as US &
Indian programmers?
• Indian projects strong in process and quality -- but not
as strong as their dominance of CMM Level 5 suggests??
15
 Hewlett Packard Pilot Study
(2003 IEEE Software, “Tradeoffs” article)
• Managers – When use iterative or waterfall?
– Survey: 35 responses, 29 projects with complete data
– Median – 170K LOC, with 70K new code; 9-person team, 14
month projects
– 59% applications, 38% systems, 28% embedded
• 74% of variation in defects explained by early
prototypes, design reviews, and
integration/regression testing on builds
– Median project: 40% of functionality complete when first
prototype released and 35.6 defects per million (.04/1000) LOC,
reported by customers in 12 months after release, and 18 LOC
per person day (360/month) 16
 Multivariate Regression Analysis
Some striking results, compared to median:
• Releasing prototype earlier with 20% of functionality 
27% reduction in defect rate
• Integration/regression testing at each code check-in 
36% reduction in defect rate
• Design reviews 
• 55% reduction in defect rate

• Releasing prototype with 20% of functionality 

35% rise in LOC output/programmer
• Daily builds 
• 93% rise in LOC output/programmer
17
 Observations from HP Survey
• Best “nominal” quality from traditional “waterfall” (fewer
cycles & late changes = less bugs, of course!!)
• Best balance of quality, flexibility, cost & speed from combining
conventional & iterative practices

• BUT: Differences in quality between waterfall & iterative

disappear if use a bundle of techniques:
– Short development subcycles (subprojects/milestones)
– Early prototypes to get customer feedback
– Frequent builds to incorporate feedback, changes
– Design/code reviews (check quality continuously)
– Regression tests on each build (check for errors, late changes,
integration problems) 18
 Where Iterative May Work Better
• Fast-paced product markets where technologies,
competition, and product features may have to change a
lot during a project.

• Product or custom projects where customers place very

high emphasis on leading-edge features.

• Custom systems where customers place high emphasis

on responsiveness to their input during a project.

• Product or custom projects that require experimentation

as in lots of short design, build, and test cycles.

• Other? 19
Where Waterfall May Work Better
• Extremely high-reliability systems (product or custom
projects), where functions are very well understood and
no changes in requirements during a project are desired
• Embedded products with hardware constraints that
cannot be easily changed
• Contract software where client requires a detailed
proposal upfront and not possible to limit scope/phases
• Complex projects (product or custom) where parts of
design & coding are outsourced, off-shored, or done in
multiple sites, AND there are weak mechanisms to
synchronize and manage distributed teams
20
 Some Comments on Good Practices
• High-Level Process
• Innovation & Design
• Architecture
• Team Management
• Project Management
• Testing & QA

21
 High-Level Process Strategy

• “You can’t do anything that’s

complex unless you have structure.”

• For creative people, that structure

should be as subtle as possible.

22
 Dave Maritz Test Mgr, Windows 95
Everything that I do here I learned in the
military.... You can't do anything that's complex
unless you have structure .... And what you have
to do is make that structure as unseen as
possible and build up this image for all these
prima donnas to think that they can do what
they like. Who cares if a guy walks around
without shoes all day? Who cares if the guy has
got his teddy bear in his office? I don't care....
[But if] somebody hasn't checked in his code by
five o'clock, then that guy knows that I am going
to get into his office.”
From Microsoft Secrets 23
 High-Level Process cont’d
• Structure:
– Process must fit the product &
market/customer
• Nature of the Product or Service
– Mission critical vs. not; new vs. derivative
– Infrastructure vs. applications, embedded, etc.
– Packaged vs. custom, or multi-system solution
• Nature of the Market or Customer:
– Speed & innovation for leading-edge markets
– Quality & support for enterprise & mass mkts
24
 Innovation & Design Strategy
• Being “slightly out of control” can stimulate
innovative thinking and creativity.
• But too few controls  chaos or “rocket science”

• Iterative a middle approach: Vision statements,

outlines of functional specs, prototypes, early
beta releases, multi-version release mentality, etc.

• Late design changes can be good: help improve

the product -- respond fast to user feedback,
competitors’ moves, or unforeseen changes in the
market & technology. 25
 Architecture Strategy
• Definition:
– How to divide a product into subsystems and
modules/components, and with what interfaces
• Strategy:
– degree of “modular” vs. “integral”
– degree of “open” vs. “closed” (proprietary)
– consider/beware of “Open but not Open”!
• Modular & “horizontal” (feature-oriented)
– de-couples small development teams
– facilitates adding or cutting functionality
– facilitates reuse of components
26
 Architecture Strategy
Tradeoff:

• Investment in architecture for the future versus

new features for the present

• Try to design architectures that will last.

– Rush-jobs “spaghetti” & hard to modify

• Or, plan to evolve architectures incrementally

27
 Bob Lisbonne,
Netscape VP Client Development
When our teams grew beyond a certain point,
they began to resemble a 200-person three-
legged race … That’s why the componentization
or the modularization of the product is so key, so
that ultimately we can get back to lots of small
teams each doing their own thing… and not
getting caught up in one another’s efforts.

From Competing on Internet Time 28

 Team Management Strategy
• Small teams of great people work best
• Large teams can work like small teams (within
limits – not Longhorn!)
• What you need:
– One strong person in charge!
– Map project architecture to product architecture
feature & subsystem teams
– Keep feature teams small  3-8 people.
– Overlap functional responsibilities (vision/scope,
requirements, QA)
– Focus everyone on shipping the product! 29
 Chris Peters,
VP Microsoft Office
Everybody in a business unit has exactly the
same job description and that is to ship
products. Your job is not to write code, your job
is not to test, your job is not to write specs. Your
job is to ship products.... When you wake up in
the morning and you come to work, you say
what is the focus? Are we trying to ship? Are
we trying to write code? The answer is we are
trying to ship.... You're trying not to write code.
If we could make all this money by not writing
code, we'd do it.
From Microsoft Secrets 30
 Project and Feature Team Organization
Microsoft Project Team Structure
Note: Bold type indicates project leaders.

Product Unit Manager

Product Planners

Group
Ggroup
Program Program Program
Manager Manager Manager

Developer Tester Development Test Developer Tester

Team Lead Team Lead Manager Manager Team Lead Team Lead

Developer ---- ----- Tester Developer Tester Developer ---- ----- Tester
Team Lead Team Lead
Developer ---- ----- Tester Developer ---- ----- Tester
Developer ---- ----- Tester
Developer ---- ----- Tester Developer ---- ----- Tester
Developer ---- ----- Tester
Developer ---- ----- Tester Developer ---- ----- Tester
Developer ---- ----- Tester

Developer ---- ----- Tester

User Education Staff

Customer Support Product Specialists

31
 Project Management Strategy
• Avoid sequential “waterfall” schedules (usually).
• Divide long projects into multiple sub-projects or
milestones, of a few weeks or months duration.
• Evolve requirements incrementally: do spec
design, development & testing concurrently.
• Impose a few rigid rules to force frequent
synchronizations and periodic stabilizations.
Synchronize-and-Stabilize!
32
 Project Management cont’d
• Prioritize Rigorously: most important features first
• Schedule “backwards” – people & time
• Let engineers schedule their own tasks
• Managers keep historical data on estimates
• Prototypes and early beta releases = rapid
feedback on designs and quality
• Use historical data to schedule buffer time (for
projects, not individuals) for changes & unknowns
33
Synchronize-&-Stabilize
Synch-and-Stabilize Process
Product Vision/Architecture Design

Functional Specification         

Milestone 1 Milestone 2 Milestone 3

Design Design Design
Code Code Code
Usability test Usability test Usability test
Test Test Test
Daily builds Daily builds Daily builds
Test Test Test
Debug Debug Debug
Integrate Integrate Integrate
Stabilize Stabilize Stabilize

Buffer time Buffer time Buffer time

Alpha release Beta release Feature complete
Beta release

Visual freeze

CODE COMPLETE
Final test
Final debug
Stabilize
Final release

34
 Synch-Up and Debug Daily
• Microsoft Secrets: Few rules, but “military-like”
discipline to force coordination & communication
– Developers can check in when they want
– Each project must build daily
– Check-in at set times; can’t leave until build is OK
– If you break the build, you must fix your code now!
– Penalties for those who break the build

• Minimize build overhead through build team and

automated tools
Objectives:
– Synching-up: 5-20 minutes; Quick test: 30 minutes
– Total Check-in: 5-60 minutes
35
 Testing and QA Strategy
• Try to build-in quality continuously
– design and code reviews
– gates or check points
– continuous customer feedback, etc.

• But continually integrate and test

– Frequent builds (daily, weekly)
– Especially check late design changes

• Test what?
– unit/feature testing, of course
– system integration testing -- from early on! 36
 Testing and QA cont’d
• Automate to test & retest design changes
quickly and frequently.

• But automation never eliminates the need for

people. Someone has to write and rewrite
(update) the automated tests.

• Also need human testers to probe real user

behavior
37
 Testing and QA cont’d
• Post-mortems: what went well, what went
poorly, and what the team should do next time.
• “Eat your own dog food”: first-hand feedback
on products as quickly as possible.
• A few quantifiable metrics: to control and
improve quality as well as monitor key product
and process characteristics.
• Early beta releases: provide feedback on
design as well as quality. 38
 Concluding Comments
• No one “best” software development process
– But “waterfall” less responsive to change
– But using a “bundle” of practices eliminates
differences in quality between waterfall and iterative!

• Process should depend on business, context,

and strategy
– Type of software, customer requirements, team
experience and culture, contract needs, etc.
– Product vs. custom, mass-market vs. niche, individual
vs. enterprise, leading-edge vs. follower, etc.
39
 Main References
• The Business of Software by M. Cusumano (Free Press/Simon & Schuster,
2004)

• Microsoft Secrets by M. Cusumano and R. Selby (Free Press/Simon & Schuster,

1995 and 1998)

• Competing on Internet Time by M. Cusumano and D. Yoffie (Free Press/Simon

& Schuster, 1998)

• Michael Cusumano, Alan MacCormack, Chris Kemerer, and Bill Crandall,

“Software Development Worldwide: The State of the Practice”, IEEE
Software, November-December 2003. (International Comparisons)

• Alan MacCormack, Chris Kemerer, Michael Cusumano, and Bill Crandall,

“Trade-offs between Productivity and Quality in Selecting Software
Development Practices”, IEEE Software, September-October 2003. (HP
Survey)

40