Results-Oriented Software Process Improvement
Jessee Ring Software Quality First 40119 San Carlos Place Fremont, CA 94539 510-915-2353 Fax: 510-573-7464 Email: [email protected] Web: www.sqa1st.com
Results-Oriented SPI: Overview
Software Process Improvement
(SPI): An effective means for improving performance in software development organizations. Results-Oriented SPI: Faster, cheaper appraisals that facilitate more timely and effective improvement action. An overall focus on results-oriented, measurable software process improvement action.
2003 Software Quality First
[email protected] www.sqa1st.com 2
Certification vs. Improvement
Use the process model as a
guide to making improvements . li mprovement projects Al shou l d be l inked to business resu l ts . Don t do anything just because the process model says so . iter ia Using the model as a cr for cert i f ication often leads to irrat iona l behav i or . lts or iented . M etrics . Be resu
2003 Software Quality First
[email protected] www.sqa1st.com 3
Performance
Time
Getting Started on SPI
Usually Begins With An
Appraisal Appraisal have tended to be:
* * * * *
SOFTWARE DEVELOPMENT AND QA ACTIVITIES
Very formal; Done by a team; Results presented in verbiage only; Identify deficiencies, not what to do about them; Approached from the perspective of certification vs. improvement.
[email protected] www.sqa1st.com 4
Questionnaire Interviews Doc. Review
PROCESS MODEL
2003 Software Quality First
Results-Oriented SPI: The Appraisal
) Done by one person ) Must have extensive
experience in multiple areas: t ware Systems and sof deve l opment i ng Test SQA SPI ividua l As an ind contr i butor & manager
[email protected] www.sqa1st.com 5
2003 Software Quality First
Results-Oriented SPI: The Appraisal Process
Reference: CMM Appraisal Framework SEI publication dated Feb., 1995
Plan
Process Data
Write Report
Kick-off
Evaluate Data
Present Results
Collect Data
2003 Software Quality First
[email protected]
Make Judgments
www.sqa1st.com 6
Take Action
Results-Oriented SPI Time Requirements For The Appraisal
One Calendar Week 40 50 hours by the appraiser Kick-off Meeting: 1 hour Participants: 2 3 hours during the week Results presentation: 1 Hour
2003 Software Quality First
[email protected]
www.sqa1st.com
Results-Oriented SPI: Appraisal Data Collection
Questionnaire Based upon the process model being used. Small group discussions Groups can be by job function or cross-functional. Document review
Selected documents, such as requirements specs, design docs, defect reports, CM records, project plan & schedule, etc.
2003 Software Quality First
[email protected]
www.sqa1st.com
Results-Oriented SPI Appraisal Questionnaire Data Analysis
Name Person 1 Person 2 Person 3 Person 4 Person 5 Person 6 Person 7 Number of Yes (1) % Yes Number of No (2) % of No Number of N.A. (3) % of N.A. Number of Don't Know (4) % of Don't Know RM-1 1 1 1 1 1 1 2 6 85.7 1 14.3 0 0.0 0 0.0
[email protected] RM-2 1 1 1 1 1 1 1 7 100.0 0 0.0 0 0.0 0 0.0 RM-3 1 1 1 1 2 2 2 4 57.1 3 42.9 0 0.0 0 0.0 9 RM-4 1 1 1 1 1 2 1 6 85.7 1 14.3 0 0.0 0 0.0 RM-5 1 1 2 2 2 2 3 2 28.6 4 57.1 1 14.3 0 0.0 RM-6 1 1 2 2 2 2 4 2 28.6 4 57.1 0 0.0 1 14.3 2.4 2.4 31.0 64.3 RM-all
2003 Software Quality First
www.sqa1st.com
Achievement Ratings
1 2 -3 4-5 67 8 9 - 10
2003 Software Quality First
[email protected]
Nothing in place. Just starting Making progress, getting some results. Some experience and better results. Fully implemented with a history of success. A model for others.
www.sqa1st.com 10
Results-Oriented SPI Process Model Based Ratings
Requirements Management
Goal 1: System requirements allocated to software are controlled to establish a baseline for software engineering and management use. Goal 2: Software plans, products and activities are kept consistent with the system requirements allocated to software. Rating Average Per KPA
10
10.0
10
Software Project Planning
Goal 1: Software estimates are documented for use in planning and tracking the software project. Goal 2: Software project activities and commitments are planned and documented. Goal 3: Affected groups and individuals agree to their commitments related to the software project.
2003 Software Quality First
[email protected] www.sqa1st.com 11
3 5 7
5.0
Results-Oriented SPI: Process Model Based Ratings Graphical Presentation
KPA Ratings
Scale of 1 10 with 10 Being Highest
10 9 8 7 6 5 4 3 2 1 0
P E F IS M RM PR TO M OP SP SQ
Software XYZ - Year 1
2003 Software Quality First
[email protected] www.sqa1st.com 12
SP
SC
OP
Software XYZ - Year 2
SP
TP
IC
Results-Oriented SPI: Appraisal Results
Summary of Results
Overall Strengths Major Issues
Consequences Recommendations
Detailed Results
Per KPA Strengths Issues Recommendations
2003 Software Quality First
[email protected] www.sqa1st.com 13
Results-Oriented SPI: Appraisal Results - Consequences
Consequences" of the
issues that were found are described. Makes the issues of more immediate interest. Answers the question: So what?. Justifies the allocation of resources.
2003 Software Quality First
[email protected] www.sqa1st.com 14
Results-Oriented SPI: Appraisal Results - Recommendations
Recommendations are a critical element of the Results-Oriented SPI approach. By including them in the Appraisal Report, people become focused on Improvement action. It also speeds up the whole process since the organization doesnt have to wait to find out what to do.
2003 Software Quality First
[email protected]
www.sqa1st.com
15
Example Of Results Reporting
Appraisal Results
Issue: Peer reviews are unstructured; no data or records are kept. Consequences: Peer review effectiveness is unknown but is probably low. Recommendation: Train participants on real inspections vs. walkthroughs (with roles, required preparation, record keeping). Track all items found in inspections to closure. Use data to evaluate inspection effectiveness and make improvements.
2003 Software Quality First
[email protected] www.sqa1st.com 16
Results-Oriented SPI: Metrics Plan
Two types of metrics: product & process
Product Metrics Number of open defects by severity Defect density Defect arrival rate Total size of the software Number of lines of code added, deleted, modified Number of new features and/or fixes contained in a new version Reliability growth Complexity of the software
2003 Software Quality First
[email protected]Process Metrics Test effectiveness (number of escaped bugs) Percent of bugs found by test and inspections. Amount of time to find a bug by test and by inspections. Cycle time of regression testing. Test coverage Inspection coverage Process maturity ratings over time
17
www.sqa1st.com
Results-Oriented SPI Sample Metrics
160 140
Number of Defects
Open Defects at Time of Release
120 100 80 60 40 20 0
Version Version Version Version Version Version Version 1.1 1.2 1.3 1.4 1.5 1.6 1.7
Crash, lock-up, etc. Functionality - minor Functionality - major Cosmetic Functionality - medium
2003 Software Quality First
[email protected]
www.sqa1st.com
18
Results-Oriented SPI: Sample Metrics
200 180 160 140 120 100 80 60 40 20 0
Defect Aging
Number of Defects
Over 30 Days
Crash, lock-up, etc. Functionality - minor
Over 60 Days
Over 90 Days
Functionality - major Cosmetic
Functionality - medium
2003 Software Quality First
[email protected]
www.sqa1st.com
19
Results-Oriented SPI: Sample Metrics
Software Reliability Growth
1000 900 800 700 600 500 400 300 200 100 0
Hours
Jan. Feb. Mar. Apr. May June July Aug. Sep. Oct. Nov. Dec. Mean time between failure
2003 Software Quality First
[email protected] www.sqa1st.com 20
The Need For Software Process Improvement
Mars Climate Orbiter
Lost because of failure to convert from English to metric units in the navigational (orbiting) program.
Mars Polar Lander
Lost because of a missing line of code that caused the braking engines to shut down prematurely.
Result
$360 million out-of -pocket cost Indefinitely postpone all future launches to Mars Address shortcomings at JPL in areas ranging from systems analysis and testing to staffing and communications.
2003 Software Quality First
[email protected] www.sqa1st.com 21
Results-Oriented SPI: Summary
Use one very experienced person to perform a
software process appraisal. Give numeric ratings to achievement levels of process model requirements. Use the process model as a guide; not a mandate. Link all process improvements to business results. Use process maturity ratings to guide improvement actions, not as a final goal. Dont publish them. In the process appraisal report, identify consequences of the issues that were found, and recommendations for addressing them. Use metrics to manage the improvement actions. KISS
[email protected] www.sqa1st.com 22
2003 Software Quality First
Results-Oriented SPI: Benefits
Faster, less expensive appraisal. A sense of urgency. Can start on improvement immediately after appraisal (dont
have to wait on another team to formulate a plan). Can track process maturity levels over time graphically. Improvement actions are focused on the issues that will impact the bottom line. Participant see it as being worthwhile. Greater likelihood of upper management support for SPI. All of which leads to a successful improvement program.
2003 Software Quality First
[email protected] www.sqa1st.com 23
Results-Oriented SPI
2003 Software Quality First
[email protected]
www.sqa1st.com
24
