Module 47. Basic Design of Experiments (DOE)
Module 47. Basic Design of Experiments (DOE)
UNCLASSIFIED / FOUO
National Guard
Black Belt Training
Module 47
Basic Design of
Experiments (DOE)
UNCLASSIFIED / FOUO
UNCLASSIFIED / FOUO
UNCLASSIFIED / FOUO
TOOLS
ACTIVITIES
•Brainstorming
• Develop Potential Solutions
•Replenishment Pull/Kanban
• Develop Evaluation Criteria
•Stocking Strategy
• Select Best Solutions
•Process Flow Improvement
• Develop Future State Process Map(s)
•Process Balancing
• Develop Pilot Plan
•Standard Work
• Pilot Solution
•Quick Change Over
• Develop Full Scale Action/
•Design of Experiments (DOE)
Implementation Plan
•Solution Selection Matrix
• Complete Improve Gate
•‘To-Be’ Process Mapping
•Poka-Yoke
•6S Visual Mgt
•RIE
Note: Activities and tools vary by project. Lists provided here are not necessarily all-inclusive. UNCLASSIFIED / FOUO 2
UNCLASSIFIED / FOUO
Learning Objectives
Learn benefits of DOE methodology
Discuss differences between DOE and trial and
error (one-factor-at-a-time) approaches to
experimentation
Learn basic DOE terminology
Distinguish between the concepts of full
and fractional factorial designs
Use Minitab to run and analyze a DOE
Use results of DOE to drive statistically
significant improvements
Helicopter Simulation
Phase One
UNCLASSIFIED / FOUO 4
UNCLASSIFIED / FOUO
UNCLASSIFIED / FOUO 5
UNCLASSIFIED / FOUO
Exercise: Constraints
Project Mission: Find the combination of factors that
maximize flight time
Project Constraints:
Budget for testing = $1.5 M
Cost to build one prototype = $100,000
Cost per flight test = $10,000
Prototype once tested can not be altered
See allowable flight test factors and parameters on the
next page
UNCLASSIFIED / FOUO 6
UNCLASSIFIED / FOUO
UNCLASSIFIED / FOUO 7
UNCLASSIFIED / FOUO
UNCLASSIFIED / FOUO 9
UNCLASSIFIED / FOUO
Introduction to DOE
UNCLASSIFIED / FOUO 10
UNCLASSIFIED / FOUO
How Do We Learn?
1. Significant Event 2. Somebody Sees It
How Do We Improve?
By creating significant events and observing them, we
can obtain knowledge faster
That is basically what occurs in a designed
experiment
Let‟s look at an example of these two things occurring
(significant event and perceptive observer)
simultaneously
Champagne Example
Wine – The fermented juice of
fresh grapes used as a beverage.
Wine has been in existence since
the beginning of recorded history
Champagne – A clear, sparkling
liquid made by way of the second
fermentation of wine. First
discovered by a French monk in
the late 1600s
People
Responses related to
Material
producing a product
Job Description
Marketing
Candidate Pool
DOE was originally used for manufacturing quality applications - it has now
expanded to many other areas where performance characteristics are of interest
Basic Design of Experiments UNCLASSIFIED / FOUO 18
UNCLASSIFIED / FOUO
Methods of Experimentation
Experimentation has been used for a long time.
Some experiments have been good, some not so
good
Our early experiments can be grouped into the
following general categories:
1. Trial and Error
2. One-Factor-at-a-Time (OFAT)
3. Full Factorial
4. Fractional Factorial
5. Others
How many more runs would you need to figure out the best configuration of
variables?
How can you explain the above results?
If there were more variables, how long would it take to get a good solution?
What if there‟s a specific combination of two or more variables that leads to the
best mileage (the optimum)?
Results
Miles per Gallon as a Function of Speed and Tire Pressure
75
17
70
How would we find
65
Speed (mph)
26 28 30 32 34 36 38
Tire Pressure (lbs.)
Basic Design of Experiments UNCLASSIFIED / FOUO 22
UNCLASSIFIED / FOUO
One-Factor-at-a-Time (OFAT)
While OFAT is simple, it is inefficient in determining
optimal results:
Unnecessary experiments may be run
Time to find causal factors is significant
Don‟t know the effects of changing one factor while
other factors are also changing (no model)
Inability to detect or learn about how factors work
together to drive the response
Full Factorial
Minimum number of tests for a full factorial experiment: Xk
X = # of levels, k = # of factors
Factors
Level 2 3 4
2 4 8 16 # of Tests
3 9 27 81
Fractional Factorial
Looks at only a fraction of all the possible combinations contained in
a full factorial.
If many factors are being investigated, information can be obtained
with smaller investment.
Resources necessary to complete a fractional factorial are
manageable.
Limitations - give up some interactions
Benefits
Economy
Speed
Fewer runs
1511
2 III Note: A deeper discussion of design resolution is beyond the scope of the lesson. The
content, above, is intended to only provide a brief explanation of the design resolution term.
55 65 85 91 30 35
27.8
26.8
55 65 85 91 30 35
3.0
2.5
is affecting standard
2.0
1.5 deviation
1.0
Speed Octane Tire Pressure
Main Effects Plot (data means) for Mileage Main Effects Plot for Standard Dev
55 65 85 91 30 35 55 65 85 91 30 35
27.8 3.0
26.8 2.5
Standard Dev
Mileage
25.8 2.0
24.8 1.5
23.8 1.0
Speed Octane Tire Pressure Speed Octane Tire Pressure
Main Effects Plot (data means) for Mileage Main Effects Plot for Standard Dev
-1 1 -1 1 -1 1 -1 1
28
3.0
27
2.5
Standard Dev
Mileage
26
2.0
25
1.5
24
1.0
Driver Radio Driver Radio
Benefits of DOE
Determine input settings which optimize results and
minimize costs
Quick screening for significant effects
Obtain a mathematical model relating inputs and results
Reduction in the number of tests required
Verification of the statistical significance of results
Identification of low-impact areas allows for increased
flexibility/tolerances
Standardized methodology provides a directed approach
DOE Review
What does DOE offer us that trial and error
experimentation and OFAT do not?
What are the differences between full and fractional
factorial DOE‟s?
What is the minimum number of runs required for a
2-level, 3-factor full factorial experiment?
Factors
Levels
Dollars spent on training 100
300
Number of Jets 50 55
Number of Employees 800
900
Basic Design of Experiments UNCLASSIFIED / FOUO 51
UNCLASSIFIED / FOUO
>
45
30
15
0
Mean
-1 1 -1 1
Employees %Overbooked
60
45
30
15
0
-1 1 -1 1
2
Mean
-1 1 -1 1
Employees % Overbooked
8
-1 1 -1 1
B
* - BD is the interaction
A between the factors Jets
AB and %Overbooked. Pareto Chart for
AC Mean Min Late
CD
0 10 20 30 40 50 60 70 80 90
Standardized Effect
AB
BD Pareto Chart for
CD
StdDev Min Late
D
A
0 1 2 3 4
Standardized Effect
Estimated Effects and Coefficients for Mean Min Late (coded units)
This data shows
Term Effect Coef SE Coef T P
Constant 30.42 0.3191 95.34 0.000 Analytically that
Training Dollars
Jets
-0.71
-0.77
-0.35
-0.39
0.3191
0.3191
-1.11
-1.21
0.319
0.279 Employees and
Employees -53.89 -26.94 0.3191 -84.44 0.000 the Jets*%
%Overbooked -1.28 -0.64 0.3191 -2.01 0.101
Training Dollars*Jets 0.68 0.34 0.3191 1.06 0.337 Overbooked
Training Dollars*Employees
Training Dollars*%Overbooked
0.12
1.13
0.06
0.56
0.3191
0.3191
0.18
1.77
0.861
0.138 interaction are
Jets*Employees -0.81 -0.41 0.3191 -1.27 0.259 statistically
Jets*%Overbooked 2.14 1.07 0.3191 3.36 0.020
Employees*%Overbooked -0.10 -0.05 0.3191 -0.16 0.881 significant.
Jets had a significant impact on Stdev Min Late. As seen from its
Effect, increasing Jets decreases Stdev.
Term Effect Coef SE Coef T P
Jets -5.930 -2.965 0.7241 -4.09 0.009
1 3
1 3
4
5
7 50
Boxplot of Baseline Data, New Data
40
30
Data
20
8
9
10
11
Helicopter Simulation
Phase Two
UNCLASSIFIED / FOUO 86
UNCLASSIFIED / FOUO
UNCLASSIFIED / FOUO 87
UNCLASSIFIED / FOUO
Exercise: Constraints
Project Mission: Find the combination of factors that
maximize flight time
Project Constraints:
Budget for testing = $1.5 M
Cost to build one prototype = $100,000
Cost per flight test = $10,000
Prototype once tested can not be altered
See allowable flight test factors and parameters on the
next page
UNCLASSIFIED / FOUO 88
UNCLASSIFIED / FOUO
UNCLASSIFIED / FOUO 89
UNCLASSIFIED / FOUO
UNCLASSIFIED / FOUO 91
UNCLASSIFIED / FOUO
Takeaways
Types of experiments – Trial and Error, OFAT, DOE
Introductory DOE terminology
Benefits of full factorial vs. fractional designs
How to use Minitab to design, run, and analyze a DOE
Use DOE results to drive statistical improvements
UNCLASSIFIED / FOUO 93
UNCLASSIFIED / FOUO
References
Schmidt & Launsby, Understanding Industrial
Designed Experiments