Applied System Engineering

Dr. Andreas Edom

8. Testing
Agenda

1 – General Test Procedure

2 – User Test

3 – Enclosure Protection Test

4 – Hardware-in-the-Loop Test (HiL)

5 – Highly Accelerated Lifetime Test (HALT)

6 – Qualification Test Program

7 – Production Sampling Test

General Test Procedure

 Purposes of a test:
Flowchart of system verification
 to verify, if the product meets all and corrective action
performance requirements as stated in the
Preparation for test
Technical Specification
 to verify – as much as possible – realistic
Test execution
customer operations

 Preparatives for test Data processing

 selection of test site (environment)

Are Yes
 shall be characteristic for user operation requirements
met?
EIS

normal ground desert vacuum No

tropics
arctic high altitude Identification of problem
air water

 selection (and maybe acquisition) of test facilities Development of system

modification kit
set of
parts to
form a
test environmental technical
chambers control system Installation of system
solution

modification kit
mock-ups exciters

3
General Test Procedure

 more Preparatives for test

 selection (and maybe acquisition) of handling equipment and software
crane remote actuators data acquisition
control system (DAS)

 selection of supply support items

spare plasticine
consumables fittings rigs
parts
tapes liquid nitrogen

 selection of environmental resources

heat water wind power air condition

 selection of approved test procedures

 documented common company practice like testing guidelines and manuals
 consideration of mandatory test procedures requested by regulatory institutions

4
General Test Procedure
system,
subsystem or
Request for Test (RFT) component
test
Contents:  Category and title
 Brief description of test front
 Test location page
 Testing period (start, end)
 Test objectives
 Test set-up requirements
-- values of environmental parameters
-- sketch of test set-up (with dimensions)
-- sketch of sensor arrangement
 Measurement campaign
 Time planning
-- build-up, measurements, visualisation/diagrams, report generation
 Reporting
-- delivery dates of first results / final report
X weeks or months after end of test 5
General Test Procedure
Reference: X##YZ#######
Request for Test
Issue: 1

Example: Front/Cover page of Date: DD.MM.YYYY

Page: 1

an RFT for component test

‘Component Test’

A3XX PtoF
Prototype Test of Courier Module Air Outlet

Brief description:
In order to comply with comfort requirements inside the Courier Area an experimental
evaluation and possible optimisation of associated air outlets is necessary. The Courier
Module Air Outlet (CMAO) is installed inside the backwall of the courier module close to
aircraft center line. Due to the (nominal) flow of ## l/s and the exposed location this outlet has
a major influence on the Courier Area air circulation. The test program comprises the

Note:
investigation of
 Outflow geometry
 Outflow velocity profile at a distance of ### mm from outlet rim by means of a two-
As it is an official document, dimensional measuring point arrangement
 Static pressure in the feed zone of the air outlet

the RFT has to be  Acoustical characteristics

The pressure measurement will also serve as input for the pressure loss coefficient of the
CMAO component in a system simulation model.

 checked by a colleague
Test Locations: Airflow Laboratory, Acoustic Laboratory, Hamburg
of the same department,
Test Campaign Reference: X##YZ#######
 approved by the supervisor
Schedule:

 released by the project manager Start of testing: CW49 YYYY

End of testing: TBD
Report available: 2 weeks after end of test

Name Department Date Signature

Compiled Design Engineer 1

Checked Design Engineer 2

Approved Supervisor

Released Project Manager

6
General Test Procedure
Final Test Report
shall contain:  reference to related RFT
 description of all test conditions
 procedures followed when conducting the test
 identification of data sources
 results and interpretation of data analyses
 recommendations for corrective action
and improvement

7
User Tests
Iterative Prototyping and User Testing
 Best way to design usable products:
The developed product is not only tested for functionality,
but also for usability by letting potential users handling the product
 can be applied during detailed design phase already
with components produced by rapid prototyping
 offers answers to the following crucial questions:
 Are special features really worth their production effort?
 Will the user accept and appreciate the special features?
 provides estimation of user error rates
 gives important input to upcoming operating manuals
 gives important information on how to train people on the product in the future

The equivalent in software development: Beta version testing

8
User Tests
Panel test
a group of test persons performs evaluation
of a system in operation
at the same place
at the same time
 During the test the test persons fill in questionnaires
Examples:  Evaluation of passenger seats and/or thermal comfort in aircraft cabins or railway cars
 Emergency evacuation of aircraft via escape slides

For user tests it is necessary to capture

 desired target group age
 enough individuals with representative mix gender
body mass index (BMI)

Advantages:  discovering and avoiding misguided design cost savings

 achieving user-friendly products
9
Enclosure Protection Test

 Objective: Assessing the ability of the product or its enclosure

to offer protection of the operating parts against
 solid foreign objects
 ingress of dust
 ingress of liquids (water)
-- drip
-- spray
-- jet
-- immersion
IPX4 testing of an
 access to hazardous parts electrical enclosure

 The ingress protection ratings are identified as IP XY

 the first digit X indicates the level of protection against solids and dust (range 1 to 6)
 the second digit Y indicates the level of protection against water (range 1 to 9)
Examples for test objects: aircraft electronics
high voltage transformers
cell phone cases

10
 Enclosure Protection Test
IP classes (as per ISO 20653)
 IP first digit:  IP second digit:
protection from solid objects or dust protection from moisture or liquids
IP0Y No protection IPX0 No protection

IP1Y Protection against solid objects > 50 mm IPX1 Protection from vertically dripping water

IP2Y Protection against solid objects > 12.5 mm IPX2 Protection from sprays / dripping water
up to 15° from vertical
IP3Y Protection against solid objects > 2.5 mm
IPX3 Protection from spraying water up to 60° from vertical
IP4Y Protection against solid objects > 1 mm
IPX4 Protection from spraying water (all directions)
IP5Y Protection against dust (limited ingress allowed, but no hazard)
(limited ingress allowed, but no harmful deposit)
IPX5 Protection from low pressure water jets (all directions)
IP6Y Fully protected from dust (limited ingress allowed, but no hazard)
(dust tight)
IPX6 Protection from powerful water jets (all directions)
(limited ingress allowed, but no hazard)

IPX7 Protection from temporary immersion in water

IPX8 Protection from continuous immersion in water

Examples: up to 1 m below surface
 Smartphones are at least IP54 certified IPX9 Protection from water applied with high pressure
 Electric outdoor torches are at least IP65 certified and high temperature
11
Hardware-in-the-Loop Test (HiL)
 Test samples are embedded into a closed loop
together with a simulated environment sensors electrical
heater/
 Environment is replaced by a precise
P1, T1, m1
humidifier

computer model (for example Matlab/Simulink) air supply

line

that is running in real time CAN* sensors

P2, T2, m2, x2

 HiL testing is often used in early stages of cabin

model
control device development

Advantages:
* Controller Area Network
 Extreme operation modes are possible, that
-- would damage a real environment
-- would be harmful for people in a Signal Processing &
Virtual
real environment
Conditioning
Environment
Simulator

 HiL helps enhancing the safety of a product

 Special operation modes can be applied, when
trouble-shooting is needed on problems in the Test sample

real world
 Independent testing of individual components
12
Highly Accelerated Lifetime Test (HALT)
 Method to accelerate ageing of test sample in operation
 provoking cumulative damage that is comparable
to damage to be expected during lifetime of product
 stimulating failures in order to detect limits of operation
 Assumption: Product shows the same failure mode in a short time under high stress,
as it would show in a longer time under real life-cycle stress
 Procedure: stepwise increase or cycling of load  temperature and vibration
 load increase continues to loads that exceed far beyond environmental conditions
test limits are set by the product itself when failure occurs
 Two types of limit can be explored:
-- operational limit test sample ceases operation (reversible or soft limit)
-- destruction limit test sample disintegrates or melts (irreversible or hard limit)
 In the case that several component design alternatives are pursued in the detailed
design phase, HALT can be useful for comparative tests of corresponding test samples.

Objectives:  to find, understand and eliminate possible overstress and wear-out failures
 to increase the distance between design load and failure limit
13
Highly Accelerated Lifetime Test (HALT)
 Five types of HALT
1 & 2 Thermal step load

Temperature
cold
test until failure steps hot
steps

Time

3 Rapid thermal transition cycling

Temperature
generally 5 cycles

Time

4 Vibration step load Reduction to low level vibration

Vibration for short functional check,
test until failure then increase to next step level
 helps to detect failure that is
provoked, but not detectable,
at higher level
Time

5 Combined environment:
Temp

Rapid thermal transition cycling

and vibration step load
Vib

generally 5 cycles Time 14

Highly Accelerated Lifetime Test (HALT)

 Required number of test samples for a representative test campaign:

not less that 15 samples minimum 3 per test
 After the tests: Root cause analysis
Error Type of load Root cause Measures

Cable insulation insufficient provide edges

Vibration
rubbed through burring with radius 2mm

Resistor on
vent holes double
PCB burned Heat
too small vent hole area
through

 Example of a chamber for HALT

-- cooling with liquid nitrogen down to -100 °C
-- heating with hot air up to 200 °C
-- rate of temperature change: 60 K/min
-- broad band vibration: 5 to 10000 Hz
with 6 degrees of freedom
(along and around 3 axes)

15
Highly Accelerated Lifetime Test (HALT)

Advantages Disadvantages
 Fast identification of design errors  Sometimes failures cannot be
and possible failure modes classified, if they are due to
 Faster than usual endurance test -- accelerated aging
(only 2 to 5 days) or
faster Time to Market -- just overcharge
 Lower probability of failure  Service life or MTBF cannot be
during later qualification tests determined
 Generalised process  Failure cases are restricted to
heat and vibration loads
no need for preparation of
special test program

Note:
HALT does not replace qualification tests!

16
Highly Accelerated Lifetime Test (HALT)

Acceptance Test vs. HALT

 checking if product meets the  stressing the product above

technical specification technical specification
to detect weak points
 test is tuned for a distinct  general test
product
 successful when no failure  successful when failure
detected (and prevented)
 pass / fail test  multi-purpose fail test

17
Qualification Test Program
 Definition:
Determined comprehensive series of tests, stated in the contract
officially agreed between customer and producer,
by which the functional, environmental and reliability performance
of a component or a system is evaluated
 Qualification tests follow the standards of the product related industry
 Qualification tests are performed once with the prototype to qualify the design
Test loads and test durations are formulated in a such way that,
if the test sample passes the tests, with a high level of confidence
identical products produced under identical conditions will survive
the expected service environments
 Qualification tests have to be repeated immediately after
 any design change
 any manufacturing change

18
Qualification Test Program
Examples of test conditions in the aircraft industry
with the tested equipment fully operative under the given test conditions!
 Severe operating temperature conditions
-- Low temperature conditions ambient temperature: -55°C
test duration: 2 hours after steady-state
-- High temperature conditions ambient temperature: 70°C
test duration: 2 hours after steady-state

 Thermal shock
-- 5 cycles at temperature of -20°C to 105°C ± 5°C

 Overpressure
-- 8 cycles: 3x 5min dwell at Pproof,1 Pproof,2
Pproof,1
4x 5min dwell at Pproof,2
Pressure

1x 30min dwell at Pproof,2

ramp 0.2x Pproof,2 /min
Time [min]

19
Qualification Test Program

 Operational vibrations
according to g=1.68

standard random
vibration test curve
Vibrating table
 Constant acceleration
Acceleration for arbitrarily mounted equipment: 10.5 g

 Sand and dust

Blowing a mixture of particles with specific sizes

 Fungus resistance
Exposure to warm and very humid environment
containing inorganic salts material test only,
equipment is
 Salt non-operative
Multiple exposures to salt fog and drying
during 500 hrs accelerated corrosion test

20
Qualification Test Program

 Explosion proofness
Exposure to gaseous mixture of combustibles
with operating equipment submitted to any kind of actuation (like knobs, buttons ...)
no ignition, no spark
 Fire proofness
Exposure to bunsen burner for 12 seconds
equipment must be operative during first 5 minutes
and non-hazardous for at least 15 minutes
 Electrostatic discharge (ESD) susceptibility
Exposure of external surface, control elements
and displays to ESD pulses of 8000 VDC

 Insulation resistance Electrostatic discharge test generator

At least 5 GΩ when exposed to 500 VDC

21
Production Sampling Test
 Regularly repeated test procedure with large number of test samples
taken directly from the production line
to check if functional and geometrical parameters of the product continue
meeting the requirements of the technical specification
Objective:  to affirm that the product has been keeping the same level of quality
throughout the production process, as it was stated when passing the
qualification tests
 With time -- the production process can change due to
 wear-out of tooling and jigs
 worsening of manufacturing processes and staff
 misalignment of tooling and fixtures
-- the properties of supplied raw material might change
Measures must be taken against unexpected variations and degradations

 Test method: A certain parameter of the retrieved samples is measured

and evaluated statistically
Examples for measured parameters: length, diameter, tensile strength, hardness,
surface finish, weight
22
Production Sampling Test
Distribution of a measured parameter

Frequency of
occurrence
 In large quantities measurements of technical
parameters follow the Gaussian distribution curve variance
specified by -- mean value (location factor)
which is the target value LTV UTV

-- variance (form factor)

lower target upper Measured
 the further values spread around the mean value, threshold
value
value threshold
value
parameter

the higher the variance and the broader the curve

Deficient processes
measurements at different times t1 and t2
t1 A shift in the mean and/or
process t2 process t1 an increase in variance
not not of key parameters can result
controlled capable in early failure of the product!
t2

change
change
in mean
in variance
23
Production Sampling Test

Target: Production process shall be

 capable and
 controlled
LTV
UTV

offset due to
misalignment or
calibration error process is capable,
but not controlled

process is not capable, process is not capable

but controlled and not controlled

24
Conclusion I
 In the development process careful testing is needed
a) to choose the best design variant, if more than one option under consideration
b) to check if the newly developed product fullfils all the requirements
as stated in the technical specification
system level
 case a) refers to the detailed design phase
where validation of requirements occurs
 left leg of the Vee model
(Definition of validation: stating the validity or correctness)
 case b) refers to the prototype production phase
where verification of requirements occurs
component level
 right leg of the Vee model

 In big companies tests are done by departments especially dedicated to testing

or even outsourced to other companies specialised on certain types of tests.
Therefore documentation request for test
is mandatory test report
to ensure -- the traceability of the development process
-- common understanding of what and how it has to be tested
25
Conclusion II
Characteristics of test types

Lifecycle usually
Type Frequency
phase outsourced?
detailed
design *,
User test once, twice no
prototype
production
Enclosure prototype
protection once yes
production
test
Hardware- several detailed
in-the-loop no
times design *
test
Highly
prototype
accelerated once yes
production
lifetime test

Qualification prototype
once yes
tests production

prototype
Acceptance production,
once, twice no
test serial
production
at regular
Production serial
time no
sample test production
intervals

* performed with supplier components

at purchaser‘s or supplier‘s premises
26