Validation of Methods & Laboratory
Systems
Yusmiati, M.Kes
Workshop “Management and Development of Clinical Laboratory”
Kongres Nasional XIV Himpunan Kimia Klinik Indonesia
21-24 April 2016
Hotel Bumi, Surabaya
VALIDATION : WHAT ?
VALIDATION VS VERIFICATION
- Non standard method
- Existing method with defined
- Laboratory designed by developed
performance
method
- Existing method used after repair
- Modified validated method
VALIDATION VERIFICATION
Before use as diagnostic test method Before use as diagnostic test method
COMPARE
COMPAREperformance
performance
DEFINE performance characteristics
characteristics,
characteristics,with
withspecifications
specifications
Alvarez, et al. 2011. Modern Approaches to Quality Control
VALIDATION : WHY ?
Why is it necessary to validate method performance when
the manufacturer has already performed extensive studies?
To demonstrate that the method Provide reliable
performs well under the operating test results for our
conditions of our laboratory. patients.
There are many factors that can affect method performance :
Different lots of calibrators and Effects of shipment and storage
reagents
Local climate control conditions
Changes in supplies and suppliers
of instrument components Quality of water
Changes in manufacturing from Stability of electric power
the production of prototypes to
final field instruments Skills of the analysts WHY
www.westgard.com
Method validation is about error assessment -
that's the secret ! (James O. Westgard)
Random Error / Imprecision Constant Error
Systematic Error / Inaccuracy Proportional Error
www.aacc.org/publications/cln/articles/2013/september/total-analytic-error
VALIDATION : HOW ?
Random Error (RE) : Systematic Error (SE) :
Affects precision Affects accuracy
May be caused by (for example) : Types of SE :
- variability in volume of sample or - Proportional --> indicated by slope
reagent delivered - Constant --> indicated by intercept
- Changes in environment - Proportional + Constant -->
- Inconsistent handling of materials combination of both
Estimated by :
Caused by (examples) : bad
- Standard deviation (SD)
calibrators, bad reagents,
- Coefficient of variation (CV)
interference
- Correlation coefficient (r)
VALIDATION : HOW ?
Accuracy
RELIABILITY
Precision
Total Analytical Error - TE
TE = 2SD + bias
Professional Practice in Clinical Chemistry
VALIDATION : HOW ?
Steps in Method Validation
• Define Goals
• Error Assessment
• Compare error vs
analytical goal
Total Allowable Error - TEA
TEA is the total error permitted, based on:
- Medical requirements
- Best available analytical method
- Compatible with proficiency testing expectations
Source: CLIA, https://www.westgard.com/biodatabase1.htm, etc.
GOAL: Total Analytical Error < Total Allowable Error
TE < TEA
Determined
- Method specific
- Measured at various Medical decision levels (Xc)
Professional Practice in Clinical Chemistry
What is the first thing to do??
www.westgard.com
VALIDATION : HOW ?
1st: Selection
Application Methodology Performance
characteristics characteristics characteristics
Factors that determine Factors that in principle Factors that in practice,
whether a method can be contribute to best demonstrate how well a
implemented in a Lab. performance method performs
Cost per test, type of Traceability of standards, Reportable range,
specimen, turn around chemical principle, precision, recovery,
time, workload, operator measurement principle, interference, accuracy,
skills, etc etc. etc.
Validation/
Verification
Westgard JO. Basic Method Validation, 3rd Ed. 2008
VALIDATION : HOW ?
Validation Guideline
Consistent with
Manufacturer's claims
VALIDATION : HOW ?
A Validation Puzzle
Non-FDA approved/LDT FDA- LDT
approved/cleared
CLIA CAP CLIA CAP
Accuracy + + + +
method comparison
Precision + + + +
replication experiment
Reportable range + + + +
linearity experiment
Establish reference range + + + +
Analytical sensitivity Not Not + +
Limit of detection study required required
Analytical specificity Not Not + +
Interference study required required
Recovery to determine proportional Not Not + Not
interferences required required required
Westgard JO. Basic Method Validation, 3rd Ed. 2008
VALIDATION : HOW ?
Performance
Validated by :
characteristic :
Imprecision
Replication study --> controls, samples
(random error)
- Comparison of methods
Inaccuraccy
- Interference (constant systematic error)
(systematic error)
- Recovery (proportional systematic error)
Sensitivity LoB, LoD, LoQ experiment
Reportable range Linearity experiment
Verified by testing samples from healthy
Reference intervals
people
Ready to validate?
Validation case study
There is a change in Cholesterol reagent and we are going to validate
whether the performance of this new reagent meets the requirement
of our lab.
- replication study
- method comparison
- interference study
- recovery study
- linearity study
Additional studies not related to cholesterol:
- analytical sensitivity
- verification of reference range
VALIDATION : HOW ?
Replication Study
At least 20 data, using control Day Control 1 Control 2
1 203 240
materials or samples (generally two or 2 202 250
three materials at concentrations that 3 204 235
are of importance) 4 201 248
5 197 236
6 200 234
7 198 242
Within run, between run, between day. 8 196 244
9 206 243
10 198 242
11 196 244
12 192 243
Calculate using excel, or other tools 13 205 240
(https://www.westgard.com/mvtools.htm) 14 190 233
15 207 237
(Mean, SD, CV).
16 198 243
17 201 231
18 195 241
19 209 240
20 186 249
CLSI EP5-A Evaluation of Precision Performance of Clinical Chemistry Devices; Approved Guideline
VALIDATION : HOW ?
Replication Study
Day Control 1 Control 2
At least 20 data, using control 1 203 240
materials or samples (generally two or 2 202 250
3 204 235
three materials at concentrations that 4 201 248
are of importance) 5 197 236
6 200 234
7 198 242
8 196 244
Within run, between run, between day. 9 206 243
10 198 242
11 196 244
12 192 243
Calculate using excel, or other tools 13 205 240
(https://www.westgard.com/mvtools.htm) 14 190 233
(Mean, SD, CV). 15 207 237
16 198 243
17 201 231
CV = SD/Mean * 100 % 18 195 241
19 209 240
20 186 249
Mean 199.20 240.75
CV range for cholesterol: < 4.5 % SD 5.84 5.22
CV % 2.93 2.17
CLSI EP5-A Evaluation of Precision Performance of Clinical Chemistry Devices; Approved Guideline
VALIDATION : HOW ?
Replication Study
https://www.westgard.com/mvtools.htm
VALIDATION : HOW ?
Method Comparison
At least 40 samples should be tested by the two methods.
Should be selected to cover the entire reportable range of the method and
represent the spectrum of diseases expected in routine application of the
method.
A minimum of 5 days is recommended, but it may be preferable to extend
the experiment for a longer period of time.
Create a scatter plot (plot the means of duplicates) if done in duplicate)
May also use a difference plot to analyze data (difference vs concentration)
Look for outliers and data gaps
- Repeat both methods for outliers
- Try to fill in gaps or eliminate highest data during analysis
Westgard JO. Basic Method Validation, 3rd Ed. 2008
CLSI, method comparison on Bias Estimation Using Patient Samples
Method x (reference) test method y
Sample (mg/dL) (mg/dL)
1 217 203
2 224 213
3 298 279
0 100 200 300 400
4 172 160
5 198 189 100
Diff x-y (mg/dL)
6 274 262 80
7 253 238
60
8 197 275
9 226 211 40
10 151 149 20
11 166 151
12 163 151 0
13 215 205 -20
14 151 133
-40
15 263 252
16 226 212 Metode x (mg/dL)
17 239 226
18 162 147
19 253 235
20 159 157
21 261 250
22 247 231
23 261 238
300
24 184 179 y = 0,941x + 3,246
Metode y (mg/dL)
25 295 284 250 R² = 0,892
26 250 232
27 201 196 200
28 209 212
29 286 275 150
30 158 142
31 288 281 100
32 161 145
50
33 183 171
34 252 239 0
35 285 277
0 100 200 300 400
36 194 190
37 240 230 Metode x (mg/dL)
38 180 177
39 297 275
40 210 188
https://www.westgard.com/mvtools.htm
check
0 100 200 300 400
r (Correlation coefficient)
5
value
Diff x-y (mg/dL)
0
-5
-10 r < 0.975 --> linear regression analysis
-15 may not be valid.
-20 r --> influenced by range of values.
-25 r < 0.975 --> may indicate that the range of
Metode x (mg/dL) data is too limited.
r --> is influenced by random errors
only, systematic error has no effect on r.
y = 0.7158x +
“r” --> a statistical term
28.037--> it indicates the
r = 0.984 between the
extent of linear relationship
300
methods.
250 y = 0,967x - 4,701
Metode y (mg/dL)
R² = 0,984
200
R = 0.992
150
100
50
0
0 100 200 300 400 if r < 0.975
Estimate bias at t mean of
Metode x (mg/dL) data from t-tests statitics
Westgard JO. Basic Method Validation, 3rd Ed. 2008
Professional practice in clinical chemistry
VALIDATION : HOW ?
Method Comparison
If r > 0.975
Calculate systematic error at medical decision levels
Use slope and intercept to calculate systematic error: Yc= mX + b
SE = Y – X
Yc = Calculated result on new method
X = Result from existing method
m = Slope observed in method comparison experiment ( proportional error)
b = Intercept observed in method comparison experiment ( constant error)
Y = 0.9672x – 4.6970
At decision level x = 200 mg/dL
Y = 188.7 mg/dL
Systematic error of 11.3 mg/dL or 5.65 %
Westgard JO. Basic Method Validation, 3rd Ed. 2008
https://www.westgard.com/mvtools.htm
VALIDATION : HOW ?
Interference Studies
ENSURE
correct result
Calculate interference (bias) interpretation !
VALIDATION : HOW ?
Interference Studies
Analyte Solution Standard solution, patient specimens
replicates recommended
Interferer solution Standard solution:
Lipemia: patient specimen/intralipid
Hemolysis: patient specimen
Icteric: bilirubin solution
Volume of Volume added should be small relative to the original test
interferer solution sample to minimize the dilution of the patient specimen.
Concentration of Should achieve a distinctly elevated level, preferably near
interferer material the maximum concentration expected in the patient
population.
Alternatively, follow criteria by manufacturer’s kit insert.
Westgard JO. Basic Method Validation, 3rd Ed. 2008
VALIDATION : HOW ?
Interference Studies
Bilirubin 48 mg/dL
0.9 mL serum + 0.1 bilirubin (yyy mg/dL)
bilirubin 48 mg/dL (total 1 mL)
V1M1 = V2M2
0.1 mL . M1 = 1 mL . 48 mg/dL
0.9 mL M1 = 48 / 0.1
serum + 0.1 M1 = 480 mg/dL
mL
saline/water
Add 0.1 mL Bilirubin 480 mg/dL to 0.9 mL serum
VALIDATION : HOW ?
Interference
spiked sample
baseline sample
0.9 mL specimen + 0.1 mL saline 0.9 mL specimen + 0.1 mL Bil standard
Patient 480 mg/dL
Patient
specimens result 1 result 2 result 3 result 4
specimens result 1 result 2 result 3 result 4
1 206 213 223 215 1 221 222 230 229
2 220 228 223 210 2 233 241 228 237
3 299 287 297 297 3 306 304 302 296
4 169 171 167 178 4 186 184 181 183
5 250 248 257 252 5 242 265 271 262
6 227 221 224 230 6 236 229 237 242
baseline sample spiked sample
0.9 mL specimen + 0.1 mL 0.9 mL specimen + 0.1 mL difference
difference (%)
saline Bil standard 480 mg/dL (mg/dL)
Patient specimens mean mean
1 214.25 225.5 11.25 5.25
2 220.25 234.75 14.5 6.58
3 295 302 7 2.37
4 171.25 183.5 12.25 7.15
5 251.75 260 8.25 3.28
6 225.5 236 10.5 4.66
Westgard JO. Basic Method Validation, 3rd Ed. 2008
VALIDATION : HOW ?
Recovery
Purpose: to estimate proportional error
Volume of analyte added:
Keep the volume of standard small relative
to the original patient sample.
Recommended: no more than 10 %.
Concentration of analyte added:
Add enough of the analyte to reach the next
decision level of the test.
mixing 0.9 mL diluting 0.9
of each mL of each
specimen with specimen Replicate: duplicate.
standard with 0.1
solution saline If low conc. Is added triplicate/quadruplicate
Westgard JO. Basic Method Validation, 3rd Ed. 2008
VALIDATION : HOW ?
Recovery
Adding cholesterol 50 mg/dL
0.9 mL serum + 0.1 standard (yyy mg/dL)
cholesterol 50 mg/dL (total 1 mL)
V1M1 = V2M2
0.1 mL . X = 1 mL . 50 mg/dL
X = 50 / 0.1
diluting 0.9 X = 500 mg/dL
mL of each
specimen
with 0.1
saline
Add 0.1 mL Cholesterol 500 mg/dL to 0.9 mL
serum (with cholesterol cons. ± 150 - 200
mg/dL)
VALIDATION : HOW ?
Recovery
baseline sample spiked sample
0.9 mL specimen + 0.1 mL saline 0.9 mL specimen + 0.1 mL chol standard
Patient
specimens result 1 result 2 result 3 result 4 result 1 result 2 result 3 result 4
1 149 151 153 146 204 196 208 194
2 210 186 178 187 224 222 228 240
3 210 204 196 206 255 243 257 257
4 180 204 184 188 235 246 233 233
5 160 157 166 159 206 207 210 210
6 187 182 191 201 235 242 246 246
spiked sample
baseline sample 0.9 mL specimen recovery
difference added
0.9 mL specimen + 0.1 mL chol (%)
Patient + 0.1 mL saline standard
specimens mean mean 50.75 50 101.5
1 149.75 200.5 45.75 50 91.5
2 182.75 228.5
49 50 98
3 204 253
47.75 50 95.5
4 189 236.75
5 160.5 208.25 47.75 50 95.5
6 190.25 242.25 52 50 104
Westgard JO. Basic Method Validation, 3rd Ed. 2008
VALIDATION : HOW ?
Linearity = Reportable Range /
Analytical Measurement Range (AMR)
Reportable range = the span of test
result values over which the laboratory
can establish or verify the accuracy of
the system.
AMR = Range of analyte where results
are proportional to the TRUE
concentration of analyte in the sample.
Westgard JO. Basic Method Validation, 3rd Ed. 2008
VALIDATION : HOW ?
Linearity = Reportable Range /
Analytical Measurement Range (AMR)
Number of levels: CLSI recommends a minimum of 4, preferably 5 –
different levels of concentrations spanning the expected reportable range
Materials: standard solution with known concentration/ manufacturer
linearity sets, dilution of patient samples/pools of samples
Diluent for use: maintain the matrix of specimen. For general chemistry:
water/saline can be used or diluent for diluting out-of-range patient specimen
Number of replicate: CLSI recommends 4 measurement on each
specimen, 3 are generally sufficient
Data analysis: measured values vs assigned values, check visually for
linearity, compare the SE + RE at concentration to allowable total error for
the test.
Westgard JO. Basic Method Validation, 3rd Ed. 2008
VALIDATION : HOW ?
Linearity = Analytical Measurement Range (AMR)
Example: Expected reportable range: 0 – 500 mg/dL
Make dilution from 500 – 0
Assign Measured value
ed Replicat Replicat Replicat mean
value e1 e2 e3
0 0 5 10 5.0
100 95 100 105 100
200 200 195 205 200
300 310 300 290 300
400 380 390 400 390
500 470 460 480 470
The reportable range clearly extends to 300 mg/dL, but does it extend to 400
Westgard JO. Basic Method Validation, 3rd Ed. 2008
mg/dL or 500 mg/dL?
Assume CV = 3 %
TEa for Cholesterol (CLIA) = 10%
500 mg/dL Assume CV = 3 %
At 500 mg/dL, SD = 15 mg/dL dan 2SD = 30 mg/dL
True value = 500, observed value = 470 mg/dL systematic error of -30 mg/dL
In addition, random error = ± 30 mg/dL
Expected value range from 440 – 500 mg/dL error as high as 60 mg/dL
CLIA criteria for TEa = 10 %, which is 50 mg/dL at 500 mg/dL
Error (60 mg/dL) >> Tea (50 mg/dL) X
400 mg/dL Assume CV = 3 %
At 400 mg/dL, SD = 12 mg/dL dan 2SD = 24 mg/dL
True value = 400, observed value = 390 mg/dL systematic error of -10 mg/dL
In addition, random error = ± 24 mg/dL
Expected value range from 366 – 414 mg/dL error as high as 34 mg/dL
CLIA criteria for TEa = 10 %, which is 40 mg/dL at 400 mg/dL
Error (34 mg/dL) << Tea (40 mg/dL) √
CLSI EP6-A Evaluation of the Linearity of Quantitative Measurement Procedures: A Statistical Approach; Approved Guideline.
VALIDATION : HOW ?
Analytical Sensitivity Studies
Limit of Blank (LoB): Highest measurement
result that is likely to be observed (with a
stated probability) for a blank sample.
LoB = meanblk + 1.65SD
Limit of Detection (LoD): Lowest amount of
analyte in a sample that can be detected
with (stated) probability, although perhaps Total
not quantified as an exact value Error ??
LoD = LoB + 1.65 SD
Limit of Quantification (LoQ): Lowest
amount of analyte that can be
quantitatively determined with stated
acceptable precision and trueness, under
stated experimental conditions
LoQ = mean @ TEa = 2 SD + bias
VALIDATION : HOW ?
Analytical Sensitivity Study
Detection limit should be verified when relevant (e.g. PSA, hsTnT)
Detection limit is not important for tests such as glucose, cholesterol, and
other constituents where thre is a “normal” or reference range.
Blank solution One aliquot for blank, one aliquot for
spiked sample
Ideally, same matrix.
Can also use zero standard
Spiked sample Concentration at LoD claimed by
manufacturer
Or at concentration of expected detection
limit
Replicate Verification: 20
Validation: 60
Time period of CLSI: LoD- several days
study LoQ at least 5 days
Analytical Sensitivity Verification
LoB Twenty (20) replicates of a blank material (Calibrator A) are run. If no
more than three replicates exceed the claimed LoB LoB is verified
LoD Twenty (20) replicates of a sample with concentration equal to the
claimed LoD will be run and an estimate of the proportion of results
exceeding the LoB is determined. If the recorded proportion is in agreement
with the expected values, that is, it “95%” is contained within the 95%
confidence limits for the recorded proportion, then the data support the
claim of the LoD. It is possible to have more than one measurement results
in 20 below the LoB and still meet this criteria.
N Lower bound of observed N Lower bound of N Lower bound of
population (%) observed population (%) observed population (%)
20 85 80 89 250 92
30 87 90 90 300 92
40 88 100 90 400 93
60 88 150 91 500 93
70 88 200 92 1000 94
CLSI EP17-A Protocols for Determination of Limits of Detection and Limits of Quantitation: Approved Guidelines
Analytical Sensitivity Verification
A minimum of thirty (30) replicates of a sample with a concentration close to
the claimed LoQ will be run.
Case
VALIDATION : HOW ?
Reference Range Verification
Reference interval is typically established by assaying specimens from
individuals that meet carefully defined criteria (reference sample group).
Resource-intensive
Many relies on manufacturers
1. Divine judgement
Acceptability of transfer may be subjectively assessed on the basis of
consistency between the “demographics” and geographics” of the study
population and the laboratory test population
2. Verification with 20 samples
Collecting 20 samples who represent the reference sample population.
If two or fewer fall outside the claimed or reported reference range verified
CLSI approved guideline C28-A2
VALIDATION : HOW ?
Reference Range Verification
3. Estimation with 60 samples (at least 40)
4. Calculation from comparative
method not recommended
Should be further verified using
20 samples
CLSI approved guideline C28-A2
References
Westgard JO. Basic Method Validation, 3rd Ed. 2008
www.westgard.com
CLSI EP5-A2. Evaluation of precision performance of quantitative measurement
methods. Approved guideline 2004.
CLSI EP9-A2. Method comparison and bias estimation using patient samples.
Approved guidelines 2002
CLSI EP6A. Evalution of the Linearity of quantitantive measurement procedures: a
statistical approach; approved guideline 2003.
CLSI EP17A. Protocols for determination of limits of detection and limit of quanitation.
Approved guidelines. 2004.
CLSI C28A2. How to define and determine reference intervals in the clinical laboratory
– 2nd edit – approved guideline. 2000.
~ Smile, Breath and Go Slowly ~
VALIDATION : HOW ?
Diagnostic Accuracy
cut-off : Disease
xx.x μg/L Presence Absence Total
Test Positive a b a+b
Result Negative c d c+d
Total a+c b+d a+b+c+d
Sensitivity = a / (a+c)
Specificity = d / (b+d)
AGREEMENT
PPV = a / (a+b)
NPV = d / (c+d) between
LR + = sens / (1-spec) methods
LR - = (1-sens) / spec
