Control of Test Methods

Methods Evaluation
VALIDATION:

Establishing standards performance through a defined

process. Long studies

Non- FDA approved testing.

VERIFICATION:

Short studies to demonstrate that a test performs in

significant compliance to previously established claims.

FDA approved testing.

Validation & Verification
Purposes of
Method Evaluation
To characterize system performance
(Quantitative wise)

To assess potential for error

To identify method-to-method differences

To meet regulatory guidelines

When to do
Method Evaluation Studies?
When placing a new system into service

At regular intervals to assess on-going

system performance.

After major troubleshooting.

When to do
Method Evaluation Studies?
When a perennial problem is shown by quality
control.

When a new (different) diagnostic

kit is being introduced.

When existing methods are being enlarged by

another purpose (e.g., Another kind of biological
material).
Approach in Method Evaluation

 Imprecision: Refers to Random Analytical

Errors

 Inaccuracy: Refers to Systematic

Errors
 Constant
 Proportional

 Total Errors: Combined Errors

Experimental Approaches to
Estimating Analytical Performance
Performance Standard
EA… Allowable Error
The amount of error that can be tolerated without:
- Invalidating the medical usefulness of the result
- Causing the test to fail a proficiency testing event

Sources of EA:
 Biological Variation
 CLIA ‘88
What is Random Error?

 Random errors refers to fluctuations of measured

values about their mean due to (random factors).

 Increased random error causes increased

variation of result both above and below
the mean.
Causes of Random Error
 Inconsistent environmental
conditions

 Electrical interferences

 Inconsistent handling
of material
Causes of Random Error

 Fluctuations in Temperature

 Fluctuations in Volume
Random Errors
Random Errors
Precision (Replication) Study
 Simple Precision:
Repeatability (within run)

 Complex Precision:
Reproducibility (between runs,
between days)
What is Systematic Error?
 Systematic error (change) is a change that is always in
one direction and will cause a shift in the mean value.

 Systematic error or change, is associated

with a change in accuracy.

 Some sources of systematic error are however

unavoidable.
Types of Systematic Error
 Constant: an error that is always in the same direction and of
the same magnitude even as the concentration of
analytes changes.

 Proportional:
an error that is always in one direction and the
magnitude of which is a percentage of the
concentration of the analytes being measured.
Systematic Errors

x
x x x x x x x
True x
Value
Causes of Systematic Errors
 Changes in reagent lot number

 Changes in calibration

 Changes in the instrument

itself
Comparison Study
 Number of test sample: Min. = 40

 Duplicates are required

 Time period: minimum 5 days.

 Data analysis: must be reviewed daily

Systematic Errors
Comparison Study
 Present graphs of data
 Outliers must be carefully examined
 Data must be linear
 Range of data must be wide
 Corr. Coef. >0.975 (according to NCCLS)
Recommendations
1. Present slope, Y-intercept

2. Present mean and SD

3. Present correlation
coefficient to demonstrate
that regression is
applicable.
Bias Plot
Linear Regression Graph
Linearity (AMR) and Recovery
 Linearity refers to a function or relationship which can
be graphically represented as a straight line.

 Reportable Range (AMR): the range of

values that the method can directly
measure without dilution or
concentration.
Linearity (AMR) and Recovery
 Recovery (Accuracy): described as the closeness of
agreement between the reference material
assigned value and the measured value .

Measured Value
Recovery % = x 100
Assigned Value
Carryover
 Contamination of sample to sample run
 Batch samples
 The incomplete washing
 New instrument
 Changed Sample Probe
 Semi-annual
Reference Range
Qualitative Methods
 includes semi quantitative testing that
use cut offs such as hepatitis testing
and some molecular testing.
 No values/concentrations are included in the
patient report. Test results are reported as
positive/negative, normal/
borderline/abnormal, reactive/nonreactive,
detected /not detected, etc.
Accuracy
 Demonstrates how close to the “true” value the
new method can achieve
 Test material can include: calibrators/controls,
reference material, proficiency testing material
with known values

 Samples tested by another lab using the same or

similar method, or by comparing results to an
established comparative method.
Diagnostic Sensitivity
 If a person has a disease, how often will the test be
positive( true positive rate).

[TP ÷ (TP + FN)] x 100

Diagnostic Specificity
 If a person does not have the disease, how often will the
test be negative ( true negative rate).

 .
[TN ÷ (TN + FP)] x 100
Experiment:
Precision:
Precision:
 2 negative samples and 2 positive samples run in triplicate
for 5 days

 Control material and reference solutions

 Precision by dividing observed results

over known results multiplied by 100
Test Method Validation
LB.10 The laboratory develops a process for test method validation.
LB.10.1 The laboratory implements policies and procedures on test method validation
including:
LB.10.1.1 Verification of accuracy/precision.
LB.10.1.2 Verification of sensitivity (lower detection limit).
LB.10.1.3 Verification of carryover acceptability.
LB.10.1.4 Verification of the Analytic Measurement Range (AMR).
LB.10.1.5 Approval of the method for clinical use.
Test Method Validation
Evidence of Compliance (LB.10)
Document Review
 Policies, process and procedures on method validation.
Documented Evidences
 Records surveyor-selected method confirms compliance with
policies and procedures.
Staff Interview
 Senior personnel are knowledgeable about the concept of method
validation.
Instruments/Methods Correlation
Evidence of Compliance (LB.13)
Document Review
 Policies, process and procedures on methods/instrument
correlation.
Documented Evidences
 Records surveyor-selected methods/instruments confirms
compliance with policies and procedures.
Staff Interview
 Senior personnel are knowledgeable about the purpose of
methods/instrument correlation.
Instruments/Methods Correlation
LB.13 The laboratory has a system for instruments/methods correlation.
LB.13.1 When the laboratory uses more than one method and/or instruments to test
for a given analyte, the laboratory develops and implements policies and
procedures on correlation to ensure the following:
LB.13.1.1 The correlation studies are conducted every six months.
LB.13.1.2 There is clear description of the correlation study.
LB.13.1.3 There are clearly defined acceptance criteria.
LB.13.1.4 There is a process for review and approval of the correlation results.
 Controlling the Quality of Test Methods
LB.14 The laboratory has a system for controlling the quality of test methods.
LB.14.1 The laboratory implements policies and procedures on quality control of test
methods to satisfy the following:
LB.14.1.1 Assignment of performance and review responsibility (control specimens are
handled and tested in the same manner and by the same laboratory
personnel testing patient samples).
LB.14.1.2 Number and frequency of running controls.
LB.14.1.3 Tolerance limits of controls results.
LB.14.1.4 Corrective action to be taken in the event of unacceptable results.
LB.14.1 The laboratory quality control system conforms to the manufacturer's
instructions.
Controlling the Quality of Test Methods
Evidence of Compliance (LB.14)
Document Review
 Policies, process and procedures on controlling the quality of test
methods.
 Policies and procedures on controlling the quality of test methods
confirm to the manufacturer instructions.
Documented Evidences
 Records surveyor-selected methods confirms compliance with
policies and procedures.
Staff Interview
 Laboratory personnel understanding of the purpose of controlling
the quality of test methods.
 Evaluation of Precision Performance of Quantitative Measurement Methods;
EP05-A2
Approved Guideline—Second Edition

Evaluation of the Linearity of Quantitative Measurement Procedures: A

EP06-A
Statistical Approach; Approved Guideline

Method Comparison and Bias Estimation Using Patient Samples; Approved

EP09-A2-IR
Guideline—Second Edition (Interim Revision)

Preliminary Evaluation of Quantitative Clinical Laboratory Measurement

EP10-A3
Procedures; Approved Guideline—Third Edition

User Protocol for Evaluation of Qualitative Test Performance; Approved

EP12-A2
Guideline—Second Edition

Defining, Establishing, and Verifying Reference Intervals in the Clinical

EP28-A3c
Laboratory; Approved Guideline—Third Edition
Questions?
More Information?

• Presenter:
• Saeed Al-maleh
• [email protected].
sa