MicroVal validation of
alternative methods
Wilma Jacobs-Reitsma
MicroVal Expert Laboratory
1 15 May 2012
Why alternative methods ?
● Compared to reference methods
● Less laborious
– Time
– Money
● Less consumables (?)
– Money
● Higher troughput
– Time
– Money
● Results available much faster!
– Time
– Money
2
Why validation of alternative methods?
● EU Microbiological Criteria Document
– Commission Regulation (EC) No 2073/2005, on microbiological criteria for
foodstuffs
– Food Business Operators
Article 5 Specific rules for testing and sampling
The use of alternative analytical methods is acceptable when the methods are validated
against the reference method in Annex I and if a proprietary method, certified by a third
party in accordance with the protocol set out in EN/ISO standard 16140 or other
internationally accepted similar protocols, is used.
● Not (yet) applicable for Official Controls
3
By whom?
● Europe:
● AFNOR
● Nordval
● MicroVal
● USA:
● (AOAC)
– not (yet) according to ISO 16140
– Nowadays ISO reference methods as a possibility
4
Validation according to ISO 16140
● EN ISO 16140:2003 ‘Microbiology of food and animal feeding stuffs
- Protocol for the validation of alternative methods’
● EN ISO 16140:2003/Amd 1:2011. Amendment 1: Interlaboratory
study on quantitative methods
● At the moment in a process of revision
– Experiences from the 2003 version
– New version to be split up in 5 parts
– Updated drafts after first round of comments world-wide in 2012
– May still take some years…
● Currently still to stick to the 2003 version
5
ISO 16140:2003 validation
Qualitative method validation (Detection)
● A. Methods comparison study (MCS):
– Relative accuracy, relative specificity and relative sensitivity
› 60 samples per Category
– Relative detection level
› 6 samples at 3-5 levels per Category
– Inclusivity and exclusivity
› 50 target strains and 30 non-target strains to be tested
● B. Interlaboratory study (ILS):
– At least 10 collaborative laboratories within 3 (European)
countries
› 8 replicates at 3 levels = 24 samples by each laboratory
6
ISO 16140:2003 validation
Quantitative method validation (Enumeration)
● A. Methods comparison study (MCS):
– Linearity and relative accuracy
› Per Category at least 1 sample in duplicate at 5 different levels
– Relative sensitivity and determination of unknown samples
› Per Category at least 10 additional samples in duplicate
– Detection and quantification limits
› 6 replicates of at least 3 levels
– Specificity, inclusivity and exclusivity
› 30 target strains and 20 non-target strains to be tested
● B. Interlaboratory study (ILS):
– At least 8 collaborative laboratories within 3 (European) countries
› 2 replicates at 4 levels = 8 samples by each laboratory
7
MicroVal Expert Laboratory (EL)
● Organisation and elaboration of the lab work
– Methods Comparison Study
– Interlaboratory Study
● To be chosen by the client
● List of labs available at www.microval.org
● Labs qualifications
– Officially approved by MicroVal
– Accreditation (eg ISO 17025)
– Confidentiality
● Collaborative laboratories for participation in the ILS:
– Contacted by the EL
– Preferably working under a QA system (eg ISO 17025)
8
RIKILT Institute of Food Safety, WUR, Wageningen, NL
as from 1 January 2010: RIVM-LZO, Bilthoven, NL
(Laboratory for Zoonoses and Environmental Microbiology)
● ISO/IEC 17025 accreditation
● Dutch Accreditation Council (www.rva.nl no. L421)
● Food microbiology
– Total Viable Count, E. coli, Salmonella, Campylobacter, S.
aureus, L. monocytogenes, etc.
– Facilities:
› Fully equipped microbiology laboratory
› Fully equipped molecular biology laboratory
9
10
RIKILT/RIVM-LZO Expert Laboratory
● Currently finished/in preparation (certifate number):
● Quantitative methods
– Enumeration of E. coli (2007LR07)
– Enumeration of Campylobacter spp. (2008LR12)
● Qualitative methods
– Detection of Salmonella (2007LR06)
– Real-time PCR Detection of Enterobacteriaceae and/or
E. sakazakii (2007LR08, 2007LR09, 2007LR19, 2007LR20)
– Real-time PCR detection of Salmonella (in preparation)
11
Example: Detection of Salmonella
● Confidentiality? (the client agrees, and the certificate is granted!)
● Example of the qualitative validation process
– Expert Lab: RIKILT Institute of Food Safety
– Certification Body: Lloyds Register
– Method reviewers: Henk Stegeman & Basil Jarvis
● Reference method: ISO 6579:2002
● Alternative method: Salmonella SOP of the client (partly ISO
6579:2002)
12
25 g sample material + 225 ml BWP
Incubation: 18 h ± 2 h at 37 °C ± 1 °C
0.1 ml enrichment 1 ml enrichment
+ +
10 ml RVS broth 10 ml MKTTn broth
Incubation: Incubation:
24 h ± 3 h at 41.5 °C ± 1 °C 24 h ± 3 h at 37 °C ± 1 °C
( )
XLD plus 2nd medium of choice
Incubation: 24 h ± 3 h at 37 °C ± 1 °C
biochemical + serological confirmation
13
Detection of Salmonella, isolation media tested
XLD BGA MLCB SMID2
14
Example: Detection of Salmonella
● Scope of the validation, e.g.: “All foods and animal feed”
– A minimum of 5 Categories to be tested (Annex B, ISO 16140)
› Meat products, poultry products, fish & seafood products, fruits and vegetable based products,
dairy products, chocolate/bakery products, other products, animal feeds
– Additional Categories: environmental samples, samples from the primary
production stage.
– Discussion in revision current ISO 16140
● One specific Category, e.g. “Poultry Products”, is also possible
● Scope for the example Salmonella validation (n=6)
– All Foods (dairy products, probiotics-containing dairy products,
chocolate products, other products)
– Animal Feed
– Environmental samples
15
Example: Detection of Salmonella
● A. Methods Comparison Study:
– Relative accuracy, relative specificity and relative
sensitivity
– Per Category: 60 samples
– Preferably around 50% positive samples
– Preferably naturally contaminated samples ! (?)
– Artificial contamination of samples is a possibility…
– Both reference and alternative methods testing the same
sample (if possible)
16
Paired results reference and alternative method:
Responses Reference method Reference method
positive (R+) negative (R-)
Alternative method Positive Positive deviation
agreement (R-A+)
(A+R+)
positive (A+) PA = 172 PD = 1
Alternative method Negative Negative
deviation (A-R+) agreement (A-R-)
negative (A-) ND = 3 NA = 188
17
Calculation of the relative AC, SE, SP
Relative Relative Relative
Accuracy AC sensitivity specificity
Matrices PA NA ND PD Sum (%) N+ SE (%) N- SP (%)
100 x (PA+NA)/ 100 x PA/ 100 x NA/
N N PA+ND N+ NA+PD N-
Dairy 25 35 0 0 60 100,0% 25 100,0% 35 100,0%
Probiotic
dairy 28 30 1 1 60 96,7% 29 96,6% 31 96,8%
Chocolates 31 33 0 0 64 100,0% 31 100,0% 33 100,0%
Other
products 27 33 0 0 60 100,0% 27 100,0% 33 100,0%
Animal feed 28 30 2 0 60 96,7% 30 93,3% 30 100,0%
Enviromental 33 27 0 0 61 100,0% 33 100,0% 27 100,0%
TOTAL 172 188 3 1 364 99,5% 175 98,3% 189 99,5%
18
Example: Detection of Salmonella
● A. Methods Comparison Study:
– Relative detection level
– Per Category: 1 sample type and 1 relevant target
organism
– Five levels per target organism and per sample type
› ISO 16140: 0, 1, 3, 9, 27 cfu per sample,
› More appropriate e.g.: 0, 0.65, 1.3, 3.25, 6.5 cfu
– Each combination repeated 6 times, using both the
reference and the alternative method
19
Vacuumcleaner residue, S. Enteritidis
Inoculation level Method Results
(cfu/25 g) Neg (-) Pos (+) Total
Reference 6 0 6
0
Alternative 6 0 6
Reference 3 3 6
0,215
Alternative 3 3 6
Reference 3 3 6
0,43
Alternative 4 2 6
Reference 0 6 6
0,86
Alternative 0 6 6
Reference 0 6 6
2,15
Alternative 0 6 6
20
Results relative detection level (1)
ISO 16140: “the relative detection level lies between the
two contamination levels giving respectively less and more
than 50% detection level. The relative detection level is
therefore expressed as a range.”
Sample type Strain Reference Alternative
method method
(cfu/25 g) (cfu/25 g)
Infant formula S. Typhimurium 0 – 0.9 0 – 0.9
Infant formula
& Probiotics S. Panama 0– 1.1 0– 1.1
Cocoa powder S. Senftenberg 0– 0.6 0– 0.6
Dried soup S. Derby 0– 0.7 0– 0.7
Dog food (dry) S. Virchow 0.4 – 0.9 0.4 -0.9
VC residues S. Enteritidis 0.2 – 0.9 0.4 -0.9
21
Results relative detection level (2)
Spearman-Kärber procedure (e.g. as used by AFNOR)
A programed non-parametric statistical procedure that calculates the microbial analyte
concentration [and 95% confidence limits] in a given food matrix that corresponds to a
50% probability of a positive result with the test method used.
Sample type Strain Reference Alternative
method method
(cfu/25 g) (cfu/25 g)
Infant formula S. Typhimurium 0,4 [0,3-0,7] 0,4 [0,3-0,7]
Infant formula
& Probiotics S. Panama 0,6 [0,3-1,2] 0,4 [0,3-0,8]
Cocoa powder S. Senftenberg 0,4 [0,2-0,6] 0,4 [0,2-0,6]
Dried soup S. Derby 0,3 [0,3-0,4] 0,3 [0,3-0,4]
Dog food (dry) S. Virchow 0,4 [0,3-0,6] 0,4 [0,3-0,6]
VC residues S. Enteritidis 0,3 [0,2-0,5] 0,3 [0,2-0,6]
22
Results relative detection level (3)
Relative Limit of Detection model (RLOD)
Defined as the LOD of the alternative method divided by the LOD of the
reference method, [with 95% confidence limits].
Proposal for the revision of ISO 16140
Matrix RLOD [95% confidence level]
Infant formula 1.0 [0.3-3.5]
Infant formula
containing probiotics 0.7 [0.2-2.3]
Cocoa powder 1.0 [0.4-2.8]
Dried chicken soup 1.0 [0.3-2.9]
Dog food (dry) 1.0 [0.4-2.4]
Vacuum cleaner residues 1.0 [0.4-2.3]
23
Example: Detection of Salmonella
● A. Methods Comparison Study:
– Inclusivity and exclusivity
– At least 50 relevant Salmonella spp. strains (inoculation at
low level)
– At least 30 relevant non-Salmonella strains (inoculation at
high level)
– Alternative method testing only
– No addition of matrix
– All strains to be from a known origin, preferably from a
relevante source (foods)
24
Results inclusivity
● All 50 Salmonella strains showed a positive result using the
alternative method, for each of the combinations of isolation
media
– The lactose-positive strains S. Agona en S. enterica subsp.
arizonae and the H2S-negative strains S. Muenster and S.
Paratyphi A showed atypical results on some of the plates,
as expected.
– One S. Dublin strain showed non-suspect white growth on
SMID2 plates (not unusual)
25
Results inclusivity
BGA / lactose-positive strain S2
(non)-suspect growth on SMID2
26
Results exclusivity
● All 30 non-Salmonella strains showed a negative result using
the alternative method
– E.g. Aeromonas, Bacillus, Citrobacter, Enterobacter,
Escherichia, Hafnia, Klebsiella, Saccharomyces, Proteus,
Pseudomonas, Serratia, Shighella, Staphylococcus,
Yersinia.
27
Conclusions Method Comparison Study
● The alternative Salmonella method shows satisfactory results
for relative accuracy, relative sensitivity and relative
specificity.
● The alternative Salmonella method and the reference method
show similar relative detection levels.
● The alternative Salmonella method is selective and specific.
28
Example: Detection of Salmonella
● B. Interlaboratory Study:
– Valid data of a minimum of 10 collaborative labs within at
least 3 (European) countries => start with 15 labs
– 1 suitable matrix (pasteurised milk)
– 3 levels of contamination (0, 3, 30 cfu/25 ml)
– 8 blind replicates per level (24 samples in total) to be
tested, using the reference mthod and the alternative
method, by each of the labs
– 1 additional sample per lab to test Total Viable Count
29
Results Interlaboratory Study
● Samples shipped on 9-2-2009
● Labs in NL, B, D, UK, E, CH.
● Arrival and start of testing on 10-2-2009
– 3 labs only on 11-2-2009
– Temperature control ok, -> valid data
● 3 Labs rapported false-positive results
– Leakage during transport reported, -> invalid data
● In the end, valid data for 12 labs (including EL)
30
Results Interlaboratory Study
Reference method
Alternative method positive negative Total
positive PA = 192 PD = 0 192
negative ND = 0 NA = 96 96
Total N+ = 192 N- = 96 N = 288
For all combinations of isolation media
31
Results Interlaboratory Study
For both the reference and the alternative method:
Contamination level Accordance Concordance COR
L0 100% 100% 1,0
L1 100% 100% 1,0
L2 100% 100% 1,0
32
Results Interlaboratory Study
Method Comparison Interlaboratory
Study Study
Relative accuracy
AC 98,9% 100%
Relative sensitivity
SE 98,3% 100%
Relative specificity
SP 99,5% 100%
33
Conclusions Interlaboratory Study
● The alternative Salmonella method shows comparable results
for relative accuracy, relative sensitivity and relative
specificity in both the Method Comparison Study and the
Interlaboratory Study.
● The alternative Salmonella method and the reference method
show similar performances (accordance, concordance and
COR).
34
Overall Conclusion:
● The results from the Method Comparison Study and the
Interlaboratory Study revealed that there was no significant
difference between the alternative method and the reference
method ISO 6579:2002 for the detection of Salmonella.
● This is valid for each of the combinations of plating media
used for the alternative method.
– XLD or BGA, XLD or MLCB, XLD or SMID2, BGA or MLCB,
BGA or SMID2, MLCB or SMID2
35
Validation process
● Information exchange (client, MV)
● Application (client)
● Contracts (client-MCB, client-EL, MCB-EL)
● Project proposal (EL)
– Method Reviewers
– Technical Committee
– Approval
● Project report (by EL, in 2 parts: MCS + ILS)
– Method Reviewers
– Technical Committee
– Approval
● Certificate (MCB)
36
Final result: certificate!
37
Thank you
for your attention !
www.microval.org
38 15 May 2012
List of abbreviations
● MV MicroVal
● MGC MicroVal General Committee
● MCB MicroVal Certification Body
● EL Expert Laboratory
● MR Method Reviewer
● MV TC MicroVal Technical Committee
● MCS Method Comparison Study
● ILS Inter Laboratory Study
39
Public Authorities Users Manufacturers Third Parties
MGC
Expert Committees MV Secretariat
MCB Group
EU Expert Labs EU Method reviewers EU Auditors
EU Collaborative Labs
40
