Foundations of Verification and Validation

A Logical Derivation from the Scientific Theory of Truth

Jonathan Smith
Compusis

NAFEMS World Congress 2015 | 21-24 June | San Diego | California | USA
 Foundations of Verification and Validation

• Simulation governance
– simulation process flow diagram (ISO 9001, NAFEMS QSS)
• framework and taxonomy of process components
• verification and validation processes

AIAA (1998) ASME (2006) NAFEMS QSS Primer (2014)

NAFEMS World Congress 2015 | 21-24 June | San Diego | California | USA
 Foundations of Verification and Validation

• Scientific method
– inductively infer an idealised model from real-world observations

𝐞𝐱𝐩𝐞𝐫𝐢𝐦𝐞𝐧𝐭𝐚𝐥 : 𝐠𝐞𝐧𝐞𝐫𝐚𝐥 ⇐ 𝐩𝐚𝐫𝐭𝐢𝐜𝐮𝐥𝐚𝐫 ∧ 𝐩𝐚𝐫𝐭𝐢𝐜𝐮𝐥𝐚𝐫

𝐦𝐞𝐚𝐬𝐮𝐫𝐞𝐦𝐞𝐧𝐭𝐬 𝑖𝑛𝑑𝑢𝑐𝑡𝑖𝑜𝑛
𝐛𝐞𝐡𝐚𝐯𝐢𝐨𝐮𝐫 𝐝𝐚𝐭𝐚 𝐛𝐞𝐡𝐚𝐯𝐢𝐨𝐮𝐫
⇓ ⇓ ⇓ ⇑
𝑖𝑑𝑒𝑎𝑙𝑖𝑠𝑎𝑡𝑖𝑜𝑛 𝑖𝑑𝑒𝑎𝑙𝑖𝑠𝑎𝑡𝑖𝑜𝑛 𝑖𝑑𝑒𝑎𝑙𝑖𝑠𝑎𝑡𝑖𝑜𝑛 𝑝𝑟𝑒𝑑𝑖𝑐𝑖𝑜𝑛

𝐢𝐝𝐞𝐚𝐥𝐢𝐬𝐞𝐝 𝐦𝐨𝐝𝐞𝐥 : 𝑦= 𝑥 ∧ 𝑥= 2 ⇒ 𝑦= 2
𝑑𝑒𝑑𝑢𝑐𝑡𝑖𝑜𝑛
𝑚𝑎𝑡ℎ𝑒𝑚𝑎𝑡𝑖𝑐𝑎𝑙 𝑝𝑎𝑟𝑎𝑚𝑒𝑡𝑟𝑖𝑐 𝑠𝑜𝑙𝑢𝑡𝑖𝑜𝑛
𝑚𝑜𝑑𝑒𝑙 𝑑𝑎𝑡𝑎

NAFEMS World Congress 2015 | 21-24 June | San Diego | California | USA
 Foundations of Verification and Validation

• Syllogistic model of a problem

major premise minor premise conclusion

𝑝→𝑞 ∧ 𝑞→𝑟 ⟹ 𝑝→𝑟

– engineering example : linear spring system 𝐾𝑢 = 𝐹

mathematical model parametric data solution

𝑢 = 𝐹/𝐾 ∧ 𝐹/𝐾 = 𝑓/𝑘 ⟹ 𝑢 = 𝑓/𝑘

mathematical model ∧ data ⟹ solution

NAFEMS World Congress 2015 | 21-24 June | San Diego | California | USA
 Foundations of Verification and Validation

• Example : Determine sources of error

model data solution

𝑦= 𝑥 ∧ 𝑥= 2 ⟹ 𝑦= 2

NAFEMS World Congress 2015 | 21-24 June | San Diego | California | USA
 Scientific tests for truth

• Coherence and correspondence tests for truth

– well-established tests for truth from the Philosophy of Science:

• correspondence theory of truth

– external conformance to observed facts
– “A judgment is said to be true when it conforms to the external reality”,
Thomas Aquinas, c.1250

• coherence theory of truth

– internal logical consistency
– “Does it make sense?”

Ref : Russell, B., The Problems of Philosophy, 1912

NAFEMS World Congress 2015 | 21-24 June | San Diego | California | USA
 Sources of error

• Correspondence test – do the premises conform to the observed facts?

model data solution

𝑦= 𝑥 ∧ 𝑥= 2 ⟹ 𝑦= 2

– modelling error - incorrect assumptions, etc.

𝑦 = log 𝑥 + 1 ∧ log 𝑥 + 1 = log 2 + 1 ⟹ 𝑦 = log 2 + 1

– data error - incorrect coefficients values, etc.

𝑦= 𝑥 ∧ 𝑥= 9 ⟹ 𝑦=3

2
log(2)+1
3

x log(x)+1

y=f(x)

NAFEMS World Congress 2015 | 21-24 June | San Diego | California | USA
 Sources of error

• Coherence test – is the model logically self-consistent?

model data solution

𝑦= 𝑥 ∧ 𝑥= 2 ⟹ 𝑦= 2

– approximation error - discretisation, truncation, etc.

𝑦= 𝑥 ∧ 𝑥 = 1.41 … 𝑛 ⟹ 𝑦 → 2
𝑛→∞

– calculation error - mistakes in arithmetic, bugs in software, etc.

𝑦= 𝑥 ∧ 𝑥= 2 ⟹ 𝑦=5

NAFEMS World Congress 2015 | 21-24 June | San Diego | California | USA
 Sources of error

• Correspondence test • Coherence test

– idealisation error – computation error
• modelling error • approximation error
– incorrect assumptions, etc. – discretisation, truncation, etc.

• data error • calculation error

– incorrect loads, coefficient values, etc – iteration, arithmetic, etc.
–
correspondence

𝐞𝐱𝐩𝐞𝐫𝐢𝐦𝐞𝐧𝐭𝐚𝐥 : 𝐠𝐞𝐧𝐞𝐫𝐚𝐥 ⇐ 𝐩𝐚𝐫𝐭𝐢𝐜𝐮𝐥𝐚𝐫 ∧ 𝐩𝐚𝐫𝐭𝐢𝐜𝐮𝐥𝐚𝐫

𝐦𝐞𝐚𝐬𝐮𝐫𝐞𝐦𝐞𝐧𝐭𝐬 𝑖𝑛𝑑𝑢𝑐𝑡𝑖𝑜𝑛
𝐛𝐞𝐡𝐚𝐯𝐢𝐨𝐮𝐫 𝐝𝐚𝐭𝐚 𝐛𝐞𝐡𝐚𝐯𝐢𝐨𝐮𝐫
⇑ ⇑ ⇑ ⇑
𝑐𝑜𝑟𝑟𝑒𝑠𝑝𝑜𝑛𝑑𝑒𝑛𝑐𝑒 𝑐𝑜𝑟𝑟𝑒𝑠𝑝𝑜𝑛𝑑𝑒𝑛𝑐𝑒 𝑐𝑜𝑟𝑟𝑒𝑠𝑝𝑜𝑛𝑑𝑒𝑛𝑐𝑒 𝑐𝑜𝑟𝑟𝑒𝑠𝑝𝑜𝑛𝑑𝑒𝑛𝑐𝑒

𝐢𝐝𝐞𝐚𝐥𝐢𝐬𝐞𝐝 𝐦𝐨𝐝𝐞𝐥 : 𝑦= 𝑥 ∧ 𝑥 = 1.41 … 𝑛 ⇒ 𝑦 → 2

𝑐𝑜ℎ𝑒𝑟𝑒𝑛𝑐𝑒 𝑛→∞
𝑛𝑢𝑚𝑒𝑟𝑖𝑐𝑎𝑙 𝑑𝑖𝑠𝑐𝑟𝑒𝑡𝑒
𝑚𝑜𝑑𝑒𝑙 𝑝𝑎𝑟𝑎𝑚𝑒𝑡𝑟𝑖𝑐 𝑑𝑎𝑡𝑎 𝑑𝑖𝑠𝑐𝑟𝑒𝑡𝑒
𝑠𝑜𝑙𝑢𝑡𝑖𝑜𝑛
coherence

NAFEMS World Congress 2015 | 21-24 June | San Diego | California | USA
 Sources of error

• Verification and validation definitions

– validation*
• the process of determining the degree to which the:
idealised model is an accurate representation of the real-world
engineering application from the perspective of the intended uses of
the model
• are we solving the correct model?
• test of external conformance to observed facts (correspondence)

– verification*
• the process of determining the degree to which the:
computational model is an accurate representation of the underlying
mathematical model and its solution
• are we solving the model correctly?
• test of internal logical consistency (coherence)

Sources* : What is Verification and Validation?, ASME/NAFEMS, 2010

NAFEMS QSS:2008, NAFEMS

NAFEMS World Congress 2015 | 21-24 June | San Diego | California | USA
 Verification and Validation ≡ Coherence and Correspondence

• Coherence and correspondence

»
correspondence

𝐞𝐱𝐩𝐞𝐫𝐢𝐦𝐞𝐧𝐭𝐚𝐥 : 𝐠𝐞𝐧𝐞𝐫𝐚𝐥 ⇐ 𝐩𝐚𝐫𝐭𝐢𝐜𝐮𝐥𝐚𝐫 ∧ 𝐩𝐚𝐫𝐭𝐢𝐜𝐮𝐥𝐚𝐫

𝐦𝐞𝐚𝐬𝐮𝐫𝐞𝐦𝐞𝐧𝐭𝐬 𝑖𝑛𝑑𝑢𝑐𝑡𝑖𝑜𝑛 𝐝𝐚𝐭𝐚 𝐛𝐞𝐡𝐚𝐯𝐢𝐨𝐮𝐫
𝐛𝐞𝐡𝐚𝐯𝐢𝐨𝐮𝐫
⇑ ⇑ ⇑ ⇑
𝑐𝑜𝑟𝑟𝑒𝑠𝑝𝑜𝑛𝑑𝑒𝑛𝑐𝑒 𝑐𝑜𝑟𝑟𝑒𝑠𝑝𝑜𝑛𝑑𝑒𝑛𝑐𝑒 𝑐𝑜𝑟𝑟𝑒𝑠𝑝𝑜𝑛𝑑𝑒𝑛𝑐𝑒 𝑐𝑜𝑟𝑟𝑒𝑠𝑝𝑜𝑛𝑑𝑒𝑛𝑐𝑒

𝐢𝐝𝐞𝐚𝐥𝐢𝐬𝐞𝐝 𝐦𝐨𝐝𝐞𝐥 : 𝑦= 𝑥 ∧ 𝑥 = 1.41 … 𝑛 ⇒ 𝑦 → 2

𝑐𝑜ℎ𝑒𝑟𝑒𝑛𝑐𝑒 𝑛→∞
𝑛𝑢𝑚𝑒𝑟𝑖𝑐𝑎𝑙 𝑑𝑖𝑠𝑐𝑟𝑒𝑡𝑒
𝑚𝑜𝑑𝑒𝑙 𝑝𝑎𝑟𝑎𝑚𝑒𝑡𝑟𝑖𝑐 𝑑𝑎𝑡𝑎 𝑑𝑖𝑠𝑐𝑟𝑒𝑡𝑒
𝑠𝑜𝑙𝑢𝑡𝑖𝑜𝑛

coherence

• Verification and validation

»
correspondence

𝐞𝐱𝐩𝐞𝐫𝐢𝐦𝐞𝐧𝐭𝐚𝐥 : 𝐠𝐞𝐧𝐞𝐫𝐚𝐥 ⇐ 𝐩𝐚𝐫𝐭𝐢𝐜𝐮𝐥𝐚𝐫 ∧ 𝐩𝐚𝐫𝐭𝐢𝐜𝐮𝐥𝐚𝐫

𝐦𝐞𝐚𝐬𝐮𝐫𝐞𝐦𝐞𝐧𝐭𝐬 𝑖𝑛𝑑𝑢𝑐𝑡𝑖𝑜𝑛 𝐝𝐚𝐭𝐚 𝐛𝐞𝐡𝐚𝐯𝐢𝐨𝐮𝐫
𝐛𝐞𝐡𝐚𝐯𝐢𝐨𝐮𝐫
⇑ ⇑ ⇑ ⇑
𝒗𝒂𝒍𝒊𝒅𝒂𝒕𝒊𝒐𝒏 𝒗𝒂𝒍𝒊𝒅𝒂𝒕𝒊𝒐𝒏 𝒗𝒂𝒍𝒊𝒅𝒂𝒕𝒊𝒐𝒏 𝒗𝒂𝒍𝒊𝒅𝒂𝒕𝒊𝒐𝒏

𝐢𝐝𝐞𝐚𝐥𝐢𝐬𝐞𝐝 𝐦𝐨𝐝𝐞𝐥 : 𝑦= 𝑥 ∧ 𝑥 = 1.41 … 𝑛 ⇒ 𝑦 → 2

𝒗𝒆𝒓𝒊𝒇𝒊𝒄𝒂𝒕𝒊𝒐𝒏 𝑛→∞
𝑛𝑢𝑚𝑒𝑟𝑖𝑐𝑎𝑙 𝑑𝑖𝑠𝑐𝑟𝑒𝑡𝑒
𝑚𝑜𝑑𝑒𝑙 𝑝𝑎𝑟𝑎𝑚𝑒𝑡𝑟𝑖𝑐 𝑑𝑎𝑡𝑎 𝑑𝑖𝑠𝑐𝑟𝑒𝑡𝑒
𝑠𝑜𝑙𝑢𝑡𝑖𝑜𝑛

coherence

NAFEMS World Congress 2015 | 21-24 June | San Diego | California | USA
 Verification and Validation

• Validation is a correspondence test

– idealisation error is reduced by the validation process
• compares model predictions against experimental measurements
• error sources tested
– mathematical model error
– parametric data error

• Verification is a coherence test

– computation error is reduced by the verification process
• compares numerical model against analytical mathematical model solution
error sources tested
– approximation error
– calculation error

NAFEMS World Congress 2015 | 21-24 June | San Diego | California | USA
 Simulation process

• Application example

verification challenges the numerical accuracy of the computation

validation challenges the behavioural accuracy of the idealisation
Source : NAFEMS QSS Primer
NAFEMS World Congress 2015 | 21-24 June | San Diego | California | USA
 Simulation process

•
engineered
1. Simulation application object
experimental data

– idealisation idealisation idealisation

• idealise model from engineered object validation

– mathematical model
idealised model
– parametric data values mathematical
model and data

– approximation
• discretise the model approximation approximation
verification
– numerical model
– discrete data values
computational
model computation

– calculation numerical model

and discrete data
verification

• compute the solution

calculation
calculation
–
verification
verification
• approximation verification
– discretisation, truncation solution

• calculation verification
– Iteration, round-off simulation process
interpretation

– validation V&V process

• idealisation validation behavioural

– mathematical model prediction

– parametric data values

NAFEMS World Congress 2015 | 21-24 June | San Diego | California | USA
 Simulation process

• 2. Software testing application

– computational model
• compute solutions using increasingly refined models
• plot mesh convergence curves for DOFs of interest

– theoretical model
• compute theoretical order of discretisation convergence
• plot mesh convergence curves for DOFs of interest

– order of convergence
• theoretical vs computational
• identify sources of numerical error
– algorithm errors, code bugs

NAFEMS World Congress 2015 | 21-24 June | San Diego | California | USA
 Simulation process
• Conclusions

– Require a simulation process flow diagram for simulation control

• crucial to use an appropriate process flow diagram engineered
object
experimental data

– Base simulation process diagram advantages idealisation idealisation

validation

• logically derived using a syllogistic model idealised model

mathematical

• structure based on logically derived sources of error model and data

• can be used as a base to derive higher level diagrams approximation approximation

verification

computational

– Verification and validation

model computation
numerical model verification
and discrete data

• shown to be correspondence and coherence tests calculation

calculation
verification

solution

• Reference
simulation process
– Smith, J. M., Quality Management in Engineering Simulation, NAFEMS 2014 interpretation
V&V process

behavioural
prediction

NAFEMS World Congress 2015 | 21-24 June | San Diego | California | USA