AMP Training
Module Ten Part Three
Audit of predetermined objectives
Reliability
Module outline
1. Reliability assertions and audit criteria
2. Basic risk rules for reliability
3. Population for testing
4. Accept-reject sampling for tests of details
5. Means of selecting items for testing
6. Accumulation and evaluation of reliability
misstatements
AoPO Reasonable assurance: Execution per subject matter (SM)
ToC
Materiality
Risk assessment and Accept-reject
response sampling
U
n
d
e Risk Register
r
Lead schedule
s SToD: Testing
SToD: Approach &
t single/once off
accumulation per target
a achievements
n
SToD: Testing
d
C,M,R,P&D
i
n SToD: Testing per
SToD: Testing per
popultation
g
population
Adjustments
Overall
Conclusion Evaluation of uncorrected misstatements and concluding
summary
For SM per indicator For SM per target Per target Per population per assertion
①
Reliability assertions and
audit criteria
Reliability assertions and audit criteria
Reliability
Assertions
Substantive/
• Validity
combined approach
• Accuracy
• Completeness
Pertaining to actual performance/target
achieved
Various working papers
depending on the target
and approach to testing
Reliability - Assurance bucket
• Approach determined per assertion for a
performance indicator
• Planned approach per indicator applies to all
related targets TARGET
• Combined vs substantive
Size 3
Tests of details
Size 2 Size 2
Substantive analytical Substantive analytical
procedures procedures
Tests of details
Tests of
Tests of controls controls
Completeness Validity Accuracy
Nature of audit procedures
Purpose
To obtain sufficient appropriate evidence at the required
level of assurance
Type
• Inspection
• Observation Nature
• Inquiry
• Confirmation
• Recalculation
• Re-performance
• Analytical procedure
Nature of audit procedures
Further
Audit
Procedures
Test of Substantive
Controls Procedures
Substantive
Test of
Analytical
Detail
Procedures
Nature of audit procedures
• Can CAATS be used?
• To what extent can CAATs be used in designing
and executing test of details:
Perform more extensive testing of electronic files
Select sample events/occurrences/items/transactions
from key electronic files
Sort events/occurrences/items/transactions with
specific characteristics
Test an entire population instead of a sample.
• ISA involvement
Direction of testing
Performance records Source documents
Validity
Accuracy Using highest
level of
Validity and Accuracy assurance
required
Performance records Source documents
Completeness
Primary vs
secondary
Substantive analytical procedures for
AoPO
Approach with care
• Suitable?
• Able to predict outcome of target?
• Logical relationship between the recorded value
and the data to be used to predict
• Plausible and predictable?
• Less effective when testing target subject to
management discretion
②
Basic risk response rules
Reliability
Basic risk response rule
Reliability
• Further audit procedures in responsive to risk of MM
• More persuasive audit evidence
• Circumstances appropriate for ToC
• SP’s for all material performance indicator/measure
and targets
• SP’s for significant risk SToD if fully substantive
approach
• Special procedures
Special consideration items
Performance information closing process
Rule 1 -
Further audit procedures responsive to
assessed risk of MM
• Further audit procedures responsive to the
assessed risks of material misstatement at
the assertion level; referring to the validity,
accuracy and completeness
TARGET
Size 3
proedures
Tests of details
Size 2 Size 2
Substantive analytical Substantive analytical
Tests of details procedures
Tests of controls Tests of
controls
Completeness Validity Accuracy
Rule 2 -
More persuasive audit evidence the
higher the risk
3 Significant
2 Standard
1 Minimal
Rule 3 –
Are test of controls appropriate?
• Sub-rule: If SP alone cannot provide sufficient
appropriate audit evidence
• Sub-rule: If the auditor plans to rely on the
operating effectiveness of identified controls
Similar to AFS
Rule 4 –
Always SPs for material performance
measure/ indicator and targets
Irrespective of the assessed risks of material
misstatement substantive procedures are
performed in relation to each material
performance measure/indicator and its targets
ALL material NB
Also test ALL non-
performance indicators
material indicators and
and related targets are
related targets
tested
Testing of non-material indicators
Always: Required procedures
relevant to performance
information closing process (rule 7)
Specific procedures if affected by
specific considerations i.e.
auditing of estimates (rule 6)
Further SPs if risk significant or
standard
No further SPs required if
minimal risk and ALL conditions
are met…..
Testing of non-material indicators …
No further SPs required if minimal
risk and ALL conditions are met
The likelihood
NO significant
NO prior year due to other
risk indicators
misstatements risk factors is
are present
LOW
Rule 5 –
Specific SP’s for significant risk
3 Significant
• Auditor’s response must include substantive
procedures specifically responsive to risk
• If substantive approach followed, must include
test of details
Rule 6 –
Specific SP’s for specific consideration
items
• When a measure/indicator and any of its targets
is affected by identified special consideration,
perform required procedures
• Specific consideration?
Auditing of estimates
Using the work of internal auditors
Using the work of an auditor’s expert
Using the work of “others”
Rule 7 –
Performance information closing process
NB
SP’s related to the collation, summary and finalisation
of information on actual target achievement
a) Agree/reconcile the reported information with
underlying records
b) Examine adjustments to the reported information
on target achievement in the process of collating,
summarising and closing-off
Lead schedule
Roll forward rules
Interim test performed
• AFS rules apply
• Refer to Chapter 10.1
③
Population for testing
Population
What is a population ?
Is the entire set of data from which the
auditor will select items for testing about
which the auditor wishes to draw
conclusions
At what level?
Actual target achievement for each
performance target represents a separate
population for testing
Population
What will influence the population for testing?
Nature and characteristics:
Unit of Assertions tested Stratification
measurement
i.e. direction of i.e. sub-
i.e. single /once – testing populations or
off achievements, stratums with
R-value, number, separately
percentage / identifiable
ratio/rate characteristics
④
Accept – Reject
Accept-reject sampling for tests of
details
⑤
Means for selecting items
for testing
Selection hierarchy
Selecting all items
•
Selecting specific items only
Using a combination of specific items and audit
sampling
Audit sampling only
SToD: Approach and
accumulation
AoPO Reasonable assurance: Execution per subject matter (SM)
ToC
Materiality
Risk assessment and Accept-reject
response sampling
U
n
d
e Risk Register
r
Lead schedule
s SToD: Testing
SToD: Approach &
t single/once off
accumulation per target
a achievements
n
SToD: Testing
d
C,M,R,P&D
i
n SToD: Testing per
SToD: Testing per
popultation
g
population
Adjustments
Overall
Conclusion Evaluation of uncorrected misstatements and concluding
summary
For SM per indicator For SM per target Per target Per population per assertion
Selecting all items
• Target is measured as a single/ once-off
achievement SToD: Testing per
population (Tab A)
• Target = 0
• Numbers, percentages, values, etc.
• Small number of items
• Significant risk with no other means of
testing
• CAATS
• Factual misstatements
Selecting specific items
• High value items associated with R – values
• Key items that exhibit some characteristic
related to the risk assessment
Not common for AoPO
SToD: Testing per
Factual misstatements population (Tab A)
SToD: Testing per
Sampling population (Tab B)
FIRST: Define the purpose, population and sampling unit
THEN FOLLOW THE FOLLOWING STEPS
Step 1 - Determine the sample size
Step 2 - Select the sample items
Step 3 - Perform the planned procedures on the selected
items
Step 4 - Evaluate the sample results and conclude on
sampling risk
Indicator: FTE Jobs
Indicator: % Coastline
SToD - AoPO - Testing single or
once-off achievements
Define the purpose, population and
sampling unit
• Define the purpose of the audit procedure
• Define the error condition
• Define the population
when audit sampling is the only means of testing?
when specific and audit sampling is used?
• The definition of the population for sampling must
Refer to the assertions being tested
Direction of testing
• Percentage unit of measure
• Stratification
• Source/source information where sample is selected is complete and
accurate
• Define the sampling unit
Step 1 - Determine the sample
size
Step 1 - Determine the sample
size
Apo tests of details sample sizes *Note*
Overall materiality = 10%
Level of assurance required
Performance Significant (= 3) Standard (= 2) Minimal (= 1)
materiality level of level of level of
assurance assurance assurance
5.0%
(10% x 50%) 60 40 20
7.5%
(10% x 75%) 40 27 14
Step 2 – Select the sample items
Step 2 – Select the sample items
Non-statistical or Statistical selection methods
Use the random selection method
Step 3 - Perform the planned procedures on the selected items
Integration with AFS and
Compliance audit
Performing AFS , AOPO and/ or compliance audit procedures concurrently
with the financial audit procedures on the same items
• Population from which the items will be selected – must be
the same for the relevant audit areas
• Individual items – must be the same for the relevant audit
areas
• Considerations and decisions clearly documented
• References and hyperlinks to the relevant AFS and/or
compliance SToD Accumulation working papers must be
included
Integration with AFS and
Compliance audit
Integration can be achieved by performing financial and/or compliance
audit procedures concurrently with the financial audit procedures on the
same items already selected for financial testing purposes
SToD Testing per population WP – Guidance note 32
Step 4 - Evaluate the sample
results
Misstatements identified
Disagreement:
Difference between information on actual performance supported by the
audit evidence and information as reported by the auditee
Limitation:
Audit evidence not available or not adequate
uncertain whether or not item is misstated
• Investigate and document nature, cause and circumstances (How, Why)
• Understand the inherent characteristics of the identified misstatements
Step 4 - Evaluate the sample results
Projected misstatement = auditors best estimate of
misstatement in population
Total factual misstatements in Total factual misstatements in
the sample (excl. anomalous) the sample (excl. anomalous)
(Total number of items in the (Total R-value of the sample) x
sample) x (Total number of (R-value of the population)
items in the population)
Conclude on sampling risk
TEMOR
• Total Estimated Misstatement Occurrence Rate
(TEMOR)
• All projected + anomalous misstatements
• Calculated for quantitative evaluation of
sampling risk only
• Not transferred for accumulation to
accumulation WP
Conclude on sampling risk
Sample risk reduced to an acceptable level?
Allowance for sample risk
Performance Overall
materiality materiality
5% / 7.5% 10%
Scenario 1 Scenario 2 Scenario 3
AoPO Reasonable assurance: Execution per subject matter (SM)
ToC
Materiality
Risk assessment and Accept-reject
response sampling
U
n
d
e Risk Register
r
Lead schedule
s SToD: Testing
SToD: Approach &
t single/once off
accumulation per target
a achievements
n
SToD: Testing
d
C,M,R,P&D
i
n SToD: Testing per
SToD: Testing per
popultation
g
population
Adjustments
Overall
Conclusion Evaluation of uncorrected misstatements and concluding
summary
For SM per indicator For SM per target Per target Per population per assertion
6
Accumulation and evaluation
of misstatements for
reliability
Accumulating reliability misstatements
All the misstatements identified in relation to a target are
accumulated including:
From ALL the different
For assertions –
populations (or sub- From the closing
Validity, accuracy and
populations/strata – if processes
completeness
applicable)
SToD: Approach and
NB accumulation
To determine the NET effect of the misstatement on the target.
Correction of misstatements
Net
Communicate uncorrected
Adjustments by Audit
all misstatements
management? adjustments
misstatements – calculate the
MOR per target
Calculate misstatement occurrence
rate
• Calculate MOR based on the total uncorrected
misstatement for the target – i.e. the net effect of all
uncorrected misstatements SToD: Approach
and accumulation
• Using the applicable formula
– Single / once-off achievement - 0% or 100%
– R-value/Numbers - Total uncorrected
misstatement/ Total misstatement
– Percentage / rate - Total uncorrected
misstatement / Reported achievement as
percentage
Programme/objective/development priority: [Provide description]
Audited value for
Performance Actual reported the target Total
measures/indicators performance uncorrected Misstatement
and their targets in the APR misstatements occurrence rate
(i.e. “Book identified for
value”) the target *Note 1*
Material measures/indicators and their related targets
Measure/indicator #1: [Provide description]
1.1 98% 87% 11% 11.2%
[Describe target] *Note 2*
> Numerator 334 296
> Denominator 340 340
1.2 590 590 0 0%
[Describe target]
1.3 1 500 1 550 50 3.3%
[Describe target]
Measure/indicator #2: [Provide description]
2.1 Approved policy No policy All 100%
[Describe target] document document
compiled
Measure/indicator #6: [Provide description]
Non-material measures/indicators and their related targets
Measure/indicator #3: [Provide description]
3.1 1 1 0 0%
[Describe target]
3.2 17 16 1 5.9%
[Describe target]
Evaluation of reliability misstatements
relating to material targets
MOR ≥ 10%
quantitatively material and
therefore material to the
programme/objective/development priority as a
whole
Evaluation of reliability misstatements
relating to material targets
MOR < 10%
Qualitative evaluation of the misstatement
• evaluate whether the misstatement is
material, individually or when considered
together with other misstatements
• nature, cause and circumstances
• qualitative considerations Chapter 15 para 65
Material misstatement relating to
reliability
• All targets to a material indicator are regarded
as a material target
• If material target is materially misstated,
indicator is materially misstated
• If only 1 material indicator is materially
misstated the subject matter is materially
misstated
AoPO Reasonable assurance: Execution per subject matter (SM)
ToC
Materiality
Risk assessment and Accept-reject
response sampling
U
n
d
e Risk Register
r
Lead schedule
s SToD: Testing
SToD: Approach &
t single/once off
accumulation per target
a achievements
n
SToD: Testing
d
C,M,R,P&D
i
n SToD: Testing per
SToD: Testing per
popultation
g
population
Adjustments
Overall
Conclusion Evaluation of uncorrected misstatements and concluding
summary
For SM per indicator For SM per target Per target Per population per assertion
