steme olutif ® ‘software testing services Thompson Information Systems Consulting Limited Sue acm eL Cc) a ee a eat od London W1M 9DL, UK Lemna ac ened cu Oe Tene De Le Nee an Remo rary nae eR cee DU cue ed Pees Ue Cg ad Tel: +44 (0)7000 NeilTh (634584) Fax: +44 (0)7000 NeilTF (634583) email: [email protected] http:liwww.TISCL.com/ SaAgenda » [Introduction « ll Risk-Based Test Strategy « Ill Risk Based Test Planning and Organisation « IV Managing Test Execution and the End-Game e V Close, Q&A « Here’s the commercial bit: — Most of the material presented today is based on: — Risk-Based E-Business Testing, Gerrard and Thompson, PN a(c1e) ae] 031 e402 — Visit www.riskbasedtesting.com for samples. Were Romer eeN cy aU e Roe em Core SrPaul Gerrard SCM eee ace suct curve cl ares! pers UU Mra Cet hele IL C LU L oaet= sets aL Co) PROM Creme ET Me a emul eel aL AP Isle) UCP ar LSS Esa DAC een ae CC n atte nas Beles) Uel-] clam) remanence] oe Le LS) across the industry by many forward-looking organisations. pM Meal BO cere ee UM alice sce Restate Ce oles cite size Peecete cheat enomcuehirinier ee mcnciueliec tc ckol STM en Le Ole mac LCA macht R ie Ia developer, designer, project manager and consultant for small and large developments. Paul has engineering degrees from the Universities of Oxford and London, is Co-Programme Chair for the BCS SIG in Software Testing, a ule mea asses este CMe el Uae CPC Umer Cee SUC ae ltt Reale Teale Rl Tester Qualification whose aim is to establish a certification scheme for testing Cee cick nea ee cme ee Cmca igri) Serge eS felt UCM OLR ea LL oc Le EU CIGi la sels aco Nem Re Moma Nec Mee) SeNeil Thompson ANTM Suen ae es a mela se Meu ons sci ger ts ec Ree Reel Ter Mees ONE ee Ree Amel Mele coy ETRE urna oma eeu MSC LMU cone ocean ale Pee eure ester M lear eet enan cers ee uals systems analysis and project management. He has worked fora computer ee eMC Ue eee Te MRM CU i ‘organisation and two global management consultancies, and currently feels SUL eR EeeNeTe ella Wet oleae Le aeRO ULM omer LU MOORE MUL ee Ue LLL) ‘Computer Society (BCS) and the IEEE. Neil is active in the BCS Specialist Interest Groups in Software Testing and Configuration Management, He SU Meuse UM Reet cue ee ee ET) Neil studied Natural Sciences at Cambridge University (BA & MA degrees). He holds the ISEB Foundation Certificate in software testing and will be taking the eee mec Le aa I Nem RRsat Mee) SryPart | Introduction Nem ReMi ease Cuca Mamet Roce Slide 6Part 1 - Agenda e How Much Testing is enough? e When is the Product Good Enough? « Introduction to Risk « The V-Model and W-Model. Were Rome eeN ny aU e RoR Core StHow Much Testing is Enough? (can risks help?) Nee ROM easre Cicero Mam Crm cere Slide 7a Mi al=\=\e my) @at=ts)(-) emcee =)(e 18] weeks...” e The Project manager says — “Four testers for six weeks and that's it!” Pe Maemo cere h — It'll take longer than six weeks — It'll cost more than the budget allocated — It’s too big/complicated/risky for us to do properly eam (Pcie le) anole oa « Was it ever possible to do “enough” testing in these circumstances? De Re mocre Ce nee le Sr)Testing is time delimited...always No upper limit to the amount of testing Even in the highest integrity environments, time and cost limit what we can do Testing is about doing the best job we can in the i anteme Welle (=) Testers should not get upset, if our estimates are cut down (or ignored) The benefits of early release may be worth the risk But who knows the status of the risks and benefits? De Re mocire Soe ee ee le Sr)Did any tester ever get enough time to test? « No, of course not « We need to separate the two responses: — the knee-jerk reaction (a back-covering exercise prior to the post-disaster “I told you so!”) —a rational evaluation of the risks and response to doing ‘too little’ « Often, just like cub-scouts, testers “promise to do their best” and don’t make waves. De Re mocire Soe ee ee le eae)Risk-based test planning e lf every test aims to address a risk, tests can be prioritised by risk e It's always going to take too long so... — Some tests are going to be dropped — Some risks are going to be taken « Proposal: — The tester is responsible for making the project aware of the risks being taken — Only if these risks are VISIBLE, will management ever reconsider. De Re mocre Ce eee lee ES EaSo how much testing is enough? Enough testing has been planned when the stakeholders (user/customer, project manager, support, developers) approve: TESTS IN SCOPE Sema (8el- Mats col meoli olan In orm emo alien) THE TESTS THAT ARE OUT OF SCOPE — Risk is low OR these tests would not give confidence The amount and rigour of testing is determined by CONSENSUS. Nee ReMi ee cy aU e cree orem SearsWhen is the Product “Good Enough”? Version 1.0a ©2002 Systeme Evolutif Ltd and TISCL PCECompulsive behaviour - Consultants, ‘gurus’, academics preach perfection through compulsive behaviour: » product perfection through process maturity and continuous process improvement » all bugs are bad, all bugs could be found, so use more and more rigorous/expensive techniques Pome (oleL6 ual nl lela See R elas » you can't manage what you can’t count » etc. etc... « Process hypochondriacs can't resist. De Remora e nee lee ea“Good Enough” « James Bach* is main advocate of the ‘Good Enough’ view (also Yourdon and others) e Areaction to compulsive formalism —if you aim at perfection, you'll never succeed — your users/customers/businesses live in the real world, why don’t you? — compromise is inevitable, don’t kid yourself — guilt and fear should not be part of the process. See mere ieee Rec mec ae md eee mee ee De Re mocMre Soe nee le aL“Good Enough” means: 1. X has sufficient benefits 2. X has no critical problems 3. Benefits of X sufficiently outweigh problems 4. In the present situation, and all things considered, improving X would cause more harm than good. All the above must apply De Remora oC anaes eile eatContribution of testing to the release decision e Have sufficient benefits been delivered? — Tests must at Jeast demonstrate that features providing benefits are delivered completely « Are there any critical problems? — Test records must show that any critical problems have been corrected, re-tested, regression tested e ls our testing Good Enough? — Have we provided sufficient evidence to be confident in our assessment? De Reece Caneel le aeWho makes the release decision? « “Good enough’ is in the eye of the stakeholder e Testers can only say: — “at the current time, our tests demonstrate that: » the following features/benefits have been delivered » the following risks have been addressed » these are the outstanding risks of release...” e Stakeholders and management, not the tester, can then make the decision. De Reece Sen ee eile CCLIntroduction to Risk Version 1.0a ©2002 Systeme Evolutif Ltd and TISCL SCEThe definition of risk e Italian dictionary: Risicare, “to dare” e Simple generic definition: —‘The probability that undesirable events will occur” e In this tutorial, we will use this definition: “A risk threatens one or more of a project’s cardinal objectives and has an uncertain probability”. De Rem ocire Se nee lee Eee}Some general statements about ath Risks only exist where there is uncertainty If the probability of a risk is zero or 100%, it is not a risk Unless there is the potential for loss, there is no risk (“nothing ventured, nothing gained”) Matec We Ml go), cooe= oscil 0l =] -10 RW ama =) Ve 0) f@)[=1e4 Software development is inherently risky. De Remo cire Soe eee le cea)Cardinal Objectives e The fundamental objectives of the system to be built ¢ Benefits of undertaking the project » Payoff(s) that underpin and justify the project e Software risks are those that threaten the Cardinal Objectives of a project. A Remora SoCal eee le eeeThree types of software risk Process Risk variances in planning and estimation, shortfalls in staffing, failure to track progress, lack of quality ‘assurance and configuration managemen} Project Risk resource constraints, external interfaces, supplier relationships, contract restrictions: Primarily a management EME uehge te cca responsibility process are the main issues here. Product Risk lack of requirements stability, complexity, design quality, coding Testers are mainly quality, non-functional concerned with issues, test specifications, Product Risk ere eR Re ects eee Pen te CUR CoCo A Re mecire Suen eee ile eeQuantitative or qualitative e Risks can be quantified: — Probability = 50% — Consequence or loss £100,000 — Nice to use numbers for calculations/comparisons — But we deal in perceived risk, not absolute risk e Or qualified: — Probability is high medium or low — Consequence is critical, moderate, low — More accessible, usable in discussions, risk workshops. NT Remora Caneel le Slide 24Uncertainty Consequence — Do you know the consequences of failure? — Need user input to determine this — Some costs could be calculated but others are intangible (credibility, embarrassment etc.) eaxe)ey=)e)) (10 — The toughest one to call — Crystal ball gazing In conducting a risk assessment, make it clear what level of uncertainty you are working with Testing (and test results) can reduce uncertainty. ews) Ne un Remora yma eum ceeThe V-Model and W-Model We Reema e nee Sle cS)Cd Nene R era ut Ae eT Rom eaDre Romer eee aU am eer) ce}W-Model and static testing eed ead aC Permit) Prenat ind Nee un Remora a nee eR ceeW-Model and dynamic testing Crear ee tca actu) er en bear ST) iity Testing Coe iain! Peo Ste aan) boriary De Reece Sue Cee ie eeWhat do we mean by a (work) product? e Project documents: — schedule, quality plan, test strategy, standards « Deliverables: — requirements, designs, specifications, user documentation, procedures — software: custom built or COTS components, sub- systems, systems, interfaces — infrastructure: hardware, O/S, network, DBMS — transition plans, conversion software, training... De Re mocire SC eee le cea)What do we mean by testing? - Testing is the process of evaluating the deliverables of a software project — detect faults so they can be removed — demonstrate products meet their requirements = gain confidence that products are ready for use — measure and reduce risk « Testing includes: — static tests: reviews, inspections etc. — dynamic tests: unit, system, acceptance tests etc. De Re mocmire Cen eee lee Perelat Risk Based Test Strategy De Remo ye Se Nee cle SiPart Il - Agenda e Risk Management Process e The Role of Testing in Product Risk Management » Identifying Benefits and Risks e From Risks to Test Objectives « Generic Test Objectives and System ieXcre (UIC = Tat =) a1) - Test Objectives, Coverage and Techniques De Rem ocire Sanaa ee Slide 34Risk Management Process Ne Remo ese nee UB CLe oe SCProcess Risk identification — what are the risks to be addressed? Risk analysis — nature, probability, consequences, exposure Risk response planning — pre-emptive or reactive risk reduction measures Risk resolution and monitoring Stakeholders should be involved at all stages. De Re mocire Soe ee ee le eiAssessing consequences (loss) Severity Description Score Critical business objectives cannot be accomplished 5 High business objectives undermined Moderate _ business objectives affected Low slight effect on business Negligible no noticeable effect sion 1.0a ©2002 Systeme Evolutif Ltd and TISCL heAssessing probability (likelihood) Probability Description >80% almost certainly, highly likely 61-80% probable, likely, we believe 41-60% better than even, 50/50, we doubt, improbable 3 21-40% unlikely, probably not 2 1-20% chances are slight, highly unlikely 1 Tes} RUM ee eR ceeRisk exposure e Risks with the highest exposure are those of most concern « Worst case scenarios drive concerns e Risk EXPOSURE is calculated as the product of the PROBABILITY and CONSEQUENCE of the risk « Asimple notation is L? —where L? = LIKELIHOOD x LOSS. De Remora Cen ee ele ecWhat do the numbers mean? « Sometimes you can use numeric assessments — We may have experience that tells us » Likelihood is high (it always seems to happen) » Loss is £50,000 (that's what it cost us last time) e But often, we are guessing — Use of categories help us to compare risks — Subjective perceptions (never the same) — E.g. Developers may not agree with users on probability! e Maybe you can only assign risk RAG numbers - RED, AMBER, GREEN » The ability to compare is what is most important. Nem Rear eeN cy aa cree orem Ee}The danger slope ery Likely 2 > |3 3 |& = — De Remora e nee le eaRisk response planning e Do nothing! « Pre-emptive risk reduction measures = information buying Se fore ee eee here ach iain =| Bethe tein — contractual transfer « Reactive risk reduction measures — contingency plans — insurance Po Male ats) eatat= ecm ema ore( r=] che Ne Re mocire Sanaa eee eeRole of Testing in Product Risk Management Version 1.0a ©2002 Systeme Evolutif Ltd and TISCL SCFaults, failure and risk « System failures are what we fear e The faults that cause failures are our ia « Uncertainty is what makes us concerned: — what type of faults are present in the ei ciLia —how many faults are in the system? — did testing remove all the serious faults? « Testing helps us to address these uncertainties. De Rem ocMre SoCal eee le eaTesting helps to reduce risk If risk assessment steers test activity -—we design tests to detect faults —we reduce the risks caused by faulty products « Faults found early reduce rework, cost and time lost in later stages « Faults found are corrected and re-tested and so the quality of all products is improved. De Re mocire Seen le oe)Testing can measure risk Testing is a measurement activity Tests that aim to find faults provide information on the quality of the product — which parts of the software are faulty — which parts of the software are not faulty Tests help us understand the risk of release Understanding the risks helps us to make a risk-based decision on release After testing, our risk assessment can be refined. De Re mocre Cen ee eee eetThe test passes... « The risk could be unchanged because: e Risk probability higher because: « Risk probability lower because: e Risk consequence higher because: « Risk consequence lower because: De Remo ce Soe Une ee lee eeThe test fails... « The risk could be unchanged because: e Risk probability higher because: « Risk probability lower because: e Risk consequence higher because: « Risk consequence lower because: De Remo cMre Se ee ele eCIdentifying Benefits and Risks Dr Remo ese eeUcLe SEBusiness benefits (cardinal objectives) « Atthe highest level, normally set out in a project initiation document or business case etc. « A benefit is any 'good thing’ required to be achieved by a project « Normally expressed in financial/business terms e.g. » Save money - one or a combination of — cut staff, stock, work in progress, time to deliver... « Increase revenues - one or a combination of — increase market share, launch new product, improve existing product, increase margins, exploit a new ioe De Remora Ce nee ele ee)Risk identification Expert interviews Independent consultant or domain expert Felctotsctola atoll Past experience (lessons learned) Checklists of common risks (risk templates) Risk workshops Brainstorming. Ne mocre SC e nee Cle erRisk workshops e Brainstorming sessions can be very productive e Make risks visible e Generates risk ideas « Generates ideas for resolution e Starts buy-in. Ne Re mocire Sen ee ee lee PeesExercise Fela Waire 0 les om la AW YW — Validation of a customers card and PIN — Cash withdrawal — On-line balance request — Request a statement e ...amongst others. De Re mocMre Soe nee Cle PeWhat kind of failures could occur for each of the four requirements? Pov POV PT Po a | a | Pee A Rema Sen ee Sle SERisks and viewpoints e Viewpoint has an influence over which risks are deemed important e Developer/supplier viewpoint —what stops us getting paid for the system? e Customer/service provider viewpoint —what could lose us business, money? e Customer viewpoint —what could lose ME money? De Re mocMire Sen eee Clee et)The testers viewpoint e Typically, testers represent either suppliers or their customers e Main stakeholders in the project: — system supplier — customer/buyer — service provider and support — end-users « Testers may work for one stakeholder, but should consider concerns of all. Ne Re mocre SoC ee eee erFrom Risks to Test Objectives Ne Reema ene Le CEEWhy use risks to define test objectives? e If we focus on risks, we know that bugs relating to the selected mode of failure are bound to be important. If we focus on particular bug types, we will probably be more effective at finding those bugs If testers provide evidence that certain failure modes do not occur in a range of test scenarios, we will become more confident that the system will work in production. De Rem ocMre SoCal eee le edRisks as failure modes or bug types e Risks describe ‘what we don’t want to intele)el- 10 ¢ Typical modes of failure: — calculations don’t work — pages don't integrate — performance is poor — user experience is uncomfortable « Think of them as ‘generic bug types’. De Rem ocmre Sanaa le eeDefining a test objective from ath e We ‘turn around’ the failure mode or risk e Risk: — a BAD thing happens and that’s a problem for us Test objective: — demonstrate using a test that the system works without the BAD thing happening Pome — execute important user tasks and verify the BAD things don't happen in a.range of scenarios. Ne Re mocire Sen ee ee le See}Risks and test objectives - examples 2 a The web site fails to function |To demonstrate that the application functions correctly on the user's client | correctly on selected combinations. of operating system and browser | operating systems and browser version configuration. combinations. (eee Cue eee BR Cesena les pce Rueme Cs Peel MM Ula cease RTT) browser do not match records | back-end legacy systems. in the back-end legacy [TONSA RSS SCL Vulnerabilities that hackers | To demonstrate through audit, scanning and could exploit exist in the web | ethical hacking that there are no security site networking infrastructure. | vulnerabilities in the web site networking (Uae SLCC es COk NR Re meee eee lee etaGeneric Test Objectives and SNC 1A gM ac slelUl (a=) at (o1a1 es) De Reema Cen ee Ue Le SLRisk-based test objectives are usually not enough e Other test objectives relate to broader issues — contractual obligations — acceptability of a system to its users — demonstrating that all or specified functional or non- functional requirements are met — non-negotiable test objectives might relate to mandatory rules imposed by an industry regulatory authority and so on « Generic test objectives complete the definition of your test stages. De Remora SoCal eee le CLGeneric test objectives Test Objective ees Pee ke eneM ene mee emeney enna Demonstrate component is ready for reuse in larger sub-system Ponenic omc clement ckesnice a assembled/combined and collaborate Demonstrate system meets functional requirements ee cut aS) Demonstrate system meets non-functional requirements | Non-Functional System aS iia) Demonstrate system meets industry regulation Sys ciumemaces sites requirements: aca) peur ces smart omc ictal ec tne) (eit paces lees Testing Wee Crem etcetera nis SEs) Demonstrate system, processes and people meet Eerie) PeTSSia=rsisMee eu) aE a) Nemo cre sen ee ee Cle eaTests as demonstrations « “Demonstrate” is most often used in test objectives Better than “Prove” which implies mathematical certainty (which is impossible) But is the word “demonstrate” too weak? — it represents exactly what we will do — we provide evidence for others to make a decision Rem uAcaecc LIM ULNVAY AIR = LD Am =e dle la elm Cor cere tn) 91-1 0k (8) what is possible — so we really are only doing a demonstration of a small, sample number of tests. De Re mocMre Cue nee le SeBut tests should aim to locate faults, shouldn't they? » The tester’s goal: to locate faults « We use boundary tests, extreme values, invalid data, exceptional conditions etc. to expose faults: — if we find faults these are fixed and re-tested — we are left with tests that were designed to detect faults, some did detect faults, but do so no longer Pe AVM lem cum MeN Ae (ci nero ATe| MUA micto1 [nom (O11 cost e Next steps are: — more detailed and confident test plans for each stage — bottom-up estimates to check against top- down — preparing to manage test specification & execution to time, cost, quality... De Remo cMre Cen eee le SasTest Plans for each testing stage eg for bel Testing: te ana hae tse thaon ol st objective F2 Precast a ie) Risk F432 ——=_ Test objective F4 ——— Risk Ul 12. ———— Test objective Ul ———— Risk U2 25. ———— Test objective U2 ——— 1 lO md Coe) (ood oO a Risk $1; 100 7 — 111) oh) a Cos) Co) eee a eee ey eer eet ey lies Peete et eae tice a in cms | | i | | | | | | | Leelee tanta ea my Cc a De Rem ocMre Sanaa le Slide 118 on a atePlan to manage risk during test execution Quality Cost mn} Ne an ROM ori rary nee ere ceTest Design: target execution schedule atone ENVIRONMENTS TEAMS TESTERS 23 456 7 8 CEU NSRP a ERE REI Eee esd toy ae) Leu = Te Me Cole Tool [es sir sre eA tL é Lz ulealey 3 Peed A Hf e eens co) ae Os RCs Ne un Rem oc Wires a Nae oR LeTest Specification and Risk Traceability We Remo yeS Ten ee cle Sara)What test specificat documents after De Master Test Flan Stage [ct att eee ete Ne un ReMi rary nae ere ce ion sign? cution & management procedures aC fe ee Cases “Scripts” Data (Test procedure specifications) Tae RARERContext-driven approach affects test documentation ta Teal aici Cet aN rr i See eae) Cid ‘ CONTEXT-DRIVEN Ps) I i ' I I i] ! ! i ! ! ) ! 1 I i) 1 Ne an ReMi rary nee eR ceVariables in test documentation » Format of the documents: automated tool, spreadsheet, word processor, or even hand-written cards! - Test procedures: detailed content of tests if needed, and separate overall process « Expected outcomes: explicit or implicit? e Test data: synthetic and lifelike. De Re mocMre Soe nee lee es)How much test documentation macro Ul incte ite « Stake out two extremes, eg: — “best practice” documentation, reviewed — spreadsheet overviews, no review Identify the requirements each “extreme pre- requisite” is really trying to address, eg: — anyone can execute; efficiency is paramount e Distil common objective, question assumptions, inject refinements; best fit might be a mixture of documentation styles tailored to your project — Sources: Software documentation superstitions (Daich); Goldratt’s Theory of Constraints Cea ae Ne un ReMi rary mae eR ceTraceability of risks e Many projects use risk analysis already, but... e Maintaining traceability of risks is a challenge e For risk-based reporting, need clerical help or automation (more work needed on this) « Beware over-complex risk relationships e Tactical solution is usually spreadsheets. De Re moc Mee nee lee Bl ae)cae Managing Test Execution and the End-Game Ne eRe ee Ne cuce Uae Om Loe SagPart IV - Agenda e Progress and Incident Management e Risk and Benefits-Based Test Reporting « Consensus and Risk-Based Acceptance. Nene Romer eee Uae em oem Bl aes)Progress and Incident Management Version 1.0a ©2002 Systeme Evolutif Ltd and TISCL Sars)Risk reduction components - Tests passed reduce perceived risk e Components of risk reduction are: — progress through tests — severities of incidents reported — progress through fault-fixing and retesting — first quantitative (have we done enough testing?), then — qualitative (can we tolerate remaining specific risks?). De Re mocre Ce nee ele ai)Incident classification and entry- exit criteria « Classifying incidents: — priority: how much testing it interrupts (“urgency”) — severity: business impact if not fixed (“importance”) — three levels of each may be enough (ten too many) » Entry and exit criteria: — entry = preparations + adequate exit from previous stage(s) — exit: build, delivery or release can proceed to next stage. NE Remora eee Cle BS aEProgress through tests « We are interested in two main aspects: — can we manage the test execution to get complete before target date? — if not, can we do it for those tests of high (and medium?) importance? Ses # tests Target tests passed — Actual tests passed Actual tests run EEG - erly De Rem ocre Cen ee ee le BaieProgress through incident fixing and retesting e Similarly, two main aspects: — can we manage the workflow to get incidents fixed and retested before target date? — if not, can we do it for those of material impact? # ineidents ane aity Cumulative incidents material impa EIS De Re mocM eS e nee le Ei)Quantitative and qualitative risk reduction from tests and retests PROGRESS & RESIDUAL RISK UP RIGHT SIDE OF W-MODEL Con] Large-Scale Acceptance 4 Treen etn atest Ld C016) Meh) ma citeleiei| Bua PROGRESS mu chcel8.e) a) INCIDENT 1a aI Tey & RETESTING Nemo cMire Cen ee ee lee aiRisk and Benefits-Based Test Reporting We Remo ese Nee Cle Si)Risk-based reporting start today ee all risks eloU risks of releasing felons Residual ey Progress through the test plan Ne Re meee e nee ee Si)Benefits of risk-based test reporting Ne an Remo rary nae eR ce Risk of release is known: — On the day you start and throughout the test phase — On the day before testing is squeezed Progress through the test plan brings positive results — risks are checked off, benefits available Pressure: to eliminate risks and for testers to provide evidence that risks are gone We assume the system does not work until we have evidence — “guilty until proven innocent” Reporting is in the language that management and stakeholders understand. ACRESBenefit & objectives based test reporting a Version 1.0a ©2002 Systeme Evolutif Ltd and TSCL Peter eel come] ts eaeBenefits of benefit-based test reporting Risk(s) that block every benefit are known: — On the day you start and throughout the test phase — Before testing is squeezed Progress through the test plan brings positive results — benefits are delivered Pressure: to eliminate risks and for testers to provide evidence that benefits are delivered We assume that the system has no benefits to deliver until we have evidence Reporting is in the language that management and stakeholders understand. Nee ae ReMi rary nae ere ce SORE)How good is our testing? e Our testing is good if it provides: — Evidence of the benefits delivered — Evidence of the CURRENT risk of release — At an acceptable cost —In an acceptable timeframe e Good testing is: — Knowing the status of benefits with confidence — Knowing the risk of release with confidence. Nee Romer eeN cy aU e orem oem CaryConsensus and Risk-Based Acceptance A Remove SCN ee Le SarStakeholders and acceptance hierarchy « Testers build consensus among differing viewpoints of stakeholders, but « Senior management will typically accept system, eg: customer-facing? PCat Lece loli ee hopes Project & as management [Gees User Acceptance Operational Acceptance tester-facilitated consensus DR Remora Cue nee le SarTest Review Boards and degrees of approval To ascend the approval and acceptance hierarchy, testers facilitate Testing Review Boards, eg: « Decisions could be: = unqualified — qualified met Oct eNf Pilot start Operational Acceptance Large-scale Integration Nee Roemer eeN cy aU e Roe em Cole aE)Slippages and trade-offs: an example « If Test Review Boards recommend delay, management may demand a trade-off, “slip in a little of that descoped functionality” « This adds benefits but also new risks: original first: ETAtteLE bears oar slip: fefem Md Err) st) date De Re mocre Caneel le aTolerable risk-benefit balance: another example « Even if we resist temptation to trade off slippage against scope, may still need to renegotiate the tolerable level of risk balanced against benefits: (risk benefits) “go for it” _ er Ce na a 3 date Sar Ne un ReMi rare nae ee ceClosing Comments A Reema Cena cle SCRE)Risk-based test approach: planning « RBT approach helps stakeholders: — They get more involved and buy-in — The have better visibility of the test process Pind 0 M19) e1cer- (ea Mt =| ecm Croll eS - Clear guidance on the focus of testing — Approval to test against risks in scope — Approval to not test against risks out of scope — Clearer test objectives upon which to design tests. De Re meee Sen eee le argRisk-based test approach: execution and reporting e RBT approach helps stakeholders: — They have better visibility of the benefits available and the risks that block benefits « RBT approach helps management: — To see progress in terms of risks addressed and benefits that are available for delivery — To manage the risks that block acceptance — To better make the release decision. De Remora Se nee lee Sar)Risk-Based Testing Close PUI Medel S110) aig Document templates can be found at www.riskbasedtesting.com De Remora eee le SCRE)