Introduction to
Software Testing
CS 3250
Software Testing
[Ammann and Offutt, “Introduction to Software Testing,” Ch. 1, Ch. 2.1]
Spring 2024 – University of Virginia © Praphamontripong 1
What is Software Testing?
• Testing = process of finding input values to check against
a software (focus of this course)
Test case consists of test values and expected results
Goal – reveal faults
Test values Program Actual vs Expected
(inputs) results results
1. Testing is fundamentally about choosing finite sets of values from
the input domain of the software being tested
2. Given the test inputs, compare the actual results with the expected
results
• Debugging = process of finding a fault given a failure
Spring 2024 – University of Virginia © Praphamontripong 2
Testing Categories
Static testing Dynamic testing
• Testing without executing the • Testing by executing the
program program with real inputs
Software inspection and
• Unit testing, integration
some forms of analysis
testing, system testing,
Effective at finding certain acceptance testing, …
kinds of problems such as
problems that can lead to
faults when the program is
modified
• Inspection, walkthrough,
code review, informal review,
…
Spring 2024 – University of Virginia © Praphamontripong 3
Testing Categories (2)
White-box Gray-box Black-box
testing testing testing
Test with Test with Test from the outside
complete information incomplete information (functionality and
of the software of the software or behavior).
(architecture, how limited knowledge of Not require
components interact, internal details source code,
functions and (may know how architecture, detailed
operations, code, components interact knowledge about
rationale, …) but not have detailed internal program
knowledge about functions,
internal program rationale, …)
functions, source code,
rationale, …)
Spring 2024 – University of Virginia © Praphamontripong 4
Testing Categories (3)
Functional testing Non-functional testing
• Unit testing • Performance testing
• Integration testing • Load testing
• System testing • Stress testing
• Smoke testing • Security testing
• Acceptance testing (Alpha / • Compatibility testing
Beta • Reliability testing
• Agile testing • Usability testing
• Regression testing • Compliance testing
• Continuous integration • Conformance testing
testing
… and many more …
… and many more …
Spring 2024 – University of Virginia © Praphamontripong 5
Validation and Verification (IEEE)
Validation Verification
• Evaluate software at the end • Evaluate software at a given
of software development phase of the development
process
• Ensure compliance with the
intended usage • Fulfill the requirements
established during the
• Done by experts in the previous phase
intended usage of the
software, not developers • Requires technical background
on the software
• Done by developers at the
various stages of development
IV&V stands for “independent verification and validation”
Spring 2024 – University of Virginia © Praphamontripong 6
Testing and
SW Development Lifecycle
Check if software
Requirements Acceptance
does what the user
Analysis Test Test needs
Design
Information Check overall
Architectural System behavior w.r.t.
Design Test specs
Check interface
Subsystem Integration between modules
Design Test in the same
subsystem
Check interactions
Detailed Module of units and
Design Test associated data
structures
Check each unit
Unit (method)
Implementation
Test individually
[based in part on AO, p.23]
Spring 2024 – University of Virginia © Praphamontripong 7
Testing and
SW Development Lifecycle
Requirements Acceptance
Analysis Test
Validation
Test
Design
Information
Architectural System
Design Test
Subsystem Integration
Design Test
Verification
Detailed Module
Design Test
Unit
Implementation
Test
[based in part on AO, p.23]
Spring 2024 – University of Virginia © Praphamontripong 8
Goals based on
Test Process Maturity
Beizer’s scale for test process maturity
§ Level 0: There is no difference between testing and debugging
§ Level 1: The purpose of testing is to show correctness
§ Level 2: The purpose of testing is to show that the software
does not work
§ Level 3: The purpose of testing is not to prove anything
specific, but to reduce the risk of using the software
§ Level 4: Testing is a mental discipline that helps all IT
professionals develop higher quality software
[AO, p.9]
Spring 2024 – University of Virginia © Praphamontripong 9
Level 0 – Testing is Debugging
§ Level 0: Testing is the same as debugging
§ Not distinguish between
incorrect behavior and mistakes
in the program
§ Not help develop software that
is reliable
§ Adopted by most CS students L
§ Get programs to compile
§ Debug with few arbitrarily
chosen inputs or those provided
by instructors
[image: http://softwaretestingandqa.blogspot.com/2007/12/software-bug.html]
Spring 2024 – University of Virginia © Praphamontripong 10
Levels 1 – Software Works
§ Level 1: To show correctness (developer-biased view)
§ Correctness is impossible to
establish or demonstrate
§ What do we know if “no failures”?
§ Good software?
§ Bad tests?
§ No strict goal, no stopping rule or
formal test technique
§ No quantitatively way to
evaluate; test managers are
powerless
[image: http://simply-the-test.blogspot.com/2010/]
Spring 2024 – University of Virginia © Praphamontripong 11
Levels 2 – Software Doesn’t Work
§ Level 2: To show failure (tester-biased view)
§ A negative view puts testers and developers into an adversarial
relationship – bad team morale
§ What do we know if “no failures”?
[image: http://simply-the-test.blogspot.com/2010/]
Spring 2024 – University of Virginia © Praphamontripong 12
“Mature” Testing
Correctness cannot generally be achieved or
demonstrated through testing.
Testing can only show the presence of failure,
not the absence.
Developers and testers should be on the same boat.
How can we move to a team approach?
Spring 2024 – University of Virginia © Praphamontripong 13
Levels 3 – Risk Reduction
§ Level 3: To reduce the risk of using the software
§ There are risks when using software
§ Some may be small with unimportant consequences
§ Some may be big with important consequences, or even
catastrophic
§ Testers and developers cooperate to reduce risk
Spring 2024 – University of Virginia © Praphamontripong 14
Levels 4 – Quality Improvement
§ Level 4: To increase quality of the software
§ Testing should be an integral part of the development process
§ Testers become technical leaders à measuring and improving
software quality
§ Help developers improve the ability to produce quality software
§ Train developers
§ Testers and developers cooperate to improve the quality
§ Example: Spell checker
§ Purpose: to improve our ability to spell
§ Change of mindset:
“find misspelled words” à “improve ability to spell”
Spring 2024 – University of Virginia © Praphamontripong 15
How “Mature” is Your Testing?
Are you at level 0, 1, 2, 3, or 4?
We hope to train you to become “change agents” (level 4)
Spring 2024 – University of Virginia © Praphamontripong 16
Principles of Software Testing
Why testing is so hard
1. Exhaustive testing is impossible
2. We need to know when to stop testing
3. There is no silver bullet in software testing
4. Faults happen in some places more than others
5. Bug-free software does not exist
6. Testing is context-dependent
7. Verification is not validation
Spring 2024 – University of Virginia © Praphamontripong 17
Tactical Goals: Each Test
”If you don’t know why you’re conducting each test,
it won’t be very helpful” – Jeff Offutt
§ What is objective and requirement of each test?
§ What fact does each test try to verify?
§ What are the threshold reliability requirements?
§ What are the planned coverage levels?
§ How many tests are needed?
Spring 2024 – University of Virginia © Praphamontripong 18
Wrap-up
• A tester’s goal is to eliminate faults as early as possible
• Testing
Improves software quality
Reduce cost
Preserve customer satisfaction
• What’s next?
• Fault, error, failure
• Reachability, Infection, Propagation, and Revealability
(RIPR) model
Spring 2024 – University of Virginia © Praphamontripong 19
