CSE 331 Read this.
Software Design and Implementation
http://blog.regehr.org/archives/199
Lecture 15
Debugging
Zach Tatlock / Winter 2017
A Bug’s Life Ways to get your code right
defect – mistake committed by a human Design + Verification
– Ensure there are no bugs in the first place
error – incorrect computation
Testing + Validation
failure – visible error: program violates its specification – Uncover problems (even in spec?) and increase confidence
Defensive programming
Debugging starts when a failure is observed – Programming with debugging in mind, failing fast
Unit testing Debugging
Integration testing – Find out why a program is not functioning as intended
In the field
Testing ≠ debugging
Goal is to go from failure back to defect – test: reveals existence of problem
– Hard: trying to solve an “inverse problem” (work backward) • test suite can also increase overall confidence
– debug: pinpoint location + cause of problem
Defense in depth First defense: Impossible by design
Levels of defense: In the language
– Java prevents type mismatches, memory overwrite bugs;
1. Make errors impossible guaranteed sizes of numeric types, …
– Examples: Java prevents type errors, memory corruption
2. Don’t introduce defects In the protocols/libraries/modules
– “get things right the first time” – TCP/IP guarantees data is not reordered
– BigInteger guarantees there is no overflow
3. Make errors immediately visible
– Examples: assertions, checkRep In self-imposed conventions
– Reduce distance from error to failure – Immutable data structure guarantees behavioral equality
4. Debug [last level/resort: needed to get from failure to defect] – finally block can prevent a resource leak
– Easier to do in modular programs with good specs & test suites Caution: You must maintain the discipline
– Use scientific method to gain information
Second defense: Correctness Strive for simplicity
Get things right the first time
There are two ways of constructing a software
– Think before you code. Don’t code before you think!
design: One way is to make it so simple that
– If you're making lots of easy-to-find defects, you're also making there are obviously no deficiencies, and the
hard-to-find defects – don't rush toward “it compiles” other way is to make it so complicated that
there are no obvious deficiencies. The first
Especially important when debugging is going to be hard
method is far more difficult.
– Concurrency, real-time environment, no access to customer Sir Anthony Hoare
environment, etc.
The key techniques are everything we have been learning:
Debugging is twice as hard as writing the
– Clear and complete specs
code in the first place. Therefore, if you write
– Well-designed modularity with no rep exposure the code as cleverly as possible, you are, by
– Testing early and often with clear goals definition, not smart enough to debug it.
– …
These techniques lead to simpler software
Brian Kernighan
Third defense: Immediate visibility Benefits of immediate visibility
If we can't prevent errors, we can try to localize them The key difficulty of debugging is to find the defect: the code
fragment responsible for an observed problem
Assertions: catch errors early, before they contaminate and are
– A method may return an erroneous result, but be itself error-
perhaps masked by further computation
free if representation was corrupted earlier
Unit testing: when you test a module in isolation, any failure is
due to a defect in that unit (or the test driver) The earlier a problem is observed, the easier it is to fix
Regression testing: run tests as often as possible when – In terms of code-writing to code-fixing
changing code. If there is a failure, chances are there's a – And in terms of window of program execution
mistake in the code you just changed
Don’t program in ways that hide errors
If you can localize problems to a single method or small module, – This lengthens distance between defect and failure
defects can usually be found simply by studying the program text
Don't hide errors Don't hide errors
// k must be present in a // k must be present in a
int i = 0; int i = 0;
while (true) { while (i < a.length) {
if (a[i]==k) break; if (a[i]==k) break;
i++; i++;
} }
This code fragment searches an array a for a value k Now the loop always terminates
– Value is guaranteed to be in the array – But no longer guaranteed that a[i]==k
– What if that guarantee is broken (by a defect)? – If other code relies on this, then problems arise later
Don't hide errors Inevitable phase: debugging
// k must be present in a Defects happen – people are imperfect
int i = 0; – Industry average (?): 10 defects per 1000 lines of code
while (i < a.length) {
if (a[i]==k) break; Defects happen that are not immediately localizable
i++; – Found during integration testing
} – Or reported by user
assert (i!=a.length) : "key not found";
step 1 – Clarify symptom (simplify input), create “minimal” test
• Assertions let us document and check invariants step 2 – Find and understand cause
• Abort/debug program as soon as problem is detected step 3 – Fix
– Turn an error into a failure step 4 – Rerun all tests, old and new
• Unfortunately, we may still be a long distance from the defect
– The defect caused k not to be in the array
The debugging process The Debugging Process
step 1 – find small, repeatable test case that produces the failure
– May take effort, but helps identify the defect and gives you a Observation
regression test
– Do not start step 2 until you have a simple repeatable test
step 2 – narrow down location and proximate cause Form Hypothesis
– Loop: (a) Study the data (b) hypothesize (c) experiment
– Experiments often involve changing the code
Design Experiment
– Do not start step 3 until you understand the cause
step 3 – fix the defect
– Is it a simple typo, or a design flaw? Run Test
– Does it occur elsewhere?
step 4 – add test case to regression suite
Fix Bug!
– Is this failure fixed? Are any other new failures introduced?
The Debugging Process Debugging and the scientific method
knowledge
• Debugging should be systematic
Observation – Carefully decide what to do
• Don’t flail!
– Keep a record of everything that you do
Form Hypothesis t – Don’t get sucked into fruitless avenues
Design Experiment • Use an iterative scientific process:
Run Test
Fix Bug!
Example Reducing absolute input size
// returns true iff sub is a substring of full Find a simple test case by divide-and-conquer
// (i.e. iff there exists A,B such that full=A+sub+B)
boolean contains(String full, String sub); Pare test down:
User bug report: Can not find "very happy" within
It can't find the string "very happy" within: "Fáilte, you are very welcome! Hi Seán! I am
"Fáilte, you are very welcome! Hi Seán! I am very very happy to see you all."
very very happy to see you all." "I am very very happy to see you all."
Poor responses: "very very happy"
– See accented characters, panic about not knowing about Can find "very happy" within
Unicode, begin unorganized web searches and inserting poorly "very happy"
understood library calls, … Can not find "ab" within "aab"
– Start tracing the execution of this example
Better response: simplify/clarify the symptom…
Reducing relative input size General strategy: simplify
Can you find two almost identical test cases where one gives the correct In general: find simplest input that will provoke failure
answer and the other does not? – Usually not the input that revealed existence of the defect
Can not find "very happy" within Start with data that revealed the defect
"I am very very happy to see you all." – Keep paring it down (“binary search” can help)
– Often leads directly to an understanding of the cause
Can find "very happy" within
"I am very happy to see you all.” When not dealing with simple method calls:
– The “test input” is the set of steps that reliably trigger the
failure
– Same basic idea
Localizing a defect Binary search on buggy code
public class MotionDetector {
private boolean first = true;
Take advantage of modularity private Matrix prev = new Matrix();
no problem yet
– Start with everything, take away pieces until failure goes away
– Start with nothing, add pieces back in until failure appears public Point apply(Matrix current) {
if (first) {
prev = current;
Take advantage of modular reasoning } Check
– Trace through program, viewing intermediate results
Matrix motion = new Matrix(); intermediate
getDifference(prev,current,motion); result
applyThreshold(motion,motion,10); at half-way point
labelImage(motion,motion);
Binary search speeds up the process
Hist hist = getHistogram(motion);
– Error happens somewhere between first and last statement int top = hist.getMostFrequent();
– Do binary search on that ordered set of statements applyThreshold(motion,motion,top,top);
Point result = getCentroid(motion);
prev.copy(current);
return result; problem exists
}
}
Binary search on buggy code Logging Events
public class MotionDetector {
private boolean first = true; Log (record) events during execution as program runs (at full speed)
private Matrix prev = new Matrix();
no problem yet
Examine logs to help reconstruct the past
public Point apply(Matrix current) { Check – Particularly on failing runs
if (first) { intermediate – And/or compare failing and non-failing runs
prev = current; result
} at half-way point The log may be all you know about a customer’s environment
Matrix motion = new Matrix();
getDifference(prev,current,motion); – Needs to tell you enough to reproduce the failure
applyThreshold(motion,motion,10); problem exists
labelImage(motion,motion); Performance / advanced issues:
Hist hist = getHistogram(motion); – To reduce overhead, store in main memory, not on disk
int top = hist.getMostFrequent(); (performance vs stable storage) (???)
applyThreshold(motion,motion,top,top); – Circular logs avoid resource exhaustion and may be good
Point result = getCentroid(motion); enough (???)
prev.copy(current);
return result;
}
}
Detecting Bugs in the Real World Debugging In Harsh Environments
Real Systems Failure is non-deterministic,
– Large and complex (duh J) difficult to reproduce
– Collection of modules, written by multiple people
– Complex input
– Many external interactions
– Non-deterministic Can’t print or use debugger
Replication can be an issue
– Infrequent failure
– Instrumentation eliminates the failure Can’t change timing of program
Defects cross abstraction barriers (or failure depends on timing)
Large time lag from corruption (defect) to detection (failure)
Look inside the machine “Heisenbugs”
In a sequential, deterministic program, failure is repeatable
Mark Oskin was hacking on a kernel.
But the real world is not that nice…
– Continuous input/environment changes
No GDB, no printf, no kprintf, … – Timing dependencies
– Concurrency and parallelism
But, did have beep from mobo! Failure occurs randomly
– Literally depends on results of random-number generation
Bugs hard to reproduce when:
– Use of debugger or assertions makes failure goes away
• Due to timing or assertions having side-effects
– Only happens when under heavy load
– Only happens once in a while
More Tricks for Hard Bugs Where is the defect?
The defect is not where you think it is
Rebuild system from scratch, or restart/reboot
– Ask yourself where it can not be; explain why
– Find the bug in your build system or persistent data structures
– Self-psychology: look forward to being wrong!
Explain the problem to a friend (or to a rubber duck)
Look for simple easy-to-overlook mistakes first, e.g.,
Make sure it is a bug – Reversed order of arguments:
– Program may be working correctly and you don’t realize it! Collections.copy(src, dest);
– Spelling of identifiers: int hashcode()
And things we already know: @Override can help catch method name typos
– Minimize input required to exercise bug (exhibit failure) – Same object vs. equal: a == b versus a.equals(b)
– Add more checks to the program – Deep vs. shallow copy
– Add more logging Make sure that you have correct source code!
– Check out fresh copy from repository; recompile everything
– Does a syntax error break the build? (it should!)
When the going gets tough Key Concepts
Testing and debugging are different
Reconsider assumptions
– Testing reveals failures, debugging pinpoints defect location
– e.g., has the OS changed? Is there room on the hard drive?
Is it a leap year? 2 full moons in the month? Debugging should be a systematic process
– Debug the code, not the comments – Use the scientific method
• Ensure that comments and specs describe the code
Understand the source of defects
Start documenting your system
– To find similar ones and prevent them in the future
– Gives a fresh angle, and highlights area of confusion
Get help
– We all develop blind spots
– Explaining the problem often helps (even to rubber duck)
Walk away
– Trade latency for efficiency – sleep!
– One good reason to start early
