background and motivation

Everyone who's worked on a nontrivial application (like Firefox) has gone @@ -143,8 +293,7 @@

background and motivation

that they're literally not the worth the time to fix with traditional techniques. However, for big projects like Firefox with its half-billion users, a bug that only reproduces 1 out of - 10,000 test runs can still have a significant negative impact on - users. + 10,000 test runs can still have a negative impact on users.

rr solves these problems by splitting debugging into two phases: first recording, in which the application's execution @@ -162,9 +311,13 @@

background and motivation

The benefit to developers is obvious: an intermittent bug can be recorded by a script over lunchtime, say, and then debugged at leisure in the afternoon. Multiple cores can be used in - parallel to record failures. Even for easily reproducible bugs, - a repeatable, deterministic, debugging session is a powerful - tool on top of traditional debugging. + parallel to record failures. If you accidentally set a + breakpoint in the wrong place and miss gathering critical + information, your precious intermittent failure isn't lost. + Just fix your breakpoint and then tell gdb to run + the recording back from the beginning again. Even for easily + reproducible bugs, a repeatable, deterministic, debugging + session is a powerful tool on top of traditional debugging.

And for projects like Firefox which run literally millions of tests a day on a vast build and test infrastructure, @@ -194,7 +347,7 @@

background and motivation

Firefox, it is very likely to be generally useful. However, there's nothing about rr that's specific to Firefox; we've just spent the majority of our testing time on Firefox. rr - support will broaden as the project matures. + will work well with more programs as the project matures.

Prioritize deployability. rr runs on a stock linux kernel, and requires little (Ubuntu) to no (Fedora) system configuration changes. And of course rr works on @@ -209,11 +362,11 @@

background and motivation

Firefox test suites, we generally see about a 1.2x slowdown when recording with rr. That means that if the suite takes 10 minutes to run by itself, it will take around 12 minutes to be - recorded running under rr. + recorded by rr.

limitations

Some of rr's limitations are inherent, and some will be removed in future releases. @@ -239,8 +392,8 @@

limitations

further reference

This presentation provides an overview of the rr implementation and is meant for @@ -254,9 +407,19 @@

further reference

+ + From 3727afa90f78933d95547674c4f62c73b6d11fd4 Mon Sep 17 00:00:00 2001 From: Chris Jones Date: Sat, 22 Mar 2014 10:31:03 -0700 Subject: [PATCH 024/100] Push new page live. --- index.html | 481 +++++++++++++++++++++++++++++++++++++++++++--------- index2.html | 425 ---------------------------------------------- 2 files changed, 397 insertions(+), 509 deletions(-) delete mode 100644 index2.html diff --git a/index.html b/index.html index 644719ec954..f6933a47561 100644 --- a/index.html +++ b/index.html @@ -2,111 +2,424 @@ - Codestin Search App + Codestin Search App + + + + - - -

WELCOME TO RR

- - - -

rr is a lightweight tool for recording and replaying execution - of applications (trees of processes and threads).

+ +

+ + +

WELCOME TO RR

+ + +

Last updated Fri Oct 18 17:19

+ +

rr

github +
mailing + list +
travis-ci +
#research on irc.mozilla.org +

+ +

rr records nondeterministic executions and debugs them + deterministically.

rr aspires to be your primary debugging tool, replacing — + well, enhancing — gdb. You record a failure once, then + debug the recording, deterministically, as many times as you want. + Every time the same execution is replayed.

+ +

the rr debugging experience

+ +

+ Start by using rr to record your application: +

+$ rr record /your/application --args
+...
+FAIL: oh no!
+

+ The entire execution, including the failure, was saved to disk. + That recording can now be debugged. +

+$ rr replay trace_0
+GNU gdb (GDB) ...
+...
+0x4cee2050 in _start () from /lib/ld-linux.so.2
+(gdb)
+

+ Remember, you're debugging the recorded trace + deterministically; not a live, nondeterministic + execution. The replayed execution's address spaces, register + contents, syscall data etc are exactly the same in every run. +

+ Most of the common gdb commands can be used. +

+(gdb) break main
+Breakpoint 1 at 0x8048797: file ...
+(gdb) continue
+Continuing.
+
+Breakpoint 1, main (argc=1, argv=0xffffd694) ...
+...
+

+ If you need to restart the debugging session, for example + because you missed breaking on some critical execution point, no + problem. Just use gdb's run command to restart + replay. +

+(gdb) run
+The program being debugged has been started already.
+Start it from the beginning? (y or n) y
+...
+Breakpoint 1, main (argc=1, argv=0xffffd694) ...
+...
+(gdb) 
+

+ The run command started another replay run of your + recording from the beginning. But after the session restarted, + the same execution was replayed again. And all your + debugging state was preserved across the restart. +

+ This technology can make your debugging more efficient. Read on + to learn more … + +

+ +

getting started

Quick-start guide

rr can currently only record and replay x86 binaries. This - restriction will be lifted in a future version. -

To install prebuilt binaries on your Fedora system,

+    
+      fedora
+       cd /tmp
 wget http://rr-project.org/releases/rr-0.5.1-Linux-$(uname -m).rpm
 sudo rpm -i rr-0.5.1-Linux-$(uname -m).rpm
-    
-    Or on Ubuntu,
-    +      
+    
+
+    
+      ubuntu
+       cd /tmp
 wget http://rr-project.org/releases/rr-0.5.1-Linux-$(uname -m).deb
 sudo dpkg -i rr-0.5.1-Linux-$(uname -m).deb
-    
-    Once rr is installed,
-      visit https://github.com/mozilla/rr/wiki/Usage
-      to learn how to use rr.
-

+ + -

Code

The project is tracked at https://github.com/mozilla/rr.

To get the sources

-git clone git://github.com/mozilla/rr.git
-

Go - to https://github.com/mozilla/rr/wiki/Installation - for instructions on how to set up your machine.

build from source

+ Follow + these instructions. +

+ +

run rr

+ Follow the + usage instructions to set up your machine (if necessary) + and learn how to use rr. +

Communication

Technical discussion happens in the "rr-dev" mailing - list. Go - to https://mail.mozilla.org/listinfo/rr-dev - to subscribe.

The current build and test status is available - at https://travis-ci.org/mozilla/rr.

Some rr folks hang out in #research on irc.mozilla.org.

background and motivation

+ Everyone who's worked on a nontrivial application + (like Firefox) has gone + through the pain of debugging an intermittently-reproducible + bug. Since nontrivial applications + are nondeterministic, each execution is different, and + you may require 5, 10, or even 100 runs just to see the bug + manifest. +

+ It's hard to debug these bugs with traditional techniques + because single stepping, setting breakpoints, inspecting program + state, etc, is all a waste of time if the program execution + you're debugging ends up not even exhibiting the bug. Even when + you can reproduce the bug consistently, important information + such as the addresses of suspect objects is unpredictable from + run to run. Given that software developers spend a lot of time + finding and fixing bugs, nondeterminism has a major impact on + their work. +

+ And there are intermittent bugs that are so hard to reproduce + that they're literally not the worth the time to fix with + traditional techniques. However, for big projects like Firefox + with its half-billion users, a bug that only reproduces 1 out of + 10,000 test runs can still have a negative impact on users. +

+ rr solves these problems by splitting debugging into two phases: + first recording, in which the application's execution + history is saved; then deterministic debugging of the + saved trace: using gdb to control replay of the trace, as many + times as you want. +

+ The saved execution history captures all nondeterminism in the + program's execution. By replaying that trace in the right way, + rr guarantees each debugging session is entirely deterministic. + The memory layout is always the same, the addresses of objects + don't change, register values are identical, syscalls return the + same data, etc. +

+ The benefit to developers is obvious: an intermittent bug can be + recorded by a script over lunchtime, say, and then debugged at + leisure in the afternoon. Multiple cores can be used in + parallel to record failures. If you accidentally set a + breakpoint in the wrong place and miss gathering critical + information, your precious intermittent failure isn't lost. + Just fix your breakpoint and then tell gdb to run + the recording back from the beginning again. Even for easily + reproducible bugs, a repeatable, deterministic, debugging + session is a powerful tool on top of traditional debugging. +

+ And for projects like Firefox which run literally millions of + tests a day on a vast build and test infrastructure, + intermittent failures in those test runs can be recorded on the + infrastructure itself and then deterministically debugged at + some later time, offline. +

+ Tools like fuzzers and randomized fault injectors become even + more powerful when used with rr. Those tools are very good at + triggering some intermittent failure, but it's often + hard to reproduce that same failure again to debug it. + With rr, the randomized execution can simply be recorded. If + the execution failed, then the saved recording can be used to + deterministically debug the problem. +

+ So rr lowers the cost of fixing intermittent bugs. This allows + a new class of bugs to be fixed with the same amount of + engineering time and money, which in turn produces + higher-quality software for the same cost. +

+ Deterministic debugging is an old idea; many systems have + preceded rr. What makes rr different, in our opinion, are the + design goals: +

Initially focus on debugging Firefox. Firefox is + a complex application, so if rr is useful for debugging + Firefox, it is very likely to be generally useful. However, + there's nothing about rr that's specific to Firefox; we've + just spent the majority of our testing time on Firefox. rr + will work well with more programs as the project matures. +
Prioritize deployability. rr runs on a stock + linux kernel, and requires little (Ubuntu) to no (Fedora) + system configuration changes. And of course rr works on + stock, relatively modern hardware. +
Make run-time overload low. We want rr to replace + gdb in your workflow. That means you need to start getting + results with rr about as quickly as you would if you were + using gdb. +

+ The overhead of rr depends on your application's workload. On + Firefox test suites, we generally see about a 1.2x slowdown when + recording with rr. That means that if the suite takes 10 + minutes to run by itself, it will take around 12 minutes to be + recorded by rr.

Documentation

Work-in-progress presentation about the technical details of - rr: http://rr-project.org/rr.html.

Peruse https://github.com/mozilla/rr/wiki.

limitations

+ Some of rr's limitations are inherent, and some will be removed + in future releases. +

rr … +

emulates a single-core machine. So, parallel programs incur + the slowdown of running on a single core. This is an inherent + feature of the design. +
cannot record processes that share memory with processes + outside the recording tree. This is an inherent feature of the + design. rr automatically disables features such as X shared + memory for recorded processes to avoid this problem. +
currently only supports x86 32-bit processes. x86-64 will be + supported in the future, it's just a matter of porting work. +
requires a reasonably modern x86 CPU. It depends on certain + performance counter features that are not available in older + CPUs, or in ARM at all currently. +
requires knowledge of every system call executed by the + recorded processes. It already supports a wide range of + syscalls — those needed by Firefox — but support + isn't comprehensive, so running rr on your application may + uncover a syscall that needs to be implemented. +

Last updated Fri Oct 18 17:19

further reference

+ This presentation + provides an overview of the rr implementation and is meant for + potential rr developers. There are some bonus slides intended + to introduce rr to record/replay researchers. +

+ The rr wiki + contains pages that cover technical topics related to rr. +

+ More information about rr will be posted in the future.

+ + + + diff --git a/index2.html b/index2.html deleted file mode 100644 index f6933a47561..00000000000 --- a/index2.html +++ /dev/null @@ -1,425 +0,0 @@ - - - - - Codestin Search App - - - - - - - - -

- -

- - -

WELCOME TO RR

- - -

Last updated Fri Oct 18 17:19

- -

rr

- -

github -
mailing - list -
travis-ci -
#research on irc.mozilla.org -

- -

rr records nondeterministic executions and debugs them - deterministically.

rr aspires to be your primary debugging tool, replacing — - well, enhancing — gdb. You record a failure once, then - debug the recording, deterministically, as many times as you want. - Every time the same execution is replayed.

- -

the rr debugging experience

- -

- Start by using rr to record your application: -

-$ rr record /your/application --args
-...
-FAIL: oh no!
-

- The entire execution, including the failure, was saved to disk. - That recording can now be debugged. -

-$ rr replay trace_0
-GNU gdb (GDB) ...
-...
-0x4cee2050 in _start () from /lib/ld-linux.so.2
-(gdb)
-

- Remember, you're debugging the recorded trace - deterministically; not a live, nondeterministic - execution. The replayed execution's address spaces, register - contents, syscall data etc are exactly the same in every run. -

- Most of the common gdb commands can be used. -

-(gdb) break main
-Breakpoint 1 at 0x8048797: file ...
-(gdb) continue
-Continuing.
-
-Breakpoint 1, main (argc=1, argv=0xffffd694) ...
-...
-

- If you need to restart the debugging session, for example - because you missed breaking on some critical execution point, no - problem. Just use gdb's run command to restart - replay. -

-(gdb) run
-The program being debugged has been started already.
-Start it from the beginning? (y or n) y
-...
-Breakpoint 1, main (argc=1, argv=0xffffd694) ...
-...
-(gdb) 
-

- The run command started another replay run of your - recording from the beginning. But after the session restarted, - the same execution was replayed again. And all your - debugging state was preserved across the restart. -

- This technology can make your debugging more efficient. Read on - to learn more … - -

- -

getting started

fedora

-cd /tmp
-wget http://rr-project.org/releases/rr-0.5.1-Linux-$(uname -m).rpm
-sudo rpm -i rr-0.5.1-Linux-$(uname -m).rpm
-

- -

ubuntu

-cd /tmp
-wget http://rr-project.org/releases/rr-0.5.1-Linux-$(uname -m).deb
-sudo dpkg -i rr-0.5.1-Linux-$(uname -m).deb
-

- -

build from source

- Follow - these instructions. -

- -

run rr

- Follow the - usage instructions to set up your machine (if necessary) - and learn how to use rr. -

- -

background and motivation

- Everyone who's worked on a nontrivial application - (like Firefox) has gone - through the pain of debugging an intermittently-reproducible - bug. Since nontrivial applications - are nondeterministic, each execution is different, and - you may require 5, 10, or even 100 runs just to see the bug - manifest. -

- It's hard to debug these bugs with traditional techniques - because single stepping, setting breakpoints, inspecting program - state, etc, is all a waste of time if the program execution - you're debugging ends up not even exhibiting the bug. Even when - you can reproduce the bug consistently, important information - such as the addresses of suspect objects is unpredictable from - run to run. Given that software developers spend a lot of time - finding and fixing bugs, nondeterminism has a major impact on - their work. -

- And there are intermittent bugs that are so hard to reproduce - that they're literally not the worth the time to fix with - traditional techniques. However, for big projects like Firefox - with its half-billion users, a bug that only reproduces 1 out of - 10,000 test runs can still have a negative impact on users. -

- rr solves these problems by splitting debugging into two phases: - first recording, in which the application's execution - history is saved; then deterministic debugging of the - saved trace: using gdb to control replay of the trace, as many - times as you want. -

- The saved execution history captures all nondeterminism in the - program's execution. By replaying that trace in the right way, - rr guarantees each debugging session is entirely deterministic. - The memory layout is always the same, the addresses of objects - don't change, register values are identical, syscalls return the - same data, etc. -

- The benefit to developers is obvious: an intermittent bug can be - recorded by a script over lunchtime, say, and then debugged at - leisure in the afternoon. Multiple cores can be used in - parallel to record failures. If you accidentally set a - breakpoint in the wrong place and miss gathering critical - information, your precious intermittent failure isn't lost. - Just fix your breakpoint and then tell gdb to run - the recording back from the beginning again. Even for easily - reproducible bugs, a repeatable, deterministic, debugging - session is a powerful tool on top of traditional debugging. -

- And for projects like Firefox which run literally millions of - tests a day on a vast build and test infrastructure, - intermittent failures in those test runs can be recorded on the - infrastructure itself and then deterministically debugged at - some later time, offline. -

- Tools like fuzzers and randomized fault injectors become even - more powerful when used with rr. Those tools are very good at - triggering some intermittent failure, but it's often - hard to reproduce that same failure again to debug it. - With rr, the randomized execution can simply be recorded. If - the execution failed, then the saved recording can be used to - deterministically debug the problem. -

- So rr lowers the cost of fixing intermittent bugs. This allows - a new class of bugs to be fixed with the same amount of - engineering time and money, which in turn produces - higher-quality software for the same cost. -

- Deterministic debugging is an old idea; many systems have - preceded rr. What makes rr different, in our opinion, are the - design goals: -

Initially focus on debugging Firefox. Firefox is - a complex application, so if rr is useful for debugging - Firefox, it is very likely to be generally useful. However, - there's nothing about rr that's specific to Firefox; we've - just spent the majority of our testing time on Firefox. rr - will work well with more programs as the project matures. -
Prioritize deployability. rr runs on a stock - linux kernel, and requires little (Ubuntu) to no (Fedora) - system configuration changes. And of course rr works on - stock, relatively modern hardware. -
Make run-time overload low. We want rr to replace - gdb in your workflow. That means you need to start getting - results with rr about as quickly as you would if you were - using gdb. -

- The overhead of rr depends on your application's workload. On - Firefox test suites, we generally see about a 1.2x slowdown when - recording with rr. That means that if the suite takes 10 - minutes to run by itself, it will take around 12 minutes to be - recorded by rr. -

- -

limitations

- Some of rr's limitations are inherent, and some will be removed - in future releases. -

rr … -

emulates a single-core machine. So, parallel programs incur - the slowdown of running on a single core. This is an inherent - feature of the design. -
cannot record processes that share memory with processes - outside the recording tree. This is an inherent feature of the - design. rr automatically disables features such as X shared - memory for recorded processes to avoid this problem. -
currently only supports x86 32-bit processes. x86-64 will be - supported in the future, it's just a matter of porting work. -
requires a reasonably modern x86 CPU. It depends on certain - performance counter features that are not available in older - CPUs, or in ARM at all currently. -
requires knowledge of every system call executed by the - recorded processes. It already supports a wide range of - syscalls — those needed by Firefox — but support - isn't comprehensive, so running rr on your application may - uncover a syscall that needs to be implemented. -

- -

further reference

- This presentation - provides an overview of the rr implementation and is meant for - potential rr developers. There are some bonus slides intended - to introduce rr to record/replay researchers. -

- The rr wiki - contains pages that cover technical topics related to rr. -

- More information about rr will be posted in the future. -

- - - - - - - From 7b5b99f096acba27137bfe65653a48ef80db5be4 Mon Sep 17 00:00:00 2001 From: Chris Jones Date: Sat, 22 Mar 2014 16:10:52 -0700 Subject: [PATCH 025/100] Justify paragraphs, add link to Firefox helpers, and add Haswell caveat. --- index.html | 17 ++++++++++++++++- 1 file changed, 16 insertions(+), 1 deletion(-) diff --git a/index.html b/index.html index f6933a47561..1a572be34e0 100644 --- a/index.html +++ b/index.html @@ -80,6 +80,12 @@ -webkit-column-width: 30rem; column-width: 30rem; } + .columnar p { + text-align: justify; + } + .columnar ul { + text-align: justify; + } code { font-family: 'Cousine'; @@ -254,7 +260,7 @@

ubuntu

build from source

or build from source

Follow these instructions.

@@ -265,6 +271,12 @@

run rr

Follow the usage instructions to set up your machine (if necessary) and learn how to use rr. +

+ If you're using rr to debug Firefox, you may find + + these special setup instructions helpful. They cover how + to build a 32-bit Firefox on a 64-bit OS, and how to use rr to + record Firefox test suites.

@@ -389,6 +401,9 @@

limitations

syscalls — those needed by Firefox — but support isn't comprehensive, so running rr on your application may uncover a syscall that needs to be implemented. +

does not support machines that have the new Intel + Haswell-based CPUs. This is just an issue of unfortunate + timing, and support will land in the next release if possible.

From 16e6eef04b0ac4ee949fc78da4484766672cb205 Mon Sep 17 00:00:00 2001 From: Chris Jones Date: Sun, 23 Mar 2014 23:54:47 -0700 Subject: [PATCH 026/100] Remove columns, improve text example, embed video. --- index.html | 44 +++++++++++++++++++++++--------------------- 1 file changed, 23 insertions(+), 21 deletions(-) diff --git a/index.html b/index.html index 1a572be34e0..3c7f2173cc5 100644 --- a/index.html +++ b/index.html @@ -75,16 +75,10 @@ font-size: 1.5rem; } - .columnar { - -moz-column-width: 30rem; - -webkit-column-width: 30rem; - column-width: 30rem; - } - .columnar p { - text-align: justify; - } - .columnar ul { - text-align: justify; + section { + margin-left: auto; + margin-right: auto; + max-width: 50rem; } code { @@ -170,14 +164,14 @@

rr

rr records nondeterministic executions and debugs them - deterministically.

+ deterministically

rr aspires to be your primary debugging tool, replacing — well, enhancing — gdb. You record a failure once, then debug the recording, deterministically, as many times as you want. Every time the same execution is replayed.

the rr debugging experience

@@ -206,12 +200,12 @@

the rr debugging experience

Most of the common gdb commands can be used.

-(gdb) break main
-Breakpoint 1 at 0x8048797: file ...
+(gdb) break mozilla::dom::HTMLMediaElement::HTMLMediaElement
+...
 (gdb) continue
 Continuing.
-
-Breakpoint 1, main (argc=1, argv=0xffffd694) ...
+...
+Breakpoint 1, mozilla::dom::HTMLMediaElement::HTMLMediaElement (this=0x61362f70, aNodeInfo=...)
 ...

@@ -224,7 +218,7 @@

the rr debugging experience

The program being debugged has been started already. Start it from the beginning? (y or n) y ... -Breakpoint 1, main (argc=1, argv=0xffffd694) ... +Breakpoint 1, mozilla::dom::HTMLMediaElement::HTMLMediaElement (this=0x61362f70, aNodeInfo=...) ... (gdb) @@ -233,10 +227,18 @@

the rr debugging experience

recording from the beginning. But after the session restarted, the same execution was replayed again. And all your debugging state was preserved across the restart. +

+ Note that the the this pointer of the + dynamically-allocated object was the same in both replay + sessions. Memory allocations are exactly the same in each + replay, meaning you can hard-code addresses you want to watch. +

+ This video demonstrates rr's basic capabilities in a bit more + detail. +

This technology can make your debugging more efficient. Read on to learn more … -

getting started

@@ -281,7 +283,7 @@

run rr

background and motivation

Everyone who's worked on a nontrivial application (like Firefox) has gone @@ -378,7 +380,7 @@

background and motivation

limitations

Some of rr's limitations are inherent, and some will be removed in future releases. @@ -408,7 +410,7 @@

limitations

further reference

This presentation provides an overview of the rr implementation and is meant for From 998b6c43e739de695d46422b3806bcb44a2b1bc2 Mon Sep 17 00:00:00 2001 From: Chris Jones Date: Mon, 24 Mar 2014 06:19:42 -0700 Subject: [PATCH 027/100] Release 1.0.0. --- index.html | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/index.html b/index.html index 3c7f2173cc5..135b2c9ebf5 100644 --- a/index.html +++ b/index.html @@ -247,8 +247,8 @@

getting started

fedora

 cd /tmp
-wget http://rr-project.org/releases/rr-0.5.1-Linux-$(uname -m).rpm
-sudo rpm -i rr-0.5.1-Linux-$(uname -m).rpm
+wget http://rr-project.org/releases/rr-1.0.0-Linux-$(uname -m).rpm
+sudo rpm -i rr-1.0.0-Linux-$(uname -m).rpm

@@ -256,8 +256,8 @@

fedora

ubuntu

 cd /tmp
-wget http://rr-project.org/releases/rr-0.5.1-Linux-$(uname -m).deb
-sudo dpkg -i rr-0.5.1-Linux-$(uname -m).deb
+wget http://rr-project.org/releases/rr-1.0.0-Linux-$(uname -m).deb
+sudo dpkg -i rr-1.0.0-Linux-$(uname -m).deb

From 2b54e366f4739f15585e2c3d749b56bea7e2eb89 Mon Sep 17 00:00:00 2001 From: Chris Jones Date: Tue, 25 Mar 2014 17:22:40 -0500 Subject: [PATCH 028/100] Be less general about perf on FF test suites. --- index.html | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-) diff --git a/index.html b/index.html index 135b2c9ebf5..d23781d948c 100644 --- a/index.html +++ b/index.html @@ -373,10 +373,12 @@

background and motivation

The overhead of rr depends on your application's workload. On - Firefox test suites, we generally see about a 1.2x slowdown when - recording with rr. That means that if the suite takes 10 - minutes to run by itself, it will take around 12 minutes to be - recorded by rr. + Firefox test suites, rr's recording performance is quite usable. + We see slowdowns down to ≤ 1.2x. A 1.2x slowdown means that if + the suite takes 10 minutes to run by itself, it will take around + 12 minutes to be recorded by rr. However, different test suites + have different performance characteristics, so they have + different overheads as well.

limitations

From ea1a4bddebf0c956c131b91d97fd5ce812b4848d Mon Sep 17 00:00:00 2001 From: Chris Jones Date: Wed, 2 Apr 2014 22:34:45 -0700 Subject: [PATCH 029/100] Release 1.1.0. --- index.html | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/index.html b/index.html index d23781d948c..3817e50a1fe 100644 --- a/index.html +++ b/index.html @@ -247,8 +247,8 @@

getting started

fedora

 cd /tmp
-wget http://rr-project.org/releases/rr-1.0.0-Linux-$(uname -m).rpm
-sudo rpm -i rr-1.0.0-Linux-$(uname -m).rpm
+wget http://rr-project.org/releases/rr-1.1.0-Linux-$(uname -m).rpm
+sudo rpm -i rr-1.1.0-Linux-$(uname -m).rpm

@@ -256,8 +256,8 @@

fedora

ubuntu

 cd /tmp
-wget http://rr-project.org/releases/rr-1.0.0-Linux-$(uname -m).deb
-sudo dpkg -i rr-1.0.0-Linux-$(uname -m).deb
+wget http://rr-project.org/releases/rr-1.1.0-Linux-$(uname -m).deb
+sudo dpkg -i rr-1.1.0-Linux-$(uname -m).deb

From a3ae3b32be51a8873774ef0ce93e8c02fd506fa3 Mon Sep 17 00:00:00 2001 From: Chris Jones Date: Tue, 15 Apr 2014 04:05:57 -0700 Subject: [PATCH 030/100] Update video demo. --- index.html | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/index.html b/index.html index 3817e50a1fe..38fc8745836 100644 --- a/index.html +++ b/index.html @@ -235,7 +235,7 @@

the rr debugging experience

This video demonstrates rr's basic capabilities in a bit more detail. - +

This technology can make your debugging more efficient. Read on to learn more … From 749365a8c2e480e08fa3b45a0b10ba409c6e5c92 Mon Sep 17 00:00:00 2001 From: Chris Jones Date: Tue, 15 Apr 2014 13:49:42 -0700 Subject: [PATCH 031/100] Release 1.2.1. --- index.html | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/index.html b/index.html index 38fc8745836..60ce4a89fcd 100644 --- a/index.html +++ b/index.html @@ -247,8 +247,8 @@

getting started

fedora

 cd /tmp
-wget http://rr-project.org/releases/rr-1.1.0-Linux-$(uname -m).rpm
-sudo rpm -i rr-1.1.0-Linux-$(uname -m).rpm
+wget http://rr-project.org/releases/rr-1.2.1-Linux-$(uname -m).rpm
+sudo rpm -i rr-1.2.1-Linux-$(uname -m).rpm

@@ -256,8 +256,8 @@

fedora

ubuntu

 cd /tmp
-wget http://rr-project.org/releases/rr-1.1.0-Linux-$(uname -m).deb
-sudo dpkg -i rr-1.1.0-Linux-$(uname -m).deb
+wget http://rr-project.org/releases/rr-1.2.1-Linux-$(uname -m).deb
+sudo dpkg -i rr-1.2.1-Linux-$(uname -m).deb

From c433262a97efee91bc19309af2abc55462202e79 Mon Sep 17 00:00:00 2001 From: Robert O'Callahan Date: Wed, 16 Apr 2014 10:40:52 +1200 Subject: [PATCH 032/100] Update Haswell comment --- index.html | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/index.html b/index.html index 60ce4a89fcd..b404848b278 100644 --- a/index.html +++ b/index.html @@ -405,9 +405,9 @@

limitations

syscalls — those needed by Firefox — but support isn't comprehensive, so running rr on your application may uncover a syscall that needs to be implemented. -

does not support machines that have the new Intel - Haswell-based CPUs. This is just an issue of unfortunate - timing, and support will land in the next release if possible. +

does not work on some systems using Intel Haswell CPUs (but does work + on many other Haswell systems). This is being investigated; follow + the github issue.

From 626d982716410ca33b4c348f5a8564a53d70c3c3 Mon Sep 17 00:00:00 2001 From: Chris Jones Date: Tue, 22 Apr 2014 22:29:49 -0700 Subject: [PATCH 033/100] Remove command-line arg in example that's not unnecessary. --- index.html | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/index.html b/index.html index b404848b278..ac6c41ce360 100644 --- a/index.html +++ b/index.html @@ -186,7 +186,7 @@

the rr debugging experience

The entire execution, including the failure, was saved to disk. That recording can now be debugged.

-$ rr replay trace_0
+$ rr replay
 GNU gdb (GDB) ...
 ...
 0x4cee2050 in _start () from /lib/ld-linux.so.2

From 8270f747bbcced7923e62c23408536aaf6163fb5 Mon Sep 17 00:00:00 2001
From: Chris Jones 
Date: Wed, 23 Apr 2014 15:16:20 -0700
Subject: [PATCH 034/100] Update rr.html slide deck from master.

---
 res/syscall-buffer.svg |   1 +
 rr.html                | 602 +++++++++++++++++++++++------------------
 2 files changed, 338 insertions(+), 265 deletions(-)
 create mode 100644 res/syscall-buffer.svg

diff --git a/res/syscall-buffer.svg b/res/syscall-buffer.svg
new file mode 100644
index 00000000000..c8e8ec5e386
--- /dev/null
+++ b/res/syscall-buffer.svg
@@ -0,0 +1 @@
+
\ No newline at end of file
diff --git a/rr.html b/rr.html
index bc4a881b4e6..97c82160d33 100644
--- a/rr.html
+++ b/rr.html
@@ -4,14 +4,16 @@
 Codestin Search App
 
 
-  How rr works
-  mozilla research
+  How rr works
+  mozilla
+      research
 
 
 
-  Low overhead recording and replay of applications (trees of
-    processes and threads).
-  Record nondeterministic inputs, replay deterministically.
+  rr records nondeterministic executions and debugs them
+    deterministically.
+  
Practical tool; version 1.2 is latest release.  Used to debug
+    Firefox.
 
 
 
@@ -373,43 +399,47 @@ Syscall buffer

ptrace traps are expensive. Do as much work in tracee process as - possible.

ptrace traps are expensive. Better to do as much work in tracee + process as possible.

Use seccomp-bpf to selectively trap syscalls.

Syscall hooks are LD_PRELOAD'd into tracees.

Hooks record kernel return value and outparam data to - the syscall buffer.

Syscall hooks are LD_PRELOAD'd into tracees. +

Hook functions record kernel return value and outparam data to + the syscall buffer.

rr monkeypatches __kernel_vsyscall() in vdso to jump - to rr trampoline.

+ to rr trampoline.

Trampoline calls dispatcher, which calls rr hook if - available.

+ available.

Untraced syscalls are recorded to syscallbuf by tracee. Traced events recorded by the rr process "flush" the - tracee's syscallbuf.

Lib falls back on traced syscalls.

+ tracee's syscallbuf. +

Lib falls back on traced syscalls.

Simplified example of wrapper function

+  Simplified example of syscallbuf hook function
+   static int sys_close(int fd)
 {
-   long ret;
-   if (!start_buffer_syscall(SYS_close))
-     /* Fall back on traced syscall. */
-     return syscall(SYS_close, fd);
-   /* Buffer this close() call. */
-   ret = untraced_syscall1(SYS_close, fd);
-   return commit_syscall(SYS_close, ret);
+	long ret;
+	if (!start_buffer_syscall(SYS_close))
+		/* Fall back on traced syscall.
+                 * This generates a ptrace trap. */
+		return syscall(SYS_close, fd);
+
+	/* Untraced syscall. Does not generate
+         * ptrace trap.*/
+	ret = untraced_syscall1(SYS_close, fd);
+        /* Save the result to syscall buffer. */
+	return commit_syscall(SYS_close, ret);
 }

@@ -421,30 +451,44 @@

How untraced syscalls are made

 asm("_untraced_syscall:\n\t"
     "int $0x80");
-

Install filter that passes calls - from _untraced_syscall, traps to rr otherwise.

+ +

Install seccomp-bpf filter that passes calls + from _untraced_syscall, traps to rr otherwise.

resume_execution changes for PTRACE_SECCOMP events

TODO

seccomp-bpf traps generate PTRACE_EVENT_SECCOMP in + tracer process. +

rr can then PTRACE_SYSCALL the tracee into traced + syscall.

Syscallbuf hooks of may-block syscalls

TODO

Problem: buffered syscalls don't trap to rr, by design. + But may-block syscalls (f.e. waitpid()) require rr to + schedule another task.

perf events to the rescue: "descheduled" event

TODO

Set event to fire on tracee context switch. Event traps to rr. +

Buffered syscall blocks → context switch → rr + trap

Handling "desched notifications"

TODO

Generating desched events

In tracee, perf_event_open() fd for + context-switch counter +
Set event "sample period" to 1 (i.e. next context + switch) just before buffered syscall +
Disarm event just after buffered syscall. +
→ tracee sent SIGSYS if context switched during + buffered syscall +

Debugger interface

Async signals were raised externally at some execution point + during recording. +

Replay to that execution point just as for time-slice + interrupts.

(gdb) set i 10 can cause replay divergence.

So you're not allowed to do it.

Replay signal delivery by emulating

If there was a sighandler, restore recorded sigframe + and registers at sighandler entry. +

Otherwise, nothing else to do.

Light wrapper around gdb protocol

TODO

Replaying buffered syscalls

Read saved buffer from trace. +

Replay each syscall as normal, but restore outparam data from + records in the read buffer.

Replayer core passes through to ptrace requests of tracee

TODO

Debugger interface

SIGTRAP, SIGTRAP, SIGTRAP; breakpoints, int3, stepi

TODOdistinguishing causes of - traps

Common commands supported. +

c, s, si, b, bt, watch, info regs, thr, info thr ...

Future work

(gdb) call foo() can cause replay divergence. +

So you're not allowed to do it … for now. Support coming.

Roadmap for near future

TODO

Small stub translates from and to gdb remote protocol. +

Then passes debugger requests up to rr replayer. +

Checkpointing

TODO

Replayer fulfills requests using ptrace() or cached + data. +

And resumes tracee execution when asked. +

Omniscient debugging

TODO

Breakpoints, int $3, stepi, watchpoints + all raise SIGTRAP. +

$ip, breakpoint table, gdb request, + and $DR6 decode trap. +

Exploratory scheduling; targeted recording

TODO

Future work

Copy traces across machines

TODO

Checkpointing

Make a "deep fork" of tracee tree during replay. +

Run code (or whatever) in copied tree, return to original.

Record shared-memory multithreading

TODO

Omniscient + debugging (aka chroniclerr)

Use chronicle-style + instrumentation to generate execution DB. +

Query state changes in DB.

Record ptrace API

TODO

CHESS-style + execution search; targeted recording

At each scheduling decision, make a checkpoint. +

If execution reaches bad state, done. Else, resume checkpoint.

Port, port, port

TODO

Other projects

GPU drivers (NVIDIA, ATI, ...)
Windows NT kernel
Darwin kernel
~~ARM port not possible with current tech~~
Copy traces across machines +
Integrate hardware shared-memory multithreading recorder + like QuickRec +
Record ptrace API; rr record rr record… +
Handle GPU drivers (NVIDIA, ATI, ...) +
Port to Darwin kernel +
Port to Windows NT kernel +
~~ARM port not possible with current generation of chips~~

Thanks from the rr team!

rr for RnR people

Appendix: rr for RnR people

Release 0.4 available today at

rr-project.org

Release 1.2 available today at +

rr-project.org

Use cases

Run on modern, commodity hardware and software: yum - install rr; rr record…
Aspire to general tool, focus on Firefox initially
Aspire to general tool, focus on Firefox initially
Record nondeterministic test failures at scale (e.g., Firefox - build/test infra), debug offline
"Super-debugger" for local development
Search execution space to actively find bugs
"Super-debugger" for local development +
Search execution space to actively find bugs

@@ -637,40 +709,40 @@

Use cases

Design concerns

Commodity HW → only record single HW thread (for - now!)
Commodity SW → stick to higher-level userspace APIs - (e.g., ptrace, PEBS)
Record tests at scale → record perf must be "economical", - but not mission-critical
"Super-debugger" → the usual, plus queries over execution - history; pretty fast replay
Search exe space → flexible scheduling and - checkpointing

rr recorder overview

Record "applications" consisting of linux tasks
Record "applications" consisting of linux tasks
Schedule CPU slices by programming precise counter interrupt - (retired branches, RBC) for k

Time slice is special case of signal: time-slice recorded as - (0, rec-RBC), signals (signum, rec-RBC)
Record kernel-created signal stack

(0, rec-RBC)

(signum, rec-RBC)

Record kernel-created signal stack
Syscall "outparams" and rdtsc generate trace - traps; results saved to log
Plus a "faster" mode we'll cover later
Plus a "faster" mode we'll cover later

Trade-off: scheduling from userspace

While it's great to fully control scheduling ...
... we have to approximate timeslices; can be unfair
While it's great to fully control scheduling ... +
... we have to approximate timeslices; can be unfair
... interactive programs don't "feel" native; rr has its own - heuristics
... can be slower
Future work is to have the option of both
... can be slower +
Future work is to have the option of both

@@ -678,14 +750,14 @@

Trade-off: scheduling from userspace

Headache: kernel writes racing with userspace

rr doesn't (can't efficiently) record reads/writes of memory - shared outside of tracee application.
But there are still hazards with the kernel:
But there are still hazards with the kernel:
... kernel-write/task-read hazard on syscall outparam buffers → rr replaces user buffers with scratch and serializes - writes
... kernel-write/task-read on random futexes, e.g. CLONE_CHILD_CLEARTID → no good solution yet; - usleep…

@@ -693,77 +765,77 @@

Headache: kernel writes racing with userspace

rr replayer overview

Replay signal/time-slice (signum, rec-RBC) by - programming interrupt for rec-RBC
Emulate signals by restoring signal stack and regs
Emulate syscalls by restoring outparams where possible
Execute non-emulatable clone/mmap/et al. as required
Serve debugger requests (maybe covered later)

rec-RBC

Emulate signals by restoring signal stack and regs +
Emulate syscalls by restoring outparams where possible +
Execute non-emulatable clone/mmap/et al. as required +
Serve debugger requests (maybe covered later)

Replayer headache: slack in counter interrupts

Interrupt programmed for RBC = k may actually fire at - up to RBC = k + slack
(Slack empirically seen to be >= 70 branches)

RBC = k + slack

(Slack empirically seen to be >= 70 branches)
So we program interrupt for RBC = k - slack and then - advance by breakpoint+stepi
"At target" when rec-RBC == rep-RBC and [rec-regs] - == [rep-regs]
Replay target therefore technically indeterminate
Replay target therefore technically indeterminate

Recorder "fast mode": syscall buffering

ptrace traps are slow
ptrace traps are slow
Idea: avoid them when possible by buffering log data - in tracee task
Implementation: LD_PRELOAD a helper library with hooks for - common syscalls (read, write, gettimeofday, etc.)
Hook makes fast untraced syscall, saves outparams in - task-local buffer
Flush buffer at traced event (including buffer overflow)
Flush buffer at traced event (including buffer overflow)

Headache: many syscalls made internally in glibc

Those syscalls can't be wrapped by usual approach of - interposing exported symbol using LD_PRELOAD
Solution: monkeypatch __kernel_vsyscall() in - vdso.
Syscalls directly made through int $0x80 still - can't be buffered.
We hope this terrible hack evolves into kernel support.
We hope this terrible hack evolves into kernel support.

Headache: buffering syscalls that may block

read/write/… may block if buffer empty/full/…
read/write/… may block if buffer empty/full/…
But, untraced syscall from wrapper means no trap to - rr for scheduling arbitration
If another tracee is blocked too, then may deadlock
If another tracee is blocked too, then may deadlock
Solution: libc wrapper programs perf_event interrupt triggered - by next context-switch of task
If the syscall blocks, task is switched out, and rr tracer - gets interrupt (SIGIO from perf_event)

Fun debugging tricks

Save register / RBC info at all events, verify in replay
Save register / RBC info at all events, verify in replay
Generate memory checksums at selected events, verify in - replay
LLVM pass to add "execution path logging" (Bell-Larus): poor man's log of retired branches. Save to "magic fd" in recording, - verify in replay.
Hack replayer itself to efficiently log arbitrary info at - arbitrary points

From 8fef2342e1650af1cd5f372e7e7df45c33c8cc3b Mon Sep 17 00:00:00 2001 From: Chris Jones Date: Mon, 28 Apr 2014 15:45:52 -0700 Subject: [PATCH 035/100] Release 1.3.0. --- index.html | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/index.html b/index.html index ac6c41ce360..3133b17a19b 100644 --- a/index.html +++ b/index.html @@ -247,8 +247,8 @@

getting started

fedora

 cd /tmp
-wget http://rr-project.org/releases/rr-1.2.1-Linux-$(uname -m).rpm
-sudo rpm -i rr-1.2.1-Linux-$(uname -m).rpm
+wget http://rr-project.org/releases/rr-1.3.0-Linux-$(uname -m).rpm
+sudo rpm -i rr-1.3.0-Linux-$(uname -m).rpm

@@ -256,8 +256,8 @@

fedora

ubuntu

 cd /tmp
-wget http://rr-project.org/releases/rr-1.2.1-Linux-$(uname -m).deb
-sudo dpkg -i rr-1.2.1-Linux-$(uname -m).deb
+wget http://rr-project.org/releases/rr-1.3.0-Linux-$(uname -m).deb
+sudo dpkg -i rr-1.3.0-Linux-$(uname -m).deb

From 3471cf420ed63c4ceae276cfb5f1b9c88f28afc1 Mon Sep 17 00:00:00 2001 From: Robert O'Callahan Date: Mon, 16 Jun 2014 20:37:01 +1200 Subject: [PATCH 036/100] Release 1.4.0. --- index.html | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/index.html b/index.html index 3133b17a19b..c2c1289841a 100644 --- a/index.html +++ b/index.html @@ -247,8 +247,8 @@

getting started

fedora

 cd /tmp
-wget http://rr-project.org/releases/rr-1.3.0-Linux-$(uname -m).rpm
-sudo rpm -i rr-1.3.0-Linux-$(uname -m).rpm
+wget http://rr-project.org/releases/rr-1.4.0-Linux-$(uname -m).rpm
+sudo rpm -i rr-1.4.0-Linux-$(uname -m).rpm

@@ -256,8 +256,8 @@

fedora

ubuntu

 cd /tmp
-wget http://rr-project.org/releases/rr-1.3.0-Linux-$(uname -m).deb
-sudo dpkg -i rr-1.3.0-Linux-$(uname -m).deb
+wget http://rr-project.org/releases/rr-1.4.0-Linux-$(uname -m).deb
+sudo dpkg -i rr-1.4.0-Linux-$(uname -m).deb

From 8267b9da2c90ee641426eeffd9b00eafe13f7027 Mon Sep 17 00:00:00 2001 From: Robert O'Callahan Date: Mon, 16 Jun 2014 20:38:28 +1200 Subject: [PATCH 037/100] Remove Haswell warning, now that the known issues are fixed. --- index.html | 3 --- 1 file changed, 3 deletions(-) diff --git a/index.html b/index.html index c2c1289841a..d2babb37314 100644 --- a/index.html +++ b/index.html @@ -405,9 +405,6 @@

limitations

syscalls — those needed by Firefox — but support isn't comprehensive, so running rr on your application may uncover a syscall that needs to be implemented. -

does not work on some systems using Intel Haswell CPUs (but does work - on many other Haswell systems). This is being investigated; follow - the github issue.

rr is a lightweight tool for recording and replaying execution + of applications (trees of processes and threads).

Communication

Code

Installation

Documentation

rr is a lightweight tool for recording and replaying execution of applications (trees of processes and threads).

Quick-start guide

Communication

Code

Installation

Building from source

Quick-start guide

rr is a lightweight tool for recording and replaying execution of applications (trees of processes and threads).

Quick-start guide

Quick-start guide

Communication

Code

Code

Building from source

Documentation

Communication

Documentation

How rr works

Why?

Overview

Examples of nondeterministic inputs

Scheduling tasks

Hardware performance counters (HPCs)

Replaying preemption

System requirements

Basic requirements

✱Performance counters are messier in reality

Buffer syscalls; flush buffer as "super event"

No ASLR or ptrace hardening

Recorder implementation

Simplified recorder loop

Scheduling a task

Tasks changing status

Resuming task execution

Multiple running tasks are nondeterministic

Resuming a task, simplified

Generating time-slice interrupts

Trapping tracees at rdtsc

Tracee events seen + by handle_event()

handle_event() structure

Non-nestable events

Scratch buffers for blocking syscalls

POSIX signals 101

Linux signals 202

Recording signal delivery

Finishing signal handlers

Delivering unhandled signals

Syscall buffer

Simplified example of wrapper function

How untraced syscalls are made

resume_execution changes for PTRACE_SECCOMP events

Syscallbuf wrappers of blocking syscalls

perf events to the rescue: "descheduled" event

Handling "desched notifications"

Saved traces

Trace directory contents

Replayer implementation

Built around PTRACE_SYSEMU

Main loop overview

Replaying time-slice interrupts, in theory

Replaying time-slice interrupts, in practice

Delivering signals

Replaying buffered syscalls

Debugger interface

Light wrapper around gdb protocol

Replayer core passes through to ptrace requests of tracee

SIGTRAP, SIGTRAP, SIGTRAP; breakpoints, int3, stepi

Future work

Roadmap for near future

Checkpointing

Omniscient debugging

Exploratory scheduling; targeted recording

Copy traces across machines

Record shared memory mappings

Record ptrace API

^✱Performance counters are messier in reality

Tracee events seen + by `handle_event()`

^✱Performance counters are messier in reality