We’ve analysed the runtime performance of Servo and Chromium in the cases of loading four websites, as of the versions below:

• “servo.1” — f8933a57353aeca14a6cbc60b3cb0cf98cab6c5d (2024-10-06)
• “servo.2” — 7015e0fb5f1a63ed13a5b07b056036e6e7adbc16 (2024-10-22)
• “servo.3” — bac1101163a436096fe0b71bed79094658bba3b2 (2024-10-30)
• “chromium” — Google Chrome 128.0.6613.113 (Official Build), as shipped with NixOS 24.11.20240905.8ce7f9f

We found the newer versions of Servo (servo.2 and servo.3) to be significantly faster than the older version (servo.1), thanks to several performance improvements we landed last month. For example, taking the average change across the pages under test with a two-CPU configuration, First Contentful Paint times are 23% lower, and overall rendering times are 38% lower.

servo.3 continues to outperform Chromium in First Contentful Paint for three of the four pages under test. We still lag behind Chromium (chromium) in FCP for www.baidu.com, but are now only 81% slower, down from 97% slower.

servo.3 performs less favourably in overall rendering time, but improves over servo.1 and servo.2. We now outperform Chromium (chromium) in all CPU configurations for one of the four pages under test (zh.wikipedia.org), and outperform it for some CPU configurations in two others (servo.org and www.baidu.com).

All of the code and data used to write this report is in this file including:

• perf-analysis-tools — our tooling
  • perf-analysis-tools/run-all-benchmarks.sh — how we ran the benchmarks
  • convert-pftrace-to-json.sh — converts Chrome tracing files to JSON
  • analyse-cpu-results.sh — computes summaries for each sample within a CPU configuration
  • generate-report-tables.sh — collates summaries into tables for this report
• */summary.txt — summary results (generated by our tooling)
• Data for each engine under test
  • *.servo.1 — data for the first version of Servo (servo.1)
  • *.servo.2 — data for the second version of Servo (servo.2)
  • *.servo.3 — data for the third version of Servo (servo.3)
  • *.chromium — data for Chromium (chromium)
  • */*.html — Servo HTML traces
  • */*.pftrace — Chrome tracing files
• Data for each page under test
  • servo.org.* — data for https://servo.org/
  • www.amazon.com.* — data for https://www.amazon.com/dp/B07S9XZYN2
  • zh.wikipedia.org.* — data for https://zh.wikipedia.org/wiki/Servo
  • www.baidu.com.* — data for https://www.baidu.com/

This report further automates the collection and analysis of data over our previous report, since there are now 64 configurations (four engines × four pages × four CPU configurations) that are each run 30 times. Changes to our procedures include:

• Merging the mitmproxy recordings, so the replay only needs to be started once
• Fixing intermittent xdotool failures caused by races in benchmark-servo.sh
• Parameterising our CPU isolation script over Linux cpu ids
• Writing scripts to run benchmarks across the 64 configurations
• Making those scripts set up the correct CPU isolation automatically
• Making measurement resumable at sample boundaries (every 30 runs)

Note that the times in this report are not directly comparable with those in the previous report, due to some minor changes in measurement procedure, such as increasing the time we run each browser from 5 seconds to 10 seconds. The results for servo.1 are the best representation of how Servo performed at the time of the previous report.

That said, due to time constraints, our methodology is otherwise unchanged from the previous report. Notably, there are several unresolved limitations in our collection and interpretation of data, which we’ll discuss below.

The test scenarios in this report only cover cold page loads, without any caching or further user interaction.

Times are not cut off after the page is “fully loaded”, which may distort results due to layout operations after page load.

In § User-facing paint metrics, we don’t have Largest Contentful Paint (LCP) data, since this metric is not yet implemented in Servo.

In § Raw events, much of the LayoutPerform time in Servo is also recorded in ParseHTML, yielding unreasonable ParseHTML times, and ScriptEvaluate times in Servo are unreasonably small compared to similar events in Chromium.

In § Rendering phases model, the Rasterise phase for Chromium is incomplete, since it only includes Layerize events, and Parse and Script phase times are heavily distorted between Servo and Chromium.

Our test environment is as follows:

• AMD 7950X (amd64)
• NixOS 24.11.20240905.8ce7f9f running X11
• Linux 6.11.0-rc6, linuxPackages_testing from NixOS (as above)
• Servo is built with ./mach build --profile production-stripped
• Chromium is google-chrome from NixOS (as above)
  • NIXPKGS_ALLOW_UNFREE=1 nix run --impure github:NixOS/nixpkgs/8ce7f9f78bdbe659a8d7c1fe376b89b3a43e4cdc\#google-chrome
• servo/perf-analysis-tools as of 01d5b4152dae0

The workloads are run in a shell created as follows:

$ newgrp mitmproxy
$ nix-shell ~/path/to/servo/shell.nix --run zsh

CPU isolation is handled in § Measurement procedure.

To ensure that the windows are kept offscreen, we use the following i3 config:

$ cat ~/.config/i3/config
for_window [instance="^servo$" class="^servo$"] floating enable
for_window [instance="^google-chrome [(]" class="^Google-chrome$"] floating enable
assign [instance="^servo$" class="^servo$"] 7
assign [instance="^google-chrome [(]" class="^Google-chrome$"] 7

To ensure that the benchmarking scripts can set up the CPU isolation automatically, we use the following sudoers(5) config:

$ cat /etc/sudoers
%wheel  ALL=(ALL:ALL)    NOPASSWD: /path/to/perf-analysis-tools/isolate-cpu-for-shell.sh

We run the benchmarks as follows:

$ cd perf-analysis-tools

$ cat run-all-benchmarks.sh
#!/usr/bin/env zsh
set -euo pipefail -o bsdecho
sudo ./isolate-cpu-for-shell.sh $$ 14 15
./run-benchmarks-for-cpu.sh ../report3/2cpu
sudo ./isolate-cpu-for-shell.sh $$ {12..15}
./run-benchmarks-for-cpu.sh ../report3/4cpu
sudo ./isolate-cpu-for-shell.sh $$ {10..15}
./run-benchmarks-for-cpu.sh ../report3/6cpu
sudo ./isolate-cpu-for-shell.sh $$ {8..15}
./run-benchmarks-for-cpu.sh ../report3/8cpu

$ ./run-all-benchmarks.sh

This in turn sets up the CPU isolation for each CPU configuration and runs the benchmarks:

$ cat run-benchmarks-for-cpu.sh
#!/usr/bin/env zsh
set -euo pipefail -o bsdecho
sample_dir=$1
./run-benchmarks-for-site.sh servo.org https://servo.org/ "$sample_dir"
./run-benchmarks-for-site.sh www.amazon.com https://www.amazon.com/dp/B07S9XZYN2 "$sample_dir"
./run-benchmarks-for-site.sh zh.wikipedia.org https://zh.wikipedia.org/wiki/Servo "$sample_dir"
./run-benchmarks-for-site.sh www.baidu.com https://www.baidu.com/ "$sample_dir"

This in turn runs the benchmarks for each site:

#!/usr/bin/env zsh
set -euo pipefail -o bsdecho
key=$1
url=$2
sample_dir=$3
./benchmark-servo.sh ~/code/servo/servo.20241006.f8933a57353aeca14a6cbc60b3cb0cf98cab6c5d/servo "$url" 30 "$sample_dir/$key.servo.1"
./benchmark-servo.sh ~/code/servo/servo.20241022.7015e0fb5f1a63ed13a5b07b056036e6e7adbc16/servo "$url" 30 "$sample_dir/$key.servo.2"
./benchmark-servo.sh ~/code/servo/servo.20241030.bac1101163a436096fe0b71bed79094658bba3b2/servo "$url" 30 "$sample_dir/$key.servo.3"
./benchmark-chromium.sh google-chrome-stable "$url" 30 "$sample_dir/$key.chromium"

This in turn runs the benchmarks for each engine.

We first convert the Chromium traces from Perfetto format to JSON format:

$ ./convert-pftrace-to-json.sh /path/to/traceconv

Then we compute summaries for each sample within a CPU configuration:

$ for i in ?cpu; do ./analyse-cpu-results.sh $i; done

Then we collate those summaries and generate the tables:

$ ./generate-report-tables.sh

The paint metrics (FP and FCP) are standard web platform concepts that should be comparable between Chromium and Servo. In Chromium, their duration is measured from the markAsMainFrame event. In Servo, they are measured from the start of the first event associated with the page URL, which is always a ScriptParseHTML event.

The results are as follows.

For a detailed discussion of these events, see this section in our previous report.

The results are as follows, but the Servo and Chromium results are not comparable.

For a detailed discussion of this model, see this section in our previous report.

The results are as follows, but we do not consider them very meaningful.

To make the most of the data we have, we can take the union of all of the events above and call that the Renderer. Whenever Servo or Chromium is recording time in Renderer, they are busy with some phase of the rendering process for the page under test (or a generic task, like Compositing in Servo).

The results are as follows.