mzprof

A dataflow profiler for Materialize.

mzprof is a CLI tool that connects to any Materialize environment and subscribes to introspection relations to collect operator-level metrics about running dataflows. It produces profiles in pprof format, which can be analyzed using any tool supporting that format.

Installation

To install the latest published version of mzprof run:

cargo install --locked mzprof

Alternatively, clone this repository and use cargo run and/or cargo build as usual.

Usage

To collect a profile you need to supply mzprof with the postgres:// URL of the target Materialize environment, as well as a cluster and replica name:

mzprof --sql-url postgres://jan@localhost:6875/materialize --cluster compute --replica r1 --profile time

This will collect a time profile over all dataflows on the target replica, with their elapsed times since they were installed.

You can instead collect a profile of live elapsed times by specifying a listen duration in seconds:

mzprof [...] --profile time --duration 10

To collect a heap size profile, specify the size profile type instead:

mzprof [...] --profile size

Profile Types

mzprof currently supports four profile types:

• time: operator elapsed times
• size: operator heap sizes
• capacity: operator heap capacities
• records: operator maintained record counts

It is possible to collect profiles for multiple types in a single mzprof invocation, by passing more than one profile type in --profile.

Note that different profile types capture different classes of metrics. Specifically, time is a rate metric (measures events per time), whereas size, capacity, and records are level metrics (measure accumulated values). Collecting a profile over a time period using --duration only makes sense for rate metrics.

Viewing Profiles

A convenient way to view profiles created by mzprof is uploading them to https://pprof.me. It renders them as flame graphs with a bunch of knobs to customize the presentation:

• To show time spent per worker, select the "worker" label in the Group by dropdown.
• To zoom in on a single dataflow, use the Filter menu to add a stack filter for the dataflow name.
• To group by operator ID instead of operator name, select "Address" under Preferences > Levels.