Thanks to visit codestin.com
Credit goes to Github.com

Skip to content

ChrisKujawa/zdb

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Zeebe Debug and Inspection tool

This repository contains a CLI tool to inspect the internal state and log of a Zeebe partition. It is a Java (21) CLI application and can be built via Maven. It was kicked off during the Camunda Summer Hackdays in 2020 and has been maintained and developed since then.

The following Zeebe versions are supported: [8.1,8.2,8.3,8.4,8.5,8.6,8.7,8.8]

Table Of Contents

What problem does it solve

When Zeebe is broken, there is currently no possibility to find out the last state of Zeebe. If there was no exporter configured, or they haven't exported for a while, it gets even worse, since it is not clear what the internal engine state is.

We built a tool called zdb - Zeebe Debugger to shed some more light in the dark. It should help you along the way during incidents and broken systems.

Usage

Note: To be on the safe side, make sure to copy Zeebe data to a separate location, not to mess with a running Zeebe process and mistakenly corrupt any data.

Docker

If you have copied data from Zeebe to your local machine, you could run the following:

 docker run -v <path>/<partitionId>/:/<partitionId>/ ghcr.io/zelldon/zdb log print -p "/<partitionId>"

Kubernetes

Important

This approach doesn't work anymore with 8.3+ and non-root containers. See related issue #332

If you have Zeebe installed in Kubernetes and want to investigate the Zeebe data, you can run zdb as an ephemeral container

kubectl debug -it -c zdb --image=ghcr.io/zelldon/zdb:latest --attach=true --target=zeebe zeebe-0 -- /bin/bash

Local CLI

Alternatively to the strategies above, you can download the fat-jar and script and run that locally.

cd /usr/bin
curl -O -L https://github.com/Zelldon/zdb/releases/latest/download/zdb
curl -O -L https://github.com/Zelldon/zdb/releases/latest/download/zdb.jar
sed -i 's/target\///' zdb
chmod u+x zdb
zdb --version

How does it solve it

Using zdb, you can inspect the internal state or the partition log.

State Inspection

Using zdb, you can inspect the internal runtime data or a snapshot. It shows information about the current state, incidents, processes, and so on from a single partition. To inspect the database, you should provide the path to the raft-partition/partitions/../runtime/ folder in a partition or one of the snapshot folders raft-partition/partitions/../snapshot/<snapshot-folder>

You can then run several commands to inspect the given state.

Inspect Zeebe Partition Status

Shows the general information of a Zeebe partition. It will show you a statistic (counts) for each existing column family in the state.

zdb state --path=<pathToDatabase>

Furthermore, the complete state can be printed as json via the list sub-command.

zdb state list --path=<pathToDatabase>

This can be more fine-tuned, and a specific column family can be given, such that only key-value pairs are printed to the console.

For example, to see all processes.

$ zdb state --path=<pathToDatabase> list -cf PROCESS_CACHE

Inspect incidents

You can inspect incidents using the following commands.

List all incidents in this partition:

zdb incident list --path=<pathToDatabase>

Returns detail to a specific incident:

zdb incident entity <IncidentKey> --path=<pathToDatabase>

Find incidents for a given processInstanceKey

zdb incident list --path=<pathToDatabase> | jq '. | map(select(.processInstanceKey==<PI_KEY>))'

Inspect Processes

You can inspect all deployed processes and get the resources of a specific process.

List all deployed processes in this partition:

zdb process list --path=<pathToDatabase>

Returns details to a specific process:

zdb process entity <ProcessKey> --path=<pathToDatabase>

List all element instances for the given process:

zdb process --path=<pathToDatabase> instances <ProcessKey>

Inspect Instances

You can inspect existing element instances and get details on their state.

Print all information to a given element instance:

zdb instance --path=<pathToDatabase> entity <elementInstanceKey>

Inspect state (generic)

There is a new (experimental) feature to inspect the state in a generic way with the state subcommand. You can either print the complete state as JSON or specify a specific column family (used in Zeebe).

Example to see all processes

$ zdb state --path=<pathToDatabase> list -cf PROCESS_CACHE

Inspect Banned Process Instances

You can check if there are any processes stuck due to banning by using the following commands.

List all banned process instances in this partition:

zdb state --path=<pathToDatabase> list -cf BANNED_INSTANCE

Log Inspection

You can also inspect the log stream using the command zdb log and its subcommands. To inspect the log, you should provide the path to a specific partition raft-partition/partitions/<partition-id>/.

Inspect Log Status

This shows the general information of a Zeebe partition log, e.g., the number of indexes, the max. Entry size, the average. Entry size, etc.

zdb log status --path=<pathToPartition>

Inspect Log

It is possible to inspect the log in more detail and search for a specific index OR position.

To search for a record position, use:

zdb log search --path=<pathToPartition> --position=<position>

It will print all related information to the record when it exists in the log.

To search for an index, use:

zdb log search --path=<pathToPartition> --index=<position>

It will print details to the specific index when it exists in the log.

Print Log

It is possible to print the complete log to standard out. This can be pretty helpful if you want to track down some records, which might have caused some issues.

To print the log:

zdb log print --path=<pathToPartition>

By default, the log is printed in JSON format. To pipe it to a file:

zdb log print --path=<pathToPartition> > output.log

The output.log file will contain all records as JSON.

Limit

The options --to and --from allow you to limit the printed log.

If you want to skip the first records or X positions, you can use zdb log print --path=<pathToPartition> --from X whereas X must be a long.

For defining a limit of the print (like until which position the log should be printed), you can use --to like this zdb log print --path=<pathToPartition> --to X whereas X must be a long.

Filter

An interesting use case is printing only certain records, for example, for specific process instances.

You can filter the printed log via: --instanceKey

 zdb log print -p <pathToPartition> --instanceKey 2251799813686738
Format

We support different formats for printing the log, like JSON, table, or dot. The JSON format is used by default. Can be set via -f or --format

Table

zdb log print --format TABLE --path=<pathToPartition>

The table format prints the complete log as a space-separated table, which can be consumed by other CSV tools.

Example:

Index Term Position SourceRecordPosition Timestamp Key RecordType ValueType Intent ProcessInstanceKey BPMNElementType 
836304301 304 6888891257 6888891180 1692869671126 2251802375814765 COMMAND PROCESS_INSTANCE ACTIVATE_ELEMENT 2251802375814765 PROCESS 
836304301 304 6888891258 6888891180 1692869671126 2251802375814770 EVENT PROCESS_INSTANCE_CREATION CREATED 2251802375814765 
836304301 304 6888891259 6888891180 1692869671126 2251802375814765 EVENT PROCESS_INSTANCE ELEMENT_ACTIVATING 2251802375814765 PROCESS 
836304301 304 6888891260 6888891180 1692869671126 2251802375814765 EVENT PROCESS_INSTANCE ELEMENT_ACTIVATED 2251802375814765 PROCESS 

Dot

zdb log print -f dot -p=<pathToPartition>

The dot format will print the complete log as a graph in dot language. This can be consumed by graphviz to generate a visual graph of the log.

Generate a dot file via: zdb log print -d -p <pathToPartition> > output.dot

Generate SVG: dot -Tsvg -o test.svg test.dot

test

Raft Inspection

Starting from 2.6.0, we support raft inspection via the zdb raft command.

Inspect Raft Status

Shows the general information of a Zeebe partition raft, e.g., current term, voted for, commit index, etc.

zdb raft status --path=<pathToPartition>

Examples

Example State Inspection

Details of a specific column family

zdb state list -p $PATH -cf EXPORTER | jq
{
  "data": [
    {
      "cf": "EXPORTER",
      "key": "00 00 00 00 00 00 00 28 00 00 00 0d 65 6c 61 73 74 69 63 73 65 61 72 63 68",
      "value": {
        "exporterPosition": 619675,
        "exporterMetadata": "eyJyZWNvcmRDb3VudGVyc0J5VmFsdWVUeXBlIjp7IkRFUExPWU1FTlQiOjE0LCJQUk9DRVNTX0lOU1RBTkNFIjo1ODcsIklOQ0lERU5UIjo1LCJNRVNTQUdFIjo3OSwiTUVTU0FHRV9TVUJTQ1JJUFRJT04iOjM5LCJQUk9DRVNTX01FU1NBR0VfU1VCU0NSSVBUSU9OIjoxNDksIk1FU1NBR0VfU1RBUlRfRVZFTlRfU1VCU0NSSVBUSU9OIjozNywiVkFSSUFCTEUiOjE5NywiUFJPQ0VTU19JTlNUQU5DRV9DUkVBVElPTiI6MSwiUFJPQ0VTUyI6MTMsIkNPTU1BTkRfRElTVFJJQlVUSU9OIjo4NH19"
      }
    },
    {
      "cf": "EXPORTER",
      "key": "00 00 00 00 00 00 00 28 00 00 00 0f 4d 65 74 72 69 63 73 45 78 70 6f 72 74 65 72",
      "value": {
        "exporterPosition": 619676,
        "exporterMetadata": ""
      }
    }
  ]
}

Retrieve a process model

$ zdb state --path=<pathToDatabase> list -cf PROCESS_CACHE | jq --raw-output '.data[0].value.resource' | base64 -d > model.bpmn

Example RAFT metadata

$ zdb raft status -p /raft-partition/partitions/2/ | jq
{
  "meta": {
    "term": 2,
    "lastFlushedIndex": 65322,
    "commitIndex": 65320,
    "votedFor": "2"
  },
  "config": {
    "index": 0,
    "term": 0,
    "time": 1760112952350,
    "force": false,
    "requiresJointConsensus": false,
    "newMembers": [
      {
        "id": "2",
        "hash": -303927213,
        "type": "ACTIVE",
        "lastUpdated": "2025-10-10T16:15:52.350Z"
      },
      {
        "id": "1",
        "hash": -126235597,
        "type": "ACTIVE",
        "lastUpdated": "2025-10-10T16:15:52.350Z"
      },
      {
        "id": "0",
        "hash": 384918240,
        "type": "ACTIVE",
        "lastUpdated": "2025-10-10T16:15:52.350Z"
      }
    ],
    "oldMembers": []
  }
}

Similar to other commands we support different formats like json (default) and table.

$ zdb raft status -p data.bk/raft-partition/partitions/2/ -f table
--------------------------------------------------------------
Raft Status for partition '2':
--------------------------------------------------------------
Meta Store:
    Term:                    2
    Last Flushed Index:      65322
    Commit Index:            65320
    Voted For:               2
--------------------------------------------------------------
Configuration:
    Index:                   0
    Term:                    0
    Time:                    1760112952350
    Force:                   false
    Requires Join Consensus: false
    New Members:             [
		Id: 2, Type: ACTIVE, Hash: -303927213, Updated: 2025-10-10T16:15:52.350Z
		Id: 1, Type: ACTIVE, Hash: -126235597, Updated: 2025-10-10T16:15:52.350Z
		Id: 0, Type: ACTIVE, Hash: 384918240, Updated: 2025-10-10T16:15:52.350Z
    ]
    Old Members:             []
--------------------------------------------------------------

Autocompletion

zdb comes with autocompletion. Just print it to a file:

zdb generate-completion >> ~/.autocompletions/zdb

and source that file in your shell profile (i.e. .bash_rc, .zsh_rc, .bash_profile, etc.):

source <(cat $HOME/.autocompletions/zdb)

About

Zeebe debug and inspection tool

Topics

Resources

Contributing

Stars

Watchers

Forks

Packages

 
 
 

Contributors 13