Wombat is a high performance and resilient stream processor, able to connect various sources and sinks in a range of brokering patterns and perform hydration, enrichments, transformations and filters on payloads.

It comes with a powerful mapping language, is easy to deploy and monitor, and ready to drop into your pipeline either as a static binary or docker image, making it cloud native as heck.

Wombat is declarative, with stream pipelines defined in as few as a single config file, allowing you to specify connectors and a list of processing stages:

input:
  gcp_pubsub:
    project: foo
    subscription: bar

pipeline:
  processors:
    - mapping: |
        root.message = this
        root.meta.link_count = this.links.length()
        root.user.age = this.user.age.number()

output:
  redis_streams:
    url: tcp://TODO:6379
    stream: baz
    max_in_flight: 20

First of all, this project is not a full fork. We still use the MIT licensed RedPanda Benthos project as our base. We even still use a large part of the Apache2 Licensed RedPanda Connect project. We did however fork some of the components RedPanda made proprietary and added some of our own.

The idea behind this move is to allow anyone to experience wombat without having to rely on a commercial entity behind it. This is a community project and always will be.

Take a look at the documentation for more information on how to get started

We're working on the release process, but you can either compile from source or pull the docker image:

docker pull ghcr.io/wombatwisdom/wombat

For more information check out the getting started guide.

wombat -c ./config.yaml

Or, with docker:

# Using a config file
docker run --rm -v /path/to/your/config.yaml:/wombat.yaml ghcr.io/wombatwisdom/wombat

# Using a series of -s flags
docker run --rm -p 4195:4195 ghcr.io/wombatwisdom/wombat \
  -s "input.type=http_server" \
  -s "output.type=kafka" \
  -s "output.kafka.addresses=kafka-server:9092" \
  -s "output.kafka.topic=wombat_topic"

Wombat includes optional IBM MQ support through the ibm_mq input and output components. Since IBM MQ requires proprietary client libraries, this feature is disabled by default and uses stub implementations for broader compatibility.

To build and test with actual IBM MQ support, you need the IBM MQ client libraries:

1. Download the IBM MQ redistributable client:

   • Visit IBM Fix Central and search for "IBM MQ redistributable client"
   • Or download directly: 9.4.1.0-IBM-MQC-Redist-LinuxX64.tar.gz

2. Extract to a local directory:

   mkdir -p ~/mqclient
   tar -xzf 9.4.1.0-IBM-MQC-Redist-LinuxX64.tar.gz -C ~/mqclient/

3. Set environment variables for building:

   export MQ_HOME="$HOME/where-you-stored-the-mq-lib"
   export CGO_ENABLED=1
   export CGO_CFLAGS="-I${MQ_HOME}/inc"
   export CGO_LDFLAGS="-L${MQ_HOME}/lib64"

Without IBM MQ client (stub implementation):

task build

This is the default build mode and doesn't require any IBM MQ libraries.

With IBM MQ client support:

# Set environment variables (see setup instructions above)
export MQ_HOME="$HOME/mqclient"
export CGO_ENABLED=1
export CGO_CFLAGS="-I${MQ_HOME}/inc"
export CGO_LDFLAGS="-L${MQ_HOME}/lib64"

# Build with mqclient tag
task build:mq

# Set environment variables
export MQ_HOME="$HOME/mqclient"
export CGO_ENABLED=1
export CGO_CFLAGS="-I${MQ_HOME}/inc"
export CGO_LDFLAGS="-L${MQ_HOME}/lib64"

# Run all tests including IBM MQ
task test:all

If you download a prebuilt wombat-mqclient binary from our releases, you'll need to:

1. Install the IBM MQ client libraries on your system:

   # Download the redistributable client
   wget https://public.dhe.ibm.com/ibmdl/export/pub/software/websphere/messaging/mqdev/redist/9.4.1.0-IBM-MQC-Redist-LinuxX64.tar.gz
   
   # Extract to a permanent location (e.g., /opt/mqm)
   sudo mkdir -p /opt/mqm
   sudo tar -xzf 9.4.1.0-IBM-MQC-Redist-LinuxX64.tar.gz -C /opt/mqm

2. Set LD_LIBRARY_PATH when running:

   export LD_LIBRARY_PATH=/opt/mqm/lib64
   ./wombat -c config.yaml

   Or add to your shell profile for permanent setup:

   echo 'export LD_LIBRARY_PATH=/opt/mqm/lib64:$LD_LIBRARY_PATH' >> ~/.bashrc
   source ~/.bashrc

I can't in all good faith take credit for the enormous amount of work that went into this project. Most of that is on Ash and the rest of the community behind the old Benthos project. I'm just a guy who forked it and made it worse.

For those of you who miss Ash too much, here are some links to some of the old content still available:

• Delivery guarantees can be a dodgy subject

Contributions are welcome, please read the guidelines, and watch your back.