Opsqueue: A lightweight batch processing queue for heavy loads

Simple 'getting started' instructions can be found in the repository README.

The Rust codebase defines both the 'server', which makes up the bulk of the Opsqueue binary itself, and the 'client', which is the common part of functionality that clients written in different other programming languages can all use.

Many datatypes are shared between this server and client, and therefore their code lives together in the same crate. Instead, we use feature-flags (client-logic and server-logic) to decide what concrete parts to include when building. The set of dependencies is based on these same feature-flags. Most interestingly, in the test suite we enable both feature-flags so we're able to do a bunch of round-trip testing immediately in Rust code.

Module setup

• The basic logic is divided in the producer and consumer modules. These both have their own db submodule.
• Common functionality and datatypes exists in the common module
• Common database helpers live in the db module.
• Reading/writing to object stores like GCS or S3 is abstracted in the object_store module.
• Finally, extra modules to have a single source of truth for configuration of the queue, and to nicely do tracing and expose metrics exist.

Opsqueue is a lightweight batch processing queue system.

This crate describes both the opsqueue queue binary, as well as the core of the various client libraries that can call it.

Installing the Opsqueue binary

cargo install opsqueue

Using a client library

Currently, we have high-level Python bindings available, c.f. https://pypi.org/project/opsqueue/.

Besides this, the Rust client itself can be used directly, by including opsqueue as a library in your project's Cargo.toml and enabling the client-logic feature-flag:

opsqueue = {version = "0.30.0", default-features = false, features = ["client-logic"]}

More info

Find full usage instructions and details at the main repository readme: https://github.com/channable/opsqueue