+1
We've been thinking about the tiered storage feature, and exploring OpenDAL integration. With that, we should be able to point the incoming messages to external destination (object storage, database, file storage, o11y engines, OLAP DBs etc).
Cc: @spetz @hubcio @numinnex

Besides the tiered-storage, we have the plugin/integration feature planned for the future. We haven't decided yet how the interface would look like, and how the "connectors" could be implemented (rust, wasm, or maybe via some network protocol like grpc).

This is certainly on our roadmap, but if I were to estimate the date, probably no sooner than in a few months.

@thyseus it would be awesome if you could propose a design for implementing this feature. Do you want to put together a proposal?

Actually, my rust skills are not yet there; in the web-ui i would image it like:

and in the cli like

Commands:
  stream           stream operations [aliases: s]
  topic            topic operations [aliases: t]
  partition        partition operations [aliases: p]
  hooks         hook operations [aliases: h]

hook operations

Usage: iggy hook <COMMAND>

Commands:
  create  Create a hook with given name
  delete  Delete hook with given id
  get     Get details for given hook
  list    List all hooks
  help    Print this message or the help of the given subcommand(s)

Create a hook topic with given name on given topic

Topic ID can be specified as a name or ID

Examples
iggy hook create my-first-hook my-topic "cd /path/to/my/other/project; php artisan execute:command {payload}"

Usage: iggy topic hook [OPTIONS] <HOOK_NAME> <TOPIC_NAME> <COMMAND_TO_EXECUTE>

Actually my rust skills had been barely enough to get a mini-consumer running that is just passing over the payload like this:

  let output = Command::new("php")
        .current_dir("/home/me/other-project/")
        .args([
            "artisan",
            "iggy:process",
            payload
        ])
        .output()
        .expect("failed to execute process");

so i am totally fine; with a hook one would not need any consumer at all to achieve something similar.

I'd recommend taking a look at Apache Camel's Connector design. We can allow people to create "connectors" and configure them with the settings. Every Connector can have a config file or we can have single "Connectors" yaml file which will be loaded into a map when Iggy server is started.

So we need to come up with a mechanism of calling these "Connectors" at important places like at the entry point, at the time of writing/persisting or reading etc.

Such Connectors can be configured at Topic level or even at the message/batch level. topic level makes the most sense but there could be use cases where we want to let the Clients decide too

Introduction

I'm going to start out by separating by usecase, of which I see a few:

• "Trigger" similar to what you'd find in most SQL databases (for example Postgres' triggers )
• A User-defined function (UDF) or User Defined (UDAF) similar to what you'd find in query engines such as Datafusion.
• An embedded program - this is where we'd have something that is embeddable into Iggy and invoked as part of it's functionality.

Usecases

Trigger

The primary use case would be @thyseus' initial idea. Some questions to answer:

• Could we potentially attach a call out to a separate user defined function based on arbitrary events inside of Iggy?
• What kinds of permissions/capabilities would these triggers have? ie run executables, make network calls (if so, at what OSI layer?)
• What events would we support?
• What will be the calling convention of some of these triggers?
• How will errors in these triggers be handled?
• Will it be important for these triggers to be sandboxed?

UDF/UDAF implementation

I've always wanted something like this in a streaming service - the use case primarily being compacted topics. For compacted topics in Kafka and Pulsar, only the most recent value for the specified key is retained and the rest are effectively deleted. If you could embed a UDAF function similar to Datafusion's implementation, one could potentially have "derivative" topics. This would also likely reduce the need for a lot of stream processing engines which are external and require quite a bit of development.

Embedded Program

This usecase surrounds itself with possibly changing core behaviour inside of Iggy that is difficult to configure with a configuration file format such as yaml. For instance, a developer can implement a script or something that will adjust the way segments get written or possibly set up custom logic for writing partitions to a separate durable store.

Embedded Scripting considerations

All of these usecases can be solved with an implementation of some sort of embedded programming language based on events. I will detail some of the idea I have:

Webassembly

Webassembly is really nice for sandboxing specifically. However, this doesn't seem to be a really important feature for Iggy as users will likely trust their own code. Webassembly is also not a target for many higher level languages without shipping the interpreter/runtime with it. Wasm also has a reputation for moving really slow with it's specs. As far as I know, this is what ScyllaDB used for their UDF implementation

Shared Library

This is probably the most straightforward one and allows the most flexibility when it comes to the API. However, there are only a handful of languages that can be compiled to a smallish shared library, so effectively we are limited to some languages that allow that type of interfacing (Rust, C, C++ and Swift come to mind). This is apparently the way Arroyo went with their implementation of UDF's

Specific language runtime

We could possibly look at embedding a specific language runtime such as RustPython, rusty-v8 or a lua engine. I don't really favour this approach as usually their is a cost to get the data into the VM and get it executed. This also brings a level of overhead as we need to consider things like language version support and all of that.

Events

We also need to consider which events might be interesting for developers to be able to "hook" into. This could be:

• Compacting a topic
• Consuming a message.
• Producing a message.
• Deleting data based on retention period.
• Dropping the to the next tier level of storage.

I think that eBPF might give us good insights as they have a really nice framework for

1. Hooking into different events in Linux and
2. compiling a library for use within the kernel.

I've just opened another discussion #1670 as the more generic continuation of this topic.