Scope

api

Desired Behavior

In order to improve FlowG's interoperability, we want to be able to match the HTTP API of third-party services, so that you can use their client libraries to send logs to FlowG without changing your code.

For example, to ingest a document into ElasticSearch, the client is performing the following HTTP request:

POST /my-index/_doc
{
  "@timestamp": "...",
  "message": "..."
}

This is equivalent to the following FlowG request:

POST /api/v1/pipelines/my-index/logs/struct
{
  "records": [
    {
      "@timestamp": "...",
      "message": "..."
    }
  ]
}

If we introduce a new collection of endpoints under /api/v1/middlewares, we can mimic a subset of third-party APIs, and translate those calls to FlowG calls automatically, for example with an hypothetical ElasticSearch middleware:

POST /api/v1/middlewares/elasticsearch/my-index/_doc
{
  "@timestamp": "...",
  "message": "..."
}

Then, you can point your ElasticSearch client to flowg:

cfg := elasticsearch.Config{
	Addresses:    []string{"http://localhost:5080/api/v1/middlewares/elasticsearch/"},
	ServiceToken: "flowg pat or jwt token",
}
client, err := elasticsearch.NewClient(cfg)

Similarly, middlewares for services like Datadog, Splunk, etc... could be implemented in the future.

NB: Obviously, 100% feature parity with the API of such services would be impossible, as many features simply don't exist in FlowG. But for log ingestion specifically, it could be a great step towards interoperability and adoption.