Pergolator

Pergolator is inspired by the capabilities of Elasticsearch's percolator, but is designed to work with Go structs. It allows you to define a struct and then create a percolator that can match queries against instances of that struct.

Pergolator can be used either as a CLI or as a //go:generate directive. It will generate code that can be used to create a percolator for a specific struct. To parse your queries, you can use our defaultparser, but you can also create your own parser, see tree for more information.

It has been designed to be as fast as possible, which led to the current implementation and APIs (see Benchmarks for more information).

Important

To my knowledge, this lib is not used in production anywhere yet, so please use it at your own risk. You should review the code generated.

Examples

Basic

You can find many examples in the tests directory, but here is a simple one:

//go:generate go run github.com/antoninferrand/pergolator <path to this file>.Log
package main

type Log struct {
    ID      int
    Level   string
    Message string
}

(use go generate to generate the percolator)

You will then be able to do:

//go:generate go run github.com/antoninferrand/pergolator <path to this package>.Log
package main

import (
    "github.com/antoninferrand/pergolator/tree/defaultparser"
)

type Log struct {
    ID      int
    Level   string
    Message string
}

func main(){ 
    // Hardcoded query but you can get it from anywhere (DB, user input, etc.)
    query := "ID:1 AND Level:INFO"
    
    // We assume here that a `StructPercolator` has been generated by the `//go:generate` directive
    percolator, err := NewLogPercolator(defaultparser.Parse, query)
    if err != nil {
        panic(err)
    }
	
    // Simulate a stream of logs, they can come from anywhere (Kafka, gRPC, etc.)
    logs := make(chan Log, 100)
	
    for log := range logs {
        if !percolator.Percolate(log) {
            continue 
        }
        // Do something
    }
}

Installation

Package (recommended)

go get github.com/antoninferrand/pergolator

Then you can use with:

//go:generate github.com/antoninferrand/pergolator <path to your package>.<struct>

CLI

go install github.com/antoninferrand/pergolator

Then you can use with either:

• (recommended) //go:generate pergolator <path to your package>.<struct> in your code
• pergolator --dest-package <package> --path <path to the directory in which the percolator will be written> --prefix <prefix for the file containing the code of the percolator> <path to your package>.<struct>

Mapping between your logical layer (query) and your physical layer (struct)

Parsers

You can use your own parser to parse the query. See tree/README.md for more information.

Modifiers

You can use modifiers to modify the AST of the query after it has been parsed. This allows you to easily rename, add, remove fields to your query. Some helpers have already been created, see modifiers/modifiers.go.

For example, you can use the FormatKeysToCamelCase to allow your users to use snake case in their queries, and remap them to camel case in your struct.

type Log struct {
    ID      int
    Level   string
    Message string
}

func main() {
	query := "id:1 AND level:INFO"
	
	percolator := NewLogPercolator(defaultparser.Parse, query, modifiers.FormatKeysToCamelCase)
}

Tags and descriptors

Tags

By adding either pergolator or json tags on your struct you can modify how the percolator code will be generated. (You can use pergolator in case you don't want to add or modify json tags).

Their first value will be used to rename the field. If the name is -, the field will be ignored (it cannot be queried). Any node in the AST that use this field will return false.

For example:

type Log struct {
    ID      int    `json:"id"`
    Level   string `json:"level"`
    Message string `pergolator:"message"`
    Ignored string `json:"-"`
}

Can be queried with:

id:2 AND level:INFO AND message:hello

Some special values are available:

• Adding !flatten will flatten a field into its parents. Example:

type StructA struct {
   Field StructB `pergolator:"field,!flatten"`
}

type StructB struct {
    Field2 string
}

Can be queried with Field2:hello instead of Field.Field2:hello. For now, it is only available for nested struct, soon for maps and slices.

Descriptors

In some cases you cannot add tags to your structs, for example when you are using a third party library. In these cases, you can use descriptors instead of tags.

You have to:

• Create a json file containing the mapping between your struct and the go tags.
• Use the --descriptor flag to pass the file to the code generator.

Example: descriptor.json

Benchmarks

In the following benchmark, each operation consists of matching 1000 logs against a query of 3 fields with AND between them. 1/3 of the logs match the query.

goos: linux
goarch: amd64
pkg: github.com/antoninferrand/pergolator/tests/bench
cpu: Intel(R) Core(TM) i7-8565U CPU @ 1.80GHz
BenchmarkMatchAll
BenchmarkMatchAll-8   	   67246	     17045 ns/op	       0 B/op	       0 allocs/op

So on average it takes 17ns per evaluation of a query.

On average (based on multiple Go benchmarks), with this hardware and 1vCPU (-core 1), we have seen the following durations:

• 3ns per int/float/bool comparison.
• 3ns per string comparison if their length are different.
• 6ns per string comparison if their length are the same. If they are large this value can be higher.
• 2ns per AND / OR / NOT operator.
• 2ns per level of depth (<struct>.<struct>.<struct>.Field = 2 additional levels of depth).

To confirm with production workload, but it looks fast.

Additional benchmarks are available in tests/benchmark.

Contributing

Please feel free to submit issues, fork the repository and send pull requests!

When submitting an issue, we ask that you to include a complete test function that demonstrates the issue. When submitting a PR, we ask that you to include a benchmark function that demonstrates the performance (of the new feature, or the performance improvement).

In particular, we are looking for contributions to the following areas:

• Support additional special types (like what has been done for time.Time)
• Support additional query languages
• Performance improvements
  • Recursive structs (structs that contain other structs)
  • Overhead of calling 1 function by operator and value

Limitations

Limitation	Reason	Status
Time to evaluate N queries grows in O(N)	We don't provide a way to match N queries at once.	Investigation in progress
Only exported fields can be used in queries.	This is a design choice that could be revisited in the future. It makes sense because unexported field are not supposed to be accessible from the outside, in particular from outside the system.	No work planned for now
Performance degradation when querying <struc>.<struct>. ... .<struct>.Field.	Currently for each new struct in depth, we call the percolator of that struct with the rest of the AST. This is not optimized as it means calling an additional function for each level of depth. However not doing this is a little bit harder to implement.	No work planned for now
Some types are not supported, in particular combination of map and slices (example map[string]map[string][]string).	Currently each "common" types (i.e. types that are not structs with their own percolator) has to be implemented by hand. So with combination of map and slice, it is complex to be exhaustive (they represent a few pathological cases).	No work planned for now
It is possible to have some conflicts when generating multiple percolators in the same package.	If you use the same sub struct in 2 different structs for example), the helper functions generated for the percolator will collide. A workaround is to generate the percolator in another package, and import it. This is definitely something we want to address in the future.	No work planned for now

Initial RFC

You can find the original RFC here.

License

This project is licensed under the terms of the Apache 2.0.