feat: new event-based decisions log buffer implementation #7446

sspaink · 2025-03-12T23:59:29Z

Why the changes in this PR are needed?

resolves: #5724

The existing buffer used by the decisions log plugin suffers from performance issues at a large scale. The heavy use of locks and when the events are compressed are the culprits for this slowdown.

What are the changes in this PR?

This PR introduces a new buffer implementation for the Decisions Log Plugin with the goal to improve performance.

Two new configuration options are introduced:

decision_logs.reporting.buffer_type - toggle to use new buffer (labeled "event") and defaults to current implementation (labeled "size")
decision_logs.reporting.buffer_size_limit_events - sets the number of events the buffer can hold, must be above zero and defaults to 100

Simple performance comparison

The following setup was used for both the old and new buffer implementation:

example.rego:

package example

allow if {
    true
}

Run the OPA server

$ ./opa_darwin_arm64 run -c opa-conf.yaml --server ./example.rego

Run a local server to consume logs (code runs a server and prints out decisions)

$ go run main.go

config

services:
  logeater:
    url: http://localhost:8080

decision_logs:
  service: logeater
  reporting:
    buffer_type: <changed per test>
    min_delay_seconds: 5
    max_delay_seconds: 10

Results using the "event" buffer type

buffer_type: event
buffer_size_limit_events: 1000

$ echo 'POST http://localhost:8181/v1/data/example/allow' | vegeta attack --duration=300s -rate=500 | tee results.bin | vegeta report

Requests      [total, rate, throughput]         150000, 500.00, 500.00
Duration      [total, attack, wait]             5m0s, 5m0s, 671.375µs
Latencies     [min, mean, 50, 90, 95, 99, max]  101.834µs, 681.405µs, 673.425µs, 797.284µs, 856.462µs, 1.41ms, 6.776ms

Results using the "size" buffer type:

buffer_type: size

$ echo 'POST http://localhost:8181/v1/data/example/allow' | vegeta attack --duration=300s -rate=500 | tee results.bin | vegeta report

Requests      [total, rate, throughput]         150000, 500.00, 500.00
Duration      [total, attack, wait]             5m0s, 5m0s, 572.833µs
Latencies     [min, mean, 50, 90, 95, 99, max]  96.042µs, 930.795µs, 585.86µs, 807.216µs, 884.121µs, 6.514ms, 80.595ms

Notes to assist PR review:

Please see the included README.md for a comparison between the current and this new buffer implementation.

netlify · 2025-03-13T00:01:06Z

✅ Deploy Preview for openpolicyagent ready!

Name	Link
🔨 Latest commit	`fa26f44`
🔍 Latest deploy log	https://app.netlify.com/sites/openpolicyagent/deploys/67e1c658f8be850008fa68bb
😎 Deploy Preview	https://deploy-preview-7446--openpolicyagent.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify site configuration.

srenatus

This looks great, especially the numbers. I've gone through the code and commented a bit, adding a few questions for my understanding. I hope you don't mind 😃

v1/plugins/logs/eventBuffer.go

v1/plugins/logs/plugin.go

v1/plugins/logs/eventBuffer.go

srenatus

LGTM, thanks for bearing with me 😃

johanfylling

Looking good 👍 😃

Some questions/comments.

To be continued ..

v1/plugins/logs/README.md

docs/content/configuration.md

v1/plugins/logs/eventBuffer.go

v1/plugins/logs/plugin.go

v1/plugins/logs/eventBuffer.go

johanfylling

Some additional notes.

Do we have some sort of comparative analysis on drop behavior between the two buffer types? E.g. is one buffer more prone to dropping events than the other while under the same pressure?

v1/plugins/logs/eventBuffer.go

v1/plugins/logs/plugin_test.go

v1/plugins/logs/eventBuffer.go

v1/plugins/logs/eventBuffer_test.go

v1/plugins/logs/plugin.go

johanfylling

Making good progress 😃

v1/plugins/logs/eventBuffer.go

v1/plugins/logs/plugin.go

v1/plugins/logs/eventBuffer.go

v1/plugins/logs/eventBuffer_test.go

This new event-based buffer provides a performance improvement over the exisiting buffer by reducing locks and allowing concurrent writes. Signed-off-by: sspaink <[email protected]>

Signed-off-by: sspaink <[email protected]>

…opping of ND cache only happens when event is read from buffer. Signed-off-by: sspaink <[email protected]>

…o 10,000. Signed-off-by: sspaink <[email protected]>

Signed-off-by: sspaink <[email protected]>

…lure. Signed-off-by: sspaink <[email protected]>

v1/plugins/logs/eventBuffer.go

…multiple chunks. Signed-off-by: sspaink <[email protected]>

Signed-off-by: sspaink <[email protected]>

…that is just used for a single log. Signed-off-by: sspaink <[email protected]>

…ure. Signed-off-by: sspaink <[email protected]>

Signed-off-by: sspaink <[email protected]>

johanfylling · 2025-03-26T20:31:59Z

Some recent measurements (500 concurrent requesting clients):

Blue is the old size type buffer, green is the new event type buffer. Blue spikes correlate with uploads to log server.

Old size type buffer:

Global metrics:
Requests: 200
Requests total: 100000
Duration: 1m19.336304917s
Max concurrency: 500
Average req/s: 1260.4569888226824
timer_server_handler_ns: Min: 81.042µs, Max: 145.218709ms, Mean: 2.116985ms, P50: 263.708µs, P75: 423.947µs, P90: 1.20862ms, P95: 10.839443ms, P99: 34.751751ms, P99.9: 145.114025ms, P99.99: 145.218709ms
timer_rego_external_resolve_ns: Min: 41ns, Max: 6.75µs, Mean: 208ns, P50: 167ns, P75: 250ns, P90: 292ns, P95: 416ns, P99: 988ns, P99.9: 6.611µs, P99.99: 6.75µs
timer_rego_query_compile_ns: Min: 10.875µs, Max: 3.265709ms, Mean: 53.166µs, P50: 35.146µs, P75: 45.864µs, P90: 70.816µs, P95: 108.539µs, P99: 362.924µs, P99.9: 3.247513ms, P99.99: 3.265709ms
timer_rego_query_eval_ns: Min: 5.875µs, Max: 685.167µs, Mean: 39.172µs, P50: 20.521µs, P75: 45.458µs, P90: 72.959µs, P95: 102.108µs, P99: 366.857µs, P99.9: 683.887µs, P99.99: 685.167µs
duration: Min: 193.666µs, Max: 163.735333ms, Mean: 3.120354ms, P50: 570.374µs, P75: 1.413905ms, P90: 5.221721ms, P95: 9.14081ms, P99: 39.370902ms, P99.9: 163.383476ms, P99.99: 163.735333ms
Peaks:
duration: 168.657916ms
timer_server_handler_ns: 164.581375ms
timer_rego_external_resolve_ns: 569.792µs
timer_rego_query_compile_ns: 17.858625ms
timer_rego_query_eval_ns: 17.963041ms
Peak duration: 168.657916ms

New event type buffer:

Global metrics:
Requests: 200
Requests total: 100000
Duration: 1m18.066683334s
Max concurrency: 500
Average req/s: 1280.9561739950016
timer_rego_query_eval_ns: Min: 5.709µs, Max: 2.339125ms, Mean: 43.701µs, P50: 24.084µs, P75: 46.916µs, P90: 76.249µs, P95: 105.041µs, P99: 295.527µs, P99.9: 2.309766ms, P99.99: 2.339125ms
timer_server_handler_ns: Min: 65.417µs, Max: 9.999292ms, Mean: 365.524µs, P50: 245.292µs, P75: 363.969µs, P90: 534.25µs, P95: 861.422µs, P99: 3.090057ms, P99.9: 9.927875ms, P99.99: 9.999292ms
timer_rego_external_resolve_ns: Min: 41ns, Max: 2.459µs, Mean: 241ns, P50: 209ns, P75: 292ns, P90: 417ns, P95: 500ns, P99: 833ns, P99.9: 2.428µs, P99.99: 2.459µs
duration: Min: 185.042µs, Max: 19.097167ms, Mean: 1.403245ms, P50: 552.854µs, P75: 1.244437ms, P90: 3.413012ms, P95: 5.84851ms, P99: 12.394373ms, P99.9: 19.062707ms, P99.99: 19.097167ms
timer_rego_query_compile_ns: Min: 11.834µs, Max: 1.517209ms, Mean: 53.961µs, P50: 40.729µs, P75: 57.5µs, P90: 84.883µs, P95: 112.292µs, P99: 273.027µs, P99.9: 1.499569ms, P99.99: 1.517209ms
Peaks:
timer_rego_query_compile_ns: 22.58375ms
timer_rego_query_eval_ns: 23.331ms
timer_server_handler_ns: 25.787417ms
timer_rego_external_resolve_ns: 757.167µs
duration: 50.463917ms
Peak duration: 50.463917ms

johanfylling

Looking great 👍 .
Only two remaining concerns.

v1/plugins/logs/eventBuffer.go

Signed-off-by: sspaink <[email protected]>

johanfylling

Awesome!
Thank you for bearing with me :)

anderseknert · 2025-03-26T21:38:44Z

142 comments (143 once I post this, I suppose) on a PR must be a new record for OPA!
Had it been me I'd probably have closed my laptop for good to pursue farming or something.

Amazing work @sspaink 👍

sspaink requested a review from johanfylling March 12, 2025 23:59

sspaink force-pushed the improvelogbuffer branch 2 times, most recently from 61267a3 to 8d17daa Compare March 13, 2025 04:30

srenatus reviewed Mar 13, 2025

View reviewed changes

srenatus reviewed Mar 14, 2025

View reviewed changes

v1/plugins/logs/eventBuffer.go Outdated Show resolved Hide resolved

srenatus reviewed Mar 14, 2025

View reviewed changes

v1/plugins/logs/eventBuffer.go Outdated Show resolved Hide resolved

srenatus approved these changes Mar 14, 2025

View reviewed changes

johanfylling reviewed Mar 14, 2025

View reviewed changes

sspaink mentioned this pull request Mar 17, 2025

Make latency impact of decision logs predictable #5724

Closed

johanfylling reviewed Mar 17, 2025

View reviewed changes

mefarazath mentioned this pull request Mar 18, 2025

filters/openpolicyagent: Add a sample benchmark to measure latency percentiles zalando/skipper#3442

Draft

mjungsbluth reviewed Mar 18, 2025

View reviewed changes

v1/plugins/logs/plugin.go Outdated Show resolved Hide resolved

This was referenced Mar 18, 2025

Decision log plugin: don't drop events when reaching buffer limit, slow down and retry #7454

Open

Decision log plugin: trigger upload as soon as the buffer limit is reached #7455

Open

johanfylling reviewed Mar 20, 2025

View reviewed changes

sspaink force-pushed the improvelogbuffer branch from 16e0278 to fa26f44 Compare March 24, 2025 20:53

sspaink and others added 13 commits March 24, 2025 15:54

feat: new event-based decisions log buffer implementation

f8a596b

This new event-based buffer provides a performance improvement over the exisiting buffer by reducing locks and allowing concurrent writes. Signed-off-by: sspaink <[email protected]>

resolve feedback, biggest change is handling errors when reconfiguring

70729ba

Signed-off-by: sspaink <[email protected]>

track dropped events

554c8cb

Signed-off-by: sspaink <[email protected]>

no need for generics

8f26cf4

Signed-off-by: sspaink <[email protected]>

resolve feedback, update readme and config validation

b297eaf

Signed-off-by: sspaink <[email protected]>

move event processing entirely to consumer side. Serialization and dr…

e1c6757

…opping of ND cache only happens when event is read from buffer. Signed-off-by: sspaink <[email protected]>

resolve feedback, mainly added more tests and changed event default t…

2e4c128

…o 10,000. Signed-off-by: sspaink <[email protected]>

update metric

047e90d

Signed-off-by: sspaink <[email protected]>

oops, broke a test.

9b0612c

Signed-off-by: sspaink <[email protected]>

re-use chunkEncoder, resolve other feedback as well.

1743d54

Signed-off-by: sspaink <[email protected]>

resolve feedback. push Eventv1 and add event to next chunk

5e1b17c

Signed-off-by: sspaink <[email protected]>

fix lint error by moving decode test function. caused by build tag.

2a3b265

Signed-off-by: sspaink <[email protected]>

prevent infinite push block by retrying, requeue events on upload fai…

ef5b5c2

…lure. Signed-off-by: sspaink <[email protected]>

sspaink force-pushed the improvelogbuffer branch from fa26f44 to ef5b5c2 Compare March 24, 2025 20:54

johanfylling reviewed Mar 25, 2025

View reviewed changes

sspaink added 9 commits March 25, 2025 15:16

fix how the chunkEncoder was being used, upload the entire buffer in …

42c4e17

…multiple chunks. Signed-off-by: sspaink <[email protected]>

comment re-worded and unused ctx removed.

b712afd

Signed-off-by: sspaink <[email protected]>

replace boolean with custom error. one less boolean to keep track of …

8a5366e

…that is just used for a single log. Signed-off-by: sspaink <[email protected]>

prevent upload to happen when channel could be closed during reconfig…

55693fc

…ure. Signed-off-by: sspaink <[email protected]>

fix error handling

5932f76

Signed-off-by: sspaink <[email protected]>

don't block dropping an event, in case an upload empties the buffer.

89da1fe

Signed-off-by: sspaink <[email protected]>

add entire chunk back into buffer to save time decoding.

1f1a150

Signed-off-by: sspaink <[email protected]>

cleanup

3aa2943

Signed-off-by: sspaink <[email protected]>

check logger

d9db575

Signed-off-by: sspaink <[email protected]>

johanfylling reviewed Mar 26, 2025

View reviewed changes

v1/plugins/logs/eventBuffer.go Outdated Show resolved Hide resolved

v1/plugins/logs/eventBuffer.go Show resolved Hide resolved

try to upload each chunk and try to drop ND cache if marshal error

0f3c19a

Signed-off-by: sspaink <[email protected]>

johanfylling approved these changes Mar 26, 2025

View reviewed changes

sspaink merged commit cd66fa3 into open-policy-agent:main Mar 26, 2025
28 checks passed

sspaink mentioned this pull request Sep 5, 2025

plugin/decision: refactor size buffer into its own type #7884

Merged

feat: new event-based decisions log buffer implementation #7446

feat: new event-based decisions log buffer implementation #7446

Uh oh!

Conversation

sspaink commented Mar 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Why the changes in this PR are needed?

What are the changes in this PR?

Simple performance comparison

Results using the "event" buffer type

Results using the "size" buffer type:

Notes to assist PR review:

Uh oh!

netlify bot commented Mar 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Deploy Preview for openpolicyagent ready!

Uh oh!

srenatus left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

srenatus left a comment

Choose a reason for hiding this comment

Uh oh!

johanfylling left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

johanfylling left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

johanfylling left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

sspaink commented Mar 12, 2025 •

edited

Loading

netlify bot commented Mar 13, 2025 •

edited

Loading