Comparing changes

BlackholeProduce makes the cluster accept produce requests and reply success without persisting records to segments: offsets still advance and idempotent/transactional sequence validation still runs, but the record bytes are discarded and no segment/index/fetch-watch state is built. This keeps broker-side storage from dominating client produce-throughput benchmarks; do not consume from a blackholed cluster. The early return in pushBatch mirrors the offset/txn/lso accounting of the normal path (high watermark, uncommittedPIDs, lastStableOffset, nbytes) so producers observe fully well-formed responses.

Batches are bounded by ProducerBatchMaxBytes measured on UNCOMPRESSED records, so with a compression ratio R, batches arrive at the broker ~R times smaller than max.message.bytes allows and a backlog costs ~R times more produce round trips per partition than necessary. The existing workaround - over-provisioning ProducerBatchMaxBytes against an assumed ratio - risks MESSAGE_TOO_LARGE (and permanently failed records, since we never split batches) whenever the ratio dips. This adds an opt-in producer option, StreamingCompression, that merges a partition's backlog of never-sent batches at drain time into batches bounded by their exact COMPRESSED size, for codecs that can stream and flush (gzip, zstd, lz4; snappy is a one-shot block codec and is unaffected): - Admission is untouched: records buffer into uncompressed-bounded batches exactly as today, and the sticky partitioner's OnNewBatch behavior is unchanged. With no backlog (an idle or keeping-up partition), draining is byte-for-byte today's behavior. - The sink's drain loop coalesces via freeze-steal-merge: (1) under recBuf.mu, the span of consecutive never-sent batches at the drain index is marked merging, which blocks admission appends and drains without removing the batches from recBuf.batches - they stay visible to failAllRecords/purge so a concurrent failure sweep still owns their promises; (2) outside recBuf.mu (admission never stalls behind compression), the span's records stream through the compressor with a worst-case decide-before-write bound: bytes since the last flush plus the candidate record are assumed incompressible, a flush tightens the bound only when that worst case would overflow, and a final overflow cuts BEFORE writing, so the compressed batch provably never exceeds ProducerBatchMaxBytes and the compressor only ever holds accepted records (sealing is always a clean close, no footer surgery, no splitting); a cut mid-source moves the source's remaining records into a fresh tail batch, legal because the source was never sent and sequences are only assigned at drain; (3) back under recBuf.mu, the splice validates that the span still sits, in order and unfrozen, at the head of what is left to drain - tolerant of finished front batches popping during the merge - and otherwise discards the merge, restoring the span records' length/timestamp stamps (phase 2 re-stamped them merge-relative; a discarded source drains on the legacy path from those stamps, and a stale stamp whose varint width differs would desync the wire records). - Only never-frozen batches merge. A batch is frozen exactly when first selected into a produce request, so frozen means possibly sent: its content and sequence never change and retries resend it verbatim, preserving idempotent and transactional semantics exactly. - Sealed merged batches cache their compressed blob and write it with a freshly built header + CRC per attempt (appendSealedTo), so retries no longer recompress - and the batch's wireLength becomes its exact wire size, so request sizing packs by real bytes. - Every odd case (unknown codec, compressor error, produce version below the codec's floor on ancient brokers) demotes the batch to the legacy path; records are always kept, so the fallback is always available. Local benchmarks (blackholed kfake, 512B ~5x-compressible values, medians of 4): gzip linger0 1970 -> 2077 ns/op (+5.5%) rec/req 7317 -> 125000 (17.1x) gzip linger1ms 1964 -> 2046 ns/op (+4.1%) rec/req 7317 -> 87500 (12.0x) zstd linger0 349 -> 487 ns/op (+39.7%) rec/req 6386 -> 55000 ( 8.6x) zstd linger1ms 429 -> 505 ns/op (+17.8%) rec/req 6470 -> 13492 ( 2.1x) lz4 linger0 383 -> 428 ns/op (+11.8%) rec/req 6818 -> 26538 ( 3.9x) none (control) ~unchanged rec/req unchanged The rec/req gain is the point: fewer round trips per partition raises per-partition throughput wherever round trips bound it (real network RTT, bounded in-flight). The zstd CPU delta reflects compression moving from the pipelined connection-writer goroutine onto the drain goroutine; the option is off by default and documents the trade.

Five tests exercising the streaming (compressed-size bound) batch path; kfake validates every produced batch's CRC, batch length, magic, compression type, and idempotent sequences, and the consume side decompresses and re-parses every record, so these prove the merged wire bytes end to end rather than merely accepting them: - RoundTrip: {gzip,zstd,lz4,snappy,none} x {stream,legacy} with mixed sizes/headers/nil values under manual flushing (batches roll purely on the size bound); byte-for-byte consume verification plus the density assertion (streaming codecs pack >= 2.5x more records per produce request; snappy/none are asserted no-ops). - Retry: the first produce request fails with NOT_LEADER after merged batches formed; the retry must resend them verbatim (kfake's sequence validation rejects any gap/reorder) and consuming verifies exactly-once. - Purge / Close: against a slow broker with merges and requests in flight, every produce promise fires exactly once and nothing hangs. - LiveDrain: merging in the live drain path (slow broker + linger, no manual flush), byte-for-byte exactly-once; its failure diagnostic dumps per-partition positions, broker end offsets, and a raw fetch of the stuck offset (this diagnostic is what pinned the discarded- merge stamp-restoration bug during development). NOTE: these tests reference kgo.StreamingCompression, which requires the kgo from this branch; build kfake with a go.work pointing at the local kgo until a kgo release carries the option.

BenchmarkProduceBlackhole measures end-to-end produce throughput (ns/record, MB/s) and batch density (records per produce request) against a BlackholeProduce cluster, across codec x linger x {legacy,stream}; BenchmarkProduceBlackholePar is the same with 4x GOMAXPROCS producing goroutines. The blackhole keeps broker-side storage out of the measurement, and the rec/req metric is the density signal that motivated kgo.StreamingCompression. Same go.work note as the streaming tests: the stream dimension needs the local kgo.

The throwaway experiment that de-risked compressed-size-bound batching before any client integration: packA (uncompressed bound, one-shot compress - the legacy strategy) vs packB (streaming decide-before-write with worst-case reservation and flush-to-tighten - what became kgo.StreamingCompression's merge bound), plus an adaptive variant, a never-overflow fuzz test, and density/CPU benchmarks over gzip/zstd at tunable compressibility. Kept only as the durable record of the design exploration; nothing imports it. DROP THIS COMMIT freely if the history should stay lean.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Comparing changes

Open a pull request

Uh oh!

Commits on Jul 7, 2026

This comparison is taking too long to generate.

Uh oh!