Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

@coderabbitai please review

@coderabbitai pls do a critical review

#!/bin/bash
# Get the list of changed files in this PR
gh pr view 6759 --json files --jq '.files[].path' | head -20

#!/bin/bash
# Look at the delta_bitpacked decoder that has 0% coverage
fd -t f "decoder.rs" qdbr/parquet2/src/encoding/delta_bitpacked/ --exec cat {}

#!/bin/bash
# Examine the bug fix around null sentinel values mentioned in PR description
rg -n "sentinel" --type rust -C 5

#!/bin/bash
# Check for unsafe code blocks in the modified decoders
rg -n "unsafe" qdbr/src/parquet_read/decoders/ --type rust -C 3

#!/bin/bash
# Look at the slice_reader with low coverage
cat qdbr/parquet2/src/read/page/slice_reader.rs

#!/bin/bash
# Examine the test files to understand what's being tested
fd -t f "test" qdbr/src/parquet_read/decoders/ --exec basename {}

#!/bin/bash
# Look at how null_count=0 optimization is implemented
rg -n "null_count.*0" qdbr/src/parquet_read/ --type rust -C 5

#!/bin/bash
# Fix paths - examine the delta_bitpacked decoder with 0% coverage
cat core/rust/qdbr/parquet2/src/encoding/delta_bitpacked/decoder.rs

#!/bin/bash
# Check for unsafe code blocks in the modified decoders
rg -n "unsafe" core/rust/qdbr/src/parquet_read/decoders/ --type rust -C 3

#!/bin/bash
# Look at the slice_reader with low coverage
cat core/rust/qdbr/parquet2/src/read/page/slice_reader.rs

#!/bin/bash
# Look at how null_count=0 optimization is implemented
rg -n "null_count" core/rust/qdbr/src/parquet_read/ --type rust -C 3 | head -80

#!/bin/bash
# Examine the main decode.rs file which has 78% coverage - what's missing?
cat core/rust/qdbr/src/parquet_read/decode.rs | head -200

#!/bin/bash
# Look at the benchmark implementation to understand what's being measured
cat core/rust/qdbr/benches/decode_page.rs | head -150

#!/bin/bash
# Check the diff for the null sentinel bug fix
gh pr diff 6759 --patch | grep -A 10 -B 10 "sentinel\|null.*=" | head -50

#!/bin/bash
# Look at test files to understand test coverage
fd -t f "test" core/rust/qdbr/src/parquet_read/ -e rs | head -10

#!/bin/bash
# Search for tests related to delta_binary_packed
rg -n "mod tests" core/rust/qdbr/src/parquet_read/decoders/delta_binary_packed.rs -A 20

#!/bin/bash
# Check what null sentinel values are used
rg -n "null_value|NULL" core/rust/qdbr/src/parquet_read/decoders/ --type rust | grep -E "(const|let|fn|null_value)" | head -40

#!/bin/bash
# Look at the delta_bitpacked batch decoder implementation more carefully for edge cases
cat core/rust/qdbr/parquet2/src/encoding/delta_bitpacked/decoder.rs | grep -A 30 "decode_batch"

#!/bin/bash
# Check if there are any overflow or boundary checks in decoders
rg -n "overflow|saturating|checked|wrapping" core/rust/qdbr/src/parquet_read/decoders/delta_binary_packed.rs -C 2

#!/bin/bash
# Look for error handling in the new batch decoder
rg -n "Error|Result|unwrap" core/rust/qdbr/parquet2/src/encoding/delta_bitpacked/decoder.rs -C 2 | head -80

#!/bin/bash
# Check how the null_count optimization is used - potential for bugs if metadata is wrong
rg -n "null_count\(\)" core/rust/qdbr/src/parquet_read/decode.rs -B 3 -A 10 | head -50

@kafka1991, conducting a critical review as requested. I have serious concerns about this PR:

🚨 Critical: Zero Test Coverage on Core Performance Path

The delta_bitpacked decoder has 0% coverage with 65 new lines. This is unacceptable for code that's in the critical decode path. The new decode_batch method in core/rust/qdbr/parquet2/src/encoding/delta_bitpacked/decoder.rs has zero tests despite:

1. Complex state management across miniblocks
2. Manual buffer management with a hardcoded 128-element stack array
3. Wrapping arithmetic for delta encoding
4. Multiple early returns and edge cases

Question: Why is there no test coverage here? The existing tests only cover the iterator interface, not the batch decoder.

🚨 Trusting Parquet Metadata Without Validation

Line 2100-2103 in decode.rs:

let nc = page.header.null_count();
if nc == Some(0) {
    return Ok(None);
}

You're trusting parquet metadata to skip definition level decoding. What happens with:

• Corrupted parquet files where null_count=0 but nulls exist?
• Maliciously crafted files?
• Files from buggy writers?

This optimization trades correctness for performance. Have you validated this against malformed inputs? Where are the fuzz tests?

🚨 Undocumented Bug Fix

The PR description mentions:

"Fixing bug with incorrect null sentinel values"

Where is this bug fix? I can't find:

• What the incorrect values were
• What they should be
• Test cases that would have caught this
• Verification the fix is complete

This is a data corruption bug and needs full documentation and regression tests.

⚠️ Unsafe Code with Fragile Assumptions

Lines 313-316, 334-337, 348-351, 358-363 in delta_binary_packed.rs and throughout other decoders:

unsafe {
    *self.buffers_ptr.add(self.buffers_offset) = self.null_value;
}

Every decoder relies on the comment:

"SAFETY: destination pointer stays in-bounds because decode paths reserve output upfront."

This assumes:

1. Reservation calculations are always correct
2. Definition levels match value counts perfectly
3. Parquet metadata is accurate

What validates these assumptions? One off-by-one error and you have UB. Where are the debug assertions? Why not use safe slice indexing in debug builds?

⚠️ Panic Points in Production

Line 286 in delta_bitpacked/decoder.rs:

let current_block = self.current_block.as_mut().unwrap();

Line 85:

Some(bitpacked::Decoder::try_new(miniblock, num_bits, length).unwrap())

Line 111, 144-145:

out[written + i] = self.min_delta + mb.next().unwrap_or_default() as i64;
// ...
.map(|x| x.next().unwrap_or_default())
.unwrap_or(0) as i64;

Multiple unwrap() calls in hot paths. Why are these safe? What prevents panics? unwrap_or_default() will silently return 0 on error - is that the intended behavior or are you hiding bugs?

⚠️ Wrapping Arithmetic Everywhere

Lines 327, 331, 391-396, 410-411, 431-436, 486-490 in delta_binary_packed.rs:

self.current_value = self.current_value.wrapping_add(...)

Why is wrapping arithmetic correct here? Delta encoding typically uses zigzag encoding for signed values. Document why overflow wrapping is semantically correct, or prove these operations can't overflow with valid inputs.

⚠️ Magic Number: 128-Element Buffer

Line 237 in delta_bitpacked/decoder.rs:

let mut delta_buf = [0i64; 128];

Why 128? Is this:

• Related to miniblock size?
• A cache line optimization?
• Arbitrary?

This should be a named constant with documentation. What's the performance impact of different sizes?

Missing Test Coverage for Edge Cases

I found no tests for:

• ❌ Single-value pages
• ❌ Empty miniblocks (bitwidth=0)
• ❌ Miniblock boundary conditions
• ❌ Maximum value counts (overflow scenarios)
• ❌ Mismatched null_count metadata
• ❌ Truncated page data
• ❌ Invalid bitwidths (>64)

The benchmarks only test the happy path with well-formed data. Where's the adversarial testing?

Questionable: SliceReader at 16.67% Coverage

slice_reader.rs has 16.67% coverage with only 1 of 6 new lines covered. This code handles page header parsing and could be an attack vector. Why is this untested?

Performance Claims Need Validation

The benchmarks show impressive numbers, but:

1. Jitter acknowledged: "there is a lot of jitter" - are results statistically significant?
2. Apple M5: These are unreleased chips. Do you mean M4/M3?
3. No error bars: The ± percentages are provided but confidence intervals missing
4. Clickbench has regressions: q5 (native 236ms → parquet 426ms), q20-22 show large slowdowns
5. Plain encoding regressions: Several plain-encoded types show slowdowns (date +6.34%, double +5.86%, float +4.65%, int +2.24%)

Challenge: Have you profiled where the remaining bottlenecks are? The string/varchar/array improvements are modest (-14% to -33%) compared to primitives (-85%+). Why?

Recommendations

Before merging, you must:

1. Add comprehensive tests for delta_bitpacked batch decoder (target 80%+ coverage)
2. Document and test the null sentinel bug fix with regression tests
3. Add validation for the null_count optimization or document the security trade-off
4. Add debug assertions for all unsafe pointer arithmetic
5. Replace unwrap() with proper error propagation or document panic conditions
6. Fuzz test the decoders with malformed parquet files
7. Document wrapping arithmetic semantics
8. Add edge case tests for all decoders
9. Clarify the M5 benchmark environment
10. Investigate Clickbench regressions (q5, q20-22)

This PR has impressive performance gains but sacrifices safety and correctness without adequate testing. The 74% overall coverage is insufficient for parser code handling untrusted input.

What's your plan to address these concerns?

Learnt from: javier
Repo: questdb/questdb PR: 0
File: :0-0
Timestamp: 2026-02-06T18:22:28.130Z
Learning: In questdb/questdb, for PRs requested by javier, provide line-cited, push-back style reviews with concrete follow-ups (tests, diffs, and performance considerations).

Learnt from: javier
Repo: questdb/questdb PR: 0
File: :0-0
Timestamp: 2026-02-06T18:22:28.130Z
Learning: For questdb/questdb, javier expects a rigorous, push-back style review; always include concrete line-cited findings and propose targeted tests/fixes when reviewing javier’s PRs.

Learnt from: jerrinot
Repo: questdb/questdb PR: 0
File: :0-0
Timestamp: 2026-02-04T11:18:36.492Z
Learning: For PRs by jerrinot, always use a thorough and critical review mode: question design decisions, challenge assumptions, scrutinize implementations for edge cases and potential issues, and avoid superficial validation. They prefer hard-core reviews that push back on the code.

Learnt from: javier
Repo: questdb/questdb PR: 0
File: :0-0
Timestamp: 2026-02-06T18:22:28.130Z
Learning: In questdb/questdb, javier prefers hard‑core, critical reviews that challenge design choices and edge cases; respond with deep, push-back style analyses for javier’s requests.

#!/bin/bash
# Fetch the full PR diff to analyze the changes
gh pr diff 6759 --repo questdb/questdb > /tmp/pr_diff.txt

# Show first part of the diff to understand the scope
head -n 500 /tmp/pr_diff.txt

# Get total line count
wc -l /tmp/pr_diff.txt

#!/bin/bash
# Extract key decoder implementation files from the diff
echo "=== Delta Binary Packed Decoder ==="
sed -n '/^diff.*delta_binary_packed\/decoder.rs/,/^diff.*[^delta_binary_packed]/p' /tmp/pr_diff.txt | head -n 400

echo -e "\n=== Plain Decoder ==="
sed -n '/^diff.*decoders\/plain.rs/,/^diff/p' /tmp/pr_diff.txt | head -n 400

echo -e "\n=== RLE Dictionary Decoder ==="
sed -n '/^diff.*decoders\/rle_dictionary.rs/,/^diff/p' /tmp/pr_diff.txt | head -n 400

#!/bin/bash
# Extract more decoder implementations and critical sections
echo "=== Delta Bitpacked Encoding Decoder (parquet2 library) ==="
sed -n '/^diff.*parquet2.*delta_bitpacked/,/^diff/p' /tmp/pr_diff.txt | head -n 500

echo -e "\n=== Decompression/Page Reader Changes ==="
sed -n '/^diff.*slice_reader.rs/,/^diff/p' /tmp/pr_diff.txt | head -n 300

echo -e "\n=== Main Decode Logic ==="
sed -n '/^diff.*parquet_read\/decode.rs/,/^diff/p' /tmp/pr_diff.txt | head -n 600

#!/bin/bash
# Look for unsafe blocks and potential issues
echo "=== Count of unsafe blocks ==="
grep -c "unsafe" /tmp/pr_diff.txt

echo -e "\n=== Unsafe pointer operations in plain decoder ==="
grep -A5 -B5 "unsafe" /tmp/pr_diff.txt | grep -A5 -B5 "PlainPrimitiveDecoder" | head -n 100

echo -e "\n=== Unsafe operations in RLE dictionary ==="
grep -A5 -B5 "unsafe" /tmp/pr_diff.txt | grep -A5 -B5 "RleDictionary" | head -n 100

echo -e "\n=== Bounds checking patterns ==="
grep -E "(bounds|overflow|len\(\)|checked_|saturating_)" /tmp/pr_diff.txt | head -n 80

echo -e "\n=== Test files added ==="
grep -E "^\+.*#\[test\]|^\+.*mod tests" /tmp/pr_diff.txt | wc -l

@kafka1991, I'll provide a critical review as requested. This PR has significant performance improvements but introduces substantial risk through extensive unsafe code with inadequate testing and questionable error handling patterns.

Critical Issues

1. Zero Test Coverage on Critical Performance Path ⚠️

The new delta_bitpacked/decoder.rs file has 0% coverage despite adding 65 lines including the performance-critical decode_batch() and decode_deltas_batch() methods. This is unacceptable for code that:

• Uses complex state machines with invariants
• Performs prefix-sum accumulation on untrusted input
• Claims wrapping arithmetic is correct

Required: Comprehensive fuzzing and unit tests covering:

• Empty blocks
• Miniblock boundary conditions
• Integer overflow in prefix sums
• Malformed block headers
• Values_remaining edge cases

2. Unsafe Pointer Arithmetic Relies on "Trusted" Malicious Input

In PlainPrimitiveDecoder::push() and push_slice():

unsafe {
    *self.buffers_ptr.add(self.buffers_offset) = self
        .converter
        .convert(self.values.add(self.values_offset).read_unaligned());
}

The comment states: "We rely on trusted parquet metadata/level streams to keep source in-bounds". This is fundamentally broken - parquet files can be malicious or corrupted. You're trusting attacker-controlled metadata to prevent memory safety violations.

What happens when:

• A malicious parquet file claims 1000 values but only provides 100 bytes?
• Level streams are corrupted and values_offset exceeds buffer bounds?
• Dictionary page promises N entries but contains fewer?

3. Silent Error Handling Creates Data Corruption Risk

RleDictionaryDecoder stores errors but continues writing to output buffers:

fn push(&mut self) -> ParquetResult<()> {
    if self.inner.error.is_some() {
        return Ok(());  // Returns "success" despite error!
    }
    // ... writes data ...
    if idx >= self.dict.len() {
        self.inner.error = Some(fmt_err!(...));
        return Ok(());  // Silently fails, may leave garbage in buffer
    }
}

This pattern means:

1. Error occurs on row 1000 of 100K
2. Decoder returns Ok(()) for remaining 99K rows
3. Output buffer contains 1K valid + 99K uninitialized/garbage data
4. Error only surfaces when someone calls result() - if they call it

Fix: Either fail-fast on errors OR explicitly document that callers MUST call result() and handle partial data correctly.

4. Batch Decoder State Invariant Violation

In delta_bitpacked/decoder.rs, the decode_batch() method claims:

// Maintain the invariant: next_value must hold the next value to emit.
// Load the next delta so next_value is ready for subsequent calls.
if self.values_remaining > 0 {
    let delta = self.load_delta()?;
    self.next_value = self.next_value.wrapping_add(delta);
}

But this is after the main loop exits. If load_delta() fails here, what state is the decoder in? The invariant is broken - next_value doesn't contain the next value, and values_remaining > 0 but we can't emit anything. The next decode_batch() call will emit stale data.

5. Pointer Invalidation Not Guarded

Decoders cache raw pointers:

buffers_ptr: *mut U,

Only refreshed in reserve():

self.buffers_ptr = self.buffers.data_vec.as_mut_ptr().cast();

Problem: If ColumnChunkBuffers is ever passed to other code that might reallocate the vec (growing in another decoder, for example), this cached pointer becomes dangling. You're betting the entire memory safety on no interleaved operations touching the same buffer.

6. Dictionary Bounds Checking Insufficient

In RleDictionaryDecoder::push_slice(), byte indices fast path:

RleIterator::ByteIndices { data, pos } => {
    for i in 0..n {
        let idx = unsafe { *bytes.get_unchecked(i) } as u32;
        if idx >= dict_len {
            self.inner.error = Some(fmt_err!(...));
            return Ok(());  // Partial write completed!
        }
    }
}

You checked avail = data.len() - *pos and n = remaining.min(avail), but then use get_unchecked(i) where i < n. If *pos isn't at the start of bytes, bytes[i] != data[*pos + i]. The slicing appears incorrect.

7. Test Coverage on Other Critical Files Inadequate

• row_groups.rs: 51.35% coverage
• array.rs: 61.75% coverage
• rle_dictionary.rs: 66.53% coverage

For decoder code handling untrusted input, anything below 90% is insufficient. You need:

• Malformed input tests
• Boundary condition tests
• Error path coverage
• Fuzzing integration

8. No Miri Validation Mentioned

With 132 unsafe blocks, you absolutely need Miri testing to detect undefined behavior like use-after-free or uninitialized memory reads. Is this PR miri-clean? Have you run it?

Design Challenges

Question: Why silent error accumulation instead of fail-fast? The pattern where decoders store errors and return Ok(()) is unprecedented in the codebase and creates subtle bugs. What's the justification?

Question: Why cache raw pointers instead of recalculating on each access? The microsecond saving isn't worth the memory safety risk of dangling pointers.

Question: What's the threat model? Are you assuming parquet files are always trusted? If so, document it. If not, every "trusted metadata" comment is a critical bug.

Required Before Merge

1. ✅ Achieve >90% test coverage on all decoder files
2. ✅ Add fuzzing for all decoders with malformed inputs
3. ✅ Run Miri and fix all UB findings
4. ✅ Add explicit bounds checks or prove mathematically why every pointer deref is in-bounds
5. ✅ Document error handling: Must callers check result()? What's in output buffers on error?
6. ✅ Fix state invariant in batch decoder error path
7. ✅ Fix dictionary byte indices slicing bug

The performance gains are impressive, but this code is a memory safety minefield. Every unsafe block needs a proof comment, every pointer deref needs bounds justification, and every error path needs coverage.

Learnt from: javier
Repo: questdb/questdb PR: 0
File: :0-0
Timestamp: 2026-02-06T18:22:28.130Z
Learning: In questdb/questdb, for PRs requested by javier, provide line-cited, push-back style reviews with concrete follow-ups (tests, diffs, and performance considerations).

Learnt from: javier
Repo: questdb/questdb PR: 0
File: :0-0
Timestamp: 2026-02-06T18:22:28.130Z
Learning: For questdb/questdb, javier expects a rigorous, push-back style review; always include concrete line-cited findings and propose targeted tests/fixes when reviewing javier’s PRs.

Learnt from: jerrinot
Repo: questdb/questdb PR: 0
File: :0-0
Timestamp: 2026-02-04T11:18:36.492Z
Learning: For PRs by jerrinot, always use a thorough and critical review mode: question design decisions, challenge assumptions, scrutinize implementations for edge cases and potential issues, and avoid superficial validation. They prefer hard-core reviews that push back on the code.

Learnt from: javier
Repo: questdb/questdb PR: 0
File: :0-0
Timestamp: 2026-02-06T18:22:28.130Z
Learning: In questdb/questdb, javier prefers hard‑core, critical reviews that challenge design choices and edge cases; respond with deep, push-back style analyses for javier’s requests.

@coderabbitai pls do a comprehensive review, appreciate your prompt response.

#!/bin/bash
# Fetch the actual PR diff to analyze specific changes
gh pr diff 6759 --repo questdb/questdb > /tmp/pr_6759_diff.txt
echo "=== Diff size ==="
wc -l /tmp/pr_6759_diff.txt

# Get list of all changed files
echo -e "\n=== Changed files ==="
gh pr diff 6759 --repo questdb/questdb --name-only

#!/bin/bash
# Examine the critical delta_bitpacked decoder with 0% coverage
echo "=== Delta Bitpacked Decoder (0% coverage) ==="
cat core/rust/qdbr/parquet2/src/encoding/delta_bitpacked/decoder.rs

#!/bin/bash
# Look at the main decode.rs file to understand null_count optimization
echo "=== Main decode.rs - null_count usage ==="
rg -n "null_count" core/rust/qdbr/src/parquet_read/decode.rs -B 5 -A 10

#!/bin/bash
# Examine unsafe code patterns in delta_binary_packed.rs
echo "=== Delta Binary Packed unsafe blocks ==="
cat core/rust/qdbr/src/parquet_read/decoders/delta_binary_packed.rs | head -500

#!/bin/bash
# Look at plain decoder implementation
echo "=== Plain Decoder Implementation ==="
cat core/rust/qdbr/src/parquet_read/decoders/plain.rs

#!/bin/bash
# Examine RLE dictionary decoder
echo "=== RLE Dictionary Decoder ==="
cat core/rust/qdbr/src/parquet_read/decoders/rle_dictionary.rs

#!/bin/bash
# Check what tests exist for decoders
echo "=== Decoder test files ==="
fd -e rs test core/rust/qdbr/src/parquet_read/decoders/
fd -e rs . core/rust/qdbr/src/parquet_read/decoders/ --exec basename {}

echo -e "\n=== Test modules in decoder files ==="
rg "#\[cfg\(test\)\]|mod tests" core/rust/qdbr/src/parquet_read/decoders/ -n

#!/bin/bash
# Look at the benchmark file to understand what's being measured
echo "=== Benchmark implementation ==="
cat core/rust/qdbr/benches/decode_page.rs

#!/bin/bash
# Check for the null sentinel bug fix mentioned in PR description
echo "=== Searching for null sentinel changes ==="
rg -n "null_value|sentinel|NULL_" core/rust/qdbr/src/parquet_read/decoders/ --type rust -C 3 | head -100

#!/bin/bash
# Look at slice_reader with low coverage
echo "=== Slice Reader (16.67% coverage) ==="
cat core/rust/qdbr/parquet2/src/read/page/slice_reader.rs

#!/bin/bash
# Check error handling patterns across decoders
echo "=== Error handling patterns ==="
rg -n "self\.error|self\.inner\.error" core/rust/qdbr/src/parquet_read/decoders/ --type rust -C 3 | head -150

#!/bin/bash
# Look for any existing fuzz tests or property-based tests
echo "=== Fuzz/Property tests ==="
fd -e rs . core/rust/qdbr/ | xargs rg "proptest|quickcheck|arbitrary|fuzz" -l

#!/bin/bash
# Examine the unpack lookup table for booleans
echo "=== Boolean unpack implementation ==="
cat core/rust/qdbr/src/parquet_read/decoders/unpack.rs

#!/bin/bash
# Check UUID byte swapping specialization
echo "=== UUID byte swapping ==="
rg -n "uuid|UUID" core/rust/qdbr/src/parquet_read/decoders/converters.rs -B 5 -A 10

#!/bin/bash
# Look at the actual diff for decode.rs to see the null_count optimization changes
echo "=== decode.rs changes in diff ==="
sed -n '/^diff.*decode\.rs/,/^diff/p' /tmp/pr_6759_diff.txt | head -800

@kafka1991, here's my comprehensive deep-dive review. I appreciate your patience.

🔴 CRITICAL: Zero Test Coverage on Batch Decoder

File: core/rust/qdbr/parquet2/src/encoding/delta_bitpacked/decoder.rs (0% coverage, 65 new lines)

Lines 99-147: decode_batch() method

This is your primary performance optimization path, yet it has zero test coverage. Critical issues:

Line 111: out[written + i] = self.min_delta + mb.next().unwrap_or_default() as i64;

• unwrap_or_default() silently returns 0 on miniblock exhaustion
• Is this semantically correct or hiding bugs? Delta encoding shouldn't have "default" values

Lines 113-126: Batch delta accumulation

let mut written = 0;
while written < count && self.values_remaining > 0 {
    if let Some(ref mut block) = self.current_block {
        let batch_count = (count - written).min(self.values_remaining);
        let mut delta_buf = [0i64; 128];

• Magic number 128: Why this size? Document relationship to miniblock/block structure
• Stack allocation of 128 i64s (1KB) - is this safe for deeply nested call stacks?
• What happens if values_remaining is corrupted in malicious files?

Lines 136-141: Wrapping arithmetic prefix sum

let mut acc = self.next_value;
for i in 0..decoded {
    acc = acc.wrapping_add(delta_buf[i]);
    out[written + i] = acc;
}

• Why is wrapping semantically correct? Delta encoding with zigzag should never overflow with valid inputs
• Document why overflow wrapping is acceptable, or prove it can't happen
• Wrapping hides potential corruption in malicious files

Lines 150-161: State invariant maintenance

if self.values_remaining > 0 {
    let delta = self.load_delta()?;
    self.next_value = self.next_value.wrapping_add(delta);
}

• CRITICAL: If load_delta() fails here (line 160), what state is decoder in?
• Comment claims "next_value must hold the next value to emit" but it won't after error
• Next decode_batch() call will emit stale/incorrect data

Required Actions:

1. Add unit tests for: empty blocks, single value, miniblock boundaries, max values, truncated data
2. Add property-based tests verifying: decode_batch() == repeated next()
3. Document or eliminate wrapping arithmetic
4. Fix error handling in invariant maintenance
5. Replace magic number 128 with named constant

🔴 CRITICAL: Trusting Untrusted Parquet Metadata

File: core/rust/qdbr/src/parquet_read/decode.rs

Lines 2100-2103: null_count optimization

let nc = page.header.null_count();
if nc == Some(0) {
    return Ok(None);
}

This is fundamentally broken for untrusted input. You're skipping definition level decoding based solely on parquet metadata. Consider:

• Malicious file: Claims null_count=0, actually has nulls → reads uninitialized memory or wrong data
• Corrupted file: Metadata corrupted → silent data corruption
• Buggy writer: Incorrect metadata → wrong results

Similar issue throughout decoders: Comments like "SAFETY: destination pointer stays in-bounds because decode paths reserve output upfront" (plain.rs:164-167, delta_binary_packed.rs:313-316) trust that:

1. Parquet value counts are accurate
2. Definition levels match value counts
3. Dictionary indices are in-bounds
4. Page sizes are correct

Every one of these assumptions can be violated by malicious/corrupt files.

Challenge: What's your threat model? If you accept only trusted parquet files, document it prominently. If you accept arbitrary files, every unsafe block needs validation.

Required Actions:

1. Document threat model (trusted vs untrusted sources)
2. If untrusted: Add validation before skipping decoding based on metadata
3. Add debug assertions checking all unsafe assumptions
4. Add fuzz tests with malformed/malicious parquet files

🔴 CRITICAL: Where's the Null Sentinel Bug Fix?

PR description states:

"Fixing bug with incorrect null sentinel values"

I cannot find this bug fix in any of the code changes. Searched for:

• Changes to null sentinel constants
• Changes to null value initialization
• Tests verifying null values
• Documentation of what was wrong

This is a data corruption bug. Where are:

1. The actual fix?
2. Regression tests demonstrating the bug?
3. Documentation of which null sentinels were wrong?
4. Verification that all null sentinels are now correct?

Required Actions:

1. Document what null values were incorrect
2. Add regression tests
3. Verify all null sentinel values across all types

🟡 HIGH: Silent Error Accumulation Anti-Pattern

File: core/rust/qdbr/src/parquet_read/decoders/rle_dictionary.rs

Lines 112-130: push() method

fn push(&mut self) -> ParquetResult<()> {
    if self.inner.error.is_some() {
        return Ok(());  // ❌ Returns success despite error!
    }
    // ...
    if idx >= self.dict.len() {
        self.inner.error = Some(fmt_err!(...));
        return Ok(());  // ❌ Silently fails, continues writing
    }
}

This pattern is dangerous:

1. Error occurs on row 1000 of 100K
2. Decoder returns Ok(()) for remaining 99K rows
3. Output buffer contains mixed valid/garbage data
4. Error only surfaces when someone calls result() - if they remember

The same pattern exists in:

• rle.rs lines 184-186
• plain.rs lines 86-94

Question: Why not fail-fast? The performance cost of checking error state is negligible compared to continuing to write potentially incorrect data.

Required Actions:

1. Either: Fail-fast on first error (recommended)
2. Or: Document that callers must check result() and how to handle partial data
3. Add tests verifying error propagation doesn't produce partial garbage output

🟡 HIGH: Pointer Invalidation Hazard

All decoder files using cached pointers (plain.rs:50, delta_binary_packed.rs:237, rle_dictionary.rs:101, etc.):

buffers_ptr: *mut T,

Refreshed only in reserve():

self.buffers_ptr = self.buffers.data_vec.as_mut_ptr().cast();

Problem: If ColumnChunkBuffers is ever reallocated elsewhere (e.g., another decoder operating on same buffer, external code), this cached pointer becomes dangling.

You're betting memory safety on:

1. No interleaved operations touching the same buffer
2. No external code reallocating the vec
3. Perfect discipline in calling reserve() before every write

Challenge: Why cache pointers instead of recalculating? The nanosecond saved isn't worth the memory safety risk.

Required Actions:

1. Add debug assertions validating cached pointer still matches vec pointer
2. Or: Remove caching, recalculate pointer each time (safe)
3. Run Miri on all tests to detect UB

🟡 MODERATE: Multiple unwrap() in Hot Paths

File: core/rust/qdbr/parquet2/src/encoding/delta_bitpacked/decoder.rs

• Line 86: bitpacked::Decoder::try_new(miniblock, num_bits, length).unwrap()
• Line 111: mb.next().unwrap_or_default()
• Line 144-145: .next().unwrap_or_default()
• Line 286: self.current_block.as_mut().unwrap()

For line 286: Comment says "At this point we must have at least one block" but what if:

• Corrupted total_count in header?
• Truncated page data?
• Bug in state machine?

Panics in production are unacceptable. Especially in parsers handling untrusted input.

Required Actions:

1. Replace unwrap() with proper error propagation
2. Or: Document invariants that make unwrap safe with debug assertions
3. Add tests triggering each unwrap path with invalid data

🟡 MODERATE: Test Coverage Insufficient for Parsers

Coverage Report:

• delta_bitpacked/decoder.rs: 0% (65 lines)
• slice_reader.rs: 16.67% (6 lines)
• row_groups.rs: 51.35%
• array.rs: 61.75%
• rle_dictionary.rs: 66.53% (but has extensive tests in reality)

For code handling untrusted input, <90% coverage is inadequate.

I found NO tests for:

• ❌ Malformed headers (truncated, oversized, invalid encodings)
• ❌ Mismatched metadata (null_count != actual nulls)
• ❌ Dictionary index out of bounds
• ❌ Invalid bitwidths (0, >64)
• ❌ Truncated page data mid-miniblock
• ❌ Empty pages, single-value pages
• ❌ Overflow scenarios in delta accumulation

Note: rle_dictionary.rs actually has excellent test coverage starting line 357 (I see 30+ test functions covering many edge cases). This is the quality level all decoders need.

Required Actions:

1. Achieve >90% coverage on all decoder files
2. Add adversarial/fuzz tests for malformed input
3. Property-based tests verifying decoder invariants
4. Integration tests with real-world malicious parquet files

🟢 GOOD: Boolean Lookup Table Optimization

File: core/rust/qdbr/src/parquet_read/decoders/plain.rs lines 29-44

const BOOLEAN_BITMAP_LUT: [[u8; 8]; 256] = {
    let mut lut = [[0u8; 8]; 256];
    let mut i = 0u16;
    while i < 256 {
        let b = i as u8;
        lut[i as usize] = [
            b & 1, (b >> 1) & 1, ...
        ];
        i += 1;
    }
    lut
};

This is excellent - compile-time lookup table generation. Explains the 97-98% performance improvement for boolean decoding.

🟢 GOOD: UUID Byte Swapping

File: core/rust/qdbr/src/parquet_read/decoders/converters.rs lines 130-145

Properly reverses byte order from Parquet big-endian to QuestDB little-endian. Explains 92.87% improvement for UUID decoding.

🟢 GOOD: RLE Dictionary Test Coverage

File: core/rust/qdbr/src/parquet_read/decoders/rle_dictionary.rs lines 357+

Exemplary test coverage: 30+ test functions covering:

• Basic operations (push, push_slice, skip)
• Null handling
• Zero bit width (single-value dictionary)
• Boundary conditions
• Mixed operations
• Out-of-bounds indices
• Large data sets
• Different bit widths
• Consistency checks

This is the standard all other decoders should meet.

📊 Benchmark Questions

1. "Apple M5" - This chip doesn't exist yet (current is M4). Do you mean M3/M4?

2. Jitter acknowledged but no statistical significance testing. Are the improvements real or noise?

3. Clickbench regressions:

   • q5: native 236ms → parquet 426ms (+80%)
   • q20-22: native 320-270ms → parquet 1200-2000ms (+275-640%)

   What's causing these regressions? Are they expected?

4. Plain encoding regressions for some types:

   • date/double/float/int: +2% to +9%

   Why are plain primitives slower?

Required Before Merge

Testing

1. ✅ Achieve >90% test coverage on all decoder files
2. ✅ Add comprehensive tests for delta_bitpacked batch decoder
3. ✅ Add fuzz tests for all decoders with malformed input
4. ✅ Property-based tests verifying decode invariants
5. ✅ Run Miri on all tests, fix all UB findings

Documentation

6. ✅ Document threat model (trusted vs untrusted input)
7. ✅ Document or fix null sentinel bug mentioned in PR description
8. ✅ Document why wrapping arithmetic is correct for delta encoding
9. ✅ Document silent error handling pattern or fix it
10. ✅ Add proof comments for all unsafe blocks

Code Quality

11. ✅ Replace unwrap() with proper error handling or document safety
12. ✅ Add validation for null_count optimization or document security trade-off
13. ✅ Fix state invariant bug in delta_bitpacked decode_batch error path
14. ✅ Add debug assertions for all pointer arithmetic
15. ✅ Explain Clickbench regressions (q5, q20-22)

The performance gains are impressive (median -87%), but this PR sacrifices correctness and safety. The delta_bitpacked batch decoder with 0% coverage is going into production handling potentially malicious input with extensive unsafe code. That's unacceptable.

What's your plan to address these concerns? Happy to discuss the architectural questions (error handling pattern, wrapping arithmetic, null_count optimization) if you have justifications I'm missing.

Learnt from: javier
Repo: questdb/questdb PR: 0
File: :0-0
Timestamp: 2026-02-06T18:22:28.130Z
Learning: In questdb/questdb, for PRs requested by javier, provide line-cited, push-back style reviews with concrete follow-ups (tests, diffs, and performance considerations).

Learnt from: javier
Repo: questdb/questdb PR: 0
File: :0-0
Timestamp: 2026-02-06T18:22:28.130Z
Learning: For questdb/questdb, javier expects a rigorous, push-back style review; always include concrete line-cited findings and propose targeted tests/fixes when reviewing javier’s PRs.

Learnt from: jerrinot
Repo: questdb/questdb PR: 0
File: :0-0
Timestamp: 2026-02-04T11:18:36.492Z
Learning: For PRs by jerrinot, always use a thorough and critical review mode: question design decisions, challenge assumptions, scrutinize implementations for edge cases and potential issues, and avoid superficial validation. They prefer hard-core reviews that push back on the code.

Learnt from: javier
Repo: questdb/questdb PR: 0
File: :0-0
Timestamp: 2026-02-06T18:22:28.130Z
Learning: In questdb/questdb, javier prefers hard‑core, critical reviews that challenge design choices and edge cases; respond with deep, push-back style analyses for javier’s requests.

@RaphDal — reviewed the Rust changes in this PR. Each finding was verified against the code on the PR branch.

Bugs

1. PlainVarSlicer::next()/skip() — unchecked read_unaligned (slicer/mod.rs)

next() reads a u32 length via ptr::read_unaligned(self.data.as_ptr().add(self.pos)) without checking self.pos + 4 <= self.data.len(). On a truncated Parquet file, self.pos can exceed the slice, causing UB before the subsequent slice indexing would panic. Same issue in skip().

Fix: bounds-check before the read, e.g.:

if self.pos + size_of::<u32>() > self.data.len() {
    // return error
}
let len = u32::from_le_bytes(self.data[self.pos..self.pos + 4].try_into().unwrap()) as usize;

2. PlainPrimitiveDecoder — no source buffer bounds checking (plain.rs)

new_with() converts the values slice to a raw *const T and discards the length. push() and push_slice() read via self.values.add(self.values_offset).read_unaligned() without verifying values_offset is in bounds. If page metadata inflates num_values beyond what the buffer contains, this causes OOB reads. (PlainBooleanDecoder is fine — it has can_read_bits().)

Fix: store the values length and validate up front or in push paths.

3. Copy-paste error message (page.rs)

split_buffer_v1, has_def branch, first error says "rep levels" instead of "def levels":

Error::oos("The number of bytes declared in v1 rep levels is higher than the page size")
//                                              ^^^ should be "def levels"

Improvements

4. BaseVarDictDecoder uses Vec instead of AcVec (dictionary.rs)

dict_values: Vec<&[u8]> bypasses QdbAllocator memory tracking. Same pattern in DecodeContext's decompress buffers.

5. RleDictionaryDecoder — uninitialized memory on error paths (rle_dictionary.rs)

reserve() calls set_len() without initializing new bytes. On sticky error, push()/push_slice() return early leaving bytes uninitialized. RleBooleanDecoder handles this by zeroing on error (ptr::write_bytes(out, 0, count)) — RleDictionaryDecoder should do the same.

6. Duplicated BOOLEAN_BITMAP_LUT (plain.rs, rle.rs)

Byte-identical 256-entry LUT in both files. Could be extracted to a shared module.

7. Test coverage gaps

Good combinatorial coverage for single-column, single-row-group files. Missing:

• Multi-row-group files
• Multi-column files
• All-nulls columns
• Zero-row files

@RaphDal The two unreachable!() calls in rle.rs (lines 336 and 373) on RleBooleanRun::None arms would panic if the decoder state machine ever reached them due to a bug or malformed parquet input. Would it be safer to return an error here instead, so that corrupt files produce a graceful error rather than a process-killing panic?

Thanks for the review !
500c2f8 addresses most of these feedbacks (mostly adding manual bound checks).
A few improvements I did not do:

• 4: This is unecessary, QdbAllocator doesn't brings any clear advantage. It may be useful for long lived allocations (e.g. RowGroupBuffers) but otherwise it's pure overhead.
• 5: If the Decoder leaves early then the error will be consumed and not the RowGroupBuffer.
• 7: There are existing Java tests to cover these cases, the new tests are here to validate the proper behavior of decoders.

@RaphDal The two unreachable!() calls in rle.rs (lines 336 and 373) on RleBooleanRun::None arms would panic if the decoder state machine ever reached them due to a bug or malformed parquet input. Would it be safer to return an error here instead, so that corrupt files produce a graceful error rather than a process-killing panic?

These branches should truly unreachable!(), run is set to RleBooleanRun::None when ensure_run` fails or there are no remaining runs to decode. Both cases prevent the match block from being evaluated:

if !self.ensure_run() { // 1st case - returns if it fails
    return;
}

let run_available = self.run.remaining().min(self.remaining_values);
if run_available == 0 { // 2nd case - iterate again if it fails
    self.run = RleBooleanRun::None;
    continue;
}
let take = count.min(run_available);

match &mut self.run {
    RleBooleanRun::Rle { remaining, .. } => {
        // Repeated run skip is just counter arithmetic.
        *remaining -= take;
    }
    RleBooleanRun::Bitpacked { byte_offset, bit_offset, remaining, .. } => {
        // Bitpacked run skip advances bit/byte cursors.
        Self::skip_bitpacked(byte_offset, bit_offset, take);
        *remaining -= take;
    }
    RleBooleanRun::None => unreachable!(),
}

Unless I'm mistaken, returning an error would only bring unnecessary overhead to the caller.

[PR Coverage check]

😍 pass : 3364 / 4476 (75.16%)

file detail