Thanks to visit codestin.com
Credit goes to github.com

Skip to content

feat(query): push Row Access Policy predicates into prewhere for storage-level pruning#19875

Merged
zhang2014 merged 4 commits into
databendlabs:mainfrom
TCeason:rap_fold
May 20, 2026
Merged

feat(query): push Row Access Policy predicates into prewhere for storage-level pruning#19875
zhang2014 merged 4 commits into
databendlabs:mainfrom
TCeason:rap_fold

Conversation

@TCeason
Copy link
Copy Markdown
Collaborator

@TCeason TCeason commented May 18, 2026

I hereby agree to the terms of the CLA available at: https://docs.databend.com/dev/policies/cla/

Summary

Enable Row Access Policy (RAP) predicates to participate in storage-level pushdown optimizations (range pruning, bloom pruning, prewhere row selection) instead of being enforced only by a post-scan Filter [SECURE] operator.

Previously, secure predicates were merged into a single combined filters field in PushDownInfo, which made them invisible in EXPLAIN and prevented fine-grained control. This PR separates them into a dedicated secure_filters field, uses effective_filters() to combine them at the storage layer for pruning, and introduces has_secure_predicates_not_applied_by_prewhere() as the safety gate for limit/sort pushdown.

Changes

Optimizer

  • RulePushDownPrewhere now matches bare Scan nodes (not just Filter -> Scan) and pushes secure predicates into prewhere via secure_prewhere_optimize().
  • Limit/sort pushdown rules (RulePushDownLimitScan, RulePushDownSortScan, RulePushDownLimitFilterScan, RulePushDownSortFilterScan) use has_secure_predicates_not_applied_by_prewhere() instead of blanket bail-out.

Physical plan

  • PushDownInfo gains a secure_filters: Option<Filters> field, kept separate from user filters for display redaction.
  • effective_filters() combines user + secure filters with AND for storage-layer consumption (range pruner, bloom pruner).
  • is_deterministic is now derived from individual user/secure computations instead of a redundant combined call.

Storage

  • FusePruner and read_partitions use effective_filters() for range/bloom pruning, ensuring RAP expressions participate in block-level index pruning.
  • TopN/limit pruning guards check secure_filters.is_none() to maintain correctness (same behavior as with regular WHERE filters).

Display

  • EXPLAIN shows secure_filters: REDACTED to indicate RAP participation without leaking policy expressions.
  • Filter [SECURE] pipeline operator is retained as the final enforcement layer.

Other

  • ResultCacheWriter::append_block calls maybe_gc() to reclaim memory from filtered blocks before caching.

Tests

  • Unit Test
  • Logic Test
  • Benchmark Test
  • No Test - Explain why

Type of change

  • Bug Fix (non-breaking change which fixes an issue)
  • New Feature (non-breaking change which adds functionality)
  • Breaking Change (fix or feature that could cause existing functionality not to work as expected)
  • Documentation Update
  • Refactoring
  • Performance Improvement
  • Other (please describe):

This change is Reviewable

@TCeason TCeason marked this pull request as draft May 18, 2026 06:37
@github-actions github-actions Bot added the pr-refactor this PR changes the code base without new features or bugfix label May 18, 2026
Copy link
Copy Markdown

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: d0fec6b30b

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread src/query/service/src/physical_plans/physical_table_scan.rs Outdated
@TCeason TCeason force-pushed the rap_fold branch 6 times, most recently from 94f38be to 02f6ab1 Compare May 19, 2026 08:52
@TCeason TCeason changed the title refactor(query): indicate secure filter presence in EXPLAIN output feat(query): push Row Access Policy predicates into prewhere for storage-level pruning May 19, 2026
@github-actions github-actions Bot added the pr-feature this PR introduces a new feature to the codebase label May 19, 2026
@TCeason TCeason marked this pull request as ready for review May 19, 2026 08:59
@b41sh b41sh self-requested a review May 19, 2026 09:06
TCeason added 4 commits May 20, 2026 09:03
  When row access policy predicates are pushed into storage-level
  filtering, EXPLAIN output now shows "secure_filters: REDACTED" to
  indicate their participation without leaking the policy expression.

  Previously the secure predicates were completely hidden from EXPLAIN,
  making it impossible to tell whether RAP was participating in pushdown.
Copy link
Copy Markdown
Member

@zhang2014 zhang2014 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Relevant tests need to be added to prevent regressions.

@TCeason TCeason enabled auto-merge May 20, 2026 02:47
@TCeason TCeason added this pull request to the merge queue May 20, 2026
@zhang2014 zhang2014 removed this pull request from the merge queue due to a manual request May 20, 2026
@zhang2014 zhang2014 merged commit 9440e16 into databendlabs:main May 20, 2026
178 of 180 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

pr-feature this PR introduces a new feature to the codebase pr-refactor this PR changes the code base without new features or bugfix

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants