Thanks to visit codestin.com
Credit goes to github.com

Skip to content

[OPIK-3846] [BE] Optimize SELECT_DATASET_ITEM_VERSIONS_WITH_EXPERIMENT_ITEMS query with CTEs #5052

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
thiagohora merged 14 commits into from
Feb 4, 2026
Merged

[OPIK-3846] [BE] Optimize SELECT_DATASET_ITEM_VERSIONS_WITH_EXPERIMENT_ITEMS query with CTEs #5052

thiagohora merged 14 commits into from
Feb 4, 2026

Conversation

@thiagohora
Copy link
Contributor

@thiagohora thiagohora commented Feb 3, 2026 •
edited
Loading

Details

  • Optimize SELECT_DATASET_ITEM_VERSIONS_WITH_EXPERIMENT_ITEMS query in DatasetItemVersionDAO.java using CTEs
  • Add CTEs for trace data pre-computation to reduce I/O on large datasets
  • Add tests for the optimized query

Optimization Strategy

The optimized query introduces CTEs to:

  1. experiment_items_trace_scope: Pre-compute distinct trace_ids from experiment items
  2. target_projects: Pre-compute distinct project_ids from traces (for potential index usage)
  3. trace_data: Pre-compute trace data with deduplication, reused in the final JOIN

Benchmark Results

Preprod Environment (Large Dataset)

Metric ORIGINAL OPTIMIZED Change
Duration 3,899 ms 4,779 ms +22% slower
Rows Read 53.46 million 28.31 million -47% fewer
Bytes Read 10.04 GiB 4.18 GiB -58% less
Memory Usage 4.65 GiB 5.84 GiB +26% more
Traces Table Scans 1 4 +3 more

Local Environment (Small Dataset)

Metric ORIGINAL OPTIMIZED Change
Duration 29 ms 16 ms -45% faster
Rows Read 8,132 8,132 Same
Memory Usage 102 MB 51 MB -50% less
Traces Table Scans 1 1 Same

Analysis

Preprod (Large Dataset):

  • The optimized query reads significantly fewer rows and bytes (-47% rows, -58% bytes)
  • However, additional CTE overhead and multiple traces table scans result in slower execution (+22%)

Local (Small Dataset):

  • The optimized query performs significantly better (-45% faster)
  • Memory usage reduced by half (-50%)

Change checklist

  • User facing
  • Documentation update

Issues

  • OPIK-3846

Testing

  • Added tests in DatasetsResourceTest.java
  • Benchmark scripts: benchmark_local.sql, benchmark_search.sql
  • Tested on both local and preprod environments

Documentation

  • Performance benchmark results documented above

@thiagohora
[OPIK-3846] [BE] Optimize experiment items stats query
0935cc3 
Replace JOINs with IN subqueries in SELECT_DATASET_ITEM_VERSIONS_WITH_EXPERIMENT_ITEMS_STATS
for better ClickHouse query performance:

- Add early trace_data CTE to resolve trace IDs and project_ids upfront
- Replace INNER JOIN experiments_resolved with IN subquery for experiment_items_scope
- Filter feedback_scores and spans by project_id IN (SELECT project_id FROM trace_data)
- Use trace_id IN subquery instead of full workspace scan on traces table

Performance improvements (benchmark with cache disabled):
- 48% faster average execution (219ms vs 420ms)
- 96% more consistent (21ms vs 479ms std dev)
- 83% better worst-case (271ms vs 1567ms max)
- 50% fewer table scans on traces and spans tables
- 38% fewer ReadFromMergeTree operations
- 42% fewer JoiningTransform operations

Also fixes test by setting correct projectName on feedback scores.
@thiagohora
Merge branch 'main' into thiaghora/OPIK-3846-optimize-experiment-item…
ac7a608 
…s-stats-query
@thiagohora
[OPIK-3846] [BE] Optimize experiment items stats query - add duration…
49269bd 
… column

- Add experiments_resolved CTE for early experiment resolution
- Add target_projects CTE for project_id index filtering
- Add experiment_items_trace_scope CTE for trace_id filtering
- Filter feedback_scores and spans by project_id and entity_id/trace_id
- Add duration materialized column to traces and spans tables
- Fix feedback_scores_percentiles WHERE clause for correct filtering

Performance improvement on 1M items:
- 3.72x faster (2.91s vs 10.92s)
- 56% fewer rows read (30.84M vs 69.49M)
- 57% less data read (4.58 GiB vs 10.74 GiB)
@thiagohora
Fix issue
6f3d753
@thiagohora
[OPIK-3846] [BE] Optimize SELECT_DATASET_ITEM_VERSIONS_WITH_EXPERIMEN…
c077e94 
…T_ITEMS_COUNT query

Add project_id filtering to feedback_scores tables to leverage ClickHouse index structure.
@thiagohora
[OPIK-3846] [BE] Optimize SELECT_DATASET_ITEM_VERSIONS_WITH_EXPERIMEN…
1f06804 
…T_ITEMS query with CTEs

Add CTEs for trace data pre-computation to reduce I/O on large datasets:
- experiment_items_trace_scope: Pre-compute distinct trace_ids
- target_projects: Pre-compute project_ids for index usage
- trace_data: Pre-compute trace data with deduplication
@thiagohora thiagohora requested a review from a team as a code owner February 3, 2026 23:32
@github-actions github-actions bot added java Pull requests that update Java code Backend tests Including test files, or tests related like configuration. labels Feb 3, 2026
@comet-ml comet-ml deleted a comment from github-actions bot Feb 3, 2026
baz-reviewer[bot]
baz-reviewer bot reviewed Feb 3, 2026
View reviewed changes
@thiagohora thiagohora changed the base branch from main to thiaghora/OPIK-3846-optimize-experiment-items-count-query February 3, 2026 23:53
@github-actions
Copy link
Contributor

github-actions bot commented Feb 4, 2026 •
edited
Loading

Backend Tests Results

  424 files  ± 0    424 suites  ±0   51m 27s ⏱️ - 11m 39s
6 638 tests + 3  6 625 ✅ + 3  13 💤 ±0  0 ❌ ±0 
6 517 runs   - 80  6 504 ✅  - 80  13 💤 ±0  0 ❌ ±0 

Results for commit f54cb3f. ± Comparison against base commit 1b847e7.

♻️ This comment has been updated with latest results.

ldaugusto
ldaugusto previously approved these changes Feb 4, 2026
View reviewed changes
Copy link
Contributor

@ldaugusto ldaugusto left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Assuming #5054 fixes the performance, there are only small things to discuss.

@thiagohora thiagohora force-pushed the thiaghora/OPIK-3846-optimize-experiment-items-count-query branch from 57165f5 to a36b9bb Compare February 4, 2026 15:02
@ldaugusto ldaugusto mentioned this pull request Feb 4, 2026
Merged
2 tasks
@thiagohora
Merge branch 'thiaghora/OPIK-3846-optimize-experiment-items-count-que…
a0e69d9 
…ry' into thiaghora/OPIK-3846-optimize-experiment-items-count-query-v2
@github-actions
Copy link
Contributor

github-actions bot commented Feb 4, 2026 •
edited
Loading

SDK E2E Tests Results

0 tests  ±0   0 ✅ ±0   0s ⏱️ ±0s
0 suites ±0   0 💤 ±0 
0 files   ±0   0 ❌ ±0 

Results for commit a0e69d9. ± Comparison against base commit c4d3b2d.

♻️ This comment has been updated with latest results.

@thiagohora thiagohora force-pushed the thiaghora/OPIK-3846-optimize-experiment-items-count-query branch from c4d3b2d to 14c9aee Compare February 4, 2026 17:08
@thiagohora thiagohora requested review from a team as code owners February 4, 2026 17:08
@thiagohora thiagohora requested review from a team as code owners February 4, 2026 17:08
Base automatically changed from thiaghora/OPIK-3846-optimize-experiment-items-count-query to main February 4, 2026 18:00
@thiagohora thiagohora dismissed ldaugusto’s stale review February 4, 2026 18:00

The base branch was changed.

@thiagohora thiagohora merged commit b615917 into main Feb 4, 2026
19 checks passed
@thiagohora thiagohora deleted the thiaghora/OPIK-3846-optimize-experiment-items-count-query-v2 branch February 4, 2026 18:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@ldaugusto ldaugusto ldaugusto approved these changes

+1 more reviewer

@baz-reviewer baz-reviewer[bot] baz-reviewer[bot] approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

@thiagohora thiagohora

Labels

Backend java Pull requests that update Java code tests Including test files, or tests related like configuration.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@thiagohora @ldaugusto