Details

This PR optimizes the SELECT_DATASET_ITEM_VERSIONS_WITH_EXPERIMENT_ITEMS_COUNT query in DatasetItemVersionDAO.java by adding project_id filtering to feedback_scores tables to leverage ClickHouse's index structure (workspace_id, project_id, ...).

Key Optimizations

1. Pre-computed experiment_items_trace_scope CTE - Pre-computes DISTINCT trace_ids for efficient filtering
2. Pre-computed target_projects CTE - Pre-computes DISTINCT project_ids from traces
3. Filtered Feedback Scores Queries - Added project_id IN (SELECT project_id FROM target_projects) filtering

Before vs After

Before:

FROM feedback_scores FINAL
WHERE entity_type = 'trace'
  AND workspace_id = :workspace_id
  AND entity_id IN (SELECT trace_id FROM experiment_items_scope)

After:

FROM feedback_scores FINAL
WHERE entity_type = 'trace'
  AND workspace_id = :workspace_id
  AND project_id IN (SELECT project_id FROM target_projects)
  AND entity_id IN (SELECT trace_id FROM experiment_items_trace_scope)

When This Optimization Helps

The optimization is defensive and will show significant improvement when:

• Workspaces have many projects where experiment traces belong to a small subset
• The target project contains a small fraction of total feedback_scores
• At scale, the project_id filtering leverages ClickHouse's index structure

Benchmark Results

Remote Server (1M records)

Note: The benchmark shows minimal improvement because the test dataset's experiment traces belong to the largest project (67.78% of all feedback_scores data). The optimization will show significant gains when traces belong to smaller projects.

Local Server (~2K records)

EXPLAIN Index Analysis

Remote Server Index Usage (Both Queries - Identical)

EXPLAIN indexes=1 Output (Remote)

ReadFromMergeTree (opik.experiment_items)
Indexes:
  PrimaryKey
    Keys: workspace_id
    Condition: (workspace_id in ['7596558c-...'])
    Parts: 6/8
    Granules: 831/1041

ReadFromMergeTree (opik.experiments)
Indexes:
  PrimaryKey
    Keys: workspace_id, dataset_id, id
    Condition: and((id in 1-element set), and((dataset_id in [...]), (workspace_id in [...])))
    Parts: 1/1
    Granules: 1/1

ReadFromMergeTree (opik.dataset_item_versions)
Indexes:
  PrimaryKey
    Keys: workspace_id, dataset_id
    Condition: and((dataset_id in [...]), (workspace_id in [...]))
    Parts: 1/6
    Granules: 128/11348

ReadFromMergeTree (opik.traces)
Indexes:
  PrimaryKey
    Keys: workspace_id, id
    Condition: and((id in 1000000-element set), (workspace_id in [...]))
    Parts: 4/7
    Granules: 149/4721

Local Server Index Usage (Both Queries - Identical)

EXPLAIN PIPELINE Analysis

Both queries produce identical execution pipelines with 32 parallel workers (remote) / 12 parallel workers (local):

(Expression)
ExpressionTransform × 32
  (Aggregating)
  Resize 32 → 32
    AggregatingTransform × 32
      (Expression)
      ExpressionTransform × 32
        (Join)
        JoiningTransform × 32 2 → 1
          (Expression)
          ExpressionTransform × 32
            (Join)
            JoiningTransform × 32 2 → 1
              (Expression)
              ExpressionTransform
                (Filter)
                FilterTransform
                  (LimitBy)
                  LimitByTransform
                    (Sorting)
                    MergingSortedTransform 32 → 1
                      MergeSortingTransform × 32
                        (ReadFromMergeTree)
                        MergeTreeSelect(pool: ReadPool, algorithm: Thread) × 32

Key pipeline stages:

• MergeTreeSelect: Parallel reads from ClickHouse tables
• JoiningTransform: Hash joins between CTEs
• LimitByTransform: Deduplication via LIMIT 1 BY
• AggregatingTransform: COUNT(DISTINCT) aggregation

Data Distribution Analysis

Feedback Scores per Project (Remote)

Key insight: All 1,000,000 experiment traces belong to the largest project (67.78% of data), so project_id filtering can only skip 32% of rows.

Change checklist

• User facing
• Documentation update

Issues

• OPIK-3846

Testing

• All 51 DatasetItemVersionDAO tests pass
• Benchmarked on remote ClickHouse server with 1M records
• Benchmarked on local ClickHouse with ~2K records
• Verified both queries return identical results
• EXPLAIN indexes and EXPLAIN PIPELINE analyzed

Documentation

• Query optimization follows ClickHouse index best practices
• Aligns with existing STATS query optimization pattern