Thanks to visit codestin.com
Credit goes to github.com

Skip to content

Expose batch progress traces on the metrics route #5956

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Draft
wants to merge 7 commits into
base: main
Choose a base branch
from
Draft

Expose batch progress traces on the metrics route #5956

wants to merge 7 commits into from
+79 −3

Conversation

@Kerollmops
Copy link
Member

@Kerollmops Kerollmops commented Oct 24, 2025 •
edited
Loading

This PR exposes more information to the metrics route, helping us debug what actually takes time when indexing documents. It can also be helpful to identify which part of the code is being executed by comparing it with other CPU/Memory/Disk graphs.

I published a prototype so that we can test on staging and expose a good Grafana visualization.

Capture d’écran 2025-11-03 à 17 09 27

To do

  • Update utoipa to feature some examples.
  • Create a linear issue to track this (internal).
  • (optional) Get rid of humantime and use a dependency we already depend on.
  • Inform doc about the new metrics.
  • Inform cloud about those new metrics, too.
  • Switch to a boolean completion ratio gauges for the currently processing step (and not the whole history).
  • Retrieve only the timings for the last processed batch (not the processing one).
  • On Grafana, compute a mean time by doing a distinct of the different values.
  • Share a screenshot of a Grafana dashboard using the live progress trace.

The batch running progress trace

# HELP meilisearch_batch_running_progress_trace The currently running progress trace
# TYPE meilisearch_batch_running_progress_trace gauge
meilisearch_batch_running_progress_trace{batch_uid="0",step_name="document"} 0.710618582519409
meilisearch_batch_running_progress_trace{batch_uid="0",step_name="extracting word proximity"} 0.2222222222222222
meilisearch_batch_running_progress_trace{batch_uid="0",step_name="indexing"} 0.6666666666666666
meilisearch_batch_running_progress_trace{batch_uid="0",step_name="processing tasks"} 0

The last batch progress trace metric

# HELP meilisearch_last_batch_progress_trace_ms The last batch progress trace in milliseconds
# TYPE meilisearch_last_batch_progress_trace_ms gauge
meilisearch_last_batch_progress_trace_ms{batch_uid="0",step_name="processing tasks"} 20170
meilisearch_last_batch_progress_trace_ms{batch_uid="0",step_name="processing tasks > computing document changes"} 383
meilisearch_last_batch_progress_trace_ms{batch_uid="0",step_name="processing tasks > computing document changes > preparing payloads"} 382
meilisearch_last_batch_progress_trace_ms{batch_uid="0",step_name="processing tasks > computing document changes > preparing payloads > payload"} 382
meilisearch_last_batch_progress_trace_ms{batch_uid="0",step_name="processing tasks > indexing"} 19760
meilisearch_last_batch_progress_trace_ms{batch_uid="0",step_name="processing tasks > indexing > building geo json"} 2
meilisearch_last_batch_progress_trace_ms{batch_uid="0",step_name="processing tasks > indexing > building geo json > clear updated items"} 0
meilisearch_last_batch_progress_trace_ms{batch_uid="0",step_name="processing tasks > indexing > building geo json > retrieve updated items"} 0
meilisearch_last_batch_progress_trace_ms{batch_uid="0",step_name="processing tasks > indexing > building geo json > retrieve updated items > item"} 0
meilisearch_last_batch_progress_trace_ms{batch_uid="0",step_name="processing tasks > indexing > extracting documents"} 207
meilisearch_last_batch_progress_trace_ms{batch_uid="0",step_name="processing tasks > indexing > extracting documents > document"} 207
meilisearch_last_batch_progress_trace_ms{batch_uid="0",step_name="processing tasks > indexing > extracting facets"} 122
meilisearch_last_batch_progress_trace_ms{batch_uid="0",step_name="processing tasks > indexing > extracting facets > document"} 122
meilisearch_last_batch_progress_trace_ms{batch_uid="0",step_name="processing tasks > indexing > extracting word proximity"} 6990
meilisearch_last_batch_progress_trace_ms{batch_uid="0",step_name="processing tasks > indexing > extracting word proximity > document"} 6990
meilisearch_last_batch_progress_trace_ms{batch_uid="0",step_name="processing tasks > indexing > extracting words"} 5290
meilisearch_last_batch_progress_trace_ms{batch_uid="0",step_name="processing tasks > indexing > extracting words > document"} 5290
meilisearch_last_batch_progress_trace_ms{batch_uid="0",step_name="processing tasks > indexing > finalizing"} 155
meilisearch_last_batch_progress_trace_ms{batch_uid="0",step_name="processing tasks > indexing > finalizing > committing"} 141
meilisearch_last_batch_progress_trace_ms{batch_uid="0",step_name="processing tasks > indexing > finalizing > computing stats"} 5
meilisearch_last_batch_progress_trace_ms{batch_uid="0",step_name="processing tasks > indexing > merging facet caches"} 3
meilisearch_last_batch_progress_trace_ms{batch_uid="0",step_name="processing tasks > indexing > merging word caches"} 1400
meilisearch_last_batch_progress_trace_ms{batch_uid="0",step_name="processing tasks > indexing > merging word caches > exact word docids"} 0
meilisearch_last_batch_progress_trace_ms{batch_uid="0",step_name="processing tasks > indexing > merging word caches > field id word count docids"} 13
meilisearch_last_batch_progress_trace_ms{batch_uid="0",step_name="processing tasks > indexing > merging word caches > word docids"} 338
meilisearch_last_batch_progress_trace_ms{batch_uid="0",step_name="processing tasks > indexing > merging word caches > word field id docids"} 368
meilisearch_last_batch_progress_trace_ms{batch_uid="0",step_name="processing tasks > indexing > merging word caches > word position docids"} 682
meilisearch_last_batch_progress_trace_ms{batch_uid="0",step_name="processing tasks > indexing > merging word proximity"} 4210
meilisearch_last_batch_progress_trace_ms{batch_uid="0",step_name="processing tasks > indexing > post processing facets"} 2
meilisearch_last_batch_progress_trace_ms{batch_uid="0",step_name="processing tasks > indexing > post processing facets > facet search"} 1
meilisearch_last_batch_progress_trace_ms{batch_uid="0",step_name="processing tasks > indexing > post processing words"} 1200
meilisearch_last_batch_progress_trace_ms{batch_uid="0",step_name="processing tasks > indexing > post processing words > exact word prefix docids"} 0
meilisearch_last_batch_progress_trace_ms{batch_uid="0",step_name="processing tasks > indexing > post processing words > word fst"} 773
meilisearch_last_batch_progress_trace_ms{batch_uid="0",step_name="processing tasks > indexing > post processing words > word prefix docids"} 70
meilisearch_last_batch_progress_trace_ms{batch_uid="0",step_name="processing tasks > indexing > post processing words > word prefix field id docids"} 104
meilisearch_last_batch_progress_trace_ms{batch_uid="0",step_name="processing tasks > indexing > post processing words > word prefix position docids"} 254
meilisearch_last_batch_progress_trace_ms{batch_uid="0",step_name="processing tasks > indexing > waiting for database writes"} 128
meilisearch_last_batch_progress_trace_ms{batch_uid="0",step_name="processing tasks > indexing > waiting for extractors"} 0
meilisearch_last_batch_progress_trace_ms{batch_uid="0",step_name="processing tasks > indexing > writing embeddings to database"} 1
meilisearch_last_batch_progress_trace_ms{batch_uid="0",step_name="processing tasks > retrieving config"} 0
meilisearch_last_batch_progress_trace_ms{batch_uid="0",step_name="writing tasks to disk"} 0
meilisearch_last_batch_progress_trace_ms{batch_uid="0",step_name="writing tasks to disk > task"} 0

@Kerollmops Kerollmops added the no db change The database didn't change label Oct 24, 2025
@Kerollmops Kerollmops force-pushed the progress-trace-in-metrics branch from efab87a to a8d5556 Compare November 3, 2025 15:01
@Kerollmops Kerollmops marked this pull request as ready for review November 5, 2025 08:54
@Kerollmops Kerollmops marked this pull request as draft November 5, 2025 11:02
@Kerollmops Kerollmops requested a review from paulden November 5, 2025 11:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@ManyTheFish ManyTheFish Awaiting requested review from ManyTheFish

@paulden paulden Awaiting requested review from paulden

At least 1 approving review is required to merge this pull request.

Assignees

No one assigned

Labels

no db change The database didn't change

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@Kerollmops