Experimental feature: Prometheus Metrics #625

irevoire · 2023-02-21T10:05:51Z

irevoire
Feb 21, 2023

Hey everyone 👋

For the v1.1 of Meilisearch (which was released on 3rd April), we will finally introduce the metrics feature as an experimental feature.

Feature Abstract

The /metrics endpoint exposes metrics to be scraped by a Prometheus collector at regular intervals and stored for analysis. By monitoring these metrics over time, teams can gain valuable insights into the behavior and performance of Meilisearch and use that information to optimize their performance, troubleshoot issues, and improve the overall reliability of their systems.

How to enable the feature in Meilisearch?

2 ways

Launch Meilisearch with the CLI flag --experimental-enable-metrics or the MEILI_EXPERIMENTAL_ENABLE_METRICS env var
Since v1.5.0, and not before, call this HTTP route:

curl \
  -X PATCH 'http://localhost:7700/experimental-features/' \
  -H 'Content-Type: application/json'  \
--data-binary '{
    "metrics": true
  }'

You can check the feature is well enabled by calling GET /experimental-features

⚠️ Do NOT use the CLI flag (or environment variable) and the HTTP call at the same time to enable the feature: this could lead to unexpected behavior when restarting the Meilisearch instance.
⚠️ An API key that has the right to see the metrics is able to see all the HTTP calls that happened for all the routes that have been hit on the instance. This may leak all the index names, some document's primary keys, or even worse, the other API keys.
=> So, be sure to consider the key that can access the API-key as a super private API-key that you must not share publicly.

How to use the feature?

The metrics are sent using the prometheus format
This exposes a new route in the API endpoint GET /metrics
- To poll this route, your API key needs the right to pull metrics through the metrics.get action and access to all indexes.
- The scrape_config must contain an Authorization header using the bearer method with the Meilisearch API Key authorized to poll the /metrics endpoint. You can find a basic config in the repository under assets/prometheus-basic-scraper.yml.

Step-by-step basic tutorial

To read before starting the tutorial:

You need to install Prometheus and Grafana. Refer to the documentation of your distribution / OS.
I won't explain how to secure your instance. Refer to this documentation if you need to learn how it works, and adapt the following command to include your master-key / api-key.

Run meilisearch with the experimental feature enabled: ./meilisearch --experimental-enable-metrics.
Download the Prometheus assets file stored on the meilisearch repository assets/prometheus-basic-scraper.yml.
If you need to use an API key, add the following lines to the end of the config file (and replace with your API key):

authorization:
	credentials: <api-key>

On another terminal or server, run Prometheus with your config file: ./prometheus --config.file=path-to-your-config-file.yml
Download the default Grafana dashboard on the following URL: https://github.com/meilisearch/meilisearch/blob/main/assets/grafana-dashboard.json.
Connect to your Grafana instance (by default, it runs on http://localhost:3000). Create an account if needed.
Add a data source that points to your Prometheus server (by default, it runs on http://localhost:9090).
Go into the dashboard menu and import a dashboard file.
Import the json previously downloaded. You should be good to go 🎉

What's currently exposed?

Name	Description	Type
`meilisearch_http_requests_total`	Returns the number of times an API resource is accessed.	counter
`meilisearch_http_response_time_seconds`	Returns a time histogram showing the number of times an API resource call goes into a time bucket (expressed in second).	histogram
`meilisearch_db_size_bytes`	Returns the “real” size of the database on disk in bytes. It includes all the lmdb memory mapped files plus all the files contained in the `data.ms` directory (mainly the updates files that were not processed yet).	gauge
`meilisearch_used_db_size_bytes`	Returns the size of the database actually used by meilisearch in bytes. Include all the same files as `meilisearch_db_size_bytes` except that when it comes to an LMDB database, we only count the pages used by meilisearch. This means if you see a large gap between both metrics, adding documents will probably re-use freed pages instead of growing `meilisearch_db_size_bytes`.	gauge
`meilisearch_index_docs_count`	Returns the number of documents for an index.	gauge
`meilisearch_index_count`	Returns the total number of index for the Meilisearch instance.	gauge
`meilisearch_nb_tasks`	Returns the total number of tasks for the Meilisearch instance parametrized by the kind of task and its value (see the table below).	counter
`meilisearch_last_update`	Returns the timestamp of the last update.	gauge
`meilisearch_is_indexing`	Returns `1` if Meilisearch is indexing or `0` if not.	gauge

What is an experimental feature

The API and the behavior of an experimental feature can break between two minor versions of Meilisearch.
The whole experimental feature process can break between 2 minor versions of Meilisearch.
The newly introduced experimental features or improvements can be removed in the future. Providing a feature as “experimental” does not guarantee it will be stable one day.
The experimental features are supposed to be unstable regarding the API usage but not bugged.

Why is this feature not stable yet?

We have yet to determine which metrics we want to expose and how. Thus this may change a lot depending on our needs and your needs.

When will the feature potentially be stable?

We don't know yet when this feature could be stable since we need to iterate on it. As explained previously, we do not guarantee it will be stable one day yet.

Depending on the feedback we get, we plan to iterate on the feature every 1 or 2 releases (Currently, our release cycle is two months long).

X metrics are missing

We know the feature is not finished yet.

Please let us comment on which information you would like to see integrated into the metrics.
And do not hesitate to create a PR as well. Most of the code is available in meilisearch/src/metrics.rs.

🗣️ You are welcome to give feedback about the feature or ask any questions; we can guide you through this new feature usage!

cc @meilisearch/cloud-team

gmourier · 2023-03-08T07:51:56Z

gmourier
Mar 8, 2023
Maintainer

Hello, everyone following this discussion 👋

We have just released the first RC (release candidate) of Meilisearch containing this experimental feature!

You can test it by using

the release assets
the Meilisearch Docker image

docker run -it --rm -p 7700:7700 -v $(pwd)/meili_data:/meili_data getmeili/meilisearch:v1.1.0-rc.0

You are welcome to communicate any feedback about this new implementation in this discussion.

If you encounter any bugs, please report them here.
Thanks in advance for your help and your involvement in Meilisearch ❤️

🎉 Official and stable release containing this change will be available on 3rd April 2023

⚠️ RC (release candidates) are not recommended for production

0 replies

gmourier · 2023-04-11T07:33:02Z

gmourier
Apr 11, 2023
Maintainer

Hey folks 👋

v1.1 has been released and features the Prometheus metrics as an experimental one 🧪

We are waiting for tests and feedback before considering any stabilization. Please, tell us if this feature is helpful or what is missing for your monitoring use-case!

0 replies

hspak · 2023-04-14T19:46:41Z

hspak
Apr 14, 2023

Thank you for building this! I had a question on how the histogram bins are set for the http_response_time_seconds metrics. It would be nice if we had more bins over 1 second.

Example

http_response_time_seconds_bucket{method="PUT",path="/indexes/myindex/documents",le="0.0005"} 0
http_response_time_seconds_bucket{method="PUT",path="/indexes/myindex/documents",le="0.0008"} 0
http_response_time_seconds_bucket{method="PUT",path="/indexes/myindex/documents",le="0.00085"} 0
http_response_time_seconds_bucket{method="PUT",path="/indexes/myindex/documents",le="0.0009"} 0
http_response_time_seconds_bucket{method="PUT",path="/indexes/myindex/documents",le="0.00095"} 0
http_response_time_seconds_bucket{method="PUT",path="/indexes/myindex/documents",le="0.001"} 0
http_response_time_seconds_bucket{method="PUT",path="/indexes/myindex/documents",le="0.00105"} 0
http_response_time_seconds_bucket{method="PUT",path="/indexes/myindex/documents",le="0.0011"} 0
http_response_time_seconds_bucket{method="PUT",path="/indexes/myindex/documents",le="0.00115"} 0
http_response_time_seconds_bucket{method="PUT",path="/indexes/myindex/documents",le="0.0012"} 0
http_response_time_seconds_bucket{method="PUT",path="/indexes/myindex/documents",le="0.0015"} 10
http_response_time_seconds_bucket{method="PUT",path="/indexes/myindex/documents",le="0.002"} 751
http_response_time_seconds_bucket{method="PUT",path="/indexes/myindex/documents",le="0.003"} 1590
http_response_time_seconds_bucket{method="PUT",path="/indexes/myindex/documents",le="1"} 2533
http_response_time_seconds_bucket{method="PUT",path="/indexes/myindex/documents",le="+Inf"} 2533
http_response_time_seconds_sum{method="PUT",path="/indexes/myindex/documents"} 20.185828975999993

This is on GCP Monitoring, but the heatmap looks like this:

(GCP doesn't expose PromQL for this, but here's their bespoke DSL that represents the heatmap):

fetch prometheus_target
| metric 'prometheus.googleapis.com/http_response_time_seconds/histogram'
| filter (metric.path != '/health' && metric.path != '/metrics')
| align delta(1m)
| every 1m
| group_by [],
    [value_http_response_time_seconds_aggregate:
       aggregate(value.http_response_time_seconds)]

5 replies

gmourier May 11, 2023
Maintainer

Hey @hspak 👋

I'm not sure to understand; please correct me if I'm wrong! What I understand is that you lack precision for the sub-second histogram is that right?

hspak May 12, 2023

Yes! I'm not sure how fast these HTTP calls (especially index requests) are in internal testing, but the default buckets are way too aggressive in my opinion. Almost all of our requests are in this 0.003s to 1s bucket and that's not very helpful in catching trends. Some additional buckets that are greater than 1s would be helpful as well so latency spikes are obvious from looking at the histogram metrics.

miiton May 23, 2023

but the default buckets are way too aggressive in my opinion

I have tried it and I feel the same way.

My PromQL

sum(rate(http_response_time_seconds_bucket{path="/multi-search"}[$__rate_interval])) by (le)

rohankmr414 Mar 25, 2024

I feel the same about the current bucket boundaries, they are too granular and only track latencies below 1s, for someone who is using a managed prometheus service the cost is going to be high for this metric because we're generating too many metric samples due to this granularity. I think we should follow the opentelemetry spec for the http duration metric, it's mentioned here that the bucket boundaries for should be set to 5ms, 10ms, 25ms, 50ms, 75ms, 100ms, 250ms, 500ms, 750ms, 1s, 2.5s, 5s, 7.5s, 10s. I can open a PR for this, @irevoire wdyt?

rohankmr414 Mar 25, 2024

PR for the same -> meilisearch/meilisearch#4530

bannarisoftwares · 2023-05-13T14:32:55Z

bannarisoftwares
May 13, 2023

Should i need MEILI_EXPERIMENTAL_ENABLE_METRICS as true ?
like this ?

export MEILI_EXPERIMENTAL_ENABLE_METRICS=true

1 reply

curquiza May 15, 2023
Maintainer

Hello @bannarisoftwares!
Yes 😊
Once you test it, it would be awesome if you take the time to let us know if the feature fits your needs.

miiton · 2023-05-17T22:55:31Z

miiton
May 17, 2023

How about having a metric, like MEILISEARCH_TASK_QUEUES, to monitor the number and increase of task queues? I believe it would be useful for tuning the application, what do you think?

6 replies

curquiza May 22, 2023
Maintainer

@irevoire do you know how complex it is to implement?

irevoire May 22, 2023
Author

I don’t think it’s that hard, and I would like to work on it; that’s actually what I was working on before the index-scheduler refactor, but we were missing a lot of helpers to access the content of the task queue.
Now everything has been introduced, and getting the info should be pretty straightforward.

curquiza May 22, 2023
Maintainer

@macraig for your information

macraig May 22, 2023
Maintainer

@curquiza I think it makes sense to add it!

curquiza May 24, 2023
Maintainer

Let us know (with @gillian-meilisearch as well) if it's something you want (and we can) add to the next sprint 😊

hspak · 2023-06-08T21:58:01Z

hspak
Jun 8, 2023

Would it be possible to include a metric for indexing latency per index? It would be helpful to get insights on end-to-end indexing latency.

2 replies

curquiza Sep 21, 2023
Maintainer

Hello @hspak, sorry for the delay
what does it mean indexing latency per index for you?

hspak Jan 11, 2024

I would expect it to be the time from receiving the index request to seeing the indexed document available through search.

The metrics here only represents the time it took to enqueue the document to be indexed.

Sorry if this is out-of-date! I have not run meilisearch in some time.

macraig · 2023-06-13T08:20:34Z

macraig
Jun 13, 2023
Maintainer

Cross-posting feedback from @gabrieltz in #669 (comment)

Hello,

It would be nice to have some additional prometheus metrics that already exist in /stats endpoint:

lastUpdate exposed as a gauge with seconds since last update, or unix timestamp to detect if something is not "feeding meilisearch with new data" so we can define alert rules to prevent stale data.
isIndexing exposed as 0/1 would be helpful to correlate slower http_response_time_seconds_bucket with indexing

and from the /tasks endpoint

exposing the number of tasks with queued status ( gauge )
exposing the number of tasks with processing status ( gauge )

thank you

1 reply

irevoire Jun 19, 2023
Author

FYI:

and from the /tasks endpoint

exposing the number of tasks with queued status ( gauge )
exposing the number of tasks with processing status ( gauge )

This as already pretty much been implemented and will come out in the next release.
I didn’t use a gauge but a graph for the first version though, you’ll see once it comes out.

konstantin83 · 2023-07-07T20:01:04Z

konstantin83
Jul 7, 2023

Hi! Thank you for such a great app!

Currenlty experimental prometheus metrics are pretty poor. Is there a way to get more statictics? Like what queries were sent, and how much hits did they return. Or how much empty results and by which queries?

Or should I do my own middlware for that metrics?

3 replies

gillian-meilisearch Jul 12, 2023
Maintainer

Cross-posting from here to make sure we capture the suggestion details in the right place for @macraig to consider in future iterations

Request count by each index
Successful / Failed requests
Which aliases were used and how much and how much results retrieved
Average load by index
RPS by index
And so on.

macraig Jul 17, 2023
Maintainer

Hi @konstantin83 , thank you for sharing your feedback. We appreciate your input. At the moment, the metrics you mentioned are not available, but we are actively considering adding them to our product in the future. However, I want to be transparent and let you know that our team is currently focused on other priorities, which may impact the timeline for implementation. I apologize for any inconvenience caused by the delay, and we will do our best to keep you updated on our progress.

madalinignisca Jul 23, 2023

Sent queries with hits returned and similar stuff look more like in the area of LOGS and not time series metrics. I think this would add a lot of CPU load extra and extremely detailed metrics should be enabled by environment variables, so everybody else should have essentials only. For me as an SRE, time series metrics are critical in essentials and almost zero performance penalty. Anything that would enforce me to have a system that spends even 1% of CPU time to expose metrics is already having a poor performance hit.

miiton · 2023-08-01T02:51:44Z

miiton
Aug 1, 2023

I see that the Prometheus process is consuming a lot of CPU/Memory due to the large number of values related to TASKs.
I think this is a point for improvement.

http_response_time_seconds_count{method="GET",path="/tasks/1"} 1
http_response_time_seconds_bucket{method="GET",path="/tasks/1",le="0.0005"} 1
http_response_time_seconds_bucket{method="GET",path="/tasks/1",le="0.0008"} 1
http_response_time_seconds_bucket{method="GET",path="/tasks/1",le="0.00085"} 1
http_response_time_seconds_bucket{method="GET",path="/tasks/1",le="0.0009"} 1
...
...
...
http_response_time_seconds_count{method="GET",path="/tasks/9977"} 1
http_response_time_seconds_bucket{method="GET",path="/tasks/9978",le="0.0005"} 1
http_response_time_seconds_bucket{method="GET",path="/tasks/9978",le="0.0008"} 1
http_response_time_seconds_bucket{method="GET",path="/tasks/9978",le="0.00085"} 1
http_response_time_seconds_bucket{method="GET",path="/tasks/9978",le="0.0009"} 1
...

with scrape meilisearch metrics

without scrape meilisearch metrics

Meilisearch v1.2.0
Prometheus v2.45.0

7 replies

irevoire Aug 8, 2023
Author

Thanks for the details; I opened an issue to track this change: meilisearch/meilisearch#3983

Hades32 Aug 15, 2023

No, the right way to handle this is to not use HTTP paths, but the routes. So e.g. instead of /tasks/1 you'd get /task/{task_id}. This would also be highly needed for the get document endpoint

irevoire Aug 16, 2023
Author

Oooh, ok, Prometheus can handle many parameters but not many routes, basically?

But do you think it really brings something to the end user to be able to see how fast ms answered on each specific document/task?

Hades32 Aug 16, 2023

It's a cardinality problem. You don't want to explode the number of things in the system basically. It's metrics and not logs after all.

And no, I don't think there is much value in getting metrics for a specific task/doc. The route metrics would be enough imho

gh2k Apr 19, 2024

This is pretty critical from my perspective. For document deletes, this has a very very high cardinality problem

gabrieltz · 2023-08-02T06:28:06Z

gabrieltz
Aug 2, 2023

I upgraded to version 1.3.0 yesterday and used some of the new metrics, however i noticed that the metric meilisearch_nb_tasks{instance="XXXXXX:7700", job="meilisearch", kind="statuses", value="processing"} is zero all the time, even while importing a few million documents in over 700 batches. The /tasks endpoint values were correct though

1 reply

curquiza Aug 8, 2023
Maintainer

Hello @gabrieltz
thanks for the report, we have opened an issue and we are working on it here

heka1024 · 2023-09-15T08:07:10Z

heka1024
Sep 15, 2023

Hi. I'm using version 1.3.4 in meilisearch cloud and I didn't see that option. Is this not available in the cloud?

1 reply

curquiza Sep 18, 2023
Maintainer

Hello @heka1024
So far no, it's not available, but we plan to do it if needed by the users.

In the meantime, feel free to contact the support to see if it's possible to activate if for your project 😊

Schtil · 2023-11-01T16:12:15Z

Schtil
Nov 1, 2023

Greetings!
At the beginning I would like to express rays of support to the development team for the implementation of metrics as such.
However, are there any plans to do away with the standard, reserved fields within label names?
I mean the label with the name "value".
In our metrics collection system (frost scrapper) this leads to errors.
And in principle, it is not customary in the community to introduce some reserved names into label names. The value is associated with the value of metrics, and this leads to collisions when collecting and aggregating data in the future.

6 replies

Schtil Nov 2, 2023

I'll give it a try)

The point is, we have these metrics.

meilisearch_nb_tasks{kind="types",value="indexUpdate"} 0
meilisearch_nb_tasks{kind="types",value="settingsUpdate"} 0
meilisearch_nb_tasks{kind="types",value="snapshotCreation"} 0
meilisearch_nb_tasks{kind="types",value="taskCancelation"} 0
meilisearch_nb_tasks{kind="types",value="taskDeletion"} 0

These use two labels name, "kind" and "value".
These metrics are then scraped (collected) by monitoring systems
And then we are going to search by these metrics in conditional kibana.

It is common in the community to use a field value (in this case 0) in the "value" field of a metric.
However, here the "value" field is occupied in labels. And there is a collision.
Therefore, it is not common in the community to use some fixed names in labels.
Now we use crutches to rename label "value" to "argument" when scraping (we just took the name from the ceiling).

But we would like to get away from this crutch and solve the problem at the root.

irevoire Nov 2, 2023
Author

It is common in the community to use a field value (in this case 0) in the "value" field of a metric.

You mean that instead of sending something like that:
meilisearch_nb_tasks{kind="types",value="indexUpdate"} 0

People send something like this:
meilisearch_nb_tasks{kind="types",whatever="indexUpdate",value=0} 0

And if the value was 2 would they update both fields or only the one in the value thingy?

And so what you're asking for is basically that we rename the value thingy we use here to something else?

Schtil Nov 2, 2023

Almost.

I'm talking about replacing it with

meilisearch_nb_tasks{kind="types",whatever="indexUpdate"} 0

In the search engine the query will look like (Prom QL) meilisearch_nb_tasks{kind="types",whatever="indexUpdate"} == 0
In grafana, the query might look like this
meilisearch_nb_tasks{kind="types",whatever="indexUpdate"}
And grafana will build a graph based on the value of the "value" field - and it will only have a number in it

irevoire Nov 6, 2023
Author

Ah, grafana will autogenerate the value field on its side?

Schtil Nov 7, 2023

Rather Victoria Metrics (metrics storage system)
Grafana only displays the results from it

capJavert · 2024-01-11T15:18:00Z

capJavert
Jan 11, 2024

Anyone else getting no data on the dashboard, I am on 1.5.x, checking manually the prometheus and grafana data is in there which points me that dashboard might be out of date?

I can also see the requests for /metrics on my meilisearch instance.

7 replies

capJavert Jan 16, 2024

@irevoire first time setting it up. I can see that indexe and instance is loaded correctly with data source and if I try to create my own dashboard the data is there.

And if yes, can you try to edit the « database size in bytes » chart to re-set the input source?

What do you mean by this, just edit something in the chart and save dashboard? (I tried it and still "No data")

capJavert Jan 16, 2024

Ok, so basically the issue was I had a custom job_name for prometheus scrape config. This has to be meilisearch as it is hardcoded in the dashboard.

capJavert Jan 16, 2024

@irevoire if you would accept I added the PR for job variable so this does not happen if user sets the custom job_name. Default would still be meilisearch as mentioned in the docs. meilisearch/meilisearch#4330

irevoire Jan 18, 2024
Author

Hey, yes I merged it thanks for your contribution 😁

capJavert Jan 18, 2024

Yep, nice, tnx for the quick turnaround 🤘

rohankmr414 · 2024-01-30T06:05:20Z

rohankmr414
Jan 30, 2024

folks, I've opened a PR to add the http status code label to the meilisearch_http_requests_total metric meilisearch/meilisearch#4373, would love to have someone's review on this, also I'm not sure if this is the right way to do it as I'm not very good with rust.

1 reply

irevoire Mar 25, 2024
Author

Thanks, I merged it

productdevbook · 2024-04-07T13:42:11Z

productdevbook
Apr 7, 2024

https://github.com/meilisearch/meilisearch/blob/main/assets/grafana-dashboard.json

Can you also make a separate json to cover the whole index?

1 reply

irevoire May 13, 2024
Author

Hey, what do you mean by the whole index?
Which information is missing currently?

mrancilio · 2024-07-17T08:55:36Z

mrancilio
Jul 17, 2024

Hi, the link for the yaml file is broken. Where can I download the file?

Download the Prometheus assets file stored on the meilisearch repository assets/prometheus-basic-scraper.yml.

1 reply

irevoire Jul 18, 2024
Author

Hey @mrancilio, thanks for the feedback!
I fixed the link : https://github.com/meilisearch/meilisearch/blob/main/assets/prometheus-basic-scraper.yml

Experimental feature: Prometheus Metrics #625

Uh oh!

Uh oh!

Feature Abstract

How to enable the feature in Meilisearch?

How to use the feature?

Step-by-step basic tutorial

What's currently exposed?

What is an experimental feature

Why is this feature not stable yet?

When will the feature potentially be stable?

X metrics are missing

Replies: 16 comments · 43 replies

Uh oh!

Uh oh!

gmourier Mar 8, 2023 Maintainer

Uh oh!

gmourier Apr 11, 2023 Maintainer

Uh oh!

Uh oh!

gmourier May 11, 2023 Maintainer

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

curquiza May 15, 2023 Maintainer

Uh oh!

Uh oh!

curquiza May 22, 2023 Maintainer

Uh oh!

irevoire May 22, 2023 Author

Uh oh!

curquiza May 22, 2023 Maintainer

Uh oh!

macraig May 22, 2023 Maintainer

Uh oh!

curquiza May 24, 2023 Maintainer

Uh oh!

Uh oh!

curquiza Sep 21, 2023 Maintainer

Uh oh!

Uh oh!

macraig Jun 13, 2023 Maintainer

Uh oh!

irevoire Jun 19, 2023 Author

Uh oh!

Uh oh!

gillian-meilisearch Jul 12, 2023 Maintainer

Uh oh!

macraig Jul 17, 2023 Maintainer

Uh oh!

Uh oh!

Uh oh!

irevoire Aug 8, 2023 Author

Uh oh!

Uh oh!

Replies: 16 comments 43 replies

gmourier
Mar 8, 2023
Maintainer

gmourier
Apr 11, 2023
Maintainer

gmourier May 11, 2023
Maintainer

curquiza May 15, 2023
Maintainer

curquiza May 22, 2023
Maintainer

irevoire May 22, 2023
Author

curquiza May 22, 2023
Maintainer

macraig May 22, 2023
Maintainer

curquiza May 24, 2023
Maintainer

curquiza Sep 21, 2023
Maintainer

macraig
Jun 13, 2023
Maintainer

irevoire Jun 19, 2023
Author

gillian-meilisearch Jul 12, 2023
Maintainer

macraig Jul 17, 2023
Maintainer

irevoire Aug 8, 2023
Author