storage for tracking rate limits #68630
Replies: 3 comments 2 replies
-
|
It is not a broker in the task-execution sense. This is just storage for the rate limiter state, so the backend keeps short-lived counters such as request counts in the current window and when the window resets. It is not meant for metrics or Airflow metadata. For Valkey vs Redis: Valkey is broadly Redis-compatible. If Redis is already available, that is the safest default. If your deployment already uses Valkey and the limiter backend supports it, that should usually work too. The βmultiple webserver/API instancesβ part just means shared state matters if you run more than one Airflow web-facing process. In that case, a shared backend like Redis or Valkey keeps the rate limiting consistent across instances. |
Beta Was this translation helpful? Give feedback.
-
Sorry for asking so many questions. |
Beta Was this translation helpful? Give feedback.
-
|
I checked the Helm-chart deployment for reference. It doesn't seem to have a deployment for that. I also checked the generated default config ( |
Beta Was this translation helpful? Give feedback.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
-
Airflow gives me this warning:
When I follow the link, I was expecting to find something I could just configure. Instead I see a lot of python code and I am completely overwhelmed and have now idea what to do.
I don't even understand what that means. I don't understand why the default is in a state that requires explicit action, instead of using the already-connected database. I mean, there is persistent storage right there. I don't understand why this has to be done by writing some python code, instead of configuring some settings in the
airflow.cfg. I don't even know if this warning is bad or not.I'm totally confused and could need some help. What is the easiest (but also production-ready) way to solve this?
Beta Was this translation helpful? Give feedback.
All reactions