OpenAI-compatible interface doesn't properly propagate 429 responses #3483
Replies: 1 comment 2 replies
-
|
Hi @ucirello - are you able to share the name of the provider and a sample raw response from them with the 429? I agree we need better error propagation in general. In the meantime, maybe the unstable setting |
Beta Was this translation helpful? Give feedback.
-
|
This setting might not be enough to surface the rate-limiting deadlines, unfortunately. If you're able to share the things I mentioned above, we can brainstorm if there's a neat solution. |
Beta Was this translation helpful? Give feedback.
-
|
Yes: here's what I get from the OpenAI's client in Go (openai-go/v2) while talking to OpenAI itself:
|
Beta Was this translation helpful? Give feedback.
Uh oh!
There was an error while loading. Please reload this page.
-
This is a bit of a curve-ball, because 429 could be handled both by the gateway and the client.
Right now, I am facing a situation in which my OpenAI-compatible provider is consistently returning 429s, but these errors are bubbled back as 502.
In a multiple-variant situation, I believe the Gateway would correctly rotate to the next available variant, but in a function with just one variant, I wonder if the Gateway should return 429?
In any case, the only I can know that I need to retry to parse the text from the 502 response and look for the string "429". However, the structured response (in which it tells me how long I would have to wait), seems to be lost.
Beta Was this translation helpful? Give feedback.
All reactions