Raise non-API exceptions #1316
Raise non-API exceptions #1316
Conversation
The original intent was to catch API exceptions (errors returned by the broker when trying to produce a message) and delegate them to the messages' futures. This is copied from the Java producer. However, we were accidentally catching all exceptions, thereby hiding exceptions from users unless they explicitly check the result of the future. Much better to raise client-side errors directly in the foreground so the user is immediately aware of them and can decide how to handle. Fix #1274
23e704b to
3cc7e27
Compare
| except Errors.KafkaTimeoutError: | ||
| raise | ||
| except AssertionError: | ||
| raise |
There was a problem hiding this comment.
I double-checked, and these are not explicitly called out in the Java producer. So it's fine to remove them as KafkaTimeoutError does not subclass BrokerResponseError.
|
Can I get a review on this? I'd like to merge this one as it means non-API problems are more likely to be noticed by my internal users themselves rather than handing the issue off to me to debug... |
|
I'm not a huge fan of this. It feels especially dangerous given that existing installations may be relying on internal retry behavior, which this would break. I would much rather encourage users to check their futures. |
|
It makes sense to me to internally retry transient errors such as network connectivity (which IIUC is already handled at the It does not make sense to me that we currently require users to check futures for non-transient errors where that message has no chance of ever being successfully produced to the cluster. I'd rather error loudly and let the application developer decide how to handle. Sometimes the message is critical and they want to stop immediately. Sometimes they don't care and can try/except to skip that particular message. I have internal users for both use cases. I'd also say that blowing up in the foreground is a much more intuitive development model for anyone new to the library. When they're just trying to get something working, it's much easier to have an exception rather than flailing for a while wondering why it's not showing up in the broker before they realize they need to check the future. Plus it would better match how the Java Producer handles this, which is one of the primary design guidelines of this library. As far as breakage, agree that we don't want to do that, at least not in a minor version release. Putting it in a 2.0 would make sense, perhaps if I finish #1196. Also, currently |
|
Ok, let me think about this some more. |
|
OK, I'm convinced. If there are exceptions raised that should not be, we can add additional exception types as-needed. The bare except is bad for the reasons you've outlined. Thanks jeff. |
|
Thanks. And for the record, I do not mind being overruled if you think there's a good reason not to do something. |
The original intent was to catch API exceptions (errors returned by the
broker when trying to produce a message) and delegate them to the
messages' futures. This is copied from the Java producer.
However, we were accidentally catching all exceptions, thereby hiding
exceptions from users unless they explicitly check the result of the
future. Much better to raise client-side errors directly in the
foreground so the user is immediately aware of them and can decide how
to handle.
Fix #1274