Core: Avoid table corruption from 409 on self conflicts after 5xx retries by throwing CommitStateUnknown #12818
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
29665a1 to
8665dc1
Compare
core/src/main/java/org/apache/iceberg/rest/RESTTableOperations.java
Outdated
Show resolved
Hide resolved
3 tasks
8665dc1 to
7e86071
Compare
7e86071 to
35bd596
Compare
singhpk234
commented
May 24, 2025
d75ab7c to
ceb0b4f
Compare
96c35f0 to
24dfd82
Compare
RussellSpitzer
approved these changes
May 27, 2025
d661510 to
afe2ee5
Compare
|
Thank you for your feedbacks Russell, I addressed it in the latest commit. |
RussellSpitzer
approved these changes
May 27, 2025
|
@amogh-jahagirdar Do you have any other feedback? |
There was a problem hiding this comment.
Overall the fix looks great @singhpk234 just had a minor comment on a public builder API, that'd be good to address before we release anything
core/src/main/java/org/apache/iceberg/rest/responses/ErrorResponse.java
Outdated
Show resolved
Hide resolved
amogh-jahagirdar
approved these changes
May 29, 2025
amogh-jahagirdar
approved these changes
May 29, 2025
There was a problem hiding this comment.
Just had a nit on the comment, but change looks good. Thank you @singhpk234 , this is an important fix!
Comment on lines
95
to
101
| // If the request was retried, and if its final error was 409, It could probably also | ||
| // mean that HTTP retries when happened the IRC service could have actually applied | ||
| // the commit in their persistence, while giving back the client still a 5xx. | ||
| // If so, since the base has changed, it could conflict with itself. | ||
| // In cases like this its best not to mark this as failed instead | ||
| // make this is a commit state unknown, so some reconciliation can happen and | ||
| // the metadata clean-up is not triggered. |
There was a problem hiding this comment.
Nit on comment, maybe something like this?
/**
* If a retried request finally fails with 409,
* the IRC service may have persisted the commit
* despite initial 5xx errors, resulting in a self-conflict on retry
* due to the base changing.
* Mark this failure as commit state unknown rather than failed to prevent file cleanup.
*/
dd34ad5 to
5f651ec
Compare
nastra
reviewed
Jun 2, 2025
singhpk234
added a commit
to singhpk234/iceberg
that referenced
this pull request
Jun 13, 2025
…ries by throwing CommitStateUnknown (apache#12818)
singhpk234
pushed a commit
to singhpk234/iceberg
that referenced
this pull request
Jun 19, 2025
… 5xx retries by throwing CommitStateUnknown (apache#12818)" This reverts commit 4927610.
RussellSpitzer
pushed a commit
that referenced
this pull request
Jul 1, 2025
singhpk234
added a commit
to singhpk234/iceberg
that referenced
this pull request
Jul 3, 2025
…ries by throwing CommitStateUnknown (apache#12818)
singhpk234
added a commit
to singhpk234/iceberg
that referenced
this pull request
Jul 3, 2025
singhpk234
added a commit
to singhpk234/iceberg
that referenced
this pull request
Jul 3, 2025
RussellSpitzer
pushed a commit
that referenced
this pull request
Jul 3, 2025
…ries by throwing CommitStateUnknown (#12818)
RussellSpitzer
pushed a commit
that referenced
this pull request
Jul 3, 2025
dcagney
pushed a commit
to Affirm/iceberg
that referenced
this pull request
Jul 7, 2025
…ries by throwing CommitStateUnknown (apache#12818)
dcagney
pushed a commit
to Affirm/iceberg
that referenced
this pull request
Jul 7, 2025
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
About the change
Solves : #12792 (comment)
This is happening in IRC client we retry on 5xx (this is a http client level retry, no reconciliation, just retrying same request without rebasing) please ref, now this a very know issue that the service despite giving 5xx applies the commit in its persistence (check this with glue non irc issue here) , but client still see its as 5xx, and retries, so when this is retried via http client retry and the persistence has already applied the commit it will now give 409 hence make table delete the metadata of the commit since we get back CommitFailed exception.
The fix here proposes to throw CommitStateUnknown so that client can reconcile the state and at the bare minumum doesn't treat this as failed and clean up the metadata files
Testing
Added new UT