fix s3 v3 multipart overwrite when key exists #9388
Merged
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Motivation
Reported in #9387: when creating a multipart upload on an existing key, we would not truncate the key before overwrite it like we do when calling
PutObject
.This was due to our optimization with
append
on the StoredS3Object, which was stateless, and bad oversight on my side.Flow for the sample in the linked issue is:
test
test
test
-> value is "first"test
test
test
, result should be "firstsecond"Issue was appending on top of existing key, resulting in "firstfirstsecond".
Changes
Added a new
truncate
method to the baseS3StoredObject
to be able to reset the file from the StoredMultipart.Add
s3_object.seek(0) + s3_object.truncate()
incomplete_multipart
so that the existing file object representing the key is properly truncated before we write and we do not append the multipart on top of the existing key.Added an AWS validated test validating this behaviour.