Thanks to visit codestin.com
Credit goes to github.com

Skip to content

S3: fix Checksum handling in UploadPartCopy #12753

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Jun 16, 2025
Merged

Conversation

bentsku
Copy link
Contributor

@bentsku bentsku commented Jun 13, 2025

Motivation

We've got a report in our Community Slack with a small sample (as represented by the new test) where Checksum handling did not work well with the UploadPartCopy operation: https://docs.aws.amazon.com/AmazonS3/latest/API/API_UploadPartCopy.html

This was due because we simply did not handle checksums as this was not implemented in our previous implementation.

We now properly handle checksums when copying an object into a part, via our storage engine.

This was the error when calling ListParts:

An error occurred (InternalError) when calling the ListParts operation (reached max retries: 4): exception while calling s3.ListParts: 'NoneType' object has no attribute 'upper'

Changes

  • add new test covering the case of multipart uploads with checksum and UploadPartCopy
  • fix the behavior to retrieve the new checksum from the stored object, calculated when being stored
  • return the checksum field when the conditions are filled for UploadPartCopy
  • add a better check in ListParts to avoid unhandled exceptions

@bentsku bentsku added this to the 4.6 milestone Jun 13, 2025
@bentsku bentsku self-assigned this Jun 13, 2025
@bentsku bentsku added aws:s3 Amazon Simple Storage Service semver: patch Non-breaking changes which can be included in patch releases labels Jun 13, 2025
Copy link

S3 Image Test Results (AMD64 / ARM64)

  2 files    2 suites   8m 26s ⏱️
496 tests 446 ✅  50 💤 0 ❌
992 runs  892 ✅ 100 💤 0 ❌

Results for commit 420747a.

Copy link

Test Results - Preflight, Unit

21 613 tests  ±0   19 958 ✅ ±0   6m 16s ⏱️ -8s
     1 suites ±0    1 655 💤 ±0 
     1 files   ±0        0 ❌ ±0 

Results for commit 420747a. ± Comparison against base commit 02ad74e.

Copy link

Test Results (amd64) - Acceptance

7 tests  ±0   5 ✅ ±0   3m 6s ⏱️ -1s
1 suites ±0   2 💤 ±0 
1 files   ±0   0 ❌ ±0 

Results for commit 420747a. ± Comparison against base commit 02ad74e.

Copy link

Test Results - Alternative Providers

987 tests   589 ✅  30m 17s ⏱️
  4 suites  398 💤
  4 files      0 ❌

Results for commit 420747a.

Copy link

Test Results (amd64) - Integration, Bootstrap

    5 files      5 suites   1h 24m 40s ⏱️
2 237 tests 1 621 ✅ 616 💤 0 ❌
2 243 runs  1 621 ✅ 622 💤 0 ❌

Results for commit 420747a.

Copy link

LocalStack Community integration with Pro

    2 files      2 suites   1h 2m 8s ⏱️
2 213 tests 1 594 ✅ 619 💤 0 ❌
2 215 runs  1 594 ✅ 621 💤 0 ❌

Results for commit 420747a.

@bentsku bentsku requested a review from k-a-il June 13, 2025 14:26
Copy link
Contributor

@k-a-il k-a-il left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice quick fix 🌟 I took a quick look at the code and snapshots - everything looks good.

@bentsku bentsku merged commit df9ebe9 into master Jun 16, 2025
52 checks passed
@bentsku bentsku deleted the fix-s3-part-checksum branch June 16, 2025 09:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
aws:s3 Amazon Simple Storage Service semver: patch Non-breaking changes which can be included in patch releases
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants