s3: fix Content-MD5 for Object Lock uploads and add GCS quirk#9200
Open
chris081519-crypto wants to merge 1 commit intorclone:masterfrom
Open
s3: fix Content-MD5 for Object Lock uploads and add GCS quirk#9200chris081519-crypto wants to merge 1 commit intorclone:masterfrom
chris081519-crypto wants to merge 1 commit intorclone:masterfrom
Conversation
AWS S3 requires Content-MD5 for PutObject with Object Lock parameters. Since rclone passes a non-seekable io.Reader, the SDK cannot compute checksums automatically. Buffer the body and compute MD5 manually for singlepart PutObject and presigned request uploads when Object Lock parameters are set. Multipart uploads are unaffected as Object Lock headers go on CreateMultipartUpload which has no body. Add object_lock_supported provider quirk (default true) to allow skipping Object Lock integration tests on providers with incomplete S3 API support. Set to false for GCS which uses non-standard x-goog-bypass-governance-retention header and doesn't implement PutObjectLegalHold/GetObjectLegalHold. Add Multipart and Presigned subtests to Object Lock integration tests to cover all three upload paths. Fixes rclone#9199
dd2f101 to
3365b91
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
AWS S3 requires Content-MD5 for PutObject with Object Lock parameters. Since rclone passes a non-seekable io.Reader, the SDK cannot compute checksums automatically. Buffer the body and compute MD5 manually for singlepart PutObject and presigned request uploads when Object Lock parameters are set. Multipart uploads are unaffected as Object Lock headers go on CreateMultipartUpload which has no body.
Add object_lock_supported provider quirk (default true) to allow skipping Object Lock integration tests on providers with incomplete S3 API support. Set to false for GCS which uses non-standard x-goog-bypass-governance-retention header and doesn't implement PutObjectLegalHold/GetObjectLegalHold.
Add Multipart and Presigned subtests to Object Lock integration tests to cover all three upload paths.
What is the purpose of this change?
Adding Content-MD5 Headers when using Object Lock on AWS for PutObject.
Adding provider quirk to skip GCP testing with Object Lock (as their implementation deviates from the standard).
Was the change discussed in an issue or in the forum before?
Fixes #9199
Checklist