fix pre-signed URL with JS SDK, enable test_presigned_url_v4_x_amz_in_qs #12164
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
S3 Image Test Results (AMD64 / ARM64) 2 files 2 suites 4m 41s ⏱️ Results for commit 0e5a983. |
LocalStack Community integration with Pro 2 files ± 0 2 suites ±0 59m 28s ⏱️ - 52m 51s Results for commit 0e5a983. ± Comparison against base commit e5e74db. This pull request removes 2333 tests. |
1 task
1 task
silv-io
approved these changes
Jan 23, 2025
| @@ -7544,6 +7543,8 @@ def test_presigned_url_v4_x_amz_in_qs( | |||
| # assert that the Javascript SDK hoists it in the URL, unlike Boto | |||
| assert StorageClass.STANDARD in presigned_url | |||
| assert "bar-complicated-no-random" in presigned_url | |||
| # the JS SDK also adds a default checksum now even for pre-signed URLs | |||
There was a problem hiding this comment.
Yeah I still don't understand how this made it into prod 😄
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Motivation
We had pipeline failures due to the
test_presigned_url_v4_x_amz_in_qstest suddenly failing. This happened at the same time we've had a big update in S3 data integrity and checksums in our boto update.I suspected a change in behavior in the Javascript SDK: what happens now is that the JS SDK is adding a checksum value for pre-signed URL, which in itself is quite a bad idea, as you cannot know in advance what the value of the object can be.
So it seems AWS is just flat out ignoring the query string parameter.
Pre-signed URLs are a funny concept: you pass some values that would normally be headers in a regular signed request as query string parameters, so that the URL can be shared. To not have to manually parse query string parameters in our S3 provider, our pre-signed URL handler is picking up the query string parameters and putting them back as headers and mutating the request before it being parsed, so that we can fully enjoy the ASF power. See #8918
But in that case, the query string parameter is important for the signature, if you try modifying it, S3 will raise an invalid signature exception. But you need to ignore it before passing it down to the provider.
edit: this actually has been reported now with #12169
Changes
x-amz-checksum-<>header that is not in the signed headers, and do not pass it down the provider\cc @k-a-il