Thanks to visit codestin.com
Credit goes to github.com

Skip to content

Fixes for POST policy checks and the x-ignore implementation #20674

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 15 commits into from
Dec 11, 2024
Merged

Fixes for POST policy checks and the x-ignore implementation #20674

merged 15 commits into from
Dec 11, 2024
+268 −133

Conversation

@marktheunissen
Copy link
Contributor

@marktheunissen marktheunissen commented Nov 21, 2024 •
edited
Loading

Community Contribution License

All community contributions in this pull request are licensed to the project maintainers
under the terms of the Apache 2 license.
By creating this pull request I represent that I have the right to license the
contributions to the project maintainers under the Apache 2 license.

Description

Related PR: minio/minio-go#1996

  • X-AWS-Ignore-... was incorrectly implemented according to S3 docs.
  • Content-Encoding should not be an ignored form field.
  • The checksum form fields should not be ignored either.
  • Simplify checkPostPolicy and add more tests

Motivation and Context

We're not correctly implementing S3 spec, and there are bugs in the implementation we do have.

Our current implementation allows form values like X-Amz-Ignore-Foo to work as directives that tell MinIO to ignore Foo form value, so you could provide Foo and X-Amz-Ignore-Foo together, and then we would not check that Foo appears in the policy. This implementation was probably just a misunderstanding or a misread of what the actual behavior should be, as specified here: https://docs.aws.amazon.com/AmazonS3/latest/API/sigv4-HTTPPOSTConstructPolicy.html

X-Amz-Ignore.. is not mentioned at all in AWS docs that I can find. Only X-Ignore-..., and the way it works is that any form key starting with X-Ignore-... will be ignored - it does not work as a directive that affects other form keys.

How to test this PR?

Types of changes

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Optimization (provides speedup with no functional changes)
  • Breaking change (fix or feature that would cause existing functionality to change)

Checklist:

  • Fixes a regression (If yes, please add commit-id or PR # here)
  • Unit tests added/updated
  • Internal documentation updated
  • Create a documentation update request here

marktheunissen
marktheunissen commented Nov 21, 2024
View reviewed changes
cmd/postpolicyform.go
"File": true,
"Policy": true,

// MinIO specific exceptions to the general S3 rule above.
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm removing the check strings.HasPrefix(key, encrypt.SseGenericHeader) below, and instead listing out the SSE related header exceptions here.

Please check my assumption that ignoring these SSE headers is still the correct course of action.

marktheunissen
marktheunissen commented Nov 21, 2024
View reviewed changes
cmd/postpolicyform_test.go
}

err = checkPostPolicy(formValues, postPolicyForm)
if err != nil && tt.expectedErr != nil && err.Error() != tt.expectedErr.Error() {
Copy link
Contributor Author

@marktheunissen marktheunissen Nov 21, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This was masking the fact that at least one of the existing tests was actually failing.

@marktheunissen marktheunissen mentioned this pull request Nov 21, 2024
Merged
@marktheunissen marktheunissen marked this pull request as ready for review November 21, 2024 02:48
marktheunissen
marktheunissen commented Nov 21, 2024
View reviewed changes
@marktheunissen
Copy link
Contributor Author

Tests will fail until the mint image is updated with this: minio/minio-go#1996

@klauspost
Copy link
Contributor

Failing mint test: https://github.com/minio/minio-go/blob/master/functional_tests.go#L5635

And regular test:

2024-11-21T04:57:21.6565040Z --- FAIL: TestPostPolicyForm (0.00s)
....
2024-11-21T04:57:21.6578393Z     --- FAIL: TestPostPolicyForm/unknown_key_Content-Encoding_is_error_as_it_does_not_appear_in_policy (0.00s)

@marktheunissen
Copy link
Contributor Author

@klauspost Thanks, I fixed the regular test, and yes, Mint tests will fail until the mint image is updated with this: minio/minio-go#1996

What procedure would you prefer to follow here?

@klauspost
Copy link
Contributor

Let's get upstream merged.

klauspost
klauspost reviewed Nov 29, 2024
View reviewed changes
Copy link
Contributor

@klauspost klauspost left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just a few overall Qs.

klauspost
klauspost approved these changes Dec 3, 2024
View reviewed changes
Copy link
Contributor

@klauspost klauspost left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@marktheunissen
Copy link
Contributor Author

Blocked currently by this failing mint test, which is unrelated:

{
  "time": "2024-12-04T05:45:17.485590419Z",
  "level": "ERROR",
  "name": "minio-go: testGetObjectAttributes",
  "duration": 474,
  "function": "GetObjectAttributes(ctx, bucketName, objectName, opts)",
  "args": {
    "bucketName": "minio-go-test-versioned-wcioaf",
    "objectName": "file1",
    "opts": "minio.ObjectAttributesOptions{}"
  },
  "status": "FAIL",
  "message": "Validating GetObjectsAttributes response failed, table test: versioned",
  "error": "Full object checksum not found"
}

I started discussion on the fix to the above test here: #20732

harshavardhana
harshavardhana approved these changes Dec 11, 2024
View reviewed changes
Copy link
Member

@harshavardhana harshavardhana left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PR already merged in aistor

@harshavardhana harshavardhana merged commit a248ed5 into minio:master Dec 11, 2024
21 checks passed
@marktheunissen marktheunissen deleted the postpolicy-fixes branch December 11, 2024 22:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@harshavardhana harshavardhana harshavardhana approved these changes

@klauspost klauspost klauspost approved these changes

@jiuker jiuker Awaiting requested review from jiuker

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@marktheunissen @klauspost @harshavardhana