Thanks to visit codestin.com
Credit goes to github.com

Skip to content

HADOOP-14334. S3 SSEC tests to downgrade when running against a mandatory encryption object store #3870

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 2 commits into from
Jan 9, 2022
Merged

HADOOP-14334. S3 SSEC tests to downgrade when running against a mandatory encryption object store #3870

merged 2 commits into from
Jan 9, 2022

Conversation

@monthonk
Copy link
Contributor

@monthonk monthonk commented Jan 6, 2022

If you run the S3A tests against a bucket with mandatory encryption, you need to set the encryption in auth keys. This breaks the SSEC tests because the encryption.key property being set breaks them.

This changes catch AccessDeniedException in the setup step of SSE tests, if it occurs then the test will be skipped. This exception is thrown by S3A when using encryption method that is not allowed by the bucket.

Tested in eu-west-1 with mvn -Dparallel-tests -DtestsThreadCount=32 clean verify

[INFO] Results:
[INFO] 
[ERROR] Failures: 
[ERROR]   ITestS3AContractUnbuffer>AbstractContractUnbufferTest.testMultipleUnbuffers:100->AbstractContractUnbufferTest.validateFullFileContents:132->AbstractContractUnbufferTest.validateFileContents:139->Assert.assertEquals:647->Assert.failNotEquals:835->Assert.fail:89 failed to read expected number of bytes from stream. This may be transient expected:<1024> but was:<605>
[ERROR]   ITestS3AInconsistency.testGetFileStatus:123->Assert.fail:89 getFileStatus should fail due to delayed visibility.
[INFO] 
[ERROR] Tests run: 1475, Failures: 2, Errors: 0, Skipped: 509

Rerun failures individually

[INFO] -------------------------------------------------------
[INFO]  T E S T S
[INFO] -------------------------------------------------------
[INFO] Running org.apache.hadoop.fs.contract.s3a.ITestS3AContractUnbuffer
[INFO] Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 4.894 s - in org.apache.hadoop.fs.contract.s3a.ITestS3AContractUnbuffer
[INFO] 
[INFO] Results:
[INFO] 
[INFO] Tests run: 6, Failures: 0, Errors: 0, Skipped: 0

[INFO] -------------------------------------------------------
[INFO]  T E S T S
[INFO] -------------------------------------------------------
[INFO] Running org.apache.hadoop.fs.s3a.ITestS3AInconsistency
[INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.108 s - in org.apache.hadoop.fs.s3a.ITestS3AInconsistency
[INFO] 
[INFO] Results:
[INFO] 
[INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0

@monthonk monthonk changed the title HADOOP-14334 S3 SSEC tests to downgrade when running against a mandatory encryption object store HADOOP-14334. S3 SSEC tests to downgrade when running against a mandatory encryption object store Jan 6, 2022
steveloughran
steveloughran approved these changes Jan 7, 2022
View reviewed changes
Copy link
Contributor

@steveloughran steveloughran left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1

makes sense. I see those transient test failures too...buffer underrun breaks assertions. but the tests are trying to validate read() so we cant move to readFully()

what is your full name for the credit in the commit?

@steveloughran
Copy link
Contributor

steveloughran commented Jan 7, 2022

and your jira user name so i can assign the issue to you there

@monthonk
Copy link
Contributor Author

monthonk commented Jan 7, 2022

My name is Monthon Klongklaew, and monthonk for the jira user name. Thank you

@steveloughran steveloughran merged commit b27732c into apache:trunk Jan 9, 2022
@apache apache deleted a comment from hadoop-yetus Jan 9, 2022
asfgit pushed a commit that referenced this pull request Jan 9, 2022
@monthonk @steveloughran
HADOOP-14334. S3 SSEC tests to downgrade when running against a manda…
7dd8e90 
…tory encryption object store (#3870)

Contributed by Monthon Klongklaew

Change-Id: Ib275c9690bbc90170c6a442ded198fe006c20bc1
sunchao pushed a commit that referenced this pull request Jan 15, 2022
@monthonk @sunchao
HADOOP-14334. S3 SSEC tests to downgrade when running against a manda…
d101e4d 
…tory encryption object store (#3870)

Contributed by Monthon Klongklaew

Change-Id: Ib275c9690bbc90170c6a442ded198fe006c20bc1
HarshitGupta11 pushed a commit to HarshitGupta11/hadoop that referenced this pull request Nov 28, 2022
@monthonk
HADOOP-14334. S3 SSEC tests to downgrade when running against a manda…
f77db17 
…tory encryption object store (apache#3870)


Contributed by Monthon Klongklaew
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@steveloughran steveloughran steveloughran approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@monthonk @steveloughran