HADOOP-19610. S3A: ITests to run under JUnit5 #7814
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Seems to kick off the tests in parallel. There are three runs, the last with no tests executed. But its elapsed time is exactly those of the other two executions added up (16.38 + 12.42 |
|
comparison execution time note that both test runs were in a single process, so even if the log showed three test suites running at the same time, only one process is running I don't think that is good, because we do have shared state (filesystem cache etc) which will interfere with each other |
|
@steveloughran Thanks for picking this up. This was blocking dev work on S3A. |
shameersss1
approved these changes
Jul 22, 2025
b2d347f to
8249bb8
Compare
|
I'm doing another local run on this with the analytics profile, after which it is ready to merge. That is even though every mini yarn cluster refuses to come up with queue problems. I'm not convinced that is related as there have been recent changes near the failing yarn code. And we need this in so urgently that reducing the number of tests down to a limited few is still a success |
|
one failure being fixed elsewhere |
4 tasks
9fea0cb to
99198ef
Compare
hadoop-aws tests which need to be parameterized on a class level are configured to do so through the @ParameterizedClass tag. Filesystem contract test suites in hadoop-common have also been parameterized as appropriate. There are custom JUnit tags declared in org.apache.hadoop.test.tags, which add tag strings to test suites/cases declaring them. They can be used on the command line and in IDEs to control which tests are/are not executed. @FlakyTest "flaky" @loadtest "load" @RootFilesystemTest "rootfilesystem" @scaletest "scale" For anyone migrating tests to JUnit 5 * Methods which subclass an existing test case MUST declare the @test tag again -it is no longer inherited. * All overridden setup/teardown methods MUST be located and @BeforeEach/@AfterEach attribute added respectively * Subclasses of a parameterized test suite MUST redeclare themselves as a @ParameterizedClass, and the binding mechanism again. * Parameterized test suites SHOULD declare a pattern to generate an informative parameter value string for logs, IDEs and stack traces, e.g. @ParameterizedClass(name="performance-{0}") * Test suites SHOULD add a org.apache.hadoop.test.tags tag to declare what kind of test it is. These tags are inherited, so it may be that only shared superclasses of test suites need to be tagged. The abstract filesystem contract tests are NOT declared as integration tests -implementations MUST do so if they are integration tests.
Maybe better to actually push this into the implementations?
99198ef to
9baa3dd
Compare
|
rebased and with core tests working, just some minor failures. Also new tag Failures now yarn minicluster and multipart. I do suspect the multipart is related but I don't want it to hold up this (critical) patch. ITestS3ACommitterMRJob may be a test setup again as it is a sequence of tests. |
|
🎊 +1 overall
This message was automatically generated. |
|
checkstyles are all about test numbering _100() etc, so invalid |
|
I need this in ASAP as there's no way to validate any other PR: failures of their tests are meaningless. @slfan1989 @ahmarsuhail @mukund-thakur can I get an upvote on this? It isn't perfect, but it's test ony code and we can tune later |
4 tasks
slfan1989
approved these changes
Jul 25, 2025
@steveloughran Thanks for your contribution! I believe this PR is good enough, and we can go ahead and merge it. |
4 tasks
|
FYI, outstanding failures.
|
|
@steveloughran I'll follow up on the Yarn MiniCluster issue, but it might be later today. I've dealt with a similar issue before, so it might be helpful. The main cause is that during parallel testing, different unit tests are accessing the same configuration file, as some of the configurations are stored in directories like |
|
ahh, I see. I think temp dirs are set per process and a base test jobID. |
|
just discovered (Iceberg code) that assertj may limit stack traces, so you need to set |
4 tasks
75 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Fixing parameterized ITests in hadoop-aws to work under Junit 5.
This is on top of PR #7785
For code changes:
LICENSE,LICENSE-binary,NOTICE-binaryfiles?