Thanks to visit codestin.com
Credit goes to github.com

Skip to content

feat: Improve slive stress tests with path scanning and better file h… #8052

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 1 commit into
base: trunk
Choose a base branch
from
Open

feat: Improve slive stress tests with path scanning and better file h… #8052

wants to merge 1 commit into from
+250 −23

Conversation

@zczhao0809
Copy link

git commit -m "feat: Improve slive stress tests with path scanning and better file handling

This commit enhances the slive (stress live) test framework:

  1. PathFinder: Add directory scanning and operation-specific path selection

    • Support for EPS format scanning existing files/directories
    • Operation-specific types (RENAME_SRC, CREATE, LS) for better reliability
    • Unique path generation using UUID for CREATE operations
    • Selective retrieval based on operation requirements

  2. RenameOp improvements:

    • Auto-create target directory before rename operation
    • Proper file selection: existing files as source, unique paths as target
    • Separate FileNotFoundException handling with detailed logging

  3. Operation classes (AppendOp, DeleteOp, ListOp, MkdirOp, ReadOp, TruncateOp):

    • Updated to use operation-specific path selection
    • Better error logging with detailed context

  4. Configuration: Add shouldUseNewAlgorithm() support

These changes reduce test failures by ensuring operations target valid
existing files and create unique paths for write operations."

@hadoop-yetus
Copy link

💔 -1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 0m 32s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 1s No case conflicting files found.
+0 🆗 codespell 0m 0s codespell was not available.
+0 🆗 detsecrets 0m 0s detect-secrets was not available.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
+1 💚 test4tests 0m 0s The patch appears to include 14 new or modified test files.
_ trunk Compile Tests _
+1 💚 mvninstall 33m 33s trunk passed
+1 💚 compile 0m 40s trunk passed with JDK Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04
+1 💚 compile 0m 40s trunk passed with JDK Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04
+1 💚 checkstyle 0m 35s trunk passed
+1 💚 mvnsite 0m 45s trunk passed
+1 💚 javadoc 0m 36s trunk passed with JDK Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04
+1 💚 javadoc 0m 32s trunk passed with JDK Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04
-1 ❌ spotbugs 0m 59s /branch-spotbugs-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient-warnings.html hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient in trunk has 2 extant spotbugs warnings.
+1 💚 shadedclient 27m 3s branch has no errors when building and testing our client artifacts.
_ Patch Compile Tests _
-1 ❌ mvninstall 0m 27s /patch-mvninstall-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient.txt hadoop-mapreduce-client-jobclient in the patch failed.
-1 ❌ compile 0m 27s /patch-compile-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient-jdkUbuntu-21.0.7+6-Ubuntu-0ubuntu120.04.txt hadoop-mapreduce-client-jobclient in the patch failed with JDK Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04.
-1 ❌ javac 0m 27s /patch-compile-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient-jdkUbuntu-21.0.7+6-Ubuntu-0ubuntu120.04.txt hadoop-mapreduce-client-jobclient in the patch failed with JDK Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04.
-1 ❌ compile 0m 28s /patch-compile-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient-jdkUbuntu-17.0.15+6-Ubuntu-0ubuntu120.04.txt hadoop-mapreduce-client-jobclient in the patch failed with JDK Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04.
-1 ❌ javac 0m 28s /patch-compile-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient-jdkUbuntu-17.0.15+6-Ubuntu-0ubuntu120.04.txt hadoop-mapreduce-client-jobclient in the patch failed with JDK Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04.
-1 ❌ blanks 0m 0s /blanks-eol.txt The patch has 22 line(s) that end in blanks. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply
-0 ⚠️ checkstyle 0m 22s /results-checkstyle-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient.txt hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient: The patch generated 11 new + 81 unchanged - 0 fixed = 92 total (was 81)
-1 ❌ mvnsite 0m 30s /patch-mvnsite-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient.txt hadoop-mapreduce-client-jobclient in the patch failed.
+1 💚 javadoc 0m 22s the patch passed with JDK Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04
+1 💚 javadoc 0m 22s the patch passed with JDK Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04
-1 ❌ spotbugs 0m 27s /patch-spotbugs-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient.txt hadoop-mapreduce-client-jobclient in the patch failed.
-1 ❌ shadedclient 15m 15s patch has errors when building and testing our client artifacts.
_ Other Tests _
-1 ❌ unit 0m 32s /patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient.txt hadoop-mapreduce-client-jobclient in the patch failed.
+1 💚 asflicense 0m 32s The patch does not generate ASF License warnings.
84m 38s
Subsystem Report/Notes
Docker ClientAPI=1.51 ServerAPI=1.51 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8052/1/artifact/out/Dockerfile
GITHUB PR #8052
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname Linux 2a3dd83924c4 5.15.0-156-generic #166-Ubuntu SMP Sat Aug 9 00:02:46 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision trunk / df70e0f
Default Java Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04
Multi-JDK versions /usr/lib/jvm/java-21-openjdk-amd64:Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-17-openjdk-amd64:Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8052/1/testReport/
Max. process+thread count 626 (vs. ulimit of 5500)
modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient U: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8052/1/console
versions git=2.25.1 maven=3.9.11 spotbugs=4.9.7
Powered by Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

@hadoop-yetus
Copy link

💔 -1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 0m 31s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 1s No case conflicting files found.
+0 🆗 codespell 0m 0s codespell was not available.
+0 🆗 detsecrets 0m 0s detect-secrets was not available.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
+1 💚 test4tests 0m 0s The patch appears to include 14 new or modified test files.
_ trunk Compile Tests _
+1 💚 mvninstall 36m 36s trunk passed
+1 💚 compile 0m 37s trunk passed with JDK Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04
+1 💚 compile 0m 36s trunk passed with JDK Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04
+1 💚 checkstyle 0m 34s trunk passed
+1 💚 mvnsite 0m 42s trunk passed
+1 💚 javadoc 0m 37s trunk passed with JDK Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04
+1 💚 javadoc 0m 32s trunk passed with JDK Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04
-1 ❌ spotbugs 1m 0s /branch-spotbugs-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient-warnings.html hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient in trunk has 2 extant spotbugs warnings.
+1 💚 shadedclient 26m 2s branch has no errors when building and testing our client artifacts.
_ Patch Compile Tests _
-1 ❌ mvninstall 0m 29s /patch-mvninstall-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient.txt hadoop-mapreduce-client-jobclient in the patch failed.
-1 ❌ compile 0m 27s /patch-compile-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient-jdkUbuntu-21.0.7+6-Ubuntu-0ubuntu120.04.txt hadoop-mapreduce-client-jobclient in the patch failed with JDK Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04.
-1 ❌ javac 0m 27s /patch-compile-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient-jdkUbuntu-21.0.7+6-Ubuntu-0ubuntu120.04.txt hadoop-mapreduce-client-jobclient in the patch failed with JDK Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04.
-1 ❌ compile 0m 27s /patch-compile-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient-jdkUbuntu-17.0.15+6-Ubuntu-0ubuntu120.04.txt hadoop-mapreduce-client-jobclient in the patch failed with JDK Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04.
-1 ❌ javac 0m 27s /patch-compile-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient-jdkUbuntu-17.0.15+6-Ubuntu-0ubuntu120.04.txt hadoop-mapreduce-client-jobclient in the patch failed with JDK Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04.
-1 ❌ blanks 0m 0s /blanks-eol.txt The patch has 22 line(s) that end in blanks. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply
-0 ⚠️ checkstyle 0m 22s /results-checkstyle-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient.txt hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient: The patch generated 11 new + 81 unchanged - 0 fixed = 92 total (was 81)
-1 ❌ mvnsite 0m 29s /patch-mvnsite-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient.txt hadoop-mapreduce-client-jobclient in the patch failed.
+1 💚 javadoc 0m 23s the patch passed with JDK Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04
+1 💚 javadoc 0m 22s the patch passed with JDK Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04
-1 ❌ spotbugs 0m 29s /patch-spotbugs-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient.txt hadoop-mapreduce-client-jobclient in the patch failed.
-1 ❌ shadedclient 15m 25s patch has errors when building and testing our client artifacts.
_ Other Tests _
-1 ❌ unit 0m 32s /patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient.txt hadoop-mapreduce-client-jobclient in the patch failed.
+1 💚 asflicense 0m 34s The patch does not generate ASF License warnings.
86m 45s
Subsystem Report/Notes
Docker ClientAPI=1.51 ServerAPI=1.51 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8052/2/artifact/out/Dockerfile
GITHUB PR #8052
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname Linux 6adfdfb5f86e 5.15.0-156-generic #166-Ubuntu SMP Sat Aug 9 00:02:46 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision trunk / e8aea80
Default Java Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04
Multi-JDK versions /usr/lib/jvm/java-21-openjdk-amd64:Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-17-openjdk-amd64:Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8052/2/testReport/
Max. process+thread count 611 (vs. ulimit of 5500)
modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient U: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8052/2/console
versions git=2.25.1 maven=3.9.11 spotbugs=4.9.7
Powered by Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

@hadoop-yetus
Copy link

💔 -1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 0m 32s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+0 🆗 codespell 0m 0s codespell was not available.
+0 🆗 detsecrets 0m 0s detect-secrets was not available.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
+1 💚 test4tests 0m 0s The patch appears to include 14 new or modified test files.
_ trunk Compile Tests _
+1 💚 mvninstall 34m 6s trunk passed
+1 💚 compile 0m 38s trunk passed with JDK Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04
+1 💚 compile 0m 38s trunk passed with JDK Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04
+1 💚 checkstyle 0m 34s trunk passed
+1 💚 mvnsite 0m 45s trunk passed
+1 💚 javadoc 0m 36s trunk passed with JDK Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04
+1 💚 javadoc 0m 32s trunk passed with JDK Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04
-1 ❌ spotbugs 0m 58s /branch-spotbugs-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient-warnings.html hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient in trunk has 2 extant spotbugs warnings.
+1 💚 shadedclient 25m 52s branch has no errors when building and testing our client artifacts.
_ Patch Compile Tests _
+1 💚 mvninstall 0m 29s the patch passed
+1 💚 compile 0m 31s the patch passed with JDK Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04
+1 💚 javac 0m 31s the patch passed
+1 💚 compile 0m 31s the patch passed with JDK Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04
+1 💚 javac 0m 31s the patch passed
-1 ❌ blanks 0m 0s /blanks-eol.txt The patch has 22 line(s) that end in blanks. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply
-0 ⚠️ checkstyle 0m 22s /results-checkstyle-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient.txt hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient: The patch generated 10 new + 81 unchanged - 0 fixed = 91 total (was 81)
+1 💚 mvnsite 0m 32s the patch passed
+1 💚 javadoc 0m 23s the patch passed with JDK Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04
+1 💚 javadoc 0m 22s the patch passed with JDK Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04
+1 💚 spotbugs 0m 54s the patch passed
+1 💚 shadedclient 25m 50s patch has no errors when building and testing our client artifacts.
_ Other Tests _
+1 💚 unit 109m 13s hadoop-mapreduce-client-jobclient in the patch passed.
+1 💚 asflicense 0m 42s The patch does not generate ASF License warnings.
207m 0s
Subsystem Report/Notes
Docker ClientAPI=1.51 ServerAPI=1.51 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8052/3/artifact/out/Dockerfile
GITHUB PR #8052
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname Linux 57c089f9e1d8 5.15.0-156-generic #166-Ubuntu SMP Sat Aug 9 00:02:46 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision trunk / 9683eb1
Default Java Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04
Multi-JDK versions /usr/lib/jvm/java-21-openjdk-amd64:Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-17-openjdk-amd64:Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8052/3/testReport/
Max. process+thread count 1239 (vs. ulimit of 5500)
modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient U: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8052/3/console
versions git=2.25.1 maven=3.9.11 spotbugs=4.9.7
Powered by Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

MapReduce trunk

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@zczhao0809 @hadoop-yetus