Thanks to visit codestin.com
Credit goes to github.com

Skip to content

HADOOP-18227. Add input stream IOStats for vectored IO api in S3A. #4636

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 2 commits into from
Jul 28, 2022
Merged

HADOOP-18227. Add input stream IOStats for vectored IO api in S3A. #4636

merged 2 commits into from
Jul 28, 2022

Conversation

@mukund-thakur
Copy link
Contributor

@mukund-thakur mukund-thakur commented Jul 26, 2022 •
edited
Loading

part of HADOOP-18103.

Description of PR

Adding input stream IOStats for vectored IO api in S3A.

How was this patch tested?

Added new tests to assert the newly added stats. Ran the whole S3A IT suite using us-east-2 bucket.

For code changes:

  • Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
  • Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
  • If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
  • If applicable, have you updated the LICENSE, LICENSE-binary, NOTICE-binary files?

@mukund-thakur mukund-thakur force-pushed the HADOOP-18227-vectored-io-iostats branch from 1751f8f to 4966eef Compare July 26, 2022 21:28
@mukund-thakur
Copy link
Contributor Author

CC @mehakmeet

@hadoop-yetus
Copy link

🎊 +1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 0m 39s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+0 🆗 codespell 0m 0s codespell was not available.
+0 🆗 detsecrets 0m 0s detect-secrets was not available.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
+1 💚 test4tests 0m 0s The patch appears to include 5 new or modified test files.
_ trunk Compile Tests _
+0 🆗 mvndep 15m 11s Maven dependency ordering for branch
+1 💚 mvninstall 25m 19s trunk passed
+1 💚 compile 23m 8s trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 💚 compile 20m 45s trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚 checkstyle 4m 23s trunk passed
+1 💚 mvnsite 3m 45s trunk passed
+1 💚 javadoc 3m 2s trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 💚 javadoc 2m 42s trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚 spotbugs 5m 5s trunk passed
+1 💚 shadedclient 22m 26s branch has no errors when building and testing our client artifacts.
_ Patch Compile Tests _
+0 🆗 mvndep 0m 34s Maven dependency ordering for patch
+1 💚 mvninstall 1m 46s the patch passed
+1 💚 compile 22m 23s the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 💚 javac 22m 23s the patch passed
+1 💚 compile 20m 43s the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚 javac 20m 43s the patch passed
+1 💚 blanks 0m 0s The patch has no blanks issues.
-0 ⚠️ checkstyle 4m 42s /results-checkstyle-root.txt root: The patch generated 8 new + 14 unchanged - 0 fixed = 22 total (was 14)
+1 💚 mvnsite 3m 47s the patch passed
+1 💚 javadoc 2m 51s the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 💚 javadoc 2m 41s the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚 spotbugs 5m 13s the patch passed
+1 💚 shadedclient 22m 41s patch has no errors when building and testing our client artifacts.
_ Other Tests _
+1 💚 unit 18m 43s hadoop-common in the patch passed.
+1 💚 unit 3m 14s hadoop-aws in the patch passed.
+1 💚 asflicense 1m 37s The patch does not generate ASF License warnings.
242m 52s
Subsystem Report/Notes
Docker ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4636/1/artifact/out/Dockerfile
GITHUB PR #4636
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname Linux 2ba262978d83 4.15.0-156-generic #163-Ubuntu SMP Thu Aug 19 23:31:58 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision trunk / 4966eef
Default Java Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
Multi-JDK versions /usr/lib/jvm/java-11-openjdk-amd64:Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4636/1/testReport/
Max. process+thread count 1259 (vs. ulimit of 5500)
modules C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: .
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4636/1/console
versions git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

steveloughran
steveloughran requested changes Jul 27, 2022
View reviewed changes
Copy link
Contributor

@steveloughran steveloughran left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM; some minor tweaks

...oop-common/src/test/java/org/apache/hadoop/fs/contract/AbstractContractVectoredReadTest.java Outdated
protected static final String VECTORED_READ_FILE_NAME = "vectored_file.txt";

private final IntFunction<ByteBuffer> allocate;
protected final IntFunction<ByteBuffer> allocate;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

keep private and add getters

@hadoop-yetus
Copy link

🎊 +1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 0m 43s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+0 🆗 codespell 0m 1s codespell was not available.
+0 🆗 detsecrets 0m 1s detect-secrets was not available.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
+1 💚 test4tests 0m 0s The patch appears to include 5 new or modified test files.
_ trunk Compile Tests _
+0 🆗 mvndep 15m 12s Maven dependency ordering for branch
+1 💚 mvninstall 25m 26s trunk passed
+1 💚 compile 23m 6s trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 💚 compile 20m 41s trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚 checkstyle 4m 20s trunk passed
+1 💚 mvnsite 3m 45s trunk passed
+1 💚 javadoc 3m 2s trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 💚 javadoc 2m 43s trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚 spotbugs 5m 4s trunk passed
+1 💚 shadedclient 22m 29s branch has no errors when building and testing our client artifacts.
_ Patch Compile Tests _
+0 🆗 mvndep 0m 34s Maven dependency ordering for patch
+1 💚 mvninstall 1m 46s the patch passed
+1 💚 compile 22m 17s the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 💚 javac 22m 17s the patch passed
+1 💚 compile 20m 43s the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚 javac 20m 43s the patch passed
+1 💚 blanks 0m 0s The patch has no blanks issues.
+1 💚 checkstyle 4m 17s the patch passed
+1 💚 mvnsite 3m 43s the patch passed
+1 💚 javadoc 2m 51s the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 💚 javadoc 2m 43s the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚 spotbugs 5m 13s the patch passed
+1 💚 shadedclient 22m 23s patch has no errors when building and testing our client artifacts.
_ Other Tests _
+1 💚 unit 18m 47s hadoop-common in the patch passed.
+1 💚 unit 3m 26s hadoop-aws in the patch passed.
+1 💚 asflicense 1m 38s The patch does not generate ASF License warnings.
242m 10s
Subsystem Report/Notes
Docker ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4636/2/artifact/out/Dockerfile
GITHUB PR #4636
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname Linux fe0f4d6db424 4.15.0-156-generic #163-Ubuntu SMP Thu Aug 19 23:31:58 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision trunk / f5a69a3
Default Java Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
Multi-JDK versions /usr/lib/jvm/java-11-openjdk-amd64:Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4636/2/testReport/
Max. process+thread count 1259 (vs. ulimit of 5500)
modules C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: .
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4636/2/console
versions git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

steveloughran
steveloughran approved these changes Jul 28, 2022
View reviewed changes
Copy link
Contributor

@steveloughran steveloughran left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1, merge at your leisure

@mukund-thakur mukund-thakur merged commit a5b12c8 into apache:trunk Jul 28, 2022
mukund-thakur added a commit that referenced this pull request Aug 11, 2022
@mukund-thakur
HADOOP-18227. Add input stream IOStats for vectored IO api in S3A. (#…
147a466 
…4636)

part of HADOOP-18103.

Contributed By: Mukund Thakur
HarshitGupta11 pushed a commit to HarshitGupta11/hadoop that referenced this pull request Nov 28, 2022
@mukund-thakur
HADOOP-18227. Add input stream IOStats for vectored IO api in S3A. (a…
06d132a 
…pache#4636)

part of HADOOP-18103.

Contributed By: Mukund Thakur
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@steveloughran steveloughran steveloughran approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@mukund-thakur @hadoop-yetus @steveloughran