Thanks to visit codestin.com
Credit goes to github.com

Skip to content

HDFS-16419. Make HDFS data transfer tools cross platform #3873

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 6 commits into from
Jan 12, 2022
Merged

HDFS-16419. Make HDFS data transfer tools cross platform #3873

merged 6 commits into from
Jan 12, 2022

Conversation

@GauthamBanasandra
Copy link
Member

@GauthamBanasandra GauthamBanasandra commented Jan 9, 2022 •
edited
Loading

Description of PR

The source files for hdfs_copyToLocal and hdfs_moveToLocal uses getopt for parsing the command line arguments. getopt is available only on Linux and thus, isn't cross platform. We need to replace getopt with boost::program_options to make these tools cross platform.

How was this patch tested?

Both of the tools were tested by running locally as follows -

hdfs_copyToLocal

$ ./hdfs_copyToLocal ~/data.txt /tmp/data.txt

Command

hdfs_moveToLocal

$ ./hdfs_moveToLocal ~/data.txt /tmp/data.txt

Command

For code changes:

  • Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
  • Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
  • If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
  • If applicable, have you updated the LICENSE, LICENSE-binary, NOTICE-binary files?

@hadoop-yetus
Copy link

hadoop-yetus commented Jan 9, 2022

🎊 +1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 37m 30s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 1s No case conflicting files found.
+0 🆗 codespell 0m 0s codespell was not available.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
+1 💚 test4tests 0m 0s The patch appears to include 6 new or modified test files.
_ trunk Compile Tests _
+1 💚 mvninstall 34m 52s trunk passed
+1 💚 compile 3m 16s trunk passed
+1 💚 mvnsite 0m 31s trunk passed
+1 💚 shadedclient 58m 38s branch has no errors when building and testing our client artifacts.
_ Patch Compile Tests _
+1 💚 mvninstall 0m 17s the patch passed
+1 💚 compile 3m 8s the patch passed
+1 💚 cc 3m 8s the patch passed
+1 💚 golang 3m 8s the patch passed
+1 💚 javac 3m 8s the patch passed
+1 💚 blanks 0m 0s The patch has no blanks issues.
+1 💚 mvnsite 0m 17s the patch passed
+1 💚 shadedclient 18m 42s patch has no errors when building and testing our client artifacts.
_ Other Tests _
+1 💚 unit 32m 22s hadoop-hdfs-native-client in the patch passed.
+1 💚 asflicense 0m 34s The patch does not generate ASF License warnings.
153m 53s
Subsystem Report/Notes
Docker ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3873/1/artifact/out/Dockerfile
GITHUB PR #3873
Optional Tests dupname asflicense compile cc mvnsite javac unit codespell golang
uname Linux 355837b91bca 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision trunk / 0234090
Default Java Red Hat, Inc.-1.8.0_312-b07
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3873/1/testReport/
Max. process+thread count 543 (vs. ulimit of 5500)
modules C: hadoop-hdfs-project/hadoop-hdfs-native-client U: hadoop-hdfs-project/hadoop-hdfs-native-client
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3873/1/console
versions git=2.9.5 maven=3.6.3
Powered by Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

@hadoop-yetus
Copy link

hadoop-yetus commented Jan 9, 2022

🎊 +1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 24m 0s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 1s No case conflicting files found.
+0 🆗 codespell 0m 0s codespell was not available.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
+1 💚 test4tests 0m 0s The patch appears to include 6 new or modified test files.
_ trunk Compile Tests _
+1 💚 mvninstall 23m 29s trunk passed
+1 💚 compile 3m 24s trunk passed
+1 💚 mvnsite 0m 45s trunk passed
+1 💚 shadedclient 49m 50s branch has no errors when building and testing our client artifacts.
_ Patch Compile Tests _
+1 💚 mvninstall 0m 23s the patch passed
+1 💚 compile 3m 19s the patch passed
+1 💚 cc 3m 19s the patch passed
+1 💚 golang 3m 19s the patch passed
+1 💚 javac 3m 19s the patch passed
+1 💚 blanks 0m 0s The patch has no blanks issues.
+1 💚 mvnsite 0m 26s the patch passed
+1 💚 shadedclient 20m 51s patch has no errors when building and testing our client artifacts.
_ Other Tests _
+1 💚 unit 32m 36s hadoop-hdfs-native-client in the patch passed.
+1 💚 asflicense 0m 47s The patch does not generate ASF License warnings.
134m 41s
Subsystem Report/Notes
Docker ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3873/1/artifact/out/Dockerfile
GITHUB PR #3873
Optional Tests dupname asflicense compile cc mvnsite javac unit codespell golang
uname Linux 0f58924ebff6 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision trunk / 0234090
Default Java Red Hat, Inc.-1.8.0_312-b07
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3873/1/testReport/
Max. process+thread count 699 (vs. ulimit of 5500)
modules C: hadoop-hdfs-project/hadoop-hdfs-native-client U: hadoop-hdfs-project/hadoop-hdfs-native-client
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3873/1/console
versions git=2.27.0 maven=3.6.3
Powered by Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

@hadoop-yetus
Copy link

hadoop-yetus commented Jan 10, 2022

🎊 +1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 12m 7s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+0 🆗 codespell 0m 0s codespell was not available.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
+1 💚 test4tests 0m 0s The patch appears to include 6 new or modified test files.
_ trunk Compile Tests _
+1 💚 mvninstall 28m 45s trunk passed
+1 💚 compile 2m 55s trunk passed
+1 💚 mvnsite 0m 21s trunk passed
+1 💚 shadedclient 59m 3s branch has no errors when building and testing our client artifacts.
_ Patch Compile Tests _
+1 💚 mvninstall 0m 17s the patch passed
+1 💚 compile 3m 4s the patch passed
+1 💚 cc 3m 4s the patch passed
+1 💚 golang 3m 4s the patch passed
+1 💚 javac 3m 4s the patch passed
+1 💚 blanks 0m 0s The patch has no blanks issues.
+1 💚 mvnsite 0m 22s the patch passed
+1 💚 shadedclient 27m 30s patch has no errors when building and testing our client artifacts.
_ Other Tests _
+1 💚 unit 32m 23s hadoop-hdfs-native-client in the patch passed.
+1 💚 asflicense 0m 33s The patch does not generate ASF License warnings.
137m 38s
Subsystem Report/Notes
Docker ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3873/1/artifact/out/Dockerfile
GITHUB PR #3873
Optional Tests dupname asflicense compile cc mvnsite javac unit codespell golang
uname Linux 3290e4277db8 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision trunk / 0234090
Default Java Debian-11.0.13+8-post-Debian-1deb10u1
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3873/1/testReport/
Max. process+thread count 720 (vs. ulimit of 5500)
modules C: hadoop-hdfs-project/hadoop-hdfs-native-client U: hadoop-hdfs-project/hadoop-hdfs-native-client
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3873/1/console
versions git=2.20.1 maven=3.6.0
Powered by Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

@hadoop-yetus
Copy link

hadoop-yetus commented Jan 10, 2022

🎊 +1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 0m 44s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+0 🆗 codespell 0m 1s codespell was not available.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
+1 💚 test4tests 0m 0s The patch appears to include 6 new or modified test files.
_ trunk Compile Tests _
+1 💚 mvninstall 21m 24s trunk passed
+1 💚 compile 3m 10s trunk passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04
+1 💚 compile 3m 12s trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚 mvnsite 0m 27s trunk passed
+1 💚 shadedclient 47m 58s branch has no errors when building and testing our client artifacts.
_ Patch Compile Tests _
+1 💚 mvninstall 0m 17s the patch passed
+1 💚 compile 3m 7s the patch passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04
+1 💚 cc 3m 7s the patch passed
+1 💚 golang 3m 7s the patch passed
+1 💚 javac 3m 7s the patch passed
+1 💚 compile 3m 7s the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚 cc 3m 7s the patch passed
+1 💚 golang 3m 7s the patch passed
+1 💚 javac 3m 7s the patch passed
+1 💚 blanks 0m 0s The patch has no blanks issues.
+1 💚 mvnsite 0m 19s the patch passed
+1 💚 shadedclient 19m 19s patch has no errors when building and testing our client artifacts.
_ Other Tests _
+1 💚 unit 32m 8s hadoop-hdfs-native-client in the patch passed.
+1 💚 asflicense 0m 34s The patch does not generate ASF License warnings.
109m 45s
Subsystem Report/Notes
Docker ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3873/1/artifact/out/Dockerfile
GITHUB PR #3873
Optional Tests dupname asflicense compile cc mvnsite javac unit codespell golang
uname Linux 61dc9f89e9ee 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision trunk / 0234090
Default Java Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
Multi-JDK versions /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3873/1/testReport/
Max. process+thread count 549 (vs. ulimit of 5500)
modules C: hadoop-hdfs-project/hadoop-hdfs-native-client U: hadoop-hdfs-project/hadoop-hdfs-native-client
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3873/1/console
versions git=2.25.1 maven=3.6.3
Powered by Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

@GauthamBanasandra GauthamBanasandra marked this pull request as ready for review January 10, 2022 17:02
@GauthamBanasandra GauthamBanasandra merged commit 271ac0e into apache:trunk Jan 12, 2022
@GauthamBanasandra GauthamBanasandra deleted the cp-mv-x-platform branch January 12, 2022 14:27
HarshitGupta11 pushed a commit to HarshitGupta11/hadoop that referenced this pull request Nov 28, 2022
@GauthamBanasandra
HDFS-16419. Make HDFS data transfer tools cross platform (apache#3873)
15d2d22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@goiri goiri goiri approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@GauthamBanasandra @hadoop-yetus @goiri