HADOOP-17934 ABFS: Make sure the AbfsHttpOperation is non-null before using it #3477

joshelser · 2021-09-23T20:51:14Z

Description of PR

It's possible that, after retries, we will have a null AbfsHttpOperation
inside of the AbfsRestOperation. AbfsClient should be aware of this and
make sure that it doesn't just assume that there's a non-null
HttpOperation (e.g. try to get the HTTP status code).

How was this patch tested?

Just existing unit tests at this point. Haven't looked to see if I can mock a realistic new test case. Have not tried to run ABFS integration tests yet.

For code changes:

Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
If applicable, have you updated the LICENSE, LICENSE-binary, NOTICE-binary files?

…ng it It's possible that, after retries, we will have a null AbfsHttpOperation inside of the AbfsRestOperation. AbfsClient should be aware of this and make sure that it doesn't just assume that there's a non-null HttpOperation (e.g. try to get the HTTP status code).

steveloughran

minor javadoc change.

now: have you run the full abfs test suite? As you know you have to...

...ols/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsRestOperation.java

joshelser · 2021-09-27T15:29:47Z

now: have you run the full abfs test suite? As you know you have to...

Nope, I haven't yet and yes I know I need to run them. I'll be working on figuring that out today.

Thanks for the pre-review Steve!

joshelser · 2021-09-27T19:03:51Z

Well, I got some stuff to run. I made my azure-auth-keys.xml file and ran it against a storage account in East US via ./dev-support/testrun-scripts/runtests.sh

Running the combination: AppendBlob-HNS-OAuth...
----- Test results -----
[INFO] Results:
[INFO]
[WARNING] Tests run: 106, Failures: 0, Errors: 0, Skipped: 26
[INFO] Results:
[INFO]
[WARNING] Tests run: 570, Failures: 0, Errors: 0, Skipped: 570
[INFO] Results:
[INFO]
[WARNING] Tests run: 259, Failures: 0, Errors: 0, Skipped: 259

Time taken: 1 mins 11 secs.
Find test logs for the combination (AppendBlob-HNS-OAuth) in: dev-support/testlogs/2021-09-27_14-56-59/Test-Logs-AppendBlob-HNS-OAuth.txt
Find consolidated test results in: dev-support/testlogs/2021-09-27_14-56-59/Test-Results.txt
----------

Running the combination: HNS-OAuth...
----- Test results -----
[INFO] Results:
[INFO]
[WARNING] Tests run: 106, Failures: 0, Errors: 0, Skipped: 26
[INFO] Results:
[INFO]
[WARNING] Tests run: 570, Failures: 0, Errors: 0, Skipped: 570
[INFO] Results:
[INFO]
[WARNING] Tests run: 259, Failures: 0, Errors: 0, Skipped: 259

Time taken: 1 mins 12 secs.
Find test logs for the combination (HNS-OAuth) in: dev-support/testlogs/2021-09-27_14-56-59/Test-Logs-HNS-OAuth.txt
Find consolidated test results in: dev-support/testlogs/2021-09-27_14-56-59/Test-Results.txt
----------

Running the combination: HNS-SharedKey...
----- Test results -----
[INFO] Results:
[INFO]
[WARNING] Tests run: 106, Failures: 0, Errors: 0, Skipped: 26
[INFO] Results:
[INFO]
[WARNING] Tests run: 570, Failures: 0, Errors: 0, Skipped: 570
[INFO] Results:
[INFO]
[WARNING] Tests run: 259, Failures: 0, Errors: 0, Skipped: 259

Time taken: 1 mins 14 secs.
Find test logs for the combination (HNS-SharedKey) in: dev-support/testlogs/2021-09-27_14-56-59/Test-Logs-HNS-SharedKey.txt
Find consolidated test results in: dev-support/testlogs/2021-09-27_14-56-59/Test-Results.txt
----------

Running the combination: NonHNS-SharedKey...
----- Test results -----
[INFO] Results:
[INFO]
[WARNING] Tests run: 106, Failures: 0, Errors: 0, Skipped: 26
[INFO] Results:
[INFO]
[WARNING] Tests run: 570, Failures: 0, Errors: 0, Skipped: 570
[INFO] Results:
[INFO]
[WARNING] Tests run: 259, Failures: 0, Errors: 0, Skipped: 259

Time taken: 1 mins 12 secs.
Find test logs for the combination (NonHNS-SharedKey) in: dev-support/testlogs/2021-09-27_14-56-59/Test-Logs-NonHNS-SharedKey.txt
Find consolidated test results in: dev-support/testlogs/2021-09-27_14-56-59/Test-Results.txt

Is this the kind of output we expect to see?

steveloughran · 2021-09-28T17:57:24Z

@joshelser -that's a lovely test run result. Happy.

steveloughran

got a q. about assert(), otherwise LGTM

hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsClient.java

joshelser · 2021-09-29T18:06:09Z

Running the combination: AppendBlob-HNS-OAuth...
Running the combination: HNS-OAuth...
Running the combination: HNS-SharedKey...
Running the combination: NonHNS-SharedKey...

Re-ran against East US. One, what looks like a, flakey test:

TestAbfsClientThrottlingAnalyzer.testManySuccessAndErrorsAndWaiting:171->fuzzyValidate:49 The actual value 9 is not within the expected range: [5.60, 8.40].

Seems to fail now and again.

hadoop-yetus · 2021-09-29T19:49:50Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	1m 23s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
-1 ❌	test4tests	0m 0s		The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
			_ trunk Compile Tests _
+1 💚	mvninstall	40m 42s		trunk passed
+1 💚	compile	0m 46s		trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	compile	0m 41s		trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	checkstyle	0m 28s		trunk passed
+1 💚	mvnsite	0m 44s		trunk passed
+1 💚	javadoc	0m 38s		trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	javadoc	0m 34s		trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	spotbugs	1m 21s		trunk passed
+1 💚	shadedclient	26m 7s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 35s		the patch passed
+1 💚	compile	0m 36s		the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	javac	0m 36s		the patch passed
+1 💚	compile	0m 33s		the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	javac	0m 33s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	0m 21s		the patch passed
+1 💚	mvnsite	0m 40s		the patch passed
+1 💚	javadoc	0m 28s		the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	javadoc	0m 23s		the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	spotbugs	1m 15s		the patch passed
+1 💚	shadedclient	22m 38s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	1m 50s		hadoop-azure in the patch passed.
+1 💚	asflicense	0m 29s		The patch does not generate ASF License warnings.
		104m 12s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3477/4/artifact/out/Dockerfile
GITHUB PR	#3477
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell
uname	Linux a8d8698c0ff0 4.15.0-147-generic #151-Ubuntu SMP Fri Jun 18 19:21:19 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `0aa4192`
Default Java	Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3477/4/testReport/
Max. process+thread count	520 (vs. ulimit of 5500)
modules	C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3477/4/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

steveloughran

+1 from me. Yes, some tests are a bit brittle over timing. the fact you discovered that reassures us that you are running the tests though

joshelser · 2021-09-30T14:27:01Z

Thanks so much, Steve!

…e using it (#3477) Contributed by: Josh Elser Change-Id: I24a2e0322d8cae2d72d65c7f3d8a74580a418317

joshelser force-pushed the 17428-abfs-npe branch from 9bae284 to e60faad Compare September 23, 2021 20:52

joshelser changed the title ~~HADOOP-17428 ABFS: Make sure the AbfsHttpOperation is non-null before using it~~ HADOOP-17934 ABFS: Make sure the AbfsHttpOperation is non-null before using it Sep 23, 2021

steveloughran requested changes Sep 24, 2021

View reviewed changes

...ols/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsRestOperation.java Outdated Show resolved Hide resolved

Add a @return to the new method on AbfsRestOperation

73a1c70

joshelser requested a review from steveloughran September 27, 2021 19:05

steveloughran reviewed Sep 28, 2021

View reviewed changes

hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsClient.java Outdated Show resolved Hide resolved

Change from assert to checkArugment

0aa4192

apache deleted a comment from hadoop-yetus Sep 30, 2021

steveloughran approved these changes Sep 30, 2021

View reviewed changes

steveloughran merged commit 6f2fa87 into apache:trunk Sep 30, 2021

asfgit pushed a commit that referenced this pull request Oct 4, 2021

HADOOP-17934. ABFS: Make sure the AbfsHttpOperation is non-null befor…

feeaebe

…e using it (#3477) Contributed by: Josh Elser Change-Id: I24a2e0322d8cae2d72d65c7f3d8a74580a418317

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

HADOOP-17934 ABFS: Make sure the AbfsHttpOperation is non-null before using it #3477

HADOOP-17934 ABFS: Make sure the AbfsHttpOperation is non-null before using it #3477

Uh oh!

joshelser commented Sep 23, 2021

Uh oh!

steveloughran left a comment

Uh oh!

Uh oh!

joshelser commented Sep 27, 2021

Uh oh!

joshelser commented Sep 27, 2021

Uh oh!

steveloughran commented Sep 28, 2021

Uh oh!

steveloughran left a comment

Uh oh!

Uh oh!

joshelser commented Sep 29, 2021

Uh oh!

hadoop-yetus commented Sep 29, 2021

Uh oh!

steveloughran left a comment

Uh oh!

joshelser commented Sep 30, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

HADOOP-17934 ABFS: Make sure the AbfsHttpOperation is non-null before using it #3477

HADOOP-17934 ABFS: Make sure the AbfsHttpOperation is non-null before using it #3477

Uh oh!

Conversation

joshelser commented Sep 23, 2021

Description of PR

How was this patch tested?

For code changes:

Uh oh!

steveloughran left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

joshelser commented Sep 27, 2021

Uh oh!

joshelser commented Sep 27, 2021

Uh oh!

steveloughran commented Sep 28, 2021

Uh oh!

steveloughran left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

joshelser commented Sep 29, 2021

Uh oh!

hadoop-yetus commented Sep 29, 2021

Uh oh!

steveloughran left a comment

Choose a reason for hiding this comment

Uh oh!

joshelser commented Sep 30, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants