HADOOP-17337. S3A NetworkBinding has a runtime dependency on shaded httpclient #2599
Conversation
…lient. Adds another class with the dependencies, uses reflection to load and invoke that. Change-Id: Iaad9ede15dc6ac3240cba3dfa80c79825dbd007c
|
testing: done the unit tests but not a full live run (which it will need) @cwensel -this look good to you? |
|
💔 -1 overall
This message was automatically generated. |
|
@steveloughran without actually running it,
|
|
if it's not found you don't get any attempts to patch in the wildfly/openssl library. HTTPS connections may be slower. You will stay with the java https factory. I think you may retain the inefficient algorithm which came with java8; there's a bit of fixup in the hadoop-common lib to strip out one of them. Other than that, if you have wildfly and openssl on the CP, you won't see any speedup, because you won't get that native library binding that wildfly/openssl binding has been a PITA; very brittle. I don't want to go near it if I can help it, and doing the fixup for unshaded JARs will add a codepath which will not get tested during normal builds. We have enough problems that the JAR and various openssl releases have been incompatible, making it brittle on some linux images. The fact that an azure SDK kept the old widfly classes hidden unshaded in its JAR made this very hard to eliminate as source of stack traces. Given you want to use the unshaded JARs for lighter images, I think you'll have to accept that a price of that is "no speedup from native openssl". That's all. What is important is: you won't get linkage errors any more |
|
this is an unfortunate tradeoff. I'll gladly opt for improved runtime performance vs improved build and deployment performance. is it not possible, if not reasonable, to have both by adding a new class that couples to the unshaded class in tandem to the current proposed class that couples to the shaded? this seems like a reasonable tax on the choice to use a shaded library and coupling to its internals. |
yes, if someone plans to do it. They'd need to do 100% reflection binding (this code compiles against the shaded one) and presumably there'd be e a binding mechanism which switches between them. I'm not going to go near this. What this patch tries to do is make your deployments work. As to why we use the shaded library, you have to look at the dependencies of aws sdk, version problems we've had with json parsers, httpclient libs and the like to understand we've got good reason to say "use the shaded SDK". There's an implicit "all issues which can be fixed by "move to shaded AWS SDK" will be closed as wontfix". We've chosen to use the shaded SDK to make those version problems go away. It's heavy, but makes versions manageable. I don't want to force people onto going onto the unshaded JARs if they really want to (hence this patch), but I'm not personally worried about the fact there may be a performance impact. Sorry. |
|
style whitespace |
|
I fully understand the situation. And appreciate the time and effort put into this issue. |
|
LGTM +1. |
|
Thanks. Merging |
…ttpclient. (#2599) Contributed by Steve Loughran. Change-Id: I0471322fc88d8bc3896ac439aefb31e6a856936c
… on shaded httpclient. (apache#2599) Contributed by Steve Loughran. Change-Id: I0471322fc88d8bc3896ac439aefb31e6a856936c
Adds another class with the dependencies, uses reflection to load and invoke
that.
Change-Id: Iaad9ede15dc6ac3240cba3dfa80c79825dbd007c