Dynamically load OpenSSL (optionally) #5974

ethomson · 2021-08-07T15:22:08Z

The OpenSSL dependency is a painful one for people who want to distribute pre-built binaries for a variety of Linux distributions. LibGit2Sharp is an example of this, and they have taken the steps to add their own https backend using .NET Core calls. This provides a second option for them.

bording · 2021-08-07T19:13:15Z

src/streams/openssl.c

+{
+	int err = 0;
+
+	if ((openssl_handle = dlopen("libssl.so.1.1", RTLD_NOW)) == NULL) {


I'm not sure this is flexible enough to work with all linux distros. For example, I recall that RHEL derivatives have a completely different naming scheme for their OpenSSL libraries. This StackExchange answer gives an example of what I'm talking about. The RHEL library is named libssl.so.10.

Would doing something like what is suggested in that answer be possible to dynamically look up the name?

Yeah, we can definitely load libssl.so.10. The tricky bit is going to be the API differences over the years, and identifying when to fallback to them because there's no "give me your version number" API in ye olde days. I think that we can determine this based on identifying whether symbols exist or not, but it's not going to be particularly pretty. 😭

I spent some more time looking into this, and it does seem that in the newest versions, RHEL/CentOS have decided to ship OpenSSL 1.1 with standard naming scheme, so they do have libssl.so.1.1. However, looking at .NET 5 - Supported OS versions, Microsoft is still supporting RHEL/CentOS 7, which just has the non-standard 1.0 name for the library. It doesn't look like this is changing for .NET 6 either.

Would it be possible to attempt to open libssl.so.1.1 and if that fails try libssl.so.10 and use that to determine which versions of the APIs to call?

Would it be possible to attempt to open libssl.so.1.1 and if that fails try libssl.so.10 and use that to determine which versions of the APIs to call?

Yes - this would be possible. But it's not yet clear to me whether libssl.so.10 always corresponds to a particular ABI - on my centos 7.5.1804 installation this is a link to libssl.1.0.2k but I don't know if that's a guarantee across all centos 7 minors and the corresponding rhel releases. Need to do a little more investigation.

boretrk · 2021-08-18T18:26:51Z

With dynamic loading it could make sense to make OpenSSL optional.

If git_openssl_stream_global_init is postponed until the first time it is needed it should be possible to use the same package with or without OpenSSL.

Perhaps a package maintainer can mark OpenSSL as recommended instead of required, and it should be possible to install without having to restart libgit2.
Another benefit is that the loading/linking overhead won't happen for offline usage.

diff --git a/src/libgit2.c b/src/libgit2.c
index 089c83590..908d14237 100644
--- a/src/libgit2.c
+++ b/src/libgit2.c
@@ -82,7 +82,6 @@ int git_libgit2_init(void)
                git_merge_driver_global_init,
                git_transport_ssh_global_init,
                git_stream_registry_global_init,
-               git_openssl_stream_global_init,
                git_mbedtls_stream_global_init,
                git_mwindow_global_init,
                git_pool_global_init,
diff --git a/src/streams/openssl.c b/src/streams/openssl.c
index 919bbe48a..1bedf4d6f 100644
--- a/src/streams/openssl.c
+++ b/src/streams/openssl.c
@@ -1005,6 +1005,9 @@ static int openssl_stream_wrap(
 {
        openssl_stream *st;
 
+       if (git__ssl_ctx == NULL && git_openssl_stream_global_init() != 0)
+               return -1;
+
        GIT_ASSERT_ARG(out);
        GIT_ASSERT_ARG(in);
        GIT_ASSERT_ARG(host);
@@ -1067,6 +1070,9 @@ int git_openssl_stream_new(git_stream **out, const char *host, const char *port)
 
 int git_openssl__set_cert_location(const char *file, const char *path)
 {
+       if (git__ssl_ctx == NULL && git_openssl_stream_global_init() != 0)
+               return -1;
+
        if (_SSL_CTX_load_verify_locations(git__ssl_ctx, file, path) == 0) {
                char errmsg[256];

ethomson · 2021-08-18T18:53:56Z

If git_openssl_stream_global_init is postponed until the first time it is needed it should be possible to use the same package with or without OpenSSL.

Yes, I've been thinking that as well. My first effort has been getting this working across multiple OpenSSL versions which has been 😡

I'm a bit stuck on our containerized build at the moment. My build on CentOS 7 seems to fail to find libssh2 when running in CI, but works great when I run it locally. Reproducibility was supposed to be the entire point of this dockerized build setup, so I'm both frustrated and stumped as to what's going on.

I can unblock this by just turning off ssh support for those builds, I suppose, but I won't have a chance to get this next iteration pushed up until the weekend.

After it's working across multiple different OpenSSL versions, I can think about delaying that dynamic load and init step until it's necessary.

Refactor the OpenSSL stream implementation so that the legacy code is better abstracted. This will enable future development.

Provide an interface around OpenSSL to dynamically load the libraries and symbols, so that users can distribute a libgit2 library that is not linked directly against OpenSSL. This enables users to target multiple distributions with a single binary. This mechanism is optional and disabled by default. Configure cmake with -DUSE_HTTPS=OpenSSL-Dynamic to use it.

dlopen sets up some thread-local state that isn't cleaned up by `dlclose`. Additionally, now that we're linking against different versions of libssh2 and OpenSSL, we're seeing different leak signatures.

The ntlmclient dependency can now dynamically load OpenSSL.

Defer dlopen until it's needed when dynamically loading OpenSSL libraries.

Add Xenial, Bionic, CentOS 7 and 8 workflows with OpenSSL-Dynamic builds nightly.

ethomson · 2021-08-24T22:28:20Z

I'm going to go ahead and merge this, I feel like it's not the prettiest of solutions but it's reasonable, it's seen a lot of local testing against different OpenSSL versions > 1.0 and has seen a lot of CI action across multiple platforms.

bording · 2021-08-24T22:34:02Z

@ethomson Since this is merged, do you feel it's ready to pull into LibGit2Sharp's native binaries package and then revert the managed HTTPS stuff, or would it be better to wait for it be part of a libgit2 release?

ethomson · 2021-08-25T00:43:00Z

@bording I'm feeling good about this but I'm also going to push for a v1.2.0 to this weekend. I want to pull in a few more PRs (esp around midx) and then 🚢

bording reviewed Aug 7, 2021

View reviewed changes

bording mentioned this pull request Aug 16, 2021

Update libgit2 to v1.1.1 libgit2/libgit2sharp.nativebinaries#114

Merged

ethomson force-pushed the ethomson/dlopen_ssl branch 3 times, most recently from 8741037 to 33224c9 Compare August 22, 2021 14:27

bording mentioned this pull request Aug 24, 2021

Update LibGit2 to version 1.1.1 libgit2/libgit2sharp#1905

Merged

ethomson added 2 commits August 24, 2021 14:09

openssl: separate legacy api

150eddd

Refactor the OpenSSL stream implementation so that the legacy code is better abstracted. This will enable future development.

ethomson force-pushed the ethomson/dlopen_ssl branch from b87d84d to 05c1487 Compare August 24, 2021 18:09

ethomson mentioned this pull request Aug 24, 2021

[CMake] ntlmclient. missing headers have been added. #5973

Closed

4 tasks

ethomson added 6 commits August 24, 2021 16:23

valgrind: suppress leaks in dlopen and newer libraries

c9b80c2

dlopen sets up some thread-local state that isn't cleaned up by `dlclose`. Additionally, now that we're linking against different versions of libssh2 and OpenSSL, we're seeing different leak signatures.

ntlmclient: update to ntlmclient 0.9.1

5158b0b

The ntlmclient dependency can now dynamically load OpenSSL.

tests: show errors when libgit2_init fails

e265eb2

openssl: dynamically load on macOS

aad497c

openssl: lazily load libraries when dynamically loading

9fce506

Defer dlopen until it's needed when dynamically loading OpenSSL libraries.

ci: run dynamically loaded openssl workflows nightly

314469f

Add Xenial, Bionic, CentOS 7 and 8 workflows with OpenSSL-Dynamic builds nightly.

ethomson force-pushed the ethomson/dlopen_ssl branch from 429fe4b to 314469f Compare August 24, 2021 20:23

ethomson merged commit c8f4b56 into main Aug 24, 2021

ethomson mentioned this pull request Aug 25, 2021

commit-graph: Add a way to write commit-graph files #5778

Merged

lolgear mentioned this pull request Sep 4, 2021

[CMake] ntlmclient. Headers for common crypto are excluded in Xcode target. #5969

Closed

ethomson deleted the ethomson/dlopen_ssl branch September 5, 2021 12:44

jkotas mentioned this pull request Apr 7, 2022

Simplify RID Model dotnet/designs#260

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Dynamically load OpenSSL (optionally) #5974

Dynamically load OpenSSL (optionally) #5974

Uh oh!

ethomson commented Aug 7, 2021

Uh oh!

bording Aug 7, 2021

Uh oh!

ethomson Aug 9, 2021

Uh oh!

bording Aug 9, 2021

Uh oh!

ethomson Aug 10, 2021

Uh oh!

boretrk commented Aug 18, 2021

Uh oh!

ethomson commented Aug 18, 2021

Uh oh!

ethomson commented Aug 24, 2021

Uh oh!

bording commented Aug 24, 2021

Uh oh!

ethomson commented Aug 25, 2021

Uh oh!

Uh oh!

Dynamically load OpenSSL (optionally) #5974

Dynamically load OpenSSL (optionally) #5974

Uh oh!

Conversation

ethomson commented Aug 7, 2021

Uh oh!

bording Aug 7, 2021

Choose a reason for hiding this comment

Uh oh!

ethomson Aug 9, 2021

Choose a reason for hiding this comment

Uh oh!

bording Aug 9, 2021

Choose a reason for hiding this comment

Uh oh!

ethomson Aug 10, 2021

Choose a reason for hiding this comment

Uh oh!

boretrk commented Aug 18, 2021

Uh oh!

ethomson commented Aug 18, 2021

Uh oh!

ethomson commented Aug 24, 2021

Uh oh!

bording commented Aug 24, 2021

Uh oh!

ethomson commented Aug 25, 2021

Uh oh!

Uh oh!