This doesn't feel like a good fit to be in the library itself. We use the OS-native cryptography for each platform, and we don't want to encourage statically-linking cryptographic code into the object produced by the library.

But the stream API is made so you can implement any TLS stream however you prefer and use it. IIRC I never got around to exposing registration after the refactoring, but git_tls_stream_new() should use a user-registered TLS stream before looking at the built-in ones.

I don't think we'd mind building this support as an external (probably shared) library, if there were a reasonably stable exposed API we could hook into for it so we don't need to carry big patches against libgit2 forever. edit: we'll have to try #3465

For context, the Julia language's package manager is now based on libgit2 thanks to the work of @wildart, and we try to build self contained generic binaries for the major platforms. To get https support on linux we have to bundle the specific version of openssl that was used on the build machine, and there's a potential license concern with distributing both openssl and other gpl dependencies (fftw and a few others) due to the incompatible license. We can't rely on a recent enough version of libgit2 to be available everywhere (centos 6 etc), so we compile and include our own build. If there were a way to dynamically register an https backend, either the version of openssl present on the os (without needing users to have root access to get dev headers), or mbedtls which has a clearly gpl compatible license, that would do the trick.

Registering your own HTTP backend has been available for a few years now and it's what Visual Studio and cargo do in order to have more control over what happens there and what to do wrt authentication.

You can take a look at the C# example in the tests or cargo for how to register your own transport. The built-in transports are built the same way as an external one would be, though they do take advantage of sharing code inside the library.

I've also opened #3465 which would allow the application to implement a TLS stream rather than a full blown backend, which should be easier to do.

May I ask for reconsideration of this proposal. Right now, a number of linux distributions (in particular Debian and derivatives) do not ship libgit2 with HTTPS support because of perceived license incompatibilities between openssl and libgit2. Not providing an alternative license-compatible crypto backend by default would essentially require every user to always bring their own https implementation for those platforms, which seems undesirable.

because of perceived license incompatibilities between openssl and libgit2.

That's odd. libgit2 is not GPL, it is GPL with the linking exception. Can somebody from debian-legal clarify why our linking exception does not satisfy the OpenSSL exception suggested by debian-legal:

In addition, as a special exception, the copyright holders give
permission to link the code of portions of this program with the
OpenSSL library under certain conditions as described in each
individual source file, and distribute linked combinations
including the two.
You must obey the GNU General Public License in all respects
for all of the code used other than OpenSSL. If you modify
file(s) with this exception, you may extend this exception to your
version of the file(s), but you are not obligated to do so. If you
do not wish to do so, delete this exception statement from your
version. If you delete this exception statement from all source
files in the program, then also delete it here.

(Except that ours is far more broad, encompassing not just OpenSSL but literally any other program at all.)

cc @petercolberg

I'm very hesitant about merging this into the mainstream if we can avoid it. We already have too much crypto code relying on too many disparate implementations. Having more and trying to maintain it would be a serious bummer on the maintainers.

I am, however, very sympathetic to the fact that this doesn't work on debian. Let's try to find an alternate solution first, though.

@ethomson Yes, that's completely understandable. FWIW, I think we'd be happy to take at least some of the maintenance burden (responding to bug reports here, making sure things keep working and being up-to-date), since we'd be shipping this configuration, at least on linux, though I understand that there would still be a serious maintenance burden left on you.

As was mentioned, we're also fine with shipping our own mbedtls based custom crypto with our own distribution (though right now we'd still have to patch due to #3850), but if libgit2 itself is provided by the system, I think at least some form of https support should be there, since otherwise we'd either ship the https implementation as our own private library or convince the distribution maintainers to ship yet another independent package to interface libgit2 with a different crypto library, neither of which seems ideal. But yes, let's figure out if we can get the Debian situation fixed in a different way first.

@Keno I appreciate your understanding, and thanks for the offer. 😀

The mbedtls folks have also been awesome in the recent past about offering code review on a similar patch to add mbedtls support to libssh2. I can ping them and see if they'd be willing to do so again here, if it would help us get to a solution. Julia would be using and testing this if it were here, and we'd be reporting bugs and submitting patches if it broke. We're not crypto qualified though, so I understand the hesitance.

@tkelman I looked over these patches; other than some whitespace damage/churn in the last patch, this looks pretty reasonable. (Looks like a previous commit added some lines with incorrect whitespace, and then a later commit fixed up the whitespace.)

I'd love to see this change go in. If it does, I'd be happy to help update the packages in Debian to use it.

@wildart Could you rebase these patches onto current HEAD? I would like to test them within Debian.

@wildart's time is in extremely high demand right now to fix critical bugs in the Julia package manager before 0.5.0, so I'd rather someone else adopted these patches if you want to get this done immediately.

I'll happily do the rebasing if @wildart gives me access to his fork, so I can push to this branch.

@Keno You may want to go ahead and work on the rebasing in a separate branch and post a link here, to avoid waiting on access; it can always become a separate pull request. (It's unfortunate that github doesn't allow changing the branch associated with a pull request.)

I rebased my branch: https://github.com/wildart/libgit2/tree/mbedtls

@Keno I gave you access to my fork

Looks like gnutls support was initially added as part of #697 but then removed within the same PR in d3e1367 before it was merged. Was it fully-functional at that stage?

I started doing both at the same time but decided to concentrate on OpenSSL as it is the library that's more widely available. The code has changed quite a bit, so it probably doesn't make a lot of sense to try to port that, but rather add the TLS stream based off of one of the ones we have now.

Since the reason for not linking to OpenSSL on a system-provided library is the license, I would prefer to go with GnuTLS as it's established and solves that problem.

👎 on GnuTLS. GnuTLS has been considered harmful and considered a poor choice. I would very much prefer mbedTLS or NSS.

I don't have a strong opinion on which TLS library to use; any library not based on OpenSSL and its license would work fine. If the already-written code for mbedTLS works, that seems like a fine choice.

@Keno Did you have any success in getting the mbedtls backend working?

@wildart did the rebase, I didn't think anything else was required, sorry if there is and I missed something.

This pull request is still based on commit d8dc2b8 from Oct 7, 2015.

git fetch --force https://github.com/libgit2/libgit2 pull/3462/head:mbedtls

Could you update the corresponding branch so the pull request can be tested and merged?

See https://github.com/wildart/libgit2/tree/mbedtls - I believe there's a github bug that means if you force push to a branch that a PR was opened from after that PR was closed, you can't reopen the PR and it doesn't refresh the diff.

I believe this is a GitHub quirk where updates to the underlying branch are not reflected in a closed PR.

@tkelman, @Keno Could you submit a new pull request? Julia 0.5.0-rc3 with LLVM 3.9-rc2 is now looking good on Debian, with the only remaining blocker being the missing TLS support in libgit2.

Just curious, why can't you use the system's libgit2 together with the system's TLS support in Debian?

On August 23, 2016 4:55:41 PM EDT, Milan Bouchet-Valat [email protected] wrote:

Just curious, why can't you use the system's libgit2 together with the
system's TLS support in Debian?

libgit2 only supports openssl right now; it doesn't yet support any other SSL library.

But why don't you use the system OpenSSL? The GPL allows linking to system libraries even when they aren't GPL-compatible. AFAIK, the problem with Julia is that it would need to ship OpenSSL directly (since the system might not include the right version), but Debian doesn't have to.

@nalimilan Quoting the GPLv2:

However, as a special exception, the source code distributed need not include anything that is normally distributed (in either source or binary form) with the major components (compiler, kernel, and so on) of the operating system on which the executable runs, unless that component itself accompanies the executable.

Two aspects of that exception prevent Debian from using it. First, OpenSSL doesn't qualify as "normally distributed with the major components of the operating system" on Debian. And second, that last bit ("unless that component itself accompanies the executable"). Thus, Debian doesn't allow GPLed software that links to OpenSSL.

The GPLv3's version doesn't have the second part of that, but retains language equivalent to the first part.

Supporting an SSL library that doesn't have that license incompatibility would help.

Ah, too bad. I thought it was "normally distributed with the major components of the operating system" (i.e. installed by default).

Has anyone had a chance to test this? I'm really interested in seeing this support available; if nobody else has tested it already, I'll try building and testing it.

(Also, could someone either reopen this or post a new PR?)

the branch linked above should be the most recent rebase. I was hoping to hear from one of the maintainers whether or not a reopened version of this would be considered.

We did a bit of testing of this in Julia while it was a work in progress pull request, but went with a different approach for now when the reception to getting it merged here was not good. If that changes and there's a chance of getting it upstreamed, we'd certainly switch to using and regularly testing this.

@carlosmn Have you reconsidered adding the mbedTLS backend to libgit2? This issue is severely impacting the usefulness of libgit2 in Debian and Debian derivatives. For example, Debian won’t be able to ship Julia 0.5 until this is resolved, since Julia is not very useful without access to the remote package repository.

@petercolberg @joshtriplett what's more likely to happen from the debian side, building a separate copy of libgit2 that links against openssl that wouldn't be usable by any GPLed software that links to libgit2, or building a separate mbedtls-enabled version of libgit2? The latter would probably also have to be packaged separately right, since I don't think multiple ssl libraries can be linked against at the same time? Or if the latter were available, would debian switch to using that in its standard libgit2 build in order to provide https support in a license-compatible way?

@tkelman If mbedTLS support worked well, most likely the default package of libgit2 would use it.

I just tried this locally (merging the mbedtls branch of https://github.com/wildart/libgit2/ into current master, resolving merge conflicts, and updating for subsequent API changes), and it seems to work. All the tests passed, including the online tests (manually run via libgit2_clar -sonline -v).

I pushed the result to #3935 for review and merging. (Note that the original mbedtls branch doesn't bisect, and I kept my changes separate for review purposes, so I'd recommend a squash-merge.)