Support mbedTLS #3935

joshtriplett · 2016-09-25T07:01:11Z

Based on the mbedtls branch from https://github.com/wildart/libgit2 ,
with merge conflicts fixed, updates for subsequent libgit2 API changes,
and cleanups.

Passes all tests, including online tests.

Patches separated for review, but this does not bisect; squash-merge
recommended.

reload properly certificates in libgit2_options

Separated for review; squash before merging. - Undo unrelated whitespace changes - Fix indentation - Remove commented-out code - Simplify preprocessor conditionals

tkelman · 2016-09-25T12:02:30Z

CMakeLists.txt

@@ -112,6 +112,10 @@ IF (HAVE_STRUCT_STAT_NSEC OR WIN32)
 	OPTION( USE_NSEC		"Care about sub-second file mtimes and ctimes"	ON  )
 ENDIF()

+IF (NOT USE_OPENSSL)
+	OPTION( USE_MBEDTLS     "Link with and use mbedTLS library"             ON )


we should default this to off, otherwise on darwin it'll try to use both the apple crypto and mbedtls if found

I think it needs to default to on, but it needs to get wrapped in the same "not darwin". Will add a patch to fix.

why should it default to on?

it might be possible to use openssl or mbedtls on darwin instead of the apple crypto, but that might not be supported here

It should default to on when not on Darwin and not using OpenSSL, to attempt to autodetect mbedTLS if it doesn't find OpenSSL. If it defaulted to off, you'd have to explicitly enable mbedTLS even if you have mbedTLS installed and OpenSSL not installed.

right that makes sense, but that's what should determine the default, not whether or not the option gets defined at all? though since the openssl option isn't defined at all for darwin, at least this would be consistent

Right; consistency seems preferable here. It might make sense for a future patch to go through CMakeLists.txt, define all options unconditionally, and change all the conditionals to just set the default values. But I don't think this patch should make that unrelated change.

tkelman · 2016-09-25T12:03:43Z

CMakeLists.txt

@@ -283,6 +287,10 @@ ELSE ()
 		FIND_PACKAGE(OpenSSL)
 	ENDIF ()

+	IF (NOT AMIGA AND USE_MBEDTLS)


is amiga relevant? wonder why openssl is gated on NOT AMIGA ...

Not sure. @sba1 made that change in commit c57c4af.

I don't think that it is relevant for the Amiga port right now.

However, the identification of OpenSSL is disabled on Amiga, because FIND_PACKAGE produced wrong results when cross compiling (it added wrong (i.e., native) include paths), at least, if I remember it correctly. Instead, it is assumed that OpenSSL is available on this platform in the (cross) compiler's standard include path and libgit2 is only statically linked.

In essence, special care for Amiga doesn't need to be done in new stuff. From time to time, I'll take care of it and submit pull requests.

tkelman · 2016-09-25T12:08:20Z

I was about to do this since we're very likely to start carrying and using this patch in Julia. We'll effectively be forking libgit2 to maintain and rebase the patch going forward if it doesn't get upstreamed.

Anyway, maybe best to tab-indent the new code for style consistency with the rest of libgit2?

Otherwise, Darwin systems would attempt to build with both their native crypto and mbedTLS.

joshtriplett · 2016-09-25T18:44:42Z

@tkelman Happy to do so as the last step before merging, but that'll completely break diffs with any of the previous code, so I think I should wait for review on this version first.

ethomson · 2016-09-28T04:04:07Z

Interesting. I'll take a look at this as soon as I have some free time. Definitely hoping to unblock debian users.

tkelman · 2016-09-29T20:55:48Z

src/mbedtls_stream.c

+#include "mbedtls/certs.h"
+
+#ifndef OPENSSLDIR
+# define OPENSSLDIR              "/usr/lib/ssl"


cross-linking to JuliaLang/julia#18693, this isn't very uniform across distributions and we might want to come up with a better certificate search mechanism?

How about taking the value for this from a cmake-time variable with /usr/lib/ssl as the default?

I guess that can already be done in CFLAGS (however cmake spells its equivalent) with a -DOPENSSLDIR mostly-equivalently. Maybe this isn't really libgit2's problem to solve.

staticfloat · 2016-10-14T20:23:24Z

The current state of this PR has an important caveat: if SSL certificate files cannot be found, git_libgit2_init() fails. This severely limits the usefulness of the ability for users of libgit2 to use the git_libgit2_opts() function to modify the SSL_CERT_LOCATIONS setting, as we cannot change options before initializing the library.

As it stands, if a user chooses to compile against MbedTLS and wants to create a portable binary (e.g. one that can run on both Ubuntu and CentOS, which store their certificates in very different locations) the software using l]ibgit2 cannot, using the C API, set the locations for libgit2 to search for certificate files. The user must set environment variables in order to change the search path.

I see two possible solutions:

Somehow change the MbedTLS certificate loading to be lazier, allowing the user the chance to change the search path for certificates before they are actually needed.
Somehow change the options setting so that global options like this can be set before init() actually occurs.

Personally, I think the first solution is the vastly superior one. What do you guys think?

libgit2/libgit2#3935 (comment)

Works around libgit2/libgit2#3935 (comment)

carlosmn · 2016-10-16T13:16:19Z

cmake/Modules/FindmbedTLS.cmake

@@ -0,0 +1,64 @@
+# - Try to find mbedTLS
+# Once done this will define


We don't need this file at all, do we? Our CMake script relies on pkg-config rather than trying to guess where it might be installed.

Mbed-TLS/mbedtls#228, so not yet 😉

carlosmn · 2016-10-16T13:18:36Z

script/cibuild.sh

+    CMAKE_INCLUDE_PATH=../mbedtls/include
+    CMAKE_LIBRARY_PATH=../mbedtls/library
+    export CMAKE_INCLUDE_PATH CMAKE_LIBRARY_PATH
+fi


This should be part of the script that installs the dependencies. The Travis rules currently only do it for osx, but we should remove the if there and run it for linux too. The instructions for installing mbedtls should go there.

carlosmn · 2016-10-16T13:20:31Z

src/global.c

+		&& (ret = git_openssl_stream_global_init()) == 0
+#elif GIT_MBEDTLS
+		&& (ret = git_mbedtls_stream_global_init()) == 0
+#endif


We didn't need to ifdef-away the OpenSSL stream init, and we definitely shouldn't need it now. If there is no mbedtls, its init function should no-op. We define an interface and program against it.

carlosmn · 2016-10-16T13:22:32Z

src/global.h

@@ -23,6 +23,12 @@ typedef struct {
 extern SSL_CTX *git__ssl_ctx;
 #endif

+#ifdef GIT_MBEDTLS
+# include "mbedtls/platform.h"
+# include "mbedtls/ssl.h"


Why are these specified as relative imports? Why woldn't <mbedtls/platform.h> and <mbedtls/ssl.h> work?

carlosmn · 2016-10-16T13:23:32Z

src/hash/hash_mbedtls.h

+#define git_hash_global_init() 0
+#define git_hash_ctx_init(ctx) git_hash_init(ctx)
+
+#endif /* INCLUDE_hash_mbedtld_h__ */


The line ending is missing.

carlosmn · 2016-10-16T13:32:00Z

src/mbedtls_stream.c

+#endif
+
+    if (error < 0)
+        return error;


This leaks the stream.

carlosmn · 2016-10-16T14:23:44Z

src/mbedtls_stream.c

+        git__free(ctr_drbg);
+        git__free(mbedtls_entropy);
+        return -1;
+    }


We are we seeding the random number generator for crypto? This is something the library should be asking the operating system whenever it needs a random number.

carlosmn · 2016-10-16T14:57:16Z

src/mbedtls_stream.c

+        return -1;
+    }
+
+    // configure TLSv1


This comment implies that we're setting the minimum SSL/TLS version to be TLS, but there's no code to do this. The documentation for mbedtls says the default is TLS 1.0 so leaving the defaults should be OK, but the comment should be saying explicitly how we're getting the minimum version.

carlosmn · 2016-10-16T15:29:46Z

src/mbedtls_stream.c

+    mbedtls_ssl_conf_authmode(git__ssl_conf, MBEDTLS_SSL_VERIFY_REQUIRED);
+    mbedtls_ssl_conf_rng(git__ssl_conf, mbedtls_ctr_drbg_random, ctr_drbg);
+
+    // find locations for which CA certificates


libgit2 is very much not in the business of guessing where SSL certificates live on everyone's machine. A distribution-provided mbedtls should know already. And if this is being shipped as part of a bundle, the application is the only one which would know what to use.

carlosmn · 2016-10-16T15:46:51Z

I've gone through a few of the higher-level aspects of adding this TLS stream. I have to say, for a library with copy on their page about OpenSSL's bad API, it's own API is disappointing. Only in very specific situations is the user of the library in a better position than the library itself to know where to get a random number.

tkelman · 2016-10-24T21:40:51Z

Thanks for the review @carlosmn.

@joshtriplett will need to include https://github.com/JuliaLang/julia/blob/e6a5cc38700302fe0986c55db8264f6c3673d1c3/deps/patches/libgit2-mbedtls-writer-fix.patch here

tkelman · 2016-12-15T17:48:23Z

ping @joshtriplett any update here?

cdluminate · 2017-03-16T03:40:13Z

Any update? Julia 0.6 will be out soon however Debian still ships 0.4 and blocked by this problem.

tkelman · 2017-03-18T06:31:04Z

Someone will have to own responding to the (much-appreciated) review feedback and updating the code to get it upstreamed. I'm fine for now with carrying the existing patch as-is in Julia, we've been using it this way for some time, but any updates can of course be incorporated in what Julia is using.

tiennou · 2017-03-19T03:12:13Z

I've tried to tackle a few things here : https://github.com/tiennou/libgit2/tree/mbedtls

Mostly, I ended up in agreement of @carlosmn sentiment above. I can understand being "modular", thus forcing the user to provide whatever "entropy pool" it likes, but providing SSL cert bundles is too much. I think the way forward is to remove the "global" CA thing and hope that mbedTLS users will set GIT_OPT_SET_SSL_CERT_LOCATIONS correctly.

@joshtriplett Would you kindly pull that here ? I'd like to know what Travis thinks of the CI changes I made.

joshtriplett · 2017-03-20T15:51:52Z

After working on this a while, I really don't have a lot of confidence in mbedTLS itself as an implementation, and I don't think I'm the right person to keep working on this. @tiennou, please feel free to make a pull request with your version.

tkelman · 2017-03-20T15:56:16Z

Will debian ever build its libgit2 package with https support then?

tiennou · 2017-03-20T18:44:23Z

@joshtriplett 😆 🚌 💥 . FWIW, the sentiment is shared w.r.t. to confidence.

@tkelman Are there any other alternative to mbedTLS that would be Debian-compatible ?

I can't help but wonder how many wrappers were written, collectively 🙄.

tkelman · 2017-03-20T21:19:01Z

What are you not confident about? Julia has switched to mbedtls for our entire network stack and it's working pretty well, certainly better than any other GPL-compatible alternative.

If Debian doesn't plan on enabling https support in their libgit2 packaging any time soon, I'm going to request that julia be removed from Debian rather than kept permanently stuck at an obsolete version.

ethomson · 2017-03-20T21:22:14Z

Same with libgit2. I don't like fielding requests from people who can't clone a repository because there's no underlying crypto stack.

joshtriplett · 2017-03-20T21:27:02Z

I'd like to see this move forward; I just don't think it makes sense for me to be an intermediary between libgit2 and the people actually doing the work. I'm not a crypto developer. If there's someone actively working on this code, I think it makes more sense for them to own the pull request. (I wish that github had a way to transfer ownership of a PR and point it at someone else's branch, rather than having to close it and open a new one.)

tkelman · 2017-03-20T21:30:45Z

Granted mbedtls isn't the only way to accomplish that, debian could just flip a switch in the way they build libgit2 (afaiu most other distros do link theirs against openssl), but I guess there's a lingering legality question about gpl compatibility of other applications that link to libgit2.

@joshtriplett okay makes sense, thanks for working on it as much as you did. We don't currently have anyone working on this code - the original author vanished. I rebase it occasionally but don't understand the code much beyond that it seems to work and passes our tests.

joshtriplett · 2017-03-20T21:57:58Z

@tkelman @tiennou If you're willing to rebase it and similar, please do consider upstreaming your changes; it doesn't seem like there are that many blockers here.

tiennou · 2017-03-20T22:08:30Z

I share @joshtriplett non-crypto background. I was just trying to help get that code in, and I have no interest in that (apart from helping Debian).

@joshtriplett "Allow edits from maintainers" maybe ? I don't know if I'd qualify for that though, but it's worth a shot. If that doesn't work, I'll open a new one.

About confidence : I was looking at @carlosmn comment on the "missing" TLSv1 setup. There's a function for that. Now, what does those constants mean ? I have no idea, and I found comments somewhere where the mapping from those "version numbers" to actual TLS version numbers was apparent. I do realize it's pretty minor, but that's how confidence works 😜.

@tkelman I'll happily try to get it in shape. It's just that I'm developing from a non-Debian OS things like certificate store paths (which macOS has no file-based equivalent of), and have to setup Travis.

tkelman · 2017-03-20T23:08:52Z

we could grant each other collaborator access to our forks and keep it in this pr, or to collaborate on a new one

joshtriplett · 2017-03-24T00:11:03Z

@tiennou

Are there any other alternative to mbedTLS that would be Debian-compatible ?

OpenSSL and the various libraries based on it, soon: https://www.openssl.org/blog/blog/2017/03/22/license/

wildart added 12 commits July 29, 2016 00:16

mbedtls cmake config

04a2b1d

define mbedtls global configuration

cfc27a3

added certificate location setup

d44d020

introduced GIT_MBEDTLS symbol

a5958d0

added git_mbedtls_stream_new

ad2b2e2

proper certificate verification & cleanup of git_mbedtls_stream_new

101ab46

use libmbedcrypto for SHA1

46db15f

added new CI configuration for mbedtls

369d23a

add mbedtls stream initialization

4f8968b

update to 0.24.1

cc156e4

turned on https feature for mbedtls

f814681

better search for CA certificates: using EVs and build-time macros

6b556a6

reload properly certificates in libgit2_options

joshtriplett mentioned this pull request Sep 25, 2016

add support of mbedTLS #3462

Closed

joshtriplett added 3 commits September 25, 2016 00:08

Merge branch 'mbedtls' of https://github.com/wildart/libgit2

a47a6bc

Update for current libgit2 API with struct git_proxy_options

05ca19a

Cleanups for mbedTLS changes

77cdc58

Separated for review; squash before merging. - Undo unrelated whitespace changes - Fix indentation - Remove commented-out code - Simplify preprocessor conditionals

joshtriplett force-pushed the mbedtls branch from c700d04 to 77cdc58 Compare September 25, 2016 07:09

mbedtls: Fix travis build

dab8d43

tkelman reviewed Sep 25, 2016

View reviewed changes

Move USE_MBEDTLS option under not-Darwin condition

81555f5

Otherwise, Darwin systems would attempt to build with both their native crypto and mbedTLS.

tkelman mentioned this pull request Sep 26, 2016

Use mbedtls instead of openssl in libgit2 on Linux JuliaLang/julia#18658

Merged

tkelman reviewed Sep 29, 2016

View reviewed changes

staticfloat added a commit to JuliaLang/julia that referenced this pull request Oct 14, 2016

Workaround

8021275

libgit2/libgit2#3935 (comment)

staticfloat mentioned this pull request Oct 14, 2016

"error initializing LibGit2 module" when CA certificates not found JuliaLang/julia#18693

Closed

staticfloat added a commit to staticfloat/julia that referenced this pull request Oct 15, 2016

Workaround

e2c5608

libgit2/libgit2#3935 (comment)

staticfloat added a commit to JuliaLang/julia that referenced this pull request Oct 15, 2016

Reorder libgit2 initialization and ssl cert location setting

397f38e

Works around libgit2/libgit2#3935 (comment)

staticfloat mentioned this pull request Oct 15, 2016

Fix libgit2 SSL certificate searching JuliaLang/julia#18936

Merged

staticfloat added a commit to staticfloat/julia that referenced this pull request Oct 15, 2016

Reorder libgit2 initialization and ssl cert location setting

3706486

Works around libgit2/libgit2#3935 (comment)

staticfloat added a commit to JuliaLang/julia that referenced this pull request Oct 15, 2016

Reorder libgit2 initialization and ssl cert location setting

971c04c

Works around libgit2/libgit2#3935 (comment)

staticfloat mentioned this pull request Oct 16, 2016

Finalize fix for libgit2 SSL certificates JuliaLang/julia#18963

Closed

staticfloat added a commit to staticfloat/julia that referenced this pull request Oct 16, 2016

Reorder libgit2 initialization and ssl cert location setting

791f9df

Works around libgit2/libgit2#3935 (comment)

carlosmn reviewed Oct 16, 2016

View reviewed changes

tkelman mentioned this pull request Jan 9, 2017

Security of unencrypted git:// protocol JuliaLang/METADATA.jl#7495

Closed

omus mentioned this pull request Mar 14, 2017

SSL_CERT_DIR can cause LibGit2 initialization error JuliaLang/julia#20439

Closed

joshtriplett closed this Mar 20, 2017

tiennou mentioned this pull request Mar 22, 2017

mbedTLS support #4173

Merged

2 tasks

omus mentioned this pull request Jun 29, 2017

Update to LibGit2 v0.26.0 JuliaLang/julia#22614

Merged

3 tasks

		@@ -0,0 +1,64 @@
		# - Try to find mbedTLS
		# Once done this will define

Support mbedTLS #3935

Support mbedTLS #3935

Conversation

joshtriplett commented Sep 25, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tkelman commented Sep 25, 2016

joshtriplett commented Sep 25, 2016

ethomson commented Sep 28, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

staticfloat commented Oct 14, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

carlosmn commented Oct 16, 2016

tkelman commented Oct 24, 2016

tkelman commented Dec 15, 2016

cdluminate commented Mar 16, 2017

tkelman commented Mar 18, 2017

tiennou commented Mar 19, 2017

joshtriplett commented Mar 20, 2017

tkelman commented Mar 20, 2017

tiennou commented Mar 20, 2017

tkelman commented Mar 20, 2017 • edited Loading

ethomson commented Mar 20, 2017

joshtriplett commented Mar 20, 2017

tkelman commented Mar 20, 2017

joshtriplett commented Mar 20, 2017

tiennou commented Mar 20, 2017

tkelman commented Mar 20, 2017

joshtriplett commented Mar 24, 2017 • edited Loading

tkelman commented Mar 20, 2017 •

edited

Loading

joshtriplett commented Mar 24, 2017 •

edited

Loading