extmod/modtls_mbedtls: Fix key_len passed to mbedtls_pk_parse_key. #14385

peterzuger · 2024-04-27T08:32:52Z

See issue #14371:

mbedtls_pk_parse_key() expects key_len to include the NULL terminator for PEM data but not for DER encoded data.

Since all PEM data starts with "-----BEGIN" this is used to check if the data is PEM.

This can be done for both v2 and v3 of mbedtls since the fundamental behaviour/expectation did not change.

What changed is that in v3 the PKCS#8 DER parser now checks that the passed key buffer was fully utilized and no bytes are remaining (all other DER formats still do not check this).

codecov · 2024-04-27T08:35:35Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 98.43%. Comparing base (288a036) to head (1473ed4).

Additional details and impacted files

@@           Coverage Diff           @@
##           master   #14385   +/-   ##
=======================================
  Coverage   98.43%   98.43%           
=======================================
  Files         161      161           
  Lines       21281    21281           
=======================================
  Hits        20948    20948           
  Misses        333      333

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

github-actions · 2024-04-27T08:48:33Z

Code size report:

   bare-arm:    +0 +0.000% 
minimal x86:    +0 +0.000% 
   unix x64:   -16 -0.002% standard
      stm32:    +0 +0.000% PYBV10
     mimxrt:    +0 +0.000% TEENSY40
        rp2:    +0 +0.000% RPI_PICO_W
       samd:    +0 +0.000% ADAFRUIT_ITSYBITSY_M4_EXPRESS

dpgeorge · 2024-07-12T01:39:17Z

Thanks for the patch. This looks OK, except that actually mbedtls is not configured to be able to parse PEM files at all. To enable that requries:

#define MBEDTLS_PEM_PARSE_C
#define MBEDTLS_BASE64_C

As such, it's not possible to write a test for the code added in this PR.

What I suggest is to change all cert/key parsing code in modtls_mbedtls.c so that it does not add a +1 at all, because only DER is supported.

And then add optional support for PEM parsing via the following (essentially what this PR does but with a if guard):

    #if MBEDTLS_PEM_PARSE_C
    // len should include terminating null if the data is PEM encoded
    if ((key_len >= 10) && (memcmp(key, "-----BEGIN", 10) == 0)) {
        key_len += 1;
    }
    #endif

peterzuger · 2024-07-15T07:45:03Z

Yes, I enabled PEM parsing while testing for this bug/patch. The support for PEM parsing was mostly added to this PR to not break the code of people who use PEM.

This Patch is mostly to fix the PKCS8 DER parsing that was broken by the update of mbedtls.

I have added the #if guard to the patch.

dpgeorge · 2024-07-15T12:03:55Z

Do the calls to mbedtls_x509_crt_parse() also need a similar fix? It looks like that function can parse both DER and PEM as well (the latter only when PEM is enabled), and so +1 should only be added to the length if it's PEM.

This Patch is mostly to fix the PKCS8 DER parsing that was broken by the update of mbedtls.

Is it possible to add a test which fails prior to this PR but passes after this PR? In other words, can we add a test that has a PKCS8 DER certificate? How to generate such a certificate?

peterzuger · 2024-07-16T14:12:43Z

After taking a look at mbedtls_x509_crt_parse() and its documentation here.

When MBEDTLS_PEM_PARSE_C is disabled this function simply passes the arguments to mbedtls_x509_crt_parse_der()

The documentation states the same thing as mbedtls_pk_parse_key():

buflen size of the buffer (including the terminating null byte for PEM data)

The problem seems to be possible, at the end of the function x509_crt_parse_der_core() in x509_crt.c mbedtls does a similar check to ensure the entire buffer was used. But they also do some custom length checking/parsing that seems to bypass this by modifying the end ptr ?

if( p != end )
{
    mbedtls_x509_crt_free( crt );
    return( MBEDTLS_ERROR_ADD( MBEDTLS_ERR_X509_INVALID_FORMAT,
            MBEDTLS_ERR_ASN1_LENGTH_MISMATCH ) );
}

After some time searching I could not find a way to create a Certificate that triggers this condition causing the parsing to fail.

As for the test, in the next iteration I will add a certificate generated like described in #14371 to the tests which will currently fail but will be fixed with this PR.

Should I include the fix for mbedtls_x509_crt_parse() in this PR or should I open a separate one since there is currently no test I can think of to see the difference for the Certificate side of things (no known way to produce a failure).

dpgeorge · 2024-07-20T12:50:28Z

Should I include the fix for mbedtls_x509_crt_parse() in this PR or should I open a separate one since there is currently no test I can think of to see the difference for the Certificate side of things (no known way to produce a failure).

Yes, please include the fix in this PR. Even if we can't find a test that fails here (passing DER with an extra null byte counted in the length), the documentation is quite clear:

buflen size of the buffer (including the terminating null byte for PEM data)

So, if it's not PEM data, it shouldn't include a terminating null byte.

peterzuger · 2024-07-26T12:01:07Z

I have addde the fix for mbedtls_x509_crt_parse() and added a test using a PKCS#8 key which fails without the fix for mbedtls_pk_parse_key().

projectgus

Thanks for digging into this, @peterzuger!

As damien noted, I ran into the same problem last week after updating ESP-IDF v5.2.2 and dropped in a separate fix here:
b180f1f

... I should have looked at open PRs first and saved myself some time! 😁

I like the idea of increasing the test coverage to include this, although as per my comment I'm not actually sure about what behaviour we should expect on most ports.

projectgus · 2024-07-30T01:34:30Z

extmod/modtls_mbedtls.c

@@ -100,6 +100,12 @@ static void mbedtls_debug(void *ctx, int level, const char *file, int line, cons
 }
 #endif

+#if defined(MBEDTLS_PEM_PARSE_C)
+static int mbedtls_is_pem(const byte *data, size_t len) {
+    return (len >= 10) && (memcmp(data, "-----BEGIN", 10) == 0);


Since all PEM data starts with "-----BEGIN" this is used to check if the data is PEM.

This is frustratingly not quite true, because PEM is an outgrowth of a mail format it can contain preceding plaintext.

Suggest using strstr() like mbedTLS does here: https://github.com/Mbed-TLS/mbedtls/blob/cfe8c4c57d7e71ea4763252e76b20fe019040eb8/tf-psa-crypto/drivers/builtin/src/pkparse.c#L1337

I was kind of aware of this, but I decided to only accept "clean" PEM data (for no particular reason).
Should I change this?
I am in favor of doing it the right way if at all.

It's relatively minor, but I'd say if there's a PEM certificate file that is accepted by desktop mbedTLS and openSSL then it should be accepted by MicroPython as well. Which means allowing preceding text, even if it's not something commonly used.

What do you think about combining my commit and your test addition for this? My version of the fix already uses strstr and it's a little more factored (less duplicate code), and they both do more or less the same thing.

@peterzuger The change b180f1f that @projectgus refers to is a little cleaner than the changes here, because it factors the extraction of cert data into a helper function, and only has one location with a conditional check on MBEDTLS_PEM_PARSE_C. So I would prefer that simpler commit.

The test from this PR is very good though, we definitely want that.

What do you think, shall we take @projectgus 's commit and your test?

I updated this to use strstr.

tests/extmod/ssl_keycert.py.exp

`mbedtls_pk_parse_key()` expects `key_len` to include the NULL terminator for PEM data but not for DER encoded data. This also applies to `mbedtls_x509_crt_parse()` and `cert_len`. Since all PEM data contains "-----BEGIN" this is used to check if the data is PEM (as per mbedtls code). This can be done for both v2 and v3 of mbedtls since the fundamental behaviour/expectation did not change. What changed is that in v3 the PKCS#8 DER parser now checks that the passed key buffer is fully utilized and no bytes are remaining (all other DER formats still do not check this). Signed-off-by: Peter Züger <[email protected]>

Signed-off-by: Peter Züger <[email protected]>

peterzuger force-pushed the mbedtls-mbedtls_pk_parse_key-key_len-fix branch 2 times, most recently from dcdd202 to bb981c8 Compare April 27, 2024 08:42

dpgeorge added the extmod Relates to extmod/ directory in source label Jun 3, 2024

peterzuger force-pushed the mbedtls-mbedtls_pk_parse_key-key_len-fix branch from bb981c8 to 9943e43 Compare July 15, 2024 07:35

peterzuger force-pushed the mbedtls-mbedtls_pk_parse_key-key_len-fix branch from 9943e43 to 6e79721 Compare July 15, 2024 08:06

peterzuger force-pushed the mbedtls-mbedtls_pk_parse_key-key_len-fix branch 2 times, most recently from 2bb03e7 to 7aa3934 Compare July 22, 2024 13:35

dpgeorge mentioned this pull request Jul 26, 2024

esp32: Update ESP-IDF to v5.2.2, optimise size #15523

Merged

6 tasks

projectgus reviewed Jul 30, 2024

View reviewed changes

projectgus mentioned this pull request Jul 31, 2024

lib/mbedtls: Update to mbedtls v3.6.2. #15547

Merged

dpgeorge force-pushed the mbedtls-mbedtls_pk_parse_key-key_len-fix branch from 7aa3934 to d24879a Compare August 13, 2024 07:26

peterzuger added 2 commits August 13, 2024 17:32

tests/extmod/ssl_keycert.py: Add test for PKCS8 formatted DER key.

1473ed4

Signed-off-by: Peter Züger <[email protected]>

dpgeorge force-pushed the mbedtls-mbedtls_pk_parse_key-key_len-fix branch from d24879a to 1473ed4 Compare August 13, 2024 07:32

dpgeorge merged commit 1473ed4 into micropython:master Aug 13, 2024
63 checks passed

projectgus mentioned this pull request Aug 14, 2024

extmod/mbedtls: Optimise the DER certificate parsing fix. #15657

Merged

dpgeorge mentioned this pull request Aug 18, 2024

SSL Context behaviour various between Raspberry Pi Pico and ESP32 Board #15674

Closed

Uh oh!

extmod/modtls_mbedtls: Fix key_len passed to mbedtls_pk_parse_key. #14385

extmod/modtls_mbedtls: Fix key_len passed to mbedtls_pk_parse_key. #14385

Uh oh!

Conversation

peterzuger commented Apr 27, 2024

Uh oh!

codecov bot commented Apr 27, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

github-actions bot commented Apr 27, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dpgeorge commented Jul 12, 2024

Uh oh!

peterzuger commented Jul 15, 2024

Uh oh!

dpgeorge commented Jul 15, 2024

Uh oh!

peterzuger commented Jul 16, 2024

Uh oh!

dpgeorge commented Jul 20, 2024

Uh oh!

peterzuger commented Jul 26, 2024

Uh oh!

projectgus left a comment

Choose a reason for hiding this comment

Uh oh!

projectgus Jul 30, 2024

Choose a reason for hiding this comment

Uh oh!

peterzuger Jul 30, 2024

Choose a reason for hiding this comment

Uh oh!

projectgus Jul 30, 2024

Choose a reason for hiding this comment

Uh oh!

dpgeorge Aug 7, 2024

Choose a reason for hiding this comment

Uh oh!

dpgeorge Aug 13, 2024

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

codecov bot commented Apr 27, 2024 •

edited

Loading

github-actions bot commented Apr 27, 2024 •

edited

Loading