Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 98.57%. Comparing base (09ea901) to head (4c54335).
Report is 6 commits behind head on master.

@@           Coverage Diff           @@
##           master   #15905   +/-   ##
=======================================
  Coverage   98.57%   98.57%           
=======================================
  Files         164      164           
  Lines       21341    21345    +4     
=======================================
+ Hits        21036    21040    +4     
  Misses        305      305           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Code size report:

   bare-arm:    +0 +0.000% 
minimal x86:    +0 +0.000% 
   unix x64:    +0 +0.000% standard
      stm32:    +0 +0.000% PYBV10
     mimxrt:    +0 +0.000% TEENSY40
        rp2:    +0 +0.000% RPI_PICO_W
       samd:    +0 +0.000% ADAFRUIT_ITSYBITSY_M4_EXPRESS
  qemu rv32:    +0 +0.000% VIRT_RV32

Anything else needs to change here ?

I agree that this is more general than #15817.

Did you consider putting the new function on the SSLContext object? Eg SSLContext.ecdsa_sign_callback? That way, each SSLContext could have a different alt implementation, which would be even more flexible. Eg some connections could use the default, and others could use the custom one. (I'm not sure exactly how this would be implemented, because there needs to be a pointer to the correct SSLContext, but it's worth thinking about this possibility.)

Did you consider putting the new function on the SSLContext object? Eg SSLContext.ecdsa_sign_callback?
I'm not sure exactly how this would be implemented, because there needs to be a pointer to the correct SSLContext, but it's worth thinking about this possibility

Yes I did, that's actually how I wanted to implement this originally, but quickly realized that there's no way to get the SSL object in mbedtls functions. It's not an issue though, because that's what the fallback is for; if another context uses regular keys, it will still function.

FWIW this is how this feature is currently used:

https://github.com/arduino/arduino-iot-cloud-py/blob/new_se_support/src/arduino_iot_cloud/ussl.py#L34

This is not merged, waiting on it to be merged upstream first.

Yes I did, that's actually how I wanted to implement this originally, but quickly realized that there's no way to get the SSL object in mbedtls functions.

I think we should try to implement it this way, as an attribute per SSLContext. It's not that difficult actually, just have a global (thread local in threaded systems) variable which points to the current SSLContext object and be sure to set that variable on entry to all C functions in modtls_mbedtls.c that will call into mbedTLS code (eg socket_read and socket_write are the main ones). Then when you need to retrieve the signing callback, access this global SSLContext variable.

I think that's a lot more flexible, and would interoperate well with other extensions like DTLS #15764 which may not want to use custom signing.

Sure, I'll get around to making all of the requested changes next week. I will retest everything and need to send a PR to micropython-lib.

I'll get around to making all of the requested changes next week

OK, great, thanks!

I think having the callback on the SSLContext will be great if you can get that working. Here are some suggestions for that:

// in py/mpconfig.h
#define MICROPY_PY_SSL_MBEDTLS_NEED_ACTIVE_CONTEXT (MICROPY_PY_SSL_ECDSA_SIGN_CALLBACK)

// in py/mpstate.h, mp_state_thread_t struct
    #if MICROPY_PY_SSL_MBEDTLS_NEED_ACTIVE_CONTEXT
    struct _mp_obj_ssl_context_t *tls_ssl_context;
    #endif

// in extmod/modtls_mbedtls.c
static inline void store_active_context(mp_obj_ssl_context_t *ssl_context) {
    #if MICROPY_PY_SSL_MBEDTLS_NEED_ACTIVE_CONTEXT
    MP_THREAD_STATE(tls_ssl_context) = ssl_context;
    #endif
}

// at start of functions that call into mbedtls
    store_active_context(ssl_context);

// could also do this at end of functions, but not necessary
    store_active_context(NULL);

@dpgeorge Can we add the callback attribute to ssl.py as well ? It's non-standard, but would make it much more convenient to set from code that uses ssl for portability. Alternatively, it could probably be set using something like:

ctx._context.ecdsa_sign_callback = callback

// could also do this at end of functions, but not necessary
store_active_context(NULL);

If the root pointer is not cleared, would this stop the socket from being collected after its closed ?

Can we add the callback attribute to ssl.py as well ? It's non-standard, but would make it much more convenient to set from code that uses ssl for portability.

We are trying hard to keep ssl compatible with CPython, so we should not add this callback there.

I'm not sure what you mean by "portability" here? It won't be portable to CPython, and any existing MicroPython code needs to update to firmware that supports this callback, and that firmware will have the new tls module.

If the root pointer is not cleared, would this stop the socket from being collected after its closed ?

It will stop the SSLContext from being collected, but I think the socket will still be freed because it won't have a reference (I don't think).

are trying hard to keep ssl compatible with CPython, so we should not add this callback there.

Ah I sent a PR but will remove it.

I'm not sure what you mean by "portability" here? It won't be portable to CPython, and any existing MicroPython code needs to update to firmware that supports this callback, and that firmware will have the new tls module

I usually just import the ssl module on CPython and MicroPython, not tls. To access this attribute from tls I would have to switch to it or use ctx._context.ecdsa_sign_callback = callback ? The latter is fine.

It will stop the SSLContext from being collected

I've updated the code, it now stores the context before certain functions, read, write, wrap_socket, and clears it on stream close. Will that be enough ?

I wanted to document this new attribute, but I couldn't find any docs for the new tls module to begin with.

This will eventually need docs... but we somehow forgot to document the tls module when that was added, and doing that is out of scope for this PR. So docs for this alternative signing callback can be added when tls itself is documented.

Also would be good to add some tests for this new functionality, that can be run with the unix coverage build. That's also very difficult, it requires at least having a loopback socket and creating a dummy TLS connection. For now, testing on hardware is good enough.