extmod/modussl_mbedtls: Updated Wire in support for DTLS. #15764
Conversation
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #15764 +/- ##
==========================================
- Coverage 98.53% 98.53% -0.01%
==========================================
Files 169 169
Lines 21806 21822 +16
==========================================
+ Hits 21487 21502 +15
- Misses 319 320 +1 ☔ View full report in Codecov by Sentry. |
79db143 to
41a04d0
Compare
|
Code size report: |
e847b8d to
1889991
Compare
a9cb25f to
e4e95c4
Compare
There was a problem hiding this comment.
Thanks for keeping this PR alive, @keenanjohnson! I can see it being quite useful for anyone using CoAP, we could even consider enabling it by default on some of the bigger ports (such as ESP32) provided the code size impact is not too big ( 🤞 ).
The main concern I have here is adding all these extra elements to the ssl module and its types. This is a CPython specific module, and we're trying to cut down on incompatibilities in these.
(If nothing else, there's also the awkward possibility that some day CPython adds DTLS support and then we have to figure out how to make this API compatible with whatever CPython adopts.)
There is potentially a way forward here. As of b802f0f, there is a tls module (non-CPython compatible) and an ssl module (CPython compatible). The current ssl module just does from tls import * but it doesn't have to do that, it could be changed to only import the CPython-compatible names.
To take the code in this direction probably means significant additional work, though. I think applying the tls/ssl split here would look something like this:
- Make a separate
tls.DTLSSocket(or similar) class, so a CPython-compatiblessl.socketdoesn't gain any of the other functions. As a bonus,tls.DTLSSocketcan also be minimal and only expose the functions used by DTLS. - Can probably leave
ssl_context_make_newimplementation the same, apart from having it return atls.DTLSSocketobject if a DTLS protocol is requested. The behaviour will stay CPython-compatible unless the caller passes in an SSLContext with the DTLS-only type set. - Update
ssl.pyin micropython-lib so it only imports the CPython-compatible names rather than everything. - Add a doc for
tlsand describe how it's the non-CPython-compatible version of thesslmodule. Docs for DTLS can be placed there.
How does that sound? Does it sound reasonable? I admit that it does seem like it could become confusing (i.e. two very similar modules with slightly different behaviour), but it's an awkward situation either way.
We can probably help with some of the steps as well (in particular the last two points), if they become sticking points.
I mentioned this to @dpgeorge and he's not keen on a new class for size reasons, so the main compatibility change may just be to change EDIT: Removed the part in this comment about not having recv/sendto/etc in the DTLS socket, as this is useful to keep a UDP-compatible API. |
|
Hey @projectgus yes that all makes sense. I see that #15905 has been merged as well. Other than fixing the test above, is there any other outstanding changes I can make here before we bring this in? |
|
Tests are now fixed. The outstanding items are improving the test coverage and fixing the commit messages. |
|
@projectgus it seems like no matter what I do I can't appease the code coverage metric. |
|
It seems like the main issue is that the coverage test is skipping my dtls test: https://github.com/micropython/micropython/actions/runs/12241647550/job/34147207388#step:5:880 @projectgus or @dpgeorge am I doing something wrong is setting up the tests here? I am enabling the DTLS support only in the unix/mbedtls port, but perhaps that is not the correct place? |
It's a bit fiddly, what is happening is that the test runner is running the test in MicroPython, where it prints nothing and passes. Then it runs it in CPython, where it prints "SKIP". Then it's comparing the output and saying "the test has failed", because the output doesn't match.! The documentation for this has just improved, see the new explanation of test types here: https://github.com/micropython/micropython/tree/master/tests#micropython-test-suite You can run this test locally and reproduce the failure by:
What I suggest for a fix is to have the test print something on success, and then add an .exp file with that expected output. For example: diff --git c/tests/ports/unix/ssl_dtls.py i/tests/ports/unix/ssl_dtls.py
index ec45eb877a..a40736c785 100644
--- c/tests/ports/unix/ssl_dtls.py
+++ i/tests/ports/unix/ssl_dtls.py
@@ -14,3 +14,4 @@ except NameError:
# Test constructing with valid arguments
dtls_client = ssl.SSLContext(PROTOCOL_DTLS_CLIENT)
dtls_server = ssl.SSLContext(PROTOCOL_DTLS_SERVER)
+print("OK")
diff --git c/tests/ports/unix/ssl_dtls.py.exp i/tests/ports/unix/ssl_dtls.py.exp
new file mode 100644
index 0000000000..d86bac9de5
--- /dev/null
+++ i/tests/ports/unix/ssl_dtls.py.exp
@@ -0,0 +1 @@
+OK |
|
Once you have the CI passing, the other step to do is to rebase the branch and squash all these commits down to one. Jimmo has a MicroPython-specific guide here: https://github.com/jimmo/git-and-micropython#README - although any git-related guide about "rebasing" and "squashing" will be able to give you the gist. If you get totally stuck, one of us can help you. Please don't solve the problem by closing this PR and opening a new one, this shouldn't be necessary! Cheers. |
|
OK
Thanks @projectgus ! I'm comfortable with rebasing, so I'll do that once all the builds are working :) |
|
Ok I have reduced the duplicated test, added the basic .exp file, and moved the enable flag to the correct file as suggested by @projectgus in discord, but it still seems to be skipping the test whether I use the tls or ssl module in the test. Am I missing something else obvious? |
|
I think the main implementation here is all done, but just trying to get the code coverage metric up. Just one more tricky timing line to hit before the code coverage bot is satisfied. Am I doing the testing in the preferred way for micropython @projectgus ? |
Looks like Damien will do the final review for this, but thanks for the kind words! |
|
@keenanjohnson I'm doing a final review here and trying to test it, but not having any luck. Do you have some (simple) example (Micro)Python code for both a DTLS server and DTLS client that can talk to each other, just send a small amount of data between each other? |
extmod/modtls_mbedtls.c
Outdated
| @@ -37,6 +37,8 @@ | |||
| #include "py/stream.h" | |||
| #include "py/objstr.h" | |||
| #include "py/reader.h" | |||
| #include "py/smallint.h" | |||
There was a problem hiding this comment.
This header shouldn't be needed anymore.
|
|
||
| def write(self, data): | ||
| self.write_buffer.extend(data) | ||
| return len(data) |
There was a problem hiding this comment.
The write_buffer is never actually used. So this function could just do return len(data).
tests/extmod/tls_dtls.py
Outdated
| def readinto(self, buf): | ||
| l = min(len(self.read_buffer), len(buf)) | ||
| if l == 0: | ||
| return None |
There was a problem hiding this comment.
The read_buffer is always empty, so this function could just always do return None.
|
I finally managed to write a test where MicroPython is both a DTLS server and client, and they connect to each other, do a handshake, and transfer some data between themselves. It works well for unix<->unix. It works well for unix<->PYBD_SF2 and unix<->RPI_PICO_W, when unix is the DTLS server. But when PYBD_SF2 or RPI_PICO_W is the DTLS server then it rarely works. I'm pretty sure this is because the lwIP driver on those boards can only queue one UDP packet at a time. With a quick hack to queue up to two UDP packets, these boards as a DTLS server work much more reliably. Note: to properly implement a DTLS server, we need to implement |
|
Hey @dpgeorge ! Thanks for taking a look. I was traveling for a few days and didn't get around to emails I guess. That makes sense that there might be some issues with the DTLS server on all ports. I've only tested on unix and esp32 and mostly with the device as a client rather than the server (which is the more common usecase I think). I've addressed your small comments above and as for the message peek feature should we try to implement that in this PR or would you prefer I remove the server functionality in this PR for now and address adding that with the peak in a separate changeset as I imagine the desire for the server side of DTLS is pretty small in the community? |
35c7b08 to
ead018c
Compare
|
If we do want to add the peek feature here, I assume we would modify the extmod/modsocket.c implementation of recvfrom()? Also if you post your test code @dpgeorge I can try to verify with the exact same code here to save the hassle of me trying to recreate what you've already done. |
85cb312 to
7be8329
Compare
|
Resolved a small merge conflict. |
7be8329 to
5b8c64f
Compare
I've now added a commit to this PR with the test that I made. It runs well under unix/unix with the multi-test running: That should pass. It should run as part of the CI as well.
No, let's not add that here, it's a separate feature. And it's not necessary for the test. And let's keep this PR as-is with both server and client support. It works fine with the unix port. |
|
Great and thanks for your support! I think the summarize there two follow up change-sets we've discussed after this is merged:
Is there anything else open on this PR that I can help with? |
Were you able to successfully run the test that I added? If you are happy with that test, then this PR should be good to go. |
|
Yes the test runs locally for me! I can also confirm that the CI system runs it successfully! |
This commit enables support for DTLS, i.e. TLS over datagram transport protocols like UDP. While support for DTLS is absent in CPython, it is worth supporting it in MicroPython because it is the basis of the ubiquitous CoAP protocol, used in many IoT projects. To select DTLS, a new set of "protocols" are added to SSLContext: - ssl.PROTOCOL_DTLS_CLIENT - ssl.PROTOCOL_DTLS_SERVER If one of these is set, the library assumes that the underlying socket is a datagram-like socket (i.e. UDP or similar). Our own timer callbacks are implemented because the out of the box implementation relies on `gettimeofday()`. This new DTLS feature is enabled on all ports that use mbedTLS. This commit is an update to a previous PR micropython#10062. Addresses issue micropython#5270 which requested DTLS support. Signed-off-by: Keenan Johnson <[email protected]>
This adds a multi-test for DTLS server and client behaviour. It works on all ports that enable this feature (eg unix, esp32, rp2, stm32), but bare-metal ports that use lwIP are not reliable as the DTLS server because the lwIP bindings only support queuing one UDP packet at a time (that needs to be fixed). Also, to properly implement a DTLS server sockets need to support `socket.recvfrom(n, MSG_PEEK)`. That can be implemented in the future. Signed-off-by: Damien George <[email protected]>
|
Thanks for testing. Now merged! |
|
Thank you for the review! |
This is an update to a previous PR: #10062, which updates the PR based on the latest refactoring of the micropython repo.
I opened this PR as I wasn't sure if the original author of #10062 was still interested in pushing the code change through, but I am quite interested in seeing it happen.
What is this PR trying to solve?
This PR enables support for DTLS (i.e. TLS over datagram transport protocols like UDP). While support for DTLS is absent in cpython, it is worth supporting it in micropython because it is the basis of the ubiquitous CoAP protocol, used in many IOT projects. See #5270
How does this PR solve the problem?
A new set of "protocols" are added to SSLContext:
ssl.PROTOCOL_DTLS_CLIENT
ssl.PROTOCOL_DTLS_SERVER
They act as you expect. If one of these is set, the library assumes that the underlying socket is a datagram-like socket (i.e. UDP or similar).
Implement our own timer callbacks as the out of the box implementation relies on gettimeofday().
TODO: To fully support asyncio for DTLS socket, we will need to return a readable or writable event in poll(MP_STREAM_POLL, ...) if _mbedtls_timing_get_delay(self) >= 1. This is left for future work so as not to interfere with #9871.