extmod/modlwip,esp32,unix: Add support for socket recv flags argument #17312

projectgus · 2025-05-16T06:10:09Z

Summary

Goal of this PR is to have support for the MSG_DONTWAIT and MSG_PEEK flags for socket recv & recvfrom on unix, esp32, and all "bare metal" LWIP ports.

Bare metal LWIP adds support for these flags in modlwip.c, with some refactoring to try and reduce the code size impact.
esp32 port now passes these flags through to the LWIP BSD socket layer, which already supports them.
unix port already had support for flags argument. MSG_PEEK constant is now exposed to Python code.
Adds multi_net tests for the new flags.
Adds docs for the flags argument.

Note: Zephyr & CC3200 ports have their own socket module implementations, support not added to these yet. Zephyr looks like it would be trivially easy, CC3200 I didn't look into.

Testing

Ran new and existing multi_net tests on ESP32_GENERIC_S3, RPI_PICO2_W, RPI_PICO_W and PYBD_SF2 boards. One outstanding issue that seems lower level than these changes, see comment.

Trade-offs and Alternatives

Support for MSG_PEEK is motivated by wanting to improve the DTLS support. Adding MSG_DONTWAIT as well was relatively easy, and both of these flags are potentially useful for socket programming on MicroPython. However I guess the obvious alternative is not to add them, or to implement them via a Python wrapper in the Python socket module (which would be hacky but possible).

github-actions · 2025-05-16T06:18:55Z

Code size report:

   bare-arm:    +0 +0.000% 
minimal x86:    +0 +0.000% 
   unix x64:   +56 +0.007% standard[incl +32(data)]
      stm32:    +0 +0.000% PYBV10
     mimxrt:    +0 +0.000% TEENSY40
        rp2:   +32 +0.003% RPI_PICO_W
       samd:    +0 +0.000% ADAFRUIT_ITSYBITSY_M4_EXPRESS
  qemu rv32:    +0 +0.000% VIRT_RV32

codecov · 2025-05-16T06:19:55Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 98.54%. Comparing base (4dff9cb) to head (ba92063).
Report is 6 commits behind head on master.

Additional details and impacted files

@@           Coverage Diff           @@
##           master   #17312   +/-   ##
=======================================
  Coverage   98.54%   98.54%           
=======================================
  Files         169      169           
  Lines       21898    21898           
=======================================
  Hits        21579    21579           
  Misses        319      319

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

dpgeorge

This looks good, thanks!

I tested on ESP8266, RPI_PICO2_W-RISCV and ESP32_GENERIC, and all the new tests pass.

ports/esp32/modsocket.c

tests/multi_net/tcp_recv_peek.py

projectgus · 2025-05-30T02:19:32Z

Have updated the tests and run on various ports (updated in description), but have found what looks like a lower level issue on RPI_PICO_W and RPI_PICO2_W.

The new udp_recv_dontwait.py test fails when instance0 is one of these rp2 boards. The UDP recv succeeds, then UDP sendto() appears to succeed and the frame appears to be passed into the CYW43 module ("ETHTX" is shown in the trace), but packet capture shows it is almost never sent (maybe <10% of the time it sends). Unclear why, other tests show occasional packet loss but nothing consistent like this.

tests/multi_net/udp_recv_dontwait.py

projectgus · 2025-05-30T02:37:01Z

The UDP recv succeeds, then UDP sendto() appears to succeed and the frame appears to be passed into the CYW43 module ("ETHTX" is shown in the trace), but packet capture shows it is almost never sent

One interesting thing in the output with wlan.config(trace=0xFF) is that the runs which succeed are always:

ETHRX (first UDP packet from instance1)
ETHTX (UDP packet sent)
ETHRX (second UDP packet from instance1)

But the runs which fail always seem to be:

ETHRX (first UDP packet from instance1)
ETHRX (second UDP packet from instance1)
ETHTX (UDP packet "sent" but not seen in the packet capture)

Weirdly this ordering seems to be happening internal to the CYW43 module, packet captures of successful runs on the Wi-Fi AP always show the second ordering over the air (RX, RX, TX) even though the trace log shows (RX, TX, RX).

tests/multi_net/udp_recv_dontwait.py

Implements MSG_PEEK and MSG_DONTWAIT. This work was funded through GitHub Sponsors. Signed-off-by: Angus Gratton <[email protected]>

Implements MSG_PEEK and MSG_DONTWAIT (both passed through to LWIP sockets API). This work was funded through GitHub Sponsors. Signed-off-by: Angus Gratton <[email protected]>

This work was funded through GitHub Sponsors. Signed-off-by: Angus Gratton <[email protected]>

Adds TCP and UDP multi_net test cases. This work was funded through GitHub Sponsors. Signed-off-by: Angus Gratton <[email protected]>

Adding multi_net case for UDP only, as TCP timing is hard to test reliably. Signed-off-by: Angus Gratton <[email protected]>

Implementation added for various ports in the parent commits. This work was funded through GitHub Sponsors. Signed-off-by: Angus Gratton <[email protected]>

dpgeorge

I've re-reviewed this, it looks good.

I also tested on PYBD_SF2, RPI_PICO_W and ESP32_GENERIC. The new tests all pass, and so do existing ones (with known failures).

projectgus added port-esp32 extmod Relates to extmod/ directory in source port-unix port-rp2 labels May 16, 2025

dpgeorge reviewed May 26, 2025

View reviewed changes

ports/esp32/modsocket.c Outdated Show resolved Hide resolved

tests/multi_net/tcp_recv_peek.py Outdated Show resolved Hide resolved

projectgus force-pushed the feature/socket_recv_peek branch from 2308215 to 3c2eaf3 Compare May 30, 2025 02:15

projectgus force-pushed the feature/socket_recv_peek branch from 3c2eaf3 to 14f7f61 Compare May 30, 2025 02:28

projectgus commented May 30, 2025

View reviewed changes

tests/multi_net/udp_recv_dontwait.py Outdated Show resolved Hide resolved

dpgeorge reviewed May 30, 2025

View reviewed changes

tests/multi_net/udp_recv_dontwait.py Outdated Show resolved Hide resolved

projectgus force-pushed the feature/socket_recv_peek branch 3 times, most recently from 355ef35 to 7f76f49 Compare May 30, 2025 05:38

projectgus requested a review from dpgeorge May 30, 2025 06:11

projectgus added 6 commits June 3, 2025 12:52

extmod/modlwip: Add optional flags argument for recv and recvfrom.

e844776

Implements MSG_PEEK and MSG_DONTWAIT. This work was funded through GitHub Sponsors. Signed-off-by: Angus Gratton <[email protected]>

esp32/modsocket: Add optional flags argument for recv and recvfrom.

4781cde

Implements MSG_PEEK and MSG_DONTWAIT (both passed through to LWIP sockets API). This work was funded through GitHub Sponsors. Signed-off-by: Angus Gratton <[email protected]>

unix/modsocket: Expose MSG_PEEK flag for recv & recvfrom.

0b22404

This work was funded through GitHub Sponsors. Signed-off-by: Angus Gratton <[email protected]>

tests/multi_net: Add test coverage for socket recv flag MSG_PEEK.

73d4ddc

Adds TCP and UDP multi_net test cases. This work was funded through GitHub Sponsors. Signed-off-by: Angus Gratton <[email protected]>

tests/multi_net: Add test coverage for socket recv flag MSG_DONTWAIT.

d23fb86

Adding multi_net case for UDP only, as TCP timing is hard to test reliably. Signed-off-by: Angus Gratton <[email protected]>

docs: Add a description of recv/recvfrom flags argument.

ba92063

Implementation added for various ports in the parent commits. This work was funded through GitHub Sponsors. Signed-off-by: Angus Gratton <[email protected]>

dpgeorge approved these changes Jun 3, 2025

View reviewed changes

dpgeorge force-pushed the feature/socket_recv_peek branch from 7f76f49 to ba92063 Compare June 3, 2025 02:56

dpgeorge merged commit ba92063 into micropython:master Jun 3, 2025
67 checks passed

projectgus deleted the feature/socket_recv_peek branch June 5, 2025 01:47

projectgus mentioned this pull request Jun 6, 2025

extmod/mbedtls: Implement DTLS HelloVerify, update test for MSG_PEEK #17441

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

extmod/modlwip,esp32,unix: Add support for socket recv flags argument #17312

extmod/modlwip,esp32,unix: Add support for socket recv flags argument #17312

Uh oh!

projectgus commented May 16, 2025 •

edited

Loading

Uh oh!

github-actions bot commented May 16, 2025

Uh oh!

codecov bot commented May 16, 2025 •

edited

Loading

Uh oh!

dpgeorge left a comment

Uh oh!

Uh oh!

Uh oh!

projectgus commented May 30, 2025 •

edited

Loading

Uh oh!

Uh oh!

projectgus commented May 30, 2025 •

edited

Loading

Uh oh!

Uh oh!

dpgeorge left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

extmod/modlwip,esp32,unix: Add support for socket recv flags argument #17312

extmod/modlwip,esp32,unix: Add support for socket recv flags argument #17312

Uh oh!

Conversation

projectgus commented May 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Testing

Trade-offs and Alternatives

Uh oh!

github-actions bot commented May 16, 2025

Uh oh!

codecov bot commented May 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

dpgeorge left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

projectgus commented May 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

projectgus commented May 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

dpgeorge left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

projectgus commented May 16, 2025 •

edited

Loading

codecov bot commented May 16, 2025 •

edited

Loading

projectgus commented May 30, 2025 •

edited

Loading

projectgus commented May 30, 2025 •

edited

Loading