unix/main: Use standard pyexec/repl for unix port. #12802

andrewleech · 2023-10-25T23:40:29Z

This improves repl usage consistency across ports.
Only enabled when MICROPY_USE_READLINE == 1 (default).

In particular I noticed that aiorepl didn't work properly on the unix port currently, it would echo the "just entered line" on a newline each time you hit enter, eg.

>>> import aiorepl, asyncio;asyncio.run(aiorepl.task())
Starting asyncio REPL...
--> print("hello")
print("hello")
hello

I feel there were also a few other inconsistencies with how I'm used to the repl working on the stm port / via mpremote which are alsos hopefully resolved now.

I've got an optional couple of lines to restore/reset stdio_raw mode when running code - however if raw mode is disabled the aiorepl issue above remains. I think for consistency with other ports it's better to keep stdio in raw mode, but not sure if this is likely to have other side effects?

This PR consilidates the repl functionality and fixes the above aiorepl glitch.

dpgeorge · 2023-10-25T23:51:56Z

Thanks for this, it's a good clean up.

I actually attempted it before, see #6008. The approach here is much simpler, so maybe it's a good first step to make.

andrewleech · 2023-10-26T01:34:26Z

Ah interesting, I did see some of the return handling and kind of assumed it shouldn't matter - if someone is dealing with the repl (live coding) they're unlikely to care about the exit code?

I didn't consider the SystemExit interrupt at all, haven't looked at how that behaves but probably should!

If running a python file directly then certainly the exit code is important but that shouldn't be impacted by this.

andrewleech · 2023-10-26T01:43:03Z

Ah, I might need to read through the other PR discussion a bit more... but in the mean time this is a little off:

$ ~/micropython/ports/unix/build-standard/micropython
MicroPython v1.22.0-preview.32.g2c1c7d8b30 on 2023-10-26; linux [GCC 10.2.1] version
Type "help()" for more information.
>>> raise ValueError()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ValueError:
>>> raise SystemExit()
>>>

Edit:
Actually that behaviour above is consistent with running a script on Unix port in that SystemExit will drop out of the running script back to console without printing anything, other exceptions drop out while printing the traceback.

jimmo · 2023-10-26T07:36:27Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 98.38%. Comparing base (40df953) to head (a686410).
⚠️ Report is 9 commits behind head on master.

Additional details and impacted files

@@           Coverage Diff           @@
##           master   #12802   +/-   ##
=======================================
  Coverage   98.38%   98.38%           
=======================================
  Files         171      171           
  Lines       22299    22299           
=======================================
  Hits        21938    21938           
  Misses        361      361

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

dpgeorge · 2024-07-05T06:24:34Z

Regarding the comment above about raise SystemExit: actually this PR changes the behaviour, now if you raise SystemExit from the REPL it does nothing. Currently on master it will exit MicroPython back to the shell.

We need to retain the existing behaviour, so we don't break existing usage, and because that's how CPython works.

Then it's a separate question how to handle this on bare-metal targets. But for now we shouldn't change the unix behaviour on this.

github-actions · 2024-07-09T07:06:44Z

Code size report:

Reference:  rp2: Fix RP2350 and RP2350B pin alt functions. [40df953]
Comparison: shared/runtime/pyexec: Set PYEXEC_FORCED_EXIT flag for SystemExit. [merge of a686410]
  mpy-cross:    +0 +0.000% 
   bare-arm:    +0 +0.000% 
minimal x86:    +0 +0.000% 
   unix x64: +1712 +0.199% standard
      stm32:   +24 +0.006% PYBV10
     mimxrt:   +16 +0.004% TEENSY40
        rp2:   +16 +0.002% RPI_PICO_W
       samd:   +20 +0.007% ADAFRUIT_ITSYBITSY_M4_EXPRESS
  qemu rv32:   +24 +0.005% VIRT_RV32

andrewleech · 2024-07-09T07:07:09Z

Regarding the comment above about raise SystemExit: actually this PR changes the behaviour, now if you raise SystemExit from the REPL it does nothing. Currently on master it will exit MicroPython back to the shell.

We need to retain the existing behaviour, so we don't break existing usage, and because that's how CPython works.

Thanks, yes I hadn't realised this behavior had been changed inadvertantly. I've now fixed it in the latest push.
There was an associates variable here pyexec_system_exit which I did a global search for - it was never actually being used (anymore) so I've stripped that out as well.
Any variable passed to SystemExit is now returned for unix port.

anl@STEP: ~ $ micropython
MicroPython v1.24.0-preview.102.g03e992d59d.dirty on 2024-07-09; linux [GCC 9.4.0] version
Type "help()" for more information.
>>> raise SystemExit
anl@STEP: ~ $ echo $?
0
anl@STEP: ~ $ micropython
MicroPython v1.24.0-preview.102.g03e992d59d.dirty on 2024-07-09; linux [GCC 9.4.0] version
Type "help()" for more information.
>>> raise SystemExit(-1)
anl@STEP: ~ $ echo $?
255

... which matches cpython correctly:

anl@ANL2-LAP:~$ python
Python 3.8.10 (default, Nov 22 2023, 10:22:35)
[GCC 9.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> raise SystemExit(-1)
anl@ANL2-LAP:~$ echo $?
255

Then it's a separate question how to handle this on bare-metal targets. But for now we shouldn't change the unix behaviour on this.

Well it looks to me like this should be treated as a soft reset.

micropython/extmod/modmachine.c

Line 66 in 358e501

mp_raise_type(&mp_type_SystemExit);

The current push also works this way:

anl@STEP: ~/micropython $ mpremote
Connected to MicroPython at /dev/ttyACM0
Use Ctrl-] or Ctrl-x to exit this shell
MicroPython v1.24.0-preview.104.g161a9c3097.dirty on 2024-07-09; WEACTV20_F411 with STM32F411CE
Type "help()" for more information.
>>> raise SystemExit
MPY: sync filesystems
MPY: soft reboot
MicroPython v1.24.0-preview.104.g161a9c3097.dirty on 2024-07-09; WEACTV20_F411 with STM32F411CE
Type "help()" for more information.
>>>

dpgeorge · 2024-07-18T04:03:19Z

There was an associates variable here pyexec_system_exit which I did a global search for - it was never actually being used (anymore) so I've stripped that out as well.

Right, it would be a nice clean up to get rid of that variable, and also change the behaviour of SystemExit on bare-metal ports to match unix/CPython.

Because this is a change in user-facing behaviour, I've split that change out in #15486.

andrewleech · 2024-08-13T12:46:15Z

This has been rebased to master, taking into account the SystemExit stuff already merged.

dpgeorge · 2025-10-08T02:17:45Z

I found that ctrl-C didn't work on the unix port 😬

I fixed that and pushed the commit to this branch. @andrewleech see what you think, it was a little tricky to get right.

dpgeorge · 2025-10-08T02:26:41Z

It looks like my fix for ctrl-C handling has slightly changed the output for using paste mode, probably due to switch in/out of raw mode.

@andrewleech do you think we can just change the tests .exp file to match the new behaviour? Or try to make it go back to the previous behaviour?

andrewleech · 2025-10-09T06:18:09Z

Thanks @dpgeorge I looked into the mode switching and agree it's required, I couldn't find any other way to make paste mode and keyboard interrupt both work. I've consolidated that in a function with comments, see if you think that helps?

I also added some new tests, one of which is sitting in the unix tests folder but needs to be run directly with cpython as a standalone script to test interactive ctrl-c handling on the unix port. It failed without your mode switch patch and passes now, it's a bit messy but validates the behaviour. It's not automatically run as part of the test suite though. Does this look useful and should it be integrated better?

I've found and included the other consolidation of do_file and do_str results handling.

I thought I had the unit test results cleaned up too but it appears not

tests/ports/unix/test_ctrl_c_interrupt.py

shared/runtime/pyexec.c

andrewleech · 2025-11-19T02:51:44Z

The unix ctrl-c tests have been split out from this PR again into #18435

This improves REPL usage consistency across ports, by utilizing the pyexec code for the unix REPL. Only enabled when MICROPY_USE_READLINE == 1 (the default). Signed-off-by: Andrew Leech <[email protected]>

When `MICROPY_MODULE___FILE__` is enabled and parsing file input, set the global `__file__` variable to the source filename. This matches the behavior of the unix port and provides the current filename to the executing script. Signed-off-by: Andrew Leech <[email protected]>

When `MICROPY_PYEXEC_COMPILE_ONLY` is enabled and the global `mp_compile_only` is True, code is compiled but not executed. Also add comprehensive tests for compile-only functionality covering both successful compilation and syntax error detection. Signed-off-by: Andrew Leech <[email protected]>

Provides support for command line `-X compile-only` option on unix port. Signed-off-by: Andrew Leech <[email protected]>

Add a general normalize_newlines() function that handles newline variations (\\r\\r\\n, \\r\\n) to \\n while preserving literal \\r characters that are part of test content. This provides a robust solution for cross-platform test compatibility, particularly addressing PTY double-newline issues that can occur with some terminal implementations. The function is applied to all test output before comparison, eliminating platform-specific newline issues. Includes a unit test to verify the normalization behavior. Signed-off-by: Andrew Leech <[email protected]>

Consolidates file and string execution to use the standard pyexec interface for consistency with other ports. Simplify execute_from_lexer for remaining usage: Remove unused LEX_SRC_VSTR and LEX_SRC_FILENAME cases, keeping only LEX_SRC_STR for REPL and LEX_SRC_STDIN for stdin execution. Signed-off-by: Andrew Leech <[email protected]>

This ensures that ctrl-C works on the unix port when executing code at the REPL. Signed-off-by: Damien George <[email protected]>

Enable `MICROPY_PYEXEC_ENABLE_EXIT_CODE_HANDLING` to propagate `sys.exit()` exit codes properly. Update `convert_pyexec_result()` to handle return values where pyexec returns the exit code with `PYEXEC_FORCED_EXIT` flag set for `SystemExit`. Extract the exit code from the lower 8 bits when the flag is set, otherwise return as-is (0 for success, 1 for exception). Signed-off-by: Andrew Leech <[email protected]>

When `MICROPY_PYEXEC_ENABLE_EXIT_CODE_HANDLING` is enabled, `SystemExit` now sets the `PYEXEC_FORCED_EXIT` flag in addition to the exit code. This allows the REPL to properly detect and exit when SystemExit is raised, while still preserving the exit code in the lower bits. Fixes `repl_lock.py` test which expects REPL to exit on `SystemExit`. Signed-off-by: Andrew Leech <[email protected]>

dpgeorge

I tested this thoroughly and fixed a minor issue: ctrl-C was not properly handled when executing a string via micropython -c "import time; time.sleep(10)".

andrewleech · 2025-11-21T21:15:55Z

Thanks for your assistance and detailed testing!

dpgeorge added the port-unix label Oct 25, 2023

andrewleech force-pushed the unix_pyexec branch 2 times, most recently from e4cc4f5 to 2c1c7d8 Compare October 26, 2023 01:38

andrewleech force-pushed the unix_pyexec branch from 2c1c7d8 to 55032e9 Compare October 26, 2023 01:44

jimmo mentioned this pull request Oct 26, 2023

ports/unix: Add embedded mode. #12807

Open

andrewleech force-pushed the unix_pyexec branch 2 times, most recently from 12f5770 to 7a0825f Compare November 16, 2023 09:37

stinos reviewed Nov 16, 2023

View reviewed changes

andrewleech force-pushed the unix_pyexec branch from 7a0825f to ad27be1 Compare November 16, 2023 10:24

andrewleech force-pushed the unix_pyexec branch from ad27be1 to 02fb334 Compare November 16, 2023 21:53

mattytrentini added the maintainer-queue label Feb 27, 2024

dpgeorge added this to the release-1.24.0 milestone Mar 26, 2024

andrewleech force-pushed the unix_pyexec branch 3 times, most recently from d08fe6e to 554ba06 Compare March 26, 2024 01:01

andrewleech force-pushed the unix_pyexec branch from 554ba06 to 03e992d Compare July 9, 2024 06:58

dpgeorge mentioned this pull request Jul 18, 2024

Make raising SystemExit do a soft reset on bare-metal targets #15486

Merged

andrewleech force-pushed the unix_pyexec branch from 03e992d to 95bce29 Compare August 13, 2024 12:26

andrewleech mentioned this pull request Oct 7, 2025

shared/runtime/pyexec: Add __file__ support for frozen modules. #18221

Open

andrewleech force-pushed the unix_pyexec branch from 3b220d7 to 9dde82b Compare October 9, 2025 06:03

dpgeorge reviewed Oct 9, 2025

View reviewed changes

tests/ports/unix/test_ctrl_c_interrupt.py Outdated Show resolved Hide resolved

dpgeorge reviewed Oct 21, 2025

View reviewed changes

shared/runtime/pyexec.c Outdated Show resolved Hide resolved

andrewleech force-pushed the unix_pyexec branch 2 times, most recently from 1be306a to 43ac816 Compare October 22, 2025 00:49

andrewleech force-pushed the unix_pyexec branch 3 times, most recently from cd45b9b to 8a0ca06 Compare November 5, 2025 02:29

andrewleech mentioned this pull request Nov 5, 2025

tests: Add Ctrl-C interrupt tests for REPL and execution. #18368

Closed

andrewleech force-pushed the unix_pyexec branch 2 times, most recently from 938d7f2 to 3d67b2a Compare November 5, 2025 05:36

andrewleech force-pushed the unix_pyexec branch from 3d67b2a to 5a984bc Compare November 19, 2025 02:17

andrewleech and others added 9 commits November 22, 2025 00:00

unix/main: Use standard pyexec REPL for unix and windows ports.

5b3c928

This improves REPL usage consistency across ports, by utilizing the pyexec code for the unix REPL. Only enabled when MICROPY_USE_READLINE == 1 (the default). Signed-off-by: Andrew Leech <[email protected]>

unix: Enable compile-only mode with shared pyexec REPL.

7ac8fcf

Provides support for command line `-X compile-only` option on unix port. Signed-off-by: Andrew Leech <[email protected]>

shared/runtime/pyexec: Call mp_hal_stdio_mode_orig/raw as appropriate.

fd1ddc3

This ensures that ctrl-C works on the unix port when executing code at the REPL. Signed-off-by: Damien George <[email protected]>

dpgeorge force-pushed the unix_pyexec branch from 5a984bc to a686410 Compare November 21, 2025 13:14

dpgeorge approved these changes Nov 21, 2025

View reviewed changes

dpgeorge merged commit a686410 into micropython:master Nov 21, 2025
93 of 95 checks passed

                   <ClCompile Include="@(PyCoreSource)" />
                   <ClCompile Include="@(PyExtModSource)" />
                   <ClCompile Include="$(PyBaseDir)shared\readline\*.c" />
+                  <ClCompile Include="$(PyBaseDir)shared\runtime\pyexec.c" />

                                       # instead of: output_mupy = subprocess.check_output(args, stdin=f)
                                       master, slave = pty.openpty()
+                                      # disable auto newline conversion on the pty terminal.
+                                      attrs = termios.tcgetattr(slave)

Uh oh!

unix/main: Use standard pyexec/repl for unix port. #12802

unix/main: Use standard pyexec/repl for unix port. #12802

Uh oh!

Conversation

andrewleech commented Oct 25, 2023

Uh oh!

dpgeorge commented Oct 25, 2023

Uh oh!

andrewleech commented Oct 26, 2023

Uh oh!

andrewleech commented Oct 26, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jimmo commented Oct 26, 2023

Uh oh!

andrewleech commented Nov 16, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

stinos Nov 16, 2023

Choose a reason for hiding this comment

Uh oh!

andrewleech Nov 16, 2023

Choose a reason for hiding this comment

Uh oh!

stinos Nov 16, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

andrewleech Nov 16, 2023

Choose a reason for hiding this comment

Uh oh!

andrewleech Nov 16, 2023

Choose a reason for hiding this comment

Uh oh!

stinos Nov 17, 2023

Choose a reason for hiding this comment

Uh oh!

codecov bot commented Nov 16, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

dpgeorge commented Jul 5, 2024

Uh oh!

github-actions bot commented Jul 9, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

andrewleech commented Jul 9, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dpgeorge commented Jul 18, 2024

Uh oh!

andrewleech commented Aug 13, 2024

Uh oh!

dpgeorge commented Oct 8, 2025

Uh oh!

dpgeorge commented Oct 8, 2025

Uh oh!

andrewleech commented Oct 9, 2025

Uh oh!

Uh oh!

Uh oh!

andrewleech commented Nov 19, 2025

Uh oh!

dpgeorge left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

andrewleech commented Nov 21, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants

andrewleech commented Oct 26, 2023 •

edited

Loading

andrewleech commented Nov 16, 2023 •

edited

Loading

stinos Nov 16, 2023 •

edited

Loading

codecov bot commented Nov 16, 2023 •

edited

Loading

github-actions bot commented Jul 9, 2024 •

edited

Loading

andrewleech commented Jul 9, 2024 •

edited

Loading