tests/run-tests.py: Wait for soft reset if a target skips a test. #15990
Conversation
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## master #15990 +/- ##
=======================================
Coverage 98.57% 98.57%
=======================================
Files 164 164
Lines 21336 21336
=======================================
Hits 21031 21031
Misses 305 305 ☔ View full report in Codecov by Sentry. |
tests/run-tests.py
Outdated
| # to skip the test. It does this via a SystemExit which triggers a soft | ||
| # reset. Wait for the soft reset to finish, so we don't interrupt the | ||
| # start-up code (eg boot.py) when preparing to run the next test. | ||
| pyb.read_until(1, b"raw REPL; CTRL-B to exit\r\n", timeout=1) |
There was a problem hiding this comment.
I'm not sure if 1 second is really long enough here?
I'm thinking of the likes of a wifi board that's used for multitests where it's common to have wifi & network setup in boot.py, this can take longer.
Considering this will continue sooner when that startup message is received I'd be inclined to set a reasonably long timeout for the less common cases that need it?
There was a problem hiding this comment.
This is not the total timeout, but actually the timeout between characters received over the serial connection.
Well, you're right that it could still take a while, with nothing printed for many seconds.
I can increase it. All boards are expected to do a soft reset after skipping a test, so it should only time out if something is wrong.
There was a problem hiding this comment.
I've now changed this to use the default timeout of 10 seconds.
Tested on stm32, rp2 and esp32 boards and the wait for the soft reset after a skip is quick (because they don't do much in their boot.py).
There was a problem hiding this comment.
That seems reasonable, looks good!
e88a784 to
59b1984
Compare
Commit 69c25ea made raising `SystemExit` do a soft reset (on bare-metal targets). This means that any test which is skipped by a target (by raising `SystemExit`) will trigger a soft reset on that target, and then it must execute its startup code, such as `boot.py`. If the timing is right, this startup code can be unintentionally interrupted by the test runner when preparing the next test, because the test runner enters the raw REPL again via a Ctrl-C Ctrl-A ctrl-D sequence (in `Pyboard.enter_raw_repl()`). When this happens (`boot.py` is interrupted) the target may not be set up correctly, and it may (in the case of stm32 boards) flash LEDs and take extra time, slowing down the test run. Fix this by explicitly waiting for the target to finish its soft reset when it skips a test. Signed-off-by: Damien George <[email protected]>
59b1984 to
7746785
Compare
|
Thanks for the reviews! |
Summary
Commit 69c25ea made raising
SystemExitdo a soft reset (on bare-metal targets). This means that any test which is skipped by a target (by raisingSystemExit) will trigger a soft reset on that target, and then it must execute its startup code, such asboot.py.If the timing is right, this startup code can be unintentionally interrupted by the test runner when preparing the next test, because the test runner enters the raw REPL again via a Ctrl-C Ctrl-A ctrl-D sequence (in
Pyboard.enter_raw_repl()).When this happens (
boot.pyis interrupted) the target may not be set up correctly, and it may (in the case of stm32 boards) flash LEDs and take extra time, slowing down the test run.Fix this by explicitly waiting (with a short 1 second timeout) for the target to finish its soft reset when it skips a test.
Testing
Run the test suite many times over and stm32 boards will occasionally have their boot.py interrupted and flash their LEDs to indicate that. With this patch applied you don't ever see that behaviour.