tests/run-tests.py: Auto detect whether the target has threading and the GIL or not. #17655

dpgeorge · 2025-07-10T06:26:42Z

Summary

This PR makes the following changes to the test runner and CI:

detect if the target has the _thread module and, if so, add the threading tests to the set of tests to run
if the target has threading, detect if the target has the GIL enabled or not, and use that information to exclude thread mutating tests if the GIL is disabled
work around threading bug in tests/extmod/select_poll_eintr.py test
make tests/thread/stress_aes.py test do less work on stackless threading builds, so it finishes before the test timeout limit
increase test timeout for unix qemu test runs
skip some threading tests on macOS and on unix qemu

Currently the information whether a port has the GIL is hard-coded: unix/PC targets and the rp2 port are considered to be GIL-less, and all other targets have the GIL enabled (if they have threading enabled). With the change here, some code is run to detect the GIL. That uses the fact that native code won't bounce the GIL and can hog it, and times how long code runs.

Overall the changes here mean that the threading tests are run in many more configurations:

on the unix port with the native emitter
on the unix port with single precision float
on the unix port with stackless mode enabled
on the unix port using clang as a compiler
on unix qemu, the architectures MIPS, ARM and RISCV-64
on macOS

Testing

Tested locally on the unix port, esp32 and rp2. Let's also see how the CI works.

Trade-offs and Alternatives

This uses a tricky technique to determine GIL/non-GIL. An alternative would be to add some indication in a function whether the GIL is enabled, eg sys.implementation._thread_info. But that will increase code size.

codecov · 2025-07-10T06:30:50Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 98.40%. Comparing base (8f8f853) to head (3c1c9ab).

Additional details and impacted files

@@            Coverage Diff             @@
##           master   #17655      +/-   ##
==========================================
- Coverage   98.44%   98.40%   -0.04%     
==========================================
  Files         171      171              
  Lines       22192    22192              
==========================================
- Hits        21847    21839       -8     
- Misses        345      353       +8

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

github-actions · 2025-07-10T06:36:04Z

Code size report:

   bare-arm:    +0 +0.000% 
minimal x86:    +0 +0.000% 
   unix x64:   +88 +0.010% standard[incl +64(data)]
      stm32:    +0 +0.000% PYBV10
     mimxrt:    +0 +0.000% TEENSY40
        rp2:   +24 +0.003% RPI_PICO_W
       samd:    +0 +0.000% ADAFRUIT_ITSYBITSY_M4_EXPRESS
  qemu rv32:    +0 +0.000% VIRT_RV32

dlech · 2025-07-10T15:15:05Z

tests/thread/stress_aes.py

+    try:
+
+        def stackless():
+            pass
+
+        micropython.heap_lock()
+        stackless()
+        micropython.heap_unlock()
+    except RuntimeError:
+        micropython.heap_unlock()
+        is_stackless = True


Suggested change

try:

def stackless():

pass

micropython.heap_lock()

stackless()

micropython.heap_unlock()

except RuntimeError:

micropython.heap_unlock()

is_stackless = True

def stackless():

pass

micropython.heap_lock()

try:

stackless()

except RuntimeError:

is_stackless = True

finally:

micropython.heap_unlock()

does it work to make the try/except a bit more focused on where the actual exception is expected?

You are right, it's probably better to be more focused like that. I will change it.

(I was just copying existing code for this from eg tests/micropython/heapalloc.py.)

dlech · 2025-07-10T15:55:32Z

ports/unix/mpthreadport.c

@@ -56,7 +56,7 @@
 #endif

 // This value seems to be about right for both 32-bit and 64-bit builds.
-#define THREAD_STACK_OVERFLOW_MARGIN (8192)
+#define THREAD_STACK_OVERFLOW_MARGIN (4 * 8192)


Suggested change

#define THREAD_STACK_OVERFLOW_MARGIN (4 * 8192)

#define THREAD_STACK_OVERFLOW_MARGIN (16 * 1024)

Also, the commit message could use an explanation of why this is changed.

I'm currently trying to work out if this change is even needed.

This change is not needed, it doesn't fix the issue I thought it would fix. So I took it out.

dlech · 2025-07-10T15:57:18Z

tests/feature_check/thread_gil.py

+    thread_done = True
+
+
+global thread_done


Do we actually need global statement at the top level?

You are right, now fixed.

dlech · 2025-07-10T16:11:19Z

tests/feature_check/thread_gil.py

+busy()
+while not thread_done:
+    time.sleep(0)
+dt = time.ticks_diff(time.ticks_ms(), t0)


Depending on time measurements seems like it could be flaky, especially on desktop systems. I'm not awake enough yet though to try to come up with an alternative that doesn't involve creating a deadlock.

Yes, it could be flaky. But from testing on bare-metal and PC, it looks reasonable. The timing is usually either very close to T or 2*T (depending if GIL is disable or enabled).

If it fails due to timing errors (running for too long), it will incorrectly think the GIL is enabled when it is not. Then it will run the mutate tests and they'll probably all fail. In such a case we can just rerun CI...

If it gets to be a real issue, could increase the timing here, eg double it (I just don't want the auto detection to take too long, because it's run at the start of each run of run-tests.py.)

I have some similar concerns, and generally prefer the approach of explicitly adding this information in sys.implementation or similar. Suggest we keep this approach now, but keep the explicit option in mind if it turns out to be flaky.

Having some information in sys.implementation could actually be useful for users, instead of them having to run a tricky detection test like this (if they ever need to know the threading implementation...).

And I guess if you have threading enabled you have some amount of flash to spare (although not necessarily on cc3200 which does enable threading...).

So maybe it's worth adding sys.implementation._thread after all?

I think it'd be useful to add. Especially if we don't add any _thread property on builds without threading support.

I've now changed this GIL detection to use sys.implementation._thread. It's a lot simpler.

And retested on PICO_W, ESP32_GENERIC and unix port. They detect the GIL correctly and the relevant tests run.

dpgeorge · 2025-07-10T23:08:17Z

Thanks @dlech for the review here. After I initially submitted this PR and all the CI failed on things I couldn't test locally, the scope of the PR increased quite a bit... so there was a lot of back and forth getting it to work.

dpgeorge · 2025-07-13T12:31:23Z

After a lot of back-and-forth with the CI, this PR is now in its final state. I updated the comment at the top with a better description of what this PR is aiming for.

Signed-off-by: Damien George <damien@micropython.org>

This is a workaround for the case where threading is enabled without a GIL. In such a configuration, creating a new global variable is not atomic and threads have race conditions resizing/accessing the global dict. Signed-off-by: Damien George <damien@micropython.org>

Builds with stackless enabled are slower than non-stackless, and this test takes around 2 minutes on the unix port of MicroPython with the standard unix parameters. So detect stackless mode and reduce the time of the test. Signed-off-by: Damien George <damien@micropython.org>

When detecting the target platform, also check if it has threading and whether the GIL is enabled or not (using the new attribute `sys.implementation._thread`). If threading is available, add the thread tests to the set of tests to run (unless the set of tests is explicitly given). With this change, the unix port no longer needs to explicitly run the set of thread tests, so that line has been removed from the Makefile. This change will make sure thread tests are run with other testing combinations. In particular, thread tests are now run: - on the unix port with the native emitter - on macOS builds - on unix qemu, the architectures MIPS, ARM and RISCV-64 Signed-off-by: Damien George <damien@micropython.org>

The qemu emulation introduces enough overhead that the `tests/thread/stress_aes.py` test overruns the default timeout. So increase it to allow this test to pass. Signed-off-by: Damien George <damien@micropython.org>

This test passes sometimes and fails other times. Eventually that should be fixed, but for now just skip this test. Signed-off-by: Damien George <damien@micropython.org>

dpgeorge added the tests Relates to tests/ directory in source label Jul 10, 2025

dpgeorge force-pushed the tests-detect-thread-gil branch from d63cbda to b90d11e Compare July 10, 2025 15:40

dpgeorge changed the title ~~tests/run-tests.py: Auto detect whether the target has the GIL or not.~~ tests/run-tests.py: Auto detect whether the target has threading and the GIL or not. Jul 10, 2025

dlech reviewed Jul 10, 2025

View reviewed changes

dpgeorge force-pushed the tests-detect-thread-gil branch 3 times, most recently from bf0c6cb to 1266375 Compare July 11, 2025 14:22

dpgeorge mentioned this pull request Jul 11, 2025

unix/make: Drop i686-linux-gnu path. #17638

Merged

dpgeorge force-pushed the tests-detect-thread-gil branch 7 times, most recently from 05a9ceb to 28535b0 Compare July 13, 2025 12:21

dpgeorge mentioned this pull request Jul 13, 2025

Allow the GIL to be enabled on the unix port, and add a CI job to test this configuration #17668

Open

dpgeorge added this to the release-1.26.0 milestone Jul 13, 2025

dpgeorge requested a review from projectgus July 14, 2025 23:29

projectgus approved these changes Jul 15, 2025

View reviewed changes

dpgeorge added 7 commits July 15, 2025 13:36

py/modsys: Add sys.implementation._thread attribute.

1401c9a

Signed-off-by: Damien George <damien@micropython.org>

tools/ci.sh: Increase timeout for unix qemu test runs.

b764f54

The qemu emulation introduces enough overhead that the `tests/thread/stress_aes.py` test overruns the default timeout. So increase it to allow this test to pass. Signed-off-by: Damien George <damien@micropython.org>

tools/ci.sh: Skip thread/stress_heap.py test on macOS test run.

133dcc9

This test passes sometimes and fails other times. Eventually that should be fixed, but for now just skip this test. Signed-off-by: Damien George <damien@micropython.org>

tools/ci.sh: Skip thread/stress_recurse.py on unix qemu test runs.

3c1c9ab

This test passes sometimes and fails other times. Eventually that should be fixed, but for now just skip this test. Signed-off-by: Damien George <damien@micropython.org>

dpgeorge force-pushed the tests-detect-thread-gil branch from 28535b0 to 3c1c9ab Compare July 15, 2025 03:38

	#define THREAD_STACK_OVERFLOW_MARGIN (4 * 8192)
	#define THREAD_STACK_OVERFLOW_MARGIN (16 * 1024)

Uh oh!

tests/run-tests.py: Auto detect whether the target has threading and the GIL or not. #17655

Are you sure you want to change the base?

tests/run-tests.py: Auto detect whether the target has threading and the GIL or not. #17655

Conversation

dpgeorge commented Jul 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Testing

Trade-offs and Alternatives

Uh oh!

codecov bot commented Jul 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

github-actions bot commented Jul 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

projectgus Jul 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dpgeorge commented Jul 10, 2025

Uh oh!

dpgeorge commented Jul 13, 2025

Uh oh!

Uh oh!

Pfad - The Proxy pFad of © 2024 Garber Painting. All rights reserved.

dpgeorge commented Jul 10, 2025 •

edited

Loading

codecov bot commented Jul 10, 2025 •

edited

Loading

github-actions bot commented Jul 10, 2025 •

edited

Loading

projectgus Jul 15, 2025 •

edited

Loading