Thanks to visit codestin.com
Credit goes to github.com

Skip to content

Fix e2e tests on macOS by replacing multiprocessing with threading #930

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Draft
wants to merge 5 commits into
base: main
Choose a base branch
from
Draft

Fix e2e tests on macOS by replacing multiprocessing with threading #930

wants to merge 5 commits into from
+25 −12

Conversation

Copy link
Contributor

Copilot AI commented Oct 30, 2025 •
edited
Loading

E2E tests segfault on macOS due to multiprocessing.set_start_method('fork'). The fork start method is unsafe on macOS where system libraries may spawn threads, causing subprocess crashes.

Changes

tests/e2e/conftest.py

  • Replace multiprocessing.Process with threading.Timer for browser timeout mechanism
  • Remove multiprocessing.set_start_method('fork') call
  • Add exception handling in timeout callback and finalizer to prevent race conditions
  • Remove unused TEST_RUNNER config (Django-specific, not applicable to Flask)

tests/conftest.py

  • Remove unused TEST_RUNNER config

Implementation

# Before: multiprocessing with fork (crashes on macOS)
timeout_process = multiprocessing.Process(target=wait_kill)
timeout_process.start()
# In finalizer:
timeout_process.terminate()

# After: threading (works cross-platform)
timeout_thread = threading.Timer(timeout, wait_kill)
timeout_thread.start()
# In finalizer:
timeout_thread.cancel()

Threading is appropriate here—the timeout is a safety mechanism to quit hung browsers, not CPU-bound work requiring process isolation. No pickling required, no platform-specific start method issues.

Original prompt

This section details on the original issue you should resolve

<issue_title>tests: end-to-end tests not working on macOS</issue_title>
<issue_description>With my current set-up (MacBook Pro with Apple M2 Pro chip running macOS Tahoe 26.0.1), running the end-to-end tests (pytest tests/e2e) gives Fatal Python error: Segmentation fault. The workarounds mentioned in #928 do not help. This might be due to:

hepdata/tests/e2e/conftest.py

Line 65 in 3b79786

multiprocessing.set_start_method('fork')

The Python docs say "The fork start method should be considered unsafe as it can lead to crashes of the subprocess as macOS system libraries may start threads. See bpo-33725."

I tried switching to the spawn start method or removing the use of multiprocessing altogether, but I got other errors. The end-to-end tests should be modified to avoid relying on the fork start method. I'll add some explanation to the installation docs in the meantime that the end-to-end tests might not work on macOS.

As an aside, I don't see that the line TEST_RUNNER="celery.contrib.test_runner.CeleryTestSuiteRunner", in both tests/conftest.py and tests/e2e/conftest.py is doing anything. This seems to be something from Django.

@ItIsJordan : maybe something for you to work on?</issue_description>

Comments on the Issue (you are @copilot in this section)

✨ Let Copilot coding agent set things up for you — coding agent works faster and does higher quality work when set up for your repo.

Copilot AI changed the title [WIP] Fix end-to-end tests not working on macOS Fix e2e tests on macOS by replacing multiprocessing with threading Oct 30, 2025
Copilot AI requested a review from GraemeWatt October 30, 2025 19:52
Copilot finished work on behalf of GraemeWatt October 30, 2025 19:52
@coveralls
Copy link

coveralls commented Oct 30, 2025

Coverage Status

coverage: 84.517%. remained the same
when pulling 1985d9a on copilot/fix-e2e-tests-on-macos
into ee2dce9 on main.

@GraemeWatt GraemeWatt mentioned this pull request Oct 31, 2025
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@GraemeWatt GraemeWatt Awaiting requested review from GraemeWatt

Assignees

@GraemeWatt GraemeWatt

Copilot code review Copilot

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

tests: end-to-end tests not working on macOS

3 participants

@coveralls @GraemeWatt