Add documentation for using the testbed runner.

python · freakboy3742 · Dec 9, 2024 · Dec 4, 2024 · Dec 4, 2024 · Dec 4, 2024
commit 89bb4359af2ae3abe5d4e39681b1c8c280dd4551
@@ -292,10 +292,12 @@ To add Python to an iOS Xcode project:
 10. Add Objective C code to initialize and use a Python interpreter in embedded
     mode. You should ensure that:
 
-   * :c:member:`UTF-8 mode <PyPreConfig.utf8_mode>` is *enabled*;
-   * :c:member:`Buffered stdio <PyConfig.buffered_stdio>` is *disabled*;
-   * :c:member:`Writing bytecode <PyConfig.write_bytecode>` is *disabled*;
-   * :c:member:`Signal handlers <PyConfig.install_signal_handlers>` are *enabled*;
+   * UTF-8 mode (:c:member:`PyPreConfig.utf8_mode`) is *enabled*;
+   * Buffered stdio (:c:member:`PyConfig.buffered_stdio`) is *disabled*;
+   * Writing bytecode (:c:member:`PyConfig.write_bytecode`) is *disabled*;
+   * Signal handlers (:c:member:`PyConfig.install_signal_handlers`) are *enabled*;
+   * System logging (:c:member:`PyConfig.use_system_logger`) is *enabled*
+     (optional, but strongly recommended);
    * ``PYTHONHOME`` for the interpreter is configured to point at the
      ``python`` subfolder of your app's bundle; and
    * The ``PYTHONPATH`` for the interpreter includes:
@@ -324,6 +326,49 @@ modules in your app, some additional steps will be required:
 * If you're using a separate folder for third-party packages, ensure that folder
   is included as part of the ``PYTHONPATH`` configuration in step 10.
 
+Testing a Python package
+------------------------
+
+The CPython source tree contains :source:`a testbed project <iOS/testbed>` that
+is used to run the CPython test suite on the iOS simulator. This testbed can also
+be used as a testbed project for running your Python library's test suite on iOS.
+
+After building or obtaining an iOS XCFramework (See :source:`iOS/README.rst`
+for details), create a clone of the Python iOS testbed project by running:
+
+.. code-block:: bash
+
+    $ python iOS/testbed clone --framework <path/to/Python.xcframework> --app <path/to/module1> --app <path/to/module2> app-testbed
+
+You will need to modify the ``iOS/testbed`` reference to point to that
+directory in the CPython source tree; any folders specified with the ``--app``
+flag will be copied into the cloned testbed project. The resulting testbed will
+be created in the ``app-testbed`` folder. In this example, the ``module1`` and
+``module2`` would be importable modules at runtime. If your project has
+additional dependencies, they can be installed into the
+``app-testbed/iOSTestbed/app_packages`` folder (using ``pip install --target
+app-testbed/iOSTestbed/app_packages`` or similar).
+
+You can then use the ``app-testbed`` folder to run the test suite for your app,
+For example, if ``module1.tests`` was the entry point to your test suite, you
+could run:
+
+.. code-block:: bash
+
+    $ python app-testbed run -- module1.tests
+
+This is the equivalent of running ``python -m module1.tests`` on a desktop
+Python build. Any arguments after the ``--`` will be passed to the testbed as
+if they were arguments to ``python -m`` on a desktop machine.
+
+You can also open the testbed project in Xcode by running:
+
+.. code-block:: bash
+
+    $ open app-testbed/iOSTestbed.xcodeproj
+
+This will allow you to use the full Xcode suite of tools for debugging.
+
 App Store Compliance
 ====================
 

@@ -285,52 +285,42 @@ This will:
 * Install the Python iOS framework into the copy of the testbed project; and
 * Run the test suite on an "iPhone SE (3rd generation)" simulator.
 
-While the test suite is running, Xcode does not display any console output.
-After showing some Xcode build commands, the console output will print ``Testing
-started``, and then appear to stop. It will remain in this state until the test
-suite completes. On a 2022 M1 MacBook Pro, the test suite takes approximately 12
-minutes to run; a couple of extra minutes is required to boot and prepare the
-iOS simulator.
-
 On success, the test suite will exit and report successful completion of the
-test suite. No output of the Python test suite will be displayed.
-
-On failure, the output of the Python test suite *will* be displayed. This will
-show the details of the tests that failed.
+test suite. On a 2022 M1 MacBook Pro, the test suite takes approximately 12
+minutes to run; a couple of extra minutes is required to compile the testbed
+project, and then boot and prepare the iOS simulator.
 
 Debugging test failures
 -----------------------
 
-The easiest way to diagnose a single test failure is to open the testbed project
-in Xcode and run the tests from there using the "Product > Test" menu item.
-
-To test in Xcode, you must ensure the testbed project has a copy of a compiled
-framework. If you've configured your build with the default install location of
-``iOS/Frameworks``, you can copy from that location into the test project. To
-test on an ARM64 simulator, run::
-
-    $ rm -rf iOS/testbed/Python.xcframework/ios-arm64_x86_64-simulator/*
-    $ cp -r iOS/Frameworks/arm64-iphonesimulator/* iOS/testbed/Python.xcframework/ios-arm64_x86_64-simulator
+Running ``make test`` generates a standalone version of the ``iOS/testbed``
+project, and runs the full test suite. It does this using ``iOS/testbed``
+itself - the folder is an executable module that can be used to create and run
+a clone of the testbed project.
 
-To test on an x86-64 simulator, run::
+You can generate your own standalone testbed instance by running::
 
-    $ rm -rf iOS/testbed/Python.xcframework/ios-arm64_x86_64-simulator/*
-    $ cp -r iOS/Frameworks/x86_64-iphonesimulator/* iOS/testbed/Python.xcframework/ios-arm64_x86_64-simulator
+    $ python iOS/testbed clone --framework iOS/Frameworks/arm64-iphonesimulator my-testbed
 
-To test on a physical device::
+This invocation assumes that ``iOS/Frameworks/arm64-iphonesimulator`` is the
+path to the iOS simulator framework for your platform (ARM64 in this case);
+``my-testbed`` is the name of the folder for the new testbed clone.
 
-    $ rm -rf iOS/testbed/Python.xcframework/ios-arm64/*
-    $ cp -r iOS/Frameworks/arm64-iphoneos/* iOS/testbed/Python.xcframework/ios-arm64
+You can then use the ``my-testbed`` folder to run the Python test suite,
+passing in any command line arguments you may require. For example, if you're
+trying to diagnose a failure in the ``os`` module, you might run::
 
-Alternatively, you can configure your build to install directly into the
-testbed project. For a simulator, use::
+    $ python my-testbed run -- test -W test_os
 
-    --enable-framework=$(pwd)/iOS/testbed/Python.xcframework/ios-arm64_x86_64-simulator
+This is the equivalent of running ``python -m test -W test_os`` on a desktop
+Python build. Any arguments after the ``--`` will be passed to testbed as if
+they were arguments to ``python -m`` on a desktop machine.
 
-For a physical device, use::
+You can also open the testbed project in Xcode by running::
 
-    --enable-framework=$(pwd)/iOS/testbed/Python.xcframework/ios-arm64
+    $ open my-testbed/iOSTestbed.xcodeproj
 
+This will allow you to use the full Xcode suite of tools for debugging.
 
 Testing on an iOS device
 ^^^^^^^^^^^^^^^^^^^^^^^^