Clearly reproduce Cygwin safe.directory submodule bug

EliahKagan · claude · EliahKagan · commit ce68322cc3b3 · 2026-05-17T23:45:34.000-04:00
Remove the Cygwin xfail decorations from `test_submodules` in `test_docs.py` and `test_repo.py`, and from `test_root_module` in `test_submodule.py`, so the tests surface the underlying failure directly. Also temporarily add 256 `reproduce-safe-dir` matrix jobs to `.github/workflows/cygwin-test.yml`, each running these three tests under the current `safe.directory` configuration. All or most of these `reproduce-safe-dir` jobs shall be removed before the associated bugfix is integrated. The existing test job's `env`, `defaults`, and setup steps gain YAML anchors so the new job can reference them without duplication. The anchors will be removed when the `reproduce-safe-dir` jobs are. Root cause ---------- The CI workflow adds the main repo and its `.git` directory to `safe.directory` but not the gitdb or smmap submodule working trees. Git matches `safe.directory` by specific paths (or wildcards, but we are not using wildcards). When GitPython opens a submodule as a `Repo` and runs `git cat-file --batch-check` against it, git rejects the repository for dubious ownership and exits. The race -------- GitPython caches the cat-file process. When git has exited, the next call to `__get_object_header` hits one of two outcomes depending on whether Python's `cmd.stdin.flush()` runs before or after the kernel marks the read end of the pipe as closed. (This is the same mechanism as the long-standing #427, in which SIGINT kills the cat-file process and the next `flush()` raises `BrokenPipeError`.) 1. If Python wins the race, `flush()` succeeds (data goes into the kernel buffer); the subsequent `stdout.readline()` returns `b""` (EOF); `_parse_object_header` raises a `ValueError` whose message names the rejected directory. 2. If git wins, `flush()` raises `BrokenPipeError`, a subclass of `OSError` (a.k.a. `IOError`). In both paths, the exception travels from `Object.new_from_sha` (outside `name_to_object`'s `except ValueError`, which covers only the `dereference_recursive` call), up through `repo.commit("HEAD")`, into `iter_items` in `git/objects/submodule/base.py`. That function's `except (IOError, BadName)` clause catches `BrokenPipeError` but not `ValueError`. So path (1) propagates `ValueError` all the way to the test, while path (2) ends with `iter_items` returning early and the test seeing an empty submodule list. Per-test outcomes ----------------- What the test sees in path (2) is determined by what the test does with the empty list: - `test_docs::test_submodules` does `sm.children()[0].name`, so the `[0]` on the empty list raises `IndexError`. - `test_repo::test_submodules` does `assertGreaterEqual(len(list(self.rorepo.iter_submodules())), 2)`. The recursive traversal yields gitdb (its `iter_items` on the *main* repo succeeds, because the main repo IS in `safe.directory`) but not smmap, so length 1 fails the assertion. - `test_submodule::test_root_module` does `assert len(rsmsp) >= 2` on a similar traversal result, failing. The race itself is non-deterministic, but the mapping from race outcome to exception type is deterministic per test. So each test has exactly two possible failure types, one per side of the race: - `test_docs`: `ValueError` -> Python wins, OR `IndexError` -> git wins. - `test_repo`: `ValueError` -> Python wins, OR `AssertionError` -> git wins. - `test_submodule`: `ValueError` -> Python wins, OR `AssertionError` -> git wins. In particular, `test_docs` never produces `AssertionError`, and `test_repo` and `test_submodule` never produce `IndexError`. Empirical confirmation ---------------------- In 1024 `reproduce-safe-dir` jobs and 4 buggy-config "test (fast)" job runs, 100% of 3084 target-test failures match the per-test prediction. `ValueError` accounts for ~99.0%, while the race-win exception types from the list above account for the others. `reproduce-safe-dir` runs: https://github.com/EliahKagan/GitPython/actions/runs/25836741324 https://github.com/EliahKagan/GitPython/actions/runs/25836329241 https://github.com/EliahKagan/GitPython/actions/runs/25836334196 https://github.com/EliahKagan/GitPython/actions/runs/25836339345 Run on the fix commit (256 jobs, 768 test outcomes, all PASSED): https://github.com/EliahKagan/GitPython/actions/runs/25836344886 The race condition, from a "test (fast)" job in a PR #2143 CI run: https://github.com/gitpython-developers/GitPython/actions/runs/25440735020/attempts/1 Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
diff --git a/.github/workflows/cygwin-test.yml b/.github/workflows/cygwin-test.yml
@@ -20,63 +20,73 @@ jobs:
 
     runs-on: windows-latest
 
-    env:
+    env: &cygwin-env
       CHERE_INVOKING: "1"
       CYGWIN_NOWINPATH: "1"
 
-    defaults:
+    defaults: &cygwin-defaults
       run:
         shell: C:\cygwin\bin\bash.exe --login --norc -eo pipefail -o igncr "{0}"
 
     steps:
-    - name: Force LF line endings
+    - &force-lf
+      name: Force LF line endings
       run: |
         git config --global core.autocrlf false  # Affects the non-Cygwin git.
       shell: pwsh  # Do this outside Cygwin, to affect actions/checkout.
 
-    - uses: actions/checkout@v6
+    - &checkout
+      uses: actions/checkout@v6
       with:
         fetch-depth: 0
 
-    - name: Install Cygwin
+    - &install-cygwin
+      name: Install Cygwin
       uses: cygwin/cygwin-install-action@v6
       with:
         packages: git python39 python-pip-wheel python-setuptools-wheel python-wheel-wheel
         add-to-path: false  # No need to change $PATH outside the Cygwin environment.
 
-    - name: Arrange for verbose output
+    - &verbose-output
+      name: Arrange for verbose output
       run: |
         # Arrange for verbose output but without shell environment setup details.
         echo 'set -x' >~/.bash_profile
 
-    - name: Special configuration for Cygwin git
+    - &safe-directory
+      name: Special configuration for Cygwin git
       run: |
         git config --global --add safe.directory "$(pwd)"
         git config --global --add safe.directory "$(pwd)/.git"
         git config --global core.autocrlf false
 
-    - name: Prepare this repo for tests
+    - &prepare-repo
+      name: Prepare this repo for tests
       run: |
         ./init-tests-after-clone.sh
 
-    - name: Set git user identity and command aliases for the tests
+    - &git-identity
+      name: Set git user identity and command aliases for the tests
       run: |
         git config --global user.email "travis@ci.com"
         git config --global user.name "Travis Runner"
         # If we rewrite the user's config by accident, we will mess it up
         # and cause subsequent tests to fail
         cat test/fixtures/.gitconfig >> ~/.gitconfig
 
-    - name: Set up virtual environment
+    - &setup-venv
+      name: Set up virtual environment
       run: |
         python3.9 -m venv .venv
         echo 'BASH_ENV=.venv/bin/activate' >>"$GITHUB_ENV"
 
-    - name: Update PyPA packages
+    - &update-pypa
+      name: Update PyPA packages
       run: |
         python -m pip install -U pip 'setuptools; python_version<"3.12"' wheel
 
-    - name: Install project and test dependencies
+    - &install-deps
+      name: Install project and test dependencies
       run: |
         pip install '.[test]'
 
@@ -91,3 +101,34 @@ jobs:
     - name: Test with pytest (${{ matrix.additional-pytest-args }})
       run: |
         pytest --color=yes -p no:sugar --instafail -vv ${{ matrix.additional-pytest-args }}
+
+  reproduce-safe-dir:
+    strategy:
+      matrix:
+        run: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100, 101, 102, 103, 104, 105, 106, 107, 108, 109, 110, 111, 112, 113, 114, 115, 116, 117, 118, 119, 120, 121, 122, 123, 124, 125, 126, 127, 128, 129, 130, 131, 132, 133, 134, 135, 136, 137, 138, 139, 140, 141, 142, 143, 144, 145, 146, 147, 148, 149, 150, 151, 152, 153, 154, 155, 156, 157, 158, 159, 160, 161, 162, 163, 164, 165, 166, 167, 168, 169, 170, 171, 172, 173, 174, 175, 176, 177, 178, 179, 180, 181, 182, 183, 184, 185, 186, 187, 188, 189, 190, 191, 192, 193, 194, 195, 196, 197, 198, 199, 200, 201, 202, 203, 204, 205, 206, 207, 208, 209, 210, 211, 212, 213, 214, 215, 216, 217, 218, 219, 220, 221, 222, 223, 224, 225, 226, 227, 228, 229, 230, 231, 232, 233, 234, 235, 236, 237, 238, 239, 240, 241, 242, 243, 244, 245, 246, 247, 248, 249, 250, 251, 252, 253, 254, 255, 256]
+      fail-fast: false
+
+    runs-on: windows-latest
+
+    env: *cygwin-env
+
+    defaults: *cygwin-defaults
+
+    steps:
+    - *force-lf
+    - *checkout
+    - *install-cygwin
+    - *verbose-output
+    - *safe-directory
+    - *prepare-repo
+    - *git-identity
+    - *setup-venv
+    - *update-pypa
+    - *install-deps
+
+    - name: Run submodule tests
+      run: |
+        python -m pytest -vv \
+          test/test_docs.py::Tutorials::test_submodules \
+          test/test_repo.py::TestRepo::test_submodules \
+          test/test_submodule.py::TestSubmodule::test_root_module
diff --git a/test/test_docs.py b/test/test_docs.py
@@ -6,9 +6,6 @@
 import gc
 import os
 import os.path
-import sys
-
-import pytest
 
 from test.lib import TestBase
 from test.lib.helper import with_rw_directory
@@ -478,11 +475,6 @@ def test_references_and_objects(self, rw_dir):
 
         repo.git.clear_cache()
 
-    @pytest.mark.xfail(
-        sys.platform == "cygwin",
-        reason="Cygwin GitPython can't find SHA for submodule",
-        raises=ValueError,
-    )
     def test_submodules(self):
         # [1-test_submodules]
         repo = self.rorepo
diff --git a/test/test_repo.py b/test/test_repo.py
@@ -877,11 +877,6 @@ def test_repo_odbtype(self):
         target_type = GitCmdObjectDB
         self.assertIsInstance(self.rorepo.odb, target_type)
 
-    @pytest.mark.xfail(
-        sys.platform == "cygwin",
-        reason="Cygwin GitPython can't find submodule SHA",
-        raises=ValueError,
-    )
     def test_submodules(self):
         self.assertEqual(len(self.rorepo.submodules), 1)  # non-recursive
         self.assertGreaterEqual(len(list(self.rorepo.iter_submodules())), 2)
diff --git a/test/test_submodule.py b/test/test_submodule.py
@@ -480,11 +480,6 @@ def test_base_rw(self, rwrepo):
     def test_base_bare(self, rwrepo):
         self._do_base_tests(rwrepo)
 
-    @pytest.mark.xfail(
-        sys.platform == "cygwin",
-        reason="Cygwin GitPython can't find submodule SHA",
-        raises=ValueError,
-    )
     @pytest.mark.xfail(
         HIDE_WINDOWS_KNOWN_ERRORS,
         reason=(
