gh-86179: Implement realpath() on Windows during getpath.py calculations #113033

zooba · 2023-12-12T20:57:32Z

Issue: venv on Windows with symlinks is broken if invoked with -I #86179

Modules/getpath.c

miss-islington-app · 2023-12-13T23:41:47Z

Thanks @zooba for the PR 🌮🎉.. I'm working now to backport this PR to: 3.11, 3.12.
🐍🍒⛏🤖 I'm not a witch! I'm not a witch!

miss-islington-app · 2023-12-13T23:41:50Z

Sorry, @zooba, I could not cleanly backport this to 3.12 due to a conflict.
Please backport using cherry_picker on command line.

cherry_picker fddc829236d7b29a522a2160e57b2d7ca23b9b95 3.12

miss-islington-app · 2023-12-13T23:42:04Z

Sorry, @zooba, I could not cleanly backport this to 3.11 due to a conflict.
Please backport using cherry_picker on command line.

cherry_picker fddc829236d7b29a522a2160e57b2d7ca23b9b95 3.11

bedevere-bot · 2023-12-14T00:07:14Z

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot AMD64 Windows11 Non-Debug 3.x has failed when building commit fddc829.

What do you need to do:

Don't panic.
Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
Go to the page of the buildbot that failed (https://buildbot.python.org/all/#builders/914/builds/3204) and take a look at the build logs.
Check if the failure is related to this commit (fddc829) or if it is a false positive.
If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/all/#builders/914/builds/3204

Failed tests:

test_embed
test_venv

Failed subtests:

test_init_is_python_build_with_home - test.test_embed.InitConfigTests.test_init_is_python_build_with_home

Summary of the results of the build (if available):

==

Click to see traceback logs

Traceback (most recent call last):
  File "B:\uildarea\3.x.ware-win11.nondebug\build\Lib\test\test_venv.py", line 269, in test_prefixes
    self.assertEqual(out.strip(), expected.encode(), prefix)
    ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError: b'b:\\uildarea\\3.x.ware-win11.nondebug\\build' != b'B:\\uildarea\\3.x.ware-win11.nondebug\\build' : base_prefix


Traceback (most recent call last):
  File "B:\uildarea\3.x.ware-win11.nondebug\build\Lib\test\test_venv.py", line 290, in test_sysconfig
    self.assertEqual(out.strip(), expected, err)
    ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError: 'b:\\uildarea\\3.x.ware-win11.nondebug\\build\\Makefile' != 'B:\\uildarea\\3.x.ware-win11.nondebug\\build\\Makefile'
- b:\uildarea\3.x.ware-win11.nondebug\build\Makefile
? ^
+ B:\uildarea\3.x.ware-win11.nondebug\build\Makefile
? ^
 : 


Traceback (most recent call last):
  File "B:\uildarea\3.x.ware-win11.nondebug\build\Lib\test\test_embed.py", line 1397, in test_init_is_python_build_with_home
    self.check_all_configs("test_init_is_python_build", config,
    ~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
                           api=API_COMPAT, env=env)
                           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "B:\uildarea\3.x.ware-win11.nondebug\build\Lib\test\test_embed.py", line 812, in check_all_configs
    self.check_config(configs, expected_config)
    ~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "B:\uildarea\3.x.ware-win11.nondebug\build\Lib\test\test_embed.py", line 738, in check_config
    self.assertEqual(config, expected)
    ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^
AssertionError: {'_co[129 chars] 0, 'faulthandler': 0, 'tracemalloc': 0, 'perf[2127 chars]': 1} != {'_co[129 chars] 0, 'int_max_str_digits': 4300, 'cpu_count': -[2130 chars]': 0}
  {'_config_init': 1,
   '_init_main': 1,
   '_install_importlib': 1,
   '_is_python_build': 1,
   'argv': [''],
   'base_exec_prefix': 'B:\\uildarea\\3.x.ware-win11.nondebug\\build',
   'base_executable': 'b:\\uildarea\\3.x.ware-win11.nondebug\\build\\PCbuild\\amd64\\_testembed.exe',
   'base_prefix': 'B:\\uildarea\\3.x.ware-win11.nondebug\\build',
   'buffered_stdio': 1,
   'bytes_warning': 0,
   'check_hash_pycs_mode': 'default',
   'code_debug_ranges': 1,
   'configure_c_stdio': 0,
   'cpu_count': -1,
   'dev_mode': 0,
   'dump_refs': 0,
   'dump_refs_file': None,
   'exec_prefix': 'B:\\uildarea\\3.x.ware-win11.nondebug\\build',
   'executable': 'b:\\uildarea\\3.x.ware-win11.nondebug\\build\\PCbuild\\amd64\\_testembed.exe',
   'faulthandler': 0,
   'hash_seed': 0,
   'home': 'B:\\uildarea\\3.x.ware-win11.nondebug\\build',
   'import_time': 0,
   'inspect': 0,
   'install_signal_handlers': 1,
   'int_max_str_digits': 4300,
   'interactive': 0,
   'isolated': 0,
   'legacy_windows_stdio': 0,
   'malloc_stats': 0,
   'module_search_paths': ['b:\\uildarea\\3.x.ware-win11.nondebug\\build\\PCbuild\\amd64\\python313.zip',
                           'B:\\uildarea\\3.x.ware-win11.nondebug\\build\\PCbuild\\amd64',
                           'B:\\uildarea\\3.x.ware-win11.nondebug\\build\\Lib',
                           'b:\\uildarea\\3.x.ware-win11.nondebug\\build\\PCbuild\\amd64\\python313.zip',
-                          'B:\\uildarea\\3.x.ware-win11.nondebug\\build\\PCbuild\\amd64',
?                           ^

zooba · 2023-12-14T00:38:43Z

The failure is because the realpath implementation will convert drive names to uppercase, which causes case-sensitive path comparisons in the tests to fail. I'm pretty sure this was reported somewhere else for os.path.realpath as well, so I guess we need to do one more comparison before returning in both cases and not return case-only changes.

bedevere-app · 2023-12-14T16:13:43Z

GH-113121 is a backport of this pull request to the 3.12 branch.

…ations (pythonGH-113033)

Yhg1s · 2024-01-16T16:04:56Z

This seems to have introduced a new test failure, but only when running tests in an installed Python and using the unversioned binary (bin/python3 rather than bin/python3.13). Rolling back the changes in this PR makes both tests pass.

% /tmp/testinstall/bin/python3.13 -m test test_venv
Using random seed: 3548105807
0:00:00 load avg: 0.25 Run 1 test sequentially
0:00:00 load avg: 0.25 [1/1] test_venv

== Tests result: SUCCESS ==

1 test OK.

Total duration: 3.6 sec
Total tests: run=29 skipped=3
Total test files: run=1/1
Result: SUCCESS

% /tmp/testinstall/bin/python3 -m test test_venv
Using random seed: 3358697499
0:00:00 load avg: 0.06 Run 1 test sequentially
0:00:00 load avg: 0.06 [1/1] test_venv
test test_venv failed -- Traceback (most recent call last):
  File "/tmp/testinstall/lib/python3.13/test/test_venv.py", line 332, in test_sysconfig_symlinks
    self.assertEqual(out.strip(), expected, err)
    ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError: '/tmp/testinstall/bin/python3.13' != '/tmp/testinstall/bin/python3'
- /tmp/testinstall/bin/python3.13
?                             ---
+ /tmp/testinstall/bin/python3
 :

test_venv failed (1 failure)

== Tests result: FAILURE ==

1 test failed:
    test_venv

Total duration: 3.7 sec
Total tests: run=29 failures=1 skipped=3
Total test files: run=1/1 failed=1
Result: FAILURE

zooba · 2024-01-16T16:46:24Z

Rather than rolling back the (primarily Windows-only) change, better to skip the test.

I'll need to take some time to learn how venv/symlinks work on POSIX in order to diagnose and fix. As it's just a test, it may be faster if we can pull in someone who already knows that side of things.

FFY00 · 2024-01-16T16:53:05Z

You can ping me on Discord, or an issue to discuss the POSIX behaviors.

zooba · 2024-01-16T17:05:07Z

I'll push a PR to skip that part of the test so it's not blocking the release. If we want to test venv later, we can figure it out then.

…ations (pythonGH-113033)

zooba added 4 commits December 12, 2023 00:17

[WIP] Figuring out why sys._base_executable isn't resolved

8987eeb

Implement realpath for Windows

01b1f82

Merge remote-tracking branch 'upstream/main' into pythongh-86179

d01aa34

Revert sysconfig change

c69f931

zooba requested a review from vsajip as a code owner December 12, 2023 20:57

bedevere-app bot added the awaiting core review label Dec 12, 2023

bedevere-app bot mentioned this pull request Dec 12, 2023

venv on Windows with symlinks is broken if invoked with -I #86179

Closed

NEWS

902a093

zooba added OS-windows needs backport to 3.11 only security fixes needs backport to 3.12 only security fixes labels Dec 12, 2023

zooba and others added 3 commits December 12, 2023 23:14

Merge branch 'main' into pythongh-86179

d37c113

Merge remote-tracking branch 'upstream/main' into pythongh-86179

01bd133

Add missed realpath and simplify sysconfig

d579fa0

zooba requested a review from FFY00 as a code owner December 13, 2023 17:48

zooba added 2 commits December 13, 2023 19:41

Alternate fix

6f77da3

Update test

823d0dd

pfmoore reviewed Dec 13, 2023

View reviewed changes

Modules/getpath.c Outdated Show resolved Hide resolved

Fix error handling

11b229c

zooba enabled auto-merge (squash) December 13, 2023 23:26

zooba merged commit fddc829 into python:main Dec 13, 2023

bedevere-app bot removed the awaiting core review label Dec 13, 2023

miss-islington-app bot assigned zooba Dec 13, 2023

zooba deleted the gh-86179 branch December 14, 2023 00:34

bedevere-app bot removed the needs backport to 3.12 only security fixes label Dec 14, 2023

zooba added a commit to zooba/cpython that referenced this pull request Dec 14, 2023

pythongh-86179: Implement realpath() on Windows for getpath.py calcul…

23703a9

…ations (pythonGH-113033)

aisk pushed a commit to aisk/cpython that referenced this pull request Feb 11, 2024

pythongh-86179: Implement realpath() on Windows for getpath.py calcul…

1af21d5

…ations (pythonGH-113033)

Glyphack pushed a commit to Glyphack/cpython that referenced this pull request Sep 2, 2024

pythongh-86179: Implement realpath() on Windows for getpath.py calcul…

390b1fa

…ations (pythonGH-113033)

ZeroIntensity removed the needs backport to 3.11 only security fixes label Feb 17, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

gh-86179: Implement realpath() on Windows during getpath.py calculations #113033

gh-86179: Implement realpath() on Windows during getpath.py calculations #113033

Uh oh!

zooba commented Dec 12, 2023 •

edited by bedevere-app bot

Loading

Uh oh!

Uh oh!

miss-islington-app bot commented Dec 13, 2023

Uh oh!

miss-islington-app bot commented Dec 13, 2023

Uh oh!

miss-islington-app bot commented Dec 13, 2023

Uh oh!

bedevere-bot commented Dec 14, 2023

Uh oh!

zooba commented Dec 14, 2023

Uh oh!

bedevere-app bot commented Dec 14, 2023

Uh oh!

Yhg1s commented Jan 16, 2024

Uh oh!

zooba commented Jan 16, 2024

Uh oh!

FFY00 commented Jan 16, 2024

Uh oh!

zooba commented Jan 16, 2024

Uh oh!

Uh oh!

Uh oh!

gh-86179: Implement realpath() on Windows during getpath.py calculations #113033

gh-86179: Implement realpath() on Windows during getpath.py calculations #113033

Uh oh!

Conversation

zooba commented Dec 12, 2023 • edited by bedevere-app bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

miss-islington-app bot commented Dec 13, 2023

Uh oh!

miss-islington-app bot commented Dec 13, 2023

Uh oh!

miss-islington-app bot commented Dec 13, 2023

Uh oh!

bedevere-bot commented Dec 14, 2023

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Uh oh!

zooba commented Dec 14, 2023

Uh oh!

bedevere-app bot commented Dec 14, 2023

Uh oh!

Yhg1s commented Jan 16, 2024

Uh oh!

zooba commented Jan 16, 2024

Uh oh!

FFY00 commented Jan 16, 2024

Uh oh!

zooba commented Jan 16, 2024

Uh oh!

Uh oh!

zooba commented Dec 12, 2023 •

edited by bedevere-app bot

Loading