bpo-34022: Stop forcing of hash-based invalidation with SOURCE_DATE_EPOCH #9607

elprans · 2018-09-27T17:19:42Z

Unconditional forcing of CHECKED_HASH invalidation was introduced in
3.7.0 in bpo-29708. The change is bad, as it unconditionally overrides
invalidation_mode, even if it was passed as an explicit argument to
py_compile.compile() or compileall. An environment variable
should never override an explicit argument to a library function.
That change leads to multiple test failures if the SOURCE_DATE_EPOCH
environment variable is set.

This changes py_compile.compile() to only look at
SOURCE_DATE_EPOCH if no explicit invalidation_mode was specified.
I also made various relevant tests run with explicit control over the
value of SOURCE_DATE_EPOCH.

While looking at this, I noticed that zipimport does not work
with hash-based .pycs at all, though I left the fixes for
subsequent commits.

https://bugs.python.org/issue34022

…POCH Unconditional forcing of ``CHECKED_HASH`` invalidation was introduced in 3.7.0 in bpo-29708. The change is bad, as it unconditionally overrides *invalidation_mode*, even if it was passed as an explicit argument to ``py_compile.compile()`` or ``compileall``. An environment variable should *never* override an explicit argument to a library function. That change leads to multiple test failures if the ``SOURCE_DATE_EPOCH`` environment variable is set. This changes ``py_compile.compile()`` to only look at ``SOURCE_DATE_EPOCH`` if no explicit *invalidation_mode* was specified. I also made various relevant tests run with explicit control over the value of ``SOURCE_DATE_EPOCH``. While looking at this, I noticed that ``zipimport`` does not work with hash-based .pycs _at all_, though I left the fixes for subsequent commits.

merwok · 2018-09-27T17:23:47Z

Doc/library/py_compile.rst

+   .. versionchanged:: 3.7.2
+      The :envvar:`SOURCE_DATE_EPOCH` environment variable no longer
+      overrides the value of the *invalidation_mode* argument, and determines
+      it's default value instead.


vstinner

LGTM, but I have a few minor comments.

I will not approve this PR since it's the work of @benjaminp and I'm not sure that the change respects his design (PEP 552) (I'm not sure that he agrees with the change).

vstinner · 2018-09-28T14:13:39Z

Doc/library/compileall.rst

+   bytecode cache files at runtime.
+   The default is ``timestamp`` if the :envvar:`SOURCE_DATE_EPOCH` environment
+   variable is not set, and ``checked-hash`` if the ``SOURCE_DATE_EPOCH``
+   environment variable is set.


"is set": I suggest "is set to a non-empty string". See other existing examples:
https://docs.python.org/dev/using/cmdline.html#envvar-PYTHONOPTIMIZE

Maybe add a reference to the PEP 552 somewhere (you can use syntax :pep:552).

By the way, should we mention SOURCE_DATE_EPOCH somewhere in https://docs.python.org/dev/using/cmdline.html ? The variable is not specific to Python, but it changes its behavior.

vstinner · 2018-09-28T14:13:53Z

Doc/library/py_compile.rst

+   enum and controls how the generated bytecode cache is invalidated at
+   runtime.  The default is :attr:`PycInvalidationMode.CHECKED_HASH` if
+   the :envvar:`SOURCE_DATE_EPOCH` environment variable is set, otherwise
+   the default is :attr:`PycInvalidationMode.TIMESTAMP`.


Maybe add a reference to the PEP 552 somewhere (you can use syntax :pep:552).

PEP 552 actually has no mention of SOURCE_DATE_EPOCH, so I'm not sure such a reference would be useful here.

invalidation_mode is a new thing coming from the PEP 552. Moreover, I proposed to update the PEP:
https://bugs.python.org/issue34022#msg326638

vstinner · 2018-09-28T14:19:04Z

Lib/test/test_py_compile.py

+    @functools.wraps(fxn)
+    def wrapper(*args, **kwargs):
+        with support.EnvironmentVarGuard() as env:
+            env.unset('SOURCE_DATE_EPOCH')


Is there at least one unit test for SOURCE_DATE_EPOCH set to an empty string?

elprans · 2018-09-28T15:34:18Z

I will not approve this PR since it's the work of @benjaminp and I'm not sure that the change respects his design (PEP 552) (I'm not sure that he agrees with the change).

Benjamin's comment on the issue:

I dislike that having SOURCE_DATE_EPOCH set magically changes command line tools behavior. In my view, this problem should be fixed by reverting ccbe581.

Reverting seems like a drastic measure, I think this PR achieves a better balance.

vstinner · 2018-09-28T15:44:21Z

cc @bmwiedemann who authored PR #5200.

bmwiedemann · 2018-09-28T16:22:34Z

I really like this PR as it is what I should have written, if I was a python guru.

We do need some way to get reproducible .pyc files across all of openSUSE's ~1800 python packages without having to touch each of them individually to add an invalidation_mode.
If there is some other preferable approach to achieve that, I'd love to hear about that.

vstinner · 2018-10-10T09:05:48Z

Hum, a few tests still fail with SOURCE_DATE_EPOCH=0:

test_multiprocessing_main_handling
test_cmd_line_script
test_runpy

@elprans: Do you want to fix these tests in the same PR, or do you prefer to fix them in a different PR?

elprans · 2018-10-10T14:16:11Z

@vstinner Separate PR. Those are unrelated to py_compile and are due to broken zipimport.

vstinner · 2018-10-10T16:44:41Z

@ned-deily, @benjaminp, @elprans: I guess that this change can wait Python 3.7.2, it's not a blocker issue nor a major bug. So I suggest to wait until 3.7.1 final is released before backporting this change to the 3.7 branch.

vstinner · 2018-10-10T16:45:07Z

@vstinner Separate PR. Those are unrelated to py_compile and are due to broken zipimport.

Perfect. In this case, I see no other reason to no merge your PR :-) I merged it, good job! Thanks @elprans.

ned-deily · 2018-10-10T16:50:43Z

Unless a release manager says otherwise, please don't wait between RC's and finals to merge PRs. It's better to get them exposed on buildbots. We'll cherry-picked fixes as necessary into the finals.

vstinner · 2018-10-10T21:13:11Z

Unless a release manager says otherwise, please don't wait between RC's and finals to merge PRs. It's better to get them exposed on buildbots. We'll cherry-picked fixes as necessary into the finals.

The merged change is a backward incompatible and it mentions the version 3.7.2 in the modified documentation :-)

ned-deily · 2018-10-10T21:20:03Z

The docs can be easily changed for the final. The question is how much of an impact is it for 3.7.1 users by not having this change in vs having it in..

Some SOURCE_DATE_EPOCH-related tests are fixed in python/cpython#9607

mcepl · 2018-10-15T11:38:10Z

@elprans @vstinner Separate PR. Those are unrelated to py_compile and are due to broken zipimport.

Is there such PR somewhere? Or at least a bug?

elprans · 2018-10-16T14:23:32Z

@mcepl I'm working on a followup PR to fix these.

mcepl · 2018-10-16T14:42:46Z

@elprans thanks

miss-islington · 2018-11-28T17:18:51Z

Thanks @elprans for the PR, and @vstinner for merging it 🌮🎉.. I'm working now to backport this PR to: 3.7.
🐍🍒⛏🤖

bedevere-bot · 2018-11-28T17:19:04Z

GH-10775 is a backport of this pull request to the 3.7 branch.

…POCH (pythonGH-9607) Unconditional forcing of ``CHECKED_HASH`` invalidation was introduced in 3.7.0 in bpo-29708. The change is bad, as it unconditionally overrides *invalidation_mode*, even if it was passed as an explicit argument to ``py_compile.compile()`` or ``compileall``. An environment variable should *never* override an explicit argument to a library function. That change leads to multiple test failures if the ``SOURCE_DATE_EPOCH`` environment variable is set. This changes ``py_compile.compile()`` to only look at ``SOURCE_DATE_EPOCH`` if no explicit *invalidation_mode* was specified. I also made various relevant tests run with explicit control over the value of ``SOURCE_DATE_EPOCH``. While looking at this, I noticed that ``zipimport`` does not work with hash-based .pycs _at all_, though I left the fixes for subsequent commits. (cherry picked from commit a6b3ec5) Co-authored-by: Elvis Pranskevichus <[email protected]>

…POCH (GH-9607) Unconditional forcing of ``CHECKED_HASH`` invalidation was introduced in 3.7.0 in bpo-29708. The change is bad, as it unconditionally overrides *invalidation_mode*, even if it was passed as an explicit argument to ``py_compile.compile()`` or ``compileall``. An environment variable should *never* override an explicit argument to a library function. That change leads to multiple test failures if the ``SOURCE_DATE_EPOCH`` environment variable is set. This changes ``py_compile.compile()`` to only look at ``SOURCE_DATE_EPOCH`` if no explicit *invalidation_mode* was specified. I also made various relevant tests run with explicit control over the value of ``SOURCE_DATE_EPOCH``. While looking at this, I noticed that ``zipimport`` does not work with hash-based .pycs _at all_, though I left the fixes for subsequent commits. (cherry picked from commit a6b3ec5) Co-authored-by: Elvis Pranskevichus <[email protected]>

the-knights-who-say-ni added the CLA signed label Sep 27, 2018

bedevere-bot added the awaiting review label Sep 27, 2018

elprans mentioned this pull request Sep 27, 2018

bpo-34022: Unbreak importlib tests when SOURCE_DATE_EPOCH is set #9484

Closed

merwok reviewed Sep 27, 2018

View reviewed changes

Fix typo

6281a5c

vstinner requested a review from benjaminp September 28, 2018 00:15

vstinner approved these changes Sep 28, 2018

View reviewed changes

bedevere-bot added awaiting merge and removed awaiting review labels Sep 28, 2018

vstinner merged commit a6b3ec5 into python:master Oct 10, 2018

bedevere-bot removed the awaiting merge label Oct 10, 2018

yan12125 added a commit to archlinuxcn/repo that referenced this pull request Oct 11, 2018

python-git: fix some tests

e4de459

Some SOURCE_DATE_EPOCH-related tests are fixed in python/cpython#9607

bmwiedemann mentioned this pull request Oct 15, 2018

WIP: bpo-34990: year 2038 problem in compileall.py #9892

Closed

vstinner added the needs backport to 3.7 label Nov 28, 2018

bedevere-bot removed the needs backport to 3.7 label Nov 28, 2018

bmwiedemann mannequin mentioned this pull request May 28, 2022

support reproducible Python builds #73894

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

bpo-34022: Stop forcing of hash-based invalidation with SOURCE_DATE_EPOCH #9607

bpo-34022: Stop forcing of hash-based invalidation with SOURCE_DATE_EPOCH #9607

elprans commented Sep 27, 2018 •

edited by bedevere-bot

Loading

merwok Sep 27, 2018

vstinner left a comment

vstinner Sep 28, 2018

vstinner Sep 28, 2018

elprans Sep 28, 2018

vstinner Sep 28, 2018

vstinner Sep 28, 2018

elprans commented Sep 28, 2018

vstinner commented Sep 28, 2018

bmwiedemann commented Sep 28, 2018

vstinner commented Oct 10, 2018

elprans commented Oct 10, 2018

vstinner commented Oct 10, 2018

vstinner commented Oct 10, 2018

ned-deily commented Oct 10, 2018

vstinner commented Oct 10, 2018

ned-deily commented Oct 10, 2018

mcepl commented Oct 15, 2018

elprans commented Oct 16, 2018

mcepl commented Oct 16, 2018

miss-islington commented Nov 28, 2018

bedevere-bot commented Nov 28, 2018

bpo-34022: Stop forcing of hash-based invalidation with SOURCE_DATE_EPOCH #9607

bpo-34022: Stop forcing of hash-based invalidation with SOURCE_DATE_EPOCH #9607

Conversation

elprans commented Sep 27, 2018 • edited by bedevere-bot Loading

merwok Sep 27, 2018

Choose a reason for hiding this comment

vstinner left a comment

Choose a reason for hiding this comment

vstinner Sep 28, 2018

Choose a reason for hiding this comment

vstinner Sep 28, 2018

Choose a reason for hiding this comment

elprans Sep 28, 2018

Choose a reason for hiding this comment

vstinner Sep 28, 2018

Choose a reason for hiding this comment

vstinner Sep 28, 2018

Choose a reason for hiding this comment

elprans commented Sep 28, 2018

vstinner commented Sep 28, 2018

bmwiedemann commented Sep 28, 2018

vstinner commented Oct 10, 2018

elprans commented Oct 10, 2018

vstinner commented Oct 10, 2018

vstinner commented Oct 10, 2018

ned-deily commented Oct 10, 2018

vstinner commented Oct 10, 2018

ned-deily commented Oct 10, 2018

mcepl commented Oct 15, 2018

elprans commented Oct 16, 2018

mcepl commented Oct 16, 2018

miss-islington commented Nov 28, 2018

bedevere-bot commented Nov 28, 2018

elprans commented Sep 27, 2018 •

edited by bedevere-bot

Loading