gh-85012: Properly reset `msgctxt` when compiling messages with `msgfmt` #130525

tomasr8 · 2025-02-24T21:03:53Z

I realized that the downside of only having binary snapshots is that it's much more difficult to review any changes. So I added human-readable snapshots as well in this PR.

These snapshots are just JSON dumps of GNUTranslations._catalog when the MO file is read back. (I thought about using PO files for the snapshots but that would require some additional changes in pygettext so I opted for this for now)

I first generated the new snapshots and then applied the fix so you can see the diff in the snapshots here: 69402a5

What you're looking for is correct msgctxt (the part before \u0004 in each message). This is the source file for reference:
https://github.com/python/cpython/blob/main/Lib/test/test_tools/msgfmt_data/general.po

Issue: Incorrect handling for msgctxt in msgfmt.py #85012

tomasr8 · 2025-02-24T21:26:59Z

cc @StanFromIreland @m-aciek just in case

serhiy-storchaka

The 69402a5 changeset does not look particularly readable. For some reasons this change causes changing the order of the _catalog items, so it is difficult to see changes in values if the items are reordered. Do you know why is it? Can the order be made more stable?

There are no new tests, so we can only visually control that the behavior is correct. But if in future the order will be change unpredictably, we can miss a regression. Please add a test for the change, preferably a translation test that does not depend on implementation details of GNUTranslations.

tomasr8 · 2025-02-25T22:11:34Z

Do you know why is it?

It was sorted, the issue was that the message context was/is part of the message and so fixing that resulted in many messages being reordered. I changed the sorting to always put plurals at the end which produces IMO a much more readable diff: 8295b2b

Please add a test for the change, preferably a translation test that does not depend on implementation details of GNUTranslations

Added in 97d0a0f. It tests all mesages from the source PO file.

serhiy-storchaka

LGTM.

miss-islington-app · 2025-03-13T18:41:24Z

Thanks @tomasr8 for the PR, and @serhiy-storchaka for merging it 🌮🎉.. I'm working now to backport this PR to: 3.12.
🐍🍒⛏🤖

miss-islington-app · 2025-03-13T18:41:25Z

Thanks @tomasr8 for the PR, and @serhiy-storchaka for merging it 🌮🎉.. I'm working now to backport this PR to: 3.13.
🐍🍒⛏🤖

miss-islington-app · 2025-03-13T18:41:27Z

Sorry, @tomasr8 and @serhiy-storchaka, I could not cleanly backport this to 3.12 due to a conflict.
Please backport using cherry_picker on command line.

cherry_picker 7ea6e88eb490635518c63c3305c03baf3e151555 3.12

miss-islington-app · 2025-03-13T18:41:32Z

Sorry, @tomasr8 and @serhiy-storchaka, I could not cleanly backport this to 3.13 due to a conflict.
Please backport using cherry_picker on command line.

cherry_picker 7ea6e88eb490635518c63c3305c03baf3e151555 3.13

… with msgfmt (pythonGH-130525) Add also human-readable snapshots for tests. (cherry picked from commit 7ea6e88) Co-authored-by: Tomas R. <[email protected]>

bedevere-app · 2025-03-13T18:51:20Z

GH-131205 is a backport of this pull request to the 3.13 branch.

… with msgfmt (pythonGH-130525) Add also human-readable snapshots for tests. (cherry picked from commit 7ea6e88) Co-authored-by: Tomas R. <[email protected]>

bedevere-app · 2025-03-13T18:52:38Z

GH-131206 is a backport of this pull request to the 3.12 branch.

tomasr8 · 2025-03-13T19:08:04Z

Thanks for dealing with the backports Serhiy, I was going to do it but you beat me to it 😄

…msgfmt (GH-130525) (GH-131206) Add also human-readable snapshots for tests. (cherry picked from commit 7ea6e88) Co-authored-by: Tomas R <[email protected]>

…msgfmt (GH-130525) (GH-131205) Add also human-readable snapshots for tests. (cherry picked from commit 7ea6e88) Co-authored-by: Tomas R <[email protected]>

bedevere-bot · 2025-03-13T19:35:07Z

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot aarch64 Android 3.13 (tier-3) has failed when building commit 78f75d3.

What do you need to do:

Don't panic.
Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
Go to the page of the buildbot that failed (https://buildbot.python.org/#/builders/1590/builds/507) and take a look at the build logs.
Check if the failure is related to this commit (78f75d3) or if it is a false positive.
If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/#/builders/1590/builds/507

Failed tests:

test_os

Failed subtests:

test_fpathconf - test.test_os.TestInvalidFD.test_fpathconf

Summary of the results of the build (if available):

==

Click to see traceback logs

Traceback (most recent call last):
  File "/data/user/0/org.python.testbed/files/python/lib/python3.13/test/test_os.py", line 2402, in test_fpathconf
    self.check(os.pathconf, "PC_NAME_MAX")
    ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data/user/0/org.python.testbed/files/python/lib/python3.13/test/test_os.py", line 2325, in check
    f(os_helper.make_bad_fd(), *args, **kwargs)
    ~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ValueError: unrecognized configuration name

… with msgfmt (pythonGH-130525) (pythonGH-131206) Add also human-readable snapshots for tests. (cherry picked from commit 7ea6e88) Co-authored-by: Tomas R <[email protected]>

…sgfmt (pythonGH-130525) Add also human-readable snapshots for tests.

tomasr8 added 3 commits February 24, 2025 21:11

Add human-readable snapshots

7e4bbb5

Reset context when adding a message

69402a5

Add news entry

741c35c

bedevere-app bot added the awaiting review label Feb 24, 2025

bedevere-app bot mentioned this pull request Feb 24, 2025

Incorrect handling for msgctxt in msgfmt.py #85012

Closed

tomasr8 requested a review from serhiy-storchaka February 24, 2025 21:25

serhiy-storchaka reviewed Feb 25, 2025

View reviewed changes

tomasr8 added 5 commits February 25, 2025 22:59

Use nicer sorting for snapshots

bcb8533

Temporarily revert fix and regenerate snapshots

bb68538

Re-apply fix and regenerate snapshots

8295b2b

Add regression tests

97d0a0f

Merge remote-tracking branch 'upstream/main' into msgfmt-ctxt

d4dcc87

tomasr8 requested a review from serhiy-storchaka February 25, 2025 22:12

serhiy-storchaka approved these changes Mar 13, 2025

View reviewed changes

bedevere-app bot added awaiting merge and removed awaiting review labels Mar 13, 2025

serhiy-storchaka merged commit 7ea6e88 into python:main Mar 13, 2025
43 checks passed

bedevere-app bot removed the awaiting merge label Mar 13, 2025

serhiy-storchaka added needs backport to 3.12 only security fixes needs backport to 3.13 bugs and security fixes labels Mar 13, 2025

miss-islington-app bot assigned serhiy-storchaka Mar 13, 2025

bedevere-app bot removed the needs backport to 3.13 bugs and security fixes label Mar 13, 2025

bedevere-app bot removed the needs backport to 3.12 only security fixes label Mar 13, 2025

tomasr8 deleted the msgfmt-ctxt branch March 13, 2025 19:08

plashchynski pushed a commit to plashchynski/cpython that referenced this pull request Mar 17, 2025

pythongh-85012: Properly reset msgctxt when compiling messages with m…

dc53092

…sgfmt (pythonGH-130525) Add also human-readable snapshots for tests.

seehwan pushed a commit to seehwan/cpython that referenced this pull request Apr 16, 2025

pythongh-85012: Properly reset msgctxt when compiling messages with m…

e05c536

…sgfmt (pythonGH-130525) Add also human-readable snapshots for tests.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

gh-85012: Properly reset `msgctxt` when compiling messages with `msgfmt` #130525

gh-85012: Properly reset `msgctxt` when compiling messages with `msgfmt` #130525

Uh oh!

tomasr8 commented Feb 24, 2025 •

edited by bedevere-app bot

Loading

Uh oh!

tomasr8 commented Feb 24, 2025

Uh oh!

serhiy-storchaka left a comment

Uh oh!

tomasr8 commented Feb 25, 2025

Uh oh!

serhiy-storchaka left a comment

Uh oh!

Uh oh!

miss-islington-app bot commented Mar 13, 2025

Uh oh!

miss-islington-app bot commented Mar 13, 2025

Uh oh!

miss-islington-app bot commented Mar 13, 2025

Uh oh!

miss-islington-app bot commented Mar 13, 2025

Uh oh!

bedevere-app bot commented Mar 13, 2025

Uh oh!

bedevere-app bot commented Mar 13, 2025

Uh oh!

tomasr8 commented Mar 13, 2025

Uh oh!

bedevere-bot commented Mar 13, 2025

Uh oh!

Uh oh!

Uh oh!

gh-85012: Properly reset msgctxt when compiling messages with msgfmt #130525

gh-85012: Properly reset msgctxt when compiling messages with msgfmt #130525

Uh oh!

Conversation

tomasr8 commented Feb 24, 2025 • edited by bedevere-app bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

tomasr8 commented Feb 24, 2025

Uh oh!

serhiy-storchaka left a comment

Choose a reason for hiding this comment

Uh oh!

tomasr8 commented Feb 25, 2025

Uh oh!

serhiy-storchaka left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

miss-islington-app bot commented Mar 13, 2025

Uh oh!

miss-islington-app bot commented Mar 13, 2025

Uh oh!

miss-islington-app bot commented Mar 13, 2025

Uh oh!

miss-islington-app bot commented Mar 13, 2025

Uh oh!

bedevere-app bot commented Mar 13, 2025

Uh oh!

bedevere-app bot commented Mar 13, 2025

Uh oh!

tomasr8 commented Mar 13, 2025

Uh oh!

bedevere-bot commented Mar 13, 2025

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Uh oh!

Uh oh!

gh-85012: Properly reset `msgctxt` when compiling messages with `msgfmt` #130525

gh-85012: Properly reset `msgctxt` when compiling messages with `msgfmt` #130525

tomasr8 commented Feb 24, 2025 •

edited by bedevere-app bot

Loading