Thanks to visit codestin.com
Credit goes to github.com

Skip to content

argparse.HelpFormatter subclasses may be broken by 3.14 changes #133653

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
iritkatriel opened this issue May 8, 2025 · 13 comments
Closed

argparse.HelpFormatter subclasses may be broken by 3.14 changes #133653

iritkatriel opened this issue May 8, 2025 · 13 comments
Assignees
@serhiy-storchaka
Labels
3.14 bugs and security fixes stdlib Python modules in the Lib dir type-bug An unexpected behavior, bug, or error

Comments

@iritkatriel
Copy link
Member

iritkatriel commented May 8, 2025 •
edited by bedevere-app bot
Loading

I came across this trying to run mypy benchmarks. The version I had contained

class AugmentedHelpFormatter(argparse.RawDescriptionHelpFormatter):
    def __init__(self, prog: str) -> None:
        super().__init__(prog=prog, max_help_position=30)

Which was rewritten in a newer version to:

class AugmentedHelpFormatter(argparse.RawDescriptionHelpFormatter):
    def __init__(self, prog: str, **kwargs: Any) -> None:
        super().__init__(prog=prog, max_help_position=30, **kwargs)

The changes in #124456 and #132323 can be made without breaking user code, if they accept the new args as **kwargs, extract their values from kwargs if they are there, and continue working as before if not. Is there a good reason not to do this?

Do these changes currently comply with Python's backwards compatibility policy?

Linked PRs
@iritkatriel iritkatriel added type-bug An unexpected behavior, bug, or error stdlib Python modules in the Lib dir 3.14 bugs and security fixes labels May 8, 2025
@iritkatriel
Copy link
Member Author

CC @hugovk @savannahostrowski .

@hugovk
Copy link
Member

hugovk commented May 8, 2025

The changes in #124456 and #132323 can be made without breaking user code, if they accept the new args as **kwargs, extract their values from kwargs if they are there, and continue working as before if not. Is there a good reason not to do this?

Let's do this.

Would you like to open a PR? I'm also happy to do it.

@iritkatriel
Copy link
Member Author

Please go ahead.

@iritkatriel
Copy link
Member Author

iritkatriel commented May 8, 2025 •
edited by hugovk
Loading

Something like

        prefix_chars = kwargs.get('prefix_chars', '-')
        color = kwargs.get('color', False)

This was referenced May 8, 2025
Closed
Merged
@The-Compiler
Copy link
Contributor

Do these changes currently comply with Python's backwards compatibility policy?

FWIW, argparse.HelpFormatter itself is undocumented and the docstrings say:

cpython/Lib/argparse.py

Lines 62 to 64 in 0ec8fc8

(Also note that HelpFormatter and RawDescriptionHelpFormatter are only
considered public as object names -- the API of the formatter objects is
still considered an implementation detail.)

cpython/Lib/argparse.py

Lines 157 to 162 in 0ec8fc8

class HelpFormatter(object):
"""Formatter for generating usage messages and argument help strings.
Only the name of this class is considered a public API. All the methods
provided by the class are considered an implementation detail.
"""

But despite that, subclassing argparse.HelpFormatter seems to be a common occurrence, and many implementations seem to not use **kwargs.

@bedevere-app bedevere-app bot mentioned this issue May 8, 2025
Closed
@hugovk
Copy link
Member

hugovk commented May 8, 2025

How does PR #133668 look?

@serhiy-storchaka
Copy link
Member

I think this is a documentation issue. The only way to fix it reliably is to properly document the HelpFormatter class and how it cab be extended.

Currently the HelpFormatter class is not documented. It's subclassing is not officially supported. You need to look in the source code to find about max_help_position.

@iritkatriel
Copy link
Member Author

The only way to fix it reliably is to properly document the HelpFormatter class and how it cab be extended.

As discussed on the PR - doc change is welcome but it is not the only way to fix this.

I think we should add a setter method for the new fields so that the signature of __init__ doesn't need to change.

We saw two libraries break already, there will be more post release. There is no point repeatedly litigating this with users over the next 3-5 years if we can avoid that with a simple change now.

@serhiy-storchaka
Copy link
Member

There is something similar issue #133745. It is more serious, because the documented contract was changed. For HelpFormatter there is no documented contract. We can apply solution similar to the code that was used before #128308 -- pass only prog to the factory, and set other parameters as private HelpFormatter attributes. This approach has many drawbacks. It still can break custom formatters, and for some users it will only defer breakage fr future versions. In long run, it is better to break the user code now. But we need the documentation to make sure that users will write correct code which will not break in future.

@iritkatriel
Copy link
Member Author

Ultimately it's up to the 3.14 release manager.

@serhiy-storchaka serhiy-storchaka self-assigned this May 10, 2025
@serhiy-storchaka
Copy link
Member

Well, I have a plan to make the transition smoother, and I'm working on it.

serhiy-storchaka added a commit to serhiy-storchaka/cpython that referenced this issue May 10, 2025
@serhiy-storchaka
pythongh-133653: Fix argparse.ArgumentParser with the formatter_class…
ef5d142 
… argument

* Fix TypeError when formatter_class is a custom subclass of
  HelpFormatter.
* Fix TypeError when formatter_class is not a subclass of
  HelpFormatter and non-standard prefix_char is used.
* Fix support of colorizing when formatter_class is not a subclass of
  HelpFormatter.
@bedevere-app bedevere-app bot mentioned this issue May 10, 2025
Merged
@serhiy-storchaka
Copy link
Member

There were other issues. Colorizing did not work when formatter_class is a function. And all failed when prefix_chars is not standard and formatter_class is a function, even without colorizing.

gentoo-bot pushed a commit to gentoo/gentoo that referenced this issue May 11, 2025
@thesamesam
dev-python/pygments: enable py3.14
92d1c6b 
Skip the help tests for now. Not ideal but it's not too bad and it's
not pygments-specific anyway.

Bug: python/cpython#133653
Bug: python/cpython#133813
Signed-off-by: Sam James <[email protected]>
tanuki-no pushed a commit to tanuki-no/gentoo that referenced this issue May 12, 2025
@thesamesam @tanuki-no
dev-python/pygments: enable py3.14
b73bf53 
Skip the help tests for now. Not ideal but it's not too bad and it's
not pygments-specific anyway.

Bug: python/cpython#133653
Bug: python/cpython#133813
Signed-off-by: Sam James <[email protected]>
serhiy-storchaka added a commit that referenced this issue May 12, 2025
@serhiy-storchaka
gh-133653: Fix argparse.ArgumentParser with the formatter_class argum…
734e15b 
…ent (GH-133813)

* Fix TypeError when formatter_class is a custom subclass of
  HelpFormatter.
* Fix TypeError when formatter_class is not a subclass of
  HelpFormatter and non-standard prefix_char is used.
* Fix support of colorizing when formatter_class is not a subclass of
  HelpFormatter.
* Remove the prefix_chars parameter of HelpFormatter.
miss-islington pushed a commit to miss-islington/cpython that referenced this issue May 12, 2025
@serhiy-storchaka @miss-islington
pythongh-133653: Fix argparse.ArgumentParser with the formatter_class…
74c8bce 
… argument (pythonGH-133813)

* Fix TypeError when formatter_class is a custom subclass of
  HelpFormatter.
* Fix TypeError when formatter_class is not a subclass of
  HelpFormatter and non-standard prefix_char is used.
* Fix support of colorizing when formatter_class is not a subclass of
  HelpFormatter.
* Remove the prefix_chars parameter of HelpFormatter.
(cherry picked from commit 734e15b)

Co-authored-by: Serhiy Storchaka <[email protected]>
@bedevere-app bedevere-app bot mentioned this issue May 12, 2025
Merged
@serhiy-storchaka
Copy link
Member

I think we should add a setter method for the new fields so that the signature of __init__ doesn't need to change.

I implemented this solution. Also, for now, there is no need to pass prefix_chars.

serhiy-storchaka added a commit that referenced this issue May 12, 2025
@miss-islington @serhiy-storchaka
[3.14] gh-133653: Fix argparse.ArgumentParser with the formatter_clas…
c15980b 
…s argument (GH-133813) (GH-133941)

* Fix TypeError when formatter_class is a custom subclass of
  HelpFormatter.
* Fix TypeError when formatter_class is not a subclass of
  HelpFormatter and non-standard prefix_char is used.
* Fix support of colorizing when formatter_class is not a subclass of
  HelpFormatter.
* Remove the prefix_chars parameter of HelpFormatter.
(cherry picked from commit 734e15b)

Co-authored-by: Serhiy Storchaka <[email protected]>
@henryiii henryiii mentioned this issue May 13, 2025
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@serhiy-storchaka serhiy-storchaka

Labels
3.14 bugs and security fixes stdlib Python modules in the Lib dir type-bug An unexpected behavior, bug, or error
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@The-Compiler @iritkatriel @hugovk @serhiy-storchaka