Thanks to visit codestin.com
Credit goes to github.com

Skip to content

feat(api): Make RESTManager generic on RESTObject class #3083

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Feb 5, 2025

Conversation

igorp-collabora
Copy link
Contributor

@igorp-collabora igorp-collabora commented Jan 14, 2025

Currently mixins like ListMixin are type hinted to return base RESTObject instead of a specific class like MergeRequest.

The GetMixin and GetWithoutIdMixin solve this problem by defining a new get method for every defined class. However, this creates a lot of duplicated code.

Make RESTManager use typing.Generic as its base class and use and assign the declared TypeVar to the _obj_cls attribute as a type of the passed class.

Make both _obj_cls and _path attributes an abstract properties so that type checkers can check that those attributes were properly defined in subclasses. Mypy will only check then the class is instantiated which makes non-final subclasses possible. Unfortunately pylint will check the declarations not instantiations so add # pylint: disable=abstract-method comments to all non-final subclasses like ListMixin.

Make _path attribute always be str instead of sometimes None. This eliminates unnecessary type checks.

Change all mixins like ListMixin or GetMixin to a subclass. This makes the attribute declarations much cleaner as for example _list_filters is now the only attribute defined by ListMixin.

Change SidekiqManager to not inherit from RESTManager and only copy its __init__ method. This is because SidekiqManager never was a real manager and does not define _path or _obj_cls.

Delete tests/unit/meta/test_ensure_type_hints.py file as the get method is no required to be defined for every class.

Closes #2228
Closes #3080

@igorp-collabora
Copy link
Contributor Author

This is a lot more verbose than #3080 but I think it is a better solution with less hacks and # type: ignore[override]

I did not make a RESTObjectList a generic or add overloads to list methods here to keep this PR more focused on RESTManager but I want to do a follow-up PR in the future.

@igorp-collabora
Copy link
Contributor Author

There was a PR #2067 in the past by @vectro but it looks like there the tox test were not passing? I ran tox on my machine and it seems to be passing everything.

Copy link

codecov bot commented Jan 14, 2025

Codecov Report

Attention: Patch coverage is 99.17355% with 3 lines in your changes missing coverage. Please review.

Project coverage is 97.25%. Comparing base (46dfc50) to head (40eb4d0).
Report is 1 commits behind head on main.

Files with missing lines Patch % Lines
gitlab/v4/objects/clusters.py 71.42% 2 Missing ⚠️
gitlab/v4/objects/pipelines.py 91.66% 1 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #3083      +/-   ##
==========================================
- Coverage   97.28%   97.25%   -0.04%     
==========================================
  Files          97       97              
  Lines        6006     5932      -74     
==========================================
- Hits         5843     5769      -74     
  Misses        163      163              
Flag Coverage Δ
api_func_v4 83.24% <97.52%> (-0.34%) ⬇️
cli_func_v4 84.37% <96.96%> (-0.20%) ⬇️
unit 90.00% <97.24%> (-0.13%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines Coverage Δ
gitlab/base.py 100.00% <100.00%> (ø)
gitlab/client.py 98.73% <100.00%> (ø)
gitlab/mixins.py 91.29% <100.00%> (-1.57%) ⬇️
gitlab/v4/cli.py 91.83% <100.00%> (+0.02%) ⬆️
gitlab/v4/objects/access_requests.py 100.00% <100.00%> (ø)
gitlab/v4/objects/appearance.py 100.00% <100.00%> (ø)
gitlab/v4/objects/applications.py 100.00% <100.00%> (ø)
gitlab/v4/objects/artifacts.py 100.00% <100.00%> (ø)
gitlab/v4/objects/audit_events.py 100.00% <100.00%> (ø)
gitlab/v4/objects/award_emojis.py 100.00% <100.00%> (ø)
... and 72 more

@igorp-collabora
Copy link
Contributor Author

I also have a 3rd implementation there I made RESTManager a Protocol as described in this comment

#2228 (comment)

It allows mixins to be "mixins" instead of subclasses using def method(sefl: RESTManager).

However, it quickly ran in to several issues:

  1. Not possible to mixin attributes as self would only have attributes from RESTManager. You can move all mixins attributes to RESTManager or define new protocols that subclass the RESTManager but it pretty ugly. It seems like the ListMixins are always mixed with RESTManager so using subclasses instead of mixins is a cleaner solution.
  2. Not possible to issubclass(some_class, RESTManager). There is something about Protocols which prevent subclass checks.

@igorp-collabora
Copy link
Contributor Author

There is also an idea about SidekiqManager. Set its _obj_cls to RESTObject and _path to "". This way it is still a RESTManager subclass and still defines required attributes.

@nejch
Copy link
Member

nejch commented Jan 27, 2025

Thanks a lot for all the work here @igorp-collabora. Technically your 3rd solution would be really clean IMO and would be nice to have proper mixins and IIRC that is also how I imagined protocols last I played around with this. But, I guess they are not real mixins from mypy's perspective anyway whenever we rely on specific instance attributes 😁

The PRs are actually quite reasonable to review but the main chunk is the removal of the manual get overrides, which makes everything a huge change. So I'd be open to reviewing that separately if that works for you (1 PR to add native typing, which should already solve it for other methods, and another for removing the custom overrides and test). Let me know if that works. It would be slightly easier to focus on the details then 🙇

I like the idea of defining a path on the SidekiqManager for consistent typing. We can actually make it /sidekiq and then use that for the methods using {self.path}/endpoint, which we already do in many cases:

path = f"{self.path}/merge"

@igorp-collabora
Copy link
Contributor Author

1 PR to add native typing, which should already solve it for other methods, and another for removing the custom overrides and test

I can do that but I want to check if keeping the def get around would cause the type checking failure. This would make the commit non-atomic and I am not sure if you are ok with it. Alternatively the get removal commit can be first and then the generic commit after.

I like the idea of defining a path on the SidekiqManager for consistent typing. We can actually make it /sidekiq and then use that for the methods using {self.path}/endpoint, which we already do in many cases:

I will add this.

@nejch
Copy link
Member

nejch commented Jan 28, 2025

@igorp-collabora separate commits but the same PR would also work for me if that's ok! 🙇

@nejch
Copy link
Member

nejch commented Jan 28, 2025

Things should be a bit calmer now after the backlog merge and release(s) so hopefully less conflict/rebasing here 😀

@igorp-collabora
Copy link
Contributor Author

I rebased the PR and updated all the new RESTManager classes added.

I will split the def get removals to the separate commit tomorrow. I think I will place the get removal commit first followed by adding a generic type hints. The reverse order requires modifying all the get requests anyway because the redundant cast would need to be removed.

Copy link
Member

@nejch nejch left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I managed to get through the diff and left some ramblings (some are non-blocking, just general notes).

In general I like that this will give users accurate typing for most resources.

It might feel a bit odd that our mixins are no longer mixins at all (not just in the typing context), but seems to be more maintainable than the protocol approach for now and the user value is more important 👍 We can revisit the protocol approach at some point if we want the code closer to the user (gitlab.v4.objects.*) to be less type-verbose but IMO this is OK.

Not sure if @JohnVillalovos would also like to give this another look as well 🙇

@igorp-collabora igorp-collabora force-pushed the rest-manager-generic branch 2 times, most recently from 4e33522 to b9a1325 Compare January 29, 2025 12:35
@igorp-collabora
Copy link
Contributor Author

igorp-collabora commented Jan 29, 2025

@nejch I think this PR is in good enough form. The only possible change I might add is splitting the RESTManager in to 2 classes. (the parent one with just _path and one that adds _obj_cls)

I think there is also room to add a new unit test that verifies that _obj_cls and _path are properly set for subclasses. Since they are no longer defined using @abstractmethod @property the type checkers will not verify that they properly defined.

If you think this PR is good enough. I can start working on splitting the def get removal to a separate commit.

Copy link
Member

@nejch nejch left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@nejch I think this PR is in good enough form. The only possible change I might add is splitting the RESTManager in to 2 classes. (the parent one with just _path and one that adds _obj_cls)

I think there is also room to add a new unit test that verifies that _obj_cls and _path are properly set for subclasses. Since they are no longer defined using @abstractmethod @property the type checkers will not verify that they properly defined.

@igorp-collabora agree, I think this is almost good to go! Now that I can focus just on a few files I just found a few more nits that I forgot earlier. I think the base manager could work well if it helps with the code using it. Let me know if that's ok.

I'm happy to see typing and linting here helps get rid of our custom checks so if possible I'd prefer to avoid more of them - maybe we can have a follow-up for that though.

@igorp-collabora igorp-collabora force-pushed the rest-manager-generic branch 2 times, most recently from 1de5b6b to eedbf9d Compare January 29, 2025 14:09
@JohnVillalovos
Copy link
Member

@nejch I'm not sure if I will have time to review this in the near future. I trust your reviews though 👍🤓

@JohnVillalovos
Copy link
Member

@nejch After this gets merged, I will propose a PR to update the black config so that it will do the equivalent of --skip-magic-trailing-comma.

Basically add to pyproject.toml:

[tool.black]
skip-magic-trailing-comma = true

@igorp-collabora
Copy link
Contributor Author

I made a separate PR to remove the get methods: #3110

Once it is merged I will rebase this PR. It should shrink by quite a bit after that.

@igorp-collabora
Copy link
Contributor Author

Rebase done.

The line diff is now +748 -536. Before it was +765 -1485.

Copy link
Member

@nejch nejch left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This LGTM, I'll just wait with merging until later today so I can do a checkout on some of our projects that make heavy use of the library. Also it'll be daytime in Oregon by then in case @JohnVillalovos has any objections 😁
This should have no impact at runtime but if mypy starts complaining a lot, it might be good if we add a breaking change so people can migrate more easily when they get their dependency/renovate bot bumps.

Thanks a lot for your patience @igorp-collabora and for all the effort put in here!

@igorp-collabora
Copy link
Contributor Author

igorp-collabora commented Jan 30, 2025

I think there shouldn't be any difference with the Get methods because of redefined get methods had the same effective type hints, however, other mixins might make certain code no longer pass type checking. Biggest changes that might impact type checking:

  1. RESTManager.path is now always str. This means checking for None is no longer required. I don't think mypy checks for unreachable code but there might be some redundant casts.
  2. ListMixin now returns a list of concrete subclasses as a part of union. I also want to make the RESTObjectList generic in the future and add overloads just like download methods.
  3. CreateMixin returns concrete subclass.
  4. SetMixin returns concrete subclass.

@JohnVillalovos
Copy link
Member

Also it'll be daytime in Oregon by then in case @JohnVillalovos has any objections 😁

No objections from me. Sorry I haven't had the time to review this yet. I trust your decision @nejch

Currently mixins like ListMixin are type hinted to return base
RESTObject instead of a specific class like `MergeRequest`.

The GetMixin and GetWithoutIdMixin solve this problem by defining
a new `get` method for every defined class. However, this creates
a lot of duplicated code.

Make RESTManager use `typing.Generic` as its base class and use
and assign the declared TypeVar to the `_obj_cls` attribute as
a type of the passed class.

Make both `_obj_cls` and `_path` attributes an abstract properties
so that type checkers can check that those attributes were properly
defined in subclasses. Mypy will only check then the class is
instantiated which makes non-final subclasses possible.
Unfortunately pylint will check the declarations not instantiations
so add `# pylint: disable=abstract-method` comments to all non-final
subclasses like ListMixin.

Make `_path` attribute always be `str` instead of sometimes `None`.
This eliminates unnecessary type checks.

Change all mixins like ListMixin or GetMixin to a subclass. This makes
the attribute declarations much cleaner as for example `_list_filters`
is now the only attribute defined by ListMixin.

Change SidekiqManager to not inherit from RESTManager and only
copy its `__init__` method. This is because SidekiqManager never
was a real manager and does not define `_path` or `_obj_cls`.

Delete `tests/unit/meta/test_ensure_type_hints.py` file as the `get`
method is no required to be defined for every class.

Signed-off-by: Igor Ponomarev <[email protected]>
@igorp-collabora
Copy link
Contributor Author

Once this gets merged I plan on adding more typing changes:

  • Make RESTObjectList generic to type hint iterated object class
  • Add overloads to the list methods so that only RESTObjectList or list gets returned based on the arguments. This can also make skipping get_all have a deprecation warning then type checked.

@nejch
Copy link
Member

nejch commented Feb 5, 2025

Once this gets merged I plan on adding more typing changes:

* Make `RESTObjectList` generic to type hint iterated object class

* Add overloads to the list methods so that only `RESTObjectList` or `list` gets returned based on the arguments. This can also make skipping `get_all` have a deprecation warning then type checked.

Thanks @igorp-collabora. I checked out python-gitlab from your branch in a few of our projects and generally it all looks great but indeed the list methods are now failing checks where we previously had workarounds for the union types. I think just for clarity I'd make that a breaking change, but we can do this in a follow-up since the list() return types will change with the incoming PRs again.

I'll go ahead and merge this for now!

@nejch nejch merged commit 91c4f18 into python-gitlab:main Feb 5, 2025
18 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Some mixins are not properly type-hinted
3 participants