Thanks to visit codestin.com
Credit goes to github.com

Skip to content

Complete SQLAlchemy inline pep484 typing #6810

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
5 of 33 tasks
CaselIT opened this issue Jul 24, 2021 · 40 comments
Open
5 of 33 tasks

Complete SQLAlchemy inline pep484 typing #6810

CaselIT opened this issue Jul 24, 2021 · 40 comments
Assignees
Labels
big job a major change that requires a lot of knowledge and work code review in progress code has been provided that's in review as PR and/or gerrit Epic Use this tag to define an Issue as an Epic that contains multiple issues for a targeted help wanted Extra attention is needed PRs (with tests!) welcome a fix or feature which is appropriate to be implemented by volunteers time consuming / tedious an issue that will take a long time to complete, even though it may be "easy" typing pep -484 typing issues. independent of "mypy"
Milestone

Comments

@CaselIT
Copy link
Member

CaselIT commented Jul 24, 2021

Status as of 2023-02

Most of the library is typed, but some modules/packages that are still work in progress.

Modules / packages that still require work

  • sqlalchemy.engine
    • cursor - passes omitting non-typed functions - internal module
    • default - passes omitting non-typed functions - internal module
  • sqlalchemy.ext
    • declarative.extentions - missing most/all types - legacy
    • backed - missing most/all types - legacy
    • compiler - missing most/all types - popular api
    • indexable - missing most/all types
    • instrumentation - missing most/all types
    • orderinglist - missing most/all types
    • serializer - missing most/all types - legacy
  • sqlalchemy.orm
    • bulk_persistence - missing most types - internal module
    • collections - missing most types - internal module
    • context - missing most types - internal module
    • dependency - missing most types - internal module
    • dynamic - missing most types - internal module
    • evaluator - missing most/all types - internal module
    • instrumentation - missing some types - internal module
    • loading - missing most types - internal module
    • persistence - missing most types - internal module
    • strategies - missing most types - internal module
    • strategies - missing most types - internal module
    • unitofwork - missing most types - internal module
    • util - missing most types - internal module
    • writeonly - missing most types - internal module
  • sqlalchemy.sql
    • base - missing most types - internal module
    • compiler - missing most types - internal module
    • crud - missing most types - internal module
    • ddl - missing most types - internal module
    • functions - missing most types - internal module
    • lambdas - missing most types - internal module
    • naming - missing most types - internal module
    • roles - missing most types - internal module
    • traversals - missing most types - internal module
    • util - missing most types - internal module
@CaselIT CaselIT added big job a major change that requires a lot of knowledge and work time consuming / tedious an issue that will take a long time to complete, even though it may be "easy" pep484 Epic Use this tag to define an Issue as an Epic that contains multiple issues for a targeted SQLA mypy plugin mypy plugin issues only. general pep-484 issues should be "typing" labels Jul 24, 2021
@CaselIT CaselIT added this to the 2.0 milestone Jul 24, 2021
@CaselIT
Copy link
Member Author

CaselIT commented Nov 22, 2021

We should try https://github.com/ambv/retype/

@CaselIT CaselIT added the help wanted Extra attention is needed label Dec 19, 2021
@sqla-tester
Copy link
Collaborator

Federico Caselli referenced this issue:

Properly type _generative https://gerrit.sqlalchemy.org/c/sqlalchemy/sqlalchemy/+/3423

sqlalchemy-bot pushed a commit that referenced this issue Dec 31, 2021
Good new is that pylance likes it and copies over the
singature and everything.
Bad news is that mypy does not support this yet python/mypy#8645
Other minor bad news is that non_generative is not typed. I've tried using a protocol
like the one in the comment but the signature is not ported over by pylance, so it's
probably best to just live without it to have the correct signature.

notes from mike:  these three decorators are at the core of getting
the library to be typed, more good news is that pylance will
do all the things we like re: public_factory, see
microsoft/pyright#2758 (comment)
.

For @_generative, we will likely move to using pep 673 once mypy
supports it which may be soon.  but overall having the explicit
"return self" in the methods, while a little inconvenient, makes
the typing more straightforward and locally present in the files
rather than being decided at a distance.   having "return self"
present, or not, both have problems, so maybe we will be able
to change it again if things change as far as decorator support.
As it is, I feel like we are barely squeaking by with our decorators,
the typing is already pretty out there.

Change-Id: Ic77e13fc861def76a5925331df85c0aa48d77807
References: #6810
@sqla-tester
Copy link
Collaborator

Mike Bayer has proposed a fix for this issue in the main branch:

initial reorganize for static typing https://gerrit.sqlalchemy.org/c/sqlalchemy/sqlalchemy/+/3447

@zzzeek
Copy link
Member

zzzeek commented Jan 9, 2022

related issues:

#7535
#7519
#7562
#7555

@zzzeek
Copy link
Member

zzzeek commented Jan 9, 2022

also there's the issue of replacing existing sqlalchemy-stubs packages in type checkers and it's suggested we use a "positive" masking approach, as a "negative" uninstall approach is not supported by type checkers that bundle the stubs: microsoft/pylance-release#840 (comment)

@sqla-tester
Copy link
Collaborator

Mike Bayer has proposed a fix for this issue in the main branch:

WIP for ORM typing https://gerrit.sqlalchemy.org/c/sqlalchemy/sqlalchemy/+/3495

@zzzeek zzzeek added typing pep -484 typing issues. independent of "mypy" and removed help wanted Extra attention is needed labels Jan 9, 2022
@zzzeek zzzeek self-assigned this Jan 9, 2022
@CaselIT CaselIT removed the pep484 label Jan 10, 2022
@zzzeek
Copy link
Member

zzzeek commented Jan 10, 2022

current goals are:

  1. get https://gerrit.sqlalchemy.org/c/sqlalchemy/sqlalchemy/+/3495, based on https://gerrit.sqlalchemy.org/c/sqlalchemy/sqlalchemy/+/3447, working. POC here is that I would like to include updating the mypy plugin and removing the dependency on sqlalchemy2-stubs, w/ mypy tests passing. that will be the proof of concept for 3495, which will also validate 3447.
  2. then merge 3447 and 3495
  3. after those two are in, that's the "hard part" of typing, that is, the column expression / operator / ORM mapping stuff. then we can do a package-by-package or module-by-module approach in separate gerrits
  4. separately, when doing the typing I hit a lot of blocks which are due to the fact that we use metaclasses for "visitable" items as well as some other things. I want to see how far we can use __subclass_init__ to get rid of all metaclasses, at the very least the metaclasses used internally by "visitable" as well as functions.py etc. The DeclarativeMeta metaclass might be a different situation if this is going to break backwards compatibility.

@CaselIT
Copy link
Member Author

CaselIT commented Jan 10, 2022

Step 4. was something that I meant to suggest since we now are guaranteed to always have at least py3.6

does it make sense as a separate change?

@zzzeek
Copy link
Member

zzzeek commented Jan 10, 2022

im going to try it now as its own thing, not including declarative, which probably also needs to be its own thing

@zzzeek
Copy link
Member

zzzeek commented Jan 10, 2022

I think we might want to skip declarative as the DeclarativeMeta class also intercepts class-level setattr and delattr events. in that sense we are still using metaclass-only features.

i have all the internal stuff off of metaclasses with one small exception in the ORM, will keep it at that.

@sqla-tester
Copy link
Collaborator

Mike Bayer referenced this issue:

remove internal use of metaclasses https://gerrit.sqlalchemy.org/c/sqlalchemy/sqlalchemy/+/3499

sqlalchemy-bot pushed a commit that referenced this issue Jan 11, 2022
All but one metaclass used internally can now
be replaced using __init_subclass__().  Within this
patch we remove:

* events._EventMeta
* sql.visitors.TraversibleType
* sql.visitors.InternalTraversibleType
* testing.fixtures.FindFixture
* testing.fixtures.FindFixtureDeclarative
* langhelpers.EnsureKWArgType
* sql.functions._GenericMeta
* sql.type_api.VisitableCheckKWArg (was a mixture of TraversibleType
  and EnsureKWArgType)

The remaining internal class is MetaOptions used by the
sql.Options object which is in turn currently mostly for
ORM internal use, as this type implements class level overrides
for the ``+`` operator.

For declarative, removing DeclarativeMeta in place of
an `__init_subclass__()` class would not be fully feasible as
it would break backwards compatibility with applications that
refer to this class explicitly, but also DeclarativeMeta intercepts
class-level attribute set and delete operations which is a widely
used pattern.   An option for declarative base to use
`__init_subclass__()` should be provided but this is out of
scope for this particular change.

Change-Id: I8aa898c7ab59d887739037d34b1cbab36521ab78
References: #6810
@sqla-tester
Copy link
Collaborator

Mike Bayer referenced this issue:

Initial ORM typing layout https://gerrit.sqlalchemy.org/c/sqlalchemy/sqlalchemy/+/3495

@CaselIT
Copy link
Member Author

CaselIT commented Jan 14, 2022

Re opened since there is still lot to do

sqlalchemy-bot pushed a commit that referenced this issue Jan 14, 2022
introduces:

1. new mapped_column() helper
2. DeclarativeBase helper
3. declared_attr has been re-typed
4. rework of Mapped[] to return InstrumentedAtribute for
   class get, so works without Mapped itself having expression
   methods
5. ORM constructs now generic on [_T]

also includes some early typing work, most of which will
be in later commits:

1. URL and History become typing.NamedTuple
2. come up with type-checking friendly way of type
   checking cy extensions, where type checking will be applied
   to the py versions, just needed to come up with a succinct
   conditional pattern for the imports

References: #6810
References: #7535
References: #7562
Change-Id: Ie5d9a44631626c021d130ca4ce395aba623c71fb
sqlalchemy-bot pushed a commit that referenced this issue Mar 13, 2025
This complements the type annotations of the `ARRAY` class, in preparation of #12384.

This pull request is:

- [ ] A documentation / typographical / small typing error fix
	- Good to go, no issue or tests are needed
- [ ] A short code fix
	- please include the issue number, and create an issue if none exists, which
	  must include a complete example of the issue.  one line code fixes without an
	  issue and demonstration will not be accepted.
	- Please include: `Fixes: #<issue number>` in the commit message
	- please include tests.   one line code fixes without tests will not be accepted.
- [x] A new feature implementation
	- please include the issue number, and create an issue if none exists, which must
	  include a complete example of how the feature would look.
	- Please include: `Fixes: #<issue number>` in the commit message
	- please include tests.

Related to #6810

Closes: #12386
Pull-request: #12386
Pull-request-sha: c9513ce

Change-Id: If9df4708c8e597eedc79ee3990792fa6c72f1afe
(cherry picked from commit 0bf7e02)
sqlalchemy-bot pushed a commit that referenced this issue Mar 13, 2025
### Description

This complements the type annotations of the `ARRAY` class, in preparation of #12384.

### Checklist

This pull request is:

- [ ] A documentation / typographical / small typing error fix
	- Good to go, no issue or tests are needed
- [ ] A short code fix
	- please include the issue number, and create an issue if none exists, which
	  must include a complete example of the issue.  one line code fixes without an
	  issue and demonstration will not be accepted.
	- Please include: `Fixes: #<issue number>` in the commit message
	- please include tests.   one line code fixes without tests will not be accepted.
- [x] A new feature implementation
	- please include the issue number, and create an issue if none exists, which must
	  include a complete example of how the feature would look.
	- Please include: `Fixes: #<issue number>` in the commit message
	- please include tests.

Related to #6810

Closes: #12386
Pull-request: #12386
Pull-request-sha: c9513ce

Change-Id: If9df4708c8e597eedc79ee3990792fa6c72f1afe
sqlalchemy-bot pushed a commit that referenced this issue Mar 17, 2025
Improved static typing for `postgresql.array()` by making the type parameter (the type of array's elements) inferred from the `clauses` and `type_` arguments while also ensuring they are consistent.

Also completed type annotations of `postgresql.ARRAY` following commit 0bf7e02 and added type annotations for functions `postgresql.Any()` and `postgresql.All()`.

Finally, fixed shadowing `typing.Any` by the `Any()` function through aliasing as `typing_Any`.

Related to #6810

Closes: #12384
Pull-request: #12384
Pull-request-sha: 78eea29

Change-Id: I5d35d15ec8ba4d58eeb9bf00abb710e2e585731f
sqlalchemy-bot pushed a commit that referenced this issue Mar 17, 2025
Improved static typing for `postgresql.array()` by making the type parameter (the type of array's elements) inferred from the `clauses` and `type_` arguments while also ensuring they are consistent.

Also completed type annotations of `postgresql.ARRAY` following commit 0bf7e02 and added type annotations for functions `postgresql.Any()` and `postgresql.All()`.

Finally, fixed shadowing `typing.Any` by the `Any()` function through aliasing as `typing_Any`.

Related to #6810

Closes: #12384
Pull-request: #12384
Pull-request-sha: 78eea29

Change-Id: I5d35d15ec8ba4d58eeb9bf00abb710e2e585731f
(cherry picked from commit 75c8e11)
dlax added a commit to dlax/sqlalchemy that referenced this issue Mar 18, 2025
Now `Column(type_=ARRAY(Integer)` is inferred as `Column[Sequence[int]]`
instead as `Column[Sequence[Any]]` previously. This only works with the `type_`
argument to Column, but that's not new.

This follows from a suggestion at
sqlalchemy#12386 (comment).

Related to sqlalchemy#6810.
dlax added a commit to dlax/sqlalchemy that referenced this issue Mar 18, 2025
Now `Column(type_=ARRAY(Integer)` is inferred as `Column[Sequence[int]]`
instead as `Column[Sequence[Any]]` previously. This only works with the `type_`
argument to Column, but that's not new.

This follows from a suggestion at
sqlalchemy#12386 (comment).

Related to sqlalchemy#6810.
sqlalchemy-bot pushed a commit that referenced this issue Mar 19, 2025
Now `Column(type_=ARRAY(Integer)` is inferred as `Column[Sequence[int]]` instead as `Column[Sequence[Any]]` previously. This only works with the `type_` argument to Column, but that's not new.

This follows from a suggestion at
#12386 (comment).

Related to #6810.

Closes: #12443
Pull-request: #12443
Pull-request-sha: 2fff4e8

Change-Id: I87b828fd82d10fbf157141db3c31f0ec8149caad
(cherry picked from commit 500adfa)
sqlalchemy-bot pushed a commit that referenced this issue Mar 19, 2025
Now `Column(type_=ARRAY(Integer)` is inferred as `Column[Sequence[int]]` instead as `Column[Sequence[Any]]` previously. This only works with the `type_` argument to Column, but that's not new.

This follows from a suggestion at
#12386 (comment).

Related to #6810.

Closes: #12443
Pull-request: #12443
Pull-request-sha: 2fff4e8

Change-Id: I87b828fd82d10fbf157141db3c31f0ec8149caad
dlax added a commit to dlax/sqlalchemy that referenced this issue Mar 25, 2025
dlax added a commit to dlax/sqlalchemy that referenced this issue Mar 25, 2025
dlax added a commit to dlax/sqlalchemy that referenced this issue Mar 25, 2025
Overloading of `__init__()` is needed, probably for the same reason as
it is in ReturnTypeFromArgs.

Related to sqlalchemy#6810.
sqlalchemy-bot pushed a commit that referenced this issue Mar 27, 2025
The return type of `array_agg()` is declared as a `Sequence[T]` where `T` is bound to the type of input argument.

This is implemented by making `array_agg()` inheriting from `ReturnTypeFromArgs` which provides appropriate overloads of `__init__()` to support this.

This usage of ReturnTypeFromArgs is a bit different from previous ones as the return type of the function is not exactly the same as that of its arguments, but a "collection" (a generic, namely a Sequence here) of the argument types.  Accordingly, we adjust the code of `tools/generate_sql_functions.py` to retrieve the "collection" type from 'fn_class' annotation and generate expected return type.

Also add a couple of hand-written typing tests for PostgreSQL.

Related to #6810

Closes: #12461
Pull-request: #12461
Pull-request-sha: ba27cbb

Change-Id: I3fd538cc7092a0492c26970f0b825bf70ddb66cd
sqlalchemy-bot pushed a commit that referenced this issue Mar 27, 2025
The return type of `array_agg()` is declared as a `Sequence[T]` where `T` is bound to the type of input argument.

This is implemented by making `array_agg()` inheriting from `ReturnTypeFromArgs` which provides appropriate overloads of `__init__()` to support this.

This usage of ReturnTypeFromArgs is a bit different from previous ones as the return type of the function is not exactly the same as that of its arguments, but a "collection" (a generic, namely a Sequence here) of the argument types.  Accordingly, we adjust the code of `tools/generate_sql_functions.py` to retrieve the "collection" type from 'fn_class' annotation and generate expected return type.

Also add a couple of hand-written typing tests for PostgreSQL.

Related to #6810

Closes: #12461
Pull-request: #12461
Pull-request-sha: ba27cbb

Change-Id: I3fd538cc7092a0492c26970f0b825bf70ddb66cd
(cherry picked from commit 543acbd)
dlax added a commit to dlax/sqlalchemy that referenced this issue Mar 28, 2025
Overloading of `__init__()` is needed, probably for the same reason as
it is in ReturnTypeFromArgs.

Related to sqlalchemy#6810.
sqlalchemy-bot pushed a commit that referenced this issue Apr 1, 2025
Related to #6810.

Closes: #12462
Pull-request: #12462
Pull-request-sha: 5a131cc

Change-Id: Ie4494d61f815edefef6a896499db4292fd94a22a
sqlalchemy-bot pushed a commit that referenced this issue Apr 1, 2025
Related to #6810.

Closes: #12462
Pull-request: #12462
Pull-request-sha: 5a131cc

Change-Id: Ie4494d61f815edefef6a896499db4292fd94a22a
(cherry picked from commit 864f79d)
dlax added a commit to dlax/sqlalchemy that referenced this issue Apr 7, 2025
Overloading of `__init__()` is needed, probably for the same reason as
it is in ReturnTypeFromArgs.

Related to sqlalchemy#6810.
dlax added a commit to dlax/sqlalchemy that referenced this issue Apr 9, 2025
Overloading of `__init__()` is needed, probably for the same reason as
it is in ReturnTypeFromArgs.

Related to sqlalchemy#6810.
sqlalchemy-bot pushed a commit that referenced this issue Apr 10, 2025
Overloading of `__init__()` is needed, probably for the same reason as it is in `ReturnTypeFromArgs`.

Related to #6810.

Closes: #12463
Pull-request: #12463
Pull-request-sha: 701d979

Change-Id: I7e1bb4d2c48dfb3461725c7079aaa72c66f1dc03
sqlalchemy-bot pushed a commit that referenced this issue Apr 10, 2025
Overloading of `__init__()` is needed, probably for the same reason as it is in `ReturnTypeFromArgs`.

Related to #6810.

Closes: #12463
Pull-request: #12463
Pull-request-sha: 701d979

Change-Id: I7e1bb4d2c48dfb3461725c7079aaa72c66f1dc03
(cherry picked from commit 09c1d3c)
Polandia94 added a commit to Polandia94/sqlalchemy that referenced this issue May 10, 2025
Tipyng of mysql dialect.

possible breaking changes:

- on `visit_sequence`, `visit_true` and `visit_false` I changed the param names to be the same tht in the super class.
- Removed  `tostring` on `array`. `tostring` is deprecated since Python 3.2 and removed in Python3.9 so i guess we can remove that option altogheter.
- create_connect_args on aiomysql, added an unused param _translate_args, only to have same signature as super class.
-  delete _check_unicode_returns method. on MySQLDialect_mysqldb. I think is not used after https://gerrit.sqlalchemy.org/c/sqlalchemy/sqlalchemy/+/1946, and is calling to a super that I think doesnt exist anymore.
- delete _set_isolation_level, is unused, and has a super that doesnt exists anymore:
sqlalchemy@af1b916
- Separate _bitmap in enumerated to _bitmap and _inversed_bitmap to have type safety
- Add inheritance from log.Identified to MySQLTableDefinitionParser, because has the decorathor @log.class_logger

Related to issue :
sqlalchemy#6810

Closes: sqlalchemy#12164
Pull-request: sqlalchemy#12164
Pull-request-sha: b877cca

Change-Id: I37bd98049ff1a64d58e9490b0e5e2ea764dd1f73
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
big job a major change that requires a lot of knowledge and work code review in progress code has been provided that's in review as PR and/or gerrit Epic Use this tag to define an Issue as an Epic that contains multiple issues for a targeted help wanted Extra attention is needed PRs (with tests!) welcome a fix or feature which is appropriate to be implemented by volunteers time consuming / tedious an issue that will take a long time to complete, even though it may be "easy" typing pep -484 typing issues. independent of "mypy"
Projects
None yet
Development

No branches or pull requests

10 participants