test: add prerelease session to test with latest dependencies, separate compliance tests #401

tswast · 2022-01-18T23:07:59Z

Thank you for opening a Pull Request! Before submitting your PR, there are a few things you can do to make sure it goes smoothly:

Make sure to open an issue as a bug/issue before writing your code! That way we can discuss the change, evaluate designs, and agree on the general idea
Ensure the tests and linter pass
Code coverage does not decrease (if any source code was changed)
Appropriate docs were updated (if necessary)

Towards googleapis/python-bigquery#1021 "Test connectors, such as pandas-gbq and sqlalchemy-bigquery with pre-release, make updates if needed."

Towards #413

Internal CL 422873148 to enable these test sessions in Kokoro.

tswast · 2022-01-18T23:13:12Z

Prerelease tests pass locally:

(dev-3.9) ➜  python-bigquery-sqlalchemy git:(tswast-prerelease) nox -s prerelease   
nox > Running session prerelease
nox > Creating virtual environment (virtualenv) using python3.8 in .nox/prerelease
nox > python -m pip install --prefer-binary --pre --upgrade google-api-core google-cloud-bigquery google-cloud-bigquery-storage google-cloud-core google-resumable-media grpcio
nox > python -m pip install freezegun google-cloud-testutils mock psutil pytest pytest-cov pytz
nox > python -m pip install sqlalchemy google-auth google-cloud-bigquery google-api-core
nox > python -m pip install --no-deps -e .[all]
nox > python -m pip freeze
attrs==21.4.0
cachetools==4.2.4
certifi==2021.10.8
charset-normalizer==2.0.10
click==8.0.3
coverage==6.2
freezegun==1.1.0
google-api-core==2.4.0
google-auth==2.3.3
google-cloud-bigquery==3.0.0b1
google-cloud-bigquery-storage==2.11.0
google-cloud-core==2.2.2
google-cloud-testutils==1.3.1
google-crc32c==1.3.0
google-resumable-media==2.1.0
googleapis-common-protos==1.54.0
greenlet==1.1.2
grpcio==1.43.0
grpcio-status==1.43.0
idna==3.3
iniconfig==1.1.1
libcst==0.4.0
mock==4.0.3
mypy-extensions==0.4.3
numpy==1.22.1
packaging==21.3
pluggy==1.0.0
proto-plus==1.19.8
protobuf==3.19.3
psutil==5.9.0
py==1.11.0
pyarrow==6.0.1
pyasn1==0.4.8
pyasn1-modules==0.2.8
pyparsing==3.0.6
pytest==6.2.5
pytest-cov==3.0.0
python-dateutil==2.8.2
pytz==2021.3
PyYAML==6.0
requests==2.27.1
rsa==4.8
six==1.16.0
SQLAlchemy==1.4.29
-e git+ssh://[email protected]/tswast/python-bigquery-sqlalchemy.git@129f2747ae820d75a4dd5eafb942a5bdbd32e3d5#egg=sqlalchemy_bigquery
toml==0.10.2
tomli==2.0.0
typing-inspect==0.7.1
typing_extensions==4.0.1
urllib3==1.26.8
nox > py.test --quiet --junitxml=prerelease_unit_3.8_sponge_log.xml tests/unit
======================================================================== test session starts =========================================================================
platform darwin -- Python 3.8.10, pytest-6.2.5, py-1.11.0, pluggy-1.0.0
rootdir: /Users/swast/src/github.com/googleapis/python-bigquery-sqlalchemy, configfile: setup.cfg
plugins: cov-3.0.0
collected 226 items / 1 skipped / 225 selected                                                                                                                       

tests/unit/test__struct.py ....................                                                                                                                [  8%]
tests/unit/test_catalog_functions.py ..........................................                                                                                [ 27%]
tests/unit/test_comments.py .....                                                                                                                              [ 29%]
tests/unit/test_compiler.py .....                                                                                                                              [ 31%]
tests/unit/test_compliance.py ..........                                                                                                                       [ 36%]
tests/unit/test_dialect_types.py .                                                                                                                             [ 36%]
tests/unit/test_engine.py ......                                                                                                                               [ 39%]
tests/unit/test_helpers.py .........                                                                                                                           [ 43%]
tests/unit/test_like_reescape.py .                                                                                                                             [ 43%]
tests/unit/test_parse_url.py ......................................                                                                                            [ 60%]
tests/unit/test_select.py ..................................................s....s...........                                                                  [ 90%]
tests/unit/test_sqlalchemy_bigquery.py .....................                                                                                                   [ 99%]
tests/unit/test_view.py .                                                                                                                                      [100%]

---------------------- generated xml file: /Users/swast/src/github.com/googleapis/python-bigquery-sqlalchemy/prerelease_unit_3.8_sponge_log.xml ----------------------
================================================================== 224 passed, 3 skipped in 14.81s ===================================================================
nox > py.test --quiet --junitxml=prerelease_system_3.8_sponge_log.xml tests/system
======================================================================== test session starts =========================================================================
platform darwin -- Python 3.8.10, pytest-6.2.5, py-1.11.0, pluggy-1.0.0
rootdir: /Users/swast/src/github.com/googleapis/python-bigquery-sqlalchemy, configfile: setup.cfg
plugins: cov-3.0.0
collected 49 items / 2 skipped / 47 selected                                                                                                                         

tests/system/test__struct.py ...                                                                                                                               [  6%]
tests/system/test_helpers.py ssss                                                                                                                              [ 14%]
tests/system/test_sqlalchemy_bigquery.py ..........................................                                                                            [100%]

--------------------- generated xml file: /Users/swast/src/github.com/googleapis/python-bigquery-sqlalchemy/prerelease_system_3.8_sponge_log.xml ---------------------
============================================================= 45 passed, 6 skipped in 284.47s (0:04:44) ==============================================================

Even though we use the latest SQLAlchemy, it is passing since we aren't running against the compliance test suite (yet?).

See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md

This reverts commit 01aca60.

See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md

tswast · 2022-01-19T20:45:04Z

noxfile.py



+@nox.session(python=DEFAULT_PYTHON_VERSION)
+def prerelease(session):


Note: Mostly taken straight from https://github.com/googleapis/python-bigquery-pandas/blob/d4539747848156446d8a3ccdb49661e72ae03109/noxfile.py#L176

googleapis/synthtool#1290 is open to add this to the templates, though it would need to be slightly templated to allow for customized prerelease dependencies.

noxfile.py

…-prerelease

tswast · 2022-02-14T21:06:40Z

FYI: a pytest release in the past month broke the compliance test suite: #413

Edit: Fixed by capping pytest version to less than 7.

parthea

LGTM, Added 1 observation

parthea · 2022-02-16T15:58:45Z

noxfile.py

+        "google-cloud-bigquery-storage",
+        "google-cloud-core",
+        "google-resumable-media",
+        "grpcio",


These 4 don't appear in setup.py . Please could you add a comment to clarify the reason that they're included here?

That's a good point. Possibly we should add google-cloud-bigquery-storage to setup.py in a separate PR. It's an "extra" but one that'll raise a warning if not present. The others we get via transitive dependencies. I'll comment.

Mailed #414 which updates some of the dependencies in setup.py to better reflect reality.

Also, remove google-resumable-media from prerelease tests, as this package does not use load jobs.

…-prerelease

…ing (#414) The wrapped DB-API uses google-cloud-bigquery-storage to download query results by default. It raises a warning and falls back to the REST API if google-cloud-bigquery-storage is not installed. The google-cloud-bigquery version 3.x library is tested in #401 and it does not include any breaking changes that should affect the SQLAlchemy connector. deps: allow google-cloud-bigquery version 3.x Thank you for opening a Pull Request! Before submitting your PR, there are a few things you can do to make sure it goes smoothly: - [ ] Make sure to open an issue as a [bug/issue](https://github.com/googleapis/python-bigquery-sqlalchemy/issues/new/choose) before writing your code! That way we can discuss the change, evaluate designs, and agree on the general idea - [ ] Ensure the tests and linter pass - [ ] Code coverage does not decrease (if any source code was changed) - [ ] Appropriate docs were updated (if necessary) Fixes #<issue_number_goes_here> 🦕

test: add prerelease session to test with latest dependencies

dfe4fc2

product-auto-label bot added the api: bigquery Issues related to the googleapis/python-bigquery-sqlalchemy API. label Jan 18, 2022

gcf-owl-bot bot and others added 5 commits January 18, 2022 23:18

🦉 Updates from OwlBot

01aca60

See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md

Merge branch 'main' into tswast-prerelease

0625f3d

Revert "🦉 Updates from OwlBot"

93b3286

This reverts commit 01aca60.

Avoid owlbot from overwriting my changes

211cc2b

split compliance and prerelease tests out

efdceac

tswast changed the title ~~test: add prerelease session to test with latest dependencies~~ test: add prerelease session to test with latest dependencies, separate compliance tests Jan 19, 2022

tswast and others added 3 commits January 19, 2022 14:14

remove newline escape

dea0707

escape regex

12900cb

🦉 Updates from OwlBot

1605357

See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md

tswast commented Jan 19, 2022

View reviewed changes

tswast marked this pull request as ready for review January 19, 2022 20:47

tswast requested a review from a team January 19, 2022 20:47

tswast requested a review from a team as a code owner January 19, 2022 20:47

tswast requested a review from steffnay January 19, 2022 20:47

tswast added the do not merge Indicates a pull request not ready for merge, due to either quality or timing. label Jan 19, 2022

This comment has been minimized.

Sign in to view

tswast commented Jan 19, 2022

View reviewed changes

noxfile.py Outdated Show resolved Hide resolved

tswast added 2 commits January 19, 2022 17:07

remove unused [all]

31bb104

Merge remote-tracking branch 'upstream/tswast-prerelease' into tswast…

288a11d

…-prerelease

tswast removed the do not merge Indicates a pull request not ready for merge, due to either quality or timing. label Jan 20, 2022

tswast mentioned this pull request Jan 20, 2022

chore(python): exclude templated GH action workflows #406

Merged

Merge branch 'main' into tswast-prerelease

4e42d8b

tswast added do not merge Indicates a pull request not ready for merge, due to either quality or timing. and removed do not merge Indicates a pull request not ready for merge, due to either quality or timing. labels Feb 14, 2022

tswast added 2 commits February 14, 2022 15:22

disallow latest pytest due to incompatibilities

6ee316f

only use single python for compliance testing

4d22f16

This was referenced Feb 14, 2022

compliance tests not running with pytest >= 7.0.0 #413

Closed

fix: POSTCOMPILE expansion in SQLAlchemy 1.4.27+ #408

Merged

chore: add custom sync repo settings #412

Merged

Merge branch 'main' into tswast-prerelease

0da9200

parthea approved these changes Feb 16, 2022

View reviewed changes

tswast mentioned this pull request Feb 17, 2022

deps: require google-cloud-bigquery-storage to avoid performance warning #414

Merged

4 tasks

tswast added 2 commits February 17, 2022 11:05

comment about transitive deps

74630c0

Also, remove google-resumable-media from prerelease tests, as this package does not use load jobs.

Merge remote-tracking branch 'upstream/tswast-prerelease' into tswast…

2b77257

…-prerelease

tswast merged commit 7f36628 into main Feb 17, 2022

tswast deleted the tswast-prerelease branch February 17, 2022 18:24

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

test: add prerelease session to test with latest dependencies, separate compliance tests #401

test: add prerelease session to test with latest dependencies, separate compliance tests #401

Uh oh!

tswast commented Jan 18, 2022 •

edited

Loading

Uh oh!

tswast commented Jan 18, 2022

Uh oh!

tswast Jan 19, 2022

Uh oh!

This comment has been minimized.

Uh oh!

tswast commented Feb 14, 2022 •

edited

Loading

Uh oh!

parthea left a comment

Uh oh!

parthea Feb 16, 2022 •

edited

Loading

Uh oh!

tswast Feb 16, 2022

Uh oh!

tswast Feb 17, 2022

Uh oh!

Uh oh!



		@nox.session(python=DEFAULT_PYTHON_VERSION)
		def prerelease(session):

test: add prerelease session to test with latest dependencies, separate compliance tests #401

test: add prerelease session to test with latest dependencies, separate compliance tests #401

Uh oh!

Conversation

tswast commented Jan 18, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

tswast commented Jan 18, 2022

Uh oh!

tswast Jan 19, 2022

Choose a reason for hiding this comment

Uh oh!

This comment has been minimized.

Uh oh!

tswast commented Feb 14, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

parthea left a comment

Choose a reason for hiding this comment

Uh oh!

parthea Feb 16, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

tswast Feb 16, 2022

Choose a reason for hiding this comment

Uh oh!

tswast Feb 17, 2022

Choose a reason for hiding this comment

Uh oh!

Uh oh!

tswast commented Jan 18, 2022 •

edited

Loading

tswast commented Feb 14, 2022 •

edited

Loading

parthea Feb 16, 2022 •

edited

Loading