Thanks to visit codestin.com
Credit goes to github.com

Skip to content

Conversation

OmriBromberg
Copy link
Contributor

@OmriBromberg OmriBromberg commented May 12, 2021

Thank you for opening a Pull Request! Before submitting your PR, there are a few things you can do to make sure it goes smoothly:

  • [V] Make sure to open an issue as a bug/issue before writing your code! That way we can discuss the change, evaluate designs, and agree on the general idea
  • [V] Ensure the tests and linter pass
  • [V] Code coverage does not decrease (if any source code was changed)
  • [V] Appropriate docs were updated (if necessary)

Fixes #173 🦕

Hey guys, I explained in the issue a bit but I wanted to implement it so users would be able to change max_results @ dataset.list_tables like the arraysize parameter so I copied how it is treated in the parse_url area and updated the tests and docs for it, the new parameter for it is called list_tables_page_size

also I saw that on line 606
there is this expression self.arraysize = self.arraysize or arraysize which always takes the default arraysize instead of the parsed querystring arraysize because self.arraysize is always initialized to a default value, so I also reversed it to be as so self.arraysize = arraysize or self.arraysize, now the querystring arraysize will take precedence over the default value

@OmriBromberg OmriBromberg requested a review from a team as a code owner May 12, 2021 21:12
@product-auto-label product-auto-label bot added the api: bigquery Issues related to the googleapis/python-bigquery-sqlalchemy API. label May 12, 2021
@google-cla
Copy link

google-cla bot commented May 12, 2021

Thanks for your pull request. It looks like this may be your first contribution to a Google open source project (if not, look below for help). Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

📝 Please visit https://cla.developers.google.com/ to sign.

Once you've signed (or fixed any issues), please reply here with @googlebot I signed it! and we'll verify it.


What to do if you already signed the CLA

Individual signers
Corporate signers

ℹ️ Googlers: Go here for more info.

@google-cla google-cla bot added the cla: no This human has *not* signed the Contributor License Agreement. label May 12, 2021
@OmriBromberg
Copy link
Contributor Author

@googlebot I signed it!

@google-cla google-cla bot added cla: yes This human has signed the Contributor License Agreement. and removed cla: no This human has *not* signed the Contributor License Agreement. labels May 12, 2021
@OmriBromberg OmriBromberg force-pushed the feature/list_tables_page_size branch from 5a97fa7 to 085df0f Compare May 12, 2021 21:28
…peration as list_tables_page_size

fix: arraysize default always takes priority over querystring
@OmriBromberg OmriBromberg force-pushed the feature/list_tables_page_size branch from 085df0f to 5331bcd Compare May 12, 2021 21:31
@jimfulton jimfulton changed the title Feature/list tables page size feat: Feature/list tables page size May 14, 2021
@jimfulton jimfulton added kokoro:force-run Add this label to force Kokoro to re-run the tests. kokoro:run Add this label to force Kokoro to re-run the tests. labels May 16, 2021
@yoshi-kokoro yoshi-kokoro removed kokoro:run Add this label to force Kokoro to re-run the tests. kokoro:force-run Add this label to force Kokoro to re-run the tests. labels May 16, 2021
@jimfulton jimfulton self-requested a review May 16, 2021 20:32
Copy link
Contributor

@jimfulton jimfulton left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a great start. In addition to adding a feature, you found two bugs!

This needs test of the logic that fetches lists of tables. A unit test would be fine and should be straightforward to write.

@OmriBromberg
Copy link
Contributor Author

OmriBromberg commented May 19, 2021

hey, regarding this can you explain this logic to me? why shouldn't it always take the "full_table_id" property? gives us inconsistencies

@jimfulton
Copy link
Contributor

hey, regarding this can you explain this logic to me? why shouldn't it always take the "full_table_id" property? gives us inconsistencies

That's a good question.

If you look at: https://github.com/sqlalchemy/sqlalchemy/blob/b20b6f8fe7ea0198f819a0fd68ca076b6c760054/lib/sqlalchemy/testing/suite/test_reflection.py#L537-L579

SQLAlchemy expects the results to be unqualified, which we satisfy if we have a schema.

Looking at other dialects:

MS SQL requires a schema: https://github.com/sqlalchemy/sqlalchemy/blob/b20b6f8fe7ea0198f819a0fd68ca076b6c760054/lib/sqlalchemy/dialects/mssql/base.py#L2877

Oracle assumes there's a default: https://github.com/sqlalchemy/sqlalchemy/blob/b20b6f8fe7ea0198f819a0fd68ca076b6c760054/lib/sqlalchemy/dialects/oracle/base.py#L1759-L1760

as does Postgres: https://github.com/sqlalchemy/sqlalchemy/blob/b20b6f8fe7ea0198f819a0fd68ca076b6c760054/lib/sqlalchemy/dialects/postgresql/base.py#L3488-L3490

As does sqlite, sort of: https://github.com/sqlalchemy/sqlalchemy/blob/b20b6f8fe7ea0198f819a0fd68ca076b6c760054/lib/sqlalchemy/dialects/sqlite/base.py#L1977-L1981

and so on.

So BQ is a little different because there might not be a default.

Amongst the options:

  1. Error if no schema is provided and there's no default.
  2. Qualify the results if we don't have a schema. I prefer using just the data set id in this case. full_table_id adds clutter IMO.

Playing with this some more, you can list schemas in other projects by specifying project_id.schema_id (e.g. "bigquery-public-data.census_bureau_acs"). That makes the use of a colon in full_table_id a little weird.

@jimfulton jimfulton marked this pull request as draft May 24, 2021 20:21
@jimfulton
Copy link
Contributor

The bigquery client now accepts a page_size parameter, so this PR wants to use that.

When your;re ready for me to review, please merge master to get rid of many spurious changes. :)

@OmriBromberg OmriBromberg marked this pull request as ready for review July 4, 2021 06:29
@jimfulton jimfulton added kokoro:force-run Add this label to force Kokoro to re-run the tests. kokoro:run Add this label to force Kokoro to re-run the tests. labels Jul 6, 2021
@yoshi-kokoro yoshi-kokoro removed kokoro:run Add this label to force Kokoro to re-run the tests. kokoro:force-run Add this label to force Kokoro to re-run the tests. labels Jul 6, 2021
@jimfulton
Copy link
Contributor

Thanks!

@jimfulton jimfulton merged commit e0f1496 into googleapis:master Jul 6, 2021
jimfulton pushed a commit to jimfulton/python-bigquery-sqlalchemy that referenced this pull request Jul 22, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
api: bigquery Issues related to the googleapis/python-bigquery-sqlalchemy API. cla: yes This human has signed the Contributor License Agreement.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

dataset.list_tables max_results parameter
3 participants