-
Notifications
You must be signed in to change notification settings - Fork 1k
feat(snowflake)!: support transpilation of BITSHIFTLEFT and BITSHIFTRIGHT #6586
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Conversation
SQLGlot Integration Test ResultsComparing:
By Dialect
Overallmain: 6173 total, 5610 passed (pass rate: 90.9%), sqlglot version: sqlglot:feature/transpile-bitshift: 6173 total, 5610 passed (pass rate: 90.9%), sqlglot version: Difference: No change |
georgesittas
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This needs testing, but I'm skeptical about the approach. May need to sync in Slack about this.
@georgesittas Please see the Jira ticket for all tests that were run prior to the first push. There's about 300 test queries against Snowflake, transpiled, run against DuckDB and results checked against one another. |
|
We need to add tests. |
9f96195 to
21052b5
Compare
21052b5 to
7c4b5e3
Compare
…DB with INT128 casts and precedence fixes
83f7591 to
526cd22
Compare
|
@fivetran-kwoodbeck @VaggelisD take a look at the refactored PR when you get the chance, took a stab at cleaning it up a bit. |
VaggelisD
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Only got one question, @fivetran-kwoodbeck if the refactored logic & tests check out feel free to merge
| self.validate_all( | ||
| "SELECT BITSHIFTRIGHT(X'FF', 4)", | ||
| write={ | ||
| "snowflake": "SELECT BITSHIFTRIGHT(255, 4)", | ||
| "duckdb": "SELECT CAST(255 AS INT128) >> 4", | ||
| }, | ||
| ) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This generates a BINARY in Snowflake but an INT in DuckDB, this is intentional given that we don't annotate the types (?)
sf> SELECT system$typeof(BITSHIFTRIGHT(X'FF', 4)), BITSHIFTRIGHT(X'FF', 4);
BINARY[LOB] | 0F
-- | --
duckdb> SELECT CAST(255 AS INT128) >> 4;
┌─────────────────────────────┐
│ (CAST(255 AS HUGEINT) >> 4) │
│ int128 │
├─────────────────────────────┤
│ 15 │
└─────────────────────────────┘There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
What if we annotate the input w/ snowflake? I think preserving the type requires type inference to run first.
|
Hi @VaggelisD The tests don't pass and I'm not sure what's the best way to deal with it (a lot of back and forth). There are 2 issues, both of which can be seen here: The first is I think it needs an Which I think can be solved with an extra |
DuckDB has support for left and right shift (<< and >>) up to INT128, but by default it assumes INT32 and needs casting in order to prevent it from throwing errors in various situations. See the Jira tickets for full testing.
Snowflake claims it always returns an INT128 (for INT input). In practice, it scales based on result and ranges from NUMBER(3,0) to NUMBER(38,0) depending on what number is returned. Snowflake also supports BINARY input.
Snowflake does not throw errors when you shift over the max int, DuckDB does.
Documentation:
https://docs.snowflake.com/en/sql-reference/functions/bitshiftleft
https://docs.snowflake.com/en/sql-reference/functions/bitshiftright