Support SqlDelight modules to implement PostgreSql extensions #5677

griffio · 2025-03-05T12:56:42Z

☢️ Experimental
🐙 Use the SqlDelight module service loader (see sqlite json-module )to implement Postgresql 3rd party extensions - rather than add in the PostgreSql grammar or having to create a new sub dialect.

sqldelight {
   databases {
       create("Sample") {
           dialect(libs.sqldelight.postgresql.dialect)
           module(project(":postgis-module")) // module can be local project or external dependency
       }
   }
}

🧪 Example extension modules using these changes:

bm25 - text search - https://github.com/griffio/sqldelight-bm25-module-app
pgvector - vector embeddings - https://github.com/griffio/sqldelight-pgvector-module-app
postgis - ever popular - https://github.com/griffio/sqldelight-postgis-module-app

Open PostgreSqlTypeResolver for inheritance to allow calling into subclasses - delegation is not enough
Public PostgreSqlType for modules to access existing PostgreSql types
Add protected booleanBinaryExprTypes as modules can add boolean binary operator types

sausti · 2025-03-23T14:29:56Z

👋 @griffio This is great! Are there any plans to move this out of a draft state and merge it for use in snapshot builds? We're using SQLDelight now with Postgres and were hoping to leverage pgvector.

griffio · 2025-03-23T18:23:03Z

OK have set for review - I implemented several interesting extensions to see what problems and work-arounds are needed.

Keep in mind SqlDelight is really for basic Sql operations -
There are still many Postgresql built-in features not implemented in SqlDelight that some of these extensions may use - e.g. Schemas, Create Functions, Triggers.

sausti · 2025-03-24T14:59:24Z

OK have set for review - I implemented several interesting extensions to see what problems and work-arounds are needed.

Keep in mind SqlDelight is really for basic Sql operations - There are still many Postgresql built-in features not implemented in SqlDelight that some of these extensions may use - e.g. Schemas, Create Functions, Triggers.

Thank you! And, of course. Our PostgreSQL use cases are straightforward (no need for Triggers, etc.). Your change here unblocks pgvector and AI Vector embedding models, which is really what we're (and I'm sure others!) are excited about.

Make the PostgreSqlType public so it can be accessed in third party modules

open PostgreSqlTypeResolver for extension and polymorphic calls booleanBinaryExprTypes is a new protected method to allow subclasses to add types

griffio · 2025-03-31T08:53:39Z

@hfhbd Can this be approved and merged for snapshot testing? I implemented 3 example modules to test it
Follows on from #5625
This is an alternative approach to including third party extensions e.g #5602 and avoids having to create a new dialect

DRSchlaubi · 2025-05-12T07:04:01Z

Any update on snapshots?

AlecKazakova · 2025-05-13T14:15:25Z

love seeing stuff being built as extensions <3

DRSchlaubi · 2025-05-20T08:27:33Z

I just tested the latest release and it works great for a single module, but if you try adding multiple modules (like bm25 and postgis) only the module added last seems to work

griffio · 2025-05-20T10:08:20Z

I just tested the latest release and it works great for a single module, but if you try adding multiple modules (like bm25 and postgis) only the module added last seems to work

Sadly - that is the big limitation - as a module is basically similar to a dialect (you can't have two dialects at the same time). Mixing in more than one module doesn't work as there is only single inheritance (or single composition) of a GrammarKit parser to work with. (e.g to use more than one module like bm25 and postgis , one module would have to inherit or compose the other).

The idea was that users can, atleast, maintain their own uber-module with the extensions they use, without waiting for features that may not be added to a core Sqldelight dialect.

AlecKazakova · 2025-05-20T14:28:27Z

i guess i never tested but the intention was to be able to load multiple modules, technically you can have modules compose together without knowing about each other, this is how sqldelight adds its special syntax: https://github.com/sqldelight/sqldelight/blob/master/sqldelight-compiler/src/main/kotlin/app/cash/sqldelight/core/sqldelight.bnf

as it technically is applied last after the dialect, but doesn't know about or extend the dialect

definitely seems worth fixing or thinking about more

| Package | Type | Package file | Manager | Update | Change | |---|---|---|---|---|---| | [com.github.jsqlparser:jsqlparser](https://github.com/JSQLParser/JSqlParser) | dependencies | misk/gradle/libs.versions.toml | gradle | minor | `5.2` -> `5.3` | | [app.cash.sqldelight](https://github.com/sqldelight/sqldelight) | plugin | misk/gradle/libs.versions.toml | gradle | minor | `2.0.2` -> `2.1.0` | | [app.cash.sqldelight:runtime](https://github.com/sqldelight/sqldelight) | dependencies | misk/gradle/libs.versions.toml | gradle | minor | `2.0.2` -> `2.1.0` | | [app.cash.sqldelight:mysql-dialect](https://github.com/sqldelight/sqldelight) | dependencies | misk/gradle/libs.versions.toml | gradle | minor | `2.0.2` -> `2.1.0` | | [app.cash.sqldelight:jdbc-driver](https://github.com/sqldelight/sqldelight) | dependencies | misk/gradle/libs.versions.toml | gradle | minor | `2.0.2` -> `2.1.0` | | [software.amazon.awssdk:sdk-core](https://aws.amazon.com/sdkforjava) | dependencies | misk/gradle/libs.versions.toml | gradle | patch | `2.31.44` -> `2.31.45` | | [software.amazon.awssdk:sqs](https://aws.amazon.com/sdkforjava) | dependencies | misk/gradle/libs.versions.toml | gradle | patch | `2.31.44` -> `2.31.45` | | [software.amazon.awssdk:dynamodb-enhanced](https://aws.amazon.com/sdkforjava) | dependencies | misk/gradle/libs.versions.toml | gradle | patch | `2.31.44` -> `2.31.45` | | [software.amazon.awssdk:dynamodb](https://aws.amazon.com/sdkforjava) | dependencies | misk/gradle/libs.versions.toml | gradle | patch | `2.31.44` -> `2.31.45` | | [software.amazon.awssdk:aws-core](https://aws.amazon.com/sdkforjava) | dependencies | misk/gradle/libs.versions.toml | gradle | patch | `2.31.44` -> `2.31.45` | | [software.amazon.awssdk:bom](https://aws.amazon.com/sdkforjava) | dependencies | misk/gradle/libs.versions.toml | gradle | patch | `2.31.44` -> `2.31.45` | | [software.amazon.awssdk:auth](https://aws.amazon.com/sdkforjava) | dependencies | misk/gradle/libs.versions.toml | gradle | patch | `2.31.44` -> `2.31.45` | --- ### Release Notes <details> <summary>sqldelight/sqldelight (app.cash.sqldelight)</summary> ### [`v2.1.0`](https://github.com/sqldelight/sqldelight/blob/HEAD/CHANGELOG.md#210---2025-05-16) [Compare Source](sqldelight/sqldelight@2.0.2...2.1.0) ##### Added - \[WASM Driver] Add support for wasmJs to web worker driver ([#5534](sqldelight/sqldelight#5534) by \[Ilya Gulya]\[IlyaGulya]) - \[PostgreSQL Dialect] Support PostgreSql UnNest Array to rows ([#5673](sqldelight/sqldelight#5673) by \[Griffio]\[griffio]) - \[PostgreSQL Dialect] PostgreSql TSRANGE/TSTZRANGE support ([#5297](sqldelight/sqldelight#5297) by \[Griffio]\[griffio]) - \[PostgreSQL Dialect] PostgreSql Right Full Join ([#5086](sqldelight/sqldelight#5086) by \[Griffio]\[griffio]) - \[PostgreSQL Dialect] Postrgesql extract from temporal types ([#5273](sqldelight/sqldelight#5273) by \[Griffio]\[griffio]) - \[PostgreSQL Dialect] PostgreSql array contains operators ([#4933](sqldelight/sqldelight#4933) by \[Griffio]\[griffio]) - \[PostgreSQL Dialect] PostgreSql drop constraint ([#5288](sqldelight/sqldelight#5288) by \[Griffio]\[griffio]) - \[PostgreSQL Dialect] Postgresql type casting ([#5089](sqldelight/sqldelight#5089) by \[Griffio]\[griffio]) - \[PostgreSQL Dialect] PostgreSql lateral join operator for subquery ([#5122](sqldelight/sqldelight#5122) by \[Griffio]\[griffio]) - \[PostgreSQL Dialect] Postgresql ILIKE operator ([#5330](sqldelight/sqldelight#5330) by \[Griffio]\[griffio]) - \[PostgreSQL Dialect] PostgreSql XML type ([#5331](sqldelight/sqldelight#5331) by \[Griffio]\[griffio]) - \[PostgreSQL Dialect] PostgreSql AT TIME ZONE ([#5243](sqldelight/sqldelight#5243) by \[Griffio]\[griffio]) - \[PostgreSQL Dialect] Support postgresql order by nulls ([#5199](sqldelight/sqldelight#5199) by \[Griffio]\[griffio]) - \[PostgreSQL Dialect] Add PostgreSQL current date/time function support ([#5226](sqldelight/sqldelight#5226) by \[Drew Dobson]\[drewd]) - \[PostgreSQL Dialect] PostgreSql Regex operators ([#5137](sqldelight/sqldelight#5137) by \[Griffio]\[griffio]) - \[PostgreSQL Dialect] add brin gist ([#5059](sqldelight/sqldelight#5059) by \[Griffio]\[griffio]) - \[MySQL Dialect] Support RENAME INDEX for MySql dialect ([#5212](sqldelight/sqldelight#5212) by \[Oren Kislev]\[orenkislev-faire]) - \[JSON Extension] Add alias to json table function ([#5372](sqldelight/sqldelight#5372) by \[Griffio]\[griffio]) ##### Changed - \[Compiler] Generated query files return row counts for simple mutators ([#4578](sqldelight/sqldelight#4578) by \[Marius Volkhart]\[MariusV]) - \[Native Driver] Update NativeSqlDatabase.kt to change readonly flag for DELETE, INSERT, and UPDATE statements ([#5680](sqldelight/sqldelight#5680) by \[Griffio]\[griffio]) - \[PostgreSQL Dialect] Change PgInterval to String ([#5403](sqldelight/sqldelight#5403) by \[Griffio]\[griffio]) - \[PostgreSQL Dialect] Support SqlDelight modules to implement PostgreSql extensions ([#5677](sqldelight/sqldelight#5677) by \[Griffio]\[griffio]) ##### Fixed - \[Compiler] fix: notify queries when executing group statements with result ([#5006](sqldelight/sqldelight#5006) by \[Vitor Hugo Schwaab]\[vitorhugods]) - \[Compiler] Fix SqlDelightModule type resolver ([#5625](sqldelight/sqldelight#5625) by \[Griffio]\[griffio]) - \[Compiler] Fix 5501 insert object escaped column ([#5503](sqldelight/sqldelight#5503) by \[Griffio]\[griffio]) - \[Compiler] Compiler: Improve error message such that path links are clickable with the correct line & char position. ([#5604](sqldelight/sqldelight#5604) by \[Niklas Baudy]\[vanniktech]) - \[Compiler] Fix issue 5298: allow keywords to be used as table names - \[Compiler] fix named executes and add test - \[Compiler] Consider foreign key table constraints when sorting initialization statements ([#5325](sqldelight/sqldelight#5325) by \[Leon Linhart]\[TheMrMilchmann]) - \[Compiler] Align error underlines properly when tabs are involved ([#5224](sqldelight/sqldelight#5224) by \[Drew Dobson]\[drewd]) - \[JDBC Driver] Fix memory leak for connectionManager during end of transaction - \[JDBC Driver] Run SQLite migrations inside transaction as mentioned in documentation ([#5218](sqldelight/sqldelight#5218) by \[Lukáš Moravec]\[morki]) - \[JDBC Driver] Fix leaking connections after transaction commit / rollback ([#5205](sqldelight/sqldelight#5205) by \[Lukáš Moravec]\[morki]) - \[Gradle Plugin] Execute `DriverInitializer` before `GenerateSchemaTask` ([#5562](sqldelight/sqldelight#5562) by \[Emeka Nwagu]\[nwagu]) - \[Runtime] Fix crash in LogSqliteDriver when real driver is Async ([#5723](sqldelight/sqldelight#5723) by \[Eric Denman]\[edenman]) - \[Runtime] Fix StringBuilder capacity ([#5192](sqldelight/sqldelight#5192) by \[Jan Bína]\[janbina]) - \[PostgreSQL Dialect] PostgreSql create or replace view ([#5407](sqldelight/sqldelight#5407) by \[Griffio]\[griffio]) - \[PostgreSQL Dialect] Postgresql to_json ([#5606](sqldelight/sqldelight#5606) by \[Griffio]\[griffio]) - \[PostgreSQL Dialect] PostgreSql numeric resolver ([#5399](sqldelight/sqldelight#5399) by \[Griffio]\[griffio]) - \[PostgreSQL Dialect] sqlite windows function ([#2799](sqldelight/sqldelight#2799) by \[Griffio]\[griffio]) - \[PostgreSQL Dialect] PostgreSql SELECT DISTINCT ON ([#5345](sqldelight/sqldelight#5345) by \[Griffio]\[griffio]) - \[PostgreSQL Dialect] alter table add column if not exists ([#5309](sqldelight/sqldelight#5309) by \[Griffio]\[griffio]) - \[PostgreSQL Dialect] Postgresql async bind parameter ([#5313](sqldelight/sqldelight#5313) by \[Griffio]\[griffio]) - \[PostgreSQL Dialect] PostgreSql boolean literals ([#5262](sqldelight/sqldelight#5262) by \[Griffio]\[griffio]) - \[PostgreSQL Dialect] PostgreSql window functions ([#5155](sqldelight/sqldelight#5155) by \[Griffio]\[griffio]) - \[PostgreSQL Dialect] PostgreSql isNull isNotNull types ([#5173](sqldelight/sqldelight#5173) by \[Griffio]\[griffio]) - \[PostgreSQL Dialect] PostgreSql select distinct ([#5172](sqldelight/sqldelight#5172) by \[Griffio]\[griffio]) - \[Paging Extension] paging refresh initial load fix ([#5615](sqldelight/sqldelight#5615) by \[Eva]\[evant]) - \[Paging Extension] Add MacOS native targets ([#5324](sqldelight/sqldelight#5324) by \[Vitor Hugo Schwaab]\[vitorhugods]) - \[IntelliJ Plugin] K2 Support </details> --- ### Configuration 📅 **Schedule**: Branch creation - "after 6pm every weekday,before 2am every weekday" in timezone Australia/Melbourne, Automerge - At any time (no schedule defined). 🚦 **Automerge**: Enabled. ♻ **Rebasing**: Never, or you tick the rebase/retry checkbox. 👻 **Immortal**: This PR will be recreated if closed unmerged. Get [config help](https://github.com/renovatebot/renovate/discussions) if that's undesired. --- - [ ] If you want to rebase/retry this PR, check this box --- This PR has been generated by [Renovate Bot](https://github.com/renovatebot/renovate). GitOrigin-RevId: 2c67f3c06e13d86329db5149abfceba98e3701a4

DRSchlaubi · 2025-05-21T08:46:57Z

I just tested the latest release and it works great for a single module, but if you try adding multiple modules (like bm25 and postgis) only the module added last seems to work

Sadly - that is the big limitation - as a module is basically similar to a dialect (you can't have two dialects at the same time). Mixing in more than one module doesn't work as there is only single inheritance (or single composition) of a GrammarKit parser to work with. (e.g to use more than one module like bm25 and postgis , one module would have to inherit or compose the other).

The idea was that users can, atleast, maintain their own uber-module with the extensions they use, without waiting for features that may not be added to a core Sqldelight dialect.

Could you share some example on how someone would implement such an uber-module since I am not all that familiar with GrammarKit and can't get it to work

griffio · 2025-05-22T07:58:19Z

I just tested the latest release and it works great for a single module, but if you try adding multiple modules (like bm25 and postgis) only the module added last seems to work

Sadly - that is the big limitation - as a module is basically similar to a dialect (you can't have two dialects at the same time). Mixing in more than one module doesn't work as there is only single inheritance (or single composition) of a GrammarKit parser to work with. (e.g to use more than one module like bm25 and postgis , one module would have to inherit or compose the other).
The idea was that users can, atleast, maintain their own uber-module with the extensions they use, without waiting for features that may not be added to a core Sqldelight dialect.

Could you share some example on how someone would implement such an uber-module since I am not all that familiar with GrammarKit and can't get it to work

I have created a branch here: https://github.com/griffio/sqldelight-pgvector-module-app/tree/ubermodule
It's not a very satisfactory solution and delicate to implement due to current limitations and requires access to source of existing modules for them to combine together - maybe create a discussion https://github.com/sqldelight/sqldelight/discussions if you have more questions

DRSchlaubi · 2025-05-22T09:25:14Z

Yeah that was pretty much what I ended up with, my issue was, that due to the uber module depending on the other two modules, the service loader still loaded both other modules, essentially overwriting the config of the uber module

My solution was to use shadow to relocate those two dependencies and strip the service definitions

griffio force-pushed the postgresql-modules branch from 30b4fe7 to fb1d1fd Compare March 23, 2025 17:56

griffio marked this pull request as ready for review March 23, 2025 18:23

griffio force-pushed the postgresql-modules branch from fb1d1fd to 09e56d6 Compare March 25, 2025 08:56

griffio added 2 commits March 27, 2025 13:21

Update PostgreSqlType.kt

432d084

Make the PostgreSqlType public so it can be accessed in third party modules

Update PostgreSqlTypeResolver.kt

7a817c1

open PostgreSqlTypeResolver for extension and polymorphic calls booleanBinaryExprTypes is a new protected method to allow subclasses to add types

griffio force-pushed the postgresql-modules branch from 09e56d6 to 7a817c1 Compare March 27, 2025 13:51

AlecKazakova approved these changes May 13, 2025

View reviewed changes

AlecKazakova merged commit 662e250 into sqldelight:master May 13, 2025
12 checks passed

AlecKazakova mentioned this pull request May 13, 2025

Postgres: Support Postgis Point type and related functions. #5602

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Support SqlDelight modules to implement PostgreSql extensions #5677

Support SqlDelight modules to implement PostgreSql extensions #5677

griffio commented Mar 5, 2025 •

edited

Loading

Uh oh!

sausti commented Mar 23, 2025

Uh oh!

griffio commented Mar 23, 2025

Uh oh!

sausti commented Mar 24, 2025

Uh oh!

griffio commented Mar 31, 2025 •

edited

Loading

Uh oh!

DRSchlaubi commented May 12, 2025

Uh oh!

AlecKazakova commented May 13, 2025

Uh oh!

Uh oh!

DRSchlaubi commented May 20, 2025

Uh oh!

griffio commented May 20, 2025

Uh oh!

AlecKazakova commented May 20, 2025

Uh oh!

DRSchlaubi commented May 21, 2025

Uh oh!

griffio commented May 22, 2025

Uh oh!

DRSchlaubi commented May 22, 2025

Uh oh!

Uh oh!

Support SqlDelight modules to implement PostgreSql extensions #5677

Support SqlDelight modules to implement PostgreSql extensions #5677

Conversation

griffio commented Mar 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

sausti commented Mar 23, 2025

Uh oh!

griffio commented Mar 23, 2025

Uh oh!

sausti commented Mar 24, 2025

Uh oh!

griffio commented Mar 31, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

DRSchlaubi commented May 12, 2025

Uh oh!

AlecKazakova commented May 13, 2025

Uh oh!

Uh oh!

DRSchlaubi commented May 20, 2025

Uh oh!

griffio commented May 20, 2025

Uh oh!

AlecKazakova commented May 20, 2025

Uh oh!

DRSchlaubi commented May 21, 2025

Uh oh!

griffio commented May 22, 2025

Uh oh!

DRSchlaubi commented May 22, 2025

Uh oh!

Uh oh!

griffio commented Mar 5, 2025 •

edited

Loading

griffio commented Mar 31, 2025 •

edited

Loading