Can't create JDBC shared memory DB #4018

silverhammermba · 2023-03-28T18:49:09Z

SQLDelight Version

1.5.4

Application Operating System

Android

Describe the Bug

JDBC supports sqlite's in-memory DB with shared cache, you can see this in their tests where connections are made with URIs such as

jdbc:sqlite:file:memdb1?mode=memory&cache=shared
jdbc:sqlite:file::memory:?cache=shared

But sqldelight's JdbcSqliteDriver does not support such URIs. I can only get in-memory DBs to work using JdbcSqliteDriver.IN_MEMORY which is hard-coded to jdbc:sqlite:.

For example, I want to use this in my tests so that I can test separate classes accessing a common DB in memory as if it exists on disk:

var dbUrl = "jdbc:sqlite:file:inMemTestDb?mode=memory&cache=shared"

val driver1 = JdbcSqliteDriver(dbUrl)
MyDB.Schema.create(driver1) // crash is here
val db1 = MyDB(driver1)

// opens a separate connection to same DB as db1 without creating anything in the filesystem
val driver2 = JdbcSqliteDriver(dbUrl)
MyDB.Schema.create(driver2)
val db2 = MyDB(driver2)

This results in the stacktrace below.

The code only runs if I use JdbcSqliteDriver.IN_MEMORY, but that gives incorrect behavior because then db1 and db2 are completely separate in-memory databases.

Stacktrace

[SQLITE_ERROR] SQL error or missing database (no such table: main.hockeyPlayer)
org.sqlite.SQLiteException: [SQLITE_ERROR] SQL error or missing database (no such table: main.hockeyPlayer)
	at app//org.sqlite.core.DB.newSQLException(DB.java:1012)
	at app//org.sqlite.core.DB.newSQLException(DB.java:1024)
	at app//org.sqlite.core.DB.throwex(DB.java:989)
	at app//org.sqlite.core.NativeDB.prepare_utf8(Native Method)
	at app//org.sqlite.core.NativeDB.prepare(NativeDB.java:134)
	at app//org.sqlite.core.DB.prepare(DB.java:257)
	at app//org.sqlite.core.CorePreparedStatement.<init>(CorePreparedStatement.java:45)
	at app//org.sqlite.jdbc3.JDBC3PreparedStatement.<init>(JDBC3PreparedStatement.java:30)
	at app//org.sqlite.jdbc4.JDBC4PreparedStatement.<init>(JDBC4PreparedStatement.java:25)
	at app//org.sqlite.jdbc4.JDBC4Connection.prepareStatement(JDBC4Connection.java:35)
	at app//org.sqlite.jdbc3.JDBC3Connection.prepareStatement(JDBC3Connection.java:241)
	at app//org.sqlite.jdbc3.JDBC3Connection.prepareStatement(JDBC3Connection.java:205)
	at app//com.squareup.sqldelight.sqlite.driver.JdbcDriver.execute(JdbcDriver.kt:109)
	at app//com.squareup.sqldelight.db.SqlDriver$DefaultImpls.execute$default(SqlDriver.kt:52)
	at app//com.foo.bar.MyDBImpl$Schema.create(MyDBImpl.kt:40)

The text was updated successfully, but these errors were encountered:

AlecKazakova · 2023-03-28T19:36:41Z

this would be an issue with the underlying driver and not sqldelight, in this case https://github.com/xerial/sqlite-jdbc

silverhammermba · 2023-03-28T19:44:55Z

@AlecStrong This is not an underlying issue with jdbc since you can see them using these connection URIs in their tests: https://github.com/xerial/sqlite-jdbc/blob/3d04d7df0c89240add2c92189adb30b6cb7e6ae0/src/test/java/org/sqlite/ConnectionTest.java#L282

The fact that the same URIs do not work in sqldelight points to it being a sqldelight issue.

dellisd · 2023-03-28T20:02:11Z

SQLDelight's SQLite driver ultimately runs the same DriverManager.getConnection() call that they do in their tests.

https://github.com/cashapp/sqldelight/blob/master/drivers/sqlite-driver/src/main/kotlin/app/cash/sqldelight/driver/jdbc/sqlite/JdbcSqliteDriver.kt#L95

05nelsonm · 2023-09-21T14:14:24Z

Would like to re-open this issue

SQLDelight's SQLite driver ultimately runs the same DriverManager.getConnection() call that they do in their tests.

https://github.com/cashapp/sqldelight/blob/master/drivers/sqlite-driver/src/main/kotlin/app/cash/sqldelight/driver/jdbc/sqlite/JdbcSqliteDriver.kt#L95

if providing a url that is not the hard coded value specified by JdbcSqliteDriver.IN_MEMORY, it will use the ThreadedConnectionManager. That behavior results in connections opening and closing after every driver.execute or driver.executeQuery. This is incorrect behavior if the uri specified mode=memory as the connection should remain open until driver.close has been invoked.

i.e., if the uri contains mode=memory, the InMemoryConnectionManager should be utilized.

05nelsonm · 2023-09-21T14:25:44Z

Here's what is being checked for in xerial/sqlite-jdbc

https://github.com/xerial/sqlite-jdbc/blob/3d04d7df0c89240add2c92189adb30b6cb7e6ae0/src/main/java/org/sqlite/SQLiteConnection.java#L195

So, JdbcSqliteDriver must also check these conditions so that it can utilize the InMemoryConnectionManager instead of the ThreadedConnectionManager (which will open and close connections instead of keeping the connection open).

Must check for the following 3 conditions:

url == ":jdbc:sqlite:"
url.startsWith(":jdbc:sqlite::memory:")
- NOTE: must use startsWith as parameters can be passed after :memory:
url.contains("mode=memory")

dellisd · 2023-09-21T17:24:22Z

Thanks for following up on this issue and investigating further. Based on what you found I think this is something that can/should be fixed in the JdbcSqliteDriver. PR welcome!

05nelsonm · 2023-09-21T17:25:33Z

Thanks for following up on this issue and investigating further. Based on what you found I think this is something that can/should be fixed in the JdbcSqliteDriver. PR welcome!

On it!

05nelsonm · 2023-09-21T19:50:18Z

So, little issue here with the difference between a temporary database, and an in-memory database.

jdbc:sqlite:uri, where uri is the expression that is served to SQLite.

If the path expression within the uri is empty (i.e. jdbc:sqlite:?pragma1=asdfasfd&pragma2=dfgh), it creates a temporary database. The temporary file is then deleted upon connection closure. It is "mostly in-memory", unless the data becomes too large which it then will dump data to the disk. (so it still creates a file).

If the path within uri is :memory:, it creates a purely in-memory database, never dumping to disk.

If the path within uri for the file expressed is file::memory:

Currently, the nomenclature used (i.e. JdbcSqliteDriver.IN_MEMORY) does not match what is actually happening. It should be named TEMPORARY. Modifying the value of it would potentially break things for library consumers. How should one proceed here, a deprecation cycle?

05nelsonm · 2023-09-21T19:58:18Z

Would something like the following work for you @dellisd ?

companion {

  /**
   *  some documentation ...
   * */
  const val URL_PREFIX = "jdbc:sqlite:"

  /**
   *  some documentation ...
   * */
  const val URL_TEMPORARY = JDBC_PREFIX + ""

  /**
   *  some documentation ...
   * */
  const val URL_IN_MEMORY = JDBC_PREFIX + ":memory:"

  @Deprecated("reasons ... TODO")
  const val IN_MEMORY = "jdbc:sqlite:"
}

dellisd · 2023-09-21T23:10:42Z

My opinion would be that the IN_MEMORY constant should be updated to jdbc:sqlite::memory: to reflect what it's actually supposed to be. A new TEMPORARY constant could also be introduced.

I can see how this could be a breaking change if someone were using IN_MEMORY to process a lot of data, but... I'll defer this to @AlecKazakova.

05nelsonm · 2023-09-21T23:30:02Z

My opinion would be that the IN_MEMORY constant should be updated to jdbc:sqlite::memory: to reflect what it's actually supposed to be. A new TEMPORARY constant could also be introduced.

I can see how this could be a breaking change if someone were using IN_MEMORY to process a lot of data, but... I'll defer this to @AlecKazakova.

100% agree, it'd be much cleaner of an API change. The behavioral change of modifying IN_MEMORY's value though was the primary factor in me leaning towards deprecation + introducing the URL_ prefixed constants; simply appending :memory: to the value could be catastrophic.

05nelsonm · 2023-09-23T05:56:04Z

Regarding the nomenclature issue about IN_MEMORY and TEMPORARY, I've created a separate issue for it and can handle it in a separate PR from the url parsing fixes.

See #4653

silverhammermba added the bug label Mar 28, 2023

silverhammermba changed the title ~~JDBC shared memory~~ Can't create JDBC shared memory DB Mar 28, 2023

AlecKazakova closed this as completed Mar 28, 2023

05nelsonm mentioned this issue Sep 21, 2023

Rework creation of in memory dbs toxicity-io/sqlite-mc#23

Closed

dellisd reopened this Sep 21, 2023

dellisd added PR WELCOME! component: drivers/jdbc labels Sep 21, 2023

This was referenced Sep 23, 2023

Fix JdbcSqliteDriver url parsing when choosing ConnectionManager type #4652

Closed

JdbcSqliteDriver.IN_MEMORY nomenclature is incorrect #4653

Open

This was referenced Sep 23, 2023

Fix JdbcSqliteDriver url parsing when choosing ConnectionManager type #4654

Closed

Fix JdbcSqliteDriver url parsing when choosing ConnectionManager type #4656

Merged

dellisd closed this as completed in #4656 Sep 25, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Can't create JDBC shared memory DB #4018

Can't create JDBC shared memory DB #4018

silverhammermba commented Mar 28, 2023 •

edited

Loading

AlecKazakova commented Mar 28, 2023

silverhammermba commented Mar 28, 2023

dellisd commented Mar 28, 2023

05nelsonm commented Sep 21, 2023 •

edited

Loading

05nelsonm commented Sep 21, 2023 •

edited

Loading

dellisd commented Sep 21, 2023

05nelsonm commented Sep 21, 2023

05nelsonm commented Sep 21, 2023 •

edited

Loading

05nelsonm commented Sep 21, 2023 •

edited

Loading

dellisd commented Sep 21, 2023 •

edited

Loading

05nelsonm commented Sep 21, 2023 •

edited

Loading

05nelsonm commented Sep 23, 2023

Can't create JDBC shared memory DB #4018

Can't create JDBC shared memory DB #4018

Comments

silverhammermba commented Mar 28, 2023 • edited Loading

SQLDelight Version

Application Operating System

Describe the Bug

Stacktrace

AlecKazakova commented Mar 28, 2023

silverhammermba commented Mar 28, 2023

dellisd commented Mar 28, 2023

05nelsonm commented Sep 21, 2023 • edited Loading

05nelsonm commented Sep 21, 2023 • edited Loading

dellisd commented Sep 21, 2023

05nelsonm commented Sep 21, 2023

05nelsonm commented Sep 21, 2023 • edited Loading

05nelsonm commented Sep 21, 2023 • edited Loading

dellisd commented Sep 21, 2023 • edited Loading

05nelsonm commented Sep 21, 2023 • edited Loading

05nelsonm commented Sep 23, 2023

silverhammermba commented Mar 28, 2023 •

edited

Loading

05nelsonm commented Sep 21, 2023 •

edited

Loading

05nelsonm commented Sep 21, 2023 •

edited

Loading

05nelsonm commented Sep 21, 2023 •

edited

Loading

05nelsonm commented Sep 21, 2023 •

edited

Loading

dellisd commented Sep 21, 2023 •

edited

Loading

05nelsonm commented Sep 21, 2023 •

edited

Loading