Allow registering DriverInitializer for VerifyMigrationTask with ServiceLoader mechanism #3986

C2H6O · 2023-03-21T23:05:14Z

This PR will allow consumers to initialize and register their custom java.sql.Driver prior to running migrations.

On the consumer side:

Create an implementation of DriverInitializer

class DriverInitializerImpl(
    // optional params
) : DriverInitializer {
    override fun execute(properties: SqlDelightDatabaseProperties, driverProperties: Properties) {
        // initialize and register the driver with DriverManager
    }
}

and add a resources/META-INF/services/app.cash.sqldelight.gradle.DriverInitializer file pointing to the DriverInitializerImpl.

In a gradle file where you setup SqlDelight database

extensions.configure<SqlDelightExtension> {
  databases.create(name) {
    dependencies.add(configurationName, "<migration driver dependency>")
  }
}

tasks.withType(VerifyMigrationTask::class.java).configureEach {
  driverProperties.set(
    objects.mapProperty(String::class.java, String::class.java).apply {
      put("projectRoot", rootDir.path)
    }
  )
}

Where <migration driver dependency> can either be project(:your_local_project) or coordinates to your published artifact group:artifact:version

sqldelight-gradle-plugin/src/main/kotlin/app/cash/sqldelight/gradle/VerifyMigrationTask.kt

C2H6O · 2023-03-22T16:31:18Z

@AlecStrong @hfhbd would appreciate a review on this. This PR solves our issue with loading a custom driver during migration verification that was caused by ClassLoader isolation.

AlecKazakova · 2023-03-26T19:58:08Z

can you share what your implementation of the MigrationDriver is? I think the property stuff I'm okay with, the driver stuff is pretty obtuse so want to understand the use case

sqldelight-gradle-plugin/src/main/kotlin/app/cash/sqldelight/gradle/SqlDelightDatabase.kt

hfhbd · 2023-03-26T20:13:45Z

Please add some tests otherwise new commits/refactoring could break it.

… of VerifyMigrationDriver

C2H6O · 2023-03-27T22:54:39Z

@AlecStrong added some documentation to the VerifyMigrationDriver. Let me know what you think and I can add more context if needed if it is still unclear.

Implementation of the driver:

class TokenizingDriver: VerifyMigrationDriver {

    companion object {
        const val PROJECT_ROOT = "projectRoot"
    }

    private val wrappedDriver: JDBC = JDBC()

    init {
        // JDBC installs itself on initialization. Let's... not.
        val driverList = DriverManager.getDrivers().toList()

        // If we have other drivers installed, houston, we have a problem!
        assert(driverList.size == 1)
        DriverManager.deregisterDriver(driverList[0])

        DriverManager.registerDriver(this)
    }

    override fun acceptsURL(url: String?): Boolean =
        wrappedDriver.acceptsURL(url)

    override fun connect(url: String?, props: Properties?): Connection {
        require(props != null && props.containsKey(PROJECT_ROOT)) {
            "Ensure connectionProperties are set on the ${VerifyMigrationTask::class} when setting up SqlDelight"
        }
        props.setProperty("enable_load_extension", "true")

        val connection = wrappedDriver.connect(url, props)
        val ext = File(props.getProperty(PROJECT_ROOT), "build/sqlitetokenizer/libsqlite-fts5-synonym-tokenizer.dylib")
        connection.prepareStatement("SELECT load_extension('${ext.absolutePath}', 'sqlite3_fts5_synonym_tokenizer_init');").execute()

        return connection
    }

    override fun getMajorVersion(): Int =
        wrappedDriver.majorVersion

    override fun getMinorVersion(): Int =
        wrappedDriver.minorVersion

    override fun getPropertyInfo(url: String?, info: Properties?): Array<DriverPropertyInfo> =
        wrappedDriver.getPropertyInfo(url, info)

    override fun jdbcCompliant(): Boolean =
        wrappedDriver.jdbcCompliant()

    override fun getParentLogger(): Logger =
        wrappedDriver.parentLogger
}

@hfhbd is there a decent starting point for these tests? I couldn't find any tests currently written that verify VerifyMigrationTask logic. 🙏

AlecKazakova · 2023-03-27T23:11:39Z

sqldelight-gradle-plugin/src/main/kotlin/app/cash/sqldelight/gradle/VerifyMigrationDriver.kt

+ *     }
+ *   }
+ */
+interface VerifyMigrationDriver: Driver


Alright, I understand now. Instead of using a service loader is it possible to just have this type (or a lambda) be another parameter to the verify migration task, I'm imagining an ideal API where you do something like

tasks.withType<VerifyMigrationTask>().configureEach { driverInitializer = { DriverManager.registerDriver(...) } }

also I'd wait to write tests until we get this API where we want it to save yourself the trouble

hmm, I didn't think it was possible since lambdas can't be serialized. Let me look into it, but if you have an example, I'd really appreciate it!

interface DriverInitializer { fun execute() }

VerifyMigrationTask:

@get:Input abstract val driverInitializer: Property<DriverInitializer>

> Cannot fingerprint input property 'driverInitializer': value 'SetupSqldelightKt$setupSqldelight$1$2$1@695e62e3' cannot be serialized.

Ah interesting, adding Serializable interface to the DriverInitializer seems to do the trick

(as in not annotated with @get:input)

Hmm, I'm getting this after removing the @get: Input:

- In plugin 'app.cash.sqldelight.gradle.SqlDelightPlugin$Inject' type 'app.cash.sqldelight.gradle.VerifyMigrationTask' property 'driverInitializer' is missing an input or output annotation. Reason: A property without annotation isn't considered during up-to-date checking. Possible solutions: 1. Add an input or output annotation. 2. Mark it as @Internal. Please refer to https://docs.gradle.org/8.0.2/userguide/validation_problems.html#missing_annotation for more details about this problem.

So perhaps we do need it?

option 2? Marking it as @Internal? And then exposing a function on the task to set it

👍 I didn't have to introduce another function to set it, it seems to work just as well

Yeah, @input is a required input that is checked and fails if it is not set. @internal is optional.

AlecKazakova · 2023-03-28T00:43:33Z

some tests for the migration task are here

hfhbd · 2023-03-28T08:07:51Z

The idea of ServiceLoader was to have a proper interface managing the connection instead of using the JDBC auto service. Having an empty interface as the requirement is (almost) useless.

There is also another option, but I am unsure if we want it:
Option 3: If you only need to fix the autoregistering of JDBC, we could just expose the classpath Gradle configuration so you simply add your classes to the classpath, without any initializer/options in the tasks. But this option would prevent getting the properties from the actual db, you would need to hardcode them in your driver.

AlecKazakova · 2023-03-28T11:59:23Z

I've definitely seen some gradle tasks expose their classpath as ways to do things like this, I'd be pretty cool with that change

C2H6O · 2023-03-28T15:30:47Z

@hfhbd I need to pass the projectRoot directory to the driver and it can't be hardcoded unfortunately

C2H6O · 2023-03-28T15:31:28Z

@AlecStrong I've greatly simplified this PR based on your suggestions 🙏

AlecKazakova

yep, cool with this

hfhbd · 2023-03-28T23:50:58Z

I still try to understand the real problem. Isn't the real problem this hardcoded line? https://github.com/cashapp/sqldelight/blob/02edf13f390229d3eb8986b5c408fd795a085e0f/sqlite-migrations/src/main/kotlin/app/cash/sqlite/migrations/CatalogDatabase.kt#L41
If so, what about refactoring this line and reusing the connectionManager from the dialect? I created a PR: #4019

C2H6O · 2023-03-29T17:34:27Z

@hfhbd now that I'm able to initialize the driver in the context of the worker's ClassLoader, I don't need to pass the properties down to the connect(...) call. I am able to initialize the driver with the right path as a constructor param

hfhbd · 2023-03-29T19:57:39Z

Could you add a test, eg using the GradleRunner, configure the parameter for example println("CALLED DriverInitializer") and check, if it is printed in the log when executing the verifying task? Or instead of printing create some file and check its existing after the task execution.

C2H6O · 2023-03-30T16:15:42Z

@AlecStrong @hfhbd if you don't have any objections, I'll merge this in and will revisit writing a unit test for this in the upcoming weeks. I spent some time on it yesterday but I haven't succeeded in writing a functional test and since there aren't any existing tests that do that (configure the VerifyMigrationTask directly, not through SqlDelightDatabase), it'll take me a bit of time to figure it out.

C2H6O · 2023-04-03T17:58:57Z

sqldelight-gradle-plugin/src/main/kotlin/app/cash/sqldelight/gradle/VerifyMigrationTask.kt

+  fun execute()
+}
+
+class DriverInitializerImpl: DriverInitializer {


Where do I place this class for testing purposes so that it is accessible during a test run?

You could create a gradle project in sqldelight-gradle-plugin/src/test, just copy an existing test project and add the test executor here too: https://github.com/cashapp/sqldelight/blob/68a4d849d234b0018c2c2b314915952564ecfe2b/sqldelight-gradle-plugin/src/test/kotlin/app/cash/sqldelight/tests/MigrationTest.kt

Oh okay, you already added it the project. This depends on your Gradle setup, for tests you could simple move this class into build.gradle. This should work, I think.

yea, just having it live in the build.gradle file should work

It looks like VerifyMigrationTask needs the DriverInitializerImpl on its classpath, since now I'm getting ClassNotFoundException

How do I add the classpath of the build.gradle file to the VerifyMigrationTask? 🤔

@hfhbd any Gradle tips for getting this working? I'm kind of running out of ideas

Yeah, I took a look and played with different implementations. The only one I got to run is using service loaders. Other implementations failed with serialization errors.

I created a PR for this PR, because I can't push to this branch: https://github.com/plangrid/sqldelight/pull/2/files

C2H6O · 2023-04-03T18:17:13Z

@AlecStrong @hfhbd I added a test, curious if you have some ideas for placing the DriverInitializerImpl class?

…properties Use service loader mechanism

hfhbd

Revert these changes

C2H6O · 2023-04-05T20:00:58Z

@hfhbd Yeah, I'll push a commit that reverts the gradle.properties changes and I believe I need to bring back the connectionProperties that I can then pass to my Driver

hfhbd · 2023-04-05T20:15:35Z

@C2H6O What properties do you want to pass? You can always use environment variables.

C2H6O · 2023-04-05T20:50:19Z

@hfhbd I need the root path of the top-level project. When TokenizingDriver gets initialized/instantiated by the ClassLoader of the gradle's worker, the driver needs access to that path. How would that (using environment variables) work in this case?

C2H6O · 2023-04-05T21:24:33Z

@AlecStrong @hfhbd I tested the PR as it is now and it works well for our use-case. I'm not sure about implementation -> api change that we made, but if you guys are cool with it then I'd love to merge this in!

gradle.properties

sqldelight-gradle-plugin/src/main/kotlin/app/cash/sqldelight/gradle/VerifyMigrationTask.kt

hfhbd · 2023-04-06T05:49:54Z

Please run spotlessApply to merge it.

C2H6O · 2023-04-06T15:32:25Z

@hfhbd I ran it but it didn't produce a diff

C2H6O · 2023-04-11T00:27:05Z

@hfhbd @AlecStrong anything else that I should address in this PR? 🙏

hfhbd · 2023-04-11T07:13:12Z

The not required run of spotless is interesting 🤔 Let's finally merge it and do it in another PR.

Allow passing properties to a migration driver

399ba49

C2H6O commented Mar 21, 2023

View reviewed changes

sqldelight-gradle-plugin/src/main/kotlin/app/cash/sqldelight/gradle/VerifyMigrationTask.kt Outdated Show resolved Hide resolved

spotlessApply

2d449c6

C2H6O commented Mar 22, 2023

View reviewed changes

sqldelight-gradle-plugin/src/main/kotlin/app/cash/sqldelight/gradle/VerifyMigrationTask.kt Outdated Show resolved Hide resolved

hfhbd self-requested a review March 22, 2023 16:52

C2H6O added 2 commits March 23, 2023 09:31

Merge branch 'master' into lx/migration_driver_connection_properties

f48896e

Merge branch 'master' into lx/migration_driver_connection_properties

3e782d9

C2H6O mentioned this pull request Mar 24, 2023

Bring back ability to disable ClassLoader isolation #3654

Closed

C2H6O added 3 commits March 24, 2023 08:22

Merge branch 'master' into lx/migration_driver_connection_properties

a5c4b49

Add findFirst() to actually load the Driver class

bd3ef8e

Merge branch 'master' into lx/migration_driver_connection_properties

693b3a2

hfhbd reviewed Mar 26, 2023

View reviewed changes

sqldelight-gradle-plugin/src/main/kotlin/app/cash/sqldelight/gradle/SqlDelightDatabase.kt Outdated Show resolved Hide resolved

C2H6O and others added 3 commits March 27, 2023 15:30

Merge branch 'master' into lx/migration_driver_connection_properties

a879695

Modify VerifyMigrationTask to accept connectionProperties and loading…

7991af0

… of VerifyMigrationDriver

remove connectionProperties from SqlDelightDatabase

bf6ac56

AlecKazakova reviewed Mar 27, 2023

View reviewed changes

C2H6O added 2 commits March 28, 2023 08:23

different strategy with DriverInitializer

d97e81a

fixups

037a269

C2H6O changed the title ~~Allow passing properties to a migration driver~~ Allow passing DriverInitializer to VerifyMigrationTask Mar 28, 2023

Merge branch 'master' into lx/migration_driver_connection_properties

1617857

AlecKazakova approved these changes Mar 28, 2023

View reviewed changes

Test DriverInitializerImpl

0a954cb

C2H6O commented Apr 3, 2023

View reviewed changes

C2H6O and others added 5 commits April 3, 2023 13:40

ClassNotFoundException

e75a3bf

forgotten rename

a8824c1

Use service loader mechanism

cb9df45

Expose the configuration name

c8dd79f

Merge pull request #2 from cashapp/hfhbd/migration_driver_connection_…

6c5a53f

…properties Use service loader mechanism

hfhbd reviewed Apr 5, 2023

View reviewed changes

connectionProperties

4e14be9

hfhbd reviewed Apr 5, 2023

View reviewed changes

gradle.properties Outdated Show resolved Hide resolved

sqldelight-gradle-plugin/src/main/kotlin/app/cash/sqldelight/gradle/VerifyMigrationTask.kt Outdated Show resolved Hide resolved

C2H6O added 2 commits April 5, 2023 16:03

pass the Properties to DriverInitializer#execute

cf7a0d3

change execute of impl to include Properties

f028b6b

C2H6O changed the title ~~Allow passing DriverInitializer to VerifyMigrationTask~~ Allow registering DriverInitializer for VerifyMigrationTask with ServiceLoader mechanism Apr 6, 2023

hfhbd merged commit b9c004f into sqldelight:master Apr 11, 2023

hfhbd mentioned this pull request Apr 11, 2023

Bump ktlint to 0.48.2 #4049

Merged

C2H6O deleted the lx/migration_driver_connection_properties branch April 11, 2023 17:46

nwagu mentioned this pull request Nov 19, 2024

Execute DriverInitializer before GenerateSchemaTask #5562

Merged

Allow registering DriverInitializer for VerifyMigrationTask with ServiceLoader mechanism #3986

Allow registering DriverInitializer for VerifyMigrationTask with ServiceLoader mechanism #3986

Conversation

C2H6O commented Mar 21, 2023 • edited Loading

C2H6O commented Mar 22, 2023

AlecKazakova commented Mar 26, 2023

hfhbd commented Mar 26, 2023

C2H6O commented Mar 27, 2023 • edited Loading

AlecKazakova Mar 27, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

AlecKazakova Mar 28, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

AlecKazakova commented Mar 28, 2023

hfhbd commented Mar 28, 2023

AlecKazakova commented Mar 28, 2023

C2H6O commented Mar 28, 2023

C2H6O commented Mar 28, 2023

AlecKazakova left a comment

Choose a reason for hiding this comment

hfhbd commented Mar 28, 2023

C2H6O commented Mar 29, 2023

hfhbd commented Mar 29, 2023

C2H6O commented Mar 30, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hfhbd Apr 4, 2023 • edited Loading

Choose a reason for hiding this comment

C2H6O commented Apr 3, 2023

hfhbd left a comment

Choose a reason for hiding this comment

C2H6O commented Apr 5, 2023

hfhbd commented Apr 5, 2023

C2H6O commented Apr 5, 2023

C2H6O commented Apr 5, 2023

hfhbd commented Apr 6, 2023

C2H6O commented Apr 6, 2023

C2H6O commented Apr 11, 2023

hfhbd commented Apr 11, 2023

C2H6O commented Mar 21, 2023 •

edited

Loading

C2H6O commented Mar 27, 2023 •

edited

Loading

AlecKazakova Mar 27, 2023 •

edited

Loading

AlecKazakova Mar 28, 2023 •

edited

Loading

hfhbd Apr 4, 2023 •

edited

Loading