Support evolutions scripts prefixed with zeros (01.sql, 001.sql, etc.) #7978

mkurz · 2017-11-01T23:26:24Z

The order of precedence which file is chosen is 1.sql if it exists, otherwise 01.sql if it exists, otherwise 001.sql and so on - until 000000000001.sql.

Can be backported because it's backwards compatible.

marcospereira · 2017-11-02T00:15:49Z

framework/src/play-jdbc-evolutions/src/main/scala/play/api/db/evolutions/EvolutionsApi.scala

  def loadResource(db: String, revision: Int) = {
-    environment.getExistingFile(Evolutions.fileName(db, revision)).map(f => java.nio.file.Files.newInputStream(f.toPath)).orElse {
-      environment.resourceAsStream(Evolutions.resourceName(db, revision))
+    val revisionPadded = List.tabulate(15)(s"${revision}".reverse.padTo(_, "0").reverse.mkString).distinct // 1, 01, 001, ... 000000000001


List.tabulate(15)(n => List.fill(n)(0).mkString + revision)

Looks simpler to me.

I didn't change this because with your approach we always would add a fixed amount of zeros in front. E.g. if we have revision 7 and 1435 the existence of files with up until following amount of zeros would be checked:

000000000000007.sql 000000000000001435.sql

I think however we just should check up until a fixed total file name length and that is what my padding approach is doing:

00000000000007.sql 00000000001435.sql

Otherwise (probably very very rare cases anyway) people might wonder why 000000000000001435.sql works (same like above) but 0000000000000007.sql doesn't (one zero more than above).

Got it.

I think you can then try:

List.tabulate(15 - revision.toString.length)(n => List.fill(n)(0).mkString + revision)

If we're scanning for the first file that matches, then something like this reads better to me:

def loadResource(db: String, revision: Int): Option[InputStream] = { @tailrec def findPaddedRevisionFile(paddedRevision: String): Option[InputStream] { if (paddedRevision.length > 15) { None // Revision string has reached max padding } else { { // Try a file on the filesystem val filename: String = Evolutions.fileName(db, paddedRevision) environment.getExistingFile().map(file => java.nio.file.Files.newInputStream(file.toPath)) } orElse { // Try a resource on the classpath val resourceName: String = Evolutions.resourceName(db, paddedRevision) environment.resource(resourceName) } match { case None => // Add an extra "0" to the padding findPaddedRevisionFile("0"+revisionString) case someStream@Some(_) => // Found something! someStream } } } findPaddedRevisionFile(revision.toString) }

marcospereira · 2017-11-02T00:16:13Z

framework/src/play-jdbc-evolutions/src/main/scala/play/api/db/evolutions/EvolutionsApi.scala

-      environment.resourceAsStream(Evolutions.resourceName(db, revision))
+    val revisionPadded = List.tabulate(15)(s"${revision}".reverse.padTo(_, "0").reverse.mkString).distinct // 1, 01, 001, ... 000000000001
+
+    revisionPadded.flatMap(revision => environment.getExistingFile(Evolutions.fileName(db, revision))).find(_ != None).map(f => java.nio.file.Files.newInputStream(f.toPath)).orElse {


find(_ != None) -> find(_.isDefined).

@marcospereira Also I did not update this because I use flatMap so _ could be just the File itself (which of course doesn't have a isDefined method)

@mkurz Got it too. You won't need the find then since flatMap removes the Nones for you (it is like flattening an empty list). For example:

val a = List(1, 2, 3, 4, 5) numbers.flatMap { case n if n % 2 == 0 => Some(n) case n => None }

Results in List(2, 4) and not List(None, 2, None, 4, None). So, no need to find. Just use headOption. Finally, maybe we can iterate over the list once like this:

revisionPadded.flatMap { revision => environment.getExistingFile(Evolutions.fileName(db, revision)) match { case Some(file) => Option(java.nio.file.Files.newInputStream(file.toPath)) case None => environment.resource(Evolutions.resourceName(db, revision)).map(_.openStream()) } }.headOption

@marcospereira If I read your suggested code (to iterate just once) right, it would mean that we would check back and forth between file and resource. Priority in your code would be:
1.sql file > 1.sql resource > 01.sql file > 01.sql resource > 001.sql file > 001.sql resource > .... > 00000000000001.sql file > 00000000000001.sql resource

Whereas priority in my code would be:
1.sql file > 01.sql file > 001.sql file > 0001.sql file > ... > 00000000000001.sql file > 1.sql resource > 01.sql resource > 001.sql resource > 0001.sql resource > ... > 00000000000001.sql resource

So my code checks all files of a revision before even starting looking into a resource for that revision.

I don't say mine is better, just questioning what is the better approach? WDYT? Does it matter somehow?

Hi @mkurz, sorry for taking so long to reply here.

What is the better approach? WDYT? Does it matter somehow?

I think it would be better to go file > resource for each iteration. That looks closer to the existing behavior to me, but I don't have a strong opinion about this. Trying all files and later the resources also sound reasonable to me too because I can't envision a case where users will have both file 01.sql (tried first in your approach) and resource 1.sql (tried after all files in your approach).

WDYT?

gmethvin · 2017-11-02T05:46:15Z

Since we're changing this, I wonder if we could also add the ability to use other names? see #6919.

If you had the ability to use arbitrary names, we could have something like:

001-create-foos-table.sql
002-add-foos-bar-field.sql
...

Is there a real advantage to only allowing numbers?

eximius313 · 2017-11-02T11:16:08Z

@gmethvin the advantage is to have at least partial solution now - #6919 was raised almost year ago

marcospereira · 2017-11-02T13:50:51Z

Rails migrations have a sustainable and simple solution in my opinion:

Migrations are stored as files in the db/migrate directory, one for each migration class. The name of the file is of the form YYYYMMDDHHMMSS_create_products.rb, that is to say a UTC timestamp identifying the migration followed by an underscore followed by the name of the migration.

They are sortable, have good information about when the migration was created, and have a clear name as suggested in #6919. So I would be more inclined to have something like this instead.

@mkurz, how would this PR handle a case where we have the following files:

evolutions
|_ 001.sql
|_ 0010.sql
|_ 010.sql
|_ 0100.sql

This is obviously a mistake, but that is hard to make today since the numbers are plain incremental.

WDYT?

mkurz · 2017-11-03T19:06:06Z

Since we're changing this, I wonder if we could also add the ability to use other names?
Is there a real advantage to only allowing numbers?

They are sortable, have good information about when the migration was created, and have a clear name as suggested in #6919. So I would be more inclined to have something like this instead.

There is no real advantage in only allowing numbers, that's just how it was implemented in Play starting with Play 1 and never got touched anymore. We could (and probably should) allow the possibility to use other names. Turns out this is possible already by implementing your own EvolutionsReader, however it would be nicer if Play comes with various built-in EvolutionsReaders so someone could just switch on the YYYYMMDDHHMMSS_create_products format by disabling the default "number" reader and enabling that specific built-in one.
However for backward compatibility reasons I wouldn't change the default reader. IMHO providing built-in readers should do.
Also I wouldn't make that part of this pull request. This pr is just about enhancing the current default EvolutionsReader.

@marcospereira

evolutions
|_ 001.sql // revision 1
|_ 0010.sql // revision 10, but ignored because 010.sql (see next line) has priority
|_ 010.sql // revision 10, chosen over 0010.sql because it has fewer leading zeros
|_ 0100.sql // revision 100

If you really would have an evolutions folder like this (with exactly that files) only 001.sql would run right now, until you add revision 2-9, then also 010.sql would run (but 0010.sql wouldn't), then you would have to add revision 11-99 so that 0100.sql would run. This isn't any different like it works right now.
Giving 010.sql priority over 0010.sql guarantees backward compatibility because fewer zeros wins (10.sql having highest priority in that case).

This is obviously a mistake, but that is hard to make today since the numbers are plain incremental.

That would also back up my idea of having different evolution readers built-in and actived by users so they don't mix different approaches.

Pull request updated, ready to reviewed again.

mkurz · 2017-11-03T19:13:27Z

Actually the format

001-create-foos-table.sql
002-add-foos-bar-field.sql

could be supported by the current default evolution reader since there is just a random string after the number (which doesn't influence ordering), however the format

YYYYMMDDHHMMSS_create_products.sql

definitely would need it's own EvolutionsReader because it's not compatible with the number format one.

However I will not add support for the former format to this pull request.

marcospereira · 2017-11-06T20:48:32Z

however it would be nicer if Play comes with various built-in EvolutionsReaders so someone could just switch on the YYYYMMDDHHMMSS_create_products format by disabling the default "number" reader and enabling that specific built-in one.

I have another opinion here: to me Play needs to have a good (opinionated) default and be extensible. Right now I think we are extensible with a not so good default (just incremental numbers). Given that and the fact we need to offer a transition, I would pick the YYYYMMDDHHMMSS_create_products.sql format as the default for new since:

It does not have to count/pad zeros at all
Has clear ordering
Has information information about when the evolution was created
Has an human name part.

And, as I said, users can replace this if they need/want. My intent here is to keep Play itself smaller, with the possibility to have a bigger ecosystem evolved by the community.

If you really would have an evolutions folder like this (with exactly that files) only 001.sql would run right now, until you add revision 2-9, then also 010.sql would run (but 0010.sql wouldn't), then you would have to add revision 11-99 so that 0100.sql would run. This isn't any different like it works right now.

Sorry for not taking proper time to better explain this, @mkurz. I was trying to draw the same scenario you did:

evolutions
|_ 001.sql
|_ 002.sql
|_ 003.sql
|_ ...
|_ 0010.sql
|_ 010.sql
|_ 011.sql
|_ 012.sql
|_ ...
|_ 099.sql
|_ 0100.sql

In this case 0010.sql is just ignored? Without a warning?

eximius313 · 2017-11-07T00:00:26Z

YYYYMMDDHHMMSS_create_products.sql looks like an overkill for many situations...
How about two formats: extended (with date) and simple (with padding) which is backwards compatibile?

mkurz · 2017-11-18T09:30:46Z

@marcospereira @richdougherty Thank you for your comments!

I had a deeper look into this issue and it eventually turned out that we actually do not need to check for a file - checking for resources on the classpath is all it needs:
Checking for a file in addition to a resource on the classpath was added a very very long time ago, at the beginning of Play 2 - because of "Better evolutions auto-reload in DEV mode": cbc542d. This file checking was probably added because in DEV mode the /conf folder was not on the classpath back then. As you know it is now, therefore I just can not see the need to check for a file at all - it is just redundant work. The resource() methods even states "The conf directory is included on the classpath, so this may be used to look up resources, relative to the conf directory". So again, this is handled by calling resource(), no need for extra file checking.
In each mode, DEV or PROD via staging, the evolutions are on the classpath. When staging and PlayKeys.externalizeResources := false they are in the generated jar, when true they are in the conf folder of the distribution which is also on the classpath.

I am pretty sure this file checking is just a leftover from long time ago and therefore I removed it. This doesn't change behaviour at all, also not in production (Again, this file checking was just introduced for DEV mode, which is working now, it was never targeting prod mode.) I am almost certainly 100% sure about that 😉

About the implementation:
I choose the approach suggested by @richdougherty using a recursion. However I customized the method a bit, so we always check all possible file names, so we can log warnings in case a file already was chosen and overrules others that will be therefore be ignored, as wished by @marcospereira 😉

I also created a test project to test all of this: https://github.com/mkurz/play-evolutions-padded/ (based on a 2.7.0-SNAPSHOT)
@marcospereira As you can see I added various evolutions scripts that should get ignored. Each evolution script that actually runs writes it's file name into a applied_evolutions_log table, which content I return when accessing the index action. And that is what it says:

Following evolutions have been applied:
---------------------------------------
001.sql
002.sql
003.sql
004.sql
005.sql
006.sql
7.sql
008.sql
009.sql
010.sql
011.sql
012.sql
00013.sql
014.sql

In the log you get following warnings:

[warn] p.a.d.e.DefaultEvolutionsApi - Ignoring evolution script 07.sql, using 7.sql instead already
[warn] p.a.d.e.DefaultEvolutionsApi - Ignoring evolution script 007.sql, using 7.sql instead already
[warn] p.a.d.e.DefaultEvolutionsApi - Ignoring evolution script 0010.sql, using 010.sql instead already
[warn] p.a.d.e.DefaultEvolutionsApi - Ignoring evolution script 0000013.sql, using 00013.sql instead already
[warn] p.a.d.e.DefaultEvolutionsApi - Ignoring evolution script 000000013.sql, using 00013.sql instead already
[warn] p.a.d.e.DefaultEvolutionsApi - Ignoring evolution script 00000000000013.sql, using 00013.sql instead already

So please check again, I think this is done and can be merged 😄

mkurz · 2017-11-18T09:39:06Z

BTW @marcospereira

I have another opinion here: to me Play needs to have a good (opinionated) default and be extensible. Right now I think we are extensible with a not so good default (just incremental numbers). Given that and the fact we need to offer a transition, I would pick the YYYYMMDDHHMMSS_create_products.sql format as the default for new

Sure we can do that. We just need to implement the YYYYMMDDHHMMSS_create_products EvolutionsReader and make it default and mention it in the 2.7 migration guide.
However that should happen in a new pull request 😉
(You say "... and the fact we need to offer a transition" -> for this pull request here we do not need to offer a transition, the padding stuff is fully backward compatible)

marcospereira

Thanks for your patience here, @mkurz.

This LGTM, but I think we can have some tests here.

You can even test the logging warning by using play.api.libs.logback.LogbackCapturingAppender. See play.api.ModeSpecificLoggerSpec for an example.

mkurz · 2017-11-24T15:52:52Z

Alright, I will have a look.

mkurz · 2017-12-03T21:51:19Z

@marcospereira Done - tests added. Please have a look, thanks!

richdougherty · 2017-12-03T23:59:43Z

...rk/src/play-jdbc-evolutions/src/test/scala/play/api/db/evolutions/EvolutionsReaderSpec.scala

+        "Ignoring evolution script 002.sql, using 2.sql instead already",
+        "Ignoring evolution script 005.sql, using 05.sql instead already",
+        "Ignoring evolution script 0010.sql, using 010.sql instead already"
      )


I'm always impressed when I see logging tests!

richdougherty

LGTM

mkurz · 2017-12-18T13:54:04Z

@marcospereira I think this pr is also waiting for your approval 😉

marcospereira

LGTM.

Thank you, @mkurz. And sorry for taking so long to look back at this PR.

mkurz · 2018-01-09T18:15:24Z

@marcospereira Thanks!

Support evolutions scripts prefixed with zeros (01.sql, 001.sql, etc.)

17649a9

mkurz added the status:needs-backport label Nov 1, 2017

mkurz mentioned this pull request Nov 1, 2017

Evolutions doesn't work when 0 is at the begining of the file #7976

Closed

marcospereira reviewed Nov 2, 2017

View reviewed changes

mkurz force-pushed the evolutionsPadding branch from ec198e2 to 17649a9 Compare November 3, 2017 19:32

No need for getExistingFile() + recursion + log warning

816857c

marcospereira reviewed Nov 24, 2017

View reviewed changes

Added tests

db6b680

richdougherty reviewed Dec 3, 2017

View reviewed changes

richdougherty approved these changes Dec 4, 2017

View reviewed changes

marcospereira approved these changes Jan 9, 2018

View reviewed changes

marcospereira merged commit d913616 into playframework:master Jan 9, 2018

mkurz deleted the evolutionsPadding branch January 9, 2018 18:15

mkurz mentioned this pull request Jan 9, 2018

[2.6.x] Support evolutions scripts prefixed with zeros #8142

Merged

marcospereira added type:improvement topic:evolutions labels Jan 11, 2018

marcospereira removed the status:needs-backport label Jan 11, 2018

Uh oh!

Support evolutions scripts prefixed with zeros (01.sql, 001.sql, etc.) #7978

Support evolutions scripts prefixed with zeros (01.sql, 001.sql, etc.) #7978

Uh oh!

Conversation

mkurz commented Nov 1, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

gmethvin commented Nov 2, 2017

Uh oh!

eximius313 commented Nov 2, 2017

Uh oh!

marcospereira commented Nov 2, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mkurz commented Nov 3, 2017

Uh oh!

mkurz commented Nov 3, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

marcospereira commented Nov 6, 2017

Uh oh!

eximius313 commented Nov 7, 2017

Uh oh!

mkurz commented Nov 18, 2017

Uh oh!

mkurz commented Nov 18, 2017

Uh oh!

marcospereira left a comment

Choose a reason for hiding this comment

Uh oh!

mkurz commented Nov 24, 2017

Uh oh!

mkurz commented Dec 3, 2017

Uh oh!

Choose a reason for hiding this comment

Uh oh!

richdougherty left a comment

Choose a reason for hiding this comment

Uh oh!

mkurz commented Dec 18, 2017

Uh oh!

marcospereira left a comment

Choose a reason for hiding this comment

Uh oh!

mkurz commented Jan 9, 2018

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

mkurz commented Nov 1, 2017 •

edited

Loading

marcospereira commented Nov 2, 2017 •

edited

Loading

mkurz commented Nov 3, 2017 •

edited

Loading