I'm looking at your new data dump design, and I don't think it works for our use case, and here's why:
Our database, which is a botanical database, we have more than one. We have one for fungi, one for lichen, moss, one for algae etc. They all have the same data structure, but some tables have different reference data. I think this has got to be a fairly common scenario.
With that in mind, these are my criticisms...
I don't get the philosophy of having config files silently get picked up, just because they're in the same directory. It's not like there is some big saving in command line arguments. The config file should be explicit --config=blah or whatever. That way we could specify a different one for each database type, and not be locked into a one setup per schema model.
I also don't get the idea of a separate exclusion and inclusion config (one hidden and one not?!?!). Why not have [data] include = ["foo", "bar", "baz"] and [schema] exclude = ["foo"]. It is good to be able to split up config over multiple files, --config=foo --config=bar, but I don't see why to have that one way only of splitting it up.
And one is definitely going to want to dump data only. pg_dump has the --data-only flag, people are going to want to (not always, but in some cases) split up the schema from the data dumping, because the schema is common and (some of) the data is not. People are going to want to do data-only dumps, data only plans without churning for 20 minutes doing a full audit of the schema. Unlike the schema, the content of tables is generally pretty independent of anything else.
I also think it's odd that you chose the csv format over the postgres native tab delimited format. For one thing, it forced you into a multi-file dump, when it wasn't necessary. For another thing, postgres distinguishes NULL from empty string, and csv has no native way of encoding that. For another thing, isn't the aim to more or less support people writing their own sql in .sql files, and have pgschema not care how you did it? So if I write COPY commands with tab delmited directly into my sql, or INSERT commands for that matter, pgschema shouldn't be worried, it should just load it and push it into postgres.. But because you've exited SQL now, into CSV, I don't really have that option anymore. With that tool I wrote pgdata, you can dump it as INSERT, or COPY, it makes no difference because whatever is there is just fed into psql. But when the program sees [data] tables=["foo"] you're not going to be looking for data/tablename.sql, you're going to be looking for data/tablename.csv, so you've completely exited the original paradigm of writing SQL to maintain your database. Now if I as my choice, as an individual user want to write COPY... FROM 'blah.csv', sure why not if that's what I want to do. If it was pure sql, I could also do that if I chose. But by forcing CSV, it's not a choice, it's mandatory.
I'm also baffled by the documentation " appends a \copy directive to the schema file. ".. I can see that for a 1 file dump, unless I'm missing it I can't see that in a multi file dump. BTW, without --file it churns there for 15 minutes before deciding it's going to fail due to usage.
I'm looking at your new data dump design, and I don't think it works for our use case, and here's why:
Our database, which is a botanical database, we have more than one. We have one for fungi, one for lichen, moss, one for algae etc. They all have the same data structure, but some tables have different reference data. I think this has got to be a fairly common scenario.
With that in mind, these are my criticisms...
I don't get the philosophy of having config files silently get picked up, just because they're in the same directory. It's not like there is some big saving in command line arguments. The config file should be explicit --config=blah or whatever. That way we could specify a different one for each database type, and not be locked into a one setup per schema model.
I also don't get the idea of a separate exclusion and inclusion config (one hidden and one not?!?!). Why not have [data] include = ["foo", "bar", "baz"] and [schema] exclude = ["foo"]. It is good to be able to split up config over multiple files, --config=foo --config=bar, but I don't see why to have that one way only of splitting it up.
And one is definitely going to want to dump data only. pg_dump has the --data-only flag, people are going to want to (not always, but in some cases) split up the schema from the data dumping, because the schema is common and (some of) the data is not. People are going to want to do data-only dumps, data only plans without churning for 20 minutes doing a full audit of the schema. Unlike the schema, the content of tables is generally pretty independent of anything else.
I also think it's odd that you chose the csv format over the postgres native tab delimited format. For one thing, it forced you into a multi-file dump, when it wasn't necessary. For another thing, postgres distinguishes NULL from empty string, and csv has no native way of encoding that. For another thing, isn't the aim to more or less support people writing their own sql in .sql files, and have pgschema not care how you did it? So if I write COPY commands with tab delmited directly into my sql, or INSERT commands for that matter, pgschema shouldn't be worried, it should just load it and push it into postgres.. But because you've exited SQL now, into CSV, I don't really have that option anymore. With that tool I wrote pgdata, you can dump it as INSERT, or COPY, it makes no difference because whatever is there is just fed into psql. But when the program sees [data] tables=["foo"] you're not going to be looking for data/tablename.sql, you're going to be looking for data/tablename.csv, so you've completely exited the original paradigm of writing SQL to maintain your database. Now if I as my choice, as an individual user want to write COPY... FROM 'blah.csv', sure why not if that's what I want to do. If it was pure sql, I could also do that if I chose. But by forcing CSV, it's not a choice, it's mandatory.
I'm also baffled by the documentation " appends a \copy directive to the schema file. ".. I can see that for a 1 file dump, unless I'm missing it I can't see that in a multi file dump. BTW, without --file it churns there for 15 minutes before deciding it's going to fail due to usage.