bpo-27645: Supporting native backup facility of SQLite #377

lelit · 2017-03-01T09:26:51Z

This is a set of patches (rebased on current master) that adds a new backup() method on sqlite3.Connection.

https://bugs.python.org/issue27645

…s do

Suggested by Aviv Palivoda.

the-knights-who-say-ni · 2017-03-01T09:26:52Z

Hello, and thanks for your contribution!

I'm a bot set up to make sure that the project can legally accept your contribution by verifying you have signed the PSF contributor agreement (CLA).

Unfortunately we couldn't find an account corresponding to your GitHub username on bugs.python.org (b.p.o) to verify you have signed the CLA. This is necessary for legal reasons before we can look at your contribution. Please follow these steps to help rectify the issue:

If you don't have an account on b.p.o, please create one
Make sure your GitHub username is listed in "Your Details" at b.p.o
If you have not already done so, please sign the PSF contributor agreement. The "bugs.python.org username " requested by the form is the "Login name" field under "Your Details".
If you just signed the CLA, please wait at least one US business day and then check "Your Details" on bugs.python.org to see if your account has been marked as having signed the CLA (the delay is due to a person having to manually check your signed CLA)
Reply here saying you have completed the above steps

Thanks again to your contribution and we look forward to looking at it!

mention-bot · 2017-03-01T09:26:54Z

@lelit, thanks for your PR! By analyzing the history of the files in this pull request, we identified @berkerpeksag, @zware, @gvanrossum, @avassalotti and @orsenthil to be potential reviewers.

lelit · 2017-03-01T09:31:50Z

I configured my GitHub username on b.p.o.

berkerpeksag · 2017-03-01T09:32:11Z

Doc/library/sqlite3.rst

@@ -521,6 +521,34 @@ Connection Objects
                 f.write('%s\n' % line)


+   .. method:: backup(filename[, pages, progress])


I'd like to make pages and progress keyword-only.

berkerpeksag · 2017-03-01T09:32:37Z

Doc/library/sqlite3.rst

@@ -521,6 +521,34 @@ Connection Objects
                 f.write('%s\n' % line)


+   .. method:: backup(filename[, pages, progress])
+
+      This method exposes the `API`__ that allows to make a backup of a SQLite


No need to add a link to SQLite docs.

berkerpeksag · 2017-03-01T09:33:48Z

Doc/library/sqlite3.rst

+
+         con = sqlite3.connect('existing_db.db')
+         con.backup('copy_of_existing_db.db', 1, progress)
+


Needs:

.. versionadded:: 3.7

berkerpeksag · 2017-03-01T09:37:01Z

Doc/library/sqlite3.rst

+      database into the mandatory argument *filename*, even while it's being accessed
+      by other clients, or concurrently by the same connection.
+
+      By default, or when *pages* is either ``0`` or a negative integer, the entire


I wonder if we really need to expose this. The default value should cover most of the use cases.

Do you mean that I should remove the pages argument? But if I do that, also the progress one becomes pointless, because with the default value there will be at most one callback, when the engine completes the backup. What am I missing?

I think it's fine to keep pages for huge databases.

berkerpeksag · 2017-03-01T09:40:45Z

Misc/NEWS

@@ -34,6 +36,9 @@ Core and Builtins

 - bpo-29546: Improve from-import error message with location

+- Issue #7063: Remove dead code from the "array" module's slice handling.


This needs to be deleted.

berkerpeksag · 2017-03-01T09:43:10Z

Modules/_sqlite/connection.c

+    }
+
+    Py_BEGIN_ALLOW_THREADS
+    rc = sqlite3_open(filename, &bckconn);


#359 could be useful here.

Not sure what you are suggesting: should I take the same approach of conditionally using sqlite3_open_v2() instead?

I was trying to say that this should be merged after #359.

This don't actually have any effect. sqlite3_open(filename, db) is the same as sqlite3_open_v2(filename, db, SQLITE_OPEN_READWRITE | SQLITE_OPEN_CREATE, NULL)

berkerpeksag · 2017-03-01T09:45:15Z

Misc/NEWS

@@ -10,6 +10,8 @@ What's New in Python 3.7.0 alpha 1?
 Core and Builtins
 -----------------

+- bpo-27645: sqlite3.Connection now exposes a backup() method.
+


Add "Patch by Your Name." to the next line.

berkerpeksag · 2017-03-01T09:46:41Z

Lib/test/test_sqlite.py

@@ -18,7 +18,7 @@ def load_tests(*args):
                               userfunctions.suite(),
                               factory.suite(), transactions.suite(),
                               hooks.suite(), regression.suite(),
-                               dump.suite()])
+                               dump.suite(), backup.suite()])


Nitpick: Add this to its own line so we won't clutter git blame next time we add something.

berkerpeksag · 2017-03-01T09:47:44Z

Doc/library/sqlite3.rst

+         import sqlite3
+
+         def progress(remaining, total):
+             print("Copied %d of %d pages..." % (total-remaining, total))


We can use f-strings here.

…f7861361d

lelit · 2017-03-01T10:57:33Z

Thank you @berkerpeksag!

palaviv · 2017-03-01T20:30:46Z

Modules/_sqlite/connection.c

+            /* Sleep for 250ms if there are still further pages to copy */
+            if (rc == SQLITE_OK || rc == SQLITE_BUSY || rc == SQLITE_LOCKED) {
+                Py_BEGIN_ALLOW_THREADS
+                sqlite3_sleep(250);


Are we sure we need to sleep when SQLITE_OK is returned? I know this is what is done in the example but we do allow other threads to take the lock between calls to sqlite3_backup_step.
In addition we might want to have some kind of timeout here. We can block forever if someone holds the db lock.

Yes, I think we should: the step function returns OK when it “successfully copies N pages and there are still more pages to be copied”.
Not sure about the timeout: maybe we could check whether there is no progress in, say, X calls to the step function?

Reconsidering, I think you're right here: there's no reason to delay next iteration when the result was SQLITE_OK.

palaviv · 2017-03-01T20:32:18Z

Modules/_sqlite/connection.c

+            rc = sqlite3_backup_step(bckhandle, pages);
+            Py_END_ALLOW_THREADS
+
+            if (progress != Py_None) {


We should call progress only when sqlite3_backup_step return SQLITE_OK

Not sure about that: maybe a better option would be pass the result status to the progress callback, so that it can eventually signal the error and do some cleanup (think about closing a progress dialog).

Now the progress callback receives three arguments.

palaviv · 2017-03-01T20:36:01Z

Modules/_sqlite/connection.c

+        Py_INCREF(Py_None);
+        retval = Py_None;
+    } else {
+        /* TODO: should the (probably incomplete/invalid) backup be removed here? */


I think you should delete the backup

Ok, will do that.

Thanks a lot for the review!

I need some help here: I expected to find an abstract/portable PyOS_UnlinkFile(), but couldn't find it.
What is the recommended idiom to remove a file from C? Is the only way to "import" the os module and then PyObject_CallFunction() on its unlink member?

I took the simplest way for now, using unlink(2) from unistd.h. Any hint for a better way is welcome.

palaviv · 2017-03-01T20:37:02Z

Modules/_sqlite/connection.c

+                                           sqlite3_backup_pagecount(bckhandle))) {
+                    /* User's callback raised an error: interrupt the loop and
+                       propagate it. */
+                    rc = -1;


You should add a test for the case that progress raise an error

Ok, will do!

palaviv · 2017-03-01T20:51:51Z

Modules/_sqlite/connection.c

+    }
+
+    if (rc != -1) {
+        rc = _pysqlite_seterror(bckconn, NULL);


I am not sure about this but from the documentation sqlite3_backup_step will not set an error code on error but will return one. I can see that in the example they do the same as here but I am not sure if this is the right behavior. Would we receive SQLITE_NOMEM error from sqlite3_backup_step?

Will try to understand the point.

I tried to address this concern in my latest patches.

palaviv · 2017-03-01T20:56:11Z

Modules/_sqlite/connection.c

+    }
+
+    Py_BEGIN_ALLOW_THREADS
+    bckhandle = sqlite3_backup_init(bckconn, "main", self->db, "main");


Shouldn't database name be a parameter?

That's not really the database name: accordingly to the documentation that name denotes the target, that can either be "main" for the main storage, "temp" for the temporary storage or the name of the attached database.
Of course we can easily expose that as a (default) argument, but I did not see as necessary when I coded the feature.

I added a source_name and a dest_name parameters.

Uhm, further investigation seems to reveal that the dest_name is pointless: attached databases are not a persistent property of a database, so that we cannot allow the following:

bck = sqlite.connect('/tmp/thebackup') bck.execute(f"ATTACH DATABASE '/tmp/theattachment' AS attached_db") bck.commit() db = sqlite.connect('/tmp/database') db.backup('/tmp/thebackup', dest_name='attached_db')

Changed back to a single name argument to specify the source database...

Suggested by Aviv Palivoda.

When sqlite3_backup_step() returns OK it means that it was able to do it's work but there are further pages to be copied: in such case it's better to immediately start the next iteration, without opening a window that allows other threads to go on and possibly locking the database. Suggested by Aviv Palivoda.

berkerpeksag · 2017-03-03T12:10:51Z

Modules/_sqlite/connection.c

@@ -1478,6 +1478,94 @@ pysqlite_connection_iterdump(pysqlite_Connection* self, PyObject* args)
 }

 static PyObject *
+pysqlite_connection_backup(pysqlite_Connection* self, PyObject* args, PyObject* kwds)


If I read SQLite release notes correctly, sqlite3_backup_init has been added in 3.6.11 so we need to expose this only if SQLite version is 3.6.11 or newer.

Ok, is it better to omit the whole function or rather raise an NotImplementedError?

Let's omit the whole function. This is what we do for other version specific features in the stdlib. Please also update the documentation to note that this feature is available from SQLite 3.6.11.

Ok, done. Thank you!

berkerpeksag · 2017-03-03T12:19:15Z

Lib/sqlite3/test/backup.py

+from tempfile import NamedTemporaryFile
+import unittest
+
+class BackupTests(unittest.TestCase):


There should be a @unittest.skipIf(...) decorator to skip the whole test case if SQLite version is not 3.6.11 or newer.

Please add tests to make sure that keyword-only parameters are really keyword-only:

self.cx.backup(bckfn.name, 1, lambda x, y: None) # should raise TypeError

….6.11

palaviv · 2017-03-03T16:23:06Z

Doc/library/sqlite3.rst

+      database is copied in a single step; otherwise the method performs a loop
+      copying up to the specified *pages* at a time.
+
+      If *progress* is specified, it must either ``None`` or a callable object that


either be ``None``

…llback Add new SQLITE_DONE integer constant to the module, that could be used by the callback to determine whether the backup is terminated.

This should address Aviv's concerns about how the sqlite3_backup_step() errors are handled.

…file

lelit · 2017-03-24T10:27:34Z

Is there anything else I could/should do to see this accepted?

dgilman · 2017-09-14T03:28:07Z

Doc/library/sqlite3.rst

+
+      This method makes a backup of a SQLite database into the mandatory argument
+      *filename*, even while it's being accessed by other clients, or concurrently by
+      the same connection.


Maybe mention that backups work for on-disk and in-memory databases like the sqlite3 C api docs mention?

lelit · 2017-10-05T19:53:07Z

FYI, I rebased these commits on top of current master: https://github.com/lelit/cpython/tree/sqlite-backup-api-v3

lelit · 2017-10-24T10:36:58Z

@Mariatta I'm confused by that "added needs rebase and removed needs rebase, what should I do?
Also, given that current implementation (rebased on master of three weeks ago) lives in a different branch (https://github.com/lelit/cpython/compare/sqlite-backup-api-v3) should I do something wrt this PR?

Mariatta · 2017-10-30T19:37:52Z

@lelit We can only review PRs in CPython repo. Currently this PR has conflict. If the conflict is resolved in a different branch, maybe you can create the PR for the other branch?
Although I would prefer that the conflict is resolved in the same branch so we get to keep the past discussions in the same PR.

lelit · 2017-10-30T21:45:16Z

Does that mean you'd prefer me to rebase the branch I opened the PR with?

brettcannon · 2017-11-02T17:25:44Z

@lelit There's a conflict because you edited Misc/NEWS and it will no longer merge cleanly as-is. See https://devguide.python.org/committing/?highlight=news.d#what-s-new-and-news-entries on how to fix this.

lelit · 2017-11-02T17:40:05Z

@brettcannon, I rebased the branch into https://github.com/lelit/cpython/compare/sqlite-backup-api-v3
rewriting the NEWS change using the new tool.

As said in this comment I'm sorry about the fuss, but it's not clear to me whether I should have closed this PR and reopened a new one on the v3 branch, or if instead I should have rewritten the same v2 branch.

As always, I'm willing to do whatever is easier/more correct for you!

brettcannon · 2017-11-02T17:44:47Z

@lelit I'm not sure what 'v2' and 'v3' are referring to (some personal branches in your fork you have?), but if you look at https://github.com/python/cpython/pull/377/files you will see that the branch your PR is based on has not been updated in regards to the NEWS file.

If it's easier, you can just close this PR, open a new one if you have some other branch that's in better shape, and then refer to this old PR in the new PR's opening comment.

lelit · 2017-11-02T17:45:32Z

Ok, doing that immediately.

lelit · 2017-11-02T17:56:25Z

As suggested by Brett, I'm closing this PR, replaced by #4238.

lelit added 5 commits March 1, 2017 10:16

Expose the SQLite Online Backup API as Connection.backup()

d93391d

See issue python#27645.

Preliminary documentation for sqlite3.Connection.backup()

7b1277a

Reduce code nesting depth by using a finally label, as other function…

1258d47

…s do

Wrap SQLite calls between BEGIN_ALLOW_THREADS and END_ALLOW_THREADS

d8ea457

Suggested by Aviv Palivoda.

Add an entry in Misc/NEWS related to issue python#27645

35fa0ad

the-knights-who-say-ni added the CLA not signed label Mar 1, 2017

berkerpeksag requested changes Mar 1, 2017

View reviewed changes

berkerpeksag removed the CLA not signed label Mar 1, 2017

the-knights-who-say-ni added the CLA signed label Mar 1, 2017

lelit added 7 commits March 1, 2017 10:55

Remove extraneous entry introduced in d4439d6c5b2a67682f3db74d395c603…

c1af5a0

…f7861361d

Attribute the patch to myself

c07d982

Make pages and progress keyword-only args in sqlite3.Connection.backup()

cdc7eda

Remove reference to SQLite docs in Connection.backup() documentation

6d3eb69

Use f-string in the sqlite3.Connection.backup() example

6fab17f

Explicitly mark sqlite3.Connection.backup() as added in v3.7

6449f3d

Reduce chances of cluttering future “git blame”

f84d375

palaviv suggested changes Mar 2, 2017

View reviewed changes

lelit added 4 commits March 3, 2017 10:09

Parametrize the name of the database involved in the backup

e748942

Suggested by Aviv Palivoda.

Test propagation of exception raised in backup's progress callback

426dad4

Use a better name for variable

895726b

berkerpeksag reviewed Mar 3, 2017

View reviewed changes

lelit added 2 commits March 3, 2017 14:23

Omit the backup method when underlying SQLite library is older than 3…

71726ba

….6.11

Assert that the non-mandatory arguments are keyword-only

c4fbc20

palaviv reviewed Mar 3, 2017

View reviewed changes

Pass also the current status of the ongoing backup to the progress ca…

ff1609b

…llback Add new SQLITE_DONE integer constant to the module, that could be used by the callback to determine whether the backup is terminated.

lelit added 2 commits March 4, 2017 10:45

Slightly different way handling backup step's error state

d688f20

This should address Aviv's concerns about how the sqlite3_backup_step() errors are handled.

When an error occurs while the backup is going on, remove the target …

4ed6c31

…file

dgilman reviewed Sep 14, 2017

View reviewed changes

Mariatta added needs rebase and removed needs rebase labels Oct 9, 2017

brettcannon closed this Nov 2, 2017

brettcannon reopened this Nov 2, 2017

brettcannon changed the title ~~bpo-27645: Supporting native backup facility of SQLite~~ bpo-27645: Supporting native backup facility of SQLite. Nov 2, 2017

brettcannon changed the title ~~bpo-27645: Supporting native backup facility of SQLite.~~ bpo-27645: Supporting native backup facility of SQLite Nov 2, 2017

lelit mentioned this pull request Nov 2, 2017

bpo-27645: Supporting native backup facility of SQLite #4238

Merged

lelit closed this Nov 2, 2017

python deleted a comment from richmoore1962 Nov 4, 2017

		@@ -521,6 +521,34 @@ Connection Objects
		f.write('%s\n' % line)


		.. method:: backup(filename[, pages, progress])


		con = sqlite3.connect('existing_db.db')
		con.backup('copy_of_existing_db.db', 1, progress)

		@@ -34,6 +36,9 @@ Core and Builtins

		- bpo-29546: Improve from-import error message with location

		- Issue #7063: Remove dead code from the "array" module's slice handling.

bpo-27645: Supporting native backup facility of SQLite #377

bpo-27645: Supporting native backup facility of SQLite #377

Conversation

lelit commented Mar 1, 2017 • edited by bedevere-bot Loading

the-knights-who-say-ni commented Mar 1, 2017

mention-bot commented Mar 1, 2017

lelit commented Mar 1, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lelit commented Mar 1, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

berkerpeksag Mar 3, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lelit commented Mar 24, 2017

Choose a reason for hiding this comment

lelit commented Oct 5, 2017

lelit commented Oct 24, 2017

Mariatta commented Oct 30, 2017

lelit commented Oct 30, 2017

brettcannon commented Nov 2, 2017

lelit commented Nov 2, 2017

brettcannon commented Nov 2, 2017

lelit commented Nov 2, 2017

lelit commented Nov 2, 2017

lelit commented Mar 1, 2017 •

edited by bedevere-bot

Loading

berkerpeksag Mar 3, 2017 •

edited

Loading