You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
After calling the ApplyRigConfig function, the exe may experience long I/O cycles (90% utilization). Investigation revealed that the Database destructor was particularly time-consuming. Further investigation revealed that this was caused by setting auto_vacuum to 1. If auto_vacuum is 0, SQLite does not clean up deleted space when deleting data, allowing it to reuse the deleted space when adding data later.
The ApplyRigConfig function primarily deletes frames and rigs. However, frames and rigs occupy very little storage space, so clearing them would not yield significant benefits.
Changing from
SQLITE3_EXEC(database->database_, "PRAGMA auto_vacuum=1", nullptr);
to
SQLITE3_EXEC(database->database_, "PRAGMA auto_vacuum=0", nullptr);
reduces the time it takes to destroy the database, especially when the SQLite database size reaches 1-10GB.
It also means that storage is not automatically freed up when data is deleted from the database?
Yes.
When auto_vacuum=0, SQLite will reuse storage marked for deletion. This should be sufficient for colmap, especially for feature extraction and feature matching.
What's the runtime savings and corresponding stats (#images/frames/rigs)?
Sorry for the late reply.
I used two datasets for experiments, with the following results:
Data 1, with 309 x 2 images (1 rig, 309 frames, 718 images), a database size of 0.93 GB, reduced ApplyRigConfig time from 30.3 seconds to 0.02 seconds.
Data 2, with 1630 x 2 images (1 rig, 1630 frames, 3260 images), a database size of 6.60 GB, reduced ApplyRigConfig time from 557.5 seconds to 0.02 seconds.
The ApplyRigConfig function primarily deletes frames and rigs. However, frames and rigs occupy very little storage space, so clearing them would not yield significant benefits.
Ideally we would set auto_vacuum=0 only when deleting frames/rigs. There are other use cases in which one might want to remove features from the database - then auto_vacuum=1 helps reduce the database size by a lot.
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
After calling the ApplyRigConfig function, the exe may experience long I/O cycles (90% utilization). Investigation revealed that the Database destructor was particularly time-consuming. Further investigation revealed that this was caused by setting auto_vacuum to 1. If auto_vacuum is 0, SQLite does not clean up deleted space when deleting data, allowing it to reuse the deleted space when adding data later.
The ApplyRigConfig function primarily deletes frames and rigs. However, frames and rigs occupy very little storage space, so clearing them would not yield significant benefits.
Changing from
SQLITE3_EXEC(database->database_, "PRAGMA auto_vacuum=1", nullptr);
to
SQLITE3_EXEC(database->database_, "PRAGMA auto_vacuum=0", nullptr);
reduces the time it takes to destroy the database, especially when the SQLite database size reaches 1-10GB.