Thanks to visit codestin.com
Credit goes to Github.com

Skip to content

Comments

Speed up cache loading during startup#3223

Open
kmws wants to merge 2 commits intonewsboat:masterfrom
kmws:kmws/startup-speedup-1
Open

Speed up cache loading during startup#3223
kmws wants to merge 2 commits intonewsboat:masterfrom
kmws:kmws/startup-speedup-1

Conversation

@kmws
Copy link
Contributor

@kmws kmws commented Dec 3, 2025

I just want to open this PR to have it out of my system, but do not stress about responding/deciding right away. I intend to polish #3219 first.

I've tried to decrease Newsboat's startup time and one idea that I had was to decrease the number of SQL queries to the cache that we currently do. I do it be loading all feeds and (non-deleted) items once in controller::run(). I benchmarked this for two cases:

  • with my personal cache of around 200MiB
  • with fake cache generated (~37MiB) and ran with .py script from High RAM usage #3210

Both scenarios were done with cold cache (vmtouch -e <cache_file> between runs). I observe around 33% speedup for real scenario and ~11x speedup for the fake one. I haven't investigated the high win for fake case, but I suspect the win is also fake. :) I've used ScopeMeasure to count the time.

+--------------------+---------------+--------------+
|                    | BEFORE Change | WITH Change  |
+--------------------+---------------+--------------+
| Fake Cache (1)     |  4.135372 s   |  0.384310 s  |
| Fake Cache (2)     |  4.136365 s   |  0.386113 s  |
| Fake Cache (3)     |  4.161611 s   |  0.386548 s  |
| Fake Cache (Avg)   |  4.144449 s   |  0.385657 s  |
|--------------------|---------------|--------------|
| Real Cache (1)     |  3.014429 s   |  2.047725 s  |
| Real Cache (2)     |  3.080632 s   |  2.087318 s  |
| Real Cache (3)     |  3.094218 s   |  2.040778 s  |
| Real Cache (Avg)   |  3.063093 s   |  2.058607 s  |
+--------------------+---------------+--------------+

The code change feels a bit copy-pasty, but I'm open to polishing it further if we intend to merge this. Let me know what you think.

@coveralls
Copy link

Coverage Status

coverage: 60.707% (+0.1%) from 60.611%
when pulling ef04abf on kmws:kmws/startup-speedup-1
into 68ace6d on newsboat:master.

@juntuu
Copy link
Contributor

juntuu commented Dec 3, 2025

I was once playing around with the sqlite pragmas to see if those could offer faster startup speed.

I tried other options as well, but most notable on my system (Mac M1) was adding this mmap_size. It reduced the time from ~3.7 s to ~2 s on a 94MiB cache file.

There are some caveats to this, so it might not be suitable as a default.

diff --git a/src/cache.cpp b/src/cache.cpp
index 4ea1bfa51..e730165cb 100644
--- a/src/cache.cpp
+++ b/src/cache.cpp
@@ -296,6 +296,9 @@ void Cache::set_pragmas()
        // then we disable case-sensitive matching for the LIKE operator in
        // SQLite, for search operations
        run_sql("PRAGMA case_sensitive_like=OFF;");
+
+        // use mmap with 0.5 GB limit
+       run_sql("PRAGMA mmap_size=536870912;");
 }

@kmws
Copy link
Contributor Author

kmws commented Dec 4, 2025

Yes, I've also looked into the mmap_size trick, but the disadvantages seem scary :D so for now I wanted to fiddle with the code and see what we can get from this approach before delving into more detailed options.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants