Replace vendored SQLite amalgamation with build-time generation.#21779
Open
vkalintiris wants to merge 1 commit intonetdata:masterfrom
Open
Replace vendored SQLite amalgamation with build-time generation.#21779vkalintiris wants to merge 1 commit intonetdata:masterfrom
vkalintiris wants to merge 1 commit intonetdata:masterfrom
Conversation
SQLite was vendored as the full amalgamation (sqlite3.c at ~9 MB) in src/database/sqlite/vendored/. With ~10 updates in git history, this accumulated ~65 MB of uncompressible blob storage, growing by ~9 MB per version bump. This replaces the vendored files with a new CMake module (NetdataSQLite.cmake) that uses ExternalProject_Add to fetch the SQLite source tree and generate the amalgamation at build time with --enable-update-limit (required for the UDL-capable parser grammar). By default, sources are fetched as a tarball from sqlite.org (~14 MB). A -DSQLITE_USE_GIT=ON option is available to use a shallow git clone from GitHub instead.
Contributor
Author
|
@stelfrag I didn't enable git sources by default, because it added >1m for clean builds. |
Contributor
There was a problem hiding this comment.
No issues found across 16 files
Confidence score: 5/5
- Automated review surfaced no issues in the provided summaries.
- No files require special attention.
Architecture diagram
sequenceDiagram
participant Build as Build System (CMake/Make)
participant Remote as SQLite Sources (sqlite.org / GitHub)
participant FS as Build Directory (Binary Tree)
participant Lib as Static Library (libsqlite3)
participant Netdata as Netdata Core (libnetdata)
Note over Build,Netdata: Build-Time SQLite Generation Flow
Build->>Build: include(NetdataSQLite)
alt NEW: Default Flow (Tarball)
Build->>Remote: Fetch sqlite-src-xxx.zip (sqlite.org)
else NEW: Git Flow (-DSQLITE_USE_GIT=ON)
Build->>Remote: Shallow clone (GitHub)
end
Remote-->>FS: SQLite Source Tree
Build->>FS: NEW: configure --enable-update-limit
Build->>FS: NEW: make sqlite3.c sqlite3.h
Note right of FS: Generates UDL-capable parser
FS->>FS: NEW: Copy artifacts to output directory
Note right of FS: sqlite3.c, sqlite3.h, sqlite3recover.c, etc.
Build->>Lib: Compile generated sources
Lib-->>Netdata: CHANGED: Link as static library
Note over Netdata,Lib: Runtime Interaction
Netdata->>Lib: API calls (e.g., sqlite3_open)
Note left of Netdata: Headers resolved via<br/>Generated Include Dirs
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
SQLite was vendored as the full amalgamation (sqlite3.c at ~9 MB) in src/database/sqlite/vendored/. With ~10 updates in git history, this accumulated ~65 MB of uncompressible blob storage, growing by ~9 MB per version bump.
This replaces the vendored files with a new CMake module (NetdataSQLite.cmake) that uses ExternalProject_Add to fetch the SQLite source tree and generate the amalgamation at build time with --enable-update-limit (required for the UDL-capable parser grammar).
By default, sources are fetched as a tarball from sqlite.org (~14 MB). A -DSQLITE_USE_GIT=ON option is available to use a shallow git clone from GitHub instead.
Summary by cubic
Replaced the vendored SQLite amalgamation with a CMake module that downloads the source and builds the amalgamation at compile time. This cuts repository growth (~65 MB to date, ~9 MB per bump) and keeps UPDATE/DELETE ... LIMIT support.
Dependencies
Migration
Written for commit 9e18d04. Summary will update on new commits.