🚀 LiteDB v6.0 Prerelease#52: Introducing Vector Search!

This release marks a significant milestone for LiteDB with the introduction of Vector Search! This powerful new feature enables you to build modern AI-powered applications, such as semantic search, recommendation engines, and Retrieval-Augmented Generation (RAG) systems, directly within LiteDB.

✨ Major New Feature: Vector Search

• Native Vector Storage & Indexing: We've added a native BsonVector type (for float[]) and a high-performance vector index based on the HNSW algorithm. You can now perform fast Approximate Nearest Neighbor (ANN) searches using standard distance metrics: Cosine (default), Euclidean, and DotProduct.

• New Fluent Query API: The query API has been extended with intuitive methods for vector operations.

  using LiteDB.Vector; // New namespace for vector extensions!
  
  // Create a vector index on your embedding property
  var docs = db.GetCollection<Document>("docs");
  docs.EnsureIndex(x => x.Embedding, new VectorIndexOptions(256));
  
  // Find the top 5 most similar documents to your query vector
  var results = docs.Query()
      .TopKNear(x => x.Embedding, queryVector, k: 5)
      .ToList();

• SQL Support & Demo: Vector searches are also supported in SQL via the VECTOR_SIM() function. To see a complete, end-to-end example, check out the new LiteDB.Demo.Tools.VectorSearch project in the repository, which builds a semantic search engine using Google Gemini for embeddings.

🛠️ Other Improvements & Fixes

• Reliability: Improved data integrity for external file storage (LiteFileStorage) by ensuring streams are correctly checkpointed on transaction commit.
• Build: The build process has been optimized to better support shallow clones in CI/CD environments.

What's Changed

• feat: Vector Search and Similarity Indexing by @JKamsker in #2678
• Ensure external streams checkpoint on commit by @JKamsker in #2652
• Disable GitVersion in shallow clones by @JKamsker in #2656

Full Changelog: v6.0.0-prerelease.0015...v6.0.0-prerelease.0052

What's Changed

• Fix package versions by @JKamsker in #2633
• Fix rollback handling for safepoint-flushed pages by @JKamsker in #2638
• Simplify rebuild crash test setup by @JKamsker in #2639

Full Changelog: v6.0.0-prerelease.9.10...v6.0.0-prerelease.0015

What's Changed

• Modernize build targets and CI/CD by @DavidNemecek in #2625
• Feat/improve test time by @JKamsker in #2627
• Add Composite sorting: OrderBy(...).ThenBy(...) by @JKamsker in #2632

Full Changelog: v6.0.0-prerelease.9...v6.0.0-prerelease.9.10

What's Changed

• Fix #2265 Stackoverflow when Self reference by @JKamsker in #2493
• Add-Badges by @JKamsker in #2496
• Fix crash with stack trace by @Pysis868 in #2476
• Fix LiteFileStream.SetReadStreamPosition by @SpaceCheetah in #2459
• Addresses issue of upgrading from v4.x databases in LiteDB versions 5.0.18-5.0.20 by @jdtkw in #2499
• Don't capture synchronization context in DiskWriterQueue by @alexbereznikov in #2447
• [♻️ Housekeeping] Update benchmark project by @pictos in #2509
• Added more blacklisted deserialization gadgets by @JKamsker in #2500
• Bugs/fix 2487 allow contains whitespaces by @JKamsker in #2498
• [♻️Housekeeping] Remove BufferPool class and use .net implementation by @pictos in #2508
• Fix "ENSURE: empty page must be defined as empty type" by @oleksii-datsiuk in #2504
• Remove DiskWriterQueue to avoid adicional thread to write on disk by @mbdavid in #2515
• Add support for on-the-fly document upgrades by @JKamsker in #2489

New Contributors

• @Pysis868 made their first contribution in #2476
• @SpaceCheetah made their first contribution in #2459
• @jdtkw made their first contribution in #2499

Full Changelog: v5.0.20...v5.0.21

Announcement

The LiteDB team is excited to announce that we have added two new maintainers to the project.
We are now working on more frequent releases, better stability and performance, and better communication with the community.
We are excited to see where this project goes and are looking forward to working with the community to make LiteDB the best it can be.

And if you haven't already, please join our Discord server:

What's Changed

• Fix LockRecursionException when doing FindById by @JKamsker in #2488
• Prevent using disposed snapshots by @oleksii-datsiuk in #2481
• Lazy string interpolation in ENSURE by @alexbereznikov in #2454
• Add support for ToLowerInvariant and ToUpperInvariant in query by @KrzysztofPajak in #2468
• Created tests that show the collections contains methods work by @visschersm in #2187

New Contributors

• @oleksii-datsiuk made their first contribution in #2481
• @alexbereznikov made their first contribution in #2454
• @visschersm made their first contribution in #2187
• @KrzysztofPajak made their first contribution in #2468

Full Changelog: v5.0.19...v5.0.20

What's Changed

• Detect infinite loops in document or index node loops
• Switch to using PackageLicenseExpression by @lahma in #2423
• Add LiteDb.Migration to readme by @JKamsker in #2429
• Fix issue: Id is null when anon. type is de/-serialized by @JKamsker in #2433

New Contributors

• @lahma made their first contribution in #2423
• @JKamsker made their first contribution in #2429

Full Changelog: v5.0.18...v5.0.19

What's New

• Added "AutoRebuild" option in Connection String

In this version, the Rebuild command has been completely redesigned to be able to recover a full or partial database in case of data corruption. When detecting a structural error (known as the ENSURE exception) the data file will be modified to indicate that this file may be corrupt. When reopening the database, a rebuild process can be performed recovering as much data as possible. Use the "auto-rebuild=true" option in the connection string to activate this functionality. Possible errors found during the rebuild process will be stored in a new collection called "_rebuild_errors". A backup of the original file is created with the "-backup" suffix

What's Changed

• Fixed left side IndexNode pointer to tail
• Added readonly to fields whose values are never modified or used as ref arguments by @Lehonti in #2352
• Clarify LiteDB.FSharp version support by @TeaDrivenDev in #2389
• Bugfix for #2385 Also adds full record support by fixing this bug. by @athendrix in #2384
• ResolveMember Extension by @pjy612 in #2236
• Fix Rebuild so that it truncates the database file correctly by @hizume in #2413
• Fix all kind of typos by @ltetak in #2410
• Simplified DiskWriterQueue with blocking concurrency by @ltetak in #2411
• Disk problems by @ltetak in #2412
• chore: throw exception when encounter unpaired surrogate instead of replace with U+FFFD by @anatawa12 in #2415
• New updates from master by @mbdavid in #2420
• v5.0.18 by @mbdavid in #2425

New Contributors

• @Lehonti made their first contribution in #2352
• @TeaDrivenDev made their first contribution in #2389
• @athendrix made their first contribution in #2384
• @hizume made their first contribution in #2413
• @ltetak made their first contribution in #2410
• @anatawa12 made their first contribution in #2415

Full Changelog: v5.0.17...v5.0.18

What's Changed

• Fix AesStream Invalid Password exception in corrupted files by @anurakt27 in #2344
• Update LockService.cs to fix "The read lock is being released without being held" error. Fix for #641, 1637, 1993, 2017 by @mathis1337 in #2280
• Ambigous match found by @Luigi6821 in #2332
• Fix for issue: #2307 by @m-omerovic in #2317
• Fix #2300 issue with big collection sort by @matsakiv in #2302
• Update README.md by @namigop in #2296
• Added Seek and Position setter for LiteFileStream.Read by @mr-sven in #2148
• Added unique error codes to encryption-related exceptions by @arcanelab in #2259

New Contributors

• @anurakt27 made their first contribution in #2344
• @mathis1337 made their first contribution in #2280
• @Luigi6821 made their first contribution in #2332
• @m-omerovic made their first contribution in #2317
• @matsakiv made their first contribution in #2302
• @namigop made their first contribution in #2296
• @mr-sven made their first contribution in #2148
• @arcanelab made their first contribution in #2259

Full Changelog: v5.0.16...v5.0.17

What's Changed

• use only mutex for Shared engine open and close synchronization and remove locks by @dev-joshi in #2293
• Fix Min(Func<T,TResult>) handling by @gcrossland-cosworth in #2287
• Change return type of ILiteRepository.Insert to match ILiteDatabase.Insert by @InvisiblePhil in #2260
• Fixed typo by @iArmanKarimi in #2267
• try fix SqlLike by @pjy612 in #2235

New Contributors

• @dev-joshi made their first contribution in #2293
• @gcrossland-cosworth made their first contribution in #2287
• @InvisiblePhil made their first contribution in #2260
• @iArmanKarimi made their first contribution in #2267
• @pjy612 made their first contribution in #2235

Full Changelog: v5.0.15...v5.0.16

Fix #1762