Thanks to visit codestin.com
Credit goes to github.com

Skip to content

Conversation

@zombocoder
Copy link
Owner

  • Implemented saveToFileBatched for efficient batch saving of large databases using buffered I/O.
  • Introduced saveToFileOptimized to automatically choose between regular and batched saving based on record count.
  • Updated handleInit to use the new optimized save method.

feat(manifest): switch to adaptive hashing for file checksums

  • Changed file checksum calculation to use crypto.blake3HashFileAdaptive for improved performance.

feat(scanner): implement parallel scanning with work queue

  • Added WorkQueue for managing scan jobs in parallel.
  • Created scanPathParallel to utilize multiple threads for scanning directories.
  • Introduced scanPathAdaptive to choose between sequential and parallel scanning based on estimated workload.
  • Added scanPathWithStringPool for memory-optimized scanning using string pooling.

feat(string_pool): add string interning for memory efficiency

  • Implemented StringPool and PathPool to reduce memory allocations for repeated strings, particularly filesystem paths.
  • Added functionality to track common prefixes for paths to optimize memory usage.

test: add comprehensive tests for new features

  • Added tests for saveToFileBatched, saveToFileOptimized, and string pooling functionality.
  • Included tests for the new work queue and parallel scanning features.

- Implemented `saveToFileBatched` for efficient batch saving of large databases using buffered I/O.
- Introduced `saveToFileOptimized` to automatically choose between regular and batched saving based on record count.
- Updated `handleInit` to use the new optimized save method.

feat(manifest): switch to adaptive hashing for file checksums

- Changed file checksum calculation to use `crypto.blake3HashFileAdaptive` for improved performance.

feat(scanner): implement parallel scanning with work queue

- Added `WorkQueue` for managing scan jobs in parallel.
- Created `scanPathParallel` to utilize multiple threads for scanning directories.
- Introduced `scanPathAdaptive` to choose between sequential and parallel scanning based on estimated workload.
- Added `scanPathWithStringPool` for memory-optimized scanning using string pooling.

feat(string_pool): add string interning for memory efficiency

- Implemented `StringPool` and `PathPool` to reduce memory allocations for repeated strings, particularly filesystem paths.
- Added functionality to track common prefixes for paths to optimize memory usage.

test: add comprehensive tests for new features

- Added tests for `saveToFileBatched`, `saveToFileOptimized`, and string pooling functionality.
- Included tests for the new work queue and parallel scanning features.
@zombocoder zombocoder merged commit f839b0d into main Sep 22, 2025
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants