What's Changed

Added

• feat: Add incremental backup support for RocksDB storage
• feat: rebuild index
• feat: Add cluster health promethues metric
• feat: support "=" filter and "OR" filter operator
• feat: Add zstd compression for backup files and implement decompressi…
• feat: add region parameter to S3 configuration for backup service

Changed

• ci: split cluster ci
• refactor: Move the compilation of faiss to env to improve CI speed
• refactor: Update metric handling in profiler and monitor service
• chore: update faiss version to v1.10.0

Fixed

• fix: update objectName format to include clusterName in backup handler
• fix: add nil checks for monitoring system initialization in Profiler …
• fix: restore from checkpoint snapshot instead of raw data
• fix: destory column family handle before close database

New Contributors

• @yanwr1 made their first contribution in #833

Full Changelog: v3.5.6...v3.5.7

What's Changed

Added

• feat: add option for get document by hash

Changed

• refactor: split master services into separate files
• feat: implement Query method in MultiFieldsRangeIndex for optimized document retrieval
• refactor: update metrics handling to use pointer types and improve protobuf generation
• perf: change space replica num default value to 3
• perf: reduce bitmap copies
• chore(deps): bump golang.org/x/net in /tools/backup

Fixed

• fix: check duplicate id when get or upsert
• fix: set table size as max_doc_id and check updating in engine
• fix: check partition num with partition rule
• fix: query from leader when update
• fix: add error handling for GetVector method in VectorManager
• fix: check space field name
• fix(python sdk): set space replica_num
• fix: optimize RawVectors access in Engine and VectorManager to fix stack crash

Full Changelog: v3.5.5...v3.5.6

What's Changed

Added

• chore: add "space_config" in master; update dependencies for bytedance/sonic and google.golang.org/grpc by @wxingda in #816
• Add langchain4j in readme and example by @Martin7-1 in #817
• feat: add interface for replace master member and auto sync router an… by @zcdb in #819
• ci: check master members in ci_cluster.yml by @zcdb in #825
• feat(python sdk): support filter operator of NOT IN and new data type
• feat: add long_search_time config to record slow search request
• feat: add router registration logic in NewServer function
• feat: add interface for replace master member and auto sync router an…
• feat: automatically recover on other healthy ps server when replica fail
• feat: add replica anti-affinity strategy
• feat: add request ID handling and space scanning in server logic; upd…

Changed

• ci: enable automatic recovery of failed nodes in configuration
• refactor: enhance storage manager with new Get and MultiGet methods, …
• feat: string array data type can set as string_array
• feat: update data types to int64_t for improved precision and add ToR…
• refactor: use timeout midware instead of timeout handling in document…

Fixed

• fix: 1. fix multi-vector search memory leak; 2. remove VIDMgr;
• fix: add timeout handling and retry logic in RegisterRouter method
• fix: check ivfpq index score is valid
• fix: check upsert result with duplicate id
• fix: add minimum training threshold
• fix: fix dead lock in range search and crash in delete scalar index
• fix: check cluster health
• fix: use topn when merging results to avoid empty results for some shards
• fix: fix string array index length limit

New Contributors

• @Martin7-1 made their first contribution in #817

Full Changelog: v3.5.4...v3.5.5

What's Changed

Added

• chore: simplify image creation process using docker multi-stage builds
• feat: set request_id in document search interface

Removed

• refactor: reduce one JSON serialization operation in the return result and remove unnecessary proto fields

Changed

• refactor: Refactor search result merging and sorting logic to reduce memory and CPU usage
• refactor: unify the engine serialization method of search responses to protobuffers
• refactor: unify python sdk
• test: set token for python sdk test
• refactor: use static library for rocksdb in build script
• refactor: optimize load field_range_index_ addition in multi-threaded environment
• perf: use field_rw_locks_ locking instead of rw_lock_ to improve scalar indexing performance.

Fixed

• refactor: optimize response handling by removing intermediate tmpItems variable
• fix: python sdk can get space detail info and set search parameters when create space
• fix: truncate doc num on loading to prevent raft duplicate insertion
• fix: 1. fix put /fail/server/ to etcd failure; 2. fix /schedule/recover_server remove failed server from space meta data
• fix: check the index parameter of the vector field of space

New Contributors

• @kaleaditya779 made their first contribution in #812

Full Changelog: v3.5.3...v3.5.4

What's Changed

Added

• ci: add arm64 image build
• ci: add error param test case in backup
• feat: dockerfile use sleep instead of vearch
• refactor: add EQ and LIKE as todo in python sdk
• ci: add cluster index
• feat: add term filter NOT IN condition

Removed

• refactor: remove malloc_trim in engine.cc

Changed

• refactor: use ShouldBindJSON instead of vjson.Unmarshal
• ci: update docker-arch comparition
• ci: use stop and start instead of restart in ci_cluster
• ci: use stream9-minimal instead of stream9 to reduce image size
• Update DeployByDockerZH_CN.md, fix vearch-dev-env url error

Fixed

• ci:fix backup test case error
• fix: check delete result whether has error
• fix: hnsw index read and write thread competition
• fix: check docid when GetDocByDocID
• fix: hnsw index read and write thread competition
• Update index.py: Fix NamedTuple Immutability Issue in IvfPQIndex and …
• ix: change index parameter buckert_max_size
• fix: fix numeric scalar index search crash when reach the last key
• fix: fix replicas faulty num and add test cases
• fix: return error information when getSpace has error

New Contributors

• @cococo2000 made their first contribution in #798

Full Changelog: v3.5.2...v3.5.3

Added

• feat: custom sharding for range sharding
• feat: add data type of Date
• feat: rbac authentication
• feat: implement modify engine config
• feat: backup with schema
• feat: benchmark scripts
• feat: resource limit
• feat: add backupSpace interface
• feat: adjust codes to compatible arm64

Removed

• chore: remove unused code and update file paths in table_io.h

Changed

• build(deps): bump github.com/gin-contrib/cors from 1.5.0 to 1.6.0
• build: bump protoc from v3.6.1 to v3.21.0
• ci: bump up image to centos:stream9
• refactor: separate query from search in engine
• refactor: move EngineStatus to entity
• build: update rocksdb version to v9.2.1
• refactor: update some default value in python sdk
• refactor: update query documents
• remove text field index in langchain sdk
• refactor: use json instead of flatbuffers in memory_info
• refactor: use json instead of flatbuffers in engine_status
• chore: Refactor WeightedRanker to use nlohmann/json for parsing
• refactor: use std::vector instead of a character array to avoid stack memory allocation issues
• refactor: using threadskv8 as the sole B+ tree

Fixed

• fix: create root user failed then check again
• fix: fix backup with multi partitions
• fix: fix partition id error for server and change partition member error
• fix: proxy retry all master node until success
• fix field_range_index thread synchronization bug
• fix: add table_->GetFieldRawValue return value check
• fix: python sdk fix auth bug
• fix: keep float32 field precision

Full Changelog: v3.5.1...v3.5.2

Added

• add go sdk
• add python sdk
• add openapi
• add trace option for document interface

Removed

• Remove some unnecessary logic in document query

Changed

• alloc FieldOperate in stack to reduce new/delete
• merge io operator to raw_vector
• use StorageManager in RawVector
• chore: use unconditional strings.TrimSuffix
• Using reference counting instead of deferred release

Fixed

• check index MetricType is empty
• delete data directionary after engine cloesed
• wait for engine closure before deleting the data directory

New Contributors

• @testwill made their first contribution in #783

Full Changelog: v3.5.0...v3.5.1

Added

• add rocksdb bitmap manager
• add engine search and createtable error msg in response
• add data type stringArray
• ivfflat support brute force search
• Add status to manage function return values
• add alias interface

Removed

• remove useless error code
• remove boost in search request
• remove OnlineLogLevel
• remove MSearch
• remove response field _source
• remove document vector field feature in query response
• remove support for search condition of document_ids
• remove upsert vector field.feature
• remove result field.extra and table.compress_mode
• remove support for indivisible nsubvector of index ivfpq
• remove id_type

Changed

• unify error return format
• reduce memory copy in table Serialize
• return status in CreateVectorTable
• separate query from search method
• Classify the HTTP return values of Vearch as standard HTTP status codes
• use recall_num instead of quick
• rename search size to limit
• reformat query/filters
• Unify return data format of document interface
• Unify return data format of cluster version interface
• Unify the format of meta define interface
• rename index_size to training_threshold
• set default value for index_params to simplify space schema
• update space operation interface
• deal with log error without directly exit

Fixed

• call free only when an error occurs
• fix request auth

Added

• support query next undeleted document with a docid value
• add retry for listen pprof port
• add interface of space/describe
• add vearch_backup tool
• add new version python sdk struct

Removed

• remove retrieval_types

Changed

• switch the router http framework to gin
• simplify error messages
• change FieldType in space to vearchpb.FieldType
• change some error return for document interface
• replace cbjson builder by sonic
• replace ioutil by os

Fixed

• fix deleteByQuery nil result panic
• fix GetMemoryInfo crash when deleting docs

Added

• add go sdk demo

Removed

Changed

• check vector field and document_id for document/upsert

Fixed

• fix multi filter search error, fix field range index boundary error
• fix multiple vector search
• fix override warning
• fix macOS compile error