a very simple vector embedding database, you can say that it is a hash-table that let you find items similar to the item you're searching for.
I'm a databases enthusiast, and this is a for fun and learning project that could be used in production ;).
P.S: I like to re-invent the wheel in my free time, because it is my free time!
I'm using the
{key => value}model,
keyshould be a unique value that represents the item.valueshould be the vector itself (List of Floats).
by default
vecdbsearches forconfig.ymlin the current working directory. but you can override it using the--config /path/to/config.ymlflag by providing your own custom file path.
# http server related configs
server:
# the address to listen on in the form of '[host]:port'
listen: "0.0.0.0:3000"
# storage related configs
store:
# the driver you want to use
# currently vecdb supports "bolt" which is based on boltdb the in process embedded the database
driver: "bolt"
# the arguments required by the driver
# for bolt, it requires a key called `database` points to the path you want to store the data in.
args:
database: "./vec.db"
# embeddings related configs
embedder:
# whether to enable the embedder and all endpoints using it or not
enabled: true
# the driver you want to use, currently vecdb supports gemini
driver: gemini
# the arguments required by the driver
# currently gemini driver requires `api_key` and `text_embedding_model`
args:
# by default vecdb will replace anything between ${..} with the actual value from the ENV var
api_key: "${GEMINI_API_KEY}"
text_embedding_model: "text-embedding-004"- Raw Vectors Layer (low-level)
- send VectorWriteRequest to
POST /v1/vectors/writewhen you have a vector and want to store it somewhere. - send VectorSearchRequest to
POST /v1/vectors/searchwhen you have a vector and want to list all similar vectors' keys/ids ordered by cosine similarity in descending order.
- send VectorWriteRequest to
- Embedding Layer (optional)
- send TextEmbeddingWriteRequest to
POST /v1/embeddings/text/writewhen you have a text and wantvecdbto build and store the vector for you using the configured embedder (gemini for now). - send TextEmbeddingSearchRequest to
POST /v1/embeddings/text/searchwhen you have a text and wantvecdbto build a vector and search for similar vectors' keys for you ordered by cosine similarity in descending order.
- send TextEmbeddingWriteRequest to
{
"bucket": "BUCKET_NAME", // consider it a collection or a table
"key": "product-id-1", // should be unique and represents a valid value in your main data store (example: the row id in your mysql/postgres ... etc)
"vector": [1.929292, 0.3848484, -1.9383838383, ... ] // the vector you want to store
}{
"bucket": "BUCKET_NAME", // consider it a collection or a table
"vector": [1.929292, 0.3848484, -1.9383838383, ... ], // you will get a list ordered by cosine-similarity in descending order
"min_cosine_similarity": 0.0, // the more you increase, the fewer data you will get
"max_result_count": 10 // max vectors to return (vecdb will first order by cosine similarity then apply the limit)
}if you set
embedder.enabledtotrue.
{
"bucket": "BUCKET_NAME", // consider it a collection or a table
"key": "product-id-1", // should be unique and represents a valid value in your main data store (example: the row id in your mysql/postgres ... etc)
"content": "This is some text representing the product" // this will be converted to a vector using the configured embedder
}if you set
embedder.enabledtotrue.
{
"bucket": "BUCKET_NAME", // consider it a collection or a table
"content": "A Product Text", // you will get a list ordered by cosine-similarity in descending order
"min_cosine_similarity": 0.0, // the more you increase, the fewer data you will get
"max_result_count": 10 // max vectors to return (vecdb will first order by cosine similarity then apply the limit)
}