| Meta |
|
ZeusDB Vector Database is a high-performance, Rust-powered vector database designed for fast similarity search across high-dimensional data. It enables efficient approximate nearest neighbor (ANN) search, ideal for use cases like document retrieval, semantic search, recommendation systems, and AI-powered assistants.
ZeusDB leverages the HNSW (Hierarchical Navigable Small World) algorithm for speed and accuracy, with native Python bindings for easy integration into data science and machine learning workflows. Whether you're indexing millions of vectors or running low-latency queries in production, ZeusDB offers a lightweight, extensible foundation for scalable vector search.
🐍 User-friendly Python API for adding vectors and running similarity searches
🔥 High-performance Rust backend optimized for speed and concurrency
🔍 Approximate Nearest Neighbor (ANN) search using HNSW for fast, accurate results
📦 Product Quantization (PQ) for compact storage and faster distance computations
🔢 Scalar quantization (INT8), one signed byte a value, on every distance metric
📥 Flexible input formats, including native Python types and NumPy arrays
🗂️ Metadata-aware filtering for precise and contextual querying
🔤 Text search over a sparse space, with a built-in tokenizer or one you supply
🔀 Hybrid search that runs dense, sparse and text arms over one candidate set and fuses them by rank
💾 Save and load complete indexes to disk
🧾 A journal beside the saved directory, so a process that stops loses nothing a call had returned from
ZeusDB Vector Database supports the following metrics for vector similarity search. All metric names are case-insensitive, so "cosine", "COSINE", and "Cosine" are treated identically.
| Metric | Description | Accepted Values (case-insensitive) | Product quantization | Scalar quantization |
|---|---|---|---|---|
| cosine | Cosine Distance (1 - Cosine Similarity) | "cosine", "COSINE", "Cosine" | supported | supported |
| l1 | Manhattan distance | "l1", "L1" | refused | supported |
| l2 | Euclidean distance | "l2", "L2" | supported | supported |
| dot | Inner product, reported as 1 - dot | "dot", "DOT" | refused | supported |
create() raises on a refused pair rather than building an index that ranks by the wrong quantity, and load() refuses a saved directory that pairs them. A product quantized graph scores from tables of squared L2 distances to a codebook, which cannot rank by an inner product or by Manhattan distance. A scalar quantized graph applies the metric's own arithmetic to the decoded values, so it takes all four. See Scalar Quantization.
All distance metrics in ZeusDB Vector Database return distance values, not similarity scores:
- Lower values = more similar
- A vector identical to the query scores 0.0, or a value within floating point error of it
This applies to all distance types, including cosine. dot is the one exception to the zero, because its score is 1 - dot and an inner product above one takes it below zero.
This is what search() returns. A query() page is scored differently. A sparse or text arm's score is a similarity, so higher is more similar, and a fused score is a reciprocal rank sum, which is higher for a record that placed well on more arms. See Hybrid Search.
Under cosine, vectors are normalized to unit length when they are stored. A vector you read back with return_vector=True or get_records() is therefore the normalized form, not the values you supplied. Under l1, l2 and dot the values are stored unchanged.
A zero vector has no direction, so under cosine it sits at distance 1.0 from everything, including itself.
On a quantized index the score is a distance to the record's reconstruction, not to the vector you inserted. Under l2 it is the euclidean distance to that reconstruction and under cosine it is the cosine distance to it, so either way the number is on the scale a raw index of the same space reports and the two are comparable. It is not equal to the raw score, because the index no longer holds the vector you gave it, and the difference is the quantization error. Rerank replaces it with an exact distance to the raw vector, and it is on by default for quantized_with_raw. A scalar quantized index keeps no raw vector and never reranks, so its score is the distance to the decoded row, which under cosine is at unit length.
from zeusdb_vector_database import VectorDatabase
index = VectorDatabase().create("hnsw", dim=4, space="cosine")
index.add({"id": "a", "values": [1.0, 0.0, 0.0, 0.0]})
print(round(index.search([1.0, 0.0, 0.0, 0.0], top_k=1)[0]["score"], 6))Output
0.0
You can install ZeusDB Vector Database with 'uv' or alternatively using 'pip'.
uv pip install zeusdb-vector-databasepip install zeusdb-vector-database# Import the vector database module
from zeusdb_vector_database import VectorDatabase
# Instantiate the VectorDatabase class
vdb = VectorDatabase()
# Initialize and set up the database resources
index = vdb.create(index_type="hnsw", dim=8)
# Vector embeddings with accompanying ID's and Metadata
records = [
{"id": "doc_001", "values": [0.1, 0.2, 0.3, 0.1, 0.4, 0.2, 0.6, 0.7], "metadata": {"author": "Alice"}},
{"id": "doc_002", "values": [0.9, 0.1, 0.4, 0.2, 0.8, 0.5, 0.3, 0.9], "metadata": {"author": "Bob"}},
{"id": "doc_003", "values": [0.11, 0.21, 0.31, 0.15, 0.41, 0.22, 0.61, 0.72], "metadata": {"author": "Alice"}},
{"id": "doc_004", "values": [0.85, 0.15, 0.42, 0.27, 0.83, 0.52, 0.33, 0.95], "metadata": {"author": "Bob"}},
{"id": "doc_005", "values": [0.12, 0.22, 0.33, 0.13, 0.45, 0.23, 0.65, 0.71], "metadata": {"author": "Alice"}},
]
# Upload records using the `add()` method
add_result = index.add(records)
print(add_result.summary())
# Perform a similarity search and print the top 2 results
query_vector = [0.1, 0.2, 0.3, 0.1, 0.4, 0.2, 0.6, 0.7]
results = index.search(vector=query_vector, filter=None, top_k=2)
for i, res in enumerate(results, 1):
print(f"{i}. ID: {res['id']}, Score: {res['score']:.6f}, Metadata: {res['metadata']}")Results Output:
5 inserted, 0 errors
1. ID: doc_001, Score: 0.000000, Metadata: {'author': 'Alice'}
2. ID: doc_003, Score: 0.000988, Metadata: {'author': 'Alice'}
add_result.summary() returns a plain ASCII string, so it prints on any console encoding. The same counts are on add_result.total_inserted and add_result.total_errors if you want the numbers rather than the sentence.
ZeusDB Vector Database makes it easy to work with high-dimensional vector data using a fast, memory-efficient HNSW index. Whether you're building semantic search, recommendation engines, or embedding-based clustering, the workflow is simple and intuitive.
Three simple steps
- Create an index using
.create() - Add data using
.add(...) - Conduct a similarity search using
.search(...)
Each step is covered below.
To get started, first initialize a VectorDatabase and create an HNSWIndex. You can configure the vector dimension, distance metric, and graph construction parameters.
# Import the vector database module
from zeusdb_vector_database import VectorDatabase
# Instantiate the VectorDatabase class
vdb = VectorDatabase()
# Initialize and set up the database resources
index = vdb.create(
index_type="hnsw",
dim=8,
space="cosine",
m=16,
ef_construction=200,
expected_size=5,
)
print(index.info())Output
HNSWIndex(dim=8, space=cosine, m=16, ef_construction=200, expected_size=5, vectors=0, quantization=none)
| Parameter | Type | Default | Description |
|---|---|---|---|
index_type |
str |
"hnsw" |
The type of vector index to create. Currently only "hnsw" is supported. Case-insensitive. |
dim |
int |
required | Dimensionality of the vectors to be indexed, from 1 to 65,536. Each vector must have this length. Required as of 0.8.0, see below. |
space |
str |
"cosine" |
Distance metric used for similarity search. One of "cosine", "l1", "l2", "dot". Case-insensitive. "l1" and "dot" cannot be combined with a quantization_config of type "pq"; type "int8" takes all four. |
m |
int |
16 or 32, see below |
Number of bi-directional connections created for each new node, from 2 to 256. Higher m improves recall but increases index size and build time. |
ef_construction |
int |
200 |
Width of the candidate search each insertion runs, from 1 to 4,096. It costs build time and buys graph quality, and it changes neither search latency nor the size of the finished index. See below. |
expected_size |
int |
10000 |
Estimated number of records to be inserted, from 1 to 100,000,000. Used for preallocating internal data structures and for choosing the default m. Not a hard limit, see below. |
quantization_config |
dict |
None |
Product or scalar quantization, chosen by its type. See Product Quantization and Scalar Quantization. |
indexed_fields |
list[str] |
None |
Metadata fields to build a column for, so that a filter naming only those fields does not read every record. Up to 32 names, no duplicates, none of $and, $or or $not. See Declaring the fields you filter on. |
sparse |
dict |
None |
Declares a sparse space beside the dense one. tokenizer makes it a text layer and defaults the weighting to "bm25"; without one the weighting defaults to "dot". name is the directory the space saves under and defaults to sparse. Also takes unlink and lazy_threshold_percent. See Text Search and Sparse Vectors. |
dim is required. There is no default, because dim has to equal the width your embedding model produces and an index built at any other width rejects every vector you add to it. Omitting it raises TypeError.
try:
vdb.create("hnsw")
except TypeError as error:
print(error)Output
create() requires 'dim', the width of the vectors this index will hold. There is no default because dim has to equal the width your embedding model produces, and an index built at any other width rejects every vector you add to it. Read it off one embedding with len(vector), or pass the width your model documents, for example dim=1536 for OpenAI text-embedding-3-small or dim=768 for most sentence-transformers models.
The default m depends on expected_size. It is 16 for an expected_size of 25,000 or less, and 32 above that. A graph too sparse for the number of records loses recall that no search width recovers, so declare expected_size honestly or set m yourself. Passing m explicitly always wins, and rebuild() changes it afterwards.
vdb.create("hnsw", dim=8, expected_size=25_000).get_stats()["m"] # '16'
vdb.create("hnsw", dim=8, expected_size=25_001).get_stats()["m"] # '32'expected_size is a hint and not a limit. An index accepts more records than it declared and the graph grows to fit them. What it does not change is m, which rebuild() does. Passing twice the declared size logs a warning once, on the add() that crosses it.
ef_construction costs build time and buys graph quality. It changes neither search latency nor the size of the finished index. Build time is linear in it above 100, so 50,000 records of dim=1536 build in 76.9 s at the default and 262.1 s at 800. Recall stops improving at or near the default on most data, and where it keeps climbing a larger ef_search buys more for less, so raise ef_search before raising this. The default does not move with m, so 200 is 6.25 times the layer zero neighbour budget of 32 at m=16 and 3.125 times the budget of 64 at m=32. Measured at 50,000 records with m=32, raising it to 400 bought 0.0010 recall at 10 on OpenAI embeddings of dim=1536, 0.0002 on SIFT and 0.0078 on GloVe, for 1.9 to 2.2 times the build time, which is why the constant stays.
Keep ef_construction above 2 × m. At or below the neighbour budget the graph keeps every candidate the insertion search returned and prunes none of them. create() and rebuild() both warn when the pair reaches that point, and both take the pair, so either remedy the warning names can be taken. The defaults are clear of it.
ZeusDB provides a flexible .add(...) method that supports multiple input formats for inserting or updating vectors in the index. Whether you're adding a single record, a list of documents, or structured arrays, the API is designed to be both intuitive and robust. Each record can include optional metadata for filtering or downstream use.
All formats return an AddResult containing total_inserted, total_errors, errors, vector_shape and ids.
index = vdb.create("hnsw", dim=2)
add_result = index.add({
"id": "doc1",
"values": [0.1, 0.2],
"metadata": {"text": "hello"}
})
print(add_result.total_inserted, add_result.total_errors)
print(add_result.is_success())Output
1 0
True
index = vdb.create("hnsw", dim=2)
add_result = index.add([
{"id": "doc1", "values": [0.1, 0.2], "metadata": {"text": "hello"}},
{"id": "doc2", "values": [0.3, 0.4], "metadata": {"text": "world"}},
])
print(add_result.total_inserted, add_result.total_errors)
print(add_result.vector_shape)
print(add_result.errors)Output
2 0
(2, 2)
[]
index = vdb.create("hnsw", dim=2)
add_result = index.add({
"ids": ["doc1", "doc2"],
"embeddings": [[0.1, 0.2], [0.3, 0.4]],
"metadatas": [{"text": "hello"}, {"text": "world"}],
})
print(add_result)Output
AddResult(inserted=2, errors=0, shape=Some((2, 2)))
The Some(...) wrapper appears only in the printed form. add_result.vector_shape is the plain tuple (2, 2).
ZeusDB also supports NumPy arrays as input for seamless integration with scientific and ML workflows.
import numpy as np
index = vdb.create("hnsw", dim=4)
data = [
{"id": "doc2", "values": np.array([0.1, 0.2, 0.3, 0.4], dtype=np.float32), "metadata": {"type": "blog"}},
{"id": "doc3", "values": np.array([0.5, 0.6, 0.7, 0.8], dtype=np.float32), "metadata": {"type": "news"}},
]
result = index.add(data)
print(result.total_inserted, result.total_errors)Output
2 0
index = vdb.create("hnsw", dim=2)
add_result = index.add({
"ids": ["doc1", "doc2"],
"embeddings": np.array([[0.1, 0.2], [0.3, 0.4]], dtype=np.float32),
"metadatas": [{"text": "hello"}, {"text": "world"}],
})
print(add_result)Output
AddResult(inserted=2, errors=0, shape=Some((2, 2)))
An index declaring a sparse space takes one more key per record. A record dict spells it text or sparse, a batch dict spells the same thing texts or sparse, one entry per record.
from zeusdb_vector_database import VectorDatabase
vdb = VectorDatabase()
index = vdb.create("hnsw", dim=2, sparse={"tokenizer": "simple"})
# A record dict carries text, one key per record.
print(index.add({"id": "doc1", "values": [0.1, 0.2], "text": "the quick brown fox"}))
# A batch dict carries texts, one per record, aligned with ids.
print(index.add({
"ids": ["doc2", "doc3"],
"embeddings": [[0.3, 0.4], [0.5, 0.6]],
"texts": ["a lazy brown dog", "the rain in spain"],
}))
# A record the space takes nothing from leaves the key out.
print(index.add({"id": "doc4", "values": [0.7, 0.8]}))
print(index.get_stats()["sparse_records"], len(index))Output
AddResult(inserted=1, errors=0, shape=Some((1, 2)))
AddResult(inserted=2, errors=0, shape=Some((2, 2)))
AddResult(inserted=1, errors=0, shape=Some((1, 2)))
3 4
| Shape | Text layer | Term ids |
|---|---|---|
| A record dict, formats 1 and 2 | text |
sparse |
| A batch dict, formats 3 and 5 | texts |
sparse |
A space takes one spelling and refuses the other. sparse on a text layer is that record's error, and text on a space declared without a tokenizer is the same. On an index declaring no sparse space at all, either key gives This collection declares no sparse space, one error per record. The parallel array rule extends to these, so a texts list short against ids raises before anything is inserted.
Each format is parsed and validated automatically. Invalid records are skipped rather than aborting the call, and the reason for each is returned in errors. A record whose vector contains NaN or an infinity is rejected this way.
add() upserts by default. Re-adding an existing ID replaces the whole record, metadata included. Metadata is not merged, so a key you leave out of the new record is gone.
index = vdb.create("hnsw", dim=2)
index.add({"id": "doc1", "values": [0.1, 0.2], "metadata": {"text": "hello", "lang": "en"}})
# "lang" is not carried over
index.add({"id": "doc1", "values": [0.3, 0.4], "metadata": {"text": "goodbye"}})
print(index.get_records("doc1", return_vector=False))
# overwrite=False rejects the record instead, and counts it as an error
rejected = index.add({"id": "doc1", "values": [0.5, 0.6]}, overwrite=False)
print(rejected.total_inserted, rejected.total_errors)
print(rejected.errors)Output
[{'id': 'doc1', 'metadata': {'text': 'goodbye'}}]
0 1
["Vector doc1: ValueError: Vector with ID 'doc1' already exists"]
A rejected record is reported in the AddResult. It does not raise. The rejection is also logged at WARNING level, which is visible on stderr under the default development settings.
Every overwrite leaves a node behind in the graph. See compact().
The add() method inserts or replaces one or more vectors in the index.
| Parameter | Type | Default | Description |
|---|---|---|---|
data |
dict, list[dict], dict of arrays, or np.ndarray |
required | Input records to upsert into the index. Supports the six formats above. |
overwrite |
bool |
True |
Whether an ID already in the index is replaced. With False, a colliding record is skipped and counted as an error. |
The parallel arrays must be the same length. Formats 3 and 5 pair ids[i] with vectors[i] and metadatas[i] by position, so a disagreement in length is a caller error and raises ValueError naming both lengths and which field is short. Nothing is inserted before the raise, so the call is safe to retry.
two_wide = vdb.create("hnsw", dim=2)
try:
two_wide.add({"ids": ["c", "d", "e"], "embeddings": [[0.1, 0.2], [0.3, 0.4]]})
except ValueError as error:
print(error)
print(len(two_wide))Output
add received 3 entries under 'ids' and 2 under 'embeddings'. A batch pairs them by position, so the two must be the same length, and 'embeddings' is the short one. Supply one id per vector, or omit 'ids' entirely.
0
The rule covers ids, metadatas and metadata, under every spelling of the vector key, on both the list and the NumPy branch. Omitting ids entirely is not a disagreement and still generates one per record. A parallel array must be a list; a tuple or an ndarray raises TypeError.
A batch is not atomic, and which failures raise is deliberate. A malformed batch raises before anything is inserted, so the call is safe to retry: that covers parallel arrays of different lengths, a parallel array of the wrong type, and an input that is not one of the five formats. A malformed record inside a well formed batch does not raise. It is counted in total_errors, described in errors, and the records around it are inserted: that covers a vector of the wrong width, a non-finite value, and a collision under overwrite=False. Check is_success() rather than assuming the call either inserted everything or nothing.
Returns:
AddResult with:
total_inserted: number of records successfully inserted or replacedtotal_errors: number of failed recordserrors: list of error messagesvector_shape: the shape of the processed batch, as(rows, dim)ids: the ID of every record that was inserted or replaced, in insertion orderis_success():Truewhentotal_errorsis zerosummary(): a one-line string of the two counts
ids is how you learn the IDs the index generated for records you supplied without one. It lines up with total_inserted and with nothing else, so a rejected record contributes no ID and errors is what names it.
index = vdb.create("hnsw", dim=2)
generated = index.add({"vectors": [[0.1, 0.2], [0.3, 0.4]]})
print(generated.ids)
supplied = index.add({"ids": ["a", "b"], "embeddings": [[0.5, 0.6], [0.7, 0.8]]})
print(supplied.ids)
partial = index.add({"ids": ["ok", "bad"], "embeddings": [[0.1, 0.2], [0.1]]})
print(partial.ids, partial.total_inserted, partial.total_errors)Output
['vec_1', 'vec_2']
['a', 'b']
['ok'] 1 1
Query the index using a new vector and retrieve the top-k nearest neighbors. You can also filter by metadata or return the stored vectors.
The examples below all run against this index:
index = vdb.create(index_type="hnsw", dim=8)
index.add([
{"id": "doc_001", "values": [0.1, 0.2, 0.3, 0.1, 0.4, 0.2, 0.6, 0.7], "metadata": {"author": "Alice"}},
{"id": "doc_002", "values": [0.9, 0.1, 0.4, 0.2, 0.8, 0.5, 0.3, 0.9], "metadata": {"author": "Bob"}},
{"id": "doc_003", "values": [0.11, 0.21, 0.31, 0.15, 0.41, 0.22, 0.61, 0.72], "metadata": {"author": "Alice"}},
{"id": "doc_004", "values": [0.85, 0.15, 0.42, 0.27, 0.83, 0.52, 0.33, 0.95], "metadata": {"author": "Bob"}},
{"id": "doc_005", "values": [0.12, 0.22, 0.33, 0.13, 0.45, 0.23, 0.65, 0.71], "metadata": {"author": "Alice"}},
])
query_vector = [0.1, 0.2, 0.3, 0.1, 0.4, 0.2, 0.6, 0.7]results = index.search(vector=query_vector, top_k=2)
for res in results:
print(res["id"], round(res["score"], 6), res["metadata"])Output
doc_001 0.0 {'author': 'Alice'}
doc_003 0.000988 {'author': 'Alice'}
results = index.search(vector=query_vector, filter={"author": "Alice"}, top_k=5)
for res in results:
print(res["id"], round(res["score"], 6), res["metadata"])Output
doc_001 0.0 {'author': 'Alice'}
doc_003 0.000988 {'author': 'Alice'}
doc_005 0.001143 {'author': 'Alice'}
The filter decides which records are ranked, not which results survive. A search asking for five results with a filter matching a hundred records returns the five nearest of those hundred. top_k is the page size and nothing else, so there is no need to raise it when you filter. See Metadata Filtering for what that costs.
Set return_vector=True to get the stored embedding alongside the metadata and score. Under cosine this is the normalized vector, not the values you supplied.
The vector is a list of Python floats, from both search and get_records.
results = index.search(vector=query_vector, top_k=1, return_vector=True)
print(results[0]["id"], round(results[0]["score"], 6))
print([round(v, 4) for v in results[0]["vector"]])Output
doc_001 0.0
[0.0913, 0.1826, 0.2739, 0.0913, 0.3651, 0.1826, 0.5477, 0.639]
Perform a similarity search on multiple query vectors at once. The result is a list of result lists, one per query, in the order the queries were given.
batch = [
[0.1, 0.2, 0.3, 0.1, 0.4, 0.2, 0.6, 0.7],
[0.9, 0.1, 0.4, 0.2, 0.8, 0.5, 0.3, 0.9],
]
results = index.search(vector=batch, top_k=2)
for q, hits in enumerate(results):
print(f"query {q}:", [(h["id"], round(h["score"], 6)) for h in hits])Output
query 0: [('doc_001', 0.0), ('doc_003', 0.000988)]
query 1: [('doc_002', 0.0), ('doc_004', 0.002238)]
query_batch = np.array(batch, dtype=np.float32)
results = index.search(vector=query_batch, top_k=2)
for q, hits in enumerate(results):
print(f"query {q}:", [h["id"] for h in hits])Output
query 0: ['doc_001', 'doc_003']
query 1: ['doc_002', 'doc_004']
The same filter is applied to every query in the batch. Each query gets the two nearest of Alice's documents, which for the second query are not among its two nearest documents overall.
results = index.search(batch, filter={"author": "Alice"}, top_k=2)
for q, hits in enumerate(results):
print(f"query {q}:", [h["id"] for h in hits])Output
query 0: ['doc_001', 'doc_003']
query 1: ['doc_005', 'doc_003']
The search() method retrieves the top-k most similar vectors from the index given an input query vector. Results include the vector ID, distance score, metadata, and optionally the stored vector.
| Parameter | Type | Default | Description |
|---|---|---|---|
vector |
List[float], List[List[float]], or np.ndarray |
required | The query vector (single: List[float] or 1D np.ndarray) or batch of query vectors (List[List[float]] or 2D np.ndarray). Must match the index dimension and contain only finite values. |
filter |
Dict[str, Any] | None |
None |
Optional metadata filter. Values may be a plain value for equality or a dict of operators, and $and, $or and $not compose them. See Filter Operators and Boolean composition. |
top_k |
int |
10 |
Number of nearest neighbors to return, from 0 to 65,536. |
ef_search |
int | None |
see below | Search complexity parameter, from 0 to 131,072. Higher values improve accuracy at the cost of speed. |
return_vector |
bool |
False |
If True, each result includes the stored embedding vector under a vector key. |
rerank |
int | None |
derived from the record count | Candidates fetched per requested result before rescoring against raw vectors. Only applies to a quantized index whose storage_mode is quantized_with_raw. See Quantized search accuracy. |
The default ef_search depends on the distance metric. It is max(2 × top_k, 100) for cosine and max(2 × top_k, 150) for l1 and l2.
A query vector containing NaN or an infinity raises ValueError rather than returning meaningless distances.
search() asks one space. To ask several in one call, and to see what a query would cost before it runs, see Hybrid Search and Query Plans.
ZeusDB Vector Database includes a suite of utility functions to help you inspect, manage, and maintain your index. You can view index configuration, attach custom metadata, list stored records, and remove vectors by ID.
print(index.info())Output
HNSWIndex(dim=8, space=cosine, m=16, ef_construction=200, expected_size=10000, vectors=5, quantization=none)
The vectors= field is the live record count, in every storage mode. get_vector_count() returns the same number. get_stats()["raw_vectors_stored"] is the one that counts raw vectors specifically, and on a trained quantized_only index it is zero.
Other single-value accessors: index.dim, index.space, index.m, index.ef_construction, index.expected_size, index.get_space(), len(index), index.get_vector_count(), index.has_quantization(), index.can_use_quantization(), and VectorDatabase.available_index_types().
index.dim, index.space, index.m, index.ef_construction and index.expected_size are read-only properties. get_space() is the same value as a method and is kept for callers already using it.
print(index.m, index.ef_construction, index.expected_size)Output
16 200 10000
Index level metadata is a flat str to str map, separate from the per-record metadata used for filtering. It is preserved by save() and load().
index.add_metadata({
"creator": "John Smith",
"version": "0.1",
"created_at": "2024-01-28T11:35:55Z",
"embedding_model": "openai/text-embedding-ada-002",
"environment": "production",
})
# View index level metadata by key
print(index.get_metadata("creator"))
# View all index level metadata
for key, value in sorted(index.get_all_metadata().items()):
print(f"{key}: {value}")Output
John Smith
created_at: 2024-01-28T11:35:55Z
creator: John Smith
embedding_model: openai/text-embedding-ada-002
environment: production
version: 0.1
get_all_metadata() returns a dict whose iteration order is not stable, which is why the example sorts it.
for record_id, metadata in index.list(number=5):
print(record_id, metadata)Output
doc_001 {'author': 'Alice'}
doc_002 {'author': 'Bob'}
doc_003 {'author': 'Alice'}
doc_004 {'author': 'Bob'}
doc_005 {'author': 'Alice'}
list() returns (id, metadata) tuples in the order the records were added, and offset pages through them. It lists every record, in every storage mode, and the order survives save() and load().
print(index.list(number=2, offset=0))
print(index.list(number=2, offset=2))
print(index.list(number=2, offset=4))
print(index.list(number=2, offset=99))Output
[('doc_001', {'author': 'Alice'}), ('doc_002', {'author': 'Bob'})]
[('doc_003', {'author': 'Alice'}), ('doc_004', {'author': 'Bob'})]
[('doc_005', {'author': 'Alice'})]
[]
An offset past the end returns an empty list rather than raising.
Deleting while you page shifts the pages under offset. Removing a record ahead of your cursor moves everything behind it up by one, so the next page skips one. Page with after instead, which names the last ID you saw.
paged = vdb.create("hnsw", dim=2)
paged.add({"ids": [f"p{n}" for n in range(5)], "embeddings": [[n, 0.0] for n in range(5)]})
first = paged.list(number=2)
print([record_id for record_id, _ in first])
paged.remove_point("p0")
print([record_id for record_id, _ in paged.list(number=2, after=first[-1][0])])
print([record_id for record_id, _ in paged.list(number=2, offset=2)])Output
['p0', 'p1']
['p2', 'p3']
['p3', 'p4']
offset skipped p2 because a record ahead of it was removed. after did not, because it names a position rather than a count.
after and offset cannot both be given. If the record after names has itself been removed there is no position to resume from, and the call raises KeyError rather than returning a page from somewhere else.
stats = index.get_stats()
for key in ["total_vectors", "graph_nodes", "stranded_graph_nodes", "storage_mode_description"]:
print(f"{key}: {stats[key]}")Output
total_vectors: 5
graph_nodes: 5
stranded_graph_nodes: 0
storage_mode_description: raw_only
get_stats() returns a str to str map. Every key it carries:
| Key | Holds |
|---|---|
dimension, space, m, ef_construction, expected_size, index_type |
The configuration the index was created with |
total_vectors |
The live record count |
graph_nodes, stranded_graph_nodes |
Nodes in the HNSW graph, and how many of them no record uses |
raw_vectors_stored, quantized_codes_stored |
Records held at full width, and records held as codes or as scalar rows |
storage_mode, storage_mode_description, storage_strategy |
What the index is storing and serving |
thread_safety |
The locking the index uses |
graph_memory_mb |
The HNSW graph, being the neighbour lists and, on a quantized index, the codes or rows it scores against. It holds no raw vector |
raw_vectors_memory_mb |
The store holding the raw vectors, at the capacity it asked for. The vectors themselves are raw_vectors_stored × dimension × 4 bytes |
quantized_codes_memory_mb |
The map holding a second copy of every product quantized code, which grows with the record count. 0.00 on a scalar index, whose rows live in the graph alone |
codebook_memory_mb, sdc_table_memory_mb, centroid_norm_memory_mb |
The trained product quantization tables, fixed by dim, subvectors and bits |
scale_memory_mb |
The scales of a scalar index, dim floats once per index |
index_bookkeeping_memory_mb |
The id store that finds a record, the per-record metadata, the declared columns, the live sets, the index level metadata and, on a product quantized index, the table of the code map |
reserved_memory_mb |
The part of total_memory_mb no record has been written into. It is committed and not resident, and it is exactly what shrink_to_fit() returns |
total_memory_mb |
The sum of the memory figures above, reserved_memory_mb being inside that sum rather than beside it, plus sparse_memory_mb and dictionary_memory_mb where the index declares a sparse space |
quantization_type |
pq, int8, or none on an unquantized index |
raw_vectors_retained |
The storage mode's policy, none_once_trained or all_records. Quantized indexes only |
On a quantized index it also carries quantization_active, quantization_trained, quantization_compression_ratio, quantization_training_size, training_progress, training_threshold_reached and training_vectors_needed. A product quantized index adds the rerank keys below. A scalar index adds quantization_scale and quantization_saturated_values, which Scalar Quantization describes, and carries no rerank_ key. A journaled index adds the five journal_ keys under Journaling an Index, and an index with a sparse space adds the keys under Statistics.
Every memory key prices the capacity its structure asked the allocator for, so total_memory_mb is what the index asked for and not what the process holds. The two differ in both directions. A buffer reserved and not yet written is asked for and not resident, and reserved_memory_mb is exactly that part. The allocator's own header on every block it hands out is resident and not asked for, and no figure the index computes can see it. Measured on 50,000 real 1,536-dimensional embeddings at the default m of 32, one index per interpreter, against the working set delta across the build:
| mode | reported | reserved | written | resident | reported / resident |
|---|---|---|---|---|---|
| no quantization | 513.40 MiB | 204.36 | 309.04 | 312.68 | 1.64 |
quantized_with_raw |
543.88 | 219.45 | 324.43 | 348.41 | 1.56 |
quantized_only |
31.88 | 0.42 | 31.46 | 54.70 | 0.58 |
int8 |
89.93 | 0.42 | 89.51 | 92.48 | 0.97 |
Size infrastructure from the resident figure, and compare two configurations by total_memory_mb minus reserved_memory_mb, which is what each has written and is free of where each landed in its doubling cycle. The two rows that hold raw vectors report far above what they hold, because at dim=1536 the store holding the vectors is reserved under a byte budget well short of the declaration and then grows by doubling, so its last doubling left capacity for far more records than the index holds; shrink_to_fit() hands that back once the inserts are done. The quantized_only row reports below what it holds, because the map that keys a code by id costs two small heap blocks a record and the allocator charges each block more than the index asked for.
index_bookkeeping_memory_mb is proportional to the record count and independent of the dimension. At 100,000 records with short decimal ids and no metadata it reads 36.6 bytes a record, being 20 for the id store and 16 for the per-record metadata entry. A uuid costs 35 more, a record carrying metadata costs 40 bytes a field beside the text of its string values, and a quantized_only product quantized index adds the code map's table and its copy of the id at 69 bytes a record. The figure steps with the id store's table rather than climbing smoothly, so it reads 41.9 bytes a record at 115,000 records, where the table has just doubled.
Figures the keys give by arithmetic. None of these is a key, because each is a division over two that are:
| Figure | Arithmetic |
|---|---|
| Bytes a record costs the index | total_memory_mb × 2^20 / total_vectors |
| What the index has written, against what it asked for | total_memory_mb - reserved_memory_mb, and the difference is what shrink_to_fit() returns |
| The raw vector payload, as opposed to the store holding it | raw_vectors_stored × dimension × 4 |
Whether to compact() |
stranded_graph_nodes / graph_nodes is the share of the graph no record uses, and every one of those nodes is still traversed |
| How much of a product quantized index is codes | quantized_codes_memory_mb / total_memory_mb; on a small index the fixed tables dominate |
| How far a reranked search over-fetches | rerank_default_fetch / 10, and rerank_fetch_capped says whether the calibration wanted more |
| Values a scalar index clipped, per stored value | quantization_saturated_values / (total_vectors × dimension); a rising rate says the sample the scales were fitted on no longer covers the data |
| Postings a record contributes | sparse_postings / sparse_records, and sparse_dead_postings / sparse_postings is the share compact() would drop |
| Bytes a journaled mutation costs on disk | journal_bytes / journal_records |
| Records until training | training_vectors_needed, directly |
It also reports what a product quantized search will fetch. rerank_default_fetch is the number of candidates a search at top_k=10 fetches and rescores at the record count the index holds now, and a search at a larger top_k fetches more than it reports. On an index that does not rerank, being quantized_only or one not yet trained, it reads 10, the page itself. It is the calibration's request held under a ceiling: rerank_requested_fetch is what the calibration asks for at this record count, rerank_fetch_ceiling is the bound, and rerank_fetch_capped reads true where the bound shortened the fetch, which Quantized search accuracy explains. rerank_calibrated is true on a trained quantized_with_raw index and false on every other one, including an index saved before the calibration existed. When it is true, these report what training measured:
| Key | Holds |
|---|---|
rerank_calibration_fetch |
The fetch measured on the training sample |
rerank_calibration_records |
The records it was measured over |
rerank_calibration_queries |
The queries it used |
rerank_calibration_target_recall |
The recall it was measured to reach |
rerank_calibration_exponent |
How the fetch is scaled as the index grows |
rerank_calibration_fit_fetches |
The fetches the exponent was fitted from, comma separated |
rerank_calibration_pages |
The page sizes the fetch was measured at |
rerank_calibration_page_fetches |
The fetch at each of those pages |
rerank_calibration_page_exponent |
The slope through those pages |
rerank_calibration_ms |
What the calibration cost |
Remove a vector and its metadata with .remove_point(id). This performs a logical deletion:
- The vector is deleted from internal storage.
- The metadata is removed.
- The vector ID is no longer returned by
.contains(),.get_records(), or.search().
index.remove_point("doc_001")
print("doc_001 present:", index.contains("doc_001"))
print("records remaining:", index.get_vector_count())Output
doc_001 present: False
records remaining: 4
compact() reclaims those nodes.
remove_points(ids) and remove_where(filter) remove a batch and a filtered set. Both are below, at the end of this section.
Both remove_point() and an overwriting add() leave a node behind in the graph. compact() rebuilds the graph in memory and returns the number of nodes it reclaimed. IDs, metadata, stored vectors, quantized codes and PQ training state all survive.
print("stranded graph nodes:", index.get_stats()["stranded_graph_nodes"])
print("reclaimed:", index.compact())
print("stranded graph nodes:", index.get_stats()["stranded_graph_nodes"])Output
stranded graph nodes: 1
reclaimed: 1
stranded graph nodes: 0
compact() costs a full rebuild, proportional to the number of live records rather than to the amount of debris, and it holds both graphs in memory while it runs. It returns 0 and does nothing when there is nothing to reclaim. It is never automatic, so schedule it when your workload has accumulated deletions.
m is chosen from expected_size when the index is created, so an index declared for far fewer records than it received runs at a degree meant for the smaller one. rebuild(m=..., expected_size=..., ef_construction=...) builds the graph again at a new configuration, in place, and every record keeps its vector, its metadata and its id. Pass any of the three.
sized_wrong = vdb.create("hnsw", dim=8, expected_size=100, m=4)
sized_wrong.add({
"ids": [f"v{i}" for i in range(400)],
"embeddings": [[float(i % 7) + j * 0.1 for j in range(8)] for i in range(400)],
})
print(sized_wrong.m, sized_wrong.expected_size, len(sized_wrong))
print(sized_wrong.rebuild(m=16, expected_size=400))
print(sized_wrong.m, sized_wrong.expected_size, len(sized_wrong))Output
4 100 400
400
16 400 400
It returns the node count of the graph it built, which is the live record count. Passing none of the three raises, because rebuilding the graph as it stands is compact(). An invalid value raises the message create() raises for it.
Raise m where an index outgrew its declaration, and schedule it. It costs a full rebuild, 27.0 seconds at 100,000 real 128 dimensional vectors, and it holds both graphs in memory while it runs. Nothing outside the graph is touched, so every filter returns what it returned and a save afterwards carries the new m.
Use get_records() to fetch one or more records by ID, with optional vector inclusion. It returns a list of dicts with id, metadata, and, when return_vector is true, vector.
# Single record
print(index.get_records("doc_002", return_vector=False))
# Multiple records
print(index.get_records(["doc_002", "doc_003"], return_vector=False))
# Missing IDs are silently skipped
print(index.get_records(["doc_002", "missing_id"], return_vector=False))
# Vectors are included by default
record = index.get_records("doc_002")[0]
print(sorted(record.keys()), len(record["vector"]))Output
[{'id': 'doc_002', 'metadata': {'author': 'Bob'}}]
[{'id': 'doc_002', 'metadata': {'author': 'Bob'}}, {'id': 'doc_003', 'metadata': {'author': 'Alice'}}]
[{'id': 'doc_002', 'metadata': {'author': 'Bob'}}]
['id', 'metadata', 'vector'] 8
get_records() only returns results for IDs that exist in the index. Missing IDs are skipped by default, so a shorter list than you asked for is how a missing ID is reported, and the result does not say which one.
strict=True raises KeyError instead, naming every ID the index does not hold.
try:
index.get_records(["doc_002", "missing_id"], strict=True)
except KeyError as error:
print(error)Output
'get_records(strict=True) was asked for 1 id the index does not hold: missing_id. Call it without strict=True to receive the records that are present, or test an id with contains(id) first.'
Under cosine the vector returned is the normalized one, not the values you supplied, because that is what the index stores. Under l1, l2 and dot it is what you supplied. There is no flag for this and no way to recover the original: the index keeps one copy of each vector and under cosine that copy is the unit vector. Keep your own copy if you need the values back.
Under quantized_only the vector returned is a reconstruction from the record's code, under the same vector key and with no marker saying so. Measured on 16 dimensional data with 4 subvectors and 8 bits, a reconstructed vector differed from the stored unit vector by 0.066 at the worst component. get_stats()["raw_vectors_stored"] is zero on such an index, which is how to tell. quantized_with_raw returns the stored vector exactly.
len(index) is the live record count. id in index tests membership. count(filter) counts the records a metadata filter matches, and count() with no filter is len(index).
print(len(index))
print("doc_002" in index, "doc_001" in index)
print(index.count())
print(index.count({"author": "Alice"}))
print(index.count({"author": "Nobody"}))Output
4
True False
4
2
0
count() is exact and therefore reads every record's metadata, so it costs what a filtered search costs. contains(id) is the same test as in and is kept for callers already using it.
update_metadata(id, metadata) replaces one record's metadata without resupplying its vector. The record keeps its vector, its quantized codes and its graph node, and no node is stranded.
print(index.get_records("doc_002", return_vector=False)[0]["metadata"])
print(index.update_metadata("doc_002", {"author": "Bob", "status": "reviewed"}))
print(sorted(index.get_records("doc_002", return_vector=False)[0]["metadata"].items()))
print(index.update_metadata("no_such_id", {"author": "Nobody"}))
print(index.get_stats()["stranded_graph_nodes"])Output
{'author': 'Bob'}
True
[('author', 'Bob'), ('status', 'reviewed')]
False
0
The example sorts the second result because a record's metadata comes back as a dict whose key order is not stable between processes. Read metadata by key rather than by position.
The replacement is wholesale, not a merge. Any key you leave out is gone, which is what add(overwrite=True) already does. It returns False for an ID the index does not hold, and writes nothing in that case.
Use this rather than reading a record back with get_records() and adding it again. Measured at 20,000 records the round trip costs 486.5 microseconds against 1.57 for this, and it strands one graph node per update.
remove_points(ids) takes the lock once for the whole batch instead of once per ID. It returns the IDs that were not in the index, so an empty list means every one was removed. A repeated ID is removed on its first occurrence and is never reported missing.
print(index.remove_points(["doc_004", "no_such_id"]))
print(len(index))Output
['no_such_id']
3
remove_where(filter) removes every record a metadata filter matches, using the same filter language as search(), and returns how many it removed.
print(index.remove_where({"author": "Alice"}))
print(len(index), index.count({"author": "Alice"}))
print(index.remove_where({"author": "Nobody"}))Output
2
1 0
0
An unrecognised operator raises ValueError before any record is removed. A filter matching nothing removes nothing and returns 0.
remove_where({}) is refused. An empty filter matches every record everywhere else in this language, and here that would destroy the index. Name the records with remove_points(ids) if that is what you want, or use clear().
Both leave one stranded graph node per record removed, exactly as remove_point() does, and neither calls compact().
delete(ids=...) dispatches to remove_points and delete(where=...) to remove_where. Both of those stay.
deletable = vdb.create("hnsw", dim=2, expected_size=10)
deletable.add({
"ids": ["doc_1", "doc_2", "doc_3", "doc_4"],
"embeddings": [[0.1, 0.2], [0.3, 0.4], [0.5, 0.6], [0.7, 0.8]],
"metadatas": [{"author": "Alice"}, {"author": "Bob"},
{"author": "Bob"}, {"author": "Alice"}],
})
print(deletable.delete(ids="doc_1"))
print(deletable.delete(ids=["doc_2", "no_such_id"]))
print(deletable.delete(where={"author": "Bob"}))
print(len(deletable))Output
1
1
1
1
It returns the number of records removed, an int, whichever argument was given. ids takes a string or a list of strings. A repeated ID counts once. An ID that was not there counts zero rather than raising.
remove_points still returns the IDs it could not find, which is more than a count, so keep calling it where you need that.
Passing both arguments raises, and so does passing neither. Use clear() when emptying the index is what you mean.
clear() drops every record and returns how many went. It replaces the graph rather than removing records one at a time, so stranded_graph_nodes reads 0 afterwards.
clearable = vdb.create("hnsw", dim=4, expected_size=10)
clearable.add({
"ids": ["a", "b", "c", "d", "e"],
"embeddings": [[1.0, 0, 0, 0], [0, 1.0, 0, 0], [0, 0, 1.0, 0],
[0, 0, 0, 1.0], [1.0, 1.0, 0, 0]],
})
clearable.remove_point("a")
print(clearable.clear())
print(len(clearable), clearable.get_stats()["stranded_graph_nodes"])
print(clearable.clear())Output
4
0 0
0
It keeps the index and drops the records. dim, space, m, ef_construction, expected_size, the index level metadata and the quantization configuration all survive, and a fitted PQ codebook or a fitted set of scales survives with them, so a trained quantized index can be refilled and searched without retraining. An index still collecting for training starts collecting again.
Clearing an empty index returns 0 and is not an error. The generated ID counter is not reset, so an ID generated after a clear continues the sequence rather than starting at vec_1 again.
An index built by inserting grows its graph buffers geometrically, so the last growth leaves the largest of them holding close to twice what they use. shrink_to_fit() returns that slack to the allocator and reports the bytes it released, and get_stats()["reserved_memory_mb"] prices the same slack before the call is made.
fresh = vdb.create("hnsw", dim=8, expected_size=300)
fresh.add({
"ids": [f"v{i}" for i in range(500)],
"embeddings": [[float(i % 7) + j * 0.1 for j in range(8)] for i in range(500)],
})
before = fresh.get_stats()
freed = fresh.shrink_to_fit()
after = fresh.get_stats()
print(before["reserved_memory_mb"], round(freed / 2**20, 2), after["reserved_memory_mb"])
print(float(after["graph_memory_mb"]) < float(before["graph_memory_mb"]))
print(fresh.shrink_to_fit())Output
0.02 0.02 0.00
True
0
The index above declared 300 records and was given 500, so its graph grew and left slack behind, and reserved_memory_mb named it before the call: the call returns exactly that figure and the key reads 0.00 afterwards. index, the index used throughout this section, returns 0 instead, because compact() was called on it earlier and compaction already shrinks the graph it rebuilds.
No node, edge or distance is touched, so every search returns the same page with the same scores.
Call it on an index that holds its records, not on one about to receive them. On an empty index it hands back the whole creation reservation that expected_size bought, so every later insertion regrows the arenas from nothing.
The index stays writable. The buffers grow again on the next add(), which costs one reallocation. That is why it is never automatic.
Five further accessors report state that get_stats() also carries.
| Method | Returns |
|---|---|
is_training_ready() |
Whether the training threshold has been reached. False on an index with no quantization configuration |
training_vectors_needed() |
Records still to collect before training triggers. 0 on an index with no quantization configuration |
rebuild_with_quantization() |
Rebuilds the graph against the quantized codes and returns whether it did. Training and load() already do this, so calling it is normally redundant |
get_performance_info() |
A str to str map describing the search and insertion paths: reads run concurrently under a reader-writer lock and scale with the readers while the index fits in cache, the memory bus bounds them at wide dimensions, and inserts are sequential |
benchmark_concurrent_reads(query_count, max_threads) |
Times sequential against threaded searches over random queries on a pool of max_threads threads, returning sequential_qps, parallel_qps, speedup, sequential_time, parallel_time and threads_used |
ready = vdb.create("hnsw", dim=8, expected_size=1200, quantization_config={
"type": "pq", "subvectors": 8, "bits": 8, "training_size": 1000,
})
print(ready.is_training_ready(), ready.training_vectors_needed())
print(sorted(ready.get_performance_info()))Output
False 1000
['benefits', 'insertion_path', 'quantization_accuracy_impact', 'quantization_compression', 'search_bottleneck', 'search_speedup_expected']
Product Quantization (PQ) is a vector compression technique that reduces memory usage by dividing each vector into subvectors and quantizing them independently. A record's compressed form is one byte per subvector, whatever the dimension, so an index over 1536-dimensional vectors with 8 subvectors stores 8 bytes per code in place of 6144 bytes of float32. It is one of two schemes. Scalar Quantization holds one signed byte a value instead, on every metric, and this section is the product quantized one.
ZeusDB Vector Database's PQ implementation features:
✅ Automatic training, triggered on the add() call that reaches the configured threshold
✅ Compact codes, one byte per subvector per record
✅ Asymmetric Distance Computation (ADC) for fast search against the codes
✅ Automatic switch from raw to quantized storage once training completes
Compression is not free, and the accuracy cost is much larger than the memory saving suggests. Read Quantized search accuracy before choosing a storage mode.
To enable PQ, pass a quantization_config dictionary to the .create() index method:
| Parameter | Type | Description | Valid Range | Default |
|---|---|---|---|---|
type |
str |
Quantization algorithm type. "int8" selects Scalar Quantization instead |
"pq" |
required |
subvectors |
int |
Number of vector subspaces. Must divide dim evenly |
1 to dim |
derived from dim, see below |
bits |
int |
Bits per quantized code, which sets the centroids per subvector to 2^bits | 1 to 8 | 8 |
training_size |
int |
Records collected before training is triggered | ≥ 1000 | 10000 |
max_training_vectors |
int | None |
Maximum records used during training | ≥ training_size |
None |
storage_mode |
str |
"quantized_only" or "quantized_with_raw" |
see below | "quantized_only" |
Compression ratio is dim × 4 / subvectors. More subvectors means a longer code, so it lowers the compression ratio and raises accuracy. Fewer subvectors means the opposite. At dim=1536, 8 subvectors gives 768x and 16 subvectors gives 384x.
subvectors defaults to dim / 32, clamped to between 8 and 192, snapped to a divisor of dim. That holds the compression ratio at 128x, which is the quantity accuracy follows.
dim |
default subvectors |
compression |
|---|---|---|
| 64 | 8 | 32x |
| 128 | 8 | 64x |
| 256 | 8 | 128x |
| 768 | 24 | 128x |
| 1536 | 48 | 128x |
| 3072 | 96 | 128x |
Going lower than 128x costs memory and build time and buys nothing on recall until the ratio reaches about 16x, where the fetch collapses and query time falls instead. See Quantized search accuracy. Below dim=256 the floor of 8 subvectors binds. Pass subvectors explicitly for a cheaper, less accurate setting.
bits does not change the size of a record's code, which is always one byte per subvector, so lowering it saves no memory per record. It sets the number of centroids per subvector to 2^bits, which sizes the codebook and the centroid distance table. Lowering it costs recall and shortens the build. Leave it at 8 unless the fixed cost or the build time is the constraint.
create() emits a UserWarning when the configuration looks unbalanced, for example when the compression ratio exceeds 50x, and another when storage_mode is quantized_with_raw. The ratio warning does not fire on a subvectors the library derived, only on one you passed.
It also warns when quantized_only cannot repay its fixed memory at the expected_size you declared, naming the record count above which it starts saving. Raise expected_size if your estimate was low, or drop quantization_config. quantized_with_raw has no such record count, because it holds more than an unquantized index at every one, and the warning that names the mode says so. A separate warning fires when expected_size is below training_size, because an index that never reaches its training threshold never trains.
from zeusdb_vector_database import VectorDatabase
import numpy as np
vdb = VectorDatabase()
quantization_config = {
"type": "pq", # `pq` for Product Quantization, `int8` for scalar
"subvectors": 8, # 8 subvectors of 192 dims each
"bits": 8, # 256 centroids per subvector (2^8)
"training_size": 1000, # Train once 1,000 records are collected
"storage_mode": "quantized_with_raw" # Keep raw vectors so results can be reranked
}
index = vdb.create(
index_type="hnsw",
dim=1536, # OpenAI `text-embedding-3-small` dimension
expected_size=2500,
quantization_config=quantization_config
)
# Add vectors. Training triggers automatically at the threshold.
rng = np.random.default_rng(0)
documents = {
"ids": [f"doc_{i}" for i in range(2500)],
"embeddings": rng.random((2500, 1536), dtype=np.float32),
"metadatas": [{"category": "tech", "year": 2026} for _ in range(2500)],
}
result = index.add(documents)
print("inserted:", result.total_inserted)
# Check quantization status
print("training progress:", f"{index.get_training_progress():.1f}%")
print("storage mode:", index.get_storage_mode())
print("is quantized:", index.is_quantized())
# Get compression statistics
quant_info = index.get_quantization_info()
print("compression ratio:", f"{quant_info['compression_ratio']:.1f}x")
print("codebook memory:", f"{quant_info['memory_mb']:.1f} MB")
# Search works the same way on a quantized index
query_vector = rng.random(1536, dtype=np.float32)
results = index.search(vector=query_vector, top_k=3)
print("results:", len(results), "| keys:", sorted(results[0].keys()))Output
inserted: 2500
training progress: 100.0%
storage mode: quantized_active
is quantized: True
compression ratio: 768.0x
codebook memory: 1.5 MB
results: 3 | keys: ['id', 'metadata', 'score']
The result IDs and scores depend on the data, so they are not shown. Production indexes use a much larger training_size; 1,000 is the minimum the validator accepts and keeps this example quick.
index.info() reports the quantization state as well:
print(index.info())Output
HNSWIndex(dim=1536, space=cosine, m=16, ef_construction=200, expected_size=2500, vectors=2500, quantization=pq(subvectors=8, bits=8, trained, active, compression=768.0x))
from zeusdb_vector_database import VectorDatabase
vdb = VectorDatabase()
quantization_config = {
"type": "pq",
"subvectors": 8,
"bits": 8,
"training_size": 10000,
"max_training_vectors": 50000,
"storage_mode": "quantized_only" # Drop raw vectors once training completes
}
index = vdb.create(
index_type="hnsw",
dim=3072, # OpenAI `text-embedding-3-large` dimension
expected_size=100000,
quantization_config=quantization_config
)| Mode | What it stores | Rerank available | Memory |
|---|---|---|---|
quantized_only |
Codes for every record; the raw vectors collected for training are released when training completes | No | Lowest of the three |
quantized_with_raw |
Codes and raw vectors for every record | Yes | Highest of the three. It adds the codes and the trained tables to everything an unquantized index holds |
A scalar index takes quantized_only alone and never reranks; Scalar Quantization says why.
Two consequences of quantized_only are worth knowing before you pick it.
The training records are held at full width only until training completes. Records collected before the threshold is reached are stored raw so the quantizer has something to train on. The moment training completes they are encoded and their raw copies are released, so a trained index in this mode holds no raw vector for any record.
Once training completes every record exists only as a code, so the vector you read back is an approximation. Every accessor sees every record. get_records(..., return_vector=True) and search(..., return_vector=True) reconstruct the vector from its code. Only quantized_with_raw reads back exactly, and get_stats()["raw_vectors_stored"] reading zero is how you confirm the release happened.
only = vdb.create("hnsw", dim=1536, expected_size=2500, quantization_config={
"type": "pq",
"subvectors": 8,
"bits": 8,
"training_size": 1000,
"storage_mode": "quantized_only",
})
only.add(documents) # the same 2,500 records used in Usage Example 1
print("storage mode:", only.get_storage_mode())
stats = only.get_stats()
print("raw vectors kept:", stats["raw_vectors_stored"])
print("quantized codes:", stats["quantized_codes_stored"])
print("records:", only.get_vector_count())
print("contains doc_0 (added before training):", only.contains("doc_0"))
print("contains doc_2000 (added after training):", only.contains("doc_2000"))
print("get_records doc_2000 returns:", len(only.get_records("doc_2000")), "record")Output
storage mode: quantized_active
raw vectors kept: 0
quantized codes: 2500
records: 2500
contains doc_0 (added before training): True
contains doc_2000 (added after training): True
get_records doc_2000 returns: 1 record
quantized_only is the memory mode and quantized_with_raw is the accuracy mode. A raw vector is held once, in a store the graph is handed. quantized_only replaces it with a code and holds a second code in the map that finds a record by id, so it saves dim × 4 - 2 × subvectors bytes per record against a codebook and a centroid distance table it holds whatever the record count. quantized_with_raw keeps the raw vector and adds both codes and both tables to it, so it holds more than an unquantized index at every record count. A scalar index holds its row in the graph's store alone, with no second copy and no table.
Measured resident, one index per interpreter, 50,000 records of real embeddings in each mode over the same data, at the default m of 32 for that declaration:
| dataset | dim | unquantized | quantized_only |
quantized_with_raw |
int8 |
|---|---|---|---|---|---|
| dbpedia-openai | 1,536 | 312.7 MiB | 54.7 MiB, 0.17x | 348.4 MiB, 1.11x | 92.5 MiB, 0.30x |
| sift-128 | 128 | 42.4 MiB | 29.5 MiB, 0.70x | 55.9 MiB, 1.32x | 24.4 MiB, 0.58x |
| glove-100 | 100 | 37.0 MiB | 30.1 MiB, 0.81x | 49.4 MiB, 1.33x | 23.4 MiB, 0.63x |
Pick quantized_only when memory is the constraint and quantized_with_raw when accuracy is, and read Scalar Quantization before either. What quantized_only saves is set by the share of a record that is the vector, and that share falls with the dimension: 19% at dim=100, 30% at dim=128 and 83% at dim=1,536 on the rows above. get_stats() prices your own index on your own records, which is the figure to size against.
create() warns when quantized_only cannot repay its fixed tables at the expected_size you declared, naming the record count above which it starts saving. Raise expected_size if your estimate was low, or drop quantization_config. quantized_with_raw never repays them, so it gets the warning that names the mode instead.
Quantized search is far less accurate than raw search, and quantized_only cannot be repaired by tuning. ADC scores candidates against the codes, and a code discards most of the information in a vector. Rerank fixes this by over-fetching candidates and rescoring them against raw vectors, which is only possible when the raw vectors are still there.
Measured on 6,000 clustered 128-dimensional vectors with 8 subvectors and 8 bits, recall at 10 against exact cosine search:
| Configuration | Recall@10 |
|---|---|
| No quantization | 1.00 |
quantized_only |
0.16 |
quantized_with_raw, rerank=0 |
0.15 |
quantized_with_raw, default rerank |
1.00 |
The exact figures depend on your data, but the shape does not. If you need quantization and you need accuracy, use quantized_with_raw and leave rerank on.
A product quantized cosine index ranks by the cosine distance to the reconstruction, and reports it. A reconstruction is assembled from independently trained per-subspace centroids and nothing renormalises it, so it is not a unit vector even where the record it stands for was. Measured on 25,000 OpenAI text-embedding-ada-002 vectors at dim=1,536 with 48 subvectors and 8 bits, reconstructed norms ran 0.85 to 0.96 against a stored norm of 1.0.
That matters because a quantized graph works from a table of squared L2 distances, and on those reconstructions the squared L2 ran at about 1.86 times the cosine distance rather than at exactly twice it. The gap is each record's own reconstruction length, which the index recovers from the codes, so the score you get back is the cosine distance and not a multiple of it. Ranking by cosine rather than by squared L2 also moves the page, and it measured better on every corpus and subvector count tried, by 0.0015 to 0.0518 of recall at 10 over 40,000 held-out queries each. A scalar cosine row carries the reciprocal of its decoded length beside its bytes, so a scalar index scores the cosine distance to the decoded vector at unit length and needs no such recovery.
How deep a search has to fetch to hold that recall depends on your data, not on the record count. Measured on three real datasets at 100,000 records with the default subvectors, the fetch that reaches mean recall at 10 of 0.99:
| dataset | dim | compression | fetch for 0.99 | share of corpus |
|---|---|---|---|---|
| dbpedia-openai (ada-002) | 1,536 | 128x | 494 | 0.49% |
| sift-128 | 128 | 64x | 426 | 0.43% |
| glove-100 | 100 | 40x | 5,143 | 5.14% |
No formula in the record count fits those three, so ZeusDB measures the fetch on your data instead. A quantized_with_raw index measures it when training completes, and scales what it measured with the record count and with the page size you ask for. get_stats()["rerank_default_fetch"] reports what a search at top_k=10 will fetch on the index as it stands, after the ceiling below.
On data with no resolvable structure no fetch works. Once the group the codes cannot separate is smaller than top_k, the true top ten span groups and nothing reaches them. Measure recall on your own data before you rely on quantization.
What rerank does:
rerank |
Effect |
|---|---|
| omitted | Uses the calibrated fetch, held under the ceiling below. It is the only setting that holds recall across corpus sizes and across datasets |
N of 1 or more |
Fetches top_k × N candidates, a fixed multiple of the page that does not move with the corpus. Use it to override the default deliberately |
0 |
Turns reranking off and returns the ADC scores and ordering |
A page below ten fetches what a page of ten fetches, so pass rerank explicitly if you want a shallower page to cost less. rerank has no effect on an unquantized index or on a quantized_only one, and both ignore it. With rerank on the scores you get back are raw-vector distances, and with it off they are distances to the reconstruction. Both are on the scale the index's own space reports, so a page is on one scale whichever you asked for.
An index trained before the calibration existed, and any index loaded from a directory saved by one, carries no calibration and falls back to a fixed fetch of 2% of the record count, held under the same ceiling. get_stats()["rerank_calibrated"] reads false for it. Rebuild the index to calibrate it.
The default fetch is held under a ceiling. The depth the calibration asks for at a page of ten is held under a tenth of the live records, with a floor of 1,500 candidates and an absolute ceiling of 25,000, and the page term scales what survives; a whole fetch never exceeds a quarter of the records. It exists because a calibrated fetch on data whose codes rank badly grows in proportion to the corpus, and every candidate is one exact distance over a full width vector. Measured at 100,000 records on the one dataset of three where it binds, glove-100, the ceiling shortened the fetch from 11,439 candidates to 10,000 for 0.0004 of recall at 10, and it changed nothing on the other two. An explicit rerank factor is not held under it. get_stats() reports the bound, the fetch performed and whether the bound changed it:
reranked = vdb.create("hnsw", dim=1536, expected_size=2500, quantization_config={
"type": "pq", "subvectors": 8, "bits": 8, "training_size": 1000,
"storage_mode": "quantized_with_raw",
})
reranked.add(documents) # the same 2,500 records used in Usage Example 1
stats = reranked.get_stats()
for key in ("rerank_fetch_ceiling", "rerank_default_fetch", "rerank_fetch_capped"):
print(key, stats[key])Output
rerank_fetch_ceiling 1500
rerank_default_fetch 625
rerank_fetch_capped true
rerank_requested_fetch is what the calibration asked for, which on the uniform random vectors of this example exceeds the corpus, so the fetch performed is the quarter of the 2,500 records the whole-fetch bound allows and rerank_fetch_capped says so. On real embeddings the request sits under the ceiling on most data and the two figures agree.
Above roughly 10,000 records a reranked quantized search is slower than an unquantized one, and the gap widens as the index grows. That is the price of the default holding recall. On dbpedia-openai at dim=1,536, paired against an unquantized index over the same records, 200 queries one each in turn:
| records | calibrated fetch | unquantized | quantized, default rerank | ratio |
|---|---|---|---|---|
| 10,000 | 277 | 0.75 ms | 0.71 ms | 0.95 |
| 25,000 | 411 | 0.79 ms | 0.97 ms | 1.23 |
| 50,000 | 554 | 1.17 ms | 1.54 ms | 1.32 |
| 100,000 | 747 | 1.18 ms | 2.12 ms | 1.79 |
Each row is one process building both indexes over the same records, so the ratio is the figure to read. Quantization remains a memory decision that costs query time. Lower rerank explicitly if query time matters more to you than recall, and measure what it costs you.
ef_search does nothing on a reranked quantized search. The graph traversal widens to the number of candidates the fetch asks for, which is already wider than any ef_search a caller is likely to set, and setting it smaller is discarded. Change rerank instead. On an unquantized search, and on a quantized search with rerank=0, ef_search applies normally.
- Training: happens once, on the
add()call that reachestraining_size. That call takes noticeably longer than the others. Onquantized_with_rawit also calibrates the rerank fetch, whichget_stats()["rerank_calibration_ms"]prices. - Memory: a record's code is
subvectorsbytes againstdim × 4for a raw vector, and a raw vector is held once.quantized_onlysaves andquantized_with_rawcosts. The table in Storage modes prices both, and a scalar row ofdimbytes, atdim=100,dim=128anddim=1,536. - Search speed: an unreranked quantized search is faster than a raw search. A reranked one is slower above roughly 10,000 records, and the table above prices it.
- Build speed: a quantized build is faster than an unquantized one, and it slows as
subvectorsrises. At 100,000 records ofdim=768it is 137 s against 231 s at the defaultsubvectors. - Accuracy: see the tables above. Treat quantization as a memory decision that costs accuracy and query time, not as a free win.
Scalar quantization holds every value of a vector as one signed byte. One scale per dimension is fitted when training triggers, as the largest magnitude that dimension reaches over the training sample divided by 127, and the graph decodes each byte through its scale inside every distance it evaluates. A record costs dim bytes, plus four under cosine for the length of its decoded vector, against dim × 4 raw.
It takes all four metrics, because the kernel applies the metric's own arithmetic to the decoded values and no table fitted to one objective sits between them. It keeps no raw vector and never reranks, so quantized_only is the one storage mode it takes.
| Parameter | Type | Description | Valid Range | Default |
|---|---|---|---|---|
type |
str |
Quantization algorithm type | "int8" |
required |
scale |
str |
How the scales are fitted, one per dimension | "per_dimension" |
"per_dimension" |
training_size |
int |
Records collected before the scales are fitted | ≥ 1000 | 10000 |
max_training_vectors |
int | None |
Maximum records used to fit the scales | ≥ training_size |
None |
storage_mode |
str |
The one mode a scalar index takes | "quantized_only" |
"quantized_only" |
subvectors and bits belong to "pq" and are refused under "int8", as scale is refused under "pq". storage_mode="quantized_with_raw" is refused, because the mode exists to rerank against raw vectors and a scalar index gives up too little for that to be worth four bytes a value.
from zeusdb_vector_database import VectorDatabase
import numpy as np
vdb = VectorDatabase()
index = vdb.create(
index_type="hnsw",
dim=1536,
space="cosine",
expected_size=2500,
quantization_config={"type": "int8", "training_size": 1000},
)
rng = np.random.default_rng(0)
index.add({
"ids": [f"doc_{i}" for i in range(2500)],
"embeddings": rng.random((2500, 1536), dtype=np.float32),
})
print(index.info())
stats = index.get_stats()
for key in ("storage_mode_description", "raw_vectors_stored", "quantized_codes_stored",
"quantization_type", "quantization_scale", "quantization_saturated_values"):
print(f"{key}: {stats[key]}")Output
HNSWIndex(dim=1536, space=cosine, m=16, ef_construction=200, expected_size=2500, vectors=2500, quantization=int8(scale=per_dimension, trained, active, compression=4.0x))
storage_mode_description: quantized_active
raw_vectors_stored: 0
quantized_codes_stored: 2500
quantization_type: int8
quantization_scale: per_dimension
quantization_saturated_values: 1458
Recall at 10 against exact search on 100,000 records, and resident memory on 50,000 records of the same data, one index per interpreter:
| dataset | dim | unquantized | quantized_only |
int8 |
|---|---|---|---|---|
| dbpedia-openai | 1,536 | 0.9929, 312.7 MiB | 0.4893, 54.7 MiB | 0.9861, 92.5 MiB |
| sift-128 | 128 | 0.9988, 42.4 MiB | 0.4038, 29.5 MiB | 0.9827, 24.4 MiB |
| glove-100 | 100 | 0.8858, 37.0 MiB | 0.2647, 30.1 MiB | 0.8788, 23.4 MiB |
A scalar index gives up two hundredths of recall for a third to two thirds of the memory, and searching it is no slower than searching an unquantized one. A trained index writes int8_scales.zdbint8 and int8_rows.zdbint8 in place of the two pq_ files, and its directory declares format version 1.2.0.
A value beyond the range its dimension's sample reached is clipped, not refused. quantization_saturated_values counts every clipped value, and a rate that climbs as records arrive says the sample no longer covers the data.
ZeusDB Vector Database can save and restore complete indexes on disk, which lets you preserve your work, move indexes between systems, and back up production deployments.
The persistence system supports:
✅ Complete state preservation for vectors, per-record metadata, index level metadata, ID mappings and quantization models
✅ Hybrid storage format, binary encoding for vectors with human-readable JSON for metadata
✅ Quantization support, both raw and quantized storage modes, including the trained codebook or scales
✅ Training state recovery, so an index saved mid-collection resumes collecting
✅ Sparse spaces, the postings and the term dictionary, under spaces/<name>/
✅ Format versioning, so a directory this build cannot interpret is refused rather than misread
✅ Atomic saves, so a reader sees the whole previous index or the whole new one
✅ A digest per artefact, checked on load, so a file that has changed since it was written is refused
✅ A journal, opened beside the directory on request, so every mutation since the last save is replayed when the directory opens
Use the .save() method to persist your index to a .zdb directory:
from zeusdb_vector_database import VectorDatabase
import numpy as np
import os
vdb = VectorDatabase()
index = vdb.create("hnsw", dim=1536, space="cosine", expected_size=1000)
rng = np.random.default_rng(1)
vectors = rng.random((1000, 1536), dtype=np.float32)
index.add({
"ids": [f"doc_{i}" for i in range(1000)],
"embeddings": vectors,
"metadatas": [{"category": f"cat_{i % 5}", "index": i} for i in range(1000)],
})
index.save("my_index.zdb")
print("saved:", sorted(os.listdir("my_index.zdb")))Output
saved: ['config.json', 'hnsw_index.zdbgraph', 'manifest.json', 'mappings.bin', 'metadata.json', 'vectors.bin']
Use the .load() method to restore a previously saved index:
vdb = VectorDatabase()
loaded_index = vdb.load("my_index.zdb")
print("vectors:", loaded_index.get_vector_count())
print(loaded_index.info())
results = loaded_index.search(vectors[0].tolist(), top_k=3)
print("top hit:", results[0]["id"])Output
vectors: 1000
HNSWIndex(dim=1536, space=cosine, m=16, ef_construction=200, expected_size=1000, vectors=1000, quantization=none)
top hit: doc_0
Loading reads the saved graph back rather than rebuilding it, so a reloaded index returns the same result pages as the index that was saved, with the same IDs and the same scores. Load time is proportional to the size of the directory rather than to the cost of building the index: 50,000 records at 1,536 dimensions load in 1.1 seconds against a 156 second build. A journaled directory adds the replay of every record in its journal, each at the cost of the add() that wrote it; see Journaling an Index.
The graph is rebuilt by re-inserting every record only when the saved graph cannot be used, which covers a directory whose graph files were lost or damaged and one written by a release too old for this build to interpret. Set ZEUSDB_LOAD_REBUILD_GRAPH=1 to ask for that rebuild on a directory whose graph is perfectly readable, which is how an index built by an earlier release picks up graph improvements made since.
A directory whose sparse space was declared with your own tokenizer needs it handed back, as vdb.load(path, tokenizer=tokenize). See Text Search.
A quantized index comes back quantized, with its codebook and training state intact:
quantization_config = {
"type": "pq",
"subvectors": 8,
"bits": 8,
"training_size": 1000,
"storage_mode": "quantized_with_raw",
}
vdb = VectorDatabase()
index = vdb.create("hnsw", dim=1536, expected_size=2000,
quantization_config=quantization_config)
rng = np.random.default_rng(2)
index.add({
"ids": [f"vec_{i}" for i in range(2000)],
"embeddings": rng.random((2000, 1536), dtype=np.float32),
})
print("quantization active:", index.is_quantized())
index.save("quantized_index.zdb")
loaded_index = vdb.load("quantized_index.zdb")
print("quantization active after load:", loaded_index.is_quantized())
print("storage mode after load:", loaded_index.get_storage_mode())
print("saved:", sorted(os.listdir("quantized_index.zdb")))Output
quantization active: True
quantization active after load: True
storage mode after load: quantized_active
saved: ['config.json', 'hnsw_index.zdbgraph', 'manifest.json', 'mappings.bin', 'metadata.json', 'pq_centroids.bin', 'pq_codes.bin', 'quantization.json', 'vectors.bin']
A scalar index writes int8_scales.zdbint8 and int8_rows.zdbint8 in place of the two pq_ files and holds no vectors.bin; Scalar Quantization shows the listing.
The .save() method creates a directory containing all index components:
my_index.zdb/
├── manifest.json # Index metadata, file inventory and, if journaled, the journal record
├── config.json # HNSW configuration and index level metadata
├── mappings.bin # ID mappings (binary format)
├── metadata.json # Per-record metadata (JSON format)
├── vectors.bin # Raw vectors (whenever the index holds any)
├── quantization.json # Quantization configuration and training state (if enabled)
├── pq_centroids.bin # Trained centroids (if PQ trained)
├── pq_codes.bin # Quantized codes (if PQ active)
├── int8_scales.zdbint8 # One scale per dimension (if scalar quantization trained)
├── int8_rows.zdbint8 # Every record's scalar row (if scalar quantization trained)
├── hnsw_index.zdbgraph # HNSW graph structure and payload
└── spaces/ # One directory per sparse space (if declared)
└── <name>/
├── postings.zdbsparse # Every live record's term ids and weights
└── terms.zdbdict # The term dictionary (text layers only)
my_index.zdb.zdbwal # The journal, beside the directory (if journaled)
<name> is the space's name, which defaults to sparse. terms.zdbdict is written only where the space was declared with a tokenizer. The journal is a sibling of the directory rather than a file inside it, and Journaling an Index says why.
manifest.json lists every file the save wrote under files_included and is the last file written, so it is the inventory of what the directory does hold. Beside the list, file_digests records each artefact's length and a digest of its contents, and for a journaled index a journal record names the sibling file, the collection id both share and the sequence the checkpoint holds.
A directory saved by 0.6.0 or earlier holds hnsw_index.hnsw.graph and hnsw_index.hnsw.data in place of hnsw_index.zdbgraph. Opening it still works: the graph is rebuilt once from the stored records, and the next .save() writes the single file.
load() refuses a directory that does not hold what its manifest names. It checks files_included before it reads anything, and the graph dump is the one exempt artefact, because every record carries what the graph is built from. A directory missing any other file will not open, and the refusal names the file and says what it held. Restore it from a copy; the missing file cannot be rebuilt from the ones that remain. A file the manifest does not name is neither read nor complained about.
It also refuses a file that is present and has changed. Each artefact is checked against the length and digest file_digests records for it, before anything parses it, so a file edited in place is refused with its name and both digests in the message. The graph dump, the two scalar artefacts and a sparse space's two files carry their own header and payload checksums instead, so the manifest records only their length. A dump that disagrees is rebuilt rather than refused, and any of the others is refused by its own frame.
A directory saved before 0.8.0 carries no digests, so nothing is verified and it loads exactly as it did.
A file that is present and does not parse is a different failure with a different message, of the form Failed to parse config.json or Failed to deserialize mappings.bin.
A full persistence lifecycle with integrity checks:
from zeusdb_vector_database import VectorDatabase
import numpy as np
# === PHASE 1: CREATE AND POPULATE INDEX ===
vdb = VectorDatabase()
original_index = vdb.create("hnsw", dim=1536, space="cosine", expected_size=500)
rng = np.random.default_rng(42)
vectors = rng.random((500, 1536), dtype=np.float32)
original_index.add({
"ids": [f"doc_{i:03d}" for i in range(500)],
"embeddings": vectors,
"metadatas": [
{
"category": ["science", "tech", "health", "finance"][i % 4],
"priority": i % 10,
"published": i % 2 == 0,
"tags": ["important", "featured"] if i % 5 == 0 else ["standard"],
}
for i in range(500)
],
})
original_index.add_metadata({
"dataset": "demo_collection",
"created_by": "data_team",
"version": "1.0",
})
query_vector = vectors[0].tolist()
original_results = original_index.search(query_vector, top_k=3)
# === PHASE 2: SAVE, THEN LOAD ===
original_index.save("demo_index.zdb")
loaded_index = vdb.load("demo_index.zdb")
# === PHASE 3: VERIFY INTEGRITY ===
assert loaded_index.get_vector_count() == original_index.get_vector_count()
assert loaded_index.info() == original_index.info()
assert loaded_index.get_all_metadata() == original_index.get_all_metadata()
loaded_results = loaded_index.search(query_vector, top_k=3)
assert [r["id"] for r in loaded_results] == [r["id"] for r in original_results]
filtered = loaded_index.search(
query_vector,
filter={"category": "science", "published": True},
top_k=20,
)
print("records:", loaded_index.get_vector_count())
print("index metadata fields:", len(loaded_index.get_all_metadata()))
print("filtered hits:", len(filtered))
print("all checks passed")Output
records: 500
index metadata fields: 3
filtered hits: 20
all checks passed
A save is a point in time, and every mutation after it lives in memory alone until the next one. journal_to(path) opens a journal beside the directory at path, saves the index into that directory first, and records every mutation to the journal from then on. load(path) replays it, so a process that stops without saving, however it stops, loses nothing a call had returned from.
index = vdb.create("hnsw", dim=4, expected_size=100)
index.add({"ids": ["a", "b"], "embeddings": [[1.0, 0.0, 0.0, 0.0], [0.0, 1.0, 0.0, 0.0]]})
# Saves the index into journaled.zdb, then records every mutation beside it
index.journal_to("journaled.zdb")
print(index.journal_path)
print(index.info())
index.add({"ids": ["c"], "embeddings": [[0.0, 0.0, 1.0, 0.0]]})
index.remove_point("a")
stats = index.get_stats()
for key in ("journal_durability", "journal_sequence", "journal_records", "journal_bytes"):
print(f"{key}: {stats[key]}")
# A checkpoint is save() into the journal's own directory, and it empties the journal
index.checkpoint()
print("after checkpoint:", index.get_stats()["journal_records"])Output
journaled.zdb.zdbwal
HNSWIndex(dim=4, space=cosine, m=16, ef_construction=200, expected_size=100, vectors=2, journal=call, quantization=none)
journal_durability: call
journal_sequence: 0
journal_records: 2
journal_bytes: 165
after checkpoint: 0
The journal is <path>.zdbwal, a sibling of the directory rather than a file inside it. durability names what a call promises about its records once it has returned, on journal_to() and again on every load():
durability |
When the call returns | A process that stops | An OS crash or power loss |
|---|---|---|---|
"call", the default |
every record is on the device, or the call raised | nothing acknowledged is lost | nothing acknowledged is lost |
"interval" |
every record is in the kernel, flushed within interval_ms, 10 by default |
nothing acknowledged is lost | up to one interval of acknowledged calls |
"none" |
every record is in the kernel until the next checkpoint | nothing acknowledged is lost | everything since the last checkpoint |
get_stats() reports five journal_ keys on a journaled index, being the four the example prints and journal_interval_ms, which is present under "interval" alone.
Nothing checkpoints for you. An index that is never checkpointed opens by replaying every record in its journal, each at the cost of the add() that wrote it, so a directory with a million records in the journal takes minutes to open where its checkpoint takes seconds. journal_records passing the record count is a reasonable trigger. A journaled directory declares format version 3.0.0.
-
Directory, not a file.
.save()creates a directory. You need write permission for the target location. -
Atomic. A save writes
<name>.zdbtmpbeside the target and renames it into place, so a reader sees the previous index or the new one and never a mixture. An interrupted save leaves the previous directory intact and loadable, and the staging directory is removed.Replacing an existing directory takes two renames rather than one, because neither Windows nor POSIX can rename a directory over a non-empty one: the target moves to
<name>.zdbold, the new directory moves in, then<name>.zdboldis removed. Between the two renames the target does not exist. A process killed in that window leaves the whole previous index at<name>.zdbold, and the next save or load moves it back. -
Overwriting is clean. The new directory is built from nothing, so an artefact from an earlier save cannot survive. Saving a plain index over a quantized one leaves no
quantization.json,pq_centroids.binorpq_codes.binbehind. -
Same volume. The staging directory is a sibling of the target, so both are on the target's volume and the move is a rename rather than a copy.
-
Version compatibility. The manifest records a format version, and this build reads any 1.x, 2.x or 3.x. What it writes depends on what the directory holds:
Directory holds Version written Opens on a dense space alone 1.1.0 every release that reads 1.x a dense space with scalar quantization 1.2.0 this release or later a sparse space 2.0.0 0.10.0 or later a sparse space and scalar quantization 2.1.0 this release or later a journal 3.0.0 this release or later a journal and scalar quantization 3.1.0 this release or later A different major version is refused with a message naming the newer release. The scalar versions are minors because an older reader refuses such a directory on the first field of its
quantization.jsonit does not know, rather than opening it wrongly. -
Integrity checks on load. Four run, in this order: the format version, then
files_includedagainst the directory, then each artefact against its recorded length and digest, then the restored record count against the count inconfig.json. A directory holding a sparse space adds two, being every record the space holds against the id mappings, and every term id the postings carry against the length of the dictionary. A journaled directory adds the journal's own, being the collection id in its header against the manifest's, its first record against the sequence the checkpoint holds, a record in its middle whose bytes changed after it was written, and every replayed record against the index it lands on. A trained scalar directory adds the bounds of its two artefacts, being one finite positive scale per declared dimension, and one row per record the mappings hold in increasing internal id order. -
save()andload()are silent. Every step they used to print to stdout is adebuglog line instead, so a library caller sees nothing on stdout. SetZEUSDB_LOG_LEVEL=debugto see the steps.
ZeusDB supports rich metadata with full type fidelity. Your metadata preserves the original Python data types, so integers stay integers and floats stay floats.
| Type | Python Example | Notes |
|---|---|---|
| String | "Alice" |
Text data, IDs, categories |
| Integer | 42, 2024 |
Counts, years, IDs |
| Float | 4.5, 29.99 |
Ratings, prices, scores |
| Boolean | True, False |
Flags, status indicators |
| Null | None |
Missing or empty values |
| Array | ["ai", "science"] |
Tags, categories, lists |
| Nested Object | {"key": "value"} |
Structured data |
Integers and floats compare by magnitude, so a stored integer 10 matches {"eq": 10.0} and {"gte": 10.0} alike. Booleans and strings do not cross into numbers.
A filter is a dict whose keys are field names, and all of them must hold. A field maps either to a plain value, which means equality, or to a dict of operators, all of which must hold. Three reserved keys, $and, $or and $not, compose whole filters rather than naming a field; see Boolean composition.
| Operator | Usage | Example | Description |
|---|---|---|---|
| Direct equality | {"field": value} |
{"author": "Alice"} |
Equality for strings, numbers, booleans, null and arrays |
eq |
{"eq": value} |
{"source": {"eq": {"kind": "web"}}} |
Equality, including for nested objects |
ne |
{"ne": value} |
{"author": {"ne": "Alice"}} |
Not equal |
gt |
{"gt": value} |
{"rating": {"gt": 4.0}} |
Greater than (numeric) |
gte |
{"gte": value} |
{"year": {"gte": 2024}} |
Greater than or equal (numeric) |
lt |
{"lt": value} |
{"price": {"lt": 30}} |
Less than (numeric) |
lte |
{"lte": value} |
{"pages": {"lte": 100}} |
Less than or equal (numeric) |
contains |
{"contains": value} |
{"tags": {"contains": "ai"}} |
String contains substring, or array contains value |
startswith |
{"startswith": value} |
{"title": {"startswith": "The"}} |
String starts with substring |
endswith |
{"endswith": value} |
{"file": {"endswith": ".pdf"}} |
String ends with substring |
in |
{"in": [values]} |
{"lang": {"in": ["en", "es"]}} |
Value is in the provided array |
nin |
{"nin": [values]} |
{"lang": {"nin": ["en", "es"]}} |
Value is not in the provided array |
any |
{"any": [values]} |
{"tags": {"any": ["ai", "ml"]}} |
Array field shares at least one element with the provided array |
all |
{"all": [values]} |
{"tags": {"all": ["ai", "ml"]}} |
Array field holds every element of the provided array |
exists |
{"exists": bool} |
{"lang": {"exists": False}} |
Whether the record carries the field at all |
is_missing |
{"is_missing": bool} |
{"lang": {"is_missing": True}} |
The complement of exists |
is_null |
{"is_null": bool} |
{"lang": {"is_null": True}} |
The record carries the field and its value is null |
any and all exist because a field maps to one condition object, so it cannot carry contains twice. They ask their question of one field's array, where $or and $and compose whole filters across fields. On a field holding a plain value rather than an array, both read it as an array of one.
Three behaviours are worth knowing.
A record that lacks the field never matches, whatever the operator. That includes ne and nin. {"lang": {"ne": "en"}} and {"lang": {"nin": ["en"]}} do not match a record with no lang at all, and they agree because nin against a one-element array means what ne means.
exists, is_missing and is_null are the three that ask about the field itself, so they are the exception to the rule above and each is decided before the value is looked up. A missing field and a stored null are different: {"lang": None} stores a null and {"lang": {"exists": False}} matches only a record with no lang key.
from zeusdb_vector_database import VectorDatabase
selector = VectorDatabase().create("hnsw", dim=2)
selector.add([
{"id": "has", "values": [1.0, 0.0], "metadata": {"lang": "en"}},
{"id": "null", "values": [0.0, 1.0], "metadata": {"lang": None}},
{"id": "none", "values": [1.0, 1.0], "metadata": {}},
])
found = lambda f: sorted(r["id"] for r in selector.search([1.0, 0.0], filter=f, top_k=9))
print(found({"lang": {"exists": True}}))
print(found({"lang": {"is_missing": True}}))
print(found({"lang": {"is_null": True}}))
print(found({"lang": {"exists": True, "is_null": False}}))Output
['has', 'null']
['none']
['null']
['has']
Each takes True or False and anything else raises. is_null: False is the complement of is_null: True, so it matches a record with no field at all; write "present and not null" as the conjunction above.
A dict value is always read as operators. Direct equality against a nested object has no plain form, because the two would be indistinguishable, so write it as {"source": {"eq": {"kind": "web"}}}. Writing {"source": {"kind": "web"}} raises ValueError: Unknown filter operation: kind.
An unrecognised operator raises ValueError before the search runs, rather than quietly matching nothing.
A filter is a conjunction of its keys. Three reserved keys compose whole filters instead of naming a field.
| Key | Takes | Means |
|---|---|---|
$and |
a list of filters | every one of them holds |
$or |
a list of filters | at least one of them holds |
$not |
one filter | that filter does not hold |
Precedence. There is none to remember, because the structure is explicit. A mapping is an AND of everything in it, fields and groups alike, so {"a": 1, "$or": [...]} means a == 1 AND the disjunction. A group's branches are each a whole filter, so a branch carrying two fields conjoins them.
Nesting. Groups nest to 10 levels, counting the filter itself as level one. A filter deeper than that raises ValueError.
Reserved keys. Exactly $and, $or and $not. The $ prefix is not reserved, so a field named $price still filters. A field literally named $or, $and or $not cannot be filtered on, and a filter naming it raises.
The empty cases. {"$and": []} matches every record and {"$or": []} matches none, which is what all and any already do with an empty array.
from zeusdb_vector_database import VectorDatabase
vdb = VectorDatabase()
index = vdb.create("hnsw", dim=4, space="l2")
index.add([
{"id": "d1", "values": [0.1, 0.1, 0.1, 0.1],
"metadata": {"lang": "en", "tier": "gold", "year": 2024}},
{"id": "d2", "values": [0.2, 0.2, 0.2, 0.2],
"metadata": {"lang": "es", "tier": "free", "year": 2023}},
{"id": "d3", "values": [0.3, 0.3, 0.3, 0.3],
"metadata": {"lang": "fr", "tier": "gold", "year": 2026}},
{"id": "d4", "values": [0.4, 0.4, 0.4, 0.4],
"metadata": {"tier": "free", "year": 2025}},
])
q = [0.1, 0.1, 0.1, 0.1]
def matched(filter):
return sorted(hit["id"] for hit in index.search(vector=q, filter=filter, top_k=10))
# Either language. A flat filter cannot ask this, because one field maps to one
# condition and two conditions on it are conjoined.
print(matched({"$or": [{"lang": "en"}, {"lang": "es"}]}))
# Gold tier, and either recent or English. A branch is a whole filter, so the
# second one carries two fields and conjoins them.
print(matched({"tier": "gold",
"$or": [{"year": {"gte": 2026}}, {"lang": "en"}]}))
# Not free tier. This is what `ne` does here, since every record has a tier.
print(matched({"$not": {"tier": "free"}}))
# Records with no lang field at all, which no operator can select.
print(matched({"$not": {"lang": {"all": []}}}))
# None of these, which is a negated disjunction over a group.
print(matched({"$not": {"$or": [{"lang": "es"}, {"tier": "gold"}]}}))Output
['d1', 'd2']
['d1', 'd3']
['d1', 'd3']
['d4']
['d4']
A filter over a field left out of indexed_fields reads every record's metadata, so it costs a great deal more than an unfiltered search. Declaring the field builds a column and removes that cost, which the next section measures.
Two paths serve a filter and the index chooses between them per search. At or below 5,000 matching records it scores every record that matched and ranks them, which is exact. Above 5,000 the graph traversal runs instead with the filter tested at every node it reaches, and recall there is the graph's own, measured at 0.96 and above on three real 100,000 record sets.
Measured on three real 100,000 record sets with no field declared, milliseconds per query, minimum of several passes. The figures were taken before the per-record metadata moved into a store indexed by internal id, which made every metadata walk about three times faster when paired inside one process at 50,000 records, so every row below the first is an upper bound until it is re-measured:
| Records matched | Path | sift, 128d | glove, 100d | dbpedia, 1536d |
|---|---|---|---|---|
| no filter | graph | 0.30 | 0.29 | 1.16 |
| 50,000 | graph | 3.2 | 3.7 | 5.0 |
| 10,000 | graph | 18.0 | 27.2 | 31.6 |
| 1,000 | exact | 33.1 | 39.4 | 38.4 |
| 100 | exact | 36.1 | 30.8 | 30.8 |
| 1 | exact | 38.3 | 31.6 | 34.7 |
Declare the fields you filter on, because undeclared a filtered search over 100,000 records costs tens of milliseconds where an unfiltered one costs a fraction of one, and it grows in proportion to the record count. Filtering on a field that few records carry does not reduce it, since the walk visits every record either way.
create(indexed_fields=[...]) builds a column for each field named, so a filter naming only declared fields is answered from those columns rather than by reading every record's metadata. It changes which records come back in no way, only what finding them costs.
from zeusdb_vector_database import VectorDatabase
catalogue = VectorDatabase().create(
"hnsw", dim=4, space="l2", indexed_fields=["lang", "tier"]
)
catalogue.add([
{"id": "c1", "values": [0.1, 0.1, 0.1, 0.1],
"metadata": {"lang": "en", "tier": "gold", "year": 2024}},
{"id": "c2", "values": [0.2, 0.2, 0.2, 0.2],
"metadata": {"lang": "es", "tier": "free", "year": 2023}},
])
query = [0.1, 0.1, 0.1, 0.1]
print(catalogue.indexed_fields)
# Answered from the columns, because every field the filter names is declared.
print([hit["id"] for hit in catalogue.search(vector=query, filter={"tier": "gold"})])
# The same record, found by reading metadata, because year was not declared.
print([hit["id"] for hit in catalogue.search(vector=query, filter={"year": 2023})])Output
['lang', 'tier']
['c1']
['c2']
The same filter answered both ways, on three real 100,000 record sets, milliseconds per query, minimum of three passes over thirty queries. The undeclared column was measured before the metadata store made every walk about three times faster, so it is an upper bound until it is re-measured:
| Records matched | Declared | Not declared |
|---|---|---|
| 1 | 0.09 to 0.15 | 28.1 to 73.9 |
| 1,000 | 0.37 to 0.48 | 31.2 to 57.2 |
| 10,000 | 3.5 to 12.6 | 20.5 to 36.9 |
| 50,000 | 0.82 to 4.1 | 3.9 to 15.7 |
Declare the fields you filter on and leave the rest out. Eight declared fields over 100,000 records cost 6.67 MB, which is a fifth of what the metadata itself costs. A field carrying a distinct value on nearly every record costs 42 bytes a record instead of 4, so declare a document id only if you filter on it.
A filter naming an undeclared field returns the same records, finds them by reading metadata rather than from a column, and logs one warning naming the field. index.indexed_fields reads the declaration back and is empty on an index created without it.
The examples below all run against this index:
from zeusdb_vector_database import VectorDatabase
vdb = VectorDatabase()
index = vdb.create("hnsw", dim=4, space="l2")
index.add([
{"id": "doc_1", "values": [0.1, 0.1, 0.1, 0.1], "metadata": {
"author": "Alice", "rating": 4.5, "year": 2024, "price": 29.99,
"published": True, "tags": ["ai", "science"], "title": "The Guide",
"filename": "report.pdf", "lang": "en"}},
{"id": "doc_2", "values": [0.2, 0.2, 0.2, 0.2], "metadata": {
"author": "Bob", "rating": 3.0, "year": 2023, "price": 45.00,
"published": False, "tags": ["cooking"], "title": "A Book",
"filename": "notes.txt", "lang": "es"}},
{"id": "doc_3", "values": [0.3, 0.3, 0.3, 0.3], "metadata": {
"author": "Charlie", "rating": 5.0, "year": 2026, "price": 25.00,
"published": True, "tags": ["ai"], "title": "Theory",
"filename": "paper.pdf", "lang": "fr"}},
])
query_embedding = [0.1, 0.1, 0.1, 0.1]def matched(filter, top_k=10):
return [hit["id"] for hit in index.search(vector=query_embedding, filter=filter, top_k=top_k)]
# doc_3 is the furthest of the three from the query, and it is still the only
# thing the filter admits, so it is what a page of one holds
print(matched({"author": "Charlie"}, top_k=1))
print(matched({"author": "Charlie"}, top_k=10))Output
['doc_3']
['doc_3']
A filter matching fewer records than top_k returns that many results, and one matching none returns an empty list. Neither is a truncation.
# Find high-quality recent documents
print(matched({"published": True, "rating": {"gte": 4.0}, "year": {"gte": 2024}}))
# Find documents by specific authors
print(matched({"author": {"in": ["Alice", "Bob"]}}))
# Find AI-related content
print(matched({"tags": {"contains": "ai"}}))
# Find documents in a price range
print(matched({"price": {"gte": 20.0, "lte": 40.0}}))
# Find documents with a specific file type
print(matched({"filename": {"endswith": ".pdf"}}))
# Match on a title prefix
print(matched({"title": {"startswith": "The"}}))
# Exclude an author
print(matched({"author": {"ne": "Alice"}}))
# Match a whole array
print(matched({"tags": ["ai"]}))
# Either a top rating or a recent year, which needs a disjunction
print(matched({"$or": [{"rating": {"gte": 5.0}}, {"year": {"gte": 2024}}]}))
# Published, and either English or cheap
print(matched({"published": True,
"$or": [{"lang": "en"}, {"price": {"lt": 26.0}}]}))
# Everything except Bob's, including any record with no author at all
print(matched({"$not": {"author": "Bob"}}))Output
['doc_1', 'doc_3']
['doc_1', 'doc_2']
['doc_1', 'doc_3']
['doc_1', 'doc_3']
['doc_1', 'doc_3']
['doc_1', 'doc_3']
['doc_2', 'doc_3']
['doc_3']
['doc_1', 'doc_3']
['doc_1', 'doc_3']
['doc_1', 'doc_3']
A sparse space indexes term counts beside the dense vectors. Declare it with a tokenizer and records carry a text field, which the tokenizer splits and the space counts.
from zeusdb_vector_database import VectorDatabase
vdb = VectorDatabase()
index = vdb.create("hnsw", dim=4, sparse={"tokenizer": "simple"})
index.add([
{"id": "d1", "values": [0.1, 0.1, 0.1, 0.1],
"text": "the quick brown fox jumps over the lazy dog"},
{"id": "d2", "values": [0.2, 0.2, 0.2, 0.2],
"text": "a quick brown dog outpaces a lazy fox"},
{"id": "d3", "values": [0.3, 0.3, 0.3, 0.3],
"text": "the rain in spain falls mainly on the plain"},
])
for hit in index.query(arms=[{"text": "quick fox"}], top_k=2):
print(hit["id"], round(hit["score"], 4))Output
d2 0.9705
d1 0.9254
| Key | Takes | Means |
|---|---|---|
tokenizer |
"simple" or a callable |
How a text becomes terms. Declaring it makes the space a text layer |
weighting |
"dot", "bm25", or a mapping |
How a term count scores. Defaults to bm25 where a tokenizer is declared. {"type": "bm25", "k1": 1.5, "b": 0.6} sets the parameters, which otherwise take 1.2 and 0.75 |
name |
a string | The directory the space saves under. Defaults to sparse |
Scores are similarities. Higher is better, unlike the dense search(), whose scores are distances. A space with a text layer takes text and refuses sparse.
The built-in tokenizer splits on anything that is not a letter or a digit and lowercases. It does no stemming, no stopword removal and no segmentation of a script written without spaces, so state-of-the-art is four terms and don't is two. A caller who needs more supplies their own.
def tokenize(text):
return [w.strip(".,!?").lower() for w in text.split() if len(w) > 2]
index = vdb.create("hnsw", dim=4, sparse={"tokenizer": tokenize})
index.add({"id": "d1", "values": [0.1, 0.1, 0.1, 0.1],
"text": "The quick brown fox."})
print(index.get_stats()["term_count"])Output
4
A tokenizer takes a str and returns an iterable of str, one per term, in order and with every repeat, since the count is what the space stores. An exception it raises comes back as itself from query(), and from add() as that record's error naming the class.
Reopening an index built with your own tokenizer needs the same tokenizer handed back, because the directory records that one was used and cannot reproduce it.
index.save("my_index.zdb")
reopened = vdb.load("my_index.zdb", tokenizer=tokenize)vdb.load("my_index.zdb") on such a directory raises, since the directory records that a tokenizer of your own was used and cannot reproduce it. It records nothing that identifies which one, so a different callable opens it without complaint and tokenizes queries its own way. An index built with "simple" reopens with nothing handed.
A caller running their own sparse encoder declares the space without a tokenizer and supplies the pairs directly. The dimensions are the caller's own term ids and the values are the caller's own weights.
from zeusdb_vector_database import VectorDatabase
vdb = VectorDatabase()
index = vdb.create("hnsw", dim=4, sparse={"name": "encoder"})
index.add([
{"id": "d1", "values": [0.1, 0.1, 0.1, 0.1],
"sparse": {"dims": [12, 977, 4021], "values": [0.41, 1.20, 0.08]}},
{"id": "d2", "values": [0.2, 0.2, 0.2, 0.2],
"sparse": {"dims": [12, 350], "values": [0.90, 0.15]}},
])
hits = index.query(
arms=[{"sparse": {"dims": [12, 977], "values": [0.9, 0.3]}}],
top_k=2,
)
for hit in hits:
print(hit["id"], round(hit["score"], 4))Output
d2 0.81
d1 0.729
| Key | Takes | Means |
|---|---|---|
dims |
a list of integers | Term ids, strictly increasing |
values |
a list of numbers | The weight of each term |
The default weighting is dot, which is the sum of products, and it leaves a trained encoder's weights alone. A space declared this way takes sparse and refuses text.
A caller counting terms themselves declares "weighting": "bm25" and supplies raw counts. The engine then applies the saturation, the length normalisation and the rarity over the records the query admits.
index = vdb.create("hnsw", dim=4, sparse={"name": "counts", "weighting": "bm25"})
index.add({"id": "d1", "values": [0.1, 0.1, 0.1, 0.1],
"sparse": {"dims": [3, 9, 41], "values": [2.0, 1.0, 1.0]}})Under bm25 a value must be a whole number above zero, since a term frequency is a count. A trained encoder's weight is refused with the value named, which is what tells you the space wants dot instead.
query() runs several arms over one set of candidates and fuses their pages by rank. An arm is a dense vector, a sparse vector or a text, and every arm shares the filter.
from zeusdb_vector_database import VectorDatabase
vdb = VectorDatabase()
index = vdb.create("hnsw", dim=4, space="l2", sparse={"tokenizer": "simple"})
index.add([
{"id": "d1", "values": [0.1, 0.1, 0.1, 0.1],
"text": "the quick brown fox", "metadata": {"lang": "en"}},
{"id": "d2", "values": [0.9, 0.9, 0.9, 0.9],
"text": "a lazy brown dog", "metadata": {"lang": "en"}},
{"id": "d3", "values": [0.2, 0.2, 0.2, 0.2],
"text": "un renard brun rapide", "metadata": {"lang": "fr"}},
])
hits = index.query(
arms=[{"vector": [0.1, 0.1, 0.1, 0.1]}, {"text": "quick fox"}],
filter={"lang": "en"},
top_k=2,
)
for hit in hits:
print(hit["id"], round(hit["score"], 4),
[(c["arm"], c["rank"]) for c in hit["contributions"]])Output
d1 0.0328 [(0, 1), (1, 1)]
d2 0.0161 [(0, 2)]
| Argument | Takes | Means |
|---|---|---|
arms |
a list of mappings | One to eight arms. Each names vector, sparse or text |
filter |
a filter | Applied once, to every arm |
top_k |
an integer | The page size |
fetch |
an integer | What each arm contributes. Defaults to top_k for one arm and five times it for several |
fusion |
"rrf" or a mapping |
How the pages combine. {"type": "rrf", "k": 60.0} sets the constant |
A hit carries its rank and score on every arm's page it appeared on, under contributions, indexed by the arm's position in arms. A fused score is a rank sum and means nothing on its own, which is what the contributions are for.
An arm takes the options its own kind takes. A dense arm takes ef_search and rerank as search() does, and a sparse or text arm takes idf.
hits = index.query(
arms=[
{"vector": [0.1, 0.1, 0.1, 0.1], "ef_search": 200},
{"text": "quick fox", "idf": "global"},
],
top_k=2,
fusion={"type": "rrf", "k": 30.0},
)| Key | On | Means |
|---|---|---|
ef_search |
a dense arm | The search width, as search() takes it |
rerank |
a dense arm | How many candidates to rescore against true vectors |
idf |
a sparse or text arm | "corpus" counts term rarity over the records the filter admits, "global" over every record. Defaults to "corpus" |
A one arm query is that arm's search. A single dense arm returns the page search() returns, id for id and score for score, with each hit carrying its contributions as well. query() is the shape to reach for whenever more than one arm is in play.
explain() takes the same arguments as query() and returns what the query would do, without running it.
plan = index.explain(
arms=[{"vector": [0.1, 0.1, 0.1, 0.1]}, {"text": "quick fox"}],
filter={"lang": "en"},
top_k=10,
)
print(plan["admit"])
for arm in plan["arms"]:
print(arm["kind"], arm["fetch"], round(arm["cost_ns"]))Output
{'shape': 'sorted', 'admitted': 2}
dense 50 235
sparse 50 22
| Key | Means |
|---|---|
admit |
The shape the filter compiled to, and how many records it admits |
arms |
One entry per arm, carrying its space, kind, fetch, estimated cost and whether its page is exact |
fusion |
The rule the pages combine under, or None for one arm |
cost_ns is an arm's own estimate of its own work, in nanoseconds, and it is a figure to compare arms of one query by. It leaves out the filter's evaluation and the page's assembly, so a wall time you measure will exceed it.
The admit shape says how the filter was answered.
| Shape | Means |
|---|---|
all |
Every live record, either no filter or one that admits all of them |
bitmap |
A filter every field of which is declared, answered from the columns |
sorted |
A short list of matching records, walked exactly |
bounded |
A walk that stopped at a bound, so the count is an upper limit |
predicate |
A walk that gave up, so each candidate is tested as it is visited |
An index holding a sparse space reports it in get_stats() beside the dense figures.
stats = index.get_stats()
for key in ("sparse_space", "sparse_weighting", "sparse_records",
"sparse_postings", "sparse_memory_mb"):
print(key, stats[key])Output
sparse_space sparse
sparse_weighting bm25
sparse_records 3
sparse_postings 12
sparse_memory_mb 0.00
| Key | Means |
|---|---|
sparse_space |
The space's name |
sparse_weighting |
dot or bm25 |
sparse_records |
Records the space holds, which may be fewer than the index holds |
sparse_postings |
Live entries across every record, one per distinct term per record |
sparse_dead_postings |
Occurrences left by removed records, reclaimed at compact() |
sparse_memory_mb |
What the postings and their index hold |
sparse_tokenizer |
simple or external, on a text layer alone |
term_count |
Distinct terms the dictionary holds, on a text layer alone |
dictionary_memory_mb |
What the dictionary holds, on a text layer alone |
These keys are present on a sparse index and absent otherwise, and total_memory_mb sums them in. A journaled index carries five journal_ keys the same way; see Journaling an Index.
ZeusDB Vector Database includes structured logging that works automatically out of the box while providing customization for advanced users.
For most users, logging works automatically with sensible defaults:
from zeusdb_vector_database import VectorDatabase
# Logging is automatically configured, no setup required
vdb = VectorDatabase()
index = vdb.create("hnsw", dim=1536)
# Operations are automatically logged with structured data
result = index.add({"ids": ids, "embeddings": vectors})
results = index.search(query_vector, top_k=5)What you get automatically:
- ✅ Quiet by default, only warnings and errors outside development
- ✅ Environment detection, appropriate defaults for dev, prod, testing, CI and notebooks
- ✅ Structured JSON logs in production environments
- ✅ Human-readable logs in development environments
- ✅ Operation timing on index creation, additions, training, compaction and saves at
info, and on searches atdebug - ✅ Cross-platform compatibility
save() and load() write their progress here too, at debug, so they print nothing on stdout. They used to print it directly and it was not affected by any of the settings below.
Control logging behavior with environment variables:
export ZEUSDB_LOG_LEVEL=debug
python your_app.pyexport ZEUSDB_LOG_LEVEL=error
export ZEUSDB_LOG_FORMAT=json
export ZEUSDB_LOG_TARGET=file
export ZEUSDB_LOG_FILE=/var/log/zeusdb/app.log
python your_app.py| Variable | Options | Default | Description |
|---|---|---|---|
ZEUSDB_LOG_LEVEL |
trace, debug, info, warn, error |
warn (dev), error (prod) |
Controls log verbosity. warning and warn are the same level, as are critical, fatal and error. An unrecognised name falls back to the default. |
ZEUSDB_LOG_FORMAT |
human, json |
human (dev), json (prod) |
Output format |
ZEUSDB_LOG_TARGET |
stdout, stderr, file |
stderr |
Where logs go |
ZEUSDB_LOG_FILE |
/path/to/file.log |
zeusdb.log |
Log file path, written exactly as given (if target=file) |
ZEUSDB_LOG_ROTATION |
daily, never |
never |
With daily, a UTC date is appended to the file name |
ZEUSDB_LOG_CONSOLE |
true, false |
Auto-detected | Force console output |
ZEUSDB_DISABLE_AUTO_LOGGING |
true, 1, yes |
unset | Skip automatic configuration entirely |
RUST_LOG |
standard env_logger syntax |
unset | Overrides ZEUSDB_LOG_LEVEL for the Rust layer |
Under ZEUSDB_LOG_ROTATION=daily with ZEUSDB_LOG_FILE=logs/app.log, two files appear: logs/app.log and a dated logs/app.log.2026-08-05. Rotation applies to the Rust layer, which writes the dated one.
The system detects your environment and applies appropriate defaults:
- 🏭 Production (
ENVIRONMENT=production, or Kubernetes or Docker markers): ERROR level, JSON format, file output - 💻 Development (default): WARNING level, human format, console output
- 🧪 Testing (
ENVIRONMENT=testing,PYTEST_CURRENT_TEST, orpytestimported): CRITICAL level, minimal output - 📓 Jupyter (
JUPYTER_SERVER_ROOT,JPY_PARENT_PID, or IPython imported): INFO level, human format - 🔄 CI/CD (
CI,GITHUB_ACTIONS,GITLAB_CI): WARNING level, human format for readability
Environment variables always override the detected defaults.
For enterprise environments with existing logging infrastructure:
import os
os.environ["ZEUSDB_DISABLE_AUTO_LOGGING"] = "1"
# Now configure your own logging before importing ZeusDB
import logging
logging.basicConfig(level=logging.INFO, format='%(message)s')
from zeusdb_vector_database import VectorDatabase # Will respect your existing logging setupimport os
os.environ["ZEUSDB_DISABLE_AUTO_LOGGING"] = "1"
import zeusdb_vector_database
# JSON to stdout
success = zeusdb_vector_database.init_logging(level="info")
# OR JSON to a directory of daily rotating files. Pick one, not both.
# success = zeusdb_vector_database.init_file_logging(
# log_dir="/var/log/myapp",
# level="debug",
# file_prefix="zeusdb"
# )
print("initialized:", success)
vdb = zeusdb_vector_database.VectorDatabase()Only the first initializer to run takes effect. Both functions return True if they installed the subscriber and False if one was already installed, so calling both leaves the second with no effect and a False return. zeusdb_vector_database.is_logging_initialized() reports whether either has run.
The file target is drained at exit. Records reach the file through a background writer, so a record emitted immediately before the process ends is still in flight when it ends. Importing the package registers the drain with atexit, which covers a normal exit and needs no call. zeusdb_vector_database.shutdown_logging() runs the same drain on demand and returns True if it drained a file appender, or False if there was nothing to drain, which is the answer for the stdout and stderr targets and for a second call. It closes the file, so records emitted after it are discarded. Nothing runs on os._exit or on a crash, and records still in flight at either are lost.
import logging
import os
# Disable auto-configuration
os.environ["ZEUSDB_DISABLE_AUTO_LOGGING"] = "1"
# Set up your own logger first
logger = logging.getLogger("myapp.zeusdb")
logger.setLevel(logging.INFO)
# Configure Rust logging to match
os.environ["ZEUSDB_LOG_LEVEL"] = "info"
os.environ["ZEUSDB_LOG_FORMAT"] = "json"
from zeusdb_vector_database import VectorDatabase
# ZeusDB will integrate with your logging setup2026-08-05T12:19:39.261318Z INFO build: HNSW index created successfully operation="index_creation_complete" dim=8 space=cosine m=16 ef_construction=200 expected_size=10000 has_quantization=false duration_ms=0
2026-08-05T12:19:39.3491294Z INFO add: Vector addition completed operation="add_vectors_complete" total_inserted=2 total_errors=0 success_rate=100.0 duration_ms=87 overwrite_mode=true final_storage_mode="raw_only"
{"timestamp":"2026-08-05T12:19:39.4853862Z","level":"INFO","fields":{"message":"HNSW index created successfully","operation":"index_creation_complete","dim":8,"space":"cosine","m":16,"ef_construction":200,"expected_size":10000,"has_quantization":false,"duration_ms":"0"},"target":"zeusdb_vector_database::hnsw_index","filename":"src\\hnsw_index.rs","line_number":1068,"threadId":"ThreadId(1)"}Every operation ends with a record that carries operation and duration_ms. What you see depends on the level: the records below are at info, a search's is at debug, and the default level is warn in development and error in production, so a process at the default writes none of them. Set ZEUSDB_LOG_LEVEL=info to see the operation records.
| Record | Level | Carries |
|---|---|---|
index_creation_complete |
info |
the configuration and duration_ms |
add_vectors_complete |
info |
total_inserted, total_errors, success_rate, duration_ms, final_storage_mode |
pq_training_complete, int8_training_complete |
info |
duration_ms |
compact_complete |
info |
nodes_before, nodes_after, nodes_reclaimed, live_records, duration_ms |
save_complete |
info |
path and duration_ms, which is also how long the save held writes |
search_complete, query_complete |
debug |
results_count and duration_ms |
Per-call search latency is the caller's own clock around the call. The engine's reading of it exists at debug alone, and enabling debug costs a search four to seven percent before a byte is written, so a process that wants a latency series measures it from the outside.
# Monitor error rates
grep '"level":"ERROR"' /var/log/zeusdb/app.log | wc -l
# Track how long each add() and save() took, at ZEUSDB_LOG_LEVEL=info
grep '"operation":"add_vectors_complete"' /var/log/zeusdb/app.log | jq '.fields.duration_ms'
grep '"operation":"save_complete"' /var/log/zeusdb/app.log | jq '.fields.duration_ms'
# Watch quantization training
grep '"operation":"pq_training' /var/log/zeusdb/app.logLogs not appearing?
# Check if auto-logging is disabled
echo $ZEUSDB_DISABLE_AUTO_LOGGING
# Verify the level is one both layers accept
ZEUSDB_LOG_LEVEL=debug python -c "import zeusdb_vector_database as z; print(z.is_logging_initialized())"File logging not working?
# Check permissions
ls -la /path/to/log/directory
# Test with console first
ZEUSDB_LOG_TARGET=stderr ZEUSDB_LOG_LEVEL=info python your_app.pyA process that ends through os._exit or a crash skips the exit drain, so its final records never reach the file. Call zeusdb_vector_database.shutdown_logging() before such an exit.
Want to see Rust logs specifically?
# Enable trace level to see all Rust operations
ZEUSDB_LOG_LEVEL=trace python your_app.py- File logging is non-blocking: records are handed to a background writer rather than written on the calling thread. The exit drain waits for that writer to finish, for up to about a second.
- Enabling
debugortracecosts a search four to seven percent before any byte is written, in the span and the events it formats, and a file or terminal target adds the write on top. Leave production aterrororwarn, and useinfowhen you want the operation records.
export ZEUSDB_LOG_LEVEL=debug
export ZEUSDB_LOG_FORMAT=humanexport ZEUSDB_LOG_LEVEL=info
export ZEUSDB_LOG_FORMAT=json
export ZEUSDB_LOG_TARGET=file
export ZEUSDB_LOG_FILE=logs/zeusdb-staging.log
export ZEUSDB_LOG_ROTATION=dailyexport ENVIRONMENT=production
export ZEUSDB_LOG_LEVEL=error
export ZEUSDB_LOG_FORMAT=json
export ZEUSDB_LOG_TARGET=file
export ZEUSDB_LOG_FILE=/var/log/zeusdb/production.log
export ZEUSDB_LOG_ROTATION=dailyThis project is licensed under the Apache License 2.0.