Skip to content

[vector store 12/13] Let a deployment tune a Qdrant collection's HNSW, optimizers and quantization (speedkick) - #1618

Draft
edwinyyyu wants to merge 73 commits into
MemMachine:speedkickfrom
edwinyyyu:feat/qdrant-collection-options-speedkick
Draft

edwinyyyu wants to merge 73 commits into
MemMachine:speedkickfrom
edwinyyyu:feat/qdrant-collection-options-speedkick

Conversation

@edwinyyyu

@edwinyyyu edwinyyyu commented Sep 14, 2026

Copy link
Copy Markdown
Contributor

Purpose of the change

Lets a deployment tune the HNSW index, the optimizers and the quantization of a Qdrant collection. Split out of #1606 on review, since it is a tangential change to the store-as-collection work; it comes over from default (a0753d3).

Stack

Slice 12 of 13, every PR targeting speedkick; merge bottom-up.

# PR change
1 #1606 (merged) Remove per-project filterable properties
2 #1597 Add session, source and expansion to EventMemory
3 #1622 Create a session's storage with the session, never on a request
4 #1623 Make the examples that write to a project create it first
5 #1624 Make no memory request create a project
6 #1625 Remove open-or-create and close from both stores
7 #1626 Rename logical collection to partition, and open to get, on both stores
8 #1627 Make a vector store one collection, with string-keyed partitions
9 #1631 Mint an incarnation per partition life; delete logically, reclaim by purge
10 #1628 Make a vector store filter only on the properties it declares
11 #1630 Bound every request to a remote vector store by a configured timeout
12 #1618 (this PR) Let a deployment tune a Qdrant collection's HNSW, optimizers and quantization
13 #1616 Close the filter union, and make negation the complement on every backend

This PR's own change is its last commit, f08d6a71 (6 files changed, 257 insertions(+), 9 deletions(-)); the rest of its diff is the slices under it, and drops out as they merge. Stacked on #1630; #1616 is stacked on it.

What changed

  • QdrantConf gains hnsw_config, optimizers_config and quantization_config: plain mappings mirroring qdrant-client's HnswConfigDiff, OptimizersConfigDiff and QuantizationConfig (for example {"turbo": {"always_ram": true, "bits": "bits2"}}), so qdrant-client stays optional for configuration parsing. DatabaseManager passes them to QdrantVectorStoreParams, which validates them against qdrant's own models and reports a failure as a configuration error.
  • They apply to the store's data collection, never to its registry collection.
  • m must be 0 or unset: the collection is multi-tenant and disables the global graph in favor of per-partition payload indexing, so a deployment tunes payload_m. It defaults to 16, as before.

Tests

Plain mappings coerce to qdrant's models; m of 0 or unset is accepted and any other value is refused naming payload_m; the native HNSW config merges overrides and pins m at 0; the overrides reach the data collection's creation and not the registry's. Configuration parsing and the manager's wiring assert the three fields round-trip.

uv run pytest packages/server/server_tests packages/client/client_tests: 2375 passed, 2 skipped, on the slice-13 tree (2026-09-15, on #1597 tip 4149fd5); this tip passes its own run. ruff check, ruff format --check and ty check --project packages/server clean.

🤖 Generated with Claude Code

https://claude.ai/code/session_01ESpWYTmCR7X3bJEpoA8SAn

@edwinyyyu
edwinyyyu marked this pull request as draft September 14, 2026 20:03
@edwinyyyu
edwinyyyu force-pushed the feat/qdrant-collection-options-speedkick branch 4 times, most recently from 28884f8 to 6ef5caf Compare September 14, 2026 20:33
@edwinyyyu
edwinyyyu force-pushed the feat/qdrant-collection-options-speedkick branch from 6ef5caf to 1364e65 Compare September 14, 2026 21:40
@edwinyyyu edwinyyyu changed the title (Depends on #1606) Let a deployment tune a Qdrant collection's HNSW, optimizers and quantization (speedkick) [vector store 12/13] Let a deployment tune a Qdrant collection's HNSW, optimizers and quantization (speedkick) Sep 14, 2026
@edwinyyyu
edwinyyyu force-pushed the feat/qdrant-collection-options-speedkick branch 5 times, most recently from 556eb26 to 2a1da2c Compare September 14, 2026 23:16
@edwinyyyu edwinyyyu changed the title [vector store 12/13] Let a deployment tune a Qdrant collection's HNSW, optimizers and quantization (speedkick) [vector store 11/12] Let a deployment tune a Qdrant collection's HNSW, optimizers and quantization (speedkick) Sep 14, 2026
@edwinyyyu
edwinyyyu force-pushed the feat/qdrant-collection-options-speedkick branch from 2a1da2c to 556eb26 Compare September 14, 2026 23:21
edwinyyyu and others added 4 commits September 14, 2026 16:35
Implements the session/source/expansion/eviction part of
design/event_memory_handoff.md from the tenant-lifecycle branch. The
context-part and block-kind model and the segmenter/deriver tables
follow in a second change; this one keeps the producer/null context
union and the `block_type` discriminator as they are.

Data models. `Event`, `Segment` and `Derivative` carry `session_id`
and `source_id` as nullable fields, copied verbatim down the
pipeline; null is encoded as a missing record key, `None` in a typed
id list selects it, and property values stay `None`-free. `SearchHit(score, seed, segments)` replaces
`ScoredSegmentContext` and `QueryResult`; `Neighborhood(before,
after)` and `EvictionOptions` are added.

Reserved keys. `common/property_keys.py` reserves the `memmachine_`
namespace; every system value a search filters on at the vector stage
sits in the record under a reserved key (`event_timestamp`,
`event_session`, `event_source`, `block_kind`). `utils.py` owns the
translation between the typed filters (`since`, `until`, `session_ids`,
`source_ids`, `block_kinds`) and filter trees; a caller key in the
namespace is rejected before any segment is written.

Segment store. `segment_store_sg` gains `session_id`, `source_id` and
`block_kind` columns, a session-led ordering index and a source index; the total order is
`(timestamp, event_uuid, index, offset)`, windows and neighborhoods
are confined to the seed's session, and a null session is one stream.
`get_segment_windows` takes `before`/`after` and the typed filters;
`get_segment_neighborhoods` returns the neighbors and never the seed, as two
lists; `delete_derivatives` unlinks without touching segments; PG
lateral reads run one statement pair per seed session. No migration:
`startup()` keeps `create_all`, and an existing speedkick database is
recreated; schema migration waits for the lifecycle/DDL changes.

EventMemory. `encode_events` forgets the batch first, so a repeat
leaves one copy, and runs eviction from the agentic_expansion branch,
cosine only: batch predecessors, one bounded neighbor query per
derivative, a cluster over `target_size` trimmed from its temporal
middle, displaced records and their links deleted, skipped ones never
written. `query` is the vector stage and returns hits with the seed's
index in its window; `rerank` is the second stage, static, for a
caller with a reranker; `expand` walks a neighborhoods from a segment
or event anchor; `render` replaces the string formatters. The reranker
and the per-call format options leave the constructor and the call,
respectively; the deriver's format is fixed per memory.

Server. `LongTermMemory` sets `Event.source_id` from the producer id,
leaves the session null and keeps the producer context; it reranks after `query` and reads
hits.

Tests: the branch's neighbor and eviction tests ported to the new
shapes on both dialects, plus session confinement, half-open bounds,
instant comparison of zoned bounds on SQLite, source and kind filters,
and link deletion. Each new store assertion was checked to fail
against a mutated store (no session predicate, an inclusive `until`, a
filtered neighborhoods seed, an unnormalized bound).

Rebased 2026-09-10 onto speedkick after MemMachine#1598 merged, adopting its
post-review names (`segment_by_derivative`, `seed_cosine_similarities`).
`common/property_keys.py` and its test, which MemMachine#1598 did not carry into
its merge, are included here.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MuAu353FiSmCJjLX1LWDQW
Claude-Session: https://claude.ai/code/session_01YBbQgZiCqeoLu83EkbEFHE
Every field of Event, Segment and Derivative carries a description;
the byte bound on ids is a validator on the model, so
EventMemory._validate_events checks only property keys; positions are
validated non-negative. SearchHit.seed is seed_index, checked to lie
inside segments, and the window around a seed is a segment window.
EvictionOptions.similarity_threshold is cosine_similarity_threshold,
the threshold at or above which eviction is considered; the other two
options say what is fetched and what is kept.

The store contract's class docstring only contrasts the two reads;
the details live on each method. Typed id lists hold ids only, and a
seed with no session walks every session: events in no session do not
belong together, so "no session" is not a value a list can name and
the only timeline to show around such a seed is everything. The
timestamp ordering index segment_store_sg__in_ts_ev_ix_of is restored
for that walk beside the session-led one.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YBbQgZiCqeoLu83EkbEFHE
The memory makes no use of either bound, so the caller discards hits
itself; the vector stage keeps its limit because the store uses it.
Every hit comes back rescored, in descending score.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YBbQgZiCqeoLu83EkbEFHE
cosine_similarity_matrix in eviction, and cosine similarity in every
docstring, comment and test name that named a bare similarity.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YBbQgZiCqeoLu83EkbEFHE
`_immediately_follows` said less than it checked: an event's first
piece immediately follows the previous event's last, and the check is
for the next piece of the same event, which is a continuation, the word
the render loop already used. The `rerank` docstring states what comes
back and nothing about the caller.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YBbQgZiCqeoLu83EkbEFHE
@edwinyyyu
edwinyyyu force-pushed the feat/qdrant-collection-options-speedkick branch from d1df9aa to 21861c6 Compare September 15, 2026 18:00
The docstring named context parts and block renderings, which arrive
with the blocks change; here a header is the timestamp and the
producer's name, and the pieces' text follows.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YBbQgZiCqeoLu83EkbEFHE
@edwinyyyu
edwinyyyu force-pushed the feat/qdrant-collection-options-speedkick branch from 21861c6 to 599d9b7 Compare September 15, 2026 19:01
edwinyyyu and others added 2 commits September 15, 2026 12:06
`render` said nothing about what it takes; `Block.render` and the
context parts' `render` are per-object, and the memory's is over
segments.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YBbQgZiCqeoLu83EkbEFHE
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YBbQgZiCqeoLu83EkbEFHE
@edwinyyyu
edwinyyyu force-pushed the feat/qdrant-collection-options-speedkick branch 2 times, most recently from 9db05f7 to 397f451 Compare September 15, 2026 19:16
edwinyyyu and others added 2 commits September 15, 2026 12:38
With three secondary indexes led by `incarnation`, PostgreSQL's planner
had, on a table it has no statistics for, a cost tie between the primary
key and `(incarnation, event_uuid)` for the link table's foreign-key
check `incarnation = $1 AND uuid = $2`, and took the latter: an index
scan on the incarnation alone with the uuid as a filter over every row
of the partition, 57 ms per thousand links against 2.6 ms on the primary
key, and growing with the table (measured with `EXPLAIN ANALYZE` on a
fresh table; the old store's identical tie fell the other way). With the
incarnation second, no secondary index can serve that lookup at all, so
the primary key is the only candidate and no statistics are needed.
Every read that used a secondary index names its leading key and the
incarnation together, so it is served as before: walks by session (and
source), lookups by event; `EXPLAIN` on both dialects shows the same
index conditions. Ingest of a thousand segments with links on a fresh
PostgreSQL table: 246 ms before, 82 ms after, the old store's 74-105.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YBbQgZiCqeoLu83EkbEFHE
The lateral walk ran one statement pair per distinct seed session, the
session a literal so the session-led index would serve it: a search
whose twenty seeds lie in twenty sessions ran forty statements and took
53 ms against the old store's 5.7 ms (twenty thousand segments, forty
sessions). The seeds are now one row set, `unnest` over six bound
arrays with the seed's session among them, and the lateral pins
`session_id` to the seed row's; the planner parameterizes that
equality per seed, so the index serves every seed in one statement per
direction (`EXPLAIN` shows the index condition on
`seeds.seed_session_id`), and six parameters whatever the seed count
keep the statement small and cacheable by shape. The seeds are located
by their order keys alone, without their payload columns or ORM rows.
The search-shaped read is 7.6-9.6 ms now: the old store's 5.7 plus the
walk's own locate statement, which keying the walk by uuid costs; a
filtered one is 8-10 ms against the old store's 42, since the filter
selects the seeds before any walk.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YBbQgZiCqeoLu83EkbEFHE
@edwinyyyu
edwinyyyu force-pushed the feat/qdrant-collection-options-speedkick branch 2 times, most recently from f2f700b to bfe1a5c Compare September 15, 2026 20:19
The original lateral shape serves the new walk: the seeds subquery,
which selects the seed rows by uuid inside the statement, gains the
seed's `session_id`, and the lateral pins `session_id` to it, an
equality the planner parameterizes per seed. That is the whole
difference from speedkick's walk, and it makes `_SeedKey`, the locate
of seed keys, the `unnest` row set and `_session_condition` unnecessary;
the seed rows the entry point already fetched drive the SQLite loop as
they did before. Same plan (`EXPLAIN`: the session-led index with the
seed's session as an index condition) and the same cost as the bound
row set: a 20-seed search 7.9-8.5 ms against the old store's 6.5-7.3 on
one container run.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YBbQgZiCqeoLu83EkbEFHE
@edwinyyyu
edwinyyyu force-pushed the feat/qdrant-collection-options-speedkick branch from bfe1a5c to 33124f2 Compare September 15, 2026 20:26
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YBbQgZiCqeoLu83EkbEFHE
@edwinyyyu
edwinyyyu force-pushed the feat/qdrant-collection-options-speedkick branch from 33124f2 to 612066d Compare September 15, 2026 20:30
edwinyyyu and others added 13 commits September 15, 2026 14:36
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YBbQgZiCqeoLu83EkbEFHE
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YBbQgZiCqeoLu83EkbEFHE
…quest

The event backend created a session's vector store collection and segment
store partition on the first request that opened the session, so a search
or a write for an unknown session created storage as a side effect, and
the service locator was the only place that knew both stores' create paths.

The owner is the session. Every path that creates a session row runs
through EpisodicMemoryManager._create_session, which inserts the row and,
when the row is new, creates the session's partitions in its segment store
and its vector store (create_episodic_memory_storage); an equivalent
re-create accepts the row and leaves the storage as it is. The request
path binds handles with the stores' lookups and raises
SessionPartitionMissingError when a partition is absent: a session without
its storage is broken, not new. Deleting a session with no open instance
deletes its partitions by key, so a session whose storage was never fully
created can still be deleted. MemMachine.create_session goes through the
manager for the same reason.

The semantic manager owns its one collection and creates it, once, at the
storage's first use. With that, nothing calls the stores' open-or-create.

The API is unchanged: the manager's open-or-create still creates a session
a memory request names, now through the same path.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ESpWYTmCR7X3bJEpoA8SAn
The simple chatbot example, the TypeScript REST demo and the Dify plugin's
add-memory tool wrote to a project without creating it, relying on the
write to create it. Each now creates its project before its first memory
request and accepts 409 as the project already existing. No behavior
changes for them; they stop depending on a write creating a project.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ESpWYTmCR7X3bJEpoA8SAn
Adding memories to, or searching, a project that did not exist created it,
with the server's default configuration, without the caller's knowledge.
Now only the create-project request creates a project: a write or a
search opens the session and answers 404 for an unknown project, as the
search endpoint already promised; the manager's open-or-create goes.

Two callers depended on the implicit creation. `org_id` and `project_id`
default to `universal`, so the API promises the project
`universal/universal`; the server creates it, once, at startup, and leaves
one that already exists as it is. The MCP add tool names its own project
and has no create-project counterpart, so it creates the project it writes
to, once, and says so. The API doc strings and the OpenAPI document say
which requests create a project.

A breaking API change on `speedkick`.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ESpWYTmCR7X3bJEpoA8SAn
Nothing calls them since a session's storage is created with the session:
`open_or_create_collection` and `close_collection` leave the vector store
interface and its four backends, `open_or_create_partition` and
`close_partition` leave the segment store interface and its
implementation, and the two config-mismatch errors that only open-or-create
raised go with them. A store creates on `create_*`, strictly, and looks up
on `open_*`, answering None; create-if-absent is the owner's, where the
key's provenance is known.

Source changes are deletions only. The tests that exercised
open-or-create as a fixture use a test-side create-if-absent instead, and
the tests of its own semantics go.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ESpWYTmCR7X3bJEpoA8SAn
…res' lookup to get_partition

A vector store's logical collection becomes a partition, the segment store's word for the same thing, and both stores' lookup is get_partition, answering None like a Python get. Identifiers only, produced by the script below; the (namespace, name) identity, the per-partition config and every docstring are as they were, and the next change gives them their meaning. The native clients' create_collection and delete_collection keep their names.

```sh
set -e
cd "$(git rev-parse --show-toplevel)"
git mv packages/server/server_tests/memmachine_server/common/vector_store/in_memory_vector_store_collection.py \
       packages/server/server_tests/memmachine_server/common/vector_store/in_memory_vector_store_partition.py
git ls-files -z 'packages/server/*.py' | xargs -0 perl -0pi -e '
  s/VectorStoreCollection(?!Config)/VectorStorePartition/g;
  s/in_memory_vector_store_collection/in_memory_vector_store_partition/g;
  s/vector_store_collection(?!_schema|_namespace)/vector_store_partition/g;
  s/open_collection/get_partition/g;
  s/def create_collection\(/def create_partition(/g;
  s/def delete_collection\(/def delete_partition(/g;
  s/\.create_collection\((\s*namespace=)/.create_partition($1/g;
  s/\.delete_collection\((\s*namespace=)/.delete_partition($1/g;
  s/\.create_collection(?=\s*=\s*AsyncMock|\.assert_)/.create_partition/g;
  s/\.delete_collection(?=\s*=\s*AsyncMock|\.assert_)/.delete_partition/g;
  s/"create_collection"/"create_partition"/g;
  s/"delete_collection"/"delete_partition"/g;
  s/only delete_collection is invoked/only delete_partition is invoked/g;
  s/test_delete_collection_/test_delete_partition_/g;
  s/open_partition/get_partition/g;
'
uv run ruff check --fix --quiet packages/server
uv run ruff format --quiet packages/server
```

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ESpWYTmCR7X3bJEpoA8SAn
…uilt by the composition root

A vector store was a factory of logical collections, each identified by a
(namespace, name) pair and created with its own dimensions and schema; a
backend that limits native collections shared one among logical
collections of equal configuration, under a name derived from a hash of
that configuration, and a registry per namespace mapped names to it.

A store is now one collection: `VectorStore(collection, vector_dimensions,
indexed_properties)` names its one native collection (or its tables and
index files) at construction, every partition of it shares the
collection's dimensions and schema, and `provision()` creates the
collection's durable resources idempotently, before `startup`.
`create_partition(key)`, `get_partition(key)` and `delete_partition(key)`
take a string key; a partition is a payload value (Qdrant), a partition-key
value (Milvus) or a pair of tables (the SQLite stores) inside the
collection, and the registry beside it records what each partition was
created under, so a store built with other dimensions or another schema
raises VectorStorePartitionSchemaMismatchError instead of reading columns
and vectors that are not there. Collection names may be 64 bytes; the
hash-derived native names go, and with them `VectorStoreCollectionConfig`
and the per-partition config.

`DatabaseManager.get_vector_store(backend, collection=, vector_dimensions=,
indexed_properties=)` builds and caches one store per (backend,
collection), keyed by the service's system keys; asking for a collection
again with other dimensions or keys is a configuration error. The event
backend's collection is `long_term_memory__<embedder>` and the semantic
memory's `semantic_memory__<embedder>`, one cell of the purpose-by-embedder
matrix each; the two SQLite stores of one backend share its engine, and
MemMachine warms the event backend's store through the locator, since
building it needs the embedder's dimensions.

The data path is as it was: a partition stores every property of a record
and filters on any key, with the declared keys indexed.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ESpWYTmCR7X3bJEpoA8SAn
…purge

A partition is identified to callers by its key and inside every store by
an incarnation the store mints when the partition is created. Records,
points and index files are keyed by the incarnation, never by the key, so
a partition deleted and re-created under the same key starts empty and its
predecessor's storage is never adopted by, or reclaimed out from under,
the successor. Handles are bound to one incarnation: once it is deleted,
every operation of the handle raises VectorStorePartitionHandleStaleError.

delete_partition becomes a registry write that makes the partition
unreachable at once; the new purge_deleted_partitions reclaims the storage
afterward, oldest deletion first, a bounded amount per call, safe to repeat
and to run from several processes. Both SQLite stores keep every partition
of a collection in shared tables (records, vec0 with the incarnation as its
partition key, pending-operation log) with a purge queue beside the
registry, and fence writes with a self-checking UPDATE or a registry SELECT
under BEGIN IMMEDIATE; Qdrant and Milvus carry the incarnation in the
payload/partition-key field, keep purge entries in the registry collection
stamped with the deletion time, and fence each operation with a registry
lookup. create_partition mints under the segment store's rules: a
collision with a live or queued incarnation re-mints, up to
_MAX_MINT_ATTEMPTS, then VectorStoreAttemptsExhaustedError.

The SQLite stores' on-disk layout changes (shared tables per collection in
place of tables per partition); existing SQLite vector store files are not
migrated.

Qdrant's custom sharding goes: `QdrantConf.is_distributed`, the CUSTOM
sharding method, the shard key per partition and the shard-key selectors
on every operation. It existed so a partition could be deleted by dropping
its shard; deletion is now the registry write, O(1) and atomic as seen by
every reader, and reclamation is the purge's filter-delete by incarnation,
so a shard per partition would only add overhead. `is_distributed` was
never documented; a configuration naming it is rejected.

partition_lifecycle_contract.py holds the contract tests every backend
mixes in: stale handles, empty re-creation, idempotent deletion, purge
reclaiming what deletion deferred and leaving live partitions alone; run
against Qdrant in local, REST and gRPC modes.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ESpWYTmCR7X3bJEpoA8SAn
The first time a vector store is handed out, the resource manager starts
the same purge loop it runs for segment stores, one per (backend,
collection); close() cancels both sets. Mechanical churn in the same
change: the loop's interval and pause constants lose their SEGMENT_STORE_
prefix and the loop takes a label for its failure log line.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ESpWYTmCR7X3bJEpoA8SAn
A partition stored every property of a record and filtered on any key,
which made a caller's arbitrary keys part of the store's schema: the SQLite
stores kept them in a JSON column and filtered with json_extract, Milvus in
a JSON field plus a dynamic field per key, and a filter on a key the store
never indexed scanned. Since EventMemory routes a filter on an undeclared
key to the segment store, the vector store need not hold undeclared keys
at all.

A partition now stores the properties its store declares and no others.
`upsert` raises UndeclaredPropertyKeyError before anything is sent for a
record naming an undeclared key, and PropertyTypeMismatchError for a value
of another type than its key declares; `query` raises
UndeclaredPropertyKeyError for a filter naming an undeclared key and
UnsupportedFilterError for a node outside the partition's
`supported_filter_nodes`. Both SQLite stores keep one typed, indexed,
nullable column per declared key on the records table (sql_columns.py);
sqlite-vec 0.1.9 rejects NULL in a vec0 metadata column and a declared key
is optional per record, so that store keeps the columns on the records
table and hands the KNN a `rowid IN (SELECT ...)` allowlist, evaluating
the filter during the search instead of after it. Qdrant and Milvus drop
the JSON copy and keep a payload or dynamic field per declared key.
Datetimes are stored as microseconds since the epoch where a backend has
no datetime type.

Since every key a filter may name is now indexed, the Qdrant store creates
its collection and its registry in strict mode (`unindexed_filtering_retrieve`
and `_update` false, Qdrant Cloud's default): a filter on an unindexed key is
refused by the server instead of scanned for. A leaf whose value is of
another type than its key declares matches nothing, as on the SQL stores;
the Qdrant compiler answers it with a filter no point satisfies, since the
server would refuse the condition for the field's index. Local mode does
not record the setting, so a unit test checks the request and integration
tests the server's answer.

`declared_schema_contract.py` states the contract every backend's test
module runs: which records a filtered search admits, over fixtures small
enough that every backend searches them exactly, checked after each
upsert so an approximate index fails on recall, by name, and not on the
filter.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ESpWYTmCR7X3bJEpoA8SAn
The Qdrant and Milvus clients were built without a timeout, so a remote
write could hang a request indefinitely. `request_timeout` on QdrantConf
and MilvusConf, in seconds, is passed to the client; it is required, with
no default, so a deployment states how long it is willing to wait, and the
configuration wizard supplies 30 seconds as the starting point. The sample
configurations and the configuration docs show the option.

A breaking configuration change on `speedkick`.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ESpWYTmCR7X3bJEpoA8SAn
…tization

`QdrantConf` gains `hnsw_config`, `optimizers_config` and
`quantization_config`, plain mappings mirroring qdrant-client's
`HnswConfigDiff`, `OptimizersConfigDiff` and `QuantizationConfig`, so
qdrant-client stays optional for configuration parsing; the store's
params validate them against qdrant's own models. They apply to the
store's data collection, never to its registry collection.

`m` must be 0 or unset: the collection is multi-tenant and disables the
global graph in favor of per-partition payload indexing, so a deployment
tunes `payload_m`, which defaults to 16 as before.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ESpWYTmCR7X3bJEpoA8SAn
@edwinyyyu
edwinyyyu force-pushed the feat/qdrant-collection-options-speedkick branch from 612066d to f08d6a7 Compare September 15, 2026 21:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant