Skip to content

[vector store 10/13] Let a deployment declare the properties a vector store indexes (speedkick) - #1629

Closed
edwinyyyu wants to merge 11 commits into
MemMachine:speedkickfrom
edwinyyyu:feat/vector-store-config-indexed-properties-speedkick
Closed

[vector store 10/13] Let a deployment declare the properties a vector store indexes (speedkick)#1629
edwinyyyu wants to merge 11 commits into
MemMachine:speedkickfrom
edwinyyyu:feat/vector-store-config-indexed-properties-speedkick

Conversation

@edwinyyyu

@edwinyyyu edwinyyyu commented Sep 14, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Purpose of the change

A store's declared schema was the system keys of the service it is built for and nothing else, so no caller property could be filtered during a search; every such predicate went to the segment store's post-filter.

indexed_properties on QdrantConf, MilvusConf, SQLiteVectorStoreConf and SQLiteVecVectorStoreConf names the property keys every collection of that backend indexes and filters on, each with its type name; DatabaseManager merges them, typed, with the service's system keys when it builds a store, and refuses a configured key the service writes with another type. The declaration is the deployment's, made once in configuration: no request and no project creates a database resource. The sample configurations and the configuration docs show the option.

Stack

Slice 10 of 13, every PR targeting speedkick; merge bottom-up.

# PR change
1 #1606 (merged) Remove per-project filterable properties
2 #1621 Route filters on undeclared keys to the segment store
3 #1622 Create a session's storage with the session, never on a request
4 #1623 Make the examples that write to a project create it first
5 #1624 Make no memory request create a project
6 #1625 Remove open-or-create and close from both stores
7 #1626 Rename logical collection to partition, and open to get, on both stores
8 #1627 Make a vector store one collection, with string-keyed partitions
9 #1628 Make a vector store filter only on the properties it declares
10 #1629 (this PR) Let a deployment declare the properties a vector store indexes
11 #1630 Bound every request to a remote vector store by a configured timeout
12 #1618 Let a deployment tune a Qdrant collection's HNSW, optimizers and quantization
13 #1616 Close the filter union, and make negation the complement on every backend

This PR's own change is its last commit, 102903f7 (10 files changed, 211 insertions(+), 28 deletions(-)); the rest of its diff is the slices under it, and drops out as they merge. Stacked on #1628; #1630 is stacked on it.

Verification

ruff check, ruff format --check, ty check (two pre-existing spacy diagnostics), pytest packages/server/server_tests packages/client/client_tests: 2215 passed, 3 skipped, on this branch.

🤖 Generated with Claude Code

https://claude.ai/code/session_01ESpWYTmCR7X3bJEpoA8SAn

This was referenced Sep 14, 2026
Merged
Closed
Draft
Draft
Draft
Draft
Draft
Draft
Draft
Draft
Draft
Draft
@edwinyyyu
edwinyyyu force-pushed the feat/vector-store-config-indexed-properties-speedkick branch from 2ee3b59 to 7ec9b19 Compare September 14, 2026 21:49
@edwinyyyu @claude
Route the part of a filter the vector store does not declare to the s…
eaf0133 
…egment store

EventMemory split a caller's filter nowhere: the whole tree went to the
vector store, which had to store every caller property and filter on any
key, and then to the segment store again. Now the vector store gets the
conjuncts naming only keys its collection declares, evaluated during the
search, and the rest is the segment store's, applied to the seeds
afterward; a disjunction or negation mixing declared and undeclared keys is
undeclared as a whole. A derivative record carries only the caller
properties the collection declares, so an undeclared key never reaches the
vector store, and a declared key's value must have the declared type.

A post-filter can drop seeds and leave the search short, so the fetch is
widened by a factor of four at a time until the limit is met or the fetch
reaches `limit * FilterOptions.max_overfetch_factor` (default 64), where
the search returns what survived. A query with no undeclared part never
widens.

`filter_parser` gains `filter_fields`, `filter_nodes` and `split_declared`;
the in-memory test collection records the filter of each query so routing
can be asserted.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ESpWYTmCR7X3bJEpoA8SAn
@edwinyyyu
edwinyyyu force-pushed the feat/vector-store-config-indexed-properties-speedkick branch from 7ec9b19 to 2883673 Compare September 14, 2026 21:54
@edwinyyyu @claude
Make the overfetch cap a field of EventMemoryParams, not a one-field …
34fa10d 
…options type

`FilterOptions` held a single integer and nothing read it but
EventMemory; a container around one knob is structure without a
second member. `max_overfetch_factor` is a field of EventMemoryParams,
with the same default and bound.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ESpWYTmCR7X3bJEpoA8SAn
@edwinyyyu
edwinyyyu force-pushed the feat/vector-store-config-indexed-properties-speedkick branch from 2883673 to 4d22990 Compare September 14, 2026 22:08
edwinyyyu and others added 9 commits September 14, 2026 15:37
@edwinyyyu @claude
Post-filter undeclared keys on the one fetch the limit allows; no wid…
27724f6 
…ening

A search widened its vector fetch by four at a time, up to
`max_overfetch_factor` times the limit, to make up for seeds the segment
store's post-filter dropped, so a selective predicate on an undeclared key
could cost 64 times the limit in vectors and segment store lookups.

The vector search is one fetch of `vector_search_limit`, as it was: that
limit bounds a query's cost, and the deployment declares the keys it
filters on so they are evaluated during the search. A predicate on an
undeclared key is applied to those seeds afterward, and the result holds
the seeds it admits, fewer than the limit when it drops some; the docstring
says so. The loop, its two constants and `max_overfetch_factor` go.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ESpWYTmCR7X3bJEpoA8SAn
@edwinyyyu @claude
Create a session's storage when the session is created, never on a re…
4266b09 
…quest

The event backend created a session's vector store collection and segment
store partition on the first request that opened the session, so a search
or a write for an unknown session created storage as a side effect, and
the service locator was the only place that knew both stores' create paths.

The owner is the session. Every path that creates a session row runs
through EpisodicMemoryManager._create_session, which inserts the row and,
when the row is new, creates the session's partitions in its segment store
and its vector store (create_episodic_memory_storage); an equivalent
re-create accepts the row and leaves the storage as it is. The request
path binds handles with the stores' lookups and raises
SessionPartitionMissingError when a partition is absent: a session without
its storage is broken, not new. Deleting a session with no open instance
deletes its partitions by key, so a session whose storage was never fully
created can still be deleted. MemMachine.create_session goes through the
manager for the same reason.

The semantic manager owns its one collection and creates it, once, at the
storage's first use. With that, nothing calls the stores' open-or-create.

The API is unchanged: the manager's open-or-create still creates a session
a memory request names, now through the same path.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ESpWYTmCR7X3bJEpoA8SAn
@edwinyyyu @claude
Make the scripts and examples that write to a project create it first
371009a 
The simple chatbot example, the TypeScript REST demo and the Dify plugin's
add-memory tool wrote to a project without creating it, relying on the
write to create it. Each now creates its project before its first memory
request and accepts 409 as the project already existing. No behavior
changes for them; they stop depending on a write creating a project.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ESpWYTmCR7X3bJEpoA8SAn
@edwinyyyu @claude
Make no memory request create a project
15ee80d 
Adding memories to, or searching, a project that did not exist created it,
with the server's default configuration, without the caller's knowledge.
Now only the create-project request creates a project: a write or a
search opens the session and answers 404 for an unknown project, as the
search endpoint already promised; the manager's open-or-create goes.

Two callers depended on the implicit creation. `org_id` and `project_id`
default to `universal`, so the API promises the project
`universal/universal`; the server creates it, once, at startup, and leaves
one that already exists as it is. The MCP add tool names its own project
and has no create-project counterpart, so it creates the project it writes
to, once, and says so. The API doc strings and the OpenAPI document say
which requests create a project.

A breaking API change on `speedkick`.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ESpWYTmCR7X3bJEpoA8SAn
@edwinyyyu @claude
Remove open-or-create and close from both stores
3bb7e89 
Nothing calls them since a session's storage is created with the session:
`open_or_create_collection` and `close_collection` leave the vector store
interface and its four backends, `open_or_create_partition` and
`close_partition` leave the segment store interface and its
implementation, and the two config-mismatch errors that only open-or-create
raised go with them. A store creates on `create_*`, strictly, and looks up
on `open_*`, answering None; create-if-absent is the owner's, where the
key's provenance is known.

Source changes are deletions only. The tests that exercised
open-or-create as a fixture use a test-side create-if-absent instead, and
the tests of its own semantics go.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ESpWYTmCR7X3bJEpoA8SAn
@edwinyyyu @claude
Rename a vector store's logical collection to partition, and both sto…
e9783d9 
…res' lookup to get_partition

A vector store's logical collection becomes a partition, the segment store's word for the same thing, and both stores' lookup is get_partition, answering None like a Python get. Identifiers only, produced by the script below; the (namespace, name) identity, the per-partition config and every docstring are as they were, and the next change gives them their meaning. The native clients' create_collection and delete_collection keep their names.

```sh
set -e
cd "$(git rev-parse --show-toplevel)"
git mv packages/server/server_tests/memmachine_server/common/vector_store/in_memory_vector_store_collection.py \
       packages/server/server_tests/memmachine_server/common/vector_store/in_memory_vector_store_partition.py
git ls-files -z 'packages/server/*.py' | xargs -0 perl -0pi -e '
  s/VectorStoreCollection(?!Config)/VectorStorePartition/g;
  s/in_memory_vector_store_collection/in_memory_vector_store_partition/g;
  s/vector_store_collection(?!_schema|_namespace)/vector_store_partition/g;
  s/open_collection/get_partition/g;
  s/def create_collection\(/def create_partition(/g;
  s/def delete_collection\(/def delete_partition(/g;
  s/\.create_collection\((\s*namespace=)/.create_partition($1/g;
  s/\.delete_collection\((\s*namespace=)/.delete_partition($1/g;
  s/\.create_collection(?=\s*=\s*AsyncMock|\.assert_)/.create_partition/g;
  s/\.delete_collection(?=\s*=\s*AsyncMock|\.assert_)/.delete_partition/g;
  s/"create_collection"/"create_partition"/g;
  s/"delete_collection"/"delete_partition"/g;
  s/only delete_collection is invoked/only delete_partition is invoked/g;
  s/test_delete_collection_/test_delete_partition_/g;
  s/open_partition/get_partition/g;
'
uv run ruff check --fix --quiet packages/server
uv run ruff format --quiet packages/server
```

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ESpWYTmCR7X3bJEpoA8SAn
@edwinyyyu @claude
Make a vector store one collection, its partitions keyed by string, b…
e602348 
…uilt by the composition root

A vector store was a factory of logical collections, each identified by a
(namespace, name) pair and created with its own dimensions and schema; a
backend that limits native collections shared one among logical
collections of equal configuration, under a name derived from a hash of
that configuration, and a registry per namespace mapped names to it.

A store is now one collection: `VectorStore(collection, vector_dimensions,
indexed_properties)` names its one native collection (or its tables and
index files) at construction, every partition of it shares the
collection's dimensions and schema, and `provision()` creates the
collection's durable resources idempotently, before `startup`.
`create_partition(key)`, `get_partition(key)` and `delete_partition(key)`
take a string key; a partition is a payload value (Qdrant), a partition-key
value (Milvus) or a pair of tables (the SQLite stores) inside the
collection, and the registry beside it records what each partition was
created under, so a store built with other dimensions or another schema
raises VectorStorePartitionSchemaMismatchError instead of reading columns
and vectors that are not there. Collection names may be 64 bytes; the
hash-derived native names go, and with them `VectorStoreCollectionConfig`
and the per-partition config.

`DatabaseManager.get_vector_store(backend, collection=, vector_dimensions=,
indexed_properties=)` builds and caches one store per (backend,
collection), keyed by the service's system keys; asking for a collection
again with other dimensions or keys is a configuration error. The event
backend's collection is `long_term_memory__<embedder>` and the semantic
memory's `semantic_memory__<embedder>`, one cell of the purpose-by-embedder
matrix each; the two SQLite stores of one backend share its engine, and
MemMachine warms the event backend's store through the locator, since
building it needs the embedder's dimensions.

The data path is as it was: a partition stores every property of a record
and filters on any key, with the declared keys indexed.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ESpWYTmCR7X3bJEpoA8SAn
@edwinyyyu @claude
Make a vector store filter only on the properties it declares
a297965 
A partition stored every property of a record and filtered on any key,
which made a caller's arbitrary keys part of the store's schema: the SQLite
stores kept them in a JSON column and filtered with json_extract, Milvus in
a JSON field plus a dynamic field per key, and a filter on a key the store
never indexed scanned. Since EventMemory routes a filter on an undeclared
key to the segment store, the vector store need not hold undeclared keys
at all.

A partition now stores the properties its store declares and no others.
`upsert` raises UndeclaredPropertyKeyError before anything is sent for a
record naming an undeclared key, and PropertyTypeMismatchError for a value
of another type than its key declares; `query` raises
UndeclaredPropertyKeyError for a filter naming an undeclared key and
UnsupportedFilterError for a node outside the partition's
`supported_filter_nodes`. Both SQLite stores keep one typed, indexed,
nullable column per declared key on the records table (sql_columns.py);
sqlite-vec 0.1.9 rejects NULL in a vec0 metadata column and a declared key
is optional per record, so that store keeps the columns on the records
table and hands the KNN a `rowid IN (SELECT ...)` allowlist, evaluating
the filter during the search instead of after it. Qdrant and Milvus drop
the JSON copy and keep a payload or dynamic field per declared key.
Datetimes are stored as microseconds since the epoch where a backend has
no datetime type.

`declared_schema_contract.py` states the contract every backend's test
module runs: which records a filtered search admits, over fixtures small
enough that every backend searches them exactly, checked after each
upsert so an approximate index fails on recall, by name, and not on the
filter.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ESpWYTmCR7X3bJEpoA8SAn
@edwinyyyu @claude
Let a deployment declare the properties a vector store indexes
102903f 
A store's declared schema was the system keys of the service it is built
for and nothing else, so no caller property could be filtered during a
search; every such predicate went to the segment store's post-filter.

`indexed_properties` on QdrantConf, MilvusConf, SQLiteVectorStoreConf and
SQLiteVecVectorStoreConf names the property keys every collection of that
backend indexes and filters on, each with its type name; DatabaseManager
merges them, typed, with the service's system keys when it builds a store,
and refuses a configured key the service writes with another type. The
declaration is the deployment's, made once in configuration: no request
and no project creates a database resource. The sample configurations and
the configuration docs show the option.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ESpWYTmCR7X3bJEpoA8SAn
@edwinyyyu
edwinyyyu force-pushed the feat/vector-store-config-indexed-properties-speedkick branch from 4d22990 to 102903f Compare September 14, 2026 22:39
@edwinyyyu

edwinyyyu commented Sep 14, 2026

Copy link
Copy Markdown
Contributor Author

Closed: user-defined properties are never evaluated in the vector store (EventMemory's property_filter is user-defined only; system_filter carries the indexed predicates), so a deployment-declared user key has no consumer. The stack is renumbered to 12; #1630 is now 10/12.

@edwinyyyu edwinyyyu closed this Sep 14, 2026
@edwinyyyu edwinyyyu mentioned this pull request Sep 15, 2026
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@edwinyyyu