Skip to content

[vector store 1/12] Remove per-project filterable properties (speedkick) - #1606

Merged
edwinyyyu merged 2 commits into
MemMachine:speedkickfrom
edwinyyyu:feat/vector-store-handoff-speedkick
Sep 14, 2026
Merged

edwinyyyu merged 2 commits into
MemMachine:speedkickfrom
edwinyyyu:feat/vector-store-handoff-speedkick

Conversation

@edwinyyyu

@edwinyyyu edwinyyyu commented Sep 10, 2026

Copy link
Copy Markdown
Contributor

Purpose of the change

A project could declare properties_schema, a set of caller property keys with types, on its long-term memory configuration; the event backend merged it into the vector store collection's indexed schema and rejected filters on any other m.<key>. That let a tenant create database resources (indexes, columns) by naming them in a request, which is what forced per-collection native resources named by a hash of their schema on the backends that limit them.

The option is removed from the server configuration, the project API and the memory-configuration API, the Python SDK, the sample configurations, the configuration docs and the OpenAPI document. A filter may name any m.<key>; the stores evaluate it on the properties they hold. What a store indexes is decided by the deployment, not per project.

A breaking API change on speedkick.

Stack

Slice 1 of 13, every PR targeting speedkick; merge bottom-up.

# PR change
1 #1606 (this PR) Remove per-project filterable properties
2 #1621 Route filters on undeclared keys to the segment store
3 #1622 Create a session's storage with the session, never on a request
4 #1623 Make the examples that write to a project create it first
5 #1624 Make no memory request create a project
6 #1625 Remove open-or-create and close from both stores
7 #1626 Rename logical collection to partition, and open to get, on both stores
8 #1627 Make a vector store one collection, with string-keyed partitions
9 #1628 Make a vector store filter only on the properties it declares
10 #1629 Let a deployment declare the properties a vector store indexes
11 #1630 Bound every request to a remote vector store by a configured timeout
12 #1618 Let a deployment tune a Qdrant collection's HNSW, optimizers and quantization
13 #1616 Close the filter union, and make negation the complement on every backend

This PR's own change is its last commit, dc1e757f (23 files changed, 12 insertions(+), 384 deletions(-)); under it, 093652b4 regenerates docs/openapi.json under the locked FastAPI (ValidationError gains input and ctx), so this and every later slice's diff of that file shows only what it changes. Sits directly on speedkick; #1621 is stacked on it.

Verification

ruff check, ruff format --check, ty check (two pre-existing spacy diagnostics), pytest packages/server/server_tests packages/client/client_tests: 2173 passed, 3 skipped, on this branch.

🤖 Generated with Claude Code

https://claude.ai/code/session_01ESpWYTmCR7X3bJEpoA8SAn

@edwinyyyu
edwinyyyu marked this pull request as draft September 10, 2026 20:37
@edwinyyyu edwinyyyu changed the title Declare indexed properties per store, close the filter union, and route undeclared predicates to the segment store (speedkick) (Depends on #1597) Declare indexed properties per store, close the filter union, and route undeclared predicates to the segment store (speedkick) Sep 11, 2026
edwinyyyu added a commit to edwinyyyu/MemMachine that referenced this pull request Sep 14, 2026
…, and close the filter union (speedkick)

A vector store is built for one collection, named at construction with its
vector dimensions and its declared `indexed_properties`; the composition
root builds one store per purpose-by-embedder cell. Within it, a partition
holds one tenant's records, and `VectorStorePartition` is the handle a
data consumer holds. Lifecycle on both storage components:

    provision()                                        # idempotent DDL
    create_partition(key)                              # strict
    get_partition(key) -> Partition | None             # None when absent
    delete_partition(key)                              # idempotent

Declared keys are typed, indexed columns of the SQLite records tables and
payload indexes on Qdrant and Milvus; a record or a filter naming an
undeclared key is rejected before anything is sent, so an undeclared key
never exists in a vector store. EventMemory routes the declared part of a
filter to the vector store and the rest to the segment store's
post-filter, widening the search up to `max_overfetch_factor`.

The filter language is a closed union: Equals, Ordering, In, IsNull, And,
Or, Not. `!=` parses as Not(Equals); Not is the complement on every
backend, so a record holding no value passes a negated predicate.
Datetimes are stored as microseconds since the epoch where a backend has
no datetime type. Each backend states its `supported_filter_nodes`.

Folds MemMachine#1613 and MemMachine#1615 into MemMachine#1606 and moves the whole off MemMachine#1597: the
routing is written against speedkick's EventMemory, and the session,
source and expansion parameters return with MemMachine#1597.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ESpWYTmCR7X3bJEpoA8SAn
@edwinyyyu
edwinyyyu force-pushed the feat/vector-store-handoff-speedkick branch from c6af2c2 to af60bb5 Compare September 14, 2026 16:41
@edwinyyyu edwinyyyu changed the title (Depends on #1597) Declare indexed properties per store, close the filter union, and route undeclared predicates to the segment store (speedkick) Declare a store's schema once, make it one collection with partitions, and close the filter union (speedkick) Sep 14, 2026
@edwinyyyu
edwinyyyu marked this pull request as ready for review September 14, 2026 16:42
@edwinyyyu
edwinyyyu marked this pull request as draft September 14, 2026 18:12
@edwinyyyu
edwinyyyu force-pushed the feat/vector-store-handoff-speedkick branch 2 times, most recently from 8a916c1 to 27dcc84 Compare September 14, 2026 19:04
@edwinyyyu edwinyyyu changed the title Declare a store's schema once, make it one collection with partitions, and close the filter union (speedkick) Declare a store's schema once, make it one collection with partitions, and create a session's storage with the session (speedkick) Sep 14, 2026
@edwinyyyu
edwinyyyu force-pushed the feat/vector-store-handoff-speedkick branch from 56dfc00 to 8393c04 Compare September 14, 2026 19:35
@edwinyyyu
edwinyyyu force-pushed the feat/vector-store-handoff-speedkick branch from 4518469 to 142ccfd Compare September 14, 2026 21:38
@edwinyyyu edwinyyyu changed the title Declare a store's schema once, make it one collection with partitions, and create a session's storage with the session (speedkick) [vector store 1/13] Remove per-project filterable properties (speedkick) Sep 14, 2026
`docs/openapi.json` predates the FastAPI release in `uv.lock`
(0.141.1), whose `ValidationError` component carries `input` and `ctx`;
regenerating the document with `docs/tools/generate_openapi.py` adds the
two fields and changes nothing else. Separate from the API changes above
it so their diffs of this file show only what they change.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ESpWYTmCR7X3bJEpoA8SAn
A project could declare `properties_schema`, a set of caller property keys
with types, on its long-term memory configuration; the event backend merged
it into the vector store collection's indexed schema and rejected filters on
any other `m.<key>`. That let a tenant create database resources (indexes,
columns) by naming them in a request, which is what forced per-collection
native resources named by a hash of their schema on the backends that limit
them.

The option is removed from the server configuration, the project API and
the memory-configuration API, the Python SDK, the sample configurations,
the configuration docs and the OpenAPI document. A filter may name any
`m.<key>`; the stores evaluate it on the properties they hold. What a store
indexes is decided by the deployment, not per project.

A breaking API change on `speedkick`.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ESpWYTmCR7X3bJEpoA8SAn
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant