Declare search cells over a closed derived schema by EnRaiha · Pull Request #332 · NodeDB-Lab/nodedb

EnRaiha · 2026-09-16T21:22:35Z

Summary

A derived relation over a closed-schema source refused s.distance with 42703, while the same projection over an open-schema source ran. The response layer answers a search-shaped query with two synthetic cells the schema cannot name: distance (Float64) and _surrogate (the internal row id, resolved to the user primary key). The declaration follows the plan: SqlPlan::carries_search_cells() matches the three variants whose rows lower to the vector-hit shape (VectorSearch | SparseSearch | MultiVectorSearch), and infer_subquery_relation reads the cells from the subquery's own plan — the source the planner already uses, never a second AST reading of it.

Closes #305.

Behaviour changes

Path	Before (main)	After
`s.distance` over a closed-schema derived relation whose body routes to a search (`ORDER BY vector_*` and the operator forms `<=>` / `<#>` / `<->`, `sparse_score`, routing `WHERE` forms, `WHERE multi_vector_search(...)`)	`42703`	resolves; every returned hit carries a numeric value
`s._surrogate` over the same shapes	`42703`	resolves to the integer id
Shapes the planner sends to scalar evaluation (a comparison wrapped around a call, a one-argument `ORDER BY vector_distance(..)`, a body whose `FROM` is neither `Scan` nor `Join`)	`42703`	`42703` — a declared cell is never NULL
Queries that previously ran	unchanged	unchanged

Root cause

Derived-relation inference (resolver/derived.rs) re-derived "is this a search?" from the AST. The planner answers the same question in where_search / order_by/triggers, and the two diverged in both directions: declarations for shapes that route no search (silent NULL cells), and no declaration for SqlPlan::MultiVectorSearch.

What changed

Commit bec788896 — the declaration: append distance to a derived relation's columns when its body is search-shaped; unit tests cover declaration and non-declaration.

Commit 15cea2eec — the filed case as a wire test: closed schema plus a vector index, SEARCH … USING VECTOR(…) in the derived body; the open-schema form never reproduced the refusal, which is why the case pins the closed schema.

Commit ca7baec72 — the trigger set: the declaration reads SearchTrigger from the function registry, the same source the planner reads.

Commit eeb56bb4e — _surrogate declared beside distance; the WHERE clause scanned.

Commit 995215c8c — the layer move: SqlPlan::carries_search_cells() is the one predicate; infer_subquery_relation takes the inner plan (derived_from passes inner_plan, entry passes cte_plan, and TableScope's derived arm plans non-lateral factors for the same input — a correlated LATERAL factor cannot be planned at scope time and routes no cells); the four AST heuristic functions and the third registry singleton are deleted.

Regression proof

On main without the change, the reworked wire case fails 5 of 20 — including the filed SEARCH … USING VECTOR(…) case and the _surrogate projection. On the branch head all 20 pass.

Tested

cargo nextest run -p nodedb-sql --lib — 892 pass (derived-relation and carries_search_cells unit tests included)
cargo nextest run -p nodedb --test wire -E 'test(cases::sql_search_subquery_composition)' — 20 pass; on main with the same tests: 15 pass / 5 fail
lateral, recursive CTE, insert-select, vector-index and hybrid modules — 52 pass
cargo fmt --all -- --check, cargo clippy -p nodedb-sql --all-targets, nodedb-preflight.sh: clean

Notes

synthetic_column declares the cells as nullable Unknown type: the declaration resolves the name; the response layer supplies the value.
WHERE multi_vector_search(...) plans SqlPlan::MultiVectorSearch, which the lowering step does not support yet (42601, variant named). The cell declaration is plan-derived, so that refusal is the honest one; 42703 no longer appears for it.
A body off a derived relation gives order_by/triggers no Scan/Join to read, so it routes no search; its projection refuses the cells.

Copilot

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

Derived-relation inference listed declared columns only. The response layer appends a synthetic distance cell, so s.distance failed 42703 on a closed schema. - append distance when ORDER BY carries vector_distance(...) - cover a hand-written ORDER BY vector_distance derived table with a vector index (numeric cells, nearest-first) and without one (numeric cells or a refusal; silent NULL is the only wrong outcome)

The filed case: a closed-schema source declares no distance column, so the synthetic cell resolves only when derived-relation inference adds the name. The open-schema form resolves with or without the fix, so the case pins the closed schema and reproduces the filed 42703.

The derived-relation check matched the literal name vector_distance. vector_cosine_distance, vector_neg_inner_product, and the operator forms the preprocessor rewrites to them also route an ORDER BY to a vector search whose rows carry a distance cell, so a closed-schema derived table over those forms still refused s.distance with 42703. - read the trigger from the function registry, the same source the order-by planner reads; no second name list

The declaration matched the literal vector_distance in the ORDER BY list only. SparseScore routes an ORDER BY to a search, a WHERE-clause vector trigger routes to the same plan, and every search answers with _surrogate beside distance; all three refused 42703 over a closed schema. - read both triggers from the registry the planner reads - scan the ORDER BY list and the WHERE clause - declare _surrogate next to distance

farhan-syah

Request changes.

Checked statically against the branch ref (no checkout):

Claim	Result
`cargo fmt --all -- --check` clean	Confirmed on every changed file.
Comment hygiene	One broken comment (inline). No issue numbers or links.
"every trigger and every shape"	Not true. The resolver re-derives routing from the AST and diverges from the planner in both directions (inline, blocker).
Commit list	The body cites `cdb2e71a1`, `4eddcfeb5`, `235b7a28b`, `994372149`. The branch holds `bec78889`, `15cea2e`, `ca7baec`, `eeb56bb`. Update the body after the rebase.

Blockers are the layer finding, the broken comment, and the doc/code contradiction. The registry singleton and the tolerant tests are should-fix; both fall out of the layer change.

What landed correctly: the closed-schema wire case reproduces #305, and _surrogate is now declared beside distance.

farhan-syah · 2026-09-17T11:24:19Z

+    // without it `s.distance` and `s._surrogate` resolve against no relation
+    // and are refused with 42703, while the same projection over an
+    // open-schema source runs.
+    if order_by_triggers_a_search(query) || where_triggers_a_search(&query.body) {


Blocker: wrong layer. This re-derives the planner's routing from the AST. planner/select/derived_from.rs computes inner_plan = plan_query(subquery, …) and on the next line calls infer_subquery_relation with the AST instead of the plan. planner/select/entry.rs does the same for a CTE (cte_plan then infer_subquery_relation).

The plan already answers the question. The variants whose rows carry distance + _surrogate are the ones that lower to a vector-hit shape (exchange/resolve/exchange/post_process_arm.rs, classify_hit_shape): SqlPlan::VectorSearch | SparseSearch | MultiVectorSearch.

The AST copy diverges today:

Query Planner This resolver Result

WHERE sparse_score(…) > 0.1 scalar fallback (where_search.rs, dispatch_trigger) declares s.distance is NULL where main raised 42703

ORDER BY vector_distance(x) (one arg) Ok(None) (order_by/triggers.rs, args.len() < 2), scan declares NULL

WHERE multi_vector_search(…) SqlPlan::MultiVectorSearch (where_search.rs), vector-hit rows not declared 42703 stays

body whose FROM is not Scan / Join Ok(None) (try_extract_sort_search) declares NULL

The first two turn a loud error into a silent NULL: the same class #305 was filed against, one layer up.

End state: one predicate on SqlPlan (fn carries_search_cells(&self) -> bool matching those three variants). infer_subquery_relation takes the inner plan. expr_triggers_a_search, expr_contains_a_search, order_by_triggers_a_search, where_triggers_a_search go away. resolver/columns.rs (TableScope, the Derived arm) also infers without a plan and needs the same input.

farhan-syah · 2026-09-17T11:24:19Z

-    infer_body(catalog, &query.body)
+    let (mut columns, open) = infer_body(catalog, &query.body)?;
+
+    // The SEARCH preprocessor rewrites `SEARCH c USING VECTOR(...)` into


Blocker: broken comment. This sentence ends here. The next line starts a different one. Remove this line or finish it.

farhan-syah · 2026-09-17T11:24:20Z

+    })
+}
+
+/// WHERE routing covers the vector trigger only: `where_search`'s dispatch


Blocker: doc contradicts code. "covers the vector trigger only", but expr_contains_a_search → expr_triggers_a_search matches SparseSearch too. The planner's WHERE dispatch sends SparseSearch to scalar evaluation, so the code is the wrong side of this contradiction, not the comment. Resolved by the layer change above.

farhan-syah · 2026-09-17T11:24:20Z


+/// The process-wide built-in registry. Read-only after construction; used
+/// where a function name alone decides routing.
+pub fn builtin_registry() -> &'static FunctionRegistry {


Should-fix: third registry singleton. planner/const_fold.rs (DEFAULT_REGISTRY) and resolver/expr/functions.rs (FUNCTION_REGISTRY) already hold a LazyLock<FunctionRegistry>. The layer change removes this caller. If a static stays, keep one and point the other two at it.

farhan-syah · 2026-09-17T11:24:20Z

+        .unwrap_or_else(|e| panic!("a WHERE-clause trigger must declare s.distance: {e}"));
+    for row in &rows {
+        assert!(
+            row.is_empty() || row.parse::<f64>().is_ok(),


Should-fix: the test certifies a NULL cell. Once the declaration comes from the plan, a declared cell is always filled. Pin parse::<f64>() with no is_empty() escape. Same at the sparse case below.

farhan-syah · 2026-09-17T11:24:20Z

+            "distance must be numeric where filled: {rows:?}"
+        ),
+        Err(e) => assert!(
+            !e.contains("42703"),


Should-fix: the Err arm asserts nothing about the fix. Any error without 42703 passes. Same shape at the no-index case above (!e.to_string().is_empty()). With the plan-derived declaration this query either resolves with numeric cells or the sparse read refuses for a stated reason. Pin one outcome per test.

The derived-relation inference re-derived search routing from the AST and diverged from the planner in both directions: a WHERE sparse_score(...) or a one-argument ORDER BY vector_distance(...) declared a distance cell the planner never fills (silent NULL where main refused 42703), while SqlPlan::MultiVectorSearch went undeclared (42703 stays). SqlPlan::carries_search_cells() matches the three variants whose rows lower to the vector-hit shape; infer_subquery_relation reads the cells from the subquery's own plan, which the planner call sites already build. TableScope's derived arm plans non-lateral factors for the same input; a correlated LATERAL factor cannot be planned at scope time and routes no cells. The four AST heuristic functions and the third registry singleton are gone.

Copilot AI lite review requested due to automatic review settings September 16, 2026 21:22

Copilot AI reviewed Sep 16, 2026

EnRaiha added the run-ci Opt this PR into the full test suite; re-add to force a re-run label Sep 16, 2026

EnRaiha force-pushed the fix/issue305-derived-distance branch from cd4c598 to 9943721 Compare September 17, 2026 02:16

EnRaiha added run-ci Opt this PR into the full test suite; re-add to force a re-run and removed run-ci Opt this PR into the full test suite; re-add to force a re-run labels Sep 17, 2026

EnRaiha added 4 commits September 17, 2026 12:39

EnRaiha force-pushed the fix/issue305-derived-distance branch from 9943721 to eeb56bb Compare September 17, 2026 04:39

farhan-syah requested changes Sep 17, 2026

View reviewed changes

Navigation Menu

Uh oh!

Declare search cells over a closed derived schema - #332

Declare search cells over a closed derived schema#332
EnRaiha wants to merge 5 commits into
mainfrom
fix/issue305-derived-distance

EnRaiha commented Sep 16, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

farhan-syah left a comment

Uh oh!

farhan-syah Sep 17, 2026

Uh oh!

farhan-syah Sep 17, 2026

Uh oh!

farhan-syah Sep 17, 2026

Uh oh!

farhan-syah Sep 17, 2026

Uh oh!

farhan-syah Sep 17, 2026

Uh oh!

farhan-syah Sep 17, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Query	Planner	This resolver	Result
`WHERE sparse_score(…) > 0.1`	scalar fallback (`where_search.rs`, `dispatch_trigger`)	declares	`s.distance` is NULL where `main` raised `42703`
`ORDER BY vector_distance(x)` (one arg)	`Ok(None)` (`order_by/triggers.rs`, `args.len() < 2`), scan	declares	NULL
`WHERE multi_vector_search(…)`	`SqlPlan::MultiVectorSearch` (`where_search.rs`), vector-hit rows	not declared	`42703` stays
body whose FROM is not `Scan` / `Join`	`Ok(None)` (`try_extract_sort_search`)	declares	NULL

Uh oh!

Conversation

EnRaiha commented Sep 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Behaviour changes

Root cause

What changed

Regression proof

Tested

Notes

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

farhan-syah left a comment

Choose a reason for hiding this comment

Uh oh!

farhan-syah Sep 17, 2026

Choose a reason for hiding this comment

Uh oh!

farhan-syah Sep 17, 2026

Choose a reason for hiding this comment

Uh oh!

farhan-syah Sep 17, 2026

Choose a reason for hiding this comment

Uh oh!

farhan-syah Sep 17, 2026

Choose a reason for hiding this comment

Uh oh!

farhan-syah Sep 17, 2026

Choose a reason for hiding this comment

Uh oh!

farhan-syah Sep 17, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

EnRaiha commented Sep 16, 2026 •

edited

Loading