fixtures: add the Qwen3-Embedding embed probe corpus so owned-cuda can certify - #11
Merged
Merged
Conversation
…n certify Owned-CUDA has no declared or inherited certification path: a measured probe row must exist before serving, and the probe resolves reference fixtures by (family, reference_model). The shipped embed fixtures only cover minilm and gte-modernbert, so a qwen3-0.6b/owned-cuda model dead-ends at reference_fixture_missing with no supported way to certify. Add the missing set: 64 items, the same corpus texts as the shipped fixtures, reference vectors generated on CPU with candle-transformers 0.10.2 f32 (last-token pool + L2), independent of the CUDA kernels under test. Wire it into probe_fixtures() and resolve the qwen3-embedding model-id family/reference names so the fixture matches without an engine build flag, mirroring the minilm/modernbert fallbacks. Verified locally: the fixture scores mean cosine 0.99999771 / worst decile 1.0 against the owned-CUDA lane (thresholds 0.999 / 0.9), and the new registry test pins family/reference/dims for all three sets.
There was a problem hiding this comment.
No issues found across 2 files
Not reviewed (too large): crates/synapse-module/src/fixtures/probe_corpus_qwen3_embedding_fp32.json (~65,932 lines) - if these are generated or fixture files, add them to ignored paths to exclude them from future reviews.
Tip: cubic can generate docs of your entire codebase and keep them up to date. Try it here.
Re-trigger cubic
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Owned-CUDA has no declared or inherited certification path:
ensure_model_certifiedrequires a measured probe row keyed on the machine-profile hash, and the only writer of those rows isprobe.start. The embed probe resolves reference fixtures by(family, reference_model), but the shipped embed fixtures only coverminilmandgte-modernbert— so a Qwen3-Embedding model on the owned-CUDA lane dead-ends at:with no supported way to certify. This PR adds the missing fixture set.
What's added
crates/synapse-module/src/fixtures/probe_corpus_qwen3_embedding_fp32.json— 64 items (same corpus texts as the shipped MiniLM fixture so cross-lane comparisons stay like-for-like), reference vectors for Qwen3-Embedding-0.6B, 1024 dims,pooling: "last",normalize: true.probe_fixtures()loads the new set and computes its reference dims, matching the existing pattern.probe_reference_keyresolves theqwen3-embeddingmodel-id family/reference names when the engine build flags carry no family (same fallback shape asminilm/gte-modernbert-base).Reference provenance (the part that matters)
The reference vectors were generated with candle-transformers 0.10.2 on CPU, f32 weights, last-token pooling + L2 normalization, from
Qwen/Qwen3-Embedding-0.6B(safetensors revision97b0c614be4d77ee51c0cef4e5f07c00f9eb65b3). The generation path shares no code with the CUDA kernels under test, which is the property the probe needs from a reference. Thegeneration_commandfield records the invocation; the generator is a small candle-based tool (happy to land it underbench/lanes/in a follow-up if you want it in-tree rather than as a documented command).Tokenization follows the module's owned-Qwen3 policy:
add_special_tokens=true, pop a trailing terminal, truncate to 2048, push the terminal — i.e. exactly whatapply_owned_tokenizer_policy+tokenize_batchproduce for this model, so the reference and the probe compare the same byte stream.Verification
cargo test -p synapse-module --lib— 423 passed, 0 failed (the new registry test included; one unrelatedworker_hosttest flaked once on this box and passes on re-run).cargo fmt --checkandcargo clippy -p synapse-module --libclean.Scope
Fixture + matching only. I did not touch the certification flow itself; with this set present, a normal
probe.starton an owned-CUDA qwen3-embedding model can produce a measured row and serve.One adjacent gap I noticed while building this, filed separately if you want it: the CUDA capability floor is env-only (
SYNAPSE_CUDA_DRIVER_API/SYNAPSE_CUDA_COMPUTE_CAPABILITY), and the driver value is a raw driver-API integer (13030here), not the marketing version string — nothing probes it automatically, so a correct machine still refuses until the operator sets both. Happy to follow up with a probe that readscuDriverGetVersion/cuDeviceGetAttributeat startup.Need help on this PR? Tag
@codesmith-botwith what you need. Autofix is disabled.Summary by cubic
Adds the Qwen3-Embedding probe fixture corpus so owned-CUDA Qwen3 embed models can certify instead of dead-ending at
reference_fixture_missing.Changes
probe_fixtures()and resolvesqwen3-embeddingmodel-ids to the new family/reference names, mirroring the existingminilm/gte-modernbertfallbacks.Verification
cargo test -p synapse-module --libpasses (423 tests, 0 failures) andcargo fmt/clippyare clean.Written for commit 6f7cdca. Summary will update on new commits.