From b304d70c4a063dd7852152e3b964ced66d48f3ec Mon Sep 17 00:00:00 2001 From: Claude Date: Sun, 23 Aug 2026 01:41:30 +0000 Subject: [PATCH] =?UTF-8?q?feat(v35):=20ML=20Lab=20=E2=80=94=20le=20chiffr?= =?UTF-8?q?e=20cesse=20de=20se=20flatter=20lui-m=C3=AAme?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Deux défauts de méthode dans du code livré, plus les deux ajouts qui en découlent. 1) LE GAGNANT ÉTAIT ÉLU SUR LE JEU DE TEST. Le leaderboard triait neuf modèles sur la métrique calculée sur test et couronnait sorted[0] : prendre le maximum de neuf tirages sur ~180 lignes biaise le chiffre vers le haut. Il y a désormais un troisième split — la validation est taillée dans le train (64/16/20 aux ratios par défaut), la sélection se fait sur validation, et la ligne du champion annonce les deux chiffres et l'écart : « choisi sur la validation à 0,974, obtient 0,917 sur le jeu de test intact ». Les indices de test sont identiques à l'octet près à ce que la même config produisait avant V35 — vérifié par un test — donc segments, seuils, incertitude et comparaison de lot voient exactement les mêmes lignes. Sous 60 lignes utilisables, le troisième split est refusé par son nom. Le classement vit maintenant dans UN module (ranking.ts) : leaderboard, historique, comparaison de runs, rapport HTML et sélection automatique du modèle inspecté. C'est précisément ce qui a cassé en cours de route — un quatrième site triait encore sur test et ouvrait un modèle différent de celui couronné. 2) LA DÉCOUPE ÉTAIT TOUJOURS ALÉATOIRE, MÊME SUR DONNÉES DATÉES. Découpe chronologique (les plus anciennes entraînent, lignes sans date exploitable écartées et comptées) et découpe par groupe (aucun groupe des deux côtés), proposées quand une colonne s'y prête et ANNONCÉES dans l'info du run. 3) DÉTECTEUR DE FUITE PRÉDICTIVE. V6 attrapait les colonnes qui MAPPENT sur la cible ; celui-ci attrape la colonne seulement prédictive — un stump à une colonne ajusté sur train, mesuré sur validation. Une colonne seule qui lit la cible à 99 % s'affiche en avertissement cuivre avec son score, jamais comme une victoire. 4) CLASSEMENT ROBUSTE À LA DEMANDE. Validation croisée 5×2 sur train+validation (pipeline réajusté dans chaque pli, jeu de test jamais touché) : moyenne, dispersion, et combien de fois le premier a réellement battu le second — « 10 plis sur 10 : l'ordre est stable » ou « 6 sur 10 : traitez-les comme à égalité ». DÉFAUT EXPOSÉ ET NOMMÉ PAR LA VAGUE. Avec le train plus petit, le Naive Bayes gaussien sur ~150 variables TF-IDF sature à exactement 0/1 : l'occlusion de V24 mesurait alors exactement zéro pour chaque mot et la carte disparaissait — ce qui se lit « aucun mot ne compte », et c'est faux. Mesuré (2 probabilités distinctes sur 48 lignes de test, contre 48 pour la régression logistique et le gradient boosting), la carte se refuse maintenant par son nom et indique un modèle capable de répondre. Les tests V25 qui encodaient l'arithmétique à deux voies sont mis à jour en préservant leur intention (le cap compte toujours les lignes utilisables ; les tailles sont ajustées pour que les caps par famille mordent encore). Vérifié : 369 tests unitaires, 65 e2e, format:check, lint (2 avertissements préexistants sur main), typecheck, build avec coquilles prérendues. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01UKw6oNC8iZ9Kn7q6x4qom4 --- PLAN.md | 50 +-- README.md | 14 +- e2e/split.spec.ts | 87 +++++ e2e/text.spec.ts | 16 +- .../ml/components/LeaderboardTable.tsx | 126 +++++-- .../ml/components/RobustRankPanel.tsx | 133 ++++++++ .../ml/components/RunArtifactsView.tsx | 50 ++- src/features/ml/components/RunsHistory.tsx | 9 +- src/features/ml/components/TrainPanel.tsx | 62 +++- .../components/insights/InsightsSection.tsx | 5 +- .../ml/components/insights/WordEffects.tsx | 19 ++ src/features/ml/data/parse.worker.ts | 18 + src/features/ml/lab-store.ts | 85 ++++- src/features/ml/projects/compare.ts | 13 +- src/features/ml/projects/report.ts | 44 ++- src/features/ml/projects/types.ts | 3 + src/features/ml/train/insights.ts | 29 +- src/features/ml/train/leakage.ts | 166 +++++++++ src/features/ml/train/pipeline.text.test.ts | 31 +- src/features/ml/train/ranking.ts | 50 +++ src/features/ml/train/robust.ts | 147 ++++++++ src/features/ml/train/sampling.test.ts | 26 +- src/features/ml/train/trainer.test.ts | 3 +- src/features/ml/train/trainer.ts | 194 ++++++++++- src/features/ml/train/types.ts | 52 +++ src/features/ml/train/v35.test.ts | 320 ++++++++++++++++++ src/features/ml/worker-protocol.ts | 7 + src/locales/en.json | 38 ++- src/locales/fr.json | 38 ++- 29 files changed, 1719 insertions(+), 116 deletions(-) create mode 100644 e2e/split.spec.ts create mode 100644 src/features/ml/components/RobustRankPanel.tsx create mode 100644 src/features/ml/train/leakage.ts create mode 100644 src/features/ml/train/ranking.ts create mode 100644 src/features/ml/train/robust.ts create mode 100644 src/features/ml/train/v35.test.ts diff --git a/PLAN.md b/PLAN.md index bb3bbcc..3c2f8e4 100644 --- a/PLAN.md +++ b/PLAN.md @@ -428,31 +428,31 @@ production on 21/08/2026. Cap 6's guiding thread: the lab meets the real world — real photos, real text, real file sizes, and the question every data budget asks. -| Wave | Content | Why | -| --------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | -| **V23 — delivered** | **Vision 2**: SqueezeNet (2012) retired for three self-hosted ONNX models — **EfficientNet-Lite4 int8** classification (1,000 ImageNet classes, 77.6% top-1), **YOLOX-Nano** object detection (80 COCO classes; the stronger-but-AGPL YOLOs were ruled out, Apache-2.0 kept) and **UltraFace RFB-320** face detection — boxes drawn on the image, FR/EN class names, plain-language counts ("1 person · 1 face"). Box decoding (grids/strides, exp, IoU, per-class NMS) is hand-written and unit-tested; letterboxed inputs (aspect squashing measurably mislocated face boxes); named thresholds (objects 35%; faces 0.9 — real faces score ≥ 0.95, measured false positives top out at 0.85); ~19 MB total, runtime-cached, offline after first use; verified on real photos (NASA portrait → 1 person + 1 face; German Shepherd → dog + breed at 99.9%) | Owner request (21/08/2026): portraits have no ImageNet class, so the old model answered off-target — and the detector must recognize a whole range of things, not just faces | -| **V24 — delivered** | **Text columns**: free text stops being skipped and enters the pipeline as a hand-written **TF-IDF** block — accent-folding bilingual tokenizer, merged FR/EN stop words, vocabulary capped at 256 terms ranked by document frequency (ties alphabetical, terms seen in a single training document dropped), smoothed IDF, L2-normalized vectors, fitted on the training split only. Features are named `column:word`, so importance, Shapley and the report speak in words; `encodedBlocks` now measures a text block by its real width (counting it as one column silently shifted every block after it). Explanations gained **signed word effects** by occlusion — erase one word from the reviews containing it and average the shift of the answer — because permutation is blind to a redundant vocabulary, and multiclass is refused rather than faked. Export bumped to **format v3** (v2 files still import). Demo `reviews.csv`: 240 bilingual orders where the text carries the signal — baseline 0.52 → 0.92, `review` top of the importance chart, `fast`/`excellent`/`avance` pushing up, `refund`/`cheap` pushing down | Real CSVs have text columns (comments, descriptions) — the lab used to drop them on the floor | -| **V25 — delivered** | **Scale**: the lab now takes 100k–1M-row files without dying, on a measure-first design. Before: a stack overflow killed every run past ~65k rows (`push(...spread)` in the split), and the slow families made big runs unusable anyway (random forest alone: **535 s at 5 000 rows**). Measured first, then fixed: (1) the split rewritten with plain loops; (2) the planned typed-array pipeline rewrite was **descoped on measurement** — the pipeline was never the bottleneck (parse 2.6 s + profile 4.9 s + fit/transform 3.1 s at 1M rows); (3) **announced seeded sampling, never silent** — a global cap of 100 000 usable rows (seeded, stratified, `summary.sampledFrom`) plus measured per-family training caps (forest 1 000 · tree 2 000 · logistic/linear/MLP 20 000 · k-NN 5 000 · GBDT 50 000) drawn as nested prefixes of one seeded order, every capped model scored on the same full test set, and every sample printed on the leaderboard, in the tuning panel and in the HTML report; k-NN's old **silent** internal 5 000-row subsample was folded into the announced mechanism; (4) a **named memory guard**: parsing streams and refuses past 20M cells (rows × columns) with the numbers spelled out, instead of letting the tab die. After, measured: a 1M-row file trains the whole 8-model zoo in **~130 s** (and a 120k-row file in the same ~126 s — cost is flat past the cap), gbdt reaches 0.991 accuracy on the 50 000-row announced sample. Every demo dataset sits under every cap: existing behavior unchanged. 283 unit tests (announced-sampling determinism, stratified nesting, cap recording), 54 e2e (a generated 120k-row CSV trains with the announcement asserted; a 21M-cell file is refused by name) | -| **V26 — delivered** | **Learning curves**: the lab answers the classic budget question — "would more data help this model, or is it time to work on features?" — with one new chart. On demand (like tuning), one model is retrained on growing seeded fractions of the train split: the SAME nested prefixes V25's announced caps draw from (same seed, same order), so for a capped family the last point is exactly the leaderboard model's diet and the verdict says out loud whether the announced cap costs accuracy. A geometric ladder of up to 6 sizes (each at least 16 rows, refused entirely when only one rung fits — one point is not a curve); at every size the pipeline is REFITTED on that prefix only (imputation, encoding, IDF, scaling all see just those rows — the strict no-leakage reading of a learning curve) and the model is scored on the same full held-out test set with a V20 bootstrap 95% band. The verdict is the V20 paired bootstrap applied to the last size step: a decisive gain reads "still climbing — more data would probably help", anything else "flattened — work on features or the model", each with the capped variant ("the cap costs accuracy" / "the cap costs nothing here"). The chart (log-spaced sizes, CI band, one dot per announced size) ships with its numbers table, joins the run artifacts (history, HTML report, share links) and refuses the baseline by name — flat by definition. Measured on Titanic/gbdt: 45 → 713 rows traces 0.753 → 0.820 with the plateau verdict at the last step. 293 unit tests, 55 e2e | -| **V27 — delivered** | **Local chat, upgraded**: a real language model — **Qwen3-0.6B-DQ, 355 MB, Apache-2.0** — running entirely in the browser, offered beside the V6 deterministic interpreter, which stays the DEFAULT and the fallback. The model never computes: it translates a question into a V6 query, and every number still comes from the deterministic engine. Its output is checked against the closed Intent grammar — invented columns, unknown operators, absurd k, a correlation of a column with itself are all REFUSED, and a refusal falls back to the keyword parser, with a badge under each answer naming which engine produced the query. Three constraints shaped the build, all discovered by measurement: (1) Cloudflare Pages refuses assets over 25 MiB, so `scripts/prepare-llm.mjs` fetches the weights at DEPLOY time (never committed — 355 MB in git would slow every clone) and splits them into **15 parts of ≤ 24 MiB**, glued back in the browser through transformers.js's `customCache` hook, with per-part size checks and named refusals (`llm-part-missing`, `llm-part-size`, `llm-short`); (2) the strict CSP forbids the library's CDN default, so its pinned ONNX Runtime build is self-hosted under `/ort-llm/` — the jsep (WebGPU) variant clears the 25 MiB limit by only 0.1 MiB, so re-check it on upgrades; (3) **WebGPU is required, not preferred**: the model's `GatherBlockQuantized` embedding kernel has no WASM implementation and needs `shader-f16`, so a device without it gets a named refusal instead of an unusable download. Verified end to end in a browser: the sharded weights download with correct cumulative progress, reassemble, and build an ONNX Runtime WebGPU session (18 s warm). **Honest limit, stated rather than hidden**: the interpretation-quality bench could NOT be run here — it needs a GPU with `shader-f16`, which the CI and dev runners lack. It ships instead as a repo tool (`npm run llm:bench`, 16 FR/EN questions over Titanic, half of them phrasings the keyword grammar cannot catch) that exercises the real production path, so the number can be measured on real hardware before the model is promoted beyond opt-in. 314 unit tests, 57 e2e | -| **V27.1 — delivered** | **The model earns its place, it does not take it**: the V27 order was wrong, and the measurement said so. With the local model selected it read EVERY question first and won whenever its JSON passed the grammar check — even when the keyword parser had a correct reading of its own. Measured in production on six reference questions over Titanic: the model turned « combien de personnes sont montées à Cherbourg ? » into `embarked = Cherbourg` → **0 rows**, where the deterministic parser had `embark_town = Cherbourg` → **168**; and read « est-ce que les femmes payaient plus cher que les hommes ? » as a plain count (314 female) instead of mean fare grouped by sex. Tally: 2 right, 2 confidently wrong, 2 refusals. The order is now **deterministic first, model as a rescue** (`resolveIntent`, unit-tested): the keyword grammar can only ever name a column that exists and a value that actually occurs in it, so when it understands, nothing overrides it — and the model is asked only about what it gives up on, which is exactly the gap that justifies its 355 MB. On the same six, **measured on the owner's GPU after deploy**: 5 right, 1 wrong, 0 refusals — up from 2 right, 2 wrong, 2 refusals. Two further defects fixed: (1) a refusal was badged « question read by the local model », claiming a reading nobody had made — refusals now name nobody and say whether the model was even consulted; (2) the system prompt had **no groupBy and no top-k example at all**, and no rule tying a filter value to the column whose value list contains it — both added, with FR phrasings and a numeric-threshold example. The bench gains the two shapes that failed (`age < 10`, a top-k) and now reports the **shipped order** as its headline number instead of the two engines separately. 326 unit tests, 57 e2e. **The number that justifies the download**: « combien d'enfants de moins de 10 ans ? » → `count age < 10` = 62 and « à quel âge moyen voyageaient les passagers ? » → `mean age` = 29.699, both of which the keyword grammar refuses outright; and « combien de personnes sont montées à Cherbourg ? » came back as the deterministic engine's 168, the model never consulted. **Still open**: one question of the six — « est-ce que les femmes payaient plus cher que les hommes ? » — is still read as a correlation (fare↔age, a column the question never names); addressed in V27.2. The full bench remains un-runnable here (no `shader-f16`), so its number still has to come from real hardware. | Measured by the owner in production (22/08/2026), the day V27 shipped. A confidently wrong answer costs more trust than a refusal — and V27 produced two of them, including a 0 where the deterministic engine already had the right 168. | -| **V27.2 — delivered** | **Two honesty defects, one measured, one found while reading the measurement**: (1) the comparison question V27.1 left wrong — « est-ce que les femmes payaient plus cher que les hommes ? » read as a correlation between `fare` and `age` — gets a rule that names both halves of the mistake: a question comparing two groups is an aggregate with `groupBy` on the column whose values name them, NEVER a correlation; and never pick a column the question does not mention. A second FR comparison example ships with it, in a **different phrasing** from the failing one, which stays a held-out bench case rather than becoming a memorised answer. (2) The answer sentence said « (sur 891 lignes) » under a mean built from 714 values: `rowsConsidered` counts rows after the filter, while `numericAt` skips missing and unparseable cells. The number was right, the sentence around it was not. Aggregates now carry `valuesUsed` (scalar) and `used` per group, set only when they differ from the row count, and the UI says « 714 valeurs utilisables sur 891 lignes » — matching what the correlation branch already did. This one predates V27 entirely: it has been there since V6. 330 unit tests, 57 e2e. **Measured after deploy**: the rule did stop the correlation — but the model then read the same question as `count fare >= 0`, still wrong. Two prompt attempts, two failure modes; see V27.3 for where that stops. | Measured by the owner on real hardware (22/08/2026): 5 of 6 reference questions right after V27.1. The sixth is a confidently wrong answer to a different question than the one asked, and the « sur 891 lignes » wording was found by reading that same screenshot closely — a right number inside a wrong sentence is exactly what this project refuses to ship. | -| **V27.3 — delivered** | **A `>=` that was quietly an `=`**: retesting the comparison question after V27.2 produced « 0 ligne correspond où fare >= 0 » — impossible on a table where all 891 fares clear zero. Root cause found by reproduction, not by reading: the model emitted `"value": "0"` as a **string**, `asFilter` accepted a string for any operator, and `matchesFilter` took the numeric branch only for `typeof value === 'number'` — so `>=` fell through to the equality branch and tested `fare == "0"` against a column whose zero fares are written `0.0`. Same intent with a real number: 891 rows. The hole is closed on both sides: `asFilter` converts a numeric string and refuses anything else on `<`, `<=`, `>`, `>=` (equality keeps text — that is how categorical filters work), and `matchesFilter` handles the ordering operators apart, throwing the named `filter-not-numeric` rather than passing a bug off as a query with no matches. V6 code, reachable only through the model: the keyword parser always built numbers. **And a limit, recorded rather than papered over**: « est-ce que les femmes payaient plus cher que les hommes ? » is still read wrong — a correlation before V27.2, a vacuous count after. The model finds `fare` every time and the shape never. Two prompt attempts are enough; a third would be sewing the prompt around one sentence, which buys a flattering bench and nothing else. The measured score stands at **5 of 6**, and the sixth is written down as what a 0.6B does not do. 337 unit tests, 57 e2e. | Found by retesting in production (22/08/2026). An arithmetically impossible answer — zero rows for a condition every row satisfies — is worse than a refusal and worse than a wrong reading: it makes the engine itself untrustworthy, which is the one thing LabML sells. | -| **V28 — delivered** | **« Ne nous croyez pas sur parole »** — a `/privacy` route that states the local-only promise once, in full, and then hands the reader the means to check it without trusting a word of it. Four verification steps, ordered by how hard they are to fake: cut the network (DevTools → Network → Offline, or the Wi-Fi switch) and watch the whole lab keep working; watch the Network tab while loading a file and training, and see nothing happen; read `Content-Security-Policy` on the document itself; open Application → IndexedDB and see exactly what was kept. The served policy is **quoted verbatim on the page and pinned to `public/_headers` by a unit test** — a page that claims a protection the site quietly dropped is worse than no page. A schematic of the Network panel is drawn rather than screenshotted (DevTools chrome differs per browser and per locale) and captioned as a diagram, not a capture. A live audit panel counts this page's own resource timings by origin and says, in the same breath, what it cannot see: worker timelines and requests the CSP blocked — a proof that oversells itself is worth less than none. Last section lists what _does_ cross the network (app files, demo datasets on click, vision models on entering Vision, LLM weights on explicit consent) and what never does. FR/EN, prerendered shell, WCAG AA verified by axe including the audit result. 344 unit tests, 60 e2e. | Owner request (22/08/2026): the promise is repeated across the site, but a user has no way to tell a true claim from a comforting one. Verifiability is the product here — anyone can write « your data stays local » in a footer. | -| **V29 — delivered** | **Analytical SQL in the browser (DuckDB-Wasm, MIT)**: the Data Studio gains a real OLAP engine — joins, window functions, aggregations — over the file you just loaded, with no server and no upload. The file is queried **as dropped, before the cleaning recipe**: the recipe belongs to the studio, and a result traceable to nothing the user can reopen would be worse than no SQL at all. Extra CSV / **Parquet** / JSON files can be attached in the same session (Parquet is a new input format for the lab), each exposed as a view named after the file; a result exports to CSV or goes to the ML Lab in one click, through the handoff path V4 already built. Errors show **DuckDB's own message** — it names the line and the token, which no paraphrase of ours would. **The measurement that set the version**: `@duckdb/duckdb-wasm` is pinned to **1.28.0**, not `latest`. From 1.29 the binaries cross Cloudflare Pages' hard 25 MiB per-file limit (eh 34.2 MiB, mvp 39.4 MiB); at 1.28.0 they are **17.3 and 21.1 MiB** and fit. Newer would have meant sharding the wasm and either widening `connect-src` to `blob:` — days after publishing a page that quotes that very directive — or rebuilding the service worker in injectManifest mode. An older engine was the cheaper honest trade, and it is written here so the next upgrade re-measures instead of rediscovering. Self-hosted under `/duckdb/` (the library defaults to jsDelivr, which the CSP refuses), **never precached** — cached on first use like the vision models, so nobody pays 18 MiB before opening the console — and the `coi` threaded build is left out entirely: no COOP/COEP, no SharedArrayBuffer, single-threaded as the assumed mode. Remote S3/HTTP querying stays out, by CSP and by intent. 352 unit tests, 61 e2e. | Owner request (21/08/2026): real analytical SQL on ~100 MB files with zero backend. Delivered after the /privacy page at the owner's request (22/08/2026). | -| V30 | **Chat that reads better, measured before it is made bigger.** The V27.1–V27.3 measurement stands at **5 of 6** reference questions, and the one failure is a _shape_ error, not missing knowledge: the model finds `fare` every time and picks the wrong intent. **First, the numbers that kill the obvious idea** — a « 600 MB model » is not an upgrade: at q4f16, Qwen3-0.6B **non-DQ is 570 MB and the same brain**, only its embeddings unquantised. The real rungs are gemma-3-1b-it **764 MB** (2×), Llama-3.2-1B **1.09 GB**, SmolLM2-1.7B **1.11 GB**, Qwen2.5-1.5B **1.22 GB**, Qwen3-1.7B **1.43 GB** — against 370 MB today. So the plan spends nothing on weights until the cheap levers are exhausted. **(A) A bench worth the name** — 40–60 FR/EN questions including the phrasings that fail, runnable and reported; today's 18 cases cannot run in CI, and without this nothing that follows is measurable. **(B) Constrained decoding** — a hand-written `LogitsProcessor` masking every token outside the grammar _during_ generation: after `{"kind":"` only seven tokens are legal. The shape error becomes unrepresentable rather than caught after the fact, and on this task that can beat a model four times larger. **(C) Examples drawn from the user's own columns** instead of frozen Titanic ones — 0 MB, and it removes the temptation to copy an example column. **(D) Two samples, one vote**, keeping the candidate that validates and invents no column the question never names — 0 MB, 2× the time. **Only then** the bigger model, and as a SECOND announced download (« reinforced model », 764 MB) with Qwen 370 MB staying the default: the V27 sharding infrastructure already handles it (32 parts of 24 MiB). VRAM (~1.2–1.5 GB estimated) and first-token latency to be measured before promising anything. | Owner question (22/08/2026): would a bigger model raise the share of correct answers? The measured failure is structural, so the plan tests that hypothesis for 0 MB before asking a visitor for twice the bandwidth. | -| V31 | **Vision that stops being asked the impossible.** Today's three models weigh **18.6 MB total** (EfficientNet-Lite4 int8 13.6, YOLOX-Nano 3.7, UltraFace 1.3) against 370 MB for the chat model — the headroom is enormous. **The main cause of the mistakes is not the network**: ImageNet-1k has **no « person » class** — 1000 labels, ~120 of them dog breeds, none for a human being — so a photo of someone comes back as « suit » or « jersey ». The model is not wrong; it is being asked a question whose answer is absent from its vocabulary. **(A) Measure first**: 30–50 public-domain images with expected label and expected boxes, replayed in e2e, so « it still makes mistakes » becomes a percentage. **(B) Fix the label space — the real correction**: CLIP ViT-B/32 zero-shot, vision q4f16 **126 MB** + text int8 **64 MB** ≈ **190 MB**, letting the visitor type their own labels (« a cat », « an invoice », « a houseplant »). It repairs the defect and makes a far better demonstration than 1000 frozen classes; open weights, self-hosted, local execution — the doctrine holds. **(C) What costs no download**: check the crop (squashing a 16:9 photo into a square skews everything — `preprocess.ts` is the suspect), average over two crops, recalibrate `OBJECT_THRESHOLD` (0.35) and `FACE_THRESHOLD` (0.9), and above all **refuse below a confidence floor** — « I am not sure » rather than a label picked at random, which is the chat's doctrine applied to pixels. **(D) A better detector**: YOLOX-S (Apache-2.0), ~35 MB, roughly +14 mAP over Nano — with acquisition and licence verified first, as in V23: the YOLOX ONNX files on the Hub are community re-uploads, not official releases. | Owner report (22/08/2026): the vision playground is better than the chat but still makes mistakes. Naming the label-space mismatch is what turns a vague complaint into a fixable defect. | -| V32 | **Documentation, the scaffolding and one finished tutorial.** A `/docs` route, linked from the footer beside « Comment ça marche », built on the **Diátaxis** split — tutorial (learning), how-to (a task), reference (lookup), explanation (the why) — because the usual failure of documentation is mixing all four on one page: a tutorial that pauses to weigh an alternative loses the beginner it was written for. A tutorial offers **no choices** and **guarantees the result**. Five rules specific to this project: **(1) the docs are tested like the code** — everything here is seeded at 42, so « you will get 0.821 accuracy » becomes an assertion in `e2e/docs.spec.ts` and a drifting page **breaks the build**; a documentation that cannot lie is the same promise as the rest of the site. **(2) Screenshots are generated** with Playwright, never hand-taken — one that cannot be regenerated does not ship. **(3) Better than a screenshot, a link that does the thing**: « try it » deep-links landing on the panel with the demo already loaded (needs small URL-parameter support), which never goes stale. **(4) Markdown lives in the repo** (`src/content/docs/**`), compiled at build with prerendered shells like every other route; no Algolia, no third-party doc host — a third-party call on a site that publishes `/privacy` would be indefensible, so search is a local index. **(5) The docs are not PLAN.md**: this file is the engineering record in English with the trade-offs; the docs are for users, FR/EN. Scope of this wave: the route, the Markdown pipeline, the table of contents, local search, and **one** complete tutorial — « premier modèle en 10 minutes » — tested end to end. It is the template every later page copies: tone, length, how figures are quoted. | Owner request (22/08/2026): document every shipped feature across /ml, /data and /ai, linked from the footer. One finished tutorial first, on purpose — writing the full reference before the template is settled means rewriting all of it. | -| V33 | **The reference, and the table of refusals.** Page-per-panel coverage of the three sections: ML Lab (leaderboard, tuning, thresholds, segments, uncertainty, learning curves, run comparison, model export/import, batch scoring), Data Studio (quality score, recipe, forced types, join, drift, anomalies, SQL console) and AI (vision, assistant, the two interpreters). Reference is dry, exhaustive and structured like the software — not prose. The page that no competitor has: **a complete table of the named refusals** — `filter-not-numeric`, `llm-part-missing`, `too-large`, `no-webgpu`, « neither interpreter understood », « the interval is not conclusive » — with what triggers each one, what it means and what to do about it. Refusing well is this project's distinguishing feature; documenting the refusals is the most honest page it can publish. Plus a formats page (CSV, Parquet, JSON, the model manifest). **Honest sizing**: this is 1–2 days of _writing_ for ~25 features in two languages. It does not automate into anything but mush. | A feature nobody can look up is a feature that does not exist for the reader; and a refusal nobody can decode reads as a bug rather than as the design it is. | -| V34 | **Explanations, how-to guides, and the pages that make the project readable as engineering.** The why-pages: why a baseline before anything else, why intervals instead of a single figure, why seed 42 everywhere, why everything runs locally (pointing at `/privacy` rather than repeating it — duplicated prose diverges), and **what LabML does not do, and why** — a project that names its limits reads as a serious one, and the limits are already measured here (the comparison question a 0.6B model does not read, the bench that needs `shader-f16`, SQL over the file before the recipe). Task-shaped how-to guides for readers who already know their way around: score a new batch, compare two runs, read a learning curve, hand a SQL result to the lab. Generated screenshots and the « try it » deep-links land here too. Every page ends with « et ensuite ? » — documentation without a next step is a dead end. | Three audiences, deliberately: the curious visitor (five minutes), the practitioner (one task), and the evaluator judging whether the engineering is rigorous. The explanation pages are what the third one reads. | -| V35 | **ML Lab: the number stops flattering itself.** Two of these are not missing features but **method defects in shipped code**, which is why they come first in a lab whose selling point is honest evaluation. **(1) The winner is picked on the test set.** `LeaderboardTable.tsx:42` sorts nine models by `primary` — the metric computed on test — and crowns `sorted[0]`. Taking the maximum of nine draws on ~180 test rows biases the headline figure upward; V20's paired bootstrap softens the comparison but the crowned number stays optimistic. Fix: a third split — train / validation / test — selecting on validation, reporting on test, and **showing the gap between the two**, which is itself the most useful lesson the lab can teach. **(2) The split is always random, even on dated data.** `splitIndices()` stratifies at random, full stop — while V10 already derives year/month/day from a date column and V8 does time series, so dated files arrive routinely. A random split puts the future in training and the past in test: the model looks excellent and collapses in production. Fix: detect a date column and offer a **chronological split** (oldest 80% trains), announced; same logic for a repeated identifier — the same customer on both sides is the same leak. Then two additions in the same spirit: a **target-leakage detector** — a lone column predicting at 99% is almost always a leak (« amount_refunded » predicting « fraud ») and must show as a warning, not a victory: small to write, striking to demonstrate, and nobody does it — and **repeated cross-validation for the leaderboard**, because ~180 test rows carry roughly ±3 points of standard deviation and ranking two models one point apart is meaningless; 5×2 CV with intervals makes the ranking defensible. | Owner request (22/08/2026): what to improve in /ml. The audit found two defects rather than gaps — a lab that sells honest evaluation cannot ship a headline figure it knows to be optimistic, nor a split that leaks on dated data. | -| V36 | **ML Lab: the gaps that were deliberately left open.** **Class imbalance** — weighting was explicitly descoped in V16; class weights in logistic regression, tree, forest and gbdt, announced like every other choice, finally complete what the threshold panel started. **Multiclass thresholds** — set aside since V16 and still open. **Choosing the ranking metric** — accuracy and RMSE are imposed today, while on an imbalanced problem F1 or recall is the right criterion and the ranking changes with it; small to build, and it makes the leaderboard answer the user's question rather than ours. **An ensemble of the best** — average or vote over the top three: typically 1–3 points, free in compute since the models are already trained, and it teaches why ensembling works. **What this wave deliberately does not do**: add a tenth model family (nine is plenty; a tenth improves neither honesty nor understanding), build an AutoML « we handle everything » mode (the opposite of a lab that shows its decisions), or bring in tabular deep learning (high cost, no gain at this scale, and no longer hand-written). | Each item was consciously deferred in an earlier wave rather than forgotten; grouping them keeps the descopes visible instead of letting them quietly become permanent. | -| V37 | **ML Lab: speed and the comfort of long sessions.** **Parallel training** — the zoo trains sequentially in a single worker; N workers means N cores, and at a million rows that is a different experience entirely. The V25 benches already exist to measure it before and after, so the gain is published rather than claimed. **Comparing more than two runs** — V21 compares two; three or four changes what the tool is for, and the diff machinery is already written. **Resuming an interrupted run** — closing the tab loses everything today, while V13 (artifacts) and V19 (persistence) already provide the storage; what is missing is a checkpoint between model families and the offer to resume. | Comes last on purpose: speed and comfort matter, but a faster wrong number is still a wrong number. V35 first, then V36, then this. | -| V38 | **Data Studio: reading the file exactly as it was written.** The headline item is a **defect in shipped code, not a missing feature**. `Papa.parse` is called with `skipEmptyLines: true` and nothing else — no encoding, no decimal separator — and `parseNumber` ends in `Number(cleaned)`. A French Excel export therefore breaks silently: `12,5` becomes `NaN`, the column is classified **text** rather than numeric, and every downstream stage one-hot encodes what should have been a number; a windows-1252 file displays `Québec`. Nothing warns, nothing refuses — the pipeline simply produces a worse model. Fix: **detect encoding and decimal separator and announce both** (« séparateur décimal : virgule, détecté sur 412 valeurs »), expose explicit **delimiter / encoding / decimal** selectors for the cases detection cannot settle, and show a **5-row preview before committing to the load** so a wrong guess is caught in two seconds rather than three panels later. The same pass covers thousands separators and dates written `31/12/2025` instead of ISO. | Owner request (22/08/2026): what to improve in /data. The audit found a defect first: a French-locale CSV — the single most likely file this owner's users will open — loses its numeric columns silently, and silence is the part that violates the project's rules. | -| V39 | **Data Studio: a recipe that works column by column.** `RecipeOptions` today applies `missing` and `clipOutliers` to the **whole file**: one strategy for every column, however different they are. A median makes sense for an age and none at all for a postcode. Make the recipe an ordered **list of per-column steps** — the current global settings becoming the defaults a column may override — and add the strategies that are missing: **median / mean / constant / a « MANQUANT » category** for categorical columns. With them comes a rule the tool should never break: **imputing without marking destroys information**, so every imputed column gains an optional **missing indicator** (`col_absent`), which is frequently predictive in its own right (a blank field is rarely blank at random). The recipe stays what it already is — a replayable, inspectable object — so the per-column version remains exportable, re-appliable to a new file, and legible as a list of named decisions. | A single global strategy is the kind of default that looks tidy and quietly makes the data worse; per-column steps cost little to build because the recipe is already an object, not a pile of checkboxes. | -| V40 | **Data Studio: validity, drift, and an auditable diff.** Quality is measured today as completeness and consistency of type; what is missing is **validity** — a value can be present, well-typed and still impossible. Named rules, each stated in plain language: an age of 200, a date in the future, a percentage at 130, a malformed postcode. Then **cross-column consistency** (`date_fin < date_debut`, `total ≠ quantité × prix`), for which **V29's DuckDB is already the engine** — the rules are SQL, and they run on the file that is already registered. Then three things that make the studio auditable rather than merely helpful: a **replayable reference profile** so a second file can be checked for drift against the first (the same idea as the V22 model manifest), a **before/after diff of the rows a recipe modified** — which rows, which columns, which values, not just a count — and a **breakdown of the quality score** so the number is explained by its parts instead of being asserted. Ends with **Parquet export**, nearly free now that DuckDB is loaded (`COPY … TO 'x.parquet'`). **What this wave deliberately does not do**: a spreadsheet-style cell editor (hand edits break reproducibility — the recipe is the record), fuzzy deduplication (guaranteed false positives on names and addresses, silently merging two real people), or model-based imputation (opaque, and it fabricates values that look plausible). | Comes last because it builds on V38's faithful read and V39's per-column recipe: validity rules on mis-parsed numbers would flag the parser, not the data. | - -**Ordering**: V38 comes before V39 and V40, and for the same reason V35 comes first in its own group: its headline item is a defect in shipped code, not a feature — a studio that promises honest data cannot silently turn `12,5` into `NaN`. V35 comes before V36 and V37 because two of its items are defects in shipped code, not features — a lab that sells honest evaluation fixes those first. V32 ships one finished tutorial before any reference page — the tutorial is the template the rest copies, and settling it late means rewriting everything. V30 and V31 both start with a bench, because neither « a bigger model » nor « it still makes mistakes » is a measurable statement today; no wave starts without an explicit launch command. V23 first (owner request); V24 keeps its vocabulary capped — +| Wave | Content | Why | +| --------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | +| **V23 — delivered** | **Vision 2**: SqueezeNet (2012) retired for three self-hosted ONNX models — **EfficientNet-Lite4 int8** classification (1,000 ImageNet classes, 77.6% top-1), **YOLOX-Nano** object detection (80 COCO classes; the stronger-but-AGPL YOLOs were ruled out, Apache-2.0 kept) and **UltraFace RFB-320** face detection — boxes drawn on the image, FR/EN class names, plain-language counts ("1 person · 1 face"). Box decoding (grids/strides, exp, IoU, per-class NMS) is hand-written and unit-tested; letterboxed inputs (aspect squashing measurably mislocated face boxes); named thresholds (objects 35%; faces 0.9 — real faces score ≥ 0.95, measured false positives top out at 0.85); ~19 MB total, runtime-cached, offline after first use; verified on real photos (NASA portrait → 1 person + 1 face; German Shepherd → dog + breed at 99.9%) | Owner request (21/08/2026): portraits have no ImageNet class, so the old model answered off-target — and the detector must recognize a whole range of things, not just faces | +| **V24 — delivered** | **Text columns**: free text stops being skipped and enters the pipeline as a hand-written **TF-IDF** block — accent-folding bilingual tokenizer, merged FR/EN stop words, vocabulary capped at 256 terms ranked by document frequency (ties alphabetical, terms seen in a single training document dropped), smoothed IDF, L2-normalized vectors, fitted on the training split only. Features are named `column:word`, so importance, Shapley and the report speak in words; `encodedBlocks` now measures a text block by its real width (counting it as one column silently shifted every block after it). Explanations gained **signed word effects** by occlusion — erase one word from the reviews containing it and average the shift of the answer — because permutation is blind to a redundant vocabulary, and multiclass is refused rather than faked. Export bumped to **format v3** (v2 files still import). Demo `reviews.csv`: 240 bilingual orders where the text carries the signal — baseline 0.52 → 0.92, `review` top of the importance chart, `fast`/`excellent`/`avance` pushing up, `refund`/`cheap` pushing down | Real CSVs have text columns (comments, descriptions) — the lab used to drop them on the floor | +| **V25 — delivered** | **Scale**: the lab now takes 100k–1M-row files without dying, on a measure-first design. Before: a stack overflow killed every run past ~65k rows (`push(...spread)` in the split), and the slow families made big runs unusable anyway (random forest alone: **535 s at 5 000 rows**). Measured first, then fixed: (1) the split rewritten with plain loops; (2) the planned typed-array pipeline rewrite was **descoped on measurement** — the pipeline was never the bottleneck (parse 2.6 s + profile 4.9 s + fit/transform 3.1 s at 1M rows); (3) **announced seeded sampling, never silent** — a global cap of 100 000 usable rows (seeded, stratified, `summary.sampledFrom`) plus measured per-family training caps (forest 1 000 · tree 2 000 · logistic/linear/MLP 20 000 · k-NN 5 000 · GBDT 50 000) drawn as nested prefixes of one seeded order, every capped model scored on the same full test set, and every sample printed on the leaderboard, in the tuning panel and in the HTML report; k-NN's old **silent** internal 5 000-row subsample was folded into the announced mechanism; (4) a **named memory guard**: parsing streams and refuses past 20M cells (rows × columns) with the numbers spelled out, instead of letting the tab die. After, measured: a 1M-row file trains the whole 8-model zoo in **~130 s** (and a 120k-row file in the same ~126 s — cost is flat past the cap), gbdt reaches 0.991 accuracy on the 50 000-row announced sample. Every demo dataset sits under every cap: existing behavior unchanged. 283 unit tests (announced-sampling determinism, stratified nesting, cap recording), 54 e2e (a generated 120k-row CSV trains with the announcement asserted; a 21M-cell file is refused by name) | +| **V26 — delivered** | **Learning curves**: the lab answers the classic budget question — "would more data help this model, or is it time to work on features?" — with one new chart. On demand (like tuning), one model is retrained on growing seeded fractions of the train split: the SAME nested prefixes V25's announced caps draw from (same seed, same order), so for a capped family the last point is exactly the leaderboard model's diet and the verdict says out loud whether the announced cap costs accuracy. A geometric ladder of up to 6 sizes (each at least 16 rows, refused entirely when only one rung fits — one point is not a curve); at every size the pipeline is REFITTED on that prefix only (imputation, encoding, IDF, scaling all see just those rows — the strict no-leakage reading of a learning curve) and the model is scored on the same full held-out test set with a V20 bootstrap 95% band. The verdict is the V20 paired bootstrap applied to the last size step: a decisive gain reads "still climbing — more data would probably help", anything else "flattened — work on features or the model", each with the capped variant ("the cap costs accuracy" / "the cap costs nothing here"). The chart (log-spaced sizes, CI band, one dot per announced size) ships with its numbers table, joins the run artifacts (history, HTML report, share links) and refuses the baseline by name — flat by definition. Measured on Titanic/gbdt: 45 → 713 rows traces 0.753 → 0.820 with the plateau verdict at the last step. 293 unit tests, 55 e2e | +| **V27 — delivered** | **Local chat, upgraded**: a real language model — **Qwen3-0.6B-DQ, 355 MB, Apache-2.0** — running entirely in the browser, offered beside the V6 deterministic interpreter, which stays the DEFAULT and the fallback. The model never computes: it translates a question into a V6 query, and every number still comes from the deterministic engine. Its output is checked against the closed Intent grammar — invented columns, unknown operators, absurd k, a correlation of a column with itself are all REFUSED, and a refusal falls back to the keyword parser, with a badge under each answer naming which engine produced the query. Three constraints shaped the build, all discovered by measurement: (1) Cloudflare Pages refuses assets over 25 MiB, so `scripts/prepare-llm.mjs` fetches the weights at DEPLOY time (never committed — 355 MB in git would slow every clone) and splits them into **15 parts of ≤ 24 MiB**, glued back in the browser through transformers.js's `customCache` hook, with per-part size checks and named refusals (`llm-part-missing`, `llm-part-size`, `llm-short`); (2) the strict CSP forbids the library's CDN default, so its pinned ONNX Runtime build is self-hosted under `/ort-llm/` — the jsep (WebGPU) variant clears the 25 MiB limit by only 0.1 MiB, so re-check it on upgrades; (3) **WebGPU is required, not preferred**: the model's `GatherBlockQuantized` embedding kernel has no WASM implementation and needs `shader-f16`, so a device without it gets a named refusal instead of an unusable download. Verified end to end in a browser: the sharded weights download with correct cumulative progress, reassemble, and build an ONNX Runtime WebGPU session (18 s warm). **Honest limit, stated rather than hidden**: the interpretation-quality bench could NOT be run here — it needs a GPU with `shader-f16`, which the CI and dev runners lack. It ships instead as a repo tool (`npm run llm:bench`, 16 FR/EN questions over Titanic, half of them phrasings the keyword grammar cannot catch) that exercises the real production path, so the number can be measured on real hardware before the model is promoted beyond opt-in. 314 unit tests, 57 e2e | +| **V27.1 — delivered** | **The model earns its place, it does not take it**: the V27 order was wrong, and the measurement said so. With the local model selected it read EVERY question first and won whenever its JSON passed the grammar check — even when the keyword parser had a correct reading of its own. Measured in production on six reference questions over Titanic: the model turned « combien de personnes sont montées à Cherbourg ? » into `embarked = Cherbourg` → **0 rows**, where the deterministic parser had `embark_town = Cherbourg` → **168**; and read « est-ce que les femmes payaient plus cher que les hommes ? » as a plain count (314 female) instead of mean fare grouped by sex. Tally: 2 right, 2 confidently wrong, 2 refusals. The order is now **deterministic first, model as a rescue** (`resolveIntent`, unit-tested): the keyword grammar can only ever name a column that exists and a value that actually occurs in it, so when it understands, nothing overrides it — and the model is asked only about what it gives up on, which is exactly the gap that justifies its 355 MB. On the same six, **measured on the owner's GPU after deploy**: 5 right, 1 wrong, 0 refusals — up from 2 right, 2 wrong, 2 refusals. Two further defects fixed: (1) a refusal was badged « question read by the local model », claiming a reading nobody had made — refusals now name nobody and say whether the model was even consulted; (2) the system prompt had **no groupBy and no top-k example at all**, and no rule tying a filter value to the column whose value list contains it — both added, with FR phrasings and a numeric-threshold example. The bench gains the two shapes that failed (`age < 10`, a top-k) and now reports the **shipped order** as its headline number instead of the two engines separately. 326 unit tests, 57 e2e. **The number that justifies the download**: « combien d'enfants de moins de 10 ans ? » → `count age < 10` = 62 and « à quel âge moyen voyageaient les passagers ? » → `mean age` = 29.699, both of which the keyword grammar refuses outright; and « combien de personnes sont montées à Cherbourg ? » came back as the deterministic engine's 168, the model never consulted. **Still open**: one question of the six — « est-ce que les femmes payaient plus cher que les hommes ? » — is still read as a correlation (fare↔age, a column the question never names); addressed in V27.2. The full bench remains un-runnable here (no `shader-f16`), so its number still has to come from real hardware. | Measured by the owner in production (22/08/2026), the day V27 shipped. A confidently wrong answer costs more trust than a refusal — and V27 produced two of them, including a 0 where the deterministic engine already had the right 168. | +| **V27.2 — delivered** | **Two honesty defects, one measured, one found while reading the measurement**: (1) the comparison question V27.1 left wrong — « est-ce que les femmes payaient plus cher que les hommes ? » read as a correlation between `fare` and `age` — gets a rule that names both halves of the mistake: a question comparing two groups is an aggregate with `groupBy` on the column whose values name them, NEVER a correlation; and never pick a column the question does not mention. A second FR comparison example ships with it, in a **different phrasing** from the failing one, which stays a held-out bench case rather than becoming a memorised answer. (2) The answer sentence said « (sur 891 lignes) » under a mean built from 714 values: `rowsConsidered` counts rows after the filter, while `numericAt` skips missing and unparseable cells. The number was right, the sentence around it was not. Aggregates now carry `valuesUsed` (scalar) and `used` per group, set only when they differ from the row count, and the UI says « 714 valeurs utilisables sur 891 lignes » — matching what the correlation branch already did. This one predates V27 entirely: it has been there since V6. 330 unit tests, 57 e2e. **Measured after deploy**: the rule did stop the correlation — but the model then read the same question as `count fare >= 0`, still wrong. Two prompt attempts, two failure modes; see V27.3 for where that stops. | Measured by the owner on real hardware (22/08/2026): 5 of 6 reference questions right after V27.1. The sixth is a confidently wrong answer to a different question than the one asked, and the « sur 891 lignes » wording was found by reading that same screenshot closely — a right number inside a wrong sentence is exactly what this project refuses to ship. | +| **V27.3 — delivered** | **A `>=` that was quietly an `=`**: retesting the comparison question after V27.2 produced « 0 ligne correspond où fare >= 0 » — impossible on a table where all 891 fares clear zero. Root cause found by reproduction, not by reading: the model emitted `"value": "0"` as a **string**, `asFilter` accepted a string for any operator, and `matchesFilter` took the numeric branch only for `typeof value === 'number'` — so `>=` fell through to the equality branch and tested `fare == "0"` against a column whose zero fares are written `0.0`. Same intent with a real number: 891 rows. The hole is closed on both sides: `asFilter` converts a numeric string and refuses anything else on `<`, `<=`, `>`, `>=` (equality keeps text — that is how categorical filters work), and `matchesFilter` handles the ordering operators apart, throwing the named `filter-not-numeric` rather than passing a bug off as a query with no matches. V6 code, reachable only through the model: the keyword parser always built numbers. **And a limit, recorded rather than papered over**: « est-ce que les femmes payaient plus cher que les hommes ? » is still read wrong — a correlation before V27.2, a vacuous count after. The model finds `fare` every time and the shape never. Two prompt attempts are enough; a third would be sewing the prompt around one sentence, which buys a flattering bench and nothing else. The measured score stands at **5 of 6**, and the sixth is written down as what a 0.6B does not do. 337 unit tests, 57 e2e. | Found by retesting in production (22/08/2026). An arithmetically impossible answer — zero rows for a condition every row satisfies — is worse than a refusal and worse than a wrong reading: it makes the engine itself untrustworthy, which is the one thing LabML sells. | +| **V28 — delivered** | **« Ne nous croyez pas sur parole »** — a `/privacy` route that states the local-only promise once, in full, and then hands the reader the means to check it without trusting a word of it. Four verification steps, ordered by how hard they are to fake: cut the network (DevTools → Network → Offline, or the Wi-Fi switch) and watch the whole lab keep working; watch the Network tab while loading a file and training, and see nothing happen; read `Content-Security-Policy` on the document itself; open Application → IndexedDB and see exactly what was kept. The served policy is **quoted verbatim on the page and pinned to `public/_headers` by a unit test** — a page that claims a protection the site quietly dropped is worse than no page. A schematic of the Network panel is drawn rather than screenshotted (DevTools chrome differs per browser and per locale) and captioned as a diagram, not a capture. A live audit panel counts this page's own resource timings by origin and says, in the same breath, what it cannot see: worker timelines and requests the CSP blocked — a proof that oversells itself is worth less than none. Last section lists what _does_ cross the network (app files, demo datasets on click, vision models on entering Vision, LLM weights on explicit consent) and what never does. FR/EN, prerendered shell, WCAG AA verified by axe including the audit result. 344 unit tests, 60 e2e. | Owner request (22/08/2026): the promise is repeated across the site, but a user has no way to tell a true claim from a comforting one. Verifiability is the product here — anyone can write « your data stays local » in a footer. | +| **V29 — delivered** | **Analytical SQL in the browser (DuckDB-Wasm, MIT)**: the Data Studio gains a real OLAP engine — joins, window functions, aggregations — over the file you just loaded, with no server and no upload. The file is queried **as dropped, before the cleaning recipe**: the recipe belongs to the studio, and a result traceable to nothing the user can reopen would be worse than no SQL at all. Extra CSV / **Parquet** / JSON files can be attached in the same session (Parquet is a new input format for the lab), each exposed as a view named after the file; a result exports to CSV or goes to the ML Lab in one click, through the handoff path V4 already built. Errors show **DuckDB's own message** — it names the line and the token, which no paraphrase of ours would. **The measurement that set the version**: `@duckdb/duckdb-wasm` is pinned to **1.28.0**, not `latest`. From 1.29 the binaries cross Cloudflare Pages' hard 25 MiB per-file limit (eh 34.2 MiB, mvp 39.4 MiB); at 1.28.0 they are **17.3 and 21.1 MiB** and fit. Newer would have meant sharding the wasm and either widening `connect-src` to `blob:` — days after publishing a page that quotes that very directive — or rebuilding the service worker in injectManifest mode. An older engine was the cheaper honest trade, and it is written here so the next upgrade re-measures instead of rediscovering. Self-hosted under `/duckdb/` (the library defaults to jsDelivr, which the CSP refuses), **never precached** — cached on first use like the vision models, so nobody pays 18 MiB before opening the console — and the `coi` threaded build is left out entirely: no COOP/COEP, no SharedArrayBuffer, single-threaded as the assumed mode. Remote S3/HTTP querying stays out, by CSP and by intent. 352 unit tests, 61 e2e. | Owner request (21/08/2026): real analytical SQL on ~100 MB files with zero backend. Delivered after the /privacy page at the owner's request (22/08/2026). | +| V30 | **Chat that reads better, measured before it is made bigger.** The V27.1–V27.3 measurement stands at **5 of 6** reference questions, and the one failure is a _shape_ error, not missing knowledge: the model finds `fare` every time and picks the wrong intent. **First, the numbers that kill the obvious idea** — a « 600 MB model » is not an upgrade: at q4f16, Qwen3-0.6B **non-DQ is 570 MB and the same brain**, only its embeddings unquantised. The real rungs are gemma-3-1b-it **764 MB** (2×), Llama-3.2-1B **1.09 GB**, SmolLM2-1.7B **1.11 GB**, Qwen2.5-1.5B **1.22 GB**, Qwen3-1.7B **1.43 GB** — against 370 MB today. So the plan spends nothing on weights until the cheap levers are exhausted. **(A) A bench worth the name** — 40–60 FR/EN questions including the phrasings that fail, runnable and reported; today's 18 cases cannot run in CI, and without this nothing that follows is measurable. **(B) Constrained decoding** — a hand-written `LogitsProcessor` masking every token outside the grammar _during_ generation: after `{"kind":"` only seven tokens are legal. The shape error becomes unrepresentable rather than caught after the fact, and on this task that can beat a model four times larger. **(C) Examples drawn from the user's own columns** instead of frozen Titanic ones — 0 MB, and it removes the temptation to copy an example column. **(D) Two samples, one vote**, keeping the candidate that validates and invents no column the question never names — 0 MB, 2× the time. **Only then** the bigger model, and as a SECOND announced download (« reinforced model », 764 MB) with Qwen 370 MB staying the default: the V27 sharding infrastructure already handles it (32 parts of 24 MiB). VRAM (~1.2–1.5 GB estimated) and first-token latency to be measured before promising anything. | Owner question (22/08/2026): would a bigger model raise the share of correct answers? The measured failure is structural, so the plan tests that hypothesis for 0 MB before asking a visitor for twice the bandwidth. | +| V31 | **Vision that stops being asked the impossible.** Today's three models weigh **18.6 MB total** (EfficientNet-Lite4 int8 13.6, YOLOX-Nano 3.7, UltraFace 1.3) against 370 MB for the chat model — the headroom is enormous. **The main cause of the mistakes is not the network**: ImageNet-1k has **no « person » class** — 1000 labels, ~120 of them dog breeds, none for a human being — so a photo of someone comes back as « suit » or « jersey ». The model is not wrong; it is being asked a question whose answer is absent from its vocabulary. **(A) Measure first**: 30–50 public-domain images with expected label and expected boxes, replayed in e2e, so « it still makes mistakes » becomes a percentage. **(B) Fix the label space — the real correction**: CLIP ViT-B/32 zero-shot, vision q4f16 **126 MB** + text int8 **64 MB** ≈ **190 MB**, letting the visitor type their own labels (« a cat », « an invoice », « a houseplant »). It repairs the defect and makes a far better demonstration than 1000 frozen classes; open weights, self-hosted, local execution — the doctrine holds. **(C) What costs no download**: check the crop (squashing a 16:9 photo into a square skews everything — `preprocess.ts` is the suspect), average over two crops, recalibrate `OBJECT_THRESHOLD` (0.35) and `FACE_THRESHOLD` (0.9), and above all **refuse below a confidence floor** — « I am not sure » rather than a label picked at random, which is the chat's doctrine applied to pixels. **(D) A better detector**: YOLOX-S (Apache-2.0), ~35 MB, roughly +14 mAP over Nano — with acquisition and licence verified first, as in V23: the YOLOX ONNX files on the Hub are community re-uploads, not official releases. | Owner report (22/08/2026): the vision playground is better than the chat but still makes mistakes. Naming the label-space mismatch is what turns a vague complaint into a fixable defect. | +| V32 | **Documentation, the scaffolding and one finished tutorial.** A `/docs` route, linked from the footer beside « Comment ça marche », built on the **Diátaxis** split — tutorial (learning), how-to (a task), reference (lookup), explanation (the why) — because the usual failure of documentation is mixing all four on one page: a tutorial that pauses to weigh an alternative loses the beginner it was written for. A tutorial offers **no choices** and **guarantees the result**. Five rules specific to this project: **(1) the docs are tested like the code** — everything here is seeded at 42, so « you will get 0.821 accuracy » becomes an assertion in `e2e/docs.spec.ts` and a drifting page **breaks the build**; a documentation that cannot lie is the same promise as the rest of the site. **(2) Screenshots are generated** with Playwright, never hand-taken — one that cannot be regenerated does not ship. **(3) Better than a screenshot, a link that does the thing**: « try it » deep-links landing on the panel with the demo already loaded (needs small URL-parameter support), which never goes stale. **(4) Markdown lives in the repo** (`src/content/docs/**`), compiled at build with prerendered shells like every other route; no Algolia, no third-party doc host — a third-party call on a site that publishes `/privacy` would be indefensible, so search is a local index. **(5) The docs are not PLAN.md**: this file is the engineering record in English with the trade-offs; the docs are for users, FR/EN. Scope of this wave: the route, the Markdown pipeline, the table of contents, local search, and **one** complete tutorial — « premier modèle en 10 minutes » — tested end to end. It is the template every later page copies: tone, length, how figures are quoted. | Owner request (22/08/2026): document every shipped feature across /ml, /data and /ai, linked from the footer. One finished tutorial first, on purpose — writing the full reference before the template is settled means rewriting all of it. | +| V33 | **The reference, and the table of refusals.** Page-per-panel coverage of the three sections: ML Lab (leaderboard, tuning, thresholds, segments, uncertainty, learning curves, run comparison, model export/import, batch scoring), Data Studio (quality score, recipe, forced types, join, drift, anomalies, SQL console) and AI (vision, assistant, the two interpreters). Reference is dry, exhaustive and structured like the software — not prose. The page that no competitor has: **a complete table of the named refusals** — `filter-not-numeric`, `llm-part-missing`, `too-large`, `no-webgpu`, « neither interpreter understood », « the interval is not conclusive » — with what triggers each one, what it means and what to do about it. Refusing well is this project's distinguishing feature; documenting the refusals is the most honest page it can publish. Plus a formats page (CSV, Parquet, JSON, the model manifest). **Honest sizing**: this is 1–2 days of _writing_ for ~25 features in two languages. It does not automate into anything but mush. | A feature nobody can look up is a feature that does not exist for the reader; and a refusal nobody can decode reads as a bug rather than as the design it is. | +| V34 | **Explanations, how-to guides, and the pages that make the project readable as engineering.** The why-pages: why a baseline before anything else, why intervals instead of a single figure, why seed 42 everywhere, why everything runs locally (pointing at `/privacy` rather than repeating it — duplicated prose diverges), and **what LabML does not do, and why** — a project that names its limits reads as a serious one, and the limits are already measured here (the comparison question a 0.6B model does not read, the bench that needs `shader-f16`, SQL over the file before the recipe). Task-shaped how-to guides for readers who already know their way around: score a new batch, compare two runs, read a learning curve, hand a SQL result to the lab. Generated screenshots and the « try it » deep-links land here too. Every page ends with « et ensuite ? » — documentation without a next step is a dead end. | Three audiences, deliberately: the curious visitor (five minutes), the practitioner (one task), and the evaluator judging whether the engineering is rigorous. The explanation pages are what the third one reads. | +| **V35 — delivered** | **ML Lab: the number stops flattering itself.** Two method defects in shipped code, fixed, plus the two additions that follow from them. **(1) The winner was picked on the test set** — the leaderboard sorted nine models by the metric computed on test and crowned `sorted[0]`, which makes the headline figure the optimistic maximum of nine draws. There is now a **third split**: validation is carved out of the train side (64/16/20 with the default ratios), ranking and crowning happen on validation, and the champion line spells out both numbers and the gap between them — « selected on validation at 0.974, scores 0.917 on the untouched test set ». The **test indices are byte-identical** to what the same config produced before V35, so every panel that reads the test set (segments, thresholds, uncertainty, batch compare) is unchanged; below 60 usable rows the third split is refused by name and the lab ranks on test as before. Ranking now lives in ONE module (`ranking.ts`) used by the leaderboard, the history, the run comparison, the report and the auto-selected insights model — the bug that shipped mid-wave was exactly that a fourth site still sorted on test and opened a different model than the one crowned. **(2) The split was always random, even on dated data.** A chronological split (oldest rows train, newest test, rows without a parseable date dropped and counted) and a group split (no group on both sides — the same customer in train and test is the same leak) are offered when a column supports one, and **announced** in the run info. **(3) A predictive leak detector**: V6 caught columns that MAP to the target; this catches the merely predictive one — a one-column stump fitted on train and scored on validation, and a lone column reading the target at ≥ 99% shows as a copper warning with its measured score, never as a victory. **(4) A robust leaderboard on demand**: 5×2 repeated cross-validation over train+validation (the pipeline refitted inside every fold, the test set never touched), reporting a mean, a spread, and how often the leader actually beat the runner-up — « 10 of 10 folds: the order is stable » or « 6 of 10: treat them as tied ». **A defect the wave exposed and named**: with the smaller train split, Gaussian Naive Bayes on ~150 TF-IDF features saturates to exactly 0/1, so V24's word-effect occlusion measured exactly zero for every word and the card simply vanished — reading as « no word matters », which is false. Measured (2 distinct probabilities out of 48 test rows, against 48 for logistic and gbdt), the card now **refuses by name** and points at a model that can answer. 369 unit tests, 65 e2e. | Owner request (22/08/2026), launched 23/08/2026. Two of the four items were defects rather than gaps: a lab that sells honest evaluation cannot ship a headline figure it knows to be optimistic, nor a split that leaks on dated data. | +| V36 | **ML Lab: the gaps that were deliberately left open.** **Class imbalance** — weighting was explicitly descoped in V16; class weights in logistic regression, tree, forest and gbdt, announced like every other choice, finally complete what the threshold panel started. **Multiclass thresholds** — set aside since V16 and still open. **Choosing the ranking metric** — accuracy and RMSE are imposed today, while on an imbalanced problem F1 or recall is the right criterion and the ranking changes with it; small to build, and it makes the leaderboard answer the user's question rather than ours. **An ensemble of the best** — average or vote over the top three: typically 1–3 points, free in compute since the models are already trained, and it teaches why ensembling works. **What this wave deliberately does not do**: add a tenth model family (nine is plenty; a tenth improves neither honesty nor understanding), build an AutoML « we handle everything » mode (the opposite of a lab that shows its decisions), or bring in tabular deep learning (high cost, no gain at this scale, and no longer hand-written). | Each item was consciously deferred in an earlier wave rather than forgotten; grouping them keeps the descopes visible instead of letting them quietly become permanent. | +| V37 | **ML Lab: speed and the comfort of long sessions.** **Parallel training** — the zoo trains sequentially in a single worker; N workers means N cores, and at a million rows that is a different experience entirely. The V25 benches already exist to measure it before and after, so the gain is published rather than claimed. **Comparing more than two runs** — V21 compares two; three or four changes what the tool is for, and the diff machinery is already written. **Resuming an interrupted run** — closing the tab loses everything today, while V13 (artifacts) and V19 (persistence) already provide the storage; what is missing is a checkpoint between model families and the offer to resume. | Comes last on purpose: speed and comfort matter, but a faster wrong number is still a wrong number. V35 first, then V36, then this. | +| V38 | **Data Studio: reading the file exactly as it was written.** The headline item is a **defect in shipped code, not a missing feature**. `Papa.parse` is called with `skipEmptyLines: true` and nothing else — no encoding, no decimal separator — and `parseNumber` ends in `Number(cleaned)`. A French Excel export therefore breaks silently: `12,5` becomes `NaN`, the column is classified **text** rather than numeric, and every downstream stage one-hot encodes what should have been a number; a windows-1252 file displays `Québec`. Nothing warns, nothing refuses — the pipeline simply produces a worse model. Fix: **detect encoding and decimal separator and announce both** (« séparateur décimal : virgule, détecté sur 412 valeurs »), expose explicit **delimiter / encoding / decimal** selectors for the cases detection cannot settle, and show a **5-row preview before committing to the load** so a wrong guess is caught in two seconds rather than three panels later. The same pass covers thousands separators and dates written `31/12/2025` instead of ISO. | Owner request (22/08/2026): what to improve in /data. The audit found a defect first: a French-locale CSV — the single most likely file this owner's users will open — loses its numeric columns silently, and silence is the part that violates the project's rules. | +| V39 | **Data Studio: a recipe that works column by column.** `RecipeOptions` today applies `missing` and `clipOutliers` to the **whole file**: one strategy for every column, however different they are. A median makes sense for an age and none at all for a postcode. Make the recipe an ordered **list of per-column steps** — the current global settings becoming the defaults a column may override — and add the strategies that are missing: **median / mean / constant / a « MANQUANT » category** for categorical columns. With them comes a rule the tool should never break: **imputing without marking destroys information**, so every imputed column gains an optional **missing indicator** (`col_absent`), which is frequently predictive in its own right (a blank field is rarely blank at random). The recipe stays what it already is — a replayable, inspectable object — so the per-column version remains exportable, re-appliable to a new file, and legible as a list of named decisions. | A single global strategy is the kind of default that looks tidy and quietly makes the data worse; per-column steps cost little to build because the recipe is already an object, not a pile of checkboxes. | +| V40 | **Data Studio: validity, drift, and an auditable diff.** Quality is measured today as completeness and consistency of type; what is missing is **validity** — a value can be present, well-typed and still impossible. Named rules, each stated in plain language: an age of 200, a date in the future, a percentage at 130, a malformed postcode. Then **cross-column consistency** (`date_fin < date_debut`, `total ≠ quantité × prix`), for which **V29's DuckDB is already the engine** — the rules are SQL, and they run on the file that is already registered. Then three things that make the studio auditable rather than merely helpful: a **replayable reference profile** so a second file can be checked for drift against the first (the same idea as the V22 model manifest), a **before/after diff of the rows a recipe modified** — which rows, which columns, which values, not just a count — and a **breakdown of the quality score** so the number is explained by its parts instead of being asserted. Ends with **Parquet export**, nearly free now that DuckDB is loaded (`COPY … TO 'x.parquet'`). **What this wave deliberately does not do**: a spreadsheet-style cell editor (hand edits break reproducibility — the recipe is the record), fuzzy deduplication (guaranteed false positives on names and addresses, silently merging two real people), or model-based imputation (opaque, and it fabricates values that look plausible). | Comes last because it builds on V38's faithful read and V39's per-column recipe: validity rules on mis-parsed numbers would flag the parser, not the data. | + +**Ordering**: V38 comes before V39 and V40, and for the same reason V35 came first in its own group: its headline item is a defect in shipped code, not a feature — a studio that promises honest data cannot silently turn `12,5` into `NaN`. V35 is delivered; V36 and V37 follow it. V32 ships one finished tutorial before any reference page — the tutorial is the template the rest copies, and settling it late means rewriting everything. V30 and V31 both start with a bench, because neither « a bigger model » nor « it still makes mistakes » is a measurable statement today; no wave starts without an explicit launch command. V23 first (owner request); V24 keeps its vocabulary capped — V25 (delivered) chose announced sampling and a named memory guard over the typed-array rewrite, which measurement showed unnecessary; widening the vocabulary stays possible later. Set aside for now: multiclass thresholds. V12 diff --git a/README.md b/README.md index 12521ba..38c9309 100644 --- a/README.md +++ b/README.md @@ -26,8 +26,11 @@ The project follows three non-negotiable principles: pinned to the served header by a unit test, so the page cannot claim a protection the site stopped shipping. - **Honest evaluation.** Every run is scored against a naive baseline on a held-out test - set. Metrics ship with 95% bootstrap intervals, per-segment breakdowns, calibration - curves, and explicit refusals when a number would be noise (tiny slices, tiny test sets). + set. Models are **selected on a validation split and reported on a third, never-selected + test split**, with the gap between the two shown — crowning the best of nine on the + reporting set is what makes a headline figure optimistic. Metrics ship with 95% bootstrap + intervals, per-segment breakdowns, calibration curves, and explicit refusals when a number + would be noise (tiny slices, tiny test sets, a model whose probabilities are saturated). - **Hand-written, deterministic ML.** The model zoo, search, explanations, and statistics are implemented from scratch in TypeScript, seeded end to end — the same seed always reproduces the same run. @@ -99,7 +102,10 @@ The project follows three non-negotiable principles: post-processing (YOLOX grid decode, IoU, non-maximum suppression). - **Leakage discipline.** Preprocessing (imputation, one-hot/ordinal encoding, standardization) is fitted on the training split only; cross-validation refits the - pipeline inside each fold; forecast backtests never peek at the future. + pipeline inside each fold; forecast backtests never peek at the future. Dated files can + be split **chronologically** and grouped files **by group**, both announced — a random + split puts the future in training. A one-column stump flags any lone column that predicts + the target at 99%: that is a leak warning, never a victory. - **Determinism.** A single seed drives splits, model initialization, search, sampling and resampling — runs are exactly reproducible, and the test suite depends on it. - **Scale, honestly.** 100k–1M-row files train comfortably: past 100 000 usable rows an @@ -110,7 +116,7 @@ The project follows three non-negotiable principles: - **Performance.** Every section serves a prerendered static shell (hero paints before JavaScript); Lighthouse mobile ≈ 0.99 on `/ml` under real throttling. Heavy dependencies (Dexie, SheetJS, ONNX Runtime) load lazily. -- **Quality bar.** 352 unit tests, 61 Playwright end-to-end tests (including offline PWA, +- **Quality bar.** 369 unit tests, 65 Playwright end-to-end tests (including offline PWA, fake-webcam and axe-core WCAG A/AA accessibility checks), strict TypeScript, ESLint, Prettier, and Lighthouse budgets — all enforced in CI. diff --git a/e2e/split.spec.ts b/e2e/split.spec.ts new file mode 100644 index 0000000..d4c1bad --- /dev/null +++ b/e2e/split.spec.ts @@ -0,0 +1,87 @@ +import { expect, test } from '@playwright/test'; + +test.use({ locale: 'en-US' }); +test.setTimeout(120_000); + +// V35: the number stops flattering itself. Four things the user must SEE: +// the third split, the champion's selection-vs-test gap, an announced +// chronological split on dated data, and a lone column caught reading the +// target. Plus the 5x2 verdict on whether the ranking is real. + +test('iris: the winner is picked on validation and reports its test gap', async ({ page }) => { + await page.goto('/ml'); + await page.getByRole('button', { name: /iris\.csv/ }).click(); + await expect(page.getByText('150 rows · 5 columns')).toBeVisible(); + await page.selectOption('#target-select', 'species'); + await page.getByTestId('train-button').click(); + await expect(page.getByTestId('train-again')).toBeVisible({ timeout: 60_000 }); + + const leaderboard = page.getByTestId('leaderboard'); + // The ranked column is now the validation metric, with test beside it. + await expect(leaderboard).toContainText('Accuracy (val)'); + await expect(leaderboard).toContainText('Test'); + // Three splits are announced, not two. + await expect(leaderboard).toContainText(/validation rows/); + + // The champion line names both numbers and the gap between them. + const gap = page.getByTestId('champion-gap'); + await expect(gap).toBeVisible(); + // Iris is separable enough that the top models reach 1.000 — the assertion + // pins the SHAPE of the sentence (two figures and a gap), not the values. + await expect(gap).toContainText(/was selected on validation at \d\.\d{3} and scores \d\.\d{3}/); + await expect(gap).toContainText('never-selected split'); +}); + +test('iris: 5x2 cross-validation says whether the ranking is real', async ({ page }) => { + await page.goto('/ml'); + await page.getByRole('button', { name: /iris\.csv/ }).click(); + await page.selectOption('#target-select', 'species'); + await page.getByTestId('train-button').click(); + await expect(page.getByTestId('train-again')).toBeVisible({ timeout: 60_000 }); + + await page.getByTestId('robust-run').click(); + const verdict = page.getByTestId('robust-verdict'); + await expect(verdict).toBeVisible({ timeout: 90_000 }); + // Either wording is correct — what matters is that it counts the folds. + await expect(verdict).toContainText(/of 10 folds/); + // The panel says out loud that the test set stayed out of the folds. + await expect(page.getByTestId('robust-rank')).toContainText('test set is never touched'); +}); + +test('titanic: re-including the mirrored column raises a named leak warning', async ({ page }) => { + await page.goto('/ml'); + await page.getByRole('button', { name: /titanic\.csv/ }).click(); + await expect(page.getByText('891 rows · 15 columns')).toBeVisible(); + await page.selectOption('#target-select', 'survived'); + + // V6 excludes `alive` automatically; the user overrides that decision. + await page.getByTestId('column-card-alive').getByRole('button', { name: 'Include' }).click(); + await page.getByTestId('train-button').click(); + await expect(page.getByTestId('train-again')).toBeVisible({ timeout: 90_000 }); + + // V35 catches it again at training time — with a measured number. + const warning = page.getByTestId('leak-warning'); + await expect(warning).toBeVisible(); + await expect(warning).toContainText('« alive » alone predicts the target at 100.0%'); + await expect(warning).toContainText('almost always leakage'); +}); + +test('energy: a dated file offers — and announces — a chronological split', async ({ page }) => { + await page.goto('/ml'); + await page.getByRole('button', { name: /energy\.csv/ }).click(); + await expect(page.getByText('240 rows · 3 columns')).toBeVisible(); + await page.selectOption('#target-select', 'kwh'); + + // The option exists because `date` is a date column — and only then. + const split = page.getByTestId('split-mode'); + await expect(split).toBeVisible(); + await split.selectOption('chronological:date'); + + await page.getByTestId('train-button').click(); + await expect(page.getByTestId('train-again')).toBeVisible({ timeout: 60_000 }); + + // Announced in the run info, like every other decision the lab makes. + await expect(page.getByTestId('leaderboard')).toContainText( + 'chronological split on date (oldest rows train, newest test)', + ); +}); diff --git a/e2e/text.spec.ts b/e2e/text.spec.ts index f690290..0808ca6 100644 --- a/e2e/text.spec.ts +++ b/e2e/text.spec.ts @@ -26,10 +26,17 @@ test('reviews: the text column trains, and the words explain the model', async ( await expect(page.getByTestId('insights')).toBeVisible({ timeout: 30_000 }); await expect(page.getByTestId('importance')).toContainText('review'); - // The words card: signed effects, one word per row, with the honest note. + // V35: on this file the crowned model is Gaussian Naive Bayes, whose + // probabilities saturate to 0/1 — occlusion then measures exactly zero. + // The card must SAY that rather than vanish, and point at a way forward. const words = page.getByTestId('word-effects'); await expect(words).toBeVisible(); - await expect(words).toContainText('Words that move the answer'); + await expect(words).toContainText('its probabilities are saturated'); + await expect(words).toContainText('Pick another model in the leaderboard'); + + // And on a model that gives graded probabilities, the words do speak. + await page.getByTestId('leaderboard').getByText('Gradient boosting').click(); + await expect(words).toContainText('Words that move the answer', { timeout: 30_000 }); // Effects are signed — at least one word pushes each way on this dataset. await expect(words).toContainText('+'); await expect(words).toContainText('−'); @@ -47,5 +54,8 @@ test('reviews in French: the words card speaks French too', async ({ page }) => const words = page.getByTestId('word-effects'); await expect(words).toBeVisible({ timeout: 30_000 }); - await expect(words).toContainText('Les mots qui font bouger la réponse'); + // Le refus est traduit lui aussi — une carte qui se tait n'apprend rien. + await expect(words).toContainText('ses probabilités sont saturées'); + await page.getByTestId('leaderboard').getByText('Gradient boosting').click(); + await expect(words).toContainText('Les mots qui font bouger la réponse', { timeout: 30_000 }); }); diff --git a/src/features/ml/components/LeaderboardTable.tsx b/src/features/ml/components/LeaderboardTable.tsx index 10cb091..3ff5278 100644 --- a/src/features/ml/components/LeaderboardTable.tsx +++ b/src/features/ml/components/LeaderboardTable.tsx @@ -1,6 +1,7 @@ -import { Eye } from 'lucide-react'; +import { AlertTriangle, Eye } from 'lucide-react'; import { useTranslation } from 'react-i18next'; import { Badge } from '@/components/ui/badge'; +import { championGap, rankingValue, sortResults } from '@/features/ml/train/ranking'; import type { TaskType } from '@/features/ml/data/types'; import type { ModelResult, TrainSummary } from '@/features/ml/train/types'; import { cn } from '@/lib/utils'; @@ -26,6 +27,13 @@ interface LeaderboardTableProps { * Presentational model ranking — used live in the lab and read-only in * stored/shared run views. Classification ranks by accuracy (higher wins), * regression by RMSE (lower wins), with the delta vs baseline made explicit. + * + * V35: when a run carries validation scores, the table ranks and crowns on + * the VALIDATION metric and shows the test metric beside it — selecting on + * the reporting set made the crowned number the optimistic max of nine + * draws. The champion line spells out the val→test gap: that gap is the + * most useful lesson the lab can teach. Runs stored before V35 carry no + * validation scores and keep their historical, test-ranked display. */ export function LeaderboardTable({ results, @@ -37,14 +45,17 @@ export function LeaderboardTable({ const { t, i18n } = useTranslation(); const lang = i18n.resolvedLanguage ?? 'en'; const isClassification = taskType !== 'regression'; - const ok = results.filter((r) => r.ok); const failed = results.filter((r) => !r.ok); - const sorted = [...ok].sort((a, b) => - isClassification ? b.primary - a.primary : a.primary - b.primary, - ); - const baseline = ok.find((r) => r.key === 'baseline'); + const sorted = sortResults(results, taskType); + const baseline = sorted.find((r) => r.key === 'baseline'); const bestKey = sorted[0]?.key; - const maxPrimary = Math.max(...ok.map((r) => r.primary), 1e-9); + const hasValidation = sorted.some((r) => r.valPrimary !== undefined); + const champion = championGap(results, taskType); + const maxPrimary = Math.max(...sorted.map((r) => rankingValue(r)), 1e-9); + const leakWarnings = summary?.leakWarnings ?? []; + + const metricsOf = (result: ModelResult) => + hasValidation ? (result.valMetrics ?? result.metrics) : result.metrics; const metricColumns: { key: keyof ModelResult['metrics']; label: string }[] = isClassification ? [ @@ -60,25 +71,51 @@ export function LeaderboardTable({ function delta(result: ModelResult): string { if (!baseline || result.key === 'baseline') return '—'; const value = isClassification - ? result.primary - baseline.primary - : baseline.primary - result.primary; + ? rankingValue(result) - rankingValue(baseline) + : rankingValue(baseline) - rankingValue(result); const sign = value > 0 ? '+' : ''; return `${sign}${value.toFixed(3)}`; } + const primaryHeader = isClassification + ? t(hasValidation ? 'ml.lab.leaderboard.accuracyVal' : 'ml.lab.leaderboard.accuracy') + : t(hasValidation ? 'ml.lab.leaderboard.rmseVal' : 'ml.lab.leaderboard.rmse'); + return (
+ {leakWarnings.length > 0 && ( +
+
+ )} - + + {hasValidation && ( + + )} {metricColumns.map(({ key, label }) => ( + )} ))} - ))}
# {t('ml.lab.leaderboard.model')} - {isClassification ? t('ml.lab.leaderboard.accuracy') : t('ml.lab.leaderboard.rmse')} - {primaryHeader} + {t('ml.lab.leaderboard.test')} + {t('ml.lab.leaderboard.delta')} @@ -121,13 +158,16 @@ export function LeaderboardTable({ )} - {formatMetric(result.primary)} + {formatMetric(rankingValue(result))} + {hasValidation && ( + {formatMetric(result.primary)} {metricColumns.map(({ key }) => ( - {formatMetric(result.metrics[key])} + {formatMetric(metricsOf(result)[key])} @@ -178,27 +218,67 @@ export function LeaderboardTable({ {t('ml.lab.leaderboard.failed')} + {result.error}
+ {champion && ( +

+ {t('ml.lab.leaderboard.championLine', { + model: t(`ml.lab.models.${champion.model.key}`), + val: formatMetric(champion.val), + test: formatMetric(champion.test), + gap: `${champion.gap > 0 ? '+' : ''}${champion.gap.toFixed(3)}`, + })}{' '} + {t('ml.lab.leaderboard.championWhy')} +

+ )} {summary && (

- {t('ml.lab.leaderboard.runInfo', { - seed: summary.seed, - train: summary.trainRows, - test: summary.testRows, - features: summary.featureCount, - })} + {summary.validationRows !== undefined + ? t('ml.lab.leaderboard.runInfoVal', { + seed: summary.seed, + train: summary.trainRows, + val: summary.validationRows, + test: summary.testRows, + features: summary.featureCount, + }) + : t('ml.lab.leaderboard.runInfo', { + seed: summary.seed, + train: summary.trainRows, + test: summary.testRows, + features: summary.featureCount, + })} + {summary.split !== undefined && ( + <> + {' '} + ·{' '} + {t( + summary.split.mode === 'chronological' + ? 'ml.lab.leaderboard.splitChronological' + : 'ml.lab.leaderboard.splitGroup', + { column: summary.split.column }, + )} + {summary.split.dropped !== undefined && + ` ${t('ml.lab.leaderboard.splitDropped', { count: summary.split.dropped })}`} + + )} {summary.sampledFrom !== undefined && ( <> {' '} ·{' '} {t('ml.lab.leaderboard.sampledFrom', { - cap: (summary.trainRows + summary.testRows).toLocaleString(lang), + cap: ( + summary.trainRows + + (summary.validationRows ?? 0) + + summary.testRows + ).toLocaleString(lang), from: summary.sampledFrom.toLocaleString(lang), })} diff --git a/src/features/ml/components/RobustRankPanel.tsx b/src/features/ml/components/RobustRankPanel.tsx new file mode 100644 index 0000000..4e4576d --- /dev/null +++ b/src/features/ml/components/RobustRankPanel.tsx @@ -0,0 +1,133 @@ +import { Loader2, Scale, Square } from 'lucide-react'; +import { useTranslation } from 'react-i18next'; +import { Button } from '@/components/ui/button'; +import { Eyebrow } from '@/components/ui/eyebrow'; +import { useLabStore } from '@/features/ml/lab-store'; +import type { RobustRankResult } from '@/features/ml/train/robust'; + +/** + * V35: the robust leaderboard — 5×2 repeated cross-validation, on demand. + * + * A single split's ranking can hinge on which rows landed in the draw; ten + * seeded fits per family produce a mean, a spread, and a plain answer to the + * only question that matters: is the order between the top two models real, + * or inside the noise? Runs on train+validation only — the test set is never + * part of the folds. + */ +export function RobustRankPanel() { + const { t, i18n } = useTranslation(); + const trainStatus = useLabStore((s) => s.trainStatus); + const results = useLabStore((s) => s.results); + const robustStatus = useLabStore((s) => s.robustStatus); + const robustProgress = useLabStore((s) => s.robustProgress); + const outcome = useLabStore((s) => s.robustOutcome); + const robustRank = useLabStore((s) => s.robustRank); + const cancelRobust = useLabStore((s) => s.cancelRobust); + + if (trainStatus !== 'done' || results.filter((r) => r.ok).length < 2) return null; + + const lang = i18n.resolvedLanguage ?? 'en'; + const running = robustStatus === 'running'; + const fmt = (v: number) => + v.toLocaleString(lang, { minimumFractionDigits: 3, maximumFractionDigits: 3 }); + + return ( +

+
+
+
+

{t('ml.lab.robust.hint')}

+
+ +
+ {running ? ( + <> + + + + + ) : ( + + )} +
+ + {outcome && !running && } +
+ ); +} + +function RobustOutcome({ + outcome, + fmt, + lang, +}: { + outcome: RobustRankResult; + fmt: (v: number) => string; + lang: string; +}) { + const { t } = useTranslation(); + const folds = outcome.reps * 2; + const pair = outcome.topPair; + const stable = pair !== null && pair.leaderWins >= folds - 1; + + return ( +
+
+ + + + + + + + + + + {outcome.entries.map((entry, rank) => ( + + + + + + + ))} + +
#{t('ml.lab.leaderboard.model')}{t('ml.lab.robust.mean')}{t('ml.lab.robust.sd')}
{rank + 1}{t(`ml.lab.models.${entry.model}`)}{fmt(entry.mean)}± {fmt(entry.sd)}
+
+ {pair && ( +

+ {t(stable ? 'ml.lab.robust.verdictStable' : 'ml.lab.robust.verdictNoise', { + leader: t(`ml.lab.models.${pair.leader}`), + runnerUp: t(`ml.lab.models.${pair.runnerUp}`), + wins: pair.leaderWins, + folds: pair.folds, + })} +

+ )} +

+ {t('ml.lab.robust.note', { + folds, + reps: outcome.reps, + rows: outcome.rows.toLocaleString(lang), + })} +

+
+ ); +} diff --git a/src/features/ml/components/RunArtifactsView.tsx b/src/features/ml/components/RunArtifactsView.tsx index 1aeb653..597d3d6 100644 --- a/src/features/ml/components/RunArtifactsView.tsx +++ b/src/features/ml/components/RunArtifactsView.tsx @@ -46,7 +46,7 @@ export function RunArtifactsView({ artifacts }: { artifacts: RunArtifacts }) { const { t, i18n } = useTranslation(); const lang = i18n.resolvedLanguage ?? 'en'; const { tuning, explanation, exploration, forecast, batchScore, threshold, segments } = artifacts; - const { uncertainty, learningCurve } = artifacts; + const { uncertainty, learningCurve, robustRank } = artifacts; if ( !tuning && !explanation && @@ -56,7 +56,8 @@ export function RunArtifactsView({ artifacts }: { artifacts: RunArtifacts }) { !threshold && !segments && !uncertainty && - !learningCurve + !learningCurve && + !robustRank ) { return null; } @@ -291,6 +292,51 @@ export function RunArtifactsView({ artifacts }: { artifacts: RunArtifacts }) { )} + {robustRank && ( + + + + + + + + + + + {robustRank.entries.map((entry) => ( + + + + + + ))} + +
{t('ml.lab.leaderboard.model')}{t('ml.lab.robust.mean')}{t('ml.lab.robust.sd')}
{t(`ml.lab.models.${entry.model}`)}{score(entry.mean)}± {score(entry.sd)}
+ {robustRank.topPair && ( +

+ {t( + robustRank.topPair.leaderWins >= robustRank.reps * 2 - 1 + ? 'ml.lab.robust.verdictStable' + : 'ml.lab.robust.verdictNoise', + { + leader: t(`ml.lab.models.${robustRank.topPair.leader}`), + runnerUp: t(`ml.lab.models.${robustRank.topPair.runnerUp}`), + wins: robustRank.topPair.leaderWins, + folds: robustRank.topPair.folds, + }, + )} +

+ )} +

+ {t('ml.lab.robust.note', { + folds: robustRank.reps * 2, + reps: robustRank.reps, + rows: robustRank.rows.toLocaleString(lang), + })} +

+ + )} + {learningCurve && (

diff --git a/src/features/ml/components/RunsHistory.tsx b/src/features/ml/components/RunsHistory.tsx index 0d1d02b..96380ae 100644 --- a/src/features/ml/components/RunsHistory.tsx +++ b/src/features/ml/components/RunsHistory.tsx @@ -12,6 +12,7 @@ import { formatSize } from '@/features/ml/projects/dataset-storage'; import { useLabStore } from '@/features/ml/lab-store'; import { cn } from '@/lib/utils'; import type { RunRecord } from '@/features/ml/projects/types'; +import { bestResult } from '@/features/ml/train/ranking'; /** Datasets kept in the browser (v19) — reopen or forget, all local. */ function SavedDatasets() { @@ -92,11 +93,9 @@ function SavedDatasets() { } function bestOf(record: RunRecord) { - const ok = record.results.filter((r) => r.ok); - const isClassification = record.taskType !== 'regression'; - return [...ok].sort((a, b) => - isClassification ? b.primary - a.primary : a.primary - b.primary, - )[0]; + // V35: same ranking rule as the leaderboard — validation when the run has + // it, test otherwise. The history must not crown a different model. + return bestResult(record.results, record.taskType) ?? undefined; } /** Local run history (IndexedDB) — rename, delete, view, compare two runs. */ diff --git a/src/features/ml/components/TrainPanel.tsx b/src/features/ml/components/TrainPanel.tsx index 2cad51c..82ff2a1 100644 --- a/src/features/ml/components/TrainPanel.tsx +++ b/src/features/ml/components/TrainPanel.tsx @@ -9,7 +9,10 @@ import { UncertaintyPanel } from '@/features/ml/components/UncertaintyPanel'; import { LearningCurvePanel } from '@/features/ml/components/LearningCurvePanel'; import { TuningPanel } from '@/features/ml/components/TuningPanel'; import { InsightsSection } from '@/features/ml/components/insights/InsightsSection'; +import { RobustRankPanel } from '@/features/ml/components/RobustRankPanel'; import { TEST_RATIO, TRAIN_SEED, useLabStore } from '@/features/ml/lab-store'; +import { VALIDATION_RATIO } from '@/features/ml/train/trainer'; +import type { SplitChoice } from '@/features/ml/train/types'; export function TrainPanel() { const { t } = useTranslation(); @@ -18,9 +21,31 @@ export function TrainPanel() { const modelProgress = useLabStore((s) => s.modelProgress); const train = useLabStore((s) => s.train); const cancelTrain = useLabStore((s) => s.cancelTrain); + const profiles = useLabStore((s) => s.profiles); + const target = useLabStore((s) => s.target); + const splitChoice = useLabStore((s) => s.splitChoice); + const setSplitChoice = useLabStore((s) => s.setSplitChoice); if (!task) return null; - const splitLabel = `${Math.round((1 - TEST_RATIO) * 100)}/${Math.round(TEST_RATIO * 100)}`; + // V35: the split is now three-way — validation is carved from the train + // side, the test share is untouched. 64/16/20 with the default ratios. + const testPct = Math.round(TEST_RATIO * 100); + const valPct = Math.round((1 - TEST_RATIO) * VALIDATION_RATIO * 100); + const splitLabel = `${100 - testPct - valPct}/${valPct}/${testPct}`; + + // V35: announced non-random splits — offered only when a column supports + // one. Chronological needs a date column; group needs a repeated identifier. + const splitOptions: SplitChoice[] = [ + ...profiles + .filter((p) => p.type === 'date' && p.name !== target) + .map((p) => ({ mode: 'chronological' as const, column: p.name })), + ...profiles + .filter( + (p) => p.type === 'id' && p.name !== target && p.cardinality < p.rowCount - p.missingCount, + ) + .map((p) => ({ mode: 'group' as const, column: p.name })), + ]; + const choiceValue = splitChoice ? `${splitChoice.mode}:${splitChoice.column}` : 'random'; return (

@@ -57,12 +82,47 @@ export function TrainPanel() { {t('ml.lab.trainConfig', { seed: TRAIN_SEED, split: splitLabel })} + {splitOptions.length > 0 && trainStatus !== 'training' && ( + + )}
+ + diff --git a/src/features/ml/components/insights/InsightsSection.tsx b/src/features/ml/components/insights/InsightsSection.tsx index 6b8546a..24b142e 100644 --- a/src/features/ml/components/insights/InsightsSection.tsx +++ b/src/features/ml/components/insights/InsightsSection.tsx @@ -54,11 +54,12 @@ export function InsightsSection() { {insights.scatter && } {insights.residuals && } - {insights.words && insights.words.length > 0 && ( + {((insights.words && insights.words.length > 0) || insights.wordsRefused) && ( )} {insights.pdp?.map(({ column, points }) => ( diff --git a/src/features/ml/components/insights/WordEffects.tsx b/src/features/ml/components/insights/WordEffects.tsx index c6d7d76..95acb0f 100644 --- a/src/features/ml/components/insights/WordEffects.tsx +++ b/src/features/ml/components/insights/WordEffects.tsx @@ -5,19 +5,38 @@ import { Eyebrow } from '@/components/ui/eyebrow'; * Signed word effects (V24): a diverging bar per word, drawn from a centre * line so the direction is the first thing you read — words pushing the * answer up go right in teal, words pushing it down go left in copper. + * + * V35: when the method cannot measure — a model whose probabilities are + * saturated shifts by exactly zero for every word — the card still appears + * and says why. Vanishing would read as "no word matters", which is false. */ export function WordEffects({ words, isClassification, positiveClass, + refusal, }: { words: { column: string; term: string; effect: number; rows: number }[]; isClassification: boolean; positiveClass?: string; + refusal?: 'saturated'; }) { const { t } = useTranslation(); const peak = Math.max(...words.map((word) => Math.abs(word.effect)), 1e-9); + if (refusal !== undefined) { + return ( +
+ {t('ml.lab.insights.wordsTitle')} +

{t('ml.lab.insights.wordsSaturated')}

+

{t('ml.lab.insights.wordsSaturatedAdvice')}

+
+ ); + } + return (
) => { ); if (cancelCurve) post({ kind: 'curve-cancelled' }); else post({ kind: 'curve-complete', payload: outcome }); + } else if (request.kind === 'cancel-robust') { + cancelRobust = true; + } else if (request.kind === 'robust-rank') { + cancelRobust = false; + if (!artifacts) throw new Error('no-run'); + const profiles: ColumnProfile[] = header.map((column, i) => + profileColumn(column, columns[i]), + ); + // V35: ten fits per family on train+validation halves. The test rows + // never enter the folds — this reranks, it does not re-test. + const payload = await robustRank(columnsAsMap(), profiles, request.config, { + onProgress: (done, total) => post({ kind: 'robust-progress', done, total }), + isCancelled: () => cancelRobust, + }); + if (payload === null) post({ kind: 'robust-cancelled' }); + else post({ kind: 'robust-complete', payload }); } else if (request.kind === 'cancel-tune') { cancelTuning = true; } else if (request.kind === 'tune') { diff --git a/src/features/ml/lab-store.ts b/src/features/ml/lab-store.ts index 53e8b2c..1986544 100644 --- a/src/features/ml/lab-store.ts +++ b/src/features/ml/lab-store.ts @@ -15,6 +15,8 @@ import type { ThresholdAnalysis } from '@/features/ml/train/threshold-analysis'; import type { UncertaintyAnalysis } from '@/features/ml/train/uncertainty'; import type { TunableKey, TuneOutcome } from '@/features/ml/train/search'; import type { LearningCurveOutcome } from '@/features/ml/train/learning-curve'; +import type { RobustRankResult } from '@/features/ml/train/robust'; +import type { SplitChoice } from '@/features/ml/train/types'; import type { ExplorationPayload } from '@/features/ml/unsupervised/explore'; import type { ForecastPayload } from '@/features/ml/timeseries/run'; import type { ShapleyExplanation } from '@/features/ml/train/shapley'; @@ -26,6 +28,7 @@ import type { WhatIfResult, } from '@/features/ml/train/types'; import type { WorkerRequest, WorkerResponse } from '@/features/ml/worker-protocol'; +import { bestResult } from '@/features/ml/train/ranking'; export type LabStatus = 'idle' | 'parsing' | 'ready' | 'error'; export type TrainStatus = 'idle' | 'training' | 'done'; @@ -49,6 +52,8 @@ interface LabState { leaks: ColumnSuggestion[]; /** Manual include/exclude decisions that override the suggestions. */ overrides: Record; + /** V35: announced non-random split, chosen in the UI. Null = seeded random. */ + splitChoice: SplitChoice | null; trainStatus: TrainStatus; modelProgress: { key: ModelKey; index: number; total: number } | null; results: ModelResult[]; @@ -63,6 +68,9 @@ interface LabState { tuneOutcome: TuneOutcome | null; curveStatus: 'idle' | 'running' | 'done'; curveProgress: { done: number; total: number } | null; + robustStatus: 'idle' | 'running' | 'done'; + robustProgress: { done: number; total: number } | null; + robustOutcome: RobustRankResult | null; /** null after a run = the worker refused (named): no curve theater. */ curveOutcome: LearningCurveOutcome | null; exploreStatus: 'idle' | 'running' | 'done'; @@ -112,6 +120,9 @@ interface LabState { tune: (model: TunableKey) => void; cancelTune: () => void; learningCurve: (model: ModelKey) => void; + robustRank: () => void; + cancelRobust: () => void; + setSplitChoice: (choice: SplitChoice | null) => void; cancelCurve: () => void; explore: () => void; forecast: (dateColumn: string, valueColumn: string) => void; @@ -147,6 +158,9 @@ const initialTraining = { curveStatus: 'idle' as const, curveProgress: null, curveOutcome: null as LearningCurveOutcome | null, + robustStatus: 'idle' as const, + robustProgress: null, + robustOutcome: null as RobustRankResult | null, exploreStatus: 'idle' as const, exploration: null, forecastStatus: 'idle' as const, @@ -164,6 +178,7 @@ const initialTraining = { const initialData = { status: 'idle' as LabStatus, + splitChoice: null as SplitChoice | null, error: null, rowsParsed: 0, meta: null, @@ -306,12 +321,10 @@ export const useLabStore = create((set, get) => { } else if (message.kind === 'train-complete') { set({ trainStatus: 'done', modelProgress: null, summary: message.summary }); // Fetch insights for the winning model right away. - const ok = get().results.filter((r) => r.ok); - if (ok.length > 0) { - const isClassification = message.summary.taskType !== 'regression'; - const best = [...ok].sort((a, b) => - isClassification ? b.primary - a.primary : a.primary - b.primary, - )[0]; + // V35: the SAME ranking rule as the leaderboard — otherwise the + // table crowns one model and the insights panel opens another. + const best = bestResult(get().results, message.summary.taskType); + if (best !== null) { send({ kind: 'model-insights', model: best.key }); // Leaderboard-wide intervals ride along with every completed run. send({ kind: 'uncertainty-analysis' }); @@ -387,6 +400,13 @@ export const useLabStore = create((set, get) => { if (message.payload) attachArtifact({ learningCurve: message.payload }); } else if (message.kind === 'curve-cancelled') { set({ curveStatus: 'idle', curveProgress: null }); + } else if (message.kind === 'robust-progress') { + set({ robustProgress: { done: message.done, total: message.total } }); + } else if (message.kind === 'robust-complete') { + set({ robustStatus: 'done', robustProgress: null, robustOutcome: message.payload }); + attachArtifact({ robustRank: message.payload }); + } else if (message.kind === 'robust-cancelled') { + set({ robustStatus: 'idle', robustProgress: null }); } else if (message.kind === 'explore-result') { set({ exploreStatus: 'done', exploration: message.payload }); attachArtifact({ exploration: message.payload }); @@ -568,7 +588,13 @@ export const useLabStore = create((set, get) => { set({ ...initialTraining, trainStatus: 'training' }); send({ kind: 'train', - config: { target: state.target, features, seed: TRAIN_SEED, testRatio: TEST_RATIO }, + config: { + target: state.target, + features, + seed: TRAIN_SEED, + testRatio: TEST_RATIO, + ...(state.splitChoice !== null && { split: state.splitChoice }), + }, }); }, @@ -618,7 +644,13 @@ export const useLabStore = create((set, get) => { send({ kind: 'tune', model, - config: { target: state.target, features, seed: TRAIN_SEED, testRatio: TEST_RATIO }, + config: { + target: state.target, + features, + seed: TRAIN_SEED, + testRatio: TEST_RATIO, + ...(state.splitChoice !== null && { split: state.splitChoice }), + }, }); }, @@ -637,7 +669,13 @@ export const useLabStore = create((set, get) => { send({ kind: 'learning-curve', model, - config: { target: state.target, features, seed: TRAIN_SEED, testRatio: TEST_RATIO }, + config: { + target: state.target, + features, + seed: TRAIN_SEED, + testRatio: TEST_RATIO, + ...(state.splitChoice !== null && { split: state.splitChoice }), + }, }); }, @@ -646,6 +684,35 @@ export const useLabStore = create((set, get) => { send({ kind: 'cancel-curve' }); }, + robustRank() { + const state = get(); + if (!state.target || state.trainStatus !== 'done' || state.robustStatus === 'running') return; + const features = state.profiles + .map((p) => p.name) + .filter((name) => name !== state.target && effectiveExclusion(state, name) === null); + set({ robustStatus: 'running', robustProgress: null, robustOutcome: null }); + send({ + kind: 'robust-rank', + config: { + target: state.target, + features, + seed: TRAIN_SEED, + testRatio: TEST_RATIO, + ...(state.splitChoice !== null && { split: state.splitChoice }), + }, + }); + }, + + cancelRobust() { + if (get().robustStatus !== 'running') return; + send({ kind: 'cancel-robust' }); + }, + + setSplitChoice(choice) { + if (get().trainStatus === 'training') return; + set({ splitChoice: choice }); + }, + explore() { const state = get(); if (state.status !== 'ready' || state.exploreStatus === 'running') return; diff --git a/src/features/ml/projects/compare.ts b/src/features/ml/projects/compare.ts index 3d6c2a0..7c5c6df 100644 --- a/src/features/ml/projects/compare.ts +++ b/src/features/ml/projects/compare.ts @@ -8,6 +8,7 @@ import type { RunRecord } from '@/features/ml/projects/types'; import type { ModelInterval } from '@/features/ml/train/uncertainty'; import type { ModelKey } from '@/features/ml/train/types'; +import { bestResult } from '@/features/ml/train/ranking'; export interface ModelDelta { key: ModelKey; @@ -40,13 +41,11 @@ export interface RunComparison { } function bestOf(record: RunRecord): { key: ModelKey; primary: number } | null { - const ok = record.results.filter((r) => r.ok); - if (ok.length === 0) return null; - const isClassification = record.taskType !== 'regression'; - const sorted = [...ok].sort((x, y) => - isClassification ? y.primary - x.primary : x.primary - y.primary, - ); - return { key: sorted[0].key, primary: sorted[0].primary }; + // V35: ranking lives in one place — a run with validation scores is ranked + // on those, so the comparison crowns the same model the leaderboard did. + // The reported figure stays the TEST metric: that is what runs compare on. + const best = bestResult(record.results, record.taskType); + return best === null ? null : { key: best.key, primary: best.primary }; } export function compareRuns(a: RunRecord, b: RunRecord): RunComparison { diff --git a/src/features/ml/projects/report.ts b/src/features/ml/projects/report.ts index faab08f..9893887 100644 --- a/src/features/ml/projects/report.ts +++ b/src/features/ml/projects/report.ts @@ -1,6 +1,7 @@ import { METRIC_ROWS, metricDelta } from '@/features/ml/train/score-view'; import type { RunRecord } from '@/features/ml/projects/types'; import type { ClusterTrait } from '@/features/ml/unsupervised/explore'; +import { championGap, sortResults } from '@/features/ml/train/ranking'; type Translate = (key: string, options?: Record) => string; @@ -369,10 +370,9 @@ function artifactSections(record: RunRecord, t: Translate, lang: string): string */ export function buildReportHtml(record: RunRecord, t: Translate, lang: string): string { const isClassification = record.taskType !== 'regression'; - const ok = record.results.filter((r) => r.ok); - const sorted = [...ok].sort((a, b) => - isClassification ? b.primary - a.primary : a.primary - b.primary, - ); + // V35: ranked through the shared rule — validation when the run has it. + const sorted = sortResults(record.results, record.taskType); + const champion = championGap(record.results, record.taskType); const metricKeys = isClassification ? (['accuracy', 'f1', 'auc', 'logLoss'] as const) : (['rmse', 'mae', 'r2'] as const); @@ -396,12 +396,43 @@ export function buildReportHtml(record: RunRecord, t: Translate, lang: string): summary.sampledFrom !== undefined ? `

${esc( t('ml.lab.leaderboard.sampledFrom', { - cap: (summary.trainRows + summary.testRows).toLocaleString(lang), + cap: ( + summary.trainRows + + (summary.validationRows ?? 0) + + summary.testRows + ).toLocaleString(lang), from: summary.sampledFrom.toLocaleString(lang), }), )}

` : ''; + // V35: the report carries the same two honesty lines the lab shows — + // the champion's selection-vs-test gap, and any suspected target leak. + const championNote = champion + ? `

${esc( + t('ml.lab.leaderboard.championLine', { + model: t(`ml.lab.models.${champion.model.key}`), + val: fmt(champion.val), + test: fmt(champion.test), + gap: `${champion.gap > 0 ? '+' : ''}${champion.gap.toFixed(3)}`, + }), + )} ${esc(t('ml.lab.leaderboard.championWhy'))}

` + : ''; + + const leakNote = + summary.leakWarnings && summary.leakWarnings.length > 0 + ? `

${summary.leakWarnings + .map((w) => + esc( + t('ml.lab.leaderboard.leakWarning', { + column: w.column, + score: (w.score * 100).toFixed(1), + }), + ), + ) + .join(' ')} ${esc(t('ml.lab.leaderboard.leakAdvice'))}

` + : ''; + const confusion = record.insights.confusion && record.insights.classes ? `

${esc(t('ml.lab.insights.confusionTitle'))} — ${esc(t(`ml.lab.models.${record.insights.model}`))}

@@ -450,6 +481,7 @@ export function buildReportHtml(record: RunRecord, t: Translate, lang: string): body { font-family: system-ui, sans-serif; color: #17221f; margin: 2rem auto; max-width: 52rem; padding: 0 1rem; } h1 { font-size: 1.6rem; } h2 { font-size: 1.1rem; margin-top: 2rem; } .meta { color: #5a6a65; font-size: .9rem; font-family: ui-monospace, monospace; } + .warn { background: #f3e6da; color: #8f4c1f; padding: .6rem .8rem; border-radius: 8px; font-size: .9rem; } .read { background: #ebf1ef; border-radius: 12px; padding: 1rem 1.25rem; margin-top: 1.5rem; } table { border-collapse: collapse; margin-top: .75rem; font-size: .9rem; } th, td { border: 1px solid #d6e0dc; padding: .4rem .7rem; text-align: left; font-variant-numeric: tabular-nums; } @@ -473,6 +505,8 @@ ${read ? `
${esc(read)}
` : ''} .join('')}${esc(t('ml.lab.leaderboard.trainTime'))} ${leaderboardRows} +${leakNote} +${championNote} ${sampledNote} ${confusion} ${importance} diff --git a/src/features/ml/projects/types.ts b/src/features/ml/projects/types.ts index b9b0880..8fee4d2 100644 --- a/src/features/ml/projects/types.ts +++ b/src/features/ml/projects/types.ts @@ -7,6 +7,7 @@ import type { UncertaintyAnalysis } from '@/features/ml/train/uncertainty'; import type { ModelKey } from '@/features/ml/train/types'; import type { TuneOutcome } from '@/features/ml/train/search'; import type { LearningCurveOutcome } from '@/features/ml/train/learning-curve'; +import type { RobustRankResult } from '@/features/ml/train/robust'; import type { ExplorationPayload } from '@/features/ml/unsupervised/explore'; import type { ForecastPayload } from '@/features/ml/timeseries/run'; import type { InsightsPayload, ModelResult, TrainSummary } from '@/features/ml/train/types'; @@ -44,6 +45,8 @@ export interface RunArtifacts { uncertainty?: UncertaintyAnalysis; /** v26: metric vs training size, with the plain-language verdict. */ learningCurve?: LearningCurveOutcome; + /** v35: 5x2 CV ranking on train+validation - mean, spread, top-pair wins. */ + robustRank?: RobustRankResult; } /** diff --git a/src/features/ml/train/insights.ts b/src/features/ml/train/insights.ts index 925ecc0..bfa9e76 100644 --- a/src/features/ml/train/insights.ts +++ b/src/features/ml/train/insights.ts @@ -67,6 +67,11 @@ export function encodedBlocks( /** Words scored per text column: enough to read, few enough to stay fast. */ const WORD_CANDIDATES = 40; +export interface WordEffectsResult { + words: { column: string; term: string; effect: number; rows: number }[]; + /** Set when the method cannot measure anything — never silently empty. */ + refusal?: 'saturated'; +} /** A word seen in fewer test rows than this cannot be measured honestly. */ const MIN_WORD_TEST_ROWS = 3; @@ -85,6 +90,13 @@ const MIN_WORD_TEST_ROWS = 3; * p(positive class)) and regression (shift of the prediction) have a single * axis to project onto — multiclass is skipped rather than faked. Candidates * are the most present words in the test split, capped at WORD_CANDIDATES. + * + * V35 named a third limit, found when the smaller training split made it + * bite: a model whose probabilities are SATURATED — Gaussian Naive Bayes on + * ~150 TF-IDF features returns exactly 1 or exactly 0 — has no axis to move + * along, so every occlusion measures exactly zero. That is not "no words + * matter"; it is "this model cannot answer the question". It is refused by + * name and said out loud, instead of the card vanishing without explanation. */ export function wordEffects( model: TrainedModel, @@ -93,16 +105,16 @@ export function wordEffects( isClassification: boolean, classCount: number, top = 12, -): { column: string; term: string; effect: number; rows: number }[] { +): WordEffectsResult { const canProject = isClassification ? classCount === 2 && typeof model.predictProba === 'function' : true; - if (!canProject) return []; + if (!canProject) return { words: [] }; const textBlocks = encodedBlocks(pipeline).filter( (block) => pipeline.specs.find((spec) => spec.name === block.column)?.kind === 'text', ); - if (textBlocks.length === 0) return []; + if (textBlocks.length === 0) return { words: [] }; /** The model's answer on one axis: p(positive class), or the prediction. */ const project = (rows: number[][]): number[] => @@ -144,10 +156,14 @@ export function wordEffects( } } - return scored + const words = scored .filter((entry) => entry.effect !== 0) .sort((a, b) => Math.abs(b.effect) - Math.abs(a.effect) || a.term.localeCompare(b.term)) .slice(0, top); + // Measured, not guessed: candidates existed and not one of them moved the + // answer at all. That is a saturated model, not a text without signal. + if (words.length === 0 && scored.length > 0) return { words: [], refusal: 'saturated' }; + return { words }; } /** @@ -274,7 +290,8 @@ export function computeInsights(artifacts: TrainArtifacts, modelKey: ModelKey): isClassification, seed, ).slice(0, 10); - const words = wordEffects(model, pipeline, testX, isClassification, classes.length); + const wordOutcome = wordEffects(model, pipeline, testX, isClassification, classes.length); + const words = wordOutcome.words; if (isClassification) { const payload: InsightsPayload = { @@ -283,6 +300,7 @@ export function computeInsights(artifacts: TrainArtifacts, modelKey: ModelKey): confusion: confusionMatrix(testY, predictions, classes.length), importance, ...(words.length > 0 ? { words } : {}), + ...(wordOutcome.refusal !== undefined ? { wordsRefused: wordOutcome.refusal } : {}), }; if (classes.length === 2 && model.predictProba) { const roc = rocCurve( @@ -302,6 +320,7 @@ export function computeInsights(artifacts: TrainArtifacts, modelKey: ModelKey): residuals: residualsHistogram(testY, predictions), importance, ...(words.length > 0 ? { words } : {}), + ...(wordOutcome.refusal !== undefined ? { wordsRefused: wordOutcome.refusal } : {}), }; const pdp = partialDependence(model, pipeline, testX, false, importance); if (pdp.length > 0) payload.pdp = pdp; diff --git a/src/features/ml/train/leakage.ts b/src/features/ml/train/leakage.ts new file mode 100644 index 0000000..5e7d57b --- /dev/null +++ b/src/features/ml/train/leakage.ts @@ -0,0 +1,166 @@ +/** + * V35: the predictive leak scan. + * + * V6's `leakSuggestions` catches columns whose values MAP to the target (the + * "alive vs survived" case) and near-perfect linear correlations. What it + * misses is the merely *predictive* leak: `amount_refunded` does not map to + * `fraud`, yet a one-column rule reads it at 99%. A lone column that predicts + * the target almost alone is nearly always information from the future — it + * must show as a warning, never as a victory. + * + * Method, deliberately boring: per column, fit the cheapest possible + * one-column model on the TRAIN split (categorical: majority class / mean per + * value; numeric: majority class / mean per quantile bin), then score it on + * the held-out selection split. No leakage in the leak detector itself. + */ +import { isMissing, parseNumber } from '@/features/ml/data/infer'; +import type { Cell, ColumnProfile } from '@/features/ml/data/types'; + +/** Score at or above which a lone column is reported as a suspected leak. */ +export const LEAK_SCORE_THRESHOLD = 0.99; +/** Below this many evaluation rows a 99% reading is noise — the scan refuses. */ +export const LEAK_MIN_EVAL_ROWS = 20; +const NUMERIC_BINS = 32; + +export interface LeakWarning { + column: string; + /** Accuracy (classification) or R² (regression) of the one-column stump. */ + score: number; +} + +export function leakScan( + columns: Map, + profiles: ColumnProfile[], + featureColumns: string[], + trainIdx: number[], + evalIdx: number[], + encode: (row: number) => number, + isClassification: boolean, +): LeakWarning[] { + if (evalIdx.length < LEAK_MIN_EVAL_ROWS) return []; + + const warnings: LeakWarning[] = []; + for (const name of featureColumns) { + const profile = profiles.find((p) => p.name === name); + const values = columns.get(name); + if (!profile || !values) continue; + + const keyOf = keyFunction(profile, values, trainIdx); + const score = isClassification + ? stumpAccuracy(trainIdx, evalIdx, encode, keyOf) + : stumpR2(trainIdx, evalIdx, encode, keyOf); + if (score !== null && score >= LEAK_SCORE_THRESHOLD) { + warnings.push({ column: name, score }); + } + } + return warnings.sort((a, b) => b.score - a.score); +} + +/** Missing cells share one bucket; numeric columns use train quantile bins. */ +function keyFunction( + profile: ColumnProfile, + values: Cell[], + trainIdx: number[], +): (row: number) => string { + if (profile.type !== 'numeric') { + return (row) => (isMissing(values[row]) ? '∅' : (values[row] as string).trim()); + } + const parsed: number[] = []; + for (const row of trainIdx) { + if (isMissing(values[row])) continue; + const n = parseNumber((values[row] as string).trim()); + if (n !== null) parsed.push(n); + } + parsed.sort((a, b) => a - b); + const cuts: number[] = []; + for (let b = 1; b < NUMERIC_BINS; b++) { + const at = Math.min(parsed.length - 1, Math.floor((b / NUMERIC_BINS) * parsed.length)); + if (parsed.length > 0) cuts.push(parsed[at]); + } + return (row) => { + if (isMissing(values[row])) return '∅'; + const n = parseNumber((values[row] as string).trim()); + if (n === null) return '∅'; + let lo = 0; + let hi = cuts.length; + while (lo < hi) { + const mid = (lo + hi) >> 1; + if (n <= cuts[mid]) hi = mid; + else lo = mid + 1; + } + return `b${lo}`; + }; +} + +function stumpAccuracy( + trainIdx: number[], + evalIdx: number[], + encode: (row: number) => number, + keyOf: (row: number) => string, +): number | null { + const perKey = new Map>(); + const global = new Map(); + for (const row of trainIdx) { + const key = keyOf(row); + const label = encode(row); + const bucket = perKey.get(key) ?? new Map(); + bucket.set(label, (bucket.get(label) ?? 0) + 1); + perKey.set(key, bucket); + global.set(label, (global.get(label) ?? 0) + 1); + } + const majority = (counts: Map): number => { + let best = -1; + let bestCount = -1; + for (const [label, count] of counts) { + if (count > bestCount || (count === bestCount && label < best)) { + best = label; + bestCount = count; + } + } + return best; + }; + const fallback = majority(global); + let hits = 0; + for (const row of evalIdx) { + const bucket = perKey.get(keyOf(row)); + const predicted = bucket ? majority(bucket) : fallback; + if (predicted === encode(row)) hits += 1; + } + return hits / evalIdx.length; +} + +function stumpR2( + trainIdx: number[], + evalIdx: number[], + encode: (row: number) => number, + keyOf: (row: number) => string, +): number | null { + const sums = new Map(); + let globalSum = 0; + for (const row of trainIdx) { + const key = keyOf(row); + const y = encode(row); + const entry = sums.get(key) ?? { sum: 0, count: 0 }; + entry.sum += y; + entry.count += 1; + sums.set(key, entry); + globalSum += y; + } + const fallback = globalSum / trainIdx.length; + + let evalMean = 0; + for (const row of evalIdx) evalMean += encode(row); + evalMean /= evalIdx.length; + + let sse = 0; + let sst = 0; + for (const row of evalIdx) { + const y = encode(row); + const entry = sums.get(keyOf(row)); + const predicted = entry ? entry.sum / entry.count : fallback; + sse += (y - predicted) ** 2; + sst += (y - evalMean) ** 2; + } + if (sst === 0) return null; + return 1 - sse / sst; +} diff --git a/src/features/ml/train/pipeline.text.test.ts b/src/features/ml/train/pipeline.text.test.ts index 6b92a7a..a518c98 100644 --- a/src/features/ml/train/pipeline.text.test.ts +++ b/src/features/ml/train/pipeline.text.test.ts @@ -146,22 +146,41 @@ describe('wordEffects', () => { const termIndex = spec.terms.indexOf('qualite'); expect(termIndex).toBeGreaterThanOrEqual(0); - const effects = wordEffects(probeModel(termIndex) as never, pipeline, testX, true, 2); - const driver = effects.find((entry) => entry.term === 'qualite'); + const { words, refusal } = wordEffects( + probeModel(termIndex) as never, + pipeline, + testX, + true, + 2, + ); + expect(refusal).toBeUndefined(); + const driver = words.find((entry) => entry.term === 'qualite'); expect(driver).toBeDefined(); // Erasing the word flips the answer down: keeping it pushes up. expect(driver!.effect).toBeCloseTo(0.8, 6); - expect(effects[0].term).toBe('qualite'); // biggest magnitude first + expect(words[0].term).toBe('qualite'); // biggest magnitude first + }); + + it('refuses a saturated model by name instead of reporting "no word matters" (V35)', () => { + // A model that answers only 0 or 1 cannot be moved by erasing a word: + // every occlusion measures exactly zero. Silence would be a lie. + const saturated = { + predict: () => testX.map(() => 1), + predictProba: () => testX.map(() => [0, 1] as number[]), + }; + const { words, refusal } = wordEffects(saturated as never, pipeline, testX, true, 2); + expect(refusal).toBe('saturated'); + expect(words).toEqual([]); }); it('refuses multiclass rather than faking a direction', () => { - const effects = wordEffects(probeModel(0) as never, pipeline, testX, true, 3); - expect(effects).toEqual([]); + const { words } = wordEffects(probeModel(0) as never, pipeline, testX, true, 3); + expect(words).toEqual([]); }); it('returns nothing when the run has no text column', () => { const numericOnly = fitPipeline(columns, profiles, ['amount'], [0, 1, 2, 3]); const rows = [0, 1, 2, 3].map((i) => numericOnly.transformRow({ amount: String(10 + i) })); - expect(wordEffects(probeModel(0) as never, numericOnly, rows, true, 2)).toEqual([]); + expect(wordEffects(probeModel(0) as never, numericOnly, rows, true, 2)).toEqual({ words: [] }); }); }); diff --git a/src/features/ml/train/ranking.ts b/src/features/ml/train/ranking.ts new file mode 100644 index 0000000..a115c03 --- /dev/null +++ b/src/features/ml/train/ranking.ts @@ -0,0 +1,50 @@ +/** + * V35: one place decides how models are ranked and who is crowned. + * + * Before V35 every surface sorted by `primary` — the metric computed on the + * test set — and crowned `sorted[0]`. Taking the maximum of nine draws on a + * few hundred test rows biases the headline figure upward. When a run carries + * validation scores, ranking and crowning happen on THOSE, and the test score + * is what gets reported for the champion, gap included. Runs stored before + * V35 have no validation scores and keep their historical ranking. + */ +import type { TaskType } from '@/features/ml/data/types'; +import type { ModelResult } from '@/features/ml/train/types'; + +/** The value a model is ranked on: validation when present, test otherwise. */ +export function rankingValue(result: ModelResult): number { + return result.valPrimary ?? result.primary; +} + +/** Successful results, best first (accuracy: higher wins; RMSE: lower wins). */ +export function sortResults(results: ModelResult[], taskType: TaskType): ModelResult[] { + const isClassification = taskType !== 'regression'; + return results + .filter((r) => r.ok) + .sort((a, b) => + isClassification ? rankingValue(b) - rankingValue(a) : rankingValue(a) - rankingValue(b), + ); +} + +export function bestResult(results: ModelResult[], taskType: TaskType): ModelResult | null { + return sortResults(results, taskType)[0] ?? null; +} + +/** + * The champion's selection-vs-test gap — the most useful lesson the lab can + * teach: the score a model was chosen on is always a little optimistic. + * Null when the run carries no validation scores. + */ +export function championGap( + results: ModelResult[], + taskType: TaskType, +): { model: ModelResult; val: number; test: number; gap: number } | null { + const best = bestResult(results, taskType); + if (!best || best.valPrimary === undefined) return null; + return { + model: best, + val: best.valPrimary, + test: best.primary, + gap: best.primary - best.valPrimary, + }; +} diff --git a/src/features/ml/train/robust.ts b/src/features/ml/train/robust.ts new file mode 100644 index 0000000..8c66a40 --- /dev/null +++ b/src/features/ml/train/robust.ts @@ -0,0 +1,147 @@ +/** + * V35: the robust leaderboard — 5×2 repeated cross-validation. + * + * A single test split of a few hundred rows carries roughly ±3 points of + * standard deviation on accuracy; ranking two models one point apart on one + * draw is meaningless. On demand (like tuning), every family is retrained on + * 5 seeded repetitions × 2 halves of the train+validation rows — ten fits per + * family, ten scores, a mean and a spread. The TEST SET IS NEVER TOUCHED: + * this is a statement about the ranking, not a second bite at the test. + */ +import { MODEL_TRAIN_CAPS, modelZoo } from '@/features/ml/train/models'; +import { fitPipeline, splitIndices } from '@/features/ml/train/pipeline'; +import { nestedSampleOrder } from '@/features/ml/train/random'; +import { prepareData, scoreModel, yieldToQueue } from '@/features/ml/train/trainer'; +import type { Cell, ColumnProfile } from '@/features/ml/data/types'; +import type { ModelKey, TrainConfig } from '@/features/ml/train/types'; + +export const ROBUST_REPS = 5; + +export interface RobustEntry { + model: ModelKey; + /** Mean of the primary metric across the 2×reps held-out halves. */ + mean: number; + /** Sample standard deviation across the same folds. */ + sd: number; + scores: number[]; +} + +export interface RobustRankResult { + /** Best mean first (accuracy: higher wins; RMSE: lower wins). */ + entries: RobustEntry[]; + reps: number; + /** Rows the folds were drawn from (train + validation; never test). */ + rows: number; + isClassification: boolean; + /** + * Fold-paired comparison of the two best means: in how many of the 2×reps + * folds the leader actually beat the runner-up. 10/10 is a stable ranking; + * 6/10 means the order between them is inside the noise. + */ + topPair: { leader: ModelKey; runnerUp: ModelKey; leaderWins: number; folds: number } | null; +} + +export interface RobustCallbacks { + onProgress(done: number, total: number): void; + isCancelled(): boolean; +} + +export async function robustRank( + columns: Map, + profiles: ColumnProfile[], + config: TrainConfig, + callbacks: RobustCallbacks, + reps = ROBUST_REPS, +): Promise { + const prepared = prepareData(columns, profiles, config); + const { isClassification, classes, featureColumns, encode } = prepared; + + // The pool is train + validation — the rows selection is allowed to see. + const pool = [...prepared.train, ...prepared.validation].sort((a, b) => a - b); + const targetValues = columns.get(config.target)!; + const poolLabels = isClassification ? pool.map((i) => (targetValues[i] as string).trim()) : null; + + const zoo = modelZoo(isClassification ? 'classification' : 'regression'); + const scores = new Map(zoo.map((def) => [def.key, []])); + const totalFits = reps * 2 * zoo.length; + let done = 0; + + const context = { + task: isClassification ? ('classification' as const) : ('regression' as const), + classCount: classes.length, + seed: config.seed, + }; + + for (let rep = 0; rep < reps; rep++) { + // Each repetition is one seeded stratified half/half split of the pool. + const halves = splitIndices(pool, poolLabels, 0.5, config.seed + 101 + rep); + const pairs: [number[], number[]][] = [ + [halves.train, halves.test], + [halves.test, halves.train], + ]; + for (const [fitIdx, evalIdx] of pairs) { + // The strict reading of cross-validation: the pipeline (imputation, + // encoding, IDF, scaling) is refitted inside every fold. + const pipeline = fitPipeline(columns, profiles, featureColumns, fitIdx); + const fullX = pipeline.transform(fitIdx); + const fullY = fitIdx.map(encode); + const evalX = pipeline.transform(evalIdx); + const evalY = evalIdx.map(encode); + const foldLabels = poolLabels ? fitIdx.map((i) => (targetValues[i] as string).trim()) : null; + let sampleOrder: number[] | null = null; + + for (const def of zoo) { + if (callbacks.isCancelled()) return null; + // V25's announced caps hold here too — ten uncapped forest fits on + // 80k rows would take minutes for a number the cap changes little. + const cap = MODEL_TRAIN_CAPS[def.key]; + let fitX = fullX; + let fitY = fullY; + if (cap !== undefined && fitIdx.length > cap) { + sampleOrder ??= nestedSampleOrder(fitIdx.length, foldLabels, config.seed); + const keep = sampleOrder.slice(0, cap).sort((a, b) => a - b); + fitX = keep.map((position) => fullX[position]); + fitY = keep.map((position) => fullY[position]); + } + try { + const model = def.train(fitX, fitY, context); + const { primary } = scoreModel(model, evalX, evalY, isClassification, classes.length); + scores.get(def.key)!.push(primary); + } catch { + // A family that fails a fold simply reports fewer folds — shown as-is. + } + done += 1; + callbacks.onProgress(done, totalFits); + await yieldToQueue(); + } + } + } + + const entries: RobustEntry[] = [...scores.entries()] + .filter(([, s]) => s.length > 0) + .map(([model, s]) => { + const mean = s.reduce((a, b) => a + b, 0) / s.length; + const variance = + s.length > 1 ? s.reduce((a, b) => a + (b - mean) ** 2, 0) / (s.length - 1) : 0; + return { model, mean, sd: Math.sqrt(variance), scores: s }; + }) + .sort((a, b) => (isClassification ? b.mean - a.mean : a.mean - b.mean)); + + let topPair: RobustRankResult['topPair'] = null; + if (entries.length >= 2 && entries[0].scores.length === entries[1].scores.length) { + const [leader, runnerUp] = entries; + let leaderWins = 0; + for (let f = 0; f < leader.scores.length; f++) { + const delta = leader.scores[f] - runnerUp.scores[f]; + if (isClassification ? delta > 0 : delta < 0) leaderWins += 1; + } + topPair = { + leader: leader.model, + runnerUp: runnerUp.model, + leaderWins, + folds: leader.scores.length, + }; + } + + return { entries, reps, rows: pool.length, isClassification, topPair }; +} diff --git a/src/features/ml/train/sampling.test.ts b/src/features/ml/train/sampling.test.ts index 0fc8bbe..d314f1d 100644 --- a/src/features/ml/train/sampling.test.ts +++ b/src/features/ml/train/sampling.test.ts @@ -12,6 +12,9 @@ import type { ModelResult } from '@/features/ml/train/types'; import type { Cell, ColumnProfile } from '@/features/ml/data/types'; // V25: sampling is ANNOUNCED, seeded and nested — these tests freeze all three. +// V35 added a third split: the announced cap still counts USABLE rows, but they +// are now shared between train, validation and test, and the per-family caps +// bite on a train split that is 20% smaller. The sizes below account for that. describe('nestedSampleOrder', () => { it('is a deterministic permutation of 0..count-1', () => { @@ -58,16 +61,21 @@ describe('prepareData — announced global sample (V25)', () => { const config = { target: 'label', features: ['x1', 'x2'], seed: 42, testRatio: 0.2 }; const prepared = prepareData(columns, profiles, config); expect(prepared.sampledFrom).toBe(120_000); - expect(prepared.train.length + prepared.test.length).toBe(GLOBAL_SAMPLE_CAP); + expect(prepared.train.length + prepared.validation.length + prepared.test.length).toBe( + GLOBAL_SAMPLE_CAP, + ); // The class balance survives the sample (quantile stratification). const labels = columns.get('label')!; const yesTotal = labels.filter((v) => v === 'yes').length; const expected = Math.round((yesTotal * GLOBAL_SAMPLE_CAP) / 120_000); - const yes = [...prepared.train, ...prepared.test].filter((i) => labels[i] === 'yes').length; + const yes = [...prepared.train, ...prepared.validation, ...prepared.test].filter( + (i) => labels[i] === 'yes', + ).length; expect(Math.abs(yes - expected)).toBeLessThanOrEqual(5); // Deterministic: the same seed picks the same rows. const again = prepareData(columns, profiles, config); expect(again.train).toEqual(prepared.train); + expect(again.validation).toEqual(prepared.validation); expect(again.test).toEqual(prepared.test); }); @@ -80,7 +88,7 @@ describe('prepareData — announced global sample (V25)', () => { testRatio: 0.2, }); expect(prepared.sampledFrom).toBeUndefined(); - expect(prepared.train.length + prepared.test.length).toBe(5_000); + expect(prepared.train.length + prepared.validation.length + prepared.test.length).toBe(5_000); }); }); @@ -108,8 +116,9 @@ async function trainOn( describe('runTraining — announced per-family caps (V25)', () => { it('records the exact trainedRows for capped and uncapped families alike', async () => { - // 2 600 rows -> 2 080 train: tree (2 000) and forest (1 000) engage, the rest do not. - const { outcome, results } = await trainOn(2_600); + // 3 300 rows -> 2 640 after the test split -> 2 112 train once validation is + // carved out: tree (2 000) and forest (1 000) engage, the rest do not. + const { outcome, results } = await trainOn(3_300); const trainRows = outcome.summary.trainRows; expect(trainRows).toBeGreaterThan(2_000); // tree's cap must actually engage expect(outcome.summary.sampledFrom).toBeUndefined(); @@ -119,14 +128,15 @@ describe('runTraining — announced per-family caps (V25)', () => { expect(results.get(key)!.trainedRows, key).toBe(trainRows); } // Deterministic: the same seed trains the capped families on the same rows. - const repeat = await trainOn(2_600); + const repeat = await trainOn(3_300); expect(repeat.results.get('forest')!.metrics).toEqual(results.get('forest')!.metrics); expect(repeat.results.get('tree')!.metrics).toEqual(results.get('tree')!.metrics); }, 60_000); it('caps k-NN at 5 000 announced rows — the old silent subsample is gone', async () => { - // 7 000 rows -> 5 600 train: knn (5 000) engages on top of tree and forest. - const { outcome, results } = await trainOn(7_000); + // 8 000 rows -> 6 400 after the test split -> 5 120 train once validation is + // carved out: knn (5 000) engages on top of tree and forest. + const { outcome, results } = await trainOn(8_000); expect(results.get('knn')!.trainedRows).toBe(MODEL_TRAIN_CAPS.knn); expect(results.get('gbdt')!.trainedRows).toBe(outcome.summary.trainRows); // Capped families still beat the baseline on this separable dataset — diff --git a/src/features/ml/train/trainer.test.ts b/src/features/ml/train/trainer.test.ts index b2fb622..20c7f02 100644 --- a/src/features/ml/train/trainer.test.ts +++ b/src/features/ml/train/trainer.test.ts @@ -49,7 +49,8 @@ describe('runTraining — classification', () => { expect(best).toBeGreaterThan(0.9); // the target is separable on f1 expect(summary.taskType).toBe('binary'); - expect(summary.trainRows + summary.testRows).toBe(N); + // V35: three splits now share the usable rows (train / validation / test). + expect(summary.trainRows + (summary.validationRows ?? 0) + summary.testRows).toBe(N); // V24: free text is a feature now — it joins the pipeline as a TF-IDF block. expect(summary.featureColumns).toEqual(['f1', 'f2', 'note']); expect(summary.skippedColumns).toEqual([]); diff --git a/src/features/ml/train/trainer.ts b/src/features/ml/train/trainer.ts index c4b0862..5bf3c80 100644 --- a/src/features/ml/train/trainer.ts +++ b/src/features/ml/train/trainer.ts @@ -3,7 +3,9 @@ import { detectTask } from '@/features/ml/data/suggest'; import { accuracy, logLoss, macroPrf, mae, r2, rmse, rocAuc } from '@/features/ml/train/metrics'; import { MODEL_TRAIN_CAPS, modelZoo, type TrainedModel } from '@/features/ml/train/models'; import { fitPipeline, splitIndices, usableRows } from '@/features/ml/train/pipeline'; -import { nestedSampleOrder } from '@/features/ml/train/random'; +import { mulberry32, nestedSampleOrder, shuffleInPlace } from '@/features/ml/train/random'; +import { leakScan } from '@/features/ml/train/leakage'; +import { parseDate } from '@/features/ml/timeseries/series'; import type { Cell, ColumnProfile } from '@/features/ml/data/types'; import type { MetricMap, @@ -44,6 +46,20 @@ const LATENCY_SAMPLE = 200; * (sampledFrom), on the leaderboard and in the report. Never silent. */ export const GLOBAL_SAMPLE_CAP = 100_000; +/** + * V35: fraction of the TRAIN split held out for model selection. The test + * split is carved first, exactly as before V35 — every panel that reads the + * test set (segments, thresholds, uncertainty, batch compare) sees the same + * rows it always did. Selection then happens on validation, so the crowned + * number is no longer the maximum of nine draws on the reporting set. + */ +export const VALIDATION_RATIO = 0.2; +/** + * V35: below this many usable rows a third split starves training and the + * validation scores would be noise — so the lab refuses the third split by + * name and ranks on test, as before V35. + */ +export const MIN_ROWS_FOR_VALIDATION = 60; // 'text' joined the list in V24: free-text columns now enter the pipeline as // TF-IDF blocks instead of being skipped. Dates and ids stay out. const TRAINABLE_TYPES = new Set(['numeric', 'categorical', 'boolean', 'text']); @@ -61,11 +77,19 @@ export interface PreparedData { featureColumns: string[]; skippedColumns: string[]; train: number[]; + /** + * V35: rows held out for model selection — carved from the train side, so + * `test` is identical to what the same config produced before V35. Empty + * when the dataset is too small for a third split (refused by name). + */ + validation: number[]; test: number[]; /** Stratification labels aligned with `train` (null for regression). */ trainLabels: (string | null)[] | null; /** Usable rows before the announced global sample, when it engaged (V25). */ sampledFrom?: number; + /** V35: the announced non-random split that was applied, if any. */ + splitInfo?: { mode: 'chronological' | 'group'; column: string; dropped?: number }; encode(i: number): number; } @@ -126,7 +150,34 @@ export function prepareData( } } - const { train, test } = splitIndices(rows, stratifyLabels, config.testRatio, config.seed); + // V35: assign rows to splits. Random (stratified) is the default; a + // chronological or group split is applied only when the config names one — + // and the summary announces it, like every other decision here. + let train: number[]; + let test: number[]; + let validation: number[] = []; + let splitInfo: PreparedData['splitInfo']; + const wantValidation = rows.length >= MIN_ROWS_FOR_VALIDATION; + + if (config.split) { + const assigned = splitNonRandom(rows, columns, config, wantValidation); + train = assigned.train; + validation = assigned.validation; + test = assigned.test; + splitInfo = assigned.info; + } else { + ({ train, test } = splitIndices(rows, stratifyLabels, config.testRatio, config.seed)); + if (wantValidation) { + // Carved from the TRAIN side with a derived seed: the test indices stay + // byte-identical to what this config produced before V35. + const labels = stratifyLabels + ? train.map((i) => (isMissing(targetValues[i]) ? null : (targetValues[i] as string).trim())) + : null; + const second = splitIndices(train, labels, VALIDATION_RATIO, config.seed + 1); + train = second.train; + validation = second.test; + } + } const trainLabels = isClassification ? train.map((i) => (isMissing(targetValues[i]) ? null : (targetValues[i] as string).trim())) @@ -139,14 +190,111 @@ export function prepareData( featureColumns, skippedColumns, train, + validation, test, trainLabels, ...(sampledFrom !== undefined && { sampledFrom }), + ...(splitInfo !== undefined && { splitInfo }), encode: (i: number): number => isClassification ? labelOf(i) : (parseNumber((targetValues[i] as string).trim()) as number), }; } +/** + * V35: the two announced non-random splits. + * + * Chronological — rows ordered by the named date column; the oldest block + * trains, the middle one validates, the newest one tests. A random split on + * dated data puts the future in training: the model looks excellent and + * collapses in production. Rows without a parseable date cannot be placed in + * time and are dropped, counted, and announced. + * + * Group — every row sharing the named column's value lands on one side only: + * the same customer in both train and test is the same leak. Groups are + * shuffled with the run's seed and dealt to test, then validation, until each + * reaches its share. Rows with a missing group value are their own group. + */ +function splitNonRandom( + rows: number[], + columns: Map, + config: TrainConfig, + wantValidation: boolean, +): { + train: number[]; + validation: number[]; + test: number[]; + info: NonNullable; +} { + const split = config.split!; + const values = columns.get(split.column); + if (!values) throw new Error('split-column-not-found'); + + if (split.mode === 'chronological') { + const dated: { row: number; at: number }[] = []; + let dropped = 0; + for (const row of rows) { + const raw = values[row]; + const at = isMissing(raw) ? null : parseDate((raw as string).trim()); + if (at === null) dropped += 1; + else dated.push({ row, at }); + } + if (dated.length < 10) throw new Error('split-column-not-dated'); + // Stable order: ties fall back to row index so the split is deterministic. + dated.sort((a, b) => a.at - b.at || a.row - b.row); + const ordered = dated.map((d) => d.row); + const testCount = Math.max(1, Math.round(ordered.length * config.testRatio)); + const rest = ordered.length - testCount; + const valCount = wantValidation ? Math.max(1, Math.round(rest * VALIDATION_RATIO)) : 0; + const train = ordered.slice(0, rest - valCount); + const validation = ordered.slice(rest - valCount, rest); + const test = ordered.slice(rest); + return { + train: [...train].sort((a, b) => a - b), + validation: [...validation].sort((a, b) => a - b), + test: [...test].sort((a, b) => a - b), + info: { mode: 'chronological', column: split.column, ...(dropped > 0 && { dropped }) }, + }; + } + + // Group mode. Missing values become singleton groups — a row that belongs + // to nobody cannot leak across the boundary. + const groups = new Map(); + let singleton = 0; + for (const row of rows) { + const raw = values[row]; + const key = isMissing(raw) ? `\u2205#${singleton++}` : (raw as string).trim(); + const bucket = groups.get(key); + if (bucket) bucket.push(row); + else groups.set(key, [row]); + } + if (groups.size < 3) throw new Error('split-column-not-groupable'); + const keys = [...groups.keys()].sort(); + shuffleInPlace(keys, mulberry32(config.seed)); + + const testTarget = Math.max(1, Math.round(rows.length * config.testRatio)); + const valTarget = wantValidation + ? Math.max(1, Math.round((rows.length - testTarget) * VALIDATION_RATIO)) + : 0; + const train: number[] = []; + const validation: number[] = []; + const test: number[] = []; + for (const key of keys) { + const bucket = groups.get(key)!; + if (test.length < testTarget) { + for (const row of bucket) test.push(row); + } else if (validation.length < valTarget) { + for (const row of bucket) validation.push(row); + } else { + for (const row of bucket) train.push(row); + } + } + if (train.length === 0 || test.length === 0) throw new Error('split-column-not-groupable'); + train.sort((a, b) => a - b); + validation.sort((a, b) => a - b); + test.sort((a, b) => a - b); + return { train, validation, test, info: { mode: 'group', column: split.column } }; +} + /** The run's metric block for one model on one evaluation set. */ export function scoreModel( model: TrainedModel, @@ -211,14 +359,38 @@ export async function runTraining( ): Promise { const startedAt = performance.now(); const prepared = prepareData(columns, profiles, config); - const { task, isClassification, classes, featureColumns, skippedColumns, train, test, encode } = - prepared; + const { + task, + isClassification, + classes, + featureColumns, + skippedColumns, + train, + validation, + test, + encode, + } = prepared; const pipeline = fitPipeline(columns, profiles, featureColumns, train); const trainX = pipeline.transform(train); const testX = pipeline.transform(test); const trainY = train.map(encode); const testY = test.map(encode); + const valX = validation.length > 0 ? pipeline.transform(validation) : null; + const valY = validation.length > 0 ? validation.map(encode) : null; + + // V35: the predictive leak scan — a lone column reading the target at 99% + // is a warning, not a victory. Fitted on train, scored on validation, so + // the detector cannot leak either; refused (empty) when validation is. + const leakWarnings = leakScan( + columns, + profiles, + featureColumns, + train, + validation, + encode, + isClassification, + ); const zoo = modelZoo(isClassification ? 'classification' : 'regression'); const models = new Map(); @@ -266,6 +438,13 @@ export async function runTraining( classes.length, ); + // V35: the same metric block on the selection split — the leaderboard + // ranks on these so the crowned number is not the max of nine test draws. + let validationScore: { metrics: MetricMap; primary: number } | null = null; + if (valX !== null && valY !== null) { + validationScore = scoreModel(model, valX, valY, isClassification, classes.length); + } + const latency = measureLatency(model, testX); callbacks.onModelResult({ key: def.key, @@ -276,6 +455,10 @@ export async function runTraining( inferP50Ms: latency.p50, inferP95Ms: latency.p95, trainedRows: fitX.length, + ...(validationScore !== null && { + valMetrics: validationScore.metrics, + valPrimary: validationScore.primary, + }), }); } catch (error) { callbacks.onModelResult({ @@ -304,6 +487,9 @@ export async function runTraining( skippedColumns, totalMs: performance.now() - startedAt, ...(prepared.sampledFrom !== undefined && { sampledFrom: prepared.sampledFrom }), + ...(validation.length > 0 && { validationRows: validation.length }), + ...(prepared.splitInfo !== undefined && { split: prepared.splitInfo }), + ...(leakWarnings.length > 0 && { leakWarnings }), }, artifacts: { models, diff --git a/src/features/ml/train/types.ts b/src/features/ml/train/types.ts index c973082..a85ede5 100644 --- a/src/features/ml/train/types.ts +++ b/src/features/ml/train/types.ts @@ -9,6 +9,20 @@ export interface TrainConfig { features: string[]; seed: number; testRatio: number; + /** + * V35: how rows are assigned to the splits. Absent = seeded random + * (stratified on classification). 'chronological' orders rows by the named + * date column — oldest train, newest test — because a random split on dated + * data puts the future in training. 'group' keeps every row sharing the + * named column's value on the same side — the same customer in both train + * and test is the same leak. + */ + split?: SplitChoice; +} + +export interface SplitChoice { + mode: 'chronological' | 'group'; + column: string; } /** Metric values per model; keys depend on the task type. */ @@ -36,6 +50,16 @@ export interface ModelResult { * Absent on runs stored before V25 and on failed models. */ trainedRows?: number; + /** + * V35: the same metrics on the validation split. When present, the + * leaderboard ranks and crowns on THESE — taking the maximum of nine test + * scores biased the headline number upward — and reports the champion's + * test score next to it, gap included. Absent on runs stored before V35 + * and on datasets too small for a third split (refused by name). + */ + valMetrics?: MetricMap; + /** Primary validation metric (accuracy or RMSE); see valMetrics. */ + valPrimary?: number; } export interface InsightsPayload { @@ -59,6 +83,14 @@ export interface InsightsPayload { * there is no single axis to project the shift onto. */ words?: { column: string; term: string; effect: number; rows: number }[]; + /** + * V35: the word-effect method could not measure anything and says so. + * 'saturated' = the model answers with only one or two distinct + * probabilities (Gaussian Naive Bayes on many TF-IDF features does), so + * every occlusion shifts it by exactly zero. Silence would read as + * "no word matters", which is a different — and false — statement. + */ + wordsRefused?: 'saturated'; /** * Partial dependence of the prediction on the top numeric columns * (binary classification: mean probability of the positive class; @@ -94,4 +126,24 @@ export interface TrainSummary { * Absent when every usable row was used — sampling is never silent. */ sampledFrom?: number; + /** + * V35: rows held out for model selection. Absent when the dataset was too + * small for a third split (refused by name — the leaderboard then ranks on + * test, as before V35, and says nothing it cannot back). + */ + validationRows?: number; + /** V35: non-random split, when one was chosen — always announced. */ + split?: { + mode: 'chronological' | 'group'; + column: string; + /** Rows excluded because the split column had no usable value there. */ + dropped?: number; + }; + /** + * V35: single columns that predict the target (almost) alone, measured by a + * one-column stump fitted on train and scored on the held-out selection + * split. A lone column at 99% is nearly always target leakage — shown as a + * warning, never as a victory. + */ + leakWarnings?: { column: string; score: number }[]; } diff --git a/src/features/ml/train/v35.test.ts b/src/features/ml/train/v35.test.ts new file mode 100644 index 0000000..5e14b3c --- /dev/null +++ b/src/features/ml/train/v35.test.ts @@ -0,0 +1,320 @@ +import { describe, expect, it } from 'vitest'; +import { profileColumn } from '@/features/ml/data/profile'; +import { leakScan } from '@/features/ml/train/leakage'; +import { splitIndices } from '@/features/ml/train/pipeline'; +import { bestResult, championGap, rankingValue, sortResults } from '@/features/ml/train/ranking'; +import { robustRank } from '@/features/ml/train/robust'; +import { MIN_ROWS_FOR_VALIDATION, prepareData, runTraining } from '@/features/ml/train/trainer'; +import type { ModelResult, TrainConfig } from '@/features/ml/train/types'; +import type { Cell, ColumnProfile } from '@/features/ml/data/types'; + +// V35: the number stops flattering itself. These tests freeze the three-way +// split (with its compatibility guarantee: the TEST indices are the same ones +// the two-way split produced), the announced chronological and group splits, +// the predictive leak scan, and the 5×2 robust ranking. + +function setup(data: Record): { + columns: Map; + profiles: ColumnProfile[]; +} { + const columns = new Map(Object.entries(data)); + const profiles = Object.entries(data).map(([name, values]) => profileColumn(name, values)); + return { columns, profiles }; +} + +/** n rows, learnable rule, ~35% positive class. */ +function classification(n: number): Record { + const x1: Cell[] = []; + const x2: Cell[] = []; + const label: Cell[] = []; + for (let i = 0; i < n; i++) { + x1.push(String(i % 23)); + x2.push(String((i * 7) % 19)); + label.push((i % 23) + ((i * 7) % 19) > 26 ? 'yes' : 'no'); + } + return { x1, x2, label }; +} + +const config = (over: Partial = {}): TrainConfig => ({ + target: 'label', + features: ['x1', 'x2'], + seed: 42, + testRatio: 0.2, + ...over, +}); + +describe('prepareData — the third split (V35)', () => { + it('keeps the test indices byte-identical to the pre-V35 two-way split', () => { + const data = classification(200); + const { columns, profiles } = setup(data); + const prepared = prepareData(columns, profiles, config()); + + const rows = Array.from({ length: 200 }, (_, i) => i); + const labels = rows.map((i) => (data.label[i] as string).trim()); + const legacy = splitIndices(rows, labels, 0.2, 42); + expect(prepared.test).toEqual(legacy.test); + }); + + it('partitions rows: train, validation and test are disjoint and complete', () => { + const { columns, profiles } = setup(classification(200)); + const prepared = prepareData(columns, profiles, config()); + expect(prepared.validation.length).toBeGreaterThan(0); + const all = [...prepared.train, ...prepared.validation, ...prepared.test].sort((a, b) => a - b); + expect(all).toEqual(Array.from({ length: 200 }, (_, i) => i)); + }); + + it('refuses the third split by name below the minimum row count', () => { + const { columns, profiles } = setup(classification(MIN_ROWS_FOR_VALIDATION - 10)); + const prepared = prepareData(columns, profiles, config()); + expect(prepared.validation).toEqual([]); + // Tiny runs keep the historical two-way behaviour. + expect(prepared.train.length + prepared.test.length).toBe(MIN_ROWS_FOR_VALIDATION - 10); + }); + + it('is deterministic: the same config reproduces the same three splits', () => { + const { columns, profiles } = setup(classification(150)); + const a = prepareData(columns, profiles, config()); + const b = prepareData(columns, profiles, config()); + expect(a.train).toEqual(b.train); + expect(a.validation).toEqual(b.validation); + expect(a.test).toEqual(b.test); + }); +}); + +describe('prepareData — announced chronological split (V35)', () => { + function dated(n: number): Record { + const base = classification(n); + const when: Cell[] = []; + for (let i = 0; i < n; i++) { + const day = new Date(Date.UTC(2024, 0, 1) + i * 86_400_000); + when.push(day.toISOString().slice(0, 10)); + } + return { ...base, when }; + } + + it('trains on the oldest rows, validates on the middle, tests on the newest', () => { + const { columns, profiles } = setup(dated(200)); + const prepared = prepareData( + columns, + profiles, + config({ split: { mode: 'chronological', column: 'when' } }), + ); + // Rows are indexed in date order, so index order IS time order here. + expect(Math.max(...prepared.train)).toBeLessThan(Math.min(...prepared.validation)); + expect(Math.max(...prepared.validation)).toBeLessThan(Math.min(...prepared.test)); + expect(prepared.splitInfo).toEqual({ mode: 'chronological', column: 'when' }); + }); + + it('drops rows without a parseable date — counted, never silent', () => { + const data = dated(100); + data.when[10] = 'not a date'; + data.when[20] = null; + const { columns, profiles } = setup(data); + const prepared = prepareData( + columns, + profiles, + config({ split: { mode: 'chronological', column: 'when' } }), + ); + expect(prepared.splitInfo?.dropped).toBe(2); + const all = [...prepared.train, ...prepared.validation, ...prepared.test]; + expect(all).toHaveLength(98); + expect(all).not.toContain(10); + expect(all).not.toContain(20); + }); +}); + +describe('prepareData — announced group split (V35)', () => { + it('never puts two rows of the same group on different sides', () => { + const base = classification(200); + const customer: Cell[] = []; + for (let i = 0; i < 200; i++) customer.push(`c${Math.floor(i / 5)}`); + const { columns, profiles } = setup({ ...base, customer }); + const prepared = prepareData( + columns, + profiles, + config({ split: { mode: 'group', column: 'customer' } }), + ); + const side = new Map(); + const check = (rows: number[], name: string) => { + for (const row of rows) { + const key = customer[row] as string; + const seen = side.get(key); + expect(seen === undefined || seen === name).toBe(true); + side.set(key, name); + } + }; + check(prepared.train, 'train'); + check(prepared.validation, 'validation'); + check(prepared.test, 'test'); + expect(prepared.test.length).toBeGreaterThan(0); + expect(prepared.splitInfo).toEqual({ mode: 'group', column: 'customer' }); + }); +}); + +describe('leakScan (V35)', () => { + it('flags a column that mirrors the target and spares an honest one', () => { + const data = classification(300); + // The leak: a perfect copy of the target under another name. + data.mirror = data.label.map((v) => (v === 'yes' ? 'oui' : 'non')); + const { columns, profiles } = setup(data); + const prepared = prepareData(columns, profiles, config({ features: ['x1', 'x2', 'mirror'] })); + const warnings = leakScan( + columns, + profiles, + ['x1', 'x2', 'mirror'], + prepared.train, + prepared.validation, + prepared.encode, + true, + ); + expect(warnings.map((w) => w.column)).toEqual(['mirror']); + expect(warnings[0].score).toBe(1); + }); + + it('flags a numeric column that reads a regression target through its bins', () => { + const n = 400; + const x: Cell[] = []; + const leak: Cell[] = []; + const y: Cell[] = []; + for (let i = 0; i < n; i++) { + const value = (i * 37) % 199; + x.push(String(i % 7)); + leak.push(String(value + 0.01 * ((i * 13) % 5))); + y.push(String(value)); + } + const { columns, profiles } = setup({ x, leak, y }); + const prepared = prepareData(columns, profiles, { + target: 'y', + features: ['x', 'leak'], + seed: 42, + testRatio: 0.2, + }); + const warnings = leakScan( + columns, + profiles, + ['x', 'leak'], + prepared.train, + prepared.validation, + prepared.encode, + false, + ); + expect(warnings.map((w) => w.column)).toEqual(['leak']); + expect(warnings[0].score).toBeGreaterThanOrEqual(0.99); + }); + + it('refuses to scan when the evaluation split is too small to mean anything', () => { + const data = classification(40); + data.mirror = [...data.label]; + const { columns, profiles } = setup(data); + const prepared = prepareData(columns, profiles, config({ features: ['x1', 'mirror'] })); + expect(prepared.validation).toEqual([]); + const warnings = leakScan( + columns, + profiles, + ['x1', 'mirror'], + prepared.train, + prepared.validation, + prepared.encode, + true, + ); + expect(warnings).toEqual([]); + }); +}); + +describe('ranking (V35)', () => { + const result = (over: Partial): ModelResult => ({ + key: 'tree', + ok: true, + metrics: {}, + primary: 0, + trainMs: 0, + inferP50Ms: 0, + inferP95Ms: 0, + ...over, + }); + + it('ranks on validation when present, on test otherwise', () => { + const a = result({ key: 'tree', primary: 0.9, valPrimary: 0.7 }); + const b = result({ key: 'forest', primary: 0.8, valPrimary: 0.85 }); + expect(rankingValue(a)).toBe(0.7); + expect(sortResults([a, b], 'binary')[0].key).toBe('forest'); + // Pre-V35 stored runs: no validation scores, historical order preserved. + const legacy = [result({ key: 'tree', primary: 0.9 }), result({ key: 'knn', primary: 0.8 })]; + expect(bestResult(legacy, 'binary')?.key).toBe('tree'); + }); + + it('reports the champion selection-vs-test gap only when validation exists', () => { + const a = result({ key: 'gbdt', primary: 0.78, valPrimary: 0.82 }); + const gap = championGap([a], 'binary'); + expect(gap?.val).toBe(0.82); + expect(gap?.test).toBe(0.78); + expect(gap?.gap).toBeCloseTo(-0.04, 12); + expect(championGap([result({ primary: 0.9 })], 'binary')).toBeNull(); + }); + + it('regression ranks low RMSE first on the validation value', () => { + const a = result({ key: 'linear', primary: 4, valPrimary: 5 }); + const b = result({ key: 'gbdt', primary: 6, valPrimary: 3 }); + expect(sortResults([a, b], 'regression')[0].key).toBe('gbdt'); + }); +}); + +describe('runTraining carries the V35 fields end to end', () => { + it('emits validation scores, announces the split and warns on a leak', async () => { + const data = classification(240); + data.mirror = [...data.label]; + const { columns, profiles } = setup(data); + const results: ModelResult[] = []; + const outcome = await runTraining( + columns, + profiles, + config({ features: ['x1', 'x2', 'mirror'] }), + { + onModelStart: () => {}, + onModelResult: (r) => results.push(r), + isCancelled: () => false, + }, + ); + expect(outcome).not.toBeNull(); + expect(outcome!.summary.validationRows).toBeGreaterThan(0); + expect(outcome!.summary.leakWarnings?.map((w) => w.column)).toEqual(['mirror']); + for (const r of results.filter((r) => r.ok)) { + expect(r.valPrimary).toBeGreaterThanOrEqual(0); + expect(r.valMetrics?.accuracy).toBe(r.valPrimary); + } + // The champion's headline pair exists and is internally consistent. + const gap = championGap(results, 'binary'); + expect(gap).not.toBeNull(); + expect(gap!.gap).toBeCloseTo(gap!.test - gap!.val, 12); + }, 30_000); +}); + +describe('robustRank — 5×2 CV (V35)', () => { + it('is deterministic, never touches the test rows, and reports the top pair', async () => { + const { columns, profiles } = setup(classification(200)); + const cfg = config(); + const run = () => + robustRank(columns, profiles, cfg, { onProgress: () => {}, isCancelled: () => false }, 2); + const a = await run(); + const b = await run(); + expect(a).not.toBeNull(); + expect(a!.entries.map((e) => e.model)).toEqual(b!.entries.map((e) => e.model)); + expect(a!.entries[0].scores).toEqual(b!.entries[0].scores); + expect(a!.entries[0].scores).toHaveLength(4); // 2 reps × 2 halves + expect(a!.entries[0].sd).toBeGreaterThanOrEqual(0); + // Pool = train + validation only: the 40 test rows stay out. + expect(a!.rows).toBe(160); + expect(a!.topPair).not.toBeNull(); + expect(a!.topPair!.folds).toBe(4); + }, 60_000); + + it('honours cancellation', async () => { + const { columns, profiles } = setup(classification(200)); + let calls = 0; + const outcome = await robustRank(columns, profiles, config(), { + onProgress: () => {}, + isCancelled: () => calls++ > 3, + }); + expect(outcome).toBeNull(); + }, 30_000); +}); diff --git a/src/features/ml/worker-protocol.ts b/src/features/ml/worker-protocol.ts index 51fb3f4..88901da 100644 --- a/src/features/ml/worker-protocol.ts +++ b/src/features/ml/worker-protocol.ts @@ -6,6 +6,7 @@ import type { BatchScore } from '@/features/ml/train/score'; import type { SegmentAnalysis } from '@/features/ml/train/segments'; import type { TunableKey, TuneOutcome } from '@/features/ml/train/search'; import type { LearningCurveOutcome } from '@/features/ml/train/learning-curve'; +import type { RobustRankResult } from '@/features/ml/train/robust'; import type { ThresholdAnalysis } from '@/features/ml/train/threshold-analysis'; import type { UncertaintyAnalysis } from '@/features/ml/train/uncertainty'; import type { ShapleyExplanation } from '@/features/ml/train/shapley'; @@ -36,6 +37,9 @@ export type WorkerRequest = // v26: learning curve — retrain on growing seeded prefixes of the train split. | { kind: 'learning-curve'; model: ModelKey; config: TrainConfig } | { kind: 'cancel-curve' } + // v35: robust ranking — 5×2 CV on train+validation; the test set stays out. + | { kind: 'robust-rank'; config: TrainConfig } + | { kind: 'cancel-robust' } | { kind: 'explore'; features: string[]; seed: number } | { kind: 'forecast'; dateColumn: string; valueColumn: string } | { kind: 'export-model'; model: ModelKey } @@ -69,6 +73,9 @@ export type WorkerResponse = // null = refused by name (baseline, or a ladder with a single rung). | { kind: 'curve-complete'; payload: LearningCurveOutcome | null } | { kind: 'curve-cancelled' } + | { kind: 'robust-progress'; done: number; total: number } + | { kind: 'robust-complete'; payload: RobustRankResult } + | { kind: 'robust-cancelled' } | { kind: 'explore-result'; payload: ExplorationPayload } | { kind: 'forecast-result'; payload: ForecastPayload } | { kind: 'model-json'; model: ModelKey; json: string | null } diff --git a/src/locales/en.json b/src/locales/en.json index 5d12aa7..dcc0b46 100644 --- a/src/locales/en.json +++ b/src/locales/en.json @@ -289,7 +289,19 @@ "skipped": "skipped (dates and identifiers are not features): {{columns}}", "trainedOn": "sample: {{rows}} rows", "trainedOnTitle": "Trained on an announced seeded sample of {{rows}} of the {{total}} training rows — never silently. The score comes from the same full test set as every other model.", - "sampledFrom": "announced sample: {{cap}} of {{from}} usable rows (seeded, stratified)" + "sampledFrom": "announced sample: {{cap}} of {{from}} usable rows (seeded, stratified)", + "accuracyVal": "Accuracy (val)", + "rmseVal": "RMSE (val)", + "test": "Test", + "testTitle": "The score on the reporting split — held out from selection. It is the number to quote.", + "championLine": "{{model}} was selected on validation at {{val}} and scores {{test}} on the untouched test set ({{gap}}).", + "championWhy": "Selecting the best of several models on a split makes that split optimistic — reporting on a third, never-selected split is what keeps the headline number honest.", + "runInfoVal": "seed {{seed}} · {{train}} train rows · {{val}} validation rows · {{test}} test rows · {{features}} features after encoding", + "splitChronological": "chronological split on {{column}} (oldest rows train, newest test)", + "splitGroup": "grouped split on {{column}} (no group on both sides)", + "splitDropped": "— {{count}} rows dropped, no usable date", + "leakWarning": "« {{column}} » alone predicts the target at {{score}}% on the validation split.", + "leakAdvice": "A single column reading the target that well is almost always leakage — information that would not exist at prediction time. Exclude it and retrain, or confirm it is legitimately available." }, "models": { "baseline": "Naive baseline", @@ -542,7 +554,9 @@ "wordsHintClassification": "Average shift of the predicted probability of “{{class}}” when the word is erased from the reviews containing it — right pushes up, left pushes down.", "wordsHintRegression": "Average shift of the prediction when the word is erased from the texts containing it — right pushes up, left pushes down.", "wordsNote": "One word at a time, on the held-out test rows. A review says the same thing several ways, so erasing one word rarely flips a prediction on its own.", - "wordsRowTitle": "“{{term}}” in {{column}} · {{rows}} test rows · {{effect}}" + "wordsRowTitle": "“{{term}}” in {{column}} · {{rows}} test rows · {{effect}}", + "wordsSaturated": "This model cannot answer the word question: its probabilities are saturated — it returns almost only 0 or 1 — so erasing a word shifts the answer by exactly zero.", + "wordsSaturatedAdvice": "That is a property of the model, not of the words. Pick another model in the leaderboard (logistic regression and gradient boosting give graded probabilities) to read the word effects." }, "export": { "model": "Model (JSON)", @@ -581,7 +595,25 @@ "note": "This link carries only metrics and charts — never the original data. It lives in the URL fragment, which browsers do not send to any server.", "invalid": "Invalid or incomplete share link." }, - "reportFooter": "Generated locally by LabML — no data ever left the browser." + "reportFooter": "Generated locally by LabML — no data ever left the browser.", + "splitMode": { + "label": "Split", + "random": "Random (seeded, stratified)", + "chronological": "Chronological — {{column}}", + "group": "By group — {{column}}" + }, + "robust": { + "title": "Is this ranking real?", + "hint": "A single split ranks on one draw of a few hundred rows, and two models a point apart can swap places on the next draw. This retrains every family on 5 repetitions × 2 halves of the training and validation rows — ten fits each — and reports a mean, a spread, and how often the leader actually beat the runner-up. The test set stays out of the folds: this re-ranks, it does not re-test.", + "run": "Rank on 5×2 cross-validation", + "preparing": "Preparing the folds…", + "progress": "Fit {{done}} / {{total}}…", + "mean": "Mean", + "sd": "Spread", + "verdictStable": "{{leader}} beat {{runnerUp}} in {{wins}} of {{folds}} folds — the order between them is stable.", + "verdictNoise": "{{leader}} beat {{runnerUp}} in only {{wins}} of {{folds}} folds — the order between them is inside the noise, so treat them as tied.", + "note": "{{folds}} folds ({{reps}} repetitions × 2 halves) over {{rows}} training and validation rows. The pipeline is refitted inside every fold; the test set is never touched." + } } }, "data": { diff --git a/src/locales/fr.json b/src/locales/fr.json index 62ddad9..b480587 100644 --- a/src/locales/fr.json +++ b/src/locales/fr.json @@ -289,7 +289,19 @@ "skipped": "écartées (dates et identifiants ne sont pas des variables) : {{columns}}", "trainedOn": "échantillon : {{rows}} lignes", "trainedOnTitle": "Entraîné sur un échantillon seedé annoncé de {{rows}} des {{total}} lignes d'entraînement — jamais en silence. Le score vient du même jeu de test complet que les autres modèles.", - "sampledFrom": "échantillon annoncé : {{cap}} lignes sur {{from}} utilisables (seedé, stratifié)" + "sampledFrom": "échantillon annoncé : {{cap}} lignes sur {{from}} utilisables (seedé, stratifié)", + "accuracyVal": "Accuracy (valid.)", + "rmseVal": "RMSE (valid.)", + "test": "Test", + "testTitle": "Le score sur le jeu de rapport — tenu à l'écart de la sélection. C'est le chiffre à citer.", + "championLine": "{{model}} a été choisi sur la validation à {{val}} et obtient {{test}} sur le jeu de test intact ({{gap}}).", + "championWhy": "Choisir le meilleur de plusieurs modèles sur un jeu rend ce jeu optimiste — rapporter sur un troisième jeu, jamais utilisé pour choisir, est ce qui garde le chiffre honnête.", + "runInfoVal": "seed {{seed}} · {{train}} lignes d'entraînement · {{val}} lignes de validation · {{test}} lignes de test · {{features}} variables après encodage", + "splitChronological": "découpe chronologique sur {{column}} (les plus anciennes entraînent, les plus récentes testent)", + "splitGroup": "découpe par groupe sur {{column}} (aucun groupe des deux côtés)", + "splitDropped": "— {{count}} lignes écartées, date inexploitable", + "leakWarning": "« {{column}} » prédit à elle seule la cible à {{score}} % sur le jeu de validation.", + "leakAdvice": "Une colonne seule qui lit aussi bien la cible est presque toujours une fuite — une information qui n'existerait pas au moment de prédire. Excluez-la et réentraînez, ou confirmez qu'elle est légitimement disponible." }, "models": { "baseline": "Baseline naïve", @@ -542,7 +554,9 @@ "wordsHintClassification": "Décalage moyen de la probabilité prédite de « {{class}} » quand le mot est effacé des avis qui le contiennent — à droite il pousse vers le haut, à gauche vers le bas.", "wordsHintRegression": "Décalage moyen de la prédiction quand le mot est effacé des textes qui le contiennent — à droite il pousse vers le haut, à gauche vers le bas.", "wordsNote": "Un mot à la fois, sur les lignes de test mises de côté. Un avis dit la même chose de plusieurs façons : effacer un seul mot renverse rarement une prédiction à lui seul.", - "wordsRowTitle": "« {{term}} » dans {{column}} · {{rows}} lignes de test · {{effect}}" + "wordsRowTitle": "« {{term}} » dans {{column}} · {{rows}} lignes de test · {{effect}}", + "wordsSaturated": "Ce modèle ne peut pas répondre à la question des mots : ses probabilités sont saturées — il ne renvoie presque que 0 ou 1 — donc effacer un mot déplace la réponse d'exactement zéro.", + "wordsSaturatedAdvice": "C'est une propriété du modèle, pas des mots. Choisissez un autre modèle dans le leaderboard (la régression logistique et le gradient boosting donnent des probabilités nuancées) pour lire les effets par mot." }, "export": { "model": "Modèle (JSON)", @@ -581,7 +595,25 @@ "note": "Ce lien ne transporte que les métriques et graphiques — jamais les données d'origine. Il vit dans le fragment d'URL, que les navigateurs n'envoient à aucun serveur.", "invalid": "Lien de partage invalide ou incomplet." }, - "reportFooter": "Généré localement par LabML — aucune donnée n'a quitté le navigateur." + "reportFooter": "Généré localement par LabML — aucune donnée n'a quitté le navigateur.", + "splitMode": { + "label": "Découpe", + "random": "Aléatoire (seedée, stratifiée)", + "chronological": "Chronologique — {{column}}", + "group": "Par groupe — {{column}}" + }, + "robust": { + "title": "Ce classement est-il réel ?", + "hint": "Une découpe unique classe sur un seul tirage de quelques centaines de lignes, et deux modèles séparés d'un point peuvent échanger leur place au tirage suivant. Ceci réentraîne chaque famille sur 5 répétitions × 2 moitiés des lignes d'entraînement et de validation — dix ajustements chacune — et rapporte une moyenne, une dispersion, et combien de fois le premier a réellement battu le second. Le jeu de test reste hors des plis : on reclasse, on ne re-teste pas.", + "run": "Classer en validation croisée 5×2", + "preparing": "Préparation des plis…", + "progress": "Ajustement {{done}} / {{total}}…", + "mean": "Moyenne", + "sd": "Dispersion", + "verdictStable": "{{leader}} a battu {{runnerUp}} sur {{wins}} plis sur {{folds}} — l'ordre entre eux est stable.", + "verdictNoise": "{{leader}} n'a battu {{runnerUp}} que sur {{wins}} plis sur {{folds}} — l'ordre entre eux est dans le bruit : traitez-les comme à égalité.", + "note": "{{folds}} plis ({{reps}} répétitions × 2 moitiés) sur {{rows}} lignes d'entraînement et de validation. Le pipeline est réajusté dans chaque pli ; le jeu de test n'est jamais touché." + } } }, "data": {