diff --git a/PLAN.md b/PLAN.md index 116480b..16b8b65 100644 --- a/PLAN.md +++ b/PLAN.md @@ -447,12 +447,19 @@ data budget asks. | V34 | **Explanations, how-to guides, and the pages that make the project readable as engineering.** The why-pages: why a baseline before anything else, why intervals instead of a single figure, why seed 42 everywhere, why everything runs locally (pointing at `/privacy` rather than repeating it — duplicated prose diverges), and **what LabML does not do, and why** — a project that names its limits reads as a serious one, and the limits are already measured here (the comparison question a 0.6B model does not read, the bench that needs `shader-f16`, SQL over the file before the recipe). Task-shaped how-to guides for readers who already know their way around: score a new batch, compare two runs, read a learning curve, hand a SQL result to the lab. Generated screenshots and the « try it » deep-links land here too. Every page ends with « et ensuite ? » — documentation without a next step is a dead end. | Three audiences, deliberately: the curious visitor (five minutes), the practitioner (one task), and the evaluator judging whether the engineering is rigorous. The explanation pages are what the third one reads. | | **V35 — delivered** | **ML Lab: the number stops flattering itself.** Two method defects in shipped code, fixed, plus the two additions that follow from them. **(1) The winner was picked on the test set** — the leaderboard sorted nine models by the metric computed on test and crowned `sorted[0]`, which makes the headline figure the optimistic maximum of nine draws. There is now a **third split**: validation is carved out of the train side (64/16/20 with the default ratios), ranking and crowning happen on validation, and the champion line spells out both numbers and the gap between them — « selected on validation at 0.974, scores 0.917 on the untouched test set ». The **test indices are byte-identical** to what the same config produced before V35, so every panel that reads the test set (segments, thresholds, uncertainty, batch compare) is unchanged; below 60 usable rows the third split is refused by name and the lab ranks on test as before. Ranking now lives in ONE module (`ranking.ts`) used by the leaderboard, the history, the run comparison, the report and the auto-selected insights model — the bug that shipped mid-wave was exactly that a fourth site still sorted on test and opened a different model than the one crowned. **(2) The split was always random, even on dated data.** A chronological split (oldest rows train, newest test, rows without a parseable date dropped and counted) and a group split (no group on both sides — the same customer in train and test is the same leak) are offered when a column supports one, and **announced** in the run info. **(3) A predictive leak detector**: V6 caught columns that MAP to the target; this catches the merely predictive one — a one-column stump fitted on train and scored on validation, and a lone column reading the target at ≥ 99% shows as a copper warning with its measured score, never as a victory. **(4) A robust leaderboard on demand**: 5×2 repeated cross-validation over train+validation (the pipeline refitted inside every fold, the test set never touched), reporting a mean, a spread, and how often the leader actually beat the runner-up — « 10 of 10 folds: the order is stable » or « 6 of 10: treat them as tied ». **A defect the wave exposed and named**: with the smaller train split, Gaussian Naive Bayes on ~150 TF-IDF features saturates to exactly 0/1, so V24's word-effect occlusion measured exactly zero for every word and the card simply vanished — reading as « no word matters », which is false. Measured (2 distinct probabilities out of 48 test rows, against 48 for logistic and gbdt), the card now **refuses by name** and points at a model that can answer. 369 unit tests, 65 e2e. | Owner request (22/08/2026), launched 23/08/2026. Two of the four items were defects rather than gaps: a lab that sells honest evaluation cannot ship a headline figure it knows to be optimistic, nor a split that leaks on dated data. | | **V36 — delivered** | **ML Lab: the gaps that were deliberately left open.** Each item was consciously deferred in an earlier wave rather than forgotten; delivering them together keeps the descopes visible instead of letting them quietly become permanent. **(1) Class imbalance**, descoped by name in V16. Two mechanisms, each NAMED per family rather than hidden behind one word: the loss is weighted where the loss is ours (logistic regression, gradient boosting — the gradient AND the hessian are scaled, since scaling only the gradient inflates leaf values instead of rebalancing), and a **seeded balanced resample** is used for the ml-cart families (tree, forest), which take no sample weights. The minority is upsampled to the majority's size — never the reverse: balancing by trimming the common class throws away real observations to fix a ratio. Off by default, because on a balanced target it changes nothing and a knob that does nothing is worse than no knob; the run announces the majority share, and the leaderboard says so when it crosses 60%. **(2) The ranking metric is a choice.** Accuracy and RMSE were imposed, which is the wrong criterion on an imbalanced problem — a model that never predicts the rare class can top an accuracy ranking and be useless. Rank on F1, recall, precision or ROC-AUC and the order genuinely changes; a model that cannot produce the chosen metric sorts last rather than being dropped. Ranking stays in the single V35 module, so the leaderboard, the history, the comparison, the report and the auto-inspected model all move together. **(3) Multiclass thresholds**, open since V16, read **one-vs-rest**: pick a class, score it against all the others, same PR and calibration curves. What the panel refuses to imply is a complete multiclass decision rule — two classes can both clear their thresholds and nothing here says which wins, so it says that instead. **(4) An ensemble of the best**: the average of the top three, built from models already fitted, so it costs one pass over the test set. The baseline is never a member (averaging a constant predictor drags the result toward the majority class), members are picked by the same V35 ranking rule, and **probabilistic members are preferred** — a mid-wave measurement showed the ensemble winning on iris with a bare vote because k-NN was in the top three, which silently closed the threshold, calibration and word-effect panels on the champion. **What this wave deliberately does not do**: add a tenth model family (nine is plenty; a tenth improves neither honesty nor understanding), build an AutoML « we handle everything » mode (the opposite of a lab that shows its decisions), or bring in tabular deep learning (high cost, no gain at this scale, and no longer hand-written). 387 unit tests, 69 e2e. | Launched 23/08/2026, right after V35. Each item was a named descope, not an oversight — and the ensemble exposed one more silent-disappearance defect, of the same family as the one V35 found. | -| V37 | **ML Lab: speed and the comfort of long sessions.** **Parallel training** — the zoo trains sequentially in a single worker; N workers means N cores, and at a million rows that is a different experience entirely. The V25 benches already exist to measure it before and after, so the gain is published rather than claimed. **Comparing more than two runs** — V21 compares two; three or four changes what the tool is for, and the diff machinery is already written. **Resuming an interrupted run** — closing the tab loses everything today, while V13 (artifacts) and V19 (persistence) already provide the storage; what is missing is a checkpoint between model families and the offer to resume. | Comes last on purpose: speed and comfort matter, but a faster wrong number is still a wrong number. V35 first, then V36, then this. | -| V38 | **Data Studio: reading the file exactly as it was written.** The headline item is a **defect in shipped code, not a missing feature**. `Papa.parse` is called with `skipEmptyLines: true` and nothing else — no encoding, no decimal separator — and `parseNumber` ends in `Number(cleaned)`. A French Excel export therefore breaks silently: `12,5` becomes `NaN`, the column is classified **text** rather than numeric, and every downstream stage one-hot encodes what should have been a number; a windows-1252 file displays `Québec`. Nothing warns, nothing refuses — the pipeline simply produces a worse model. Fix: **detect encoding and decimal separator and announce both** (« séparateur décimal : virgule, détecté sur 412 valeurs »), expose explicit **delimiter / encoding / decimal** selectors for the cases detection cannot settle, and show a **5-row preview before committing to the load** so a wrong guess is caught in two seconds rather than three panels later. The same pass covers thousands separators and dates written `31/12/2025` instead of ISO. | Owner request (22/08/2026): what to improve in /data. The audit found a defect first: a French-locale CSV — the single most likely file this owner's users will open — loses its numeric columns silently, and silence is the part that violates the project's rules. | -| V39 | **Data Studio: a recipe that works column by column.** `RecipeOptions` today applies `missing` and `clipOutliers` to the **whole file**: one strategy for every column, however different they are. A median makes sense for an age and none at all for a postcode. Make the recipe an ordered **list of per-column steps** — the current global settings becoming the defaults a column may override — and add the strategies that are missing: **median / mean / constant / a « MANQUANT » category** for categorical columns. With them comes a rule the tool should never break: **imputing without marking destroys information**, so every imputed column gains an optional **missing indicator** (`col_absent`), which is frequently predictive in its own right (a blank field is rarely blank at random). The recipe stays what it already is — a replayable, inspectable object — so the per-column version remains exportable, re-appliable to a new file, and legible as a list of named decisions. | A single global strategy is the kind of default that looks tidy and quietly makes the data worse; per-column steps cost little to build because the recipe is already an object, not a pile of checkboxes. | -| V40 | **Data Studio: validity, drift, and an auditable diff.** Quality is measured today as completeness and consistency of type; what is missing is **validity** — a value can be present, well-typed and still impossible. Named rules, each stated in plain language: an age of 200, a date in the future, a percentage at 130, a malformed postcode. Then **cross-column consistency** (`date_fin < date_debut`, `total ≠ quantité × prix`), for which **V29's DuckDB is already the engine** — the rules are SQL, and they run on the file that is already registered. Then three things that make the studio auditable rather than merely helpful: a **replayable reference profile** so a second file can be checked for drift against the first (the same idea as the V22 model manifest), a **before/after diff of the rows a recipe modified** — which rows, which columns, which values, not just a count — and a **breakdown of the quality score** so the number is explained by its parts instead of being asserted. Ends with **Parquet export**, nearly free now that DuckDB is loaded (`COPY … TO 'x.parquet'`). **What this wave deliberately does not do**: a spreadsheet-style cell editor (hand edits break reproducibility — the recipe is the record), fuzzy deduplication (guaranteed false positives on names and addresses, silently merging two real people), or model-based imputation (opaque, and it fabricates values that look plausible). | Comes last because it builds on V38's faithful read and V39's per-column recipe: validity rules on mis-parsed numbers would flag the parser, not the data. | +| **V37 — delivered** | **ML Lab: speed and the comfort of long sessions.** The wave opened, as the plan demanded, with a measurement — and the measurement moved the wave. **(1) Parallel training** was the headline: the zoo trains sequentially in one worker, so the heavy families were shipped to helper cores. Models cannot cross a worker boundary — `predict` is a closure and structured clone drops functions — so each helper returns `toJSON()` **as a JSON string** and the caller rebuilds through the V22 import path, which makes a parallel model byte-identical to an imported one. That string is not a detail: posting the object instead let structured clone keep shapes JSON drops, and ml-cart's `load()` then rebuilt a tree whose first prediction threw `this.root.classify(...).maxRowIndex is not a function` — a defect that would only have surfaced later, when the user opened insights. Helpers are split by measured cost, heaviest first, to the lightest helper (greedy longest-processing-time); k-NN never leaves the main worker, being the one family with no `toJSON` and also the one that fits in 0 ms; any failure — no Worker support, a helper that throws, an unserialisable family — silently falls back to the sequential trainer, because parallelism may change how long a run takes and never which models it produces. Every family's inference latency is still measured **here**, on the rebuilt predictor: a helper's timing would describe another core under contention, and the column would otherwise read 0 ms for exactly the families that ran in parallel. **(2) The measurement then found the real bottleneck, which was not training at all.** On a 60 000-row run, k-NN inference cost **59.6 s of a 68.8 s wall time — 87% of the whole run** — because the neighbour search allocated 5 000 objects and sorted them for every single prediction, and the scorer asked for labels and then for probabilities, searching twice. Fixed with a bounded top-k insertion over a flat `Float64Array` (ties keep the row seen first, exactly as the stable sort did) and an explicit `predictWithProba` for families where both answers come out of one computation. The old sorted implementation is kept verbatim in the tests as the oracle: the fast path is asserted to predict identically, row for row. **(3) Comparing more than two runs** — V21 compares two; three to six read against the **oldest** of the selection, so the deltas say what the session's changes did rather than what the newest run happens to be. Deliberately not a second diff engine: the matrix is the same V35 ranking and the feature columns are set algebra over the same `summary.featureColumns` V21 reads. **Measured, same machine, same 60 000-row file, same seed** — four arms: | -**Ordering**: V38 comes before V39 and V40, and for the same reason V35 came first in its own group: its headline item is a defect in shipped code, not a feature — a studio that promises honest data cannot silently turn `12,5` into `NaN`. V35 and V36 are delivered; V37 follows them. V32 ships one finished tutorial before any reference page — the tutorial is the template the rest copies, and settling it late means rewriting everything. V30 and V31 both start with a bench, because neither « a bigger model » nor « it still makes mistakes » is a measurable statement today; no wave starts without an explicit launch command. V23 first (owner request); V24 keeps its vocabulary capped — +| | k-NN as shipped before | k-NN fixed | +| ---------- | ---------------------- | ------------ | +| sequential | 73 950 ms | 14 654 ms | +| parallel | 69 738 ms | **9 509 ms** | + +So parallel training alone was worth **1.06×** — real work, drowned by a bottleneck nobody had measured; the k-NN fix alone **5.0×**; parallelism on top of it **1.54×**; together **7.8×**, with every leaderboard number identical. No minimum dataset size guards the helpers, because that was measured too: the gain is already positive at 1 000 rows (2 919 → 2 375 ms), at 3 000 (6 479 → 5 446 ms) and at 8 000 (7 496 → 5 937 ms). **Descoped by name: resuming an interrupted run.** It was the third item and it is not shipping, for a stated reason rather than for lack of time. A resume needs the fitted **pipeline** persisted as well as the models; k-NN has no `toJSON` at all, so the resumed leaderboard would be missing a family the interrupted one had; and it would only work for runs whose dataset the user opted to keep under V19's 50 MB budget — a feature that works for some runs and silently produces a different leaderboard for others is worse than no feature. The exposure it protects against also shrank by 7.8× in this very wave. It stays open, honestly, rather than shipping as a half-feature. **What this wave deliberately does not do**: cache trained models across runs (the cache key is the entire config plus the data — a stale hit is a silently wrong leaderboard), or move k-NN scoring to a helper (it is now 700 ms, and the model must exist in the main worker anyway for insights and what-if). 422 unit tests, 71 e2e. | Launched 23/08/2026, after V36. The plan said « measure before and after so the gain is published rather than claimed » — and the measurement is what turned the wave around: the promised parallelism was worth 6%, while the bottleneck it revealed was worth 5×. Two latent defects fell out of the same instrumentation: models corrupted by structured clone, and an inference column reading 0 ms for every parallel family. | +| V38 | **Data Studio: reading the file exactly as it was written.** The headline item is a **defect in shipped code, not a missing feature**. `Papa.parse` is called with `skipEmptyLines: true` and nothing else — no encoding, no decimal separator — and `parseNumber` ends in `Number(cleaned)`. A French Excel export therefore breaks silently: `12,5` becomes `NaN`, the column is classified **text** rather than numeric, and every downstream stage one-hot encodes what should have been a number; a windows-1252 file displays `Québec`. Nothing warns, nothing refuses — the pipeline simply produces a worse model. Fix: **detect encoding and decimal separator and announce both** (« séparateur décimal : virgule, détecté sur 412 valeurs »), expose explicit **delimiter / encoding / decimal** selectors for the cases detection cannot settle, and show a **5-row preview before committing to the load** so a wrong guess is caught in two seconds rather than three panels later. The same pass covers thousands separators and dates written `31/12/2025` instead of ISO. | Owner request (22/08/2026): what to improve in /data. The audit found a defect first: a French-locale CSV — the single most likely file this owner's users will open — loses its numeric columns silently, and silence is the part that violates the project's rules. | +| V39 | **Data Studio: a recipe that works column by column.** `RecipeOptions` today applies `missing` and `clipOutliers` to the **whole file**: one strategy for every column, however different they are. A median makes sense for an age and none at all for a postcode. Make the recipe an ordered **list of per-column steps** — the current global settings becoming the defaults a column may override — and add the strategies that are missing: **median / mean / constant / a « MANQUANT » category** for categorical columns. With them comes a rule the tool should never break: **imputing without marking destroys information**, so every imputed column gains an optional **missing indicator** (`col_absent`), which is frequently predictive in its own right (a blank field is rarely blank at random). The recipe stays what it already is — a replayable, inspectable object — so the per-column version remains exportable, re-appliable to a new file, and legible as a list of named decisions. | A single global strategy is the kind of default that looks tidy and quietly makes the data worse; per-column steps cost little to build because the recipe is already an object, not a pile of checkboxes. | +| V40 | **Data Studio: validity, drift, and an auditable diff.** Quality is measured today as completeness and consistency of type; what is missing is **validity** — a value can be present, well-typed and still impossible. Named rules, each stated in plain language: an age of 200, a date in the future, a percentage at 130, a malformed postcode. Then **cross-column consistency** (`date_fin < date_debut`, `total ≠ quantité × prix`), for which **V29's DuckDB is already the engine** — the rules are SQL, and they run on the file that is already registered. Then three things that make the studio auditable rather than merely helpful: a **replayable reference profile** so a second file can be checked for drift against the first (the same idea as the V22 model manifest), a **before/after diff of the rows a recipe modified** — which rows, which columns, which values, not just a count — and a **breakdown of the quality score** so the number is explained by its parts instead of being asserted. Ends with **Parquet export**, nearly free now that DuckDB is loaded (`COPY … TO 'x.parquet'`). **What this wave deliberately does not do**: a spreadsheet-style cell editor (hand edits break reproducibility — the recipe is the record), fuzzy deduplication (guaranteed false positives on names and addresses, silently merging two real people), or model-based imputation (opaque, and it fabricates values that look plausible). | Comes last because it builds on V38's faithful read and V39's per-column recipe: validity rules on mis-parsed numbers would flag the parser, not the data. | + +**Ordering**: V38 comes before V39 and V40, and for the same reason V35 came first in its own group: its headline item is a defect in shipped code, not a feature — a studio that promises honest data cannot silently turn `12,5` into `NaN`. V35, V36 and V37 are delivered. V32 ships one finished tutorial before any reference page — the tutorial is the template the rest copies, and settling it late means rewriting everything. V30 and V31 both start with a bench, because neither « a bigger model » nor « it still makes mistakes » is a measurable statement today; no wave starts without an explicit launch command. V23 first (owner request); V24 keeps its vocabulary capped — V25 (delivered) chose announced sampling and a named memory guard over the typed-array rewrite, which measurement showed unnecessary; widening the vocabulary stays possible later. Set aside for now: multiclass thresholds. V12 diff --git a/README.md b/README.md index c449b67..d88dff6 100644 --- a/README.md +++ b/README.md @@ -41,19 +41,20 @@ The project follows three non-negotiable principles: ### ML Lab — `/ml` -| Area | What it does | -| -------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| Data in | Drag & drop CSV/Excel (parsed in a worker), demo datasets, per-column profiling, automatic task detection, smart exclusions and **target-leakage detection**; **free-text columns** join the pipeline as hand-written TF-IDF (FR/EN tokenization, capped vocabulary, fitted on the training split only) | -| Models | 8 classifiers / 7 regressors trained live: naive baseline, linear/logistic regression, k-NN, Gaussian Naive Bayes, decision tree, random forest, **hand-written histogram gradient boosting** (LightGBM-style) and **MLP** (seeded He init, Adam) | -| Leaderboard | Accuracy/F1/ROC-AUC/log-loss or RMSE/MAE/R², delta vs baseline, train time, inference latency p50/p95, **95% bootstrap intervals** with a paired winner-vs-baseline verdict | -| Understanding | Confusion matrix, ROC, permutation importance, partial dependence, live what-if with **exact Shapley explanations**, **signed word effects** for text columns (which words push the answer up or down), plain-language read (FR/EN, rule-generated) | -| Where it fails | **Per-segment analysis**: the test set sliced by every categorical column — including excluded ones, where proxy effects hide — worst gaps first | -| Imbalance | Precision-recall curve (AP), calibration curve (Brier), **cost-priced decision threshold** with the optimal cut computed by exhaustive sweep | -| Tuning | Seeded random search scored by stratified 3-fold cross-validation, pipeline refitted inside each fold — the test set is scored exactly once | -| More data? | **Learning curve** on demand: one model retrained on growing seeded nested fractions, 95% bootstrap band, plain verdict — still climbing (collect more rows) or flattened (work on features) — including whether an announced training cap costs accuracy | -| No target? | Seeded k-means (k chosen by silhouette) + power-iteration PCA projection, groups described in plain language; date column? **Holt-Winters forecasting** validated by rolling-origin backtest | -| MLOps loop | Score a **new batch** with honest test-vs-batch metrics; **compare two runs** side by side with cross-run uncertainty verdicts; **export a model as JSON and re-import it later** — the exact predictor is rebuilt (byte-identical predictions) and scores any CSV without retraining | -| Persistence | Local run history with attached artifacts, opted-in dataset storage (compressed, explicit 50 MB budget), self-contained HTML reports, data-free share links | +| Area | What it does | +| -------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| Data in | Drag & drop CSV/Excel (parsed in a worker), demo datasets, per-column profiling, automatic task detection, smart exclusions and **target-leakage detection**; **free-text columns** join the pipeline as hand-written TF-IDF (FR/EN tokenization, capped vocabulary, fitted on the training split only) | +| Models | 8 classifiers / 7 regressors trained live: naive baseline, linear/logistic regression, k-NN, Gaussian Naive Bayes, decision tree, random forest, **hand-written histogram gradient boosting** (LightGBM-style) and **MLP** (seeded He init, Adam) | +| Leaderboard | Accuracy/F1/ROC-AUC/log-loss or RMSE/MAE/R², delta vs baseline, train time, inference latency p50/p95, **95% bootstrap intervals** with a paired winner-vs-baseline verdict | +| Understanding | Confusion matrix, ROC, permutation importance, partial dependence, live what-if with **exact Shapley explanations**, **signed word effects** for text columns (which words push the answer up or down), plain-language read (FR/EN, rule-generated) | +| Where it fails | **Per-segment analysis**: the test set sliced by every categorical column — including excluded ones, where proxy effects hide — worst gaps first | +| Imbalance | Precision-recall curve (AP), calibration curve (Brier), **cost-priced decision threshold** with the optimal cut computed by exhaustive sweep | +| Tuning | Seeded random search scored by stratified 3-fold cross-validation, pipeline refitted inside each fold — the test set is scored exactly once | +| More data? | **Learning curve** on demand: one model retrained on growing seeded nested fractions, 95% bootstrap band, plain verdict — still climbing (collect more rows) or flattened (work on features) — including whether an announced training cap costs accuracy | +| No target? | Seeded k-means (k chosen by silhouette) + power-iteration PCA projection, groups described in plain language; date column? **Holt-Winters forecasting** validated by rolling-origin backtest | +| MLOps loop | Score a **new batch** with honest test-vs-batch metrics; **compare two runs** side by side with cross-run uncertainty verdicts, or **up to six at once** read against the oldest of the selection; **export a model as JSON and re-import it later** — the exact predictor is rebuilt (byte-identical predictions) and scores any CSV without retraining | +| Speed | Heavy families train on **helper cores** (announced on the leaderboard, split by measured cost, never silent), and a model crosses back as JSON so it is rebuilt through the same path an imported model uses. Measured on a 60 000-row run: **74 s → 9.5 s**, with every leaderboard number identical | +| Persistence | Local run history with attached artifacts, opted-in dataset storage (compressed, explicit 50 MB budget), self-contained HTML reports, data-free share links | ### Data Studio — `/data` @@ -118,7 +119,7 @@ The project follows three non-negotiable principles: - **Performance.** Every section serves a prerendered static shell (hero paints before JavaScript); Lighthouse mobile ≈ 0.99 on `/ml` under real throttling. Heavy dependencies (Dexie, SheetJS, ONNX Runtime) load lazily. -- **Quality bar.** 387 unit tests, 69 Playwright end-to-end tests (including offline PWA, +- **Quality bar.** 422 unit tests, 71 Playwright end-to-end tests (including offline PWA, fake-webcam and axe-core WCAG A/AA accessibility checks), strict TypeScript, ESLint, Prettier, and Lighthouse budgets — all enforced in CI. diff --git a/e2e/parallel.spec.ts b/e2e/parallel.spec.ts new file mode 100644 index 0000000..5f82e25 --- /dev/null +++ b/e2e/parallel.spec.ts @@ -0,0 +1,83 @@ +import { expect, test } from '@playwright/test'; + +test.use({ locale: 'en-US' }); +test.setTimeout(180_000); + +/** + * V37 — speed, and the comfort of a long session. + * + * The two things worth pinning in a real browser: helper cores are ANNOUNCED + * like every other decision the lab makes, and a run trained across several + * cores is the same run — nothing about the leaderboard changes because of + * where a family happened to be fitted. + */ +test('the helper cores are announced by name, and the run is unchanged', async ({ page }) => { + await page.goto('/ml'); + await page.getByRole('button', { name: /titanic\.csv/ }).click(); + await expect(page.getByText('891 rows · 15 columns')).toBeVisible(); + await page.selectOption('#target-select', 'survived'); + await page.getByTestId('train-button').click(); + await expect(page.getByTestId('train-again')).toBeVisible({ timeout: 120_000 }); + + const leaderboard = page.getByTestId('leaderboard'); + // The announcement names how many cores helped and which families they took. + await expect(leaderboard).toContainText(/\d helper cores?:/); + await expect(leaderboard).toContainText('trained in parallel'); + // Parallelism is an optimisation: no family may go missing because of it. + await expect(leaderboard).not.toContainText('failed'); + await expect(leaderboard.locator('tbody tr')).toHaveCount(9); + + // A model fitted in a helper crosses back as JSON and is rebuilt here. If + // that rebuild were broken (it was, through structured clone) the first + // prediction would throw — the inference column is where that shows. + const rows = await leaderboard.locator('tbody tr').allInnerTexts(); + expect(rows.every((row) => row.includes('ms'))).toBe(true); +}); + +/** + * V37 — three or more runs read against the oldest, which is where the + * session started. Three separate pairwise diffs would make the reader do the + * joining; this table does it for them. + */ +test('iris: three runs compare in one table, against the oldest', async ({ page }) => { + await page.goto('/ml'); + await page.getByRole('button', { name: /iris\.csv/ }).click(); + await expect(page.getByText('150 rows · 5 columns')).toBeVisible(); + await page.selectOption('#target-select', 'species'); + + // Run A: everything. Run B: one feature dropped. Run C: a second one too. + await page.getByTestId('train-button').click(); + await expect(page.getByTestId('train-again')).toBeVisible({ timeout: 60_000 }); + await page + .getByTestId('column-card-petal_width') + .getByRole('button', { name: 'Exclude' }) + .click(); + await page.getByTestId('train-button').click(); + await expect(page.getByTestId('train-again')).toBeVisible({ timeout: 60_000 }); + await page + .getByTestId('column-card-sepal_width') + .getByRole('button', { name: 'Exclude' }) + .click(); + await page.getByTestId('train-button').click(); + await expect(page.getByTestId('train-again')).toBeVisible({ timeout: 60_000 }); + + const history = page.getByTestId('runs-history'); + const checkboxes = history.getByRole('checkbox'); + await expect(checkboxes).toHaveCount(3); + await checkboxes.nth(2).check(); // oldest — the reference + await checkboxes.nth(1).check(); + await checkboxes.nth(0).check(); + await page.getByTestId('compare-many-open').click(); + + const table = page.getByTestId('compare-many-page'); + await expect(table).toBeVisible(); + await expect(table).toContainText('Session comparison'); + // The oldest run is labelled as the reference, exactly once. + await expect(table.getByText('reference', { exact: true })).toHaveCount(1); + // The champion row and the per-model matrix both read across all three runs. + await expect(table).toContainText('Best model'); + await expect(table).toContainText('Naive baseline'); + // The features card names what each later run dropped relative to the first. + await expect(table).toContainText('petal_width'); + await expect(table).toContainText('sepal_width'); +}); diff --git a/e2e/text.spec.ts b/e2e/text.spec.ts index 0808ca6..dbb94fd 100644 --- a/e2e/text.spec.ts +++ b/e2e/text.spec.ts @@ -35,7 +35,12 @@ test('reviews: the text column trains, and the words explain the model', async ( await expect(words).toContainText('Pick another model in the leaderboard'); // And on a model that gives graded probabilities, the words do speak. - await page.getByTestId('leaderboard').getByText('Gradient boosting').click(); + // The row, not any mention of the name: V37's footer announces which + // families trained on helper cores, so the name also appears there. + await page + .getByTestId('leaderboard') + .getByRole('row', { name: /Gradient boosting/ }) + .click(); await expect(words).toContainText('Words that move the answer', { timeout: 30_000 }); // Effects are signed — at least one word pushes each way on this dataset. await expect(words).toContainText('+'); @@ -56,6 +61,11 @@ test('reviews in French: the words card speaks French too', async ({ page }) => await expect(words).toBeVisible({ timeout: 30_000 }); // Le refus est traduit lui aussi — une carte qui se tait n'apprend rien. await expect(words).toContainText('ses probabilités sont saturées'); - await page.getByTestId('leaderboard').getByText('Gradient boosting').click(); + // The row, not any mention of the name: V37's footer announces which + // families trained on helper cores, so the name also appears there. + await page + .getByTestId('leaderboard') + .getByRole('row', { name: /Gradient boosting/ }) + .click(); await expect(words).toContainText('Les mots qui font bouger la réponse', { timeout: 30_000 }); }); diff --git a/src/app/router.tsx b/src/app/router.tsx index f00bd4c..4faa82c 100644 --- a/src/app/router.tsx +++ b/src/app/router.tsx @@ -27,6 +27,12 @@ export const router = createBrowserRouter([ Component: (await import('@/features/ml/pages/MlComparePage')).default, }), }, + { + path: 'ml/compare-many/:ids', + lazy: async () => ({ + Component: (await import('@/features/ml/pages/MlCompareManyPage')).default, + }), + }, { path: 'ml/share', lazy: async () => ({ diff --git a/src/features/ml/components/LeaderboardTable.tsx b/src/features/ml/components/LeaderboardTable.tsx index cc0341a..52badad 100644 --- a/src/features/ml/components/LeaderboardTable.tsx +++ b/src/features/ml/components/LeaderboardTable.tsx @@ -361,6 +361,18 @@ export function LeaderboardTable({ ) )} + {summary.parallel !== undefined && summary.parallel.families.length > 0 && ( + <> + {' '} + ·{' '} + {t('ml.lab.leaderboard.parallel', { + count: summary.parallel.helpers, + families: summary.parallel.families + .map((key) => t(`ml.lab.models.${key}`)) + .join(', '), + })} + + )} {summary.skippedColumns.length > 0 && ( <> {' '} diff --git a/src/features/ml/components/RunsHistory.tsx b/src/features/ml/components/RunsHistory.tsx index 96380ae..bd0b0f1 100644 --- a/src/features/ml/components/RunsHistory.tsx +++ b/src/features/ml/components/RunsHistory.tsx @@ -13,6 +13,7 @@ import { useLabStore } from '@/features/ml/lab-store'; import { cn } from '@/lib/utils'; import type { RunRecord } from '@/features/ml/projects/types'; import { bestResult } from '@/features/ml/train/ranking'; +import { MAX_RUNS } from '@/features/ml/projects/compare-many'; /** Datasets kept in the browser (v19) — reopen or forget, all local. */ function SavedDatasets() { @@ -113,9 +114,13 @@ export function RunsHistory() { .map((id) => runs.find((r) => r.id === id)) .filter((r): r is RunRecord => Boolean(r)); + // V37: up to MAX_RUNS selections, not two. Three or four runs answer a + // different question — « which of the things I tried actually worked? » function toggleCompare(id: number) { setCompare((current) => - current.includes(id) ? current.filter((v) => v !== id) : [...current.slice(-1), id], + current.includes(id) + ? current.filter((v) => v !== id) + : [...current.slice(-(MAX_RUNS - 1)), id], ); } @@ -252,6 +257,18 @@ export function RunsHistory() { )} + {compared.length > 2 && ( +
+ r.id).join('-')}`} + data-testid="compare-many-open" + className={cn(buttonVariants({ size: 'sm' }))} + > +
+ )} diff --git a/src/features/ml/data/parse.worker.ts b/src/features/ml/data/parse.worker.ts index bbb9ea8..a0dc170 100644 --- a/src/features/ml/data/parse.worker.ts +++ b/src/features/ml/data/parse.worker.ts @@ -6,6 +6,7 @@ import { analyzeTarget, baselineSuggestions } from '@/features/ml/data/suggest'; import { computeInsights, computeWhatIf } from '@/features/ml/train/insights'; import { runLearningCurve } from '@/features/ml/train/learning-curve'; import { robustRank } from '@/features/ml/train/robust'; +import { trainInParallel } from '@/features/ml/train/parallel-run'; import { runSearch } from '@/features/ml/train/search'; import { runExploration } from '@/features/ml/unsupervised/explore'; import { runForecast } from '@/features/ml/timeseries/run'; @@ -16,7 +17,7 @@ import { scoreBatch, scoreRows } from '@/features/ml/train/score'; import { analyzeSegments } from '@/features/ml/train/segments'; import { analyzeThresholds } from '@/features/ml/train/threshold-analysis'; import { analyzeUncertainty, type ModelLosses } from '@/features/ml/train/uncertainty'; -import { runTraining, type TrainArtifacts } from '@/features/ml/train/trainer'; +import { detectTaskType, runTraining, type TrainArtifacts } from '@/features/ml/train/trainer'; import type { Cell, ColumnProfile, ParseResultPayload } from '@/features/ml/data/types'; import type { ModelKey } from '@/features/ml/train/types'; import type { WorkerRequest, WorkerResponse } from '@/features/ml/worker-protocol'; @@ -264,15 +265,50 @@ self.onmessage = async (event: MessageEvent) => { const profiles: ColumnProfile[] = header.map((column, i) => profileColumn(column, columns[i]), ); - const outcome = await runTraining(columnsAsMap(), profiles, request.config, { - onModelStart: (key, index, total) => post({ kind: 'model-start', key, index, total }), - onModelResult: (result) => post({ kind: 'model-result', result }), - isCancelled: () => cancelTraining, - }); + // V37: the heavy families first, on other cores. Best-effort — anything + // that does not come back is fitted by the sequential loop below. An + // undetectable task skips the parallel phase entirely so that runTraining + // stays the code that raises the named error. + const taskType = detectTaskType(columnsAsMap(), profiles, request.config); + const parallel = + taskType === null + ? { pretrained: undefined, report: null } + : await trainInParallel( + header, + columns, + request.config, + taskType === 'classification', + () => undefined, + () => cancelTraining, + ); + + const outcome = await runTraining( + columnsAsMap(), + profiles, + request.config, + { + onModelStart: (key, index, total) => post({ kind: 'model-start', key, index, total }), + onModelResult: (result) => post({ kind: 'model-result', result }), + isCancelled: () => cancelTraining, + }, + parallel.pretrained, + ); if (outcome) { artifacts = outcome.artifacts; lastFeatureColumns = outcome.summary.featureColumns; - post({ kind: 'train-complete', summary: outcome.summary }); + post({ + kind: 'train-complete', + summary: + parallel.report === null + ? outcome.summary + : { + ...outcome.summary, + parallel: { + helpers: parallel.report.helpers, + families: parallel.report.families, + }, + }, + }); } else { post({ kind: 'train-cancelled' }); } diff --git a/src/features/ml/pages/MlCompareManyPage.tsx b/src/features/ml/pages/MlCompareManyPage.tsx new file mode 100644 index 0000000..96fc3d7 --- /dev/null +++ b/src/features/ml/pages/MlCompareManyPage.tsx @@ -0,0 +1,163 @@ +import { useLiveQuery } from 'dexie-react-hooks'; +import { ArrowLeft } from 'lucide-react'; +import { useTranslation } from 'react-i18next'; +import { Link, useParams } from 'react-router'; +import { buttonVariants } from '@/components/ui/button'; +import { Card } from '@/components/ui/card'; +import { Eyebrow } from '@/components/ui/eyebrow'; +import { db } from '@/features/ml/projects/db'; +import { compareMany } from '@/features/ml/projects/compare-many'; +import { cn } from '@/lib/utils'; + +/** + * V37: three or more runs, side by side, all read against the OLDEST — which + * is where the session started, so the deltas say « what my changes did » + * rather than « what the newest run happens to be ». + */ +export default function MlCompareManyPage() { + const { t, i18n } = useTranslation(); + const lang = i18n.resolvedLanguage ?? 'en'; + const { ids } = useParams<{ ids: string }>(); + const wanted = (ids ?? '') + .split('-') + .map((value) => Number(value)) + .filter((value) => Number.isFinite(value) && value > 0); + + const runs = useLiveQuery(() => db.runs.bulkGet(wanted), [ids]); + const found = (runs ?? []).filter((run) => run !== undefined); + const comparison = found.length > 0 ? compareMany(found) : null; + + const fmt = (value: number | null) => + value === null + ? '—' + : value.toLocaleString(lang, { minimumFractionDigits: 3, maximumFractionDigits: 3 }); + const signed = (value: number | null) => + value === null ? '' : `${value > 0 ? '+' : ''}${value.toFixed(3)}`; + + return ( +
+ +
+ ); +} diff --git a/src/features/ml/projects/compare-many.test.ts b/src/features/ml/projects/compare-many.test.ts new file mode 100644 index 0000000..b30727a --- /dev/null +++ b/src/features/ml/projects/compare-many.test.ts @@ -0,0 +1,134 @@ +import { describe, expect, it } from 'vitest'; +import { compareMany, MAX_RUNS } from '@/features/ml/projects/compare-many'; +import type { RunRecord } from '@/features/ml/projects/types'; +import type { ModelKey } from '@/features/ml/train/types'; + +// V37: N runs, read against the oldest. Not a second diff engine — the same +// ranking rule and the same feature lists, in a wider shape. + +function run( + id: number, + createdAt: number, + scores: Partial>, + features: string[] = ['a', 'b'], + over: Partial = {}, +): RunRecord { + return { + id, + name: `run ${id}`, + createdAt, + dataset: { name: 'demo.csv', rowCount: 100, columnCount: 3 }, + target: 'label', + taskType: 'binary', + seed: 42, + results: Object.entries(scores).map(([key, value]) => ({ + key: key as ModelKey, + ok: true, + metrics: { accuracy: value as number }, + primary: value as number, + valPrimary: value as number, + valMetrics: { accuracy: value as number }, + trainMs: 0, + inferP50Ms: 0, + inferP95Ms: 0, + })), + summary: { featureColumns: features } as RunRecord['summary'], + insights: {} as RunRecord['insights'], + artifacts: {}, + ...over, + } as RunRecord; +} + +describe('compareMany (V37)', () => { + it('refuses fewer than two and more than six runs', () => { + expect(compareMany([run(1, 1, { tree: 0.8 })])).toBeNull(); + const many = Array.from({ length: MAX_RUNS + 1 }, (_, i) => run(i, i, { tree: 0.8 })); + expect(compareMany(many)).toBeNull(); + }); + + it('reads every run against the OLDEST, whatever the selection order', () => { + const a = run(1, 100, { tree: 0.7, gbdt: 0.75 }); + const b = run(2, 200, { tree: 0.72, gbdt: 0.8 }); + const c = run(3, 300, { tree: 0.6, gbdt: 0.65 }); + const cmp = compareMany([c, a, b])!; // deliberately out of order + expect(cmp.referenceId).toBe(1); + expect(cmp.columns.map((col) => col.id)).toEqual([1, 2, 3]); + expect(cmp.columns[0].delta).toBeNull(); // the reference has no delta + expect(cmp.columns[1].delta).toBeCloseTo(0.05, 10); // 0.80 − 0.75 + expect(cmp.columns[2].delta).toBeCloseTo(-0.1, 10); // 0.65 − 0.75 + }); + + it('names the leader — the run whose champion is best', () => { + const cmp = compareMany([ + run(1, 100, { gbdt: 0.75 }), + run(2, 200, { gbdt: 0.8 }), + run(3, 300, { gbdt: 0.65 }), + ])!; + expect(cmp.leaderId).toBe(2); + }); + + it('picks the LOWEST champion on regression', () => { + const rmse = (id: number, at: number, value: number) => + run(id, at, { gbdt: value }, ['a', 'b'], { taskType: 'regression' }); + const cmp = compareMany([rmse(1, 100, 5), rmse(2, 200, 3), rmse(3, 300, 8)])!; + expect(cmp.isClassification).toBe(false); + expect(cmp.leaderId).toBe(2); + }); + + it('builds a matrix with a hole where a model did not run', () => { + const cmp = compareMany([ + run(1, 100, { tree: 0.7, gbdt: 0.75 }), + run(2, 200, { gbdt: 0.8, mlp: 0.9 }), + ])!; + const byKey = new Map(cmp.models.map((m) => [m.key, m.values])); + expect(byKey.get('tree')).toEqual([0.7, null]); + expect(byKey.get('mlp')).toEqual([null, 0.9]); + expect(byKey.get('gbdt')).toEqual([0.75, 0.8]); + }); + + it('orders the matrix by the reference run own ranking', () => { + const cmp = compareMany([ + run(1, 100, { tree: 0.6, gbdt: 0.9, mlp: 0.75 }), + run(2, 200, { tree: 0.99 }), + ])!; + // gbdt led the reference, so it leads the table — not the newest winner. + expect(cmp.models.map((m) => m.key)).toEqual(['gbdt', 'mlp', 'tree']); + }); + + it('reports feature moves against the reference, and the shared core', () => { + const cmp = compareMany([ + run(1, 100, { gbdt: 0.7 }, ['a', 'b', 'c']), + run(2, 200, { gbdt: 0.8 }, ['a', 'b', 'd']), + run(3, 300, { gbdt: 0.9 }, ['a', 'd']), + ])!; + expect(cmp.columns[1].added).toEqual(['d']); + expect(cmp.columns[1].removed).toEqual(['c']); + expect(cmp.sharedFeatures).toEqual(['a']); + }); + + it('refuses to call numbers deltas when the targets differ', () => { + const cmp = compareMany([ + run(1, 100, { gbdt: 0.7 }), + run(2, 200, { gbdt: 0.8 }, ['a', 'b'], { target: 'autre' }), + ])!; + expect(cmp.comparable).toBe(false); + expect(cmp.columns[1].delta).toBeNull(); + expect(cmp.leaderId).toBeNull(); + }); + + it('ranks on the chosen metric, like every other surface (V36)', () => { + const withRecall = (id: number, at: number, acc: number, recall: number): RunRecord => { + const r = run(id, at, { gbdt: acc }); + r.results[0].valMetrics = { accuracy: acc, recall }; + r.results[0].metrics = { accuracy: acc, recall }; + return r; + }; + const onAccuracy = compareMany([withRecall(1, 100, 0.9, 0.1), withRecall(2, 200, 0.8, 0.9)])!; + expect(onAccuracy.leaderId).toBe(1); + const onRecall = compareMany( + [withRecall(1, 100, 0.9, 0.1), withRecall(2, 200, 0.8, 0.9)], + 'recall', + )!; + expect(onRecall.leaderId).toBe(2); + }); +}); diff --git a/src/features/ml/projects/compare-many.ts b/src/features/ml/projects/compare-many.ts new file mode 100644 index 0000000..40f29b3 --- /dev/null +++ b/src/features/ml/projects/compare-many.ts @@ -0,0 +1,153 @@ +/** + * V37: comparing more than two runs. + * + * V21 answered "did my cleaning help?" — one change, two runs, a diff. Three + * or four runs answer a different question: "which of the things I tried + * actually worked?" That is what a long session produces, and reading it as + * three separate pairwise diffs makes the reader do the joining. + * + * Deliberately NOT a second diff engine: the per-model table is a matrix over + * the same `bestResult` ranking every other surface uses (V35/V36), and the + * feature columns are a set algebra over the same `summary.featureColumns` + * V21 reads. What is new is only the shape — N columns instead of a left and + * a right, and a baseline run every other one is measured against. + */ +import { bestResult, rankingValue } from '@/features/ml/train/ranking'; +import type { RunRecord } from '@/features/ml/projects/types'; +import type { ModelKey, RankingMetric } from '@/features/ml/train/types'; + +/** At least two runs to compare; past six the table stops being readable. */ +export const MIN_RUNS = 2; +export const MAX_RUNS = 6; + +export interface RunColumn { + id: number; + name: string; + createdAt: number; + /** The run's champion under the shared ranking rule. */ + best: { key: ModelKey; value: number } | null; + /** Champion value minus the reference run's — null for the reference. */ + delta: number | null; + featureCount: number; + /** Columns this run uses that the reference does not, and vice versa. */ + added: string[]; + removed: string[]; +} + +export interface ManyComparison { + /** The oldest selected run: everything else is read against it. */ + referenceId: number; + isClassification: boolean; + metric: RankingMetric | null; + /** Runs in selection order, reference first. */ + columns: RunColumn[]; + /** Every model key seen, with each run's value — null where it did not run. */ + models: { key: ModelKey; values: (number | null)[] }[]; + /** Columns present in every run — the stable core of the comparison. */ + sharedFeatures: string[]; + /** False when targets or task families differ: the numbers are not deltas. */ + comparable: boolean; + /** The run with the best champion, when the set is comparable. */ + leaderId: number | null; +} + +export function compareMany(runs: RunRecord[], metric?: RankingMetric): ManyComparison | null { + if (runs.length < MIN_RUNS || runs.length > MAX_RUNS) return null; + + // Oldest first: the reference is where the session started, so the deltas + // read as "what my changes did", not "what the newest run happens to be". + const ordered = [...runs].sort((a, b) => a.createdAt - b.createdAt); + const reference = ordered[0]; + const isClassification = reference.taskType !== 'regression'; + const comparable = ordered.every( + (run) => + run.target === reference.target && (run.taskType !== 'regression') === isClassification, + ); + + const bestOf = (run: RunRecord) => { + const best = bestResult(run.results, run.taskType, metric); + return best === null ? null : { key: best.key, value: rankingValue(best, metric) }; + }; + const referenceBest = bestOf(reference); + + const referenceFeatures = new Set(reference.summary.featureColumns); + const columns: RunColumn[] = ordered.map((run) => { + const best = bestOf(run); + const features = new Set(run.summary.featureColumns); + return { + id: run.id ?? 0, + name: run.name, + createdAt: run.createdAt, + best, + delta: + comparable && best !== null && referenceBest !== null && run.id !== reference.id + ? best.value - referenceBest.value + : null, + featureCount: features.size, + added: [...features].filter((f) => !referenceFeatures.has(f)).sort(), + removed: [...referenceFeatures].filter((f) => !features.has(f)).sort(), + }; + }); + + // The per-model matrix, ordered by the reference run's own ranking so the + // table reads top-down like the leaderboard the user already knows. + const seen = new Map(); + ordered.forEach((run, index) => { + for (const result of run.results) { + if (result.ok && !seen.has(result.key)) seen.set(result.key, index); + } + }); + const referenceOrder = new Map(); + reference.results + .filter((r) => r.ok) + .sort((a, b) => + isClassification + ? rankingValue(b, metric) - rankingValue(a, metric) + : rankingValue(a, metric) - rankingValue(b, metric), + ) + .forEach((r, i) => referenceOrder.set(r.key, i)); + + const models = [...seen.keys()] + .sort((a, b) => { + const ai = referenceOrder.get(a) ?? Number.MAX_SAFE_INTEGER; + const bi = referenceOrder.get(b) ?? Number.MAX_SAFE_INTEGER; + return ai !== bi ? ai - bi : a.localeCompare(b); + }) + .map((key) => ({ + key, + values: ordered.map((run) => { + const found = run.results.find((r) => r.key === key && r.ok); + return found === undefined ? null : rankingValue(found, metric); + }), + })); + + const sharedFeatures = [...referenceFeatures] + .filter((f) => ordered.every((run) => run.summary.featureColumns.includes(f))) + .sort(); + + let leaderId: number | null = null; + if (comparable) { + let bestValue: number | null = null; + for (const column of columns) { + if (column.best === null) continue; + const better = + bestValue === null || + (isClassification ? column.best.value > bestValue : column.best.value < bestValue); + if (better) { + bestValue = column.best.value; + leaderId = column.id; + } + } + } + + return { + referenceId: reference.id ?? 0, + isClassification, + metric: metric ?? null, + columns, + models, + sharedFeatures, + comparable, + leaderId, + }; +} diff --git a/src/features/ml/train/deserialize.ts b/src/features/ml/train/deserialize.ts index bc07f8e..a360623 100644 --- a/src/features/ml/train/deserialize.ts +++ b/src/features/ml/train/deserialize.ts @@ -56,6 +56,30 @@ function argmaxRows(proba: (rows: number[][]) => number[][]) { } /* eslint-disable @typescript-eslint/no-explicit-any */ +/** + * V37: exported for the parallel trainer. A model cannot cross a worker + * boundary — `predict` is a closure, and structured clone drops functions — + * so a helper worker returns `toJSON()` as a JSON string and the main worker + * parses it and rebuilds the predictor here. Passing the object instead of the + * string does NOT work: structured clone keeps shapes JSON drops, and ml-cart's + * `load()` then rebuilds a tree that throws on its first prediction. Going + * through JSON is what makes a parallel run and an imported model the same + * object rather than merely similar ones. + */ +export function rebuildTrainedModel( + kind: string, + params: any, + isClassification: boolean, +): TrainedModel { + // The rebuild path produces a predictor, not an exporter: nothing in + // `rebuildModel` defines `toJSON`, because an imported model has no reason + // to be re-exported. A family trained in a helper does — it is a normal row + // of a normal run, and the export button must still work on it. The + // parameters we were handed ARE what V22 writes to the file, so they go + // straight back out. + return { ...rebuildModel(kind, params, isClassification), toJSON: () => params }; +} + function rebuildModel(kind: string, p: any, isClassification: boolean): TrainedModel { if (kind === 'baseline') { if (p.task === 'regression') return { predict: (X) => X.map(() => p.mean as number) }; diff --git a/src/features/ml/train/family.worker.ts b/src/features/ml/train/family.worker.ts new file mode 100644 index 0000000..9f64dbd --- /dev/null +++ b/src/features/ml/train/family.worker.ts @@ -0,0 +1,174 @@ +/** + * V37: one helper worker that trains a SUBSET of the model families. + * + * Why a second worker type at all: the zoo trains sequentially in a single + * worker, and the measurement (see PLAN V37) says four families carry 97% of + * the cost and are roughly the same size — so N workers really do mean N + * cores here, not a rounding error. + * + * The constraint that shapes the protocol: a TrainedModel cannot cross a + * worker boundary. `predict` is a closure and structured clone drops + * functions. So this worker returns `toJSON()` and the caller rebuilds the + * predictor through the V22 deserialisation path — which also means a + * parallel run and an imported model are provably the same object. + * + * k-NN is deliberately NOT trainable here: it is the one family with no + * `toJSON` (it keeps its training rows), and it is also the one that costs + * 0 ms to fit. It stays in the main worker, where it belongs. + * + * Each worker re-derives the splits and the pipeline from the same seeded + * config, so nothing large is cloned in: the same seed gives the same rows, + * which is the whole point of seeding everything. + */ +import { profileColumn } from '@/features/ml/data/profile'; +import { MODEL_TRAIN_CAPS, modelZoo } from '@/features/ml/train/models'; +import { fitPipeline } from '@/features/ml/train/pipeline'; +import { nestedSampleOrder } from '@/features/ml/train/random'; +import { balancedWeights } from '@/features/ml/train/class-weight'; +import { prepareData, scoreModel } from '@/features/ml/train/trainer'; +import type { Cell } from '@/features/ml/data/types'; +import type { MetricMap, ModelKey, TrainConfig } from '@/features/ml/train/types'; + +/** Families this worker will never train — see the k-NN note above. */ +export const NOT_PARALLELISABLE: ReadonlySet = new Set(['knn']); + +export interface FamilyRequest { + header: string[]; + columns: Cell[][]; + config: TrainConfig; + families: ModelKey[]; +} + +export interface FamilyOutcome { + key: ModelKey; + ok: boolean; + error?: string; + /** + * `toJSON()` of the fitted model, as a JSON STRING — rebuilt by the caller + * through the V22 path. A string, not an object, and that is not a detail: + * structured clone keeps shapes JSON drops, and ml-cart's `load()` silently + * rebuilds a tree whose `classify` returns a plain object instead of a + * matrix. Measured: a structured-cloned tree throws + * `this.root.classify(...).maxRowIndex is not a function` on its first + * prediction. Going through JSON makes a parallel model byte-identical to an + * imported one — which is exactly what this protocol claims. + */ + serialized?: { kind: string; json: string }; + metrics?: MetricMap; + primary?: number; + valMetrics?: MetricMap; + valPrimary?: number; + trainMs?: number; + trainedRows?: number; +} + +export type FamilyResponse = + | { kind: 'family-done'; outcome: FamilyOutcome } + | { kind: 'batch-done' } + | { kind: 'batch-error'; message: string }; + +function post(message: FamilyResponse): void { + (self as unknown as Worker).postMessage(message); +} + +self.onmessage = (event: MessageEvent) => { + const { header, columns, config, families } = event.data; + try { + const map = new Map(); + header.forEach((name, i) => map.set(name, columns[i])); + const profiles = header.map((name, i) => profileColumn(name, columns[i])); + + const prepared = prepareData(map, profiles, config); + const { isClassification, classes, featureColumns, encode } = prepared; + const pipeline = fitPipeline(map, profiles, featureColumns, prepared.train); + const trainX = pipeline.transform(prepared.train); + const trainY = prepared.train.map(encode); + const testX = pipeline.transform(prepared.test); + const testY = prepared.test.map(encode); + const valX = prepared.validation.length > 0 ? pipeline.transform(prepared.validation) : null; + const valY = prepared.validation.length > 0 ? prepared.validation.map(encode) : null; + + const weights = + config.classWeighting === 'balanced' && isClassification + ? balancedWeights(trainY, classes.length) + : undefined; + const context = { + task: isClassification ? ('classification' as const) : ('regression' as const), + classCount: classes.length, + seed: config.seed, + ...(weights !== undefined && { classWeights: weights }), + }; + + // The SAME seeded order the sequential trainer uses, so a capped family + // sees the same rows whichever path trained it. + let sampleOrder: number[] | null = null; + const zoo = modelZoo(isClassification ? 'classification' : 'regression'); + + for (const key of families) { + const def = zoo.find((d) => d.key === key); + if (!def || NOT_PARALLELISABLE.has(key)) { + post({ kind: 'family-done', outcome: { key, ok: false, error: 'not-parallelisable' } }); + continue; + } + try { + const cap = MODEL_TRAIN_CAPS[key]; + let fitX = trainX; + let fitY = trainY; + if (cap !== undefined && trainX.length > cap) { + sampleOrder ??= nestedSampleOrder(trainX.length, prepared.trainLabels, config.seed); + const keep = sampleOrder.slice(0, cap).sort((a, b) => a - b); + fitX = keep.map((position) => trainX[position]); + fitY = keep.map((position) => trainY[position]); + } + const started = performance.now(); + const model = def.train(fitX, fitY, context); + const trainMs = performance.now() - started; + + const serialized = model.toJSON?.() as { kind?: string } | undefined; + if (serialized === undefined || typeof serialized.kind !== 'string') { + // Refused by name rather than returning a model the caller cannot + // rebuild — the sequential path will train this family instead. + post({ kind: 'family-done', outcome: { key, ok: false, error: 'not-serialisable' } }); + continue; + } + const test = scoreModel(model, testX, testY, isClassification, classes.length); + const validation = + valX !== null && valY !== null + ? scoreModel(model, valX, valY, isClassification, classes.length) + : null; + + post({ + kind: 'family-done', + outcome: { + key, + ok: true, + serialized: { kind: serialized.kind, json: JSON.stringify(serialized) }, + metrics: test.metrics, + primary: test.primary, + trainMs, + trainedRows: fitX.length, + ...(validation !== null && { + valMetrics: validation.metrics, + valPrimary: validation.primary, + }), + }, + }); + } catch (error) { + post({ + kind: 'family-done', + outcome: { + key, + ok: false, + error: error instanceof Error ? error.message : String(error), + }, + }); + } + } + post({ kind: 'batch-done' }); + } catch (error) { + post({ + kind: 'batch-error', + message: error instanceof Error ? error.message : String(error), + }); + } +}; diff --git a/src/features/ml/train/models.ts b/src/features/ml/train/models.ts index bb4e748..17a73e1 100644 --- a/src/features/ml/train/models.ts +++ b/src/features/ml/train/models.ts @@ -9,6 +9,14 @@ export interface TrainedModel { predict(X: number[][]): number[]; /** Class probabilities (n × k) — only for models that can produce them. */ predictProba?(X: number[][]): number[][]; + /** + * V37: labels and probabilities from a SINGLE pass, for families where the + * two answers come out of the same expensive computation. Scorers use it when + * present; everything else keeps calling `predict` and `predictProba`, which + * must stay exactly as authoritative. Only k-NN implements it today — there, + * calling both meant searching every neighbour twice. + */ + predictWithProba?(X: number[][]): { labels: number[]; proba: number[][] }; /** Serializable parameters for export — absent when not exportable (k-NN). */ toJSON?(): unknown; } @@ -221,18 +229,59 @@ export function trainKnn( ): TrainedModel { // V25: no silent subsample here any more — callers (trainer, search) cap the // training set through the announced mechanism before this function runs. - const trainX = X; + const rowCount = X.length; + const width = X[0]?.length ?? 0; + const k = Math.min(kWanted, rowCount); + + // V37: the training rows flattened once. Measured on a 60 000-row run, k-NN + // inference was 59.6 s of a 68.8 s wall time — 87% of the whole run, and by + // far the largest single cost in the app. The maths below is unchanged; what + // changed is that it no longer allocates 5 000 objects and sorts them for + // every single prediction. + const flat = new Float64Array(rowCount * width); + for (let i = 0; i < rowCount; i++) { + const row = X[i]; + for (let j = 0; j < width; j++) flat[i * width + j] = row[j]; + } const trainY = y; - const k = Math.min(kWanted, trainX.length); - function neighbors(row: number[]): number[] { - const distances = trainX.map((trainRow, i) => { + // Scratch buffers for the k best neighbours, reused across predictions. + const bestDistance = new Float64Array(k); + const bestLabel = new Float64Array(k); + + /** + * The k nearest labels, kept sorted by distance ascending. + * + * A bounded insertion instead of a full sort: k is 5, the training set is up + * to 5 000 rows, so almost every candidate fails the single `>= worst` test + * and costs nothing beyond its distance. Ties keep the row seen FIRST — the + * strict `<` below — which is exactly what the stable sort it replaces did, + * so the chosen neighbours are identical row for row. + */ + function neighbors(row: number[]): Float64Array { + let filled = 0; + let worst = Number.POSITIVE_INFINITY; + for (let i = 0; i < rowCount; i++) { + const base = i * width; let sum = 0; - for (let j = 0; j < row.length; j++) sum += (row[j] - trainRow[j]) ** 2; - return { i, d: sum }; - }); - distances.sort((a, b) => a.d - b.d); - return distances.slice(0, k).map(({ i }) => trainY[i]); + for (let j = 0; j < width; j++) { + const diff = row[j] - flat[base + j]; + sum += diff * diff; + } + if (filled === k && sum >= worst) continue; + // Slide the larger entries right, drop the last one when already full. + let position = filled < k ? filled : k - 1; + while (position > 0 && bestDistance[position - 1] > sum) { + bestDistance[position] = bestDistance[position - 1]; + bestLabel[position] = bestLabel[position - 1]; + position--; + } + bestDistance[position] = sum; + bestLabel[position] = trainY[i]; + if (filled < k) filled++; + worst = bestDistance[filled - 1]; + } + return bestLabel; } if (ctx.task === 'regression') { @@ -240,19 +289,32 @@ export function trainKnn( predict: (rows) => rows.map((row) => { const near = neighbors(row); - return near.reduce((a, v) => a + v, 0) / near.length; + let sum = 0; + for (let i = 0; i < k; i++) sum += near[i]; + return sum / k; }), }; } + const proba = (rows: number[][]) => rows.map((row) => { + const near = neighbors(row); const votes = new Array(ctx.classCount).fill(0); - for (const label of neighbors(row)) votes[label] += 1; + for (let i = 0; i < k; i++) votes[near[i] | 0] += 1; return votes.map((v) => v / k); }); + const labelsOf = (probabilities: number[][]) => + probabilities.map((p) => p.indexOf(Math.max(...p))); + return { - predict: (rows) => proba(rows).map((p) => p.indexOf(Math.max(...p))), + predict: (rows) => labelsOf(proba(rows)), predictProba: proba, + // V37: a scorer wants both, and for k-NN both come out of the same + // neighbour search — asking twice searched 5 000 rows twice per prediction. + predictWithProba: (rows) => { + const probabilities = proba(rows); + return { labels: labelsOf(probabilities), proba: probabilities }; + }, }; } diff --git a/src/features/ml/train/parallel-run.ts b/src/features/ml/train/parallel-run.ts new file mode 100644 index 0000000..2351a30 --- /dev/null +++ b/src/features/ml/train/parallel-run.ts @@ -0,0 +1,132 @@ +/** + * V37: runs the heavy families in helper workers and hands the results back + * as rebuilt predictors. Called from the ML worker; see parallel.ts for the + * measurement and the three rules this follows. + * + * Everything here is best-effort by design: any failure — no Worker support, + * a helper that throws, a family that comes back unserialisable — simply + * leaves that family out of the returned map, and the sequential trainer + * fits it as it always did. Parallelism must never change WHICH models a run + * produces, only how long it takes to produce them. + */ +import { rebuildTrainedModel } from '@/features/ml/train/deserialize'; +import { modelZoo } from '@/features/ml/train/models'; +import { familyCost, helperCount, planBatches } from '@/features/ml/train/parallel'; +import type { Cell } from '@/features/ml/data/types'; +import type { FamilyRequest, FamilyResponse } from '@/features/ml/train/family.worker'; +import type { PretrainedFamily } from '@/features/ml/train/trainer'; +import type { ModelKey, TrainConfig } from '@/features/ml/train/types'; + +export interface ParallelReport { + helpers: number; + families: ModelKey[]; + /** Wall time of the parallel phase, so the gain can be published. */ + ms: number; +} + +export async function trainInParallel( + header: string[], + columns: Cell[][], + config: TrainConfig, + isClassification: boolean, + onFamilyDone: (key: ModelKey) => void, + isCancelled: () => boolean, +): Promise<{ pretrained: Map; report: ParallelReport | null }> { + const empty = { pretrained: new Map(), report: null }; + if (typeof Worker === 'undefined') return empty; + + const zoo = modelZoo(isClassification ? 'classification' : 'regression').map((d) => d.key); + const cores = typeof navigator !== 'undefined' ? (navigator.hardwareConcurrency ?? 2) : 2; + const helpers = helperCount(zoo, cores); + if (helpers === 0) return empty; + + const batches = planBatches( + zoo.filter((key) => familyCost(key) > 1), + helpers, + familyCost, + ); + if (batches.length === 0) return empty; + + const started = performance.now(); + const pretrained = new Map(); + const workers: Worker[] = []; + + try { + await Promise.all( + batches.map( + (families) => + new Promise((resolve) => { + let worker: Worker; + try { + worker = new Worker(new URL('./family.worker.ts', import.meta.url), { + type: 'module', + }); + } catch { + resolve(); // no helper: the sequential path covers these families + return; + } + workers.push(worker); + + const finish = () => resolve(); + worker.onerror = finish; + worker.onmessageerror = finish; + worker.onmessage = (event: MessageEvent) => { + const message = event.data; + if (message.kind === 'batch-done' || message.kind === 'batch-error') { + finish(); + return; + } + const outcome = message.outcome; + if (!outcome.ok || outcome.serialized === undefined) return; + try { + const model = rebuildTrainedModel( + outcome.serialized.kind, + JSON.parse(outcome.serialized.json), + isClassification, + ); + pretrained.set(outcome.key, { + model, + result: { + key: outcome.key, + ok: true, + metrics: outcome.metrics ?? {}, + primary: outcome.primary ?? Number.NaN, + trainMs: outcome.trainMs ?? 0, + // Latency is measured in the main worker on the rebuilt + // predictor: a helper's timing would describe its own core. + inferP50Ms: 0, + inferP95Ms: 0, + ...(outcome.trainedRows !== undefined && { + trainedRows: outcome.trainedRows, + }), + ...(outcome.valMetrics !== undefined && { + valMetrics: outcome.valMetrics, + valPrimary: outcome.valPrimary, + }), + }, + }); + onFamilyDone(outcome.key); + } catch { + // Rebuild failed — leave it to the sequential trainer. + } + }; + + const request: FamilyRequest = { header, columns, config, families }; + worker.postMessage(request); + }), + ), + ); + } finally { + for (const worker of workers) worker.terminate(); + } + + if (isCancelled()) return empty; + return { + pretrained, + report: { + helpers: batches.length, + families: [...pretrained.keys()], + ms: performance.now() - started, + }, + }; +} diff --git a/src/features/ml/train/parallel.test.ts b/src/features/ml/train/parallel.test.ts new file mode 100644 index 0000000..5db19df --- /dev/null +++ b/src/features/ml/train/parallel.test.ts @@ -0,0 +1,86 @@ +import { describe, expect, it } from 'vitest'; +import { + familyCost, + helperCount, + planBatches, + HEAVY_FAMILIES, + MAX_HELPERS, +} from '@/features/ml/train/parallel'; +import { NOT_PARALLELISABLE } from '@/features/ml/train/family.worker'; +import type { ModelKey } from '@/features/ml/train/types'; + +// V37: orchestration only — the maths lives in the families. What must hold +// is that the split is balanced, bounded, and never silently drops a family. + +const ZOO: ModelKey[] = [ + 'baseline', + 'logistic', + 'knn', + 'naiveBayes', + 'tree', + 'forest', + 'gbdt', + 'mlp', +]; + +describe('helperCount (V37)', () => { + it('leaves a core for the UI and never exceeds the heavy families', () => { + expect(helperCount(ZOO, 8)).toBe(MAX_HELPERS); + expect(helperCount(ZOO, 3)).toBe(2); // cores - 1 + expect(helperCount(ZOO, 1)).toBe(1); + }); + + it('refuses to parallelise when there is nothing to gain', () => { + // A single heavy family in a helper is pure overhead. + expect(helperCount(['baseline', 'knn', 'tree'], 8)).toBe(0); + expect(helperCount(['baseline', 'gbdt'], 8)).toBe(0); + expect(helperCount(['gbdt', 'mlp'], 8)).toBe(2); + }); +}); + +describe('planBatches (V37)', () => { + const heavy = ZOO.filter((key) => familyCost(key) > 1); + + it('never loses a family and never duplicates one', () => { + for (const helpers of [1, 2, 3, 4]) { + const batches = planBatches(heavy, helpers, familyCost); + const flat = batches.flat(); + expect([...flat].sort()).toEqual([...heavy].sort()); + expect(new Set(flat).size).toBe(heavy.length); + } + }); + + it('balances the load — the slowest helper is what the user waits for', () => { + const batches = planBatches(heavy, 4, familyCost); + const loads = batches.map((b) => b.reduce((a, k) => a + familyCost(k), 0)); + // Four heavy families of similar cost: one each, so the spread is small. + expect(Math.max(...loads) - Math.min(...loads)).toBeLessThanOrEqual( + Math.max(...heavy.map(familyCost)), + ); + }); + + it('puts the heaviest family first, alone, when helpers are scarce', () => { + const batches = planBatches(heavy, 2, familyCost); + expect(batches).toHaveLength(2); + // mlp is the most expensive in the V37 measurement — it opens a batch. + expect(batches[0][0]).toBe('mlp'); + }); + + it('is deterministic — the same zoo plans the same way', () => { + expect(planBatches(heavy, 3, familyCost)).toEqual(planBatches(heavy, 3, familyCost)); + }); + + it('returns nothing when asked for no helpers', () => { + expect(planBatches(heavy, 0, familyCost)).toEqual([]); + }); +}); + +describe('the k-NN rule (V37)', () => { + it('keeps the one family that cannot be serialised out of the helpers', () => { + // k-NN has no toJSON (it keeps its training rows) AND costs 0 ms to fit, + // so shipping it to another core would be both impossible and pointless. + expect(NOT_PARALLELISABLE.has('knn')).toBe(true); + expect(HEAVY_FAMILIES).not.toContain('knn'); + expect(familyCost('knn')).toBe(1); + }); +}); diff --git a/src/features/ml/train/parallel.ts b/src/features/ml/train/parallel.ts new file mode 100644 index 0000000..9d2118f --- /dev/null +++ b/src/features/ml/train/parallel.ts @@ -0,0 +1,85 @@ +/** + * V37: the parallel trainer — orchestration only, no maths. + * + * Measured before it was built (PLAN V37, 60 000 rows, 38 400 training rows): + * the zoo takes ~10.4 s sequentially and four families carry 97% of it — + * MLP 3.1 s, logistic 2.7 s, forest 2.5 s, gbdt 1.8 s — while baseline, + * naive Bayes, tree and k-NN together cost about 0.2 s. Four roughly equal + * heavy families is the shape parallelism actually helps: the wall time + * floor is the slowest one, not the sum. + * + * Three rules the design follows: + * + * 1. **k-NN never leaves.** It is the only family with no `toJSON` — it keeps + * its training rows — and it is also the one that fits in 0 ms. It trains + * in the main worker with the other instant families. + * 2. **Models come back as JSON, not as objects.** `predict` is a closure and + * structured clone drops functions, so each helper returns `toJSON()` and + * the caller rebuilds through the V22 path. + * 3. **A failure is never fatal.** If workers cannot be created, a helper + * errors, or a family comes back unserialisable, that family falls back to + * the sequential trainer. Parallelism is an optimisation; it may not + * change which models a run produces. + */ +import type { ModelKey } from '@/features/ml/train/types'; + +/** Families that cost enough to be worth shipping to another core. */ +export const HEAVY_FAMILIES: readonly ModelKey[] = ['mlp', 'logistic', 'forest', 'gbdt', 'linear']; + +/** + * Never more than this many helpers, whatever the machine reports: past the + * number of heavy families the extra workers only cost memory, and leaving a + * core for the UI thread is what keeps the page responsive while training. + */ +export const MAX_HELPERS = 4; + +/** How many helpers to spawn for a given zoo, on this machine. */ +export function helperCount(families: readonly ModelKey[], cores: number): number { + const heavy = families.filter((key) => HEAVY_FAMILIES.includes(key)).length; + if (heavy < 2) return 0; // one heavy family in parallel is just overhead + return Math.max(0, Math.min(MAX_HELPERS, heavy, Math.max(1, cores - 1))); +} + +/** + * Splits families across helpers by measured cost, heaviest first, always to + * the currently lightest helper. Greedy longest-processing-time — the classic + * makespan heuristic, and the right one here: four buckets, known costs. + */ +export function planBatches( + families: readonly ModelKey[], + helpers: number, + cost: (key: ModelKey) => number, +): ModelKey[][] { + if (helpers <= 0) return []; + const batches: ModelKey[][] = Array.from({ length: helpers }, () => []); + const load = new Array(helpers).fill(0); + const ordered = [...families].sort((a, b) => cost(b) - cost(a) || a.localeCompare(b)); + for (const key of ordered) { + let lightest = 0; + for (let i = 1; i < helpers; i++) if (load[i] < load[lightest]) lightest = i; + batches[lightest].push(key); + load[lightest] += cost(key); + } + return batches.filter((batch) => batch.length > 0); +} + +/** + * Relative training cost per family, from the V37 measurement. Only the + * ORDER and rough ratios matter — this feeds the makespan heuristic, not any + * reported number, so it never has to be re-measured to stay correct. + */ +export const MEASURED_COST: Record = { + mlp: 31, + logistic: 27, + forest: 25, + gbdt: 17, + linear: 17, + tree: 2, + naiveBayes: 1, + knn: 1, + baseline: 1, +}; + +export function familyCost(key: ModelKey): number { + return MEASURED_COST[key] ?? 1; +} diff --git a/src/features/ml/train/score.ts b/src/features/ml/train/score.ts index 63e325d..d4329eb 100644 --- a/src/features/ml/train/score.ts +++ b/src/features/ml/train/score.ts @@ -121,8 +121,12 @@ export function scoreRows( X.push(scorer.transformRow(record)); } - const predictions = model.predict(X); - const probabilities = isClassification && model.predictProba ? model.predictProba(X) : null; + // V37: one pass where the family offers one (k-NN) — same numbers, half the + // neighbour searches. See `predictWithProba` on TrainedModel. + const both = model.predictWithProba?.(X) ?? null; + const predictions = both?.labels ?? model.predict(X); + const probabilities = + isClassification && model.predictProba ? (both?.proba ?? model.predictProba(X)) : null; const label = (value: number): string => isClassification ? (classes[value] ?? String(value)) : String(value); diff --git a/src/features/ml/train/trainer.ts b/src/features/ml/train/trainer.ts index a5efd72..0ddb38b 100644 --- a/src/features/ml/train/trainer.ts +++ b/src/features/ml/train/trainer.ts @@ -27,6 +27,16 @@ export interface TrainerCallbacks { isCancelled(): boolean; } +/** + * V37: a family already trained elsewhere (a helper worker), handed back as a + * rebuilt predictor plus its measured scores. The sequential loop then skips + * that family instead of fitting it twice. + */ +export interface PretrainedFamily { + model: TrainedModel; + result: ModelResult; +} + /** Everything kept in worker memory after a run, for insights and what-if. */ export interface TrainArtifacts { models: Map; @@ -103,6 +113,26 @@ export interface PreparedData { * Everything up to the split, shared by training and hyperparameter search so * both see the exact same rows (the search never touches the test indices). */ +/** + * V37: classification or regression, resolved without preparing the data. The + * parallel planner needs the answer before any split happens, and it must not + * be the code that fails first: this returns null where `prepareData` throws a + * named error, so an unusable target is still reported by the trainer that has + * always reported it. + */ +export function detectTaskType( + columns: Map, + profiles: ColumnProfile[], + config: TrainConfig, +): 'classification' | 'regression' | null { + const targetProfile = profiles.find((p) => p.name === config.target); + const targetValues = columns.get(config.target); + if (!targetProfile || !targetValues) return null; + const task = detectTask(targetProfile, targetValues); + if (!task) return null; + return task.type === 'regression' ? 'regression' : 'classification'; +} + export function prepareData( columns: Map, profiles: ColumnProfile[], @@ -309,7 +339,11 @@ export function scoreModel( isClassification: boolean, classCount: number, ): { metrics: MetricMap; primary: number } { - const predictions = model.predict(X); + // V37: one pass where the family offers one — for k-NN, labels and + // probabilities are the same neighbour search, and asking twice doubled the + // single largest cost in the app. The numbers are identical either way. + const both = model.predictWithProba?.(X) ?? null; + const predictions = both?.labels ?? model.predict(X); const metrics: MetricMap = {}; let primary: number; if (isClassification) { @@ -319,7 +353,7 @@ export function scoreModel( metrics.recall = prf.recall; metrics.f1 = prf.f1; if (model.predictProba) { - const probabilities = model.predictProba(X); + const probabilities = both?.proba ?? model.predictProba(X); metrics.logLoss = logLoss(y, probabilities); if (classCount === 2) { const auc = rocAuc( @@ -362,6 +396,8 @@ export async function runTraining( profiles: ColumnProfile[], config: TrainConfig, callbacks: TrainerCallbacks, + /** V37: families already fitted in parallel — skipped by the loop below. */ + pretrained?: Map, ): Promise { const startedAt = performance.now(); const prepared = prepareData(columns, profiles, config); @@ -433,6 +469,26 @@ export async function runTraining( await yieldToQueue(); if (callbacks.isCancelled()) return null; + // V37: a helper already fitted this one — reuse it rather than refit. + const ready = pretrained?.get(def.key); + if (ready !== undefined) { + models.set(def.key, ready.model); + // Latency is the one number a helper cannot report: its timing would + // describe another core under contention. Measure it here, on the + // rebuilt predictor, exactly as the sequential path does — otherwise the + // column would silently read 0 ms for every parallel family. + const latency = measureLatency(ready.model, testX); + const result: ModelResult = { + ...ready.result, + inferP50Ms: latency.p50, + inferP95Ms: latency.p95, + }; + emitted.set(def.key, result); + callbacks.onModelResult(result); + await yieldToQueue(); + continue; + } + try { const cap = MODEL_TRAIN_CAPS[def.key]; let fitX = trainX; diff --git a/src/features/ml/train/types.ts b/src/features/ml/train/types.ts index d36e8f8..8efab93 100644 --- a/src/features/ml/train/types.ts +++ b/src/features/ml/train/types.ts @@ -180,4 +180,10 @@ export interface TrainSummary { classWeighting?: 'balanced'; /** V36: the ensemble's members and the method used to combine them. */ ensemble?: { members: ModelKey[]; method: 'probability' | 'vote' | 'mean' }; + /** + * V37: families that were fitted on other cores, and how many helpers ran. + * Absent when the run was fully sequential — parallelism is announced like + * every other decision, never assumed. + */ + parallel?: { helpers: number; families: ModelKey[] }; } diff --git a/src/features/ml/train/v37.test.ts b/src/features/ml/train/v37.test.ts new file mode 100644 index 0000000..b295a2d --- /dev/null +++ b/src/features/ml/train/v37.test.ts @@ -0,0 +1,214 @@ +/** + * V37 — speed. Every test here defends the same promise: the run got faster + * without a single number moving. + */ +import { describe, expect, it } from 'vitest'; +import { modelZoo, trainKnn } from '@/features/ml/train/models'; +import { rebuildTrainedModel } from '@/features/ml/train/deserialize'; +import { helperCount, planBatches, familyCost } from '@/features/ml/train/parallel'; +import { mulberry32 } from '@/features/ml/train/random'; +import type { ModelContext } from '@/features/ml/train/models'; + +/** + * The implementation V37 replaced, kept verbatim as the oracle: every distance + * in an object, a full stable sort, the first k labels. Slow, obviously right, + * and the only honest way to claim the fast path changed nothing. + */ +function referenceKnn(X: number[][], y: number[], ctx: ModelContext, kWanted: number) { + const k = Math.min(kWanted, X.length); + const neighbors = (row: number[]): number[] => { + const distances = X.map((trainRow, i) => { + let sum = 0; + for (let j = 0; j < row.length; j++) sum += (row[j] - trainRow[j]) ** 2; + return { i, d: sum }; + }); + distances.sort((a, b) => a.d - b.d); + return distances.slice(0, k).map(({ i }) => y[i]); + }; + if (ctx.task === 'regression') { + return { + predict: (rows: number[][]) => + rows.map((row) => { + const near = neighbors(row); + return near.reduce((a, v) => a + v, 0) / near.length; + }), + predictProba: undefined, + }; + } + const proba = (rows: number[][]) => + rows.map((row) => { + const votes = new Array(ctx.classCount).fill(0); + for (const label of neighbors(row)) votes[label] += 1; + return votes.map((v) => v / k); + }); + return { + predict: (rows: number[][]) => proba(rows).map((p) => p.indexOf(Math.max(...p))), + predictProba: proba, + }; +} + +function dataset(rows: number[], width: number, classCount: number, seed: number) { + const rng = mulberry32(seed); + const X: number[][] = []; + const y: number[] = []; + for (let i = 0; i < rows.length; i++) { + X.push(Array.from({ length: width }, () => Math.round(rng() * 6) / 2)); + y.push(rows[i]); + } + return { X, y, classCount }; +} + +describe('V37 — k-NN: faster, and provably the same model', () => { + it('predicts exactly what the sorted implementation predicted', () => { + const labels = Array.from({ length: 300 }, (_, i) => i % 3); + const { X, y } = dataset(labels, 4, 3, 42); + const ctx: ModelContext = { task: 'classification', classCount: 3, seed: 42 }; + const queries = X.slice(0, 120).map((row) => row.map((v) => v + 0.25)); + + const fast = trainKnn(X, y, ctx, 5); + const slow = referenceKnn(X, y, ctx, 5); + expect(fast.predict(queries)).toEqual(slow.predict(queries)); + expect(fast.predictProba!(queries)).toEqual(slow.predictProba!(queries)); + }); + + it('breaks ties the way a stable sort did — the earlier row wins', () => { + // Every training row sits at exactly the same distance from the query, so + // the answer is decided purely by the tie rule. Rows 0..4 are class 0 and + // the rest class 1: a first-seen rule votes 0, any other rule may not. + const X = Array.from({ length: 40 }, () => [1, 1]); + const y = Array.from({ length: 40 }, (_, i) => (i < 5 ? 0 : 1)); + const ctx: ModelContext = { task: 'classification', classCount: 2, seed: 42 }; + const query = [[0, 0]]; + expect(trainKnn(X, y, ctx, 5).predict(query)).toEqual( + referenceKnn(X, y, ctx, 5).predict(query), + ); + expect(trainKnn(X, y, ctx, 5).predict(query)).toEqual([0]); + }); + + it('regresses to the same values', () => { + const targets = Array.from({ length: 200 }, (_, i) => (i % 17) * 1.5); + const { X, y } = dataset(targets, 3, 0, 7); + const ctx: ModelContext = { task: 'regression', classCount: 0, seed: 7 }; + const queries = X.slice(0, 80).map((row) => row.map((v) => v - 0.1)); + expect(trainKnn(X, y, ctx, 5).predict(queries)).toEqual( + referenceKnn(X, y, ctx, 5).predict(queries), + ); + }); + + it('handles k larger than the training set, as before', () => { + const X = [ + [0, 0], + [1, 1], + [2, 2], + ]; + const y = [0, 1, 1]; + const ctx: ModelContext = { task: 'classification', classCount: 2, seed: 1 }; + const queries = [ + [0.1, 0.1], + [5, 5], + ]; + expect(trainKnn(X, y, ctx, 50).predict(queries)).toEqual( + referenceKnn(X, y, ctx, 50).predict(queries), + ); + }); + + it('predictWithProba returns exactly predict and predictProba', () => { + const labels = Array.from({ length: 150 }, (_, i) => i % 4); + const { X, y } = dataset(labels, 5, 4, 99); + const ctx: ModelContext = { task: 'classification', classCount: 4, seed: 99 }; + const queries = X.slice(0, 60); + const model = trainKnn(X, y, ctx, 5); + const both = model.predictWithProba!(queries); + expect(both.labels).toEqual(model.predict(queries)); + expect(both.proba).toEqual(model.predictProba!(queries)); + }); + + it('offers the single-pass path only where it is really one search', () => { + // Regression k-NN has no probabilities at all, so there is nothing to fuse. + const ctx: ModelContext = { task: 'regression', classCount: 0, seed: 3 }; + const model = trainKnn([[0], [1]], [0, 1], ctx, 1); + expect(model.predictWithProba).toBeUndefined(); + expect(model.predictProba).toBeUndefined(); + }); +}); + +describe('V37 — a family crossing the worker boundary', () => { + const rng = mulberry32(42); + const X = Array.from({ length: 600 }, () => [rng(), rng(), rng(), rng()]); + const y = X.map((row) => (row[0] + row[1] > 1 ? 1 : 0)); + const ctx: ModelContext = { task: 'classification', classCount: 2, seed: 42 }; + + /** Every family the helper is allowed to send back, i.e. every one with toJSON. */ + const serialisable = modelZoo('classification') + .map((def) => ({ key: def.key, model: def.train(X, y, ctx) })) + .filter((entry) => entry.model.toJSON !== undefined); + + it('covers the whole zoo except k-NN', () => { + expect(serialisable.map((e) => e.key).sort()).toEqual([ + 'baseline', + 'forest', + 'gbdt', + 'logistic', + 'mlp', + 'naiveBayes', + 'tree', + ]); + }); + + it.each(serialisable.map((e) => [e.key, e] as const))( + 'rebuilds %s from JSON and predicts exactly what the original predicted', + (_key, entry) => { + const json = entry.model.toJSON!() as { kind: string }; + // The protocol: a JSON string across postMessage, parsed on the far side. + const rebuilt = rebuildTrainedModel(json.kind, JSON.parse(JSON.stringify(json)), true); + expect(rebuilt.predict(X.slice(0, 100))).toEqual(entry.model.predict(X.slice(0, 100))); + }, + ); + + it('stays exportable after the round trip — parallelism takes no feature away', () => { + // A family fitted in a helper is a normal leaderboard row: the export + // button must still work on it. The rebuild path itself defines no + // `toJSON` (an imported model has no reason to be re-exported), so + // `rebuildTrainedModel` re-attaches the parameters it was handed. + for (const entry of serialisable) { + const json = entry.model.toJSON!() as { kind: string }; + const params = JSON.parse(JSON.stringify(json)); + const rebuilt = rebuildTrainedModel(json.kind, params, true); + expect(rebuilt.toJSON).toBeDefined(); + // Compared as written to the file, not as objects: the originals carry + // ml-cart's TreeNode prototypes and the round trip carries plain ones, + // which is exactly the difference an export erases anyway. + expect(JSON.stringify(rebuilt.toJSON!())).toBe(JSON.stringify(json)); + } + }); + + it('refuses to travel as a structured clone — the bug this protocol avoids', () => { + // structuredClone keeps shapes JSON drops, and ml-cart's `load()` then + // builds a tree whose `classify` returns a plain object instead of a + // matrix: the first prediction throws. This test exists so nobody + // "simplifies" the protocol back to posting the object directly. + const tree = serialisable.find((e) => e.key === 'tree')!; + const json = tree.model.toJSON!() as { kind: string }; + const cloned = rebuildTrainedModel(json.kind, structuredClone(json), true); + expect(() => cloned.predict(X.slice(0, 1))).toThrow(); + }); +}); + +describe('V37 — planning the helpers', () => { + it('spawns nothing when a single heavy family would run alone', () => { + expect(helperCount(['knn', 'tree', 'mlp'], 8)).toBe(0); + }); + + it('never asks for more helpers than the machine has cores to spare', () => { + expect(helperCount(['mlp', 'logistic', 'forest', 'gbdt'], 2)).toBe(1); + expect(helperCount(['mlp', 'logistic', 'forest', 'gbdt'], 16)).toBe(4); + }); + + it('balances the batches by measured cost, heaviest first', () => { + const batches = planBatches(['mlp', 'logistic', 'forest', 'gbdt', 'tree'], 2, familyCost); + const loads = batches.map((batch) => batch.reduce((total, key) => total + familyCost(key), 0)); + // Greedy LPT on 31/27/25/17/2 splits 58 against 44 — under a fifth apart. + expect(Math.abs(loads[0] - loads[1]) / Math.max(...loads)).toBeLessThan(0.25); + expect(batches.flat().sort()).toEqual(['forest', 'gbdt', 'logistic', 'mlp', 'tree']); + }); +}); diff --git a/src/locales/en.json b/src/locales/en.json index 10dac22..e4e47b2 100644 --- a/src/locales/en.json +++ b/src/locales/en.json @@ -310,7 +310,9 @@ "weighting": "class weighting: balanced ({{loss}} weight the loss, {{resample}} use a seeded balanced resample)", "imbalanceHint": "The largest class holds {{share}}% of the training rows. Accuracy rewards always answering it — rank on F1 or recall, and try class weighting.", "weightToggle": "Balance the classes", - "weightToggleHint": "Weights the training loss (logistic regression, gradient boosting) or resamples the rows (tree, forest) so the rare class counts as much as the common one." + "weightToggleHint": "Weights the training loss (logistic regression, gradient boosting) or resamples the rows (tree, forest) so the rare class counts as much as the common one.", + "parallel_one": "{{count}} helper core: {{families}} trained in parallel", + "parallel_other": "{{count}} helper cores: {{families}} trained in parallel" }, "models": { "baseline": "Naive baseline", @@ -485,7 +487,16 @@ "intervalsTitle": "Winner intervals (95%, from each run)", "intervalsDisjoint": "The intervals do not overlap — the gap between the two runs exceeds both uncertainties. Probably real.", "intervalsOverlap": "The intervals overlap — on these test sets, the gap between the two runs could be noise.", - "intervalsNote": "Each interval comes from its own run's test draw (v20). Two runs are never paired — read this as an indication, not a test." + "intervalsNote": "Each interval comes from its own run's test draw (v20). Two runs are never paired — read this as an indication, not a test.", + "openMany": "Compare these {{count}} runs", + "manyTitle": "Session comparison", + "manyHint": "Every run is read against {{reference}}, the oldest of the selection — so the deltas say what your changes did, not what the newest run happens to be.", + "manyIncomparable": "These runs do not predict the same thing — the numbers are shown, but they are not deltas. Read each column on its own.", + "reference": "reference", + "champion": "Best model", + "manyShared_one": "{{count}} feature present in every run: {{columns}}", + "manyShared_other": "{{count}} features present in every run: {{columns}}", + "manyMissing": "Not enough runs to compare — select between 2 and 6 saved runs from the lab history." }, "imported": { "title": "Reuse an exported model", diff --git a/src/locales/fr.json b/src/locales/fr.json index 68e30a0..14a0789 100644 --- a/src/locales/fr.json +++ b/src/locales/fr.json @@ -310,7 +310,9 @@ "weighting": "pondération de classes : équilibrée ({{loss}} pondèrent la perte, {{resample}} utilisent un rééchantillonnage équilibré seedé)", "imbalanceHint": "La classe majoritaire occupe {{share}} % des lignes d'entraînement. L'accuracy récompense le fait de toujours la répondre — classez sur F1 ou rappel, et essayez la pondération de classes.", "weightToggle": "Équilibrer les classes", - "weightToggleHint": "Pondère la perte à l'entraînement (régression logistique, gradient boosting) ou rééchantillonne les lignes (arbre, forêt) pour que la classe rare compte autant que la commune." + "weightToggleHint": "Pondère la perte à l'entraînement (régression logistique, gradient boosting) ou rééchantillonne les lignes (arbre, forêt) pour que la classe rare compte autant que la commune.", + "parallel_one": "{{count}} cœur d'appoint : {{families}} entraînés en parallèle", + "parallel_other": "{{count}} cœurs d'appoint : {{families}} entraînés en parallèle" }, "models": { "baseline": "Baseline naïve", @@ -485,7 +487,16 @@ "intervalsTitle": "Intervalles des gagnants (95 %, propres à chaque run)", "intervalsDisjoint": "Les intervalles ne se recouvrent pas — l'écart entre les deux runs dépasse leurs deux incertitudes. Probablement réel.", "intervalsOverlap": "Les intervalles se recouvrent — sur ces jeux de test, l'écart entre les deux runs peut être du bruit.", - "intervalsNote": "Chaque intervalle vient du tirage de test de son propre run (v20). Deux runs ne sont jamais appariés — à lire comme une indication, pas un test." + "intervalsNote": "Chaque intervalle vient du tirage de test de son propre run (v20). Deux runs ne sont jamais appariés — à lire comme une indication, pas un test.", + "openMany": "Comparer ces {{count}} runs", + "manyTitle": "Comparaison de session", + "manyHint": "Chaque run est lu par rapport à {{reference}}, le plus ancien de la sélection — les écarts disent donc ce qu'ont donné vos changements, pas ce que vaut le dernier run.", + "manyIncomparable": "Ces runs ne prédisent pas la même chose — les chiffres sont affichés, mais ce ne sont pas des écarts. Lisez chaque colonne pour elle-même.", + "reference": "référence", + "champion": "Meilleur modèle", + "manyShared_one": "{{count}} variable présente dans tous les runs : {{columns}}", + "manyShared_other": "{{count}} variables présentes dans tous les runs : {{columns}}", + "manyMissing": "Pas assez de runs à comparer — sélectionnez entre 2 et 6 runs enregistrés dans l'historique du lab." }, "imported": { "title": "Réutiliser un modèle exporté",