feat(v35): ML Lab — le chiffre cesse de se flatter lui-même

README.md

@@ -26,8 +26,11 @@ The project follows three non-negotiable principles:
   pinned to the served header by a unit test, so the page cannot claim a protection the
   site stopped shipping.
 - **Honest evaluation.** Every run is scored against a naive baseline on a held-out test
-  set. Metrics ship with 95% bootstrap intervals, per-segment breakdowns, calibration
-  curves, and explicit refusals when a number would be noise (tiny slices, tiny test sets).
+  set. Models are **selected on a validation split and reported on a third, never-selected
+  test split**, with the gap between the two shown — crowning the best of nine on the
+  reporting set is what makes a headline figure optimistic. Metrics ship with 95% bootstrap
+  intervals, per-segment breakdowns, calibration curves, and explicit refusals when a number
+  would be noise (tiny slices, tiny test sets, a model whose probabilities are saturated).
 - **Hand-written, deterministic ML.** The model zoo, search, explanations, and statistics
   are implemented from scratch in TypeScript, seeded end to end — the same seed always
   reproduces the same run.
@@ -99,7 +102,10 @@ The project follows three non-negotiable principles:
   post-processing (YOLOX grid decode, IoU, non-maximum suppression).
 - **Leakage discipline.** Preprocessing (imputation, one-hot/ordinal encoding,
   standardization) is fitted on the training split only; cross-validation refits the
-  pipeline inside each fold; forecast backtests never peek at the future.
+  pipeline inside each fold; forecast backtests never peek at the future. Dated files can
+  be split **chronologically** and grouped files **by group**, both announced — a random
+  split puts the future in training. A one-column stump flags any lone column that predicts
+  the target at 99%: that is a leak warning, never a victory.
 - **Determinism.** A single seed drives splits, model initialization, search, sampling
   and resampling — runs are exactly reproducible, and the test suite depends on it.
 - **Scale, honestly.** 100k–1M-row files train comfortably: past 100 000 usable rows an
@@ -110,7 +116,7 @@ The project follows three non-negotiable principles:
 - **Performance.** Every section serves a prerendered static shell (hero paints before
   JavaScript); Lighthouse mobile ≈ 0.99 on `/ml` under real throttling. Heavy
   dependencies (Dexie, SheetJS, ONNX Runtime) load lazily.
-- **Quality bar.** 352 unit tests, 61 Playwright end-to-end tests (including offline PWA,
+- **Quality bar.** 369 unit tests, 65 Playwright end-to-end tests (including offline PWA,
   fake-webcam and axe-core WCAG A/AA accessibility checks), strict TypeScript, ESLint,
   Prettier, and Lighthouse budgets — all enforced in CI.
 

e2e/split.spec.ts

@@ -0,0 +1,87 @@
+import { expect, test } from '@playwright/test';
+
+test.use({ locale: 'en-US' });
+test.setTimeout(120_000);
+
+// V35: the number stops flattering itself. Four things the user must SEE:
+// the third split, the champion's selection-vs-test gap, an announced
+// chronological split on dated data, and a lone column caught reading the
+// target. Plus the 5x2 verdict on whether the ranking is real.
+
+test('iris: the winner is picked on validation and reports its test gap', async ({ page }) => {
+  await page.goto('/ml');
+  await page.getByRole('button', { name: /iris\.csv/ }).click();
+  await expect(page.getByText('150 rows · 5 columns')).toBeVisible();
+  await page.selectOption('#target-select', 'species');
+  await page.getByTestId('train-button').click();
+  await expect(page.getByTestId('train-again')).toBeVisible({ timeout: 60_000 });
+
+  const leaderboard = page.getByTestId('leaderboard');
+  // The ranked column is now the validation metric, with test beside it.
+  await expect(leaderboard).toContainText('Accuracy (val)');
+  await expect(leaderboard).toContainText('Test');
+  // Three splits are announced, not two.
+  await expect(leaderboard).toContainText(/validation rows/);
+
+  // The champion line names both numbers and the gap between them.
+  const gap = page.getByTestId('champion-gap');
+  await expect(gap).toBeVisible();
+  // Iris is separable enough that the top models reach 1.000 — the assertion
+  // pins the SHAPE of the sentence (two figures and a gap), not the values.
+  await expect(gap).toContainText(/was selected on validation at \d\.\d{3} and scores \d\.\d{3}/);
+  await expect(gap).toContainText('never-selected split');
+});
+
+test('iris: 5x2 cross-validation says whether the ranking is real', async ({ page }) => {
+  await page.goto('/ml');
+  await page.getByRole('button', { name: /iris\.csv/ }).click();
+  await page.selectOption('#target-select', 'species');
+  await page.getByTestId('train-button').click();
+  await expect(page.getByTestId('train-again')).toBeVisible({ timeout: 60_000 });
+
+  await page.getByTestId('robust-run').click();
+  const verdict = page.getByTestId('robust-verdict');
+  await expect(verdict).toBeVisible({ timeout: 90_000 });
+  // Either wording is correct — what matters is that it counts the folds.
+  await expect(verdict).toContainText(/of 10 folds/);
+  // The panel says out loud that the test set stayed out of the folds.
+  await expect(page.getByTestId('robust-rank')).toContainText('test set is never touched');
+});
+
+test('titanic: re-including the mirrored column raises a named leak warning', async ({ page }) => {
+  await page.goto('/ml');
+  await page.getByRole('button', { name: /titanic\.csv/ }).click();
+  await expect(page.getByText('891 rows · 15 columns')).toBeVisible();
+  await page.selectOption('#target-select', 'survived');
+
+  // V6 excludes `alive` automatically; the user overrides that decision.
+  await page.getByTestId('column-card-alive').getByRole('button', { name: 'Include' }).click();
+  await page.getByTestId('train-button').click();
+  await expect(page.getByTestId('train-again')).toBeVisible({ timeout: 90_000 });
+
+  // V35 catches it again at training time — with a measured number.
+  const warning = page.getByTestId('leak-warning');
+  await expect(warning).toBeVisible();
+  await expect(warning).toContainText('« alive » alone predicts the target at 100.0%');
+  await expect(warning).toContainText('almost always leakage');
+});
+
+test('energy: a dated file offers — and announces — a chronological split', async ({ page }) => {
+  await page.goto('/ml');
+  await page.getByRole('button', { name: /energy\.csv/ }).click();
+  await expect(page.getByText('240 rows · 3 columns')).toBeVisible();
+  await page.selectOption('#target-select', 'kwh');
+
+  // The option exists because `date` is a date column — and only then.
+  const split = page.getByTestId('split-mode');
+  await expect(split).toBeVisible();
+  await split.selectOption('chronological:date');
+
+  await page.getByTestId('train-button').click();
+  await expect(page.getByTestId('train-again')).toBeVisible({ timeout: 60_000 });
+
+  // Announced in the run info, like every other decision the lab makes.
+  await expect(page.getByTestId('leaderboard')).toContainText(
+    'chronological split on date (oldest rows train, newest test)',
+  );
+});

e2e/text.spec.ts

@@ -26,10 +26,17 @@ test('reviews: the text column trains, and the words explain the model', async (
   await expect(page.getByTestId('insights')).toBeVisible({ timeout: 30_000 });
   await expect(page.getByTestId('importance')).toContainText('review');
 
-  // The words card: signed effects, one word per row, with the honest note.
+  // V35: on this file the crowned model is Gaussian Naive Bayes, whose
+  // probabilities saturate to 0/1 — occlusion then measures exactly zero.
+  // The card must SAY that rather than vanish, and point at a way forward.
   const words = page.getByTestId('word-effects');
   await expect(words).toBeVisible();
-  await expect(words).toContainText('Words that move the answer');
+  await expect(words).toContainText('its probabilities are saturated');
+  await expect(words).toContainText('Pick another model in the leaderboard');
+
+  // And on a model that gives graded probabilities, the words do speak.
+  await page.getByTestId('leaderboard').getByText('Gradient boosting').click();
+  await expect(words).toContainText('Words that move the answer', { timeout: 30_000 });
   // Effects are signed — at least one word pushes each way on this dataset.
   await expect(words).toContainText('+');
   await expect(words).toContainText('−');
@@ -47,5 +54,8 @@ test('reviews in French: the words card speaks French too', async ({ page }) =>
 
   const words = page.getByTestId('word-effects');
   await expect(words).toBeVisible({ timeout: 30_000 });
-  await expect(words).toContainText('Les mots qui font bouger la réponse');
+  // Le refus est traduit lui aussi — une carte qui se tait n'apprend rien.
+  await expect(words).toContainText('ses probabilités sont saturées');
+  await page.getByTestId('leaderboard').getByText('Gradient boosting').click();
+  await expect(words).toContainText('Les mots qui font bouger la réponse', { timeout: 30_000 });
 });

src/features/ml/components/LeaderboardTable.tsx

@@ -1,6 +1,7 @@
-import { Eye } from 'lucide-react';
+import { AlertTriangle, Eye } from 'lucide-react';
 import { useTranslation } from 'react-i18next';
 import { Badge } from '@/components/ui/badge';
+import { championGap, rankingValue, sortResults } from '@/features/ml/train/ranking';
 import type { TaskType } from '@/features/ml/data/types';
 import type { ModelResult, TrainSummary } from '@/features/ml/train/types';
 import { cn } from '@/lib/utils';
@@ -26,6 +27,13 @@ interface LeaderboardTableProps {
  * Presentational model ranking — used live in the lab and read-only in
  * stored/shared run views. Classification ranks by accuracy (higher wins),
  * regression by RMSE (lower wins), with the delta vs baseline made explicit.
+ *
+ * V35: when a run carries validation scores, the table ranks and crowns on
+ * the VALIDATION metric and shows the test metric beside it — selecting on
+ * the reporting set made the crowned number the optimistic max of nine
+ * draws. The champion line spells out the val→test gap: that gap is the
+ * most useful lesson the lab can teach. Runs stored before V35 carry no
+ * validation scores and keep their historical, test-ranked display.
  */
 export function LeaderboardTable({
   results,
@@ -37,14 +45,17 @@ export function LeaderboardTable({
   const { t, i18n } = useTranslation();
   const lang = i18n.resolvedLanguage ?? 'en';
   const isClassification = taskType !== 'regression';
-  const ok = results.filter((r) => r.ok);
   const failed = results.filter((r) => !r.ok);
-  const sorted = [...ok].sort((a, b) =>
-    isClassification ? b.primary - a.primary : a.primary - b.primary,
-  );
-  const baseline = ok.find((r) => r.key === 'baseline');
+  const sorted = sortResults(results, taskType);
+  const baseline = sorted.find((r) => r.key === 'baseline');
   const bestKey = sorted[0]?.key;
-  const maxPrimary = Math.max(...ok.map((r) => r.primary), 1e-9);
+  const hasValidation = sorted.some((r) => r.valPrimary !== undefined);
+  const champion = championGap(results, taskType);
+  const maxPrimary = Math.max(...sorted.map((r) => rankingValue(r)), 1e-9);
+  const leakWarnings = summary?.leakWarnings ?? [];
+
+  const metricsOf = (result: ModelResult) =>
+    hasValidation ? (result.valMetrics ?? result.metrics) : result.metrics;
 
   const metricColumns: { key: keyof ModelResult['metrics']; label: string }[] = isClassification
     ? [
@@ -60,25 +71,51 @@ export function LeaderboardTable({
   function delta(result: ModelResult): string {
     if (!baseline || result.key === 'baseline') return '—';
     const value = isClassification
-      ? result.primary - baseline.primary
-      : baseline.primary - result.primary;
+      ? rankingValue(result) - rankingValue(baseline)
+      : rankingValue(baseline) - rankingValue(result);
     const sign = value > 0 ? '+' : '';
     return `${sign}${value.toFixed(3)}`;
   }
 
+  const primaryHeader = isClassification
+    ? t(hasValidation ? 'ml.lab.leaderboard.accuracyVal' : 'ml.lab.leaderboard.accuracy')
+    : t(hasValidation ? 'ml.lab.leaderboard.rmseVal' : 'ml.lab.leaderboard.rmse');
+
   return (
     <div
       data-testid="leaderboard"
       className="overflow-x-auto rounded-2xl border border-line bg-surface"
     >
+      {leakWarnings.length > 0 && (
+        <div
+          data-testid="leak-warning"
+          className="flex items-start gap-2 border-b border-line bg-copper-soft px-3 py-2.5 text-sm"
+        >
+          <AlertTriangle className="mt-0.5 h-4 w-4 shrink-0 text-copper" aria-hidden="true" />
+          <div>
+            {leakWarnings.map((warning) => (
+              <p key={warning.column}>
+                {t('ml.lab.leaderboard.leakWarning', {
+                  column: warning.column,
+                  score: (warning.score * 100).toFixed(1),
+                })}
+              </p>
+            ))}
+            <p className="text-xs text-muted">{t('ml.lab.leaderboard.leakAdvice')}</p>
+          </div>
+        </div>
+      )}
       <table className="w-full text-left text-sm">
         <thead>
           <tr className="bg-surface-2 font-mono text-[0.68rem] tracking-wider uppercase">
             <th className="px-3 py-2 font-medium">#</th>
             <th className="px-3 py-2 font-medium">{t('ml.lab.leaderboard.model')}</th>
-            <th className="px-3 py-2 font-medium">
-              {isClassification ? t('ml.lab.leaderboard.accuracy') : t('ml.lab.leaderboard.rmse')}
-            </th>
+            <th className="px-3 py-2 font-medium">{primaryHeader}</th>
+            {hasValidation && (
+              <th className="px-3 py-2 font-medium" title={t('ml.lab.leaderboard.testTitle')}>
+                {t('ml.lab.leaderboard.test')}
+              </th>
+            )}
             <th className="px-3 py-2 font-medium">{t('ml.lab.leaderboard.delta')}</th>
             {metricColumns.map(({ key, label }) => (
               <th key={key} className="px-3 py-2 font-medium">
@@ -121,13 +158,16 @@ export function LeaderboardTable({
                     <span className="h-1.5 w-20 shrink-0 overflow-hidden rounded-full bg-surface-2">
                       <span
                         className="block h-full rounded-full bg-accent/75"
-                        style={{ width: `${(result.primary / maxPrimary) * 100}%` }}
+                        style={{ width: `${(rankingValue(result) / maxPrimary) * 100}%` }}
                       />
                     </span>
                   )}
-                  <span className="font-medium">{formatMetric(result.primary)}</span>
+                  <span className="font-medium">{formatMetric(rankingValue(result))}</span>
                 </span>
               </td>
+              {hasValidation && (
+                <td className="px-3 py-2 text-muted">{formatMetric(result.primary)}</td>
+              )}
               <td
                 className={cn(
                   'px-3 py-2',
@@ -138,7 +178,7 @@ export function LeaderboardTable({
               </td>
               {metricColumns.map(({ key }) => (
                 <td key={key} className="px-3 py-2">
-                  {formatMetric(result.metrics[key])}
+                  {formatMetric(metricsOf(result)[key])}
                 </td>
               ))}
               <td className="px-3 py-2 font-mono text-xs whitespace-nowrap">
@@ -178,27 +218,67 @@ export function LeaderboardTable({
                   <Badge variant="copper">{t('ml.lab.leaderboard.failed')}</Badge>
                 </span>
               </td>
-              <td className="px-3 py-2" colSpan={4 + metricColumns.length}>
+              <td
+                className="px-3 py-2"
+                colSpan={4 + metricColumns.length + (hasValidation ? 1 : 0)}
+              >
                 <span className="font-mono text-xs">{result.error}</span>
               </td>
             </tr>
           ))}
         </tbody>
       </table>
+      {champion && (
+        <p data-testid="champion-gap" className="border-t border-line px-3 py-2 text-xs">
+          {t('ml.lab.leaderboard.championLine', {
+            model: t(`ml.lab.models.${champion.model.key}`),
+            val: formatMetric(champion.val),
+            test: formatMetric(champion.test),
+            gap: `${champion.gap > 0 ? '+' : ''}${champion.gap.toFixed(3)}`,
+          })}{' '}
+          <span className="text-muted">{t('ml.lab.leaderboard.championWhy')}</span>
+        </p>
+      )}
       {summary && (
         <p className="border-t border-line px-3 py-2 font-mono text-[0.68rem] text-muted">
-          {t('ml.lab.leaderboard.runInfo', {
-            seed: summary.seed,
-            train: summary.trainRows,
-            test: summary.testRows,
-            features: summary.featureCount,
-          })}
+          {summary.validationRows !== undefined
+            ? t('ml.lab.leaderboard.runInfoVal', {
+                seed: summary.seed,
+                train: summary.trainRows,
+                val: summary.validationRows,
+                test: summary.testRows,
+                features: summary.featureCount,
+              })
+            : t('ml.lab.leaderboard.runInfo', {
+                seed: summary.seed,
+                train: summary.trainRows,
+                test: summary.testRows,
+                features: summary.featureCount,
+              })}
+          {summary.split !== undefined && (
+            <>
+              {' '}
+              ·{' '}
+              {t(
+                summary.split.mode === 'chronological'
+                  ? 'ml.lab.leaderboard.splitChronological'
+                  : 'ml.lab.leaderboard.splitGroup',
+                { column: summary.split.column },
+              )}
+              {summary.split.dropped !== undefined &&
+                ` ${t('ml.lab.leaderboard.splitDropped', { count: summary.split.dropped })}`}
+            </>
+          )}
           {summary.sampledFrom !== undefined && (
             <>
               {' '}
               ·{' '}
               {t('ml.lab.leaderboard.sampledFrom', {
-                cap: (summary.trainRows + summary.testRows).toLocaleString(lang),
+                cap: (
+                  summary.trainRows +
+                  (summary.validationRows ?? 0) +
+                  summary.testRows
+                ).toLocaleString(lang),
                 from: summary.sampledFrom.toLocaleString(lang),
               })}
             </>