Skip to content

fix(promotion): refuse to call a candidate cheaper on dollars nobody measured - #949

Merged
drewstone merged 1 commit into
mainfrom
fix/promotion-gate-refuses-unknown-cost
Aug 21, 2026
Merged

fix(promotion): refuse to call a candidate cheaper on dollars nobody measured#949
drewstone merged 1 commit into
mainfrom
fix/promotion-gate-refuses-unknown-cost

Conversation

@drewstone

Copy link
Copy Markdown
Contributor

Problem

BenchmarkCell (run-benchmark.ts:77) carries usd: number and usdKnown: boolean as required siblings. promotionGate read usd from both arms (:88-89) and never read usdKnown — zero occurrences in the file.

In non-inferiority mode a significant paired cost saving is a promotion criterion (:151): "(b) cost SAVINGS (incumbent − candidate, usd/task) significantly > 0". So a candidate whose dollars were a catalog estimate or an unreported floor could be promoted for being cheaper than a receipted incumbent — comparing a number no provider ever billed against one that was.

Spend.usdKnown states the rule this gate has to obey, in its own words: "A false value must not be treated as $0 when enforcing a dollar-denominated comparison or limit." Promotion on cost savings is that comparison. budget.ts already refuses unknown dollars under a dollar cap; this gate did not.

Measured: with one of 24 paired tasks marked usdKnown: false on the candidate arm, the gate returned promoted: true, reason: 'non-inferior-and-cheaper'.

Change

Non-inferiority mode collects the tasks where either arm reports usdKnown === false and refuses before running the cost bootstrap, with a new reason: 'cost-unknown' and a costUnknownTasks list naming them.

Why long-term right

This is the refusal the type was already asking for. The alternative — comparing anyway and hoping the estimate is close — is precisely the failure mode usdKnown exists to make impossible, and it fails in the expensive direction: a candidate looks cheaper because its cost went unmeasured. Naming the offending tasks makes the refusal actionable: it says which evidence to go and measure, rather than returning a bare no.

superiority mode is untouched, because it never reads dollars.

Cost

3 files. This is a consumer-visible behaviour change: a caller that exhaustively switches on PromotionVerdict.reason must handle 'cost-unknown', and a non-inferiority promotion built on unmeasured dollars now refuses. Version bumped 0.153.1 → 0.154.0 with a CHANGELOG entry and the docs/canonical-api.md pin.

Worth flagging: check:version-bump reported "consumer surface unchanged" both before and after the bump — it compares export names, not their shapes, so it does not see a union gaining a member. The bump here is a judgement call, not a gate result.

Proof

  • pnpm run typecheck, pnpm run lint (613 files), pnpm run docs:check, pnpm run check:version-bump — clean.
  • tests/kernel/strategy-suite.test.ts — 38 passed, including the existing four non-inferiority cases, which still promote/refuse exactly as before.
  • New test verified to FAIL without the refusal: with the guard disabled the same fixture yields promoted: true (expected true to be false).
  • src/improvement/improve.test.ts shows 4 failures; that file is first on the clean-origin/main baseline failing-file list (the documented macOS git-worktree class) and is untouched here.

Simplification

Simplification: none — this is a net addition, and the addition is irreducible. The measurement that found it was the usd/usdKnown flag-pair census (187 usdKnown occurrences over 30 files, 56 read sites guarding on it, and 10 files that read .usd without ever mentioning usdKnown); this is the one of those ten where the unguarded read decides a promotion. Collapsing the flag pair into a cost discriminated union would make the omission unrepresentable rather than caught by hand, but that migration touches 143 files and remains out of scope.
Net: +38 / -2 lines, 3 files, 0 copies removed.
Not done here: waterfall.ts:86 is the other consequential unguarded read — span.usd = p.spent?.usd ?? 0, rendered as $0.0000. That one is a display concern whose fix is to label the number unmeasured rather than to refuse, so it is a different change with a different test.
Tests: +1 (a candidate cannot be promoted as cheaper when an arm's dollars were never measured — the exact defect, promotes without this change), -0 deleted.

@drewstone
drewstone force-pushed the fix/promotion-gate-refuses-unknown-cost branch from 9122439 to 2784ddb Compare August 21, 2026 10:41
tangletools
tangletools previously approved these changes Aug 21, 2026

@tangletools tangletools left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Auto-approved drewstone PR — 2784ddbb

This PR was opened by the trusted drewstone account.

This approval is provisional and was applied by the local stand-in because the pr-reviewer webhook host is unreachable (2026-08-21). CI on this head is fully green. The full PR reviewer audit re-runs via the resweep when the service returns and will publish findings if it detects issues.

…measured

BenchmarkCell carries usd and usdKnown as required siblings. promotionGate read
usd at both arms and never read usdKnown, while non-inferiority mode promotes on
a significant paired cost saving. A candidate whose dollars were a catalog
estimate or an unreported floor could therefore be promoted for being cheaper
than a receipted incumbent.

Spend.usdKnown states the rule this gate has to obey: a false value must not be
treated as a measurement when enforcing a dollar-denominated comparison or
limit. Promotion on cost savings is that comparison.

The verdict now carries cost-unknown and names the tasks that caused it, so the
refusal says which evidence to go and measure.
@drewstone
drewstone force-pushed the fix/promotion-gate-refuses-unknown-cost branch from 2784ddb to 9ab6064 Compare August 21, 2026 10:50

@tangletools tangletools left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Auto-approved drewstone PR — 9ab60645

This PR was opened by the trusted drewstone account.

This approval is provisional and was applied by the local stand-in because the pr-reviewer webhook host is unreachable (2026-08-21). CI on this head is fully green. The full PR reviewer audit re-runs via the resweep when the service returns and will publish findings if it detects issues.

@drewstone
drewstone merged commit 7f05cdb into main Aug 21, 2026
4 checks passed
@drewstone
drewstone deleted the fix/promotion-gate-refuses-unknown-cost branch August 21, 2026 10:56
drewstone added a commit that referenced this pull request Aug 21, 2026
check:version-bump compared export names only, so it reported "consumer
surface unchanged" for #946, which removed WorkerView.cwd from a public
interface, and for #949, which added a member to the public
PromotionVerdict.reason union. Neither moved a name, so api-surface.json was
byte-identical across both and the 0.154.0 bump was a human judgement rather
than a gate result.

The record now states a shape digest beside each kind, taken over the built
declaration with comments removed, whitespace collapsed, the declaration's own
local name blanked, and every type reference rewritten to a stable token. A
reference to a symbol this package exports contributes that symbol's public
name, so an edit is reported once, on the line of the symbol that changed.

A shape move is classified breaking: telling an added optional field from a
removed required one is a subtyping question, and the record states structure.

Refs #946, #949
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants