feat(speculation): generator contract and bestfirst impl

[behinddwalls]

Add submitqueue/extension/speculation/generator, the candidate-stream composition point (Generator/PathIterator) the standard Speculator pulls from, plus the bestfirst implementation and mocks.

Open takes only the queue's batches. The generator offers every coherent path in the space, including paths whose builds already ran — suppressing finished paths belongs to the Allocator, the piece that already reconciles candidates against the stored path sets by ID. Keeping the generator ignorant of path sets leaves it a pure enumerator and removes the expand-before-skip subtlety from its iterator: Next pops, expands, returns, with no filtering loop. Ranking stays implementation-defined at the contract; only bestfirst commits to a scale.

bestfirst ranks each path by how likely all its assumptions are to hold, and names paths long before it builds them. Each head roots a tree of its paths, and a single heap holds the frontier of every head's tree at once — as nodes, a node being a head plus the ascending set of alternatives that path takes rather than a built path. Handing a node out builds that one path and adds its one or two children. Pulling k paths therefore builds exactly k paths and sorts at most k heads' alternatives, however large the space behind them and however many heads the queue holds.

Scores are summed log probabilities, which preserves ordering without underflow. Multiplying the probabilities instead loses the ordering outright: two heads waiting on 1200 and 1300 coin-flip dependencies both round to exactly 0.0, tie, and the heap falls back to its tie-break — so it can emit the 1300-dependency path first, one that is 2^100 times less probable. The ranking does not degrade, it inverts. Summing logs keeps the arithmetic in range at any depth, preserves the order because log is monotonic, and gives an assumption that cannot happen a score of negative infinity, which compares correctly against every finite score. A wide-head test pins this, asserting both that two 7100-dependency heads order correctly and that exponentiating either score yields zero.

Scores from the injected scorer are validated on the way in. Everything downstream treats a score as a probability — its log is the ranking key, its complement is the other side's probability — so a value outside [0, 1] would not fail loudly, it would quietly produce a positive log, an inverted ordering, or a NaN that makes every comparison false.

Each unresolved dependency has a preferred assumption — the one with the higher probability — and an unpreferred one. The head's best path takes preferred everywhere and roots the tree; every other node takes some subset of the head's alternatives, each adding its relativeScore. Subsets are walked by a canonical expansion under which every subset has exactly one parent, so each path is reached exactly once with no visited set, and a child never outranks its parent — which, since a node is the heap's maximum when it is popped, is what makes the emitted sequence non-increasing.

A node's score is always summed from its head's best-path score in ascending alternative order, never carried over from the parent it grew out of. The two are not the same number: advancing {0,1} to {0,2} gives bestPathScore+r0+r2 summed against (s-r1)+r2 carried, and floating-point addition does not associate. Carrying would break the ordering guarantee silently and only on some inputs, so a test pins every emitted score to its canonical sum.

Equal scores break on depth first, then on the alternatives taken, then on the head. Depth first offers every head's root before any head's deeper nodes, which matters because a dependency at probability exactly 0.5 deviates for free: ordering on the head instead would let one head's tied subtree drain a caller's build budget before another head's root was offered at all. Comparing the alternatives before the head then interleaves heads at equal depth, since position i is the i-th cheapest deviation for whichever head it belongs to. This replaces a tie-break on path ID, which needed a hash of every path the moment it entered the heap.

Everything a dependency contributes — which assumption is preferred, what the preferred side scores, what deviating costs — follows from that batch alone, so it is worked out once per run rather than once per head waiting on it. Since entity.Batch.Dependencies holds a transitive closure, a chain of N batches has O(N^2) edges over N distinct dependencies, and the difference is not marginal: on a 3000-head chain Open drops from roughly 1.0s and 953MB to roughly 0.1s and 2MB. What Open cannot defer is the scoring itself, because ranking heads against each other needs every head's best-path score up front.

Open holds the batches and their dependency slices rather than copying them, for as long as the iterator lives. The Generator contract now says so, since it constrains every caller and every implementation even though no signature changed. It is consistent with batches being replaced rather than edited in place, and it is what lets a head cost a struct rather than a slice per dependency.

Generation honors context cancellation. Open checks before each head and newHead before each dependency, since a head can carry hundreds; Next checks before each pull, which is where a caller that has stopped consuming stops paying. All return the context error with no iterator and no candidate. Nothing after the heap pop can fail, so a node is never taken off the heap and then dropped: a pull that ends in an error leaves the stream exactly where it was, which a test pins.

Both sides of each unresolved dependency are carried through as computed rather than derived from one another: 1-(1-score) is not score in floating point, and deriving one side would perturb every score below the cutoff.

There is no depth bound. It existed to cap 2^n enumeration, and lazy generation removed that cost, so a head may now have as many unresolved dependencies as it likes.

A dependency absent from the live batches cannot be scored, and gets a high default rather than an even split. It stays unresolved rather than becoming a certainty — both outcomes remain in the space. The cutoff used to pick which side is preferred is a separate constant from that default, so the two move independently; sharing one constant would have silently inverted the preferred assumption for every dependency below the default.

Also sharpens scorer.Scorer's contract: it returns the probability that a batch's build succeeds, not "the likelihood of a successful land", which conflated the build with the merge. That is exactly the quantity bestfirst needs, since an assumption that a dependency succeeds is refuted when its builds cannot pass. The scorer README is rewritten around entity.Batch, which the interface has taken for some time.

The package is three files by phase: bestfirst.go holds the generator, the per-dependency derivation and the scoring done at Open, head.go holds a head and everything derived from it including path materialization, and iterator.go holds the heap and the pull-time walk.

Stack

1. refactor(entity)!: speculation paths carry assumptions, not bets #473
2. @ feat(speculation): generator contract and bestfirst impl #446
3. feat(speculation): allocator contract and sticky impl #450
4. feat(speculation): standard composed speculator #451