uber · behinddwalls · Jul 24, 2026
@@ -1,64 +1,17 @@
-# Scorer
+# scorer
 
-Vendor-agnostic interface for computing success probability scores for code changes.
+A `Scorer` returns the probability that a batch's build succeeds — a number between 0.0 and 1.0. It is handed the batch identity and resolves the batch's changes itself through an injected `changeset.Resolver`, so callers pass an `entity.Batch` and nothing more.
 
-## Interface
+Callers may score every batch a queue is waiting on, so implementations should be cheap. A speculation run scores each batch at most once, but it does not carry results across runs; anything expensive belongs behind the implementation's own cache.
 
-### Scorer
-
-Computes a success probability for a given change.
-
-```go
-type Scorer interface {
-    Score(ctx context.Context, change entity.Change) (float64, error)
-}
-```
-
-- **change**: A `entity.Change` identifying the code change to score.
-- **Score**: Returns a probability between 0.0 and 1.0 indicating the likelihood of a successful land. Returns an error if scoring fails.
+Like the other extensions, a `Scorer` is selected **per queue** by the wiring layer through the `Config` (queue name) and `Factory` interface.
 
 ## Implementations
 
-### Heuristic
-
-Scores a change by extracting a numeric value via a `ValueFunc` and matching it against ordered buckets. Each bucket maps a `[Min, Max]` range to a probability.
-
-```go
-s := heuristic.New(
-    []heuristic.Bucket{
-        {Min: 0, Max: 5, Score: 0.95},
-        {Min: 6, Max: 20, Score: 0.75},
-        {Min: 21, Max: 100, Score: 0.5},
-    },
-    func(ctx context.Context, change entity.Change) (int, error) {
-        // resolve the change into a numeric metric
-        return filesChanged, nil
-    },
-)
-
-score, err := s.Score(ctx, change)
-```
-
-### Composite
-
-Combines multiple named scorers into a single score using a reduce function. The reduce function receives a `map[string]float64` mapping scorer names to their scores, enabling domain-aware aggregation.
-
-Built-in reduce functions: `Min`, `Max`, `Avg`.
-
-```go
-s := composite.New(
-    map[string]scorer.Scorer{
-        "files": fileScorer,
-        "deps":  depScorer,
-    },
-    composite.Min,
-)
+**`heuristic`** scores a batch by extracting one number from its changes and matching that against ordered buckets, each mapping a `[Min, Max]` range to a probability. The extraction is a caller-supplied `ValueFunc` over the resolved `entity.BatchChanges`, so the same bucketing works for files touched, lines changed, or any other metric.
 
-score, err := s.Score(ctx, change)
-```
+**`composite`** runs several named scorers and reduces their scores to one. The reduce function receives the scores keyed by scorer name, so it can weigh sources differently rather than treating them as interchangeable; `Min`, `Max`, and `Avg` are provided.
 
-## Implementing a Backend
+## Adding a backend
 
-1. Create `extension/scorer/{backend}/` directory
-2. Implement the `Scorer` interface
-3. Accept `entity.Change` and resolve it into whatever data the implementation needs
+Create a package under `scorer/<backend>/` whose `New(...)` returns a `scorer.Scorer`, injecting whatever it needs at construction — a `changeset.Resolver` to reach the batch's changes, a metrics scope, any client. Do not add a `Config` or `Factory` implementation here; per-queue routing and the factory adapter live in the wiring layer.
@@ -22,11 +22,17 @@ import (
 	"github.com/uber/submitqueue/submitqueue/entity"
 )
 
-// Scorer computes a success probability score for a batch based on its changes.
+// Scorer computes the probability that a batch's build succeeds, based on its
+// changes.
 type Scorer interface {
-	// Score returns a probability between 0.0 and 1.0 indicating the likelihood
-	// of a successful land for the given batch. It is handed the batch identity
-	// and resolves the batch's changes itself through an injected changeset.Resolver.
+	// Score returns a probability between 0.0 and 1.0 that the given batch's
+	// build succeeds. It is handed the batch identity and resolves the batch's
+	// changes itself through an injected changeset.Resolver.
+	//
+	// Callers may score every batch a queue is waiting on, so implementations
+	// should be cheap: a speculation run scores each batch at most once, but it
+	// does not carry results over to the next run, so anything expensive to
+	// compute belongs behind the implementation's own cache.
 	Score(ctx context.Context, batch entity.Batch) (float64, error)
 }
 

@@ -0,0 +1,9 @@
+load("@rules_go//go:def.bzl", "go_library")
+
+go_library(
+    name = "go_default_library",
+    srcs = ["generator.go"],
+    importpath = "github.com/uber/submitqueue/submitqueue/extension/speculation/generator",
+    visibility = ["//visibility:public"],
+    deps = ["//submitqueue/entity:go_default_library"],
+)
@@ -0,0 +1,9 @@
+# generator
+
+The `generator` package is a piece the `standard` `Speculator` is built from: a `Generator` produces the queue's candidate paths as one stream, best first, across all heads. It is **not** controller-facing — the speculate controller only knows the `Speculator` contract, and a different `Speculator` need not split its work this way. So there is no `Config` or `Factory` here; a `Generator` is chosen when the `standard` `Speculator` is constructed.
+
+`Open` starts the stream over the queue's live batches and returns a `PathIterator`. The caller pulls one candidate at a time and the generator does only the work that answer needs. A cancelled or expired context ends the stream with its error.
+
+Candidates never repeat and never contradict a known fact. Beyond that, the order is the `Generator`'s own: it yields candidates in whatever ranking it implements, and each carries the score it ranked by — higher first, on a scale the generator defines. Consumers take the iterator in the order given and do not interpret the score. Scores mean something only within the run and are never stored.
+
+The generator offers every path in the space, including paths whose builds already ran. Suppressing finished paths is the `Allocator`'s job, since that is the piece reconciling candidates against the stored path sets.
@@ -0,0 +1,30 @@
+load("@rules_go//go:def.bzl", "go_library", "go_test")
+
+go_library(
+    name = "go_default_library",
+    srcs = [
+        "bestfirst.go",
+        "head.go",
+        "iterator.go",
+    ],
+    importpath = "github.com/uber/submitqueue/submitqueue/extension/speculation/generator/bestfirst",
+    visibility = ["//visibility:public"],
+    deps = [
+        "//submitqueue/entity:go_default_library",
+        "//submitqueue/extension/scorer:go_default_library",
+        "//submitqueue/extension/speculation/generator:go_default_library",
+    ],
+)
+
+go_test(
+    name = "go_default_test",
+    srcs = ["bestfirst_test.go"],
+    embed = [":go_default_library"],
+    deps = [
+        "//submitqueue/entity:go_default_library",
+        "//submitqueue/extension/scorer:go_default_library",
+        "//submitqueue/extension/speculation/generator:go_default_library",
+        "@com_github_stretchr_testify//assert:go_default_library",
+        "@com_github_stretchr_testify//require:go_default_library",
+    ],
+)
@@ -0,0 +1,100 @@
+# bestfirst
+
+A *head* — the batch we want to build — waiting on unfinished dependencies has one path per combination of outcomes — 2ⁿ of them — while callers want the best few. `bestfirst` hands them out in score order without building the rest.
+
+Dependencies that already resolved carry a forced assumption: succeeded means the head builds on top of it, failed or cancelled means it builds without. They stay in the path but drop out of the search. Each unresolved one is a two-way assumption whose *probability* — that the dependency's build succeeds — comes from an injected `scorer.Scorer`, remembered per dependency so one shared by several heads is scored once. A dependency that is not a live batch cannot be scored at all, and gets a high default, since nearly every batch a queue accepts does build successfully. A path's score follows from the probability that all its assumptions hold.
+
+**The best path is not "assume everything succeeds."** Each dependency takes its *preferred* assumption — the one with the higher probability — so a dependency that will probably fail is assumed to fail. Every other path takes the unpreferred assumption on some of them, and each of those costs a known factor.
+
+**Scores are logarithms.** A probability is the `[0, 1]` value the scorer gives; a score is always its logarithm. Scores use summed log probabilities to preserve ordering without underflow: a plain product across a wide head rounds to zero, ties, and loses the order entirely. A score is at most 0, and an assumption that cannot happen scores negative infinity. The package doc in `bestfirst.go` shows the failure it avoids.
+
+**Laziness.** Each head roots a tree of its paths: the root takes no alternatives, and every other node takes some subset. One heap holds the frontier of all those trees at once — the nodes reached but not yet handed out. Handing a node out adds its children, so the heap always holds each head's current best and the top of it is the best path in the queue.
+
+A node is its head, the alternatives it takes, and a score, so reaching one costs nothing near the size of the head. The path itself is built only when the node is handed out, and a head works out and sorts its alternatives only when its root is expanded — a head nobody pulls from does neither. Pulling *k* paths builds *k* paths and works out the alternatives of at most *k* heads, whatever the 2ⁿ behind them.
+
+What cannot be deferred is scoring. Ranking heads against each other needs every head's best-path score up front, so `Open` walks every dependency of every head once. That is linear in the queue's dependency edges — and since `Dependencies` holds a transitive closure, a chain of batches has quadratically many edges in its depth. `Open` pays that whether or not anything is pulled.
+
+`bestfirst` discards nothing: pull long enough and every path for every head comes out. It does no conflict relaxation (`ignored` assumptions) — that is the `Speculator`'s call, not the generator's.
+
+## Worked example
+
+A head waits on three unresolved dependencies. The scorer gives each the probability its build succeeds:
+
+| dependency | probability | preferred assumption |
+| --- | --- | --- |
+| `d0` | 0.9 | succeeds |
+| `d1` | 0.8 | succeeds |
+| `d2` | 0.6 | succeeds |
+
+All three are likelier to succeed than not, so the head's **best path** assumes all three succeed. Its probability is the product `0.9 × 0.8 × 0.6 = .432`, and its score is the sum of the logs, `−0.105 + −0.223 + −0.511 = −0.839`. Probabilities read more easily, so the tables below stay in them; the code only ever compares logs.
+
+### Step 1 — the alternatives, cheapest first
+
+Flipping one dependency to its other side swaps its factor in that product, multiplying the result by `unpreferred / preferred`. Sorted by what that costs:
+
+| | flips | ratio | log |
+| --- | --- | --- | --- |
+| `a0` | `d2` (0.6) | 0.4 / 0.6 = .67 | −0.41 |
+| `a1` | `d1` (0.8) | 0.2 / 0.8 = .25 | −1.39 |
+| `a2` | `d0` (0.9) | 0.1 / 0.9 = .11 | −2.20 |
+
+`a0` is `d2`, not `d0`. The cheapest deviation is from the *least* confident dependency — it is the assumption we mind changing least.
+
+This is the only sorting that happens, it is per head, and it happens the first time something pulls from that head.
+
+### Step 2 — the tree
+
+Every path is the best path with some set of alternatives taken. `expand` grows those sets one step at a time: either take the next alternative as well, or move the last one along. That lays all 2³ = 8 paths out as a tree, rooted at the best path:
+
+```
+{}                        .432   flip nothing — the best path
+└── {a0}                  .288   flip d2
+    ├── {a0,a1}           .072   flip d2, d1
+    │   ├── {a0,a1,a2}    .008   flip all three — the worst path
+    │   └── {a0,a2}       .032   flip d2, d0
+    └── {a1}              .108   flip d1
+        ├── {a1,a2}       .012   flip d1, d0
+        └── {a2}          .048   flip d0
+```
+
+Two things to read off it. Every node scores at or below its parent — that is what the heap relies on, and it holds because the alternatives are sorted, so both moves reach for a later, never-cheaper one. But **siblings are not ordered**: `{a0,a1}` at .072 sits above `{a1}` at .108 in the tree while scoring below it. The tree only guarantees the parent relation. Global order is the heap's job.
+
+### Step 3 — scoring a node
+
+A node's score is the head's best-path score plus each taken alternative's log, and never anything carried over from its parent. For `{a0,a2}`:
+
+```
+−0.8393  best path
+−0.4055  a0
+−2.1972  a2
+───────
+−3.4420
+```
+
+Which is the same path read directly: `d0` fails, `d1` succeeds, `d2` fails → `0.1 × 0.8 × 0.4 = .032`, and `log(.032) = −3.4420`.
+
+### Step 4 — pulling
+
+`Open` puts just the root in the heap. Each pull takes the best node there, builds *that* path, and puts back its children:
+
+```
+handed out         heap afterwards
+{}         .432    {a0}·.288
+{a0}       .288    {a1}·.108  {a0,a1}·.072
+{a1}       .108    {a0,a1}·.072  {a2}·.048  {a1,a2}·.012
+{a0,a1}    .072    {a2}·.048  {a0,a2}·.032  {a1,a2}·.012  {a0,a1,a2}·.008
+{a2}       .048    {a0,a2}·.032  {a1,a2}·.012  {a0,a1,a2}·.008
+{a0,a2}    .032    {a1,a2}·.012  {a0,a1,a2}·.008
+{a1,a2}    .012    {a0,a1,a2}·.008
+{a0,a1,a2} .008    —
+```
+
+The scores come out in descending order without the eight paths ever being enumerated or sorted together, and the heap never holds more than four nodes. Stop after two pulls and only two paths were ever built — everything else reached is still just a set of indexes.
+
+With several heads in the queue there is still only this one heap. Every head's root goes in at `Open`, they compete on score, and a head nobody reaches never even sorts its alternatives.
+
+### Ties
+
+Equal scores break on depth first, so every head's root is offered before any head's deeper nodes; then on the alternatives taken, which interleaves heads at equal depth; then on the head itself. Depth has to come first because a dependency at probability exactly 0.5 deviates for free, and ordering on the head instead would let one head's tied subtree drain a caller's budget before another head's root was offered at all.
+
+The proof that this walk reaches every path exactly once, in score order, is on `expand` in `iterator.go`. The behavior is pinned by tests in `bestfirst_test.go`.