fix: solve the image build error for big images

cmd/main.go

@@ -61,6 +61,15 @@ import (
 // a fallback for managerNamespace when POD_NAMESPACE is unset.
 const defaultNamespace = "nebula-system"
 
+// restConfigQPS and restConfigBurst size the client-go bucket shared by every API call in the
+// process — each controller's, and the virtual kubelet's status pushes. controller-runtime's
+// default 20/30 is what binds first at fleet scale, and it binds invisibly: throttled calls
+// wait in our own process, so it reads as API-server or provider slowness.
+const (
+	restConfigQPS   = 50
+	restConfigBurst = 100
+)
+
 var (
 	scheme   = runtime.NewScheme()
 	setupLog = ctrl.Log.WithName("setup")
@@ -207,7 +216,11 @@ func main() {
 		})
 	}
 
-	mgr, err := ctrl.NewManager(ctrl.GetConfigOrDie(), ctrl.Options{
+	restConfig := ctrl.GetConfigOrDie()
+	restConfig.QPS = restConfigQPS
+	restConfig.Burst = restConfigBurst
+
+	mgr, err := ctrl.NewManager(restConfig, ctrl.Options{
 		Scheme:                 scheme,
 		Metrics:                metricsServerOptions,
 		WebhookServer:          webhookServer,

docs/add-a-provider.md

@@ -34,17 +34,21 @@ claim identity). Do not duplicate Pod fields onto the request.
 
 ### Wrap the errors your `Provision` returns
 
-`ClassifyProvisionError` decides *how widely* to blocklist, but a separate predicate —
-`provider.IsRejection` — decides *whether to blocklist at all*, and whether the Pod is
-failed. It answers: did the provider make a decision about this request ("no capacity",
-"over quota", "bad credentials"), or did we merely fail to find out what it would have
-decided (a transport error, a timeout, a 503)?
-
-Only a **decision** is acted on. An unattributable failure leaves the Pod
-non-terminal at `Provisioning` for the pod controller to retry, and records nothing —
-because failing a Pod there would stamp a terminal verdict on a request the provider may
-well have accepted, reaping the Pod out from under a paid instance whose id was never
-returned.
+`provider.ClassifyError` — which `ClassifyProvisionError` delegates to before adding
+anything provider-specific — is the *single* answer to what a failure blocklists, and its
+**zero scope means "block nothing"**. Three outcomes are possible: a decision about the
+candidate ("no capacity", "over quota", "I do not offer that accelerator") is scoped to
+that accelerator and tier so failover routes around it; bad credentials widen to the whole
+provider; and two cases block nothing at all — a failure that belongs to the *request*
+rather than the candidate (an image it cannot pull or build), and one that could not be
+attributed to either (a transport error, a 503, an unparseable response).
+
+The Pod fails either way, so failover can pick a different candidate rather than sit behind
+an attempt nothing re-enters. What your wrapping decides is whether a candidate is fenced
+off for the blocklist TTL — and getting it wrong is expensive in both directions: an
+unwrapped image failure whose text says `unauthorized` reads as auth and fences off your
+entire provider, while an unwrapped capacity refusal lets the next Pod fail exactly the
+same way.
 
 What this asks of an adapter: **wrap every error your `Provision` path returns with the
 matching sentinel** (`fmt.Errorf("...: %w", provider.ErrNoCapacity)`). A wrapped sentinel

docs/architecture.md

@@ -601,13 +601,15 @@ Components designed to degrade without leaks:
   suppresses both a duplicate `CreatePod` and a premature `DeletePod` until the
   provider answers. Conflating the two would let one failed list mark a healthy Pod
   `Failed` for reaping while the paid instance kept running behind a zero instance id.
-- **Provider unreachable during `Provision`.** Distinguished from a *rejection*
-  (`provider.IsRejection`). Only a rejection — no capacity, quota, auth, unsupported
-  accelerator — fails the Pod and files a blocklist entry. A transport error, timeout
-  or 503 leaves the Pod non-terminal at `Provisioning` with the error as its message
-  and records nothing, because it is not evidence about the request: the provider may
-  have accepted it. `Provision` is idempotent on `ClaimName`, so the retry adopts
-  whatever the failed attempt created rather than doubling it.
+- **Provider unreachable during `Provision`.** Every provision failure fails the Pod, so
+  placement can fail over rather than sit behind an attempt nothing re-enters. What the
+  failure's *kind* still decides is the blocklist, and `provider.ClassifyError` is the
+  single answer: a transport error or 503 files nothing, because it is not evidence about
+  the candidate — the provider may well have accepted the request. An exhausted provision
+  timeout is candidate-scoped; other candidate decisions (no capacity, quota, unsupported
+  accelerator) also file an entry, and only auth widens it to the whole provider.
+  `ClaimName`, so a re-provision adopts whatever the failed attempt created rather than
+  doubling it.
 
 Leader election (`LeaderElectionID: nebula.inftyai.com`) keeps a single active
 manager reconciling controllers and owning virtual-node leases.

docs/metrics.md

@@ -87,7 +87,7 @@ What the external call cost, and how it failed.
 | --- | --- | --- |
 | `nebula_provision_attempts_total{result}` | counter | Provisioning volume and error rate, per candidate. |
 | `nebula_provision_failures_total{reason}` | counter | *Why* provisioning fails. |
-| `nebula_provision_duration_seconds{result}` | histogram | Latency of the `Provision` call alone. AWS sweeps a region's availability zones inside it, so a capacity shortage shows up as latency *here*; Modal returns as soon as the sandbox is accepted and the wait moves to the next metric. |
+| `nebula_provision_duration_seconds{result}` | histogram | Latency of the `Provision` call alone. What lands here is provider-specific: AWS sweeps a region's availability zones inside the call, so a capacity shortage shows up as latency *here*; Modal builds the image inside it, so a build-cache miss does. Bucketed to 600s, the largest `ProvisionTimeout` — so `+Inf` means a call outran its own deadline. |
 | `nebula_instance_ready_duration_seconds` | histogram | The whole user-visible wait, from `CreatePod` to the first poll tick reporting `Running` — including provider-side queueing, image pull, GPU attach and up to one poll interval of detection lag. |
 
 `nebula_provision_failures_total` deliberately overlaps
@@ -105,8 +105,7 @@ The failure `reason` is a coarse, closed set, mapped from the shared sentinels i
 | `auth` | `ErrAuth` — credentials or permissions. |
 | `unsupported_accelerator` | `ErrUnsupportedAccelerator` — the request cannot be honoured here at all. |
 | `timeout` | The `Provision` call hit its own deadline without a capacity cause. |
-| `unreachable` | The provider never told us what it decided. |
-| `other` | The provider *did* reject the request, but the adapter returned a raw API error without wrapping a sentinel, so the category was unavailable. |
+| `other` | No sentinel, so the category was unavailable — either the adapter returned a raw API error without wrapping one, or the provider never told us what it decided at all. |
 
 Fine-grained detail is deliberately *not* here: it stays where it is already available
 (the Pod's `Failed` status message and the `vnode-handler` error log). These labels exist
@@ -149,11 +148,11 @@ Five label values are load-bearing:
 - **`region`** is the provider's own token, not necessarily one region. For a provider that
   collapses every declared region into a single candidate (Modal) it is the joined form —
   the same value `NodeClaimSpec.Region` carries.
-- **`reason="unreachable"`** means the provider never told us what it decided: a transport
-  failure, a 503, an unparseable response. It is the one failure reason for which Nebula
-  deliberately does *not* fail the Pod or blocklist the candidate (see
-  `provider.IsRejection`), so a spike here alongside flat `capacity`/`auth` series is a
-  network or provider-outage problem, not a placement one.
+- **`reason="other"`** covers both an adapter that did not wrap its errors and a provider
+  that never told us what it decided (a transport failure, a 503). Neither blocklists the
+  candidate on its own, and the two are separated in the `vnode-handler` error log rather
+  than in the label. A spike here alongside flat `capacity`/`auth` series is the shape of a
+  network or provider outage.
 
 Cardinality is bounded by *configuration*, not by workload: providers x regions x tiers x
 accelerator pools, all of which come from NodePools and provider catalogs. Nothing

pkg/metrics/provision.go

@@ -17,7 +17,6 @@ limitations under the License.
 package metrics
 
 import (
-	"context"
 	"errors"
 	"time"
 
@@ -45,19 +44,12 @@ const (
 	ReasonAuth        = "auth"
 	ReasonUnsupported = "unsupported_accelerator"
 	ReasonTimeout     = "timeout"
-	// ReasonUnreachable: the provider never told us what it decided — a transport
-	// failure, a 503, an unparseable response. Kept separate from every other reason
-	// because it is the one that is NOT about capacity, credentials or the request: it
-	// says the integration itself is unhealthy, and it is the only reason for which
-	// Nebula deliberately does not fail the Pod or blocklist the candidate (see
-	// provider.IsRejection). A spike here alongside flat capacity/auth series is a
-	// network or provider-outage signal, not a placement one.
-	ReasonUnreachable = "unreachable"
-	// ReasonOther: the provider DID reject the request, but not through a sentinel, so
-	// the category is unavailable. In practice that means the adapter returned a raw API
-	// error without wrapping it, which makes a sustained rate on this series a to-do
-	// rather than an incident: wrap the condition in the adapter (see
-	// docs/add-a-provider.md) and the failure moves onto its real category.
+	// ReasonOther: the failure carried no sentinel, so its category is unavailable — either
+	// the adapter returned a raw API error without wrapping it, or the provider never told
+	// us what it decided at all (a transport failure, a 503, an unparseable response). A
+	// sustained rate here is a to-do rather than an incident: wrap the condition in the
+	// adapter (see docs/add-a-provider.md) and the failure moves onto its real category.
+	// Which of the two it was is in the vnode-handler error log.
 	ReasonOther = "other"
 )
 
@@ -83,17 +75,19 @@ var (
 	}, withExtra("reason"))
 
 	// ProvisionDuration measures the provider's Provision call alone — not the
-	// wait for the instance to become usable. The two differ enormously and for
-	// different reasons: AWS sweeps a region's availability zones inside this call
-	// (so a capacity shortage shows up as latency HERE), while Modal returns as soon
-	// as the sandbox is accepted and the wait moves to InstanceReadyDuration.
-	// Bucketed out to 300s because the call is bounded by
-	// Capabilities.ProvisionTimeout, which AWS raises above the 90s default.
+	// wait for the instance to become usable. What lands here is provider-specific
+	// and worth knowing per provider: AWS sweeps a region's availability zones
+	// inside the call (so a capacity shortage shows up as latency HERE), Modal
+	// builds the image inside it (so a cache miss does).
+	//
+	// The top bucket tracks the largest Capabilities.ProvisionTimeout — Modal's 10
+	// minutes. A lower ceiling would bury every slow build in +Inf; at 600s, +Inf
+	// means one thing only: a call outran its own deadline.
 	ProvisionDuration = prometheus.NewHistogramVec(prometheus.HistogramOpts{
 		Name: "nebula_provision_duration_seconds",
 		Help: "Latency of the provider's Provision call, by provider, region, capacity type, " +
 			"accelerator type, accelerator count and outcome.",
-		Buckets: []float64{0.5, 1, 2.5, 5, 10, 20, 30, 45, 60, 90, 120, 180, 300},
+		Buckets: []float64{0.5, 1, 2.5, 5, 10, 20, 30, 45, 60, 90, 120, 180, 300, 420, 600},
 	}, withExtra("result"))
 
 	// InstanceReadyDuration measures the whole user-visible wait: from the moment
@@ -170,17 +164,8 @@ func FailureReason(err error) string {
 		return ReasonUnsupported
 	case errors.Is(err, provider.ErrNoCapacity):
 		return ReasonCapacity
-	// Checked after the sentinels: a provider that hits its own deadline while
-	// sweeping for capacity may wrap both, and the capacity cause is the more useful
-	// of the two.
-	case errors.Is(err, context.DeadlineExceeded):
+	case provider.IsDeadline(err):
 		return ReasonTimeout
-	// Everything left is either a rejection whose category we could not name, or a
-	// failure to reach the provider at all. Splitting them is the whole point of having
-	// this label: the first is a placement problem, the second an integration one, and
-	// they are fixed by completely different people.
-	case !provider.IsRejection(err):
-		return ReasonUnreachable
 	default:
 		return ReasonOther
 	}

pkg/metrics/provision_test.go

@@ -27,7 +27,7 @@ import (
 
 // The reason label is a CLOSED set, so this pins every mapping into it. The label is
 // what an operator reads to decide whether a provisioning problem is theirs (quota,
-// credentials), the provider's (capacity, unreachable) or neither.
+// credentials), the provider's (capacity) or neither.
 func TestFailureReason(t *testing.T) {
 	tests := []struct {
 		name string
@@ -45,18 +45,26 @@ func TestFailureReason(t *testing.T) {
 		// the capacity cause is the more useful of the two, so the sentinels come first.
 		{"capacity beats timeout", fmt.Errorf("%w: %w", provider.ErrNoCapacity, context.DeadlineExceeded), ReasonCapacity},
 		{"bare timeout", context.DeadlineExceeded, ReasonTimeout},
+		// The form errors.Is CANNOT see, and the one a timed-out Modal build actually arrives
+		// in: grpc-go renders an expired context as a status error wrapping nothing. This read
+		// as "other" until FailureReason moved onto provider.IsDeadline — a real timeout
+		// reported as an unwrapped-adapter to-do, which is the one thing "other" must not mean.
+		{"grpc deadline during an image build",
+			fmt.Errorf("modal: image build: %w",
+				errors.New("rpc error: code = DeadlineExceeded desc = context deadline exceeded")),
+			ReasonTimeout},
 
-		// The split that matters operationally: an integration outage must not read as a
-		// capacity shortfall, or the failure series points at the wrong problem entirely.
-		{"grpc transport", errors.New("rpc error: code = Unavailable desc = transport is closing"), ReasonUnreachable},
-		{"connection refused", errors.New("dial tcp: connect: connection refused"), ReasonUnreachable},
-		{"unrecognized", errors.New("weird transient blip"), ReasonUnreachable},
+		// What matters here is that a transport failure does NOT read as a capacity
+		// shortfall, which would point the failure series at the wrong problem entirely.
+		// "Unavailable" in a gRPC status text is the misread this guards.
+		{"grpc transport", errors.New("rpc error: code = Unavailable desc = transport is closing"), ReasonOther},
+		{"connection refused", errors.New("dial tcp: connect: connection refused"), ReasonOther},
+		{"unrecognized", errors.New("weird transient blip"), ReasonOther},
 
-		// An unwrapped provider message is recognized as a REJECTION (so not
-		// "unreachable"), but its category is unavailable here: FailureReason matches
-		// sentinels only, on purpose, rather than re-running the string heuristics. So it
-		// lands on "other", which is precisely the signal that an adapter is not wrapping
-		// its errors — actionable, unlike a guess at the category.
+		// An unwrapped provider message lands on "other" too: FailureReason matches
+		// sentinels only, on purpose, rather than re-running the string heuristics. That is
+		// precisely the signal that an adapter is not wrapping its errors — actionable,
+		// unlike a guess at the category.
 		{"unwrapped rejection", errors.New("InsufficientInstanceCapacity"), ReasonOther},
 	}
 	for _, tt := range tests {
@@ -93,7 +101,7 @@ func TestObserveProvision_CountersStayInStep(t *testing.T) {
 	// A success must never touch the failure-reason counter, whatever the reason.
 	allReasons := []string{
 		ReasonCapacity, ReasonAuth, ReasonQuota,
-		ReasonUnsupported, ReasonTimeout, ReasonUnreachable, ReasonOther,
+		ReasonUnsupported, ReasonTimeout, ReasonOther,
 	}
 	for _, reason := range allReasons {
 		if reason == ReasonCapacity {

pkg/provider/aws/aws_test.go

@@ -559,25 +559,26 @@ func TestClassifyProvisionError(t *testing.T) {
 		{"wrapped capacity is regional", fmt.Errorf("run: %w", provider.ErrNoCapacity), onDemandRegional},
 		{"spot capacity blocks only Spot in region", spotNoCapacity, spotRegional},
 		{"string no-capacity is regional OnDemand", stringNoCapacity, onDemandRegional},
-		// An unrecognized error is confined to this accelerator + tier + region, NOT
-		// DenyAll: a whole-provider block on an unidentifiable failure is too broad, so
-		// failover routes around the one failing candidate instead.
-		{"unknown is regional OnDemand", fmt.Errorf("weird transient blip"), onDemandRegional},
-		// InvalidFleetConfiguration, in a CreateFleet per-override error, means the
-		// instance type is not offered in that subnet's AZ — a zone-local availability
-		// gap classifyEC2Error maps to no-capacity, so it blocks only this
-		// accelerator/tier/region (a sibling AZ or region may still serve it), never
-		// DenyAll.
-		{"invalid fleet config is regional OnDemand",
+		// An error we cannot attribute blocks NOTHING, so no region is stamped either: a
+		// transport failure or an unknown code is no evidence against this candidate.
+		{"unknown blocks nothing", fmt.Errorf("weird transient blip"), provider.BlockScope{}},
+		// The RAW code reaches here only if it escaped translation, and unrecognized means
+		// unattributable, so it blocks nothing. The real path never presents it this way:
+		// classifyEC2Error wraps InvalidFleetConfiguration as no-capacity first (it means
+		// "not offered in this subnet's AZ" — see translate.go), which lands on the
+		// ErrNoCapacity rows above and blocks this accelerator/tier/region.
+		{"raw invalid fleet config blocks nothing",
 			&smithy.GenericAPIError{Code: "InvalidFleetConfiguration", Message: "not supported in AZ"},
-			onDemandRegional},
+			provider.BlockScope{}},
 		{"nil", nil, provider.BlockScope{}},
 		// An image pull credential this adapter cannot honour is a fact about the POD, so it
 		// blocks nothing. Region must NOT be stamped on: a scope carrying only a region
 		// fences off every accelerator and tier there, on behalf of one Pod.
-		{"image pull blocks nothing", provider.ErrImagePull, provider.BlockScope{}},
-		{"wrapped image pull blocks nothing",
-			fmt.Errorf("aws: unsupported image pull credential: %w", provider.ErrImagePull),
+		{"unsupported image pull blocks nothing",
+			(&provider.RegistryAuth{Registry: "ghcr.io"}).Unsupported("aws"),
+			provider.BlockScope{}},
+		{"malformed image pull blocks nothing",
+			(&provider.RegistryAuth{Registry: "ghcr.io", Basic: &provider.BasicAuth{Username: "u"}}).Validate(),
 			provider.BlockScope{}},
 	}
 	for _, tt := range tests {