Skip to content

feat(kubernetes): add template-driven warm pooling - #3508

Open
grs wants to merge 1 commit into
NVIDIA:mainfrom
grs:warm-pooling
Open

grs wants to merge 1 commit into
NVIDIA:mainfrom
grs:warm-pooling

Conversation

@grs

@grs grs commented Sep 21, 2026

Copy link
Copy Markdown
Contributor

Summary

Add Kubernetes warm pooling for template-based sandboxes. The Kubernetes driver prepares single-use workload/supervisor pairs ahead of demand, allowing compatible sandbox creates to claim ready capacity while preserving cold provisioning as a fallback.

Warm pooling is enabled by default with bounded capacity, but only templates with startup.ready_within and a positive startup.max_burst request spare pairs.

Example:

openshell sandbox template create my-template \
  --ready-within 1s \
  --max-burst 2

openshell sandbox create \
  --template my-template \
  -- echo ready

Related Issue

Closes #2157

Changes

  • Add a leader-elected warm-pool controller to the Kubernetes driver that prepares, validates, replenishes, expires, and retires spare pairs using core Agent Sandbox resources.
  • Add durable gateway allocation claims and atomic target reservations for safe concurrent allocation, retry, restart recovery, deletion, and provisioning-timeout cleanup.
  • Claim compatible ready pairs for template-based creates, preserve their physical identity through stop/start, and fall back to cold provisioning when no eligible pair is available.
  • Add the RegisterSupervisor flow so prepared supervisors can register before logical assignment and receive their operational credentials and launch configuration after the gateway verifies the physical runtime and persisted assignment.
  • Bind the workload to its logical sandbox through a gateway-signed resource binding, while keeping operational credentials out of Kubernetes bootstrap Secrets.
  • Add Kubernetes and Helm configuration for enabling pooling, limiting total spare capacity, and controlling preparation and idle timeouts.
  • Add Pod and Sandbox metadata reconciliation for logical sandbox and template labels, including SPIFFE sandbox identity annotations after assignment.
  • Extend Kubernetes RBAC for Lease-based controller election and bootstrap Secret validation and cleanup.
  • Document the allocation, authentication, recovery, configuration, and troubleshooting behavior.
  • Regenerate the affected Go protobuf bindings.

Testing

  • mise run pre-commit passes
  • Unit tests added/updated
  • E2E tests added/updated (if applicable)

The branch adds focused coverage for pool reconciliation, target selection and reservation conflicts, claim/retirement CAS behavior, restart recovery, late-bound authentication, configuration rendering, and persistence.

The opt-in Kubernetes E2E scenario verifies that:

  • unassigned pairs reach standby readiness without a logical sandbox identity;
  • sandbox creation claims an existing prepared pair;
  • both prepared Pods retain their physical identity after assignment;
  • the pool replenishes claimed capacity;
  • deleting a template retires spare inventory without deleting assigned pairs;
  • assigned storage survives stop/start;
  • sandbox deletion removes the assigned pair; and
  • cold provisioning remains available.

Checklist

  • Follows Conventional Commits
  • Commits are signed off (DCO)
  • Architecture docs updated (if applicable)

Signed-off-by: Gordon Sim <gsim@redhat.com>
@grs
grs requested review from a team, derekwaynecarr, mrunalp and sjenning as code owners September 21, 2026 13:16
@copy-pr-bot

copy-pr-bot Bot commented Sep 21, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat: add warm-pool provisioning for Kubernetes sandboxes

1 participant