You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
As an OpenShell cluster operator, I want multi-replica gateways to expose actionable capacity signals and redistribute supervisor ownership during planned changes, so that scaling and rolling upgrades do not leave one replica overloaded while new replicas remain idle.
Problem Statement
PR #1868 makes multi-replica gateway routing possible, but several production-scale behaviors remain incomplete:
Supervisor ownership stays with the original connection. Rolling restarts and failover can leave the last surviving replica holding most sessions until sandboxes naturally reconnect.
The gateway does not expose held-session, pending-relay, or peer-relay rate and latency metrics suitable for capacity planning or autoscaling.
Cross-object mutations use one fleet-wide PostgreSQL advisory-lock key, so unrelated sandbox and provider mutations serialize across all replicas.
WatchSandbox performs one database lookup per actively watched sandbox, per polling interval, per replica.
Impact / Why This Matters
Operators can add gateway replicas, but cannot tell when capacity is exhausted or rely on replicas receiving a balanced share of established supervisor sessions. The current workaround is to overprovision, manually inspect logs, and wait for sandbox churn or force reconnects after a rollout. That does not prevent pending-relay exhaustion, database amplification, or fleet-wide mutation contention at larger scale.
Proposed Design
Provide an observable, documented HA lifecycle with graceful draining and bounded shared-database load:
A terminating gateway stops accepting new ownership, releases or transfers existing ownership, and prompts affected supervisors to reconnect before the termination grace period expires.
Gateway metrics expose held supervisor sessions, pending relays versus capacity, and peer-relay request rate, failure rate, and latency.
The Helm chart documents those signals and provides an optional autoscaling configuration, or a documented integration contract for a Kubernetes metrics adapter.
Unrelated object mutations do not share one fleet-wide serialization point when their invariants cannot overlap.
Cross-replica sandbox watches use a batched or notification-based mechanism whose database work does not grow as one query per watched sandbox on every replica.
Acceptance Criteria
During a graceful gateway rollout, established supervisor ownership redistributes without waiting for natural sandbox churn, within the configured termination grace period.
Operators can observe per-replica held sessions, pending relays/capacity, and peer-relay rate, failures, and latency through the supported metrics endpoint.
The Helm documentation explains how to use these signals for horizontal scaling; any chart-provided HPA behavior is optional and configurable.
Mutations for unrelated sandboxes or providers can proceed concurrently across replicas unless they share a documented cross-object invariant.
Cross-replica watch synchronization avoids one database query per watched sandbox per interval per replica.
Load or integration tests cover rollout redistribution, relay saturation signals, concurrent unrelated mutations, and large watch sets.
HA architecture and Kubernetes operator documentation describe the lifecycle, capacity signals, and database-scaling behavior.
Alternatives Considered
Keeping ownership sticky is simple and preserves correctness, but leaves rollout skew entirely dependent on sandbox churn. Relying only on logs does not provide stable autoscaling inputs. Increasing the PostgreSQL connection pool masks neither fleet-wide lock serialization nor query amplification.
Agent Investigation
This groups the remaining production-scale review discussions from PR #1868:
User Story
As an OpenShell cluster operator, I want multi-replica gateways to expose actionable capacity signals and redistribute supervisor ownership during planned changes, so that scaling and rolling upgrades do not leave one replica overloaded while new replicas remain idle.
Problem Statement
PR #1868 makes multi-replica gateway routing possible, but several production-scale behaviors remain incomplete:
WatchSandboxperforms one database lookup per actively watched sandbox, per polling interval, per replica.Impact / Why This Matters
Operators can add gateway replicas, but cannot tell when capacity is exhausted or rely on replicas receiving a balanced share of established supervisor sessions. The current workaround is to overprovision, manually inspect logs, and wait for sandbox churn or force reconnects after a rollout. That does not prevent pending-relay exhaustion, database amplification, or fleet-wide mutation contention at larger scale.
Proposed Design
Provide an observable, documented HA lifecycle with graceful draining and bounded shared-database load:
Acceptance Criteria
Alternatives Considered
Keeping ownership sticky is simple and preserves correctness, but leaves rollout skew entirely dependent on sandbox churn. Relying only on logs does not provide stable autoscaling inputs. Increasing the PostgreSQL connection pool masks neither fleet-wide lock serialization nor query amplification.
Agent Investigation
This groups the remaining production-scale review discussions from PR #1868:
Related umbrella issue: #1021.