fix(connect): remove tunnels after hosts go offline

[t3dotgg]

Cloudflare tunnels stayed provisioned after an environment crashed or went offline. Environments linked through web or mobile could not recreate a deleted tunnel.

The relay now removes tunnels that stay disconnected for at least five minutes. Updated hosts recreate tunnels under the same URL, while older hosts remain protected until they register recovery support.

Verified with 146 focused tests, 129 server integration tests, and typechecks for the relay, server, shared client runtime, web app, and mobile app.

Built by GPT-5.6 Sol with the Codex harness.

---

Note

High Risk
Changes authentication-adjacent relay APIs, optimistic concurrency on tunnel allocations, and automatic tunnel lifecycle on startup—errors could leave environments unreachable or delete tunnels incorrectly.

Overview
Adds managed tunnel recovery so linked environments can replace a deleted or rejected Cloudflare tunnel without re-pairing, while the relay can retire unused tunnels after extended downtime.

On the T3 server, the cloud managed endpoint runtime now exposes a recovery request stream, a link-state lock, and logic to detect repeated tunnel authorization failures from cloudflared (distinct from generic transport errors). Connector exit or sustained rejection enqueues recovery; startup registers recovery with the relay when possible and otherwise reconciles the CLI link, then listens for recovery events and calls the relay with the stored environment credential and signed proofs.

Cloud HTTP adds registerManagedCloudTunnelRecovery and recoverManagedCloudTunnel, maps relay 401/403/409 to permanent failures, and wraps relay config / unlink / reconcile paths in the link lock so persisted tunnel config cannot race recovery.

On the relay, allocation rows gain generation (replacing updatedAt as the CAS token), plus recovery registration fields. Allocation mutations increment generation, gate on tunnel id where needed, and add withClaimedTunnel (FOR UPDATE) so tunnel delete/release cannot race a new provision. ManagedEndpointProvider.release can require expected tunnel id, status, and inactivity time; provisioning steps fail if ownership changes mid-flight.

A Postgres migration adds recovery_enabled_at, recovery_environment_public_key, and generation on relay_managed_endpoint_allocations. User and internal docs describe automatic replacement after long offline periods and the same public hostname.

^{Reviewed by Cursor Bugbot for commit 4831d73. Bugbot is set up for automated code reviews on this repo. Configure here.}

Note

Remove offline managed tunnels and add recovery via ManagedEndpointReaper

• Adds ManagedEndpointReaper service that sweeps inactive and down managed tunnels past a 5-minute grace period, deletes orphans after revalidation, and calls provider.release with expected tunnel preconditions for owned allocations. Wired into the */5 cron in worker.ts.
• Switches ManagedEndpointAllocations from updatedAt-based to generation-based concurrency in ManagedEndpointAllocations.ts. recordTunnel, recordDns, markReady, claimRelease, claimDeprovision, and removeClaimed now require and return a numeric generation. New withClaimedTunnel provides transactional row locking with FOR UPDATE. Migration adds generation, recovery_enabled_at, and recovery_environment_public_key columns to relay_managed_endpoint_allocations.
• Adds managed tunnel recovery: two new relay HTTP endpoints (/v1/environments/:id/tunnel/recovery and /v1/environments/:id/tunnel), JWT proof verification, registerManagedCloudTunnelRecovery and recoverManagedCloudTunnel in http.ts, and server-layer recovery orchestration in server.ts that listens for runtime recoveryRequests and retries with backoff.
• CloudManagedEndpointRuntime in ManagedEndpointRuntime.ts now emits recovery requests on connector exit and after 4 unauthorized tunnel registration errors, annotates failed statuses with a failure discriminator (unsupported-platform, not-installed, spawn-failed), and exposes withLinkStateLock to serialize link-state mutations.
• Risk: ManagedEndpointProvider.Service.release now returns Effect<boolean> instead of Effect<void> — all in-tree callers and mocks are updated but out-of-tree consumers must adapt. ManagedEndpointAllocation gains a required generation field; all mutating allocation calls must pass the current generation or the update is a no-op (returns null/false).

^{Macroscope summarized 4831d73.}

[coderabbitai]

Important

Review skipped

Auto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Pro Plus

Run ID: f753c5f0-d4ad-42a1-8fb7-a8aa18584bc4

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[github-actions]

Thread transfer impact

✅ Thread transfer remains within every enforced ceiling.

Provider	Metric	Main baseline	This PR	Impact	PR ceiling
Codex	Total thread wire	13.3 KiB	13.3 KiB	−34 B (−0.2%)	15.1 KiB	✅
Codex	Thread snapshot wire	6.9 KiB	6.9 KiB	−8 B (−0.1%)	7.3 KiB	✅
Codex	Live turn WebSocket wire	6.5 KiB	6.4 KiB	−26 B (−0.4%)	7.8 KiB	✅
Codex	Live turn WebSocket decoded	55.6 KiB	55.6 KiB	0 B (0.0%)	66.4 KiB	✅
Codex	Live turn messages	11	11	0 (0.0%)	21	✅
Claude	Total thread wire	13.3 KiB	13.4 KiB	+19 B (+0.1%)	15.1 KiB	✅
Claude	Thread snapshot wire	6.9 KiB	6.9 KiB	+3 B (+0.0%)	7.3 KiB	✅
Claude	Live turn WebSocket wire	6.4 KiB	6.5 KiB	+16 B (+0.2%)	7.8 KiB	✅
Claude	Live turn WebSocket decoded	56.4 KiB	56.4 KiB	0 B (0.0%)	66.4 KiB	✅
Claude	Live turn messages	11	11	0 (0.0%)	21	✅

Baseline: c034f51 · PR result: 4831d73 · Source CI: success

Scenario and decoded snapshot size

10 historical turns, 5 command tools per turn, 878.9 KiB retained MCP result per historical turn, and a 1.05 MiB retained result in the measured turn.

• Codex decoded thread snapshot: 109.4 KiB
• Claude decoded thread snapshot: 110.1 KiB

Updated in place by a trusted workflow. PR artifacts are strictly validated and never executed.

[macroscopeapp]

Approvability

Verdict: Not approved

Macroscope's review found this PR not approvable — This change adds end-to-end managed-tunnel recovery and automated Cloudflare tunnel deletion, including new signed relay APIs, persistence fields, startup behavior, and scheduled production cleanup. Its cross-service scope and external resource side effects warrant human review.

^{You can add or adjust custom eligibility rules. Learn more.}

[macroscopeapp]

Effect service conventions: four findings, all in newly added error-handling code (three Effect.catchTag usages and one startup Effect.catchCause). Details inline.

^{Posted via Macroscope — Effect Service Conventions}

[macroscopeapp]

One convention finding on the new retry predicate. Previously reported Effect.catchTag usages in ManagedEndpointReaper.ts, ManagedEndpointProvider.ts and the Effect.catchCause in server.ts are resolved in this head commit.

^{Posted via Macroscope — Effect Service Conventions}

[cursor]

Cursor Bugbot has reviewed your changes using high effort and found 2 potential issues.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit a9386bf. Configure here.}