diff --git a/docs/lambda-microvm/README.md b/docs/lambda-microvm/README.md index 2283bbd..f7d7935 100644 --- a/docs/lambda-microvm/README.md +++ b/docs/lambda-microvm/README.md @@ -200,6 +200,108 @@ again after expiry. Static assets and request-shaped handlers should remain on cheaper stateless delivery paths; this target is only the resident-server adapter. +#### Hosted-app control plane and preview gateway + +The service-side control plane is stateful-profile only. It snapshots the +source runtime session under its existing lock, records an exact checkpoint and +AWS idempotency intent in the fenced Redis registry, then launches/restores the +dedicated app-host VM on an isolated BullMQ queue. API pods enqueue lifecycle +work and proxy preview bytes; only worker pods need Lambda MicroVM IAM. +Start requests share the authenticated execution rate limiter. Before enabling +this feature broadly for untrusted multi-tenant traffic, add a plan-aware cap on +active hosted-app leases per owner; the initial feature-flagged slice relies on +the deployment's Lambda MicroVM quota as its hard fleet ceiling. + +Authenticated API contract: + +```http +POST /v1/hosted-apps +Content-Type: application/json + +{ + "runtime_session_hint": "conversation-123", + "app_id": "my-app", + "revision": "rev-1", + "language": "node", + "version": ">=22", + "entrypoint": "server.js", + "cwd": ".", + "args": [], + "env": {} +} +``` + +`GET /v1/hosted-apps/:app_id?runtime_session_hint=...` returns status and a +fresh five-minute `preview_url`; `DELETE` on the same resource terminates the +lease. A revision is immutable. Retrying the identical spec reasserts the +resident process; changing code or launch settings requires a new revision and +captures a new exact checkpoint. An ambiguous provider launch is replayed only +with its persisted token and can never be overwritten by a newer revision. + +Preview traffic uses a wildcard **unprivileged origin**, not a path below the +CodeAPI or LibreChat origin. Configure wildcard DNS and TLS such that +`*.apps.example.net` reaches the stateful CodeAPI API service, then set the bare +origin `https://apps.example.net`. The short-lived URL capability is exchanged +for an HttpOnly, Secure, host-only cookie and redirected to `/`; every app gets +its own `happ-.apps.example.net` origin, so absolute asset paths work +without exposing privileged-origin cookies to AI-generated JavaScript. Use a +dedicated registrable domain in production—do not set broad parent-domain +cookies that also match the app domain. + +Set these on both API and worker pods: + +| Env | Default | Meaning | +|---|---|---| +| `CODEAPI_HOSTED_APPS_ENABLED` | `false` | Enables the stateful-only lifecycle API, isolated preview gateway, and worker. | +| `CODEAPI_HOSTED_APP_CREDENTIAL_KEY` | — | Base64 of 32 random bytes; AES-GCM encrypts the AWS preview credential stored in Redis. | + +Set these only on API pods (the signing key must differ from the credential +key): + +| Env | Default | Meaning | +|---|---|---| +| `CODEAPI_HOSTED_APP_PREVIEW_SIGNING_KEY` | — | Different base64 32-byte key for owner-bound preview URL/cookie capabilities. | +| `CODEAPI_HOSTED_APP_PREVIEW_ORIGIN` | — | Bare HTTPS origin for wildcard app hosts, for example `https://apps.example.net`. | + +Set these on worker pods in addition to the ordinary stateful/checkpoint +configuration: + +| Env | Default | Meaning | +|---|---|---| +| `LAMBDA_MICROVM_APP_IMAGE_ARN` | — | Dedicated `lambda-microvm-app-host` image ARN. | +| `LAMBDA_MICROVM_APP_IMAGE_VERSION` | — | Required pinned image version. | +| `LAMBDA_MICROVM_APP_MAX_DURATION_SECONDS` | `28800` | App-VM hard lifetime. The control plane relaunches an immutable revision after expiry. | +| `LAMBDA_MICROVM_APP_IDLE_SECONDS` | `300` | Seconds idle before AWS suspends the VM. | +| `LAMBDA_MICROVM_APP_SUSPEND_SECONDS` | `900` | Seconds suspended before AWS terminates the VM. Suspended VMs still consume quota. | + +The pinned app-host image contract fixes the root-owned control/checkpoint +listener at port 8080, the resident app at port 3000, and resident readiness at +30 seconds. `RunMicrovm` cannot override the image environment; changing this +contract requires publishing a matching image and control-plane revision. + +Generate the two keys independently: + +```bash +openssl rand -base64 32 # CODEAPI_HOSTED_APP_CREDENTIAL_KEY +openssl rand -base64 32 # CODEAPI_HOSTED_APP_PREVIEW_SIGNING_KEY +``` + +The preview proxy strips CodeAPI authorization, cookies, forwarded identity, +caller-provided AWS headers, app `Set-Cookie`, app-controlled caching, +cross-origin policy, and external redirects. Gateway responses are private and +non-storable so an older revision cannot survive through the browser cache. It +supports streamed HTTP/SSE and same-origin redirects. WebSockets +are not part of this first resident adapter. A gateway-owned CSP constrains +fetches and subresources to the app origin and disables workers/service workers, +so one app revision cannot leave a persistent worker controlling a later +revision. Top-level app JavaScript can still navigate the owner's browser to an +external origin; treat this experimental viewer as owner-trusted. Before broad +untrusted enablement, serve app content from a separate origin inside a sandboxed +gateway wrapper. The request `env` map is persisted +with the immutable launch spec in the registry; it is configuration, not a +secret store. Add a dedicated secret-reference flow before passing application +secrets to hosted code. + ### 3. Generate the split execution-manifest keys The worker signs each execution manifest; the runner only receives the public diff --git a/helm/codeapi/README.md b/helm/codeapi/README.md index 9b3efbb..d0c169d 100644 --- a/helm/codeapi/README.md +++ b/helm/codeapi/README.md @@ -90,6 +90,15 @@ hardening variables documented in `docs/lambda-microvm/README.md`. This chart still renders its bundled sandbox-runner, though a Lambda worker does not call it; a platform-specific stateful deployment may omit that component. +Resident hosted apps are an opt-in capability of that stateful deployment. +Configure `CODEAPI_HOSTED_APPS_ENABLED`, the preview origin, and both hosted-app +keys through `api.extraEnv`; configure only the feature flag and credential key +on `workerSandbox.extraEnv`, along with the dedicated app image settings. +Wildcard DNS and TLS for the preview origin must route to the API service +separately from the normal CodeAPI host. See “Hosted-app control plane and preview gateway” in +`docs/lambda-microvm/README.md`; do not serve previews beneath the privileged +LibreChat/CodeAPI origin. + For an existing affinity/strict deployment from before execution profiles, first roll the new binary to API and worker pods with `CODEAPI_EXECUTION_PROFILE` still unset. The inferred stateful compatibility diff --git a/service/.env.example b/service/.env.example index e06ec4c..c9b5bea 100644 --- a/service/.env.example +++ b/service/.env.example @@ -9,6 +9,14 @@ REDIS_HOST=localhost REDIS_PORT=6379 REDIS_PASSWORD=mysecretpassword +# Stateful Lambda resident hosted apps (optional; see docs/lambda-microvm/README.md) +# CODEAPI_HOSTED_APPS_ENABLED=true +# LAMBDA_MICROVM_APP_IMAGE_ARN=arn:aws:lambda:REGION:ACCOUNT:microvm-image:codeapi-app-host +# LAMBDA_MICROVM_APP_IMAGE_VERSION=1 +# CODEAPI_HOSTED_APP_PREVIEW_ORIGIN=https://apps.example.net +# CODEAPI_HOSTED_APP_CREDENTIAL_KEY= +# CODEAPI_HOSTED_APP_PREVIEW_SIGNING_KEY= + # ----------------------------------------------------------------------------- # Stripe: https://shipfa.st/docs/features/payments # ----------------------------------------------------------------------------- @@ -20,4 +28,4 @@ STRIPE_WEBHOOK_SECRET= # Mailgun: https://shipfa.st/docs/features/emails # ----------------------------------------------------------------------------- # EMAIL_SERVER=smtp://postmaster@[mail.yourdomain.com]:[copied_password]@smtp.mailgun.org:587 (without the brackets) -EMAIL_SERVER= \ No newline at end of file +EMAIL_SERVER= diff --git a/service/openapi.yml b/service/openapi.yml index 913e780..78ba082 100644 --- a/service/openapi.yml +++ b/service/openapi.yml @@ -227,7 +227,155 @@ components: type: string enum: [default, stateful] + HostedAppStartRequest: + type: object + required: + - runtime_session_hint + - app_id + - revision + - language + - version + - entrypoint + properties: + runtime_session_hint: + type: string + maxLength: 128 + pattern: '^[A-Za-z0-9._:-]+$' + adapter: + type: string + enum: [resident] + default: resident + app_id: + type: string + maxLength: 64 + pattern: '^[A-Za-z0-9][A-Za-z0-9._-]{0,63}$' + revision: + type: string + maxLength: 128 + pattern: '^[A-Za-z0-9][A-Za-z0-9._:-]{0,127}$' + language: + type: string + version: + type: string + entrypoint: + type: string + description: Canonical path relative to the restored stateful workspace. + cwd: + type: string + default: . + args: + type: array + maxItems: 64 + items: + type: string + env: + type: object + additionalProperties: + type: string + + HostedAppStatus: + type: object + required: [app_id, revision, state, preview_id, updated_at] + properties: + app_id: + type: string + revision: + type: string + state: + type: string + enum: [starting, running, stopping, stopped, failed] + preview_id: + type: string + description: Opaque hosted-app lease identity. + preview_url: + type: string + format: uri + description: Five-minute owner capability exchange URL on the isolated app origin. + hard_deadline_at: + type: integer + format: int64 + updated_at: + type: integer + format: int64 + error: + type: string + paths: + /hosted-apps: + post: + summary: Start or reassert a resident hosted app + description: >- + Stateful-profile only. Captures an exact workspace checkpoint and runs + the immutable revision in a dedicated Lambda MicroVM app-host image. + operationId: startHostedApp + parameters: + - $ref: '#/components/parameters/ExpectedExecutionProfile' + requestBody: + required: true + content: + application/json: + schema: + $ref: '#/components/schemas/HostedAppStartRequest' + responses: + '200': + description: Hosted app is running + content: + application/json: + schema: + $ref: '#/components/schemas/HostedAppStatus' + '400': + $ref: '#/components/responses/BadRequest' + '409': + $ref: '#/components/responses/Conflict' + '429': + description: Hosted app start rate limit exceeded + '503': + description: Lifecycle or provider operation unavailable + content: + application/json: + schema: + $ref: '#/components/schemas/Error' + + /hosted-apps/{app_id}: + parameters: + - $ref: '#/components/parameters/ExpectedExecutionProfile' + - name: app_id + in: path + required: true + schema: + type: string + - name: runtime_session_hint + in: query + required: true + schema: + type: string + maxLength: 128 + pattern: '^[A-Za-z0-9._:-]+$' + get: + summary: Get hosted app status and a fresh preview URL + operationId: getHostedApp + responses: + '200': + description: Hosted app status + content: + application/json: + schema: + $ref: '#/components/schemas/HostedAppStatus' + '404': + description: Hosted app not found + delete: + summary: Terminate a hosted app lease + operationId: stopHostedApp + responses: + '200': + description: Hosted app stopped + content: + application/json: + schema: + $ref: '#/components/schemas/HostedAppStatus' + '404': + description: Hosted app not found + /exec: post: summary: Execute code diff --git a/service/src/api-server.ts b/service/src/api-server.ts index 7868982..deab2df 100644 --- a/service/src/api-server.ts +++ b/service/src/api-server.ts @@ -25,6 +25,8 @@ import { executionProfileMiddleware } from './middleware/execution-profile'; import { traceHttpRequest } from './telemetry'; import { env } from './config'; import logger from './logger'; +import hostedAppRouter from './hosted-app/router'; +import { hostedAppPreviewGateway } from './hosted-app/preview-gateway'; const { LOCAL_MODE: isLocalMode } = env; @@ -34,6 +36,7 @@ app.set('trust proxy', 1); app.use(traceHttpRequest('codeapi.api.request')); app.use(httpMetricsMiddleware); app.use(executionProfileMiddleware); +app.use(hostedAppPreviewGateway); const v1 = Router(); @@ -53,6 +56,7 @@ app.get('/v1/health', async (_, res) => { v1.use(isLocalMode ? localAuth : apiKeyAuth); +v1.use('/hosted-apps', hostedAppRouter); v1.use(serviceRouter); v1.use(programmaticRouter); diff --git a/service/src/config.ts b/service/src/config.ts index ecf3566..e5eebc7 100644 --- a/service/src/config.ts +++ b/service/src/config.ts @@ -403,6 +403,34 @@ export const env = { ), CHECKPOINT_TIMEOUT_MS: configuredNumber(process.env.CODEAPI_CHECKPOINT_TIMEOUT_MS, 60_000), CHECKPOINT_PREFIX: process.env.CODEAPI_CHECKPOINT_PREFIX ?? 'rtsx-checkpoints/', + /** Dedicated Lambda MicroVM resident-server fleet. This remains an explicit + * stateful-stack capability; the ordinary/default HTTP profile never starts + * or preserves application processes. */ + HOSTED_APPS_ENABLED: process.env.CODEAPI_HOSTED_APPS_ENABLED === 'true', + HOSTED_APP_IMAGE_ARN: process.env.LAMBDA_MICROVM_APP_IMAGE_ARN ?? '', + HOSTED_APP_IMAGE_VERSION: process.env.LAMBDA_MICROVM_APP_IMAGE_VERSION || undefined, + /* These values are part of the pinned app-host image contract. RunMicrovm + * cannot inject environment variables into the image, so exposing overrides + * here would only make the control plane call ports the runner never opened. */ + HOSTED_APP_CONTROL_PORT: 8080 as number, + HOSTED_APP_PREVIEW_PORT: 3000 as number, + HOSTED_APP_MAX_DURATION_SECONDS: configuredNumber( + process.env.LAMBDA_MICROVM_APP_MAX_DURATION_SECONDS, + 28_800, + ), + HOSTED_APP_IDLE_SECONDS: configuredNumber( + process.env.LAMBDA_MICROVM_APP_IDLE_SECONDS, + 300, + ), + HOSTED_APP_SUSPEND_SECONDS: configuredNumber( + process.env.LAMBDA_MICROVM_APP_SUSPEND_SECONDS, + 900, + ), + HOSTED_APP_START_TIMEOUT_MS: 30_000 as number, + HOSTED_APP_CREDENTIAL_KEY: process.env.CODEAPI_HOSTED_APP_CREDENTIAL_KEY ?? '', + HOSTED_APP_PREVIEW_ORIGIN: process.env.CODEAPI_HOSTED_APP_PREVIEW_ORIGIN ?? '', + HOSTED_APP_PREVIEW_SIGNING_KEY: + process.env.CODEAPI_HOSTED_APP_PREVIEW_SIGNING_KEY ?? '', }; const default_run_memory_limit = 256 * 1024 * 1024; diff --git a/service/src/hosted-app/control-plane.test.ts b/service/src/hosted-app/control-plane.test.ts new file mode 100644 index 0000000..66c27c6 --- /dev/null +++ b/service/src/hosted-app/control-plane.test.ts @@ -0,0 +1,551 @@ +import { describe, expect, test } from 'bun:test'; +import { randomBytes } from 'node:crypto'; +import type { CheckpointStore } from '../runtime-session/checkpoint-store'; +import type { MicrovmDescription } from '../runtime-session/lambda-client'; +import type { RuntimeSessionRecord } from '../runtime-session/registry'; +import { + HostedAppControlPlane, + HostedAppControlPlaneError, + hostedAppPublicStatus, + type HostedAppRegistry, +} from './control-plane'; +import { openHostedAppCredential } from './credential'; +import { + hostedAppLaunchClientToken, + hostedAppLaunchFingerprint, + hostedAppLaunchGenerationSeed, + hostedAppLaunchRequestFingerprint, + HostedAppMicrovmError, + type HostedAppMicrovmConfig, + type HostedAppMicrovmRuntime, +} from './microvm-runtime'; +import { hostedAppSpecFingerprint, type ResidentHostedAppSpec } from './spec'; + +const appSpec: ResidentHostedAppSpec = { + adapter: 'resident', + app_id: 'demo', + revision: 'rev-1', + language: 'node', + version: '>=22', + entrypoint: 'server.js', + cwd: '.', + args: [], + env: {}, +}; + +const runtimeConfig: HostedAppMicrovmConfig = { + imageArn: 'arn:aws:lambda:us-east-2:1:microvm-image:app-host', + imageVersion: '7', + executionRoleArn: 'arn:aws:iam::1:role/app-host', + ingressConnectorArns: ['arn:ingress/private'], + controlPort: 8080, + previewPort: 3000, + maximumDurationSeconds: 28_800, + idleSeconds: 300, + suspendedSeconds: 900, + authTokenTtlSeconds: 3_600, + launchTimeoutMs: 5_000, + healthTimeoutMs: 500, + appStartTimeoutMs: 2_000, + launchTps: 4, + tokenTps: 8, +}; + +class MemoryRegistry implements HostedAppRegistry { + record: RuntimeSessionRecord | null = null; + generation = hostedAppLaunchGenerationSeed(runtimeConfig); + writes: RuntimeSessionRecord[] = []; + allocations = 0; + + async waitForLock(): Promise { return 'lock'; } + async renewLock(): Promise<'held'> { return 'held'; } + async releaseLock(): Promise {} + async read(): Promise { + return this.record ? structuredClone(this.record) : null; + } + async write(record: RuntimeSessionRecord, token: string): Promise { + if (token !== 'lock') return false; + this.record = structuredClone(record); + this.writes.push(structuredClone(record)); + return true; + } + async allocateGeneration(): Promise { + this.allocations += 1; + return this.generation; + } +} + +class FakeRuntime { + readonly config = runtimeConfig; + launches: string[] = []; + starts: Array<{ vm: string; source: string; spec: ResidentHostedAppSpec }> = []; + healthChecks: string[] = []; + terminations: string[] = []; + previewMints = 0; + previewExpiresAt = 1_900_000_000_000; + startedAtMs?: number; + terminateSucceeds = true; + launchError?: Error; + healthError?: Error; + + async launch(clientToken: string): Promise<{ vm: MicrovmDescription; clientToken: string }> { + this.launches.push(clientToken); + if (this.launchError) throw this.launchError; + return { + vm: { + microvmId: 'vm-app-1', + endpoint: 'https://vm-app-1.test', + state: 'RUNNING', + startedAtMs: this.startedAtMs, + imageArn: runtimeConfig.imageArn, + imageVersion: runtimeConfig.imageVersion, + }, + clientToken, + }; + } + async waitForControlReady(vm: MicrovmDescription): Promise { + this.healthChecks.push(vm.microvmId); + if (this.healthError) { + const error = this.healthError; + this.healthError = undefined; + throw error; + } + } + async startResidentApp( + vm: MicrovmDescription, + source: string, + spec: ResidentHostedAppSpec, + ): Promise { + this.starts.push({ vm: vm.microvmId, source, spec }); + } + async previewToken() { + this.previewMints += 1; + return { + headerName: 'X-aws-proxy-auth', + token: `secret-token-${this.previewMints}`, + expiresAtMs: this.previewExpiresAt, + }; + } + async terminate(microvmId: string): Promise { + this.terminations.push(microvmId); + return this.terminateSucceeds; + } +} + +function fixture(options: { + registry?: MemoryRegistry; + runtime?: FakeRuntime; + restore?: 'restored' | 'absent' | 'fetch_failed' | 'push_failed'; +} = {}) { + const registry = options.registry ?? new MemoryRegistry(); + const runtime = options.runtime ?? new FakeRuntime(); + const credentialKey = randomBytes(32); + const captures: string[] = []; + const restores: string[] = []; + const control = new HostedAppControlPlane({ + registry, + runtime: runtime as unknown as HostedAppMicrovmRuntime, + checkpointStore: {} as CheckpointStore, + checkpointConfig: { + port: 8080, + authTokenTtlSeconds: 3_600, + maxBytes: 1024, + timeoutMs: 1_000, + }, + credentialKey, + lockWaitMs: 50, + lockTtlMs: 5_000, + captureCheckpoint: async source => { + captures.push(source); + return `rtsx-checkpoints/${source}/0001.tar.gz`; + }, + restoreCheckpoint: async args => { + restores.push(args.checkpointKey); + return options.restore ?? 'restored'; + }, + startHeartbeat: () => ({ stop() {} }), + now: () => 1_800_000_000_000, + }); + return { control, registry, runtime, credentialKey, captures, restores }; +} + +const input = { + hostedAppRuntimeId: 'happ_123', + sourceRuntimeSessionId: 'rt_source', + tenantId: 'tenant-1', + canonicalUserId: 'user-1', + spec: appSpec, + signal: new AbortController().signal, +}; + +function pendingRecord(): RuntimeSessionRecord { + const generation = hostedAppLaunchGenerationSeed(runtimeConfig); + return { + runtime_session_id: input.hostedAppRuntimeId, + tenant_id: input.tenantId, + canonical_user_id: input.canonicalUserId, + state: 'PENDING', + generation, + launched_at: 1_799_999_000_000, + hard_deadline_at: 1_800_010_000_000, + last_seen_at: 1_799_999_000_000, + image_arn: runtimeConfig.imageArn, + image_version: runtimeConfig.imageVersion, + port: runtimeConfig.previewPort, + launch_fingerprint: hostedAppLaunchFingerprint(runtimeConfig), + launch_request_fingerprint: hostedAppLaunchRequestFingerprint(runtimeConfig), + launch_client_token: hostedAppLaunchClientToken(input.hostedAppRuntimeId, generation), + hosted_app: { + source_runtime_session_id: input.sourceRuntimeSessionId, + app_id: appSpec.app_id, + revision: appSpec.revision, + spec_fingerprint: hostedAppSpecFingerprint(appSpec), + spec: appSpec, + checkpoint_key: 'rtsx-checkpoints/rt_source/exact.tar.gz', + }, + }; +} + +describe('HostedAppControlPlane', () => { + test('does not advertise an expired AWS lease as a running preview', () => { + const expired = { + ...pendingRecord(), + state: 'RUNNING' as const, + microvm_id: 'vm-expired', + endpoint: 'https://vm-expired.test', + hard_deadline_at: 99, + }; + expect(hostedAppPublicStatus(expired, 100).state).toBe('stopped'); + }); + + test('does not expose provider details persisted in an internal failure record', () => { + const failed = { + ...pendingRecord(), + state: 'TERMINATED' as const, + last_error: 'AccessDenied for arn:aws:iam::123456789012:role/private', + }; + const status = hostedAppPublicStatus(failed); + expect(status.state).toBe('failed'); + expect(status.error).toBe('Hosted app operation failed'); + expect(JSON.stringify(status)).not.toContain('123456789012'); + }); + + test('checkpoints, launches, restores, starts, and persists only a sealed preview credential', async () => { + const f = fixture(); + + const status = await f.control.start(input); + + expect(status).toMatchObject({ state: 'running', preview_id: 'happ_123', revision: 'rev-1' }); + expect(f.captures).toEqual(['rt_source']); + expect(f.restores).toEqual(['rtsx-checkpoints/rt_source/0001.tar.gz']); + expect(f.runtime.starts).toHaveLength(1); + expect(f.registry.writes.map(record => record.state)).toEqual(['PENDING', 'PENDING', 'RUNNING']); + expect(JSON.stringify(f.registry.record)).not.toContain('secret-token-1'); + const sealed = f.registry.record?.hosted_app?.preview_credential as string; + expect(openHostedAppCredential('happ_123', sealed, f.credentialKey).token).toBe('secret-token-1'); + }); + + test('derives the advertised lease deadline from the provider start time', async () => { + const runtime = new FakeRuntime(); + runtime.startedAtMs = 1_800_000_000_500; + const f = fixture({ runtime }); + + await f.control.start(input); + + expect(f.registry.record?.launched_at).toBe(runtime.startedAtMs); + expect(f.registry.record?.hard_deadline_at).toBe( + runtime.startedAtMs + runtimeConfig.maximumDurationSeconds * 1_000 - 60_000, + ); + }); + + test('replays an exact pending launch intent without taking a different checkpoint', async () => { + const registry = new MemoryRegistry(); + registry.record = pendingRecord(); + const f = fixture({ registry }); + + await f.control.start(input); + + expect(f.captures).toEqual([]); + expect(registry.allocations).toBe(0); + expect(f.runtime.launches).toEqual([pendingRecord().launch_client_token as string]); + expect(f.restores).toEqual(['rtsx-checkpoints/rt_source/exact.tar.gz']); + }); + + test('rejects changed launch settings under an immutable revision', async () => { + const registry = new MemoryRegistry(); + registry.record = pendingRecord(); + const f = fixture({ registry }); + const changed = { ...input, spec: { ...appSpec, args: ['--changed'] } }; + + const error = await f.control.start(changed).catch(value => value); + + expect(error).toBeInstanceOf(HostedAppControlPlaneError); + expect(error.code).toBe('hosted_app_revision_conflict'); + expect(f.runtime.launches).toEqual([]); + }); + + test('does not overwrite an ambiguous pending provider launch with a new revision', async () => { + const registry = new MemoryRegistry(); + registry.record = pendingRecord(); + const f = fixture({ registry }); + + const error = await f.control.start({ + ...input, + spec: { ...appSpec, revision: 'rev-2' }, + }).catch(value => value); + + expect(error.code).toBe('hosted_app_launch_in_progress'); + expect(f.captures).toEqual([]); + expect(registry.allocations).toBe(0); + expect(f.runtime.launches).toEqual([]); + }); + + test('reasserts an exact running revision and rotates its preview credential', async () => { + const registry = new MemoryRegistry(); + registry.record = { + ...pendingRecord(), + state: 'RUNNING', + microvm_id: 'vm-existing', + endpoint: 'https://vm-existing.test', + }; + const f = fixture({ registry }); + + await f.control.start(input); + + expect(f.captures).toEqual([]); + expect(f.runtime.launches).toEqual([]); + expect(f.runtime.healthChecks).toEqual(['vm-existing']); + expect(f.runtime.starts).toHaveLength(1); + expect(f.runtime.previewMints).toBe(1); + }); + + test('recycles a dead app VM and restores the exact immutable revision checkpoint', async () => { + const registry = new MemoryRegistry(); + registry.record = { + ...pendingRecord(), + state: 'RUNNING', + microvm_id: 'vm-dead', + endpoint: 'https://vm-dead.test', + }; + const runtime = new FakeRuntime(); + runtime.healthError = new HostedAppMicrovmError( + 'hosted_app_unhealthy', + 'endpoint is gone', + true, + ); + const f = fixture({ registry, runtime }); + + await f.control.start(input); + + expect(f.captures).toEqual([]); + expect(runtime.terminations).toEqual(['vm-dead']); + expect(runtime.launches).toHaveLength(1); + expect(f.restores).toEqual(['rtsx-checkpoints/rt_source/exact.tar.gz']); + }); + + test('records replacement cleanup before terminating the prior revision', async () => { + const registry = new MemoryRegistry(); + registry.record = { + ...pendingRecord(), + state: 'RUNNING', + microvm_id: 'vm-old-revision', + endpoint: 'https://vm-old-revision.test', + }; + registry.allocateGeneration = async () => { + throw new Error('generation store unavailable'); + }; + const runtime = new FakeRuntime(); + const f = fixture({ registry, runtime }); + + await f.control.start({ + ...input, + spec: { ...appSpec, revision: 'rev-2' }, + }).catch(() => undefined); + + expect(runtime.terminations).toEqual(['vm-old-revision']); + expect(registry.record).toMatchObject({ + state: 'TERMINATING', + microvm_id: 'vm-old-revision', + endpoint: 'https://vm-old-revision.test', + }); + }); + + test('terminates and retires a VM whose exact checkpoint cannot be restored', async () => { + const f = fixture({ restore: 'fetch_failed' }); + + const error = await f.control.start(input).catch(value => value); + + expect(error.code).toBe('hosted_app_restore_failed'); + expect(f.runtime.terminations).toEqual(['vm-app-1']); + expect(f.registry.record).toMatchObject({ state: 'TERMINATED', microvm_id: undefined }); + }); + + test('retains a failed launch VM id until termination can be confirmed', async () => { + const runtime = new FakeRuntime(); + runtime.terminateSucceeds = false; + const f = fixture({ restore: 'push_failed', runtime }); + + await f.control.start(input).catch(() => undefined); + + expect(f.registry.record).toMatchObject({ + state: 'TERMINATING', + microvm_id: 'vm-app-1', + endpoint: 'https://vm-app-1.test', + }); + }); + + test('a failed stop keeps the possibly-live app running and retryable', async () => { + const registry = new MemoryRegistry(); + registry.record = { + ...pendingRecord(), + state: 'RUNNING', + microvm_id: 'vm-existing', + endpoint: 'https://vm-existing.test', + }; + const runtime = new FakeRuntime(); + runtime.terminateSucceeds = false; + const f = fixture({ registry, runtime }); + + const error = await f.control.stop( + input.hostedAppRuntimeId, + input, + input.signal, + ).catch(value => value); + + expect(error.code).toBe('hosted_app_stop_failed'); + expect(registry.record).toMatchObject({ + state: 'RUNNING', + microvm_id: 'vm-existing', + last_error: 'Could not terminate the hosted app', + }); + }); + + test('stop replays an ambiguous pending launch before terminating it', async () => { + const registry = new MemoryRegistry(); + registry.record = pendingRecord(); + const f = fixture({ registry }); + + const status = await f.control.stop(input.hostedAppRuntimeId, input, input.signal); + + expect(status.state).toBe('stopped'); + expect(f.runtime.launches).toEqual([pendingRecord().launch_client_token as string]); + expect(f.runtime.terminations).toEqual(['vm-app-1']); + expect(registry.writes.map(record => record.state)).toEqual([ + 'PENDING', + 'TERMINATING', + 'TERMINATED', + ]); + expect(registry.record).toMatchObject({ + state: 'TERMINATED', + microvm_id: undefined, + endpoint: undefined, + }); + }); + + test('stop preserves an ambiguous pending intent when recovery fails', async () => { + const registry = new MemoryRegistry(); + registry.record = pendingRecord(); + const runtime = new FakeRuntime(); + runtime.launchError = new HostedAppMicrovmError( + 'hosted_app_launch_failed', + 'connection reset after provider accepted the request', + true, + ); + const f = fixture({ registry, runtime }); + + const error = await f.control.stop(input.hostedAppRuntimeId, input, input.signal) + .catch(value => value); + + expect(error.code).toBe('hosted_app_launch_failed'); + expect(registry.writes).toEqual([]); + expect(registry.record).toEqual(pendingRecord()); + }); + + test('stop does not overwrite a pending intent that current config cannot replay', async () => { + const registry = new MemoryRegistry(); + registry.record = { ...pendingRecord(), launch_fingerprint: 'different-image' }; + const f = fixture({ registry }); + + const error = await f.control.stop(input.hostedAppRuntimeId, input, input.signal) + .catch(value => value); + + expect(error.code).toBe('hosted_app_stop_pending'); + expect(error.transient).toBe(true); + expect(f.runtime.launches).toEqual([]); + expect(registry.writes).toEqual([]); + }); + + test('a failed cleanup never promotes a partial launch to running', async () => { + const registry = new MemoryRegistry(); + registry.record = { + ...pendingRecord(), + state: 'TERMINATING', + microvm_id: 'vm-partial', + endpoint: 'https://vm-partial.test', + }; + const runtime = new FakeRuntime(); + runtime.terminateSucceeds = false; + const f = fixture({ registry, runtime }); + + await f.control.stop(input.hostedAppRuntimeId, input, input.signal) + .catch(() => undefined); + + expect(registry.record).toMatchObject({ + state: 'TERMINATING', + microvm_id: 'vm-partial', + }); + }); + + test('retires definite boot exhaustion but preserves an ambiguous launch intent', async () => { + const definiteRuntime = new FakeRuntime(); + definiteRuntime.launchError = new HostedAppMicrovmError( + 'hosted_app_boot_failed', + 'both attempts terminated', + true, + ); + const definite = fixture({ runtime: definiteRuntime }); + await definite.control.start(input).catch(() => undefined); + expect(definite.registry.record).toMatchObject({ state: 'TERMINATED' }); + + const ambiguousRuntime = new FakeRuntime(); + ambiguousRuntime.launchError = new HostedAppMicrovmError( + 'hosted_app_launch_failed', + 'connection reset after write', + true, + ); + const ambiguous = fixture({ runtime: ambiguousRuntime }); + await ambiguous.control.start(input).catch(() => undefined); + expect(ambiguous.registry.record?.state).toBe('PENDING'); + expect(ambiguous.registry.record?.microvm_id).toBeUndefined(); + }); + + test('does not refresh a preview after its advertised hard deadline', async () => { + const registry = new MemoryRegistry(); + registry.record = { + ...pendingRecord(), + state: 'RUNNING', + microvm_id: 'vm-existing', + endpoint: 'https://vm-existing.test', + hard_deadline_at: 1_800_000_000_000, + }; + const f = fixture({ registry }); + + const error = await f.control.refreshPreview(input.hostedAppRuntimeId, input, input.signal) + .catch(value => value); + + expect(error.code).toBe('hosted_app_not_running'); + expect(f.runtime.previewMints).toBe(0); + }); + + test('rejects an already-expired credential instead of publishing it', async () => { + const runtime = new FakeRuntime(); + runtime.previewExpiresAt = 1_800_000_000_000; + const f = fixture({ runtime }); + + const error = await f.control.start(input).catch(value => value); + + expect(error.code).toBe('hosted_app_preview_unavailable'); + expect(runtime.terminations).toEqual(['vm-app-1']); + expect(f.registry.record).toMatchObject({ state: 'TERMINATED' }); + }); +}); diff --git a/service/src/hosted-app/control-plane.ts b/service/src/hosted-app/control-plane.ts new file mode 100644 index 0000000..ff4010e --- /dev/null +++ b/service/src/hosted-app/control-plane.ts @@ -0,0 +1,698 @@ +import type { CheckpointConfig } from '../runtime-session/checkpoint'; +import type { CheckpointStore } from '../runtime-session/checkpoint-store'; +import type { MicrovmDescription } from '../runtime-session/lambda-client'; +import type { RuntimeSessionRecord } from '../runtime-session/registry'; +import type { LockHeartbeat } from '../runtime-session/lock-heartbeat'; +import { sealHostedAppCredential } from './credential'; +import { + hostedAppLaunchClientToken, + hostedAppLaunchFingerprint, + hostedAppLaunchGenerationSeed, + hostedAppLaunchRequestFingerprint, + HostedAppMicrovmError, + type HostedAppMicrovmRuntime, +} from './microvm-runtime'; +import type { HostedAppPublicStatus } from './record'; +import { + hostedAppSpecFingerprint, + type ResidentHostedAppSpec, +} from './spec'; + +const HOSTED_APP_DEADLINE_HEADROOM_MS = 60_000; + +export class HostedAppControlPlaneError extends Error { + constructor( + readonly code: string, + message: string, + readonly status: number, + readonly transient = false, + readonly cause?: unknown, + ) { + super(message); + this.name = 'HostedAppControlPlaneError'; + } +} + +export interface HostedAppOwner { + tenantId: string; + canonicalUserId: string; +} + +export interface HostedAppStartInput extends HostedAppOwner { + hostedAppRuntimeId: string; + sourceRuntimeSessionId: string; + spec: ResidentHostedAppSpec; + signal: AbortSignal; +} + +export interface HostedAppRegistry { + waitForLock(runtimeId: string, args: { + waitMs: number; + ttlMs: number; + signal: AbortSignal; + }): Promise; + renewLock(runtimeId: string, token: string, ttlMs: number, args: { + signal: AbortSignal; + onLateLost: () => void; + }): Promise<'held' | 'lost' | 'error'>; + releaseLock(runtimeId: string, token: string): Promise; + read(runtimeId: string, args: { signal?: AbortSignal }): Promise; + write(record: RuntimeSessionRecord, token: string, args: { + signal?: AbortSignal; + }): Promise; + allocateGeneration(runtimeId: string, seed: number, args: { + signal: AbortSignal; + }): Promise; +} + +export interface HostedAppControlPlaneDeps { + registry: HostedAppRegistry; + runtime: HostedAppMicrovmRuntime; + checkpointStore: CheckpointStore; + checkpointConfig: CheckpointConfig; + credentialKey: Buffer; + lockWaitMs: number; + lockTtlMs: number; + captureCheckpoint( + runtimeSessionId: string, + owner: HostedAppOwner, + signal: AbortSignal, + ): Promise; + restoreCheckpoint(args: { + runtimeSessionId: string; + checkpointKey: string; + vm: MicrovmDescription; + store: CheckpointStore; + config: CheckpointConfig; + signal: AbortSignal; + }): Promise<'restored' | 'absent' | 'fetch_failed' | 'push_failed'>; + startHeartbeat(args: { + renew: () => Promise<'held' | 'lost' | 'error'>; + fence: AbortController; + ttlMs: number; + }): LockHeartbeat; + now?: () => number; +} + +function publicState( + record: RuntimeSessionRecord, + now: number, +): HostedAppPublicStatus['state'] { + if ( + record.state === 'RUNNING' + && record.hard_deadline_at != null + && record.hard_deadline_at <= now + ) return 'stopped'; + if (record.state === 'RUNNING') return 'running'; + if (record.state === 'PENDING') return 'starting'; + if (record.state === 'TERMINATED') { + return record.last_error ? 'failed' : 'stopped'; + } + if (record.state === 'TERMINATING') return 'stopping'; + return 'starting'; +} + +export function hostedAppPublicStatus( + record: RuntimeSessionRecord, + now = Date.now(), +): HostedAppPublicStatus { + const app = record.hosted_app; + if (!app) { + throw new HostedAppControlPlaneError( + 'hosted_app_record_invalid', + 'Hosted app registry record is missing app metadata', + 503, + true, + ); + } + return { + app_id: app.app_id, + revision: app.revision, + state: publicState(record, now), + preview_id: record.runtime_session_id, + hard_deadline_at: record.hard_deadline_at, + updated_at: record.last_seen_at, + ...(record.last_error ? { + /* Provider, endpoint, and checkpoint errors stay in the internal record + * and worker logs. Status is a public API and must not replay them. */ + error: record.state === 'TERMINATING' + ? 'Hosted app cleanup is pending' + : record.state === 'RUNNING' + ? 'Hosted app could not be stopped' + : 'Hosted app operation failed', + } : {}), + }; +} + +export function assertHostedAppOwned( + record: RuntimeSessionRecord, + owner: HostedAppOwner, + sourceRuntimeSessionId?: string, +): void { + if ( + record.tenant_id !== owner.tenantId + || record.canonical_user_id !== owner.canonicalUserId + || (sourceRuntimeSessionId != null + && record.hosted_app?.source_runtime_session_id !== sourceRuntimeSessionId) + ) { + /* Do not disclose whether another owner's opaque id exists. */ + throw new HostedAppControlPlaneError('hosted_app_not_found', 'Hosted app not found', 404); + } +} + +export class HostedAppControlPlane { + private readonly now: () => number; + + constructor(private readonly deps: HostedAppControlPlaneDeps) { + this.now = deps.now ?? Date.now; + } + + async start(input: HostedAppStartInput): Promise { + return this.withLease(input.hostedAppRuntimeId, input.signal, async (signal, lockToken) => { + const fingerprint = hostedAppSpecFingerprint(input.spec); + let prior = await this.deps.registry.read(input.hostedAppRuntimeId, { signal }); + if (prior) { + assertHostedAppOwned(prior, input, input.sourceRuntimeSessionId); + if ( + prior.hosted_app?.revision === input.spec.revision + && prior.hosted_app.spec_fingerprint !== fingerprint + ) { + throw new HostedAppControlPlaneError( + 'hosted_app_revision_conflict', + 'An app revision is immutable; use a new revision for changed launch settings', + 409, + ); + } + } + + const exactRevision = prior?.hosted_app?.revision === input.spec.revision + && prior.hosted_app.spec_fingerprint === fingerprint; + if ( + exactRevision + && prior?.state === 'RUNNING' + && prior.microvm_id + && prior.endpoint + && (prior.hard_deadline_at == null + || prior.hard_deadline_at > this.now() + HOSTED_APP_DEADLINE_HEADROOM_MS) + ) { + /* Reassert both the runner and credential. This resumes a suspended VM, + * heals a dead resident process, and never trusts an expired token. */ + const vm = this.recordedVm(prior); + try { + await this.deps.runtime.waitForControlReady(vm, signal); + await this.deps.runtime.startResidentApp( + vm, + input.sourceRuntimeSessionId, + input.spec, + signal, + ); + return hostedAppPublicStatus( + await this.persistPreviewCredential(prior, vm, lockToken, signal), + this.now(), + ); + } catch (error) { + if (!(error instanceof HostedAppMicrovmError) || !error.transient) throw error; + const terminating: RuntimeSessionRecord = { + ...prior, + state: 'TERMINATING', + last_seen_at: this.now(), + }; + /* Record the destructive transition before acting on AWS. If the + * subsequent generation allocation or Redis write fails, callers see + * a recoverable cleanup state rather than a stale RUNNING endpoint + * for a VM we already terminated. */ + await this.writeOrFence(terminating, lockToken, signal); + const terminated = await this.deps.runtime.terminate(vm.microvmId); + if (!terminated) throw error; + prior = { + ...terminating, + microvm_id: undefined, + endpoint: undefined, + state: 'TERMINATED', + last_seen_at: this.now(), + last_error: error.message, + }; + } + } + + const replayPending = Boolean( + exactRevision + && prior?.state === 'PENDING' + && !prior.microvm_id + && prior.launch_client_token + && prior.hosted_app?.checkpoint_key + && prior.launch_fingerprint === hostedAppLaunchFingerprint(this.deps.runtime.config) + && prior.launch_request_fingerprint + === hostedAppLaunchRequestFingerprint(this.deps.runtime.config) + && prior.hard_deadline_at != null + && prior.hard_deadline_at > this.now() + HOSTED_APP_DEADLINE_HEADROOM_MS + ); + const pendingProviderCouldStillBeLive = Boolean( + prior?.state === 'PENDING' + && !prior.microvm_id + && ( + prior.hard_deadline_at == null + || prior.hard_deadline_at + + HOSTED_APP_DEADLINE_HEADROOM_MS + + this.deps.runtime.config.launchTimeoutMs > this.now() + ) + ); + if ( + pendingProviderCouldStillBeLive + && !replayPending + ) { + /* A provider call may have succeeded before its response was lost. We + * can recover only by replaying the exact revision/config token; never + * overwrite that intent with a different revision and orphan a VM. */ + throw new HostedAppControlPlaneError( + 'hosted_app_launch_in_progress', + 'The prior hosted app launch must be recovered before it can be replaced', + 409, + true, + ); + } + /* An exact revision always reuses its immutable source snapshot. A dead + * app VM must not silently pick up later workspace edits under the same + * revision; changed bytes require a new revision. */ + const checkpointKey = exactRevision && prior?.hosted_app?.checkpoint_key + ? prior.hosted_app.checkpoint_key + : await this.deps.captureCheckpoint(input.sourceRuntimeSessionId, input, signal); + + if (prior?.microvm_id) { + const terminating: RuntimeSessionRecord = { + ...prior, + state: 'TERMINATING', + last_seen_at: this.now(), + }; + await this.writeOrFence(terminating, lockToken, signal); + const terminated = await this.deps.runtime.terminate(prior.microvm_id); + if (!terminated) { + throw new HostedAppControlPlaneError( + 'hosted_app_replace_failed', + 'Could not terminate the previous hosted app revision', + 503, + true, + ); + } + prior = terminating; + } + + const generation = replayPending && prior + ? prior.generation + : await this.deps.registry.allocateGeneration( + input.hostedAppRuntimeId, + hostedAppLaunchGenerationSeed(this.deps.runtime.config), + { signal }, + ); + const clientToken = replayPending && prior?.launch_client_token + ? prior.launch_client_token + : hostedAppLaunchClientToken(input.hostedAppRuntimeId, generation); + const launchedAt = replayPending && prior?.launched_at + ? prior.launched_at + : this.now(); + const hardDeadlineAt = replayPending && prior?.hard_deadline_at + ? prior.hard_deadline_at + : launchedAt + + this.deps.runtime.config.maximumDurationSeconds * 1_000 + - HOSTED_APP_DEADLINE_HEADROOM_MS; + let launchIntent: RuntimeSessionRecord = { + runtime_session_id: input.hostedAppRuntimeId, + tenant_id: input.tenantId, + canonical_user_id: input.canonicalUserId, + port: this.deps.runtime.config.previewPort, + image_arn: this.deps.runtime.config.imageArn, + image_version: this.deps.runtime.config.imageVersion, + launch_fingerprint: hostedAppLaunchFingerprint(this.deps.runtime.config), + launch_client_token: clientToken, + launch_request_fingerprint: hostedAppLaunchRequestFingerprint(this.deps.runtime.config), + state: 'PENDING', + generation, + launched_at: launchedAt, + last_seen_at: this.now(), + hard_deadline_at: hardDeadlineAt, + hosted_app: { + source_runtime_session_id: input.sourceRuntimeSessionId, + app_id: input.spec.app_id, + revision: input.spec.revision, + spec_fingerprint: fingerprint, + spec: input.spec, + checkpoint_key: checkpointKey, + }, + }; + await this.writeOrFence(launchIntent, lockToken, signal); + + let vm: MicrovmDescription | undefined; + try { + const launched = await this.deps.runtime.launch(clientToken, signal); + vm = launched.vm; + const providerStartedAt = Number.isFinite(vm.startedAtMs) + ? vm.startedAtMs + : undefined; + launchIntent = { + ...launchIntent, + launch_client_token: launched.clientToken, + microvm_id: vm.microvmId, + endpoint: vm.endpoint, + image_arn: vm.imageArn ?? launchIntent.image_arn, + image_version: vm.imageVersion ?? launchIntent.image_version, + launched_at: providerStartedAt ?? launchIntent.launched_at, + hard_deadline_at: providerStartedAt == null + ? launchIntent.hard_deadline_at + : providerStartedAt + + this.deps.runtime.config.maximumDurationSeconds * 1_000 + - HOSTED_APP_DEADLINE_HEADROOM_MS, + last_seen_at: this.now(), + }; + await this.writeOrFence(launchIntent, lockToken, signal); + await this.deps.runtime.waitForControlReady(vm, signal); + const restored = await this.deps.restoreCheckpoint({ + runtimeSessionId: input.sourceRuntimeSessionId, + checkpointKey, + vm, + store: this.deps.checkpointStore, + config: this.deps.checkpointConfig, + signal, + }); + if (restored !== 'restored') { + throw new HostedAppControlPlaneError( + 'hosted_app_restore_failed', + `Could not restore the exact app workspace revision (${restored})`, + 503, + true, + ); + } + await this.deps.runtime.startResidentApp( + vm, + input.sourceRuntimeSessionId, + input.spec, + signal, + ); + const running: RuntimeSessionRecord = { + ...launchIntent, + state: 'RUNNING', + last_seen_at: this.now(), + }; + return hostedAppPublicStatus( + await this.persistPreviewCredential(running, vm, lockToken, signal), + this.now(), + ); + } catch (error) { + const terminated = vm ? await this.deps.runtime.terminate(vm.microvmId) : false; + /* Preserve a no-id PENDING intent after an ambiguous provider failure: + * the successor replays the same token. Once a VM id is known, it was + * terminated above and the intent must be retired. */ + if (vm) { + const failed: RuntimeSessionRecord = { + ...launchIntent, + microvm_id: terminated ? undefined : vm.microvmId, + endpoint: terminated ? undefined : vm.endpoint, + state: terminated ? 'TERMINATED' : 'TERMINATING', + last_seen_at: this.now(), + last_error: terminated + ? (error instanceof Error ? error.message : 'Hosted app launch failed') + : 'Hosted app launch failed and its MicroVM still requires termination', + }; + await this.deps.registry.write(failed, lockToken, { signal }) + .catch(() => false); + } else if ( + error instanceof HostedAppMicrovmError + && (error.code === 'hosted_app_boot_failed' || !error.transient) + ) { + /* No VM id escaped launch(), and these outcomes prove AWS did not + * leave a live resource: deterministic rejection, or both boot + * attempts reached a terminal state. Let the next request allocate a + * fresh generation instead of replaying dead tokens forever. */ + await this.deps.registry.write({ + ...launchIntent, + state: 'TERMINATED', + last_seen_at: this.now(), + last_error: error.message, + }, lockToken, { signal }).catch(() => false); + } + throw error; + } + }); + } + + async status( + hostedAppRuntimeId: string, + owner: HostedAppOwner, + signal: AbortSignal, + ): Promise { + const record = await this.deps.registry.read(hostedAppRuntimeId, { signal }); + if (!record?.hosted_app) { + throw new HostedAppControlPlaneError('hosted_app_not_found', 'Hosted app not found', 404); + } + assertHostedAppOwned(record, owner); + return hostedAppPublicStatus(record, this.now()); + } + + async stop( + hostedAppRuntimeId: string, + owner: HostedAppOwner, + signal: AbortSignal, + ): Promise { + return this.withLease(hostedAppRuntimeId, signal, async (leaseSignal, lockToken) => { + let record = await this.deps.registry.read(hostedAppRuntimeId, { signal: leaseSignal }); + if (!record?.hosted_app) { + throw new HostedAppControlPlaneError('hosted_app_not_found', 'Hosted app not found', 404); + } + assertHostedAppOwned(record, owner); + if ( + record.state === 'PENDING' + && !record.microvm_id + && ( + record.hard_deadline_at == null + || record.hard_deadline_at + + HOSTED_APP_DEADLINE_HEADROOM_MS + + this.deps.runtime.config.launchTimeoutMs > this.now() + ) + ) { + const replayable = Boolean( + record.launch_client_token + && record.launch_fingerprint === hostedAppLaunchFingerprint(this.deps.runtime.config) + && record.launch_request_fingerprint + === hostedAppLaunchRequestFingerprint(this.deps.runtime.config) + ); + if (!replayable) { + /* The provider may still have accepted this request. Retain the + * intent until its maximum provider lifetime passes rather than + * claiming it was stopped and losing the only safe recovery key. */ + throw new HostedAppControlPlaneError( + 'hosted_app_stop_pending', + 'The pending hosted app launch must be recovered before it can be stopped', + 409, + true, + ); + } + const launched = await this.deps.runtime.launch( + record.launch_client_token as string, + leaseSignal, + ); + const recovered: RuntimeSessionRecord = { + ...record, + launch_client_token: launched.clientToken, + microvm_id: launched.vm.microvmId, + endpoint: launched.vm.endpoint, + image_arn: launched.vm.imageArn ?? record.image_arn, + image_version: launched.vm.imageVersion ?? record.image_version, + launched_at: Number.isFinite(launched.vm.startedAtMs) + ? launched.vm.startedAtMs + : record.launched_at, + hard_deadline_at: Number.isFinite(launched.vm.startedAtMs) + ? (launched.vm.startedAtMs as number) + + this.deps.runtime.config.maximumDurationSeconds * 1_000 + - HOSTED_APP_DEADLINE_HEADROOM_MS + : record.hard_deadline_at, + last_seen_at: this.now(), + }; + try { + await this.writeOrFence(recovered, lockToken, leaseSignal); + } catch (error) { + /* A recovered VM must never escape merely because we lost the Redis + * fence while recording its id. */ + await this.deps.runtime.terminate(launched.vm.microvmId).catch(() => false); + throw error; + } + record = recovered; + } + if (record.microvm_id) { + const terminating: RuntimeSessionRecord = { + ...record, + state: 'TERMINATING', + last_seen_at: this.now(), + }; + await this.writeOrFence(terminating, lockToken, leaseSignal); + if (!await this.deps.runtime.terminate(record.microvm_id)) { + await this.writeOrFence({ + ...record, + state: record.state === 'RUNNING' ? 'RUNNING' : 'TERMINATING', + last_seen_at: this.now(), + last_error: 'Could not terminate the hosted app', + }, lockToken, leaseSignal); + throw new HostedAppControlPlaneError( + 'hosted_app_stop_failed', + 'Could not terminate the hosted app', + 503, + true, + ); + } + } + const stopped: RuntimeSessionRecord = { + ...record, + microvm_id: undefined, + endpoint: undefined, + state: 'TERMINATED', + last_seen_at: this.now(), + last_error: undefined, + hosted_app: { + ...(record.hosted_app as NonNullable), + preview_credential: undefined, + preview_credential_expires_at: undefined, + }, + }; + await this.writeOrFence(stopped, lockToken, leaseSignal); + return hostedAppPublicStatus(stopped, this.now()); + }); + } + + async refreshPreview( + hostedAppRuntimeId: string, + owner: HostedAppOwner, + signal: AbortSignal, + ): Promise { + return this.withLease(hostedAppRuntimeId, signal, async (leaseSignal, lockToken) => { + const record = await this.deps.registry.read(hostedAppRuntimeId, { signal: leaseSignal }); + if (!record?.hosted_app) { + throw new HostedAppControlPlaneError( + 'hosted_app_not_running', + 'Hosted app is not running', + 409, + true, + ); + } + assertHostedAppOwned(record, owner); + if ( + record.state !== 'RUNNING' + || !record.microvm_id + || !record.endpoint + || record.hard_deadline_at == null + || record.hard_deadline_at <= this.now() + ) { + throw new HostedAppControlPlaneError( + 'hosted_app_not_running', + 'Hosted app is not running', + 409, + true, + ); + } + return hostedAppPublicStatus( + await this.persistPreviewCredential( + record, + this.recordedVm(record), + lockToken, + leaseSignal, + ), + this.now(), + ); + }); + } + + private recordedVm(record: RuntimeSessionRecord): MicrovmDescription { + return { + microvmId: record.microvm_id as string, + endpoint: record.endpoint, + state: 'RUNNING', + imageArn: record.image_arn, + imageVersion: record.image_version, + }; + } + + private async persistPreviewCredential( + record: RuntimeSessionRecord, + vm: MicrovmDescription, + lockToken: string, + signal: AbortSignal, + ): Promise { + const credential = await this.deps.runtime.previewToken(vm.microvmId, signal); + if (credential.expiresAtMs <= this.now()) { + throw new HostedAppControlPlaneError( + 'hosted_app_preview_unavailable', + 'Hosted app preview credential is unavailable', + 503, + true, + ); + } + const running: RuntimeSessionRecord = { + ...record, + state: 'RUNNING', + last_seen_at: this.now(), + last_error: undefined, + hosted_app: { + ...(record.hosted_app as NonNullable), + preview_credential: sealHostedAppCredential( + record.runtime_session_id, + credential, + this.deps.credentialKey, + ), + preview_credential_expires_at: credential.expiresAtMs, + }, + }; + await this.writeOrFence(running, lockToken, signal); + return running; + } + + private async writeOrFence( + record: RuntimeSessionRecord, + lockToken: string, + signal: AbortSignal, + ): Promise { + if (!await this.deps.registry.write(record, lockToken, { signal })) { + signal.throwIfAborted(); + throw new HostedAppControlPlaneError( + 'hosted_app_fenced', + 'Lost the hosted app lease while updating it', + 409, + true, + ); + } + } + + private async withLease( + runtimeId: string, + callerSignal: AbortSignal, + operation: (signal: AbortSignal, lockToken: string) => Promise, + ): Promise { + const lockToken = await this.deps.registry.waitForLock(runtimeId, { + waitMs: this.deps.lockWaitMs, + ttlMs: this.deps.lockTtlMs, + signal: callerSignal, + }); + if (!lockToken) { + throw new HostedAppControlPlaneError( + 'hosted_app_busy', + 'Another hosted app transition is in progress', + 409, + true, + ); + } + const fence = new AbortController(); + const signal = AbortSignal.any([callerSignal, fence.signal]); + const heartbeat = this.deps.startHeartbeat({ + renew: () => this.deps.registry.renewLock( + runtimeId, + lockToken, + this.deps.lockTtlMs, + { signal: callerSignal, onLateLost: () => fence.abort() }, + ), + fence, + ttlMs: this.deps.lockTtlMs, + }); + try { + return await operation(signal, lockToken); + } finally { + heartbeat.stop(); + await this.deps.registry.releaseLock(runtimeId, lockToken); + } + } +} diff --git a/service/src/hosted-app/credential.test.ts b/service/src/hosted-app/credential.test.ts new file mode 100644 index 0000000..e14236b --- /dev/null +++ b/service/src/hosted-app/credential.test.ts @@ -0,0 +1,48 @@ +import { describe, expect, test } from 'bun:test'; +import { + openHostedAppCredential, + parseHostedAppCredentialKey, + sealHostedAppCredential, +} from './credential'; + +const key = Buffer.alloc(32, 7); +const credential = { + headerName: 'X-aws-proxy-auth', + token: 'secret-jwe', + expiresAtMs: 1_787_300_000_000, +}; + +describe('hosted-app credential envelope', () => { + test('round-trips a preview token without emitting plaintext', () => { + const sealed = sealHostedAppCredential('happ_owner_a', credential, key); + expect(sealed).not.toContain(credential.token); + expect(openHostedAppCredential('happ_owner_a', sealed, key)).toEqual(credential); + }); + + test('binds ciphertext to one hosted-app identity', () => { + const sealed = sealHostedAppCredential('happ_owner_a', credential, key); + expect(() => openHostedAppCredential('happ_owner_b', sealed, key)).toThrow( + 'could not be authenticated', + ); + }); + + test('rejects tampering and malformed key material', () => { + const sealed = sealHostedAppCredential('happ_owner_a', credential, key); + const parts = sealed.split('.'); + const tag = Buffer.from(parts[2] as string, 'base64url'); + tag[0] ^= 1; + parts[2] = tag.toString('base64url'); + const tampered = parts.join('.'); + expect(() => openHostedAppCredential('happ_owner_a', tampered, key)).toThrow( + 'could not be authenticated', + ); + expect(() => openHostedAppCredential( + 'happ_owner_a', + `${sealed.slice(0, -1)}${sealed.endsWith('A') ? 'B' : 'A'}`, + key, + )).toThrow('could not be authenticated'); + expect(() => parseHostedAppCredentialKey(Buffer.alloc(31).toString('base64'))) + .toThrow('exactly 32 bytes'); + expect(parseHostedAppCredentialKey(key.toString('base64'))).toEqual(key); + }); +}); diff --git a/service/src/hosted-app/credential.ts b/service/src/hosted-app/credential.ts new file mode 100644 index 0000000..836fc0f --- /dev/null +++ b/service/src/hosted-app/credential.ts @@ -0,0 +1,98 @@ +import { + createCipheriv, + createDecipheriv, + randomBytes, +} from 'node:crypto'; +import type { MicrovmAuthToken } from '../runtime-session/lambda-client'; + +const TOKEN_FORMAT = 'v1'; +const IV_BYTES = 12; +const TAG_BYTES = 16; + +export class HostedAppCredentialError extends Error {} + +export function parseHostedAppCredentialKey(raw: string): Buffer { + let key: Buffer; + try { + key = Buffer.from(raw, 'base64'); + } catch { + throw new HostedAppCredentialError('CODEAPI_HOSTED_APP_CREDENTIAL_KEY must be base64'); + } + if (key.length !== 32 || key.toString('base64').replace(/=+$/, '') !== raw.trim().replace(/=+$/, '')) { + throw new HostedAppCredentialError( + 'CODEAPI_HOSTED_APP_CREDENTIAL_KEY must encode exactly 32 bytes', + ); + } + return key; +} + +export function sealHostedAppCredential( + hostedAppRuntimeId: string, + credential: MicrovmAuthToken, + key: Buffer, +): string { + if (key.length !== 32) throw new HostedAppCredentialError('credential key must be 32 bytes'); + const iv = randomBytes(IV_BYTES); + const cipher = createCipheriv('aes-256-gcm', key, iv); + cipher.setAAD(Buffer.from(hostedAppRuntimeId, 'utf8')); + const plaintext = Buffer.from(JSON.stringify(credential), 'utf8'); + const ciphertext = Buffer.concat([cipher.update(plaintext), cipher.final()]); + const tag = cipher.getAuthTag(); + return [ + TOKEN_FORMAT, + iv.toString('base64url'), + tag.toString('base64url'), + ciphertext.toString('base64url'), + ].join('.'); +} + +export function openHostedAppCredential( + hostedAppRuntimeId: string, + sealed: string, + key: Buffer, +): MicrovmAuthToken { + if (key.length !== 32) throw new HostedAppCredentialError('credential key must be 32 bytes'); + const [version, ivRaw, tagRaw, ciphertextRaw, extra] = sealed.split('.'); + if (version !== TOKEN_FORMAT || !ivRaw || !tagRaw || !ciphertextRaw || extra !== undefined) { + throw new HostedAppCredentialError('hosted-app credential is malformed'); + } + try { + const decode = (raw: string, expectedBytes?: number): Buffer => { + if (!/^[A-Za-z0-9_-]+$/.test(raw)) throw new Error('credential encoding is malformed'); + const decoded = Buffer.from(raw, 'base64url'); + if ( + decoded.toString('base64url') !== raw + || (expectedBytes != null && decoded.length !== expectedBytes) + ) { + throw new Error('credential encoding is malformed'); + } + return decoded; + }; + const iv = decode(ivRaw, IV_BYTES); + const tag = decode(tagRaw, TAG_BYTES); + const ciphertext = decode(ciphertextRaw); + const decipher = createDecipheriv('aes-256-gcm', key, iv); + decipher.setAAD(Buffer.from(hostedAppRuntimeId, 'utf8')); + decipher.setAuthTag(tag); + const plaintext = Buffer.concat([ + decipher.update(ciphertext), + decipher.final(), + ]).toString('utf8'); + const parsed = JSON.parse(plaintext) as Partial; + if ( + typeof parsed.headerName !== 'string' + || parsed.headerName.length === 0 + || typeof parsed.token !== 'string' + || parsed.token.length === 0 + || !Number.isSafeInteger(parsed.expiresAtMs) + || (parsed.expiresAtMs as number) <= 0 + ) { + throw new Error('credential payload is invalid'); + } + return parsed as MicrovmAuthToken; + } catch (error) { + throw new HostedAppCredentialError( + `hosted-app credential could not be authenticated: ${(error as Error).message}`, + ); + } +} diff --git a/service/src/hosted-app/factory.ts b/service/src/hosted-app/factory.ts new file mode 100644 index 0000000..9d3cb6e --- /dev/null +++ b/service/src/hosted-app/factory.ts @@ -0,0 +1,142 @@ +import { env } from '../config'; +import { checkpointSession, restoreSession } from '../runtime-session/checkpoint'; +import { MinioCheckpointStore } from '../runtime-session/checkpoint-store'; +import { AwsLambdaMicrovmClient } from '../runtime-session/lambda-client-aws'; +import { startRuntimeSessionLockHeartbeat } from '../runtime-session/lock-heartbeat'; +import { + allocateRuntimeSessionGeneration, + RUNTIME_SESSION_LOCK_TTL_MS, + readRuntimeSessionRecord, + releaseRuntimeSessionLock, + renewRuntimeSessionLock, + waitForRuntimeSessionLock, + writeRuntimeSessionRecord, +} from '../runtime-session/registry'; +import { + HostedAppControlPlane, + HostedAppControlPlaneError, +} from './control-plane'; +import { parseHostedAppCredentialKey } from './credential'; +import { + HostedAppMicrovmRuntime, + normalizeHostedAppMicrovmEndpoint, +} from './microvm-runtime'; +import { captureHostedAppSourceCheckpoint } from './source-checkpoint'; + +let controlPlane: HostedAppControlPlane | undefined; + +export function getHostedAppControlPlane(): HostedAppControlPlane { + if (controlPlane) return controlPlane; + if (!env.HOSTED_APPS_ENABLED) { + throw new HostedAppControlPlaneError( + 'hosted_apps_disabled', + 'Hosted apps are not enabled on this execution profile', + 404, + ); + } + + const client = new AwsLambdaMicrovmClient({ region: env.LAMBDA_MICROVM_REGION }); + const runtime = new HostedAppMicrovmRuntime(client, { + imageArn: env.HOSTED_APP_IMAGE_ARN, + imageVersion: env.HOSTED_APP_IMAGE_VERSION as string, + executionRoleArn: env.LAMBDA_MICROVM_EXECUTION_ROLE_ARN, + logGroup: env.LAMBDA_MICROVM_LOG_GROUP, + ingressConnectorArns: env.LAMBDA_MICROVM_INGRESS_CONNECTOR_ARNS, + controlPort: env.HOSTED_APP_CONTROL_PORT, + previewPort: env.HOSTED_APP_PREVIEW_PORT, + maximumDurationSeconds: env.HOSTED_APP_MAX_DURATION_SECONDS, + idleSeconds: env.HOSTED_APP_IDLE_SECONDS, + suspendedSeconds: env.HOSTED_APP_SUSPEND_SECONDS, + authTokenTtlSeconds: env.LAMBDA_MICROVM_AUTH_TOKEN_TTL_SECONDS, + launchTimeoutMs: env.LAMBDA_MICROVM_LAUNCH_TIMEOUT_MS, + healthTimeoutMs: env.LAMBDA_MICROVM_HEALTH_TIMEOUT_MS, + appStartTimeoutMs: env.HOSTED_APP_START_TIMEOUT_MS, + launchTps: env.LAMBDA_MICROVM_LAUNCH_TPS, + tokenTps: env.LAMBDA_MICROVM_TOKEN_TPS, + }); + const store = new MinioCheckpointStore(); + const checkpointConfig = { + port: env.HOSTED_APP_CONTROL_PORT, + authTokenTtlSeconds: env.LAMBDA_MICROVM_AUTH_TOKEN_TTL_SECONDS, + maxBytes: env.CHECKPOINT_MAX_BYTES, + timeoutMs: env.CHECKPOINT_TIMEOUT_MS, + }; + + controlPlane = new HostedAppControlPlane({ + registry: { + waitForLock: (runtimeId, args) => waitForRuntimeSessionLock(runtimeId, args), + renewLock: (runtimeId, token, ttlMs, args) => renewRuntimeSessionLock( + runtimeId, + token, + ttlMs, + args, + ), + releaseLock: releaseRuntimeSessionLock, + read: readRuntimeSessionRecord, + write: (record, token, args) => writeRuntimeSessionRecord( + record, + token, + undefined, + args, + ), + allocateGeneration: allocateRuntimeSessionGeneration, + }, + runtime, + checkpointStore: store, + checkpointConfig, + credentialKey: parseHostedAppCredentialKey(env.HOSTED_APP_CREDENTIAL_KEY), + lockWaitMs: env.RUNTIME_SESSION_LOCK_WAIT_MS, + lockTtlMs: RUNTIME_SESSION_LOCK_TTL_MS, + startHeartbeat: startRuntimeSessionLockHeartbeat, + captureCheckpoint: (runtimeSessionId, owner, signal) => ( + captureHostedAppSourceCheckpoint({ + runtimeSessionId, + owner, + signal, + lockWaitMs: env.RUNTIME_SESSION_LOCK_WAIT_MS, + deps: { + waitForLock: (sourceId, args) => waitForRuntimeSessionLock(sourceId, args), + releaseLock: releaseRuntimeSessionLock, + read: readRuntimeSessionRecord, + checkpoint: ({ runtimeSessionId: sourceId, lockToken, signal: sourceSignal }) => ( + checkpointSession({ + mintToken: microvmId => runtime.mintToken( + microvmId, + env.LAMBDA_MICROVM_PORT, + sourceSignal, + ), + store, + runtimeSessionId: sourceId, + config: { + ...checkpointConfig, + port: env.LAMBDA_MICROVM_PORT, + }, + normalizeEndpoint: normalizeHostedAppMicrovmEndpoint, + lockToken, + signal: sourceSignal, + }) + ), + }, + }) + ), + restoreCheckpoint: args => restoreSession({ + mintToken: microvmId => runtime.mintToken( + microvmId, + env.HOSTED_APP_CONTROL_PORT, + args.signal, + ), + store: args.store, + runtimeSessionId: args.runtimeSessionId, + microvmId: args.vm.microvmId, + endpointBase: normalizeHostedAppMicrovmEndpoint(args.vm.endpoint ?? ''), + config: args.config, + signal: args.signal, + checkpointKey: args.checkpointKey, + }), + }); + return controlPlane; +} + +export function resetHostedAppControlPlaneForTests(): void { + controlPlane = undefined; +} diff --git a/service/src/hosted-app/jobs.ts b/service/src/hosted-app/jobs.ts new file mode 100644 index 0000000..fb34ff7 --- /dev/null +++ b/service/src/hosted-app/jobs.ts @@ -0,0 +1,31 @@ +import type { Job } from 'bullmq'; +import type { HostedAppPublicStatus } from './record'; +import type { ResidentHostedAppSpec } from './spec'; + +export const HOSTED_APP_JOBS = [ + 'hosted-app:start', + 'hosted-app:stop', + 'hosted-app:refresh-preview', +] as const; +export type HostedAppJobName = (typeof HOSTED_APP_JOBS)[number]; + +interface HostedAppJobBase { + hostedAppRuntimeId: string; + tenantId: string; + canonicalUserId: string; + _otel?: Record; +} + +export interface HostedAppStartJobData extends HostedAppJobBase { + operation: 'start'; + sourceRuntimeSessionId: string; + spec: ResidentHostedAppSpec; +} + +export interface HostedAppStopJobData extends HostedAppJobBase { + operation: 'stop' | 'refresh-preview'; +} + +export type HostedAppJobData = HostedAppStartJobData | HostedAppStopJobData; +export type HostedAppJobResult = HostedAppPublicStatus; +export type HostedAppJob = Job; diff --git a/service/src/hosted-app/microvm-runtime.test.ts b/service/src/hosted-app/microvm-runtime.test.ts new file mode 100644 index 0000000..307b948 --- /dev/null +++ b/service/src/hosted-app/microvm-runtime.test.ts @@ -0,0 +1,209 @@ +import { describe, expect, test } from 'bun:test'; +import { FakeLambdaMicrovmClient } from '../runtime-session/lambda-client-fake'; +import { LambdaMicrovmApiError } from '../runtime-session/lambda-client'; +import { + hostedAppLaunchFingerprint, + hostedAppLaunchGenerationSeed, + hostedAppLaunchRequestFingerprint, + HostedAppMicrovmError, + HostedAppMicrovmRuntime, + type HostedAppMicrovmConfig, +} from './microvm-runtime'; +import type { ResidentHostedAppSpec } from './spec'; + +function config(): HostedAppMicrovmConfig { + return { + imageArn: 'arn:aws:lambda:us-east-2:1:microvm-image:app-host', + imageVersion: '7', + executionRoleArn: 'arn:aws:iam::1:role/app-host', + logGroup: '/aws/lambda-microvm/codeapi-app-host', + ingressConnectorArns: ['arn:ingress/private'], + controlPort: 8080, + previewPort: 3000, + maximumDurationSeconds: 28_800, + idleSeconds: 300, + suspendedSeconds: 900, + authTokenTtlSeconds: 3_600, + launchTimeoutMs: 5_000, + healthTimeoutMs: 500, + appStartTimeoutMs: 2_000, + launchTps: 4, + tokenTps: 8, + }; +} + +const spec: ResidentHostedAppSpec = { + adapter: 'resident', + app_id: 'demo', + revision: 'rev-1', + language: 'node', + version: '>=22', + entrypoint: 'server.js', + cwd: '.', + args: [], + env: {}, +}; + +function runtime( + fake: FakeLambdaMicrovmClient, + fetchImpl: (input: string | URL | Request, init?: RequestInit) => Promise + = async () => new Response('{}', { status: 200 }), +) { + const reservations: string[] = []; + return { + reservations, + runtime: new HostedAppMicrovmRuntime(fake, config(), { + reserveOp: async op => { reservations.push(op); }, + poisonOp: async () => {}, + fetch: fetchImpl, + sleep: async () => {}, + }), + }; +} + +describe('HostedAppMicrovmRuntime', () => { + test('seeds idempotency from exact wire inputs while keeping semantic matching order-independent', () => { + const first = { ...config(), ingressConnectorArns: ['arn:b', 'arn:a'] }; + const reordered = { ...config(), ingressConnectorArns: ['arn:a', 'arn:b'] }; + expect(hostedAppLaunchFingerprint(first)).toBe(hostedAppLaunchFingerprint(reordered)); + expect(hostedAppLaunchRequestFingerprint(first)).not.toBe( + hostedAppLaunchRequestFingerprint(reordered), + ); + expect(hostedAppLaunchGenerationSeed(first)).not.toBe(hostedAppLaunchGenerationSeed(reordered)); + }); + + test('launches the dedicated image with bounded idle policy and no egress connector', async () => { + const fake = new FakeLambdaMicrovmClient(); + const fixture = runtime(fake); + + const launched = await fixture.runtime.launch('sess-happ-1', new AbortController().signal); + + expect(launched.clientToken).toBe('sess-happ-1'); + expect(launched.vm.state).toBe('RUNNING'); + expect(fixture.reservations).toEqual(['run']); + const args = fake.callsFor('runMicrovm')[0].args as Record; + expect(args).toMatchObject({ + imageIdentifier: config().imageArn, + imageVersion: '7', + maximumDurationSeconds: 28_800, + idlePolicy: { + maxIdleSeconds: 300, + suspendedSeconds: 900, + autoResume: true, + }, + }); + expect(args.egressConnectorArns).toBeUndefined(); + }); + + test('retries a definite boot-time death once under a distinct token', async () => { + const fake = new FakeLambdaMicrovmClient(); + fake.terminateNextLaunch(); + const fixture = runtime(fake); + + const launched = await fixture.runtime.launch('sess-happ-2', new AbortController().signal); + + expect(launched.clientToken).toBe('sess-happ-2-r1'); + expect(fake.callsFor('runMicrovm').map(call => ( + call.args as { clientToken?: string } + ).clientToken)).toEqual(['sess-happ-2', 'sess-happ-2-r1']); + }); + + test('resumes a suspended same-token launch instead of provisioning a second VM', async () => { + const fake = new FakeLambdaMicrovmClient(); + const fixture = runtime(fake); + const first = await fixture.runtime.launch('sess-happ-recovered', new AbortController().signal); + await fake.suspendMicrovm(first.vm.microvmId); + + const recovered = await fixture.runtime.launch( + 'sess-happ-recovered', + new AbortController().signal, + ); + + expect(recovered.vm.microvmId).toBe(first.vm.microvmId); + expect(recovered.clientToken).toBe('sess-happ-recovered'); + expect(fake.vms.size).toBe(1); + expect(fake.callsFor('resumeMicrovm')).toHaveLength(1); + expect(fake.callsFor('runMicrovm').map(call => ( + call.args as { clientToken?: string } + ).clientToken)).toEqual(['sess-happ-recovered', 'sess-happ-recovered']); + }); + + test('does not rotate the idempotency token after an ambiguous provider failure', async () => { + const fake = new FakeLambdaMicrovmClient(); + fake.failNext('runMicrovm', new LambdaMicrovmApiError( + 'other', + 'RunMicrovm', + 'connection reset after request write', + )); + const fixture = runtime(fake); + + const error = await fixture.runtime.launch( + 'sess-happ-3', + new AbortController().signal, + ).catch(value => value); + + expect(error.code).toBe('hosted_app_launch_failed'); + expect(fake.callsFor('runMicrovm')).toHaveLength(1); + }); + + test('reuses one control credential while polling health', async () => { + const fake = new FakeLambdaMicrovmClient({ endpointProvider: () => 'http://app-host.test' }); + let probes = 0; + const fixture = runtime(fake, async () => { + probes += 1; + return new Response('{}', { status: probes < 3 ? 503 : 200 }); + }); + const { vm } = await fixture.runtime.launch('sess-happ-4', new AbortController().signal); + + await fixture.runtime.waitForControlReady(vm, new AbortController().signal); + + expect(probes).toBe(3); + expect(fake.callsFor('createMicrovmAuthToken')).toHaveLength(1); + }); + + test('starts the resident app through the control port with its session binding', async () => { + const fake = new FakeLambdaMicrovmClient({ endpointProvider: () => 'http://app-host.test' }); + let captured: { url: string; init?: RequestInit } | undefined; + const fixture = runtime(fake, async (input, init) => { + captured = { url: String(input), init }; + return new Response('{}', { status: 200 }); + }); + const { vm } = await fixture.runtime.launch('sess-happ-5', new AbortController().signal); + + await fixture.runtime.startResidentApp( + vm, + 'rt_source_session', + spec, + new AbortController().signal, + ); + + expect(captured?.url).toBe('http://app-host.test/api/v2/hosted-app/start'); + expect(captured?.init?.method).toBe('POST'); + expect(captured?.init?.headers).toMatchObject({ + 'X-aws-proxy-auth': expect.any(String), + 'X-Runtime-Session-Id': 'rt_source_session', + 'Content-Type': 'application/json', + }); + expect(JSON.parse(String(captured?.init?.body))).toEqual(spec); + }); + + test('preserves a runner validation status as a non-retryable typed failure', async () => { + const fake = new FakeLambdaMicrovmClient({ endpointProvider: () => 'http://app-host.test' }); + const fixture = runtime(fake, async () => new Response(JSON.stringify({ + error: 'hosted_app_runtime_not_found', + message: 'runtime node@99 is not installed', + }), { status: 400 })); + const { vm } = await fixture.runtime.launch('sess-happ-6', new AbortController().signal); + + const error = await fixture.runtime.startResidentApp( + vm, + 'rt_source_session', + spec, + new AbortController().signal, + ).catch(value => value); + + expect(error).toBeInstanceOf(HostedAppMicrovmError); + expect(error.httpStatus).toBe(400); + expect(error.transient).toBe(false); + }); +}); diff --git a/service/src/hosted-app/microvm-runtime.ts b/service/src/hosted-app/microvm-runtime.ts new file mode 100644 index 0000000..627b5a8 --- /dev/null +++ b/service/src/hosted-app/microvm-runtime.ts @@ -0,0 +1,444 @@ +import { + LambdaMicrovmApiError, + microvmPortHeaders, + type LambdaMicrovmClient, + type MicrovmAuthToken, + type MicrovmDescription, +} from '../runtime-session/lambda-client'; +import { + MicrovmOpThrottledError, + acquireOpBudget, + poisonOpBucket, + type ThrottledOp, +} from '../runtime-session/throttle'; +import type { ResidentHostedAppSpec } from './spec'; +import { createHash } from 'node:crypto'; + +export interface HostedAppMicrovmConfig { + imageArn: string; + imageVersion: string; + executionRoleArn?: string; + logGroup?: string; + ingressConnectorArns?: string[]; + controlPort: number; + previewPort: number; + maximumDurationSeconds: number; + idleSeconds: number; + suspendedSeconds: number; + authTokenTtlSeconds: number; + launchTimeoutMs: number; + healthTimeoutMs: number; + appStartTimeoutMs: number; + launchTps: number; + tokenTps: number; +} + +/** Order-independent identity for every immutable app-host launch input. */ +export function hostedAppLaunchFingerprint(config: HostedAppMicrovmConfig): string { + return JSON.stringify({ + imageArn: config.imageArn, + imageVersion: config.imageVersion, + executionRoleArn: config.executionRoleArn ?? '', + logGroup: config.logGroup ?? '', + ingressConnectorArns: [...(config.ingressConnectorArns ?? [])].sort(), + controlPort: config.controlPort, + previewPort: config.previewPort, + maximumDurationSeconds: config.maximumDurationSeconds, + idlePolicy: { + maxIdleSeconds: config.idleSeconds, + suspendedSeconds: config.suspendedSeconds, + autoResume: true, + }, + }); +} + +/** Exact RunMicrovm request identity. Preserve connector order because AWS + * idempotency compares the submitted request, not our semantic policy. */ +export function hostedAppLaunchRequestFingerprint(config: HostedAppMicrovmConfig): string { + return JSON.stringify({ + launchFingerprint: hostedAppLaunchFingerprint(config), + runMicrovm: { + imageIdentifier: config.imageArn, + imageVersion: config.imageVersion, + executionRoleArn: config.executionRoleArn, + logGroup: config.logGroup, + ingressConnectorArns: config.ingressConnectorArns, + maximumDurationSeconds: config.maximumDurationSeconds, + idlePolicy: { + maxIdleSeconds: config.idleSeconds, + suspendedSeconds: config.suspendedSeconds, + autoResume: true, + }, + }, + }); +} + +/** Keep reset Redis counters in an image/config-specific safe-integer range. */ +export function hostedAppLaunchGenerationSeed(config: HostedAppMicrovmConfig): number { + const offset = Number.parseInt( + createHash('sha256') + .update(hostedAppLaunchRequestFingerprint(config), 'utf8') + .digest('hex') + .slice(0, 13), + 16, + ); + return 1_000_000_000_000_000 + offset; +} + +export function hostedAppLaunchClientToken(runtimeId: string, generation: number): string { + if (!Number.isSafeInteger(generation) || generation < 1) { + throw new Error('Hosted app generation must be a positive safe integer'); + } + const token = `happ-${runtimeId}-${generation}`; + /* launch() reserves a single `-r1` suffix after a definite boot failure. */ + if (token.length > 125) { + throw new Error('Hosted app clientToken exceeds the AWS length limit'); + } + return token; +} + +export class HostedAppMicrovmError extends Error { + constructor( + readonly code: string, + message: string, + readonly transient: boolean, + readonly cause?: unknown, + readonly httpStatus = 503, + ) { + super(message); + this.name = 'HostedAppMicrovmError'; + } +} + +interface HostedAppMicrovmDeps { + reserveOp: ( + op: ThrottledOp, + args: { limitPerSecond: number; deadlineAtMs: number; signal: AbortSignal }, + ) => Promise; + poisonOp: (op: ThrottledOp, deadlineAtMs: number, signal: AbortSignal) => Promise; + fetch: (input: string | URL | Request, init?: RequestInit) => Promise; + sleep: (ms: number, signal: AbortSignal) => Promise; +} + +const abortableSleep = (ms: number, signal: AbortSignal): Promise => new Promise( + (resolve, reject) => { + if (signal.aborted) { + reject(signal.reason instanceof Error ? signal.reason : new Error('operation aborted')); + return; + } + const finish = (): void => { + signal.removeEventListener('abort', onAbort); + resolve(); + }; + const timer = setTimeout(finish, ms); + const onAbort = (): void => { + clearTimeout(timer); + reject(signal.reason instanceof Error ? signal.reason : new Error('operation aborted')); + }; + signal.addEventListener('abort', onAbort, { once: true }); + }, +); + +export function normalizeHostedAppMicrovmEndpoint(endpoint: string): string { + if (endpoint.startsWith('http://') || endpoint.startsWith('https://')) { + return endpoint.replace(/\/+$/, ''); + } + return `https://${endpoint.replace(/\/+$/, '')}`; +} + +function launchFailure(error: unknown): HostedAppMicrovmError { + if (error instanceof HostedAppMicrovmError) return error; + if (error instanceof LambdaMicrovmApiError) { + const transient = error.kind === 'throttled' || error.kind === 'other'; + return new HostedAppMicrovmError( + error.kind === 'throttled' ? 'hosted_app_launch_throttled' : 'hosted_app_launch_failed', + error.message, + transient, + error, + ); + } + return new HostedAppMicrovmError( + 'hosted_app_launch_failed', + error instanceof Error ? error.message : 'Hosted app MicroVM launch failed', + false, + error, + ); +} + +export class HostedAppMicrovmRuntime { + private readonly deps: HostedAppMicrovmDeps; + + constructor( + private readonly client: LambdaMicrovmClient, + readonly config: HostedAppMicrovmConfig, + deps: Partial = {}, + ) { + this.deps = { + reserveOp: (op, args) => acquireOpBudget(op, { + limitPerSecond: args.limitPerSecond, + budgetMs: Math.max(1, args.deadlineAtMs - Date.now()), + deadlineAtMs: args.deadlineAtMs, + signal: args.signal, + }), + poisonOp: (op, deadlineAtMs, signal) => poisonOpBucket(op, undefined, { + deadlineAtMs, + signal, + }), + fetch, + sleep: abortableSleep, + ...deps, + }; + } + + async launch(clientToken: string, callerSignal: AbortSignal): Promise<{ + vm: MicrovmDescription; + clientToken: string; + }> { + const deadlineAtMs = Date.now() + this.config.launchTimeoutMs; + const deadlineSignal = AbortSignal.timeout(this.config.launchTimeoutMs); + const signal = AbortSignal.any([callerSignal, deadlineSignal]); + try { + const first = await this.launchOnce(clientToken, deadlineAtMs, signal, deadlineSignal); + return { vm: first, clientToken }; + } catch (error) { + const failure = deadlineSignal.aborted && !callerSignal.aborted + ? new HostedAppMicrovmError( + 'hosted_app_launch_timeout', + `Hosted app MicroVM did not reach RUNNING within ${this.config.launchTimeoutMs}ms`, + true, + error, + ) + : launchFailure(error); + /* Only a definite boot-time death is safe to retry with a new token. + * Ambiguous RunMicrovm failures retain the original token so a successor + * can replay and recover the provider resource. */ + if (failure.code !== 'hosted_app_boot_failed' || callerSignal.aborted) throw failure; + const retryToken = `${clientToken}-r1`; + const vm = await this.launchOnce(retryToken, deadlineAtMs, signal, deadlineSignal) + .catch(second => { throw launchFailure(second); }); + return { vm, clientToken: retryToken }; + } + } + + private async launchOnce( + clientToken: string, + deadlineAtMs: number, + signal: AbortSignal, + reconcileSignal: AbortSignal, + ): Promise { + try { + await this.deps.reserveOp('run', { + limitPerSecond: this.config.launchTps, + deadlineAtMs, + signal, + }); + } catch (error) { + if (error instanceof MicrovmOpThrottledError) { + throw new HostedAppMicrovmError( + 'hosted_app_launch_throttled', + error.message, + true, + error, + ); + } + throw error; + } + + let vm: MicrovmDescription; + try { + vm = await this.client.runMicrovm({ + imageIdentifier: this.config.imageArn, + imageVersion: this.config.imageVersion, + executionRoleArn: this.config.executionRoleArn, + logGroup: this.config.logGroup, + ingressConnectorArns: this.config.ingressConnectorArns, + /* No egress connector is passed. The app-host runner additionally + * blocks all new OUTPUT traffic from the untrusted app UID. */ + maximumDurationSeconds: this.config.maximumDurationSeconds, + idlePolicy: { + maxIdleSeconds: this.config.idleSeconds, + suspendedSeconds: this.config.suspendedSeconds, + autoResume: true, + }, + clientToken, + }, signal, reconcileSignal); + } catch (error) { + if (error instanceof LambdaMicrovmApiError && error.kind === 'throttled') { + await this.deps.poisonOp('run', deadlineAtMs, signal).catch(() => {}); + } + throw error; + } + + let current = vm; + let resumeRequested = false; + for (;;) { + if (Date.now() >= deadlineAtMs) { + throw new HostedAppMicrovmError( + 'hosted_app_launch_timeout', + 'Hosted app MicroVM launch timed out', + true, + ); + } + if (current.state === 'RUNNING' && current.endpoint) return current; + if (current.state === 'TERMINATING' || current.state === 'TERMINATED') { + throw new HostedAppMicrovmError( + 'hosted_app_boot_failed', + `Hosted app MicroVM entered ${current.state} before becoming ready`, + true, + ); + } + /* Same-token recovery can find an older accepted launch after its idle + * policy suspended it. That is the resource we must recover, not evidence + * of a failed boot: rotating the token here would provision a second VM + * and abandon the first one until its hard deadline. Resume it once, then + * keep polling the same id while AWS completes the transition. */ + if (current.state === 'SUSPENDED' && !resumeRequested) { + resumeRequested = true; + try { + current = await this.client.resumeMicrovm(current.microvmId, signal); + continue; + } catch (error) { + if (error instanceof LambdaMicrovmApiError && error.kind === 'not_found') { + throw new HostedAppMicrovmError( + 'hosted_app_boot_failed', + 'Recovered hosted app MicroVM no longer exists', + true, + error, + ); + } + /* A conflicting resume commonly means auto-resume or another request + * won the state transition. Poll the known id instead of discarding + * it or rotating the launch token. */ + if (!(error instanceof LambdaMicrovmApiError) || error.kind !== 'conflict') { + throw error; + } + } + } + await this.deps.sleep(Math.min(250, Math.max(1, deadlineAtMs - Date.now())), signal); + current = await this.client.getMicrovm(current.microvmId, signal); + } + } + + async mintToken( + microvmId: string, + port: number, + callerSignal: AbortSignal, + ): Promise { + const deadlineAtMs = Date.now() + this.config.launchTimeoutMs; + const signal = AbortSignal.any([ + callerSignal, + AbortSignal.timeout(this.config.launchTimeoutMs), + ]); + try { + await this.deps.reserveOp('token', { + limitPerSecond: this.config.tokenTps, + deadlineAtMs, + signal, + }); + return await this.client.createMicrovmAuthToken({ + microvmId, + port, + ttlSeconds: this.config.authTokenTtlSeconds, + }, signal); + } catch (error) { + if (error instanceof LambdaMicrovmApiError && error.kind === 'throttled') { + await this.deps.poisonOp('token', deadlineAtMs, signal).catch(() => {}); + } + throw new HostedAppMicrovmError( + 'hosted_app_auth_failed', + error instanceof Error ? error.message : 'Could not authorize hosted app endpoint', + true, + error, + ); + } + } + + async waitForControlReady(vm: MicrovmDescription, callerSignal: AbortSignal): Promise { + const endpoint = normalizeHostedAppMicrovmEndpoint(vm.endpoint ?? ''); + const deadlineAtMs = Date.now() + this.config.launchTimeoutMs; + let lastError: unknown; + let token: MicrovmAuthToken | undefined; + while (Date.now() < deadlineAtMs) { + callerSignal.throwIfAborted(); + try { + if (token == null || token.expiresAtMs <= Date.now() + this.config.healthTimeoutMs) { + token = await this.mintToken(vm.microvmId, this.config.controlPort, callerSignal); + } + const response = await this.deps.fetch(`${endpoint}/api/v2/health`, { + headers: { + [token.headerName]: token.token, + ...microvmPortHeaders(this.config.controlPort), + }, + signal: AbortSignal.any([ + callerSignal, + AbortSignal.timeout(this.config.healthTimeoutMs), + ]), + }); + if (response.ok) return; + lastError = new Error(`health returned ${response.status}`); + } catch (error) { + lastError = error; + } + await this.deps.sleep(250, callerSignal); + } + throw new HostedAppMicrovmError( + 'hosted_app_unhealthy', + `Hosted app control listener did not become ready: ${lastError instanceof Error ? lastError.message : 'unknown error'}`, + true, + lastError, + ); + } + + async startResidentApp( + vm: MicrovmDescription, + runtimeSessionId: string, + spec: ResidentHostedAppSpec, + callerSignal: AbortSignal, + ): Promise { + const token = await this.mintToken(vm.microvmId, this.config.controlPort, callerSignal); + const response = await this.deps.fetch( + `${normalizeHostedAppMicrovmEndpoint(vm.endpoint ?? '')}/api/v2/hosted-app/start`, + { + method: 'POST', + headers: { + [token.headerName]: token.token, + ...microvmPortHeaders(this.config.controlPort), + 'X-Runtime-Session-Id': runtimeSessionId, + 'Content-Type': 'application/json', + }, + body: JSON.stringify(spec), + signal: AbortSignal.any([ + callerSignal, + AbortSignal.timeout(this.config.appStartTimeoutMs), + ]), + }, + ); + if (!response.ok) { + const body = await response.text().catch(() => ''); + throw new HostedAppMicrovmError( + 'hosted_app_start_failed', + `Hosted app runner rejected start (${response.status}): ${body.slice(0, 1_024)}`, + response.status >= 500, + undefined, + response.status, + ); + } + } + + previewToken(microvmId: string, signal: AbortSignal): Promise { + return this.mintToken(microvmId, this.config.previewPort, signal); + } + + async terminate(microvmId: string): Promise { + try { + await this.client.terminateMicrovm( + microvmId, + AbortSignal.timeout(this.config.launchTimeoutMs), + ); + return true; + } catch (error) { + if (error instanceof LambdaMicrovmApiError && error.kind === 'not_found') return true; + return false; + } + } +} diff --git a/service/src/hosted-app/preview-access.test.ts b/service/src/hosted-app/preview-access.test.ts new file mode 100644 index 0000000..1f383bf --- /dev/null +++ b/service/src/hosted-app/preview-access.test.ts @@ -0,0 +1,72 @@ +import { describe, expect, test } from 'bun:test'; +import { + hostedAppPreviewAuthorizeUrl, + hostedAppPreviewHostname, + hostedAppPreviewOwnerBinding, + hostedAppRuntimeIdFromHostname, + signHostedAppPreviewAccess, + verifyHostedAppPreviewAccess, +} from './preview-access'; + +const key = Buffer.alloc(32, 9); +const claims = { + hostedAppRuntimeId: `happ_${'a'.repeat(40)}`, + revision: 'rev-1', + ownerBinding: hostedAppPreviewOwnerBinding({ + tenantId: 'tenant-1', + canonicalUserId: 'user-1', + }, key), + expiresAt: 2_000_000, +}; + +describe('hosted app preview access', () => { + test('round-trips owner-bound claims and rejects tampering or expiry', () => { + const token = signHostedAppPreviewAccess(claims, key); + expect(verifyHostedAppPreviewAccess(token, key, 1_000_000)).toEqual(claims); + const parts = token.split('.'); + const signature = Buffer.from(parts[2] as string, 'base64url'); + signature[0] ^= 1; + parts[2] = signature.toString('base64url'); + expect(() => verifyHostedAppPreviewAccess(parts.join('.'), key, 1_000_000)).toThrow('invalid'); + expect(() => verifyHostedAppPreviewAccess(`${token}=`, key, 1_000_000)).toThrow('malformed'); + expect(() => verifyHostedAppPreviewAccess(token, key, claims.expiresAt)).toThrow('expired'); + }); + + test('binds the capability to one immutable app revision', () => { + const token = signHostedAppPreviewAccess(claims, key); + expect(verifyHostedAppPreviewAccess(token, key, 1_000_000).revision).toBe('rev-1'); + expect(() => signHostedAppPreviewAccess({ + ...claims, + revision: '../rev-2', + }, key)).toThrow('claims are invalid'); + }); + + test('blinds tenant and user identities into a stable keyed owner binding', () => { + const first = hostedAppPreviewOwnerBinding({ + tenantId: 'tenant-1', canonicalUserId: 'user-1', + }, key); + const second = hostedAppPreviewOwnerBinding({ + tenantId: 'tenant-1', canonicalUserId: 'user-2', + }, key); + expect(first).not.toContain('tenant-1'); + expect(first).not.toContain('user-1'); + expect(first).not.toBe(second); + }); + + test('maps the opaque runtime id to one wildcard host and clean exchange URL', () => { + const host = hostedAppPreviewHostname(claims.hostedAppRuntimeId, 'https://apps.example.test'); + expect(host).toBe(`happ-${'a'.repeat(40)}.apps.example.test`); + expect(hostedAppRuntimeIdFromHostname(host, 'https://apps.example.test')).toBe( + claims.hostedAppRuntimeId, + ); + expect(hostedAppRuntimeIdFromHostname('apps.example.test', 'https://apps.example.test')).toBeUndefined(); + const url = new URL(hostedAppPreviewAuthorizeUrl( + claims.hostedAppRuntimeId, + 'https://apps.example.test', + 'signed-token', + )); + expect(url.hostname).toBe(host); + expect(url.pathname).toBe('/__codeapi/authorize'); + expect(url.searchParams.get('token')).toBe('signed-token'); + }); +}); diff --git a/service/src/hosted-app/preview-access.ts b/service/src/hosted-app/preview-access.ts new file mode 100644 index 0000000..374b34e --- /dev/null +++ b/service/src/hosted-app/preview-access.ts @@ -0,0 +1,144 @@ +import { createHmac, timingSafeEqual } from 'node:crypto'; +import type { HostedAppOwner } from './control-plane'; +import { HOSTED_APP_REVISION_PATTERN } from './spec'; + +const PREVIEW_TOKEN_VERSION = 'v1'; +const RUNTIME_ID_PATTERN = /^happ_[0-9a-f]{40}$/; + +export interface HostedAppPreviewClaims { + hostedAppRuntimeId: string; + revision: string; + ownerBinding: string; + expiresAt: number; +} + +export class HostedAppPreviewAccessError extends Error {} + +export function hostedAppPreviewOwnerBinding(owner: HostedAppOwner, key: Buffer): string { + if (!owner.tenantId || !owner.canonicalUserId) { + throw new HostedAppPreviewAccessError('preview owner is invalid'); + } + return createHmac('sha256', key) + .update('hosted-app-preview-owner-v1\0') + .update(owner.tenantId, 'utf8') + .update('\0') + .update(owner.canonicalUserId, 'utf8') + .digest('base64url'); +} + +function signature(payload: string, key: Buffer): Buffer { + if (key.length !== 32) throw new HostedAppPreviewAccessError('preview signing key must be 32 bytes'); + return createHmac('sha256', key).update(PREVIEW_TOKEN_VERSION).update('.').update(payload).digest(); +} + +function decodeBase64url(raw: string, maxBytes: number): Buffer { + if (!/^[A-Za-z0-9_-]+$/.test(raw)) { + throw new HostedAppPreviewAccessError('preview access token is malformed'); + } + const decoded = Buffer.from(raw, 'base64url'); + if (decoded.length > maxBytes || decoded.toString('base64url') !== raw) { + throw new HostedAppPreviewAccessError('preview access token is malformed'); + } + return decoded; +} + +export function signHostedAppPreviewAccess( + claims: HostedAppPreviewClaims, + key: Buffer, +): string { + if (!RUNTIME_ID_PATTERN.test(claims.hostedAppRuntimeId)) { + throw new HostedAppPreviewAccessError('hosted app runtime id is malformed'); + } + if ( + !Number.isSafeInteger(claims.expiresAt) + || claims.expiresAt <= 0 + || !HOSTED_APP_REVISION_PATTERN.test(claims.revision) + || !/^[A-Za-z0-9_-]{43}$/.test(claims.ownerBinding) + ) { + throw new HostedAppPreviewAccessError('preview claims are invalid'); + } + const payload = Buffer.from(JSON.stringify({ + r: claims.hostedAppRuntimeId, + v: claims.revision, + o: claims.ownerBinding, + e: claims.expiresAt, + }), 'utf8').toString('base64url'); + return `${PREVIEW_TOKEN_VERSION}.${payload}.${signature(payload, key).toString('base64url')}`; +} + +export function verifyHostedAppPreviewAccess( + token: string, + key: Buffer, + now = Date.now(), +): HostedAppPreviewClaims { + if (token.length > 1_024) { + throw new HostedAppPreviewAccessError('preview access token is malformed'); + } + const [version, payload, signatureRaw, extra] = token.split('.'); + if (version !== PREVIEW_TOKEN_VERSION || !payload || !signatureRaw || extra !== undefined) { + throw new HostedAppPreviewAccessError('preview access token is malformed'); + } + const received = decodeBase64url(signatureRaw, 32); + const expected = signature(payload, key); + if (received.length !== expected.length || !timingSafeEqual(received, expected)) { + throw new HostedAppPreviewAccessError('preview access token is invalid'); + } + try { + const parsed = JSON.parse(decodeBase64url(payload, 512).toString('utf8')) as { + r?: unknown; v?: unknown; o?: unknown; e?: unknown; + }; + if ( + typeof parsed.r !== 'string' + || !RUNTIME_ID_PATTERN.test(parsed.r) + || typeof parsed.v !== 'string' + || !HOSTED_APP_REVISION_PATTERN.test(parsed.v) + || typeof parsed.o !== 'string' + || !/^[A-Za-z0-9_-]{43}$/.test(parsed.o) + || !Number.isSafeInteger(parsed.e) + || (parsed.e as number) <= now + ) { + throw new Error('claims invalid or expired'); + } + return { + hostedAppRuntimeId: parsed.r, + revision: parsed.v, + ownerBinding: parsed.o, + expiresAt: parsed.e as number, + }; + } catch (error) { + throw new HostedAppPreviewAccessError( + `preview access token claims are invalid: ${(error as Error).message}`, + ); + } +} + +export function hostedAppPreviewHostname(runtimeId: string, previewOrigin: string): string { + if (!RUNTIME_ID_PATTERN.test(runtimeId)) { + throw new HostedAppPreviewAccessError('hosted app runtime id is malformed'); + } + return `${runtimeId.replace('_', '-')}.${new URL(previewOrigin).hostname}`; +} + +export function hostedAppRuntimeIdFromHostname( + hostname: string, + previewOrigin: string, +): string | undefined { + const suffix = new URL(previewOrigin).hostname.toLowerCase(); + const lower = hostname.toLowerCase().replace(/\.$/, ''); + if (!lower.endsWith(`.${suffix}`)) return undefined; + const label = lower.slice(0, -(suffix.length + 1)); + if (!/^happ-[0-9a-f]{40}$/.test(label)) return undefined; + return label.replace('-', '_'); +} + +export function hostedAppPreviewAuthorizeUrl( + runtimeId: string, + previewOrigin: string, + token: string, +): string { + const origin = new URL(previewOrigin); + origin.hostname = hostedAppPreviewHostname(runtimeId, previewOrigin); + origin.pathname = '/__codeapi/authorize'; + origin.searchParams.set('token', token); + return origin.toString(); +} diff --git a/service/src/hosted-app/preview-gateway.ts b/service/src/hosted-app/preview-gateway.ts new file mode 100644 index 0000000..4ce6ed5 --- /dev/null +++ b/service/src/hosted-app/preview-gateway.ts @@ -0,0 +1,165 @@ +import type { NextFunction, Request, Response } from 'express'; +import { env } from '../config'; +import type { AuthenticatedRequest } from '../types'; +import { readRuntimeSessionRecord } from '../runtime-session/registry'; +import { HostedAppControlPlaneError } from './control-plane'; +import { + hostedAppRuntimeIdFromHostname, + HostedAppPreviewAccessError, + hostedAppPreviewOwnerBinding, + signHostedAppPreviewAccess, + verifyHostedAppPreviewAccess, +} from './preview-access'; +import { proxyHostedAppPreview } from './preview-proxy'; +import { applyHostedAppPreviewSecurityHeaders } from './proxy-policy'; + +const COOKIE_NAME = '__Host-codeapi-app'; +const PREVIEW_COOKIE_TTL_MS = 60 * 60_000; + +function rawHostname(req: Request): string | undefined { + const host = req.headers.host; + if (!host || /[\s/@\\]/.test(host)) return undefined; + try { + return new URL(`http://${host}`).hostname; + } catch { + return undefined; + } +} + +function cookie(req: Request, name: string): string | undefined { + for (const item of (req.headers.cookie ?? '').split(';')) { + const separator = item.indexOf('='); + if (separator < 0 || item.slice(0, separator).trim() !== name) continue; + try { + return decodeURIComponent(item.slice(separator + 1).trim()); + } catch { + return undefined; + } + } + return undefined; +} + +function previewKey(): Buffer { + return Buffer.from(env.HOSTED_APP_PREVIEW_SIGNING_KEY, 'base64'); +} + +function reject(res: Response, status: number, message: string): Response { + res.setHeader('Cache-Control', 'no-store'); + res.setHeader('Referrer-Policy', 'no-referrer'); + return res.status(status).type('text/plain').send(message); +} + +export async function hostedAppPreviewGateway( + req: Request, + res: Response, + next: NextFunction, +): Promise { + if (!env.HOSTED_APPS_ENABLED || !env.HOSTED_APP_PREVIEW_ORIGIN) return next(); + const hostname = rawHostname(req); + const runtimeId = hostname + ? hostedAppRuntimeIdFromHostname(hostname, env.HOSTED_APP_PREVIEW_ORIGIN) + : undefined; + if (!runtimeId) return next(); + + /* A wildcard app host is a separate, unprivileged origin. Never fall through + * from it into CodeAPI routes, even when authentication fails. */ + /* App routes are arbitrary user data. Collapse them before the outer metrics + * middleware records its completion event so Prometheus labels stay bounded. */ + res.locals.codeapiMetricPath = '/hosted-app-preview/*'; + applyHostedAppPreviewSecurityHeaders(res); + try { + if (req.path === '/__codeapi/authorize') { + if (req.method !== 'GET' || typeof req.query.token !== 'string') { + reject(res, 400, 'Invalid preview authorization request'); + return; + } + const linkClaims = verifyHostedAppPreviewAccess(req.query.token, previewKey()); + if (linkClaims.hostedAppRuntimeId !== runtimeId) { + reject(res, 403, 'Preview authorization does not match this app'); + return; + } + const record = await readRuntimeSessionRecord(runtimeId); + if ( + !record?.hosted_app + || record.state !== 'RUNNING' + || record.hosted_app.revision !== linkClaims.revision + || !record.microvm_id + || !record.endpoint + || record.hard_deadline_at == null + || record.hard_deadline_at <= Date.now() + ) { + reject(res, 409, 'Hosted app is not running'); + return; + } + if (hostedAppPreviewOwnerBinding({ + tenantId: record.tenant_id, + canonicalUserId: record.canonical_user_id, + }, previewKey()) !== linkClaims.ownerBinding) { + reject(res, 403, 'Preview authorization does not match this owner'); + return; + } + const expiresAt = Math.min( + record.hard_deadline_at ?? Date.now() + PREVIEW_COOKIE_TTL_MS, + Date.now() + PREVIEW_COOKIE_TTL_MS, + ); + if (expiresAt <= Date.now()) { + reject(res, 409, 'Hosted app lease has expired'); + return; + } + const sessionToken = signHostedAppPreviewAccess({ + ...linkClaims, + expiresAt, + }, previewKey()); + const maxAge = Math.max(1, Math.floor((expiresAt - Date.now()) / 1_000)); + res.setHeader('Set-Cookie', [ + `${COOKIE_NAME}=${encodeURIComponent(sessionToken)}`, + 'Path=/', + 'HttpOnly', + 'Secure', + 'SameSite=Strict', + `Max-Age=${maxAge}`, + ].join('; ')); + res.setHeader('Cache-Control', 'no-store'); + res.redirect(303, '/'); + return; + } + + const sessionToken = cookie(req, COOKIE_NAME); + if (!sessionToken) { + reject(res, 401, 'Preview authorization required'); + return; + } + const claims = verifyHostedAppPreviewAccess(sessionToken, previewKey()); + if (claims.hostedAppRuntimeId !== runtimeId) { + reject(res, 403, 'Preview authorization does not match this app'); + return; + } + const publicOrigin = new URL(env.HOSTED_APP_PREVIEW_ORIGIN); + publicOrigin.hostname = hostname as string; + await proxyHostedAppPreview( + req as AuthenticatedRequest, + res, + { + hostedAppRuntimeId: runtimeId, + revision: claims.revision, + ownerBinding: claims.ownerBinding, + publicHost: publicOrigin.host, + }, + req.path, + ); + } catch (error) { + if (res.headersSent) { + res.destroy(error instanceof Error ? error : undefined); + return; + } + if (error instanceof HostedAppPreviewAccessError) { + reject(res, 401, 'Preview authorization failed'); + return; + } + if (error instanceof HostedAppControlPlaneError) { + reject(res, error.status, error.message); + return; + } + reject(res, 502, 'Hosted app preview is unavailable'); + } +} diff --git a/service/src/hosted-app/preview-proxy.test.ts b/service/src/hosted-app/preview-proxy.test.ts new file mode 100644 index 0000000..1fedb49 --- /dev/null +++ b/service/src/hosted-app/preview-proxy.test.ts @@ -0,0 +1,102 @@ +import { describe, expect, test } from 'bun:test'; +import type { RuntimeSessionRecord } from '../runtime-session/registry'; +import { + hostedAppPreviewCredentialUsable, + hostedAppForwardedQuery, + hostedAppUpstreamUrl, + rewriteHostedAppLocation, +} from './preview-proxy'; + +function previewRecord(): RuntimeSessionRecord { + return { + runtime_session_id: `happ_${'a'.repeat(40)}`, + tenant_id: 'tenant-1', + canonical_user_id: 'user-1', + state: 'RUNNING', + generation: 1, + launched_at: 1_000, + last_seen_at: 1_000, + hard_deadline_at: 20_000, + microvm_id: 'vm-1', + endpoint: 'https://vm.aws.example', + hosted_app: { + source_runtime_session_id: 'rt-source', + app_id: 'demo', + revision: 'rev-2', + spec_fingerprint: 'fingerprint', + spec: { + adapter: 'resident', + app_id: 'demo', + revision: 'rev-2', + language: 'node', + version: '22', + entrypoint: 'server.js', + cwd: '.', + args: [], + env: {}, + }, + checkpoint_key: 'checkpoint', + preview_credential: 'sealed', + preview_credential_expires_at: 15_000, + }, + }; +} + +describe('hosted app preview upstream URL', () => { + test('keeps protocol-relative and ordinary request paths on the AWS endpoint origin', () => { + const endpoint = 'https://vm.aws.example/'; + expect(hostedAppUpstreamUrl(endpoint, '/assets/app.js').toString()) + .toBe('https://vm.aws.example/assets/app.js'); + expect(hostedAppUpstreamUrl(endpoint, '//attacker.example/steal').origin) + .toBe('https://vm.aws.example'); + expect(hostedAppUpstreamUrl(endpoint, '//attacker.example/steal').pathname) + .toBe('//attacker.example/steal'); + }); + + test('rejects non-HTTPS and credential-bearing endpoints', () => { + expect(() => hostedAppUpstreamUrl('http://vm.aws.example', '/')).toThrow( + 'Hosted app endpoint is invalid', + ); + expect(() => hostedAppUpstreamUrl('https://user@vm.aws.example', '/')).toThrow( + 'Hosted app endpoint is invalid', + ); + }); + + test('resolves relative redirects against the current app route', () => { + const current = 'https://vm.aws.example/projects/demo/start?from=preview'; + expect(rewriteHostedAppLocation('../login?next=demo', current)) + .toBe('/projects/login?next=demo'); + expect(rewriteHostedAppLocation('?ready=true', current)) + .toBe('/projects/demo/start?ready=true'); + expect(rewriteHostedAppLocation('https://attacker.example/steal', current)).toBeUndefined(); + }); + + test('preserves signed and repeated query bytes without Express re-encoding', () => { + expect(hostedAppForwardedQuery('/download?sig=a%2Fb+c&tag=one&tag=two')) + .toBe('?sig=a%2Fb+c&tag=one&tag=two'); + expect(hostedAppForwardedQuery('/download')).toBe(''); + }); + + test('rejects stale revisions, expired leases, and expired preview credentials', () => { + const record = previewRecord(); + const target = { + hostedAppRuntimeId: record.runtime_session_id, + revision: 'rev-2', + ownerBinding: 'owner', + }; + expect(hostedAppPreviewCredentialUsable(record, target, 10_000)).toBe(true); + expect(hostedAppPreviewCredentialUsable(record, { ...target, revision: 'rev-1' }, 10_000)) + .toBe(false); + expect(hostedAppPreviewCredentialUsable({ + ...record, + hard_deadline_at: 10_000, + }, target, 10_000)).toBe(false); + expect(hostedAppPreviewCredentialUsable({ + ...record, + hosted_app: { + ...record.hosted_app!, + preview_credential_expires_at: 10_000, + }, + }, target, 10_000)).toBe(false); + }); +}); diff --git a/service/src/hosted-app/preview-proxy.ts b/service/src/hosted-app/preview-proxy.ts new file mode 100644 index 0000000..a4b657d --- /dev/null +++ b/service/src/hosted-app/preview-proxy.ts @@ -0,0 +1,296 @@ +import type { Response } from 'express'; +import { Readable, Transform } from 'node:stream'; +import { env } from '../config'; +import { readRuntimeSessionRecord } from '../runtime-session/registry'; +import type { RuntimeSessionRecord } from '../runtime-session/registry'; +import { captureTraceCarrier } from '../telemetry'; +import type { AuthenticatedRequest } from '../types'; +import { assertHostedAppOwned, HostedAppControlPlaneError } from './control-plane'; +import { openHostedAppCredential, parseHostedAppCredentialKey } from './credential'; +import { + hostedAppProxyRequestHeaders, + hostedAppProxyResponseHeaders, +} from './proxy-policy'; +import { submitHostedAppJob } from './queue'; +import { hostedAppPreviewOwnerBinding } from './preview-access'; + +const PREVIEW_REFRESH_SKEW_MS = 60_000; + +export interface HostedAppPreviewTarget { + hostedAppRuntimeId: string; + revision?: string; + sourceRuntimeSessionId?: string; + identity?: { tenantId: string; canonicalUserId: string }; + ownerBinding?: string; + publicHost?: string; +} + +export function hostedAppPreviewRecordUsable( + record: RuntimeSessionRecord | null | undefined, + resolved: HostedAppPreviewTarget, + now = Date.now(), +): record is RuntimeSessionRecord & { + hosted_app: NonNullable; + microvm_id: string; + endpoint: string; +} { + return Boolean( + record?.hosted_app + && record.state === 'RUNNING' + && record.microvm_id + && record.endpoint + && record.hard_deadline_at != null + && record.hard_deadline_at > now + && (resolved.revision == null || record.hosted_app.revision === resolved.revision) + ); +} + +export function hostedAppPreviewCredentialUsable( + record: RuntimeSessionRecord | null | undefined, + resolved: HostedAppPreviewTarget, + now = Date.now(), + minimumTtlMs = 0, +): record is RuntimeSessionRecord & { + hosted_app: NonNullable & { + preview_credential: string; + preview_credential_expires_at: number; + }; + microvm_id: string; + endpoint: string; +} { + return hostedAppPreviewRecordUsable(record, resolved, now) + && Boolean( + record.hosted_app.preview_credential + && record.hosted_app.preview_credential_expires_at != null + && record.hosted_app.preview_credential_expires_at > now + minimumTtlMs + ); +} + +async function previewRecord( + resolved: HostedAppPreviewTarget, + signal: AbortSignal, +) { + let record = await readRuntimeSessionRecord(resolved.hostedAppRuntimeId, { signal }); + if (!hostedAppPreviewRecordUsable(record, resolved)) { + throw new HostedAppControlPlaneError( + 'hosted_app_not_running', + 'Hosted app is not running', + 409, + true, + ); + } + assertPreviewRecordAuthorized(record, resolved); + if (!hostedAppPreviewCredentialUsable(record, resolved, Date.now(), PREVIEW_REFRESH_SKEW_MS)) { + await submitHostedAppJob('hosted-app:refresh-preview', { + operation: 'refresh-preview', + hostedAppRuntimeId: resolved.hostedAppRuntimeId, + tenantId: record.tenant_id, + canonicalUserId: record.canonical_user_id, + _otel: captureTraceCarrier(), + }, `happ-refresh-${resolved.hostedAppRuntimeId}-${Math.floor(Date.now() / 30_000)}`); + record = await readRuntimeSessionRecord(resolved.hostedAppRuntimeId, { signal }); + } + if (!hostedAppPreviewCredentialUsable(record, resolved)) { + throw new HostedAppControlPlaneError( + 'hosted_app_preview_unavailable', + 'Hosted app preview credential is unavailable', + 503, + true, + ); + } + /* Refresh waits on another worker and then rereads durable state. Recheck + * ownership on that second snapshot instead of relying on the pre-await + * authorization decision. */ + assertPreviewRecordAuthorized(record, resolved); + return record; +} + +function assertPreviewRecordAuthorized( + record: RuntimeSessionRecord, + resolved: HostedAppPreviewTarget, +): void { + if (resolved.identity) { + assertHostedAppOwned(record, resolved.identity, resolved.sourceRuntimeSessionId); + } else { + const key = Buffer.from(env.HOSTED_APP_PREVIEW_SIGNING_KEY, 'base64'); + const expected = hostedAppPreviewOwnerBinding({ + tenantId: record.tenant_id, + canonicalUserId: record.canonical_user_id, + }, key); + if (!resolved.ownerBinding || resolved.ownerBinding !== expected) { + throw new HostedAppControlPlaneError('hosted_app_not_found', 'Hosted app not found', 404); + } + } +} + +function requestBody(req: AuthenticatedRequest): BodyInit | undefined { + if (req.method === 'GET' || req.method === 'HEAD') return undefined; + const declaredLength = Number(req.headers['content-length']); + if (Number.isFinite(declaredLength) && declaredLength > env.MAX_FILE_SIZE) { + throw new HostedAppControlPlaneError( + 'hosted_app_request_too_large', + `Hosted app request exceeds ${env.MAX_FILE_SIZE} bytes`, + 413, + ); + } + if (req.body != null) { + const body = Buffer.isBuffer(req.body) || typeof req.body === 'string' + ? req.body + : JSON.stringify(req.body); + if (Buffer.byteLength(body) > env.MAX_FILE_SIZE) { + throw new HostedAppControlPlaneError( + 'hosted_app_request_too_large', + `Hosted app request exceeds ${env.MAX_FILE_SIZE} bytes`, + 413, + ); + } + return body as unknown as BodyInit; + } + let bytes = 0; + const limiter = new Transform({ + transform(chunk: Buffer, _encoding, callback) { + bytes += chunk.length; + callback( + bytes > env.MAX_FILE_SIZE + ? new HostedAppControlPlaneError( + 'hosted_app_request_too_large', + `Hosted app request exceeds ${env.MAX_FILE_SIZE} bytes`, + 413, + ) + : null, + chunk, + ); + }, + }); + return req.pipe(limiter) as unknown as BodyInit; +} + +export function rewriteHostedAppLocation( + location: string, + currentUpstreamUrl: string, +): string | undefined { + try { + const upstreamOrigin = new URL(currentUpstreamUrl); + const destination = new URL(location, upstreamOrigin); + if (destination.origin !== upstreamOrigin.origin) return undefined; + return `${destination.pathname}${destination.search}${destination.hash}`; + } catch { + return undefined; + } +} + +/** Preserve the AWS endpoint origin even when a hostile request path begins + * with `//` (which URL resolution would otherwise treat as a new host). */ +export function hostedAppUpstreamUrl(endpoint: string, upstreamPath: string): URL { + let upstream: URL; + try { + upstream = new URL(endpoint); + } catch { + throw new HostedAppControlPlaneError( + 'hosted_app_endpoint_invalid', + 'Hosted app endpoint is invalid', + 502, + true, + ); + } + if ( + upstream.protocol !== 'https:' + || upstream.username + || upstream.password + || !upstream.hostname + ) { + throw new HostedAppControlPlaneError( + 'hosted_app_endpoint_invalid', + 'Hosted app endpoint is invalid', + 502, + true, + ); + } + upstream.pathname = upstreamPath.startsWith('/') ? upstreamPath : `/${upstreamPath}`; + upstream.search = ''; + upstream.hash = ''; + return upstream; +} + +export function hostedAppForwardedQuery(originalUrl: string): string { + const queryStart = originalUrl.indexOf('?'); + return queryStart < 0 ? '' : originalUrl.slice(queryStart); +} + +export async function proxyHostedAppPreview( + req: AuthenticatedRequest, + res: Response, + resolved: HostedAppPreviewTarget, + upstreamPath: string, +): Promise { + const controller = new AbortController(); + const abort = (): void => controller.abort(new Error('Preview client disconnected')); + req.once('aborted', abort); + res.once('close', abort); + try { + const record = await previewRecord(resolved, controller.signal); + const key = parseHostedAppCredentialKey(env.HOSTED_APP_CREDENTIAL_KEY); + const token = openHostedAppCredential( + resolved.hostedAppRuntimeId, + record.hosted_app?.preview_credential as string, + key, + ); + if (token.expiresAtMs <= Date.now()) { + throw new HostedAppControlPlaneError( + 'hosted_app_preview_unavailable', + 'Hosted app preview credential is unavailable', + 503, + true, + ); + } + const endpoint = `${record.endpoint?.replace(/\/+$/, '')}/`; + const upstream = hostedAppUpstreamUrl(endpoint, upstreamPath); + /* A reverse proxy must not parse and rebuild signed/repeated query strings: + * decoding and re-encoding changes their byte representation, while nested + * values can disappear entirely through Express's query parser. Preserve + * the original query bytes and constrain only the upstream origin/path. */ + upstream.search = hostedAppForwardedQuery(req.originalUrl); + const init: RequestInit & { duplex?: 'half' } = { + method: req.method, + headers: hostedAppProxyRequestHeaders( + req.headers, + token, + env.HOSTED_APP_PREVIEW_PORT, + resolved.publicHost ? { + host: resolved.publicHost, + protocol: new URL(env.HOSTED_APP_PREVIEW_ORIGIN).protocol === 'http:' ? 'http' : 'https', + } : undefined, + ), + body: requestBody(req), + redirect: 'manual', + signal: controller.signal, + }; + if (init.body != null && !Buffer.isBuffer(init.body) && typeof init.body !== 'string') { + init.duplex = 'half'; + } + const response = await fetch(upstream, init); + res.status(response.status); + hostedAppProxyResponseHeaders(response.headers).forEach((value, name) => { + res.setHeader(name, value); + }); + const location = response.headers.get('location'); + /* Resolve relative redirects against the request URL, not the AWS endpoint + * root. Framework redirects such as `Location: ../login` depend on the + * current route while the same-origin check still strips the AWS origin. */ + const safeLocation = location + ? rewriteHostedAppLocation(location, upstream.toString()) + : undefined; + if (location && !safeLocation) { + await response.body?.cancel().catch(() => {}); + return res.status(502).type('text/plain').send('Hosted app returned an unsafe redirect'); + } + if (safeLocation) res.setHeader('Location', safeLocation); + if (req.method === 'HEAD' || response.body == null) return res.end(); + const body = Readable.fromWeb(response.body as never); + body.once('error', error => res.destroy(error)); + body.pipe(res); + } finally { + req.removeListener('aborted', abort); + if (res.writableEnded) res.removeListener('close', abort); + } +} diff --git a/service/src/hosted-app/proxy-policy.test.ts b/service/src/hosted-app/proxy-policy.test.ts new file mode 100644 index 0000000..0553362 --- /dev/null +++ b/service/src/hosted-app/proxy-policy.test.ts @@ -0,0 +1,67 @@ +import { describe, expect, test } from 'bun:test'; +import { + applyHostedAppPreviewSecurityHeaders, + hostedAppProxyRequestHeaders, + hostedAppProxyResponseHeaders, +} from './proxy-policy'; + +describe('hosted app preview proxy policy', () => { + test('constrains browser fetches and disables persistent workers', () => { + const headers = new Map(); + applyHostedAppPreviewSecurityHeaders({ + setHeader(name, value) { + headers.set(name.toLowerCase(), value); + }, + }); + + expect(headers.get('content-security-policy')).toContain("connect-src 'self'"); + expect(headers.get('content-security-policy')).toContain("worker-src 'none'"); + expect(headers.get('content-security-policy')).toContain("form-action 'self'"); + expect(headers.get('permissions-policy')).toContain('camera=()'); + expect(headers.get('x-dns-prefetch-control')).toBe('off'); + expect(headers.get('cross-origin-opener-policy')).toBe('same-origin'); + expect(headers.get('cache-control')).toBe('private, no-store'); + }); + + test('keeps CodeAPI identity and caller-supplied AWS headers out of the app', () => { + const headers = hostedAppProxyRequestHeaders({ + authorization: 'Bearer codeapi-secret', + cookie: 'librechat=session-secret', + 'x-api-key': 'api-secret', + 'x-forwarded-user': 'user-1', + 'x-aws-proxy-auth': 'attacker-token', + accept: 'text/event-stream', + 'x-app-action': 'move', + }, { + headerName: 'X-aws-proxy-auth', + token: 'worker-minted-token', + expiresAtMs: Date.now() + 60_000, + }, 3000, { + host: 'happ-safe.apps.example.test', + protocol: 'https', + }); + + expect(Object.fromEntries(headers)).toEqual({ + accept: 'text/event-stream', + 'x-app-action': 'move', + 'x-aws-proxy-auth': 'worker-minted-token', + 'x-aws-proxy-port': '3000', + 'x-forwarded-host': 'happ-safe.apps.example.test', + 'x-forwarded-proto': 'https', + }); + }); + + test('does not let a hosted app set cookies, caching, redirects, or security policy', () => { + const headers = hostedAppProxyResponseHeaders(new Headers({ + 'content-type': 'text/html', + 'cache-control': 'public, max-age=31536000', + expires: 'Wed, 21 Oct 2037 07:28:00 GMT', + 'set-cookie': 'session=owned', + location: 'https://internal-microvm.example/secret', + 'content-security-policy': "default-src *", + 'access-control-allow-origin': '*', + })); + + expect(Object.fromEntries(headers)).toEqual({ 'content-type': 'text/html' }); + }); +}); diff --git a/service/src/hosted-app/proxy-policy.ts b/service/src/hosted-app/proxy-policy.ts new file mode 100644 index 0000000..8f9bbe9 --- /dev/null +++ b/service/src/hosted-app/proxy-policy.ts @@ -0,0 +1,105 @@ +import type { IncomingHttpHeaders } from 'node:http'; +import { microvmPortHeaders, type MicrovmAuthToken } from '../runtime-session/lambda-client'; + +export interface HostedAppPreviewHeaderWriter { + setHeader(name: string, value: string): unknown; +} + +/** Browser-enforced guardrails owned by the trusted gateway, not user code. */ +export function applyHostedAppPreviewSecurityHeaders( + response: HostedAppPreviewHeaderWriter, +): void { + response.setHeader('Referrer-Policy', 'no-referrer'); + response.setHeader('X-Content-Type-Options', 'nosniff'); + response.setHeader('X-DNS-Prefetch-Control', 'off'); + response.setHeader('Cross-Origin-Opener-Policy', 'same-origin'); + /* The app origin and browser cache outlive an individual capability and app + * revision. User-controlled caching could otherwise replay old HTML/JS after + * the revision-bound cookie has expired or a replacement has landed. */ + response.setHeader('Cache-Control', 'private, no-store'); + /* User code has no server-side egress and should not regain it through the + * owner's browser. Disabling workers also prevents a service worker from one + * revision persisting on this stable app origin into a later revision. */ + response.setHeader('Content-Security-Policy', [ + "default-src 'self' data: blob:", + "connect-src 'self'", + "script-src 'self' 'unsafe-inline' 'unsafe-eval'", + "style-src 'self' 'unsafe-inline'", + "img-src 'self' data: blob:", + "media-src 'self' data: blob:", + "font-src 'self' data:", + "worker-src 'none'", + "child-src 'none'", + "frame-src 'none'", + "object-src 'none'", + "base-uri 'none'", + "form-action 'self'", + "frame-ancestors 'self'", + ].join('; ')); + response.setHeader( + 'Permissions-Policy', + 'camera=(), microphone=(), geolocation=(), payment=(), usb=()', + ); +} + +const SAFE_REQUEST_HEADERS = new Set([ + 'accept', + 'accept-language', + 'cache-control', + 'content-type', + 'if-match', + 'if-modified-since', + 'if-none-match', + 'if-range', + 'if-unmodified-since', + 'range', + 'user-agent', +]); + +const SAFE_RESPONSE_HEADERS = new Set([ + 'accept-ranges', + 'content-disposition', + 'content-language', + 'content-range', + 'content-type', + 'etag', + 'last-modified', + 'vary', +]); + +/** Build a capability-minimal upstream request. In particular, never expose + * LibreChat/CodeAPI auth cookies, API keys, forwarded identity, or an + * attacker-supplied AWS proxy credential to the untrusted hosted app. */ +export function hostedAppProxyRequestHeaders( + source: IncomingHttpHeaders, + token: MicrovmAuthToken, + previewPort: number, + publicOrigin?: { host: string; protocol: 'https' | 'http' }, +): Headers { + const headers = new Headers({ + [token.headerName]: token.token, + ...microvmPortHeaders(previewPort), + }); + for (const [name, value] of Object.entries(source)) { + const lower = name.toLowerCase(); + if (!SAFE_REQUEST_HEADERS.has(lower) && !lower.startsWith('x-app-')) continue; + if (value == null) continue; + headers.set(name, Array.isArray(value) ? value.join(', ') : value); + } + headers.delete('content-length'); + if (publicOrigin) { + headers.set('X-Forwarded-Host', publicOrigin.host); + headers.set('X-Forwarded-Proto', publicOrigin.protocol); + } + return headers; +} +/** Cookies, redirects, CORS, and security-policy headers from user code must + * not mutate the CodeAPI/LibreChat origin. The narrow representation headers + * below are sufficient for HTML, assets, ranges, and SSE. */ +export function hostedAppProxyResponseHeaders(source: Headers): Headers { + const headers = new Headers(); + source.forEach((value, name) => { + if (SAFE_RESPONSE_HEADERS.has(name.toLowerCase())) headers.set(name, value); + }); + return headers; +} diff --git a/service/src/hosted-app/queue.ts b/service/src/hosted-app/queue.ts new file mode 100644 index 0000000..1855593 --- /dev/null +++ b/service/src/hosted-app/queue.ts @@ -0,0 +1,77 @@ +import { Queue, QueueEvents } from 'bullmq'; +import { setMaxListeners } from 'node:events'; +import { nanoid } from 'nanoid'; +import { connection } from '../queue'; +import { env } from '../config'; +import logger from '../logger'; +import type { HostedAppJobData, HostedAppJobName, HostedAppJobResult } from './jobs'; + +/** Hosted apps are stateful-only. A fixed isolated queue prevents an ordinary + * stateless worker from ever receiving an AWS lifecycle job. */ +export const HOSTED_APP_QUEUE_NAME = 'stateful-hosted-app-queue'; + +type HostedAppQueue = Queue< + HostedAppJobData, + HostedAppJobResult, + HostedAppJobName +>; + +let hostedAppQueue: HostedAppQueue | undefined; +let hostedAppQueueEvents: QueueEvents | undefined; + +function resources(): { queue: HostedAppQueue; events: QueueEvents } { + if (!hostedAppQueue || !hostedAppQueueEvents) { + hostedAppQueue = new Queue( + HOSTED_APP_QUEUE_NAME, + { connection }, + ); + hostedAppQueueEvents = new QueueEvents(HOSTED_APP_QUEUE_NAME, { connection }); + setMaxListeners(0, hostedAppQueue, hostedAppQueueEvents); + /* These resources are created after lifecycle startup, on first use, so the + * ordinary queue listener registration never sees them. An unhandled + * EventEmitter `error` would otherwise crash an API process during a Redis + * failover. */ + hostedAppQueue.on('error', error => { + logger.error('Hosted app queue error', { error }); + }); + hostedAppQueueEvents.on('error', error => { + logger.error('Hosted app queue events error', { error }); + }); + } + return { queue: hostedAppQueue, events: hostedAppQueueEvents }; +} + +/* A cold start can capture (pull + store), restore (load + push), launch, wait + * for control, start the process, and mint preview credentials. Budget every + * independently bounded leg so the queue waiter cannot abandon valid work. */ +const HOSTED_APP_OPERATION_WAIT_MS = env.CHECKPOINT_TIMEOUT_MS * 5 + + env.LAMBDA_MICROVM_LAUNCH_TIMEOUT_MS * 7 + + env.HOSTED_APP_START_TIMEOUT_MS + + 45_000; + +export async function submitHostedAppJob( + name: HostedAppJobName, + data: HostedAppJobData, + jobId = `happ-${nanoid()}`, +): Promise { + const { queue, events } = resources(); + const job = await queue.add(name, data, { + jobId, + removeOnComplete: { age: 3_600, count: 1_000 }, + removeOnFail: { age: 86_400, count: 1_000 }, + }); + return job.waitUntilFinished(events, HOSTED_APP_OPERATION_WAIT_MS); +} + +/** No-op unless this process submitted hosted-app work. Keeping construction + * lazy means disabled/default-profile services create no extra Redis clients. */ +export async function closeHostedAppQueueResources(): Promise { + const queue = hostedAppQueue; + const events = hostedAppQueueEvents; + hostedAppQueue = undefined; + hostedAppQueueEvents = undefined; + await Promise.all([ + ...(queue ? [queue.close()] : []), + ...(events ? [events.close()] : []), + ]); +} diff --git a/service/src/hosted-app/record.ts b/service/src/hosted-app/record.ts new file mode 100644 index 0000000..84defb8 --- /dev/null +++ b/service/src/hosted-app/record.ts @@ -0,0 +1,28 @@ +import type { ResidentHostedAppSpec } from './spec'; + +/** Durable hosted-app fields carried by the existing fenced MicroVM registry. + * The registry's top-level runtime_session_id is the opaque `happ_*` lease id; + * `source_runtime_session_id` identifies the coding workspace checkpoint that + * was copied into this independent app-host VM. */ +export interface HostedAppRecordDetails { + source_runtime_session_id: string; + app_id: string; + revision: string; + spec_fingerprint: string; + spec: ResidentHostedAppSpec; + checkpoint_key: string; + preview_credential?: string; + preview_credential_expires_at?: number; +} + +export interface HostedAppPublicStatus { + app_id: string; + revision: string; + state: 'starting' | 'running' | 'stopping' | 'stopped' | 'failed'; + preview_id: string; + /** Short-lived owner capability exchange URL on the isolated app origin. */ + preview_url?: string; + hard_deadline_at?: number; + updated_at: number; + error?: string; +} diff --git a/service/src/hosted-app/router.ts b/service/src/hosted-app/router.ts new file mode 100644 index 0000000..6aed37f --- /dev/null +++ b/service/src/hosted-app/router.ts @@ -0,0 +1,207 @@ +import { Router, type Response } from 'express'; +import type { AuthenticatedRequest } from '../types'; +import { env } from '../config'; +import { getExecutionIdentity } from '../execution-identity'; +import { checkServiceShutDown, checkServiceStartUp } from '../lifecycle'; +import { readRuntimeSessionRecord } from '../runtime-session/registry'; +import { + deriveRuntimeSessionId, + validateRuntimeSessionHint, + RuntimeSessionHintError, +} from '../runtime-session/id'; +import { captureTraceCarrier } from '../telemetry'; +import { executionLimiter } from '../middleware/limits'; +import { + assertHostedAppOwned, + HostedAppControlPlaneError, + hostedAppPublicStatus, +} from './control-plane'; +import { submitHostedAppJob } from './queue'; +import { + deriveHostedAppRuntimeId, + HostedAppSpecError, + parseHostedAppStartRequest, + validateHostedAppId, +} from './spec'; +import { + hostedAppPreviewAuthorizeUrl, + hostedAppPreviewOwnerBinding, + signHostedAppPreviewAccess, +} from './preview-access'; +import type { HostedAppPublicStatus } from './record'; +import type { HostedAppPreviewTarget } from './preview-proxy'; + +const router = Router(); +const PREVIEW_LINK_TTL_MS = 5 * 60_000; + +function presentStatus( + status: HostedAppPublicStatus, + owner: { tenantId: string; canonicalUserId: string }, +): HostedAppPublicStatus { + if (status.state !== 'running') return status; + const key = Buffer.from(env.HOSTED_APP_PREVIEW_SIGNING_KEY, 'base64'); + const token = signHostedAppPreviewAccess({ + hostedAppRuntimeId: status.preview_id, + revision: status.revision, + ownerBinding: hostedAppPreviewOwnerBinding(owner, key), + expiresAt: Date.now() + PREVIEW_LINK_TTL_MS, + }, key); + return { + ...status, + preview_url: hostedAppPreviewAuthorizeUrl( + status.preview_id, + env.HOSTED_APP_PREVIEW_ORIGIN, + token, + ), + }; +} + +function unavailable(res: Response): Response | undefined { + if (!env.HOSTED_APPS_ENABLED) { + return res.status(404).json({ error: 'hosted_apps_disabled', message: 'Not Found' }); + } + if (checkServiceShutDown()) { + return res.status(503).json({ error: 'service_shutting_down', message: 'Service is shutting down' }); + } + if (checkServiceStartUp()) { + return res.status(503).json({ error: 'service_starting', message: 'Service is starting up' }); + } + return undefined; +} + +function target( + req: AuthenticatedRequest, + rawAppId: unknown, + rawHint: unknown, +): HostedAppPreviewTarget & { + appId: string; + sourceRuntimeSessionId: string; + identity: { tenantId: string; canonicalUserId: string }; +} { + const appId = validateHostedAppId(rawAppId); + const hint = validateRuntimeSessionHint(rawHint); + if (!hint) throw new HostedAppSpecError('runtime_session_hint is required'); + const identity = getExecutionIdentity(req); + const sourceRuntimeSessionId = deriveRuntimeSessionId({ + storageNamespace: identity.storageNamespace, + canonicalUserId: identity.canonicalUserId, + hint, + }); + return { + appId, + identity, + sourceRuntimeSessionId, + hostedAppRuntimeId: deriveHostedAppRuntimeId(sourceRuntimeSessionId, appId), + }; +} + +function parseWorkerFailure(error: unknown): HostedAppControlPlaneError | undefined { + const message = error instanceof Error ? error.message : String(error); + const jsonStart = message.indexOf('{'); + if (jsonStart < 0) return undefined; + try { + const parsed = JSON.parse(message.slice(jsonStart)) as Record; + if ( + typeof parsed.code === 'string' + && typeof parsed.message === 'string' + && typeof parsed.status === 'number' + ) { + return new HostedAppControlPlaneError( + parsed.code, + parsed.message, + parsed.status, + parsed.transient === true, + ); + } + } catch { + // Fall through to the generic failure below. + } + return undefined; +} + +function sendFailure(error: unknown, res: Response): Response { + if ( + error instanceof HostedAppSpecError + || error instanceof RuntimeSessionHintError + ) { + return res.status(error.status).json({ error: 'invalid_hosted_app_request', message: error.message }); + } + const known = error instanceof HostedAppControlPlaneError + ? error + : parseWorkerFailure(error); + if (known) { + return res.status(known.status).json({ + error: known.code, + message: known.message, + retryable: known.transient, + }); + } + return res.status(503).json({ + error: 'hosted_app_operation_failed', + message: 'Hosted app operation failed', + retryable: true, + }); +} + +router.post('/', executionLimiter, async (req: AuthenticatedRequest, res) => { + if (unavailable(res)) return; + try { + const parsed = parseHostedAppStartRequest(req.body); + const resolved = target(req, parsed.spec.app_id, parsed.runtimeSessionHint); + const result = await submitHostedAppJob('hosted-app:start', { + operation: 'start', + hostedAppRuntimeId: resolved.hostedAppRuntimeId, + sourceRuntimeSessionId: resolved.sourceRuntimeSessionId, + tenantId: resolved.identity.tenantId, + canonicalUserId: resolved.identity.canonicalUserId, + spec: parsed.spec, + _otel: captureTraceCarrier(), + }); + return res.status(200).json(presentStatus(result, { + tenantId: resolved.identity.tenantId, + canonicalUserId: resolved.identity.canonicalUserId, + })); + } catch (error) { + return sendFailure(error, res); + } +}); + +router.get('/:appId', async (req: AuthenticatedRequest, res) => { + if (unavailable(res)) return; + try { + const resolved = target(req, req.params.appId, req.query.runtime_session_hint); + const record = await readRuntimeSessionRecord(resolved.hostedAppRuntimeId); + if (!record?.hosted_app) { + throw new HostedAppControlPlaneError('hosted_app_not_found', 'Hosted app not found', 404); + } + assertHostedAppOwned(record, { + tenantId: resolved.identity.tenantId, + canonicalUserId: resolved.identity.canonicalUserId, + }, resolved.sourceRuntimeSessionId); + return res.status(200).json(presentStatus(hostedAppPublicStatus(record), { + tenantId: resolved.identity.tenantId, + canonicalUserId: resolved.identity.canonicalUserId, + })); + } catch (error) { + return sendFailure(error, res); + } +}); + +router.delete('/:appId', async (req: AuthenticatedRequest, res) => { + if (unavailable(res)) return; + try { + const resolved = target(req, req.params.appId, req.query.runtime_session_hint); + const result = await submitHostedAppJob('hosted-app:stop', { + operation: 'stop', + hostedAppRuntimeId: resolved.hostedAppRuntimeId, + tenantId: resolved.identity.tenantId, + canonicalUserId: resolved.identity.canonicalUserId, + _otel: captureTraceCarrier(), + }); + return res.status(200).json(result); + } catch (error) { + return sendFailure(error, res); + } +}); + +export default router; diff --git a/service/src/hosted-app/source-checkpoint.test.ts b/service/src/hosted-app/source-checkpoint.test.ts new file mode 100644 index 0000000..a7561f6 --- /dev/null +++ b/service/src/hosted-app/source-checkpoint.test.ts @@ -0,0 +1,74 @@ +import { describe, expect, test } from 'bun:test'; +import type { RuntimeSessionRecord } from '../runtime-session/registry'; +import { captureHostedAppSourceCheckpoint } from './source-checkpoint'; + +const owner = { tenantId: 'tenant-1', canonicalUserId: 'user-1' }; + +function record(state: RuntimeSessionRecord['state']): RuntimeSessionRecord { + return { + runtime_session_id: 'rt_source', + tenant_id: owner.tenantId, + canonical_user_id: owner.canonicalUserId, + state, + generation: 1, + launched_at: 1, + last_seen_at: 1, + workspace_checkpoint: 'rtsx-checkpoints/rt_source/0001.tar.gz', + ...(state === 'RUNNING' + ? { microvm_id: 'vm-source', endpoint: 'https://vm-source.test' } + : {}), + }; +} + +function fixture(source: RuntimeSessionRecord | null) { + let current = source; + const calls: string[] = []; + return { + calls, + deps: { + waitForLock: async () => { calls.push('lock'); return 'source-lock'; }, + releaseLock: async () => { calls.push('release'); }, + read: async () => { calls.push('read'); return current; }, + checkpoint: async () => { + calls.push('checkpoint'); + current = current ? { + ...current, + workspace_checkpoint: 'rtsx-checkpoints/rt_source/0002.tar.gz', + } : null; + return 'stored' as const; + }, + }, + }; +} + +describe('hosted app source checkpoint capture', () => { + test('holds one lock across a fresh live checkpoint and its committed pointer read', async () => { + const f = fixture(record('RUNNING')); + const key = await captureHostedAppSourceCheckpoint({ + runtimeSessionId: 'rt_source', owner, signal: new AbortController().signal, + lockWaitMs: 100, deps: f.deps, + }); + expect(key).toBe('rtsx-checkpoints/rt_source/0002.tar.gz'); + expect(f.calls).toEqual(['lock', 'read', 'checkpoint', 'read', 'release']); + }); + + test('reuses a stopped workspace checkpoint without requiring a live source VM', async () => { + const f = fixture(record('TERMINATED')); + const key = await captureHostedAppSourceCheckpoint({ + runtimeSessionId: 'rt_source', owner, signal: new AbortController().signal, + lockWaitMs: 100, deps: f.deps, + }); + expect(key).toBe('rtsx-checkpoints/rt_source/0001.tar.gz'); + expect(f.calls).toEqual(['lock', 'read', 'release']); + }); + + test('fails closed on owner mismatch and still releases the source lock', async () => { + const f = fixture({ ...record('TERMINATED'), canonical_user_id: 'user-2' }); + const error = await captureHostedAppSourceCheckpoint({ + runtimeSessionId: 'rt_source', owner, signal: new AbortController().signal, + lockWaitMs: 100, deps: f.deps, + }).catch(value => value); + expect(error.code).toBe('hosted_app_source_not_found'); + expect(f.calls).toEqual(['lock', 'read', 'release']); + }); +}); diff --git a/service/src/hosted-app/source-checkpoint.ts b/service/src/hosted-app/source-checkpoint.ts new file mode 100644 index 0000000..4e7bd74 --- /dev/null +++ b/service/src/hosted-app/source-checkpoint.ts @@ -0,0 +1,84 @@ +import type { RuntimeSessionRecord } from '../runtime-session/registry'; +import { HostedAppControlPlaneError, type HostedAppOwner } from './control-plane'; + +export interface HostedAppSourceCheckpointDeps { + waitForLock( + runtimeSessionId: string, + args: { waitMs: number; signal: AbortSignal }, + ): Promise; + releaseLock(runtimeSessionId: string, lockToken: string): Promise; + read( + runtimeSessionId: string, + args: { signal: AbortSignal }, + ): Promise; + checkpoint(args: { + runtimeSessionId: string; + lockToken: string; + signal: AbortSignal; + }): Promise<'stored' | 'skipped_busy' | 'skipped_state' | 'failed'>; +} + +/** Freeze one exact source workspace revision for an app. A live VM gets a + * fresh checkpoint; a stopped VM can reuse its last committed immutable + * checkpoint. The source lock spans commit and pointer read. */ +export async function captureHostedAppSourceCheckpoint(args: { + runtimeSessionId: string; + owner: HostedAppOwner; + signal: AbortSignal; + lockWaitMs: number; + deps: HostedAppSourceCheckpointDeps; +}): Promise { + const lockToken = await args.deps.waitForLock(args.runtimeSessionId, { + waitMs: args.lockWaitMs, + signal: args.signal, + }); + if (!lockToken) { + throw new HostedAppControlPlaneError( + 'hosted_app_source_busy', + 'The stateful workspace is busy; retry after its execution completes', + 409, + true, + ); + } + try { + let source = await args.deps.read(args.runtimeSessionId, { signal: args.signal }); + if ( + !source + || source.tenant_id !== args.owner.tenantId + || source.canonical_user_id !== args.owner.canonicalUserId + ) { + throw new HostedAppControlPlaneError( + 'hosted_app_source_not_found', + 'Stateful source workspace not found', + 404, + ); + } + if (source.state === 'RUNNING' && source.microvm_id && source.endpoint) { + const result = await args.deps.checkpoint({ + runtimeSessionId: args.runtimeSessionId, + lockToken, + signal: args.signal, + }); + if (result !== 'stored') { + throw new HostedAppControlPlaneError( + 'hosted_app_checkpoint_failed', + 'Could not capture the current stateful workspace revision', + 503, + true, + ); + } + source = await args.deps.read(args.runtimeSessionId, { signal: args.signal }); + } + if (!source?.workspace_checkpoint) { + throw new HostedAppControlPlaneError( + 'hosted_app_checkpoint_missing', + 'The stateful workspace has no durable checkpoint to host', + 503, + true, + ); + } + return source.workspace_checkpoint; + } finally { + await args.deps.releaseLock(args.runtimeSessionId, lockToken); + } +} diff --git a/service/src/hosted-app/spec.test.ts b/service/src/hosted-app/spec.test.ts new file mode 100644 index 0000000..26417c7 --- /dev/null +++ b/service/src/hosted-app/spec.test.ts @@ -0,0 +1,70 @@ +import { describe, expect, test } from 'bun:test'; +import { + deriveHostedAppRuntimeId, + hostedAppSpecFingerprint, + parseHostedAppStartRequest, +} from './spec'; + +const valid = () => ({ + runtime_session_hint: 'conversation-1', + app_id: 'demo-app', + revision: 'rev-1', + language: 'node', + version: '>=22', + entrypoint: 'src/server.js', +}); + +describe('hosted app spec', () => { + test('normalizes the resident adapter defaults', () => { + expect(parseHostedAppStartRequest(valid())).toEqual({ + runtimeSessionHint: 'conversation-1', + spec: { + adapter: 'resident', + app_id: 'demo-app', + revision: 'rev-1', + language: 'node', + version: '>=22', + entrypoint: 'src/server.js', + cwd: '.', + args: [], + env: {}, + }, + }); + }); + + test('requires the stateful session hint and rejects unsupported adapters', () => { + expect(() => parseHostedAppStartRequest({ ...valid(), runtime_session_hint: '' })) + .toThrow('runtime_session_hint is required'); + expect(() => parseHostedAppStartRequest({ ...valid(), adapter: 'static' })) + .toThrow('adapter must be "resident"'); + }); + + test('rejects traversal, non-canonical paths, and runner-owned networking env', () => { + expect(() => parseHostedAppStartRequest({ ...valid(), entrypoint: '../server.js' })) + .toThrow('canonical relative path'); + expect(() => parseHostedAppStartRequest({ ...valid(), cwd: 'src/../src' })) + .toThrow('canonical relative path'); + expect(() => parseHostedAppStartRequest({ ...valid(), env: { port: '9999' } })) + .toThrow('runner-controlled'); + expect(() => parseHostedAppStartRequest({ ...valid(), env: { BASH_ENV: 'bootstrap.sh' } })) + .toThrow('runner-controlled'); + expect(() => parseHostedAppStartRequest({ ...valid(), env: { LD_PRELOAD: './evil.so' } })) + .toThrow('runner-controlled'); + }); + + test('fingerprints equivalent env maps identically and changed launch settings differently', () => { + const a = parseHostedAppStartRequest({ ...valid(), env: { B: '2', A: '1' } }).spec; + const b = parseHostedAppStartRequest({ ...valid(), env: { A: '1', B: '2' } }).spec; + const changed = { ...b, args: ['--changed'] }; + expect(hostedAppSpecFingerprint(a)).toBe(hostedAppSpecFingerprint(b)); + expect(hostedAppSpecFingerprint(a)).not.toBe(hostedAppSpecFingerprint(changed)); + }); + + test('derives a stable owner-scoped opaque runtime id', () => { + const first = deriveHostedAppRuntimeId('rt_owner_a', 'demo'); + expect(first).toMatch(/^happ_[0-9a-f]{40}$/); + expect(deriveHostedAppRuntimeId('rt_owner_a', 'demo')).toBe(first); + expect(deriveHostedAppRuntimeId('rt_owner_b', 'demo')).not.toBe(first); + expect(deriveHostedAppRuntimeId('rt_owner_a', 'other')).not.toBe(first); + }); +}); diff --git a/service/src/hosted-app/spec.ts b/service/src/hosted-app/spec.ts new file mode 100644 index 0000000..6ee2201 --- /dev/null +++ b/service/src/hosted-app/spec.ts @@ -0,0 +1,228 @@ +import { createHash } from 'node:crypto'; +import * as path from 'node:path'; +import { validateRuntimeSessionHint } from '../runtime-session/id'; + +const APP_ID_PATTERN = /^[A-Za-z0-9][A-Za-z0-9._-]{0,63}$/; +export const HOSTED_APP_REVISION_PATTERN = /^[A-Za-z0-9][A-Za-z0-9._:-]{0,127}$/; +const ENV_NAME_PATTERN = /^[A-Za-z_][A-Za-z0-9_]*$/; +const MAX_ARGS = 64; +const MAX_ARG_BYTES = 4_096; +const MAX_ENV_VARS = 64; +const MAX_ENV_VALUE_BYTES = 4_096; +const MAX_ENV_BYTES = 32 * 1_024; +const MAX_PATH_LENGTH = 256; +const MAX_PATH_DEPTH = 10; +/* Mirror the app-host runner's root-launch filter. Rejecting here makes the + * immutable control-plane spec match what the runner actually applies. */ +const RESERVED_ENV_KEYS = new Set([ + 'OPENBLAS_NUM_THREADS', + 'MKL_NUM_THREADS', + 'OMP_NUM_THREADS', + 'SANDBOX_LANGUAGE', + 'HOME', + 'PATH', + 'TOOL_CALL_SOCKET', + 'PYTHONPATH', + 'PYTHONSTARTUP', + 'PYTHONHOME', + 'PYTHONEXECUTABLE', + 'PYTHONIOENCODING', + 'NODE_OPTIONS', + 'NODE_PATH', + 'BASH_ENV', + 'ENV', + 'PROMPT_COMMAND', + 'IFS', + 'SHELLOPTS', + 'BASHOPTS', + 'GLIBC_TUNABLES', + 'PTC_HISTORY_PATH', + 'PORT', + 'HOST', +]); +const RESERVED_ENV_PREFIXES = ['LD_', 'DYLD_', 'PTC_']; + +export interface ResidentHostedAppSpec { + adapter: 'resident'; + app_id: string; + revision: string; + language: string; + version: string; + entrypoint: string; + cwd: string; + args: string[]; + env: Record; +} + +export interface HostedAppStartRequest extends Omit { + adapter?: 'resident'; + runtime_session_hint: string; + cwd?: string; + args?: string[]; + env?: Record; +} + +export class HostedAppSpecError extends Error { + readonly status = 400; + + constructor(message: string) { + super(message); + this.name = 'HostedAppSpecError'; + } +} + +export function validateHostedAppId(value: unknown): string { + if (typeof value !== 'string' || !APP_ID_PATTERN.test(value)) { + throw new HostedAppSpecError('app_id is malformed'); + } + return value; +} + +function isPlainObject(value: unknown): value is Record { + if (typeof value !== 'object' || value === null || Array.isArray(value)) return false; + const proto = Object.getPrototypeOf(value); + return proto === Object.prototype || proto === null; +} + +function boundedString( + source: Record, + key: string, + maxBytes: number, +): string { + const value = source[key]; + if ( + typeof value !== 'string' + || value.length === 0 + || value.includes('\0') + || Buffer.byteLength(value, 'utf8') > maxBytes + ) { + throw new HostedAppSpecError(`${key} must be a non-empty bounded string`); + } + return value; +} + +function canonicalRelativePath(value: string, field: string, allowDot: boolean): string { + if ( + value.includes('\0') + || path.posix.isAbsolute(value) + || path.posix.normalize(value) !== value + || value.endsWith('/') + || (!allowDot && value === '.') + || value === '' + || value.length > MAX_PATH_LENGTH + || value.split('/').filter(Boolean).length > MAX_PATH_DEPTH + || value === '..' + || value.startsWith('../') + ) { + throw new HostedAppSpecError(`${field} must be a canonical relative path`); + } + return value; +} + +export function parseHostedAppStartRequest(raw: unknown): { + runtimeSessionHint: string; + spec: ResidentHostedAppSpec; +} { + if (!isPlainObject(raw)) throw new HostedAppSpecError('request body must be an object'); + if (raw.adapter !== undefined && raw.adapter !== 'resident') { + throw new HostedAppSpecError('adapter must be "resident"'); + } + + const runtimeSessionHint = validateRuntimeSessionHint(raw.runtime_session_hint); + if (!runtimeSessionHint) { + throw new HostedAppSpecError('runtime_session_hint is required'); + } + const appId = boundedString(raw, 'app_id', 64); + validateHostedAppId(appId); + const revision = boundedString(raw, 'revision', 128); + if (!HOSTED_APP_REVISION_PATTERN.test(revision)) { + throw new HostedAppSpecError('revision is malformed'); + } + const language = boundedString(raw, 'language', 64); + const version = boundedString(raw, 'version', 128); + const entrypoint = canonicalRelativePath( + boundedString(raw, 'entrypoint', MAX_PATH_LENGTH), + 'entrypoint', + false, + ); + const rawCwd = raw.cwd ?? '.'; + if (typeof rawCwd !== 'string') throw new HostedAppSpecError('cwd must be a string'); + const cwd = canonicalRelativePath(rawCwd, 'cwd', true); + + const rawArgs = raw.args ?? []; + if ( + !Array.isArray(rawArgs) + || rawArgs.length > MAX_ARGS + || rawArgs.some(value => ( + typeof value !== 'string' + || value.includes('\0') + || Buffer.byteLength(value, 'utf8') > MAX_ARG_BYTES + )) + ) { + throw new HostedAppSpecError(`args must contain at most ${MAX_ARGS} bounded strings`); + } + + const rawEnv = raw.env ?? {}; + if (!isPlainObject(rawEnv) || Object.keys(rawEnv).length > MAX_ENV_VARS) { + throw new HostedAppSpecError(`env must be an object with at most ${MAX_ENV_VARS} entries`); + } + const env: Record = {}; + let envBytes = 0; + for (const [key, value] of Object.entries(rawEnv)) { + if ( + !ENV_NAME_PATTERN.test(key) + || typeof value !== 'string' + || value.includes('\0') + || Buffer.byteLength(value, 'utf8') > MAX_ENV_VALUE_BYTES + ) { + throw new HostedAppSpecError(`env.${key} is invalid`); + } + const upperKey = key.toUpperCase(); + if ( + RESERVED_ENV_KEYS.has(upperKey) + || RESERVED_ENV_PREFIXES.some(prefix => upperKey.startsWith(prefix)) + ) { + throw new HostedAppSpecError(`env.${key} is runner-controlled`); + } + envBytes += Buffer.byteLength(key, 'utf8') + Buffer.byteLength(value, 'utf8'); + if (envBytes > MAX_ENV_BYTES) throw new HostedAppSpecError('env is too large'); + env[key] = value; + } + + return { + runtimeSessionHint, + spec: { + adapter: 'resident', + app_id: appId, + revision, + language, + version, + entrypoint, + cwd, + args: [...rawArgs] as string[], + env, + }, + }; +} + +function canonicalSpec(spec: ResidentHostedAppSpec): string { + return JSON.stringify({ + ...spec, + env: Object.fromEntries(Object.entries(spec.env).sort(([a], [b]) => a.localeCompare(b))), + }); +} + +export function hostedAppSpecFingerprint(spec: ResidentHostedAppSpec): string { + return createHash('sha256').update(canonicalSpec(spec), 'utf8').digest('hex'); +} + +/** Opaque, owner-scoped identity used by Redis and preview URLs. */ +export function deriveHostedAppRuntimeId(runtimeSessionId: string, appId: string): string { + const digest = createHash('sha256') + .update(runtimeSessionId, 'utf8') + .update('\0', 'utf8') + .update(appId, 'utf8') + .digest('hex') + .slice(0, 40); + return `happ_${digest}`; +} diff --git a/service/src/hosted-app/worker.test.ts b/service/src/hosted-app/worker.test.ts new file mode 100644 index 0000000..2b87012 --- /dev/null +++ b/service/src/hosted-app/worker.test.ts @@ -0,0 +1,42 @@ +import { describe, expect, test } from 'bun:test'; +import { HostedAppControlPlaneError } from './control-plane'; +import { HostedAppMicrovmError } from './microvm-runtime'; +import { serializedHostedAppFailure } from './worker'; + +function wire(error: unknown): Record { + return JSON.parse(serializedHostedAppFailure(error).message) as Record; +} + +describe('hosted app worker error boundary', () => { + test('preserves safe control-plane and runner validation classifications', () => { + expect(wire(new HostedAppControlPlaneError('busy', 'Try later', 409, true))) + .toEqual({ code: 'busy', status: 409, message: 'Try later', transient: true }); + expect(wire(new HostedAppMicrovmError( + 'hosted_app_start_failed', + 'runtime node@99 is not installed', + false, + undefined, + 400, + ))).toEqual({ + code: 'hosted_app_start_failed', + status: 400, + message: 'runtime node@99 is not installed', + transient: false, + }); + }); + + test('redacts provider details while preserving retryability', () => { + const result = wire(new HostedAppMicrovmError( + 'hosted_app_launch_failed', + 'AWS arn:secret leaked detail', + true, + )); + expect(result).toEqual({ + code: 'hosted_app_launch_failed', + status: 503, + message: 'Hosted app infrastructure operation failed', + transient: true, + }); + expect(JSON.stringify(result)).not.toContain('arn:secret'); + }); +}); diff --git a/service/src/hosted-app/worker.ts b/service/src/hosted-app/worker.ts new file mode 100644 index 0000000..368f494 --- /dev/null +++ b/service/src/hosted-app/worker.ts @@ -0,0 +1,123 @@ +import { Worker } from 'bullmq'; +import { env } from '../config'; +import { connection } from '../queue'; +import { withSpan, withTraceContext } from '../telemetry'; +import { HostedAppControlPlaneError } from './control-plane'; +import { HostedAppMicrovmError } from './microvm-runtime'; +import type { HostedAppJob, HostedAppJobData, HostedAppJobName, HostedAppJobResult } from './jobs'; +import { HOSTED_APP_QUEUE_NAME } from './queue'; +import logger from '../logger'; +import { workerRunning } from '../metrics'; + +export function serializedHostedAppFailure(error: unknown): Error { + if (error instanceof HostedAppControlPlaneError) { + return new Error(JSON.stringify({ + code: error.code, + status: error.status, + message: error.message, + transient: error.transient, + })); + } + if (error instanceof HostedAppMicrovmError) { + const status = error.httpStatus >= 400 && error.httpStatus < 500 + ? error.httpStatus + : error.code === 'hosted_app_start_failed' && error.httpStatus === 504 + ? 504 + : 503; + return new Error(JSON.stringify({ + code: error.code, + status, + message: status < 500 + ? error.message + : status === 504 + ? 'Hosted app did not become ready before the startup deadline' + : 'Hosted app infrastructure operation failed', + transient: error.transient, + })); + } + return error instanceof Error ? error : new Error('Hosted app operation failed'); +} + +export async function processHostedAppJob(job: HostedAppJob): Promise { + return withTraceContext(job.data._otel, () => withSpan('codeapi.hosted_app.process', { + 'messaging.system': 'bullmq', + 'messaging.operation.name': job.name, + 'messaging.message.id': String(job.id ?? ''), + 'codeapi.hosted_app.id': job.data.hostedAppRuntimeId, + }, async () => { + const controller = new AbortController(); + const timeoutMs = env.CHECKPOINT_TIMEOUT_MS * 5 + + env.LAMBDA_MICROVM_LAUNCH_TIMEOUT_MS * 7 + + env.HOSTED_APP_START_TIMEOUT_MS + + 30_000; + const timer = setTimeout( + () => controller.abort(new Error(`Hosted app operation timed out after ${timeoutMs}ms`)), + timeoutMs, + ); + timer.unref?.(); + try { + /* Keep AWS SDK and checkpoint-store construction out of default-profile + * workers; this import is reached only by an enabled hosted-app job. */ + const control = (await import('./factory')).getHostedAppControlPlane(); + if (job.name === 'hosted-app:start' && job.data.operation === 'start') { + return await control.start({ + ...job.data, + signal: controller.signal, + }); + } + if (job.name === 'hosted-app:stop' && job.data.operation === 'stop') { + return await control.stop( + job.data.hostedAppRuntimeId, + job.data, + controller.signal, + ); + } + if ( + job.name === 'hosted-app:refresh-preview' + && job.data.operation === 'refresh-preview' + ) { + return await control.refreshPreview( + job.data.hostedAppRuntimeId, + job.data, + controller.signal, + ); + } + throw new HostedAppControlPlaneError( + 'hosted_app_job_invalid', + 'Hosted app job name and payload do not match', + 400, + ); + } catch (error) { + throw serializedHostedAppFailure(error); + } finally { + clearTimeout(timer); + } + }, 'CONSUMER')); +} + +export const hostedAppWorker: Worker< + HostedAppJobData, + HostedAppJobResult, + HostedAppJobName +> | undefined = env.HOSTED_APPS_ENABLED + ? new Worker(HOSTED_APP_QUEUE_NAME, processHostedAppJob, { + connection, + /* Lifecycle transitions are serialized again by their per-app Redis lock. + * This modest concurrency allows unrelated apps to launch in parallel while + * the fleet-wide AWS throttle remains authoritative. */ + concurrency: Math.max(1, Math.min(env.OTHER_CONCURRENCY, 4)), + }) + : undefined; + +if (hostedAppWorker) workerRunning.set({ worker_type: 'hosted-app' }, 1); + +hostedAppWorker?.on('failed', (job, error) => { + logger.error('Hosted app job failed', { jobId: job?.id, error }); +}); +hostedAppWorker?.on('error', error => { + logger.error('Hosted app worker error', { error }); + workerRunning.set({ worker_type: 'hosted-app' }, 0); +}); +hostedAppWorker?.on('closed', () => { + workerRunning.set({ worker_type: 'hosted-app' }, 0); +}); diff --git a/service/src/lifecycle.ts b/service/src/lifecycle.ts index 8fc0adc..0de35c3 100644 --- a/service/src/lifecycle.ts +++ b/service/src/lifecycle.ts @@ -6,12 +6,14 @@ import { env } from './config'; import { validateApiHardenedConfig, validateExecutionProfilePolicy, + validateHostedAppsApiConfig, validateSandboxBackendPolicy, validateWorkerHardenedConfig, } from './secure-startup'; import logger from './logger'; import { shutdownTelemetry } from './telemetry'; import { configureExecutionProfileMetrics } from './metrics'; +import { closeHostedAppQueueResources } from './hosted-app/queue'; const { INSTANCE_ID } = env; let isShuttingDown = false; @@ -90,6 +92,7 @@ export async function startupApiOnly(): Promise { logger.info('Starting API service (no workers)...'); validateApiHardenedConfig(); validateExecutionProfilePolicy({ requireBackendMatch: false }); + validateHostedAppsApiConfig(); /* No validateSandboxBackendPolicy() here: an API-only pod authenticates and * enqueues jobs, it never constructs the Lambda backend or checkpoint store. * Validating that policy would force worker-only config (LAMBDA_MICROVM_* and @@ -119,6 +122,9 @@ export async function startupWorkerOnly(): Promise { // Dynamically import workers to start them const { pyWorker, otherWorker } = await import('./workers'); + const hostedAppWorker = env.HOSTED_APPS_ENABLED + ? (await import('./hosted-app/worker')).hostedAppWorker + : undefined; registerWorkers(); @@ -134,6 +140,9 @@ export async function startupWorkerOnly(): Promise { throw new Error('Other worker is not running'); } logger.info('Workers health check passed'); + if (env.HOSTED_APPS_ENABLED && !hostedAppWorker?.isRunning()) { + throw new Error('Hosted app worker is not running'); + } }; checkWorkers(); @@ -150,6 +159,7 @@ async function gracefulStartup(): Promise { validateApiHardenedConfig(); validateWorkerHardenedConfig(); validateExecutionProfilePolicy(); + validateHostedAppsApiConfig(); validateSandboxBackendPolicy(); await validateLifecycleAuthConfig(); configureProfileMetrics(); @@ -159,6 +169,9 @@ async function gracefulStartup(): Promise { // Import workers (this starts them) const { pyWorker, otherWorker } = await import('./workers'); + const hostedAppWorker = env.HOSTED_APPS_ENABLED + ? (await import('./hosted-app/worker')).hostedAppWorker + : undefined; registerWorkers(); @@ -178,6 +191,9 @@ async function gracefulStartup(): Promise { throw new Error('Other worker is not running'); } logger.info('Workers health check passed'); + if (env.HOSTED_APPS_ENABLED && !hostedAppWorker?.isRunning()) { + throw new Error('Hosted app worker is not running'); + } }; checkWorkers(); @@ -219,12 +235,18 @@ export async function gracefulShutdown(): Promise { if (hasWorkers) { // Worker shutdown: close workers gracefully const { pyWorker, otherWorker } = await import('./workers'); + const hostedAppWorker = env.HOSTED_APPS_ENABLED + ? (await import('./hosted-app/worker')).hostedAppWorker + : undefined; // Pause workers and wait for active jobs to complete // Note: We pause workers, NOT queues (queues are shared) // pause(false) = wait for active jobs to finish before resolving (doNotWaitActive=false) // pause(true) = return immediately without waiting for active jobs - const pauseAndDrain = async (worker: typeof pyWorker, name: string): Promise => { + const pauseAndDrain = async ( + worker: { pause(doNotWaitActive?: boolean): Promise }, + name: string, + ): Promise => { logger.info(`Pausing ${name} worker and waiting for active jobs to drain...`); try { // doNotWaitActive=false means wait for active jobs to complete @@ -237,13 +259,15 @@ export async function gracefulShutdown(): Promise { await Promise.all([ pauseAndDrain(pyWorker, 'Python'), - pauseAndDrain(otherWorker, 'Other') + pauseAndDrain(otherWorker, 'Other'), + ...(hostedAppWorker ? [pauseAndDrain(hostedAppWorker, 'Hosted app')] : []), ]); // Close workers await Promise.all([ pyWorker.close(), - otherWorker.close() + otherWorker.close(), + ...(hostedAppWorker ? [hostedAppWorker.close()] : []), ]); logger.info('Workers closed'); } @@ -253,7 +277,8 @@ export async function gracefulShutdown(): Promise { pyQueue.close(), otherQueue.close(), pyQueueEvents.close(), - otherQueueEvents.close() + otherQueueEvents.close(), + closeHostedAppQueueResources(), ]); logger.info('Queue connections closed'); diff --git a/service/src/middleware/httpMetrics.test.ts b/service/src/middleware/httpMetrics.test.ts new file mode 100644 index 0000000..0a507a7 --- /dev/null +++ b/service/src/middleware/httpMetrics.test.ts @@ -0,0 +1,15 @@ +import { describe, expect, test } from 'bun:test'; +import type { Request, Response } from 'express'; +import { httpMetricPath } from './httpMetrics'; + +describe('HTTP metric path overrides', () => { + test('collapses arbitrary hosted-app routes to one bounded label', () => { + const req = { path: '/generated/assets/nonce-123.js' } as Request; + const res = { + locals: { codeapiMetricPath: '/hosted-app-preview/*' }, + } as unknown as Response; + + expect(httpMetricPath(req, res)).toBe('/hosted-app-preview/*'); + expect(httpMetricPath(req, { locals: {} } as unknown as Response)).toBe(req.path); + }); +}); diff --git a/service/src/middleware/httpMetrics.ts b/service/src/middleware/httpMetrics.ts index 19a0d31..a202450 100644 --- a/service/src/middleware/httpMetrics.ts +++ b/service/src/middleware/httpMetrics.ts @@ -8,6 +8,11 @@ function expressRouteLabel(req: Request): string { return 'unmatched'; } +export function httpMetricPath(req: Request, res: Response): string { + const override = res.locals?.codeapiMetricPath; + return typeof override === 'string' && override.length > 0 ? override : req.path; +} + export function httpMetricsMiddleware(req: Request, res: Response, next: NextFunction): void { const start = httpLatencyStartMs(); let recorded = false; @@ -22,7 +27,7 @@ export function httpMetricsMiddleware(req: Request, res: Response, next: NextFun recordHttpRequest({ method: req.method, route: expressRouteLabel(req), - rawPath: req.path, + rawPath: httpMetricPath(req, res), statusCode, durationSeconds, }); diff --git a/service/src/runtime-session/registry.ts b/service/src/runtime-session/registry.ts index f8d6650..deed37d 100644 --- a/service/src/runtime-session/registry.ts +++ b/service/src/runtime-session/registry.ts @@ -6,6 +6,7 @@ import { RUNTIME_SESSION_REDIS_COMMAND_TIMEOUT_MS, } from '../config'; import logger from '../logger'; +import type { HostedAppRecordDetails } from '../hosted-app/record'; export { RUNTIME_SESSION_REDIS_COMMAND_TIMEOUT_MS } from '../config'; @@ -54,6 +55,10 @@ export interface RuntimeSessionRecord { workspace_checkpoint?: string; checkpointed_at?: number; last_error?: string; + /** Present only for a dedicated hosted-app MicroVM lease. Reusing this + * registry gives app launches the same lock fencing, crash recovery, and + * provider-idempotency guarantees as stateful execution VMs. */ + hosted_app?: HostedAppRecordDetails; } const SESS_PREFIX = 'rtsx:sess:'; diff --git a/service/src/secure-startup.test.ts b/service/src/secure-startup.test.ts index 4aa603e..f951976 100644 --- a/service/src/secure-startup.test.ts +++ b/service/src/secure-startup.test.ts @@ -4,6 +4,7 @@ import { validateApiHardenedConfig, validateEgressGatewayHardenedConfig, validateExecutionProfilePolicy, + validateHostedAppsApiConfig, validateSandboxBackendPolicy, validateWorkerHardenedConfig, } from './secure-startup'; @@ -41,6 +42,18 @@ const saved = { ledgerRequired: env.EGRESS_LEDGER_REQUIRED, fileServerUrl: env.EGRESS_GATEWAY_FILE_SERVER_URL, toolCallUrl: env.EGRESS_GATEWAY_TOOL_CALL_SERVER_URL, + hostedAppsEnabled: env.HOSTED_APPS_ENABLED, + hostedAppImageArn: env.HOSTED_APP_IMAGE_ARN, + hostedAppImageVersion: env.HOSTED_APP_IMAGE_VERSION, + hostedAppControlPort: env.HOSTED_APP_CONTROL_PORT, + hostedAppPreviewPort: env.HOSTED_APP_PREVIEW_PORT, + hostedAppMaxDuration: env.HOSTED_APP_MAX_DURATION_SECONDS, + hostedAppIdle: env.HOSTED_APP_IDLE_SECONDS, + hostedAppSuspend: env.HOSTED_APP_SUSPEND_SECONDS, + hostedAppStartTimeout: env.HOSTED_APP_START_TIMEOUT_MS, + hostedAppCredentialKey: env.HOSTED_APP_CREDENTIAL_KEY, + hostedAppPreviewOrigin: env.HOSTED_APP_PREVIEW_ORIGIN, + hostedAppPreviewSigningKey: env.HOSTED_APP_PREVIEW_SIGNING_KEY, }; function restore(): void { @@ -79,6 +92,18 @@ function restore(): void { env.EGRESS_LEDGER_REQUIRED = saved.ledgerRequired; env.EGRESS_GATEWAY_FILE_SERVER_URL = saved.fileServerUrl; env.EGRESS_GATEWAY_TOOL_CALL_SERVER_URL = saved.toolCallUrl; + env.HOSTED_APPS_ENABLED = saved.hostedAppsEnabled; + env.HOSTED_APP_IMAGE_ARN = saved.hostedAppImageArn; + env.HOSTED_APP_IMAGE_VERSION = saved.hostedAppImageVersion; + env.HOSTED_APP_CONTROL_PORT = saved.hostedAppControlPort; + env.HOSTED_APP_PREVIEW_PORT = saved.hostedAppPreviewPort; + env.HOSTED_APP_MAX_DURATION_SECONDS = saved.hostedAppMaxDuration; + env.HOSTED_APP_IDLE_SECONDS = saved.hostedAppIdle; + env.HOSTED_APP_SUSPEND_SECONDS = saved.hostedAppSuspend; + env.HOSTED_APP_START_TIMEOUT_MS = saved.hostedAppStartTimeout; + env.HOSTED_APP_CREDENTIAL_KEY = saved.hostedAppCredentialKey; + env.HOSTED_APP_PREVIEW_ORIGIN = saved.hostedAppPreviewOrigin; + env.HOSTED_APP_PREVIEW_SIGNING_KEY = saved.hostedAppPreviewSigningKey; } afterEach(restore); @@ -431,3 +456,91 @@ describe('sandbox backend policy', () => { expect(() => validateSandboxBackendPolicy()).toThrow('LAMBDA_MICROVM_ALLOW_SHELL'); }); }); + +describe('hosted app startup policy', () => { + function configureHostedApps(): void { + env.HOSTED_APPS_ENABLED = true; + env.EXECUTION_PROFILE = 'stateful'; + env.EXECUTION_PROFILE_SOURCE = 'explicit'; + env.SANDBOX_BACKEND = 'lambda-microvm'; + env.RUNTIME_SESSION_MODE = 'affinity'; + env.SESSION_CHECKPOINTS = true; + env.HOSTED_APP_IMAGE_ARN = 'arn:aws:lambda:us-east-2:1:microvm-image:app-host'; + env.HOSTED_APP_IMAGE_VERSION = '4'; + env.HOSTED_APP_CONTROL_PORT = 8080; + env.HOSTED_APP_PREVIEW_PORT = 3000; + env.HOSTED_APP_MAX_DURATION_SECONDS = 28_800; + env.HOSTED_APP_IDLE_SECONDS = 300; + env.HOSTED_APP_SUSPEND_SECONDS = 900; + env.HOSTED_APP_START_TIMEOUT_MS = 30_000; + env.HOSTED_APP_CREDENTIAL_KEY = Buffer.alloc(32, 7).toString('base64'); + env.HOSTED_APP_PREVIEW_ORIGIN = 'https://apps.example.test'; + env.HOSTED_APP_PREVIEW_SIGNING_KEY = Buffer.alloc(32, 8).toString('base64'); + } + + test('API-only pods require the stateful profile and a 32-byte credential key', () => { + configureHostedApps(); + expect(() => validateHostedAppsApiConfig()).not.toThrow(); + + env.EXECUTION_PROFILE = 'default'; + expect(() => validateHostedAppsApiConfig()).toThrow('stateful execution profile'); + + env.EXECUTION_PROFILE = 'stateful'; + env.HOSTED_APP_CREDENTIAL_KEY = 'not-a-key'; + expect(() => validateHostedAppsApiConfig()).toThrow('exactly 32 bytes'); + + env.HOSTED_APP_CREDENTIAL_KEY = Buffer.alloc(32, 7).toString('base64'); + env.HOSTED_APP_PREVIEW_SIGNING_KEY = env.HOSTED_APP_CREDENTIAL_KEY; + expect(() => validateHostedAppsApiConfig()).toThrow('must be distinct'); + + env.HOSTED_APP_PREVIEW_SIGNING_KEY = Buffer.alloc(32, 8).toString('base64'); + env.HOSTED_APP_PREVIEW_ORIGIN = 'https://apps.example.test/path'; + expect(() => validateHostedAppsApiConfig()).toThrow('bare HTTPS origin'); + }); + + test('worker policy requires a pinned dedicated image and its fixed listener contract', () => { + configureHostedApps(); + /* Reuse the suite's valid Lambda/checkpoint baseline. */ + env.PTC_MODE = 'replay'; + env.HARDENED_SANDBOX_MODE = false; + env.LAMBDA_MICROVM_IMAGE_ARN = 'arn:aws:lambda:us-east-2:1:microvm-image:codeapi'; + env.LAMBDA_MICROVM_IMAGE_VERSION = '3'; + env.LAMBDA_MICROVM_PORT = 8080; + env.LAMBDA_MICROVM_MAX_DURATION_SECONDS = 28_800; + env.LAMBDA_MICROVM_IDLE_SECONDS = 1_800; + env.LAMBDA_MICROVM_SUSPEND_SECONDS = 1_800; + env.LAMBDA_MICROVM_AUTH_TOKEN_TTL_SECONDS = 300; + env.LAMBDA_MICROVM_LAUNCH_TIMEOUT_MS = 60_000; + env.LAMBDA_MICROVM_HEALTH_TIMEOUT_MS = 5_000; + env.LAMBDA_MICROVM_LAUNCH_TPS = 4; + env.LAMBDA_MICROVM_TOKEN_TPS = 8; + env.LAMBDA_MICROVM_ALLOW_SHELL = false; + env.JOB_TIMEOUT = 300_000; + env.RUNTIME_SESSION_LOCK_WAIT_MS = 15_000; + env.CHECKPOINT_MAX_BYTES = 512 * 1024 * 1024; + env.CHECKPOINT_TIMEOUT_MS = 60_000; + process.env.MINIO_ENDPOINT = 'minio'; + process.env.CODEAPI_CHECKPOINT_BUCKET = 'codeapi-checkpoints'; + + expect(() => validateSandboxBackendPolicy()).not.toThrow(); + env.HOSTED_APP_PREVIEW_SIGNING_KEY = ''; + env.HOSTED_APP_PREVIEW_ORIGIN = ''; + expect(() => validateSandboxBackendPolicy()).not.toThrow(); + env.HOSTED_APP_MAX_DURATION_SECONDS = 60; + expect(() => validateSandboxBackendPolicy()).toThrow( + 'LAMBDA_MICROVM_APP_MAX_DURATION_SECONDS', + ); + env.HOSTED_APP_MAX_DURATION_SECONDS = 28_800; + env.HOSTED_APP_IMAGE_VERSION = undefined; + expect(() => validateSandboxBackendPolicy()).toThrow('LAMBDA_MICROVM_APP_IMAGE_VERSION'); + env.HOSTED_APP_IMAGE_VERSION = '4'; + env.HOSTED_APP_PREVIEW_PORT = 3001; + expect(() => validateSandboxBackendPolicy()).toThrow('preview port 3000'); + env.HOSTED_APP_PREVIEW_PORT = 3000; + env.HOSTED_APP_START_TIMEOUT_MS = 30_001; + expect(() => validateSandboxBackendPolicy()).toThrow('30000ms'); + env.HOSTED_APP_START_TIMEOUT_MS = 30_000; + process.env.LAMBDA_MICROVM_APP_PREVIEW_PORT = '4000'; + expect(() => validateSandboxBackendPolicy()).toThrow('cannot override'); + }); +}); diff --git a/service/src/secure-startup.ts b/service/src/secure-startup.ts index a82dabf..a9947f6 100644 --- a/service/src/secure-startup.ts +++ b/service/src/secure-startup.ts @@ -63,6 +63,68 @@ export function validateWorkerHardenedConfig(): void { requireValue('CODEAPI_EXECUTION_MANIFEST_PRIVATE_KEY', env.EXECUTION_MANIFEST_PRIVATE_KEY); } +function hostedAppKey(name: string, raw: string): Buffer { + const normalized = raw.trim(); + const key = Buffer.from(normalized, 'base64'); + if ( + key.length !== 32 + || key.toString('base64').replace(/=+$/, '') !== normalized.replace(/=+$/, '') + ) { + throw new SecureStartupConfigError( + `${name} must be base64 encoding exactly 32 bytes`, + ); + } + return key; +} + +function validateHostedAppsSharedConfig(): Buffer { + if (env.EXECUTION_PROFILE !== 'stateful' || env.RUNTIME_SESSION_MODE === 'stateless') { + throw new SecureStartupConfigError( + 'CODEAPI_HOSTED_APPS_ENABLED=true requires the stateful execution profile', + ); + } + return hostedAppKey('CODEAPI_HOSTED_APP_CREDENTIAL_KEY', env.HOSTED_APP_CREDENTIAL_KEY); +} + +/** API pods decrypt short-lived preview credentials but never receive AWS IAM + * control-plane permissions. Validate only their routing/key contract; the + * worker validator below owns image and checkpoint configuration. */ +export function validateHostedAppsApiConfig(): void { + if (!env.HOSTED_APPS_ENABLED) return; + const credentialKey = validateHostedAppsSharedConfig(); + const previewSigningKey = hostedAppKey( + 'CODEAPI_HOSTED_APP_PREVIEW_SIGNING_KEY', + env.HOSTED_APP_PREVIEW_SIGNING_KEY, + ); + if (credentialKey.equals(previewSigningKey)) { + throw new SecureStartupConfigError( + 'Hosted app credential and preview signing keys must be distinct', + ); + } + let previewOrigin: URL; + try { + previewOrigin = new URL(env.HOSTED_APP_PREVIEW_ORIGIN); + } catch { + throw new SecureStartupConfigError( + 'CODEAPI_HOSTED_APP_PREVIEW_ORIGIN must be an absolute URL', + ); + } + if ( + !['https:', ...(process.env.NODE_ENV === 'production' ? [] : ['http:'])].includes( + previewOrigin.protocol, + ) + || previewOrigin.username + || previewOrigin.password + || previewOrigin.pathname !== '/' + || previewOrigin.search + || previewOrigin.hash + ) { + throw new SecureStartupConfigError( + 'CODEAPI_HOSTED_APP_PREVIEW_ORIGIN must be a bare HTTPS origin', + ); + } +} + /** * Make the endpoint identity trustworthy. Callers route by execution profile, * so accepting a contradictory backend/session tuple would silently send work @@ -201,6 +263,58 @@ export function validateSandboxBackendPolicy(): void { ); } } + + if (env.HOSTED_APPS_ENABLED) { + /* Workers encrypt AWS port credentials but do not serve previews, so they + * need the credential key—not the separate URL-signing key or app origin. */ + validateHostedAppsSharedConfig(); + if (!env.SESSION_CHECKPOINTS) { + throw new SecureStartupConfigError( + 'CODEAPI_HOSTED_APPS_ENABLED=true requires CODEAPI_SESSION_CHECKPOINTS=true', + ); + } + requireValue('LAMBDA_MICROVM_APP_IMAGE_ARN', env.HOSTED_APP_IMAGE_ARN); + requireValue('LAMBDA_MICROVM_APP_IMAGE_VERSION', env.HOSTED_APP_IMAGE_VERSION); + for (const [name, expected] of [ + ['LAMBDA_MICROVM_APP_CONTROL_PORT', '8080'], + ['LAMBDA_MICROVM_APP_PREVIEW_PORT', '3000'], + ['LAMBDA_MICROVM_APP_START_TIMEOUT_MS', '30000'], + ] as const) { + const configured = process.env[name]?.trim(); + if (configured && configured !== expected) { + throw new SecureStartupConfigError( + `${name} cannot override the pinned app-host image contract (${expected})`, + ); + } + } + requireSafeWholeNumber('LAMBDA_MICROVM_APP_CONTROL_PORT', env.HOSTED_APP_CONTROL_PORT, 1_024); + requireSafeWholeNumber('LAMBDA_MICROVM_APP_PREVIEW_PORT', env.HOSTED_APP_PREVIEW_PORT, 1_024); + if (env.HOSTED_APP_CONTROL_PORT !== 8080 || env.HOSTED_APP_PREVIEW_PORT !== 3000) { + throw new SecureStartupConfigError( + 'Hosted app image contract requires control port 8080 and preview port 3000', + ); + } + requireSafeWholeNumber( + 'LAMBDA_MICROVM_APP_MAX_DURATION_SECONDS', + env.HOSTED_APP_MAX_DURATION_SECONDS, + 120, + ); + requireSafeWholeNumber('LAMBDA_MICROVM_APP_IDLE_SECONDS', env.HOSTED_APP_IDLE_SECONDS, 60); + requireSafeWholeNumber('LAMBDA_MICROVM_APP_SUSPEND_SECONDS', env.HOSTED_APP_SUSPEND_SECONDS, 0); + requireSafeWholeNumber('LAMBDA_MICROVM_APP_START_TIMEOUT_MS', env.HOSTED_APP_START_TIMEOUT_MS, 1); + if (env.HOSTED_APP_START_TIMEOUT_MS !== 30_000) { + throw new SecureStartupConfigError( + 'Hosted app image contract requires a 30000ms resident startup timeout', + ); + } + if ( + env.HOSTED_APP_MAX_DURATION_SECONDS > 28_800 + || env.HOSTED_APP_IDLE_SECONDS > 28_800 + || env.HOSTED_APP_SUSPEND_SECONDS > 28_800 + ) { + throw new SecureStartupConfigError('Hosted app lifetime controls must be at most 28800 seconds'); + } + } } export function validateEgressGatewayHardenedConfig(): void { diff --git a/service/src/service-api.ts b/service/src/service-api.ts index ec15b1a..67237ca 100644 --- a/service/src/service-api.ts +++ b/service/src/service-api.ts @@ -8,11 +8,14 @@ import programmaticRouter from './service/programmatic-router'; import { connection } from './queue'; import { env } from './config'; import logger from './logger'; +import hostedAppRouter from './hosted-app/router'; +import { hostedAppPreviewGateway } from './hosted-app/preview-gateway'; const app = express(); app.disable('x-powered-by'); app.set('trust proxy', 1); app.use(executionProfileMiddleware); +app.use(hostedAppPreviewGateway); const v1 = Router(); @@ -30,6 +33,7 @@ app.get('/v1/health', async (_, res) => { v1.use(apiKeyAuth); +v1.use('/hosted-apps', hostedAppRouter); v1.use(serviceRouter); v1.use(programmaticRouter); diff --git a/service/src/worker-server.ts b/service/src/worker-server.ts index 8904904..43f3574 100644 --- a/service/src/worker-server.ts +++ b/service/src/worker-server.ts @@ -47,6 +47,7 @@ import { connection } from './queue'; * The workers are singletons, not created fresh on each import. */ import { pyWorker, otherWorker } from './workers'; +import { hostedAppWorker } from './hosted-app/worker'; const HEALTH_PORT = Number(process.env.WORKER_HEALTH_PORT) || 3113; @@ -101,14 +102,16 @@ const healthServer = http.createServer(async (req, res) => { // Check workers are running const pyRunning = pyWorker.isRunning(); const otherRunning = otherWorker.isRunning(); + const hostedAppsRunning = !env.HOSTED_APPS_ENABLED || hostedAppWorker?.isRunning() === true; - if (pyRunning && otherRunning) { + if (pyRunning && otherRunning && hostedAppsRunning) { res.writeHead(200, { 'Content-Type': 'application/json' }); res.end(JSON.stringify({ status: 'healthy', workers: { python: pyRunning, - other: otherRunning + other: otherRunning, + hostedApps: hostedAppsRunning, }, config: { pythonConcurrency: env.PYTHON_CONCURRENCY, @@ -120,7 +123,7 @@ const healthServer = http.createServer(async (req, res) => { res.writeHead(503, { 'Content-Type': 'application/json' }); res.end(JSON.stringify({ status: 'unhealthy', - workers: { python: pyRunning, other: otherRunning } + workers: { python: pyRunning, other: otherRunning, hostedApps: hostedAppsRunning } })); } } catch (error) { @@ -136,8 +139,9 @@ const healthServer = http.createServer(async (req, res) => { await connection.ping(); const pyRunning = pyWorker.isRunning(); const otherRunning = otherWorker.isRunning(); + const hostedAppsRunning = !env.HOSTED_APPS_ENABLED || hostedAppWorker?.isRunning() === true; - if (pyRunning && otherRunning) { + if (pyRunning && otherRunning && hostedAppsRunning) { res.writeHead(200); res.end('ready'); } else {