diff --git a/.agents/skills/debug-openshell-cluster/SKILL.md b/.agents/skills/debug-openshell-cluster/SKILL.md index cbff462d45..caa2cfe3c4 100644 --- a/.agents/skills/debug-openshell-cluster/SKILL.md +++ b/.agents/skills/debug-openshell-cluster/SKILL.md @@ -268,6 +268,39 @@ If the gateway exits with `failed to read sandbox JWT signing key from `sandbox-jwt` secret at `/etc/openshell-jwt`. The sandbox JWT mount is required even when local Helm values disable TLS. +If `certManager.serverIssuerRef` points the server certificate at an external +Issuer or ClusterIssuer (for example an ACME issuer, for a publicly-trusted +cert on an OpenShift `Route` with TLS passthrough — see +`openshiftRoute.enabled`), check the `Certificate`/`CertificateRequest`/ +`Challenge` resources directly when the secret never becomes Ready: + +```bash +kubectl -n openshell get certificate,certificaterequest,challenge +kubectl -n openshell describe certificate openshell-server +oc -n openshell get route +``` + +ACME issuers reject certificate requests that include internal-only names +(`*.svc.cluster.local`, `localhost`, loopback IPs) and require the +`commonName` to also be a SAN — the server `Certificate` only requests the +hostnames in `certManager.serverDnsNames` when `serverIssuerRef` is set, for +exactly this reason. + +If sandbox supervisors fail their TLS handshake to the gateway with +`UnknownCA` after configuring `serverIssuerRef`, the client (mTLS) certificate +and the server certificate now come from different CAs, and the gateway's +client-verification CA is misconfigured. The chart now fails at render time if +`clientCaFromServerTlsSecret` is still true when `serverIssuerRef` is set, but +this can still occur with pre-existing releases or manual overrides. Set +`certManager.clientCaFromServerTlsSecret=false` and +`server.tls.clientCaSecretName` to a secret that actually contains the CA that +signs the client certificate (`certManager.caSecretName`'s value by default): + +```bash +helm -n openshell get values openshell | grep -E 'clientCaFromServerTlsSecret|clientCaSecretName|serverIssuerRef|caSecretName' +kubectl -n openshell get secret openshell-ca-tls -o jsonpath='{.data.ca\.crt}' | base64 -d | openssl x509 -noout -subject +``` + If `server.providerTokenGrants.spiffe.enabled=true`, the gateway should still render `[openshell.gateway.gateway_jwt]` and mount the `sandbox-jwt` Secret. SPIRE is used only by sandbox pods for dynamic provider token grants. Verify @@ -452,6 +485,8 @@ openshell logs | `K8s namespace not ready` with `envoy-gateway-openshell.yaml: the server could not find the requested resource` | Optional Gateway API manifest was applied without Envoy Gateway CRDs, or k3s Helm controller startup exceeded the namespace wait | Apply `deploy/kube/manifests/envoy-gateway-openshell.yaml` manually only after Envoy Gateway is installed and `grpcRoute` is enabled | | HTTPS ingress (`grpcRoute.gateway.listener.protocol=HTTPS`) connection resets or TLS handshake hangs | Envoy terminates TLS but the gateway pod still expects TLS, so the plaintext backend hop fails | Set `server.disableTls=true` so Envoy forwards plaintext to the pod; verify the listener `certificateRefs` Secret exists in the release namespace and `openshell status` over `https://` | | HTTPS ingress returns `Unauthenticated` after connecting | TLS terminates at Envoy, so the gateway never sees a client cert; no OIDC issuer is configured for identity | Configure `server.oidc.issuer` and register with `openshell gateway add https:// --oidc-issuer `, or set `server.auth.allowUnauthenticatedUsers=true` for a trusted-proxy/dev cluster | +| Server `Certificate` never becomes Ready with `certManager.serverIssuerRef` set | ACME issuer rejected internal-only SANs, a loopback IP, or a `commonName` absent from the SANs | `kubectl -n openshell describe certificate openshell-server`; confirm `certManager.serverDnsNames` lists only real, externally-resolvable hostnames | +| Sandbox supervisors fail TLS handshake with `UnknownCA` after configuring `certManager.serverIssuerRef` | Client (mTLS) cert and server cert now come from different CAs, and `server.tls.clientCaSecretName` still points at the wrong (or default, unpopulated) secret | Set `certManager.clientCaFromServerTlsSecret=false` and `server.tls.clientCaSecretName` to the secret named by `certManager.caSecretName` | ## Reporting diff --git a/Cargo.lock b/Cargo.lock index 9f3f7dcdca..03aa8386e1 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -6942,6 +6942,7 @@ dependencies = [ "tower-layer", "tower-service", "tracing", + "webpki-roots 1.0.7", ] [[package]] diff --git a/crates/openshell-core/Cargo.toml b/crates/openshell-core/Cargo.toml index e138e1eee1..658ebff3f5 100644 --- a/crates/openshell-core/Cargo.toml +++ b/crates/openshell-core/Cargo.toml @@ -14,7 +14,7 @@ repository.workspace = true glob = { workspace = true } prost = { workspace = true } prost-types = { workspace = true } -tonic = { workspace = true, features = ["channel", "tls-native-roots"] } +tonic = { workspace = true, features = ["channel", "tls-webpki-roots"] } tonic-prost = { workspace = true } tokio = { workspace = true } thiserror = { workspace = true } diff --git a/crates/openshell-core/src/grpc_client.rs b/crates/openshell-core/src/grpc_client.rs index 579ee4a5b3..c4272bdbe6 100644 --- a/crates/openshell-core/src/grpc_client.rs +++ b/crates/openshell-core/src/grpc_client.rs @@ -167,8 +167,23 @@ async fn build_plain_channel(endpoint: &str) -> Result { .into_diagnostic() .wrap_err_with(|| format!("failed to read client key from {key_path}"))?; + // Trust the configured CA (self-signed deployments sign the gateway's + // cert with it) *in addition to* the compiled-in WebPKI roots — this + // covers deployments where the gateway's server cert comes from a + // public CA (e.g. an ACME issuer) instead of the same private CA that + // signs this client's identity. + // + // Do NOT add `.with_native_roots()` here: the supervisor runs inside + // the user-selected sandbox image (Docker/Podman drivers), so the + // image's CA bundle is not operator-controlled. An attacker who can + // influence the image and DNS/routing could install their own CA and + // intercept the supervisor→gateway TLS connection. Additionally, + // tonic returns NativeCertsNotFound when the native store is empty, + // which would break minimal BYOC images even though OPENSHELL_TLS_CA + // is valid. let mut tls_config = ClientTlsConfig::new() .ca_certificate(Certificate::from_pem(ca_pem)) + .with_webpki_roots() .identity(Identity::from_pem(cert_pem, key_pem)); if let Ok(server_name) = std::env::var(sandbox_env::GATEWAY_TLS_SERVER_NAME) && !server_name.is_empty() diff --git a/crates/openshell-driver-docker/src/lib.rs b/crates/openshell-driver-docker/src/lib.rs index dd4d9ef0f0..88e4ef8063 100644 --- a/crates/openshell-driver-docker/src/lib.rs +++ b/crates/openshell-driver-docker/src/lib.rs @@ -2250,6 +2250,10 @@ fn build_environment_for_oci_user( environment.remove(openshell_core::sandbox_env::SANDBOX_TOKEN); environment.remove(openshell_core::sandbox_env::SANDBOX_TOKEN_FILE); + // Prevent user-supplied env from overriding the hostname the supervisor + // verifies during TLS. The K8s driver already strips this; Docker and + // Podman must do the same because sandbox images are not operator-controlled. + environment.remove(openshell_core::sandbox_env::GATEWAY_TLS_SERVER_NAME); environment.insert( openshell_core::sandbox_env::OCI_IMAGE_USER.to_string(), oci_user.to_string(), diff --git a/crates/openshell-driver-podman/src/container.rs b/crates/openshell-driver-podman/src/container.rs index 005f688a19..f4cd37d685 100644 --- a/crates/openshell-driver-podman/src/container.rs +++ b/crates/openshell-driver-podman/src/container.rs @@ -483,6 +483,10 @@ fn build_env( env.remove(openshell_core::sandbox_env::SANDBOX_TOKEN); env.remove(openshell_core::sandbox_env::SANDBOX_TOKEN_FILE); + // Prevent user-supplied env from overriding the hostname the supervisor + // verifies during TLS. The K8s driver already strips this; Docker and + // Podman must do the same because sandbox images are not operator-controlled. + env.remove(openshell_core::sandbox_env::GATEWAY_TLS_SERVER_NAME); env.insert( openshell_core::sandbox_env::OCI_IMAGE_USER.into(), oci_user.to_string(), diff --git a/deploy/helm/openshell/README.md b/deploy/helm/openshell/README.md index d4310cb9a7..e8b660df25 100644 --- a/deploy/helm/openshell/README.md +++ b/deploy/helm/openshell/README.md @@ -144,10 +144,11 @@ add `ci/values-spire.yaml` to the OpenShell release values files. | certManager.caSecretName | string | `"openshell-ca-tls"` | Secret created for the intermediate CA (Certificate with isCA: true). | | certManager.certificateDuration | string | `"8760h"` | Duration for cert-manager-issued certificates. | | certManager.certificateRenewBefore | string | `"720h"` | Renewal window for cert-manager-issued certificates. | -| certManager.clientCaFromServerTlsSecret | bool | `true` | Mount gateway client CA from the server TLS secret's ca.crt (populated by cert-manager for certs issued by a CA Issuer). Avoids a separate openshell-server-client-ca Secret. | +| certManager.clientCaFromServerTlsSecret | bool | `true` | Mount gateway client CA from the server TLS secret's ca.crt (populated by cert-manager for certs issued by a CA Issuer). Set to false when serverIssuerRef points at an external issuer (its secret's ca.crt would be that issuer's chain, not the CA that signs the client cert). When false, also set server.tls.clientCaSecretName to a secret containing the actual client CA — for example caSecretName's value, since that's the CA the client certificate above is issued from by default. | | certManager.enabled | bool | `false` | Create cert-manager Issuer and Certificate resources. When enabled, cert-manager owns TLS and the chart runs a JWT-only certgen hook to create the sandbox JWT signing Secret that cert-manager does not manage. | | certManager.serverDnsNames | list | `["openshell","openshell.openshell.svc","openshell.openshell.svc.cluster.local","localhost","openshell.localhost","*.openshell.localhost","host.docker.internal"]` | DNS SANs on the cert-manager-issued server certificate. | | certManager.serverIpAddresses | list | `["127.0.0.1"]` | IP SANs on the cert-manager-issued server certificate. | +| certManager.serverIssuerRef | object | `{"group":"","kind":"","name":""}` | Override the issuerRef for the server Certificate (e.g. a real ACME ClusterIssuer for a publicly-trusted cert on an external hostname). Leave name empty to use the chart's own self-signed CA issuer (default). | | fullnameOverride | string | `""` | Override the full generated resource name. | | grpcRoute.enabled | bool | `false` | Create a Gateway API GRPCRoute for the gateway service. | | grpcRoute.gateway.className | string | `"eg"` | GatewayClass to reference. Envoy Gateway installs one named "eg". | @@ -166,6 +167,9 @@ add `ci/values-spire.yaml` to the OpenShell release values files. | nameOverride | string | `"openshell"` | Override the chart name used in generated resource names. | | networkPolicy.enabled | bool | `true` | Create a NetworkPolicy restricting SSH ingress on sandbox pods to the gateway. | | nodeSelector | object | `{}` | Node selector for the gateway pod. | +| openshiftRoute.annotations | object | `{}` | Extra annotations on the Route (e.g. haproxy.router.openshift.io/*). | +| openshiftRoute.enabled | bool | `false` | Create an OpenShift Route with TLS passthrough. | +| openshiftRoute.host | string | `""` | Hostname for the Route. Must match a SAN on the gateway's server cert. | | pkiInitJob.enabled | bool | `true` | Run a pre-install/pre-upgrade Job that creates gateway and client mTLS Secrets. When certManager.enabled=true, cert-manager owns TLS and this same hook runs in JWT-only mode even if pkiInitJob.enabled remains true. | | pkiInitJob.serverDnsNames | list | `[]` | Extra DNS SANs to append to the server certificate. | | pkiInitJob.serverIpAddresses | list | `[]` | Extra IP SANs to append to the server certificate. | diff --git a/deploy/helm/openshell/ci/values-openshift-route-cert-manager.yaml b/deploy/helm/openshell/ci/values-openshift-route-cert-manager.yaml new file mode 100644 index 0000000000..d06283ee15 --- /dev/null +++ b/deploy/helm/openshell/ci/values-openshift-route-cert-manager.yaml @@ -0,0 +1,33 @@ +# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +# Render-coverage overlay for cert-manager issuing the server certificate from +# an external Issuer/ClusterIssuer (e.g. a real ACME issuer), plus an +# OpenShift Route with TLS passthrough. Merge after values.yaml: +# helm lint deploy/helm/openshell -f ci/values-openshift-route-cert-manager.yaml +# +# The ClusterIssuer name below is a placeholder for render coverage; a real +# deployment must reference an Issuer/ClusterIssuer that's actually installed +# and Ready in the target cluster. See docs/kubernetes/managing-certificates.mdx. + +server: + disableTls: false + grpcEndpoint: "https://openshell.example.com:443" + tls: + # Must name a secret containing the actual client CA when + # clientCaFromServerTlsSecret is false — here, the same CA secret + # cert-manager creates for signing the client (mTLS) certificate. + clientCaSecretName: openshell-ca-tls + +certManager: + enabled: true + clientCaFromServerTlsSecret: false + serverIssuerRef: + name: letsencrypt-prod + kind: ClusterIssuer + serverDnsNames: + - openshell.example.com + +openshiftRoute: + enabled: true + host: openshell.example.com diff --git a/deploy/helm/openshell/templates/cert-manager-pki.yaml b/deploy/helm/openshell/templates/cert-manager-pki.yaml index fdd702a305..a139ecbe14 100644 --- a/deploy/helm/openshell/templates/cert-manager-pki.yaml +++ b/deploy/helm/openshell/templates/cert-manager-pki.yaml @@ -42,6 +42,21 @@ spec: ca: secretName: {{ .Values.certManager.caSecretName | quote }} --- +{{- $externalServerIssuer := .Values.certManager.serverIssuerRef.name }} +{{- if and $externalServerIssuer .Values.certManager.clientCaFromServerTlsSecret }} +{{- fail "certManager.serverIssuerRef.name is set but certManager.clientCaFromServerTlsSecret is still true \u2014 the server cert Secret's ca.crt comes from the external issuer, not the CA that signs the client (mTLS) certificate. Set certManager.clientCaFromServerTlsSecret=false and set server.tls.clientCaSecretName to the secret containing the client CA (e.g. the value of certManager.caSecretName, which defaults to openshell-ca-tls)." }} +{{- end }} +{{- if and (not .Values.certManager.clientCaFromServerTlsSecret) (eq .Values.server.tls.clientCaSecretName "openshell-server-client-ca") }} +{{- fail "certManager.clientCaFromServerTlsSecret is false but server.tls.clientCaSecretName is still the default (openshell-server-client-ca) which is not created by the chart when cert-manager is enabled. Set server.tls.clientCaSecretName to the Secret containing the CA that signs the client (mTLS) certificate (e.g. the value of certManager.caSecretName, which defaults to openshell-ca-tls), or set it to \"\" to disable client-certificate verification entirely." }} +{{- end }} +{{- if $externalServerIssuer }} +{{- range .Values.certManager.serverDnsNames }} +{{- /* Single-label names (e.g. "openshell") are also rejected by ACME CAs but are intentionally not checked here — the guard targets recognisable internal-network patterns. */ -}} +{{- if or (eq . "localhost") (hasSuffix ".localhost" .) (hasSuffix ".svc.cluster.local" .) (hasSuffix ".svc" .) (eq . "host.docker.internal") (eq . "host.containers.internal") }} +{{- fail (printf "certManager.serverIssuerRef.name is set (external issuer) but certManager.serverDnsNames contains %q — external CAs (e.g. ACME / Let's Encrypt) reject internal-only names per CA/Browser Forum baseline requirements. Override certManager.serverDnsNames with your externally-resolvable hostname(s)." .) }} +{{- end }} +{{- end }} +{{- end }} apiVersion: cert-manager.io/v1 kind: Certificate metadata: @@ -53,7 +68,30 @@ spec: secretName: {{ .Values.server.tls.certSecretName | quote }} duration: {{ .Values.certManager.certificateDuration | quote }} renewBefore: {{ .Values.certManager.certificateRenewBefore | quote }} - commonName: openshell-server + {{- if $externalServerIssuer }} + # External issuers (e.g. ACME) reject internal-only names (cluster-local, + # localhost, loopback IPs) per CA/Browser Forum baseline requirements, and + # require any commonName to also appear in dnsNames. Only request the + # externally-resolvable SANs the operator configured — no defaults, no IPs. + {{- if .Values.certManager.serverDnsNames }} + commonName: {{ first .Values.certManager.serverDnsNames | quote }} + {{- end }} + dnsNames: + {{- range .Values.certManager.serverDnsNames }} + - {{ . | quote }} + {{- end }} + {{- else }} + # The chart fullname is always the first entry of defaultServerDnsNames + # below, so it's always valid as a commonName. + # + # Upgrade note: this replaces the previously-hardcoded "openshell-server" + # commonName. The ACME commonName-must-be-a-SAN constraint this exists for + # only applies to the external-issuer branch above, but this branch was + # changed too for consistency. Existing self-signed/internal-CA deployments + # will get a new server cert with a different Subject CN on next reissue + # after upgrading — SAN-based hostname verification (what TLS clients + # actually check) is unaffected. + commonName: {{ include "openshell.fullname" . }} dnsNames: {{- range (include "openshell.defaultServerDnsNames" . | fromYamlArray) }} - {{ . | quote }} @@ -65,6 +103,7 @@ spec: ipAddresses: {{- toYaml .Values.certManager.serverIpAddresses | nindent 4 }} {{- end }} + {{- end }} privateKey: algorithm: ECDSA size: 256 @@ -73,9 +112,15 @@ spec: - digital signature - key encipherment issuerRef: + {{- if .Values.certManager.serverIssuerRef.name }} + name: {{ .Values.certManager.serverIssuerRef.name }} + kind: {{ .Values.certManager.serverIssuerRef.kind | default "Issuer" }} + group: {{ .Values.certManager.serverIssuerRef.group | default "cert-manager.io" }} + {{- else }} name: {{ include "openshell.fullname" . }}-ca-issuer kind: Issuer group: cert-manager.io + {{- end }} --- apiVersion: cert-manager.io/v1 kind: Certificate diff --git a/deploy/helm/openshell/templates/route.yaml b/deploy/helm/openshell/templates/route.yaml new file mode 100644 index 0000000000..7bef6e58c2 --- /dev/null +++ b/deploy/helm/openshell/templates/route.yaml @@ -0,0 +1,31 @@ +# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +{{- if .Values.openshiftRoute.enabled }} +{{- if .Values.server.disableTls }} +{{- fail "openshiftRoute.enabled=true requires TLS (server.disableTls must be false) \u2014 a passthrough Route forwards encrypted traffic by SNI, so the gateway must terminate its own TLS." }} +{{- end }} +apiVersion: route.openshift.io/v1 +kind: Route +metadata: + name: {{ include "openshell.fullname" . }} + namespace: {{ .Release.Namespace }} + labels: + {{- include "openshell.labels" . | nindent 4 }} + {{- with .Values.openshiftRoute.annotations }} + annotations: + {{- toYaml . | nindent 4 }} + {{- end }} +spec: + {{- if .Values.openshiftRoute.host }} + host: {{ .Values.openshiftRoute.host }} + {{- end }} + to: + kind: Service + name: {{ include "openshell.fullname" . }} + port: + targetPort: grpc + tls: + termination: passthrough + wildcardPolicy: None +{{- end }} diff --git a/deploy/helm/openshell/tests/cert_manager_pki_test.yaml b/deploy/helm/openshell/tests/cert_manager_pki_test.yaml new file mode 100644 index 0000000000..8abe8ef1b9 --- /dev/null +++ b/deploy/helm/openshell/tests/cert_manager_pki_test.yaml @@ -0,0 +1,169 @@ +# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +suite: cert-manager PKI issuerRef overrides +templates: + - templates/cert-manager-pki.yaml +release: + name: openshell + namespace: my-namespace + +tests: + - it: defaults both server and client Certificates to the chart's own CA issuer + template: templates/cert-manager-pki.yaml + set: + certManager.enabled: true + asserts: + - equal: + path: spec.issuerRef.name + value: openshell-ca-issuer + documentIndex: 3 + - equal: + path: spec.issuerRef.kind + value: Issuer + documentIndex: 3 + - equal: + path: spec.issuerRef.name + value: openshell-ca-issuer + documentIndex: 4 + - equal: + path: spec.issuerRef.kind + value: Issuer + documentIndex: 4 + + - it: default server Certificate includes internal SANs, IPs, and a fixed commonName + template: templates/cert-manager-pki.yaml + set: + certManager.enabled: true + asserts: + - equal: + path: spec.commonName + value: openshell + documentIndex: 3 + - contains: + path: spec.dnsNames + content: openshell.my-namespace.svc.cluster.local + documentIndex: 3 + - contains: + path: spec.dnsNames + content: localhost + documentIndex: 3 + - equal: + path: spec.ipAddresses[0] + value: 127.0.0.1 + documentIndex: 3 + + - it: overrides the server Certificate issuerRef when serverIssuerRef is set + template: templates/cert-manager-pki.yaml + set: + certManager.enabled: true + certManager.serverIssuerRef.name: letsencrypt-prod + certManager.serverIssuerRef.kind: ClusterIssuer + certManager.clientCaFromServerTlsSecret: false + server.tls.clientCaSecretName: openshell-ca-tls + certManager.serverDnsNames: + - openshell.example.com + asserts: + - equal: + path: spec.issuerRef.name + value: letsencrypt-prod + documentIndex: 3 + - equal: + path: spec.issuerRef.kind + value: ClusterIssuer + documentIndex: 3 + - equal: + path: spec.issuerRef.group + value: cert-manager.io + documentIndex: 3 + + - it: scopes server Certificate SANs to only the configured external hostnames when serverIssuerRef is set + template: templates/cert-manager-pki.yaml + set: + certManager.enabled: true + certManager.serverIssuerRef.name: letsencrypt-prod + certManager.serverIssuerRef.kind: ClusterIssuer + certManager.clientCaFromServerTlsSecret: false + server.tls.clientCaSecretName: openshell-ca-tls + certManager.serverDnsNames: + - openshell.example.com + asserts: + - equal: + path: spec.commonName + value: openshell.example.com + documentIndex: 3 + - equal: + path: spec.dnsNames + value: + - openshell.example.com + documentIndex: 3 + - notExists: + path: spec.ipAddresses + documentIndex: 3 + - notContains: + path: spec.dnsNames + content: localhost + documentIndex: 3 + - notContains: + path: spec.dnsNames + content: openshell.my-namespace.svc.cluster.local + documentIndex: 3 + + - it: fails when serverIssuerRef is set but clientCaFromServerTlsSecret is true + template: templates/cert-manager-pki.yaml + set: + certManager.enabled: true + certManager.serverIssuerRef.name: letsencrypt-prod + certManager.serverIssuerRef.kind: ClusterIssuer + certManager.clientCaFromServerTlsSecret: true + certManager.serverDnsNames: + - openshell.example.com + asserts: + - failedTemplate: + errorMessage: "certManager.serverIssuerRef.name is set but certManager.clientCaFromServerTlsSecret is still true \u2014 the server cert Secret's ca.crt comes from the external issuer, not the CA that signs the client (mTLS) certificate. Set certManager.clientCaFromServerTlsSecret=false and set server.tls.clientCaSecretName to the secret containing the client CA (e.g. the value of certManager.caSecretName, which defaults to openshell-ca-tls)." + + - it: fails when serverIssuerRef is set but serverDnsNames contains internal-only names + template: templates/cert-manager-pki.yaml + set: + certManager.enabled: true + certManager.serverIssuerRef.name: letsencrypt-prod + certManager.clientCaFromServerTlsSecret: false + server.tls.clientCaSecretName: openshell-ca-tls + # serverDnsNames is left at the default which contains "openshell.openshell.svc", "localhost", etc. + asserts: + - failedTemplate: + errorMessage: "certManager.serverIssuerRef.name is set (external issuer) but certManager.serverDnsNames contains \"openshell.openshell.svc\" \u2014 external CAs (e.g. ACME / Let's Encrypt) reject internal-only names per CA/Browser Forum baseline requirements. Override certManager.serverDnsNames with your externally-resolvable hostname(s)." + + - it: fails when clientCaFromServerTlsSecret is false but clientCaSecretName is the default + template: templates/cert-manager-pki.yaml + set: + certManager.enabled: true + certManager.serverIssuerRef.name: letsencrypt-prod + certManager.serverIssuerRef.kind: ClusterIssuer + certManager.clientCaFromServerTlsSecret: false + certManager.serverDnsNames: + - openshell.example.com + # server.tls.clientCaSecretName left at the default openshell-server-client-ca + asserts: + - failedTemplate: + errorMessage: "certManager.clientCaFromServerTlsSecret is false but server.tls.clientCaSecretName is still the default (openshell-server-client-ca) which is not created by the chart when cert-manager is enabled. Set server.tls.clientCaSecretName to the Secret containing the CA that signs the client (mTLS) certificate (e.g. the value of certManager.caSecretName, which defaults to openshell-ca-tls), or set it to \"\" to disable client-certificate verification entirely." + + - it: client Certificate issuerRef is unaffected by serverIssuerRef + template: templates/cert-manager-pki.yaml + set: + certManager.enabled: true + certManager.serverIssuerRef.name: letsencrypt-prod + certManager.serverIssuerRef.kind: ClusterIssuer + certManager.clientCaFromServerTlsSecret: false + server.tls.clientCaSecretName: openshell-ca-tls + certManager.serverDnsNames: + - openshell.example.com + asserts: + - equal: + path: spec.issuerRef.name + value: openshell-ca-issuer + documentIndex: 4 + - equal: + path: spec.issuerRef.kind + value: Issuer + documentIndex: 4 diff --git a/deploy/helm/openshell/tests/route_test.yaml b/deploy/helm/openshell/tests/route_test.yaml new file mode 100644 index 0000000000..e12b7ce2b7 --- /dev/null +++ b/deploy/helm/openshell/tests/route_test.yaml @@ -0,0 +1,70 @@ +# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +suite: OpenShift Route +templates: + - templates/route.yaml +release: + name: openshell + namespace: my-namespace + +tests: + - it: renders nothing by default + asserts: + - hasDocuments: + count: 0 + + - it: renders a passthrough Route when enabled + set: + openshiftRoute.enabled: true + openshiftRoute.host: openshell.apps.example.com + asserts: + - isKind: + of: Route + - equal: + path: apiVersion + value: route.openshift.io/v1 + - equal: + path: spec.host + value: openshell.apps.example.com + - equal: + path: spec.to.kind + value: Service + - equal: + path: spec.to.name + value: openshell + - equal: + path: spec.port.targetPort + value: grpc + - equal: + path: spec.tls.termination + value: passthrough + - equal: + path: spec.wildcardPolicy + value: None + + - it: omits host when not set + set: + openshiftRoute.enabled: true + asserts: + - notExists: + path: spec.host + + - it: fails when passthrough Route is enabled with TLS disabled + set: + openshiftRoute.enabled: true + server.disableTls: true + asserts: + - failedTemplate: + errorMessage: "openshiftRoute.enabled=true requires TLS (server.disableTls must be false) \u2014 a passthrough Route forwards encrypted traffic by SNI, so the gateway must terminate its own TLS." + + - it: renders custom annotations + set: + openshiftRoute.enabled: true + openshiftRoute.host: openshell.apps.example.com + openshiftRoute.annotations: + haproxy.router.openshift.io/balance: roundrobin + asserts: + - equal: + path: metadata.annotations["haproxy.router.openshift.io/balance"] + value: roundrobin diff --git a/deploy/helm/openshell/tests/statefulset_client_ca_test.yaml b/deploy/helm/openshell/tests/statefulset_client_ca_test.yaml index a7b02310cf..1d744b35aa 100644 --- a/deploy/helm/openshell/tests/statefulset_client_ca_test.yaml +++ b/deploy/helm/openshell/tests/statefulset_client_ca_test.yaml @@ -53,13 +53,14 @@ tests: certManager.enabled: true certManager.clientCaFromServerTlsSecret: false pkiInitJob.enabled: true + server.tls.clientCaSecretName: openshell-ca-tls asserts: - equal: path: spec.template.spec.volumes[3].name value: tls-client-ca - equal: path: spec.template.spec.volumes[3].secret.secretName - value: openshell-server-client-ca + value: openshell-ca-tls - notExists: path: spec.template.spec.volumes[3].secret.items diff --git a/deploy/helm/openshell/values.yaml b/deploy/helm/openshell/values.yaml index 0525ed475d..c12d093d0b 100644 --- a/deploy/helm/openshell/values.yaml +++ b/deploy/helm/openshell/values.yaml @@ -345,7 +345,7 @@ pkiInitJob: serverIpAddresses: [] # cert-manager Certificate/Issuer resources (requires cert-manager CRDs in-cluster). -# Uses namespaced Issuers only (no ClusterIssuer). Does not install cert-manager itself. +# Does not install cert-manager itself. certManager: # -- Create cert-manager Issuer and Certificate resources. When enabled, # cert-manager owns TLS and the chart runs a JWT-only certgen hook to create @@ -353,9 +353,20 @@ certManager: enabled: false # -- Secret created for the intermediate CA (Certificate with isCA: true). caSecretName: openshell-ca-tls + # -- Override the issuerRef for the server Certificate (e.g. a real ACME + # ClusterIssuer for a publicly-trusted cert on an external hostname). Leave + # name empty to use the chart's own self-signed CA issuer (default). + serverIssuerRef: + name: "" + kind: "" + group: "" # -- Mount gateway client CA from the server TLS secret's ca.crt (populated by - # cert-manager for certs issued by a CA Issuer). Avoids a separate - # openshell-server-client-ca Secret. + # cert-manager for certs issued by a CA Issuer). Set to false when + # serverIssuerRef points at an external issuer (its secret's ca.crt would be + # that issuer's chain, not the CA that signs the client cert). When false, + # also set server.tls.clientCaSecretName to a secret containing the actual + # client CA — for example caSecretName's value, since that's the CA the + # client certificate above is issued from by default. clientCaFromServerTlsSecret: true # -- Duration for cert-manager-issued certificates. certificateDuration: 8760h @@ -415,3 +426,15 @@ grpcRoute: # or the existing openshell-server-tls Secret (its SANs must include the # external hostname). certificateRefs: [] + +# OpenShift Route with TLS passthrough. The gateway terminates its own +# TLS/mTLS; the router only forwards based on SNI, so it never sees plaintext +# or the client certificate. Requires server.disableTls=false and a server +# cert whose SANs include the Route host (see certManager.serverIssuerRef). +openshiftRoute: + # -- Create an OpenShift Route with TLS passthrough. + enabled: false + # -- Hostname for the Route. Must match a SAN on the gateway's server cert. + host: "" + # -- Extra annotations on the Route (e.g. haproxy.router.openshift.io/*). + annotations: {} diff --git a/docs/kubernetes/managing-certificates.mdx b/docs/kubernetes/managing-certificates.mdx index b66419b505..3aadd0b002 100644 --- a/docs/kubernetes/managing-certificates.mdx +++ b/docs/kubernetes/managing-certificates.mdx @@ -60,6 +60,53 @@ The chart also runs a pre-install hook in JWT-only mode to create the gateway's sandbox JWT signing Secret. That Secret is separate from the cert-manager TLS certificate Secrets and is mounted at `/etc/openshell-jwt`. +## Using a real Issuer for the server certificate + +By default, cert-manager issues both the server and client certificates from +a self-signed CA the chart creates — this rotates automatically, but the +server certificate is still not publicly trusted. `certManager.serverIssuerRef` +overrides the `issuerRef` on the server `Certificate` resource to point at a +real `Issuer` or `ClusterIssuer` instead, for example an ACME issuer: + +```shell +helm upgrade --install openshell \ + oci://ghcr.io/nvidia/openshell/helm-chart \ + --version \ + --namespace openshell \ + --set certManager.enabled=true \ + --set certManager.clientCaFromServerTlsSecret=false \ + --set server.tls.clientCaSecretName=openshell-ca-tls \ + --set certManager.serverIssuerRef.name=letsencrypt-prod \ + --set certManager.serverIssuerRef.kind=ClusterIssuer \ + --set certManager.serverDnsNames[0]=openshell.example.com +``` + + +Public CAs such as Let's Encrypt reject certificate requests that include +internal-only names (`*.svc.cluster.local`, `localhost`, loopback IPs) per +CA/Browser Forum baseline requirements. The chart validates this at install +time and fails with an actionable error if `certManager.serverDnsNames` +contains internal-only entries while `serverIssuerRef` is set. When +`serverIssuerRef` is set, the server certificate's SANs are limited to the +hostnames in `certManager.serverDnsNames` — the chart's usual internal +cluster-local SANs are omitted. Sandboxes calling back into the gateway must then use the same +external hostname; see `server.grpcEndpoint` in +[Gateway Configuration](/reference/gateway-config). + + +Set `certManager.clientCaFromServerTlsSecret=false` whenever `serverIssuerRef` +is set, and set `server.tls.clientCaSecretName` to name a secret that actually +contains the client CA — by default that's the same secret named by +`certManager.caSecretName` (`openshell-ca-tls`), since that's the CA the chart +issues the client certificate from. The client (mTLS) certificate used by sandbox supervisors +stays on the chart's own internal CA — it doesn't need public trust, and +getting a public CA to sign it isn't practical: ACME only validates domain +identifiers, not arbitrary workload identity, and issuing one certificate per +sandbox would put a public CA's issuance rate limits directly in the +sandbox-creation path. + ## Next Steps -Return to [Setup](/kubernetes/setup) to complete the installation. +Return to [Setup](/kubernetes/setup) to complete the installation. For +exposing the gateway externally on OpenShift with a real certificate, see +[OpenShift](/kubernetes/openshift). diff --git a/docs/kubernetes/openshift.mdx b/docs/kubernetes/openshift.mdx index 7512eaa65e..23022e13cd 100644 --- a/docs/kubernetes/openshift.mdx +++ b/docs/kubernetes/openshift.mdx @@ -87,8 +87,61 @@ openshell gateway add http://127.0.0.1:8080 --local --name openshift openshell status ``` +## Production: expose externally with a real certificate + +The steps above run the gateway over plaintext HTTP for quick evaluation. For +a real deployment, cert-manager can issue the gateway's server certificate +from a real Issuer or ClusterIssuer (for example, an ACME issuer), and an +OpenShift Route with TLS passthrough exposes it externally while the gateway +keeps terminating its own TLS and mTLS. + +Install cert-manager and configure a working `ClusterIssuer` first — see +[Managing Certificates](/kubernetes/managing-certificates) for the +`certManager.serverIssuerRef` details. Configure an OIDC provider as described +in [Access Control](/kubernetes/access-control) — remote gateways authenticate +CLI users via OIDC, not mTLS, so the gateway must know the OIDC issuer URL. +Install the chart with: + +```shell +helm install openshell oci://ghcr.io/nvidia/openshell/helm-chart \ + --version \ + --namespace openshell \ + --set podSecurityContext.fsGroup=null \ + --set securityContext.runAsUser=null \ + --set server.disableTls=false \ + --set server.grpcEndpoint=https://:443 \ + --set certManager.enabled=true \ + --set certManager.clientCaFromServerTlsSecret=false \ + --set server.tls.clientCaSecretName=openshell-ca-tls \ + --set certManager.serverIssuerRef.name= \ + --set certManager.serverIssuerRef.kind=ClusterIssuer \ + --set certManager.serverDnsNames[0]= \ + --set openshiftRoute.enabled=true \ + --set openshiftRoute.host= \ + --set server.oidc.issuer= \ + --set server.oidc.audience= +``` + +| Override | Reason | +|---|---| +| `server.grpcEndpoint` | Sandboxes call back into the gateway using this hostname. A cert from an external issuer only carries the external SANs you configure, not the internal cluster-local ones, so sandboxes must use the same externally-valid hostname to pass TLS verification. | +| `certManager.clientCaFromServerTlsSecret=false` + `server.tls.clientCaSecretName` | The server cert's issuer no longer shares a CA with the client (mTLS) cert, so this points the gateway's client-verification CA at `certManager.caSecretName` (the chart's own CA secret, `openshell-ca-tls` by default) instead of the server secret. | +| `certManager.serverIssuerRef` | Points the server certificate at your real Issuer or ClusterIssuer instead of the chart's built-in self-signed CA. | +| `openshiftRoute.enabled` / `openshiftRoute.host` | Creates an OpenShift Route with TLS passthrough — the router forwards the encrypted connection by SNI without decrypting, so the gateway keeps terminating its own TLS and mTLS. | +| `server.oidc.issuer` / `server.oidc.audience` | Configures server-side OIDC validation. Without these, the gateway expects mTLS client certificates and rejects OIDC-only CLI connections. See [Access Control](/kubernetes/access-control). | + +Register the gateway with the CLI over OIDC. Remote gateways authenticate CLI +users via OIDC, not mTLS — see [Access Control](/kubernetes/access-control): + +```shell +openshell gateway add https:// \ + --name openshift \ + --oidc-issuer +openshell gateway login openshift +``` + ## Next Steps -- For TLS-enabled deployments, refer to [Managing Certificates](/kubernetes/managing-certificates). -- To expose the gateway externally, refer to [Ingress](/kubernetes/ingress). +- For more on certificate provisioning modes, refer to [Managing Certificates](/kubernetes/managing-certificates). +- To expose the gateway externally through the Kubernetes Gateway API instead of a Route, refer to [Ingress](/kubernetes/ingress). - To configure OIDC authentication, refer to [Access Control](/kubernetes/access-control). diff --git a/target b/target new file mode 120000 index 0000000000..e58e2c52c0 --- /dev/null +++ b/target @@ -0,0 +1 @@ +/mnt/build-artifacts/cargo-target \ No newline at end of file