diff --git a/content/en/docs/next/operations/oidc/enable_oidc.md b/content/en/docs/next/operations/oidc/enable_oidc.md index 26859428..ceae15f3 100644 --- a/content/en/docs/next/operations/oidc/enable_oidc.md +++ b/content/en/docs/next/operations/oidc/enable_oidc.md @@ -82,7 +82,7 @@ kubectl patch packages.cozystack.io cozystack.cozystack-platform --type=merge -p ``` {{% alert color="info" %}} -**Optional**: If you want the dashboard to reach Keycloak via the internal cluster network instead of the external ingress, set `keycloakInternalUrl`. This is useful in environments with self-signed certificates or restricted external access. See [Self-Signed Certificates]({{% ref "/docs/next/operations/oidc/self-signed-certificates" %}}) for details. +Dashboard and LINSTOR backend OIDC requests use the internal `keycloak-http` Service by default, while browser-facing redirects keep the external issuer URL. Override `keycloakInternalUrl` only when Keycloak is deployed at a different in-cluster address, or set it to an empty string to restore external OIDC discovery. See [Private CA and Let's Encrypt Staging]({{% ref "/docs/next/operations/oidc/self-signed-certificates" %}}) for private and development certificate trust. {{% /alert %}} Within one minute, CozyStack will reconcile and create three new `HelmRelease` resources: diff --git a/content/en/docs/next/operations/oidc/self-signed-certificates.md b/content/en/docs/next/operations/oidc/self-signed-certificates.md index 032652e6..cf5827a7 100644 --- a/content/en/docs/next/operations/oidc/self-signed-certificates.md +++ b/content/en/docs/next/operations/oidc/self-signed-certificates.md @@ -1,187 +1,301 @@ --- -title: "Self-Signed Certificates" -linkTitle: "Self-Signed Certificates" -description: "How to configure OIDC with self-signed certificates" +title: "Private CA and Let's Encrypt Staging" +linkTitle: "Private CA and Staging" +description: "Configure OIDC trust for private, air-gapped, and development environments" weight: 60 aliases: - /docs/oidc/self-signed-certificates - /docs/operations/oidc/self-signed-certificates --- -This guide explains how to configure Kubernetes API server for OIDC authentication with Keycloak when using self-signed certificates. By default, Cozystack issues certificates via LetsEncrypt, but some environments (e.g., air-gapped or private enterprise networks) may use a custom CA instead. +This guide explains how to serve Keycloak with a certificate that is not publicly trusted and how to establish trust in every OIDC client. It covers private certificate authorities (CAs), Let's Encrypt staging, wildcard certificates, Talos, the Kubernetes API server, browsers, and kubelogin. -## Prerequisites +## Choose a certificate source -- Cozystack cluster with OIDC enabled (see [Enable OIDC Server]({{% ref "/docs/next/operations/oidc/enable_oidc" %}})) -- Talos Linux control plane nodes -- `talosctl` configured for your cluster -- `kubelogin` installed +| Source | Recommended use | Publicly trusted | Notes | +| --- | --- | --- | --- | +| Let's Encrypt production | Production | Yes | Subject to production rate limits. Prefer a DNS-01 wildcard when many endpoints share a domain. | +| Let's Encrypt staging | Ephemeral and development clusters | No | Uses separate accounts and much higher limits than production. Trust staging roots only in isolated development clients. | +| Private CA | Air-gapped and corporate environments | No | Distribute the CA certificate to every trust boundary. Keep the CA stable and rotate leaf certificates beneath it. | +| Direct self-signed leaf | Temporary diagnostics only | No | Every leaf replacement changes the trust anchor. Use a self-signed issuer to bootstrap a CA instead. | -## Step 1: Retrieve the Keycloak Certificate +Let's Encrypt explicitly warns against adding its staging roots to a trust store used for ordinary browsing because the staging hierarchy is not audited to production standards. A dedicated browser profile, disposable VM, or isolated development workstation avoids extending that trust to unrelated browsing. See the [Let's Encrypt staging environment](https://letsencrypt.org/docs/staging-environment/) for the current staging root certificates and limits. -Get the certificate from the ingress controller: +{{% alert color="warning" %}} +Trust a CA certificate, not the certificate currently served by Keycloak. Pinning a rotating leaf certificate causes OIDC to fail on the next renewal. +{{% /alert %}} + +## Reduce certificate orders with a wildcard + +Cozystack can serve one certificate for the root domain and its single-label subdomains. The certificate must contain both the apex name and the wildcard, for example `example.org` and `*.example.org`. + +### Let Cozystack issue a staging wildcard + +For the ingress-nginx path, enable DNS-01 and shared wildcard issuance in the Platform package values: + +```yaml +publishing: + certificates: + issuerName: letsencrypt-stage + solver: dns01 + wildcard: true +``` + +Configure the selected DNS-01 provider as described in [Gateway API and DNS-01]({{% ref "/docs/next/networking/gateway-api" %}}). HTTP-01 cannot issue wildcard certificates. When Gateway API is enabled, each tenant Gateway manages its own certificate and the `wildcard` value above is ignored. + +The staging certificate remains untrusted until the active Let's Encrypt staging root certificates are installed at each trust boundary described below. Staging and production ACME accounts and rate limits are separate. + +### Supply an existing wildcard certificate + +For a corporate CA or an externally issued certificate, create a `kubernetes.io/tls` Secret in the publishing namespace, which is `tenant-root` by default: ```bash -echo | openssl s_client -connect :443 \ - -servername keycloak.example.org 2>/dev/null | openssl x509 +kubectl create secret tls cozystack-wildcard-tls \ + --namespace=tenant-root \ + --cert=wildcard-fullchain.pem \ + --key=wildcard.key ``` -Replace `` with your ingress controller IP address, and `keycloak.example.org` with your actual Keycloak domain. +Then set the Secret name in the Platform package values: -Save the output (the certificate between `-----BEGIN CERTIFICATE-----` and `-----END CERTIFICATE-----`) for the next step. +```yaml +publishing: + certificates: + wildcardSecretName: cozystack-wildcard-tls +``` + +The root ingress controller uses this Secret as its default TLS certificate. System Ingress resources, including Keycloak, omit their per-host certificate and use that default. Cozystack replicates the Secret for inherited tenant ingress controllers. A wildcard such as `*.example.org` does not cover `service.tenant.example.org`; include every required tenant apex in the certificate SANs or use per-tenant certificates. + +{{% alert color="warning" %}} +Do not upload a corporate root private key to the cluster. Ask the corporate PKI to issue the wildcard certificate, or use a dedicated intermediate CA whose scope and lifetime are limited to the cluster. +{{% /alert %}} -## Step 2: Configure Talos Control Plane Nodes +### Bootstrap a development CA with cert-manager -For each control plane node, add the following to your machine configuration: +For a disposable development environment without an existing PKI, the built-in `selfsigned-cluster-issuer` can bootstrap a CA certificate. Do not use it directly for Keycloak leaf certificates. ```yaml -machine: - network: - extraHostEntries: - - ip: - aliases: - - keycloak.example.org - files: - - content: | - -----BEGIN CERTIFICATE----- - - -----END CERTIFICATE----- - permissions: 0o644 - path: /var/oidc-ca.crt - op: create +apiVersion: cert-manager.io/v1 +kind: Certificate +metadata: + name: development-root-ca + namespace: cozy-cert-manager +spec: + isCA: true + commonName: Cozystack development root CA + subject: + organizations: + - Cozystack development + secretName: development-root-ca + duration: 87600h + privateKey: + algorithm: ECDSA + size: 256 + issuerRef: + name: selfsigned-cluster-issuer + kind: ClusterIssuer +--- +apiVersion: cert-manager.io/v1 +kind: ClusterIssuer +metadata: + name: development-ca +spec: + ca: + secretName: development-root-ca +--- +apiVersion: cert-manager.io/v1 +kind: Certificate +metadata: + name: cozystack-wildcard-tls + namespace: tenant-root +spec: + secretName: cozystack-wildcard-tls + dnsNames: + - example.org + - "*.example.org" + issuerRef: + name: development-ca + kind: ClusterIssuer +``` + +The CA Secret is in `cozy-cert-manager` because cert-manager reads `ClusterIssuer` credentials from its cluster resource namespace. After the wildcard Secret becomes ready, set `publishing.certificates.wildcardSecretName` as shown above. Back up the CA Secret if the development environment must survive cluster recreation; losing it requires distributing a new trust anchor. + +## Understand the trust boundaries + +The same CA bundle may need to be installed in several independent places: + +- Talos trusts it for host-level HTTPS clients. +- The Kubernetes API server uses it to verify the OIDC issuer during discovery and JWKS retrieval. +- kubelogin uses it to contact the OIDC issuer from an administrator or tenant workstation. +- Browsers use it to open the Keycloak login page. + +The CA embedded in a kubeconfig under `clusters[].cluster.certificate-authority-data` verifies the Kubernetes API server. It does not verify Keycloak. Cozystack-generated tenant kubeconfigs currently include the API server CA but do not include an OIDC issuer CA. + +## Configure Talos and the Kubernetes API server +Create a PEM bundle containing the private root CA, the active Let's Encrypt staging roots, or both old and new roots during a CA rotation. Do not put a leaf certificate in this bundle. + +Set the issuer and CA path in the Talm project `values.yaml`: + +```yaml +oidcIssuerUrl: "https://keycloak.example.org/realms/cozy" + +extraApiServerArgs: + oidc-ca-file: /etc/ssl/certs/ca-certificates.crt +``` + +Create a side-patch such as `oidc-private-ca.yaml`: + +```yaml cluster: apiServer: - extraArgs: - oidc-issuer-url: https://keycloak.example.org/realms/cozy - oidc-client-id: kubernetes - oidc-username-claim: preferred_username - oidc-groups-claim: groups - oidc-ca-file: /etc/kubernetes/oidc/ca.crt extraVolumes: - - hostPath: /var/oidc-ca.crt - mountPath: /etc/kubernetes/oidc/ca.crt + - hostPath: /etc/ssl/certs/ca-certificates.crt + mountPath: /etc/ssl/certs/ca-certificates.crt + readonly: true +--- +apiVersion: v1alpha1 +kind: TrustedRootsConfig +name: oidc-ca +certificates: |- + -----BEGIN CERTIFICATE----- + + -----END CERTIFICATE----- ``` -Apply the configuration to each control plane node: +`TrustedRootsConfig` appends the CA to the Talos host bundle. The explicit `extraVolumes` entry is still required because the kube-apiserver container does not inherit that host file automatically. The `readonly` key is lowercase in the Talos schema. -```bash -talosctl apply-config -n -f nodes/.yaml -``` - -{{% alert color="info" %}} -The `extraHostEntries` configuration ensures that the Keycloak domain resolves correctly within the cluster, which is essential when using internal ingress IPs. -{{% /alert %}} +If control-plane nodes cannot resolve the external Keycloak hostname, add a `StaticHostConfig` document to the same side-patch: -## Optional: Configure Internal Keycloak URL for Dashboard +```yaml +--- +apiVersion: v1alpha1 +kind: StaticHostConfig +name: 192.0.2.10 +hostnames: + - keycloak.example.org +``` -By default, the Cozystack Dashboard's oauth2-proxy connects to Keycloak through the external ingress URL. In environments with self-signed certificates or restricted external access, you can configure the dashboard to use Keycloak's internal cluster service for backend requests (token exchange, JWKS validation, userinfo, logout) while keeping browser redirects on the external URL. +`name` is the destination IP address and `hostnames` is a top-level list. `StaticHostConfig` replaces the deprecated `machine.network.extraHostEntries` configuration. Omit it when normal DNS already resolves the issuer correctly. -Patch the Platform Package: +Preview and apply the composed configuration to each control-plane node. A non-empty Talm side-patch requires a single-node anchor: ```bash -kubectl patch packages.cozystack.io cozystack.cozystack-platform --type=merge -p '{ - "spec": { - "components": { - "platform": { - "values": { - "authentication": { - "oidc": { - "keycloakInternalUrl": "http://keycloak-http.cozy-keycloak.svc:8080/realms/cozy" - } - } - } - } - } - } -}' +talm apply --dry-run -f nodes/controlplane-1.yaml -f oidc-private-ca.yaml +talm apply -f nodes/controlplane-1.yaml -f oidc-private-ca.yaml ``` +Repeat the operation for every control-plane node. Keep an administrative kubeconfig available while changing OIDC configuration. + {{% alert color="info" %}} -This only affects the dashboard's oauth2-proxy (pod-to-pod communication). The Kubernetes API server still requires `extraHostEntries` to reach Keycloak, since `kube-apiserver` uses host-level DNS and cannot resolve cluster service names. +An OIDC issuer hosted in the same cluster does not create a kube-apiserver startup cycle. Kubernetes initializes OIDC discovery asynchronously and retries while Keycloak is unavailable. The API server starts normally, but OIDC token authentication fails until discovery succeeds. {{% /alert %}} -## Step 3: Configure kubelogin +## Configure Cozystack OIDC clients -Install kubelogin if you haven't already: +Enable OIDC as described in [Enable OIDC Server]({{% ref "/docs/next/operations/oidc/enable_oidc" %}}). Keep the default internal Keycloak URL and TLS verification enabled: -```bash -# Homebrew (macOS and Linux) -brew install int128/kubelogin/kubelogin +```yaml +authentication: + oidc: + enabled: true + insecureSkipVerify: false + keycloakInternalUrl: "http://keycloak-http.cozy-keycloak.svc:8080/realms/cozy" +``` -# Krew (macOS, Linux, Windows and ARM) -kubectl krew install oidc-login +Dashboard and LINSTOR oauth2-proxy instances retain the external issuer and browser login URLs, but send token, JWKS, userinfo, and logout requests to the internal HTTP Service. They therefore do not need the private ingress CA. `authentication.oidc.insecureSkipVerify` controls those oauth2-proxy instances only; it does not configure kube-apiserver, kubelogin, or browser trust. -# Chocolatey (Windows) -choco install kubelogin -``` +### Keycloak truststore -Save the CA certificate from Step 1 to a file on your local machine: +Keycloak does not need to trust the CA that issued its own ingress certificate. TLS terminates at the ingress controller and the Keycloak pod receives HTTP on the `keycloak-http` Service. -```bash -# Save the certificate to a file (e.g., ~/.kube/oidc-ca.pem) -cat > ~/.kube/oidc-ca.pem < ------END CERTIFICATE----- -EOF -``` +Keycloak needs a private CA in its truststore only when it initiates TLS to a private endpoint, such as an external identity provider, LDAPS server, or SMTP server. Upstream Keycloak accepts PEM or PKCS12 files through `KC_TRUSTSTORE_PATHS`, but the current Cozystack Keycloak chart does not expose an additional volume mount for such a file. `extraEnv` alone is not sufficient because the CA must also be mounted. Add first-class chart support before configuring this case; do not patch the generated StatefulSet manually because Flux will reconcile it back. -Set up OIDC login (this will open a browser for authentication): +## Configure browsers and kubelogin + +Install the private root CA or active Let's Encrypt staging roots in a dedicated browser profile or isolated workstation trust store. Verify that the certificate SAN contains the exact external Keycloak hostname. + +Save the CA bundle on each kubelogin client and test issuer discovery: ```bash kubectl oidc-login setup \ --oidc-issuer-url=https://keycloak.example.org/realms/cozy \ --oidc-client-id=kubernetes \ - --certificate-authority=~/.kube/oidc-ca.pem + --certificate-authority=/path/to/oidc-ca.pem ``` -Configure kubectl credentials: +Add the same option to the generated kubeconfig under the kubelogin exec arguments: -```bash -kubectl config set-credentials oidc \ - --exec-api-version=client.authentication.k8s.io/v1 \ - --exec-interactive-mode=IfAvailable \ - --exec-command=kubectl \ - --exec-arg=oidc-login \ - --exec-arg=get-token \ - --exec-arg="--oidc-issuer-url=https://keycloak.example.org/realms/cozy" \ - --exec-arg="--oidc-client-id=kubernetes" \ - --exec-arg="--certificate-authority=~/.kube/oidc-ca.pem" -``` - -Switch to the OIDC user and verify: - -```bash -kubectl config set-context --current --user=oidc -kubectl get nodes +```yaml +users: + - name: oidc + user: + exec: + apiVersion: client.authentication.k8s.io/v1 + command: kubectl + args: + - oidc-login + - get-token + - --oidc-issuer-url=https://keycloak.example.org/realms/cozy + - --oidc-client-id=kubernetes + - --certificate-authority=/path/to/oidc-ca.pem ``` -{{% alert color="info" %}} -If your organization's CA is already installed in the system trust store (common in enterprise environments), you can omit the `--certificate-authority` flag entirely — kubelogin will use the system CA bundle automatically. -{{% /alert %}} +kubelogin also supports `--certificate-authority-data` for embedding a base64-encoded public CA bundle. If the CA is already present in the workstation system trust store, omit both options. {{% alert color="warning" %}} -Avoid using `--insecure-skip-tls-verify`. If you cannot install the CA certificate on your machine or pass it via `--certificate-authority`, you can use `--insecure-skip-tls-verify` as a temporary workaround, but this disables TLS verification and is not recommended for production use. +The global kubectl `--certificate-authority` option verifies the Kubernetes API server, while `kubectl oidc-login ... --certificate-authority` verifies Keycloak. Setting one does not configure the other. {{% /alert %}} -## Troubleshooting +## Verify the complete path + +Verify the served hostname and chain against the intended CA bundle: + +```bash +openssl s_client \ + -connect keycloak.example.org:443 \ + -servername keycloak.example.org \ + -CAfile /path/to/oidc-ca.pem \ + -verify_return_error