Feat: AIAC Event Broker + Keycloak SPI listener (phase 2, PR 2)

aiac/docs/specs/components/event-broker.md

@@ -69,7 +69,7 @@ No authentication credentials are required. The NATS server runs with no-auth co
 
 ## Runtime
 
-- Image: `nats:latest` with JetStream enabled (`-js` flag)
+- Image: `nats:2.14-alpine` with JetStream enabled (`-js` flag)
 - Bind: `0.0.0.0:4222` (NATS client port)
 - Kubernetes ClusterIP service: `aiac-event-broker-service:4222`
 - Base image: official `nats` Docker image

aiac/k8s/agent-deployment.yaml

@@ -90,9 +90,34 @@ spec:
         fsGroup: 10001
         seccompProfile:
           type: RuntimeDefault
-      # aiac-init init container is deferred to Phase 2 (issue 4.21).
-      # It will gate startup on NATS + IdP Config + PDP Policy + RAG Ingest health
-      # and provision the aiac-events JetStream stream.
+      initContainers:
+        # Gates Agent startup on NATS + IdP Config + PDP Policy (+ optional RAG
+        # Ingest) health, then provisions the aiac-events JetStream stream.
+        # Reuses the aiac-agent image (see src/aiac/agent/init/wait_and_provision.py).
+        - name: aiac-init
+          image: localhost/aiac-agent:local
+          imagePullPolicy: Never
+          command: ["python", "-m", "aiac.agent.init.wait_and_provision"]
+          envFrom:
+            - configMapRef:
+                name: aiac-pdp-config
+          # Same hardening baseline as the app container; temp writes go to /tmp.
+          securityContext:
+            allowPrivilegeEscalation: false
+            readOnlyRootFilesystem: true
+            capabilities:
+              drop:
+                - ALL
+          resources:
+            requests:
+              cpu: 50m
+              memory: 64Mi
+            limits:
+              cpu: 250m
+              memory: 128Mi
+          volumeMounts:
+            - name: tmp
+              mountPath: /tmp
       containers:
         - name: aiac-agent
           image: localhost/aiac-agent:local

aiac/k8s/aiac-deployment-guide.md

@@ -8,7 +8,8 @@ This guide covers the full AIAC deployment in the `aiac-system` namespace.
 |---|---|---|
 | `pdp-interface-deployment.yaml` | Rossoctl Interface Pod (IdP Configuration Service + PDP Policy Writer **Phase 1 rego-file mock** `aiac-pdp-policy-opa`) + 2 ClusterIP Services | 7071, 7072 |
 | `policy-model-store-statefulset.yaml` | Policy Model Store StatefulSet + 1 Gi PVC + headless Service + ClusterIP Service | 7074 |
-| `agent-deployment.yaml` | Agent Pod Deployment (AIAC Agent) + ClusterIP Service | 7070 |
+| `event-broker-deployment.yaml` | NATS JetStream Event Broker Deployment + ClusterIP Service | 4222 |
+| `agent-deployment.yaml` | Agent Pod Deployment (`aiac-init` init container + AIAC Agent) + ClusterIP Service | 7070 |
 
 ## Prerequisites
 
@@ -37,11 +38,14 @@ docker build -f aiac/src/aiac/pdp/service/policy/opa/Dockerfile \
 docker build -f aiac/src/aiac/policy/model_store/service/Dockerfile \
   -t localhost/aiac-policy-model-store:local aiac/src/
 
-# AIAC Agent
+# AIAC Agent (also used as the aiac-init init container, via a command override) — context: aiac/src/
 docker build -f aiac/src/aiac/agent/controller/Dockerfile \
   -t localhost/aiac-agent:local aiac/src/
 ```
 
+The Event Broker uses the stock `nats:2.14-alpine` image (pinned in
+`event-broker-deployment.yaml`) — no build step.
+
 ## 2 — Load images into the cluster
 
 **Kind (local development)**
@@ -53,15 +57,26 @@ kind load docker-image localhost/aiac-policy-model-store:local     --name <clust
 kind load docker-image localhost/aiac-agent:local            --name <cluster-name>
 ```
 
+For a fully air-gapped Kind cluster (no outbound network access), also pull and load the
+NATS image; `event-broker-deployment.yaml` uses `imagePullPolicy: IfNotPresent`, so a
+networked cluster can skip this and pull it directly:
+
+```bash
+docker pull nats:2.14-alpine
+kind load docker-image nats:2.14-alpine --name <cluster-name>
+```
+
 **Remote registry** — tag, push, then update the `image:` fields in the manifests to match.
 
-> **Note:** the manifests set `imagePullPolicy: Never` because images are side-loaded
-> into a local Kind cluster (dev only). For a real cluster that pulls from a registry,
-> change these to `imagePullPolicy: IfNotPresent` (or `Always`).
+## 3 — Create the secrets
+
+Two Secrets must exist in `aiac-system` before applying the manifests. Create the namespace first, then both secrets.
 
-## 3 — Create the admin secret
+```bash
+kubectl create namespace aiac-system
+```
 
-The Interface Pod requires a `keycloak-admin-secret` Secret. Create it once per cluster before applying the manifests:
+**`keycloak-admin-secret`** — required by the Interface Pod:
 
 ```bash
 kubectl create secret generic keycloak-admin-secret \
@@ -114,10 +129,13 @@ Edit the `aiac-pdp-config` ConfigMap in `pdp-interface-deployment.yaml` to match
 | `AIAC_PDP_POLICY_URL` | `http://aiac-pdp-policy-service:7072` | Agent |
 | `AIAC_POLICY_MODEL_STORE_URL` | `http://aiac-policy-model-store-service:7074` | Agent |
 | `SERVICEPOLICY_DB_PATH` | `/data/policy_model.db` | Policy Model Store |
-| `NATS_URL` | `nats://aiac-event-broker-service:4222` | Agent — **added in Phase 2** (Event Broker, issue 4.19) |
+| `NATS_URL` | `nats://aiac-event-broker-service:4222` | Agent, `aiac-init` — Event Broker ClusterIP address |
 | `AIAC_RAG_INGEST_URL` | `http://aiac-rag-service:7073` | Init container — **added in Phase 3** (RAG Pod, issue 4.20) |
 | `AIAC_CHROMADB_URL` | `http://aiac-rag-service:8000` | Agent — **added in Phase 3** (RAG Pod, issue 4.20) |
 
+`aiac-init` treats `AIAC_RAG_INGEST_URL` as optional and skips the RAG Ingest health check
+when it is unset (the current phase has no RAG pod deployed yet).
+
 ## 5 — Deploy
 
 Apply in dependency order:
@@ -126,17 +144,22 @@ Apply in dependency order:
 # 1. Interface Pod — creates the namespace, ConfigMap, Secret, and ClusterIP Services
 kubectl apply -f aiac/k8s/pdp-interface-deployment.yaml
 
-# 2. Policy Model Store — needs the aiac-system namespace
+# 2. Event Broker — NATS JetStream, no dependencies
+kubectl apply -f aiac/k8s/event-broker-deployment.yaml
+
+# 3. Policy Model Store — needs the aiac-system namespace
 kubectl apply -f aiac/k8s/policy-model-store-statefulset.yaml
 
-# 3. Agent — depends on the Interface Pod + Policy Model Store already being healthy
+# 4. Agent — aiac-init waits for NATS + Interface Pod to be healthy (it does not
+#    currently gate on Policy Model Store readiness)
 kubectl apply -f aiac/k8s/agent-deployment.yaml
 ```
 
 Wait for all pods to be ready:
 
 ```bash
 kubectl wait deployment/aiac-interface     -n aiac-system --for=condition=Available --timeout=120s
+kubectl wait deployment/aiac-event-broker  -n aiac-system --for=condition=Available --timeout=120s
 kubectl wait statefulset/aiac-policy-model-store -n aiac-system --for=jsonpath='{.status.readyReplicas}'=1 --timeout=120s
 kubectl wait deployment/aiac-agent         -n aiac-system --for=condition=Available --timeout=120s
 ```
@@ -148,25 +171,62 @@ Port-forward each service and check its health endpoint:
 ```bash
 # IdP Configuration Service
 kubectl port-forward svc/aiac-pdp-config-service 7071:7071 -n aiac-system &
+pf_pids=$!
 curl http://localhost:7071/health
 # {"status":"ok"}
 
 # PDP Policy Writer
 kubectl port-forward svc/aiac-pdp-policy-service 7072:7072 -n aiac-system &
+pf_pids="$pf_pids $!"
 curl http://localhost:7072/health
 # {"status":"ok"}
 
 # Policy Model Store
 kubectl port-forward svc/aiac-policy-model-store-service 7074:7074 -n aiac-system &
+pf_pids="$pf_pids $!"
 curl http://localhost:7074/health
 # {"status":"ok"}
 
 # AIAC Agent
 kubectl port-forward svc/aiac-agent-service 7070:7070 -n aiac-system &
+pf_pids="$pf_pids $!"
 curl http://localhost:7070/health
 # {"status":"ok"}
 
-pkill -f "port-forward"
+# cleanup only the tunnels started above (not unrelated port-forward sessions)
+kill $pf_pids
+```
+
+### NATS Event Broker — end-to-end check
+
+Requires the [`nats` CLI](https://github.com/nats-io/natscli).
+
+```bash
+kubectl port-forward svc/aiac-event-broker-service 4222:4222 -n aiac-system &
+pf_pid=$!
+
+# Wait for the tunnel to accept connections before publishing through it — port-forward
+# starts in the background and needs a moment; fail fast if it exits instead of connecting.
+for i in $(seq 1 30); do
+  if ! kill -0 "$pf_pid" 2>/dev/null; then
+    echo "port-forward exited before becoming ready" >&2
+    exit 1
+  fi
+  (exec 3<>/dev/tcp/localhost/4222) 2>/dev/null && exec 3>&- && break
+  sleep 0.5
+done
+
+nats context save aiac --server nats://localhost:4222
+nats context select aiac
+
+# Publish a test service-onboarding event (use a real IdP client UUID to see it
+# processed end to end; any string will demonstrate delivery either way):
+nats pub aiac.apply.service.<test-uuid> '{"id":"<test-uuid>"}'
+
+# Confirm the Agent processed and acked it (no redelivery):
+kubectl logs deployment/aiac-agent -n aiac-system -c aiac-agent --tail=50
+
+kill "$pf_pid"
 ```
 
 Run the IdP data smoke test:
@@ -181,9 +241,10 @@ pkill -f "port-forward.*7071"
 ## Redeploying after a code change
 
 ```bash
-# Rebuild the changed image, e.g. IdP Configuration Service:
+# Rebuild the changed image, e.g. IdP Configuration Service (context: the service dir):
 docker build -f aiac/src/aiac/idp/service/configuration/keycloak/Dockerfile \
-  -t localhost/aiac-pdp-config:local aiac/src/
+  -t localhost/aiac-pdp-config:local \
+  aiac/src/aiac/idp/service/configuration/keycloak/
 kind load docker-image localhost/aiac-pdp-config:local --name <cluster-name>
 
 # Restart the affected deployment:

aiac/k8s/event-broker-deployment.yaml

@@ -0,0 +1,87 @@
+---
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: aiac-event-broker
+  namespace: aiac-system
+spec:
+  replicas: 1
+  selector:
+    matchLabels:
+      app: aiac-event-broker
+  template:
+    metadata:
+      labels:
+        app: aiac-event-broker
+    spec:
+      # The stock image runs as root by default. fsGroup makes the mounted JetStream data
+      # volume group-writable by UID 10001, matching the rest of AIAC's non-root convention.
+      securityContext:
+        runAsNonRoot: true
+        runAsUser: 10001
+        runAsGroup: 10001
+        fsGroup: 10001
+        seccompProfile:
+          type: RuntimeDefault
+      containers:
+        - name: nats
+          image: nats:2.14-alpine
+          imagePullPolicy: IfNotPresent
+          args: ["-js", "-sd", "/data/jetstream"]
+          ports:
+            - containerPort: 4222
+          # JetStream data lives on its own volume; the rest of the root filesystem is
+          # read-only, so temp writes go to the /tmp emptyDir.
+          securityContext:
+            allowPrivilegeEscalation: false
+            readOnlyRootFilesystem: true
+            capabilities:
+              drop:
+                - ALL
+          resources:
+            requests:
+              cpu: 50m
+              memory: 64Mi
+            limits:
+              cpu: 250m
+              memory: 256Mi
+          # No HTTP health endpoint on a plain "-js" NATS server (no -m monitoring port), so
+          # this uses tcpSocket like the repo's other non-HTTP workloads.
+          readinessProbe:
+            tcpSocket:
+              port: 4222
+            initialDelaySeconds: 5
+            periodSeconds: 10
+          livenessProbe:
+            tcpSocket:
+              port: 4222
+            initialDelaySeconds: 10
+            periodSeconds: 20
+          volumeMounts:
+            - name: jetstream-data
+              mountPath: /data/jetstream
+            - name: tmp
+              mountPath: /tmp
+      volumes:
+        # Dev-only: emptyDir means a pod restart or eviction drops the aiac-events stream and
+        # any unacked messages. aiac-init recreates the stream, but in-flight messages are lost.
+        # Fine while this broker is opt-in/inactive; switch to a PersistentVolumeClaim before
+        # this is wired live.
+        - name: jetstream-data
+          emptyDir: {}
+        - name: tmp
+          emptyDir: {}
+
+---
+apiVersion: v1
+kind: Service
+metadata:
+  name: aiac-event-broker-service
+  namespace: aiac-system
+spec:
+  selector:
+    app: aiac-event-broker
+  ports:
+    - name: nats
+      port: 4222
+      targetPort: 4222

aiac/k8s/pdp-interface-deployment.yaml

@@ -22,7 +22,8 @@ data:
   # PLATFORM_SOURCE_CLIENTS is the comma-separated list of platform bypass
   # clients the inbound Rego admits without a user role.
   PLATFORM_SOURCE_CLIENTS: "rossoctl"
-  # NATS_URL is added to this ConfigMap in Phase 2 (Event Broker, issue 4.19).
+  # NATS_URL points the PDP services at the Event Broker (Phase 2, issue 4.19).
+  NATS_URL: "nats://aiac-event-broker-service:4222"
   # AIAC_RAG_INGEST_URL and AIAC_CHROMADB_URL are added in Phase 3 (RAG Pod, issue 4.20).
 
 ---

aiac/keycloak-spi/.dockerignore

@@ -0,0 +1,8 @@
+target/
+.idea/
+.vscode/
+.git/
+.gitignore
+*.iml
+README.md
+Makefile

aiac/keycloak-spi/.gitignore

@@ -0,0 +1,2 @@
+# Maven build output — regenerated by `mvn package`, never committed.
+target/

aiac/keycloak-spi/Dockerfile

@@ -0,0 +1,27 @@
+# syntax=docker/dockerfile:1.7
+
+ARG KEYCLOAK_IMAGE=quay.io/keycloak/keycloak:26.6.3
+
+# Stage 1: build the SPI JAR (shaded — bundles jnats, since it isn't on Keycloak's classpath).
+FROM maven:3.9-eclipse-temurin-17 AS jar-builder
+WORKDIR /build
+COPY pom.xml .
+COPY src ./src
+RUN --mount=type=cache,target=/root/.m2 mvn -B -DskipTests package
+
+# Stage 2: drop the JAR into Keycloak and run `kc.sh build` so the augmented server is baked
+# into the final image (no per-pod build at startup).
+FROM ${KEYCLOAK_IMAGE} AS keycloak-builder
+COPY --from=jar-builder /build/target/aiac-event-listener-*.jar /opt/keycloak/providers/
+RUN /opt/keycloak/bin/kc.sh build
+
+# Stage 3: final runtime image — copy the augmented Keycloak from stage 2.
+FROM ${KEYCLOAK_IMAGE}
+COPY --from=keycloak-builder /opt/keycloak/ /opt/keycloak/
+# Match the AIAC non-root UID convention (10001) used by every other service container in this
+# repo, instead of the base image's default UID 1000. /opt/keycloak is group-writable by GID 0
+# (the base image's own "arbitrary UID" convention), so UID 10001 in group 0 already has the
+# access it needs — no useradd/chown required.
+USER 10001:0
+ENTRYPOINT ["/opt/keycloak/bin/kc.sh"]
+CMD ["start"]