feat: add Gemini via Vertex AI, authenticated with Workload Identity Federation

[jcschaff]

Adds gemini-model as a third option next to openai-model and local-model, serving
gemini-2.5-flash from us-central1 on the CAIA grant project caia-fac-uconn-2-225-2026.

Draft until Burwood creates the identity pool (REQ0040169). The pool and provider IDs are
REPLACE_ME placeholders in two places — the projected volume's audience in
kustomize/base/litellm.yaml and the audience field of kustomize/base/gcp-wif.json. Everything
else is finished and verified.

Why this isn't just an API key

Every other model here authenticates with a key in a sealed secret. Gemini can't: org policy
iam.managed.disableServiceAccountKeyCreation blocks service-account key creation on the CAIA
project. Burwood offered to exclude our project from that policy; we'd rather not — it would put a
non-expiring, portable credential in this repo, make us the org's standing exception to a policy
that exists for good reason, and leave us owning rotation.

So the proxy authenticates with Workload Identity Federation instead. The only credential in the pod
is a 1-hour projected ServiceAccount token that kubelet rotates in place:

projected token (/var/run/secrets/gcp/token)
  -> external-account config (/etc/gcp/wif.json, via GOOGLE_APPLICATION_CREDENTIALS)
  -> sts.googleapis.com          k8s token -> federated token
  -> iamcredentials.googleapis.com          impersonate the Google SA
  -> bearer token for aiplatform.googleapis.com

The credential config holds no key material, so it rides in the plain litellm-config-file
ConfigMap. Adding Gemini required no kubeseal round-trip and no change to secrets.dat or
sealed_secret_litellm.sh — that's the practical payoff.

The non-obvious constraint

Our API server is VPN-only and the issuer https://kubernetes.default.svc.cluster.local doesn't
resolve publicly, so Google cannot fetch our JWKS. The provider must be created with the keys
uploaded inline via --jwk-json-path. This is a documented path, not a workaround — Google's
Workload Identity Federation with Kubernetes
covers self-hosted clusters and states "The cluster doesn't need to be accessible over the internet."

This was the one correction we sent back on the ticket: Burwood's first reply asked for the issuer
URL and JWKS, which was right, but the provider has to be told to trust the uploaded keys rather
than go and get them.

What was verified against the live cluster

Measured against vcell-ai-rke-dev and the running litellm pod, rather than assumed:

Custom-audience token minting	works — sub = system:serviceaccount:vcell-ai-rke-dev:<sa>, iss as above. --api-audiences does not restrict projected-token audiences
Pod egress to Google	sts, iamcredentials, us-central1-aiplatform all reachable; no firewall or proxy in the way
LiteLLM	1.92.0 with google-auth 2.52.0 — vertex_llm_base.py already branches on type == "external_account" and calls identity_pool.Credentials.from_info(). Nothing needs upgrading
End-to-end rehearsal	the real rendered gcp-wif.json plus a real cluster token was fed to google.auth inside the pod: parsed as an identity-pool credential, wired impersonation, read the token, and reached Google's STS, which rejected only the POOL_ID placeholder. Every link except the pool is proven

Design choices

• A dedicated litellm ServiceAccount, not the namespace default — the pool's attribute
  condition names exactly one subject, so federating default would let any pod in the namespace
  mint Vertex credentials.
• GOOGLE_APPLICATION_CREDENTIALS, not litellm_params.vertex_credentials. Both work, but
  leaving the variable unset makes google.auth.default() fall back to a developer's own gcloud
  ADC, so the same alias works locally with no config change. Pointing vertex_credentials at a
  path that doesn't exist locally would instead fail, because LiteLLM treats a non-existent path as
  inline JSON and tries to parse it. Hence vcell-ai-local sets the project and location but not
  the credentials path.
• Impersonation, not direct resource access — the better-established path for Vertex AI, and it
  sidesteps the configuration where LiteLLM's since-fixed missing-scopes bug used to bite. One extra
  IAM binding.
• The two audience strings differ by scheme on purpose — https:// in the projected volume
  (Google's manifest form), schemeless // in gcp-wif.json (what create-cred-config emits).
  Both are default allowed audiences; there's a comment saying not to "fix" it.

Ongoing maintenance to accept

The uploaded JWKS is a static snapshot. If the cluster's service-account signing key is ever
rotated, every Gemini call starts failing with no alerting until someone re-uploads it. Documented
in docs/gcp-wif-findings.md with the recovery command and an STS-error-to-cause table.

Testing

• kubectl kustomize on the vcell-ai-rke-dev overlay builds clean; ServiceAccount, projected
  volume, both ConfigMap mounts and the env vars all render as intended.
• The alias name agrees across all three layers that must match — LiteLLM config, the backend
  LLMModel Literal, and the frontend ModelId union.
• vcell-ai-rke and vcell-ai-local do not kustomize build, but that is pre-existing (their
  sealed secrets are gitignored and absent from disk) and reproduces identically on main.

Still to do once the pool exists: replace the placeholders, then walk the 5-step verification ladder
in docs/gcp-wif-findings.md — token shape, STS exchange, the LiteLLM alias, end-to-end through the
UI, and a call more than an hour after pod start to confirm token rotation.

🤖 Generated with Claude Code

https://claude.ai/code/session_014ZbiJ9tw2PWi8A4coNs5w5