docs(dha): PoC finding — CNPG scale subresource has no selector; reopen actuation mechanism

design-proposals/database-horizontal-autoscaling/README.md

@@ -4,21 +4,38 @@
 - **Author(s):** `@scooby87`
 - **Date:** `2026-07-08`; revised `2026-07-24` (mechanism), `2026-07-29` and `2026-07-31` (addressing @lllamnyp and @IvanHunters review on PR #44), with earlier review by @IvanHunters, Gemini, and CodeRabbit
 - **Status:** Draft
+- **PoC:** `2026-08-10` — live validation confirmed the mechanism and pinned a hard precondition: **CNPG ≥ 1.28.4** (the `Cluster` scale subresource must serve `status.selector`); see [PoC finding](#poc-finding-2026-08-10--cnpg-version-floor-for-hpa-actuation)
 
 ## Overview
 
 This proposal adds automatic horizontal scaling of a managed database's **read replicas** in response to load. The mechanism is **entirely stock**: the application chart renders a **KEDA `ScaledObject`** next to the database; KEDA queries VictoriaMetrics for the read load, computes the desired count with a plain `HorizontalPodAutoscaler` it manages, and drives the engine operator's **`scale` subresource** (CloudNativePG `Cluster.spec.instances`). There is **no bespoke operator and no new CRD** — the net-new surface of this proposal is a Helm helper, one `autoscaling` values block, one PromQL query, and KEDA added as a platform component.
 
 The proposal is deliberately scoped to **horizontal scaling of read replicas**: a stateful primary cannot be scaled horizontally the way a stateless Deployment can. The MVP targets **PostgreSQL (CloudNativePG)**; see [Scope](#scope-and-related-proposals) for the engine ladder.
 
+> **Note (2026-08-10):** a live PoC validated this design and pinned a version floor — a stock HPA can only drive the CNPG `Cluster` scale subresource on **CNPG ≥ 1.28.4**, where the subresource serves `status.selector` (cloudnative-pg#8996). The mechanism is unchanged; see the [PoC finding](#poc-finding-2026-08-10--cnpg-version-floor-for-hpa-actuation) below.
+
 ### Why this changed
 
-This design converged over three revisions, each removing machinery the previous one thought it needed. Rev1 proposed a bespoke `db-autoscaler` operator that *owned* the application's `replicas` value and enforced that ownership; an implementation spike proved the enforcement premise unbuildable on the aggregated apps API, and showed the whole conflict was self-imposed — it exists only because the chart unconditionally templates the replica field (full findings in the [Appendix](#appendix-findings-from-the-implementation-spike)). Rev2/rev3 therefore moved to a stock HPA on the engine's `scale` subresource with the chart omitting the field, keeping only a thin controller and CRD to render the HPA and drive a synthesized metric. Review then showed even that is unnecessary: the metric can be *queried* into existence rather than emitted per-pod, and once the query exists, KEDA renders and manages everything declaratively — so the controller and CRD are gone too. The guiding principle throughout: reuse the platform Kubernetes ships, do not reimplement it.
+This design converged over three revisions, each removing machinery the previous one thought it needed. Rev1 proposed a bespoke `db-autoscaler` operator that *owned* the application's `replicas` value and enforced that ownership; an implementation spike proved the enforcement premise unbuildable on the aggregated apps API, and showed the whole conflict was self-imposed — it exists only because the chart unconditionally templates the replica field (full findings in the [Appendix](#appendix-findings-from-the-implementation-spike)). Rev2/rev3 therefore moved to a stock HPA on the engine's `scale` subresource with the chart omitting the field, keeping only a thin controller and CRD to render the HPA and drive a synthesized metric. Review then showed even that is unnecessary: the metric can be *queried* into existence rather than emitted per-pod, and once the query exists, KEDA renders and manages everything declaratively — so the controller and CRD are gone too. The guiding principle throughout: reuse the platform Kubernetes ships, do not reimplement it. The PoC below shows that last step relied on one thing the vendored CNPG version did not yet provide — since fixed upstream, leaving the mechanism unchanged behind a version floor.
+
+## PoC finding (2026-08-10) — CNPG version floor for HPA actuation
+
+A live PoC on a dev cluster (cozystack v45, CloudNativePG 1.27.3, Kubernetes 1.34.3) validated this design and pinned one hard precondition: the actuation path works only on **CNPG ≥ 1.28.4**. This section is a record, not an open decision — the mechanism is unchanged.
+
+**What the PoC confirmed.** The single-value metric works on real data: for a live CNPG cluster the query `Σ(active read connections over the standby pods) + target` returned `151` at `target = 150` (one active connection on the replica), so `desired = ceil(151/150) = 2 = 1 primary + 1 read replica` — the §1 arithmetic holds. The required series and labels exist in VictoriaMetrics (`cnpg_backends_total{state="active"}`, and `kube_pod_labels` carrying `label_cnpg_io_cluster` and `label_cnpg_io_instance_role`). The KEDA package installs, its `external.metrics.k8s.io` APIService becomes Available, and KEDA renders the `ScaledObject` into a managed HPA.
+
+**What blocked it — and why it is version-bound.** On the CNPG the PoC ran (1.27.x), the HPA never scales: it reports `ScalingActive=False, reason=InvalidSelector` — *"the HPA target's scale is missing a selector"*. That CNPG's `Cluster` `/scale` subresource returned only `status: {replicas: N}` with no `status.selector`, and the CRD declared no `labelSelectorPath`; the Kubernetes HPA controller requires `scale.status.selector` unconditionally, before any metric-type branching, so it fails for every target type (confirmed with both `AverageValue` and `Value`). This is fixed upstream. cloudnative-pg#2574 ("Support the scale sub-resource for Clusters") was implemented by cloudnative-pg#8996 ("feat: add label selector to the Cluster scale subresource"), which populates `Cluster.status.selector` on every status update and wires `subresources.scale.labelSelectorPath: .status.selector` into the CRD, naming HPA/VPA explicitly. It shipped on 2026-06-29 in **CNPG v1.28.4, v1.29.2, and v1.30.0**, and was not backported to the 1.27 line the PoC ran on. (The earlier issue [#7923](https://github.com/cloudnative-pg/cloudnative-pg/issues/7923) asking for the same thing was not declined — it went stale as an unnoticed duplicate; #2574/#8996 is the accepted, completed work.)
+
+**Resolution — no design change.** On any CNPG ≥ 1.28.4 the approved mechanism (chart conditional + KEDA `ScaledObject` + the `Σ + target` query) works byte-for-byte on stock CNPG against a real operator-maintained selector. The fix is therefore a routine vendored-chart bump, tracked as a separate cozystack PR (`packages/system/postgres-operator` currently vendors CNPG 1.28.2, two patch releases below the floor — the bump is 1.28.2 → ≥ 1.28.4), followed by a re-run of this same PoC as the verification gate. Two alternatives were considered on the way to this record and are **rejected** as unnecessary once the selector ships: (A) KEDA plus a thin mirror shim behind a proxy object that exposes a selector; (B) a lean actuation controller writing `Cluster.spec.instances` directly. (A) adds a CRD and a controller to work around a gap that no longer exists; (B) re-acquires the hand-rolled decision loop this design spent three revisions deleting. Neither would have re-introduced rev1's ownership machinery — §3 (the chart omitting the field) already removed that — but neither is needed.
+
+**Fail-safe (carried into implementation).** When VictoriaMetrics returns no sample, `NaN`, stale data, or a timeout, the actuator must not read it as zero and scale down to the quorum floor during a monitoring outage; it holds the last-known-good count and does not scale — the same "never scale blind" behavior already stated in [Failure and edge cases](#failure-and-edge-cases).
 
 ## Scope and related proposals
 
 This proposal covers **horizontal** autoscaling (read replicas) only. Two sibling axes are deferred to separate proposals: **vertical autoscaling** (stepping the `resourcesPreset` ladder / in-place resize) and **storage autoscaling** (automatic PVC expansion). Write-path scaling that requires data rebalancing (Kafka broker addition, ClickHouse/MongoDB sharding) is out of scope — it is an orchestrated procedure, not a counter change.
 
+**Precondition — CNPG ≥ 1.28.4.** The `Cluster` scale subresource must serve `status.selector`, or the HPA rejects it with `InvalidSelector` and never scales (see the [PoC finding](#poc-finding-2026-08-10--cnpg-version-floor-for-hpa-actuation)). The selector was added in cloudnative-pg#8996 and shipped in CNPG 1.28.4 / 1.29.2 / 1.30.0; cozystack must vendor CNPG at or above that floor (it currently ships 1.28.2) for autoscaling to function.
+
 **Engine scope of the MVP.** The mechanism applies to engines whose operator CR exposes a `scale` subresource: PostgreSQL (CloudNativePG `Cluster.spec.instances`) and MariaDB (`MariaDB.spec.replicas`). The MVP ships **PostgreSQL**; MariaDB follows once its cozystack chart supports on-the-fly scale-out (today it does not — see [Failure and edge cases](#failure-and-edge-cases)). **Redis (spotahome RedisFailover) and MongoDB (Percona) expose no `scale` subresource**, so a stock HPA cannot drive them; they are deferred to a follow-up that adds a thin actuation shim (see [Alternatives considered](#alternatives-considered)).
 
 ## Context