From de5992855e2d4f37d196a3b99f6c1cfcfdd17238 Mon Sep 17 00:00:00 2001 From: Alexey Artamonov Date: Mon, 10 Aug 2026 16:57:20 +0300 Subject: [PATCH 1/3] =?UTF-8?q?docs(dha):=20PoC=20finding=20=E2=80=94=20CN?= =?UTF-8?q?PG=20scale=20subresource=20has=20no=20selector;=20reopen=20mech?= =?UTF-8?q?anism?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A live PoC validated the metric encoding (Σ+target drives the right count on real data) but found the load-bearing mechanism — a stock HPA via KEDA driving the CNPG Cluster scale subresource — is not buildable on stock CNPG: the Cluster /scale exposes no status.selector (no labelSelectorPath), so the HPA controller fails with InvalidSelector for every metric type. Upstream CNPG #7923 (add the selector) is closed as not planned; not fixable by a version bump. Record the finding and two resolution options — (A) KEDA + a thin mirror shim behind a proxy object with a selector, or (B) a lean actuation controller (no KEDA) writing Cluster.spec.instances directly. Note that neither returns rev1's ownership machinery, which §3 already removed. This reopens the actuation mechanism decision. Signed-off-by: Alexey Artamonov --- .../database-horizontal-autoscaling/README.md | 24 +++++++++++++++++-- 1 file changed, 22 insertions(+), 2 deletions(-) diff --git a/design-proposals/database-horizontal-autoscaling/README.md b/design-proposals/database-horizontal-autoscaling/README.md index a1630a4..f92c0bf 100644 --- a/design-proposals/database-horizontal-autoscaling/README.md +++ b/design-proposals/database-horizontal-autoscaling/README.md @@ -3,7 +3,8 @@ - **Title:** `Database Horizontal Autoscaler for Cozystack` - **Author(s):** `@scooby87` - **Date:** `2026-07-08`; revised `2026-07-24` (mechanism), `2026-07-29` and `2026-07-31` (addressing @lllamnyp and @IvanHunters review on PR #44), with earlier review by @IvanHunters, Gemini, and CodeRabbit -- **Status:** Draft +- **Status:** Draft — mechanism reopened by the `2026-08-10` PoC finding below +- **PoC:** `2026-08-10` — live validation surfaced a blocking constraint in CNPG; see [PoC finding](#poc-finding-2026-08-10--blocking-the-cnpg-scale-subresource-exposes-no-selector) ## Overview @@ -11,9 +12,28 @@ This proposal adds automatic horizontal scaling of a managed database's **read r The proposal is deliberately scoped to **horizontal scaling of read replicas**: a stateful primary cannot be scaled horizontally the way a stateless Deployment can. The MVP targets **PostgreSQL (CloudNativePG)**; see [Scope](#scope-and-related-proposals) for the engine ladder. +> **Note (2026-08-10):** a live PoC validated the metric side of this design but found that a stock HPA cannot drive the CNPG `Cluster` scale subresource (it exposes no selector), which reopens the actuation mechanism. See [PoC finding](#poc-finding-2026-08-10--blocking-the-cnpg-scale-subresource-exposes-no-selector) below; the sections after it describe the mechanism as it stood before that finding. + ### Why this changed -This design converged over three revisions, each removing machinery the previous one thought it needed. Rev1 proposed a bespoke `db-autoscaler` operator that *owned* the application's `replicas` value and enforced that ownership; an implementation spike proved the enforcement premise unbuildable on the aggregated apps API, and showed the whole conflict was self-imposed — it exists only because the chart unconditionally templates the replica field (full findings in the [Appendix](#appendix-findings-from-the-implementation-spike)). Rev2/rev3 therefore moved to a stock HPA on the engine's `scale` subresource with the chart omitting the field, keeping only a thin controller and CRD to render the HPA and drive a synthesized metric. Review then showed even that is unnecessary: the metric can be *queried* into existence rather than emitted per-pod, and once the query exists, KEDA renders and manages everything declaratively — so the controller and CRD are gone too. The guiding principle throughout: reuse the platform Kubernetes ships, do not reimplement it. +This design converged over three revisions, each removing machinery the previous one thought it needed. Rev1 proposed a bespoke `db-autoscaler` operator that *owned* the application's `replicas` value and enforced that ownership; an implementation spike proved the enforcement premise unbuildable on the aggregated apps API, and showed the whole conflict was self-imposed — it exists only because the chart unconditionally templates the replica field (full findings in the [Appendix](#appendix-findings-from-the-implementation-spike)). Rev2/rev3 therefore moved to a stock HPA on the engine's `scale` subresource with the chart omitting the field, keeping only a thin controller and CRD to render the HPA and drive a synthesized metric. Review then showed even that is unnecessary: the metric can be *queried* into existence rather than emitted per-pod, and once the query exists, KEDA renders and manages everything declaratively — so the controller and CRD are gone too. The guiding principle throughout: reuse the platform Kubernetes ships, do not reimplement it. The PoC below shows that last step reused one thing Kubernetes cannot actually provide here. + +## PoC finding (2026-08-10) — blocking: the CNPG scale subresource exposes no selector + +A live PoC on a dev cluster (cozystack v45, CloudNativePG 1.27.3, Kubernetes 1.34.3) validated the metric side of this design but uncovered a constraint the mechanism above does not survive as written. + +**What the PoC confirmed.** The single-value metric works on real data: for a live CNPG cluster the query `Σ(active read connections over the standby pods) + target` returned `151` at `target = 150` (one active connection on the replica), so `desired = ceil(151/150) = 2 = 1 primary + 1 read replica` — the §1 arithmetic holds. The required series and labels exist in VictoriaMetrics (`cnpg_backends_total{state="active"}`, and `kube_pod_labels` carrying `label_cnpg_io_cluster` and `label_cnpg_io_instance_role`). The KEDA package installs, its `external.metrics.k8s.io` APIService becomes Available, and KEDA renders the `ScaledObject` into a managed HPA. + +**The blocker.** That HPA never scales: it reports `ScalingActive=False, reason=InvalidSelector` — *"the HPA target's scale is missing a selector"*. The CNPG `Cluster` `/scale` subresource returns only `status: {replicas: N}` — no `status.selector` — and the CRD declares no `labelSelectorPath`. The Kubernetes HPA controller requires `scale.status.selector` unconditionally, before any metric-type branching, so this fails for **every** target type (confirmed with both `AverageValue` and `Value`). It is not a calibration detail and not fixable by a version bump: upstream CNPG issue [#7923](https://github.com/cloudnative-pg/cloudnative-pg/issues/7923), which requested exactly this selector for HPA/KEDA, is **closed as not planned**, and the CNPG 1.30 docs explicitly recommend against HPA for a `Cluster`. + +**Consequence.** The load-bearing mechanism of this revision — a stock HPA (via KEDA) driving the CNPG `Cluster` scale subresource — cannot be built on stock CNPG; an actuation bridge is required after all. Crucially, what returns is **not** the machinery that got rev1 rejected: the ownership/enforcement layer (SSA, marker annotation, HelmRelease webhook, terminal-freeze) existed only because the chart declared `replicas`, and §3 (the chart omitting the field under autoscaling) removes it regardless of mechanism. What returns is only the small write-the-count actuator. + +**Two options to resolve (decision needed).** + +- **Option A — KEDA + a thin mirror shim.** Keep KEDA's hardened decision loop by pointing its HPA at a proxy object that *does* expose a selector (a small owned CRD, or a placeholder workload), and add a tiny controller that mirrors the proxy's computed count into `Cluster.spec.instances`. Preserves the stock decision loop, but adds a shim, a proxy object, and the platform-wide KEDA dependency for a value KEDA cannot deliver end-to-end on its own. +- **Option B — a lean actuation controller, no KEDA.** A small controller reads the read-load metric from VictoriaMetrics and writes `Cluster.spec.instances` directly, applying the `min`/`max`/quorum-floor bounds and stabilization. This is close to rev1 **minus the ownership machinery** (which §3 already eliminates) and minus the aggregated-API enforcement — a much smaller component than the rejected operator, with no KEDA platform dependency, at the cost of a modest amount of stabilization logic KEDA would otherwise provide. + +Both keep §1 (the validated metric encoding) and §3 (chart omits the field). The choice is where the desired-count computation lives — stock KEDA behind a proxy, or a lean purpose-built loop — and whether to take on KEDA as a platform dependency. This reopens the mechanism decision made in the previous revision; the sections below describe the pre-finding mechanism and stand until that decision is made. ## Scope and related proposals From 2c126855dcf290e038fb657d640a769ec0a17c1a Mon Sep 17 00:00:00 2001 From: Alexey Artamonov Date: Tue, 18 Aug 2026 17:04:16 +0300 Subject: [PATCH 2/3] =?UTF-8?q?docs(dha):=20rework=20PoC=20finding=20into?= =?UTF-8?q?=20a=20closed=20record=20=E2=80=94=20CNPG=20>=3D=201.28.4=20ver?= =?UTF-8?q?sion=20floor?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Address @lllamnyp's review: the finding is real but version-bound, not a mechanism blocker. Correct the upstream story (cloudnative-pg#7923 went stale as a duplicate, not declined; #2574 was implemented by #8996, which adds Cluster.status.selector + labelSelectorPath and shipped in CNPG 1.28.4 / 1.29.2 / 1.30.0, not backported to 1.27). Record options A and B as rejected — unnecessary once the selector ships; the fix is a routine CNPG vendored-chart bump (1.28.2 -> >= 1.28.4) plus a PoC re-run as the gate. Add the CNPG >= 1.28.4 precondition to Scope, restore the Status line (mechanism not reopened), and carry the fail-safe-metric note into implementation. Signed-off-by: Alexey Artamonov --- .../database-horizontal-autoscaling/README.md | 23 ++++++++----------- 1 file changed, 10 insertions(+), 13 deletions(-) diff --git a/design-proposals/database-horizontal-autoscaling/README.md b/design-proposals/database-horizontal-autoscaling/README.md index f92c0bf..e3383b7 100644 --- a/design-proposals/database-horizontal-autoscaling/README.md +++ b/design-proposals/database-horizontal-autoscaling/README.md @@ -3,8 +3,8 @@ - **Title:** `Database Horizontal Autoscaler for Cozystack` - **Author(s):** `@scooby87` - **Date:** `2026-07-08`; revised `2026-07-24` (mechanism), `2026-07-29` and `2026-07-31` (addressing @lllamnyp and @IvanHunters review on PR #44), with earlier review by @IvanHunters, Gemini, and CodeRabbit -- **Status:** Draft — mechanism reopened by the `2026-08-10` PoC finding below -- **PoC:** `2026-08-10` — live validation surfaced a blocking constraint in CNPG; see [PoC finding](#poc-finding-2026-08-10--blocking-the-cnpg-scale-subresource-exposes-no-selector) +- **Status:** Draft +- **PoC:** `2026-08-10` — live validation confirmed the mechanism and pinned a hard precondition: **CNPG ≥ 1.28.4** (the `Cluster` scale subresource must serve `status.selector`); see [PoC finding](#poc-finding-2026-08-10--cnpg-version-floor-for-hpa-actuation) ## Overview @@ -12,33 +12,30 @@ This proposal adds automatic horizontal scaling of a managed database's **read r The proposal is deliberately scoped to **horizontal scaling of read replicas**: a stateful primary cannot be scaled horizontally the way a stateless Deployment can. The MVP targets **PostgreSQL (CloudNativePG)**; see [Scope](#scope-and-related-proposals) for the engine ladder. -> **Note (2026-08-10):** a live PoC validated the metric side of this design but found that a stock HPA cannot drive the CNPG `Cluster` scale subresource (it exposes no selector), which reopens the actuation mechanism. See [PoC finding](#poc-finding-2026-08-10--blocking-the-cnpg-scale-subresource-exposes-no-selector) below; the sections after it describe the mechanism as it stood before that finding. +> **Note (2026-08-10):** a live PoC validated this design and pinned a version floor — a stock HPA can only drive the CNPG `Cluster` scale subresource on **CNPG ≥ 1.28.4**, where the subresource serves `status.selector` (cloudnative-pg#8996). The mechanism is unchanged; see the [PoC finding](#poc-finding-2026-08-10--cnpg-version-floor-for-hpa-actuation) below. ### Why this changed This design converged over three revisions, each removing machinery the previous one thought it needed. Rev1 proposed a bespoke `db-autoscaler` operator that *owned* the application's `replicas` value and enforced that ownership; an implementation spike proved the enforcement premise unbuildable on the aggregated apps API, and showed the whole conflict was self-imposed — it exists only because the chart unconditionally templates the replica field (full findings in the [Appendix](#appendix-findings-from-the-implementation-spike)). Rev2/rev3 therefore moved to a stock HPA on the engine's `scale` subresource with the chart omitting the field, keeping only a thin controller and CRD to render the HPA and drive a synthesized metric. Review then showed even that is unnecessary: the metric can be *queried* into existence rather than emitted per-pod, and once the query exists, KEDA renders and manages everything declaratively — so the controller and CRD are gone too. The guiding principle throughout: reuse the platform Kubernetes ships, do not reimplement it. The PoC below shows that last step reused one thing Kubernetes cannot actually provide here. -## PoC finding (2026-08-10) — blocking: the CNPG scale subresource exposes no selector +## PoC finding (2026-08-10) — CNPG version floor for HPA actuation -A live PoC on a dev cluster (cozystack v45, CloudNativePG 1.27.3, Kubernetes 1.34.3) validated the metric side of this design but uncovered a constraint the mechanism above does not survive as written. +A live PoC on a dev cluster (cozystack v45, CloudNativePG 1.27.3, Kubernetes 1.34.3) validated this design and pinned one hard precondition: the actuation path works only on **CNPG ≥ 1.28.4**. This section is a record, not an open decision — the mechanism is unchanged. **What the PoC confirmed.** The single-value metric works on real data: for a live CNPG cluster the query `Σ(active read connections over the standby pods) + target` returned `151` at `target = 150` (one active connection on the replica), so `desired = ceil(151/150) = 2 = 1 primary + 1 read replica` — the §1 arithmetic holds. The required series and labels exist in VictoriaMetrics (`cnpg_backends_total{state="active"}`, and `kube_pod_labels` carrying `label_cnpg_io_cluster` and `label_cnpg_io_instance_role`). The KEDA package installs, its `external.metrics.k8s.io` APIService becomes Available, and KEDA renders the `ScaledObject` into a managed HPA. -**The blocker.** That HPA never scales: it reports `ScalingActive=False, reason=InvalidSelector` — *"the HPA target's scale is missing a selector"*. The CNPG `Cluster` `/scale` subresource returns only `status: {replicas: N}` — no `status.selector` — and the CRD declares no `labelSelectorPath`. The Kubernetes HPA controller requires `scale.status.selector` unconditionally, before any metric-type branching, so this fails for **every** target type (confirmed with both `AverageValue` and `Value`). It is not a calibration detail and not fixable by a version bump: upstream CNPG issue [#7923](https://github.com/cloudnative-pg/cloudnative-pg/issues/7923), which requested exactly this selector for HPA/KEDA, is **closed as not planned**, and the CNPG 1.30 docs explicitly recommend against HPA for a `Cluster`. +**What blocked it — and why it is version-bound.** On the CNPG the PoC ran (1.27.x), the HPA never scales: it reports `ScalingActive=False, reason=InvalidSelector` — *"the HPA target's scale is missing a selector"*. That CNPG's `Cluster` `/scale` subresource returned only `status: {replicas: N}` with no `status.selector`, and the CRD declared no `labelSelectorPath`; the Kubernetes HPA controller requires `scale.status.selector` unconditionally, before any metric-type branching, so it fails for every target type (confirmed with both `AverageValue` and `Value`). This is fixed upstream. cloudnative-pg#2574 ("Support the scale sub-resource for Clusters") was implemented by cloudnative-pg#8996 ("feat: add label selector to the Cluster scale subresource"), which populates `Cluster.status.selector` on every status update and wires `subresources.scale.labelSelectorPath: .status.selector` into the CRD, naming HPA/VPA explicitly. It shipped on 2026-06-29 in **CNPG v1.28.4, v1.29.2, and v1.30.0**, and was not backported to the 1.27 line the PoC ran on. (The earlier issue [#7923](https://github.com/cloudnative-pg/cloudnative-pg/issues/7923) asking for the same thing was not declined — it went stale as an unnoticed duplicate; #2574/#8996 is the accepted, completed work.) -**Consequence.** The load-bearing mechanism of this revision — a stock HPA (via KEDA) driving the CNPG `Cluster` scale subresource — cannot be built on stock CNPG; an actuation bridge is required after all. Crucially, what returns is **not** the machinery that got rev1 rejected: the ownership/enforcement layer (SSA, marker annotation, HelmRelease webhook, terminal-freeze) existed only because the chart declared `replicas`, and §3 (the chart omitting the field under autoscaling) removes it regardless of mechanism. What returns is only the small write-the-count actuator. +**Resolution — no design change.** On any CNPG ≥ 1.28.4 the approved mechanism (chart conditional + KEDA `ScaledObject` + the `Σ + target` query) works byte-for-byte on stock CNPG against a real operator-maintained selector. The fix is therefore a routine vendored-chart bump, tracked as a separate cozystack PR (`packages/system/postgres-operator` currently vendors CNPG 1.28.2, two patch releases below the floor — the bump is 1.28.2 → ≥ 1.28.4), followed by a re-run of this same PoC as the verification gate. Two alternatives were considered on the way to this record and are **rejected** as unnecessary once the selector ships: (A) KEDA plus a thin mirror shim behind a proxy object that exposes a selector; (B) a lean actuation controller writing `Cluster.spec.instances` directly. (A) adds a CRD and a controller to work around a gap that no longer exists; (B) re-acquires the hand-rolled decision loop this design spent three revisions deleting. Neither would have re-introduced rev1's ownership machinery — §3 (the chart omitting the field) already removed that — but neither is needed. -**Two options to resolve (decision needed).** - -- **Option A — KEDA + a thin mirror shim.** Keep KEDA's hardened decision loop by pointing its HPA at a proxy object that *does* expose a selector (a small owned CRD, or a placeholder workload), and add a tiny controller that mirrors the proxy's computed count into `Cluster.spec.instances`. Preserves the stock decision loop, but adds a shim, a proxy object, and the platform-wide KEDA dependency for a value KEDA cannot deliver end-to-end on its own. -- **Option B — a lean actuation controller, no KEDA.** A small controller reads the read-load metric from VictoriaMetrics and writes `Cluster.spec.instances` directly, applying the `min`/`max`/quorum-floor bounds and stabilization. This is close to rev1 **minus the ownership machinery** (which §3 already eliminates) and minus the aggregated-API enforcement — a much smaller component than the rejected operator, with no KEDA platform dependency, at the cost of a modest amount of stabilization logic KEDA would otherwise provide. - -Both keep §1 (the validated metric encoding) and §3 (chart omits the field). The choice is where the desired-count computation lives — stock KEDA behind a proxy, or a lean purpose-built loop — and whether to take on KEDA as a platform dependency. This reopens the mechanism decision made in the previous revision; the sections below describe the pre-finding mechanism and stand until that decision is made. +**Fail-safe (carried into implementation).** When VictoriaMetrics returns no sample, `NaN`, stale data, or a timeout, the actuator must not read it as zero and scale down to the quorum floor during a monitoring outage; it holds the last-known-good count and does not scale — the same "never scale blind" behavior already stated in [Failure and edge cases](#failure-and-edge-cases). ## Scope and related proposals This proposal covers **horizontal** autoscaling (read replicas) only. Two sibling axes are deferred to separate proposals: **vertical autoscaling** (stepping the `resourcesPreset` ladder / in-place resize) and **storage autoscaling** (automatic PVC expansion). Write-path scaling that requires data rebalancing (Kafka broker addition, ClickHouse/MongoDB sharding) is out of scope — it is an orchestrated procedure, not a counter change. +**Precondition — CNPG ≥ 1.28.4.** The `Cluster` scale subresource must serve `status.selector`, or the HPA rejects it with `InvalidSelector` and never scales (see the [PoC finding](#poc-finding-2026-08-10--cnpg-version-floor-for-hpa-actuation)). The selector was added in cloudnative-pg#8996 and shipped in CNPG 1.28.4 / 1.29.2 / 1.30.0; cozystack must vendor CNPG at or above that floor (it currently ships 1.28.2) for autoscaling to function. + **Engine scope of the MVP.** The mechanism applies to engines whose operator CR exposes a `scale` subresource: PostgreSQL (CloudNativePG `Cluster.spec.instances`) and MariaDB (`MariaDB.spec.replicas`). The MVP ships **PostgreSQL**; MariaDB follows once its cozystack chart supports on-the-fly scale-out (today it does not — see [Failure and edge cases](#failure-and-edge-cases)). **Redis (spotahome RedisFailover) and MongoDB (Percona) expose no `scale` subresource**, so a stock HPA cannot drive them; they are deferred to a follow-up that adds a thin actuation shim (see [Alternatives considered](#alternatives-considered)). ## Context From f491243f149be9ff22d1b0629fe78391713ed6cf Mon Sep 17 00:00:00 2001 From: Timofei Larkin Date: Tue, 18 Aug 2026 17:48:15 +0300 Subject: [PATCH 3/3] docs(dha): unstale the convergence note The Why-this-changed paragraph still said the PoC hit something Kubernetes cannot provide; the reworked record says the opposite (CNPG >= 1.28.4 provides it). Align the sentence with the record. Assisted-By: Claude Signed-off-by: Timofei Larkin --- design-proposals/database-horizontal-autoscaling/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/design-proposals/database-horizontal-autoscaling/README.md b/design-proposals/database-horizontal-autoscaling/README.md index e3383b7..bfd96a5 100644 --- a/design-proposals/database-horizontal-autoscaling/README.md +++ b/design-proposals/database-horizontal-autoscaling/README.md @@ -16,7 +16,7 @@ The proposal is deliberately scoped to **horizontal scaling of read replicas**: ### Why this changed -This design converged over three revisions, each removing machinery the previous one thought it needed. Rev1 proposed a bespoke `db-autoscaler` operator that *owned* the application's `replicas` value and enforced that ownership; an implementation spike proved the enforcement premise unbuildable on the aggregated apps API, and showed the whole conflict was self-imposed — it exists only because the chart unconditionally templates the replica field (full findings in the [Appendix](#appendix-findings-from-the-implementation-spike)). Rev2/rev3 therefore moved to a stock HPA on the engine's `scale` subresource with the chart omitting the field, keeping only a thin controller and CRD to render the HPA and drive a synthesized metric. Review then showed even that is unnecessary: the metric can be *queried* into existence rather than emitted per-pod, and once the query exists, KEDA renders and manages everything declaratively — so the controller and CRD are gone too. The guiding principle throughout: reuse the platform Kubernetes ships, do not reimplement it. The PoC below shows that last step reused one thing Kubernetes cannot actually provide here. +This design converged over three revisions, each removing machinery the previous one thought it needed. Rev1 proposed a bespoke `db-autoscaler` operator that *owned* the application's `replicas` value and enforced that ownership; an implementation spike proved the enforcement premise unbuildable on the aggregated apps API, and showed the whole conflict was self-imposed — it exists only because the chart unconditionally templates the replica field (full findings in the [Appendix](#appendix-findings-from-the-implementation-spike)). Rev2/rev3 therefore moved to a stock HPA on the engine's `scale` subresource with the chart omitting the field, keeping only a thin controller and CRD to render the HPA and drive a synthesized metric. Review then showed even that is unnecessary: the metric can be *queried* into existence rather than emitted per-pod, and once the query exists, KEDA renders and manages everything declaratively — so the controller and CRD are gone too. The guiding principle throughout: reuse the platform Kubernetes ships, do not reimplement it. The PoC below shows that last step relied on one thing the vendored CNPG version did not yet provide — since fixed upstream, leaving the mechanism unchanged behind a version floor. ## PoC finding (2026-08-10) — CNPG version floor for HPA actuation