feat(ci): report clusteralerts fired during release rollover - #2686
Draft
universal-itengineer wants to merge 23 commits into
Draft
feat(ci): report clusteralerts fired during release rollover#2686universal-itengineer wants to merge 23 commits into
universal-itengineer wants to merge 23 commits into
Conversation
The release e2e pipeline had no visibility into alerts of the module itself: nothing in the repo ever looked at ClusterAlert objects, and the rules in monitoring/prometheus-rules were only covered by promtool unit tests. A dedicated job now watches the nested cluster for the whole test and upgrade sequence and collects every firing D8Virtualization* alert. A ClusterAlert object exists only while the alert fires, so the watch polls instead of taking a single snapshot at the end. It stops on a ConfigMap marker placed in the nested cluster by a separate always-running job: runners share no filesystem, and the marker must also appear when test-new-release never started. The report job renders a table into the job summary, emits a warning annotation per alert and exits non-zero, which paints it red in the UI. Being continue-on-error, it leaves the workflow conclusion successful. Alerts are attributed to pre-upgrade, upgrade or post-upgrade by comparing the observation time with the upgrade window, so an alert that was already firing before the rollover is not mistaken for its result. Signed-off-by: Nikita Korolev <nikita.korolev@flant.com>
The report job only depended on the upgrade job and on the watch, so in the run graph it sat next to the new-release tests instead of after them. Worse, if the watch job died early - on the kubeconfig step, say - the report started while the new-release tests were still running and declared that no alerts had been firing. It now depends on the test jobs as well, and an empty result is reported as "not monitored" instead of "nothing was firing" when the watch job did not succeed. Signed-off-by: Nikita Korolev <nikita.korolev@flant.com>
Pin the release-upgrade defaults to the versions under test and delete the nested cluster right after the ClusterAlerts report, so a test run does not hold the cluster until the nightly cleanup. Drop this commit before merge. Signed-off-by: Nikita Korolev <nikita.korolev@flant.com>
universal-itengineer
force-pushed
the
chore/ci/check-clusteralerts-during-overroling-release
branch
from
August 10, 2026 18:52
d647fce to
3eeef5f
Compare
The release e2e config hardcoded a feature gate that the 1.9 release line never shipped, so ModulePullOverride validation failed and the module was never installed: the pipeline only reported that it timed out waiting for virtualization to become ready. Gates are now derived from the release under test. The upgrade narrows the list to what both releases support before switching the image tag, then enables everything the new release supports once its images are running, so each release is tested with all the gates it has. Signed-off-by: Nikita Korolev <nikita.korolev@flant.com>
The release pipeline waited for one Evict operation per running virtual machine after every upgrade, and failed after 20 minutes when none appeared. Two releases that ship the same virt-handler and virt-launcher never move a virtual machine, so the wait could not be satisfied and reported a timeout instead of an upgrade that simply had nothing to migrate. The expectation is now derived from the workload image digests of both releases: unchanged images end the step with a message, and anything that cannot be determined keeps the previous waiting behaviour. Signed-off-by: Nikita Korolev <nikita.korolev@flant.com>
Runners force these actions onto Node.js 24 already and annotate every run with a deprecation warning. Moved each one to its first major built for Node.js 24: cache v4 to v5, setup-go v5 to v6, github-script v7 to v8, setup-kubectl and k8s-set-context v4 to v5. Signed-off-by: Nikita Korolev <nikita.korolev@flant.com>
setup-go v6 pins GOTOOLCHAIN to the requested version instead of letting Go fetch a newer one, and the pipelines asked for a Go older than the go.work of the sources they test, so every Go build failed. Both the release and the nightly pipelines now ask for the version the project builds with, and keep the toolchain automatic for refs that need another one. Signed-off-by: Nikita Korolev <nikita.korolev@flant.com>
Signed-off-by: Nikita Korolev <nikita.korolev@flant.com>
…ate vms The release smoke test compares the iperf window against the migration of the iperf server, and failed when no migration operation existed at all. Releases that ship the same virt-handler and virt-launcher never move a virtual machine, so the pipeline now passes its own verdict to the tests and the comparison is skipped when there is nothing to compare against. An unset value still expects a migration, so a missing one keeps failing the test. Signed-off-by: Nikita Korolev <nikita.korolev@flant.com>
The watch deleted the stop marker on start to ignore leftovers from earlier runs, which could delete the marker another job had just created for it: the watch then kept polling until its own timeout hours later, with the report job waiting on it. The marker name now carries the run and attempt, so foreign markers are invisible and none has to be deleted. Signed-off-by: Nikita Korolev <nikita.korolev@flant.com>
The report shows a fixed set of fields, and the ClusterAlert itself disappears from the cluster as soon as the alert stops firing - together with the cluster the pipeline deletes. Every matching alert is now also archived as the whole object in YAML, as first seen, next to the collected log. Signed-off-by: Nikita Korolev <nikita.korolev@flant.com>
Watching the ClusterAlerts objects took a job that lived as long as the whole pipeline, a stop marker in the nested cluster to end it, and an artifact round trip to hand the collected log to the report - because a ClusterAlert exists only while its alert is active. The ALERTS series keep that history, so one range query at the end of the pipeline replaces all of it. The phase now comes from the intersection of the firing interval with the upgrade window instead of the moment a poll happened to catch the alert, and pending alerts are reported as well: with a 'for' clause they are what a short rollover produces. Only a fired alert paints the job red, or components restarting would make that the norm. The rendered summary and description are kept: their templates come from the alerting rules of the same Prometheus and the labels of the series render them. Signed-off-by: Nikita Korolev <nikita.korolev@flant.com>
Collecting in the same job that reports means the job now needs the nested cluster, and a pipeline that died before the cluster existed painted it red - which reads as a firing alert. Neither the kubeconfig setup nor the query is allowed to fail the job any more; the report step turns both into the "not checked" verdict it already had. Signed-off-by: Nikita Korolev <nikita.korolev@flant.com>
Signed-off-by: Nikita Korolev <nikita.korolev@flant.com>
The templates of every alerting rule of a cluster add up to more than the 128 KiB Linux allows for a single command line argument, so passing them with --argjson failed the collection with E2BIG on the runner while it worked on a developer machine with a larger limit. They go through a file now. Signed-off-by: Nikita Korolev <nikita.korolev@flant.com>
The GPU gate was missing from the gates an e2e run enables, so nothing covered it. It ships in no release yet, and a gate the pulled module does not know fails ModulePullOverride validation, so the gates now come from a list that carries the release each one shipped in - an empty version meaning only a build off main or a pull request has it. Adding the next gate is one line. Signed-off-by: Nikita Korolev <nikita.korolev@flant.com>
Alert annotations are Go templates, and only $labels references were substituted, so a description built on $value - node evacuation stuck, outdated firmware - reached the report with the raw template in it. The sample value is not carried by the ALERTS series, so whatever template action survives the label substitution now becomes a "?" placeholder. Drop the error branches that set -e made unreachable: the jsonpath of an empty item list and curl --fail both abort the script before the check meant to explain the failure, so the range query now reports its own failure instead. Drop the environment knobs nothing sets, the FAIL_ON_ALERTS switch a continue-on-error job has no use for, and the trap - ERR that never fires on an explicit exit. Signed-off-by: Nikita Korolev <nikita.korolev@flant.com>
Identical virt-handler and virt-launcher digests were described as a guarantee that no virtual machine moves. They are not: a template-only change - new args, env, resources or tolerations - rolls the virt-handler DaemonSet at unchanged digests and migrates virtual machines all the same. Say so where the comparison is made, and note that a wrong verdict is not silent, since test-new-release then fails on an Evict VMOP still InProgress. Signed-off-by: Nikita Korolev <nikita.korolev@flant.com>
Both branches of the post-upgrade check asserted the same two totals, which do not depend on whether the upgrade migrates virtual machines. Assert them before the branch instead. Signed-off-by: Nikita Korolev <nikita.korolev@flant.com>
…bundle The gates an e2e run enables came from a hand-kept table of the release each gate first shipped in, which had to be updated by hand and, for a gate no release carries yet, was never enabled on a release tag at all. The nightly pipeline carried a second, plain hardcoded copy of the list. Every module bundle ships its own openapi/config-values.yaml, and the enum in it is the very schema the ModuleConfig is validated against, so each release now states its own gate list and no table is needed. The list of releases is intersected for the step that runs before the image tag is switched. A gate the live cluster refuses - locked in this edition, or needing a newer Kubernetes - is now dropped with a warning instead of failing the run: the point of the release e2e is the upgrade, not the gate. Since the moduleconfig webhook declares sideEffects: None, a server-side dry run answers whether a gate would be admitted without writing anything, and the dropped gates land in the job summary so the lost coverage stays visible. That webhook is served by virtualization-controller itself and has no failurePolicy, so a patch right after an image switch is rejected for reasons that say nothing about the gate. Get an answer out of the webhook first and treat lasting silence as the defect it is. A patch that goes through re-renders the module, so wait for it to settle before the next job starts. Signed-off-by: Nikita Korolev <nikita.korolev@flant.com>
The gates were rendered into the ModuleConfig at creation time, where nothing validates them: the module webhook guards updates only, and the controller that serves it is not running yet. A gate that lands in a fresh config is therefore never checked, and the per-gate dry run cannot drop it afterwards either - by then it is not a gate being added any more. A gate this edition locks makes the controller exit on start, and since it serves that very webhook, the config can no longer be repaired. Create the config without the key and let patch-virtualization-feature-gates.sh add the gates once the module is Ready. Every gate then goes through the webhook as an addition, install and upgrade share one code path, and the enum-to-YAML rendering leaves both install scripts. The guard against an unreadable bundle moves into virtualization_feature_gates, where an empty enum for one release is told apart from a legitimately empty intersection of two. In the nightly pipeline maintenance mode becomes a step of its own after the gate patch: it stops module reconciliation, so gates applied later would never reach the rendered resources. Signed-off-by: Nikita Korolev <nikita.korolev@flant.com>
curl -f throws the response body away, and Prometheus puts the reason a query was rejected exactly there, so a failed collection reported nothing but the script's own guess at what had happened. --fail-with-body keeps the non-zero exit code and the body both, which also makes the status check in fetch_rules meaningful again instead of unreachable behind the same flag. Sample the ALERTS series every 10 seconds instead of 30. The series exists only while its alert is active, so an alert shorter than the step can fall between two samples and be reported as no alert at all - the single alert that fired during the last run lasted two samples at the old step. A rollover window holds a few hundred points at the new one, far below the 11000 Prometheus allows per series. Say what actually makes the module config probe a reachability check: the webhook skips its whole validator chain when the generation does not change, so a no-op patch can fail on nothing but being unable to reach it. Record the other side of that predicate too - a real gate patch runs every validator against the whole config and can be refused over CIDRs or storage classes, with a message that says nothing about gates. Signed-off-by: Nikita Korolev <nikita.korolev@flant.com>
The report listed pending alerts as well, and a rollover puts almost every rule with a `for` clause into pending for a moment, so the alerts that actually fired were buried among a dozen that never did. Collect alertstate="firing" only - the same set a cluster's ClusterAlerts hold, since such an object exists only while its alert fires. Filtering in the query and not in the report keeps the whole pipeline free of records nobody wants. With pending gone the state is a constant, so it leaves the record, the dedup key, the table column and the notice-versus-warning branch: every alert that reaches the report now deserves the warning and the red job. Signed-off-by: Nikita Korolev <nikita.korolev@flant.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The release e2e pipeline had no visibility into alerts of the module itself: nothing in the repo ever looked at ClusterAlert objects, and the rules in monitoring/prometheus-rules were only covered by promtool unit tests.
A dedicated job now watches the nested cluster for the whole test and upgrade sequence and collects every firing D8Virtualization* alert. A ClusterAlert object exists only while the alert fires, so the watch polls instead of taking a single snapshot at the end. It stops on a ConfigMap marker placed in the nested cluster by a separate always-running job: runners share no filesystem, and the marker must also appear when test-new-release never started.
The report job renders a table into the job summary, emits a warning annotation per alert and exits non-zero, which paints it red in the UI. Being continue-on-error, it leaves the workflow conclusion successful. Alerts are attributed to pre-upgrade, upgrade or post-upgrade by comparing the observation time with the upgrade window, so an alert that was already firing before the rollover is not mistaken for its result.
Checklist
Changelog entries