Conversation
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #105 +/- ##
==========================================
+ Coverage 50.05% 53.97% +3.92%
==========================================
Files 4 4
Lines 975 1032 +57
==========================================
+ Hits 488 557 +69
+ Misses 473 449 -24
- Partials 14 26 +12 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
vishesh92
force-pushed
the
add-simulator-e2e-environment
branch
from
August 31, 2026 10:19
9e0a52b to
b3a11f1
Compare
vishesh92
force-pushed
the
add-simulator-e2e-environment
branch
from
September 1, 2026 04:54
b3a11f1 to
193dabc
Compare
Contributor
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 24 out of 25 changed files in this pull request and generated 4 comments.
Suppressed comments (1)
cloudstack_loadbalancer.go:487
- The netErr branch returns the wrong error variable (it returns err from the previous GetPublicIpAddressByID call). If the first call succeeded (err==nil) but GetNetworkByID fails, this currently returns a nil error and masks the failure.
network, _, netErr := cs.client.Network.GetNetworkByID(ip.Associatednetworkid, cloudstack.WithProject(cs.projectID))
if netErr != nil {
klog.Errorf("Failed to fetch the network for id: %v", ip.Associatednetworkid)
return "", err
}
vishesh92
force-pushed
the
add-simulator-e2e-environment
branch
from
September 1, 2026 07:26
193dabc to
f656173
Compare
vishesh92
force-pushed
the
add-simulator-e2e-environment
branch
from
September 1, 2026 09:22
f656173 to
98f7d4a
Compare
Contributor
There was a problem hiding this comment.
Warning
Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.
Pull request overview
Copilot reviewed 24 out of 25 changed files in this pull request and generated 5 comments.
vishesh92
force-pushed
the
add-simulator-e2e-environment
branch
from
September 1, 2026 09:59
98f7d4a to
0ac220a
Compare
Contributor
There was a problem hiding this comment.
Warning
Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.
Pull request overview
Copilot reviewed 26 out of 27 changed files in this pull request and generated 6 comments.
vishesh92
force-pushed
the
add-simulator-e2e-environment
branch
from
September 1, 2026 10:59
0ac220a to
80f3681
Compare
vishesh92
force-pushed
the
add-simulator-e2e-environment
branch
from
September 1, 2026 11:23
80f3681 to
f5f9f69
Compare
vishesh92
force-pushed
the
add-simulator-e2e-environment
branch
from
September 2, 2026 07:09
f5f9f69 to
13692b4
Compare
Contributor
There was a problem hiding this comment.
Warning
Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.
Pull request overview
Copilot reviewed 26 out of 27 changed files in this pull request and generated 2 comments.
vishesh92
marked this pull request as ready for review
September 2, 2026 07:53
vishesh92
force-pushed
the
add-simulator-e2e-environment
branch
from
September 7, 2026 12:48
13692b4 to
83e49a5
Compare
Contributor
There was a problem hiding this comment.
🟡 Changes recommended
There is at least one gofmt/formatting issue in a changed Go file (cloudstack_loadbalancer.go) that is likely to fail CI/lint and should be corrected before approval.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Review details
- Files reviewed: 26/27 changed files
- Comments generated: 1
- Review effort level: Lite
DaanHoogland
approved these changes
Sep 10, 2026
DaanHoogland
left a comment
DaanHoogland
left a comment
Contributor
There was a problem hiding this comment.
clgtm , no production code.
| branches: | ||
| - main | ||
| pull_request: | ||
| workflow_dispatch: |
Contributor
There was a problem hiding this comment.
should we've to still keep this, or can remove
Member
Author
There was a problem hiding this comment.
workflow_dispatch is to allow manual execution of the action. IMO, it's good to have
Contributor
There was a problem hiding this comment.
🔵 Needs a closer look
The harness teardown and simulator bring-up scripts have a couple of concrete operational/security issues (persisted cmk credentials; simulator version overrides can be ignored when reusing an existing container) that should be addressed before merge.
Review details
Suppressed comments (2)
Previously missed (2) — in code that hasn't changed since the last review.
hack/e2e/10-simulator-up.sh:52
- 10-simulator-up.sh reuses an existing container named $SIM_NAME without checking whether it was created from the currently requested $SIM_IMAGE/$SIM_TAG. If a developer changes SIM_TAG (or SIM_IMAGE) and reruns the script, they can silently keep running the old simulator version while thinking the override took effect. Consider comparing the existing container image to $SIM_IMAGE and recreating the container when they differ.
hack/e2e/99-down.sh:35 - 99-down.sh leaves behind hack/e2e/_out/cmk.ini (created by cmk_init) which contains CloudStack admin credentials. Since this script is documented as tearing down everything the harness created, it should remove cmk.ini as well (and optionally the artifacts directory) to avoid persisting credentials and large logs on disk.
- Files reviewed: 26/27 changed files
- Comments generated: 0 new
- Review effort level: Lite
Contributor
|
@vishesh92 can you check the below.
|
vishesh92
force-pushed
the
add-simulator-e2e-environment
branch
from
September 11, 2026 09:11
fe56d78 to
8f34e10
Compare
Contributor
There was a problem hiding this comment.
🟡 Changes recommended
Unresolved critical and moderate findings remain in load-balancer cleanup and e2e image rebuilding.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Review details
Suppressed comments (1)
cloudstack_loadbalancer.go:821
- The unusable-port/protocol branch in
deleteDuplicateRulereturns nil while leaving the CloudStack rule in place (lines 831-833), and this unconditional clear drops the only record of that rule. On the next reconcile or delete,getLoadBalancerkeeps the valid twin inlb.rules, so this duplicate is never retried and its rule/firewall/IP can remain orphaned. Preserve failed entries or return an error that causes deletion to retry instead of clearing them as successfully processed.
lb.duplicateRules = nil
- Files reviewed: 26/27 changed files
- Comments generated: 4
- Review effort level: Lite
Comment on lines
+835
to
+836
| if _, err := lb.deleteFirewallRule(lbRule.Publicipid, port, protocol); err != nil { | ||
| return err |
| } | ||
| if count == 0 { | ||
| return "", err | ||
| return "", fmt.Errorf("no public IP address found with ID %v", publicIpId) |
The repository had no way to exercise the CCM end to end. The only "run against real CloudStack" hook was configFromEnv() in cloudstack_test.go, which skips unless CS_API_URL and friends are set, and nothing set them. As a result EnsureLoadBalancer, UpdateLoadBalancer and EnsureLoadBalancerDeleted -- the three functions holding nearly all of the load balancer branching -- had no test coverage at all, and the README pointed at a Docker Hub image (cloudstack/simulator) that no longer exists. Add hack/e2e, which brings up a CloudStack simulator, deploys its advanced zone, mints admin API keys, creates a kind cluster and deploys CloudStack VMs matching its nodes, then runs the CCM against both. CloudStack calls go through cmk, so the scripts run the same commands the documentation tells you to run, and cmk's own async job handling removes any need to poll queryAsyncJobResult. docs/development.md walks through the same steps by hand so the environment is understandable rather than magic. Add a Go e2e suite under test/e2e covering load balancer lifecycle, node initialization, service annotations and the VPC/network ACL path. It is behind the e2e build tag, so it stays out of `make test` and `go build ./...`, and it needs no new module dependencies. Run all of it in CI as a matrix of the latest two Kubernetes minors against CloudStack 4.22.1.0 and 4.20.2.0. The CloudStack axis is not only version coverage: 4.22 and later update a load balancer rule's CIDR list in place while earlier releases delete and recreate the rule, so both branches are exercised. Cells run in parallel and share a single image build, and the simulator and kind node images are cached between runs, so the workflow costs about as much wall-clock as a single run. Fix several latent bugs in the load balancer path that the new suite exposed. Three call sites fetched CloudStack resources without the configured project: updateNetworkACL (the network, its ACL list and the ACL rule listing), getNetworkIDFromIPAddress (the public IP and its network), and the disassociation check in EnsureLoadBalancerDeleted. On a VPC owned by a project this made every LoadBalancer service fail with "error fetching Network with ID" and never get an ingress address, and it leaked the public IP on deletion. getNetworkIDFromIPAddress also reported a failed network lookup as success by returning the wrong error variable, and guarded on Networkid while looking up Associatednetworkid; either could hand the caller an empty network ID, which GetNetworkByID does not reject but looks up as an unfiltered network list, so it could resolve to an arbitrary network instead of failing. Fix a load balancer rule leak. CloudStack does not enforce unique rule names, but loadBalancer.rules is keyed by name, so a duplicate silently displaced its twin in the map and then survived EnsureLoadBalancerDeleted with no service left to reference it. Duplicates are now tracked separately and removed on both reconcile and delete, together with the firewall rule and public IP that only the duplicate used, leaving the network ACL rules it shares with the kept rule in place. The rule that survives is the one on the address the service is published on -- spec.loadBalancerIP when set, otherwise the IP already in status.loadBalancer.ingress -- so the sweep cannot delete the rule clients are pointing at. On the delete path a failed sweep is logged rather than returned, so a duplicate that cannot be removed leaks a rule instead of holding the service in Terminating for ever. Harden getManagementServerVersion, which sliced the version string to three components without checking its length. A management server reporting fewer than three, such as "4.22" or a bare "24" under CloudStack's new versioning scheme, panicked the controller at startup. Also add the local cloud-config, cmk-config and kube-config files to .gitignore. They hold live credentials and were previously untracked but not ignored. Fixes #4 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
vishesh92
force-pushed
the
add-simulator-e2e-environment
branch
from
September 11, 2026 11:32
8f34e10 to
b4ee398
Compare
Contributor
There was a problem hiding this comment.
🟡 Changes recommended
Unresolved issues remain in e2e caching, simulator defaults, cleanup behavior, and e2e test handling.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Review details
Suppressed comments (8)
Makefile:36
- Exporting this fallback URL globally changes the existing opt-in acceptance-test behavior: a developer who has
CS_API_KEYandCS_SECRET_KEYin the shell but intentionally has noCS_API_URLnow passes all three checks inconfigFromEnv()andmake testattemptshttp://localhost:8080instead of skipping. Keep the simulator default scoped to the e2e/CI invocation, or require an explicitly configured endpoint before enabling acceptance tests.
CS_API_URL ?= http://localhost:$(SIM_HOST_PORT)/client/api
README.md:58
getManagementServerVersionreturns the API error immediately whenlistManagementServersMetricsis unauthorized; it only producesno management servers foundafter a successful response withCount == 0. This text therefore gives the wrong symptom for the missing-role permission failure and can mislead users debugging startup.
The account must also be allowed to call `listManagementServersMetrics`, which the controller uses
on startup to determine the management server version. This is a root admin API and is **not**
included in the default `User` role; without it the controller exits immediately with
`no management servers found`.
cloudstack_loadbalancer.go:836
- When the duplicate has an invalid port or protocol,
getLoadBalancerhas already omitted it fromlb.rules, but this branch returns success anddeleteDuplicateRulesclearsduplicateRules. A delete can therefore remove the kept rule and report success while orphaning this duplicate and its public IP; later reconciles repeat the no-op. Keep the item in a cleanup set or route it through a safe direct rule-deletion/retry path instead of silently forgetting it.
if err != nil || protocol == LoadBalancerProtocolInvalid {
klog.Warningf("Leaving duplicate rule %v (%v) in place: unusable public port %q or protocol %q", lbRule.Name, lbRule.Id, lbRule.Publicport, lbRule.Protocol)
return nil
cloudstack_loadbalancer.go:438
- This still returns
sweepErrfromEnsureLoadBalancerDeleted, so any duplicate-cleanup failure is propagated to the service controller and can keep the Service inTerminating. That contradicts the PR's stated deletion behavior that sweep failures are logged rather than returned so deletion can complete. Please choose and document one contract: return nil after the primary resources are deleted, or retain retry semantics and update the PR description and test accordingly.
return sweepErr
docs/development.md:417
- The controller returns the API authorization error directly when this call is denied;
no management servers foundis reserved for a successful empty response. Please describe the permission failure as an authorization/startup error here so the troubleshooting entry matches the actual failure path.
| CCM exits with `no management servers found` | The account cannot call `listManagementServersMetrics`. This is a root-admin API; the default `User` role does not include it. |
docs/development.md:265
- This section says to skip step 4, but
cloud-config-hostis generated by40-ccm-deploy.shin step 4; following the documented manual sequence leaves the--cloud-configpath absent. The scale-down command is also shown only after the host process is launched, so a user can start two CCMs. Update the walkthrough to generate the host config and scale the in-cluster deployment down before starting the binary.
For interactive development and debugging, skip step 4 and run the binary
directly against the same environment:
test/e2e/annotations_test.go:171
PublicIPByAddressexplicitly returns(nil, nil)when the address is absent, and the regular delete test already treats that as a successful release. Returning(false, nil)forip == nilhere makes this test time out if CloudStack removes the released address from the listing instead of retaining a free record. Treat a nil record as released.
if err != nil || ip == nil {
return false, err
}
return ip.Allocated == "", nil
test/e2e/vpc_test.go:259
PublicIPByAddressexplicitly returns(nil, nil)when the address is absent, but this callback turns that valid released state intofalseand eventually times out. The non-VPC delete test handlesip == nilas success; this VPC regression test should do the same.
if err != nil || ip == nil {
return false, err
}
return ip.Allocated == "", nil
- Files reviewed: 26/27 changed files
- Comments generated: 3
- Review effort level: Lite
Member
Author
I have addressed these. |
kiranchavala
approved these changes
Sep 15, 2026
kiranchavala
left a comment
kiranchavala
left a comment
Member
There was a problem hiding this comment.
LGTM , tested manually
make e2e-up
make test-e2e # phase 1: load balancer, nodes, annotations
make e2e-vpc # switch to a VPC in a project
make test-e2e-vpc # phase 2: VPC / network ACL
make e2e-down
ubuntu@Kind-cluster:~/cloudstack-kubernetes-provider$ make e2e-up
hack/e2e/up.sh
[07:52:23] simulator container cloudstack-simulator already running, reusing it
[07:52:23] waiting up to 600s for: jetty answering http://localhost:8080/client/api
[07:52:23] ready after 0s: jetty answering http://localhost:8080/client/api
[07:52:23] waiting up to 300s for: CloudStack API accepting admin credentials
[07:52:23] ready after 0s: CloudStack API accepting admin credentials
[07:52:23] waiting up to 300s for: management server registered
[07:52:23] ready after 0s: management server registered
Loaded in-built API cache. Failed to read API cache, please run 'sync'.
[07:52:23] deploying zone Sandbox-simulator (this takes a few minutes)
==== Log Folder Path: /tmp/MarvinLogs/DeployDataCenter__Sep_15_2026_07_52_24_A7LYC0 All logs will be available here ====
==== Deploy DC Started ====
=== Data Center Settings are dumped to /tmp/MarvinLogs/DeployDataCenter__Sep_15_2026_07_52_24_A7LYC0/dc_entries.obj===
====Deploy DC Successful=====
[07:54:39] waiting up to 600s for: zone Sandbox-simulator enabled
[07:54:40] ready after 1s: zone Sandbox-simulator enabled
[07:54:40] waiting up to 300s for: at least one routing host up
[07:54:40] ready after 0s: at least one routing host up
Loaded in-built API cache. Failed to read API cache, please run 'sync'.
Loaded in-built API cache. Failed to read API cache, please run 'sync'.
[07:54:40] simulator ready; admin API keys written to /home/ubuntu/cloudstack-kubernetes-provider/hack/e2e/_out/keys.env
[07:54:40] creating kind cluster cs-ccm-e2e (image kindest/node:v1.37.0) on network cs-ccm-e2e
Creating cluster "cs-ccm-e2e" ...
WARNING: Overriding docker network due to KIND_EXPERIMENTAL_DOCKER_NETWORK
WARNING: Here be dragons! This is not supported currently.
✓ Ensuring node image (kindest/node:v1.37.0) 🖼️
✓ Preparing nodes 📦 📦 📦
✓ Writing configuration 📜
✓ Starting control-plane 🕹️
✓ Installing CNI 🔌
✓ Installing StorageClass 💾
✓ Joining worker nodes 🚜
✓ Waiting ≤ 3m0s for control-plane = Ready ⏳
• Ready after 9s 💚
Set kubectl context to "kind-cs-ccm-e2e"
You can now use your cluster with:
kubectl cluster-info --context kind-cs-ccm-e2e
Have a nice day! 👋
[07:55:21] node IPs:
cs-ccm-e2e-worker 172.30.0.4
cs-ccm-e2e-worker2 172.30.0.3
cs-ccm-e2e-control-plane 172.30.0.5
[07:55:21] kind cluster ready; kubeconfig at /home/ubuntu/cloudstack-kubernetes-provider/hack/e2e/_out/kubeconfig
Loaded in-built API cache. Failed to read API cache, please run 'sync'.
Loaded in-built API cache. Failed to read API cache, please run 'sync'.
Loaded in-built API cache. Failed to read API cache, please run 'sync'.
Loaded in-built API cache. Failed to read API cache, please run 'sync'.
[07:55:29] creating isolated network ccm-e2e-iso (172.30.0.0/24)
Loaded in-built API cache. Failed to read API cache, please run 'sync'.
Loaded in-built API cache. Failed to read API cache, please run 'sync'.
Loaded in-built API cache. Failed to read API cache, please run 'sync'.
Loaded in-built API cache. Failed to read API cache, please run 'sync'.
[07:55:31] deploying VM cs-ccm-e2e-worker with IP 172.30.0.4
Loaded in-built API cache. Failed to read API cache, please run 'sync'.
Loaded in-built API cache. Failed to read API cache, please run 'sync'.
[07:55:35] deploying VM cs-ccm-e2e-worker2 with IP 172.30.0.3
Loaded in-built API cache. Failed to read API cache, please run 'sync'.
Loaded in-built API cache. Failed to read API cache, please run 'sync'.
[07:55:40] deploying VM cs-ccm-e2e-control-plane with IP 172.30.0.5
Loaded in-built API cache. Failed to read API cache, please run 'sync'.
Loaded in-built API cache. Failed to read API cache, please run 'sync'.
Loaded in-built API cache. Failed to read API cache, please run 'sync'.
Loaded in-built API cache. Failed to read API cache, please run 'sync'.
[07:55:44] isolated topology ready (network e980e175-79a0-48e7-acd0-7a703920a01d)
[07:55:44] building apache/cloudstack-kubernetes-provider:e2e from /home/ubuntu/cloudstack-kubernetes-provider/hack/e2e/../..
[+] Building 154.3s (15/15) FINISHED docker:default
=> [internal] load build definition from Dockerfile 0.1s
=> => transferring dockerfile: 1.53kB 0.0s
=> [internal] load metadata for gcr.io/distroless/static:nonroot 3.4s
=> [internal] load metadata for docker.io/library/golang:1.23 2.3s
=> [internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
=> [builder 1/7] FROM docker.io/library/golang:1.23@sha256:60deed95d3888cc5e4d9ff8a10c54e5edc008c6ae3fba6187be6fb592e19e8c0 32.0s
=> => resolve docker.io/library/golang:1.23@sha256:60deed95d3888cc5e4d9ff8a10c54e5edc008c6ae3fba6187be6fb592e19e8c0 0.0s
=> => sha256:d5791340ef181f05ef1e99760e4be81cb2008d99b2bc1e56b2faa9b6295d0ac3 74.07MB / 74.07MB 27.8s
=> => sha256:01b3c4a0d5c8f9abfa7395525fc8c3e2d9b8d3fec47edaf0d9c58f6253797d34 125B / 125B 0.5s
=> => sha256:d64c83132511ab5226f18a66abc03228a24fffcaf1ab1b4e9c9cdb64d16ad877 92.38MB / 92.38MB 18.0s
=> => sha256:cffef7dc6f99e0837fd18f5ab2b363aff8d1c12ed377199f6d6478f80b458c05 64.40MB / 64.40MB 26.3s
=> => sha256:6d6401b7636bba3fe2d916b154ba44abe2081a737e117b2c736167ca6ea42334 24.02MB / 24.02MB 9.5s
=> => sha256:f014853ae2033c0e173500a9c5027c3ecffe2ffacbd09b789ac2f2dc63ddaa63 48.49MB / 48.49MB 17.8s
=> => extracting sha256:f014853ae2033c0e173500a9c5027c3ecffe2ffacbd09b789ac2f2dc63ddaa63 0.5s
=> => extracting sha256:6d6401b7636bba3fe2d916b154ba44abe2081a737e117b2c736167ca6ea42334 0.2s
=> => extracting sha256:cffef7dc6f99e0837fd18f5ab2b363aff8d1c12ed377199f6d6478f80b458c05 0.6s
=> => extracting sha256:d64c83132511ab5226f18a66abc03228a24fffcaf1ab1b4e9c9cdb64d16ad877 0.8s
=> => extracting sha256:d5791340ef181f05ef1e99760e4be81cb2008d99b2bc1e56b2faa9b6295d0ac3 1.2s
=> => extracting sha256:01b3c4a0d5c8f9abfa7395525fc8c3e2d9b8d3fec47edaf0d9c58f6253797d34 0.1s
=> => extracting sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1 0.1s
=> [internal] load build context 0.2s
=> => transferring context: 56.39MB 0.1s
=> [stage-1 1/2] FROM gcr.io/distroless/static:nonroot@sha256:e2e927ec666bae08560abb3c55d0659eceabb657f56b6782ab500a9fc7f555e3 6.7s
=> => resolve gcr.io/distroless/static:nonroot@sha256:e2e927ec666bae08560abb3c55d0659eceabb657f56b6782ab500a9fc7f555e3 0.0s
=> => sha256:c4bc6f35ff5e25aa1afe7a148a0b6f10a760f54e298f86a4a4d44b91f1afbfab 143.34kB / 143.34kB 2.7s
=> => sha256:ebddc55facdc6b1f7e0f30816a5fc7cc62f38abdf76c0a8b0a0ce52085754795 311B / 311B 0.6s
=> => sha256:b839dfae01f66e15c6a8b63520557ed315bdfe036342fa7a0c537259f10d7a9a 351B / 351B 0.6s
=> => sha256:dd64bf2dd177757451a98fcdc999a339c35dee5d9872d8f4dc69c8f3c4dd0112 80B / 80B 1.6s
=> => sha256:52630fc75a18675c530ed9eba5f55eca09b03e91bd5bc15307918bbc1a7e7296 162B / 162B 1.1s
=> => sha256:3214acf345c0cc6bbdb56b698a41ccdefc624a09d6beb0d38b5de0b2303ecaf4 123B / 123B 1.1s
=> => sha256:7c12895b777bcaa8ccae0605b4de635b68fc32d60fa08f421dc3818bf55ee212 188B / 188B 0.5s
=> => sha256:2780920e5dbfbe103d03a583ed75345306e572ec5a48cb10361f046767d9f29a 67B / 67B 0.5s
=> => sha256:d6b1b89eccacc15c2420b2776d72c1dae334a00805ed9af54bf2f71e4d536f28 32.09kB / 32.09kB 2.1s
=> => sha256:f6069939f718601b38d25ec5ed5631ed047e4da24acec2e252f8f29d251be718 254.67kB / 254.67kB 3.5s
=> => sha256:218cf840d0d95a86231eb9d33d26ac8a520a4299c6096cb70a0c2fb2a87c9e0a 288.06kB / 288.06kB 3.1s
=> => sha256:c172f21841dff4c8cf45cde46589c1c2616cefe7e819965e92e6d3475c428aa0 12.68kB / 12.68kB 0.8s
=> => sha256:2cc7ee286bf3a9e6af5f71756d7fc8e22e23ce65fec9047d774511ffcef79fa8 124.53kB / 124.53kB 2.0s
=> => extracting sha256:2cc7ee286bf3a9e6af5f71756d7fc8e22e23ce65fec9047d774511ffcef79fa8 0.0s
=> => extracting sha256:c172f21841dff4c8cf45cde46589c1c2616cefe7e819965e92e6d3475c428aa0 0.0s
=> => extracting sha256:218cf840d0d95a86231eb9d33d26ac8a520a4299c6096cb70a0c2fb2a87c9e0a 0.1s
=> => extracting sha256:f6069939f718601b38d25ec5ed5631ed047e4da24acec2e252f8f29d251be718 0.1s
=> => extracting sha256:d6b1b89eccacc15c2420b2776d72c1dae334a00805ed9af54bf2f71e4d536f28 0.0s
=> => extracting sha256:2780920e5dbfbe103d03a583ed75345306e572ec5a48cb10361f046767d9f29a 0.0s
=> => extracting sha256:7c12895b777bcaa8ccae0605b4de635b68fc32d60fa08f421dc3818bf55ee212 0.0s
=> => extracting sha256:3214acf345c0cc6bbdb56b698a41ccdefc624a09d6beb0d38b5de0b2303ecaf4 0.0s
=> => extracting sha256:52630fc75a18675c530ed9eba5f55eca09b03e91bd5bc15307918bbc1a7e7296 0.0s
=> => extracting sha256:dd64bf2dd177757451a98fcdc999a339c35dee5d9872d8f4dc69c8f3c4dd0112 0.0s
=> => extracting sha256:b839dfae01f66e15c6a8b63520557ed315bdfe036342fa7a0c537259f10d7a9a 0.0s
=> => extracting sha256:ebddc55facdc6b1f7e0f30816a5fc7cc62f38abdf76c0a8b0a0ce52085754795 0.0s
=> => extracting sha256:c4bc6f35ff5e25aa1afe7a148a0b6f10a760f54e298f86a4a4d44b91f1afbfab 0.0s
=> [builder 2/7] WORKDIR /go/src/github.com/apache/cloudstack-kubernetes-provider 14.6s
=> [builder 3/7] COPY go.mod /go/src/github.com/apache/cloudstack-kubernetes-provider/go.mod 0.1s
=> [builder 4/7] COPY go.sum /go/src/github.com/apache/cloudstack-kubernetes-provider/go.sum 0.1s
=> [builder 5/7] RUN go mod download 72.5s
=> [builder 6/7] COPY . /go/src/github.com/apache/cloudstack-kubernetes-provider 0.2s
=> [builder 7/7] RUN make clean && CGO_ENABLED=0 GOOS=linux GOARCH=amd64 make 26.7s
=> [stage-1 2/2] COPY --from=builder /go/src/github.com/apache/cloudstack-kubernetes-provider/cloudstack-ccm /app/cloudstack-ccm 1.2s
=> exporting to image 2.8s
=> => exporting layers 2.3s
=> => exporting manifest sha256:54060d2d98bc5464361e1b71af1330fc31f6b8f598aa7f64409e04d4d55a79ad 0.0s
=> => exporting config sha256:818869320c28f4a7201fe58c2b2a781961f1a73300f3c11e9fd5214370af0cd9 0.0s
=> => exporting attestation manifest sha256:98d20d2da441e91330fb1335c5da41211dd5dc9ab2448454d61d784135a69465 0.1s
=> => exporting manifest list sha256:96d3fd2b8b87e647447d0ef9de240779f5936bd6153f1fd63f1a7f5af13a9e93 0.0s
=> => naming to docker.io/apache/cloudstack-kubernetes-provider:e2e 0.0s
=> => unpacking to docker.io/apache/cloudstack-kubernetes-provider:e2e 0.2s
Image: "apache/cloudstack-kubernetes-provider:e2e" with ID "sha256:96d3fd2b8b87e647447d0ef9de240779f5936bd6153f1fd63f1a7f5af13a9e93" not yet present on node "cs-ccm-e2e-worker", loading...
Image: "apache/cloudstack-kubernetes-provider:e2e" with ID "sha256:96d3fd2b8b87e647447d0ef9de240779f5936bd6153f1fd63f1a7f5af13a9e93" not yet present on node "cs-ccm-e2e-worker2", loading...
Image: "apache/cloudstack-kubernetes-provider:e2e" with ID "sha256:96d3fd2b8b87e647447d0ef9de240779f5936bd6153f1fd63f1a7f5af13a9e93" not yet present on node "cs-ccm-e2e-control-plane", loading...
secret/cloudstack-secret created
serviceaccount/cloud-controller-manager created
clusterrole.rbac.authorization.k8s.io/system:cloud-controller-manager created
clusterrolebinding.rbac.authorization.k8s.io/system:cloud-controller-manager created
rolebinding.rbac.authorization.k8s.io/system:cloud-controller-manager:extension-apiserver-authentication-reader created
deployment.apps/cloud-controller-manager created
deployment.apps/cloud-controller-manager patched
Waiting for deployment spec update to be observed...
Waiting for deployment "cloud-controller-manager" rollout to finish: 0 out of 1 new replicas have been updated...
Waiting for deployment "cloud-controller-manager" rollout to finish: 1 old replicas are pending termination...
Waiting for deployment "cloud-controller-manager" rollout to finish: 1 old replicas are pending termination...
Waiting for deployment "cloud-controller-manager" rollout to finish: 1 old replicas are pending termination...
deployment "cloud-controller-manager" successfully rolled out
[07:58:27] waiting up to 300s for: all nodes initialized by the CCM
[07:58:32] ready after 5s: all nodes initialized by the CCM
[07:58:32] CCM deployed and all nodes initialized
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
cs-ccm-e2e-control-plane Ready control-plane 3m34s v1.37.0 172.30.0.5 <none> Debian GNU/Linux 13 (trixie) 6.8.0-139-generic (amd64) containerd://2.3.4
cs-ccm-e2e-worker Ready <none> 3m20s v1.37.0 172.30.0.4 <none> Debian GNU/Linux 13 (trixie) 6.8.0-139-generic (amd64) containerd://2.3.4
cs-ccm-e2e-worker2 Ready <none> 3m20s v1.37.0 172.30.0.3 <none> Debian GNU/Linux 13 (trixie) 6.8.0-139-generic (amd64) containerd://2.3.4
Environment is up. Try it:
export KUBECONFIG=hack/e2e/_out/kubeconfig
kubectl create deployment web --image=nginx
kubectl expose deployment web --port=80 --type=LoadBalancer
kubectl get svc web -w # EXTERNAL-IP appears from 192.168.2.0/24
Run the e2e suite: make test-e2e
Tear down: hack/e2e/99-down.sh
ubuntu@Kind-cluster:~/cloudstack-kubernetes-provider$ make test-e2e
. hack/e2e/_out/keys.env && \
KUBECONFIG=/home/ubuntu/cloudstack-kubernetes-provider/hack/e2e/_out/kubeconfig \
CS_API_URL=http://localhost:8080/client/api \
go test -tags e2e -v -timeout 30m ./test/e2e/... -run 'TestLB|TestNode|TestAnnot'
go: downloading github.com/apache/cloudstack-go/v2 v2.19.0
go: downloading k8s.io/api v0.24.17
go: downloading k8s.io/apimachinery v0.24.17
go: downloading k8s.io/client-go v0.24.17
go: downloading github.com/google/gofuzz v1.1.0
go: downloading k8s.io/klog/v2 v2.80.1
go: downloading sigs.k8s.io/structured-merge-diff/v4 v4.2.3
go: downloading k8s.io/utils v0.0.0-20221107191617-1a15be271d1d
go: downloading sigs.k8s.io/json v0.0.0-20220713155537-f223a00ba0e2
go: downloading go.uber.org/mock v0.5.0
go: downloading golang.org/x/net v0.38.0
go: downloading github.com/imdario/mergo v0.3.6
go: downloading github.com/spf13/pflag v1.0.5
go: downloading golang.org/x/term v0.30.0
go: downloading github.com/json-iterator/go v1.1.12
go: downloading github.com/google/gnostic v0.5.7-v3refs
go: downloading golang.org/x/time v0.3.0
go: downloading golang.org/x/sys v0.0.0-20220502124256-b6088ccd6cba
go: downloading google.golang.org/protobuf v1.34.1
go: downloading k8s.io/kube-openapi v0.0.0-20220328201542-3ee0da9b0b42
go: downloading golang.org/x/oauth2 v0.18.0
go: downloading sigs.k8s.io/yaml v1.3.0
go: downloading github.com/davecgh/go-spew v1.1.1
go: downloading github.com/modern-go/reflect2 v1.0.2
go: downloading github.com/go-logr/logr v1.4.1
go: downloading golang.org/x/text v0.23.0
go: downloading github.com/emicklei/go-restful v2.16.0+incompatible
go: downloading github.com/go-openapi/swag v0.19.14
go: downloading github.com/go-openapi/jsonreference v0.20.0
go: downloading github.com/go-openapi/jsonpointer v0.19.5
go: downloading github.com/mailru/easyjson v0.7.6
go: downloading github.com/josharian/intern v1.0.0
=== RUN TestAnnot_SourceCIDRs
--- PASS: TestAnnot_SourceCIDRs (12.10s)
=== RUN TestAnnot_Hostname
--- PASS: TestAnnot_Hostname (20.34s)
=== RUN TestAnnot_SessionAffinity
--- PASS: TestAnnot_SessionAffinity (12.12s)
=== RUN TestAnnot_ExplicitLoadBalancerIP
--- PASS: TestAnnot_ExplicitLoadBalancerIP (9.39s)
=== RUN TestLB_CreateSingleTCPPort
--- PASS: TestLB_CreateSingleTCPPort (9.07s)
=== RUN TestLB_MultiPort
--- PASS: TestLB_MultiPort (15.07s)
=== RUN TestLB_NodeMembership
--- PASS: TestLB_NodeMembership (9.11s)
=== RUN TestLB_PortChange
--- PASS: TestLB_PortChange (15.09s)
=== RUN TestLB_Delete
--- PASS: TestLB_Delete (9.10s)
=== RUN TestNode_Initialized
--- PASS: TestNode_Initialized (0.03s)
=== RUN TestNode_ProviderID
node_test.go:69: node cs-ccm-e2e-control-plane has a foreign provider ID "kind://docker/cs-ccm-e2e/cs-ccm-e2e-control-plane" (set by the infrastructure, not the CCM); expected CloudStack provider ID would be "external-cloudstack://479353e8-c548-4fde-8174-d2bc3bbb422b"
node_test.go:69: node cs-ccm-e2e-worker has a foreign provider ID "kind://docker/cs-ccm-e2e/cs-ccm-e2e-worker" (set by the infrastructure, not the CCM); expected CloudStack provider ID would be "external-cloudstack://b97948d4-b54c-473c-b9fb-7ffdb518edfe"
node_test.go:69: node cs-ccm-e2e-worker2 has a foreign provider ID "kind://docker/cs-ccm-e2e/cs-ccm-e2e-worker2" (set by the infrastructure, not the CCM); expected CloudStack provider ID would be "external-cloudstack://40ebbb0e-96b8-45f4-ba0c-c26e02b1b6cc"
node_test.go:80: every node has a provider ID assigned by the infrastructure; the CCM's provider ID assignment is not exercised by this environment
--- SKIP: TestNode_ProviderID (0.14s)
=== RUN TestNode_Labels
--- PASS: TestNode_Labels (0.10s)
=== RUN TestNode_InternalIP
--- PASS: TestNode_InternalIP (0.09s)
PASS
ok github.com/apache/cloudstack-kubernetes-provider/test/e2e 111.764s
ubuntu@Kind-cluster:~/cloudstack-kubernetes-provider$ make e2e-vpc
hack/e2e/50-topology-vpc.sh
Loaded in-built API cache. Failed to read API cache, please run 'sync'.
[08:45:34] creating project ccm-e2e-vpc
Loaded in-built API cache. Failed to read API cache, please run 'sync'.
Loaded in-built API cache. Failed to read API cache, please run 'sync'.
Loaded in-built API cache. Failed to read API cache, please run 'sync'.
[08:45:37] creating VPC ccm-e2e-vpc (172.30.0.0/22)
Loaded in-built API cache. Failed to read API cache, please run 'sync'.
Loaded in-built API cache. Failed to read API cache, please run 'sync'.
[08:45:40] creating ACL list ccm-e2e-acl
Loaded in-built API cache. Failed to read API cache, please run 'sync'.
Loaded in-built API cache. Failed to read API cache, please run 'sync'.
Loaded in-built API cache. Failed to read API cache, please run 'sync'.
[08:45:42] creating VPC tier ccm-e2e-tier (172.30.0.0/24)
Loaded in-built API cache. Failed to read API cache, please run 'sync'.
Loaded in-built API cache. Failed to read API cache, please run 'sync'.
[08:45:43] deploying project VM cs-ccm-e2e-worker with IP 172.30.0.4
Loaded in-built API cache. Failed to read API cache, please run 'sync'.
Loaded in-built API cache. Failed to read API cache, please run 'sync'.
[08:45:45] deploying project VM cs-ccm-e2e-worker2 with IP 172.30.0.3
Loaded in-built API cache. Failed to read API cache, please run 'sync'.
Loaded in-built API cache. Failed to read API cache, please run 'sync'.
[08:45:50] deploying project VM cs-ccm-e2e-control-plane with IP 172.30.0.5
Loaded in-built API cache. Failed to read API cache, please run 'sync'.
[08:45:53] building apache/cloudstack-kubernetes-provider:e2e from /home/ubuntu/cloudstack-kubernetes-provider/hack/e2e/../..
[+] Building 31.8s (15/15) FINISHED docker:default
=> [internal] load build definition from Dockerfile 0.0s
=> => transferring dockerfile: 1.53kB 0.0s
=> [internal] load metadata for gcr.io/distroless/static:nonroot 1.2s
=> [internal] load metadata for docker.io/library/golang:1.23 1.9s
=> [internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
=> [builder 1/7] FROM docker.io/library/golang:1.23@sha256:60deed95d3888cc5e4d9ff8a10c54e5edc008c6ae3fba6187be6fb592e19e8c0 0.1s
=> => resolve docker.io/library/golang:1.23@sha256:60deed95d3888cc5e4d9ff8a10c54e5edc008c6ae3fba6187be6fb592e19e8c0 0.0s
=> CACHED [stage-1 1/2] FROM gcr.io/distroless/static:nonroot@sha256:e2e927ec666bae08560abb3c55d0659eceabb657f56b6782ab500a9fc7f555e3 0.1s
=> => resolve gcr.io/distroless/static:nonroot@sha256:e2e927ec666bae08560abb3c55d0659eceabb657f56b6782ab500a9fc7f555e3 0.0s
=> [internal] load build context 0.0s
=> => transferring context: 6.29kB 0.0s
=> CACHED [builder 2/7] WORKDIR /go/src/github.com/apache/cloudstack-kubernetes-provider 0.0s
=> CACHED [builder 3/7] COPY go.mod /go/src/github.com/apache/cloudstack-kubernetes-provider/go.mod 0.0s
=> CACHED [builder 4/7] COPY go.sum /go/src/github.com/apache/cloudstack-kubernetes-provider/go.sum 0.0s
=> CACHED [builder 5/7] RUN go mod download 0.0s
=> [builder 6/7] COPY . /go/src/github.com/apache/cloudstack-kubernetes-provider 0.2s
=> [builder 7/7] RUN make clean && CGO_ENABLED=0 GOOS=linux GOARCH=amd64 make 25.9s
=> [stage-1 2/2] COPY --from=builder /go/src/github.com/apache/cloudstack-kubernetes-provider/cloudstack-ccm /app/cloudstack-ccm 0.9s
=> exporting to image 2.5s
=> => exporting layers 2.0s
=> => exporting manifest sha256:d996b036e5d9c74ffed9a3b67bcf7e849b5ed2c8d6a0453fa29b1b8a9c0a4182 0.0s
=> => exporting config sha256:f59c3715c5c88940323712b69b2959d4ddcacd0b27ae2b0ceb8df4a510224f80 0.0s
=> => exporting attestation manifest sha256:e5570d7575f668a616ed80cf693214717861b565738b05efcd69019ced49cbe0 0.1s
=> => exporting manifest list sha256:b4d727dbeeee21b662cb354accd1766661aaf6ac20c9e6e61970b99501913f27 0.0s
=> => naming to docker.io/apache/cloudstack-kubernetes-provider:e2e 0.0s
=> => unpacking to docker.io/apache/cloudstack-kubernetes-provider:e2e 0.2s
Image: "apache/cloudstack-kubernetes-provider:e2e" with ID "sha256:b4d727dbeeee21b662cb354accd1766661aaf6ac20c9e6e61970b99501913f27" not yet present on node "cs-ccm-e2e-worker", loading...
Image: "apache/cloudstack-kubernetes-provider:e2e" with ID "sha256:b4d727dbeeee21b662cb354accd1766661aaf6ac20c9e6e61970b99501913f27" not yet present on node "cs-ccm-e2e-worker2", loading...
Image: "apache/cloudstack-kubernetes-provider:e2e" with ID "sha256:b4d727dbeeee21b662cb354accd1766661aaf6ac20c9e6e61970b99501913f27" not yet present on node "cs-ccm-e2e-control-plane", loading...
secret/cloudstack-secret configured
serviceaccount/cloud-controller-manager unchanged
clusterrole.rbac.authorization.k8s.io/system:cloud-controller-manager unchanged
clusterrolebinding.rbac.authorization.k8s.io/system:cloud-controller-manager unchanged
rolebinding.rbac.authorization.k8s.io/system:cloud-controller-manager:extension-apiserver-authentication-reader unchanged
deployment.apps/cloud-controller-manager configured
deployment.apps/cloud-controller-manager patched
Waiting for deployment spec update to be observed...
Waiting for deployment "cloud-controller-manager" rollout to finish: 1 old replicas are pending termination...
Waiting for deployment "cloud-controller-manager" rollout to finish: 1 old replicas are pending termination...
deployment "cloud-controller-manager" successfully rolled out
[08:46:30] waiting up to 300s for: all nodes initialized by the CCM
[08:46:30] ready after 0s: all nodes initialized by the CCM
[08:46:30] CCM deployed and all nodes initialized
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
cs-ccm-e2e-control-plane Ready control-plane 51m v1.37.0 172.30.0.5 <none> Debian GNU/Linux 13 (trixie) 6.8.0-139-generic (amd64) containerd://2.3.4
cs-ccm-e2e-worker Ready <none> 51m v1.37.0 172.30.0.4 <none> Debian GNU/Linux 13 (trixie) 6.8.0-139-generic (amd64) containerd://2.3.4
cs-ccm-e2e-worker2 Ready <none> 51m v1.37.0 172.30.0.3 <none> Debian GNU/Linux 13 (trixie) 6.8.0-139-generic (amd64) containerd://2.3.4
deployment.apps/cloud-controller-manager restarted
Waiting for deployment "cloud-controller-manager" rollout to finish: 1 old replicas are pending termination...
Waiting for deployment "cloud-controller-manager" rollout to finish: 1 old replicas are pending termination...
deployment "cloud-controller-manager" successfully rolled out
[08:46:31] VPC topology ready (project 5b5f562e-b281-4ca1-bd95-8e676477c080, tier 73b8e2ce-67b5-4af2-9f45-0bba26648b46, acl d2e95ccc-2b52-48f5-bdc8-b86ddc278408)
ubuntu@Kind-cluster:~/cloudstack-kubernetes-provider$ make test-e2e-vpc
. hack/e2e/_out/keys.env && . hack/e2e/_out/ids.env && \
KUBECONFIG=/home/ubuntu/cloudstack-kubernetes-provider/hack/e2e/_out/kubeconfig \
CS_API_URL=http://localhost:8080/client/api \
CS_PROJECT_ID="$E2E_PROJECT_ID" \
E2E_ACL_ID="$E2E_ACL_ID" E2E_VPC_ID="$E2E_VPC_ID" \
go test -tags e2e -v -timeout 30m ./test/e2e/... -run 'TestVPC'
=== RUN TestVPC_LoadBalancer
--- PASS: TestVPC_LoadBalancer (9.11s)
=== RUN TestVPC_NodesReinitialized
--- PASS: TestVPC_NodesReinitialized (0.11s)
=== RUN TestVPC_ACLRuleNotDuplicatedOnResync
--- PASS: TestVPC_ACLRuleNotDuplicatedOnResync (12.09s)
=== RUN TestVPC_ExplicitLoadBalancerIPReleased
--- PASS: TestVPC_ExplicitLoadBalancerIPReleased (9.43s)
PASS
ok github.com/apache/cloudstack-kubernetes-provider/test/e2e 30.759s
ubuntu@Kind-cluster:~/cloudstack-kubernetes-provider$ make e2e-down
hack/e2e/99-down.sh
[09:11:24] deleting kind cluster cs-ccm-e2e
[09:11:25] removing simulator container cloudstack-simulator
cloudstack-simulator
[09:11:25] removing docker network cs-ccm-e2e
cs-ccm-e2e
[09:11:25] done
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #4, which asked for documentation on running kube-apiserver and
cloudstack-simulator locally so automated test workloads could follow.
Today nothing exercises the CCM end to end.
configFromEnv()skips unlessCS_API_URLand friends are set and nothing sets them, soEnsureLoadBalancer,UpdateLoadBalancerandEnsureLoadBalancerDeletedhave no coverage at all. TheREADME also links a Docker Hub image that no longer exists.
What this adds
A one-command local environment under
hack/e2e: CloudStack simulator,advanced zone, admin API keys, a kind cluster, CloudStack VMs matching the kind
nodes, and the CCM deployed from
deployment.yaml. Every CloudStack call is aplain
cmkinvocation, so the scripts run the same commands the docs tell you torun.
docs/development.md, which walks the same steps by hand: what each port isfor, why
cloud-provider: externalis needed, the node-name and node-IP contractbetween kind and CloudStack, both CCM run modes, the VPC scenario, and a
symptom-to-cause troubleshooting table.
A Go e2e suite in
test/e2e, 17 tests across load balancer lifecycle, nodeinitialization, annotations and session affinity, and the VPC/network ACL path.
It sits behind the
e2ebuild tag, somake testandgo build ./...areunaffected, and it needs no new module dependencies.
Using it
Bugs this surfaced, fixed here
project. On a project-owned VPC every LoadBalancer Service failed with
error fetching Network with IDand never got an ingress address, and the public IPleaked on delete.
getNetworkIDFromIPAddress. It returned the wrong error variable after afailed network lookup, guarded on
Networkidwhile looking upAssociatednetworkid, and could return an empty ID with a nil error.GetNetworkByIDdoes not reject an empty ID, it matches an unfiltered networklist, so that resolved to an arbitrary network instead of failing.
names, but
loadBalancer.rulesis keyed by name, so a duplicate displaced itstwin and then survived deletion with no Service referencing it. Duplicates are
now swept on reconcile and on delete, along with the firewall rule and public
IP only they used. The rule that survives is the one on the published address,
and a sweep that fails during deletion is logged rather than returned so a
Service cannot stick in
Terminating.getManagementServerVersionsliced the version string tothree components unchecked, so a server reporting
4.22, or a bare24underCloudStack's new versioning scheme, crashed the controller at startup.
CI
.github/workflows/e2e-simulator.ymlruns on pull requests, pushes tomain,and
workflow_dispatch. The matrix is the latest two Kubernetes minors againstCloudStack
4.22.1.0and4.20.2.0. The CloudStack axis is real branchcoverage, not just version coverage: 4.22 and later update a rule's CIDR list in
place, earlier releases delete and recreate it. Cells run in parallel behind one
shared image build, with the simulator and kind node images cached, so wall-clock
is close to a single run. The job needs no secrets.
Notes for reviewers
listManagementServersMetrics, which is a hard startup gate. Noted in theREADME.
exercised and that test skips under kind.
rule delete succeeds and the release then fails, the duplicate sweep's in-use
check counts only load balancer rules, and
UpdateLoadBalancerdoes not sweep..asf.yamluntil the job proves stable.