Skip to content

development.md

Latest commit

 

History

History
424 lines (333 loc) · 18 KB

development.md

File metadata and controls

424 lines (333 loc) · 18 KB
Raw

Development

This document describes how to run a fully simulated development and test environment for the CloudStack Kubernetes Provider: a real Kubernetes API server, a real CloudStack management server and the CCM itself, all in containers on your workstation. Nothing is mocked — the CCM makes genuine CloudStack API calls and the resulting load balancer rules, public IPs and firewall/ACL rules are real database objects you can inspect.

The pieces are:

Component What provides it
Kubernetes API server + kubelets a kind cluster
CloudStack management server the apache/cloudstack-simulator container
Cloud controller manager this repository, either in-cluster or as a host process

Prerequisites

  • Docker
  • kind v0.30 or later
  • kubectl
  • Go 1.23 or later
  • cmk (CloudMonkey), the CloudStack CLI — the harness drives the CloudStack API through it
  • jq and curl
  • About 12 GB of free disk and 8 GB of RAM

linux/amd64 only. apache/cloudstack-simulator is not published for arm64. On Apple Silicon you can run it under emulation with --platform linux/amd64 (expect the simulator to take three to five times longer to start), point the environment at a simulator running on an x86 host, or build an arm64 image yourself from tools/docker/ in the apache/cloudstack repository.

Quickstart

make e2e-up          # simulator + zone + kind cluster + CloudStack VMs + CCM
make test-e2e        # phase 1: load balancer, nodes, annotations
make e2e-vpc         # switch the environment to a VPC in a project
make test-e2e-vpc    # phase 2: VPC / network ACL
make e2e-down        # tear everything down

Phase 2 builds on phase 1, so run them in that order; both test targets stop with a clear message if the environment they need is not up.

make e2e-up takes about seven minutes once the simulator image is pulled — roughly 90 seconds for the simulator to start, two and a half minutes to deploy the zone, and the rest for the kind cluster, the VMs and the CCM. The first run also has to pull a ~2 GB image.

Once it is up, try the thing the CCM exists for:

export KUBECONFIG=hack/e2e/_out/kubeconfig
kubectl create deployment web --image=nginx
kubectl expose deployment web --port=80 --type=LoadBalancer
kubectl get svc web -w

The service gets an EXTERNAL-IP from the simulator's public IP range (192.168.2.0/24), and the corresponding rule shows up in CloudStack:

cmk -c hack/e2e/_out/cmk.ini listLoadBalancerRules listall=true

The CloudStack UI is also available: run the simulator with -p 8081:5050 and open http://localhost:8081/, logging in as admin / password.

How the harness talks to CloudStack

The scripts call cmk directly, always in the form

cmk -c hack/e2e/_out/cmk.ini <command> [key=value ...]

so every CloudStack command in the harness is one you can paste into a shell. The config file is generated by the harness rather than read from ~/.cmk/config, so your own cmk profiles are left alone. (cmk takes its config path only from -c or $HOME, with no environment variable for it, which is why the flag is repeated rather than hidden behind a wrapper.)

The generated profile authenticates with admin/password rather than API keys, because the harness has to talk to CloudStack before any keys exist — it is what mints them. Two cmk defaults do real work here:

  • asyncblock = true — cmk waits for async jobs such as deployVirtualMachine and returns the finished result, so nothing in the harness polls queryAsyncJobResult.
  • output = json — responses come back without the <command>response envelope, so a zone list is .zone[0].id, not .listzonesresponse.zone[0].id. Worth knowing if you compare the scripts against raw API output.

What the scripts do

hack/e2e/up.sh chains four numbered scripts. Each is independently runnable and safe to re-run. All tunables live in hack/e2e/env.sh and can be overridden from the environment.

1. 10-simulator-up.sh — simulator and zone

Creates a docker bridge network (cs-ccm-e2e, 172.30.0.0/24) that both the simulator and the kind nodes will join, then starts the simulator on it:

docker network create --subnet 172.30.0.0/24 cs-ccm-e2e
docker run -d --name cloudstack-simulator --network cs-ccm-e2e \
    -p 127.0.0.1:8080:8080 apache/cloudstack-simulator:4.22.1.0

The image exposes three ports and it matters which one you use:

Port What it is
8080 the management server API (/client/api) — use this one
8096 the unauthenticated integration API, used by marvin
5050 the Vue UI development server, which proxies to 8080

The upstream simulator README suggests -p 8080:5050, which publishes the UI. For API access, publish container port 8080 directly.

Readiness is checked in three stages rather than with a fixed sleep: jetty answering at all, then the API accepting admin credentials, then listManagementServersMetrics returning a server. The last one matters because the CCM makes exactly that call on startup and refuses to run until it succeeds.

The zone is then deployed with marvin, which is preinstalled in the image:

docker exec cloudstack-simulator python3 \
    /root/tools/marvin/marvin/deployDataCenter.py -i /root/setup/dev/advanced.cfg

This creates the Sandbox-simulator advanced zone with a public IP range of 192.168.2.2–192.168.2.200.

Finally the script mints admin API keys for the CCM:

cmk -c hack/e2e/_out/cmk.ini listUsers username=admin      # -> the user id
cmk -c hack/e2e/_out/cmk.ini getUserKeys id=<user id>      # -> apikey, secretkey

listUsers is not a substitute for getUserKeys — it returns the API key but never the secret. registerUserKeys is used only when no key pair exists yet, because it rotates the keys, which would break a simulator you are reusing.

Keys land in hack/e2e/_out/keys.env.

2. 20-kind-up.sh — the Kubernetes cluster

KIND_EXPERIMENTAL_DOCKER_NETWORK=cs-ccm-e2e kind create cluster \
    --name cs-ccm-e2e --config hack/e2e/kind-config.yaml

The cluster config does two important things:

  • cloud-provider: external in every node's kubeletExtraArgs, so nodes register with the node.cloudprovider.kubernetes.io/uninitialized taint. Removing that taint is the CCM's job, and is how you know it works.
  • kubelet-preferred-address-types: InternalIP on the API server. Once the CCM initializes a node it sets the node's Hostname address to the CloudStack instance's hostname, which for the simulator is the simulated hypervisor agent and is not resolvable. Without this setting, kubectl logs and kubectl exec stop working after node initialization.

The cluster is named so that node names are deterministic: cs-ccm-e2e-control-plane, cs-ccm-e2e-worker, cs-ccm-e2e-worker2. Two workers exist so tests can check that the control plane node — which kubeadm labels node.kubernetes.io/exclude-from-external-load-balancers — is left out of load balancer membership.

The script records each node's IP on the shared docker network into hack/e2e/_out/node-ips. The next step depends on it.

CoreDNS stays Pending until the CCM removes the uninitialized taint. That is expected; don't wait for it.

3. 30-topology-isolated.sh — matching CloudStack VMs

This is the part that makes or breaks the environment. The CCM looks up each Kubernetes node by name in CloudStack, so a VM must exist whose name exactly matches the node name. On top of that, kind starts kubelet with --node-ip=<the node's docker IP>, and the CCM's node controller refuses to initialize a node whose kubelet-reported IP is not among the addresses the cloud provider reports for it. So the VMs must also carry the same IP addresses as the kind node containers.

The script therefore aligns the zone's guest CIDR with the docker subnet, creates an isolated network on it, and deploys one VM per node pinned to that node's IP:

cmk updateZone id=$ZONE guestcidraddress=172.30.0.0/24
cmk createNetwork name=ccm-e2e-iso networkofferingid=$OFFERING \
    gateway=172.30.0.1 netmask=255.255.255.0 zoneid=$ZONE
cmk deployVirtualMachine name=cs-ccm-e2e-worker displayname=cs-ccm-e2e-worker \
    ipaddress=172.30.0.4 networkids=$NET ...

The offering used is DefaultIsolatedNetworkOfferingWithSourceNatService, which provides the Firewall service — so on this network the CCM manages firewall rules. (The VPC scenario below uses an offering that provides NetworkACL instead, exercising the other branch.)

All offerings and templates are looked up by name, because their UUIDs differ between simulator deployments.

4. 40-ccm-deploy.sh — the controller

Generates two cloud-config files that differ only in api-url:

  • hack/e2e/_out/cloud-config — used by the in-cluster deployment, pointing at the simulator's IP on the cs-ccm-e2e docker network.
  • hack/e2e/_out/cloud-config-host — used when you run the CCM as a host process, pointing at http://localhost:8080/client/api.

The in-cluster config must use the simulator's IP address, not its container name or network alias: pods have their own network namespace and cannot reach Docker's embedded DNS resolver, and host.docker.internal does not exist on Linux Docker Engine.

Both configs set zone explicitly. If zone is empty the CCM tries to detect it by looking up its own pod, which cannot work when running as a host process.

The script then loads the image into kind, applies the repository's deployment.yaml and patches it for testing: the local image with imagePullPolicy: Never, --leader-elect=false (single replica, faster startup), --v=4 for useful logs, and higher CPU limits — the stock manifest's limits.cpu: 50m throttles informer startup badly on shared CI runners.

Finally it waits for every node to lose the uninitialized taint.

Running the CCM as a host process

For interactive development and debugging, skip step 4 and run the binary directly against the same environment:

make
./cloudstack-ccm \
    --cloud-provider=external-cloudstack \
    --cloud-config=hack/e2e/_out/cloud-config-host \
    --kubeconfig=hack/e2e/_out/kubeconfig \
    --leader-elect=false \
    --v=4

If the in-cluster CCM is already running, scale it down first so the two do not fight over the same services:

kubectl -n kube-system scale deployment/cloud-controller-manager --replicas=0

Debugging

You can use the VS Code extension Go to debug the CCM. Add the following to .vscode/launch.json:

{
    "version": "0.2.0",
    "configurations": [
        {
            "name": "Launch CloudStack CCM",
            "type": "go",
            "request": "launch",
            "mode": "auto",
            "program": "${workspaceFolder}/cmd/cloudstack-ccm",
            "env": {},
            "args": [
                "--cloud-provider=external-cloudstack",
                "--cloud-config=${workspaceFolder}/hack/e2e/_out/cloud-config-host",
                "--kubeconfig=${workspaceFolder}/hack/e2e/_out/kubeconfig",
                "--leader-elect=false",
                "--v=4"
            ],
            "showLog": true,
            "trace": "verbose"
        },
        {
            "name": "Attach to Process",
            "type": "go",
            "request": "attach",
            "mode": "local",
            "processId": 0
        }
    ]
}

To debug against a real CloudStack installation instead of the simulator, point --cloud-config at your own cloud-config and --kubeconfig at your cluster's kubeconfig.

The VPC scenario

make e2e-vpc (that is, hack/e2e/50-topology-vpc.sh) switches the environment to a VPC network so the Network ACL code path can be exercised. It creates a VPC, a custom ACL list, a tier network and per-node VMs, then re-points the CCM at them and restarts it.

Two details are worth knowing:

  • The ACL list must be a custom one. The CCM deliberately refuses to add rules to the built-in default_allow and default_deny lists.
  • Everything is created inside a CloudStack project. The CCM matches VM names across the whole account, and fails with found hosts that belong to different networks if the matched VMs are spread over several networks. Because CloudStack hides project resources from non-project queries and vice versa, putting the VPC VMs in a project makes the two scenarios mutually invisible without needing a second cluster or a second account.

The tier reuses the same subnet as the isolated network, so the VMs keep the same IP addresses and node initialization continues to work after the switch.

Running the tests

Unit tests need nothing but Go:

make test

The end-to-end suite needs the environment above. It is behind the e2e build tag, so it never runs as part of make test or go build ./...:

make test-e2e        # phase 1
make test-e2e-vpc    # phase 2, after `make e2e-vpc`

Configuration comes from the environment, using the same variable names as the existing opt-in acceptance tests in cloudstack_test.go. The make targets set these for you from hack/e2e/_out/; the table matters if you invoke go test directly:

Variable Meaning
KUBECONFIG cluster under test
CS_API_URL CloudStack API endpoint as reachable from the test process
CS_API_KEY, CS_SECRET_KEY CloudStack credentials
CS_PROJECT_ID optional; set during the VPC phase

When any of them is missing the tests skip rather than fail. The same suite runs against a real CloudStack installation — just point the variables at it.

Each test creates its own namespace and cleans up after itself. Because load balancer provisioning is asynchronous, all assertions poll rather than assuming immediate consistency.

Known limitation: provider IDs

kind starts kubelet with --provider-id=kind://docker/<cluster>/<node>, and Kubernetes only allows a node's provider ID to be set once. In this environment the CCM therefore never assigns the external-cloudstack://<instance UUID> provider ID it would set on a real cluster. TestNode_ProviderID detects this, logs the value it would have assigned, and reports itself as skipped, so the gap stays visible instead of quietly passing. Everything else about node initialization — taint removal, labels, addresses — is exercised normally.

Continuous integration

.github/workflows/e2e-simulator.yml runs this environment on every pull request and every push to main, as a matrix of the latest two Kubernetes versions against the latest two CloudStack releases. The CloudStack axis is not only version coverage: CloudStack 4.22 and later update a load balancer rule's CIDR list in place, while earlier versions delete and recreate the rule, so both branches get tested.

All matrix cells run in parallel and a shared build job compiles the CCM image once, so the whole workflow takes about as long as a single run — roughly fifteen minutes, most of it the simulator image pull and zone deployment.

To change the versions under test, edit the k8s and acs lists in the matrix. Both use explicit patch-level tags (kindest/node and apache/cloudstack-simulator), so a run is reproducible; avoid floating tags like latest.

Troubleshooting

Symptom Cause
LB service provider cannot support this rule on a VPC The VPC virtual router accepts only a restricted set of public load balancer ports. 80 and 8080 work; an arbitrary high port such as 8081 is rejected. Pick a port the router supports when adding a VPC test.
CCM exits with no management servers found The account cannot call listManagementServersMetrics. This is a root-admin API; the default User role does not include it.
Nodes keep the uninitialized taint; CCM logs provided node ip for node "..." is not valid The CloudStack VM's NIC IP does not match the IP kubelet registered with. Recreate the VM with ipaddress= set to the kind node's docker IP.
Services stay <pending>; CCM logs none of the hosts matched the list of VMs retrieved from CS API No CloudStack VM has a name matching a Kubernetes node name.
CCM logs found hosts that belong to different networks VMs matching the node names exist on more than one network — typically leftovers from a previous scenario.
No ACL rules are created on a VPC network The tier uses default_allow or default_deny. The CCM only manages rules on custom ACL lists.
CoreDNS stuck Pending Expected until the CCM removes the uninitialized taint. If it persists, the CCM is not working — check its logs.
kubectl logs/exec fail after nodes initialize The API server is preferring the Hostname address, which the CCM set to the CloudStack instance hostname. Use kubelet-preferred-address-types: InternalIP as the provided kind config does.
Simulator never becomes ready It runs mvn jetty:run and fetches from Maven Central at startup. Check docker logs cloudstack-simulator.