Add 'balancedweighted' VM allocation algorithm

[bhouse-nexthop]

Description

This PR adds an opt-in VM allocation algorithm, balancedweighted, for vm.allocation.algorithm. Existing algorithms and the default are untouched.

The problem. The existing algorithms rank hosts on allocated capacity alone. Under a large overprovisioning factor that reads badly: allocation is measured against a total that has already been multiplied by the factor, so a host under real strain can still report a low percentage allocated and keep attracting new VMs. Anything allocation cannot see - guests using far more than they asked for, VMs the management server has lost track of - is invisible.

Concurrent deployments make it worse. Capacity is only charged once a VM starts, so every decision taken in the same moment reads the same figures, and strict ordering makes them all agree on one host.

What it ranks on. A blend, lower is better:

Term	Source
CPU allocated	op_host_capacity, over the overprovisioned total
CPU utilisation	moving average of measured usage
Memory allocated	op_host_capacity, over the overprovisioned total
Memory utilisation	moving average of measured usage
VM count	VMs on the host
Recent starts	VMs started within the last few minutes

Recent starts exist because a VM that started moments ago is invisible to both allocation lag and a moving average, while often working hardest.

A dominant-resource term is added on top of the weighted mean, taking the larger of allocated and measured per resource, so that a host nearly out of any one resource does not rank well on a good average.

Then two things happen that ranking alone does not do:

• hosts measurably too busy are held back, unless that would leave nowhere to deploy
• selection is random among the best few rather than strictly ordered, so simultaneous deployments do not all pick the same host

The algorithm

Every candidate host gets a score in [0, 1]; lower is better.

Terms. Each is a fraction of that host's capacity for the resource, so the weights are directly comparable:

cpu_alloc  = (used + reserved CPU)    / (raw total CPU    x cluster cpu overcommit ratio)
mem_alloc  = (used + reserved memory) / (raw total memory x cluster memory overcommit ratio)
cpu_util   = EWMA of measured CPU utilisation      (fraction of real cores)
mem_util   = EWMA of measured memory in use        (fraction of real memory)
vm_count   = VMs on the host            / host.weighted.expected.vms.per.host
recent     = VMs started within window  / host.weighted.expected.vms.per.host

All six are clamped to [0, 1]. The allocated terms divide by the overprovisioned total because that is what a host can hand out; dividing by the raw total makes every host on an overcommitted cluster read as full.

Score.

             w_ca*cpu_alloc + w_cu*cpu_util + w_ma*mem_alloc + w_mu*mem_util + w_vc*vm_count + w_rs*recent
  mean(h) = ---------------------------------------------------------------------------------------------
                              w_ca + w_cu + w_ma + w_mu + w_vc + w_rs

  dominant(h) = max( max(cpu_alloc, cpu_util), max(mem_alloc, mem_util) )

                mean(h) + w_dom * dominant(h)
  score(h)  =  ------------------------------
                        1 + w_dom

dominant is the host's most stressed resource, and it takes the larger of allocated and measured per resource — memory reclaimed from idle guests is taken back the moment those guests get busy, so the optimistic reading should not win. Adding it on top of the mean stops a host that is fine on average but nearly out of one resource from ranking well. Both parts are convex combinations of values in [0, 1], so the score stays in [0, 1].

When a host has no usable utilisation samples, w_cu and w_mu drop out of both numerator and denominator, and the host is ranked behind every host that can be measured rather than being assumed idle.

Moving average. Weighted by elapsed time, so a missed poll decays by the right amount instead of over-weighting the previous value:

  value  <-  value + a * (sample - value),    a = 1 - exp( -dt * ln2 / half_life )

Selection. Ranking alone is not enough: capacity is only charged once a VM starts, so concurrent deployments all read the same figures and strict ordering makes them agree on one host.

1. split measured hosts into healthy / over-threshold
2. sort each group by score, ascending
3. shuffle the first N healthy hosts        (N = host.weighted.selection.spread)
4. append hosts that cannot be measured
5. append hosts held back by a threshold    (used only if nothing above them fits)

Settings

Weights are relative to each other; only their ratios matter. 0 disables a term.

The first four are defined in api rather than alongside the allocator, because they describe how
loaded a host is rather than anything specific to placement, and a DRS algorithm that ranks hosts
the same way should read the same settings instead of carrying a second copy that could disagree.

Setting	Default	Scope	Meaning
host.weighted.cpu.allocated.weight	1.0	Cluster	Weight of CPU allocated
host.weighted.cpu.used.weight	2.0	Cluster	Weight of measured CPU utilisation
host.weighted.memory.allocated.weight	1.0	Cluster	Weight of memory allocated
host.weighted.memory.used.weight	2.0	Cluster	Weight of measured memory utilisation
host.weighted.vm.count.weight	1.0	Cluster	Weight of the number of VMs on the host
host.weighted.recent.start.weight	2.0	Cluster	Weight of VMs started recently
host.weighted.dominant.resource.weight	1.0	Cluster	Weight of the host's most stressed resource
host.weighted.recent.start.window	300	Global	Seconds a VM counts as recently started
host.weighted.expected.vms.per.host	50	Cluster	Normaliser for the two count terms. Not a limit and never enforced
host.weighted.cpu.utilisation.threshold	0.85	Cluster	Hold a host back above this measured CPU. 1 disables
host.weighted.memory.utilisation.threshold	0.90	Cluster	Hold a host back above this measured memory. 1 disables
host.weighted.selection.spread	3	Cluster	How many of the best hosts to choose between at random. 1 restores strict ordering
host.load.sample.interval	60	Global	Seconds between utilisation samples. Read at startup; should not be below host.stats.interval
host.load.half.life	300	Global	Half life of the moving average
host.load.stale.after	600	Global	Seconds after which a host's average is discarded and it ranks as unmeasured

All are dynamic except host.load.sample.interval.

Measured effect. Simulation over a churning, heavily overprovisioned fleet where part of the real load is invisible to allocation, measuring how unevenly real load ends up distributed (max/mean across hosts, lower is better):

Ranking	Load skew
allocation only	1.84 - 2.01
allocation only, plus the random spread	1.65 - 2.08
weighted	1.27 - 1.40

The middle row is a control: it isolates what the scoring contributes from what the randomisation contributes.

Utilisation figures come from a moving average of what StatsCollector already polls but nothing used for placement. It is per management server and not persisted; every management server polls every host, so they converge. Until a server has samples, ranking falls back to allocation figures, and hosts that cannot be measured rank behind hosts that can rather than being assumed idle.

Note the utilisation terms are only meaningful as intended on KVM. getCpuUtilization reports a reservation figure on VMware and is scaled by core count on XenServer; this is documented on the collector.

All weights and thresholds are settings, most cluster scoped.

Types of changes

• Breaking change (fix or feature that would cause existing functionality to change)
• New feature (non-breaking change which adds functionality)
• Bug fix (non-breaking change which fixes an issue)
• Enhancement (improves an existing feature and functionality)
• Cleanup (Code refactoring and cleanup, that may add test cases)
• Build/CI
• Test (unit or integration test code)

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

• Major
• Minor

How Has This Been Tested?

Unit tests, 39 new cases:

• WeightedHostScorerTest - the scoring function: each term's direction, the dominant-resource term, unmeasured hosts, the utilisation gates, the selection spread.
• WeightedHostScorerRankTest - rank() end to end as the allocator calls it, with the capacity and VM-count queries mocked. Covers the overprovisioned denominator, a busy host ranking behind a quiet one at equal allocation, an over-threshold host never leading while a healthy one exists, unmeasured hosts ranking last, fallback to allocation when nothing is measured, and a host missing a capacity row.
• HostLoadTrackerTest - the moving average: first sample, a single spike not dominating, convergence, half-life, missed samples decaying by elapsed time rather than sample count, unchanged readings not folded twice, and a host that stops reporting becoming unusable.
• WeightedPlacementDistributionTest - the simulation above. Deterministic; the workload is fixed before any arm runs so all arms see identical VMs.

Full mvn test on api, engine/schema and server, checkstyle and license checks enabled: 0 failures.

How did you try to break this feature and the system with this change?

This went through two rounds of adversarial review. The defects found and fixed are worth listing, since they are the interesting part:

• The allocated fraction was measured against the wrong total. op_host_capacity stores totals raw and applies overprovisioning when they are read. Dividing by the stored total made the fraction reach 1 at the host's physical size, so on a cluster overcommitted ten times every host clamped to 1 and both the allocation term and the dominant-resource term went dead - on exactly the clusters this is for.
• The utilisation threshold could be bypassed. Held-back hosts were re-appended before the random spread was applied, so with one healthy host and a spread of three, two thirds of deployments picked an over-threshold host.
• A host with no samples was treated as idle, exempt from the thresholds and scoring well on the dominant term - so a host with broken statistics became a deployment magnet.
• A host that stopped reporting kept vouching for itself. StatsCollector hands back the previous entry when a poll fails, and that unchanged reading was folded again every interval.
• The sampler ran on a Timer, which dies permanently and silently on one escaping error, leaving placement quietly back on allocation alone. It also ran regardless of whether the algorithm was selected.
• A negative weight would have ranked the most loaded host first.
• The simulation drew different random streams per arm, so the arms were not seeing the same workload; and it bypassed rank(), which is why it caught none of the above. Both fixed, and the spread-only control arm added.

Other things checked: hosts missing a capacity row, all-zero weights, podId/clusterId being null, concurrent access to the shared average, and that the two per-ranking queries became one.

[codecov]

Codecov Report

❌ Patch coverage is 65.56886% with 115 lines in your changes missing coverage. Please review.
✅ Project coverage is 19.82%. Comparing base (036493f) to head (726e632).

Files with missing lines	Patch %	Lines
.../agent/manager/allocator/impl/HostLoadTracker.java	54.20%	47 Missing and 2 partials ⚠️
...ent/manager/allocator/impl/WeightedHostScorer.java	81.17%	21 Missing and 11 partials ⚠️
.../main/java/com/cloud/vm/dao/VMInstanceDaoImpl.java	0.00%	28 Missing ⚠️
...m/cloud/agent/manager/allocator/impl/HostLoad.java	81.81%	3 Missing and 1 partial ⚠️
...gent/manager/allocator/impl/FirstFitAllocator.java	0.00%	2 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##               main   #14109      +/-   ##
============================================
+ Coverage     19.79%   19.82%   +0.03%     
- Complexity    20015    20069      +54     
============================================
  Files          6371     6375       +4     
  Lines        575954   576287     +333     
  Branches      70521    70562      +41     
============================================
+ Hits         113997   114262     +265     
- Misses       449530   449577      +47     
- Partials      12427    12448      +21     

Flag	Coverage Δ
uitests	3.52% <ø> (ø)
unittests	21.10% <65.56%> (+0.03%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.