Add 'weighted' DRS algorithm, and speed up DRS plan generation - #14111
Draft
bhouse-nexthop wants to merge 16 commits into
Draft
Add 'weighted' DRS algorithm, and speed up DRS plan generation#14111bhouse-nexthop wants to merge 16 commits into
bhouse-nexthop wants to merge 16 commits into
Conversation
Counts VMs occupying each host in a zone, pod or cluster, optionally only those that changed state recently. One query for the whole scope rather than one per host. Signed-off-by: Brad House <bhouse@nexthop.ai>
StatsCollector already polls CPU and memory utilisation for every host, but it keeps only the newest sample and nothing uses it for placement. - fold those samples into an exponentially weighted moving average - weight by elapsed time, so a missed poll decays correctly instead of over-weighting the previous value - report nothing usable until a host has been sampled, so callers can fall back to allocation figures Two settings: host.load.sample.interval and host.load.half.life. Signed-off-by: Brad House <bhouse@nexthop.ai>
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #14111 +/- ##
============================================
+ Coverage 19.79% 19.86% +0.06%
- Complexity 20015 20117 +102
============================================
Files 6371 6376 +5
Lines 575954 576590 +636
Branches 70521 70601 +80
============================================
+ Hits 113997 114516 +519
- Misses 449530 449593 +63
- Partials 12427 12481 +54
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Opt-in via vm.allocation.algorithm. Existing algorithms and the default are untouched. Allocated capacity alone is a poor ranking under heavy overprovisioning: it is measured against a total already multiplied by the overprovisioning factor, so a host under real strain still reports a low percentage and keeps being chosen. Anything allocation cannot see - VMs the scheduler has lost track of, guests using more than they asked for - is invisible. Ranks on a blend, lower is better: | term | source | |---------------------|-------------------------------------------| | CPU allocated | op_host_capacity, over the inflated total | | CPU utilisation | moving average of real usage | | memory allocated | op_host_capacity, over the inflated total | | memory utilisation | moving average of real usage | | VM count | VMs on the host | | recent starts | VMs started within the last few minutes | A dominant resource term is added on top so a host nearly out of any one resource does not rank well on a good average. Hosts measurably too busy are held back, unless that would leave nowhere to deploy. Selection is random among the best few rather than strictly ordered: capacity is only charged once a VM starts, so concurrent deployments all read the same figures and strict ordering makes them agree on one host. All weights and thresholds are settings, most cluster scoped. Signed-off-by: Brad House <bhouse@nexthop.ai>
Allocated fraction was measured against the wrong total. op_host_capacity stores totals raw and overprovisioning is applied when they are read, so dividing by the stored total made the fraction reach 1 at the host's physical size. On a cluster overcommitted 10 times every host clamped to 1, killing both the allocation term and the dominant resource term - on exactly the clusters this algorithm is for. - apply the cluster ratio to the denominator - drop hosts missing a CPU or memory capacity row instead of scoring the missing resource as untouched, which made them rank first Utilisation thresholds could be bypassed. Held-back hosts were appended before the random spread was applied, so the spread could shuffle a busy host into the lead. With 1 healthy host and a spread of 3, two thirds of deployments picked an over-threshold host. - spread over healthy hosts only, before anything else is appended A host with no load samples was treated as idle. It was exempt from the thresholds and its dominant resource term fell back to allocation, so a host with broken stats outranked every measured host and collected the deployments. - rank hosts we cannot measure behind every host we can - when nothing can be measured, ranking falls back to allocation as before Signed-off-by: Brad House <bhouse@nexthop.ai>
Utilisation average
- a host whose agent stops reporting kept vouching for itself forever:
StatsCollector hands back the previous entry when a poll fails, and
that unchanged reading was folded again every minute. Detect the
repeat, and expire an average that stops being updated
- sample on a scheduled executor catching Throwable, not a Timer, which
dies permanently and silently on one escaping error
- only collect when an algorithm that reads the figures is selected
- read the half life once per sample rather than inside the map update
- document what getCpuUtilization means per hypervisor: it is what this
assumes on KVM, a reservation figure on VMware, and scaled by core
count on XenServer
Scoring
- a negative weight would rank the most loaded host first; floor at
zero and say so
- zeroing all six terms no longer discards the dominant resource term
- read weights once per ranking instead of once per host
Queries
- one query per ranking instead of two, returning both counts
- count Stopping VMs, which still hold their host
- correct the doc: every host in scope is returned, including empty ones
Tests
- cover rank() end to end, which is where the defects were: the
capacity denominator, the thresholds, the spread and the ordering of
measured against unmeasured hosts
- the distribution simulation drew different random streams per arm, so
the arms saw different workloads. Fix the workload up front and add
an allocation-only-plus-spread control, which shows the scoring and
not the spread is what evens out real load
Signed-off-by: Brad House <bhouse@nexthop.ai>
The reserved-capacity branch was chained to the wrong condition, so it only ran when the group VM was null or removed: - a Stopped VM still holding reserved capacity never had its last host avoided - the branch was dead for every live VM - a group mapping pointing at a deleted VM hit the branch with a null and threw NullPointerException Restructured to match NonStrictHostAffinityProcessor, which already had the intended shape: skip null/removed, then avoid the current host, else the last host while capacity is still reserved. Signed-off-by: Brad House <bhouse@nexthop.ai>
The processor already accepted a vmList of placements and ignored it, reading every group member's host from the database instead. DRS builds a multi-migration plan in memory and persists it only at the end, so while the plan is being built the database still shows the old host for every VM the plan has already moved. Anti-affinity was therefore evaluated against stale placements, and a plan could put two anti-affine VMs on the same host. - resolve each group member from vmList first, fall back to the database - mirrors what HostAffinityProcessor already does with the same argument - no change when vmList is empty, which is every non-DRS caller Signed-off-by: Brad House <bhouse@nexthop.ai>
Same gap as the strict processor: vmList was accepted and ignored, so host priorities were adjusted from database placements even when the caller supplied newer ones. NonStrictHostAntiAffinityProcessor extends this class and only overrides the priority direction, so it is fixed by the same change. Signed-off-by: Brad House <bhouse@nexthop.ai>
A DRS plan is generated once and executed later, and nothing downstream re-checks it - migrateVirtualMachine does not enforce affinity groups. By execution time the cluster may have changed, so a plan that was valid when generated can violate anti-affinity when it runs. - validate each migration against current state before queueing it - skip and mark failed instead of migrating into a violation - track destinations already queued in this run, since the jobs are asynchronous and the database does not reflect them yet Signed-off-by: Brad House <bhouse@nexthop.ai>
Three defects found reviewing the previous commits, all in how DRS decides whether a planned migration is still allowed. Non-strict anti-affinity was discarded entirely. Non-strict groups express themselves by lowering a host's priority on the deployment plan rather than by excluding it, and DRS built a plan, handed it to the processors, read only the exclude list and threw the plan away. Non-strict means the rule may be broken when there is nowhere else to put a VM. That cannot arise while rebalancing: the VM already runs somewhere that satisfies the group and leaving it there is always an option. Being better balanced is not a reason to break it. A host was treated as free the moment a migration away from it was queued. The jobs are asynchronous and can fail, so a queued migration occupies both ends until it completes. In a swap - A from host1 to host3, B from host2 to host1 - B was cleared for host1 while A was still on it. - track the hosts queued migrations have not actually left - refuse a destination that is one of them A VM that stopped between planning and execution threw NPE inside the affinity check, which was then logged without a stack trace. - treat a VM that is no longer running as an out of date plan - log the exception rather than its message Signed-off-by: Brad House <bhouse@nexthop.ai>
- a skipped migration is CANCELLED, not FAILED. Nothing went wrong; the plan went out of date. FAILED was indistinguishable from a migration that genuinely broke, and left no record of why - record an event when one is skipped, so it is visible rather than a silent no-op in an otherwise successful plan - the processors also cover dedicated resources and DPDK, so the refusal message no longer claims every skip is about an affinity group - hoist the third copy of getVmIdVmMap into AffinityProcessorBase Signed-off-by: Brad House <bhouse@nexthop.ai>
Plan generation took minutes on a busy cluster. Two causes, both about repeating work rather than about the algorithm. Working out where a VM could go runs the host allocators and inspects every volume, and it was done once per VM. VMs that would get the same answer - same offering, template, current host, storage and affinity groups - now share one pass. A cluster of similar VMs goes from one pass per VM to one per handful. Affinity was re-evaluated for every VM in the cluster on every iteration, up to drs.max.migrations times. Only VMs sharing a group with the one just moved can have changed, so only those are re-evaluated after the first pass. Also: a VM whose grouping key cannot be worked out is now considered on its own rather than silently dropped from the plan. Signed-off-by: Brad House <bhouse@nexthop.ai>
Opt-in per cluster via drs.algorithm. balanced and condensed are
untouched and the default is unchanged.
The existing algorithms balance one metric chosen by drs.metric, so
whichever is not chosen goes unwatched: a cluster can sit inside its
imbalance threshold on memory while CPU load varies several fold, and
DRS correctly does nothing. Allocation is also a poor stand-in for load
under overprovisioning, where a saturated host still reports a small
percentage allocated.
'weighted' blends four figures per host - CPU and memory allocated, CPU
and memory in use - and balances the result. Imbalance keeps its
existing definition, standard deviation over the mean, so drs.imbalance
still means what it did.
- allocation is measured against what a host can hand out, so the
overcommit ratio is applied
- a migration is only worth making if the cluster ends up more even,
and one that has to move storage costs more
- hosts without utilisation samples fall back to allocation figures
needsDrs decides whether to look at a cluster at all, and it was handed
lists with no host identity, so measured load could not be attributed to
a host. Added a map form alongside it; algorithms that only need the
values keep the default and are unaffected.
Signed-off-by: Brad House <bhouse@nexthop.ai>
Two of these meant the algorithm could not run at all.
The plugin had no module.properties, so it was never discovered, never
instantiated and never registered. Selecting it would have failed every
DRS run with "Invalid algorithm configured".
The plugin injected a service that lives in a sibling Spring module and
is therefore not visible to it, so once the module did load, the
management server would fail to start.
- the caller now works out the load once per plan and passes it in
- which also fixes the cost of getting it: getMetrics runs for every
candidate VM and host, and it was reading two cluster details and
four settings from the database on each call
Grouping VMs to share one candidate-host lookup was unsound. The key
listed the inputs it thought mattered, and missed several that are held
per VM rather than per offering - a custom offering's size, boot mode,
device settings. Two VMs could then share a host list neither of them
should have had.
- only group VMs that carry none of those, rather than trying to
enumerate everything that could matter
- key on disk offering as well as pool, since storage tags come from
the offering
Reserved capacity was multiplied by the overcommit ratio instead of being
subtracted from the scaled total, so a host reading 11% fuller than its
peer read 67% fuller, and DRS would evacuate hosts merely for holding
reserved capacity.
Hosts were compared on different bases. A host with no utilisation
samples was measured on allocation alone while its peers were measured on
a blend, reporting a difference that was an artefact of the monitoring.
Utilisation is now used only when every host has it.
The cost and benefit terms cancelled out, so storage motion was free
despite the claim otherwise. A migration that has to move storage now has
to earn more than one that does not, by a configurable margin.
Also: allocation beyond what a host can hand out is no longer flattened
to a single value, and the settings that shape the other algorithms'
single metric are documented as not applying here.
Signed-off-by: Brad House <bhouse@nexthop.ai>
The grouping could have been disabled and every test would still have passed. Asserts the expensive call is made once for two interchangeable VMs. Signed-off-by: Brad House <bhouse@nexthop.ai>
bhouse-nexthop
force-pushed
the
drs-weighted
branch
from
September 10, 2026 01:48
11d99d4 to
ebc8e74
Compare
It was declared alongside the allocators, in a module context that is a descendant of the one the DRS service lives in. A bean in a descendant context is not visible to its parent, so injecting it into the DRS service failed and the management server did not start. Unit tests cannot catch this - the field is a mock there. It showed up as every integration test job running to its six hour limit waiting for a management server that was never coming up. Declared in the core context instead, which is an ancestor of the allocator module and the same context the DRS service is declared in. Also add the trailing newlines pre-commit asked for. Signed-off-by: Brad House <bhouse@nexthop.ai>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Warning
Draft — stacked on two other PRs. Please do not review the whole diff.
This branch contains the commits from #14108 and #14109 as well as its own, because it depends on
both. Only the last four commits are this PR:
Stop DRS planning from redoing the same workAdd 'weighted' DRS algorithmFix defects found in review of the weighted DRS algorithmVerify equivalent VMs share one candidate host lookupEverything before those belongs to:
4.22; the commits appear here cherry-pickedforward, and will disappear from this diff once that merges forward)
balancedweightedallocation algorithm, which this reuses for itsutilisation figures
It will be taken out of draft and rebased once both have merged. Raising it now so the design can
be discussed alongside them, since the three are easier to judge together than apart.
Description
Adds an opt-in DRS algorithm,
weighted, fordrs.algorithm, alongsidebalancedandcondensed.The default is unchanged and the existing algorithms are untouched.
The problem. The existing algorithms balance a single metric chosen by
drs.metric, so whicheveris not chosen goes unwatched. A cluster can sit comfortably inside its imbalance threshold on memory
while CPU load varies several fold across its hosts, and DRS will correctly do nothing.
Allocation is also a poor stand-in for load under overprovisioning. Allocated figures are measured
against a total that has already been multiplied by the overprovisioning factor, so a host whose
cores are saturated can still report a small percentage allocated.
What it balances. A blend of four figures per host - CPU and memory allocated, CPU and memory in
use:
where the allocated fractions are measured against what a host can hand out - its total scaled by
the cluster's overcommit ratio, with reserved capacity taken off that.
Imbalance keeps the shape the other algorithms use, so
drs.imbalancestill behaves the same way,but it is now computed over a blend rather than over one metric and is therefore not calibrated
the same. Worth re-checking that setting after switching.
drs.metric,drs.metric.typeanddrs.metric.use.ratiochoose and shape the single metric theother algorithms balance. They do not apply here and are ignored.
Utilisation is used only when every host in the cluster has been sampled. Imbalance compares
hosts against each other, so measuring one host on a blend and another on allocation alone would
report a difference that is an artefact of the monitoring rather than of the load - and would
evacuate whichever host had stopped reporting. When any host is unmeasured the cluster falls back to
allocation figures for all of them.
Cost and benefit. Both are expressed in units of imbalance, so the caller's
benefit > costtestmeans "is this worth what it costs". A migration that has to move storage has to earn more than one
that does not, by a configurable margin.
Settings
The four weights are the same
host.weighted.*settings thebalancedweightedallocationalgorithm uses (#14109), deliberately rather than a second copy:
host.weighted.cpu.allocated.weight1.0host.weighted.cpu.used.weight2.0host.weighted.memory.allocated.weight1.0host.weighted.memory.used.weight2.0drs.weighted.storage.motion.cost0.02If placement and rebalancing weighted host load differently they would disagree about which host is
the better one, and rebalancing could move VMs off hosts placement had just chosen, only for
placement to put them back. Sharing the weights makes them agree by construction. The terms that
only make sense for one of the two - VM count and recent starts, which are placement-only - stay
where they are.
Weights are relative to each other;
0disables a term.Also: DRS plan generation is much faster
Independent of the algorithm, and useful on its own. Plan generation took minutes on a large
cluster, for two reasons that are both about repeating work:
done once per VM. VMs that would get the same answer now share one pass. A VM is only grouped when
it carries none of the per-VM inputs that change the answer - a custom offering whose size comes
from the VM, a boot mode, a device setting - rather than trying to enumerate everything that could
matter and risk missing one.
drs.max.migrationstimes. Only VMs sharing a group with the one just moved can have changed.Types of changes
Feature/Enhancement Scale or Bug Severity
Feature/Enhancement Scale
How Has This Been Tested?
WeightedTest- 13 cases: an even cluster reads as balanced; CPU load imbalance is seen whenmemory is even and vice versa; allocation beyond what a host can hand out stays distinguishable;
reserved capacity does not make a host look fuller; utilisation is dropped when any host cannot be
measured and used when all can; moving a VM off the busier host is an improvement and onto it is
not; storage motion has to earn more.
ClusterDrsEquivalenceTest- 11 cases on what may and may not share a candidate-host lookup,including custom offerings and boot modes, which must never be grouped.
ClusterDrsServiceImplTest- 34 cases, including one asserting that two interchangeable VMs costone lookup rather than two, so the grouping cannot be silently disabled.
Full
mvn testonapi,serverand all three DRS plugins with checkstyle and license checksenabled: 0 failures.
How did you try to break this feature and the system with this change?
Adversarial review found four defects that would have made this unusable, all fixed:
module.properties, so it was never discovered, never instantiated andnever registered. Selecting it would have failed every DRS run with "Invalid algorithm
configured".
module did load, the management server would have failed to start - for anyone with the plugin on
the classpath, regardless of configuration. The caller now works out the load once per plan and
passes it in.
getMetricsread two cluster details and four settings from the database on every call, andit is called for every candidate VM and host - hundreds of thousands of times for a large cluster.
That would have swamped the performance work in the same PR.
that are held per VM rather than per offering. Two VMs could share a host list neither should have
had. Inverted to only group VMs that provably carry none of them.
Also found and fixed: reserved capacity was multiplied by the overcommit ratio instead of subtracted
from the scaled total, so a host 11% fuller than its peer read 67% fuller and DRS would have
evacuated it for no reason; hosts with and without utilisation samples were compared on different
bases; and the cost and benefit terms cancelled out, making storage motion free despite the claim
otherwise.