Add 'weighted' DRS algorithm, and speed up DRS plan generation

PendingReleaseNotes

@@ -39,3 +39,49 @@ example.ver.1 > example.ver.2:
     which can now be attached to Instances. This is to prevent the Secondary
     Storage to grow to enormous sizes as Linux Distributions keep growing in
     size while a stripped down Linux should fit on a 2.88MB floppy.
+
+  * New VM allocation algorithm 'balancedweighted' for vm.allocation.algorithm. Existing
+    algorithms and the default are unchanged; the new one is opt-in.
+
+    Existing algorithms rank hosts on allocated capacity alone. Under a large overprovisioning
+    factor that reads badly: allocation is measured against a total that has already been
+    multiplied by the factor, so a host under real strain can still report a low percentage and
+    keep attracting new VMs. Concurrent deployments make it worse, since they all read the same
+    figures before any of them is accounted for.
+
+    'balancedweighted' ranks hosts on a blend of allocated CPU and memory, measured CPU and
+    memory utilisation, VM count, and how many VMs started on the host recently. It holds back
+    hosts that are measurably too busy, and chooses at random from among the best scoring hosts
+    so that simultaneous deployments do not all pick the same one.
+
+    Tuned with the host.weighted.* settings, most of which are cluster scoped. Measured
+    utilisation comes from a moving average configured with host.load.sample.interval and
+    host.load.half.life. Until a management server has collected samples, ranking falls back to
+    allocation figures alone.
+
+  * New DRS algorithm 'weighted' for drs.algorithm, alongside 'balanced' and 'condensed'. The
+    default is unchanged; the new one is opt-in per cluster.
+
+    The existing algorithms balance a single metric chosen by drs.metric, so choosing one leaves
+    the other unwatched: a cluster can be even on memory while its CPU load varies several fold and
+    nothing moves. Allocation is also a poor stand-in for load under overprovisioning, where a
+    saturated host can still report a small percentage allocated.
+
+    'weighted' blends CPU and memory allocated with CPU and memory in use, and balances the result.
+    Imbalance keeps its existing shape - standard deviation over the mean - but it is computed over
+    a blend rather than over one metric, so drs.imbalance is not calibrated the same way and is
+    worth re-checking after switching. drs.metric, drs.metric.type and drs.metric.use.ratio choose
+    and shape the single metric the other algorithms balance; they do not apply to 'weighted' and
+    are ignored.
+
+    Utilisation is only used when every host in the cluster has been sampled. Comparing a host
+    measured on utilisation against one measured on allocation alone would report a difference that
+    is an artefact of the monitoring rather than of the load, so the cluster falls back to
+    allocation figures until every host can be measured.
+
+    Tuned with the drs.weighted.* settings, all cluster scoped.
+
+  * DRS plan generation is considerably faster on large clusters. Working out where a VM could go
+    is now done once for each group of VMs that would get the same answer rather than once per VM,
+    and affinity is re-evaluated only for VMs affected by the migration just planned rather than
+    for every VM on every iteration.

api/src/main/java/com/cloud/deploy/DeploymentClusterPlanner.java

@@ -62,11 +62,14 @@ public interface DeploymentClusterPlanner extends DeploymentPlanner {
             "vm.allocation.algorithm",
             "Advanced",
             "random",
-            "Order in which hosts within a cluster will be considered for VM allocation. The value can be 'random', 'firstfit', 'userdispersing', or 'firstfitleastconsumed'.",
+            "Order in which hosts within a cluster will be considered for VM allocation. The value can be 'random', "
+                    + "'firstfit', 'userdispersing', 'firstfitleastconsumed', or 'balancedweighted'. 'balancedweighted' "
+                    + "ranks hosts on a blend of allocated capacity, measured utilisation, VM count and how many VMs "
+                    + "started recently, and is tuned with the host.weighted.* settings.",
             true,
             ConfigKey.Scope.Global, null, null, null, null, null,
             ConfigKey.Kind.Select,
-            "random,firstfit,userdispersing,firstfitleastconsumed");
+            "random,firstfit,userdispersing,firstfitleastconsumed,balancedweighted");
 
     /**
      * This is called to determine list of possible clusters where a virtual

api/src/main/java/com/cloud/deploy/DeploymentPlanner.java

@@ -70,7 +70,7 @@ public interface DeploymentPlanner extends Adapter {
     boolean canHandle(VirtualMachineProfile vm, DeploymentPlan plan, ExcludeList avoid);
 
     public enum AllocationAlgorithm {
-        random, firstfit, userdispersing, firstfitleastconsumed;
+        random, firstfit, userdispersing, firstfitleastconsumed, balancedweighted;
     }
 
     public enum PlannerResourceUsage {

api/src/main/java/com/cloud/host/HostLoad.java

@@ -0,0 +1,60 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+package com.cloud.host;
+
+/**
+ * Smoothed view of what a host is actually doing, as opposed to what has been allocated on it.
+ * Fractions are of the host's real capacity and ignore overprovisioning.
+ */
+public class HostLoad {
+
+    public static final HostLoad UNKNOWN = new HostLoad(0, 0, 0);
+
+    private final double cpuUtilisation;
+    private final double memoryUtilisation;
+    private final long samples;
+
+    public HostLoad(double cpuUtilisation, double memoryUtilisation, long samples) {
+        this.cpuUtilisation = cpuUtilisation;
+        this.memoryUtilisation = memoryUtilisation;
+        this.samples = samples;
+    }
+
+    public double getCpuUtilisation() {
+        return cpuUtilisation;
+    }
+
+    public double getMemoryUtilisation() {
+        return memoryUtilisation;
+    }
+
+    public long getSamples() {
+        return samples;
+    }
+
+    /**
+     * False until enough has been observed to rank on. Callers fall back to allocation figures.
+     */
+    public boolean isUsable() {
+        return samples > 0;
+    }
+
+    @Override
+    public String toString() {
+        return String.format("HostLoad[cpu=%.3f, memory=%.3f, samples=%d]", cpuUtilisation, memoryUtilisation, samples);
+    }
+}

api/src/main/java/com/cloud/host/HostLoadService.java

@@ -0,0 +1,32 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+package com.cloud.host;
+
+/**
+ * A smoothed view of what each host is actually doing, as opposed to what has been allocated on it.
+ *
+ * Allocation figures say what was promised; under overprovisioning they can be far from what a host
+ * is really carrying. Placement and rebalancing both need the second view.
+ */
+public interface HostLoadService {
+
+    /**
+     * @return the host's smoothed load, or a value reporting itself unusable when the host has not
+     *         been sampled recently enough to rank on
+     */
+    HostLoad getLoad(long hostId);
+}

api/src/main/java/com/cloud/host/HostScoringWeights.java

@@ -0,0 +1,57 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+package com.cloud.host;
+
+import org.apache.cloudstack.framework.config.ConfigKey;
+
+/**
+ * How much each signal counts when ranking hosts by how loaded they are.
+ *
+ * Shared by initial placement and by rebalancing on purpose. If the two weighted these differently
+ * they would disagree about which host is the better one, and rebalancing could move VMs off hosts
+ * that placement had just chosen, only for placement to put them back.
+ *
+ * Weights are relative to each other; only their ratios matter, and zero disables a term. Terms that
+ * only make sense for one of the two - how many VMs a host carries, how many started recently - stay
+ * with whichever uses them.
+ */
+public interface HostScoringWeights {
+
+    String WEIGHT_DESCRIPTION_SUFFIX = " Relative weight, only meaningful compared with the other "
+            + "host.weighted.* weights. Zero disables the term. Used by both the 'balancedweighted' "
+            + "allocation algorithm and the 'weighted' DRS algorithm.";
+
+    ConfigKey<Double> CpuAllocatedWeight = new ConfigKey<>(ConfigKey.CATEGORY_ADVANCED,
+            Double.class, "host.weighted.cpu.allocated.weight", "1.0",
+            "How much CPU allocated on a host counts against it." + WEIGHT_DESCRIPTION_SUFFIX,
+            true, ConfigKey.Scope.Cluster);
+
+    ConfigKey<Double> CpuUsedWeight = new ConfigKey<>(ConfigKey.CATEGORY_ADVANCED,
+            Double.class, "host.weighted.cpu.used.weight", "2.0",
+            "How much measured CPU utilisation counts against a host." + WEIGHT_DESCRIPTION_SUFFIX,
+            true, ConfigKey.Scope.Cluster);
+
+    ConfigKey<Double> MemoryAllocatedWeight = new ConfigKey<>(ConfigKey.CATEGORY_ADVANCED,
+            Double.class, "host.weighted.memory.allocated.weight", "1.0",
+            "How much memory allocated on a host counts against it." + WEIGHT_DESCRIPTION_SUFFIX,
+            true, ConfigKey.Scope.Cluster);
+
+    ConfigKey<Double> MemoryUsedWeight = new ConfigKey<>(ConfigKey.CATEGORY_ADVANCED,
+            Double.class, "host.weighted.memory.used.weight", "2.0",
+            "How much measured memory utilisation counts against a host." + WEIGHT_DESCRIPTION_SUFFIX,
+            true, ConfigKey.Scope.Cluster);
+}

api/src/main/java/org/apache/cloudstack/affinity/AffinityProcessorBase.java

@@ -25,7 +25,9 @@
 import com.cloud.vm.VirtualMachineProfile;
 
 import java.util.Collections;
+import java.util.HashMap;
 import java.util.List;
+import java.util.Map;
 
 public class AffinityProcessorBase extends AdapterBase implements AffinityGroupProcessor {
 
@@ -44,6 +46,23 @@ public void process(VirtualMachineProfile vm, DeploymentPlan plan, ExcludeList a
 
     }
 
+
+    /**
+     * Indexes placements supplied by the caller. Callers such as DRS build a plan of several moves
+     * in memory and persist it only at the end, so during planning the database still shows the old
+     * host for every VM the plan has already moved.
+     */
+    protected Map<Long, VirtualMachine> getVmIdVmMap(List<VirtualMachine> vmList) {
+        Map<Long, VirtualMachine> vmIdVmMap = new HashMap<>();
+        if (vmList == null) {
+            return vmIdVmMap;
+        }
+        for (VirtualMachine vm : vmList) {
+            vmIdVmMap.put(vm.getId(), vm);
+        }
+        return vmIdVmMap;
+    }
+
     @Override
     public String getType() {
         return _type;

api/src/main/java/org/apache/cloudstack/cluster/ClusterDrsAlgorithm.java

@@ -20,6 +20,7 @@
 package org.apache.cloudstack.cluster;
 
 import com.cloud.host.Host;
+import com.cloud.host.HostLoad;
 import com.cloud.offering.ServiceOffering;
 import com.cloud.org.Cluster;
 import com.cloud.utils.Ternary;
@@ -62,6 +63,35 @@ public interface ClusterDrsAlgorithm extends Adapter {
     boolean needsDrs(Cluster cluster, List<Ternary<Long, Long, Long>> cpuList,
                      List<Ternary<Long, Long, Long>> memoryList) throws ConfigurationException;
 
+    /**
+     * As above, but keeping host identity.
+     *
+     * An algorithm that considers anything beyond the figures in the two maps - measured load, for
+     * instance - cannot attribute it without knowing which host each entry belongs to. Algorithms
+     * that only need the values keep the default.
+     *
+     * @param hostCpuMap
+     *         host id to a Ternary of used, reserved and total CPU
+     * @param hostMemoryMap
+     *         host id to a Ternary of used, reserved and total memory
+     */
+    default boolean needsDrs(Cluster cluster, Map<Long, Ternary<Long, Long, Long>> hostCpuMap,
+            Map<Long, Ternary<Long, Long, Long>> hostMemoryMap, Map<Long, HostLoad> hostLoadMap)
+            throws ConfigurationException {
+        return needsDrs(cluster, new ArrayList<>(hostCpuMap.values()), new ArrayList<>(hostMemoryMap.values()));
+    }
+
+    /**
+     * Called once per plan, before any migration is considered, so that an algorithm can do work
+     * that would otherwise be repeated for every candidate VM and host.
+     *
+     * @param hostLoadMap
+     *         measured load per host, empty when nothing has been sampled
+     */
+    default void prepare(Cluster cluster, Map<Long, Ternary<Long, Long, Long>> hostCpuMap,
+            Map<Long, Ternary<Long, Long, Long>> hostMemoryMap, Map<Long, HostLoad> hostLoadMap) {
+    }
+
     /**
      * Calculates the metrics (improvement, cost, benefit) for migrating a VM to a destination host. Improvement is
      * calculated based on the change in cluster imbalance before and after the migration.

api/src/main/java/org/apache/cloudstack/cluster/ClusterDrsService.java

@@ -51,9 +51,9 @@ public interface ClusterDrsService extends Manager, Configurable, Scheduler {
             true, ConfigKey.Scope.Cluster, null, "Maximum number of migrations for DRS", null, null, null);
 
     ConfigKey<String> ClusterDrsAlgorithm = new ConfigKey<>(String.class, "drs.algorithm",
-            ConfigKey.CATEGORY_ADVANCED, "balanced", "The DRS algorithm to be executed on the cluster. Possible values are condensed, balanced.",
+            ConfigKey.CATEGORY_ADVANCED, "balanced", "The DRS algorithm to be executed on the cluster. Possible values are condensed, balanced, weighted. 'weighted' balances CPU and memory together, and on measured load as well as allocation, tuned with the drs.weighted.* settings.",
             true, ConfigKey.Scope.Cluster, null, "DRS algorithm", null, null,
-            null, ConfigKey.Kind.Select, "condensed,balanced");
+            null, ConfigKey.Kind.Select, "condensed,balanced,weighted");
 
     ConfigKey<Float> ClusterDrsImbalanceThreshold = new ConfigKey<>(Float.class, "drs.imbalance",
             ConfigKey.CATEGORY_ADVANCED, "0.4",

client/pom.xml

@@ -592,6 +592,11 @@
             <artifactId>cloud-plugin-cluster-drs-condensed</artifactId>
             <version>${project.version}</version>
         </dependency>
+<dependency>
+            <groupId>org.apache.cloudstack</groupId>
+            <artifactId>cloud-plugin-cluster-drs-weighted</artifactId>
+            <version>${project.version}</version>
+        </dependency>
         <dependency>
             <groupId>org.apache.cloudstack</groupId>
             <artifactId>cloud-plugin-database-quota</artifactId>

engine/schema/src/main/java/com/cloud/vm/dao/VMInstanceDao.java

@@ -131,6 +131,18 @@ public interface VMInstanceDao extends GenericDao<VMInstanceVO, Long>, StateDao<
 
     List<Long> listHostIdsByVmCount(long dcId, Long podId, Long clusterId, long accountId);
 
+    /**
+     * Counts the VMs occupying each host in a zone, pod or cluster, in one query.
+     *
+     * @param changedStateAfter
+     *         cut-off for the second count: VMs whose state last changed after this. Approximates
+     *         the VMs still working through their startup load, which neither allocation nor a
+     *         utilisation average has caught up with yet.
+     * @return host id to {total VMs, VMs that changed state recently}. Every host in scope appears,
+     *         including hosts with no VMs.
+     */
+    Map<Long, Pair<Long, Long>> countVmsByHost(long dcId, Long podId, Long clusterId, Date changedStateAfter);
+
     Long countRunningAndStartingByAccount(long accountId);
 
     Long countByZoneAndState(long zoneId, State state);