Graphical Abstract

BalCon – resource balancing algorithm for VM consolidation

Andrei Gudkov , Pavel Popov , Stepan Romanov

Classes:
Does Ample Force step
VM fit into free space Balanced on VM4
of any host? Lopsided
Yes N o VM4 VM1
Are VM4
Place VM VM1
with BestFit hosts balanced?
VM3 VM3
No
Yes VM2 VM2
Place VM with Place VM with Host 1 Host 2 Host 1 Host 2
ForceFitBalanced ForceFitLopsided
Highlights
BalCon – resource balancing algorithm for VM consolidation
Andrei Gudkov , Pavel Popov , Stepan Romanov

• We introduce the BalCon algorithm for solving the migration-

aware consolidation problem. Force Fit is a key part
of BalCon that allows for solving instances with Force
Steps.
• Huawei and synthetic datasets are used to validate Bal-
Con.
• Integer linear programming flavor flow model is proposed
to produce optimal solutions and lower bounds.
• A Sercon heuristic is modified for comparison with Bal-
Con.
• The balance factor is proposed as a measure of free space
distribution in a datacenter.
 BalCon – resource balancing algorithm for VM consolidation
a a a
Andrei Gudkov , Pavel Popov , Stepan Romanov
a Huawei Technologies Company Ltd, Russian Research Institute, Moscow, 121099, Russian Federation

Abstract
Cloud providers handle substantial number of requests to create and delete virtual machines (VMs) on a daily basis, where the
unknown sequence of requests eventually leads to resource fragmentation. To mitigate this issue, periodic consolidation of VMs
into fewer number of physical hosts is an important cost-saving procedure, closely related to the vector bin-packing problem. In
this paper, we propose the BalCon algorithm for consolidation that steadily reduces the number of active hosts and keeps migration
costs low. BalCon classifies the cluster’s state and selects one of three heuristics to balance resources for superior consolidation.
To evaluate BalCon’s performance with respect to optimality, we introduce integer programming models. BalCon finds 99.7% of
the optimal solutions for over 750 problem instances. This outstanding result was achieved due to the Force Step of our algorithm,
which is the key improvement detail for common heuristics. We compare BalCon with a modified Sercon heuristic using Huawei
and synthetic datasets with two resources for allocation.
Keywords: Consolidation, Bin packing, High-multiplicity, Cloud computing, Resource scheduling, Heuristic algorithms

Capacity a) Initial Fragmented Optimal

1. Introduction Host Flavors
VM2 VM3
1

r off
Virtualization technology allows the sharing of a physi- Delete Migrate
3

Powe
VM2 VM3
cal machine (host) between multiple isolated virtual machines VM1 VM1 VM1
(VMs) [1, 2]. Resources for virtualization include the central 2 VM3 VM3

processing unit (CPU), random access memory (RAM), net- Host 1 Host 2 Host 1 Host 2 Host 1 Host 2
b) Initial Fragmented Optimal
work, drives, etc. Several users can run VMs with different
operating system (OS) on the same host simultaneously. The Create VM2 VM1

r off
r off

off

Migrate
resource sharing prospect of virtualization opens up variety of VM2 VM2
r

Powe
Powe

Powe

VM3 VM3 VM4 VM1

opportunities for business. VM2
VM3 VM4
VM1 VM4 VM1
Cloud providers operate datacenters with thousands of hosts
Host 1 Host 2 Host 3 Host 1 Host 2 Host 3 Host 1 Host 2 Host 3
and exploit virtualization to lease VMs to clients [3, 4]. Re-
sources combinations for the VMs are usually predefined and Figure 1: An example of resource fragmentation caused by delete and cre-
known as flavors. For instance, two dimensional flavors (CPU, ate operations followed by optimisation via the migration operation. Panel a)
RAM) can be (2 cores, 4 GiB), (8 cores, 24 GiB), (64 cores, demonstrates how a delete operation of VM2 led to resource fragmentation,
which was optimized by migration of VM3. Panel b) shows how a sequence
560 GiB), etc. Clients choose a flavor and an OS image for the of create operation for the trace of VMs (VM2, VM3, VM4) by the First Fit
VM and remotely use the VM on a ”pay-as-you-go” basis. The heuristic caused resource fragmentation, which was optimized by the migration
guarantees from providers to clients regarding VMs availability of VM1 and VM2.
and reliability are specified in a service level agreement (SLA)
[5, 6], where violation leads to the providers being penalised. The VMs in datacenters are managed through three essen-
Therefore, during datacenters operation the providers strive to tial operations: 1) create VM on host, 2) delete VM from host,
minimize SLA violations and operational costs. and 3) migrate VM from one host to another. The periodic dele-
Hosts in an idle state consume about 70% of maximal power tion of VMs (see Figure 1a) along with unpredictable requests
consumption[7]. To minimize operational costs, providers seek to create VMs (see Figure 1b) leads to resource fragmentation
to allocate VMs into the fewest number of hosts and power off on the hosts and non-optimal VM placement.
the unused ones. The scope for optimization arises from the Live migration is an important technique for the optimiza-
uncertainty of requests during cloud operation, which leads to tion of datacenters, as it allows running VMs to be migrated
fragmentation of resources. between hosts without interruption or client notice [8]. How-
ever, frequent live migration is undesirable due to the increased
DOI: https://doi.org/10.1016/j.future.2023.05.001
load on the management side of the cloud, where the load de-
Email addresses: [email protected] (Andrei Gudkov ), pends on various factors, such as the state of the VMs (pro-
[email protected] (Pavel Popov ), cesses, memory, network, etc.). Therefore, it is preferable to
[email protected] (Stepan Romanov ) migrate less active VMs with less RAM.

Accepted to Future Generation Computer Systems May 5, 2023

1.1. Consolidation the consolidation problem is equivalent to removing all the
Consolidation of virtual machines [9] is a procedure that VMs from the hosts and placing them again. Such actions re-
searches for a better assignment of VMs to hosts. The improved quire unacceptably large amounts of memory for migration.
assignment benefits one or more of the following metrics:
power consumption [10, 7], number of active hosts [11, 12], mi- 1.3. Methods for solving the consolidation problem and pro-
gration cost [13, 11, 14, 15], resource fragmentation [12, 16], posed approach
number of SLA violations [11], etc. Typically, minimization of To solve the multidimensional consolidation problems re-
the number of active hosts is a primary goal of consolidation, searchers mainly utilize three methods: 1) integer linear pro-
however taking migration cost into account is also crucial. In- gramming (ILP), 2) metaheuristics, and 3) heuristics. ILP
deed, large number of migrations introduce undesirable loads formulations obtain the optimal solution only for the small
on the management side of a datacenter. To avoid overloads, a problem instances because of the exponential time complex-
balance between migration cost and the number of active hosts ity of solvers [26, 27, 28]. Metaheuristics work faster than
must be preserved. ILP solvers and provide approximate solutions for larger in-
Consolidation relies on two major approaches: static and stances. Such metaheuristics algorithms include: genetic
dynamic [9]. Static consolidation reserves resources for a VM algorithms[29, 30], ant colony optimization [16, 31], particle
based on its maximal consumption, which is specified in the fla- swarm optimization [32], artificial bee colony [33], etc. Typ-
vor. Dynamic consolidation assigns VMs to hosts using either ically, the performance of metaheuristics is worse than that of
their current or predicted, by statistical model, resource require- the heuristics in terms of the solvable instance size, parameters
ments of VMs. These requirements are usually lower than the dependence, and time complexity[34].
flavor size. In other words, static consolidation guarantees the Heuristics for the consolidation problem practically used
availability of the resources on the host for the clients, whereas due to time efficiency, small number of parameters, and imple-
dynamic consolidation provides denser packing at the cost of mentation simplicity. The majority of migration-aware heuris-
possible SLA violations – if demand of VMs on a host rises to tics rely on this common approach: sort hosts, sort VMs, and try
a higher level. to place largest VM into the most loaded host [14, 11, 35, 12].
Furthermore, algorithms for consolidation are divided into If the VM does not fit into a host because of a lack of free space,
centralized [9, 10] and decentralized [17, 18, 19] depending on then the heuristic skips the host and try the next one. We call
the cloud architecture. In centralized algorithms, any host pos- such skipping heuristics that exploit only available free space –
sesses full information about the entire cluster. In decentralized ”Sercon-like”. The heuristics differ in the criteria for sorting of
algorithms, all hosts pursue the same goal, however each host d-dimensional objects, additional classifications, and algorithm
has information only about its neighborhood. Consolidation for parameters. Murtazaev and Oh performed sorting using ”surro-
centralized architectures works faster and provides better re- gate weight” and introduced parameters for the maximum num-
sults, whereas decentralized architectures are free from a single ber of migrations and minimum migration efficiency [14]. Fer-
point of failure and have better scalability. reto et al. used lexicographic order for sorting and held steady
VMs[11]. Rao and Thilagam proposed hosts classification into
1.2. d-dimensional vector bin packing receivers and donors based on the theoretical minimum of re-
Consolidation of VMs is closely related to the well-known quired hosts and introduced a defragmentation procedure to re-
d-dimensional vector bin packing (d-VBP) problem. In the d- duce resource fragmentation after consolidation [12].
VBP problem, items are represented as d-dimensional vectors The main disadvantage of Sercon-like heuristics becomes
which are packed into the fewest number of d-dimensional bins evident in consolidation problems with 2 resources. The heuris-
such that: 1) all items are placed 2) the sum of items in each tics are poorly adapted to solving instances with imbalanced
bin is less than or equal to the bin capacity in all dimensions. hosts lacking free space for one of the resources (see Figure 2a,
The d-VBP problem is known to be an NP-hard problem which RAM limits Host 2, whereas CPU limits Host 3). Sercon-like
is exponentially hard to solve [20]. However, if d = 1, there heuristics are unable to free any of the hosts from the assign-
are two types of restrictions of the d-VBP problem that are of- ment demonstrated in Figure 2a. However, migrations of yel-
ten applicable to consolidation and lead to optimal solutions in low and green VMs between Host 2 and Host 3 increase free
polynomial time: 1) divisible items – in a list of sorted items space and release Host 1 (see Figure 2b).
each item is a divisor of next items – can be optimally packed In this paper, we propose the BalCon algorithm for solving
with the First Fit Decreasing heuristic [21], 2) high-multiplicity 2D centralized consolidation problems with a controlled trade-
VBP problem – items form a few groups and the items in each off between the number of released hosts and the amount of
group are identical – can be solved in polynomial time [22]. To migrated memory. The key component of BalCon is the Force
the best of our knowledge, the d-VBP problem for d , 1 is still Step, which places a VM into the host with a lack of free space
exponentially complex even with the above restrictions. using induced migrations. For instance, Figure 2c illustrates
The hardness of the d-VBP problem requires the use of two Force Steps that are required to consolidate the initial as-
heuristics to obtain a solution in reasonable time [23, 24, 25]. signment (Figure 2a) into the optimal assignment (Figure 2b).
Unfortunately, the d-VBP misses objective of minimizing of To efficiently place 2D VMs, we introduce the balance factor to
migration cost, which restricts its application to migration- classify the cluster state and choose one of three heuristic for
aware consolidation [14]. Application of d-VBP heuristics to VM placement. To evaluate BalCon we use two datasets: real
2
VM assignments from Huawei Cloud datacenters and synthet- 2. capacity of all hosts is satisfied in both (CPU and RAM)
ically generated hard instances for the purpose of stress test- dimensions:
P P
ing. Also we introduce ILP models and a modified the Ser- ∀h ∈ H, v.cpu ≤ h.cpu ∧ v.mem ≤ h.mem
v∈h v∈h
con heuristic for comparison. The results indicate the supe-
rior performance of the BalCon towards solving consolidation Our goal is to consolidate the VMs of the cluster. Given
problems under resource imbalance. The code with the imple- an initial feasible mapping µ0 of the VMs to the hosts, we try
mentation of BalCon and the synthetic datasets are presented in to compute a different feasible mapping µb that minimizes the
https://github.com/andreigudkov/BalCon. number of active hosts and the amount of migrated memory.
This paper is organized as follows. In Section 2, we for-
Definition 2. For a given mapping µ, host h is called active if
mulate the cloud consolidation problem. In Section 3, we in-
it contains at least one VM v. ∃v : µ(v) = h
troduce integer programming models that were used to obtain
optimal solutions and lower bounds for performance evaluation. Definition 3. For a given mapping µ, a VM v is called migrated
In Section 4, we provide a description of the BalCon algorithm, if it changes host: µ(v) , µ0 (v)
including the basic ideas (as shown in the diagram in Section
4.1) and the formal pseudo code (as presented in Section 4.3). To control our preference between the number of active
In Section 5, we describe the datasets and compare BalCon with hosts A(µb ) and the amount of migrated RAM M(µb , µ0 ), we
Sercon and optimal solutions. In Section 6, we conclude the pa- minimize the following linear objective function:
per and propose possible directions for future work.
Ob j(µb , µ0 ) = wa · A(µb ) + wm · M(µb , µ0 ) (1)
a)
6
b)
6 where wa , wm are non-negative weights. It is important to note
5 ForceFit 5 that the resulting mapping of the minimization problem de-
of Host 1
4 4
pends on the ratio of the weights instead of their absolute val-
ues. To investigate the algorithms performance under different
f
of

Induced
3 3
er

migration
w

to Host 2 objective functions, we introduce the maximal memory for mi-

Po

2 2

1
Induced
1
gration per host
migration
to Host 3
0 CPU RAM 0 wa
Host 1
CPU RAM
Host 2
CPU RAM
Host 3
CPU RAM
Host 1
CPU RAM
Host 2
CPU RAM
Host 3 MPH = TiB. (2)
c) Initial Optimal wm
1 2 3 The parameter is used for evaluation (as described in Section
Force step Force step To
to host 2 to host 3 host 2 5.2). Additionally, in our algorithm, if the migration cost to re-
To To lease any host exceeds MPH, then releasing the host decreases
host 1 host 1
Host 1 Host 2 Host 3 Host 1 Host 2 Host 3 Host 1 Host 2 Host 3 Host 1 Host 2 Host 3 the objective function. In the case of the classical VBP prob-
lem, MPH = ∞ TiB.
Figure 2: A consolidation problem instance with two resources (CPU, RAM)
which is solvable by BalCon and unsolvable by Sercon-like heuristics. The
problem involves four color-coded VMs that need to be assigned to three hosts, 3. Integer programming solutions
each with a capacity of (6, 6) a.u. Panel a) shows the initial assignment of VMs
to hosts. Panel b) shows the optimal assignment of VMs to hosts. Panel c) 3.1. Allocation model
shows the sequence of actions taken by the BalCon algorithm to optimize the
initial assignment panel a) to the final assignment panel b). Bartók and Mann [26] presented a straightforward Integer
Linear Programming (ILP) formulation of the similar consol-
idation problem. However, our problem statement differs in
the calculation of migration cost. Their formulation counts
2. Problem statement
the number of migrated VMs, whereas we compute the total
In our scenario, we operate a cluster with sets of physical amount of migrated memory. The updated ILP allocation model
hosts h ∈ H, VMs v ∈ V, and flavors f ∈ F. Every host h pro- is expressed in our terms as follows.
vides a pair of integer resources for scheduling: CPU (cores) The binary variables are:
and RAM (tebibytes). We denote the capacity of host h as Allocv,h = 1 if v is allocated on h and 0 otherwise
(h.cpu, h.mem), and the capacity of other resources is denoted
in the same way. The demand for resources of a VM v is de- Activeh = 1 if h is active and 0 otherwise
fined in a flavor f , that is, (v.cpu, v.mem) = ( f.cpu, f.mem). Migrv = 1 if v is migrated and 0 otherwise
Each host h reserves resources for the allocated VM v.
With these variables the objective is to
Definition 1. A mapping µ is the assignment of VMs to hosts, X X
where some VMs can be unassigned. The mapping µ is feasible minimize wa Activeh + wm v.mem · Migrv . (3)
h∈H v∈V
if:
In other words, our objective is to minimize the number of
1. every VM is assigned to exactly one host. active hosts and the amount of migrated memory, subject to the
following constraints:
3
 Table 1: Comparison of the models by domain of variables and number of
constraints.
X
Allocv,h = 1 ∀v ∈ V (4) Model Domain of variables Number of constraints
h∈H
Allocation |V||H|+|V|+|H| |V||H| + 2|H| + 2|V|
X B
v.cpu · Allocv,h ≤ h.cpu ∀h ∈ H (5)
v∈V
X Flavor flow B|H| × Z2|F||H| 2|F||H| + 2|H| + |F|
v.mem · Allocv,h ≤ h.mem ∀h ∈ H (6) Relaxed Flavor flow
v∈V (Lower Bound)
B|H| × R2|F||H| 2|F||H| + 2|H| + |F|
Allocv,h ≤ Activeh ∀(v, h) ∈ V × H (7)
Migrv = 1 − Allocv,µ0 (v) ∀v ∈ V (8)
Constraint 10 ensures that the net flow of VMs is zero for
Constraint 4 requires that every VM is scheduled on exactly each flavor f . In other words, the number of VMs migrated out
one host (Definition 1.1). Resource Constraints 5 and 6 en- from all hosts must be equal to the number of VMs migrated
sure that the CPU and RAM capacities of the hosts are satisfied onto all hosts for each flavor f . Constraint 11 forbids mov-
(Definition 1.2). Constraint 7 guarantees that hosts with VMs ing out of a host more VMs of a flavor f than were initially
are active (Definition 2). According to Constraint 8, a VM is present on the host. For inactive hosts, all VMs must be moved
migrated if it changes its host (Definition 3). out according to Constraint 12. Resource Constraints 13 and
14 ensure that the CPU and RAM capacities of each host are
3.2. Flavor flow model satisfied.
We introduce the flavor flow model that takes advantage The flavor flow model has asymptotically |V|/|F| fewer vari-
of the fewer number of flavors than VMs. The allocation ables than the allocation model, which allows for solving much
model uses |H| variables for each VM, however many of the larger problem instances when |F| ≪ |V| – a typical real cloud
VMs are of the same flavor and have equal resource demands situation.
( f.cpu, f.mem). Permutation of these similar VMs leads to the
same values of objective function and an undesirable symme- 3.3. Lower bounds
try. Instead of defining the exact location for every VM, we
Allocation
keep track on the flow of the VMs of the same flavor between
hosts. Such a flavor flow model breaks the symmetry of the Relaxed
Flavor flow
allocation model because all VMs of the same flavor are equal.
Formal definition of the flavor flow model requires defining
the following variables: Figure 3: Comparison of the ILP models on progressively larger instance sizes.
The problem instances of different sizes were solved by the CBC solver for 10
In f,h ∈ Z is the number of VMs of flavor f migrated into minutes each. The mean number of flavors per problem instance was 27.
host h
We detected that the running time of ILP solvers is sensitive
Out f,h ∈ Z is the number of VMs of flavor f migrated out
to the value of MPH. When MPH is close to 0 or infinity,
from host h
solvers manage to find the optimal solutions µopt in a reasonable
Activeh = 1 if h is active and 0 otherwise time. Unfortunately, for intermediate values of MPH, solvers
are unable to provide the optimal solutions even in a few days.
With these variables the objective is to Nevertheless, relaxation of the flavor flow model allows one
X X to obtain a lower bound (LB) µLB on the optimal solution for
minimize wa Activeh + wm f.mem · Out f,h . (9) the intermediate MPH values. To perform the relaxation, we
h∈H f ∈F,h∈H
kept the |H| variables Activeh as binary and relaxed the domain
Let n f,h be the number of VMs of flavor f initially located of the 2|F||H| variables In f,h and Out f,h to R. This relaxation
on host h, according to the mapping µ0 . Then the constraints of significantly simplifies the flow model for the solver and sets
the flavor flow model are: the lower bound on the optimal solution

Ob j(µLB , µ0 ) ≤ Ob j(µopt , µ0 ).
X X
Out f,h = In f,h ∀f ∈ F (10) (15)
h∈H h∈H
The comparison of the allocation, flavor flow, and relaxed
Out f,h ≤ n f,h ∀( f, h) ∈ F × H (11) flavor flow models in terms of variable domain and number of
n f,h · (1 − Activeh ) ≤ Out f,h ∀( f, h) ∈ F × H (12) constraints is presented in Table 1. To compare the performance
X of the models, we searched for the optimal solution of problem
f.cpu·(n f,h + In f,h − Out f,h ) ⩽ h.cpu· Activeh ∀h ∈ H (13) instances with different sizes using the CBC solver [36] for 10
f ∈F minutes (see Figure 3). As expected, the flavor flow models
X
f.mem·(n f,h + In f,h − Out f,h ) ⩽ h.mem· Activeh ∀h ∈ H (14) provided the optimal solutions for much larger instances than
f ∈F the allocation model.

4
4. Algorithms Section 4.2) and chooses one of two heuristics with Force Steps:
ForceFitBalanced or ForceFitLopsided. These heuristics pick a
4.1. Basic description of BalCon destination host h and free space for a given VM v by ejecting
The BalCon algorithm was created to cope with imbalanced VMs from h into the stash. This procedure repeats and VMs
situations in a datacenter. The need for such an algorithms from the stash are assigned to new or their original hosts. The
arises from the poor performance of Sercon-like heuristics in emptying of the stash indicates the generation of a new feasible
terms of host balancing. For example, Sercon-like heuristics mapping. In the new mapping some of the ejected VMs could
are unable to consolidate the imbalanced mapping from Figure migrate between active hosts.
2a because of insufficient amount of free space in the host for
one of the resources (RAM limits Host 2, whereas CPU limits Definition 6. A migration of a VM is called induced if it takes
Host 3). However, the consolidation is possible by shuffling the place between active hosts.
yellow and green VMs between Host 2 and Host 3 (see Figure Using induced migrations BalCon solves the problem of de-
2b). The shuffling in the BalCon is implemented with Force ficiency of free space in a datacenter. Induced migrations allow
Steps, which free space in a host by ejecting VMs from the host one to shuffle VMs and release the host, at the cost of an in-
into a temporary buffer (stash S ). The VMs from the stash are crease in the amount of migrated memory.
later moved into other hosts, which leads to induced migrations.
To perform the consolidation in the example, BalCon uses two a) Stop Start
b) 1.00s 1.36s
Yes No Choose smallest host
Force Steps. The sequence of actions for the consolidation is Are by migration cost (Def. 4)
presented in Figure 2c, where Host 1 illustrates the role of the all hosts
tried? Place VMs from host
into a stash S pcap(s,H)
stash. In the first step, to place the red VM to Host 2 we eject the Yes = 1.36s
No In the S choose the largest
green VM from Host 2. In the second step, to place the green VM based on Eq. 16
VM to Host 3 we eject the yellow VM from Host 3. Finally, we Does
Classes: CPU RAM
VM fit into free space Ample Stash
place the yellow VM into the free space of Host 2. As a result, Is of any host? Balanced
S empty? No Lopsided cap(s,H) = 0.99s
we released Host 1 and induced two migrations: 1) the yellow Yes
Place VM Are 0.25s 0.24s 0.50s
VM was migrated from Host 3 to Host 2 and 2) the green VM with BestFit hosts balanced?
No
was migrated from Host 2 to Host 3. Yes
Place VM with Place VM with
In general, BalCon is a greedy heuristic that attempts to re- ForceFitBalanced ForceFitLopsided
lease one active host in each step. The basic ideas of the algo- CPU RAM
Place ejected VMs into S CPU RAM CPU RAM CPU RAM Stacked
rithm are presented in Figure 4a, whereas a formal definition h1 h2 h3 hosts

will be given in Section 4.3. In each iteration, BalCon chooses

Figure 4: a) A diagram illustrates the workflow of the BalCon algorithm b)
the smallest host according to migration cost and places a VM An illustration of the calculation of cap (Eq. 20) and pcap (Eq. 21) for the
from the host into the stash S . balance factor. The stash resource vector s (red color) represents a unit vector
to measure the amount of the free space. The green color corresponds to the
Definition 4. The migration cost of VM v is v.mem, and the occupied space in the hosts by VMs.
P
migration cost of host h is w.mem, where w are unmigrated
w∈h
VMs: µ0 (w) = h.
4.2. Balance factor
Definition 5. The stash S is a temporary buffer for VMs v with We introduce the balance factor as a measure of the distri-
a resource vector s = v∈S v.cpu, v∈S v.mem
P P

bution of free space in a datacenter. The factor allows us to es-
timate the potential for VM shuffling operation to increase VM
For the success of a host release, S must be emptied. The
placements. In each step, BalCon attempts to fit the stash S into
BalCon takes the largest VM from the stash and uses the Best
the hosts. Therefore, the distribution of free space between the
Fit heuristic until it encounters a problem with free space on
hosts is improved if more stashes can be accommodated by the
all the hosts. The largest VM v is defined using the following
hosts. The balance factor is defined as the ratio of the current
formula:
amount of stashes in the free space of the hosts cap(s, H) to the
v.cpu v.mem potential amount of stashes in the combined free space of all
|v| = P + P . (16)
w.cpu w.mem hosts pcap(s, H)
w∈V w∈V

The free space of a host h is defined by the mean of its load and cap(s, H)
BF(s, H) = , (19)
allocated VMs v ∈ h pcap(s, H)

X X
 where
load(h) = 

v.cpu, v.mem (17) !
X free(h).cpu free(h).mem
v∈h v∈h cap(s, H) = min , (20)
s.cpu s.mem
free(h) = (h.cpu − load(h).cpu, h.mem − load(h).mem) (18) h∈H
 
X free(h).cpu X free(h).mem 
In the case of a lack of free space on the hosts, BalCon clas- pcap(s, H) = min  ,  . (21)
s.cpu s.mem 
sifies the situation using the balance factor (defined formally in h∈H h∈H

5
 The domain of BF(s, H) is [0, 1], where the value of 1 cor- Listing 1: High-level BalCon algorithm structure
responds to a purely balanced situation without potential for Input : H is the list of hosts
improvement, whereas a value of 0 corresponds to a purely lop- V is the list of VMs
sided situation that can be improved. In the balanced case most µ0 is the initial feasible mapping
hosts reach their limit by the same resource, unlike in the lop- Global Parameters: α = 0.95
sided case where some hosts are limited by cpu and some by b = 4000
mem. γ=3
Note that both pcap(s, H) and cap(s, H) provide relaxed Output : At the end, µb is a feasible
values that ignore the actual sizes of separate VMs in the stash. mapping of VMs to hosts
Thus, pcap(s, H) and cap(s, H) set upper bounds on the amount 1 Procedure BalCon(H, V, µ0 )
of stashes in any mapping and current mapping respectively. 2 µb := copy µ0
For instance, if cap(s, H) < 1 the allocation of all VMs from 3 H := sort H by migration cost
the stash is impossible without induced migrations. Similarly, 4 for h in H do
if pcap(s, H) < 1 the stash is implacable into the hosts by any 5 µtmp := copy µb
mapping. To distinguish between Balanced and Lopsided sit- 6 S := get VMs from h according to µtmp
uations, we introduce the parameter α. If BF(s, H) < α or 7 µtmp := Unassign all VMs in S from µtmp
cap(s, H) < 1 the situation is Lopsided and Balanced other- 8 A := get active hosts from µtmp
wise. In practice we use α = 0.95. 9 µtmp := ForceFit(S , A, µtmp )
An example of capacities calculation is presented in Fig- 10 if µtmp is f easible and
ure 4b. The stash with a resource capacity of s = (8.0, 4.0) Ob j(µtmp , µ0 ) ≤ Ob j(µb , µ0 )
is used as a unit vector of free space. Additionally, there are 11 µb :=µtmp
three partially filled hosts with equal capacities of (6.0, 6.0)
12 return µb
and free spaces free(h1 ) = (5.0, 1.0), free(h2 ) = (1.9, 5.0), and
free(h3 ) = (4.0, 2.0). In total, the hosts can allocate at most

! ! The BalCon sorts hosts by migration cost (see Definition 4) and

5.0 1.0 1.9 5.0 attempts to release the hosts one by one (rows 3-4). To release
cap(s, H) = min , + min , +
8.0 4.0 8.0 4.0 host h, the procedure searches for a better feasible mapping µtmp
(22)
in rows 5-9. In the better mapping, h has to be turned off, and
!
4.0 2.0
+ min , = 0.25 + 0.24 + 0.50 = 0.99 VMs from h are reassigned to active hosts A. The VMs from
8.0 4.0
h are placed into the stash S and unassigned from the mapping
of the stashes without induced migrations. Potential capacity is µtmp in rows 6-7. In rows 9-11 ForceFit procedure tries to fit
computed by combining the three hosts together that results in VMs from the stash S into active hosts A. If a new mapping
µtmp is feasible and leads to an improvement of the objective
function, then host h is successfully released (rows 10-11).
5.0 + 1.9 + 40 1.0 + 5.0 + 2.0
!
pcap(s, H) = min , = The ForceFit heuristic is presented in Listing 2. In rows 2-3,
8.0 4.0 (23) we define two variables for the procedure: ForceS teps and p.
= 1.36 The integer variable ForceS teps is used to control the number
of iterations in the while loop (rows 4-19). To avoid coming
of the stashes sizes. Finally, we compute the balance factor back to previously-visited solutions, we prohibit a host from
BF(s, H) = cap(H)/pcap(H) = 0.73. The fact that the BF is being selected more than γ times in a row during Force Steps.
lower than α = 0.95 indicates a resource imbalance and poten- The state information is maintained in the object instance p.
tial for improvement by induced migrations. The ForceFit heuristic works until the stash is empty or the
Force Steps limit is reached. In each step, ForceFit tries to in-
4.3. The BalCon algorithm sert the largest VM v (Eq. 16) from the stash into the destina-
The core structure of our algorithm is presented in the List- tion host h. The choice of the destination host depends on the
ing 1. There are three global parameters in the algorithm: cluster state, which is determined by the Classify procedure
(rows 22-29). The procedure implements ideas from Section
• α is used as a threshold to classify imbalanced 4.2 and classifies the state into three classes: Ample (Section
(BF(s, H) < α) and balanced (BF(s, H) ≥ α) situations 4.4), Balanced (Section 4.5), and Lopsided (Section 4.6). The
using Eq. 19. last two classes use Force Steps and differ in their approach to
• b is the maximal number of Force Steps to try during host choose the destination host h and VMs Ve to eject from h.
release. The choice of VMs Ve influences the number of induced
migrations and the amount of migrated memory at the end of
• γ is the maximal number of tries of the same destination ForceFit. To reduce the amount of migrated memory, we pre-
host in a row. fer to eject VMs Ve that were migrated to the destination host
h during previous steps of the algorithm. The shuffling of these

6
 Listing 2: ForceFit heuristic of hosts are low in the same resource. Such scenarios makes
the situation closer to a one-dimensional problem, where a
1 Procedure ForceFit(S , A, µ)
smaller VM is easier to place than a larger one. Therefore,
2 ForceS teps := 0
ChooseHostBalanced (row 12) chooses the destination host
3 p := RepeatsProhibitor(A, γ)
h with the greatest number of VMs smaller than v using Eq. 16.
4 while S is not empty and ForceS teps < b do
Afterwards, ForceFitBalanced (row 13) sorts the VMs in
5 v := peek largest v in S
h according to lexicographical order:
6 class := Classify(S , A, µ, v)
7 S := remove v from S 1. Prefer VMs previously migrated to h.
8 if class is ”Ample”
9 µ := BestFit(v, A, µ) 2. In case of a tie, prefer VMs with smaller migration cost.
10 if class is ”Balanced”
11 ForceS teps +=1 Next, ForceFitBalanced iterates over the sorted list of VMs
12 h, p := ChooseHostBalanced(v, A, µ, p) and excludes them from h one by one until there is enough free
13 µ, Ve := ForceFitBalanced(v, h, µ) space in h for v. After placing v into h, the procedure tries to
14 S := add all VMs from Ve to stash S return the excluded VMs which fit into h in reverse order. The
15 if class is ”Lopsided” remaining VMs are returned as Ve (rows 15).
16 ForceS teps +=1
17 h, p := ChooseHostLopsided(v, A, µ, p) 4.6. Lopsided class
18 µ, Ve := ForceFitLopsided(v, h, µ) In the Lopsided class (Listing 2 rows 16-19) we also need
19 S := add all VMs from Ve to stash S to eject some VMs from the destination host h into the stash
20 return µ to place a given VM v. However, unlike the balanced case,
21
both resources limit VMs placement. The Lopsided class is a
22 Procedure Classify(S , A, µ, v) key tool of the BalCon from a balancing point of view. The
23 if ∃h ∈ A : v fits h balance of the whole datacenter improves by the choices of the
24 return ”Ample” destination host and VMs to eject for future induced migrations.
25 cap := Capacity(S , A, µ) Choice of destination host:
26 pcap := PotentialCapacity(S , A, µ) a) 100 b) 100
27 if cap < 1 or cap < α · pcap h1 h2 h1 h2
28 return ”Lopsided” 75
h3 h3
75
return ”Balanced”
CPU (%)

CPU (%)
29
50 h4 50 h4
v1 v2
25 h5 25 h5
migrated VMs in ForceFit is unable to increase the objective
function. In other words, only new migrations in ForceFit in- v3
0 0
0 25 50 75 100 0 25 50 75 100
crease the objective function compared to the best step of Bal- MEM(%) MEM(%)
Con µb . In terms of memory, improving the objective func-
Choice of VMs for ejection (prefer VMs in red zone):
tion (row 10 of Listing 1) requires the amount of new mem- c) d) 100
100
ory for migration during ForceFit to be no more than MPH h1
(M(µtmp , µ0 ) − M(µb , µ0 ) ≤ MPH). ed
fer
75 75 pre
CPU (%)

CPU (%)

d
4.4. Ample class 50 fere 50
pre
In the Ample class one or more hosts have enough free
25
v2 h5 25
v2
space to accommodate a VM. We employ a Best Fit heuristic
to allocate VM v. The destination host h is chosen based on 0 0
surrogate load 0 25 50 75 100 0 25 50 75 100
MEM(%) MEM(%)
load(h).cpu load(h).mem
|h| = + . (24) Figure 5: Illustration of choices of destination host and VMs for ejection in
h.cpu h.mem the Lopsided class. a-b) Illustrate the choice of the destination host (h1 , h5 )
Among all hosts with enough free space, we assign a VM depending on the VM (v1 , v2 , v3 ) to place. In a) the choice is explicit, and the
destination host has the opposite load angle (for v1 is h5 and for v3 is h1 ),
to the host with the highest load. whereas in b) the choice is ambiguous and depends on the previous steps (v2
can be allocated on h1 and h5 ). c-d) Illustrates choices of VMs for ejection.
4.5. Balanced class The preferable VMs are in the red areas that are located on the same side from
the VM as the destination host (h1 or h5 ).
To place a VM v in the Balanced class (Listing 2 rows
11-14), some other VMs must be ejected and placed into the Figure 5 presents an example of the resource requirements
stash. The balanced situation is special because the majority of the VMs and the loads of the hosts. Some hosts lack CPU but
7
have plenty of memory (h1 ), some hosts lack memory but have 1. In Sercon, the authors limited the total number of allowed
plenty of CPU (h4 , h5 ), and some are low on both resources. To migrations over all hosts. The SerconModified heuristic
reduce imbalance, we prefer to place a given VM into the host limits the amount of migrated memory in each algorithm
with minimal or maximal load angles (in example h1 or h5 ). step. A step is accepted if the new amount of migrated
v.cpu memory in the step is no more than MPH (Listing 1 row
Definition 7. The load angle of a VM is arctan v.mem , and the 10).
load(h).cpu
load angle of a host is arctan load(h).mem .
2. Unlike Sercon, we try to release each host once.
To choose the destination host h, the procedure
ChooseHostLopsided (row 17) uses the global variable 3. The destination host for a VM in the original version is
r ∈ {”mem”, ”cpu”} and follows rules: chosen with First Fit, whereas we use Best Fit. Also, we
choose the destination host among all active hosts.
1. If the load angle of v is the largest or smallest among the
load angles of all the hosts, choose the destination host 4. We omitted the migration efficiency parameter.
with the most opposite load angle and set r to the host’s
The second modification leads to |H| times less complexity
largest resource.
of SerconModified compared to SerconOriginal (see Table 2),
For instance, in Figure 5a, v1 has the largest angle and
where Vmax is the maximum number of VMs in a host. The
is moved to host h5 , which has the smallest angle (r set
complexity of BalCon is only b times worse than that of Ser-
to ”mem”). Similarly, v3 has the smallest angle and is
conModified. The upper bounds on Vmax can be |V|.
moved to h1 , which has the largest angle (r set to ”cpu”).

2. Else, switch r and choose the host with the largest load Table 2: Comparison of time complexity of the algorithms.
by the resource in r. Algorithm Time complexity
For instance, with v2 from Figure 5b, if r was ”cpu”, it SerconOriginal O(|H|2 · Vmax · log(Vmax ) + Vmax · |H|)


switches to ”mem”, and v2 is moved to h5 (host with the SerconModified O(|H| · Vmax · log(Vmax ) + Vmax · |H|)


largest ”mem” load). Otherwise, it switches to ”cpu” and BalCon O(b · |H| · Vmax · log(Vmax ) + Vmax · |H|)


is moved to h1 (host with the largest ”cpu” load).

The implementation of the ForceFitLopsided procedure

(row 18) is similar to that of the Balanced class, except it has an
5. Evaluation
additional requirements on the load angle of the VMs for ejec-
tion. To fix the destination hosts lopsidedness, the load angle 5.1. Datasets
of the VMs for ejection has to be close to the load angle of the
Algorithm evaluation was performed using Huawei and
host. The procedure follows lexicographical sorting of the VMs
synthetic datasets. Synthetic datasets was generated to test the
in h:
algorithms under complex inputs, whereas the Huawei datasets
1. Prefer VMs with a direction to the same side of v as h. validate algorithm performance on real data. The comparison
For instance, if h has a lower angle than v, we prefer VMs of the main characteristics of the datasets is presented in Table 3
with angles less than v (red zone Figure 5c). Otherwise, and Firgure 6. In general, synthetic problem instances are less
we prefer VMs with angle larger than v (red zone Figure balanced, have more VMs and more flavors than the Huawei
5d). datasets.

2. In case of a tie, prefer VMs which were previously mi- Table 3: Statistics of Huawei and synthetic datasets, where mean values are
grated to h. given per instance.
Instances Hosts VMs Flavors Balance factor
3. In case of a second tie, prefer VMs with a smaller migra- Dataset
Total Mean Mean Mean/Total Mean
tion cost. Huawei 555 96 1.0k 8 / 69 0.99
Synthetic 200 50 2.7k 28 / 30 0.07
After sorting, the ForceFitLopsided procedure frees
space in h for v exactly like in ForceFitBalanced and returns
The Huawei dataset contains 555 cluster snapshots from an
the VMs for ejection Ve in row 18.
operational Huawei Cloud. The snapshots were gathered from
clusters of various sizes and roles. The flavor distribution over
4.7. Modified Sercon heuristics and time complexity
all snapshots is given in Figure 6a, whereas the distribution of
To compare BalCon with well-known approaches, we mod- hosts within a range of the number of VMs is presented in Fig-
ified the Sercon heuristic to fit our objectives. The SerconMod- ure 6c.
ified heuristic derives from BalCon if we forbid using Force To generate a complex synthetic dateset, we created in-
Steps. The differences between the original Sercon heuristic stances with severe lopsidedness of resources. Briefly, the VMs
and the SerconModified heuristic include: were generated from 30 flavors with decreasing probabilities
by CPU size (see Figure 6b). Then, the VMs were sorted by
8
 Huawei dataset Synthetic dataset
a) b) Table 4: The comparison of execution time between methods in the case of the
VBP problem and Huawei dataset.
Relaxed ILP Sercon Sercon
RAM (GiB)

RAM (GiB)
Method ILP Flow Model BalCon
Flow model (LB) Original Modified

Counts
Mean
4h 0.3s 9.5s 0.3s 0.1s
execution time

unable to solve any because of lack of free space of one of the

resources (see Figure 7b and Section 5.1). The iterative nature
CPU (cores) CPU (cores)
c) d) of the algorithms allows for easy implementation of a running
time limit. The time limit for all algorithms was set to 60 sec-
Hosts in range 103

Hosts in range 103

onds on one physical core of Intel Xeon E5-2690. However, on

average, BalCon solved instances in less time (see Table 4). Ad-
ditionally, the measured execution time is well-correlated with
the algorithms time complexity from Table 2.

a) Huawei dataset b) Synthetic dataset

Number of instances

Number of instances
Range of number of VMs Range of number of VMs BalCon MPH = ∞ (TiB)
BalCon MPH = 0.4(TiB)
BalCon MPH = 0.2(TiB)
Figure 6: The comparison between the Huawei (a,c) and synthetic (b,d) SerconModified MPH = ∞ (TiB)

datasets. Panels a-b present the flavor distribution over all problem instances, MPH = ∞ (TiB)
whereas panels c-d) demonstrate the number of hosts with VMs in the given BalCon
SerconModified
range. SerconOriginal

Gap(μalg, μopt) (%) Gap(μalg, μopt) (%)

the load angle (Def. 7) and packed into hosts using the First
Fit heuristic. Such packing method complicates the ability of Figure 7: The performance profiles of BalCon, SerconModified, and SerconO-
Sercon-like heuristic because of the lack of free space in hosts. riginal are compared to the optimal solution µopt of the Flavor flow model for
Also, the packing produces an imbalance that requires many the VBP problem when MPH = ∞ (TiB). The Gap is calculated using Eq. 25.
Panels a) and b) show the comparison for the Huawei and Synthetic datasets,
Force Steps in BalCon to place VMs with ForceFit. To vali- respectively.
date the lopsidedness of resources in our synthetic datasets we
calculated the mean balance factor. The balance factor of an To investigate the algorithms performance towards
instance from the datasets was calculated with Eq. 19, where migration-aware consolidation, we chose a few values of
s = |H|
1 P
h∈H (h.cpu, h.mem) is the mean hosts capacity in the in- MPH. Solvers with the Flavor flow model were unable to
stance. The mean balance factor (Table 3) of the initial feasible obtain optimal solutions for all MPH values in a reasonable
mappings in the Huawei instances is 0.99, which corresponds time. Therefore, we use the LB obtained with the relaxed
to a balanced situation. In contrast, for the synthetic data, the flavor flow model instead of the true optimum (Section 3.3).
mean balance factor is 0.07 (Table 3), indicating a highly lop- In the example of the VBP problem the execution time for the
sided situation. LB is significantly lower (see Table 4). The mean values of
Gap(µalg , µLB ) over non-trivial solutions – at least one host
5.2. BalCon performance is released – are presented in Figure 8. As expected from
We begin evaluation of algorithm performance with the ex- the VBP results, BalCon outperforms SerconModified on
treme case of the classical VBP problem, where the maximal the Huawei dataset and dominates on the Synthetic dataset,
memory for migration per host (Eq. 2) is infinite MPH = ∞ where SerconModified is unable to solve any instance. The
TiB. To compare BalCon with SerconModified (Section 4.7), Force Steps of BalCon allow to provide closer solutions to
we measured the gap between the values of the objective func- the LB than SerconModified on the Huawei dataset (Figure
tion of the algorithm solution µalg and the optimal solution µopt 8a) at large values of MPH from 1 TiB to 10 TiB. However,
when MPH reaches the capacities of the hosts ∼ 0.7TiB, the
Ob j(µalg , µ0 ) − Ob j(µopt , µ0 ) number of non-trivial instances decreases, and performance of
Gap(µalg , µopt ) = , (25) both algorithms equalizes, due to the insufficient memory for
Ob j(µ0 , µ0 ) − Ob j(µopt , µ0 )
induced migrations. Even at MPH = 0.2 TiB, Force Steps lead
where µopt were obtained with the Flavor flow model (Section to a little bit worse performance of BalCon. Qualitatively Bal-
3.2). The performance profile [37] indicates the advantage of con demonstrates similar operation in the Synthetic datasets,
BalCon over SerconModified, which optimally solved 535 and however the Gap to LB is larger and the number of non-trivial
345 instances respectively on Huawei dataset (see Figure 7a). solutions is more sensitive to MPH. The last facts are because
Also, the original version of Sercon (see Section 4.7) optimally of the high imbalance of the Synthetic datasets, which require
solved 245 instances. On the synthetic datasets BalCon op- additional amounts of memory for migration.
timally solved all the problem instances, whereas Sercon was In general, BalCon’s high performance is determined by
9
 a) Huawei dataset b) Synthetic dataset
438 519 554 555 555 555 57
be considered for further algorithm improvement. In general,
BalCon BalCon
17 the number of Force Steps and total migrated memory increase

Mean Gap(μalg, μLB) (%)

Mean Gap(μalg, μLB) (%)

Sercon
Modified Number of along with MPH. Also, on average BalCon demonstrates out-
Number of 8
non-trivial
non-trivial 107 solutions standing performance, especially in the case of large MPH and
solutions SerconModified
solves none imbalanced situations (Figure 8).
323 200

200 6. Conclusions
197 200 200

Maximal memory for Maximal memory for

We proposed the BalCon algorithm which efficiently solves
migration per host (MPH) (TiB) migration per host (MPH) (TiB) the migration-aware consolidation problems. The algorithm
was compared with a modified Sercon heuristic and ILP mod-
Figure 8: Comparison of BalCon and SerconModified with lower bound (LB) at els. The advantages of BalCon over Sercon-like heuristics were
different values of MPH. The mean Gap is calculated over non-trivial solutions
using Eq. 25. Panels a) and b) show the results on the Huawei dataset and
achieved due to Force Steps. The Force Steps allow BalCon to
Synthetic dataset, respectively. optimally solve imbalanced problem instances that Sercon-like
heuristics are unable to optimize. The performance of BalCon
is very close to optimal at large values of MPH. Time complex-
the Ample, Balanced, and Lopsided classes and their respec- ity of BalCon is only b times larger than that of modified Ser-
tive heuristics (see Sections 4.4 - 4.6). Other factors – such con heuristic, where b is the maximum number of Force Steps.
as cost functions for VMs ordering – are details of minor im- Note that the BalCon implementation is independent of flavor
portance. For instance, a random ordering of VMs in the stash set F and therefore directly applicable to dynamic consolida-
leads to less than 1% of Gap increase compared to ordering tion. Also, we used the amount of RAM as a migration cost,
with Eq. 16. To demonstrate the influence of MPH on Bal- however one can replace it with other metrics such as number
Con’s performance and areas for further improvement of the of VM migrations, predicted time for migration of VM, proba-
algorithm, we selected one representative instance from each bility of SLA violation, etc.
dataset (see Figure 9). The number of active hosts and total
migrated memory of SerconModified and BalCon coincide at
low MPH (Figure 9a). Then, at larger MPH, SerconModified 7. Data availability
reaches constant values, whereas BalCon releases more hosts at The data and code used in this article are available by the
the cost of a larger amount of migrated memory and improved link https://github.com/andreigudkov/BalCon.
objective function.
a) Huawei dataset b) Synthetic dataset 8. Declaration of competing interest
130
Mean number of Force steps
Total migrated memory (TiB)

Total migrated memory (TiB)

Sercon BalCon
Number of active hosts

Modified
Number of active hosts

BalCon The authors declare that they have no known competing fi-
nancial interests or personal relationships that could have ap-
peared to influence the work reported in this paper.

9. Acknowledgments
Maximal memory for Maximal memory for
migration per host (MPH) (TiB) migration per host (MPH) (TiB) Authors are gratefull to Mr. Oliver Slumbers for helping
with language polishing of this article. Dr. Stepan Romanov
Figure 9: Examples of BalCon and SerconModified performance at different thanks Dr. Mafuda for fruitfull discussions and organizational
MPH values. The evaluation is demonstrated in terms of the number of active help.
hosts, the amount of migrated memory, and the number of Force Steps. The
problem instances are from a) Huawei dataset and b) Synthetic dataset.
10. CRediT authorship contribution statement
Because of the greedy nature, BalCon sometimes performs
non-optimal steps by increasing the amount of migrated mem- The authors equally contributed to this work.
ory, the number of active hosts, or the number of Force Steps.
For instance, the number of active hosts is increased by one in References
the range of [3.4,3.8] TiB, which could be avoided using the
solution at 3 TiB (see Figure 9a). Also, at the same number [1] P. Barham, B. Dragovic, K. Fraser, S. Hand, T. Harris, A. Ho, R. Neuge-
bauer, I. Pratt, A. Warfield, Xen and the art of virtualization, ACM
of active hosts, the algorithm uses larger memory than it could SIGOPS operating systems review 37 (2003) 164–177.
in the range [5.0, 6.0] TiB because the better solution is at 4 [2] R. Uhlig, G. Neiger, D. Rodgers, A. L. Santoni, F. C. Martins, A. V. An-
TiB. The mean number of Force Steps changes, although the derson, S. M. Bennett, A. Kagi, F. H. Leung, L. Smith, Intel virtualization
number of active hosts remains the same for the range [2.8, 3.6] technology, Computer 38 (2005) 48–56.
[3] M. Armbrust, A. Fox, R. Griffith, A. D. Joseph, R. Katz, A. Konwin-
TiB (see Figure 9b). We would like to emphasize that those are ski, G. Lee, D. Patterson, A. Rabkin, I. Stoica, et al., A view of cloud
only examples of non-optimal BalCon operations, which might computing, Communications of the ACM 53 (2010) 50–58.

10
 [4] R. Buyya, C. S. Yeo, S. Venugopal, J. Broberg, I. Brandic, Cloud com- server consolidation problems in virtualized data centers, IEEE Transac-
puting and emerging it platforms: Vision, hype, and reality for delivering tions on services computing 3 (2010) 266–278.
computing as the 5th utility, Future Generation computer systems 25 [28] C. Ghribi, M. Hadji, D. Zeghlache, Energy efficient vm scheduling for
(2009) 599–616. cloud data centers: Exact allocation and migration algorithms, in: 2013
[5] P. Patel, A. H. Ranabahu, A. P. Sheth, Service level agreement in cloud 13th IEEE/ACM International Symposium on Cluster, Cloud, and Grid
computing (2009). Computing, IEEE, 2013, pp. 671–678.
[6] G. J. Mirobi, L. Arockiam, Service level agreement in cloud comput- [29] J. Xu, J. A. Fortes, Multi-objective virtual machine placement in virtu-
ing: An overview, in: 2015 International Conference on Control, In- alized data center environments, in: 2010 IEEE/ACM Int’l Conference
strumentation, Communication and Computational Technologies (ICCI- on Green Computing and Communications & Int’l Conference on Cyber,
CCT), IEEE, 2015, pp. 753–758. Physical and Social Computing, IEEE, 2010, pp. 179–188.
[7] A. Beloglazov, J. Abawajy, R. Buyya, Energy-aware resource allocation [30] H. Hallawi, J. Mehnen, H. He, Multi-capacity combinatorial ordering ga
heuristics for efficient management of data centers for cloud computing, in application to cloud resources allocation and efficient virtual machines
Future generation computer systems 28 (2012) 755–768. consolidation, Future Generation Computer Systems 69 (2017) 1–10.
[8] A. Ruprecht, D. Jones, D. Shiraev, G. Harmon, M. Spivak, M. Krebs, [31] A. Ashraf, I. Porres, Multi-objective dynamic virtual machine consol-
M. Baker-Harvey, T. Sanderson, Vm live migration at scale, ACM SIG- idation in the cloud using ant colony system, International Journal of
PLAN Notices 53 (2018) 45–56. Parallel, Emergent and Distributed Systems 33 (2018) 103–120.
[9] R. W. Ahmad, A. Gani, S. H. A. Hamid, M. Shiraz, A. Yousafzai, F. Xia, [32] H. Li, G. Zhu, C. Cui, H. Tang, Y. Dou, C. He, Energy-efficient migration
A survey on virtual machine migration and server consolidation frame- and consolidation algorithm of virtual machines in data centers for cloud
works for cloud data centers, Journal of network and computer applica- computing, Computing 98 (2016) 303–317.
tions 52 (2015) 11–25. [33] J. Jiang, Y. Feng, J. Zhao, K. Li, Dataabc: A fast abc based energy-
[10] A. Verma, P. Ahuja, A. Neogi, pmapper: power and migration cost aware efficient live vm consolidation policy with data-intensive energy evalua-
application placement in virtualized systems, in: ACM/IFIP/USENIX tion model, Future generation computer systems 74 (2017) 132–141.
international conference on distributed systems platforms and open dis- [34] R. Ponto, G. Kecskeméti, Z. Á. Mann, Comparison of workload consoli-
tributed processing, Springer, 2008, pp. 243–264. dation algorithms for cloud data centers, Concurrency and Computation:
[11] T. C. Ferreto, M. A. Netto, R. N. Calheiros, C. A. De Rose, Server con- Practice and Experience 33 (2021) e6138.
solidation with migration control for virtualized data centers, Future Gen- [35] Y. Ho, P. Liu, J.-J. Wu, Server consolidation algorithms with bounded
eration Computer Systems 27 (2011) 1027–1034. migration cost and performance guarantees in cloud computing, in: 2011
[12] K. S. Rao, P. S. Thilagam, Heuristics based server consolidation with Fourth IEEE International Conference on Utility and Cloud Computing,
residual resource defragmentation in cloud data centers, Future Genera- IEEE, 2011, pp. 154–161.
tion Computer Systems 50 (2015) 87–98. [36] J. Forrest, T. Ralphs, H. Santos, et al., coin-or/cbc: Release releases/2.10.
[13] F. Hermenier, X. Lorca, J.-M. Menaud, G. Muller, J. Lawall, Entropy: 8, 2022.
a consolidation manager for clusters, in: Proceedings of the 2009 ACM [37] E. D. Dolan, J. J. Moré, Benchmarking optimization software with per-
SIGPLAN/SIGOPS international conference on Virtual execution envi- formance profiles, Mathematical programming 91 (2002) 201–213.
ronments, 2009, pp. 41–50.
[14] A. Murtazaev, S. Oh, Sercon: Server consolidation algorithm using live
migration of virtual machines for green computing, IETE Technical Re-
view 28 (2011) 212–231.
[15] Q. Wu, F. Ishikawa, Q. Zhu, Y. Xia, Energy and migration cost-aware dy-
namic virtual machine consolidation in heterogeneous cloud datacenters,
IEEE transactions on Services Computing 12 (2016) 550–563.
[16] Y. Gao, H. Guan, Z. Qi, Y. Hou, L. Liu, A multi-objective ant colony sys-
tem algorithm for virtual machine placement in cloud computing, Journal
of computer and system sciences 79 (2013) 1230–1242.
[17] M. Marzolla, O. Babaoglu, F. Panzieri, Server consolidation in clouds
through gossiping, in: 2011 IEEE International Symposium on a World
of Wireless, Mobile and Multimedia Networks, IEEE, 2011, pp. 1–6.
[18] E. Feller, C. Morin, A. Esnault, A case for fully decentralized dynamic
vm consolidation in clouds, in: 4th IEEE International Conference on
Cloud Computing Technology and Science Proceedings, IEEE, 2012, pp.
26–33.
[19] A. Ashraf, B. Byholm, I. Porres, Distributed virtual machine consolida-
tion: A systematic mapping study, Computer Science Review 28 (2018)
118–130.
[20] M. R. Garey, D. S. Johnson, Computers and intractability, A Guide to the
(1979).
[21] E. G. Coffman Jr, M. R. Garey, D. S. Johnson, Bin packing with divisible
item sizes, Journal of Complexity 3 (1987) 406–428.
[22] M. X. Goemans, T. Rothvoss, Polynomiality for bin packing with a con-
stant number of item types, Journal of the ACM (JACM) 67 (2020) 1–21.
[23] H. I. Christensen, A. Khan, S. Pokutta, P. Tetali, Approximation and
online algorithms for multidimensional bin packing: A survey, Computer
Science Review 24 (2017) 63–79.
[24] R. Aringhieri, D. Duma, A. Grosso, P. Hosteins, Simple but effective
heuristics for the 2-constraint bin packing problem, Journal of Heuristics
24 (2018) 345–357.
[25] R. Panigrahy, K. Talwar, L. Uyeda, U. Wieder, Heuristics for vector bin
packing, research. microsoft. com (2011).
[26] D. Bartók, Z. Á. Mann, A branch-and-bound approach to virtual machine
placement, in: Proceedings of the 3rd HPI cloud symposium “operating
the cloud, 2015, pp. 49–63.
[27] B. Speitkamp, M. Bichler, A mathematical programming approach for

11