Research ArticleA Robust Optimization Based Energy-Aware Virtual NetworkFunction Placement Proposal for Small Cell 5G Networks withMobile Edge Computing Capabilities

Bego Blanco Ianire Taboada Jose Oscar Fajardo and Fidel Liberal

Faculty of Engineering of Bilbao University of the Basque Country (UPVEHU) Alameda Urquijo sn 48013 Bilbao Spain

Correspondence should be addressed to Bego Blanco begonablancoehueus

Received 4 May 2017 Revised 8 August 2017 Accepted 28 August 2017 Published 12 October 2017

Academic Editor Wolfgang Kellerer

Copyright copy 2017 Bego Blanco et al This is an open access article distributed under the Creative Commons Attribution Licensewhich permits unrestricted use distribution and reproduction in any medium provided the original work is properly cited

In the context of cloud-enabled 5G radio access networks with network function virtualization capabilities we focus on the virtualnetwork function placement problem for a multitenant cluster of small cells that provide mobile edge computing services Underan emerging distributed network architecture and hardware infrastructure we employ cloud-enabled small cells that integratemicroservers for virtualization execution equipped with additional hardware appliances We develop an energy-aware placementsolution using a robust optimization approach based on service demand uncertainty in order to minimize the power consumptionin the system constrained by network service latency requirements and infrastructure terms Then we discuss the results of theproposed placement mechanism in 5G scenarios that combine several service flavours and robust protection values Once theimpact of the service flavour and robust protection on the global power consumption of the system is analyzed numerical resultsindicate that our proposal succeeds in efficiently placing the virtual network functions that compose the network services in theavailable hardware infrastructure while fulfilling service constraints

1 Introduction

In this new era of an always-connected and hypercommuni-cated society 5G targets a set of ambitious requirements interms of network performance-related key performance indi-cators such as very low delays and high data rates [1ndash3] Forthis reason from an operatorrsquos perspective finding a tradeoffbetween user performance improvement and CAPEXOPEXreduction becomes essential

In this context Cloud-Enabled Radio Access Networks(CE-RANs) become a feasible solution to enhance userexperience and decrease total costs [4] CE-RAN architectureleverages network function virtualization (NFV) [5ndash7] to splitthe radio protocol stack between physical and virtualizedfunctions the latter running in edge clouds This edge cloudmay also be used to place service instances at the edge of themobile network and in the proximity of end users Thus thisinclusion of edge cloud service capabilities in the network notonly improves quality of service but also simplifies deploy-ment and management in multitenant scenarios [8]

Through the deployment of Virtualized Network Func-tions (VNFs) on Virtual Machines (VMs) operators willexecute edge radio and service VNFs inside data-centers oncommodity hardware with the advantage of reducing capitalexpenses In this novel paradigm general-purpose comput-ing in networks has been realized along with the virtualiza-tion of network functions which enables the automation ofnetwork service provisioning and management In this workwe focus on an emerging 5G scenario amultitenant cluster ofsmall cells (SCs) that provides edge radio and service func-tions by means of virtualization technologies In particularwe employ cloud-enabled SCs that integrate microservers forvirtualization execution equipped with IT resources (CPURAM and storage) and additional hardware appliances suchas GPUs DSPs or FPGAs (Figure 1) Thus in this whole pic-ture we could define a network service (NS) as a collection ofVNFs required to deploy a complete 5G mobile service (eginnovative video content delivery at the edge integrated withradio monitoring and management) for the end users of theSC network operator

HindawiMobile Information SystemsVolume 2017 Article ID 2603410 14 pageshttpsdoiorg10115520172603410

2 Mobile Information Systems

SC small cell NF network function

PNF physical NF VNF virtualized NF

Mobile core network

PNF PNF PNF

Mobile edgecomputing

SC SC SC

Dist

ribut

ed cl

oud-

enab

led

radi

o ac

cess

net

wor

k

VNF VNF VNF

Figure 1 Cloud-enabled RAN architecture based on small cells

Nevertheless the success of the presented novel approachundoubtedly depends on the VNF placement strategy usedto allocate VNFs into the available resources given a set ofservice constraints However defining efficient mechanismsto allocate the software-based components of NS onto thenetworked data-centers (set ofmicroservers that conform thecloud-enabled SC cluster in our case) is a challenging taskTherefore the relevance of designing intelligent VNF place-ment algorithms becomes fundamental while it is notdefined in standards [9]

Besides current trends towards green information andcommunication technologies target the improvement ofenergy consumption in all the nodes including the networknodes themselves In this paper we will particularly discusshow to derive an energy-aware VNF placement solution fora cluster of cloud-enabled SCs in a 5G deployment sce-nario With this purpose we will deal with the objective ofminimizing power consumption of both the microserversand SDN-enabled switches to deploy the overall virtualizedinfrastructure needed (including virtual switches within eachSC and physical ones interconnecting SCs)

In order to address the VNF placement problem underthe aforementioned novel 5G scenario this paper proposesa placement mechanism that minimizes power consumptionsubject to bit rate and latency constraints imposed by thenetwork services and is aware of the available underlyingresources Existing literature provides a wide variety of VNFplacement approaches ranging from accurate Integer LinearProgramming (ILP) formulations to heuristic algorithmsHowever these solutions aremore focused on the use of NFVover big centralized data-centers rather than over small edgeclouds devoted to provide radio access and edge networkingservices In addition the analyzed studies rarely consideruncertainty in the definition of the problem

Hence the main contribution of this work is the designof an energy-aware VNF placement expert system for

a CE-RAN environment with 5G SCs We use constraintprogramming to solve the discrete optimization problemof resource assignment combined with robust optimizationtechniques (RO) [10] to develop an energy-awareVNFplace-ment policy Thus our approach differentiates from othersin the use of RO which allows us to generate a suboptimalplacement policy that enables the introduction of uncertaintyin the problem In particular we introduce service demanduncertainty in order to capture the variable nature of trafficflows size and per-service job execution burden

The paper has been organized in six sections Section 2reviews the related work In Section 3 we describe the VNFplacement problem under a 5G environment Section 4presents our placement proposal using the RO approach pre-viously providing the problem model Next in Section 5 weanalyze the performance of the proposed placement solutionFinally Section 6 gathers the conclusions of this work

2 Related Work

Nowadays mobile operators see network virtualization asthe key towards enabling flexible 5G network architectureFor this reason the problem of VNF placement has beenattracting increasing attention during the last years [11ndash25]Nonetheless the optimal placement of virtual elements tophysical resources aimed at minimizing power consumptionwith service performance constraints for the 5G scenariounder study is a complex issue

VNFplacement is a resource allocation discrete or combi-natorial optimization problem that can be considered equiv-alent to a knapsack problem In consequence it can be solvedusing exact methods or heuristics depending on the numberof VNF instances and the complexity of the resources incompetition As typically provided in the literature (see [12ndash14]) an ILP approach optimally solves the placement problemin reasonable time when the number of service instances islow In reference to heuristics-based placement proposals thevariety of tools employed for solving the problem is hugefor example [15] uses a greedy algorithm whereas [16] usesa binary search heuristic

Even though some of the aforementioned relevant worksdeal with energy-related optimization taking into accountdelay bounds (eg the objective in [12 13] is minimizing thenumber of CPUs constrained by latency) those solutions donot usually consider optimalsuboptimal power consumptionitself However it is worth mentioning the few works wehave found deal with power-based VNF placement proposals[17ndash21] for our case study Reference [17] aims at minimizingpower consumption allowing users to meet delay require-ments using a genetic algorithm approach while [18 19] pro-pose heuristic algorithms that give power reduction improve-ment Besides [20] presents a suboptimal placement solutionthat combines traffic and energy cost optimization derivedby means of a Markov approximation with matching theoryApart from that research works in [22ndash24] go a step forwardand cope with the migration problem of active microserversneeded in low traffic conditions to turn off or suspendmicroservers so as to achieve energy savings

Mobile Information Systems 3

NS request Service orchestrator NS catalog

VNF catalog

Available virtualizedresources

Placement algorithm

VNF1 VNF2 VNF3 VNF4 VNF5VNFchain

Infrastructure manager

SLA

Cluster network

microserver microserver microserver

Clou

d-RA

N

VM

VM

Virtualization layer

HW resources HW resources HW resources

VNF1VM

VNF3VM

VNF5VM

VNF4VM

VNF2VM

VM

Figure 2 VNF placement process

Moreover a common assumption of the different opti-mization models proposed to solve different versions of theVNFs placement problem is that input data is perfectlyknown which is difficult in practice Unfortunately smalldeviations in input data values may usually lead to situationswhere a previously found optimal solution is not even feasibleanymore and consequently the presence of uncertain datamay produce useless placement solutions In order to copewith uncertain input parameters RO [10] is applied to solveoptimization problems Indeed RO has been successfullyemployed in different contexts such as in resource allocationin OFDM networks [26] with uncertainty in channel stateinformation in cloud resource provisioning without perfectknowledge of demands and price [21] and even in the virtualnetwork embedding problem on a physical network substratein recent works [27ndash30] that usually deal with uncertainservice demands

Nevertheless to the best of our knowledge except thework presented in [21] there is no VNF placement proposalbased on RO Therefore to achieve our goal of designingan energy-aware placement solution using RO approach thatconsiders service demand uncertainty [21] has been relevantAuthors develop a novel power-based placement solutionwith uncertainty inVNFsCPUdemands based onRO theoryTo this aim they take into account the physical serversand network resources along with their energy efficiencyNonetheless they focus on the core network part whereas ourresearch is oriented to combined deployment of radioservicenetwork functions at the edge Additionally the placementproblem considered in this paper takes into account a het-erogeneous edge cloudmade up of a cluster of interconnectedSCs

3 Problem Description

Thenetworking environment targeted in ourwork is a 5Gdis-tributed edge cloudThis environment uses SCs that evolve tobuild multitenant CE-RANs The enhanced SCs deploymentis offered to multiple operatorsservice providers to servetheir network services in the virtualized infrastructure andengage them in a multitenant system Figure 2 shows the NSrequest and subsequent placement decision process in thisjoint radio-cloud architecture

In such an architecture whenever a tenant needs todeploy a new service itmakes a request to the ServiceOrches-trator functional element which is implemented as theNFV Orchestrator (NFVO) in ETSI-MANO terminologyThe orchestrator analyzes its NS catalog to extract thecorrespondent service chain A service chain is the sequenceof VNFs that composes the requested 5G service

The VNF chain extracted from the catalog is later used asan input for the placement algorithm together with the ser-vice constraints imposed by the key performance indicators(KPI) of the corresponding Service Level Agreement (SLA)between the tenant and the CE-RAN operator In our workthe SLA KPIs include the following network parametersmaximum accepted latency for the NS aggregated user bitrate and a robust protection parameterThis protection para-meter is related to the uncertainty of the user traffic demandWe assume that the network service has a mean aggregatedtraffic demand that is used to size the necessary resourcesin order to serve tenantrsquos user But along the service timethe user traffic demand fluctuates around this mean valueaffecting the amount of resources needed The robust pro-tection parameter is therefore related to the expected peak

4 Mobile Information Systems

traffic demand deviation and will be used to overestimate theallocation of virtualized resources in order to be prepared foreventual user peak demands

Next the placement algorithm inspects the VNF catalogfor the parametrization of theVNF instances needed tomatchthe SLA and checks the available resources provided throughthe virtualized infrastructure VNFs are characterized by theuse of resources (ie number of cores RAM and storage)according to the aggregated bit rate (ie characterized aggre-gated traffic flows of a NS) served by the NS they belongtoMoreover the service latency of eachVNF also depends onthe traffic load supported by the VNF instance Lastly virtu-alized resources may also include other hardware appliancessuch as hardware accelerators (HWA) that can improve theperformance of a VNF in terms of latency This way theseadditional appliances help matching SLA with the tenant butat a higher energy cost

All the VNFs composing the service chains of therequested network services must be allocated in the availablevirtualized resources including a given number of CPUshardware appliances RAM and storage space and networkbandwidth We assume that each core runs a single VM thatis devoted to the execution of a single VNF instance but oneVNF instance can scale from one to many VMs upon the ser-vice requirements of theNS Besides the transmission latencyalong the network topology between two VNFs allocatedin different microservers depends on the traffic load of theinvolved links and the size of the flow to be transmittedWith this information the placement algorithm assigns eachVNF instance to one or more VMs with the objective ofminimizing the energy consumption of the whole systemwhile matching the latency constraints for the NS

Finally the Infrastructure Manager functional elementVirtualized Infrastructure Manager (VIM) in ETSI-MANOterminology receives the outcome of the placement decisionand maps each VNF to the assigned resources

It is important to point out that a VNF chain includesradio-related and service-related instances The placementalgorithm introduced in this work considers a scenario withseveral tenants each of them requesting one or more NSswhich share the available resources Considering the afore-mentioned communication particularities of 5G Figure 3shows an example of a simple complete placement result ofa multitenant environment

This example considers a distributed edge cloud shared bytwo tenants Tenant 1 offers two network services the first onechaining a virtualized firewall (vFW) an intrusion detectionfunction (vID) a watermarking process (vWM) and finally acaching service (vCache) The second NS of Tenant 1 chainsthe vFW and a video analytics (VA) functionality for onlinesurveillance Tenant 2 offers a single network service thanchains the vFW a vCache service and a virtual transcodingunit (vTU) All NSs are associated with their correspondentlatency constraint

In the example both NSs belonging to the same tenantshare the radio VNF (small cell VNF or SCVNF) In this waythe common radio operations are performed in a coherentway between the different edge services of a tenant and thedata paths of the specific service instances per tenant are split

SCVNF SCVNF

vFW

VA

vFW

vWM

vCachevFW

vCache

vTU

HW

Acc

CPU

s

vID

NS2

NS1

NS1

NS1

NS1

NS1

NS1

NS1 N

S3NS3

NS1

NS2

Network topology

vFW vCache vTU

vFW VA

vFW vID vWM vCache ltlat1

ltlat2

ltlat3

Tena

nt 1

SCVNF

SCVNF

Tena

nt 2

NS1

NS2

NS3

vIDlowast

Figure 3 Multitenant placement example lowast indicates that ldquovIDrdquouses a hardware acceleration appliance

The SCVNF of each tenant will be shared by all theNSs of thistenant which means that the use of resources of the SCVNFmust then consider the sumof the aggregated flows of theNSsof the same tenant

The distinctive features of 5G communications have twoimportant implications over the placement algorithm Firstlyevery tenant allocates a unique SCVNF instance at the edgecloud in order to support the control plane and data planeoperations of all the SCs associated with that tenant Andsecondly since the SCVNFmanages the uplink and downlinktraffic between themobile users and the core network it leadsto circular forwarding paths when including edge serviceinstances In this paper we consider that the core networkfunctionalities of themobile network operator are outside theCE-RAN which means that the CE-RAN only deploys radioand edge service instances In order to keep the data pathbetween the mobile users and the core network functions alltheNSs need to start and end at the SCVNFwhichwill handlethe user plane traffic tofrom the core network

The example also shows how some VNFs needmore thanoneVM to execute the associated functionalities and howoneVNF uses a HWA tomeet the latency constraint of the server

Section 5 will show the result of more complex and com-plete placement experiments For the sake of generality thispaper does not focus on the specific modelling of the trafficpatterns the energy consumption schemes or the VNF net-work service performance Section 5 provides some referenceoutcomes on the potential benefits of the introduced VNFplacement algorithm in the considered network scenario but

Mobile Information Systems 5

the applicability of the solution to other scenarios shouldbe fed by experimental modelling data specific to the targetscenario

4 Robust Constraint-Based VNFPlacement Proposal

Once the 5G VNF placement problem under study is pre-sented in this section we propose a placement solution tothat problem using the RO approach To this end first weformulate the energy-aware VNF placement model in thenext subsection Then we provide our RO-based placementproposal in Section 42

41 ProblemModelling As stated before the VNF placementproblem addressed in this work focuses on the assignment ofthe VNFs composing each NS to the provided 5G networkinfrastructure Therefore it constitutes a discrete or com-binatorial optimization problem where the minimizationof energy consumption constitutes the optimality criterionSummary of Model Indexes Sets Parameters and DecisionVariables reports and summarizes the mathematical nota-tions used for the model indexes sets input parameters anddecision variables of VNF placement problem that is definedin the following

The Problem Model The energy-aware VNF placement opti-mization problem

min120587isinΠ

119875120587 (1)

st 119889120587119894 lt 119863max119894 forall119894 isin I 120587 isin Π (2)

119898120587119903119909 lt 119872119903119909 forall119903 isin R 119909 isin X 120587 isin Π (3)

119905119897120587119909 lt BW119909 forall119909 isin X 120587 isin Π (4)

119875120587 = sum119894isinI

sum119895isinJ119894

119875120587119894119895 (5)

119875120587119894119895

= sum119909isinX

sum119910isinY119909

119875cpu119909119910119894119895 119860119909119910

119894119895 + sum119894isinI

sum119895isinJ119894

119875switch119894119895119871 119894119895

+ sum119909isinX

119875sc119909119860119909119910

119894119895

(6)

119889120587119894 = sum119895isinJ119894

119889120587119894119895 (7)

119889120587119894119895

= sum119909isinX

sum119910isinY119909

119889119901119909119910119894119895 119860119909119910

119894119895

+ sum1199091isinX

sum1199101isinY1199091

sum1199092isinX

sum1199102isinY1199092

11988911989711990911199101 11990921199102119894119895 11986011990911199101119894119895 11986011990921199102119894119895+1

(8)

sum119894isinI

sum119895isinJ119894

119860119909119910119894119895 = 1 forall119909 isin X 119910isin Y119909 (9)

119898120587119903119909 = sum119894isinI

sum119895isinJ119894

sum119910isinY

119898119903119894119895119860119909119910

119894119895 (10)

119905119897120587119909 = sum119894isinI

sum119895isinJ119894

sum119910isinY

119879119894119860119909119910

119894119895 1198601199091015840119910

119894119895+1 (11)

We consider a set X of small cells that compose the CE-RAN and a set Y119909 of cores included in the microserver ofthe SC 119909 Additionally we also consider a set of I networkservices offered by the CE-RAN operator to the differenttenants and a set J119894 of VNFs that form the functionalchain of the NS 119894 Finally R represents the set of differentresources available in the microservers for the execution ofthe VNFs (storage RAM hardware accelerators etc) Henceour constraint-based expert systemmust determine in whichVM 119910 of a SC 119909 each VNF 119895 of a NS 119894 is located

Let Π denote the set of all possible power-aware andlatency- and resource-constrained VNF placement policieswhich decide the mapping of each VNF of available NSs toavailable VMs in the provided SCs We formulate our VNFplacement optimization problem aimed atminimizing powerconsumption with delay requirements as (1)

In reference to the energy part we define the total powerconsumption 119875120587 as the sum of the energy demand of all theallocated VNFs 119875120587119894119895 (5) Then for each VNF instance thepower consumption is expressed in (6) as the sum of thepower consumption of VNF 119895 of NS 119894 due to the CPU useof VM 119910 of SC 119909 the power consumption of the physicalswitch due to the forwarded traffic of the VNF and the powerconsumption of SC 119909 because of being switched on

The power consumption of the cores that run the VMsdepends of the aggregated user traffic served by the corre-spondent VNF and this relationship may be nonlinear Itmust be also understood that when a VNF instance scales toa higher number of cores the individual load of the involvedCPUs decreases leading to a lower power consumption percore Additionally the energy consumption of the networkingdevices (eg an Ethernet switch) depends on the aggregatedflow that traverses the device to forward data packets betweenSCs and the number of active ports Finally it is realized thatthe power wasted because the SC is active is constant notaffecting the optimization results Besides we express per-NSdelay in (7) as the sum of per-VNF delays and the per-VNFdelay in (8) as the sum of the processing delays introducedby the execution of the VNF instance plus the path delayintroduced by the traffic going from one VNF to the next onein the chain defined in the correspondent NS

On the one hand the processing delay in the CPU growswith the supported user traffic load On the other hand weconsider that the pathlink delay is composed of several linkdelays fromtoVMs tofrom the switches plus the delay in theswitches Hence depending on the adopted placement policy120587 each NS will follow a different path through the networkIn such a way if the VNFs belonging to a NS are placed in thesame SC the service chain traverses virtualinternal links andthe virtual switch of the selected SC whereas if the VNFs of

6 Mobile Information Systems

a NS are located in different SCs external links and the physi-cal switch is also used therefore generally 11988911989711990911199101 11990921199102119894119895 =11988911990911199101vs + 119889vs

1199091+ 119889vs1199091 119904 + 119889119904 + 119889119904vs1199092 + 119889vs

1199092+ 119889vs11990921199102 but for

a given VNF if the next VNF in its chain holds in the sameSC its link delay is only affected because of passing throughthe virtual switch Note that in case a VNF is the last VNF inthe chain the link delay is null with the sums referring to thelink delay for that 119895 being null

In order to guarantee that each VM holds only a singleVNF constraint (9) must be satisfied Nevertheless weconsider that each VNF of a NS can be instantiated as a VMthat is executed over several cores of the same SC Apart fromthat as previously detailed in Section 3 the first VNF and thelast VNF in a service chain are the same (the SCVNF) in orderto perform the specific functions that allow the split betweencontrol and data plane required in 5G communicationsTherefore it implies that 1198601199091199101198941 = 119860119909119869119894119894119895

Finally the model must also include the constraintsreferred to as the available resources In this sense (3) statesthat the consumption of the resource 119903 due to the placementof VNFs in the microserver of the SC 119909119898120587119903119909 must not exceedthe total amount of resource 119872119903119909 where 119898120587119903119909 is defined in(10) Similarly (4) constrains the traffic through the networklinks 119905119897120587119909 to the maximum capacity of the links BW119909 as thesum of the traffic flows of the network services that must beforwarded from a VNF in one SC to the next VNF in anotherSC

42 Robust Constraint-Based Solution As described inSection 2 many existing placement works assume perfectknowledge on input parameters pointing thus to the for-mulation of a deterministic optimization problem Unfortu-nately in the real world the estimated values of the inputparameters may differ due to biased data unrealistic assump-tions or numerical errors affecting the obtained optimalsolution and its performance This potential deviation ofthe nominal or expected input may lead to the violation ofthe problem constraints and therefore make the obtainedsolution suboptimal or even unfeasible

The uncertainty of modelling parameters has been tra-ditionally handled through stochastic programming andsensitivity analysis but robust optimization techniques haverecently arisen as a powerful tool to manage the impact ofuncertain input sets RO seeks an uncertainty-immunizedsolution with an acceptable performance under the real-ization of the uncertain inputs becoming a conservativeworst-case oriented methodology The main tools of RO areuncertainty sets and a robust counterpart problem [31] Theuncertainty of the input data is described later in this sectionwhile the robust counterpart problem is the deterministicmodel previously detailed in Section 41

Here we propose a robust constraint-based placementsolution that deals with service demand uncertainty Withthis RO approach we consider a tradeoff between theproblem optimization and the protection from deviationscaused by input parameters uncertainty This uncertainty inservice demand has an impact on several components on theformulation previously presented in Section 41 On the one

hand we consider that a VM which is running a specificVNF requires an expected use of CPU and thus a certainconstant power consumption due to CPU Neverthelessthe uncertainty in traffic demand may cause uncertainty inthe CPU requirementuse and consequently in its energyconsumption and processing delays On the other hand bothenergy consumption and delays in switches and links dependon that demand uncertainty as well

In order to obtain a proposal for the placement problemwith uncertain service demand we employ the Soyster [10]method used in the framework of RO That technique isalso known as the worst-case approach where the solutionsare achieved using the most extreme expected values of theuncertain variables Therefore that kind of resolution guar-antees feasible solutions for any value of the uncertain servicedemand

In reference to the modelling of the uncertain trafficdemand per NS 119894 we consider an expected traffic demand119864[119879119894] = 119879119894 plus a term due to uncertainty in the servicedemandUTmax119894 We assume that the randomvariableUTmax119894is symmetrically distributed within [minusΔ119879119894 +Δ119879119894] sdot 119879119894 andwith mean zero Furthermore the sum of deviations of theuncertain service demand should not exceed the worst-casemaximum value 119879max119894 fulfilling (12) In this way the valueof 119879max119894 controls the tradeoff between the robustness andthe impact on the optimization higher 119879max119894 leads to worseobjective function values but protects from more parameterdeviations

119880119879119894Δ119879119894

le 119879max119894 forall119894 (12)

5 Analysis of the Results

This section presents the results obtained with the simulationof the proposed robust constraint-based expert system for theplacement of VNF chains in an emerging 5G distributed edgecloud As previously introduced in Section 2 the mathemat-ical VNF placement problemmodel presented in Section 4 isa combinatorial or discrete optimization problem equivalentto the well-known knapsack problem This means that thedecision problem is NP-complete while the optimizationproblem is NP-hard [11] The proposed optimization modelhas been solved using constraint programming as a powerfulparadigm for solving combinatorial search problems [32]

We usedMinizinc [33] a free and open-source constraintmodelling language to model the optimization and con-straint satisfaction VNF placement problem in a high-levelsolver-independent wayThen Minizinc compiles this modelinto FlatZinc a solver input language that is understoodby a wide range of solvers In our case Gecode solver [34]provided the best performance in the search of the optimalsolution

The experiments that are described and discussed belowhave been conducted in a single computer with a 27GHzIntel Core i5 processor and 16GB RAM This simple con-figuration provides solutions within minutes for simpleexperiments with loose constraint satisfaction When theexperiments demand tighter latency or resource constraints

Mobile Information Systems 7

Virt

ual i

nfra

struc

ture

Corenetwork

Figure 4 Evaluation scenario

the solving time increases to hours depending on the specificconfiguration of the experiment

Next we first describe the analysis scenario and the con-figuration of the performed experiments to later discuss theresults of those experimentsThe objective of the experimentsis to demonstrate that the placement mechanism is able toallocate multitenant service chains to the available resourcesconsidering latency constraints For this reason the numeri-cal values employed in the next section are just experimentaland do not respond to real world measurements

51 Description of the Analysis Scenario Our evaluationscenario is depicted in Figure 4 We consider a 10-SC deploy-ment for example to cover a football stadium for periodicalflash eventsThe SCs are connected in a star topology throughan Ethernet switch Each SC is composed of 8 cores 16GBRAM 100GB of storage and 1HWA (eg GPU)

We also consider 2 NSs that can be offered at 3 serviceflavours (bronze silver and gold) that determine the latencyconstraint for eachNS and the nominal aggregate user bit rateserved as shown in Figure 5

Table 1 shows the experimental energy model and Table 2shows the modelling of the VNFs that compose the proposedNSs for the user traffic demand that will be employed in theevaluation experiments The user traffic values include thenominal bit rates of services NS1 and NS2 (130Mbps and90Mbps resp) as well as the values when a 20 protectionparameter is applied to the nominal values (156Mbps and108Mbps resp) in order to study the effect of robust optimi-zation techniques on the placement decision

As a final remark the placement behavior of VNFs canbe altered by the use of HWA The assignment of a HWA toa VNF implies that the service latency of the correspondentinstance is reduced 3 times at a cost of incrementing thepower consumption by 10 It is important to note that whenthe VNF instance needs more than 1 CPU for executingwithout the assistance of a HWA then if the placementalgorithm assigns it to a HWA the VNF will need a singlecore

52 Discussion of the Placement Results As stated before theevaluation scenario includes 3 tenants that hire a combination

8 Mobile Information Systems

VNF1 VNF2 VNF3 VNF5 VNF6

VNF1 VNF2 VNF3 VNF4

NS1

NS2

Service flavourBronze Silver Gold

NS1

NS2 Max Flow = 90MbpsMax Lat = 90ms Max Lat = 80ms

Max Flow = 130－＜ＪＭ

Max Lat = 150ＧＭ

Max Lat = 110ＧＭ

Max Flow = 90Mbps

Max Flow = 130－＜ＪＭ

Max Lat = 120ＧＭ

Max Flow = 90Mbps

Max Flow = 130－＜ＪＭ

Max Lat = 100ＧＭ

Figure 5 Description of NSs and service levels

Table 1 Energy models of the cores of the microservers and of the network switch related to the usage percentage

Usage 0 10 20 30 40 50 60 70 80 90 100119875router 0 1 2 5 7 9 11 13 14 15 15119875cpu 5 50 70 100 130 150 165 170 180 185 190

Table 2 VNF characterization for the service flavours of the evaluation scenario

VNF1 VNF2 VNF3 VNF4 VNF5 VNF6Number of cores per instance

90Mbps 1 2 1 1 1 1108Mbps 1 2 2 1 1 1130Mbps 1 2 2 1 1 1156Mbps 1 2 2 1 1 1

CPU usage90Mbps 30 20 90 50 50 20108Mbps 40 30 60 60 60 30130Mbps 40 30 60 60 60 30156Mbps 40 30 90 40 20 30

Service latency90Mbps 10ms 15ms 30ms 35ms 20ms 25ms108Mbps 10ms 15ms 35ms 40ms 25ms 30ms130Mbps 10ms 15ms 35ms 40ms 25ms 30ms156Mbps 12ms 16ms 40ms 50ms 30ms 35ms

of network servicesflavours to the SC operator In this wayin the following first we are going to evaluate the behavior ofthe placement algorithm for the three service levels and nextwe will show the effect of applying RO techniques to thoseresults

521 Bronze Flavour Scenario Results with 0 Robust Pro-tection Figure 6 shows the placement results of an scenariowhere all the network services operate at bronze flavourDifferent colours represent each tenant in order to dif-ferentiate the VNFs that belong to its NS and differenttones distinguish NSs of the same tenant As a result ofthe placement algorithm all the NSs are allocated in theavailable virtualized resources with energy consumption costof 3712W meeting all the latency constraints The figure

displays the assignment of VNFs to the SCs highlightingsome placement particularities As explained in Section 3the VNF chains of the same tenant share their SCVNF Inour example Tenant 1 and Tenant 2 both have two NSsthat are served with a single SCVNF Furthermore eachCPU executes a single VM but some VNF instances needtwo cores to manage the user demand In connection withthis behavior the VNF scalability feature is also visible inFigure 6 observing the placement of for example VNF3for the different NSs NS1 with a higher aggregated userdemand needs two CPUs to instantiate VNF3 but since NS2operates at a lower user demand the instantiation of VNF3needs just a single core Note as well that bronze serviceflavour does not require HW acceleration to meet the latencyconstraints of the NSs Finally it can also be observed that

Mobile Information Systems 9

Tenant 1 Tenant 2 Tenant 3

NS1 NS2 NS1 NS2 NS1

Service type Bronze Bronze Bronze Bronze Bronze

Max ow

Max latency

Deviation 0 0 0 0 0

Peak ow

VNF1VNF3

VNF1

VNF6

VNF1

SCVNF

VNF3VNF2

VNF1VNF3

VNF1VNF2

VNF2VNF3

VNF5

VNF5VNF2

VNF5

SCVNF

VNF1

VNF5

VNF3

VNF4

VNF5

VNF6

SCVNF

VNF1

VNF1VNF2

VNF3

VNF4

VNF1

960Mbps

960Mbps

780Mbps

780Mbps

780Mbps

260Mbps

VNF2

VNF6

CPU

s

Finallatency

Tenant 1NS1

NS2

Tenant 2NS1

NS2

Tenant 3 NS1

Total energy consumption

Switc

hH

A A

cc

132ＧＭ

102ＧＭ

132ＧＭ

102ＧＭ

130ＧＭ

3712７

130－＜ＪＭ 90－＜ＪＭ 130－＜ＪＭ 130－＜ＪＭ90－＜ＪＭ

130－＜ＪＭ 90－＜ＪＭ 130－＜ＪＭ 130－＜ＪＭ90－＜ＪＭ

150ＧＭ 110ＧＭ 150ＧＭ 110ＧＭ 150ＧＭ

Figure 6 Placement results of bronze flavour setting with 0 robust protection

the algorithm tries to group asmanyVNFs as possible in eachserver in order to reduce the data traffic across the switch andminimize its load and therefore its energy waste

522 Silver Flavour Scenario Results with 0 Robust Pro-tection The evaluation experiment analyzed in this sectionstudies the outcome of the placement algorithm when thelatency constraints become more strict with silver serviceflavour Due to the further limitation of maximum latencyvalues (20 stricter) the placement mechanism is compelledto use HW acceleration units Figure 7 shows that at thisservice level all the NSs must use HW acceleration in oneof the VNFs composing the service chain As a consequencethe resulting total latency of the NSs decreases to meet theservice level requirements but in return the global powerconsumption grows by 22 up to 4547W

Apart from that the placement results illustrate the samebehavior as the bronze flavoured example of Section 521regarding the allocation of VNFs in the available resourcesscale-inout features and multitenancy

The evaluation experiments of Sections 521 and 522operate with a nominal aggregated user traffic to perform theoptimization decision-making process However a perfectlyknown user traffic demand is seldom available Usuallythere is a certain level of uncertainty in the behavior of theuser demand We include robust optimization techniques to

allow the placement algorithm to insert a defined level ofuncertainty and protect the system against the undesirableeffects of demand peaksThe evaluation examples in Sections523 and 524 include a robust protection parameter andanalyze its effect on the global power consumption

523 Silver Flavour Scenario Results with 20 Robust Protec-tion This section studies the placement outcome of the eval-uation scenario described in the previous section introducinga 20 robust protection parameter This modification on thescenario configuration implies an increase of the 20 of theaggregated user traffic as an input of the decision-makingprocess in order to be prepared for potential demand peaksduring the service operation

Figure 8 exhibits that VNF3 of NS2 must scale as a con-sequence of the traffic growth and now needs two CPUs foreach instance The traffic increase also has an impact on theprocessing latency of the VNFs causing an adjustment in theuse of HW accelerators As a consequence some VNFs thatrequired 2CPUs for their operation in the previous evalu-ation scenario now only need a single core in combinationwith the HWA Therefore in some cases the higher powerconsumption that is implicit to the use of HWAs may bebalanced with a decrease on the number of CPUs required forthe VNF Besides the higher aggregated user traffic involvesa more intense use of the network switch These factors have

10 Mobile Information Systems

SCVNF

VNF6

VNF1VNF1

VNF2

VNF3

VNF3

VNF3

VNF5

VNF2

VNF1

VNF2

SCVNF

VNF1

VNF2

VNF4

VNF5

VNF6

SCVNF

VNF2

VNF3

VNF4

VNF5

VNF6

VNF3

VNF1

Finallatency

Tenant 1NS1

NS2

Tenant 2NS1

NS2

Tenant 3 NS1

Total energy consumption

VNF6VNF1VNF1VNF2VNF2

Tenant 1 Tenant 2 Tenant 3

NS1 NS2 NS1 NS2 NS1

Service type Silver Silver Silver Silver Silver

Max ow

Max latency

Deviation 0 0 0 0 0

Peak ow

960Mbps

960Mbps

780Mbps

780Mbps

260Mbps

780MbpsSw

itch

100ＧＭ

118ＧＭ

116ＧＭ

4547７

130－＜ＪＭ 90－＜ＪＭ 130－＜ＪＭ 130－＜ＪＭ90－＜ＪＭ

130－＜ＪＭ 90－＜ＪＭ 130－＜ＪＭ 130－＜ＪＭ90－＜ＪＭ

120ＧＭ 120ＧＭ 120ＧＭ

CPU

sH

A A

cc

88ＧＭ

88ＧＭ

90ＧＭ 90ＧＭ

Figure 7 Placement results of silver flavour setting with 0 robust protection

an impact on the power consumption of the componentsof the network resulting in a global energy consumption of5047W which implies an increment of 11 compared to theunprotected placement

524 Gold Flavour Scenario Results with 20 Robust Protec-tion Finally the last evaluation example sets even stricterlatency constraints to theNSs to be placed and includes a 20robust protection parameter Figure 9 depicts the results ofthe placement algorithm with this configuration Again thereduction of latency values forces the increment of the use ofHW acceleration In this case the main consequence is thespreading of the VNFs across the SCs to use the single HWAavailable in each cellThe use of HWAs the increase of powerconsumption due to higher user aggregated traffic and thehigher network traffic traversing the switch cause the globalenergy consumption to grow up to 5569W

As a final remark Figure 10 shows the comparison ofthe energy consumption and use of resources (cores activeCESCs and HWA) related to the robust protection level ina wider collection of scenarios In particular the charts ofFigure 10 compare the results of the placement algorithm forthe three service levels described in the previous sections(bronze silver and gold) combined with three robust protec-tion levels (0 or no protection 10 and 20) Obviouslythe main trend is that both power and resource consumptionincrease with higher values of protection but the associatedcost can be assumed in favor of reducing the impact of

unexpected user demand peaks However there are somecases in which the protection parameter does not have soadverse implications For example it can be observed howthe number of used cores in the silver service level with a10 protection level is lower than in the case of no robustprotection The reason of this behavior is how the algorithmuses HWA appliances in this case The increase of the usertraffic demand in the protected services implies the scaling upof someVNFs that nowmust usemore cores for the operationof theNSsThen the placement algorithm decides to useHWaccelerators for those specific VNFs compensating this waythe higher energy cost of HWAs with a lower use of CPUcores With all these considerations that can be extractedfrom a deeper analysis of robust placement results robustoptimization can be applied to decision-making problemsnot ending just with the optimization problem

The proposed steps in this paper can be used in anyplacement decision-making context and its goal is to providea tool that helps system administrator to evaluate the possibleimpact of any decision on the performance of the deploymentunderneath

6 Conclusions

This paper introduces a constraint-based expert system tosupport the decision-making process of VNF chain place-ment in distributed edge cloud networks The placement

Mobile Information Systems 11

SCVNF

VNF5

VNF6VNF5VNF2

VNF3

VNF5

VNF1

VNF2

VNF3

VNF1

VNF2

VNF3

VNF6

SCVNF

VNF1

VNF1

VNF3

VNF4

SCVNF

VNF1

VNF2

VNF3

VNF4

840Mbps

840Mbps

936Mbps

624Mbps

312Mbps

312Mbps

936Mbps

VNF6

Tenant 1NS1

NS2

Tenant 2NS1

NS2

Tenant 3 NS1

Total energy consumption

VNF5VNF6VNF5VNF1VNF4VNF4

VNF2

Finallatency

Tenant 1 Tenant 2 Tenant 3

NS1 NS2 NS1 NS2 NS1

Service type Silver Silver Silver Silver Silver

Max ow

Max latency

Deviation 20 20 20 20 20

Peak ow

116ＧＭ

110ＧＭ

118ＧＭ

5047７

130－＜ＪＭ 90－＜ＪＭ 130－＜ＪＭ 130－＜ＪＭ90－＜ＪＭ

156－＜ＪＭ 108－＜ＪＭ 156－＜ＪＭ 108－＜ＪＭ 156－＜ＪＭ

150ＧＭ 110ＧＭ 150ＧＭ 110ＧＭ 150ＧＭ

79ＧＭ

79ＧＭ

CPU

sH

A A

ccSw

itch

Figure 8 Placement results of silver flavour setting with 20 robust protection

algorithm applies robust optimization techniques to theminimization of the global power consumption of the systemsubject to the bit rate and latency constraints of the networkservices The power consumption of the components of thesystem depends on the usage percentage which is a con-sequence of the aggregated user traffic demand The modelassumes that the relationships between the user trafficdemand the power consumption and the use of the resourcesmay be nonlinear Besides the placement algorithmconsidersthe 5G communication particularities of both control anddata plane and allows multitenancy Finally the algorithmalso includes VNF scaling features and the use of hardwareacceleration

The results show that the placement algorithm succeeds inallocating theVNFs that compose theNSs of different tenantsin the available resourcesmeeting the latency constraints andminimizing the energy consumption of thewhole systemThepresented evaluation examples demonstrate the features ofthe constraint-based expert system and illustrate the impactof the selected service flavour and robust protection on theglobal power consumption and therefore the exploitationcost of the 5G distributed edge cloud

Further researchwill extend the characterization of VNFsto include variable output bit rate This feature will modelthe behavior of for example transcoding functions A future

version of the algorithm will also incorporate the existenceof a virtual switch to connect the VMs running in the samemicroserver Another enhancement of the model will bethe combination of heterogeneous SCs with different con-figurations and resource consumption models Finally theobtained placement results will be used for a technoeconomicstudy that will analyze the relationship between the serveraggregated user traffic cost of the required infrastructure andthe pricing of the services This study will also consider theintegration of the placement decision problem with the radioside

Summary of Model Indexes SetsParameters and Decision Variables

Indexes

120587 Index for VNF placement policy119909 Index for SC microserver119910 Index for core number in each microserver119894 Index for NS119895 Index for VNF in a network service chain119903 Index for physical resources in

SC microserver

12 Mobile Information Systems

VNF6VNF5VNF1VNF6VNF2SCVNF

VNF2

VNF3

VNF5

VNF3

VNF1

VNF2

VNF3

VNF5

SCVNF

VNF1

VNF1

VNF3

VNF4

SCVNF

VNF2

VNF3

VNF4

VNF1

Tenant 1NS1

NS2

Tenant 2NS1

NS2

Tenant 3 NS1

Total energy consumption

VNF6VNF5VNF1VNF6VNF2VNF5VNF6VNF4VNF4

VNF2

840Mbps

840Mbps

936Mbps

936Mbps

312Mbps

312Mbps

312Mbps

312Mbps

312Mbps

Finallatency

Tenant 1 Tenant 2 Tenant 3

NS1 NS2 NS1 NS2 NS1

Service type Gold Gold Gold Gold Gold

Max ow

Max latency

Deviation 20 20 20 20 20

Peak ow

5569７

130－＜ＪＭ 90－＜ＪＭ 130－＜ＪＭ 130－＜ＪＭ90－＜ＪＭ

156－＜ＪＭ 108－＜ＪＭ 156－＜ＪＭ 108－＜ＪＭ 156－＜ＪＭ

100ＧＭ 100ＧＭ 100ＧＭ

79ＧＭ

95ＧＭ

89ＧＭ

79ＧＭ

86ＧＭ

CPU

sH

A A

ccSw

itch

80ＧＭ 80ＧＭ

Figure 9 Placement results of gold flavour setting with 20 robust protection

34

32

30

28

260

10

20Robust protection ()

Num

ber o

f use

dco

res

10864200

10

20Robust protection ()

HW

AN

umbe

r of u

sed

0

10

20Robust protection ()

987654N

umbe

r of a

ctiv

eCE

SCs

6000

5500

5000

4500

4000

35003000

010

20

Ener

gy co

nsum

ptio

n(W

)

Robust protection ()

Gold

Silver

Bronze

Gold

Silver

Bronze

Gold

Silver

Bronze

Gold

Silver

Bronze

Figure 10 Comparison of energy and resource consumption for different levels of robust protection

Mobile Information Systems 13

Sets

Π Set of possible VNF placement policies119883 Set of SC microservers119884 Set of cores composing a SC microserver119868 Set of offered NSs119869119894 Set of VNFs in NS 119894119877 Set of resources available in

each SC microserver

Input Parameters

119879119894 Contracted maximum aggregated usertraffic for network service 119894

119863max119894 Maximum latency allowed for NS 119894119872119903119909 Amount of resource 119903 available in

SC microserver 119909119875sc119909 Power consumption of SC 119909 because of

being switched on119875cpu119909119910119894119895 Power consumption of VM 119910 of SC 119909 due

to CPU use of VNF 119895 of NS 119894119875switch119894119895 Power consumption in the physical switch

due to CPU use of VNF 119895 of NS 119894

Decision Variables

119875120587 Total power consumption ofplacement policy 120587

119875120587119894119895 Power consumption of VNF 119895 of NS 119894 inplacement policy 120587

119898120587119903119909 Consumption of resource 119903 in SCmicroserver 119909 in placement policy 120587

119905119897120587119909 Traffic of the link that connects SCmicroserver 119909 with the switchinplacement policy 120587

119860119909119910119894119895 Binary variable that expressesthe assignment of VNF 119895 of NS 119894 toVM 119910 in SC 119909

119871 119894119895 Binary variable to express the use of thenetwork switch to forward the flow ofVNF 119895 of NS 119894 to the next VNF

119889119894 Delay of NS 119894119889119894119895 Delay of VNF 119895 of NS 119894119889119901119909119910119894119895 Processing delay in the CPU of VNF 119895 of

NS 119894 in VM 119910 of SC 1199091198891198971199091119910111990921199102119894119895 Path delay for NS 119894 between VNF 119895

(located in VM 1199101 of SC 1199091) and VNF119895 + 1 (located in VM 1199102 of SC 1199092)

119889vs119909119910 Link delay between the virtual switch inSC 119909 and the VM 119910 in SC 119909

119889vs119909 119904 Link delay between the virtual switch inSC 119909 and the physical switch

119889vs119909 Delay in the virtual switch of SC 119909

119889119904 Delay in the physical switch

Conflicts of Interest

The authors declare that they have no conflicts of interest

Acknowledgments

The research leading to these results has been supportedby the EU funded H2020 5G-PPP Project SESAME (GrantAgreement 671596) and the Spanish MINECO Project5GRANVIR (TEC2016-80090-C2-2-R)

References

[1] ITU-R M2083 IMT Vision-Framework and overall objectivesof the future development of IMT for 2020 and beyond 2015

[2] P Demestichas A Georgakopoulos D Karvounas et al ldquo5G onthe Horizon key challenges for the radio-access networkrdquo IEEEVehicular Technology Magazine vol 8 no 3 pp 47ndash53 2013

[3] I F Akyildiz S Nie S-C Lin and M Chandrasekaran ldquo5Groadmap 10 key enabling technologiesrdquo Computer Networksvol 106 pp 17ndash48 2016

[4] ETSI Mobile-edge computing Introductory technical whitepaper 2014

[5] B Han V Gopalakrishnan L Ji and S Lee ldquoNetwork functionvirtualization challenges and opportunities for innovationsrdquoIEEE Communications Magazine vol 53 no 2 pp 90ndash97 2015

[6] R Mijumbi J Serrat J Gorricho N Bouten F De Turckand R Boutaba ldquoNetwork function virtualization state-of-the-art and research challengesrdquo IEEE Communications Surveys ampTutorials vol 18 no 1 pp 236ndash262 2016

[7] Ericsson White Paper Cloud-RANndash the benefits of virtualiza-tion centralization and coordination 2015

[8] B Blanco J O Fajardo I Giannoulakis et al ldquoTechnologypillars in the architecture of future 5G mobile networks NFVMEC and SDNrdquo Computer Standards and Interfaces vol 54 pp216ndash228 2017

[9] ETSI Network Functions Virtualization (NFV) Managementand Orchestration 2014

[10] A Ben-Tal L E Ghaoui and A Nemirovski Robust Optimiza-tion Aharon Ben-Tal Laurent El Ghaoui Arkadi Nemirovski-Google Books Princeton University Press Princeton NewJersey NY USA 2009

[11] J Gil Herrera and J F Botero ldquoResource Allocation in NFVA Comprehensive Surveyrdquo IEEE Transactions on Network andService Management vol 13 no 3 pp 518ndash532 2016

[12] B Addis D Belabed M Bouet and S Secci ldquoVirtual networkfunctions placement and routing optimizationrdquo in Proceedingsof the 4th IEEE International Conference on Cloud NetworkingCloudNet rsquo15 pp 171ndash177 October 2015

[13] H Moens and F De Turck ldquoVNF-P a model for efficientplacement of virtualized network functionsrdquo in Proceedingsof the 10th International Conference on Network and ServiceManagement (CNSM rsquo14) pp 418ndash423 Rio de Janeiro BrazilNovember 2014

[14] A Gupta M F Habib P Chowdhury M Tornatore and BMukherjee ldquoOn service chaining using Virtual Network Func-tions in Network-enabled Cloud systemsrdquo in Proceedings of the9th IEEE International Conference on Advanced Networks andTelecommuncations Systems ANTS rsquo15 pp 1ndash3 December 2015

[15] M Bouet J Leguay T Combe and V Conan ldquoCost-basedplacement of vDPI functions in NFV infrastructuresrdquo Interna-tional Journal of Network Management vol 25 no 6 pp 490ndash506 2015

[16] M C Luizelli L R Bays L S Buriol M P Barcellos andL P Gaspary ldquoPiecing together the NFV provisioning puzzle

14 Mobile Information Systems

efficient placement and chaining of virtual network functionsrdquoin Proceedings of the 14th International Symposium on IntegratedNetwork Management IM rsquo15 pp 98ndash106 IEEE TorontoCanada May 2015

[17] S Kim Y Han and S Park ldquoAn energy-Aware service functionchaining and reconfiguration algorithm in NFVrdquo in Proceedingsof the 1st International Workshops on Foundations and Applica-tions of Self-Systems FAS-W rsquo16 pp 54ndash59 September 2016

[18] V Eramo A Tosti and E Miucci ldquoServer resource dimen-sioning and routing of service function chain in NFV networkarchitecturesrdquo Journal of Electrical and Computer Engineeringvol 2016 Article ID 7139852 12 pages 2016

[19] K Hida and S-I Kuribayashi ldquoVirtual routing function allo-cation method for minimizing total network power consump-tionrdquo World Academy of Science Engineering and TechnologyInternational Journal of Electrical Computer Energetic Elec-tronic and Communication Engineering vol 10 pp 997ndash10022016

[20] C Pham N H Tran S Ren W Saad and C S Hong ldquoTraffic-aware and energy-efficient vnf placement for service chainingjoint sampling and matching approachrdquo IEEE Transactions onServices Computing 2017

[21] A Marotta and A Kassler ldquoA power efficient and robust virtualnetwork functions placement problemrdquo in Proceedings of the28th International Teletraffic Congress ITC rsquo16 pp 331ndash339September 2016

[22] C Pham H D Tran S I Moon K Thar and C S Hong ldquoAgeneral and practical consolidation framework in CloudNFVrdquoin Proceedings of the 2015 International Conference on Infor-mation Networking ICOIN rsquo15 pp 295ndash300 IEEE CambodiaCambodia January 2015

[23] V Eramo M Ammar and F G Lavacca ldquoMigration energyaware reconfigurations of virtual network function instances inNFV architecturesrdquo IEEE Access vol 5 pp 4927ndash4938 2017

[24] N E Khoury S Ayoubi and C Assi ldquoEnergy-aware placementand scheduling of network trafficflowswith deadlines on virtualnetwork functionsrdquo in Proceedings of the 5th IEEE InternationalConference on CloudNetworking CloudNet rsquo16 pp 89ndash94 IEEEPisa Italy October 2016

[25] V Eramo E Miucci M Ammar and F G Lavacca ldquoAnapproach for service function chain routing and virtual func-tion network instance migration in network function virtual-ization architecturesrdquo IEEEACM Transactions on Networking2017

[26] V Jumba S Parsaeefard M Derakhshani and T Le-NgocldquoEnergy-efficient robust resource provisioning in virtualizedwireless networksrdquo in Proceedings of the IEEE InternationalConference on Ubiquitous Wireless Broadband ICUWB rsquo15 pp1ndash5 IEEE Montreal Canada October 2015

[27] S Coniglio A Koster and M Tieves ldquoData uncertainty invirtual network embedding robust optimization and protectionlevelsrdquo Journal of Network and SystemsManagement vol 24 no3 pp 681ndash710 2016

[28] I Takouna K Sachs and C Meinel ldquoMultiperiod robustoptimization for proactive resource provisioning in virtualizeddata centersrdquo Journal of Supercomputing vol 70 no 3 pp 1514ndash1536 2014

[29] G Chochlidakis and V Friderikos ldquoRobust virtual networkembedding for mobile networksrdquo in Proceedings of the 26thIEEE Annual International Symposium on Personal Indoor andMobile Radio Communications PIMRC rsquo15 pp 1867ndash1871 IEEEKowloon Hong Kong September 2015

[30] S Coniglio A M Koster and M Tieves ldquoVirtual networkembedding under uncertainty exact and heuristic approachesrdquoin Proceedings of the 11th International Conference on the Designof Reliable Communication Networks DRCN rsquo15 pp 1ndash8 IEEEKansas Kan USA March 2015

[31] D Bertsimas and M Sim ldquoThe price of robustnessrdquoOperationsResearch vol 52 no 1 pp 35ndash53 2004

[32] F Rossi P Van Beek and T Walsh Handbook of ConstraintProgramming Elsevier Amsterdam Netherland 2006

[33] httpwwwminizincorg[34] C Schulte G Tack and M Z Lagerkvist Modeling and pro-

gramming with gecode 2010 Schulte Christian and Tack Guidoand Lagerkvist Mikael

Mobile Information Systems 3

NS request Service orchestrator NS catalog

VNF catalog

Available virtualizedresources

Placement algorithm

VNF1 VNF2 VNF3 VNF4 VNF5VNFchain

Infrastructure manager

SLA

Cluster network

microserver microserver microserver

Clou

d-RA

N

VM

VM

Virtualization layer

HW resources HW resources HW resources

VNF1VM

VNF3VM

VNF5VM

VNF4VM

VNF2VM

VM

Figure 2 VNF placement process

Moreover a common assumption of the different opti-mization models proposed to solve different versions of theVNFs placement problem is that input data is perfectlyknown which is difficult in practice Unfortunately smalldeviations in input data values may usually lead to situationswhere a previously found optimal solution is not even feasibleanymore and consequently the presence of uncertain datamay produce useless placement solutions In order to copewith uncertain input parameters RO [10] is applied to solveoptimization problems Indeed RO has been successfullyemployed in different contexts such as in resource allocationin OFDM networks [26] with uncertainty in channel stateinformation in cloud resource provisioning without perfectknowledge of demands and price [21] and even in the virtualnetwork embedding problem on a physical network substratein recent works [27ndash30] that usually deal with uncertainservice demands

Nevertheless to the best of our knowledge except thework presented in [21] there is no VNF placement proposalbased on RO Therefore to achieve our goal of designingan energy-aware placement solution using RO approach thatconsiders service demand uncertainty [21] has been relevantAuthors develop a novel power-based placement solutionwith uncertainty inVNFsCPUdemands based onRO theoryTo this aim they take into account the physical serversand network resources along with their energy efficiencyNonetheless they focus on the core network part whereas ourresearch is oriented to combined deployment of radioservicenetwork functions at the edge Additionally the placementproblem considered in this paper takes into account a het-erogeneous edge cloudmade up of a cluster of interconnectedSCs

3 Problem Description

Thenetworking environment targeted in ourwork is a 5Gdis-tributed edge cloudThis environment uses SCs that evolve tobuild multitenant CE-RANs The enhanced SCs deploymentis offered to multiple operatorsservice providers to servetheir network services in the virtualized infrastructure andengage them in a multitenant system Figure 2 shows the NSrequest and subsequent placement decision process in thisjoint radio-cloud architecture

In such an architecture whenever a tenant needs todeploy a new service itmakes a request to the ServiceOrches-trator functional element which is implemented as theNFV Orchestrator (NFVO) in ETSI-MANO terminologyThe orchestrator analyzes its NS catalog to extract thecorrespondent service chain A service chain is the sequenceof VNFs that composes the requested 5G service

The VNF chain extracted from the catalog is later used asan input for the placement algorithm together with the ser-vice constraints imposed by the key performance indicators(KPI) of the corresponding Service Level Agreement (SLA)between the tenant and the CE-RAN operator In our workthe SLA KPIs include the following network parametersmaximum accepted latency for the NS aggregated user bitrate and a robust protection parameterThis protection para-meter is related to the uncertainty of the user traffic demandWe assume that the network service has a mean aggregatedtraffic demand that is used to size the necessary resourcesin order to serve tenantrsquos user But along the service timethe user traffic demand fluctuates around this mean valueaffecting the amount of resources needed The robust pro-tection parameter is therefore related to the expected peak

4 Mobile Information Systems

traffic demand deviation and will be used to overestimate theallocation of virtualized resources in order to be prepared foreventual user peak demands

Next the placement algorithm inspects the VNF catalogfor the parametrization of theVNF instances needed tomatchthe SLA and checks the available resources provided throughthe virtualized infrastructure VNFs are characterized by theuse of resources (ie number of cores RAM and storage)according to the aggregated bit rate (ie characterized aggre-gated traffic flows of a NS) served by the NS they belongtoMoreover the service latency of eachVNF also depends onthe traffic load supported by the VNF instance Lastly virtu-alized resources may also include other hardware appliancessuch as hardware accelerators (HWA) that can improve theperformance of a VNF in terms of latency This way theseadditional appliances help matching SLA with the tenant butat a higher energy cost

All the VNFs composing the service chains of therequested network services must be allocated in the availablevirtualized resources including a given number of CPUshardware appliances RAM and storage space and networkbandwidth We assume that each core runs a single VM thatis devoted to the execution of a single VNF instance but oneVNF instance can scale from one to many VMs upon the ser-vice requirements of theNS Besides the transmission latencyalong the network topology between two VNFs allocatedin different microservers depends on the traffic load of theinvolved links and the size of the flow to be transmittedWith this information the placement algorithm assigns eachVNF instance to one or more VMs with the objective ofminimizing the energy consumption of the whole systemwhile matching the latency constraints for the NS

Finally the Infrastructure Manager functional elementVirtualized Infrastructure Manager (VIM) in ETSI-MANOterminology receives the outcome of the placement decisionand maps each VNF to the assigned resources

It is important to point out that a VNF chain includesradio-related and service-related instances The placementalgorithm introduced in this work considers a scenario withseveral tenants each of them requesting one or more NSswhich share the available resources Considering the afore-mentioned communication particularities of 5G Figure 3shows an example of a simple complete placement result ofa multitenant environment

This example considers a distributed edge cloud shared bytwo tenants Tenant 1 offers two network services the first onechaining a virtualized firewall (vFW) an intrusion detectionfunction (vID) a watermarking process (vWM) and finally acaching service (vCache) The second NS of Tenant 1 chainsthe vFW and a video analytics (VA) functionality for onlinesurveillance Tenant 2 offers a single network service thanchains the vFW a vCache service and a virtual transcodingunit (vTU) All NSs are associated with their correspondentlatency constraint

In the example both NSs belonging to the same tenantshare the radio VNF (small cell VNF or SCVNF) In this waythe common radio operations are performed in a coherentway between the different edge services of a tenant and thedata paths of the specific service instances per tenant are split

SCVNF SCVNF

vFW

VA

vFW

vWM

vCachevFW

vCache

vTU

HW

Acc

CPU

s

vID

NS2

NS1

NS1

NS1

NS1

NS1

NS1

NS1 N

S3NS3

NS1

NS2

Network topology

vFW vCache vTU

vFW VA

vFW vID vWM vCache ltlat1

ltlat2

ltlat3

Tena

nt 1

SCVNF

SCVNF

Tena

nt 2

NS1

NS2

NS3

vIDlowast

Figure 3 Multitenant placement example lowast indicates that ldquovIDrdquouses a hardware acceleration appliance

The SCVNF of each tenant will be shared by all theNSs of thistenant which means that the use of resources of the SCVNFmust then consider the sumof the aggregated flows of theNSsof the same tenant

The distinctive features of 5G communications have twoimportant implications over the placement algorithm Firstlyevery tenant allocates a unique SCVNF instance at the edgecloud in order to support the control plane and data planeoperations of all the SCs associated with that tenant Andsecondly since the SCVNFmanages the uplink and downlinktraffic between themobile users and the core network it leadsto circular forwarding paths when including edge serviceinstances In this paper we consider that the core networkfunctionalities of themobile network operator are outside theCE-RAN which means that the CE-RAN only deploys radioand edge service instances In order to keep the data pathbetween the mobile users and the core network functions alltheNSs need to start and end at the SCVNFwhichwill handlethe user plane traffic tofrom the core network

The example also shows how some VNFs needmore thanoneVM to execute the associated functionalities and howoneVNF uses a HWA tomeet the latency constraint of the server

Section 5 will show the result of more complex and com-plete placement experiments For the sake of generality thispaper does not focus on the specific modelling of the trafficpatterns the energy consumption schemes or the VNF net-work service performance Section 5 provides some referenceoutcomes on the potential benefits of the introduced VNFplacement algorithm in the considered network scenario but

Mobile Information Systems 5

the applicability of the solution to other scenarios shouldbe fed by experimental modelling data specific to the targetscenario

4 Robust Constraint-Based VNFPlacement Proposal

Once the 5G VNF placement problem under study is pre-sented in this section we propose a placement solution tothat problem using the RO approach To this end first weformulate the energy-aware VNF placement model in thenext subsection Then we provide our RO-based placementproposal in Section 42

41 ProblemModelling As stated before the VNF placementproblem addressed in this work focuses on the assignment ofthe VNFs composing each NS to the provided 5G networkinfrastructure Therefore it constitutes a discrete or com-binatorial optimization problem where the minimizationof energy consumption constitutes the optimality criterionSummary of Model Indexes Sets Parameters and DecisionVariables reports and summarizes the mathematical nota-tions used for the model indexes sets input parameters anddecision variables of VNF placement problem that is definedin the following

The Problem Model The energy-aware VNF placement opti-mization problem

min120587isinΠ

119875120587 (1)

st 119889120587119894 lt 119863max119894 forall119894 isin I 120587 isin Π (2)

119898120587119903119909 lt 119872119903119909 forall119903 isin R 119909 isin X 120587 isin Π (3)

119905119897120587119909 lt BW119909 forall119909 isin X 120587 isin Π (4)

119875120587 = sum119894isinI

sum119895isinJ119894

119875120587119894119895 (5)

119875120587119894119895

= sum119909isinX

sum119910isinY119909

119875cpu119909119910119894119895 119860119909119910

119894119895 + sum119894isinI

sum119895isinJ119894

119875switch119894119895119871 119894119895

+ sum119909isinX

119875sc119909119860119909119910

119894119895

(6)

119889120587119894 = sum119895isinJ119894

119889120587119894119895 (7)

119889120587119894119895

= sum119909isinX

sum119910isinY119909

119889119901119909119910119894119895 119860119909119910

119894119895

+ sum1199091isinX

sum1199101isinY1199091

sum1199092isinX

sum1199102isinY1199092

11988911989711990911199101 11990921199102119894119895 11986011990911199101119894119895 11986011990921199102119894119895+1

(8)

sum119894isinI

sum119895isinJ119894

119860119909119910119894119895 = 1 forall119909 isin X 119910isin Y119909 (9)

119898120587119903119909 = sum119894isinI

sum119895isinJ119894

sum119910isinY

119898119903119894119895119860119909119910

119894119895 (10)

119905119897120587119909 = sum119894isinI

sum119895isinJ119894

sum119910isinY

119879119894119860119909119910

119894119895 1198601199091015840119910

119894119895+1 (11)

We consider a set X of small cells that compose the CE-RAN and a set Y119909 of cores included in the microserver ofthe SC 119909 Additionally we also consider a set of I networkservices offered by the CE-RAN operator to the differenttenants and a set J119894 of VNFs that form the functionalchain of the NS 119894 Finally R represents the set of differentresources available in the microservers for the execution ofthe VNFs (storage RAM hardware accelerators etc) Henceour constraint-based expert systemmust determine in whichVM 119910 of a SC 119909 each VNF 119895 of a NS 119894 is located

Let Π denote the set of all possible power-aware andlatency- and resource-constrained VNF placement policieswhich decide the mapping of each VNF of available NSs toavailable VMs in the provided SCs We formulate our VNFplacement optimization problem aimed atminimizing powerconsumption with delay requirements as (1)

In reference to the energy part we define the total powerconsumption 119875120587 as the sum of the energy demand of all theallocated VNFs 119875120587119894119895 (5) Then for each VNF instance thepower consumption is expressed in (6) as the sum of thepower consumption of VNF 119895 of NS 119894 due to the CPU useof VM 119910 of SC 119909 the power consumption of the physicalswitch due to the forwarded traffic of the VNF and the powerconsumption of SC 119909 because of being switched on

The power consumption of the cores that run the VMsdepends of the aggregated user traffic served by the corre-spondent VNF and this relationship may be nonlinear Itmust be also understood that when a VNF instance scales toa higher number of cores the individual load of the involvedCPUs decreases leading to a lower power consumption percore Additionally the energy consumption of the networkingdevices (eg an Ethernet switch) depends on the aggregatedflow that traverses the device to forward data packets betweenSCs and the number of active ports Finally it is realized thatthe power wasted because the SC is active is constant notaffecting the optimization results Besides we express per-NSdelay in (7) as the sum of per-VNF delays and the per-VNFdelay in (8) as the sum of the processing delays introducedby the execution of the VNF instance plus the path delayintroduced by the traffic going from one VNF to the next onein the chain defined in the correspondent NS

On the one hand the processing delay in the CPU growswith the supported user traffic load On the other hand weconsider that the pathlink delay is composed of several linkdelays fromtoVMs tofrom the switches plus the delay in theswitches Hence depending on the adopted placement policy120587 each NS will follow a different path through the networkIn such a way if the VNFs belonging to a NS are placed in thesame SC the service chain traverses virtualinternal links andthe virtual switch of the selected SC whereas if the VNFs of

6 Mobile Information Systems

a NS are located in different SCs external links and the physi-cal switch is also used therefore generally 11988911989711990911199101 11990921199102119894119895 =11988911990911199101vs + 119889vs

1199091+ 119889vs1199091 119904 + 119889119904 + 119889119904vs1199092 + 119889vs

1199092+ 119889vs11990921199102 but for

a given VNF if the next VNF in its chain holds in the sameSC its link delay is only affected because of passing throughthe virtual switch Note that in case a VNF is the last VNF inthe chain the link delay is null with the sums referring to thelink delay for that 119895 being null

In order to guarantee that each VM holds only a singleVNF constraint (9) must be satisfied Nevertheless weconsider that each VNF of a NS can be instantiated as a VMthat is executed over several cores of the same SC Apart fromthat as previously detailed in Section 3 the first VNF and thelast VNF in a service chain are the same (the SCVNF) in orderto perform the specific functions that allow the split betweencontrol and data plane required in 5G communicationsTherefore it implies that 1198601199091199101198941 = 119860119909119869119894119894119895

Finally the model must also include the constraintsreferred to as the available resources In this sense (3) statesthat the consumption of the resource 119903 due to the placementof VNFs in the microserver of the SC 119909119898120587119903119909 must not exceedthe total amount of resource 119872119903119909 where 119898120587119903119909 is defined in(10) Similarly (4) constrains the traffic through the networklinks 119905119897120587119909 to the maximum capacity of the links BW119909 as thesum of the traffic flows of the network services that must beforwarded from a VNF in one SC to the next VNF in anotherSC

42 Robust Constraint-Based Solution As described inSection 2 many existing placement works assume perfectknowledge on input parameters pointing thus to the for-mulation of a deterministic optimization problem Unfortu-nately in the real world the estimated values of the inputparameters may differ due to biased data unrealistic assump-tions or numerical errors affecting the obtained optimalsolution and its performance This potential deviation ofthe nominal or expected input may lead to the violation ofthe problem constraints and therefore make the obtainedsolution suboptimal or even unfeasible

The uncertainty of modelling parameters has been tra-ditionally handled through stochastic programming andsensitivity analysis but robust optimization techniques haverecently arisen as a powerful tool to manage the impact ofuncertain input sets RO seeks an uncertainty-immunizedsolution with an acceptable performance under the real-ization of the uncertain inputs becoming a conservativeworst-case oriented methodology The main tools of RO areuncertainty sets and a robust counterpart problem [31] Theuncertainty of the input data is described later in this sectionwhile the robust counterpart problem is the deterministicmodel previously detailed in Section 41

Here we propose a robust constraint-based placementsolution that deals with service demand uncertainty Withthis RO approach we consider a tradeoff between theproblem optimization and the protection from deviationscaused by input parameters uncertainty This uncertainty inservice demand has an impact on several components on theformulation previously presented in Section 41 On the one

hand we consider that a VM which is running a specificVNF requires an expected use of CPU and thus a certainconstant power consumption due to CPU Neverthelessthe uncertainty in traffic demand may cause uncertainty inthe CPU requirementuse and consequently in its energyconsumption and processing delays On the other hand bothenergy consumption and delays in switches and links dependon that demand uncertainty as well

In order to obtain a proposal for the placement problemwith uncertain service demand we employ the Soyster [10]method used in the framework of RO That technique isalso known as the worst-case approach where the solutionsare achieved using the most extreme expected values of theuncertain variables Therefore that kind of resolution guar-antees feasible solutions for any value of the uncertain servicedemand

In reference to the modelling of the uncertain trafficdemand per NS 119894 we consider an expected traffic demand119864[119879119894] = 119879119894 plus a term due to uncertainty in the servicedemandUTmax119894 We assume that the randomvariableUTmax119894is symmetrically distributed within [minusΔ119879119894 +Δ119879119894] sdot 119879119894 andwith mean zero Furthermore the sum of deviations of theuncertain service demand should not exceed the worst-casemaximum value 119879max119894 fulfilling (12) In this way the valueof 119879max119894 controls the tradeoff between the robustness andthe impact on the optimization higher 119879max119894 leads to worseobjective function values but protects from more parameterdeviations

119880119879119894Δ119879119894

le 119879max119894 forall119894 (12)

5 Analysis of the Results

This section presents the results obtained with the simulationof the proposed robust constraint-based expert system for theplacement of VNF chains in an emerging 5G distributed edgecloud As previously introduced in Section 2 the mathemat-ical VNF placement problemmodel presented in Section 4 isa combinatorial or discrete optimization problem equivalentto the well-known knapsack problem This means that thedecision problem is NP-complete while the optimizationproblem is NP-hard [11] The proposed optimization modelhas been solved using constraint programming as a powerfulparadigm for solving combinatorial search problems [32]

We usedMinizinc [33] a free and open-source constraintmodelling language to model the optimization and con-straint satisfaction VNF placement problem in a high-levelsolver-independent wayThen Minizinc compiles this modelinto FlatZinc a solver input language that is understoodby a wide range of solvers In our case Gecode solver [34]provided the best performance in the search of the optimalsolution

The experiments that are described and discussed belowhave been conducted in a single computer with a 27GHzIntel Core i5 processor and 16GB RAM This simple con-figuration provides solutions within minutes for simpleexperiments with loose constraint satisfaction When theexperiments demand tighter latency or resource constraints

Mobile Information Systems 7

Virt

ual i

nfra

struc

ture

Corenetwork

Figure 4 Evaluation scenario

the solving time increases to hours depending on the specificconfiguration of the experiment

Next we first describe the analysis scenario and the con-figuration of the performed experiments to later discuss theresults of those experimentsThe objective of the experimentsis to demonstrate that the placement mechanism is able toallocate multitenant service chains to the available resourcesconsidering latency constraints For this reason the numeri-cal values employed in the next section are just experimentaland do not respond to real world measurements

51 Description of the Analysis Scenario Our evaluationscenario is depicted in Figure 4 We consider a 10-SC deploy-ment for example to cover a football stadium for periodicalflash eventsThe SCs are connected in a star topology throughan Ethernet switch Each SC is composed of 8 cores 16GBRAM 100GB of storage and 1HWA (eg GPU)

We also consider 2 NSs that can be offered at 3 serviceflavours (bronze silver and gold) that determine the latencyconstraint for eachNS and the nominal aggregate user bit rateserved as shown in Figure 5

Table 1 shows the experimental energy model and Table 2shows the modelling of the VNFs that compose the proposedNSs for the user traffic demand that will be employed in theevaluation experiments The user traffic values include thenominal bit rates of services NS1 and NS2 (130Mbps and90Mbps resp) as well as the values when a 20 protectionparameter is applied to the nominal values (156Mbps and108Mbps resp) in order to study the effect of robust optimi-zation techniques on the placement decision

As a final remark the placement behavior of VNFs canbe altered by the use of HWA The assignment of a HWA toa VNF implies that the service latency of the correspondentinstance is reduced 3 times at a cost of incrementing thepower consumption by 10 It is important to note that whenthe VNF instance needs more than 1 CPU for executingwithout the assistance of a HWA then if the placementalgorithm assigns it to a HWA the VNF will need a singlecore

52 Discussion of the Placement Results As stated before theevaluation scenario includes 3 tenants that hire a combination

8 Mobile Information Systems

VNF1 VNF2 VNF3 VNF5 VNF6

VNF1 VNF2 VNF3 VNF4

NS1

NS2

Service flavourBronze Silver Gold

NS1

NS2 Max Flow = 90MbpsMax Lat = 90ms Max Lat = 80ms

Max Flow = 130－＜ＪＭ

Max Lat = 150ＧＭ

Max Lat = 110ＧＭ

Max Flow = 90Mbps

Max Flow = 130－＜ＪＭ

Max Lat = 120ＧＭ

Max Flow = 90Mbps

Max Flow = 130－＜ＪＭ

Max Lat = 100ＧＭ

Figure 5 Description of NSs and service levels

Table 1 Energy models of the cores of the microservers and of the network switch related to the usage percentage

Usage 0 10 20 30 40 50 60 70 80 90 100119875router 0 1 2 5 7 9 11 13 14 15 15119875cpu 5 50 70 100 130 150 165 170 180 185 190

Table 2 VNF characterization for the service flavours of the evaluation scenario

VNF1 VNF2 VNF3 VNF4 VNF5 VNF6Number of cores per instance

90Mbps 1 2 1 1 1 1108Mbps 1 2 2 1 1 1130Mbps 1 2 2 1 1 1156Mbps 1 2 2 1 1 1

CPU usage90Mbps 30 20 90 50 50 20108Mbps 40 30 60 60 60 30130Mbps 40 30 60 60 60 30156Mbps 40 30 90 40 20 30

Service latency90Mbps 10ms 15ms 30ms 35ms 20ms 25ms108Mbps 10ms 15ms 35ms 40ms 25ms 30ms130Mbps 10ms 15ms 35ms 40ms 25ms 30ms156Mbps 12ms 16ms 40ms 50ms 30ms 35ms

of network servicesflavours to the SC operator In this wayin the following first we are going to evaluate the behavior ofthe placement algorithm for the three service levels and nextwe will show the effect of applying RO techniques to thoseresults

521 Bronze Flavour Scenario Results with 0 Robust Pro-tection Figure 6 shows the placement results of an scenariowhere all the network services operate at bronze flavourDifferent colours represent each tenant in order to dif-ferentiate the VNFs that belong to its NS and differenttones distinguish NSs of the same tenant As a result ofthe placement algorithm all the NSs are allocated in theavailable virtualized resources with energy consumption costof 3712W meeting all the latency constraints The figure

displays the assignment of VNFs to the SCs highlightingsome placement particularities As explained in Section 3the VNF chains of the same tenant share their SCVNF Inour example Tenant 1 and Tenant 2 both have two NSsthat are served with a single SCVNF Furthermore eachCPU executes a single VM but some VNF instances needtwo cores to manage the user demand In connection withthis behavior the VNF scalability feature is also visible inFigure 6 observing the placement of for example VNF3for the different NSs NS1 with a higher aggregated userdemand needs two CPUs to instantiate VNF3 but since NS2operates at a lower user demand the instantiation of VNF3needs just a single core Note as well that bronze serviceflavour does not require HW acceleration to meet the latencyconstraints of the NSs Finally it can also be observed that

Mobile Information Systems 9

Tenant 1 Tenant 2 Tenant 3

NS1 NS2 NS1 NS2 NS1

Service type Bronze Bronze Bronze Bronze Bronze

Max ow

Max latency

Deviation 0 0 0 0 0

Peak ow

VNF1VNF3

VNF1

VNF6

VNF1

SCVNF

VNF3VNF2

VNF1VNF3

VNF1VNF2

VNF2VNF3

VNF5

VNF5VNF2

VNF5

SCVNF

VNF1

VNF5

VNF3

VNF4

VNF5

VNF6

SCVNF

VNF1

VNF1VNF2

VNF3

VNF4

VNF1

960Mbps

960Mbps

780Mbps

780Mbps

780Mbps

260Mbps

VNF2

VNF6

CPU

s

Finallatency

Tenant 1NS1

NS2

Tenant 2NS1

NS2

Tenant 3 NS1

Total energy consumption

Switc

hH

A A

cc

132ＧＭ

102ＧＭ

132ＧＭ

102ＧＭ

130ＧＭ

3712７

130－＜ＪＭ 90－＜ＪＭ 130－＜ＪＭ 130－＜ＪＭ90－＜ＪＭ

130－＜ＪＭ 90－＜ＪＭ 130－＜ＪＭ 130－＜ＪＭ90－＜ＪＭ

150ＧＭ 110ＧＭ 150ＧＭ 110ＧＭ 150ＧＭ

Figure 6 Placement results of bronze flavour setting with 0 robust protection

the algorithm tries to group asmanyVNFs as possible in eachserver in order to reduce the data traffic across the switch andminimize its load and therefore its energy waste

522 Silver Flavour Scenario Results with 0 Robust Pro-tection The evaluation experiment analyzed in this sectionstudies the outcome of the placement algorithm when thelatency constraints become more strict with silver serviceflavour Due to the further limitation of maximum latencyvalues (20 stricter) the placement mechanism is compelledto use HW acceleration units Figure 7 shows that at thisservice level all the NSs must use HW acceleration in oneof the VNFs composing the service chain As a consequencethe resulting total latency of the NSs decreases to meet theservice level requirements but in return the global powerconsumption grows by 22 up to 4547W

Apart from that the placement results illustrate the samebehavior as the bronze flavoured example of Section 521regarding the allocation of VNFs in the available resourcesscale-inout features and multitenancy

The evaluation experiments of Sections 521 and 522operate with a nominal aggregated user traffic to perform theoptimization decision-making process However a perfectlyknown user traffic demand is seldom available Usuallythere is a certain level of uncertainty in the behavior of theuser demand We include robust optimization techniques to

allow the placement algorithm to insert a defined level ofuncertainty and protect the system against the undesirableeffects of demand peaksThe evaluation examples in Sections523 and 524 include a robust protection parameter andanalyze its effect on the global power consumption

523 Silver Flavour Scenario Results with 20 Robust Protec-tion This section studies the placement outcome of the eval-uation scenario described in the previous section introducinga 20 robust protection parameter This modification on thescenario configuration implies an increase of the 20 of theaggregated user traffic as an input of the decision-makingprocess in order to be prepared for potential demand peaksduring the service operation

Figure 8 exhibits that VNF3 of NS2 must scale as a con-sequence of the traffic growth and now needs two CPUs foreach instance The traffic increase also has an impact on theprocessing latency of the VNFs causing an adjustment in theuse of HW accelerators As a consequence some VNFs thatrequired 2CPUs for their operation in the previous evalu-ation scenario now only need a single core in combinationwith the HWA Therefore in some cases the higher powerconsumption that is implicit to the use of HWAs may bebalanced with a decrease on the number of CPUs required forthe VNF Besides the higher aggregated user traffic involvesa more intense use of the network switch These factors have

10 Mobile Information Systems

SCVNF

VNF6

VNF1VNF1

VNF2

VNF3

VNF3

VNF3

VNF5

VNF2

VNF1

VNF2

SCVNF

VNF1

VNF2

VNF4

VNF5

VNF6

SCVNF

VNF2

VNF3

VNF4

VNF5

VNF6

VNF3

VNF1

Finallatency

Tenant 1NS1

NS2

Tenant 2NS1

NS2

Tenant 3 NS1

Total energy consumption

VNF6VNF1VNF1VNF2VNF2

Tenant 1 Tenant 2 Tenant 3

NS1 NS2 NS1 NS2 NS1

Service type Silver Silver Silver Silver Silver

Max ow

Max latency

Deviation 0 0 0 0 0

Peak ow

960Mbps

960Mbps

780Mbps

780Mbps

260Mbps

780MbpsSw

itch

100ＧＭ

118ＧＭ

116ＧＭ

4547７

130－＜ＪＭ 90－＜ＪＭ 130－＜ＪＭ 130－＜ＪＭ90－＜ＪＭ

130－＜ＪＭ 90－＜ＪＭ 130－＜ＪＭ 130－＜ＪＭ90－＜ＪＭ

120ＧＭ 120ＧＭ 120ＧＭ

CPU

sH

A A

cc

88ＧＭ

88ＧＭ

90ＧＭ 90ＧＭ

Figure 7 Placement results of silver flavour setting with 0 robust protection

an impact on the power consumption of the componentsof the network resulting in a global energy consumption of5047W which implies an increment of 11 compared to theunprotected placement

524 Gold Flavour Scenario Results with 20 Robust Protec-tion Finally the last evaluation example sets even stricterlatency constraints to theNSs to be placed and includes a 20robust protection parameter Figure 9 depicts the results ofthe placement algorithm with this configuration Again thereduction of latency values forces the increment of the use ofHW acceleration In this case the main consequence is thespreading of the VNFs across the SCs to use the single HWAavailable in each cellThe use of HWAs the increase of powerconsumption due to higher user aggregated traffic and thehigher network traffic traversing the switch cause the globalenergy consumption to grow up to 5569W

As a final remark Figure 10 shows the comparison ofthe energy consumption and use of resources (cores activeCESCs and HWA) related to the robust protection level ina wider collection of scenarios In particular the charts ofFigure 10 compare the results of the placement algorithm forthe three service levels described in the previous sections(bronze silver and gold) combined with three robust protec-tion levels (0 or no protection 10 and 20) Obviouslythe main trend is that both power and resource consumptionincrease with higher values of protection but the associatedcost can be assumed in favor of reducing the impact of

unexpected user demand peaks However there are somecases in which the protection parameter does not have soadverse implications For example it can be observed howthe number of used cores in the silver service level with a10 protection level is lower than in the case of no robustprotection The reason of this behavior is how the algorithmuses HWA appliances in this case The increase of the usertraffic demand in the protected services implies the scaling upof someVNFs that nowmust usemore cores for the operationof theNSsThen the placement algorithm decides to useHWaccelerators for those specific VNFs compensating this waythe higher energy cost of HWAs with a lower use of CPUcores With all these considerations that can be extractedfrom a deeper analysis of robust placement results robustoptimization can be applied to decision-making problemsnot ending just with the optimization problem

The proposed steps in this paper can be used in anyplacement decision-making context and its goal is to providea tool that helps system administrator to evaluate the possibleimpact of any decision on the performance of the deploymentunderneath

6 Conclusions

This paper introduces a constraint-based expert system tosupport the decision-making process of VNF chain place-ment in distributed edge cloud networks The placement

Mobile Information Systems 11

SCVNF

VNF5

VNF6VNF5VNF2

VNF3

VNF5

VNF1

VNF2

VNF3

VNF1

VNF2

VNF3

VNF6

SCVNF

VNF1

VNF1

VNF3

VNF4

SCVNF

VNF1

VNF2

VNF3

VNF4

840Mbps

840Mbps

936Mbps

624Mbps

312Mbps

312Mbps

936Mbps

VNF6

Tenant 1NS1

NS2

Tenant 2NS1

NS2

Tenant 3 NS1

Total energy consumption

VNF5VNF6VNF5VNF1VNF4VNF4

VNF2

Finallatency

Tenant 1 Tenant 2 Tenant 3

NS1 NS2 NS1 NS2 NS1

Service type Silver Silver Silver Silver Silver

Max ow

Max latency

Deviation 20 20 20 20 20

Peak ow

116ＧＭ

110ＧＭ

118ＧＭ

5047７

130－＜ＪＭ 90－＜ＪＭ 130－＜ＪＭ 130－＜ＪＭ90－＜ＪＭ

156－＜ＪＭ 108－＜ＪＭ 156－＜ＪＭ 108－＜ＪＭ 156－＜ＪＭ

150ＧＭ 110ＧＭ 150ＧＭ 110ＧＭ 150ＧＭ

79ＧＭ

79ＧＭ

CPU

sH

A A

ccSw

itch

Figure 8 Placement results of silver flavour setting with 20 robust protection

algorithm applies robust optimization techniques to theminimization of the global power consumption of the systemsubject to the bit rate and latency constraints of the networkservices The power consumption of the components of thesystem depends on the usage percentage which is a con-sequence of the aggregated user traffic demand The modelassumes that the relationships between the user trafficdemand the power consumption and the use of the resourcesmay be nonlinear Besides the placement algorithmconsidersthe 5G communication particularities of both control anddata plane and allows multitenancy Finally the algorithmalso includes VNF scaling features and the use of hardwareacceleration

The results show that the placement algorithm succeeds inallocating theVNFs that compose theNSs of different tenantsin the available resourcesmeeting the latency constraints andminimizing the energy consumption of thewhole systemThepresented evaluation examples demonstrate the features ofthe constraint-based expert system and illustrate the impactof the selected service flavour and robust protection on theglobal power consumption and therefore the exploitationcost of the 5G distributed edge cloud

Further researchwill extend the characterization of VNFsto include variable output bit rate This feature will modelthe behavior of for example transcoding functions A future

version of the algorithm will also incorporate the existenceof a virtual switch to connect the VMs running in the samemicroserver Another enhancement of the model will bethe combination of heterogeneous SCs with different con-figurations and resource consumption models Finally theobtained placement results will be used for a technoeconomicstudy that will analyze the relationship between the serveraggregated user traffic cost of the required infrastructure andthe pricing of the services This study will also consider theintegration of the placement decision problem with the radioside

Summary of Model Indexes SetsParameters and Decision Variables

Indexes

120587 Index for VNF placement policy119909 Index for SC microserver119910 Index for core number in each microserver119894 Index for NS119895 Index for VNF in a network service chain119903 Index for physical resources in

SC microserver

12 Mobile Information Systems

VNF6VNF5VNF1VNF6VNF2SCVNF

VNF2

VNF3

VNF5

VNF3

VNF1

VNF2

VNF3

VNF5

SCVNF

VNF1

VNF1

VNF3

VNF4

SCVNF

VNF2

VNF3

VNF4

VNF1

Tenant 1NS1

NS2

Tenant 2NS1

NS2

Tenant 3 NS1

Total energy consumption

VNF6VNF5VNF1VNF6VNF2VNF5VNF6VNF4VNF4

VNF2

840Mbps

840Mbps

936Mbps

936Mbps

312Mbps

312Mbps

312Mbps

312Mbps

312Mbps

Finallatency

Tenant 1 Tenant 2 Tenant 3

NS1 NS2 NS1 NS2 NS1

Service type Gold Gold Gold Gold Gold

Max ow

Max latency

Deviation 20 20 20 20 20

Peak ow

5569７

130－＜ＪＭ 90－＜ＪＭ 130－＜ＪＭ 130－＜ＪＭ90－＜ＪＭ

156－＜ＪＭ 108－＜ＪＭ 156－＜ＪＭ 108－＜ＪＭ 156－＜ＪＭ

100ＧＭ 100ＧＭ 100ＧＭ

79ＧＭ

95ＧＭ

89ＧＭ

79ＧＭ

86ＧＭ

CPU

sH

A A

ccSw

itch

80ＧＭ 80ＧＭ

Figure 9 Placement results of gold flavour setting with 20 robust protection

34

32

30

28

260

10

20Robust protection ()

Num

ber o

f use

dco

res

10864200

10

20Robust protection ()

HW

AN

umbe

r of u

sed

0

10

20Robust protection ()

987654N

umbe

r of a

ctiv

eCE

SCs

6000

5500

5000

4500

4000

35003000

010

20

Ener

gy co

nsum

ptio

n(W

)

Robust protection ()

Gold

Silver

Bronze

Gold

Silver

Bronze

Gold

Silver

Bronze

Gold

Silver

Bronze

Figure 10 Comparison of energy and resource consumption for different levels of robust protection

Mobile Information Systems 13

Sets

Π Set of possible VNF placement policies119883 Set of SC microservers119884 Set of cores composing a SC microserver119868 Set of offered NSs119869119894 Set of VNFs in NS 119894119877 Set of resources available in

each SC microserver

Input Parameters

119879119894 Contracted maximum aggregated usertraffic for network service 119894

119863max119894 Maximum latency allowed for NS 119894119872119903119909 Amount of resource 119903 available in

SC microserver 119909119875sc119909 Power consumption of SC 119909 because of

being switched on119875cpu119909119910119894119895 Power consumption of VM 119910 of SC 119909 due

to CPU use of VNF 119895 of NS 119894119875switch119894119895 Power consumption in the physical switch

due to CPU use of VNF 119895 of NS 119894

Decision Variables

119875120587 Total power consumption ofplacement policy 120587

119875120587119894119895 Power consumption of VNF 119895 of NS 119894 inplacement policy 120587

119898120587119903119909 Consumption of resource 119903 in SCmicroserver 119909 in placement policy 120587

119905119897120587119909 Traffic of the link that connects SCmicroserver 119909 with the switchinplacement policy 120587

119860119909119910119894119895 Binary variable that expressesthe assignment of VNF 119895 of NS 119894 toVM 119910 in SC 119909

119871 119894119895 Binary variable to express the use of thenetwork switch to forward the flow ofVNF 119895 of NS 119894 to the next VNF

119889119894 Delay of NS 119894119889119894119895 Delay of VNF 119895 of NS 119894119889119901119909119910119894119895 Processing delay in the CPU of VNF 119895 of

NS 119894 in VM 119910 of SC 1199091198891198971199091119910111990921199102119894119895 Path delay for NS 119894 between VNF 119895

(located in VM 1199101 of SC 1199091) and VNF119895 + 1 (located in VM 1199102 of SC 1199092)

119889vs119909119910 Link delay between the virtual switch inSC 119909 and the VM 119910 in SC 119909

119889vs119909 119904 Link delay between the virtual switch inSC 119909 and the physical switch

119889vs119909 Delay in the virtual switch of SC 119909

119889119904 Delay in the physical switch

Conflicts of Interest

The authors declare that they have no conflicts of interest

Acknowledgments

The research leading to these results has been supportedby the EU funded H2020 5G-PPP Project SESAME (GrantAgreement 671596) and the Spanish MINECO Project5GRANVIR (TEC2016-80090-C2-2-R)

References

[1] ITU-R M2083 IMT Vision-Framework and overall objectivesof the future development of IMT for 2020 and beyond 2015

[2] P Demestichas A Georgakopoulos D Karvounas et al ldquo5G onthe Horizon key challenges for the radio-access networkrdquo IEEEVehicular Technology Magazine vol 8 no 3 pp 47ndash53 2013

[3] I F Akyildiz S Nie S-C Lin and M Chandrasekaran ldquo5Groadmap 10 key enabling technologiesrdquo Computer Networksvol 106 pp 17ndash48 2016

[4] ETSI Mobile-edge computing Introductory technical whitepaper 2014

[5] B Han V Gopalakrishnan L Ji and S Lee ldquoNetwork functionvirtualization challenges and opportunities for innovationsrdquoIEEE Communications Magazine vol 53 no 2 pp 90ndash97 2015

[6] R Mijumbi J Serrat J Gorricho N Bouten F De Turckand R Boutaba ldquoNetwork function virtualization state-of-the-art and research challengesrdquo IEEE Communications Surveys ampTutorials vol 18 no 1 pp 236ndash262 2016

[7] Ericsson White Paper Cloud-RANndash the benefits of virtualiza-tion centralization and coordination 2015

[8] B Blanco J O Fajardo I Giannoulakis et al ldquoTechnologypillars in the architecture of future 5G mobile networks NFVMEC and SDNrdquo Computer Standards and Interfaces vol 54 pp216ndash228 2017

[9] ETSI Network Functions Virtualization (NFV) Managementand Orchestration 2014

[10] A Ben-Tal L E Ghaoui and A Nemirovski Robust Optimiza-tion Aharon Ben-Tal Laurent El Ghaoui Arkadi Nemirovski-Google Books Princeton University Press Princeton NewJersey NY USA 2009

[11] J Gil Herrera and J F Botero ldquoResource Allocation in NFVA Comprehensive Surveyrdquo IEEE Transactions on Network andService Management vol 13 no 3 pp 518ndash532 2016

[12] B Addis D Belabed M Bouet and S Secci ldquoVirtual networkfunctions placement and routing optimizationrdquo in Proceedingsof the 4th IEEE International Conference on Cloud NetworkingCloudNet rsquo15 pp 171ndash177 October 2015

[13] H Moens and F De Turck ldquoVNF-P a model for efficientplacement of virtualized network functionsrdquo in Proceedingsof the 10th International Conference on Network and ServiceManagement (CNSM rsquo14) pp 418ndash423 Rio de Janeiro BrazilNovember 2014

[14] A Gupta M F Habib P Chowdhury M Tornatore and BMukherjee ldquoOn service chaining using Virtual Network Func-tions in Network-enabled Cloud systemsrdquo in Proceedings of the9th IEEE International Conference on Advanced Networks andTelecommuncations Systems ANTS rsquo15 pp 1ndash3 December 2015

[15] M Bouet J Leguay T Combe and V Conan ldquoCost-basedplacement of vDPI functions in NFV infrastructuresrdquo Interna-tional Journal of Network Management vol 25 no 6 pp 490ndash506 2015

[16] M C Luizelli L R Bays L S Buriol M P Barcellos andL P Gaspary ldquoPiecing together the NFV provisioning puzzle

14 Mobile Information Systems

efficient placement and chaining of virtual network functionsrdquoin Proceedings of the 14th International Symposium on IntegratedNetwork Management IM rsquo15 pp 98ndash106 IEEE TorontoCanada May 2015

[17] S Kim Y Han and S Park ldquoAn energy-Aware service functionchaining and reconfiguration algorithm in NFVrdquo in Proceedingsof the 1st International Workshops on Foundations and Applica-tions of Self-Systems FAS-W rsquo16 pp 54ndash59 September 2016

[18] V Eramo A Tosti and E Miucci ldquoServer resource dimen-sioning and routing of service function chain in NFV networkarchitecturesrdquo Journal of Electrical and Computer Engineeringvol 2016 Article ID 7139852 12 pages 2016

[19] K Hida and S-I Kuribayashi ldquoVirtual routing function allo-cation method for minimizing total network power consump-tionrdquo World Academy of Science Engineering and TechnologyInternational Journal of Electrical Computer Energetic Elec-tronic and Communication Engineering vol 10 pp 997ndash10022016

[20] C Pham N H Tran S Ren W Saad and C S Hong ldquoTraffic-aware and energy-efficient vnf placement for service chainingjoint sampling and matching approachrdquo IEEE Transactions onServices Computing 2017

[21] A Marotta and A Kassler ldquoA power efficient and robust virtualnetwork functions placement problemrdquo in Proceedings of the28th International Teletraffic Congress ITC rsquo16 pp 331ndash339September 2016

[22] C Pham H D Tran S I Moon K Thar and C S Hong ldquoAgeneral and practical consolidation framework in CloudNFVrdquoin Proceedings of the 2015 International Conference on Infor-mation Networking ICOIN rsquo15 pp 295ndash300 IEEE CambodiaCambodia January 2015

[23] V Eramo M Ammar and F G Lavacca ldquoMigration energyaware reconfigurations of virtual network function instances inNFV architecturesrdquo IEEE Access vol 5 pp 4927ndash4938 2017

[24] N E Khoury S Ayoubi and C Assi ldquoEnergy-aware placementand scheduling of network trafficflowswith deadlines on virtualnetwork functionsrdquo in Proceedings of the 5th IEEE InternationalConference on CloudNetworking CloudNet rsquo16 pp 89ndash94 IEEEPisa Italy October 2016

[25] V Eramo E Miucci M Ammar and F G Lavacca ldquoAnapproach for service function chain routing and virtual func-tion network instance migration in network function virtual-ization architecturesrdquo IEEEACM Transactions on Networking2017

[26] V Jumba S Parsaeefard M Derakhshani and T Le-NgocldquoEnergy-efficient robust resource provisioning in virtualizedwireless networksrdquo in Proceedings of the IEEE InternationalConference on Ubiquitous Wireless Broadband ICUWB rsquo15 pp1ndash5 IEEE Montreal Canada October 2015

[27] S Coniglio A Koster and M Tieves ldquoData uncertainty invirtual network embedding robust optimization and protectionlevelsrdquo Journal of Network and SystemsManagement vol 24 no3 pp 681ndash710 2016

[28] I Takouna K Sachs and C Meinel ldquoMultiperiod robustoptimization for proactive resource provisioning in virtualizeddata centersrdquo Journal of Supercomputing vol 70 no 3 pp 1514ndash1536 2014

[29] G Chochlidakis and V Friderikos ldquoRobust virtual networkembedding for mobile networksrdquo in Proceedings of the 26thIEEE Annual International Symposium on Personal Indoor andMobile Radio Communications PIMRC rsquo15 pp 1867ndash1871 IEEEKowloon Hong Kong September 2015

[30] S Coniglio A M Koster and M Tieves ldquoVirtual networkembedding under uncertainty exact and heuristic approachesrdquoin Proceedings of the 11th International Conference on the Designof Reliable Communication Networks DRCN rsquo15 pp 1ndash8 IEEEKansas Kan USA March 2015

[31] D Bertsimas and M Sim ldquoThe price of robustnessrdquoOperationsResearch vol 52 no 1 pp 35ndash53 2004

[32] F Rossi P Van Beek and T Walsh Handbook of ConstraintProgramming Elsevier Amsterdam Netherland 2006

[33] httpwwwminizincorg[34] C Schulte G Tack and M Z Lagerkvist Modeling and pro-

gramming with gecode 2010 Schulte Christian and Tack Guidoand Lagerkvist Mikael

4 Mobile Information Systems

traffic demand deviation and will be used to overestimate theallocation of virtualized resources in order to be prepared foreventual user peak demands

Next the placement algorithm inspects the VNF catalogfor the parametrization of theVNF instances needed tomatchthe SLA and checks the available resources provided throughthe virtualized infrastructure VNFs are characterized by theuse of resources (ie number of cores RAM and storage)according to the aggregated bit rate (ie characterized aggre-gated traffic flows of a NS) served by the NS they belongtoMoreover the service latency of eachVNF also depends onthe traffic load supported by the VNF instance Lastly virtu-alized resources may also include other hardware appliancessuch as hardware accelerators (HWA) that can improve theperformance of a VNF in terms of latency This way theseadditional appliances help matching SLA with the tenant butat a higher energy cost

All the VNFs composing the service chains of therequested network services must be allocated in the availablevirtualized resources including a given number of CPUshardware appliances RAM and storage space and networkbandwidth We assume that each core runs a single VM thatis devoted to the execution of a single VNF instance but oneVNF instance can scale from one to many VMs upon the ser-vice requirements of theNS Besides the transmission latencyalong the network topology between two VNFs allocatedin different microservers depends on the traffic load of theinvolved links and the size of the flow to be transmittedWith this information the placement algorithm assigns eachVNF instance to one or more VMs with the objective ofminimizing the energy consumption of the whole systemwhile matching the latency constraints for the NS

Finally the Infrastructure Manager functional elementVirtualized Infrastructure Manager (VIM) in ETSI-MANOterminology receives the outcome of the placement decisionand maps each VNF to the assigned resources

It is important to point out that a VNF chain includesradio-related and service-related instances The placementalgorithm introduced in this work considers a scenario withseveral tenants each of them requesting one or more NSswhich share the available resources Considering the afore-mentioned communication particularities of 5G Figure 3shows an example of a simple complete placement result ofa multitenant environment

This example considers a distributed edge cloud shared bytwo tenants Tenant 1 offers two network services the first onechaining a virtualized firewall (vFW) an intrusion detectionfunction (vID) a watermarking process (vWM) and finally acaching service (vCache) The second NS of Tenant 1 chainsthe vFW and a video analytics (VA) functionality for onlinesurveillance Tenant 2 offers a single network service thanchains the vFW a vCache service and a virtual transcodingunit (vTU) All NSs are associated with their correspondentlatency constraint

In the example both NSs belonging to the same tenantshare the radio VNF (small cell VNF or SCVNF) In this waythe common radio operations are performed in a coherentway between the different edge services of a tenant and thedata paths of the specific service instances per tenant are split

SCVNF SCVNF

vFW

VA

vFW

vWM

vCachevFW

vCache

vTU

HW

Acc

CPU

s

vID

NS2

NS1

NS1

NS1

NS1

NS1

NS1

NS1 N

S3NS3

NS1

NS2

Network topology

vFW vCache vTU

vFW VA

vFW vID vWM vCache ltlat1

ltlat2

ltlat3

Tena

nt 1

SCVNF

SCVNF

Tena

nt 2

NS1

NS2

NS3

vIDlowast

Figure 3 Multitenant placement example lowast indicates that ldquovIDrdquouses a hardware acceleration appliance

The SCVNF of each tenant will be shared by all theNSs of thistenant which means that the use of resources of the SCVNFmust then consider the sumof the aggregated flows of theNSsof the same tenant

The distinctive features of 5G communications have twoimportant implications over the placement algorithm Firstlyevery tenant allocates a unique SCVNF instance at the edgecloud in order to support the control plane and data planeoperations of all the SCs associated with that tenant Andsecondly since the SCVNFmanages the uplink and downlinktraffic between themobile users and the core network it leadsto circular forwarding paths when including edge serviceinstances In this paper we consider that the core networkfunctionalities of themobile network operator are outside theCE-RAN which means that the CE-RAN only deploys radioand edge service instances In order to keep the data pathbetween the mobile users and the core network functions alltheNSs need to start and end at the SCVNFwhichwill handlethe user plane traffic tofrom the core network

The example also shows how some VNFs needmore thanoneVM to execute the associated functionalities and howoneVNF uses a HWA tomeet the latency constraint of the server

Section 5 will show the result of more complex and com-plete placement experiments For the sake of generality thispaper does not focus on the specific modelling of the trafficpatterns the energy consumption schemes or the VNF net-work service performance Section 5 provides some referenceoutcomes on the potential benefits of the introduced VNFplacement algorithm in the considered network scenario but

Mobile Information Systems 5

the applicability of the solution to other scenarios shouldbe fed by experimental modelling data specific to the targetscenario

4 Robust Constraint-Based VNFPlacement Proposal

Once the 5G VNF placement problem under study is pre-sented in this section we propose a placement solution tothat problem using the RO approach To this end first weformulate the energy-aware VNF placement model in thenext subsection Then we provide our RO-based placementproposal in Section 42

41 ProblemModelling As stated before the VNF placementproblem addressed in this work focuses on the assignment ofthe VNFs composing each NS to the provided 5G networkinfrastructure Therefore it constitutes a discrete or com-binatorial optimization problem where the minimizationof energy consumption constitutes the optimality criterionSummary of Model Indexes Sets Parameters and DecisionVariables reports and summarizes the mathematical nota-tions used for the model indexes sets input parameters anddecision variables of VNF placement problem that is definedin the following

The Problem Model The energy-aware VNF placement opti-mization problem

min120587isinΠ

119875120587 (1)

st 119889120587119894 lt 119863max119894 forall119894 isin I 120587 isin Π (2)

119898120587119903119909 lt 119872119903119909 forall119903 isin R 119909 isin X 120587 isin Π (3)

119905119897120587119909 lt BW119909 forall119909 isin X 120587 isin Π (4)

119875120587 = sum119894isinI

sum119895isinJ119894

119875120587119894119895 (5)

119875120587119894119895

= sum119909isinX

sum119910isinY119909

119875cpu119909119910119894119895 119860119909119910

119894119895 + sum119894isinI

sum119895isinJ119894

119875switch119894119895119871 119894119895

+ sum119909isinX

119875sc119909119860119909119910

119894119895

(6)

119889120587119894 = sum119895isinJ119894

119889120587119894119895 (7)

119889120587119894119895

= sum119909isinX

sum119910isinY119909

119889119901119909119910119894119895 119860119909119910

119894119895

+ sum1199091isinX

sum1199101isinY1199091

sum1199092isinX

sum1199102isinY1199092

11988911989711990911199101 11990921199102119894119895 11986011990911199101119894119895 11986011990921199102119894119895+1

(8)

sum119894isinI

sum119895isinJ119894

119860119909119910119894119895 = 1 forall119909 isin X 119910isin Y119909 (9)

119898120587119903119909 = sum119894isinI

sum119895isinJ119894

sum119910isinY

119898119903119894119895119860119909119910

119894119895 (10)

119905119897120587119909 = sum119894isinI

sum119895isinJ119894

sum119910isinY

119879119894119860119909119910

119894119895 1198601199091015840119910

119894119895+1 (11)

We consider a set X of small cells that compose the CE-RAN and a set Y119909 of cores included in the microserver ofthe SC 119909 Additionally we also consider a set of I networkservices offered by the CE-RAN operator to the differenttenants and a set J119894 of VNFs that form the functionalchain of the NS 119894 Finally R represents the set of differentresources available in the microservers for the execution ofthe VNFs (storage RAM hardware accelerators etc) Henceour constraint-based expert systemmust determine in whichVM 119910 of a SC 119909 each VNF 119895 of a NS 119894 is located

Let Π denote the set of all possible power-aware andlatency- and resource-constrained VNF placement policieswhich decide the mapping of each VNF of available NSs toavailable VMs in the provided SCs We formulate our VNFplacement optimization problem aimed atminimizing powerconsumption with delay requirements as (1)

In reference to the energy part we define the total powerconsumption 119875120587 as the sum of the energy demand of all theallocated VNFs 119875120587119894119895 (5) Then for each VNF instance thepower consumption is expressed in (6) as the sum of thepower consumption of VNF 119895 of NS 119894 due to the CPU useof VM 119910 of SC 119909 the power consumption of the physicalswitch due to the forwarded traffic of the VNF and the powerconsumption of SC 119909 because of being switched on

The power consumption of the cores that run the VMsdepends of the aggregated user traffic served by the corre-spondent VNF and this relationship may be nonlinear Itmust be also understood that when a VNF instance scales toa higher number of cores the individual load of the involvedCPUs decreases leading to a lower power consumption percore Additionally the energy consumption of the networkingdevices (eg an Ethernet switch) depends on the aggregatedflow that traverses the device to forward data packets betweenSCs and the number of active ports Finally it is realized thatthe power wasted because the SC is active is constant notaffecting the optimization results Besides we express per-NSdelay in (7) as the sum of per-VNF delays and the per-VNFdelay in (8) as the sum of the processing delays introducedby the execution of the VNF instance plus the path delayintroduced by the traffic going from one VNF to the next onein the chain defined in the correspondent NS

On the one hand the processing delay in the CPU growswith the supported user traffic load On the other hand weconsider that the pathlink delay is composed of several linkdelays fromtoVMs tofrom the switches plus the delay in theswitches Hence depending on the adopted placement policy120587 each NS will follow a different path through the networkIn such a way if the VNFs belonging to a NS are placed in thesame SC the service chain traverses virtualinternal links andthe virtual switch of the selected SC whereas if the VNFs of

6 Mobile Information Systems

a NS are located in different SCs external links and the physi-cal switch is also used therefore generally 11988911989711990911199101 11990921199102119894119895 =11988911990911199101vs + 119889vs

1199091+ 119889vs1199091 119904 + 119889119904 + 119889119904vs1199092 + 119889vs

1199092+ 119889vs11990921199102 but for

a given VNF if the next VNF in its chain holds in the sameSC its link delay is only affected because of passing throughthe virtual switch Note that in case a VNF is the last VNF inthe chain the link delay is null with the sums referring to thelink delay for that 119895 being null

In order to guarantee that each VM holds only a singleVNF constraint (9) must be satisfied Nevertheless weconsider that each VNF of a NS can be instantiated as a VMthat is executed over several cores of the same SC Apart fromthat as previously detailed in Section 3 the first VNF and thelast VNF in a service chain are the same (the SCVNF) in orderto perform the specific functions that allow the split betweencontrol and data plane required in 5G communicationsTherefore it implies that 1198601199091199101198941 = 119860119909119869119894119894119895

Finally the model must also include the constraintsreferred to as the available resources In this sense (3) statesthat the consumption of the resource 119903 due to the placementof VNFs in the microserver of the SC 119909119898120587119903119909 must not exceedthe total amount of resource 119872119903119909 where 119898120587119903119909 is defined in(10) Similarly (4) constrains the traffic through the networklinks 119905119897120587119909 to the maximum capacity of the links BW119909 as thesum of the traffic flows of the network services that must beforwarded from a VNF in one SC to the next VNF in anotherSC

42 Robust Constraint-Based Solution As described inSection 2 many existing placement works assume perfectknowledge on input parameters pointing thus to the for-mulation of a deterministic optimization problem Unfortu-nately in the real world the estimated values of the inputparameters may differ due to biased data unrealistic assump-tions or numerical errors affecting the obtained optimalsolution and its performance This potential deviation ofthe nominal or expected input may lead to the violation ofthe problem constraints and therefore make the obtainedsolution suboptimal or even unfeasible

The uncertainty of modelling parameters has been tra-ditionally handled through stochastic programming andsensitivity analysis but robust optimization techniques haverecently arisen as a powerful tool to manage the impact ofuncertain input sets RO seeks an uncertainty-immunizedsolution with an acceptable performance under the real-ization of the uncertain inputs becoming a conservativeworst-case oriented methodology The main tools of RO areuncertainty sets and a robust counterpart problem [31] Theuncertainty of the input data is described later in this sectionwhile the robust counterpart problem is the deterministicmodel previously detailed in Section 41

Here we propose a robust constraint-based placementsolution that deals with service demand uncertainty Withthis RO approach we consider a tradeoff between theproblem optimization and the protection from deviationscaused by input parameters uncertainty This uncertainty inservice demand has an impact on several components on theformulation previously presented in Section 41 On the one

hand we consider that a VM which is running a specificVNF requires an expected use of CPU and thus a certainconstant power consumption due to CPU Neverthelessthe uncertainty in traffic demand may cause uncertainty inthe CPU requirementuse and consequently in its energyconsumption and processing delays On the other hand bothenergy consumption and delays in switches and links dependon that demand uncertainty as well

In order to obtain a proposal for the placement problemwith uncertain service demand we employ the Soyster [10]method used in the framework of RO That technique isalso known as the worst-case approach where the solutionsare achieved using the most extreme expected values of theuncertain variables Therefore that kind of resolution guar-antees feasible solutions for any value of the uncertain servicedemand

In reference to the modelling of the uncertain trafficdemand per NS 119894 we consider an expected traffic demand119864[119879119894] = 119879119894 plus a term due to uncertainty in the servicedemandUTmax119894 We assume that the randomvariableUTmax119894is symmetrically distributed within [minusΔ119879119894 +Δ119879119894] sdot 119879119894 andwith mean zero Furthermore the sum of deviations of theuncertain service demand should not exceed the worst-casemaximum value 119879max119894 fulfilling (12) In this way the valueof 119879max119894 controls the tradeoff between the robustness andthe impact on the optimization higher 119879max119894 leads to worseobjective function values but protects from more parameterdeviations

119880119879119894Δ119879119894

le 119879max119894 forall119894 (12)

5 Analysis of the Results

This section presents the results obtained with the simulationof the proposed robust constraint-based expert system for theplacement of VNF chains in an emerging 5G distributed edgecloud As previously introduced in Section 2 the mathemat-ical VNF placement problemmodel presented in Section 4 isa combinatorial or discrete optimization problem equivalentto the well-known knapsack problem This means that thedecision problem is NP-complete while the optimizationproblem is NP-hard [11] The proposed optimization modelhas been solved using constraint programming as a powerfulparadigm for solving combinatorial search problems [32]

We usedMinizinc [33] a free and open-source constraintmodelling language to model the optimization and con-straint satisfaction VNF placement problem in a high-levelsolver-independent wayThen Minizinc compiles this modelinto FlatZinc a solver input language that is understoodby a wide range of solvers In our case Gecode solver [34]provided the best performance in the search of the optimalsolution

The experiments that are described and discussed belowhave been conducted in a single computer with a 27GHzIntel Core i5 processor and 16GB RAM This simple con-figuration provides solutions within minutes for simpleexperiments with loose constraint satisfaction When theexperiments demand tighter latency or resource constraints

Mobile Information Systems 7

Virt

ual i

nfra

struc

ture

Corenetwork

Figure 4 Evaluation scenario

the solving time increases to hours depending on the specificconfiguration of the experiment

Next we first describe the analysis scenario and the con-figuration of the performed experiments to later discuss theresults of those experimentsThe objective of the experimentsis to demonstrate that the placement mechanism is able toallocate multitenant service chains to the available resourcesconsidering latency constraints For this reason the numeri-cal values employed in the next section are just experimentaland do not respond to real world measurements

51 Description of the Analysis Scenario Our evaluationscenario is depicted in Figure 4 We consider a 10-SC deploy-ment for example to cover a football stadium for periodicalflash eventsThe SCs are connected in a star topology throughan Ethernet switch Each SC is composed of 8 cores 16GBRAM 100GB of storage and 1HWA (eg GPU)

We also consider 2 NSs that can be offered at 3 serviceflavours (bronze silver and gold) that determine the latencyconstraint for eachNS and the nominal aggregate user bit rateserved as shown in Figure 5

Table 1 shows the experimental energy model and Table 2shows the modelling of the VNFs that compose the proposedNSs for the user traffic demand that will be employed in theevaluation experiments The user traffic values include thenominal bit rates of services NS1 and NS2 (130Mbps and90Mbps resp) as well as the values when a 20 protectionparameter is applied to the nominal values (156Mbps and108Mbps resp) in order to study the effect of robust optimi-zation techniques on the placement decision

As a final remark the placement behavior of VNFs canbe altered by the use of HWA The assignment of a HWA toa VNF implies that the service latency of the correspondentinstance is reduced 3 times at a cost of incrementing thepower consumption by 10 It is important to note that whenthe VNF instance needs more than 1 CPU for executingwithout the assistance of a HWA then if the placementalgorithm assigns it to a HWA the VNF will need a singlecore

52 Discussion of the Placement Results As stated before theevaluation scenario includes 3 tenants that hire a combination

8 Mobile Information Systems

VNF1 VNF2 VNF3 VNF5 VNF6

VNF1 VNF2 VNF3 VNF4

NS1

NS2

Service flavourBronze Silver Gold

NS1

NS2 Max Flow = 90MbpsMax Lat = 90ms Max Lat = 80ms

Max Flow = 130－＜ＪＭ

Max Lat = 150ＧＭ

Max Lat = 110ＧＭ

Max Flow = 90Mbps

Max Flow = 130－＜ＪＭ

Max Lat = 120ＧＭ

Max Flow = 90Mbps

Max Flow = 130－＜ＪＭ

Max Lat = 100ＧＭ

Figure 5 Description of NSs and service levels

Table 1 Energy models of the cores of the microservers and of the network switch related to the usage percentage

Usage 0 10 20 30 40 50 60 70 80 90 100119875router 0 1 2 5 7 9 11 13 14 15 15119875cpu 5 50 70 100 130 150 165 170 180 185 190

Table 2 VNF characterization for the service flavours of the evaluation scenario

VNF1 VNF2 VNF3 VNF4 VNF5 VNF6Number of cores per instance

90Mbps 1 2 1 1 1 1108Mbps 1 2 2 1 1 1130Mbps 1 2 2 1 1 1156Mbps 1 2 2 1 1 1

CPU usage90Mbps 30 20 90 50 50 20108Mbps 40 30 60 60 60 30130Mbps 40 30 60 60 60 30156Mbps 40 30 90 40 20 30

Service latency90Mbps 10ms 15ms 30ms 35ms 20ms 25ms108Mbps 10ms 15ms 35ms 40ms 25ms 30ms130Mbps 10ms 15ms 35ms 40ms 25ms 30ms156Mbps 12ms 16ms 40ms 50ms 30ms 35ms

of network servicesflavours to the SC operator In this wayin the following first we are going to evaluate the behavior ofthe placement algorithm for the three service levels and nextwe will show the effect of applying RO techniques to thoseresults

521 Bronze Flavour Scenario Results with 0 Robust Pro-tection Figure 6 shows the placement results of an scenariowhere all the network services operate at bronze flavourDifferent colours represent each tenant in order to dif-ferentiate the VNFs that belong to its NS and differenttones distinguish NSs of the same tenant As a result ofthe placement algorithm all the NSs are allocated in theavailable virtualized resources with energy consumption costof 3712W meeting all the latency constraints The figure

displays the assignment of VNFs to the SCs highlightingsome placement particularities As explained in Section 3the VNF chains of the same tenant share their SCVNF Inour example Tenant 1 and Tenant 2 both have two NSsthat are served with a single SCVNF Furthermore eachCPU executes a single VM but some VNF instances needtwo cores to manage the user demand In connection withthis behavior the VNF scalability feature is also visible inFigure 6 observing the placement of for example VNF3for the different NSs NS1 with a higher aggregated userdemand needs two CPUs to instantiate VNF3 but since NS2operates at a lower user demand the instantiation of VNF3needs just a single core Note as well that bronze serviceflavour does not require HW acceleration to meet the latencyconstraints of the NSs Finally it can also be observed that

Mobile Information Systems 9

Tenant 1 Tenant 2 Tenant 3

NS1 NS2 NS1 NS2 NS1

Service type Bronze Bronze Bronze Bronze Bronze

Max ow

Max latency

Deviation 0 0 0 0 0

Peak ow

VNF1VNF3

VNF1

VNF6

VNF1

SCVNF

VNF3VNF2

VNF1VNF3

VNF1VNF2

VNF2VNF3

VNF5

VNF5VNF2

VNF5

SCVNF

VNF1

VNF5

VNF3

VNF4

VNF5

VNF6

SCVNF

VNF1

VNF1VNF2

VNF3

VNF4

VNF1

960Mbps

960Mbps

780Mbps

780Mbps

780Mbps

260Mbps

VNF2

VNF6

CPU

s

Finallatency

Tenant 1NS1

NS2

Tenant 2NS1

NS2

Tenant 3 NS1

Total energy consumption

Switc

hH

A A

cc

132ＧＭ

102ＧＭ

132ＧＭ

102ＧＭ

130ＧＭ

3712７

130－＜ＪＭ 90－＜ＪＭ 130－＜ＪＭ 130－＜ＪＭ90－＜ＪＭ

130－＜ＪＭ 90－＜ＪＭ 130－＜ＪＭ 130－＜ＪＭ90－＜ＪＭ

150ＧＭ 110ＧＭ 150ＧＭ 110ＧＭ 150ＧＭ

Figure 6 Placement results of bronze flavour setting with 0 robust protection

the algorithm tries to group asmanyVNFs as possible in eachserver in order to reduce the data traffic across the switch andminimize its load and therefore its energy waste

522 Silver Flavour Scenario Results with 0 Robust Pro-tection The evaluation experiment analyzed in this sectionstudies the outcome of the placement algorithm when thelatency constraints become more strict with silver serviceflavour Due to the further limitation of maximum latencyvalues (20 stricter) the placement mechanism is compelledto use HW acceleration units Figure 7 shows that at thisservice level all the NSs must use HW acceleration in oneof the VNFs composing the service chain As a consequencethe resulting total latency of the NSs decreases to meet theservice level requirements but in return the global powerconsumption grows by 22 up to 4547W

Apart from that the placement results illustrate the samebehavior as the bronze flavoured example of Section 521regarding the allocation of VNFs in the available resourcesscale-inout features and multitenancy

The evaluation experiments of Sections 521 and 522operate with a nominal aggregated user traffic to perform theoptimization decision-making process However a perfectlyknown user traffic demand is seldom available Usuallythere is a certain level of uncertainty in the behavior of theuser demand We include robust optimization techniques to

allow the placement algorithm to insert a defined level ofuncertainty and protect the system against the undesirableeffects of demand peaksThe evaluation examples in Sections523 and 524 include a robust protection parameter andanalyze its effect on the global power consumption

523 Silver Flavour Scenario Results with 20 Robust Protec-tion This section studies the placement outcome of the eval-uation scenario described in the previous section introducinga 20 robust protection parameter This modification on thescenario configuration implies an increase of the 20 of theaggregated user traffic as an input of the decision-makingprocess in order to be prepared for potential demand peaksduring the service operation

Figure 8 exhibits that VNF3 of NS2 must scale as a con-sequence of the traffic growth and now needs two CPUs foreach instance The traffic increase also has an impact on theprocessing latency of the VNFs causing an adjustment in theuse of HW accelerators As a consequence some VNFs thatrequired 2CPUs for their operation in the previous evalu-ation scenario now only need a single core in combinationwith the HWA Therefore in some cases the higher powerconsumption that is implicit to the use of HWAs may bebalanced with a decrease on the number of CPUs required forthe VNF Besides the higher aggregated user traffic involvesa more intense use of the network switch These factors have

10 Mobile Information Systems

SCVNF

VNF6

VNF1VNF1

VNF2

VNF3

VNF3

VNF3

VNF5

VNF2

VNF1

VNF2

SCVNF

VNF1

VNF2

VNF4

VNF5

VNF6

SCVNF

VNF2

VNF3

VNF4

VNF5

VNF6

VNF3

VNF1

Finallatency

Tenant 1NS1

NS2

Tenant 2NS1

NS2

Tenant 3 NS1

Total energy consumption

VNF6VNF1VNF1VNF2VNF2

Tenant 1 Tenant 2 Tenant 3

NS1 NS2 NS1 NS2 NS1

Service type Silver Silver Silver Silver Silver

Max ow

Max latency

Deviation 0 0 0 0 0

Peak ow

960Mbps

960Mbps

780Mbps

780Mbps

260Mbps

780MbpsSw

itch

100ＧＭ

118ＧＭ

116ＧＭ

4547７

130－＜ＪＭ 90－＜ＪＭ 130－＜ＪＭ 130－＜ＪＭ90－＜ＪＭ

130－＜ＪＭ 90－＜ＪＭ 130－＜ＪＭ 130－＜ＪＭ90－＜ＪＭ

120ＧＭ 120ＧＭ 120ＧＭ

CPU

sH

A A

cc

88ＧＭ

88ＧＭ

90ＧＭ 90ＧＭ

Figure 7 Placement results of silver flavour setting with 0 robust protection

an impact on the power consumption of the componentsof the network resulting in a global energy consumption of5047W which implies an increment of 11 compared to theunprotected placement

524 Gold Flavour Scenario Results with 20 Robust Protec-tion Finally the last evaluation example sets even stricterlatency constraints to theNSs to be placed and includes a 20robust protection parameter Figure 9 depicts the results ofthe placement algorithm with this configuration Again thereduction of latency values forces the increment of the use ofHW acceleration In this case the main consequence is thespreading of the VNFs across the SCs to use the single HWAavailable in each cellThe use of HWAs the increase of powerconsumption due to higher user aggregated traffic and thehigher network traffic traversing the switch cause the globalenergy consumption to grow up to 5569W

As a final remark Figure 10 shows the comparison ofthe energy consumption and use of resources (cores activeCESCs and HWA) related to the robust protection level ina wider collection of scenarios In particular the charts ofFigure 10 compare the results of the placement algorithm forthe three service levels described in the previous sections(bronze silver and gold) combined with three robust protec-tion levels (0 or no protection 10 and 20) Obviouslythe main trend is that both power and resource consumptionincrease with higher values of protection but the associatedcost can be assumed in favor of reducing the impact of

unexpected user demand peaks However there are somecases in which the protection parameter does not have soadverse implications For example it can be observed howthe number of used cores in the silver service level with a10 protection level is lower than in the case of no robustprotection The reason of this behavior is how the algorithmuses HWA appliances in this case The increase of the usertraffic demand in the protected services implies the scaling upof someVNFs that nowmust usemore cores for the operationof theNSsThen the placement algorithm decides to useHWaccelerators for those specific VNFs compensating this waythe higher energy cost of HWAs with a lower use of CPUcores With all these considerations that can be extractedfrom a deeper analysis of robust placement results robustoptimization can be applied to decision-making problemsnot ending just with the optimization problem

The proposed steps in this paper can be used in anyplacement decision-making context and its goal is to providea tool that helps system administrator to evaluate the possibleimpact of any decision on the performance of the deploymentunderneath

6 Conclusions

This paper introduces a constraint-based expert system tosupport the decision-making process of VNF chain place-ment in distributed edge cloud networks The placement

Mobile Information Systems 11

SCVNF

VNF5

VNF6VNF5VNF2

VNF3

VNF5

VNF1

VNF2

VNF3

VNF1

VNF2

VNF3

VNF6

SCVNF

VNF1

VNF1

VNF3

VNF4

SCVNF

VNF1

VNF2

VNF3

VNF4

840Mbps

840Mbps

936Mbps

624Mbps

312Mbps

312Mbps

936Mbps

VNF6

Tenant 1NS1

NS2

Tenant 2NS1

NS2

Tenant 3 NS1

Total energy consumption

VNF5VNF6VNF5VNF1VNF4VNF4

VNF2

Finallatency

Tenant 1 Tenant 2 Tenant 3

NS1 NS2 NS1 NS2 NS1

Service type Silver Silver Silver Silver Silver

Max ow

Max latency

Deviation 20 20 20 20 20

Peak ow

116ＧＭ

110ＧＭ

118ＧＭ

5047７

130－＜ＪＭ 90－＜ＪＭ 130－＜ＪＭ 130－＜ＪＭ90－＜ＪＭ

156－＜ＪＭ 108－＜ＪＭ 156－＜ＪＭ 108－＜ＪＭ 156－＜ＪＭ

150ＧＭ 110ＧＭ 150ＧＭ 110ＧＭ 150ＧＭ

79ＧＭ

79ＧＭ

CPU

sH

A A

ccSw

itch

Figure 8 Placement results of silver flavour setting with 20 robust protection

algorithm applies robust optimization techniques to theminimization of the global power consumption of the systemsubject to the bit rate and latency constraints of the networkservices The power consumption of the components of thesystem depends on the usage percentage which is a con-sequence of the aggregated user traffic demand The modelassumes that the relationships between the user trafficdemand the power consumption and the use of the resourcesmay be nonlinear Besides the placement algorithmconsidersthe 5G communication particularities of both control anddata plane and allows multitenancy Finally the algorithmalso includes VNF scaling features and the use of hardwareacceleration

The results show that the placement algorithm succeeds inallocating theVNFs that compose theNSs of different tenantsin the available resourcesmeeting the latency constraints andminimizing the energy consumption of thewhole systemThepresented evaluation examples demonstrate the features ofthe constraint-based expert system and illustrate the impactof the selected service flavour and robust protection on theglobal power consumption and therefore the exploitationcost of the 5G distributed edge cloud

Further researchwill extend the characterization of VNFsto include variable output bit rate This feature will modelthe behavior of for example transcoding functions A future

version of the algorithm will also incorporate the existenceof a virtual switch to connect the VMs running in the samemicroserver Another enhancement of the model will bethe combination of heterogeneous SCs with different con-figurations and resource consumption models Finally theobtained placement results will be used for a technoeconomicstudy that will analyze the relationship between the serveraggregated user traffic cost of the required infrastructure andthe pricing of the services This study will also consider theintegration of the placement decision problem with the radioside

Summary of Model Indexes SetsParameters and Decision Variables

Indexes

120587 Index for VNF placement policy119909 Index for SC microserver119910 Index for core number in each microserver119894 Index for NS119895 Index for VNF in a network service chain119903 Index for physical resources in

SC microserver

12 Mobile Information Systems

VNF6VNF5VNF1VNF6VNF2SCVNF

VNF2

VNF3

VNF5

VNF3

VNF1

VNF2

VNF3

VNF5

SCVNF

VNF1

VNF1

VNF3

VNF4

SCVNF

VNF2

VNF3

VNF4

VNF1

Tenant 1NS1

NS2

Tenant 2NS1

NS2

Tenant 3 NS1

Total energy consumption

VNF6VNF5VNF1VNF6VNF2VNF5VNF6VNF4VNF4

VNF2

840Mbps

840Mbps

936Mbps

936Mbps

312Mbps

312Mbps

312Mbps

312Mbps

312Mbps

Finallatency

Tenant 1 Tenant 2 Tenant 3

NS1 NS2 NS1 NS2 NS1

Service type Gold Gold Gold Gold Gold

Max ow

Max latency

Deviation 20 20 20 20 20

Peak ow

5569７

130－＜ＪＭ 90－＜ＪＭ 130－＜ＪＭ 130－＜ＪＭ90－＜ＪＭ

156－＜ＪＭ 108－＜ＪＭ 156－＜ＪＭ 108－＜ＪＭ 156－＜ＪＭ

100ＧＭ 100ＧＭ 100ＧＭ

79ＧＭ

95ＧＭ

89ＧＭ

79ＧＭ

86ＧＭ

CPU

sH

A A

ccSw

itch

80ＧＭ 80ＧＭ

Figure 9 Placement results of gold flavour setting with 20 robust protection

34

32

30

28

260

10

20Robust protection ()

Num

ber o

f use

dco

res

10864200

10

20Robust protection ()

HW

AN

umbe

r of u

sed

0

10

20Robust protection ()

987654N

umbe

r of a

ctiv

eCE

SCs

6000

5500

5000

4500

4000

35003000

010

20

Ener

gy co

nsum

ptio

n(W

)

Robust protection ()

Gold

Silver

Bronze

Gold

Silver

Bronze

Gold

Silver

Bronze

Gold

Silver

Bronze

Figure 10 Comparison of energy and resource consumption for different levels of robust protection

Mobile Information Systems 13

Sets

Π Set of possible VNF placement policies119883 Set of SC microservers119884 Set of cores composing a SC microserver119868 Set of offered NSs119869119894 Set of VNFs in NS 119894119877 Set of resources available in

each SC microserver

Input Parameters

119879119894 Contracted maximum aggregated usertraffic for network service 119894

119863max119894 Maximum latency allowed for NS 119894119872119903119909 Amount of resource 119903 available in

SC microserver 119909119875sc119909 Power consumption of SC 119909 because of

being switched on119875cpu119909119910119894119895 Power consumption of VM 119910 of SC 119909 due

to CPU use of VNF 119895 of NS 119894119875switch119894119895 Power consumption in the physical switch

due to CPU use of VNF 119895 of NS 119894

Decision Variables

119875120587 Total power consumption ofplacement policy 120587

119875120587119894119895 Power consumption of VNF 119895 of NS 119894 inplacement policy 120587

119898120587119903119909 Consumption of resource 119903 in SCmicroserver 119909 in placement policy 120587

119905119897120587119909 Traffic of the link that connects SCmicroserver 119909 with the switchinplacement policy 120587

119860119909119910119894119895 Binary variable that expressesthe assignment of VNF 119895 of NS 119894 toVM 119910 in SC 119909

119871 119894119895 Binary variable to express the use of thenetwork switch to forward the flow ofVNF 119895 of NS 119894 to the next VNF

119889119894 Delay of NS 119894119889119894119895 Delay of VNF 119895 of NS 119894119889119901119909119910119894119895 Processing delay in the CPU of VNF 119895 of

NS 119894 in VM 119910 of SC 1199091198891198971199091119910111990921199102119894119895 Path delay for NS 119894 between VNF 119895

(located in VM 1199101 of SC 1199091) and VNF119895 + 1 (located in VM 1199102 of SC 1199092)

119889vs119909119910 Link delay between the virtual switch inSC 119909 and the VM 119910 in SC 119909

119889vs119909 119904 Link delay between the virtual switch inSC 119909 and the physical switch

119889vs119909 Delay in the virtual switch of SC 119909

119889119904 Delay in the physical switch

Conflicts of Interest

The authors declare that they have no conflicts of interest

Acknowledgments

The research leading to these results has been supportedby the EU funded H2020 5G-PPP Project SESAME (GrantAgreement 671596) and the Spanish MINECO Project5GRANVIR (TEC2016-80090-C2-2-R)

References

[1] ITU-R M2083 IMT Vision-Framework and overall objectivesof the future development of IMT for 2020 and beyond 2015

[2] P Demestichas A Georgakopoulos D Karvounas et al ldquo5G onthe Horizon key challenges for the radio-access networkrdquo IEEEVehicular Technology Magazine vol 8 no 3 pp 47ndash53 2013

[3] I F Akyildiz S Nie S-C Lin and M Chandrasekaran ldquo5Groadmap 10 key enabling technologiesrdquo Computer Networksvol 106 pp 17ndash48 2016

[4] ETSI Mobile-edge computing Introductory technical whitepaper 2014

[5] B Han V Gopalakrishnan L Ji and S Lee ldquoNetwork functionvirtualization challenges and opportunities for innovationsrdquoIEEE Communications Magazine vol 53 no 2 pp 90ndash97 2015

[6] R Mijumbi J Serrat J Gorricho N Bouten F De Turckand R Boutaba ldquoNetwork function virtualization state-of-the-art and research challengesrdquo IEEE Communications Surveys ampTutorials vol 18 no 1 pp 236ndash262 2016

[7] Ericsson White Paper Cloud-RANndash the benefits of virtualiza-tion centralization and coordination 2015

[8] B Blanco J O Fajardo I Giannoulakis et al ldquoTechnologypillars in the architecture of future 5G mobile networks NFVMEC and SDNrdquo Computer Standards and Interfaces vol 54 pp216ndash228 2017

[9] ETSI Network Functions Virtualization (NFV) Managementand Orchestration 2014

[10] A Ben-Tal L E Ghaoui and A Nemirovski Robust Optimiza-tion Aharon Ben-Tal Laurent El Ghaoui Arkadi Nemirovski-Google Books Princeton University Press Princeton NewJersey NY USA 2009

[11] J Gil Herrera and J F Botero ldquoResource Allocation in NFVA Comprehensive Surveyrdquo IEEE Transactions on Network andService Management vol 13 no 3 pp 518ndash532 2016

[12] B Addis D Belabed M Bouet and S Secci ldquoVirtual networkfunctions placement and routing optimizationrdquo in Proceedingsof the 4th IEEE International Conference on Cloud NetworkingCloudNet rsquo15 pp 171ndash177 October 2015

[13] H Moens and F De Turck ldquoVNF-P a model for efficientplacement of virtualized network functionsrdquo in Proceedingsof the 10th International Conference on Network and ServiceManagement (CNSM rsquo14) pp 418ndash423 Rio de Janeiro BrazilNovember 2014

[14] A Gupta M F Habib P Chowdhury M Tornatore and BMukherjee ldquoOn service chaining using Virtual Network Func-tions in Network-enabled Cloud systemsrdquo in Proceedings of the9th IEEE International Conference on Advanced Networks andTelecommuncations Systems ANTS rsquo15 pp 1ndash3 December 2015

[15] M Bouet J Leguay T Combe and V Conan ldquoCost-basedplacement of vDPI functions in NFV infrastructuresrdquo Interna-tional Journal of Network Management vol 25 no 6 pp 490ndash506 2015

[16] M C Luizelli L R Bays L S Buriol M P Barcellos andL P Gaspary ldquoPiecing together the NFV provisioning puzzle

14 Mobile Information Systems

efficient placement and chaining of virtual network functionsrdquoin Proceedings of the 14th International Symposium on IntegratedNetwork Management IM rsquo15 pp 98ndash106 IEEE TorontoCanada May 2015

[17] S Kim Y Han and S Park ldquoAn energy-Aware service functionchaining and reconfiguration algorithm in NFVrdquo in Proceedingsof the 1st International Workshops on Foundations and Applica-tions of Self-Systems FAS-W rsquo16 pp 54ndash59 September 2016

[18] V Eramo A Tosti and E Miucci ldquoServer resource dimen-sioning and routing of service function chain in NFV networkarchitecturesrdquo Journal of Electrical and Computer Engineeringvol 2016 Article ID 7139852 12 pages 2016

[19] K Hida and S-I Kuribayashi ldquoVirtual routing function allo-cation method for minimizing total network power consump-tionrdquo World Academy of Science Engineering and TechnologyInternational Journal of Electrical Computer Energetic Elec-tronic and Communication Engineering vol 10 pp 997ndash10022016

[20] C Pham N H Tran S Ren W Saad and C S Hong ldquoTraffic-aware and energy-efficient vnf placement for service chainingjoint sampling and matching approachrdquo IEEE Transactions onServices Computing 2017

[21] A Marotta and A Kassler ldquoA power efficient and robust virtualnetwork functions placement problemrdquo in Proceedings of the28th International Teletraffic Congress ITC rsquo16 pp 331ndash339September 2016

[22] C Pham H D Tran S I Moon K Thar and C S Hong ldquoAgeneral and practical consolidation framework in CloudNFVrdquoin Proceedings of the 2015 International Conference on Infor-mation Networking ICOIN rsquo15 pp 295ndash300 IEEE CambodiaCambodia January 2015

[23] V Eramo M Ammar and F G Lavacca ldquoMigration energyaware reconfigurations of virtual network function instances inNFV architecturesrdquo IEEE Access vol 5 pp 4927ndash4938 2017

[24] N E Khoury S Ayoubi and C Assi ldquoEnergy-aware placementand scheduling of network trafficflowswith deadlines on virtualnetwork functionsrdquo in Proceedings of the 5th IEEE InternationalConference on CloudNetworking CloudNet rsquo16 pp 89ndash94 IEEEPisa Italy October 2016

[25] V Eramo E Miucci M Ammar and F G Lavacca ldquoAnapproach for service function chain routing and virtual func-tion network instance migration in network function virtual-ization architecturesrdquo IEEEACM Transactions on Networking2017

[26] V Jumba S Parsaeefard M Derakhshani and T Le-NgocldquoEnergy-efficient robust resource provisioning in virtualizedwireless networksrdquo in Proceedings of the IEEE InternationalConference on Ubiquitous Wireless Broadband ICUWB rsquo15 pp1ndash5 IEEE Montreal Canada October 2015

[27] S Coniglio A Koster and M Tieves ldquoData uncertainty invirtual network embedding robust optimization and protectionlevelsrdquo Journal of Network and SystemsManagement vol 24 no3 pp 681ndash710 2016

[28] I Takouna K Sachs and C Meinel ldquoMultiperiod robustoptimization for proactive resource provisioning in virtualizeddata centersrdquo Journal of Supercomputing vol 70 no 3 pp 1514ndash1536 2014

[29] G Chochlidakis and V Friderikos ldquoRobust virtual networkembedding for mobile networksrdquo in Proceedings of the 26thIEEE Annual International Symposium on Personal Indoor andMobile Radio Communications PIMRC rsquo15 pp 1867ndash1871 IEEEKowloon Hong Kong September 2015

[30] S Coniglio A M Koster and M Tieves ldquoVirtual networkembedding under uncertainty exact and heuristic approachesrdquoin Proceedings of the 11th International Conference on the Designof Reliable Communication Networks DRCN rsquo15 pp 1ndash8 IEEEKansas Kan USA March 2015

[31] D Bertsimas and M Sim ldquoThe price of robustnessrdquoOperationsResearch vol 52 no 1 pp 35ndash53 2004

[32] F Rossi P Van Beek and T Walsh Handbook of ConstraintProgramming Elsevier Amsterdam Netherland 2006

[33] httpwwwminizincorg[34] C Schulte G Tack and M Z Lagerkvist Modeling and pro-

gramming with gecode 2010 Schulte Christian and Tack Guidoand Lagerkvist Mikael

Mobile Information Systems 5

the applicability of the solution to other scenarios shouldbe fed by experimental modelling data specific to the targetscenario

4 Robust Constraint-Based VNFPlacement Proposal

Once the 5G VNF placement problem under study is pre-sented in this section we propose a placement solution tothat problem using the RO approach To this end first weformulate the energy-aware VNF placement model in thenext subsection Then we provide our RO-based placementproposal in Section 42

41 ProblemModelling As stated before the VNF placementproblem addressed in this work focuses on the assignment ofthe VNFs composing each NS to the provided 5G networkinfrastructure Therefore it constitutes a discrete or com-binatorial optimization problem where the minimizationof energy consumption constitutes the optimality criterionSummary of Model Indexes Sets Parameters and DecisionVariables reports and summarizes the mathematical nota-tions used for the model indexes sets input parameters anddecision variables of VNF placement problem that is definedin the following

The Problem Model The energy-aware VNF placement opti-mization problem

min120587isinΠ

119875120587 (1)

st 119889120587119894 lt 119863max119894 forall119894 isin I 120587 isin Π (2)

119898120587119903119909 lt 119872119903119909 forall119903 isin R 119909 isin X 120587 isin Π (3)

119905119897120587119909 lt BW119909 forall119909 isin X 120587 isin Π (4)

119875120587 = sum119894isinI

sum119895isinJ119894

119875120587119894119895 (5)

119875120587119894119895

= sum119909isinX

sum119910isinY119909

119875cpu119909119910119894119895 119860119909119910

119894119895 + sum119894isinI

sum119895isinJ119894

119875switch119894119895119871 119894119895

+ sum119909isinX

119875sc119909119860119909119910

119894119895

(6)

119889120587119894 = sum119895isinJ119894

119889120587119894119895 (7)

119889120587119894119895

= sum119909isinX

sum119910isinY119909

119889119901119909119910119894119895 119860119909119910

119894119895

+ sum1199091isinX

sum1199101isinY1199091

sum1199092isinX

sum1199102isinY1199092

11988911989711990911199101 11990921199102119894119895 11986011990911199101119894119895 11986011990921199102119894119895+1

(8)

sum119894isinI

sum119895isinJ119894

119860119909119910119894119895 = 1 forall119909 isin X 119910isin Y119909 (9)

119898120587119903119909 = sum119894isinI

sum119895isinJ119894

sum119910isinY

119898119903119894119895119860119909119910

119894119895 (10)

119905119897120587119909 = sum119894isinI

sum119895isinJ119894

sum119910isinY

119879119894119860119909119910

119894119895 1198601199091015840119910

119894119895+1 (11)

We consider a set X of small cells that compose the CE-RAN and a set Y119909 of cores included in the microserver ofthe SC 119909 Additionally we also consider a set of I networkservices offered by the CE-RAN operator to the differenttenants and a set J119894 of VNFs that form the functionalchain of the NS 119894 Finally R represents the set of differentresources available in the microservers for the execution ofthe VNFs (storage RAM hardware accelerators etc) Henceour constraint-based expert systemmust determine in whichVM 119910 of a SC 119909 each VNF 119895 of a NS 119894 is located

Let Π denote the set of all possible power-aware andlatency- and resource-constrained VNF placement policieswhich decide the mapping of each VNF of available NSs toavailable VMs in the provided SCs We formulate our VNFplacement optimization problem aimed atminimizing powerconsumption with delay requirements as (1)

In reference to the energy part we define the total powerconsumption 119875120587 as the sum of the energy demand of all theallocated VNFs 119875120587119894119895 (5) Then for each VNF instance thepower consumption is expressed in (6) as the sum of thepower consumption of VNF 119895 of NS 119894 due to the CPU useof VM 119910 of SC 119909 the power consumption of the physicalswitch due to the forwarded traffic of the VNF and the powerconsumption of SC 119909 because of being switched on

The power consumption of the cores that run the VMsdepends of the aggregated user traffic served by the corre-spondent VNF and this relationship may be nonlinear Itmust be also understood that when a VNF instance scales toa higher number of cores the individual load of the involvedCPUs decreases leading to a lower power consumption percore Additionally the energy consumption of the networkingdevices (eg an Ethernet switch) depends on the aggregatedflow that traverses the device to forward data packets betweenSCs and the number of active ports Finally it is realized thatthe power wasted because the SC is active is constant notaffecting the optimization results Besides we express per-NSdelay in (7) as the sum of per-VNF delays and the per-VNFdelay in (8) as the sum of the processing delays introducedby the execution of the VNF instance plus the path delayintroduced by the traffic going from one VNF to the next onein the chain defined in the correspondent NS

On the one hand the processing delay in the CPU growswith the supported user traffic load On the other hand weconsider that the pathlink delay is composed of several linkdelays fromtoVMs tofrom the switches plus the delay in theswitches Hence depending on the adopted placement policy120587 each NS will follow a different path through the networkIn such a way if the VNFs belonging to a NS are placed in thesame SC the service chain traverses virtualinternal links andthe virtual switch of the selected SC whereas if the VNFs of

6 Mobile Information Systems

a NS are located in different SCs external links and the physi-cal switch is also used therefore generally 11988911989711990911199101 11990921199102119894119895 =11988911990911199101vs + 119889vs

1199091+ 119889vs1199091 119904 + 119889119904 + 119889119904vs1199092 + 119889vs

1199092+ 119889vs11990921199102 but for

a given VNF if the next VNF in its chain holds in the sameSC its link delay is only affected because of passing throughthe virtual switch Note that in case a VNF is the last VNF inthe chain the link delay is null with the sums referring to thelink delay for that 119895 being null

In order to guarantee that each VM holds only a singleVNF constraint (9) must be satisfied Nevertheless weconsider that each VNF of a NS can be instantiated as a VMthat is executed over several cores of the same SC Apart fromthat as previously detailed in Section 3 the first VNF and thelast VNF in a service chain are the same (the SCVNF) in orderto perform the specific functions that allow the split betweencontrol and data plane required in 5G communicationsTherefore it implies that 1198601199091199101198941 = 119860119909119869119894119894119895

Finally the model must also include the constraintsreferred to as the available resources In this sense (3) statesthat the consumption of the resource 119903 due to the placementof VNFs in the microserver of the SC 119909119898120587119903119909 must not exceedthe total amount of resource 119872119903119909 where 119898120587119903119909 is defined in(10) Similarly (4) constrains the traffic through the networklinks 119905119897120587119909 to the maximum capacity of the links BW119909 as thesum of the traffic flows of the network services that must beforwarded from a VNF in one SC to the next VNF in anotherSC

42 Robust Constraint-Based Solution As described inSection 2 many existing placement works assume perfectknowledge on input parameters pointing thus to the for-mulation of a deterministic optimization problem Unfortu-nately in the real world the estimated values of the inputparameters may differ due to biased data unrealistic assump-tions or numerical errors affecting the obtained optimalsolution and its performance This potential deviation ofthe nominal or expected input may lead to the violation ofthe problem constraints and therefore make the obtainedsolution suboptimal or even unfeasible

The uncertainty of modelling parameters has been tra-ditionally handled through stochastic programming andsensitivity analysis but robust optimization techniques haverecently arisen as a powerful tool to manage the impact ofuncertain input sets RO seeks an uncertainty-immunizedsolution with an acceptable performance under the real-ization of the uncertain inputs becoming a conservativeworst-case oriented methodology The main tools of RO areuncertainty sets and a robust counterpart problem [31] Theuncertainty of the input data is described later in this sectionwhile the robust counterpart problem is the deterministicmodel previously detailed in Section 41

Here we propose a robust constraint-based placementsolution that deals with service demand uncertainty Withthis RO approach we consider a tradeoff between theproblem optimization and the protection from deviationscaused by input parameters uncertainty This uncertainty inservice demand has an impact on several components on theformulation previously presented in Section 41 On the one

hand we consider that a VM which is running a specificVNF requires an expected use of CPU and thus a certainconstant power consumption due to CPU Neverthelessthe uncertainty in traffic demand may cause uncertainty inthe CPU requirementuse and consequently in its energyconsumption and processing delays On the other hand bothenergy consumption and delays in switches and links dependon that demand uncertainty as well

In order to obtain a proposal for the placement problemwith uncertain service demand we employ the Soyster [10]method used in the framework of RO That technique isalso known as the worst-case approach where the solutionsare achieved using the most extreme expected values of theuncertain variables Therefore that kind of resolution guar-antees feasible solutions for any value of the uncertain servicedemand

In reference to the modelling of the uncertain trafficdemand per NS 119894 we consider an expected traffic demand119864[119879119894] = 119879119894 plus a term due to uncertainty in the servicedemandUTmax119894 We assume that the randomvariableUTmax119894is symmetrically distributed within [minusΔ119879119894 +Δ119879119894] sdot 119879119894 andwith mean zero Furthermore the sum of deviations of theuncertain service demand should not exceed the worst-casemaximum value 119879max119894 fulfilling (12) In this way the valueof 119879max119894 controls the tradeoff between the robustness andthe impact on the optimization higher 119879max119894 leads to worseobjective function values but protects from more parameterdeviations

119880119879119894Δ119879119894

le 119879max119894 forall119894 (12)

5 Analysis of the Results

This section presents the results obtained with the simulationof the proposed robust constraint-based expert system for theplacement of VNF chains in an emerging 5G distributed edgecloud As previously introduced in Section 2 the mathemat-ical VNF placement problemmodel presented in Section 4 isa combinatorial or discrete optimization problem equivalentto the well-known knapsack problem This means that thedecision problem is NP-complete while the optimizationproblem is NP-hard [11] The proposed optimization modelhas been solved using constraint programming as a powerfulparadigm for solving combinatorial search problems [32]

We usedMinizinc [33] a free and open-source constraintmodelling language to model the optimization and con-straint satisfaction VNF placement problem in a high-levelsolver-independent wayThen Minizinc compiles this modelinto FlatZinc a solver input language that is understoodby a wide range of solvers In our case Gecode solver [34]provided the best performance in the search of the optimalsolution

The experiments that are described and discussed belowhave been conducted in a single computer with a 27GHzIntel Core i5 processor and 16GB RAM This simple con-figuration provides solutions within minutes for simpleexperiments with loose constraint satisfaction When theexperiments demand tighter latency or resource constraints

Mobile Information Systems 7

Virt

ual i

nfra

struc

ture

Corenetwork

Figure 4 Evaluation scenario

the solving time increases to hours depending on the specificconfiguration of the experiment

Next we first describe the analysis scenario and the con-figuration of the performed experiments to later discuss theresults of those experimentsThe objective of the experimentsis to demonstrate that the placement mechanism is able toallocate multitenant service chains to the available resourcesconsidering latency constraints For this reason the numeri-cal values employed in the next section are just experimentaland do not respond to real world measurements

51 Description of the Analysis Scenario Our evaluationscenario is depicted in Figure 4 We consider a 10-SC deploy-ment for example to cover a football stadium for periodicalflash eventsThe SCs are connected in a star topology throughan Ethernet switch Each SC is composed of 8 cores 16GBRAM 100GB of storage and 1HWA (eg GPU)

We also consider 2 NSs that can be offered at 3 serviceflavours (bronze silver and gold) that determine the latencyconstraint for eachNS and the nominal aggregate user bit rateserved as shown in Figure 5

Table 1 shows the experimental energy model and Table 2shows the modelling of the VNFs that compose the proposedNSs for the user traffic demand that will be employed in theevaluation experiments The user traffic values include thenominal bit rates of services NS1 and NS2 (130Mbps and90Mbps resp) as well as the values when a 20 protectionparameter is applied to the nominal values (156Mbps and108Mbps resp) in order to study the effect of robust optimi-zation techniques on the placement decision

As a final remark the placement behavior of VNFs canbe altered by the use of HWA The assignment of a HWA toa VNF implies that the service latency of the correspondentinstance is reduced 3 times at a cost of incrementing thepower consumption by 10 It is important to note that whenthe VNF instance needs more than 1 CPU for executingwithout the assistance of a HWA then if the placementalgorithm assigns it to a HWA the VNF will need a singlecore

52 Discussion of the Placement Results As stated before theevaluation scenario includes 3 tenants that hire a combination

8 Mobile Information Systems

VNF1 VNF2 VNF3 VNF5 VNF6

VNF1 VNF2 VNF3 VNF4

NS1

NS2

Service flavourBronze Silver Gold

NS1

NS2 Max Flow = 90MbpsMax Lat = 90ms Max Lat = 80ms

Max Flow = 130－＜ＪＭ

Max Lat = 150ＧＭ

Max Lat = 110ＧＭ

Max Flow = 90Mbps

Max Flow = 130－＜ＪＭ

Max Lat = 120ＧＭ

Max Flow = 90Mbps

Max Flow = 130－＜ＪＭ

Max Lat = 100ＧＭ

Figure 5 Description of NSs and service levels

Table 1 Energy models of the cores of the microservers and of the network switch related to the usage percentage

Usage 0 10 20 30 40 50 60 70 80 90 100119875router 0 1 2 5 7 9 11 13 14 15 15119875cpu 5 50 70 100 130 150 165 170 180 185 190

Table 2 VNF characterization for the service flavours of the evaluation scenario

VNF1 VNF2 VNF3 VNF4 VNF5 VNF6Number of cores per instance

90Mbps 1 2 1 1 1 1108Mbps 1 2 2 1 1 1130Mbps 1 2 2 1 1 1156Mbps 1 2 2 1 1 1

CPU usage90Mbps 30 20 90 50 50 20108Mbps 40 30 60 60 60 30130Mbps 40 30 60 60 60 30156Mbps 40 30 90 40 20 30

Service latency90Mbps 10ms 15ms 30ms 35ms 20ms 25ms108Mbps 10ms 15ms 35ms 40ms 25ms 30ms130Mbps 10ms 15ms 35ms 40ms 25ms 30ms156Mbps 12ms 16ms 40ms 50ms 30ms 35ms

of network servicesflavours to the SC operator In this wayin the following first we are going to evaluate the behavior ofthe placement algorithm for the three service levels and nextwe will show the effect of applying RO techniques to thoseresults

521 Bronze Flavour Scenario Results with 0 Robust Pro-tection Figure 6 shows the placement results of an scenariowhere all the network services operate at bronze flavourDifferent colours represent each tenant in order to dif-ferentiate the VNFs that belong to its NS and differenttones distinguish NSs of the same tenant As a result ofthe placement algorithm all the NSs are allocated in theavailable virtualized resources with energy consumption costof 3712W meeting all the latency constraints The figure

displays the assignment of VNFs to the SCs highlightingsome placement particularities As explained in Section 3the VNF chains of the same tenant share their SCVNF Inour example Tenant 1 and Tenant 2 both have two NSsthat are served with a single SCVNF Furthermore eachCPU executes a single VM but some VNF instances needtwo cores to manage the user demand In connection withthis behavior the VNF scalability feature is also visible inFigure 6 observing the placement of for example VNF3for the different NSs NS1 with a higher aggregated userdemand needs two CPUs to instantiate VNF3 but since NS2operates at a lower user demand the instantiation of VNF3needs just a single core Note as well that bronze serviceflavour does not require HW acceleration to meet the latencyconstraints of the NSs Finally it can also be observed that

Mobile Information Systems 9

Tenant 1 Tenant 2 Tenant 3

NS1 NS2 NS1 NS2 NS1

Service type Bronze Bronze Bronze Bronze Bronze

Max ow

Max latency

Deviation 0 0 0 0 0

Peak ow

VNF1VNF3

VNF1

VNF6

VNF1

SCVNF

VNF3VNF2

VNF1VNF3

VNF1VNF2

VNF2VNF3

VNF5

VNF5VNF2

VNF5

SCVNF

VNF1

VNF5

VNF3

VNF4

VNF5

VNF6

SCVNF

VNF1

VNF1VNF2

VNF3

VNF4

VNF1

960Mbps

960Mbps

780Mbps

780Mbps

780Mbps

260Mbps

VNF2

VNF6

CPU

s

Finallatency

Tenant 1NS1

NS2

Tenant 2NS1

NS2

Tenant 3 NS1

Total energy consumption

Switc

hH

A A

cc

132ＧＭ

102ＧＭ

132ＧＭ

102ＧＭ

130ＧＭ

3712７

130－＜ＪＭ 90－＜ＪＭ 130－＜ＪＭ 130－＜ＪＭ90－＜ＪＭ

130－＜ＪＭ 90－＜ＪＭ 130－＜ＪＭ 130－＜ＪＭ90－＜ＪＭ

150ＧＭ 110ＧＭ 150ＧＭ 110ＧＭ 150ＧＭ

Figure 6 Placement results of bronze flavour setting with 0 robust protection

the algorithm tries to group asmanyVNFs as possible in eachserver in order to reduce the data traffic across the switch andminimize its load and therefore its energy waste

522 Silver Flavour Scenario Results with 0 Robust Pro-tection The evaluation experiment analyzed in this sectionstudies the outcome of the placement algorithm when thelatency constraints become more strict with silver serviceflavour Due to the further limitation of maximum latencyvalues (20 stricter) the placement mechanism is compelledto use HW acceleration units Figure 7 shows that at thisservice level all the NSs must use HW acceleration in oneof the VNFs composing the service chain As a consequencethe resulting total latency of the NSs decreases to meet theservice level requirements but in return the global powerconsumption grows by 22 up to 4547W

Apart from that the placement results illustrate the samebehavior as the bronze flavoured example of Section 521regarding the allocation of VNFs in the available resourcesscale-inout features and multitenancy

The evaluation experiments of Sections 521 and 522operate with a nominal aggregated user traffic to perform theoptimization decision-making process However a perfectlyknown user traffic demand is seldom available Usuallythere is a certain level of uncertainty in the behavior of theuser demand We include robust optimization techniques to

allow the placement algorithm to insert a defined level ofuncertainty and protect the system against the undesirableeffects of demand peaksThe evaluation examples in Sections523 and 524 include a robust protection parameter andanalyze its effect on the global power consumption

523 Silver Flavour Scenario Results with 20 Robust Protec-tion This section studies the placement outcome of the eval-uation scenario described in the previous section introducinga 20 robust protection parameter This modification on thescenario configuration implies an increase of the 20 of theaggregated user traffic as an input of the decision-makingprocess in order to be prepared for potential demand peaksduring the service operation

Figure 8 exhibits that VNF3 of NS2 must scale as a con-sequence of the traffic growth and now needs two CPUs foreach instance The traffic increase also has an impact on theprocessing latency of the VNFs causing an adjustment in theuse of HW accelerators As a consequence some VNFs thatrequired 2CPUs for their operation in the previous evalu-ation scenario now only need a single core in combinationwith the HWA Therefore in some cases the higher powerconsumption that is implicit to the use of HWAs may bebalanced with a decrease on the number of CPUs required forthe VNF Besides the higher aggregated user traffic involvesa more intense use of the network switch These factors have

10 Mobile Information Systems

SCVNF

VNF6

VNF1VNF1

VNF2

VNF3

VNF3

VNF3

VNF5

VNF2

VNF1

VNF2

SCVNF

VNF1

VNF2

VNF4

VNF5

VNF6

SCVNF

VNF2

VNF3

VNF4

VNF5

VNF6

VNF3

VNF1

Finallatency

Tenant 1NS1

NS2

Tenant 2NS1

NS2

Tenant 3 NS1

Total energy consumption

VNF6VNF1VNF1VNF2VNF2

Tenant 1 Tenant 2 Tenant 3

NS1 NS2 NS1 NS2 NS1

Service type Silver Silver Silver Silver Silver

Max ow

Max latency

Deviation 0 0 0 0 0

Peak ow

960Mbps

960Mbps

780Mbps

780Mbps

260Mbps

780MbpsSw

itch

100ＧＭ

118ＧＭ

116ＧＭ

4547７

130－＜ＪＭ 90－＜ＪＭ 130－＜ＪＭ 130－＜ＪＭ90－＜ＪＭ

130－＜ＪＭ 90－＜ＪＭ 130－＜ＪＭ 130－＜ＪＭ90－＜ＪＭ

120ＧＭ 120ＧＭ 120ＧＭ

CPU

sH

A A

cc

88ＧＭ

88ＧＭ

90ＧＭ 90ＧＭ

Figure 7 Placement results of silver flavour setting with 0 robust protection

an impact on the power consumption of the componentsof the network resulting in a global energy consumption of5047W which implies an increment of 11 compared to theunprotected placement

524 Gold Flavour Scenario Results with 20 Robust Protec-tion Finally the last evaluation example sets even stricterlatency constraints to theNSs to be placed and includes a 20robust protection parameter Figure 9 depicts the results ofthe placement algorithm with this configuration Again thereduction of latency values forces the increment of the use ofHW acceleration In this case the main consequence is thespreading of the VNFs across the SCs to use the single HWAavailable in each cellThe use of HWAs the increase of powerconsumption due to higher user aggregated traffic and thehigher network traffic traversing the switch cause the globalenergy consumption to grow up to 5569W

As a final remark Figure 10 shows the comparison ofthe energy consumption and use of resources (cores activeCESCs and HWA) related to the robust protection level ina wider collection of scenarios In particular the charts ofFigure 10 compare the results of the placement algorithm forthe three service levels described in the previous sections(bronze silver and gold) combined with three robust protec-tion levels (0 or no protection 10 and 20) Obviouslythe main trend is that both power and resource consumptionincrease with higher values of protection but the associatedcost can be assumed in favor of reducing the impact of

unexpected user demand peaks However there are somecases in which the protection parameter does not have soadverse implications For example it can be observed howthe number of used cores in the silver service level with a10 protection level is lower than in the case of no robustprotection The reason of this behavior is how the algorithmuses HWA appliances in this case The increase of the usertraffic demand in the protected services implies the scaling upof someVNFs that nowmust usemore cores for the operationof theNSsThen the placement algorithm decides to useHWaccelerators for those specific VNFs compensating this waythe higher energy cost of HWAs with a lower use of CPUcores With all these considerations that can be extractedfrom a deeper analysis of robust placement results robustoptimization can be applied to decision-making problemsnot ending just with the optimization problem

The proposed steps in this paper can be used in anyplacement decision-making context and its goal is to providea tool that helps system administrator to evaluate the possibleimpact of any decision on the performance of the deploymentunderneath

6 Conclusions

This paper introduces a constraint-based expert system tosupport the decision-making process of VNF chain place-ment in distributed edge cloud networks The placement

Mobile Information Systems 11

SCVNF

VNF5

VNF6VNF5VNF2

VNF3

VNF5

VNF1

VNF2

VNF3

VNF1

VNF2

VNF3

VNF6

SCVNF

VNF1

VNF1

VNF3

VNF4

SCVNF

VNF1

VNF2

VNF3

VNF4

840Mbps

840Mbps

936Mbps

624Mbps

312Mbps

312Mbps

936Mbps

VNF6

Tenant 1NS1

NS2

Tenant 2NS1

NS2

Tenant 3 NS1

Total energy consumption

VNF5VNF6VNF5VNF1VNF4VNF4

VNF2

Finallatency

Tenant 1 Tenant 2 Tenant 3

NS1 NS2 NS1 NS2 NS1

Service type Silver Silver Silver Silver Silver

Max ow

Max latency

Deviation 20 20 20 20 20

Peak ow

116ＧＭ

110ＧＭ

118ＧＭ

5047７

130－＜ＪＭ 90－＜ＪＭ 130－＜ＪＭ 130－＜ＪＭ90－＜ＪＭ

156－＜ＪＭ 108－＜ＪＭ 156－＜ＪＭ 108－＜ＪＭ 156－＜ＪＭ

150ＧＭ 110ＧＭ 150ＧＭ 110ＧＭ 150ＧＭ

79ＧＭ

79ＧＭ

CPU

sH

A A

ccSw

itch

Figure 8 Placement results of silver flavour setting with 20 robust protection

algorithm applies robust optimization techniques to theminimization of the global power consumption of the systemsubject to the bit rate and latency constraints of the networkservices The power consumption of the components of thesystem depends on the usage percentage which is a con-sequence of the aggregated user traffic demand The modelassumes that the relationships between the user trafficdemand the power consumption and the use of the resourcesmay be nonlinear Besides the placement algorithmconsidersthe 5G communication particularities of both control anddata plane and allows multitenancy Finally the algorithmalso includes VNF scaling features and the use of hardwareacceleration

The results show that the placement algorithm succeeds inallocating theVNFs that compose theNSs of different tenantsin the available resourcesmeeting the latency constraints andminimizing the energy consumption of thewhole systemThepresented evaluation examples demonstrate the features ofthe constraint-based expert system and illustrate the impactof the selected service flavour and robust protection on theglobal power consumption and therefore the exploitationcost of the 5G distributed edge cloud

Further researchwill extend the characterization of VNFsto include variable output bit rate This feature will modelthe behavior of for example transcoding functions A future

version of the algorithm will also incorporate the existenceof a virtual switch to connect the VMs running in the samemicroserver Another enhancement of the model will bethe combination of heterogeneous SCs with different con-figurations and resource consumption models Finally theobtained placement results will be used for a technoeconomicstudy that will analyze the relationship between the serveraggregated user traffic cost of the required infrastructure andthe pricing of the services This study will also consider theintegration of the placement decision problem with the radioside

Summary of Model Indexes SetsParameters and Decision Variables

Indexes

120587 Index for VNF placement policy119909 Index for SC microserver119910 Index for core number in each microserver119894 Index for NS119895 Index for VNF in a network service chain119903 Index for physical resources in

SC microserver

12 Mobile Information Systems

VNF6VNF5VNF1VNF6VNF2SCVNF

VNF2

VNF3

VNF5

VNF3

VNF1

VNF2

VNF3

VNF5

SCVNF

VNF1

VNF1

VNF3

VNF4

SCVNF

VNF2

VNF3

VNF4

VNF1

Tenant 1NS1

NS2

Tenant 2NS1

NS2

Tenant 3 NS1

Total energy consumption

VNF6VNF5VNF1VNF6VNF2VNF5VNF6VNF4VNF4

VNF2

840Mbps

840Mbps

936Mbps

936Mbps

312Mbps

312Mbps

312Mbps

312Mbps

312Mbps

Finallatency

Tenant 1 Tenant 2 Tenant 3

NS1 NS2 NS1 NS2 NS1

Service type Gold Gold Gold Gold Gold

Max ow

Max latency

Deviation 20 20 20 20 20

Peak ow

5569７

130－＜ＪＭ 90－＜ＪＭ 130－＜ＪＭ 130－＜ＪＭ90－＜ＪＭ

156－＜ＪＭ 108－＜ＪＭ 156－＜ＪＭ 108－＜ＪＭ 156－＜ＪＭ

100ＧＭ 100ＧＭ 100ＧＭ

79ＧＭ

95ＧＭ

89ＧＭ

79ＧＭ

86ＧＭ

CPU

sH

A A

ccSw

itch

80ＧＭ 80ＧＭ

Figure 9 Placement results of gold flavour setting with 20 robust protection

34

32

30

28

260

10

20Robust protection ()

Num

ber o

f use

dco

res

10864200

10

20Robust protection ()

HW

AN

umbe

r of u

sed

0

10

20Robust protection ()

987654N

umbe

r of a

ctiv

eCE

SCs

6000

5500

5000

4500

4000

35003000

010

20

Ener

gy co

nsum

ptio

n(W

)

Robust protection ()

Gold

Silver

Bronze

Gold

Silver

Bronze

Gold

Silver

Bronze

Gold

Silver

Bronze

Figure 10 Comparison of energy and resource consumption for different levels of robust protection

Mobile Information Systems 13

Sets

Π Set of possible VNF placement policies119883 Set of SC microservers119884 Set of cores composing a SC microserver119868 Set of offered NSs119869119894 Set of VNFs in NS 119894119877 Set of resources available in

each SC microserver

Input Parameters

119879119894 Contracted maximum aggregated usertraffic for network service 119894

119863max119894 Maximum latency allowed for NS 119894119872119903119909 Amount of resource 119903 available in

SC microserver 119909119875sc119909 Power consumption of SC 119909 because of

being switched on119875cpu119909119910119894119895 Power consumption of VM 119910 of SC 119909 due

to CPU use of VNF 119895 of NS 119894119875switch119894119895 Power consumption in the physical switch

due to CPU use of VNF 119895 of NS 119894

Decision Variables

119875120587 Total power consumption ofplacement policy 120587

119875120587119894119895 Power consumption of VNF 119895 of NS 119894 inplacement policy 120587

119898120587119903119909 Consumption of resource 119903 in SCmicroserver 119909 in placement policy 120587

119905119897120587119909 Traffic of the link that connects SCmicroserver 119909 with the switchinplacement policy 120587

119860119909119910119894119895 Binary variable that expressesthe assignment of VNF 119895 of NS 119894 toVM 119910 in SC 119909

119871 119894119895 Binary variable to express the use of thenetwork switch to forward the flow ofVNF 119895 of NS 119894 to the next VNF

119889119894 Delay of NS 119894119889119894119895 Delay of VNF 119895 of NS 119894119889119901119909119910119894119895 Processing delay in the CPU of VNF 119895 of

NS 119894 in VM 119910 of SC 1199091198891198971199091119910111990921199102119894119895 Path delay for NS 119894 between VNF 119895

(located in VM 1199101 of SC 1199091) and VNF119895 + 1 (located in VM 1199102 of SC 1199092)

119889vs119909119910 Link delay between the virtual switch inSC 119909 and the VM 119910 in SC 119909

119889vs119909 119904 Link delay between the virtual switch inSC 119909 and the physical switch

119889vs119909 Delay in the virtual switch of SC 119909

119889119904 Delay in the physical switch

Conflicts of Interest

The authors declare that they have no conflicts of interest

Acknowledgments

The research leading to these results has been supportedby the EU funded H2020 5G-PPP Project SESAME (GrantAgreement 671596) and the Spanish MINECO Project5GRANVIR (TEC2016-80090-C2-2-R)

References

[1] ITU-R M2083 IMT Vision-Framework and overall objectivesof the future development of IMT for 2020 and beyond 2015

[2] P Demestichas A Georgakopoulos D Karvounas et al ldquo5G onthe Horizon key challenges for the radio-access networkrdquo IEEEVehicular Technology Magazine vol 8 no 3 pp 47ndash53 2013

[3] I F Akyildiz S Nie S-C Lin and M Chandrasekaran ldquo5Groadmap 10 key enabling technologiesrdquo Computer Networksvol 106 pp 17ndash48 2016

[4] ETSI Mobile-edge computing Introductory technical whitepaper 2014

[5] B Han V Gopalakrishnan L Ji and S Lee ldquoNetwork functionvirtualization challenges and opportunities for innovationsrdquoIEEE Communications Magazine vol 53 no 2 pp 90ndash97 2015

[6] R Mijumbi J Serrat J Gorricho N Bouten F De Turckand R Boutaba ldquoNetwork function virtualization state-of-the-art and research challengesrdquo IEEE Communications Surveys ampTutorials vol 18 no 1 pp 236ndash262 2016

[7] Ericsson White Paper Cloud-RANndash the benefits of virtualiza-tion centralization and coordination 2015

[8] B Blanco J O Fajardo I Giannoulakis et al ldquoTechnologypillars in the architecture of future 5G mobile networks NFVMEC and SDNrdquo Computer Standards and Interfaces vol 54 pp216ndash228 2017

[9] ETSI Network Functions Virtualization (NFV) Managementand Orchestration 2014

[10] A Ben-Tal L E Ghaoui and A Nemirovski Robust Optimiza-tion Aharon Ben-Tal Laurent El Ghaoui Arkadi Nemirovski-Google Books Princeton University Press Princeton NewJersey NY USA 2009

[11] J Gil Herrera and J F Botero ldquoResource Allocation in NFVA Comprehensive Surveyrdquo IEEE Transactions on Network andService Management vol 13 no 3 pp 518ndash532 2016

[12] B Addis D Belabed M Bouet and S Secci ldquoVirtual networkfunctions placement and routing optimizationrdquo in Proceedingsof the 4th IEEE International Conference on Cloud NetworkingCloudNet rsquo15 pp 171ndash177 October 2015

[13] H Moens and F De Turck ldquoVNF-P a model for efficientplacement of virtualized network functionsrdquo in Proceedingsof the 10th International Conference on Network and ServiceManagement (CNSM rsquo14) pp 418ndash423 Rio de Janeiro BrazilNovember 2014

[14] A Gupta M F Habib P Chowdhury M Tornatore and BMukherjee ldquoOn service chaining using Virtual Network Func-tions in Network-enabled Cloud systemsrdquo in Proceedings of the9th IEEE International Conference on Advanced Networks andTelecommuncations Systems ANTS rsquo15 pp 1ndash3 December 2015

[15] M Bouet J Leguay T Combe and V Conan ldquoCost-basedplacement of vDPI functions in NFV infrastructuresrdquo Interna-tional Journal of Network Management vol 25 no 6 pp 490ndash506 2015

[16] M C Luizelli L R Bays L S Buriol M P Barcellos andL P Gaspary ldquoPiecing together the NFV provisioning puzzle

14 Mobile Information Systems

efficient placement and chaining of virtual network functionsrdquoin Proceedings of the 14th International Symposium on IntegratedNetwork Management IM rsquo15 pp 98ndash106 IEEE TorontoCanada May 2015

[17] S Kim Y Han and S Park ldquoAn energy-Aware service functionchaining and reconfiguration algorithm in NFVrdquo in Proceedingsof the 1st International Workshops on Foundations and Applica-tions of Self-Systems FAS-W rsquo16 pp 54ndash59 September 2016

[18] V Eramo A Tosti and E Miucci ldquoServer resource dimen-sioning and routing of service function chain in NFV networkarchitecturesrdquo Journal of Electrical and Computer Engineeringvol 2016 Article ID 7139852 12 pages 2016

[19] K Hida and S-I Kuribayashi ldquoVirtual routing function allo-cation method for minimizing total network power consump-tionrdquo World Academy of Science Engineering and TechnologyInternational Journal of Electrical Computer Energetic Elec-tronic and Communication Engineering vol 10 pp 997ndash10022016

[20] C Pham N H Tran S Ren W Saad and C S Hong ldquoTraffic-aware and energy-efficient vnf placement for service chainingjoint sampling and matching approachrdquo IEEE Transactions onServices Computing 2017

[21] A Marotta and A Kassler ldquoA power efficient and robust virtualnetwork functions placement problemrdquo in Proceedings of the28th International Teletraffic Congress ITC rsquo16 pp 331ndash339September 2016

[22] C Pham H D Tran S I Moon K Thar and C S Hong ldquoAgeneral and practical consolidation framework in CloudNFVrdquoin Proceedings of the 2015 International Conference on Infor-mation Networking ICOIN rsquo15 pp 295ndash300 IEEE CambodiaCambodia January 2015

[23] V Eramo M Ammar and F G Lavacca ldquoMigration energyaware reconfigurations of virtual network function instances inNFV architecturesrdquo IEEE Access vol 5 pp 4927ndash4938 2017

[24] N E Khoury S Ayoubi and C Assi ldquoEnergy-aware placementand scheduling of network trafficflowswith deadlines on virtualnetwork functionsrdquo in Proceedings of the 5th IEEE InternationalConference on CloudNetworking CloudNet rsquo16 pp 89ndash94 IEEEPisa Italy October 2016

[25] V Eramo E Miucci M Ammar and F G Lavacca ldquoAnapproach for service function chain routing and virtual func-tion network instance migration in network function virtual-ization architecturesrdquo IEEEACM Transactions on Networking2017

[26] V Jumba S Parsaeefard M Derakhshani and T Le-NgocldquoEnergy-efficient robust resource provisioning in virtualizedwireless networksrdquo in Proceedings of the IEEE InternationalConference on Ubiquitous Wireless Broadband ICUWB rsquo15 pp1ndash5 IEEE Montreal Canada October 2015

[27] S Coniglio A Koster and M Tieves ldquoData uncertainty invirtual network embedding robust optimization and protectionlevelsrdquo Journal of Network and SystemsManagement vol 24 no3 pp 681ndash710 2016

[28] I Takouna K Sachs and C Meinel ldquoMultiperiod robustoptimization for proactive resource provisioning in virtualizeddata centersrdquo Journal of Supercomputing vol 70 no 3 pp 1514ndash1536 2014

[29] G Chochlidakis and V Friderikos ldquoRobust virtual networkembedding for mobile networksrdquo in Proceedings of the 26thIEEE Annual International Symposium on Personal Indoor andMobile Radio Communications PIMRC rsquo15 pp 1867ndash1871 IEEEKowloon Hong Kong September 2015

[30] S Coniglio A M Koster and M Tieves ldquoVirtual networkembedding under uncertainty exact and heuristic approachesrdquoin Proceedings of the 11th International Conference on the Designof Reliable Communication Networks DRCN rsquo15 pp 1ndash8 IEEEKansas Kan USA March 2015

[31] D Bertsimas and M Sim ldquoThe price of robustnessrdquoOperationsResearch vol 52 no 1 pp 35ndash53 2004

[32] F Rossi P Van Beek and T Walsh Handbook of ConstraintProgramming Elsevier Amsterdam Netherland 2006

[33] httpwwwminizincorg[34] C Schulte G Tack and M Z Lagerkvist Modeling and pro-

gramming with gecode 2010 Schulte Christian and Tack Guidoand Lagerkvist Mikael

6 Mobile Information Systems

a NS are located in different SCs external links and the physi-cal switch is also used therefore generally 11988911989711990911199101 11990921199102119894119895 =11988911990911199101vs + 119889vs

1199091+ 119889vs1199091 119904 + 119889119904 + 119889119904vs1199092 + 119889vs

1199092+ 119889vs11990921199102 but for

a given VNF if the next VNF in its chain holds in the sameSC its link delay is only affected because of passing throughthe virtual switch Note that in case a VNF is the last VNF inthe chain the link delay is null with the sums referring to thelink delay for that 119895 being null

In order to guarantee that each VM holds only a singleVNF constraint (9) must be satisfied Nevertheless weconsider that each VNF of a NS can be instantiated as a VMthat is executed over several cores of the same SC Apart fromthat as previously detailed in Section 3 the first VNF and thelast VNF in a service chain are the same (the SCVNF) in orderto perform the specific functions that allow the split betweencontrol and data plane required in 5G communicationsTherefore it implies that 1198601199091199101198941 = 119860119909119869119894119894119895

Finally the model must also include the constraintsreferred to as the available resources In this sense (3) statesthat the consumption of the resource 119903 due to the placementof VNFs in the microserver of the SC 119909119898120587119903119909 must not exceedthe total amount of resource 119872119903119909 where 119898120587119903119909 is defined in(10) Similarly (4) constrains the traffic through the networklinks 119905119897120587119909 to the maximum capacity of the links BW119909 as thesum of the traffic flows of the network services that must beforwarded from a VNF in one SC to the next VNF in anotherSC

42 Robust Constraint-Based Solution As described inSection 2 many existing placement works assume perfectknowledge on input parameters pointing thus to the for-mulation of a deterministic optimization problem Unfortu-nately in the real world the estimated values of the inputparameters may differ due to biased data unrealistic assump-tions or numerical errors affecting the obtained optimalsolution and its performance This potential deviation ofthe nominal or expected input may lead to the violation ofthe problem constraints and therefore make the obtainedsolution suboptimal or even unfeasible

The uncertainty of modelling parameters has been tra-ditionally handled through stochastic programming andsensitivity analysis but robust optimization techniques haverecently arisen as a powerful tool to manage the impact ofuncertain input sets RO seeks an uncertainty-immunizedsolution with an acceptable performance under the real-ization of the uncertain inputs becoming a conservativeworst-case oriented methodology The main tools of RO areuncertainty sets and a robust counterpart problem [31] Theuncertainty of the input data is described later in this sectionwhile the robust counterpart problem is the deterministicmodel previously detailed in Section 41

Here we propose a robust constraint-based placementsolution that deals with service demand uncertainty Withthis RO approach we consider a tradeoff between theproblem optimization and the protection from deviationscaused by input parameters uncertainty This uncertainty inservice demand has an impact on several components on theformulation previously presented in Section 41 On the one

hand we consider that a VM which is running a specificVNF requires an expected use of CPU and thus a certainconstant power consumption due to CPU Neverthelessthe uncertainty in traffic demand may cause uncertainty inthe CPU requirementuse and consequently in its energyconsumption and processing delays On the other hand bothenergy consumption and delays in switches and links dependon that demand uncertainty as well

In order to obtain a proposal for the placement problemwith uncertain service demand we employ the Soyster [10]method used in the framework of RO That technique isalso known as the worst-case approach where the solutionsare achieved using the most extreme expected values of theuncertain variables Therefore that kind of resolution guar-antees feasible solutions for any value of the uncertain servicedemand

In reference to the modelling of the uncertain trafficdemand per NS 119894 we consider an expected traffic demand119864[119879119894] = 119879119894 plus a term due to uncertainty in the servicedemandUTmax119894 We assume that the randomvariableUTmax119894is symmetrically distributed within [minusΔ119879119894 +Δ119879119894] sdot 119879119894 andwith mean zero Furthermore the sum of deviations of theuncertain service demand should not exceed the worst-casemaximum value 119879max119894 fulfilling (12) In this way the valueof 119879max119894 controls the tradeoff between the robustness andthe impact on the optimization higher 119879max119894 leads to worseobjective function values but protects from more parameterdeviations

119880119879119894Δ119879119894

le 119879max119894 forall119894 (12)

5 Analysis of the Results

This section presents the results obtained with the simulationof the proposed robust constraint-based expert system for theplacement of VNF chains in an emerging 5G distributed edgecloud As previously introduced in Section 2 the mathemat-ical VNF placement problemmodel presented in Section 4 isa combinatorial or discrete optimization problem equivalentto the well-known knapsack problem This means that thedecision problem is NP-complete while the optimizationproblem is NP-hard [11] The proposed optimization modelhas been solved using constraint programming as a powerfulparadigm for solving combinatorial search problems [32]

We usedMinizinc [33] a free and open-source constraintmodelling language to model the optimization and con-straint satisfaction VNF placement problem in a high-levelsolver-independent wayThen Minizinc compiles this modelinto FlatZinc a solver input language that is understoodby a wide range of solvers In our case Gecode solver [34]provided the best performance in the search of the optimalsolution

The experiments that are described and discussed belowhave been conducted in a single computer with a 27GHzIntel Core i5 processor and 16GB RAM This simple con-figuration provides solutions within minutes for simpleexperiments with loose constraint satisfaction When theexperiments demand tighter latency or resource constraints

Mobile Information Systems 7

Virt

ual i

nfra

struc

ture

Corenetwork

Figure 4 Evaluation scenario

the solving time increases to hours depending on the specificconfiguration of the experiment

Next we first describe the analysis scenario and the con-figuration of the performed experiments to later discuss theresults of those experimentsThe objective of the experimentsis to demonstrate that the placement mechanism is able toallocate multitenant service chains to the available resourcesconsidering latency constraints For this reason the numeri-cal values employed in the next section are just experimentaland do not respond to real world measurements

51 Description of the Analysis Scenario Our evaluationscenario is depicted in Figure 4 We consider a 10-SC deploy-ment for example to cover a football stadium for periodicalflash eventsThe SCs are connected in a star topology throughan Ethernet switch Each SC is composed of 8 cores 16GBRAM 100GB of storage and 1HWA (eg GPU)

We also consider 2 NSs that can be offered at 3 serviceflavours (bronze silver and gold) that determine the latencyconstraint for eachNS and the nominal aggregate user bit rateserved as shown in Figure 5

Table 1 shows the experimental energy model and Table 2shows the modelling of the VNFs that compose the proposedNSs for the user traffic demand that will be employed in theevaluation experiments The user traffic values include thenominal bit rates of services NS1 and NS2 (130Mbps and90Mbps resp) as well as the values when a 20 protectionparameter is applied to the nominal values (156Mbps and108Mbps resp) in order to study the effect of robust optimi-zation techniques on the placement decision

As a final remark the placement behavior of VNFs canbe altered by the use of HWA The assignment of a HWA toa VNF implies that the service latency of the correspondentinstance is reduced 3 times at a cost of incrementing thepower consumption by 10 It is important to note that whenthe VNF instance needs more than 1 CPU for executingwithout the assistance of a HWA then if the placementalgorithm assigns it to a HWA the VNF will need a singlecore

52 Discussion of the Placement Results As stated before theevaluation scenario includes 3 tenants that hire a combination

8 Mobile Information Systems

VNF1 VNF2 VNF3 VNF5 VNF6

VNF1 VNF2 VNF3 VNF4

NS1

NS2

Service flavourBronze Silver Gold

NS1

NS2 Max Flow = 90MbpsMax Lat = 90ms Max Lat = 80ms

Max Flow = 130－＜ＪＭ

Max Lat = 150ＧＭ

Max Lat = 110ＧＭ

Max Flow = 90Mbps

Max Flow = 130－＜ＪＭ

Max Lat = 120ＧＭ

Max Flow = 90Mbps

Max Flow = 130－＜ＪＭ

Max Lat = 100ＧＭ

Figure 5 Description of NSs and service levels

Table 1 Energy models of the cores of the microservers and of the network switch related to the usage percentage

Usage 0 10 20 30 40 50 60 70 80 90 100119875router 0 1 2 5 7 9 11 13 14 15 15119875cpu 5 50 70 100 130 150 165 170 180 185 190

Table 2 VNF characterization for the service flavours of the evaluation scenario

VNF1 VNF2 VNF3 VNF4 VNF5 VNF6Number of cores per instance

90Mbps 1 2 1 1 1 1108Mbps 1 2 2 1 1 1130Mbps 1 2 2 1 1 1156Mbps 1 2 2 1 1 1

CPU usage90Mbps 30 20 90 50 50 20108Mbps 40 30 60 60 60 30130Mbps 40 30 60 60 60 30156Mbps 40 30 90 40 20 30

Service latency90Mbps 10ms 15ms 30ms 35ms 20ms 25ms108Mbps 10ms 15ms 35ms 40ms 25ms 30ms130Mbps 10ms 15ms 35ms 40ms 25ms 30ms156Mbps 12ms 16ms 40ms 50ms 30ms 35ms

of network servicesflavours to the SC operator In this wayin the following first we are going to evaluate the behavior ofthe placement algorithm for the three service levels and nextwe will show the effect of applying RO techniques to thoseresults

521 Bronze Flavour Scenario Results with 0 Robust Pro-tection Figure 6 shows the placement results of an scenariowhere all the network services operate at bronze flavourDifferent colours represent each tenant in order to dif-ferentiate the VNFs that belong to its NS and differenttones distinguish NSs of the same tenant As a result ofthe placement algorithm all the NSs are allocated in theavailable virtualized resources with energy consumption costof 3712W meeting all the latency constraints The figure

displays the assignment of VNFs to the SCs highlightingsome placement particularities As explained in Section 3the VNF chains of the same tenant share their SCVNF Inour example Tenant 1 and Tenant 2 both have two NSsthat are served with a single SCVNF Furthermore eachCPU executes a single VM but some VNF instances needtwo cores to manage the user demand In connection withthis behavior the VNF scalability feature is also visible inFigure 6 observing the placement of for example VNF3for the different NSs NS1 with a higher aggregated userdemand needs two CPUs to instantiate VNF3 but since NS2operates at a lower user demand the instantiation of VNF3needs just a single core Note as well that bronze serviceflavour does not require HW acceleration to meet the latencyconstraints of the NSs Finally it can also be observed that

Mobile Information Systems 9

Tenant 1 Tenant 2 Tenant 3

NS1 NS2 NS1 NS2 NS1

Service type Bronze Bronze Bronze Bronze Bronze

Max ow

Max latency

Deviation 0 0 0 0 0

Peak ow

VNF1VNF3

VNF1

VNF6

VNF1

SCVNF

VNF3VNF2

VNF1VNF3

VNF1VNF2

VNF2VNF3

VNF5

VNF5VNF2

VNF5

SCVNF

VNF1

VNF5

VNF3

VNF4

VNF5

VNF6

SCVNF

VNF1

VNF1VNF2

VNF3

VNF4

VNF1

960Mbps

960Mbps

780Mbps

780Mbps

780Mbps

260Mbps

VNF2

VNF6

CPU

s

Finallatency

Tenant 1NS1

NS2

Tenant 2NS1

NS2

Tenant 3 NS1

Total energy consumption

Switc

hH

A A

cc

132ＧＭ

102ＧＭ

132ＧＭ

102ＧＭ

130ＧＭ

3712７

130－＜ＪＭ 90－＜ＪＭ 130－＜ＪＭ 130－＜ＪＭ90－＜ＪＭ

130－＜ＪＭ 90－＜ＪＭ 130－＜ＪＭ 130－＜ＪＭ90－＜ＪＭ

150ＧＭ 110ＧＭ 150ＧＭ 110ＧＭ 150ＧＭ

Figure 6 Placement results of bronze flavour setting with 0 robust protection

the algorithm tries to group asmanyVNFs as possible in eachserver in order to reduce the data traffic across the switch andminimize its load and therefore its energy waste

522 Silver Flavour Scenario Results with 0 Robust Pro-tection The evaluation experiment analyzed in this sectionstudies the outcome of the placement algorithm when thelatency constraints become more strict with silver serviceflavour Due to the further limitation of maximum latencyvalues (20 stricter) the placement mechanism is compelledto use HW acceleration units Figure 7 shows that at thisservice level all the NSs must use HW acceleration in oneof the VNFs composing the service chain As a consequencethe resulting total latency of the NSs decreases to meet theservice level requirements but in return the global powerconsumption grows by 22 up to 4547W

Apart from that the placement results illustrate the samebehavior as the bronze flavoured example of Section 521regarding the allocation of VNFs in the available resourcesscale-inout features and multitenancy

The evaluation experiments of Sections 521 and 522operate with a nominal aggregated user traffic to perform theoptimization decision-making process However a perfectlyknown user traffic demand is seldom available Usuallythere is a certain level of uncertainty in the behavior of theuser demand We include robust optimization techniques to

allow the placement algorithm to insert a defined level ofuncertainty and protect the system against the undesirableeffects of demand peaksThe evaluation examples in Sections523 and 524 include a robust protection parameter andanalyze its effect on the global power consumption

523 Silver Flavour Scenario Results with 20 Robust Protec-tion This section studies the placement outcome of the eval-uation scenario described in the previous section introducinga 20 robust protection parameter This modification on thescenario configuration implies an increase of the 20 of theaggregated user traffic as an input of the decision-makingprocess in order to be prepared for potential demand peaksduring the service operation

Figure 8 exhibits that VNF3 of NS2 must scale as a con-sequence of the traffic growth and now needs two CPUs foreach instance The traffic increase also has an impact on theprocessing latency of the VNFs causing an adjustment in theuse of HW accelerators As a consequence some VNFs thatrequired 2CPUs for their operation in the previous evalu-ation scenario now only need a single core in combinationwith the HWA Therefore in some cases the higher powerconsumption that is implicit to the use of HWAs may bebalanced with a decrease on the number of CPUs required forthe VNF Besides the higher aggregated user traffic involvesa more intense use of the network switch These factors have

10 Mobile Information Systems

SCVNF

VNF6

VNF1VNF1

VNF2

VNF3

VNF3

VNF3

VNF5

VNF2

VNF1

VNF2

SCVNF

VNF1

VNF2

VNF4

VNF5

VNF6

SCVNF

VNF2

VNF3

VNF4

VNF5

VNF6

VNF3

VNF1

Finallatency

Tenant 1NS1

NS2

Tenant 2NS1

NS2

Tenant 3 NS1

Total energy consumption

VNF6VNF1VNF1VNF2VNF2

Tenant 1 Tenant 2 Tenant 3

NS1 NS2 NS1 NS2 NS1

Service type Silver Silver Silver Silver Silver

Max ow

Max latency

Deviation 0 0 0 0 0

Peak ow

960Mbps

960Mbps

780Mbps

780Mbps

260Mbps

780MbpsSw

itch

100ＧＭ

118ＧＭ

116ＧＭ

4547７

130－＜ＪＭ 90－＜ＪＭ 130－＜ＪＭ 130－＜ＪＭ90－＜ＪＭ

130－＜ＪＭ 90－＜ＪＭ 130－＜ＪＭ 130－＜ＪＭ90－＜ＪＭ

120ＧＭ 120ＧＭ 120ＧＭ

CPU

sH

A A

cc

88ＧＭ

88ＧＭ

90ＧＭ 90ＧＭ

Figure 7 Placement results of silver flavour setting with 0 robust protection

an impact on the power consumption of the componentsof the network resulting in a global energy consumption of5047W which implies an increment of 11 compared to theunprotected placement

524 Gold Flavour Scenario Results with 20 Robust Protec-tion Finally the last evaluation example sets even stricterlatency constraints to theNSs to be placed and includes a 20robust protection parameter Figure 9 depicts the results ofthe placement algorithm with this configuration Again thereduction of latency values forces the increment of the use ofHW acceleration In this case the main consequence is thespreading of the VNFs across the SCs to use the single HWAavailable in each cellThe use of HWAs the increase of powerconsumption due to higher user aggregated traffic and thehigher network traffic traversing the switch cause the globalenergy consumption to grow up to 5569W

As a final remark Figure 10 shows the comparison ofthe energy consumption and use of resources (cores activeCESCs and HWA) related to the robust protection level ina wider collection of scenarios In particular the charts ofFigure 10 compare the results of the placement algorithm forthe three service levels described in the previous sections(bronze silver and gold) combined with three robust protec-tion levels (0 or no protection 10 and 20) Obviouslythe main trend is that both power and resource consumptionincrease with higher values of protection but the associatedcost can be assumed in favor of reducing the impact of

unexpected user demand peaks However there are somecases in which the protection parameter does not have soadverse implications For example it can be observed howthe number of used cores in the silver service level with a10 protection level is lower than in the case of no robustprotection The reason of this behavior is how the algorithmuses HWA appliances in this case The increase of the usertraffic demand in the protected services implies the scaling upof someVNFs that nowmust usemore cores for the operationof theNSsThen the placement algorithm decides to useHWaccelerators for those specific VNFs compensating this waythe higher energy cost of HWAs with a lower use of CPUcores With all these considerations that can be extractedfrom a deeper analysis of robust placement results robustoptimization can be applied to decision-making problemsnot ending just with the optimization problem

The proposed steps in this paper can be used in anyplacement decision-making context and its goal is to providea tool that helps system administrator to evaluate the possibleimpact of any decision on the performance of the deploymentunderneath

6 Conclusions

This paper introduces a constraint-based expert system tosupport the decision-making process of VNF chain place-ment in distributed edge cloud networks The placement

Mobile Information Systems 11

SCVNF

VNF5

VNF6VNF5VNF2

VNF3

VNF5

VNF1

VNF2

VNF3

VNF1

VNF2

VNF3

VNF6

SCVNF

VNF1

VNF1

VNF3

VNF4

SCVNF

VNF1

VNF2

VNF3

VNF4

840Mbps

840Mbps

936Mbps

624Mbps

312Mbps

312Mbps

936Mbps

VNF6

Tenant 1NS1

NS2

Tenant 2NS1

NS2

Tenant 3 NS1

Total energy consumption

VNF5VNF6VNF5VNF1VNF4VNF4

VNF2

Finallatency

Tenant 1 Tenant 2 Tenant 3

NS1 NS2 NS1 NS2 NS1

Service type Silver Silver Silver Silver Silver

Max ow

Max latency

Deviation 20 20 20 20 20

Peak ow

116ＧＭ

110ＧＭ

118ＧＭ

5047７

130－＜ＪＭ 90－＜ＪＭ 130－＜ＪＭ 130－＜ＪＭ90－＜ＪＭ

156－＜ＪＭ 108－＜ＪＭ 156－＜ＪＭ 108－＜ＪＭ 156－＜ＪＭ

150ＧＭ 110ＧＭ 150ＧＭ 110ＧＭ 150ＧＭ

79ＧＭ

79ＧＭ

CPU

sH

A A

ccSw

itch

Figure 8 Placement results of silver flavour setting with 20 robust protection

algorithm applies robust optimization techniques to theminimization of the global power consumption of the systemsubject to the bit rate and latency constraints of the networkservices The power consumption of the components of thesystem depends on the usage percentage which is a con-sequence of the aggregated user traffic demand The modelassumes that the relationships between the user trafficdemand the power consumption and the use of the resourcesmay be nonlinear Besides the placement algorithmconsidersthe 5G communication particularities of both control anddata plane and allows multitenancy Finally the algorithmalso includes VNF scaling features and the use of hardwareacceleration

The results show that the placement algorithm succeeds inallocating theVNFs that compose theNSs of different tenantsin the available resourcesmeeting the latency constraints andminimizing the energy consumption of thewhole systemThepresented evaluation examples demonstrate the features ofthe constraint-based expert system and illustrate the impactof the selected service flavour and robust protection on theglobal power consumption and therefore the exploitationcost of the 5G distributed edge cloud

Further researchwill extend the characterization of VNFsto include variable output bit rate This feature will modelthe behavior of for example transcoding functions A future

version of the algorithm will also incorporate the existenceof a virtual switch to connect the VMs running in the samemicroserver Another enhancement of the model will bethe combination of heterogeneous SCs with different con-figurations and resource consumption models Finally theobtained placement results will be used for a technoeconomicstudy that will analyze the relationship between the serveraggregated user traffic cost of the required infrastructure andthe pricing of the services This study will also consider theintegration of the placement decision problem with the radioside

Summary of Model Indexes SetsParameters and Decision Variables

Indexes

120587 Index for VNF placement policy119909 Index for SC microserver119910 Index for core number in each microserver119894 Index for NS119895 Index for VNF in a network service chain119903 Index for physical resources in

SC microserver

12 Mobile Information Systems

VNF6VNF5VNF1VNF6VNF2SCVNF

VNF2

VNF3

VNF5

VNF3

VNF1

VNF2

VNF3

VNF5

SCVNF

VNF1

VNF1

VNF3

VNF4

SCVNF

VNF2

VNF3

VNF4

VNF1

Tenant 1NS1

NS2

Tenant 2NS1

NS2

Tenant 3 NS1

Total energy consumption

VNF6VNF5VNF1VNF6VNF2VNF5VNF6VNF4VNF4

VNF2

840Mbps

840Mbps

936Mbps

936Mbps

312Mbps

312Mbps

312Mbps

312Mbps

312Mbps

Finallatency

Tenant 1 Tenant 2 Tenant 3

NS1 NS2 NS1 NS2 NS1

Service type Gold Gold Gold Gold Gold

Max ow

Max latency

Deviation 20 20 20 20 20

Peak ow

5569７

130－＜ＪＭ 90－＜ＪＭ 130－＜ＪＭ 130－＜ＪＭ90－＜ＪＭ

156－＜ＪＭ 108－＜ＪＭ 156－＜ＪＭ 108－＜ＪＭ 156－＜ＪＭ

100ＧＭ 100ＧＭ 100ＧＭ

79ＧＭ

95ＧＭ

89ＧＭ

79ＧＭ

86ＧＭ

CPU

sH

A A

ccSw

itch

80ＧＭ 80ＧＭ

Figure 9 Placement results of gold flavour setting with 20 robust protection

34

32

30

28

260

10

20Robust protection ()

Num

ber o

f use

dco

res

10864200

10

20Robust protection ()

HW

AN

umbe

r of u

sed

0

10

20Robust protection ()

987654N

umbe

r of a

ctiv

eCE

SCs

6000

5500

5000

4500

4000

35003000

010

20

Ener

gy co

nsum

ptio

n(W

)

Robust protection ()

Gold

Silver

Bronze

Gold

Silver

Bronze

Gold

Silver

Bronze

Gold

Silver

Bronze

Figure 10 Comparison of energy and resource consumption for different levels of robust protection

Mobile Information Systems 13

Sets

Π Set of possible VNF placement policies119883 Set of SC microservers119884 Set of cores composing a SC microserver119868 Set of offered NSs119869119894 Set of VNFs in NS 119894119877 Set of resources available in

each SC microserver

Input Parameters

119879119894 Contracted maximum aggregated usertraffic for network service 119894

119863max119894 Maximum latency allowed for NS 119894119872119903119909 Amount of resource 119903 available in

SC microserver 119909119875sc119909 Power consumption of SC 119909 because of

being switched on119875cpu119909119910119894119895 Power consumption of VM 119910 of SC 119909 due

to CPU use of VNF 119895 of NS 119894119875switch119894119895 Power consumption in the physical switch

due to CPU use of VNF 119895 of NS 119894

Decision Variables

119875120587 Total power consumption ofplacement policy 120587

119875120587119894119895 Power consumption of VNF 119895 of NS 119894 inplacement policy 120587

119898120587119903119909 Consumption of resource 119903 in SCmicroserver 119909 in placement policy 120587

119905119897120587119909 Traffic of the link that connects SCmicroserver 119909 with the switchinplacement policy 120587

119860119909119910119894119895 Binary variable that expressesthe assignment of VNF 119895 of NS 119894 toVM 119910 in SC 119909

119871 119894119895 Binary variable to express the use of thenetwork switch to forward the flow ofVNF 119895 of NS 119894 to the next VNF

119889119894 Delay of NS 119894119889119894119895 Delay of VNF 119895 of NS 119894119889119901119909119910119894119895 Processing delay in the CPU of VNF 119895 of

NS 119894 in VM 119910 of SC 1199091198891198971199091119910111990921199102119894119895 Path delay for NS 119894 between VNF 119895

(located in VM 1199101 of SC 1199091) and VNF119895 + 1 (located in VM 1199102 of SC 1199092)

119889vs119909119910 Link delay between the virtual switch inSC 119909 and the VM 119910 in SC 119909

119889vs119909 119904 Link delay between the virtual switch inSC 119909 and the physical switch

119889vs119909 Delay in the virtual switch of SC 119909

119889119904 Delay in the physical switch

Conflicts of Interest

The authors declare that they have no conflicts of interest

Acknowledgments

The research leading to these results has been supportedby the EU funded H2020 5G-PPP Project SESAME (GrantAgreement 671596) and the Spanish MINECO Project5GRANVIR (TEC2016-80090-C2-2-R)

References

[1] ITU-R M2083 IMT Vision-Framework and overall objectivesof the future development of IMT for 2020 and beyond 2015

[2] P Demestichas A Georgakopoulos D Karvounas et al ldquo5G onthe Horizon key challenges for the radio-access networkrdquo IEEEVehicular Technology Magazine vol 8 no 3 pp 47ndash53 2013

[3] I F Akyildiz S Nie S-C Lin and M Chandrasekaran ldquo5Groadmap 10 key enabling technologiesrdquo Computer Networksvol 106 pp 17ndash48 2016

[4] ETSI Mobile-edge computing Introductory technical whitepaper 2014

[5] B Han V Gopalakrishnan L Ji and S Lee ldquoNetwork functionvirtualization challenges and opportunities for innovationsrdquoIEEE Communications Magazine vol 53 no 2 pp 90ndash97 2015

[6] R Mijumbi J Serrat J Gorricho N Bouten F De Turckand R Boutaba ldquoNetwork function virtualization state-of-the-art and research challengesrdquo IEEE Communications Surveys ampTutorials vol 18 no 1 pp 236ndash262 2016

[7] Ericsson White Paper Cloud-RANndash the benefits of virtualiza-tion centralization and coordination 2015

[8] B Blanco J O Fajardo I Giannoulakis et al ldquoTechnologypillars in the architecture of future 5G mobile networks NFVMEC and SDNrdquo Computer Standards and Interfaces vol 54 pp216ndash228 2017

[9] ETSI Network Functions Virtualization (NFV) Managementand Orchestration 2014

[10] A Ben-Tal L E Ghaoui and A Nemirovski Robust Optimiza-tion Aharon Ben-Tal Laurent El Ghaoui Arkadi Nemirovski-Google Books Princeton University Press Princeton NewJersey NY USA 2009

[11] J Gil Herrera and J F Botero ldquoResource Allocation in NFVA Comprehensive Surveyrdquo IEEE Transactions on Network andService Management vol 13 no 3 pp 518ndash532 2016

[12] B Addis D Belabed M Bouet and S Secci ldquoVirtual networkfunctions placement and routing optimizationrdquo in Proceedingsof the 4th IEEE International Conference on Cloud NetworkingCloudNet rsquo15 pp 171ndash177 October 2015

[13] H Moens and F De Turck ldquoVNF-P a model for efficientplacement of virtualized network functionsrdquo in Proceedingsof the 10th International Conference on Network and ServiceManagement (CNSM rsquo14) pp 418ndash423 Rio de Janeiro BrazilNovember 2014

[14] A Gupta M F Habib P Chowdhury M Tornatore and BMukherjee ldquoOn service chaining using Virtual Network Func-tions in Network-enabled Cloud systemsrdquo in Proceedings of the9th IEEE International Conference on Advanced Networks andTelecommuncations Systems ANTS rsquo15 pp 1ndash3 December 2015

[15] M Bouet J Leguay T Combe and V Conan ldquoCost-basedplacement of vDPI functions in NFV infrastructuresrdquo Interna-tional Journal of Network Management vol 25 no 6 pp 490ndash506 2015

[16] M C Luizelli L R Bays L S Buriol M P Barcellos andL P Gaspary ldquoPiecing together the NFV provisioning puzzle

14 Mobile Information Systems

efficient placement and chaining of virtual network functionsrdquoin Proceedings of the 14th International Symposium on IntegratedNetwork Management IM rsquo15 pp 98ndash106 IEEE TorontoCanada May 2015

[17] S Kim Y Han and S Park ldquoAn energy-Aware service functionchaining and reconfiguration algorithm in NFVrdquo in Proceedingsof the 1st International Workshops on Foundations and Applica-tions of Self-Systems FAS-W rsquo16 pp 54ndash59 September 2016

[18] V Eramo A Tosti and E Miucci ldquoServer resource dimen-sioning and routing of service function chain in NFV networkarchitecturesrdquo Journal of Electrical and Computer Engineeringvol 2016 Article ID 7139852 12 pages 2016

[19] K Hida and S-I Kuribayashi ldquoVirtual routing function allo-cation method for minimizing total network power consump-tionrdquo World Academy of Science Engineering and TechnologyInternational Journal of Electrical Computer Energetic Elec-tronic and Communication Engineering vol 10 pp 997ndash10022016

[20] C Pham N H Tran S Ren W Saad and C S Hong ldquoTraffic-aware and energy-efficient vnf placement for service chainingjoint sampling and matching approachrdquo IEEE Transactions onServices Computing 2017

[21] A Marotta and A Kassler ldquoA power efficient and robust virtualnetwork functions placement problemrdquo in Proceedings of the28th International Teletraffic Congress ITC rsquo16 pp 331ndash339September 2016

[22] C Pham H D Tran S I Moon K Thar and C S Hong ldquoAgeneral and practical consolidation framework in CloudNFVrdquoin Proceedings of the 2015 International Conference on Infor-mation Networking ICOIN rsquo15 pp 295ndash300 IEEE CambodiaCambodia January 2015

[23] V Eramo M Ammar and F G Lavacca ldquoMigration energyaware reconfigurations of virtual network function instances inNFV architecturesrdquo IEEE Access vol 5 pp 4927ndash4938 2017

[24] N E Khoury S Ayoubi and C Assi ldquoEnergy-aware placementand scheduling of network trafficflowswith deadlines on virtualnetwork functionsrdquo in Proceedings of the 5th IEEE InternationalConference on CloudNetworking CloudNet rsquo16 pp 89ndash94 IEEEPisa Italy October 2016

[25] V Eramo E Miucci M Ammar and F G Lavacca ldquoAnapproach for service function chain routing and virtual func-tion network instance migration in network function virtual-ization architecturesrdquo IEEEACM Transactions on Networking2017

[26] V Jumba S Parsaeefard M Derakhshani and T Le-NgocldquoEnergy-efficient robust resource provisioning in virtualizedwireless networksrdquo in Proceedings of the IEEE InternationalConference on Ubiquitous Wireless Broadband ICUWB rsquo15 pp1ndash5 IEEE Montreal Canada October 2015

[27] S Coniglio A Koster and M Tieves ldquoData uncertainty invirtual network embedding robust optimization and protectionlevelsrdquo Journal of Network and SystemsManagement vol 24 no3 pp 681ndash710 2016

[28] I Takouna K Sachs and C Meinel ldquoMultiperiod robustoptimization for proactive resource provisioning in virtualizeddata centersrdquo Journal of Supercomputing vol 70 no 3 pp 1514ndash1536 2014

[29] G Chochlidakis and V Friderikos ldquoRobust virtual networkembedding for mobile networksrdquo in Proceedings of the 26thIEEE Annual International Symposium on Personal Indoor andMobile Radio Communications PIMRC rsquo15 pp 1867ndash1871 IEEEKowloon Hong Kong September 2015

[30] S Coniglio A M Koster and M Tieves ldquoVirtual networkembedding under uncertainty exact and heuristic approachesrdquoin Proceedings of the 11th International Conference on the Designof Reliable Communication Networks DRCN rsquo15 pp 1ndash8 IEEEKansas Kan USA March 2015

[31] D Bertsimas and M Sim ldquoThe price of robustnessrdquoOperationsResearch vol 52 no 1 pp 35ndash53 2004

[32] F Rossi P Van Beek and T Walsh Handbook of ConstraintProgramming Elsevier Amsterdam Netherland 2006

[33] httpwwwminizincorg[34] C Schulte G Tack and M Z Lagerkvist Modeling and pro-

gramming with gecode 2010 Schulte Christian and Tack Guidoand Lagerkvist Mikael

Mobile Information Systems 7

Virt

ual i

nfra

struc

ture

Corenetwork

Figure 4 Evaluation scenario

the solving time increases to hours depending on the specificconfiguration of the experiment

Next we first describe the analysis scenario and the con-figuration of the performed experiments to later discuss theresults of those experimentsThe objective of the experimentsis to demonstrate that the placement mechanism is able toallocate multitenant service chains to the available resourcesconsidering latency constraints For this reason the numeri-cal values employed in the next section are just experimentaland do not respond to real world measurements

51 Description of the Analysis Scenario Our evaluationscenario is depicted in Figure 4 We consider a 10-SC deploy-ment for example to cover a football stadium for periodicalflash eventsThe SCs are connected in a star topology throughan Ethernet switch Each SC is composed of 8 cores 16GBRAM 100GB of storage and 1HWA (eg GPU)

We also consider 2 NSs that can be offered at 3 serviceflavours (bronze silver and gold) that determine the latencyconstraint for eachNS and the nominal aggregate user bit rateserved as shown in Figure 5

Table 1 shows the experimental energy model and Table 2shows the modelling of the VNFs that compose the proposedNSs for the user traffic demand that will be employed in theevaluation experiments The user traffic values include thenominal bit rates of services NS1 and NS2 (130Mbps and90Mbps resp) as well as the values when a 20 protectionparameter is applied to the nominal values (156Mbps and108Mbps resp) in order to study the effect of robust optimi-zation techniques on the placement decision

As a final remark the placement behavior of VNFs canbe altered by the use of HWA The assignment of a HWA toa VNF implies that the service latency of the correspondentinstance is reduced 3 times at a cost of incrementing thepower consumption by 10 It is important to note that whenthe VNF instance needs more than 1 CPU for executingwithout the assistance of a HWA then if the placementalgorithm assigns it to a HWA the VNF will need a singlecore

52 Discussion of the Placement Results As stated before theevaluation scenario includes 3 tenants that hire a combination

8 Mobile Information Systems

VNF1 VNF2 VNF3 VNF5 VNF6

VNF1 VNF2 VNF3 VNF4

NS1

NS2

Service flavourBronze Silver Gold

NS1

NS2 Max Flow = 90MbpsMax Lat = 90ms Max Lat = 80ms

Max Flow = 130－＜ＪＭ

Max Lat = 150ＧＭ

Max Lat = 110ＧＭ

Max Flow = 90Mbps

Max Flow = 130－＜ＪＭ

Max Lat = 120ＧＭ

Max Flow = 90Mbps

Max Flow = 130－＜ＪＭ

Max Lat = 100ＧＭ

Figure 5 Description of NSs and service levels

Table 1 Energy models of the cores of the microservers and of the network switch related to the usage percentage

Usage 0 10 20 30 40 50 60 70 80 90 100119875router 0 1 2 5 7 9 11 13 14 15 15119875cpu 5 50 70 100 130 150 165 170 180 185 190

Table 2 VNF characterization for the service flavours of the evaluation scenario

VNF1 VNF2 VNF3 VNF4 VNF5 VNF6Number of cores per instance

90Mbps 1 2 1 1 1 1108Mbps 1 2 2 1 1 1130Mbps 1 2 2 1 1 1156Mbps 1 2 2 1 1 1

CPU usage90Mbps 30 20 90 50 50 20108Mbps 40 30 60 60 60 30130Mbps 40 30 60 60 60 30156Mbps 40 30 90 40 20 30

Service latency90Mbps 10ms 15ms 30ms 35ms 20ms 25ms108Mbps 10ms 15ms 35ms 40ms 25ms 30ms130Mbps 10ms 15ms 35ms 40ms 25ms 30ms156Mbps 12ms 16ms 40ms 50ms 30ms 35ms

of network servicesflavours to the SC operator In this wayin the following first we are going to evaluate the behavior ofthe placement algorithm for the three service levels and nextwe will show the effect of applying RO techniques to thoseresults

521 Bronze Flavour Scenario Results with 0 Robust Pro-tection Figure 6 shows the placement results of an scenariowhere all the network services operate at bronze flavourDifferent colours represent each tenant in order to dif-ferentiate the VNFs that belong to its NS and differenttones distinguish NSs of the same tenant As a result ofthe placement algorithm all the NSs are allocated in theavailable virtualized resources with energy consumption costof 3712W meeting all the latency constraints The figure

displays the assignment of VNFs to the SCs highlightingsome placement particularities As explained in Section 3the VNF chains of the same tenant share their SCVNF Inour example Tenant 1 and Tenant 2 both have two NSsthat are served with a single SCVNF Furthermore eachCPU executes a single VM but some VNF instances needtwo cores to manage the user demand In connection withthis behavior the VNF scalability feature is also visible inFigure 6 observing the placement of for example VNF3for the different NSs NS1 with a higher aggregated userdemand needs two CPUs to instantiate VNF3 but since NS2operates at a lower user demand the instantiation of VNF3needs just a single core Note as well that bronze serviceflavour does not require HW acceleration to meet the latencyconstraints of the NSs Finally it can also be observed that

Mobile Information Systems 9

Tenant 1 Tenant 2 Tenant 3

NS1 NS2 NS1 NS2 NS1

Service type Bronze Bronze Bronze Bronze Bronze

Max ow

Max latency

Deviation 0 0 0 0 0

Peak ow

VNF1VNF3

VNF1

VNF6

VNF1

SCVNF

VNF3VNF2

VNF1VNF3

VNF1VNF2

VNF2VNF3

VNF5

VNF5VNF2

VNF5

SCVNF

VNF1

VNF5

VNF3

VNF4

VNF5

VNF6

SCVNF

VNF1

VNF1VNF2

VNF3

VNF4

VNF1

960Mbps

960Mbps

780Mbps

780Mbps

780Mbps

260Mbps

VNF2

VNF6

CPU

s

Finallatency

Tenant 1NS1

NS2

Tenant 2NS1

NS2

Tenant 3 NS1

Total energy consumption

Switc

hH

A A

cc

132ＧＭ

102ＧＭ

132ＧＭ

102ＧＭ

130ＧＭ

3712７

130－＜ＪＭ 90－＜ＪＭ 130－＜ＪＭ 130－＜ＪＭ90－＜ＪＭ

130－＜ＪＭ 90－＜ＪＭ 130－＜ＪＭ 130－＜ＪＭ90－＜ＪＭ

150ＧＭ 110ＧＭ 150ＧＭ 110ＧＭ 150ＧＭ

Figure 6 Placement results of bronze flavour setting with 0 robust protection

the algorithm tries to group asmanyVNFs as possible in eachserver in order to reduce the data traffic across the switch andminimize its load and therefore its energy waste

522 Silver Flavour Scenario Results with 0 Robust Pro-tection The evaluation experiment analyzed in this sectionstudies the outcome of the placement algorithm when thelatency constraints become more strict with silver serviceflavour Due to the further limitation of maximum latencyvalues (20 stricter) the placement mechanism is compelledto use HW acceleration units Figure 7 shows that at thisservice level all the NSs must use HW acceleration in oneof the VNFs composing the service chain As a consequencethe resulting total latency of the NSs decreases to meet theservice level requirements but in return the global powerconsumption grows by 22 up to 4547W

Apart from that the placement results illustrate the samebehavior as the bronze flavoured example of Section 521regarding the allocation of VNFs in the available resourcesscale-inout features and multitenancy

The evaluation experiments of Sections 521 and 522operate with a nominal aggregated user traffic to perform theoptimization decision-making process However a perfectlyknown user traffic demand is seldom available Usuallythere is a certain level of uncertainty in the behavior of theuser demand We include robust optimization techniques to

allow the placement algorithm to insert a defined level ofuncertainty and protect the system against the undesirableeffects of demand peaksThe evaluation examples in Sections523 and 524 include a robust protection parameter andanalyze its effect on the global power consumption

523 Silver Flavour Scenario Results with 20 Robust Protec-tion This section studies the placement outcome of the eval-uation scenario described in the previous section introducinga 20 robust protection parameter This modification on thescenario configuration implies an increase of the 20 of theaggregated user traffic as an input of the decision-makingprocess in order to be prepared for potential demand peaksduring the service operation

Figure 8 exhibits that VNF3 of NS2 must scale as a con-sequence of the traffic growth and now needs two CPUs foreach instance The traffic increase also has an impact on theprocessing latency of the VNFs causing an adjustment in theuse of HW accelerators As a consequence some VNFs thatrequired 2CPUs for their operation in the previous evalu-ation scenario now only need a single core in combinationwith the HWA Therefore in some cases the higher powerconsumption that is implicit to the use of HWAs may bebalanced with a decrease on the number of CPUs required forthe VNF Besides the higher aggregated user traffic involvesa more intense use of the network switch These factors have

10 Mobile Information Systems

SCVNF

VNF6

VNF1VNF1

VNF2

VNF3

VNF3

VNF3

VNF5

VNF2

VNF1

VNF2

SCVNF

VNF1

VNF2

VNF4

VNF5

VNF6

SCVNF

VNF2

VNF3

VNF4

VNF5

VNF6

VNF3

VNF1

Finallatency

Tenant 1NS1

NS2

Tenant 2NS1

NS2

Tenant 3 NS1

Total energy consumption

VNF6VNF1VNF1VNF2VNF2

Tenant 1 Tenant 2 Tenant 3

NS1 NS2 NS1 NS2 NS1

Service type Silver Silver Silver Silver Silver

Max ow

Max latency

Deviation 0 0 0 0 0

Peak ow

960Mbps

960Mbps

780Mbps

780Mbps

260Mbps

780MbpsSw

itch

100ＧＭ

118ＧＭ

116ＧＭ

4547７

130－＜ＪＭ 90－＜ＪＭ 130－＜ＪＭ 130－＜ＪＭ90－＜ＪＭ

130－＜ＪＭ 90－＜ＪＭ 130－＜ＪＭ 130－＜ＪＭ90－＜ＪＭ

120ＧＭ 120ＧＭ 120ＧＭ

CPU

sH

A A

cc

88ＧＭ

88ＧＭ

90ＧＭ 90ＧＭ

Figure 7 Placement results of silver flavour setting with 0 robust protection

an impact on the power consumption of the componentsof the network resulting in a global energy consumption of5047W which implies an increment of 11 compared to theunprotected placement

524 Gold Flavour Scenario Results with 20 Robust Protec-tion Finally the last evaluation example sets even stricterlatency constraints to theNSs to be placed and includes a 20robust protection parameter Figure 9 depicts the results ofthe placement algorithm with this configuration Again thereduction of latency values forces the increment of the use ofHW acceleration In this case the main consequence is thespreading of the VNFs across the SCs to use the single HWAavailable in each cellThe use of HWAs the increase of powerconsumption due to higher user aggregated traffic and thehigher network traffic traversing the switch cause the globalenergy consumption to grow up to 5569W

As a final remark Figure 10 shows the comparison ofthe energy consumption and use of resources (cores activeCESCs and HWA) related to the robust protection level ina wider collection of scenarios In particular the charts ofFigure 10 compare the results of the placement algorithm forthe three service levels described in the previous sections(bronze silver and gold) combined with three robust protec-tion levels (0 or no protection 10 and 20) Obviouslythe main trend is that both power and resource consumptionincrease with higher values of protection but the associatedcost can be assumed in favor of reducing the impact of

unexpected user demand peaks However there are somecases in which the protection parameter does not have soadverse implications For example it can be observed howthe number of used cores in the silver service level with a10 protection level is lower than in the case of no robustprotection The reason of this behavior is how the algorithmuses HWA appliances in this case The increase of the usertraffic demand in the protected services implies the scaling upof someVNFs that nowmust usemore cores for the operationof theNSsThen the placement algorithm decides to useHWaccelerators for those specific VNFs compensating this waythe higher energy cost of HWAs with a lower use of CPUcores With all these considerations that can be extractedfrom a deeper analysis of robust placement results robustoptimization can be applied to decision-making problemsnot ending just with the optimization problem

The proposed steps in this paper can be used in anyplacement decision-making context and its goal is to providea tool that helps system administrator to evaluate the possibleimpact of any decision on the performance of the deploymentunderneath

6 Conclusions

This paper introduces a constraint-based expert system tosupport the decision-making process of VNF chain place-ment in distributed edge cloud networks The placement

Mobile Information Systems 11

SCVNF

VNF5

VNF6VNF5VNF2

VNF3

VNF5

VNF1

VNF2

VNF3

VNF1

VNF2

VNF3

VNF6

SCVNF

VNF1

VNF1

VNF3

VNF4

SCVNF

VNF1

VNF2

VNF3

VNF4

840Mbps

840Mbps

936Mbps

624Mbps

312Mbps

312Mbps

936Mbps

VNF6

Tenant 1NS1

NS2

Tenant 2NS1

NS2

Tenant 3 NS1

Total energy consumption

VNF5VNF6VNF5VNF1VNF4VNF4

VNF2

Finallatency

Tenant 1 Tenant 2 Tenant 3

NS1 NS2 NS1 NS2 NS1

Service type Silver Silver Silver Silver Silver

Max ow

Max latency

Deviation 20 20 20 20 20

Peak ow

116ＧＭ

110ＧＭ

118ＧＭ

5047７

130－＜ＪＭ 90－＜ＪＭ 130－＜ＪＭ 130－＜ＪＭ90－＜ＪＭ

156－＜ＪＭ 108－＜ＪＭ 156－＜ＪＭ 108－＜ＪＭ 156－＜ＪＭ

150ＧＭ 110ＧＭ 150ＧＭ 110ＧＭ 150ＧＭ

79ＧＭ

79ＧＭ

CPU

sH

A A

ccSw

itch

Figure 8 Placement results of silver flavour setting with 20 robust protection

algorithm applies robust optimization techniques to theminimization of the global power consumption of the systemsubject to the bit rate and latency constraints of the networkservices The power consumption of the components of thesystem depends on the usage percentage which is a con-sequence of the aggregated user traffic demand The modelassumes that the relationships between the user trafficdemand the power consumption and the use of the resourcesmay be nonlinear Besides the placement algorithmconsidersthe 5G communication particularities of both control anddata plane and allows multitenancy Finally the algorithmalso includes VNF scaling features and the use of hardwareacceleration

The results show that the placement algorithm succeeds inallocating theVNFs that compose theNSs of different tenantsin the available resourcesmeeting the latency constraints andminimizing the energy consumption of thewhole systemThepresented evaluation examples demonstrate the features ofthe constraint-based expert system and illustrate the impactof the selected service flavour and robust protection on theglobal power consumption and therefore the exploitationcost of the 5G distributed edge cloud

Further researchwill extend the characterization of VNFsto include variable output bit rate This feature will modelthe behavior of for example transcoding functions A future

version of the algorithm will also incorporate the existenceof a virtual switch to connect the VMs running in the samemicroserver Another enhancement of the model will bethe combination of heterogeneous SCs with different con-figurations and resource consumption models Finally theobtained placement results will be used for a technoeconomicstudy that will analyze the relationship between the serveraggregated user traffic cost of the required infrastructure andthe pricing of the services This study will also consider theintegration of the placement decision problem with the radioside

Summary of Model Indexes SetsParameters and Decision Variables

Indexes

120587 Index for VNF placement policy119909 Index for SC microserver119910 Index for core number in each microserver119894 Index for NS119895 Index for VNF in a network service chain119903 Index for physical resources in

SC microserver

12 Mobile Information Systems

VNF6VNF5VNF1VNF6VNF2SCVNF

VNF2

VNF3

VNF5

VNF3

VNF1

VNF2

VNF3

VNF5

SCVNF

VNF1

VNF1

VNF3

VNF4

SCVNF

VNF2

VNF3

VNF4

VNF1

Tenant 1NS1

NS2

Tenant 2NS1

NS2

Tenant 3 NS1

Total energy consumption

VNF6VNF5VNF1VNF6VNF2VNF5VNF6VNF4VNF4

VNF2

840Mbps

840Mbps

936Mbps

936Mbps

312Mbps

312Mbps

312Mbps

312Mbps

312Mbps

Finallatency

Tenant 1 Tenant 2 Tenant 3

NS1 NS2 NS1 NS2 NS1

Service type Gold Gold Gold Gold Gold

Max ow

Max latency

Deviation 20 20 20 20 20

Peak ow

5569７

130－＜ＪＭ 90－＜ＪＭ 130－＜ＪＭ 130－＜ＪＭ90－＜ＪＭ

156－＜ＪＭ 108－＜ＪＭ 156－＜ＪＭ 108－＜ＪＭ 156－＜ＪＭ

100ＧＭ 100ＧＭ 100ＧＭ

79ＧＭ

95ＧＭ

89ＧＭ

79ＧＭ

86ＧＭ

CPU

sH

A A

ccSw

itch

80ＧＭ 80ＧＭ

Figure 9 Placement results of gold flavour setting with 20 robust protection

34

32

30

28

260

10

20Robust protection ()

Num

ber o

f use

dco

res

10864200

10

20Robust protection ()

HW

AN

umbe

r of u

sed

0

10

20Robust protection ()

987654N

umbe

r of a

ctiv

eCE

SCs

6000

5500

5000

4500

4000

35003000

010

20

Ener

gy co

nsum

ptio

n(W

)

Robust protection ()

Gold

Silver

Bronze

Gold

Silver

Bronze

Gold

Silver

Bronze

Gold

Silver

Bronze

Figure 10 Comparison of energy and resource consumption for different levels of robust protection

Mobile Information Systems 13

Sets

Π Set of possible VNF placement policies119883 Set of SC microservers119884 Set of cores composing a SC microserver119868 Set of offered NSs119869119894 Set of VNFs in NS 119894119877 Set of resources available in

each SC microserver

Input Parameters

119879119894 Contracted maximum aggregated usertraffic for network service 119894

119863max119894 Maximum latency allowed for NS 119894119872119903119909 Amount of resource 119903 available in

SC microserver 119909119875sc119909 Power consumption of SC 119909 because of

being switched on119875cpu119909119910119894119895 Power consumption of VM 119910 of SC 119909 due

to CPU use of VNF 119895 of NS 119894119875switch119894119895 Power consumption in the physical switch

due to CPU use of VNF 119895 of NS 119894

Decision Variables

119875120587 Total power consumption ofplacement policy 120587

119875120587119894119895 Power consumption of VNF 119895 of NS 119894 inplacement policy 120587

119898120587119903119909 Consumption of resource 119903 in SCmicroserver 119909 in placement policy 120587

119905119897120587119909 Traffic of the link that connects SCmicroserver 119909 with the switchinplacement policy 120587

119860119909119910119894119895 Binary variable that expressesthe assignment of VNF 119895 of NS 119894 toVM 119910 in SC 119909

119871 119894119895 Binary variable to express the use of thenetwork switch to forward the flow ofVNF 119895 of NS 119894 to the next VNF

119889119894 Delay of NS 119894119889119894119895 Delay of VNF 119895 of NS 119894119889119901119909119910119894119895 Processing delay in the CPU of VNF 119895 of

NS 119894 in VM 119910 of SC 1199091198891198971199091119910111990921199102119894119895 Path delay for NS 119894 between VNF 119895

(located in VM 1199101 of SC 1199091) and VNF119895 + 1 (located in VM 1199102 of SC 1199092)

119889vs119909119910 Link delay between the virtual switch inSC 119909 and the VM 119910 in SC 119909

119889vs119909 119904 Link delay between the virtual switch inSC 119909 and the physical switch

119889vs119909 Delay in the virtual switch of SC 119909

119889119904 Delay in the physical switch

Conflicts of Interest

The authors declare that they have no conflicts of interest

Acknowledgments

The research leading to these results has been supportedby the EU funded H2020 5G-PPP Project SESAME (GrantAgreement 671596) and the Spanish MINECO Project5GRANVIR (TEC2016-80090-C2-2-R)

References

[1] ITU-R M2083 IMT Vision-Framework and overall objectivesof the future development of IMT for 2020 and beyond 2015

[2] P Demestichas A Georgakopoulos D Karvounas et al ldquo5G onthe Horizon key challenges for the radio-access networkrdquo IEEEVehicular Technology Magazine vol 8 no 3 pp 47ndash53 2013

[3] I F Akyildiz S Nie S-C Lin and M Chandrasekaran ldquo5Groadmap 10 key enabling technologiesrdquo Computer Networksvol 106 pp 17ndash48 2016

[4] ETSI Mobile-edge computing Introductory technical whitepaper 2014

[5] B Han V Gopalakrishnan L Ji and S Lee ldquoNetwork functionvirtualization challenges and opportunities for innovationsrdquoIEEE Communications Magazine vol 53 no 2 pp 90ndash97 2015

[6] R Mijumbi J Serrat J Gorricho N Bouten F De Turckand R Boutaba ldquoNetwork function virtualization state-of-the-art and research challengesrdquo IEEE Communications Surveys ampTutorials vol 18 no 1 pp 236ndash262 2016

[7] Ericsson White Paper Cloud-RANndash the benefits of virtualiza-tion centralization and coordination 2015

[8] B Blanco J O Fajardo I Giannoulakis et al ldquoTechnologypillars in the architecture of future 5G mobile networks NFVMEC and SDNrdquo Computer Standards and Interfaces vol 54 pp216ndash228 2017

[9] ETSI Network Functions Virtualization (NFV) Managementand Orchestration 2014

[10] A Ben-Tal L E Ghaoui and A Nemirovski Robust Optimiza-tion Aharon Ben-Tal Laurent El Ghaoui Arkadi Nemirovski-Google Books Princeton University Press Princeton NewJersey NY USA 2009

[11] J Gil Herrera and J F Botero ldquoResource Allocation in NFVA Comprehensive Surveyrdquo IEEE Transactions on Network andService Management vol 13 no 3 pp 518ndash532 2016

[12] B Addis D Belabed M Bouet and S Secci ldquoVirtual networkfunctions placement and routing optimizationrdquo in Proceedingsof the 4th IEEE International Conference on Cloud NetworkingCloudNet rsquo15 pp 171ndash177 October 2015

[13] H Moens and F De Turck ldquoVNF-P a model for efficientplacement of virtualized network functionsrdquo in Proceedingsof the 10th International Conference on Network and ServiceManagement (CNSM rsquo14) pp 418ndash423 Rio de Janeiro BrazilNovember 2014

[14] A Gupta M F Habib P Chowdhury M Tornatore and BMukherjee ldquoOn service chaining using Virtual Network Func-tions in Network-enabled Cloud systemsrdquo in Proceedings of the9th IEEE International Conference on Advanced Networks andTelecommuncations Systems ANTS rsquo15 pp 1ndash3 December 2015

[15] M Bouet J Leguay T Combe and V Conan ldquoCost-basedplacement of vDPI functions in NFV infrastructuresrdquo Interna-tional Journal of Network Management vol 25 no 6 pp 490ndash506 2015

[16] M C Luizelli L R Bays L S Buriol M P Barcellos andL P Gaspary ldquoPiecing together the NFV provisioning puzzle

14 Mobile Information Systems

efficient placement and chaining of virtual network functionsrdquoin Proceedings of the 14th International Symposium on IntegratedNetwork Management IM rsquo15 pp 98ndash106 IEEE TorontoCanada May 2015

[17] S Kim Y Han and S Park ldquoAn energy-Aware service functionchaining and reconfiguration algorithm in NFVrdquo in Proceedingsof the 1st International Workshops on Foundations and Applica-tions of Self-Systems FAS-W rsquo16 pp 54ndash59 September 2016

[18] V Eramo A Tosti and E Miucci ldquoServer resource dimen-sioning and routing of service function chain in NFV networkarchitecturesrdquo Journal of Electrical and Computer Engineeringvol 2016 Article ID 7139852 12 pages 2016

[19] K Hida and S-I Kuribayashi ldquoVirtual routing function allo-cation method for minimizing total network power consump-tionrdquo World Academy of Science Engineering and TechnologyInternational Journal of Electrical Computer Energetic Elec-tronic and Communication Engineering vol 10 pp 997ndash10022016

[20] C Pham N H Tran S Ren W Saad and C S Hong ldquoTraffic-aware and energy-efficient vnf placement for service chainingjoint sampling and matching approachrdquo IEEE Transactions onServices Computing 2017

[21] A Marotta and A Kassler ldquoA power efficient and robust virtualnetwork functions placement problemrdquo in Proceedings of the28th International Teletraffic Congress ITC rsquo16 pp 331ndash339September 2016

[22] C Pham H D Tran S I Moon K Thar and C S Hong ldquoAgeneral and practical consolidation framework in CloudNFVrdquoin Proceedings of the 2015 International Conference on Infor-mation Networking ICOIN rsquo15 pp 295ndash300 IEEE CambodiaCambodia January 2015

[23] V Eramo M Ammar and F G Lavacca ldquoMigration energyaware reconfigurations of virtual network function instances inNFV architecturesrdquo IEEE Access vol 5 pp 4927ndash4938 2017

[24] N E Khoury S Ayoubi and C Assi ldquoEnergy-aware placementand scheduling of network trafficflowswith deadlines on virtualnetwork functionsrdquo in Proceedings of the 5th IEEE InternationalConference on CloudNetworking CloudNet rsquo16 pp 89ndash94 IEEEPisa Italy October 2016

[25] V Eramo E Miucci M Ammar and F G Lavacca ldquoAnapproach for service function chain routing and virtual func-tion network instance migration in network function virtual-ization architecturesrdquo IEEEACM Transactions on Networking2017

[26] V Jumba S Parsaeefard M Derakhshani and T Le-NgocldquoEnergy-efficient robust resource provisioning in virtualizedwireless networksrdquo in Proceedings of the IEEE InternationalConference on Ubiquitous Wireless Broadband ICUWB rsquo15 pp1ndash5 IEEE Montreal Canada October 2015

[27] S Coniglio A Koster and M Tieves ldquoData uncertainty invirtual network embedding robust optimization and protectionlevelsrdquo Journal of Network and SystemsManagement vol 24 no3 pp 681ndash710 2016

[28] I Takouna K Sachs and C Meinel ldquoMultiperiod robustoptimization for proactive resource provisioning in virtualizeddata centersrdquo Journal of Supercomputing vol 70 no 3 pp 1514ndash1536 2014

[29] G Chochlidakis and V Friderikos ldquoRobust virtual networkembedding for mobile networksrdquo in Proceedings of the 26thIEEE Annual International Symposium on Personal Indoor andMobile Radio Communications PIMRC rsquo15 pp 1867ndash1871 IEEEKowloon Hong Kong September 2015

[30] S Coniglio A M Koster and M Tieves ldquoVirtual networkembedding under uncertainty exact and heuristic approachesrdquoin Proceedings of the 11th International Conference on the Designof Reliable Communication Networks DRCN rsquo15 pp 1ndash8 IEEEKansas Kan USA March 2015

[31] D Bertsimas and M Sim ldquoThe price of robustnessrdquoOperationsResearch vol 52 no 1 pp 35ndash53 2004

[32] F Rossi P Van Beek and T Walsh Handbook of ConstraintProgramming Elsevier Amsterdam Netherland 2006

[33] httpwwwminizincorg[34] C Schulte G Tack and M Z Lagerkvist Modeling and pro-

gramming with gecode 2010 Schulte Christian and Tack Guidoand Lagerkvist Mikael

8 Mobile Information Systems

VNF1 VNF2 VNF3 VNF5 VNF6

VNF1 VNF2 VNF3 VNF4

NS1

NS2

Service flavourBronze Silver Gold

NS1

NS2 Max Flow = 90MbpsMax Lat = 90ms Max Lat = 80ms

Max Flow = 130－＜ＪＭ

Max Lat = 150ＧＭ

Max Lat = 110ＧＭ

Max Flow = 90Mbps

Max Flow = 130－＜ＪＭ

Max Lat = 120ＧＭ

Max Flow = 90Mbps

Max Flow = 130－＜ＪＭ

Max Lat = 100ＧＭ

Figure 5 Description of NSs and service levels

Table 1 Energy models of the cores of the microservers and of the network switch related to the usage percentage

Usage 0 10 20 30 40 50 60 70 80 90 100119875router 0 1 2 5 7 9 11 13 14 15 15119875cpu 5 50 70 100 130 150 165 170 180 185 190

Table 2 VNF characterization for the service flavours of the evaluation scenario

VNF1 VNF2 VNF3 VNF4 VNF5 VNF6Number of cores per instance

90Mbps 1 2 1 1 1 1108Mbps 1 2 2 1 1 1130Mbps 1 2 2 1 1 1156Mbps 1 2 2 1 1 1

CPU usage90Mbps 30 20 90 50 50 20108Mbps 40 30 60 60 60 30130Mbps 40 30 60 60 60 30156Mbps 40 30 90 40 20 30

Service latency90Mbps 10ms 15ms 30ms 35ms 20ms 25ms108Mbps 10ms 15ms 35ms 40ms 25ms 30ms130Mbps 10ms 15ms 35ms 40ms 25ms 30ms156Mbps 12ms 16ms 40ms 50ms 30ms 35ms

of network servicesflavours to the SC operator In this wayin the following first we are going to evaluate the behavior ofthe placement algorithm for the three service levels and nextwe will show the effect of applying RO techniques to thoseresults

521 Bronze Flavour Scenario Results with 0 Robust Pro-tection Figure 6 shows the placement results of an scenariowhere all the network services operate at bronze flavourDifferent colours represent each tenant in order to dif-ferentiate the VNFs that belong to its NS and differenttones distinguish NSs of the same tenant As a result ofthe placement algorithm all the NSs are allocated in theavailable virtualized resources with energy consumption costof 3712W meeting all the latency constraints The figure

displays the assignment of VNFs to the SCs highlightingsome placement particularities As explained in Section 3the VNF chains of the same tenant share their SCVNF Inour example Tenant 1 and Tenant 2 both have two NSsthat are served with a single SCVNF Furthermore eachCPU executes a single VM but some VNF instances needtwo cores to manage the user demand In connection withthis behavior the VNF scalability feature is also visible inFigure 6 observing the placement of for example VNF3for the different NSs NS1 with a higher aggregated userdemand needs two CPUs to instantiate VNF3 but since NS2operates at a lower user demand the instantiation of VNF3needs just a single core Note as well that bronze serviceflavour does not require HW acceleration to meet the latencyconstraints of the NSs Finally it can also be observed that

Mobile Information Systems 9

Tenant 1 Tenant 2 Tenant 3

NS1 NS2 NS1 NS2 NS1

Service type Bronze Bronze Bronze Bronze Bronze

Max ow

Max latency

Deviation 0 0 0 0 0

Peak ow

VNF1VNF3

VNF1

VNF6

VNF1

SCVNF

VNF3VNF2

VNF1VNF3

VNF1VNF2

VNF2VNF3

VNF5

VNF5VNF2

VNF5

SCVNF

VNF1

VNF5

VNF3

VNF4

VNF5

VNF6

SCVNF

VNF1

VNF1VNF2

VNF3

VNF4

VNF1

960Mbps

960Mbps

780Mbps

780Mbps

780Mbps

260Mbps

VNF2

VNF6

CPU

s

Finallatency

Tenant 1NS1

NS2

Tenant 2NS1

NS2

Tenant 3 NS1

Total energy consumption

Switc

hH

A A

cc

132ＧＭ

102ＧＭ

132ＧＭ

102ＧＭ

130ＧＭ

3712７

130－＜ＪＭ 90－＜ＪＭ 130－＜ＪＭ 130－＜ＪＭ90－＜ＪＭ

130－＜ＪＭ 90－＜ＪＭ 130－＜ＪＭ 130－＜ＪＭ90－＜ＪＭ

150ＧＭ 110ＧＭ 150ＧＭ 110ＧＭ 150ＧＭ

Figure 6 Placement results of bronze flavour setting with 0 robust protection

the algorithm tries to group asmanyVNFs as possible in eachserver in order to reduce the data traffic across the switch andminimize its load and therefore its energy waste

522 Silver Flavour Scenario Results with 0 Robust Pro-tection The evaluation experiment analyzed in this sectionstudies the outcome of the placement algorithm when thelatency constraints become more strict with silver serviceflavour Due to the further limitation of maximum latencyvalues (20 stricter) the placement mechanism is compelledto use HW acceleration units Figure 7 shows that at thisservice level all the NSs must use HW acceleration in oneof the VNFs composing the service chain As a consequencethe resulting total latency of the NSs decreases to meet theservice level requirements but in return the global powerconsumption grows by 22 up to 4547W

Apart from that the placement results illustrate the samebehavior as the bronze flavoured example of Section 521regarding the allocation of VNFs in the available resourcesscale-inout features and multitenancy

The evaluation experiments of Sections 521 and 522operate with a nominal aggregated user traffic to perform theoptimization decision-making process However a perfectlyknown user traffic demand is seldom available Usuallythere is a certain level of uncertainty in the behavior of theuser demand We include robust optimization techniques to

allow the placement algorithm to insert a defined level ofuncertainty and protect the system against the undesirableeffects of demand peaksThe evaluation examples in Sections523 and 524 include a robust protection parameter andanalyze its effect on the global power consumption

523 Silver Flavour Scenario Results with 20 Robust Protec-tion This section studies the placement outcome of the eval-uation scenario described in the previous section introducinga 20 robust protection parameter This modification on thescenario configuration implies an increase of the 20 of theaggregated user traffic as an input of the decision-makingprocess in order to be prepared for potential demand peaksduring the service operation

Figure 8 exhibits that VNF3 of NS2 must scale as a con-sequence of the traffic growth and now needs two CPUs foreach instance The traffic increase also has an impact on theprocessing latency of the VNFs causing an adjustment in theuse of HW accelerators As a consequence some VNFs thatrequired 2CPUs for their operation in the previous evalu-ation scenario now only need a single core in combinationwith the HWA Therefore in some cases the higher powerconsumption that is implicit to the use of HWAs may bebalanced with a decrease on the number of CPUs required forthe VNF Besides the higher aggregated user traffic involvesa more intense use of the network switch These factors have

10 Mobile Information Systems

SCVNF

VNF6

VNF1VNF1

VNF2

VNF3

VNF3

VNF3

VNF5

VNF2

VNF1

VNF2

SCVNF

VNF1

VNF2

VNF4

VNF5

VNF6

SCVNF

VNF2

VNF3

VNF4

VNF5

VNF6

VNF3

VNF1

Finallatency

Tenant 1NS1

NS2

Tenant 2NS1

NS2

Tenant 3 NS1

Total energy consumption

VNF6VNF1VNF1VNF2VNF2

Tenant 1 Tenant 2 Tenant 3

NS1 NS2 NS1 NS2 NS1

Service type Silver Silver Silver Silver Silver

Max ow

Max latency

Deviation 0 0 0 0 0

Peak ow

960Mbps

960Mbps

780Mbps

780Mbps

260Mbps

780MbpsSw

itch

100ＧＭ

118ＧＭ

116ＧＭ

4547７

130－＜ＪＭ 90－＜ＪＭ 130－＜ＪＭ 130－＜ＪＭ90－＜ＪＭ

130－＜ＪＭ 90－＜ＪＭ 130－＜ＪＭ 130－＜ＪＭ90－＜ＪＭ

120ＧＭ 120ＧＭ 120ＧＭ

CPU

sH

A A

cc

88ＧＭ

88ＧＭ

90ＧＭ 90ＧＭ

Figure 7 Placement results of silver flavour setting with 0 robust protection

an impact on the power consumption of the componentsof the network resulting in a global energy consumption of5047W which implies an increment of 11 compared to theunprotected placement

524 Gold Flavour Scenario Results with 20 Robust Protec-tion Finally the last evaluation example sets even stricterlatency constraints to theNSs to be placed and includes a 20robust protection parameter Figure 9 depicts the results ofthe placement algorithm with this configuration Again thereduction of latency values forces the increment of the use ofHW acceleration In this case the main consequence is thespreading of the VNFs across the SCs to use the single HWAavailable in each cellThe use of HWAs the increase of powerconsumption due to higher user aggregated traffic and thehigher network traffic traversing the switch cause the globalenergy consumption to grow up to 5569W

As a final remark Figure 10 shows the comparison ofthe energy consumption and use of resources (cores activeCESCs and HWA) related to the robust protection level ina wider collection of scenarios In particular the charts ofFigure 10 compare the results of the placement algorithm forthe three service levels described in the previous sections(bronze silver and gold) combined with three robust protec-tion levels (0 or no protection 10 and 20) Obviouslythe main trend is that both power and resource consumptionincrease with higher values of protection but the associatedcost can be assumed in favor of reducing the impact of

unexpected user demand peaks However there are somecases in which the protection parameter does not have soadverse implications For example it can be observed howthe number of used cores in the silver service level with a10 protection level is lower than in the case of no robustprotection The reason of this behavior is how the algorithmuses HWA appliances in this case The increase of the usertraffic demand in the protected services implies the scaling upof someVNFs that nowmust usemore cores for the operationof theNSsThen the placement algorithm decides to useHWaccelerators for those specific VNFs compensating this waythe higher energy cost of HWAs with a lower use of CPUcores With all these considerations that can be extractedfrom a deeper analysis of robust placement results robustoptimization can be applied to decision-making problemsnot ending just with the optimization problem

The proposed steps in this paper can be used in anyplacement decision-making context and its goal is to providea tool that helps system administrator to evaluate the possibleimpact of any decision on the performance of the deploymentunderneath

6 Conclusions

This paper introduces a constraint-based expert system tosupport the decision-making process of VNF chain place-ment in distributed edge cloud networks The placement

Mobile Information Systems 11

SCVNF

VNF5

VNF6VNF5VNF2

VNF3

VNF5

VNF1

VNF2

VNF3

VNF1

VNF2

VNF3

VNF6

SCVNF

VNF1

VNF1

VNF3

VNF4

SCVNF

VNF1

VNF2

VNF3

VNF4

840Mbps

840Mbps

936Mbps

624Mbps

312Mbps

312Mbps

936Mbps

VNF6

Tenant 1NS1

NS2

Tenant 2NS1

NS2

Tenant 3 NS1

Total energy consumption

VNF5VNF6VNF5VNF1VNF4VNF4

VNF2

Finallatency

Tenant 1 Tenant 2 Tenant 3

NS1 NS2 NS1 NS2 NS1

Service type Silver Silver Silver Silver Silver

Max ow

Max latency

Deviation 20 20 20 20 20

Peak ow

116ＧＭ

110ＧＭ

118ＧＭ

5047７

130－＜ＪＭ 90－＜ＪＭ 130－＜ＪＭ 130－＜ＪＭ90－＜ＪＭ

156－＜ＪＭ 108－＜ＪＭ 156－＜ＪＭ 108－＜ＪＭ 156－＜ＪＭ

150ＧＭ 110ＧＭ 150ＧＭ 110ＧＭ 150ＧＭ

79ＧＭ

79ＧＭ

CPU

sH

A A

ccSw

itch

Figure 8 Placement results of silver flavour setting with 20 robust protection

algorithm applies robust optimization techniques to theminimization of the global power consumption of the systemsubject to the bit rate and latency constraints of the networkservices The power consumption of the components of thesystem depends on the usage percentage which is a con-sequence of the aggregated user traffic demand The modelassumes that the relationships between the user trafficdemand the power consumption and the use of the resourcesmay be nonlinear Besides the placement algorithmconsidersthe 5G communication particularities of both control anddata plane and allows multitenancy Finally the algorithmalso includes VNF scaling features and the use of hardwareacceleration

The results show that the placement algorithm succeeds inallocating theVNFs that compose theNSs of different tenantsin the available resourcesmeeting the latency constraints andminimizing the energy consumption of thewhole systemThepresented evaluation examples demonstrate the features ofthe constraint-based expert system and illustrate the impactof the selected service flavour and robust protection on theglobal power consumption and therefore the exploitationcost of the 5G distributed edge cloud

Further researchwill extend the characterization of VNFsto include variable output bit rate This feature will modelthe behavior of for example transcoding functions A future

version of the algorithm will also incorporate the existenceof a virtual switch to connect the VMs running in the samemicroserver Another enhancement of the model will bethe combination of heterogeneous SCs with different con-figurations and resource consumption models Finally theobtained placement results will be used for a technoeconomicstudy that will analyze the relationship between the serveraggregated user traffic cost of the required infrastructure andthe pricing of the services This study will also consider theintegration of the placement decision problem with the radioside

Summary of Model Indexes SetsParameters and Decision Variables

Indexes

120587 Index for VNF placement policy119909 Index for SC microserver119910 Index for core number in each microserver119894 Index for NS119895 Index for VNF in a network service chain119903 Index for physical resources in

SC microserver

12 Mobile Information Systems

VNF6VNF5VNF1VNF6VNF2SCVNF

VNF2

VNF3

VNF5

VNF3

VNF1

VNF2

VNF3

VNF5

SCVNF

VNF1

VNF1

VNF3

VNF4

SCVNF

VNF2

VNF3

VNF4

VNF1

Tenant 1NS1

NS2

Tenant 2NS1

NS2

Tenant 3 NS1

Total energy consumption

VNF6VNF5VNF1VNF6VNF2VNF5VNF6VNF4VNF4

VNF2

840Mbps

840Mbps

936Mbps

936Mbps

312Mbps

312Mbps

312Mbps

312Mbps

312Mbps

Finallatency

Tenant 1 Tenant 2 Tenant 3

NS1 NS2 NS1 NS2 NS1

Service type Gold Gold Gold Gold Gold

Max ow

Max latency

Deviation 20 20 20 20 20

Peak ow

5569７

130－＜ＪＭ 90－＜ＪＭ 130－＜ＪＭ 130－＜ＪＭ90－＜ＪＭ

156－＜ＪＭ 108－＜ＪＭ 156－＜ＪＭ 108－＜ＪＭ 156－＜ＪＭ

100ＧＭ 100ＧＭ 100ＧＭ

79ＧＭ

95ＧＭ

89ＧＭ

79ＧＭ

86ＧＭ

CPU

sH

A A

ccSw

itch

80ＧＭ 80ＧＭ

Figure 9 Placement results of gold flavour setting with 20 robust protection

34

32

30

28

260

10

20Robust protection ()

Num

ber o

f use

dco

res

10864200

10

20Robust protection ()

HW

AN

umbe

r of u

sed

0

10

20Robust protection ()

987654N

umbe

r of a

ctiv

eCE

SCs

6000

5500

5000

4500

4000

35003000

010

20

Ener

gy co

nsum

ptio

n(W

)

Robust protection ()

Gold

Silver

Bronze

Gold

Silver

Bronze

Gold

Silver

Bronze

Gold

Silver

Bronze

Figure 10 Comparison of energy and resource consumption for different levels of robust protection

Mobile Information Systems 13

Sets

Π Set of possible VNF placement policies119883 Set of SC microservers119884 Set of cores composing a SC microserver119868 Set of offered NSs119869119894 Set of VNFs in NS 119894119877 Set of resources available in

each SC microserver

Input Parameters

119879119894 Contracted maximum aggregated usertraffic for network service 119894

119863max119894 Maximum latency allowed for NS 119894119872119903119909 Amount of resource 119903 available in

SC microserver 119909119875sc119909 Power consumption of SC 119909 because of

being switched on119875cpu119909119910119894119895 Power consumption of VM 119910 of SC 119909 due

to CPU use of VNF 119895 of NS 119894119875switch119894119895 Power consumption in the physical switch

due to CPU use of VNF 119895 of NS 119894

Decision Variables

119875120587 Total power consumption ofplacement policy 120587

119875120587119894119895 Power consumption of VNF 119895 of NS 119894 inplacement policy 120587

119898120587119903119909 Consumption of resource 119903 in SCmicroserver 119909 in placement policy 120587

119905119897120587119909 Traffic of the link that connects SCmicroserver 119909 with the switchinplacement policy 120587

119860119909119910119894119895 Binary variable that expressesthe assignment of VNF 119895 of NS 119894 toVM 119910 in SC 119909

119871 119894119895 Binary variable to express the use of thenetwork switch to forward the flow ofVNF 119895 of NS 119894 to the next VNF

119889119894 Delay of NS 119894119889119894119895 Delay of VNF 119895 of NS 119894119889119901119909119910119894119895 Processing delay in the CPU of VNF 119895 of

NS 119894 in VM 119910 of SC 1199091198891198971199091119910111990921199102119894119895 Path delay for NS 119894 between VNF 119895

(located in VM 1199101 of SC 1199091) and VNF119895 + 1 (located in VM 1199102 of SC 1199092)

119889vs119909119910 Link delay between the virtual switch inSC 119909 and the VM 119910 in SC 119909

119889vs119909 119904 Link delay between the virtual switch inSC 119909 and the physical switch

119889vs119909 Delay in the virtual switch of SC 119909

119889119904 Delay in the physical switch

Conflicts of Interest

The authors declare that they have no conflicts of interest

Acknowledgments

The research leading to these results has been supportedby the EU funded H2020 5G-PPP Project SESAME (GrantAgreement 671596) and the Spanish MINECO Project5GRANVIR (TEC2016-80090-C2-2-R)

References

[1] ITU-R M2083 IMT Vision-Framework and overall objectivesof the future development of IMT for 2020 and beyond 2015

[2] P Demestichas A Georgakopoulos D Karvounas et al ldquo5G onthe Horizon key challenges for the radio-access networkrdquo IEEEVehicular Technology Magazine vol 8 no 3 pp 47ndash53 2013

[3] I F Akyildiz S Nie S-C Lin and M Chandrasekaran ldquo5Groadmap 10 key enabling technologiesrdquo Computer Networksvol 106 pp 17ndash48 2016

[4] ETSI Mobile-edge computing Introductory technical whitepaper 2014

[5] B Han V Gopalakrishnan L Ji and S Lee ldquoNetwork functionvirtualization challenges and opportunities for innovationsrdquoIEEE Communications Magazine vol 53 no 2 pp 90ndash97 2015

[6] R Mijumbi J Serrat J Gorricho N Bouten F De Turckand R Boutaba ldquoNetwork function virtualization state-of-the-art and research challengesrdquo IEEE Communications Surveys ampTutorials vol 18 no 1 pp 236ndash262 2016

[7] Ericsson White Paper Cloud-RANndash the benefits of virtualiza-tion centralization and coordination 2015

[8] B Blanco J O Fajardo I Giannoulakis et al ldquoTechnologypillars in the architecture of future 5G mobile networks NFVMEC and SDNrdquo Computer Standards and Interfaces vol 54 pp216ndash228 2017

[9] ETSI Network Functions Virtualization (NFV) Managementand Orchestration 2014

[10] A Ben-Tal L E Ghaoui and A Nemirovski Robust Optimiza-tion Aharon Ben-Tal Laurent El Ghaoui Arkadi Nemirovski-Google Books Princeton University Press Princeton NewJersey NY USA 2009

[11] J Gil Herrera and J F Botero ldquoResource Allocation in NFVA Comprehensive Surveyrdquo IEEE Transactions on Network andService Management vol 13 no 3 pp 518ndash532 2016

[12] B Addis D Belabed M Bouet and S Secci ldquoVirtual networkfunctions placement and routing optimizationrdquo in Proceedingsof the 4th IEEE International Conference on Cloud NetworkingCloudNet rsquo15 pp 171ndash177 October 2015

[13] H Moens and F De Turck ldquoVNF-P a model for efficientplacement of virtualized network functionsrdquo in Proceedingsof the 10th International Conference on Network and ServiceManagement (CNSM rsquo14) pp 418ndash423 Rio de Janeiro BrazilNovember 2014

[14] A Gupta M F Habib P Chowdhury M Tornatore and BMukherjee ldquoOn service chaining using Virtual Network Func-tions in Network-enabled Cloud systemsrdquo in Proceedings of the9th IEEE International Conference on Advanced Networks andTelecommuncations Systems ANTS rsquo15 pp 1ndash3 December 2015

[15] M Bouet J Leguay T Combe and V Conan ldquoCost-basedplacement of vDPI functions in NFV infrastructuresrdquo Interna-tional Journal of Network Management vol 25 no 6 pp 490ndash506 2015

[16] M C Luizelli L R Bays L S Buriol M P Barcellos andL P Gaspary ldquoPiecing together the NFV provisioning puzzle

14 Mobile Information Systems

efficient placement and chaining of virtual network functionsrdquoin Proceedings of the 14th International Symposium on IntegratedNetwork Management IM rsquo15 pp 98ndash106 IEEE TorontoCanada May 2015

[17] S Kim Y Han and S Park ldquoAn energy-Aware service functionchaining and reconfiguration algorithm in NFVrdquo in Proceedingsof the 1st International Workshops on Foundations and Applica-tions of Self-Systems FAS-W rsquo16 pp 54ndash59 September 2016

[18] V Eramo A Tosti and E Miucci ldquoServer resource dimen-sioning and routing of service function chain in NFV networkarchitecturesrdquo Journal of Electrical and Computer Engineeringvol 2016 Article ID 7139852 12 pages 2016

[19] K Hida and S-I Kuribayashi ldquoVirtual routing function allo-cation method for minimizing total network power consump-tionrdquo World Academy of Science Engineering and TechnologyInternational Journal of Electrical Computer Energetic Elec-tronic and Communication Engineering vol 10 pp 997ndash10022016

[20] C Pham N H Tran S Ren W Saad and C S Hong ldquoTraffic-aware and energy-efficient vnf placement for service chainingjoint sampling and matching approachrdquo IEEE Transactions onServices Computing 2017

[21] A Marotta and A Kassler ldquoA power efficient and robust virtualnetwork functions placement problemrdquo in Proceedings of the28th International Teletraffic Congress ITC rsquo16 pp 331ndash339September 2016

[22] C Pham H D Tran S I Moon K Thar and C S Hong ldquoAgeneral and practical consolidation framework in CloudNFVrdquoin Proceedings of the 2015 International Conference on Infor-mation Networking ICOIN rsquo15 pp 295ndash300 IEEE CambodiaCambodia January 2015

[23] V Eramo M Ammar and F G Lavacca ldquoMigration energyaware reconfigurations of virtual network function instances inNFV architecturesrdquo IEEE Access vol 5 pp 4927ndash4938 2017

[24] N E Khoury S Ayoubi and C Assi ldquoEnergy-aware placementand scheduling of network trafficflowswith deadlines on virtualnetwork functionsrdquo in Proceedings of the 5th IEEE InternationalConference on CloudNetworking CloudNet rsquo16 pp 89ndash94 IEEEPisa Italy October 2016

[25] V Eramo E Miucci M Ammar and F G Lavacca ldquoAnapproach for service function chain routing and virtual func-tion network instance migration in network function virtual-ization architecturesrdquo IEEEACM Transactions on Networking2017

[26] V Jumba S Parsaeefard M Derakhshani and T Le-NgocldquoEnergy-efficient robust resource provisioning in virtualizedwireless networksrdquo in Proceedings of the IEEE InternationalConference on Ubiquitous Wireless Broadband ICUWB rsquo15 pp1ndash5 IEEE Montreal Canada October 2015

[27] S Coniglio A Koster and M Tieves ldquoData uncertainty invirtual network embedding robust optimization and protectionlevelsrdquo Journal of Network and SystemsManagement vol 24 no3 pp 681ndash710 2016

[28] I Takouna K Sachs and C Meinel ldquoMultiperiod robustoptimization for proactive resource provisioning in virtualizeddata centersrdquo Journal of Supercomputing vol 70 no 3 pp 1514ndash1536 2014

[29] G Chochlidakis and V Friderikos ldquoRobust virtual networkembedding for mobile networksrdquo in Proceedings of the 26thIEEE Annual International Symposium on Personal Indoor andMobile Radio Communications PIMRC rsquo15 pp 1867ndash1871 IEEEKowloon Hong Kong September 2015

[30] S Coniglio A M Koster and M Tieves ldquoVirtual networkembedding under uncertainty exact and heuristic approachesrdquoin Proceedings of the 11th International Conference on the Designof Reliable Communication Networks DRCN rsquo15 pp 1ndash8 IEEEKansas Kan USA March 2015

[31] D Bertsimas and M Sim ldquoThe price of robustnessrdquoOperationsResearch vol 52 no 1 pp 35ndash53 2004

[32] F Rossi P Van Beek and T Walsh Handbook of ConstraintProgramming Elsevier Amsterdam Netherland 2006

[33] httpwwwminizincorg[34] C Schulte G Tack and M Z Lagerkvist Modeling and pro-

gramming with gecode 2010 Schulte Christian and Tack Guidoand Lagerkvist Mikael

Mobile Information Systems 9

Tenant 1 Tenant 2 Tenant 3

NS1 NS2 NS1 NS2 NS1

Service type Bronze Bronze Bronze Bronze Bronze

Max ow

Max latency

Deviation 0 0 0 0 0

Peak ow

VNF1VNF3

VNF1

VNF6

VNF1

SCVNF

VNF3VNF2

VNF1VNF3

VNF1VNF2

VNF2VNF3

VNF5

VNF5VNF2

VNF5

SCVNF

VNF1

VNF5

VNF3

VNF4

VNF5

VNF6

SCVNF

VNF1

VNF1VNF2

VNF3

VNF4

VNF1

960Mbps

960Mbps

780Mbps

780Mbps

780Mbps

260Mbps

VNF2

VNF6

CPU

s

Finallatency

Tenant 1NS1

NS2

Tenant 2NS1

NS2

Tenant 3 NS1

Total energy consumption

Switc

hH

A A

cc

132ＧＭ

102ＧＭ

132ＧＭ

102ＧＭ

130ＧＭ

3712７

130－＜ＪＭ 90－＜ＪＭ 130－＜ＪＭ 130－＜ＪＭ90－＜ＪＭ

130－＜ＪＭ 90－＜ＪＭ 130－＜ＪＭ 130－＜ＪＭ90－＜ＪＭ

150ＧＭ 110ＧＭ 150ＧＭ 110ＧＭ 150ＧＭ

Figure 6 Placement results of bronze flavour setting with 0 robust protection

the algorithm tries to group asmanyVNFs as possible in eachserver in order to reduce the data traffic across the switch andminimize its load and therefore its energy waste

522 Silver Flavour Scenario Results with 0 Robust Pro-tection The evaluation experiment analyzed in this sectionstudies the outcome of the placement algorithm when thelatency constraints become more strict with silver serviceflavour Due to the further limitation of maximum latencyvalues (20 stricter) the placement mechanism is compelledto use HW acceleration units Figure 7 shows that at thisservice level all the NSs must use HW acceleration in oneof the VNFs composing the service chain As a consequencethe resulting total latency of the NSs decreases to meet theservice level requirements but in return the global powerconsumption grows by 22 up to 4547W

Apart from that the placement results illustrate the samebehavior as the bronze flavoured example of Section 521regarding the allocation of VNFs in the available resourcesscale-inout features and multitenancy

The evaluation experiments of Sections 521 and 522operate with a nominal aggregated user traffic to perform theoptimization decision-making process However a perfectlyknown user traffic demand is seldom available Usuallythere is a certain level of uncertainty in the behavior of theuser demand We include robust optimization techniques to

allow the placement algorithm to insert a defined level ofuncertainty and protect the system against the undesirableeffects of demand peaksThe evaluation examples in Sections523 and 524 include a robust protection parameter andanalyze its effect on the global power consumption

523 Silver Flavour Scenario Results with 20 Robust Protec-tion This section studies the placement outcome of the eval-uation scenario described in the previous section introducinga 20 robust protection parameter This modification on thescenario configuration implies an increase of the 20 of theaggregated user traffic as an input of the decision-makingprocess in order to be prepared for potential demand peaksduring the service operation

Figure 8 exhibits that VNF3 of NS2 must scale as a con-sequence of the traffic growth and now needs two CPUs foreach instance The traffic increase also has an impact on theprocessing latency of the VNFs causing an adjustment in theuse of HW accelerators As a consequence some VNFs thatrequired 2CPUs for their operation in the previous evalu-ation scenario now only need a single core in combinationwith the HWA Therefore in some cases the higher powerconsumption that is implicit to the use of HWAs may bebalanced with a decrease on the number of CPUs required forthe VNF Besides the higher aggregated user traffic involvesa more intense use of the network switch These factors have

10 Mobile Information Systems

SCVNF

VNF6

VNF1VNF1

VNF2

VNF3

VNF3

VNF3

VNF5

VNF2

VNF1

VNF2

SCVNF

VNF1

VNF2

VNF4

VNF5

VNF6

SCVNF

VNF2

VNF3

VNF4

VNF5

VNF6

VNF3

VNF1

Finallatency

Tenant 1NS1

NS2

Tenant 2NS1

NS2

Tenant 3 NS1

Total energy consumption

VNF6VNF1VNF1VNF2VNF2

Tenant 1 Tenant 2 Tenant 3

NS1 NS2 NS1 NS2 NS1

Service type Silver Silver Silver Silver Silver

Max ow

Max latency

Deviation 0 0 0 0 0

Peak ow

960Mbps

960Mbps

780Mbps

780Mbps

260Mbps

780MbpsSw

itch

100ＧＭ

118ＧＭ

116ＧＭ

4547７

130－＜ＪＭ 90－＜ＪＭ 130－＜ＪＭ 130－＜ＪＭ90－＜ＪＭ

130－＜ＪＭ 90－＜ＪＭ 130－＜ＪＭ 130－＜ＪＭ90－＜ＪＭ

120ＧＭ 120ＧＭ 120ＧＭ

CPU

sH

A A

cc

88ＧＭ

88ＧＭ

90ＧＭ 90ＧＭ

Figure 7 Placement results of silver flavour setting with 0 robust protection

an impact on the power consumption of the componentsof the network resulting in a global energy consumption of5047W which implies an increment of 11 compared to theunprotected placement

524 Gold Flavour Scenario Results with 20 Robust Protec-tion Finally the last evaluation example sets even stricterlatency constraints to theNSs to be placed and includes a 20robust protection parameter Figure 9 depicts the results ofthe placement algorithm with this configuration Again thereduction of latency values forces the increment of the use ofHW acceleration In this case the main consequence is thespreading of the VNFs across the SCs to use the single HWAavailable in each cellThe use of HWAs the increase of powerconsumption due to higher user aggregated traffic and thehigher network traffic traversing the switch cause the globalenergy consumption to grow up to 5569W

As a final remark Figure 10 shows the comparison ofthe energy consumption and use of resources (cores activeCESCs and HWA) related to the robust protection level ina wider collection of scenarios In particular the charts ofFigure 10 compare the results of the placement algorithm forthe three service levels described in the previous sections(bronze silver and gold) combined with three robust protec-tion levels (0 or no protection 10 and 20) Obviouslythe main trend is that both power and resource consumptionincrease with higher values of protection but the associatedcost can be assumed in favor of reducing the impact of

unexpected user demand peaks However there are somecases in which the protection parameter does not have soadverse implications For example it can be observed howthe number of used cores in the silver service level with a10 protection level is lower than in the case of no robustprotection The reason of this behavior is how the algorithmuses HWA appliances in this case The increase of the usertraffic demand in the protected services implies the scaling upof someVNFs that nowmust usemore cores for the operationof theNSsThen the placement algorithm decides to useHWaccelerators for those specific VNFs compensating this waythe higher energy cost of HWAs with a lower use of CPUcores With all these considerations that can be extractedfrom a deeper analysis of robust placement results robustoptimization can be applied to decision-making problemsnot ending just with the optimization problem

The proposed steps in this paper can be used in anyplacement decision-making context and its goal is to providea tool that helps system administrator to evaluate the possibleimpact of any decision on the performance of the deploymentunderneath

6 Conclusions

This paper introduces a constraint-based expert system tosupport the decision-making process of VNF chain place-ment in distributed edge cloud networks The placement

Mobile Information Systems 11

SCVNF

VNF5

VNF6VNF5VNF2

VNF3

VNF5

VNF1

VNF2

VNF3

VNF1

VNF2

VNF3

VNF6

SCVNF

VNF1

VNF1

VNF3

VNF4

SCVNF

VNF1

VNF2

VNF3

VNF4

840Mbps

840Mbps

936Mbps

624Mbps

312Mbps

312Mbps

936Mbps

VNF6

Tenant 1NS1

NS2

Tenant 2NS1

NS2

Tenant 3 NS1

Total energy consumption

VNF5VNF6VNF5VNF1VNF4VNF4

VNF2

Finallatency

Tenant 1 Tenant 2 Tenant 3

NS1 NS2 NS1 NS2 NS1

Service type Silver Silver Silver Silver Silver

Max ow

Max latency

Deviation 20 20 20 20 20

Peak ow

116ＧＭ

110ＧＭ

118ＧＭ

5047７

130－＜ＪＭ 90－＜ＪＭ 130－＜ＪＭ 130－＜ＪＭ90－＜ＪＭ

156－＜ＪＭ 108－＜ＪＭ 156－＜ＪＭ 108－＜ＪＭ 156－＜ＪＭ

150ＧＭ 110ＧＭ 150ＧＭ 110ＧＭ 150ＧＭ

79ＧＭ

79ＧＭ

CPU

sH

A A

ccSw

itch

Figure 8 Placement results of silver flavour setting with 20 robust protection

algorithm applies robust optimization techniques to theminimization of the global power consumption of the systemsubject to the bit rate and latency constraints of the networkservices The power consumption of the components of thesystem depends on the usage percentage which is a con-sequence of the aggregated user traffic demand The modelassumes that the relationships between the user trafficdemand the power consumption and the use of the resourcesmay be nonlinear Besides the placement algorithmconsidersthe 5G communication particularities of both control anddata plane and allows multitenancy Finally the algorithmalso includes VNF scaling features and the use of hardwareacceleration

The results show that the placement algorithm succeeds inallocating theVNFs that compose theNSs of different tenantsin the available resourcesmeeting the latency constraints andminimizing the energy consumption of thewhole systemThepresented evaluation examples demonstrate the features ofthe constraint-based expert system and illustrate the impactof the selected service flavour and robust protection on theglobal power consumption and therefore the exploitationcost of the 5G distributed edge cloud

Further researchwill extend the characterization of VNFsto include variable output bit rate This feature will modelthe behavior of for example transcoding functions A future

version of the algorithm will also incorporate the existenceof a virtual switch to connect the VMs running in the samemicroserver Another enhancement of the model will bethe combination of heterogeneous SCs with different con-figurations and resource consumption models Finally theobtained placement results will be used for a technoeconomicstudy that will analyze the relationship between the serveraggregated user traffic cost of the required infrastructure andthe pricing of the services This study will also consider theintegration of the placement decision problem with the radioside

Summary of Model Indexes SetsParameters and Decision Variables

Indexes

120587 Index for VNF placement policy119909 Index for SC microserver119910 Index for core number in each microserver119894 Index for NS119895 Index for VNF in a network service chain119903 Index for physical resources in

SC microserver

12 Mobile Information Systems

VNF6VNF5VNF1VNF6VNF2SCVNF

VNF2

VNF3

VNF5

VNF3

VNF1

VNF2

VNF3

VNF5

SCVNF

VNF1

VNF1

VNF3

VNF4

SCVNF

VNF2

VNF3

VNF4

VNF1

Tenant 1NS1

NS2

Tenant 2NS1

NS2

Tenant 3 NS1

Total energy consumption

VNF6VNF5VNF1VNF6VNF2VNF5VNF6VNF4VNF4

VNF2

840Mbps

840Mbps

936Mbps

936Mbps

312Mbps

312Mbps

312Mbps

312Mbps

312Mbps

Finallatency

Tenant 1 Tenant 2 Tenant 3

NS1 NS2 NS1 NS2 NS1

Service type Gold Gold Gold Gold Gold

Max ow

Max latency

Deviation 20 20 20 20 20

Peak ow

5569７

130－＜ＪＭ 90－＜ＪＭ 130－＜ＪＭ 130－＜ＪＭ90－＜ＪＭ

156－＜ＪＭ 108－＜ＪＭ 156－＜ＪＭ 108－＜ＪＭ 156－＜ＪＭ

100ＧＭ 100ＧＭ 100ＧＭ

79ＧＭ

95ＧＭ

89ＧＭ

79ＧＭ

86ＧＭ

CPU

sH

A A

ccSw

itch

80ＧＭ 80ＧＭ

Figure 9 Placement results of gold flavour setting with 20 robust protection

34

32

30

28

260

10

20Robust protection ()

Num

ber o

f use

dco

res

10864200

10

20Robust protection ()

HW

AN

umbe

r of u

sed

0

10

20Robust protection ()

987654N

umbe

r of a

ctiv

eCE

SCs

6000

5500

5000

4500

4000

35003000

010

20

Ener

gy co

nsum

ptio

n(W

)

Robust protection ()

Gold

Silver

Bronze

Gold

Silver

Bronze

Gold

Silver

Bronze

Gold

Silver

Bronze

Figure 10 Comparison of energy and resource consumption for different levels of robust protection

Mobile Information Systems 13

Sets

Π Set of possible VNF placement policies119883 Set of SC microservers119884 Set of cores composing a SC microserver119868 Set of offered NSs119869119894 Set of VNFs in NS 119894119877 Set of resources available in

each SC microserver

Input Parameters

119879119894 Contracted maximum aggregated usertraffic for network service 119894

119863max119894 Maximum latency allowed for NS 119894119872119903119909 Amount of resource 119903 available in

SC microserver 119909119875sc119909 Power consumption of SC 119909 because of

being switched on119875cpu119909119910119894119895 Power consumption of VM 119910 of SC 119909 due

to CPU use of VNF 119895 of NS 119894119875switch119894119895 Power consumption in the physical switch

due to CPU use of VNF 119895 of NS 119894

Decision Variables

119875120587 Total power consumption ofplacement policy 120587

119875120587119894119895 Power consumption of VNF 119895 of NS 119894 inplacement policy 120587

119898120587119903119909 Consumption of resource 119903 in SCmicroserver 119909 in placement policy 120587

119905119897120587119909 Traffic of the link that connects SCmicroserver 119909 with the switchinplacement policy 120587

119860119909119910119894119895 Binary variable that expressesthe assignment of VNF 119895 of NS 119894 toVM 119910 in SC 119909

119871 119894119895 Binary variable to express the use of thenetwork switch to forward the flow ofVNF 119895 of NS 119894 to the next VNF

119889119894 Delay of NS 119894119889119894119895 Delay of VNF 119895 of NS 119894119889119901119909119910119894119895 Processing delay in the CPU of VNF 119895 of

NS 119894 in VM 119910 of SC 1199091198891198971199091119910111990921199102119894119895 Path delay for NS 119894 between VNF 119895

(located in VM 1199101 of SC 1199091) and VNF119895 + 1 (located in VM 1199102 of SC 1199092)

119889vs119909119910 Link delay between the virtual switch inSC 119909 and the VM 119910 in SC 119909

119889vs119909 119904 Link delay between the virtual switch inSC 119909 and the physical switch

119889vs119909 Delay in the virtual switch of SC 119909

119889119904 Delay in the physical switch

Conflicts of Interest

The authors declare that they have no conflicts of interest

Acknowledgments

The research leading to these results has been supportedby the EU funded H2020 5G-PPP Project SESAME (GrantAgreement 671596) and the Spanish MINECO Project5GRANVIR (TEC2016-80090-C2-2-R)

References

[1] ITU-R M2083 IMT Vision-Framework and overall objectivesof the future development of IMT for 2020 and beyond 2015

[2] P Demestichas A Georgakopoulos D Karvounas et al ldquo5G onthe Horizon key challenges for the radio-access networkrdquo IEEEVehicular Technology Magazine vol 8 no 3 pp 47ndash53 2013

[3] I F Akyildiz S Nie S-C Lin and M Chandrasekaran ldquo5Groadmap 10 key enabling technologiesrdquo Computer Networksvol 106 pp 17ndash48 2016

[4] ETSI Mobile-edge computing Introductory technical whitepaper 2014

[5] B Han V Gopalakrishnan L Ji and S Lee ldquoNetwork functionvirtualization challenges and opportunities for innovationsrdquoIEEE Communications Magazine vol 53 no 2 pp 90ndash97 2015

[6] R Mijumbi J Serrat J Gorricho N Bouten F De Turckand R Boutaba ldquoNetwork function virtualization state-of-the-art and research challengesrdquo IEEE Communications Surveys ampTutorials vol 18 no 1 pp 236ndash262 2016

[7] Ericsson White Paper Cloud-RANndash the benefits of virtualiza-tion centralization and coordination 2015

[8] B Blanco J O Fajardo I Giannoulakis et al ldquoTechnologypillars in the architecture of future 5G mobile networks NFVMEC and SDNrdquo Computer Standards and Interfaces vol 54 pp216ndash228 2017

[9] ETSI Network Functions Virtualization (NFV) Managementand Orchestration 2014

[10] A Ben-Tal L E Ghaoui and A Nemirovski Robust Optimiza-tion Aharon Ben-Tal Laurent El Ghaoui Arkadi Nemirovski-Google Books Princeton University Press Princeton NewJersey NY USA 2009

[11] J Gil Herrera and J F Botero ldquoResource Allocation in NFVA Comprehensive Surveyrdquo IEEE Transactions on Network andService Management vol 13 no 3 pp 518ndash532 2016

[12] B Addis D Belabed M Bouet and S Secci ldquoVirtual networkfunctions placement and routing optimizationrdquo in Proceedingsof the 4th IEEE International Conference on Cloud NetworkingCloudNet rsquo15 pp 171ndash177 October 2015

[13] H Moens and F De Turck ldquoVNF-P a model for efficientplacement of virtualized network functionsrdquo in Proceedingsof the 10th International Conference on Network and ServiceManagement (CNSM rsquo14) pp 418ndash423 Rio de Janeiro BrazilNovember 2014

[14] A Gupta M F Habib P Chowdhury M Tornatore and BMukherjee ldquoOn service chaining using Virtual Network Func-tions in Network-enabled Cloud systemsrdquo in Proceedings of the9th IEEE International Conference on Advanced Networks andTelecommuncations Systems ANTS rsquo15 pp 1ndash3 December 2015

[15] M Bouet J Leguay T Combe and V Conan ldquoCost-basedplacement of vDPI functions in NFV infrastructuresrdquo Interna-tional Journal of Network Management vol 25 no 6 pp 490ndash506 2015

[16] M C Luizelli L R Bays L S Buriol M P Barcellos andL P Gaspary ldquoPiecing together the NFV provisioning puzzle

14 Mobile Information Systems

efficient placement and chaining of virtual network functionsrdquoin Proceedings of the 14th International Symposium on IntegratedNetwork Management IM rsquo15 pp 98ndash106 IEEE TorontoCanada May 2015

[17] S Kim Y Han and S Park ldquoAn energy-Aware service functionchaining and reconfiguration algorithm in NFVrdquo in Proceedingsof the 1st International Workshops on Foundations and Applica-tions of Self-Systems FAS-W rsquo16 pp 54ndash59 September 2016

[18] V Eramo A Tosti and E Miucci ldquoServer resource dimen-sioning and routing of service function chain in NFV networkarchitecturesrdquo Journal of Electrical and Computer Engineeringvol 2016 Article ID 7139852 12 pages 2016

[19] K Hida and S-I Kuribayashi ldquoVirtual routing function allo-cation method for minimizing total network power consump-tionrdquo World Academy of Science Engineering and TechnologyInternational Journal of Electrical Computer Energetic Elec-tronic and Communication Engineering vol 10 pp 997ndash10022016

[20] C Pham N H Tran S Ren W Saad and C S Hong ldquoTraffic-aware and energy-efficient vnf placement for service chainingjoint sampling and matching approachrdquo IEEE Transactions onServices Computing 2017

[21] A Marotta and A Kassler ldquoA power efficient and robust virtualnetwork functions placement problemrdquo in Proceedings of the28th International Teletraffic Congress ITC rsquo16 pp 331ndash339September 2016

[22] C Pham H D Tran S I Moon K Thar and C S Hong ldquoAgeneral and practical consolidation framework in CloudNFVrdquoin Proceedings of the 2015 International Conference on Infor-mation Networking ICOIN rsquo15 pp 295ndash300 IEEE CambodiaCambodia January 2015

[23] V Eramo M Ammar and F G Lavacca ldquoMigration energyaware reconfigurations of virtual network function instances inNFV architecturesrdquo IEEE Access vol 5 pp 4927ndash4938 2017

[24] N E Khoury S Ayoubi and C Assi ldquoEnergy-aware placementand scheduling of network trafficflowswith deadlines on virtualnetwork functionsrdquo in Proceedings of the 5th IEEE InternationalConference on CloudNetworking CloudNet rsquo16 pp 89ndash94 IEEEPisa Italy October 2016

[25] V Eramo E Miucci M Ammar and F G Lavacca ldquoAnapproach for service function chain routing and virtual func-tion network instance migration in network function virtual-ization architecturesrdquo IEEEACM Transactions on Networking2017

[26] V Jumba S Parsaeefard M Derakhshani and T Le-NgocldquoEnergy-efficient robust resource provisioning in virtualizedwireless networksrdquo in Proceedings of the IEEE InternationalConference on Ubiquitous Wireless Broadband ICUWB rsquo15 pp1ndash5 IEEE Montreal Canada October 2015

[27] S Coniglio A Koster and M Tieves ldquoData uncertainty invirtual network embedding robust optimization and protectionlevelsrdquo Journal of Network and SystemsManagement vol 24 no3 pp 681ndash710 2016

[28] I Takouna K Sachs and C Meinel ldquoMultiperiod robustoptimization for proactive resource provisioning in virtualizeddata centersrdquo Journal of Supercomputing vol 70 no 3 pp 1514ndash1536 2014

[29] G Chochlidakis and V Friderikos ldquoRobust virtual networkembedding for mobile networksrdquo in Proceedings of the 26thIEEE Annual International Symposium on Personal Indoor andMobile Radio Communications PIMRC rsquo15 pp 1867ndash1871 IEEEKowloon Hong Kong September 2015

[30] S Coniglio A M Koster and M Tieves ldquoVirtual networkembedding under uncertainty exact and heuristic approachesrdquoin Proceedings of the 11th International Conference on the Designof Reliable Communication Networks DRCN rsquo15 pp 1ndash8 IEEEKansas Kan USA March 2015

[31] D Bertsimas and M Sim ldquoThe price of robustnessrdquoOperationsResearch vol 52 no 1 pp 35ndash53 2004

[32] F Rossi P Van Beek and T Walsh Handbook of ConstraintProgramming Elsevier Amsterdam Netherland 2006

[33] httpwwwminizincorg[34] C Schulte G Tack and M Z Lagerkvist Modeling and pro-

gramming with gecode 2010 Schulte Christian and Tack Guidoand Lagerkvist Mikael

10 Mobile Information Systems

SCVNF

VNF6

VNF1VNF1

VNF2

VNF3

VNF3

VNF3

VNF5

VNF2

VNF1

VNF2

SCVNF

VNF1

VNF2

VNF4

VNF5

VNF6

SCVNF

VNF2

VNF3

VNF4

VNF5

VNF6

VNF3

VNF1

Finallatency

Tenant 1NS1

NS2

Tenant 2NS1

NS2

Tenant 3 NS1

Total energy consumption

VNF6VNF1VNF1VNF2VNF2

Tenant 1 Tenant 2 Tenant 3

NS1 NS2 NS1 NS2 NS1

Service type Silver Silver Silver Silver Silver

Max ow

Max latency

Deviation 0 0 0 0 0

Peak ow

960Mbps

960Mbps

780Mbps

780Mbps

260Mbps

780MbpsSw

itch

100ＧＭ

118ＧＭ

116ＧＭ

4547７

130－＜ＪＭ 90－＜ＪＭ 130－＜ＪＭ 130－＜ＪＭ90－＜ＪＭ

130－＜ＪＭ 90－＜ＪＭ 130－＜ＪＭ 130－＜ＪＭ90－＜ＪＭ

120ＧＭ 120ＧＭ 120ＧＭ

CPU

sH

A A

cc

88ＧＭ

88ＧＭ

90ＧＭ 90ＧＭ

Figure 7 Placement results of silver flavour setting with 0 robust protection

an impact on the power consumption of the componentsof the network resulting in a global energy consumption of5047W which implies an increment of 11 compared to theunprotected placement

524 Gold Flavour Scenario Results with 20 Robust Protec-tion Finally the last evaluation example sets even stricterlatency constraints to theNSs to be placed and includes a 20robust protection parameter Figure 9 depicts the results ofthe placement algorithm with this configuration Again thereduction of latency values forces the increment of the use ofHW acceleration In this case the main consequence is thespreading of the VNFs across the SCs to use the single HWAavailable in each cellThe use of HWAs the increase of powerconsumption due to higher user aggregated traffic and thehigher network traffic traversing the switch cause the globalenergy consumption to grow up to 5569W

As a final remark Figure 10 shows the comparison ofthe energy consumption and use of resources (cores activeCESCs and HWA) related to the robust protection level ina wider collection of scenarios In particular the charts ofFigure 10 compare the results of the placement algorithm forthe three service levels described in the previous sections(bronze silver and gold) combined with three robust protec-tion levels (0 or no protection 10 and 20) Obviouslythe main trend is that both power and resource consumptionincrease with higher values of protection but the associatedcost can be assumed in favor of reducing the impact of

unexpected user demand peaks However there are somecases in which the protection parameter does not have soadverse implications For example it can be observed howthe number of used cores in the silver service level with a10 protection level is lower than in the case of no robustprotection The reason of this behavior is how the algorithmuses HWA appliances in this case The increase of the usertraffic demand in the protected services implies the scaling upof someVNFs that nowmust usemore cores for the operationof theNSsThen the placement algorithm decides to useHWaccelerators for those specific VNFs compensating this waythe higher energy cost of HWAs with a lower use of CPUcores With all these considerations that can be extractedfrom a deeper analysis of robust placement results robustoptimization can be applied to decision-making problemsnot ending just with the optimization problem

The proposed steps in this paper can be used in anyplacement decision-making context and its goal is to providea tool that helps system administrator to evaluate the possibleimpact of any decision on the performance of the deploymentunderneath

6 Conclusions

This paper introduces a constraint-based expert system tosupport the decision-making process of VNF chain place-ment in distributed edge cloud networks The placement

Mobile Information Systems 11

SCVNF

VNF5

VNF6VNF5VNF2

VNF3

VNF5

VNF1

VNF2

VNF3

VNF1

VNF2

VNF3

VNF6

SCVNF

VNF1

VNF1

VNF3

VNF4

SCVNF

VNF1

VNF2

VNF3

VNF4

840Mbps

840Mbps

936Mbps

624Mbps

312Mbps

312Mbps

936Mbps

VNF6

Tenant 1NS1

NS2

Tenant 2NS1

NS2

Tenant 3 NS1

Total energy consumption

VNF5VNF6VNF5VNF1VNF4VNF4

VNF2

Finallatency

Tenant 1 Tenant 2 Tenant 3

NS1 NS2 NS1 NS2 NS1

Service type Silver Silver Silver Silver Silver

Max ow

Max latency

Deviation 20 20 20 20 20

Peak ow

116ＧＭ

110ＧＭ

118ＧＭ

5047７

130－＜ＪＭ 90－＜ＪＭ 130－＜ＪＭ 130－＜ＪＭ90－＜ＪＭ

156－＜ＪＭ 108－＜ＪＭ 156－＜ＪＭ 108－＜ＪＭ 156－＜ＪＭ

150ＧＭ 110ＧＭ 150ＧＭ 110ＧＭ 150ＧＭ

79ＧＭ

79ＧＭ

CPU

sH

A A

ccSw

itch

Figure 8 Placement results of silver flavour setting with 20 robust protection

algorithm applies robust optimization techniques to theminimization of the global power consumption of the systemsubject to the bit rate and latency constraints of the networkservices The power consumption of the components of thesystem depends on the usage percentage which is a con-sequence of the aggregated user traffic demand The modelassumes that the relationships between the user trafficdemand the power consumption and the use of the resourcesmay be nonlinear Besides the placement algorithmconsidersthe 5G communication particularities of both control anddata plane and allows multitenancy Finally the algorithmalso includes VNF scaling features and the use of hardwareacceleration

The results show that the placement algorithm succeeds inallocating theVNFs that compose theNSs of different tenantsin the available resourcesmeeting the latency constraints andminimizing the energy consumption of thewhole systemThepresented evaluation examples demonstrate the features ofthe constraint-based expert system and illustrate the impactof the selected service flavour and robust protection on theglobal power consumption and therefore the exploitationcost of the 5G distributed edge cloud

Further researchwill extend the characterization of VNFsto include variable output bit rate This feature will modelthe behavior of for example transcoding functions A future

version of the algorithm will also incorporate the existenceof a virtual switch to connect the VMs running in the samemicroserver Another enhancement of the model will bethe combination of heterogeneous SCs with different con-figurations and resource consumption models Finally theobtained placement results will be used for a technoeconomicstudy that will analyze the relationship between the serveraggregated user traffic cost of the required infrastructure andthe pricing of the services This study will also consider theintegration of the placement decision problem with the radioside

Summary of Model Indexes SetsParameters and Decision Variables

Indexes

120587 Index for VNF placement policy119909 Index for SC microserver119910 Index for core number in each microserver119894 Index for NS119895 Index for VNF in a network service chain119903 Index for physical resources in

SC microserver

12 Mobile Information Systems

VNF6VNF5VNF1VNF6VNF2SCVNF

VNF2

VNF3

VNF5

VNF3

VNF1

VNF2

VNF3

VNF5

SCVNF

VNF1

VNF1

VNF3

VNF4

SCVNF

VNF2

VNF3

VNF4

VNF1

Tenant 1NS1

NS2

Tenant 2NS1

NS2

Tenant 3 NS1

Total energy consumption

VNF6VNF5VNF1VNF6VNF2VNF5VNF6VNF4VNF4

VNF2

840Mbps

840Mbps

936Mbps

936Mbps

312Mbps

312Mbps

312Mbps

312Mbps

312Mbps

Finallatency

Tenant 1 Tenant 2 Tenant 3

NS1 NS2 NS1 NS2 NS1

Service type Gold Gold Gold Gold Gold

Max ow

Max latency

Deviation 20 20 20 20 20

Peak ow

5569７

130－＜ＪＭ 90－＜ＪＭ 130－＜ＪＭ 130－＜ＪＭ90－＜ＪＭ

156－＜ＪＭ 108－＜ＪＭ 156－＜ＪＭ 108－＜ＪＭ 156－＜ＪＭ

100ＧＭ 100ＧＭ 100ＧＭ

79ＧＭ

95ＧＭ

89ＧＭ

79ＧＭ

86ＧＭ

CPU

sH

A A

ccSw

itch

80ＧＭ 80ＧＭ

Figure 9 Placement results of gold flavour setting with 20 robust protection

34

32

30

28

260

10

20Robust protection ()

Num

ber o

f use

dco

res

10864200

10

20Robust protection ()

HW

AN

umbe

r of u

sed

0

10

20Robust protection ()

987654N

umbe

r of a

ctiv

eCE

SCs

6000

5500

5000

4500

4000

35003000

010

20

Ener

gy co

nsum

ptio

n(W

)

Robust protection ()

Gold

Silver

Bronze

Gold

Silver

Bronze

Gold

Silver

Bronze

Gold

Silver

Bronze

Figure 10 Comparison of energy and resource consumption for different levels of robust protection

Mobile Information Systems 13

Sets

Π Set of possible VNF placement policies119883 Set of SC microservers119884 Set of cores composing a SC microserver119868 Set of offered NSs119869119894 Set of VNFs in NS 119894119877 Set of resources available in

each SC microserver

Input Parameters

119879119894 Contracted maximum aggregated usertraffic for network service 119894

119863max119894 Maximum latency allowed for NS 119894119872119903119909 Amount of resource 119903 available in

SC microserver 119909119875sc119909 Power consumption of SC 119909 because of

being switched on119875cpu119909119910119894119895 Power consumption of VM 119910 of SC 119909 due

to CPU use of VNF 119895 of NS 119894119875switch119894119895 Power consumption in the physical switch

due to CPU use of VNF 119895 of NS 119894

Decision Variables

119875120587 Total power consumption ofplacement policy 120587

119875120587119894119895 Power consumption of VNF 119895 of NS 119894 inplacement policy 120587

119898120587119903119909 Consumption of resource 119903 in SCmicroserver 119909 in placement policy 120587

119905119897120587119909 Traffic of the link that connects SCmicroserver 119909 with the switchinplacement policy 120587

119860119909119910119894119895 Binary variable that expressesthe assignment of VNF 119895 of NS 119894 toVM 119910 in SC 119909

119871 119894119895 Binary variable to express the use of thenetwork switch to forward the flow ofVNF 119895 of NS 119894 to the next VNF

119889119894 Delay of NS 119894119889119894119895 Delay of VNF 119895 of NS 119894119889119901119909119910119894119895 Processing delay in the CPU of VNF 119895 of

NS 119894 in VM 119910 of SC 1199091198891198971199091119910111990921199102119894119895 Path delay for NS 119894 between VNF 119895

(located in VM 1199101 of SC 1199091) and VNF119895 + 1 (located in VM 1199102 of SC 1199092)

119889vs119909119910 Link delay between the virtual switch inSC 119909 and the VM 119910 in SC 119909

119889vs119909 119904 Link delay between the virtual switch inSC 119909 and the physical switch

119889vs119909 Delay in the virtual switch of SC 119909

119889119904 Delay in the physical switch

Conflicts of Interest

The authors declare that they have no conflicts of interest

Acknowledgments

The research leading to these results has been supportedby the EU funded H2020 5G-PPP Project SESAME (GrantAgreement 671596) and the Spanish MINECO Project5GRANVIR (TEC2016-80090-C2-2-R)

References

[1] ITU-R M2083 IMT Vision-Framework and overall objectivesof the future development of IMT for 2020 and beyond 2015

[2] P Demestichas A Georgakopoulos D Karvounas et al ldquo5G onthe Horizon key challenges for the radio-access networkrdquo IEEEVehicular Technology Magazine vol 8 no 3 pp 47ndash53 2013

[3] I F Akyildiz S Nie S-C Lin and M Chandrasekaran ldquo5Groadmap 10 key enabling technologiesrdquo Computer Networksvol 106 pp 17ndash48 2016

[4] ETSI Mobile-edge computing Introductory technical whitepaper 2014

[5] B Han V Gopalakrishnan L Ji and S Lee ldquoNetwork functionvirtualization challenges and opportunities for innovationsrdquoIEEE Communications Magazine vol 53 no 2 pp 90ndash97 2015

[6] R Mijumbi J Serrat J Gorricho N Bouten F De Turckand R Boutaba ldquoNetwork function virtualization state-of-the-art and research challengesrdquo IEEE Communications Surveys ampTutorials vol 18 no 1 pp 236ndash262 2016

[7] Ericsson White Paper Cloud-RANndash the benefits of virtualiza-tion centralization and coordination 2015

[8] B Blanco J O Fajardo I Giannoulakis et al ldquoTechnologypillars in the architecture of future 5G mobile networks NFVMEC and SDNrdquo Computer Standards and Interfaces vol 54 pp216ndash228 2017

[9] ETSI Network Functions Virtualization (NFV) Managementand Orchestration 2014

[10] A Ben-Tal L E Ghaoui and A Nemirovski Robust Optimiza-tion Aharon Ben-Tal Laurent El Ghaoui Arkadi Nemirovski-Google Books Princeton University Press Princeton NewJersey NY USA 2009

[11] J Gil Herrera and J F Botero ldquoResource Allocation in NFVA Comprehensive Surveyrdquo IEEE Transactions on Network andService Management vol 13 no 3 pp 518ndash532 2016

[12] B Addis D Belabed M Bouet and S Secci ldquoVirtual networkfunctions placement and routing optimizationrdquo in Proceedingsof the 4th IEEE International Conference on Cloud NetworkingCloudNet rsquo15 pp 171ndash177 October 2015

[13] H Moens and F De Turck ldquoVNF-P a model for efficientplacement of virtualized network functionsrdquo in Proceedingsof the 10th International Conference on Network and ServiceManagement (CNSM rsquo14) pp 418ndash423 Rio de Janeiro BrazilNovember 2014

[14] A Gupta M F Habib P Chowdhury M Tornatore and BMukherjee ldquoOn service chaining using Virtual Network Func-tions in Network-enabled Cloud systemsrdquo in Proceedings of the9th IEEE International Conference on Advanced Networks andTelecommuncations Systems ANTS rsquo15 pp 1ndash3 December 2015

[15] M Bouet J Leguay T Combe and V Conan ldquoCost-basedplacement of vDPI functions in NFV infrastructuresrdquo Interna-tional Journal of Network Management vol 25 no 6 pp 490ndash506 2015

[16] M C Luizelli L R Bays L S Buriol M P Barcellos andL P Gaspary ldquoPiecing together the NFV provisioning puzzle

14 Mobile Information Systems

efficient placement and chaining of virtual network functionsrdquoin Proceedings of the 14th International Symposium on IntegratedNetwork Management IM rsquo15 pp 98ndash106 IEEE TorontoCanada May 2015

[17] S Kim Y Han and S Park ldquoAn energy-Aware service functionchaining and reconfiguration algorithm in NFVrdquo in Proceedingsof the 1st International Workshops on Foundations and Applica-tions of Self-Systems FAS-W rsquo16 pp 54ndash59 September 2016

[18] V Eramo A Tosti and E Miucci ldquoServer resource dimen-sioning and routing of service function chain in NFV networkarchitecturesrdquo Journal of Electrical and Computer Engineeringvol 2016 Article ID 7139852 12 pages 2016

[19] K Hida and S-I Kuribayashi ldquoVirtual routing function allo-cation method for minimizing total network power consump-tionrdquo World Academy of Science Engineering and TechnologyInternational Journal of Electrical Computer Energetic Elec-tronic and Communication Engineering vol 10 pp 997ndash10022016

[20] C Pham N H Tran S Ren W Saad and C S Hong ldquoTraffic-aware and energy-efficient vnf placement for service chainingjoint sampling and matching approachrdquo IEEE Transactions onServices Computing 2017

[21] A Marotta and A Kassler ldquoA power efficient and robust virtualnetwork functions placement problemrdquo in Proceedings of the28th International Teletraffic Congress ITC rsquo16 pp 331ndash339September 2016

[22] C Pham H D Tran S I Moon K Thar and C S Hong ldquoAgeneral and practical consolidation framework in CloudNFVrdquoin Proceedings of the 2015 International Conference on Infor-mation Networking ICOIN rsquo15 pp 295ndash300 IEEE CambodiaCambodia January 2015

[23] V Eramo M Ammar and F G Lavacca ldquoMigration energyaware reconfigurations of virtual network function instances inNFV architecturesrdquo IEEE Access vol 5 pp 4927ndash4938 2017

[24] N E Khoury S Ayoubi and C Assi ldquoEnergy-aware placementand scheduling of network trafficflowswith deadlines on virtualnetwork functionsrdquo in Proceedings of the 5th IEEE InternationalConference on CloudNetworking CloudNet rsquo16 pp 89ndash94 IEEEPisa Italy October 2016

[25] V Eramo E Miucci M Ammar and F G Lavacca ldquoAnapproach for service function chain routing and virtual func-tion network instance migration in network function virtual-ization architecturesrdquo IEEEACM Transactions on Networking2017

[26] V Jumba S Parsaeefard M Derakhshani and T Le-NgocldquoEnergy-efficient robust resource provisioning in virtualizedwireless networksrdquo in Proceedings of the IEEE InternationalConference on Ubiquitous Wireless Broadband ICUWB rsquo15 pp1ndash5 IEEE Montreal Canada October 2015

[27] S Coniglio A Koster and M Tieves ldquoData uncertainty invirtual network embedding robust optimization and protectionlevelsrdquo Journal of Network and SystemsManagement vol 24 no3 pp 681ndash710 2016

[28] I Takouna K Sachs and C Meinel ldquoMultiperiod robustoptimization for proactive resource provisioning in virtualizeddata centersrdquo Journal of Supercomputing vol 70 no 3 pp 1514ndash1536 2014

[29] G Chochlidakis and V Friderikos ldquoRobust virtual networkembedding for mobile networksrdquo in Proceedings of the 26thIEEE Annual International Symposium on Personal Indoor andMobile Radio Communications PIMRC rsquo15 pp 1867ndash1871 IEEEKowloon Hong Kong September 2015

[30] S Coniglio A M Koster and M Tieves ldquoVirtual networkembedding under uncertainty exact and heuristic approachesrdquoin Proceedings of the 11th International Conference on the Designof Reliable Communication Networks DRCN rsquo15 pp 1ndash8 IEEEKansas Kan USA March 2015

[31] D Bertsimas and M Sim ldquoThe price of robustnessrdquoOperationsResearch vol 52 no 1 pp 35ndash53 2004

[32] F Rossi P Van Beek and T Walsh Handbook of ConstraintProgramming Elsevier Amsterdam Netherland 2006

[33] httpwwwminizincorg[34] C Schulte G Tack and M Z Lagerkvist Modeling and pro-

gramming with gecode 2010 Schulte Christian and Tack Guidoand Lagerkvist Mikael

Mobile Information Systems 11

SCVNF

VNF5

VNF6VNF5VNF2

VNF3

VNF5

VNF1

VNF2

VNF3

VNF1

VNF2

VNF3

VNF6

SCVNF

VNF1

VNF1

VNF3

VNF4

SCVNF

VNF1

VNF2

VNF3

VNF4

840Mbps

840Mbps

936Mbps

624Mbps

312Mbps

312Mbps

936Mbps

VNF6

Tenant 1NS1

NS2

Tenant 2NS1

NS2

Tenant 3 NS1

Total energy consumption

VNF5VNF6VNF5VNF1VNF4VNF4

VNF2

Finallatency

Tenant 1 Tenant 2 Tenant 3

NS1 NS2 NS1 NS2 NS1

Service type Silver Silver Silver Silver Silver

Max ow

Max latency

Deviation 20 20 20 20 20

Peak ow

116ＧＭ

110ＧＭ

118ＧＭ

5047７

130－＜ＪＭ 90－＜ＪＭ 130－＜ＪＭ 130－＜ＪＭ90－＜ＪＭ

156－＜ＪＭ 108－＜ＪＭ 156－＜ＪＭ 108－＜ＪＭ 156－＜ＪＭ

150ＧＭ 110ＧＭ 150ＧＭ 110ＧＭ 150ＧＭ

79ＧＭ

79ＧＭ

CPU

sH

A A

ccSw

itch

Figure 8 Placement results of silver flavour setting with 20 robust protection

algorithm applies robust optimization techniques to theminimization of the global power consumption of the systemsubject to the bit rate and latency constraints of the networkservices The power consumption of the components of thesystem depends on the usage percentage which is a con-sequence of the aggregated user traffic demand The modelassumes that the relationships between the user trafficdemand the power consumption and the use of the resourcesmay be nonlinear Besides the placement algorithmconsidersthe 5G communication particularities of both control anddata plane and allows multitenancy Finally the algorithmalso includes VNF scaling features and the use of hardwareacceleration

The results show that the placement algorithm succeeds inallocating theVNFs that compose theNSs of different tenantsin the available resourcesmeeting the latency constraints andminimizing the energy consumption of thewhole systemThepresented evaluation examples demonstrate the features ofthe constraint-based expert system and illustrate the impactof the selected service flavour and robust protection on theglobal power consumption and therefore the exploitationcost of the 5G distributed edge cloud

Further researchwill extend the characterization of VNFsto include variable output bit rate This feature will modelthe behavior of for example transcoding functions A future

version of the algorithm will also incorporate the existenceof a virtual switch to connect the VMs running in the samemicroserver Another enhancement of the model will bethe combination of heterogeneous SCs with different con-figurations and resource consumption models Finally theobtained placement results will be used for a technoeconomicstudy that will analyze the relationship between the serveraggregated user traffic cost of the required infrastructure andthe pricing of the services This study will also consider theintegration of the placement decision problem with the radioside

Summary of Model Indexes SetsParameters and Decision Variables

Indexes

120587 Index for VNF placement policy119909 Index for SC microserver119910 Index for core number in each microserver119894 Index for NS119895 Index for VNF in a network service chain119903 Index for physical resources in

SC microserver

12 Mobile Information Systems

VNF6VNF5VNF1VNF6VNF2SCVNF

VNF2

VNF3

VNF5

VNF3

VNF1

VNF2

VNF3

VNF5

SCVNF

VNF1

VNF1

VNF3

VNF4

SCVNF

VNF2

VNF3

VNF4

VNF1

Tenant 1NS1

NS2

Tenant 2NS1

NS2

Tenant 3 NS1

Total energy consumption

VNF6VNF5VNF1VNF6VNF2VNF5VNF6VNF4VNF4

VNF2

840Mbps

840Mbps

936Mbps

936Mbps

312Mbps

312Mbps

312Mbps

312Mbps

312Mbps

Finallatency

Tenant 1 Tenant 2 Tenant 3

NS1 NS2 NS1 NS2 NS1

Service type Gold Gold Gold Gold Gold

Max ow

Max latency

Deviation 20 20 20 20 20

Peak ow

5569７

130－＜ＪＭ 90－＜ＪＭ 130－＜ＪＭ 130－＜ＪＭ90－＜ＪＭ

156－＜ＪＭ 108－＜ＪＭ 156－＜ＪＭ 108－＜ＪＭ 156－＜ＪＭ

100ＧＭ 100ＧＭ 100ＧＭ

79ＧＭ

95ＧＭ

89ＧＭ

79ＧＭ

86ＧＭ

CPU

sH

A A

ccSw

itch

80ＧＭ 80ＧＭ

Figure 9 Placement results of gold flavour setting with 20 robust protection

34

32

30

28

260

10

20Robust protection ()

Num

ber o

f use

dco

res

10864200

10

20Robust protection ()

HW

AN

umbe

r of u

sed

0

10

20Robust protection ()

987654N

umbe

r of a

ctiv

eCE

SCs

6000

5500

5000

4500

4000

35003000

010

20

Ener

gy co

nsum

ptio

n(W

)

Robust protection ()

Gold

Silver

Bronze

Gold

Silver

Bronze

Gold

Silver

Bronze

Gold

Silver

Bronze

Figure 10 Comparison of energy and resource consumption for different levels of robust protection

Mobile Information Systems 13

Sets

Π Set of possible VNF placement policies119883 Set of SC microservers119884 Set of cores composing a SC microserver119868 Set of offered NSs119869119894 Set of VNFs in NS 119894119877 Set of resources available in

each SC microserver

Input Parameters

119879119894 Contracted maximum aggregated usertraffic for network service 119894

119863max119894 Maximum latency allowed for NS 119894119872119903119909 Amount of resource 119903 available in

SC microserver 119909119875sc119909 Power consumption of SC 119909 because of

being switched on119875cpu119909119910119894119895 Power consumption of VM 119910 of SC 119909 due

to CPU use of VNF 119895 of NS 119894119875switch119894119895 Power consumption in the physical switch

due to CPU use of VNF 119895 of NS 119894

Decision Variables

119875120587 Total power consumption ofplacement policy 120587

119875120587119894119895 Power consumption of VNF 119895 of NS 119894 inplacement policy 120587

119898120587119903119909 Consumption of resource 119903 in SCmicroserver 119909 in placement policy 120587

119905119897120587119909 Traffic of the link that connects SCmicroserver 119909 with the switchinplacement policy 120587

119860119909119910119894119895 Binary variable that expressesthe assignment of VNF 119895 of NS 119894 toVM 119910 in SC 119909

119871 119894119895 Binary variable to express the use of thenetwork switch to forward the flow ofVNF 119895 of NS 119894 to the next VNF

119889119894 Delay of NS 119894119889119894119895 Delay of VNF 119895 of NS 119894119889119901119909119910119894119895 Processing delay in the CPU of VNF 119895 of

NS 119894 in VM 119910 of SC 1199091198891198971199091119910111990921199102119894119895 Path delay for NS 119894 between VNF 119895

(located in VM 1199101 of SC 1199091) and VNF119895 + 1 (located in VM 1199102 of SC 1199092)

119889vs119909119910 Link delay between the virtual switch inSC 119909 and the VM 119910 in SC 119909

119889vs119909 119904 Link delay between the virtual switch inSC 119909 and the physical switch

119889vs119909 Delay in the virtual switch of SC 119909

119889119904 Delay in the physical switch

Conflicts of Interest

The authors declare that they have no conflicts of interest

Acknowledgments

The research leading to these results has been supportedby the EU funded H2020 5G-PPP Project SESAME (GrantAgreement 671596) and the Spanish MINECO Project5GRANVIR (TEC2016-80090-C2-2-R)

References

[1] ITU-R M2083 IMT Vision-Framework and overall objectivesof the future development of IMT for 2020 and beyond 2015

[2] P Demestichas A Georgakopoulos D Karvounas et al ldquo5G onthe Horizon key challenges for the radio-access networkrdquo IEEEVehicular Technology Magazine vol 8 no 3 pp 47ndash53 2013

[3] I F Akyildiz S Nie S-C Lin and M Chandrasekaran ldquo5Groadmap 10 key enabling technologiesrdquo Computer Networksvol 106 pp 17ndash48 2016

[4] ETSI Mobile-edge computing Introductory technical whitepaper 2014

[5] B Han V Gopalakrishnan L Ji and S Lee ldquoNetwork functionvirtualization challenges and opportunities for innovationsrdquoIEEE Communications Magazine vol 53 no 2 pp 90ndash97 2015

[6] R Mijumbi J Serrat J Gorricho N Bouten F De Turckand R Boutaba ldquoNetwork function virtualization state-of-the-art and research challengesrdquo IEEE Communications Surveys ampTutorials vol 18 no 1 pp 236ndash262 2016

[7] Ericsson White Paper Cloud-RANndash the benefits of virtualiza-tion centralization and coordination 2015

[8] B Blanco J O Fajardo I Giannoulakis et al ldquoTechnologypillars in the architecture of future 5G mobile networks NFVMEC and SDNrdquo Computer Standards and Interfaces vol 54 pp216ndash228 2017

[9] ETSI Network Functions Virtualization (NFV) Managementand Orchestration 2014

[10] A Ben-Tal L E Ghaoui and A Nemirovski Robust Optimiza-tion Aharon Ben-Tal Laurent El Ghaoui Arkadi Nemirovski-Google Books Princeton University Press Princeton NewJersey NY USA 2009

[11] J Gil Herrera and J F Botero ldquoResource Allocation in NFVA Comprehensive Surveyrdquo IEEE Transactions on Network andService Management vol 13 no 3 pp 518ndash532 2016

[12] B Addis D Belabed M Bouet and S Secci ldquoVirtual networkfunctions placement and routing optimizationrdquo in Proceedingsof the 4th IEEE International Conference on Cloud NetworkingCloudNet rsquo15 pp 171ndash177 October 2015

[13] H Moens and F De Turck ldquoVNF-P a model for efficientplacement of virtualized network functionsrdquo in Proceedingsof the 10th International Conference on Network and ServiceManagement (CNSM rsquo14) pp 418ndash423 Rio de Janeiro BrazilNovember 2014

[14] A Gupta M F Habib P Chowdhury M Tornatore and BMukherjee ldquoOn service chaining using Virtual Network Func-tions in Network-enabled Cloud systemsrdquo in Proceedings of the9th IEEE International Conference on Advanced Networks andTelecommuncations Systems ANTS rsquo15 pp 1ndash3 December 2015

[15] M Bouet J Leguay T Combe and V Conan ldquoCost-basedplacement of vDPI functions in NFV infrastructuresrdquo Interna-tional Journal of Network Management vol 25 no 6 pp 490ndash506 2015

[16] M C Luizelli L R Bays L S Buriol M P Barcellos andL P Gaspary ldquoPiecing together the NFV provisioning puzzle

14 Mobile Information Systems

efficient placement and chaining of virtual network functionsrdquoin Proceedings of the 14th International Symposium on IntegratedNetwork Management IM rsquo15 pp 98ndash106 IEEE TorontoCanada May 2015

[17] S Kim Y Han and S Park ldquoAn energy-Aware service functionchaining and reconfiguration algorithm in NFVrdquo in Proceedingsof the 1st International Workshops on Foundations and Applica-tions of Self-Systems FAS-W rsquo16 pp 54ndash59 September 2016

[18] V Eramo A Tosti and E Miucci ldquoServer resource dimen-sioning and routing of service function chain in NFV networkarchitecturesrdquo Journal of Electrical and Computer Engineeringvol 2016 Article ID 7139852 12 pages 2016

[19] K Hida and S-I Kuribayashi ldquoVirtual routing function allo-cation method for minimizing total network power consump-tionrdquo World Academy of Science Engineering and TechnologyInternational Journal of Electrical Computer Energetic Elec-tronic and Communication Engineering vol 10 pp 997ndash10022016

[20] C Pham N H Tran S Ren W Saad and C S Hong ldquoTraffic-aware and energy-efficient vnf placement for service chainingjoint sampling and matching approachrdquo IEEE Transactions onServices Computing 2017

[21] A Marotta and A Kassler ldquoA power efficient and robust virtualnetwork functions placement problemrdquo in Proceedings of the28th International Teletraffic Congress ITC rsquo16 pp 331ndash339September 2016

[22] C Pham H D Tran S I Moon K Thar and C S Hong ldquoAgeneral and practical consolidation framework in CloudNFVrdquoin Proceedings of the 2015 International Conference on Infor-mation Networking ICOIN rsquo15 pp 295ndash300 IEEE CambodiaCambodia January 2015

[23] V Eramo M Ammar and F G Lavacca ldquoMigration energyaware reconfigurations of virtual network function instances inNFV architecturesrdquo IEEE Access vol 5 pp 4927ndash4938 2017

[24] N E Khoury S Ayoubi and C Assi ldquoEnergy-aware placementand scheduling of network trafficflowswith deadlines on virtualnetwork functionsrdquo in Proceedings of the 5th IEEE InternationalConference on CloudNetworking CloudNet rsquo16 pp 89ndash94 IEEEPisa Italy October 2016

[25] V Eramo E Miucci M Ammar and F G Lavacca ldquoAnapproach for service function chain routing and virtual func-tion network instance migration in network function virtual-ization architecturesrdquo IEEEACM Transactions on Networking2017

[26] V Jumba S Parsaeefard M Derakhshani and T Le-NgocldquoEnergy-efficient robust resource provisioning in virtualizedwireless networksrdquo in Proceedings of the IEEE InternationalConference on Ubiquitous Wireless Broadband ICUWB rsquo15 pp1ndash5 IEEE Montreal Canada October 2015

[27] S Coniglio A Koster and M Tieves ldquoData uncertainty invirtual network embedding robust optimization and protectionlevelsrdquo Journal of Network and SystemsManagement vol 24 no3 pp 681ndash710 2016

[28] I Takouna K Sachs and C Meinel ldquoMultiperiod robustoptimization for proactive resource provisioning in virtualizeddata centersrdquo Journal of Supercomputing vol 70 no 3 pp 1514ndash1536 2014

[29] G Chochlidakis and V Friderikos ldquoRobust virtual networkembedding for mobile networksrdquo in Proceedings of the 26thIEEE Annual International Symposium on Personal Indoor andMobile Radio Communications PIMRC rsquo15 pp 1867ndash1871 IEEEKowloon Hong Kong September 2015

[30] S Coniglio A M Koster and M Tieves ldquoVirtual networkembedding under uncertainty exact and heuristic approachesrdquoin Proceedings of the 11th International Conference on the Designof Reliable Communication Networks DRCN rsquo15 pp 1ndash8 IEEEKansas Kan USA March 2015

[31] D Bertsimas and M Sim ldquoThe price of robustnessrdquoOperationsResearch vol 52 no 1 pp 35ndash53 2004

[32] F Rossi P Van Beek and T Walsh Handbook of ConstraintProgramming Elsevier Amsterdam Netherland 2006

[33] httpwwwminizincorg[34] C Schulte G Tack and M Z Lagerkvist Modeling and pro-

gramming with gecode 2010 Schulte Christian and Tack Guidoand Lagerkvist Mikael

12 Mobile Information Systems

VNF6VNF5VNF1VNF6VNF2SCVNF

VNF2

VNF3

VNF5

VNF3

VNF1

VNF2

VNF3

VNF5

SCVNF

VNF1

VNF1

VNF3

VNF4

SCVNF

VNF2

VNF3

VNF4

VNF1

Tenant 1NS1

NS2

Tenant 2NS1

NS2

Tenant 3 NS1

Total energy consumption

VNF6VNF5VNF1VNF6VNF2VNF5VNF6VNF4VNF4

VNF2

840Mbps

840Mbps

936Mbps

936Mbps

312Mbps

312Mbps

312Mbps

312Mbps

312Mbps

Finallatency

Tenant 1 Tenant 2 Tenant 3

NS1 NS2 NS1 NS2 NS1

Service type Gold Gold Gold Gold Gold

Max ow

Max latency

Deviation 20 20 20 20 20

Peak ow

5569７

130－＜ＪＭ 90－＜ＪＭ 130－＜ＪＭ 130－＜ＪＭ90－＜ＪＭ

156－＜ＪＭ 108－＜ＪＭ 156－＜ＪＭ 108－＜ＪＭ 156－＜ＪＭ

100ＧＭ 100ＧＭ 100ＧＭ

79ＧＭ

95ＧＭ

89ＧＭ

79ＧＭ

86ＧＭ

CPU

sH

A A

ccSw

itch

80ＧＭ 80ＧＭ

Figure 9 Placement results of gold flavour setting with 20 robust protection

34

32

30

28

260

10

20Robust protection ()

Num

ber o

f use

dco

res

10864200

10

20Robust protection ()

HW

AN

umbe

r of u

sed

0

10

20Robust protection ()

987654N

umbe

r of a

ctiv

eCE

SCs

6000

5500

5000

4500

4000

35003000

010

20

Ener

gy co

nsum

ptio

n(W

)

Robust protection ()

Gold

Silver

Bronze

Gold

Silver

Bronze

Gold

Silver

Bronze

Gold

Silver

Bronze

Figure 10 Comparison of energy and resource consumption for different levels of robust protection

Mobile Information Systems 13

Sets

Π Set of possible VNF placement policies119883 Set of SC microservers119884 Set of cores composing a SC microserver119868 Set of offered NSs119869119894 Set of VNFs in NS 119894119877 Set of resources available in

each SC microserver

Input Parameters

119879119894 Contracted maximum aggregated usertraffic for network service 119894

119863max119894 Maximum latency allowed for NS 119894119872119903119909 Amount of resource 119903 available in

SC microserver 119909119875sc119909 Power consumption of SC 119909 because of

being switched on119875cpu119909119910119894119895 Power consumption of VM 119910 of SC 119909 due

to CPU use of VNF 119895 of NS 119894119875switch119894119895 Power consumption in the physical switch

due to CPU use of VNF 119895 of NS 119894

Decision Variables

119875120587 Total power consumption ofplacement policy 120587

119875120587119894119895 Power consumption of VNF 119895 of NS 119894 inplacement policy 120587

119898120587119903119909 Consumption of resource 119903 in SCmicroserver 119909 in placement policy 120587

119905119897120587119909 Traffic of the link that connects SCmicroserver 119909 with the switchinplacement policy 120587

119860119909119910119894119895 Binary variable that expressesthe assignment of VNF 119895 of NS 119894 toVM 119910 in SC 119909

119871 119894119895 Binary variable to express the use of thenetwork switch to forward the flow ofVNF 119895 of NS 119894 to the next VNF

119889119894 Delay of NS 119894119889119894119895 Delay of VNF 119895 of NS 119894119889119901119909119910119894119895 Processing delay in the CPU of VNF 119895 of

NS 119894 in VM 119910 of SC 1199091198891198971199091119910111990921199102119894119895 Path delay for NS 119894 between VNF 119895

(located in VM 1199101 of SC 1199091) and VNF119895 + 1 (located in VM 1199102 of SC 1199092)

119889vs119909119910 Link delay between the virtual switch inSC 119909 and the VM 119910 in SC 119909

119889vs119909 119904 Link delay between the virtual switch inSC 119909 and the physical switch

119889vs119909 Delay in the virtual switch of SC 119909

119889119904 Delay in the physical switch

Conflicts of Interest

The authors declare that they have no conflicts of interest

Acknowledgments

The research leading to these results has been supportedby the EU funded H2020 5G-PPP Project SESAME (GrantAgreement 671596) and the Spanish MINECO Project5GRANVIR (TEC2016-80090-C2-2-R)

References

[1] ITU-R M2083 IMT Vision-Framework and overall objectivesof the future development of IMT for 2020 and beyond 2015

[2] P Demestichas A Georgakopoulos D Karvounas et al ldquo5G onthe Horizon key challenges for the radio-access networkrdquo IEEEVehicular Technology Magazine vol 8 no 3 pp 47ndash53 2013

[3] I F Akyildiz S Nie S-C Lin and M Chandrasekaran ldquo5Groadmap 10 key enabling technologiesrdquo Computer Networksvol 106 pp 17ndash48 2016

[4] ETSI Mobile-edge computing Introductory technical whitepaper 2014

[5] B Han V Gopalakrishnan L Ji and S Lee ldquoNetwork functionvirtualization challenges and opportunities for innovationsrdquoIEEE Communications Magazine vol 53 no 2 pp 90ndash97 2015

[6] R Mijumbi J Serrat J Gorricho N Bouten F De Turckand R Boutaba ldquoNetwork function virtualization state-of-the-art and research challengesrdquo IEEE Communications Surveys ampTutorials vol 18 no 1 pp 236ndash262 2016

[7] Ericsson White Paper Cloud-RANndash the benefits of virtualiza-tion centralization and coordination 2015

[8] B Blanco J O Fajardo I Giannoulakis et al ldquoTechnologypillars in the architecture of future 5G mobile networks NFVMEC and SDNrdquo Computer Standards and Interfaces vol 54 pp216ndash228 2017

[9] ETSI Network Functions Virtualization (NFV) Managementand Orchestration 2014

[10] A Ben-Tal L E Ghaoui and A Nemirovski Robust Optimiza-tion Aharon Ben-Tal Laurent El Ghaoui Arkadi Nemirovski-Google Books Princeton University Press Princeton NewJersey NY USA 2009

[11] J Gil Herrera and J F Botero ldquoResource Allocation in NFVA Comprehensive Surveyrdquo IEEE Transactions on Network andService Management vol 13 no 3 pp 518ndash532 2016

[12] B Addis D Belabed M Bouet and S Secci ldquoVirtual networkfunctions placement and routing optimizationrdquo in Proceedingsof the 4th IEEE International Conference on Cloud NetworkingCloudNet rsquo15 pp 171ndash177 October 2015

[13] H Moens and F De Turck ldquoVNF-P a model for efficientplacement of virtualized network functionsrdquo in Proceedingsof the 10th International Conference on Network and ServiceManagement (CNSM rsquo14) pp 418ndash423 Rio de Janeiro BrazilNovember 2014

[14] A Gupta M F Habib P Chowdhury M Tornatore and BMukherjee ldquoOn service chaining using Virtual Network Func-tions in Network-enabled Cloud systemsrdquo in Proceedings of the9th IEEE International Conference on Advanced Networks andTelecommuncations Systems ANTS rsquo15 pp 1ndash3 December 2015

[15] M Bouet J Leguay T Combe and V Conan ldquoCost-basedplacement of vDPI functions in NFV infrastructuresrdquo Interna-tional Journal of Network Management vol 25 no 6 pp 490ndash506 2015

[16] M C Luizelli L R Bays L S Buriol M P Barcellos andL P Gaspary ldquoPiecing together the NFV provisioning puzzle

14 Mobile Information Systems

efficient placement and chaining of virtual network functionsrdquoin Proceedings of the 14th International Symposium on IntegratedNetwork Management IM rsquo15 pp 98ndash106 IEEE TorontoCanada May 2015

[17] S Kim Y Han and S Park ldquoAn energy-Aware service functionchaining and reconfiguration algorithm in NFVrdquo in Proceedingsof the 1st International Workshops on Foundations and Applica-tions of Self-Systems FAS-W rsquo16 pp 54ndash59 September 2016

[18] V Eramo A Tosti and E Miucci ldquoServer resource dimen-sioning and routing of service function chain in NFV networkarchitecturesrdquo Journal of Electrical and Computer Engineeringvol 2016 Article ID 7139852 12 pages 2016

[19] K Hida and S-I Kuribayashi ldquoVirtual routing function allo-cation method for minimizing total network power consump-tionrdquo World Academy of Science Engineering and TechnologyInternational Journal of Electrical Computer Energetic Elec-tronic and Communication Engineering vol 10 pp 997ndash10022016

[20] C Pham N H Tran S Ren W Saad and C S Hong ldquoTraffic-aware and energy-efficient vnf placement for service chainingjoint sampling and matching approachrdquo IEEE Transactions onServices Computing 2017

[21] A Marotta and A Kassler ldquoA power efficient and robust virtualnetwork functions placement problemrdquo in Proceedings of the28th International Teletraffic Congress ITC rsquo16 pp 331ndash339September 2016

[22] C Pham H D Tran S I Moon K Thar and C S Hong ldquoAgeneral and practical consolidation framework in CloudNFVrdquoin Proceedings of the 2015 International Conference on Infor-mation Networking ICOIN rsquo15 pp 295ndash300 IEEE CambodiaCambodia January 2015

[23] V Eramo M Ammar and F G Lavacca ldquoMigration energyaware reconfigurations of virtual network function instances inNFV architecturesrdquo IEEE Access vol 5 pp 4927ndash4938 2017

[24] N E Khoury S Ayoubi and C Assi ldquoEnergy-aware placementand scheduling of network trafficflowswith deadlines on virtualnetwork functionsrdquo in Proceedings of the 5th IEEE InternationalConference on CloudNetworking CloudNet rsquo16 pp 89ndash94 IEEEPisa Italy October 2016

[25] V Eramo E Miucci M Ammar and F G Lavacca ldquoAnapproach for service function chain routing and virtual func-tion network instance migration in network function virtual-ization architecturesrdquo IEEEACM Transactions on Networking2017

[26] V Jumba S Parsaeefard M Derakhshani and T Le-NgocldquoEnergy-efficient robust resource provisioning in virtualizedwireless networksrdquo in Proceedings of the IEEE InternationalConference on Ubiquitous Wireless Broadband ICUWB rsquo15 pp1ndash5 IEEE Montreal Canada October 2015

[27] S Coniglio A Koster and M Tieves ldquoData uncertainty invirtual network embedding robust optimization and protectionlevelsrdquo Journal of Network and SystemsManagement vol 24 no3 pp 681ndash710 2016

[28] I Takouna K Sachs and C Meinel ldquoMultiperiod robustoptimization for proactive resource provisioning in virtualizeddata centersrdquo Journal of Supercomputing vol 70 no 3 pp 1514ndash1536 2014

[29] G Chochlidakis and V Friderikos ldquoRobust virtual networkembedding for mobile networksrdquo in Proceedings of the 26thIEEE Annual International Symposium on Personal Indoor andMobile Radio Communications PIMRC rsquo15 pp 1867ndash1871 IEEEKowloon Hong Kong September 2015

[30] S Coniglio A M Koster and M Tieves ldquoVirtual networkembedding under uncertainty exact and heuristic approachesrdquoin Proceedings of the 11th International Conference on the Designof Reliable Communication Networks DRCN rsquo15 pp 1ndash8 IEEEKansas Kan USA March 2015

[31] D Bertsimas and M Sim ldquoThe price of robustnessrdquoOperationsResearch vol 52 no 1 pp 35ndash53 2004

[32] F Rossi P Van Beek and T Walsh Handbook of ConstraintProgramming Elsevier Amsterdam Netherland 2006

[33] httpwwwminizincorg[34] C Schulte G Tack and M Z Lagerkvist Modeling and pro-

gramming with gecode 2010 Schulte Christian and Tack Guidoand Lagerkvist Mikael

Mobile Information Systems 13

Sets

Π Set of possible VNF placement policies119883 Set of SC microservers119884 Set of cores composing a SC microserver119868 Set of offered NSs119869119894 Set of VNFs in NS 119894119877 Set of resources available in

each SC microserver

Input Parameters

119879119894 Contracted maximum aggregated usertraffic for network service 119894

119863max119894 Maximum latency allowed for NS 119894119872119903119909 Amount of resource 119903 available in

SC microserver 119909119875sc119909 Power consumption of SC 119909 because of

being switched on119875cpu119909119910119894119895 Power consumption of VM 119910 of SC 119909 due

to CPU use of VNF 119895 of NS 119894119875switch119894119895 Power consumption in the physical switch

due to CPU use of VNF 119895 of NS 119894

Decision Variables

119875120587 Total power consumption ofplacement policy 120587

119875120587119894119895 Power consumption of VNF 119895 of NS 119894 inplacement policy 120587

119898120587119903119909 Consumption of resource 119903 in SCmicroserver 119909 in placement policy 120587

119905119897120587119909 Traffic of the link that connects SCmicroserver 119909 with the switchinplacement policy 120587

119860119909119910119894119895 Binary variable that expressesthe assignment of VNF 119895 of NS 119894 toVM 119910 in SC 119909

119871 119894119895 Binary variable to express the use of thenetwork switch to forward the flow ofVNF 119895 of NS 119894 to the next VNF

119889119894 Delay of NS 119894119889119894119895 Delay of VNF 119895 of NS 119894119889119901119909119910119894119895 Processing delay in the CPU of VNF 119895 of

NS 119894 in VM 119910 of SC 1199091198891198971199091119910111990921199102119894119895 Path delay for NS 119894 between VNF 119895

(located in VM 1199101 of SC 1199091) and VNF119895 + 1 (located in VM 1199102 of SC 1199092)

119889vs119909119910 Link delay between the virtual switch inSC 119909 and the VM 119910 in SC 119909

119889vs119909 119904 Link delay between the virtual switch inSC 119909 and the physical switch

119889vs119909 Delay in the virtual switch of SC 119909

119889119904 Delay in the physical switch

Conflicts of Interest

The authors declare that they have no conflicts of interest

Acknowledgments

The research leading to these results has been supportedby the EU funded H2020 5G-PPP Project SESAME (GrantAgreement 671596) and the Spanish MINECO Project5GRANVIR (TEC2016-80090-C2-2-R)

References

[1] ITU-R M2083 IMT Vision-Framework and overall objectivesof the future development of IMT for 2020 and beyond 2015

[2] P Demestichas A Georgakopoulos D Karvounas et al ldquo5G onthe Horizon key challenges for the radio-access networkrdquo IEEEVehicular Technology Magazine vol 8 no 3 pp 47ndash53 2013

[3] I F Akyildiz S Nie S-C Lin and M Chandrasekaran ldquo5Groadmap 10 key enabling technologiesrdquo Computer Networksvol 106 pp 17ndash48 2016

[4] ETSI Mobile-edge computing Introductory technical whitepaper 2014

[5] B Han V Gopalakrishnan L Ji and S Lee ldquoNetwork functionvirtualization challenges and opportunities for innovationsrdquoIEEE Communications Magazine vol 53 no 2 pp 90ndash97 2015

[6] R Mijumbi J Serrat J Gorricho N Bouten F De Turckand R Boutaba ldquoNetwork function virtualization state-of-the-art and research challengesrdquo IEEE Communications Surveys ampTutorials vol 18 no 1 pp 236ndash262 2016

[7] Ericsson White Paper Cloud-RANndash the benefits of virtualiza-tion centralization and coordination 2015

[8] B Blanco J O Fajardo I Giannoulakis et al ldquoTechnologypillars in the architecture of future 5G mobile networks NFVMEC and SDNrdquo Computer Standards and Interfaces vol 54 pp216ndash228 2017

[9] ETSI Network Functions Virtualization (NFV) Managementand Orchestration 2014

[10] A Ben-Tal L E Ghaoui and A Nemirovski Robust Optimiza-tion Aharon Ben-Tal Laurent El Ghaoui Arkadi Nemirovski-Google Books Princeton University Press Princeton NewJersey NY USA 2009

[11] J Gil Herrera and J F Botero ldquoResource Allocation in NFVA Comprehensive Surveyrdquo IEEE Transactions on Network andService Management vol 13 no 3 pp 518ndash532 2016

[12] B Addis D Belabed M Bouet and S Secci ldquoVirtual networkfunctions placement and routing optimizationrdquo in Proceedingsof the 4th IEEE International Conference on Cloud NetworkingCloudNet rsquo15 pp 171ndash177 October 2015

[13] H Moens and F De Turck ldquoVNF-P a model for efficientplacement of virtualized network functionsrdquo in Proceedingsof the 10th International Conference on Network and ServiceManagement (CNSM rsquo14) pp 418ndash423 Rio de Janeiro BrazilNovember 2014

[14] A Gupta M F Habib P Chowdhury M Tornatore and BMukherjee ldquoOn service chaining using Virtual Network Func-tions in Network-enabled Cloud systemsrdquo in Proceedings of the9th IEEE International Conference on Advanced Networks andTelecommuncations Systems ANTS rsquo15 pp 1ndash3 December 2015

[15] M Bouet J Leguay T Combe and V Conan ldquoCost-basedplacement of vDPI functions in NFV infrastructuresrdquo Interna-tional Journal of Network Management vol 25 no 6 pp 490ndash506 2015

[16] M C Luizelli L R Bays L S Buriol M P Barcellos andL P Gaspary ldquoPiecing together the NFV provisioning puzzle

14 Mobile Information Systems

efficient placement and chaining of virtual network functionsrdquoin Proceedings of the 14th International Symposium on IntegratedNetwork Management IM rsquo15 pp 98ndash106 IEEE TorontoCanada May 2015

[17] S Kim Y Han and S Park ldquoAn energy-Aware service functionchaining and reconfiguration algorithm in NFVrdquo in Proceedingsof the 1st International Workshops on Foundations and Applica-tions of Self-Systems FAS-W rsquo16 pp 54ndash59 September 2016

[18] V Eramo A Tosti and E Miucci ldquoServer resource dimen-sioning and routing of service function chain in NFV networkarchitecturesrdquo Journal of Electrical and Computer Engineeringvol 2016 Article ID 7139852 12 pages 2016

[19] K Hida and S-I Kuribayashi ldquoVirtual routing function allo-cation method for minimizing total network power consump-tionrdquo World Academy of Science Engineering and TechnologyInternational Journal of Electrical Computer Energetic Elec-tronic and Communication Engineering vol 10 pp 997ndash10022016

[20] C Pham N H Tran S Ren W Saad and C S Hong ldquoTraffic-aware and energy-efficient vnf placement for service chainingjoint sampling and matching approachrdquo IEEE Transactions onServices Computing 2017

[21] A Marotta and A Kassler ldquoA power efficient and robust virtualnetwork functions placement problemrdquo in Proceedings of the28th International Teletraffic Congress ITC rsquo16 pp 331ndash339September 2016

[22] C Pham H D Tran S I Moon K Thar and C S Hong ldquoAgeneral and practical consolidation framework in CloudNFVrdquoin Proceedings of the 2015 International Conference on Infor-mation Networking ICOIN rsquo15 pp 295ndash300 IEEE CambodiaCambodia January 2015

[23] V Eramo M Ammar and F G Lavacca ldquoMigration energyaware reconfigurations of virtual network function instances inNFV architecturesrdquo IEEE Access vol 5 pp 4927ndash4938 2017

[24] N E Khoury S Ayoubi and C Assi ldquoEnergy-aware placementand scheduling of network trafficflowswith deadlines on virtualnetwork functionsrdquo in Proceedings of the 5th IEEE InternationalConference on CloudNetworking CloudNet rsquo16 pp 89ndash94 IEEEPisa Italy October 2016

[25] V Eramo E Miucci M Ammar and F G Lavacca ldquoAnapproach for service function chain routing and virtual func-tion network instance migration in network function virtual-ization architecturesrdquo IEEEACM Transactions on Networking2017

[26] V Jumba S Parsaeefard M Derakhshani and T Le-NgocldquoEnergy-efficient robust resource provisioning in virtualizedwireless networksrdquo in Proceedings of the IEEE InternationalConference on Ubiquitous Wireless Broadband ICUWB rsquo15 pp1ndash5 IEEE Montreal Canada October 2015

[27] S Coniglio A Koster and M Tieves ldquoData uncertainty invirtual network embedding robust optimization and protectionlevelsrdquo Journal of Network and SystemsManagement vol 24 no3 pp 681ndash710 2016

[28] I Takouna K Sachs and C Meinel ldquoMultiperiod robustoptimization for proactive resource provisioning in virtualizeddata centersrdquo Journal of Supercomputing vol 70 no 3 pp 1514ndash1536 2014

[29] G Chochlidakis and V Friderikos ldquoRobust virtual networkembedding for mobile networksrdquo in Proceedings of the 26thIEEE Annual International Symposium on Personal Indoor andMobile Radio Communications PIMRC rsquo15 pp 1867ndash1871 IEEEKowloon Hong Kong September 2015

[30] S Coniglio A M Koster and M Tieves ldquoVirtual networkembedding under uncertainty exact and heuristic approachesrdquoin Proceedings of the 11th International Conference on the Designof Reliable Communication Networks DRCN rsquo15 pp 1ndash8 IEEEKansas Kan USA March 2015

[31] D Bertsimas and M Sim ldquoThe price of robustnessrdquoOperationsResearch vol 52 no 1 pp 35ndash53 2004

[32] F Rossi P Van Beek and T Walsh Handbook of ConstraintProgramming Elsevier Amsterdam Netherland 2006

[33] httpwwwminizincorg[34] C Schulte G Tack and M Z Lagerkvist Modeling and pro-

gramming with gecode 2010 Schulte Christian and Tack Guidoand Lagerkvist Mikael

14 Mobile Information Systems

efficient placement and chaining of virtual network functionsrdquoin Proceedings of the 14th International Symposium on IntegratedNetwork Management IM rsquo15 pp 98ndash106 IEEE TorontoCanada May 2015

[17] S Kim Y Han and S Park ldquoAn energy-Aware service functionchaining and reconfiguration algorithm in NFVrdquo in Proceedingsof the 1st International Workshops on Foundations and Applica-tions of Self-Systems FAS-W rsquo16 pp 54ndash59 September 2016

[18] V Eramo A Tosti and E Miucci ldquoServer resource dimen-sioning and routing of service function chain in NFV networkarchitecturesrdquo Journal of Electrical and Computer Engineeringvol 2016 Article ID 7139852 12 pages 2016

[19] K Hida and S-I Kuribayashi ldquoVirtual routing function allo-cation method for minimizing total network power consump-tionrdquo World Academy of Science Engineering and TechnologyInternational Journal of Electrical Computer Energetic Elec-tronic and Communication Engineering vol 10 pp 997ndash10022016

[20] C Pham N H Tran S Ren W Saad and C S Hong ldquoTraffic-aware and energy-efficient vnf placement for service chainingjoint sampling and matching approachrdquo IEEE Transactions onServices Computing 2017

[21] A Marotta and A Kassler ldquoA power efficient and robust virtualnetwork functions placement problemrdquo in Proceedings of the28th International Teletraffic Congress ITC rsquo16 pp 331ndash339September 2016

[22] C Pham H D Tran S I Moon K Thar and C S Hong ldquoAgeneral and practical consolidation framework in CloudNFVrdquoin Proceedings of the 2015 International Conference on Infor-mation Networking ICOIN rsquo15 pp 295ndash300 IEEE CambodiaCambodia January 2015

[23] V Eramo M Ammar and F G Lavacca ldquoMigration energyaware reconfigurations of virtual network function instances inNFV architecturesrdquo IEEE Access vol 5 pp 4927ndash4938 2017

[24] N E Khoury S Ayoubi and C Assi ldquoEnergy-aware placementand scheduling of network trafficflowswith deadlines on virtualnetwork functionsrdquo in Proceedings of the 5th IEEE InternationalConference on CloudNetworking CloudNet rsquo16 pp 89ndash94 IEEEPisa Italy October 2016

[25] V Eramo E Miucci M Ammar and F G Lavacca ldquoAnapproach for service function chain routing and virtual func-tion network instance migration in network function virtual-ization architecturesrdquo IEEEACM Transactions on Networking2017

[26] V Jumba S Parsaeefard M Derakhshani and T Le-NgocldquoEnergy-efficient robust resource provisioning in virtualizedwireless networksrdquo in Proceedings of the IEEE InternationalConference on Ubiquitous Wireless Broadband ICUWB rsquo15 pp1ndash5 IEEE Montreal Canada October 2015

[27] S Coniglio A Koster and M Tieves ldquoData uncertainty invirtual network embedding robust optimization and protectionlevelsrdquo Journal of Network and SystemsManagement vol 24 no3 pp 681ndash710 2016

[28] I Takouna K Sachs and C Meinel ldquoMultiperiod robustoptimization for proactive resource provisioning in virtualizeddata centersrdquo Journal of Supercomputing vol 70 no 3 pp 1514ndash1536 2014

[29] G Chochlidakis and V Friderikos ldquoRobust virtual networkembedding for mobile networksrdquo in Proceedings of the 26thIEEE Annual International Symposium on Personal Indoor andMobile Radio Communications PIMRC rsquo15 pp 1867ndash1871 IEEEKowloon Hong Kong September 2015

[30] S Coniglio A M Koster and M Tieves ldquoVirtual networkembedding under uncertainty exact and heuristic approachesrdquoin Proceedings of the 11th International Conference on the Designof Reliable Communication Networks DRCN rsquo15 pp 1ndash8 IEEEKansas Kan USA March 2015

[31] D Bertsimas and M Sim ldquoThe price of robustnessrdquoOperationsResearch vol 52 no 1 pp 35ndash53 2004

[32] F Rossi P Van Beek and T Walsh Handbook of ConstraintProgramming Elsevier Amsterdam Netherland 2006

[33] httpwwwminizincorg[34] C Schulte G Tack and M Z Lagerkvist Modeling and pro-

gramming with gecode 2010 Schulte Christian and Tack Guidoand Lagerkvist Mikael

Submit your manuscripts athttpswwwhindawicom

Computer Games Technology

International Journal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Distributed Sensor Networks

International Journal of

Advances in

FuzzySystems

Hindawi Publishing Corporationhttpwwwhindawicom

Volume 2014

International Journal of

ReconfigurableComputing

Hindawi Publishing Corporation httpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 201

Applied Computational Intelligence and Soft Computing

thinspAdvancesthinspinthinsp

Artificial Intelligence

HindawithinspPublishingthinspCorporationhttpwwwhindawicom Volumethinsp2014

Advances inSoftware EngineeringHindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Electrical and Computer Engineering

Journal of

Hindawi Publishing Corporation

httpwwwhindawicom Volume 2014

Advances in

Multimedia

International Journal of

Biomedical Imaging

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Advances in

Hindawi Publishing Corporationhttpwwwhindawicom Volume 201

RoboticsJournal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Computational Intelligence and Neuroscience

Industrial EngineeringJournal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Modelling amp Simulation in EngineeringHindawi Publishing Corporation httpwwwhindawicom Volume 2014

The Scientific World JournalHindawi Publishing Corporation httpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Human-ComputerInteraction

Advances in

Computer EngineeringAdvances in

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014