Telco Clouds: Modelling and Simulation

Jakub Krzywda1, William Tarneberg2, Per-Olov Ostberg1, Maria Kihl2, and Erik Elmroth1

1Dept. of Computing Science, Umea University, SE-901 87 Umea, Sweden2Dept. of Electrical and Information Technology, Lund University, SE-223 63 Lund, Sweden

{jakub, p-o, elmroth}@cs.umu.se, {william.tarneberg, maria.kihl}@eit.lth.se

Keywords: Mobile cloud computing, Telecommunication infrastructure, Cloud infrastructure, Modelling, Simulation

Abstract: In this paper, we propose a telco cloud meta-model that can be used to simulate different infrastructure config-urations and explore their consequences for system performance and costs. To achieve this, we analyse currenttelecommunication and data centre infrastructure paradigms, describe the architecture of the telco cloud, anddetail the benefits of merging both infrastructures in a unified system. Next, we detail the dynamics of thetelco cloud and identify the components that are the most relevant from the perspective of modelling perfor-mance and cost. As a number of well established simulation technologies exist for most of the telco cloudcomponents, we survey existing models in an attempt to construct a suitable composite meta-model. Finally,we present a showcase scenario to demonstrate the scope of our telco cloud simulator.

1 INTRODUCTION

Recent technological developments have enabled aunion of telecommunication and cloud computing.Joint management of telecommunication infrastruc-ture, such as Radio Base Stations (RBS), and DataCentres (DC) may help to achieve better performanceof hosted applications and reduce the operation costs.In this paper we refer to this paradigm as telco cloudcomputing (Bosch et al., 2011).

Despite the interest in this paradigm, there are nosimulation models capable of simultaneously mod-elling the dynamics of Mobile Devices (MD), place-ment and capacity of DCs, and network infrastructure.Understanding of these relations is important for telcocloud stakeholders, e.g., Infrastructure Providers (IP)that can use that knowledge to reduce infrastructurecosts while still delivering competitive performance.

We propose a meta-model of the telco cloud thatfacilitates experimentation and evaluation of possi-ble configurations, such as placement and capacityof DCs in a joint telecommunication-cloud infrastruc-ture. The meta-model uses existing, well establishedsimulation models, e.g., for Radio Access Networks(RANs) or DCs, for modelling of individual parts ofthe infrastructure behaviour.

The contributions of this paper are: describing thedynamics of the telco cloud, including Quality of Ser-vice (QoS) and the associated costs of this paradigm(Section 4); surveying existing models of telco cloud

building blocks (Section 5); and establishing a meta-model that captures the described dynamics using ex-isting and composite models (Section 6).

This paper is structured as follows. Section 2outlines the architecture of the proposed telco cloud.Next, the simulation motivations, challenges, and re-quirements that the meta-model has to fulfill are pre-sented in Section 3. Section 4 describes the telcocloud dynamics, followed by a survey of existingmodels in Section 5. Section 6 introduces the telcocloud meta-model. Section 7 presents a showcasesimulation scenario, using a prototype implementa-tion of the introduced meta-model. In Section 8 welist a few relevant research topics that can be exploredusing the proposed model and conclude the paper.

2 THE TELCO CLOUDARCHITECTURE

In this section we present issues with current telecom-munication and cloud infrastructures, an overview ofthe proposed telco cloud topology, and how the telcocloud paradigm can help to remedy these issues.

2.1 Current Infrastructure

Currently, telecommunication and cloud computinginfrastructures are separated and managed indepen-dently. The telecommunication infrastructure is

placed in close proximity to end users and is built us-ing specialised hardware. The cloud computing in-frastructure consists of remote DCs that are signifi-cantly geographically separated from end users, con-sists of commodity hardware, and is connected withthe telecommunication infrastructure via the Internet.

We identify several issues of the current infras-tructure. Performance (especially latency) of cloudservices is not predictable, which makes computationoffloading difficult. All data processed in the cloudare sent over the Internet to DCs, which adds com-munication latency. For the Internet of Things, withmillions of sensors generating huge amounts of data,the volume of traffic can cause network congestion.Moreover, specialised telecommunication hardware isexpensive and hard to upgrade.

As a consequence of the performance bottlenecks,particularly latency sensitive applications such as in-dustrial process control and augmented reality con-text recognition applications have mostly not yet beencloudified. The low latency, jitter free, and highthroughput connections required by such applicationscannot be provided by the existing telecommunica-tion and cloud infrastructures (Barker and Shenoy,2010). Moreover, compute and battery resources inMDs are limited and coupled. An approach to aug-menting MD’s capabilities is to offload the executionof applications to a cloud infrastructure. Such perfor-mance augmentation can only be seamless if the com-munication latency is low enough, and both networkbandwidth and service availability are high.

Devices are at an increasing rate gaining accessto the Internet (Atzori et al., 2010). Anything fromsmall sensors to petrol pumps, flowerpots, helmets,tumblers, windows, and spark plugs is being con-nected to the Internet to communicate and monitorits quantified performance metrics. Most of the con-nected devices are generating correlated contextualinformation, incurring large amounts of Wide AreaNetwork’s (WAN) traffic. The traffic typically con-verges to a handful of DCs for analysis and processingwhich introduces congestion in the capacity-sparseand increasingly congested RANs, core networks, andWANs. Additionally, a portion of the transported in-formation is only locally relevant and will add no orvery little entropy to a service in a Remote DC.

Wireless access network virtualisation and cloud-ification of telecom equipment and services proposedby (Wang et al., 2013), requires careful placement ofcompute capacity as not to introduce significant prop-agation delay. The placement of the compute nodeshas to reflect the demand for telecom- and cloud-services in the geographic area which the RAN cov-ers. Current telecom signalling standards and remote

Mobile Device

Radio Base Station

Radio Base Station

Controller Radio Catchment Border

Network

Proximal Data Centers

Remote Data Centre

Figure 1: Overview of telco cloud.

DCs do not interoperate well and are often not able tomeet telecom latency requirements.

2.2 Introducing the Telco Cloud

A telco cloud is an infrastructure consisting of MDs(known also as user equipment), stationary devices(sensors), access networks, intermediate WANs (con-necting the access networks to the backbone net-works), backbone (Internet) networks, and DCs. Wehere include two main types of DCs: Remote DataCentres (large DCs located far from the access net-works) and Proximal Data Centres (smaller DCs lo-cated close to the access networks).

The telco cloud topology paradigm proposesa closer integration between access networks anda cloud infrastructure than the current topologicalmodel where the telecom and cloud infrastructuresare unaware of each other. When coexisting withcloud infrastructure, telecommunication functionalitycan be virtualised and augmented to the adjacent DCs.When virtualising RAN functions large portions of anRBS and the RAN control functions can be executedin a DC (Baroncelli et al., 2010).

Cloud capacity will reside in geographically dis-tributed DCs. The telco cloud topology is composedof multiple DCs that are dispersed in a mesh structure,ranging from complete adjacency with the telecom in-frastructure, Proximal DCs, to more traditional, Re-mote DCs, as depicted in Figure 1.

A group of telco cloud DCs can be provisionedand load balanced as one resource or act as indepen-dent DCs. In the former case, a control plane willneed to coordinate services and the shared resourcesby for example optimising locality and proximity toall entities by geographically placing services accord-ingly. Proximal DCs will thus have to be managed in

a distributed and coordinated manner.The cloud services hosted in Proximal DCs are ac-

cessed through the RAN. The MDs are assumed topossess varying degrees of mobility, with an equiv-alent likelihood of passing between a particular setof macro/micro-RBS1 over time. In an effort to beable to enforce DC management constraints and toglobally avoid unnecessary migration, an MD is notstrictly associated with the closest DC or the DC thatits current RBS cell is associated with.

The telco cloud infrastructure topology will needto reflect the capacity and latency objectives of thevirtualised RBSs, and to give access to the telcocloud-hosted services given the networks local capa-bility and diversity at a sufficient service level. Theprevailing 4G/LTE topology is centred around hierar-chical macro- and micro-cells that very much resem-bles that of its predecessors. The telco cloud topol-ogy will evolve with mobile access technology gen-erations, shifting and/or dispersing compute capacityat various levels of mobile, Metropolitan Area Net-work (MAN), and WAN networks to best suit the pre-vailing services and throughput channels.

2.3 Benefits of the Telco Cloud

We identify four main benefits of the telco cloud.First, it provides cloud applications with better andmore predictable performance. Second, it supportscomputation off-loading for resource-bounded MDs.Third, the telco cloud reduces network utilization byprocessing part of data closer to its producer or con-sumer. Fourth, it enhances cost-efficiency and flexi-bility of telecommunication infrastructure.

Thanks to a geographically distributed cloud in-frastructure, application developers and telecommu-nication operators can take advantage of the signif-icantly lower round-trip times. Moreover, we ex-pect that the average network throughput will increasewhen communicating with Proximal DCs in compar-ison to Remote DCs. Additionally, users offloadingMD applications will benefit from a low latency com-munication with a DC, where the code is executed.

The large amount of information generated bysensors can be filtered through the intermediate hier-archical cascade of DCs to prevent congestion in theintermediate WAN. Data that is only locally relevantcan be kept and consumed locally, while redundantand highly covariant information can be more easilyidentified in a local context and discarded.

Through the telco cloud traditional proprietaryhardware-bound telecom services can be virtualised,

1What is considered a traditional rural/urban cell, con-stituted by a high power RBS, mounted on a tower.

migrated, and executed in Proximal or Remote DCs.Multiple RBSs can be consolidated to increase theaggregate utilisation of the infrastructure. ExecutingRBSs on cloud infrastructure will allow for greateruse of cost-effective commodity hardware and genericsoftware. With the availability of the telco cloud, weexpect that an RBS in future mobile infrastructuregenerations will only consist of a radio interface. Themanagement of the RAN, individual radio channels,signalling, services, and signal processing, will be allvirtualised and executed in a Proximal DC.

3 SIMULATION CHALLENGES

Simulation of a telco cloud is motivated by severalfactors, e.g., lack of existing infrastructure and appro-priate control plane standards. The desired simulatorhas to fulfil many requirements, such as, to be able tosimulate hundreds of thousands of various entities atfine grained time granularity (milliseconds) for longperiods of time (hours). We here motivate and de-scribe identified requirements and challenges in sim-ulation of the telco cloud. Addressing the challengesand fulfilling the requirements is crucial while design-ing a meta-model and implementing a simulator.

Telco cloud stakeholders will benefit from beingable to investigate the consequences of possible in-frastructure configurations. For example, IPs respon-sible for building and maintaining the infrastructuremay use the simulator for planning placement and ca-pacity for new DCs, as well as modifying capacityand connectivity of existing DCs. Service providersthat use infrastructures to host services are interestedin comparing different strategies for placement of ap-plication components. Moreover, developers that im-plement mobile applications utilising a telco cloud,need to determine what application components couldbenefit from offloading. To answer these and simi-lar questions, tests with various infrastructure config-urations need to be performed and results compared.There are two options for performing these tests: bysimulation or by running them in real test beds.

Currently, there is no existing infrastructure thatcan be used for testing the telco cloud. Creatinga physical test bed for large-scale testing of a telcocloud in different configurations is economically in-feasible and small-scale test beds will not be able tocapture phenomena occurring in reality, e.g., user mo-bility patterns or latency issues.

For these reasons, we believe that simulation isthe most feasible option to evaluate the telco cloud.However, we have identified several requirements thatmake simulation of telco clouds challenging. First of

Workload- Application- Mobility

Objectives- QoS- Cost

Setup- Topology- Capacity- Service placement

Figure 2: Dependencies between elements of a telco cloud.

all, the scale of simulation is large in terms of numberand types of entities. The simulation of a telco cloudhas to concurrently cover hundreds of thousands ofMDs moving around a simulated area, each gener-ating requests; hundreds of RBSs providing an ac-cess to the core network; and tens of DCs, runningservices that process requests. Another challenge isthe ratio between time precision and length of simu-lation. We are interested in a very fine-grained latencysimulation, that requires precision of at least millisec-onds. However, to capture the daily patterns of MDmovement (e.g., moving between home, work, andshops) caused by migrations of users carrying them,the whole system needs to simulate several run-timehours. Moreover, simulation of application stateful-ness, and of transferring those states when MDs aremoving, have not yet been described in the literatureand requires new models to be developed.

4 TELCO CLOUD DYNAMICS

Before constructing a meta-model of the telco cloudwe first need to understand fundamental telco clouddynamics, in terms of the relations between systeminput, configuration and output. Later, we will use theknowledge of these dynamics to build the intendedsimulation meta-model.

Figure 2 visualizes the dynamics inside the telcocloud. The workload is an input to the system that IPshave no influence over. It includes: applications, witha request generation model (rate and size), a resourcerequirements model (the amount of resources neededto process a request), and application statefulness (theoverhead of transferring user’s state between DC); aswell as, the mobility of users carrying MDs.

Next, objectives describe required output charac-teristics of the system. We have identified two funda-mental objectives. Firstly, QoS, which imposes per-formance requirements, e.g., latency or throughputthrough Service Level Objectives (SLO). Secondly,the monetary cost associated with energy consumedfor computation and maintenance of an infrastructure.The objectives can be used when constructing an op-timisation problem with QoS as conditions and costas the function that should be minimised.

Finally, setup is that part of the system that canbe adjusted by designers or operators to achieve de-sired objectives when processing existing workloads.Setup includes topology, location, capacity of DCsand the network that interconnects MDs and RBSswith DCs, as well as resource management policiesthat control placement and migration of services.

The above mentioned elements are all dependenton each other. The setup of a telco cloud is relatedto the existing workload. The capacity of a DC is de-fined by the resource demands of the services, e.g.how memory-, CPU-, network-, and disk-intensivethe services are. The locations and topology of DCsare defined by the geographic and demographic scopeof the services, the number of MDs that reside in thatdomain, and the capacity of the associated telecom-munication infrastructure.

In addition, workload influences objectives. Forexample, user mobility is inducing delays during ser-vice migrations and potentially causes QoS penalties.Moreover, application statefulness introduces addi-tional costs of storing data in a DC and transmittingdata between Proximal DCs. It may also increase la-tency due to an additional time necessary to send thestate data before processing of the request can begin.

Finally, objectives depend on the setup: QoS isproportional to the proximity and capacity of DC –a smaller DC catchment (the geographic area the DCserves) translates to greater locality and reduced prop-agation latency, while higher capacity allows hostingof more services. Moreover, the capacity and catch-ment of DCs determine telco cloud costs. Costs areproportional to the dispersion of computing capac-ity. There is an overhead of each DC, irregardless ofits capacity, e.g., building, cooling infrastructure, andconnection to energy or network. In addition, disper-sion increases costs of maintenance, e.g., technicianshave to travel between locations. Therefore, costs areproportionally higher in smaller, dispersed DCs be-cause of high initial costs and proportionally lower inlarger, centralized DCs due to the economy of scale(Armbrust et al., 2010).

5 EXISTING MODELS

To support creation of a meta-model that incorporatesworkload, setup, and objectives of the telco cloud de-scribed in the previous section, we here survey exist-ing models in the following categories: application re-quest generation and resource requirements, MD mo-bility, networks, DCs, and infrastructure costs.

Most of the models and simulators are assignedto only one of the above mentioned categories, how-

ever capabilities of four surveyed simulators extend tomany categories, so we summarise them in Table 1.

Table 1: Overview of surveyed simulators.

Framework RG RR M N DCNS-3 X X XOMNeT++ X X XCloudSim X XGreenCloud X X

RG – Request Generation, RR – Resource Require-ments, M – Mobility, N – Network, DC – Data Centre.

5.1 Workload Models

Applications running in the telco cloud consist of amobile client and a server processing offloaded com-putations. Therefore, they should be modelled fromtwo perspectives: request generation that describeshow requests are created and sent to the DCs; and re-source requirements that describes how much compu-tational resources are needed to process the requests.

Request Generation

Request generation models capture the user’s be-haviour by primarily representing interaction times orthe timing clicks through a stochastic process, oftenPoissonian in nature. A user behaviour model canbe further refined by introducing a stochastic modelfor how long time a user consumes a certain type ofcontent. Additionally, the transition between types ofcontent is often modelled as a Markov process.

Furthermore, the request generation characteris-tics are commonly modelled with multiple stochas-tic processes, encompassing the number of packetsin a session, and the size of each packet. Requestgeneration models are either closed or open looped.In an open loop model, the generation of each newsession is typically a Poisson process independent ofthe resulting DC action. Conversely, in a closed loopmodel, the generation of new sessions is dependenton timing of the response from the DC and thus theproperties of the previously generated session.

In the packet-level event driven network simula-tors, NS-3 (Riley and Henderson, 2010) and OM-NeT++ (Varga et al., 2001), a node can act as eithera client or server, by the mechanism of either send-ing packets provided by a stochastic model, at a givenrate, within a certain time period, or at a certain in-terval; or processing received packets from a buffer,at a given rate. Both server and client models can beaugmented with a more complex system of queues tosuch an extent that they can represent an abstract DCthat hosts multiple applications.

Resource Requirements

CloudSim (Calheiros et al., 2011), a simulator ofcloud infrastructure, provides an application modelthat describes computational requirements – theamount of resources that needs to be available (e.g.number of cores, memory and storage); and com-municational requirements – the amount of data thatneeds to be transferred. GreenCloud (Kliazovichet al., 2012), a packet level simulator based on NS-2, apart from computational and communicationalrequirements, describes also QoS requirements, ex-pressed by an execution deadline. The applicationmodel may also include the size of the code that hasto be offloaded and dependencies on other services,e.g., in terms of amount of data that has to be sent orreceived (Kovachev, 2012).

Mobility

The NS-3 and OMNeT++ nodes described above canbe set into motion given a certain stochastic mobilitymodel. They can for example traverse the space aspedestrians, or in auto mobiles, with correspondingvelocity and rate of change. The spatial relationshipbetween nodes and RBS affects the channel propertiesand RBS-to-node associations. Node mobility willalso result in handover between RBSs, which in turnwill alter the paths of the node-generated workload inthe network.

5.2 Setup Models

Below, we describe existing models and simulators ofnetworks and DCs, which can be used to configure thesetup of the telco cloud meta-model.

Network

There are several well-established event-drivenframeworks that are capable of modelling computernetworks, mobile networks, applications, packet-levelnetwork traffic, infrastructure, and independent users.The two primary examples are, as mentioned in theprevious section, NS-3 and OMNeT++. These twoare commonly deployed in academic network re-search and provide detailed results on network utili-sation, throughput, congestion, and latency.

Both NS-3 and OMNeT++ are comprehensivepacket-level network simulation frameworks that in-clude wired and wireless standards, and are able tosimulate communication channel conditions. Fur-thermore, both frameworks have detailed models forchannel definition, such as propagation delay, inter-ference, data rate, and medium access schemes. In

addition, both NS-3 and OMNeT++, support controlplane signalling for a number of wireless standardsand complex network topologies.

Both frameworks have support for modelling dif-ferent types of network nodes, ranging from com-puters to routers and switches. Each edge and nodepair has a defined communication and medium accessstandard, such as TCP/IP and Ethernet. Each packetthat is sent over the network is treated in accordancewith the prevailing network’s transport protocols androuting standard. In both, the events of arrival anddeparture of packets drive the simulation clock.

Furthermore, they require detailed configurationof all communication modes and node behaviour,making it very time-consuming to implement and ver-ify systems with different levels of abstractions, andare thus cumbersome to model abstract systems.

As the telco cloud topology is yet to be de-fined with unspecified control planes, it would becounter-intuitive and time consuming to implementtelco cloud topologies in either NS-3 or OMNeT++.In some instances, some modules would have to becompletely redesigned, and others would have to bespecified to a much greater detail than the telco cloudcan offer at this stage.

Data Centre

The purpose of this section is to survey the DC mod-els that are the most suitable for inclusion in thetelco cloud meta-model. An extensive list of mathe-matical models, simulation approaches, and test bedscan be found in (Sakellari and Loukas, 2013), while(Ahmed and Sabyasachi, 2014) provides a survey oftwelve cloud simulators. After careful examination,we chose the ones that best suit our goals.

We compare DC models and simulators based ondescriptions provided by the authors of the simulators.For each model we describe: Resource Provisioning– what resources are included and how they are mod-elled; QoS – what performance indicators are mea-sured; Costs of computation in the DC; Performanceof simulator – an estimation of the time needed to per-form a simulation.

CloudSim is an event-based simulator imple-mented in Java, for simulation of cloud computingsystem and application provisioning environments.Resource Provisioning. The CloudSim simulationlayer offers dedicated management interfaces forCPU, memory, storage and bandwidth allocation, aswell as, defining policies in allocating hosts to Vir-tual Machines (VM) – VM provisioning. Hosts aredescribed by processing capabilities (in MIPS) anda core provisioning policy, together with an amount

of available memory and storage. A model sup-ports time-sharing and space-sharing core provision-ing policies on both host and VM levels.Latency (QoS). The latency model is based on con-ceptual networking abstraction, where the communi-cation delays between each pair of entity type (e.g.host, storage, end-user) are described in a latency ma-trix as a constant value expressed in simulation timeunits (e.g. milliseconds).Costs. CloudSim provides a two-layered cost model,where the first layer relates to Infrastructure as a Ser-vice (IaaS), with costs per unit of resources, while thesecond one relates to Software as a Service (SaaS),with costs per task units (application requests). Thismodel allows calculation of the costs of using thecloud from the end-user perspective or the revenuefrom the IP perspective.Performance. CloudSim is able to perform large-scalesimulations, e.g., it can instantiate an experiment with1 million hosts in 12 seconds. Moreover, memoryusage grows linearly with the host number and evenwith 1 million hosts it does not exceed 320 MB.

CloudAnalyst (Wickremasinghe et al., 2010) isa simulator of geographically distributed large-scalecloud applications, that is developed in Java andutilises CloudSim and SimJava.Resource Provisioning. Cloud Analyst uses the re-source provisioning model of CloudSim.Latency (QoS). A latency model allows configurationof network delays, available bandwidth between re-gions, and current traffic levels. CloudAnalyst fa-cilitates experiments with latency by producing thefollowing statistical metrics: average, minimum, andmaximum response time of all user requests; and re-sponse time grouped by time of the day, location, andDC.Costs. CloudAnalyst supports calculation of costs forusing cloud resources, such as cost per VM per hourand cost per Gigabit of data transfer.Performance. To improve performance of simulationentities are grouped at three levels: clusters of users,cluster of requests generated by users, and clusters ofrequests processed by VM.

GreenCloud is a packet level simulator based onNS-2, for simulation of energy-aware clouds.Resource Provisioning. Servers are modelled as sin-gle core nodes with defined processing power limit (inMIPS or FLOPS), size of memory and storage, andimplementing different task scheduling mechanisms.Latency (QoS). Full support for the TCP/IP protocolreference model is provided and the simulator calcu-lates communication latency with high accuracy.Costs. GreenCloud allows detailed modelling of en-ergy consumption by implementing energy models

for every DC element.Performance. Given that GreenCloud has to simulatethe full stack of Internet protocols, each simulationmay take even tens of minutes for a DC with a fewthousands of nodes.

5.3 Costs Models

The above mentioned DC models focus mostly on thecosts of running applications in DCs from the end-user perspective. Since we want to model costs ofDCs from the IP point of view, additional models areneeded for capital expenditures (CAPEX) and operat-ing expenditures (OPEX).CAPEX includes costs of infrastructure that needs tobe built and servers that have to be bought.Infrastructure Costs. Costs of building, power distri-bution, (and cooling can be estimated using a follow-ing equation: $200M · (1+ cm)/ai, where cm is thecost of money2, and ai is the time of infrastructureamortisation [in years] (Greenberg et al., 2008).Server Costs. Costs of servers can be modelled asns · ps ·(1+ cm)/as, where ns is the number of servers,ps is the price of one server [in $], cm is the cost ofmoney, and as is the time of server amortisation [inyears] (Greenberg et al., 2008).OPEX consists of power and personnel costs.Power Costs. To estimate costs of power, the follow-ing equation can be used, ns · pcs/1000 ·PUE · pkWh ·24 · 365, where ns is the number of servers, pcs isthe power consumption of one server [in W], PUEis Power Usage Efficiency, and pkWh is the price ofelectricity [in $ per kWh] (Greenberg et al., 2008).Personnel Costs. Costs of personnel can be calcu-lated using M1 ·C1 +M2 ·C2 +M3 ·C3, where M1 isthe number of IT personnel per rack, M2 is the num-ber of facility personnel per rack, M3 is the numberof administrative personnel per rack, and C1, C2, C3are the average costs per person for each of the abovementioned categories (Patel and Shah, 2005).

From another perspective, Munoz et al., providean evaluation of telecom, storage and computing costsfor cloud infrastructure (Munoz et al., 2011)

6 TELCO CLOUD META-MODEL

We here detail how we have composed the above sur-veyed models into a telco cloud meta-model. Figure 3depicts the visualisation of the meta-model. MDs,such as cell phones or laptops, are carried by end-users, who are in motion. The MDs generate requests

2Cost of money is the rate of interest or dividend pay-ment on borrowed capital.

Radio Base Station

Data Centre

User StateApplication Queue

Request

Figure 3: Visualisation of telco cloud meta-model.

which are sent over the network to a DC. It is alsopossible that requests are generated by sensors thatmay be static (e.g. traffic cameras) or mobile (e.g.trains). The requests are processed in the DC and theresponse is sent back to the MD or sensor. Processingrequests, in case of stateful applications, generates auser state that has to be migrated with the end-user ifhe moves to the catchment of another DC.

The primary objective of the meta-model is to cap-ture the interactions between application workload,MD mobility, network topology, and DC character-istics, and their influence on QoS and costs of telcocloud. The parameters that define the meta-model arepresented in Table 2 and described in detail below.

6.1 Workload Model

The first group of parameters in Table 2 describes themobility of end-users carrying MD and the character-istics of requests generated by these MDs.

Request Generation

Many services may run in the telco cloud at the sametime and their number is defined by Nser. We modela service application as a stateful web service. Eachsession is separated in time with a Poisson processλses (Reyes-Lecuona et al., 1999). Each session pro-duces Nreq requests, sampled from an inverse Gaus-sian distribution, where each request is separated intime by Log-Normal distributed delay Dreq in sec-onds. The size of each request is given by Sreq KBand is drawn from a Pareto distribution.

Resource Requirements

To model application resource requirements we pro-pose a linear model specifying the needed amount

Table 2: Fundamental meta-model parameters.

Type Parameters Unit DescriptionWORKLOAD

RequestGeneration

Nser Total number of servicesλi

ses, where i = 1,2, . . . ,Nser s Session arrival rate to DCNi

req, where i = 1,2, . . . ,Nser Number of requests per sessionSi

req, where i = 1,2, . . . ,Nser KB Size of requestsDi

req, where i = 1,2, . . . ,Nser s Inter-request time

ResourceRequirements

CPU iidle, CPU i

req, where i = 1,2, . . . ,Nser MI CPU cycles used by servicememi

idle, memireq, where i = 1,2, . . . ,Nser MB Size of memory used by service

diskiidle, diski

req, where i = 1,2, . . . ,Nser MB Size of storage used by servicestatei, where i = 1,2, . . . ,Nser MB Size of user’s state per request

Mobility NMD Number of Mobile Devicessi

t , ait , θi

t , ωit , where i = 1,2, . . . ,NMD Movements of Mobile Devices

SETUP

NetworkNRBS Number of Radio Base StationsdRBS m Dimensions of an RBS cellDnet s Cumulative network delay

Data Centre

NDC Number of Data CentresNi

S, where i = 1,2, . . . ,NDC Number of servers in Data CentreN j

CPU , where j = 1,2, . . . ,NiS Number of CPUs per server

s jCPU , where j = 1,2, . . . ,Ni

S MIPS CPU’s speedmemory j, where j = 1,2, . . . ,Ni

S MB Amount of memory per serverstorage j, where j = 1,2, . . . ,Ni

S GB Amount of storage per servernetworki

bw, where i = 1,2, . . . ,NDC Mb/s Network bandwidthtinit , tidle, tterm s Times of VM transitions

Service Placement placement ={every, n-closests, . . .} Service placement policyOBJECTIVES

Qualityof Service

RT i, where i = 1,2, . . . ,Nser s Application response timeT Pi, where i = 1,2, . . . ,Nser req/s Application throughput

Costs Cost $ Total costs of infrastructure

of resources, both for an idle service and per pro-cessing each request. An idle service uses CPUidleCPU operations, memidle amount of memory, anddiskidle amount of storage. Additionally for each pro-cessed request, the service uses CPUreq CPU opera-tions, memreq amount of memory, and diskreq amountof storage. The amount of user’s state data createdby each request is defined by state and expressed inabsolute value or percentage of request size Sreq.

Mobility

The network is populated by NMD MDs, each sub-scribing to a subset of the Nser available services. The2-dimensional, multi modal, mobility model detailedin (Bettstetter, 2001) provides us with an on-averageuniform distribution of users, with movement propor-tional to the duration of a session and the scale of themobile network. The aforementioned model definesthe properties of an MD’s movement. A MD’s mo-

mentary movement is defined by its velocity consti-tuted by the current speed s and current direction θ.Changes in mobility are defined by multiple stochas-tic processes that describe the duration of its state. Anentity’s speed s is independent of direction θ and ismaintained for Ts seconds, after which acceleration abetween amin and amax is applied for time Ta, until itreaches smin or smax. Furthermore, direction θ is main-tained for time Tθ until the next change-event wherethe direction θ is altered for Tω seconds with at therate of ω radians per second. Ts, Ta, Tθ, and Tω, de-scribing the timing of each change-event, are set foreach mobility mode, and are each defined by a proba-bility distribution bounded by a maxima and minima.

6.2 Setup Model

The second group of parameters in Table 2 charac-terises the network and DCs.

Network

In our model, the core network introduces a cumu-lative propagation, switching, and routing delay andit is modelled with a Weibull delay Dnet in multiplesof the number of network nodes between the sourceand the destination (Papagiannaki et al., 2003). Thenetwork distance between RBSs is equal to the cell di-mension dRBS. The associated RBSs are equidistant totheir common DC, and are for the sake of simplicityassumed to be separated by one network edge.

Furthermore, forthcoming cell planing practicesaim to increase area energy efficiency by favouringsmaller cells in urban areas (Shahab et al., 2013;Fehske et al., 2009). Our model employs a smallhomogeneous mobile network composed of NRBSequidistantly distributed RBSs.

In the absence of a specific mobile generationstandard, an MD is handed over between RBSs at thegeographic point where they cross the cell boundarydistinguishing two independent RBSs defined by thewidth of the rectangular cells dRBS.

Data Centre

The DC model captures the influence that its capac-ity has on performance and costs of computation, asdescribed in Section 4.

To capture the influence on performance, quan-tity and quality of each DC resource is described.DC consists of NS servers, that can differ in spec-ification. Server contains NCPU CPUs capable ofexecuting sCPU operations in every second. Valuesof memory and storage specify the total amount ofavailable memory and storage, respectively. The net-work bandwidth is specified with networkbw. The DCmodel includes also a provisioning model, that de-scribes how available resources are shared among sev-eral applications, e.g., time-sharing or space-sharing.

A DC hosts services in VMs. A service can bedistributed over multiple VMs. Incoming workloadis load-balanced by either a method of round-robin,random selection, or placed in the VM with the low-est load. However, a user’s requests are always for-warded to the VM that served his first request. A ser-vice can specify a minimum and maximum number ofVMs it requires. The DC scales the application withinthese bounds based on the load-balancing outcome.

To emulate the life-cycle of a VM we have definedsix VM states as described in Table 3. The transitionsbetween the states are presented in Figure 4. At thebeginning all VMs are in INACTIVE state. A VMis initiated when the first request arrives to a DC. Ittakes tinit seconds before VM is ready to start process-ing requests or receiving migrated requests and user

Table 3: States of Virtual Machine.

Name DescriptionINACTIVE VM is turned off.

INITIATING VM is booting up.PROCESSING VM is serving requests.

IDLE VM is waiting for requests.MIGRATING VM is transmitting data.

TERMINATING VM is shutting down.

Inactive Initiating Idle

Processing

Migrating

Terminating

Figure 4: Transitions between Virtual Machine states.

state from other DC. We assume that a VM is not ableto process requests and handle migrations at the sametime, so it changes state between PROCESSING andMIGRATION over the time. Moreover, we give mi-grations a higher priority than processing, so process-ing is paused if there are any migrations to perform.When there are no requests to process and no migra-tions to handle a VM goes into IDLE state. A VMis terminated if IDLE state lasts for longer than tidleseconds, and the VM termination takes tterm seconds.

Service Placement

Service placement policies define in what DC(s) a ser-vice should be hosted, what number of replicas shouldbe running, and when a service should be migratedbetween DCs. These decisions depend on the mo-bility of users, the size of users’ state that has to bemigrated, and QoS requirements. For example, a ser-vice can be hosted in n Proximal DCs closest to themajority of its users (n-closests), or in the case of la-tency sensitive services in every Proximal DC that isneeded to provide acceptable QoS (every).

6.3 Objectives Model

The third group of parameters in Table 2 describesQoS and costs of the telco cloud.

Quality of Service

Combining the resource requirements model, whichdescribes the amount of resources an application

needs, with a DC model, allows to simulate how co-location of different services in a DC influences theirresponse times RT i and throughputs T Pi.

Costs

In our opinion the cost models available in the liter-ature and described in Section 5.3 are very ”countrydependent”, because of the inclusion of variable pa-rameters such as salaries, costs of energy or costs ofproperty. They are also not taking into account pa-rameters important from the perspective of the telcocloud, such as the size of DC. Therefore, we modelthe costs of the telco cloud using a basic heuristicbased on observation that dispersion of infrastructurecauses additional costs, e.g.: increase of administratortravel time between locations, and higher unit costsof computation in proximal DCs because of smallerscale and high initial costs.

Cost ∝NDC

∑DC NS

(1)

As shown in Equation 1, the total cost of a telcocloud is directly proportional to the number of DCsand inversely proportional to the total number ofservers in all DCs. It means that distributing the samenumber of servers among many DCs is more expen-sive than placing them in a single DC.

6.4 Limitations

The proposed meta-model has several limitations.The application model assumes that all requests gen-erated by one application are homogeneous and thateach of them consumes the same amount of resources.The mobile access network model does not take intoaccount the physical layer, channel provisioning, andcell load balancing. Additionally, the radio accessnetwork functions as a mechanism to associate MDswith DCs propagation, and system processing delaysare thus not modelled.

7 SIMULATION SHOWCASE

We have implemented a coarse grained simulator us-ing SimJava (Howell and McNab, 1998) as the under-lying event-driven simulation framework. All mod-ules are implemented from scratch but are based onthe meta-model presented in Section 6. The simula-tor fully implements the proposed request generationand network models, but has implemented more ab-stract mobility, resource requirements, DC, and ser-vice placement models.

To demonstrate the scope of the telco cloud meta-model and the simulator we introduce an elementaryshowcase scenario below. The scenario is designedto reveal the basic relationship between workload –MD mobility, setup – Proximal DC catchment, andobjectives – the aggregate utilisation of a telco cloud.

7.1 Experiments

For the sake of clarity we present a simplified sce-nario. Only one service is considered and the size ofthe simulation is reduced when compared to the de-sired scale. The VM scalability and placement mod-els are included as proof-of-concepts. The goal isto obtain conclusions about the relation between MDmobility and DC catchment, and avoid the interfer-ence of other elements. The scenario is described indetail below.

The telecommunication infrastructure is com-posed of 16 RBSs, in a 4x4 layout. The cells aretangent but not overlapping and are dimensioned as atypical LTE micro-cell at 750 m, as detailed in (Sha-hab et al., 2013). The number of DCs varies be-tween the experiments and thus so, also the DC catch-ment defined as the ratio between DCs and RBSs,changes between (1:1) and (1:16). In abstract terms,the (1:1) catchment represents a setup with one Prox-imal DCs per RBS. In contrast, the (1:16) catchmentapproaches a more traditional case of a Remote DCserving all users in the domain.

To reveal the effects of DC catchment, all DCs areof the same capacity. The number of VMs in eachDC is scaled proportionally to the number of usersthey serve. The DC in the (1:16) catchment scenariohas 16 VMs, while the DC in the (1:1) scenario hasjust one VM. The workload is balanced among avail-able VMs, and new sessions are forwarded to the leastloaded VM. To reveal the full extent of the effect ofuser mobility, user states and requests are strictly mi-grated to the geographically nearest DC.

We use a request generation model with a ses-sion arrival rate of λses described by a Log-Normaldistribution with the parameters µ = 3 and σ = 1.1.The number of requests per session Nreq is taken froman inverse Gaussian distribution with the parametersλ = 5 and µ = 3. Inter-request time is Dreq secondsand is modelled with with an exponential distributionwith λ = 0.1. The simulation domain is populated by480 MDs, all subscribing to the same service. Dueto the size and simplicity of the network topology inthe scenario, we are deploying a Markov-based mo-bility model. The mobility model is based on a carand is as specified in Section 6.1, with parametersfrom (Bettstetter, 2001). To allow the mobility and

workload models to jointly reach a steady state, thesimulation is run for 8 simulated hours. This resultsin an average processing load of 30%, which givesenough time for migrations to complete successfully.

The user state is proportional to the aggregate sizeof that user’s sessions with the application it sub-scribes to and is defined by a 5th order AR-processwith linearly decaying parameters. In our simulation,initialisation of a VM takes tinit = 81s, similarly as form1.small VM type in Amazon EC2 (Ostermann et al.,2010). A VM is terminated if it remains in the IDLEstate longer than tidle which is equal to the mean inter-session time. It takes tterm = 21s to terminate a VM.

To investigate the influence of Proximal DC catch-ment on the performance of the telco cloud we ob-serve the life cycle of the VMs by recording theamount of time they spend in each state. We runtwo sets of experiments. In the first set, end usersare static. The second set introduces mobility.

7.2 Results

Figure 5(a) shows the breakdown of the mean timespent in each VM state in the system per DC catch-ment. With a (1:1) DC catchment the utilisation suf-fers from the proportion of time spent in IDLE statedue to the relatively low request arrival rate gener-ated by one sixteenth of all users. The inefficiencyis caused by the time the system spends in the IDLE,INITIATING, and TERMINATING states. The com-position of time spent in these states changes withDC catchment, and is a reflection of the number ofVMs in a DC and load-balancing effort. Reducingthe time spent on starting and terminating VMs wouldfree up more resources and perhaps also make the sys-tem more reactive to sudden workload changes. Themanagement of VM scalability and placement in telcoclouds is clearly something that requires optimisation.

Figure 5(b) reveals the overhead of user mobilityand the migration effort it incurs. Depending on theDC catchment, different migration dynamics comeinto play. As migrations are more frequent in the (1:1)case than in the (1:8) case, user states do not have thetime to grow as much between migrations in the for-mer case. The migration effort is therefore not a fac-tor eight lower in the (1:8) case versus the (1:1) case,but rather, they spend 26% and 47% of their time inthe MIGRATING state, respectively. The system dy-namics revealed by Figure 5(b), where at worst, 47%of the execution time is spent migrating users, pointsto the need to find scaling mechanisms for the telcocloud that take into account mobility and inactivity,so that resources can be freed dynamically for otheractive applications. A policy of strictly migrating user

(a) Static users

(b) Mobile usersFigure 5: DC catchment vs. time spent in each VM state.

states and requests to the geographically closest DC,irregardless of DC catchment, in order to obtain min-imal propagation and communication latency, is sub-optimal.

8 CONCLUSIONS

In this paper we present a way to combine existingmodels of user mobility, mobile and core networks,and DCs into a meta-model capable of capturing dy-namics of the telco cloud. We also implement a pro-totype simulator based on a simplified meta-model.

The meta-model can be used by telecommuni-cation operators as well as equipment developersto model existing infrastructures and to plan futurechanges. Researchers can test algorithms for resourcemanagement, e.g., migration of services between geo-distributed DCs. Also developers can benefit from us-ing the simulator to observe how their mobile appli-cations behave in telco cloud environments.

Future work will be focused on enhancing thefunctionality of the simulator to incorporate other pa-rameters from the presented meta-model. Then, usingthe simulator we would like to explore the followingtelco cloud challenges: minimising the trade-offs be-tween costs and performance of telco cloud depend-ing on the DC placement and capacity, and optimalplacement and migration of services between DCs.

ACKNOWLEDGEMENTS

This work is funded by the Swedish Research Council(VR) project Cloud Control and the European Union’sSeventh Framework Programme under grant agree-ment 610711 (CACTOS). Maria Kihl and WilliamTarneberg are members of the Lund Center for Con-trol of Complex Engineering Systems (LCCC) fundedby the Swedish Research Council. Also, they aremembers of the Excellence Center Linkoping – Lundin Information Technology (ELLIIT).

The authors thank Johan Eker (Ericsson Research)who contributed with feedback during several dis-cussions and Tania Lorido-Botran (University of theBasque Country) for her critical comments on earlydrafts of the paper.

REFERENCES

Ahmed, A. and Sabyasachi, A. S. (2014). Cloud computingsimulators: A detailed survey and future direction. In2014 IEEE International Advance Computing Confer-ence (IACC), pages 866–872. IEEE.

Armbrust, M., Fox, A., Griffith, R., Joseph, A. D., Katz,R., Konwinski, A., Lee, G., Patterson, D., Rabkin, A.,Stoica, I., et al. (2010). A view of cloud computing.Communications of the ACM, 53(4):50–58.

Atzori, L., Iera, A., and Morabito, G. (2010). The internetof things: A survey. Comp. net., 54(15):2787–2805.

Barker, S. K. and Shenoy, P. (2010). Empirical evaluationof latency-sensitive application performance in thecloud. In Proc. of the 1st annual ACM SIGMM con-ference on Multimedia systems, pages 35–46. ACM.

Baroncelli, F., Martini, B., and Castoldi, P. (2010). Net-work virtualization for cloud computing. Annals oftelecommunications, 65(11-12):713–721.

Bettstetter, C. (2001). Smooth is better than sharp: A ran-dom mobility model for simulation of wireless net-works. In Proc. of the 4th ACM Int. Workshop on Mod-eling, Analysis and Simulation of Wireless and MobileSystems, MSWIM ’01, pages 19–27. ACM.

Bosch, P., Duminuco, A., Pianese, F., and Wood, T. L.(2011). Telco clouds and virtual telco: Consolidation,convergence, and beyond. In 2011 IFIP/IEEE Inter-national Symposium on Integrated Network Manage-ment (IM), pages 982–988. IEEE.

Calheiros, R., Ranjan, R., Beloglazov, A., De Rose, C., andBuyya, R. (2011). CloudSim: a toolkit for modelingand simulation of cloud computing environments andevaluation of resource provisioning algorithms. Soft-ware: Practice and Experience, 41(1):23–50.

Fehske, A., Richter, F., and Fettweis, G. (2009). Energy ef-ficiency improvements through micro sites in cellularmobile radio networks. In GLOBECOM Workshops,2009 IEEE, pages 1–5.

Greenberg, A., Hamilton, J., Maltz, D. A., and Patel, P.(2008). The cost of a cloud: Research problems indata center networks. SIGCOMM Comput. Commun.Rev., 39(1):68–73.

Howell, F. and McNab, R. (1998). Simjava: A discreteevent simulation library for java. Simulation Series,30:51–56.

Kliazovich, D., Bouvry, P., and Khan, S. (2012). Green-cloud: a packet-level simulator of energy-aware cloudcomputing data centers. The Journal of Supercomput-ing, 62(3):1263–1283.

Kovachev, D. (2012). Framework for computation offload-ing in mobile cloud computing. Int. J. of InteractiveMultimedia and Artificial Intelligence, 1(7):6–15.

Munoz, V. M., Kaci, M., Gadea, A., and Salt, J. (2011).On the Economics of Huge Requirements of the MassStorage-A Case Study of the AGATA Project. InCLOSER, pages 507–511.

Ostermann, S., Iosup, A., Yigitbasi, N., Prodan, R.,Fahringer, T., and Epema, D. (2010). A performanceanalysis of EC2 cloud computing services for scien-tific computing. pages 115–131. Springer.

Papagiannaki, K., Moon, S., Fraleigh, C., Thiran, P., andDiot, C. (2003). Measurement and analysis of single-hop delay on an IP backbone network. IEEE J. onSelected Areas in Communications, 21(6):908–921.

Patel, C. D. and Shah, A. J. (2005). Cost model for plan-ning, development and operation of a data center.

Reyes-Lecuona, A., Gonzalez-Parada, E., Casilari, E.,Casasola, J., and Diaz-Estrella, A. (1999). A page-oriented www traffic model for wireless system simu-lations. In Proc. ITC, volume 16, pages 1271–1280.

Riley, G. F. and Henderson, T. R. (2010). The ns-3 net-work simulator. In Modeling and Tools for NetworkSimulation, pages 15–34. Springer.

Sakellari, G. and Loukas, G. (2013). A survey of mathemat-ical models, simulation approaches and testbeds usedfor research in cloud computing. Simulation Mod-elling Practice and Theory, 39(0):92 – 103.

Shahab, S., Kiong, T., and Abdulkafi, A. (2013). A Frame-work for Energy Efficiency Evaluation of LTE Net-work in Urban, Suburban and Rural Areas. AustralianJ. of Basic and Applied Sciences, 7(7):404–413.

Varga, A. et al. (2001). The OMNeT++ discrete event simu-lation system. In Proceedings of the European simula-tion multiconference (ESM’2001), volume 9, page 65.

Wang, A., Iyer, M., Dutta, R., Rouskas, G. N., and Baldine,I. (2013). Network virtualization: Technologies, per-spectives, and frontiers. Journal of Lightwave Tech-nology, 31(4):523–537.

Wickremasinghe, B., Calheiros, R., and Buyya, R. (2010).Cloudanalyst: A cloudsim-based visual modeller foranalysing cloud computing environments and applica-tions. In 2010 24th IEEE International Conference onAdvanced Information Networking and Applications(AINA), pages 446–452.