programmable edge-to-cloud virtualization fabric for the ... · caas container as a service co...

Project co-funded by the European Commission under the Horizon 2020 Programme.

Programmable edge-to-cloud virtualization fabric for

the 5G Media industry

D3.1 - Initial Design of the 5G-MEDIA Operations and Configuration Platform

Work Package: WP3 - Operations and Configurations Framework

Lead partner: SiLO

Author(s):

Stamatia Rizou [SiLO], Takis Athanasoulis [SiLO], Francesco Iadanza [ENG], Daniele Pavia [ENG], David Breitgand [IBM], Avi Weit [IBM], George Agapiou [OTE], David Griffin [UCL], Gino Carrozzo [NXW], Francesca Moscatelli [NXW], Ugur Acar [NET], David Lopez Meco [TID]

Delivery date (DoA): 28 February, 2018

Actual delivery date: 2 March, 2018

Dissemination level: Public

Version number: 1.0 Status: Final

Grant Agreement N°: 761699

Project Acronym: 5G-MEDIA

Project Title: Programmable edge-to-cloud virtualization fabric for the 5G Media industry

Instrument: IA

Call identifier: H2020-ICT-2016-2

Topic: ICT-08-2017, 5G PPP Convergent Technologies, Strand 2: Flexible network applications

Start date of the project: June 1st, 2017

Duration: 30 months

5G-MEDIA - Grant Agreement number: 761699 D3.1 – Initial Design of the 5G-MEDIA Operations and Configuration Platform

2

Revision History

Revision Date Who Description

0.1 January, 11th

2018

SiLO Document structure

0.2 January, 30th

2018

SiLO, NETAS, ENG, TID, NXW, IBM

Draft version for internal review

0.2 February, 6th 2018

ENG, NXW First round internal review

0.3 February, 22nd SiLO, ENG, UPM, IBM, NETAS, UCL

Address comments and revise contributions

0.4 February 26th ENG Final internal review

1.0 February 28th SiLO Integration of comments, finalization of deliverable

Quality Control

Role Date Who Approved/Comment

Internal Reviewer 6/2, 26/2 ENG Approved with minor comments addressed in the final version

Internal Reviewer 6/2 NXW Approved with minor comments addressed in the final version


3

Disclaimer

This document may contain material that is copyright of certain 5G-MEDIA project beneficiaries, and may not be reproduced or copied without permission. The commercial use of any information contained in this document may require a license from the proprietor of that information. The 5G-MEDIA project is part of the European Community's Horizon 2020 Program for research and development and is as such funded by the European Commission. All information in this document is provided "as is" and no guarantee or warranty is given that the information is fit for any particular purpose. The user thereof uses the information at its sole risk and liability. For the avoidance of all doubts, the European Commission has no liability with respect to this document, which is merely representing the authors’ view.

The 5G-MEDIA Consortium is the following:

Participant number

Participant organisation name Short name

Country

01 ENGINEERING – INGEGNERIA INFORMATICA SPA ENG Italy

02 IBM ISRAEL - SCIENCE AND TECHNOLOGY LTD IBM Israel

03 SINGULARLOGIC ANONYMI ETAIREIA PLIROFORIAKON SYSTIMATON KAI EFARMOGON PLIROFORIKIS

SiLO Greece

04 HELLENIC TELECOMMUNICATIONS ORGANIZATION S.A. - OTE AE (ORGANISMOS TILEPIKOINONION TIS ELLADOS OTE AE)

OTE Greece

05 CORPORACION DE RADIO Y TELEVISION ESPANOLA SA RTVE Spain

06 UNIVERSITY COLLEGE LONDON UCL

United Kingdom

07 TELEFONICA INVESTIGACION Y DESARROLLO SA TID Spain

08 UNIVERSIDAD POLITECNICA DE MADRID UPM Spain

09 INSTITUT FUER RUNDFUNKTECHNIK GMBH IRT Germany

10 NEXTWORKS NXW Italy

11 ETHNIKO KENTRO EREVNAS KAI TECHNOLOGIKIS ANAPTYXIS CERTH Greece

12 NETAS TELEKOMUNIKASYON ANONIM SIRKETI NET Turkey

13 INTERINNOV SAS IINV France

14 BITTUBES GMBH BIT Germany


4

Table of Contents

EXECUTIVE SUMMARY ..................................................................................................... 10

1. INTRODUCTION ........................................................................................................ 12

1.1. SCOPE OF THE DELIVERABLE .......................................................................................................................12

2. OVERALL 5G-MEDIA ARCHITECTURE ......................................................................... 13

3. SPECIFICATIONS FOR 5G-MEDIA SERVICE VIRTUALIZATION PLATFORM ..................... 15

3.1. SPECIFICATIONS FOR OPEN SOURCE MANO .................................................................................................16

3.1.1 Specifications for Service Orchestrator, VNF Configuration and Abstraction and Resource Orchestrator .................................................................................................................................................19

3.1.2 Specifications for Catalogues and Repositories ..............................................................................21

3.1.3 Specifications for Monitoring module ............................................................................................23

3.1.4 5G-MEDIA extensions and adaptations ..........................................................................................25

3.2. SPECIFICATIONS FOR THE MEDIA SERVICE MAPE ...........................................................................................30

3.2.1 QoS/QoE Monitoring Service ..........................................................................................................31

3.2.2 Cognitive Network Optimizer .........................................................................................................31

3.2.3 5G-MEDIA adaptations and extensions ..........................................................................................32

4. SPECIFICATIONS FOR 5G-MEDIA EDGE-TO-CLOUD HIERARCHY AND INSTANTIATED NFVIS/VIMS ..................................................................................................................... 36

4.1. ENGINEERING CLOUD - FIWARE PLATFORM .................................................................................................36

4.2. OTE INTEGRATED INFRASTRUCTURE .............................................................................................................38

4.2.1 VMware cloud ................................................................................................................................38

4.2.2 OpenStack cloud .............................................................................................................................39

4.3. TELEFONICA CLOUD – ONLIFE PLATFORM ......................................................................................................42

4.4. INTEGRATION OF GPUS IN 5G-MEDIA NFVIS ..............................................................................................44

4.4.1. Integration of GPUs into the cloud environment ............................................................................44

4.4.2. GPU provision over containerization technology ...........................................................................46

4.5. MULTI-POP (INTER-NFVI) NETWORK CONNECTIVITY ......................................................................................50

4.5.1 Multi-type VIMs ..............................................................................................................................50

4.5.2 Multi-region OpenStack..................................................................................................................52

4.5.3 Multi-region OpenStack in hybrid NFVI ..........................................................................................54

5. SPECIFICATIONS OF VIMS SUPPORTED BY 5G-MEDIA PLATFORM .............................. 55

5.1. INTEGRATION WITH OPENNEBULA ...............................................................................................................55


5

5.2. FAAS VIM ARCHITECTURE AND SPECIFICATIONS .............................................................................................57

5.2.1 Background.....................................................................................................................................57

5.2.2 FaaS VIM Architecture ....................................................................................................................58

CONCLUSIONS ................................................................................................................. 67

ANNEX I – INSTALLATION STEPS OF OSM V3.0 .................................................................. 68

ANNEX II – DATA MODEL FOR VNFDS IN OSM V3.0 ........................................................... 70

ANNEX III – DATA MODEL FOR NSDS IN OSM V3.0 ............................................................ 74


6

List of Figures

Figure 1: Architecture of 5G-MEDIA platform.......................................................................... 14

Figure 2: Integration of OSM MANO in 5G-MEDIA architecture ............................................. 16

Figure 3: Data flow of 5G-MEDIA monitoring services ............................................................ 24

Figure 4: Data flow and specifications of Media Service MAPE ............................................... 31

Figure 5: Physical layer and NFVIs in 5G-MEDIA project .......................................................... 36

Figure 6: FIWARE Vicenza node in ENG cloud .......................................................................... 37

Figure 7: OTE VMWare component architecture..................................................................... 39

Figure 8: OTE Open Stack all component in one Infrastructure .............................................. 40

Figure 9: Cloud architecture that based on OpenStack ........................................................... 41

Figure 10: OTE Lab mobile infrastructure ................................................................................ 42

Figure 11: Integrated infrastructure in OTE’s lab ..................................................................... 42

Figure 12: Components of CORD .............................................................................................. 43

Figure 13: PCI passthrough in virtual environment ................................................................. 45

Figure 14: NVidia-docker high-level architecture .................................................................... 47

Figure 15: NVidia-docker internal architecture ........................................................................ 48

Figure 16: Container orchestrators support to GPU (Source: OpenStack Summit 2017 ......... 49

Figure 17: Interconnecting multiple NFVI-PoPs ....................................................................... 51

Figure 18: Architecture of Tricircle ........................................................................................... 53

Figure 19: Kingbird architecture ............................................................................................... 54

Figure 20: Kingbird centralized deployment over different OpenStack multi-region PoPs ..... 54

Figure 21: Integration between OpenNebula and OSM .......................................................... 55

Figure 22: Plugin of OpenNebula in RO of OSM ....................................................................... 56

Figure 23: Use of python-oca library for XMLRPC OpenNebula Cloud API .............................. 57

Figure 24: Reference Architecture for Integration of FaaS Frameworks with 5G-MEDIA ....... 58

Figure 25: Software architecture for FaaS integration within 5G-MEDIA Platform................. 60

Figure 26: FaaS VNF image upload and on-going management .............................................. 61

Figure 27: Onboarded FaaS VNFD for a VNF vTranscoder being part of Star Ball Game VNS ....................................................................................................................................... 62

Figure 28: Onboarded VNS Star Ball Game that comprises two vTranscoder FaaS VNFs interconnected by a virtual link ........................................................................................... 63

Figure 29: A User selects a VNS to instantiate ......................................................................... 64


7

Figure 30: OSM view of a running FaaS VNS ............................................................................ 64

Figure 31: Status of a running FaaS VNS .................................................................................. 65

List of Tables

Table 1 - Specifications of LXD containers of OSM Specifications of LXD containers of OSM ..................................................................................................................................... 17

Table 2 - CLI commands in OSM ............................................................................................... 17

Table 3 - SAUvo_SACbr REST API from SELFNET ...................................................................... 22

Table 4 - List of requests and responses through the message bus of the monitoring module ................................................................................................................................. 24

Table 5 - The list of normalized metrics supported by the OSM Monitoring module per VIM ....................................................................................................................................... 25

Table 6 – List of normalized alarms supported by OSM release THREE. ................................. 27

Table 7 - Specifications of the Media Service MAPE component ............................................ 33

Table 8 - NFVI – PoP nodes ....................................................................................................... 41

Table 9 - ETSI Standard: vnfd:vdu information related to hypervisors .................................... 66


8

Definitions and acronyms

AAA Authentication, Authorization, and Accounting

BSS Business Support System

CaaS Container as a Service

CO Central Office

CNO Cognitive Network Optimization

CLI Command Line Interface

CTpd Central Telefónica Procesadora de Datos

FaaS Function As A Service

GPU Graphical Processing Unit

IaaS Infrastucture as a Service

KPI Key Performance Indicator

NBI North-Bound Interface

NFV Network Function Virtualization

NFVI Network Function Virtualization Infrastructure

NFVO Network Function Virtualization Orchestrator

NS Network Service

NSD Network Service Descriptor

NVML NVidia Management Library

OCP Open Compute Project

OSM Open Source MANO

OSS Operation Support System

PaaS Platform as a Service

PCI Peripheral Component Interconnect

PNF Physical Network Function

PoP Point of Presence

RBAC Role-Based Access Control

RO Resource Orchestrator

SBI South-Bound Interface

SDK Service Development Kit


9

SDN Software-Defined Networking

SLA Service Level Agreements

SO Service Orchestrator

SVF Serverless Virtual Function

SVP Service Virtualization Platform

VCA VNF Configuration and Abstraction

VDU Virtualization Deployment Unit

VF Virtual Functions

VIM Virtualized Infrastructure Manager

VLD Virtual Link Descriptors

VNF Virtual Network Function

VNFC VNF Components

VNFD Virtual Network Function Descriptor

VNFFG VNF Forwarding Graph

VoD Video on Demand

WIM WAN Infrastructure Manager


10

Executive summary

As one of the most demanding 5G-enabling verticals in terms of ultra fast transmission speed and low latency, media related applications will gain great value from the significant innovations and technical improvements in network management capabilities of 5G technologies.

The 5G-MEDIA project will extend the 5G-PPP ongoing work on NFV and SDN key technologies to offer an advanced management environment for network services and media-related applications that directly links their online lifecycle management with user experience and decisions in resources and infrastructures usage optimization. In this scope, 5G-MEDIA Operations and Configuration Platform aims to establish an open source, ETSI-NFV compliant, agile programming orchestrating and verification DevOps platform for network media services and applications, accompanied by a large set of open source network services and functions, responding to the needs of the media network challenges of H2020 according to the 5G vision.

The aim of this document is to provide the initial design and specifications of the 5G-MEDIA Operations and Configuration Platform from a technical and user experience point of view. The document builds upon the architectural specifications of the 5G-MEDIA platform carried out within Work Package 2 (“Architecture Analysis and Tools”) and represents the baseline for subsequent development activities in Work Package 3 and the rest deliverables during the project lifetime therein.

The first chapter provides an overview of the purpose of this deliverable, its relation to the rest deliverables of the project and its structure.

The second chapter introduces the initial architecture of the 5G-MEDIA ecosystem and briefly discusses the role and responsibilities of its main parts, in conformance with the KPIs that have been set by the beginning of the project with respect to the number of components integrated from other 5G-PPP projects, the accommodation of several NFVIs and VIMs, beyond those supported by the MANOs.

The third chapter presents the technical specifications and the architectural design of the 5G-MEDIA Service Virtualization Platform (SVP), which is the central component of the 5G-MEDIA architecture. SVP leverages on ETSI compliant Open Source MANO (OSM) stack to orchestrate resources and realize a multi-technology environment, where several cutting edge open source and commercial technologies, such as OpenStack, VMWare, Docker, unikernels, LXD containers and OpenWhisk serverless computing, are integrated with prominent software solutions of 5G-PPP phase 1 projects, such as the Service Development Kit (SDK) of SONATA, the NS/VNFs Catalogues and Repositories of SELFNET and the Monitoring-Analysis-Planning-Execution (MAPE) loop of CogNet. Beyond providing the specifications of the various components, interfaces and infrastructure to realize the 5G-MEDIA SVP, the analysis herein emphasizes especially on the extensions, adaptations and innovations which are delivered by the 5G-MEDIA project over the used enabling technologies to meet its business and use case requirements. In particular, sections 3.1.4 and 3.2.3 provide a clear view on the extensions and adaptations related to: 1) the addition of active monitoring metrics to the OSM monitoring module that are required by the 5G-MEDIA use cases to support intensive networking


11

characteristics, such as high bandwidth and low latency measurements and thus support SLA and QoE/QoS, 2) the integration of metrics related alerts in order to trigger actions on the NFVI and thus optimize the performance of the running network services, 3) specific extensions to OSM related to scaling groups capabilities and VCA configuration, 4) integration of websockets to provide real-time monitoring data to Cognitive Network Optimization components, 5) extension and adaptation of VNFFG characteristics supported by OSM, and 6) support of unikernels for fast deployment of microservice-based VNFs, 7) adaptation of CogNet components and tools related to MAPE, 8) extensions to Machine learning algorithms provided by CigNet or open source in order to fulfill the requirements of the 5G-MEDIA use cases, 9) integration with SELFNET Catalogue and functionalities.

Fourth chapter presents the specifications for the hierarchical, edge-to-core cloud environment and the various NFV Infrastructures used in 5G-MEDIA project to support the considered scenarios and use cases. The technical specifications of the three cloud environments used for demonstration purposes are analyzed (in ENG, TID and OTE premises) and the way the 5G-MEDIA project realizes multi-NFVI network connectivity is presented. Finally, the chapter discusses how GPUs, as a technology offering one order of magnitude higher computational power compared to traditional CPUs, are integrated into the resources provided by the 5G-MEDIA cloud computing platforms to address demanding needs of media-related and entertainment services and applications.

Fifth chapter discusses the technical aspects of two main innovations delivered by the 5G-MEDIA project constituting potential contributions to the OSM community, i.e. the integration of the OpenNebula and OpenWhisk Function as a Service (FaaS) VIMs into the SVP, as enablers of integration with OnLife cloud virtualization platform and serverless computing paradigm, respectively.

Finally, the sixth and final chapter summarizes key conclusions of the deliverable.


12

1. Introduction

The goal of this deliverable is to provide the initial design of the 5G-MEDIA Operations and Configuration Platform, which constitutes the backbone of the 5G-MEDIA service platform and enables the management and orchestration of media services through the implementation and integration of the MANO framework and the extension of the Service Virtualization Platform with the seamless integration with external proprietary and/or open source systems and infrastructures.

The core of the 5G-MEDIA Operations and Configuration Platform is the Service Virtualization Platform (SVP). The implementation of the 5G-MEDIA SVP will build upon the Open Source Mano (OSM) v3.0 NFV-MANO stack and will focus on the development, adaptation and extension of the following components:

NFV MANO Service Orchestrator: This component will contain the NSD/VNFD catalogues and NS/VNF repositories customized to 5G-MEDIA project needs as well as the Network Service Orchestrator and the VNF Manager.

Media Service MAPE (Monitoring-Analyze-Planning-Execute): This component will realize a MAPE loop for the optimization of media services in terms of network requirements, e.g., latency, throughput etc. The 5G-MEDIA project will focus on the implementation of the Cognitive Network Optimizer (CNO), which represents the intelligence module integrated in the MAPE loop.

NFV MANO Resource Orchestrator: This component will contain the mechanisms for the deployment of media services on the Network Function Virtualization Infrastructures (NFVIs). The 5G-MEDIA project will provide two new Virtualized Infrastructure Manager (VIMs), in addition to those supported by OSM, i.e. the Function as a Service (FaaS) VIM, which enables the use of the so-called FaaS concept and the OpenNebula VIM, which enables the connection of OSM to OnLife NFVI provided by TID.

Last but not least, one of the important 5G-MEDIA goals to be achieved also in this Work Package is the integration of GPUs in the 5G-MEDIA Service Virtualization Platform, providing computational power that exceeds by far the one offered by CPUs.

The 5G-MEDIA SVP will enable the connection to different NFVIs, which will host the media services of the 5G-MEDIA project. Currently, we foresee that three different NFVIs provided by the project partners (ENG, OTE and TID) will be used in the frame of the project. In this line, we provide an overview of the characteristics of each NFVI and a reference instantiation of the proposed 5G-MEDIA platform.

1.1. Scope of the Deliverable

This deliverable focuses on the expected outcomes of Task 3.1 “MANO Framework and NFVI Interoperability” and provides an initial design of the core 5G-MEDIA SVP components (Service Orchestrator, Resource Orchestrator). The detailed description of the Media Service MAPE is not part of this deliverable since it will be discussed in “D3.3 Specification of the 5G-MEDIA QoS Control and Management Tools” but a brief description for its scope, architectural design and specifications is also provided as reference.


13

In addition, to keep the deliverable self-contained, we include an overview on the overall 5G-MEDIA architecture. Details on the assessment of background technologies and the design of the platform architecture will be part of “D2.3 5G-MEDIA Platform Architecture”.

2. Overall 5G-MEDIA Architecture

The 5G-MEDIA approach delivers an integrated programmable service platform for the development, design and operation of media applications in 5G networks. Figure 1 shows the high-level architecture of the 5G-MEDIA operations and configuration platform. The main building blocks comprising the 5G-MEDIA architecture are:

An Application/Service Development Kit that enables access to media applications development tools, such as editor, validator, emulator, etc.

A Service Virtualization Platform (SVP) that hosts the components related to the ETSI MANO framework1, the Virtual Network Functions and the Media Application Catalogue and Repository, the Media Service MAPE and the corresponding Virtualized Infrastructure Manager (VIM) and WAN Infrastructure Manager (WIM) plugins enabling the integration to different NFVI platforms.

Different Network Function Virtualization Infrastructures (NFVIs) comprising the “Physical Layer” that provide computing resources by different operators and supporting different cloud technologies to host generic and media-specific VNFs depicted at the “Virtualized Resource Layer”.

The Application/Service Development Kit (5G-MEDIA SDK) provides a set of open source tools to support the rapid development of media applications using a DevOps approach. In particular, these tools allow the definition of Media Service forwarding graphs (also using already existing VNFs, stored in the 5G-MEDIA VNFD/NSD Catalogue), to proof and package the various functions, as well as to emulate behaviors of the virtualized infrastructure, to accelerate application development and provide a testing environment to be utilized prior to service deployment in the runtime SVP. In addition, the 5G-MEDIA SDK tools enable the use of the innovative concept of the Function-as-a-Service (FaaS). Particularly, 5G-MEDIA pioneers in applying FaaS to VNF management, complementing traditional VM based VNFs with FaaS based media specific functions, aiming at dramatically reducing development cycles and slashing operational costs to 5G-MEDIA users. In this perspective, developers do not have to care about the low-level details related to the virtual computing and storage infrastructure (e.g. virtual server profiling in terms of CPU, RAM, etc.), thus drastically contributing to reduce the service creation time cycle and maintenance effort. In this line, the service developers will be able to create the so-called FaaS VNFs, i.e., VNFs that are instantiated upon the detection of specific events or VNFs whose events trigger specific actions on other components. The combination of the FaaS approach with the VNF packaging and the enablement of inserting FaaS VNFs in a typical VNF forwarding graph is one of the main innovation aspects of the proposed 5G-MEDIA approach.

The 5G-MEDIA SDK interacts with the Service Virtualization Platform (5G-MEDIA SVP), which hosts the components related to the ETSI MANO framework (NFV Orchestrator, VNF

1 http://www.etsi.org/technologies-clusters/technologies/nfv/open-source-mano


14

Manager(s), Infrastructure Manager(s), the VNF/NS Catalogue and Repositories etc.), as well as unique components that are used to support 5G-MEDIA internal services and deliver its applications to the external stakeholders (e.g. the Media Service MAPE).

Figure 1: Architecture of 5G-MEDIA platform

A major innovation of the project is the development of the Media Service MAPE component, which is composed of the Monitoring system, the Cognitive Network Optimization (CNO)


15

engine and the Policy Manager. The development of MAPE will be based, where applicable, to the design, utilization of tools and architectural diagram being part of the outcomes of CogNet 5G-PPP phase 1 project, while achieving integration to OSM components, including Monitoring plugin and Service Orchestrator. The monitoring services aims to monitor the performance/behavior of the instantiated Network Services (NSs), the integrated NFVIs, the interconnecting networks and the applications themselves. The measured performance metrics are directly used by the CNO engine which comprises mechanisms that take advantage of Machine Learning (ML) techniques and optimization policies to trigger the dynamic instantiation and update of VNF Forwarding Graphs (VNFFGs) or scaling options according to the capabilities provided by OSM (i.e. Scaling Groups construct in the NS descriptors) over the different NFVIs. Alternatively, after the proper execution of the policy classification procedure, 5G-MEDIA SVP will be also able to take advantage of the benefits provided by the VCA Engine (and in particular the Juju adapter provided by OSM) to execute commands on the VNFs themselves, according to the specific use case scenario needs. As it becomes clear, the CNO is able to respond on dynamic changes of the environment (e.g., location change of end users, varying QoS demands) and in cooperation with the Policy Manager to make suggestions to the MANO NFVO and VNFMs how to manage/update VNFFGs, perform scaling options, execute commands via Juju charms or even start new components enforced by the FaaS VNFs in order to meet expected QoS/SLA requirements in the most effective way. Last but not least, the QoS/QoE monitoring modules are able to provide both NSs individual and aggregated monitoring views and graphical user interfaces for metrics specified by the developer per NS/VNF, which are part of the generic monitoring system of the SVP.

In the context of 5G-MEDIA approach, as depicted in Figure 1 at the “Virtualized Resource Layer”, each media service comprises a chain of media-specific and network-specific atomic services, all interconnected to deliver an expected output to the end user (media consumer). The resulting graph (i.e., a graph where nodes refer to computing tasks and edges refer to network communication links) is referred as Media Service Forwarding Graph. During the design, development and testing phases, the 5G-MEDIA platform provides appropriate programming tools to abstract the details of the underlying 5G infrastructure and allow developers to focus on the functionality of the services. Once the media service is deployed in the virtualized infrastructure, the 5G-MEDIA platform provides mechanisms to flexibly adapt service operations to dynamic conditions and react upon events (e.g. to transparently accommodate auto-scaling of resources, VNF re-placement, etc.).

Finally, in terms of the supporting physical infrastructure (Physical Layer), the proposed architecture considers that several cloud-based NFVIs will be connected to the Service Virtualization Platform via the appropriate VIM and WIM components, enabling the use of computing and networking resources in the respective NFVI.

3. Specifications for 5G-MEDIA Service Virtualization Platform

In this section, the architectural design of the 5G-MEDIA SVP is presented and the specifications of its main components are identified, i.e. the NFV MANO and the MEDIA Service MAPE.


16

3.1. Specifications for Open Source MANO

The 5G-MEDIA SVP leverages on Open Source MANO (OSM) v3.0 to realize functionalities of the NFV-MANO stack2. In Figure 2, it is shown how OSM is integrated into the reference architecture of the 5G-MEDIA SVP. The components in purple color, and the corresponding services, are parts of OSM stack while the components and services in blue color are innovations, adaptations and extensions to be developed and integrated in the scope of 5G-MEDIA.

Figure 2: Integration of OSM MANO in 5G-MEDIA architecture

In 5G-MEDIA project, OSM has been installed in the core cloud of the Physical layer (see reference architecture in Figure 1) which is hosted in the ENG cloud. The Devstack installation comprises of 8vCPUs, 16GB RAM, and 80GB disk, running OS Ubuntu 16.04 (64 bit). The detailed installation steps are provided in Annex I. As a result of the installation, three LXD containers are created in the host VM, namely Resource Orchestrator (RO), VNF Configuration and Abstraction (VCA), and Service Orchestrator (SO-ub), with the specifications shown in Table 1:

2 https://osm.etsi.org/


17

Table 1 - Specifications of LXD containers of OSM Specifications of LXD containers of OSM

NAME STATE IPV4 TYPE

RO RUNNING 10.207.216.65 (eth0) PERSISTENT

SO-ub RUNNING 10.207.216.75 (eth0) PERSISTENT

VCA RUNNING 10.44.127.1 (lxdbr0)

10.207.216.190 (eth0)

PERSISTENT

Open Source MANO provides a Northbound REST API3 that is intended to allow the invocation of several actions by external systems, such as an Operation Support System (OSS), SDK and the Media Service MAPE. This API supports all the operations that OSM provides and it is the unique entry point for all the interactions with the system (OSS, UI and CLI). The SO is in charge of triggering all actions and requests to the rest of OSM components, e.g. any action requiring direct access to VCA and RO APIs (and the internal elements and services). In this way, for instance, access to the following entities of the RO is provided4:

● Tenants: Intended to create groups of resources. In OSM v3.0 no security mechanisms are implemented.

● Datacenters: Represents the VIM information stored by OpenMano. ● VIMs: used to perform an action over a datacenter (specific pool of resources) ● VNFs: SW-based network function, composed of one or several VMs that can be

deployed on an NFV datacenter. ● Scenarios: topologies of VNFs and their interconnections. ● Instances: Each one of the Scenarios deployed in a datacenter.

The OSM Client provides a Command Line Interface (CLI) client to remotely interact with the Northbound REST API. The client gives a number of endpoints to onboard, instantiate and terminate NSs/VNFs and interact with VIMs, the most important of which are presented in Table 25.

Table 2 - CLI commands in OSM

Command Description

config-agent-add Add a new configuration agent

config-agent-delete Delete an existing configuration agent

config-agent-list List of configuration agents (i.e OSM Juju)

3 OSM RELEASE THREE, A TECHNICAL OVERVIEW, Oct. 2017 [https://osm.etsi.org/images/OSM-Whitepaper-TechContent-ReleaseTHREE-FINAL.PDF], fig. 4, p.20

4 https://osm.etsi.org/wikipub/index.php/RO_Northbound_Interface

5 https://osm.etsi.org/wikipub/index.php/OsmClient


18

Command Description

ns-create Instantiate a new NS

ns-delete Delete an instantiated NS

ns-list List of the running NSs (in NFVI)

ns-monitoring-show Display monitoring details for an instantiated NS

ns-scale Scale NS based on the corresponding scaling group descriptor

ns-scaling-show Display details for a specific scaling group descriptor

ns-show Display details for a specific running NS

nsd-delete Delete an existing NS descriptor (included in catalogue)

nsd-list List of the NS descriptors (included in catalogue)

ro-dump

upload-package Upload a VNF package (tar.gz)

vcs-list List of internal services and their status

vim-create Register a new VIM

vim-delete Delete an existing VIM

vim-list List of the integrated VIMs

vim-show Display details for a specific integrated VIM

vnf-list List of the running VNFs (in NFVI)

vnf-monitoring-show Display monitoring details for a specific running VNF

vnf-show Display details for a specific running VNF

vnfd-delete Delete an existing VNF descriptor (included in catalogue)

vnfd-list List of the VNF descriptors (included in catalogue)

Apart from CLI, a GUI (Launchpad) is also provided to accelerate NS design, VNFD/NSD creation as a catalog item, VNFD/NSD onboarding to the catalog, VNF/NS instantiation and user management.


19

In the following subsections, the specifications of the 5G-MEDIA MANO are described in relation with capabilities and specifications of the OSM technology. In the last subsection, a number of new extensions and adapted functionalities for OSM are summarized as they are planned to be developed in the scope of 5G-MEDIA.

3.1.1 Specifications for Service Orchestrator, VNF Configuration and Abstraction and Resource Orchestrator

The overview of 5G-Media user management and the low-level specifications of the three main modules of the 5G-MEDIA MANO (i.e. SO, VCA and RO) are provided below.

User Management

OSM Release Three has a number of advancements in terms of SVP security. Although OSM community considers security to represent a much broader set of topics, OSM Release Three has the focus of access control security and provides extensions of role-based access control (RBAC) and multi-project management.

In terms of RBAC, the OSM SO requires a significant set of capabilities and privileges to perform all its required tasks such as VNF on-boarding, NS design on-boarding, NS deployment, NS shutdown, addition of new datacenters/VIMs, etc. However, not all of these tasks are expected to be performed by the same user or type of user in the organization. Each of the stages in the lifecycle of a VNF or NS may have different implications in terms of service continuity, validation, access to credentials, etc. OSM Release Three allows the definition of different roles, defined by admin user, with different sets of privileges. All users are mapped to at least one of these roles.

Secondly, OSM has the concept of multi-project access control. A “project” is used to group things such as VNFDs, NSDs, NS instances, and datacenters (VIMs). The RBAC can be applied on the project definition to apply consistent access to the defined grouping of resources that OSM manages. Users have roles and can belong to more than one project. Each user is associated with one or more projects and each user has one or more roles on each project. The role is system-wide and is defined based on permissions on API endpoints. Permissions at some VIMs are restricted by the admin of the infrastructure. This is particularly true of public cloud infrastructure. In general, OSM would not have sufficient rights to create or edit tenants at VIM level. The use of the project definition enables more flexibility in grouping access to resources than if OSM allowed a VIM tenant view to permeate up the MANO stack. During the creation of OSM projects, the admin should indicate in which datacenters the new OSM project is authorized. Per datacenter, the credentials and the VIM tenant should be provided.

Media function developer, media application developer, media service developer, SVP operator are different user roles for SVP specified on D2.2. In 5G-MEDIA, the authentication and authorization of the users to specific services of the SVP are named as “User Management”. Since the OSM Release Three provides a GUI (Launchpad) for user management, new roles and specifications will be defined on further iterations adjusting the existing roles from the one provided by OSM.


20

OSM Service Orchestrator

In 5G-MEDIA, media applications (called also as network services) are defined as chains of one or more VNFs, the orchestration of which realizes the desirable end-to-end functionality. Each such chain is constructed and maintained by the SO module based on the information included in the corresponding Network Service Descriptor (NSD) regarding the following aspects (a detailed description of the information model defined in OSM about NSDs is provided in Annex II):

● The VNFs which are composing the service chain. ● The Physical Network Functions (PNFs), e.g. GPUs, which are used in the service chain. ● The Virtual Links Descriptors (VLDs) which describe the resource requirements needed

for links between VNFs, PNFs and the endpoints of the network service. ● The VNF forwarding graph (VNFFG) information which determines the traffic flow and

behavior over the service chain.

Summarizing the specifications of the SO in 5G-MEDIA, this module aims at supporting the following operations:

● NS/VNF instantiation: deployment of NS/VNF instances, according to the lifecycle events defined in the corresponding NSDs/VNFDs. In 5G-MEDIA, it is possible to instantiate NSs over a hierarchical and distributed NFVI (e.g. distributed NSs), where different PoPs can be managed and operated by (at least) three different types of VIMs, i.e. OpenStack, VMWare and OpenNebula.

● NS/VNF configuration: modifications in the configuration of NSs/VNFs through their descriptors. This can be done either prior to the instantiation of a NS/VNF or as an active process while the NS/VNF is running.

● NS/VNF performance monitoring: Several different performance’s metrics from the computing instances and the network/virtual links (e.g. bandwidth, latency etc.), as they are collected by the SVP monitoring module, are provided to the SO to track critical KPIs and trigger corrective actions.

● NS/VNF scaling: increase/decrease of the NSs/VNFs instances according to the scaling policies defined per NSDs and VNFDs. This may result to creation/termination of VNF instances or updating the virtual links over them.

● VNFFG updates: update VNFFGs based on VNFFGDs and also recommendations provided by the Media Service MAPE CNO engine. This may result to re-ordering VNF list and modifying traffic routes over it.

● NS/VNF termination: release any given NS/VNF instance and its associated resources.

Service Orchestrator interfaces a Catalogue and a Repository (the specs of which are presented in §3.1.2) aiming at persistently storing information about onboarded and running VNFs and NSs, namely the public NS/VNF Catalogue and the NS/VNF Repository.

VNF Configuration and Abstraction

The VCA layer has the responsibility to enable configurations, actions and notifications to/from the VNFs and/or Element Managers. Following ETSI NFV directives, in 5G-MEDIA each VNF is composed by a set of discrete software components, called VNF Components (VNFCs),


21

where each one realizes a full description of some software functionality that is appropriately packetized as JUJU charms6, and it is onboarded at runtime into a VM image or a container. The way the VCA handles VNFs during both their instantiation and lifecycle management is described in the corresponding VNFDs, containing information with respect the following aspects:

● VNF images. ● Connectivity (connection points and virtual links), interface, and KPI requirements that

can be used by MANO functional blocks to establish appropriate virtual links, either over the VNFCs of the same VNF or between a VNF instance and the outside network and the other VNFs of the network service.

● VDUs that specify the VM/VNFC compute, storage, and network requirements. ● Platform resource requirements, such as CPU, memory, interfaces, and network. ● Special characteristics related to Enhanced Platform Awareness (EPA) attributes and

performance capabilities. ● Scaling properties.

A detailed description of the requirements and capabilities of VNF descriptors in OSM are provided in Annex III.

OSM Resource Orchestrator

Resource Orchestrator enables the interaction between the SO and the NFVIs by managing resources as required and interacting with multiple VIMs and SDN controllers. In OSM v.3, the RO is implemented by OpenMano and it has already been integrated with Openstack, VMware, Openvim and Amazon Web Services (AWS) VIMs. 5G-MEDIA aims to extend several RO functionalities of OSM to support the following operations:

● Multi-PoP deployment: the RO would be able to deploy the VNFs across different PoPs and domains, controlled either by the same or different NFVIs/VIMs (see §0).

● PNF integration: ability to deploy a NS that requires the deployment of one or more network functions on physical machines (i.e. GPUs).

● Support of multiple virtualizations technologies: support the deployment of VNFs on multiple virtualization technologies, including VMs, containers and unikernels.

● Computational resilience: automatic re-provisioning of resources over server failure or network disruption.

● Serverless support: deploy, run and manage VNFs packetized under the FaaS computing model.

3.1.2 Specifications for Catalogues and Repositories

One of the existing solutions that have been studied in order to create and instantiate VNFs/NSs into the catalogue of the 5G-MEDIA MANO block is the 5GPPP Phase 1 Project SELFNET7. One of the main reasons to choose this platform is that it is a 5GPPP Phase 1 Project

6 https://osm.etsi.org/wikipub/index.php/Creating_your_own_VNF_charm_(Release_THREE)

7 https://selfnet-5g.eu/


22

so, in order to achieve our goal of using existing 5GPPP solutions, SELFNET could help. This allows us to create an integrated catalogue using some tools from SELFNET over the OSM catalogue.

Talking about the SELFNET tools that 5G-MEDIA could use, this platform provides a set of REST APIs interfaces in order to perform the VNF packages management. Concretely, the REST API SAUvo_SACbr allows the following operations:

Table 3 - SAUvo_SACbr REST API from SELFNET

Operation Description

Onboard The content of the APP Package tar archive file is extracted and matched against the defined information model. Package content is validated, parsed, stored and notified through message bus. The unique app-id is returned by this operation.

Enable Set the package status to ENABLED so it is available for instantiation.

Disable Set the package status to DISABLED so it is not available for instantiation.

Update It updates the content of the APP Package tar archive to a new version.

Offboard It removes the APP Package from the platform. The Package has to be on DISABLED status in order to be offboarded. Then, all the information related to the APP Package is removed from the catalogue database.

Query It returns all the package content information in a JSON file.

In accordance with the operations shown in Table 3, SELFNET proposes the following APP specifications:

● metadata.json: it includes generic onboarding information about the app. This means, at least, family, class, type, name and version; as well as some more optional information.

● app-descriptor/app-d.json: it includes the VNFD. SELFNET has its own VNFD data structure specified on its D2.3. Since the catalogue used is OSM r3.0, this specifications are presented in Annex II.

● configuration/configuration.json: it includes parameters and actions to model the management and control of the APP behaviour.

● monitoring/monitoring.json: provides some metrics exposed by the sensor APPs. This information could be used by the monitoring module.

SELFNET platform has also a query REST API in order to retrieve individually specific information. The information that could be retrieved is:

● List of onboarded apps, including some additional information. ● List of onboarded apps filtered by their class, including some additional information. ● Status of a single APP: DISABLED or ENABLED. ● Metadata json of a single APP. ● Descriptor json of a single APP. ● Configuration json of a single APP.


23

● Monitoring json of a single APP. These are the basic characteristics from SELFNET platform and the two basic interfaces that are intended to be leveraged by 5G-MEDIA. Since the SVP choice is based on the OSM release three, new interfaces and specifications will be deployed on further iterations adjusting the catalogue from the one provided by OSM.

The two key parts of the GUI (Launchpad) in OSM that relate to a design-time focus are the VNF Package Generator and the VNF/NS Catalog Composer. The VNF Package Generator is a tool that assists VNF providers to create a properly formed package for on-boarding in OSM. The VNF/NS Catalog Composer is a model-driven GUI (Launchpad) that supports both VNF providers and an Operator’s Network Service designers to rapidly develop descriptors that accurately represent the essence of the entity being modelled for deployments. After adjusting the SELFNET catalogue from the one provided by OSM, the Launchpad application will also be adapted in case of any modifications in the REST APIs of the catalogue in the scope of 5G-MEDIA Package Management.

3.1.3 Specifications for Monitoring module

In October 2017, the latest release of OSM has been announced, providing many features missing from the previous releases. One of them was the addition of the Monitoring module8 (although in experimental version). Based on the decision of OSM Community not to develop and implement another monitoring solution, OSM provides interfaces to already existing monitoring tools for all supported VIMs, namely OpenStack, VMWare and AWS. In this perspective, OSM Monitoring module is not intended to replicate or compete with these systems, but to provide the means for easy integration under a commonly agreed data model.

In the current release, OSM supports a flexible, plugin-based method for integrating several external (existing) monitoring tools. In particular, it provides plugins for Aodh and Gnocchi for OpenStack, vRealize Operations Manager for VMWare and CloudWatch for AWS. On top of the plugins, a well-defined Information Model parses incoming data and represents them in the format accepted by the internal modules of OSM, e.g. SO and/or VCA while communicating with the Monitoring module.

The data flow of 5G-MEDIA monitoring services is shown in Figure 3. The OSM Monitoring module executes two main functions: the first one is its ability to interact with the external monitoring tool in order to perform configuration updates related to monitoring metrics and alerting rules. The second one is the ability to steer actionable events to the Service Orchestrator. Moreover, the monitoring module is capable of correlating monitoring data related to the VNFs that consist a NS and thus provides an improved user experience compared to other solutions. It is highlighted that the module includes an instance of an Apache Kafka that is used as the message bus for the communication of all sub-modules of Monitoring. Kafka provides a publish-subscribe model, based on topics that interested users subscribe to.

8 https://osm.etsi.org/wikipub/index.php/OSM_MON_Module_Installation_Guide_(Release_THREE)


24

Figure 3: Data flow of 5G-MEDIA monitoring services

The generic procedure followed for the message exchange is the following: the SO sends a request to the OSM monitoring module through Apache Kafka message bus. The plugin consumer reads the messages and talks to the appropriate VIM monitoring tools to configure them according to the encapsulated command in JSON format. In this release, the alarms and metrics have dedicated topics in the message bus, as shown in Table 4. Each message is sent on the message bus in JSON format, along with a unique request key and its topic.

Table 4 - List of requests and responses through the message bus of the monitoring module

Request & Response Description

create_alarm Message through Kafka topic to create alarm

create_metric Message through Kafka topic to create metric

list_alarm Message through Kafka topic to list alarm

list_metric Message through Kafka topic to list metric

delete_alarm Message through Kafka topic to delete alarm

delete_metric Message through Kafka topic to delete metric


25

Request & Response Description

update_alarm Message through Kafka topic to update alarm

update_metric Message through Kafka topic to update metric

acknowledge_alarm Message through Kafka topic to acknowledge alarm

acknowledge_metric Message through Kafka topic to acknowledge metric

3.1.4 5G-MEDIA extensions and adaptations

Extensions with respect to monitoring

Due to the fact that release three is the first OSM release where monitoring is supported (in experimental state), there are several adaptations and possible extensions that have to be considered for development and implementation in order to fulfill the Use Case requirements of 5G-MEDIA. First of all, the number as well as the “nature” of metrics supported at this version is neither large nor covering the demands that could be set by the majority of the use cases in the future, including those that 5G-MEDIA project is focusing on. More specifically, Table 5 depicts the supported metrics per VIM, along with a short description and the respective VIM.

Table 5 - The list of normalized metrics supported by the OSM Monitoring module per VIM9

Normalized Metric Name

Unit VMware vROPs Metric

Amazon CloudWatch Metric

OpenStack Metric

AVERAGE_MEMORY_UTILIZATION

% mem|usage_average

Not Supported Memory.total (%)

READ_LATENCY_<DISK_NO>

msec

virtualDisk|totalReadLatency_average

Not Supported Not Supported

WRITE_LATENCY_<DISK_NO>

msec

virtualDisk|totalWriteLatency_average

Not Supported Not Supported

DISK_READ_OPS

Nos

Not Supported

DiskReadOps

Disk_ops.DISK

DISK_WRITE_OPS Nos Not Supported DiskReadOps Disk_ops.DISK

9 https://osm.etsi.org/images/OSM-Whitepaper-TechContent-ReleaseTHREE-FINAL.PDF


26

Normalized Metric Name

Unit VMware vROPs Metric


OpenStack Metric

DISK_READ_BYTES

Bytes or bytes/sec

Not Supported

DiskReadBytes

Disk_octets.DISK

DISK_WRITE_BYTES

Bytes or bytes/sec

Not Supported

DiskReadBytes

Disk_octets.DISK

PACKETS_DROPPED_<NIC_NO>

Nos

net|dropped

Not Supported

if_dropped.INTERFACE

PACKETS_RECEIVED

Nos

net:Aggregate of all instances|packetsRxPerSec

NetworkPacketsIn if_packets.INTERFACE

PACKETS_SENT

Nos

net:Aggregate of all instances|packetsTxPerSec

NetworkPacketsOut

If_packets.INTERFACE

CPU_UTILIZATION

% cpu|usage_aver

age

CPUUtilization

Percent.virt_cpu_total

It is obvious that the list of Table 5 lacks support for networking monitoring metrics, such as bandwidth, latency, packet loss, capacity, etc. This is critical for 5G-MEDIA, since the scenarios are mostly based on bandwidth and latency requirements, while QoS is supposed to be supported through a Graphical User Interface. In this respect, 5G-MEDIA will extend this list by providing the means to include measurements from sources other than those defined in the external monitoring tools supported by OSM currently. This requirement covers both passive and active monitoring measurements within the 5G-MEDIA NFVI under question. Active monitoring deploys probes and transmits packets into the network between two endpoints. Commonly used tools include ping, which measures delay and packet loss, as well as traceroute, which helps to determine topology of the network. The main problem with active monitoring is that by the deployment of monitoring probes and the injection of packets, interference to normal traffic can be observed. On the other hand, active network monitoring allows full control over additional packets that are required to be sent over the network, as they can be sent whenever required by any specific monitoring application hence are more flexible. Unlike active monitoring, passive monitoring does not inject packets in the network


27

but rather collects information about one point in the network that is being measured rather than between two endpoints, as it is the case on active monitoring concept. Passive monitoring can be achieved by the utilization of any sniffing application and the offline processing of collected data. As a conclusion, passive monitoring is helpful when network administrator needs to know details such as network topology, running services, ports used frequently, etc. This is, in most of the cases, achieved by scanning the TCP and IP headers using various packet sniffers (e.g. tcpdump). Utilizing the advantages of both monitoring methodologies, 5G-MEDIA will extend the list of supported metrics and fully cover the requirements posed by the use case scenarios, implementing well-known, open source active monitoring tools such as iperf to cover specific needs where required.

Moreover, as shown in Table 6, the list of alarms does not cover the demands of 5G-MEDIA and in general the developers’ and service providers’ needs with respect the performance monitoring of the network service and the application on top of it as deployed in the 5G-MEDIA infrastructure. The extensions foreseen in 5G-MEDIA complement the monitored metrics that might be reflected by the definition of an alarm to trigger specific actions and inform the respective users (infrastructure owner, service developer, service provider).

Table 6 – List of normalized alarms supported by OSM release THREE.

Normalized Alarm Name

VMware vROPs Metric


OpenStack Metric

Average_Memory_Usage_Above_Threshold


Not Supported


Read_Latency_Above_Threshold

Read_Latency_Above_Threshold

Not Supported

Not Supported

Write_Latency_Above_Threshold

Write_Latency_Above_Threshold

Not Supported

Not Supported

Disk_read_ops_above_threshold

Not Supported

Disk_read_ops_above_threshold

Disk_ops.DISK

Disk_write_ops_above_threshold

Not Supported

Disk_write_ops_above_threshold

Disk_ops.DISK

Disk_read_bytes_above_threshold

Not Supported

Disk_read_bytes_above_threshold

Disk_octets.DISK


28

Normalized Alarm Name

VMware vROPs Metric


OpenStack Metric

Disk_write_bytes_above_threshold

Not Supported

Disk_write_bytes_above_threshold

Disk_octets.DISK

Net_Packets_Dropped

Net_Packets_Dropped

Not Supported

Net_Packets_Dropped

Packets_in_Above_Threshold



if_packets.INTERFACE

Packets_out_Above_Threshold



if_packets.INTERFACE

CPU_Utilization_Above_Threshold




Close the DevOps loop through utilization of monitoring data

Another extension related to monitoring is based on the shortcoming of OSM to implement the northbound interface of the Monitoring module10. This has a direct impact on “closing the cycle” between Development and Operations (DevOps) in the respect that data (and respective alarms) coming to the Monitoring module from external monitoring tools are not consumed by any actors/modules. Thus, OSM functionalities such as Scaling Groups and VCA charms remain underutilized. The Scaling Groups construct within the NSD can be used to identify a group of VNFs (sharing common characteristics) that need to be scaled under certain performance (or fault) conditions. In this release, the scaling actions supported include scaling in and out; in the case of scaling out, the added VNFs instances are attached to the same network as the initial VNF instances while the scaling action is triggered manually using the OSM GUI or the Northbound API. The certain conditions that would trigger this action must be imposed by the Monitoring module and thus a better support for such cases is dictated within 5G-MEDIA. Additionally, the VNF Configuration and Abstraction (VCA) layer supports configuration updates, actions and notifications to/from already running instances of VNFs. Backed-up by interfaces to trigger these actions, VNFs can be updated during runtime, providing several practical advantages during use case testing and demonstration execution.

10 https://osm.etsi.org/images/OSM-Whitepaper-TechContent-ReleaseTHREE-FINAL.PDF


29

Providing streaming monitoring data over websockets technology

During the study of the methodologies and technologies that have been used by 5G-PPP Phase 1 projects, it has been discovered that SONATA monitoring framework has implemented a streaming “channel” over websockets technology to provide a (near) real-time stream of monitoring data to interested user whose services are running on the 5G-MEDIA distributed infrastructure. The concept is based on the instantiation of a Tornado server that creates websockets that are passed to previously authenticated users of the SONATA Service Platform. Apart from the obvious advantage compared to the provisioning of an API call, there are two interesting things that 5G-MEDIA would like to further investigate and probably build upon: the need for streaming data might be proved really useful to developers, having the need to check the service parameters in real-time. The second remark to be taken into consideration is the introduction of the CNO in the 5G-MEDIA architecture. In the case that real-time data need to be consumed, websockets could be an alternative to the utilization of a message bus that has been implemented in CogNet and that 5G-MEDIA will leverage.

Providing VNF Forwarding Graphs capabilities for dynamic modifications

One of the most important characteristics of the MANO frameworks that have been considered for adoption in 5G-MEDIA was related to their ability to support the definition and enforcement of forwarding graphs between VNFs consisting network services. This feature has been further highlighted in 5G-MEDIA, since one of the main objectives of Cognitive Network Optimizer is the ability to provide recommendations to the operator in order to optimize the use of resources through VNF placement. By investigating offerings from these MANO frameworks, it became evident that none of these solutions could be used “as-is”, and it was clear that adaptations or even extensions to current approaches would be required. In OSM, a Virtual Network Function Forwarding Graph (VNFFG) is considered a deployment template that defines the network service topology and end-to-end traffic behavior. It is defined in NS’s descriptor file and orchestrated by the NFVO. The VNFFG may cover virtual links, VNFs, and NFs. A VNFFG is a graph, specified by a network service provider, of bi-directional logical links that connect network function nodes, where at least one node is a VNF through which network traffic is directed. The VNFFG model is defined at the network service level, where one or more VNFFG descriptors can be defined in the respective descriptor (NSD).

Handling changes on VNF forwarding graph level in a dynamic way is of paramount importance for the project. However, dealing with dynamic re-configuration of VNFs is not a trivial task, while in some cases might even be impossible to be realized without loss of the service at least for a period of time.

Extensions with respect to unikernel

Unikernels are specialised virtual machine images constructed by compiling high-level languages that run directly on a hypervisor 11. They provide only the strictly necessary code and, as a consequence, they have an improved security, smaller footprint and shorter boot time compared to traditional operating systems on virtual machines. Unikernels in 5G-MEDIA project will be used to support the development of application layer services (e.g. streaming

11 http://unikernel.org/


30

service) and support the adoption of microservices architecture12. 5G-MEDIA project will integrate the packaging tools (such as "capstan13”) provided by Mikelangelo research project14 to generate specialised machine images hosting Java, Javascript or Python microservices running on OSv unikernel images15. Such machine images will be fully integrated in the CI/CD pipeline and be referred by the VNF descriptor in a seamless manner and they will support OpenStack16 using KVM hypervisor 17.

3.2. Specifications for the Media Service MAPE

The 5G-MEDIA Media Service Monitoring-Analysis-Planning and Execution (MAPE) component implements an automation mechanism aiming at assisting MANO to dynamically manage and provide infrastructure resources according to the changes in user demand patterns, and the availability and performance of network and computational resources. The MAPE execution loop consists of the following four concrete sequential steps of activities:

1) Monitor: Collect data from integrated infrastructure and network elements used to monitor network conditions, resources performance and services status and enable the analysis step.

2) Analyse: Process collected data by using machine learning to improve specific performance KPIs of individual and aggregated network services and the network environment.

3) Plan: Recommend decisions and output concrete control actions aiming to mitigate certain risks with respect to the network infrastructure and the provided services.

4) Execute: Execute commands upon available resources and infrastructure by invoking the MANO stack and its modules according to the recommended actions.

The software architectural design and data flow of the 5G-MEDIA MAPE are shown in Figure 4.

12 https://smartbear.com/learn/api-design/what-are-microservices/

13 https://github.com/mikelangelo-project/capstan-packages

14 https://www.mikelangelo-project.eu/

15 https://github.com/mikelangelo-project/osv

16 https://www.openstack.org/

17 https://www.linux-kvm.org/page/Main_Page


31

Figure 4: Data flow and specifications of Media Service MAPE

As shown in Figure 4, the 5G-MEDIA Media Service MAPE is realized on top of two main sub-components, namely the QoS/QoE Monitoring Service and the Cognitive Network Optimizer (CNO), which implement all the services introduced by the MAPE chain. The specifications of these two sub-components are provided below:

3.2.1 QoS/QoE Monitoring Service

The QoS/QoE monitoring service collects data from running NSs/VNFs and the SDN controller. A detailed list of the available monitoring data is provided in section §3.1.3. Among other, the collected data include %CPU, %RAM, BW in-out, latency, packets drop/received/send etc.

The QoS/QoE monitoring services are responsible to i) retrieve data from OpenStack, VMWare and OpenNebula NFVIs, ii) implement a data modelling/indexing schema that enables flexible management and analysis of the collected data, iii) permanently store data in a metrics DataBase which is accessible by the ML and optimization algorithms of the CNO, and iv) share data to the CNO in real-time through a high performance, distributed, and scalable message queue based on Apache Kafka (to enable near real-time analysis capabilities).

3.2.2 Cognitive Network Optimizer

Cognitive Network Optimizer (CNO) is an analysis and prediction engine that uses the raw measurement data from the QoS/QoE monitoring service and employs statistical analysis based on machine learning and optimization techniques for classifying network and


32

computational resource service status, predicting/forecasting future resource conditions, and triggering corrective actions for the running NSs.

The CNO consists of two main sub-components; the first is the Machine Learning Engine and the second the Policy/Optimization Engine. The Machine Learning Engine builds upon the existing specifications and open source software developed by the CogNet 5G-PPP phase 1 project:18 the CogNet Lightweight Smart Engine. 5G-MEDIA’s Machine Learning Engine groups together a set of Machine Learning (ML) modules quantifying network status and providing services, such as (big) data dimensionality reduction, feature selection/extraction, traffic classification, anomaly detection, performance degradation detection and demand prediction. It is used to analyse the behavior of the instantiated network services (e.g. classify traffic, predict resource demand and utilisation, evaluate SLA violation risks, etc.) to deliver insights to the Policy/Optimization Engine on both current performance metrics and forecasted changes in demand and expected performance levels. The inputs from the Machine Learning Engine provide the necessary inputs to the multi-objective optimization algorithms that will determine the assignment of the available network and computational resources to maximizing performance metrics within defined cost and efficiency constraints.

Table 7 presents an initial set of ML modules that have been identified to assist in the optimization of 5G-MEDIA scenarios and use cases. The Machine Learning Engine applies both to: stream processing algorithms, i.e. applied on top of data streams provided by QoS/QoE monitoring services in near real-time scale; and batch processing algorithms, i.e. applied over new and historical data (as they are retrieved by accessing the metrics DB) to identify possible time/space and events-related correlations in an offline way. The output of the Machine Learning Engine is passed after the necessary adaptation phase as an active indication to the Policy/Optimization Engine. The latter aims at applying the most appropriate algorithms to optimize usage of infrastructure resources and performance of instantiated NSs, under the constraints set by the service providers and the network administrator (e.g. wrt latency, maximum resource utilization, budget cost, etc.), both in individual and aggregated levels. For instance, Integer Linear Programming can be used over small input datasets related to a specific NSs (or a suitable heuristic algorithm can be employed for larger input datasets) to determine when is the optimal time to scale in/out its VNF-FG by removing/adding further resources. By interfacing to the northbound API of the SO, the output of the Policy/Optimization Engine is conveyed to the MANO stack which will enforce the recommended policies over the resources of the integrated NFVIs.

3.2.3 5G-MEDIA adaptations and extensions

The extensions and adaptations foreseen for 5G-MEDIA are partially related to the work that has been conducted in CogNet with respect to the testing infrastructure, the ML tools as well as the algorithms and their adaptation to fulfill the specific 5G-MEDIA use case requirements. As discussed below, the first contribution of 5G-MEDIA is related to the adaptation of the CNO with Ceilometer, Gnocchi and Aodh, being the monitoring tools utilized by OSM, replacing

18 http://www.cognet.5g-ppp.eu/


33

Monasca that it was the selected tool of CogNet for collecting monitoring data from OpenStack environment. As a second extension, 5G-MEDIA plans to capitalize on the ML algorithms that will become available from CogNet after the end of the project. 5G-MEDIA consortium has already provided a first filtering of the algorithms that could potentially be used and extended for the purposes of the use case scenarios involved in the project, as mentioned in the following Table. Moreover, also in the Table below, the consortium has classified the four main types of algorithms for the optimization of media services under consideration in the three 5G-MEDIA use cases, to find similarities and differences with CogNet.

In Table 7, we summarize the specifications of the Media Service MAPE sub-components. Note that the final architectural design and the complete technical specifications for this component will be defined in D3.3.

Table 7 - Specifications of the Media Service MAPE component

Sub-component Specifications

QoS/QoE monitoring 1) Data are collected by consuming/exposing interfaces from/to: a. Ceilometer, Gnocchi and Aodh to collect data from

OpenStack NFVIs, b. vRealise to collect data from VMware NFVIs, c. the monitoring module of OnLife NFVIs

2) The Metrics DB is based on a time series Influx DB. Depending on the needs of the considered scenarios, distributed deployments (clustering) will be explored.

3) Real-time data sharing between QoS/QoE monitoring services and CNO will be served by an Apache Kafka.


34


CNO Machine Learning Engine

ML modules, partially derived from the CogNet project19 and adapted for 5G-MEDIA use case scenarios, include:

1) Mobile User Throughput Prediction (ML4MQ): detection of SLA breaches at service level.

2) Anomaly Detection Engine (ADE) with Long Short Term Memory Neural Networks: detection of resource performance degradation.

3) Network Traffic classification (NTC): multi-classification model of NSs based on labelled network-traffic KPI data.

4) Distributed Application Performance Optimizer (DAPO): classifier of network load and predictor of critical KPIs

ML modules are encapsulated in Docker containers and process metrics from InfluxDB and the Apache Kafka message bus using Apache Spark to produce outputs. They are implemented in Python and Hive on Spark. Specifically, the SW involved is:

1) Machine Learning: Apache Spark MLlib (scalable machine learning library with common learning algorithms for classification, regression and clustering).

2) Distributed Data System (in cases required): Apache Hadoop. Framework for distributed processing of large data sets.

3) Machine Learning for data streams: Apache Hive (data warehouse infrastructure on top of Hadoop for data summarization, query and analysis).

19 http://www.cognet.5g-ppp.eu/wp-content/uploads/2018/01/D34_version10_final.pdf


35


Policy/Optimization Engine

The Policy/Optimization Engine takes input from the Machine Learning Engine and from operator-defined policies (e.g. maximum cost budget and lowest acceptable performance level to meet SLA objectives) about the status of the network resources and the instantiated NSs and executes its optimisation algorithms. The output of which are implemented as policy directives to be executed by the SO. The communication with the northbound API of the OSM SO is through JSON data models and the RESTful architectural style.

Algorithms for the optimization of media services under consideration in the three 5G-MEDIA use cases are of four main types:

Service placement optimisation to determine which VNFI instance/edge node should house each VNF for a NS by trading-off cost with performance of the network and computational infrastructure. This can run at various timescales, including initial NS deployment and ongoing reconfiguration to migrate existing VNFs, instantiate new VNFs, undertake service scaling as demand patterns change.

VNFFG optimisation to determine which instances of VNFs should be interconnected to meet performance and cost objectives for specific user session requests. This can be undertaken at initial session establishment as well as for the optimisation of already running sessions/VNFFG instances.

Infrastructure adaptation to overcome streaming difficulties, e.g. to reserve network capacity, allocate greater computational capacity for stream processing, establish expedited paths or reroute flows to avoid congested parts of the network.

Application-specific adaptation and intelligent network-wide congestion avoidance, for example to configure the capturing or transcoding of 3D models to defined quality levels to match dynamically varying network throughput capabilities and available processing capacity along the VNFI nodes and clusters implementing the VNFFG instance.


36

4. Specifications for 5G-MEDIA edge-to-cloud hierarchy and instantiated NFVIs/VIMs

The 5G-MEDIA NFVI, shown in Figure 5, is composed of three different cloud sites geographically distributed and belonging to different organization, i.e. ENG in Italy, TID in Spain and OTE in Greece). The core cloud, hosting SVP and also an OpenStack NFVI providing compute, storage and network resources for NSs and VNFs, is provided by the ENG cloud, while other two cloud edge environments are provided by TID and OTE to support the various considered scenarios and use cases of the project. The networking interconnection and communication between the core and the two edge cloud environments will be achieved by using HTTPS connections, while other security features could be taken into account (i.e. limit the source address to those used in ENG) depending on the admin policies in OTE/TID. In the following subsections, the specifications and other technical aspects of the three environments are analyzed.

Figure 5: Physical layer and NFVIs in 5G-MEDIA project

4.1. Engineering cloud - FIWARE Platform

The core network cloud of 5G-MEDIA project is hosted in the FIWARE platform premises of ENG. From a technical standpoint, the FIWARE Platform is an OpenStack-based federated infrastructure comprised of several nodes, with some peculiar aspects:

● a Keystone20 based centralized Authentication, Authorization and Accounting service shared across all the FIWARE nodes

● a Ceilometer21 and Monasca22 based centralized monitoring service shared across all the FIWARE nodes, which constitute the pillar above which the infographics23 that show the overall status of the FIWARE platform has been built.

● daily sanity checks are performed on every node of the platform ● a customized, Horizon24 based user interface

20 Keystone, the OpenStack Identity Service: https://docs.openstack.org/keystone/latest/

21 Ceilometer’s documentation: https://docs.openstack.org/ceilometer/latest/

22 Monasca: https://wiki.openstack.org/wiki/Monasca

23 FIWARE Lab Nodes: http://infographic.lab.fiware.org/

24 Horizon: The OpenStack Dashboard Project: https://docs.openstack.org/horizon/latest/


37

● a wide Generic Enablers offering available through a software catalogue25 for rapid and easy deployment of software architecture blocks

Regarding the 5G-MEDIA project, Engineering Ingegneria Informatica s.p.a. is fielding its FIWARE Vicenza node, shown in Figure 6.

Figure 6: FIWARE Vicenza node in ENG cloud

The FIWARE Vicenza node is based on the OpenStack Kilo release, however an update to a newer OpenStack version has been scheduled across all FIWARE nodes in the upcoming months. Despite FIWARE user interface customizations, this deployment is fully compliant with the RESTful APIs offered by a standard OpenStack installation (please refer to the official OpenStack API Documentation26 for more info). The node is comprised of three redundant, highly available controller nodes, five compute nodes and two CEPH27 storage nodes. In terms of available resources, here is a quick recap:

432 vCores

384GB of RAM

16TB of NAS/SAN disk space, plus 300GB of local disk space for each node

8 10Gb interfaces per node, connected to two 10Gb switches

Moreover, the hosts network interfaces are bound in couples to achieve high networking availability. Four virtual LANs are employed to isolate different kinds of network traffic, such as internal server, internal Virtual Machine, public, and administrative data. Two firewall

25 Generic Enablers FIWARE Catalogue: https://catalogue.fiware.org/enablers

26 OpenStack API Documentation: https://developer.openstack.org/api-guide/quick-start/

27 Ceph: https://ceph.com/


38

layers and port filtering techniques are leveraged to address network security. Common port ranges used in typical network services such as HTTP, HTTPS, SSH, FTP and others may be freely bound on any instantiated Virtual Machine that has a public floating IP address. Two mirrored CEPH storage nodes offer object, block and file storage to the virtualization infrastructure. The FIWARE node infrastructure is hosted at a tier 4 Data Center, thus providing 99.995% uptime, 96 hours power outage protection and fully redundant network, power and cooling infrastructures.

In the context of the 5G-MEDIA project, the Vicenza FIWARE node is hosting several Central Cloud services that are expected to be employed for the project lifetime, such as Open Source MANO, OpenWhisk, a standalone, all-in-one OpenStack instance for development purposes and the CI/CD support services (such as Jenkins slave node, container/unikernel repositories, etc.) that will be updated to satisfy the project’s needs. Please refer to the 5G-MEDIA Central cloud resources28 wiki page for further information.

4.2. OTE integrated infrastructure

OTE will provide two separate infrastructures. The first one is based on a VMware platform and the second one on an Open stack platform, also providing two wireless networks based on WiFi and 4G LTE, which offer an end-to-end connectivity. The Open Stack platform will soon be upgraded to one where all the components, controller and nodes will installed in independent servers.

4.2.1 VMware cloud

The VMware ESXi cloud operates independently of the other operating systems, which is based on a VMware ESXiv5.5 server supporting the entire deployment. It is a compact architecture supporting the entire VMware infrastructure suite of products including:

VMware file system,

VMware scheduler and

VMware manager

Its operating system is called VMkernel and all processes run on top of it. VMkernel provides the means for running all processes on the system, including the management apps, agents and all virtual machines. In the following Figure 7, the VMware component architecture is presented.

28 Central cloud resources (ENG): https://production.eng.it/portal/group/prj_5g_media/wiki/-

/wiki/Main/central+cloud+(ENG)


39

Figure 7: OTE VMWare component architecture

The virtual machine monitor, VMM, provides the execution environment for a VM.

The console user interface, (CUI), is the low level configuration and management interface for basic configuration processes.

The Common Information Model, (CIM), module provides the interface for installation of a set of APIs

4.2.2 OpenStack cloud

4.2.2.1 Current Open Stack Infrastructure

The current Open Stack infrastructure OpenStack is an open source project facilitating us to build our own cloud computing setup. In other words, it creates an Infrastructure as a Service (IaaS) on our own infrastructure. It is a project originally started by NASA and Rackspace for delivering a cloud computing and storage platform. OpenStack lets users deploy virtual machines and other instances that handle different tasks for managing a cloud environment on the fly.

OpenStack software delivers a massively scalable cloud operating system consisting of three major components:

1. Compute: open source software designed to provision and manage large networks of

virtual machines, creating a redundant and scalable cloud computing platform.

2. Object Storage: open source software for creating redundant, scalable object storage

using clusters of standardized servers to store petabytes of accessible data (code-

named "Swift").

3. Image Service: provides discovery, registration, and delivery services for virtual disk

images (code-named "Glance").


40

4.2.2.2 Open Stack Lab Topology

OpenStack all-in-one is installed in our lab premises and based on DevStack, which is a specific project used for the basic installation. According to DevStack information, “queens” version has been used and it is installed in an HP Pro Liant DL380 Gen9 server with 2 4-cores CPUs, in total 8-cores and 32 GB RAM with Ubuntu 16.04 xenial OS version. Right now, OpenStack topology for the time being, includes all elements illustrated in Figure 8. As 5G-Media will move on, furthermore instances will be created, such as OpenWhisk, or anything else that 5G-Media requires.

Figure 8: OTE Open Stack all component in one Infrastructure

The connectivity to the internet is done through NAT services with a CISCO ASA network element. The NAT can be configured to be dynamic so that all hosts can reach the internet, or static NAT can be applied to allow also access to particular services from the internal OTE network to be reachable from outside the firewall.

Additionally, the 5G-Media partners are able to remotely to the OTE infrastructure via SSL clients through open VPN connections. This allows connectivity to the infrastructure from other developers in the 5G-Media project, (ex. SILO, ENG, IBM, etc.), or from other infrastructure (ex. TID) which is located in Spain. Different credentials is needed to provide to different partners for security but also for management reasons.

The upcoming infrastructure in OTE lab is going to be as follows.


41

Table 8 - NFVI – PoP nodes

A/A component

1 Cloud controller

2 Compute Node 1

3 Compute Node 2

4 Compute Node 3

The future OTE open stack based infrastructure it will be based on a nova controller together with two-three compute nodes based on neutron agents. The gateway will provide connection to the internet.

Figure 9: Cloud architecture that based on OpenStack

4.2.2.3 Mobile end-to-end system

A mobile system is set-up in the OTE lab that can be used for the delivery of services such as video, etc. It is constituted by two indoor Nokia small base stations of type Flexi zone and a virtual enhanced packet core, (vEPC). The mobility, authentication and roaming of the user’s mobile is performed by the vEPC system. The small base stations are carrier aggregated in two frequencies of 2600 MHz and 1800 MHz and are WiFi capable.


42

Figure 10: OTE Lab mobile infrastructure

The integrated infrastructure in the OTE’s lab is shown in the following Figure 11.

Figure 11: Integrated infrastructure in OTE’s lab

4.3. Telefonica cloud – OnLife platform

TID Cloud is based on OnLife platform, which is a new architecture for Central Offices (COs) based on edge computing that allows the virtualization of its access network and offers third-party application developers and content providers cloud-computing capabilities at the edge of the network. OnLife proposes a design based on NFV, SDN and Cloud Computing. In particular, there were three main principles that guided the design, namely:


43

Figure 12: Components of CORD

1. Greenfield: No support/integration with legacy systems nor infrastructure,

2. Extreme use of simplification and NFV principles: Making NFVs as simple as possible and only implementing the protocols and pieces of “Central Office Re-architected as a Datacenter” (CORD) that were necessary for the actual services to be provided, but leaving room to increase functionality in the future if needed, and,

3. Open Source Software and Open Compute Project29 (OCP) hardware.

Following the simplification principle, CORD, shown in Figure 12, is implemented using OpenNebula30, which is a lightweight and flexible solution to enable fast prototyping, service orchestration and management of the “Central Telefónica Procesadora de Datos” (CTpd) infrastructure. In this design, several CORD modules, such as Authentication, Authorization, and Accounting (AAA) or Virtual Tenant Networks (VTN) are not implemented, as its functionality is redundant to other elements to be deployed in CTpd or not required to provide the selected services. The cloud provides the flexibility needed to deploy scalable services as well as the management layer for the VMs implementing multiple network functions. In particular, OpenNebula represents a clear advantage in this area compared with other approaches, namely:

Smaller footprint, in terms of hardware requirements, software components and operational costs.

Simple and extensible architecture, with well-defined interfaces that allow an easy integration of new and innovative scenarios.

Rich and consistent feature set, to support novel networking services.

29 http://www.opencompute.org/

30 https://opennebula.org/


44

Moreover, aiming to a future-proof solution and aligned with the Greenfield principle, CTpd supports only IPv6 and considers IPv4 as a service that interconnects to IPv4 Internet by means of NAT64 at the edge of the data center or tunneling the IPv4 customer traffic to that point. Finally, the characteristics of OpenNebula also allow the CTpd to design a network service development framework. This framework will open the CO to third-party applications to explore new business use cases.

ONOS is the selected SDN platform for the deployment of CTpd, as most of CORD functionality relies on applications developed in this platform. More specifically, CTpd focuses on the R-CORD use case, which involves the use of the vOLT and the vRouter applications, already developed for ONOS. Further deployment into the Enterprise and Mobile segments will follow the same approach as the access network becomes one and the same for all segments. Other reasons of choosing ONOS are its simple and dynamic deployment via the Apache Karaf environment, its high-quality documentation and its community support.

4.4. Integration of GPUs in 5G-MEDIA NFVIs

Computing power has traditionally been associated with CPU and RAM availability and so has been, consequently, the demand of resources to cloud providers. In the last years, another kind of processors have gained attention, i.e. Graphical Processing Units (GPUs). Once the last confined to traditional PCs market, now spanning over the area of parallel computing on servers and, therefore, on the cloud.

The reason for the interest on GPUs is the ability to perform high parallel computing based on thousands of cores, while on the other side multicore CPUs, mainly designed to execute sequential calculation, have only a few dozens of cores and are much less efficient for such kind of computation31.

4.4.1. Integration of GPUs into the cloud environment

Recently, the top cloud providers have started offering GPU based VMs that take advantage of such parallel models and are suitable for the GPU intensive computation (i.e. video rendering, gaming, etc.) and even deep learning computation models. Among the top GPUs suppliers32, NVidia is intensively involved in this transition and has developed a programming model for its GPU called “CUDA”33 to enable developers to build parallel applications, together with powerful datacenter GPUs boards like the Tesla series34 or the Quadro series boards for desktop and mobile devices35 . According to that, the common ways to access GPUs from VM36 are:

31 https://www.forbes.com/sites/janakirammsv/2017/08/07/in-the-era-of-artificial-intelligence-gpus-are-the-new-cpus/

32 https://wccftech.com/nvidia-amd-discrete-gpu-market-share-report-q3-2017/

33 https://developer.nvidia.com/cuda-zone

34 http://www.nvidia.com/object/tesla-servers.html

35 https://developer.nvidia.com/cuda-gpus

36 GPU in virtual environment: https://www.isi.edu/sites/default/files/users/jwalters/papers/HPGC_2014.pdf

https://www.forbes.com/sites/janakirammsv/2017/08/07/in-the-era-of-artificial-intelligence-gpus-are-the-new-cpus/1

https://www.forbes.com/sites/janakirammsv/2017/08/07/in-the-era-of-artificial-intelligence-gpus-are-the-new-cpus/1

https://wccftech.com/nvidia-amd-discrete-gpu-market-share-report-q3-2017/

https://developer.nvidia.com/cuda-zone

http://www.nvidia.com/object/tesla-servers.html


45

through Remote API invocation, that allows applications executed within VMs to leverage hardware acceleration with API interception and redirection and achieve high computing performance in a transparent way;

through Peripheral Component Interconnect (PCI) passthrough 37, that allows a VM to see a host device (such as a GPU board) as if it was physically attached to the guest operating system (see Figure 13) and with near-native performance although with an exclusive usage of such device from the guest operating system;

through emulated devices (for example with devices supporting SR-IOV38), which introduces a considerable amount of overhead39 especially with GPU devices.

Figure 13: PCI passthrough in virtual environment

Until now, the Remote API invocation approach has been implemented in a number of different ways, including the lately introduced support to CUDA devices (for example vCUDA40 or rCUDA41). However, recent studies show that there are still strong performance issues42, while on the contrary, the PCI passthrough approach (shown in Figure 13) only introduces a small overhead compared to native device usage which represents the best possible use case. Thus, PCI passthrough is a promising candidate solution to be implemented in the scope of 5G-MEDIA project.

37 PCI passthrough: https://www.ibm.com/developerworks/library/l-pci-passthrough/index.html

38 SR-IOV: https://en.wikipedia.org/wiki/Single-root_input/output_virtualization

39 overhead in emulated devices: https://www.ibm.com/developerworks/library/l-pci-passthrough/index.html

40 vCUDA: http://ieeexplore.ieee.org/abstract/document/5928326/?reload=true and http://dsc.soic.indiana.edu/presentations/Cloud_2014_GPU.pdf

41 rCUDA: http://www.rcuda.net/

42 vCUDA and rCUDA performance compared to native PCI express: https://www.isi.edu/sites/default/files/users/jwalters/papers/HPGC_2014.pdf


46

4.4.2. GPU provision over containerization technology

In the last years, cloud providers have also started using containerization as a virtualization technology along with the VMs; for this reason, the provisioning of GPUs in the cloud has also evolved embracing such technology (e.g. Docker43) as a more efficient way to manage resources, according to the model known as “Containers-as-a-Service” (CaaS). NVidia has recently released to developers an extension of Docker to support its own GPUs as resources inside containers, with the high-level architecture depicted in the Figure 14. With this Docker extension, GPU resources are made available to containers through a “CUDA Toolkit” library, relying on the CUDA driver installed on the host that mounts the NVidia GPU(s). The most important features are basic GPU isolation support44 and the ability to support different version of kernel modules and keep Docker container images portability45, overcoming the dependency on the specific kernel module version due to the necessity of loading user-level NVIDIA GPU driver libraries46.

In Figure 15, the internal architecture of NVidia Docker extension is depicted, as well the actual flow when specific Docker images supporting CUDA are retrieved by the registry47. Each container relies on an NVidia Docker plugin48 that discovers the host driver files and makes the GPU resources available through the NVidia management library49 and even provide GPU monitoring information through REST API interface. The creation of containers is done through the NVidia Docker CLI wrapper, that mounts a host volume containing the hard links to the specific NVidia driver files that are looked up and loaded by the dynamic linker 50 (with ldconfig using the ldcache51 cache) in the predefined Docker images supporting CUDA from the Docker Registry. Such CLI wrapper is also able to manage containers on a remote host using network protocols (SSH/HTTP).

43 Docker: https://www.docker.com/

44 Nvidia-docker GPU isolation to assign resources to a single container: https://github.com/NVIDIA/nvidia-

docker/wiki/GPU-isolation-%28version-1.0%29

45 Nvidia-docker image portability: https://github.com/NVIDIA/nvidia-docker/wiki/NVIDIA-driver-(version-1.0)

46 VM and GPU passthrough setup - comments: https://devblogs.nvidia.com/parallelforall/nvidia-docker-gpu-

server-application-deployment-made-easy/ 47 nvidia-docker internal architecture: https://github.com/NVIDIA/nvidia-docker/wiki/NVIDIA-driver-(version-1.0)

48 nvidia-docker-plugin: https://github.com/NVIDIA/nvidia-docker/wiki/nvidia-docker-plugin

49 NVML: https://developer.nvidia.com/nvidia-management-library-nvml

50 dynamic linker: https://en.wikipedia.org/wiki/Dynamic_linker

51 ldconfig and local cache: http://man7.org/linux/man-pages/man8/ldconfig.8.html


47

Figure 14: NVidia-docker high-level architecture

While the solution provided by NVidia focuses on physical hosts using Docker as virtualization technology, it can also run on a VM-based virtual environment itself, although with some limitations52. For instance, in an approach where PCI passthrough is implemented over OpenStack53, having a VM with NVidia Docker running on top is a simple example 54, but it may potentially become inefficient from the point of view of a cloud provider because of the exclusive usage of the GPU from a single VM. For this reason, the most important container orchestrators have been starting supporting GPUs, and a presentation at the OpenStack Summit in Boston of mid 201755 compared the features available on Kubernetes56, OpenStack57, Apache Mesos58, and Docker Swarm59, highlighting the actual support to GPU and their limitations, as shown in the Figure 16.

52 Docker with GPU on Hyper-V limitations: https://github.com/NVIDIA/nvidia-docker/issues/197

53 PCI passthrough on OpenStack: https://docs.openstack.org/nova/pike/admin/pci-passthrough.html

54 OpenStack GPU passthrough example: https://gist.github.com/claudiok/890ab6dfe76fa45b30081e58038a9215

55 OpenStack and GPU: https://www.OpenStack.org/assets/presentation-media/OpenStack-Boston-Summit-Presentation.pdf

56 Kubernetes: https://kubernetes.io/

57 OpenStack: https://www.openstack.org/

58 Apache Mesos: http://mesos.apache.org/

59 Docker Swarm: https://docs.docker.com/engine/swarm/

https://www.openstack.org/assets/presentation-media/OpenStack-Boston-Summit-Presentation.pdf

https://www.openstack.org/assets/presentation-media/OpenStack-Boston-Summit-Presentation.pdf


48

Figure 15: NVidia-docker internal architecture

Under this scope, one important feature, particularly in a multi-tenant cloud environment, is the ability to provide GPU isolation, to restrict the usage of a specific GPU to a particular container60. At the time of such comparison (mid 2017), this feature was not completely supported by “nvidia-docker” (different container could get the same GPU61) but only by Kubernetes (version 1.6.x) and Apache Mesos, with the latter using a different containerizer than Docker but using the same approach nvidia-docker adopts to mount the NVidia driver62.

The configuration of such cloud orchestrators and resources together with GPUs on top of different cloud providers has evolved and simplified over time. Tools like Juju from Ubuntu63 allows to model, deploy and manage complex environments through set of scripts (“charms”) publicly available to developers. An example is reported on the official Ubuntu website64 that shows how easy is, with a charm and a few commands, to allocate Kubernetes configured to provide (NVidia) GPU based containers65 on top of Amazon Web Services66; similar configuration is going to be available for different cloud providers as well67. On the other hand, support to Kubernetes and GPU in OpenStack is still in progress, as shown in the OpenStack

60 Nvidia-docker GPU isolation: https://github.com/NVIDIA/nvidia-docker/wiki/GPU-isolation-(version-1.0)

61 nvidia-docker GPU isolation details: slide 24 https://www.OpenStack.org/assets/presentation-media/OpenStack-Boston-Summit-Presentation.pdf

62 GPU support on Apache Mesos: http://mesos.apache.org/documentation/latest/gpu-support/

63 Ubuntu juju: https://jujucharms.com/

64 https://insights.ubuntu.com/2017/04/19/how-we-commoditized-gpus-for-kubernetes/

65 NVidia GPU container support on Kubernetes: https://kubernetes.io/docs/tasks/manage-gpus/scheduling-gpus/

66 Amazon AWS with GPU: https://aws.amazon.com/ec2/elastic-gpus/

67 Kubernetes with GPU on Azure: https://medium.com/@wbuchwalter/creating-a-kubernetes-cluster-with-gpu-support-on-azure-for-ml-training-and-predictions-with-a551a19b8859

https://insights.ubuntu.com/2017/04/19/how-we-commoditized-gpus-for-kubernetes/


49

Summit in Sydney of November 201768, where has been shown that the support to multi-tenancy requires some specific configuration and effort. The option to deploy Kubernetes 1.6.x. for each tenant with Magnum69 or Heat70 has still some issues71 and possible alternatives are under evaluation72.

Figure 16: Container orchestrators support to GPU (Source: OpenStack Summit 2017

In the context of 5G-MEDIA project, GPUs are used to support the specific tasks described in its use cases73, such as 3D gaming and video rendering. On the other hand, OpenStack is the main NFVI provider as it is supported by different SVPs such as SONATA74 and OSM75, while Kubernetes is going to be used by OpenWhisk76 to manage clusters of containers. The solution that will be evaluated for 5G-MEDIA project is based on Kubernetes (with a version supporting NVidia GPUs77) running on top of OpenStack and may rely on Magnum78 to supports also other container orchestration engines such as Docker Swarm and Apache Mesos. This may be

68 https://www.OpenStack.org/videos/sydney-2017/better-gpu-container-as-a-service-realize-multi-tenancy-with-OpenStack

69 OpenStack Magnum to manage Kubernetes: https://wiki.OpenStack.org/wiki/Magnum

70 OpenStack Heat: https://wiki.OpenStack.org/wiki/Heat

71 limitations of Kubernetes with GPU on OpenStack in multitenant environment: see 4:05 and 4:09 of https://www.OpenStack.org/videos/sydney-2017/better-gpu-container-as-a-service-realize-multi-tenancy-with-OpenStack

72 using Keystone with Kubernetes for GPU multitenancy: see 9:40 of https://www.OpenStack.org/videos/sydney-2017/better-gpu-container-as-a-service-realize-multi-tenancy-with-OpenStack

73 5G-MEDIA usecases: http://www.5gmedia.eu/use-cases/

74 SONATA: http://www.sonata-nfv.eu/

75 OSM: https://osm.etsi.org/

76 OpenWhisk: https://openwhisk.apache.org/

77 Kubernetes NVidia GPU support: https://kubernetes.io/docs/tasks/manage-gpus/scheduling-gpus/

78 Magnum: https://wiki.openstack.org/wiki/Magnum

https://www.openstack.org/videos/sydney-2017/better-gpu-container-as-a-service-realize-multi-tenancy-with-openstack

https://www.openstack.org/videos/sydney-2017/better-gpu-container-as-a-service-realize-multi-tenancy-with-openstack


50

subject to change over the project time life according to the availability of valuable updates about GPU support from the opensource community.

4.5. Multi-PoP (inter-NFVI) network connectivity

The 5G-MEDIA NFVI is composed of different cloud sites geographically distributed and belonging to different organization (i.e. ENG in Italy, TID in Spain and OTE in Greece). Each 5G-MEDIA cloud platform corresponds to a PoP in the NFVI and offers different kind of virtualized resources depending on the Network Service deployment needs. In particular, the 5G-MEDIA NFVI is hierarchically structured with a central cloud platform and edge cloud platforms. The motivation behind this kind of hierarchy is the envisioning of distributed NSs, where VNFs in a VNFFG can be instantiated across the different PoPs depending on their requirements and functionalities. If we take into consideration a typical media service like VoD, reducing latency is a key factor for achieving a high QoE from the end-users. The service provider might want to deploy a cache with the requested content in proximity to the end-user, maintaining the core service functionalities in the central cloud and deploying the new service component in the edge cloud nearest to the end-user. This means that, from the NS management and orchestration point of view, the 5G-MEDIA SVP should have a comprehensive unified view of the different NFVI-PoPs virtualized resources and should be capable of establishing the proper network connectivity between these PoPs in order to guarantee the network communication between VNFs belonging to the same NS.

Depending on the different types of VIM instances deployed in each cloud platform, the implementation of this inter-NFVI PoPs network connectivity functionality may vary. In particular, we distinguish between three different cases:

● Each PoP is under the control of a different type of VIM (e.g. OpenStack, OpenNebula, VMware, etc.).

● The same VIM is used in each PoP (in a following section we are considering the case of OpenStack in a multi-region deployment).

● Hybrid case, where multiple instances of the same VIM coexist with other types of VIM (e.g. multi-region OpenStack plus OpenNebula).

4.5.1 Multi-type VIMs

In the case of an NFVI composed of PoPs managed by different types of VIM (e.g. OpenStack, OpenNebula, VMware, etc.) the inter-NFVI network connectivity should be handled at the NFVO level, in particular integrating the use of a WAN Infrastructure Manager (WIM) component within the NS provisioning workflow, as shown in Figure 17. A WIM is a specialized VIM used for establishing connectivity between endpoints in different NFVI-PoPs.


51

Figure 17: Interconnecting multiple NFVI-PoPs

Looking at the standards, in order to interconnect the different NFVI-PoPs, the WAN between them should provide Virtual Links, based on the network connectivity services, as requested by the NFVO through the Or-Vi reference point. This implies the introduction of a further level of resources orchestration at the NFVO, where the orchestrator should have an abstracted view of the network topology and interfaces to underlying VIMs and WIMs. In particular, a WIM should implement the following functionalities related with the NFVI connectivity services:

● Path computation according to QoS input parameters ● Connectivity establishing over the physical network ● North Bound APIs to be used by the NFVO ● South Bound APIs / Drivers to SDN Controllers in order to configure the underlying

network The SDN controller invoked by the WIM is responsible for the resource management and for collecting information and statistics about the underlying network (e.g. bandwidth etc.), for such reasons it should implement proper drivers at its South-Bound Interface (SBI) in order to interact properly with the network data-plane.

Different software alternatives already exist for the SDN Controller (e.g. OpenDaylight, Onos, Floodlight etc.), while the WIM component with its functionalities should be designed from scratch. The integration within an ETSI compliant MANO framework can be handled through the implementation of the Or-Vi reference point as the North-Bound Interface (NBI) of the WIM for the interaction with the NFVO.


52

4.5.2 Multi-region OpenStack

OpenStack is a flexible open source platform providing a well-established framework for managing virtualized resources. However, OpenStack’s networking model, enabled through the Neutron service, was not designed to exchange networking information with external networking entities, in particular other OpenStack instances. In the contrary, the Neutron network model is designed to operate within a single domain and anything outside of that is treated as “external,” preventing the Layer-2 connectivity and requiring the configuration of floating IPs on VMs to be able to reach the “external world”. This subsection will present an architecture alternative for enabling the end-to-end orchestration of distributed NSs across different NFVI-PoPs, where each PoP in the NFVI corresponds to an OpenStack region in a multisite OpenStack deployment.

An OpenStack region can be identified as a regular data-centre, where one OpenStack instance is completely deployed except from the KeyStone component. The KeyStone DB will be centralised or shared between the different regions, enabling authorization and authentication over multiple clouds. What is missing from the NS orchestration point of view is the possibility to have a unified view of the whole NFVI resources offered from the different regions and a way to establish the L2 network connectivity between VNFs within the same NS chain but distributed across different PoPs.

From the networking point of view, a first approach could be the use of Tricircle79, a plugin for networking automation that works across Neutron instances in multi-region deployments, as shown in Figure 18. In order to allow the VNFs inter-connection across regions, Tricircle enables the L2 networking among distributed Neutron instances (e.g. IP address space management, IP allocation and L2 network segment global management). In terms of resource orchestration, Tricircle allows Neutron to work as one cluster in multi-region OpenStack clouds, actuating the orchestration of virtualized networking resources across multiple OpenStack clouds. All VNFs, in an OpenStack tenant domain and provisioned in different clouds, can be interconnected via the global virtualized networking resources.

79 https://wiki.openstack.org/wiki/Tricircle


53

Figure 18: Architecture of Tricircle

The main Tricircle components are the following:

● The Tricircle plugin should be deployed coupled with each Neutron component in the multisite OpenStack deployment. In particular, a kind of hierarchy is needed, deploying a Central Neutron instance with the Tricircle Central Neutron Plugin, which runs under the Central Neutron Server and serves for tenant L2/L3 networking automation. The Tricircle Local Neutron Plugins, deployed in the other PoPs, are responsible for cross Neutron networking automation triggering.

● The Admin API module manages the mappings between OpenStack instances and availability zone, retrieve object uuid routing and exposes API for maintenance.

● The XJob module receives and processes cross-region OpenStack functionalities and other asynchronous jobs from Admin API or Tricircle Central Neutron Plugin.

Concerning the unified view of virtualized distributed resources across different regions, the Kingbird80 project offers a centralized OpenStack service that provides resource operation and management across multiple OpenStack instances in a multi-region OpenStack deployment. In particular, Kingbird provides features like centralized quota management, centralized view for distributed virtual resources, global view for tenant level IP/MAC address space management, synchronisation of ssh keys, images, flavors, security groups, etc. across different PoPs. Kingbird is part of the OPNFV Multisite project that intends to address the use cases related to distributed cloud environments.

In OpenStack, quotas are usually defined per region and are administrated by the Openstack components, i.e. Neutron, Nova and Cinder. Kingbird, shown in Figure 19 and Figure 20, introduces a new centralised quota management function that allows defining quotas not for

80 https://wiki.openstack.org/wiki/Kingbird


54

region but across the whole NFVI and with a minimal impact on the existing Openstack services. In particular, the Kingbird quota management function:

● uses existing APIs to dynamically balance quota values ● calculates resource usage upon synchronization ● stores the default/tenant quota limits in the Kingbird DB, providing CRUD operations

for the known quota limits

Figure 19: Kingbird architecture

Figure 20: Kingbird centralized deployment over different OpenStack multi-region PoPs

4.5.3 Multi-region OpenStack in hybrid NFVI

This last case presents the deployment of different types of VIMs for different cloud platforms and the aggregation of some cloud platforms in a multi-region OpenStack deployment as detailed before. This hybrid deployment foresees the use of Tricircle and Kingbird for the OpenStack multi-region installation, as well as the use of a WIM for inter-connecting the OpenStack multi-region deployment, looking as a single VIM and a single PoP, with the other NFVI PoPs managed by different VIMs. Then, also in this case, the inter-NFVI PoPs network connectivity is handled at the NFVO level, where the orchestrator should request the creation of VLs to the WIM through the Or-Vi reference point. The WIM will be responsible for the computation of network paths between the OpenStack multi-region gateway and the other PoPs gateways as well as between the non-OpenStack PoPs. In terms of virtualized resources aggregation across the whole NFVI, this kind of deployment implies also the establishing of a kind of federation in terms of trusts between different administrative domains, since the authentication and authorization systems cannot be shared across the different PoPs in an automatic way and complex configuration might be needed.


55

5. Specifications of VIMs supported by 5G-MEDIA platform

5.1. Integration with OpenNebula

OpenNebula is an Open Source platform oriented of Cloud Computing for data centers, which allows you to create a virtual infrastructure to build private/public/hybrid Cloud, following the paradigm of Infrastucture as a Service (IaaS). It is free software with Apache2 license. Additionally, OpenNebula orchestrates storage, network, and services. It takes control of the hosts where it deploys multi-tier services such as machines or virtual networks, enabling resources sharing from the own data center within remote cloud resources.

OpenNebula aims to break through the legacy datacenter infrastructure with specific requirements (as other Cloud Computing platforms), providing flexibility and simple management of the cloud, through the automation and orchestration of simply network operations, storage, virtualization and existing monitoring (Bridges, KVM, VNC, etc.). The main advantage and difference with other similar platforms (e.g. OpenStack) is that OpenNebula is responsible only for the cloud management and has no additional responsibilities. As an example, OpenNebula does not interconnect virtual machines. Instead, it is the SDN controller or the bridge/OVS at a host level that does the work. Another example is the utilization of KVM as a virtualization technology or the use of templates instead of flavours. The difference between templates and flavours is that the last ones are description archives that contain several parameters of the virtual machines (eg: Network and execution scripts) while flavours only has CPU, storage and RAM information. The rest of configuration are performed out of the scope of OpenNebula. These are the reasons why OpenNebula is lighter, simpler and easier to install and configure.

In 5G-MEDIA, the integration between OpenNebula and OSM will be developed using a plugin allocated at the OSM’s RO component similar to other cloud platforms, as shown in Figure 21. The objective is that OpenNebula behaves as a VIM additionally to the official VIMs supported Out-of-the-Box by OSM.

Figure 21: Integration between OpenNebula and OSM


56

The plugin will make requests using the XML-RPC API of OpenNebula to fulfill the diverse operations (creation, actualization, etc.) of the necessary virtual services, such as machines or networks. As shown in Figure 22, a similar structure to the existing plugins will be followed, but the content of each method will be modified to fulfill the related request to OpenNebula. At the next step, the received data from OpenNebula will be treated and sent in the proper format for the complete execution of OSM.

Figure 22: Plugin of OpenNebula in RO of OSM

To request XML-RPC from the connector the Python-Oca81 library will be used. This library is in charge of building the HTTP-XML requests and send them to OpenNebula. Below it is shown an example of the Vimconn_opennebula and the use of the Python-Oca library:

81 https://pypi.python.org/pypi/oca


57

Figure 23: Use of python-oca library for XMLRPC OpenNebula Cloud API

5.2. FaaS VIM architecture and Specifications

5.2.1 Background

5G-MEDIA architecture introduces a novel concept of Serverless Virtual Function (SVF). Typically, SVF is an application level Virtual Function (VF) providing a reusable generic or media specific functionality. The complete list of VFs considered in 5G-MEDIA project is included in D2.2. Examples are transcoding, buffering, media clips production, cognitive services, such as speech and image recognition, etc.

In a traditional MANO stack, VFs are provisioned as VMs similarly to the VNFs without any regard to the function life time duration. Nevertheless, an innovation of 5G-MEDIA allows to slash operational costs dramatically, by allowing short lived VFs to be provisioned on demand using Function-as-a-Service (FaaS) programming model. As an example, consider a short-lived media intensive game bout between two players, where a session lasts only a few minutes. During that session, many VFs should be instantiated: two transcoders for processing the media stream, a buffering function to buffer the last few seconds of the game for on demand clip creation, rendering functions for the two players and for the on-line spectators, and the repeat clip creation function that might be instantiated any time throughout the session to create a clip from the buffered media. FaaS allows to provision all these VFs on demand (typically, in the form of containers) where function provisioning and deprovisioning is handled by a FaaS platform transparently providing for seamless elasticity and scalability. Much of the FaaS benefit comes from the billing model. For each VF/VNF there exists a break even point of time utilization after which FaaS is not justified economically. If, for example, a VF or a VNF should run all the time, FaaS does not offer better cost efficiency than using


58

traditional VMs. To that end, 5G-MEDIA offers FaaS as a virtualization option that can be combined with other forms of virtualization allowing 5G-MEDIA service developers to mix and match the most suitable technologies to attain cost-efficiency.

5.2.2 FaaS VIM Architecture

Figure 24 depicts a reference architecture of integrating FaaS VIM with an ETSI compatible MANO stack. In the 5G-MEDIA reference implementation that uses OSM R3 some deviations from the standard are unavoidable, because of the gaps that exist between the current OSM implementation and ETSI MANO. In what follows, wherever deviations from the ETSI standard influence the VNFI flows, we specifically mention this and explain 5G-MEDIA specific solution.

5G-MEDIA treats FaaS platforms as virtualization platforms that are managed by their corresponding VIMs, where main reference points Vi-Vnfm, Nf-Vi, and Or-Vi conform to the ETSI MANO specification82. Therefore, communication between NFVO and FaaS VIM VNFM and FaaS VIM is seamless.

Figure 24: Reference Architecture for Integration of FaaS Frameworks with 5G-MEDIA

FaaS is a form of PaaS that takes away the burden of managing underlying virtualized resources, allowing to instantiate VNFs on-demand as processes (e.g., as containers) seamlessly managing their lifecycle. On the south-bound interfaces, FaaS VIM communicates

82 ETSI GS NFV-MAN 001 V1.1.1 (2014-12)


59

with a specific FaaS framework (e.g., Apache OpenWhisk), a specific PaaS (e.g., Kubernetes) and a specific IaaS Cloud Orchestrator (e.g., OpenStack or OpenNebula). In particular, communication between the FaaS VIM and the FaaS framework is required in all VNF lifecycle flows (e.g., instantiation, where instantiation corresponds to invocation of an action). Communication with PaaS might be required in some VNF lifecycle flows, e.g., in one embodiment of this architecture, FaaS VIM communicates with PaaS to provision and deprovision private networks for VNF instances (e.g., as part of instantiation and termination flows respectively). Also, FaaS VIM can communicate with the PaaS framework for the purposes of detailed monitoring. Communication between FaaS VIM and the underlying IaaS Cloud Orchestrator can be required in some flows that are not mandated by the ETSI MANO and this communication is optional. Some cases, where this communication might be required might be related to provisioning of logical networks that can be utilized by the FaaS framework through the PaaS one for providing more separation between the executing actions. Also, this communication might be useful for on demand provisioning/deprovisioning of resources that can be added to PaaS and, therefore, made available to FaaS at time of load peaks/valleys.

Presently there is no standard specified by ETSI for cloud native VNF onboarding, instantiation and management. Network Service Descriptors (NSD), Virtual Network Function Descriptors (VNFD) and Virtual Deployment Units (VDU) assume hypervisor-based virtualization or bare-metal based infrastructure. In absence of ETSI standard for cloud native PaaS, such as container service and FaaS, we complement the standard specifications with optional elements as explained below. In the future we plan to reach out to ETSI and join forces in standardizing activity with respect to the cloud native PaaS83. A concrete software architecture embodying the presented reference architecture is shown in Figure 25.

83 ETSI GR NFV-IFA 029 v0.50


60

Figure 25: Software architecture for FaaS integration within 5G-MEDIA Platform

Network Service Descriptor (NSD)

Network Services Descriptors are not affected by integrating FaaS with the ETSI MANO framework. In 5G-MEDIA platform architecture we allow mix and match of VNFs within the same VNS, where some VNFs might be deployed as traditional VMs on the virtualized or bare-metal infrastructure managed by their corresponding VIMs and other VNFs might be deployed as containers through the FaaS VIM.

The on-boarding flow of OSM does not change. A VNFD of a FaaS VNF is identical to that of a regular VNF in all mandatory fields. In contrast to the ETSI MANO specified on-boarding flow, OSM R3 does not validate presence of the VNF image in VIM. In case of FaaS VNFD, the mandatory image reference field contains a fully qualified name of an Open Whisk action implementing this VNF. It is assumed, as for on-boarding of any VNF type that at an arbitrary time, but before the first VNF instantiation, the VNF image should be uploaded to a VIM through which this VNF will be instantiated. Otherwise instantiation will fail. In case of FaaS VNF, uploading of the FaaS VNF image is being done via deployment tools offered by OpenWhisk and extended in this project. In other words, uploading of the FaaS VNF image is done outside of OSM R3. Figure 26 depicts image upload to FaaS VIM (i.e., to OpenWhisk) process for the general distributed environment for a sample use case corresponding to UC1 of the 5G-MEDIA project, as it is described in D22.


61

Figure 26: FaaS VNF image upload and on-going management

Figure 26 depicts a scenario with a central cloud (at the top) and two edges (at the bottom). This figure focuses on the OpenWhisk assets management and therefore does not show any other components of the platform. OpenWhisk of a central cloud is designated as a “leader”, a single point of truth, with respect to the assets (i.e., FaaS VNFs implemented as Open Whisk Docker actions) of a VNS “Star Ball Game”. The leader OpenWhisk defines a desired state for all edges of the type “game”. The relationships between a leader OpenWhisk and a follower OpenWhisk are logical and can be reversed. However, for maintainability, 5G-MEDIA recommends using central cloud as a central location where FaaS VNFs should be deployed and from which they will be seamlessly propagated by a federation management protocol of OpenWhisk (currently under development). It should be stressed that both leader and follower OpenWhisk instances are independent fully functional OpenWhisk instances.

In Step 1 in Figure 26, the user needs to push a VNF container image to a repository. It can either be a public or a private repository. This should be done for every FaaS VNF in the VNS. In Step 2, the user utilises wskdeploy tool84 to deploy the metadata of all actions pushed to container repository in Step 1. The tool uses a manifest file that describes all the actions which are treated as a managed “project”. After Step 2 completes, the FaaS VNF action metadata is in OpenWhisk of the central cloud and the Docker container image resides in the repository

84 https://github.com/apache/incubator-openwhisk-wskdeploy


62

of the user’s choice and the FaaS VNFs can now be instantiated in the central cloud via OSM’s FaaS VIM plugin that connects to the central cloud’s OpenWhisk.

The assets in the managed project created at the central cloud OpenWhisk will get automatically propagated to the edges that follow that project in Step 4, which occurs as a timer based OpenWhisk action. In case the user wishes to force assets propagation it calls a special OpenWhisk federated management action push_state in Step 4’ specifying the follower OpenWhisk type, game. Configuration of a follower type is part of the OpenWhisk federation management configuration and is performed via a special action create_follower (not shown in the figure) that establishes a logical relationship between the leader and follower OpenWhisk instances with respect to a specific managed project. All federation management actions are provided as system ones to prevent users from accidentally creating actions with the same name in a regular user space.

After the FaaS VNF images and metadata are predeployed as explained, VNF and VNS onboarding via OSM can be executed. This process does not change because of FaaS integration. More specifically, each FaaS VIM instance is treated as yet another VIM representing its datacentre and a user is given a flexibility to associate a VNF that should be provisioned as serverless with a FaaS VIM of her choice. Figure 27 provides a screenshot of a FaaS VNF on-boarding done via OSM R3. The reader should notice reference to the “image” of the OpenWhisk action implementing this FaaS VNF. This is a fully qualified action name by which it is accessible in OpenWhisk through the OpenWhisk REST API.

Figure 27: Onboarded FaaS VNFD for a VNF vTranscoder being part of Star Ball Game VNS

After all VNFs of a VNS are onboarded, the user can on-board the VNS itself. Figure 28 shows on-boarded VNS Star Ball Game comprising two FaaS VNFs interconnected by a virtual link.


63

Figure 28: Onboarded VNS Star Ball Game that comprises two vTranscoder FaaS VNFs interconnected by a virtual link

Next, as shown in Figure 29, the user selects a VNS for instantiation and launches it as shown in Figure 30 and it starts execution as shown in Figure 30. The details about the running VNS can be obtained as shown in Figure 31.


64

Figure 29: A User selects a VNS to instantiate

Figure 30: OSM view of a running FaaS VNS


65

Figure 31: Status of a running FaaS VNS

Virtual Network Function Descriptor (VNFD)

VNFD contains all information necessary to deploy a VNF. Since the assumed underlying technology is hypervisor based virtualization, vnfd:vdu information elements related to hypervisors include hypervisor_parameter elements with at least one parameter being mandatory. There is one special case, where there is no virtualization and a VNF should be provisioned on a non-virtualized system, but still managed through Virtualized Infrastructure Manager. To allow for this use case, ETSI standard85 allows to specify a reserved label “BARE” as a hypervisor type parameter (see Table 9). This constitutes some minor abuse of notation, but allows for a practical and efficient workaround. In the same spirit, we propose to use vnfd:vdu information elements related to hypervisors to describe information pertinent to FaaS based deployment of a VNF.

OSM R3 ignores vnfd:vdu related to hypervisors. However, we believe that in the future releases this might change and OSM will validate vfnfd:vdu hypervisor related information. To this end in a forward looking manner, we propose “FaaS” to be used to describe a hypervisor type and also add FaaS related parameters, such as specific FaaS framework type (e.g., OpenWhisk), OpenWhisk version, required PaaS service (e.g., Kubernetes), specific version of Kubernetes, etc. The resulting vdu will be a comprehensive description of the underlying FaaS technology required for this VNFD. This allows easy integration with any FaaS and PaaS and facilitates validation during the onboarding and instantiation flows. It should be stressed that

85 ETSI GS NFV-MAN 001 V1.1.1 (2014-12), p.49


66

this is the only mandatory extension in VNFD that is required to seamlessly integrate FaaS with the rest of the MANO stack.

Table 9 - ETSI Standard: vnfd:vdu information related to hypervisors

Identifier Type Cardinality Description

hypervisor_parameter Leaf 1…N There are a number of hypervisor related parameters that can have a significant impact on the deployment and performance of the VDU. These include: • Hypervisor type (see note 1) • Hypervisor version as a VDU may be validated with a particular version. • Hypervisor Address Translation support parameters including:

o Second Level Address Translation (see note 2). o Second Level Address Translation with Large page support. o Second Level Address Translation for I/O. o Second Level Address Translation for I/O with Large page support. Where "Large" is considered to be 1 GB or greater. o Support for interrupt remapping, i.e. supporting the IOMMU in the hypervisor. o Support of data processing acceleration libraries in the hypervisor, i.e. for acceleration libraries which require hypervisor support for high performance

NOTE 1: "BARE" could be included in this list if the network function component was to be deployed on a non-virtualised system, but still managed via the Virtualised Infrastructure Manager. NOTE 2: This needs to be included as the platform may support this feature but the hypervisor may not.


67

Conclusions

In this deliverable, we have presented the initial design and main technical specifications of the 5G-MEDIA Service Virtualization Platform. The SVP constitutes the central component of the 5G-MEDIA architecture, being responsible for the service orchestration and the intelligent deployment of the media VNFs/NSs on the physical resources. In 5G-MEDIA, the development of the SVP will rely on the Open Source MANO (OSM) release 3.0 and will integrate, extend and adapt the results from 5GPPP Phase 1 projects (CogNET, SELFNET, SONATA). The resulting platform will be instantiated and proof-of-concept validated considering three NFVIs with different characteristics (e.g., different virtualization technologies used) provided by the project partners ENG, OTE, TID.

The main contributions of this deliverable lie in the technical specifications design for each 5G-MEDIA SVP component (Service Orchestrator, Media Service MAPE, Resource Orchestrator), in light of the existing technologies and prototypes to be used during development as well as in the presentation of the target NFVIs, their characteristics and peculiarities to be considered during their integration in the 5G-MEDIA platform. More specifically, in the context of the service orchestrator, 5G-MEDIA will support the following characteristics: i) flexible configuration of the VNFs/NSs through their descriptors to allow intelligent instantiation supported by Media Service MAPE ii) deployment of VNFs/NSs in distributed NFVIs; iii) continuous monitoring of deployed VNFs/NSs and capability for upcscaling/downscaling depending on the application needs; iv) updates of VNFs/NSs instantiation based on Media Service MAPE outputs. In the case of Media Service MAPE, the main specifications identified are: i) selected monitoring tools for different virtualization technologies (OpenStack, VMWare, OpenNebula); ii) specification for real-time data communication between monitoring and CNO module; iii) specification of the scenarios supported by the machine learning algorithms including in the CNO, such as Mobile User Throughput Prediction, Anomaly Detection etc. iv) specification for the policy engine, part of the CNO, which will support intelligent VNF/NS placement, VNFFG optimization etc. For the Resource Orchestrator, the main achievements foreseen in 5G-MEDIA are: i) the implementation of the OpenNebula VIM to allow OnLife integration; ii) the development of FaaS VIM to enable the exploitation of the FaaS concepts in 5G. Finally, the three target NFVIs are presented illustrating the differences in terms of virtualization technologies (OpenStack, OpenNebula, VMWare), topologies and specific requirements (e.g., need for GPUs integration, need for Multi-PoP connectivity).

Overall, the deliverable provides an initial point of reference for the development teams to start the implementation of the integration and extensions of the baseline technologies. This initial design and specification is expected to evolve during the implementation phase and it will be refined in the deliverables “D3.2 Specification of the G-MEDIA Serverless Computing Framework” and “D3.3 Specification of the 5G-MEDIA QoS Control and Management Tools”.


68

Annex I – Installation steps of OSM v3.0

The flow of the commands that we used are:

> ssh -i <pem> ubuntu@ipv4 # Access the OSM VM

$ sudo apt-get update

$ sudo apt-get install -y lxd

$ newgrp lxd # add lxd user/group

$ sudo lxd init # Configure LXD - select the default options in the dialog

# Enable the IPv4 and NAT

# Disable the IPv6

# LXD creates a bridge named lxdbr0.

$ lxc list # This will drive initialization of lxdbr0

$ ip address show # Check the interfaces

$ ip address show ens3 # Inspect the default interface and check the MTU (i.e. 1450)

$ ip address show lxdbr0 # Inspect the bridge

# If MTU of ens3 and lxdbr0 differs, we use the ens3 MTU

$ sudo lxc profile device set default eth0 mtu 1450 # Use the appropriate MTU value

# Check if LXD has installed properly

$ lxc launch ubuntu:16.04 test # Create a container based on Ubuntu 16.04 with name 'test'

$ lxc exec test bash # Access the container

$ root@test:~# apt-get update # Run command 'apt-get update' from inside the container

$ root@test:~# exit # Exit from the container

$ lxc stop test # Stop the container

$ lxc delete test # Delete the container

# Download the OSM release 3

$ cd $HOME # /home/ubuntu

$ wget https://osm-download.etsi.org/ftp/osm-3.0-three/install_osm.sh

$ chmod +x install_osm.sh


69

# Install it through the bash script

$ ./install_osm.sh

# After approx half an hour, the installation will be completed...


70

Annex II – Data model for VNFDs in OSM v3.0

OSM key Name OSM type

High level Description

Group of fields Description of the fields

id String Identifier for the NSD

name String NSD name

short-name String NSD short name to use as a label in the UI

vendor String Provider of the NSD.

logo String File path of the vendor-specific logo

description String Description of the NSD

version String Version of the NSD

connection-point List List of the connection points of the NS.

● Name ● type

The Name and the type of the NS connection-point. Only value VPORT (virtual port) is supported.

vLd List List of the Virtual Link Descriptors of the NS.

● ID ● name ● short-name ● vendor

ID is the [key] identifier for the VLD. Vendor holds the name of the provider of the VLD.

constituent-vnfd List List of the VNF Descriptors that are part of this Network Service.

● Member-vnfd-index ● vnfd-id-ref ● start-by-default

Holds the unique id of the VNFD.

Holds the path of the VNFD [id].

States if the VNFD is started as part of NS instantiation.

scaling-group-descriptor

List List of scaling groups of the NS. The scaling groups consist of one or more VNFs. For each

● Name Name of the scaling-group .


71




scaling group a different scaling action may be applied.

● scaling-policy

Contains Scale-in/out criteria for this network service.

● vnfd-member (ui32)

Contains the list of the member VNF index and the number of each VNF's instances when a scaling action targets this group.

● min-instance-count ● max-instance-count

Minimum and maximum instances of that are allowed.

● scaling-config-action Contains the scaling trigger type (pre/post - scale in/out) and a reference to the NSC primitive name.

placement-groups

List List of placement groups. Each placement group defines the compute resource placement strategy in cloud environment.

● Name ● requirement ● strategy ● member-vnfd

Contains the resource placement strategy (COLLOCATION or ISOLATION) for a particular set of NS VNFDs.

Ip-profiles-list List List of IP profiles that describe the IP characteristics for the Virtual Link.

● Name ● Description ● ip-profile-params

Contains IP parameters for each profile. (IPv, subnet, gateway, DNS list,DHCP,subnet-prefix-pool/ VIM-specific reference)

vnf-dependency List List of VNF dependencies.

● Vnf-source-ref ● vnf-depends-on-ref

Describes which VNF depends on the source VNF.


72




vnffgd List List of VNF forwarding graph descriptor.

● Id ● name ● short-name ● vendor ● description ● version ● RSP ● classifier

Contains fields that uniquely describe each FG. Each VNFFG model consists of a list of rendered service path (RSP) and a list of classifier components. RSP is an ordered list of references to SF. The SF reference exists in the form of references to connection-points of the constituent-VNFDs. Classifier defines a list of rules for the VNFFGD.

monitoring-param

List List of network parameters for the NS.

● Id ● name ● monitoring-param-ui-data ● monitoring-param-value ● aggregation-type ● vnfd-monitoring-param,

Contains fields that refer to parameters for:

● UI-data (eg. Histogram, bar, units/Mbps)

● values (eg. INT,STRING)

● aggregation-types (eg. AVG,SUM)

● Reference to VNFD or VNFD monitoring parameter

input-parameter-xpath

List List of xpath to parameters inside the NSD that can be customized during instantiation.

● Xpath ● Label ● default-value

Xpath that specifies the element in a descriptor, along with a descriptive string (label) and a default value for this input parameter.

parameter-pool List Pool of parameter values that must be pulled from during configuration.

● Name ● range

Name of the conf value pool and a Range of values from which to populate the pool. (Start/End values)

service-primitive List Network service level configuration primitives.

● Name, ● parameter ● parameter-group ● vnf-primitive-group ● user-defined-script

Holds the name of the service primitrive, a grouping of parameters that are logically grouped in UI, a List of service parameters grouped by the


73




VNF and a user defined script.

Parameters of a Service-Primitive include:

● name ● data-type

(string,int,boolean)

● mandatory (boolean)

● def_value

initial-config-primitive

List Set of configuration primitives to be executed when the network service comes up.

● seq ● name ● user-defined-script ● parameter

Holds the sequence number (seq) for the configuration primitive, its name, a user-defined script and a list of parameters with the subfields (name, value).

terminate-config-primitive

List Set of configuration primitives to be executed before during termination of the network service.

● seq ● name ● user-defined-script ● parameter

cloud-config List Configures the list of users and public keys to be injected as part of network service instantiation.

● key-pair (name,key) ● user (name, user-info, key-

pair)


74

Annex III – Data model for NSDs in OSM v3.0

OSM key Name

OSM type

High level Description Group of fields Description of the fields

id String Identifier for the VNFD

name String VNFD name

short-name String VNFD short name to use as a label in the UI

vendor String Provider of the VNFD.

logo String File path of the vendor-specific logo

description String Description of the VNFD

version String Version of the VNFD

vnf-configuration

container Information about the VNF configuration for the management interface.

● config-method

Defines the configuration method for the VNF

● script (BASH, EXPECT)

● Juju charm to use with the VNF

● config-access IP, username and password to be used to configure this VNF

● config-attributes

● config-priority ● config-delay

● service-primitive

Holds the name for the config primitive, a list of parameters and a user defined script.

● initial-config-primitive

Initial set of configuration primitives.


75

OSM key Name

OSM type


● config-template

Configuration template for each VNF.

mgmt-interface

container Interface over which the VNF is managed.

● endpoint type (IP, vdu-id,cp)

● port ● dashboard-

params (path, https, port)

internal-vld List List of Internal Virtual Link Descriptors (VLD).

● ID ● name ● short-name ● vendor ● description ● version

Unique description of the internal virtual link

● type

ELAN - multipoint service that connects a set of VDUs

● root-bandwidth

For ELAN this is the aggregate bandwidth

● leaf-bandwidth

For ELAN this is the bandwidth of branches

● internal-connection-point

List of internal points in this VLD represented as id-refs

● virtual-connection-points

List of (available) virtual-connection points associated with this virtual Link.

● name, id, short-name

● type (VPORT)


76

OSM key Name

OSM type


● port-security-enabled (if true RO passes the value to the VIM to filter traffic.)

● static-ip-address

● associated-cps (list of id-refs of cp associated with vcp )

● provider-network

● physical-network

● overlay-type (identifies the type of the overlay network - LOCAL, FLAT, VLAN, VXLAN, GRE)

● segmentation-id

init-params ● vim-network-

name ● ip-profile-ref

Ip-profiles List List of IP profiles that describe the IP characteristics for the

Virtual Link.

● Name ● description ● ip-profile-

params

Contains IP parameters for each profile. (IPv, subnet, gateway, DNS list,DHCP,subnet-prefix-pool/ VIM-specific reference)

connection-point

List List for the external connection points

● name ● id ● short-name

Unique description of the cp.

● type VPORT

● port-security-enabled

When set to True, the resource orchestrator passes the value to the


77

OSM key Name

OSM type


VIM when the connection-point is created to filter traffic. (Openstuck VIM only)

● static-ip-address

IPv4 or IPv6

vdu List List of virtual deployment units (VDUs).

● id ● name ● description ● count ● mgmt-vpci ● vm-flavor ● guest-epa ● vswitch-epa ● hypervision-

epa ● host-epa ● alarm ● image-

properties ● cloud-input ● supplemental-

boot-data ● internal-

connection-point

● internal-interface

● external-interface

● volumes

VDUs are virtual machines that host the network function, such as:

● Virtual machine specification

● Computation properties (RAM size, disk size, memory page size, number of CPUs

● number of cores per CPU, number of threads per core)

● Storage requirements

● Initiation and termination scripts

● High availability redundancy model

● Scale out/scale in limits

vdu-dependency

List List of VDU dependencies, from which the orchestrator determines the order of startup for VDUs.

● vdu-source-ref

● vdu-depends-on-ref

A reference to the VDU on which the source VDU depends on.


78

OSM key Name

OSM type


service-function-chain

enum Type of node in the service function chaining (SFC) architecture: UNAWARE (default), CLASSIFIER, SF, SFF

service-function-type

String Type of service function. This field is temporarily set to string data type for ease of use.

monitoring-param

List List of network parameters for the VNF.

● http-endpoint ● monitoring-

param ● monitoring-

param-ui-data ● monitoring-

param-value

Contains fields that refer to parameters for:

● http-endpoint (list of http endpoints to be used by monitoring params)

● monitoring-param/ui-data (includes parameters for JSON queries, http-endpoint reference and UI monitoring parameters as described for NSD)

placement-groups

List List of placement groups at VNF level. Thew group construct defines the compute resource placement strategy in cloud environment

● name ● requirement ● strategy ● member-

VDUs

Contains the resource placement strategy (COLLOCATION or ISOLATION) for a particular set of NSs/VNFDs. Also contains a list of VDUs that area part of this placement-group.

programmable edge-to-cloud virtualization fabric for the ... · caas container as a service co...

Documents