Heartbleed

...and why yours should, too

You are in the right session

_ This  is  an  emergency  service  announcement  _ Due  to  events  that  transpired  on  Tuesday  _ I  thought  it‘d  be  good  to  have  some  info  

10.04.14 OSDC 2014 2  

About me

_ Dr.  Christopher  Kunz  _ Studied  CompSci  in  Hannover,  PhD  in  2012  _ Works  as  a  hoster  for  15  years  

_ Some  admin  experience  

_ Used  to  do  a  lot  of  PHP  _ Author,  „PHP-­‐Sicherheit“,  ed.  1-­‐3  

_ And  don‘t  get  me  started  about  swords!  

10.04.14 OSDC 2014 3  

About filoo

_ hQps://www.filoo.de  _ Quickly-­‐growing  hosVng  company  _ Data  center  in  Frankfurt,  Germany  _ Developed  own  IaaS  middleware  

_ QEMU/KVM,  OVS,  Ceph  

_ Offer  hosVng,  co-­‐locaVon,  cloud  services  _ 100%  subsidiary  of  Thomas-­‐Krenn.AG  _ Visit  their  booth!  

10.04.14 OSDC 2014 4  

Heartbleed in a nutshell

_ A  bug  with  a  cute  name  _ ...and  not  so  cute  effects  _ Pre-­‐auth,  pre-­‐logging  universal  TLS/SSL  bug  _ Introduced  in  OpenSSL  1.0.1a  (2012)  _ Allows  to  make  64kb  memory  dumps  of  the  server‘s  memory  

10.04.14 OSDC 2014 5  

Wait. What?

_ Yes,  remote  memory  dumps  _ Due  to  an  unchecked  buffer  length,  a  TLS  enabled  server  may  dump  memory  contents  to  the  client  _ Limit  of  64k  per  reply  _ MulVple  replies  possible  _ Memdump  may  contain...  

_ URLs  and  GET  /  POST  variables  _ Random  excerpts  from  whatever  _ Source  code  of  scripts/whatever  else  _ SSL  cerVficate  private  keys  

10.04.14 OSDC 2014 6  

About DTLS heartbeats

_ RFC  6520,  Transport  Layer  Security  (TLS)  and  Datagram  Transport  Layer  Security  (DTLS)  Heartbeat  Extension  _ Provides  a  heartbeat  for  TLS  (TCP)  and  DTLS  (mostly  UDP)  sessions  _ Intended  to  add  stability  to  unstable  connecVons  and  prevent  renegoVaVons  _ Implemented  in  OpenSSL  as  part  of  a  PhD  thesis  _ Patch  commiQed  Dec  15,  2011    

10.04.14 OSDC 2014 7  

What this bug is not

_ This  is  not  a  crypto  bug  _ At  least  not  in  its  primary  funcVon  

_ This  is  not  a  fully  arbitrary  mem  disclosure  _ Only  memory  belonging  to  aQacked  daemon  can  be  dumped  

_ This  is  not  a  remote  root  hole  _ Hence  the  relaVvely  low  CVE  score  of  5.0  

10.04.14 OSDC 2014 8  

Anatomy of the bug 1

struct { HeartbeatMessageType type; uint16 payload_length; opaque payload[HeartbeatMessage.payload_length]; opaque padding[padding_length];

} HeartbeatMessage;

_ From RFC6520: _ payload_length: The length of the payload. _ payload: The payload consists of arbitrary content.

10.04.14 OSDC 2014 9  

Anatomy of the bug 2

_ ssl/d1_both.c,  line  1474+:  buffer = OPENSSL_malloc(1 + 2 + payload + padding);

bp = buffer;

[..]

memcpy(bp, pl, payload);

_ From: https://github.com/openssl/openssl/commit/4817504d069b4c5082161b02a22116ad75f822b1

10.04.14 OSDC 2014 10  

Anatomy of the bug

_ The  heartbeat  extension  allocates  payload+19  bytes  of  memory  _ Copies  pl  bytes  of  arbitrary  user-­‐supplied  data  payload  via  memcpy()  to  construct  response  _ Client  sets  pl  to  65535  _ Client  sends  only  1  byte  of  data  in  payload

_ Response  contains  1  byte  of  client-­‐supplied  payload  _ ...and  64K  of  RAM  from  the  memcpy()  call  _ Analysis  in:  hQp://blog.existenValize.com/diagnosis-­‐of-­‐the-­‐openssl-­‐heartbleed-­‐bug.html  

10.04.14 OSDC 2014 11  

Test vulnerability

_ Python  script  at:  hQps://gist.github.com/takeshixx/10107280  _ Can  test  any  SSL/TLS  enabled  TCP  service  

_ Has  support  for  StartTLS  (-­‐s  opVon)  _ Conveniently  dumps  64kb  of  memory  for  you  

10.04.14 OSDC 2014 12  

00d0: 10 00 11 00 23 00 00 00 0F 00 01 01 33 41 31 25 ....#.......3A1% 00e0: 32 43 25 32 32 5F 6D 6F 64 65 25 32 32 25 33 41 2C%22_mode%22%3A 00f0: 25 32 32 6A 73 6F 6E 25 32 32 25 32 43 25 32 32 %22json%22%2C%22 0100: 5F 69 64 25 32 32 25 33 41 25 32 32 70 5F 33 30 _id%22%3A%22p_30 0110: 33 34 35 38 31 38 25 32 32 25 32 43 25 32 32 5F 345818%22%2C%22_ 0120: 63 6F 6E 74 61 69 6E 65 72 25 32 32 25 33 41 30 container%22%3A0 0130: 25 32 43 25 32 32 5F 61 63 74 69 6F 6E 25 32 32 %2C%22_action%22 0140: 25 33 41 25 32 32 76 69 65 77 25 32 32 25 32 43 %3A%22view%22%2C

Memdump

_ From:  hQps://twiQer.com/markloman/status/453502888447586304  

10.04.14 OSDC 2014 13  

Memdump

_ Memory  contents  is  non-­‐determinisVc  _ SomeVmes  exciVng,  mostly  boring  

_ while true do python hb-test.py yahoo.com | grep -C 2 login >> /tmp/out; sleep 1; done"

_ Profit!  

10.04.14 OSDC 2014 14  

Detect exploitation

_ No  logging  on  the  machine  _ All  exploitaVon  is  pre-­‐logging,  pre-­‐applicaVon  _ IDS  vendors  are  pushing  out  signatures  already  

10.04.14 OSDC 2014 15  

Affected services

_ Above  all,  SSL-­‐enabled  web  servers  _ Any  that  uses  OpenSSL,  anyway  

_ Mail  servers  _  IMAP  over  SSL,  POP  over  SSL,  SMTP  over  SSL,  StartTLS  

_ VPN  tunnels  _ OpenVPN  when  using  cert  auth  (maybe?)  _ PotenVally  others  

_ IRC  servers,  XMPP,  FTP  over  TLS  _ Android  4.1.1  is  vulnerable  _ OpenSSH  is  not  vulnerable  

10.04.14 OSDC 2014 16  

Linux versions affected

_ OpenSSL  1.0.1  a  thru  f  _ Debian  Wheezy,  Jessie,  Sid  

_ Fixed  for  Wheezy  &  Sid  

_ Ubuntu  10.04,  12.04,  12.10,  13.10,  14.04  _ Fixed  packages  exist  

_ RHEL  6  _ Patch  exists  

_ And  all  others  that  ship  OpenSSL  _ Clients  are  also  vulnerable!    

10.04.14 OSDC 2014 17  

Other affected stuff

_ Cisco  devices:  „We  use  Cisco  SSL  which  is  not  OpenSSL.“;  SSL  VPN  products  potenVally  affected  _ Juniper  has  released  fixes  for  their  SSL  VPN,  none  for  J-­‐Web  etc.  yet  _ Big  IP?  Kemp?  Fritz.Box?  Your  home  NAS?    _ More  info  (hopefully)  here:  hQp://www.kb.cert.org/vuls/byvendor?searchview&Query=FIELD+Reference=720951&SearchOrder=4  

10.04.14 OSDC 2014 18  

Mitigation & cleanup

_ First,  upgrade  to  fixed  openssl  _ apt-­‐get  install  openssl  libssl-­‐1.0.0  

_ Next,  restart  all  services  that  load  old  lib  _ Use  checkrestart  or  lsof  –n  |  grep  DEL  |  grep  ssl  

_ If  you  use  staVc  binaries,  recompile  everything  _ If  you  use  Google‘s  mod_spdy  on  Apache2.2,  don‘t  

_  It  has  its  own  staVcally  linked  mod_ssl  which  is  shamefully  out  of  date  

10.04.14 OSDC 2014 19  

What about certs?

_ It  is  possible  that  privkeys  have  leaked  _ If  so,  you  need  to  revoke&reissue  certs  _ Some  CAs  offer  free  reissue  _ If  you  don‘t  have  PFS,  you  have  a  problem  _ AQackers  who  sniffed  your  traffic  might  be  able  to  decode  it  

10.04.14 OSDC 2014 20  

Thank you

_ .Do  not  despair,  there  is  hope!  

_ ...and  now,  back  to  our  regular  scheduled  programme!  

10.04.14 OSDC 2014 21  

hQp://xkcd.com/1353/  

Software-defined Networking

In an open-source cloud

Agenda

_ High-­‐Level  overview:  What  is  this  about?  _ The  use  case  –  virtualized  networks  for  IaaS  _ Intro  to  OpenVSwitch  _ How-­‐to:  Deploy  OpenVSwitch  _ Frontnet,  Backnet,  public  net  _ Firewalling  _ Tying  it  all  together  

10.04.14 OSDC 2014 23  

So what‘s the hype?

_ Sovware-­‐Defined  Networking  is  the  hype  _  I‘m  not  good  with  hype  

_ Networking  is  decoupled  from  bare  metal  _ EssenVally  you  virtualize  parts  of  your  network  _ Control  and  data  plane  are  decoupled  

_ Many  vendors  jumped  on  the  train  _ HP,  Cisco,  VMWare,  you  name  it  

10.04.14 OSDC 2014 24  

OpenFlow

_ ImperaVve  control  _ Switches  are  dumb  –  they  only  forward  according  to  rules  _ OpenFlow  controllers  make  the  rules  _ First  packet  of  each  type  is  sent  thru  OpenFlow  controller  _ Subsequent  ones  go  directly  through  switch  

10.04.14 OSDC 2014 25  

OpFlex

_ Cisco‘s  answer  to  OpenFlow  _ Other  vendors  on  board:  Citrix,  MSFT,  RHAT,  Canonical  _ Not  on  board:  J,  HP,  Huawei,  vmWare  

_ Balance  intelligence  between  switch  and  controller  _  „DeclaraVve  control“;  just  declare  how  you  want  it  and  the  switch  interprets  that  rule  

_ IETF  proposed  standard  _ Drav-­‐smith-­‐opflex  _ Open  APIs  

_ AltruisVc  goal:  Eliminate  SPOF  (the  controller)  _ EgoisVc  goal:  Sell  smarter  (=$++)  switches  

10.04.14 OSDC 2014 26  

The OSS Contender

_ OpenVSwitch  _ Openvswitch.org  

_ Open  Source  _ Apache  2.0  license,  non-­‐viral  _ GPLv2  

_ MulVlayer  (2,3)  virtual  switch  

_ Supports  lots  of  interesVng  features  _ VLANs,  Ne{low,  sFlow,  LACP,  filtering,  ...  

10.04.14 OSDC 2014 27  

OVS Overview

_ Shamelessly  lived  from  [1]  

10.04.14 OSDC 2014 28  

ovs-vswitchd

OVS Kernel Module

Control Cluster

ovsdb-server

Off-box

User Kernel

Management Protocol (6632/TCP)OpenFlow (6633/TCP)Netlink

OSVDB

_ Database  holds  configuraVon  items  _ DefiniVons  for  bridges,  tunnels,  interfaces  _ Controller  addresses  

_ ConfiguraVon  is  reboot-­‐safe  _ Custom  database  system,  not  MySQLiteMongoDB    _ Speaks  custom  protocol  (OSVDB)  _ Log  based  

_ osvdb-­‐tool  show-­‐log  shows  all  changes    _ Nivy  for  debug  /  change  management!  

10.04.14 OSDC 2014 29  

How ovs works

_ ImperaVve  control  _ All  intelligence  is  in  the  controller  _ Data  path  only  carries  out  instrucVons  

_ Data  Path  _ Kernel  module    _  Licensed  under  GPLv2  

_ Controller  _  Lives  in  userland  _  Licensed  under  Apache  2.0  

10.04.14 OSDC 2014 30  

Flow flow

_ Everything  is  a  flow  _ CombinaVon  of  input  port,  VLAN,  MAC,  IP,  TCP/UDP  port  

10.04.14 OSDC 2014 31  

OVS management

_ Command-­‐line  tools  _ Ovs-­‐vsctl  for  switch  management  _ Ovs-­‐ofctl  for  flow  management  _ Ovsdb-­‐tool  for  database  management  

10.04.14 OSDC 2014 32  

What‘s our angle here?

_ filoo  is  a  hoster.  

_ We  host  VMs.  

_ VMs  need  networking.  

_ See  where  this  goes?  

10.04.14 OSDC 2014 33  

What we wanted

_ Internet-­‐facing  front-­‐net  interface  _ Private  LAN  for  VMs  _ VM  isolaVon  _ Firewalling  _ Traffic  shaping  _ Fine-­‐grained  accounVng  _ Live  migraVon  

10.04.14 OSDC 2014 34  

Overview - physical

10.04.14 OSDC 2014 35  

Back-­‐end  switch  

Front-­‐end  switch  

Overview - virtual

10.04.14 OSDC 2014 36  

Firewall   Firewall   Firewall  

Overview – OVS stack

10.04.14 OSDC 2014 37  

OVS   OVS   OVS  

Let‘s get started

_ We  usually  compile  ovs  ourselves  _ There  are  also  packages  in  apt  _ Those  might  work  or  not  

_ Download  &  compile  OVS  _  Latest  stable:  2.1.0,  latest  LTS:  1.9.3  _  ./boot.sh  &&  ./configure  &&  make  &&  make  install  

_ Kernel  module  from  3.3+  _ Enable  in  Kernel  Networking  -­‐>  OpVons  -­‐>  Open  Vswitch  _ modprobe  openvswitch  

10.04.14 OSDC 2014 38  

Let‘s get started 2

_ Set  up  ovs  db  _ Ovsdb-­‐tool  create  conf.db  vswitch.ovsschema  _ Conf.db  is  in  /usr/localetc/openvswitch  _  /usr/src/openvswitch-­‐1.9.3/vswitchd/vswitch.ovsschema    

_ Make  sure    ovs-­‐vswitchd  and  ovsdb-­‐server  start  before  networking  _ Add  startup  entries  to  rc.local  _ Remove  networking  from  rc.d  _  start  networking  in  rc.local  

10.04.14 OSDC 2014 39  

Initial bridges

_ Front-­‐net  vlan:  199  _ Same  procedure  for  back-­‐net  VLAN  _ Add  bridges  

_ ovs-­‐vsctl  add-­‐br  vmbr1  _ ovs-­‐vsctl  add-­‐port  vmbr1  vlan199  tag=199  _ ovs-­‐vsctl  set  interface  vlan199  type=internal  

_ Log  in  via  IPMI  _ ovs-­‐vsctl  add-­‐port  vmbr1  eth1  _ Machine  is  offline  now  

_ Modify  physical  switching  

10.04.14 OSDC 2014 40  

VM networking

_ We  use  KVM/QEMU  _ Add  the  TAP  interface  

_  /sbin/ip  tuntap  add  dev  tap1i0d0  mode  tap  user  fcms  _ qemu-­‐system-­‐x86_64  ...  -­‐device  rtl8139,mac=00:F1:70:00:00:10,netdev=vlan0d0  -­‐netdev  type=tap,id=vlan0d0,ifname=tap1i0d0  

_ Bring  up  the  port  _  /usr/local/bin/ovs-­‐vsctl  add-­‐port  vmbr0  tap1i0d0  199  other_config:stp-­‐enable=false  

10.04.14 OSDC 2014 41  

From TAP to port to flow

_ We  have  a  tap  interface  tap1i0d0  

_ Find  the  corresponding  bridge  port:  _ ovs-­‐ofctl  show  vmbr0  |  grep  tap1i0d0    _ 1820(tap1i0d0):  addr:fa:7a:67:e3:5d:�    

_ Now  we  have  a  port  number:  1820  

_ We  use  this  port  for  flow  management  

10.04.14 OSDC 2014 42  

Multiple interfaces

_ Add  more  TAP  interfaces  _ Assign  one  VLAN  per  customer  _ Internal  network  across  VMs  on  same  node    

_ Make  VLAN  known  on  inter-­‐node  switches  _ Via  whatever  switch  automaVon  you  have  

_ Cross-­‐node  internal  networking  _ VLAN  limits  apply  –  hard  cut  at  ~4090    _ Overlay  networks  to  the  rescue  

10.04.14 OSDC 2014 43  

Prevent MAC spoofing

_ PORT=1820  _ ovs-­‐ofctl  add-­‐flow  vmbr0  "in_port="${PORT}"  arp  idle_Vmeout=0  priority=39500  acVon=resubmit("${PORT}",2)“  _ ovs-­‐ofctl  add-­‐flow  vmbr0  "in_port="${PORT}"  table=2  arp  priority=200  idle_Vmeout=0  arp_sha=00:F1:70:00:00:10  nw_src=192.168.1.1  acVon=normal"    _ ovs-­‐ofctl  add-­‐flow  vmbr0  "in_port="${PORT}"  table=2  priority=100  idle_Vmeout=0  acVon=drop"  

10.04.14 OSDC 2014 44  

We  know  this  MAC  because  we  control  the  hypervisor!  

We  know  this  address  too!  

Caveats for MAC/ARP

_ SomeVmes  you  want  customers  to  spoof  _ HA  soluVons  that  switch  „cluster  IP  addresses“  _ You  can  cater  for  this  in  case  you  know  the  corresponding  MACs  _ Assign  sequenVal  MACs  and  wildcard  _ Or  set  specific  rules  _ OpVonal  „HA  feature“  for  VMs  _ Never  allow  customers  to  wildcard  here!  

10.04.14 OSDC 2014 45  

Firewalling with flows

_ ovs-­‐ofctl  add-­‐flow  vmbr0  "in_port="${PORT}"  table=1  tcp  idle_Vmeout=0  nw_dst=192.168.12.13/32  nw_src=192.168.1.123/32  tp_dst="80"  priority=38000  acVon=drop“  _ From  192.168.1.123    _ To  192.168.12.13  _ Port  80  _ Drop  

10.04.14 OSDC 2014 46  

Port ranges

_ ovs-­‐ofctl  add-­‐flow  vmbr0  "in_port="${PORT}"  table=1  tcp  idle_Vmeout=0  nw_src=192.168.1.123/32  nw_dst=192.168.12.13/24  tp_src="0x05E8/0xFFFC"  priority=37960  acVon=drop“  _ Source  192.168.1.123  _ DesVnaVon  192.168.12.0/24  _ Source  port  =  0x05E8/0xFFFC  _ 0x05E8/0xFFFC  =  1512/65532  _ Port  1512  –  1516  

_ OVS  1.11  supports  „Megaflows“,  i.e  universal  wildcarding  

10.04.14 OSDC 2014 47  

Default accept

_ ovs-­‐ofctl  add-­‐flow  vmbr0  "in_port="${PORT}"  table=1  priority=100  acVon=normal“  _ Fallthru  rule  _ Match  everything  else  

10.04.14 OSDC 2014 48  

Accounting

_ We  grab  interface  counters  from  the  tap  interfaces  _ You  can  also  use  Ne{low/sFlow  or  ipfix    _ We  didn‘t  go  there  yet,  experiences  welcome  

10.04.14 OSDC 2014 49  

Shaping

_ Simple  shaping:  _ ovs-­‐vsctl  set  Interface  tap0  ingress_policing_rate=100000  _ ovs-­‐vsctl  set  Interface  tap0  ingress_policing_burst=1000  

_ QoS  policies:  _ ovs-­‐vsctl  set  port  eth1  qos=@newqos  \  id=@newqos  create  qos  type=linux-­‐htb  \  other-­‐config:max-­‐rate=200000000  queues=0=@q0,1=@q1  \  

_ We  don‘t  do  QoS  policies,  shaping  works  mostly  as  intended  

10.04.14 OSDC 2014 50  

Live migration

_ We  don‘t  actually  do  OVS‘s  own  live  migraVon  _ Start  VM  on  target  host  in  suspend-­‐to-­‐RAM  mode  _ Stop  VM  on  losing  host;  down  interface  _ Resume  VM  on  target  host  

_ There  are  live  migraVon  mechanisms  in  OVS  _  L2  based    _  Inter-­‐OVS  GRE  tunnel  _ Honestly,  I  have  no  clue.  

10.04.14 OSDC 2014 51  

Thank you

_ I  hope  you  learned  something  _ If  not,  I  hope  you  had  a  laugh  at  my  expense  _ If  neither,  I‘m  really  sorry.  Beer?  

_ QuesVons?  

10.04.14 OSDC 2014 52  

Literature

_ [1]  hQp://openvswitch.org/slides/OpenStack-­‐131107.pdf  –  OVS  Deep  Dive  _ OVS  IntroducVon:  hQp://horms.net/projects/openvswitch/2010-­‐10/openvswitch.en.pdf  

10.04.14 OSDC 2014 53