29 May 2017 1

DeiC DDoS Prevention System - DDPS

Niels Thomas Haugård, DeiC

Consultant since 1995 in the security and network field

See https://www.linkedin.com/in/uninth

and https://www.deic.dk

29 May 2017 2

> About DeiC> Danish NREN

> DeiC – Danish e-Infrastructure Cooperation has the stated aim of supporting the development of Denmark as an eScience nation through the delivery of e-infrastructure(computing, data warehousing, network connections and auxiliary services), guidance, and initiatives at national level. DeiC is a virtual unit under the Danish Ministry of Higher Education and Science and the result of an agreement concluded between the eight Danish universities and the Danish Agency for Science and Higher Education. For details, see www.deic.dk

29 May 2017 3

> Why present it> Benefit for others

> Exchange of ideas never made something worse

> Open source and on github:https://github.com/deic-dk/DDPS-documentation

29 May 2017 4

> The next 42 minutes> Why we made DDPS - A short description of DDoS attacks

> How we designed it and why

> What it looks like, how to handle failures etc.

> Lessons learned

> Current Status

> Who made DDPS

> Questions ?

29 May 2017 5

> Why do it - 20 years of DDoS history

29 May 2017 6

> akamai’s state of the internet / security Q2 2017

2017, Q1

29 May 2017 7

> State of Forskningsnet> Early warnings only

> One small incident in the summer of 2016

> Attacking student dormitories in Ålborg (World of Warcraft) and Ålborg University

> Not visible on traffic graphs

> Our infrastructure attacked

> Hosting Itslearning AS with national tests

> Regular attacked

29 May 2017 8

> DDoS characteristic > Stateless connections (UDP, TCP syn, GRE, ICMP echo request)

> Amplification attack with spoofed source addresses

> Attack on kernel and network layer (Ping of Death, Teardrop osv.)

> Application layer attack (Slowloris, R-U-Dead-Yet osv.)

> Massive legitimate connections

ISP focus area

29 May 2017 9

> DDoS mitigation challenges for ISP’s> Volumetric only - will / cannot take responsibility for application layer attacks

> Attacker has more bandwidth than the target

> Manual blocking is slow, labor intensive and error prone

> The blocking usually takes place to late

29 May 2017 10

> DDoS mitigation ISP - customer expectations> Very short responce time

> Mitigation must be implemented in uplink and peers

> (as close as possible on the source)

> ISP’s should cooperate

29 May 2017 11

> Mitigation methods> BGP based Blackhole routing: scarify attacked addresses (først RFC3882 fra 2004)

> Predefined route communities (all, ISP, … )

> Described in rfc5635 og rfc7999

> BGP flowspec (RFC 5575 fra 2009) match/action firewall rules in edge routers

> Attacked addresses not scarified: filtering on 12 network parameters

> Implanted ca 2013 in Juniper routers, Cisco ca 2014

> See Nokia/Alcatel, Cisco, Huawei og Juniper: BGP Flow Specification: Multi Vendor and Inter AS Interoperability January 2, 2017 (https://www.nextlayer.at/flowspec-paper.pdf) - a sad and long story, but the most general option

29 May 2017 12

> Other methods> Loadshare

> Cloudflare and others

> Rewrite your application and hand over your SSL keys - not an option for all

> Assumes service provider have more bandwidth than the attacker (usually they do)

> Any cast

> Multiple hosts one IP address - Service configured as a cluster

> Our MX960’ers must filter differently on traffic from NORDunet and forskningsnet

> Assumes attach only through NORDunet - wrong assumption

29 May 2017 13

> No matter the solution> Your investment in DDoS protection is worth nothing

while you are not under attack

> Defending is more expensive than attacking

> Attacks never worsens over time

> You cannot predict what kind of attack will be used next time

> IPv6 not really supported see https://tools.ietf.org/html/draft-ietf-idr-flow-spec-v6-03 (2012)

29 May 2017 14

> Existing solutions> Onsite: Arbor Network, F5 m.fl.:

> Expensive, proprietary, and will they fit our infrastructure and customers?

> Cloud: Cloud Flare, incapsula m.fl.

> Expensive cloud based load share, requires your SSL keys

> Open source alternatives

> FastNetMon

> Firewall on Demand (grnet / GÉANT)

29 May 2017 15

> Firewall on Demand (grnet / GÉANT)> No longer maintained at the time we started the project

> Configuration based on netconf - not used here

> requires shibboleth for authentication while we would use WAYF — the digital recycling centre

> Lack of automatic detection

> A good source for inspiration

29 May 2017 16

> Mitigation Opportunities> Do not change existing customer infrastructure nor scarify hosts for the network

> black hole

> BGP flowspec

> load share

> multicast

> Price, Performance (and maintainable code)

> Commercial

> Open source

> Home grown

29 May 2017 17

> What we would like …> Existing services should not be changed to match a solution

> BGP flowspec benefits:

> May be implemented in existing (aka nor new) network equipment

> No changes to services needed (unlike e.g. load share)

> Builds on top of existing BGP peering

> Requires

> Manual changes implemented ahead of the attack

> Automatic detection and mitigation

29 May 2017 18

> BGP flowspec as a security tool> eBGP: announce what network you would like to receive traffic for from a given peer

> only as long as the BGP session is active (TCP)

> BGP flowspec :Add discrimination to your eBGP announcement

> Will receive traffic for our network

> except traffic matching some criteria

> Think of BGP flowspec as a transport protocol for firewall rules

> Rules are implemented in the edge routers

> Rules: closest match wins (cleanup not really required)

> Add start and end time on top

29 May 2017 19

> Simple static filtering - not opt-in

announce flow route default-discard-udp-fragments { match { ; protocol udp; fragment [ is-fragment first-fragment last-fragment ]; } then { discard } }

announce flow route default-discard-ntp-amplification { match { ; protocol udp; source-port =123; packet-length =468; } then { discard } }

announce flow route default-discard-dns-amplification { match { ; protocol udp; source-port =53; packet-length =512; } then { discard } }

announce flow route default-discard-dns-amplification { match { ; protocol udp; source-port =19; } then { discard } }

announce flow route default-discard-chargen { match { ; protocol udp; source-port =19; } then { discard } }

announce flow route default-discard-chargen { match { ; protocol tcp; source-port =19; } then { discard } }

announce flow route default-discard-QOTD { match { ; protocol udp; source-port =17; } then { discard } }

announce flow route default-discard-QOTD { match { ; protocol tcp; source-port =17; } then { discard } }

announce flow route default-discard-gre { match { ; protocol =47; source-port =17; } then { discard } }

announce flow route default-ratelimit-SSDP { match { ; protocol udp; source-port =1900 } then { rate-limit 9600; } }

announce flow route default-ratelimit-snmp { match { ; protocol udp; source-port =161&=162; } then { rate-limit 9600; } }

29 May 2017 20

> A better solution> Prefer BGP flowspec: integrates seamlessly with existing infrastructure and services

> Opt-in: use on-site probe (FastNetMon)

> Use WAYF for Web-UI authorization

> Non commercial version of FastNetMon, home grown Web-UI, daemon etc

> GUI and decentralized probes to inject rules in a central database

> No direct database access, no fancy interface, use sftp and file upload

> Access only granted through OpenVPN - with few exceptions (protecting the GUI)

> Short-lived blocking only: Not first line firewall with permeant rules

29 May 2017 21

> Design principles> Precautionary principle: Opt-in

> Both Self-service and Automatic mitigation

> Least privilege / least authority: restrict rule creation to specific network(s)

> Fail safe: do nothing on e.g data failure

> Automatic remove access for administrators who are no longer employed

> Automatic mitigation

> Time critical, mitigation start time/end time, logging and status

> Modular building blocks: software components should be replaceable

> Limit hardware requirements

> install on-site: On-site is 10Gb, core is 100Gb

> 2 x interfaces: one for monitoring (mirror port) one for uplink (assume behind NAT)

29 May 2017 22

> Design> Redundancy where technical and economical feasible

> Redundant functionality or fast recovery procedure elsewhere

> Decentral probes based on FastNetMon

> Central rule database with Web-UI etc.

> ExaBGP hosts, firewalls, routers etc.

29 May 2017 23

> Components> FastNetMon host

> Debian8, ixgbe and ixgbe drivers

> 2 connections: mirror port and uplink: (out of band: ADSL/LTE/3G … - behind NAT)

> FastNetMon, influxdb, notify_script, OpenVPN, ssh, configuration

> Two way connections: VPN required for access to influxdb

> DDPS host

> OS, Ubuntu 16.04

> PostgresDB, daemon,

> Web-UI, cli tools

> tool to create ISO images for unattended installation (Debian and Ubuntu)

> ExaBGP, routers etc.

29 May 2017 24

> FastNetMon> Hardware: Supermicro X10SLL-F with Intel NIC 82599

> Debian 8, looking at Ubuntu 16.04 / FreeBSD 11.1 / Debian 9

> Pktgen-DPDK

> The latest version is located on dpdk.org at http://dpdk.org/download

> Traffic generator powered by Intel's DPDK at 10Gbit wire rate traffic with 64 byte frames.

> pkt-gen -i eth1 -z -f tx -n 500111222 -l 60 -d a.b.c.d:123

> Handles up to 14 million 64 bit UDP packets

> Tested with FastNetMon / package generator host back-to-back on - Debian 8

29 May 2017 25

> FastNetMon> Commercial: all the nice options but license requires routable IANA address

> Community edition: perl script, which execute git, cmake etc.

> A new version each time some changes something on git

> Looking at FreeBSD - is in ports together with netmap drivers

> We use OpenBSD, Ubuntu and Debian (and GAiA and SPLAT)

> Working towards compile once - use everywhere on similar OS

> statical linking and cmake (not funny)

> statifier fails (Kernel unble to load executable where phrs took more then PAGE_SIZE)

> More work here

29 May 2017 26

> FastNetMon on Debian 9> echo 'deb http://ftp.de.debian.org/debian sid main' > /etc/apt/sources.list.d/fastnetmon.list

> apt-get -y update; apt-get -y upgrade; apt-get -y dist-upgrade

> apt-get -y install fastnetmon

> but for now drivers ixgbe and igb must be compiled from source each time the kernel is upgraded

> Currently lack hardware to test:

Comparison of performance

and options

Debian 9 Native Intel Pavel Odintsov’s patched

igb ? ? ?

ixgbe ? ? ?

29 May 2017 27

> Connections> FastNetMon mirror port,

> FastNetMon alerts to database,

> FastNetMon default gateway may be behind NAT / cellular uplink may have

> Download limitations

> Keep influxdb data with FastNetMon

> Web-UI to influxdb

> LAN VPN required

29 May 2017 28

> Automatic mitigation walk through> FastNetMon connected to mirror port detects abnormality

> FastNetMon runs alert script

> Alert script reads TCP dump from STDIN, creates an rulefile and uploads it to the database host with sftp through OpenVPN tunnel

> Daemon on DDPS host reads rulefile and reduces it to one rule which is inserted in the database

> Daemon on DDPS reads database and activate new rules and deactivate expired rules

> ExaBGP announces rule changes to infrastructure

29 May 2017 29

> Automatic mitigation walk through - drawing

29 May 2017 30

> Implementation and security> Limit attack surface

> Do not expose Web-UI unrestricted to the public

> Place behind VPN to limit attach surface

> Limit access to FastNetMon: no console / customer access

> All (relevant) parameters available to customer through Web-UI

29 May 2017 31

> Security and Data validation> Hierarchy of customers, their administrator right and networks

> FastNetMon:

> Only upload valid data (rule files based on TCP dump output)

> Web-UI: select from predefined choices

> validate data in Web-UI

> CLI:

> Require SSH access, implements full flowspec, minimal sanity check

> DDPS daemon

> Validate data before inserting in database

> Validate data from database before sending to ExaBGP

> Last check: destination addresses must be ours

29 May 2017 32

> Rule file format> Originally based on tcpdump output from

FastNetMon

> Upload with sftp

> Read literately or optimize before insert in database

> format:

> Header:

> Lines ..

> last-line

29 May 2017 33

> Rule optimization> Optimize output from FastNetMon (TCP dump): reduce to one rule

> One stream one rule file

> Find closest match for all non-null fields:

> Don’t match on stateless spoofed sources

> Do match on state full source (e.g established tcp connections)

> Source and destination ports or port ranges

> Package lengths: same or in a range

> Match on fragment?

> Match on TCP flags

> Etc

29 May 2017 34

> Rule optimization cont.Attack type Mitigation Match on

syn_flood rate-limit tcp option (syn) protocol, destination port, tcp flags, size, (ttl would be nice but

is still in draft), size, and source any

udp_flood rate-limit protocol and destination, size, host and port

icmp flood discard protocol and destination

ip_fragmentation_flood rate-limit protocol size, and destination

DNS amplification rate-limit protocol, size, port and destination

NTP amplification rate-limit protocol, size, port and destination

SSDP amplification discard protocol, size, port 1900, source any

SNMP amplification discard protocol, size, port, destination

29 May 2017 35

> How to handle failure and errors> Phone

> Panic button: restart 2 x ExaBGP

> Add more specific accept rule (cli only)

> Expire rule(s) now (cli only)

29 May 2017 36

> ddpsrules (cli)/opt/db2dps/bin/ddpsrules [-v] add [-h] ... | del ... | active | log

active:Print active rules with rule id's from database

del:Set expire time to now for rule matching (list of) rule id(s)

add:--blocktime|b minutes--dst|D destination: one cidr only (database type limitation)--src|S source: one cidr only (database type limitation)--protocol|P protocol:--dport|d destination port--sport|s source port--icmp_type|t icmp type--icmp_code|c icmp code--tcpflags|T TCP tcpflags--length|l package length--dscp|C DSCP flags--frag|f fragments--action|a action: accept discard or 'rate-limit 9600'

flowspec syntax (exabgp) is accepted for all parameters but IP addressese.g.Specify http and https only-P '=80 =443'

Specify length: 3 specific all more than 300 or less than 302-l '=205 =206 =207 >=300&<=302'

Specify fragments and TCP tcpflags-f '[not-a-fragment dont-fragment is-fragment first-fragment last-fragment]'-T '[fin syn rst push ack urgent]'

29 May 2017 37

> Web-UI> All Rules Web-UI -

connections etc.

29 May 2017 38

> Web-UI — see https://github.com/deic-dk/gossamer> Dashboard with the

most relevant network information

29 May 2017 39

> Firewalls and VPN> OpenBSD cluster and OpenVPN in cluster

> OpenVPN:

> HMAC in first UDP packet

> Re-establish a client session, after a disconnection

> OpenVPN clients (FastNetMon) served DNS, info, internal authoritative domain

> OpenVPN users: no DNS, split VPN

> Unbound & nsd - recursive, and caching and authoritative DNS.

> pf: force DNS and NTP to predefined servers (localhost), without the clients knowledge

> Same time for all system components

29 May 2017 40

> ExaBGP, routers etc> Part of our network infrastructure

> Configuration is memory only: deliberately designed not to survive restart

29 May 2017 41

> Deployment and development> GitHub

> Unattended installation from ISO

> FastNetMon, 10Gb drivers etc. customer specific OpenVPN configurations

> DDPS database server, Web-UI and tools

> Tool to modify ISO images

29 May 2017 42

> Status> Rough documentation and not everything on github

> No code review yet: probably full of errors

> GUI has missing parts

> Web-UI / Web-server and database not yet cluster aware

> (Probably something else I’ve forgot)

29 May 2017 43

> FastNetMon detection limitation> Only certain volumetric attacks

> Not attacks masked as legitimate traffic

> Not attacks on the application layer (expensive API calls, Slow Loris based attacks (sending a partial request filling the web-servers maximum concurrent connection pool) - one source many requests, small amount of data

29 May 2017 44

> Who made DDPS> Anders Mundt Due

> Ashokaditya Mohanty

> Kasper Sort

> Nicolai Ernst

> Niels Thomas Haugård

> Tangui Coulouarn

> Who made DDPS possible

> Pavel Odintsov - FastNetMon - and a long list of the people who made postgres, languages, drivers and operating system

S29 May 2017 45

> Questions