lessons from surviving a 300gbps ddos...
TRANSCRIPT
Matthew Prince Co-founder & CEO, CloudFlare
@eastdakota www.cloudflare.com
Lessons from Surviving a 300Gbps DDoS Attack
March 18 – March 25 Series of very high volume DDoS attacks targeting one of CloudFlare’s customers
The Story 1. The nature of the attack 2. What we did to stop it 3. Practical steps to protect your own
networks
1. The Attack
Monday, 18 March 2013
March 18 – 21 “Annoyance” attacks (10 – 80Gbps range)
Wednesday, March 20 ~75Gbps attack
100Gbps Magic ceiling in DDoS attacks
How?
What you don’t need… 1. Botnets 2. A lot of people 3. Significant technical skill
What you need… 1. A list of open DNS resolvers 2. Some servers running on networks
that allow source IP spoofing
1. Open DNS resolvers
Misconfigured DNS servers running without limits on
what they respond to
Amplification
64-byte query
!; <<>> DiG 9.8.3-P1 <<>> ANY isc.org @63.217.84.76 +edns=0 +notcp +bufsize=4096 ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 20458 ;; flags: qr rd ra; QUERY: 1, ANSWER: 26, AUTHORITY: 4, ADDITIONAL: 12 !;; OPT PSEUDOSECTION: ; EDNS: version: 0, flags:; udp: 4096 ;; QUESTION SECTION: ;isc.org. IN ANY !;; ANSWER SECTION: isc.org. 7147 IN SOA ns-int.isc.org. hostmaster.isc.org. 2013073000 7200 3600 24796800 3600 isc.org. 7147 IN NS ns.isc.afilias-nst.info. isc.org. 7147 IN NS ord.sns-pb.isc.org. isc.org. 7147 IN NS ams.sns-pb.isc.org. isc.org. 7147 IN NS sfba.sns-pb.isc.org. isc.org. 7 IN A 149.20.64.69 isc.org. 7147 IN MX 10 mx.pao1.isc.org. isc.org. 7147 IN TXT "v=spf1 a mx ip4:204.152.184.0/21 ip4:149.20.0.0/16 ip6:2001:04F8::0/32 ip6:2001:500:60::65/128 ~all" isc.org. 7147 IN TXT "$Id: isc.org,v 1.1835 2013-07-24 00:15:22 dmahoney Exp $" isc.org. 7 IN AAAA 2001:4f8:0:2::69 isc.org. 7147 IN NAPTR 20 0 "S" "SIP+D2U" "" _sip._udp.isc.org. isc.org. 3547 IN NSEC _adsp._domainkey.isc.org. A NS SOA MX TXT AAAA NAPTR RRSIG NSEC DNSKEY SPF isc.org. 7147 IN DNSKEY 256 3 5 BQEAAAABwuHz9Cem0BJ0JQTO7C/a3McR6hMaufljs1dfG/inaJpYv7vH XTrAOm/MeKp+/x6eT4QLru0KoZkvZJnqTI8JyaFTw2OM/ItBfh/hL2lm Cft2O7n3MfeqYtvjPnY7dWghYW4sVfH7VVEGm958o9nfi79532Qeklxh x8pXWdeAaRU= isc.org. 7147 IN DNSKEY 257 3 5 BEAAAAOhHQDBrhQbtphgq2wQUpEQ5t4DtUHxoMVFu2hWLDMvoOMRXjGr hhCeFvAZih7yJHf8ZGfW6hd38hXG/xylYCO6Krpbdojwx8YMXLA5/kA+ u50WIL8ZR1R6KTbsYVMf/Qx5RiNbPClw+vT+U8eXEJmO20jIS1ULgqy3 47cBB1zMnnz/4LJpA0da9CbKj3A254T515sNIMcwsB8/2+2E63/zZrQz Bkj0BrN/9Bexjpiks3jRhZatEsXn3dTy47R09Uix5WcJt+xzqZ7+ysyL KOOedS39Z7SDmsn2eA0FKtQpwA6LXeG2w+jxmw3oA8lVUgEf/rzeC/bB yBNsO70aEFTd isc.org. 7147 IN SPF "v=spf1 a mx ip4:204.152.184.0/21 ip4:149.20.0.0/16 ip6:2001:04F8::0/32 ip6:2001:500:60::65/128 ~all" isc.org. 7147 IN RRSIG SPF 5 2 7200 20130828233259 20130729233259 50012 isc.org. XDoOYzkTHEV1W1V4TT50SsqXn4cxNhPvEuz3iFjq/oskLY9UOaK4GYDO GqHAjwNT0B6pUakKTQ3GvBjUBufPcEauCOl7L7kb8/cC6zYifUCoW0pS moiQxmyqfrPDTzyVA894myUONGgMmB6QW68HGPVvc6HzGWx9bOmjvFyX uOs= isc.org. 7147 IN RRSIG DNSKEY 5 2 7200 20130828230128 20130729230128 12892 isc.org. COfF8fU6a8TBUG97SI/X+u2eKv7/mw
3,363-byte response
50x Amplification factor
2. A network that allows source IP spoofing
Good networks don’t let packets originate from IPs they don’t own (BCP38)
Not all networks are good
UDP = no handshake
Spoofed source: 190.93.243.93 !
dig ANY isc.org @63.217.84.76 +edns=0 +notcp +bufsize=4096 !
Response sent to: 190.93.243.93
Stunningly simple
Don’t need to be a genius…
How common are these ingredients?
28 million open resolvers
24.6% networks allow spoofing
Ingredients for the Spamhaus attack?
Spamhaus Ingredients 309Gbps for 28 minutes 30,956 open resolver IPs 3 networks that allowed spoofing
1 attacker’s laptop controlling 5–7 compromised servers on 3 networks that allowed spoofing of 9Gbps DNS requests to 0.1% of open resolvers resulted in 300Gbps+ of DDoS attack traffic.
+ + + +
1 attacker’s laptop controlling 10-12 compromised servers on 5 networks that allowed spoofing of 18Gbps DNS requests to 0.2% of open resolvers resulted in 600Gbps+ of DDoS attack traffic.
+ + + +
1 attacker’s laptop controlling 50-70 compromised servers on 20 networks that allowed spoofing of 87Gbps DNS requests to 1% of open resolvers resulted in 3Tbps+ of DDoS attack traffic.
+ + + +
1 attacker’s laptop controlling 200-400 compromised servers on 100 networks that allowed spoofing of 280Gbps DNS requests to 8% of open resolvers resulted in 12Tbps+ of DDoS attack traffic.
+ + + +
24Tbps Total NAM Internet Backbone Traffic in 2012
(Estimated by Minnesota Internet Traffic Studies)
…but that’s yesterday’s news.
Monday, February 10
400Gbps
That’s not the real story…
NTP
MONLIST (?!*#@!!)
200x Amplification factor for NTP
1 attacker’s laptop controlling 1 compromised servers on 1 networks that allowed spoofing of 2Gbps NTP MONLIST requests to 4,532 vulnerable NTP servers 400Gbps+ of DDoS attack traffic.
+ + + +
What’s coming next…?
SNMP
650x Amplification factor for SNMP
And no reason an attacker couldn’t combine
DNS, NTP, SNMP + more
2. What we did to stop it
2 million customers
Seattle
LondonAmsterdam
ParisFrankfurt
StockholmWarsawPragueVienna
TokyoSeoul
Hong KongBangkok1H 2014
Manila1H 2014
New Dehli1H 2014
Mumbai1H 2014
Chennai1H 2014
SingaporeJakarta
1H 2014
Sydney
Miami
San Jose
Los Angeles
NewarkWashington DCAtlanta
TorontoChicago
Dallas
Santiago4Q 2013
Riyadh1H 2014
Sao Paulo 4Q 2013
Milan1H 2014
Cairo1H 2014
Madrid1H 2014
Mombasa1H 2014
Johannesburg1H 2014
Anycast
Inherently “dilutes” the attack
300Gbps 23 Anycasted PoPs 13Gbps/PoP
÷
Attacker could do the math
Started attacking our upstreams
London
traceroute to www.spamhaus.org (141.101.123.93), 30 hops max, 60 byte packets ! 1 192.168.0.1 (192.168.0.1) 0.573 ms 2 XXXXX.skybroadband.com (151.231.X.X) 16.367 ms 3 02780XXX.bb.sky.com (2.120.9.136) 19.768 ms 4 te0-7-0-5.er11.thlon.ov.easynet.net (89.200.134.59) 22.188 ms 5 195.66.225.179 (195.66.225.179) 16.902 ms 6 141.101.123.93 (141.101.123.93) 17.323 ms
LINX
Caused temporary regional disruptions
Also went after our upstream transit providers
Worked with IXs and providers
“Next Hop Self ” internal routing
Edge filtering of IPs/protocols with an understanding of
our application
Flowspec is our friend
Special shout-out to nLayer (now GTT)
3. How to protect your own network
Four suggestions
First, make sure you’re not part of the problem…
Are you running open resolvers?
Implement BCP38 (uRPF)
Second, practice good protocol hygene…
Separate protocols onto distinct IPs and filter
E.g., UDP packets should never transit the IP of your HTTP server
Third, implement infrastructure ACLs…
At the edge, don’t allow packets destined for your infrastructure.
“Next Hop Self ”
Fourth, know your upstreams…
Work with them closely to extend the edge of your network
