dns rate limiting a hard lesson - apnic · dns rate limiting a hard lesson apops / singapore...
TRANSCRIPT
![Page 1: DNS Rate Limiting a Hard Lesson - APNIC · DNS Rate Limiting a Hard Lesson APOPS / Singapore 2013.02.26 Randy Bush 2013.02.16 DNS Rate Limit 1](https://reader033.vdocuments.site/reader033/viewer/2022060821/609aaeda9b70a85aa0119812/html5/thumbnails/1.jpg)
DNS Rate Limiting a Hard Lesson
APOPS / Singapore 2013.02.26
Randy Bush <[email protected]>
2013.02.16 DNS Rate Limit 1
![Page 2: DNS Rate Limiting a Hard Lesson - APNIC · DNS Rate Limiting a Hard Lesson APOPS / Singapore 2013.02.26 Randy Bush 2013.02.16 DNS Rate Limit 1](https://reader033.vdocuments.site/reader033/viewer/2022060821/609aaeda9b70a85aa0119812/html5/thumbnails/2.jpg)
First Symptoms • I was in a boring meeting and dealing
with email • Service to my email server was suddenly
unusable • The PoP in trouble also contained my
MRTG and other measurement <blush> • But I could log into the ‘outside’ IP
address of one of the border routers 2013.02.16 DNS Rate Limit 2
![Page 3: DNS Rate Limiting a Hard Lesson - APNIC · DNS Rate Limiting a Hard Lesson APOPS / Singapore 2013.02.26 Randy Bush 2013.02.16 DNS Rate Limit 1](https://reader033.vdocuments.site/reader033/viewer/2022060821/609aaeda9b70a85aa0119812/html5/thumbnails/3.jpg)
I am the Attacker? 5 minute input rate 720000 bits/sec, 210
packets/sec 5 minute output rate 740230000 bits/sec, 72520 packets/sec
2013.02.16 DNS Rate Limit 3
![Page 4: DNS Rate Limiting a Hard Lesson - APNIC · DNS Rate Limiting a Hard Lesson APOPS / Singapore 2013.02.26 Randy Bush 2013.02.16 DNS Rate Limit 1](https://reader033.vdocuments.site/reader033/viewer/2022060821/609aaeda9b70a85aa0119812/html5/thumbnails/4.jpg)
But it was Very Hard to reach MRTG and Other Tools
2013.02.16 DNS Rate Limit 4
![Page 5: DNS Rate Limiting a Hard Lesson - APNIC · DNS Rate Limiting a Hard Lesson APOPS / Singapore 2013.02.26 Randy Bush 2013.02.16 DNS Rate Limit 1](https://reader033.vdocuments.site/reader033/viewer/2022060821/609aaeda9b70a85aa0119812/html5/thumbnails/5.jpg)
MRTG for Router
2013.02.16 DNS Rate Limit 5
and a DNS Server
![Page 6: DNS Rate Limiting a Hard Lesson - APNIC · DNS Rate Limiting a Hard Lesson APOPS / Singapore 2013.02.26 Randy Bush 2013.02.16 DNS Rate Limit 1](https://reader033.vdocuments.site/reader033/viewer/2022060821/609aaeda9b70a85aa0119812/html5/thumbnails/6.jpg)
Really My Server? • Managed to get to APC Power Bar which
supplied server
• Shut the Server Down
• Problem Went Away!!!
• Powered Server Back Up
• OK for a Minute, but Then Back to Bad 2013.02.16 DNS Rate Limit 6
![Page 7: DNS Rate Limiting a Hard Lesson - APNIC · DNS Rate Limiting a Hard Lesson APOPS / Singapore 2013.02.26 Randy Bush 2013.02.16 DNS Rate Limit 1](https://reader033.vdocuments.site/reader033/viewer/2022060821/609aaeda9b70a85aa0119812/html5/thumbnails/7.jpg)
SSH To Server -
Took Three Tries
Over 15 Minutes
2013.02.16 DNS Rate Limit 7
![Page 8: DNS Rate Limiting a Hard Lesson - APNIC · DNS Rate Limiting a Hard Lesson APOPS / Singapore 2013.02.26 Randy Bush 2013.02.16 DNS Rate Limit 1](https://reader033.vdocuments.site/reader033/viewer/2022060821/609aaeda9b70a85aa0119812/html5/thumbnails/8.jpg)
tcpdump
2013.02.16 DNS Rate Limit 8
06:28:26.448024 IP rip.psg.com.domain > 108.178.55.192.9463: 54533*- 19/0/14 SOA, RRSIG, RRSIG, Type51, RRSIG, RRSIG, RRSIG, RRSIG, RRSIG, DNSKEY[|domain]!06:28:26.448026 IP rip.psg.com > 108.178.55.192: udp!06:28:26.448071 IP rip.psg.com.domain > 108.178.55.192.9463: 54533*- 19/0/14 SOA, RRSIG, RRSIG, Type51, RRSIG, RRSIG, RRSIG, RRSIG, RRSIG, DNSKEY[|domain]!06:28:26.448072 IP rip.psg.com > 108.178.55.192: udp!06:28:26.448168 IP rip.psg.com.domain > 108.178.55.192.9463: 54533*- 19/0/14 SOA, RRSIG, RRSIG, Type51, RRSIG, RRSIG, RRSIG, RRSIG, RRSIG, DNSKEY[|domain]!06:28:26.448171 IP rip.psg.com > 108.178.55.192: udp!06:28:26.448174 IP rip.psg.com.domain > 108.178.55.192.9463: 54533*- 19/0/14 SOA, RRSIG, RRSIG, Type51, RRSIG, RRSIG, RRSIG, RRSIG, RRSIG, DNSKEY[|domain]!06:28:26.448176 IP rip.psg.com > 108.178.55.192: udp!06:28:26.448234 IP rip.psg.com.domain > 108.178.55.192.9463: 54533*- 19/0/14 SOA, RRSIG, RRSIG, Type51, RRSIG, RRSIG, RRSIG, RRSIG, RRSIG, DNSKEY[|domain]!06:28:26.448237 IP rip.psg.com > 108.178.55.192: udp!06:28:26.448247 IP rip.psg.com.domain > 108.178.55.192.9463: 54533*- 19/0/14 SOA, RRSIG, RRSIG, Type51, RRSIG, RRSIG, RRSIG, RRSIG, RRSIG, DNSKEY[|domain]!
So It Was a DNS Reflector Attack!
![Page 9: DNS Rate Limiting a Hard Lesson - APNIC · DNS Rate Limiting a Hard Lesson APOPS / Singapore 2013.02.26 Randy Bush 2013.02.16 DNS Rate Limit 1](https://reader033.vdocuments.site/reader033/viewer/2022060821/609aaeda9b70a85aa0119812/html5/thumbnails/9.jpg)
But the Server Was NOT a Recursive Resolver
2013.02.16 DNS Rate Limit 9
![Page 10: DNS Rate Limiting a Hard Lesson - APNIC · DNS Rate Limiting a Hard Lesson APOPS / Singapore 2013.02.26 Randy Bush 2013.02.16 DNS Rate Limit 1](https://reader033.vdocuments.site/reader033/viewer/2022060821/609aaeda9b70a85aa0119812/html5/thumbnails/10.jpg)
Turned off DNS • Used /etc/ipfw.conf, IP Firwall to add deny udp from any to any 53!• I Could Now Breathe and Think • But the Server was Critical to DNS,
serving 20 ccTLDs • A Quick Mailing List Question Showed
that this was a DNSsec-based Query Reflector Attack
2013.02.16 DNS Rate Limit 10
![Page 11: DNS Rate Limiting a Hard Lesson - APNIC · DNS Rate Limiting a Hard Lesson APOPS / Singapore 2013.02.26 Randy Bush 2013.02.16 DNS Rate Limit 1](https://reader033.vdocuments.site/reader033/viewer/2022060821/609aaeda9b70a85aa0119812/html5/thumbnails/11.jpg)
With a Highly Signed CH ccTLD
One Byte of Query Produced > 1KB
of DNSsec Response 2013.02.16 DNS Rate Limit 11
![Page 12: DNS Rate Limiting a Hard Lesson - APNIC · DNS Rate Limiting a Hard Lesson APOPS / Singapore 2013.02.26 Randy Bush 2013.02.16 DNS Rate Limit 1](https://reader033.vdocuments.site/reader033/viewer/2022060821/609aaeda9b70a85aa0119812/html5/thumbnails/12.jpg)
Attacker Used Spoofed Source Address, the
Address of the Victim, for UDP Query
2013.02.16 DNS Rate Limit 12
![Page 13: DNS Rate Limiting a Hard Lesson - APNIC · DNS Rate Limiting a Hard Lesson APOPS / Singapore 2013.02.26 Randy Bush 2013.02.16 DNS Rate Limit 1](https://reader033.vdocuments.site/reader033/viewer/2022060821/609aaeda9b70a85aa0119812/html5/thumbnails/13.jpg)
The Solution Would Be Rate-Limiting
Throttle Queries From
a Single Source 2013.02.16 DNS Rate Limit 13
![Page 14: DNS Rate Limiting a Hard Lesson - APNIC · DNS Rate Limiting a Hard Lesson APOPS / Singapore 2013.02.26 Randy Bush 2013.02.16 DNS Rate Limit 1](https://reader033.vdocuments.site/reader033/viewer/2022060821/609aaeda9b70a85aa0119812/html5/thumbnails/14.jpg)
Upgraded BIND to 9.9.2
with Patch rl005.12-P1
2013.02.16 DNS Rate Limit 14
![Page 15: DNS Rate Limiting a Hard Lesson - APNIC · DNS Rate Limiting a Hard Lesson APOPS / Singapore 2013.02.26 Randy Bush 2013.02.16 DNS Rate Limit 1](https://reader033.vdocuments.site/reader033/viewer/2022060821/609aaeda9b70a85aa0119812/html5/thumbnails/15.jpg)
Options { rate-limit { responses-per-second 5; window 5; }; };!
2013.02.16 DNS Rate Limit 15
![Page 16: DNS Rate Limiting a Hard Lesson - APNIC · DNS Rate Limiting a Hard Lesson APOPS / Singapore 2013.02.26 Randy Bush 2013.02.16 DNS Rate Limit 1](https://reader033.vdocuments.site/reader033/viewer/2022060821/609aaeda9b70a85aa0119812/html5/thumbnails/16.jpg)
2013.02.16 DNS Rate Limit 16
![Page 17: DNS Rate Limiting a Hard Lesson - APNIC · DNS Rate Limiting a Hard Lesson APOPS / Singapore 2013.02.26 Randy Bush 2013.02.16 DNS Rate Limit 1](https://reader033.vdocuments.site/reader033/viewer/2022060821/609aaeda9b70a85aa0119812/html5/thumbnails/17.jpg)
The Problem Was Solved!
2013.02.16 DNS Rate Limit 17
![Page 18: DNS Rate Limiting a Hard Lesson - APNIC · DNS Rate Limiting a Hard Lesson APOPS / Singapore 2013.02.26 Randy Bush 2013.02.16 DNS Rate Limit 1](https://reader033.vdocuments.site/reader033/viewer/2022060821/609aaeda9b70a85aa0119812/html5/thumbnails/18.jpg)
From: CH ccTLD Admin As you have seen today the CH-zone got hit with a DNS ANY query storm. I assume the traffic has been sent to most CH secondary name-servers. We saw the following kind of query towards our name-servers which resulted in an amplification factor of 75:
dig +edns=0 +bufsize=9000 CH. ANY!
2013.02.16 DNS Rate Limit 18
![Page 19: DNS Rate Limiting a Hard Lesson - APNIC · DNS Rate Limiting a Hard Lesson APOPS / Singapore 2013.02.26 Randy Bush 2013.02.16 DNS Rate Limit 1](https://reader033.vdocuments.site/reader033/viewer/2022060821/609aaeda9b70a85aa0119812/html5/thumbnails/19.jpg)
Lessons • OOB Access Really Needed to Be Out
Of Band <blush> • Set Up a Second Measurement System
to Measure the First?
• Install and Configure DNS Flow-Limiting Before This Happens to You!!
2013.02.16 DNS Rate Limit 19
![Page 20: DNS Rate Limiting a Hard Lesson - APNIC · DNS Rate Limiting a Hard Lesson APOPS / Singapore 2013.02.26 Randy Bush 2013.02.16 DNS Rate Limit 1](https://reader033.vdocuments.site/reader033/viewer/2022060821/609aaeda9b70a85aa0119812/html5/thumbnails/20.jpg)
Unbound Measurement of Plasma Unbound
Unconjugated Bilirubin
Monitoring changes in bilirubin concentration using diazo derivatives, and correcting for rate-limiting dissociation of bilirubin from albumin.
Google does not always work 2013.02.16 DNS Rate Limit 20
![Page 21: DNS Rate Limiting a Hard Lesson - APNIC · DNS Rate Limiting a Hard Lesson APOPS / Singapore 2013.02.26 Randy Bush 2013.02.16 DNS Rate Limit 1](https://reader033.vdocuments.site/reader033/viewer/2022060821/609aaeda9b70a85aa0119812/html5/thumbnails/21.jpg)
NSD
Use the configure script option ./configure –enable-ratelimit! The default parameters are a good start
2013.02.16 DNS Rate Limit 21