DDoS mitigation in the real world

19th May 2014Vienna DevOps/Security Meetup

Michael Renner@terrorobe

My name is Michael Renner

In IT since 2001, focus on system administrations and web operations.

Freelancing since 2010 for multiple companies doing a hodge-podge of tasks - from building data centers, DBA work, troubleshooting, performance optimization...

If you've got questions, just ask!

Company I'm freelancing for.

Viennese company, founded around 2010 - our offices are on Mariahilfer Strasse

Operations, Development, Customer Care, payment solutions, SMS delivery, you name it ;)

And since a year ago also DDoS protection

DDoS?

In computing, a denial-of-service (DoS) or distributed denial-of-service (DDoS) attack is an attempt to make a machine or network resource unavailable to its intended users.

Source: http://en.wikipedia.org/wiki/Denial-of-service_attack

Why people DDoS

...which leads us to the topic of our today's talk.

DDoS is something that has gotten more and more news coverage in the last few years.Chances are high that you already tried using a service while it was being DDoSed or that you run a service on your own which was the target of an attack

So why do people DDoS?

...to extort money

This one is easy.

Was very prevalent in the gambling and sports betting area. Back at Inode in 2003 people where already DDoSing gambling companies!

"Comes with the business"

...to underline a political agenda

Operation Payback in 2010 - done by Anonymous - targeted against recording industry prosecuting piracy as well as finance companies denying service to wikileaks

...to escalate issues in customer care

Has happened in the past - if you annoy the wrong customer he might escalate things by his own means

...because somebody pissed them off on IRC

User chats from your server/network, annoys the wrong people and they show them that they mean business

...to prove a point

...or just to prove a point, whatever that might be.

The more high profile/recognized your site is the higher the chance that somebody will attack it just for the fun of it.

DDoS attacks have asymmetrical costs

The problem with DDoS attacks is that they have highly asymmetrical costs.

The attack itself can be quite cheap whereas the lost revenue as well as time and money to bandaid-patch an attack can be quite expensive

Attacks are cheap

"Prices per hour may depend on your target. Price starts 10$ per hour to 100$ per hour depending on how Huge or Protected the target is."

When we're talking money one must know that attacks are quite cheap.

This is an example from end-customer site, completely with youtube video and everything!

Depending on how smart the attackers are and how good you're connected attacks can be significantly cheaper.

Attacks happen when people have free time

• Weekends, holidays, bridging days

• In their respective timezone!

• More often than not:

• the time when you'd like to have free time as well

Another problem is that the attacks will happen when people have free time.

Attacks are hard to identify

• "The DB server has too many connections open"

• "I'm seeing packet loss and a large amount of softinterrupts on CPU0"

• "The server is down"

DDoS attacks are very disorienting

Depending on how much instrumentation you have for your infrastructure you might not even be able to tell that you're getting DDoSed

"Newrelic might not be enough to spot it"

Depending on attack you might need OS, application level even network packet statistics.

Mitigating is hard without proper infrastructure

• "Let's see if I can whip up an u32 classifier to filter out this pattern I'm seeing"

And when the attack is already running you'll often have a hard time jury-rigging yourself out of the attack.

Hard to find the sources of attacks

• Most attacks use spoofed source addresses

• If it's a reflection or amplification attack you won't even get those.

• And even if you'v got real sources, it's a hacked server in nearly all of the cases

More often than not management will be interested on who's doing the attack and why.

This is nigh impossible if you're not the NSA, having a big picture of all the traffic going on.

And even if you've got "real" source addresses nearly all of them will be hacked servers connected to an anonymous C&C servers

In the end...

• ...it's you and your customers who're unhappy

so it's best to try not to get knocked off the net in the first place!

DDoS war stories

To give the users who haven't experienced a thorough attack yet a glimpse of what it may look like I've collected a few things we faced in the past.

So you run a website...

Most of you operate a web/online service I guess, which generates you fame, fortune or both.

And along the way you picked up a few people who don't have the best interests regarding your site, and so they decide to...

Somebody does a slow-read attack

...open up hundreds of connections, read a few bytes at a time...

basically exhausting open workers, tcp connections, file descriptors, memory, whatever comes first.

so you get a proxy setup like CloudFlare up and running, all HTTP requests proxied through

They detect the botnet IPs either through pattern matching or existing botnet IP lists and start filtering those out or presenting them with captchas. Attack solved.

But they still know your server's IP address

Then the attacker goes on netcraft.com, checks the former IP address of your site ...and decides to send a few hundred megabits worth of UDP packets your way.

You ask your ISP for a new subnet, renumber your servers, add the new IP addresses to your proxy setup and hope for the best

Things are quiet for now.

Suddenly you get a ERR_NAME_NOT_RESOLVED

when visiting your site

After a few hours the monitoring starts to page you. You check your site and you get a NAME NOT RESOLVED.

...turns out the cheap registrar where you bought your domain

...who still does your DNS

...isn't quite DDoS resilient.

So you shop around, see companies like dyn.com or Amazon route 53, move domains over, wait for cached entries to expire and after a few hours you should be back online!

You notice that you're sending mails

[..]Received: from web01.yourcoolapp.com ([198.51.100.42])Subject: WELCOME TO YOURCOOLAPP!From: noreply@yourcoolapp.comTo: evilattacker@example.com

But then you notice that you're sending emails! And your attacker noticed that the email address contain interesting information in their headers.

...and the DDoS attack on your server resumes.

...so you configure your mail gateway to not include servers IP addresses

...get new IP addresses

...hope for the best

...so you think you're good for now.

And then your ISP goes offline

By then you think you've seen it all and you think you're good to go.

...most smaller ISPs are not equipped to sustain attacks even 10 to 50 gigabits in size.

If regular traffic is in the single-gigabit per second range it's easy to knock them offline. Usually they then just filter the destination IPs at their carriers and be done with it.

Mitigating attacks...in Theory!

...so, this should give you a nice overview what can happen in the real world.

So what can we do about these attacks?

In the end all of the DDoS Mitigation services out there are doing the same thing.

Increase your attack surface

• Geo-distribution is the way to go

• either via DNS or IP anycast

• -> more traffic sinks!

First part is increasing the attack surface

Most customers have just a single datacenter, amazon availability zone and so on. Very easy to concentrate your attack on.

If you operate multiple Datacenters it's much harder to knock all of them offline - loadbalancing done either via DNS & geoip or IP anycast setups - same prefix announced on multiple locations

This all boils down to have more traffic sinks to filter the traffic out on

150 Gbit/sec sounds scary

• Less scary when it's:

• 40 Gbit Hongkong

• 20 Gbit Frankfurt

• 35 Gbit London

• 25 Gbit US West

• 30 Gbit US East

I'm just pulling a few numbers out of my hat to give you an example. Suppose you've got a 150gbit attack running - which to this day is still a very large number when it comes to DDoS attacks - this gets far more manageable when you operate multiple sites where the traffic has to go to

And even if they manage to knock one of the sites off the net all the others are still online and answering requests.

Scrubbing the traffic

• In general: you want to filter out all "DDoSy" or anomalous requests from your traffic

• Statistical models, signature-based filters, hand-written rules, etc.

• Specialized hardware with in-ASIC rule processing

• Beyond the scope of this talk

The other big part is to scrub the traffic and remove all packets which belong to "probably unwanted" traffic.

The further down the traffic goes through your stack the more expensive is the processing of a malicious request. Compare the time needed to throw away a request on the firewall vs. throwing it away in the web application.

Big Companies use specialized hardware with network chips, being capable of doing complex rule processing at wirespeed of the interface.

They often use statistical models to detect network anomalies, have rules for known attack patterns (e.g. NTP reflection attacks) and can also deploy custom rules specific to attacks currently going on.

Don't build your own scrubbing setup unless

you've got good reasons

In most cases it's much cheaper to just lease scrubbing services from one of the established players instead of building your own infrastructure.

Mitigating attacks......in Practice!

...so much for basics - what can we actually do to improve our security?

Don't do DDoS mitigation as an afterthought

First and foremost...

it's much nicer to talk and discuss about these things during office hours with sober colleagues than on a saturday night at 4 am after 8 beers

Decide on a scope

• Threat modeling

• Build playbooks

• Prepare & implement low-hanging fruits right now

You need to do threat modeling - which attacks do you expect, how you're going to manage them. Prepare contacts up front even if you don't want to buy something right now.

Have a playbook describing how you're going to react

Implement low hanging fruits right now!

Have good OpSec

One example for a low-hanging fruit: Operational Security

Make sure you don't leak IP addresses you don't want to

It makes sense to have two different network segments - one for public-facing IPs, another one for the actual servers. Actual server IPs don't even have to be public ones if you can avoid it.

Things to check: Mail HeadersTest systems like beta.yourcoolapp.comMX records!

Choose an approach

And as with every idea - there's more than one way to do it!

Proxies

• The easy/cheap solution

• Protocol proxies (HTTP, SMTP, etc.)

• Geo-distributed

The cheap approach would be a proxy setup. You get a list of protocol-level proxies where you point your DNS records to - you configure the destination for the proxies and all the traffic hits these servers first.

Well-known company is CloudFlare, offering this service even for free

Good if you need a fast solution

Proxy Caveats• Few control over proxy servers

• Can hurt troubleshooting times

• Applications don't see end-user IPs anymore

• Fix for HTTP: X-Forward-For

• If your real IPs are leaked it's game over again

• Not feasible for all protocols

This comes not without draw-backs though.

If you need to troubleshoot things you might not have the level of control you'd like, since somebody else is already terminating your HTTP requests for you.

Your applications don't see IPs anymore

If you should leak your IPs it's easy for an attacker to just go for your real servers.

And it's not feasible for all protocols, VoIP, UDP-based gameserver protocols, etc.

Routed solution

• You need a /24 network prefix

• IP tunnel to scrubbing center

• All scrubbing locations announce the prefix

• Traffic gets sent to closest location, scrubbed and forwarded to you

Those get easier to manage if you've got a routed setup

Routed approaches are more expensive and complex to maintain

Pro: Protocol- and application agnostic

Routing caveats

• Takes time to implement & test properly

• More expensive to setup and maintain

• Routing & BGP can be scary

It's not without it's drawbacks though:

Appliances

There's still appliance vendors in the DDoS market, e.g. Arbor or Radware.

Was a bit surprised by that, but it absolutely makes sense

Appliances targeted at end users should always come with a traffic scrubbing offer, otherwise they're snakeoil - it's easy to DDoS a single device running behind even a 10Gbit link.

They can make live easier though since they automate tasks like route changes and provide additional information for the scrubbing center

Ask your ISP

something you shouldn't neglect is that your ISP may already have DDoS infrastructure set up - it doesn't hurt to ask them what it is they're running and if they can make you an offer

Common caveats

• Your applications latency is most likely going to increase

• There can be false positives & overfiltering

• Your system is becoming more complex

Something to consider with any choice you're going to make is

Latency is going to increase

false positives and overfiltering - how are you going to spot this?

your system is getting more complex

Interested?Talk to us!

info@ixolit.comm.renner@ixolit.com

I hope that gave you a bit of an overview of how the current situation looks like when dealing with DDoS

If you're interested - please talk to us

Ixolit is both a Prolexic and a Dyn Partner

Prolexic is one of the major traffic scrubbing operators in the world. Got recently acquired by Akamai.Dyn.com offers anycasted DNS setups which are DDoS resilient

Already have a few production setups running for _huge_ sites which solved all our DDoS issues to this day

Questions?

Do you have any questions?

Thanks!