How a major ISP built a new anti-abuse platformMike O’Reirdan

Comcast Distinguished EngineerInternet Systems EngineeringComcast National Engineering & Technical Operations

Outline

Comcast facts and figures

Why build a new platform

Fundamentals of anti spam

Size of the problem

Previous approach

Current solution

Migration methods

Current status

Why a new platform?

Moved from a hosted to an in-house platformNeed to improve customer experience by further reducing volumes of spam to the mailboxDeploy a platform which can economically and easily scaleEmerging threats in abuse landscape

• Image spam• Botnets• VoIP spam (SPIT)

Need to have a plug-and-play architecture• Firmly believe that no one vendor will be the best forever• We need a mix of vendors and approaches to hedge our bets and

reduce risk• Somebody in this room may be our next vendor when you have

gone from the lab to the VC and into beta

Size of the problem

Volumes of spam are astronomical• 596 million connection attempts (Jan25th 2008)

• 539 million connection attempts rejected• 93% spam• 76 million messages delivered

Connection attempts increases massively above this around holidays such as Thanksgiving.The problems is criminality at massive scale

Fundamentals of anti-spam

Not much differentiation between major mail box hosters and other ISPs with regard to spam percentages and volumesThree stages

• Blocking based on IP (reputation and DUL space)– 5% of CPU cycles– Removes ~70% of the spam

• Blocking based on message protocol and heuristics– 10% of CPU cycles– Removes ~15% of the spam

• Blocking based on content– 85% of CPU cycles– Remove ~10% of the spam

Idea is to use the least cycles to remove the most messages

Previous approach

100s of Linux blade servers• No site fail over

Multiple RBLs using BIND for DNS

Heuristics and protocol filtering

Spam content filtering using industry standard software

Virus filtering using industry standard software

New Approach

Fewer Linux Blade servers distributed over two sites• Full dual site redundancy with each site fully capable of

carrying 100% of trafficRBLs hosted on a specialised DNS based platform

• Trend• Spamhaus• Return Path

Protocol and heuristics filtering performed on the Bizanga IMP MTAs which run on LinuxSpam content filtering technologyAnti-virus technology

Heuristics employed

Directory Harvest attack

Dictionary attack

rDNS check

Throttling

Dynamic space blocking

Non-existent user block

Content filtering-detecting spammy content

Cloudmark• Relies on multiple sources of data

– Spam / no Spam reports from end users– Honeypots

• Initially based on Vipul’s Razor• Applies algorithmically derived signatures to

incoming email (Proprietary)• Zero hour anti virus

Trend Anti-virus• Signature analysis• Heuristics

Migration

Relatively simple process to migrate from old platformMoved traffic across by re-pointing comcast.net MX records to new platform and making lots of involved highly planned DNS configuration changesPerformed a series of increasing short duration burst test scaleThen moved 5% of the traffic. After platform rules proved stable, traffic was moved across in slightly larger increments over several days to the new platform.This method allowed us to quickly revert back (under 30 minutes) to old platform in the event of any issues without customer impact

Lessons learned

It always helps to be able to test the new platform against an existing live e-mail flow but this is difficult at our scale with a multi-Gbps mail flowFailing that, heavy reliance has to be placed on cooperation with vendors and existing platform technology usersRules used on an old platform do not always map across neatly to a new one