Common problems
• Prevent DoS• Prevent brute-force attacks• Throttle• Kick• Ban• Limit *anything*
Rely on memcached
• Because it is easy• Lightweight, very fast• Use LRU and TTL for auto cleanup
Dead-simple principle
• Store TTL epochs only (time the event happened => counter was incremented)
• Number of timestamps *is* the counter• memcached key name indicates what is
counted
No, really,it *is* simple!
• Just do a grep to eliminate expired timestamps
• Is the remaining number of timestamps above threshold? => return 0 / 1
• Use built-in LRU for ‘block’ records
Throttle to 100 requests per
minute for a single
IP + User-Agent combination
Use case #1
my ( $status, $messages ) = BorderPatrol->authorize( memcached_client => Cache::Memcached::Fast->new(...), all => { ip_ua => { max => 100, ttl => 60, message => 'This IP+UA combination ran more ' .'than 100 queries in the last minute', value => "${client_ip}_${user_agent}", }, }, identifier => 'robot_connect',);
CODE
robot_connect#ip_ua#92.243.24.26_Googlebot => [ 1275731713, 1275731713, 1275731713, 1275731713, 1275731713, 1275731713, 1275731713, 1275731713, 1275731713, 1275731713, 1275731713, 1275731713, 1275731713, 1275731713, 1275731713, 1275731713, 1275731713, 1275731713, 1275731713, 1275731713, 1275731713, 1275731713, 1275731713, 1275731713, 1275731713, 1275731713, 1275731713, 1275731713, 1275731713, 1275731714, 1275731714, ... ]
RECORD
Prevent access to an authentication form for 5 minutes if:• more than 5 login attempts for a single username
• OR more than 50 login attempts from a single IP
Use case #2
use BorderPatrol;my ( $status, $messages ) = BorderPatrol->authorize( memcached_client => Cache::Memcached::Fast->new(...), either => { ip => { max => 50, ttl => 300, message => 'More than 50 login tries from this IP ' .'in the last 5 minutes', value => $client_ip, }, login => { max => 5, ttl => 60, message => 'More than 5 login tries for this ' .'username in the last minute', value => $username, }, }, lockout => 600, identifier => 'user_login_tries',);
user_login_tries#ip#92.243.24.26 => [ 1275731713, 1275731713, 1275731713, 1275731714, 1275731714, ]
user_login_tries#username#dmorel => [ 1275731713, 1275731713, 1275731714, ]
2 RECORDS
IMPLEMENTATIONDETAILS
$frozen_time = time;
# $record is an arrayref of timestamps# - contains one timestamp per previous access# - timestamps can be duplicated many times # if many accesses in the same second... # it doesn't matter!
if ( ref $record eq 'ARRAY' ) {
# purge the expired timestamps # the magic happens *HERE*
@$record = grep { $_ > $frozen_time } @$record;
# Since we are about to add a record:# - if we already have the max number of allowed records, # set to blocked# - if no lockout time specified, use a bucket, # so return false, but do not touch the record
if ( @$record >= $condition->{max} ) { if ($lockout) { $memcached_client->set(
$memcached_key, 'block', $lockout ); } push @$messages_notok, $condition->{message};}
# if under threshold, push the current timestamp + TTL to# the record, and set the record expiration in memcached # at TTL seconds from now
else { push @$record, $frozen_time + $condition->{ttl}; $memcached_client->set( $memcached_key, $record, $condition->{ttl} ); $conditions_ok++;}
# Do we have a 'block' value in the record?# if so, simple return the message.# The 'block' record will be automatically # removed from storage at the object's # eviction time, so just let memcached do # its job.
elsif ( $record eq 'block' ) { push @$messages_notok, $condition->{message};}
# Check the conditions stack and determine # the final answer
# If logic was 'either', one 'notok' (or more) # should block. If logic was 'all', we should # have no 'ok' at all, or else we allow.
if ( $condition_type eq 'either' ) { return ( @$messages_notok > 0 ) ? ( AC_BLOCKED, $messages_notok ) : ( AC_AUTHORIZED, undef );}else { # condition is 'all' return ( $conditions_ok == 0 ) ? ( AC_BLOCKED, $messages_notok ) : ( AC_AUTHORIZED, undef );}
MANY OTHER USE CASES
The same sliding time window approach can be used in many ways.
It really is just a generic
event hit rate counter
ON YOUR FAVORITEMIRROR SOON
(hopefully)
© Booking.com 2010
Thanks for listening!
Special thanks to Ruud, Kris, Philippe, Dennis and Liz
