redis networking nerd down: for lovers of packets and jumbo frames- john bullard, distil networks

For Lovers of Packets and Jumbo FramesRedis Networking Nerd Down

Intro

John BullardVP Engineering

Benji TaylorDevOps Lead

We Block BotsDistil

How the Distil Bot Detection Solution Works

As web traffic passes through Distil, the system

1. Fingerprints each incoming connection and compares it to our Known Violators Database

2. If it’s a new fingerprint, validates the browser to determine if it’s a Bot or Not

3. Based on your preferences, automatically tags, challenges, or blocks the bot

Bad Bots Cause Problems

Fortune 500 & Alexa Global 10,000 CustomersEcommerce

Travel

Publishers

Directories

Traditional Media

Marketplace

Services

We Block BotsDistil Architecture

Distil Architecture

Topology - Single Master

Topology - Master : Master

Topology - Master : Slave

Every Packet MattersCustomer Story

Customer Story

○Large Online MarketplaceHigh traffic volume Approaching seasonal peak - 3x trafficAWS EC2 Classic

○ Soft 100k PPS Limit○ Customer concern

Customer Story

○Scaling Up + OutMaster : Slave Topology

○ 9 Node: 1 Master, 8 SlavesC3.8xlarge Instances

○ 36 vCPU, 60 GiB, 10 GigabitProof of Concept

○ Onboard <5% of traffic

POC - Slave Measurements

POC - Master Measurements

POC - Master Measurements

Customer Story

Customer Story

Customer Story

○Bottleneck - Master PPSRedis Master : Write + Replicate

○ All nodes write○ All writes replicate to all nodes

○RequirementsMaster PPS < 100kNo partitioningEC2 Classic$0 Cost Increase

We Block BotsPackets

Packets

Packets

Packets

Packets

Overview

○Jumbo CaveatsNot supported everywhere (Virtualbox)“Jumbo” differs among vendorsEvery component on path must support itSwitches - drop jumbo framesRouters - must support ingress and egress

Overview

○ Tuning - Jumbo frames - AWS Support for jumbo frames:

○ Compute optimized: C3, C4, CC2 ○ General purpose: M3, M4, T2 - GPU: CG1, G2 ○ Memory optimized: CR1, R3 ○ Storage optimized: D2, HI1, HS1, I2

MTU 9001 bytes

Packets

Packets

Packets

Packets

Packets

○TCP Small Packet ProblemApplication sending 1 byte packetsTCP packets have a 40 byte header

○ 20 bytes for TCP○ 20 bytes for IPv4

41 byte packet ○ For 1 byte of useful information

Packets

○Nagle’s algorithm (RFC 896)Combine small outgoing messages

Reduce # of packetsImprove TCP/IP efficiency Trade-off: increased latency

○ Problematic for real-time systems○ Per-socket: TCP_NODELAY

Packets

# Disable TCP_NODELAY on the slave socket after SYNC?## If you select "yes" Redis will use a smaller number of TCP packets and# less bandwidth to send data to slaves. But this can add a delay for# the data to appear on the slave side, up to 40 milliseconds with# Linux kernels using a default configuration.## If you select "no" the delay for data to appear on the slave side will# be reduced but more bandwidth will be used for replication.## By default we optimize for low latency, but in very high traffic conditions# or when the master and slaves are many hops away, turning this to "yes" may# be a good idea.repl-disable-tcp-nodelay no

Packets

Packets

Packets

Master/Slave ChainsCommon Architecture for Read Intensive

ScenariosBalance Write Duplication to Read-SlavesShift PPS to Read-SlavesConfiguration Change - No DevNo Added Cost

○ 1 Master 2 Read-Slaves 6 Edgenodes

Customer Story

Customer Story

Packets

○ApproachEthernet tuning - Enable Jumbo PacketsTCP/IP tuning - Enable “Naggling”Topology - Read-Slave fan-out

Demo