network visibility and control using industry standard sflow telemetry
TRANSCRIPT
Network visibility and control using industry standard sFlow telemetry
Peter Phaal InMon Corp.March, 2016
Twitter: @sFlow Blog: blog.sflow.com
San Francisco Network Visibility Meetup
Why monitor?
“If you can’t measure it, you can’t improve it”Lord Kelvin
Time
Capacity
Demand
Static provisioning
$ Unused capacity
$$$ Service failure
$$ Unused capacity
$$ Savings
Time
Capacity
Demand
Dynamic provisioning
Feedback control
Measure
Control
System
desired output
measured output
Controllability and Observability
Basic concept is simple, a stable feedback control system requires:1. ability to influence all important system states (controllable)2. ability to monitor all important system states (observable)
It’s hard to stay on the road if you can’t see the road, or keep to the speed limit without a speedometer
It’s hard to stay on the road or maintain speed if your brakes, engine or steering fail
Controllability and Observability driving example
Observability
Controllability
States location, speed, direction, ...Tule fog in California Central Valley
Effect of delay on stability
Measurement delay Planning delay
Time
Configuration delayDisturbance Response delay
EffectLoop delay
DDoS launched Identify target, attacker Black hole, mark, re-route? Switch CLI commands Route propagation Traffic dropped
Components of loop delay
e.g. Slow reaction time causes tired / drunk / distracted
driver to weave, very slow reaction time and they leave
the road
What is sFlow?
“In God we trust. All others bring data.”Dr. Edwards Deming
Industry standard measurement technology integrated in switcheshttp://sflow.org/
Open source agents for hosts, hypervisors and applications
Host sFlow project (http://sflow.net) is center of an ecosystem of related open source projects embedding sFlow in popular
operating systems and applications
Host agent extends network visibility into public / private cloud
Network (maintained by hardware in network devices)- MIB-2 ifTable: ifInOctets, ifInUcastPkts, ifInMulticastPkts, ifInBroadcastPkts, ifInDiscards, ifInErrors, ifUnkownProtos,
ifOutOctets, ifOutUcastPkts, ifOutMulticastPkts, ifOutBroadcastPkts, ifOutDiscards, ifOutErrors
Host (maintained by operating system kernel)- CPU: load_one, load_five, load_fifteen, proc_run, proc_total, cpu_num, cpu_speed, uptime, cpu_user, cpu_nice,
cpu_system, cpu_idle, cpu_wio, cpu_intr, cpu_sintr, interupts, contexts
- Memory: mem_total, mem_free, mem_shared, mem_buffers, mem_cached, swap_total, swap_free, page_in, page_out, swap_in, swap_out
- Disk IO: disk_total, disk_free, part_max_used, reads, bytes_read, read_time, writes, bytes_written, write_time
- Network IO: bytes_in, packets_in, errs_in, drops_in, bytes_out, packet_out, errs_out, drops_out
Application (maintained by application)- HTTP: method_option_count, method_get_count, method_head_count, method_post_count, method_put_count,
method_delete_count, method_trace_count, method_connect_count, method_other_count, status_1xx_count, status_2xx_count, status_3xx_count, status_4xx_count, status_5xx_count, status_other_count
- Memcache: cmd_set, cmd_touch, cmd_flush, get_hits, get_misses, delete_hits, delete_misses, incr_hits, incr_misses, decr_hists, decr_misses, cas_hits, cas_misses, cas_badval, auth_cmds, auth_errors, threads, con_yields, listen_disabled_num, curr_connections, rejected_connections, total_connections, connection_structures, evictions, reclaimed, curr_items, total_items, bytes_read, bytes_written, bytes, limit_maxbytes
Standard counters
Simple
- standard structures - densely packed blocks of counters
- extensible (tag, length, value)
- RFC 1832: XDR encoded (big endian, quad-aligned, binary) - simple to encode /decode
- unicast UDP transport
Minimal configuration
- collector address
- polling interval
Cloud friendly
- flat, two tier architecture: many embedded agents → central “smart” collector
- sFlow agents automatically start sending metrics on startup, automatically discovered
- eliminates complexity of maintaining polling daemons (and associated configurations)
Scaleable push protocol
• Counters tell you there is a problem, but not why.
• Counters summarize performance by dropping high cardinality attributes:
- IP addresses
- URLs
- Memcache keys
• Need to be able to efficiently disaggregate counter by attributes in order to understand root cause of performance problems.
• How do you get this data when there are millions of transactions per second?
Counters aren’t enough
Why the spike in traffic?(100Gbit link carrying 14,000,000 packets/second)
• Random sampling is lightweight
• Critical path roughly cost of maintaining one counter: if(--skip == 0) sample();
• Sampling is easy to distribute among modules, threads, processes without any synchronization
• Minimal resources required to capture attributes of sampled transactions
• Easily identify top keys, connections, clients, servers, URLs etc.
• Unbiased results with known accuracy
Break out traffic by client, server and port (graph based on samples from100Gbit link carrying 14,000,000 packets/second)
sFlow also exports random samples
Integrated data model
Packet Header
Source Destination
TCP/UDP Socket TCP/UDP Socket
MAC Address MAC Address
Sampled Packet Headers +
Forwarding State
I/F CountersNETWORK
HOST
CPU
Memory
I/O
Adapter MACs
APPLICATION
Sampled Transactions
Transaction Counters
TCP/UDP Socket
Independent agents sFlow analyzer joins data for integrated view
Embedded monitoring of all switches, all
Virtual Servers
ApplicationsApache/PHP
Tomcat/Java
Memcached
Virtual Network
Servers
Network
Embedded monitoring of all switches, all servers, all applications, all the time
Consistent measurements shared between multiple
management tools
Comprehensive data center wide visibility
Picking the right tools
“This is the Unix philosophy: Write programs that do one thing and do it well. Write programs to work together.”Doug McIlroy
packets
decode hash sendflow cache flushsample
Flow Records
flow cache embedded on switchswitch
NetFlowIPFIX…
decode hash sendflow cache flush
Flow Records
packets
send
polli/f counters
sample
multiple switches export sFlowpackets
send
polli/f counters
sample
...
centralized software flow cache
switch
switchJSON/RESTNetFlowIPFIX…
• Reduce ASIC cost / complexity • Fast response (data not sitting on switch) • Centralized, network-wide visibility • Increase flexibility → software defined analytics
Move flow cache from ASIC to external software
Scale-out alternative to SNMP polling
Traffic analytics with sFlow
sFlow-RT.com analytics engine
• Low latency flow analytics for real-time control applications
• Disaggregates flow cache from database. Choose external database(s) for history (InfluxDB, Logstash, etc.)
• Programmable analytics pipeline through open APIs
RESTful API for defining flows
http://blog.sflow.com/2013/08/restflow.html
curl -H "Content-Type:application/json" -X PUT —data \ '{"keys":"ipsource,ipdestination,tcpsourceport,tcpdestinationport", \"value":"bytes", "ipfixCollectors":["10.0.0.1"]}'\http://127.0.0.1:8008/flow/tcp/json
curl -H "Content-Type:application/json" -X PUT --data \'{"keys":"ipdestination,icmpunreachableport", "value":"frames"}' \http://127.0.0.1:8008/flow/unreachableport/json
• Instantly enables network wide monitoring of flows • All switches, all ports, including hosts and virtual switches • Contrast with task of re-configuring Flexible Netflow/IPFIX caches on
every switch in multi-vendor network. How many simultaneous flow definitions are allowed? What key / value combinations are allowed?
curl -H "Content-Type:application/json" -X PUT --data \'{"value":"frames"}' \http://127.0.0.1:8008/flow/frames/json
InMon sFlow-RT
active timeout active timeout
NetFlow
Open vSwitch
SolarWinds Real-Time NetFlow Analyzer
• sFlow does not use flow cache, so realtime charts more accurately reflect traffic trend • NetFlow spikes caused by flow cache active-timeout for long running connections
Rapid detection of large flows
Flow cache active timeout delays large flow detection,limits value of signal for real-time control applications
Counters and packet samples
http://blog.sflow.com/2013/02/measurement-delay-counters-vs-packet.html
• Packet samples give a fast signal that operates at scale • Counters are maintained in hardware and provide precise traffic totals. • Counters capture rare events, like packet discards, that can severely
impact performance. • Counters report important link state information, like link speed, LAG
group membership etc.
Metrics and Events
Metrics Events
Sources SNMP, sFlow, collectdTraps, syslog, IPFIX,
NetFlow
Crossover Event count Threshold event
Collectors InfluxDB, OpenTSDB, Graphite, rrdtool
Logstash, flowtools, Splunk
Application Performance Security and alerts
Data models and transportssFlow SNMP NetFlow
version 5 OpenConfig Telemetry IPFIX syslog
Model
standard measurements published by sFlow.org, Dataplane
focus: based on IEEE, IETF, APIs (MIB-2,
LAG-MIB, libvirt, JMX, …)
standard MIBs
defined by IETF
standard tcp / udp / icmp flow
record defined by
Cisco
Telemetry defined as part of YANG
configuration models by
OpenConfig.orgControl plane focus: BGP,
MPLS, VLAN, etc.
Encoding XDR (RFC 4506)
ASN1 (IETF)
NetFlow (Cisco)
protobufs, JSON,
NetConf
IPFIX (IETF)
Syslog (RFC 5424)
Transport UDP UDP UDP UDP, HTTPSCTP, TCP, UDP
UDP
Mode Push Pull Push Push Push Push
Easy to combine multiple data sources if you disaggregate tool chain e.g. separate agents from collectors, feed data from all sources into InfluxDB / Logstash
sflowtool
https://github.com/sflow/sflowtool
sflowtool
replicate
asciiPerl, Python, awk, grep
pcap wireshark, tcpdumpNetflow
Open source command line tool
Network visibility for DevOps tools
• Streaming filtering and summarization reduces data volume and increases scaleability of backend tools
• Streaming flow analytics to generate application metrics
Feedback control of cloud infrastructure
“You can’t control what you can’t measure”Tom DeMarco
Cloud depends on network
• Server costs (both capex and power consumption) far exceed networking costs in the data center.
• Network congestion caused server to wait, resulting in poor utilization of cloud infrastructure.
• Optimize network to increase data center efficiency
http://perspectives.mvdirona.com/2010/09/overall-data-center-costs/
“Typically the resource that is most scarce is the network.” Amin Vahdat, Google, ONS2015 Keynote
http://blog.sflow.com/2015/06/optimizing-software-defined-data-center.html
http://blog.sflow.com/2013/02/sdn-and-large-flows.html
Elephant flows are the small number of long lived large flows responsible
for majority of bytes on network
Large “Elephant” flows
ECMP visibility and control
Default sFlow settings
Port Speed Large Flow Sampling Rate Polling Interval
1 Gbit/s >= 100 Mbit/s 1-in-1,000 30 seconds
10 Gbit/s >= 1 Gbit/s 1-in-10,000 30 seconds
25 Gbit/s >= 2.5 Gbit/s 1-in-25,000 30 seconds
40 Gbit/s >= 4 Gbit/s 1-in-40,000 30 seconds
50 Gbit/s >= 5 Gbit/s 1-in-50,000 30 seconds
100 Gbit/s >= 10 Gbit/s 1-in-100,000 30 seconds
Supports real-time detection of large “Elephant” flows where large flow is defined as flow consuming >10% of link bandwidth for >1 second
http://blog.sflow.com/2013/06/large-flow-detection.html
Elephant flow collisions
http://blog.sflow.com/2015/03/ecmp-visibility-with-cumulus-linux.html
Elephant flow collisions
http://blog.sflow.com/2015/03/ecmp-visibility-with-cumulus-linux.html
ECMP monitoring challenge
• large number of links, 12 x 10G links
• all links need to be monitored continuously, 180G total bandwidth
• real-time detection of congested links
• real-time detection of Elephant flows
http://blog.sflow.com/2015/03/ecmp-visibility-with-cumulus-linux.html
Fabric level performance metrics
• Fabric View application runs on sFlow-RT
• Downloadable from sFlow-RT.com, includes captured data set from 4 node 10G ECMP fabric
• Elephant collisions on spine links occur frequently
• Collisions halve throughput
• Collisions cause packet discards
http://blog.sflow.com/2015/10/fabric-view.html
http://www.slideshare.net/martin_casado/elephants-and-mice-elephant-detection-in-the-vswitch-with-hardware-handling
Large “Elephant” flows delay small “Mice” flows
Separating Elephants and Mice into different queues significantly improves latency and response time
Large flow marking
http://blog.sflow.com/2014/06/restful-control-of-cumulus-linux-acls.html
Large flow marking
ONS2015 SDN ShowcaseCORD: Open-source spine-leaf fabric
http://blog.sflow.com/2015/06/leaf-and-spine-traffic-engineering.html
CORD: Open-source spine-leaf fabric
RackRack
RackRack
Spine
http://blog.sflow.com/2015/08/cord-open-source-spine-leaf-fabric.html
SDN Central Demo Friday
SDN Optimization of Hybrid Packet / Optical Data Center Fabric
http://blog.sflow.com/2014/09/sdn-control-of-hybrid-packet-optical.html
Cumulus, Calient and InMon PoC. Large flows detected in real-time and diverted to direct ToR to ToR optical circuit
SDN Optimization of Hybrid Packet / Optical Data Center Fabric
Software Defined Internet Router (SIR)
https://eos.arista.com/spotifys-sdn-internet-router/
IXP Routes Installed Routes not installedLINX 18672 95711
DECIX 12518 108164EQIX-ASH 27687 75146
https://eos.arista.com/arista-eos-bgp-selective-route-download/
White box Internet router PoC
Internet Router PoC
http://blog.sflow.com/2015/07/white-box-internet-router-poc.html
Peer 1 Peer 2 Peer N
DDoS Mitigation
REST APIsFlow-RT controller
real-time analytics OpenFlow, REST, BGP, …
Large flow marking
Routing Offload
BGP
…
BGP
Router
Switch
http://blog.sflow.com/2015/10/active-route-manager.html
Open Networking Summit SDN Idol winning solution
Real-time SDN Analytics for DDoS mitigation
DDoS Mitigation Market Opportunity
DDoS Attack Megatrends [Reference 1]
• High bandwidth, volumetric infrastructure layer (Layer 3 & 4) attacks increased approximately 30 percent
• DDoS attack volume also increased month-to-month in 2013, with 10 out of 12 months showing higher attack volume compared to 2012
• Average DDoS attack sizes continued to increase – many over 100 Gbps, the largest peaking at 179 Gbps
DDoS Mitigation Market Growth
• $870M market by 2017, 18.2% CAGR – Source: IDC: Worldwide DDoS Prevention Products and Services 2013-2017 Forecast
• $1049M market by 2017, 25% CAGR – Source: Infonetics: Global DDoS Prevention Appliances 2012-2017 Forecast
Reference 1: Top DDoS Attack Trends http://www.itbriefcase.net/top-ddos-attack-trends-for-2013
DDoS Mitigation Use Case (1)
ISP 1
ISP 2
ISP N
• ISP/IX is uniquely positioned to protect customers from DDoS flood attacks• New revenue from DDoS mitigation service + differentiates ISP/IX service
Attacker
User Prevent attack from overwhelming customer access link
Filter attack traffic in real-time
Customer network
DDoS target hostAttack on single host can take out entire customer data center. Customer cannot mitigate flood attack without upstream help
ISP / IX
ISP/IX Market Segment
Customer portal
DDoS Mitigation Service
Web UI + RESTful programmatic API• real-time TopN analytics• programmable filtering of traffic• set thresholds + automatic blocking
Real-time sFlow visibility, Hybrid OpenFlow Control capability of Brocade switches/routers
REST API
InM
on s
Flow
-RT
REST API
Ope
nFlo
w C
ontr
olle
r
DDoS Mitigation Application
Customer Network
Internet
1. Flood attack overloads customer port
2. Attack maps to large flows [Ref. 2]. sFlow-RT detects attack (maps to large flows) and characterizes attack (srcip, dstip, protocol, ports, etc.)
3. mitigation application takes signature, applies customer policy, selects optimal control and push OpenFlow rule(s) to switch(es)
5. OpenFlow rule(s) applied to switch forwarding path to drop / mark traffic and protect link
HTTPS HTTPS
4. Controller pushes OpenFlow rule(s) to switch(es)
OpenFlow 1.3 Match Fieldsline rate filtering using Brocade switches
Reference 2: IETF I2RS Working Group Draft - https://ietf.org/doc/draft-krishnan-i2rs-large-flow-use-case/
Demonstration
http://blog.sflow.com/2014/03/ons2014-sdn-idol-finalist-demonstrations.html
DDoS Mitigation
Cloudflare DDoS mitigation
http://blog.sflow.com/2016/02/cloudflare-ddos-mitigation-pipeline.html
• sFlow used to identify attack signature • BPF created to match signature • Detailed signatures minimize collateral
damage
Big Switch Network Webinar
Big Tap sFlow: Enabling Pervasive Flow-level Visibility
Pervasive monitoring
http://blog.sflow.com/2015/04/big-tap-sflow-enabling-pervasive-flow.html
Targeted capture
http://blog.sflow.com/2015/04/big-tap-sflow-enabling-pervasive-flow.html
Comments
• sFlow instrumentation is widely available in switches http://sflow.org/products/network.php
• Host sFlow (sFlow.net) agent extends visibility into servers (works with libpcap, iptables, Open vSwitch to efficiently sample packets in host data plane)
• Common data model ensures strong interoperability across sFlow data sources
• Streaming counter and packet telemetry across network, compute and application tiers makes data center observable
• Observability makes it possible to apply feedback controls
Extra slides
Host sFlow monitoring of Linux datapath
Technology Reference
Adapter, bridge, macvlan, ipvlan
Berkeley Packet Filter (BPF) sampling
function
http://blog.sflow.com/2016/02/linux-bridge-
macvlan-ipvlan-adapters.html
Open vSwitch Kernel datapath has sFlow support
http://openvswitch.org/support/config-
cookbooks/sflow/
Linux Firewalliptables statistic module random
function with ulog
http://blog.sflow.com/2010/12/ulog.html
Top of Rack Switch
ASIC provides wirespeed monitoring
of attached servers
http://blog.sflow.com/2010/04/hybrid-server-
monitoring.html
Efficient monitoring of high traffic production workloads
Open vSwitch Fall Conference
New OVS instrumentation features aimed at real-time monitoring of virtual networks
Overlay / Underlay Visibility
http://blog.sflow.com/2015/11/network-virtualization-visibility-demo.html
Open vSwitch Fall Conference
OVN service injection demonstration
Service injection
http://blog.sflow.com/2015/11/ovn-service-injection-demonstration.html
White Paper
Actionable Intelligence in the SDN Ecosystem: Optimizing Network Traffic through FRSA
WAN Optimization
http://blog.sflow.com/2015/06/wan-optimization-using-real-time.html
Dell NFV Summit 2015
Demystifying NFV Infrastructure Hotspots End-to-end Monitoring, Analytics & SDN Control
Demystifying NFV Infrastructure Hotspots
http://blog.sflow.com/2016/01/demystifying-nfv-infrastructure-hotspots.html
Hybrid OpenFlow ECMP testbed
Hybrid OpenFlow ECMP testbed
http://blog.sflow.com/2015/01/hybrid-openflow-ecmp-testbed.html
http://mininet.org/
• Simulated ECMP network for developing visibility and control applications
• sFlow support in Open vSwitch
• OpenFlow for control
The sFlow Standard: Scalable, Unified Monitoring of Networks, Systems and Applications
2012 Velocity Conference
Tagged.com case study
http://blog.sflow.com/2013/04/velocity-conference-talk.html