enhanced flow - internet2 · 2017. 10. 12. · but also includes l7 (bgp and dns) - with...
TRANSCRIPT
Enhanced Flow:
Using BGP, Performance, DNS, and URL data to make your AS go!
Jim MeehanDirector, Product Marketing
All contents © Kentik Inc. 2
What’s the Goal? Answer Critical Questions!
Network Operations
Network Engineering
SecOps DevOps Finance Sales / BD
Is the network the problem?
Are we fast?Slow?
Broken?Confused?
Are we under DDoS attack?
What does this traffic
cost?
Where should we invest
going forward?
All contents © Kentik Inc. 3
Modern Flow AnalyticsThe network doesn’t just serve the business, it sees the business.
The traffic data you already have can give you the actionable intelligence you really need.
It starts with flow data...
But also includes L7 (BGP and DNS) -with performance data! Internal
& InternetScalable
High-resolution
Usable for all teams
Real-time &
Historical
Open &
Easy
06 01
02
0304
05
What’s Needed
Network Traffic Intelligence:Modern NPM with real-time BI
All contents © Kentik Inc. 4
Modern Network Traffic Intelligence
https
NetFlow UDP
BGP
SNMP
Public IP
MitigationDevices
https
RTBH
Data Engine
Alerts
Data ExplorerDashboardsPeering AnalyticsAnomaly Detection
SQL/RESTful APIs
All contents © Kentik Inc. 5
Modern Network Traffic Intelligence - Use Cases
Anomaly Detection
Planning and Peering
Traffic Engineering
DDoS DefensePerformanceManagement
ThreatDetection
ServiceCreation
Network Forensics
Business Analytics
… for technical and business operations!
Now Possible:
Data-DrivenNetwork Operations
All contents © Kentik Inc. 7
What is Data-Driven Network Operations?
Get network traffic intelligence (NetOps + network-savvy BI) by using data to drive your technical and business operations!
• Plays well with app and DevOps groups• Web companies and enterprises are data and analytics driven• DevOps is as well (logs, APM, metrics-at-scale)• But the network world has some catchup to do• Network nerds can have nice things too• Can share with tech peers (systems/apps), and the business side
All contents © Kentik Inc. 8
Key Network Operator Requirements
Formodern Data-DrivenNetworkOperations:
• Nodataaggregation orpre-filtering• Correlation (fusing)betweendatatypes• Fullresolution,searchableandstoredformonths• FAST: Lessthan10sforresults.Cannotwaitminutestoexplore• Network-savvy UIsandAPIs(understandsroutingandCIDR)• Detectanomalies:Shouldnothavetowatchgraphsmanually• Dataandalertsavailableacrossthecompany• “0”-to-usableinminutes toweeks,notmonthstoyears
…But it all needs ‘Food’
All contents © Kentik Inc. 10
Fusing Data for Richer Traffic Analytics
Flow or BGP or SNMP or DNS or logs alone are not enough.
Flow becomes much richer when combined with:• Performance and layer 7 information• BGP attributes• Geography• DNS lookups• Tags (rack, department, customer…)• Config changes and software versions• Threat intelligence and known-bad IPs
Fusing should be real-time, performed at ingest and data-specific
All contents © Kentik Inc. 11
Modern Architecture
DATA FUSION
DecoderModules
MemTable
esNetFlow v5
NetFlow v9
IPFIX
BGP RIB
Custom Tags
SNMP Poller
BGP Daemons
Enrichment DB
DATA FUSION
Geo ←→ IP
ASN ←→ IP
SFlow
ROUTER
TRAFFIC-SAVVY DATASTORE
Single flowfused row
sent to storage
PCAP
PCAPagent
proxy
All contents © Kentik Inc. 12
Where to Get Enhanced Flow?
• You can get data like: Queue depth, re-xmit, latency, L7 data from:
• On-server or sensor software - kprobe, nprobe, argus• Commercial sensors - nBox, nPulse, and others• Packet Brokers - Ixia and Gigamon (IPFIX, potentially more)• IDS (bro) – a superset of most flow fields, + app decode• Web servers (nginx, varnish) – web logs + tcp_info for perf• Load balancers – already see HTTPS-decoded URLs• CISCO AVC, Netflow Lite – generally only on small devices
Use CasesFor “Flow ++”
All contents © Kentik Inc. 14
Flow + BGP -> Network Planning
Flow-based traffic + BGP can be used to help show:
• Path, neighbor, transit, origin, and country of traffic
• Strategic peering and transit changes that can improve perf and costs
• Potential new peers and locations to peer
• Evaluate the potential of new peering exchanges or facilities• Transit relationships that are of high or little value
• Understand ROI before extending backbone links or capacity
All contents © Kentik Inc. 15
Use Case: Network Planning, Traffic by BGP HOP
All contents © Kentik Inc. 16
Use Case: Traffic Cost Analysis / Optimization
All contents © Kentik Inc. 17
Flow + Business Data
IP (or even AS) by itself is often not interesting.
What user (enterprise) or customer (SP) is it?
Answer: Add dimensions for site, app, user, customer.
Group by and ask questions about cost, security, performance, …
So you can see how the network supports the business.
All contents © Kentik Inc. 18
Flow + Business Data = Costs + Competitive Analysis
Internal Customer IDs
And the Main Topic:Flow + HTTP/DNS =
Performance + Security
All contents © Kentik Inc. 20
Use Case: Network Performance Analytics
Flow-based traffic + BGP + network performance data can show:
• Whether issues are in the application or network layer (+ where)
• And where?• And in a way expose-able to internal dev +
app ops• And to pinpoint performance issues by
peer or remote AS path or prefix or DC• Or provably not in the network ☺• With client vs. server vs. app latency
All contents © Kentik Inc. 21
Augmented Flow: App Latency
All contents © Kentik Inc. 22
Augmented Flow: App Latency
All contents © Kentik Inc. 23
Use Case: Network Performance Analysis
Packet Loss per Host
Before After
All contents © Kentik Inc. 24
Perf-Enhanced Flow: TCP Latency / ASN
All contents © Kentik Inc. 25
Perf-Enhanced Flow: TCP Latency / Prefix
All contents © Kentik Inc. 26
Flow + HTTP Augmentation
Types of Data Augmentation:
• URL—what website?• Return Code– 404?• Referrer—Reddit? Duckduckgo?• User Agent---IoS? Android? Windows?• Host Header—foo.com
The Big Problem: HTTPS – but you can still get the cert/site name
All contents © Kentik Inc. 27
HTTP User Agents
All contents © Kentik Inc. 28
HTTP Use Case: Application-Level Attacks
• What was your network transporting last night?• With URL and performance data, many kinds of application
attacks can be detected. • To get * URL info in an HTTPS world, will need to get data
from load balancers or web logs.• Simplest is WAF – looking for SQL fragments, binary, or other
known attack vectors.• Can hook alerts to mitigation methods, even if running OOB
(for example, send TCP FIN/RST in both directions)
All contents © Kentik Inc. 29
DNS Augmented Flow
• DNS logs alone• Must be manually correlated
with other information
• DNS is volumetric (hundreds of queries per second)• Good news: flow export is
even more volumetric
• Flow-generating agents can capture DNS logs!
All contents © Kentik Inc. 30
DNS Augmented Flow
• Combines DNS log message content with flow, BGP, geoIP, performance data, such as: DNS query, query type, return code, response.
• Allowing analyses such as:• Top DNS queries• Top DNS query by src or dest IP
• Against various metrics:• Flows per second• Bits/packets per second
All contents © Kentik Inc. 31
DNS Return Code by PPS
All contents © Kentik Inc. 32
DDoS Detection with Flow + DNS
• Change in top N queried domain• Otherwise anomalous domains• Domains with weird names• Dyn attack:
• Not a traditional DNS flood-style reflection/amplification attack, not detectable by typical volumetric DDoS detection
• Rather, broadly distributed set of nodes making tons of randomized queries that exhausted Dyn server resources
• Unusual domains and random, weird subdomain names• Detectable by regex filtering to show that top N domains
queried had changed• Also can look at source geo or ASN distribution
All contents © Kentik Inc. 33
Augmented Flow: DNS at CYA.Net
All contents © Kentik Inc. 34
Augmented Flow: DNS at CYA.Net
All contents © Kentik Inc. 35
Detecting Emerging RATs (Remote Access Trojans)
• Recent report published on new RAT executed via cleverly disguised Word document that executes four layers of Powershell scripts• https://threatpost.com/new-fileless-attack-using-dns-
queries-to-carry-out-powershell-commands/124078/
• Ultimately, scripts use DNS TXT queries to fetch malicious PowerShell commands stored remotely at C2 servers
• Flow + DNS can help:• Weird request types by source IPs
All contents © Kentik Inc. 36
Takeaways
• Network traffic data can tell you many things about your business.• Even un-augmented!• But with business metadata (like customer, site, app);
BGP, performance, and layer 7 data…• You get even better visibility into network vs. app stack issues• And find+fix security issues!• Requires a bit of fusing, and some big data magic.• But well worth it…
Questions?
kentik.com
Thank you!
Jim MeehanDirector, Solutions Engineering