ip agnostic bot detection - owaspowasp.org/...jan 18, 2017 · travel industry good bots ?!...
TRANSCRIPT
![Page 1: IP Agnostic Bot Detection - OWASPowasp.org/...Jan 18, 2017 · Travel Industry Good Bots ?! Protecting the Most valuable Asset - Data •Competitors’ bots are extracting your data:](https://reader034.vdocuments.site/reader034/viewer/2022050511/5f9bab3e5f4e34260d09be59/html5/thumbnails/1.jpg)
Director, Web Application Security
OWASP January 2017
IP Agnostic Bot Detection
Michael Groskop
![Page 2: IP Agnostic Bot Detection - OWASPowasp.org/...Jan 18, 2017 · Travel Industry Good Bots ?! Protecting the Most valuable Asset - Data •Competitors’ bots are extracting your data:](https://reader034.vdocuments.site/reader034/viewer/2022050511/5f9bab3e5f4e34260d09be59/html5/thumbnails/2.jpg)
About me
Director of Web Application Security @Radware
Over 12 years in the Security Space:
• Web Applications Security
• Authentication & SSO
• Cloud solutions
• Database Security
2
https://www.linkedin.com/in/michaelgroskop
![Page 3: IP Agnostic Bot Detection - OWASPowasp.org/...Jan 18, 2017 · Travel Industry Good Bots ?! Protecting the Most valuable Asset - Data •Competitors’ bots are extracting your data:](https://reader034.vdocuments.site/reader034/viewer/2022050511/5f9bab3e5f4e34260d09be59/html5/thumbnails/3.jpg)
Scoping the Bot Problem
Bots Behavioral Attributes
IP Agnostic Bot Detection
![Page 4: IP Agnostic Bot Detection - OWASPowasp.org/...Jan 18, 2017 · Travel Industry Good Bots ?! Protecting the Most valuable Asset - Data •Competitors’ bots are extracting your data:](https://reader034.vdocuments.site/reader034/viewer/2022050511/5f9bab3e5f4e34260d09be59/html5/thumbnails/4.jpg)
Bots Generate ~½ of the Internet Traffic
![Page 5: IP Agnostic Bot Detection - OWASPowasp.org/...Jan 18, 2017 · Travel Industry Good Bots ?! Protecting the Most valuable Asset - Data •Competitors’ bots are extracting your data:](https://reader034.vdocuments.site/reader034/viewer/2022050511/5f9bab3e5f4e34260d09be59/html5/thumbnails/5.jpg)
~30% of the Web traffic is generated by Bad Bots
![Page 6: IP Agnostic Bot Detection - OWASPowasp.org/...Jan 18, 2017 · Travel Industry Good Bots ?! Protecting the Most valuable Asset - Data •Competitors’ bots are extracting your data:](https://reader034.vdocuments.site/reader034/viewer/2022050511/5f9bab3e5f4e34260d09be59/html5/thumbnails/6.jpg)
330 Million Records Breached in the US since 2011
US population projected to Jan 2017 is 325,400,000
Sensitive data records breached by hacking in the United States 2011 - 2016
![Page 7: IP Agnostic Bot Detection - OWASPowasp.org/...Jan 18, 2017 · Travel Industry Good Bots ?! Protecting the Most valuable Asset - Data •Competitors’ bots are extracting your data:](https://reader034.vdocuments.site/reader034/viewer/2022050511/5f9bab3e5f4e34260d09be59/html5/thumbnails/7.jpg)
Scraping services are “Just a Google Away”
![Page 8: IP Agnostic Bot Detection - OWASPowasp.org/...Jan 18, 2017 · Travel Industry Good Bots ?! Protecting the Most valuable Asset - Data •Competitors’ bots are extracting your data:](https://reader034.vdocuments.site/reader034/viewer/2022050511/5f9bab3e5f4e34260d09be59/html5/thumbnails/8.jpg)
Are you a Bot?
Alan Turing Test (1950) Eugene Goostman (2013) ChatBots (2017)
In a “reverse” Turing test, a computer is to determine whether it is interacting with a human or another computer.
![Page 9: IP Agnostic Bot Detection - OWASPowasp.org/...Jan 18, 2017 · Travel Industry Good Bots ?! Protecting the Most valuable Asset - Data •Competitors’ bots are extracting your data:](https://reader034.vdocuments.site/reader034/viewer/2022050511/5f9bab3e5f4e34260d09be59/html5/thumbnails/9.jpg)
Google reCAPTCHA ?
![Page 10: IP Agnostic Bot Detection - OWASPowasp.org/...Jan 18, 2017 · Travel Industry Good Bots ?! Protecting the Most valuable Asset - Data •Competitors’ bots are extracting your data:](https://reader034.vdocuments.site/reader034/viewer/2022050511/5f9bab3e5f4e34260d09be59/html5/thumbnails/10.jpg)
~25% of the Web traffic is generated by Good Bots
![Page 11: IP Agnostic Bot Detection - OWASPowasp.org/...Jan 18, 2017 · Travel Industry Good Bots ?! Protecting the Most valuable Asset - Data •Competitors’ bots are extracting your data:](https://reader034.vdocuments.site/reader034/viewer/2022050511/5f9bab3e5f4e34260d09be59/html5/thumbnails/11.jpg)
Travel Industry Good Bots ?!
![Page 12: IP Agnostic Bot Detection - OWASPowasp.org/...Jan 18, 2017 · Travel Industry Good Bots ?! Protecting the Most valuable Asset - Data •Competitors’ bots are extracting your data:](https://reader034.vdocuments.site/reader034/viewer/2022050511/5f9bab3e5f4e34260d09be59/html5/thumbnails/12.jpg)
Protecting the Most valuable Asset - Data
• Competitors’ bots are extracting your data:
– Price comparison to beat your prices
– Content Theft & Data Aggregation
– Faux buyers: continuously creating but never completing reservations
![Page 13: IP Agnostic Bot Detection - OWASPowasp.org/...Jan 18, 2017 · Travel Industry Good Bots ?! Protecting the Most valuable Asset - Data •Competitors’ bots are extracting your data:](https://reader034.vdocuments.site/reader034/viewer/2022050511/5f9bab3e5f4e34260d09be59/html5/thumbnails/13.jpg)
Scoping the Bot Problem
Bots Behavioral Attributes
IP Agnostic Bot Detection
![Page 14: IP Agnostic Bot Detection - OWASPowasp.org/...Jan 18, 2017 · Travel Industry Good Bots ?! Protecting the Most valuable Asset - Data •Competitors’ bots are extracting your data:](https://reader034.vdocuments.site/reader034/viewer/2022050511/5f9bab3e5f4e34260d09be59/html5/thumbnails/14.jpg)
Single Request Attack
SQL Injection
XSS
CSRF
…
Continuous Attack
Application DDoS
Password Cracking / Brute Force
Site Scraping / Data Harvesting
Account Lockdown
…
Single Request vs. Continuous Attacks
![Page 15: IP Agnostic Bot Detection - OWASPowasp.org/...Jan 18, 2017 · Travel Industry Good Bots ?! Protecting the Most valuable Asset - Data •Competitors’ bots are extracting your data:](https://reader034.vdocuments.site/reader034/viewer/2022050511/5f9bab3e5f4e34260d09be59/html5/thumbnails/15.jpg)
Commonly Used Frameworks
![Page 16: IP Agnostic Bot Detection - OWASPowasp.org/...Jan 18, 2017 · Travel Industry Good Bots ?! Protecting the Most valuable Asset - Data •Competitors’ bots are extracting your data:](https://reader034.vdocuments.site/reader034/viewer/2022050511/5f9bab3e5f4e34260d09be59/html5/thumbnails/16.jpg)
Single Source with multiple IPs
The Problem:• Single Attack source
• Attacker dynamically changes its IP
• DHCP reset, Anonymous proxies etc.
![Page 17: IP Agnostic Bot Detection - OWASPowasp.org/...Jan 18, 2017 · Travel Industry Good Bots ?! Protecting the Most valuable Asset - Data •Competitors’ bots are extracting your data:](https://reader034.vdocuments.site/reader034/viewer/2022050511/5f9bab3e5f4e34260d09be59/html5/thumbnails/17.jpg)
Scoping the Bot Problem
Bots Behavioral Attributes
IP Agnostic Bot Detection
![Page 18: IP Agnostic Bot Detection - OWASPowasp.org/...Jan 18, 2017 · Travel Industry Good Bots ?! Protecting the Most valuable Asset - Data •Competitors’ bots are extracting your data:](https://reader034.vdocuments.site/reader034/viewer/2022050511/5f9bab3e5f4e34260d09be59/html5/thumbnails/18.jpg)
Solution Requirements
IP Agnostic Unique Cross Platform Correlation
![Page 19: IP Agnostic Bot Detection - OWASPowasp.org/...Jan 18, 2017 · Travel Industry Good Bots ?! Protecting the Most valuable Asset - Data •Competitors’ bots are extracting your data:](https://reader034.vdocuments.site/reader034/viewer/2022050511/5f9bab3e5f4e34260d09be59/html5/thumbnails/19.jpg)
Device Fingerprint
Identify a browser/bot through info collection.
Dozens of browser attributes can be collected on the client side.
JavaScript allows collecting detailed browser info.
The power of the fingerprint is in the consolidated information.
Operating System
System Fonts
Browser Plug-ins
Screen Resolution
Local IPs
![Page 20: IP Agnostic Bot Detection - OWASPowasp.org/...Jan 18, 2017 · Travel Industry Good Bots ?! Protecting the Most valuable Asset - Data •Competitors’ bots are extracting your data:](https://reader034.vdocuments.site/reader034/viewer/2022050511/5f9bab3e5f4e34260d09be59/html5/thumbnails/20.jpg)
How many bits of information are required to uniquely identify an individual from the entire population?
How distinct does a fingerprint need to be?
The current estimated world population: 7,477,780,179.
log2 (7,477,780,179) < 33
![Page 21: IP Agnostic Bot Detection - OWASPowasp.org/...Jan 18, 2017 · Travel Industry Good Bots ?! Protecting the Most valuable Asset - Data •Competitors’ bots are extracting your data:](https://reader034.vdocuments.site/reader034/viewer/2022050511/5f9bab3e5f4e34260d09be59/html5/thumbnails/21.jpg)
Entropy of Browser Fingerprint
To differentiate between 1,000,000 unique users, who access the secured environment requires 20 bits of information:
log2 (1,000,000) < 20
![Page 22: IP Agnostic Bot Detection - OWASPowasp.org/...Jan 18, 2017 · Travel Industry Good Bots ?! Protecting the Most valuable Asset - Data •Competitors’ bots are extracting your data:](https://reader034.vdocuments.site/reader034/viewer/2022050511/5f9bab3e5f4e34260d09be59/html5/thumbnails/22.jpg)
Navigator variable
UserAgentApp NameApp Code NameApp VersionBuild IDPlatformCPU ClassOS CPUProduct
Product SubVendorVendor SubLanguageUser LanguageBrowser LanguageSystem Language
Screen variable
Screen W x HAvailable W x HColor DepthPixel DepthDevice DPI (X, Y)Logical DPI (X, Y)Update IntervalSystem DPI (X, Y)
JavaScript Variables
![Page 23: IP Agnostic Bot Detection - OWASPowasp.org/...Jan 18, 2017 · Travel Industry Good Bots ?! Protecting the Most valuable Asset - Data •Competitors’ bots are extracting your data:](https://reader034.vdocuments.site/reader034/viewer/2022050511/5f9bab3e5f4e34260d09be59/html5/thumbnails/23.jpg)
HTTP Headers
User AgentAccept Accept-LanguageAccept-CharsetAccept-EncodingX-FORWARDED-FORTRUE-CLIENT-IPViaDNT (Do not track)
TCP Packet Parameters
Initial packet sizeIP Initial TTL TCP Window sizeWindow scaling value Max segment sizeTCP OptionsIP flagsIP Type of serviceIP Total Length
"don't fragment" flag"sackOK" flag"nop" flag
Other Fingerprinting Approaches
![Page 24: IP Agnostic Bot Detection - OWASPowasp.org/...Jan 18, 2017 · Travel Industry Good Bots ?! Protecting the Most valuable Asset - Data •Competitors’ bots are extracting your data:](https://reader034.vdocuments.site/reader034/viewer/2022050511/5f9bab3e5f4e34260d09be59/html5/thumbnails/24.jpg)
navigator.plugins
for (plugin of navigator.plugins) { console.log(plugin.name); }
"Shockwave Flash“
"QuickTime Plug-in 7.7.3“
"Default Browser Helper“
"Unity Player“
"Google Earth Plug-in“
"Silverlight Plug-In“
"Java Applet Plug-in“
"Adobe Acrobat NPAPI Plug-in, Version 11.0.02“
"WacomTabletPlugin“
![Page 25: IP Agnostic Bot Detection - OWASPowasp.org/...Jan 18, 2017 · Travel Industry Good Bots ?! Protecting the Most valuable Asset - Data •Competitors’ bots are extracting your data:](https://reader034.vdocuments.site/reader034/viewer/2022050511/5f9bab3e5f4e34260d09be59/html5/thumbnails/25.jpg)
Fingerprint Example
![Page 26: IP Agnostic Bot Detection - OWASPowasp.org/...Jan 18, 2017 · Travel Industry Good Bots ?! Protecting the Most valuable Asset - Data •Competitors’ bots are extracting your data:](https://reader034.vdocuments.site/reader034/viewer/2022050511/5f9bab3e5f4e34260d09be59/html5/thumbnails/26.jpg)
Mobile App Support
![Page 27: IP Agnostic Bot Detection - OWASPowasp.org/...Jan 18, 2017 · Travel Industry Good Bots ?! Protecting the Most valuable Asset - Data •Competitors’ bots are extracting your data:](https://reader034.vdocuments.site/reader034/viewer/2022050511/5f9bab3e5f4e34260d09be59/html5/thumbnails/27.jpg)
Summary
![Page 28: IP Agnostic Bot Detection - OWASPowasp.org/...Jan 18, 2017 · Travel Industry Good Bots ?! Protecting the Most valuable Asset - Data •Competitors’ bots are extracting your data:](https://reader034.vdocuments.site/reader034/viewer/2022050511/5f9bab3e5f4e34260d09be59/html5/thumbnails/28.jpg)
What have we talked about?
Scoping the Bot Problem
Bots Behavioral Attributes
IP Agnostic Bot Detection
Device Fingerprint