Download - Shmoocon 2015 - httpscreenshot
![Page 1: Shmoocon 2015 - httpscreenshot](https://reader033.vdocuments.site/reader033/viewer/2022052602/55a963db1a28ab4a108b4627/html5/thumbnails/1.jpg)
httpscreenshot A Tool for Both Teams
![Page 2: Shmoocon 2015 - httpscreenshot](https://reader033.vdocuments.site/reader033/viewer/2022052602/55a963db1a28ab4a108b4627/html5/thumbnails/2.jpg)
Outline
• Who we are
• The problem • Our solution
• Demo
• How we’ve used it
• Q&A
![Page 3: Shmoocon 2015 - httpscreenshot](https://reader033.vdocuments.site/reader033/viewer/2022052602/55a963db1a28ab4a108b4627/html5/thumbnails/3.jpg)
Who We Are – Steve Breen
• Senior penetration tester
• Former “Enterprise” developer – Current hacky script developer
• Vulnerability and exploit development hobbyist by night
@breenmachine
breenmachine.blogspot.com
![Page 4: Shmoocon 2015 - httpscreenshot](https://reader033.vdocuments.site/reader033/viewer/2022052602/55a963db1a28ab4a108b4627/html5/thumbnails/4.jpg)
Who we are – Justin Kennedy
• Lifelong security hobbyist, actively for ~15 years
• Intern -> Computer Tech -> Help Desk -> SOC -> Network Security -> Junior PT -> Senior PT -> Team Lead
• Terrible at making slides look pretty… if you came to see pretty slides, you may be in the wrong talk. If you came to see an awesome tool that is available (and is OSS) as of today, you’re in the right place.
• @jstnkndy / juken (freenode)
![Page 5: Shmoocon 2015 - httpscreenshot](https://reader033.vdocuments.site/reader033/viewer/2022052602/55a963db1a28ab4a108b4627/html5/thumbnails/5.jpg)
Before we get started
_o/
![Page 6: Shmoocon 2015 - httpscreenshot](https://reader033.vdocuments.site/reader033/viewer/2022052602/55a963db1a28ab4a108b4627/html5/thumbnails/6.jpg)
Blue Team challenge
Let’s start off with a question…
Can you account for every device or application on your network?
Why not?
![Page 7: Shmoocon 2015 - httpscreenshot](https://reader033.vdocuments.site/reader033/viewer/2022052602/55a963db1a28ab4a108b4627/html5/thumbnails/7.jpg)
Blue Team challenge reasons
1. You’ve inherited an infrastructure that you didn’t build and (of course) not everything was documented by your predecessor.
2. You work in an environment where business units don't necessarily communicate as much as they should and another business unit has spun up some demo or test application without telling you.
3. You forgot about that old NT4 or tomcat box that no one has touched in the past 10 years.
4. Or someone just plugged some shit into a network jack.
![Page 8: Shmoocon 2015 - httpscreenshot](https://reader033.vdocuments.site/reader033/viewer/2022052602/55a963db1a28ab4a108b4627/html5/thumbnails/8.jpg)
Red Team challenge
1. We are constantly attempting to compromise organizations that we don't know anything about (besides our recon).
2. It's our job to identify what the target attack surface looks like.
3. Anyone in here ever masscan a /8 for common web ports?
4. Let’s face it, we don’t always have as much time as we’d like on an assessment.
![Page 9: Shmoocon 2015 - httpscreenshot](https://reader033.vdocuments.site/reader033/viewer/2022052602/55a963db1a28ab4a108b4627/html5/thumbnails/9.jpg)
*Disclaimer: Some of these may look familiar
![Page 10: Shmoocon 2015 - httpscreenshot](https://reader033.vdocuments.site/reader033/viewer/2022052602/55a963db1a28ab4a108b4627/html5/thumbnails/10.jpg)
![Page 11: Shmoocon 2015 - httpscreenshot](https://reader033.vdocuments.site/reader033/viewer/2022052602/55a963db1a28ab4a108b4627/html5/thumbnails/11.jpg)
root
calvin
![Page 12: Shmoocon 2015 - httpscreenshot](https://reader033.vdocuments.site/reader033/viewer/2022052602/55a963db1a28ab4a108b4627/html5/thumbnails/12.jpg)
![Page 13: Shmoocon 2015 - httpscreenshot](https://reader033.vdocuments.site/reader033/viewer/2022052602/55a963db1a28ab4a108b4627/html5/thumbnails/13.jpg)
![Page 14: Shmoocon 2015 - httpscreenshot](https://reader033.vdocuments.site/reader033/viewer/2022052602/55a963db1a28ab4a108b4627/html5/thumbnails/14.jpg)
![Page 15: Shmoocon 2015 - httpscreenshot](https://reader033.vdocuments.site/reader033/viewer/2022052602/55a963db1a28ab4a108b4627/html5/thumbnails/15.jpg)
Our Solution HTTPScreenshot/Cluster
• HTTPScreenshot: A python script to screenshot thousands of websites really quickly (and reliably)
• Cluster: A script to do “fuzzy matching” on HTML pages. Produce immediately usable output with “similar” pages grouped together.
![Page 16: Shmoocon 2015 - httpscreenshot](https://reader033.vdocuments.site/reader033/viewer/2022052602/55a963db1a28ab4a108b4627/html5/thumbnails/16.jpg)
HTTPScreenshot.py
• Goals: Fast, Thorough, Automagic • Challenges: Code was hacked together during
assessments – needs some TLC • Fun features:
– Input is nmap/masscan output – Javascript parsed and executed – SSL autodetect – SSL Certificate domain scraping for vhosts – Headless (configurable fail-over to FireFox) – Threaded – Saves PNG and HTML (good for grep’ing) – Attempts tls 1.0 and falls back to sslv3 when necessary
![Page 17: Shmoocon 2015 - httpscreenshot](https://reader033.vdocuments.site/reader033/viewer/2022052602/55a963db1a28ab4a108b4627/html5/thumbnails/17.jpg)
Cluster.py
• Identify “similar” websites and group them together
• Displays the resulting groups in a useful way (HTML output with JS “hoverzoom”)
• Algorithm - Reduces to DBSCAN: – Needed a clustering algorithm that didn’t require definition of “k” – Uses HTML tag/attr values – computes a “similarity” score for two
sites • href, name, src, id, class, title, h1
– Works fairly well – could DEFINITELY be improved upon
• Supports “diff” reports. Sites that have been changed/added/removed since the last scan
![Page 18: Shmoocon 2015 - httpscreenshot](https://reader033.vdocuments.site/reader033/viewer/2022052602/55a963db1a28ab4a108b4627/html5/thumbnails/18.jpg)
Demo!
![Page 19: Shmoocon 2015 - httpscreenshot](https://reader033.vdocuments.site/reader033/viewer/2022052602/55a963db1a28ab4a108b4627/html5/thumbnails/19.jpg)
How we’ve used it
![Page 20: Shmoocon 2015 - httpscreenshot](https://reader033.vdocuments.site/reader033/viewer/2022052602/55a963db1a28ab4a108b4627/html5/thumbnails/20.jpg)
Story 1 - SQLi
![Page 21: Shmoocon 2015 - httpscreenshot](https://reader033.vdocuments.site/reader033/viewer/2022052602/55a963db1a28ab4a108b4627/html5/thumbnails/21.jpg)
Story 2 – WTF is SuperGoose?
![Page 22: Shmoocon 2015 - httpscreenshot](https://reader033.vdocuments.site/reader033/viewer/2022052602/55a963db1a28ab4a108b4627/html5/thumbnails/22.jpg)
Story 2 – WTF is SuperGoose?
![Page 23: Shmoocon 2015 - httpscreenshot](https://reader033.vdocuments.site/reader033/viewer/2022052602/55a963db1a28ab4a108b4627/html5/thumbnails/23.jpg)
![Page 24: Shmoocon 2015 - httpscreenshot](https://reader033.vdocuments.site/reader033/viewer/2022052602/55a963db1a28ab4a108b4627/html5/thumbnails/24.jpg)
Story 3 – Bug Bounties
![Page 25: Shmoocon 2015 - httpscreenshot](https://reader033.vdocuments.site/reader033/viewer/2022052602/55a963db1a28ab4a108b4627/html5/thumbnails/25.jpg)
Cheat Sheet: Usage
• masscan --iL scope.txt --oG out.gnmap –rate 100000
• httpscreenshot.py -i out.gnmap --headless --workers 50 --timeout 45 --autodetect --vhosts --retries 1
• cluster.py –d <httpscreenshot directory>
![Page 26: Shmoocon 2015 - httpscreenshot](https://reader033.vdocuments.site/reader033/viewer/2022052602/55a963db1a28ab4a108b4627/html5/thumbnails/26.jpg)
masshttp
#!/bin/sh ~/tools/masscan –p1-65535 -iL scope.txt -oG out.gnmap --rate 100000 mkdir httpscreenshot cd httpscreenshot python ~/tools/httpscreenshot.py -i ../out.gnmap -p -t 50 -w 45 -a -vH -r 1 python ~/tools/httpscreenshot.py -i ../out.gnmap -p -t 10 -w 5 -a -vH -r 1 cd .. python ~/tools/cluster.py -d httpscreenshot/
![Page 27: Shmoocon 2015 - httpscreenshot](https://reader033.vdocuments.site/reader033/viewer/2022052602/55a963db1a28ab4a108b4627/html5/thumbnails/27.jpg)
Questions / Suggestions? Code: github.com/breenmachine/