![Page 1: 2013 Trend Micro 25th Anniversary Threat Connect : a visualized cyber-threats entity reporting system backed with Hadoop ecosystem Scott Miao, Trend Micro](https://reader038.vdocuments.site/reader038/viewer/2022110208/56649dbe5503460f94ab18a9/html5/thumbnails/1.jpg)
2013 Trend Micro 25th Anniversary
Threat Connect : a visualized cyber-threats entity reporting system backed with Hadoop ecosystem
Scott Miao, Trend [email protected]
@takeshi.miao
![Page 2: 2013 Trend Micro 25th Anniversary Threat Connect : a visualized cyber-threats entity reporting system backed with Hadoop ecosystem Scott Miao, Trend Micro](https://reader038.vdocuments.site/reader038/viewer/2022110208/56649dbe5503460f94ab18a9/html5/thumbnails/2.jpg)
2013 Trend Micro 25th Anniversary
Who am I
• RD, SPN, Trend Micro• 3+ years for Hadoop eco system• Expertise in HDFS/MR/HBase• @takeshi.miao
![Page 3: 2013 Trend Micro 25th Anniversary Threat Connect : a visualized cyber-threats entity reporting system backed with Hadoop ecosystem Scott Miao, Trend Micro](https://reader038.vdocuments.site/reader038/viewer/2022110208/56649dbe5503460f94ab18a9/html5/thumbnails/3.jpg)
2013 Trend Micro 25th Anniversary
Agenda
• Threat intelligence problem• Challenges and Solutions• Summary
![Page 4: 2013 Trend Micro 25th Anniversary Threat Connect : a visualized cyber-threats entity reporting system backed with Hadoop ecosystem Scott Miao, Trend Micro](https://reader038.vdocuments.site/reader038/viewer/2022110208/56649dbe5503460f94ab18a9/html5/thumbnails/4.jpg)
2013 Trend Micro 25th Anniversary
THREAT INTELLIGENCE PROBLEM
“I want to quickly get an overview of the incident, including its scope, timeline, and impact.”
![Page 5: 2013 Trend Micro 25th Anniversary Threat Connect : a visualized cyber-threats entity reporting system backed with Hadoop ecosystem Scott Miao, Trend Micro](https://reader038.vdocuments.site/reader038/viewer/2022110208/56649dbe5503460f94ab18a9/html5/thumbnails/5.jpg)
2013 Trend Micro 25th Anniversary
![Page 6: 2013 Trend Micro 25th Anniversary Threat Connect : a visualized cyber-threats entity reporting system backed with Hadoop ecosystem Scott Miao, Trend Micro](https://reader038.vdocuments.site/reader038/viewer/2022110208/56649dbe5503460f94ab18a9/html5/thumbnails/6.jpg)
2013 Trend Micro 25th Anniversary
![Page 7: 2013 Trend Micro 25th Anniversary Threat Connect : a visualized cyber-threats entity reporting system backed with Hadoop ecosystem Scott Miao, Trend Micro](https://reader038.vdocuments.site/reader038/viewer/2022110208/56649dbe5503460f94ab18a9/html5/thumbnails/7.jpg)
2013 Trend Micro 25th Anniversary 7
![Page 8: 2013 Trend Micro 25th Anniversary Threat Connect : a visualized cyber-threats entity reporting system backed with Hadoop ecosystem Scott Miao, Trend Micro](https://reader038.vdocuments.site/reader038/viewer/2022110208/56649dbe5503460f94ab18a9/html5/thumbnails/8.jpg)
2013 Trend Micro 25th Anniversary
![Page 9: 2013 Trend Micro 25th Anniversary Threat Connect : a visualized cyber-threats entity reporting system backed with Hadoop ecosystem Scott Miao, Trend Micro](https://reader038.vdocuments.site/reader038/viewer/2022110208/56649dbe5503460f94ab18a9/html5/thumbnails/9.jpg)
2013 Trend Micro 25th Anniversary
Threat Connect
• A Web Service for Threat Information Report– RESTful Interface to access– Integrated with TM Deep Discovery products
• Relevant and Actionable Intelligence
![Page 10: 2013 Trend Micro 25th Anniversary Threat Connect : a visualized cyber-threats entity reporting system backed with Hadoop ecosystem Scott Miao, Trend Micro](https://reader038.vdocuments.site/reader038/viewer/2022110208/56649dbe5503460f94ab18a9/html5/thumbnails/10.jpg)
2013 Trend Micro 25th Anniversary
![Page 11: 2013 Trend Micro 25th Anniversary Threat Connect : a visualized cyber-threats entity reporting system backed with Hadoop ecosystem Scott Miao, Trend Micro](https://reader038.vdocuments.site/reader038/viewer/2022110208/56649dbe5503460f94ab18a9/html5/thumbnails/11.jpg)
2013 Trend Micro 25th Anniversary
IP, domain, URL, filename, process, file hash, Virus detection, registry key, etc.
Product 1 Product 2 Product 3 …
Threat Conne
ct
Sand-box File
Detection
Threat
Web
Web Reputatio
nFamil
y Write-up
TE
Virus DB
APT KB
Most relevant threat report with actionable
intelligenceon a single portal
Process and correlates different data sources
![Page 12: 2013 Trend Micro 25th Anniversary Threat Connect : a visualized cyber-threats entity reporting system backed with Hadoop ecosystem Scott Miao, Trend Micro](https://reader038.vdocuments.site/reader038/viewer/2022110208/56649dbe5503460f94ab18a9/html5/thumbnails/12.jpg)
2013 Trend Micro 25th Anniversary
CHALLENGES AND SOLUTIONS
![Page 13: 2013 Trend Micro 25th Anniversary Threat Connect : a visualized cyber-threats entity reporting system backed with Hadoop ecosystem Scott Miao, Trend Micro](https://reader038.vdocuments.site/reader038/viewer/2022110208/56649dbe5503460f94ab18a9/html5/thumbnails/13.jpg)
2013 Trend Micro 25th Anniversary
StoringReal Time AccessPick Your right tool
Big DataMoving
Process & CorrelateGraph Problem
![Page 14: 2013 Trend Micro 25th Anniversary Threat Connect : a visualized cyber-threats entity reporting system backed with Hadoop ecosystem Scott Miao, Trend Micro](https://reader038.vdocuments.site/reader038/viewer/2022110208/56649dbe5503460f94ab18a9/html5/thumbnails/14.jpg)
2013 Trend Micro 25th Anniversary
MOVING
![Page 15: 2013 Trend Micro 25th Anniversary Threat Connect : a visualized cyber-threats entity reporting system backed with Hadoop ecosystem Scott Miao, Trend Micro](https://reader038.vdocuments.site/reader038/viewer/2022110208/56649dbe5503460f94ab18a9/html5/thumbnails/15.jpg)
2013 Trend Micro 25th Anniversary
Hadoop
Event Logs
FBSFBS
FBS
Feed Back log ServiceDear users/services
Accumulate small files
![Page 16: 2013 Trend Micro 25th Anniversary Threat Connect : a visualized cyber-threats entity reporting system backed with Hadoop ecosystem Scott Miao, Trend Micro](https://reader038.vdocuments.site/reader038/viewer/2022110208/56649dbe5503460f94ab18a9/html5/thumbnails/16.jpg)
2013 Trend Micro 25th Anniversary
STORING
![Page 17: 2013 Trend Micro 25th Anniversary Threat Connect : a visualized cyber-threats entity reporting system backed with Hadoop ecosystem Scott Miao, Trend Micro](https://reader038.vdocuments.site/reader038/viewer/2022110208/56649dbe5503460f94ab18a9/html5/thumbnails/17.jpg)
2013 Trend Micro 25th Anniversary
Cost
Easy Process
Archive
HDFS
![Page 18: 2013 Trend Micro 25th Anniversary Threat Connect : a visualized cyber-threats entity reporting system backed with Hadoop ecosystem Scott Miao, Trend Micro](https://reader038.vdocuments.site/reader038/viewer/2022110208/56649dbe5503460f94ab18a9/html5/thumbnails/18.jpg)
2013 Trend Micro 25th Anniversary
PROCESS & CORRELATE
![Page 19: 2013 Trend Micro 25th Anniversary Threat Connect : a visualized cyber-threats entity reporting system backed with Hadoop ecosystem Scott Miao, Trend Micro](https://reader038.vdocuments.site/reader038/viewer/2022110208/56649dbe5503460f94ab18a9/html5/thumbnails/19.jpg)
2013 Trend Micro 25th Anniversary
Pig/MR
• UDFs• MRs for special
cases
Store
• HDFS• Hbase• Solr• RDB
Time
• Batch• Performance
![Page 20: 2013 Trend Micro 25th Anniversary Threat Connect : a visualized cyber-threats entity reporting system backed with Hadoop ecosystem Scott Miao, Trend Micro](https://reader038.vdocuments.site/reader038/viewer/2022110208/56649dbe5503460f94ab18a9/html5/thumbnails/20.jpg)
2013 Trend Micro 25th Anniversary
REAL TIME ACCESS
![Page 21: 2013 Trend Micro 25th Anniversary Threat Connect : a visualized cyber-threats entity reporting system backed with Hadoop ecosystem Scott Miao, Trend Micro](https://reader038.vdocuments.site/reader038/viewer/2022110208/56649dbe5503460f94ab18a9/html5/thumbnails/21.jpg)
2013 Trend Micro 25th Anniversary
Real Time
Access
Free form
search
Random Access
Solr Cloud
HBase
• EX. Sandbox Reports
EX. Threat Detection DBs
![Page 22: 2013 Trend Micro 25th Anniversary Threat Connect : a visualized cyber-threats entity reporting system backed with Hadoop ecosystem Scott Miao, Trend Micro](https://reader038.vdocuments.site/reader038/viewer/2022110208/56649dbe5503460f94ab18a9/html5/thumbnails/22.jpg)
2013 Trend Micro 25th Anniversary
GRAPH MODEL
![Page 23: 2013 Trend Micro 25th Anniversary Threat Connect : a visualized cyber-threats entity reporting system backed with Hadoop ecosystem Scott Miao, Trend Micro](https://reader038.vdocuments.site/reader038/viewer/2022110208/56649dbe5503460f94ab18a9/html5/thumbnails/23.jpg)
2013 Trend Micro 25th Anniversary
Massive scalable ?
Active community ?
Analyzable ?
![Page 24: 2013 Trend Micro 25th Anniversary Threat Connect : a visualized cyber-threats entity reporting system backed with Hadoop ecosystem Scott Miao, Trend Micro](https://reader038.vdocuments.site/reader038/viewer/2022110208/56649dbe5503460f94ab18a9/html5/thumbnails/24.jpg)
2013 Trend Micro 25th Anniversary
• We use HBase as a Graph Storage– Google BigTable and PageRank– HBaseCon2012
![Page 25: 2013 Trend Micro 25th Anniversary Threat Connect : a visualized cyber-threats entity reporting system backed with Hadoop ecosystem Scott Miao, Trend Micro](https://reader038.vdocuments.site/reader038/viewer/2022110208/56649dbe5503460f94ab18a9/html5/thumbnails/25.jpg)
2013 Trend Micro 25th Anniversary
HGraph
Schema Design
Blueprints API
Graph Analysis MRs
https://github.com/tinkerpop/blueprints/wiki
![Page 26: 2013 Trend Micro 25th Anniversary Threat Connect : a visualized cyber-threats entity reporting system backed with Hadoop ecosystem Scott Miao, Trend Micro](https://reader038.vdocuments.site/reader038/viewer/2022110208/56649dbe5503460f94ab18a9/html5/thumbnails/26.jpg)
2013 Trend Micro 25th Anniversary
PICK RIGHT TOOL
![Page 27: 2013 Trend Micro 25th Anniversary Threat Connect : a visualized cyber-threats entity reporting system backed with Hadoop ecosystem Scott Miao, Trend Micro](https://reader038.vdocuments.site/reader038/viewer/2022110208/56649dbe5503460f94ab18a9/html5/thumbnails/27.jpg)
2013 Trend Micro 25th Anniversary
Pick right tool for right usecases
• Silver bullet ?• No one project fits all• One problem may has several choices
http://www.neevtech.com/blog/2013/03/18/hadoop-ecosystem-at-a-glance/
![Page 28: 2013 Trend Micro 25th Anniversary Threat Connect : a visualized cyber-threats entity reporting system backed with Hadoop ecosystem Scott Miao, Trend Micro](https://reader038.vdocuments.site/reader038/viewer/2022110208/56649dbe5503460f94ab18a9/html5/thumbnails/28.jpg)
2013 Trend Micro 25th Anniversary
SUMMARY
![Page 29: 2013 Trend Micro 25th Anniversary Threat Connect : a visualized cyber-threats entity reporting system backed with Hadoop ecosystem Scott Miao, Trend Micro](https://reader038.vdocuments.site/reader038/viewer/2022110208/56649dbe5503460f94ab18a9/html5/thumbnails/29.jpg)
2013 Trend Micro 25th Anniversary
Small files
• Namenode fsimage would explore the memory
• Too many map tasks to run for a job
FBSFBSFBS
![Page 30: 2013 Trend Micro 25th Anniversary Threat Connect : a visualized cyber-threats entity reporting system backed with Hadoop ecosystem Scott Miao, Trend Micro](https://reader038.vdocuments.site/reader038/viewer/2022110208/56649dbe5503460f94ab18a9/html5/thumbnails/30.jpg)
2013 Trend Micro 25th Anniversary
Store your data anyway
• Store all the raw data on the HDFS– Break invisible isolation from different data
sources• Archive your data with deduced easy to use
FileFormat– Trenvi, RC file, ORC file
![Page 31: 2013 Trend Micro 25th Anniversary Threat Connect : a visualized cyber-threats entity reporting system backed with Hadoop ecosystem Scott Miao, Trend Micro](https://reader038.vdocuments.site/reader038/viewer/2022110208/56649dbe5503460f94ab18a9/html5/thumbnails/31.jpg)
2013 Trend Micro 25th Anniversary
Know MR more
• Even you are the pig developer– Deal with MR issues– Write better pig-latin– Sometimes you can only use MR
![Page 32: 2013 Trend Micro 25th Anniversary Threat Connect : a visualized cyber-threats entity reporting system backed with Hadoop ecosystem Scott Miao, Trend Micro](https://reader038.vdocuments.site/reader038/viewer/2022110208/56649dbe5503460f94ab18a9/html5/thumbnails/32.jpg)
2013 Trend Micro 25th Anniversary
Know your data & usecases
• Realtime ? Batch ?• Access Pattern ?• Therefore, you can pick right tool
![Page 33: 2013 Trend Micro 25th Anniversary Threat Connect : a visualized cyber-threats entity reporting system backed with Hadoop ecosystem Scott Miao, Trend Micro](https://reader038.vdocuments.site/reader038/viewer/2022110208/56649dbe5503460f94ab18a9/html5/thumbnails/33.jpg)
2013 Trend Micro 25th Anniversary
THANK YOU GUYS