scott miao, trend micro s [email protected] @ takeshi.miao
DESCRIPTION
Threat Connect : a visualized cyber-threats entity reporting system backed with H adoop ecosystem. Scott Miao, Trend Micro s [email protected] @ takeshi.miao. Who am I. RD, SPN, Trend Micro 3+ years for Hadoop eco system Expertise in HDFS/MR/ HBase @ takeshi.miao. Agenda. - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Scott Miao, Trend Micro s cott_miao@trend.tw @ takeshi.miao](https://reader035.vdocuments.site/reader035/viewer/2022081419/568163c7550346895dd4fb71/html5/thumbnails/1.jpg)
2013 Trend Micro 25th Anniversary
Threat Connect : a visualized cyber-threats entity reporting system backed with Hadoop ecosystem
Scott Miao, Trend [email protected]
@takeshi.miao
![Page 2: Scott Miao, Trend Micro s cott_miao@trend.tw @ takeshi.miao](https://reader035.vdocuments.site/reader035/viewer/2022081419/568163c7550346895dd4fb71/html5/thumbnails/2.jpg)
2013 Trend Micro 25th Anniversary
Who am I
• RD, SPN, Trend Micro• 3+ years for Hadoop eco system• Expertise in HDFS/MR/HBase• @takeshi.miao
![Page 3: Scott Miao, Trend Micro s cott_miao@trend.tw @ takeshi.miao](https://reader035.vdocuments.site/reader035/viewer/2022081419/568163c7550346895dd4fb71/html5/thumbnails/3.jpg)
2013 Trend Micro 25th Anniversary
Agenda• Threat intelligence problem• Challenges and Solutions• Summary
![Page 4: Scott Miao, Trend Micro s cott_miao@trend.tw @ takeshi.miao](https://reader035.vdocuments.site/reader035/viewer/2022081419/568163c7550346895dd4fb71/html5/thumbnails/4.jpg)
2013 Trend Micro 25th Anniversary
THREAT INTELLIGENCE PROBLEM
“I want to quickly get an overview of the incident, including its scope, timeline, and impact.”
![Page 5: Scott Miao, Trend Micro s cott_miao@trend.tw @ takeshi.miao](https://reader035.vdocuments.site/reader035/viewer/2022081419/568163c7550346895dd4fb71/html5/thumbnails/5.jpg)
2013 Trend Micro 25th Anniversary
![Page 6: Scott Miao, Trend Micro s cott_miao@trend.tw @ takeshi.miao](https://reader035.vdocuments.site/reader035/viewer/2022081419/568163c7550346895dd4fb71/html5/thumbnails/6.jpg)
2013 Trend Micro 25th Anniversary
![Page 7: Scott Miao, Trend Micro s cott_miao@trend.tw @ takeshi.miao](https://reader035.vdocuments.site/reader035/viewer/2022081419/568163c7550346895dd4fb71/html5/thumbnails/7.jpg)
2013 Trend Micro 25th Anniversary 7
![Page 8: Scott Miao, Trend Micro s cott_miao@trend.tw @ takeshi.miao](https://reader035.vdocuments.site/reader035/viewer/2022081419/568163c7550346895dd4fb71/html5/thumbnails/8.jpg)
2013 Trend Micro 25th Anniversary
![Page 9: Scott Miao, Trend Micro s cott_miao@trend.tw @ takeshi.miao](https://reader035.vdocuments.site/reader035/viewer/2022081419/568163c7550346895dd4fb71/html5/thumbnails/9.jpg)
2013 Trend Micro 25th Anniversary
Threat Connect
• A Web Service for Threat Information Report– RESTful Interface to access– Integrated with TM Deep Discovery products
• Relevant and Actionable Intelligence
![Page 10: Scott Miao, Trend Micro s cott_miao@trend.tw @ takeshi.miao](https://reader035.vdocuments.site/reader035/viewer/2022081419/568163c7550346895dd4fb71/html5/thumbnails/10.jpg)
2013 Trend Micro 25th Anniversary
![Page 11: Scott Miao, Trend Micro s cott_miao@trend.tw @ takeshi.miao](https://reader035.vdocuments.site/reader035/viewer/2022081419/568163c7550346895dd4fb71/html5/thumbnails/11.jpg)
2013 Trend Micro 25th Anniversary
IP, domain, URL, filename, process, file hash, Virus detection, registry key, etc.
Product 1 Product 2 Product 3 …
Threat Conne
ct
Sand-box File
Detection
Threat
Web
Web Reputatio
nFamil
y Write-up
TE
Virus DB
APT KB
Most relevant threat report with actionable
intelligenceon a single portal
Process and correlates different data sources
![Page 12: Scott Miao, Trend Micro s cott_miao@trend.tw @ takeshi.miao](https://reader035.vdocuments.site/reader035/viewer/2022081419/568163c7550346895dd4fb71/html5/thumbnails/12.jpg)
2013 Trend Micro 25th Anniversary
CHALLENGES AND SOLUTIONS
![Page 13: Scott Miao, Trend Micro s cott_miao@trend.tw @ takeshi.miao](https://reader035.vdocuments.site/reader035/viewer/2022081419/568163c7550346895dd4fb71/html5/thumbnails/13.jpg)
2013 Trend Micro 25th Anniversary
StoringReal Time AccessPick Your right tool
Big DataMoving
Process & CorrelateGraph Problem
![Page 14: Scott Miao, Trend Micro s cott_miao@trend.tw @ takeshi.miao](https://reader035.vdocuments.site/reader035/viewer/2022081419/568163c7550346895dd4fb71/html5/thumbnails/14.jpg)
2013 Trend Micro 25th Anniversary
MOVING
![Page 15: Scott Miao, Trend Micro s cott_miao@trend.tw @ takeshi.miao](https://reader035.vdocuments.site/reader035/viewer/2022081419/568163c7550346895dd4fb71/html5/thumbnails/15.jpg)
2013 Trend Micro 25th Anniversary
Hadoop
Event Logs
FBSFBS
FBS
Feed Back log ServiceDear users/services
Accumulate small files
![Page 16: Scott Miao, Trend Micro s cott_miao@trend.tw @ takeshi.miao](https://reader035.vdocuments.site/reader035/viewer/2022081419/568163c7550346895dd4fb71/html5/thumbnails/16.jpg)
2013 Trend Micro 25th Anniversary
STORING
![Page 17: Scott Miao, Trend Micro s cott_miao@trend.tw @ takeshi.miao](https://reader035.vdocuments.site/reader035/viewer/2022081419/568163c7550346895dd4fb71/html5/thumbnails/17.jpg)
2013 Trend Micro 25th Anniversary
Cost
Easy Process
Archive
HDFS
![Page 18: Scott Miao, Trend Micro s cott_miao@trend.tw @ takeshi.miao](https://reader035.vdocuments.site/reader035/viewer/2022081419/568163c7550346895dd4fb71/html5/thumbnails/18.jpg)
2013 Trend Micro 25th Anniversary
PROCESS & CORRELATE
![Page 19: Scott Miao, Trend Micro s cott_miao@trend.tw @ takeshi.miao](https://reader035.vdocuments.site/reader035/viewer/2022081419/568163c7550346895dd4fb71/html5/thumbnails/19.jpg)
2013 Trend Micro 25th Anniversary
Pig/MR
• UDFs• MRs for special
cases
Store
• HDFS• Hbase• Solr• RDB
Time
• Batch• Performance
![Page 20: Scott Miao, Trend Micro s cott_miao@trend.tw @ takeshi.miao](https://reader035.vdocuments.site/reader035/viewer/2022081419/568163c7550346895dd4fb71/html5/thumbnails/20.jpg)
2013 Trend Micro 25th Anniversary
REAL TIME ACCESS
![Page 21: Scott Miao, Trend Micro s cott_miao@trend.tw @ takeshi.miao](https://reader035.vdocuments.site/reader035/viewer/2022081419/568163c7550346895dd4fb71/html5/thumbnails/21.jpg)
2013 Trend Micro 25th Anniversary
Real Time
Access
Free form
search
Random Access
Solr Cloud
HBase
• EX. Sandbox Reports
EX. Threat Detection DBs
![Page 22: Scott Miao, Trend Micro s cott_miao@trend.tw @ takeshi.miao](https://reader035.vdocuments.site/reader035/viewer/2022081419/568163c7550346895dd4fb71/html5/thumbnails/22.jpg)
2013 Trend Micro 25th Anniversary
GRAPH MODEL
![Page 23: Scott Miao, Trend Micro s cott_miao@trend.tw @ takeshi.miao](https://reader035.vdocuments.site/reader035/viewer/2022081419/568163c7550346895dd4fb71/html5/thumbnails/23.jpg)
2013 Trend Micro 25th Anniversary
Massive scalable ?
Active community ?
Analyzable ?
![Page 24: Scott Miao, Trend Micro s cott_miao@trend.tw @ takeshi.miao](https://reader035.vdocuments.site/reader035/viewer/2022081419/568163c7550346895dd4fb71/html5/thumbnails/24.jpg)
2013 Trend Micro 25th Anniversary
• We use HBase as a Graph Storage– Google BigTable and PageRank– HBaseCon2012
![Page 25: Scott Miao, Trend Micro s cott_miao@trend.tw @ takeshi.miao](https://reader035.vdocuments.site/reader035/viewer/2022081419/568163c7550346895dd4fb71/html5/thumbnails/25.jpg)
2013 Trend Micro 25th Anniversary
HGraph
Schema Design
Blueprints API
Graph Analysis MRs
https://github.com/tinkerpop/blueprints/wiki
![Page 26: Scott Miao, Trend Micro s cott_miao@trend.tw @ takeshi.miao](https://reader035.vdocuments.site/reader035/viewer/2022081419/568163c7550346895dd4fb71/html5/thumbnails/26.jpg)
2013 Trend Micro 25th Anniversary
PICK RIGHT TOOL
![Page 27: Scott Miao, Trend Micro s cott_miao@trend.tw @ takeshi.miao](https://reader035.vdocuments.site/reader035/viewer/2022081419/568163c7550346895dd4fb71/html5/thumbnails/27.jpg)
2013 Trend Micro 25th Anniversary
Pick right tool for right usecases
• Silver bullet ?• No one project fits all• One problem may has several choices
http://www.neevtech.com/blog/2013/03/18/hadoop-ecosystem-at-a-glance/
![Page 28: Scott Miao, Trend Micro s cott_miao@trend.tw @ takeshi.miao](https://reader035.vdocuments.site/reader035/viewer/2022081419/568163c7550346895dd4fb71/html5/thumbnails/28.jpg)
2013 Trend Micro 25th Anniversary
SUMMARY
![Page 29: Scott Miao, Trend Micro s cott_miao@trend.tw @ takeshi.miao](https://reader035.vdocuments.site/reader035/viewer/2022081419/568163c7550346895dd4fb71/html5/thumbnails/29.jpg)
2013 Trend Micro 25th Anniversary
Small files
• Namenode fsimage would explore the memory
• Too many map tasks to run for a job
FBSFBSFBS
![Page 30: Scott Miao, Trend Micro s cott_miao@trend.tw @ takeshi.miao](https://reader035.vdocuments.site/reader035/viewer/2022081419/568163c7550346895dd4fb71/html5/thumbnails/30.jpg)
2013 Trend Micro 25th Anniversary
Store your data anyway
• Store all the raw data on the HDFS– Break invisible isolation from different data
sources• Archive your data with deduced easy to use
FileFormat– Trenvi, RC file, ORC file
![Page 31: Scott Miao, Trend Micro s cott_miao@trend.tw @ takeshi.miao](https://reader035.vdocuments.site/reader035/viewer/2022081419/568163c7550346895dd4fb71/html5/thumbnails/31.jpg)
2013 Trend Micro 25th Anniversary
Know MR more
• Even you are the pig developer– Deal with MR issues– Write better pig-latin– Sometimes you can only use MR
![Page 32: Scott Miao, Trend Micro s cott_miao@trend.tw @ takeshi.miao](https://reader035.vdocuments.site/reader035/viewer/2022081419/568163c7550346895dd4fb71/html5/thumbnails/32.jpg)
2013 Trend Micro 25th Anniversary
Know your data & usecases
• Realtime ? Batch ?• Access Pattern ?• Therefore, you can pick right tool
![Page 33: Scott Miao, Trend Micro s cott_miao@trend.tw @ takeshi.miao](https://reader035.vdocuments.site/reader035/viewer/2022081419/568163c7550346895dd4fb71/html5/thumbnails/33.jpg)
2013 Trend Micro 25th Anniversary
THANK YOU GUYS