How KKBOX use mrjob to link python, hadoop, aws

Download How KKBOX use mrjob to link python, hadoop, aws

Post on 19-Aug-2014

526 views

Category:

Engineering

4 download

DESCRIPTION

How KKBOX use mrjob to link python, hadoop, aws

TRANSCRIPT

Aaronlin KKBOX mrjob Python, hadoop, aws About KKBOX! About KKBOX! Aaron Lin" " aaronlin@kkbox.com" http://about.me/aaron.yclin" KKBOX About me! " " " MORE THAN 10 MILLION USERS MORE THAN 10 MILLION SONGS Need to use map-reduce to perform experiments" map-reduce: map sort reduce" ! What is mrjob" Open source project founded by Yelp" https://github.com/Yelp/mrjob" Docs: https://pythonhosted.org/mrjob/" A python library for writing map-reduce job" Can cooperate with hadoop cluster and AWS very easily" mrjob?! Why python?" Because of we love python" Why AWS Elastic MapReduce (EMR)?" if hadoop cluster has no resources left, use EMR" If hadoop cluster cannot nish the job in time, use EMR" mrjob can audit your expense and effectiveness of each job" mrjob?! Three steps" Dene your question into map-reduce" Write your mapper(s)" Write your reducer(s)" Thats it!" First mrjob program! First mrjob program! mrjob can run in three ways" Locally" Hadoop" AWS EMR" First mrjob program! Either way works" python wordcount news.txt cat news.txt | python wordcount.py cat news.txt | python wordcount.py --mapper | sort | python wordcount.py --reducer Run mrjob locally! Easy to test since mapper/reducer can be run individually" cat news.txt | python wordcount.py --mapper cat news.txt | python wordcount.py --mapper | sort | python wordcount.py --reducer Good for Development" Run mrjob locally! Write .mrjob.conf in HOME folder" Run mrjob in EMR! Instance type of each group! task c3.2xlarge c3.2xlarge m1.small Use -r to specify the runner" python wordcount.py -r emr news.txt python wordcount.py -r emr s3://xxxx/news.txt Run mrjob in EMR! Run mrjob in EMR! How to audit emr usage" mrjob audit-emr-usage If you have ValueError due to mismatched datetime format" Fix it in /audit_usage.py Run mrjob in EMR! Write a cool program to compute it" But we dont know which AWS instance type is the best ! http://docs.aws.amazon.com/ElasticMapReduce/latest /DeveloperGuide/emr-plan-instances.html" If you check the ofcial document! I like brute force! Memory optimized Compute optimized General purpose For instances with Similar Cost and same number of vCPU, Current generation instance is better" Focus on compute optimized instance! For instances with Similar Cost and same number of vCPU, Current generation instance is better" Focus on compute optimized instance! Conguration of number of mapper/reducer is different" Focus on compute optimized instance! Conguration of number of mapper/reducer is different" Focus on compute optimized instance! Evaluation is specic to this task" Brute force search is too lazy" Cost about 1500 NTD per run" Hadoop/AWS is a buzz word" The money you spend is real" Buying some low-cost computers is always an option" Conclusion! Mrjob" https://github.com/Yelp/mrjob" Docs: https://pythonhosted.org/mrjob/" Hardware spec of each instance type" http://aws.amazon.com/ec2/instance-types/" http://aws.amazon.com/ec2/previous-generation/" Number of mapper/reducer of instance type" http://docs.aws.amazon.com/ElasticMapReduce/latest /DeveloperGuide/TaskConguration_H1.0.3.html" Reference! Slides and script" https://github.com/KKBOX/coscup.tw.2014" Reference! z We are hiring! h,p://www.kkbox.com/jobs/