oct 2012 hug: hadoop .next (0.23) - customer impact and deployment
DESCRIPTION
Hadoop 0.23 contains major architectural changes in both HDFS and Map-Reduce frameworks. The fundamental changes include HDFS (Hadoop Distributed File System) Federation and YARN (Yet Another Resource Negotiator) to overcome the current scalability limitations of both HDFS and Job Tracker. Despite major architectural changes, the impact on user applications and programming model has been kept to a minimal to ensure that existing user Hadoop applications written in Hadoop 20 will continue to function with minimal changes. In this talk we will discuss the architectural changes which Hadoop 23 introduces and compare it to Hadoop 20. Since this is the biggest major release of Hadoop that has been adopted at Yahoo! (after Hadoop 20) in 3 years, we will talk about the customer impact and potential deployment issues of Hadoop 23 and its ecosystems. The deployment of Hadoop 23 at Yahoo! is an ongoing process and is being conducted in a phased manner on our clusters. Presenter: Viraj Bhat, Principal Engineer, Yahoo!TRANSCRIPT
![Page 1: Oct 2012 HUG: Hadoop .Next (0.23) - Customer Impact and Deployment](https://reader036.vdocuments.site/reader036/viewer/2022062511/54c66e674a7959e6368b46a5/html5/thumbnails/1.jpg)
Hadoop 23 (dotNext): Experiences,
Customer Impact & Deployment
Hadoop User Group Sunnyvale Meet up – 17 October 2012Viraj Bhat: [email protected]
![Page 2: Oct 2012 HUG: Hadoop .Next (0.23) - Customer Impact and Deployment](https://reader036.vdocuments.site/reader036/viewer/2022062511/54c66e674a7959e6368b46a5/html5/thumbnails/2.jpg)
- 2 -
About Me• Principal Engg in the Yahoo! Grid Team since May 2008• PhD from Rutgers University, NJ
– Specialization in Data Streaming, Grid, Autonomic Computing
• Worked on streaming data from live simulations executing in NERSC (CA), ORNL (TN) to Princeton Plasma Physics Lab (PPPL - NJ)– Library introduce less then 5% overhead on computation
• PhD Thesis on In-Transit data processing for peta-scale simulation workflows
• Developed CorbaCoG kit for Globus• Active contributor to Hadoop Apache, Pig, HCat and
developer of Hadoop Vaidya
![Page 3: Oct 2012 HUG: Hadoop .Next (0.23) - Customer Impact and Deployment](https://reader036.vdocuments.site/reader036/viewer/2022062511/54c66e674a7959e6368b46a5/html5/thumbnails/3.jpg)
- 3 -
Agenda
• Overview and Introduction• YARN• Federation• Hadoop 23 Experiences
![Page 4: Oct 2012 HUG: Hadoop .Next (0.23) - Customer Impact and Deployment](https://reader036.vdocuments.site/reader036/viewer/2022062511/54c66e674a7959e6368b46a5/html5/thumbnails/4.jpg)
- 4 -
Hadoop Technology Stack at Yahoo!
• HDFS – Distributed File System
• Map/Reduce – Data Processing Paradigm
• HBase and HFile – columnar storage
• PIG – Data Processing Language
• HIVE – SQL like query processing language
• HCatalog – Table abstraction on top of big data allows interaction with Pig and Hive
• Oozie – Workflow Management System
44
HDFS
File Format (HFile)
HBase
Map Reduce
Hive PIG
HCatalog
Oozie
![Page 5: Oct 2012 HUG: Hadoop .Next (0.23) - Customer Impact and Deployment](https://reader036.vdocuments.site/reader036/viewer/2022062511/54c66e674a7959e6368b46a5/html5/thumbnails/5.jpg)
- 5 -
Hadoop 0.23 (dotNext) Highlights
• Major Hadoop release adopted by Yahoo! in over 2 years (after Hadoop 0.20)– Built and stabilized by the Yahoo! Champaign Hadoop
team
• Primary focus is scalability– YARN aka MRv2 – Job run reliability
• Agility & Evolution
– HDFS Federation – larger namespace & scalability• Larger aggregated namespace• Helps for better storage consolidation in Yahoo!• Undergoing customer testing
• Hadoop 23 release does not target availability• Addressed in Hadoop 2.0 and beyond
![Page 6: Oct 2012 HUG: Hadoop .Next (0.23) - Customer Impact and Deployment](https://reader036.vdocuments.site/reader036/viewer/2022062511/54c66e674a7959e6368b46a5/html5/thumbnails/6.jpg)
- 6 -
Hadoop 23 Story at Yahoo!• Extra effort was taken in Yahoo! to certify
applications with Hadoop 23• Sufficient time was provided for users to test
their applications in Hadoop 23• Users are encouraged to get accounts to test if
their applications run on a sandbox cluster which has Hadoop 23 installed
• Roll Out Plan – In Progress– Q4-2012 through Q1 2013 Hadoop 23 will be
installed in a phased manner on 50k nodes at Yahoo!
– 3 Large Customer Grids were successfully upgraded to Hadoop 23
![Page 7: Oct 2012 HUG: Hadoop .Next (0.23) - Customer Impact and Deployment](https://reader036.vdocuments.site/reader036/viewer/2022062511/54c66e674a7959e6368b46a5/html5/thumbnails/7.jpg)
- 7 -
YET ANOTHER RESOURCE NEGOTIATOR (YARN)
NEXT GENERATION OF HADOOP MAP-REDUCE
![Page 8: Oct 2012 HUG: Hadoop .Next (0.23) - Customer Impact and Deployment](https://reader036.vdocuments.site/reader036/viewer/2022062511/54c66e674a7959e6368b46a5/html5/thumbnails/8.jpg)
- 8 -
Hadoop MapReduce in Hadoop 1.0.2
• JobTracker– Manages cluster resources
and job scheduling
• TaskTracker– Per-node agent– Manage tasks
![Page 9: Oct 2012 HUG: Hadoop .Next (0.23) - Customer Impact and Deployment](https://reader036.vdocuments.site/reader036/viewer/2022062511/54c66e674a7959e6368b46a5/html5/thumbnails/9.jpg)
- 9 -
Paradigm shift with Hadoop 23
• Split up the two major functions of JobTracker– Cluster resource management– Application life-cycle management
• MapReduce becomes user-land library
![Page 10: Oct 2012 HUG: Hadoop .Next (0.23) - Customer Impact and Deployment](https://reader036.vdocuments.site/reader036/viewer/2022062511/54c66e674a7959e6368b46a5/html5/thumbnails/10.jpg)
- 10 -
Components of YARN
• Resource Manager– Global resource scheduler– Hierarchical queues
• Node Manager– Per-machine agent– Manages the life-cycle of container– Container resource monitoring
• Application Master– Per-application– Manages application scheduling and task execution
![Page 11: Oct 2012 HUG: Hadoop .Next (0.23) - Customer Impact and Deployment](https://reader036.vdocuments.site/reader036/viewer/2022062511/54c66e674a7959e6368b46a5/html5/thumbnails/11.jpg)
- 11 -
Architecture of YARN
![Page 12: Oct 2012 HUG: Hadoop .Next (0.23) - Customer Impact and Deployment](https://reader036.vdocuments.site/reader036/viewer/2022062511/54c66e674a7959e6368b46a5/html5/thumbnails/12.jpg)
- 12 -
Architecture of YARN
![Page 13: Oct 2012 HUG: Hadoop .Next (0.23) - Customer Impact and Deployment](https://reader036.vdocuments.site/reader036/viewer/2022062511/54c66e674a7959e6368b46a5/html5/thumbnails/13.jpg)
- 13 -
Architecture of YARN
![Page 14: Oct 2012 HUG: Hadoop .Next (0.23) - Customer Impact and Deployment](https://reader036.vdocuments.site/reader036/viewer/2022062511/54c66e674a7959e6368b46a5/html5/thumbnails/14.jpg)
- 14 -
• Scalable– Largest YARN cluster in the world built at Yahoo! running
on (Hadoop 0.23.3), with no scalability issues so far – Ran tests to validate that YARN should scale to 10,000
nodes.
• Surprisingly Stable• Web Services• Better Utilization of Resources at Yahoo!
– No fixed partitioning between Map and Reduce Tasks– Latency from resource available to resource re-assigned
is far better than 1.x in big clusters
Experiences of YARN – High Points
![Page 15: Oct 2012 HUG: Hadoop .Next (0.23) - Customer Impact and Deployment](https://reader036.vdocuments.site/reader036/viewer/2022062511/54c66e674a7959e6368b46a5/html5/thumbnails/15.jpg)
- 15 -
• HDFS
– Read (Throughput 5.37% higher)
• MapReduce
– Sort (Runtime 4.59% smaller, Throughput 3.98%
higher)
– Shuffle (Shuffle Time 13.25% smaller)
– Gridmix (Runtime 5.29% smaller)– Small Jobs – Uber AM (Word Count 3.5x faster,
27.7x fewer resources)
Performance (0.23.3 vs. 1.0.2)
![Page 16: Oct 2012 HUG: Hadoop .Next (0.23) - Customer Impact and Deployment](https://reader036.vdocuments.site/reader036/viewer/2022062511/54c66e674a7959e6368b46a5/html5/thumbnails/16.jpg)
- 16 -
• MPI (www.open-mpi.org nightly snapshot)• Machine Learning (Spark)• Real-time Streaming (S4 and Storm coming
soon) • Graph Processing (GIRAPH-13 coming soon)
Synergy with new Compute Paradigms
![Page 17: Oct 2012 HUG: Hadoop .Next (0.23) - Customer Impact and Deployment](https://reader036.vdocuments.site/reader036/viewer/2022062511/54c66e674a7959e6368b46a5/html5/thumbnails/17.jpg)
- 17 -
• Oozie on YARN can have potential deadlocks (MAPREDUCE-4304)– UberAM can mitigate this
• Some UI scalability issues (YARN-151, MAPREDUCE-4720)– Some pages download very large tables and paginate in
JavaScript
• Minor incompatibilities in the distributed cache• No generic history server (MAPREDUCE-3061)• AM failures hard to debug (MAPREDUCE-4428,
MAPREDUCE-3688)
The Not So Good
![Page 18: Oct 2012 HUG: Hadoop .Next (0.23) - Customer Impact and Deployment](https://reader036.vdocuments.site/reader036/viewer/2022062511/54c66e674a7959e6368b46a5/html5/thumbnails/18.jpg)
- 18 -
HADOOP 23 FEATURESHDFS FEDERATION
![Page 19: Oct 2012 HUG: Hadoop .Next (0.23) - Customer Impact and Deployment](https://reader036.vdocuments.site/reader036/viewer/2022062511/54c66e674a7959e6368b46a5/html5/thumbnails/19.jpg)
- 19 -
Non Federated HDFS Architecture
![Page 20: Oct 2012 HUG: Hadoop .Next (0.23) - Customer Impact and Deployment](https://reader036.vdocuments.site/reader036/viewer/2022062511/54c66e674a7959e6368b46a5/html5/thumbnails/20.jpg)
- 20 -
Non Federated HDFS Architecture
• Single Namespace Volume
– Namespace Volume = Namespace
+ Blocks
• Single namenode with a
namespace
– Entire namespace is in memory
– Provides Block Management
• Datanodes store block replicas
– Block files stored on local file
system
Blo
ck S
tora
geN
am
esp
ace
Namenode
Block Management
NS
Storage
Datanode Datanode
NS
![Page 21: Oct 2012 HUG: Hadoop .Next (0.23) - Customer Impact and Deployment](https://reader036.vdocuments.site/reader036/viewer/2022062511/54c66e674a7959e6368b46a5/html5/thumbnails/21.jpg)
- 21 -
Limitation - Single Namespace• Scalability
– Storage scales horizontally - namespace doesn’t – Limited number of files, dirs and blocks
• 250 million files and blocks at 64GB Namenode heap size
• Performance– File system operations throughput limited by a single node
• 120K read ops/sec and 6000 write ops/sec• Poor Isolation
– All the tenants share a single namespace• Separate volume for tenants is not possible
– Lacks separate namespace for different categories of applications
• Experimental apps can affect production apps• Example - HBase could use its own namespace
• Isolation is problem, even in a small cluster
![Page 22: Oct 2012 HUG: Hadoop .Next (0.23) - Customer Impact and Deployment](https://reader036.vdocuments.site/reader036/viewer/2022062511/54c66e674a7959e6368b46a5/html5/thumbnails/22.jpg)
- 22 -
HDFS Federation
• An administrative/operational feature for better managing resources required at Yahoo!
• Multiple independent Namenodes and Namespace Volumes in a cluster› Namespace Volume = Namespace + Block Pool
• Block Storage as generic storage service› Set of blocks for a Namespace Volume is called a Block Pool
› DNs store blocks for all the Namespace Volumes – no partitioning
Datanode 1 Datanode 2 Datanode m... ... ...
NS1Foreign NS
n
... ... NS k
Block Pools
Pool nPool kPool 1
NN-1 NN-k
NN-n
Common Storage
Blo
ck S
tora
ge
Nam
esp
ace
![Page 23: Oct 2012 HUG: Hadoop .Next (0.23) - Customer Impact and Deployment](https://reader036.vdocuments.site/reader036/viewer/2022062511/54c66e674a7959e6368b46a5/html5/thumbnails/23.jpg)
- 23 -
Managing Namespaces• Federation has multiple
namespaces • Client-side implementation of
mount tables– No single point of failure– No hotspot for root and top level
directories
• Applications using Federation should use the viewfs:// schema– The viewfs:// URI schema can be
used as the default file system replacing the hdfs:// schema
homeproject
NS1 NS3NS2
NS4
tmp
/Client-side mount-table
data
![Page 24: Oct 2012 HUG: Hadoop .Next (0.23) - Customer Impact and Deployment](https://reader036.vdocuments.site/reader036/viewer/2022062511/54c66e674a7959e6368b46a5/html5/thumbnails/24.jpg)
- 24 -
Hadoop 23 Federation • Federation Testing is underway
– Many ecosystems such as Pig have completed testing – Real load testing will only be possible when multiple co-
located Grids transition to Hadoop 23
• Adoption of Federation will allow for better consolidation storage resources– Many data feeds are duplicated across various Grids
![Page 25: Oct 2012 HUG: Hadoop .Next (0.23) - Customer Impact and Deployment](https://reader036.vdocuments.site/reader036/viewer/2022062511/54c66e674a7959e6368b46a5/html5/thumbnails/25.jpg)
- 25 -
HADOOP 23 IMPACT ON END USERS AND ECOSYSTEM DEVELOPERS
![Page 26: Oct 2012 HUG: Hadoop .Next (0.23) - Customer Impact and Deployment](https://reader036.vdocuments.site/reader036/viewer/2022062511/54c66e674a7959e6368b46a5/html5/thumbnails/26.jpg)
- 26 -
Hadoop 23 Command Line
• New environment variables:– $HADOOP_COMMON_HOME – $HADOOP_MAPRED_HOME – $HADOOP_HDFS_HOME
• hadoop command to execute mapred or hdfs sub-commands has been deprecated– Old usage (will work)
– hadoop queue –showacls– hadoop fs -ls– hadoop mapred job -kill <job_id>
– New Usage– mapred queue -showacls – hdfs dfs –ls <path>– mapred job -kill <job_id>
![Page 27: Oct 2012 HUG: Hadoop .Next (0.23) - Customer Impact and Deployment](https://reader036.vdocuments.site/reader036/viewer/2022062511/54c66e674a7959e6368b46a5/html5/thumbnails/27.jpg)
- 27 -
Hadoop 23 Map Reduce
• An application that is using a version of Hadoop 1.0.2 will not work in Hadoop 0.23
• Hadoop 0.23 version is API compatible with Hadoop 0.20.205 and Hadoop 1.0.2– Not binary compatible
• Hadoop Java programs will not require any code change, However users have to recompile with Hadoop 0.23– If code change is required, please let us know.
• Streaming applications should work without modifications
• Hadoop Pipes (using C/C++ interface) application will require a re-compilation with new libraries
![Page 28: Oct 2012 HUG: Hadoop .Next (0.23) - Customer Impact and Deployment](https://reader036.vdocuments.site/reader036/viewer/2022062511/54c66e674a7959e6368b46a5/html5/thumbnails/28.jpg)
- 28 -
Hadoop 23 Compatibility - Pig• Pig versions 0.9.2, 0.10 and beyond will be fully supported on
Hadoop 0.23– Packaging problem: Generating 2 different pig.jar with different
versions of Hadoop
• No Changes in Pig script if it uses relative paths in HDFS
• Changes in pig script is required if HDFS absolute path (hdfs:// ) is used – HDFS Federation part of Hadoop 23 requires the usage of
viewfs:// (HDFS discussion to follow)– Change hdfs:// schema to use viewfs:// schema
• Java UDF’s must be re-compiled with Hadoop 23 compatible jar– Customer Loaders and Storers in Pig are affected
![Page 29: Oct 2012 HUG: Hadoop .Next (0.23) - Customer Impact and Deployment](https://reader036.vdocuments.site/reader036/viewer/2022062511/54c66e674a7959e6368b46a5/html5/thumbnails/29.jpg)
- 29 -
• Oozie 3.1.4 and later versions compatible with Hadoop 23
• No changes in workflow definition or job properties– No need to redeploy the Oozie coordinator jobs
• Java code, streaming, pipes apps need to be recompiled with Hadoop 0.23 jars for binary compatibility
• Existing user workflow and coordinator definition (XML) should continue to work as expected
• Users “responsibility” to package the right Hadoop 23 compatible jars• Hadoop 23 compatible pig.jar needs to be packaged for Pig
action
Hadoop 23 Compatibility - Oozie
![Page 30: Oct 2012 HUG: Hadoop .Next (0.23) - Customer Impact and Deployment](https://reader036.vdocuments.site/reader036/viewer/2022062511/54c66e674a7959e6368b46a5/html5/thumbnails/30.jpg)
- 30 -
Hadoop 23 - Oozie Dev Challenges
• Learning curve for maven builds– Build iterations, local maven staging repo staleness
• Queue configurations, container allocations require revisiting the design
• Many iterations of Hadoop 23 deployment– Overhead to test Oozie compatibility with new release
• Initial deployment of YARN did not have a view of the Application Master (AM) logs– Manual ssh to AM for debugging launcher jobs
![Page 31: Oct 2012 HUG: Hadoop .Next (0.23) - Customer Impact and Deployment](https://reader036.vdocuments.site/reader036/viewer/2022062511/54c66e674a7959e6368b46a5/html5/thumbnails/31.jpg)
- 31 -
• Hive version 0.9 and upwards are fully supported
• Hive SQL/scripts should continue to work without any modification
• Java UDF’s in Hive must be re-compiled with Hadoop 23 compatible hive.jar
Hadoop 23 Compatibility - Hive
![Page 32: Oct 2012 HUG: Hadoop .Next (0.23) - Customer Impact and Deployment](https://reader036.vdocuments.site/reader036/viewer/2022062511/54c66e674a7959e6368b46a5/html5/thumbnails/32.jpg)
- 32 -
Hadoop 23 – Hive Dev Challenges• Deprecation of code in MiniMRCluster that fetches
the stack trace from the JobTracker “no longer” works– Extra amount of time in debugging and rewriting test
cases
• Incompatibility of HDFS commands between Hadoop 1.0.2 and 0.23– -rmr vs. -rm -r – mkdir vs. mkdir –p– Results in fixing tests in new ways or inventing
workarounds so that they run in both Hadoop 1.0.2 and Hadoop 0.23
• As Hive uses MapRed API’s; more work required for certification – Would be good to move to MapReduce API’s (for
example: Pig)
![Page 33: Oct 2012 HUG: Hadoop .Next (0.23) - Customer Impact and Deployment](https://reader036.vdocuments.site/reader036/viewer/2022062511/54c66e674a7959e6368b46a5/html5/thumbnails/33.jpg)
- 33 -
• HCat 0.4 and upwards version is certified to work with Hadoop 23
Hadoop 23 - HCat
![Page 34: Oct 2012 HUG: Hadoop .Next (0.23) - Customer Impact and Deployment](https://reader036.vdocuments.site/reader036/viewer/2022062511/54c66e674a7959e6368b46a5/html5/thumbnails/34.jpg)
- 34 -
Hadoop 23 Job History Log Format
• History API & Log format have changed– Affects all applications and tools that directly use
Hadoop History API– Logs stored as Avro serialization in JSon format
• Affected many tools which rely on Job Logs– Hadoop Vaidya – had to be rewritten with the new
JobHistoryParser
![Page 35: Oct 2012 HUG: Hadoop .Next (0.23) - Customer Impact and Deployment](https://reader036.vdocuments.site/reader036/viewer/2022062511/54c66e674a7959e6368b46a5/html5/thumbnails/35.jpg)
- 35 -
Hadoop 23 Queues• Hadoop 23 has support for Hierarchical Queues
– In Yahoo! it has been configured as a flat queue to limit customer disruption
– Customer testing is being conducted
![Page 36: Oct 2012 HUG: Hadoop .Next (0.23) - Customer Impact and Deployment](https://reader036.vdocuments.site/reader036/viewer/2022062511/54c66e674a7959e6368b46a5/html5/thumbnails/36.jpg)
- 36 -
32/64 bit JDK 1.7• Currently certifying Hadoop 23 and its ecosystems on
32 bit 1.7 JDK
• 64 bit 1.7 JDK certification for Hadoop and its ecosystems would be taken up in Q1 2013
![Page 37: Oct 2012 HUG: Hadoop .Next (0.23) - Customer Impact and Deployment](https://reader036.vdocuments.site/reader036/viewer/2022062511/54c66e674a7959e6368b46a5/html5/thumbnails/37.jpg)
- 37 -
Hadoop 23 Operations and Services
• Grid Operations at Yahoo! transitioned Hadoop 1.0.2 Namenode to Hadoop 23 smoothly– No data was lost
• Matched the container configurations on Hadoop 23 clusters with the old Map Reduce slots– Map Reduce slots were configured based on memory
hence transition was smooth
• Scheduling, planning and migration of Hadoop 1.0.2 applications to Hadoop 23 for about 100+ customers was a major task for solutions– Many issues were caught in the last minute needed
emergency fixes (globbing, pig.jar packaging, change in mkdir command )
– Hadoop 0.23.4 build planned
![Page 38: Oct 2012 HUG: Hadoop .Next (0.23) - Customer Impact and Deployment](https://reader036.vdocuments.site/reader036/viewer/2022062511/54c66e674a7959e6368b46a5/html5/thumbnails/38.jpg)
- 38 -
Acknowledgements• YARN – Robert Evans, Thomas Graves, Jason Lowe• Pig - Rohini Paliniswamy• Hive and HCatalog – Chris Drome• Oozie – Mona Chitnis and Mohammad Islam• Services and Operations – Rajiv Chittajallu and
Kimsukh Kundu
![Page 39: Oct 2012 HUG: Hadoop .Next (0.23) - Customer Impact and Deployment](https://reader036.vdocuments.site/reader036/viewer/2022062511/54c66e674a7959e6368b46a5/html5/thumbnails/39.jpg)
- 39 -
References• 0.23 Documentation
– http://people.apache.org/~acmurthy/hadoop-0.23/
• 0.23 Release Notes– http://people.apache.org/~acmurthy/hadoop-0.23/hado
op-project-dist/hadoop-common/releasenotes.html
• YARN Documentation– http://people.apache.org/~acmurthy/hadoop-0.23/hado
op-yarn/hadoop-yarn-site/YARN.html
• HDFS Federation Documentation – http://people.apache.org/~acmurthy/hadoop-0.23/hado
op-yarn/hadoop-yarn-site/Federation.html