oozie high availability (hadoop summit 2014 meetup)
DESCRIPTION
by Robert Kanter (Cloudera)TRANSCRIPT
![Page 1: Oozie High Availability (Hadoop Summit 2014 meetup)](https://reader037.vdocuments.site/reader037/viewer/2022110114/545676e5af7959b8078b46bf/html5/thumbnails/1.jpg)
1
Oozie High Availability (HA)Robert Kanter
![Page 2: Oozie High Availability (Hadoop Summit 2014 meetup)](https://reader037.vdocuments.site/reader037/viewer/2022110114/545676e5af7959b8078b46bf/html5/thumbnails/2.jpg)
2
High Availability
• A system without non-planned downtime when partial failures occur• Typically achieved by having redundancies and removing
single-points of failure
• Our Goals• Don’t change the API or usage patterns• User doesn’t even have to know its HA
![Page 3: Oozie High Availability (Hadoop Summit 2014 meetup)](https://reader037.vdocuments.site/reader037/viewer/2022110114/545676e5af7959b8078b46bf/html5/thumbnails/3.jpg)
3
The HA SolutionArchitectural Overview
![Page 4: Oozie High Availability (Hadoop Summit 2014 meetup)](https://reader037.vdocuments.site/reader037/viewer/2022110114/545676e5af7959b8078b46bf/html5/thumbnails/4.jpg)
4
The HA Solution: Database
• Oozie stores all state in a database• (submitted jobs, workflow definitions, etc)
• Instead of a failover model, we want to run many Oozie servers against the same database• Active-Active HA• Also provides horizontal scalability
• ZooKeeper for coordination
![Page 5: Oozie High Availability (Hadoop Summit 2014 meetup)](https://reader037.vdocuments.site/reader037/viewer/2022110114/545676e5af7959b8078b46bf/html5/thumbnails/5.jpg)
5
The HA Solution: Database
![Page 6: Oozie High Availability (Hadoop Summit 2014 meetup)](https://reader037.vdocuments.site/reader037/viewer/2022110114/545676e5af7959b8078b46bf/html5/thumbnails/6.jpg)
6
The HA Solution: Access
• Users and client programs need a single address to connect (Web UI, REST/Java API, JobTracker callbacks, etc)
• Load Balancer, Virtual IP, or DNS round-robin can be used to provide a single entry point to the Oozie servers• Technically also needs to be HA
![Page 7: Oozie High Availability (Hadoop Summit 2014 meetup)](https://reader037.vdocuments.site/reader037/viewer/2022110114/545676e5af7959b8078b46bf/html5/thumbnails/7.jpg)
7
The HA Solution: Access
![Page 8: Oozie High Availability (Hadoop Summit 2014 meetup)](https://reader037.vdocuments.site/reader037/viewer/2022110114/545676e5af7959b8078b46bf/html5/thumbnails/8.jpg)
8
The HA Solution: Log Streaming
• Oozie’s log files are not in the database• Each Oozie Server only has access to its own logs
• Jobs are not assigned to a specific Oozie server
• What if Oozie Server A wants to get logs for a job processed by Oozie Server B?• Oozie Server A can ask Oozie Server B for its logs
• Caveat: If an Oozie Server goes down, any logs from it will be unavailable until it is brought back up
![Page 9: Oozie High Availability (Hadoop Summit 2014 meetup)](https://reader037.vdocuments.site/reader037/viewer/2022110114/545676e5af7959b8078b46bf/html5/thumbnails/9.jpg)
9
The HA Solution: Log Streaming
![Page 10: Oozie High Availability (Hadoop Summit 2014 meetup)](https://reader037.vdocuments.site/reader037/viewer/2022110114/545676e5af7959b8078b46bf/html5/thumbnails/10.jpg)
10
How to Enable HAConfiguration and Security
![Page 11: Oozie High Availability (Hadoop Summit 2014 meetup)](https://reader037.vdocuments.site/reader037/viewer/2022110114/545676e5af7959b8078b46bf/html5/thumbnails/11.jpg)
11
How to Enable HA
• Setup Load balancer, ZooKeeper ensemble, HA database, and multiple identically configured Oozie servers
• Enable Oozie HA services:<property> <name>oozie.services.ext</name> <value> org.apache.oozie.service.ZKLocksService, org.apache.oozie.service.ZKXLogStreamingService, org.apache.oozie.service.ZKJobsConcurrencyService </value></property>
![Page 12: Oozie High Availability (Hadoop Summit 2014 meetup)](https://reader037.vdocuments.site/reader037/viewer/2022110114/545676e5af7959b8078b46bf/html5/thumbnails/12.jpg)
12
How to Enable HA
• Point Oozie to ZooKeeper Ensemble:<property> <name>oozie.zookeeper.connection.string</name> <value>ZK_HOST1:2181,ZK_HOST2:2181</value></property>
• Point environment variable for callbacks to load balancer:
export OOZIE_BASE_URL="http://loadbalancer:11000/oozie"
![Page 13: Oozie High Availability (Hadoop Summit 2014 meetup)](https://reader037.vdocuments.site/reader037/viewer/2022110114/545676e5af7959b8078b46bf/html5/thumbnails/13.jpg)
13
How to Enable HA: Security
• Extra step to configure Kerberos with Load Balancer:<property> <name> oozie.authentication.kerberos.principal </name> <value>HTTP/loadbalancer@REALM</value></property>
• Note: this currently prevents clients from talking directly to any Oozie server
![Page 14: Oozie High Availability (Hadoop Summit 2014 meetup)](https://reader037.vdocuments.site/reader037/viewer/2022110114/545676e5af7959b8078b46bf/html5/thumbnails/14.jpg)
14
How to Enable HA: Security
• Enable Kerberos connection to ZooKeeper and ACLs:<property> <name>oozie.zookeeper.secure</name> <value>true</value></property>
• ACLs prevent malicious users or programs from interfering with Oozie’s znodes
![Page 15: Oozie High Availability (Hadoop Summit 2014 meetup)](https://reader037.vdocuments.site/reader037/viewer/2022110114/545676e5af7959b8078b46bf/html5/thumbnails/15.jpg)
15
Using Oozie with HA
![Page 16: Oozie High Availability (Hadoop Summit 2014 meetup)](https://reader037.vdocuments.site/reader037/viewer/2022110114/545676e5af7959b8078b46bf/html5/thumbnails/16.jpg)
16
Using Oozie with HA
• New Oozie CLI/REST API command to list all servers
$ oozie admin -oozie http://loadbalancer:11000/oozie -servershostA : http://hostA:11000/ooziehostB : http://hostB:11000/ooziehostC : http://hostC:11000/oozie
• Log messages now include which server wrote them2013-09-29 16:46:20,182 WARN org.apache.oozie.command.wf.ActionStartXCommand: SERVER[hostA] USER[root] GROUP[-] TOKEN[] APP[demo-wf] JOB[0000000-130925230553293-oozie-oozi-W] ACTION[0000000-130925230553293-oozie-oozi-W@streaming-node] [***0000000-130925230553293-oozie-oozi-W@streaming-node***]Action status=RUNNING
![Page 17: Oozie High Availability (Hadoop Summit 2014 meetup)](https://reader037.vdocuments.site/reader037/viewer/2022110114/545676e5af7959b8078b46bf/html5/thumbnails/17.jpg)
17
To DoWhat’s left
![Page 18: Oozie High Availability (Hadoop Summit 2014 meetup)](https://reader037.vdocuments.site/reader037/viewer/2022110114/545676e5af7959b8078b46bf/html5/thumbnails/18.jpg)
18
To Do
• HA support for SLAs and HCatalog integration• Sharelib Purging with HA• Log Streaming HA• With Kerberos, Oozie servers can’t talk to each other
• Breaks log streaming, sharelibupdate• Other misc improvements
![Page 19: Oozie High Availability (Hadoop Summit 2014 meetup)](https://reader037.vdocuments.site/reader037/viewer/2022110114/545676e5af7959b8078b46bf/html5/thumbnails/19.jpg)
19