streamline hadoop devops with apache ambari
TRANSCRIPT
![Page 1: Streamline Hadoop DevOps with Apache Ambari](https://reader034.vdocuments.site/reader034/viewer/2022042907/588003001a28ab421b8b459b/html5/thumbnails/1.jpg)
Streamline Hadoop DevOps with Apache Ambari
Jayush Luniya
Hadoop Summit, Tokyo
![Page 2: Streamline Hadoop DevOps with Apache Ambari](https://reader034.vdocuments.site/reader034/viewer/2022042907/588003001a28ab421b8b459b/html5/thumbnails/2.jpg)
2 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Speaker
Jayush LuniyaStaff Software Engineer @ HortonworksApache Ambari [email protected]
![Page 3: Streamline Hadoop DevOps with Apache Ambari](https://reader034.vdocuments.site/reader034/viewer/2022042907/588003001a28ab421b8b459b/html5/thumbnails/3.jpg)
3 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Apache Ambari is the open-source platform to provision, manage and monitor Hadoop clusters
![Page 4: Streamline Hadoop DevOps with Apache Ambari](https://reader034.vdocuments.site/reader034/viewer/2022042907/588003001a28ab421b8b459b/html5/thumbnails/4.jpg)
4 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
![Page 5: Streamline Hadoop DevOps with Apache Ambari](https://reader034.vdocuments.site/reader034/viewer/2022042907/588003001a28ab421b8b459b/html5/thumbnails/5.jpg)
5 © Hortonworks Inc. 2011 – 2016. All Rights Reserved 4 years old
![Page 6: Streamline Hadoop DevOps with Apache Ambari](https://reader034.vdocuments.site/reader034/viewer/2022042907/588003001a28ab421b8b459b/html5/thumbnails/6.jpg)
6 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Exciting Enterprise Features in Ambari 2.4
New Services: Log Search, Zeppelin, Hive LLAP
Role Based Access Control
Management Packs
Grafana UI for Ambari Metrics System
New Views: Zeppelin, Storm
![Page 7: Streamline Hadoop DevOps with Apache Ambari](https://reader034.vdocuments.site/reader034/viewer/2022042907/588003001a28ab421b8b459b/html5/thumbnails/7.jpg)
7 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
More in Ambari 2.4
• Alerts: Customizable props and thresholds (AMBARI-14898)
• Alerts: Retry tolerance (AMBARI-15686)
• Alerts: New HDFS Alerts (AMBARI-14800)
• New Host Page Filtering (AMBARI-15210)
• Remove Service from UI (AMBARI-14759)
• Support for SLES 12 (AMBARI-16007)
• Stability: Database Consistency Checking (AMBARI-16258)
• Customizable Ambari Log + PID Dirs (AMBARI-15300)
• New Version Registration Experience (AMBARI-15724)
• Log Search Technical Preview (AMBARI-14927)
• Operational Audit Logging (AMBARI-15241)
• Role-Based Access Control (AMBARI-13977)
• Automated Setup of Ambari Kerberos through Blueprints (AMBARI-15561)
• Automated Setup of Ambari Proxy User (AMBARI-15561)
• Customizable Host Reg. SSH Port (AMBARI-13450)
Core Features Security Features
• View URLs for bookmarks (AMBARI-15821), View Refresh (AMBARI-15682)
• Inherit Cluster Permissions (AMBARI-16177)
• Remote Cluster Registration (AMBARI-16274)
Views Framework Features
![Page 8: Streamline Hadoop DevOps with Apache Ambari](https://reader034.vdocuments.site/reader034/viewer/2022042907/588003001a28ab421b8b459b/html5/thumbnails/8.jpg)
8 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Deploy
Secure/LDAP
Smart Configs
Upgrade
Monitor
Scale, Extend, Analyze
Simply Operations - Lifecycle
Ease-of-Use Deploy
![Page 9: Streamline Hadoop DevOps with Apache Ambari](https://reader034.vdocuments.site/reader034/viewer/2022042907/588003001a28ab421b8b459b/html5/thumbnails/9.jpg)
9 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Deploy On Premise
Ambari UI wizard handles all of these combinations and makes recommendations
based on host specs.
![Page 10: Streamline Hadoop DevOps with Apache Ambari](https://reader034.vdocuments.site/reader034/viewer/2022042907/588003001a28ab421b8b459b/html5/thumbnails/10.jpg)
10 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Deploy On The Cloud
Certified environmentsSysprepped VMsHundreds of similar clusters
![Page 11: Streamline Hadoop DevOps with Apache Ambari](https://reader034.vdocuments.site/reader034/viewer/2022042907/588003001a28ab421b8b459b/html5/thumbnails/11.jpg)
11 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Deploy with Blueprints
Systematic way of defining a cluster
Export existing cluster into blueprint/api/v1/clusters/:clusterName?format=blueprint
Configs
Topology Hosts Cluste
r
![Page 12: Streamline Hadoop DevOps with Apache Ambari](https://reader034.vdocuments.site/reader034/viewer/2022042907/588003001a28ab421b8b459b/html5/thumbnails/12.jpg)
12 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Create a cluster with Blueprints
{ "configurations" : [ { "hdfs-site" : {
"dfs.datanode.data.dir" : "/hadoop/1, /hadoop/2,/hadoop/3" } } ], "host_groups" : [ { "name" : "master-host", "components" : [ { "name" : "NAMENODE” }, { "name" : "RESOURCEMANAGER” }, … ], "cardinality" : "1" }, { "name" : "worker-host", "components" : [ { "name" : "DATANODE" }, { "name" : "NODEMANAGER” }, … ], "cardinality" : "1+" }, ], "Blueprints" : { "stack_name" : "HDP", "stack_version" : "2.5" }}
{ "blueprint" : "my-blueprint", "host_groups" :[ { "name" : "master-host", "hosts" : [ { "fqdn" : "master001.ambari.apache.org"
} ] }, { "name" : "worker-host", "hosts" : [ { "fqdn" : "worker001.ambari.apache.org"
}, { "fqdn" : "worker002.ambari.apache.org"
}, … { "fqdn" : "worker099.ambari.apache.org"
} ] } ]}
1. POST /api/v1/blueprints/my-blueprint
2. POST /api/v1/clusters/my-cluster
![Page 13: Streamline Hadoop DevOps with Apache Ambari](https://reader034.vdocuments.site/reader034/viewer/2022042907/588003001a28ab421b8b459b/html5/thumbnails/13.jpg)
13 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Create a cluster with Blueprints
{ "configurations" : [ { "hdfs-site" : {
"dfs.datanode.data.dir" : "/hadoop/1, /hadoop/2,/hadoop/3" } } ], "host_groups" : [ { "name" : "master-host", "components" : [ { "name" : "NAMENODE” }, { "name" : "RESOURCEMANAGER” }, … ], "cardinality" : "1" }, { "name" : "worker-host", "components" : [ { "name" : "DATANODE" }, { "name" : "NODEMANAGER” }, … ], "cardinality" : "1+" }, ], "Blueprints" : { "stack_name" : "HDP", "stack_version" : "2.5" }}
{ "blueprint" : "my-blueprint", "host_groups" :[ { "name" : "master-host", "hosts" : [ { "fqdn" : "master001.ambari.apache.org"
} ] }, { "name" : "worker-host", "hosts" : [ { "fqdn" : "worker001.ambari.apache.org"
}, { "fqdn" : "worker002.ambari.apache.org"
}, … { "fqdn" : "worker099.ambari.apache.org"
} ] } ]}
1. POST /api/v1/blueprints/my-blueprint
2. POST /api/v1/clusters/my-cluster
![Page 14: Streamline Hadoop DevOps with Apache Ambari](https://reader034.vdocuments.site/reader034/viewer/2022042907/588003001a28ab421b8b459b/html5/thumbnails/14.jpg)
14 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Create a cluster with Blueprints
{ "configurations" : [ { "hdfs-site" : {
"dfs.datanode.data.dir" : "/hadoop/1, /hadoop/2,/hadoop/3" } } ], "host_groups" : [ { "name" : "master-host", "components" : [ { "name" : "NAMENODE” }, { "name" : "RESOURCEMANAGER” }, … ], "cardinality" : "1" }, { "name" : "worker-host", "components" : [ { "name" : "DATANODE" }, { "name" : "NODEMANAGER” }, … ], "cardinality" : "1+" }, ], "Blueprints" : { "stack_name" : "HDP", "stack_version" : "2.5" }}
{ "blueprint" : "my-blueprint", "host_groups" :[ { "name" : "master-host", "hosts" : [ { "fqdn" : "master001.ambari.apache.org"
} ] }, { "name" : "worker-host", "hosts" : [ { "fqdn" : "worker001.ambari.apache.org"
}, { "fqdn" : "worker002.ambari.apache.org"
}, … { "fqdn" : "worker099.ambari.apache.org"
} ] } ]}
1. POST /api/v1/blueprints/my-blueprint
2. POST /api/v1/clusters/my-cluster
![Page 15: Streamline Hadoop DevOps with Apache Ambari](https://reader034.vdocuments.site/reader034/viewer/2022042907/588003001a28ab421b8b459b/html5/thumbnails/15.jpg)
15 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Create a cluster with Blueprints
{ "configurations" : [ { "hdfs-site" : {
"dfs.datanode.data.dir" : "/hadoop/1, /hadoop/2,/hadoop/3" } } ], "host_groups" : [ { "name" : "master-host", "components" : [ { "name" : "NAMENODE” }, { "name" : "RESOURCEMANAGER” }, … ], "cardinality" : "1" }, { "name" : "worker-host", "components" : [ { "name" : "DATANODE" }, { "name" : "NODEMANAGER” }, … ], "cardinality" : "1+" }, ], "Blueprints" : { "stack_name" : "HDP", "stack_version" : "2.5" }}
{ "blueprint" : "my-blueprint", "host_groups" :[ { "name" : "master-host", "hosts" : [ { "fqdn" : "master001.ambari.apache.org"
} ] }, { "name" : "worker-host", "hosts" : [ { "fqdn" : "worker001.ambari.apache.org"
}, { "fqdn" : "worker002.ambari.apache.org"
}, … { "fqdn" : "worker099.ambari.apache.org"
} ] } ]}
1. POST /api/v1/blueprints/my-blueprint
2. POST /api/v1/clusters/my-cluster
![Page 16: Streamline Hadoop DevOps with Apache Ambari](https://reader034.vdocuments.site/reader034/viewer/2022042907/588003001a28ab421b8b459b/html5/thumbnails/16.jpg)
16 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Blueprints for Large Scale
• Kerberos, secure out-of-the-box
• High Availability is setup initially for NameNode, YARN, Hive, Oozie, etc
• Host Discovery allows Ambari to automatically install services for a Host when it comes online
• Stack Advisor recommendations
![Page 17: Streamline Hadoop DevOps with Apache Ambari](https://reader034.vdocuments.site/reader034/viewer/2022042907/588003001a28ab421b8b459b/html5/thumbnails/17.jpg)
17 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
POST /api/v1/clusters/MyCluster/hosts
[ { "blueprint" : "single-node-hdfs-test2", "host_groups" :[ { "host_group" : "slave", "host_count" : 3, "host_predicate" : "Hosts/cpu_count>1” }, { "host_group" : "super-slave", "host_count" : 5, "host_predicate" : "Hosts/cpu_count>2& Hosts/total_mem>3000000" } ] }]
Blueprint Host Discovery
![Page 18: Streamline Hadoop DevOps with Apache Ambari](https://reader034.vdocuments.site/reader034/viewer/2022042907/588003001a28ab421b8b459b/html5/thumbnails/18.jpg)
18 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Comprehensive Security
LDAP/AD• User auth• Sync
Kerberos• MIT KDC• Keytab
management
Atlas• Governance• Compliance• Linage & history• Data
classification
Ranger• Security policies• Audit• Authorization
Knox• Perimeter
security• Supports
LDAP/AD• Sec. for
REST/HTTP• SSL
![Page 19: Streamline Hadoop DevOps with Apache Ambari](https://reader034.vdocuments.site/reader034/viewer/2022042907/588003001a28ab421b8b459b/html5/thumbnails/19.jpg)
19 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Kerberos
Ambari manages Kerberos principals and keytabs
Works with existing MIT KDC or Active Directory
Once Kerberized, handles
1. Adding hosts
2. Adding componentsto existing hosts
3. Adding services
4. Moving componentsto different hosts
![Page 20: Streamline Hadoop DevOps with Apache Ambari](https://reader034.vdocuments.site/reader034/viewer/2022042907/588003001a28ab421b8b459b/html5/thumbnails/20.jpg)
20 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Management Packs
Improved Release Management:Decouple Ambari core from stacks releases
Support Add-ons:–Release vehicle for 3rd party services, views–Self-contained release artifacts–Stack is an overlay of multiple management
packs
![Page 21: Streamline Hadoop DevOps with Apache Ambari](https://reader034.vdocuments.site/reader034/viewer/2022042907/588003001a28ab421b8b459b/html5/thumbnails/21.jpg)
21 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Overlay of Management Packs
inclu
ded
by
includ
ed by
included byincluded byinherits from 2.3
inherits from 2.4
inherits from 2.5
![Page 22: Streamline Hadoop DevOps with Apache Ambari](https://reader034.vdocuments.site/reader034/viewer/2022042907/588003001a28ab421b8b459b/html5/thumbnails/22.jpg)
22 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Management Pack++
Short Term Goals (Ambari 2.4) Retrofit in Stack Processing Framework Enable 3rd party to ship add-on services
Future Goals Management Pack Framework Deliver Views
![Page 23: Streamline Hadoop DevOps with Apache Ambari](https://reader034.vdocuments.site/reader034/viewer/2022042907/588003001a28ab421b8b459b/html5/thumbnails/23.jpg)
23 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Role Based Access Control (RBAC)
As Ambari & organizations grow,so do security needs
Ambari integrates with external authentication systems & LDAP
![Page 24: Streamline Hadoop DevOps with Apache Ambari](https://reader034.vdocuments.site/reader034/viewer/2022042907/588003001a28ab421b8b459b/html5/thumbnails/24.jpg)
24 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
RBAC Terms
Users belong to groups
A group has a role
Users can also have additional roles
Roles are applied to Resources. E.g.,Ambari, particular Cluster, particular View
Roles have permissionse.g., add services to cluster
![Page 25: Streamline Hadoop DevOps with Apache Ambari](https://reader034.vdocuments.site/reader034/viewer/2022042907/588003001a28ab421b8b459b/html5/thumbnails/25.jpg)
25 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
New RBAC Roles
only view
↑, except change configs
↑, except alter cluster topologyor install components
Ambari Admin
Cluster Admin
Cluster Op
Service Admin
Service Op
Read-Only
↑, except add services, Kerberos,manage alerts & upgrades
↑, except manage permissions
all
![Page 26: Streamline Hadoop DevOps with Apache Ambari](https://reader034.vdocuments.site/reader034/viewer/2022042907/588003001a28ab421b8b459b/html5/thumbnails/26.jpg)
26 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Service LayoutCommon Services Stack Override
![Page 27: Streamline Hadoop DevOps with Apache Ambari](https://reader034.vdocuments.site/reader034/viewer/2022042907/588003001a28ab421b8b459b/html5/thumbnails/27.jpg)
27 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Stack Advisor
KerberosHTTPSZookeeper ServersMemory Settings…High Availability
atlas.rest.address = http(s)://host:port
# Atlas Serversatlas.enabletTLS = true|falseatlas.server.http.port = 21000atlas.server.https.port = 21443
Example
Configurations
![Page 28: Streamline Hadoop DevOps with Apache Ambari](https://reader034.vdocuments.site/reader034/viewer/2022042907/588003001a28ab421b8b459b/html5/thumbnails/28.jpg)
28 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Background: Upgrade Terminology
Manual Upgrade
The user follows instructions to upgrade the stack Incurs downtime
![Page 29: Streamline Hadoop DevOps with Apache Ambari](https://reader034.vdocuments.site/reader034/viewer/2022042907/588003001a28ab421b8b459b/html5/thumbnails/29.jpg)
29 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Background: Upgrade Terminology
Manual Upgrade
The user follows instructions to upgrade the stack Incurs downtime
Rolling Upgrade
Automated Upgrades one component per host at a time Preserves cluster operation and minimizes service impact
![Page 30: Streamline Hadoop DevOps with Apache Ambari](https://reader034.vdocuments.site/reader034/viewer/2022042907/588003001a28ab421b8b459b/html5/thumbnails/30.jpg)
30 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Background: Upgrade Terminology
ExpressUpgrade
Automated Runs in parallel across hosts Incurs downtime
Manual Upgrade
The user follows instructions to upgrade the stack Incurs downtime
Rolling Upgrade
Automated Upgrades one component per host at a time Preserves cluster operation and minimizes service impact
![Page 31: Streamline Hadoop DevOps with Apache Ambari](https://reader034.vdocuments.site/reader034/viewer/2022042907/588003001a28ab421b8b459b/html5/thumbnails/31.jpg)
31 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Automated Upgrade: Rolling or Express
Check Prerequisites
Review the prereqs to confirm your cluster configs are ready
Prepare
Take backups of critical cluster metadata
Perform Upgrade
Perform the HDP upgrade. The steps depend on upgrade method: Rolling or Express
Register + Install
Register the HDP repository and install the target HDP version on the cluster
Finalize
Finalize the upgrade, making the target version the current version
![Page 32: Streamline Hadoop DevOps with Apache Ambari](https://reader034.vdocuments.site/reader034/viewer/2022042907/588003001a28ab421b8b459b/html5/thumbnails/32.jpg)
32 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Process: Rolling Upgrade
ZooKeeper
Ranger
Hive
Oozie
Falcon
Kafka
Knox
Storm
Slider
Flume
Finalize or Downgrade
Clients HDFS, YARN, MR, Tez, HBase, Pig. Hive, etc.
Core Masters
Core Slaves
HDFS
YARN
HBase
![Page 33: Streamline Hadoop DevOps with Apache Ambari](https://reader034.vdocuments.site/reader034/viewer/2022042907/588003001a28ab421b8b459b/html5/thumbnails/33.jpg)
33 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Alerting Framework
Alert Type Description Thresholds (units)WEB Connects to a Web URL. Alert status is based on
the HTTP response codeResponse Code (n/a)Connection Timeout (seconds)
PORT Connects to a port. Alert status is based on response time
Response (seconds)
METRIC Checks the value of a service metric. Units vary, based on the metric being checked
Metric Value (units vary)Connection Timeout (seconds)
AGGREGATE Aggregates the status for another alert % Affected (percentage)
SCRIPT Executes a script to handle the alert check Varies
SERVER Executes a server-side runnable class to handle the alert check
Varies
![Page 34: Streamline Hadoop DevOps with Apache Ambari](https://reader034.vdocuments.site/reader034/viewer/2022042907/588003001a28ab421b8b459b/html5/thumbnails/34.jpg)
34 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Alert Check Counts
• Customize the number of times an alert ischecked before dispatching a notification
• Avoid dispatching an alert notification (email, snmp)in case of transient issues
![Page 35: Streamline Hadoop DevOps with Apache Ambari](https://reader034.vdocuments.site/reader034/viewer/2022042907/588003001a28ab421b8b459b/html5/thumbnails/35.jpg)
35 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Alerts - Configuring the Check Count
Set globally for all alerts, or override for a specific alert
Global Setting Alert
Override
![Page 36: Streamline Hadoop DevOps with Apache Ambari](https://reader034.vdocuments.site/reader034/viewer/2022042907/588003001a28ab421b8b459b/html5/thumbnails/36.jpg)
36 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Storm Monitoring View
![Page 37: Streamline Hadoop DevOps with Apache Ambari](https://reader034.vdocuments.site/reader034/viewer/2022042907/588003001a28ab421b8b459b/html5/thumbnails/37.jpg)
37 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Grafana for Ambari Metrics
Grafana as a “Native UI” for Ambari Metrics
Pre-built DashboardsHost-level, Service-level
Supports HTTPS
System Home, Servers
HDFS Home, NameNodes, DataNodes
YARN Home, Applications, Job History Server
HBase Home, Performance
FEATURES DASHBOARDS
![Page 38: Streamline Hadoop DevOps with Apache Ambari](https://reader034.vdocuments.site/reader034/viewer/2022042907/588003001a28ab421b8b459b/html5/thumbnails/38.jpg)
38 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Grafana includes pre-built dashboards for visualizing the most important cluster metrics.
![Page 39: Streamline Hadoop DevOps with Apache Ambari](https://reader034.vdocuments.site/reader034/viewer/2022042907/588003001a28ab421b8b459b/html5/thumbnails/39.jpg)
39 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
The HDFS NameNodedashboard highlightsfile system activity.
![Page 40: Streamline Hadoop DevOps with Apache Ambari](https://reader034.vdocuments.site/reader034/viewer/2022042907/588003001a28ab421b8b459b/html5/thumbnails/40.jpg)
40 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Log Search
Search and index HDP logs!
Capabilities• Rapid Search of all HDP component logs• Search across time ranges, log levels, and for
keywords
Solr
LogsearchAmbari
![Page 41: Streamline Hadoop DevOps with Apache Ambari](https://reader034.vdocuments.site/reader034/viewer/2022042907/588003001a28ab421b8b459b/html5/thumbnails/41.jpg)
41 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Log Search
W O R K E RN O D E
L O G F E E D E R
Solr
L O G S E A R C H
U I
Solr
Solr
A M B A R I
Java ProcessMulti-output SupportGrok filters
Solr CloudLocal Disk Storage
![Page 42: Streamline Hadoop DevOps with Apache Ambari](https://reader034.vdocuments.site/reader034/viewer/2022042907/588003001a28ab421b8b459b/html5/thumbnails/42.jpg)
42 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Future of Ambari Cloud features Service multi-instance (two ZK quorums) Service multi-versions (Spark 1.6 & Spark 2.0) YARN assemblies Patch Upgrades: upgrade individual components in
the same stack version, e.g., just DN and RM in HDP 2.5.*.* with zero downtime
Ambari High Availability