manage add-on services in apache ambari
TRANSCRIPT
1 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Manage Add-on Services in Apache Ambari
Jayush LuniyaDataworks Summit Sydney, 2017
2 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Speakers
Apache Ambari PMC
Engineering Manager @ Hortonworks
Jayush Luniya
3 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Agenda
Ambari Overview
Management Packs – Mpacks v1
Add-On Services – Primer
vNext Architecture – Ambari 3.x
Mpack Based Operations Model – Mpacks v2
Summary
4 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Ambari Overview
5 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Apache Ambari
Ambari ServerProvision | Manage | Monitor
Ambari REST API
Ambari Stacks
Apache Ambari is a 100% open sourceplatform for provisioning, managing andmonitoring Apache Hadoop clusters
Other Operations Tools
Am
bar
i Age
nts
6 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Apache Ambari Themes
• Deliver the core operational capabilities to provision, manage and monitor Hadoop clusters at scale.
Operate Hadoop at
Scale
• Robust API for integration with existing enterprise systems
Integrate with the
Enterprise
• Provide extensible platform for Customers, Partners and the Community (Stacks, Views)
Extend for the
Ecosystem
7 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Ambari Architecture
8 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Extensibility Features
• No vendor lock-in with Ambari
• Customize a stack for customer specific environments
• Add new Services (ISV or otherwise) beyond vendor stackStacks
• Use Ambari for automating cluster installations
• Share best practices on layout and cluster configurationBlueprints
• Extend and customize the Ambari Web UI
• Add new capabilities, customize existing capabilitiesViews
9 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Lifecycle Management
Deploy
Configure
MonitorSecure
Upgrade
Extend
10 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Management Pack – V1
11 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Ambari Management Packs
Motivation
12 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Ambari Management Packs
Decouple stack releases from Ambari core release
Delivery artifact to release stacks and services
Release vehicle for add-on services
Tarballs with metadata of how to install the mpack
EPIC: AMBARI-14854
Ambari Wiki
Release: 2.4.0.0
13 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Ambari Management Packs
14 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Management Pack Use Cases
Hortonworks Data Flow (HDF)
Open Data Platform (ODP)
Hortonworks CyberSecurity (HCS)
Lucidworks HDP Search (Solr)
Stack Mpacks Add-on Service Mpacks
15 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Add-On Services
16 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Add-on Service
Mpack Definition– Specify services, stacks in mpack– Format: mpack.json
Service Definition– Specify components, dependencies, configs– Format: metainfo.xml
Lifecycle Management– Scripts for install/start/stop/upgrade commands– Format: python scripts
Role Command Order– Dependency order for start/stop commands – Format: role_command_order.json
Service Repo – Where to install the service binaries from?– Format: repoinfo.xml
Configurations– Key, value, description, allow empty, password, etc.– Format: <config-type>.xml
Starter Pack
17 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Add-on Service
Enhanced UI– Enhanced UI configuration and dashboard layouts– Format: theme.json, widgets.json
Service Advisor– Recommend/validate configurations– Format: service-advisor.py
Alerts– Defines periodic checks to perform and alert on– Format: alerts.json
Metrics– Defines metrics to be sent from your service to Ambari– Format: metrics.json
Kerberos– Defines service principals, keytab & rules for configuration changes – Format: kerberos.json
Service Upgrade Pack– Steps to perform to upgrade the service (express, rolling)– Format: upgrades/*.xml
Starter Pack
18 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Mpack Definition
Mpack.json{
"name": "my-ambari-mpack",
"version": “1.0.0.0",
"description": "Ambari Management Pack for MyService-1 & MyService-2",
"prerequisites": {
"min-ambari-version": "2.5.0.0",
"min-stack-versions": [
{
"stack_name": "HDP",
"stack_version": "2.5.0"
}
]
},
"artifacts": [
{
"name": "my-service-definitions",
"type" : "service-definitions",
"source_dir" : "common-services"
}
],
19 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Mpack DefinitionMpack.json
{
"name" : "my-addon-services",
"type" : "stack-addon-service-definitions",
"source_dir": "addon-services",
"service_versions_map": [
{
"service_name" : "MyService-1",
"service_version" : "2.1.1",
"applicable_stacks" : [
{
"stack_name" : "HDP",
"stack_version" : "2.5"
}
]
},
{
"service_name" : "MyService-2",
"service_version" : "1.0.1",
"applicable_stacks" : [
{
"stack_name" : "HDP",
"stack_version" : "2.6"
}
]
}
]
20 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Service DefinitionMetainfo.xml<service>
<name>SAMPLESRV</name>
<displayName>New Sample Service</displayName>
<comment>A New Sample Service</comment>
<version>1.0.0</version>
<components>
<component>
<name>SAMPLESRV_MASTER</name>
<displayName>Sample Srv Master</displayName>
<category>MASTER</category>
<cardinality>1</cardinality>
<commandScript>
<script>scripts/master.py</script>
<scriptType>PYTHON</scriptType>
<timeout>600</timeout>
</commandScript>
</component>
<component>
<name>SAMPLESRV_SLAVE</name>
<displayName>Sample Slave</displayName>
<category>SLAVE</category>
<cardinality>1+</cardinality>
...
21 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Service DefinitionMetainfo.xml...
<commandScript>
<script>scripts/slave.py</script>
<scriptType>PYTHON</scriptType>
</commandScript>
<customCommands>
<customCommand>
<name>MY_CUSTOM_COMMAND</name>
<commandScript>
<script>scripts/my_custom_command.py</script>
<scriptType>PYTHON</scriptType>
<timeout>1200</timeout>
</commandScript>
</customCommand>
<customCommands>
<dependencies>
<dependency>
<name>HDFS/HDFS_CLIENT</name>
<scope>cluster | host</scope>
<auto-deploy>
<enabled>true | false</enabled>
</auto-deploy>
</dependency>
<dependencies>
22 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Service DefinitionMetainfo.xml...
</component>
</components>
<requiredServices>
<service>HDFS</service>
</requiredServices>
<configuration-dependencies>
<config-type>service-env</config-type>
<config-type>service-site</config-type>
</configuration-dependencies>
<osSpecifics>
<osSpecific>
<osFamily>any</osFamily>
<packages>
<package>
<name>rpm_apt_pkg_name</name>
</package>
</packages>
</osSpecific>
</osSpecifics>
</service>
</services>
</metainfo>
23 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Lifecycle Management
import sys
from resource_management import Script
class Master(Script):
def install(self, env):
print 'Install the Sample Srv Master'
def stop(self, env):
print 'Stop the Sample Srv Master'
def start(self, env):
print 'Start the Sample Srv Master'
def status(self, env):
print 'Status of the Sample Srv Master'
def configure(self, env):
print 'Configure the Sample Srv Master'
if __name__ == "__main__":
Master().execute()
Python Scripts
24 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Lifecycle ManagementRole Command Order{
"_comment" : "Record format:",
"_comment" : "blockedRole-blockedCommand: [blockerRole1-
blockerCommand1, blockerRole2-blockerCommand2, ...]",
"general_deps" : {
"_comment" : “Dependencies for SAMPLESRV",
"SAMPLESRV_SLAVE-START": ["SAMPLESRV_MASTER-START"],
"SAMPLESRV_SLAVE-STOP": ["SAMPLESRV_MASTER-STOP"]
}
}
25 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Enhanced Configurations
Too many configurations– Which ones are important?
Too easy to mess up– What are valid/reasonable values?
– What are the units?
– Ok, what about dependencies?
Gets harder with combinations of services, host assignments, enabled features, CPU/RAM/disks, etc– Any recommendations? What am I doing wrong?
Solution: – Enhanced Configurations
– Stack/Service Advisor
Hadoop Configuration Challenges
26 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Enhanced Configurations - UI
Customizable layout- Tabs- Sections- Sub-sections- Simple grid layout
(Advanced Tab contains remaining configurations)
New Widgets
Sliders- Recommended- Minimum- Maximum- Increment StepCombos- Enumerated valuesToggles- Binary optionsSpinners- Splits value into multiple
controls. - Time in milliseconds split
into days, hours, minutes.
Lists- Enumerated values- Single select- Multi select
27 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Service Advisor
Replaces monolithic stack advisors
Defined at service level
Config recommendations and validations
Scenarios– Kerberos Enabled/Disabled
– HTTPS Enabled/Disabled
– Zookeeper Quorum
– Memory Settings
– High Availability Enabled/Disabled
28 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Alerting Framework
Alert Type Description Thresholds (units)
WEB Connects to a Web URL. Alert status is
based on the HTTP response code
Response Code (n/a)
Connection Timeout
(seconds)
PORT Connects to a port. Alert status is based
on response time
Response (seconds)
METRIC Checks the value of a service metric.
Units vary, based on the metric being
checked
Metric Value (units vary)
Connection Timeout
(seconds)
AGGREGATE Aggregates the status for another alert % Affected (percentage)
SCRIPT Executes a script to handle the alert
check
Varies
SERVER Executes a server-side runnable class to
handle the alert check
Varies
29 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Ambari Metrics System
Metric Sinks – HDFS, YARN, HBase, Storm etc.
Metric Monitors – lightweight daemon for system metrics
Collector – API daemon + HBase (embedded / distributed)
Phoenix schema designed for fast reads
Grafana support from version 2.2.2
Ambari
Collector API
Grafana
Phoenix
HDP
Services
System
MO
NIT
OR
SS
INK
S
Metrics Collector
30 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Kerberos
Ambari manages Kerberos principals and keytabs
Works with existing MIT KDC or Active Directory
Once kerberized, handles
– Adding host components
– Adding hosts
– Adding services
– Moving components
31 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
vNext Architecture
32 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
LimitationsOne-One Relationships
Monolithic Stacks are monolithic
Entire cluster upgrade
Ambari upgrades for refreshing stack definitions
Ambari Cluster
Cluster Stack
StackService Version
ClusterService
Instance
HostComponent
Instance
AmbariHosting Platform
33 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Vision
Replace monolithic stacks
Modular upgrades
Decouple Ambari core release
Independent release cycles
Refactor at service level
3rd party onboarding platform
Multiple Clusters
Multiple Stacks
Multiple Services
Multiple Service Versions
Multiple Host Components
Multiple Hosting Platforms
Modular Multi Everything
34 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Cluster2
Cluster1
Ambari Hosting Platform
v.Next Architecture
ZK1 HDFS YARN HBASE
Core SG
NIFIZK2 KAFKA
Stream SG
Mpack Repository
HDP3.0.0
HDF3.1.0HDF3.2.0
Ambari
BROKER-1
BROKER-2
BROKER-3
HBASE2
Yapp SG
YarnApp1.0.0
Yarn Hosting Platform
HMaster
RServer-1
RServer-2
35 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Cluster2
Cluster1
Ambari Hosting Platform
v.Next Architecture
ZK1 HDFS YARN HBASE
Core SG
NIFIZK2 KAFKA
Stream SG
Mpack Repository
HDP3.0.0
HDF3.1.0HDF3.2.0
Ambari
BROKER-1
BROKER-2
BROKER-3
HBASE2
Yapp SG
YarnApp1.0.0
Yarn Hosting Platform
HMaster
RServer-1
RServer-2
Modular
Multi-Everything
36 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Mpack Based Operations Model
37 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
New Concepts - Delivery
Deployment artifact
Individual ownership of services
Release container for services
Examples:o HDFS-3.0.0-packlet
o SPARK-2.0.0-packlet
Distribution artifact
Composed of multiple packlets
Modular release & upgrades
Examples:o HDP-3.0.0-mpack
o HDF-3.1.0-mpack
Packlets Management Packs (Mpacks)
Mpack Repository
Catalog of management packs
Compatibility between mpacks
Examples:o Hortonworks Mpack Repository
38 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Management Packs – V2
Decouple Ambari Core release
Replacement for monolithic stacks
Composition of verified combination of packlets
Support dynamic installation
Mpacks released and upgraded independently
Release Management
39 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Management Packs – V2
Support multiple mpack repositories
Search and discover management packs
Compatibility between management packs
Recommendations for mpack bundles
Add-on Mpacks First class support for 3rd party add-on services
Provide a third party onboarding platform
Independently upgradable services
Mpack Repositories
40 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Upgrades
Modular upgrades
Ability to upgrade subset of services in a cluster
Patch services without affecting other parts of the cluster
Performance improvements
Goals
Mpack Upgrades
Upgrade packs defined at mpack level
Upgrade subset of services in a cluster that belong to a mpack
Download new version of a mpack and kickoff an upgrade
41 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Upgrades
Upgrade mechanism for hotfix releases
Selectively patch components without restarting other services
Purely Version Definition File (VDF) driven; no configuration changes
Does not require a new management pack; uses same upgrade pack
Applied patches can be reverted; no finalization step
Patch Upgrades
Service Upgrades
VDF driven if no configuration changes needed
Mpack driven for more complicated service upgrades
No revert capability (similar to current cluster upgrades)
42 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Summary
43 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Key Takeaways
Stack Based Operations Model
Refactoring at service level
Mpacks v1 provided delivery vehicle for add-on services
Architecture limitations remain
Mpack Based Operations Model
Re-architect from ground up
First class support for add-on services
Modular deployments & upgrades
Current Future
A
M P
44 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Resources
https://cwiki.apache.org/confluence/display/AMBARI/Stacks+and+Services
https://cwiki.apache.org/confluence/display/AMBARI/Custom+Services
https://cwiki.apache.org/confluence/display/AMBARI/Management+Packs
https://cwiki.apache.org/confluence/display/AMBARI/Enhanced+Configs
https://cwiki.apache.org/confluence/display/AMBARI/Enhanced+Service+Dashboard
https://cwiki.apache.org/confluence/display/AMBARI/Stack+Defined+Metrics
https://cwiki.apache.org/confluence/display/AMBARI/Alerts
https://cwiki.apache.org/confluence/display/AMBARI/Quick+Links
https://cwiki.apache.org/confluence/display/AMBARI/Automated+Kerberizaton
https://cwiki.apache.org/confluence/display/AMBARI/Service+Upgrade
https://cwiki.apache.org/confluence/display/AMBARI/Role+Command+Order
45 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Apache JIRAs - vNext
AMBARI-14714 – Multi Everything ArchitectureUmbrella Apache EPIC
AMBARI-19621 – Mpack Based Operations Model
AMBARI-20463 – Multi Service Instances
AMBARI-20465 – Multi Host Component Instances
AMBARI-17353 – Yarn Hosted Services
AMBARI-12556 – Patch Upgrades
AMBARI-20466 – Multi Cluster
AMBARI-20435 - Swagger Integration
Other Apache EPICs
Release Timelines Features will be incrementally delivered in Apache Ambari 3.x releases.
Join the Apache Ambari Community
46 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Thank You