manage add-on services in apache ambari

1 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Manage Add-on Services in Apache Ambari

Jayush LuniyaDataworks Summit Sydney, 2017

2 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Speakers

Apache Ambari PMC

Engineering Manager @ Hortonworks

jluniya@apache.org

Jayush Luniya

3 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Agenda

Ambari Overview

Management Packs – Mpacks v1

Add-On Services – Primer

vNext Architecture – Ambari 3.x

Mpack Based Operations Model – Mpacks v2

Summary

4 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Ambari Overview

5 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Apache Ambari

Ambari ServerProvision | Manage | Monitor

Ambari REST API

Ambari Stacks

Apache Ambari is a 100% open sourceplatform for provisioning, managing andmonitoring Apache Hadoop clusters

Other Operations Tools

Am

bar

i Age

nts

6 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Apache Ambari Themes

• Deliver the core operational capabilities to provision, manage and monitor Hadoop clusters at scale.

Operate Hadoop at

Scale

• Robust API for integration with existing enterprise systems

Integrate with the

Enterprise

• Provide extensible platform for Customers, Partners and the Community (Stacks, Views)

Extend for the

Ecosystem

7 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Ambari Architecture

8 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Extensibility Features

• No vendor lock-in with Ambari

• Customize a stack for customer specific environments

• Add new Services (ISV or otherwise) beyond vendor stackStacks

• Use Ambari for automating cluster installations

• Share best practices on layout and cluster configurationBlueprints

• Extend and customize the Ambari Web UI

• Add new capabilities, customize existing capabilitiesViews

9 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Lifecycle Management

Deploy

Configure

MonitorSecure

Upgrade

Extend

10 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Management Pack – V1

11 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Ambari Management Packs

Motivation

12 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Ambari Management Packs

Decouple stack releases from Ambari core release

Delivery artifact to release stacks and services

Release vehicle for add-on services

Tarballs with metadata of how to install the mpack

EPIC: AMBARI-14854

Ambari Wiki

Release: 2.4.0.0

13 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Ambari Management Packs

14 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Management Pack Use Cases

Hortonworks Data Flow (HDF)

Open Data Platform (ODP)

Hortonworks CyberSecurity (HCS)

Lucidworks HDP Search (Solr)

Stack Mpacks Add-on Service Mpacks

15 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Add-On Services

16 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Add-on Service

Mpack Definition– Specify services, stacks in mpack– Format: mpack.json

Service Definition– Specify components, dependencies, configs– Format: metainfo.xml

Lifecycle Management– Scripts for install/start/stop/upgrade commands– Format: python scripts

Role Command Order– Dependency order for start/stop commands – Format: role_command_order.json

Service Repo – Where to install the service binaries from?– Format: repoinfo.xml

Configurations– Key, value, description, allow empty, password, etc.– Format: <config-type>.xml

Starter Pack

17 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Add-on Service

Enhanced UI– Enhanced UI configuration and dashboard layouts– Format: theme.json, widgets.json

Service Advisor– Recommend/validate configurations– Format: service-advisor.py

Alerts– Defines periodic checks to perform and alert on– Format: alerts.json

Metrics– Defines metrics to be sent from your service to Ambari– Format: metrics.json

Kerberos– Defines service principals, keytab & rules for configuration changes – Format: kerberos.json

Service Upgrade Pack– Steps to perform to upgrade the service (express, rolling)– Format: upgrades/*.xml

Starter Pack

18 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Mpack Definition

Mpack.json{

"name": "my-ambari-mpack",

"version": “1.0.0.0",

"description": "Ambari Management Pack for MyService-1 & MyService-2",

"prerequisites": {

"min-ambari-version": "2.5.0.0",

"min-stack-versions": [

{

"stack_name": "HDP",

"stack_version": "2.5.0"

}

]

},

"artifacts": [

{

"name": "my-service-definitions",

"type" : "service-definitions",

"source_dir" : "common-services"

}

],

19 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Mpack DefinitionMpack.json

{

"name" : "my-addon-services",

"type" : "stack-addon-service-definitions",

"source_dir": "addon-services",

"service_versions_map": [

{

"service_name" : "MyService-1",

"service_version" : "2.1.1",

"applicable_stacks" : [

{

"stack_name" : "HDP",

"stack_version" : "2.5"

}

]

},

{

"service_name" : "MyService-2",

"service_version" : "1.0.1",

"applicable_stacks" : [

{

"stack_name" : "HDP",

"stack_version" : "2.6"

}

]

}

]

20 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Service DefinitionMetainfo.xml<service>

<name>SAMPLESRV</name>

<displayName>New Sample Service</displayName>

<comment>A New Sample Service</comment>

<version>1.0.0</version>

<components>

<component>

<name>SAMPLESRV_MASTER</name>

<displayName>Sample Srv Master</displayName>

<category>MASTER</category>

<cardinality>1</cardinality>

<commandScript>

<script>scripts/master.py</script>

<scriptType>PYTHON</scriptType>

<timeout>600</timeout>

</commandScript>

</component>

<component>

<name>SAMPLESRV_SLAVE</name>

<displayName>Sample Slave</displayName>

<category>SLAVE</category>

<cardinality>1+</cardinality>

...

21 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Service DefinitionMetainfo.xml...

<commandScript>

<script>scripts/slave.py</script>

<scriptType>PYTHON</scriptType>

</commandScript>

<customCommands>

<customCommand>

<name>MY_CUSTOM_COMMAND</name>

<commandScript>

<script>scripts/my_custom_command.py</script>

<scriptType>PYTHON</scriptType>

<timeout>1200</timeout>

</commandScript>

</customCommand>

<customCommands>

<dependencies>

<dependency>

<name>HDFS/HDFS_CLIENT</name>

<scope>cluster | host</scope>

<auto-deploy>

<enabled>true | false</enabled>

</auto-deploy>

</dependency>

<dependencies>

22 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Service DefinitionMetainfo.xml...

</component>

</components>

<requiredServices>

<service>HDFS</service>

</requiredServices>

<configuration-dependencies>

<config-type>service-env</config-type>

<config-type>service-site</config-type>

</configuration-dependencies>

<osSpecifics>

<osSpecific>

<osFamily>any</osFamily>

<packages>

<package>

<name>rpm_apt_pkg_name</name>

</package>

</packages>

</osSpecific>

</osSpecifics>

</service>

</services>

</metainfo>

23 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Lifecycle Management

import sys

from resource_management import Script

class Master(Script):

def install(self, env):

print 'Install the Sample Srv Master'

def stop(self, env):

print 'Stop the Sample Srv Master'

def start(self, env):

print 'Start the Sample Srv Master'

def status(self, env):

print 'Status of the Sample Srv Master'

def configure(self, env):

print 'Configure the Sample Srv Master'

if __name__ == "__main__":

Master().execute()

Python Scripts

24 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Lifecycle ManagementRole Command Order{

"_comment" : "Record format:",

"_comment" : "blockedRole-blockedCommand: [blockerRole1-

blockerCommand1, blockerRole2-blockerCommand2, ...]",

"general_deps" : {

"_comment" : “Dependencies for SAMPLESRV",

"SAMPLESRV_SLAVE-START": ["SAMPLESRV_MASTER-START"],

"SAMPLESRV_SLAVE-STOP": ["SAMPLESRV_MASTER-STOP"]

}

}

25 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Enhanced Configurations

Too many configurations– Which ones are important?

Too easy to mess up– What are valid/reasonable values?

– What are the units?

– Ok, what about dependencies?

Gets harder with combinations of services, host assignments, enabled features, CPU/RAM/disks, etc– Any recommendations? What am I doing wrong?

Solution: – Enhanced Configurations

– Stack/Service Advisor

Hadoop Configuration Challenges

26 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Enhanced Configurations - UI

Customizable layout- Tabs- Sections- Sub-sections- Simple grid layout

(Advanced Tab contains remaining configurations)

New Widgets

Sliders- Recommended- Minimum- Maximum- Increment StepCombos- Enumerated valuesToggles- Binary optionsSpinners- Splits value into multiple

controls. - Time in milliseconds split

into days, hours, minutes.

Lists- Enumerated values- Single select- Multi select

27 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Service Advisor

Replaces monolithic stack advisors

Defined at service level

Config recommendations and validations

Scenarios– Kerberos Enabled/Disabled

– HTTPS Enabled/Disabled

– Zookeeper Quorum

– Memory Settings

– High Availability Enabled/Disabled

28 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Alerting Framework

Alert Type Description Thresholds (units)

WEB Connects to a Web URL. Alert status is

based on the HTTP response code

Response Code (n/a)

Connection Timeout

(seconds)

PORT Connects to a port. Alert status is based

on response time

Response (seconds)

METRIC Checks the value of a service metric.

Units vary, based on the metric being

checked

Metric Value (units vary)

Connection Timeout

(seconds)

AGGREGATE Aggregates the status for another alert % Affected (percentage)

SCRIPT Executes a script to handle the alert

check

Varies

SERVER Executes a server-side runnable class to

handle the alert check

Varies

29 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Ambari Metrics System

Metric Sinks – HDFS, YARN, HBase, Storm etc.

Metric Monitors – lightweight daemon for system metrics

Collector – API daemon + HBase (embedded / distributed)

Phoenix schema designed for fast reads

Grafana support from version 2.2.2

Ambari

Collector API

Grafana

Phoenix

HDP

Services

System

MO

NIT

OR

SS

INK

S

Metrics Collector

30 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Kerberos

Ambari manages Kerberos principals and keytabs

Works with existing MIT KDC or Active Directory

Once kerberized, handles

– Adding host components

– Adding hosts

– Adding services

– Moving components

31 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

vNext Architecture

32 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

LimitationsOne-One Relationships

Monolithic Stacks are monolithic

Entire cluster upgrade

Ambari upgrades for refreshing stack definitions

Ambari Cluster

Cluster Stack

StackService Version

ClusterService

Instance

HostComponent

Instance

AmbariHosting Platform

33 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Vision

Replace monolithic stacks

Modular upgrades

Decouple Ambari core release

Independent release cycles

Refactor at service level

3rd party onboarding platform

Multiple Clusters

Multiple Stacks

Multiple Services

Multiple Service Versions

Multiple Host Components

Multiple Hosting Platforms

Modular Multi Everything

34 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Cluster2

Cluster1

Ambari Hosting Platform

v.Next Architecture

ZK1 HDFS YARN HBASE

Core SG

NIFIZK2 KAFKA

Stream SG

Mpack Repository

HDP3.0.0

HDF3.1.0HDF3.2.0

Ambari

BROKER-1

BROKER-2

BROKER-3

HBASE2

Yapp SG

YarnApp1.0.0

Yarn Hosting Platform

HMaster

RServer-1

RServer-2

35 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Cluster2

Cluster1

Ambari Hosting Platform

v.Next Architecture

ZK1 HDFS YARN HBASE

Core SG

NIFIZK2 KAFKA

Stream SG

Mpack Repository

HDP3.0.0

HDF3.1.0HDF3.2.0

Ambari

BROKER-1

BROKER-2

BROKER-3

HBASE2

Yapp SG

YarnApp1.0.0

Yarn Hosting Platform

HMaster

RServer-1

RServer-2

Modular

Multi-Everything

36 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Mpack Based Operations Model

37 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

New Concepts - Delivery

Deployment artifact

Individual ownership of services

Release container for services

Examples:o HDFS-3.0.0-packlet

o SPARK-2.0.0-packlet

Distribution artifact

Composed of multiple packlets

Modular release & upgrades

Examples:o HDP-3.0.0-mpack

o HDF-3.1.0-mpack

Packlets Management Packs (Mpacks)

Mpack Repository

Catalog of management packs

Compatibility between mpacks

Examples:o Hortonworks Mpack Repository

38 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Management Packs – V2

Decouple Ambari Core release

Replacement for monolithic stacks

Composition of verified combination of packlets

Support dynamic installation

Mpacks released and upgraded independently

Release Management

39 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Management Packs – V2

Support multiple mpack repositories

Search and discover management packs

Compatibility between management packs

Recommendations for mpack bundles

Add-on Mpacks First class support for 3rd party add-on services

Provide a third party onboarding platform

Independently upgradable services

Mpack Repositories

40 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Upgrades

Modular upgrades

Ability to upgrade subset of services in a cluster

Patch services without affecting other parts of the cluster

Performance improvements

Goals

Mpack Upgrades

Upgrade packs defined at mpack level

Upgrade subset of services in a cluster that belong to a mpack

Download new version of a mpack and kickoff an upgrade

41 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Upgrades

Upgrade mechanism for hotfix releases

Selectively patch components without restarting other services

Purely Version Definition File (VDF) driven; no configuration changes

Does not require a new management pack; uses same upgrade pack

Applied patches can be reverted; no finalization step

Patch Upgrades

Service Upgrades

VDF driven if no configuration changes needed

Mpack driven for more complicated service upgrades

No revert capability (similar to current cluster upgrades)

42 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Summary

43 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Key Takeaways

Stack Based Operations Model

Refactoring at service level

Mpacks v1 provided delivery vehicle for add-on services

Architecture limitations remain

Mpack Based Operations Model

Re-architect from ground up

First class support for add-on services

Modular deployments & upgrades

Current Future

A

M P

44 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Resources

https://cwiki.apache.org/confluence/display/AMBARI/Stacks+and+Services

https://cwiki.apache.org/confluence/display/AMBARI/Custom+Services

https://cwiki.apache.org/confluence/display/AMBARI/Management+Packs

https://cwiki.apache.org/confluence/display/AMBARI/Enhanced+Configs

https://cwiki.apache.org/confluence/display/AMBARI/Enhanced+Service+Dashboard

https://cwiki.apache.org/confluence/display/AMBARI/Stack+Defined+Metrics

https://cwiki.apache.org/confluence/display/AMBARI/Alerts

https://cwiki.apache.org/confluence/display/AMBARI/Quick+Links

https://cwiki.apache.org/confluence/display/AMBARI/Automated+Kerberizaton

https://cwiki.apache.org/confluence/display/AMBARI/Service+Upgrade

https://cwiki.apache.org/confluence/display/AMBARI/Role+Command+Order

45 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Apache JIRAs - vNext

AMBARI-14714 – Multi Everything ArchitectureUmbrella Apache EPIC

AMBARI-19621 – Mpack Based Operations Model

AMBARI-20463 – Multi Service Instances

AMBARI-20465 – Multi Host Component Instances

AMBARI-17353 – Yarn Hosted Services

AMBARI-12556 – Patch Upgrades

AMBARI-20466 – Multi Cluster

AMBARI-20435 - Swagger Integration

Other Apache EPICs

Release Timelines Features will be incrementally delivered in Apache Ambari 3.x releases.

Join the Apache Ambari Community

46 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Thank You