apache ambari bof - overview - hadoop summit 2013
DESCRIPTION
Apache Ambari BOF Meet Up @ Hadoop Summit 2013 Overview Slides http://www.meetup.com/Apache-Ambari-User-Group/events/119184782/TRANSCRIPT
© Hortonworks Inc. 2013
Welcome to Ambari BoF Meetup!
Hadoop SummitJune 2013
© Hortonworks Inc. 2013
Agenda: June 25, 2013
1. Ambari Overview, Current State, Recent Developments
Yusaku Sako @ Hortonworks
Siddharth Wagle @ Hortonworks
2. APIs and SPIs – How to Integrate with AmbariTom Beerbower @ Hortonworks
Nate Cole @ Hortonworks
3. Ambari BlueprintBrian Swan @ Microsoft
Sumit Mohanty @ Hortonworks
4. Ambari and OpenStackHimanshu Bari @ Hortonworks
Page 2
© Hortonworks Inc. 2013
“Make Hadoop Management Easy for Everyone”
Page 3
Provision, Manage,
Monitor
Hadoop Clusters
Hadoop Operators
Analyze Jobs
(MR, Hive, Pig, etc)
Hadoop Users
Integrate Hadoop management/monitoring capabilities to software
Software Vendors
Ambari’s Mission
© Hortonworks Inc. 2013
Apache Ambari – 100% Open Source!
• Active and growing community• 50+ Contributors / 20+ Committers• 140 Ambari User Group Members
• Steady progress / short release cycle
Page 4
Release Version
Release Date
JIRAs Resolved
0.9.0 Sep 2012 402
1.2.0 Feb 2013 441
1.2.1 Mar 2013 134
1.2.2 Apr 2013 106
1.2.3 Jun 2013 515
1.2.4 Jul 2013 109+
1.2.5 Jul 2013 131+
Current Release
Today’s Demo
© Hortonworks Inc. 2013
Ambari System Architecture
5
Ambari Server
Host
Agent
gmond
Host
Agent
gmond
GangliaServer
Agent
Host
Agent
gmondgmetadgmond
Ambari Web
DB
REST/clusters
NagiosServer
Agent
© Hortonworks Inc. 2013
Ambari Today
• Live Demo of a cluster installed with Ambari 1.2.5
Page 6
© Hortonworks Inc. 2013
Ambari Architecture
Page 7
DBOrchestrator SPI
REST API
Request Dispatcher
Ambari Web
Ambari
Server
Metrics
Auth P
rovider
/clusters/services/hosts/workflows/jobs/users, …
UserStore
java
RDBMS
javascript
RDBMS
AD/ LDAP
REST API for integration
Auth Provider
Cluster Configurations
Web Client100% REST
Ambari
Agents ganglia nagios
AlertsPluggable
Service Providersfalcon
Data Mgmt
jmx
python
puppet
© Hortonworks Inc. 2013
Hadoop Security - Background
Page 8
• Non secure mode = No auth• Kerberos tickets allow NN/JT to mutually
authenticate (principals/keytabs)• Guidelines for Kerberos based security
– Daemon principals and keytabs– Mapping of secure user to FS user– Re-configuring individual daemons– Re-starting services and testing for secure setup
KDC
HM
JT
NN Kerberos client Client
TT
DNKerberos client
© Hortonworks Inc. 2013
Hadoop Security with Ambari
Page 9
• Ambari 1.2.5 introduces Security Wizard– Re-configuring services with security enabled– Mapping of secure users to FS user– Re-starting services in secure mode and run
smoke tests– Note: User needs to generate principals and
keytabs
• Future:– Automatically create principals and keytabs– Secure distributions of keytabs
© Hortonworks Inc. 2013
Password Encryption and Web Access
Page 10
• Allow trusted signed certs to be imported into Ambari– Keystore: Open SSL with PKCS 12 keystore– User provided cert is used for server-agent
communication
• Database and LDAP passwords no longer in clear-text– JCEKS keystore to store credentials, locked with
a user-provided master key
© Hortonworks Inc. 2013
REST API – Centralized & Consistent
Page 11
Ambari REST API
Alerts
Job History
MetricsConfigurations
ConfigDB
Nagios
ServerGanglia
Server
…
HTTP GET, POST, PUT, DELETE
:8080
HTTP Status Code / JSON
core-site.xml
core-site.xml
Config files
Config filesConfig files
JMX
Realtime Historical*-site.xml…
Job HistoryDB
Hosts / ServicesCluster
© Hortonworks Inc. 2013
Ambari + Teradata Viewpoint Integration
Page 12
• Ambari = Key enabler for integrating Hadoop monitoring capabilities to Teradata Viewpoint
• Get Hadoop metrics from a non-Ambari deployed cluster– Ambari REST API + Custom Service
Providers
© Hortonworks Inc. 2013
Stack Definitions
• Design Goals–Ambari should be able to support choice of Hadoop stacks–Ambari should enable adding new components to an existing stack
• Define which Services are available (services)• Define where to get the packages (repos)
13
S S S SStack Brepos
services
S S S SStack Arepos
services
S S S SStack CextendsStack B
repos
services
S
S+
© Hortonworks Inc. 2013
Ambari + Redhat GlusterFS Integration
• Using Ambari to deploy / manage cluster with distributed file system other than HDFS–HCFS: GlusterFS as first implementation–Pluggability with other HCFS’s–See AMBARI-1817
Page 14
MapReduce
Hive
Distributed File System
HDFS
GlusterFS
HBasePig
Other HCFS …
© Hortonworks Inc. 2013
Ambari + Accumulo Integration
• Using Ambari to deploy / manage cluster with Accumulo–Google Summer of Code project–See AMBARI-1930
MapReduce
Hive
Distributed File System
HBasePig
© Hortonworks Inc. 2013
Splunk + Ambari Integration
• Head over to Splunk’s booth tomorrow to see Ambari integrated into Splunk’s Management UI for Hadoop monitoring and management
Page 16
+
© Hortonworks Inc. 2013
Thanks!
• Next Up: APIs and SPIs – How to Integrate with Ambari
Page 17