double your hadoop hardware performance with smartsense
Post on 21-Mar-2017
354 Views
Preview:
TRANSCRIPT
1 © Hortonworks Inc. 2011 – 2017. All Rights Reserved1 © Hortonworks Inc. 2011 – 2017. All Rights Reserved
Boost Apache Hadoop Hardware Performance 2X with SmartSense
Paul CoddingProduct Management Director
2 © Hortonworks Inc. 2011 – 2017. All Rights Reserved
Hortonworks Connected Data Platforms and Solutions
Data Services
Hortonworks Solutions
Enterprise DataWarehouse Optimization
Cyber Security andThreat Management
Internet of Thingsand Streaming Analytics
Data CenterHortonworks Data Suite
HDFHDP
HortonworksConnection
CloudHortonworks Data CloudAWS HDInsight
Hortonworks ConnectionEnablement Subscription
SmartSense™
Premier Operational Support
Educational Services
Professional Services
Community Connection
3 © Hortonworks Inc. 2011 – 2017. All Rights Reserved
Hortonworks Connection Ensures Success of Your Big Data Journey
4 © Hortonworks Inc. 2011 – 2017. All Rights Reserved
5 Reasons Why You Need More Than Just Open Source Software
The open source community doesn’t ensure everything works together and is certified for the data center and cloud platforms you rely on. Hortonworks does.1The unprecedented pace of open source innovation is both a benefit and a challenge. Hortonworks can help; it’s what we do.2Your enterprise needs more than just support for the latest open source versions. Hortonworks supports and maintains the versions you rely on.3The community doesn’t ensure that consistent security, governance, and operations are built in. Hortonworks takes enterprise needs seriously.4The community is not responsible for your success with open source technologies and tools. Hortonworks success is built on your success. 5
5 © Hortonworks Inc. 2011 – 2017. All Rights Reserved
Increase Performance
Prevent Issues
Accelerate Case Resolution
Understand Your Cluster
6 © Hortonworks Inc. 2011 – 2017. All Rights Reserved
Issue: YARN @ capacity, struggling to add more use cases
Before SmartSense
Could only run 500 jobs concurrently
1100 jobs would be pending waiting for
resources at peak hours
7 © Hortonworks Inc. 2011 – 2017. All Rights Reserved
After Applying only 3 SmartSense Recommendations
They can now run 1200 concurrent jobs
...with only 350 waiting jobs at peak hours
Issue: YARN @ capacity, struggling to add more use cases
Before SmartSense
Could only run 500 jobs concurrently
1100 jobs would be pending waiting for
resources at peak hours
8 © Hortonworks Inc. 2011 – 2017. All Rights Reserved
After Applying only 3 SmartSense Recommendations
They can now run 1200 concurrent jobs
...with only 350 waiting jobs at peak hours
Issue: YARN @ capacity, struggling to add more use cases
Before SmartSense
Could only run 500 jobs concurrently
1100 jobs would be pending waiting for
resources at peak hours
With SmartSense = 2X Throughput Improvement
9 © Hortonworks Inc. 2011 – 2017. All Rights Reserved
Hardware ($$$)
Hadoop Performance
• Type of CPU & Core Count• Type & Amount of Memory• Type & Number of Disks
10 © Hortonworks Inc. 2011 – 2017. All Rights Reserved
Hardware ($$$)
Operating System
Hadoop Performance
• Type of CPU & Core Count• Type & Amount of Memory• Type & Number of Disks
• Kernel Configuration• Disk Mount/Tuning• Network Configuration
11 © Hortonworks Inc. 2011 – 2017. All Rights Reserved
Hardware ($$$)
Operating System
Hadoop Daemons
Hadoop Performance
• Type of CPU & Core Count• Type & Amount of Memory• Type & Number of Disks
• Kernel Configuration• Disk Mount/Tuning• Network Configuration
• YARN/MR/Tez Memory Configuration• HDFS Configuration• ZooKeeper Configuration
12 © Hortonworks Inc. 2011 – 2017. All Rights Reserved
Hardware ($$$)
Operating System
Hadoop Daemons
Hadoop Performance
• Type of CPU & Core Count• Type & Amount of Memory• Type & Number of Disks
• Kernel Configuration• Disk Mount/Tuning• Network Configuration
• YARN/MR/Tez Memory Configuration• HDFS Configuration• ZooKeeper Configuration
13 © Hortonworks Inc. 2011 – 2017. All Rights Reserved
What we do
A M B A R I
O P S S m a r t S e n s eS E R V E R
B U N D L EG AT E W AY
S m a r t S e n s eA n a l y ti c s
S m a r t S e n s eS E R V I C E
Collection Diagnostic Information Secure & Send Analyze & Recommend
14 © Hortonworks Inc. 2011 – 2017. All Rights Reserved
Hardware ($$$)
Operating System
Hadoop Daemons
Hadoop Performance
• Type of CPU & Core Count• Type & Amount of Memory• Type & Number of Disks
• Kernel Configuration• Disk Mount/Tuning• Network Configuration
• YARN/MR/Tez Memory Configuration• HDFS Configuration• ZooKeeper Configuration
15 © Hortonworks Inc. 2011 – 2017. All Rights Reserved
YARN Memory Configuration
ContainersUnit of allocation for memory and compute
Scheduler Configuration Minimum Container Size Maximum Container Size
YARN NodeManager Configuration How much memory can be used by YARN on each cluster node
YARN Cluster1
2
3
4
5
6
7
64 GB
16 © Hortonworks Inc. 2011 – 2017. All Rights Reserved
YARN Memory Configuration
ContainersUnit of allocation for memory and compute
Scheduler Configuration Minimum Container Size: 5GB Maximum Container Size: 35GB
YARN NodeManager Configuration How much memory can be used by YARN on each cluster node
– 35GB
YARN Cluster1
2
3
4
5
6
7
64 GB
17 © Hortonworks Inc. 2011 – 2017. All Rights Reserved
YARN Memory Configuration
5
YARN ClusterApplication YARN Scheduler
I need 5 2GB containers
Min: 5GBMax: 35GB
201
2
3
4
5
6
7
18 © Hortonworks Inc. 2011 – 2017. All Rights Reserved
YARN Memory Configuration
Application YARN Scheduler
I need 5 2GB containers
Min: 5GBMax: 35GB
5
5
5
5
5
YARN Cluster
5
201
2
3
4
5
6
7
19 © Hortonworks Inc. 2011 – 2017. All Rights Reserved
YARN Memory Configuration
Application YARN Scheduler
I need 5 2GB containers
Min: 5GBMax: 35GB
Application is taking 25GB of resources when it only needs 10GB
5
5
5
5
5
YARN Cluster
5
201
2
3
4
5
6
7
20 © Hortonworks Inc. 2011 – 2017. All Rights Reserved
YARN Memory Configuration
Gatorade: $2.50Machine only takes CashEXACT CHANGE REQUIRED!
21 © Hortonworks Inc. 2011 – 2017. All Rights Reserved
YARN Memory Configuration
Gatorade: $2.50Machine only takes CashEXACT CHANGE REQUIRED!
Minimum Withdrawal: $20Maximum Withdrawal: $500
22 © Hortonworks Inc. 2011 – 2017. All Rights Reserved
YARN Memory Configuration
ContainersUnit of allocation for memory and compute
Scheduler Configuration Minimum Container Size: 2GB vs 5GB Maximum Container Size: 10GB vs 35GB
YARN NodeManager Configuration How much memory can be used by YARN on each cluster node
– 56GB vs 35GB
YARN Cluster1
2
3
4
5
6
7
64 GB
23 © Hortonworks Inc. 2011 – 2017. All Rights Reserved
After Applying only 3 SmartSense Recommendations
They can now run 1200 concurrent jobs
...with only 350 waiting jobs at peak hours
Issue: YARN @ capacity, struggling to add more use cases
Before SmartSense
Could only run 500 jobs concurrently
1100 jobs would be pending waiting for
resources at peak hours
With SmartSense = 2X Throughput Improvement
24 © Hortonworks Inc. 2011 – 2017. All Rights Reserved
Increase Performance
Prevent Issues
Accelerate Case Resolution
Understand Your Cluster
25 © Hortonworks Inc. 2011 – 2017. All Rights Reserved
Support Cases by Type
ConfigurationEnvironmentEducationNo ResponseProduct DefectUnreproducibleUse Case AdviceWorks as DesignedOther
SmartSense Today – Prevent Issues
26 © Hortonworks Inc. 2011 – 2017. All Rights Reserved
Support Cases by Type
ConfigurationEnvironmentEducationNo ResponseProduct DefectUnreproducibleUse Case AdviceWorks as DesignedOther
SmartSense Today – Prevent Issues
27 © Hortonworks Inc. 2011 – 2017. All Rights Reserved
Support Cases by Type
ConfigurationEnvironmentEducationNo ResponseProduct DefectUnreproducibleUse Case AdviceWorks as DesignedOther
SmartSense Today – Prevent Issues
30% of support cases are configuration issues—this is where SmartSense adds incredible value
28 © Hortonworks Inc. 2011 – 2017. All Rights Reserved
SmartSense Today – Prevent Issues
SmartSense analyzes Bundles for configuration issues – recommendations are produced and made available for each cluster in the Hortonworks Support Portal
Recommendations prevent operational issues, and improve performance and overall cluster throughput.
29 © Hortonworks Inc. 2011 – 2017. All Rights Reserved
SmartSense Today – Prevent Issues
SmartSense analyzes Bundles for configuration issues – recommendations are produced and made available for each cluster in the Hortonworks Support Portal
Recommendations prevent operational issues, and improve performance and overall cluster throughput.
30 © Hortonworks Inc. 2011 – 2017. All Rights Reserved
Increase Performance
Prevent Issues
Accelerate Case Resolution
Understand Your Cluster
31 © Hortonworks Inc. 2011 – 2017. All Rights Reserved
SmartSense Today – Accelerate Case Resolution
SmartSense provides Hadoop Operators with an Ambari Integrated tool to quickly capture diagnostic information for specific services and hosts into a single “Bundle” that’s automatically uploaded to Hortonworks Support.
Significantly reduces the back-and-forth nature of troubleshooting issues.
A M B A R I
O P SH O R T O N W O R K S
S U P P O R T
S U P P O R TC A S E
S m a r t S e n s eS E R V E R
B U N D L EG AT E W AY
32 © Hortonworks Inc. 2011 – 2017. All Rights Reserved
SmartSense Today – Accelerate Case Resolution
SmartSense provides Hadoop Operators with an Ambari Integrated tool to quickly capture diagnostic information for specific services and hosts into a single “Bundle” that’s automatically uploaded to Hortonworks Support.
Significantly reduces the back-and-forth nature of troubleshooting issues.
A M B A R I
O P SH O R T O N W O R K S
S U P P O R T
S U P P O R TC A S E
S m a r t S e n s eS E R V E R
B U N D L EG AT E W AY
33 © Hortonworks Inc. 2011 – 2017. All Rights Reserved
SmartSense Today – Accelerate Case Resolution
SmartSense provides Hadoop Operators with an Ambari Integrated tool to quickly capture diagnostic information for specific services and hosts into a single “Bundle” that’s automatically uploaded to Hortonworks Support.
Significantly reduces the back-and-forth nature of troubleshooting issues.
A M B A R I
O P SH O R T O N W O R K S
S U P P O R T
S U P P O R TC A S E
S m a r t S e n s eS E R V E R
B U N D L EG AT E W AY
34 © Hortonworks Inc. 2011 – 2017. All Rights Reserved
SmartSense Today – Accelerate Case Resolution
SmartSense provides Hadoop Operators with an Ambari Integrated tool to quickly capture diagnostic information for specific services and hosts into a single “Bundle” that’s automatically uploaded to Hortonworks Support.
Significantly reduces the back-and-forth nature of troubleshooting issues.
A M B A R I
O P SH O R T O N W O R K S
S U P P O R T
S U P P O R TC A S E
S m a r t S e n s eS E R V E R
B U N D L EG AT E W AY
35 © Hortonworks Inc. 2011 – 2017. All Rights Reserved
SmartSense Data Capture Architecture
L A N D I N G Z O N E
S E R V E RG AT E W AY
A M B A R I
A G E N T A G E N T
A G E N TA G E N TA G E N T
A G E N T
W O R K E RN O D E
W O R K E RN O D E
W O R K E RN O D E
W O R K E RN O D E
W O R K E RN O D E
W O R K E RN O D E
S m a r t S e n s eA n a l y ti c s
36 © Hortonworks Inc. 2011 – 2017. All Rights Reserved
SmartSense Data Capture Architecture
L A N D I N G Z O N E
S E R V E RG AT E W AY
A M B A R I
A G E N T A G E N T
A G E N TA G E N TA G E N T
A G E N T
B U N D L E
W O R K E RN O D E
W O R K E RN O D E
W O R K E RN O D E
W O R K E RN O D E
W O R K E RN O D E
W O R K E RN O D E
Agent to Server: TLS
Bundle: AES 256/RSA 1024
Landing Zone: SOC2 Certified
S m a r t S e n s eA n a l y ti c s
37 © Hortonworks Inc. 2011 – 2017. All Rights Reserved
SmartSense Data Capture Architecture
L A N D I N G Z O N E
S E R V E RG AT E W AY
A M B A R I
A G E N T A G E N T
A G E N TA G E N TA G E N T
A G E N T
B U N D L E
W O R K E RN O D E
W O R K E RN O D E
W O R K E RN O D E
W O R K E RN O D E
W O R K E RN O D E
W O R K E RN O D E
Agent to Server: TLS
Bundle: AES 256/RSA 1024
Server to Gateway: TLS
Landing Zone: SOC2 Certified
S m a r t S e n s eA n a l y ti c s
38 © Hortonworks Inc. 2011 – 2017. All Rights Reserved
SmartSense Data Capture Architecture
L A N D I N G Z O N E
S E R V E RG AT E W AY
A M B A R I
A G E N T A G E N T
A G E N TA G E N TA G E N T
A G E N T
B U N D L E
W O R K E RN O D E
W O R K E RN O D E
W O R K E RN O D E
W O R K E RN O D E
W O R K E RN O D E
W O R K E RN O D E
Agent to Server: TLS
Bundle: AES 256/RSA 1024
Server to Gateway: TLS
Landing Zone: SOC2 Certified
Gateway to Landing Zone: HTTPS (TLS 1.2) or SFTP (AES)
S m a r t S e n s eA n a l y ti c s
39 © Hortonworks Inc. 2011 – 2017. All Rights Reserved
Increase Performance
Prevent Issues
Accelerate Case Resolution
Understand Your Cluster
40 © Hortonworks Inc. 2011 – 2017. All Rights Reserved
“Who’s creating all of these small files in HDFS!?”
“What are my top 10 most active users, and longest running jobs?”
“How much should I charge users for their cluster resource use?”
SmartSense Today – Understand Your Cluster
41 © Hortonworks Inc. 2011 – 2017. All Rights Reserved
SmartSense Today – Understand Your ClusterOn-Premise
Chargeback Reporting
42 © Hortonworks Inc. 2011 – 2017. All Rights Reserved
SmartSense Today – Understand Your ClusterOn-Premise
Chargeback Reporting
HDFS Dashboards
43 © Hortonworks Inc. 2011 – 2017. All Rights Reserved
SmartSense Today – Understand Your ClusterOn-Premise
Chargeback Reporting
HDFS Dashboards
YARN Dashboards
44 © Hortonworks Inc. 2011 – 2017. All Rights Reserved
Impact of Hortonworks SmartSense
Without SmartSense
With SmartSense
0200400600800
100012001400
Concurrent Jobs
B U N D L E
2X Throughput Improvement
Address 30% of Issues
Configuration Issues
Avoid 10% of Sev1 Issues
Production Down
Single-Bundle Case Resolution 25% of the Time
SmartSense Troubleshooting Bundle
45 © Hortonworks Inc. 2011 – 2017. All Rights Reserved
Questions
46 © Hortonworks Inc. 2011 – 2017. All Rights Reserved
Hortonworks Connected Data Platforms and Solutions
Data Services
Hortonworks Solutions
Enterprise DataWarehouse Optimization
Cyber Security andThreat Management
Internet of Thingsand Streaming Analytics
Data CenterHortonworks Data Suite
HDFHDP
HortonworksConnection
CloudHortonworks Data CloudAWS HDInsight
Hortonworks ConnectionEnablement Subscription
SmartSense™
Premier Operational Support
Educational Services
Professional Services
Community Connection
© DataWorks Summit and Hadoop Summit 2017. All Rights Reserved47
DataWorks Summit 2017
http://dataworkssummit.com
top related