beyond tco
TRANSCRIPT
![Page 1: Beyond TCO](https://reader035.vdocuments.site/reader035/viewer/2022062401/586fdeb81a28ab18428b6d21/html5/thumbnails/1.jpg)
2016-06-29
Beyond TCOArchitecting Hadoop for adoption and data applications
Reid Levesque – Head, Solution Engineering
![Page 2: Beyond TCO](https://reader035.vdocuments.site/reader035/viewer/2022062401/586fdeb81a28ab18428b6d21/html5/thumbnails/2.jpg)
Introduction
![Page 3: Beyond TCO](https://reader035.vdocuments.site/reader035/viewer/2022062401/586fdeb81a28ab18428b6d21/html5/thumbnails/3.jpg)
Topics
Technology
Use cases
Deployment Impact Next
steps
![Page 4: Beyond TCO](https://reader035.vdocuments.site/reader035/viewer/2022062401/586fdeb81a28ab18428b6d21/html5/thumbnails/4.jpg)
Technology – Let’s talk Hadoop
![Page 5: Beyond TCO](https://reader035.vdocuments.site/reader035/viewer/2022062401/586fdeb81a28ab18428b6d21/html5/thumbnails/5.jpg)
Every company is a technology company…
some just don’t know it yet.
![Page 6: Beyond TCO](https://reader035.vdocuments.site/reader035/viewer/2022062401/586fdeb81a28ab18428b6d21/html5/thumbnails/6.jpg)
Traditional systems under pressure
Conventional wisdom• Put the code on an Application Server• Move the data to/from database• Move the data to/from NASReality check• This works well for small amounts of data• As data volumes increase this design falls apart
![Page 7: Beyond TCO](https://reader035.vdocuments.site/reader035/viewer/2022062401/586fdeb81a28ab18428b6d21/html5/thumbnails/7.jpg)
Hadoop to the rescue
Enterprise
![Page 8: Beyond TCO](https://reader035.vdocuments.site/reader035/viewer/2022062401/586fdeb81a28ab18428b6d21/html5/thumbnails/8.jpg)
How do we get Hadoop into the organization?
![Page 9: Beyond TCO](https://reader035.vdocuments.site/reader035/viewer/2022062401/586fdeb81a28ab18428b6d21/html5/thumbnails/9.jpg)
How about these use cases?
File archive +Hadoop
Data-intensive grid compute analytics
Database replacement
ETL off-load +Hadoop
+Hadoop
+Hadoop
• Data is online; no need for tape backup
• Cheaper than NAS / SAN
• Increased performance / scalability
• Metadata is easier to get; all the data is in one spot
• Improved performance
• Lower TCO
• Reduced dependence on proprietary software
• Reduce RDBMS licensing
• Reduced operational cost for analysis
• Improved functionality with stored XML
• Lower TCO
• Additional analytic capability
• Better hardware utilization
• Lower platform management
![Page 10: Beyond TCO](https://reader035.vdocuments.site/reader035/viewer/2022062401/586fdeb81a28ab18428b6d21/html5/thumbnails/10.jpg)
Not so much
File archive +Hadoop
Data-intensive grid compute analytics
Database replacement
ETL off-load +Hadoop
+Hadoop
+HadoopTCO
![Page 11: Beyond TCO](https://reader035.vdocuments.site/reader035/viewer/2022062401/586fdeb81a28ab18428b6d21/html5/thumbnails/11.jpg)
Which use case did work?
Current batch was taking 4 hours; which limited the way they did their job
Users wanted interactive response times to design and test their financial models
This was net new functionality that could only be achieved in Hadoop
![Page 12: Beyond TCO](https://reader035.vdocuments.site/reader035/viewer/2022062401/586fdeb81a28ab18428b6d21/html5/thumbnails/12.jpg)
Now TCO makes more sense
File archive +Hadoop
Data-intensive grid compute analytics
Database replacement
ETL off-load +Hadoop
+Hadoop
+Hadoop
With Hadoop TCO covered, previous use cases are now more compelling.
![Page 13: Beyond TCO](https://reader035.vdocuments.site/reader035/viewer/2022062401/586fdeb81a28ab18428b6d21/html5/thumbnails/13.jpg)
How do we deploy this?
![Page 14: Beyond TCO](https://reader035.vdocuments.site/reader035/viewer/2022062401/586fdeb81a28ab18428b6d21/html5/thumbnails/14.jpg)
Which distribution?
Pick one:
![Page 15: Beyond TCO](https://reader035.vdocuments.site/reader035/viewer/2022062401/586fdeb81a28ab18428b6d21/html5/thumbnails/15.jpg)
Time to pick the hardware
Is this true?
![Page 16: Beyond TCO](https://reader035.vdocuments.site/reader035/viewer/2022062401/586fdeb81a28ab18428b6d21/html5/thumbnails/16.jpg)
Commodity hardware + commodity networking = bad architecture
![Page 17: Beyond TCO](https://reader035.vdocuments.site/reader035/viewer/2022062401/586fdeb81a28ab18428b6d21/html5/thumbnails/17.jpg)
Before there was Hadoop, there were enterprise IT standards
To name a few conflicts during the rollout…
• Local account UID / names• OS settings• Root access• File locations• Standard mount sizes• Enterprise Active Directory• Monitoring systems
Hadoop is NOT flexible on deployment requirements
![Page 18: Beyond TCO](https://reader035.vdocuments.site/reader035/viewer/2022062401/586fdeb81a28ab18428b6d21/html5/thumbnails/18.jpg)
Who does the work?
Single team including:• Dedicated infrastructure team (Compute, Network, Data Center, Operations)• Dedicated Hadoop team (sysadmin/operations, engineering)• Hardware vendor engineers• Hadoop distribution engineers
![Page 19: Beyond TCO](https://reader035.vdocuments.site/reader035/viewer/2022062401/586fdeb81a28ab18428b6d21/html5/thumbnails/19.jpg)
Into production we go!
![Page 20: Beyond TCO](https://reader035.vdocuments.site/reader035/viewer/2022062401/586fdeb81a28ab18428b6d21/html5/thumbnails/20.jpg)
What was the impact?
![Page 21: Beyond TCO](https://reader035.vdocuments.site/reader035/viewer/2022062401/586fdeb81a28ab18428b6d21/html5/thumbnails/21.jpg)
Changing perceptions
![Page 22: Beyond TCO](https://reader035.vdocuments.site/reader035/viewer/2022062401/586fdeb81a28ab18428b6d21/html5/thumbnails/22.jpg)
Impact across the organization
Infrastructure• Networking / Data Center designs• Relationship with storage, cloud,
virtualization capabilities• Generating analytic use cases
Development• Mega-attractor for talent• Application consolidation• Shifting from IT to business focus
Management• Understanding (or accepting) new
paradigm• Cross-department architecture
alignment• Data-focus rather than application-
focus
Business• Continuously evolving understanding of
capability / possibilities• Next generation IT w/ rapidly evolving
ecosystem• Self-service innovation for business
users
![Page 23: Beyond TCO](https://reader035.vdocuments.site/reader035/viewer/2022062401/586fdeb81a28ab18428b6d21/html5/thumbnails/23.jpg)
Lessons Learned
Hadoop doesn’t remove hardware maintenance
Hadoop development is still development!
New paradigm – requires skilled developers
A whole new set of error messages to decode
There aren’t that many experts
![Page 24: Beyond TCO](https://reader035.vdocuments.site/reader035/viewer/2022062401/586fdeb81a28ab18428b6d21/html5/thumbnails/24.jpg)
Where do we go next?
![Page 25: Beyond TCO](https://reader035.vdocuments.site/reader035/viewer/2022062401/586fdeb81a28ab18428b6d21/html5/thumbnails/25.jpg)
Self-service tools
![Page 26: Beyond TCO](https://reader035.vdocuments.site/reader035/viewer/2022062401/586fdeb81a28ab18428b6d21/html5/thumbnails/26.jpg)
Selling Hadoop internally• This journey has taught me a lot about Hadoop; more than most people at the organization• The biggest tasks are educating the organization and doing simple things as a first step
![Page 27: Beyond TCO](https://reader035.vdocuments.site/reader035/viewer/2022062401/586fdeb81a28ab18428b6d21/html5/thumbnails/27.jpg)
Thank You