mysql london tech tour march 2015 - big data
TRANSCRIPT
Copyright © 2015, Oracle and/or its affiliates. All rights reserved. |
Unlocking Big Data Insights with MySQL
Mark Swarbrick Principle Presales Consultant MySQL UK&I March 2015
1
Copyright © 2015, Oracle and/or its affiliates. All rights reserved. |
Safe Harbor Statement The following is intended to outline our general product direcRon. It is intended for informaRon purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or funcRonality, and should not be relied upon in making purchasing decisions. The development, release, and Rming of any features or funcRonality described for Oracle’s products remains at the sole discreRon of Oracle.
2
Copyright © 2015, Oracle and/or its affiliates. All rights reserved. |
Cloud
Web & Enterprise OEM & ISVs
Industry Leaders Rely on MySQL
3
Copyright © 2015, Oracle and/or its affiliates. All rights reserved. |
MySQL Powers The Web
Over 500 million Tweets/day. 143,200 Tweets/sec in Aug 2013 ”Many petabytes” of data. 11.2 Million Row changes & 2.5 billion rows read /sec handled in MySQL 6 billion hours of video watched each month Globally-‐distributed database with 100 terabytes of user-‐related data based on MySQL Cluster
4
Copyright © 2015, Oracle and/or its affiliates. All rights reserved. |
Why Is Big Data Important? Value CreaRon
HEALTH CARE MANUFACTURING COMMUNICATIONS
“In a big data world, a compeRtor that fails to sufficiently develop its capabiliRes will be le_ behind.”
Reduce PrescripRon Fraud
Accelerate Test Cycles to Reduce
Backlog
Offering New Services based on LocaRon
Data
McKinsey Global Ins0tute
RETAIL
Becer Predict Product Success
PUBLIC SECTOR
Improve Student Outcomes
Copyright © 2015, Oracle and/or its affiliates. All rights reserved. |
Create Value
Big Data What It Is, What it Means
Volume Variety Velocity
Copyright © 2015, Oracle and/or its affiliates. All rights reserved. |
Big Data: Strategic TransformaRon
• From REPORTING to ANALYTICS
• From REAR-‐VIEW MIRROR to PREDICT/EXPLORE
• From SOME DATA to BIG DATA
Copyright © 2015, Oracle and/or its affiliates. All rights reserved. |
What’s Changed? • Enablers
– DigiRzaRon – nearly everything has a digital heartbeat – Ability to store much larger data volumes (distributed file systems) – Ability to process much larger data volumes (parallel processing)
• Why is this different from BI/DW? – Business formulated quesRons to ask upfront – Drove what was data collected, data model, query design
Ø Big Data Enables what-‐if analysis, real-‐Rme discovery
Copyright © 2015, Oracle and/or its affiliates. All rights reserved. |
Big Data AdopRon • Web RecommendaRons • SenRment Analysis • MarkeRng Campaign Analysis • Customer Churn Modeling • Fraud DetecRon • Research and Development • Risk Modeling • Machine Learning
Copyright 2013, Oracle and/or its affiliates. All rights reserved.
Copyright © 2015, Oracle and/or its affiliates. All rights reserved. |
Big Data Can Help You …
Chief MarkeRng Officer
Sell More
Chief Financial Officer
Manage Risk
Chief InformaRon Officer
Reduce Cost
Copyright © 2015, Oracle and/or its affiliates. All rights reserved. |
Leading Use-‐Case, On-‐Line Retail Users
Browsing
RecommendaRons Profile, Purchase History
Web Logs: Pages Viewed
Comments Posted Social media updates
Preferences Brands “Liked”
RecommendaRons
Copyright © 2015, Oracle and/or its affiliates. All rights reserved. |
Why Hadoop? • Scales to thousands of nodes, PB of structured and unstructured data
– Combines data from mulRple sources, schema-‐less – Run queries against all of the data
• Runs on commodity servers, handle storage and processing • Data replicated, self-‐healing • IniRally just batch (Map/Reduce) processing
– Extending with interacRve querying, via Apache Drill, Cloudera Impala, SRnger etc.
Copyright 2014, Oracle and/or its affiliates. All rights reserved.
Copyright © 2015, Oracle and/or its affiliates. All rights reserved. |
Big Data Lifecycle BeBer Decisions Using Big Data
Oracle ConfidenRal | #BeyondBigData 14 Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
ANALYZE
DECIDE ACQUIRE
ORGANIZE
CREATE VALUE FROM DATA
Copyright © 2015, Oracle and/or its affiliates. All rights reserved. |
Big Data Lifecycle BeBer Decisions Using Big Data
Oracle ConfidenRal | #BeyondBigData 15 Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
ANALYZE
DECIDE ACQUIRE
ORGANIZE
CREATE VALUE FROM DATA
Copyright © 2015, Oracle and/or its affiliates. All rights reserved. |
Big Data Lifecycle BeBer Decisions Using Big Data
Oracle ConfidenRal | #BeyondBigData 16 Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
ACQUIRE
CREATE VALUE FROM DATA
NoSQL Interfaces MySQL Database MySQL Cluster MySQL Fabric
Copyright © 2015, Oracle and/or its affiliates. All rights reserved. |
MySQL NoSQL Interfaces: Fast, Flexible, Safe
Blazing Fast Key/Value Queries
Fully TransacRonal/ ACID
NoSQL And SQL Across the same
data Set
17
Combined with Schema Flexibility: Online DDL
Copyright © 2015, Oracle and/or its affiliates. All rights reserved. |
MySQL Strategy: Best of Both Worlds
• Mix Key Value & RelaRonal Queries
• TransacRonal Integrity
• Complex Queries
• Standards & Skillsets
Copyright 2014, Oracle and/or its affiliates. All rights reserved.
Copyright © 2015, Oracle and/or its affiliates. All rights reserved. |
Performance Up to 9X higher ”SET/INSERT” Throughput
19
Copyright © 2015, Oracle and/or its affiliates. All rights reserved. |
MySQL 5.7: InnoDB, NoSQL With Memcached 6x Faster than MySQL 5.6 Thank you, Facebook
0
200,000
400,000
600,000
800,000
1,000,000
1,200,000
8 16 32 64 128 256 512 1,024
Que
ries p
er Secon
d
Connec`ons
MySQL 5.7 vs 5.6 -‐ InnoDB & Memcached
MySQL 5.7
MySQL 5.6
1 Million QPS
Intel(R) Xeon(R) CPU X7560 x86_64 4 sockets x 10 cores-‐HT (80 CPU threads) 2.3 GHz, 512 GB RAM Oracle Linux 6.5
Copyright © 2015, Oracle and/or its affiliates. All rights reserved. |
MySQL Cluster: MulRple NoSQL Interfaces Mix & Match
Copyright 2014, Oracle and/or its affiliates. All rights reserved.
Copyright © 2015, Oracle and/or its affiliates. All rights reserved. |
MySQL Cluster Benchmark
• NoSQL C++ API, flexaSynch benchmark
• 30 x Intel E5-‐2600 Intel Servers, 2 socket, 64GB
• ACID TransacRons, with Synchronous ReplicaRon
Copyright 2014, Oracle and/or its affiliates. All rights reserved.
Copyright © 2015, Oracle and/or its affiliates. All rights reserved. |
MySQL Cluster Schema Flexibility Configure with or without Schema
Copyright 2014, Oracle and/or its affiliates. All rights reserved.
<town:maidenhead,SL6>
key value
<town:maidenhead,SL6>
key value
Key Value
town:maidenhead SL6
generic table
ApplicaRon view
SQL view
Copyright © 2015, Oracle and/or its affiliates. All rights reserved. |
MySQL Fabric Scale out with Data Sharding + High Availability
• Scale-‐out through sharding • Read AND Write • Standard framework,
no more custom soluRons
• HA out of the box • On top of ReplicaRon • AutomaRc failover • AutomaRc rouRng
MySQL Fabric
Connector
ApplicaRon
Read-‐slaves
mappings
SQL
Master group Read-‐slaves Master group
Copyright © 2015, Oracle and/or its affiliates. All rights reserved. |
Big Data Lifecycle BeBer Decisions Using Big Data
Oracle ConfidenRal | #BeyondBigData 25 Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
ACQUIRE
ORGANIZE
CREATE VALUE FROM DATA
Import Data Apache Sqoop MySQL Hadoop Applier
Copyright © 2015, Oracle and/or its affiliates. All rights reserved. |
Apache Sqoop
• Apache TLP, part of Hadoop project • Developed by Cloudera
• Bulk data import and export • Between Hadoop (HDFS) and external data stores
• JDBC Connector architecture • Supports plug-‐ins for specific funcRonality
• “Fast Path” Connector developed for MySQL
Copyright © 2015, Oracle and/or its affiliates. All rights reserved. |
MySQL Applier for Hadoop
Copyright 2014, Oracle and/or its affiliates. All rights reserved.
• Real-‐Rme streaming of events from MySQL to Hadoop
Ø Supports move towards “Speed of Thought” analyRcs
• Connects to the binary log, writes events to HDFS via libhdfs library
• Each database table mapped to a Hive data warehouse directory
• Enables eco-‐system of Hadoop tools to integrate with MySQL data
• Available for download now: labs.mysql.com
labs.mysql.com
Copyright © 2015, Oracle and/or its affiliates. All rights reserved. |
• Replicates rows inserted into a table in MySQL to Hadoop Distributed File System • Uses an API provided by libhdfs, a C library to manipulate files in HDFS
• The library comes pre-‐compiled with Hadoop DistribuRons
• Connects to the MySQL master (or reads the binary log generated by MySQL) to: • Fetch the row insert events occurring on the master • Decode these events, extracRng data inserted into each field of the row • Separate the data by the desired field delimiters and row delimiters • Use content handlers to get it in the format required • Append it to a text file in HDFS
Copyright 2014, Oracle and/or its affiliates. All rights reserved.
MySQL Applier for Hadoop: ImplementaRon labs.mysql.com
Copyright © 2015, Oracle and/or its affiliates. All rights reserved. |
MySQL Applier for Hadoop
29
labs.mysql.com
Copyright © 2015, Oracle and/or its affiliates. All rights reserved. |
Big Data Lifecycle BeBer Decisions Using Big Data
Oracle ConfidenRal | #BeyondBigData 30 Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
ANALYZE
DECIDE
CREATE VALUE FROM DATA
Analyze Export Data Decide
Copyright © 2015, Oracle and/or its affiliates. All rights reserved. |
MySQL ReporRng Database for BI
Copyright 2014, Oracle and/or its affiliates. All rights reserved.
Copyright © 2015, Oracle and/or its affiliates. All rights reserved. |
Management Tools Advanced Features Support
• Scalability • High Availability • Security • Audit • EncrypRon
• Monitoring • Backup • Development • AdministraRon • MigraRon
• Technical Support • ConsultaRve Support • Oracle CerRficaRons
Data Analysis with MySQL Enterprise EdiRon
33
Copyright © 2015, Oracle and/or its affiliates. All rights reserved. |
MySQL Enterprise Monitor with Query Analyzer Tune AnalyRcal Queries Enhance DevOps Agility
Copyright 2014, Oracle and/or its affiliates. All rights reserved.
Copyright © 2015, Oracle and/or its affiliates. All rights reserved. |
Scaling, Security and Data ProtecRon
MySQL Enterprise Scalability MySQL Enterprise Backup MySQL Enterprise Security MySQL Enterprise EncrypRon MySQL Enterprise Audit
Copyright © 2015, Oracle and/or its affiliates. All rights reserved. |
MySQL Enterprise Support • Largest MySQL engineering and support organizaRon • Backed by the MySQL developers • World-‐class support, in 29 languages • Hot fixes & maintenance releases • 24x7x365 • Unlimited incidents • ConsultaRve support • Global scale and reach Get immediate help for any MySQL
issue, plus expert advice
36
Copyright © 2015, Oracle and/or its affiliates. All rights reserved. |
MySQL ConsultaRve Support Make the Most of your Deployments
• Remote troubleshooRng • ReplicaRon review • ParRRoning review • Schema review • Query review • Performance tuning • ...and more
37
Copyright © 2015, Oracle and/or its affiliates. All rights reserved. |
Why MySQL Enterprise EdiRon? In Addi`on to all the MySQL Features you Love
Insure Your Deployments Get the Best Results Delight Customers
Improve Performance & Scalability
Enhance Agility & Produc`vity
Reduce TCO
Mi`gate Risks
Get Immediate Help if/when
Needed
Increase Customer Sa`sfac`on
38
Copyright © 2015, Oracle and/or its affiliates. All rights reserved. |
Company Overview boo-‐box is one of the largest adverRsing networks in South America, with a focus on the Brazilian social media market. Applica`on boo-‐box relies on MySQL and Hadoop to display 1 billion adverRsements to 60 million people across 430,000 web sites and social network profiles every month. Why MySQL? "MySQL is a core part of our big data strategy. Simple integra0on with Hadoop enables us to improve our digital adver0sing service and grow our business with maximum speed and agility.“ Josafá Santos, IT Manager, boo-‐box
boo-‐box
39
Copyright © 2015, Oracle and/or its affiliates. All rights reserved. |
Leveraging Other Oracle SoluRons For Data Aquired in MySQL
Acquire Organize Analyze Decide
Web Data Acquired in MySQL
Analyzed with oracle Exadata
Organized with Oracle Big Data
Appliance
Decide Using the power of Oracle
ExalyRcs
Copyright 2014, Oracle and/or its affiliates. All rights reserved.
Copyright © 2015, Oracle and/or its affiliates. All rights reserved. |
Oracle Enterprise Manager for MySQL
41
Performance Security
Availability
• Availability monitoring • Performance monitoring • ConfiguraRon monitoring • All available metrics collected
– Allowing for custom threshold based incident reports
• MySQL auto-‐detecRon
Copyright © 2015, Oracle and/or its affiliates. All rights reserved. |
MySQL Enterprise Oracle CerRficaRons • Oracle Linux • Oracle VM • Oracle GoldenGate • Oracle Solaris Clustering • Oracle Clusterware
• Oracle Enterprise Manager • Oracle Fusion Middleware • Oracle Audit Vault & Database Firewall • Oracle Secure Backup • MyOracle Online Support
MySQL Integrates into the Oracle Environment
Copyright © 2015, Oracle and/or its affiliates. All rights reserved. |
Summary • Create value from Big Data with MySQL
• MySQL + Hadoop: widely deployed soluRon
• “Best of both worlds” SQL + NoSQL Access • Scale Out & data sharding with MySQL Fabric
• Tools and experRse to support you • End to end Oracle soluRons for Big Data
Copyright 2014, Oracle and/or its affiliates. All rights reserved.