mysql london tech tour march 2015 - big data

44
Copyright © 2015, Oracle and/or its affiliates. All rights reserved. | Unlocking Big Data Insights with MySQL Mark Swarbrick Principle Presales Consultant MySQL UK&I March 2015

Upload: mark-swarbrick

Post on 03-Aug-2015

88 views

Category:

Technology


0 download

TRANSCRIPT

Copyright  ©  2015,  Oracle  and/or  its  affiliates.  All  rights  reserved.    |  

Unlocking  Big  Data    Insights  with  MySQL  

Mark  Swarbrick  Principle  Presales  Consultant  MySQL  UK&I  March  2015  

1  

Copyright  ©  2015,  Oracle  and/or  its  affiliates.  All  rights  reserved.    |  

Safe  Harbor  Statement  The  following  is  intended  to  outline  our  general  product  direcRon.  It  is  intended  for  informaRon  purposes  only,  and  may  not  be  incorporated  into  any  contract.  It  is  not  a  commitment  to  deliver  any  material,  code,  or  funcRonality,  and  should  not  be  relied  upon  in  making  purchasing  decisions.  The  development,  release,  and  Rming  of  any  features  or  funcRonality  described  for  Oracle’s  products  remains  at  the  sole  discreRon  of  Oracle.  

2  

Copyright  ©  2015,  Oracle  and/or  its  affiliates.  All  rights  reserved.    |  

Cloud  

Web  &  Enterprise     OEM  &  ISVs  

Industry  Leaders  Rely  on  MySQL  

3  

Copyright  ©  2015,  Oracle  and/or  its  affiliates.  All  rights  reserved.    |  

MySQL  Powers  The  Web  

 Over  500  million  Tweets/day.  143,200  Tweets/sec  in  Aug  2013        ”Many  petabytes”  of  data.  11.2  Million  Row  changes  &  2.5  billion    rows  read  /sec  handled  in  MySQL        6  billion  hours  of  video  watched  each  month        Globally-­‐distributed  database  with  100  terabytes  of  user-­‐related    data  based  on  MySQL  Cluster  

4  

Copyright  ©  2015,  Oracle  and/or  its  affiliates.  All  rights  reserved.    |  

An  Avalanche  of  Data  

Copyright  ©  2015,  Oracle  and/or  its  affiliates.  All  rights  reserved.    |  

Why  Is  Big  Data  Important?  Value  CreaRon  

HEALTH  CARE     MANUFACTURING   COMMUNICATIONS  

“In  a  big  data  world,  a  compeRtor  that  fails  to  sufficiently  develop  its  capabiliRes  will  be  le_  behind.”  

Reduce  PrescripRon  Fraud  

Accelerate  Test  Cycles  to  Reduce  

Backlog  

Offering  New  Services  based  on  LocaRon  

Data    

McKinsey  Global  Ins0tute  

RETAIL  

Becer  Predict  Product  Success  

PUBLIC  SECTOR  

Improve  Student  Outcomes  

Copyright  ©  2015,  Oracle  and/or  its  affiliates.  All  rights  reserved.    |  

Create Value

Big  Data  What  It  Is,  What  it  Means  

Volume Variety Velocity

Copyright  ©  2015,  Oracle  and/or  its  affiliates.  All  rights  reserved.    |  

Big  Data:  Strategic  TransformaRon  

•  From  REPORTING      to  ANALYTICS  

•  From  REAR-­‐VIEW  MIRROR    to    PREDICT/EXPLORE  

•  From    SOME  DATA        to    BIG  DATA  

Copyright  ©  2015,  Oracle  and/or  its  affiliates.  All  rights  reserved.    |  

What’s  Changed?  •  Enablers  

– DigiRzaRon  –  nearly  everything  has  a  digital  heartbeat  – Ability  to  store  much  larger  data  volumes  (distributed  file  systems)  – Ability  to  process  much  larger  data  volumes  (parallel  processing)  

• Why  is  this  different  from  BI/DW?  – Business  formulated  quesRons  to  ask  upfront  – Drove  what  was  data  collected,  data  model,  query  design  

Ø Big  Data  Enables  what-­‐if  analysis,  real-­‐Rme  discovery  

Copyright  ©  2015,  Oracle  and/or  its  affiliates.  All  rights  reserved.    |  

Big  Data  AdopRon  • Web  RecommendaRons  •  SenRment  Analysis  • MarkeRng  Campaign  Analysis  •  Customer  Churn  Modeling  •  Fraud  DetecRon  •  Research  and  Development  •  Risk  Modeling  • Machine  Learning  

Copyright  2013,  Oracle  and/or  its  affiliates.  All  rights  reserved.  

Copyright  ©  2015,  Oracle  and/or  its  affiliates.  All  rights  reserved.    |  

Big  Data  Can  Help  You  …  

Chief  MarkeRng  Officer  

 

Sell  More    

Chief  Financial  Officer  

 

Manage    Risk  

Chief  InformaRon  Officer  

 

Reduce  Cost  

Copyright  ©  2015,  Oracle  and/or  its  affiliates.  All  rights  reserved.    |  

Leading  Use-­‐Case,  On-­‐Line  Retail  Users  

Browsing  

RecommendaRons  Profile,  Purchase  History  

Web  Logs:  Pages  Viewed  

Comments  Posted  Social  media  updates  

Preferences  Brands  “Liked”  

RecommendaRons  

Copyright  ©  2015,  Oracle  and/or  its  affiliates.  All  rights  reserved.    |  

Why  Hadoop?  •  Scales  to  thousands  of  nodes,  PB  of  structured  and  unstructured  data  

– Combines  data  from  mulRple  sources,  schema-­‐less  – Run  queries  against  all  of  the  data  

• Runs  on  commodity  servers,  handle  storage  and  processing  • Data  replicated,  self-­‐healing  •  IniRally  just  batch  (Map/Reduce)  processing  

– Extending  with  interacRve  querying,  via  Apache  Drill,  Cloudera  Impala,  SRnger  etc.  

Copyright 2014, Oracle and/or its affiliates. All rights reserved.

Copyright  ©  2015,  Oracle  and/or  its  affiliates.  All  rights  reserved.    |  

Big  Data  Lifecycle  BeBer  Decisions  Using  Big  Data  

Oracle  ConfidenRal  |  #BeyondBigData   14  Copyright  ©  2014  Oracle  and/or  its  affiliates.  All  rights  reserved.    |  

ANALYZE  

DECIDE   ACQUIRE  

ORGANIZE  

CREATE  VALUE  FROM  DATA  

Copyright  ©  2015,  Oracle  and/or  its  affiliates.  All  rights  reserved.    |  

Big  Data  Lifecycle  BeBer  Decisions  Using  Big  Data  

Oracle  ConfidenRal  |  #BeyondBigData   15  Copyright  ©  2014  Oracle  and/or  its  affiliates.  All  rights  reserved.    |  

ANALYZE  

DECIDE   ACQUIRE  

ORGANIZE  

CREATE  VALUE  FROM  DATA  

Copyright  ©  2015,  Oracle  and/or  its  affiliates.  All  rights  reserved.    |  

Big  Data  Lifecycle  BeBer  Decisions  Using  Big  Data  

Oracle  ConfidenRal  |  #BeyondBigData   16  Copyright  ©  2014  Oracle  and/or  its  affiliates.  All  rights  reserved.    |  

ACQUIRE  

CREATE  VALUE  FROM  DATA  

NoSQL  Interfaces    MySQL  Database  MySQL  Cluster  MySQL  Fabric  

Copyright  ©  2015,  Oracle  and/or  its  affiliates.  All  rights  reserved.    |  

MySQL  NoSQL  Interfaces:  Fast,  Flexible,  Safe  

Blazing  Fast  Key/Value  Queries  

Fully  TransacRonal/  ACID  

NoSQL  And  SQL  Across  the  same  

data  Set  

17  

Combined  with  Schema  Flexibility:  Online  DDL  

Copyright  ©  2015,  Oracle  and/or  its  affiliates.  All  rights  reserved.    |  

MySQL  Strategy:  Best  of  Both  Worlds  

•  Mix  Key  Value  &  RelaRonal  Queries  

•  TransacRonal  Integrity  

•  Complex  Queries  

•  Standards  &  Skillsets  

Copyright  2014,  Oracle  and/or  its  affiliates.  All  rights  reserved.  

Copyright  ©  2015,  Oracle  and/or  its  affiliates.  All  rights  reserved.    |  

Performance  Up  to  9X  higher  ”SET/INSERT”  Throughput  

19  

Copyright  ©  2015,  Oracle  and/or  its  affiliates.  All  rights  reserved.    |  

MySQL  5.7:  InnoDB,  NoSQL  With  Memcached    6x  Faster  than  MySQL  5.6  Thank  you,  Facebook                            

0  

200,000  

400,000  

600,000  

800,000  

1,000,000  

1,200,000  

8   16   32   64   128   256   512   1,024  

Que

ries  p

er  Secon

d  

Connec`ons  

MySQL  5.7  vs  5.6  -­‐  InnoDB  &  Memcached  

MySQL  5.7  

MySQL  5.6  

1  Million  QPS  

Intel(R)  Xeon(R)  CPU  X7560  x86_64  4  sockets  x  10  cores-­‐HT  (80  CPU  threads)  2.3  GHz,  512  GB  RAM  Oracle  Linux  6.5  

Copyright  ©  2015,  Oracle  and/or  its  affiliates.  All  rights  reserved.    |  

MySQL  Cluster:  MulRple  NoSQL  Interfaces  Mix  &  Match  

Copyright 2014, Oracle and/or its affiliates. All rights reserved.

Copyright  ©  2015,  Oracle  and/or  its  affiliates.  All  rights  reserved.    |  

MySQL  Cluster  Benchmark  

•  NoSQL  C++  API,  flexaSynch  benchmark  

•  30  x  Intel  E5-­‐2600  Intel  Servers,  2  socket,  64GB  

•  ACID  TransacRons,  with  Synchronous  ReplicaRon  

Copyright 2014, Oracle and/or its affiliates. All rights reserved.

Copyright  ©  2015,  Oracle  and/or  its  affiliates.  All  rights  reserved.    |  

MySQL  Cluster  Schema  Flexibility  Configure  with  or  without  Schema  

Copyright  2014,  Oracle  and/or  its  affiliates.  All  rights  reserved.  

<town:maidenhead,SL6>

key value

<town:maidenhead,SL6>

key value

Key   Value  

town:maidenhead   SL6  

generic  table  

ApplicaRon  view  

SQL  view  

Copyright  ©  2015,  Oracle  and/or  its  affiliates.  All  rights  reserved.    |  

MySQL  Fabric      Scale  out  with  Data  Sharding  +  High  Availability  

•  Scale-­‐out  through  sharding  •  Read  AND  Write  •  Standard  framework,    

no  more  custom  soluRons  

• HA  out  of  the  box  •  On  top  of  ReplicaRon  •  AutomaRc  failover  •  AutomaRc  rouRng  

MySQL  Fabric  

Connector  

ApplicaRon  

Read-­‐slaves  

mappings  

SQL  

Master  group  Read-­‐slaves  Master  group  

Copyright  ©  2015,  Oracle  and/or  its  affiliates.  All  rights  reserved.    |  

Big  Data  Lifecycle  BeBer  Decisions  Using  Big  Data  

Oracle  ConfidenRal  |  #BeyondBigData   25  Copyright  ©  2014  Oracle  and/or  its  affiliates.  All  rights  reserved.    |  

ACQUIRE  

ORGANIZE  

CREATE  VALUE  FROM  DATA  

Import  Data  Apache  Sqoop  MySQL  Hadoop  Applier  

Copyright  ©  2015,  Oracle  and/or  its  affiliates.  All  rights  reserved.    |  

Apache  Sqoop  

• Apache  TLP,  part  of  Hadoop  project  •  Developed  by  Cloudera  

• Bulk  data  import  and  export  •  Between  Hadoop  (HDFS)  and  external  data  stores  

•   JDBC  Connector  architecture  •  Supports  plug-­‐ins  for  specific  funcRonality  

• “Fast  Path”  Connector  developed  for  MySQL  

Copyright  ©  2015,  Oracle  and/or  its  affiliates.  All  rights  reserved.    |  

MySQL  Applier  for  Hadoop  

Copyright  2014,  Oracle  and/or  its  affiliates.  All  rights  reserved.  

• Real-­‐Rme  streaming  of  events  from  MySQL  to  Hadoop  

Ø Supports  move  towards  “Speed  of  Thought”  analyRcs  

• Connects  to  the  binary  log,  writes  events  to  HDFS  via  libhdfs  library  

• Each  database  table  mapped  to  a  Hive  data  warehouse  directory  

• Enables  eco-­‐system  of  Hadoop  tools  to  integrate  with  MySQL  data    

• Available  for  download  now:  labs.mysql.com  

labs.mysql.com  

Copyright  ©  2015,  Oracle  and/or  its  affiliates.  All  rights  reserved.    |  

•  Replicates  rows  inserted  into  a  table  in  MySQL  to  Hadoop  Distributed  File  System  •  Uses  an  API  provided    by  libhdfs,  a  C  library  to  manipulate  files  in  HDFS  

•  The  library  comes  pre-­‐compiled  with  Hadoop  DistribuRons  

•  Connects  to  the  MySQL  master  (or  reads  the  binary  log  generated  by  MySQL)  to:  •  Fetch  the  row  insert  events  occurring  on  the  master  •  Decode  these  events,  extracRng  data  inserted  into  each  field  of  the  row  •  Separate  the  data  by  the  desired  field  delimiters  and  row  delimiters  •  Use  content  handlers  to  get  it  in  the  format  required  •  Append  it  to  a  text  file  in  HDFS  

Copyright  2014,  Oracle  and/or  its  affiliates.  All  rights  reserved.  

MySQL  Applier  for  Hadoop:  ImplementaRon  labs.mysql.com  

Copyright  ©  2015,  Oracle  and/or  its  affiliates.  All  rights  reserved.    |  

MySQL  Applier  for  Hadoop  

29  

labs.mysql.com  

Copyright  ©  2015,  Oracle  and/or  its  affiliates.  All  rights  reserved.    |  

Big  Data  Lifecycle  BeBer  Decisions  Using  Big  Data  

Oracle  ConfidenRal  |  #BeyondBigData   30  Copyright  ©  2014  Oracle  and/or  its  affiliates.  All  rights  reserved.    |  

ANALYZE  

DECIDE  

CREATE  VALUE  FROM  DATA  

Analyze  Export  Data  Decide  

Copyright  ©  2015,  Oracle  and/or  its  affiliates.  All  rights  reserved.    |  

Analyze  Big  Data  in  Hadoop  

Copyright  ©  2015,  Oracle  and/or  its  affiliates.  All  rights  reserved.    |  

MySQL  ReporRng  Database  for  BI  

Copyright 2014, Oracle and/or its affiliates. All rights reserved.

Copyright  ©  2015,  Oracle  and/or  its  affiliates.  All  rights  reserved.    |  

Management  Tools  Advanced  Features   Support  

• Scalability  • High  Availability  • Security  • Audit  • EncrypRon  

• Monitoring  • Backup  • Development  • AdministraRon  • MigraRon  

• Technical  Support  • ConsultaRve  Support  • Oracle  CerRficaRons  

Data  Analysis  with  MySQL  Enterprise  EdiRon  

33  

Copyright  ©  2015,  Oracle  and/or  its  affiliates.  All  rights  reserved.    |  

MySQL  Enterprise  Monitor  with  Query  Analyzer    Tune  AnalyRcal  Queries                  Enhance  DevOps  Agility  

Copyright 2014, Oracle and/or its affiliates. All rights reserved.

Copyright  ©  2015,  Oracle  and/or  its  affiliates.  All  rights  reserved.    |  

Scaling,  Security  and  Data  ProtecRon    

MySQL  Enterprise  Scalability    MySQL  Enterprise  Backup    MySQL  Enterprise  Security    MySQL  Enterprise  EncrypRon    MySQL  Enterprise  Audit    

Copyright  ©  2015,  Oracle  and/or  its  affiliates.  All  rights  reserved.    |  

MySQL  Enterprise  Support  •  Largest  MySQL  engineering  and  support  organizaRon  •  Backed  by  the  MySQL  developers  •  World-­‐class  support,  in  29  languages  •  Hot  fixes  &  maintenance  releases  •  24x7x365  •  Unlimited  incidents  •  ConsultaRve  support  •  Global  scale  and  reach   Get  immediate  help  for  any  MySQL  

issue,  plus  expert  advice

36  

Copyright  ©  2015,  Oracle  and/or  its  affiliates.  All  rights  reserved.    |  

MySQL  ConsultaRve  Support    Make  the  Most  of  your  Deployments  

• Remote  troubleshooRng  • ReplicaRon  review  • ParRRoning  review  •  Schema  review  • Query  review  • Performance  tuning  •  ...and  more  

37  

Copyright  ©  2015,  Oracle  and/or  its  affiliates.  All  rights  reserved.    |  

Why  MySQL  Enterprise  EdiRon?  In  Addi`on  to  all  the  MySQL  Features  you  Love  

Insure  Your  Deployments    Get  the  Best  Results    Delight  Customers  

Improve  Performance  &  Scalability  

Enhance  Agility  &  Produc`vity  

Reduce  TCO  

Mi`gate  Risks  

Get  Immediate  Help  if/when  

Needed  

Increase  Customer  Sa`sfac`on  

38  

Copyright  ©  2015,  Oracle  and/or  its  affiliates.  All  rights  reserved.    |  

Company  Overview      boo-­‐box  is  one  of  the  largest  adverRsing  networks  in  South  America,  with  a  focus  on  the  Brazilian  social  media  market.    Applica`on    boo-­‐box  relies  on  MySQL  and  Hadoop  to  display  1  billion  adverRsements  to  60  million  people  across  430,000  web  sites  and  social  network  profiles  every  month.    Why  MySQL?  "MySQL  is  a  core  part  of  our  big  data  strategy.  Simple  integra0on  with  Hadoop  enables  us  to  improve  our  digital  adver0sing  service  and  grow  our  business  with  maximum  speed  and  agility.“  Josafá  Santos,  IT  Manager,  boo-­‐box    

boo-­‐box      

39  

Copyright  ©  2015,  Oracle  and/or  its  affiliates.  All  rights  reserved.    |  

Leveraging  Other  Oracle  SoluRons  For  Data  Aquired  in  MySQL  

Acquire   Organize   Analyze   Decide  

Web  Data  Acquired  in  MySQL  

Analyzed  with  oracle  Exadata  

Organized  with  Oracle  Big  Data  

Appliance  

Decide  Using  the  power  of  Oracle  

ExalyRcs  

Copyright 2014, Oracle and/or its affiliates. All rights reserved.

Copyright  ©  2015,  Oracle  and/or  its  affiliates.  All  rights  reserved.    |  

           Oracle  Enterprise  Manager  for  MySQL  

41  

Performance  Security  

Availability  

• Availability  monitoring  • Performance  monitoring  • ConfiguraRon  monitoring  • All  available  metrics  collected  

– Allowing  for  custom  threshold  based  incident  reports  

• MySQL  auto-­‐detecRon    

Copyright  ©  2015,  Oracle  and/or  its  affiliates.  All  rights  reserved.    |  

MySQL  Enterprise  Oracle  CerRficaRons    • Oracle  Linux    • Oracle  VM  • Oracle  GoldenGate  • Oracle  Solaris  Clustering  • Oracle  Clusterware  

• Oracle  Enterprise  Manager  • Oracle  Fusion  Middleware  • Oracle  Audit  Vault  &  Database  Firewall  • Oracle  Secure  Backup  • MyOracle  Online  Support  

MySQL  Integrates  into  the  Oracle  Environment  

Copyright  ©  2015,  Oracle  and/or  its  affiliates.  All  rights  reserved.    |  

Summary  •  Create  value  from  Big  Data  with  MySQL  

• MySQL  +  Hadoop:  widely  deployed  soluRon  

•  “Best  of  both  worlds”  SQL  +  NoSQL  Access  •  Scale  Out  &  data  sharding  with  MySQL  Fabric  

•  Tools  and  experRse  to  support  you  •  End  to  end  Oracle  soluRons  for  Big  Data  

Copyright  2014,  Oracle  and/or  its  affiliates.  All  rights  reserved.