cloud & big data: lessons learnt

Download Cloud & Big Data: Lessons Learnt

Post on 16-Apr-2017

224 views

Category:

Technology

1 download

Embed Size (px)

TRANSCRIPT

  • Cloud-Based Architecture and Big Data

    Philip Balinov,

    DevOps Engineer

  • SOME DEFINITIONS

    def. CLOUD COMPUTING l - distributed computing over a networkl - the ability to run a program or application on many connected computers at

    the same time.

    def. BIG DATAl data sets so large and complex that it becomes difficult to process using

    traditional data processing applications

  • Retail

    Finance

    E-Commerce

    Telecommunication

    B2B

    Publishing/Media

    Government & NGO

    Automotive

    Travel

    KOMFO'S CLIENTS

  • KOMFO PLATFORM WORKFLOW OVERVIEW

  • EXTERNAL PROVIDERS

  • MODERN COMPANY STRUCTURE

    Agile software development: Scrum, Kanban

    Short product feature lifecycle

    Continuous delivery

    Self-organizing teams

    Cross-functional teams

  • AGILE DEVELOPMENT, CONTD.

    Scrum

    Excellent for product development

    Short, iterative cycles (sprints), commit to deadlines

    Self-organizing teams

    Kanban

    Developed at Toyota to optimize supply-chain

    Good for ad-hoc tasks, e.g. support, bugfixing

  • AGILE DEVELOPMENT, CONTD.

    Agile development requires agile operations

    Underlying architecture must scale together with the product

    Continuous integration

    Quality assurance

    Deployment

    The solution: de-couple development and operations

    DevOps

  • USE CASE

    Fast and dependable communication with external providers

    On-demand resource scaling

    Flexibility

    Storage, indexing & analysis of huge amounts of data

    Dependability

  • CLOUD CONCEPTS

    Three main service models

    IaaS EC2, Rackspace, Azure, HP, Oracle

    PaaS Google App Engine, Heroku

    SaaS Gmail, Wordpress.com, Salesforce, Office 365

  • CLOUD CONCEPTS, CONTD

    Four deployment models

    Public

    Private

    Community

    Hybrid

  • CLOUD INTERNALS

    SERVERSSTORAGE

    EXTERNAL SERVICES

    (CDN, NETWORKING, SaaS)VMs VMs

  • PROS AND CONS

    Dedicated Cloud

    + simplicity

    + performance

    + predictability

    + tried and tested

    + agility

    + ease of use

    + scalability

    + stability

    - inflexible

    - ineffective

    - nightmare in case of disaster

    - vendor lock-in

    - blackboxed

    - fluctuations

    - helpless in case of disaster

  • CLOUD ARCHITECTURE

    Application servers scale up and down based on load

    Application software written for parallelism

    Communication between nodes via messaging service

    Write for eventual consistency

  • OK, so we have an (endlessly) scalable cloud app

    now.

    But didn't we just create a bottleneck elsewhere?

  • MIX & MATCH

    Crunch numbers in the cloud

    Application servers

    Slow running tasks

    Temporary services

    Test servers

  • MIX & MATCH, CONTD.

    Traditional servers for:

    Incompatible apps (single-threaded, memory, disk intensive, specialized

    hardware) do not work well in cloud environments

    Database servers are best kept on dedicated machines in our experience

  • DATABASES, NOSQL

    def. NoSQLl a mechanism for storage and retrieval of data that is modeled in means other

    than the tabular relations used in relational databases.

  • DATABASES

    Use the best tool for the job depending on the task

    NoSQL Advantages Social networks generate a lot of data

    Complex interconnections, cyclical dependencies

    Aggregations must be performed on both new and old data

    Structure of foreign sources may change on short notice

  • DATABASES, NOSQL CONTD

    Riak, Hadoop, MongoDB, Cassandra, Redis

    In-memory dataset for faster operation

    No predefined structure

    Integrated sharding, load-balancing and failover

    Versatility - can be used for anything from data storage to real-time messaging to search

    indexes

  • DATABASES, LONG TERM STRATEGY

    Data quickly becomes irrelevant

    Archive it, but still be accessible

    Online Data Warehouse solutions

    Amazon Redshift

    Keep Everything

    Terabytes for pennies

  • WE ARE LOOKING FOR

    Senior Web Developer

    Junior Web Developer

    Junior QA

    DevOps Engineer

  • Questions?

    Slide 1Slide 2Slide 3Slide 4Slide 5Slide 6Slide 7Slide 8Slide 9Slide 10Slide 11Slide 12Slide 13Slide 14Slide 15Slide 16Slide 17Slide 18Slide 19Slide 20Slide 21Slide 22Slide 23Slide 24Slide 25