Cloud-Based Architecture and Big Data

Philip Balinov,

DevOps Engineer

SOME DEFINITIONS

►def. CLOUD COMPUTING

l - distributed computing over a network

l

- the ability to run a program or application on many connected computers at

the same time.

►def. BIG DATA

l

data sets so large and complex that it becomes difficult to process using

traditional data processing applications

Retail

Finance

E-Commerce

Telecommunication

B2B

Publishing/Media

Government & NGO

Automotive

Travel

KOMFO'S CLIENTS

KOMFO PLATFORM WORKFLOW OVERVIEW

EXTERNAL PROVIDERS

MODERN COMPANY STRUCTURE

►Agile software development: Scrum, Kanban

► Short product feature lifecycle

► Continuous delivery

► Self-organizing teams

► Cross-functional teams

AGILE DEVELOPMENT, CONTD.

►Scrum

► Excellent for product development

► Short, iterative cycles (sprints), commit to deadlines

► Self-organizing teams

►Kanban

► Developed at Toyota to optimize supply-chain

► Good for ad-hoc tasks, e.g. support, bugfixing

AGILE DEVELOPMENT, CONTD.

►Agile development requires agile operations

► Underlying architecture must scale together with the product

► Continuous integration

► Quality assurance

► Deployment

►The solution: de-couple development and operations

► DevOps

USE CASE

►Fast and dependable communication with external providers

►On-demand resource scaling

►Flexibility

►Storage, indexing & analysis of huge amounts of data

►Dependability

CLOUD CONCEPTS

►Three main service models

► IaaS – EC2, Rackspace, Azure, HP, Oracle

► PaaS – Google App Engine, Heroku

► SaaS – Gmail, Wordpress.com, Salesforce, Office 365

CLOUD CONCEPTS, CONTD

►Four deployment models

► Public

► Private

► Community

► Hybrid

CLOUD INTERNALS

SERVERS

STORAGE

EXTERNAL SERVICES

(CDN, NETWORKING, SaaS)

VMs VMs

PROS AND CONS

Dedicated Cloud

+ simplicity

+ performance

+ predictability

+ tried and tested

+ agility

+ ease of use

+ scalability

+ stability

- inflexible

- ineffective

- nightmare in case of disaster

- vendor lock-in

- blackboxed

- fluctuations

- helpless in case of disaster

CLOUD ARCHITECTURE

►Application servers scale up and down based on load

►Application software written for parallelism

►Communication between nodes via messaging service

►Write for eventual consistency

OK, so we have an (endlessly) scalable cloud app

now.

But didn't we just create a bottleneck elsewhere?

MIX & MATCH

►Crunch numbers in the cloud

► Application servers

► Slow running tasks

► Temporary services

► Test servers

MIX & MATCH, CONTD.

►Traditional servers for:

► Incompatible apps (single-threaded, memory, disk intensive, specialized

hardware) do not work well in cloud environments

► Database servers are best kept on dedicated machines in our experience

DATABASES, NOSQL

►def. NoSQL

l

a mechanism for storage and retrieval of data that is modeled in means other

than the tabular relations used in relational databases.

DATABASES

• Use the best tool for the job depending on the task

• NoSQL Advantages

► Social networks generate a lot of data

► Complex interconnections, cyclical dependencies

► Aggregations must be performed on both new and old data

► Structure of foreign sources may change on short notice

DATABASES, NOSQL CONTD

►Riak, Hadoop, MongoDB, Cassandra, Redis

►In-memory dataset for faster operation

►No predefined structure

►Integrated sharding, load-balancing and failover

►Versatility - can be used for anything from data storage to real-time messaging to search

indexes

DATABASES, LONG TERM STRATEGY

►Data quickly becomes irrelevant

►Archive it, but still be accessible

►Online Data Warehouse solutions

► Amazon Redshift

► Keep Everything

► Terabytes for pennies

WE ARE LOOKING FOR…

Senior Web Developer

Junior Web Developer

Junior QA

DevOps Engineer

Questions?