unlocked workshop oscon 2013 - part ii

5 PILLARS OF CLOUDINESS

wayne wallscloud evangelist

alex brandtsr developer

Tuesday, July 23, 13

FIVE PILLARS OF CLOUDINESS

1 2 3 4 5

Parallel Modular Horizontal Agile Secure

2


Pillar #1:Parallel


1MRecords

Answer

Node

4


1MRecords

Answer

Node1-250K

Node251K-500K

Node501K-750K

Node751K-1M

5


6

1MRecords

Answer

Node1-250K

Node251K-500K

Node501K-750K

Node751K-1M


7


DATA

CALCULATIONMAP

CALCULATIONREDUCE

DATA

CALC

D

C

R

D

D

C

R

D

D

C

R

D

SHUFFLE

DATA

7


8


Thread 1

Deadlock

9

Thread 2


File A

Thread 1

Deadlock

9

Thread 2


File A File B

Thread 1

Deadlock

9

Thread 2


File A File B

Thread 1

Deadlock

9

File B

Thread 2


File A File B

File A

Thread 1

Deadlock

9

File B

Thread 2


File A File B

File A

Thread 1

Deadlock

9

X

File B

Thread 2


File A File B

File A

Thread 1

Deadlock

9

X X

File B

Thread 2


Solution? Scheduling

Y

XA

B

A

B

Y

X

1

2

3

4

10


1M records

Answer

Node

1M records

Answer

Node1-250K

Node251K-500K

Node501K-750K

Node750K-1M

11


12


13


pillar #2: modular


ONE PERSON BUILDS A CAR

15


CHASSIS ENGINE BODY INTERIOR

16



17



18



SEATS FLOOR

DASHBOARD

GASAND

BRAKE PEDALS

18



19



20


defect


20


defect defect


20


defect defect defect


20


defect defect defectdefectCHASSIS ENGINE BODY INTERIOR

20


defect defect defectdefect LEMONCHASSIS ENGINE BODY INTERIOR

20


Pillar #3:horizontal


2 GB2 CORES

22

8 GB4 CORES


2 GB2 CORES

VERTICAL

22

8 GB4 CORES


2 GB2 CORES

VERTICAL

HORIZONTAL

22

8 GB4 CORES


2 GB2 CORES

2 GB2 CORES

VERTICAL

HORIZONTAL

22

8 GB4 CORES


2 GB2 CORES

2 GB2 CORES

2 GB2 CORES

VERTICAL

HORIZONTAL

22

8 GB4 CORES


2 GB2 CORES

2 GB2 CORES

2 GB2 CORES

2 GB2 CORES

VERTICAL

HORIZONTAL

22

8 GB4 CORES


2 GB2 CORES

2 GB2 CORES

2 GB2 CORES

2 GB2 CORES

2 GB2 CORES

VERTICAL

HORIZONTAL

22

8 GB4 CORES


2 GB2 CORES

2 GB2 CORES

2 GB2 CORES

2 GB2 CORES

2 GB2 CORES

2 GB2 CORES

VERTICAL

HORIZONTAL

22

8 GB4 CORES


2 GB2 CORES

2 GB2 CORES

2 GB2 CORES

2 GB2 CORES

2 GB2 CORES

2 GB2 CORES

2 GB2 CORES

VERTICAL

HORIZONTAL

22

8 GB4 CORES


2 GB2 CORES

2 GB2 CORES

2 GB2 CORES

2 GB2 CORES

2 GB2 CORES

2 GB2 CORES

2 GB2 CORES

VERTICAL

HORIZONTAL

SHARED NOTHING THROUGH STATELESSNESS

22

8 GB4 CORES


2 GB2 CORES

2 GB2 CORES

2 GB2 CORES

2 GB2 CORES

2 GB2 CORES

2 GB2 CORES

VERTICAL

HORIZONTAL


22

8 GB4 CORES


2 GB2 CORES

2 GB2 CORES

2 GB2 CORES

2 GB2 CORES

2 GB2 CORES

2 GB2 CORES

VERTICAL

HORIZONTAL


2 GB2 CORES

22

8 GB4 CORES


23


24


ONE SERVER NEAR

CAPACITY

24


ONE SERVER NEAR

CAPACITY

ADD ANOTHER SERVER

24


ONE SERVER NEAR

CAPACITY

ADD ANOTHER SERVER

25


1234 MESSAGE QUEUE

26


1234MESSAGE QUEUE

27


1

2

3

4

28

MESSAGE QUEUE


1

2

3

4

28

56789101112 MESSAGE QUEUE


29

5678 9101112MESSAGE QUEUE

1

2

3

4


29

5678 9101112

ADD MORE SERVERSBASED ON

MESSAGE QUEUE

MESSAGE QUEUE

1

2

3

4


MESSAGE QUEUE

30

5

6

7

8

9

10

11

12

13141516


MESSAGE QUEUE

1

2

3

4


31

5678 9101112

13141516


MESSAGE QUEUE

1234

MESSAGE QUEUE


32

5678 9101112

13141516


MESSAGE QUEUE

1234

MESSAGE QUEUE


33

5678 9101112

13141516


MESSAGE QUEUE1234

MESSAGE QUEUE


33

5678 9101112

13141516


MESSAGE QUEUE1234

MESSAGE QUEUE

DECREASE SERVERSBASED ON

MESSAGE QUEUE


MESSAGE QUEUE

34

5678 9101112

13

14

15

16


MESSAGE QUEUE1234

DECREASE SERVERSBASED ON

MESSAGE QUEUE


Pillar #4:agile


36


Features In Progress Blocked Tabled Testing Complete

Feature A

Feature B

Feature C

37

Agile - why Monday


Agile - why


Feature A

Feature B

Feature C

38

Tuesday


Agile - why


Feature A

Feature B

Feature C

Feature E

Feature D

38

Tuesday



Feature AFeature BFeature C

Feature E

Feature G

Agile - why

39

Wednesday



Feature AFeature BFeature C

Feature E

Feature D

Feature F

Feature G

Agile - why

39

Wednesday



Feature A

Feature B

Feature C

Feature E

Feature D

Feature F

Feature G

Agile - why

40

Friday



Feature A

Feature B

Feature C

Feature E

Feature D

Feature F

Feature G

Feature H

Feature I

Agile - why

40

Friday



Feature A

Feature B

Feature C

Feature EFeature D

Feature F

Feature G

Feature H

Feature I

Agile - why

41

Friday - WK 2


Agile - why

57%

14%

29%

Failed Successful Challenged

49%

42%

9%

Waterfall Agile

Source: The CHAOS Manifesto, The Standish Group, 2012

42


43

Continuous Integration/Continuous Delivery


44

Quality


45

Release Cycle


IMAGE: blogs.bu.edu

46


47

Lint Testing

Test Driven Development

Functional Testing

Black Box

Integration Testing

Unit Testing

Behavior Driven Development

White Box

Smoke Testing

Acceptance Testing


ENSURES QUALITY

47

Lint Testing


Functional Testing

Black Box

Integration Testing

Unit Testing


White Box

Smoke Testing

Acceptance Testing


48

Lint Testing


Functional Testing

Black Box

Integration Testing

Unit Testing


White Box

Smoke Testing

Acceptance Testing


ENSURES CONFIDENCE

48

Lint Testing


Functional Testing

Black Box

Integration Testing

Unit Testing


White Box

Smoke Testing

Acceptance Testing


Pillar #5:secure


50


51

PRESENTATION

APPLICATION

STORAGE


51

PRESENTATION

APPLICATION

STORAGE

INTERFACESECURITYACROSS ALL LAYERS


52


53

Host-based Authentication


54

User Authentication


55

Hybrid Cloud


56

Trustworthiness as an Organization


APPLICATION dissection

alex brandtsr developer

wayne wallscloud evangelist


Data Schema

•articles• url

• text

• tags-index

• notations

• location

• note

• votes

• created_at-index

• etag

• parsed_at

•users• username-unique index

• email

• name

• hash (md5)

58

•subscribers• uuid4-user

• uuid4-bookmark

• subscribed_at


LB

FRONT END API

MQ

CONSUMER

LB

KEY/VALUESTORE

DOCUMENTSTORE

INTERNET

Routing

•Topic

•users

•articles

•Fanout

•article.sanitize

•article.references

Consumers

•create user

•password user

•article reference

•email user

•create article

•article sanitize

OBJECTSTORE


LB

FRONT END API

MQ

CONSUMER

LB

KEY/VALUESTORE

DOCUMENTSTORE

INTERNET

Routing

•Topic

•users

•articles

•Fanout

•article.sanitize


Consumers

•create user

•password user


•email user

•create article

•article sanitize

OBJECTSTORE Parallel

Modular

Horizontally scalable

Agile

Secure


KEY/VALUESTORE

OBJECTSTORE

LB

FRONT END API

MQ

CONSUMER

LB

DOCUMENTSTORE

INTERNET

Routing

•Topic

•users

•articles

•Fanout

•article.sanitize


Consumers

•create user

•password user


•email user

•create article

•article sanitize

Parallel

Modular

Horizontally scalable

Agile

Secure


65

https://github.com/raxsavvy/margarine

Thank You!

@rackspace@waynewalls@hhoover@alunduil




unlocked workshop oscon 2013 - part ii

Technology

cores vertical horizontal

file b file

file b thread

records answer node

scheduling y xa b

message queue

server near capacity

brake pedals