vertically scaled design patters
TRANSCRIPT
![Page 1: Vertically Scaled Design Patters](https://reader033.vdocuments.site/reader033/viewer/2022042611/587f3a601a28ab43318b47bd/html5/thumbnails/1.jpg)
Vertically Scaled Design Patterns
Jeff Malnick & Paul AmbrosiniSystems Engineering @ SRC:CLR
srcclr.com | @srcclr
![Page 2: Vertically Scaled Design Patters](https://reader033.vdocuments.site/reader033/viewer/2022042611/587f3a601a28ab43318b47bd/html5/thumbnails/2.jpg)
How this is going down:
srcclr.com | @srcclr
● History & Evolution
● Principles
● Notes & Examples
● Demo
![Page 3: Vertically Scaled Design Patters](https://reader033.vdocuments.site/reader033/viewer/2022042611/587f3a601a28ab43318b47bd/html5/thumbnails/3.jpg)
History & Evolution
srcclr.com | @srcclr
![Page 4: Vertically Scaled Design Patters](https://reader033.vdocuments.site/reader033/viewer/2022042611/587f3a601a28ab43318b47bd/html5/thumbnails/4.jpg)
In the beginning there was
srcclr.com | @srcclr
● AWS● ELB● Ansible ● SCP● Tomcat
● We all start somewhere
![Page 5: Vertically Scaled Design Patters](https://reader033.vdocuments.site/reader033/viewer/2022042611/587f3a601a28ab43318b47bd/html5/thumbnails/5.jpg)
Then there was
srcclr.com | @srcclr
![Page 6: Vertically Scaled Design Patters](https://reader033.vdocuments.site/reader033/viewer/2022042611/587f3a601a28ab43318b47bd/html5/thumbnails/6.jpg)
srcclr.com | @srcclr
Paul heroically rolled it out in 2 weeks
![Page 7: Vertically Scaled Design Patters](https://reader033.vdocuments.site/reader033/viewer/2022042611/587f3a601a28ab43318b47bd/html5/thumbnails/7.jpg)
Massive Gains
srcclr.com | @srcclr
● Huge improvement
● Split up Frontend and Backend Code
● Teams deploying separately
● Productivity went up dramatically
![Page 8: Vertically Scaled Design Patters](https://reader033.vdocuments.site/reader033/viewer/2022042611/587f3a601a28ab43318b47bd/html5/thumbnails/8.jpg)
Wasn’t perfect
srcclr.com | @srcclr
● Not fully dynamic
● CI hooks not perfect
● Cookbook merging*
● Service health management lacking
![Page 9: Vertically Scaled Design Patters](https://reader033.vdocuments.site/reader033/viewer/2022042611/587f3a601a28ab43318b47bd/html5/thumbnails/9.jpg)
Then we migrated...
srcclr.com | @srcclr
● *hired Jeff○ ‘ops role’○ or ‘devops’
![Page 10: Vertically Scaled Design Patters](https://reader033.vdocuments.site/reader033/viewer/2022042611/587f3a601a28ab43318b47bd/html5/thumbnails/10.jpg)
We got a shiny new CM system.
srcclr.com | @srcclr
![Page 11: Vertically Scaled Design Patters](https://reader033.vdocuments.site/reader033/viewer/2022042611/587f3a601a28ab43318b47bd/html5/thumbnails/11.jpg)
"Type" : "AWS::EC2::Instance",
"Properties" : {
"KeyName" : "secure",
"SecurityGroupIds" : ["sg-1234abcd","sg-5678efg"],
"SubnetId" : "subnet-1234abcd",
"UserData" : {
"Fn::Base64" : {
"Fn::Join" : ["", [
"#!/bin/sh -v\n",
"ROLE='qa_nginx_static'\n",
"MASTER='https://puppet.srcclr.com'\n",
"D_ROLE=$(echo $ROLE | tr '_' '-')\n",
"D_IP=$(hostname -I | tr '.' '-' | cut -d' ' -f1)\n",
"HOSTNAME=\"$(echo $D_ROLE)-$(echo $D_IP)\"\n",
"echo \"127.0.0.1 $HOSTNAME\" >> /etc/hosts",
"apt-get update\n",
"echo $HOSTNAME > /etc/hostname\n",
"/bin/hostname $HOSTNAME\n",
"echo $ROLE > /etc/role\n",
"sudo curl -k $MASTER:8140/packages/current/install.bash | sudo bash\n"
]]}
Cloud Formation for Bare Metal
![Page 12: Vertically Scaled Design Patters](https://reader033.vdocuments.site/reader033/viewer/2022042611/587f3a601a28ab43318b47bd/html5/thumbnails/12.jpg)
Continuous Integration
srcclr.com | @srcclr
![Page 13: Vertically Scaled Design Patters](https://reader033.vdocuments.site/reader033/viewer/2022042611/587f3a601a28ab43318b47bd/html5/thumbnails/13.jpg)
srcclr.com | @srcclr
# Executed in $WORKSPACE on jenkins
VERSION=$(some_mvn_command)
curl \
-X POST \
-d@- \
puppet.myorg.com:1015/deploy <\<EOF
{
"service": "analytics",
"version": "$VERSION",
"environment": "qa",
"role": "qa_service"
}
EOF
How we do CI
![Page 14: Vertically Scaled Design Patters](https://reader033.vdocuments.site/reader033/viewer/2022042611/587f3a601a28ab43318b47bd/html5/thumbnails/14.jpg)
New CM was better but...
srcclr.com | @srcclr
![Page 15: Vertically Scaled Design Patters](https://reader033.vdocuments.site/reader033/viewer/2022042611/587f3a601a28ab43318b47bd/html5/thumbnails/15.jpg)
Shameless use of hooks
srcclr.com | @srcclr
include http
http::listener {‘kick_haproxy’:
port => '1234',
routes => {
'kick_haproxy' => {
'method' => 'get',
'command' => "su - peadmin -c 'mco puppet runonce -F
role == haproxy_internal'"
},
![Page 16: Vertically Scaled Design Patters](https://reader033.vdocuments.site/reader033/viewer/2022042611/587f3a601a28ab43318b47bd/html5/thumbnails/16.jpg)
“Service Discovery”
srcclr.com | @srcclr
http { 'kick_haproxy':
ensure => get,
port => '1234',
fqdn => 'puppet.myorg.com'
}
![Page 17: Vertically Scaled Design Patters](https://reader033.vdocuments.site/reader033/viewer/2022042611/587f3a601a28ab43318b47bd/html5/thumbnails/17.jpg)
srcclr.com | @srcclr
sc_services::service { 'librarian':
deploy_stage => 'production',
version => $version,
enable_newrelic_apm => true,
loadbalancer_role => 'production_internal',
xmx_setting => '1024',
xms_setting => '1024',
proc_opts => {
'1' => {
'env_profile' => 'prod',
'port' => '11100',
'mgmt_port' => '11110',
'override' => $properties_file,
},
'2' => {
'env_profile' => 'prod',
'port' => '11101',
'mgmt_port' => '11111',
'override' => $properties_file,
}
}
Still static...
![Page 18: Vertically Scaled Design Patters](https://reader033.vdocuments.site/reader033/viewer/2022042611/587f3a601a28ab43318b47bd/html5/thumbnails/18.jpg)
srcclr.com | @srcclr
sc_services::service { 'librarian':
deploy_stage => 'app',
version => $version,
enable_newrelic_apm => true,
loadbalancer_role => 'production_internal',
xmx_setting => '1024',
xms_setting => '1024',
proc_opts => {
'1' => {
'env_profile' => 'app',
'port' => '11100',
'mgmt_port' => '11110',
'override' => $properties_file,
},
'2' => {
'env_profile' => 'app',
'port' => '11101',
'mgmt_port' => '11111',
'override' => $properties_file,
}
}
Still static...
{Statically
assigning ports!
![Page 19: Vertically Scaled Design Patters](https://reader033.vdocuments.site/reader033/viewer/2022042611/587f3a601a28ab43318b47bd/html5/thumbnails/19.jpg)
srcclr.com | @srcclr
sc_services::service { 'librarian':
deploy_stage => 'app',
version => $version,
enable_newrelic_apm => true,
loadbalancer_role => 'production_internal',
xmx_setting => '1024',
xms_setting => '1024',
proc_opts => {
'1' => {
'env_profile' => 'app',
'port' => '11100',
'mgmt_port' => '11110',
'override' => $properties_file,
},
'2' => {
'env_profile' => 'app',
'port' => '11101',
'mgmt_port' => '11111',
'override' => $properties_file,
}
}
Still static...
What S3 Bucketis this Jar from?
![Page 20: Vertically Scaled Design Patters](https://reader033.vdocuments.site/reader033/viewer/2022042611/587f3a601a28ab43318b47bd/html5/thumbnails/20.jpg)
srcclr.com | @srcclr
sc_services::service { 'librarian':
deploy_stage => 'app',
version => $version,
enable_newrelic_apm => true,
loadbalancer_role => 'production_internal',
xmx_setting => '1024',
xms_setting => '1024',
proc_opts => {
'1' => {
'env_profile' => 'app',
'port' => '11100',
'mgmt_port' => '11110',
'override' => $properties_file,
},
'2' => {
'env_profile' => 'app',
'port' => '11101',
'mgmt_port' => '11111',
'override' => $properties_file,
}
}
Still static...
“Service Discovery”
for HaProxy
![Page 21: Vertically Scaled Design Patters](https://reader033.vdocuments.site/reader033/viewer/2022042611/587f3a601a28ab43318b47bd/html5/thumbnails/21.jpg)
srcclr.com | @srcclr
sc_services::service { 'librarian':
deploy_stage => 'app',
version => $version,
enable_newrelic_apm => true,
loadbalancer_role => 'production_internal',
xmx_setting => '1024',
xms_setting => '1024',
proc_opts => {
'1' => {
'env_profile' => 'app',
'port' => '11100',
'mgmt_port' => '11110',
'override' => $properties_file,
},
'2' => {
'env_profile' => 'app',
'port' => '11101',
'mgmt_port' => '11111',
'override' => $properties_file,
}
}
Still static...Service statically
scheduled to execute on a
node classified by Puppet
![Page 22: Vertically Scaled Design Patters](https://reader033.vdocuments.site/reader033/viewer/2022042611/587f3a601a28ab43318b47bd/html5/thumbnails/22.jpg)
“Scaling”
srcclr.com | @srcclr
$ports = {
'librarian' => {
'svc' => '20060-20070',
'mgmt' => '20071-20080',
},
'notifications' => {
'svc' => '20000-20010',
'mgmt' => '20011-20020',
},
'cloud_agent' => {
'svc' => '31040-31050',
'mgmt' => '31051-31060',
},
'search' => {
'svc' => '41140-41150',
'mgmt' => '41151-41160',
},
![Page 23: Vertically Scaled Design Patters](https://reader033.vdocuments.site/reader033/viewer/2022042611/587f3a601a28ab43318b47bd/html5/thumbnails/23.jpg)
“Scaling”
srcclr.com | @srcclr
$ports = {
'librarian' => {
'svc' => '20060-20070',
'mgmt' => '20071-20080',
},
'notifications' => {
'svc' => '20000-20010',
'mgmt' => '20011-20020',
},
'cloud_agent' => {
'svc' => '31040-31050',
'mgmt' => '31051-31060',
},
'search' => {
'svc' => '41140-41150',
'mgmt' => '41151-41160',
}
...
{ Scaling to maximum of 10 instances per box.
![Page 24: Vertically Scaled Design Patters](https://reader033.vdocuments.site/reader033/viewer/2022042611/587f3a601a28ab43318b47bd/html5/thumbnails/24.jpg)
Recap
srcclr.com | @srcclr
● CM statically schedules services to run on node
● Can scale to maximum number of known ports
● This does “scale” vertically & horizontally...
![Page 25: Vertically Scaled Design Patters](https://reader033.vdocuments.site/reader033/viewer/2022042611/587f3a601a28ab43318b47bd/html5/thumbnails/25.jpg)
Automation Principles
![Page 26: Vertically Scaled Design Patters](https://reader033.vdocuments.site/reader033/viewer/2022042611/587f3a601a28ab43318b47bd/html5/thumbnails/26.jpg)
A Standard Factory...
srcclr.com | @srcclr
● Bare Metal Provisioning
● Configuration Management
● Remote Execution Framework
![Page 27: Vertically Scaled Design Patters](https://reader033.vdocuments.site/reader033/viewer/2022042611/587f3a601a28ab43318b47bd/html5/thumbnails/27.jpg)
Software Principles
![Page 28: Vertically Scaled Design Patters](https://reader033.vdocuments.site/reader033/viewer/2022042611/587f3a601a28ab43318b47bd/html5/thumbnails/28.jpg)
The Monolith…
srcclr.com | @srcclr
![Page 29: Vertically Scaled Design Patters](https://reader033.vdocuments.site/reader033/viewer/2022042611/587f3a601a28ab43318b47bd/html5/thumbnails/29.jpg)
As a framework it is:
srcclr.com | @srcclr
● Hard to diagnose what is broken
● One part breaks, the entire thing breaks
● Unwieldy code bases
● Works for some, wasn’t for us
![Page 30: Vertically Scaled Design Patters](https://reader033.vdocuments.site/reader033/viewer/2022042611/587f3a601a28ab43318b47bd/html5/thumbnails/30.jpg)
The Monolith Factory
srcclr.com | @srcclr
● Statically schedules services
● Doesn’t require service discovery
![Page 31: Vertically Scaled Design Patters](https://reader033.vdocuments.site/reader033/viewer/2022042611/587f3a601a28ab43318b47bd/html5/thumbnails/31.jpg)
The Micro-Service
srcclr.com | @srcclr
![Page 32: Vertically Scaled Design Patters](https://reader033.vdocuments.site/reader033/viewer/2022042611/587f3a601a28ab43318b47bd/html5/thumbnails/32.jpg)
As a framework it is:
srcclr.com | @srcclr
● Atomic services
● Easy(er) to figure out what is broken
● Scale or add individual pieces
![Page 33: Vertically Scaled Design Patters](https://reader033.vdocuments.site/reader033/viewer/2022042611/587f3a601a28ab43318b47bd/html5/thumbnails/33.jpg)
The Micro-Services Factory
srcclr.com | @srcclr
● Dynamic task scheduling
● Service discovery required
![Page 34: Vertically Scaled Design Patters](https://reader033.vdocuments.site/reader033/viewer/2022042611/587f3a601a28ab43318b47bd/html5/thumbnails/34.jpg)
The Scaling Problem...
srcclr.com | @srcclr
● Service ‘x’ requires known port assignments
● Service ‘x’ configuration changes across env’s
● Service ‘x’ should be able to run on a box with other instances of Service ‘x’ as well as Service ‘y’ and Service ‘z’
![Page 35: Vertically Scaled Design Patters](https://reader033.vdocuments.site/reader033/viewer/2022042611/587f3a601a28ab43318b47bd/html5/thumbnails/35.jpg)
The Vertical Component
srcclr.com | @srcclr
● Start locally, on a single box
● Find a solution that works vertically, master it
● Once the vertical solution is found, move horizontally
![Page 36: Vertically Scaled Design Patters](https://reader033.vdocuments.site/reader033/viewer/2022042611/587f3a601a28ab43318b47bd/html5/thumbnails/36.jpg)
Dynamic Task Scheduling
srcclr.com | @srcclr
● Purpose built remote execution framework
![Page 37: Vertically Scaled Design Patters](https://reader033.vdocuments.site/reader033/viewer/2022042611/587f3a601a28ab43318b47bd/html5/thumbnails/37.jpg)
What we really need...
srcclr.com | @srcclr
● Purpose built remote execution framework
● … micro services are like streakers…
![Page 38: Vertically Scaled Design Patters](https://reader033.vdocuments.site/reader033/viewer/2022042611/587f3a601a28ab43318b47bd/html5/thumbnails/38.jpg)
They don’t care where they’re running, they just want to be exposed.
srcclr.com | @srcclr
● Dynamic tasks scheduling
● … micro services are like streaker`
![Page 39: Vertically Scaled Design Patters](https://reader033.vdocuments.site/reader033/viewer/2022042611/587f3a601a28ab43318b47bd/html5/thumbnails/39.jpg)
What we really need...
srcclr.com | @srcclr
● Purpose built systems
● … CM does CM; SD does SD; divide and conquer
![Page 40: Vertically Scaled Design Patters](https://reader033.vdocuments.site/reader033/viewer/2022042611/587f3a601a28ab43318b47bd/html5/thumbnails/40.jpg)
Configuration Management
srcclr.com | @srcclr
● Is still important!
● Is still necessary!
![Page 41: Vertically Scaled Design Patters](https://reader033.vdocuments.site/reader033/viewer/2022042611/587f3a601a28ab43318b47bd/html5/thumbnails/41.jpg)
srcclr.com | @srcclr
Our Remote Execution Framework
![Page 42: Vertically Scaled Design Patters](https://reader033.vdocuments.site/reader033/viewer/2022042611/587f3a601a28ab43318b47bd/html5/thumbnails/42.jpg)
Our Initial Deployment
srcclr.com | @srcclr
● Apache Mesos
● Marathon
![Page 43: Vertically Scaled Design Patters](https://reader033.vdocuments.site/reader033/viewer/2022042611/587f3a601a28ab43318b47bd/html5/thumbnails/43.jpg)
"Type" : "AWS::EC2::Instance",
"Properties" : {
"KeyName" : "secure",
"SecurityGroupIds" : ["sg-1234abcd","sg-5678efg"],
"SubnetId" : "subnet-1234abcd",
"UserData" : {
"Fn::Base64" : {
"Fn::Join" : ["", [
"#!/bin/sh -v\n",
"ROLE='qa_nginx_static'\n",
"MASTER='https://puppet.srcclr.com'\n",
"D_ROLE=$(echo $ROLE | tr '_' '-')\n",
"D_IP=$(hostname -I | tr '.' '-' | cut -d' ' -f1)\n",
"HOSTNAME=\"$(echo $D_ROLE)-$(echo $D_IP)\"\n",
"echo \"127.0.0.1 $HOSTNAME\" >> /etc/hosts",
"apt-get update\n",
"echo $HOSTNAME > /etc/hostname\n",
"/bin/hostname $HOSTNAME\n",
"echo $ROLE > /etc/role\n",
"sudo curl -k $MASTER:8140/packages/current/install.bash | sudo bash\n"
]]}
Design Pattern : Bare Metal
![Page 44: Vertically Scaled Design Patters](https://reader033.vdocuments.site/reader033/viewer/2022042611/587f3a601a28ab43318b47bd/html5/thumbnails/44.jpg)
class roles::mesos_master { include ::profiles::mesos::basic include ::profiles::mesos::master include ::profiles::docker::marathon
include ::profiles::haproxy::marathon_template }
Design Pattern : Config Mgmt
![Page 45: Vertically Scaled Design Patters](https://reader033.vdocuments.site/reader033/viewer/2022042611/587f3a601a28ab43318b47bd/html5/thumbnails/45.jpg)
Design Pattern : Application Layer
Mesos for Layer 7:- Deploy application layer via Docker - Ensure service ‘x’ is running ‘n’ number of
times
![Page 46: Vertically Scaled Design Patters](https://reader033.vdocuments.site/reader033/viewer/2022042611/587f3a601a28ab43318b47bd/html5/thumbnails/46.jpg)
{ "container": { "type": "DOCKER", "docker": { "forcePullImage": true, "image": "malnick/microbrew",
Design Pattern : Service Discovery Marathon for Service Discovery/Deployment:- API to Apache Mesos- Queried for IP/Port assignments of layer 7
resources to configure haproxy
![Page 47: Vertically Scaled Design Patters](https://reader033.vdocuments.site/reader033/viewer/2022042611/587f3a601a28ab43318b47bd/html5/thumbnails/47.jpg)
LIVE DEMO
Are you sure you want to do this?
Pray to the demo gods
![Page 48: Vertically Scaled Design Patters](https://reader033.vdocuments.site/reader033/viewer/2022042611/587f3a601a28ab43318b47bd/html5/thumbnails/48.jpg)
Jeff Malnick Paul Ambrosini malnick at gmail dot com
github.com/malnick
@malnick
technoblogic.iosrcclr.com | @srcclr
c at c4 dot vcpaul at srcclr dot com
github.com/cl4r1ty
@cl4r1ty
c4.vc