Download - An Introduction To Yelp's SOA Infrastructure
![Page 1: An Introduction To Yelp's SOA Infrastructure](https://reader031.vdocuments.site/reader031/viewer/2022022123/58a0ba4d1a28ab75368b7059/html5/thumbnails/1.jpg)
Yelp is (always) hiring!Talk to a Speaker or Recruiter if
interested!
Obligatory Welcome Slide!
SF Devops Meetup Organizers Thank You!
![Page 2: An Introduction To Yelp's SOA Infrastructure](https://reader031.vdocuments.site/reader031/viewer/2022022123/58a0ba4d1a28ab75368b7059/html5/thumbnails/2.jpg)
Yelp’s Mission:Connecting people with great
local businesses.
![Page 3: An Introduction To Yelp's SOA Infrastructure](https://reader031.vdocuments.site/reader031/viewer/2022022123/58a0ba4d1a28ab75368b7059/html5/thumbnails/3.jpg)
A Behind the Scenes Tour of Yelp's SOA Architecture
![Page 4: An Introduction To Yelp's SOA Infrastructure](https://reader031.vdocuments.site/reader031/viewer/2022022123/58a0ba4d1a28ab75368b7059/html5/thumbnails/4.jpg)
What Are We Going To Talk About?
● YelpSOA Configs - linchpin of our stuff
● Service Discovery (Smartstack)+ Docker
● Mesos-based Deployment (PaaSTA)
● Dynamic Monitoring (Sensu)
![Page 5: An Introduction To Yelp's SOA Infrastructure](https://reader031.vdocuments.site/reader031/viewer/2022022123/58a0ba4d1a28ab75368b7059/html5/thumbnails/5.jpg)
First - Terminology
● Services ~= “microservices?”○ I don’t care how small your services are.
● SOA == Service Oriented Architecture○ Opposite of a monolithic app architecture. Usually
accompanied with teams (reverse Conway’s law?)
![Page 6: An Introduction To Yelp's SOA Infrastructure](https://reader031.vdocuments.site/reader031/viewer/2022022123/58a0ba4d1a28ab75368b7059/html5/thumbnails/6.jpg)
Yelp’s Architecture● Still has a significant amount of code in the
monolith○ it “pays the bills”○ Treated mostly like any other service
● Still though, lots of teams, lots of services○ Years of pushing for “the SOA dream”?
● In AWS and Datacenter in 3 “super” regions (yes, hybrid)
● We think we have some good“behind the scenes ideas”
![Page 7: An Introduction To Yelp's SOA Infrastructure](https://reader031.vdocuments.site/reader031/viewer/2022022123/58a0ba4d1a28ab75368b7059/html5/thumbnails/7.jpg)
What Would I Do If I Did It Again?
1. Have an idea2. Make a git repo
![Page 8: An Introduction To Yelp's SOA Infrastructure](https://reader031.vdocuments.site/reader031/viewer/2022022123/58a0ba4d1a28ab75368b7059/html5/thumbnails/8.jpg)
YelpSOA Configs: The Big Idea
● Let’s just use config files to be the source of truth about what our SOA infrastructure looks like instead of being in databases, zookeeper, or just from “whoever ran the deploy command last”
● Be declarative instead of imperative● “Just a git repo”
![Page 9: An Introduction To Yelp's SOA Infrastructure](https://reader031.vdocuments.site/reader031/viewer/2022022123/58a0ba4d1a28ab75368b7059/html5/thumbnails/9.jpg)
YelpSOA Configs: Things We WON'T do
● Let’s try not to be implementation specific● Let’s try not to do service discovery● No secrets in the config● All configs in one repo, no dev/prod split
![Page 10: An Introduction To Yelp's SOA Infrastructure](https://reader031.vdocuments.site/reader031/viewer/2022022123/58a0ba4d1a28ab75368b7059/html5/thumbnails/10.jpg)
Architecture: Good Ol Rsync + Cron
Plain-ole git repo
Normal Commits
Rsync Pulls
Machine generated files added
Read-Only
Slaves
Most Servers At Yelp
Most Servers At Yelp
Most Servers At Yelp
Read-Only
Slaves
Read-Only
SlavesMost Servers At Yelp
![Page 11: An Introduction To Yelp's SOA Infrastructure](https://reader031.vdocuments.site/reader031/viewer/2022022123/58a0ba4d1a28ab75368b7059/html5/thumbnails/11.jpg)
Architecture: Mostly Doesn’t Matter● Git repo - Works fine
● Gitolite for ACLs - Sure
● Pre-receive hooks for validation - Cool
● Cron Jobs to Deploy the Git repo - What else?
● Rsync Fanout to distribute the files - Heh, fine
● Yelp Puppet-Cron makes this easy and high-frequency: https://github.com/Yelp/puppet-cron
![Page 12: An Introduction To Yelp's SOA Infrastructure](https://reader031.vdocuments.site/reader031/viewer/2022022123/58a0ba4d1a28ab75368b7059/html5/thumbnails/12.jpg)
Demo: Seeing == Believing
What does it look like when you have your services metadata in an easy-to-reach place?
![Page 13: An Introduction To Yelp's SOA Infrastructure](https://reader031.vdocuments.site/reader031/viewer/2022022123/58a0ba4d1a28ab75368b7059/html5/thumbnails/13.jpg)
Example Ideas of Things to Put In There
● team data (ownership)● general service metadata (what is it?)● monitoring data (who should get alerts?)● deployments data (where should it run?)● routing data (where should it be
advertised?)
![Page 14: An Introduction To Yelp's SOA Infrastructure](https://reader031.vdocuments.site/reader031/viewer/2022022123/58a0ba4d1a28ab75368b7059/html5/thumbnails/14.jpg)
Conclusion / Big Ideas● Every team has people with enough privileges to edit
them● They are configs about the services, not for the
services (they don’t exist in the code repo)● Super fast deployment. No code, only config.● Describe the concepts, not the implementation● Build the metadata and the tools will follow!
![Page 15: An Introduction To Yelp's SOA Infrastructure](https://reader031.vdocuments.site/reader031/viewer/2022022123/58a0ba4d1a28ab75368b7059/html5/thumbnails/15.jpg)
A. To describe how cool Yelp's SOA configs areB. To persuade viewers buy Yelp’s SOA configs enterprise product offeringC. To Inspire viewers to build their own SOA configs - based on some of these ideas presentedD. To bore viewers about how Yelp rsyncs YAML files around with cron jobs.
Reading Comprehension Question:(what is the purpose of this talk?)
![Page 16: An Introduction To Yelp's SOA Infrastructure](https://reader031.vdocuments.site/reader031/viewer/2022022123/58a0ba4d1a28ab75368b7059/html5/thumbnails/16.jpg)
A. To describe how cool Yelp's SOA configs areB. To persuade viewers buy Yelp’s SOA configs enterprise product offeringC. To Inspire viewers to build their own SOA configs - based on some of these ideas presentedD. To bore viewers about how Yelp rsyncs YAML files around with cron jobs.
Reading Comprehension Question:(what is the purpose of this talk?)
![Page 17: An Introduction To Yelp's SOA Infrastructure](https://reader031.vdocuments.site/reader031/viewer/2022022123/58a0ba4d1a28ab75368b7059/html5/thumbnails/17.jpg)
● Helps puppet-cron type for high-frequency crons:https://github.com/Yelp/puppet-cron
● Gitolite for ACLS and hooks: http://gitolite.com/● rsync + cron
Tools used:
![Page 18: An Introduction To Yelp's SOA Infrastructure](https://reader031.vdocuments.site/reader031/viewer/2022022123/58a0ba4d1a28ab75368b7059/html5/thumbnails/18.jpg)
Questions