June 3-5, 2014 | Berlin, Germany
Robin Fernandes, Dev Team Lead, Atlassian
Building a Production-Ready Connect Add-On
Who’s Looking?
19 500 users 950 hosts
Overview / Stack
× many!
× many!
Who’s Looking?
Multi-tenancy
Scalabilityhttps://bitbucket.org/atlassian/whoslooking-connect
What Connect ProvidesTenant registration
Authentication & tenant context
Webhooks
Integrated look & feel
Local dev & test
…Handled by Connect!
A Production-Ready Add-On
Hosting
Upgrading
Scaling
Monitoring
…
Lots of decisions…
Hosting
Stateless Request Processing
Load Balancer
Data Store
Hosting on a PaaS can encourage
statelessness, which enables
horizontal scalability and
improves resilience.
Requests
app processes
(dynos)
whoslooking.herokuapp.com
Rich ecosystem of external services
3rd Party Services
Basic deployment is just a git push
Low Barrier to Entry
> git push heroku master
Free tiers allow you to experiment at no cost
Testing
TestingUnit testing: business as usual
Browser tests: similar to in-process add-ons
Run add-on and host app locally
Atlassian SDK, WebDriver, Atlassian Page Objects…
Add post-deployment verification tests
Reliable sanity tests, run after every deployment
Must be runnable against a production instance
Testing & Muti-tenancyMock out products’ REST APIs to test multi-tenant interactions
(see Who’s Looking source on Bitbucket for examples)
ac-mock-tenants.herokuapp.com
Connect add-on
Load TestingLoad test on free tier, and extrapolate
Use e.g. New Relic to monitor the add-on under load
Load Testing
Tool: ac-load-test
Sets up Gatling-based load tests for Connect add-ons
Simulates authenticated requests from 1000s of hosts & users
Connect add-on
ac-mock-tenants> ./sbt runac-load-test
(Gatling)
heavy load
bitbucket.org/atlassianlabs/ac-load-test.g8
Mock OD instances
JWT-signed requests
Load Testing
> ./sbt run … ================================================================================ 2014-05-17 17:36:22 20s elapsed ---- A mixture of iframe and XHR requests -------------------------------------- [--------------------------------------------------------------------------] 0% waiting: 0 / running: 200 / done:0 ---- Requests ------------------------------------------------------------------ > Global (OK=899 KO=0 ) > iframe-request (OK=200 KO=0 ) > token-based-xhr-request (OK=699 KO=0 ) ================================================================================ …
bitbucket.org/atlassianlabs/ac-load-test.g8
Deployment
Staged Deployments
whoslooking-dev. herokuapp.com
dev
run PDV tests
deploy
Staged Deployments
whoslooking-stg. herokuapp.com
stg
run PDV tests
promote from dev to stg
(using pipelines)
Green dev deployment?
+ soak time
Staged Deployments
whoslooking. herokuapp.com
run PDV tests
All good on stg?
prod
promote from stg to prod
(using pipelines)
Further Precautions
the goggles…
Things can still go wrong!Log management
Set up alerts on log patterns, search logs
Add-ons: Logentries, FlyData, Papertrail…
Feature flags
Turn new features on & off via configuration
Metrics / analytics
Get insight into application behaviour to detect issues
Libraries for in-memory metrics: Simon, Metrics, Servo…
Add-ons for externalised analytics: New Relic, StatsMix, Hosted Graphite…
Healthcheck & Monitoring
http://whoslooking.herokuapp.com/healthcheck
Implement a Health Check
REST resource that runs sanity-checks & reports results
Monitor it (Pingdom, New Relic…)
Example:
(Checks Postgres & Redis connections, reports config
settings, resource usage, weekly/daily active users/hosts)
Healthcheck & Monitoring
To build a production-ready add-on…Design for multi-tenancy and horizontal scalability
Ensure deploying is easy
Make the most of 3rd party services
Load test to help estimate running cost
Implement a health check resource & monitor it
Run post-deployment verification tests
Start simple!
Who’s Looking Stats
2-4 dynos (average ~3) 5 external services (3 free, 2 paid) 19800 active users last week 960 active instances last week 3500 average rpm (~7500 peak) <1s average page load time !
$89.50 / month (~$0.005 / user / month)
Slides:
go.atlassian.com/prod-connect
[email protected] / @rewbs
