atlascamp 2014: building a production ready connect add-on

June 3-5, 2014 | Berlin, Germany

Robin Fernandes, Dev Team Lead, Atlassian

Building a Production-Ready Connect Add-On

Who’s Looking?

19 500 users 950 hosts

Overview / Stack

× many!

× many!

Who’s Looking?

Multi-tenancy

Scalabilityhttps://bitbucket.org/atlassian/whoslooking-connect

What Connect ProvidesTenant registration

Authentication & tenant context

Webhooks

Integrated look & feel

Local dev & test

…Handled by Connect!

A Production-Ready Add-On

Hosting

Upgrading

Scaling

Monitoring

…

Lots of decisions…

Hosting

Stateless Request Processing

Load Balancer

Data Store

Hosting on a PaaS can encourage

statelessness, which enables

horizontal scalability and

improves resilience.

Requests

app processes

(dynos)

whoslooking.herokuapp.com

Rich ecosystem of external services

3rd Party Services

Basic deployment is just a git push

Low Barrier to Entry

> git push heroku master

Free tiers allow you to experiment at no cost

Testing

TestingUnit testing: business as usual

Browser tests: similar to in-process add-ons

Run add-on and host app locally

Atlassian SDK, WebDriver, Atlassian Page Objects…

Add post-deployment verification tests

Reliable sanity tests, run after every deployment

Must be runnable against a production instance

Testing & Muti-tenancyMock out products’ REST APIs to test multi-tenant interactions

(see Who’s Looking source on Bitbucket for examples)

ac-mock-tenants.herokuapp.com

Connect add-on

Load TestingLoad test on free tier, and extrapolate

Use e.g. New Relic to monitor the add-on under load

Load Testing

Tool: ac-load-test

Sets up Gatling-based load tests for Connect add-ons

Simulates authenticated requests from 1000s of hosts & users

Connect add-on

ac-mock-tenants> ./sbt runac-load-test

(Gatling)

heavy load

bitbucket.org/atlassianlabs/ac-load-test.g8

Mock OD instances

JWT-signed requests

Load Testing

> ./sbt run … ================================================================================ 2014-05-17 17:36:22 20s elapsed ---- A mixture of iframe and XHR requests -------------------------------------- [--------------------------------------------------------------------------] 0% waiting: 0 / running: 200 / done:0 ---- Requests ------------------------------------------------------------------ > Global (OK=899 KO=0 ) > iframe-request (OK=200 KO=0 ) > token-based-xhr-request (OK=699 KO=0 ) ================================================================================ …

bitbucket.org/atlassianlabs/ac-load-test.g8

Deployment

Staged Deployments

whoslooking-dev. herokuapp.com

dev

run PDV tests

deploy

Staged Deployments

whoslooking-stg. herokuapp.com

stg

run PDV tests

promote from dev to stg

(using pipelines)

Green dev deployment?

+ soak time

Staged Deployments

whoslooking. herokuapp.com

run PDV tests

All good on stg?

prod

promote from stg to prod

(using pipelines)

Further Precautions

the goggles…

Things can still go wrong!Log management

Set up alerts on log patterns, search logs

Add-ons: Logentries, FlyData, Papertrail…

Feature flags

Turn new features on & off via configuration

Metrics / analytics

Get insight into application behaviour to detect issues

Libraries for in-memory metrics: Simon, Metrics, Servo…

Add-ons for externalised analytics: New Relic, StatsMix, Hosted Graphite…

Healthcheck & Monitoring

http://whoslooking.herokuapp.com/healthcheck

Implement a Health Check

REST resource that runs sanity-checks & reports results

Monitor it (Pingdom, New Relic…)

Example:

(Checks Postgres & Redis connections, reports config

settings, resource usage, weekly/daily active users/hosts)

Healthcheck & Monitoring

To build a production-ready add-on…Design for multi-tenancy and horizontal scalability

Ensure deploying is easy

Make the most of 3rd party services

Load test to help estimate running cost

Implement a health check resource & monitor it

Run post-deployment verification tests

Start simple!

Who’s Looking Stats

2-4 dynos (average ~3) 5 external services (3 free, 2 paid) 19800 active users last week 960 active instances last week 3500 average rpm (~7500 peak) <1s average page load time !

$89.50 / month (~$0.005 / user / month)

Slides:

go.atlassian.com/prod-connect

robin@atlassian.com / @rewbs