the last mile

The Last Mile

Disclaimer: I’m only using Windows for PowerPoint. Please don’t judge.

@StephenMelroseLead Software Engineer - BSkyB

Photo by Ousseynou Cisséhttps://www.flickr.com/photos/afuelcalledlove/

Are Recruiting!Developers, DevOps, Scala/Hadoop,

Node.js Super Heroes, Principle Test Engineers,an Engineering Manager, and more!

victoria.howling@bskyb.comFor more info, please see/contact:

SPIN Connector

Context

Image source: http://blog.burningman.com/wp-content/uploads/2012/09/blog4.jpg

(a.k.a. SPIN)

Image source: http://www.psdgraphics.com/file/adhesive-plaster.jpg

Image source: http://patriciaknight.files.wordpress.com/2011/08/change-architect-sign1.jpg

Image source: http://3.bp.blogspot.com/-Kdzzg4pimz0/Udk6bE5tDvI/AAAAAAAAFes/MxIGspcwu3M/s1600/photo.JPG

TestingPerformanceExtendibility

Supportability

Logging

Request failed

Connection lost

Failed to perform action

Request failed

Connection lost

Failed to perform action

Request failed

Connection lost

Failed to perform action

Request failed

Connection lost

Failed to perform action

UsefulInformative

Failed to start-up controller for fixture [gC6RFPZ0IW7Zv_vqFR78Aj6_05c]: Failed to get AMQP details from SPIN Connect API: Failed to get stream AMQP details: HTTP request failed: ETIMEDOUT - Request: {"url":"http://<api-domain>/UnifiedDataAPI/stream/IceHockey/gC6RFPZ0IW7Zv_vqFR78Aj6_05c","options":{"headers":{"Accept-Encoding":"gzip","X-Auth-Token":"XXXXXX"},"timeout":30000,"attempt":2,"uri":"http://<api-domain>/UnifiedDataAPI/stream/IceHockey/gC6RFPZ0IW7Zv_vqFR78Aj6_05c"}}

Failed to start-up controller for fixture [gC6RFPZ0IW7Zv_vqFR78Aj6_05c]: Failed to get AMQP details from SPIN Connect API: Failed to get stream AMQP details: HTTP request failed: ETIMEDOUT - Request: {"url":"http://<api-domain>/UnifiedDataAPI/stream/IceHockey/gC6RFPZ0IW7Zv_vqFR78Aj6_05c","options":{"headers":{"Accept-Encoding":"gzip","X-Auth-Token":"XXXXXX"},"timeout":30000,"attempt":2,"uri":"http://<api-domain>/UnifiedDataAPI/stream/IceHockey/gC6RFPZ0IW7Zv_vqFR78Aj6_05c"}}

Failed to start-up controller for fixture [gC6RFPZ0IW7Zv_vqFR78Aj6_05c]: Failed to get AMQP details from SPIN Connect API: Failed to get stream AMQP details: HTTP request failed: ETIMEDOUT - Request: {"url":"http://<api-domain>/UnifiedDataAPI/stream/IceHockey/gC6RFPZ0IW7Zv_vqFR78Aj6_05c","options":{"headers":{"Accept-Encoding":"gzip","X-Auth-Token":"XXXXXX"},"timeout":30000,"attempt":2,"uri":"http://<api-domain>/UnifiedDataAPI/stream/IceHockey/gC6RFPZ0IW7Zv_vqFR78Aj6_05c"}}

Failed to start-up controller for fixture [gC6RFPZ0IW7Zv_vqFR78Aj6_05c]: Failed to get AMQP details from SPIN Connect API: Failed to get stream AMQP details: HTTP request failed: ETIMEDOUT - Request: {"url":"http://<api-domain>/UnifiedDataAPI/stream/IceHockey/gC6RFPZ0IW7Zv_vqFR78Aj6_05c","options":{"headers":{"Accept-Encoding":"gzip","X-Auth-Token":"XXXXXX"},"timeout":30000,"attempt":2,"uri":"http://<api-domain>/UnifiedDataAPI/stream/IceHockey/gC6RFPZ0IW7Zv_vqFR78Aj6_05c"}}

Failed to start-up controller for fixture [gC6RFPZ0IW7Zv_vqFR78Aj6_05c]: Failed to get AMQP details from SPIN Connect API: Failed to get stream AMQP details: HTTP request failed: ETIMEDOUT - Request: {"url":"http://<api-domain>/UnifiedDataAPI/stream/IceHockey/gC6RFPZ0IW7Zv_vqFR78Aj6_05c","options":{"headers":{"Accept-Encoding":"gzip","X-Auth-Token":"XXXXXX"},"timeout":30000,"attempt":2,"uri":"http://<api-domain>/UnifiedDataAPI/stream/IceHockey/gC6RFPZ0IW7Zv_vqFR78Aj6_05c"}}

Log Tail Video

Nothing

The Syslog Protocol (RFC 5424)

DEBUG Verbose and raw debug information. Use in development only.e.g. request/response bodies

INFO 95% of our logging. Stuff we want to monitor, e.g. HTTP request complete, took X seconds, performed Y actions

NOTICE Similar to INFO, but significant events.Also, errors that CAN happen and we HAVE HANDLED

WARN Errors that SHOULDN’T happen, but we HAVE HANDLED

ERROR Errors that SHOULDN’T happen and we CAN’T HANDLE

CRITICAL Fatal errors that cause the application to crash

Graphs

Node.js LogProcessor Graphitestatsd

Monitoring

Load Testing

SPIN’s API OpenBetSPIN Connector

SPIN Recorder

SPIN’s API OpenBetSPIN Connector

Fake SPIN API Our EntireFrontend Stack

Released to Live

Information exposure is paramountKnow what your app is doing

Automate the monitoringMake sure it can do it

Thank youAny questions?

@StephenMelrosehttp://stephenmelrose.co.uk/