qcon sp 2016 - video authorization: from chaos to 25ms response time
TRANSCRIPT
![Page 1: QCon SP 2016 - Video Authorization: from chaos to 25ms response time](https://reader031.vdocuments.site/reader031/viewer/2022030308/58ed120f1a28aba5398b46fb/html5/thumbnails/1.jpg)
Video AuthorizationFrom chaos to
25ms response time
![Page 2: QCon SP 2016 - Video Authorization: from chaos to 25ms response time](https://reader031.vdocuments.site/reader031/viewer/2022030308/58ed120f1a28aba5398b46fb/html5/thumbnails/2.jpg)
Emerson Macedo@emerleite
https://emerleite.com
![Page 3: QCon SP 2016 - Video Authorization: from chaos to 25ms response time](https://reader031.vdocuments.site/reader031/viewer/2022030308/58ed120f1a28aba5398b46fb/html5/thumbnails/3.jpg)
![Page 4: QCon SP 2016 - Video Authorization: from chaos to 25ms response time](https://reader031.vdocuments.site/reader031/viewer/2022030308/58ed120f1a28aba5398b46fb/html5/thumbnails/4.jpg)
![Page 5: QCon SP 2016 - Video Authorization: from chaos to 25ms response time](https://reader031.vdocuments.site/reader031/viewer/2022030308/58ed120f1a28aba5398b46fb/html5/thumbnails/5.jpg)
Content authorizationis always
a challenge
![Page 6: QCon SP 2016 - Video Authorization: from chaos to 25ms response time](https://reader031.vdocuments.site/reader031/viewer/2022030308/58ed120f1a28aba5398b46fb/html5/thumbnails/6.jpg)
Video Authorization process
![Page 7: QCon SP 2016 - Video Authorization: from chaos to 25ms response time](https://reader031.vdocuments.site/reader031/viewer/2022030308/58ed120f1a28aba5398b46fb/html5/thumbnails/7.jpg)
Video Authorization process
![Page 8: QCon SP 2016 - Video Authorization: from chaos to 25ms response time](https://reader031.vdocuments.site/reader031/viewer/2022030308/58ed120f1a28aba5398b46fb/html5/thumbnails/8.jpg)
![Page 9: QCon SP 2016 - Video Authorization: from chaos to 25ms response time](https://reader031.vdocuments.site/reader031/viewer/2022030308/58ed120f1a28aba5398b46fb/html5/thumbnails/9.jpg)
![Page 10: QCon SP 2016 - Video Authorization: from chaos to 25ms response time](https://reader031.vdocuments.site/reader031/viewer/2022030308/58ed120f1a28aba5398b46fb/html5/thumbnails/10.jpg)
Video authorizationmust be really
very fast
![Page 11: QCon SP 2016 - Video Authorization: from chaos to 25ms response time](https://reader031.vdocuments.site/reader031/viewer/2022030308/58ed120f1a28aba5398b46fb/html5/thumbnails/11.jpg)
Authorization Provider
Rules
![Page 12: QCon SP 2016 - Video Authorization: from chaos to 25ms response time](https://reader031.vdocuments.site/reader031/viewer/2022030308/58ed120f1a28aba5398b46fb/html5/thumbnails/12.jpg)
1 - User has a valid session
![Page 13: QCon SP 2016 - Video Authorization: from chaos to 25ms response time](https://reader031.vdocuments.site/reader031/viewer/2022030308/58ed120f1a28aba5398b46fb/html5/thumbnails/13.jpg)
2 - User has the channel of this video in his Pay TV
subscription
![Page 14: QCon SP 2016 - Video Authorization: from chaos to 25ms response time](https://reader031.vdocuments.site/reader031/viewer/2022030308/58ed120f1a28aba5398b46fb/html5/thumbnails/14.jpg)
3 - User parental control matches video content
rating
![Page 15: QCon SP 2016 - Video Authorization: from chaos to 25ms response time](https://reader031.vdocuments.site/reader031/viewer/2022030308/58ed120f1a28aba5398b46fb/html5/thumbnails/15.jpg)
OR ...
![Page 16: QCon SP 2016 - Video Authorization: from chaos to 25ms response time](https://reader031.vdocuments.site/reader031/viewer/2022030308/58ed120f1a28aba5398b46fb/html5/thumbnails/16.jpg)
1 - User has a valid session
![Page 17: QCon SP 2016 - Video Authorization: from chaos to 25ms response time](https://reader031.vdocuments.site/reader031/viewer/2022030308/58ed120f1a28aba5398b46fb/html5/thumbnails/17.jpg)
2 - Video channel has a trial happening
![Page 18: QCon SP 2016 - Video Authorization: from chaos to 25ms response time](https://reader031.vdocuments.site/reader031/viewer/2022030308/58ed120f1a28aba5398b46fb/html5/thumbnails/18.jpg)
3 - User is able to join the current trial
![Page 19: QCon SP 2016 - Video Authorization: from chaos to 25ms response time](https://reader031.vdocuments.site/reader031/viewer/2022030308/58ed120f1a28aba5398b46fb/html5/thumbnails/19.jpg)
Video Authorization process
![Page 20: QCon SP 2016 - Video Authorization: from chaos to 25ms response time](https://reader031.vdocuments.site/reader031/viewer/2022030308/58ed120f1a28aba5398b46fb/html5/thumbnails/20.jpg)
response time > 5 secondsCircuit Breaker opens
![Page 21: QCon SP 2016 - Video Authorization: from chaos to 25ms response time](https://reader031.vdocuments.site/reader031/viewer/2022030308/58ed120f1a28aba5398b46fb/html5/thumbnails/21.jpg)
![Page 22: QCon SP 2016 - Video Authorization: from chaos to 25ms response time](https://reader031.vdocuments.site/reader031/viewer/2022030308/58ed120f1a28aba5398b46fb/html5/thumbnails/22.jpg)
Nobody can playany subscriber only
video
![Page 23: QCon SP 2016 - Video Authorization: from chaos to 25ms response time](https://reader031.vdocuments.site/reader031/viewer/2022030308/58ed120f1a28aba5398b46fb/html5/thumbnails/23.jpg)
We need todo something
about this
![Page 24: QCon SP 2016 - Video Authorization: from chaos to 25ms response time](https://reader031.vdocuments.site/reader031/viewer/2022030308/58ed120f1a28aba5398b46fb/html5/thumbnails/24.jpg)
1 - split the circuit breaker
![Page 25: QCon SP 2016 - Video Authorization: from chaos to 25ms response time](https://reader031.vdocuments.site/reader031/viewer/2022030308/58ed120f1a28aba5398b46fb/html5/thumbnails/25.jpg)
The circuit breaker was like this
![Page 26: QCon SP 2016 - Video Authorization: from chaos to 25ms response time](https://reader031.vdocuments.site/reader031/viewer/2022030308/58ed120f1a28aba5398b46fb/html5/thumbnails/26.jpg)
The new circuit breaker is per provider
![Page 27: QCon SP 2016 - Video Authorization: from chaos to 25ms response time](https://reader031.vdocuments.site/reader031/viewer/2022030308/58ed120f1a28aba5398b46fb/html5/thumbnails/27.jpg)
![Page 28: QCon SP 2016 - Video Authorization: from chaos to 25ms response time](https://reader031.vdocuments.site/reader031/viewer/2022030308/58ed120f1a28aba5398b46fb/html5/thumbnails/28.jpg)
A circuit open for one provider
should not affect any other providers
![Page 29: QCon SP 2016 - Video Authorization: from chaos to 25ms response time](https://reader031.vdocuments.site/reader031/viewer/2022030308/58ed120f1a28aba5398b46fb/html5/thumbnails/29.jpg)
2 - Optimize Auth Provider
![Page 30: QCon SP 2016 - Video Authorization: from chaos to 25ms response time](https://reader031.vdocuments.site/reader031/viewer/2022030308/58ed120f1a28aba5398b46fb/html5/thumbnails/30.jpg)
Globosat PlayAuthorization Provider
![Page 31: QCon SP 2016 - Video Authorization: from chaos to 25ms response time](https://reader031.vdocuments.site/reader031/viewer/2022030308/58ed120f1a28aba5398b46fb/html5/thumbnails/31.jpg)
Usually~ 500ms response time
![Page 32: QCon SP 2016 - Video Authorization: from chaos to 25ms response time](https://reader031.vdocuments.site/reader031/viewer/2022030308/58ed120f1a28aba5398b46fb/html5/thumbnails/32.jpg)
Sometimes5s to 10s response time
![Page 33: QCon SP 2016 - Video Authorization: from chaos to 25ms response time](https://reader031.vdocuments.site/reader031/viewer/2022030308/58ed120f1a28aba5398b46fb/html5/thumbnails/33.jpg)
Video authorizationmust be really
very fast
![Page 34: QCon SP 2016 - Video Authorization: from chaos to 25ms response time](https://reader031.vdocuments.site/reader031/viewer/2022030308/58ed120f1a28aba5398b46fb/html5/thumbnails/34.jpg)
![Page 35: QCon SP 2016 - Video Authorization: from chaos to 25ms response time](https://reader031.vdocuments.site/reader031/viewer/2022030308/58ed120f1a28aba5398b46fb/html5/thumbnails/35.jpg)
New Relic Transaction Trace
![Page 36: QCon SP 2016 - Video Authorization: from chaos to 25ms response time](https://reader031.vdocuments.site/reader031/viewer/2022030308/58ed120f1a28aba5398b46fb/html5/thumbnails/36.jpg)
New Relic Transaction Trace
![Page 37: QCon SP 2016 - Video Authorization: from chaos to 25ms response time](https://reader031.vdocuments.site/reader031/viewer/2022030308/58ed120f1a28aba5398b46fb/html5/thumbnails/37.jpg)
2.1 - Snapshot someexternal services data
![Page 38: QCon SP 2016 - Video Authorization: from chaos to 25ms response time](https://reader031.vdocuments.site/reader031/viewer/2022030308/58ed120f1a28aba5398b46fb/html5/thumbnails/38.jpg)
Globosat PlayAuthorization Provider
![Page 39: QCon SP 2016 - Video Authorization: from chaos to 25ms response time](https://reader031.vdocuments.site/reader031/viewer/2022030308/58ed120f1a28aba5398b46fb/html5/thumbnails/39.jpg)
After every user sign in
![Page 40: QCon SP 2016 - Video Authorization: from chaos to 25ms response time](https://reader031.vdocuments.site/reader031/viewer/2022030308/58ed120f1a28aba5398b46fb/html5/thumbnails/40.jpg)
Before User InfoSnapshot data
![Page 41: QCon SP 2016 - Video Authorization: from chaos to 25ms response time](https://reader031.vdocuments.site/reader031/viewer/2022030308/58ed120f1a28aba5398b46fb/html5/thumbnails/41.jpg)
After User InfoSnapshot data
![Page 42: QCon SP 2016 - Video Authorization: from chaos to 25ms response time](https://reader031.vdocuments.site/reader031/viewer/2022030308/58ed120f1a28aba5398b46fb/html5/thumbnails/42.jpg)
On user video authorization
![Page 43: QCon SP 2016 - Video Authorization: from chaos to 25ms response time](https://reader031.vdocuments.site/reader031/viewer/2022030308/58ed120f1a28aba5398b46fb/html5/thumbnails/43.jpg)
Before Profile ConfigSnapshot data
![Page 44: QCon SP 2016 - Video Authorization: from chaos to 25ms response time](https://reader031.vdocuments.site/reader031/viewer/2022030308/58ed120f1a28aba5398b46fb/html5/thumbnails/44.jpg)
After Profile ConfigSnapshot data
![Page 45: QCon SP 2016 - Video Authorization: from chaos to 25ms response time](https://reader031.vdocuments.site/reader031/viewer/2022030308/58ed120f1a28aba5398b46fb/html5/thumbnails/45.jpg)
On any video authorization
![Page 46: QCon SP 2016 - Video Authorization: from chaos to 25ms response time](https://reader031.vdocuments.site/reader031/viewer/2022030308/58ed120f1a28aba5398b46fb/html5/thumbnails/46.jpg)
Before Video infoSnapshot data
![Page 47: QCon SP 2016 - Video Authorization: from chaos to 25ms response time](https://reader031.vdocuments.site/reader031/viewer/2022030308/58ed120f1a28aba5398b46fb/html5/thumbnails/47.jpg)
After Vide InfoSnapshot data
![Page 48: QCon SP 2016 - Video Authorization: from chaos to 25ms response time](https://reader031.vdocuments.site/reader031/viewer/2022030308/58ed120f1a28aba5398b46fb/html5/thumbnails/48.jpg)
In the worst scenarioWe have THREE external calls
![Page 49: QCon SP 2016 - Video Authorization: from chaos to 25ms response time](https://reader031.vdocuments.site/reader031/viewer/2022030308/58ed120f1a28aba5398b46fb/html5/thumbnails/49.jpg)
In the best scenarioWe have NO external calls
![Page 50: QCon SP 2016 - Video Authorization: from chaos to 25ms response time](https://reader031.vdocuments.site/reader031/viewer/2022030308/58ed120f1a28aba5398b46fb/html5/thumbnails/50.jpg)
2.2 - Be safe with allexternal services calls
![Page 68: QCon SP 2016 - Video Authorization: from chaos to 25ms response time](https://reader031.vdocuments.site/reader031/viewer/2022030308/58ed120f1a28aba5398b46fb/html5/thumbnails/68.jpg)
![Page 70: QCon SP 2016 - Video Authorization: from chaos to 25ms response time](https://reader031.vdocuments.site/reader031/viewer/2022030308/58ed120f1a28aba5398b46fb/html5/thumbnails/70.jpg)
2.3 - Cache logged requests
when it's safe
![Page 71: QCon SP 2016 - Video Authorization: from chaos to 25ms response time](https://reader031.vdocuments.site/reader031/viewer/2022030308/58ed120f1a28aba5398b46fb/html5/thumbnails/71.jpg)
Logged request - Success
![Page 72: QCon SP 2016 - Video Authorization: from chaos to 25ms response time](https://reader031.vdocuments.site/reader031/viewer/2022030308/58ed120f1a28aba5398b46fb/html5/thumbnails/72.jpg)
Logged request - Failure
![Page 73: QCon SP 2016 - Video Authorization: from chaos to 25ms response time](https://reader031.vdocuments.site/reader031/viewer/2022030308/58ed120f1a28aba5398b46fb/html5/thumbnails/73.jpg)
Nginx Cache Config
![Page 74: QCon SP 2016 - Video Authorization: from chaos to 25ms response time](https://reader031.vdocuments.site/reader031/viewer/2022030308/58ed120f1a28aba5398b46fb/html5/thumbnails/74.jpg)
With Nginx cache
![Page 75: QCon SP 2016 - Video Authorization: from chaos to 25ms response time](https://reader031.vdocuments.site/reader031/viewer/2022030308/58ed120f1a28aba5398b46fb/html5/thumbnails/75.jpg)
We dida very intesive job
and reduced theresponse time
![Page 76: QCon SP 2016 - Video Authorization: from chaos to 25ms response time](https://reader031.vdocuments.site/reader031/viewer/2022030308/58ed120f1a28aba5398b46fb/html5/thumbnails/76.jpg)
Usually~25ms response time
![Page 77: QCon SP 2016 - Video Authorization: from chaos to 25ms response time](https://reader031.vdocuments.site/reader031/viewer/2022030308/58ed120f1a28aba5398b46fb/html5/thumbnails/77.jpg)
Usually~25ms response time
![Page 78: QCon SP 2016 - Video Authorization: from chaos to 25ms response time](https://reader031.vdocuments.site/reader031/viewer/2022030308/58ed120f1a28aba5398b46fb/html5/thumbnails/78.jpg)
BUT ...
![Page 79: QCon SP 2016 - Video Authorization: from chaos to 25ms response time](https://reader031.vdocuments.site/reader031/viewer/2022030308/58ed120f1a28aba5398b46fb/html5/thumbnails/79.jpg)
Sometimes5s to 10s response time
![Page 80: QCon SP 2016 - Video Authorization: from chaos to 25ms response time](https://reader031.vdocuments.site/reader031/viewer/2022030308/58ed120f1a28aba5398b46fb/html5/thumbnails/80.jpg)
Sometimes5s to 10s response time
![Page 81: QCon SP 2016 - Video Authorization: from chaos to 25ms response time](https://reader031.vdocuments.site/reader031/viewer/2022030308/58ed120f1a28aba5398b46fb/html5/thumbnails/81.jpg)
3 - Take a look at ourinfrastructure
![Page 82: QCon SP 2016 - Video Authorization: from chaos to 25ms response time](https://reader031.vdocuments.site/reader031/viewer/2022030308/58ed120f1a28aba5398b46fb/html5/thumbnails/82.jpg)
3.1 - MongoDB
![Page 83: QCon SP 2016 - Video Authorization: from chaos to 25ms response time](https://reader031.vdocuments.site/reader031/viewer/2022030308/58ed120f1a28aba5398b46fb/html5/thumbnails/83.jpg)
MongoDB has a bad reputation in software developer community
![Page 84: QCon SP 2016 - Video Authorization: from chaos to 25ms response time](https://reader031.vdocuments.site/reader031/viewer/2022030308/58ed120f1a28aba5398b46fb/html5/thumbnails/84.jpg)
Everybody Blames Mongo everytime 2
2 http://rhaas.blogspot.com.br/2014/04/why-clock-is-ticking-for-mongodb.html
![Page 85: QCon SP 2016 - Video Authorization: from chaos to 25ms response time](https://reader031.vdocuments.site/reader031/viewer/2022030308/58ed120f1a28aba5398b46fb/html5/thumbnails/85.jpg)
Everybody Blames Mongo everytime 3
3 http://cryto.net/~joepie91/blog/2015/07/19/why-you-should-never-ever-ever-use-mongodb/
![Page 86: QCon SP 2016 - Video Authorization: from chaos to 25ms response time](https://reader031.vdocuments.site/reader031/viewer/2022030308/58ed120f1a28aba5398b46fb/html5/thumbnails/86.jpg)
Database change isan architectural decision
![Page 87: QCon SP 2016 - Video Authorization: from chaos to 25ms response time](https://reader031.vdocuments.site/reader031/viewer/2022030308/58ed120f1a28aba5398b46fb/html5/thumbnails/87.jpg)
architectural decisionsare hard and very
expensive to change
![Page 88: QCon SP 2016 - Video Authorization: from chaos to 25ms response time](https://reader031.vdocuments.site/reader031/viewer/2022030308/58ed120f1a28aba5398b46fb/html5/thumbnails/88.jpg)
Sometimes5s to 10s response time
![Page 89: QCon SP 2016 - Video Authorization: from chaos to 25ms response time](https://reader031.vdocuments.site/reader031/viewer/2022030308/58ed120f1a28aba5398b46fb/html5/thumbnails/89.jpg)
Strange query convultions
![Page 90: QCon SP 2016 - Video Authorization: from chaos to 25ms response time](https://reader031.vdocuments.site/reader031/viewer/2022030308/58ed120f1a28aba5398b46fb/html5/thumbnails/90.jpg)
Traces confirms convultions
![Page 91: QCon SP 2016 - Video Authorization: from chaos to 25ms response time](https://reader031.vdocuments.site/reader031/viewer/2022030308/58ed120f1a28aba5398b46fb/html5/thumbnails/91.jpg)
Logs confirms convultions
![Page 92: QCon SP 2016 - Video Authorization: from chaos to 25ms response time](https://reader031.vdocuments.site/reader031/viewer/2022030308/58ed120f1a28aba5398b46fb/html5/thumbnails/92.jpg)
We checked all indexesand found nothinghurting our system
![Page 93: QCon SP 2016 - Video Authorization: from chaos to 25ms response time](https://reader031.vdocuments.site/reader031/viewer/2022030308/58ed120f1a28aba5398b46fb/html5/thumbnails/93.jpg)
3.2 - Servers
![Page 94: QCon SP 2016 - Video Authorization: from chaos to 25ms response time](https://reader031.vdocuments.site/reader031/viewer/2022030308/58ed120f1a28aba5398b46fb/html5/thumbnails/94.jpg)
CPU Wait confirms convultions
![Page 95: QCon SP 2016 - Video Authorization: from chaos to 25ms response time](https://reader031.vdocuments.site/reader031/viewer/2022030308/58ed120f1a28aba5398b46fb/html5/thumbnails/95.jpg)
Load average confirms convultions
![Page 96: QCon SP 2016 - Video Authorization: from chaos to 25ms response time](https://reader031.vdocuments.site/reader031/viewer/2022030308/58ed120f1a28aba5398b46fb/html5/thumbnails/96.jpg)
MongoDB docs confirms thatSomething was wrong 4
4 https://docs.mongodb.org/manual/administration/production-notes/#remote-filesystems
![Page 97: QCon SP 2016 - Video Authorization: from chaos to 25ms response time](https://reader031.vdocuments.site/reader031/viewer/2022030308/58ed120f1a28aba5398b46fb/html5/thumbnails/97.jpg)
MongoDB users group confirms thatSomething was wrong 5
5 https://groups.google.com/forum/#!msg/mongodb-user/Kd85b2HHVn8/7SnwTyeQKsEJ
![Page 98: QCon SP 2016 - Video Authorization: from chaos to 25ms response time](https://reader031.vdocuments.site/reader031/viewer/2022030308/58ed120f1a28aba5398b46fb/html5/thumbnails/98.jpg)
We removed NFSfrom our Mongodb
servers setup
![Page 99: QCon SP 2016 - Video Authorization: from chaos to 25ms response time](https://reader031.vdocuments.site/reader031/viewer/2022030308/58ed120f1a28aba5398b46fb/html5/thumbnails/99.jpg)
convultions stopped
![Page 100: QCon SP 2016 - Video Authorization: from chaos to 25ms response time](https://reader031.vdocuments.site/reader031/viewer/2022030308/58ed120f1a28aba5398b46fb/html5/thumbnails/100.jpg)
Usually~25ms response time
![Page 101: QCon SP 2016 - Video Authorization: from chaos to 25ms response time](https://reader031.vdocuments.site/reader031/viewer/2022030308/58ed120f1a28aba5398b46fb/html5/thumbnails/101.jpg)
Usually~25ms response time
![Page 102: QCon SP 2016 - Video Authorization: from chaos to 25ms response time](https://reader031.vdocuments.site/reader031/viewer/2022030308/58ed120f1a28aba5398b46fb/html5/thumbnails/102.jpg)
AND ...
![Page 103: QCon SP 2016 - Video Authorization: from chaos to 25ms response time](https://reader031.vdocuments.site/reader031/viewer/2022030308/58ed120f1a28aba5398b46fb/html5/thumbnails/103.jpg)
Sometimes100ms to 500ms response time
![Page 104: QCon SP 2016 - Video Authorization: from chaos to 25ms response time](https://reader031.vdocuments.site/reader031/viewer/2022030308/58ed120f1a28aba5398b46fb/html5/thumbnails/104.jpg)
Lessons Learned
![Page 105: QCon SP 2016 - Video Authorization: from chaos to 25ms response time](https://reader031.vdocuments.site/reader031/viewer/2022030308/58ed120f1a28aba5398b46fb/html5/thumbnails/105.jpg)
1 - Go deep into your problem
![Page 106: QCon SP 2016 - Video Authorization: from chaos to 25ms response time](https://reader031.vdocuments.site/reader031/viewer/2022030308/58ed120f1a28aba5398b46fb/html5/thumbnails/106.jpg)
2 - Don't panic
![Page 107: QCon SP 2016 - Video Authorization: from chaos to 25ms response time](https://reader031.vdocuments.site/reader031/viewer/2022030308/58ed120f1a28aba5398b46fb/html5/thumbnails/107.jpg)
3 - Most of the timethe solution is not
to rewrite from scratch
![Page 108: QCon SP 2016 - Video Authorization: from chaos to 25ms response time](https://reader031.vdocuments.site/reader031/viewer/2022030308/58ed120f1a28aba5398b46fb/html5/thumbnails/108.jpg)
Questions?
![Page 109: QCon SP 2016 - Video Authorization: from chaos to 25ms response time](https://reader031.vdocuments.site/reader031/viewer/2022030308/58ed120f1a28aba5398b46fb/html5/thumbnails/109.jpg)