webcamp 2016: python. Вячеслав Каковский: real-time мессенджер на...
TRANSCRIPT
![Page 1: WebCamp 2016: Python. Вячеслав Каковский: Real-time мессенджер на Python. Особенности разработки back-end](https://reader033.vdocuments.site/reader033/viewer/2022042723/58d1da671a28ab66108b5667/html5/thumbnails/1.jpg)
![Page 2: WebCamp 2016: Python. Вячеслав Каковский: Real-time мессенджер на Python. Особенности разработки back-end](https://reader033.vdocuments.site/reader033/viewer/2022042723/58d1da671a28ab66108b5667/html5/thumbnails/2.jpg)
Me!
@kakovskyi
Python Developer at SoftServeContributor of Atlassian HipChat — Python 2, TwistedMaintainer of KPIdata — Python 3, asyncio
2
![Page 3: WebCamp 2016: Python. Вячеслав Каковский: Real-time мессенджер на Python. Особенности разработки back-end](https://reader033.vdocuments.site/reader033/viewer/2022042723/58d1da671a28ab66108b5667/html5/thumbnails/3.jpg)
Agenda
● What is 'instant messenger'?● Related projects from my experience● Messaging protocols● Life of messaging platform● Lessons learned● Summary● Further reading
3
![Page 4: WebCamp 2016: Python. Вячеслав Каковский: Real-time мессенджер на Python. Особенности разработки back-end](https://reader033.vdocuments.site/reader033/viewer/2022042723/58d1da671a28ab66108b5667/html5/thumbnails/4.jpg)
What is 'instant messenger'?
4
![Page 5: WebCamp 2016: Python. Вячеслав Каковский: Real-time мессенджер на Python. Особенности разработки back-end](https://reader033.vdocuments.site/reader033/viewer/2022042723/58d1da671a28ab66108b5667/html5/thumbnails/5.jpg)
What is 'instant messenger'?
● online chat● real-time delivery● short messages
5
![Page 6: WebCamp 2016: Python. Вячеслав Каковский: Real-time мессенджер на Python. Особенности разработки back-end](https://reader033.vdocuments.site/reader033/viewer/2022042723/58d1da671a28ab66108b5667/html5/thumbnails/6.jpg)
What is 'instant messenger'?
● history search● file sharing● mobile push notifications● video calling● bots and integrations
6
![Page 7: WebCamp 2016: Python. Вячеслав Каковский: Real-time мессенджер на Python. Особенности разработки back-end](https://reader033.vdocuments.site/reader033/viewer/2022042723/58d1da671a28ab66108b5667/html5/thumbnails/7.jpg)
Related projects from my experience
● Hosted chat for teams and enterprises● Founded in 2009 by 3 students● 100 000+ connected users● 100+ nodes● REST API for integrations and bots● Built with Python 2 and Twisted
7
![Page 8: WebCamp 2016: Python. Вячеслав Каковский: Real-time мессенджер на Python. Особенности разработки back-end](https://reader033.vdocuments.site/reader033/viewer/2022042723/58d1da671a28ab66108b5667/html5/thumbnails/8.jpg)
Messaging protocols Protocol is about:
● Message format● Allowed types of messages● Limitations● Routine
○ How to encode data?○ How to establish/close connection?○ How to authenticate?○ How to encrypt?
8
![Page 9: WebCamp 2016: Python. Вячеслав Каковский: Real-time мессенджер на Python. Особенности разработки back-end](https://reader033.vdocuments.site/reader033/viewer/2022042723/58d1da671a28ab66108b5667/html5/thumbnails/9.jpg)
Messaging protocols ● OSCAR (1997)
● XMPP (1999)
● Skype (2003)
● WebSocket-based (2011)
● MQTT, MTProto, DHT-based, etc.
9
![Page 10: WebCamp 2016: Python. Вячеслав Каковский: Real-time мессенджер на Python. Особенности разработки back-end](https://reader033.vdocuments.site/reader033/viewer/2022042723/58d1da671a28ab66108b5667/html5/thumbnails/10.jpg)
XMPP
● XMPP - signaling protocol● BOSH - transport protocol● Started from Jabber in 1999● XML as a message format● Stanza - basic unit in XMPP● Types of stanzas:
○ Message○ Presence○ Info/Query
10
![Page 11: WebCamp 2016: Python. Вячеслав Каковский: Real-time мессенджер на Python. Особенности разработки back-end](https://reader033.vdocuments.site/reader033/viewer/2022042723/58d1da671a28ab66108b5667/html5/thumbnails/11.jpg)
XMPP
● Extensions defined by XEPs (XMPP Extension Protocols):○ Bidirectional-streams Over Synchronous
HTTP (BOSH)○ Serverless messaging○ File transfer and etc.
11
![Page 12: WebCamp 2016: Python. Вячеслав Каковский: Real-time мессенджер на Python. Особенности разработки back-end](https://reader033.vdocuments.site/reader033/viewer/2022042723/58d1da671a28ab66108b5667/html5/thumbnails/12.jpg)
XMPP: Establishing a connection
12
Client:
<?xml version='1.0'?> <stream:stream to='example.com' xmlns='jabber:client' xmlns:stream='http://etherx.jabber.org/streams' version='1.0'>
Server:
<?xml version='1.0'?> <stream:stream from='example.com' id='someid' xmlns='jabber:client' xmlns:stream='http://etherx.jabber.org/streams' version='1.0'>
![Page 13: WebCamp 2016: Python. Вячеслав Каковский: Real-time мессенджер на Python. Особенности разработки back-end](https://reader033.vdocuments.site/reader033/viewer/2022042723/58d1da671a28ab66108b5667/html5/thumbnails/13.jpg)
XMPP: Sending a message
13
Client:
<message from='[email protected]' to='[email protected]' xml:lang='en'> <body>Art thou not Romeo, and a Montague?</body></message>
Server:
<message from='[email protected]' to='[email protected]' xml:lang='en'> <body> Neither, fair saint, if either thee dislike. </body></message>
![Page 14: WebCamp 2016: Python. Вячеслав Каковский: Real-time мессенджер на Python. Особенности разработки back-end](https://reader033.vdocuments.site/reader033/viewer/2022042723/58d1da671a28ab66108b5667/html5/thumbnails/14.jpg)
XMPP: Closing a connection
14
Client:
</stream:stream>
Server:
</stream:stream>
![Page 15: WebCamp 2016: Python. Вячеслав Каковский: Real-time мессенджер на Python. Особенности разработки back-end](https://reader033.vdocuments.site/reader033/viewer/2022042723/58d1da671a28ab66108b5667/html5/thumbnails/15.jpg)
XMPP: Pros
● Robust and standardized● Extendable via XEPs● Secured● Native support of multi-sessions● A lot of clients implementations
15
![Page 16: WebCamp 2016: Python. Вячеслав Каковский: Real-time мессенджер на Python. Особенности разработки back-end](https://reader033.vdocuments.site/reader033/viewer/2022042723/58d1da671a28ab66108b5667/html5/thumbnails/16.jpg)
XMPP: Cons
● Overhead○ Presence ○ Downloading the World on startup
● XML○ Large documents○ Expensive parsing
16
![Page 17: WebCamp 2016: Python. Вячеслав Каковский: Real-time мессенджер на Python. Особенности разработки back-end](https://reader033.vdocuments.site/reader033/viewer/2022042723/58d1da671a28ab66108b5667/html5/thumbnails/17.jpg)
XMPP and Python
● Servers:○ TwistedWords - good place to start○ Tornado-based example○ aioxmpp○ XMPPFlask○ Punjab - BOSH-server on Twisted
17
![Page 18: WebCamp 2016: Python. Вячеслав Каковский: Real-time мессенджер на Python. Особенности разработки back-end](https://reader033.vdocuments.site/reader033/viewer/2022042723/58d1da671a28ab66108b5667/html5/thumbnails/18.jpg)
XMPP and Python
● Clients:○ SleekXMPP - mature and solid○ Slixmpp - asyncio-support○ TwistedWords○ Wokkel - Twisted-based○ xmpp.py
● JS-client: Strophe.js
18
![Page 19: WebCamp 2016: Python. Вячеслав Каковский: Real-time мессенджер на Python. Особенности разработки back-end](https://reader033.vdocuments.site/reader033/viewer/2022042723/58d1da671a28ab66108b5667/html5/thumbnails/19.jpg)
WebSocket-based solutions
● WebSocket - transport protocol● Standardized in 2011 by W3C● Full-duplex communication channel● JSON as a message format● Custom message types
19
![Page 20: WebCamp 2016: Python. Вячеслав Каковский: Real-time мессенджер на Python. Особенности разработки back-end](https://reader033.vdocuments.site/reader033/viewer/2022042723/58d1da671a28ab66108b5667/html5/thumbnails/20.jpg)
WebSocket: Establishing a connection
20
Client:
GET /chat HTTP/1.1
Host: server.example.com
Upgrade: websocket
Connection: Upgrade
Sec-WebSocket-Key: dGhlIHNhbXBsZSBub25jZQ==
Origin: http://example.com
Sec-WebSocket-Protocol: chat, superchat
Sec-WebSocket-Version: 13
Server:
HTTP/1.1 101 Switching Protocols
Upgrade: websocket
Connection: Upgrade
Sec-WebSocket-Accept: s3pPLMBiTxaQ9kYGzzhZRbK+xOo=
![Page 21: WebCamp 2016: Python. Вячеслав Каковский: Real-time мессенджер на Python. Особенности разработки back-end](https://reader033.vdocuments.site/reader033/viewer/2022042723/58d1da671a28ab66108b5667/html5/thumbnails/21.jpg)
WebSocket: Sending a message
21
Client:
{
"type": "message",
"ts": 1469563519,
"user": "kakovskyi",
"text": "Hello, @WebCamp!"
}
Server:
{
"type": "notification",
"ts": 1469563519,
"user": "WebCamp Bot",
"text": "Howdy @kakovskyi?"
}
![Page 22: WebCamp 2016: Python. Вячеслав Каковский: Real-time мессенджер на Python. Особенности разработки back-end](https://reader033.vdocuments.site/reader033/viewer/2022042723/58d1da671a28ab66108b5667/html5/thumbnails/22.jpg)
WebSocket: Closing a connection
22
Client:
0x8
Server:
0x8
![Page 23: WebCamp 2016: Python. Вячеслав Каковский: Real-time мессенджер на Python. Особенности разработки back-end](https://reader033.vdocuments.site/reader033/viewer/2022042723/58d1da671a28ab66108b5667/html5/thumbnails/23.jpg)
WebSocket: Pros
● Supported by majority of browsers● Low latency● Small bandwidth● Easy to start development
23
![Page 24: WebCamp 2016: Python. Вячеслав Каковский: Real-time мессенджер на Python. Особенности разработки back-end](https://reader033.vdocuments.site/reader033/viewer/2022042723/58d1da671a28ab66108b5667/html5/thumbnails/24.jpg)
WebSocket: Cons
● Needs development of signaling protocol ● Timeouts/reconnections should be additionally
handled
24
![Page 25: WebCamp 2016: Python. Вячеслав Каковский: Real-time мессенджер на Python. Особенности разработки back-end](https://reader033.vdocuments.site/reader033/viewer/2022042723/58d1da671a28ab66108b5667/html5/thumbnails/25.jpg)
WebSocket and Python
● Servers:○ Autobahn - Twisted and asyncio
implementations○ aiohttp○ Tornado○ Flask-SocketIO○ Flask-Sockets
25
![Page 26: WebCamp 2016: Python. Вячеслав Каковский: Real-time мессенджер на Python. Особенности разработки back-end](https://reader033.vdocuments.site/reader033/viewer/2022042723/58d1da671a28ab66108b5667/html5/thumbnails/26.jpg)
WebSocket and Python
● Clients:○ Autobahn○ aiohttp○ Tornado-based example○ Vanilla websocket-client
● JS-client: SocketIO
26
![Page 27: WebCamp 2016: Python. Вячеслав Каковский: Real-time мессенджер на Python. Особенности разработки back-end](https://reader033.vdocuments.site/reader033/viewer/2022042723/58d1da671a28ab66108b5667/html5/thumbnails/27.jpg)
Life of messaging platform● Authentication● Access control checks● Delivery
○ Messages○ User's presence○ Push notifications
● History retrieval● History search
27
![Page 28: WebCamp 2016: Python. Вячеслав Каковский: Real-time мессенджер на Python. Особенности разработки back-end](https://reader033.vdocuments.site/reader033/viewer/2022042723/58d1da671a28ab66108b5667/html5/thumbnails/28.jpg)
Life of messaging platform● Parsing
○ Protocol○ Message content
● Dealing with file uploads○ Security checks○ Thumbnails distribution
● Multi-session support● Reconnection handling● Rate-limiting
28
![Page 29: WebCamp 2016: Python. Вячеслав Каковский: Real-time мессенджер на Python. Особенности разработки back-end](https://reader033.vdocuments.site/reader033/viewer/2022042723/58d1da671a28ab66108b5667/html5/thumbnails/29.jpg)
Life of messaging platform● Server keeps connections open for every client● High amount of long-lived concurrent connections● Multithreaded approach isn't efficient due to overhead● Requires usage of a select implementation on backend:
○ poll○ epoll○ kqueue
● Usage of asynchronous Python frameworks is preferred for high loaded solutions
29
![Page 30: WebCamp 2016: Python. Вячеслав Каковский: Real-time мессенджер на Python. Особенности разработки back-end](https://reader033.vdocuments.site/reader033/viewer/2022042723/58d1da671a28ab66108b5667/html5/thumbnails/30.jpg)
Life of messaging platform● Authentication
○ OAuth2○ Run encryption operations in a separate Python thread○ Cache users identities with Redis/Memcached
● Access-control checks○ Make the checks lightweight and cheap○ Raise an exception when operation isn't permitted
30
EAFP: Easier to ask for forgiveness than permission
![Page 31: WebCamp 2016: Python. Вячеслав Каковский: Real-time мессенджер на Python. Особенности разработки back-end](https://reader033.vdocuments.site/reader033/viewer/2022042723/58d1da671a28ab66108b5667/html5/thumbnails/31.jpg)
Delivery● Make message delivery fault-tolerant● Limit size of a message● Filter content of messages:
○ Users like to send chars that break all the things● Reduce presence traffic, it could be a bottleneck for large chats● Use asynchronous broker for delivery when a user is offline
(email or push)○ Celery○ RQ○ Amazon Simple Queue Service○ Huey
31
![Page 32: WebCamp 2016: Python. Вячеслав Каковский: Real-time мессенджер на Python. Особенности разработки back-end](https://reader033.vdocuments.site/reader033/viewer/2022042723/58d1da671a28ab66108b5667/html5/thumbnails/32.jpg)
Life of messaging platform● Push notifications
■ Vendors● Amazon SNS● APNS● Google Cloud Messaging● Firebase Cloud Messaging
■ Python tools● PyAPNs● Python-GCM● Pusher
● Be careful with device registration● Make delivery of pushes fault-tolerant
32
![Page 33: WebCamp 2016: Python. Вячеслав Каковский: Real-time мессенджер на Python. Особенности разработки back-end](https://reader033.vdocuments.site/reader033/viewer/2022042723/58d1da671a28ab66108b5667/html5/thumbnails/33.jpg)
History retrieval● Return last messages for every chat instantly
○ Use double writes■ In-memory queue only for last messages■ Persistent storage for all the things
● Majority of history retrievals is for the last days○ Let's optimize the case
● Index messages by date
33
![Page 34: WebCamp 2016: Python. Вячеслав Каковский: Real-time мессенджер на Python. Особенности разработки back-end](https://reader033.vdocuments.site/reader033/viewer/2022042723/58d1da671a28ab66108b5667/html5/thumbnails/34.jpg)
History search● ElasticSearch is the default solution for full-
text search● @a_soldatenko: What is the best full text
search engine for Python?● Add timing for search requests
34
![Page 35: WebCamp 2016: Python. Вячеслав Каковский: Real-time мессенджер на Python. Особенности разработки back-end](https://reader033.vdocuments.site/reader033/viewer/2022042723/58d1da671a28ab66108b5667/html5/thumbnails/35.jpg)
Parsing● Protocol
○ Avoid to use Pure Python parsers■ ujson■ lxml
○ Run benchmarks against your typical cases● Message content
○ Be careful with regular expressions■ re2■ pyre2
○ Alternative parsers in Python
35
![Page 36: WebCamp 2016: Python. Вячеслав Каковский: Real-time мессенджер на Python. Особенности разработки back-end](https://reader033.vdocuments.site/reader033/viewer/2022042723/58d1da671a28ab66108b5667/html5/thumbnails/36.jpg)
Dealing with file uploads● Security checks
○ File upload vulnerabilities○ Image upload
■ Decompression bomb■ Other vulnerabilities with Pillow
○ Amazon S3 as file storage■ boto■ aiobotocore■ botornado
● Thumbnails distribution○ Delegate that to S3○ Requested by a client even if not needed
36
![Page 37: WebCamp 2016: Python. Вячеслав Каковский: Real-time мессенджер на Python. Особенности разработки back-end](https://reader033.vdocuments.site/reader033/viewer/2022042723/58d1da671a28ab66108b5667/html5/thumbnails/37.jpg)
Life of messaging platform● Multi-session support
○ Set expiration time○ Be ready to handle up to 4x sessions per user simultaneously
■ Desktop■ Mobile■ Tablet■ Laptop
● Reconnection handling○ Spin a proxy layer between messaging server and clients
● Rate-limiting○ Limit amount of operations per user/group for heavy stuff○ Leaky bucket○ Throttling
37
![Page 38: WebCamp 2016: Python. Вячеслав Каковский: Real-time мессенджер на Python. Особенности разработки back-end](https://reader033.vdocuments.site/reader033/viewer/2022042723/58d1da671a28ab66108b5667/html5/thumbnails/38.jpg)
Lessons learned
● Bursty traffic○ Load testing is a must, but not always enough
■ Locust■ Yandex Tank
● Reconnect storm could be a big deal○ We should handle that on platform and client-side
● AWS issues make bad customers experience○ Put nodes in Multi-AZ
38
![Page 39: WebCamp 2016: Python. Вячеслав Каковский: Real-time мессенджер на Python. Особенности разработки back-end](https://reader033.vdocuments.site/reader033/viewer/2022042723/58d1da671a28ab66108b5667/html5/thumbnails/39.jpg)
Lessons learned
● Incidents prevention is cheaper than resolution○ Grab stats and metrics about your services and
storages■ Redis for per-chat stats■ StatsD■ Grafana
○ Be notified when something starts going wrong■ Elastalert■ Monit■ DataDog
39
![Page 40: WebCamp 2016: Python. Вячеслав Каковский: Real-time мессенджер на Python. Особенности разработки back-end](https://reader033.vdocuments.site/reader033/viewer/2022042723/58d1da671a28ab66108b5667/html5/thumbnails/40.jpg)
Lessons learned
● Don't stick with one language/stack○ Python is great, but for some cases Go, Ruby or
PHP are more suitable from product side○ Avoid business logic duplication in several repos,
spin a service and just call the endpoint● Releasing new features only for certain groups makes
product management easier○ LaunchDarkly
40
![Page 41: WebCamp 2016: Python. Вячеслав Каковский: Real-time мессенджер на Python. Особенности разработки back-end](https://reader033.vdocuments.site/reader033/viewer/2022042723/58d1da671a28ab66108b5667/html5/thumbnails/41.jpg)
Lessons learned
● Don’t F**k the Customer○ Provide unit/integration tests with every PR○ Have development environment same as prod○ Have staging environment same as prod○ Make deployments fast○ Rollback faster○ Have a fallback plan
41
![Page 42: WebCamp 2016: Python. Вячеслав Каковский: Real-time мессенджер на Python. Особенности разработки back-end](https://reader033.vdocuments.site/reader033/viewer/2022042723/58d1da671a28ab66108b5667/html5/thumbnails/42.jpg)
Summary
42
![Page 43: WebCamp 2016: Python. Вячеслав Каковский: Real-time мессенджер на Python. Особенности разработки back-end](https://reader033.vdocuments.site/reader033/viewer/2022042723/58d1da671a28ab66108b5667/html5/thumbnails/43.jpg)
Summary● Select a messaging protocol which aligns with your needs● WebSocket + JSON could be the thing for new projects● Usage of asynchronous frameworks is preferred● Execute blocking operations in a separate thread● Collect metrics for common services operations● Caching saves a lot of time● Use C or Cython-based solutions for CPU-bound tasks● Have fast release/deploy/rollback cycle● Python is great, but don't hesitate to pick other tools
43
![Page 44: WebCamp 2016: Python. Вячеслав Каковский: Real-time мессенджер на Python. Особенности разработки back-end](https://reader033.vdocuments.site/reader033/viewer/2022042723/58d1da671a28ab66108b5667/html5/thumbnails/44.jpg)
Further reading
● How HipChat Stores and Indexes Billions of Messages Using ElasticSearch● @kakovskyi: Maintaining a high load Python project for newcomers● HipChat: Important improvements to staging, presence & database storage● HipChat and the little connection that could● Elasticsearch at HipChat: 10x faster queries● Atlassian: How IT and SRE use ChatOps to run incident management● A Study of Internet Instant Messaging and Chat Protocols● What Is Async, How Does It Work, And When Should I Use It?● Leaky Bucket & Tocken Bucket - Traffic shaping● A guide to analyzing Python performance● Why Leading Companies Dark Launch - LaunchDarkly Blog● WebCamp 2016: Asyncio-stack for web development (soon)
44
![Page 45: WebCamp 2016: Python. Вячеслав Каковский: Real-time мессенджер на Python. Особенности разработки back-end](https://reader033.vdocuments.site/reader033/viewer/2022042723/58d1da671a28ab66108b5667/html5/thumbnails/45.jpg)
Questions?
45
Viacheslav [email protected]
@kakovskyi
Instant messenger with PythonBack-end development
![Page 46: WebCamp 2016: Python. Вячеслав Каковский: Real-time мессенджер на Python. Особенности разработки back-end](https://reader033.vdocuments.site/reader033/viewer/2022042723/58d1da671a28ab66108b5667/html5/thumbnails/46.jpg)
Instant messenger with PythonBack-end development
Viacheslav Kakovskyi WebCamp 2016