python twisted
DESCRIPTION
Introduction to twisted programming (Pycon 2010)TRANSCRIPT
![Page 1: Python twisted](https://reader034.vdocuments.site/reader034/viewer/2022042503/55844890d8b42afc4e8b4984/html5/thumbnails/1.jpg)
Python Twisted
Mahendra M@mahendra
http://creativecommons.org/licenses/by-sa/3.0/
![Page 2: Python twisted](https://reader034.vdocuments.site/reader034/viewer/2022042503/55844890d8b42afc4e8b4984/html5/thumbnails/2.jpg)
Methods of concurrency
Workers Threads and processes
Event driven
Let us examine this with the case of a web server
![Page 3: Python twisted](https://reader034.vdocuments.site/reader034/viewer/2022042503/55844890d8b42afc4e8b4984/html5/thumbnails/3.jpg)
Worker model
dispatch()request worker_1()
worker_n()
read(fp)
db_rd()
db_wr()
sock_wr()
![Page 4: Python twisted](https://reader034.vdocuments.site/reader034/viewer/2022042503/55844890d8b42afc4e8b4984/html5/thumbnails/4.jpg)
Worker model
dispatch()request worker_1()
worker_n()
read(fp)
db_rd()
db_wr()
sock_wr()
BLOCKING!!
![Page 5: Python twisted](https://reader034.vdocuments.site/reader034/viewer/2022042503/55844890d8b42afc4e8b4984/html5/thumbnails/5.jpg)
How does it scale ?
A worker gets CPU when it is not blocking When it makes a blocking call, it sleeps till the sys
call is requested At this time another worker gets CPU Worker might block before it completes it's
allocated timeslice. This model has worked great and still works great Eg: Apache, Squid
![Page 6: Python twisted](https://reader034.vdocuments.site/reader034/viewer/2022042503/55844890d8b42afc4e8b4984/html5/thumbnails/6.jpg)
Overheads
Worker management Thread creation Process creation and management Synchronization Scheduling (though this is left to OS)
More system calls for blocking operations
![Page 7: Python twisted](https://reader034.vdocuments.site/reader034/viewer/2022042503/55844890d8b42afc4e8b4984/html5/thumbnails/7.jpg)
Event Driven Code
Non blocking code blocks Code execution on events
Data on sockets, timer, new connection
Execution triggered from an event loop Full use of CPU timeslice
![Page 8: Python twisted](https://reader034.vdocuments.site/reader034/viewer/2022042503/55844890d8b42afc4e8b4984/html5/thumbnails/8.jpg)
Visually ...
block_on_events( .. )
event_1
event_2
event_n
hdler_1()
hdler_2()
hdler_n()
Events are posted
Non blocking functions
![Page 9: Python twisted](https://reader034.vdocuments.site/reader034/viewer/2022042503/55844890d8b42afc4e8b4984/html5/thumbnails/9.jpg)
Visually ...
block_on_events()
event_1
event_2
event_n
hdler_1()
hdler_2()
hdler_n()
Events are posted
Non blocking functions
ev()
![Page 10: Python twisted](https://reader034.vdocuments.site/reader034/viewer/2022042503/55844890d8b42afc4e8b4984/html5/thumbnails/10.jpg)
Web Server
event_loop()
request
opened
sql_read
open(fp)
wri_sql()
Non blocking functions
reg()
parse()
read_sql() reg()
reg()
sql_writ
responded
sock_wr() reg()
close()
![Page 11: Python twisted](https://reader034.vdocuments.site/reader034/viewer/2022042503/55844890d8b42afc4e8b4984/html5/thumbnails/11.jpg)
Event Driven Designs
Nginx, Tornado Varnish Memcached OS support
epoll – Linux kqueue – BSDs
![Page 12: Python twisted](https://reader034.vdocuments.site/reader034/viewer/2022042503/55844890d8b42afc4e8b4984/html5/thumbnails/12.jpg)
Event Driven Libraries
Libevent Pythontwisted Java NIO
Apache MINA, Tomcat (not default) Jetty
QT
![Page 13: Python twisted](https://reader034.vdocuments.site/reader034/viewer/2022042503/55844890d8b42afc4e8b4984/html5/thumbnails/13.jpg)
Drawbacks
Tougher to code, design and maintain Workers required to make use of multiple CPUs All callbacks must be nonblocking
Tough to get nonblocking libraries for all modules
No isolation A block in any event loop can freeze everything
![Page 14: Python twisted](https://reader034.vdocuments.site/reader034/viewer/2022042503/55844890d8b42afc4e8b4984/html5/thumbnails/14.jpg)
Python Twisted
Event driven programming framework MIT licensed 8 years old and stable Support for large number of protocols
Client and server support HTTP – SOAP, REST, CouchDB, XMLRPC, .... Sockets, TCP/IP, Multicast, TLS, SSH, IMAP … SMTP, NNTP, FTP, Memcached, AMQP, XMPP, ...
![Page 15: Python twisted](https://reader034.vdocuments.site/reader034/viewer/2022042503/55844890d8b42afc4e8b4984/html5/thumbnails/15.jpg)
Deferred
The central concept of twisted A callback returns a deferred to indicate that the job
is not done yet. The caller can add callbacks to a deferred. The callbacks are invoked then the job is done
Eh ?
![Page 16: Python twisted](https://reader034.vdocuments.site/reader034/viewer/2022042503/55844890d8b42afc4e8b4984/html5/thumbnails/16.jpg)
Deferred examplefrom twisted.internet import reactor
# Define a success callbackdef cb( arg1, arg2 ): print ”Timeout after %d %s” % ( arg1, arg2 )
# Define an error callbackdef eb( error ): Print ”error %s” % error
# Invoke a non blocking taskdeferred = someTimeout( 4 )
# Register the callbacks on the returned deferreddeferred.addCallback( cb, 4, 'twisted is great' )deferred.addErrback( eb )
# Run the event loopreactor.run()
![Page 17: Python twisted](https://reader034.vdocuments.site/reader034/viewer/2022042503/55844890d8b42afc4e8b4984/html5/thumbnails/17.jpg)
Twisted Server
from twisted.internet.protocol import Protocol, Factoryfrom twisted.internet import reactor
class QOTD(Protocol): def connectionMade(self): self.transport.write("Welcome\r\n") self.transport.loseConnection()
# Next lines are magic:factory = Factory()factory.protocol = QOTD
# 8007 is the port you want to run under.reactor.listenTCP(8007, factory)reactor.run()
![Page 18: Python twisted](https://reader034.vdocuments.site/reader034/viewer/2022042503/55844890d8b42afc4e8b4984/html5/thumbnails/18.jpg)
Deferred chaining
A callback can register and return another deferred This callback can return another deferred In short we have a deferred chain … Web server example:
![Page 19: Python twisted](https://reader034.vdocuments.site/reader034/viewer/2022042503/55844890d8b42afc4e8b4984/html5/thumbnails/19.jpg)
Deferred Chaining
![Page 20: Python twisted](https://reader034.vdocuments.site/reader034/viewer/2022042503/55844890d8b42afc4e8b4984/html5/thumbnails/20.jpg)
Advanced twisted
Twisted application support Mixing and matching twisted applications
Command line 'twistd' runner Predefined twisted apps Web, telnet, xmpp, dns, conch (ssh), mail, …
Plugin architecture Live debugging ADBAPI – for RDBMS
![Page 21: Python twisted](https://reader034.vdocuments.site/reader034/viewer/2022042503/55844890d8b42afc4e8b4984/html5/thumbnails/21.jpg)
Advanced twisted
Perspective Broker RPC and object sharing Spreading out servers
Cred – Authentication framework Deferring to threads External loops (GTK, QT) Streaming support MVC framework (Complex. Very complex)
![Page 22: Python twisted](https://reader034.vdocuments.site/reader034/viewer/2022042503/55844890d8b42afc4e8b4984/html5/thumbnails/22.jpg)
Drawbacks
Single threaded by design Makes use of only one core/CPU Need external modules for using multiple CPU
Run multiple instances of twisted on a box num_instances = num_cpus Use nginx (HTTP), HAProxy for load balancing
![Page 23: Python twisted](https://reader034.vdocuments.site/reader034/viewer/2022042503/55844890d8b42afc4e8b4984/html5/thumbnails/23.jpg)
Demos
![Page 24: Python twisted](https://reader034.vdocuments.site/reader034/viewer/2022042503/55844890d8b42afc4e8b4984/html5/thumbnails/24.jpg)
Links
http://twistedmatrix.com/trac/