twisted
TRANSCRIPT
2
Based on
Dave Peticolas: Twisted Introductionhttp://krondo.com/?page_id=1327
Orestis Markou: Asynchronous programming with Twisted
http://ep2011.europython.eu/conference/talks/asynchronous-programming-with-twisted
https://github.com/orestis/twisted-tutorial
Twisted documentation
http://twistedmatrix.com/documents
3
Outline
● What is Twisted?● Why Twisted?
● Async vs. Sync● Blocking vs. Non Blocking
● Parts:● Reactor● Factory● Protocol● Transfer
4
What is Twisted
Twisted is a networking engine written in Python, supporting numerous protocols.
It contains a web server, numerous chat clients, chat servers, mail servers, and more.
http://twistedmatrix.com/trac/wiki/TwistedProjects
5
Why use Twisted
● Python... 2 :(● Asynchronous and event-based● Full-featured
● Mail, web, news, chat, DNS, SSH, Telnet, RPC, database access, and more
● Flexible● Open source
http://www.eecho.info/Echo/python/why-use-twisted/
6
Async? Eventbased?
● Single thread● Synchronous● Asynchronous
● Multi-thread
7
Synchronous model
● 3 tasks● One after another● Simple flow
8
Threaded model
● Parallel execution● Complex flow coordination
● IPC, critical section, race condition, synchronisation
9
Asynchronous model
● Interleaved tasks● Sequence of small steps● More complex than sync
No parallelism!
Q: Same execution time as in sync model?
10
Typical network server blocksimport sockets = socket.socket(socket.AF_INET, socket.SOCK_STREAM)s.bind((HOST, PORT))s.listen(1)while 1: conn, addr = s.accept() while 1: data = conn.recv(1024) if not data: break conn.send(data) conn.close()
11
Blocking
● Some function calls blocks and sync programs waits
● I/O● CPU much faster then:
● Disk● Network
12
When and why async?
● Many tasks● Interactive● Independent tasks
● Non-blocking● Responsive● Less interprocess
communication means less waiting for processes
Network services
13
How?
● Low level I/O - Platform dependent● Socket – setblocking(0)● Async Disk I/O
– Python 3 - http://www.python.org/dev/peps/pep-3116/– Twisted
http://twistedmatrix.com/documents/8.1.0/api/twisted.internet.fdesc.html#setNonBlocking
● Reactor pattern and async libs - Twisted
14
Twisted basic parts
● Reactor● Endpoint● Factory● Protocol● Transfer● Deferred
15
16
Reactor
● Wait for I/O● Handle Event
● Q: What if event handler blocks?
17
Blocking event handlerstops application
18
● potentially blocking operations● reading or writing from a non-socket file descriptor
(pipe)● waiting for a subprocess to finish
● Switch from blocking to non-blocking?● Many standard Python functions are blocking only● Example: os.system – always block● Use Twisted API :)
19
First server
● The reactor isn’t created explicitly, just imported.
● reactor.run(), reactor.stop()
● The reactor loop runs in the same thread it was started in.
● Once the loop starts up, it just keeps going.
● If it doesn’t have anything to do, the reactor loop does not consume CPU.
● Default reactor is twisted.internet.selectreactor
from twisted.internet import reactorreactor.run()
20
ReactorStatus T
CP
SSL
UDP
Threading Processes Scheduling
Platforms
select() Stable Y Y Y Y Y Y Unix, Win32
poll Stable Y Y Y Y Y Y Unix
WaitForMultipleObjects
Experimental Y Y Y Y Y Y Win32
Input/Output Completion Port
Experimental Y Y N N N Y Win32
CoreFoundation Unmaintained Y Y Y Y Y Y Mac OS X
epoll Stable Y Y Y Y Y Y Linux 2.6
GTK+ Stable Y Y Y Y Y Y Unix, Win32
wx Experimental Y Y Y Y Y Y Unix, Win32
kqueue Experimental Y Y Y Y Y Y FreeBSD
http://twistedmatrix.com/documents/current/core/howto/choosing-reactor.html
21
from twisted.internet import pollreactorpollreactor.install()
from twisted.internet import reactorreactor.run()
Change reactor
22
23
Factory
● persistent configuration common to all connections
● instantiates “protocol”● access persistence: protocol_instance.factory● factory does not listen to connections● same service on multiple ports or network
addresses
24
25
Protocol
● Per connection instance● Protocol handling● DataReceived(): Called whenever data is received.● ConnectionLost(): Called when the connection is
shut down.● MakeConnection(): Make a connection to a transport
and a server.● ConnectionMade(): Called when a connection is
made.
26
27
Transport
● Represents physical connection● write(): send data over connection● GetPeer()● GetHost()● loseConnection()
28
29
Endpoint
● Listener or connector abstraction
cep = clientFromString(reactor "tcp:host=www.example.com:port=80")
cep.connect(factory)
cep = clientFromString(reactor, "ssl:host=web.example.com:port=443:"
"privateKey=foo.pem:certKey=foo.pem")cep.connect(factory)
30
sep = serverFromString(reactor, "tcp:80:interface=127.0.0.1")
sep.listen(factory)
sep = serverFromString(reactor, "unix:/var/run/finger:mode=660")
sep.listen(factory)
31
Summary
● Reactor is most important● reactor.run()● Appropriate endpoint.● Protocol factory to create protocol instances● Protocol instances only deal with a single connection● Shared state in factory● Data arrive in chunks● Implementing protocols is hard. Reuse as much code as
possible
32
What's next?
● Server example● Deferred - promise● Database handling – Async?● Error handling – reverse error flow● Trial
33
First server
● Proxy at localhost:8000● Input: URL● Output: Fetched data
34
import timeimport urllib2
from twisted.protocols import basic
class ProxyProtocol(basic.LineReceiver): def lineReceived(self, url): if not url.startswith('http://'): return start = time.time() print 'fetching', url connection = urllib2.urlopen(url) data = connection.read() print 'fetched', url, self.transport.write(data) self.transport.loseConnection() print 'in', time.time() start
First server
35
from twisted.internet import protocolclass ProxyFactory(protocol.ServerFactory): protocol = ProxyProtocol
from twisted.internet import reactor, endpoints
endpoint = endpoints.TCP4ServerEndpoint(reactor, 8000)factory = ProxyFactory()endpoint.listen(factory)
reactor.run()
36
Run
Site Server Client===== ====== ======A 0.771 3.817B 0.567 2.026C 1.457 2.026D 1.019 3.815 Total: 3.813 3.817
Why same download times?
c = urllib2.urlopen(url)
data=c.read()
37
We need async url data fetcher
● twisted.web.client.getPage● http://launchpad.net/tx
38
import timefrom twisted.web import clientfrom twisted.protocols import basic
class ProxyProtocol(basic.LineReceiver):
def lineReceived(self, url): if not url.startswith('http://'): return start = time.time() print 'fetching', url deferredData = client.getPage(url)
def urlFetched(data): self.transport.write(data) self.transport.loseConnection() print 'fetched', url, print 'in', time.time() start
deferredData.addCallback(urlFetched)
39
from twisted.internet import protocolclass ProxyFactory(protocol.ServerFactory): protocol = ProxyProtocol
from twisted.internet import reactor, endpoints
endpoint = endpoints.TCP4ServerEndpoint(reactor, 8000)factory = ProxyFactory()endpoint.listen(factory)
reactor.run()
40
Site Server Client===== ====== ======A 0.850 0.853B 0.486 0.488C 1.582 1.584D 0.999 1.000 Total: 3.918 1.585
Better :)