Friday, September 17, 2010

Learning Twisted (part 6) : Understanding protocol factory

Since most of us will be reusing the transport implementations that are already provided by Twisted. Our focus will be to create protocols and protocol factory that ties up a transport with protocol instance.

from twisted.web import proxy
from twisted.internet import reactor
from twisted.python import log
import sys
log.startLogging(sys.stdout)

class myProtocolFactory():
    protocol = proxy.Proxy

reactor.connectTCP('localhost', 80, myProtocolFactory())
reactor.run()

This barebones throws a whole lot of trace backs which helps understand the code flow a little bit easily. You can keep supplying the functions and rerun to see all the required methods of protocol factory and how the code flows or is structured.



Sequence is:
  1. Reactor calls : tcp.connectTCP
    1. Create a connector instance - connection
    2. connection->connect()
      1. protocol factory -> doStart()
      2. connection->_makeTransport()
      3. factory -> startedConnecting(connection)
Since this is select based reactor connection.read and connection.write are set to connection.doConnect
  1. When connection is made , connection -> _connectDone() is called
    1. Now is the time to build a protocol instance, by calling connector-> buildProtocol()
      1. protocol factory -> buildProtocol (address), returns the protocol instance
    2. protocol instance -> makeConnection ()
Visually, the workflow looks something like this:
    +-------+
    |reactor|------------connect interface
    +-------+                  | 1
                           tcp.connectTCP
    +---------+                | 2               +----------+
    |connector|............. connection instance |connection|
    +---------+                | 3               +----------+
       | |...................  connect()
       | |.........            | 4                  +----------------+
       |          .          5 |_ doStart() ....... |protocol factory|
       |     _makeTransport() _|  6                 +----------------+
       |                       |_ startedConnecting(connection)....||
       |                                                            |
       |                                                            |
       |                  When connection is made                   |
       |                       | 21                                 |
       | .................  _connectDone()                          |
       |                       | 22                                 |
       | ......................|_buildProtocol()                    |
                               |    | 23                            |
                               |    |_buildProtocol(address)........|
                               |24      |
                               |       returns protocol   +--------+
                   makeConnection().....  instance ...... |Protocol|
                                                          +--------+

Sequence of Failure is also called upwards:
  1. Failure in select : self._disconnectSelectable(selectable, why, method=="doRead")
  2. Calls selectable.connectionLost(failure.Failure(why))
  3. Communicate to Transport : Connection.connectionLost(self, reason)
  4. Communicate upwards: protocol.connectionLost(reason)
  5. Calls factory -> doStop()
Visually:
failure in select -----> _disconnectSelectable(selectable,
                                 .   |              why,
           +-------+             .   |              method)
           |reactor|..............   |
           +-------+             .   |
                              selectable.connectionLost(failure(why))
                                     |
                                     |
                                     |
          +----------+               |
          |connection|....... connectionLost(reason)
          +----------+               |
                                     |
           +--------+                |
           |protocol|........ connectionLost(reason)
           +--------+                |
                                     |
             +-------+               |
             |factory|........... doStop()
             +-------+

So finally, here are the few methods that a protocol factory must support:

from twisted.web import proxy
from twisted.internet import reactor
from twisted.python import log
import sys
log.startLogging(sys.stdout)


class myProtocolFactory():
    protocol = proxy.Proxy
    
    def doStart(self):
        pass
        
    def startedConnecting(self, connectorInstance):
        print connectorInstance
        
    def buildProtocol(self, address):
        print address
        return self.protocol()
        
    def clientConnectionLost(self, connection, reason):
        print reason
        print connection
        
    def doStop(self):
        pass


reactor.connectTCP('localhost', 80, myProtocolFactory())
reactor.run()

2 comments:

  1. Hi,
    Your post are great. There is only one thing that I don't understand. I discovered that Factory instance is never deleted. I have a memory leak in my program because this. For example, if I call connectTCP() 100 of times, I got 100 instances of factory that are only deleted when the program shutdown. They are not deleted even if the connection are lost.

    Any hint about this?

    ReplyDelete
    Replies
    1. I suggest you call reactor.connectTCP as described in the example above:
      # this is wrong when you are calling for connectTCP
      # multiple times
      reactor.connectTCP('localhost', 80, myProtocolFactory())

      You should create factory only once and then reuse factory instance:

      f = myProtocolFactory()
      for i in range(0,100):
      reactor.connectTCP('localhost', 80, f) # <- same factory instance

      Delete