Learning Twisted (part 5) : Low-Level Networking

So far we have explored the basic pieces of twisted core - reactors , event handlers, scheduled & looping calls. It also provides support for cooperative multitasking or loop parallelism.
In this post, I will explore the framework that implements low level protocol and networking support in twisted. This one for me was really twisted. No wonder, it is named as such!

In retrospection, generality of twisted as a networking framework is greatly improved by the underlying complexity of it's implementation. How ever, tackling this complexity does pay off in the awesome power this tool provides in building networking applications.

I am a picture guy. I can't hold all these dense information without some sort of picture in place. So I guess, I will put the picture first and then delve in the details.

Sorry folks, no code in this post.

+----------------+       connectSSL       +-----------+
|PosixReactorBase|.......listentTCP ......|IReactorTCP|
+----------------+       connectTCP       +-----------+
                             |
                             |
    +----------+             |
    |IConnector|________connector obj <.......You pass a factory
    +----------+        obj -> connect        This is a protocol
     |   |                                    factory.
     |   |                                 ,-'
     |  stopDisconnecting               ,-'
     |  getDestination                ,'
     |   ..............> factory.dostart()
  connect..............> _makeTransport()
                               |
        +-------------+    connection obj
        |ISystemHandle|________|   |__calls factory.buildProtocol(addr)
        |             |                               |
        |ITCPTransport|.......>+----------+           |
        +-------------+        |ITransport|    +-------------+
                               +----------+    |IProtocol obj|
                                    |          +-------------+
                       .............:................
                       |                            |
                     Client..___                __Server
                                `'`---..----''''
                               These are initialized differently

Some key pieces of networking jigsaw puzzle includes transport layer that shuffles bytes around and once the bytes have been shuffled around, you need a protocol to handle it.
Consider, what should be happening in any networking framework. It allows you to setup transport to send and receive bytes, tie it up with a protocol that knows how handle data once it is received or send data on behalf of application layer and manage the connections (Transport Layer). The implementation of this tie up between a transport and protocol is usually implemented through the factory pattern and in twisted parlance "ProtocolFactory".

This is how, it is implemented in Twisted. Simple! You might ask, so how does all this tie up with the reactor or event loop?

This is done through a connector, that creates a connection. See IConnector.

+----------+
|Transport |      +--------+               +---------------+
|or        |......|Protocol|<-----------|ProtocolFactory|
|connection|  |   +--------+  provided     +-------=-------+
+----------+  binding                    ,.--''
      |                          __..--''
 created by               __.,-''provided to create the protocol
      |            _,.--''
 +---------+<--.-''         +-----------------+
 |Connector|<---------------|reactor interface|
 +---------+    used by        +-----------------+

Twisted reactor has some basic interfaces that utilizes these connectors to create connections. Remember that connection is essentially a transport instance (See ITransport)
Various transports supported by twisted :

• ITCPTransport, ITLSTransport,  ISSLTransport, IProcessTransport

In Twisted, if you want some thing, look into the code or documentation, it may already be there for you to use!

Entry points in twisted reactor which utilizes connector to create connections (transport) are provided by reactor interface.

• IReactorTCP, IReactorSSL, IReactorUNIX, IReactorUNIXDatagram, IReactorUDP, IReactorMulticast 

Most of these provide two types of interface - connect and listen . For ex : An implementation of IReactorTCP, needs to implement methods - connectTCP and listenTCP.
For some connectionless oriented transports, only listen interface is required like UDP and Multicast.

Since all of these follow the same pattern, I will investigate the TCP connection is little more detail.
IReactorTCP is implemented in posixbase.py for select based reactor, which is usually available as default reactor on almost all platforms. Approach to create server and client are slightly different and same as you would do socket programming using native API provided by your OS. We will investigate connectTCP that is used to create client connections.

so here is the sequence and commentary, which allows us to understand the above diagram:

1) connectTCP creates an "connection instance" by creating an instance of connector: tcp.Connector and then call connect function on this connection instance. We need to provide host, port, factory to this connector.
2) This connector provides the specific functionality to create a connection via _makeTransport (a transport), which is called by the "connect" function of base class of connector. ("connect" is called by reactor interface)
The sequence is to call : factory.dostart(), _makeTransport , factory.startedConnecting

connectTcp  -------cobj----[]tcpConnector--provides-_makeTransport
     (recator interface) (1)          |                       +
               |                      |                       |
               |                   (base class)               |
               |                      BaseConnector           |
             cobj (2)                  |                      | (4)
               |                     (methods)                |
               |-------------------> connect      (3)         |
                                       |->protocol.do_start   |
                                       |-----------------------
                                       |
                                       |->protocol.startedConnecting
                                                   (5)