Learning Twisted (part 5) : Low-Level Networking

So far we have explored the basic pieces of twisted core - reactors , event handlers, scheduled & looping calls. It also provides support for cooperative multitasking or loop parallelism.
In this post, I will explore the framework that implements low level protocol and networking support in twisted. This one for me was really twisted. No wonder, it is named as such!

In retrospection, generality of twisted as a networking framework is greatly improved by the underlying complexity of it's implementation. How ever, tackling this complexity does pay off in the awesome power this tool provides in building networking applications.

I am a picture guy. I can't hold all these dense information without some sort of picture in place. So I guess, I will put the picture first and then delve in the details.

Sorry folks, no code in this post.

+----------------+       connectSSL       +-----------+
|PosixReactorBase|.......listentTCP ......|IReactorTCP|
+----------------+       connectTCP       +-----------+
                             |
                             |
    +----------+             |
    |IConnector|________connector obj <.......You pass a factory
    +----------+        obj -> connect        This is a protocol
     |   |                                    factory.
     |   |                                 ,-'
     |  stopDisconnecting               ,-'
     |  getDestination                ,'
     |   ..............> factory.dostart()
  connect..............> _makeTransport()
                               |
        +-------------+    connection obj
        |ISystemHandle|________|   |__calls factory.buildProtocol(addr)
        |             |                               |
        |ITCPTransport|.......>+----------+           |
        +-------------+        |ITransport|    +-------------+
                               +----------+    |IProtocol obj|
                                    |          +-------------+
                       .............:................
                       |                            |
                     Client..___                __Server
                                `'`---..----''''
                               These are initialized differently

Learning Twisted (part 4) : Cooperative multitasking

EventLoop dispatches the events to be handled by eventHandlers. If an eventHandler plays fraud and do not return immediately, event or reactor loop can not service other events. This means that all our event handlers need to return quickly for this model to work.
i.e functions that have substantial waiting period and are non-blocking. Typically, network or file I/O can be are non-blocking.

Twisted provides facility to interleave execution of various functions, where the period of waiting can be used to do something useful for other functions that are waiting to be serviced.

+--------------------------++--------------------------+
|task1 | wait period | comp||task2 | wait period | comp|
+--------------------------++--------------------------+
+--------------------------+
|task1 | wait period | comp|
+--------------------------+
       +--------------------------+
       |task2 | wait period | comp|
       +--------------------------+

Learning Twisted (part 3) : Scheduling with reactor

In previous installments, we found that twisted does provide methods to schedule a call at some time later, after the reactor loop starts running. It also provides method for registering a call that will be called repeatedly after a delay.

The facility is provided in twisted.internet.task.

from twisted.internet import reactor
from twisted.python import log
import sys, time
from twisted.internet.task import LoopingCall, deferLater

log.startLogging(sys.stdout) 

Learning Twisted (Part 2): Async IO

  +---------------+
  |ILoggingContext|......... logPrefix
  +---------------+
  +---------------+
  |IFileDescriptor|'''''''''''  fileno
  +-------+-------+'''''''''''  connectionLost
        | |  +---------------+.......doRead
        | +--|IReadDescriptor|.....__init__ : open the descriptor or
        |    +---------------+                  
       +----------------+..|.......__init__ : open a connection
       |IWriteDescriptor|..|.........doWrite
       +--------.-------+  |
                |          |
         addWriter        addReader
               +-------------+
               |IReactorFDSet|
               +-------------+

Well, this will become clear soon.

We know that the core of twisted is reactor loop that listens for events and dispatches those events to "event handlers" (Event based computing paradigm). We also learnt that how to generate custom events and register methods as "event handlers" for the custom events and some special events.

Various frameworks are available within twisted that provide facility to register events, add & remove the producer and consumer of these events. These framework usually follow certain well known paradigms. We will follow through some of those implementations in a short while as we continue exploring the default reactor "SelectReactor".

Non-blocking synchronisation shows better performance in certain application than blocking synchronization. Select based Non-blocking I/O allows us to implement this paradigm and is supported by SelectReactor.

Learning Twisted (part 1) : reactor basics

Twisted framework is an event based networking framework in Python. This description suffices for initial impression about this framework.

These and subsequent posts will serve as my notes for learning Twisted.

At the core of event based programming is reactor loop that run endlessly unless asked to stop.
While in the loop, reactor listens to events and dispatches these events to event handlers for processing of events. The example below is the simplest way of starting up the reactor.

print("To stop this example, press ctrl-c")
from twisted.internet import reactor
reactor.run()

reactor.run() should be the last line of code that gets executed immediately. After this, the control is handed to reactor loop, which listens for events endlessly. Some of the methods of this class also allows you to register for events and event handlers to process something real.

There are many reactors available, which needs to be installed. Default reactor is select based reactor - "SelectReactor"