My experiences with Fabric based deployment automation

Many good tools are available for configuration management and application deployment.
Puppet, Chef have attained cult status among the dev-ops team. There are good tools available in Python too. Salt may soon become a viable alternative and looks definitely promising to me. Push-Pull is commonly used to explain various types of tools in the eco-system.

Fabric is an excellent tool that allows you to weave operation locally and remotely on cluster of machines, allowing you to deploy applications, start/stop services and perform other operations on a cluster of machines. There are few good tutorials available to help familiarize with Fabric. If you haven't read it already, you should.

1. An example on deploying django using Fabric
2. A presentation on using Fabric
3. A video on Fabric usage

I have used Fabric to automate deployment of Hadoop / Hive application, Nagios deployment on cluster of machines on EC2, private cloud based on Cloudstack and commodity machines.

The code grew from nifty little commands / functions like setting up the "Fully qualified domain hostname" (FQDN), creating users and groups on Linux, installing yum packages to a complete system of commands that installs and brings up Kerberos enabled secure hadoop cluster using Cloudera hadoop packages.

Code soon became unwieldily.

There are a few practices that helps contain the level of complexity that grows when using Fabric enthusiastically.

First and most important is the fact that Fabric exposes functions as fab commands through a file called fabfile.py. All functions imported here can be executed as commands. Over a period of time, adding and removing functions from fabefile.py becomes a tenuous task.

The solution is to expose class methods as fabric commands. The excellent details are provided in the following post by Eliot.

I preferred to make a small twist to allow me to add test functions which could be removed through a simple flag later on.

def add_classmethods_as_fabric_functions(instance, 
                                         module_name, 
                                         ignore_test_func=True):
    '''Utility to take the methods of the instance of a class, instance,
    and add them as fab command '''
    module_obj = sys.modules[module_name]
    for method in inspect.getmembers(instance, predicate=inspect.ismethod):
        method_name, method_obj = method
        if not method_name.startswith('_'):
            if not (ignore_test_func and method_name.startswith('Test_')):
                # get the bound method
                func = getattr(instance, method_name)
                # add the function to the current module
                setattr(module_obj, method_name, func)

And in fabfile.py, you can add a class methods as fabric commands:

from fab_util import fab_util.add_classmethods_as_fabric_functions
from fab_node import _Bare_Node_Setup
instance = _Bare_Node_Setup()
fab_util.add_classmethods_as_fabric_functions(instance, __name__)   

This will insure that methods of instance of Class _Bare_Node_Setup are made available as fab commands.

Second issue is that important attributes of how a command is executed, gets mixed up.

1. On which remote node / host the commands are executed
2. As  what user credentials are the command executed
3. The actual functionality or logic of the commands

I learnt over many iterations that it is a good idea to separate these aspects in different functions.

Credentials are easily separated through a following class which can be inherited and used in the classes that compose the fab common logic.

class _Credential_Mixin(object):
    """Credential with which to execute commands"""
    
    def _run_as_user(self, username, userpassword, function, *args, **kwargs):
        with settings(user=username, password=userpassword):
            return function(*args, **kwargs)    
    
    def _run_as_root(self, cfg,  function, *args, **kwargs):
        root_password = .....
        return self._run_as_user('root', 
                                 root_password, 
                                 function, *args, **kwargs)

Node operations can also be separated by providing a separate Class

class _Node_Mixin(object):
    """Support tp run functions on a particular node"""
    
    def _run_on_node(self, node, function, *args, **kwargs):
        with settings(host_string=node):
            function(*args, **kwargs)

It is very easy to compose fabric recipes using this approach.

class _Nagios_Node_Mixin(_Node_Mixin):
    """Provides support for Ganglia server"""
        
    def _run_on_nagiosserver(self, cfg, function, *args, **kwargs):
        node = cfg.nagios.nagios_server
        self._run_on_node(node, function, *args, **kwargs)

   
class _Nagios_Node_Setup( _Nagios_Bootstrap_Mixin,
                          _Nagios_Node_Mixin,
                                        ):
   def setup_nagios_server(self, project):
        """Configure and run nagios server"""
        if project: cfg = load_property_file(project)
        if not cfg: 
            print "No such project properties could be found"
            return 
        
        self._run_on_nagiosserver(cfg, self._upload_nagios_conf_files, 
                                  cfg, project)
        self._run_on_nagiosserver(cfg, self._create_nagios_data_dir, 
                                  cfg)
        self._run_on_nagiosserver(cfg, self._service_nagios, 
                                  cfg, action='start')

You can now switch from using root to a normal user, based on some configuration.

        if cfg.access.minimize_root_usage == 'yes':
            self._run_as_hdfs(cfg, _func, cfg, False) 
        else:
            self._run_as_root(cfg, _func, cfg, True) 

I hope this helps. I will leave other techniques for later posts.

Do let me know, if you have more ideas on how to tame complexity of Fabric commands.