Thursday, December 1, 2011

My experiences with Fabric based deployment automation

Many good tools are available for configuration management and application deployment.
Puppet, Chef have attained cult status among the dev-ops team. There are good tools available in Python too. Salt may soon become a viable alternative and looks definitely promising to me. Push-Pull is commonly used to explain various types of tools in the eco-system.

Fabric is an excellent tool that allows you to weave operation locally and remotely on cluster of machines, allowing you to deploy applications, start/stop services and perform other operations on a cluster of machines. There are few good tutorials available to help familiarize with Fabric. If you haven't read it already, you should.
  1. An example on deploying django using Fabric
  2. A presentation on using Fabric
  3. A video on Fabric usage
I have used Fabric to automate deployment of Hadoop / Hive application, Nagios deployment on cluster of machines on EC2, private cloud based on Cloudstack and commodity machines.

The code grew from nifty little commands / functions like setting up the "Fully qualified domain hostname" (FQDN), creating users and groups on Linux, installing yum packages to a complete system of commands that installs and brings up Kerberos enabled secure hadoop cluster using Cloudera hadoop packages.

Code soon became unwieldily.

There are a few practices that helps contain the level of complexity that grows when using Fabric enthusiastically.

Installing funkload on Mac

Funkload is  a useful tool for understanding the characteristics of the application server under stress and load conditions.

Installation of Funkload is very straightforward using virtualenv and macports on Mac.
If you aren't using it already, you should think of checking it out.

Create a isolated environment for installing Funkload.

virtualenv --no-site-packages loadtest
source loadtest/bin/activate
pip install yolk