Using buildbot for continuos integration development

Continuos integration in it's simplicity embodies certain agile tenets like frequent integration of code and automated verification of the integrated code to provide continuous feedback to the team on development and reducing heart burns during large integrations. It also avoids silent creeping in of broken builds into the code repository. At the heart of this process is a tool that can be integrated with the workflow of code check-in to trigger automated testing of frequently checked in development artifacts.

This helps in providing the developer an immediate feedback and assurance that things are moving in a positive direction.

Buildbot is a "continuos integration" tool.

BuildBot can automate the compile/test cycle required by most software projects to validate code changes.

I had a chance to set it up some time back. What follows is a snippet of that experience on getting it up and running quickly.


Brief info on buildbot architecture

+---+  changes   +-----------+................ +-------+
|VCS|----------- |BuildMaster|---------------- |Browser|
+---+            +-----------+---------------  +-------+
                  |       |     build status| +-----+
                  |       |                 | |Email|
                  | commands/results        | +-----+
                  |       |                 --+-------------------+
                  |       |                   |Other Status Client|
       +----------+   +----------+            +-------------------+
       |BuildSlave|   |BuildSlave|
       +----------+   +----------+

Buildbot has a master - slave architecture with a single buildmaster to which notification can be sent through integration hooks with various version control systems, this results in the buildmaster issuing various commands to available slave build systems that can lie on the same machine or in the network. The instructions are carried out by the build slaves and the result is reported back to the build master. The results can be notified via mail , irc etc depending upon the configuration.

Some key aspects about the internal architecture of buildbot include

• Change source - Change Sources, which create a Change object each time something is modified in the VC repository
• Schedulers - which decide when builds should be performed. They collect Changes into BuildRequests, which are then queued for delivery to Builders until a buildslave is available.
• Builders - Builders, which control exactly how each build is performed Each build is run on a  single buildslave.
• Status plugins - delivers information about the build results through protocol like HTTP, mail and IRC.

All these information is provided to buildbot through a configuration file that is loaded up during the starting of a buildbot server.

Setting up a buildbot involves providing the the details in this configuration file called - master.cfg

Setting up buildbot
Before we discuss the configuration details, I have listed some of the basic requirements for setting up of buildbot.

Key dependencies are twisted and zope.interface, install them prior to installing buildbot

• Create a directory structure for setting up buildbot
  
  
  

  +---------+
  |Setup    |
  |Directory|
  +---------+
       |
       |        Build Master directory
       |        +-------+      +-----------+(don't
       |________|masters| .....|buildmaster| create
       |        +-------+      +-----------+directly)
       |
       |        Directory where slave
       |        +------+  information resides
       '........|slaves|.......
                +------+      |    +----------+
                              |____|buildslave| single
                                   +----------+ slave
                                        |       information
                                        |
  

  
  
  
• Create buildmaster

cd masters
buildbot create-master buildmaster

• Copy the default configuration file located in buildbot sources to build master directory
• Create slave

mkdir -p slaves/buildslave

• Set up a slave, remember the information you have provided. The information has to be supplied to buildmaster via master.cfg

buildbot create-slave <slave-dir> <ip-address>:<port> <slave-name><password>

Some key buildbot commands

Starting master and slaves

cd pathto/masters
buildbot start buildmaster

Starting slaves

cd pathto/slaves
buildbot start buildslave

Stopping master

cd pathto/masters
buildbot stop buildmaster

Stopping slave

cd pathto/slaves
buildbot stop buildslave

Restarting master

cd pathto/masters
buildbot restart buildmaster

Restarting slave

cd pathto/slaves
buildbot restart buildslave

Updating config

cd pathto/masters
buildbot reconfig buildmaster

Cleaning up master and slaves for new setup

rm -rf pathto/masters
rm -rf pathto/slaves