Building and packaging a python application for distribution

I find it messy to build a python application package that is easy to distribute . Though, python tools have come a long way.

"pip/wheel" helps in installing, managing and distributing individual packages. "virtualenv" provides an approachable way to create isolated environment and avoid the pollution of the main distribution. "zc.buildout" lets you reproduce / assemble and deploy a python application through configuration. Together, they provide a powerful framework for building and distributing python applications.

However, It is not as simple as build once and distribute everywhere model of executable / jar package distribution. In all likelihood, you will be creating an isolated virtual environment, installing the dependencies and the application into it.

Build and packaging tools

Two excellent tools from twitter namely "pex", "pants" and packaging tool called "fpm" helps in providing a system that allows me to build once and distribute everywhere. Along with the tools mentioned above, they make a complete tool chain for building application packages.
I use a simple shell script to build and create an installation package for distribution. Rest of blog post is about how I have used these tools to create an easy to deploy and distribe application package.

PEX

A pex file is a self-bootstrapping python environment. You can consider it as an isolated virtual environment packed in a file.
This makes for an environment super easy to distribute. It can be built once and used everywhere with a few caveats, mostly due to platform dependencies. Most likely, you should be able to build platform specific "pex" distribution file.

You can use it to create a file that acts like a python interpreter with all the dependencies added already. You can use pex to create a sphinx documentation tool executable - "sphinx.pex". Executing the tool, you will find that it acts like "sphinx-build" script. You can read more about it in pex documentation.

pex -r sphinx  -e sphinx:main -o sphinx.pex

./sphinx.pex -h
Sphinx v1.2.3
Usage: ./sphinx.pex [options] sourcedir outdir [filenames...]
......

Pants

Pants is a build tool similar to makefile, gradle, ant. It can handle Java, Scala and Python code base. Pants builds "targets" which could be a library, binary based on a hierarchal tree of "BUILD" files. Pants can be used to build a part of your code base by targeting a branch of "BUILD" files tree. Another useful aspect about building python code base is that it builds a python binary target as "pex" file. There is more to pants than described here. Go through pants documentation.

FPM

FPM eases creation of distribution packages like "rpm" and "deb" formats for different target platforms. You can easily package a tarball into an rpm.

fpm -s tar -t rpm -n "${package_name}"
    -v version
    --license license
    --vendor vendor
    --maintainer "maintainer"
    --url url
    --description "description"
    --iteration iteration
    path_to_package.tar.gz

Now you have a rpm package called "package_name-version-iteration.arch.rpm" which can be distributed and deployed.

Using these tools

I use a build script to build, test an application and package it into an installable format. I use virtualenv to create a isolated environment:

virtualenv build_dir/virtual_env

I install the pants build system in the virtual environment that I just created using pip.

source build_dir/virtual_env/bin/activate
pip install "http://effbot.org/media/downloads/elementtree-1.2.7-20070827-preview.zip"
pip install coverage
pip install twitter.pants

For mac osx, use: Mac: export ARCHFLAGS="-Wno-error=unused-command-line-argument-hard-error-in-future"; pip install pkg_toinstall

Initializing the pants build tool requires access to PyPI and should be enabled in pants.ini. How ever, I turn it off while building the application. I depend on local repository of wheel packages. To make this kind of switch, it is better to generate the pants.ini file from a template. I also substitute the paths to a few variables to isolate the pants working directory per project.

PANTS_DIR=build_dir/pants
PANTS_CACHEDIR=$PANTS_DIR/.pants.d
PANTS_WORKDIR=$PANTS_DIR/.pants.d
PANTS_SUPPORTDIR=$PANTS_DIR/build-support
PANTS_PYTHONS=$PANTS_DIR/.python
PANTS_WHEELHOUSE=$PANTS_DIR/wheelpackage
PANTS_WHEELHOUSE_CACHE=$PANTS_DIR/wheelpackage/cache

and use sed to transform the template into usable "pants.ini" file.

cp pants_template_file.ini pants.ini
sed -i.bak "s|PANTS_CACHEDIR|${PANTS_CACHEDIR}|g" pants.ini
.......

With pypi repository enabled in pants.ini.

indices:  ['pypi.python.org']
allow_pypi: True 

we can initialize pants build system.

pants py

To prepare for building application, I turn off pypi repository and enable local wheel repository.

[python-repos]
repos: [
  'PANTS_WHEELHOUSE/',
  'https://raw.github.com/twitter/commons/binaries/pants/third_party/python/dist/index.html',
  'https://raw.github.com/twitter/commons/binaries/pants/third_party/python/index.html']
#indices:  ['pypi.python.org']
allow_pypi: False 

Now we are read to build our application. This is defined as a binary target in "BUILD" file. The binary target can be dependent on external and internal packages.

python_binary(
  name = 'demoapplication',
  source = 'src/demoapplication.py',
  dependencies = [
    pants(":external_library"),
  ]
)

python_binary(name = "supervisord",
  entry_point = "supervisor.supervisord:main",
  dependencies = [
    pants(":supervisord_library"),
  ]
)

python_library(
  name = "supervisord_library",
  dependencies = [
    python_requirement('meld3'),
    python_requirement("supervisor"),
  ]
)

python_library(
  name = "external_library",
  dependencies = [
    python_requirement('argparse'),
    pants("src/demoapplication_lib"),
  ]
)

This build file defines two binary targets, namely "demoapplication" and "supervisord". "demoapplication" is also dependent on libraries located in directory src/demoapplication_lib. While, "supervisord" is dependent only on external libraries meld3 and supervisor. "src/demoapplication_lib" directory should contain another build file that tells what pants should do to with this target.

python_library(
  name = "demoapplication_lib",
  sources = globs('*.py'),
  dependencies = [
  ]
)

Next step is to store the wheel packages of external dependency in the cache.

pip wheel --use-wheel -w "$PANTS_WHEELHOUSE" -f "$PANTS_WHEELHOUSE" --download-cache "$PANTS_WHEELHOUSE_CACHE" -r requirements.txt

I am now ready to build the application as executable binary "demoapplication.pex".

PANTS_VERBOSE=1 PYTHON_VERBOSE=2 pants build -v -i "CPython>=2.7,<3" :demoapplication

I usually use sphinx for documentation with module documenting feature. This requires that all the application dependencies are packaged with the documentation tool binary. It is easy to get that information from requirements.txt file. I then use the binary to generate the documents.

ALL_DOC_DEPS=`cat requirements.txt | fmt -1 | xargs -I placeme echo "-r placeme" | xargs`
pex $ALL_DOC_DEPS -e sphinx:main --no-pypi --repo=$PANTS_WHEELHOUSE -o $SCRIPTDIR/dist/doctool.pex
doctool.pex -a -c docs/source/ -b html docs/source docs/build/

I typically use pytest to drive unit testing. It works well with the target application binary (pex file) too. To do that I usually include pytest as an external dependency while building the tool.

PEX_MODULE=pytest demoapplication.pex test/test*.py

By now, I have built the application, generated documentation and ran the unit tests. Now its time to package it. First, I package what ever is required into a tar ball - "demoapplication.tar.gz". I use fpm to convert it into an easy to deploy rpm package.

fpm -s tar -t rpm -n "${PACKAGE_NAME}"
    -v $VERSION
    --license $LICENSE
    --vendor $VENDOR
    --maintainer "${MAINTAINER}"
    --url URL
    --description "${DESCRIPTION}"
    --iteration $ITERATION
    demoapplication.tar.gz