Wednesday, September 17, 2014

Building and packaging a python application for distribution

I find it messy to build a python application package that is easy to distribute . Though, python tools have come a long way.

"pip/wheel" helps in installing, managing and distributing individual packages. "virtualenv" provides an approachable way to create isolated environment and avoid the pollution of the main distribution. "zc.buildout" lets you reproduce / assemble and deploy a python application through configuration. Together, they provide a powerful framework for building and distributing python applications.

However, It is not as simple as build once and distribute everywhere model of executable / jar package distribution. In all likelihood, you will be creating an isolated virtual environment, installing the dependencies and the application into it.

Build and packaging tools

Two excellent tools from twitter namely "pex", "pants" and packaging tool called "fpm" helps in providing a system that allows me to build once and distribute everywhere. Along with the tools mentioned above, they make a complete tool chain for building application packages.
I use a simple shell script to build and create an installation package for distribution. Rest of blog post is about how I have used these tools to create an easy to deploy and distribe application package.
PEX
A pex file is a self-bootstrapping python environment. You can consider it as an isolated virtual environment packed in a file.
This makes for an environment super easy to distribute. It can be built once and used everywhere with a few caveats, mostly due to platform dependencies. Most likely, you should be able to build platform specific "pex" distribution file.

You can use it to create a file that acts like a python interpreter with all the dependencies added already. You can use pex to create a sphinx documentation tool executable - "sphinx.pex". Executing the tool, you will find that it acts like "sphinx-build" script. You can read more about it in pex documentation.
pex -r sphinx  -e sphinx:main -o sphinx.pex

./sphinx.pex -h
Sphinx v1.2.3
Usage: ./sphinx.pex [options] sourcedir outdir [filenames...]
......

Pants
Pants is a build tool similar to makefile, gradle, ant. It can handle Java, Scala and Python code base. Pants builds "targets" which could be a library, binary based on a hierarchal tree of "BUILD" files. Pants can be used to build a part of your code base by targeting a branch of "BUILD" files tree. Another useful aspect about building python code base is that it builds a python binary target as "pex" file. There is more to pants than described here. Go through pants documentation.
FPM
FPM eases creation of distribution packages like "rpm" and "deb" formats for different target platforms. You can easily package a tarball into an rpm.
fpm -s tar -t rpm -n "${package_name}"
    -v version
    --license license
    --vendor vendor
    --maintainer "maintainer"
    --url url
    --description "description"
    --iteration iteration
    path_to_package.tar.gz
Now you have a rpm package called "package_name-version-iteration.arch.rpm" which can be distributed and deployed.

Using these tools

I use a build script to build, test an application and package it into an installable format. I use virtualenv to create a isolated environment:
virtualenv build_dir/virtual_env
I install the pants build system in the virtual environment that I just created using pip.
source build_dir/virtual_env/bin/activate
pip install "http://effbot.org/media/downloads/elementtree-1.2.7-20070827-preview.zip"
pip install coverage
pip install twitter.pants

For mac osx, use: Mac: export ARCHFLAGS="-Wno-error=unused-command-line-argument-hard-error-in-future"; pip install pkg_toinstall
Initializing the pants build tool requires access to PyPI and should be enabled in pants.ini. How ever, I turn it off while building the application. I depend on local repository of wheel packages. To make this kind of switch, it is better to generate the pants.ini file from a template. I also substitute the paths to a few variables to isolate the pants working directory per project.
PANTS_DIR=build_dir/pants
PANTS_CACHEDIR=$PANTS_DIR/.pants.d
PANTS_WORKDIR=$PANTS_DIR/.pants.d
PANTS_SUPPORTDIR=$PANTS_DIR/build-support
PANTS_PYTHONS=$PANTS_DIR/.python
PANTS_WHEELHOUSE=$PANTS_DIR/wheelpackage
PANTS_WHEELHOUSE_CACHE=$PANTS_DIR/wheelpackage/cache
and use sed to transform the template into usable "pants.ini" file.
cp pants_template_file.ini pants.ini
sed -i.bak "s|PANTS_CACHEDIR|${PANTS_CACHEDIR}|g" pants.ini
.......
With pypi repository enabled in pants.ini.
indices:  ['pypi.python.org']
allow_pypi: True 
we can initialize pants build system.
pants py
To prepare for building application, I turn off pypi repository and enable local wheel repository.
[python-repos]
repos: [
  'PANTS_WHEELHOUSE/',
  'https://raw.github.com/twitter/commons/binaries/pants/third_party/python/dist/index.html',
  'https://raw.github.com/twitter/commons/binaries/pants/third_party/python/index.html']
#indices:  ['pypi.python.org']
allow_pypi: False 
Now we are read to build our application. This is defined as a binary target in "BUILD" file. The binary target can be dependent on external and internal packages.
python_binary(
  name = 'demoapplication',
  source = 'src/demoapplication.py',
  dependencies = [
    pants(":external_library"),
  ]
)

python_binary(name = "supervisord",
  entry_point = "supervisor.supervisord:main",
  dependencies = [
    pants(":supervisord_library"),
  ]
)

python_library(
  name = "supervisord_library",
  dependencies = [
    python_requirement('meld3'),
    python_requirement("supervisor"),
  ]
)

python_library(
  name = "external_library",
  dependencies = [
    python_requirement('argparse'),
    pants("src/demoapplication_lib"),
  ]
)
This build file defines two binary targets, namely "demoapplication" and "supervisord". "demoapplication" is also dependent on libraries located in directory src/demoapplication_lib. While, "supervisord" is dependent only on external libraries meld3 and supervisor. "src/demoapplication_lib" directory should contain another build file that tells what pants should do to with this target.
python_library(
  name = "demoapplication_lib",
  sources = globs('*.py'),
  dependencies = [
  ]
)
Next step is to store the wheel packages of external dependency in the cache.
pip wheel --use-wheel -w "$PANTS_WHEELHOUSE" -f "$PANTS_WHEELHOUSE" --download-cache "$PANTS_WHEELHOUSE_CACHE" -r requirements.txt
I am now ready to build the application as executable binary "demoapplication.pex".
PANTS_VERBOSE=1 PYTHON_VERBOSE=2 pants build -v -i "CPython>=2.7,<3" :demoapplication
I usually use sphinx for documentation with module documenting feature. This requires that all the application dependencies are packaged with the documentation tool binary. It is easy to get that information from requirements.txt file. I then use the binary to generate the documents.
ALL_DOC_DEPS=`cat requirements.txt | fmt -1 | xargs -I placeme echo "-r placeme" | xargs`
pex $ALL_DOC_DEPS -e sphinx:main --no-pypi --repo=$PANTS_WHEELHOUSE -o $SCRIPTDIR/dist/doctool.pex
doctool.pex -a -c docs/source/ -b html docs/source docs/build/
I typically use pytest to drive unit testing. It works well with the target application binary (pex file) too. To do that I usually include pytest as an external dependency while building the tool.
PEX_MODULE=pytest demoapplication.pex test/test*.py
By now, I have built the application, generated documentation and ran the unit tests. Now its time to package it. First, I package what ever is required into a tar ball - "demoapplication.tar.gz". I use fpm to convert it into an easy to deploy rpm package.
fpm -s tar -t rpm -n "${PACKAGE_NAME}"
    -v $VERSION
    --license $LICENSE
    --vendor $VENDOR
    --maintainer "${MAINTAINER}"
    --url URL
    --description "${DESCRIPTION}"
    --iteration $ITERATION
    demoapplication.tar.gz

22 comments:

  1. Nice Website...
    Hey JOIN now fblikesbot.com and Increase Facebook Likes your profile and websites.
    Increase Facebook Likes and check your website worth worth my websites
    its may be very beneficial for you also really

    ReplyDelete
  2. Quality Services- By booking an all-inclusive package tour you enjoy best facilities throughout the trip. Travel agencies and tour operators spend a lot of time in assessing all the aspects of a package tour to deliver high standard services to its customers. They choose best hotels and cover most frequented areas of a particular place for customers to visit.

    Packaging is Fun

    ReplyDelete
  3. Nice content in this blog to get knowledge about application packaging. Keep me more updates.
    application packaging
    upgrade from ie8 to ie11

    ReplyDelete
  4. Wow amazing i saw the article with execution models you had posted for the python models. It was such informative. Really its a wonderful article. Thank you for sharing and please keep update like this type of article because i want to learn more relevant to this topic.

    Salesforce Training

    ReplyDelete
  5. Thanks a lot for all your valuable article! We are really happy about the your thoughts... SAP Training in chennai

    ReplyDelete
  6. This blog is having a wonderful talk. The technology are discussed and provide a great knowledge toall. This helps to learn more details about technology. All this details are important for this technology. Thank you for this blog.
    Hadoop Training in Chennai

    ReplyDelete
  7. Really an amazing post..! By reading your blog post i gained more information. Thanks a lot for posting unique information and made me more knowledgeable person. Keep on blogging!!
    Android Training in Chennai Adyar

    ReplyDelete
  8. thank you for sharing such a nice and interesting blog with us. hope it might be much useful for us. keep on updating...
    ROI Services in Chennai

    ReplyDelete
  9. Your application concept is really nice and thus it is very well doing and it is really good thanks for sharing these valuable information.

    Digital Marketing Company in Chennai

    ReplyDelete
  10. It's interesting that many of the bloggers to helped clarify a few things for me as well as giving.Most of ideas can be nice content.The people to give them a good shake to get your point and across the command .

    PPC Services in Chennai

    ReplyDelete
  11. Such a great articles in my carrier, It's wonderful commands like easiest understand words of knowledge in information's.

    Branding Services in Chennai

    ReplyDelete
  12. Someone essentially lend a hand to make severely posts I would state. That is the very first time I frequented your website page and thus far? I surprised with the analysis you made to create this particular submit incredible. Fantastic job!

    SMO Services Chennai

    ReplyDelete
  13. Wowwww... really great information. Having interesting topic. The pictorial representation helps me to easy understanding of this. Have a great idea. Thank you.
    SEO Web Design Company in Chennai

    ReplyDelete
  14. Why everyone is commenting with a linking to spam. This is quality post and you guys don't respect it.

    ReplyDelete
  15. Thank you for sharing such a nice and interesting blog with us. Hope it might be much useful for us. keep on updating...!!
    seo company in india
    digital marketing company in india
    seo company in chennai
    digital marketing company in chennai

    ReplyDelete
  16. Superb. I really enjoyed very much with this article here. Really it is an amazing article I had ever read. I hope it will help a lot for all. Thank you so much for this amazing posts and please keep update like this excellent article.thank you for sharing such a great blog with us. expecting for your.
    seo company in india

    ReplyDelete
  17. Superb. I really enjoyed very much with this article here. Really it is an amazing article I had ever read. I hope it will help a lot for all. Thank you so much for this amazing posts and please keep update like this excellent article.thank you for sharing such a great blog with us. expecting for your.

    seo company in india

    ReplyDelete
  18. Wonderful bloggers like yourself who would positively reply encouraged me to be more open and engaging in commenting.So know it's helpful.

    SEO Company in Chennai

    ReplyDelete
  19. I feel so much smarter now. I like the trend that content is becoming more and more important.I like the trend that content is becoming more and more important.
    Best IELTS Coaching Center in Chennai | IELTS Training Centers in Chennai | IELTS Coaching Classes in Chennai

    ReplyDelete
  20. All the details are explained clearly with the great explanation. Thanks for this wonderful blog. Step by step processes execution are given clearly.Know the details about different thing.
    Selenium Training in Chennai

    ReplyDelete
  21. It's interesting that many of the bloggers your tips helped to clarify a few things for me as well as giving.. very specific nice content. And tell people specific ways to live their lives.Sometimes you just have to yell at people and give them a good shake to get your point across.

    Best Laser Clinic In Chennai

    Best Implant Clinic In Chennai


    ReplyDelete

  22. This article is so informatic and it really helped me to know more about the Selenium Testing. So keep updating the content regularly.
    Selenium Training in Chennai | Selenium Training | Selenium Course in Chennai

    ReplyDelete