Wednesday, September 17, 2014

Building and packaging a python application for distribution

I find it messy to build a python application package that is easy to distribute . Though, python tools have come a long way.

"pip/wheel" helps in installing, managing and distributing individual packages. "virtualenv" provides an approachable way to create isolated environment and avoid the pollution of the main distribution. "zc.buildout" lets you reproduce / assemble and deploy a python application through configuration. Together, they provide a powerful framework for building and distributing python applications.

However, It is not as simple as build once and distribute everywhere model of executable / jar package distribution. In all likelihood, you will be creating an isolated virtual environment, installing the dependencies and the application into it.

Build and packaging tools

Two excellent tools from twitter namely "pex", "pants" and packaging tool called "fpm" helps in providing a system that allows me to build once and distribute everywhere. Along with the tools mentioned above, they make a complete tool chain for building application packages.
I use a simple shell script to build and create an installation package for distribution. Rest of blog post is about how I have used these tools to create an easy to deploy and distribe application package.
PEX
A pex file is a self-bootstrapping python environment. You can consider it as an isolated virtual environment packed in a file.
This makes for an environment super easy to distribute. It can be built once and used everywhere with a few caveats, mostly due to platform dependencies. Most likely, you should be able to build platform specific "pex" distribution file.

You can use it to create a file that acts like a python interpreter with all the dependencies added already. You can use pex to create a sphinx documentation tool executable - "sphinx.pex". Executing the tool, you will find that it acts like "sphinx-build" script. You can read more about it in pex documentation.
pex -r sphinx  -e sphinx:main -o sphinx.pex

./sphinx.pex -h
Sphinx v1.2.3
Usage: ./sphinx.pex [options] sourcedir outdir [filenames...]
......

Pants
Pants is a build tool similar to makefile, gradle, ant. It can handle Java, Scala and Python code base. Pants builds "targets" which could be a library, binary based on a hierarchal tree of "BUILD" files. Pants can be used to build a part of your code base by targeting a branch of "BUILD" files tree. Another useful aspect about building python code base is that it builds a python binary target as "pex" file. There is more to pants than described here. Go through pants documentation.
FPM
FPM eases creation of distribution packages like "rpm" and "deb" formats for different target platforms. You can easily package a tarball into an rpm.
fpm -s tar -t rpm -n "${package_name}"
    -v version
    --license license
    --vendor vendor
    --maintainer "maintainer"
    --url url
    --description "description"
    --iteration iteration
    path_to_package.tar.gz
Now you have a rpm package called "package_name-version-iteration.arch.rpm" which can be distributed and deployed.

Using these tools

I use a build script to build, test an application and package it into an installable format. I use virtualenv to create a isolated environment:
virtualenv build_dir/virtual_env
I install the pants build system in the virtual environment that I just created using pip.
source build_dir/virtual_env/bin/activate
pip install "http://effbot.org/media/downloads/elementtree-1.2.7-20070827-preview.zip"
pip install coverage
pip install twitter.pants

For mac osx, use: Mac: export ARCHFLAGS="-Wno-error=unused-command-line-argument-hard-error-in-future"; pip install pkg_toinstall
Initializing the pants build tool requires access to PyPI and should be enabled in pants.ini. How ever, I turn it off while building the application. I depend on local repository of wheel packages. To make this kind of switch, it is better to generate the pants.ini file from a template. I also substitute the paths to a few variables to isolate the pants working directory per project.
PANTS_DIR=build_dir/pants
PANTS_CACHEDIR=$PANTS_DIR/.pants.d
PANTS_WORKDIR=$PANTS_DIR/.pants.d
PANTS_SUPPORTDIR=$PANTS_DIR/build-support
PANTS_PYTHONS=$PANTS_DIR/.python
PANTS_WHEELHOUSE=$PANTS_DIR/wheelpackage
PANTS_WHEELHOUSE_CACHE=$PANTS_DIR/wheelpackage/cache
and use sed to transform the template into usable "pants.ini" file.
cp pants_template_file.ini pants.ini
sed -i.bak "s|PANTS_CACHEDIR|${PANTS_CACHEDIR}|g" pants.ini
.......
With pypi repository enabled in pants.ini.
indices:  ['pypi.python.org']
allow_pypi: True 
we can initialize pants build system.
pants py
To prepare for building application, I turn off pypi repository and enable local wheel repository.
[python-repos]
repos: [
  'PANTS_WHEELHOUSE/',
  'https://raw.github.com/twitter/commons/binaries/pants/third_party/python/dist/index.html',
  'https://raw.github.com/twitter/commons/binaries/pants/third_party/python/index.html']
#indices:  ['pypi.python.org']
allow_pypi: False 
Now we are read to build our application. This is defined as a binary target in "BUILD" file. The binary target can be dependent on external and internal packages.
python_binary(
  name = 'demoapplication',
  source = 'src/demoapplication.py',
  dependencies = [
    pants(":external_library"),
  ]
)

python_binary(name = "supervisord",
  entry_point = "supervisor.supervisord:main",
  dependencies = [
    pants(":supervisord_library"),
  ]
)

python_library(
  name = "supervisord_library",
  dependencies = [
    python_requirement('meld3'),
    python_requirement("supervisor"),
  ]
)

python_library(
  name = "external_library",
  dependencies = [
    python_requirement('argparse'),
    pants("src/demoapplication_lib"),
  ]
)
This build file defines two binary targets, namely "demoapplication" and "supervisord". "demoapplication" is also dependent on libraries located in directory src/demoapplication_lib. While, "supervisord" is dependent only on external libraries meld3 and supervisor. "src/demoapplication_lib" directory should contain another build file that tells what pants should do to with this target.
python_library(
  name = "demoapplication_lib",
  sources = globs('*.py'),
  dependencies = [
  ]
)
Next step is to store the wheel packages of external dependency in the cache.
pip wheel --use-wheel -w "$PANTS_WHEELHOUSE" -f "$PANTS_WHEELHOUSE" --download-cache "$PANTS_WHEELHOUSE_CACHE" -r requirements.txt
I am now ready to build the application as executable binary "demoapplication.pex".
PANTS_VERBOSE=1 PYTHON_VERBOSE=2 pants build -v -i "CPython>=2.7,<3" :demoapplication
I usually use sphinx for documentation with module documenting feature. This requires that all the application dependencies are packaged with the documentation tool binary. It is easy to get that information from requirements.txt file. I then use the binary to generate the documents.
ALL_DOC_DEPS=`cat requirements.txt | fmt -1 | xargs -I placeme echo "-r placeme" | xargs`
pex $ALL_DOC_DEPS -e sphinx:main --no-pypi --repo=$PANTS_WHEELHOUSE -o $SCRIPTDIR/dist/doctool.pex
doctool.pex -a -c docs/source/ -b html docs/source docs/build/
I typically use pytest to drive unit testing. It works well with the target application binary (pex file) too. To do that I usually include pytest as an external dependency while building the tool.
PEX_MODULE=pytest demoapplication.pex test/test*.py
By now, I have built the application, generated documentation and ran the unit tests. Now its time to package it. First, I package what ever is required into a tar ball - "demoapplication.tar.gz". I use fpm to convert it into an easy to deploy rpm package.
fpm -s tar -t rpm -n "${PACKAGE_NAME}"
    -v $VERSION
    --license $LICENSE
    --vendor $VENDOR
    --maintainer "${MAINTAINER}"
    --url URL
    --description "${DESCRIPTION}"
    --iteration $ITERATION
    demoapplication.tar.gz

100 comments:

  1. Quality Services- By booking an all-inclusive package tour you enjoy best facilities throughout the trip. Travel agencies and tour operators spend a lot of time in assessing all the aspects of a package tour to deliver high standard services to its customers. They choose best hotels and cover most frequented areas of a particular place for customers to visit.

    Packaging is Fun

    ReplyDelete
  2. Nice content in this blog to get knowledge about application packaging. Keep me more updates.
    application packaging
    upgrade from ie8 to ie11

    ReplyDelete
  3. Wow amazing i saw the article with execution models you had posted for the python models. It was such informative. Really its a wonderful article. Thank you for sharing and please keep update like this type of article because i want to learn more relevant to this topic.

    Salesforce Training

    ReplyDelete
  4. Thanks a lot for all your valuable article! We are really happy about the your thoughts... SAP Training in chennai

    ReplyDelete
  5. This blog is having a wonderful talk. The technology are discussed and provide a great knowledge toall. This helps to learn more details about technology. All this details are important for this technology. Thank you for this blog.
    Hadoop Training in Chennai

    ReplyDelete
  6. Really an amazing post..! By reading your blog post i gained more information. Thanks a lot for posting unique information and made me more knowledgeable person. Keep on blogging!!
    Android Training in Chennai Adyar

    ReplyDelete
  7. Your application concept is really nice and thus it is very well doing and it is really good thanks for sharing these valuable information.

    Digital Marketing Company in Chennai

    ReplyDelete
  8. Such a great articles in my carrier, It's wonderful commands like easiest understand words of knowledge in information's.

    Branding Services in Chennai

    ReplyDelete
  9. Wowwww... really great information. Having interesting topic. The pictorial representation helps me to easy understanding of this. Have a great idea. Thank you.
    SEO Web Design Company in Chennai

    ReplyDelete
  10. Why everyone is commenting with a linking to spam. This is quality post and you guys don't respect it.

    ReplyDelete
  11. Superb. I really enjoyed very much with this article here. Really it is an amazing article I had ever read. I hope it will help a lot for all. Thank you so much for this amazing posts and please keep update like this excellent article.thank you for sharing such a great blog with us. expecting for your.
    seo company in india

    ReplyDelete
  12. Superb. I really enjoyed very much with this article here. Really it is an amazing article I had ever read. I hope it will help a lot for all. Thank you so much for this amazing posts and please keep update like this excellent article.thank you for sharing such a great blog with us. expecting for your.

    seo company in india

    ReplyDelete
  13. It's interesting that many of the bloggers your tips helped to clarify a few things for me as well as giving.. very specific nice content. And tell people specific ways to live their lives.Sometimes you just have to yell at people and give them a good shake to get your point across.

    Best Laser Clinic In Chennai

    Best Implant Clinic In Chennai


    ReplyDelete

  14. This article is so informatic and it really helped me to know more about the Selenium Testing. So keep updating the content regularly.
    Selenium Training in Chennai | Selenium Training | Selenium Course in Chennai

    ReplyDelete


  15. very useful info, and please keep updating........
    Best Online Software Training

    ReplyDelete
  16. Very nice interested blog very nice blog full information. Please Visit:

    windows 7 to 10 migration
    16-bit application
    Windows Packaging

    ReplyDelete
  17. nice article great post comment information thanks for sharing
    goldenslot

    ReplyDelete
  18. Eduwizz online training is one of the Best Online Training Institute in Hyderabad, Bangalore. Eduwizz provide courses Hybris , Guidewire, Adobe, RPA , Machine Learning, AWS, Statistics for Beginners, Commvault Training, Devops, Netapps, TSM, EMC, Data Science, Internet of Things , IBM Blue-Mix , Hybris ,Angular JS , Node JS , Express JS , Business Analyst, Selenium testing with webdriver.

    ReplyDelete

  19. Interesting blog about buliding and packaging python which attracted me more.Spend a worthful time.keep updating more.
    Informatica Training in Chennai

    ReplyDelete
  20. I really enjoyed reading the Post. It was very informative and useful for me.
    Selenium Training In Bangalore

    ReplyDelete
  21. Superb explanation & it's too clear to understand the concept as well, keep sharing admin with some updated information with right examples.Keep update more posts.


    SEO Company in chennai

    Digital Marketing Company in Chennai

    SEO Company in India

    Digital Marketing Company in India

    ReplyDelete
  22. 2. Best Digital Marketing company Anantapur
    helpful information, thanks for writing and share this information

    ReplyDelete
  23. Graceful written content on this blog is really useful for everyone same as I got to know. Difficult to locate relevant and useful informative blog as I found this one to get more knowledge but this is really a nice one.
    wholesale cake box printing

    ReplyDelete
  24. Learning new technology would help oneself at hard part of their career. And staying updated is the only way to survive in current position. Your content tells the same. Thanks for sharing this information in here. Keep blogging like this.
    Best Data Science Online Training Institute In Hyderabad | Online Data Science Training
    Data Science Online Training Institute In Hyderabad
    Data science online training in hyderabad
    Best data science training in hyderabad


    ReplyDelete
  25. It’s great to come across a blog every once in a while that isn’t the same out of date rehashed material. Fantastic read.

    Digital Marketing Training in Mumbai

    Six Sigma Training in Dubai

    Six Sigma Abu Dhabi

    ReplyDelete
  26. Become a JAVA Expert in 3 Months using our fast-track course. Our expert trainers ensure that your learning is based on the industry best practices.

    ReplyDelete
  27. This comment has been removed by the author.

    ReplyDelete
  28. Great Article… I love to read your articles because your writing style is too good, its is very very helpful for all of us and I never get bored while reading your article because, they are becomes a more and more interesting from the starting lines until the end. 
    microsoft azure training in bangalore
    rpa training in bangalore
    best rpa training in bangalore

    ReplyDelete
  29. Thanks for your informative article, Your post helped me to understand the future and career prospects & Keep on updating your blog with such awesome article.
    Best Devops online Training
    Online DevOps Certification Course - Gangboard
    Best Devops Training institute in Chennai

    ReplyDelete
  30. I always enjoy reading quality articles by an individual who is obviously knowledgeable on their chosen subject. Ill be watching this post with much interest. Keep up the great work, I will be back
    python Course in Pune
    python Course institute in Chennai
    python Training institute in Bangalore

    ReplyDelete
  31. What is web and digital design?
    The aim of the Master’s course in net and Digital design is that the creation of various skilled profiles with in the web media design space that ar ready to set up, turn out and share through the online communities a replacement generation of content and services in an efficient manner. If you want courses oriented any details to contact us : Web Design Training Courses

    ReplyDelete
  32. Great blog! Thanks for the info. Keep going.

    https://socialprachar.com/data-science-course-online-training-institute/

    ReplyDelete
  33. I glad I came across this piece of writings. Keep up the great work. Software Development Company in India

    ReplyDelete
  34. This comment has been removed by the author.

    ReplyDelete
  35. Thanks for sharing information with us. If someone wants to know about Taxi Service App and Health Management Software I think this is the right place for you.
    Taxi Dispatch App | Taxi Service Providers | Safety and Health Management System

    ReplyDelete
  36. Thank you for excellent article.You made an article that is interesting.
    AWS Solutions Architect courses in Bangalore with certifications.
    https://onlineidealab.com/aws-training-in-bangalore/


    ReplyDelete
  37. Thanks for your post! Really interesting blogs. Here is the some more interesting and most related links.

    Best digital marketing company in Dubai, United Arab Emirates. Brandstory is one of the top and best digital marketing companies in Dubai UAE. As a leading digital marketing agency in Dubai, We offer search engine optimization services, online marketing services, UI UX design services, search engine marketing services, email marketing services, Google / Facebook / Bing pay per click services, Internet marketing services, website design services and website development services, social media marketing services. Hire ROI based digital marketing services company in dubai to get digital leads for your business.

    Digital marketing company in Dubai | Digital Marketing Agency in Dubai | SEO Company in Dubai | SEO Agency in Dubai | Best Digital Marketing Companies in Dubai | Top Digital Marketing Agencies in Dubai | Best SEO Companies in Dubai | SEO Agencies in Dubai | Online Marketing Company in Dubai | SEO Services Company in Dubai | PPC Company in Dubai | PPC Agency in Dubai | | PPC Services in Dubai | Social Media Marketing Company in Dubai | Social Media Marketing Services in Dubai | Social Media Marketing Agencies in Dubai | Web Design Company in Dubai | Website Designers in Dubai | Website Development Services Company in Dubai | Web Design Companies in Dubai

    ReplyDelete
  38. best beauty parlour
    Learn Spoken English from Native English Teachers. Our Unique Approach is 6X Faster than any other Teaching Method. Improve in Weeks!

    ReplyDelete
  39. There is plainly a ton to consider this. Keep working, remarkable work!
    iot training in delhi

    ReplyDelete
  40. I really enjoyed reading this blog. It was explained and structured with perfection; Best Digital Marketing Company in Delhi

    ReplyDelete
  41. That was an extremely interesting and knowledgeable blog consisting of very relevant and valid points. Here is a referred link same as yours   oracle apex training.  Thanks for sharing this with us as it was quite useful and informative.

    ReplyDelete

  42. We are well established IT and outsourcing firm working in the market since 2013. We are providing training to the people ,
    like- Web Design , Graphics Design , SEO, CPA Marketing & YouTube Marketing.Call us Now whatsapp: +(88) 01537587949
    : SEO training course
    Free bangla sex video:careful
    good post Freelancing Training in bangladesh

    ReplyDelete
  43. I like viewing web sites which comprehend the price of delivering the excellent useful resource free of charge. I truly adored reading your posting. Thank you!
    Best Digital Marketing Institute in Hyderabad

    ReplyDelete
  44. I really appreciate your articles im a daily reader of your articles i love your informative tips and paragraph which are rich of useful knowledge thanks for sharing
    Mobile App Development Companies In Bangalore|
    |Android App Development Company Bangalore|
    |UX Design Companies In Bangalore|
    |Mobile App Development Company In Bangalore
    |UI UX Design Companies In Bangalore |

    ReplyDelete
  45. Awesome blog post, thanks for sharing.
    Digital Marketing is right now the most stable job you could have. There are many openings for Digital Marketers around the globe. Hence we suggest you to attend our Digital Marketing Course in Hyderabad to acquire skills that a Digital Marketer needs.

    ReplyDelete
  46. I think you did mgsu bsc 3rd year result an awesome job explaining it. Sure Hpu ba 3rd Year Result beats having to research it on my own. Thanks

    ReplyDelete
  47. Nice post really useful information. We are the leading Ecommerce Website Designers in Bangalore . Hire our ecommerce website designers in bangalore for ecommerce website development services.

    ReplyDelete
  48. This comment has been removed by the author.

    ReplyDelete
  49. Normal visits recorded here are the least demanding strategy to see the value in your energy, which is the reason why I am going to the site regular, looking for new, intriguing data. Many, much obliged! Software development services company Thanks for sharing this informative article now visit this Hire PHP Developers

    ReplyDelete
  50. Thanks for such a great article. It has helped me a lot to troubleshoot my offline printer and I can easily set it up to overcome the problem.
    wordpress Casino
    ufa88kh.blogspot Casino
    youtube Casino

    ហ្គេមបាញ់ត្រីអនឡាញ

    ReplyDelete

  51. I have truly glad to these perusing your post. This item control and the upkeep of our well-being. The everyday schedule can help you weight lose rapidly and securely. My life is revised once I followed this eating routine. I feeling pleasant concerning myself. Hire PHP Developers
    Hire Full Stack Developer

    ReplyDelete
  52. Hi Thanks for Sharing this Valuable Information with us: this is very useful for me. Keep it Up.
    data scientist courses in aurangabad

    ReplyDelete
  53. Nice post. I am happy after reading the post that you have posted on this blog. Thanks for sharing a wonderful post and hoping to post more.
    What are the Most Important Benefits of CBD or Cannabidiol

    ReplyDelete
  54. I was very pleased to find this site.I wanted to thank you for this great read!! I definitely enjoying every little bit of it and I have you bookmarked to check out new stuff you post. data science course in surat

    ReplyDelete
  55. Never miss a stock price change from Acam Stock with our Live, Real Time Stock Market Overview! Most of the data is updated every 5 minutes to ensure that you are getting all of the latest Acam Stock change alerts as soon as they happen.

    ReplyDelete
  56. Fxit Stock Overview: Stay ahead of the market with our live and real time stock market overview. Get all of your favorite stocks in one place, and get alerted to live news at the same time!

    ReplyDelete
  57. Gteh Stocktwits Real-Time Overview Of A Stock, Including Recent And Historical Price Charts, News, Events, Analyst Rating Changes And Other Key Stock Information.

    ReplyDelete

  58. Extremely overall quite fascinating post. I was searching for this sort of data and delighted in perusing this one. Continue posting. A debt of gratitude is in order for sharing.data science training in warangal

    ReplyDelete
  59. I am very thankful to you for providing such a great information. It is simple but very accurate information.

    MLSU BA 1st Year Result
    MLSU BA 2nd Year Result
    MLSU BA 3rd Year Result

    ReplyDelete
  60. Indus Valley Mukhteshwar’s team was truly a collaborative partner working with in a cost effective
    , efficient and quality construction product
    The knowledge of their team was truly a value add to the overall success
    ofour build-out project.

    ReplyDelete
  61. F Awesome article! Your explanations on building and packaging Python applications for distribution were concise and easy to understand. Thanks for sharing your knowledge! get more information about! Commercial Vehicle Insurance

    ReplyDelete
  62. Great post Thanks for sharing valuable information, keep posting more Python Course in Pune
    best python training institute in Pune

    ReplyDelete
  63. your blog on building-and-packaging-python is really helpful for us , thank you so much for this amazing writeup keep posting, and if you want to learn some new python technology and checkout our blog python classes in pune

    ReplyDelete