Python - virtualenv is your friend Source

May 25, 2012 Markdown source

1---
2title: 'Python - virtualenv is your friend'
3date: '2012-05-25'
4published_at: '2012-05-25T17:25:00.000+10:00'
5tags: ['development', 'pip', 'programming', 'python', 'sysadmin', 'virtualev']
6author: 'Gavin Jackson'
7excerpt: 'My manager (www.nathanr.net) recently shared the following technique that we use to manage our python instances, I thought it was so good, I wanted to share it with you. No longer stress about what ve...'
8updated_at: '2012-06-18T12:03:14.113+10:00'
9legacy_url: 'http://www.gavinj.net/2012/05/python-virtualenv-is-your-friend.html'
10---
11
12[![](https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEicCX1zBSSPc1RezUh3I15Tq9aCJWJJMZwWzBcQaRfLK4C_qLIXLcLq9mpQZ0cF-xSAwBTZJaJzjL1X60swoH4UARzndR6Ee7IQYoPnu3BIf2rFFiMp9xqwBUR1kubcT8dWqRa56WVXZHA/s1600/woma-python.jpg)](https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEicCX1zBSSPc1RezUh3I15Tq9aCJWJJMZwWzBcQaRfLK4C_qLIXLcLq9mpQZ0cF-xSAwBTZJaJzjL1X60swoH4UARzndR6Ee7IQYoPnu3BIf2rFFiMp9xqwBUR1kubcT8dWqRa56WVXZHA/s1600/woma-python.jpg)
13
14My manager (www.nathanr.net) recently shared the following technique that we use to manage our python instances, I thought it was so good, I wanted to share it with you.
15
16No longer stress about what version of python is installed with your Linux distribution.
17
18Python source + virtualenv + pip is your savior.
19
20**Virtualenv** ([http://pypi.python.org/pypi/virtualenv](http://pypi.python.org/pypi/virtualenv)) is a bit like a vm for a python install, it takes a vanilla python build (typically from source) and allows you to "activate" it (replacing your default python binaries and libraries with the ones within the virtualenv instance.
21
22This has the benefit of freeing you from the outdated python binaries found in most Linux distributions, and allowing you to tailor specific library configurations for each environment.
23
24**Pip** ([http://pypi.python.org/pypi/pip/1.1](http://pypi.python.org/pypi/pip/1.1)) is a tool for installing and managing python packages (a bit like CPAN for python). We can use this within the virtualenv instance to manage our installed libraries.
25
26By default it queries the online python package index (PyPI - [http://pypi.python.org/pypi](http://pypi.python.org/pypi)).
27
28##  Building python and installing virtualenv into it
29
30Before running the following, you will need to download the python source code (http://www.python.org/download/) and virtualenv (http://pypi.python.org/pypi/virtualenv).
31
32**#install dev libraries (SLES SP2)** **#SUSE build tool pattern (same as build-essential on Ubuntu) ** zypper install zypper install -t pattern Basis-Devel zypper install libbz2-devel zypper install readline-devel zypper install ncurses-devel zypper install libopenssl-devel zypper install libxslt-devel
33
34**#build python**
35
36tar -xjf Python-2.7.3.tar.bz2
37
38cd Python-2.7.3/
39
40./configure --prefix=/usr/local/python-2.7.3
41
42make
43
44make install
45
46ln -s python-2.7.3 python-2.7
47
48ln -s python-2.7 python
49
50**#install virtualenv ** tar -xzf ~/Downloads/virtualenv-1.7.1.2.tar.gz cd virtualenv-1.7.1.2/
51
52/usr/local/python/bin/python setup.py install
53
54Ok, we now have a Python 2.7 install with virtualenv in it. Now, create a virtualenv for our development and make it your working virtual environment:
55
56**#set up virtualenv in environment-name** /usr/local/python/bin/virtualenv environment-name New python executable in environment-name/bin/python Installing setuptools............done. Installing pip...............done.
57
58**#activate the virtual environment ** cd environment-name/. bin/activate (environment-name)user@host:~/tmp/environment-name>
59
60##  Installing python libraries into the virtualenv instance
61
62Note that some libraries in the online pip repositories are somewhat dated, to ensure that we have the correct versions we maintain a local cache (which we keep checked into our mercurial repository).
63
64**pip install --requirement=~/work/libraries/python/python-pip-freeze.txt --find-links=file://$HOME/work/libraries/python/cache --no-index**
65
66Where the cache folder contains the tar.gz files for all of the libraries contained in the python-pip-freeze.txt file (this file can easily be regenerated using the **pip freeze** command).
67
68In our case the file contains the following libs:
69
70**CherryPy==3.2.2 Mako==0.7.0 MarkupSafe==0.15 PyXB-base==1.1.3 SQLAlchemy==0.7.7 Twisted==12.0.0 psycopg2==2.4.5 pyOpenSSL==0.13 reportlab==2.5 suds==0.4 wsgiref==0.1.2 zope.interface==3.8.0 pyclamdplus==1.0b2 python-magic==0.4.2 six==1.1.0 python-dateutil==2.1 lxml==2.3.4 python-daemon==1.5.5**
71
72##  Pulling it all together
73
74We use the following helper script to create a new virtualenv environment and deploy our required libraries into this environment:
75
76**#!/bin/bash
77
78# Need two parameters:
79# 1. Which installed python to use
80# 2. Where the virtualenv should be
81if [[ $#!= 2 ]]; then echo "Usage: build-python-env.sh " echo "eg. For the distribution shipped python, you might do: \"build-python-env.sh /usr pyenv-test\"" exit 1 fi ZKDIR=`dirname $(readlink -f $0)` echo $ZKDIR $1/bin/virtualenv $2. $2/bin/activate pip install --requirement=$ZKDIR/libraries/python/python-pip-freeze.txt --find-links=file://$ZKDIR/libraries/python/cache --no-index ** ** ** ** **
82
83
84

---
title: 'Python - virtualenv is your friend'
date: '2012-05-25'
published_at: '2012-05-25T17:25:00.000+10:00'
tags: ['development', 'pip', 'programming', 'python', 'sysadmin', 'virtualev']
author: 'Gavin Jackson'
excerpt: 'My manager (www.nathanr.net) recently shared the following technique that we use to manage our python instances, I thought it was so good, I wanted to share it with you. No longer stress about what ve...'
updated_at: '2012-06-18T12:03:14.113+10:00'
legacy_url: 'http://www.gavinj.net/2012/05/python-virtualenv-is-your-friend.html'
---

[![](https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEicCX1zBSSPc1RezUh3I15Tq9aCJWJJMZwWzBcQaRfLK4C_qLIXLcLq9mpQZ0cF-xSAwBTZJaJzjL1X60swoH4UARzndR6Ee7IQYoPnu3BIf2rFFiMp9xqwBUR1kubcT8dWqRa56WVXZHA/s1600/woma-python.jpg)](https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEicCX1zBSSPc1RezUh3I15Tq9aCJWJJMZwWzBcQaRfLK4C_qLIXLcLq9mpQZ0cF-xSAwBTZJaJzjL1X60swoH4UARzndR6Ee7IQYoPnu3BIf2rFFiMp9xqwBUR1kubcT8dWqRa56WVXZHA/s1600/woma-python.jpg)

My manager (www.nathanr.net) recently shared the following technique that we use to manage our python instances, I thought it was so good, I wanted to share it with you.

No longer stress about what version of python is installed with your Linux distribution.

Python source + virtualenv + pip is your savior.

**Virtualenv** ([http://pypi.python.org/pypi/virtualenv](http://pypi.python.org/pypi/virtualenv)) is a bit like a vm for a python install, it takes a vanilla python build (typically from source) and allows you to "activate" it (replacing your default python binaries and libraries with the ones within the virtualenv instance.

This has the benefit of freeing you from the outdated python binaries found in most Linux distributions, and allowing you to tailor specific library configurations for each environment.

**Pip** ([http://pypi.python.org/pypi/pip/1.1](http://pypi.python.org/pypi/pip/1.1)) is a tool for installing and managing python packages (a bit like CPAN for python). We can use this within the virtualenv instance to manage our installed libraries.

By default it queries the online python package index (PyPI - [http://pypi.python.org/pypi](http://pypi.python.org/pypi)).

##  Building python and installing virtualenv into it

Before running the following, you will need to download the python source code (http://www.python.org/download/) and virtualenv (http://pypi.python.org/pypi/virtualenv).

**#install dev libraries (SLES SP2)** **#SUSE build tool pattern (same as build-essential on Ubuntu) ** zypper install zypper install -t pattern Basis-Devel zypper install libbz2-devel zypper install readline-devel zypper install ncurses-devel zypper install libopenssl-devel zypper install libxslt-devel

**#build python**

tar -xjf Python-2.7.3.tar.bz2

cd Python-2.7.3/

./configure --prefix=/usr/local/python-2.7.3

make

make install

ln -s python-2.7.3 python-2.7

ln -s python-2.7 python

**#install virtualenv ** tar -xzf ~/Downloads/virtualenv-1.7.1.2.tar.gz cd virtualenv-1.7.1.2/

/usr/local/python/bin/python setup.py install

Ok, we now have a Python 2.7 install with virtualenv in it. Now, create a virtualenv for our development and make it your working virtual environment:

**#set up virtualenv in environment-name** /usr/local/python/bin/virtualenv environment-name New python executable in environment-name/bin/python Installing setuptools............done. Installing pip...............done.

**#activate the virtual environment ** cd environment-name/. bin/activate (environment-name)user@host:~/tmp/environment-name>

##  Installing python libraries into the virtualenv instance

Note that some libraries in the online pip repositories are somewhat dated, to ensure that we have the correct versions we maintain a local cache (which we keep checked into our mercurial repository).

**pip install --requirement=~/work/libraries/python/python-pip-freeze.txt --find-links=file://$HOME/work/libraries/python/cache --no-index**

Where the cache folder contains the tar.gz files for all of the libraries contained in the python-pip-freeze.txt file (this file can easily be regenerated using the **pip freeze** command).

In our case the file contains the following libs:

**CherryPy==3.2.2 Mako==0.7.0 MarkupSafe==0.15 PyXB-base==1.1.3 SQLAlchemy==0.7.7 Twisted==12.0.0 psycopg2==2.4.5 pyOpenSSL==0.13 reportlab==2.5 suds==0.4 wsgiref==0.1.2 zope.interface==3.8.0 pyclamdplus==1.0b2 python-magic==0.4.2 six==1.1.0 python-dateutil==2.1 lxml==2.3.4 python-daemon==1.5.5**

##  Pulling it all together

We use the following helper script to create a new virtualenv environment and deploy our required libraries into this environment:

**#!/bin/bash

# Need two parameters:
# 1. Which installed python to use
# 2. Where the virtualenv should be
if [[ $#!= 2 ]]; then echo "Usage: build-python-env.sh " echo "eg. For the distribution shipped python, you might do: \"build-python-env.sh /usr pyenv-test\"" exit 1 fi ZKDIR=`dirname $(readlink -f $0)` echo $ZKDIR $1/bin/virtualenv $2. $2/bin/activate pip install --requirement=$ZKDIR/libraries/python/python-pip-freeze.txt --find-links=file://$ZKDIR/libraries/python/cache --no-index ** ** ** ** **