How to Compile Python from Source

Here we are going compile python from source. I assume you have a clean installation of Debian testing. Here are few packages required for compilation.

apt-get -y install build-essential zlib1g-dev libbz2-dev \
    libncurses5-dev libreadline-gplv2-dev libsqlite3-dev \
    libssl-dev libgdbm-dev

Once above installation is complete, download python source code from here: http://www.python.org/ftp/python/. Suppose you choose to download python 2.5.2.

cd /usr/local/src
wget http://www.python.org/ftp/python/2.5.2/Python-2.5.2.tar.bz2
tar xjf Python-2.5.2.tar.bz2
cd Python-2.5.2

Since most of libraries in Debian moved from /usr/lib to /usr/lib/i386-linux-gnu we need create symbolic links in old location so the build scripts can find them all. This is far easier than specify a valid library location for each case. Here are links:

ln -s /usr/lib/i386-linux-gnu/libssl.so \
    /usr/lib/libssl.so
ln -s /usr/lib/i386-linux-gnu/libcrypt.so \
    /usr/lib/libcrypt.so
ln -s /usr/lib/i386-linux-gnu/libcrypto.so  \
    /usr/lib/libcrypto.so
ln -s /usr/lib/i386-linux-gnu/libbz2.so  \
    /usr/lib/libbz2.so
ln -s /usr/lib/i386-linux-gnu/libgdbm.so  \
    /usr/lib/libgdbm.so
ln -s /usr/lib/i386-linux-gnu/libcurses.so  \
    /usr/lib/libcurses.so
ln -s /usr/lib/i386-linux-gnu/libz.so  \
    /usr/lib/libz.so
ln -s /usr/lib/i386-linux-gnu/libsqlite3.so  \
    /usr/lib/libsqlite3.so

Compilation

Before we start compile we need to configure it first. You can run it with all defaults (this will install python to /usr/local/).

./configure

Or you can specify some other location:

./configure --prefix=/usr/local

The configuration process take few seconds. Next issue make command to actually compile it (this may take few minutes). The -s option prints warning only and -j 2 utilizes 2 CPU cores during the compilation).

make -s -j 2

You can optionally test it before installing with:

make test

or run specific tests of your interest:

./python Lib/test/test_hashlib.py

Install

make install

Python executable should be located at /usr/local/bin/python2.5.

Extra Packages

While python is perfectly working at this moment you might need install some extra packages (e.g. virtualenv) with easy_install.

wget -O - -q http://python-distribute.org/distribute_setup.py | python2.5
easy_install-2.5 virtualenv

This way you can install as many python versions as you like.

Troubleshooting

While working with some third party package (e.g. django) you got the following error:

ImportError: ...undefined symbol: PyUnicodeUCS2_Replace

There reason is described here. You have to re-configure the python:

./configure --enable-unicode=ucs4

and build/install it again.

How to install XMPP (Jabber) IM server in Debian

XMPP is an open-standard communications protocol for message-oriented middleware based on XML (originally named Jabber). Prerequisites:

• XMPP domain: dev.local
• ejabberd server name: im1.dev.local
• Administrative account: admin@dev.local

In order to install the ejabberd IM server in Debian:

apt-get -y install ejabberd

Add administrative account (user, host, password):

ejabberdctl register admin dev.local P@ssw0rd

Configuration

You will need a certificate file for your domain. While obtaining self signed certificate, please ensure:

1. Common Name is the XMPP domain name, e.g. dev.local.
2. Resulting pem file has both key and certificate

Take a look how to can create a self signed certificate here and add it to trusted certificates here. All configuration is stored in /etc/ejabberd/ejabberd.cfg file.

%% Admin user
{acl, admin, {user, "", "dev.local"}}.

%% Hostname (The list of domains we are going to serve)
{hosts, ["dev.local"]}.

...

%% domain_certfile: Specify a different certificate for 
%% each served hostname    .
{domain_certfile, "dev.local", "/etc/ejabberd/dev.local.pem"}.

...

%% To enable in-band registration, replace 'deny' with 
%% 'allow'. This let users create IM account from theirs
%% client applications.
{access, register, [{allow, all}]}.

XMPP DNS Discovery

You need to add the following records to your dns server:

$ORIGIN _tcp.dev.local.
$TTL 900    ; 15 minutes
_jabber         SRV 5 0 5269 im1.dev.local.
_xmpp-client    SRV 5 0 5222 im1.dev.local.
_xmpp-server    SRV 5 0 5269 im1.dev.local.

If you are editing zone of your dynamic dns server consider have a look here.

How to Install django on Debian using nginx uwsgi

Here we are going install a basic django application using uwsgi application server and nginx as load balancer. Requirements:

• django code location: /usr/local/lib
• django project: hello

Let install few packages we need:

apt-get -y install nginx uwsgi \
    uwsgi-plugin-python python-django

Create a simple django application:

cd /usr/local/lib
django-admin startproject hello

uWSGI

Create uWSGI configuration (file /etc/uwsgi/apps-available/hello.yaml):

uwsgi:
    uid: www-data
    gid: www-data
    socket: /tmp/uwsgi-hello.sock
    plugins: http, python
    module: django.core.handlers.wsgi:WSGIHandler()
    pythonpath: /usr/local/lib
    chdir: /usr/local/lib/hello
    env: DJANGO_SETTINGS_MODULE=hello.settings 

Enable this configuration and restart uwsgi server:

ln -s /etc/uwsgi/apps-available/hello.yaml \
    /etc/uwsgi/apps-enabled/hello.yaml
/etc/init.d/uwsgi restart

nginx

Create nginx site configuration (file /etc/nginx/sites-available/hello):

upstream backend {
    server unix:///tmp/uwsgi-hello.sock;
}

server {
    location / { 
        uwsgi_pass  backend;
        include     uwsgi_params;
    }   

    location /static/admin/ {
        alias /usr/lib/pymodules/python2.6/django/contrib/admin/media/;
        access_log off;
        expires 7d;
    }

    location  /static/ {
        alias  /usr/local/lib/hello/static/;
        access_log off;
        expires 7d;
    }
}

Remove default site, enable this configuration and restart nginx server:

rm /etc/nginx/sites-enabled/default
ln -s /etc/nginx/sites-available/hello \
    /etc/nginx/sites-enabled/hello
/etc/init.d/nginx restart

nginx - uwsgi_cache

Web sites that serve almost static content (updated let say every 15 mins) can benefit by utilizing content caching:

   ...
uwsgi_cache_path   /var/cache/hello levels=1:2
                   keys_zone=NAME:15m
                   inactive=5m;
server {
   location / {
       uwsgi_pass  backend;
       include     uwsgi_params;
       uwsgi_cache   NAME;
       uwsgi_cache_valid   200 302 15m;
       uwsgi_cache_key $request_uri;
       expires 15m;
   }
   ...
}

If you experience performance issue with content rendering, overhead of framework internal time, have a willing to fine control content caching consider take a look at the following post and this one.

Simple Load Balancing with nginx on Debian

Suppose you need distribute http workload across multiple computers/processes. Here we will be using nginx to implement load balancing (using round-robin scheduling). Let start by installing nginx package:

apt-get -y install nginx

Place the following into /etc/nginx/sites-available/my-site:

upstream backend {
    // every 7 requests: 5 requests to s1 server 
    // and one request to the second and the third
    server s1 weight=5;
    // after 3 unsuccessful attempts within 
    // the 30 seconds s2 is considered 
    // inoperative
    server unix:/tmp/s2 max_fails=3 fail_timeout=30s;
    server 127.0.0.1:8080;
}   
 
server {
    location / {
       proxy_pass http://backend;
    }   
}  

Now disable default site configuration, enable a new one and restart nginx:

rm /etc/nginx/sites-enabled/default
ln -s /etc/nginx/sites-available/my-site \
    /etc/nginx/sites-enabled/my-site
/etc/init.d/nginx restart

Read more about nginx upstream module here.

How to Serve Django Static Files by Apache

Serving static files out from python is not good idea since there is more effective way to do that. Why not let apache do this? Suppose you followed one of two previous posts that let you configure apache to server django application using mod_python or mod_wsgi. The idea is simple, we define an alias for the media directory of the django application and set it handler to none. Here is a partial example of apache site configuration file:

    ...

    Alias "/static/admin/" "/usr/lib/pymodules/python2.7/django/contrib/admin/media/"
    <Location "/static/admin/">
        SetHandler None
    </Location>

    Alias "/static/" "/usr/local/lib/hello/static/"
    <Location "/static/">
        SetHandler None
    </Location>

And finally let turn on static content expiration.

    ...

    <IfModule mod_expires.c>
        <FilesMatch "\.(jpg|gif|png|css|js)$">
            ExpiresActive on
            ExpiresDefault "access plus 7 days"
        </FilesMatch>
    </IfModule>

Enable expires module and restart apache2.

a2enmod expires
/etc/init.d/apache2 restart