Building Python EGGs in Batch

While working with different versions of python you deal with libraries that comes with source code and doesn't provide a compiled python egg file.

Method1: Offline

Here is a script that develops eggs per python version you have installed for all packages found in lib directory (you must create that directory at the same place where the script below and drop downloaded source distribution archives in).

#!/bin/sh

rm -rf eggs
mkdir eggs
for p in /usr/local/bin/python?.?
do
    echo -n $p
    rm -rf env/
    virtualenv --no-site-packages --python=$p -q env > /dev/null 2>/dev/null
    echo -n .
    env/bin/easy_install -i lib -U -O2 -z distribute > /dev/null 2>/dev/null
    for f in lib/*  # packages to build
    do
        echo -n .
        env/bin/easy_install -i lib -O2 -z $f > /dev/null 2>/dev/null
    done
    cp env/lib/python*.*/site-packages/*.egg eggs/ 2>/dev/null
    echo done
done
rm -rf env/
rm -f eggs/setuptools* eggs/distribute*

Run script with:

./sh make_eggs.sh

Python eggs are now in eggs directory.

Method2: Online

In this case the script relies on internet connection and downloads sources right from pypi.

#!/bin/sh

rm -rf eggs
mkdir eggs
for p in /usr/local/bin/python?.?
do
    echo -n $p
    rm -rf env/
    virtualenv --no-site-packages --python=$p -q env > /dev/null 2>/dev/null
    echo -n .
    env/bin/easy_install -U -O2 -z distribute > /dev/null 2>/dev/null
    for f in $@
    do
        echo -n .
        env/bin/easy_install -O2 -z $f > /dev/null 2>/dev/null
    done
    cp env/lib/python*.*/site-packages/*.egg eggs/ 2>/dev/null
    echo done
done
rm -rf env/
rm -f eggs/setuptools* eggs/distribute*

Run script by passing packages you want to build:

./sh make_eggs.sh pycrypto lxml

Python eggs are now in eggs directory.

Using Buildbot with Multiple Projects

The BuildBot is a system to automate the compile/test cycle required to validate code changes. Here we are going setup buildbot master and slaves with the following requirements:

1. We have serveral projects: project1, project2, project3
2. These projects live under Mercurial (e.g. hosted at bitbucket.org) with base url https://scm.dev.local/hg/ followed by project name (e.g. https://scm.dev.local/hg/project1)
3. We should be able run at least 2 builds in parallel (this is where we can use buildbot slaves: linux-slave1 and linux-slave2)
4. The build should trigger automatically if the changes detected every 30 minutes during working day.
5. The builder should execute make all that ultimatelly does everything we need to verify integrity.
6. The builder should be checked agains certain python versions: 2.4, 2.5, 2.6, 2.7 and 3.2

How to chroot SFTP (Secure File Transfer)

SFTP (SSH File Transfer Protocol) is a network protocol that provides file transfer functionality over reliable data stream. It has nothing related with old ftp protocol however it treated as a secure replacement. Here we are going to achieve the following:

1. Service root directory: /srv/sftp
2. Each user must have isolated sftp location, e.g. /srv/sftp/user1
3. User top level directory include directories: files, archive
4. User session is chrooted
5. User is limited to sftp only, no shell, no ssh access

If you have ssh installed, you have sftp. Just in case:

apt-get -y install ssh

SSH Configuration for SFTP

You need to ensure the sftp subsystem is enabled in ssh. We are going to use internal-sftp implementation, that is in-process ssh subsystem (file /etc/ssh/sshd_config):

#Subsystem sftp /usr/lib/openssh/sftp-server
Subsystem sftp internal-sftp

Let use sftp group to identify users for sftp. Here is a matching rule for ssh, add it at the end of /etc/ssh/sshd_config file:

Match group sftp
    ChrootDirectory /srv/sftp/%u
    X11Forwarding no
    AllowTcpForwarding no
    MaxAuthTries 2
    ForceCommand internal-sftp

Restart ssh so the changes take place:

/etc/init.d/ssh restart

Users

Let create a security group for our sftp users:

groupadd sftp

Here is a script that does the rest (file sftp-add.sh).

#!/bin/bash

sftproot=/srv/sftp

genpasswd() {
    local l=$1
    [ "$l" == "" ] && l=20
    tr -dc A-Za-z0-9_ < /dev/urandom \
        | head -c ${l} | xargs
}

if [ -z $1 ]; then 
    echo "Usage $0 username"
    exit 1
fi

# 1. User is created with home directory set to /, 
# this is the directory sftp change once chroot.
# 2. User added to group sftp.
# 3. Do not create home directory.
# 4. User has no shell, ssh login impossible.
useradd -d / -G sftp -M -s /bin/false $1

echo "Auto generated password:"
genpasswd
passwd $1

mkdir -p $sftproot/$1/{files,archive}
# Chroot directory must be owned by root
chown root:$1 $sftproot/$1 
# User has read-only access
chmod -R 750 $sftproot/$1
# User owns everything below chroot directory
chown $1:$1 $sftproot/$1/*

Just invoke it this way:

./sftp-add.sh user1

Now you should be able use sftp.

How to Compile Python from Source

Here we are going compile python from source. I assume you have a clean installation of Debian testing. Here are few packages required for compilation.

apt-get -y install build-essential zlib1g-dev libbz2-dev \
    libncurses5-dev libreadline-gplv2-dev libsqlite3-dev \
    libssl-dev libgdbm-dev

Once above installation is complete, download python source code from here: http://www.python.org/ftp/python/. Suppose you choose to download python 2.5.2.

cd /usr/local/src
wget http://www.python.org/ftp/python/2.5.2/Python-2.5.2.tar.bz2
tar xjf Python-2.5.2.tar.bz2
cd Python-2.5.2

Since most of libraries in Debian moved from /usr/lib to /usr/lib/i386-linux-gnu we need create symbolic links in old location so the build scripts can find them all. This is far easier than specify a valid library location for each case. Here are links:

ln -s /usr/lib/i386-linux-gnu/libssl.so \
    /usr/lib/libssl.so
ln -s /usr/lib/i386-linux-gnu/libcrypt.so \
    /usr/lib/libcrypt.so
ln -s /usr/lib/i386-linux-gnu/libcrypto.so  \
    /usr/lib/libcrypto.so
ln -s /usr/lib/i386-linux-gnu/libbz2.so  \
    /usr/lib/libbz2.so
ln -s /usr/lib/i386-linux-gnu/libgdbm.so  \
    /usr/lib/libgdbm.so
ln -s /usr/lib/i386-linux-gnu/libcurses.so  \
    /usr/lib/libcurses.so
ln -s /usr/lib/i386-linux-gnu/libz.so  \
    /usr/lib/libz.so
ln -s /usr/lib/i386-linux-gnu/libsqlite3.so  \
    /usr/lib/libsqlite3.so

Compilation

Before we start compile we need to configure it first. You can run it with all defaults (this will install python to /usr/local/).

./configure

Or you can specify some other location:

./configure --prefix=/usr/local

The configuration process take few seconds. Next issue make command to actually compile it (this may take few minutes). The -s option prints warning only and -j 2 utilizes 2 CPU cores during the compilation).

make -s -j 2

You can optionally test it before installing with:

make test

or run specific tests of your interest:

./python Lib/test/test_hashlib.py

Install

make install

Python executable should be located at /usr/local/bin/python2.5.

Extra Packages

While python is perfectly working at this moment you might need install some extra packages (e.g. virtualenv) with easy_install.

wget -O - -q http://python-distribute.org/distribute_setup.py | python2.5
easy_install-2.5 virtualenv

This way you can install as many python versions as you like.

Troubleshooting

While working with some third party package (e.g. django) you got the following error:

ImportError: ...undefined symbol: PyUnicodeUCS2_Replace

There reason is described here. You have to re-configure the python:

./configure --enable-unicode=ucs4

and build/install it again.

How to install XMPP (Jabber) IM server in Debian

XMPP is an open-standard communications protocol for message-oriented middleware based on XML (originally named Jabber). Prerequisites:

• XMPP domain: dev.local
• ejabberd server name: im1.dev.local
• Administrative account: admin@dev.local

In order to install the ejabberd IM server in Debian:

apt-get -y install ejabberd

Add administrative account (user, host, password):

ejabberdctl register admin dev.local P@ssw0rd

Configuration

You will need a certificate file for your domain. While obtaining self signed certificate, please ensure:

1. Common Name is the XMPP domain name, e.g. dev.local.
2. Resulting pem file has both key and certificate

Take a look how to can create a self signed certificate here and add it to trusted certificates here. All configuration is stored in /etc/ejabberd/ejabberd.cfg file.

%% Admin user
{acl, admin, {user, "", "dev.local"}}.

%% Hostname (The list of domains we are going to serve)
{hosts, ["dev.local"]}.

...

%% domain_certfile: Specify a different certificate for 
%% each served hostname    .
{domain_certfile, "dev.local", "/etc/ejabberd/dev.local.pem"}.

...

%% To enable in-band registration, replace 'deny' with 
%% 'allow'. This let users create IM account from theirs
%% client applications.
{access, register, [{allow, all}]}.

XMPP DNS Discovery

You need to add the following records to your dns server:

$ORIGIN _tcp.dev.local.
$TTL 900    ; 15 minutes
_jabber         SRV 5 0 5269 im1.dev.local.
_xmpp-client    SRV 5 0 5222 im1.dev.local.
_xmpp-server    SRV 5 0 5269 im1.dev.local.

If you are editing zone of your dynamic dns server consider have a look here.