How to share network connection with iptables

While working in isolated environment you might need to share your machine internet connection with other computers or virtual machines (e.g. host only network in VirtualBox). Ensure you have iptables installed.

apt-get install iptables

There are two thing we need to do: let kernel know that it is permitted to forward network traffic.

echo "sysctl net.ipv4.ip_forward=1" >> \
    /etc/sysctl.d/ip_forward.conf

and apply masquerading for the interface that we what to share (eth0), add the following line to /etc/rc.local:

iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE

You have to restart your computer so the changes take place during the next system boot.

How to rewrite all to https in nginx

Here we are going redirect all http traffic to https with nginx. I suppose you already have nginx installed, if not have a look here. We will store SSL certificates in /etc/nginx/ssl directory.

cd /etc/nginx
mkdir ssl
openssl req -new -x509 -sha256 -days 9999 -nodes \
    -out ssl/cert.pem -keyout ssl/cert.key
chown -R www-data:www-data ssl
chmod -R 700 ssl

Here is nginx configuration:

upstream backend {
    server 127.0.0.1:8080;
}

server {
    listen  *:80;
    return 301 https://$host$request_uri;
    #if ( $scheme = "http" ) {
    #    rewrite  ^/(.*)$  https://$host/$1 permanent;
    #}
}

server {
    listen  *:443;

    ssl on;
    ssl_protocols TLSv1;
    ssl_certificate /etc/nginx/ssl/cert.pem;
    ssl_certificate_key /etc/nginx/ssl/cert.key;

    location / {
        proxy_pass http://backend;
    }
}

You have to reload nginx so the changes take place.

Building Python EGGs in Batch

While working with different versions of python you deal with libraries that comes with source code and doesn't provide a compiled python egg file.

Method1: Offline

Here is a script that develops eggs per python version you have installed for all packages found in lib directory (you must create that directory at the same place where the script below and drop downloaded source distribution archives in).

#!/bin/sh

rm -rf eggs
mkdir eggs
for p in /usr/local/bin/python?.?
do
    echo -n $p
    rm -rf env/
    virtualenv --no-site-packages --python=$p -q env > /dev/null 2>/dev/null
    echo -n .
    env/bin/easy_install -i lib -U -O2 -z distribute > /dev/null 2>/dev/null
    for f in lib/*  # packages to build
    do
        echo -n .
        env/bin/easy_install -i lib -O2 -z $f > /dev/null 2>/dev/null
    done
    cp env/lib/python*.*/site-packages/*.egg eggs/ 2>/dev/null
    echo done
done
rm -rf env/
rm -f eggs/setuptools* eggs/distribute*

Run script with:

./sh make_eggs.sh

Python eggs are now in eggs directory.

Method2: Online

In this case the script relies on internet connection and downloads sources right from pypi.

#!/bin/sh

rm -rf eggs
mkdir eggs
for p in /usr/local/bin/python?.?
do
    echo -n $p
    rm -rf env/
    virtualenv --no-site-packages --python=$p -q env > /dev/null 2>/dev/null
    echo -n .
    env/bin/easy_install -U -O2 -z distribute > /dev/null 2>/dev/null
    for f in $@
    do
        echo -n .
        env/bin/easy_install -O2 -z $f > /dev/null 2>/dev/null
    done
    cp env/lib/python*.*/site-packages/*.egg eggs/ 2>/dev/null
    echo done
done
rm -rf env/
rm -f eggs/setuptools* eggs/distribute*

Run script by passing packages you want to build:

./sh make_eggs.sh pycrypto lxml

Python eggs are now in eggs directory.

Using Buildbot with Multiple Projects

The BuildBot is a system to automate the compile/test cycle required to validate code changes. Here we are going setup buildbot master and slaves with the following requirements:

1. We have serveral projects: project1, project2, project3
2. These projects live under Mercurial (e.g. hosted at bitbucket.org) with base url https://scm.dev.local/hg/ followed by project name (e.g. https://scm.dev.local/hg/project1)
3. We should be able run at least 2 builds in parallel (this is where we can use buildbot slaves: linux-slave1 and linux-slave2)
4. The build should trigger automatically if the changes detected every 30 minutes during working day.
5. The builder should execute make all that ultimatelly does everything we need to verify integrity.
6. The builder should be checked agains certain python versions: 2.4, 2.5, 2.6, 2.7 and 3.2

How to chroot SFTP (Secure File Transfer)

SFTP (SSH File Transfer Protocol) is a network protocol that provides file transfer functionality over reliable data stream. It has nothing related with old ftp protocol however it treated as a secure replacement. Here we are going to achieve the following:

1. Service root directory: /srv/sftp
2. Each user must have isolated sftp location, e.g. /srv/sftp/user1
3. User top level directory include directories: files, archive
4. User session is chrooted
5. User is limited to sftp only, no shell, no ssh access

If you have ssh installed, you have sftp. Just in case:

apt-get -y install ssh

SSH Configuration for SFTP

You need to ensure the sftp subsystem is enabled in ssh. We are going to use internal-sftp implementation, that is in-process ssh subsystem (file /etc/ssh/sshd_config):

#Subsystem sftp /usr/lib/openssh/sftp-server
Subsystem sftp internal-sftp

Let use sftp group to identify users for sftp. Here is a matching rule for ssh, add it at the end of /etc/ssh/sshd_config file:

Match group sftp
    ChrootDirectory /srv/sftp/%u
    X11Forwarding no
    AllowTcpForwarding no
    MaxAuthTries 2
    ForceCommand internal-sftp

Restart ssh so the changes take place:

/etc/init.d/ssh restart

Users

Let create a security group for our sftp users:

groupadd sftp

Here is a script that does the rest (file sftp-add.sh).

#!/bin/bash

sftproot=/srv/sftp

genpasswd() {
    local l=$1
    [ "$l" == "" ] && l=20
    tr -dc A-Za-z0-9_ < /dev/urandom \
        | head -c ${l} | xargs
}

if [ -z $1 ]; then 
    echo "Usage $0 username"
    exit 1
fi

# 1. User is created with home directory set to /, 
# this is the directory sftp change once chroot.
# 2. User added to group sftp.
# 3. Do not create home directory.
# 4. User has no shell, ssh login impossible.
useradd -d / -G sftp -M -s /bin/false $1

echo "Auto generated password:"
genpasswd
passwd $1

mkdir -p $sftproot/$1/{files,archive}
# Chroot directory must be owned by root
chown root:$1 $sftproot/$1 
# User has read-only access
chmod -R 750 $sftproot/$1
# User owns everything below chroot directory
chown $1:$1 $sftproot/$1/*

Just invoke it this way:

./sftp-add.sh user1

Now you should be able use sftp.