Python packaging with setuptools

We are going create a python egg distribution for a simple helloworld module.

Install tools

Let install two tools we need (consider switch to debian sid repository in order to get latest version of the tools):

deby:~# apt-get -y install python-setuptools python-virtualenv

We are going to work in isolated environment:

user1@deby:~$ virtualenv devenv
New python executable in devenv/bin/python
Installing setuptools............done.
user1@deby:~$ cd devenv/
user1@deby:~/devenv$

Directory structure

Suppose our directory structure looks this way:

~/devenv/
`-- trunk/
    |-- src/
    |   `-- greatings/
    |       |-- __init__.py
    |       |-- helloworld.py
    |       `-- tests/
    |           |-- __init__.py
    |           `-- test_helloworld.py
    `-- README.txt

We are going to place all our python code in src directory.

mkdir -p trunk/src/greatings/tests

Code

The file __init__.py is left empty and makes greating a python package. Here is content of helloworld.py (note that we are using docunits in order to demonstrate dependencies later, main function will be an entry point of our script):

import sys

def say():
    """
    >>> say()
    'hello world'
    """
    return 'hello world'

def main():
    print(say())
    return 0

if __name__ == '__main__':
    sys.exit(main())

The tests will be combined into test suites so they can be easier added for testing our setup later. Here is test_helloworld.py:

from greatings import helloworld
import unittest

class HelloworldTestCase(unittest.TestCase):

    def test_say(self):
        assert 'hello world' == helloworld.say()


def suite():
    loader = unittest.TestLoader()
    suite = unittest.TestSuite()
    suite.addTest(loader.loadTestsFromTestCase(HelloworldTestCase))
    return suite

if __name__ == '__main__':
    unittest.TextTestRunner(verbosity=2).run(suite())

Here is tests package __init__.py file:

from greatings import helloworld
import test_helloworld

def suite():
    import unittest
    import doctest
    suite = unittest.TestSuite()
    suite.addTests(doctest.DocTestSuite(helloworld))
    suite.addTests(test_helloworld.suite())
    return suite

if __name__ == '__main__':
    unittest.TextTestRunner(verbosity=2).run(suite())

Setup files

Here is out ~/devenv/trunk/setup.py file:

import os
from setuptools import setup, find_packages

setup(
    name = 'greatings',
    version = '0.1',

    # Package structure
    #
    # find_packages searches through a set of directories 
    # looking for packages
    packages = find_packages('src', exclude = ['ez_setup',
        '*.tests', '*.tests.*', 'tests.*', 'tests']),
    # package_dir directive maps package names to directories.
    # package_name:package_directory
    package_dir = {'': 'src'},

    # Not all packages are capable of running in compressed form, 
    # because they may expect to be able to access either source 
    # code or data files as normal operating system files.
    zip_safe = True,

    # Entry points
    #
    # install the executable
    entry_points = {
        'console_scripts': ['helloworld = greatings.helloworld:main']
    },

    # Dependencies
    #
    # Dependency expressions have a package name on the left-hand 
    # side, a version on the right-hand side, and a comparison 
    # operator between them, e.g. == exact version, >= this version
    # or higher
    install_requires = [
        '',
    ],

    # Tests
    #
    # Tests must be wrapped in a unittest test suite by either a
    # function, a TestCase class or method, or a module or package
    # containing TestCase classes. If the named suite is a package,
    # any submodules and subpackages are recursively added to the
    # overall test suite.
    test_suite = 'greatings.tests.suite',
    # Download dependencies in the current directory
    tests_require = 'docutils >= 0.6',

    # Meta information
    #
    author = 'Me',
    author_email = 'my@e-mail.com',
    description = 'A sample hello world application',
    url = 'http://mindref.blogspot.com'
)

And configuration (file ~/devenv/setup.cfg):

[global]
# Just silently do your job
quiet = 1

[easy_install]
# Where we are going to look for thrirdparty dependencies
find_links = thirdparty

[build_py]
# No optimization for now
optimize = 0
# Force build everything?
force = True

[egg_info]
# We are doing development build
tag_build = dev
# Do we want to have date in file name?
tag_date = 0
# Add svn revision to the file name
tag_svn_revision = 1

[bdist_egg]
# We do not want to distribute binary with source code
exclude-source-files = True

[rotate]
# Keep only last 10 eggs, clean up older
match = .egg
keep = 10

Third party dependencies

The next thing, we would like keep thirdparty dependencies (e.g. docutils) in a separate folder so each time we build the project it doesn't download dependencies from internet instead look at our folder, so we always have a proper version there. So let create directory thirdparty at the same level as src and download there docutils.

user1@deby:~/devenv/trunk$ mkdir thirdparty
user1@deby:~/devenv/trunk$ wget -P thirdparty/ http://pypi.python.org\
/packages/source/d/docutils/docutils-0.6.tar.gz

The directory structure should look like this:

~/devenv/
`-- trunk/
    |-- src/
    |   `-- greatings/
    |        ...
    `-- thirdparty/
        `-- docutils-0.6.tar.gz

Install docutils that we downloaded into our environment:

../bin/easy_install thirdparty/*

Test, EGG, Source

Let ensure tests are passed:

master@deby:~/devenv/trunk$ ../bin/python setup.py test
..
---------------------------------------------
Ran 2 tests in 0.015s

OK

Here is how to create a binary distribution in egg format (look outcome at ~/devenv/trunk/dist directory):

../bin/python setup.py bdist_egg

... and source code:

../bin/python setup.py sdist

Both source and binary distributions are in ~/devenv/trunk/dist directory.

user1@deby:~/devenv/trunk$ ls dist/
greatings-0.1dev-py2.6.egg  greatings-0.1dev.tar.gz

Version control

Before adding the project to version control (e.g. svn), ensure the following directories are ignored:

1. build
2. dist
3. src/greatings.egg-info

That's it.

How to disable autoindent in VIM

When copy/paste from one source to the other (e.g. sample code), vim auto-indent makes too much indentation, so you want to quickly turn it off. Here is a command:

:setl noai nocin nosi inde=

Here is a mapping (add to file ~/.vimrc):

" Disable autoindent in VIM
nnoremap <F8> :setl noai nocin nosi inde=<CR>

Alternatively, you can use:

" Turning off auto indent when pasting text into vim
set pastetoggle=<F8>

So now, before pasting something in, you press F8 to disable auto-indent.

Sync time with external server in Debian

You can synchronize your local computer time with external time servers. You need install ntpdate:

apt-get -y install ntpdate

Try update your time manually:

deby:~# ntpdate pool.ntp.org
12 Jun 00:04:16 ntpdate[1903]: step time server 62.80.187.114 offset -291.468062 sec

Here is a script that syncs the time with external server, adjusts clock drift and finally sets the hardware clock to current system time (file /usr/local/sbin/sync-time).

#!/bin/bash

server=pool.ntp.org

# Sleep a random amount, not greater than 2 minutes
sleep_time=$(($RANDOM % 120))

echo "Sleeping for $sleep_time seconds..."
/bin/sleep $sleep_time

# Sync the time with external server
echo "Sync time with $server."
/usr/sbin/ntpdate -s $server || exit 1

# Adjusts clock drift
/sbin/hwclock --adjust

# Set the hardware clock to current system time
/sbin/hwclock --systohc

Here we are going to schedule a cron job on system startup (file /usr/local/sbin/sync-time, symbolic link from /etc/cron.d/sync-time):

#
# Regular cron job for time synchronization
#
PATH=/usr/local/sbin
HOME=/
LOG=/dev/null

# Every 23 hours, e.g. 2:11, 13:11, etc
11 */23 * * * root test -x /usr/local/sbin/sync-time && sync-time > $LOG

The next time your system restarts it will automatically synchronizes clock with external server. Please note that the system sync the time each time the network interface is up (see ntpdate script in /etc/network/if-up.d/).