Monday, June 21, 2010

Python static code analysis with pylint

Pylint is a python tool that checks if a module satisfies a coding standard. You can install pylint with easy_install (pylint has dependencies on logilab_common and logilab_astng, so consider download them as well in case of offline install).
easy_install pylint
Running checks is pretty easy:
user1@deby:~/devenv/trunk/src$ ../../bin/pylint greatings
No config file found, using default configuration
************* Module greatings
C:  1: Missing docstring
************* Module greatings.helloworld
C:  1: Missing docstring
C: 10:main: Missing docstring
************* Module greatings.tests
C:  1: Missing docstring
W:  4: Relative import 'test_helloworld', should be 
'greatings.tests.test_helloworld'
W:  7:suite: Redefining name 'suite' from outer scope (line 6)
C:  6:suite: Missing docstring
************* Module greatings.tests.test_helloworld
C:  1: Missing docstring
C:  4:HelloworldTestCase: Missing docstring
C:  6:HelloworldTestCase.test_say: Missing docstring
R:  6:HelloworldTestCase.test_say: Method could be a function
W: 12:suite: Redefining name 'suite' from outer scope (line 10)
C: 10:suite: Missing docstring
  ...
Global evaluation
-----------------
Your code has been rated at 5.94/10
The tool also produces a report that include:
  • Statistics by type
  • External dependencies
  • Duplication
  • Raw metrics
  • Messages by category
  • % errors / warnings by module
  • Messages
If you want to change the default behaviour, you can define options in pylintrc file (use --rcfile option to specify a location). Here is an example.
user1@deby:~/devenv/trunk$ wget -P tools/ http://www.\
logilab.org/cgi-bin/hgwebdir.cgi/pylint/raw-file\
/df8f34aa3dd2/examples/pylintrc
user1@deby:~/devenv/trunk$ cd src/
user1@deby:~/devenv/trunk/src$ ../../bin/pylint \
--rcfile=../tools/pylintrc greatings
Read more about pylint here.

Python static code analysis with pychecker

Python tool PyChecker is used to find typical programming errors in your source code. In order to install it you need download it and install manually (if you try install it using easy_install it will not work, unfortunately). We will assume the following directory structure (note, devenv is virtual environment, you can create it issuing virtualenv devenv).
~/devenv/
`-- trunk/
    |-- src/
    |   `-- greatings/
    |       |-- __init__.py
    |       |-- helloworld.py
    |       `-- tests/
    |           |-- __init__.py
    |           `-- test_helloworld.py
    `-- tools/
Let download pychecker into tools directory and proceed with installation:
user1@deby:~/devenv/trunk$ wget -P tools/ http://downloads.\
sourceforge.net/project/pychecker/pychecker/0.8.18/\
pychecker-0.8.18.tar.gz
...
user1@deby:~/devenv/trunk$  cd tools && tar zxf \
pychecker-0.8.18.tar.gz \
 && cd  pychecker-0.8.18
user1@deby:~/devenv/trunk/tools/pychecker-0.8.18$ \
../../../bin/python setup.py install && cd ../.. && \
rm -rf tools/pychecker-0.8.18
...
The next step you need to "fix" a bit a pychecker file installed into your virtual environment bin directory so the content look like this.
#!/bin/sh

site_packages_dir=$(dirname $0)/../lib/python2.6/site-packages
python $site_packages_dir/pychecker/checker.py "$@"
Since pychecker import all the files it is going to check, you need to start the tool in correct directory, in our case src:
user1@deby:~/devenv/trunk$ cd src/
user1@deby:~/devenv/trunk/src$ ../../bin/pychecker greatings/*.py
Processing module helloworld (greatings/helloworld.py)...
Processing module __init__ (greatings/__init__.py)...

Warnings...

None
Please note that pychecker doesn't drill into sub-packages, so in order to analyze tests issue the following command:
../../bin/pychecker greatings/tests/*.py
If you want to change the default behaviour, you can define options in pycheckrc file (use -F option to specify a location). Here is an example.
user1@deby:~/devenv/trunk$ wget -P tools/ http://pychecker.\
cvs.sourceforge.net/viewvc/pychecker/pychecker/pycheckrc
user1@deby:~/devenv/trunk$ cd src/
user1@deby:~/devenv/trunk/src$ ../../bin/pychecker -F \
../tools/pycheckrc greatings/*.py
Read more about pychecker here and here.

Saturday, June 19, 2010

Python packaging with setuptools

We are going create a python egg distribution for a simple helloworld module.

Install tools

Let install two tools we need (consider switch to debian sid repository in order to get latest version of the tools):
deby:~# apt-get -y install python-setuptools python-virtualenv
We are going to work in isolated environment:
user1@deby:~$ virtualenv devenv
New python executable in devenv/bin/python
Installing setuptools............done.
user1@deby:~$ cd devenv/
user1@deby:~/devenv$

Directory structure

Suppose our directory structure looks this way:
~/devenv/
`-- trunk/
    |-- src/
    |   `-- greatings/
    |       |-- __init__.py
    |       |-- helloworld.py
    |       `-- tests/
    |           |-- __init__.py
    |           `-- test_helloworld.py
    `-- README.txt
We are going to place all our python code in src directory.
mkdir -p trunk/src/greatings/tests

Code

The file __init__.py is left empty and makes greating a python package. Here is content of helloworld.py (note that we are using docunits in order to demonstrate dependencies later, main function will be an entry point of our script):
import sys

def say():
    """
    >>> say()
    'hello world'
    """
    return 'hello world'

def main():
    print(say())
    return 0

if __name__ == '__main__':
    sys.exit(main())
The tests will be combined into test suites so they can be easier added for testing our setup later. Here is test_helloworld.py:
from greatings import helloworld
import unittest

class HelloworldTestCase(unittest.TestCase):

    def test_say(self):
        assert 'hello world' == helloworld.say()


def suite():
    loader = unittest.TestLoader()
    suite = unittest.TestSuite()
    suite.addTest(loader.loadTestsFromTestCase(HelloworldTestCase))
    return suite

if __name__ == '__main__':
    unittest.TextTestRunner(verbosity=2).run(suite())
Here is tests package __init__.py file:
from greatings import helloworld
import test_helloworld

def suite():
    import unittest
    import doctest
    suite = unittest.TestSuite()
    suite.addTests(doctest.DocTestSuite(helloworld))
    suite.addTests(test_helloworld.suite())
    return suite

if __name__ == '__main__':
    unittest.TextTestRunner(verbosity=2).run(suite())

Setup files

Here is out ~/devenv/trunk/setup.py file:
import os
from setuptools import setup, find_packages

setup(
    name = 'greatings',
    version = '0.1',

    # Package structure
    #
    # find_packages searches through a set of directories 
    # looking for packages
    packages = find_packages('src', exclude = ['ez_setup',
        '*.tests', '*.tests.*', 'tests.*', 'tests']),
    # package_dir directive maps package names to directories.
    # package_name:package_directory
    package_dir = {'': 'src'},

    # Not all packages are capable of running in compressed form, 
    # because they may expect to be able to access either source 
    # code or data files as normal operating system files.
    zip_safe = True,

    # Entry points
    #
    # install the executable
    entry_points = {
        'console_scripts': ['helloworld = greatings.helloworld:main']
    },

    # Dependencies
    #
    # Dependency expressions have a package name on the left-hand 
    # side, a version on the right-hand side, and a comparison 
    # operator between them, e.g. == exact version, >= this version
    # or higher
    install_requires = [
        '',
    ],

    # Tests
    #
    # Tests must be wrapped in a unittest test suite by either a
    # function, a TestCase class or method, or a module or package
    # containing TestCase classes. If the named suite is a package,
    # any submodules and subpackages are recursively added to the
    # overall test suite.
    test_suite = 'greatings.tests.suite',
    # Download dependencies in the current directory
    tests_require = 'docutils >= 0.6',

    # Meta information
    #
    author = 'Me',
    author_email = 'my@e-mail.com',
    description = 'A sample hello world application',
    url = 'http://mindref.blogspot.com'
)
And configuration (file ~/devenv/setup.cfg):
[global]
# Just silently do your job
quiet = 1

[easy_install]
# Where we are going to look for thrirdparty dependencies
find_links = thirdparty

[build_py]
# No optimization for now
optimize = 0
# Force build everything?
force = True

[egg_info]
# We are doing development build
tag_build = dev
# Do we want to have date in file name?
tag_date = 0
# Add svn revision to the file name
tag_svn_revision = 1

[bdist_egg]
# We do not want to distribute binary with source code
exclude-source-files = True

[rotate]
# Keep only last 10 eggs, clean up older
match = .egg
keep = 10

Third party dependencies

The next thing, we would like keep thirdparty dependencies (e.g. docutils) in a separate folder so each time we build the project it doesn't download dependencies from internet instead look at our folder, so we always have a proper version there. So let create directory thirdparty at the same level as src and download there docutils.
user1@deby:~/devenv/trunk$ mkdir thirdparty
user1@deby:~/devenv/trunk$ wget -P thirdparty/ http://pypi.python.org\
/packages/source/d/docutils/docutils-0.6.tar.gz
The directory structure should look like this:
~/devenv/
`-- trunk/
    |-- src/
    |   `-- greatings/
    |        ...
    `-- thirdparty/
        `-- docutils-0.6.tar.gz
Install docutils that we downloaded into our environment:
../bin/easy_install thirdparty/*

Test, EGG, Source

Let ensure tests are passed:
master@deby:~/devenv/trunk$ ../bin/python setup.py test
..
---------------------------------------------
Ran 2 tests in 0.015s

OK
Here is how to create a binary distribution in egg format (look outcome at ~/devenv/trunk/dist directory):
../bin/python setup.py bdist_egg
... and source code:
../bin/python setup.py sdist
Both source and binary distributions are in ~/devenv/trunk/dist directory.
user1@deby:~/devenv/trunk$ ls dist/
greatings-0.1dev-py2.6.egg  greatings-0.1dev.tar.gz

Version control

Before adding the project to version control (e.g. svn), ensure the following directories are ignored:
  1. build
  2. dist
  3. src/greatings.egg-info
That's it.

Wednesday, June 16, 2010

How to disable autoindent in VIM

When copy/paste from one source to the other (e.g. sample code), vim auto-indent makes too much indentation, so you want to quickly turn it off. Here is a command:
:setl noai nocin nosi inde=
Here is a mapping (add to file ~/.vimrc):
" Disable autoindent in VIM
nnoremap <F8> :setl noai nocin nosi inde=<CR>
Alternatively, you can use:
" Turning off auto indent when pasting text into vim
set pastetoggle=<F8>
So now, before pasting something in, you press F8 to disable auto-indent.

Monday, June 14, 2010

Sync time with external server in Debian

You can synchronize your local computer time with external time servers. You need install ntpdate:
apt-get -y install ntpdate
Try update your time manually:
deby:~# ntpdate pool.ntp.org
12 Jun 00:04:16 ntpdate[1903]: step time server 62.80.187.114 offset -291.468062 sec
Here is a script that syncs the time with external server, adjusts clock drift and finally sets the hardware clock to current system time (file /usr/local/sbin/sync-time).
#!/bin/bash

server=pool.ntp.org

# Sleep a random amount, not greater than 2 minutes
sleep_time=$(($RANDOM % 120))

echo "Sleeping for $sleep_time seconds..."
/bin/sleep $sleep_time

# Sync the time with external server
echo "Sync time with $server."
/usr/sbin/ntpdate -s $server || exit 1

# Adjusts clock drift
/sbin/hwclock --adjust

# Set the hardware clock to current system time
/sbin/hwclock --systohc

Here we are going to schedule a cron job on system startup (file /usr/local/sbin/sync-time, symbolic link from /etc/cron.d/sync-time):
#
# Regular cron job for time synchronization
#
PATH=/usr/local/sbin
HOME=/
LOG=/dev/null

# Every 23 hours, e.g. 2:11, 13:11, etc
11 */23 * * * root test -x /usr/local/sbin/sync-time && sync-time > $LOG
The next time your system restarts it will automatically synchronizes clock with external server. Please note that the system sync the time each time the network interface is up (see ntpdate script in /etc/network/if-up.d/).