How to build Python deb package from source

You need a newer python version that is not available with the Debian package management system. You can compile it from source and install but that usually takes a lot of time. It would be better to build a deb instead and install it on as many machines as you need. You can use checkinstall for this.

apt-get install checkinstall

The instruction how to compile python from source is here. Once you compiled python here is a simple command that does the rest for you:

checkinstall -y --pkgname=python2.7

Once it finishes you will get a deb file in the current directory. You can install it this way:

dpkg -i python2.7_2.7.6-1_i386.deb

Here is how to remove it:

apt-get remove --purge python2.7

Note, packages built with checkinstall are not updated by OS package management system, thus updates need to be provided manually.

PostgreSQL Streaming Replication Hot Standby

Streaming replication allows a standby server to stay up-to-date with primary. The standby connects to the primary, which streams WAL records to the standby as they're generated.

Hot standby is the term used to describe the ability to connect to the server and run read-only queries.

This method of replication is completely transparent to the client, it doesn't require any changes to database, allows query information from standby server and requires minimum administrative effort.

Python HTTP Client API

The modern web services expose public API to the world and JSON is de-facto standard in this communication. Here is a simple use case that integrates with buildbot public API.

Lets setup a virtual environment and install wheezy.core package:

virtualenv env
env/bin/easy_install wheezy.core

Launch python from virtual environment (env/bin/python) and try this:

>>> from wheezy.core.httpclient import HTTPClient
>>> c = HTTPClient('http://buildbot.buildbot.net/json/')
>>> c.get('project')
200
>>> project = c.json
>>>> str(project.title)
Buildbot

Here is another example that demonstarates etag handling (the second time we request events the server responds with HTTP status code 304, not modified):

>>> base = 'https://api.github.com/repos/python/cpython/'
>>> c = HTTPClient(base)
>>> c.get('events')
200
>>> c.headers['content-encoding']
['gzip']
>>> c.get('events')
304

The HTTPClient supports HTTP(S) GET/HEAD/POST verbs, follows redirects, handles cookies and etags between requests, gzip content decoding.

Keep FreeBSD up to date with subversion

The freebsd ports tree is quite big and sometimes you need just few packages. You can do that with subversion sparse checkouts.

pkg_add -r subversion

Checkout only immediate top level. The list of svn mirrors is here.

svn checkout --depth=immediates \
    http://svn0.eu.freebsd.org/ports/head /usr/ports

Update tree on as needed basis:

cd /usr/ports
# common
svn up --set-depth=infinity Mk Templates Tools/scripts
# portmaster
svn up ports-mgmt/portmaster ports-mgmt/dialog4ports 
# python
svn up lang/python lang/python2 lang/python27 \
    devel/gettext converters/libiconv
# mercurial
svn up devel/mercurial
# python development
svn up devel/py-distribute py-setuptools \
    devel/py-virtualenv
# vim
svn up editors/vim
# subversion
svn up devel/subversion devel/libtool devel/apr1 \
    databases/db42 databases/gdbm databases/sqlite3 \
    textproc/expat2 www/serf

Consider configure /etc/make.conf as advised here and install packages:

make install clean -sC /usr/ports/ports-mgmt/portmaster
portmaster lang/python27 devel/mercurial \
    devel/py-virtualenv editors/vim \
    devel/subversion

The above ports require only 12Mb in /usr/ports and next time ports update takes few seconds.

Convenient Remote Access with SSH Config

If you are working with a lot of remote ssh hosts it becomes hard to remember all that host specific information: username, ip address, identity file, non-standard port or local/remote port forwarding. ssh_config to rescue. Here is a sample to give you an idea (file ~/.ssh/config):

Compression yes
IdentityFile ~/.ssh/id_rsa
LogLevel ERROR
Port 22

Host h1
    HostName 192.168.91.57
    User master
    IdentityFile ~/.ssh/h1.pem

Host db1
    HostName usca45d1.example.com
    User pg
    LocalForward 5432 127.0.0.1:5432

The above configuration let me access those hosts simply by name, e.g.:

ssh h1
scp schema.sql db1:~/

How to manage Git or Mercurial repositories

Managing version control repositories can be a challenge in multi-user environment especially when simplification of user collaboration is your goal. There are usually two primary concerns while considering enterprise deployment for version control repositories: access control and safety of your data. Both are not directly addressed by version control itself, thus a sort of security facade is necessary.

wheezy web: Actors

The application design of wheezy.web application consists of several actors, each playing its unifying role. Actors are grouped into packages, e.g. models, repository, service, web, content, tests, etc. These packages (or layers) shape a subsystem.

The diagram below visualizes relations between actors, boundaries between packages, subsystems and application, giving a better picture of whole:

The sequence diagram explains request processing and interaction between actors:

You can download a pdf copy here.

Thoughts on SQL vs ORM

The question of persistence implementation arise often. I found repository pattern very valuable due to separation of concerns, mediate between domain model and data source (mock, file, database, web service, etc). The database data source is somewhat specific since you can proceed with SQL functions or ORM. Here are some thoughts why you might prefer SQL functions over ORM in your next project:

wheezy web: Quick Start i18n Project

wheezy.web is a lightweight, high performance, high concurrency WSGI web framework with the key features to build modern, efficient web. Here we will use an i18n (multilingual internationalization) project quick start to build a new project.

1. Download quickstart-i18n.zip and extract.
2. Rename extracted directory `quickstart-i18n` to meet your project name, e.g. mysite and open terminal in that directory.

The i18n project quick start comes with Makefile with several handy targets. Setup development environment:

make env

Internationalization is built using gettext, thus you need extract gettext messages and compile `*.po` files that you can find in i18n directory.

make po

Let run test, look at coverage and finally benchmark tests:

$ make test nose-cover benchmark
....................................
..........
Name                 Stmts   Miss  Cover   Missing
--------------------------------------------------
public                   3      0   100%   
public.web               0      0   100%   
public.web.profile       5      0   100%   
public.web.urls         10      0   100%   
public.web.views        26      0   100%   
--------------------------------------------------
TOTAL                   44      0   100%   
-----------------------------------------------------
Ran 10 tests in 0.458s

OK
public: 5 x 1000
baseline throughput change target
  100.0%    5303rps  +0.0% test_root
   94.4%    5005rps  -5.6% test_home
   98.2%    5206rps  -1.8% test_error_400
   99.2%    5258rps  -0.8% test_error_403
   97.9%    5191rps  -2.1% test_error_404
static: 3 x 1000
baseline throughput change target
  100.0%    5172rps  +0.0% test_static_files
   47.4%    2453rps -52.6% test_static_file_not_found
   52.5%    2715rps -47.5% test_static_file_forbidden
-----------------------------------------------------
Ran 2 tests in 2.286s

OK

The tests are passed let run it in web browser:

$ make run
Visit http://localhost:8080/

Alternatively you can run it using uwsgi

env/bin/easy_install uwsgi
make uwsgi

Enjoy!

wheezy web: Quick Start Empty Project

wheezy.web is a lightweight, high performance, high concurrency WSGI web framework with the key features to build modern, efficient web. Here we will use an empty project quick start to build a new project.

1. Download quickstart-empty.zip and extract.
2. Rename extracted directory `quickstart-empty` to meet your project name, e.g. mysite and open terminal in that directory.

The empty project quick start comes with Makefile with several handy targets. Setup development environment:

make env

Let run test, look at coverage and finally benchmark tests:

$ make test nose-cover benchmark
....................................
..........
Name                 Stmts   Miss  Cover   Missing
--------------------------------------------------
public                   3      0   100%   
public.web               0      0   100%   
public.web.profile       5      0   100%   
public.web.urls         10      0   100%   
public.web.views        20      0   100%   
--------------------------------------------------
TOTAL                   38      0   100%   
--------------------------------------------------
Ran 10 tests in 0.423s

OK
public: 5 x 1000
baseline throughput change target
  100.0%    5041rps  +0.0% test_root
   81.3%    4097rps -18.7% test_home
   99.4%    5008rps  -0.6% test_error_400
  100.3%    5055rps  +0.3% test_error_403
   98.7%    4975rps  -1.3% test_error_404
static: 3 x 1000
baseline throughput change target
  100.0%    4846rps  +0.0% test_static_files
   49.8%    2415rps -50.2% test_static_file_not_found
   55.0%    2663rps -45.0% test_static_file_forbidden
-----------------------------------------------------
Ran 2 tests in 2.375s

OK

The tests are passed let run it in web browser:

$ make run
Visit http://localhost:8080/

Alternatively you can run it using uwsgi

env/bin/easy_install uwsgi
make uwsgi

Enjoy!