Debian: Clean up dpkg rc packages

rc means that the package is not completely removed.

dpkg --list | grep "^rc" | cut -d " " -f 3 | \
	xargs sudo dpkg --purge

Debian: Possible missing firmware

If you encounter issue like the following, you need to download/update firmware files.

W: Possible missing firmware /lib/firmware/i915

Wheezy Projects Update

All libraries related wheezy.web and wheezy.template have been recently migrated from bitbucket to github.

Python ASGI CLI

Call ASGI Python application module from command line (without an application server), just like CURL. Interested?

Authorization in Microservices Landscape

Overview

This article outlines a high-level design of authentication and authorization options in Azure per a typical SPA use case interacting with multiple microservices. Compares API gateway pattern versus a direct client-to-API communication in microservices environment.

Use Case

A user should be able to request the following information:

1. User profile via Azure Graph API.
2. User documents via SharePoint Online API.
3. Aggregated data from two other internal systems.

                +----------------------+
            +-> | Azure Graph API      |
            |   +----------------------+
+---------+ |   +----------------------+
| browser |-+-> | Azure SharePoint API |
+---------+ |   +----------------------+
            |   +----------------------+     +--------+
            +-> | App API              |---> | API #1 |
                +----------------------+     +--------+
  +---------------------------+       |      +--------+
  |Microsoft Identity Platform|       +----> | API #2 |
  +---------------------------+              +--------+

Asynchronous Python in Web Applications

Asynchronous or non-blocking IO allows worker to process other requests before current request has finished. That is possible in case a processing of current web request is related to an operation that is IO bound.

Prerequisites

What is important to understand about applicability of async pattern for IO bound operations, they must:

1. Utilize the same event loop, in this case worker's event loop can switch to next in the queue (otherwise it is blocked).
2. Have long latency (otherwise you are wasting event loop queue for short requests).

If operation is not IO bound it is considered blocking.

wheezy template: static website generator

Sometimes you need a quick way to generate a static web site and put it up on a server. Here is how you can generate a static website with wheezy.template. Let's suppose the following structure of the site:

 content/
 `- index.html
  - main.html
  shared/
  `- master.html
   snippet/
   `- script.html
    - widget.html

wheezy web: RESTful API Design

In this article we are going to explore a simple RESTful API created with wheezy.web framework. The demo implements a CRUD for tasks. Includes entity validation, content caching with dependencies and functional test cases. The source code is structured with well defined actors (you can read more about it here).

Design

The following convention is used with respect to operation, HTTP method (verb) and URL:

List:   GET    /api/v1/tasks
Add:    POST   /api/v1/tasks
Get:    GET    /api/v1/task/{task_id}
Update: PUT    /api/v1/task/{task_id}
Remove: DELETE /api/v1/task/{task_id}

The task entity consists of the following attributes: task_id, title and status. The source code is available here. Please download before proceeding any further.

wheezy web: deploy nginx + uwsgi + memcached

wheezy.web is a lightweight, high performance, high concurrency WSGI web framework with the key features to build modern, efficient web. Here we will deploy quick start with nginx, uwsgi and memcached to clean debian stable installation.

memcached statistics status

Here is a "top" emulator for memcached statistics:

watch -n 5 "echo stats | socat unix-connect:/var/tmp/memcached.sock -"

The screen will be refreshed every 5 seconds and display results of memcached stats command.

How to build Python deb package from source

You need a newer python version that is not available with the Debian package management system. You can compile it from source and install but that usually takes a lot of time. It would be better to build a deb instead and install it on as many machines as you need. You can use checkinstall for this.

apt-get install checkinstall

The instruction how to compile python from source is here. Once you compiled python here is a simple command that does the rest for you:

checkinstall -y --pkgname=python2.7

Once it finishes you will get a deb file in the current directory. You can install it this way:

dpkg -i python2.7_2.7.6-1_i386.deb

Here is how to remove it:

apt-get remove --purge python2.7

Note, packages built with checkinstall are not updated by OS package management system, thus updates need to be provided manually.

PostgreSQL Streaming Replication Hot Standby

Streaming replication allows a standby server to stay up-to-date with primary. The standby connects to the primary, which streams WAL records to the standby as they're generated.

Hot standby is the term used to describe the ability to connect to the server and run read-only queries.

This method of replication is completely transparent to the client, it doesn't require any changes to database, allows query information from standby server and requires minimum administrative effort.

Python HTTP Client API

The modern web services expose public API to the world and JSON is de-facto standard in this communication. Here is a simple use case that integrates with buildbot public API.

Lets setup a virtual environment and install wheezy.core package:

virtualenv env
env/bin/easy_install wheezy.core

Launch python from virtual environment (env/bin/python) and try this:

>>> from wheezy.core.httpclient import HTTPClient
>>> c = HTTPClient('http://buildbot.buildbot.net/json/')
>>> c.get('project')
200
>>> project = c.json
>>>> str(project.title)
Buildbot

Here is another example that demonstarates etag handling (the second time we request events the server responds with HTTP status code 304, not modified):

>>> base = 'https://api.github.com/repos/python/cpython/'
>>> c = HTTPClient(base)
>>> c.get('events')
200
>>> c.headers['content-encoding']
['gzip']
>>> c.get('events')
304

The HTTPClient supports HTTP(S) GET/HEAD/POST verbs, follows redirects, handles cookies and etags between requests, gzip content decoding.

Keep FreeBSD up to date with subversion

The freebsd ports tree is quite big and sometimes you need just few packages. You can do that with subversion sparse checkouts.

pkg_add -r subversion

Checkout only immediate top level. The list of svn mirrors is here.

svn checkout --depth=immediates \
    http://svn0.eu.freebsd.org/ports/head /usr/ports

Update tree on as needed basis:

cd /usr/ports
# common
svn up --set-depth=infinity Mk Templates Tools/scripts
# portmaster
svn up ports-mgmt/portmaster ports-mgmt/dialog4ports 
# python
svn up lang/python lang/python2 lang/python27 \
    devel/gettext converters/libiconv
# mercurial
svn up devel/mercurial
# python development
svn up devel/py-distribute py-setuptools \
    devel/py-virtualenv
# vim
svn up editors/vim
# subversion
svn up devel/subversion devel/libtool devel/apr1 \
    databases/db42 databases/gdbm databases/sqlite3 \
    textproc/expat2 www/serf

Consider configure /etc/make.conf as advised here and install packages:

make install clean -sC /usr/ports/ports-mgmt/portmaster
portmaster lang/python27 devel/mercurial \
    devel/py-virtualenv editors/vim \
    devel/subversion

The above ports require only 12Mb in /usr/ports and next time ports update takes few seconds.

Convenient Remote Access with SSH Config

If you are working with a lot of remote ssh hosts it becomes hard to remember all that host specific information: username, ip address, identity file, non-standard port or local/remote port forwarding. ssh_config to rescue. Here is a sample to give you an idea (file ~/.ssh/config):

Compression yes
IdentityFile ~/.ssh/id_rsa
LogLevel ERROR
Port 22

Host h1
    HostName 192.168.91.57
    User master
    IdentityFile ~/.ssh/h1.pem

Host db1
    HostName usca45d1.example.com
    User pg
    LocalForward 5432 127.0.0.1:5432

The above configuration let me access those hosts simply by name, e.g.:

ssh h1
scp schema.sql db1:~/

How to manage Git or Mercurial repositories

Managing version control repositories can be a challenge in multi-user environment especially when simplification of user collaboration is your goal. There are usually two primary concerns while considering enterprise deployment for version control repositories: access control and safety of your data. Both are not directly addressed by version control itself, thus a sort of security facade is necessary.

wheezy web: Actors

The application design of wheezy.web application consists of several actors, each playing its unifying role. Actors are grouped into packages, e.g. models, repository, service, web, content, tests, etc. These packages (or layers) shape a subsystem.

The diagram below visualizes relations between actors, boundaries between packages, subsystems and application, giving a better picture of whole:

The sequence diagram explains request processing and interaction between actors:

You can download a pdf copy here.

Thoughts on SQL vs ORM

The question of persistence implementation arise often. I found repository pattern very valuable due to separation of concerns, mediate between domain model and data source (mock, file, database, web service, etc). The database data source is somewhat specific since you can proceed with SQL functions or ORM. Here are some thoughts why you might prefer SQL functions over ORM in your next project:

wheezy web: Quick Start i18n Project

wheezy.web is a lightweight, high performance, high concurrency WSGI web framework with the key features to build modern, efficient web. Here we will use an i18n (multilingual internationalization) project quick start to build a new project.

1. Download quickstart-i18n.zip and extract.
2. Rename extracted directory `quickstart-i18n` to meet your project name, e.g. mysite and open terminal in that directory.

The i18n project quick start comes with Makefile with several handy targets. Setup development environment:

make env

Internationalization is built using gettext, thus you need extract gettext messages and compile `*.po` files that you can find in i18n directory.

make po

Let run test, look at coverage and finally benchmark tests:

$ make test nose-cover benchmark
....................................
..........
Name                 Stmts   Miss  Cover   Missing
--------------------------------------------------
public                   3      0   100%   
public.web               0      0   100%   
public.web.profile       5      0   100%   
public.web.urls         10      0   100%   
public.web.views        26      0   100%   
--------------------------------------------------
TOTAL                   44      0   100%   
-----------------------------------------------------
Ran 10 tests in 0.458s

OK
public: 5 x 1000
baseline throughput change target
  100.0%    5303rps  +0.0% test_root
   94.4%    5005rps  -5.6% test_home
   98.2%    5206rps  -1.8% test_error_400
   99.2%    5258rps  -0.8% test_error_403
   97.9%    5191rps  -2.1% test_error_404
static: 3 x 1000
baseline throughput change target
  100.0%    5172rps  +0.0% test_static_files
   47.4%    2453rps -52.6% test_static_file_not_found
   52.5%    2715rps -47.5% test_static_file_forbidden
-----------------------------------------------------
Ran 2 tests in 2.286s

OK

The tests are passed let run it in web browser:

$ make run
Visit http://localhost:8080/

Alternatively you can run it using uwsgi

env/bin/easy_install uwsgi
make uwsgi

Enjoy!

wheezy web: Quick Start Empty Project

wheezy.web is a lightweight, high performance, high concurrency WSGI web framework with the key features to build modern, efficient web. Here we will use an empty project quick start to build a new project.

1. Download quickstart-empty.zip and extract.
2. Rename extracted directory `quickstart-empty` to meet your project name, e.g. mysite and open terminal in that directory.

The empty project quick start comes with Makefile with several handy targets. Setup development environment:

make env

Let run test, look at coverage and finally benchmark tests:

$ make test nose-cover benchmark
....................................
..........
Name                 Stmts   Miss  Cover   Missing
--------------------------------------------------
public                   3      0   100%   
public.web               0      0   100%   
public.web.profile       5      0   100%   
public.web.urls         10      0   100%   
public.web.views        20      0   100%   
--------------------------------------------------
TOTAL                   38      0   100%   
--------------------------------------------------
Ran 10 tests in 0.423s

OK
public: 5 x 1000
baseline throughput change target
  100.0%    5041rps  +0.0% test_root
   81.3%    4097rps -18.7% test_home
   99.4%    5008rps  -0.6% test_error_400
  100.3%    5055rps  +0.3% test_error_403
   98.7%    4975rps  -1.3% test_error_404
static: 3 x 1000
baseline throughput change target
  100.0%    4846rps  +0.0% test_static_files
   49.8%    2415rps -50.2% test_static_file_not_found
   55.0%    2663rps -45.0% test_static_file_forbidden
-----------------------------------------------------
Ran 2 tests in 2.375s

OK

The tests are passed let run it in web browser:

$ make run
Visit http://localhost:8080/

Alternatively you can run it using uwsgi

env/bin/easy_install uwsgi
make uwsgi

Enjoy!