Friday, December 24, 2010

Configuring OpenLDAP with SSL/TLS on Debian

It is recommended that communication between clients and ldap server be encrypted. Before we enable encryption for ldap server we need SSL private key and certificate signed by certificate authority. Have a look at OpenSSL Certificates. Suppose here are your files: ldap.dev.local-key.pem and ldap.dev.local-cert.pem.

Server

  1. Install CA certificate:
    cp ~/ca/demoCA/cacert.pem /etc/ssl/certs/
    chmod go+r /etc/ssl/certs/cacert.pem
    
  2. Copy ldap key and certificate files to /etc/ldap/ssl
    mkdir /etc/ldap/ssl/
    cp ~/ca/ldap.dev.local-*.pem /etc/ldap/ssl/
    
  3. Secure certificates:
    ldap1:~# chown -R root:openldap /etc/ldap/ssl
    ldap1:~# chmod -R o-rwx /etc/ldap/ssl
    
  4. Enable ldaps protocol (file /etc/default/slapd)
    LAPD_SERVICES="ldap://127.0.0.1:389/ ldaps:/// ldapi:///"
    
  5. Create tls configuration file (tls-config.ldif):
    dn: cn=config
    add: olcTLSCACertificateFile
    olcTLSCACertificateFile: /etc/ssl/certs/cacert.pem
    -
    add: olcTLSCertificateFile
    olcTLSCertificateFile: /etc/ldap/ssl/ldap.dev.local-cert.pem
    -
    add: olcTLSCertificateKeyFile
    olcTLSCertificateKeyFile: /etc/ldap/ssl/ldap.dev.local-key.pem
    
  6. Apply it:
    ldapmodify -QY EXTERNAL -H ldapi:/// -f tls-config.ldif
    
  7. Restart slapd:
    /etc/init.d/slapd restart
    
  8. Ensure started:
    netstat -tunlp | grep slapd
    tcp        0      0 0.0.0.0:636             0.0.0.0:*               LISTEN      2462/slapd      
    tcp        0      0 127.0.0.1:389           0.0.0.0:*               LISTEN      2462/slapd  
    

Client

  1. Install ldap-utils package:
    apt-get install ldap-utils
    
  2. Configure (file /etc/ldap/ldap.conf)
    BASE    dc=dev,dc=local
    URI     ldaps://ldap.dev.local
    
    TLS_CACERT /etc/ssl/certs/cacert.pem
    TLS_REQCERT demand
    
  3. Ensure working:
    ldapsearch -x
    
  4. Have a look at server log file, the communication must go through port 636 now
    ldap1 slapd[2462]: conn=1005 fd=15 ACCEPT from IP=192.168.10.8:38344 (IP=0.0.0.0:636)
    ldap1 slapd[2462]: conn=1005 fd=15 TLS established tls_ssf=128 ssf=128
    ldap1 slapd[2462]: conn=1005 op=0 BIND dn="" method=128
    ldap1 slapd[2462]: conn=1005 op=0 RESULT tag=97 err=0 text=
    ldap1 slapd[2462]: conn=1005 op=1 SRCH base="dc=dev,dc=local" scope=2 deref=0 filter="(objectClass=*)"
    ldap1 slapd[2462]: conn=1005 op=1 SEARCH RESULT tag=101 err=0 nentries=6 text=
    ldap1 slapd[2462]: conn=1005 op=2 UNBIND
    ldap1 slapd[2462]: conn=1005 fd=15 closed
    

How to create Certificates using OpenSSL

In order to create a new certificate you basically need to follow two steps: (a) create certificate request, (b) sign request by certificate authority. Since for step (b) you need certificate authority please have a look at previous post that details it.

Certificate Request

The process of creating a certificate request is the same as for certificate authority, except it is important to set valid Common Name that should be a FQDN (e.g. ldap1.dev.local) for the server that this request it for (the name that the client will access your host remotely).
ldap1:~/ca# openssl req -new -nodes -keyout newreq.pem -out newreq.pem
Generating a 2048 bit RSA private key
.....................+++
....................................+++
writing new private key to 'newreq.pem'
...
Country Name (2 letter code) [UA]:
State or Province Name (full name) [LV]:
Locality Name (eg, city) []:Lviv
Organization Name (eg, company) [XYZ Co]:
Organizational Unit Name (eg, section) []:
Common Name (eg, YOUR name) []:ldap1.dev.local
Email Address []:

Please enter the following 'extra' attributes
to be sent with your certificate request
A challenge password []:
An optional company name []:

Sign Request

Here we are going to sign the client certificate request by our certificate authority:
ldap1:~/ca# /usr/lib/ssl/misc/CA.sh -sign
Using configuration from /usr/lib/ssl/openssl.cnf
Enter pass phrase for ./demoCA/private/cakey.pem: *******
Check that the request matches the signature
Signature ok
Certificate Details:
...       
Certificate is to be certified until XXX (365 days)
Sign the certificate? [y/n]:y

1 out of 1 certificate requests certified, commit? [y/n]y
Write out database with 1 new entries
Data Base Updated
Certificate:
    ...
Signed certificate is in newcert.pem
There are two important files we created: newreq.pem and newcert.pem. Consider rename those file to match the service they are created for, e.g. ldap1-key.pem and ldap1-cert.pem. You can combine them into a single file:
cat newreq.pem newcert.pem > new.pem

How to create Certificate Authority using OpenSSL

The Certificate Authority (CA) is used to verify the authenticity of a certificate. Start by installing openssl package:
apt-get install openssl

Create Private Certificate Authority

  1. OpenSSL (version 0.9.8) is installed to path /usr/lib/ssl. The CA.sh script is not in search path, we are going to add it for just current session.
    export PATH=$PATH:/usr/lib/ssl/misc
    
  2. Let customize a bit configuration file (/usr/lib/ssl/openssl.cnf) that is used for certificate creation, but first make a backup copy. Make the following changes:
    ...
    [ req ]
    default_bits    = 2048
    ...
    [ req_distinguished_name ]
    countryName_default             = UA
    stateOrProvinceName_default     = LV
    0.organizationName_default      = XYZ Co
    ...
    
  3. Create a directory for all certificates (it can be any directory, we will create in home):
    mkdir ~/ca && cd ~/ca
    
  4. Answer few questions (hit enter to create a new when prompted for CA filename):
    ldap1:~/ca# CA.sh -newca
    CA certificate filename (or enter to create)
    
    Making CA certificate ...
    Generating a 2048 bit RSA private key
    ............+++
    ........+++
    writing new private key to './demoCA/private/./cakey.pem'
    Enter PEM pass phrase: **************
    Verifying - Enter PEM pass phrase: **************
    ...
    Country Name (2 letter code) [UA]:
    State or Province Name (full name) [LV]:
    Locality Name (eg, city) []:Lviv
    Organization Name (eg, company) [XYZ Co]:
    Organizational Unit Name (eg, section) []:
    Common Name (eg, YOUR name) []:XYZ Root CA
    Email Address []:
    
    Please enter the following 'extra' attributes
    to be sent with your certificate request
    A challenge password []:
    An optional company name []:
    Using configuration from /usr/lib/ssl/openssl.cnf
    Enter pass phrase for ./demoCA/private/./cakey.pem: *****
    Check that the request matches the signature
    Signature ok
    Certificate Details:
    ...
    Write out database with 1 new entries
    Data Base Updated
    
  5. Secure Certificate Authority:
    chmod -R go-rwx ~/ca
    
Your Certificate Authority file is cacert.pem (it is located in ~/ca/demoCA directory).

Thursday, December 23, 2010

How to create a new user in OpenLDAP

We are going create a new account for John Smith. Here are few simple steps:
  1. We need create a template for a new user account jsmith (file add-user.ldif):
    # User primary group
    dn: cn=jsmith,ou=groups,dc=dev,dc=local
    cn: jsmith
    objectClass: top
    objectClass: posixGroup
    gidNumber: 10000
    
    # User account
    dn: uid=jsmith,ou=people,dc=dev,dc=local
    cn: John Smith
    givenName: John
    sn: Smith
    uid: jsmith
    uidNumber: 10000
    gidNumber: 10000
    homeDirectory: /home/jsmith
    mail: jsmith@dev.local
    objectClass: top
    objectClass: posixAccount
    objectClass: shadowAccount
    objectClass: inetOrgPerson
    objectClass: organizationalPerson
    objectClass: person
    loginShell: /bin/bash
    userPassword: {CRYPT}*
    
  2. Load user to ldap:
    ldapadd -cxWD cn=admin,dc=dev,dc=local -f add-user.ldif
    
    or if you are authenticated by Kerberos:
    ldapadd -f add-user.ldif
    
  3. Try to find it:
    ldapsearch -x uid=jsmith
    
  4. Set user password (consider store user password in kerberos instead):
    ldappasswd -xWD cn=admin,dc=dev,dc=local -S uid=jsmith,ou=people,dc=dev,dc=local
    
Read more about openldap here.

Debian OpenLDAP

OpenLDAP is a free, open source implementation of the Lightweight Directory Access Protocol (LDAP).

Install OpenLDAP Server

  1. Ensure the host name is FQDN:
    ldap1:~# hostname 
    ldap1.dev.local
    
    If it is not, issue the following:
    echo "ldap1.dev.local" > /etc/hostname
    hostname -F /etc/hostname
    
  2. Install necessary packages (during a package configuration phase set admin password and accept all default options):
    apt-get -y install rsyslog slapd ldap-utils
    
  3. Setup system-wide defaults for LDAP clients (file /etc/ldap/ldap.conf):
    BASE    dc=dev,dc=local
    URI     ldap://ldap1.dev.local
    
  4. Disable ipv6 support for slapd (file /etc/default/slapd):
    # Additional options to pass to slapd
    SLAPD_OPTIONS="-4"
    
    Restart slapd:
    /etc/init.d/slapd restart
    netstat -tunlp | grep slapd
    
    Output:
    tcp        0      0 0.0.0.0:389             0.0.0.0:*               LISTEN      1557/slapd
    

Logging

  1. Create a file that enable ldap logging (file log-stats.ldif):
    # Enable LDAP logging
    dn: cn=config
    changetype: modify
    replace: olcLogLevel
    olcLogLevel: stats
    
  2. ... disable ldap logging (file log-none.ldif):
    # Disable LDAP logging
    dn: cn=config
    changetype: modify
    replace: olcLogLevel
    olcLogLevel: none
    
  3. And here is a command (changes are applied immediately, no need to restart slapd):
    ldapmodify -QY EXTERNAL -H ldapi:/// -f log-stats.ldif
    

What to index

  1. Create indexes to match the actual filter terms used in search queries. Read more here. We are going to add the following indexes: uid, cn. So here is our index file (file db-index.ldif):
    dn: olcDatabase={1}hdb,cn=config
    changetype: modify
    add: olcDbIndex
    olcDbIndex: uid eq
    -
    add: olcDbIndex
    olcDbIndex: cn eq
    -
    add: olcDbIndex
    olcDbIndex: ou eq
    -
    add: olcDbIndex
    olcDbIndex: dc eq
    -
    add: olcDbIndex
    olcDbIndex: uniqueMember eq
    -
    add: olcDbIndex
    olcDbIndex: uidNumber eq
    -
    add: olcDbIndex
    olcDbIndex: gidNumber eq
    
    Apply changes:
    ldapmodify -QY EXTERNAL -H ldapi:/// -f db-index.ldif
    

Reindex database

  1. Here is a simple script to reindex database (file /usr/local/sbin/slap-reindex). You do not need to run it often, that is depends how big is your database and how many changes occur, consider run it monthly:
    #!/bin/sh
    /etc/init.d/slapd stop > /dev/null
    su openldap -c "slapindex"
    /etc/init.d/slapd start > /dev/null
    

Simple tree structure

  1. Here is our simple structure:
    dev.local
    |--people
    `--groups
    
  2. It correspond to the following (file init-tree.ldif):
    dn: ou=people,dc=dev,dc=local
    ou: people
    objectClass: organizationalUnit
    
    dn: ou=groups,dc=dev,dc=local
    ou: groups
    objectClass: organizationalUnit
    
  3. Add it to ldap:
    ldapadd -cxWD cn=admin,dc=dev,dc=local -f init-tree.ldif
    
  4. Test if we can find it:
    ldapsearch -x ou=people
    
    Here is search result:
    # extended LDIF
    #
    # LDAPv3
    # base  (default) with scope subtree
    # filter: ou=people
    # requesting: ALL
    #
    
    # people, dev.local
    dn: ou=people,dc=dev,dc=local
    ou: people
    objectClass: organizationalUnit
    
    # search result
    search: 2
    result: 0 Success
    
    # numResponses: 2
    # numEntries: 1
    

Wednesday, December 22, 2010

Debian Kerberos Slave

Slave KDCs provide an additional source of Kerberos ticket-granting services in the event of inaccessibility of the master KDC. It recommended that your KDCs have a predefined set of CNAME records (DNS hostname aliases), such as krb for the master KDC and kdc1, kdc2, ... for the slave KDCs. This way, if you need to swap a machine, you only need to change a DNS entry, rather than having to change hostnames.

Master (Primary) Kerberos Server

  1. Add a new slave (kdc2.dev.local) to file /etc/krb5.conf (for the master and any other slaves):
    [realms]
            DEV.LOCAL = {
                    kdc = kdc1.dev.local
                    kdc = kdc2.dev.local
                    admin_server = krb.dev.local
            }
    
    Alternatively (preferred way) consider setup DNS discovery. Read here how.
  2. Add slave host principal:
    kadmin.local -q "addprinc -randkey host/kdc2.dev.local"
    
    kadmin.local -q "ktadd host/kdc2.dev.local"
    
  3. Create database propagation host list (file /etc/krb5kdc/kpropd.acl):
    host/kdc1.dev.local@DEV.LOCAL
    host/kdc2.dev.local@DEV.LOCAL
    
  4. Create a dump of the kerberos database (that is a default path for kprop utility):
    kdb5_util dump /var/lib/krb5kdc/slave_datatrans
    

Secondary (Slave, Read-Only) Kerberos Server

  1. Install Kerberos Server and xinetd (to be used for database propagation):
    apt-get install krb5-kdc xinetd
    
  2. Copy (a) realm configuration (file /etc/krb5.conf), (b) database propagation list (file /etc/krb5kdc/kpropd.acl), (c) keytab (file /etc/krb5.keytab), (d) logrotate settings from master, e.g. using ssh copy:
    scp kdc1:/etc/krb5.conf /etc
    scp kdc1:/etc/krb5kdc/kpropd.acl /etc/krb5kdc
    scp kdc1:/etc/krb5.keytab /etc
    scp kdc1:/etc/logrotate.d/krb5 /etc/logrotate.d
    mkdir /var/log/krb5
    
  3. Setup database propagation service (file /etc/xinetd.d/krb_prop):
    service krb_prop
    {
            disable         = no
            socket_type     = stream
            protocol        = tcp
            user            = root
            wait            = no
            server          = /usr/sbin/kpropd
    }
    
    Restart xinetd service:
    /etc/init.d/xinetd restart
    

Propagate database

  1. Propagate database from Master to Slave
    kdc1:~# kprop kdc2.dev.local
    Database propagation to kdc2.dev.local: SUCCEEDED
    
  2. Create database stash key on slave
    kdb5_util stash
    
  3. Start Kerberos Slave service:
    /etc/init.d/krb5-kdc start
    

Automate database propagation

  1. Here is a script that populates master database to all slaves (run on master, file /usr/local/sbin/krb5-prop):
    #!/bin/sh
    
    #slaves="kdc2.dev.local kdc3.dev.local"
    slaves="kdc2.dev.local"
    
    /usr/sbin/kdb5_util dump /var/lib/krb5kdc/slave_datatrans
    error=$?
    if [ $error -ne 0 ]; then
      echo "Kerberos database dump failed."
      exit 1
    fi
    
    for slave in $slaves; do
      /usr/sbin/kprop $slave > /dev/null
      error=$?
      if [ $error -ne 0 ]; then
        echo "Kerberos propagation to host $slave failed."
      fi
    done
    exit 0
    
    Ensure the file is executable:
    chmod +x /usr/local/sbin/krb5-prop
    
  2. Schedule a cron job (/usr/local/sbin/cron-krb5-prop):
    #
    # Regular cron job for Kerberos database propagation
    #
    PATH=/usr/local/sbin
    HOME=/
    LOG=/dev/null
    
    # Every 53 minutes
    53 * * * * root test -x /usr/local/sbin/krb5-prop && krb5-prop >> $LOG
    
    .. and let cron know about it:
    ln -s /usr/local/sbin/cron-krb5-prop /etc/cron.d/cron-krb5-prop
    
Finally here is how to test it is working:
  1. Stop Master Kerberos server:
    /etc/init.d/krb5-kdc stop
    
  2. Open log file on Slave:
    tail -f /var/log/krb5/kdc.log
    
  3. Login to kerberos client:
    ssh user1@deby01
    
  4. Watch the log on Slave, you should see authentication messages.
Read more about kerberos here.

Tuesday, December 21, 2010

How to setup Kerberos DNS discovery

Kerberos DNS discovery can simplify the client hosts setup. The following need to be added to zone file.
$ORIGIN dev.local.
_kerberos-adm._tcp      SRV     0 0 749 kdc1
$ORIGIN _udp.dev.local.
_kerberos               SRV     10 0 88 kdc1.dev.local.
_kerberos               SRV     20 0 88 kdc2.dev.local.
_kerberos-master        SRV     0 0 88 kdc1.dev.local.
_kpasswd                SRV     0 0 464 kdc1.dev.local.
The client configuration can now look like this (file /etc/krb5.conf):
[libdefaults]
        default_realm = DEV.LOCAL
# ...
[realms]
        DEV.LOCAL = {
        }

[domain_realm]
Let test this:
deby01:~$ host -t SRV _kerberos._udp
_kerberos._udp.dev.local has SRV record 10 0 88 kdc1.dev.local.

Debian Kerberos Client

You must have Kerberos server running on the network, read here how to get it up. We are going to add host deby01 as a client for dev.local Kerberos realm.
  1. Ensure the host name is FQDN:
    ldap1:~# hostname -f
    deby01.dev.local
    
    If it is not, issue the following:
    echo "deby01" > /etc/hostname
    hostname -F /etc/hostname
    
  2. Install Kerberos client:
    apt-get -y install krb5-user libpam-krb5
    
  3. Configure client (file /etc/krb5.conf):
    [libdefaults]
            default_realm = DEV.LOCAL
    # ...
    [realms]
            DEV.LOCAL = {
                    # The entry below can be commented 
                    # out in case there is dns 
                    # resolution for kdc
                    kdc = kdc1.dev.local
                    admin_server = krb.dev.local
            }
    
    [domain_realm]
    
    
  4. Add host principal:
    kadmin -p admin -q "addprinc -randkey host/deby01.dev.local"
    
    kadmin -p admin -q "ktadd host/deby01.dev.local"
    
Let verify it:
  1. List kerberos principals:
    deby01:~# kadmin -p admin -q "list_principals"
    ...
    host/deby01.dev.local@DEV.LOCAL
    ...
    user1@DEV.LOCAL
    ...
    
  2. List keys in keytab:
    klist -ke
    
  3. Now you can login to deby01 as user1.
  4. Have a look at log on kerberos server (file /var/log/krb5/kdc.log):
    kdc1.dev.local krb5kdc[988](info): AS_REQ (4 etypes {18 17 16 23}) 192.168.10.41: NEEDED_PREAUTH: user1@DEV.LOCAL for krbtgt/DEV.LOCAL@DEV.LOCAL, Additional pre-authentication required
    kdc1.dev.local krb5kdc[988](info): AS_REQ (4 etypes {18 17 16 23}) 192.168.10.41: ISSUE: authtime 1293635137, etypes {rep=18 tkt=18 ses=18}, user1@DEV.LOCAL for krbtgt/DEV.LOCAL@DEV.LOCAL
    kdc1.dev.local krb5kdc[988](info): TGS_REQ (4 etypes {18 17 16 23}) 192.168.10.41: ISSUE: authtime 1293635137, etypes {rep=18 tkt=18 ses=18}, user1@DEV.LOCAL for host/deby01.dev.local@DEV.LOCAL
    
The pam authentication by default is configured to authenticate user with kerberos with fallback to local authentication, that is fine so nothing need to be configured there.

How to add a new user to Kerberos

The Kerberos is used only for authentication purpose that means that user we are going to add must exists as a normal unix account (or ldap account).
root@kdc1:~# kadmin.local -q "addprinc user1"
...
Principal "user1@DEV.LOCAL" created.
Let test it out:
root@kdc1:~# kinit user1 && klist && kdestroy 
...
Default principal: user1@DEV.LOCAL
...
The operations must be performed on kdc1 that is Kerberos administrative server.

How to add a new host to Kerberos

Each host (a client computer) that need to be a part of kerberos realm must have principal and keytab. Let do that for host deby01. The command must be invoked on Kerberos administrative server (kdc1):
kadmin.local -q "addprinc -randkey host/deby01.dev.local"
This is run on client (deby01):
kadmin -p admin -q "ktadd host/deby01.dev.local"
or consider using the following script (file /usr/local/sbin/kdc-add):
#!/bin/sh

type=host
if [ ! -z $1 ]; then type=$1; fi

sh -c "`cat /etc/hostname | xargs -t -I {} echo \
"kadmin -p admin -q \\\"addprinc -randkey $type/{}\\\""`"

sh -c "`cat /etc/hostname | xargs -t -I {} echo \
"kadmin -p admin -q \\\"ktadd $type/{}\\\""`"
You can run it on a machine you wish to add (you will be prompted to enter password two times):
kdc-add
Please note that each host added to kerberos must have fully qualified hostname. Both forward and reverse mapping must work properly. Here are few simple tests:
deby01:~# hostname
deby01.dev.local

deby01:~# dig deby01.dev.local +short
192.168.2.41

deby01:~# dig -x 192.168.2.41 +short
deby01.dev.local.
Read more here.

Debian Kerberos Master

Kerberos is a network authentication protocol. The idea is to be a secure, single sign-on authentication provider.
  1. Install Kerberos Server.
    apt-get -y install rsyslog krb5-{admin-server,user,doc}
    
  2. Create realm (this may take a long time, up to few minutes).
    krb5_newrealm
    
  3. Activate Kerberos administration by authorizing admin access (file /etc/krb5kdc/kadm5.acl).
    # ...
    */admin *
    admin *
    
  4. Setup logging (new file /etc/logrotate.d/krb5):
    /var/log/krb5/kadmin.log /var/log/krb5/kdc.log {
            daily
            missingok
            rotate 7
            compress
            delaycompress
            notifempty
    }
    
    Create log directory
    mkdir /var/log/krb5
    
  5. Realm Configuration (file /etc/krb5.conf). In our case the kerberos server name is kdc1 and there is alias to it krb (used for administration purpose).
    [libdefaults]
            default_realm = DEV.LOCAL
    
    [realms]
            DEV.LOCAL = {
                    kdc = kdc1.dev.local
                    # kdc = kdc2.dev.local
                    admin_server = krb.dev.local
            }
    
    [domain_realm]
            .dev.local = DEV.LOCAL
            dev.local = DEV.LOCAL
    
    [logging]
            kdc = FILE:/var/log/krb5/kdc.log
            admin_server = FILE:/var/log/krb5/kadmin.log
    
  6. Restart kerberos services.
    invoke-rc.d krb5-admin-server restart ; \
    invoke-rc.d krb5-kdc restart
    
  7. Open another console and have a look at log files.
    cd /var/log/krb5/ ; \
    tail -f kadmin.log kdc.log
    
  8. Ensure services are running.
    root@kdc1:~# netstat -tunlp
    Active Internet connections (only servers)
    Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
    tcp        0      0 0.0.0.0:464             0.0.0.0:*               LISTEN      840/kadmind     
    tcp        0      0 0.0.0.0:749             0.0.0.0:*               LISTEN      840/kadmind     
    tcp6       0      0 :::464                  :::*                    LISTEN      840/kadmind     
    udp        0      0 0.0.0.0:464             0.0.0.0:*                           840/kadmind     
    udp        0      0 0.0.0.0:88              0.0.0.0:*                           861/krb5kdc     
    udp        0      0 0.0.0.0:750             0.0.0.0:*                           861/krb5kdc     
    
So far we have the services up and running, however in order to administer it we need create an administrative account:
  1. Add admin principal:
    kadmin.local -q "addprinc admin"
    
  2. Add host (kdc1) principal:
    kadmin.local -q "addprinc -randkey host/kdc1.dev.local"
    
    kadmin.local -q "ktadd host/kdc1.dev.local"
    
Now let test it:
root@kdc1:~# kinit admin && klist && kdestroy 
Password for admin@DEV.LOCAL: 
Ticket cache: FILE:/tmp/krb5cc_0
Default principal: admin@DEV.LOCAL

Valid starting     Expires            Service principal
12/21/10 18:05:14  12/22/10 04:05:14  krbtgt/DEV.LOCAL@DEV.LOCAL
 renew until 12/22/10 18:05:11
Read more here.

Thursday, December 16, 2010

How to edit Dynamic DNS zone

All changes made to a zone using dynamic update are stored in the zone's journal file. The zone file is updated every 15 min. The zone files of dynamic zones cannot normally be edited by hand because they are not guaranteed to contain the most recent dynamic changes (those are only in the journal file). Here are few steps that let you edit entries in dynamic dns zone:
  1. Suspend updates to all dynamic zones.
    rndc freeze
    
  2. Edit zone file
  3. Enable updates to all dynamic zones and reload them.
    rndc thaw
    
Read more about advanced dns features here.

Debian DHCP server failover

Before we start I assume you followed previous two posts: setup and dynamic-dns. Our primary dhcp server located at 192.168.10.4 and secondary at 192.168.10.5.

Primary DHCP Server

  1. You need declare failover section that identifies the primary dhcp server (file /etc/dhcp/dhcpd.conf).
    failover peer "dhcp-failover" {
      primary; # declare this to be the primary server
      address 192.168.10.4;
      port 647;
      peer address 192.168.10.5;
      peer port 647;
      max-response-delay 30;
      max-unacked-updates 10;
      load balance max seconds 3;
      mclt 1800;
      split 128;
    }
    
  2. Failover peer needs to be referenced by concrete subnet:
    subnet 192.168.10.0 netmask 255.255.255.0 {
      pool {
        # In order to turn off failover just comment out
        # the line below
        failover peer "dhcp-failover";
        range 192.168.10.41 192.168.10.254;
      }
      option subnet-mask 255.255.255.0;
      option broadcast-address 192.168.10.255;
      option routers gw1.dev.local;
    }
    

Secondary DHCP Server

  1. Failover secondary peer declaration (file /etc/dhcp/dhcpd.conf):
    failover peer "dhcp-failover" {
      secondary; # declare this to be the secondary server
      address 192.168.10.5;
      port 647;
      peer address 192.168.10.4;
      peer port 647;
      max-response-delay 30;
      max-unacked-updates 10;
      load balance max seconds 3;
    }
    
  2. subnet 192.168.10.0 netmask 255.255.255.0 {
      pool {
        # In order to turn off failover just comment out
        # the line below
        failover peer "dhcp-failover";
        range 192.168.10.41 192.168.10.254;
      }
      option subnet-mask 255.255.255.0;
      option broadcast-address 192.168.10.255;
      option routers gw1.dev.local;
    }
    
That pretty much you need to do. Read more here.

Dynamic DNS update with DHCP on Debian

If you have many dhcp clients it is much convenient to find them by name than remember ip addresses. This is what dynamic dns update with dhcp is for. I assume you followed few previous posts on dns and dhcp topic.

Configure DNS server

  1. We would like to accept only authorized secure updates, so let generate a secure key:
    dnssec-keygen -r /dev/urandom -a hmac-md5 -b 256 -n host key
    cat Kkey.*.private
    rm Kkey*
    
    Here is sample output:
    Private-key-format: v1.3
    Algorithm: 157 (HMAC_MD5)
    Key: 9rHjOgEuZ8O8LpsoJcl4zORqbeOCaPc3WfYDd5Mq3FHI=
    ...
    
  2. Add the following (replace md5 key with the one you generated) to a new file /etc/bind/dynamic-dns.key
    key DYNAMICDNS {
            algorithm hmac-md5;
            secret "9rHjOgEuZ8O8LpsoJcl4zORqbeOCaPc3WfYDd5Mq3FHI=";
    };
    
  3. Secure key:
    chmod o-r /etc/bind/dynamic-dns.key
    
  4. Ensure bind is the owner of the configucation directory, since it save some files there during dynamic updates:
    chmod -R g+w /etc/bind/
    
  5. Update zone registration file to allow dynamic updates (file /etc/bind/named.conf.local):
    include "/etc/bind/dynamic-dns.key";
    
    zone "dev.local" IN {
           type master;
           file "/etc/bind/db.dev.local";
           allow-update { key DYNAMICDNS; };
    };
    
    zone "10.168.192.IN-ADDR.ARPA" IN {
           type master;
           file "/etc/bind/db.10.168.192";
           allow-update { key DYNAMICDNS; };
    };
    
  6. Restart bind9

Test DNS Settings

  1. Let configure DNS for a new host test with ip 192.168.10.7:
    root@ns1:/etc/bind# nsupdate 
    > server 127.0.0.1
    > key DYNAMICDNS 9rHjOgEuZ8O8LpsoJcl4zORqbeOCaPc3WfYDd5Mq3FHI=
    > zone dev.local
    > update add test.dev.local. 600 IN A 192.168.10.7
    > send
    > zone 10.168.192.IN-ADDR.ARPA
    > update add 7.10.168.192.in-addr.arpa 600 IN PTR test.dev.local.
    > send
    
  2. And now verify:
    user1@deby01:~$ host test
    test.dev.local has address 192.168.10.7
    
    user1@deby01:~$ host 192.168.10.7
    7.10.168.192.in-addr.arpa domain name pointer test.dev.local.
    

Configure DHCP server

  1. Add the following (replace md5 key with the one you generated) to a new file /etc/dhcp/dynamic-dns.key
    key DYNAMICDNS {
            algorithm hmac-md5;
            secret "9rHjOgEuZ8O8LpsoJcl4zORqbeOCaPc3WfYDd5Mq3FHI=";
    };
    
  2. Create a new file /etc/dhcp/dhcpd.conf.local with the following content:
    include "/etc/dhcp/dynamic-dns.key";
    
    zone dev.local. {
            primary ns1.dev.local;
            key DYNAMICDNS;
    }
    
    zone 10.168.192.IN-ADDR.ARPA. {
            primary ns1.dev.local;
            key DYNAMICDNS;
    }
    
  3. Open file /etc/dhcp/dhcpd.conf and ensure:
    ddns-update-style interim;
    include "/etc/dhcp/dhcpd.conf.local";
    
  4. Restart dhcp server so our change take effect.
    /etc/init.d/isc-dhcp-server restart
    

Test DHCP server with Debian client

  1. First of all in order to identify your debian client by name you must ensure it send host name to dhcp server. You can check this in file /etc/dhcp/dhclient.conf:
    send host-name "deby01";
    
  2. Assuming the dhcp client interface is configured for eth1, here is a command to re-new ip address from server:
    dhclient -v eth1
    

Debian DHCP Server Setup

Dynamic Host Configuration Protocol (DHCP) is a protocol. It gives client machines "leases" for IP addresses and can automatically set their network configuration.
apt-get -y install rsyslog isc-dhcp-server
Before we start configuring the dhcp server let set our requirements:
  • Domain name: dev.local
  • Network: 192.168.10.0/24
  • DNS Servers: ns1.dev.local, ns2.dev.local
  • Gateway: gw1.dev.local
  • First 40 ip addresses are reserved for servers
  • DHCP pool is 41 - 254

Server Configuration

  1. The server will be listening on eth0 interface (file /etc/default/isc-dhcp-server):
    # On what interfaces should the DHCP server (dhcpd) 
    # serve DHCP requests? Separate multiple interfaces 
    # with spaces, e.g. "eth0 eth1".
    INTERFACES="eth0"
    
  2. Configure DHCP per our requirements (file /etc/dhcp/dhcpd.conf)
    # The ddns-updates-style parameter controls whether or
    # not the server will attempt to do a DNS update when 
    # a lease is confirmed. We default to the behavior of 
    # the version 2 packages ('none', since DHCP v2 didn't
    # have support for DDNS.)
    ddns-update-style none;
    
    # option definitions common to all supported networks
    option domain-name "dev.local";
    option domain-name-servers ns1.dev.local, ns2.dev.local;
    option ip-forwarding off;
    
    # This way you can specify multiple search domains.
    # For Windows clients it doesn't work and need to be
    # setup manually
    option domain-search "dev.local", "corp.local";
    
    # Lease time is in seconds
    default-lease-time 600;
    max-lease-time 7200;
    
    # If this DHCP server is the official DHCP server for 
    # the local network, the authoritative directive should 
    # be uncommented.
    authoritative;
    
    # Use this to send dhcp log messages to a different log 
    # file (you also have to hack syslog.conf to complete 
    # the redirection).
    log-facility local7;
    
    subnet 192.168.10.0 netmask 255.255.255.0 {
      pool {
        range 192.168.10.41 192.168.10.254;
      }
      option subnet-mask 255.255.255.0;
      option broadcast-address 192.168.10.255;
      option routers gw1.dev.local;         
    }
    
    

How to test from Debian client

  1. You need a network interface configured for dhcp (file /etc/network/interfaces):
    allow-hotplug eth0
    iface eth0 int dhcp
    
  2. Obtain ip address and check your up:
    root@dh1:~# dhclient eth0 && ifconfig eth0 | grep inet
              inet addr:192.168.10.41 ...
    
  3. Try some lookups (notice multiple dns search list; in order to use host command you need to install dnsutils package):
    root@dh1:~# host ns1
    ns1.dev.local has address 192.168.10.2
    
    root@dh1:~# host mail
    mail.corp.local has address 192.168.11.10
    

How to enable multi-domain search in Windows client

  1. Choose Advanced TCP/IP Settings
  2. In DNS tab choose "Append these DNS suffixes (in order)"
  3. Add as many as you need domain to search
  4. Try some lookups (notice multiple dns search list):
    C:\>nslookup ns1
    ...
    C:\>nslookup mail
    ...
    
Read more here.

Wednesday, December 15, 2010

Debian Slave DNS Server Setup

The setup of Slave (Secondary) DNS Server is pretty easy. You need to follow two previous posts of setting up a simple DNS server and chroot bind9.

Master (Primary) DNS Server

  1. Add the following to /etc/bind/named.conf.options
    dnssec-enable yes;
    
  2. Generate MD5 hash key:
    dnssec-keygen -r /dev/urandom -a hmac-md5 \
     -b 256 -n host rndc ; cat Krndc.*.private \
     | grep Key ; rm Krndc*
    
    Here is output:
    Key: 9EKQM+7+EnJzO7TWyayUf0vks4k+SZPf9DAs8fOeREM=
    
  3. Add the following (replace md5 key with the one you generated) to a new file /etc/bind/transfer.key
    key TRANSFER {
            algorithm hmac-md5;
            secret "9EKQM+7+EnJzO7TWyayUf0vks4k+SZPf9DAs8fOeREM=";
    };
    
  4. Secure key:
    chmod o-r /etc/bind/transfer.key
    
  5. Add the following to a new file /etc/bind/named.conf.transfer
    include "/etc/bind/transfer.key";
    
    // Slave IP Address
    server 192.168.10.3 {
            keys {
            TRANSFER;
        };
    };
    
  6. Add the following to file /etc/bind/named.conf
    include "/etc/bind/named.conf.transfer";
    

Slave (Secondary) DNS Server

  1. Add the following to /etc/bind/named.conf.options
    dnssec-enable yes;
    
  2. Add the following (replace md5 key with the one you generated) to file /etc/bind/transfer.key
    key TRANSFER {
            algorithm hmac-md5;
            secret "9EKQM+7+EnJzO7TWyayUf0vks4k+SZPf9DAs8fOeREM=";
    };
    
  3. Secure key:
    chmod o-r /etc/bind/transfer.key
    
  4. Add the following to file /etc/bind/named.conf.transfer
    include "/etc/bind/transfer.key";
    
    // Master IP Address
    server 192.168.10.2 {
            keys {
            TRANSFER;
        };
    };
    
  5. Add the following to file /etc/bind/named.conf
    include "/etc/bind/named.conf.transfer";
    
  6. Specify slave zones in file /etc/bind/named.conf.local:
    zone "dev.local" IN {
           type slave;
           file "/etc/bind/db.dev.local";
           masters { 192.168.10.2; };
           allow-notify { 192.168.10.2; };
    };
    
    zone "10.168.192.IN-ADDR.ARPA" IN {
           type slave;
           file "/etc/bind/db.10.168.192";
           masters { 192.168.10.2; };
           allow-notify { 192.168.10.2; };
    };
    
  7. Copy forwards to slave (file /etc/bind/named.conf.forward):
    zone "corp.local" IN {
           type forward;
           forwarders { 192.168.11.2; 192.168.11.3; };
    };
    
  8. Ensure bind:bind is the owner of the configuration so it can update the files received from master.
    chown -R bind:bind /var/chroot/bind9/etc/*
    
In order to keep both servers in sync, setup ntpdate (on master and slave dns servers):
root@ns2:/etc/bind# apt-get install ntpdate
...
root@ns2:/etc/bind# ntpdate pool.ntp.org
Now you can restart bind9 on both servers and ensure that slave received zone files.

Troubleshooting

Have a look at system log file (/var/log/syslog) for any errors reported by named. If you will see something telling you permission denied while dumping a file, ensure bind:bind is the owner as following:
chown -R bind:bind /var/chroot/bind9/etc/*
/etc/init.d/bind9 restart
ls -l /ent/bind/db.*

Debian LAN DNS setup

We are going setup a LAN DNS server for a dev.local domain. I assume you already have dns server up and running, if not please follow previous post here.

Configure Forward Lookup Zone

First of all let start from forward lookup zone (file /etc/bind/db.dev.local):
$TTL 2d
dev.local.    IN     SOA     ns1.dev.local. hostmaster.dev.local. (
                             2010122201 ; se = serial number
                             6h         ; ref = refresh
                             15m        ; ret = update retry
                             3w         ; ex = expiry
                             3h         ; min = minimum
                             )
              IN     NS      ns1.dev.local.
              IN     NS      ns2.dev.local.
              IN     MX  10  mail.dev.local.
              IN     A       192.168.10.5
; hosts
gw1           IN     A       192.168.10.1
ns1           IN     A       192.168.10.2
ns2           IN     A       192.168.10.3
mail          IN     A       192.168.10.4
www           IN     CNAME   dev.local.

Configure Reverse Lookup Zone

Each forward lookup zone can have a reverse lookup zone, here is ours (file /etc/bind/db.10.168.192):
$ORIGIN .
$TTL 2d
10.168.192.IN-ADDR.ARPA      IN   SOA   ns1.dev.local. hostmaster.dev.local. (
                             2010122201 ; se = serial number
                             6h         ; ref = refresh
                             15m        ; ret = update retry
                             3w         ; ex = expiry
                             3h         ; min = minimum
                             )
              IN     NS      ns1.dev.local.
              IN     NS      ns2.dev.local.
$ORIGIN 10.168.192.IN-ADDR.ARPA.
1             IN     PTR     gw1.dev.local.
2             IN     PTR     ns1.dev.local.
3             IN     PTR     ns2.dev.local.
4             IN     PTR     mail.dev.local.
5             IN     PTR     www.dev.local.

Add Zone to DNS server

Now that we have both forward and reverse dns lookup zones for dev.local domain, we need to let dns server to know about it. All we need to do is add the following to /etc/bind/named.conf.local:
//
// Do any local configuration here
//

zone "dev.local" IN {
       type master;
       file "/etc/bind/db.dev.local";
};

zone "10.168.192.IN-ADDR.ARPA" IN {
       type master;
       file "/etc/bind/db.10.168.192";
};

// Consider adding the 1918 zones here, if they are not 
// used in your organization
include "/etc/bind/zones.rfc1918";

Forwarding to other LAN DNS Servers

Let do DNS forwarding for corp.local LAN domain with dns server on 192.168.11.2 (create a new file /etc/bind/named.conf.forward).
zone "corp.local" IN {
       type forward;
       forwarders { 192.168.11.2; 192.168.11.3; };
};
zone "11.168.192.IN-ADDR.ARPA" IN {
       type forward;
       forwarders { 192.168.11.2; 192.168.11.3; };
};
Let include it into the /etc/bind/named.conf
include "/etc/bind/named.conf.forward";
We need to ask bind9 reload the changes:
/etc/init.d/bind9 reload

Forwarding to IPS DNS Servers

You can optimize the dns queries to use dns servers supplied by your ISP (they are much closer to you than any others). In case your dns server can not resolve some domains, instead of contacting root servers it will contact ISP's servers first. This can be configured in /etc/bind/named.conf.options file:
// forwarders {
//      0.0.0.0;
// };
forwarders {
        192.168.123.123;    # ns1.your-isp.net
        192.168.321.321;    # ns2.your-isp.net
};

Client Configuration

Our dns server for dev.local is up and running, so now it is time configure client machines to use it. Ensure the following in /etc/resolv.conf:
search dev.local
nameserver 192.168.10.2
Please read more here.

How to chroot bind9 in Debian

Here are few simple steps to chroot bind9 in debian squeeze.
#!/bin/bash
/etc/init.d/bind9 stop
mkdir -p /var/chroot/bind9/{etc,dev,var/cache/bind,var/run/bind/run}
chown -R bind:bind /var/chroot/bind9/var/*
mknod /var/chroot/bind9/dev/null c 1 3
mknod /var/chroot/bind9/dev/random c 1 8
chmod 666 /var/chroot/bind9/dev/{null,random}
mv /etc/bind /var/chroot/bind9/etc
ln -s /var/chroot/bind9/etc/bind /etc/bind
chown -R bind:bind /etc/bind/*
echo "\$AddUnixListenSocket /var/chroot/bind9/dev/log" > /etc/rsyslog.d/bind-chroot.conf
Switch bind9 to use the chroot (file /etc/default/bind9):
OPTIONS="-u bind -t /var/chroot/bind9"
Finally restart rsyslogd and start bind9.
/etc/init.d/rsyslog restart ; /etc/init.d/bind9 start
You can download script from here.

Debian simple DNS server setup

We are going setup a simple Debian DNS server for local purpose using bind9.
apt-get install -y rsyslog bind9 bind9-doc dnsutils
Once the server installed let our system know which dns server to use (a one we just installed), ensure that 127.0.0.1 is the first nameserver in the list (file /etc/resolv.conf):
nameserver 127.0.0.1
In case you do no need the server to listen on ipv6 set the following option (file /etc/bind/named.conf.options):
listen-on-v6 { none; };
Restart bind9 daemon:
/etc/init.d/bind9 restart
and verify with:
root@ns1:~# netstat -tunlp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 192.168.10.2:53         0.0.0.0:*               LISTEN      816/named       
tcp        0      0 127.0.0.1:53            0.0.0.0:*               LISTEN      816/named       
tcp        0      0 127.0.0.1:953           0.0.0.0:*               LISTEN      816/named       
udp        0      0 192.168.10.2:53         0.0.0.0:*                           816/named       
udp        0      0 127.0.0.1:53            0.0.0.0:*                           816/named       
That pretty it, let ensure its working. First we need install dnsutils package that comes with dig command, so here we go:
root@ns1:~# dig debian.org
; <<>> DiG 9.7.2-P3 <<>> debian.org
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 64434
;; flags: qr rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 3, ADDITIONAL: 3

;; QUESTION SECTION:
;debian.org.   IN A

;; ANSWER SECTION:
debian.org.  3600 IN A 128.31.0.51
debian.org.  3600 IN A 206.12.19.7

;; AUTHORITY SECTION:
debian.org.  28606 IN NS ns2.debian.org.
debian.org.  28606 IN NS ns4.debian.com.
debian.org.  28606 IN NS ns1.debian.org.

;; ADDITIONAL SECTION:
ns1.debian.org.  28606 IN AAAA 2607:f8f0:610:4000:214:38ff:feee:b65a
ns4.debian.com.  28606 IN A 194.177.211.209
ns4.debian.com.  28606 IN AAAA 2001:648:2ffc:deb::10:10

;; Query time: 96 msec
;; SERVER: 127.0.0.1#53(127.0.0.1)
;; WHEN: Wed Dec 15 21:47:12 2010
;; MSG SIZE  rcvd: 196
Notice the server responded to our request was 127.0.0.1. Read more here and here. Consider chroot your dns server, details here.

How to solve eth0 missing in VirtualBox

Suppose you setup a linux virtual machine in VirtualBox and once you clone that hard disk and attach to a new virtual machine you notice that eth0 is not available. The problem is related to fact that since the MAC address of network adapter has changed (you created a new virtual machine) kernel has reconfigured it to be used by next available name, e.g. eth1. So what you need is simply open file /etc/udev/rules.d/70-persistent-net.rules in your favorite editor and remove a line that uses currently eth0 and change the line with NAME="eth1" to NAME="eth0". Here is an example:
# PCI device 0x1022:0x2000 (pcnet32)
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", \
ATTR{address}=="08:00:27:43:0b:0f", ATTR{dev_id}=="0x0", \
ATTR{type}=="1", KERNEL=="eth*", NAME="eth0"
Probably simplest way to do this:
echo > /etc/udev/rules.d/70-persistent-net.rules
reboot

How to disable ipv6 in Debian

Here are simple steps to disable ipv6 in Debian:
  1. Comment out anything related to ipv6 in /etc/hosts
  2. SSH. Ensure AddressFamily inet is set in /etc/ssh/sshd_config. Restart ssh.
  3. BIND. Ensure listen-on-v6 { none; }; in /etc/bind/named.conf.options. Restart bind9.
  4. NTP. Ensure -4 option is set in /etc/default/ntp (e.g. NTPD_OPTS='-4 -g'). Restart ntp.
  5. APACHE2. Ensure Listen 0.0.0.0:80 in /etc/apache2/ports.conf file. Restart apache2.
  6. RPCBIND (rpc.statd, rpc.mountd). Comment out the appropriate entries in /etc/netconfig:
    udp        tpi_clts      v     inet     udp     - -
    tcp        tpi_cots_ord  v     inet     tcp     - -
    #udp6       tpi_clts      v     inet6    udp    - -
    #tcp6       tpi_cots_ord  v     inet6    tcp    - -
    rawip      tpi_raw       -     inet      -      - -
    local      tpi_cots_ord  -     loopback  -      - -
    unix       tpi_cots_ord  -     loopback  -      - -
    
  7. PostgreSQL 9. Ensure ipv4 in listen_addresses (file /etc/postgresql/9.1/main/postgresql.conf):
    # - Connection Settings
    listen_addresses = '0.0.0.0'
    
    Comment out lines related to ipv6 (file /etc/postgresql/9.1/main/pg_hba.conf):
    # IPv6 local connections:
    #host  all     all     ::1/128   md5
    
    Restart postgresql.
  8. Disable ipv6 in kernel:
    echo net.ipv6.conf.all.disable_ipv6=1 \
    > /etc/sysctl.d/disableipv6.conf
    
  9. Disable ipv6 in kernel modules (file /etc/modprobe.d/aliases.conf):
    # alias net-pf-10 ipv6
    alias net-pf-10 off
    alias ipv6 off
    
The next time the system boots it will have ipv6 disabled. Let verify it with:
netstat -tunlp
Here is a sample output:
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 192.168.10.2:53         0.0.0.0:*               LISTEN      895/named       
tcp        0      0 127.0.0.1:53            0.0.0.0:*               LISTEN      895/named       
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      734/sshd        
tcp        0      0 127.0.0.1:953           0.0.0.0:*               LISTEN      895/named       
udp        0      0 192.168.10.2:53         0.0.0.0:*                           895/named       
udp        0      0 127.0.0.1:53            0.0.0.0:*                           895/named         
Read more about ipv6 here.

Tuesday, December 14, 2010

How to lock console in Debian

There is an easy way to lock your console session in Linux. Just install vlock:
apt-get install vlock
Once installed, issue command vlock, here is what you will see:
This TTY is now locked.

Please press [ENTER] to unlock.
Read more about vlock here.

Thursday, November 25, 2010

How to setup static IP address on Debian

Let check few pre-requirements we need to obtain from the network admin:
  1. Host name: deby01.dev.local
  2. Ip address: 192.168.10.21
  3. Network: 192.168.10/24
  4. Gateway: 192.168.10.1
  5. DNS Servers: 192.168.10.2 and 192.168.10.3
  6. Local Domains: dev.local, corp.local
Here are few simple steps to give your Debian box static IP address:
  1. Configure hostname:
    echo "deby01" > /etc/hostname
    
  2. Ensure /etc/hosts file properly resolves localhost:
    127.0.0.1       localhost
    # In some circumstance it is not recommended to
    # resolve machine name to local ip address
    #127.0.1.1      deby01.dev.local  deby01
    
  3. Here is content of /etc/network/interfaces
    # The primary network interface
    auto eth0
    iface eth0 inet static
         address 192.168.10.21
         netmask 255.255.255.0
         network 192.168.10.0
         broadcast 192.168.10.255
         gateway 192.168.10.1
    
  4. Resolver configuration file /etc/resolv.conf:
    domain dev.local
    search dev.local dev.local. corp.local.
    nameserver 192.168.10.2
    nameserver 192.168.10.3
    
If you need setup a dynamic ip address with DHCP, take a look here.

How to setup DHCP client on Debian

The Dynamic Host Configuration Protocol (DHCP) is an auto configuration protocol used on IP networks. Before we proceed let assume the hostname we are going to setup is deby01.dev.local.
  1. Configure hostname:
    echo "deby01" > /etc/hostname
    
  2. Ensure hosts file properly resolves localhost:
    127.0.0.1       localhost
    # In some circumstance it is not recommended to
    # resolve machine name to local ip address
    #127.0.1.1      deby01.dev.local  deby01
    
  3. Here is content of /etc/network/interfaces
    # The primary network interface
    allow-hotplug eth0
    iface eth0 inet dhcp
    
    # If you experience issues with obtaining
    # default gateway consider uncomment lines below.
    #       up route add default gw 192.168.10.1
    #       down route del default gw 192.168.10.1
    
  4. Let your dhcp client publish our name (so it can be resolved by name). Ensure the following in file /etc/dhcp/dhcpclient.conf:
    send host-name "deby01";
    
  5. If you experience issues with DNS you can set them manually (file /etc/dhcp/dhcpclient.conf)
    prepend domain-name-servers 8.8.8.8, 8.8.4.4;
    
  6. Reboot in order to changes take effect or issue the following commands:
    ifdown eth0 && ifup eth0
    
  7. Check your dynamically obtained ip address:
    deby01:~# ifconfig eth0 | grep inet
     inet addr:192.168.10.41 ...
    
If you need to setup a static ip address, take a look here.

Wednesday, November 24, 2010

How to Compile a Kernel in Debian

Here you will see how to build Debian deb package for linux kernel source.

Install Tools

First of all install few tools:
apt-get install kernel-package libncurses5-dev fakeroot bzip2 build-essential

Prepare Working Directory

  1. Add users to group src:
    usermod -a -G src user1
    
  2. Ensure group src is the owner of /usr/src:
    chgrp -R src /usr/src
    chmod g+s /usr/src
    chmod -R g+w /usr/src
    

Download Kernel Source

  1. Change your working directory to /usr/src/:
    cd /usr/src
    
  2. Download kernel source from kernel.org:
    wget -c http://www.kernel.org/pub/linux/kernel/v2.6/linux-2.6.36.tar.bz2
    
  3. Decompress kernel source and create a symbolic link:
    tar xjf linux-2.6.36.tar.bz2
    test -L linux && rm linux 
    ln -s linux-2.6.36 linux
    cd linux
    

Configure Kernel

  1. Let use existing kernel configuration as a start point:
    cp /boot/config-`uname -r` /usr/src/linux/.config
    
  2. Launch kernel configuration tool:
    make menuconfig
    
  3. Start kernel build (this may take from 20 mins to few hours depending on your hardware and number of CPUs):
    time fakeroot make-kpkg -j 2 --initrd --append-to-version=-custom kernel_image kernel_headers
    
  4. Once build finishes you should get two .deb files in parent directory:
    deby01:/usr/src$ ls -l *.deb
    
    linux-headers-2.6.36-custom_2.6.36-custom-10.00.Custom_amd64.deb
    linux-image-2.6.36-custom_2.6.36-custom-10.00.Custom_amd64.deb
    

Install Kernel

  1. Install linux kernel and image deb packages by issuing the following command:
    dpkg -i *.deb
    
  2. Restart your computer:
    shutdown -r now
    
  3. Once it restarts check the kernel version you are using:
    deby01:~$ uname -r
    2.6.36-custom
    
Enjoy your custom kernel.

Tuesday, November 23, 2010

How to quickly reboot you Linux

Here is a quick way to reboot you Linux with kexec command. In debian first install kexec-tools package:
apt-get install kexec-tools
Once installed make your current kernel the one you want to quickly reboot into:
kexec -e
In case you would like disable fast reboot, open file located at /etc/default/kexec and set LOAD_KEXEC to false:

# Defaults for kexec initscript
# sourced by /etc/init.d/kexec and /etc/init.d/kexec-load

# Load a kexec kernel (true/false)
LOAD_KEXEC=true

# Kernel and initrd image
KERNEL_IMAGE="/vmlinuz"
INITRD="/initrd.img"

# If empty, use current /proc/cmdline
APPEND=""

How to get Linux partition UUID

Ext3 file system UUID

tune2fs -l /dev/sda11 | grep UUID
Here is an output:
Filesystem UUID: 0827dce3-d1c0-41b1-bc6f-0d4cfa0a1849

XFS file system UUID

xfs_admin -u /dev/sda13
Here is an output:
UUID = f128170a-1ee4-4b4f-abe7-0acf169bb8ae

Thursday, November 18, 2010

How to compact VirtualBox VDI drive

While working with dynamically expanding virtual drives you might notice that it actually virtual drive takes more space than it actually use by the virtual machine, e.g. you removed a bunch of files, defragmented volume, etc. Here is how you can compact VDI drive:
VBoxManage modifyhd --compact my-virtual-drive.vdi
Please note, that you can compact only drives that are registered with VirtualBox Media Manager.

Wednesday, November 17, 2010

How to change VirtualBox VDI drive UUID

While working with virtual machines you often need to copy an existing virtual machine and go from there, however once you try add it to VirtualBox you will get a message like this:
A hard disk with UUID ... is already registered.
In this case what you need is re-generate UUID for the VDI image. Here is how you can do that:
VBoxManage internalcommands setvdiuuid my-virtual-drive.vdi
This way the virtual drive my-virtual-drive.vdi will be assigned a new UUID so it can be registered in VirtualBox Media Manager.

Timing disk read/write performance with dd

There is easy way to measure your partition performance with dd. Here is how to measure write performance by simply coping a file of length 1 Gb:
deby:~# time dd if=/dev/zero of=file bs=1024 count=1000000
1000000+0 records in
1000000+0 records out
1024000000 bytes (1.0 GB) copied, 13.454 s, 76.1 MB/s

real 0m13.696s
user 0m0.240s
sys 0m5.924s
Same way we can read that file back into a memory (/dev/shm):
deby:~# time dd if=file of=/dev/shm/file
2000000+0 records in
2000000+0 records out
1024000000 bytes (1.0 GB) copied, 4.19182 s, 244 MB/s

real 0m4.334s
user 0m0.592s
sys 0m3.740s
Read more about dd command here.

Tuesday, November 16, 2010

How to defragment XFS

There is an easy way to find out if your XFS partition needs defragmentation. Here is the command (it is part of package xfsprogs):
deby:~# xfs_db -c frag -r /dev/sda10
actual 2903, ideal 2418, fragmentation factor 16.71%
Once you see it is pretty high, e.g. above 40% you would need to issue the following command that defragment the drive:
deby:~# xfs_fsr -v /dev/sda10
/home start inode=0

Thursday, October 28, 2010

Working with Vim Explore plugin

Vim Explore command let you browse files.

Hiding files and folders: Ctrl + H

Sometimes you would like exclude some files or folders from the list, e.g. pyc files that are results of python compilation, or .svn folders, etc. There is an easy way to get this done. Add the following to your ~/.vimrc file:
let g:netrw_list_hide = '.pyc,.svn,.egg-info'

Back to Explore: Ctrl + 6

When you click on a file in explore view Vim displays its content. Once you finished with your changes to the file there is a way to quickly get back to the explore view. Just press Ctrl+6.

Change current folder: c

When you are in Explore view you can use the following command to create a new file:
:e filename
This command create a file in a folder you started Vim, but sometimes that is not what you want, usually need a new file in a folder you currently browsing. Just press c to make a folder in explore view a current folder for Vim.
Read more about explore here.

Wednesday, October 27, 2010

Vim - Save in Insert Mode

You can use the following combination:
  • Press Ctrl+O
  • Than enter command :w and hit enter
But there is a better way. If you come from Windows there is Ctrl+S combination that saves changes, you can use it in your VIM editor (try issue the following in Vim command mode):
" Use Ctrl+S to save file is edit and command modes
inoremap <c-s> <c-o>:w<cr>
nnoremap <c-s> :w<cr>

Consider add it to your .vimrc file. If you are using Vim in putty console, please have a look here. Read more about mapping keys here.

Recovering from Ctrl+S in Putty

The problem is related to XON/XOFF command that is mapped to Ctrl+S sequence. The terminal doesn't echo the commands you issue, so you need to remember press Ctrl+Q in order to turn flow control ON. There is a way to ignore such behaviour. What you need to do is to change your terminal characteristics.
stty -ixon
Consider add this command to your /etc/profile.d/ixon.sh file.

Thursday, June 24, 2010

Python duplicate code detection with clonedigger

The python tool clonedigger can be used to examine your source code for duplication. It can be installed with easy_install clonedigger. Here is how to run it:
user1@deby:~/devenv/trunk$ ../bin/clonedigger src/
Parsing  src/greatings/helloworld.py ... done
Parsing  src/greatings/__init__.py ... done
Parsing  src/greatings/tests/__init__.py ... done
Parsing  src/greatings/tests/test_helloworld.py ... done
3 sequences
average sequence length: 2.666667
maximum sequence length: 3
Number of statements:  8
Calculating size for each statement... done
Building statement hash... done
Number of different hash values:  5
Building patterns... 6 patterns were discovered
Choosing pattern for each statement... done
Finding similar sequences of statements... 0  sequences were found
Refining candidates... 0 clones were found
Removing dominated clones... 0 clones were removed
Read more about clonedigger here.

Tuesday, June 22, 2010

Python code metrics with pymetrics

The python tool pymetrics can be used to measure your source code complexity. You can install it with easy_install pymetrics. Here is an example (running in virtual environment devenv; source code is located in src):
user1@deby:~/devenv/trunk$ ../bin/pymetrics src/greatings\
/helloworld.py

=== File: src/greatings/helloworld.py ===
Module src/greatings/helloworld.py is missing a module doc string.
Detected at line 1

Basic Metrics for module src/greatings/helloworld.py
----------------------------------------------------
          1    maxBlockDepth
          4    numBlocks
        189    numCharacters
          1    numDocStrings
          1    numFcnDocStrings
          2    numFunctions
          7    numKeywords
         16    numLines
         73    numTokens

         50.00 %FunctionsHavingDocStrings

Functions DocString present(+) or missing(-)
--------------------------------------------
- main
+ say

McCabe Complexity Metric for file src/greatings/helloworld.py
--------------------------------------------------------------
          2    __main__
          1    main
          1    say

COCOMO 2's SLOC Metric for src/greatings/helloworld.py
-------------------------------------------------------
          8    src/greatings/helloworld.py

*** Processed 1 module in run ***
Here is a script that let you generate report for all your files in src directory:
#!/bin/sh

working_dir=/tmp/$USER/pymetrics
mkdir -p $working_dir

find src/ -name \*.py > $working_dir/files.txt
../bin/pymetrics --nosql --nocsv -f $working_dir/files.txt
Here is a guideline for understanding cyclomatic complexity number:
  • 1 - 15: simple code, minimum risk, can be easily covered by tests
  • 15 - 30: complicated code, consider refactoring before writing tests
  • 30 - 50+: complex code, refactor now, almost impossible to write a good tests
Read more about software complexity here and here.

How to count source lines of code in Linux

You can use debian package sloccount for SLOC:
deby:~# apt-get install sloccount
  ...
user1@deby:~/devenv/trunk$ sloccount src/ | grep ^python
python:          31 (100.00%)
Read more about sloccount here.

Monday, June 21, 2010

Python static code analysis with pyflakes

Python tool pyflakes is focused on identifying common errors quickly without executing Python code. It can be installed with easy_install pyflakes. Since the tool doesn't import your code, you just specify a folder with source code:
user1@deby:~/devenv/trunk$ ../bin/pyflakes src/
Using pyflakes with IDE like PyDev make good sense since you see errors while you are typing.

Python static code analysis with pylint

Pylint is a python tool that checks if a module satisfies a coding standard. You can install pylint with easy_install (pylint has dependencies on logilab_common and logilab_astng, so consider download them as well in case of offline install).
easy_install pylint
Running checks is pretty easy:
user1@deby:~/devenv/trunk/src$ ../../bin/pylint greatings
No config file found, using default configuration
************* Module greatings
C:  1: Missing docstring
************* Module greatings.helloworld
C:  1: Missing docstring
C: 10:main: Missing docstring
************* Module greatings.tests
C:  1: Missing docstring
W:  4: Relative import 'test_helloworld', should be 
'greatings.tests.test_helloworld'
W:  7:suite: Redefining name 'suite' from outer scope (line 6)
C:  6:suite: Missing docstring
************* Module greatings.tests.test_helloworld
C:  1: Missing docstring
C:  4:HelloworldTestCase: Missing docstring
C:  6:HelloworldTestCase.test_say: Missing docstring
R:  6:HelloworldTestCase.test_say: Method could be a function
W: 12:suite: Redefining name 'suite' from outer scope (line 10)
C: 10:suite: Missing docstring
  ...
Global evaluation
-----------------
Your code has been rated at 5.94/10
The tool also produces a report that include:
  • Statistics by type
  • External dependencies
  • Duplication
  • Raw metrics
  • Messages by category
  • % errors / warnings by module
  • Messages
If you want to change the default behaviour, you can define options in pylintrc file (use --rcfile option to specify a location). Here is an example.
user1@deby:~/devenv/trunk$ wget -P tools/ http://www.\
logilab.org/cgi-bin/hgwebdir.cgi/pylint/raw-file\
/df8f34aa3dd2/examples/pylintrc
user1@deby:~/devenv/trunk$ cd src/
user1@deby:~/devenv/trunk/src$ ../../bin/pylint \
--rcfile=../tools/pylintrc greatings
Read more about pylint here.

Python static code analysis with pychecker

Python tool PyChecker is used to find typical programming errors in your source code. In order to install it you need download it and install manually (if you try install it using easy_install it will not work, unfortunately). We will assume the following directory structure (note, devenv is virtual environment, you can create it issuing virtualenv devenv).
~/devenv/
`-- trunk/
    |-- src/
    |   `-- greatings/
    |       |-- __init__.py
    |       |-- helloworld.py
    |       `-- tests/
    |           |-- __init__.py
    |           `-- test_helloworld.py
    `-- tools/
Let download pychecker into tools directory and proceed with installation:
user1@deby:~/devenv/trunk$ wget -P tools/ http://downloads.\
sourceforge.net/project/pychecker/pychecker/0.8.18/\
pychecker-0.8.18.tar.gz
...
user1@deby:~/devenv/trunk$  cd tools && tar zxf \
pychecker-0.8.18.tar.gz \
 && cd  pychecker-0.8.18
user1@deby:~/devenv/trunk/tools/pychecker-0.8.18$ \
../../../bin/python setup.py install && cd ../.. && \
rm -rf tools/pychecker-0.8.18
...
The next step you need to "fix" a bit a pychecker file installed into your virtual environment bin directory so the content look like this.
#!/bin/sh

site_packages_dir=$(dirname $0)/../lib/python2.6/site-packages
python $site_packages_dir/pychecker/checker.py "$@"
Since pychecker import all the files it is going to check, you need to start the tool in correct directory, in our case src:
user1@deby:~/devenv/trunk$ cd src/
user1@deby:~/devenv/trunk/src$ ../../bin/pychecker greatings/*.py
Processing module helloworld (greatings/helloworld.py)...
Processing module __init__ (greatings/__init__.py)...

Warnings...

None
Please note that pychecker doesn't drill into sub-packages, so in order to analyze tests issue the following command:
../../bin/pychecker greatings/tests/*.py
If you want to change the default behaviour, you can define options in pycheckrc file (use -F option to specify a location). Here is an example.
user1@deby:~/devenv/trunk$ wget -P tools/ http://pychecker.\
cvs.sourceforge.net/viewvc/pychecker/pychecker/pycheckrc
user1@deby:~/devenv/trunk$ cd src/
user1@deby:~/devenv/trunk/src$ ../../bin/pychecker -F \
../tools/pycheckrc greatings/*.py
Read more about pychecker here and here.

Saturday, June 19, 2010

Python packaging with setuptools

We are going create a python egg distribution for a simple helloworld module.

Install tools

Let install two tools we need (consider switch to debian sid repository in order to get latest version of the tools):
deby:~# apt-get -y install python-setuptools python-virtualenv
We are going to work in isolated environment:
user1@deby:~$ virtualenv devenv
New python executable in devenv/bin/python
Installing setuptools............done.
user1@deby:~$ cd devenv/
user1@deby:~/devenv$

Directory structure

Suppose our directory structure looks this way:
~/devenv/
`-- trunk/
    |-- src/
    |   `-- greatings/
    |       |-- __init__.py
    |       |-- helloworld.py
    |       `-- tests/
    |           |-- __init__.py
    |           `-- test_helloworld.py
    `-- README.txt
We are going to place all our python code in src directory.
mkdir -p trunk/src/greatings/tests

Code

The file __init__.py is left empty and makes greating a python package. Here is content of helloworld.py (note that we are using docunits in order to demonstrate dependencies later, main function will be an entry point of our script):
import sys

def say():
    """
    >>> say()
    'hello world'
    """
    return 'hello world'

def main():
    print(say())
    return 0

if __name__ == '__main__':
    sys.exit(main())
The tests will be combined into test suites so they can be easier added for testing our setup later. Here is test_helloworld.py:
from greatings import helloworld
import unittest

class HelloworldTestCase(unittest.TestCase):

    def test_say(self):
        assert 'hello world' == helloworld.say()


def suite():
    loader = unittest.TestLoader()
    suite = unittest.TestSuite()
    suite.addTest(loader.loadTestsFromTestCase(HelloworldTestCase))
    return suite

if __name__ == '__main__':
    unittest.TextTestRunner(verbosity=2).run(suite())
Here is tests package __init__.py file:
from greatings import helloworld
import test_helloworld

def suite():
    import unittest
    import doctest
    suite = unittest.TestSuite()
    suite.addTests(doctest.DocTestSuite(helloworld))
    suite.addTests(test_helloworld.suite())
    return suite

if __name__ == '__main__':
    unittest.TextTestRunner(verbosity=2).run(suite())

Setup files

Here is out ~/devenv/trunk/setup.py file:
import os
from setuptools import setup, find_packages

setup(
    name = 'greatings',
    version = '0.1',

    # Package structure
    #
    # find_packages searches through a set of directories 
    # looking for packages
    packages = find_packages('src', exclude = ['ez_setup',
        '*.tests', '*.tests.*', 'tests.*', 'tests']),
    # package_dir directive maps package names to directories.
    # package_name:package_directory
    package_dir = {'': 'src'},

    # Not all packages are capable of running in compressed form, 
    # because they may expect to be able to access either source 
    # code or data files as normal operating system files.
    zip_safe = True,

    # Entry points
    #
    # install the executable
    entry_points = {
        'console_scripts': ['helloworld = greatings.helloworld:main']
    },

    # Dependencies
    #
    # Dependency expressions have a package name on the left-hand 
    # side, a version on the right-hand side, and a comparison 
    # operator between them, e.g. == exact version, >= this version
    # or higher
    install_requires = [
        '',
    ],

    # Tests
    #
    # Tests must be wrapped in a unittest test suite by either a
    # function, a TestCase class or method, or a module or package
    # containing TestCase classes. If the named suite is a package,
    # any submodules and subpackages are recursively added to the
    # overall test suite.
    test_suite = 'greatings.tests.suite',
    # Download dependencies in the current directory
    tests_require = 'docutils >= 0.6',

    # Meta information
    #
    author = 'Me',
    author_email = 'my@e-mail.com',
    description = 'A sample hello world application',
    url = 'http://mindref.blogspot.com'
)
And configuration (file ~/devenv/setup.cfg):
[global]
# Just silently do your job
quiet = 1

[easy_install]
# Where we are going to look for thrirdparty dependencies
find_links = thirdparty

[build_py]
# No optimization for now
optimize = 0
# Force build everything?
force = True

[egg_info]
# We are doing development build
tag_build = dev
# Do we want to have date in file name?
tag_date = 0
# Add svn revision to the file name
tag_svn_revision = 1

[bdist_egg]
# We do not want to distribute binary with source code
exclude-source-files = True

[rotate]
# Keep only last 10 eggs, clean up older
match = .egg
keep = 10

Third party dependencies

The next thing, we would like keep thirdparty dependencies (e.g. docutils) in a separate folder so each time we build the project it doesn't download dependencies from internet instead look at our folder, so we always have a proper version there. So let create directory thirdparty at the same level as src and download there docutils.
user1@deby:~/devenv/trunk$ mkdir thirdparty
user1@deby:~/devenv/trunk$ wget -P thirdparty/ http://pypi.python.org\
/packages/source/d/docutils/docutils-0.6.tar.gz
The directory structure should look like this:
~/devenv/
`-- trunk/
    |-- src/
    |   `-- greatings/
    |        ...
    `-- thirdparty/
        `-- docutils-0.6.tar.gz
Install docutils that we downloaded into our environment:
../bin/easy_install thirdparty/*

Test, EGG, Source

Let ensure tests are passed:
master@deby:~/devenv/trunk$ ../bin/python setup.py test
..
---------------------------------------------
Ran 2 tests in 0.015s

OK
Here is how to create a binary distribution in egg format (look outcome at ~/devenv/trunk/dist directory):
../bin/python setup.py bdist_egg
... and source code:
../bin/python setup.py sdist
Both source and binary distributions are in ~/devenv/trunk/dist directory.
user1@deby:~/devenv/trunk$ ls dist/
greatings-0.1dev-py2.6.egg  greatings-0.1dev.tar.gz

Version control

Before adding the project to version control (e.g. svn), ensure the following directories are ignored:
  1. build
  2. dist
  3. src/greatings.egg-info
That's it.

Wednesday, June 16, 2010

How to disable autoindent in VIM

When copy/paste from one source to the other (e.g. sample code), vim auto-indent makes too much indentation, so you want to quickly turn it off. Here is a command:
:setl noai nocin nosi inde=
Here is a mapping (add to file ~/.vimrc):
" Disable autoindent in VIM
nnoremap <F8> :setl noai nocin nosi inde=<CR>
Alternatively, you can use:
" Turning off auto indent when pasting text into vim
set pastetoggle=<F8>
So now, before pasting something in, you press F8 to disable auto-indent.

Monday, June 14, 2010

Sync time with external server in Debian

You can synchronize your local computer time with external time servers. You need install ntpdate:
apt-get -y install ntpdate
Try update your time manually:
deby:~# ntpdate pool.ntp.org
12 Jun 00:04:16 ntpdate[1903]: step time server 62.80.187.114 offset -291.468062 sec
Here is a script that syncs the time with external server, adjusts clock drift and finally sets the hardware clock to current system time (file /usr/local/sbin/sync-time).
#!/bin/bash

server=pool.ntp.org

# Sleep a random amount, not greater than 2 minutes
sleep_time=$(($RANDOM % 120))

echo "Sleeping for $sleep_time seconds..."
/bin/sleep $sleep_time

# Sync the time with external server
echo "Sync time with $server."
/usr/sbin/ntpdate -s $server || exit 1

# Adjusts clock drift
/sbin/hwclock --adjust

# Set the hardware clock to current system time
/sbin/hwclock --systohc

Here we are going to schedule a cron job on system startup (file /usr/local/sbin/sync-time, symbolic link from /etc/cron.d/sync-time):
#
# Regular cron job for time synchronization
#
PATH=/usr/local/sbin
HOME=/
LOG=/dev/null

# Every 23 hours, e.g. 2:11, 13:11, etc
11 */23 * * * root test -x /usr/local/sbin/sync-time && sync-time > $LOG
The next time your system restarts it will automatically synchronizes clock with external server. Please note that the system sync the time each time the network interface is up (see ntpdate script in /etc/network/if-up.d/).

Saturday, May 29, 2010

Python mock testing with pymock module

Python module pymock is based on EasyMock. Install pymock module:
easy_install pymock
Suppose you need to test withdraw operation in ATM (file atm.py):
from datetime import datetime

class Atm:
    def signin(self, account):
        self._account = account

    def signout(self):
        self._account = None

    def withdraw(self, amount):
        try:
            self._account.withdraw(amount)
            self._account.comission(amount * 0.005)
        except ValueError:
            self._account.comission(amount * 0.001)
        return self._account.balance(datetime.now())
Pymock uses a recording and replay model. Here is our test (file pymockexample.py):
from datetime import datetime
from atm import Atm
import unittest
from pymock import Controller, Any

class TestAtm(unittest.TestCase):

    def setUp(self):
        self._mocker = Controller()
        self._mock_account = self._mocker.mock()
        self._atm = Atm()
        self._atm.signin(self._mock_account)

    def tearDown(self):
        self._atm.signout()
        self._mocker.verify()

    def test_withdraw(self):
        # Arrange
        self._mock_account.deposit(150)
        self._mock_account.withdraw(100)
        self._mock_account.comission(0.5)

        # WARNING: This doesn't work
        # self._mock_account.balance(Any())
        self._mock_account.balance(datetime.now())
        self._mocker.returns(49.5)
        self._mocker.replay()

        # Act
        self._mock_account.deposit(150)
        remaining_balance = self._atm.withdraw(100)

        # Assert
        assert remaining_balance == 49.5

    def test_withdraw_insufficient_funds(self):
        # Arrange
        self._mock_account.deposit(50)
        self._mock_account.withdraw(100)
        self._mocker.raises(ValueError('Insufficient funds'))
        self._mock_account.comission(0.1)
        self._mock_account.balance(datetime.now())
        self._mocker.returns(49.9)
        self._mocker.replay()

        # Act
        self._mock_account.deposit(50)
        remaining_balance = self._atm.withdraw(100)

        # Assert
        assert remaining_balance == 49.9

if __name__ == '__main__':
    unittest.main()
Run tests:
python pymockexample.py
Read more about pymock here.

Python mock testing with mox module

Python module mox is based on EasyMock. Install mox module:
easy_install mox
Suppose you need to test withdraw operation in ATM (file atm.py):
from datetime import datetime

class Atm:
    def signin(self, account):
        self._account = account

    def signout(self):
        self._account = None

    def withdraw(self, amount):
        try:
            self._account.withdraw(amount)
            self._account.comission(amount * 0.005)
        except ValueError:
            self._account.comission(amount * 0.001)
        return self._account.balance(datetime.now())
When you create a mock object, it is in record mode. You record the behavior by calling the expected methods. Once you are done, switch to replay mode. Here is our test (file moxexample.py):
from datetime import datetime
from atm import Atm
import unittest
from mox import Mox, IgnoreArg, Func

class TestAtm(unittest.TestCase):

    def setUp(self):
        self._mox = Mox()
        self._mock_account = self._mox.CreateMockAnything()
        self._atm = Atm()
        self._atm.signin(self._mock_account)

    def tearDown(self):
        self._atm.signout()
        self._mox.VerifyAll()

    def test_withdraw(self):
        # Arrange
        self._mock_account.deposit(150)
        self._mock_account.withdraw(100)
        self._mock_account.comission(0.5)
        self._mock_account.balance(\
                Func(lambda d: d <= datetime.now()))\
                .AndReturn(49.5)
        self._mox.ReplayAll()

        # Act
        self._mock_account.deposit(150)
        remaining_balance = self._atm.withdraw(100)

        # Assert
        assert remaining_balance == 49.5

    def test_withdraw_insufficient_funds(self):
        # Arrange
        self._mock_account.deposit(50)
        self._mock_account.withdraw(100)\
                .AndRaise(ValueError('Insufficient funds'))
        self._mock_account.comission(0.1)
        self._mock_account.balance(IgnoreArg())\
                .AndReturn(49.9)
        self._mox.ReplayAll()

        # Act
        self._mock_account.deposit(50)
        remaining_balance = self._atm.withdraw(100)

        # Assert
        assert remaining_balance == 49.9

if __name__ == '__main__':
    unittest.main()
Run tests:
python moxexample.py
Read more about mox here.

Python mock testing with mock module

Install mock module:
easy_install mock
Suppose you need to test withdraw operation in ATM (file atm.py):
from datetime import datetime

class Atm:
    def signin(self, account):
        self._account = account

    def signout(self):
        self._account = None

    def withdraw(self, amount):
        try:
            self._account.withdraw(amount)
            self._account.comission(amount * 0.005)
        except ValueError:
            self._account.comission(amount * 0.001)
        return self._account.balance(datetime.now())
After performing an action on Mock instance, you can make assertions about which methods / attributes were used and arguments they were called with. Here is our test (file mockexample.py):
from datetime import datetime
from atm import Atm
import unittest
from mock import Mock

class TestAtm(unittest.TestCase):

    def setUp(self):
        self._mock_account = Mock()
        self._atm = Atm()
        self._atm.signin(self._mock_account)

    def tearDown(self):
        self._atm.signout()

    def test_withdraw(self):
        # Arrange
        def balance(d):
            self.assertTrue(d >= datetime.now()) 
            return 49.5
        self._mock_account.balance.side_effect = balance

        # Act
        self._mock_account.deposit(150)
        remaining_balance = self._atm.withdraw(100)

        # Assert
        assert remaining_balance == 49.5
        self._mock_account.deposit.assert_called_with(150)
        self._mock_account.withdraw.assert_called_with(100)
        self._mock_account.comission.assert_called_with(0.5)

    def test_withdraw_insufficient_funds(self):
        # Arrange
        self._mock_account.withdraw\
                .side_effect = ValueError('Insufficient funds')
        self._mock_account.balance.return_value = 49.9

        # Act
        self._mock_account.deposit(50)
        remaining_balance = self._atm.withdraw(100)

        # Assert
        assert remaining_balance == 49.9
        self._mock_account.deposit.assert_called_with(50)
        self._mock_account.withdraw.assert_called_with(100)
        self._mock_account.comission.assert_called_with(0.1)

if __name__ == '__main__':
    unittest.main()
Run tests:
python mockexample.py
Read more about mock here.

Python mock testing with mocker module

First of all install mocker:
easy_install mocker
Suppose you need to test withdraw operation in ATM (file atm.py):
from datetime import datetime

class Atm:
    def signin(self, account):
        self._account = account

    def signout(self):
        self._account = None

    def withdraw(self, amount):
        try:
            self._account.withdraw(amount)
            self._account.comission(amount * 0.005)
        except ValueError:
            self._account.comission(amount * 0.001)
        return self._account.balance(datetime.now())
A Mocker instance is used for expectations record/replay. Here is our test (file mockerexample.py):
import unittest
from datetime import datetime

from mocker import Mocker, ANY, expect
from atm import Atm


class TestAtm(unittest.TestCase):

    def setUp(self):
        self._mocker = Mocker()
        self._mock_account = self._mocker.mock()
        self._atm = Atm()
        self._atm.signin(self._mock_account)

    def tearDown(self):
        self._atm.signout()
        self._mocker.restore()
        self._mocker.verify()

    def test_withdraw(self):
        # Arrange
        self._mock_account.deposit(150)
        self._mock_account.withdraw(100)
        self._mock_account.comission(0.5)
        expect(self._mock_account.balance(ANY))\
                .result(49.5)\
                .call(lambda d: self.assertTrue(d <= datetime.now()))
        self._mocker.replay()

        # Act
        self._mock_account.deposit(150)
        remaining_balance = self._atm.withdraw(100)

        # Assert
        assert remaining_balance == 49.5

    def test_withdraw_insufficient_funds(self):
        # Arrange
        self._mock_account.deposit(50)
        expect(self._mock_account.withdraw(100))\
                .throw(ValueError('Insufficient funds'))
        self._mock_account.comission(0.1)
        expect(self._mock_account.balance(ANY))\
                .result(49.9)
        self._mocker.replay()

        # Act
        self._mock_account.deposit(50)
        remaining_balance = self._atm.withdraw(100)

        # Assert
        assert remaining_balance == 49.9

if __name__ == '__main__':
    unittest.main()
Run tests:
python mockerexample.py
Read more about mocker here.

Python mock testing with mockito module

This module is a port of the Mockito mocking framework to Python. Install mockito with easy_install:
easy_install mockito
Suppose you need to test withdraw operation in ATM (file atm.py):
from datetime import datetime

class Atm:
    def signin(self, account):
        self._account = account

    def signout(self):
        self._account = None

    def withdraw(self, amount):
        try:
            self._account.withdraw(amount)
            self._account.comission(amount * 0.005)
        except ValueError:
            self._account.comission(amount * 0.001)
        return self._account.balance(datetime.now())
Here is our test (file mockitoexample.py):
from datetime import datetime
from atm import Atm
import unittest
from mockito import Mock, when, verify, any, verifyNoMoreInteractions

class TestAtm(unittest.TestCase):

    def setUp(self):
        self._mock_account = Mock()
        self._atm = Atm()
        self._atm.signin(self._mock_account)

    def tearDown(self):
        self._atm.signout()
        verifyNoMoreInteractions(self._mock_account)

    def test_withdraw(self):
        # Arrange
        when(self._mock_account).balance(any(datetime))\
                .thenReturn(49.5)

        # Act
        self._mock_account.deposit(150)
        remaining_balance = self._atm.withdraw(100)

        # Assert
        assert remaining_balance == 49.5
        verify(self._mock_account).deposit(150)
        verify(self._mock_account).withdraw(100)
        verify(self._mock_account).comission(0.5)
        verify(self._mock_account).balance(any(datetime))

    def test_withdraw_insufficient_funds(self):
        # Arrange
        when(self._mock_account).withdraw(100)\
                .thenRaise(ValueError('Insufficient funds'))
        when(self._mock_account).balance(any(datetime))\
                .thenReturn(49.9)

        # Act
        self._mock_account.deposit(50)
        remaining_balance = self._atm.withdraw(100)

        # Assert
        assert remaining_balance == 49.9
        verify(self._mock_account).deposit(50)
        verify(self._mock_account).withdraw(100)
        verify(self._mock_account).comission(0.1)
        verify(self._mock_account).balance(any(datetime))

if __name__ == '__main__':
    unittest.main()
Run tests:
python mockitoexample.py
Read more about mockito here.

Python mock testing with ludibrio module

In a unit test, mock objects are simulated objects that mimic the behavior of real objects in controlled ways. You can install ludibrio mock framework module issuing the following command:
easy_install ludibrio
This framework comes with:
  • Mock - recorded expectations, strict behaviour
  • Dummy - never validated, ignore behaviour
  • Stub - canned answers, loose behaviour, give me as much as you can or Dummy
  • Spy - proxy, forwarder behaviour, give what you setup, everything else from wrapped type
Suppose you need to test withdraw operation in ATM (file atm.py):
from datetime import datetime

class Atm:
    def signin(self, account):
        self._account = account

    def signout(self):
        self._account = None

    def withdraw(self, amount):
        try:
            self._account.withdraw(amount)
            self._account.comission(amount * 0.005)
        except ValueError:
            self._account.comission(amount * 0.001)
        return self._account.balance(datetime.now())
Here is our test (file ludibrioexample.py):
from atm import Atm
import unittest
from ludibrio import Mock, Dummy

class TestAtm(unittest.TestCase):

    def setUp(self):
        self._mock_account = Mock()
        self._atm = Atm()
        self._atm.signin(self._mock_account)

    def tearDown(self):
        self._atm.signout()
        self._mock_account.validate()

    def test_withdraw(self):
        # Arrange
        with self._mock_account as a:
            a.deposit(150) >> None
            a.withdraw(100) >> None
            a.comission(0.5) >> None
            a.balance(Dummy()) >> 49.5

        # Act
        self._mock_account.deposit(150)
        remaining_balance = self._atm.withdraw(100)

        # Assert
        assert remaining_balance == 49.5

    def test_withdraw_insufficient_funds(self):
        # Arrange
        with self._mock_account as a:
            a.deposit(50) >> None
            a.withdraw(100) >> ValueError('Insufficient funds')
            a.comission(0.1) >> None
            a.balance(Dummy()) >> 49.9

        # Act
        self._mock_account.deposit(50)
        remaining_balance = self._atm.withdraw(100)

        # Assert
        assert remaining_balance == 49.9

if __name__ == '__main__':
    unittest.main()
Run tests:
python ludibrioexample.py
Read more about ludibrio here.

Python unit testing with nose module

You can install nose issuing the following command:
easy_install nose
nose collects tests automatically, as long as you follow some simple guidelines for organizing test code.
from nose import SkipTest
from nose.tools import raises

class Counter:
    def __init__(self, value = 0):
        self.value = value

    def add(self, x):
        if not x:
            raise ValueError
        self.value += x
        return self.value

class TestCounter:
    def setup(self):
        """Automatically called by nose before and
           for each test method invoked
        """
        self._counter = Counter()

    def teardown(self):
        """Automatically called by nose after and
           for each test method invoked
        """
        self._counter = None

    def test_initial_value(self):
        assert self._counter.value == 0

    def test_add(self):
        assert 5 == self._counter.add(5)
        assert 5 == self._counter.value

    @raises(ValueError)
    def test_add_zero_raises_error(self):
        self._counter.add(0)

    def test_skip_me(self):
        raise SkipTest("Not ready yet")
        assert False

if __name__ == '__main__':
    import nose
    nose.runmodule()
In order to execute all tests just run the following:
test1@deby:~$ python noseexample.py
...S
----------------------------------------------------------------------
Ran 4 tests in 0.000s

OK (SKIP=1)
You can run doctests with nose. The easiest way to do so is to add the following to your setup.cfg file:
[nosetests]
verbosity=1
with-doctest=1
And use command-line:
nosetest noseexample.py
Read more about nose testing tools here.

Friday, May 28, 2010

Python unit testing with py.test module

You can install py.test issuing the following command:
easy_install py
py.test offers automatic test discovery:
from py.test import raises
from py.test import skip

class Counter:
    def __init__(self, value = 0):
        self.value = value

    def add(self, x):
        if not x:
            raise ValueError
        self.value += x
        return self.value

class TestCounter:
    def setup_method(self, state):
        """Automatically called by py.test before and
           for each test method invoked
        """
        self._counter = Counter()

    def teardown_method(self, state):
        """Automatically called by py.test after and
           for each test method invoked
        """
        self._counter = None

    def test_initial_value(self):
        assert self._counter.value == 0

    def test_add(self):
        assert 5 == self._counter.add(5)
        assert 5 == self._counter.value

    def test_add_zero_raises_error(self):
        raises(ValueError, self._counter.add, 0)

    def test_skip_me(self):
        skip('Not ready yet')
        assert False

if __name__ == '__main__':
    pass
In order to execute all tests just run the following:
test1@deby:~$ py.test pytestexample.py
============================= test session starts ==================
test object 1: pytestexample.py

pytestexample.py ...s

===================== 3 passed, 1 skipped in 0.08 seconds ==========
Read more about py.test features here.

Thursday, May 27, 2010

Python unit testing with doctest module

The doctest module is part of the Python standard library. There is no need to write separate test functions/methods in doctest, you simply copy the expected results and paste them in the docstring that corresponds to the tested function.
class Counter:
    def __init__(self, value = 0):
        """
        >>> Counter().value
        0
        >>> Counter(100).value
        100
        """
        self.value = value
    
    def add(self, x):
        """
        >>> c = Counter()
        >>> c.add(5)
        5
        >>> c.value
        5
        >>> c.add(0)
        Traceback (most recent call last):
        ...
        ValueError
        >>> c.skip_me # doctest: +SKIP
        """
        if not x:
            raise ValueError
        self.value += x
        return self.value

if __name__ == '__main__':
    import doctest
    doctest.testmod()
In order to execute all tests just run the following (on success no output is given):
python doctestexample.py
Alternatively you can run doctest with unittest:
if __name__ == '__main__':
    import doctest, unittest
    suite = unittest.TestSuite()
    suite.addTest(doctest.DocTestSuite(__name__))
    runner = unittest.TextTestRunner()
    runner.run(suite)
Read more about doctest here.

Python unit testing with unittest module

The unittest module (also known as pyunit) is a unit test framework included in Python standard library. Let see how you can test the following:
class Counter:
    def __init__(self, value = 0):
        self.value = value
    
    def add(self, x):
        if not x:
            raise ValueError
        self.value += x
        return self.value
All test methods should start with word test by convention. Here is a unit test (file unittestexample.py):
import unittest

class CounterTestCase(unittest.TestCase):
    def setUp(self):
        """Automatically called by TestCase before and
           for each test method invoked
        """
        self._counter = Counter()

    def tearDown(self):
        """Automatically called by TestCase after and
           for each test method invoked
        """
        self._counter = None

    def test_initial_value(self):
        self.assertFalse(self._counter.value)
        
    def test_add(self):        
        self.assertEqual(5, self._counter.add(5))
        self.assertEqual(5, self._counter.value)

    def test_add_zero_raises_error(self):
        self.assertRaises(ValueError, lambda: self._counter.add(0))

    def skip_test_skip_me(self):
        assert False

if __name__ == '__main__':
    unittest.main()
In order to execute all tests just run the following (each dot corresponds to a test run):
test1@deby:~$ python unittestexample.py
...
---------------------------------------
Ran 3 tests in 0.000s
Verbose output:
python unittestexample.py -v
A selected test run:
python unittestexample.py CounterTestCase.test_add
Alternatively you can combine several test cases into test suites.
 ...

def suite():
    loader = unittest.TestLoader()
    suite = unittest.TestSuite()
    suite.addTest(loader.loadTestsFromTestCase(CounterTestCase))
    return suite

if __name__ == '__main__':
    unittest.TextTestRunner(verbosity=2).run(suite())
Here is file __init__.py
import unittest

import unittestexample

def suite():
    suite = unittest.TestSuite()
    suite.addTests(unittestexample.suite())
    return suite

if __name__ == '__main__':
    unittest.TextTestRunner(verbosity=2).run(suite())
You can read more about unittest module here and here.