Performance Monitoring in Linux

There are few useful tools that can help find out a bottleneck of your Linux box performance.

What to monitor first?

The system load is a measure of the amount of work that a computer system performs. You can use this command to read system load:

uptime

Here is a sample output:

... load average: 1.07, 1.63, 2.81

The three values of load average refer to the past 1, 5, and 15 minutes of system operation. These numbers should be read this way: the number represents how well a single CPU can handle load, thus if the number is 1 or less - it is pretty comfortable (the 4-CPU system works well at load number 4 or less); 1.5 - means at least 50% of load is not handled on time, it is queued for processing and is a subject for attention.

System Monitoring

Real time monitoring can be observed with top and htop commands. Command htop gives you more convenient way of what top does. Particularly it is handy to add two more columns (via 'F2' Setup) related to IO read and IO write.

htop

Processors related statistics with mpstat:

watch -n 1 mpstat

Disk Monitoring

IO can be a one of possible bottleneck of system performance degradation. The tool iotop tracks disk I/O by process, and prints a summary report that is refreshed every second.

iotop

Statistic for IO devices and partitions can be monitored with iostat:

watch -n 1 iostat

Who is waiting and blocked?

It is useful to know how the system load goes across processes, however most interest is related to processes that keep waiting for the operation to complete, thus cause delays. Here is a simple command to get this kind of report every second:

watch -n 1 "(ps aux | awk '\$8 ~ /D/  { print \$0 }')"

Network Monitoring

Intensive network related operation can cause the high load as well. Here is a tool that let you have a better idea of your network traffic utilization - iftop:

iftop

Troubleshooting: Could not update .ICEauthority file

Here is the error message that might encounter duing gnome desktop startup.

Could not update .ICEauthority file 
/var/lib/gdm3/.ICEauthority

I have resolved this problem by simply removing that file and let gdm3 daemon re-create it. The commands below must be executed with root privileges.

/etc/init.d/gdm3 stop
rm /var/lib/gdm3/.ICEauthority*
/etc/init.d/gdm3 start

Note, restarting gdm not helps all the time, so try reboot computer instead.

How to Backup and Restore OpenLDAP Database

Instead of backup / restore of actual ldap database (hdb, etc) we will export/import ldap directory tree into ldif format that ultimately let us do the same, however without any particular database implementation specifics.

Backup

The backup will be stored in backup.ldif text file.

#!/bin/sh

slapcat -v -l ldap.diff

Restore

The restore will go through replacing current database from a one we have in ldif backup.

#!/bin/sh

# Stop slapd daemon
/etc/init.d/slapd stop

# Remove current database
rm -rf /var/lib/ldap/*

# Import directory tree from backup
slapadd -l backup.ldif

# Fix permissions
chown -R openldap:openldap /var/lib/ldap/*

# Start slapd daemon
/etc/init.d/slapd start

Debian Link Aggregation (Network Bonding)

Here we are going setup link aggregation for Debian. Your system must have at least 2 network interfaces (but not limited to).

Bonding

1. Install ifenslave-2.6 package:

   apt-get install ifenslave-2.6
   

2. Load bonding kernel module:

   modprobe bonding
   

3. Note that you shouldn't have any configuration for eth0 and eth1, they are attached as slaves for bond0 interface. Network configuration (file /etc/network/interfaces):

   auto bond0
   iface bond0 inet static
        address 192.168.10.14
        netmask 255.255.255.0
        network 192.168.10.0
        broadcast 192.168.10.255
        gateway 192.168.10.1
   
        slaves eth0 eth1
   
        # Transmit packets in sequential order 
        # from the first available slave through 
        # the last. This mode provides load 
        # balancing and fault tolerance.
        bond-mode balance-rr
   
        # Only one slave in the bond is active. 
        # A different slave becomes active if, 
        # and only if, the active slave fails. 
        #This mode provides fault tolerance.
        #bond-mode active-backup
   
        bond-miimon 100
        bond-downdelay 200
        bond-updelay 200
   

4. Once above is complete you need to reboot your server so the network changes take place (that should be faster than bringing up/down network interfaces and restarting networking service).

Bridge

You can use link aggregation with network bridge, for example if you are using lxc / kvm virtualization. Here is network configuration (file /etc/network/interfaces):

auto bond0
iface bond0 inet manual
     slaves eth0 eth1
     bond-mode balance-rr
     bond-miimon 100
     bond-downdelay 200
     bond-updelay 200

auto br0
iface br0 inet static
     address 192.168.10.14
     netmask 255.255.255.0
     network 192.168.10.0
     broadcast 192.168.10.255
     gateway 192.168.10.1
     bridge_ports bond0
     bridge_fd 0
     bridge_maxwait 0
     bridge_stp off

Network Switch

If your switch supports port trunking you should add the switch ports used by eth0 and eth1 to a single trunk.

How to change default OS in grub2

Change the following line in file /etc/default/grub (the number corresponds to the grub2 boot menu item starting from zero):

GRUB_DEFAULT=0

Once you made your changes issue the following command:

update-grub

Once you reboot the default selected item will be changed per your changes above.