How to shrink qcow2 file

While working with kvm/qemu virtual environment you might encounter need to shrink image file after a removal of unnecessary files, etc. You will be surprised that the space you freed in guest virtual machine is not actually released in host file. It's size remain the same. Here you will know how to shrink it to minimum.

Windows Guest

The idea here is simple, there are few things you have to do:

1. Delete all unnecessary files, empty recycle bin
2. Defragment drive (you might need to do this several times, until you see it "compacted" well)
3. Use sdelete to zero free disk space. Please note that this operation will cause that all drive free space will be filled by zero, so the virtual machine image will grow to the maximum size.

   sdelete -c c:
   

Linux/FreeBSD Guest

dd if=/dev/zero of=./zero bs=1M
sync
rm -f ./zero

Note, the bs parameter is important, since it greatly reduce time necessary to complete this task.

Host

Convert image to the same format that is currently is (e.g. qcow2 => qcow2)... during this procedure it will release unused space.

qemu-img convert -O qcow2 w2k3.qcow2 \
 w2k3-shrinked.qcow2

The process is time consuming and each phase greatly depends on physical disk IO performance and available free space.

Debian Samba Server

Samba is the protocol by which a lot of PC-related machines share files and printers.

apt-get install samba

Temporary File Space

We will create a simple network share tmp. All users in WORKGROUP will have read-write access. Let create a directory tree (under /srv/smb/) we are going to serve.

mkdir -p /srv/smb/tmp

Place the following in /etc/samba/smb.conf (backup the original file first):

[global]
workgroup = WORKGROUP
server string = Public File Server

# When clients connect to a share level security server, 
# they need not log onto the server with a valid 
# username and password before attempting to connect to
# a shared resource. Instead, the clients send 
# authentication information (passwords) on a per-share
# basis, at the time they attempt to connect to that 
# share.
security = share

# This parameter determines whether or not smbclient(8) 
# and other samba client tools will attempt to 
# authenticate itself to servers using the weaker 
# LANMAN password hash.
client lanman auth = no

# This parameter determines whether or not smbd(8) will 
# attempt to authenticate users or permit password 
# changes using the LANMAN password hash.
lanman auth = no

[tmp]
# If this parameter is yes for a service, then no 
# password is required to connect to the service.
public = yes
comment = Temporary File Space (cleaned up at midnight)
path = /srv/smb/tmp
read only = no
hide files = lost+found

Now from Windows machine you can access this share as \\samba-server-name\tmp. Here is a cron job to make daily clean ups (file /etc/cron.d/clean-samba-tmp):

#
# Regular cron jobs for removing everything 
# in samba tmp share
#
SHELL=/bin/sh

# Run daily at 1:35 AM
# m h dom mon dow user  command
35 1 * * * root /bin/rm -rf /srv/smb/tmp/*

See more configuration options here.

Performance Monitoring in Linux

There are few useful tools that can help find out a bottleneck of your Linux box performance.

What to monitor first?

The system load is a measure of the amount of work that a computer system performs. You can use this command to read system load:

uptime

Here is a sample output:

... load average: 1.07, 1.63, 2.81

The three values of load average refer to the past 1, 5, and 15 minutes of system operation. These numbers should be read this way: the number represents how well a single CPU can handle load, thus if the number is 1 or less - it is pretty comfortable (the 4-CPU system works well at load number 4 or less); 1.5 - means at least 50% of load is not handled on time, it is queued for processing and is a subject for attention.

System Monitoring

Real time monitoring can be observed with top and htop commands. Command htop gives you more convenient way of what top does. Particularly it is handy to add two more columns (via 'F2' Setup) related to IO read and IO write.

htop

Processors related statistics with mpstat:

watch -n 1 mpstat

Disk Monitoring

IO can be a one of possible bottleneck of system performance degradation. The tool iotop tracks disk I/O by process, and prints a summary report that is refreshed every second.

iotop

Statistic for IO devices and partitions can be monitored with iostat:

watch -n 1 iostat

Who is waiting and blocked?

It is useful to know how the system load goes across processes, however most interest is related to processes that keep waiting for the operation to complete, thus cause delays. Here is a simple command to get this kind of report every second:

watch -n 1 "(ps aux | awk '\$8 ~ /D/  { print \$0 }')"

Network Monitoring

Intensive network related operation can cause the high load as well. Here is a tool that let you have a better idea of your network traffic utilization - iftop:

iftop

Troubleshooting: Could not update .ICEauthority file

Here is the error message that might encounter duing gnome desktop startup.

Could not update .ICEauthority file 
/var/lib/gdm3/.ICEauthority

I have resolved this problem by simply removing that file and let gdm3 daemon re-create it. The commands below must be executed with root privileges.

/etc/init.d/gdm3 stop
rm /var/lib/gdm3/.ICEauthority*
/etc/init.d/gdm3 start

Note, restarting gdm not helps all the time, so try reboot computer instead.

How to Backup and Restore OpenLDAP Database

Instead of backup / restore of actual ldap database (hdb, etc) we will export/import ldap directory tree into ldif format that ultimately let us do the same, however without any particular database implementation specifics.

Backup

The backup will be stored in backup.ldif text file.

#!/bin/sh

slapcat -v -l ldap.diff

Restore

The restore will go through replacing current database from a one we have in ldif backup.

#!/bin/sh

# Stop slapd daemon
/etc/init.d/slapd stop

# Remove current database
rm -rf /var/lib/ldap/*

# Import directory tree from backup
slapadd -l backup.ldif

# Fix permissions
chown -R openldap:openldap /var/lib/ldap/*

# Start slapd daemon
/etc/init.d/slapd start