FreeBSD Bourne Shell Tricks

Bourne shell is default shell for user account in FreeBSD. You can control settings through the following configuration files:

~/.shrc - user specific settings
/etc/profile - global settings

Color Prompt: display hostname in green color for shell prompt (read more here):

# vi ^[: ctrl+v and then Esc
#PS1="^[[32m$^[[0m "
PS1="^[[1;32m`whoami`@\h ^[[34m\W ^[[0m$ "

List directories in color:

alias ls='ls -G'

Color case insensitive grep:

export GREP_OPTIONS='-i --color=auto'

FreeBSD Virtual Machine Loader Settings

It is recommended to lower kernel internal timer rate and delay before auto booting: Add the following to /boot/loader.conf:

kern.hz=100 # The kernel interval timer rate
autoboot_delay=3 # Delay in seconds before autobooting

Read more here.

Python Fastest Template Engine

What is the fastest template system for Python? See also:

• Comprehensive benchmarks for various template include, extends and widget directives are here.

The benchmark is based on wheezy.template big table test for rendering a 10x1000 HTML table in unicode. Latest available versions (April 2, 2015):

1. bottle 0.12.8
2. chameleon 2.22
3. cheetah 2.4.4
4. django 1.8
5. jinja2 2.7.3
6. mako 1.0.1
7. durusworks qpy 1.2
8. spitfire 0.7.15
9. tenjin 1.1.1
10. tornado 4.1
11. web2py 2.0.9
12. wheezy.template 0.1.159

Let setup virtualenv environment:

virtualenv env
env/bin/easy_install -O2 bottle chameleon cheetah django \
  jinja2 mako tenjin webext tornado wheezy.html \
  wheezy.template

Download bigtable.py test:

wget https://bitbucket.org/akorn/wheezy.template/raw/tip/demos/bigtable/bigtable.py

And run it:

env/bin/python bigtable.py

Troubleshooting slapd error: too many open files

Debian testing comes with openldap (slapd package) version 2.4.28. I noticed that ldap clients start receiving error and cannot contact ldap service any more. While the slapd daemon was running I found a number of errors in syslog (file /var/log/syslog):

... ldap1 slapd[4894]: SASL [conn=1611] Failure: GSSAPI 
  Error: Unspecified GSS failure.  Minor code may 
  provide more information (Too many open files)
... ldap1 slapd[4894]: warning: cannot open 
  /etc/hosts.allow: Too many open files
... ldap1 slapd[4894]: warning: cannot open 
  /etc/hosts.deny: Too many open files
... ldap1 slapd[4894]: SASL [conn=1611] Failure: GSSAPI
  Error: Unspecified GSS failure.  Minor code may
  provide more information (Too many open files)

Take a look at slapd process max open files soft limit:

cat /proc/`pidof slapd`/limits

Check number of files open by slapd process:

pidof slapd | xargs lsof -a -p | wc -l

The slapd process must not exceed max open files soft limit. Take a look at files opened by slapd process:

pidof slapd | xargs lsof -a -p | tail

Here is a sample output that shows there are a number of deleted files... actually one file /var/tmp/ldap_103.

slapd   4894 openldap  117u   REG  0,197 3371 705975 
  /var/tmp/ldap_103 (deleted)
slapd   4894 openldap  118u   REG  0,197 3371 705975 
  /var/tmp/ldap_103 (deleted)
slapd   4894 openldap  119u   REG  0,197 3371 705975 
  /var/tmp/ldap_103 (deleted)

There is definitely a bug in slapd that cause max open files limit exceed.

Recycle Process

We can write a script that does a check and restart daemon if it reaches certain limit (file /usr/local/sbin/slapd-restart).

#!/bin/sh

# Restart slapd daemon if it has open more than 512 files
if [ `pidof slapd | xargs lsof -a -p | wc -l` -gt 512 ]
then
    /etc/init.d/slapd restart
fi

Let cron run this script hourly:

ln -s /usr/local/sbin/slapd-restart \
  /etc/cron.hourly/slapd-restart

If slapd process exceed max open files soft limit too quickly consider schedule cron job more frequently.

Mount /var/tmp in tmpfs

While slapd process creates small files in /var/tmp quite quickly I found reasonable to mount it with tmpfs (file /etc/fstab):

tmpfs /var/tmp tmpfs noatime,nodev,noexec,nosuid,size=1M 0 0

Restart slapd daemon so changes take place:

/etc/init.d/slapd stop
# Ensure the /var/tmp is empty
rm /var/tmp/ldap_103
mount /var/tmp
/etc/init.d/slapd start
# Ensure the /var/tmp mounted
df -h | grep /var/tmp

Ensure the /var/tmp mounted:

tmpfs           1.0M  4.0K 1020K   1% /var/tmp

Regularly take a look a /var/log/syslog if there are any errors reported.

How to Convert FLAC to MP3 in a Batch

Converting flac to mp3 package pre-requirements are the same as published in another post (How to Convert APE+CUE to MP3). So I will add what is necessary. You will need flac package.

apt-get -y install flac

Here is a script that does the rest (file flac-mp3.sh):

#!/bin/bash

for f in *.flac
do
    metaflac --export-tags-to=- "$f" | \
        sed 's/=\(.*\)/="\1"/' | \
        sed 's/\(.*\)=/\L&/' > tags.sh
    . ./tags.sh
    rm ./tags.sh
   
    out_dir="mp3/$artist/$album"
    if [ ! -d "$out_dir" ]; then
        mkdir -p "$out_dir"
    fi
    flac -cd "$f" | lame -h - -v --preset cd \
        --tt "$title" \
        --tn "$tracknumber" \
        --tg "$genre" \
        --ty "$date" \
        --ta "$artist" \
        --tl "$album" \
        --add-id3v2 \
        "$out_dir/${f%.*}.mp3"
done

Drop that file into a directory that has flac files. Run the script and in few minutes you will get a mp3 directory with your mp3 tracks folded by artist/album.