Using Buildbot with Multiple Projects

The BuildBot is a system to automate the compile/test cycle required to validate code changes. Here we are going setup buildbot master and slaves with the following requirements:

1. We have serveral projects: project1, project2, project3
2. These projects live under Mercurial (e.g. hosted at bitbucket.org) with base url https://scm.dev.local/hg/ followed by project name (e.g. https://scm.dev.local/hg/project1)
3. We should be able run at least 2 builds in parallel (this is where we can use buildbot slaves: linux-slave1 and linux-slave2)
4. The build should trigger automatically if the changes detected every 30 minutes during working day.
5. The builder should execute make all that ultimatelly does everything we need to verify integrity.
6. The builder should be checked agains certain python versions: 2.4, 2.5, 2.6, 2.7 and 3.2

Install Buildbot Packages

Installation in debian is pretty straight forward:

 
apt-get -y install buildbot subversion mercurial git make

This installs both master and slave. By default debian installer places buildbot working directories to /var/lib/buildbot. This directory includes 2 others: masters, slaves. So basic installation supports multiple masters and slaves.

Create Master

Let call our master buildbot instance master1 so in future in case we need more (that happens) we can easily add seconds, etc.

cd /var/lib/buildbot/masters
buildbot create-master --relocatable --log-size=512000 \
--log-count=3 master1
cp master1/master.cfg.sample master1/master.cfg
chown -R buildbot:buildbot master1/

This command creates a buildmaster working directory and buildbot.tac file. The master will live in master1. At runtime, the master will read a configuration file (named master.cfg by default) in its basedir.

Configure Buildbot Daemon

The Debian installer for buildbot comes with configuration file (/etc/default/buildmaster) that let you auto start build masters.

MASTER_RUNNER=/usr/bin/buildbot

# 1-enabled, 0-disabled
MASTER_ENABLED[1]=1                    
# short name printed on start/stop
MASTER_NAME[1]="buildmaster1"          
# user to run master as
MASTER_USER[1]="buildbot"              
# basedir to master (absolute path)
MASTER_BASEDIR[1]="/var/lib/buildbot/masters/master1" 
# buildbot options
MASTER_OPTIONS[1]=""                   
# prefix command, i.e. nice, linux32, dchroot
MASTER_PREFIXCMD[1]=""                 

By this point we should be able to start buildbot with default configuration.

/etc/init.d/buildmaster start

Look at master1/twisted.log for errors:

tail -f master1/twisted.log

Let verify it is up and running (web front end listen on port 8010, slaves enter point is 9989):

netstat -tunlp
tcp  0 0 0.0.0.0:9989 0.0.0.0:* LISTEN 3557/python     
tcp  0 0 0.0.0.0:8010 0.0.0.0:* LISTEN 3557/python

Open browser to http://localhost:8010/ (change localhost to the buildbot server name) to see buildbot welcome page.

Master Configuration

Open master.cfg (located at /var/lib/buildbot/masters/master1) and define our pre-requirements:

# -*- python -*-
# ex: set syntax=python:

# This is a sample buildmaster config file. It must be installed as
# 'master.cfg' in your buildmaster's base directory.

# This is the dictionary that the buildmaster pays attention to. We also use
# a shorter alias to save typing.
c = BuildmasterConfig = {}
source_root = 'https://scm.dev.local/hg/'
source_user = ''
projects = ['project1', 'project2', 'project3']
python_versions = ['2.4', '2.5', '2.6', '2.7', '3.2']

passwd = {
    'source': '',
    'linux-slave1': '',
    'linux-slave2': ''
}

from buildbot.locks import MasterLock
from buildbot.locks import SlaveLock
source_lock = MasterLock('source')
build_lock = SlaveLock('build',
     maxCount = 1,
     maxCountForSlave = {
         'linux-slave1': 2, 
         'linux-slave2': 2 
})

Consider read strong password generator.

Build Slaves

Build slaves represent a standard, manually started machine that will try to connect to the buildbot master as a slave (file master.cfg):

###### BUILDSLAVES

# The 'slaves' list defines the set of recognized buildslaves. Each element is
# a BuildSlave object, specifying a unique slave name and password.  The same
# slave name and password must be configured on the slave.
from buildbot.buildslave import BuildSlave
c['slaves'] = [
    BuildSlave('linux-slave1', passwd['linux-slave1']),
    BuildSlave('linux-slave2', passwd['linux-slave2'])
]

# 'slavePortnum' defines the TCP port to listen on for connections from slaves.
# This must match the value configured into the buildslaves (with their
# --master option)
c['slavePortnum'] = 9989

Change Sources

The sources are checked every 30 mins, with 1 minute shift. This let lower source control workload in case you have many projects. Here is an example for svn:

####### CHANGESOURCES

# the 'change_source' setting tells the buildmaster how it should find out
# about source code changes.
from buildbot.changes.svnpoller import SVNPoller
from buildbot.changes.svnpoller import split_file_alwaystrunk
c['change_source'] = []
c['change_source'].extend(SVNPoller(
            svnurl=source_root + name,
            split_file=split_file_alwaystrunk,  
            svnuser=source_user,
            svnpasswd=passwd['source'],
            project=name,
            pollinterval=1800 + i*60
        ) for i, name in enumerate(projects))

Schedulers

Schedulers are responsible for initiating builds on builders. Several schedulers perform filtering on an incoming set of changes, here we are filtering on project name.

####### SCHEDULERS

# Configure the Schedulers, which decide how to react to incoming changes.  In this
# case, just kick off a 'runtests' build

# buildbot version 0.8.5
from buildbot.changes.filter import ChangeFilter
from buildbot.schedulers.basic import SingleBranchScheduler
c['schedulers'] = []
c['schedulers'].extend([
        SingleBranchScheduler(
            name=name,
            change_filter=ChangeFilter(
                branch=None,
                project=name,
            ),
            treeStableTimer=5 * 60,
            builderNames=["%s-%s" % (name, v) for v in python_versions]
        ) for name in projects
])

# buildbot version 0.8.6p1
from buildbot.schedulers.forcesched import ForceScheduler
c['schedulers'].extend([
             ForceScheduler(
                 name=name + '-forced',
                 builderNames=[name])
             for name in projects
])

Builders

A BuildFactory defines the steps that every build will follow (sort of script).

####### BUILDERS

# The 'builders' list defines the Builders, which tell Buildbot how to perform a build:
# what steps, and which slaves can execute them.  Note that any particular build will
# only take place on one slave.

from buildbot.config import BuilderConfig
from buildbot.process.factory import BuildFactory
from buildbot.steps.shell import ShellCommand
from buildbot.steps.source.mercurial import Mercurial

c['builders'] = []
c['builders'].extend([
        BuilderConfig(
            name="%s-%s" % (name, v),
            slavenames=["linux-slave1", "linux-slave2"],
            factory=BuildFactory([
                Mercurial(
                    repourl=source_root + name, 
                    branchType='inrepo',
                    mode='full',
                    method='fresh',
                    haltOnFailure=True,
                    locks=[source_lock.access('exclusive')]
                ),
                ShellCommand(
                    command=["make", "all"], 
                    haltOnFailure=True,
                    locks=[build_lock.access('counting')]
                ),
                ShellCommand(
                    command=["make", "clean"],
                    alwaysRun=True
                )
            ]))
        for name in projects
        for v in python_versions
])

Status Targets

The Buildmaster has a variety of ways to present build status to various users. Each such delivery method is a status target. Here we define web status and notification via email.

####### STATUS TARGETS

# 'status' is a list of Status Targets. The results of each build will be
# pushed to these targets. buildbot/status/*.py has a variety to choose from,
# including web pages, email senders, and IRC bots.

c['status'] = []

from buildbot.status import html
from buildbot.status.web import authz
authz_cfg=authz.Authz(
    # change any of these to True to enable; see the manual for more
    # options
    gracefulShutdown = False,
    forceBuild = True, # use this to test your slave once it is set up
    forceAllBuilds = False,
    pingBuilder = False,
    stopBuild = False,
    stopAllBuilds = False,
    cancelPendingBuild = False,
)
c['status'].append(html.WebStatus(http_port=8010, authz=authz_cfg))

from buildbot.status.mail import MailNotifier
c['status'].append(MailNotifier(
            fromaddr="bb1@dev.local",
            extraRecipients=["buildbot@dev.local"],
            sendToInterestedUsers=False,
            mode='problem'
))

Create Slave

The buildslaves are typically run on a variety of separate machines, at least one per platform of interest. These machines connect to the buildmaster over a TCP connection to a publically-visible port. As a result, the buildslaves can live behind a NAT box or similar firewalls, as long as they can get to buildmaster. Here the build master is located at localhost.

buildslave create-slave --umask=027 --relocatable \
  --keepalive=120 --maxdelay=30 --log-size=512000 \
  --log-count=3 linux-slave1 localhost linux-slave1 \
  <password>

chown -R buildbot:buildbot linux-slave1/

Repeat above but for linux-slave2.

Configure Buildbot Daemon

The Debian installer for buildbot comes with configuration file (/etc/default/buildslave) that let you auto start build slaves.

SLAVE_RUNNER=/usr/bin/buildslave

# NOTE: SLAVE_ENABLED has changed its behaviour in version 0.8.4. Use
# 'true|yes|1' to enable instance and 'false|no|0' to disable. Other
# values will be considered as syntax error.

# 1-enabled, 0-disabled
SLAVE_ENABLED[1]=1
# short name printed on start/stop
SLAVE_NAME[1]="linux-slave1"
# user to run slave as
SLAVE_USER[1]="buildbot"
# basedir to slave (absolute path)
SLAVE_BASEDIR[1]="/var/lib/buildbot/slaves/linux-slave1"
# buildbot options
SLAVE_OPTIONS[1]=""
# prefix command, i.e. nice, linux32, dchroot
SLAVE_PREFIXCMD[1]=""

# 1-enabled, 0-disabled
SLAVE_ENABLED[2]=1
# short name printed on start/stop
SLAVE_NAME[2]="linux-slave2"
# user to run slave as
SLAVE_USER[2]="buildbot"
# basedir to slave (absolute path)
SLAVE_BASEDIR[2]="/var/lib/buildbot/slaves/linux-slave2"
# buildbot options
SLAVE_OPTIONS[2]=""
# prefix command, i.e. nice, linux32, dchroot
SLAVE_PREFIXCMD[2]=""            

You should be able to start buildbot slaves.

# /etc/init.d/buildslave start
Restarting buildslave "linux-slave1".
Restarting buildslave "linux-slave2".

Look at linux-slave1/twisted.log for errors:

tail -f linux-slave1/twisted.log

Open browser to http://localhost:8010/ (change localhost to the buildbot server name) to trigger your build.