path: root/README.buildd-admin

                                                    -*- mode: indented-text -*-

                  Overview of buildd Administration
                  =================================

This document should give you a rough idea how to administer a build
daemon, i.e. basics of how it works, and how to reply to the mails sent
to you. It does not yet cover configuration of buildd in a proper way,
sorry.


1) Overview of Operation
========================

The buildd system consists of the following parts:

  buildd: This is the daemon itself. It basically runs in a loop,
    first requesting some package in state Needs-Build from the
    wanna-build database and then compiling those packages.

  buildd-mail: This script processes mails directed to buildd, i.e.
    your answers to build logs etc.

  buildd-uploader: This script uploads finished jobs regularly and is
    called by cron.

  buildd-watcher: This does some regular maintenance stuff, like
    archiving log files and finding outdated files so that the disk
    doesn't fill up with them. It also restarts buildd should it
    have died for some reason.

1.1) buildd
-----------

The most important component is of course buildd itself. It runs as a
daemon, i.e. all the time, no need to start it by cron or the like. It
usually lowers its priority (config var $nice_level), so that other
foreground jobs aren't delayed too much.

buildd runs in a permanent loop where it tries to compile packages. To
do this, it needs to know which packages to build. To determine this, it
calls wanna-build --list=needs-build for all distributions (in order
stable -> frozen -> unstable) and if the output is non-empty, it takes
some of those packages needing rebuilding. You can restrict which
distributions to look at with the @take_from_dists config var, and how
many packages to take at a time with $max_build.

Additionally, there is a file called REDO in ~/build, where
some packages are occasionally dumped to be picked up later by buildd.
This file is checked before new packages are requested from
wanna-build. REDO also contains distributions for the packages, and
here again stable has priority before frozen, which has priority over
unstable. The rationale is that stable/frozen packages are usually more
urgent than unstable ones.

If buildd has some packages from wanna-build, it tries to take them for
building (wanna-build --take). This should usually go ok, but in rare
cases it can happen that somebody else takes the package in the time
between listing and taking (race condition). Those packages are ignored.
Another exception that can happen are packages that failed in the previous
version. To avoid unnecessary work, buildd doesn't immediately start to build
those packages but asks its administrator first. This will be discussed in
greater length in a following section ("Should I build" mails).

If a set of packages are taken (either from wanna-build or from REDO),
buildd starts an sbuild run to build all the packages. sbuild will send
log mails to the admin, who will (later) send answers, but this doesn't
touch buildd anymore. It goes back to the start of the loop and tries
to take more packages and build them...

Some special notes:

buildd automatically detects if either the config file ~/buildd.conf
or its binary has been changed while it is running, and it takes
appropriate action. So there's no need to restart the daemon or send
it some signal.

You can interrupt a running buildd by sending it a SIGINT or SIGTERM.
It will then kill a subsidiary sbuild process (if currently running).
sbuild also has shutdown handling for this, and dumps unfinished jobs
into the REDO file, so they'll be retried later. buildd blocks these
signals if they could lead to inconsistencies.

There is also some way to influence which packages buildd takes (or
more precisely) doesn't take to build. The config variable
@no_auto_build lists packages that should never be taken by buildd,
either because auto-building doesn't work or because they need to much
time on this machine, or ... There is also a @weak_no_auto_build for
packages that usually are too (time-)expensive to build here, but
should be taken nevertheless if the daemon would be idle otherwise.
Packages in @weak_no_auto_build are taken only if there are no other
packages left in state Needs-Build in any distribution, and also REDO
is empty.

If sbuild gives back a package to wanna-build (because source couldn't
be downloaded or the like), buildd will find the SBUILD-GIVEN-BACK
file created by sbuild and it will remember when those give-backs have
happened. To avoid trying the same package(s) over and over, such
given back packages aren't taken immediately again. They have a
waiting period of $delay_after_give_back minutes.


1.2 buildd-mail
---------------

buildd-mail is the mail processor of the buildd system. It should be
listed in the user buildd's ~/.forward file, e.g.:

  "|/usr/bin/buildd-mail"

The wrapper is necessary to allow setuid-ness, and also queues up
incoming mails in ~/mqueue, so that the MTA process can exit and
doesn't need to wait for buildd-mail to finish.

There's not much to say here about operating buildd-mail, because
it just receives your answers to log files etc., and this will be
described in more detail in the next section.

The only (a bit) special thing is that buildd-mail also receives the
"INSTALLED" mails from dinstall and analyzes them. It deletes all
files from ~/upload when they're installed in the master archive.
It also receives the "new version of ..." mails from wanna-build, that
are sent whenever a new version of a package in state Building
arrives.

There is also a file ~/mail-blacklist that contains a list of (Perl)
regular expressions. If a mail arrives from an address that matches
one of those patterns, the mail is deleted. This is simply to avoid
spam.


1.3 buildd-uploader
-------------------

buildd-uploader is started by cron a few times per day and uploads all
finished jobs which are in ~/upload. It simply uses dupload, as you would
expect. It of course ignores jobs for which an *.upload file already
exists, i.e. those which already have been uploaded. After uploading,
it calls wanna-build --uploaded for all the packages it processed.

The --to option of dupload can be set with the $dupload_to config var;
default is "erlangen". In all other respects, dupload is configured
with ~/.duploadrc.

In case of upload errors it sends a log mail to the admin, who has to
fix the situation manually. However, errors should be seldom. The
*.upload file of a failed job is deleted, so it's retried as a whole
in the next run.


1.4 buildd-watcher
------------------

buildd-watcher performs some regular maintenance tasks:

 - If checks if buildd is running and restarts it if not. This is
   meant as a safeguard against spuriously crashing daemons, no manual
   intervention is required to restart it.

   If you don't really need to have a daemon running, create a file
   NO-DAEMON-PLEASE in the daemon's home dir. Then buildd-watcher
   won't restart buildd.

 - It purges build directories that are no longer needed. This would
   be actually the job of buildd-mail, but since the latter is started
   setuid, it can't use sudo and the build dirs can contain root-owned
   files. So buildd-mail just appends the name of a directory to
   delete to ~/build/PURGE, and buildd-watcher reads that file and
   removes all the directories.

 - It checks for old files in the build/ and upload/ directories, so
   that no forgotten files can fill up the disk over time. Files are
   considered "old" if they're older than $warning_age days. There's
   additionally a Perl regexp $no_warn_pattern. buildd-watcher doesn't
   warn about files matching this pattern. So files expected to be
   there should be included in it, like build/buildd.pid, build/REDO,
   ...

 - It archives old log files into ~/old-logs/ and rotates the main log
   file ~/daemon.log. Files to be archived are:

    - package logs: logs/*                   -> old-logs/plog-<DATE>.tar.gz
    - sbuild logs:  build/build*.log         -> old-logs/blog-<DATE>.tar.gz
    - daemon logs:  old-logs/daemon-*.log.gz -> old-logs/dlog-<DATE>.tar.gz

   Additionally daemon.log is renamed to daemon.log.old, and SIGHUP is
   sent to buildd to open the new log file. If a daemon.log.old already
   exists, it's compressed and moved to old-logs/daemon-<DATE>.log.gz.

   There are several config vars that control long to keep the various
   log files until they're archived: $pkg_log_keep, $build_log_keep,
   $daemon_log_rotate, and $daemon_log_keep. All are measured in days.


2) Replying to Mails
====================

You'll receive lots of mail from buildd and its friends, most of it
build logs. But there are also some other kinds of mail: "Should I
build?" from buildd, dupload errors, and warnings about old files. The
latter two are not complicated and are obvious to handle. But the former
two require that you reply in a certain way.

To make daily administration easier, there is a set of Emacs Lisp
functions (for Rmail and Gnus) that allow you to send most answers
with two keystrokes. The library is called buildd-reply.el and can be
found in the wanna-build Git archive, where other buildd scripts reside.
The keystrokes in Emacs are given in square brackets.

2.1) Build Logs
---------------

All build log replies are recognized by their subject. It should be
of the form

  Re: Log for (successful|failed) build of PACKAGE_VERSION (dist=DIST)

The subject line provides buildd-mail with a lot of needed info: The
package name, version, and for which distribution it was compiled. The
successful/failed information is ignored.

The body of the reply mail is in most cases only a single keyword,
sometimes with arguments. Despite this, there should be no excess blank
lines or similar.


2.1.1 Successful [C-c C-o]
................

This is the most often used reply, and also different from the others
because it does not consist of a single keyword. Instead, you send
back a signed version of the .changes file of the package. The
.changes can be found at the end of the build log. The Emacs function
automatically extracts the .changes from the log and signs it (with
mailcrypt). If you do not Emacs, you have to cut the .changes manually
somehow (depending on the mailer) and invoke some kind of PGP backend.

buildd-mail recognizes "ok" mails by the "--- BEGIN PGP SIGNED MESSAGE"
line. If such a mail is received, it first checks that the package
isn't outdated.


2.1.2 retry [C-c C-r]
...........

2.1.3 fail [C-c C-f]
..........

2.1.4 dep-wait [C-c C-d]
.............

2.1.5 giveback [C-c M-g]
..............

2.1.6 manual [C-c C-m]
............

2.1.7 newvers [C-c C-n]
.............

2.1.8 not-for-us [C-c M-n]
................

2.1.9 upload-remove [C-c C-]
...................

2.1.10 purge [C-c C-p]
............


2.2 Other Emacs Actions
-----------------------

"\C-c\C-s" 'buildd-edit-manual-source-deps
"\C-c\C-b" 'buildd-bug
"\C-c\C-a" 'buildd-bug-ack-append


3) Log Files
============