diff options
Diffstat (limited to '')
| -rw-r--r-- | docs/FAQ | 219 | ||||
| -rw-r--r-- | docs/Makefile.am | 4 | ||||
| -rwxr-xr-x | docs/bzip.example | 35 | ||||
| -rwxr-xr-x | docs/changelogs.example | 246 | ||||
| -rw-r--r-- | docs/changestool.1 | 172 | ||||
| -rwxr-xr-x | docs/copybyhand.example | 28 | ||||
| -rw-r--r-- | docs/di.example/DI-filter.sh | 40 | ||||
| -rw-r--r-- | docs/di.example/README | 13 | ||||
| -rw-r--r-- | docs/di.example/distributions | 23 | ||||
| -rw-r--r-- | docs/di.example/updates | 5 | ||||
| -rwxr-xr-x | docs/mail-changes.example | 69 | ||||
| -rw-r--r-- | docs/manual.html | 1497 | ||||
| -rwxr-xr-x | docs/outsftphook.py | 589 | ||||
| -rwxr-xr-x | docs/outstore.py | 237 | ||||
| -rwxr-xr-x | docs/pdiff.example | 255 | ||||
| -rw-r--r-- | docs/recovery | 67 | ||||
| -rw-r--r-- | docs/reprepro.1 | 2847 | ||||
| -rw-r--r-- | docs/reprepro.bash_completion | 742 | ||||
| -rw-r--r-- | docs/reprepro.zsh_completion | 554 | ||||
| -rw-r--r-- | docs/rredtool.1 | 90 | ||||
| -rwxr-xr-x | docs/sftp.py | 886 | ||||
| -rw-r--r-- | docs/short-howto | 209 | ||||
| -rw-r--r-- | docs/xz.example | 30 |
23 files changed, 8857 insertions, 0 deletions
diff --git a/docs/FAQ b/docs/FAQ new file mode 100644 index 0000000..f5d42a4 --- /dev/null +++ b/docs/FAQ @@ -0,0 +1,219 @@ +This is a list of "frequently" asked questions. + +1.1) What can I do when reprepro complains about a missing .orig.tar.gz? +1.2) Why does it refuse a file when one in another suite has the same name? +1.4) The key to sign my Release files needs a passphrase, what to do? +1.5) How do I change how files are downloaded. +1.6) How to omit packages missing files when updating. +2.1) Does reprepro support to generate Release.gpg files? +2.2) Does reprepro support tildes ('~') in versions? +2.3) Does reprepro support generation of Contents-<arch>.gz files? +3.1) Can I have two versions of a package in the same distribution? +3.2) Can reprepro pass through a server-supplied Release.gpg? +9) Feature ... is missing, can you add it? + + +1.1) What can I do when reprepro complains about a missing .orig.tar.gz? +------------------------------------------------------------------------ +When 'include'ing a .changes file reprepro by default only adds files +referenced in the .changes file into the pool/-hierarchy and does not +search for files referenced in a .dsc file and thus fails if this .orig.tar.gz +is not already in the pool. +You are facing the choice: +- copy the .orig.tar.gz by hand into the appropriate place within pool/ + and try again. reprepro will find it there when you try it the next time + and add it to its database. +- use --ignore=missingfile to tell reprepro to search for such files + in the directory the .changes file resides in. +- modify the .changes file by hand to reference the .orig.tar.gz +- use changestool (comes with reprepro since version 1.3.0) to + list the file. ("changestool <.changesfile> includeallsources") +- use dpkg-buildpackage -sa the next time you build a package so that + it calls dpkg-genchanges with -sa which then always lists .orig.tar.gz + and not only if it ends in -0 or -1. + +1.2) Why does it refuse a file when one in another suite has the same name? +---------------------------------------------------------------------------- +Reprepro uses Debian's way to organize the pool hierarchy, which means +that the directory and name a file is saved under is only determined by +its sourcename, its name and its version and especially not by the +distribution it belongs to. (This is the intent of having a pool directory, +so that if two distributions have the same version, the disk space is only +used once). This means that if different versions of a packaged having the +same version string are put in the same reprepro repository (even if put +into different distributions within that), reprepro will refuse to do so. +(Unless you have a md5sum collision, in which case it will put the one and +just replace the second with the first). + +The only way to work around, is too put the different distributions into +different repositories. But in general it is really worth the effort to +get the versioning right instead: Having multiple packages with the same +version make it hard to track down problems, because it is easy to mix +them up. Also up/downgrading a host from one distribution to the other +will not change the package but just keep the old (as they are the +same version, so they have to be the same, apt and dpkg will think). + +How to deal with this without separating repositories depends on how +you reached this situation: + +- in the past Debian's stable and stable-security buildds sometimes both + built a package and for some architectures the one version entered + security.debian.org and the other ftp.debian.org with the next + point release. (This should be fixed now. And it is always considered + a bug, so if you hit this, please report it). If you mirror such + a situation, just update one of the distributions and manually + include the package into the other distribution. As the versions + are the same, reprepro will keep with this and not try to download + the other version, err other same version, err ... +- backports (i.e. packages rebuild for older distributions) + Common practise is to append the version with reducing ~, + i.e. 1.0-1 becomes 1.0-1~bpo.7, or 3.0 becomes 3.0~sarge. + (This makes sure that if a host is updated the backport is + replaced by the real package). + If backporting to multiple distributions you get bonus points + for making sure newer distributions have higher version numbers. + (To make sure which version is considered newer by dpkg use + dpkg's --compare-versions action). +- a package built for multiple distributions + is equivalent to the backports case +- locally modified packages that should be replace by newer official + versions: append something like "a0myname". If it should be + replaced by security updates of the official package, make sure (using + dpkg's --compare-versions) that a security update would have + a higher version. +- locally modified packages that should not be replaced by newer + official versions: prefix the version with "99:" and perhaps appending + it with something like "-myname". (appending only makes it easier to + distinguish, as some tools do not show epochs). + +1.4) The key to sign my Release files needs a passphrase, what to do? +--------------------------------------------------------------------- +Please take a look at gpg-agent. +You can also use the --ask-passphrase option, but please note this is quite insecure. + +1.5) How do I change how files are downloaded. +---------------------------------------------- +reprepro just calls apt's methods for file retrieval. +You can give them options in conf/updates like +Config: Acquire::Http::Proxy=http://proxy.yours.org:8080 +or replace them with other programs speaking the same +interface. + +1.6) How to omit packages missing files when updating. +------------------------------------------------------ +reprepro does not like broken upstream repositories and just splits out +errors and does not process the rest. (Implementing simply a ignore for +that is not that easy, as I would have special case holding an old version +in that case when unavailable packages should be deleted, and make some +costly information-pushing between layers (after all, each file can belong +to multiple packages and packages can have more than one file, so keeping +track which package should get a mark that files where missing needs a +n-to-n relation that should never be uses expect the case where such a +error happens)). +What you can do when a upstream repository you update from misses a file: +- try once with a different mirror not missing those files. You can either + change the mirror to use once and change it back afterwards. Or if both + mirrors have the same inner directory structure (they usually have) and + are accessible via the same method (like both http or both ftp) you can + also use the Fallback: option in conf/updates to tell reprepro to get + missing files from the other Mirror. This an even be used for things not + being a mirror of the same thing, but only having some files at the same + place. For example to work around etch r1 listing many older kernel + packages but no longer having the needed files, a line + Fallback: http://snapshot.debian.net/archive/2007/04/02/debian/ + can help. (But make sure to look at the run and remove this line + once reprepro downloaded the missing files. With this line active and + the primary mirror you list in Method: unreachable, reprepro will also + download index files from snapshot and make your repository a copy of + unstable from 2007-04-02 instead of an updated etch version.) +- get the file elsewhere (with the correct md5sum), place it in the + appropriate place in the pool/ hierarchy and do the update. Reprepro will + see the file is already there, add it to its database and just continue + with the update. +- tell reprepro to exclude this package +* There are multiple ways to do so. Easiest is adding something like + FilterFormula: package (!= xen-shell) + or + FilterFormula: package (!= xen-shell) | version (!=1.0-2) | !files + to your rule in conf/updates. ( the "| ! files" tells it to only + omit the source package xen-shell, as source packages have a files + field. Make sure the package in question does not require you to + make the source available or you are not making your repository + accessible to others). +* Another way is adding something like + FilterList: install excludefile + and adding a file conf/excludefile with content + xen-shell deinstall + (the install will tell it to install what is not listed in the file, + the deinstall on xen-shell will tell it to not install that package) +* Finally you can also supply a ListHook: with a script copying + its first argument to the second argument, removing all occurrences + of the package you do not want (take a look intro the dctrl-tool + package for tools helping you with this). +- the worst solution is to just propagate the problem further, by just + telling reprepro the file is there with the correct md5sum while it + is not. (Via the _addmd5sums command of reprepro). Unless you + run checkpool reprepro will not notice what you have done and will + not even try to download that file once it becomes available. So + don't do this. + +2.1) Does reprepro support to generate Release.gpg files? +--------------------------------------------------------- +Yes, add a SignWith in the suite's definition in conf/distributions. +(and take a look what the man page says about SignWith) + +2.2) Does reprepro support tildes ('~') in versions? +---------------------------------------------------- +Yes, but in .changes files only since version 0.5. +(You can work around this in older versions by using includedeb and + includedsc on the .deb and .dsc files within the .changes file, though) + +2.3) Does reprepro support generation of Contents-<arch>.gz files? +------------------------------------------------------------------ +Yes, since version 1.1.0 (well, actually since 0.8.2 but a bug +caused the generated files to not be up to date unless manually +exporting the distributions in question). +Look for "Contents" in the man page. + +3.1) Can I have two versions of a package in the same distribution? +------------------------------------------------------------------- +Sorry, this is not possible right now, as reprepro heavily optimizes +at only having one version of a package in a suite-type-component-architecture +quadruple. +You can have different versions in different architectures and/or components +within the same suite. (Even different versions of a architecture all package +in different architectures of the same suite). But within the same +architecture and the same component of a distribution it is not possible. + +3.2) Can reprepro pass through a server-supplied Release.gpg? +------------------------------------------------------------------- +No. +The reason for this is that the Release file will be different, +so a Release.gpg would not match. +The reason that the Release file looks differently is that reprepro +mirrors packages. While it can create a distribution with the same +packages as a distribution it mirrors. It will decide on its own where +to put the files, so their Filename: or Directory: might differ. It may +create a different set of compressions for the generated index files. +It does not mirror Packages.diff directories (but only comes with helpers +to create diffs between different states of the mirror). It does not mirror +Contents files but creates them; and so on. So to be able to mirror +distribution signatures almost all the functionality of reprepro would need +to be duplicated (once supporting literal mirroring, once support local +packages, partial mirroring, merging mirroring, pool condensing), thus I +decided that this is better a task for another program. (Note that if +you already have a local literal mirror, you can also use that as upstream +for partial/merged/extended mirrored distributions of that. If you use +the file:/// in Method: (as opposed to copy:///), reprepro will make +hardlinks for files in pool/ if possible). + +9) Feature ... is missing, can you add it? +------------------------------------------ +First, please take another look at the man page. My documentation is not +very good, so it is easy to overlook some feature even when it is described +already. If it is not there, just write me a mail (or better write a wishlist +report to the Debian BTS, then it cannot get lost). Some things I add quite +fast, other stuff takes a bit. Things incompatible with the current underlying +infrastructures or past design decisions may never come, but if I have it on the +TODO list of things to add, it help the code to develop in a direction that +things like that might be possible in the future. diff --git a/docs/Makefile.am b/docs/Makefile.am new file mode 100644 index 0000000..197b7fe --- /dev/null +++ b/docs/Makefile.am @@ -0,0 +1,4 @@ + +EXTRA_DIST = short-howto reprepro.1 changestool.1 rredtool.1 recovery bzip.example xz.example pdiff.example di.example/README di.example/DI-filter.sh di.example/distributions di.example/updates reprepro.bash_completion reprepro.zsh_completion FAQ changelogs.example manual.html copybyhand.example outstore.py sftp.py outsftphook.py +man_MANS = reprepro.1 changestool.1 rredtool.1 +MAINTAINERCLEANFILES = $(srcdir)/Makefile.in diff --git a/docs/bzip.example b/docs/bzip.example new file mode 100755 index 0000000..ca5db42 --- /dev/null +++ b/docs/bzip.example @@ -0,0 +1,35 @@ +#!/bin/sh +# since reprepro 0.8 this is no longer needed, as it can +# create .bz2 files on its own (when compiled with libbz2-dev +# present). It's still here for reference how to such a filter works. + +# Copy this script to your conf/ dir as bzip2.sh, make it executable +# and add to some definition in conf/distributions +# DscIndices: Sources Release . .gz bzip2.sh +# DebIndices: Packages Release . .gz bzip2.sh +# UDebIndices: Packages . .gz bzip2.sh +# and you have .bz2'd Packages and Sources. +# (alternatively, if you are very brave, put the full path to this file in there) + +DIROFDIST="$1" +NEWNAME="$2" +OLDNAME="$3" +# this can be old($3 exists), new($2 exists) or change (both): +STATUS="$4" +BASENAME="`basename "$OLDNAME"`" + +# with reprepro <= 0.7 this could also be Packages.gz or Sources.gz, +# but now only the uncompressed name is given. (even if not generated) +if [ "xPackages" = "x$BASENAME" ] || [ "xSources" = "x$BASENAME" ] ; then + if [ "x$STATUS" = "xold" ] ; then + if [ -f "$DIROFDIST/$OLDNAME.bz2" ] ; then + echo "$OLDNAME.bz2" >&3 + else + bzip2 -c -- "$DIROFDIST/$OLDNAME" >"$DIROFDIST/$OLDNAME.bz2.new" 3>/dev/null + echo "$OLDNAME.bz2.new" >&3 + fi + else + bzip2 -c -- "$DIROFDIST/$NEWNAME" >"$DIROFDIST/$OLDNAME.bz2.new" 3>/dev/null + echo "$OLDNAME.bz2.new" >&3 + fi +fi diff --git a/docs/changelogs.example b/docs/changelogs.example new file mode 100755 index 0000000..22ec3f9 --- /dev/null +++ b/docs/changelogs.example @@ -0,0 +1,246 @@ +#!/bin/sh +# This is an example script that can be hooked into reprepro +# to either generate a hierarchy like packages.debian.org/changelogs/ +# or to generate changelog files in the "third party sites" +# location apt-get changelogs looks if it is not found in +# Apt::Changelogs::Server. +# +# All you have to do is to: +# - copy it into you conf/ directory, +# - if you want "third party site" style changelogs, edit the +# CHANGELOGDIR variable below, +# and +# - add the following to any distribution in conf/distributions +# you want to have changelogs and copyright files extracted: +#Log: +# --type=dsc changelogs.example +# (note the space at the beginning of the second line). +# This will cause this script to extract changelogs for all +# newly added source packages. (To generate them for already +# existing packages, call "reprepro rerunnotifiers"). + +# DEPENDENCIES: dpkg >= 1.13.9 + +if test "x${REPREPRO_OUT_DIR:+set}" = xset ; then + # Note: due to cd, REPREPRO_*_DIR will no longer + # be usable. And only things relative to outdir will work... + cd "${REPREPRO_OUT_DIR}" || exit 1 +else + # this will also trigger if reprepro < 3.5.1 is used, + # in that case replace this with a manual cd to the + # correct directory... + cat "changelog.example needs to be run by reprepro!" >&2 + exit 1 +fi + +# CHANGELOGDIR set means generate full hierarchy +# (clients need to set Apt::Changelogs::Server to use that) +CHANGELOGDIR=changelogs + +# CHANGELOGDIR empty means generate changelog (and only changelog) files +# in the new "third party site" place apt-get changelog is using as fallback: +#CHANGELOGDIR= + +# Set to avoid using some predefined TMPDIR or even /tmp as +# tempdir: + +# TMPDIR=/var/cache/whateveryoucreated + +if test -z "$CHANGELOGDIR" ; then +addsource() { + DSCFILE="$1" + CANONDSCFILE="$(readlink --canonicalize "$DSCFILE")" + CHANGELOGFILE="${DSCFILE%.dsc}.changelog" + BASEDIR="$(dirname "$CHANGELOGFILE")" + if ! [ -f "$CHANGELOGFILE" ] ; then + EXTRACTDIR="$(mktemp -d)" + (cd -- "$EXTRACTDIR" && dpkg-source --no-copy -x "$CANONDSCFILE" > /dev/null) + install --mode=644 -- "$EXTRACTDIR"/*/debian/changelog "$CHANGELOGFILE" + chmod -R u+rwX -- "$EXTRACTDIR" + rm -r -- "$EXTRACTDIR" + fi + if [ -L "$BASEDIR"/current."$CODENAME" ] ; then + # should not be there, just to be sure + rm -f -- "$BASEDIR"/current."$CODENAME" + fi + # mark this as needed by this distribution + ln -s -- "$(basename "$CHANGELOGFILE")" "$BASEDIR/current.$CODENAME" + JUSTADDED="$CHANGELOGFILE" +} +delsource() { + DSCFILE="$1" + CHANGELOGFILE="${DSCFILE%.dsc}.changelog" + BASEDIR="$(dirname "$CHANGELOGFILE")" + BASENAME="$(basename "$CHANGELOGFILE")" + if [ "x$JUSTADDED" = "x$CHANGELOGFILE" ] ; then + exit 0 + fi +# echo "delete, basedir=$BASEDIR changelog=$CHANGELOGFILE, dscfile=$DSCFILE, " + if [ "x$(readlink "$BASEDIR/current.$CODENAME")" = "x$BASENAME" ] ; then + rm -- "$BASEDIR/current.$CODENAME" + fi + NEEDED=0 + for c in "$BASEDIR"/current.* ; do + if [ "x$(readlink -- "$c")" = "x$BASENAME" ] ; then + NEEDED=1 + fi + done + if [ "$NEEDED" -eq 0 -a -f "$CHANGELOGFILE" ] ; then + rm -r -- "$CHANGELOGFILE" + # to remove the directory if now empty + rmdir --ignore-fail-on-non-empty -- "$BASEDIR" + fi +} + +else # "$CHANGELOGDIR" set: + +addsource() { + DSCFILE="$1" + CANONDSCFILE="$(readlink --canonicalize "$DSCFILE")" + TARGETDIR="${CHANGELOGDIR}/${DSCFILE%.dsc}" + SUBDIR="$(basename $TARGETDIR)" + BASEDIR="$(dirname $TARGETDIR)" + if ! [ -d "$TARGETDIR" ] ; then + #echo "extract $CANONDSCFILE information to $TARGETDIR" + mkdir -p -- "$TARGETDIR" + EXTRACTDIR="$(mktemp -d)" + (cd -- "$EXTRACTDIR" && dpkg-source --no-copy -x "$CANONDSCFILE" > /dev/null) + install --mode=644 -- "$EXTRACTDIR"/*/debian/copyright "$TARGETDIR/copyright" + install --mode=644 -- "$EXTRACTDIR"/*/debian/changelog "$TARGETDIR/changelog" + chmod -R u+rwX -- "$EXTRACTDIR" + rm -r -- "$EXTRACTDIR" + fi + if [ -L "$BASEDIR"/current."$CODENAME" ] ; then + # should not be there, just to be sure + rm -f -- "$BASEDIR"/current."$CODENAME" + fi + # mark this as needed by this distribution + ln -s -- "$SUBDIR" "$BASEDIR/current.$CODENAME" + JUSTADDED="$TARGETDIR" +} +delsource() { + DSCFILE="$1" + TARGETDIR="${CHANGELOGDIR}/${DSCFILE%.dsc}" + SUBDIR="$(basename $TARGETDIR)" + BASEDIR="$(dirname $TARGETDIR)" + if [ "x$JUSTADDED" = "x$TARGETDIR" ] ; then + exit 0 + fi +# echo "delete, basedir=$BASEDIR targetdir=$TARGETDIR, dscfile=$DSCFILE, " + if [ "x$(readlink "$BASEDIR/current.$CODENAME")" = "x$SUBDIR" ] ; then + rm -- "$BASEDIR/current.$CODENAME" + fi + NEEDED=0 + for c in "$BASEDIR"/current.* ; do + if [ "x$(readlink -- "$c")" = "x$SUBDIR" ] ; then + NEEDED=1 + fi + done + if [ "$NEEDED" -eq 0 -a -d "$TARGETDIR" ] ; then + rm -r -- "$TARGETDIR" + # to remove the directory if now empty + rmdir --ignore-fail-on-non-empty -- "$BASEDIR" + fi +} +fi # CHANGELOGDIR + +ACTION="$1" +CODENAME="$2" +PACKAGETYPE="$3" +if [ "x$PACKAGETYPE" != "xdsc" ] ; then +# the --type=dsc should cause this to never happen, but better safe than sorry. + exit 1 +fi +COMPONENT="$4" +ARCHITECTURE="$5" +if [ "x$ARCHITECTURE" != "xsource" ] ; then + exit 1 +fi +NAME="$6" +shift 6 +JUSTADDED="" +if [ "x$ACTION" = "xadd" -o "x$ACTION" = "xinfo" ] ; then + VERSION="$1" + shift + if [ "x$1" != "x--" ] ; then + exit 2 + fi + shift + while [ "$#" -gt 0 ] ; do + case "$1" in + *.dsc) + addsource "$1" + ;; + --) + exit 2 + ;; + esac + shift + done +elif [ "x$ACTION" = "xremove" ] ; then + OLDVERSION="$1" + shift + if [ "x$1" != "x--" ] ; then + exit 2 + fi + shift + while [ "$#" -gt 0 ] ; do + case "$1" in + *.dsc) + delsource "$1" + ;; + --) + exit 2 + ;; + esac + shift + done +elif [ "x$ACTION" = "xreplace" ] ; then + VERSION="$1" + shift + OLDVERSION="$1" + shift + if [ "x$1" != "x--" ] ; then + exit 2 + fi + shift + while [ "$#" -gt 0 -a "x$1" != "x--" ] ; do + case "$1" in + *.dsc) + addsource "$1" + ;; + esac + shift + done + if [ "x$1" != "x--" ] ; then + exit 2 + fi + shift + while [ "$#" -gt 0 ] ; do + case "$1" in + *.dsc) + delsource "$1" + ;; + --) + exit 2 + ;; + esac + shift + done +fi + +exit 0 +# Copyright 2007,2008,2012 Bernhard R. Link <brlink@debian.org> +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License version 2 as +# published by the Free Software Foundation. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program; if not, write to the Free Software +# Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02111-1301 USA diff --git a/docs/changestool.1 b/docs/changestool.1 new file mode 100644 index 0000000..e7e49f0 --- /dev/null +++ b/docs/changestool.1 @@ -0,0 +1,172 @@ +.TH CHANGESTOOL 1 "2010-03-19" "reprepro" REPREPRO +.SH NAME +changestool \- verify, dump, modify, create or fix Debian .changes files +.SH SYNOPSIS +.B changestool \-\-help + +.B changestool +[ +\fIoptions\fP +] +\fI.changes-filename\fP +\fIcommand\fP +[ +\fIper-command-arguments\fP +] +.SH DESCRIPTION +changestool is a little program to operate on Debian .changes files, +as they are produced by \fBdpkg\-genchanges\fP(1) and used to feed +built Debian packages into Debian repository managers like +.BR reprepro (1) +or +.BR dak . + +.SH EXAMPLES +.P +.B changestool \fIbloat.changes\fP setdistribution \fIlocal\fP +.br +will modify the \fBDistribution:\fP header inside \fIbloat.changes\fP +to say \fIlocal\fP instead of what was there before. +.P +.B changestool \fIreprepro_1.2.0\-1.local_sparc.changes\fP includeallsources +.br +will modify the given file to also list \fB.orig.tar.gz\fP it does not list +because you forgot to build it with +.BR "dpkg\-buildpackage \-sa" . +.P +.B changestool \fIblafasel_1.0_abacus.changes\fP updatechecksums +.br +will update the md5sums to those of the files referenced by this file. +(So one can do quick'n'dirty corrections to them before uploading to +your private package repository) +.P +.B changestool \-\-create \fItest.changes\fP add \fIbla_1\-1.dsc bla_1\-1_abacus.deb\fP +.br +will add the specified files (format detected by filename, +use \fBadddeb\fP or \fBadddsc\fP if you know it). +If the file \fItest.changes\fP does not exist yet, a minimal one will be +generated. Though that might be too minimal for most direct uses. + +.SH "GLOBAL OPTIONS" +Options can be specified before the command. Each affects a different +subset of commands and is ignored by other commands. +.TP +.B \-h \-\-help +Displays a short list of options and commands with description. +.TP +.B \-o \-\-outputdir \fIdir\fP +Not yet implemented. +.TP +.B \-s \-\-searchpath \fIpath\fP +A colon-separated list of directories to search for files if they +are not found in the directory of the .changes file. +.TP +.B \-\-create +Flag for the commands starting with \fBadd\fP to create the \fB.changes\fP +file if it does not yet exists. +.TP +.B \-\-create\-with\-all\-fields +Flag for the commands starting with \fBadd\fP to create the \fB.changes\fP +file if it does not yet exists. +Unlike \fB\-\-create\fP, this creates more fields to make things like dupload +happier. +Currently that creates fake \fBUrgency\fP and \fBChanges\fP fields. +.TP +.B \-\-unlzma \fIcommand\fP +External uncompressor used to uncompress lzma files to look +into .diff.lzma, .tar.lzma or .tar.lzma within .debs. +.TP +.B \-\-unxz \fIcommand\fP +External uncompressor used to uncompress xz files to look +into .diff.xz, .tar.xz or .tar.xz within .debs. +.TP +.B \-\-lunzip \fIcommand\fP +External uncompressor used to uncompress lzip files to look +into .diff.lz, .tar.lz or .tar.lz within .debs. +.TP +.B \-\-bunzip2 \fIcommand\fP +External uncompressor used to uncompress bz2 when compiled without +libbz2. +.SH COMMANDS +.TP +.BR verify +Check for inconsistencies in the specified \fB.changes\fP file and the +files referenced by it. +.TP +.BR updatechecksums " [ " \fIfilename\fP " ]" +Update the checksum (md5sum and size) information within the specified +\fB.changes\fP file and all \fB.dsc\fP files referenced by it. +Without arguments, all files will be updated. +To only update specific files, give their filename (without path) as +arguments. +.TP +.BR setdistribution " [ " \fIdistributions\fP " ]" +Change the \fBDistribution:\fP header to list the remaining arguments. +.TP +.BR includeallsources " [ " \fIfilename\fP " ]" +List all files referenced by \fB.dsc\fP files mentioned in the \fB.changes\fP +file in said file. +Without arguments, all missing files will be included. +To only include specific files, give their filename (without path) as +arguments. + +Take a look at the description of \fB\-si\fP, \fB\-sa\fP and \fB\-sd\fP in +the manpage of \fBdpkg\-genchanges\fP/\fBdpkg\-buildpackage\fP how to avoid +to have to do this at all. + +Note that while \fBreprepro\fP will just ignore files listed in a \fB.changes\fP +file when it already has the file with the same size and md5sum, \fBdak\fP +might choke in that case. +.TP +.B adddeb \fIfilenames\fP +Add the \fB.deb\fP and \fB.udeb\fP files specified by their filenames to +the \fB.changes\fP file. +Filenames without a slash will be searched +in the current directory, +the directory the changes file resides in +and in the directories specified by the \fB\-\-searchpath\fP. +.TP +.B adddsc \fIfilenames\fP +Add the \fB.dsc\fP files specified by their filenames to +the \fB.changes\fP file. +Filenames without a slash will be searched +in the current directory, +in the directory the changes file resides in +and in the directories specified by the \fB\-\-searchpath\fP. +.TP +.B addrawfile \fIfilenames\fP +Add the files specified by their filenames to +the \fB.changes\fP file. +Filenames without a slash will be searched +in the current directory, +in the directory the changes file resides in +and in the directories specified by the \fB\-\-searchpath\fP. +.TP +.B add \fIfilenames\fP +Behave like \fBadddsc\fP for filenames ending in \fB.dsc\fP, +like \fBadddeb\fP for filenames ending in \fB.deb\fP or \fB.udeb\fP, +and like \fBaddrawfile\fP for all other filenames +.TP +.B dumbremove \fIfilenames\fP +Remove the specified files from the .changes file. +No other fields (Architectures, Binaries, ...) are updated and +no related files is removed. +Just the given files (which must be specified without any \fB/\fP) +are no longer listen in the .changes file (and only no longer in the +changes file). + +.SH "SEE ALSO" +.BR reprepro (1), +.BR dpkg\-genchanges (1), +.BR dpkg\-buildpackage (1), +.BR md5sum (1). +.SH "REPORTING BUGS" +Report bugs or wishlist requests the Debian BTS +(e.g. by using \fBreportbug reperepro\fP) +or directly to <brlink@debian.org>. +.br +.SH COPYRIGHT +Copyright \(co 2006-2009 Bernhard R. Link +.br +This is free software; see the source for copying conditions. There is NO +warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. diff --git a/docs/copybyhand.example b/docs/copybyhand.example new file mode 100755 index 0000000..0b1762d --- /dev/null +++ b/docs/copybyhand.example @@ -0,0 +1,28 @@ +#!/bin/sh +# This is an example script for a byhandhook. +# Add to you conf/distributions something like +##ByhandHooks: +## * * * copybyhand.sh +# and copy this script as copybyhand.sh in your conf/ +# directory (or give the full path), and processincoming +# will copy all byhand/raw files to dists/codename/extra/* + +set -e + +if [ $# != 5 ] ; then + echo "to be called by reprepro as byhandhook" >&2 + exit 1 +fi +if [ -z "$REPREPRO_DIST_DIR" ] ; then + echo "to be called by reprepro as byhandhook" >&2 + exit 1 +fi + +codename="$1" +section="$2" +priority="$3" +basefilename="$4" +fullfilename="$5" + +mkdir -p "$REPREPRO_DIST_DIR/$codename/extra" +install -T -- "$fullfilename" "$REPREPRO_DIST_DIR/$codename/extra/$basefilename" diff --git a/docs/di.example/DI-filter.sh b/docs/di.example/DI-filter.sh new file mode 100644 index 0000000..38696a3 --- /dev/null +++ b/docs/di.example/DI-filter.sh @@ -0,0 +1,40 @@ +#!/bin/sh +# +# Select only debs needed for a D-I netinstall cd + +IN="$1" +OUT="$2" + +DIR=`dirname "$IN"` +FILE=`basename "$IN"` +CODENAME=`echo $FILE | cut -d"_" -f1` +COMPONENT=`echo $FILE | cut -d"_" -f4` +ARCH=`echo $FILE | cut -d"_" -f5` + +echo "### $IN" +echo "# Source: $IN" +echo "# Debs: $DIR/$FILE.debs" +echo "# Out: $OUT" +echo + +# generate list of packages needed +DEBCDDIR="/usr/share/debian-cd" +export ARCH CODENAME DEBCDDIR DIR +make -f "$DEBCDDIR/Makefile" \ + BDIR='$(DIR)' \ + INSTALLER_CD='2' \ + TASK='$(DEBCDDIR)/tasks/debian-installer+kernel' \ + BASEDIR='$(DEBCDDIR)' \ + forcenonusoncd1='0' \ + VERBOSE_MAKE='yes' \ + "$DIR/list" + +# hotfix abi name for sparc kernel +sed -e 's/-1-/-2-/' "$DIR/list" > "$DIR/$FILE.debs" +rm -f "$DIR/list" + +# filter only needed packages +grep-dctrl `cat "$DIR/$FILE.debs" | while read P; do echo -n " -o -X -P $P"; done | cut -b 5-` "$IN" >"$OUT" + +# cleanup +rm -f "$DIR/$FILE.debs" diff --git a/docs/di.example/README b/docs/di.example/README new file mode 100644 index 0000000..983ff5f --- /dev/null +++ b/docs/di.example/README @@ -0,0 +1,13 @@ +This is an example from Goswin Brederlow <brederlo@informatik.uni-tuebingen.de> +for the ListHook directive. + +He describes the example as: +> attached a sample config that mirrors only packages from the debian-cd +> netinstall CD image task. I think this would make a usefull example +> for making a partial mirror by filtering. + +The speciality of the example needing the ListHook and not +easer possible with FilterList is the need for different +packages in different architectured. (Though extending +FilterList to support this is on my TODO-List) + diff --git a/docs/di.example/distributions b/docs/di.example/distributions new file mode 100644 index 0000000..316a051 --- /dev/null +++ b/docs/di.example/distributions @@ -0,0 +1,23 @@ +Origin: Debian-Installer +Label: Debian-Installer +Suite: testing +Codename: sarge +Version: 3.1 +Architectures: sparc i386 +Components: main +UDebComponents: main +Description: Debian Installer partial mirror +Update: - debian +#SignWith: yes + +Origin: Debian-Installer +Label: Debian-Installer +Suite: unstable +Codename: sid +Version: 3.2 +Architectures: sparc i386 +Components: main +UDebComponents: main +Description: Debian Installer partial mirror +Update: - debian +#SignWith: yes diff --git a/docs/di.example/updates b/docs/di.example/updates new file mode 100644 index 0000000..af39f5d --- /dev/null +++ b/docs/di.example/updates @@ -0,0 +1,5 @@ +Name: debian +Architectures: sparc i386 +Method: http://ftp.de.debian.org/debian +#VerifyRelease: FBC60EA91B67D3C0 +ListHook: /mnt/mirror/DI/DI-filter.sh diff --git a/docs/mail-changes.example b/docs/mail-changes.example new file mode 100755 index 0000000..2402aaf --- /dev/null +++ b/docs/mail-changes.example @@ -0,0 +1,69 @@ +#!/bin/sh +# +# +# Copyright 2016 Luca Capello <luca.capello@infomaniak.com> +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License version 2 as +# published by the Free Software Foundation. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program; if not, write to the Free Software +# Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02111-1301 USA +# +# +# This is an example script that can be hooked into reprepro +# to send an email after a .changes file is processed. +# +# All you have to do is to: +# - copy it into you conf/ directory, +# - add the following to any distribution in conf/distributions +# you want to have emails sent for: +#Log: +# --changes mail-changes.example +# (note the space at the beginning of the second line). +# +# DEPENDENCIES: mailx + + +set -e + + +if test "x${REPREPRO_OUT_DIR:+set}" = xset ; then + # Note: due to cd, REPREPRO_*_DIR will no longer + # be usable. And only things relative to outdir will work... + cd "${REPREPRO_OUT_DIR}" || exit 1 +else + # this will also trigger if reprepro < 3.5.1 is used, + # in that case replace this with a manual cd to the + # correct directory... + cat "mail-accepted.example needs to be run by reprepro!" >&2 + exit 1 +fi + + +MAIL_TO="$USER" + +ACTION="$1" +CODENAME="$2" +PACKAGENAME="$3" +PACKAGEVERSION="$4" +CHANGESFILE="$5" + +if [ "x$ACTION" = "xaccepted" ]; then + MAIL_FROM="$(grep Changed-By $CHANGESFILE | \ + sed -e 's/Changed-By/From/')" + ARCHITECTURE="$(grep Architecture $CHANGESFILE | \ + sed -e 's/Architecture: //')" + MAIL_SUBJECT="Accepted $PACKAGENAME $PACKAGEVERSION ($ARCHITECTURE) into $CODENAME" + cat "$CHANGESFILE" | \ + mail -a "$MAIL_FROM" -s "$MAIL_SUBJECT" "$MAIL_TO" +fi + + +exit 0 diff --git a/docs/manual.html b/docs/manual.html new file mode 100644 index 0000000..af6a993 --- /dev/null +++ b/docs/manual.html @@ -0,0 +1,1497 @@ +<html><head> +<title>reprepro manual</title> +<!-- some style elements stolen from or inspired by bugs.debian.org /--> +<style> +<!-- +html { color: #000; background: #fefefe; font-family: serif; margin: 1em; border: 0; padding: 0; line-height: 120%; } +body { margin: 0; border: 0; padding: 0; } +pre { text-align: left; border: #f0f0f0 1px solid; padding: 1px;} +pre.shell { text-align: left; border: none; border-left: #f0f0f0 1px solid; padding: 2px;} +h1, h2, h3 { text-align: left; font-family: sans-serif; background-color: #f0f0ff; color: #3c3c3c; border: #a7a7a7 1px solid; padding: 10px;} +h1 { font-size: 180%; line-height: 150%; } +h2 { font-size: 150% } +h3 { font-size: 100% } +ul.dir { list-style-type: disc; } +ul { list-style-type: square; } +dt.dir, dt.file, dt.symlink { font-weight:bold; font-family: sans-serif; } +/--> +</style> +</head> +<body> +<h1>reprepro manual</h1> +This manual documents reprepro, a tool to generate and administer +Debian package repositories. +<br> +Other useful resources: +<ul> +<li> the <a href="http://mirrorer.alioth.debian.org/">homepage</a> of reprepro.</li> +<li> <a href="file://localhost/usr/share/doc/reprepro/">local directory</a> with documentation and examples, if you have reprepro installed.</li> +<li> the <a href="http://git.debian.org/?p=mirrorer/reprepro.git;a=blob_plain;f=docs/FAQ;hb=HEAD">Frequently Asked Questions</a></li> +</ul> +<h2>Table of contents</h2> +Sections of this document: +<ul> +<li><a href="#introduction">Introduction</a></li> +<li><a href="#firststeps">First steps</a></li> +<li><a href="#dirbasics">Repository basics</a></li> +<li><a href="#config">Config files</a></li> +<li><a href="#export">Generation of index files</a> + <ul> + <li><a href="#compression">Compression and file names</a></li> + <li><a href="#signing">Signing</a></li> + <li><a href="#contents">Contents files</a></li> + <li><a href="#exporthook">Additional index files (like .diff)</a></li> + </ul></li> +<li><a href="#localpackages">Local packages</a> + <ul> + <li><a href="#include">Including via command line</a></li> + <li><a href="#incoming">Processing an incoming queue</a></li> + </ul></li> +<li><a href="#mirroring">Mirroring</a></li> +<li><a href="#propagation">Propagation of packages</a></li> +<li><a href="#snapshosts">Snapshots</a> (TODO)</li> +<li><a href="#tracking">Source package tracking</a> (TODO)</li> +<li><a href="#hooks">Extending reprepro / Hooks and more</a></li> +<li><a href="#maintenance">Maintenance</a></li> +<li><a href="#internals">Internals</a></li> +<li><a href="#recovery">Disaster recovery</a></li> +<li><a href="#paranoia">Paranoia</a></li> +<li><a href="#counterindications">What reprepro cannot do</a></li> +</ul> +<h2><a name="introduction">Introduction</a></h2> +<h3>What reprepro does</h3> +Reprepro is a tool to take care of a repository of Debian packages +(<tt>.dsc</tt>,<tt>.deb</tt> and <tt>.udeb</tt>). +It installs them to the proper places, generates indices of packages +(<tt>Packages</tt> and <tt>Sources</tt> and their compressed variants) +and of index files (<tt>Release</tt> and optionally <tt>Release.gpg</tt>), +so tools like <tt>apt</tt> know what is available and where to get it from. +It will keep track which file belongs to where and remove files no longer +needed (unless told to not do so). +It can also make (partial) partial mirrors of remote repositories, +including merging multiple sources and +automatically (if explicitly requested) removing packages no longer available +in the source. +And many other things (sometimes I fear it got a few features too much). +<h3>What reprepro needs</h3> +It needs some libraries (<tt>zlib</tt>, <tt>libgpgme</tt>, <tt>libdb</tt> (Version 3, 4.3 or 4.4)) and can be compiled with some more for additional features (<tt>libarchive</tt>, +<tt>libbz2</tt>). +Otherwise it only needs +<tt>apt</tt>'s methods (only when downloading stuff), +<tt>gpg</tt> (only when signing or checking signatures), +and if compiled without <tt>libarchive</tt> it needs <tt>tar</tt> and <tt>ar</tt> installed. +<br> +If you tell reprepro to call scripts for you, you will of course need the interpreters for these scripts: +The included example to generate pdiff files needs python. The example to extract +changelogs needs dpkg-source. +<h3>What this manual aims to do</h3> +This manual aims to give some overview over the most important features, +so people can use them and so that I do not implement something a second +time because I forgot support is already there. +For a full reference of all possible commands and config options take a +look at the man page, as this manual might miss some of the more obscure +options. +<h2><a name="firststeps">First steps</a></h2> +<h3>generate a repository with local packages</h3> +<ul> +<li>Choose a directory (or create it).</li> +<li>Create a subdirectory called <tt>conf</tt> in there.</li> +<li>In the <tt>conf/</tt> subdirectory create a file called <tt>distributions</tt>, +with content like: +<pre class="file"> +Codename: mystuff +Components: main bad +Architectures: sparc i386 source +</pre> +or with content like: +<pre class="file"> +Codename: andy +Suite: rusty +Components: main bad +Architectures: sparc i386 source +Origin: myorg +Version: 20.3 +Description: my first little repository +</pre> +(Multiple distributions are separated by empty lines, Origin, Version and Description +are just copied to the generated Release files, more things controlling reprepro can +appear which are described later). +</li> +<li>If your <tt>conf/distributions</tt> file contained a <tt>Suite:</tt> and you +are too lazy to generate the symbolic links yourself, call: +<pre class="shell"> +reprepro -b $YOURBASEDIR createsymlinks +</pre> +</li> +<li>Include some package, like: +<pre class="shell"> +reprepro -b $YOURBASEDIR include mystuff mypackage.changes +</pre> +or: +<pre class="shell"> +reprepro -b $YOURBASEDIR includedeb mystuff mypackage.deb +</pre> +</li> +<li>Take a look at the generated <tt>pool</tt> and <tt>dists</tt> +directories. They contain everything needed to apt-get from. +Tell apt to include it by adding the following to your <tt>sources.list</tt>: +<pre class="file"> +deb file:///$YOURBASEDIR mystuff main bad +</pre> +or make it available via http or ftp and do the same <tt>http://</tt> or <tt>ftp://</tt> source.</li> +</ul> +<h3>mirroring packages from other repositories</h3> +This example shows how to generate a mirror of a single architecture with +all packages of etch plus security updates: +<ul> +<li>Choose a directory (or create it).</li> +<li>Create a subdirectory called <tt>conf</tt> in there (if not already existent).</li> +<li>In the <tt>conf/</tt> subdirectory create a file called <tt>distributions</tt>, +with content like (or add to that file after an empty line): +<pre class="file"> +Origin: Debian +Label: Debian +Suite: stable +Version: 4.0 +Codename: etch +Architectures: i386 +Components: main +Description: Debian 4.0 etch + security updates +Update: - debian security +Log: logfile +</pre> +Actually only <tt>Codename</tt>, <tt>Components</tt>, <tt>Architecture</tt> and <tt>Update</tt> is needed, the rest is just information for clients. +The <tt>Update</tt> line tells to delete everything no longer available (<tt>-</tt>), +then add the <tt>debian</tt> and <tt>security</tt> rules, which still have to be defined: +</li> +<li>In the <tt>conf/</tt> subdirectory create a file called <tt>updates</tt>, +with content like (or add to that file after an empty line:): +or with content like: +<pre class="file"> +Name: security +Method: http://security.debian.org/debian-security +Fallback: ftp://klecker.debian.org/debian-security +Suite: */updates +VerifyRelease: A70DAF536070D3A1|B5D0C804ADB11277 +Architectures: i386 +Components: main +UDebComponents: + +Name: debian +Method: http://ftp2.de.debian.org/debian +Config: Acquire::Http::Proxy=http://proxy.myorg.de:8080 +VerifyRelease: A70DAF536070D3A1|B5D0C804ADB11277 +</pre> +(If there are no Architecture, Components or UDebComponents, it will try all the distribution to update has. Fallback means a URL to try when the first cannot offer some file (Has to be the same method)). +</li> +<li>Tell reprepro to update: +<pre class="shell"> +reprepro -b $YOURBASEDIR update etch +</pre> +</li> +<li>Take a look at the generated <tt>pool</tt> and <tt>dists</tt> +directories. They contain everything needed to apt-get from. +Tell apt to include it by adding the following to your <tt>sources.list</tt>: +<pre class="shell"> +deb file:///$YOURBASEDIR etch main +</pre> +or make it available via http or ftp.</li> +</ul> +<h2><a name="dirbasics">Repository basics</a></h2> +An <tt>apt-get</tt>able repository of Debian packages consists of two parts: +the index files describing what is available and where it is and the actual +Debian binary (<tt class="suffix">.deb</tt>), +installer binary (<tt class="suffix">.udeb</tt>), +and source (<tt class="suffix">.dsc</tt> together with +<tt class="suffix">.tar.gz</tt> or +<tt class="suffix">.orig.tar.gz</tt> and +<tt class="suffix">.diff.gz</tt>) packages. +<br> +While you do not know how these look like to use reprepro, it's always a good +idea to know what you are creating. +<h3>Index files</h3> +All index files are in subdirectories of a directory called +<tt class="dirname">dists</tt>. Apt is very decided what names those should +have, including the name of <tt class="dirname">dists</tt>. +Including all optional and extensional files, the hierarchy looks like this: + +<dl class="dir"> +<dt class="dir">dists</dt><dd> + <dl class="dir"> + <dt class="dir">CODENAME</dt><dd> +Each distribution has it's own subdirectory here, named by it's codename. + <dl class="dir"> + <dt class="file">Release</dt><dd> +This file describes what distribution this is and the checksums of +all index files included. + </dd> + <dt class="file">Release.gpg</dt><dd> +This is the optional detached gpg signature of the Release file. +Take a look at the <a name="#signing">section about signing</a> for how to +active this. + </dd> + <dt class="file">Contents-ARCHITECTURE.gz</dt><dd> +This optional file lists all files and which packages they belong to. +It's downloaded and used by tools like +<a href="http://packages.debian.org/apt-file">apt-file</a> +to allow users to determine which package to install to get a specific file. +<br> +To activate generating of these files by reprepro, you need a <a href="#contents">Contents</a> +header in your distribution declaration. + </dd> + <dt class="dir">COMPONENT1</dt><dd> +Each component has it's own subdirectory here. They can be named whatever users +can be bothered to write into their <tt class="filename">sources.list</tt>, but +things like <tt>main</tt>, <tt>non-free</tt> and <tt>contrib</tt> are common. +But funny names like <tt>bad</tt> or <tt>universe</tt> are just as possible. + <dl class="dir"> + <dt class="dir">source</dt><dd> +If this distribution supports sources, this directory lists which source +packages are available in this component. + <dl class="dir"> + <dt class="file">Release</dt><dd> +This file contains a copy of those information about the distribution +applicable to this directory. + </dd> + <dt class="file">Sources</dt> + <dt class="file">Sources.gz</dt> + <dt class="file">Sources.bz2</dt><dd> +These files contain the actual description of the source Packages. By default +only the <tt class="suffix">.gz</tt> file created, to create all three add the +following to the declarations of the distributions: +<pre class="config"> +DscIndices Sources Release . .gz .bz2 +</pre> +That header can also be used to name those files differently, but then apt +will no longer find them... + </dd> + <dt class="dir">Sources.diff</dt><dd> +This optional directory contains diffs, so that only parts of the index +file must be downloaded if it changed. While reprepro cannot generate these +so-called <tt>pdiff</tt>s itself, it ships both with a program called rredtool +and with an example python script to generate those. + </dd> + </dl> + </dd> + </dl> + <dl class="dir"> + <dt class="dir">binary-ARCHITECTURE</dt><dd> +Each architecture has its own directory in each component. + <dl class="dir"> + <dt class="file">Release</dt><dd> +This file contains a copy of those information about the distribution +applicable to this directory. + </dd> + <dt class="file">Packages</dt> + <dt class="file">Packages.gz</dt> + <dt class="file">Packages.bz2</dt><dd> +These files contain the actual description of the binary Packages. By default +only the uncompressed and <tt class="suffix">.gz</tt> files are created. +To create all three, add the following to the declarations of the distributions: +<pre class="config"> +DebIndices Packages Release . .gz .bz2 +</pre> +That header can also be used to name those files differently, but then apt +will no longer find them... + </dd> + <dt class="dir">Packages.diff</dt><dd> +This optional directory contains diffs, so that only parts of the index +file must be downloaded if it changed. While reprepro cannot generate these +so-called <tt>pdiff</tt>s itself, it ships both with a program called rredtool +and with an example python script to generate those. + </dd> + </dl> + </dd> + <dt class="dir">debian-installer</dt><dd> +This directory contains information about the <tt class="suffix">.udeb</tt> +modules for the <a href="http://www.debian.org/devel/debian-installer/">Debian-Installer</a>. +Those are actually just a very stripped down form of normal <tt class="suffix">.deb</tt> +packages and this the hierarchy looks very similar: + + <dl class="dir"> + <dt class="dir">binary-ARCHITECTURE</dt><dd> + <dl class="dir"> + <dt class="file">Packages</dt><dd></dd> + <dt class="file">Packages.gz</dt><dd></dd> + </dl> + </dd> + </dl> + </dd> + </dl> + </dd> + <dt class="dir">COMPONENT2</dt><dd> +There is one dir for every component. All look just the same. + </dd> + </dl> + </dd> + <dt class="symlink">SUITE -> CODENAME</dt><dd> +To allow accessing distribution by function instead of by name, there are often +symbolic links from suite to codenames. That way users can write +<pre class="config"> +deb http://some.domain.tld/debian SUITE COMPONENT1 COMPONENT2 +</pre> +instead of +<pre class="config"> +deb http://some.domain.tld/debian CODENAME COMPONENT1 COMPONENT2 +</pre> +in their <tt class="filename">/etc/apt/sources.list</tt> and totally get +surprised by getting something new after a release. + </dd> + </dl> +</dd></dl> +<h3>Package pool</h3> +While the index files have a required filename, the actual files +are given just as relative path to the base directory you specify +in your sources list. That means apt can get them no matter what +scheme is used to place them. The classical way Debian used till +woody was to just put them in subdirectories of the +<tt class="dir">binary-ARCHITECTURE</tt> directories, with the exception +of the architecture-independent packages, which were put into a +artificial <tt class="dir">binary-all</tt> directory. This was replaced +for the official repository with package pools, which reprepro also uses. +(Actually reprepro stores everything in pool a bit longer than the official +repositories, that's why it recalculates all filenames without exception). +<br> +In a package pool, all package files of all distributions in that repository +are stored in a common directory hierarchy starting with <tt class="dir">pool/</tt>, +only separated by the component they belong to and the source package name. +As everything this has disadvantages and advantages: +<ul><li>disadvantages + <ul><li>different files in different distributions must have different filenames + </li><li>it's impossible to determine which distribution a file belongs to by path and filename (think mirroring) + </li><li>packages can no longer be grouped together in common subdirectories by having similar functions + </li></ul> +</li><li>advantages + <ul><li>the extremely confusing situation of having differently build packages with the same version if different distributions gets impossible by design. + </li><li>the source (well, if it exists) is in the same directory as the binaries generated from it + </li><li>same files in different distributions need disk-space and bandwidth only once + </li><li>each package can be found only knowing component and sourcename + </li></ul> +</li></ul> +Now let's look at the actual structure of a pool (there is currently no difference +between the pool structure of official Debian repositories and those generated by +reprepro): + +<dl class="dir"> +<dt class="dir">pool</dt><dd> + The directory all this resides in is normally called <tt class="dir">pool</tt>. + That's nowhere hard coded in apt but that only looks at the relative + directory names in the index files. But there is also no reason to name + it differently. + <dl class="dir"> + <dt class="dir">COMPONENT1</dt><dd> +Each component has it's own subdirectory here. +They can be named whatever users +can be bothered to write into their <tt class="filename">sources.list</tt>, but +things like <tt>main</tt>, <tt>non-free</tt> and <tt>contrib</tt> are common. +But funny names like <tt>bad</tt> or <tt>universe</tt> are just as possible. + <dl class="dir"> + <dt class="dir">a</dt><dd> +As there are really many different source packages, +the directory would be too full when all put here. +So they are separated in different directories. +Source packages starting with <tt class="constant">lib</tt> are put into a +directory named after the first four letters of the source name. +Everything else is put in a directory having the first letter as name. + <dl class="dir"> + <dt class="dir">asource</dt><dd> +Then the source package name follows. +So this directory <tt class="dir">pool/COMPONENT1/a/asource/</tt> would contain +all files of different versions of the hypothetical package <tt class="constant">asource</tt>. + <dl class="dir"> + <dt class="dir">asource</dt><dd> + <dt class="file">a-source_version.dsc</dt> + <dt>a-source_version.tar.gz</dt><dd> +The actual source package consists of its description file (<tt class="suffix">.dsc</tt>) +and the files references by that. + </dd> + <dt class="file">binary_version_ARCH1deb</dt> + <dt class="file">binary_version_ARCH2.deb</dt> + <dt class="file">binary2_version_all.deb</dt><dd> + <dt class="file">di-module_version_ARCH1.udeb</dt><dd> +Binary packages are stored here to. +So to know where a binary package is stored you need to know what its source package +name is. + </dd> + </dl> + </dd> + </dl> + </dd> + <dt class="dir">liba</dt><dd> +As described before packages starting with <tt class="constant">lib</tt> are not stored +in <tt class="dir">l</tt> but get a bit more context. + </dd> + </dl> + </dd> + <dt class="dir">COMPONENT2</dt><dd> +There is one dir for every component. All look just the same. + </dd> + </dl> +</dd></dl> +As said before, you don't need to know this hierarchy in normal operation. +reprepro will put everything to where it belong, keep account what is there +and needed by what distribution or snapshot, and delete files no longer needed. +(Unless told otherwise or when you are using the low-level commands). +<h2><a name="config">Config files</a></h2> +Configuring a reprepro repository is done by writing some config files +into a directory. +This directory is currently the <tt class="dir">conf</tt> subdirectory of the +base directory of the repository, +unless you specify <tt class="option">--confdir</tt> or set the +environment variable <tt class="env">REPREPRO_CONFIG_DIR</tt>. + +<dl class="dir"> +<dt class="dir">options</dt><dd> +If this file exists, reprepro will consider each line an additional +command line option. +Arguments must be in the same line after an equal sign. + +Options specified on the command line take precedence. +</dd> +<dt class="dir">distributions</dt><dd> +This is the main configuration file and the only that is needed in all +cases. +It lists the distributions this repository contains and their properties. +<br> +See <a href="#firststeps">First steps</a> for a short example or the manpage +for a list of all possible fields. +</dd> +<dt class="dir">updates</dt><dd> +Rules about where to download packages from other repositories. +See the section <a href="#mirroring">Mirroring / Updating</a> +for more examples or the man page for a full reference. +</dd> +<dt class="dir">pulls</dt><dd> +Rules about how to move packages in bulk between distributions +where to download packages from other repositories. +See the section <a href="#propagation">Propagation of packages</a> +for an example or the man page for full reference. +</dd> +<dt class="dir">incoming</dt><dd> +Rules for incoming queues as processed by <tt class="command">processincoming</tt>. +See <a href="#processincoming-incoming-config">Processing an incoming queue</a> for more information. +</dd> +</dl> +<h2><a name="export">Generation of index files</a></h2> +<h3>Deciding when to generate</h3> +As reprepro stores all state in its database, +you can decide when you want them to be written to the <tt class="dir">dists/</tt> +directory. +You can always tell reprepro to generate those files with the <tt>export</tt> command: +<pre class="command"> +reprepro -b $YOURBASEDIR export $CODENAMES +</pre> +This can be especially useful, if you just edited <tt class="file">conf/distributions</tt> +and want to test what it generates. +<p> +While that command regenerates all files, in normal operation reprepro will only +regenerate files where something just changed or that are missing. +With <tt class="option">--export</tt> option you can control when this fill happen: +<dl><dt>never</dt><dd>Don't touch any index files. +This can be useful for doing multiple operations in a row and not wanting to regenerate +the indices all the time. +Note that unless you do an explicit export or change the same parts later without that +option, the generated index files may be permanently out of date. +</dd><dt>silent-never</dt><dd>Like never, but be more silent about it. +</dd><dt>changed</dt><dd>This is the default behaviour since 3.0.1. +Only export distributions where something changed +(and no error occurred that makes an inconsistent state likely). +And in those distributions only (re-)generate files which content should have been changed +by the current action or which are missing. +</dd><dt>lookedat</dt><dd>New name for <tt>normal</tt> since 3.0.1. +</dd><dt>normal</dt><dd>This was the default behaviour until 3.0.0 (changed in 3.0.1). +In this mode all distributions are processed that were looked at without error +(where error means only errors happening while the package was open so have a chance +to cause strange contents). +This ensures that even after a operation that had nothing to do the looked at +distribution has all the files exported needed to access it. (But still only files +missing or that content would change with this action are regenerated). +</dd><dt>force</dt><dd>Also try to write the current state if some error occurred. +In all other modes reprepro will not write the index files if there was a problem. +While this keeps the repository usable for users, it means that you will need an +explicit export to write possible other changes done before that in the same run. +(reprepro will tell you that at the end of the run with error, but you should not +miss it). +</dd></dl> +<h3>Distribution specific fields</h3> +There are a lot of <tt class="file">conf/distributions</tt> headers to control +what index files to generate for some distribution, how to name +them, how to postprocess them and so on. The most important are: +<h4>Fields for the Release files</h4> +The following headers are copied verbatim to the Release file, if they exist: +<tt class="header">Origin</tt>, +<tt class="header">Label</tt>, +<tt class="header">Codename</tt>, +<tt class="header">Suite</tt>, +<tt class="header">Architectures</tt> (excluding a possible value "<tt>source</tt>"), +<tt class="header">Components</tt>, +<tt class="header">Description</tt>, and +<tt class="header">NotAutomatic</tt>, +<tt class="header">ButAutomaticUpgrades</tt>. +<h4><a name="compression">Choosing compression and file names</a></h4> +Depending on the type of the index files, different files are generated. +No specifying anything is equivalent to: +<pre class="config"> + DscIndices Sources Release .gz + DebIndices Packages Release . .gz + UDebIndices Packages . .gz +</pre> +This means to generate <tt>Release</tt>, <tt>Sources.gz</tt> for sources, +<tt>Release</tt>, <tt>Packages</tt> and <tt>Packages.gz</tt> for binaries +and <tt>Packages</tt> and <tt>Packages.gz</tt> for installer modules. +<br> +The format of these headers is the name of index file to generate, followed +by the optional name for a per-directory release description +(when no name is specified, no file is generated). +Then a list of compressions: +A single dot (<tt>.</tt>) means generating an uncompressed index, +<tt>.gz</tt> means generating a gzipped output, +while <tt>.bz2</tt> requests and bzip2ed file. +(<tt>.bz2</tt> is not available when disabled at compile time). +After the compressions a script can be given that is called to generate/update +additional forms, see <a href="#exporthook">"Additional index files"</a>. +<h4><a name="signing">Signing</a></h4> +If there is a <tt class="config">SignWith</tt> header, reprepro will try +to generate a <tt class="file">Release.gpg</tt> file using libgpgme. +If the value of the header is <tt>yes</tt> it will use the first key +it finds, otherwise it will give the option to libgpgme to determine the +key. (Which means fingerprints and keyids work fine, and whatever libgpgme +supports, which might include most that gpg supports to select a key). +<br> +The best way to deal with keys needing passphrases is to use +<a href="http://packages.debian.org/gnupg-agent">gpg-agent</a>. +The only way to specify which keyring to use is to set the +<tt class="env">GNUPGHOME</tt> environment variable, which will effect all +distributions. +<h4><a name="contents">Contents files</a></h4> +Reprepro can generate files called +<tt class="file">dists/CODENAME/Contents-ARCHITECTURE.gz</tt> +listing all files in all binary packages available for the selected +architecture in that distribution and which package they belong to. +<br> +This file can either be used by humans directly or via downloaded +and searched with tools like +<a href="http://packages.debian.org/apt-file">apt-file</a>. +<br> +To activate generating of these files by reprepro, you need a <tt class="config">Contents</tt> header in that distribution's declaration in <tt class="file">conf/distributions</tt>, +like: +<pre class="config"> +Contents: +</pre> +Versions before 3.0.0 need a ratio number there, like: +<pre class="config"> +Contents: 1 +</pre> +The number is the inverse ratio of not yet looked at and cached files to process in +every run. The larger the more packages are missing. 1 means to list everything. +<br> +The arguments of the Contents field and other fields control +which Architectures to generate Contents files for and which +Components to include in those. For example +<pre class="config"> +Contents: udebs nodebs . .gz .bz2 +ContentsArchitectures: ia64 +ContentsComponents: +ContentsUComponents: main +</pre> +means to not skip any packages, generate Contents for <tt class="suffix">.udeb</tt> +files, not generating Contents for <tt class="suffix">.deb</tt>s. Also it is only +generated for the <tt>ia64</tt> architecture and only packages in component +<tt>main</tt> are included. +<h4><a name="exporthook">Additional index files (like .diff)</a></h4> +Index files reprepro cannot generate itself, can be generated by telling +it to call a script. +<h5>using rredtool to generate pdiff files</h5> +Starting with version 4.1.0, the <tt>rredtool</tt> coming with reprepro +can be used as hook to create and update <tt>Packages.diff/Index</tt> files. +<br> +Unlike dak (which created the official Debian repositories) or the pdiff.py +script (see below) derived from dak, an user will only need to download +one of those patches, as new changes are merged into the old files. +<br> +To use it, make sure you have +<a href="http://packages.debian.org/diff">diff</a> and +<a href="http://packages.debian.org/gzip">gzip</a> +installed. +Then add something like the following to the headers of the distributions +that should use this in <tt class="file">conf/distributions</tt>: +<pre class="config"> + DscIndices: Sources Release . .gz /usr/bin/rredtool + DebIndices: Packages Release . .gz /usr/bin/rredtool +</pre> +<h5>the pdiff example hook script (generates pdiff files)</h5> +This example generates <tt class="file">Packages.diff</tt> and/or +<tt class="file">Sources.diff</tt> directories containing a set of +ed-style patches, so that people do not redownload the whole index +for just some small changes. +<br> +To use it, copy <tt class="file">pdiff.example</tt> from the examples directory +into your <tt class="dir">conf</tt> directory. +(or any other directory, then you will need to give an absolute path later). +Unpack, if needed. Rename it to pdiff.py and make it executable. +Make sure you have +<a href="http://packages.debian.org/python3-apt">python3-apt</a>, +<a href="http://packages.debian.org/diff">diff</a> and +<a href="http://packages.debian.org/gzip">gzip</a> +installed. +Then add something like the following to the headers of the distributions +that should use this in <tt class="file">conf/distributions</tt>: +<pre class="config"> + DscIndices: Sources Release . .gz pdiff.py + DebIndices: Packages Release . .gz pdiff.py +</pre> +More information can be found in the file itself. You should read it. +<h5>the bzip2 example hook script</h5> +This is an very simple example. +Simple and mostly useless, +as reprepro has built in <tt>.bz2</tt> generation support, +unless you compiled it your own with <tt>--without-libbz2</tt> or +with no <tt>libbz2-dev</tt> installed. +<br> +To use it, copy <tt class="file">bzip.example</tt> from the examples directory +into your <tt class="dir">conf</tt> directory. +(or any other directory, then you will need to give an absolute path later). +Unpack, if needed. Rename it to bzip2.sh and make it executable. +Then add something like the following to the headers of the distributions +that should use this in <tt class="file">conf/distributions</tt>: +<pre class="config"> + DscIndices: Sources Release . .gz bzip2.sh + DebIndices: Packages Release . .gz bzip2.sh + UDebIndices: Packages . .gz bzip2.sh +</pre> +The script will compress the index file using the +<a href="http://packages.debian.org/bzip2">bzip2</a> program and tell +reprepro which files to include in the Release file of the distribution. +<h5>internals</h5> +TO BE CONTINUED +<h4>...</h4> +TO BE CONTINUED +<h2><a name="localpackages">Local packages</a></h2> +There are two ways to get packages not yet in any repository into yours. +<dl><dt>includedsc, includedeb, include</dt><dd> +These are for including packages at the command line. +Many options are available to control what actually happens. +You can easily force components, section and priority and/or choose to +include only some files or only in specific architectures. +(Can be quite useful for architecture all packages depending on some +packages you will some time before building for some of your architectures). +Files can be moved instead of copied and most sanity checks overwritten. +They are also optimized towards being fast and simply try things instead of +checking a long time if they would succeed. +</dd><dt>processincoming</dt><dd> +This command checks for changes files in an incoming directory. +Being optimized for automatic processing (i.e. trying to checking +everything before actually doing anything), it can be slower +(as every file is copied at least once to sure the owner is correct, +with multiple partitions another copy can follow). +Component, section and priority can only be changed via the distribution's +override files. Every inclusion needs a <tt class="suffix">.changes</tt> file. +<br> +This method is also relatively new (only available since 2.0.0), thus +optimisation for automatic procession will happen even more. +</dd></dl> +<h3><a name="include">Including via command line</a></h3> +There are three commands to directly include packages into your repository: +<tt class="command">includedeb</tt>, <tt class="command">includedsc</tt> +and <tt class="command">includechanges</tt>. +Each needs to codename of the distribution you want to put your package into +as first argument and a file of the appropriate type +(<tt class="suffix">.deb</tt>, <tt class="suffix">.dsc</tt> or + <tt class="suffix">.changes</tt>, respectively) as second argument. +<br> +If no component is specified via <tt class="option">--component</tt> +(or short <tt class="option">-C</tt>), it will be guessed looking at its +section and the components of that distribution. +<br> +If there are no <tt class="option">--section</tt> +(or short <tt class="option">-S</tt>) option, and it is not specified +by the (binary or source, depending on the type) override file of the +distribution, the value from the <tt class="suffix">.changes</tt>-file +is used (if the command is <tt class="command">includechanges</tt>) +or it is extracted out of the file (if it is a +<tt class="suffix">.deb</tt>-file, future versions might also try to +extract it from a <tt class="suffix">.dsc</tt>'s diff or tarball). +<br> +Same with the priority and the <tt class="option">--priority</tt> +(or short <tt class="option">-P</tt>) option. +<br> +With the <tt class="option">--architecture</tt> (or short <tt class="option">-A</tt>) +option, the scope of the command is limited to that architecture. +<tt class="command">includdeb</tt> will add a Architecture <tt>all</tt> +packages only to that architecture (and complain about Debian packages for +other architectures). +<tt class="command">include</tt> will do the same and ignore packages for +other architectures (source packages will only be included if the value +for <tt class="option">--architecture</tt> is <tt>source</tt>). +<br> +To limit the scope to a specify type of package, use the +<tt class="option">--packagetype</tt> or short <tt class="option">-T</tt> +option. Possible values are <tt>deb</tt>, <tt>udeb</tt> and <tt>dsc</tt>. +<br> +When using the <tt class="option">--delete</tt> option, files will +be moved or deleted after copying them. +Repeating the <tt class="option">--delete</tt> option will also delete +unused files. +<br> +TO BE CONTINUED. +<h3><a name="incoming">Processing an incoming queue</a></h3> +Using the <tt class="command">processincoming</tt> command reprepro +can automatically process incoming queues. +While this is still improveable (reprepro still misses ways to send +mails and especially an easy way to send rejection mails to the +uploader directly), it makes it easy to have an directory where you +place your packages and reprepro will automatically include them. +<br> +To get this working you need three things: +<ul> +<li><a href="#processincoming-incoming-config"> +a file <tt class="file">conf/incoming</tt> describing your incoming directories, +</a></li> +<li><a href="#processincoming-dist-config"> +a <tt class="file">conf/distribution</tt> file describing your distributions +(as always with reprepro) +and +</a></li> +<li><a href="#processincoming-calling"> +a way to get reprepro called to process it. +</a></li> +</ul> +<a name="processincoming-incoming-config"> +<h4>The file <tt class="file">conf/incoming</tt></h4></a> +describes the different incoming queues. +As usual the different chunks are separated by empty lines. +Each chunk can have the following fields: +<dl><dt>Name</dt><dd>This +is the name of the incoming queue, that <tt class="command">processincoming</tt> +wants as argument.</dd> +<dt>IncomingDir</dt><dd>The actual directory to look for +<tt class="suffix">.changes</tt> files.</dd> +<dt>TempDir</dt><dd>To ensure integrity of the processed files and their +permissions, +every file is first copied from the incoming directory to this directory. +Only the user reprepro runs as needs write permissions here. +It speeds things up if this directory is in the same partition as the pool. +<dt>Allow</dt><dd> +This field lists the distributions this incoming queue might inject packages +into. +Each item can be a pair of a name of a distribution to accept and a distribution +to put it into. +Each upload has each item in its <tt class="field">Distribution:</tt> field +compared first to last to each of this items and is put in the first distribution +accepting it. For example +<pre class="line"> +Allow: stable>etch stable>etch-proposed-updates mystuff unstable>sid +</pre> +will put a <tt class="suffix">.changes</tt> file with +<tt class="field">Distribution: stable</tt> into etch. +If that is not possible (e.g. because etch has a +<tt class="field">UploadersList</tt> option not allowing this) it will +be put into etch-proposed-updates. +And a <tt class="suffix">.changes</tt> file with +<tt class="field">Distribution: unstable</tt> will be put into sid, while +with <tt class="field">Distribution: mystuff</tt> will end up in mystuff. +<br> +If there is a <tt class="field">Default</tt> field, the <tt class="field">Allow</tt> +field is optional.</dd> +<dt>Default</dt><dd> +Every upload not caught by an item of the <tt class="field">Allow</tt> +field is put into the distribution specified by this. +<br> +If there is a <tt class="field">Allow</tt> field, the <tt class="field">Default</tt> +field is optional.</dd> +<dt>Multiple</dt><dd> +This field only makes a difference if a <tt class="suffix">.changes</tt> file +has multiple distributions listed in its <tt class="field">Distribution:</tt> +field. +Without this field each of those distributions is tried according to the +above rules until the package is added to one (or none accepts it). +With this field it is tried for each distribution, so a package can be upload +to multiple distributions at the same time. +</dd> +<dt>Permit</dt><dd> +A list of options to allow things otherwise causing errors. +(see the manpage for possible values). +<br>This field os optional.</dd> +<dt>Cleanup</dt><dd> +Determines when and what files to delete from the incoming queue. +By default only successfully processed <tt class="suffix">.changes</tt> files +and the files references by those are deleted. +For a list of possible options take a look into the man page. +<br>This field os optional.</dd> +</dl> +<a name="processincoming-dist-config"> +<h4><tt class="file">conf/distribution</tt> for <tt class="command">processincoming</tt></h4></a> +There are no special requirements on the <tt class="file">conf/distribution</tt> +file by processincoming. So even a simple +<pre class="file"> +Codename: mystuff +Architectures: i386 source +Components: main non-free contrib bad +</pre> +will work. +<br> +The <tt class="field">Uploaders</tt> field can list a file limiting +uploads to this distribution to specific keys and +<tt class="field">AlsoAcceptFor</tt> is used to resolve unknown names +in <tt class="file">conf/incoming</tt>'s <tt class="field">Allow</tt> +and <tt class="field">Default</tt> fields. +<a name="processincoming-calling"> +<h4>Getting <tt class="command">processincoming</tt> called.</h4></a> +While you can just call <tt class="command">reprepro processincoming</tt> manually, +having an incoming queue needing manual intervention takes all the fun out of +having an incoming queue, so usually so automatic way is chosen: +<ul> +<li>Dupload and dput have ways to call an hook after an package was uploaded. +This can be an ssh to the host calling reprepro. +The disavantage is having to configure this in every +<tt class="file">.dupload.conf</tt> on every host you want to upload and give +everyone access to ssh and permissions on the archive who should upload. +The advantage is you can configure reprepro to have interactive scripts or +ask for passphrases. +</li> +<li>Install a cron-job calling reprepro every 5 minutes. Cron is usually +available everywhere and getting the output sent by mail to you or a mailing +list is easy. +The annoying part is having to wait almost 5 minutes for the processing. +</li> +<li>Use something like <a href="http://packages.debian.org/inoticoming"><tt class="external">inoticoming</tt></a>. +Linux has a syscall called inotify, allowing a program to be run whenever +something happens to a file. +One program making use of this is inoticoming. It watches a directory using +this facility and whenever a <tt class="suffix">.changes</tt> file is completed +it can call reprepro for you. +(As this happens directly, make sure you always upload the <tt class="suffix">.changes</tt> +file last, dupload and dput always ensure this). +This can be combined with Debian's cron-extension to have a program started at +boot time with the <tt>@reboot</tt> directive. +For example with a crontab like: +<pre class="file"> +MAILTO=myaddress@somewhere.tld + +@reboot inoticoming --logfile /my/basedir/logs/i.log /my/basedir/incoming/ --stderr-to-log --stdout-to-log --suffix '.changes' --chdir /my/basedir reprepro -b /my/basedir --waitforlock 100 processincoming local {} \; +</pre> +</li> +</ul> +<h2><a name="mirroring">Mirroring / Updating</a></h2> +Reprepro can fetch packages from other repositories. +For this it uses apt's methods from <tt class="dir">/usr/lib/apt/methods/</tt> +so everything (http, ftp, ...) that works with apt should also work with reprepro. +Note that this works on the level of packages, even though you can tell reprepro +to create a distribution having always the same packages as some remote repository, +the repository as a whole may not look exactly the same but only have the same set +of packages in the same versions. +<br> +You can also only mirror a specific subset of packages, merge multiple repositories +into one distribution, or even have distributions mixing remote and local packages. +<br> +Each distribution to receive packages from other repositories needs an +<tt class="field">Update:</tt> field listing the update rules applied to it. +Those update rules are listed in <tt class="file">conf/updates</tt>. +There is also the magic <tt>-</tt> update rule, which tells reprepro to delete +all packages not re-added by later rules. +<br> +To make reprepro to update all distributions call <tt>reprepro update</tt> +without further arguments, or give the distributions to update as additional +arguments. +<br> +Let's start with some examples: +<h3><a name="update-examples">Updating examples</a></h3> +Let's assume you have the following <tt class="file">conf/distributions</tt> +<pre class="file"> +Codename: etch +Architectures: i386 source +Components: main contrib +Update: local - debian security + +Codename: mystuff +Architectures: abacus source +Components: main bad +Update: debiantomystuff +</pre> +and the following <tt class="file">conf/updates</tt> +<pre class="file"> +Name: local +Method: http://ftp.myorg.tld/debian + +Name: debian +Method: http://ftp.de.debian.org/debian +VerifyRelease: A70DAF536070D3A1 +Config: Acquire::Http::Proxy=http://proxy.yours.org:8080 + +Name: security +Suite: */updates +Method: http://security.eu.debian.org/ +Fallback: http://security.debian.org/ +VerifyRelease: A70DAF536070D3A1 +Config: Acquire::Http::Proxy=http://proxy.yours.org:8080 + +Name: debiantomystuff +Suite: sid +Method: http://ftp.de.debian.org/debian +Architectures: i386>abacus source +Components: main non-free>bad contrib>bad +FilterFormula: Architecture (== all)| !Architecture +FilterList: deinstall list +</pre> +and a file <tt class="file">conf/list</tt> with some +output as <tt>dpkg --get-selections</tt> is printing. +<br> +If you then run +<tt class="command">reprepro update etch</tt> or +<tt class="command">reprepro checkupdate etch</tt>, +reprepro looks at etch's <tt class="field">Update:</tt> line +and finds four rules. The first is the <tt>local</tt> rule, +which only has a method, so that means it will download the +<tt class="file">Release</tt> file from +<tt>http://ftp.myorg.tld/debian/dists/etch/Release</tt> and +(unless it already has downloaded them before or that +repository does not have all of them) downloads the +<tt>binary-i386/Packages.gz</tt> +and <tt>source/Sources.gz</tt> files for main and contrib. +The same is done for the <tt>debian</tt> and <tt>security</tt> +rules. +As they have a <tt class="field">VerifyRelease</tt> field, +Release.gpg is also downloaded and checked to be signed with the +given key +(which you should have imported to you <tt class="external">gpg</tt> +keyring before). +As security has a <tt class="field">Suite:</tt> field, not the codename, +but the content of this field (with an possible<tt>*</tt> replaced by the codename), +is used as distribution to get. +<br> +Then it will parse for each part of the distribution, parse the files it +get from left to right. +For each package it starts with the version currently in the distribution, +if there is a newer on in <tt>local</tt> it will mark this. +Then there is the delete rule <tt>-</tt>, which will mark it to be deleted +(but remembers what was there, so if later the version in the distribution +or the version in <tt>local</tt> are newest, it will get them from here avoiding +slow downloads from far away). Then it will look into <tt>debian</tt> and then +in <tt>security</tt>, if they have a newer version (or the same version, clearing +the deletion mark). +<br> +If you issued <tt class="command">checkupdate</tt> reprepro will print what it would +do now, otherwise it tries to download all the needed files and when it got all, +change the packages in the distribution to the new ones, export the index files +for this distribution and finally delete old files no longer needed. +<br> +TO BE CONTINUED. +<h2><a name="propagation">Propagation of packages</a></h2> +You can copy packages between distributions using the +<tt class="command">pull</tt> and <tt class="command">copy</tt> commands. +<br> +With the <tt class="command">copy</tt> command you can copy packages +by name from one distribution to the other within the same repository. +<br> +With the <tt class="command">pull</tt> command you can pull all packages +(or a subset defined by some list, or exceptions by some list, or by some +formula, or ...) from one distribution to another within the same formula. +<br> +Note that both assume the filenames of the corresponding packages in the +pool will not differ, so you cannot move packages from one component to another. +<br> +Let's just look at a little example, more information can be found in the man page. +<br> +Assume you upload all new packages to a distribution and you want another +so you can keep using an old version until you know the newer works, too. +One way would be to use something like the following +<tt class="file">conf/distributions</tt>: +<pre class="file"> +Codename: development +Suite: unstable +Components: main extra +Architectures: i386 source + +Codename: bla +Suite: testing +Components: main extra +Architectures: i386 source +Pull: from_development +</pre> +and <tt class="file">conf/pulls</tt>: +<pre class="file"> +Name: from_development +From: development +</pre> +i.e. you have two distributions, bla and development. +Now you can just upload stuff to development (or it's alias unstable). +And when you want a single package to go to testing, you can use the copy +command: +<pre class="shell"> +reprepro copy bla development name1 name2 name3 +</pre> +If you do not want to copy all packages of a given name, but only some +of them, you can use <tt>-A</tt>, <tt>-T</tt> and <tt>-C</tt>: +<pre class="shell"> +reprepro -T deb -A i386 copy bla development name1 +</pre> +will copy <tt class="suffix">.deb</tt> packages called name1 from the i386 +parts of the distribution. +<br> +TO BE CONTINUED +<h2><a name="snapshosts">Snapshots</a></h2> +There is a gensnapshot command.<br> +TO BE DOCUMENTED +<h2><a name="tracking">Source package tracking</a></h2> +TO BE DOCUMENTED +<h2><a name="hooks">Extending reprepro / Hooks and more</a></h2> +When reprepro misses some functionality, +it often can be added by some kind of hook. +<br> +Currently you can execute your own scripts at the following occasions: +<ul> +<li><a href="#addhook">after adding or removing packages</a></li> +<li><a href="#byhandhook">to process byhand files</a></li> +<li><a href="#exporthook">when creating index files (Packages.gz, Sources.gz)</a></li> +<li><a href="#signhook">when signing releases</a></li> +<li><a href="#outhook">after changing the visible files of the repository managed</a></li> +<li><a href="#endhook">when reprepro finished</a></li> +</ul> +<h3><a name="addhook">Scripts to be run when adding or removing packages</a></h3> +Whenever a package is added or removed, +you can tell reprepro to log that to some file and/or call a script using the +<tt>Log:</tt> directive in <tt class="file">conf/distributions</tt>. +<br> +This script can send out mails and do other logging stuff, +but despite the name, it is not restricted to logging. +<br> +<h4>Automatically extracting changelog and copyright information</h4> +reprepro ships with an example script to extract <tt class="file">debian/changelog</tt> +and <tt class="file">debian/copyright</tt> +files from source packages into a hierarchy loosely resembling the way changelogs +are made available at +<a href="http://packages.debian.org/changelogs/">http://packages.debian.org/changelogs/</a>. +<br> +All you have to do is to copy (or unpack if compressed) the file +<tt class="file">changelogs.example</tt> from the examples directory +in the reprepro source or +<a href="file:///usr/share/doc/reprepro/examples/">/usr/share/doc/reprepro/examples/</a> +of your installed reprepro package into your <tt class="directory">conf/</tt> directory +(or somewhere else, then you will need an absolute path later), perhaps +change some directories specified in it +and add something like the following lines +to all distributions in <tt class="file">conf/distributions</tt> that should use +this feature: +<pre class="config"> +Log: + --type=dsc changelogs.example +</pre> +If you still want to log to some file, just keep the filename there: +<pre class="config"> +Log: mylogfilename + --type=dsc changelogs.example +</pre> +Then cause those files to be generated for all existing files via +<pre class="command"> +reprepro rerunnotifiers +</pre> +and all future source packages added or removed will get this list automatically +updated. +<h4>Writing your own Log: scripts</h4> +You can list an arbitrary amount of scripts, to be called at specified times +(which can overlap or even be the same): +<pre class="config"> +Log: logfilename + --type=dsc script-to-run-on-source-package-changes + script-to-run-on-package-changes + another-script-to-run-on-package-changes + --type=dsc --component=main script-to-run-on-main-source-packages + --architecture=i386 --type=udeb script-to-run-on-i386-udebs + --changes script-to-run-on-include-or-processincoming +</pre> +There are two kind of scripts: +The first one is called when a package was added or removed. +Using the <tt class="option">--archtecture=</tt>, +<tt class="option">--component=</tt> and +<tt class="option">--type=</tt> options you can limit it to specific parts +of the distribution. +The second kind is marked with <tt class="option">--changes</tt> and is +called when a <tt class="suffix">.changes</tt>-file was added with +<tt class="command">include</tt> or <tt class="command">processincoming</tt>. +Both are called asynchronous in the background <emph>after</emph> everything was done, +but before no longer referenced files are deleted (so the files of the +replaced or deleted package are still around). +<h5>Calling conventions for package addition/removal scripts</h5> +This type of script is called with a variable number of arguments. +The first argument is the action. This is either +<tt>add</tt>, <tt>remove</tt> or <tt>replace</tt>. +The next four arguments are the codename of the affected distribution +and the packagetype, component and architecture in that distribution +affected. +The sixth argument is the package's name. +After that is the version of the added package (<tt>add</tt> and <tt>replace</tt>) +and the version of the removed package (<tt>remove</tt> and <tt>replace</tt>). +Finally the filekeys of the new (<tt>add</tt> and <tt>replace</tt>) and/or +removed (<tt>remove</tt> and <tt>replace</tt>) package are listed +starting with the marker "<tt>--</tt>" followed by each filekey +(the name of the file in the <tt class="dir">pool/</tt> +relative to <tt class="env">REPREPRO_OUT_DIR</tt>) +as its own argument. +<br> +The environment variable <tt class="env">REPREPRO_CAUSING_COMMAND</tt> +contains the command of the action causing this change. +The environment variable +<tt class="env">REPREPRO_CAUSING_FILE</tt> contains the name of the file +given at the command line causing this package to be changed, +if there is one. +(i.e. with <tt class="command">includedeb</tt>, +<tt class="command">includedsc</tt> and <tt class="command">include</tt>). +The environment variables +<tt class="env">REPREPRO_CAUSING_RULE</tt> and +<tt class="env">REPREPRO_FROM</tt> are +the name of the update or pull rule pulling in a package +and the name of the distribution a package is coming from. +What this name is depends on the command and for most commands +it is simply not set at all. +And of course all the <tt class="env">REPREPRO_*_DIR</tt> variables are set. +<h5>Calling conventions for <tt class="suffix">.changes</tt> scripts</h5> +This type of script is called with 5 or 6 arguments. +The first is always "<tt>accepted</tt>", to make it easier to +check it is configured the right way. +The second argument is the codename of the distribution the +<tt class="suffix">.changes</tt>-file was added to. +The third argument is the source name, the forth the version. +The fifth name is the <tt class="suffix">.changes</tt> itself +(in case of <tt class="command">processingcoming</tt> the secure copy in the +temporary dir). +There is a sixth argument if the <tt class="suffix">.changes</tt>-file was +added to the <tt class="dir">pool/</tt>: +The filekey of the added .changes file +(i.e. the filename relative to <tt class="env">REPREPRO_OUT_DIR</tt>). +<br> +The environment variable <tt class="env">REPREPRO_CAUSING_COMMAND</tt> +contains the command of the action causing this change. +The environment variable +<tt class="env">REPREPRO_CAUSING_FILE</tt> contains the name of the file +given at the command line, if there is one +(e.g. with <tt class="command">include</tt>). +And of course all the <tt class="env">REPREPRO_*_DIR</tt> variables are set. +<h3><a name="byhandhook">Scripts to be run to process byhand files</a></h3> +<tt class="suffix">.changes</tt> files can (beside the usual packages files +to be included in the repository) contain additional files to be processed +specially. +Those are marked by the special section <tt class="constant">byhand</tt> (in Debian) +or <tt class="constant">raw-</tt>something (in Ubuntu). +Besides storing them just in the pool besides the packages using the +<tt class="constant">includebyhand</tt> value in the <tt class="field">Tracking</tt> +settings you can also let reprepro process a hook to process them when encountering +them in the <tt class="action">processincomming</tt> action +(Typical usages are uploading documentation files this way that are unpacked next +to the repository, or installer images or stuff like that). + +To use them add to the distribution's defining stanca in <tt class="filename">conf/distributions</tt> a field like: +<pre class="config"> +ByhandHooks: + byhand * manifesto.txt handle-byhand.sh +</pre> +This will call the hook script <tt class="constant">handle-byhand.sh</tt> for every byhand file with section <tt class="constant">byhand</tt>, any priority and filename <tt class="constant">manifesto.txt</tt>. (The first three fields allow glob characters for matching). + +The script will then be called with 5 arguments: +the codename of the distribution, +the section, +the priority, +the filename as found in the changes file and +the filename of where the script can find the actual file. + +<h3>Scripts to be run when creating index files (Packages.gz, Sources.gz)</h3> +this hook is described in the section <a href="#exporthook">"Additional index files"</a>. + +<h3><a name="signhook">Scripts to be run when signing releases</a></h3> +Instead of creating <tt class="filename">InRelease</tt> and +<tt class="filename">Release.gpg</tt> files using libgpgme, +the <tt class="option">SignWith</tt> option can also contain +a exclamation mark followed by a space and the name of a hook script to call. + +The script gets three arguments: +The filename to sign, +the filename of the InRelease file to create and +the filename of the Release.gpg to create +(a Release.gpg does not need to be created. reprepro will assume you do not care about that legacy file if it is not created). + +Reprepro will wait for the script to continue and only do the renaming +and deleting of old files after that, so the script might wait for example +for someone to copy files from the system, signing and copying them it, +for example. + +<h3><a name="outhook">Scripts to be run after changing the visible files of the repository managed</a></h3> +When using the <tt class="option">--outhook</tt> command line option (or the corresponding +<tt class="constant">outhook</tt> in the <tt class="filename">options</tt> file), +reprepro will create a <tt class="suffix">.outlog</tt> file in the log directory describing +any changes done to the out dir and calls the hook script given as argument with this +file as argument. + +The <tt class="suffix">.outlog</tt> file consists of lines each starting with a keyword +and then some arguments separated by tab characters. + +The possible keywords are: +<ul> +<li><tt class="constant">POOLNEW</tt>: +One argument is the filekey of a file newly added to the pool. +<li><tt class="constant">POOLDELETE</tt>: +One argument is the filekey of a file removed from the pool. +<li><tt class="constant">START-DISTRIBUTION</tt>, <tt class="constant">END-DISTRIBUTION</tt>: +two or three arguments: the codename, the directory, +and the suite (if set). +<li><tt class="constant">START-SNAPSHOT</tt>, <tt class="constant">END-SNAPSHOT</tt>: +three arguments: the codename, the directory, and the name of the snapshot generated. +<li><tt class="constant">DISTFILE</tt>: +three arguments: the directory of the distribution (relative to out dir), the name relative to that directory, and the filename generated by reprepro. +<li><tt class="constant">DISTSYMLINK</tt>: +three arguments: the directory of the distribution (relative to out dir), the name relative to that directory, and the symlink target (relative to that directory). +<li><tt class="constant">DISTDELETE</tt>: +two arguments: the directory of the distribution (relative to out dir), the name relative to that directory of a file no longer there. +<li><tt class="constant">DISTKEEP</tt> (not yet generated): +two arguments: the directory of the distribution (relative to out dir), the name relative to that directory. +</ul> + +All <tt class="constant">POOLNEW</tt> come before any distribution changes referencing them +and all <tt class="constant">POOLDELETE</tt> will be afterwards. +Each line belonging to a distribution is guaranteed to be between the corresponding +<tt class="constant">START-DISTRIBUTION</tt> and +<tt class="constant">END-DISTRIBUTION</tt> or between a +<tt class="constant">START-SNAPSHOT</tt> and +<tt class="constant">END-SNAPSHOT</tt> or between a +with the same directory +(i.e. there is some redundancy so you can choose to parse the information where it is more convenient for you). + +The lines starting with <tt class="constant">DIST</tt> describe new or modified files in the distribution description exported by reprepro. No hint is given if that file was previously non-existent, a proper file or a symlink (i.e. if you copy stuff, do not make any assumptions about that). +Future versions of reprepro might create <tt class="constant">DISTKEEP</tt> lines to denote files that have not changed (i.e. just ignore those lines to be future-proof). + +The directories for the distribution entries are what apt expects them (i.e. always starting with <tt class="constant">dists/</tt>, while the third argument to <tt class="constant">DISTFILE</tt> is the name reprepro generated (i.e. starts with the distdir value, which can be configured to not end with <tt class="constant">dists/</tt>). + +<h3><a name="endhook">when reprepro finished</a></h3> +With the <tt class="option">--endhook</tt> command line option (or the corresponding +<tt class="constant">endhook</tt> in the <tt class="filename">options</tt> file) you +can specify a hook to be executed after reprepro finished but before reprepro returns +the to calling process. +The hook gets all the command line arguments after the options (i.e. starting with +the name of the action) and the exit code reprepro would have produces. +For an example see the man page. +<h2><a name="maintenance">Maintenance</a></h2> +This section lists some commands you can use to check and improve the health +of you repository. +<br> +Normally nothing of this should be needed, but taking a look from time to time +cannot harm. +<pre class="command"> +reprepro -b $YOURBASEDIR dumpunreferenced +</pre> +This lists all files reprepro knows about that are not marked as needed by anything. +Unless you called reprepro with the <tt class="option">--keepunreferenced</tt> +option, those should never occur. Though if reprepro is confused or interrupted it +may sometimes prefer keeping files around instead of deleting them. +<pre class="command"> +reprepro -b $YOURBASEDIR deleteunreferenced +</pre> +This is like the command before, only that such files are directly forgotten and +deleted. +<pre class="command"> +reprepro -b $YOURBASEDIR check +</pre> +Look if all needed files are in fact marked needed and known. +<pre class="command"> +reprepro -b $YOURBASEDIR checkpool +</pre> +Make sure all known files are still there and still have the same checksum. +<pre class="command"> +reprepro -b $YOURBASEDIR checkpool fast +</pre> +As the command above, but do not compute checksums. +<pre class="command"> +reprepro -b $YOURBASEDIR tidytracks +</pre> +If you use source package tracking, check for files kept because of this +that should no longer by the current rules. +<br> +If you fear your tracking data could have became outdated, +you can also try the retrack command: +<pre class="command"> +reprepro -b $YOURBASEDIR retrack +</pre> +That refreshes the tracking information about packages used and then +runs a tidytracks. (Beware: don't do this with reprepro versions before 3.0.0). +<h2><a name="internals">Internals</a></h2> +reprepro stores the data it collects in Berkeley DB file (<tt class="suffix">.db</tt>) +in a directory called <tt class="dir">db/</tt> or whatever you specified via command +line. With a few exceptions, those files are NO CACHES, but the actual data. +While some of those data can be regained when you lose those files, they are better +not deleted. +<h3>packages.db</h3> +This file contains the actual package information. +<br> +It contains a database for every (codename,component,architecture,packagetype) quadruple +available. +<br> +Each is indexed by package name and essentially contains the information written do +the Packages and Sources files. +<br> +Note that if you change your <tt class="file">conf/distributions</tt> to no longer +list some codenames, architectures or components, +that will not remove the associated databases in this file. +That needs an explicit call to <tt class="command">clearvanished</tt>. +<h3>references.db</h3> +This file contains a single database that lists for every file why this file +is still needed. +This is either an identifier for a package database, an tracked source package, +or a snapshot. +<br> +Some low level commands to access this are (take a look at the manpage for how to use them): +<dl class="commands"> +<dt class="command">rereference</dt><dd>recreate references (i.e. forget old and create newly)</dd> +<dt class="command">dumpreferences</dt><dd>print a list of all references</dd> +<dt class="command">_removereferences</dt><dd>remove everything referenced by a given identifier</dd> +<dt class="command">_addreference</dt><dd>manually add a reference</dd> +<dt class="command">_addreferences</dt><dd>manually add multiple references</dd> +</dl> +<h3>files.db / checksums.db</h3> +These files contains what reprepro knows about your <tt class="dir">pool/</tt> directory, +i.e. what files it things are there with what sizes and checksums. +The file <tt class="filename">files.db</tt> is used by reprepro before version 3.3 +and kept for backwards compatibility. +If your repository was only used with newer versions you can safely delete it. +Otherwise you should run <tt class="command">collectnewchecksums</tt> before deleting +it. +The file <tt class="filename">checksums.db</tt> is the new file used since +version 3.3. +It can store more checksums types (<tt class="filename">files.db</tt> only contained +md5sums, <tt class="filename">checksums.db</tt> can store arbitrary checksums and +reprepro can even cope with it containing checksum types it does not yet know of) +but for compatibility with pre-3.3 versions is not the canonical source of information +as long as a <tt class="filename">files.db</tt> file exists). +<br> +If you manually put files in the pool or remove them, you should tell reprepro about that. +(it sometimes looks for files there without being told, but it never forgets files +except when it would have deleted them anyway). +Some low level commands (take a look at the man page for how to use them): +<dl class="commands"> +<dt class="command">collectnewchecksums</dt><dd>Make sure every file is listed in <tt class="filename">checksums.db</tt> and with all checksum types your reprepro supports.</dd> +<dt class="command">checkpool fast</dt><dd>Make sure all files are still there.</dd> +<dt class="command">checkpool</dt><dd>Make sure all files are still there and correct.</dd> +<dt class="command">dumpunreferenced</dt><dd>Show all known files without reference.</dd> +<dt class="command">deleteunreferenced</dt><dd>Delete all known files without reference.</dd> +<dt class="command">_listmd5sums</dt><dd>Dump this database (old style)</dd> +<dt class="command">_listchecksums</dt><dd>Dump this database (new style)</dd> +<dt class="command">_detect</dt><dd>Add files to the database</dd> +<dt class="command">_forget</dt><dd>Forget that some file is there</dd> +<dt class="command">_addmd5sums</dt><dd>Create the database from dumped data</dd> +<dt class="command">_addchecksums</dt><dd>dito</dd> +<h3>release.cache.db</h3> +In this file reprepro remembers what it already wrote to the <tt class="dir">dists</tt> +directory, +so that it can write their checksums (including the checksums of the uncompressed variant, +even if that was never written to disk) +in a newly to create <tt class="file">Release</tt> +file without having to trust those files or having to unpack them. +<h3>contents.cache.db</h3> +This file contains all the lists of files of binary package files where reprepro +already needed them. (which can only happen if you requested Contents files to be +generated). +<h3>tracking.db</h3> +This file contains the information of the <a href="#tracking">source package tracking</a>. +<h2><a name="recovery">Disaster recovery</a></h2> +TO BE DOCUMENTED (see the +<a href="http://git.debian.org/?p=mirrorer/reprepro.git;a=blob_plain;f=docs/recovery;hb=HEAD">recovery</a> +file until then) +<h2><a name="paranoia">Paranoia</a></h2> +As all software, reprepro might have bugs. +And it uses libraries not written by myself, +which I'm thus even more sure that they will have bugs. +Some of those bugs might be security relevant. +This section contains some tips, to reduce the impact of those. +<ul> +<li>Never run reprepro as root.<br> +All reprepro needs to work are permissions to files, +there is no excuse for running it as root. +</li> +<li>Don't publish your db/ directory.<br> +The contents of the db directory are not needed by everyone else. +Having them available to everyone may make it easier for them to +exploit some hypothetical problem in libdb and makes it easier to +know in advance how exactly reprepro will act in a given circumstances, +thus easier to exploit some hypothetical problem. +</li> +<li>Don't accept untrusted data without need.<br> +If an attacker cannot do anything, they cannot do anything harmful, either. +So if there is no need, don't offer an anonymous incoming queue. +<tt class="program">dput</tt> supports uploading via scp, so just having +an only group-writable incoming directory, or even better multiple incoming +directories can be a better alternative. +</li> +</ul> +External stuff being used and attack vectors opened by it: +<dl> +<dt>libgpgme/gpg</dt><dd> +Almost anything is run through <tt>libgpgme</tt> and thus <tt>gpg</tt>. +It will be used to check the <tt class="filename">Release.gpg</tt> file, +or to read <tt class="suffix">.dsc</tt> and <tt class="suffix">.changes</tt> +files (even when there is no key to look for specified, +as that is the best way to get the data from the signed block). +Avoiding this by just accepting stuff without looking for signatures on +untrusted data is not really an option, so I know nothing to prevent this +type of problems. +</dd> +<dt>libarchive</dt><dd> +The <tt class="suffix">.tar</tt> files within <tt class="suffix">.deb</tt> +files are normally (unless that library was +not available while compiling) read using libarchive. +This happens when a <tt class="suffix">.deb</tt> file is to be added +(though only after deciding if it should be added, so if it does not have +the correct checksum or the .changes did not have the signatures you specified, +it is not) or when the file list is to be extracted +(when creating <tt class="filename">Contents</tt> files). +Note that they are not processed when only mirroring them (of course unless +<tt class="filename">Contents</tt> files are generated), as then only the +information from the Packages file is copied. +</dd> +<dt>dpkg-deb/tar</dt><dd> +If reprepro was compiled without libarchive, +<tt class="program">dpkg-deb</tt> is used instead, which most likely will +call <tt class="program">tar</tt>. Otherwise just the same like the last +item. +</dd> +<dt>zlib</dt><dd> +When mirroring packages, the downloaded +<tt class="filename">Packages.gz</tt> and <tt class="filename">Sources.gz</tt> files +are read using zlib. Also the generated <tt class="suffix">.gz</tt> files +are generated using it. There is no option but hoping there is no security +relevant problem in that library. +</dd> +<dt>libbz2</dt><dd> +Only used to generate <tt class="suffix">.bz2</tt> files. +If you fear simple blockwise writing using that library has a security problem +that can be exploited by data enough harmless looking to be written to the +generated index files, you can always decide to no tell reprepro to generate +<tt class="suffix">.bz2</tt> files. +</dd> +</dl> +<h2><a name="counterindications">What reprepro cannot do</a></h2> +There are some things reprepro does not do: +<dl><dt>Verbatim mirroring</dt><dd> +Reprepro aims to put all files into a coherent <tt>pool/</tt> hierarchy. +Thus it cannot guarantee that files will have the same relatives path as in the +original repository (especially if those have no pool). +It also creates the index files from its own indices. +While this leads to a tidy repository and possible savings of disk-space, the +signatures of the repositories you mirror cannot be used to authenticate the mirror, +but you will have to sign (or tell reprepro to sign for you) the result. +While this is perfect when you only mirror some parts or specific packages or +also have local packages that need local signing anyway, reprepro is no suitable tool +for creating a full mirror that can be authenticated without adding the key of this +repository. +</dd> +<dt>Placing your files on your own</dt><dd> +Reprepro does all the calculation of filenames to save files as, +bookkeeping what files are there and what are needed and so on. +This cannot be switched off or disabled. +You can place files where reprepro will expect them and reprepro will use +them if their md5sum matches. +But reprepro is not suited if you want those files outside of a pool or in +places reprepro does not consider their canonical ones. +</dd> +<dt>Having different files with the same name</dt><dd> +take a look in the <a href="http://git.debian.org/?p=mirrorer/reprepro.git;a=blob_plain;f=docs/FAQ;hb=HEAD">FAQ</a> (currently question 1.2) why and how to avoid the problem. + +</dd> +</dl> +</body> +</html> diff --git a/docs/outsftphook.py b/docs/outsftphook.py new file mode 100755 index 0000000..91c1c9c --- /dev/null +++ b/docs/outsftphook.py @@ -0,0 +1,589 @@ +#!/usr/bin/python3 +# Copyright (C) 2013 Bernhard R. Link +# +# This example script is free software; you can redistribute it +# and/or modify it under the terms of the GNU General Public License +# version 2 as published by the Free Software Foundation. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program; if not, write to the Free Software +# Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02111-1301 USA + +# Those can be set here or in conf/outsftphook.conf: +servername = None +username = None +targetdir = "" + +import sys, os, subprocess, select, sftp + +class Round(sftp.Enum, + DONE = -2, + INDIRECT = -1, + POOLFILES = 0, + DISTFILES = 1, + DELETES = 2, +): + pass + +errors = 0 +def printe(s): + global errors + print(s, file=sys.stderr) + errors += 1 + +# renaming file, assuming all directories exist... +def renamefile(dst, src, donefunc): + a = yield [sftp.REMOVE(targetdir + dst), sftp.RENAME(targetdir + src, targetdir + dst, [sftp.SSH_FXF_RENAME.OVERWRITE])] + while True: + l = [] + if not isinstance(a, sftp.STATUS): + raise SftpUnexpectedAnswer(a, "expecting STATUS") + if isinstance(a.forr, sftp.REMOVE): + if a.status != sftp.SSH_FX.OK and a.status != sftp.SSH_FX.NO_SUCH_FILE: + printe("%s failed: %s" % (a.forr, a)) + elif isinstance(a.forr, sftp.RENAME): + if a.status != sftp.SSH_FX.OK: + printe("%s failed: %s" % (a.forr, a)) + else: + l = donefunc(dst) + else: + raise SftpUnexpectedAnswer(a, a.forr) + a.forr.done() + a = yield l + +# create symlink, assuming all directories exist... +def symlinkfile(dst, src, donefunc): + a = yield [sftp.REMOVE(targetdir + dst), sftp.SYMLINK(targetdir + dst, targetdir + src)] + while True: + l = [] + if not isinstance(a, sftp.STATUS): + raise SftpUnexpectedAnswer(a, "expecting STATUS") + if isinstance(a.forr, sftp.REMOVE): + if a.status != sftp.SSH_FX.OK and a.status != sftp.SSH_FX.NO_SUCH_FILE: + printe("%s failed: %s" % (a.forr, a)) + elif isinstance(a.forr, sftp.SYMLINK): + if a.status != sftp.SSH_FX.OK: + printe("%s failed: %s" % (a.forr, a)) + else: + l = donefunc(dst, message="symlink done") + else: + raise SftpUnexpectedAnswer(a, a.forr) + a.forr.done() + a = yield l + +def deletefile(dst, donefunc): + a = yield [sftp.REMOVE(targetdir + dst)] + if not isinstance(a, sftp.STATUS): + raise SftpUnexpectedAnswer(a, "expecting STATUS") + if a.status == sftp.SSH_FX.OK: + l = donefunc(dst, message="deleted") + elif a.status == sftp.SSH_FX.NO_SUCH_FILE: + l = donefunc(dst, message="already deleted") + else: + printe("%s failed: %s" % (a.forr, a)) + l = [] + a.forr.done() + a = yield l + raise SftpUnexpectedAnswer(a, a.forr) + +def writefile(fname, filetocopy, donefunc): + filename = targetdir + fname + fd = open(filetocopy, 'rb') + dirname = os.path.dirname(filename) + if dirname: + mode = yield [('waitingfor', sftp.Dirlock, dirname)] + else: + mode = "top-level" + a = yield [('lock', sftp.Semaphore, 'openfile')] + if a != "unlock": + raise SftpUnexpectedAnswer(a, "waiting for unlock event") + a = yield [sftp.OPEN(filename, "CREAT|WRITE")] + if mode == "tryandtell" and isinstance(a, sftp.STATUS) and a.status == a.status.NO_SUCH_FILE: + a.forr.done() + a = yield [('missing', sftp.Dirlock, dirname), + ('release', sftp.Semaphore, 'openfile')] + if a != "createnew": + raise SftpUnexpectedAnswer(a, "waiting for %s" % dirname) + mode = a + a = yield [('lock', sftp.Semaphore, 'openfile')] + if a != "unlock": + raise SftpUnexpectedAnswer(a, "waiting for unlock event") + a = yield [sftp.OPEN(filename, "CREAT|WRITE")] + if not isinstance(a, sftp.HANDLE): + a.forr.done() + printe("Failed to create %s: %s" % (filename, a)) + return + # raise SftpException("Failed to create %s: %s" % (filename, a)) + h = a.handle + a.forr.done() + if mode == "tryandtell": + f = [('found', sftp.Dirlock, dirname), 'wantwrite'] + else: + f = ['wantwrite'] + a = yield f + if a != 'canwrite': + raise SftpUnexpectedAnswer(a, "waiting for 'canwrite'") + ofs = 0 + while True: + b = fd.read(16376) + if len(b) == 0: + break + a = yield [sftp.WRITE(h, ofs, b), 'wantwrite'] + ofs += len(b) + b = None + while a != 'canwrite': + a.forr.done() + fd.close() + a = yield [sftp.CLOSE(h), ('release', sftp.Semaphore, 'openfile')] + while True: + if type(a.forr) == sftp.CLOSE: + if a.status != sftp.SSH_FX.OK: + printe("%s failed: %s" % (a.forr, a)) + l = donefunc(fname) + else: + if a.status != sftp.SSH_FX.OK: + printe("%s failed: %s" % (a.forr, a)) + l = [] + a.forr.done() + a = yield l + +class CriticalError(Exception): + pass +class ParseError(CriticalError): + pass +class ParseErrorWrongCount(ParseError): + def __init__(field): + super().__init__("Wrong number of arguments for %s" % field) + +class CollectedDistDir: + def __init__(self, dir): + self.done = False + self.failed = False + self.dir = dir + self.files = dict() + self.deletes = dict() + self.symlinks = dict() + self.transfered = 0 + def onedone(self, filename): + assert(filename.endswith(".new")) + filename = filename[:-4] + assert (filename in self.files) + self.transfered += 1 + self.files[filename].markpartial(filename, "asdotnew") + return self.finalizeifready() + def finalizeifready(self): + assert (not self.done) + if len(self.files) != self.transfered: + assert (len(self.files) > self.transfered) + return [] + # everything copied as .new as needed, let's start finalisation + self.done = True + l = [] + for m,e in self.files.items(): + l.append(sftp.TaskFromGenerator(renamefile(m, m + ".new", e.doneone))) + for m,e in self.deletes.items(): + l.append(sftp.TaskFromGenerator(deletefile(m, e.doneone))) + for m,(t,e) in self.symlinks.items(): + l.append(sftp.TaskFromGenerator(symlinkfile(m, t, e.doneone))) + return l + +class DistDir: + def __init__(self, dir, onelog=True): + self.dir = dir + self.files = [] + self.deletes = [] + self.symlinks = [] + def queue(self, todo, distdirs, logfile): + if not self.dir in distdirs: + collection = CollectedDistDir(self.dir) + distdirs[self.dir] = collection + else: + collection = distdirs[self.dir] + for fn, fr in self.files: + ffn = self.dir + "/" + fn + if logfile.alreadydone.get(ffn, "") == "asdotnew": + if logfile.enqueue(todo, ffn, Round.INDIRECT): + collection.files[ffn] = logfile + collection.transfered += 1 + else: + if logfile.enqueue(todo, ffn, + Round.DISTFILES, ffn + ".new", + fr, collection.onedone): + collection.files[ffn] = logfile + for fn in self.deletes: + ffn = self.dir + "/" + fn + if logfile.enqueue(todo, ffn, Round.INDIRECT): + collection.deletes[ffn] = logfile + for fn, flt in self.symlinks: + ffn = self.dir + "/" + fn + if logfile.enqueue(todo, ffn, Round.INDIRECT): + collection.symlinks[ffn] = (flt, logfile) + +class LogFile: + def parselogline(self, fields): + if fields[0] == 'POOLNEW': + if len(fields) != 2: + raise ParseErrorWrongCount(fields[0]) + self.newpoolfiles.append(fields[1]) + elif fields[0] == 'POOLDELETE': + if len(fields) != 2: + raise ParseErrorWrongCount(fields[0]) + self.deletepoolfiles.append(fields[1]) + elif fields[0].startswith('BEGIN-'): + pass + elif fields[0].startswith('END-'): + pass + elif fields[0].startswith('DIST'): + command = fields[0][4:] + if command not in ['KEEP', 'FILE', 'DELETE', 'SYMLINK']: + raise ParseError("Unknown command %s" % command) + if not fields[1] in self.dists: + d = self.dists[fields[1]] = DistDir(fields[1]) + else: + d = self.dists[fields[1]] + if command == 'FILE': + if len(fields) != 4: + raise ParseErrorWrongCount(fields[0]) + d.files.append((fields[2], fields[3])) + elif command == 'DELETE': + if len(fields) != 3: + raise ParseErrorWrongCount(fields[0]) + d.deletes.append(fields[2]) + elif command == 'SYMLINK': + if len(fields) != 4: + raise ParseErrorWrongCount(fields[0]) + d.symlinks.append((fields[2], fields[3])) + elif fields[0] == "DONE": + self.alreadydone[fields[2]] = fields[1] + else: + raise ParseError("Unknown command %s" % fields[0]) + def __init__(self, logfile, donefile): + self.alreadydone = dict() + self.logfile = logfile + self.donefile = donefile + try: + lf = open(logfile, 'r', encoding='utf-8') + except Exception as e: + raise CriticalError("Cannot open %s: %s" % (repr(logfile), e)) + self.newpoolfiles = [] + self.dists = {} + self.deletepoolfiles = [] + self.todocount = 0 + for l in lf: + if l[-1] != '\n': + raise ParseError("not a text file") + self.parselogline(l[:-1].split('\t')) + lf.close() + def queue(self, todo, distdirs): + self.todo = set() + for f in self.deletepoolfiles: + self.enqueue(todo, f, Round.DELETES, f, None, self.doneone) + for f in self.newpoolfiles: + self.enqueue(todo, f, Round.POOLFILES, f, options.outdir + "/" + f, self.doneone) + for d in self.dists.values(): + d.queue(todo, distdirs, self) + if not self.todocount: + # nothing to do left, mark as done: + os.rename(self.logfile, self.donefile) + del self.todo + return self.todocount > 0 + def enqueue(self, dic, elem, *something): + if elem in self.alreadydone and self.alreadydone[elem] != "asdotnew": + if not elem in dic: + dic[elem] = (Round.DONE,) + return False + elif not elem in dic: + self.todo.add(elem) + self.todocount += 1 + dic[elem] = something + return True + else: + self.markpartial(elem, "obsoleted") + return False + def markpartial(self, filename, message="done"): + if options.verbose: + print("%s: %s" % (message, repr(filename))) + f = open(self.logfile, "a", encoding="utf-8") + print("DONE\t%s\t%s" % (message, filename), file=f) + f.close() + def doneone(self, filename, message="done"): + assert (filename in self.todo) + self.todo.discard(filename) + assert (self.todocount > 0) + self.todocount -= 1 + self.markpartial(filename, message=message) + if self.todocount == 0: + os.rename(self.logfile, self.donefile) + return [] + + +def doround(s, r, todo): + for p,v in todo.items(): + assert (isinstance(v[0], Round)) + if v[0] != r: + continue + round, filename, source, donefunc = v + if round != r: + continue + if source is None: + s.start(sftp.TaskFromGenerator(deletefile(filename, donefunc))) + else: + s.start(sftp.TaskFromGenerator(writefile(filename, source, donefunc))) + s.dispatch() + + +class Options: + def __init__(self): + self.verbose = None + self.pending = False + self.autoretry = None + self.ignorepending = False + self.forceorder = False + self.confdir = None + self.basedir = None + self.outdir = None + self.logdir = None + self.debugsftp = None + +options = Options() + +def parseoptions(args): + while args and args[0].startswith("--"): + arg = args.pop(0) + if arg == "--verbose" or arg == "-v": + options.verbose = True + elif arg.startswith("--debug-sftp="): + options.debugsftp = int(arg[13:]) + elif arg == "--pending": + options.pending = True + elif arg == "--ignore-pending": + options.ignorepending = True + elif arg == "--force-order": + options.forceorder = True + elif arg == "--basedir=": + options.basedir = arg[:10] + elif arg == "--basedir": + options.basedir = args.pop(0) + elif arg == "--outdir=": + options.outdir = arg[:9] + elif arg == "--outdir": + options.outdir = args.pop(0) + elif arg == "--logdir=": + options.logdir = arg[:9] + elif arg == "--logdir": + options.logdir = args.pop(0) + elif arg == "--help": + print("""outsftphook.py: an reprepro outhook example using sftp +This hook sends changed files over sftp to a remote host. It is usually put into +conf/options as outhook, but may also be called manually. +Options: + --verbose tell what you did + --basedir DIR sets the following to default values + --outdir DIR directory to find pool/ and dist/ directories in + --logdir DIR directory to check for unprocessed outlog files + --pending process pending files instead of arguments + --autoretry reprocess older pending files, too + --ignore-pending ignore pending files + --force-order do not bail out if the given files are not ordered + --debug-sftp=N debug sftp.py (or your remote sftp server) +""") + raise SystemExit(0) + else: + raise CriticalError("Unexpected command line option %s" %repr(arg)) + if options.pending and options.ignorepending: + raise CriticalError("Cannot do both --pending and --ignore-pending") + if options.autoretry and options.forceorder: + raise CriticalError("Cannot do both --pending and --force-order") + if options.autoretry and options.ignorepending: + raise CriticalError("Cannot do both --autoretry and --ignore-pending") + # we need confdir, logdir and outdir, if they are given, all is done + if options.logdir is not None and options.outdir is not None and options.confdir is not None: + return + # otherwise it gets more complicated... + preconfdir = options.confdir + if preconfdir is None: + preconfdir = os.environ.get("REPREPRO_CONFIG_DIR", None) + if preconfdir is None: + if options.basedir is not None: + preconfdir = options.basedir + "/conf" + elif "REPREPRO_BASE_DIR" in os.environ: + preconfdir = os.environ["REPREPRO_BASE_DIR"] + "/conf" + else: + raise CriticalError("If not called by reprepro, please either give (--logdir and --outdir) or --basedir!") + optionsfile = preconfdir + "/options" + if os.path.exists(optionsfile): + f = open(optionsfile, "r") + for line in f: + line = line.strip() + if len(line) == 0 or line[0] == '#' or line[0] == ';': + continue + line = line.split() + if line[0] == "basedir" and options.basedir is None: + options.basedir = line[1] + elif line[0] == "confdir" and options.confdir is None: + options.confdir = line[1] + elif line[0] == "logdir" and options.logdir is None: + options.logdir = line[1] + elif line[0] == "outdir" and options.outdir is None: + options.outdir = line[1] + f.close() + if options.basedir is None: + options.basedir = os.environ.get("REPREPRO_BASE_DIR", None) + if options.outdir is None: + if options.basedir is None: + raise CriticalError("Need --basedir if not called by reprepro") + options.outdir = options.basedir + if options.logdir is None: + if options.basedir is None: + raise CriticalError("Need --basedir if not called by reprepro") + options.logdir = options.basedir + "/logs" + if options.confdir is None: + if "REPREPRO_CONFIG_DIR" in os.environ: + options.confdir = os.environ["REPREPRO_CONFIG_DIR"] + else: + if options.basedir is None: + raise CriticalError("Need --basedir if not called by reprepro") + options.confdir = options.basedir + "/conf" + +def main(args): + global errors, servername, username, targetdir + if "REPREPRO_OUT_DIR" in os.environ or "REPREPRO_LOG_DIR" in os.environ: + # assume being called by reprepro if one of those variable + # is set, so they all should be set: + options.outdir = os.environ["REPREPRO_OUT_DIR"] + options.logdir = os.environ["REPREPRO_LOG_DIR"] + options.confdir = os.environ["REPREPRO_CONFIG_DIR"] + else: + parseoptions(args) + assert (options.outdir and (options.ignorepending or options.logdir) and options.confdir) + conffilename = options.confdir + "/outsftphook.conf" + if os.path.exists(conffilename): + conffile = open(conffilename, "r") + for line in conffile: + line = line.strip().split(None, 1) + if len(line) == 0 or line[0].startswith("#"): + continue + if line[0] == "servername": + servername = line[1] + elif line[0] == "username": + username = line[1] + elif line[0] == "targetdir": + targetdir = line[1] + elif line[0] == "debug": + if options.debugsftp is None: + try: + options.debugsftp = int(line[1]) + except Exception: + raise CriticalError(("Cannot parse %s: " + + "unparseable number %s") % + (repr(conffilename), repr(line[1]))) + elif line[0] == "verbose": + if line[1].lower() in {'yes', 'on', '1', 'true'}: + if options.verbose is None: + options.verbose = True + elif line[1].lower() in {'no', 'off', '0', 'false'}: + if options.verbose is None: + options.verbose = False + else: + raise CriticalError(("Cannot parse %s: " + + "unparseable truth value %s") % + (repr(conffilename), repr(line[1]))) + elif line[0] == "autoretry": + if line[1].lower() in {'yes', 'on', '1', 'true'}: + if options.autoretry is None: + options.autoretry = True + elif line[1].lower() in {'no', 'off', '0', 'false'}: + if options.autoretry is None: + options.autoretry = False + else: + raise CriticalError(("Cannot parse %s: " + + "unparseable truth value %s") % + (repr(conffilename), repr(line[1]))) + else: + raise CriticalError("Cannot parse %s: unknown option %s" % + (repr(conffilename), repr(line[0]))) + conffile.close() + if options.debugsftp is None: + options.debugsftp = 0 + if targetdir and not targetdir.endswith("/"): + targetdir = targetdir + "/" + if not servername: + raise CriticalError("No servername configured!") + if not username: + raise CriticalError("No username configured!") + + if len(args) <= 0: + if not options.pending: + raise CriticalError("No .outlog files given at command line!") + else: + if options.pending: + raise CriticalError("--pending might not be combined with arguments!") + if options.ignorepending: + pendinglogs = set() + else: + pendinglogs = set(name for name in os.listdir(options.logdir) + if name.endswith(".outlog")) + maxbasename = None + for f in args: + if len(f) < 8 or f[-7:] != ".outlog": + raise CriticalError("command line argument '%s' does not look like a .outlog file!" % f) + bn = os.path.basename(f) + pendinglogs.discard(bn) + if maxbasename: + if maxbasename < bn: + maxbasename = bn + elif not options.forceorder: + raise CriticalError("The arguments are not in order (%s <= %s). Applying in this order might not be safe. (use --force-order to proceed in this order anyway)" % (bn, maxbasename)) + else: + maxbasename = bn + if options.pending: + pendinglogs = sorted(pendinglogs) + else: + pendinglogs = sorted(filter(lambda bn: bn < maxbasename, pendinglogs)) + if pendinglogs and not options.autoretry: + raise CriticalError("Unprocessed earlier outlogs found: %s\nYou need to process them first (or use --autoretry or autoretry true in outsftphook.conf to automatically process them)" % repr(pendinglogs)) + if pendinglogs and len(args) > 1: + raise CriticalError("autoretry does not work with multiple log files given (yet).") + args = list(map(lambda bn: options.logdir + "/" + bn, pendinglogs)) + args + outlogfiles = [] + for f in args: + donefile = f[:-7] + ".done" + if options.verbose: + print("Parsing '%s'" % f) + try: + outlogfiles.append(LogFile(f, donefile)) + except ParseError as e: + raise CriticalError("Error parsing %s: %s" %(f, str(e))) + todo = {} + distdirs = {} + workpending = False + for o in reversed(outlogfiles): + workpending |= o.queue(todo, distdirs) + if not workpending: + if options.verbose: + print("Nothing to do") + raise SystemExit(0) + s = sftp.Connection(servername=servername, username=username, debug=options.debugsftp) + doround(s, Round.POOLFILES, todo) + if errors: + raise SystemExit(1) + for d in distdirs.values(): + for t in d.finalizeifready(): + s.start(t) + doround(s, Round.DISTFILES, todo) + if errors: + raise SystemExit(1) + doround(s, Round.DELETES, todo) + if errors: + raise SystemExit(1) + +try: + main(sys.argv[1:]) +except CriticalError as e: + print(str(e), file=sys.stderr) + raise SystemExit(1) diff --git a/docs/outstore.py b/docs/outstore.py new file mode 100755 index 0000000..d0cc14e --- /dev/null +++ b/docs/outstore.py @@ -0,0 +1,237 @@ +#!/usr/bin/python3 +# Copyright (C) 2012 Bernhard R. Link +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License version 2 as +# published by the Free Software Foundation. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program; if not, write to the Free Software +# Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02111-1301 USA + +# This is an example outhook script. +# Actually it is part of the testsuite and does many things +# an actual outhook script would never do. +# But it checks so many aspects of how a outhook script is called +# that it should make quite clear what a outhookscript can expect. + +import sys, os, subprocess, select, dbm + +def poolfile(outdir, name): + s = os.lstat(outdir + '/' + name) + return "poolfile %d bytes" % s.st_size +def distfile(outdir, name): + s = os.lstat(outdir + '/' + name) + return "distfile %d bytes" % s.st_size +def distsymlink(distdir, target): + return "distsymlink -> %s/%s" % (distdir,target) +def collecteddistfile(outdir, name): + if os.path.islink(outdir + '/' + name): + l = os.readlink(outdir + '/' + name) + d = os.path.dirname(name) + while d and l[0:3] == '../': + d = os.path.dirname(d) + l = l[3:] + if d: + d = d + '/' + return "distsymlink -> %s%s" % (d,l) + else: + return distfile(outdir, name) + +def processfile(logfile, donefile, db): + # print("Parsing '%s'" % logfile) + lf = open(logfile, 'r', encoding='utf-8') + newpoolfiles = [] + distributions = [] + deletepoolfiles = [] + mode = 'POOLNEW' + # This parser is wasteful and unnecessarily complicated, but it's + # purpose is mainly making sure the output of reprepro is + # well-formed and no so much targeted at doing actual work. + for l in lf: + if l[-1] != '\n': + raise CriticalError("Malformed file '%s' (not a text file)" % logfile) + l = l[:-1] + fields = l.split('\t') + if fields[0] != 'POOLNEW': + break + if len(fields) != 2: + raise CriticalError("Malformed file '%s': POOLNEW with more than one argument" % logfile) + newpoolfiles.append(fields[1]) + else: + fields = ['EOF'] + while fields[0] == 'BEGIN-DISTRIBUTION' or fields[0] == 'BEGIN-SNAPSHOT': + beginmarker = fields[0] + endmarker = 'END-' + beginmarker[6:] + if len(fields) != 3 and len(fields) != 4: + raise CriticalError("Malformed file '%s': wrong number of arguments for %s" % (logfile,beginmarker)) + distname = fields[1] + distdir = fields[2] + distfiles = [] + distsymlinks = [] + distdeletes = [] + for l in lf: + if l[-1] != '\n': + raise CriticalError("Malformed file '%s' (not a text file)" % logfile) + l = l[:-1] + fields = l.split('\t') + if fields[0] == endmarker: + if len(fields) != 3 and len(fields) != 4: + raise CriticalError("Malformed file '%s': wrong number of arguments for %s" % (logfile, endmarker)) + if fields[1] != distname or fields[2] != distdir: + raise CriticalError("Malformed file '%s': %s not matching previous %s" % (logfile, endmarker, beginmarker)) + break + elif fields[0] == 'DISTKEEP': + continue + elif not fields[0] in ['DISTFILE', 'DISTSYMLINK', 'DISTDELETE']: + raise CriticalError("Malformed file '%s': Unexpected '%s'" % (logfile, fields[0])) + if len(fields) < 3: + raise CriticalError("Malformed file '%s': wrong number of arguments for %s" % (logfile, fields[0])) + if fields[1] != distdir: + raise CriticalError("Malformed file '%s': wrong distdir '%s' in '%s'" %(logfile, fields[1], fields[0])) + if fields[0] == 'DISTFILE': + if len(fields) != 4: + raise CriticalError("Malformed file '%s': wrong number of arguments for %s" % (logfile, fields[0])) + distfiles.append((fields[2], fields[3])) + elif fields[0] == 'DISTDELETE': + if len(fields) != 3: + raise CriticalError("Malformed file '%s': wrong number of arguments for %s" % (logfile, fields[0])) + distdeletes.append(fields[2]) + elif fields[0] == 'DISTSYMLINK': + if len(fields) != 4: + raise CriticalError("Malformed file '%s': wrong number of arguments for %s" % (logfile, fields[0])) + distsymlinks.append((fields[2], fields[3])) + else: + raise CriticalError("Malformed file '%s': unexpected end of file (%s missing)" % (logfile, endmarker)) + distributions.append((distname, distdir, distfiles, distsymlinks, distdeletes)) + l = next(lf, 'EOF\n') + if l[-1] != '\n': + raise CriticalError("Malformed file '%s' (not a text file)" % logfile) + l = l[:-1] + fields = l.split('\t') + while fields[0] == 'POOLDELETE': + if len(fields) != 2: + raise CriticalError("Malformed file '%s': wrong number of arguments for POOLDELETE" % logfile) + deletepoolfiles.append(fields[1]) + l = next(lf, 'EOF\n') + if l[-1] != '\n': + raise CriticalError("Malformed file '%s' (not a text file)" % logfile) + l = l[:-1] + fields = l.split('\t') + if fields[0] != 'EOF' or next(lf, None) != None: + raise CriticalError("Malformed file '%s': Unexpected command '%s'" % (logfile, fields[0])) + # print("Processing '%s'" % logfile) + # Checked input to death, no actualy do something + outdir = os.environ['REPREPRO_OUT_DIR'] + for p in newpoolfiles: + bp = bytes(p, encoding="utf-8") + if bp in db: + raise Exception("duplicate pool file %s" % p) + db[bp] = poolfile(outdir, p) + for distname, distdir, distfiles, distsymlinks, distdeletes in distributions: + for name, orig in distfiles: + db[distdir + '/' + name] = distfile(outdir, orig) + for name, target in distsymlinks: + db[distdir + '/' + name] = distsymlink(distdir, target) + for name in distdeletes: + del db[distdir + '/' + name] + for p in deletepoolfiles: + bp = bytes(p, encoding="utf-8") + if not bp in db: + raise Exception("deleting non-existant pool file %s" % p) + del db[bp] + +def collectfiles(dir, name): + for l in os.listdir(dir + '/' + name): + n = name + '/' + l + if os.path.isdir(dir + '/' + n): + for x in collectfiles(dir, n): + yield x + else: + yield n + +def collectpool(outdir): + if os.path.isdir(outdir + '/pool'): + return ["%s: %s" % (filename, poolfile(outdir, filename)) for filename in collectfiles(outdir, 'pool')] + else: + return [] + +def collectdists(outdir): + if os.path.isdir(outdir + '/dists'): + return ["%s: %s" % (filename, collecteddistfile(outdir, filename)) for filename in collectfiles(outdir, 'dists')] + else: + return [] + +def showdiff(i1, i2): + clean = True + l1 = next(i1, None) + l2 = next(i2, None) + while l1 or l2: + if l1 == l2: + l1 = next(i1, None) + l2 = next(i2, None) + elif l1 != None and (l2 == None or l1 < l2): + print("+ %s" % l1) + clean = False + l1 = next(i1, None) + elif l2 != None and (l1 == None or l1 > l2): + print("- %s" % l2) + clean = False + l2 = next(i2, None) + else: + raise("unexpected") + return clean + +def check(db): + outdir = os.environ['REPREPRO_OUT_DIR'] + actualfiles = collectpool(outdir) + actualfiles.extend(collectdists(outdir)) + + expectedfiles = [] + for k in db.keys(): + expectedfiles.append("%s: %s" % (k.decode(encoding='utf-8'), db[k].decode(encoding='utf-8'))) + expectedfiles.sort() + actualfiles.sort() + if not showdiff(iter(expectedfiles), iter(actualfiles)): + raise CriticalError("outdir does not match expected state") + +class CriticalError(Exception): + pass + +def main(args): + if len(args) <= 0: + raise CriticalError("No .outlog files given at command line!") + + if len(args) == 1 and args[0] == '--print': + db = dbm.open(os.environ['REPREPRO_OUT_DB'], 'r') + for k in sort(db.keys()): + print("%s: %s" % (k, db[k])) + return + if len(args) == 1 and args[0] == '--check': + db = dbm.open(os.environ['REPREPRO_OUT_DB'], 'r') + check(db) + return + + for f in args: + if len(f) < 8 or f[-7:] != ".outlog": + raise CriticalError("command line argument '%s' does not look like a .outlog file!" % f) + + db = dbm.open(os.environ['REPREPRO_OUT_DB'], 'c') + + for f in args: + donefile = f[:-7] + ".outlogdone" + if os.path.exists(donefile): + print("Ignoring '%s' as '%s' already exists!" % (f,donefile), file=sys.stderr) + continue + processfile(f, donefile, db) + +try: + main(sys.argv[1:]) +except CriticalError as e: + print(str(e), file=sys.stderr) + raise SystemExit(1) diff --git a/docs/pdiff.example b/docs/pdiff.example new file mode 100755 index 0000000..bf58d64 --- /dev/null +++ b/docs/pdiff.example @@ -0,0 +1,255 @@ +#!/usr/bin/env python3 + +# Note that rrdtool can also generate .diff files and +# is usually more efficient at it than this script + +############################################################################# +# generates partial package updates list as reprepro hook +# (to be used by apt-get >= 0.6.44, apt-qupdate or things compatible with that) + +# changes Copyright 2005 Bernhard R. Link <brlink@debian.org> +# as this is used as hook, it does not need any parsing of +# Configuration or Handling of architectures and components. +# Also reprepro will present old and new file, so it does not +# need to store a permanent copy of the last version. +# This needs either python3-apt installed or you have to change +# it to use another sha1 calculation method. + +# HOW TO USE: +# - install python3-apt +# - make sure your paths contain no ' characters. +# - be aware this is still quite experimental and might not +# report some errors properly +# - uncompress this file if it is compressed +# - copy this file to your conf/ directory as "pdiff" +# - make it executable +# - add something like the following to the every distribution +# in conf/distributions you want to have diffs for: +# +# DscIndices: Sources Release . .gz pdiff +# DebIndices: Packages Release . .gz pdiff +# +# The first line is for source indices, the second for binary indices. +# Make sure uncompressed index files are generated (the single dot in those +# lines), as this version only diffs the uncompressed files. + +# This file is a heavily modified version of apt-qupdate's tiffany, +# (downloaded from http://ftp-master.debian.org/~ajt/tiffani/tiffany +# 2005-02-20)which says: +#-------------------------------------------------------------------- +# idea and basic implementation by Anthony, some changes by Andreas +# parts are stolen from ziyi +# +# Copyright (C) 2004-5 Anthony Towns <aj@azure.humbug.org.au> +# Copyright (C) 2004-5 Andreas Barth <aba@not.so.argh.org> +#-------------------------------------------------------------------- + +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2 of the License, or +# (at your option) any later version. + +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. + +# You should have received a copy of the GNU General Public License +# along with this program; if not, write to the Free Software +# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA + +############################################################################# + +import sys, os, time +import apt_pkg + +################################################################################ + +def usage (exit_code=0): + print("""Usage: pdiff.example directory newfile oldfile mode 3>releaselog +Write out ed-style diffs to Packages/Source lists +This file is intended to be called by reprepro as hook +given to DebIndices, UDebIndices or DscIndices. + + """) + sys.exit(exit_code); + + +def tryunlink(file): + try: + os.unlink(file) + except OSError: + print("warning: removing of %s denied" % (file)) + +class Updates: + def __init__(self, readpath = None): + self.can_path = None + self.history = {} + self.max = 14 + self.readpath = readpath + self.filesizesha1 = None + + if readpath: + try: + f = open(readpath + "/Index") + x = f.readline() + + def read_hashs(ind, f, self, x=x): + while 1: + x = f.readline() + if not x or x[0] != " ": break + l = x.split() + if l[2] not in self.history: + self.history[l[2]] = [None,None] + self.history[l[2]][ind] = (l[0], int(l[1])) + return x + + while x: + l = x.split() + + if len(l) == 0: + x = f.readline() + continue + + if l[0] == "SHA1-History:": + x = read_hashs(0,f,self) + continue + + if l[0] == "SHA1-Patches:": + x = read_hashs(1,f,self) + continue + + if l[0] == "Canonical-Name:" or l[0]=="Canonical-Path:": + self.can_path = l[1] + + if l[0] == "SHA1-Current:" and len(l) == 3: + self.filesizesha1 = (l[1], int(l[2])) + + x = f.readline() + + except IOError: + 0 + + def dump(self, out=sys.stdout): + if self.can_path: + out.write("Canonical-Path: %s\n" % (self.can_path)) + + if self.filesizesha1: + out.write("SHA1-Current: %s %7d\n" % (self.filesizesha1)) + + hs = self.history + l = list(self.history.keys()) + l.sort() + + cnt = len(l) + if cnt > self.max: + for h in l[:cnt-self.max]: + tryunlink("%s/%s.gz" % (self.readpath, h)) + del hs[h] + l = l[cnt-self.max:] + + out.write("SHA1-History:\n") + for h in l: + out.write(" %s %7d %s\n" % (hs[h][0][0], hs[h][0][1], h)) + out.write("SHA1-Patches:\n") + for h in l: + out.write(" %s %7d %s\n" % (hs[h][1][0], hs[h][1][1], h)) + +def sizesha1(f): + size = os.fstat(f.fileno())[6] + f.seek(0) + sha1sum = apt_pkg.sha1sum(f) + return (sha1sum, size) + +def getsizesha1(name): + f = open(name, "r") + r = sizesha1(f) + f.close() + return r + +def main(): + if len(sys.argv) != 5: + usage(1) + + directory = sys.argv[1] + newrelfile = sys.argv[2] + oldrelfile = sys.argv[3] + mode = sys.argv[4] + + # this is only needed with reprepro <= 0.7 + if oldrelfile.endswith(".gz"): + sys.exit(0); + + + oldfile = "%s/%s" % (directory,oldrelfile) + newfile= "%s/%s" % (directory,newrelfile) + + outdir = oldfile + ".diff" + + if mode == "old": + # Nothing to do... + if os.path.isfile(outdir + "/Index"): + os.write(3,(oldrelfile + ".diff/Index").encode("utf-8")) + sys.exit(0); + + if mode == "new": + # TODO: delete possible existing Index and patch files? + sys.exit(0); + + print("making diffs between %s and %s: " % (oldfile, newfile)) + + o = os.popen("date +%Y-%m-%d-%H%M.%S") + patchname = o.readline()[:-1] + o.close() + difffile = "%s/%s" % (outdir, patchname) + + upd = Updates(outdir) + + oldsizesha1 = getsizesha1(oldfile) + + # should probably early exit if either of these checks fail + # alternatively (optionally?) could just trim the patch history + + if upd.filesizesha1: + if upd.filesizesha1 != oldsizesha1: + print("old file seems to have changed! %s %s => %s %s" % (upd.filesizesha1 + oldsizesha1)) + sys.exit(1); + + newsizesha1 = getsizesha1(newfile) + + if newsizesha1 == oldsizesha1: + print("file unchanged, not generating diff") + if os.path.isfile(outdir + "/Index"): + os.write(3,(oldrelfile + ".diff/Index\n").encode("utf-8")) + else: + if not os.path.isdir(outdir): os.mkdir(outdir) + print("generating diff") + while os.path.isfile(difffile + ".gz"): + print("This was too fast, diffile already there, waiting a bit...") + time.sleep(2) + o = os.popen("date +%Y-%m-%d-%H%M.%S") + patchname = o.readline()[:-1] + o.close() + difffile = "%s/%s" % (outdir, patchname) + + # TODO make this without shell... + os.system("diff --ed '%s' '%s' > '%s'" % + (oldfile,newfile, difffile)) + difsizesha1 = getsizesha1(difffile) + # TODO dito + os.system("gzip -9 '%s'" %difffile) + + + upd.history[patchname] = (oldsizesha1, difsizesha1) + upd.filesizesha1 = newsizesha1 + + f = open(outdir + "/Index.new", "w") + upd.dump(f) + f.close() +# Specifying the index should be enough, it contains checksums for the diffs + os.write(3,(oldrelfile + ".diff/Index.new\n").encode("utf-8")) + +################################################################################ + +if __name__ == '__main__': + main() diff --git a/docs/recovery b/docs/recovery new file mode 100644 index 0000000..80740c0 --- /dev/null +++ b/docs/recovery @@ -0,0 +1,67 @@ +Some tips what to do if (hopefully never), your database gets +corrupted: + +First there are three different databases used, residing in three +files in your --dbdir (normally db/): + +1) references.db +This file only contains the information which file in the pool/ +is needed by which target (i.e. which type/distribution/ +component/architecture quadruple). This is simply repairable by +deleting references.db and running "rereference". + +The current state of this database can be seen with "dumpreferences". +All references from some specific target can be removed with +"_removereferences". + +2) files.db and checksums.db +These files contain the information about which files in the pool/ dir +are known and what checksums they have. Files not in here will not be +deleted with "deleteunreferenced". Files being wrong here will not realized +(and thus not corrected even if told to be newly included) + +If both files exist, files.db is the canonical information and checksums.db +can be regenerated with a call to collectnewchecksums. +If only checksums.db is there, only that it used. (This means: if you +have called collectnewchecksums since you last used a version prior to 3.3 +with this repository, you can just delete files.db. But make sure to +never ever use a version prior to 3.0 on this repository after that.) + +To get this database in text form use "_listchecksums" without argument, +to add items manually pipe it into "_addchecksums". (Filenames +are handled as strings, so be careful). + +If the database is completely lost or broken, you can regain it by moving +files.db and checksums.db out of the way and running: +find $BASEDIR/pool -type f -printf "pool/%P\n" | reprepro -b $BASEDIR _detect +(or cd $BASEDIR && find pool -type f -print | reprepro -b . _detect) + +Also single files can be removed or added by "_forget" and "_detect". +(Again note filekeys will be handled as strings, so leading "./", double + slashes, "/./", symlinks and the like make them differ). + +4) packages.db +This file contains multiple databases, one for each target, containing +the chunks from the Packages or Sources files, indexed by package name. + +This one is the hardest to reconstruct. If you have still an uncorrupted +"dists/" directory around, (e.g. you just deleted db/ accidentally), +it can be reconstructed by moving your dists/ directory to some other place, +moving the packages.db file (if still existent) away, and set every distribution +in conf/distributions a "Update: localreadd" with localreadd in conf/updates like: +Name: localreadd +Suite: * +Method: copy:/<otherplace> + +with otherplace being the place you moved the dists/ directory too. + +If the packages database is corrupt, the described way can at least reconstruct +the Packages still landing in the Packages.gz and Sources.gz files. +If references.db is still accessible via dumpreferences, it can give hints +where the other files belong to. Otherwise removing references.db and calling +"rereference" and then "dumpunreferenced" will give you a list of files not +yet anywhere. + +Last but not least, there are also the "check" and "checkpool" commands, which +can give some hints about inconsistencies. (Check will also read files missing +from files.db+checksums.db if they are needed by packages but in the pool). diff --git a/docs/reprepro.1 b/docs/reprepro.1 new file mode 100644 index 0000000..34f1a3c --- /dev/null +++ b/docs/reprepro.1 @@ -0,0 +1,2847 @@ +.TH REPREPRO 1 "2013-05-04" "reprepro" REPREPRO +.SH NAME +reprepro \- produce, manage and sync a local repository of Debian packages +.mso www.tmac +.SH SYNOPSIS +.B reprepro \-\-help + +.B reprepro +[ +\fIoptions\fP +] +\fIcommand\fP +[ +\fIper\-command\-arguments\fP +] +.SH DESCRIPTION +reprepro is a tool to manage a repository of Debian packages +(.deb, .udeb, .dsc, ...). +It stores files either being injected manually or +downloaded from some other repository (partially) mirrored +into a pool/ hierarchy. +Managed packages and checksums of files are stored in a +Berkeley DB database file, +so no database server is needed. +Checking signatures of mirrored repositories and creating +signatures of the generated Package indices is supported. + +Former working title of this program was mirrorer. +.SH "GLOBAL OPTIONS" +Options can be specified before the command. Each affects a different +subset of commands and is ignored by other commands. +.TP +.B \-h \-\-help +Displays a short list of options and commands with description. +.TP +.B \-v, \-V, \-\-verbose +Be more verbose. Can be applied multiple times. One uppercase +.B \-V +counts as five lowercase +.B \-v. +.TP +.B \-\-silent +Be less verbose. Can be applied multiple times. One +.B \-v +and one +.B \-s +cancel each other out. +.TP +.B \-f, \-\-force +This option is ignored, as it no longer exists. +.TP +.B \-b, \-\-basedir \fIbasedir\fP +Sets the base\-dir all other default directories are relative to. +If none is supplied and the +.B REPREPRO_BASE_DIR +environment variable is not set either, the current directory +will be used. +.TP +.B \-\-outdir \fIoutdir\fP +Sets the base\-dir of the repository to manage, i.e. where the +.B pool/ +subdirectory resides. And in which the +.B dists/ +directory is placed by default. +If this starts with '\fB+b/\fP', it is relative to basedir. + +The default for this is \fIbasedir\fP. +.TP +.B \-\-confdir \fIconfdir\fP +Sets the directory where the configuration is searched in. + +If this starts with '\fB+b/\fP', it is relative to basedir. + +If none is given, \fB+b/conf\fP (i.e. \fIbasedir\fP\fB/conf\fP) will be used. +.TP +.B \-\-distdir \fIdistdir\fP +Sets the directory to generate index files relatively to. (i.e. things like +Packages.gz, Sources.gz and Release.gpg) + +If this starts with '\fB+b/\fP', it is relative to basedir, +if starting with '\fB+o/\fP' relative to outdir. + +If none is given, \fB+o/dists\fP (i.e. \fIoutdir\fP\fB/dists\fP) is used. + +.B Note: +apt has +.B dists +hard-coded in it, so this is mostly only useful for testing or when your webserver +pretends another directory structure than your physical layout. + +.B Warning: +Beware when changing this forth and back between two values not ending +in the same directory. +Reprepro only looks if files it wants are there. If nothing of the content +changed and there is a file it will not touch it, assuming it is the one it +wrote last time, assuming any different \fB\-\-distdir\fP ended in the same +directory. +So either clean a directory before setting \fB\-\-distdir\fP to it or +do an \fBexport\fP with the new one first to have a consistent state. +.TP +.B \-\-logdir \fIlogdir\fP +The directory where files generated by the \fBLog:\fP directive are +stored if they have no absolute path. + +If this starts with '\fB+b/\fP', it is relative to basedir, +if starting with '\fB+o/\fP' relative to outdir, +with '\fB+c/\fP' relative to confdir. + +If none is given, \fB+b/logs\fP (i.e. \fIbasedir\fP\fB/logs\fP) is used. +.TP +.B \-\-dbdir \fIdbdir\fP +Sets the directory where reprepro keeps its databases. + +If this starts with '\fB+b/\fP', it is relative to basedir, +if starting with '\fB+o/\fP' relative to outdir, +with '\fB+c/\fP' relative to confdir. + +If none is given, \fB+b/db\fP (i.e. \fIbasedir\fP\fB/db\fP) is used. + +.B Note: +This is permanent data, no cache. One has almost to regenerate the whole +repository when this is lost. +.TP +.B \-\-listdir \fIlistdir\fP +Sets the directory where it downloads indices to when importing +from other repositories. This is temporary data and can be safely deleted +when not in an update run. + +If this starts with '\fB+b/\fP', it is relative to basedir, +if starting with '\fB+o/\fP' relative to outdir, +with '\fB+c/\fP' relative to confdir. + +If none is given, \fB+b/lists\fP (i.e. \fIbasedir\fP\fB/lists\fP) is used. +.TP +.B \-\-morguedir \fImorguedir\fP +Files deleted from the pool are stored into \fImorguedir\fP. + +If this starts with '\fB+b/\fP', it is relative to basedir, +if starting with '\fB+o/\fP' relative to outdir, +with '\fB+c/\fP' relative to confdir. + +If none is given, deleted files are just deleted. +.TP +.B \-\-methoddir \fImethoddir\fP +Look in \fImethoddir\fP instead of +.B /usr/lib/apt/methods +for methods to call when importing from other repositories. +.TP +.B \-C, \-\-component \fIcomponents\fP +Limit the specified command to this components only. +This will force added packages to this components, +limit removing packages from this components, +only list packages in this components, +and/or otherwise only look at packages in this components, +depending on the command in question. + +Multiple components are specified by separating them with \fB|\fP, +as in \fB\-C 'main|contrib'\fP. +.TP +.B \-A, \-\-architecture \fIarchitectures\fP +Limit the specified command to this architectures only. +(i.e. only list such packages, +only remove packages from the specified architectures, +or otherwise only look at/act on this architectures +depending on the specific command). + +Multiple architectures are specified by separating them with \fB|\fP, +as in \fB\-A 'sparc|i386'\fP. + +Note that architecture \fBall\fP packages can be included to each +architecture but are then handled separately. +Thus by using \fB\-A\fP in a specific way one can have different versions of +an architecture \fBall\fP package in different architectures of the +same distribution. +.TP +.B \-T, \-\-type \fRdsc|deb|udeb +Limit the specified command to this packagetypes only. +(i.e. only list such packages, only remove such packages, only include +such packages, ...) +.TP +.B \-S, \-\-section \fIsection\fP +Overrides the section of inclusions. (Also override possible override files) +.TP +.B \-P, \-\-priority \fIpriority\fP +Overrides the priority of inclusions. (Also override possible override files) +.TP +.BR \-\-export= ( silent\-never | never | changed | lookedat | force ) +This option specify whether and how the high level actions +(e.g. install, update, pull, delete) +should export the index files of the distributions they work with. +.TP +.BR \-\-export=lookedat +In this mode every distribution the action handled will be exported, +unless there was an error possibly corrupting it. +.br +\fINote\fP that only missing files and files whose intended content changed +between before and after the action will be written. +To get a guaranteed current export, use the \fBexport\fP action. +.br +For backwards compatibility, \fBlookedat\fP is also available under the +old name \fBnormal\fP. +The name \fBnormal\fP is deprecated and will be removed in future versions. +.TP +.BR \-\-export=changed +In this mode every distribution actually changed will be exported, +unless there was an error possibly corrupting it. +(i.e. if nothing changed, not even missing files will be created.) +.br +\fINote\fP that only missing files and files whose intended content changed +between before and after the action will be written. +To get a guaranteed current export, use the \fBexport\fP action. +.TP +.BR \-\-export=force +Always export all distributions looked at, even if there was some +error possibly bringing it into a inconsistent state. +.TP +.BR \-\-export=never +No index files are exported. You will have to call \fBexport\fP later. +.br +\fINote\fP that you most likely additionally need the \fB\-\-keepunreferencedfiles\fP +option, if you do not want some of the files pointed to by the untouched index +files to vanish. +.TP +.BR \-\-export=silent-never +Like never, but suppress most output about that. +.TP +.B \-\-ignore=\fIwhat\fP +Ignore errors of type \fIwhat\fP. See the section \fBERROR IGNORING\fP +for possible values. +.TP +.B \-\-nolistsdownload +When running \fBupdate\fP, \fBcheckupdate\fP or \fBpredelete\fP do not download +any Release or index files. +This is hardly useful except when you just run one of those +command for the same distributions. +And even then reprepro is usually good in +not downloading except \fBRelease\fP and \fBRelease.gpg\fP files again. +.TP +.B \-\-nothingiserror +If nothing was done, return with exitcode 1 instead of the usual 0. + +Note that "nothing was done" means the primary purpose of the action +in question. +Auxiliary actions (opening and closing the database, exporting missing +files with \-\-export=lookedat, ...) usually do not count. +Also note that this is not very well tested. +If you find an action that claims to have done something in some cases +where you think it should not, please let me know. +.TP +.B \-\-keeptemporaries +Do not delete temporary \fB.new\fP files when exporting a distribution +fails. +(reprepro first create \fB.new\fP files in the \fBdists\fP directory and +only if everything is generated, all files are put into their final place +at once. +If this option is not specified and something fails, all are deleted +to keep \fBdists\fP clean). +.TP +.B \-\-keepunreferencedfiles +Do not delete files that are no longer used because the package they +are from is deleted/replaced with a newer version from the last distribution +it was in. +.TP +.B \-\-keepunusednewfiles +The include, includedsc, includedeb and processincoming by default delete +any file they added to the pool that is not marked used at the end of the +operation. +While this keeps the pool clean and allows changing before trying to add again, +this needs copying and checksum calculation every time one tries to add a file. +.TP +.B \-\-keepdirectories +Do not try to rmdir parent directories after files or directories +have been removed from them. +(Do this if your directories have special permissions you want keep, +do not want to be pestered with warnings about errors to remove them, +or have a buggy rmdir call deleting non-empty directories.) +.TP +.B \-\-ask\-passphrase +Ask for passphrases when signing things and one is needed. This is a quick +and dirty and unsafe implementation using the obsolete \fBgetpass(3)\fP +function with the description gpgme is supplying. +So the prompt will look quite funny and support for passphrases with +more than 8 characters depend on your libc. +Use of this option is not recommended. Use gpg-agent with pinentry instead. + +(With current versions of gnupg you need to set \fBpinentry-mode loopback\fP +in your .gnupg/gpg.conf file to use \fB\-\-ask\-passphrase\fP. +Without that option gnupg uses the much safer and recommended pinentry instead). +.TP +.B \-\-noskipold +When updating do not skip targets where no new index files and no files +marked as already processed are available. + +If you changed a script to preprocess downloaded index files or +changed a Listfilter, you most likely want to call reprepro with \-\-noskipold. +.TP +.B \-\-waitforlock \fIcount +If there is a lockfile indicating another instance of reprepro is currently +using the database, retry \fIcount\fP times after waiting for 10 seconds +each time. +The default is 0 and means to error out instantly. +.TP +.B \-\-spacecheck full\fR|\fPnone +The default is \fBfull\fR: +.br +In the update commands, check for every to be downloaded file which filesystem +it is on and how much space is left. +.br +To disable this behaviour, use \fBnone\fP. +.TP +.BI \-\-dbsafetymargin " bytes-count" +If checking for free space, reserve \fIbyte-count\fP bytes on the filesystem +containing the \fBdb/\fP directory. +The default is 104857600 (i.e. 100MB), which is quite large. +But as there is no way to know in advance how large the databases will +grow and libdb is extremely touchy in that regard, lower only when you know +what you do. +.TP +.BI \-\-safetymargin " bytes-count" +If checking for free space, reserve \fIbyte-count\fP bytes on filesystems +not containing the \fBdb/\fP directory. +The default is 1048576 (i.e. 1MB). +.TP +.B \-\-noguessgpgtty +Don't set the environment variable +.BR GPG_TTY , +even when it is not set, stdin is terminal and +.B /proc/self/fd/0 +is a readable symbolic link. +.TP +.B \-\-gnupghome +Set the +.B GNUPGHOME +evnironment variable to the given directory as argument to this option. +And your gpg will most likely use the content of this variable +instead of "~/.gnupg". +Take a look at +.BR gpg (1) +to be sure. +This option in the command line is usually not very useful, as it is possible +to set the environment variable directly. +Its main reason for existence is that it can be used in \fIconf\fP\fB/options\fP. +.TP +.BI \-\-gunzip " gz-uncompressor" +While reprepro links against \fBlibz\fP, it will look for the program given +with this option (or \fBgunzip\fP if not given) and use that when uncompressing +index files while downloading from remote repositories. +(So that downloading and uncompression can happen at the same time). +If the program is not found or is \fBNONE\fP (all-uppercase) then uncompressing +will always be done using the built in uncompression method. +The program has to accept the compressed file as stdin and write +the uncompressed file into stdout. +.TP +.BI \-\-bunzip2 " bz2-uncompressor" +When uncompressing downloaded index files or if not linked against \fBlibbz2\fP +reprepro will use this program to uncompress \fB.bz2\fP files. +The default value is \fBbunzip2\fP. +If the program is not found or is \fBNONE\fP (all-uppercase) then uncompressing +will always be done using the built in uncompression method or not be possible +if not linked against \fBlibbz2\fP. +The program has to accept the compressed file as stdin and write +the uncompressed file into stdout. +.TP +.BI \-\-unlzma " lzma-uncompressor" +When uncompressing downloaded index files or if not linked against \fBliblzma\fP +reprepro will use this program to uncompress \fB.lzma\fP files. +The default value is \fBunlzma\fP. +If the program is not found or is \fBNONE\fP (all-uppercase) +then uncompressing lzma files will always be done using +the built in uncompression method +or not be possible if not linked against \fBliblzma\fP. +The program has to accept the compressed file as stdin and write +the uncompressed file into stdout. +.TP +.BI \-\-unxz " xz-uncompressor" +When uncompressing downloaded index files or if not linked against \fBliblzma\fP +reprepro will use this program to uncompress \fB.xz\fP files. +The default value is \fBunxz\fP. +If the program is not found or is \fBNONE\fP (all-uppercase) +then uncompressing xz files will always be done using +the built in uncompression method +or not be possible if not linked against \fBliblzma\fP. +The program has to accept the compressed file as stdin and write +the uncompressed file into stdout. +.TP +.BI \-\-lunzip " lzip-uncompressor" +When trying to uncompress or read \fPlzip\fP compressed files, this program +will be used. +The default value is \fBlunzip\fP. +If the program is not found or is \fBNONE\fP (all-uppercase) then uncompressing +lz files will not be possible. +The program has to accept the compressed file as stdin and write +the uncompressed file into stdout. +Note that .lz support is \fBDEPRECATED\fP and will be removed in the future. +.TP +.BI \-\-list\-max " count" +Limits the output of \fBlist\fP, \fBlistmatched\fP and \fBlistfilter\fP to the first \fIcount\fP +results. +The default is 0, which means unlimited. +.TP +.BI \-\-list\-skip " count" +Omitts the first \fIcount\fP results from the output of +\fBlist\fP, \fBlistmatched\fP and \fBlistfilter\fP. +.TP +.BI \-\-list\-format " format" +Set the output format of \fBlist\fP, \fBlistmatched\fP and \fBlistfilter\fP commands. +The format is similar to dpkg\-query's \fB\-\-showformat\fP: +fields are specified as +.BI ${ fieldname } +or +.BI ${ fieldname ; length }\fR.\fP +Zero length or no length means unlimited. +Positive numbers mean fill with spaces right, negative fill with spaces left. + +.BR \[rs]n ", " \[rs]r ", " \[rs]t ", " \[rs]0 +are new-line, carriage-return, tabulator and zero-byte. +Backslash (\fB\[rs]\fP) can be used to escape every non-letter-or-digit. + +The special field names +.BR $identifier ", " $architecture ", " $component ", " $type ", " $codename +denote where the package was found. + +The special field names +.BR $source " and " $sourceversion +denote the source and source version a package belongs to. +(i.e. +.B ${$source} +will either be the same as +.B ${source} +(without a possible version in parentheses at the end) +or the same as +.BR ${package} . + +The special field names +.BR $basename ", " $filekey " and " $fullfilename +denote the first package file part of this entry +(i.e. usually the .deb, .udeb or .dsc file) +as basename, as filekey (filename relative to the outdir) +and the full filename with outdir prepended +(i.e. as relative or absolute as your +outdir (or basedir if you did not set outdir) is). + +When \fB\-\-list\-format\fP is not given or \fBNONE\fP, then the default +is equivalent to +.br +.BR "${$identifier} ${package} ${version}\[rs]n" . + +Escaping digits or letters not in above list, +using dollars not escaped outside specified constructs, +or any field names not listed as special and not consisting entirely out of +letters, digits and minus signs have undefined behaviour +and might change meaning without any further notice. + +If you give this option on the command line, +don't forget that $ is also interpreted by your shell. +So you have to properly escape it. +For example by putting the whole argument to \-\-list\-format in single quotes. +.TP +.B \-\-show\-percent +When downloading packages, show each completed percent of completed package +downloads together with the size of completely downloaded packages. +(Repeating this option increases the frequency of this output). +.TP +.B \-\-onlysmalldeletes +The pull and update commands will skip every distribution in which one +target loses more than 20% of its packages (and at least 10). + +Using this option (or putting it in the options config file) can +avoid removing large quantities of data but means you might often +give +.B \-\-noonlysmalldeletes +to override it. +.TP +.B \-\-restrict \fIsrc\fP\fR[\fP=\fIversion\fP\fR|\fP:\fItype\fP\fR]\fP +Restrict a \fBpull\fP or \fBupdate\fP to only act on packages belonging +to source-package \fIsrc\fP. +Any other package will not be updated (unless it matches a \fB\-\-restrict\-bin\fP). +Only packages that would otherwise be updated or are at least marked with \fBhold\fP +in a \fBFilterList\fP or \fBFilerSrcList\fP will be updated. + +The action can be restricted to a source version using a equal sign or +changed to another type (see FilterList) using a colon. + +This option can be given multiple times to list multiple packages, but each +package may only be named once (even when there are different versions or types). +.TP +.B \-\-restrict\-binary \fIname\fP\fR[\fP=\fIversion\fP\fR|\fP:\fItype\fP\fR]\fP +Like \fB\-\-restrict\fP but restrict to binary packages (\fB.deb\fP and \fB.udeb\fP). +Source packages are not upgraded unless they appear in a \fB\-\-restrict\fP. +.TP +.B \-\-restrict\-file \fIfilename\fP +Like \fB\-\-restrict\fP but read a whole file in the \fBFilterSrcList\fP format. +.TP +.B \-\-restrict\-file\-bin \fIfilename\fP +Like \fB\-\-restrict\-bin\fP but read a whole file in the \fBFilterList\fP format. +.TP +.B \-\-endhook \fIhookscript\fP + +Run the specified \fIhookscript\fP once reprepro exits. +It will get the usual \fBREPREPRO_\fP* environment variables set (or unset) +and additionally a variable \fBREPREPRO_EXIT_CODE\fP that is the exit code +with which reprepro would have exited (the hook is always called once the +initial parsing of global options and the command name is done, no matter +if reprepro did anything or not). +Reprepro will return to the calling process with the exitcode of this script. +Reprepro has closed all its databases and removed all its locks, +so you can run reprepro again in this script +(unless someone else did so in the same repository before, of course). + +The only advantage over running that command always directly after reprepro +is that you can some environment variables set and cannot so easily forget +it if this option is in conf/options. + +The script is supposed to be located relative to \fIconfdir\fP, unless its +name starts with \fB/\fP, \fB./\fP, \fB+b/\fP, \fB+o/\fP, or \fB+c/\fP +and the name may not start (except in the cases given before) with a \fB+\fP. + +An example script looks like: \fB + #!/bin/sh + + if [ "$REPREPRO_EXIT_CODE" \-ne 0 ] ; then + exit "$REPREPRO_EXIT_CODE" + fi + + echo "congratulations, reprepro with arguments: $*" + echo "seems to have run successfully. REPREPRO_ part of the environment is:" + set | grep ^REPREPRO_ + + exit 0 + \fP +.TP +.B \-\-outhook \fIhookscript\fP +\fIhookscript\fP is called with a \fB.outlog\fP file as argument (located +in \fIlogdir\fP) containing a description of all changes made to \fIoutdir\fP. + +The script is supposed to be located relative to \fIconfdir\fP, unless its +name starts with \fB/\fP, \fB./\fP, \fB+b/\fP, \fB+o/\fP, or \fB+c/\fP +and the name may not start (except in the cases given before) with a \fB+\fP. + +For a format of the \fB.outlog\fP files generated for this script see the +\fBmanual.html\fP shipped with reprepro. +.SH COMMANDS +.TP +.BR export " [ " \fIcodenames\fP " ]" +Generate all index files for the specified distributions. + +This regenerates all files unconditionally. +It is only useful if you want to be sure \fBdists\fP is up to date, +you called some other actions with \fB\-\-export=never\fP before or +you want to create an initial empty but fully equipped +.BI dists/ codename +directory. +.TP +.RB " [ " \-\-delete " ] " createsymlinks " [ " \fIcodenames\fP " ]" +Creates \fIsuite\fP symbolic links in the \fBdists/\fP-directory pointing +to the corresponding \fIcodename\fP. + +It will not create links, when multiple of the given codenames +would be linked from the same suite name, or if the link +already exists (though when \fB\-\-delete\fP is given it +will delete already existing symlinks) +.TP +.B list \fIcodename\fP \fR[\fP \fIpackagename\fP \fR]\fP +List all packages (source and binary, except when +.B \-T +or +.B \-A +is given) with the given name in all components (except when +.B \-C +is given) and architectures (except when +.B \-A +is given) of the specified distribution. +If no package name is given, list everything. +The format of the output can be changed with \fB\-\-list\-format\fP. +To only get parts of the result, use \fB\-\-list\-max\fP and +\fB\-\-list\-skip\fP. +.TP +.B listmatched \fIcodename\fP \fIglob\fP +as list, but does not list a single package, but all packages +matching the given shell-like \fIglob\fP. +(i.e. \fB*\fP, \fB?\fP and \fB[\fP\fIchars\fP\fB]\fP are allowed). + +Examples: + +.B reprepro \-b . listmatched test2 'linux\-*' +lists all packages starting with \fBlinux\-\fP. + +.TP +.B listfilter \fIcodename\fP \fIcondition\fP +as list, but does not list a single package, but all packages +matching the given condition. + +The format of the formulas is those of the dependency lines in +Debian packages' control files with some extras. +That means a formula consists of names of fields with a possible +condition for its content in parentheses. +These atoms can be combined with +an exclamation mark '!' (meaning not), +a pipe symbol '|' (meaning or) and +a comma ',' (meaning and). +Additionally parentheses can be used to change binding +(otherwise '!' binds more than '|' than ','). + +The values given in the search expression are directly alphabetically +compared to the headers in the respective index file. +That means that each part \fIFieldname\fP\fB (\fP\fIcmp\fP\fB \fP\fIvalue\fP\fB)\fP +of the formula will be true for exactly those package that have +in the \fBPackage\fP or \fBSources\fP file a line starting with \fIfieldname\fP +and a value is alphabetically \fIcmp\fP to \fIvalue\fP. + +Additionally since reprepro 3.11.0, '\fB%\fP' can be used as comparison operator, +denoting matching a name with shell like wildcard +(with '\fB*\fP', '\fB?\fP' and '\fB[\fP..\fB]\fP'). + +The special field names starting with '\fB$\fP' have special meaning +(available since 3.11.1): + +.B $Version + +The version of the package, comparison is not alphabetically, but as +Debian version strings. + +.B $Source + +The source name of the package. + +.B $SourceVersion + +The source version of the package. + +.B $Architecture + +The architecture the package is in (listfilter) or to be put into. + +.B $Component + +The component the package is in (listfilter) or to be put into. + +.B $Packagetype + +The packagetype of the package. + +Examples: + +.B reprepro \-b . listfilter test2 'Section (== admin)' +will list all packages in distribution test2 with a Section field and the value +of that field being \fBadmin\fP. + +.B reprepro \-b . \-T deb listfilter test2 'Source (== \fIblub\fP) | ( !Source , Package (== \fIblub\fP) )' +will find all .deb Packages with either a Source field blub or +no Source field and a Package field blub. +(That means all package generated by a source package \fIblub\fP, +except those also specifying a version number with its Source). + +.B reprepro \-b . \-T deb listfilter test2 '$Source (==\fIblub\fP)' +is the better way to do this (but only available since 3.11.1). + +.B reprepro \-b . listfilter test2 '$PackageType (==deb), $Source (==\fIblub\fP)' +is another (less efficient) way. + +.B reprepro \-b . listfilter test2 'Package (% linux\-*\-2.6*)' +lists all packages with names starting with \fBlinux\-\fP and later +having an \fB\-2.6\fP. +.TP +.B ls \fIpackage-name\fP +List the versions of the specified package in all distributions. +.TP +.B lsbycomponent \fIpackage-name\fP +Like ls, but group by component (and print component names). +.TP +.B remove \fIcodename\fP \fIpackage-names\fP\fR[\fP=\fIversion\fP\fR]\fP \fI...\fP +Delete packages in the specified distribution, +that have package name listed as argument. +Package versions must be specified by appending '\fB=\fP' and the +version to the name (without spaces). When no version is specified, the latest +package version is removed. + +Note that like any other operation removing or replacing a package, +the old package's files are unreferenced and thus may be automatically +deleted if this was their last reference and no \fB\-\-keepunreferencedfiles\fP +specified. +.TP +.B removematched \fIcodename\fP \fIglob\fP +Delete all packages \fBlistmatched\fP with the same arguments would list. +.TP +.B removefilter \fIcodename\fP \fIcondition\fP +Delete all packages \fBlistfilter\fP with the same arguments would list. +.TP +.B removesrc \fIcodename\fP \fIsource-name\fP \fR[\fP\fIversion\fP\fR]\fP +Remove all packages in distribution \fIcodename\fP belonging to source +package \fIsource-name\fP. +(Limited to those with source version \fIversion\fP if specified). + +If package tracking is activated, it will use that information to find the +packages, otherwise it traverses all package indices for the distribution. +.TP +.B removesrcs \fIcodename\fP \fIsource-name\fP\fR[\fP=\fIversion\fP\fR]\fP \fI...\fP +Like \fBremovesrc\fP, but can be given multiple source names +and source versions must be specified by appending '\fB=\fP' and the version +to the name (without spaces). +.TP +.BR update " [ " \fIcodenames\fP " ]" +Sync the specified distributions (all if none given) as +specified in the config with their upstreams. See the +description of +.B conf/updates +below. +.TP +.BR checkupdate " [ " \fIcodenames\fP " ]" +Same like +.BR update , +but will show what it will change instead of actually changing it. +.TP +.BR dumpupdate " [ " \fIcodenames\fP " ]" +Same like +.BR checkupdate , +but less suitable for humans and more suitable for computers. +.TP +.BR predelete " [ " \fIcodenames\fP " ]" +This will determine which packages a \fBupdate\fP would delete or +replace and remove those packages. +This can be useful for reducing space needed while upgrading, but +there will be some time where packages are vanished from the +lists so clients will mark them as obsolete. +Plus if you cannot +download a updated package in the (hopefully) following update +run, you will end up with no package at all instead of an old one. +This will also blow up \fB.diff\fP files if you are using the pdiff +example or something similar. +So be careful when using this option or better get some more space so +that update works. +.TP +.B cleanlists +Delete all files in \fIlistdir\fP (default \fIbasedir\fP\fB/lists\fP) that do not +belong to any update rule for any distribution. +I.e. all files are deleted in that directory that no \fBupdate\fP +command in the current configuration can use. +(The files are usually left there, so if they are needed again they +do not need to be downloaded again. Though in many easy cases not +even those files will be needed.) +.TP +.BR pull " [ " \fIcodenames\fP " ]" +pull in newer packages into the specified distributions (all if none given) +from other distributions in the same repository. +See the description of +.B conf/pulls +below. +.TP +.BR checkpull " [ " \fIcodenames\fP " ]" +Same like +.BR pull , +but will show what it will change instead of actually changing it. +.TP +.BR dumppull " [ " \fIcodenames\fP " ]" +Same like +.BR checkpull , +but less suitable for humans and more suitable for computers. +.TP +.B includedeb \fIcodename\fP \fI.deb-filename\fP +Include the given binary Debian package (.deb) in the specified +distribution, applying override information and guessing all +values not given and guessable. +.TP +.B includeudeb \fIcodename\fP \fI.udeb-filename\fP +Same like \fBincludedeb\fP, but for .udeb files. +.TP +.B includedsc \fIcodename\fP \fI.dsc-filename\fP +Include the given Debian source package (.dsc, including other files +like .orig.tar.gz, .tar.gz and/or .diff.gz) in the specified +distribution, applying override information and guessing all values +not given and guessable. + +Note that .dsc files do not contain section or priority, but the +Sources.gz file needs them. +reprepro tries to parse .diff and .tar files for +it, but is only able to resolve easy cases. +If reprepro fails to extract those automatically, +you have to either specify a DscOverride or give them via +.B \-S +and +.B \-P +.TP +.B include \fIcodename\fP \fI.changes-filename\fP +Include in the specified distribution all packages found and suitable +in the \fI.changes\fP file, applying override information guessing all +values not given and guessable. +.TP +.B processincoming \fIrulesetname\fP \fR[\fP\fI.changes-file\fP\fR]\fP +Scan an incoming directory and process the .changes files found there. +If a filename is supplied, processing is limited to that file. +.I rulesetname +identifies which rule-set in +.B conf/incoming +determines which incoming directory to use +and in what distributions to allow packages into. +See the section about this file for more information. +.TP +.BR check " [ " \fIcodenames\fP " ]" +Check if all packages in the specified distributions have all files +needed properly registered. +.TP +.BR checkpool " [ " fast " ]" +Check if all files believed to be in the pool are actually still there and +have the known md5sum. When +.B fast +is specified md5sum is not checked. +.TP +.BR collectnewchecksums +Calculate all supported checksums for all files in the pool. +(Versions prior to 3.3 did only store md5sums, 3.3 added sha1, 3.5 added sha256). +.TP +.BR translatelegacychecksums +Remove the legacy \fBfiles.db\fP file after making sure all information +is also found in the new \fBchecksums.db\fP file. +(Alternatively you can call \fBcollecnewchecksums\fP and remove the file +on your own.) +.TP +.B rereference +Forget which files are needed and recollect this information. +.TP +.B dumpreferences +Print out which files are marked to be needed by whom. +.TP +.B dumpunreferenced +Print a list of all filed believed to be in the pool, that are +not known to be needed. +.TP +.B deleteunreferenced +Remove all known files (and forget them) in the pool not marked to be +needed by anything. +.TP +.BR deleteifunreferenced " [ " \fIfilekeys\fP " ]" +Remove the given files (and forget them) in the pool if they +are not marked to be used by anything. +If no command line arguments are given, +stdin is read and every line treated as one filekey. +This is mostly useful together with \fB\-\-keepunreferenced\fP +in \fBconf/options\fP or in situations where one does not want +to run \fBdeleteunreferenced\fP, which removes all files eligible +to be deleted with this command. +.TP +.BR reoverride " [ " \fIcodenames\fP " ]" +Reapply the override files to the given distributions (Or only parts +thereof given by \fB\-A\fP,\fB\-C\fP or \fB\-T\fP). + +Note: only the control information is changed. Changing a section +to a value, that would cause another component to be guessed, will +not cause any warning. +.TP +.BR redochecksums " [ " \fIcodenames\fP " ]" +Re-add the information about file checksums to the package indices. + +Usually the package's control information is created at inclusion +time or imported from some remote source and not changed later. +This command modifies it to re-add missing checksum types. + +Only checksums already known are used. +To update known checksums about files run \fBcollectnewchecksums\fP first. + +.TP +.BR dumptracks " [ " \fIcodenames\fP " ]" +Print out all information about tracked source packages in the +given distributions. +.TP +.BR retrack " [ " \fIcodenames\fP " ]" +Recreate a tracking database for the specified distributions. +This contains ouf of three steps. +First all files marked as part of a source package are set to +unused. +Then all files actually used are marked as thus. +Finally tidytracks is called remove everything no longer needed +with the new information about used files. + +(This behaviour, though a bit longsome, keeps even files only +kept because of tracking mode \fBkeep\fP and files not otherwise +used but kept due to \fBincludechanges\fP or its relatives. +Before version 3.0.0 such files were lost by running retrack). +.TP +.BR removealltracks " [ " \fIcodenames\fP " ]" +Removes all source package tracking information for the +given distributions. +.TP +.B removetrack " " \fIcodename\fP " " \fIsourcename\fP " " \fIversion\fP +Remove the trackingdata of the given version of a given sourcepackage +from a given distribution. This also removes the references for all +used files. +.TP +.BR tidytracks " [ " \fIcodenames\fP " ]" +Check all source package tracking information for the given distributions +for files no longer to keep. +.TP +.B copy \fIdestination-codename\fP \fIsource-codename\fP \fIpackage\fP\fR[\fP=\fIversion\fP\fR]\fP \fI...\fP +Copy the given packages from one distribution to another. +The packages are copied verbatim, no override files are consulted. +Only components and architectures present in the source distribution are +copied. Package versions must be specified by appending '\fB=\fP' and the +version to the name (without spaces). When no version is specified, the latest +package version is copied. +.TP +.B copysrc \fIdestination-codename\fP \fIsource-codename\fP \fIsource-package\fP \fR[\fP\fIversions\fP\fR]\fP +look at each package +(where package means, as usual, every package be it dsc, deb or udeb) +in the distribution specified by \fIsource-codename\fP +and identifies the relevant source package for each. +All packages matching the specified \fIsource-package\fP name +(and any \fIversion\fP if specified) +are copied to the \fIdestination-codename\fP distribution. +The packages are copied verbatim, no override files are consulted. +Only components and architectures present in the source distribution are +copied. +.TP +.B copymatched \fIdestination-codename\fP \fIsource-codename\fP \fIglob\fP +Copy packages matching the given glob (see \fBlistmatched\fP). + +The packages are copied verbatim, no override files are consulted. +Only components and architectures present in the source distribution are +copied. +.TP +.B copyfilter \fIdestination-codename\fP \fIsource-codename\fP \fIformula\fP +Copy packages matching the given formula (see \fBlistfilter\fP). +(all versions if no version is specified). +The packages are copied verbatim, no override files are consulted. +Only components and architectures present in the source distribution are +copied. +.TP +.B move \fIdestination-codename\fP \fIsource-codename\fP \fIpackage\fP\fR[\fP=\fIversion\fP\fR]\fP \fI...\fP +Move the given packages from one distribution to another. +The packages are moved verbatim, no override files are consulted. +Only components and architectures present in the source distribution are +moved. Package versions must be specified by appending '\fB=\fP' and the +version to the name (without spaces). When no version is specified, the latest +package version is moved. +.TP +.B movesrc \fIdestination-codename\fP \fIsource-codename\fP \fIsource-package\fP \fR[\fP\fIversions\fP\fR]\fP +look at each package +(where package means, as usual, every package be it dsc, deb or udeb) +in the distribution specified by \fIsource-codename\fP +and identifies the relevant source package for each. +All packages matching the specified \fIsource-package\fP name +(and any \fIversion\fP if specified) +are moved to the \fIdestination-codename\fP distribution. +The packages are moved verbatim, no override files are consulted. +Only components and architectures present in the source distribution are +moved. +.TP +.B movematched \fIdestination-codename\fP \fIsource-codename\fP \fIglob\fP +Move packages matching the given glob (see \fBlistmatched\fP). + +The packages are moved verbatim, no override files are consulted. +Only components and architectures present in the source distribution are +moved. +.TP +.B movefilter \fIdestination-codename\fP \fIsource-codename\fP \fIformula\fP +Move packages matching the given formula (see \fBlistfilter\fP). +(all versions if no version is specified). +The packages are moved verbatim, no override files are consulted. +Only components and architectures present in the source distribution are +moved. +.TP +.B restore \fIcodename\fP \fIsnapshot\fP \fIpackages...\fP +.TP +.B restoresrc \fIcodename\fP \fIsnapshot\fP \fIsource-epackage\fP \fR[\fP\fIversions\fP\fR]\fP +.TP +.B restorefilter \fIdestination-codename\fP \fIsnapshot\fP \fIformula\fP +.TP +.B restorematched \fIdestination-codename\fP \fIsnapshot\fP \fIglob\fP +Like the copy commands, but do not copy from another distribution, +but from a snapshot generated with \fBgensnapshot\fP. +Note that this blindly trusts the contents of the files in your \fBdists/\fP +directory and does no checking. +.TP +.B clearvanished +Remove all package databases that no longer appear in \fBconf/distributions\fP. +If \fB\-\-delete\fP is specified, it will not stop if there are still +packages left. +Even without \fB\-\-delete\fP it will unreference +files still marked as needed by this target. +(Use \fB\-\-keepunreferenced\fP to not delete them if that was the last +reference.) + +Do not forget to remove all exported package indices manually. +.TP +.B gensnapshot " " \fIcodename\fP " " \fIdirectoryname\fP +Generate a snapshot of the distribution specified by \fIcodename\fP +in the directory \fIdists\fB/\fIcodename\fB/snapshots/\fIdirectoryname\fB/\fR +and reference all needed files in the pool as needed by that. +No Content files are generated and no export hooks are run. + +Note that there is currently no automated way to remove that snapshot +again (not even clearvanished will unlock the referenced files after the +distribution itself vanished). +You will have to remove the directory yourself and tell reprepro +to \fBunreferencesnapshot \fP\fIcodename\fP\fB \fP\fIdirectoryname\fP before +\fBdeleteunreferenced\fP will delete the files from the pool locked by this. + +To access such a snapshot with apt, add something like the following to +your sources.list file: +.br +\fBdeb method://as/without/snapshot \fIcodename\fB/snapshots/\fIname\fB main\fR +.TP +.B unreferencesnapshot " " \fIcodename\fP " " \fIdirectoryname\fP +Remove all references generated by an \fBgenshapshot\fP with the +same arguments. +This allows the next \fBdeleteunferenced\fP call to delete those files. +(The indices in \fBdists/\fP for the snapshot are not removed.) +.TP +.BR rerunnotifiers " [ " \fIcodenames\fP " ]" +Run all external scripts specified in the \fBLog:\fP options of the +specified distributions. +.TP +.B build\-needing \fIcodename\fP \fIarchitecture\fP \fR[\fP \fIglob\fP \fR]\fP +List source packages (matching \fIglob\fP) that likely need a build on the +given architecture. + +List all source package in the given distribution without a binary package +of the given architecture built from that version of the source, +without a \fB.changes\fP or \fB.log\fP file for the given architecture, +with an Architecture field including \fBany\fP, \fIos\fP\fB-any\fP (with +\fIos\fP being the part before the hyphen in the architecture or \fBlinux\fP +if there is no hyphen) or the architecture and +at least one package in the Binary field not yet available. + +If instead of \fIarchitecture\fP the term \fBany\fP is used, +all architectures are iterated and the architecture is printed as +fourth field in every line. + +If the \fIarchitecture\fP is \fBall\fP, then only source packages +with an Architecture field including \fBall\fP are considered +(i.e. as above with real architectures but \fBany\fP does not suffice). +Note that dpkg\-dev << 1.16.1 does not both set \fBany\fP and \fBall\fP +so source packages building both architecture dependent and independent +packages will never show up unless built with a new enough dpkg\-source). + +.TP +.B translatefilelists +Translate the file list cache within +.IB db /contents.cache.db +into the new format used since reprepro 3.0.0. + +Make sure you have at least half of the space of the current +.IB db /contents.cache.db +file size available in that partition. +.TP +.B flood \fIdistribution\fP \fR[\fP\fIarchitecture\fP\fR]\fP +For each architecture of \fIdistribution\fP (or for the one specified) +add architecture \fBall\fP packages from other architectures +(but the same component or packagetype) under the following conditions: + + Packages are only upgraded, never downgraded. + If there is a package not being architecture \fPall\fP, +then architecture \fBall\fP packages of the same source from the same +source version are preferred over those that have no such binary sibling. + Otherwise the package with the highest version wins. + +You can restrict with architectures are looked for architecture \fPall\fP +packages using \fB\-A\fP and which components/packagetypes are flooded by +\fB\-C\fP/\fB\-T\fP as usual. + +There are mostly two use cases for this command: +If you added an new architecture to an distribution and want to copy all +architecture \fBall\fP packages to it. +Or if you included some architecture all packages only to some architectures +using \fB\-A\fP to avoid breaking the other architectures for which the binary +packages were still missing and now want to copy it to those architectures were +they are unlikely to break something (because a newbinary is already available). +.TP +.B unusedsources \fR[\fP\fIdistributions\fP\fR]\fP +List all source packages for which no binary package build from them is found. +.TP +.B sourcemissing \fR[\fP\fIdistributions\fP\fR]\fP +List all binary packages for which no source package is found +(the source package must be in the same distribution, +but source packages only kept by package tracking is enough). +.TP +.B reportcruft \fR[\fP\fIdistributions\fP\fR]\fP +List all source package versions that either have a source package +and no longer a binary package or binary packages left without +source package in the index. (Unless sourcemissing also list packages +where the source package in only in the pool due to enabled tracking +but no longer in the index). +.TP +.BR sizes " [ " \fIcodenames\fP " ]" +List the size of all packages in the distributions specified or +in all distributions. + +Each row contains 4 numbers, each being a number of bytes in a set +of packages, which are: +The packages in this distribution +(including anything only kept because of tracking), +the packages only in this distribution +(anything in this distribution and a snapshot of this distribution +counts as only in this distribution), +the packages in this distribution and its snapshots, +the packages only in this distribution or its snapshots. + +If more than one distribution is selected, also list a sum of those +(in which 'Only' means only in selected ones, and not only only in +one of the selected ones). + +.TP +.BR repairdescriptions " [ " \fIcodenames\fP " ]" +Look for binary packages only having a short description +and try to get the long description from the .deb file +(and also remove a possible Description-md5 in this case). +.SS internal commands +These are hopefully never needed, but allow manual intervention. +.B WARNING: +Is is quite easy to get into an inconsistent and/or unfixable state. +.TP +.BR _detect " [ " \fIfilekeys\fP " ]" +Look for the files, which \fIfilekey\fP +is given as argument or as a line of the input +(when run without arguments), and calculate +their md5sum and add them to the list of known files. +(Warning: this is a low level operation, no input validation +or normalization is done.) +.TP +.BR _forget " [ " \fIfilekeys\fP " ]" +Like +.B _detect +but remove the given \fIfilekey\fP from the list of known +files. +(Warning: this is a low level operation, no input validation +or normalization is done.) +.TP +.B _listmd5sums +Print a list of all known files and their md5sums. +.TP +.B _listchecksums +Print a list of all known files and their recorded checksums. +.TP +.B _addmd5sums +alias for the newer +.TP +.B _addchecksums +Add information of known files (without any check done) +in the strict format of _listchecksums output (i.e. don't dare to +use a single space anywhere more than needed). +.TP +.BI _dumpcontents " identifier" +Printout all the stored information of the specified +part of the repository. (Or in other words, the content +the corresponding Packages or Sources file would get) + +This command is deprecated and will be removed in a future version. +.TP +.BI "_addreference " filekey " " identifier +Manually mark \fIfilekey\fP to be needed by \fIidentifier\fP +.TP +.BI "_addreferences " identifier " \fR[\fR " filekeys " \fR]\fR" +Manually mark one or more \fIfilekeys\fP to be needed by \fIidentifier\fP. +If no command line arguments are given, +stdin is read and every line treated as one filekey. +.TP +.BI "_removereference " identifier " " filekey +Manually remove the given mark that the file is needed by this identifier. +.TP +.BI "_removereferences " identifier +Remove all references what is needed by +.I identifier. +.TP +.BI __extractcontrol " .deb-filename" +Look what reprepro believes to be the content of the +.B control +file of the specified .deb-file. +.TP +.BI __extractfilelist " .deb-filename" +Look what reprepro believes to be the list of files +of the specified .deb-file. +.TP +.BI _fakeemptyfilelist " filekey" +Insert an empty filelist for \fIfilekey\fP. This is a evil +hack around broken .deb files that cannot be read by reprepro. +.TP +.B _addpackage \fIcodenam\fP \fIfilename\fP \fIpackages...\fP +Add packages from the specified filename to part specified +by \fB\-C\fP \fB\-A\fP and \fB\-T\fP of the specified distribution. +Very strange things can happen if you use it improperly. +.TP +.B __dumpuncompressors +List what compressions format can be uncompressed and how. +.TP +.BI __uncompress " format compressed-file uncompressed-file" +Use builtin or external uncompression to uncompress the specified +file of the specified format into the specified target. +.TP +.BR _listcodenames +Print - on per line - the codenames of all configured distributions. +.TP +.B _listconfidentifiers \fIidentifier\fP \fR[\fP \fIdistributions...\fP \fR]\fP +Print - one per line - all identifiers of subdatabases as derived from the +configuration. +If a list of distributions is given, only identifiers of those are printed. + +.TP +.B _listdbidentifiers \fIidentifier\fP \fR[\fP \fIdistributions...\fP \fR]\fP +Print - one per line - all identifiers of subdatabases in the current +database. +This will be a subset of the ones printed by \fB_listconfidentifiers\fP or +most commands but \fBclearvanished\fP will refuse to run, and depending +on the database compatibility version, will include all those if reprepro +was run since the config was last changed. + +.SH "CONFIG FILES" +.B reprepo +uses three config files, which are searched in +the directory specified with +.B \-\-confdir +or in the +.B conf/ +subdirectory of the \fIbasedir\fP. + +If a file +.B options +exists, it is parsed line by line. +Each line can be the long +name of a command line option (without the \-\-) +plus an argument, where possible. +Those are handled as if they were command line options given before +(and thus lower priority than) any other command line option. +(and also lower priority than any environment variable). + +To allow command line options to override options file options, +most boolean options also have a corresponding form starting with \fB\-\-no\fP. + +(The only exception is when the path to look for config files +changes, the options file will only opened once and of course +before any options within the options file are parsed.) + +The file +.B distributions +is always needed and describes what distributions +to manage, while +.B updates +is only needed when syncing with external repositories and +.B pulls +is only needed when syncing with repositories in the same reprepro database. + +The last three are in the format control files in Debian are in, +i.e. paragraphs separated by empty lines consisting of +fields. Each field consists of a fieldname, followed +by a colon, possible whitespace and the data. A field +ends with a newline not followed by a space or tab. + +Lines starting with # as first character are ignored, +while in other lines the # character and +everything after it till the newline character are ignored. + +A paragraph can also consist of only a single field +.RB \(dq !include: \(dq +which causes the named file (relative to confdir unless starting +with +.BR ~/ ", " +b/ ", " +c/ " or " / " )" +to be read as if it was found at this place. + +Each of the three files or a file included as described above +can also be a directory, in which case all files it contains +with a filename ending in +.B .conf +and not starting with +.B . +are read. +.SS conf/distributions +.TP +.B Codename +This required field is the unique identifier of a distribution +and used as directory name within +.B dists/ +It is also copied into the Release files. + +Note that this name is not supposed to change. +You most likely \fBnever ever\fP want a name like \fBtesting\fP +or \fBstable\fP here (those are suite names and supposed to point +to another distribution later). +.TP +.B Suite +This optional field is simply copied into the +Release files. In Debian it contains names like +stable, testing or unstable. To create symlinks +from the Suite to the Codename, use the +\fBcreatesymlinks\fP command of reprepro. +.TP +.B FakeComponentPrefix +If this field is present, +its argument is added - separated by a slash - before every +Component written to the main Release file +(unless the component already starts with it), +and removed from the end of the Codename and Suite fields in that file. +Also if a component starts with it, its directory in the dists dir +is shortened by this. +.br +So \fB + + Codename: bla/updates + Suite: foo/updates + FakeComponentPrefix: updates + Components: main bad\fP + +will create a Release file with \fB + + Codename: bla + Suite: foo + Components: updates/main updates/bad\fP + +in it, but otherwise nothing is changed, while\fB + + Codename: bla/updates + Suite: foo/updates + FakeComponentPrefix: updates + Components: updates/main updates/bad\fP + +will also create a Release file with \fB + + Codename: bla + Suite: foo + Components: updates/main updates/bad\fP + +but the packages will actually be in the components +\fBupdates/main\fP and \fBupdates/bad\fP, +most likely causing the same file using duplicate storage space. + +This makes the distribution look more like Debian's security archive, +thus work around problems with apt's workarounds for that. +.TP +.B AlsoAcceptFor +A list of distribution names. +When a \fB.changes\fP file is told to be included +into this distribution with the \fBinclude\fP command +and the distribution header of that file is neither +the codename, nor the suite name, nor any name from the +list, a \fBwrongdistribution\fP error is generated. +The \fBprocess_incoming\fP command will also use this field, +see the description of \fBAllow\fP and \fBDefault\fP +from the \fBconf/incoming\fP file for more information. +.TP +.B Version +This optional field is simply copied into the +Release files. +.TP +.B Origin +This optional field is simply copied into the +Release files. +.TP +.B Label +This optional field is simply copied into the +Release files. +.TP +.B NotAutomatic +This optional field is simply copied into the +Release files. +(The value is handled as an arbitrary string, +though anything but \fByes\fP does not make much +sense right now.) +.TP +.B ButAutomaticUpgrades +This optional field is simply copied into the +Release files. +(The value is handled as an arbitrary string, +though anything but \fByes\fP does not make much +sense right now.) +.TP +.B Description +This optional field is simply copied into the +Release files. +.TP +.B Architectures +This required field lists the binary architectures within +this distribution and if it contains +.B source +(i.e. if there is an item +.B source +in this line this Distribution has source. All other items +specify things to be put after "binary\-" to form directory names +and be checked against "Architecture:" fields.) + +This will also be copied into the Release files. (With exception +of the +.B source +item, which will not occur in the topmost Release file whether +it is present here or not) +.TP +.B Components +This required field lists the component of a +distribution. See +.B GUESSING +for rules which component packages are included into +by default. This will also be copied into the Release files. +.TP +.B DDebComponents +List of components containing .ddebs. +.TP +.B UDebComponents +Components with a debian\-installer subhierarchy containing .udebs. +(E.g. simply "main") +.TP +.B Update +When this field is present, it describes which update rules are used +for this distribution. There also can be a magic rule minus ("\-"), +see below. +.TP +.B Pull +When this field is present, it describes which pull rules are used +for this distribution. +Pull rules are like Update rules, +but get their stuff from other distributions and not from external sources. +See the description for \fBconf/pulls\fP. +.TP +.B SignWith +When this field is present, a Release.gpg file will be generated. +If the value is "yes" or "default", the default key of gpg is used. +If the field starts with an exclamation mark ("!"), the given script +is executed to do the signing. +Otherwise the value will be given to libgpgme to determine to key to +use. + +If there are problems with signing, you can try +.br +.B gpg \-\-list\-secret\-keys \fIvalue\fP +.br +to see how gpg could interpret the value. +If that command does not list any keys or multiple ones, +try to find some other value (like the keyid), +that gpg can more easily associate with a unique key. + +If this key has a passphrase, you need to use gpg\-agent +or the insecure option \fB\-\-ask\-passphrase\fP. + +A '\fB!\fP' hook script is looked for in the confdir, +unless it starts with +.BR ~/ ", " ./ ", " +b/ ", " +o/ ", " +c/ " or " / " ." +Is gets three command line arguments: The filename to sign, +an empty argument or the filename to create with an inline +signature (i.e. InRelease) and +an empty argument or the filename to create an detached signature +(i.e. Release.gpg). +The script may generate no Release.gpg file if it choses to +(then the repository will look like unsigned for older clients), +but generating empty files is not allowed. +Reprepro waits for the script to finish and will abort the exporting +of the distribution this signing is part of unless the scripts +returns normally with exit code 0. +Using a space after ! is recommended to avoid incompatibilities +with possible future extensions. +.TP +.B DebOverride +When this field is present, it describes the override file used +when including .deb files. +.TP +.B UDebOverride +When this field is present, it describes the override file used +when including .udeb files. +.TP +.B DscOverride +When this field is present, it describes the override file used +when including .dsc files. +.TP +.B DebIndices\fR, \fBUDebIndices\fR, \fBDscIndices +Choose what kind of Index files to export. The first +part describes what the Index file shall be called. +The second argument determines the name of a Release +file to generate or not to generate if missing. +Then at least one of "\fB.\fP", "\fB.gz\fP", "\fB.xz\fP" or "\fB.bz2\fP" +specifying whether to generate uncompressed output, gzipped +output, bzip2ed output or any combination. +(bzip2 is only available when compiled with bzip2 support, +so it might not be available when you compiled it on your +own, same for xz and liblzma). +If an argument not starting with dot follows, +it will be executed after all index files are generated. +(See the examples for what argument this gets). +The default is: +.br +DebIndices: Packages Release . .gz +.br +UDebIndices: Packages . .gz +.br +DscIndices: Sources Release .gz +.TP +.B ExportOptions +Options to modify how and if exporting is done: +.br +.B noexport +Never export this distribution. +That means there will be no directory below \fBdists/\fP generated and the distribution is only useful to copy packages to other distributions. +.br +.B keepunknown +Ignore unknown files and directories in the exported directory. +This is currently the only available option and the default, but might change in the future, so it can already be requested explicitly. +.TP +.B Contents +Enable the creation of Contents files listing all the files +within the binary packages of a distribution. +(Which is quite slow, you have been warned). + +In earlier versions, the first argument was a rate at which +to extract file lists. +As this did not work and was no longer easily possible after +some factorisation, this is no longer supported. + +The arguments of this field is a space separated list of options. +If there is a \fBudebs\fP keyword, \fB.udeb\fPs are also listed +(in a file called \fBuContents\-\fP\fIarchitecture\fP.) +If there is a \fBnodebs\fP keyword, \fB.deb\fPs are not listed. +(Only useful together with \fBudebs\fP) +If there is at least one of the keywords +\fB.\fP, \fB.gz\fP, \fB\.xz\fP and/or \fB.bz2\fP, +the Contents files are written uncompressed, gzipped and/or bzip2ed instead +of only gzipped. + +If there is a \fBpercomponent\fP then one Contents\-\fIarch\fP file +per component is created. +If there is a \fBallcomponents\fP then one global Contents\-\fIarch\fP +file is generated. +If both are given, both are created. +If none of both is specified then \fBpercomponent\fP is taken +as default (earlier versions had other defaults). + +The switches \fBcompatsymlink\fP or \fBnocompatsymlink\fP +(only possible if \fBallcomponents\fP was not specified explicitly) +control whether a compatibility symlink is created so old versions +of apt\-file looking for the component independent filenames at +least see the contents of the first component. + +Unless \fBallcomponents\fP is given, \fBcompatsymlinks\fP +currently is the default, but that will change +in some future (current estimate: after wheezy was released) + +.TP +.B ContentsArchitectures +Limit generation of Contents files to the architectures given. +If this field is not there, all architectures are processed. +An empty field means no architectures are processed, thus not +very useful. +.TP +.B ContentsComponents +Limit what components are processed for the \fBContents\-\fP\fIarch\fP +files to the components given. +If this field is not there, all components are processed. +An empty field is equivalent to specify \fBnodebs\fP in the +\fBContents\fP field, while a non-empty field overrides a +\fBnodebs\fP there. +.TP +.B ContentsUComponents +Limit what components are processed for the uContents files to +the components given. +If this field is not there and there is the \fBudebs\fP keyword +in the Contents field, all .udebs of all components are put +in the \fBuContents.\fP\fIarch\fP files. +If this field is not there and there is no \fBudebs\fP keyword +in the Contents field, no \fBuContents\-\fP\fIarch\fP files are +generated at all. +A non-empty fields implies generation of \fBuContents\-\fP\fIarch\fP +files (just like the \fBudebs\fP keyword in the Contents field), +while an empty one causes no \fBuContents\-\fP\fIarch\fP files to +be generated. +.TP +.B Uploaders +Specifies a file (relative to confdir if not starting with +.BR ~/ ", " +b/ ", " +c/ " or " / " )" +to specify who is allowed to upload packages. Without this there are no +limits, and this file can be ignored via \fB\-\-ignore=uploaders\fP. +See the section \fBUPLOADERS FILES\fP below. +.TP +.B Tracking +Enable the (experimental) tracking of source packages. +The argument list needs to contain exactly one of the following: +.br +.B keep +Keeps all files of a given source package, until that +is deleted explicitly via \fBremovetrack\fP. This is +currently the only possibility to keep older packages +around when all indices contain newer files. +.br +.B all +Keep all files belonging to a given source package until +the last file of it is no longer used within that +distribution. +.br +.B minimal +Remove files no longer included in the tracked distribution. +(Remove changes, logs and includebyhand files once no file is +in any part of the distribution). +.br +And any number of the following (or none): +.br +.B includechanges +Add the .changes file to the tracked files of a source package. +Thus it is also put into the pool. +.br +.B includebyhand +Add \fBbyhand\fP and \fBraw\-\fP\fI*\fP files to the tracked +files and thus in the pool. +.br +.B includebuildinfos +Add buildinfo files to the tracked files and thus in the pool. +.br +.B includelogs +Add log files to the tracked files and thus in the pool. +(Not that putting log files in changes files is a reprepro +extension not found in normal changes files) +.br +.B embargoalls +Not yet implemented. +.br +.B keepsources +Even when using minimal mode, do not remove source files +until no file is needed any more. +.br +.B needsources +Not yet implemented. +.TP +.B Limit +Limit the number of versions of a package per distribution, architecture, +component, and type. The limit must be a number. If the number is positive, +all old package version that exceed these limit will be removed or archived +(see +.B Archive +option), when a new package version is added. If the number is zero or negative, +all package version will be kept. By default only one package version will be +kept. +.TP +.B Archive +Specify a codename which must be declared before (to avoid loops). When packages +exceed the version count limit (specified in \fBLimit\fR), these packages will +be moved to the specified distribution instead of being removed. +.TP +.B Log +Specify a file to log additions and removals of this distribution +into and/or external scripts to call when something is added or +removed. +The rest of the \fBLog:\fP line is the filename, +every following line (as usual, have to begin with a single space) +the name of a script to call. +The name of the script may be preceded with options of the +form \fB\-\-type=\fP(\fBdsc\fP|\fBdeb\fP|\fBudeb\fP), +\fB\-\-architecture=\fP\fIname\fP or +\fB\-\-component=\fP\fIname\fP to only call the script for some +parts of the distribution. +An script with argument \fB\-\-changes\fP is called when a \fB.changes\fP +file was accepted by \fBinclude\fP or \fBprocessincoming\fP (and with other +arguments). +Both type of scripts can have a \fB\-\-via=\fP\fIcommand\fP specified, +in which case it is only called when caused by reprepro command \fIcommand\fP. + +For information how it is called and some examples take a look +at manual.html in reprepro's source or +.B /usr/share/doc/reprepro/ + +If the filename for the log files does not start with a slash, +it is relative to the directory specified with \fB\-\-logdir\fP, +the scripts are relative to \fB\-\-confdir\fP unless starting with +.BR ~/ ", " +b/ ", " +c/ " or " / . +.TP +.B ValidFor +If this field exists, an Valid\-Until field is put into generated +.B Release +files for this distribution with an date as much in the future as the +argument specifies. + +The argument has to be an number followed by one of the units +.BR d ", " m " or " y , +where \fBd\fP means days, \fBm\fP means 31 days and \fBy\fP means +365 days. +So +.B ValidFor: 1m 11 d +causes the generation of a +.B Valid\-Until: +header in Release files that points 42 days into the future. +.TP +.B ReadOnly +Disallow all modifications of this distribution or its directory +in \fBdists/\fP\fIcodename\fP (with the exception of snapshot subdirectories). +.TP +.B ByHandHooks +This species hooks to call for handling byhand/raw files by processincoming +(and in future versions perhaps by include). + +Each line consists out of 4 arguments: +A glob pattern for the section +(classically \fBbyhand\fP, though Ubuntu uses \fBraw\-\fP*), +a glob pattern for the priority (not usually used), +and a glob pattern for the filename. + +The 4th argument is the script to be called when all of the above match. +It gets 5 arguments: the codename of the distribution, +the section (usually \fBbyhand\fP), +the priority (usually only \fB\-\fP), +the filename in the changes file and +the full filename (with processincoming in the secure TempDir). +.TP +.B Signed\-By +This optional field is simply copied into the Release files. +It is used to tell apt which keys to trust for this Release +in the future. +(see SignWith for how to tell reprepro whether and how to sign). +.SS conf/updates +.TP +.B Name +The name of this update\-upstream as it can be used in the +.B Update +field in conf/distributions. +.TP +.B Method +An URI as one could also give it apt, e.g. +.I http://ftp.debian.de/debian +which is simply given to the corresponding +.B apt\-get +method. (So either +.B apt\-get has to be installed, or you have to point with +.B \-\-methoddir +to a place where such methods are found. +.TP +.B Fallback +(Still experimental:) A fallback URI, where all files are +tried that failed the first one. They are given to the +same method as the previous URI (e.g. both http://), and +the fallback-server must have everything at the same place. +No recalculation is done, but single files are just retried from +this location. +.TP +.B Config +This can contain any number of lines, each in the format +.B apt\-get \-\-option +would expect. (Multiple lines \(hy as always \(hy marked with +leading spaces). +.P +For example: Config: Acquire::Http::Proxy=http://proxy.yours.org:8080 +.TP +.B From +The name of another update rule this rules derives from. +The rule containing the \fBFrom\fP may not contain +.BR Method ", " Fallback " or " Config "." +All other fields are used from the rule referenced in \fBFrom\fP, unless +found in this containing the \fBFrom\fP. +The rule referenced in \fBFrom\fP may itself contain a \fBFrom\fP. +Reprepro will only assume two remote index files are the same, +if both get their \fBMethod\fP information from the same rule. +.TP +.B Suite +The suite to update from. If this is not present, the codename +of the distribution using this one is used. Also "*/whatever" +is replaced by "<codename>/whatever" +.TP +.B Components +The components to update. Each item can be either the name +of a component or a pair of a upstream component and a local +component separated with ">". (e.g. "main>all contrib>all non\-free>notall") + +If this field is not there, all components from the distribution +to update are tried. + +An empty field means no source or .deb packages are updated by this rule, +but only .udeb packages, if there are any. + +A rule might list components not available in all distributions +using this rule. In this case unknown components are silently +ignored. +(Unless you start reprepro with the \fB\-\-fast\fP option, +it will warn about components unusable in all distributions using +that rule. As exceptions, unusable components called \fBnone\fP +are never warned about, for compatibility with versions prior to +3.0.0 where and empty field had a different meaning.) +.TP +.B Architectures +The architectures to update. If omitted all from the distribution +to update from. (As with components, you can use ">" to download +from one architecture and add into another one. (This only determine +in which Package list they land, it neither overwrites the Architecture +line in its description, nor the one in the filename determined from this +one. In other words, it is no really useful without additional filtering)) +.TP +.B UDebComponents +Like +.B Components +but for the udebs. +.TP +.B VerifyRelease +Download the +.B Release.gpg +file and check if it is a signature of the +.B Releasefile +with the key given here. (In the Format as +"gpg \-\-with\-colons \-\-list\-key" prints it, i.e. the last +16 hex digits of the fingerprint) Multiple keys can be specified +by separating them with a "\fB|\fP" sign. Then finding a signature +from one of the will suffice. +To allow revoked or expired keys, add a "\fB!\fP" behind a key. +(but to accept such signatures, the appropriate \fB\-\-ignore\fP +is also needed). +To also allow subkeys of a specified key, add a "\fB+\fP" behind a key. +.TP +.B IgnoreRelease: yes +If this is present, no +.B InRelease +or +.B Release +file will be downloaded and thus the md5sums of the other +index files will not be checked. +.TP +.B GetInRelease: no +IF this is present, no +.B InRelease +file is downloaded but only +.B Release +(and +.B Release.gpg +) +are tried. +.TP +.B Flat +If this field is in an update rule, it is supposed to be a +flat repository, i.e. a repository without a \fBdists\fP +dir and no subdirectories for the index files. +(If the corresponding \fBsources.list\fP line has the suite +end with a slash, then you might need this one.) +The argument for the \fBFlat:\fP field is the Component to +put those packages into. +No \fBComponents\fP or \fBUDebComponents\fP +fields are allowed in a flat update rule. +If the \fBArchitecture\fP field has any \fB>\fP items, +the part left of the "\fB>\fP" is ignored. +.br +For example the \fBsources.list\fP line + deb http://cran.r\-project.org/bin/linux/debian etch\-cran/ +.br +would translate to +.br + Name: R + Method: http://cran.r\-project.org/bin/linux/debian + Suite: etch\-cran + Flat: whatevercomponentyoudlikethepackagesin +.TP +.B IgnoreHashes +This directive tells reprepro to not check the listed +hashes in the downloaded Release file (and only in the Release file). +Possible values are currently \fBmd5\fP, \fBsha1\fP and \fBsha256\fP. + +Note that this does not speed anything +up in any measurable way. The only reason to specify this if +the Release file of the distribution you want to mirror from +uses a faulty algorithm implementation. +Otherwise you will gain nothing and only lose security. +.TP +.B FilterFormula +This can be a formula to specify which packages to accept from +this source. The format is misusing the parser intended for +Dependency lines. To get only architecture all packages use +"architecture (== all)", to get only at least important +packages use "priority (==required) | priority (==important)". + +See the description of the listfilter command for the semantics +of formulas. +.TP +.B FilterList\fR, \fPFilterSrcList +These two options each take at least two arguments: +The first argument is the fallback (default) action. +All following arguments are treated as file names of lists. + +The filenames are considered to be relative to +.B \-\-confdir\fR, +if not starting with +.BR ~/ ", " +b/ ", " +c/ " or " / "." + +Each list file consists of lines with a package name +followed by whitespaced followed by an action. + +Each list may only contain a single line for a given package name. +The action to be taken is the action specified by the first file +mentioning that package. +If no list file mentions a package, the fallback action is used instead. + +This format is inspired by dpkg \-\-get\-selections before multiarch +and the names of the actions likely only make sense if you imagine the +file to be the output of this command of an existing system. + +For each package available in the distribution to be updated from/pulled from +this action is determined and affects the current decision what to do +to the target distribution. +(Only after all update/pull rules for a given target distribution have been +processed something is actually done). + +The possible action keywords are: +.RS +.TP +.B install +mark the available package to be added to the target distribution unless +the same version or a higher version is already marked as to be added/kept. +(Note that without a prior delete rule (\fB\-\fP) or \fBsupersede\fP action, +this will never downgrade a package as the already existing version +is marked to be kept). +.TP +.B upgradeonly +like \fBinstall\fP but will not add new packages to a distribution. +.TP +.B supersede +unless the current package version is higher than the available package version, +mark the package to be deleted in the target distribution. +(Useful to remove packages in add-on distributions once they reached the base distribution). +.TP +.BR deinstall " or " purge +ignore the newly available package. +.TP +.B warning +print a warning message to stderr if a new package/newer version is available. +Otherwise ignore the new package (like with \fBdeinstall\fP or \fBpurge\fP). +.TP +.B hold +the new package is ignored, but every previous decision to +downgrade or delete the package in the target distribution is reset. +.TP +.B error +abort the whole upgrade/pull if a new package/newer version is available +.TP +.B "= \fIversion\fP" +If the candidate package has the given version, behave like \fBinstall\fP. +Otherwise continue as if this list file did not mention this package +(i.e. look in the remaining list files or use the fallback action). +Only one such entry per package is currently supported and the version +is currently compared as string. +.RE +.PP +.RS +If there is both \fBFilterList\fP and \fBFilterSrcList\fP then +the first is used for \fB.deb\fP and \fB.udeb\fP and the second for +\fB.dsc\fP packages. +.PP +If there is only \fBFilterList\fP that is applied to everything. +.PP +If there is only \fBFilterSrcList\fP that is applied to everything, too, +but the source package name (and source version) is used to do the lookup. +.RE +.TP +.B OmitExtraSourceOnly +This field controls whether source packages with Extra-Source-Only +set are ignore when getting source packages. +Without this option or if it is true, those source packages +are ignored, while if set to no or false, those source packages +are also candidates if no other filter excludes them. +(The default of true will likely change once reprepro supports +multiple versions of a package or has other means to keep the +source packages around). +.TP +.B ListHook +If this is given, it is executed for all downloaded index files +with the downloaded list as first and a filename that will +be used instead of this. (e.g. "ListHook: /bin/cp" works +but does nothing.) + +If a file will be read multiple times, it is processed multiple +times, with the environment variables +.BR REPREPRO_FILTER_CODENAME ", " REPREPRO_FILTER_PACKAGETYPE ", " +.BR REPREPRO_FILTER_COMPONENT " and " REPREPRO_FILTER_ARCHITECTURE +set to the where this file will be added and +.B REPREPRO_FILTER_PATTERN +to the name of the update rule causing it. + +.TP +.B ListShellHook +This is like ListHook, but the whole argument is given to the shell +as argument, and the input and output file are stdin and stdout. + +i.e.: +.br +ListShellHook: cat +.br +works but does nothing but useless use of a shell and cat, while +.br +ListShellHook: grep\-dctrl \-X \-S apt \-o \-X \-S dpkg || [ $? \-eq 1 ] +.br +will limit the update rule to packages from the specified source packages. +.TP +.B DownloadListsAs +The arguments of this field specify which index files reprepro +will download. + +Allowed values are +.BR . ", " .gz ", " .bz2 ", " .lzma ", " .xz ", " .diff ", " +.BR force.gz ", " force.bz2 ", " force.lzma ", " +.BR force.xz ", and " force.diff "." + +Reprepro will try the first supported variant in the list given: +Only compressions compiled in or for which an uncompressor was found +are used. +Unless the value starts with \fBforce.\fP, +it is only tried if is found in the Release or InRelease file. + +The default value is \fB.diff .xz .lzma .bz2 .gz .\fP, i.e. +download Packages.diff if listed in the Release file, +otherwise or if not usable download .xz if +listed in the Release file and there is a way to uncompress it, +then .lzma if usable, +then .bz2 if usable, +then .gz and then uncompressed). + +Note there is no way to see if an uncompressed variant +of the file is available (as the Release file always lists their +checksums, even if not there), +so putting '\fB.\fP' anywhere but as the last argument can mean +trying to download a file that does not exist. + +Together with \fBIgnoreRelease\fP reprepro will download the first +in this list that could be unpacked (i.e. \fBforce\fP is always assumed) +and the default value is \fB.gz .bzip2 . .lzma .xz\fP. +.SS conf/pulls +This file contains the rules for pulling packages from one +distribution to another. +While this can also be done with update rules using the file +or copy method and using the exported indices of that other +distribution, this way is faster. +It also ensures the current files are used and no copies +are made. +(This also leads to the limitation that pulling from one +component to another is not possible.) + +Each rule consists out of the following fields: +.TP +.B Name +The name of this pull rule as it can be used in the +.B Pull +field in conf/distributions. +.TP +.B From +The codename of the distribution to pull packages from. +.TP +.B Components +The components of the distribution to get from. + +If this field is not there, +all components from the distribution to update are tried. + +A rule might list components not available in all distributions using this +rule. In this case unknown components are silently ignored. +(Unless you start reprepro with the \-\-fast option, +it will warn about components unusable in all distributions using that rule. +As exception, unusable components called \fBnone\fP are never warned about, +for compatibility with versions prior to 3.0.0 where and empty field had +a different meaning.) +.TP +.B Architectures +The architectures to update. +If omitted all from the distribution to pull from. +As in +.BR conf/updates , +you can use ">" to download +from one architecture and add into another one. (And again, only useful +with filtering to avoid packages not architecture \fBall\fP to migrate). +.TP +.B UDebComponents +Like +.B Components +but for the udebs. +.TP +.B FilterFormula +.TP +.B FilterList +.TP +.B FilterSrcList +The same as with update rules. +.SH "OVERRIDE FILES" +The format of override files used by reprepro +should resemble the extended ftp\-archive format, +to be specific it is: + +.B \fIpackagename\fP \fIfield name\fP \fInew value\fP + +For example: +.br +.B kernel\-image\-2.4.31\-yourorga Section protected/base +.br +.B kernel\-image\-2.4.31\-yourorga Priority standard +.br +.B kernel\-image\-2.4.31\-yourorga Maintainer That's me <me@localhost> +.br +.B reprepro Priority required + +All fields of a given package will be replaced by the new value specified +in the override file +with the exception of special fields starting with a dollar sign ($). +While the field name is compared case-insensitive, it is copied in +exactly the form in the override file there. +(Thus I suggest to keep to the exact case it is normally found in +index files in case some other tool confuses them.) +More than copied is the Section header (unless \fB\-S\fP is supplied), +which is also used to guess the component (unless \fB\-C\fP is there). + +Some values like \fBPackage\fP, \fBFilename\fP, \fBSize\fP or \fBMD5sum\fP +are forbidden, as their usage would severely confuse reprepro. + +As an extension reprepro also supports patterns instead of packagenames. +If the package name contains '*', '[' or '?', +it is considered a pattern +and applied to each package +that is not matched by any non-pattern override nor by any previous pattern. + +Fieldnames starting with a dollar ($) are not be placed in the +exported control data but have special meaning. +Unknown ones are loudly ignored. +Special fields are: + + \fB$Component\fP: includedeb, includedsc, include and processincoming +will put the package in the component given as value +(unless itself overridden with \fB\-C\fP). +Note that the proper way to specify the component is by setting the +section field and using this extension will most likely confuse people +and/or tools. + + \fB$Delete\fP: the value is treated a fieldname and fields of that +name are removed. +(This way one can remove fields previously added without removing and +re-adding the package. +And fields already included in the package can be removed, too). + +.SS conf/incoming +Every chunk is a rule set for the +.B process_incoming +command. +Possible fields are: +.TP +.B Name +The name of the rule-set, used as argument to the scan command to specify +to use this rule. +.TP +.B IncomingDir +The Name of the directory to scan for +.B .changes +files. +.TP +.B TempDir +A directory where the files listed in the processed .changes files +are copied into before they are read. +You can avoid some copy operations by placing this directory +within the same mount point the pool hierarchy +is (at least partially) in. +.TP +.B LogDir +A directory where .changes files, .log files, .buildinfo files +and otherwise unused .byhand files are stored upon procession. +.TP +.B Allow +Each argument is either a pair \fIname1\fB>\fIname2\fR or simply +\fIname\fP which is short for \fIname\fB>\fIname\fR. +Each \fIname2\fP must identify a distribution, +either by being Codename, a unique Suite, or a unique AlsoAcceptFor +from \fBconf/distributions\fP. +Each upload has each item in its +.B Distribution: +header compared first to last with each \fIname1\fP in the rules +and is put in the first one accepting this package. e.g.: +.br +Allow: local unstable>sid +.br +or +.br +Allow: stable>security\-updates stable>proposed\-updates +.br +(Note that this makes only sense if Multiple is set to true or if +there are people only allowed to upload to proposed\-updates but +not to security\-updates). +.TP +.B Default \fIdistribution +Every upload not put into any other distribution because +of an Allow argument is put into \fIdistribution\fP if that +accepts it. +.TP +.B Multiple +Old form of Options: multiple_distributions. +.TP +.B Options +A list of options +.br +.B multiple_distributions +.br +Allow including a upload in multiple distributions. + +If a .changes file lists multiple distributions, +then reprepro will start with the first name given, +check all Accept and Default options till it finds +a distribution this upload can go into. + +If this found no distribution or if this option was given, +reprepro will then do the same with the second distribution name +given in the .changes file and so on. +.br +.B limit_arch_all +.br +If an upload contains binaries from some architecture and architecture +all packages, +the architecture all packages are only put into the architectures within +this upload. +Useful to combine with the \fBflood\fP command. +.TP +.B Permit +A list of options to allow things otherwise causing errors: +.br +.B unused_files +.br +Do not stop with error if there are files listed in the \fB.changes\fP +file if it lists files not belonging to any package in it. +.br +.B older_version +.br +Ignore a package not added because there already is a strictly newer +version available instead of treating this as an error. +.br +.B unlisted_binaries +.br +Do not abort with an error if a .changes file contains .deb files that +are not listed in the Binaries header. +.TP +.B Cleanup \fIoptions +A list of options to cause more files in the incoming directory to be +deleted: +.br +.B unused_files +.br +If there is \fBunused_files\fP in \fBPermit\fP then also delete those +files when the package is deleted after successful processing. +.br +.B unused_buildinfo_files +.br +If .buildinfo files of processed .changes files are not used (neither +stored by LogDir nor with Tracking: includebuildinfos) then delete them +from the incoming dir. +(This option has no additional effect if \fBunused_files\fP is already used.) +.br +.B on_deny +.br +If a \fB.changes\fP file is denied processing because of missing signatures +or allowed distributions to be put in, delete it and all the files it references. +.br +.B on_error +.br +If a \fB.changes\fP file causes errors while processing, delete it and the files +it references. + +Note that allowing cleanup in publicly accessible incoming queues allows a denial +of service by sending in .changes files deleting other peoples files before they +are completed. +Especially when .changes files are handled directly (e.g. by inoticoming). + +.TP +.B MorgueDir +If files are to be deleted by Cleanup, they are instead moved to a subdirectory +of the directory given as value to this field. +This directory has to be on the same partition as the incoming directory and +files are moved (i.e. owner and permission stay the same) and never copied. + +.SH "UPLOADERS FILES" +These files specified by the \fBUploaders\fP header in the distribution +definition as explained above describe what key a \fB.changes\fP file +as to be signed with to be included in that distribution. +.P +Empty lines and lines starting with a hash are ignored, every other line +must be of one of the following nine forms or an include directive: +.TP +.B allow \fIcondition\fP by anybody +which allows everyone to upload packages matching \fIcondition\fP, +.TP +.B allow \fIcondition\fP by unsigned +which allows everything matching that has no pgp/gpg header, +.TP +.B allow \fIcondition\fP by any key +which allows everything matching with any valid signature in or +.TP +.B allow \fIcondition\fP by key \fIkey-id\fP +which allows everything matching signed by this \fIkey-id\fP +(to be specified without any spaces). +If the \fIkey-id\fP ends with a \fB+\fP (plus), a signature with a subkey of +this primary key also suffices. + +\fIkey-id\fP must be a suffix of the id libgpgme uses to identify this key, +i.e. a number of hexdigits from the end of the fingerprint of the key, but +no more than what libgpgme uses. +(The maximal number should be what gpg \-\-list-key \-\-with\-colons prints, +as of the time of this writing that is at most 16 hex-digits). +.TP +.B allow \fIcondition\fP by group \fIgroupname\fP +which allows every member of group \fIgroupname\fP. +Groups can be manipulated by +.TP +.B group \fIgroupname\fP add \fIkey-id\fP +to add a \fIkey-id\fP (see above for details) to this group, or +.TP +.B group \fIgroupname\fP contains \fIgroupname\fP +to add a whole group to a group. + +To avoid warnings in incomplete config files there is also +.TP +.B group \fIgroupname\fP empty +to declare a group has no members (avoids warnings that it is used without those) +and +.TP +.B group \fIgroupname\fP unused +to declare that a group is not yet used (avoid warnings that it is not used). +.PP +A line starting with \fBinclude\fP causes the rest of the line to be +interpreted as filename, which is opened and processed before the rest +of the file is processed. + +The only conditions currently supported are: +.TP +.B * +which means any package, +.TP +.BI "source '" name ' +which means any package with source \fIname\fP. +('\fB*\fP', '\fB?\fP' and '\fB[\fP..\fB]\fP' are treated as in shell wildcards). +.TP +.B sections '\fIname\fP'\fR(\fP|'\fIname\fP'\fR)*\fP +matches an upload in which each section matches one of the names +given. +As upload conditions are checked very early, +this is the section listed in the .changes file, +not the one from the override file. +(But this might change in the future, +if you have the need for the one or the other behavior, let me know). +.TP +.B sections contain '\fIname\fP'\fR(\fP|'\fIname\fP'\fR)*\fP +The same, but not all sections must be from the given set, +but at least one source or binary package needs to have one of those given. +.TP +.B binaries '\fIname\fP'\fR(\fP|'\fIname\fP'\fR)*\fP +matches an upload in which each binary (type deb or udeb) +matches one of the names given. +.TP +.B binaries contain '\fIname\fP'\fR(\fP|'\fIname\fP'\fR)*\fP +again only at least one instead of all is required. +.TP +.B architectures '\fIarchitecture\fP'\fR(\fP|'\fIname\fP'\fR)*\fP +matches an upload in which each package has only architectures +from the given set. +\fBsource\fP and \fBall\fP are treated as unique architectures. +Wildcards are not allowed. +.TP +.B architectures contain '\fIarchitecture\fP'\fR(\fP|'\fIarchitecture\fP'\fR)*\fP +again only at least one instead of all is required. +.TP +.B byhand +matches an upload with at least one byhand file +(i.e. a file with section \fBbyhand\fP or \fBraw\-\fP\fIsomething\fP). +.TP +.B byhand '\fIsection\fP'\fR(\fP|'\fIsection\fP'\fR)*\fP +matches an upload with at least one byhand file and +all byhand files having a section listed in the list of given section. +(i.e. \fBbyhand 'byhand'|'raw\-*'\fP is currently is the same as \fBbyhand\fP). +.TP +.BI "distribution '" codename ' +which means any package when it is to be included in \fIcodename\fP. +As the uploaders file is given by distribution, this is only useful +to reuse a complex uploaders file for multiple distributions. +.PP +Putting \fBnot\fP in front of a condition, inverses it's meaning. +For example +.br +\fBallow not source 'r*' by anybody\fP +.br +means anybody may upload packages which source name does not start +with an 'r'. +.PP +Multiple conditions can be connected with \fBand\fP and \fBor\fP, +with \fBor\fP binding stronger (but both weaker than \fBnot\fP). +That means +.br +\fBallow source 'r*' and source '*xxx' or source '*o' by anybody\fP +.br +is equivalent to +.br +\fBallow source 'r*xxx' by anybody\fP +.br +\fBallow source 'r*o' by anybody\fP + +(Other conditions +will follow once somebody tells me what restrictions are useful. +Currently planned is only something for architectures). +.SH "ERROR IGNORING" +With \fB\-\-ignore\fP on the command line or an \fIignore\fP +line in the options file, the following type of errors can be +ignored: +.TP +.B brokenold \fR(hopefully never seen) +If there are errors parsing an installed version of package, do not +error out, but assume it is older than anything else, has not files +or no source name. +.TP +.B brokensignatures +If a .changes or .dsc file contains at least one invalid signature +and no valid signature (not even expired or from an expired or revoked key), +reprepro assumes the file got corrupted and refuses to use it unless this +ignore directive is given. +.TP +.B brokenversioncmp \fR(hopefully never seen) +If comparing old and new version fails, assume the new one is newer. +.TP +.B dscinbinnmu +If a .changes file has an explicit Source version that is different the +to the version header of the file, +than reprepro assumes it is binary non maintainer upload (NMU). +In that case, source files are not permitted in .changes files +processed by +.B include +or +.BR processincoming . +Adding \fB\-\-ignore=dscinbinnmu\fP allows it for the \fBinclude\fP +command. +.TP +.B emptyfilenamepart \fR(insecure) +Allow strings to be empty that are used to construct filenames. +(like versions, architectures, ...) +.TP +.B extension +Allow one to \fBincludedeb\fP files that do not end with \fB.deb\fP, +to \fBincludedsc\fP files not ending in \fB.dsc\fP and to +\fBinclude\fP files not ending in \fB.changes\fP. +.TP +.B forbiddenchar \fR(insecure) +Do not insist on Debian policy for package and source names +and versions. +Thus allowing all 7-bit characters but slashes (as they would +break the file storage) and things syntactically active +(spaces, underscores in filenames in .changes files, opening +parentheses in source names of binary packages). +To allow some 8-bit chars additionally, use \fB8bit\fP additionally. +.TP +.B 8bit \fR(more insecure) +Allow 8-bit characters not looking like overlong UTF-8 sequences +in filenames and things used as parts of filenames. +Though it hopefully rejects overlong UTF-8 sequences, there might +be other characters your filesystem confuses with special characters, +thus creating filenames possibly equivalent to +\fB/mirror/pool/main/../../../etc/shadow\fP +(Which should be safe, as you do not run reprepro as root, do you?) +or simply overwriting your conf/distributions file adding some commands +in there. So do not use this if you are paranoid, unless you are paranoid +enough to have checked the code of your libs, kernel and filesystems. +.TP +.B ignore \fR(for forward compatibility) +Ignore unknown ignore types given to \fI\-\-ignore\fP. +.TP +.B flatandnonflat \fR(only suppresses a warning) +Do not warn about a flat and a non-flat distribution from the same +source with the same name when updating. +(Hopefully never ever needed.) +.TP +.B malformedchunk \fR(I hope you know what you do) +Do not stop when finding a line not starting with a space but +no colon(:) in it. These are otherwise rejected as they have no +defined meaning. +.TP +.B missingfield \fR(safe to ignore) +Ignore missing fields in a .changes file that are only checked but +not processed. +Those include: Format, Date, Urgency, Maintainer, Description, Changes +.TP +.B missingfile \fR(might be insecure) +When including a .dsc file from a .changes file, +try to get files needed but not listed in the .changes file +(e.g. when someone forgot to specify \-sa to dpkg\-buildpackage) +from the directory the .changes file is in instead of erroring out. +(\fB\-\-delete\fP will not work with those files, though.) +.TP +.B spaceonlyline \fR(I hope you know what you do) +Allow lines containing only (but non-zero) spaces. As these +do not separate chunks as thus will cause reprepro to behave +unexpected, they cause error messages by default. +.TP +.B surprisingarch +Do not reject a .changes file containing files for a +architecture not listed in the Architecture-header within it. +.TP +.B surprisingbinary +Do not reject a .changes file containing .deb files containing +packages whose name is not listed in the "Binary:" header +of that changes file. +.TP +.B undefinedtarget \fR(hope you are not using the wrong db directory) +Do not stop when the packages.db file contains databases for +codename/packagetype/component/architectures combinations that are +not listed in your distributions file. + +This allows you to temporarily remove some distribution from the config files, +without having to remove the packages in it with the \fBclearvanished\fP +command. +You might even temporarily remove single architectures or components, +though that might cause inconsistencies in some situations. +.TP +.B undefinedtracking \fR(hope you are not using the wrong db directory) +Do not stop when the tracking file contains databases for +distributions that are not listed in your \fBdistributions\fP file. + +This allows you to temporarily remove some distribution from the config files, +without having to remove the packages in it with the \fBclearvanished\fP +command. +You might even temporarily disable tracking in some distribution, but that +is likely to cause inconsistencies in there, if you do not know, what you +are doing. +.TP +.B unknownfield \fR(for forward compatibility) +Ignore unknown fields in the config files, instead of refusing to run +then. +.TP +.B unusedarch \fR(safe to ignore) +No longer reject a .changes file containing no files for any of the +architectures listed in the Architecture-header within it. +.TP +.B unusedoption +Do not complain about command line options not used by the +specified action (like \fB\-\-architecture\fP). +.TP +.B uploaders +The include command will accept packages that would otherwise been +rejected by the uploaders file. +.TP +.B wrongarchitecture \fR(safe to ignore) +Do not warn about wrong "Architecture:" lines in downloaded +Packages files. +(Note that wrong Architectures are always ignored when getting +stuff from flat repositories or importing stuff from one architecture +to another). +.TP +.B wrongdistribution \fR(safe to ignore) +Do not error out if a .changes file is to be placed in a +distribution not listed in that files' Distributions: header. +.TP +.B wrongsourceversion +Do not reject a .changes file containing .deb files with +a different opinion on what the version of the source package is. +.br +(Note: reprepro only compares literally here, not by meaning.) +.TP +.B wrongversion +Do not reject a .changes file containing .dsc files with +a different version. +.br +(Note: reprepro only compares literally here, not by meaning.) +.TP +.B expiredkey \fR(I hope you know what you do) +Accept signatures with expired keys. +(Only if the expired key is explicitly requested). +.TP +.B expiredsignature \fR(I hope you know what you do) +Accept expired signatures with expired keys. +(Only if the key is explicitly requested). +.TP +.B revokedkey \fR(I hope you know what you do) +Accept signatures with revoked keys. +(Only if the revoked key is explicitly requested). +.SH GUESSING +When including a binary or source package without explicitly +declaring a component with +.B \-C +it will take the +first component with the name of the section, being +prefix to the section, being suffix to the section +or having the section as prefix or any. (In this order) + +Thus having specified the components: +"main non\-free contrib non\-US/main non\-US/non\-free non\-US/contrib" +should map e.g. +"non\-US" to "non\-US/main" and "contrib/editors" to "contrib", +while having only "main non\-free and contrib" as components should +map "non\-US/contrib" to "contrib" and "non\-US" to "main". + +.B NOTE: +Always specify main as the first component, if you want things +to end up there. + +.B NOTE: +unlike in dak, non\-US and non\-us are different things... +.SH NOMENCLATURE +.B Codename +the primary identifier of a given distribution. This are normally +things like \fBsarge\fP, \fBetch\fP or \fBsid\fP. +.TP +.B basename +the name of a file without any directory information. +.TP +.B byhand +Changes files can have files with section 'byhand' (Debian) or 'raw\-' (Ubuntu). +Those files are not packages but other data generated (usually together with +packages) and then uploaded together with this changes files. + +With reprepro those can be stored in the pool next to their packages with +tracking, put in some log directory when using processincoming, or given to +an hook script (currently only possible with processincoming). +.TP +.B filekey +the position relative to the outdir. (as found in "Filename:" in Packages.gz) +.TP +.B "full filename" +the position relative to / +.TP +.B architecture +The term like \fBsparc\fP, \fBi386\fP, \fBmips\fP, ... . +To refer to the source packages, \fBsource\fP +is sometimes also treated as architecture. +.TP +.B component +Things like \fBmain\fP, \fBnon\-free\fP and \fBcontrib\fP +(by policy and some other programs also called section, reprepro follows +the naming scheme of apt here.) +.TP +.B section +Things like \fBbase\fP, \fBinterpreters\fP, \fBoldlibs\fP and \fBnon\-free/math\fP +(by policy and some other programs also called subsections). +.TP +.B md5sum +The checksum of a file in the format +"\fI<md5sum of file>\fP \fI<length of file>\fP" +.SH Some note on updates +.SS A version is not overwritten with the same version. +.B reprepro +will never update a package with a version it already has. This would +be equivalent to rebuilding the whole database with every single upgrade. +To force the new same version in, remove it and then update. +(If files of +the packages changed without changing their name, make sure the file is +no longer remembered by reprepro. +Without \fB\-\-keepunreferencedfiled\fP +and without errors while deleting it should already be forgotten, otherwise +a \fBdeleteunreferenced\fP or even some \fB__forget\fP might help.) +.SS The magic delete rule ("\-"). +A minus as a single word in the +.B Update: +line of a distribution marks everything to be deleted. The mark causes later rules +to get packages even if they have (strict) lower versions. The mark will +get removed if a later rule sets the package on hold (hold is not yet implemented, +in case you might wonder) or would get a package with the same version +(Which it will not, see above). If the mark is still there at the end of the processing, +the package will get removed. +.P +Thus the line "Update: \- +.I rules +" will cause all packages to be exactly the +highest Version found in +.I rules. +The line "Update: +.I near +\- +.I rules +" will do the same, except if it needs to download packages, it might download +it from +.I near +except when too confused. (It will get too confused e.g. when +.I near +or +.I rules +have multiple versions of the package and the highest in +.I near +is not the first one in +.I rules, +as it never remember more than one possible spring for a package. +.P +Warning: This rule applies to all type/component/architecture triplets +of a distribution, not only those some other update rule applies to. +(That means it will delete everything in those!) +.SH ENVIRONMENT VARIABLES +Environment variables are always overwritten by command line options, +but overwrite options set in the \fBoptions\fP file. (Even when the +options file is obviously parsed after the environment variables as +the environment may determine the place of the options file). +.TP +.B REPREPRO_BASE_DIR +The directory in this variable is used instead of the current directory, +if no \fB\-b\fP or \fB\-\-basedir\fP options are supplied. +.br +It is also set in all hook scripts called by reprepro +(relative to the current directory or absolute, +depending on how reprepro got it). +.TP +.B REPREPRO_CONFIG_DIR +The directory in this variable is used when no \fB\-\-confdir\fP is +supplied. +.br +It is also set in all hook scripts called by reprepro +(relative to the current directory or absolute, +depending on how reprepro got it). +.TP +.B REPREPRO_OUT_DIR +This is not used, but only set in hook scripts called by reprepro +to the directory in which the \fBpool\fP subdirectory resides +(relative to the current directory or absolute, +depending on how reprepro got it). +.TP +.B REPREPRO_DIST_DIR +This is not used, but only set in hook scripts called by reprepro +to the \fBdists\fP directory (relative to the current directory or +absolute, depending on how reprepro got it). +.TP +.B REPREPRO_LOG_DIR +This is not used, but only set in hook scripts called by reprepro +to the value setable by \fB\-\-logdir\fP. +.TP +.B REPREPRO_CAUSING_COMMAND +.TP +.B REPREPRO_CAUSING_FILE +Those two environment variable are set (or unset) in +\fBLog:\fP and \fBByHandHooks:\fP scripts and hint what command +and what file caused the hook to be called (if there is some). +.TP +.B REPREPRO_CAUSING_RULE +This environment variable is set (or unset) in +\fBLog:\fP scripts and hint what update or pull rule caused +this change. +.TP +.B REPREPRO_FROM +This environment variable is set (or unset) in +\fBLog:\fP scripts and denotes what other distribution a +package is copied from (with pull and copy commands). +.TP +.B REPREPRO_FILTER_ARCHITECTURE +.TP +.B REPREPRO_FILTER_CODENAME +.TP +.B REPREPRO_FILTER_COMPONENT +.TP +.B REPREPRO_FILTER_PACKAGETYPE +.TP +.B REPREPRO_FILTER_PATTERN +Set in \fBFilterList:\fP and \fBFilterSrcList:\fP scripts. +.TP +.B GNUPGHOME +Not used by reprepro directly. +But reprepro uses libgpgme, which calls gpg for signing and verification +of signatures. +And your gpg will most likely use the content of this variable +instead of "~/.gnupg". +Take a look at +.BR gpg (1) +to be sure. +You can also tell reprepro to set this with the \fB\-\-gnupghome\fP option. +.TP +.B GPG_TTY +When there is a gpg\-agent running that does not have the passphrase +cached yet, gpg will most likely try to start some pinentry program +to get it. +If that is pinentry\-curses, that is likely to fail without this +variable, because it cannot find a terminal to ask on. +In this cases you might set this variable to something like +the value of +.B $(tty) +or +.B $SSH_TTY +or anything else denoting a usable terminal. (You might also +want to make sure you actually have a terminal available. +With ssh you might need the +.B \-t +option to get a terminal even when telling gpg to start a specific command). + +By default, reprepro will set this variable to what the symbolic link +.B /proc/self/fd/0 +points to, if stdin is a terminal, unless you told with +.B \-\-noguessgpgtty +to not do so. +.SH BUGS +Increased verbosity always shows those things one does not want to know. +(Though this might be inevitable and a corollary to Murphy) + +Reprepro uses berkeley db, which was a big mistake. +The most annoying problem not yet worked around is database corruption +when the disk runs out of space. +(Luckily if it happens while downloading packages while updating, +only the files database is affected, which is easy (though time consuming) +to rebuild, see \fBrecovery\fP file in the documentation). +Ideally put the database on another partition to avoid that. + +While the source part is mostly considered as the architecture +.B source +some parts may still not use this notation. +.SH "WORK-AROUNDS TO COMMON PROBLEMS" +.TP +.B gpgme returned an impossible condition +With the woody version this normally meant that there was no .gnupg +directory in $HOME, but it created one and reprepro succeeds when called +again with the same command. +Since sarge the problem sometimes shows up, too. But it is no longer +reproducible and it does not fix itself, neither. Try running +\fBgpg \-\-verify \fP\fIfile-you-had-problems-with\fP manually as the +user reprepro is running and with the same $HOME. This alone might +fix the problem. It should not print any messages except perhaps +.br +gpg: no valid OpenPGP data found. +.br +gpg: the signature could not be verified. +.br +if it was an unsigned file. +.TP +.B not including .orig.tar.gz when a .changes file's version does not end in \-0 or \-1 +If dpkg\-buildpackage is run without the \fB\-sa\fP option to build a version with +a Debian revision not being \-0 or \-1, it does not list the \fB.orig.tar.gz\fP file +in the \fB.changes\fP file. +If you want to \fBinclude\fP such a file with reprepro +when the .orig.tar.gz file does not already exist in the pool, reprepro will report +an error. +This can be worked around by: +.br +call \fBdpkg\-buildpackage\fP with \fB\-sa\fP (recommended) +.br +copy the .orig.tar.gz file to the proper place in the pool before +.br +call reprepro with \-\-ignore=missingfile (discouraged) +.TP +.B leftover files in the pool directory. +reprepro is sometimes a bit too timid of deleting stuff. When things +go wrong and there have been errors it sometimes just leaves everything +where it is. +To see what files reprepro remembers to be in your pool directory but +does not know anything needing them right know, you can use +.br +\fBreprepro dumpunreferenced\fP +.br +To delete them: +.br +\fBreprepro deleteunreferenced\fP +.SH INTERRUPTING +Interrupting reprepro has its problems. +Some things (like speaking with apt methods, database stuff) can cause +problems when interrupted at the wrong time. +Then there are design problems of the code making it hard to distinguish +if the current state is dangerous or non-dangerous to interrupt. +Thus if reprepro receives a signal normally sent to tell a process to +terminate itself softly, +it continues its operation, but does not start any new operations. +(I.e. it will not tell the apt\-methods any new file to download, it will +not replace a package in a target, unless it already had started with it, +it will not delete any files gotten dereferenced, and so on). + +\fBIt only catches the first signal of each type. The second signal of a +given type will terminate reprepro. You will risk database corruption +and have to remove the lockfile manually.\fP + +Also note that even normal interruption leads to code-paths mostly untested +and thus expose a multitude of bugs including those leading to data corruption. +Better think a second more before issuing a command than risking the need +for interruption. +.SH "REPORTING BUGS" +Report bugs or wishlist requests to the Debian BTS +.br +(e.g. by using \fBreportbug reprepro\fP under Debian) +.br +or directly to +.MTO brlink@debian.org +.SH COPYRIGHT +Copyright \(co 2004,2005,2006,2007,2008,2009,2010,2011,2012 +.URL http://www.brlink.eu "Bernhard R. Link" +.br +This is free software; see the source for copying conditions. There is NO +warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. diff --git a/docs/reprepro.bash_completion b/docs/reprepro.bash_completion new file mode 100644 index 0000000..062397e --- /dev/null +++ b/docs/reprepro.bash_completion @@ -0,0 +1,742 @@ +_reprepro() +{ + local cur prev commands options noargoptions i state cmd ignores hiddencommands commands codenames confdir outdir basedir architectures components importrules snapshots + + confdir="" + basedir="" + outdir="" + distdir="" + + # for parsing configs consider: + # - command line arguments take priority over <confdir>/options take priority over environment variables + # - all the ways to set confdir may start with '+b/' to denote the basedir to be used. + # - basedir might also be set in <confdir>/options (which obviously does not change where this file is) + + function parse_config() { + local conffile distfile + if [[ -n "$confdir" ]] ; then + conffile="$confdir/options" + distfile="$confdir/distributions" + elif [[ -n "${REPREPRO_CONFIG_DIR:+set}" ]] ; then + conffile="$REPREPRO_CONFIG_DIR/options" + distfile="$REPREPRO_CONFIG_DIR/distributions" + else + conffile="+b/conf/options" + distfile="+b/conf/distributions" + fi + confbasedir="${basedir:-${REPREPRO_BASE_DIR:-.}}" + if [ x"${conffile#+b/}" != x"${conffile}" ] ; then + conffile="$confbasedir/${conffile#+b/}" + fi + if [ -z "$basedir" ] && [[ -e "$conffile" ]] && grep -q '^basedir ' -- "$conffile" 2>/dev/null ; then + confbasedir="$(grep '^basedir ' -- "$conffile" 2>/dev/null | sed -e 's/^basedir *//')" + fi + if [ -z "$confdir" ] && [[ -e "$conffile" ]] && grep -q '^confdir ' -- "$conffile" 2>/dev/null ; then + distfile="$(grep '^confdir ' -- "$conffile" 2>/dev/null | sed -e 's/^confdir *//')/distributions" + fi + if [ x"${distfile#+b/}" != x"${distfile}" ] ; then + distfile="$confbasedir/${distfile#+b/}" + fi + if [[ -d "$distfile" ]] ; then + codenames="$(awk -- '/^[Cc][Oo][Dd][Ee][Nn][Aa][Mm][Ee]: / {$1="";print}' "$distfile"/*.conf)" + architectures="$(awk -- '/^[Aa][Rr][Cc][Hh][Ii][Tt][Ee][Cc][Tt][Uu][Rr][Ee][Ss]: / {$1="";print}' "$distfile"/*.conf)" + components="$(awk -- '/^[Cc][Oo][Mm][Pp][Oo][Nn][Ee][Nn][Tt][Ss]: / {$1="";print}' "$distfile"/*.conf)" + elif [[ -e "$distfile" ]] ; then + codenames="$(awk -- '/^[Cc][Oo][Dd][Ee][Nn][Aa][Mm][Ee]: / {$1="";print}' "$distfile")" + architectures="$(awk -- '/^[Aa][Rr][Cc][Hh][Ii][Tt][Ee][Cc][Tt][Uu][Rr][Ee][Ss]: / {$1="";print}' "$distfile")" + components="$(awk -- '/^[Cc][Oo][Mm][Pp][Oo][Nn][Ee][Nn][Tt][Ss]: / {$1="";print}' "$distfile")" + else + codenames="experimental sid whatever-you-defined" + architectures="source i386 abacus whatever-you-defined" + components="main contrib non-free whatever-you-defined" + fi + } + function parse_config_for_distdir() { + local conffile + if [[ -n "$confdir" ]] ; then + conffile="$confdir/options" + elif [[ -n "${REPREPRO_CONFIG_DIR:+set}" ]] ; then + conffile="$REPREPRO_CONFIG_DIR/options" + else + conffile="+b/conf/options" + fi + if [ x"${conffile#+b/}" != x"${conffile}" ] ; then + conffile="${basedir:-${REPREPRO_BASE_DIR:-.}}/${conffile#+b/}" + fi + if [ -z "$basedir" ] && [[ -e "$conffile" ]] ; then + if grep -q '^basedir ' -- "$conffile" 2>/dev/null ; then + basedir="$(grep '^basedir ' -- "$conffile" 2>/dev/null | sed -e 's/^basedir *//')" + fi + fi + if [ -z "$outdir" ] && [[ -e "$conffile" ]] ; then + if grep -q '^outdir ' -- "$conffile" 2>/dev/null ; then + outdir="$(grep '^outdir ' -- "$conffile" 2>/dev/null | sed -e 's/^outdir *//')" + fi + fi + if [ -z "$distdir" ] && [[ -e "$conffile" ]] ; then + if grep -q '^distdir ' -- "$conffile" 2>/dev/null ; then + distdir="$(grep '^distdir ' -- "$conffile" 2>/dev/null | sed -e 's/^distdir *//')" + fi + fi + if [ -z "$basedir" ] ; then + basedir="${REPREPRO_BASE_DIR:-.}" + fi + if [ -z "$outdir" ] ; then + outdir="${REPREPRO_OUT_DIR:-$basedir}" + fi + if [ x"${outdir#+b/}" != x"${outdir}" ] ; then + outdir="$basedir/${outdir#+b/}" + fi + if [ -z "$distdir" ] ; then + distdir="${REPREPRO_DIST_DIR:-$outdir/dists}" + fi + if [ x"${distdir#+o/}" != x"${distdir}" ] ; then + distdir="$outdir/${distdir#+o/}" + elif [ x"${distdir#+b/}" != x"${distdir}" ] ; then + distdir="$basedir/${distdir#+b/}" + fi + } + function parse_incoming() { + local conffile incomingfile + if [[ -n "$confdir" ]] ; then + conffile="$confdir/options" + incomingfile="$confdir/incoming" + elif [[ -n "${REPREPRO_CONFIG_DIR:+set}" ]] ; then + conffile="$REPREPRO_CONFIG_DIR/options" + incomingfile="$REPREPRO_CONFIG_DIR/incoming" + else + conffile="+b/conf/options" + incomingfile="+b/conf/incoming" + fi + confbasedir="${basedir:-${REPREPRO_BASE_DIR:-.}}" + if [ x"${conffile#+b/}" != x"${conffile}" ] ; then + conffile="$confbasedir/${conffile#+b/}" + fi + if [ -z "$basedir" ] && [[ -e "$conffile" ]] && grep -q '^basedir ' -- "$conffile" 2>/dev/null ; then + confbasedir="$(grep '^basedir ' -- "$conffile" 2>/dev/null | sed -e 's/^basedir *//')" + fi + if [ -z "$confdir" ] && [[ -e "$conffile" ]] && grep -q '^confdir ' -- "$conffile" 2>/dev/null ; then + incomingfile="$(grep '^confdir ' -- "$conffile" 2>/dev/null | sed -e 's/^confdir //')/incoming" + fi + if [ x"${incomingfile#+b/}" != x"${incomingfile}" ] ; then + incomingfile="$confbasedir/${incomingfile#+b/}" + fi + if [[ -d "$incomingfile" ]] ; then + importrules="$(awk -- '/^[Nn][Aa][Mm][Ee]: / {$1="";print}' "$incomingfile"/*.conf)" + elif [[ -e "$incomingfile" ]] ; then + importrules="$(awk -- '/^[Nn][Aa][Mm][Ee]: / {$1="";print}' "$incomingfile")" + else + importrules="rule-name" + fi + } + + COMPREPLY=() + + ignores='ignore flatandnonflat forbiddenchar 8bit emptyfilenamepart\ + spaceonlyline malformedchunk unknownfield\ + wrongdistribution missingfield brokenold\ + undefinedtracking undefinedtarget unusedoption\ + brokenversioncmp extension unusedarch surprisingarch\ + surprisingbinary wrongsourceversion wrongversion dscinbinnmu\ + brokensignatures uploaders missingfile longkeyid\ + expiredkey expiredsignature revokedkey oldfile wrongarchitecture' + noargoptions='--delete --nodelete --help -h --verbose -v\ + --nothingiserror --nolistsdownload --keepunreferencedfiles --keepunusednewfiles\ + --keepdirectories --keeptemporaries --keepuneededlists\ + --ask-passphrase --nonothingiserror --listsdownload\ + --nokeepunreferencedfiles --nokeepdirectories --nokeeptemporaries\ + --nokeepuneededlists --nokeepunusednewfiles\ + --noask-passphrase --skipold --noskipold --show-percent \ + --version --guessgpgtty --noguessgpgtty --verbosedb --silent -s --fast' + options='-b -i --basedir --outdir --ignore --unignore --methoddir --distdir --dbdir\ + --listdir --confdir --logdir --morguedir \ + --section -S --priority -P --component -C\ + --architecture -A --type -T --export --waitforlock \ + --spacecheck --safetymargin --dbsafetymargin\ + --gunzip --bunzip2 --unlzma --unxz --lunzip --gnupghome --list-format --list-skip --list-max\ + --outhook --endhook' + + i=1 + prev="" + cmd="XYZnoneyetXYZ" + while [[ $i -lt $COMP_CWORD ]] ; do + cur=${COMP_WORDS[i]} + prev="" + case "$cur" in + --basedir=*) + basedir="${cur#--basedir=}" + i=$((i+1)) + ;; + --outdir=*) + outdir="${cur#--basedir=}" + i=$((i+1)) + ;; + --distdir=*) + distdir="${cur#--basedir=}" + i=$((i+1)) + ;; + --confdir=*) + confdir="${cur#--confdir=}" + i=$((i+1)) + ;; + --*=*) + i=$((i+1)) + ;; + -b|--basedir) + prev="$cur" + basedir="${COMP_WORDS[i+1]}" + i=$((i+2)) + ;; + --outdir) + prev="$cur" + outdir="${COMP_WORDS[i+1]}" + i=$((i+2)) + ;; + --distdir) + prev="$cur" + distdir="${COMP_WORDS[i+1]}" + i=$((i+2)) + ;; + --confdir) + prev="$cur" + confdir="${COMP_WORDS[i+1]}" + i=$((i+2)) + ;; + -i|--ignore|--unignore|--methoddir|--distdir|--dbdir|--listdir|--section|-S|--priority|-P|--component|-C|--architecture|-A|--type|-T|--export|--waitforlock|--spacecheck|--checkspace|--safetymargin|--dbsafetymargin|--logdir|--gunzip|--bunzip2|--unlzma|--unxz|--lunzip|--gnupghome|--morguedir) + + prev="$cur" + i=$((i+2)) + ;; + --*|-*) + i=$((i+1)) + ;; + *) + cmd="$cur" + i=$((i+1)) + break + ;; + esac + done + cur=${COMP_WORDS[COMP_CWORD]} + if [[ $i -gt $COMP_CWORD && -n "$prev" ]]; then + case "$prev" in + -b|--basedir|--outdir|--methoddir|--distdir|--dbdir|--listdir|--confdir) + COMPREPLY=( $( compgen -d -- $cur ) ) + + return 0 + ;; + -T|--type) + COMPREPLY=( $( compgen -W "dsc deb udeb" -- $cur ) ) + return 0 + ;; + -i|--ignore|--unignore) + COMPREPLY=( $( compgen -W "$ignores" -- $cur ) ) + return 0 + ;; + -P|--priority) + COMPREPLY=( $( compgen -W "required important standard optional extra" -- $cur ) ) + return 0 + ;; + -S|--section) + COMPREPLY=( $( compgen -W "admin base comm contrib devel doc editors electronics embedded games gnome graphics hamradio interpreters kde libs libdevel mail math misc net news non-free oldlibs otherosfs perl python science shells sound tex text utils web x11 contrib/admin contrib/base contrib/comm contrib/contrib contrib/devel contrib/doc contrib/editors contrib/electronics contrib/embedded contrib/games contrib/gnome contrib/graphics contrib/hamradio contrib/interpreters contrib/kde contrib/libs contrib/libdevel contrib/mail contrib/math contrib/misc contrib/net contrib/news contrib/non-free contrib/oldlibs contrib/otherosfs contrib/perl contrib/python contrib/science contrib/shells contrib/sound contrib/tex contrib/text contrib/utils contrib/web contrib/x11 non-free/admin non-free/base non-free/comm non-free/contrib non-free/devel non-free/doc non-free/editors non-free/electronics non-free/embedded non-free/games non-free/gnome non-free/graphics non-free/hamradio non-free/interpreters non-free/kde non-free/libs non-free/libdevel non-free/mail non-free/math non-free/misc non-free/net non-free/news non-free/non-free non-free/oldlibs non-free/otherosfs non-free/perl non-free/python non-free/science non-free/shells non-free/sound non-free/tex non-free/text non-free/utils non-free/web non-free/x11" -- $cur ) ) + return 0 + ;; + -A|--architecture) + parse_config + COMPREPLY=( $( compgen -W "$architectures" -- $cur ) ) + return 0 + ;; + -C|--component) + parse_config + COMPREPLY=( $( compgen -W "$components" -- $cur ) ) + return 0 + ;; + --export) + COMPREPLY=( $( compgen -W "silent-never never changed lookedat force" -- $cur ) ) + return 0 + ;; + --waitforlock) + COMPREPLY=( $( compgen -W "0 60 3600 86400" -- $cur ) ) + return 0 + ;; + --spacecheck) + COMPREPLY=( $( compgen -W "none full" -- $cur ) ) + return 0 + ;; + --safetymargin) + COMPREPLY=( $( compgen -W "0 1048576" -- $cur ) ) + return 0 + ;; + --dbsafetymargin) + COMPREPLY=( $( compgen -W "0 104857600" -- $cur ) ) + return 0 + ;; + esac + fi + + if [[ "XYZnoneyetXYZ" = "$cmd" ]] ; then + commands='build-needing\ + check\ + checkpool\ + checkpull\ + checkupdate\ + cleanlists\ + clearvanished\ + collectnewchecksums\ + copy\ + copyfilter\ + copymatched\ + copysrc\ + createsymlinks\ + deleteunreferenced\ + deleteifunreferenced\ + dumpreferences\ + dumptracks\ + dumppull\ + dumpunreferenced\ + dumpupdate\ + export\ + forcerepairdescriptions\ + flood\ + generatefilelists\ + gensnapshot\ + unreferencesnapshot\ + include\ + includedeb\ + includedsc\ + includeudeb\ + list\ + listfilter\ + listmatched\ + ls\ + lsbycomponent\ + move\ + movefilter\ + movematched\ + movesrc\ + predelete\ + processincoming\ + pull\ + remove\ + removealltracks\ + removefilter\ + removematched\ + removesrc\ + removesrcs\ + removetrack\ + reoverride\ + repairdescriptions\ + reportcruft\ + rereference\ + rerunnotifiers\ + restore\ + restorefilter\ + restorematched\ + restoresrc\ + retrack\ + sourcemissing\ + tidytracks\ + translatefilelists\ + translatelegacychecksums\ + unusedsources\ + update' + hiddencommands='__d\ + __dumpuncompressors + __extractcontrol\ + __extractfilelist\ + __extractsourcesection\ + __uncompress\ + _addchecksums\ + _addpackage\ + _addreference\ + _addreferences\ + _detect\ + _dumpcontents\ + _fakeemptyfilelist\ + _forget\ + _listchecksums\ + _listcodenames\ + _listconfidentifiers\ + _listdbidentifiers\ + _listmd5sums\ + _removereference\ + _removereferences\ + _versioncompare' + + if [[ "$cur" == -* ]]; then + case "$cur" in + --ignore=*) + COMPREPLY=( $( compgen -W "$ignores" -- ${cur#--ignore=} ) ) + ;; + --unignore=*) + COMPREPLY=( $( compgen -W "$ignores" -- ${cur#--unignore=} ) ) + ;; + --component=*) + parse_config + COMPREPLY=( $( compgen -W "$components" -- {cur#--component=} ) ) + ;; + --architectures=*) + parse_config + COMPREPLY=( $( compgen -W "$architectures" -- {cur#--architectures=} ) ) + ;; + + *) + COMPREPLY=( $( compgen -W "$options $noargoptions" -- $cur ) ) + ;; + esac + elif [[ "$cur" == _* ]]; then + COMPREPLY=( $( compgen -W "$hiddencommands" -- $cur ) ) + else + COMPREPLY=( $( compgen -W "$commands" -- $cur ) ) + fi + return 0 + fi + + case "$cmd" in + remove|list|listfilter|removefilter|removetrack|listmatched|removematched|removesrc|removesrcs) + # first argument is the codename + if [[ $i -eq $COMP_CWORD ]] ; then + parse_config + COMPREPLY=( $( compgen -W "$codenames" -- $cur ) ) + return 0 + fi + # these later could also look for stuff, but + # that might become a bit slow + ;; + export|update|checkupdate|pull|checkpull|rereference|retrack|removealltracks|tidytracks|dumptracks|check|repairdescriptions|forcerepairdescriptions|reoverride|rerunnotifiers|dumppull|dumpupdate|unusedsources|sourcemissing|reportcruft) + # all arguments are codenames + parse_config + COMPREPLY=( $( compgen -W "$codenames" -- $cur ) ) + return 0 + ;; + + processincoming) + # arguments are rule-name from conf/incoming + parse_config + parse_incoming + if [[ $i -eq $COMP_CWORD ]] ; then + COMPREPLY=( $( compgen -W "$importrules" -- $cur ) ) + return 0 + fi + ;; + + collectnewchecksums|cleanlists|_listcodenames) + return 0 + ;; + + checkpool) + # first argument can be fast + if [[ $i -eq $COMP_CWORD ]] ; then + COMPREPLY=( $( compgen -W "fast" -- $cur ) ) + return 0 + fi + return 0 + ;; + flood) + # first argument is the codename + if [[ $i -eq $COMP_CWORD ]] ; then + parse_config + COMPREPLY=( $( compgen -W "$codenames" -- $cur ) ) + return 0 + fi + # then an architecture might follow + if [[ $(( $i + 1 )) -eq $COMP_CWORD ]] ; then + parse_config + COMPREPLY=( $( compgen -W "$architectures" -- $cur ) ) + return 0 + fi + # then nothing else + return 0 + ;; + build-needing) + # first argument is the codename + if [[ $i -eq $COMP_CWORD ]] ; then + parse_config + COMPREPLY=( $( compgen -W "$codenames" -- $cur ) ) + return 0 + fi + # then an architecture + if [[ $(( $i + 1 )) -eq $COMP_CWORD ]] ; then + parse_config + COMPREPLY=( $( compgen -W "$architectures" -- $cur ) ) + return 0 + fi + # then a glob + if [[ $(( $i + 2 )) -eq $COMP_CWORD ]] ; then + COMPREPLY=( $( compgen -W "$cur'\*'" -- $cur ) ) + return 0 + fi + return 0 + ;; + __uncompress) + # first argument is method + if [[ $i -eq $COMP_CWORD ]] ; then + COMPREPLY=( $( compgen -W ".gz .bz2 .lzma .xz .lz" -- $cur ) ) + return 0 + fi + if [[ $(( $i + 1 )) -eq $COMP_CWORD ]] ; then + COMPREPLY=( $( compgen -f -- $cur ) ) + return 0 + fi + if [[ $(( $i + 2 )) -eq $COMP_CWORD ]] ; then + COMPREPLY=( $( compgen -f -- $cur ) ) + return 0 + fi + return 0 + ;; + __extractsourcesection) + if [[ $i -eq $COMP_CWORD ]] ; then + _filedir dsc + fi + return 0 + ;; + includedeb) + # first argument is the codename + if [[ $i -eq $COMP_CWORD ]] ; then + parse_config + COMPREPLY=( $( compgen -W "$codenames" -- $cur ) ) + return 0 + fi + # then one .deb file follows + if [[ $(( $i + 1 )) -eq $COMP_CWORD ]] ; then + _filedir deb + fi + return 0 + ;; + includedsc) + # first argument is the codename + if [[ $i -eq $COMP_CWORD ]] ; then + parse_config + COMPREPLY=( $( compgen -W "$codenames" -- $cur ) ) + return 0 + fi + # then one .dsc file follows + if [[ $(( $i + 1 )) -eq $COMP_CWORD ]] ; then + _filedir dsc + fi + return 0 + ;; + include) + # first argument is the codename + if [[ $i -eq $COMP_CWORD ]] ; then + parse_config + COMPREPLY=( $( compgen -W "$codenames" -- $cur ) ) + return 0 + fi + # then one .changes file follows + if [[ $(( $i + 1 )) -eq $COMP_CWORD ]] ; then + _filedir changes + fi + return 0 + ;; + gensnapshot|unreferencesnapshot) + # first argument is a codename + if [[ $i -eq $COMP_CWORD ]] ; then + parse_config + COMPREPLY=( $( compgen -W "$codenames" -- $cur ) ) + return 0 + fi + # then the name of a snapshot, add a suggestion + if [[ $(( $i + 1 )) -eq $COMP_CWORD ]] ; then + COMPREPLY=( $( compgen -W "$(date +%Y/%m/%d)" -- $cur ) ) + return 0 + fi + return 0; + ;; + copy|copysrc|copyfilter|copymatched|move|movesrc|movefilter|movematched) + # first argument is a codename + if [[ $i -eq $COMP_CWORD ]] ; then + parse_config + COMPREPLY=( $( compgen -W "$codenames" -- $cur ) ) + return 0 + fi + # second argument is a codename + if [[ $(( $i + 1 )) -eq $COMP_CWORD ]] ; then + parse_config + COMPREPLY=( $( compgen -W "$codenames" -- $cur ) ) + return 0 + fi + # here we could look for package names existing in + # that distribution, but that would be slow... + ;; + restore|restoresrc|restorefilter|restorematched) + # first argument is a codename + if [[ $i -eq $COMP_CWORD ]] ; then + parse_config + COMPREPLY=( $( compgen -W "$codenames" -- $cur ) ) + return 0 + fi + # second argument is snapshot of that name + if [[ $(( $i + 1 )) -eq $COMP_CWORD ]] ; then + parse_config_for_distdir + snapshots="$( ls "$distdir/${COMP_WORDS[i]}/snapshots" )" + COMPREPLY=( $( compgen -W "$snapshots" -- $cur ) ) + return 0 + fi + # here we could look for package names existing in + # that distribution, but that would be slow... + ;; + __dumpuncompressors|translatelageacychecksums|deleteunreferenced) + # no arguments + return 0 + ;; + deleteifunreferenced) + # less useful than the output of dumpunreferenced, + # but this way it should be massively faster: + parse_config_for_distdir + COMPREPLY=( $(cd "$outdir" && compgen -o filenames -f -- $cur) ) + return 0 + ;; + esac + COMPREPLY=( $( compgen -f -- $cur ) ) + return 0 +} +# This -o filename has its problems when there are directories named like +# commands in you current directory. But it makes adding filenames so much +# easier. I wished I knew a way to only active it for those parts that are +# filenames. +complete -o filenames -F _reprepro reprepro + +_changestool() +{ + local cur prev commands options noargoptions i j cmd ignores wascreate changesfilename + + COMPREPLY=() + + ignores=' notyetimplemented ' + noargoptions='--help --create' + options='--ignore --searchpath' + wascreate=no + + i=1 + prev="" + while [[ $i -lt $COMP_CWORD ]] ; do + cur=${COMP_WORDS[i]} + prev="" + case "$cur" in + --*=*) + i=$((i+1)) + ;; + -i|--ignore|--unignore|-s|--searchpath) + prev="$cur" + i=$((i+2)) + ;; + --create|-c) + i=$((i+1)) + wascreate=yes + ;; + --*|-*) + i=$((i+1)) + ;; + *) + break + ;; + esac + done + cur=${COMP_WORDS[COMP_CWORD]} + if [[ $i -gt $COMP_CWORD && -n "$prev" ]]; then + case "$prev" in + -i|--ignore|--unignore) + COMPREPLY=( $( compgen -W "$ignores" -- $cur ) ) + return 0 + ;; + -s|--searchpath) + COMPREPLY=( $( compgen -d -- $cur ) ) + return 0 + ;; + esac + fi + + if [[ $i -ge $COMP_CWORD ]] ; then + # No changes filename yet specified: + commands='addrawfile adddsc adddeb add includeallsources setdistribution updatechecksums verify' + + if [[ "$cur" == -* ]]; then + case "$cur" in + *) + COMPREPLY=( $( compgen -W "$options $noargoptions" -- $cur ) ) + ;; + esac + return 0 + fi + if [ "$wascreate" = "yes" ] ; then + _filedir + else + _filedir changes + fi + return 0 + fi + changesfilename=${COMP_WORDS[i]} + i=$((i+1)) + if [[ $i -ge $COMP_CWORD ]] ; then + # No command yet specified: + commands='addrawfile adddsc adddeb add includeallsources setdistribution updatechecksums verify' + # todo: restrict to add commands when --create and file not yet existing? + COMPREPLY=( $( compgen -W "$commands" -- $cur ) ) + return 0 + fi + cmd=${COMP_WORDS[i]} + + case "$cmd" in +# with searchpath it should also list the files available there, +# but I know no easy way to get that done... + addrawfile) + _filedir + return 0 + ;; + adddsc) + _filedir dsc + return 0 + ;; + adddeb) + _filedir deb + return 0 + ;; + adddeb) + _filedir + return 0 + ;; + includeallsources) + prev="$(grep '^ [0-9a-f]\{32\} \+[0-9]\+ \+[a-zA-Z/0-9.:-]\+ \+[a-zA-Z/0-9.:-]\+ \+[^ ]\+\.dsc$' -- "$changesfilename" | sed -e 's/^ [0-9a-f]\+ \+[0-9]\+ \+[^ ]\+ \+[^ ]\+ \+//')" + j=0 + options=() + for i in $prev ; do + if [ -f "$i" ] ; then + options=(${options[@]:-} $(grep '^ [0-9a-f]\{32\} \+[0-9]\+ \+[^ ]\+$' -- "$i" | sed -e 's/^ [0-9a-f]\+ \+[0-9]\+ \+//') ) + elif [ -f "$(dirname $changesfilename)/$i" ] ; then + options=(${options[@]:-} $(grep '^ [0-9a-f]\{32\} \+[0-9]\+ \+[^ ]\+$' -- "$(dirname $changesfilename)/$i" | sed -e 's/^ [0-9a-f]\+ \+[0-9]\+ \+//') ) + else + cmd="missing" + fi + done + COMPREPLY=( $( compgen -W "${options[@]}" -- $cur ) ) + # if some .dsc cannot be found or read, offer everything additionally + if [ "$cmd" = "missing" ] ; then + _filedir + fi + return 0 + ;; + setdistribution) + COMPREPLY=( $( compgen -W "unstable testing stable sarge etch lenny sid backports local" -- $cur ) ) + return 0 + ;; + updatechecksums) + options="$(grep '^ [0-9a-f]\{32\} \+[0-9]\+ \+[a-zA-Z/0-9.:-]\+ \+[a-zA-Z/0-9.:-]\+ \+[^ ]\+$' -- "$changesfilename" | sed -e 's/^ [0-9a-f]\+ \+[0-9]\+ \+[^ ]\+ \+[^ ]\+ \+//')" + if [ -n "$options" ] ; then + COMPREPLY=( $( compgen -W "$options" -- $cur ) ) + else + _filedir + fi + return 0 + ;; + verify) + return 0 + ;; + esac + COMPREPLY=( $( compgen -f -- $cur ) ) + return 0 +} +# same problem as above with -o filenames, +# but I guess still better than without... +complete -o filenames -F _changestool changestool + diff --git a/docs/reprepro.zsh_completion b/docs/reprepro.zsh_completion new file mode 100644 index 0000000..44a132b --- /dev/null +++ b/docs/reprepro.zsh_completion @@ -0,0 +1,554 @@ +#compdef reprepro + +# This is a zsh completion script for reprepro. +# To make use of it make sure it is stored as _reprepro in your +# zsh's fpath (like /usr/local/share/zsh/site-functions/). +# +# to install as user: +# +# mkdir ~/.zshfiles +# cp reprepro.zsh_completion ~/.zshfiles/_reprepro +# echo 'fpath=(~/.zshfiles $fpath)' >> ~/.zshrc +# echo 'autoload -U ~/.zshfiles*(:t)' >> ~/.zshrc +# +# make sure compinit is called after those lines in .zshrc + +local context state line confdir distfile incomingfile incomingdir outdir basedir confdirset basedirset +typeset -A opt_args +local -a codenames architectures list commands hiddencommands + +function _reprepro_calcbasedir () +{ + if [[ -n "$opt_args[-b]" ]]; then + basedir=${opt_args[-b]} + basedirset=true + elif [[ -n "$opt_args[--basedir]" ]]; then + basedir=${opt_args[--basedir]} + basedirset=true + elif [[ -n "$REPREPRO_BASE_DIR" ]]; then + basedir=${REPREPRO_BASE_DIR} + basedirset=true + else + basedir=$PWD + basedirset=false + fi + if [[ -n "$opt_args[--confdir]" ]]; then + confdir=${opt_args[--confdir]} + elif [[ -n "$REPREPRO_CONFIG_DIR" ]]; then + confdir=${REPREPRO_CONFIG_DIR} + else + confdir=$basedir/conf + fi + if [[ -e "$confdir/options" ]] ; then + if [ "$basedirset" != "true" ] && grep -q '^basedir ' -- "$confdir/options" 2>/dev/null ; then + basedir="$(grep '^basedir ' -- "$confdir/options" 2>/dev/null | sed -e 's/^basedir *//')" + fi + fi +} +function _reprepro_filekeys () +{ + _reprepro_calcbasedir + if [[ -n "$opt_args[--outdir]" ]]; then + outdir=${opt_args[--outdir]} + else + outdir=$basedir + fi + list=( $outdir ) + _files -W list +} + +function _reprepro_calcconfdir () +{ + if [[ -n "$opt_args[--confdir]" ]]; then + confdir=${opt_args[--confdir]} + confdirset=direct + elif [[ -n "$REPREPRO_CONFIG_DIR" ]]; then + confdir=${REPREPRO_CONFIG_DIR} + confdirset=direct + elif [[ -n "$opt_args[-b]" ]]; then + confdir=${opt_args[-b]}/conf + confdirset=basedir + basedirset=true + elif [[ -n "$opt_args[--basedir]" ]]; then + confdir=${opt_args[--basedir]}/conf + confdirset=basedir + basedirset=true + elif [[ -n "$REPREPRO_BASE_DIR" ]]; then + confdir=${REPREPRO_BASE_DIR}/conf + confdirset=basedir + basedirset=true + else + confdir=$PWD/conf + confdirset=default + basedirset=false + fi + if [ "$confdirset" != "direct" ] && [[ -e "$confdir/options" ]] ; then + if grep -q '^confdir ' -- "$confdir/options" 2>/dev/null ; then + confdir="$(grep '^confdir ' -- "$confdir/options" 2>/dev/null | sed -e 's/^confdir *//')" + elif [ "$basedirset" = "false" ] \ + && grep -q '^basedir ' -- "$confdir/options" 2>/dev/null ; then + confdir="$(grep '^basedir ' -- "$confdir/options" 2>/dev/null | sed -e 's/^basedir *//')/conf" + fi + fi +} + +function _reprepro_finddistributions () +{ + _reprepro_calcconfdir + distfile="$confdir"/distributions + test -e "$distfile" +} + +function _reprepro_findincoming () +{ + _reprepro_calcconfdir + incomingfile="$confdir"/incoming + test -e "$incomingfile" +} + +function _reprepro_grepdistfile () +{ + _reprepro_finddistributions && + if test -d "$distfile" ; then + sed -n -e 's#^'"$1"': \(.*\)#\1#p' "$distfile"/*.conf + else + sed -n -e 's#^'"$1"': \(.*\)#\1#p' "$distfile" + fi +} + +function _reprepro_architectures () +{ + architectures=($(_reprepro_grepdistfile '[Aa][Rr][Cc][Hh][Ii][Tt][Ee][Cc][Tt][Uu][Rr][Ee][Ss]')) \ + || architectures=(i386 m68k sparc alpha powerpc arm mips mipsel hppa ia64 s390 amd64 ppc64 sh armeb m32r hurd-i386 netbsd-i386 netbsd-alpha kfreebsd-gnu) + _wanted -V 'architectures' expl 'architecture' compadd -a architectures +} + +function _reprepro_components () +{ + components=($(_reprepro_grepdistfile '[Cc][Oo][Mm][Pp][Oo][Nn][Ee][Nn][Tt][Ss]')) \ + || components=(main contrib non-free bad) + _wanted -V 'components' expl 'component' compadd -a components +} +function _reprepro_codenames () { + codenames=($(_reprepro_grepdistfile '[Cc][Oo][Dd][Ee][Nn][Aa][Mm][Ee]')) \ + || codenames=(sid lenny etch sarge unstable testing stable local) + _wanted -V 'codenames' expl 'codename' compadd -a codenames +} +function _reprepro_identifiers () { + _reprepro_finddistributions \ + && list=($(if test -d "$distfile" ; then set -- "$distfile"/*.conf ; + else set -- "$distfile" ; fi && awk ' + /^$/ {for(a=2;a<=acount;a++){ + for(c=2;c<=ccount;c++){ + print codename "|" components[c] "|" architectures[a] + } + if( architectures[a] != "source" ) { + for(c=2;c<=uccount;c++){ + print "u|" codename "|" ucomponents[c] "|" architectures[a] + } + } + }; acount=0;ccount=0;ucount=0} + /^[Cc][Oo][Mm][Pp][Oo][Nn][Ee][Nn][Tt][Ss]: / {ccount = split($0,components); next} + /^[Uu][Dd][Ee][Bb][Cc][Oo][Mm][Pp][Oo][Nn][Ee][Nn][Tt][Ss]: / {uccount = split($0,ucomponents); next} + /^[Aa][Rr][Cc][Hh][Ii][Tt][Ee][Cc][Tt][Uu][Rr][Ee][Ss]: / {acount = split($0,architectures); next} + /^[Cc][Oo][Dd][Ee][Nn][Aa][Mm][Ee]: / {codename = $2; next} + END {for(a=2;a<=acount;a++){ + for(c=2;c<=ccount;c++){ + print codename "|" components[c] "|" architectures[a] + } + if( architectures[a] != "source" ) { + for(c=2;c<=uccount;c++){ + print "u|" codename "|" ucomponents[c] "|" architectures[a] + } + } + }; acount=0;ccount=0;ucount=0} + {next} + ' "$@" )) \ + || list=(identifier) + _wanted -V 'identifiers' expl 'identifier' compadd -a list +} +function _reprepro_incomings () { + _reprepro_findincoming \ + && list=($(if test -d "$incomingfile" ; then set -- "$incomingfile"/*.conf ; else set -- "$incomingfile" ; fi && awk '/^[Nn][Aa][Mm][Ee]: / {print $2}' "$@")) \ + || list=(rule-name) + _wanted -V 'rule names' expl 'rule name' compadd -a list +} +function _reprepro_incomingdir () { + local rulename=$1 + shift + _reprepro_findincoming \ + && incomingdir=($(if test -d "$incomingfile" ; then set -- "$incomingfile"/*.conf ; else set -- "$incomingfile" ; fi && awk ' + /^[Ii][Nn][Cc][Oo][Mm][Ii][Nn][Gg][Dd][Ii][Rr]: / {dir=$2; next} + /^[Nn][Aa][Mm][Ee]: / {name=$2; next} + /^$/ { if( name="'"$rulename"'" ) { print dir } ; next } + END { if( name="'"$rulename"'" ) { print dir }} + {next} + ' "$@")) + # needs to be an array, as it might not be absolute... + list=( $incomingdir ) +} +function _reprepro_package_names () { +#todo get package names?... + _wanted -V 'package names' expl 'package name' compadd name +} +function _reprepro_source_package_names () { +#todo get package names?... + _wanted -V 'source package names' expl 'source package name' compadd name +} + +commands=( + build-needing:"list packages likely needing a build" + check:"check if all references are correct" + checkpool:"check if all files are still there and correct" + checkpull:"check what would be pulled" + checkupdate:"check what would be updated" + cleanlists:"clean unneeded downloaded list files" + clearvanished:"remove empty databases" + collectnewchecksums:"calculate missing file hashes" + copy:"copy a package from one distribution to another" + copyfilter:"copy packages from one distribution to another" + copymatched:"copy packages from one distribution to another" + copysrc:"copy packages belonging to a specific source from one distribution to another" + createsymlinks:"create suite symlinks" + deleteunreferenced:"delete files without reference" + dumpreferences:"dump reference information" + dumppull:"dump what would be pulled" + dumptracks:"dump tracking information" + dumpupdate:"dump what would be updated" + dumpunreferenced:"dump files without reference (i.e. unneded)" + export:"export index files" + forcerepairdescriptions:"forcefully readd lost long descriptions from .deb file" + flood:"copy architecture all packages within a distribution" + generatefilelists:"pre-prepare filelist caches for all binary packages" + gensnapshot:"generate a snapshot" + includedeb:"include a .deb file" + includedsc:"include a .dsc file" + include:"include a .changes file" + includeudeb:"include a .udeb file" + listfilter:"list packages matching filter" + listmatched:"list packages matching filter" + list:"list packages" + ls:"list versions of package" + lsbycomponent:"list versions of package (grouped by component)" + predelete:"delete what would be removed or superseded by an update" + processincoming:"process files from an incoming directory" + pull:"update from another local distribtuion" + removealltracks:"remove tracking information" + remove:"remove packages" + removefilter:"remove packages matching a formula" + removematched:"remove packages matching a glob" + removesrc:"remove packages belonging to a source package" + removesrcs:"remove packages belonging to names source packages" + removetrack:"remove a single tracking data" + reoverride:"apply override information to already existing packages" + repairdescriptions:"readd lost long descriptions from .deb file" + reportcruft:"report source packages without binaries and vice versa" + rereference:"recreate references" + rerunnotifiers:"call notificators as if all packages were just included" + restore:"restore a package from a distribution's snapshot" + restorefilter:"restore packages matching a filter from a snapshot" + restorematched:"restore packages matching a glob from a snapshot" + restoresrc:"restore packages belonging to a specific source from a snapshot" + retrack:"refresh tracking information" + sourcemissing:"list binary packages with no source package" + tidytracks:"look for files referened by tracks but no longer needed" + translatefilelists:"translate pre-3.0.0 contents.cache.db into new format" + translatelegacychecksums:"get rid of obsolete files.db" + unreferencesnapshot:"no longer mark files used by an snapshot" + unusedsources:"list source packages with no binary packages" + update:"update from external source" + ) +hiddencommands=( + __dumpuncompressors:"list what external uncompressors are available" + __extractcontrol:"extract the control file from a .deb file" + __extractfilelist:"extract the filelist from a .deb file" + __extractsourcesection:"extract source and priority from a .dsc" + __uncompress:"uncompress a file" + _addchecksums:"add checksums to database" + _addmd5sums:"add checksums to database" + _addreference:"mark a filekey needed by an identifier" + _addreferences:"mark multiple filekeys needed by an identifier" + _detect:"look if the file belonging to a filekey exists and add to the database." + _dumpcontents:"output contents of a part of the repository" + _fakeemptyfilelist:"create an empty fake filelist cache item for a filekey" + _forget:"forget a file specified by filekey." + _listchecksums:"print a list of filekeys and their checksums" + _listcodenames:"list configured codenames" + _listconfidentifiers:"list parts of the repository in the configuration" + _listdbidentifiers:"list parts of the repository in the database" + _listmd5sums:"print a list of filekeys and their md5 hashes" + _removereference:"manuall remove a reference" + _removereferences:"remove all references by an identifier" + ) + +_arguments \ + '*'{-v,-V,--verbose}'[be more verbose]' \ + '*--silent[be less verbose]' \ + '*--delete[Delete files after inclusion]' \ + '(-b --basedir)'{-b,--basedir}'[Base drectory]:basedir:_files -/' \ + '--outdir[Directory where pool and dist are in]:out dir:_files -/' \ + '--confdir[Directory where config files are]:config dir:_files -/' \ + '--distdir[Directory where index files will be exported to]:dist dir:_files -/' \ + '--logdir[Directory where log files will be generated]:log dir:_files -/' \ + '--morguedir[Directory where files removed from the pool are stored]:morgue dir:_files -/' \ + '--dbdir[Directory where the database is stored]:database dir:_files -/' \ + '--listdir[Directory where downloaded index files will be stored]:list dir:_files -/' \ + '--methoddir[Directory to search apt methods in]:method dir:_files -/' \ + '(-C --component)'{-C,--component}'[Override component]:component:{_reprepro_components}' \ + '(-A --architecture)'{-A,--architecture}'[Limit to a specific architecture]:architecture:{_reprepro_architectures}' \ + '(-T --type)'{-T,--type}'[Limit to a specific type]:file type:(dsc deb udeb)' \ + '(-S --section)'{-S,--section}'[Override section]:section:(admin base comm contrib devel doc editors electronics embedded games gnome graphics hamradio interpreters kde libs libdevel mail math misc net news non-free oldlibs otherosfs perl python science shells sound tex text utils web x11 contrib/admin contrib/base contrib/comm contrib/contrib contrib/devel contrib/doc contrib/editors contrib/electronics contrib/embedded contrib/games contrib/gnome contrib/graphics contrib/hamradio contrib/interpreters contrib/kde contrib/libs contrib/libdevel contrib/mail contrib/math contrib/misc contrib/net contrib/news contrib/non-free contrib/oldlibs contrib/otherosfs contrib/perl contrib/python contrib/science contrib/shells contrib/sound contrib/tex contrib/text contrib/utils contrib/web contrib/x11 non-free/admin non-free/base non-free/comm non-free/contrib non-free/devel non-free/doc non-free/editors non-free/electronics non-free/embedded non-free/games non-free/gnome non-free/graphics non-free/hamradio non-free/interpreters non-free/kde non-free/libs non-free/libdevel non-free/mail non-free/math non-free/misc non-free/net non-free/news non-free/non-free non-free/oldlibs non-free/otherosfs non-free/perl non-free/python non-free/science non-free/shells non-free/sound non-free/tex non-free/text non-free/utils non-free/web non-free/x11)' \ + '(-P --priority)'{-P,--priority}'[Override priority]:priority:(required important standard optional extra)' \ + '--export=[]:when:(silent-never never changed lookedat force)' \ + '*--ignore=[Do ignore errors of some type]:error type:((\ + ignore\:"ignore unknown ignore tags"\ + flatandnonflat\:"ignore warnings about flat and non-flat distribution"\ + forbiddenchar\:"allow more 7bit characters for names and versions"\ + 8bit\:"allow 8 bit characters"\ + emptyfilenamepart\:"allow strings used to construct filenames to be empty"\ + spaceonlyline\:"do not warn about lines containing only spaces"\ + malformedchunk\:"ignore lines without colons"\ + unknownfield\:"ignore unknown fields"\ + wrongdistribution\:"put .changes files in distributed they were not made for"\ + wrongarchitecture\:"do not warn about wrong Architecture fields in downloaded Packages files"\ + missingfield\:"allow missing fields"\ + brokenold\:"ignore broken packages in database"\ + brokenversioncmp\:"ignore versions not parseable"\ + extension\:"ignore unexpected suffixes of files"\ + unusedarch\:"allow changes files to list architectures not used"\ + unusedoption\:"ignore command line options not used by an action"\ + undefinedtarget\:"allow unspecified package databases"\ + undefinedtracking\:"allow unspecified tracking databases"\ + surprisingarch\:"do not protest when a changes file does not list a architecture it has files for"\ + surprisingbinary\:"do not demand a .changes Binaries header to list all binaries"\ + wrongsourceversion\:"do not demand coherent source versions in a .changes"\ + wrongversion\:"do not demand coherent version of source packages in a .changes"\ + dscinbinnmu\:"do not reject source files in what looks like a binMNU"\ + brokensignatures\:"ignore corrupted signatures if there is a valid one"\ + uploaders\:"allow even when forbidden by uploaders file"\ + missingfile\:"include commands search harder for missing files like .orig.tar.gz"\ + expiredkey\:"allow signatures with expired keys"\ + expiredsignature\:"allow expired signatures"\ + revokedkey\:"allow signatures with revoked keys"\ + oldfile\:"silence warnings about strange old files in dists"\ + longkeyid\:"do not warn about keyid in uploaders files gpgme might not accept"\ + ))' \ + '*--unignore=[Do not ignore errors of type]:error type:( + ignore flatandnonflat forbiddenchar 8bit emptyfilenamepart\ + spaceonlyline malformedchunk unknownfield unusedoption\ + wrongdistribution missingfield brokenold brokenversioncmp\ + extension unusedarch surprisingarch surprisingbinary\ + wrongsourceversion wrongversion brokensignatures\ + missingfile uploaders undefinedtarget undefinedtracking\ + expiredkey expiredsignature revokedkey wrongarchitecture)' \ + '--waitforlock=[Time to wait if database is locked]:count:(0 3600)' \ + '--spacecheck[Mode for calculating free space before downloading packages]:behavior:(full none)' \ + '--dbsafetymargin[Safety margin for the partition with the database]:bytes count:' \ + '--safetymargin[Safety margin per partition]:bytes count:' \ + '--gunzip[external Program to extract .gz files]:gunzip binary:_files' \ + '--bunzip2[external Program to extract .bz2 files]:bunzip binary:_files' \ + '--unlzma[external Program to extract .lzma files]:unlzma binary:_files' \ + '--unxz[external Program to extract .xz files]:unxz binary:_files' \ + '--lunzip[external Program to extract .lz files]:lunzip binary:_files' \ + '--list-format[Format for list output]:listfilter format:' \ + '--list-skip[Number of packages to skip in list output]:list skip:' \ + '--list-max[Maximum number of packages in list output]:list max:' \ + '(--nonothingiserror)--nothingiserror[Return error code when nothing was done]' \ + '(--listsdownload --nonolistsdownload)--nolistsdownload[Do not download Release nor index files]' \ + '(--nokeepunneededlists)--keepunneededlists[Do not delete list/ files that are no longer needed]' \ + '(--nokeepunreferencedfiles)--keepunreferencedfiles[Do not delete files that are no longer used]' \ + '(--nokeepunusednewfiles)--keepunusednewfiles[Do not delete newly added files that later were found to not be used]' \ + '(--nokeepdirectories)--keepdirectories[Do not remove directories when they get empty]' \ + '(--nokeeptemporaries)--keeptemporaries[When exporting fail do not remove temporary files]' \ + '(--noask-passphrase)--ask-passphrase[Ask for passphrases (insecure)]' \ + '(--nonoskipold --skipold)--noskipold[Do not ignore parts where no new index file is available]' \ + '(--guessgpgtty --nonoguessgpgtty)--noguessgpgtty[Do not set GPG_TTY variable even when unset and stdin is a tty]' \ + ':reprepro command:->commands' \ + '2::arguments:->first' \ + '3::arguments:->second' \ + '4::arguments:->third' \ + '*::arguments:->argument' && return 0 + +case "$state" in + (commands) + if [[ -prefix _* ]] ; then + _describe "reprepro command" hiddencommands + else + _describe "reprepro command" commands + fi + ;; + + (first argument|second argument|third argument|argument) + case "$words[1]" in + (export|update|checkupdate|predelete|pull|checkpull|check|reoverride|repairdescriptions|forcerepairdescriptions|rereference|dumptracks|retrack|removealltracks|tidytracks|dumppull|dumpupdate|rerunnotifiers|unusedsources|sourcemissing|reportcruft) + _reprepro_codenames + ;; + (checkpool) + if [[ "$state" = "first argument" ]] ; then + _wanted -V 'modifiers' expl 'modifier' compadd fast + fi + ;; + + (cleanlists|clearvanished|dumpreferences|dumpunreferened|deleteunreferenced|_listmd5sums|_listchecksums|_addmd5sums|_addchecksums|__dumpuncompressors|transatelegacychecksums|_listcodenames) + ;; + (_dumpcontents|_removereferences) + if [[ "$state" = "first argument" ]] ; then + _reprepro_identifiers + fi + ;; + (_removereference) + if [[ "$state" = "first argument" ]] ; then + _reprepro_identifiers + elif [[ "$state" = "second argument" ]] ; then + _reprepro_filekeys + fi + ;; + (list|listfilter|listmatched) + if [[ "$state" = "first argument" ]] ; then + _reprepro_codenames + fi + ;; + (remove) + if [[ "$state" = "first argument" ]] ; then + _reprepro_codenames + else + _reprepro_package_names "$words[2]" + fi + ;; + # removesrcs might be improveable... + (removesrc|removesrcs) + if [[ "$state" = "first argument" ]] ; then + _reprepro_codenames + else + _reprepro_source_package_names "$words[2]" + fi + ;; + (removefilter|removematched) + if [[ "$state" = "first argument" ]] ; then + _reprepro_codenames + fi + ;; + (gensnapshot|unreferencesnapshot) + # TODO: for unreferencesnapshot get instead a list of existing ones + if [[ "$state" = "first argument" ]] ; then + _reprepro_codenames + elif [[ "$state" = "second argument" ]] ; then + _wanted -V 'snapshot names' expl 'snapshot name' compadd $(date -I) + fi + ;; + (removetrack) + if [[ "$state" = "first argument" ]] ; then + _reprepro_codenames + elif [[ "$state" = "second argument" ]] ; then + _reprepro_source_package_names "$words[2]" + elif [[ "$state" = "third argument" ]] ; then +#and version... + fi + ;; + (includedeb) + if [[ "$state" = "first argument" ]] ; then + _reprepro_codenames + elif [[ "$state" = "second argument" ]] ; then + _files -g "*.deb" + fi + ;; + (includedsc) + if [[ "$state" = "first argument" ]] ; then + _reprepro_codenames + elif [[ "$state" = "second argument" ]] ; then + _files -g "*.dsc" + fi + ;; + (__extractsourcesection) + if [[ "$state" = "first argument" ]] ; then + _files -g "*.dsc" + fi + ;; + (copy|copysrc|copyfilter|copymatched) + if [[ "$state" = "first argument" ]] ; then + _reprepro_codenames + elif [[ "$state" = "second argument" ]] ; then + _reprepro_codenames + fi + ;; + (restore|restoresrc|restorefilter|restorematched) + if [[ "$state" = "first argument" ]] ; then + _reprepro_codenames +# TODO: +# elif [[ "$state" = "second argument" ]] ; then +# _reprepro_codenames + fi + ;; + (include) + if [[ "$state" = "first argument" ]] ; then + _reprepro_codenames + elif [[ "$state" = "second argument" ]] ; then + _files -g "*.changes" + fi + ;; + (__extractfilelist|__extractcontrol) + _files -g "*.deb" + ;; + (processincoming) + if [[ "$state" = "first argument" ]] ; then + _reprepro_incomings + elif [[ "$state" = "second argument" ]] ; then + _reprepro_incomingdir "$words[2]" \ + && _files -g "*.changes" -W list \ + || _files -g "*.changes" + fi + ;; + (_detect|_forget) + _reprepro_filekeys + ;; + (_fakeemptyfilelist) + if [[ "$state" = "first argument" ]] ; then + _reprepro_filekeys + fi + ;; + (_addreference) + if [[ "$state" = "first argument" ]] ; then + _reprepro_filekeys + elif [[ "$state" = "second argument" ]] ; then + _reprepro_identifiers + fi + ;; + (_addreferences) + if [[ "$state" = "first argument" ]] ; then + _reprepro_identifiers + else + _reprepro_filekeys + fi + ;; + (__uncompress) + if [[ "$state" = "first argument" ]] ; then + uncompressions=(.gz .bz2 .lzma .xz .lz) + _wanted -V 'uncompressions' expl 'uncompression' compadd -a uncompressions + elif [[ "$state" = "second argument" ]] ; then + _files + elif [[ "$state" = "third argument" ]] ; then + _files + fi + ;; + (build-needing) + if [[ "$state" = "first argument" ]] ; then + _reprepro_codenames + elif [[ "$state" = "second argument" ]] ; then + _reprepro_architectures +##TODO elif [[ "$state" = "third argument" ]] ; then +##TODO _reprepro_glob + fi + ;; + (flood) + if [[ "$state" = "first argument" ]] ; then + _reprepro_codenames + elif [[ "$state" = "second argument" ]] ; then + _reprepro_architectures + fi + ;; + (*) + _files + ;; + esac + ;; +esac diff --git a/docs/rredtool.1 b/docs/rredtool.1 new file mode 100644 index 0000000..f217128 --- /dev/null +++ b/docs/rredtool.1 @@ -0,0 +1,90 @@ +.TH RREDTOOL 1 "2009-11-12" "reprepro" REPREPRO +.SH NAME +rredtool \- merge or apply a very restricted subset of ed patches +.SH SYNOPSIS +.B rredtool \-\-help + +.B rredtool +[ +\fIoptions\fP +] +.B \-\-merge +.I patches... + +.B rredtool +[ +\fIoptions\fP +] +.B \-\-patch +.IR file-to-patch " " patches... + +.B rredtool +.IR directory " " newfile " " oldfile " " mode +.SH DESCRIPTION +rredtool is a tool to handle a subset of ed patches in a safe way. +It is especially targeted at ed patches as used in Packages.diff +and Sources.diff. +Is also has a mode supposed to be called from reprepro as Index Hook +to generate and update a \fBPackages.diff/Index\fP file. +.SH "MODI" +One of the following has to be given, so that rredtool know that to +do. +.TP +.B \-\-version +Print the version of this tool +(or rather the version of reprepro which it is coming with). +.TP +.B \-\-help +Print a short overview of the modi. +.TP +.B \-\-patch +The first argument of rredtool is the file to patch, +the other arguments are ed patches to apply on this one. +.TP +.B \-\-merge +The arguments are treated as ed patches, which are merged into +a single one. +.TP +.BR \-\-reprepro\-hook " (or no other mode flag) +Act as reprepro index hook to manage a \fBPackages.diff/index\fP file. +That means it expects to get exactly 4 arguments +and writes the names of files to place into filedescriptor 3. + +If neither \-\-patch nor \-\-merge is given, +this mode is used, so you can just put + + \fBDebIndices: Packages Release . .gz /usr/bin/rredtool\fP + +into reprepro's \fBconf/distributions\fP file to have a Packages.diff +directory generated. +(Note that you have to generate an uncompressed file (the single dot). +You will need to have patch, gzip and gunzip available in your path.) + +.SH "OPTIONS" +.TP +.B \-\-debug +Print intermediate results or other details that might be interesting +when trying to track down bugs in rredtool but not intresting otherwise. +.TP +.B \-\-max\-patch\-count=\fIcount\fP +When generating a \fIPackages\fP\fB.diff/Index\fP file, +put at most \fIcount\fP patches in it +(not counting possible apt workaround patches). +.TP +.BR \-o | \-\-output +Not yet implemented. +.SH "ENVIRONMENT" +.TP +.BR TMPDIR ", " TEMPDIR +temporary files are created in $\fITEMPDIR\fP if set, +otherwise in $\fITMPDIR\fP if set, otherwise in \fB/tmp/\fP. +.SH "REPORTING BUGS" +Report bugs or wishlist requests the Debian BTS +(e.g. by using \fBreportbug reperepro\fP) +or directly to <brlink@debian.org>. +.br +.SH COPYRIGHT +Copyright \(co 2009 Bernhard R. Link +.br +This is free software; see the source for copying conditions. There is NO +warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. diff --git a/docs/sftp.py b/docs/sftp.py new file mode 100755 index 0000000..072dcf1 --- /dev/null +++ b/docs/sftp.py @@ -0,0 +1,886 @@ +# -*- coding: utf-8 -*- +# +# Copyright 2013 Bernhard R. Link <brlink@debian.org> +# +# Permission is hereby granted, free of charge, to any person obtaining a +# copy of this software and associated documentation files (the "Software"), +# to deal in the Software without restriction, including without limitation +# the rights to use, copy, modify, merge, publish, distribute, sublicense, +# and/or sell copies of the Software, and to permit persons to whom the +# Software is furnished to do so, subject to the following conditions: +# +# The above copyright notice and this permission notice shall be included in +# all copies or substantial portions of the Software. +# +# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL +# SOFTWARE IN THE PUBLIC INTEREST, INC. BE LIABLE FOR ANY CLAIM, DAMAGES OR +# OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, +# ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER +# DEALINGS IN THE SOFTWARE. +# + +""" +This is a sftp module to be used in reprepro's outsftphook example. +Like the sftp binary it calls ssh to do the connection in a secure +way and then speaks the sftp subsystem language over that connection. +""" + + +import os, subprocess, select + +class EnumInternException(Exception): + def __init__(self, v): + super().__init__(v) + self.value = v + +class _EnumType(type): + """ + Metaclass for Enum. Allows one to set values as parameters. + """ + def __new__(cls, name, bases, namespace, **values): + return type.__new__(cls, name, bases, namespace) + def __init__(self, name, bases, namespace, **values): + super().__init__(name, bases, namespace) + if bases: + self._byvalue = dict() + self._byname = dict() + if values: + for k,v in values.items(): + self._create_instance(k, v) + +class Enum(metaclass=_EnumType): + """ + An enum is a class with a fixed set of instances. + Each instance has a name and a integer value. + If a new new instance is to be created, one of those + fix instances is returned instead. + """ + @classmethod + def _create_instance(cls, name, value): + # create a new instance: + result = super(Enum, cls).__new__(cls) + if isinstance(name, str): + result.name = name + else: + result.name = name[0] + result.__name__ = result.name + result.value = value + cls._byvalue[value] = result + if isinstance(name, str): + cls._byname[name] = result + setattr(cls, name, result) + else: + for n in name: + cls._byname[n] = result + setattr(cls, n, result) + return result + def __new__(cls, l): + try: + if isinstance(l, cls): + return l + elif isinstance(l, int): + return cls._byvalue[l] + elif isinstance(l, str): + return cls._byname[l] + else: + raise EnumInternException(repr(l)) + except KeyError: + raise EnumInternException(repr(l)) + def __init__(self, l): + pass + def __int__(self): + return self.value + def __str__(self): + return self.name + def __repr__(self): + return "%s.%s.%s" % (type(self).__module__, type(self).__name__, self.name) + +class _BitmaskType(type): + """ + Metaclass for Bitmask types. Allows one to set values as parameters. + """ + @classmethod + def __prepare__(cls, name, bases, **values): + namespace = type.__prepare__(cls, name, bases) + if values: + flagcls = _EnumType.__new__(type, "flags of " + name, (Enum,), dict()) + flagcls._byvalue = dict() + flagcls._byname = dict() + namespace["_Values"] = flagcls + for (k,v) in values.items(): + if isinstance(v, int): + e = flagcls._create_instance(k, v) + e.mask = v + else: + e = flagcls._create_instance(k, v[0]) + e.mask = v[1] + namespace[k] = e + return namespace + def __new__(cls, name, bases, namespace, **values): + return type.__new__(cls, name, bases, namespace) + def __init__(self, name, bases, namespace, **values): + return super().__init__(name, bases, namespace) + +class Bitmask(set, metaclass=_BitmaskType): + def __init__(self, l): + if isinstance(l, int): + super().__init__([i + for (k,i) + in self._Values._byvalue.items() + if (l & i.mask) == k]) + if l != int(self): + raise Exception("Unrepresentable number %d (got parsed as %s = %d)" % + (l, str(self), int(self))) + elif isinstance(l, str): + try: + super().__init__([self._Values(i) + for i + in l.split("|")]) + # test for inconsistencies: + type(self)(int(self)) + except EnumInternException as e: + raise Exception("Invalid value '%s' in value '%s' for %s" % + (e.value, str(l), type(self).__name__)) + else: + try: + super().__init__([self._Values(i) for i in l]) + # test for inconsistencies: + type(self)(int(self)) + except EnumInternException as e: + raise Exception("Invalid value '%s' in value '%s' for %s" % + (e.value, str(l), type(self).__name__)) + def __int__(self): + v = 0 + for i in self: + v = v | int(i) + return v + def __str__(self): + return "|".join([str(i) for i in self]) + +class SSH_FILEXFER(Bitmask, ATTR_SIZE = 0x00000001, + ATTR_UIDGID = 0x00000002, + ATTR_PERMISSIONS = 0x00000004, + ATTR_ACMODTIME = 0x00000008, + ATTR_EXTENDED = 0x80000000): + pass + +def ssh_data(b): + return len(b).to_bytes(4, byteorder='big') + b +def ssh_string(s): + b = str(s).encode(encoding='utf-8') + return len(b).to_bytes(4, byteorder='big') + b +def ssh_u8(i): + return int(i).to_bytes(1, byteorder='big') +def ssh_u32(i): + return int(i).to_bytes(4, byteorder='big') +def ssh_u64(i): + return int(i).to_bytes(8, byteorder='big') +def ssh_attrs(**opts): + flags = SSH_FILEXFER(0) + extended = [] + for key in opts: + if key == 'size': + flags.add(SSH_FILEXFER.ATTR_SIZE) + elif key == 'uid' or key == 'gid': + flags.add(SSH_FILEXFER.ATTR_UIDGID) + elif key == 'permissions': + flags.add(SSH_FILEXFER.ATTR_PERMISSIONS) + elif key == 'atime' or key == 'mtime': + flags.add(SSH_FILEXFER.ATTR_ACMODTIME) + elif '@' in key: + extended.add(opts[key]) + else: + raise SftpException("Unsupported file attribute type %s" % repr(key)) + if extended: + flags.add(SSH_FILEXFER.ATTR_EXTENDED) + b = ssh_u32(int(flags)) + if SSH_FILEXFER.ATTR_SIZE in flags: + b = b + ssh_u64(opts['size']) + if SSH_FILEXFER.ATTR_UIDGID in flags: + b = b + ssh_u32(opts['uid']) + b = b + ssh_u32(opts['gid']) + if SSH_FILEXFER.ATTR_PERMISSIONS in flags: + b = b + ssh_u32(opts['permissions']) + if SSH_FILEXFER.ATTR_ACMODTIME in flags: + b = b + ssh_u32(opts['atime']) + b = b + ssh_u32(opts['mtime']) + if SSH_FILEXFER.ATTR_EXTENDED in flags: + b = b + ssh_u32(len(extended)) + for key in extended: + b = b + ssh_string(key) + b = b + ssh_data(opts[key]) + return b + +def ssh_getu32(m): + v = int.from_bytes(m[:4], byteorder='big') + return v, m[4:] +def ssh_getstring(m): + l = int.from_bytes(m[:4], byteorder='big') + return (m[4:4+l].decode(encoding='utf-8'), m[4+l:]) +def ssh_getdata(m): + l = int.from_bytes(m[:4], byteorder='big') + return (m[4:4+l], m[4+l:]) +def ssh_getattrs(m): + attrs = dict() + flags, m = ssh_getu32(m) + flags = SSH_FILEXFER(flags) + if SSH_FILEXFER.ATTR_SIZE in flags: + attrs['size'], m = ssh_getu64(m) + if SSH_FILEXFER.ATTR_UIDGID in flags: + attrs['uid'], m = ssh_getu32(m) + attrs['gid'], m = ssh_getu32(m) + if SSH_FILEXFER.ATTR_PERMISSIONS in flags: + attrs['permissions'], m = ssh_getu32(m) + if SSH_FILEXFER.ATTR_ACMODTIME in flags: + attrs['atime'], m = ssh_getu32(m) + attrs['mtime'], m = ssh_getu32(m) + if SSH_FILEXFER.ATTR_EXTENDED in flags: + count, m = ssh_getu32(m) + while count > 0: + count -= 1 + key, m = ssh_getstring(m) + attrs[key], m = ssh_getdata(m) + return (attrs, m) + +class SftpException(Exception): + pass + +class SftpStrangeException(SftpException): + """Unparseable stuff from server""" + pass + +class SftpUnexpectedAnswerException(SftpStrangeException): + def __init__(self, answer, request): + super().__init__("Unexpected answer '%s' to request '%s'" % + (str(answer), str(request))) + +class SftpTooManyRequestsException(SftpException): + def __init__(self): + super().__init__("Too many concurrent requests (out of request ids)") + +class SftpInternalException(SftpException): + """a programming or programmer mistake""" + pass + +class Request: + def __init__(self, **args): + self.data = args + pass + def __int__(self): + return self.requestid + def __str__(self): + return type(self).__name__ + "(" + " ".join(["%s=%s" % (key, repr(val)) + for (key, val) in self.data.items()]) + ")" + @classmethod + def bin(cls, conn, req, *payload): + s = 5 + for b in payload: + s = s + len(b) + # print("Sending packet of type %d and size %d" % (cls.typeid, s)) + r = ssh_u32(s) + ssh_u8(cls.typeid) + ssh_u32(int(req)) + for b in payload: + r = r + b + return r + def send(self, conn): + conn.requests[int(self)] = self + self.conn = conn + conn.send(self.bin(conn, self, **self.data)) + def done(self): + if self.requestid != None: + del self.conn.requests[self.requestid] + self.requestid = None + +class NameRequest(Request): + """Base class for requests with a single name as argument""" + def __init__(self, name): + super().__init__(name = name) + @classmethod + def bin(cls, conn, req, name): + return super().bin(conn, req, ssh_string(name)) + +class HandleRequest(Request): + """Base class for requests with a single name as argument""" + def __init__(self, handle): + super().__init__(handle = handle) + @classmethod + def bin(cls, conn, req, handle): + return super().bin(conn, req, ssh_data(handle)) + +class NameAttrRequest(Request): + """Base class for requests with a name and attributes as argument""" + def __init__(self, name, **attrs): + super().__init__(name = name, attrs = attrs) + @classmethod + def bin(cls, conn, req, name, attrs): + return super().bin(conn, req, + ssh_string(name), + ssh_attrs(**attrs)) + +class INIT(Request): + typeid = 1 + @classmethod + def bin(cls, conn, version): + # INIT has no request id but instead sends a protocol version + return super().bin(conn, int(version)) + +class SSH_FXF(Bitmask, READ = 0x00000001, + WRITE = 0x00000002, + APPEND = 0x00000004, + CREAT = 0x00000008, + TRUNC = 0x00000010, + EXCL = 0x00000020): + pass + +class OPEN(Request): + typeid = 3 + def __init__(self, name, flags, **attributes): + super().__init__(name = name, flags = SSH_FXF(flags), attrs = attributes) + @classmethod + def bin(cls, conn, req, name, flags, attrs): + return super().bin(conn, req, + ssh_string(name), + ssh_u32(flags), + ssh_attrs(**attrs)) + +class CLOSE(HandleRequest): + typeid = 4 + +class READ(Request): + typeid = 5 + def __init__(self, handle, start, length): + super().__init__(handle = handle, start = start, length = int(length)) + @classmethod + def bin(cls, conn, req, handle, start, length): + return super().bin(conn, req, ssh_data(handle), ssh_u64(start), ssh_u32(length)) + +class WRITE(Request): + typeid = 6 + def __init__(self, handle, start, data): + super().__init__(handle = handle, start = start, data = bytes(data)) + @classmethod + def bin(cls, conn, req, handle, start, data): + return super().bin(conn, req, ssh_data(handle), ssh_u64(start), ssh_data(data)) + +class LSTAT(NameRequest): + typeid = 7 + +class FSTAT(HandleRequest): + typeid = 8 + +class SETSTAT(NameAttrRequest): + typeid = 9 + +class FSETSTAT(Request): + typeid = 10 + def __init__(self, handle, **attrs): + super().__init__(handle = handle, attrs = attrs) + @classmethod + def bin(cls, conn, req, name, attrs): + return super().bin(conn, req, + ssh_data(handle), + ssh_attrs(**attrs)) + +class OPENDIR(NameRequest): + typeid = 11 + +class READDIR(HandleRequest): + typeid = 12 + +class REMOVE(NameRequest): + typeid = 13 + +class MKDIR(NameAttrRequest): + typeid = 14 + +class RMDIR(NameRequest): + typeid = 15 + +class REALPATH(NameRequest): + typeid = 16 + +class STAT(NameRequest): + typeid = 17 + +class SSH_FXF_RENAME(Bitmask, OVERWRITE = 0x00000001, + ATOMIC = 0x00000002, + NATIVE = 0x00000004): + pass + +class RENAME(Request): + typeid = 18 + def __init__(self, src, dst, flags): + if not isinstance(flags, SSH_FXF_RENAME): + flags = SSH_FXF_RENAME(flags) + super().__init__(src = src, dst = dst, flags = flags) + @classmethod + def bin(cls, conn, req, src, dst, flags): + # TODO: Version 3 has no flags (though they do not seem to harm) + return super().bin(conn, req, ssh_string(src), + ssh_string(dst), ssh_u32(flags)) + +class READLINK(NameRequest): + typeid = 19 + +class SYMLINK(Request): + typeid = 20 + def __init__(self, name, dest): + super().__init__(name = name, dest = dest) + @classmethod + def bin(cls, conn, req, name, dest): + # TODO: this is openssh and not the standard (they differ) + return super().bin(conn, req, ssh_string(dest), + ssh_string(name)) + +class EXTENDED(Request): + typeid = 200 + # TODO? + +################ Answers ################ + +class Answer: + def __int__(self): + return self.id + # Fallbacks, can be removed once all are done: + def __init__(self, m): + self.data = m + def __str__(self): + return "%s %s" % (type(self).__name__, repr(self.data)) + +class VERSION(Answer): + id = 2 + +class SSH_FX(Enum, + OK = 0, + EOF = 1, + NO_SUCH_FILE = 2, + PERMISSION_DENIED = 3, + FAILURE = 4, + BAD_MESSAGE = 5, + NO_CONNECTION = 6, + CONNECTION_LOST = 7, + OP_UNSUPPORTED = 8, + INVALID_HANDLE = 9, + NO_SUCH_PATH = 10, + FILE_ALREADY_EXISTS = 11, + WRITE_PROTECT = 12, + NO_MEDIA = 13 +): + pass + +class STATUS(Answer): + id = 101 + def __init__(self, m): + s, m = ssh_getu32(m) + self.status = SSH_FX(s) + self.message, m = ssh_getstring(m) + self.lang, m = ssh_getstring(m) + def __str__(self): + return "STATUS %s: %s[%s]" % ( + str(self.status), + self.message, + self.lang) + +class HANDLE(Answer): + id = 102 + def __init__(self, m): + self.handle, m = ssh_getdata(m) + def __str__(self): + return "HANDLE %s" % repr(self.handle) + +class DATA(Answer): + id = 103 + def __init__(self, m): + self.data, m = ssh_getdata(m) + def __str__(self): + return "DATA %s" % repr(self.data) + +class NAME(Answer): + id = 104 + def __init__(self, m): + count, m = ssh_getu32(m) + self.names = [] + while count > 0: + count -= 1 + filename, m = ssh_getstring(m) + longname, m = ssh_getstring(m) + attrs, m = ssh_getattrs(m) + self.append((filename, longname, attrs)) + + def __str__(self): + return "NAME" + "".join(("%s:%s:%s" % (repr(fn), repr(ln), str(attrs)) + for (fn,ln,attrs) in self.names)) + +class ATTRS(Answer): + id = 105 + def __init__(self, m): + self.attrs, m = ssh_getattrs(m) + + def __str__(self): + return "ATTRS %s" % str(self.attrs) + +class EXTENDED_REPLY(Answer): + id = 201 + # TODO? + +################ Tasks ################ + +class Task: + """A task is everything that sends requests, + receives answers, uses collectors or is + awakened by collectors. + """ + def start(self, connection): + self.connection = connection + def enqueueRequest(self, request): + request.task = self + self.connection.enqueueRequest(request) + def sftpanswer(self, a): + raise SftpInternalException("unimplemented sftpanswer called") + def writeready(self): + raise SftpInternalException("unimplemented writeready called") + def parentinfo(self, command): + raise SftpInternalException("unimplemented parentinfo called") + +class TaskFromGenerator(Task): + """A wrapper around a python corotine (generator)""" + def __init__(self, gen): + super().__init__() + self.gen = gen + def start(self, connection): + super().start(connection) + self.enqueue(next(self.gen)) + def parentinfo(self, command): + self.enqueue(self.gen.send(command)) + def sftpanswer(self, answer): + self.enqueue(self.gen.send(answer)) + def writeready(self): + self.enqueue(self.gen.send('canwrite')) + def __str__(self): + return "Task(by %s)" % self.gen + def enqueue(self, joblist): + if len(joblist) == 0: + return + for job in joblist: + if isinstance(job, Request): + self.enqueueRequest(job) + elif job == 'wantwrite': + self.connection.enqueueTask(self) + elif (isinstance(job, tuple) and len(job) == 2 and + isinstance(job[0], Task)): + if DebugMode.LOCKS in self.debug: + print("parentinfo", job, + **self.debugopts) + job[0].parentinfo(job[1]) + elif (isinstance(job, tuple) and len(job) >= 2 and + issubclass(job[1], Collector)): + self.connection.collect(self, *job) + elif isinstance(job, Task): + self.connection.start(job) + else: + raise SftpInternalException("strange result from generator") + + +class Collector(Task): + """ Collectors collect information from Tasks and send them + triggers at requested events (parent directory created, + another file can be processed, ...) + """ + def childinfo(self, who, command): + raise SftpInternalException("unimplemented parentinfo called") + +class DebugMode(Bitmask, **{ + 'COOKED_IN': 1, + 'COOKED_OUT': 2, + 'RAW_IN_STAT': 4, + 'RAW_OUT_STAT': 8, + 'RAW_IN': 16, + 'RAW_OUT': 32, + 'ENQUEUE': 64, + 'LOCKS': 128, +}): + pass + +class Connection: + def next_request_id(self): + i = self.requestid_try_next + while i in self.requests: + i = (i + 1) % 0x100000000 + if i == self.requestid_try_next: + raise SftpTooManyRequestsException() + self.requestid_try_next = (i + 1) % 0x100000000 + return i + def __init__(self, servername, sshcommand="ssh", username=None, ssh_options=[], debug=0, debugopts=dict(), maxopenfiles=10): + self.debug = DebugMode(debug) + self.debugopts = debugopts + self.requests = dict() + self.collectors = dict() + self.queue = list() + self.wantwrite = list() + self.requestid_try_next = 17 + self.semaphores = {'openfile': maxopenfiles} + + commandline = [sshcommand] + if ssh_options: + commandline.extend(ssh_options) + # those defaults are after the user-supplied ones so they can be overridden. + # (earlier ones win with ssh). + commandline.extend(["-oProtocol 2", # "-oLogLevel DEBUG", + "-oForwardX11 no", "-oForwardAgent no", + "-oPermitLocalCommand no", + "-oClearAllForwardings yes"]) + if username: + commandline.extend(["-l", username]) + commandline.extend(["-s", "--", servername, "sftp"]) + self.connection = subprocess.Popen(commandline, + close_fds = True, + stdin = subprocess.PIPE, + stdout = subprocess.PIPE, + bufsize = 0) + self.poll = select.poll() + self.poll.register(self.connection.stdout, select.POLLIN) + self.inbuffer = bytes() + self.send(INIT.bin(self, 3)) + t,b = self.getpacket() + if t != VERSION.id: + raise SftpUnexpectedAnswerException(b, "INIT") + # TODO: parse answer data (including available extensions) + def close(self): + self.connection.send_signal(15) + def getmoreinput(self, minlen): + while len(self.inbuffer) < minlen: + o = self.connection.stdout.read(minlen - len(self.inbuffer)) + if o == None: + continue + if len(o) == 0: + raise SftpStrangeException("unexpected EOF") + self.inbuffer = self.inbuffer + o + def getpacket(self): + self.getmoreinput(5) + s = int.from_bytes(self.inbuffer[:4], byteorder='big') + if s < 1: + raise SftpStrangeException("Strange size field in Paket from server!") + t = self.inbuffer[4] + if DebugMode.RAW_IN_STAT in self.debug: + print("receiving packet of length %d and type %d " % + (s, t), **self.debugopts) + s = s - 1 + self.inbuffer = self.inbuffer[5:] + self.getmoreinput(s) + d = self.inbuffer[:s] + self.inbuffer = self.inbuffer[s:] + if DebugMode.RAW_IN in self.debug: + print("received packet(type %d):" % t, repr(d), + **self.debugopts) + return (t, d) + def send(self, b): + if not isinstance(b, bytes): + raise SftpInternalException("send not given byte sequence") + if DebugMode.RAW_OUT_STAT in self.debug: + print("sending packet of %d bytes" % len(b), + **self.debugopts) + if DebugMode.RAW_OUT in self.debug: + print("sending packet:", repr(b), + **self.debugopts) + self.connection.stdin.write(b) + def enqueueRequest(self, job): + if DebugMode.ENQUEUE in self.debug: + print("enqueue", job, + **self.debugopts) + if len(self.queue) == 0 and len(self.wantwrite) == 0: + self.poll.register(self.connection.stdin, + select.POLLOUT) + job.requestid = self.next_request_id() + self.queue.append(job) + def enqueueTask(self, task): + if DebugMode.ENQUEUE in self.debug: + print("enqueue", task, **self.debugopts) + if len(self.queue) == 0 and len(self.wantwrite) == 0: + self.poll.register(self.connection.stdin, + select.POLLOUT) + self.wantwrite.append(task) + def collect(self, who, command, collectortype, *collectorargs): + if DebugMode.LOCKS in self.debug: + print("collector", command, collectortype.__name__, + *collectorargs, **self.debugopts) + """Tell the (possibly to be generated) """ + collectorid = (collectortype, collectorargs) + if not collectorid in self.collectors: + l = collectortype(*collectorargs) + self.collectors[collectorid] = l + l.start(self) + else: + l = self.collectors[collectorid] + l.childinfo(who, command) + def start(self, task): + task.start(self) + def dispatchanswer(self, answer): + task = answer.forr.task + try: + task.sftpanswer(answer) + except StopIteration: + orphanreqs = [ r + for r in self.requests.values() + if r.task == task ] + for r in orphanreqs: + r.done() + def readdata(self): + t,m = self.getpacket() + for answer in Answer.__subclasses__(): + if t == answer.id: + break + else: + raise SftpUnexpectedAnswerException("Unknown answer type %d" % t, "") + id, m = ssh_getu32(m) + a = answer(m) + if DebugMode.COOKED_IN in self.debug: + print("got answer for request %d: %s" % + (id, str(a)), **self.debugopts) + if not id in self.requests: + raise SftpUnexpectedAnswerException(a, "unknown-id-%d" % id) + else: + a.forr = self.requests[id] + self.dispatchanswer(a) + def senddata(self): + if len(self.queue) == 0: + while len(self.wantwrite) > 0: + w = self.wantwrite.pop(0) + if len(self.wantwrite) == 0 and len(self.queue) == 0: + self.poll.unregister(self.connection.stdin) + w.writeready() + if len(self.queue) > 0: + request = self.queue.pop(0) + if len(self.queue) == 0 and len(self.wantwrite) == 0: + self.poll.unregister(self.connection.stdin) + if DebugMode.COOKED_OUT in self.debug: + print("sending request %d: %s" % + (request.requestid, str(request)), + **self.debugopts) + request.send(self) + def dispatch(self): + while self.requests or self.queue: + for (fd, event) in self.poll.poll(): + if event == select.POLLIN: + self.readdata() + elif event == select.POLLHUP: + raise SftpStrangeException( + "Server disconnected unexpectedly" + " or ssh client process terminated") + elif event == select.POLLOUT: + self.senddata() + else: + raise SftpException("Unexpected event %d from poll" % event) + +class Dirlock(Collector): + def __init__(self, name): + super().__init__() + self.name = name + self.dirname = os.path.dirname(name) + self.queue = [] + def start(self, connection): + super().start(connection) + if self.dirname and (self.name != self.dirname): + self.mode = "wait-for-parent" + self.connection.collect(self, 'waitingfor', + Dirlock, self.dirname) + else: + self.tellparent = False + self.mode = "wait-for-client" + self.isnew = False + def sftpanswer(self, a): + assert(self.mode == "creating") + if not isinstance(a, STATUS): + raise SftpUnexpectedAnswer(a, a.forr) + # Only one answer is expected: + a.forr.done() + if a.status == SSH_FX.OK: + self.mode = "exists" + self.isnew = True + self.releaseallqueued() + elif self.tellparent and a.status == SSH_FX.NO_SUCH_FILE: + self.mode = "wait-for-parent" + self.connection.collect(self, 'missing', + Dirlock, self.dirname) + else: + raise SftpException("Cannot create directory %s: %s" % (self.name, a)) + def parentinfo(self, command): + assert(self.mode == "wait-for-parent") + if command == "createnew": + self.tellparent = False + self.isnew = True + self.createdir() + return + if command != "tryandtell" and command != "ready": + raise SftpInternalException( + "Unexpected parent info %s" % + command) + self.tellparent = command == "tryandtell" + if len(self.queue) > 0: + self.mode = "testing" + self.queue.pop(0).parentinfo("tryandtell") + else: + self.mode = "wait-for-client" + def childinfo(self, who, command): + if command == "waitingfor": + if self.mode == "exists": + if self.isnew: + who.parentinfo("createnew") + else: + who.parentinfo("ready") + elif self.mode == "wait-for-client": + self.mode = "testing" + who.parentinfo("tryandtell") + else: + self.queue.append(who) + elif command == "found": + assert(self.mode == "testing") + self.mode = "exists" + self.isnew = False + self.releaseallqueued() + elif command == "missing": + self.queue.append(who) + self.mode = "creating" + self.createdir() + else: + raise SftpInternalException( + "Unexpected child information: %s" % + command) + def createdir(self): + self.mode = "creating" + self.enqueueRequest(MKDIR(self.name)) + def releaseallqueued(self): + if self.tellparent: + self.connection.collect(self, 'found', + Dirlock, self.dirname) + self.tellparent = False + if self.isnew: + command = "createnew" + else: + command = "ready" + # This assumes out mode cannot change any more: + while self.queue: + self.queue.pop(0).parentinfo(command) + +class Semaphore(Collector): + def __init__(self, name): + super().__init__() + self.name = name + self.queue = [] + self.allowed = 10 + def start(self, connection): + self.allowed = connection.semaphores[self.name] + def childinfo(self, who, command): + if command == "lock": + if self.allowed > 0: + self.allowed -= 1 + who.parentinfo("unlock") + else: + self.queue.append(who) + elif command == "release": + if self.allowed == 0 and self.queue: + self.queue.pop(0).parentinfo("unlock") + else: + self.allowed += 1 + else: + raise SftpInternalException("Semaphore.childinfo called with invalid command") diff --git a/docs/short-howto b/docs/short-howto new file mode 100644 index 0000000..bf55bce --- /dev/null +++ b/docs/short-howto @@ -0,0 +1,209 @@ +This short HOW-TO describes how to setup a repository using reprepro. + +First choose a directory where you want to store your repository, + +1) Configuration: + +Generate a directory named conf/. + +Create a file named "distributions" there. + +Add entries such as: + +Origin: Debian +Label: Debian-All +Suite: stable +Codename: woody +Version: 3.0 +Architectures: i386 sparc mips source +Components: main non-free contrib +Description: Debian woody + woody/non-US + woody/updates +#Update: debian non-US security +#SignWith: yes + +Or: + +Origin: PCPool +Label: PCPool +Suite: stable +Codename: pcpool +Version: 3.0 +Architectures: i386 source +Components: main non-free contrib bad protected server +UDebComponents: main +Description: PCPool specific (or backported) packages +SignWith: yes +DebOverride: override +UDebOverride: override +DscOverride: srcoverride + +Multiple entries are separated with an empty line. + +The codename of the distribution is specified with Codename:. +It is the primary name of a distribution and e.g. used to determine the +directory to create and put the index files into. + +Update: is described later. + +If SignWith: is there, it will try to sign it: either use "yes" or give +something gpg can use to identify the key you want to use. + +The other fields are copied into the appropriate "Release" files generated. + +2) Adding files to the repository: + +To add a .deb manually: + +reprepro -Vb . includedeb pcpool /var/cache/apt/archives/libc6_2.2.5-11.8_i386.deb + +to add a .changes file: + +reprepro -Vb . include pcpool test.changes + +Hint: you can add "-C component", "-A architecture", "-S section" and "-P +priority" to give additional hints where it should go. Note -A will not +overwrite something to go into another architecture, but simply ignore those +not fitting, only "Architecture: all" packages are placed exactly in these +architecture. Helps when it is not available for all architectures and each +binary version needs a fitting version of the "Architecture: all" package. + +3) Removing files from the repository: + +reprepro -Vb . remove pcpool libc6 + +to only remove from a specific component or architecture: + +reprepro -Vb . -C main -A i386 remove pcpool libc6 + +4) Getting information about a package: + +To see in which architectures/components a package exists and which version it +uses. + +reprepro -b . list pcpool libc6 + +5) Override-Files: + +When including packages via "includedeb", "includedsc" or "include" +the applicable override file from the distribution it is placed +into is used. The file given by DebOverride: for ".deb"s, the +file given by UDebOverride: for ".udeb"s and the file given by +DscOverride: for ".dsc"s. If the filename starts with +a slash (/) it is not relative to the conf directory given +with --conf, defaulting to "conf" in the current directory (or in the +directory specified with --basedir, if that is given). + +Note that the Format is those of apt-ftparchive's ExtraOverride, not the old format. +An (stupid) example line for that file would be: +libc6 Priority extra + +6) importing from upstream repositories: + +The file conf/updates can contain entries like this: + +Name: debian +Method: http://ftp.debian.de/debian +VerifyRelease: F1D53D8C4F368D5D + +Name: non-US +Method: http://ftp.debian.de/debian-non-US +Suite: */non-US +Architectures: i386 sparc mips source +Components: main>main non-free>non-free contrib>contrib +UDebComponents: +VerifyRelease: B629A24C38C6029A + +Name: security +Method: http://security.debian.org/debian-security +Suite: */updates +UDebComponents: +VerifyRelease: F1D53D8C4F368D5D + +Which of those are used is determined by the Update: line +in the description in conf/distributions. When Suite:, +Architecture:, Components: or UDebComponents: are not given, +those of the distribution to be added are used. +The suite of the target can be used as "*" in the Suite: here. +VerifyRelease: tells which GPG key to use checking the Release.gpg. + +Add a "IgnoreRelease: yes" to ignore any Release files. + +To import components in other components, use the source>target +syntax. + +Method: describes an apt-method, for which the programs +from /usr/lib/apt/methods are used... + +To update everything possible do: + +reprepro -b . update + +To only update some distributions do: + +reprepro -b . update woody + +There is no support for updating a distribution from only specific + upstreams yet. You will have to edit conf/distributions for that. + +The value for VerifyRelease: can be retrieved using: + +gpg --with-colons --list-keys <whatever> + +=============================================================================== +The following is from V. Stanley Jaddoe <debian@terabytemusic.cjb.net>. +Make sure to include all sources when allowing everyone access to software +only available under GPL to you. Well, you should always supply sources, +but in some cases not doing so might cause you trouble. + +Using reprepro with apache2 (sarge, etch, sid) + +This example assumes the reprepro repository is under /srv/reprepro/ and that +apache2 has been correctly installed and configured. + +The first step is to create a virtual directory called debian/. Assuming your +server runs the host http://www.example.com/, the web repository will be +placed at http://www.example.com/debian/. + +Create an apache2 config file in the conf dir of your reprepro repository, +using the following command: + +cat > /srv/reprepro/conf/apache.conf << EOF +Alias /debian /srv/reprepro/ +<Directory /srv/reprepro> + Options +Indexes + AllowOverride None + order allow,deny + allow from all +</Directory> +EOF + +To enable this virtual directory, a symlink has to be created. This can be done +using the following command: + +ln -s /srv/reprepro/conf/apache.conf /etc/apache2/conf.d/reprepro.conf + +The second step is setting the permissions in such a way that web users can +browse the repository, but cannot view the reprepro specific configuration. +This can be done using the following commands: + +chown -R root:root /srv/reprepro/ +chmod 755 /srv/reprepro/ +chown -R root:www-data /srv/reprepro/dists/ /srv/reprepro/pool/ +chmod 750 /srv/reprepro/* + +Reload apache2: + +/etc/init.d/apache2 reload + +Check if the repository is viewable by web-users, by pointing your browser to + +http://www.example.com/debian/ + +If there are no problems with your reprepro repository and the apache2 +configuration, you should see two directories, dists/ and pool/. + +The last step is to add this new repository to your sources.list. + +This is as easy as: + +echo "deb http://www.example.com/debian pcpool main non-free contrib" >> /etc/apt/sources.list diff --git a/docs/xz.example b/docs/xz.example new file mode 100644 index 0000000..2725bb0 --- /dev/null +++ b/docs/xz.example @@ -0,0 +1,30 @@ +#!/bin/sh + +# Copy this script to your conf/ dir as xz.sh, make it executable +# and add to some definition in conf/distributions +# DscIndices: Sources Release . .gz xz.sh +# DebIndices: Packages Release . .gz xz.sh +# UDebIndices: Packages . .gz xz.sh +# and you have .xz'd Packages and Sources. +# (alternatively, if you are very brave, put the full path to this file in there) + +DIROFDIST="$1" +NEWNAME="$2" +OLDNAME="$3" +# this can be old($3 exists), new($2 exists) or change (both): +STATUS="$4" +BASENAME="`basename "$OLDNAME"`" + +if [ "xPackages" = "x$BASENAME" ] || [ "xSources" = "x$BASENAME" ] ; then + if [ "x$STATUS" = "xold" ] ; then + if [ -f "$DIROFDIST/$OLDNAME.xz" ] ; then + echo "$OLDNAME.xz" >&3 + else + xz -c -- "$DIROFDIST/$OLDNAME" >"$DIROFDIST/$OLDNAME.xz.new" 3>/dev/null + echo "$OLDNAME.xz.new" >&3 + fi + else + xz -c -- "$DIROFDIST/$NEWNAME" >"$DIROFDIST/$OLDNAME.xz.new" 3>/dev/null + echo "$OLDNAME.xz.new" >&3 + fi +fi |