# README for Debian packaging contributors

This documentation describes how to contribute to the official Debian packages
of MariaDB. The packaging in Debian repositories is not identical to the packaging
in mariadb.org repositories, but whatever is in Debian repositories will eventually
be upstreamed.


## Development environment and tools

Use a recent version of Debian or Ubuntu as the environment for Debian packaging
testing and development. Preferred environment is Debian Sid (unstable).

Install the tool used to manage and build the source:

    sudo apt-get install git-buildpackage


## Getting the source

The official Debian package source is hosted on the Debian Gitlab server under
the MariaDB/MySQL packaging team at https://salsa.debian.org/mariadb-team/. You
are welcome to fork it and make merge requests.

To get the latest official Debian packaging source of `mariadb`, clone the
source repository with all relevant branches (main branch `debian/latest`) to
your local environment using _git-buildpackage_:

    gbp clone https://salsa.debian.org/mariadb-team/mariadb-server.git

If you have your own fork and SSH keys set up on Salsa, you can run:

    gbp clone git@salsa.debian.org:<username>/mariadb-server.git


The clone needs to be run only once. On later runs you can refresh your clone with
relevant branches using:

    gbp pull --force


## Building the packages

Build binaries, run testsuite and build Debian packages with:

    gbp buildpackage

On the first run git-buildpackage will complain if some of the build dependencies
defined in debian/control are missing. Simply install those packages and run the
build again.

A quick command to install all dependencies:

    sudo mk-build-deps -r -i debian/control -t "apt-get -y -o Debug::pkgProblemResolver=yes --no-install-recommends"

If the build fails, the easiest way to clean up before a new run is

    git clean -fdx && git reset --hard


### Build options

If you want to skip the mysql-test-run step (which takes a lot of time) set
the following environment variable:

    export DEB_BUILD_OPTIONS="nocheck"

If you want to run the build in parallel on 2 CPUs and have verbose output:

    export DEB_BUILD_OPTIONS="parallel=2 verbose"

The options above can also be combined freely to get desired behavior.


### Using special build environments

If you want to ensure all build dependencies are clean, you can build inside a
Docker or sbuild (Debian tool) environment.

#### Build in Docker

First make a working directory for the build artifacts. Inside that directory
clone the repository. Then start a Docker session using whatever Debian/Ubuntu
image you want with the command:

    docker run -it -v ${PWD}:/build -w /build debian:sid bash

This will start a session, where you are as the root user in the path /build
inside the Docker container. Here you can `cd` into the source directory,
install dependencies and start the build. Note that when you exit the session,
everything will be lost apart from the files you had inside the mounted volume
in `/build`.

#### Build using sbuild

If you prefer sbuild, you can build with something like:

    gbp buildpackage --git-builder=sbuild -A -v -d unstable

## Creating a feature or bugfix branch

The repository layout follows the DEP-14 standard:
https://dep-team.pages.debian.net/deps/dep14/

All new features and also bug fixes are done only in the `debian/latest` branch.
The release branches for Debian and Ubuntu are only used for security updates.

To prepare the Salsa pull request, create a bugfix branch from master with:

    git checkout -b bugfix/NNNNNN-example-name

After this you can develop with all the usual git commit and push commands
until you have in your fork at Salsa the desired change and you are ready
to open the merge request.


### Notes about how to make changes in the proper way

First consider submitting your patch upstream. Upstream MariaDB makes frequent
maintenance releases and any fix done upstream will therefore be included in
Debian relatively quickly. You can send email to the developers mailing list
or open a pull request at https://github.com/MariaDB/server.

Follow these instructions if your fix is about packaging in Debian specifically.
Start by using `gitk --all` or similar tool to browse the previous changes. Try
to follow similar pattern in your new changes.

Keep in mind that all changes must done only for files residing in the `debian/`
sub-directory. If you need to create changes outside the `debian/` directory,
then you need to create a patch file using the same pattern as the patches
found in `debian/patches` and activated by a line in `debian/patches/series`.

Do not bundle in your commit any changes to `debian/changelog`. The correct
changelog entries will be created later by the maintainer using `git-dch` in an
automated fashion.

For an example of a patch adding commit see
https://salsa.debian.org/mariadb-team/mariadb-server/-/commit/7972a38e


# Quality assurance tips

Ensure most packaging files are formatted correctly:

    wrap-and-sort -vast

Check man pages for syntax errors:

    LC_ALL=en_US.UTF-8 MANROFFSEQ='' MANWIDTH=80 man --warnings -E UTF-8 -l -Tutf8 -Z mariadb.1 >/dev/null

Find spelling errors:

    find * -type f | xargs spellintian


# Debugging tips

## Debug mariadb-test-run failures

If the test suite is failing on Launchpad or some other CI system where you
don't have access to the build artifacts, you can extend the debian/rules file
to print out valuable information with the commands:

    cd $(BUILDDIR)/mysql-test && find var/log/ -ls || true
    cd $(BUILDDIR)/mysql-test && cat var/log/*.err || true
    cd $(BUILDDIR)/mysql-test && tail -n 1000 var/log/*.log || true

The `cd` is required on every line since it is a Makefile and the actual command
needs to run in the correct directory. Also, the `|| true` at the end ensures
the build will complete and not abort if any of those debug steps fail.

## Debugging with gdb

If the `mariadb-test-run` fails on a `mariadbd` crash it should produce a core
dump file, from which a full stack trace can be produced with:

    cd $(BUILDDIR)/mysql-test && gdb --batch --ex 'thr a a bt' var/log/*/mysqld.1/core || true

To attach `gdb` on a running process and get a stack trace run:

    gdb -p $(pgrep -x mariadbd) /usr/sbin/mariadbd
    set height 0
    set logging file /tmp/mysqld.log
    set logging on
    thread apply all backtrace full

The readability of the stack traces depends on if symbols are available on the
system. In Debian and Ubuntu all (C/C++) software is automatically built with
debug symbols, but to save disk space they are distributed in separate packages
(usually with `-dbg` or `-dbgsym` suffix) which users need to install in the
rare case stack traces are needed. See the Debian and Ubuntu documentation on
how to enable the repository that has the debug symbol packages.

* https://wiki.ubuntu.com/Debug%20Symbol%20Packages
* https://wiki.debian.org/HowToGetABacktrace

## Debug build

A debug build can be created using the following build flags:

    -DCMAKE_BUILD_TYPE=Debug \
    -DMYSQL_MAINTAINER_MODE=OFF \

The latter flag ensures a build does not stop on warnings but continues to the
end.

A 'mariadbd' binary from a debug build can be started with argument '--debug' to
be verbose about what is going on in the server. Debug binaries should not be
used in production as they are slower than normal binaries.

Core dumps and stack traces can be produced on any build running with
`--core-file --stack-trace` and *debug builds are not needed to run `gdb`*.

## Debugging a running server

Linux distros come standard with tools like `strace` and `lsof` which can also
be used to inspect what processes are doing (no need for debug build). For
example to see what `mariadbd` is writing to the database files can be viewed
with:

    strace -ffp $(pgrep -x mariadbd) -e pwrite,write,fsync,fdatasync,sync,send,sendto,sendmsg
    lsof -a -p $(pgrep -x mariadbd) | grep "/var/lib/mysql"

## Compare changes between builds

Diffoscope can be used to investigate small changes between recent builds:

    docker run --rm -t -w $(pwd) -v $(pwd):$(pwd) registry.salsa.debian.org/reproducible-builds/diffoscope --html-dir report mariadb-server-1.deb mariadb-server-2.deb
    firefox report/index.html

## Test autopkgtest locally

If Debian CI fails (or Ubuntu CI) one might need to debug the autopkgtests
manually. The easiest way to do it is to start a Docker container that has
access to the packaging source directory via a local mount:

    laptop$ docker run -it -v ${PWD}:/build -w /build debian:sid bash
    container$ apt update && apt install -y autopkgtest
    container$ autopkgtest --no-built-binaries --shell-fail -- null

Edit the files in `debian/tests` in your favorite code editor and re-run the
`autopkgtest -- null` until the tests are passing. When the autopkgtests work
the container can be shut down and the valid `debian/tests` committed in git.

If you want to iterate on a single test, use `--test-name`, e.g.
`autopkgtest --no-built-binaries --test-name=configuration-tracing -- null`.

If you don't want to use the MariaDB binaries from Debian Sid but instead build
them from the source tree to be used in autopkgtest directly, simply omit
`--no-built-binaries` from the `autopkgtest` command.

For more information please read:
* https://manpages.debian.org/unstable/autopkgtest/autopkgtest.1.en.html
* https://salsa.debian.org/ci-team/autopkgtest/-/tree/master/doc

## Debug installation/upgrade

To see what exactly the Debian maintainer scripts run, they can be made verbose with:

    export DEBIAN_SCRIPT_DEBUG=1
    apt install ...

The source files of the Debian maintainer scripts are not the final ones, as the
package building stage may make changes and additions to them. To view a
maintainer script in the final form on an installed system run:

    cat /var/lib/dpkg/info/mariadb-server.postinst

To review the my.cnf status run:

    find /etc/mysql/ -ls
    update-alternatives --display my.cnf

## Debug apt Depends/Conflicts/Breaks

It can be quite frustrating to debug situations where `apt` (or `apt-get`) fails
on an install or upgrade with an error message like:

    The following packages have unmet dependencies:
     mariadb-client : Depends: mariadb-client-10.5 but 1:10.5.12 is to be installed
     mariadb-server : Depends: mariadb-server-10.5 but 1:10.5.12 is to be installed
     mariadb-test : Depends: mariadb-client-10.5 but 1:10.5.12 is to be installed
                    Depends: mariadb-server-10.5 but 1:10.5.12 is to be installed
    E: Unable to correct problems, you have held broken packages.

To make apt show debug information on what it tried to resolve and how it failed
enable debug features by addin a file in `/etc/apt/apt.conf.d/` with:

   Debug::pkgProblemResolver 1;
   Debug::pkgDepCache::AutoInstall 1;
   Debug::pkgDepCache::Marker 1;

>lternatively append options directly to `apt` commands:

    apt dist-upgrade -o Debug::pkgProblemResolver=1

It can be also quite annoying to rebuild the entire package to debug small
changes in the `debian/control` file. To have a much faster change->test->change
cycle one can simply instruct `apt` to use a custom `Packages` file to read the
`control` data.

First ensure `apt` forgets all repositories:

    rm /etc/apt/sources.list
    apt clean
    apt update

Download a Packages file for so it can be edited:

    curl -O http://ftp.debian.org/debian/dists/sid/main/binary-amd64/Packages.xz
    unxz Packages.xz
    cp Packages Packages.orig

Open the file in an editor, scroll down to the MariadB packages and make any
changes you want and then test them:

    nano Packages
    apt install --with-source ./Packages -s mariadb-server -o Debug::pkgDepCache::Marker=1 -o Debug::pkgDepCache::AutoInstall=1 -o Debug::pkgProblemResolver=1

The example uses maximum verbosity but it is naturally not mandatory. When the
solution has been found, compare to the original and transfer the changes into
the actual debian/control in the MariaDB packaging:

    diff -u Packages.orig Packages

## Test install/upgrade with local package repository

Normally the fastest way to test that the built *.deb files install and upgrade
properly is simply to run `apt` directly on them inside a container that has
access to the .deb files via a local mount:

    laptop$ docker run -it -v ${PWD}:/build -w /build debian:sid bash
    container$ apt update && apt install ./*.deb

Some bugs however occur only when apt does various dependency resolving and can
only be tested with an installation from an actual apt repository. The fastest
way to get a directory with deb files served via a local repository is to run:

    apt install apt-utils
    apt-ftparchive packages . > Packages
    apt-ftparchive release . > Release
    echo 'deb [trusted=yes] file:/build/mariadb-bionic ./' >> /etc/apt/sources.list
    apt update
    apt install mariadb-server

The example above assumes that the .deb files are in path `/build`.

## Check Breaks/Replaces

MariaDB is not only a massive package by itself, it also has several parallel
major releases at any given time and also other variants (MySQL, Percona Server)
the packaging might interact with.

The standard Salsa-CI pipeline checks Breaks/Replaces for what is currently in
the Debian repositories, but to check Breaks/Replaces across all known
repositories one needs to run:

    docker run -it -v ${PWD}:/build -w /build debian:sid bash
    apt update
    apt install --yes python3-junit.xml python3-debian apt-file
    curl -O https://salsa.debian.org/salsa-ci-team/pipeline/-/raw/master/images/scripts/check_for_missing_breaks_replaces.py
    chmod +x check_for_missing_breaks_replaces.py
    apt install --no-install-recommends --yes gpg gpg-agent dirmngr ca-certificates curl debian-archive-keyring
    curl -sS https://mariadb.org/mariadb_release_signing_key.asc -o /etc/apt/trusted.gpg.d/mariadb.asc
    gpg --list-keys # Initialize default keyring
    gpg --no-default-keyring --keyring gnupg-ring:/etc/apt/trusted.gpg.d/mariadb.gpg --keyserver hkps://keyserver.ubuntu.com:443 --recv-keys 871920D1991BC93C 3B4FE6ACC0B21F32 CBF8D6FD518E17E1 7638D0442B90D010 8C718D3B5072E1F5 9334A25F8507EFA5 CBCB082A1BB943DB 467B942D3A79BD29
    chmod 644 /etc/apt/trusted.gpg.d/mariadb.gpg
    cat > /etc/apt/sources.list.d/mariadb.list <<EOF
    deb http://deb.debian.org/debian bullseye main
    deb http://deb.debian.org/debian buster main
    deb http://deb.debian.org/debian stretch main
    deb [trusted=yes] http://deb.debian.org/debian jessie main
    deb http://archive.ubuntu.com/ubuntu/ jammy main restricted universe multiverse
    deb http://archive.ubuntu.com/ubuntu/ focal main restricted universe multiverse
    deb http://archive.ubuntu.com/ubuntu/ bionic main restricted universe multiverse
    deb http://archive.ubuntu.com/ubuntu/ xenial main restricted universe multiverse
    deb http://archive.ubuntu.com/ubuntu/ trusty main restricted universe multiverse
    deb https://archive.mariadb.org/mariadb-10.11/repo/debian bullseye main
    deb https://archive.mariadb.org/mariadb-10.10/repo/debian bullseye main
    deb https://archive.mariadb.org/mariadb-10.9/repo/debian bullseye main
    deb https://archive.mariadb.org/mariadb-10.8/repo/debian bullseye main
    deb https://archive.mariadb.org/mariadb-10.7/repo/debian bullseye main
    deb https://archive.mariadb.org/mariadb-10.6/repo/debian buster main
    deb https://archive.mariadb.org/mariadb-10.5/repo/debian buster main
    deb https://archive.mariadb.org/mariadb-10.4/repo/debian buster main
    deb https://archive.mariadb.org/mariadb-10.3/repo/debian buster main
    deb https://archive.mariadb.org/mariadb-10.2/repo/debian buster main
    deb https://archive.mariadb.org/mariadb-10.1/repo/debian stretch main
    deb [trusted=yes] https://archive.mariadb.org/mariadb-10.0/repo/debian jessie main
    deb [trusted=yes] https://archive.mariadb.org/mariadb-5.5/repo/debian wheezy main
    deb https://repo.mysql.com/apt/ubuntu/ jammy mysql-8.0
    deb https://repo.mysql.com/apt/ubuntu/ focal mysql-8.0
    deb https://repo.mysql.com/apt/debian/ buster mysql-8.0
    deb https://repo.mysql.com/apt/debian/ buster mysql-5.7
    deb https://repo.mysql.com/apt/debian/ buster mysql-5.6
    deb https://repo.mysql.com/apt/debian/ buster mysql-cluster-8.0
    deb https://repo.mysql.com/apt/debian/ buster mysql-cluster-7.6
    deb https://repo.mysql.com/apt/debian/ buster mysql-cluster-7.5
    deb https://repo.mysql.com/apt/debian/ buster mysql-tools
    deb https://repo.percona.com/apt/ bullseye main
    deb https://repo.percona.com/apt/ buster main
    deb https://repo.percona.com/apt/ stretch main
    deb https://repo.percona.com/apt/ jessie main
    deb https://repo.percona.com/apt/ wheezy main
    EOF
    apt-file update
    ./check_for_missing_breaks_replaces.py --changes-file mariadb-*.changes --debug

## Check reverse dependencies

When making changes to the MariaDB packaging in Debian and Ubuntu, keep in mind
that there are hundreds of packages that depend on MariaDB. Most of them can be
found by running:

    apt rdepends 'default-mysql*' 'default-libmysql*' 'mariadb*' 'libmariadb*'

The separate command/package 'apt-rdepends' can also check for reverse
build-dependencies.

Please be diligent in all changes to not wreak havoc in Debian.