diff options
Diffstat (limited to 'debian/README.Contributor')
| -rw-r--r-- | debian/README.Contributor | 420 |
1 files changed, 420 insertions, 0 deletions
diff --git a/debian/README.Contributor b/debian/README.Contributor new file mode 100644 index 00000000..45708b52 --- /dev/null +++ b/debian/README.Contributor @@ -0,0 +1,420 @@ +# README for Debian packaging contributors + +This documentation describes how to contribute to the official Debian packages +of MariaDB. The packaging in Debian repositories is not identical to the packaging +in mariadb.org repositories, but whatever is in Debian repositories will eventually +be upstreamed. + + +## Development environment and tools + +Use a recent version of Debian or Ubuntu as the environment for Debian packaging +testing and development. Preferred environment is Debian Sid (unstable). + +Install the tool used to manage and build the source: + + sudo apt-get install git-buildpackage + + +## Getting the source + +The official Debian package source is hosted on the Debian Gitlab server under +the MariaDB/MySQL packaging team at https://salsa.debian.org/mariadb-team/. You +are welcome to fork it and make merge requests. + +To get the latest official Debian packaging source of `mariadb`, clone the +source repository with all relevant branches (main branch `debian/latest`) to +your local environment using _git-buildpackage_: + + gbp clone https://salsa.debian.org/mariadb-team/mariadb-server.git + +If you have your own fork and SSH keys set up on Salsa, you can run: + + gbp clone git@salsa.debian.org:<username>/mariadb-server.git + + +The clone needs to be run only once. On later runs you can refresh your clone with +relevant branches using: + + gbp pull --force + + +## Building the packages + +Build binaries, run testsuite and build Debian packages with: + + gbp buildpackage + +On the first run git-buildpackage will complain if some of the build dependencies +defined in debian/control are missing. Simply install those packages and run the +build again. + +A quick command to install all dependencies: + + sudo mk-build-deps -r -i debian/control -t "apt-get -y -o Debug::pkgProblemResolver=yes --no-install-recommends" + +If the build fails, the easiest way to clean up before a new run is + + git clean -fdx && git reset --hard + + +### Build options + +If you want to skip the mysql-test-run step (which takes a lot of time) set +the following environment variable: + + export DEB_BUILD_OPTIONS="nocheck" + +If you want to run the build in parallel on 2 CPUs and have verbose output: + + export DEB_BUILD_OPTIONS="parallel=2 verbose" + +The options above can also be combined freely to get desired behavior. + + +### Using special build environments + +If you want to ensure all build dependencies are clean, you can build inside a +Docker or sbuild (Debian tool) environment. + +#### Build in Docker + +First make a working directory for the build artifacts. Inside that directory +clone the repository. Then start a Docker session using whatever Debian/Ubuntu +image you want with the command: + + docker run -it -v ${PWD}:/build -w /build debian:sid bash + +This will start a session, where you are as the root user in the path /build +inside the Docker container. Here you can `cd` into the source directory, +install dependencies and start the build. Note that when you exit the session, +everything will be lost apart from the files you had inside the mounted volume +in `/build`. + +#### Build using sbuild + +If you prefer sbuild, you can build with something like: + + gbp buildpackage --git-builder=sbuild -A -v -d unstable + +## Creating a feature or bugfix branch + +The repository layout follows the DEP-14 standard: +https://dep-team.pages.debian.net/deps/dep14/ + +All new features and also bug fixes are done only in the `debian/latest` branch. +The release branches for Debian and Ubuntu are only used for security updates. + +To prepare the Salsa pull request, create a bugfix branch from master with: + + git checkout -b bugfix/NNNNNN-example-name + +After this you can develop with all the usual git commit and push commands +until you have in your fork at Salsa the desired change and you are ready +to open the merge request. + + +### Notes about how to make changes in the proper way + +First consider submitting your patch upstream. Upstream MariaDB makes frequent +maintenance releases and any fix done upstream will therefore be included in +Debian relatively quickly. You can send email to the developers mailing list +or open a pull request at https://github.com/MariaDB/server. + +Follow these instructions if your fix is about packaging in Debian specifically. +Start by using `gitk --all` or similar tool to browse the previous changes. Try +to follow similar pattern in your new changes. + +Keep in mind that all changes must done only for files residing in the `debian/` +sub-directory. If you need to create changes outside the `debian/` directory, +then you need to create a patch file using the same pattern as the patches +found in `debian/patches` and activated by a line in `debian/patches/series`. + +Do not bundle in your commit any changes to `debian/changelog`. The correct +changelog entries will be created later by the maintainer using `git-dch` in an +automated fashion. + +For an example of a patch adding commit see +https://salsa.debian.org/mariadb-team/mariadb-server/-/commit/7972a38e + + +# Quality assurance tips + +Ensure most packaging files are formatted correctly: + + wrap-and-sort -av + +Check man pages for syntax errors: + + LC_ALL=en_US.UTF-8 MANROFFSEQ='' MANWIDTH=80 man --warnings -E UTF-8 -l -Tutf8 -Z mariadb.1 >/dev/null + +Find spelling errors: + + find * -type f | xargs spellintian + + +# Debugging tips + +## Debug mariadb-test-run failures + +If the test suite is failing on Launchpad or some other CI system where you +don't have access to the build artifacts, you can extend the debian/rules file +to print out valuable information with the commands: + + cd $(BUILDDIR)/mysql-test && find var/log/ -ls || true + cd $(BUILDDIR)/mysql-test && cat var/log/*.err || true + cd $(BUILDDIR)/mysql-test && tail -n 1000 var/log/*.log || true + +The `cd` is required on every line since it is a Makefile and the actual command +needs to run in the correct directory. Also, the `|| true` at the end ensures +the build will complete and not abort if any of those debug steps fail. + +## Debugging with gdb + +If the `mariadb-test-run` fails on a `mariadbd` crash it should produce a core +dump file, from which a full stack trace can be produced with: + + cd $(BUILDDIR)/mysql-test && gdb --batch --ex 'thr a a bt' var/log/*/mysqld.1/core || true + +To attach `gdb` on a running process and get a stack trace run: + + gdb -p $(pgrep -x mariadbd) /usr/sbin/mariadbd + set height 0 + set logging file /tmp/mysqld.log + set logging on + thread apply all backtrace full + +The readability of the stack traces depends on if symbols are available on the +system. In Debian and Ubuntu all (C/C++) software is automatically built with +debug symbols, but to save disk space they are distributed in separate packages +(usually with `-dbg` or `-dbgsym` suffix) which users need to install in the +rare case stack traces are needed. See the Debian and Ubuntu documentation on +how to enable the repository that has the debug symbol packages. + +* https://wiki.ubuntu.com/Debug%20Symbol%20Packages +* https://wiki.debian.org/HowToGetABacktrace + +## Debug build + +A debug build can be created using the following build flags: + + -DCMAKE_BUILD_TYPE=Debug \ + -DMYSQL_MAINTAINER_MODE=OFF \ + +The latter flag ensures a build does not stop on warnings but continues to the +end. + +A 'mariadbd' binary from a debug build can be started with argument '--debug' to +be verbose about what is going on in the server. Debug binaries should not be +used in production as they are slower than normal binaries. + +Core dumps and stack traces can be produced on any build running with +`--core-file --stack-trace` and *debug builds are not needed to run `gdb`*. + +## Debugging a running server + +Linux distros come standard with tools like `strace` and `lsof` which can also +be used to inspect what processes are doing (no need for debug build). For +example to see what `mariadbd` is writing to the database files can be viewed +with: + + strace -ffp $(pgrep -x mariadbd) -e pwrite,write,fsync,fdatasync,sync,send,sendto,sendmsg + lsof -a -p $(pgrep -x mariadbd) | grep "/var/lib/mysql" + +## Compare changes between builds + +Diffoscope can be used to investigate small changes between recent builds: + + docker run --rm -t -w $(pwd) -v $(pwd):$(pwd) registry.salsa.debian.org/reproducible-builds/diffoscope --html-dir report mariadb-server-1.deb mariadb-server-2.deb + firefox report/index.html + +## Test autopkgtest locally + +If Debian CI fails (or Ubuntu CI) one might need to debug the autopkgtests +manually. The easiest way to do it is to start a Docker container that has +access to the packaging source directory via a local mount: + + laptop$ docker run -it -v ${PWD}:/build -w /build debian:sid bash + container$ apt update && apt install -y autopkgtest + container$ autopkgtest --no-built-binaries --shell-fail -- null + +Edit the files in `debian/tests` in your favorite code editor and re-run the +`autopkgtest -- null` until the tests are passing. When the autopkgtests work +the container can be shut down and the valid `debian/tests` committed in git. + +If you want to iterate on a single test, use `--test-name`, e.g. +`autopkgtest --no-built-binaries --test-name=configuration-tracing -- null`. + +If you don't want to use the MariaDB binaries from Debian Sid but instead build +them from the source tree to be used in autopkgtest directly, simply omit +`--no-built-binaries` from the `autopkgtest` command. + +For more information please read: +* https://manpages.debian.org/unstable/autopkgtest/autopkgtest.1.en.html +* https://salsa.debian.org/ci-team/autopkgtest/-/tree/master/doc + +## Debug installation/upgrade + +To see what exactly the Debian maintainer scripts run, they can be made verbose with: + + export DEBIAN_SCRIPT_DEBUG=1 + apt install ... + +The source files of the Debian maintainer scripts are not the final ones, as the +package building stage may make changes and additions to them. To view a +maintainer script in the final form on an installed system run: + + cat /var/lib/dpkg/info/mariadb-server.postinst + +To review the my.cnf status run: + + find /etc/mysql/ -ls + update-alternatives --display my.cnf + +## Debug apt Depends/Conflicts/Breaks + +It can be quite frustrating to debug situations where `apt` (or `apt-get`) fails +on an install or upgrade with an error message like: + + The following packages have unmet dependencies: + mariadb-client : Depends: mariadb-client-10.5 but 1:10.5.12 is to be installed + mariadb-server : Depends: mariadb-server-10.5 but 1:10.5.12 is to be installed + mariadb-test : Depends: mariadb-client-10.5 but 1:10.5.12 is to be installed + Depends: mariadb-server-10.5 but 1:10.5.12 is to be installed + E: Unable to correct problems, you have held broken packages. + +To make apt show debug information on what it tried to resolve and how it failed +enable debug features by addin a file in `/etc/apt/apt.conf.d/` with: + + Debug::pkgProblemResolver 1; + Debug::pkgDepCache::AutoInstall 1; + Debug::pkgDepCache::Marker 1; + +>lternatively append options directly to `apt` commands: + + apt dist-upgrade -o Debug::pkgProblemResolver=1 + +It can be also quite annoying to rebuild the entire package to debug small +changes in the `debian/control` file. To have a much faster change->test->change +cycle one can simply instruct `apt` to use a custom `Packages` file to read the +`control` data. + +First ensure `apt` forgets all repositories: + + rm /etc/apt/sources.list + apt clean + apt update + +Download a Packages file for so it can be edited: + + curl -O http://ftp.debian.org/debian/dists/sid/main/binary-amd64/Packages.xz + unxz Packages.xz + cp Packages Packages.orig + +Open the file in an editor, scroll down to the MariadB packages and make any +changes you want and then test them: + + nano Packages + apt install --with-source ./Packages -s mariadb-server -o Debug::pkgDepCache::Marker=1 -o Debug::pkgDepCache::AutoInstall=1 -o Debug::pkgProblemResolver=1 + +The example uses maximum verbosity but it is naturally not mandatory. When the +solution has been found, compare to the original and transfer the changes into +the actual debian/control in the MariaDB packaging: + + diff -u Packages.orig Packages + +## Test install/upgrade with local package repository + +Normally the fastest way to test that the built *.deb files install and upgrade +properly is simply to run `apt` directly on them inside a container that has +access to the .deb files via a local mount: + + laptop$ docker run -it -v ${PWD}:/build -w /build debian:sid bash + container$ apt update && apt install ./*.deb + +Some bugs however occur only when apt does various dependency resolving and can +only be tested with an installation from an actual apt repository. The fastest +way to get a directory with deb files served via a local repository is to run: + + apt install apt-utils + apt-ftparchive packages . > Packages + apt-ftparchive release . > Release + echo 'deb [trusted=yes] file:/build/mariadb-bionic ./' >> /etc/apt/sources.list + apt update + apt install mariadb-server + +The example above assumes that the .deb files are in path `/build`. + +## Check Breaks/Replaces + +MariaDB is not only a massive package by itself, it also has several parallel +major releases at any given time and also other variants (MySQL, Percona Server) +the packaging might interact with. + +The standard Salsa-CI pipeline checks Breaks/Replaces for what is currently in +the Debian repositories, but to check Breaks/Replaces across all known +repositories one needs to run: + + docker run -it -v ${PWD}:/build -w /build debian:sid bash + apt update + apt install --yes python3-junit.xml python3-debian apt-file + curl -O https://salsa.debian.org/salsa-ci-team/pipeline/-/raw/master/images/scripts/check_for_missing_breaks_replaces.py + chmod +x check_for_missing_breaks_replaces.py + apt install --no-install-recommends --yes gpg gpg-agent dirmngr ca-certificates curl debian-archive-keyring + curl -sS https://mariadb.org/mariadb_release_signing_key.asc -o /etc/apt/trusted.gpg.d/mariadb.asc + gpg --list-keys # Initialize default keyring + gpg --no-default-keyring --keyring gnupg-ring:/etc/apt/trusted.gpg.d/mariadb.gpg --keyserver hkps://keyserver.ubuntu.com:443 --recv-keys 871920D1991BC93C 3B4FE6ACC0B21F32 CBF8D6FD518E17E1 7638D0442B90D010 8C718D3B5072E1F5 9334A25F8507EFA5 CBCB082A1BB943DB 467B942D3A79BD29 + chmod 644 /etc/apt/trusted.gpg.d/mariadb.gpg + cat > /etc/apt/sources.list.d/mariadb.list <<EOF + deb http://deb.debian.org/debian bullseye main + deb http://deb.debian.org/debian buster main + deb http://deb.debian.org/debian stretch main + deb [trusted=yes] http://deb.debian.org/debian jessie main + deb http://archive.ubuntu.com/ubuntu/ jammy main restricted universe multiverse + deb http://archive.ubuntu.com/ubuntu/ focal main restricted universe multiverse + deb http://archive.ubuntu.com/ubuntu/ bionic main restricted universe multiverse + deb http://archive.ubuntu.com/ubuntu/ xenial main restricted universe multiverse + deb http://archive.ubuntu.com/ubuntu/ trusty main restricted universe multiverse + deb https://archive.mariadb.org/mariadb-10.11/repo/debian bullseye main + deb https://archive.mariadb.org/mariadb-10.10/repo/debian bullseye main + deb https://archive.mariadb.org/mariadb-10.9/repo/debian bullseye main + deb https://archive.mariadb.org/mariadb-10.8/repo/debian bullseye main + deb https://archive.mariadb.org/mariadb-10.7/repo/debian bullseye main + deb https://archive.mariadb.org/mariadb-10.6/repo/debian buster main + deb https://archive.mariadb.org/mariadb-10.5/repo/debian buster main + deb https://archive.mariadb.org/mariadb-10.4/repo/debian buster main + deb https://archive.mariadb.org/mariadb-10.3/repo/debian buster main + deb https://archive.mariadb.org/mariadb-10.2/repo/debian buster main + deb https://archive.mariadb.org/mariadb-10.1/repo/debian stretch main + deb [trusted=yes] https://archive.mariadb.org/mariadb-10.0/repo/debian jessie main + deb [trusted=yes] https://archive.mariadb.org/mariadb-5.5/repo/debian wheezy main + deb https://repo.mysql.com/apt/ubuntu/ jammy mysql-8.0 + deb https://repo.mysql.com/apt/ubuntu/ focal mysql-8.0 + deb https://repo.mysql.com/apt/debian/ buster mysql-8.0 + deb https://repo.mysql.com/apt/debian/ buster mysql-5.7 + deb https://repo.mysql.com/apt/debian/ buster mysql-5.6 + deb https://repo.mysql.com/apt/debian/ buster mysql-cluster-8.0 + deb https://repo.mysql.com/apt/debian/ buster mysql-cluster-7.6 + deb https://repo.mysql.com/apt/debian/ buster mysql-cluster-7.5 + deb https://repo.mysql.com/apt/debian/ buster mysql-tools + deb https://repo.percona.com/apt/ bullseye main + deb https://repo.percona.com/apt/ buster main + deb https://repo.percona.com/apt/ stretch main + deb https://repo.percona.com/apt/ jessie main + deb https://repo.percona.com/apt/ wheezy main + EOF + apt-file update + ./check_for_missing_breaks_replaces.py --changes-file mariadb-*.changes --debug + +## Check reverse dependencies + +When making changes to the MariaDB packaging in Debian and Ubuntu, keep in mind +that there are hundreds of packages that depend on MariaDB. Most of them can be +found by running: + + apt rdepends 'default-mysql*' 'default-libmysql*' 'mariadb*' 'libmariadb*' + +The separate command/package 'apt-rdepends' can also check for reverse +build-dependencies. + +Please be diligent in all changes to not wreak havoc in Debian. |