path: root/GETTING-STARTED-WITH-dh-debputy.md

# Getting started with `dh-debputy`

_This is [how-to guide] and is primarily aimed at getting a task done._

<!-- To writers and reviewers: Check the documentation against https://documentation.divio.com/ -->

This document will help you convert a Debian source package using the `dh` sequencer from `debhelper` to
use `dh-debputy`, where `debputy` is integrated with `debhelper`.  Prerequisites for this how-to guide:

 * You have a Debian source package using the `dh` sequencer on `debhelper` compat level 12 or later.
 * It is strongly recommended that your package is bit-for-bit reproducible before starting the conversion
   as that makes it easier to spot bugs introduced by the conversion!  The output of `debputy` will *not*
   be bit-for-bit reproducible with `debhelper` in all cases. However, the differences should be easy to
   review with `diffoscope` if the package was already bit-for-bit reproducible.

Note that during the conversion (particularly Step 2 and Step 3), you may find that `debputy` cannot
support the requirements for your package for now.  Feel free to file an issue for what is holding you
back in the [debputy issue tracker].

Prerequisites
-------------

This guide assumes familiarity with Debian packaging in general.  Notably, you should understand
the different between a (Debian) source package and a (Debian) binary package (e.g., `.deb`) plus
how these concepts relates to `debian/control` (the source control file).

Additionally, since this is about `debputy` integration with debhelper, the reader is expected to
be familiar with `debhelper` (notably the `dh` style `debian/rules`).

## Step 1: Choose the level of migration target

When migrating a package to `debputy`, you have to choose at which level you want `debputy` to manage
your packaging. At the time of writing, your options are:

 1) `dh-sequence-zz-debputy-rrr`: Minimal integration necessary to get support for `Rules-Requires-Root: no`
    in all packages.  It is compatible with most `dh` based packages as it only replaces very few helpers
    and therefore usually also requires very little migration.

 2) `dh-sequence-zz-debputy`: In this integration mode, `debhelper` manages the upstream build system
    (that is, anything up to and including `dh_auto_install`). Everything else is managed by `debputy`.
    This mode is less likely to be compatible with complex packaging at the time of writing. Notably,
    most `debhelper` add-ons will not be compatible with `debputy` in this integration mode.
    - For this integration mode, you are recommended to pick a simple package as many packages cannot be
      converted at this time. Note that this mode does *not* interact well with most third-party
      `dh` addons. You are recommended to start with source packages without third-party `dh` addons.

Since you can always migrate from "less integrated" to "more integrated", you are recommended to start
with `dh-sequence-zz-debputy-rrr` first. If that works, you can re-run the migration with
`dh-sequence-zz-debputy` as the target to see if further integration seems feasible / desirable.

Note: More options may appear in the future.

## Step 2: Have `debputy` convert relevant `debhelper` files

The `debputy` integration with `debhelper` removes (replaces) some debhelper tools, but does **not**
read their relevant config files.  These should instead be converted in to the new format. 

You can have `debputy` convert these for you by using the following command:

    # Dry-run conversion, creates `debian/debputy-manifest.new` if anything is migrated and prints a summary
    # of what is done and what you manually need to fix.  Replace `--no-apply-changes` with `--apply-changes`
    # when you are ready to commit the conversion.
    #  - Replace MIGRATION_TARGET in the command with the concrete target from step 1
    $ debputy migrate-from-dh --migration-target MIGRATION_TARGET --no-apply-changes

Note: Running `debputy migrate-from-dh ...` multiple times is supported and often recommended as it can
help you assert that all detectable issues have been fixed.

If relevant, `debputy` may inform you that:

 1) the package needs to change build dependencies or/and activate `dh` add-ons. Concretely, it will ask you to
    ensure that the relevant `dh` add-on is added for `debputy` sequence matching the migration target. Additionally,
    it may ask you to add build-dependencies for activating `debputy` plugins to match certain `dh` add-ons.
    - Note that if `debputy` asks you to add a `debputy` plugin without removing the `dh` add-on it supposedly
      replaces then keep the `dh` active. Sometimes the add-on is only partly converted and sometimes the `dh`
      sequence is used to pull relevant build-dependencies.

 2) the package is using an unsupported feature.  Unless that functionality can easily be removed (e.g., it is now
    obsolete) or easily replaced, then you probably do not want to convert the package to `debputy` at this time.
    * One common source of unsupported features are dh hook targets (such as override targets), which will be covered
      in slightly more detail in the next section.  If `debputy` detected any hook targets, it is probably worth it to
      check if these can be migrated before continuing or run earlier in a hook target that is not removed.
    * Other cases can be that the package uses a feature, where `debputy` behaves differently than `debhelper` would
      at the given compat level. In this case, `debputy` will recommend that you perform a compat upgrade in
      `debhelper` before migrating to `debputy`.
    * It is worth noting that the migration tool will update an existing manifest when re-run. You can safely "jump"
      around in the order of the steps, when you try to migrate, if that better suits you.

 3) the migration would trigger a conflict.  This can occur for two reasons:
    * The debhelper configuration has the conflict (example [#934499]), where debhelper is being lenient and ignores
       the problem.  In this case, you need to resolve the conflict in the debhelper config and re-run `debputy`.
    * The package has a manifest already with a similar (but not identical) definition of what the migration would
       generate.  In this case, you need to reconcile the definitions manually (and removing one of them).  After that
       you can re-run `debputy`.

As an example, if you had a `debian/foo.links` (with `foo` being defined in `debian/control`) containing the following:

    usr/share/foo/my-first-symlink usr/share/bar/symlink-target
    usr/lib/{{DEB_HOST_MULTIARCH}}/my-second-symlink usr/lib/{{DEB_HOST_MULTIARCH}}/baz/symlink-target

The `debputy migrate-from-dh --migration-target dh-sequence-zz-debputy` tool would generate a manifest looking
something like this:

    manifest-version: "0.1"
    packages:
        foo:
            transformations:
             - create-symlink:
                  path: usr/share/foo/my-first-symlink
                  target: /usr/share/bar/symlink-target
             - create-symlink:
                  path: usr/lib/{{DEB_HOST_MULTIARCH}}/my-second-symlink
                  target: /usr/lib/{{DEB_HOST_MULTIARCH}}/baz/symlink-target


## Step 3: Migrate override/hook-targets in debian/rules

Have a look at the hook targets that `debputy migrate-from-dh` flags as unsupported and migrate them to
`debputy` features or move them to earlier hook targets that are supported.  See also the subsections
below for concrete advice on how to deal with override or hook targets for some of these tools. However,
since `debhelper` hooks are arbitrary code execution hooks, there will be many cases that the guide will
not be able to cover or where `debputy` may not have the feature to support your hook target.

While you could manually force any of the removed `debhelper` tools to be run via a hook target, they are
very likely to feature  interact with `debputy`. Either causing `debputy` to replace their output completely
or by having the tool overwrite what `debputy` did (depending on the exact order).  If you find, you
*really* need to run one of these tools, because `debputy` is not supporting a particular feature they have,
then you are probably better off not migrate to this level of `debputy` integration at this time.


### Affected debhelper command list for `dh-sequence-zz-debputy-rrr` integration mode

The following `debhelper` commands are replaced in the `dh-sequence-zz-debputy-rrr` integration mode. Generally,
`debputy migrate-from-dh` will warn you if there is anything to worry about in relation to these commands.

 * `dh_fixperms`
 * `dh_gencontrol`
 * `dh_md5sums`
 * `dh_builddeb`

In case something is flagged, it is most likely a hook target, which either have to be converted to `debputy`'s
features or moved earlier. The most common cases are hook targets for `dh_fixperms` and `dh_gencontrol`, which
have sections below advising how to approach those.

### Affected debhelper command list for `dh-sequence-zz-debputy` integration mode

The following `debhelper` commands are replaced in the `dh-sequence-zz-debputy` integration mode. The
`debputy migrate-from-dh` command will warn you where it can when some of these are used. However, some
usage may only become apparent during package build. Therefore, `debputy migrate-from-dh` can be silent
even though `debputy` during the build will flag an unsupported feature.

You are recommended to skim through this list for commands where your package might be using non-trivial
or non-default features, as those are likely to prevent migration at this time. Pay extra attention to
commands marked with **(!)** as `debputy` has zero or almost zero support for features from these
commands. Other tools will have some form of support (often at least a commonly used flow/feature set).

 * `dh_install`
 * `dh_installdocs`
 * `dh_installchangelogs`
 * `dh_installexamples`
 * `dh_installman`
 * `dh_installcatalogs` **(!)**
 * `dh_installcron`
 * `dh_installifupdown`
 * `dh_installdebconf` **(!)**
 * `dh_installemacsen` **(!)**
 * `dh_installinfo`
 * `dh_installinit`
    - Only default service mode (enable, standard restart) are supported. Any use of `--no-start`, etc.
      is unsupported
 * `dh_installsysusers`
 * `dh_installtmpfiles`
 * `dh_installsystemd`
    - Only default service mode (enable, standard restart) are supported. Any use of `--no-start`, etc.
      is unsupported
 * `dh_installsystemduser` **(!)**
 * `dh_installmenu` **(!)**
 * `dh_installmime`
 * `dh_installmodules`
 * `dh_installlogcheck`
 * `dh_installlogrotate`
 * `dh_installpam`
 * `dh_installppp`
 * `dh_installudev`  **(!)**
 * `dh_installgsettings`
 * `dh_installinitramfs`
 * `dh_installalternatives`
 * `dh_bugfiles`
 * `dh_ucf` **(!)**
 * `dh_lintian`
 * `dh_icons`
 * `dh_perl`
 * `dh_usrlocal` **(!)**
 * `dh_links`
 * `dh_installwm` **(!)**
 * `dh_installxfonts`
 * `dh_strip_nondeterminism`
 * `dh_compress`
 * `dh_fixperms`
 * `dh_dwz`
 * `dh_strip`
 * `dh_makeshlibs`
 * `dh_shlibdeps`
 * `dh_missing`
 * `dh_installdeb`
 * `dh_gencontrol`
 * `dh_md5sums`

As mentioned, `debputy migrate-from-dh --no-act` will detect these completely unsupported tools via existence
of their config files or indirectly debhelper hook targets for these tools where possible.  However, some
tools may only be detected late into the build (which is the case with `dh_usrlocal` as a concrete
example).

### Review and migrate any installation code from `dh_install`, `dh_installdocs`, etc. (if any)

_This is section does **not** apply to the `dh-sequence-zz-debputy-rrr` integration mode._

All code in `debian/rules` that involves copying or moving files into packages or around in packages must
be moved to the manifest's `installations` section.  The migration tool will attempt to auto-migrate
any rules from `debian/install` (etc.). However, be aware of:

 1) The migration tool assumes none of install rules overlap.  Most packages do not have overlapping
    install rules as it tends to lead to file conflicts.  If the install rules overlap, `debputy` will
    detect it at *runtime* (not migration time) and stop with an error. In that case, you will have to
    tweak the migrated rules manually.

 2) Any hook target that copies or moves files around in packages must be moved to `installations`
    (per source) or `transformations` (per package) depending on the case.

    - For source packages installing content via `debian/tmp`, you can use `install` to rename paths as
      you install them and `discard` (under `installations`) to ignore paths that should not be installed.

    - For source packages installing content via `debian/<pkg>`, then everything in there is "auto-installed".
      If you need to tweak that content, you can use `remove` or `move` transformations (under `transformations`)
      for manipulation the content.

Keep in mind that the migrator "blindly" appends new rules to the bottom of `installations` if you have any existing
rules (per "none of the install rules overlap"-logic mentioned above).  If you cannot convert all debhelper config
files in one go, or you manually created some installation rules before running the migrator, you may need to
manually re-order the generated installation rules to avoid conflicts due to inadequate ordering.  You probably
want to do so any way for readability/grouping purposes.

Note: For very complex hook targets that manipulate context in packages, it is possible to keep the logic in
`debian/rules` by moving the logic to `execute_after_dh_auto_install`. This will mainly be useful if you are
using complex rules for pruning or moving around files in a way that are not easily representable via globs.

#### Double-check the `language` settings on all `install-man` rules in `installations`

_This is section does **not** apply to the `dh-sequence-zz-debputy-rrr` integration mode._

The `dh_installman` tool auto-detects the language for manpages via two different methods:

 1) By path name (Does the installation path look like `man/<language>/man<section>/...`?)
 2) By basename (Does the basename look like `foo.<language>.<section>`?)

Both methods are used in order with first match being the method of choice.  Unfortunately, the second
method is prune to false-positives.  Does `foo.pl.1` mean a Polish translation of `foo.1` or is it the
manpage for a Perl script called `foo.pl` (similar happens for other languages/file extensions).

To avoid this issue, `debputy` uses 1) by default and only that.  Option 2) can be chosen by setting
`language: derive-from-basename` on the concrete installation rule.  The problem is that the migration tool
has to guess, and it is hard to tell if rules like `man/*.1` would need option 2).

Therefore, take a critical look at the generated `install-man` rules and the generated `language` property
(or lack thereof).

### Convert your overrides or excludes for `dh_fixperms` (if any)

The `debputy` tool will normalize permissions like `dh_fixperms` during package build.  If you have
any special requirements that `dh_fixperms` did not solve for you, you will have to tell `debputy`
about them.

If you had something like:

    override_dh_fixperms:
        dh_fixperms -X bin/sudo

and the goal was to have `dh_fixperms` not touch the mode but the ownership (root:root) was fine, you
would have the manifest `debian/debputy.manifest`:

    manifest-version: "0.1"
    packages:
        foo:
            transformations:
             - path-metadata:
                  path: usr/bin/sudo
                  mode: "04755"

Note you have to spell out the desired mode for this file.

On the other hand, if your `debian/rules` had something like:

    execute_after_dh_fixperms:
        chown www-data:www-data debian/foo/var/lib/something-owned-by-www-data

Then the manifest would look something like:

    manifest-version: "0.1"
    packages:
        foo:
            transformations:
             - path-metadata:
                  path: var/lib/something-owned-by-www-data
                  owner: www-data
                  group: www-data

This can be combined with an explicit `mode: "02755"` if you also need a non-default mode.

The paths provided here support substitution variables (`usr/lib/{{DEB_HOST_MULTIARCH}}/...`) and
some _limited_ glob support (`usr/bin/sudo*`).

_Remember to merge your manifest with previous steps rather than replacing it!_  Note that
`debputy migrate-from-dh` will merge its changes into existing manifests and can safely be re-run
after adding/writing this base manifest.

### Convert your overrides for `dh_gencontrol` (if any)

If the package uses an override to choose a custom version for a binary package, then it is possible in `debputy`
by  using the `binary-version` key under the package.  Here is an example to force the package `foo` to have
epoch `1`:

    manifest-version: "0.1"
    packages:
        foo:
            # The foo package needs a different epoch because we took it over from a different
            # source package with higher epoch version
            binary-version: '1:{{DEB_VERSION_UPSTREAM_REVISION}}'

Useful if the source took over a binary package from a different source and that binary had a higher
epoch version.

Note that only limited manipulation of the version is supported, since your options are generally
limited to expanding one of the following version related variables:

 * `DEB_VERSION` - same definition as the one from `/usr/share/dpkg/pkg-info.mk` (from `dpkg`)
 * `DEB_VERSION_EPOCH_UPSTREAM` - ditto
 * `DEB_VERSION_UPSTREAM_REVISION` - ditto
 * `DEB_VERSION_UPSTREAM` - ditto

If the override is needed for dynamic substitution variables or binary versions that cannot be done with
the above substitutions, then it might be better to hold off on the conversion.

_Remember to merge your manifest with previous steps rather than replacing it!_  Note that
`debputy migrate-from-dh` will merge its changes into existing manifests and can safely be re-run
after adding/writing this base manifest.

## Step 4: Verify the build

At this stage, if there are no errors in your previous steps, you should be able to build your
changed package with `debputy`.  We recommend that you take time to verify this.  For some packages,
there was no conversion to do in the previous steps, and you would not even need a manifest at all
in this case.  However, we still recommend that you verify the build is successful here and now.

The `debputy` supports bit-for-bit reproducibility in its output. However, the output is not bit-for-bit
reproducible with `debhelper`. You are recommended to use `diffoscope` to compare the `debhelper`
built-version with the `debputy` built version to confirm that all the changes are benign.

However, `debputy` is bit-for-bit reproducible in its output with `(fakeroot) dpkg-deb -b`. Should you
spot a difference where `debputy` does not produce bit-for-bit identical results with `dpkg-deb` (tar
format, file ordering, etc.), then please file a bug against `debputy` with a reproducing test case.

Once you have verified your built, the conversion is done. :)  At this point, you can consider
looking at other features that `debputy` supports that might be useful to you.

[how-to guide]: https://documentation.divio.com/how-to-guides/
[#885580]: https://bugs.debian.org/885580
[#934499]: https://bugs.debian.org/934499
[debputy issue tracker]: https://salsa.debian.org/debian/debputy/-/issues