Adding upstream version 0.1.21.upstream/0.1.21

Signed-off-by: Daniel Baumann <daniel.baumann@progress-linux.org>

1 files changed, 536 insertions, 0 deletions

diff --git a/MIGRATING-A-DH-PLUGIN.md b/MIGRATING-A-DH-PLUGIN.md
new file mode 100644
index 0000000..3e4e227
--- /dev/null
+++ b/MIGRATING-A-DH-PLUGIN.md
@@ -0,0 +1,536 @@
+# Migrating a `debhelper` plugin to `debputy`
+
+_This is [how-to guide] and is primarily aimed at getting a task done._
+
+<!-- To writers and reviewers: Check the documentation against https://documentation.divio.com/ -->
+
+This document will help you convert a `debhelper` plugin / `debhelper` tool into a `debputy` plugin.
+Prerequisites for this how-to guide:
+
+ * You have a `debhelper` tool/plugin that you want to migrate.  Ideally a simple one as not all tools
+   can be migrated at this time.
+ * Many debhelper tools do not come with test cases, because no one has created a decent test framework
+   for them.  Therefore, consider how you intend to validate that the `debputy` plugin does not have any
+   (unplanned) regressions compared to `debhelper` tool.
+ * Depending on the features needed, you may need to provide a python hook for `debputy` to interact
+   with.
+   - Note: `debputy` will handle byte-compilation for you per
+     [Side note: Python byte-compilation](#side-note-python-byte-compilation)
+
+Note that during the conversion, you may find that `debputy` cannot support the requirements for your
+debhelper tool for now.  Feel free to file an issue for what is holding you back in the
+[debputy issue tracker].
+
+Prerequisites
+-------------
+
+This guide assumes familiarity with Debian packaging and the debhelper tool stack in general.  Notably,
+you are expected to be familiar with the `Dh_Lib.pm` API to the point of recognising references to said
+API and how to look up document for methods from said API.
+
+If the debhelper tool is not written in `Dh_Lib.pm`, then you will need to understand how to map the
+`Dh_Lib.pm` reference into the language/tool equivalent on your own.
+
+## Step 0: The approach taken
+
+The guide will assume you migrate one tool (a `dh_foo` command) at a time. If you have multiple tools
+that need to migrate together, you may want to review "Step 1" below for all tools before migrating to
+further steps.
+
+## Step 1: Analyze what features are required by the tools and the concept behind the helper
+
+For the purpose of this guide, we can roughly translate debhelper tools into one or more
+of the following categories.
+
+
+### Supported categories
+
+ * Install `debian/pkg.foo` *as-is* into a directory.
+    - This category uses a mix of `pkgfile` + `install_dir` + `install_file` / `install_prog`
+    - Example: `dh_installtmpfiles`
+ * If some file is installed in or beneath a directory, then (maybe) analyze the file, and apply metadata
+    (substvars, maintscripts, triggers, etc.). Note this does *not* apply to special-case of services.
+    While services follow this pattern, `debputy` will have special support for services.
+    - Typically, this category uses a bit of glob matching + (optionally) `open` +
+      `addsubstvars` / `autoscript` / `autotrigger`
+    - Example: `dh_installtmpfiles`
+    - *Counter* examples: `dh_installsystemd` (due to service rule, below).
+
+### Unsupported categories
+
+ * Read `debian/pkg.foo` and do something based on the content of said file.
+    - Typically, the category uses a mix of `pkgfile` + `filedoublearray` / `filearray` / `open(...)`.
+      The most common case of this is to install a list of files in the `debian/pkg.foo` file.
+    - In this scenario, the migration strategy should likely involve replacing `debian/pkg.foo` with
+      a section inside the `debian/debputy.manifest` file.
+    - Example: `dh_install`
+ * Any tool that manages services like `systemd`, `init.d` or `runit`.
+    - Typically, this category uses a bit of glob matching + (optionally) `open` +
+      `addsubstvars` / `autoscript` / `autotrigger`.
+    - This is unsupported because services will be a first-class feature in `debputy`, but the feature
+      is not fully developed yet.
+    - Example: `dh_installsystemd`
+ * Based on a set of rules, modify a set of files if certain criteria are met.
+    - Example: `dh_strip`, `dh_compress`, `dh_dwz`, `dh_strip_nondeterminism`, `dh_usrlocal`
+ * Run custom build system logic that cannot or has not been fit into the `debhelper` Buildsystem API.
+    - Example: `dh_cmake_install`, `dh_raku_build`, etc.
+ * "None of the above". There are also tools that have parts not fitting into any of the above
+    - Which just means the guide has no good help to offer you for migrating.
+    - Example: `dh_quilt_patch`
+
+As mentioned, a tool can have multiple categories at the same time.  As an example:
+
+ * The `dh_installtmpfiles` tool from debhelper is a mix between "installing `debian/pkg.tmpfiles` in to
+   `usr/lib/tmpfiles.d`" and "Generate a maintscript based on `<prefix>/tmpfiles.d/*.conf` globs".
+
+ * The `dh_usrlocal` tool from debhelper is a mix between "Generate a maintscript to create dirs in
+   `usr/local` as necessary on install and clean up on removal" and "Remove any directory from `usr/local`
+   installed into the package".
+
+
+When migrating a tool (or multiple tools), it is important to assert that all categories are supported by
+the `debputy` plugin API.  Otherwise, you will end with a half-finished plugin and realize you cannot
+complete the migration because you are missing a critical piece that `debputy` currently do not support.
+
+If your tool does not fit inside those two base categories, you cannot fully migrate the tool. You should
+consider whether it makes sense to continue without the missing features.
+
+## Step 2: Setup basic infrastructure
+
+This how-to guide assumes you will be using the debhelper integration via `dh-sequence-installdebputy`. To
+do that, add `dh-sequence-installdebputy` in the `Build-Depends` in `debian/control`. With this setup,
+any `debputy` plugin should be provided in the directory `debian/<package>.debputy-plugins` (replace `<package>`
+with the name of the package that should provide the plugin).
+
+In this directory, for each plugin, you can see the following files:
+
+    debian/package.debputy-plugins/my-plugin.json      # Metadata file (mandatory)
+    debian/package.debputy-plugins/my_plugin.py        # Python implementation (optional, note "_" rather than "_")
+    debian/package.debputy-plugins/my_plugin_check.py  # tests (optional, run with py.test, note "_" rather than "_")
+                                                       # Alternative names such as _test.py or _check_foo.py works too
+
+A basic version of the JSON plugin metadata file could be:
+
+
+```json
+{
+  "plugin-initializer": "initialize_my_plugin",
+  "api-compat-level": 1,
+  "packager-provided-files": [
+    {
+      "stem": "foo",
+      "installed-path": "/usr/share/foo/{name}.conf"
+    }
+  ]
+}
+```
+
+This example JSON assumes that you will be providing both python code (`plugin-intializer`, requires a Python
+implementation file) and packager provided files (`packager-provided-files`). In some cases, you will *not*
+need all of these features. Notably, if you find that you do not need any feature requiring python code,
+you are recommended to remove `plugin-initializer` from the plugin JSON file.
+
+A Python-based plugin for `debputy` plugin starts with an initialization function like this:
+
+```python
+from debputy.plugin.api import DebputyPluginInitializer
+
+def initialize_my_plugin(api: DebputyPluginInitializer):
+    pass
+```
+
+Remember to replace the values in the JSON, so they match your plugin. The keys are:
+
+ * `plugin-initializer`: (Python plugin-only) The function `debputy` should call to initialize your plugin. This is
+   the function we  just defined in the previous example). The plugin loader requires this initialization function to
+   be a top level function of the module (that is, `getattr(module, plugin_initializer)` must return the initializer
+   function).
+ * `module`: (Python plugin-only, optional) The python module the `plugin-initializer` function is defined in.
+   If omitted, `debputy` will derive the module name from the plugin name (replace `-` with `_`). When omitted,
+   the Python module can be placed next to the `.json` file.  This is useful single file plugins.
+ * `api-compat-level`: This is the API compat level of `debputy` required to load the
+   plugin. This defines how `debputy` will load the plugin and is to ensure that
+   `debputy`'s plugin API can evolve gracefully.  For now, only one version is supported
+   and that is `1`.
+ * `packager-provided-files`: Declares packager provided files. This keyword is covered in the section below.
+
+This file then has to be installed into the `debputy` plugin directory.
+
+With this you have an empty plugin that `debputy` can load, but it does not provide any features.
+
+## Step 3: Provide packager provided files (Category 1 tools)
+
+*This step only applies if the tool in question automatically installs `debian/pkg.foo` in some predefined path
+like `dh_installtmpfiles` does.  If not, please skip this section as it is not relevant to your case.*
+
+You can ask `debputy` to automatically detect `debian/pkg.foo` files and install them into a concrete directory
+via the plugin. You have two basic options for providing packager provided files.
+
+ 1) A pure-JSON plugin variant.
+ 2) A Python plugin variant.
+
+This guide will show you both. The pure-JSON variant is recommended assuming it satisfies your needs as it is
+the simplest to get started with and have fewer moving parts.  The Python plugin has slightly more features
+for the "1% special cases".
+
+### Packager provided files primer on naming convention
+
+This section will break the filename `debian/g++-3.0.name.segment.my.file.type.amd64` down into parts and name
+the terms `debputy` uses for them and how they are used. If you already know the terms, you can skip this section.
+
+This example breaks into 4 pieces, in order:
+
+ * An optional package name (`g++-3.0`). Decides which package the file applies to (defaulting to the main package
+   if omitted).  It is also used as the default "installed as name".
+
+ * An optional "name segment" (`name.segment`). Named so after the `--name` parameter from `debhelper` that is
+   needed for `debhelper` to detect files with the segment and because it also changes the default "installed as
+   name" (both in `debhelper` and `debputy`). When omitted, the package name decides the "installed as name".
+
+ * The "stem" (`my.file.type`). This part never had an official name in `debhelper` other than `filename`
+   or `basename`.
+
+ * An optional architecture restriction. It is used in special cases like `debian/foo.symbols.amd64` where you
+   have architecture specific details in the file.
+
+In `debputy`, when you register a packager provided file, you have some freedom in which of these should apply
+to your file.  The architecture restriction is rarely used and disabled by default, whereas the "name segment"
+is available by default. When the "name segment" is enabled, the packager is able to:
+
+ 1) choose a different filename than the package name (by using `debian/package.desired-name.foo` instead of
+    `debian/package.foo`)
+
+ 2) provide multiple files for the same package (`debian/package.foo` *and* `debian/package.some-name.foo`).
+
+If it is important that a package can provide at most one file, and it must be named after the package itself,
+you are advised to disable to name segment.
+
+### JSON-based packager provided files (Category 1 tools)
+
+With the pure JSON based method, the plugin JSON file should contain all the relevant details. A minimal
+example is:
+
+```json
+{
+  "api-compat-level": 1,
+  "packager-provided-files": [
+    {
+      "stem": "foo",
+      "installed-path": "/usr/share/foo/{name}.conf",
+      "reference-documentation": {
+        "description": "Some possibly multi-line description related to foo",
+        "format-documentation-uris": ["man:foo.conf(5)"]
+      }
+    }
+  ]
+}
+```
+(This file should be saved as `debian/package.debputy-plugins/my-plugin.json`.)
+
+This plugin snippet would provide one packager provided files and nothing else. When loading the plugin, `debputy`
+would detect files such as `debian/package.foo` and install them into `/usr/share/foo/package.conf`.
+
+As shown in the example. the packager provided files are declared as a list in the attribute
+`packager-provided-files`. Each element in that list is an object with the following keys:
+
+ * `name` (required): The "stem" of the file. In the example above, `"foo"` is used meaning that `debputy`
+   would detect `debian/package.foo`. Note that this value must be unique across all packager provided files known
+   by `debputy` and all loaded plugins.
+
+ * `installed-path` (required): A format string describing where the file should be installed. This is
+   `"/usr/share/foo/{name}.conf"` from the example above and leads to `debian/package.foo` being installed
+   as `/usr/share/foo/package.conf`.
+
+   The following placeholders are supported:
+
+     * `{name}` - The name in the name segment (defaulting the package name if no name segment is given)
+     * `{priority}` / `{priority:02}` - The priority of the file. Only provided priorities are used (that
+       is, `default-priority` is provided).  The latter variant ensuring that the priority takes at least
+       two characters and the `0` character is left-padded for priorities that takes less than two
+       characters.
+     * `{owning_package}` - The name of the package.  Should only be used when `{name}` alone is insufficient.
+       If you do not want the "name" segment in the first place, set `allow-name-segment` to `false` instead.
+
+     The path is always interpreted as relative to the binary package root.
+
+ * `default-mode` (optional): If provided, it must be an octal mode (such as `"0755"`), which defines the mode
+   that `debputy` will use by default for this kind of file. Note that the mode must be provided as a string.
+
+ * `default-priority` (optional): If provided, it must be an integer declaring the default priority of the file,
+   which will be a part of the filename.  The `installed-path` will be required to have the `{priority}` or
+   `{priority:02}` placeholder. This attribute is useful for directories where the files are read in "sorted"
+   and there is a convention of naming files like `20-foo.conf` to ensure files are processed in the correct
+   order.
+
+ * `allow-name-segment` (optional): If provided, it must be a boolean (defaults to `true`), which determines
+   whether `debputy` should allow a name segment for the file.
+
+ * `allow-architecture-segment` (optional): If provided, it must be a boolean (defaults to `false`), which determines
+   whether `debputy` should allow an architecture restriction for the file.
+
+ * `reference-documentation` (optional): If provided, the following keys can be used:
+
+    * `description` (optional): If provided, it is used as a description for the file if the user requests
+       documentation about the file.
+
+    * `format-documentation-uris` (optional): If provided, it must be a list of URIs that describes the format
+       of the file. `http`, `https` and `man` URIs are recommended.
+
+
+### Python-based packager provided files (Category 1 tools)  [NOT RECOMMENDED]
+
+**This section uses a Python-based API, which is not recommended at this time as the logistics are not finished**
+
+With the Python based method, the plugin JSON file should contain a reference to the python module. A minimal
+example is:
+
+```json
+{
+  "api-compat-level": 1,
+  "plugin-initializer": "initialize_my_plugin"
+}
+```
+(This file should be saved as `debian/package.debputy-plugins/my-plugin.json`.)
+
+The python module file should then provide the `initialize_my_plugin` function, which could look something like this:
+
+```python
+from debputy.plugin.api import DebputyPluginInitializer
+
+def initialize_my_plugin(api: DebputyPluginInitializer):
+    api.packager_provided_file(
+        "foo",  # This is the "foo" in "debian/pkg.foo"
+        "/usr/share/foo/{name}.conf",  # This is the directory to install them at.
+    )
+```
+(This file would be saved as `debian/package.debputy-plugins/my_plugin.py` assuming `my-plugin.json` was
+used for the metadata file)
+
+This example code would make `debputy` install `debian/my-pkg.foo` as `/usr/share/foo/my-pkg.conf` provided the
+plugin is  loaded. Please review the API docs for the full details of options.
+
+This can be done via the interactive python shell with:
+
+```python
+import sys
+sys.path.insert(0, "/usr/share/dh-debputy/")
+from debputy.plugin.api import DebputyPluginInitializer
+help(DebputyPluginInitializer.packager_provided_file)
+```
+
+### Testing your plugin
+
+If you are the type that like to provide tests for your code, the following `py.test` snippet can get you started:
+
+```python
+from debputy.plugin.api.test_api import initialize_plugin_under_test
+
+
+def test_packager_provided_files():
+    plugin = initialize_plugin_under_test()
+    ppf_by_stem = plugin.packager_provided_files_by_stem()
+    assert ppf_by_stem.keys() == {'foo'}
+    foo_file = ppf_by_stem['foo']
+
+    assert foo_file.stem == 'foo'
+
+    # Note, the discard part is the installed into directory, and it is skipped because `debputy`
+    # normalize the directory as an implementation detail and the test would depend on said detail
+    # for no good reason in this case.  If your case have the variable in the directory part, tweak
+    # the test as necessary.
+    _, basename = foo_file.compute_dest("my-package")
+    assert basename == 'my-package.conf'
+    # Test other things you might have configured:
+    #   assert foo_file.default_mode == 0o755   # ... if the file is to be executable
+    #   assert foo_file.default_priority == 20  # ... if the file has priority
+    #   ...
+```
+(This file would be saved as `debian/package.debputy-plugins/my_plugin_check.py` assuming `my-plugin.json` was
+used for the metadata file)
+
+This test works the same regardless of whether the JSON-based or Python-based method was chosen.
+
+## Step 4: Migrate metadata detection (Category 3 tools)  [NOT RECOMMENDED]
+
+*This step only applies if the tool in question generates substvars, maintscripts or triggers based on
+certain paths being present or having certain content like `dh_installtmpfiles` does.  However,
+this section does **NOT** apply to service management tools (such as `dh_installsystemd`). If not, please
+skip this section as it is not relevant to your case.*
+
+For dealing with substvars, maintscripts and triggers, the plugin will need to register a function that
+can perform the detection. The `debputy` API refers to it as a "detector" and functionally it behaves like
+a "callback" or "hook". The "detector" will be run once per package that it applies to with some context and
+is expected to register the changes it wants.
+
+A short example is:
+
+```python
+from debputy.plugin.api import (
+   DebputyPluginInitializer,
+   VirtualPath,
+   BinaryCtrlAccessor,
+   PackageProcessingContext,
+)
+
+
+def initialize_my_plugin(api: DebputyPluginInitializer):
+   # ... remember to preserve any existing code here that you may have had from previous steps.
+   api.metadata_or_maintscript_detector(
+      "foo-detector",  # This is an ID of the detector. It is part of the plugins API and should not change.
+      # Packagers see it if it triggers an error and will also be able to disable by this ID.
+      detect_foo_files,  # This is the detector (hook) itself.
+   )
+
+
+def detect_foo_files(fs_root: VirtualPath,
+                     ctrl: BinaryCtrlAccessor,
+                     context: PackageProcessingContext,
+                     ) -> None:
+   # If for some reason the hook should not apply to all packages, and `metadata_or_maintscript_detector` does not
+   # provide a filter for it, then you just do an `if <should not apply>: return`
+   if not context.binary_package.is_arch_all:
+      # For some reason, our hook only applies to arch:all packages.
+      return
+
+   foo_dir = fs_root.lookup("usr/share/foo")
+   if not foo_dir:
+      return
+
+   conf_files = [path.absolute for path in foo_dir.iterdir if path.is_file and path.name.endswith(".conf")]
+   if not conf_files:
+      return
+   ctrl.substvars.add_dependency("misc:Depends", "foo-utils")
+   conf_files_escaped = ctrl.maintscript.escape_shell_words(*conf_files)
+   # With multi-line snippets, consider:
+   #
+   # snippet = textwrap.dedent("""\
+   #     ... content here using {var}
+   # """).format(var=value)
+   #
+   # (As that tends to result in more readable snippets, when the dedent happens before formatting)
+   snippet = f"foo-analyze --install {conf_files_escaped}"
+   ctrl.maintscript.on_configure(snippet)
+```
+(This file would be saved as `debian/package.debputy-plugins/my_plugin.py`)
+
+This code would register the `detect_foo_files` function as a metadata hook. It would be run for all regular `deb`
+packages processed by `debputy` (`udeb` requires opt-in, auto-generated packages such as `-dbgsym` cannot be
+targeted).
+
+The hook conditionally generates a dependency (via the `${misc:Depends}` substvar) on `foo-utils` and a `postinst`
+snippet to be run when the package is configured.
+
+An important thing to note is that `debputy` have *NOT* materialized the package anywhere. Instead, `debputy`
+provides an in-memory view of the file system (`fs_root`) and related path metadata that the plugin should base its
+analysis of. The in-memory view of the file system can have virtual paths that are not backed by any real
+path on the file system.  This commonly happens for directories and symlinks - and during tests, also for files.
+
+
+In addition to the python code above, remember that the plugin JSON file should contain a reference to the python
+module. A minimal example for this is:
+
+```json
+{
+  "api-compat-level": 1,
+  "plugin-initializer": "initialize_my_plugin"
+}
+```
+(This file should be saved into `debian/package.debputy-plugins/my-plugin.json` assuming `my_plugin.py` was
+used for the module file)
+
+
+If you are the type that like to provide tests for your code, the following `py.test` snippet can get you started:
+
+```python
+from debputy.plugin.api.test_api import initialize_plugin_under_test, build_virtual_file_system, \
+    package_metadata_context
+
+
+def test_packager_provided_files():
+    plugin = initialize_plugin_under_test()
+    detector_id = 'foo-detector'
+
+    fs_root = build_virtual_file_system([
+        '/usr/share/foo/foo.conf'  # Creates a virtual (no-content) file.
+        # Use virtual_path_def(..., fs_path="/path") if your detector needs to read the file
+        # NB: You have to create that "/path" yourself.
+    ])
+
+    metadata = plugin.run_metadata_detector(
+        detector_id,
+        fs_root,
+        # Test with an arch:any package. The test framework will supply a minimum number of fields
+        # (e.g., "Package") so you do not *have* to provide them if you do not need them.
+        # That is also why providing `Architecture` alone works here.
+        context=package_metadata_context(package_fields={'Architecture': 'any'})
+    )
+    # Per definition of our detector, there should be no dependency added (even though the file is there)
+    assert 'misc:Depends' not in metadata.substvars
+    # Nor should any maintscripts have been added
+    assert metadata.maintscripts() == []
+
+    metadata = plugin.run_metadata_detector(
+        detector_id,
+        fs_root,
+        # This time, we test with an arch:all package
+        context=package_metadata_context(package_fields={'Architecture': 'all'})
+    )
+
+    assert metadata.substvars['misc:Depends'] == 'foo-utils'
+
+    # You could also have added `maintscript='postinst'` to filter by which script it was added to.
+    snippets = metadata.maintscripts()
+    # There should be exactly one snippet
+    assert len(snippets) == 1
+    snippet = snippets[0]
+    # And we can verify that the snippet is as expected.
+    assert snippet.maintscript == 'postinst'
+    assert snippet.registration_method == 'on_configure'
+    assert 'foo-analyze --install /usr/share/foo/foo.conf' in snippet.plugin_provided_script
+```
+(This file should be saved into `debian/package.debputy-plugins/my_plugin_check.json` assuming `my_plugin.py` was
+used for the module file)
+
+This test works the same regardless of whether the JSON-based or Python-based method was chosen.
+
+## Step 4: Have your package provide `debputy-plugin-X`
+
+All third-party `debputy` plugins are loaded by adding a build dependency on `debputy-plugin-X`,
+where `X` is the basename of the plugin JSON file.  Accordingly, any package providing a `debputy` plugin
+must either be named `debputy-plugin-X` or provide `debputy-plugin-X` (where `X` is replaced with the concrete
+plugin name).
+
+## Step 5: Running the tests
+
+To run the tests, you have two options:
+
+ 1) Add `python3-pytest <!nocheck>` to the `Build-Depends` in `debian/control`. This will cause
+    `dh_installdebputy` to run the tests when the package is built. The tests will be skipped
+    if `DEB_BUILD_OPTIONS` contains `nocheck` per Debian Policy.  You will also need to have
+    the `debputy` command in PATH. This generally happens as a side effect of the
+    `dh-sequence-installdebputy` build dependency.
+
+ 2) Add `autopkgtest-pkg-debputy` to the `Testsuite` field in `debian/control`.  This will cause
+    the Debian CI framework (via the `autodep8` command) to generate an autopkgtest that will
+    run the plugin tests against the installed plugin.
+
+Using both options where possible is generally preferable.
+
+If your upstream uses a Python test framework that auto-detects tests such as `py.test`, you may
+find that it picks up the `debputy` plugin or its tests. If this is causing you issues, please have
+a  look at the `dh_installdebputy` manpage, which have a section dedicated to how to resolve these
+issues.
+
+## Side note: Python byte-compilation
+
+When you install a `debputy` plugin into `/usr/share/debputy/debputy/plugins`, then `debputy` will
+manage the Python byte-compilation for you.
+
+## Closing
+
+You should now either have done all the basic steps of migrating the debhelper tool to `debputy`
+or discovered some feature that the guide did not cover. In the latter case, please have a look
+at the [debputy issue tracker] and consider whether you should file a feature request for it.
+
+[how-to guide]: https://documentation.divio.com/how-to-guides/
+[debputy issue tracker]: https://salsa.debian.org/debian/debputy/-/issues