diff options
Diffstat (limited to 'MIGRATING-A-DH-PLUGIN.md')
| -rw-r--r-- | MIGRATING-A-DH-PLUGIN.md | 536 |
1 files changed, 536 insertions, 0 deletions
diff --git a/MIGRATING-A-DH-PLUGIN.md b/MIGRATING-A-DH-PLUGIN.md new file mode 100644 index 0000000..3e4e227 --- /dev/null +++ b/MIGRATING-A-DH-PLUGIN.md @@ -0,0 +1,536 @@ +# Migrating a `debhelper` plugin to `debputy` + +_This is [how-to guide] and is primarily aimed at getting a task done._ + +<!-- To writers and reviewers: Check the documentation against https://documentation.divio.com/ --> + +This document will help you convert a `debhelper` plugin / `debhelper` tool into a `debputy` plugin. +Prerequisites for this how-to guide: + + * You have a `debhelper` tool/plugin that you want to migrate. Ideally a simple one as not all tools + can be migrated at this time. + * Many debhelper tools do not come with test cases, because no one has created a decent test framework + for them. Therefore, consider how you intend to validate that the `debputy` plugin does not have any + (unplanned) regressions compared to `debhelper` tool. + * Depending on the features needed, you may need to provide a python hook for `debputy` to interact + with. + - Note: `debputy` will handle byte-compilation for you per + [Side note: Python byte-compilation](#side-note-python-byte-compilation) + +Note that during the conversion, you may find that `debputy` cannot support the requirements for your +debhelper tool for now. Feel free to file an issue for what is holding you back in the +[debputy issue tracker]. + +Prerequisites +------------- + +This guide assumes familiarity with Debian packaging and the debhelper tool stack in general. Notably, +you are expected to be familiar with the `Dh_Lib.pm` API to the point of recognising references to said +API and how to look up document for methods from said API. + +If the debhelper tool is not written in `Dh_Lib.pm`, then you will need to understand how to map the +`Dh_Lib.pm` reference into the language/tool equivalent on your own. + +## Step 0: The approach taken + +The guide will assume you migrate one tool (a `dh_foo` command) at a time. If you have multiple tools +that need to migrate together, you may want to review "Step 1" below for all tools before migrating to +further steps. + +## Step 1: Analyze what features are required by the tools and the concept behind the helper + +For the purpose of this guide, we can roughly translate debhelper tools into one or more +of the following categories. + + +### Supported categories + + * Install `debian/pkg.foo` *as-is* into a directory. + - This category uses a mix of `pkgfile` + `install_dir` + `install_file` / `install_prog` + - Example: `dh_installtmpfiles` + * If some file is installed in or beneath a directory, then (maybe) analyze the file, and apply metadata + (substvars, maintscripts, triggers, etc.). Note this does *not* apply to special-case of services. + While services follow this pattern, `debputy` will have special support for services. + - Typically, this category uses a bit of glob matching + (optionally) `open` + + `addsubstvars` / `autoscript` / `autotrigger` + - Example: `dh_installtmpfiles` + - *Counter* examples: `dh_installsystemd` (due to service rule, below). + +### Unsupported categories + + * Read `debian/pkg.foo` and do something based on the content of said file. + - Typically, the category uses a mix of `pkgfile` + `filedoublearray` / `filearray` / `open(...)`. + The most common case of this is to install a list of files in the `debian/pkg.foo` file. + - In this scenario, the migration strategy should likely involve replacing `debian/pkg.foo` with + a section inside the `debian/debputy.manifest` file. + - Example: `dh_install` + * Any tool that manages services like `systemd`, `init.d` or `runit`. + - Typically, this category uses a bit of glob matching + (optionally) `open` + + `addsubstvars` / `autoscript` / `autotrigger`. + - This is unsupported because services will be a first-class feature in `debputy`, but the feature + is not fully developed yet. + - Example: `dh_installsystemd` + * Based on a set of rules, modify a set of files if certain criteria are met. + - Example: `dh_strip`, `dh_compress`, `dh_dwz`, `dh_strip_nondeterminism`, `dh_usrlocal` + * Run custom build system logic that cannot or has not been fit into the `debhelper` Buildsystem API. + - Example: `dh_cmake_install`, `dh_raku_build`, etc. + * "None of the above". There are also tools that have parts not fitting into any of the above + - Which just means the guide has no good help to offer you for migrating. + - Example: `dh_quilt_patch` + +As mentioned, a tool can have multiple categories at the same time. As an example: + + * The `dh_installtmpfiles` tool from debhelper is a mix between "installing `debian/pkg.tmpfiles` in to + `usr/lib/tmpfiles.d`" and "Generate a maintscript based on `<prefix>/tmpfiles.d/*.conf` globs". + + * The `dh_usrlocal` tool from debhelper is a mix between "Generate a maintscript to create dirs in + `usr/local` as necessary on install and clean up on removal" and "Remove any directory from `usr/local` + installed into the package". + + +When migrating a tool (or multiple tools), it is important to assert that all categories are supported by +the `debputy` plugin API. Otherwise, you will end with a half-finished plugin and realize you cannot +complete the migration because you are missing a critical piece that `debputy` currently do not support. + +If your tool does not fit inside those two base categories, you cannot fully migrate the tool. You should +consider whether it makes sense to continue without the missing features. + +## Step 2: Setup basic infrastructure + +This how-to guide assumes you will be using the debhelper integration via `dh-sequence-installdebputy`. To +do that, add `dh-sequence-installdebputy` in the `Build-Depends` in `debian/control`. With this setup, +any `debputy` plugin should be provided in the directory `debian/<package>.debputy-plugins` (replace `<package>` +with the name of the package that should provide the plugin). + +In this directory, for each plugin, you can see the following files: + + debian/package.debputy-plugins/my-plugin.json # Metadata file (mandatory) + debian/package.debputy-plugins/my_plugin.py # Python implementation (optional, note "_" rather than "_") + debian/package.debputy-plugins/my_plugin_check.py # tests (optional, run with py.test, note "_" rather than "_") + # Alternative names such as _test.py or _check_foo.py works too + +A basic version of the JSON plugin metadata file could be: + + +```json +{ + "plugin-initializer": "initialize_my_plugin", + "api-compat-level": 1, + "packager-provided-files": [ + { + "stem": "foo", + "installed-path": "/usr/share/foo/{name}.conf" + } + ] +} +``` + +This example JSON assumes that you will be providing both python code (`plugin-intializer`, requires a Python +implementation file) and packager provided files (`packager-provided-files`). In some cases, you will *not* +need all of these features. Notably, if you find that you do not need any feature requiring python code, +you are recommended to remove `plugin-initializer` from the plugin JSON file. + +A Python-based plugin for `debputy` plugin starts with an initialization function like this: + +```python +from debputy.plugin.api import DebputyPluginInitializer + +def initialize_my_plugin(api: DebputyPluginInitializer): + pass +``` + +Remember to replace the values in the JSON, so they match your plugin. The keys are: + + * `plugin-initializer`: (Python plugin-only) The function `debputy` should call to initialize your plugin. This is + the function we just defined in the previous example). The plugin loader requires this initialization function to + be a top level function of the module (that is, `getattr(module, plugin_initializer)` must return the initializer + function). + * `module`: (Python plugin-only, optional) The python module the `plugin-initializer` function is defined in. + If omitted, `debputy` will derive the module name from the plugin name (replace `-` with `_`). When omitted, + the Python module can be placed next to the `.json` file. This is useful single file plugins. + * `api-compat-level`: This is the API compat level of `debputy` required to load the + plugin. This defines how `debputy` will load the plugin and is to ensure that + `debputy`'s plugin API can evolve gracefully. For now, only one version is supported + and that is `1`. + * `packager-provided-files`: Declares packager provided files. This keyword is covered in the section below. + +This file then has to be installed into the `debputy` plugin directory. + +With this you have an empty plugin that `debputy` can load, but it does not provide any features. + +## Step 3: Provide packager provided files (Category 1 tools) + +*This step only applies if the tool in question automatically installs `debian/pkg.foo` in some predefined path +like `dh_installtmpfiles` does. If not, please skip this section as it is not relevant to your case.* + +You can ask `debputy` to automatically detect `debian/pkg.foo` files and install them into a concrete directory +via the plugin. You have two basic options for providing packager provided files. + + 1) A pure-JSON plugin variant. + 2) A Python plugin variant. + +This guide will show you both. The pure-JSON variant is recommended assuming it satisfies your needs as it is +the simplest to get started with and have fewer moving parts. The Python plugin has slightly more features +for the "1% special cases". + +### Packager provided files primer on naming convention + +This section will break the filename `debian/g++-3.0.name.segment.my.file.type.amd64` down into parts and name +the terms `debputy` uses for them and how they are used. If you already know the terms, you can skip this section. + +This example breaks into 4 pieces, in order: + + * An optional package name (`g++-3.0`). Decides which package the file applies to (defaulting to the main package + if omitted). It is also used as the default "installed as name". + + * An optional "name segment" (`name.segment`). Named so after the `--name` parameter from `debhelper` that is + needed for `debhelper` to detect files with the segment and because it also changes the default "installed as + name" (both in `debhelper` and `debputy`). When omitted, the package name decides the "installed as name". + + * The "stem" (`my.file.type`). This part never had an official name in `debhelper` other than `filename` + or `basename`. + + * An optional architecture restriction. It is used in special cases like `debian/foo.symbols.amd64` where you + have architecture specific details in the file. + +In `debputy`, when you register a packager provided file, you have some freedom in which of these should apply +to your file. The architecture restriction is rarely used and disabled by default, whereas the "name segment" +is available by default. When the "name segment" is enabled, the packager is able to: + + 1) choose a different filename than the package name (by using `debian/package.desired-name.foo` instead of + `debian/package.foo`) + + 2) provide multiple files for the same package (`debian/package.foo` *and* `debian/package.some-name.foo`). + +If it is important that a package can provide at most one file, and it must be named after the package itself, +you are advised to disable to name segment. + +### JSON-based packager provided files (Category 1 tools) + +With the pure JSON based method, the plugin JSON file should contain all the relevant details. A minimal +example is: + +```json +{ + "api-compat-level": 1, + "packager-provided-files": [ + { + "stem": "foo", + "installed-path": "/usr/share/foo/{name}.conf", + "reference-documentation": { + "description": "Some possibly multi-line description related to foo", + "format-documentation-uris": ["man:foo.conf(5)"] + } + } + ] +} +``` +(This file should be saved as `debian/package.debputy-plugins/my-plugin.json`.) + +This plugin snippet would provide one packager provided files and nothing else. When loading the plugin, `debputy` +would detect files such as `debian/package.foo` and install them into `/usr/share/foo/package.conf`. + +As shown in the example. the packager provided files are declared as a list in the attribute +`packager-provided-files`. Each element in that list is an object with the following keys: + + * `name` (required): The "stem" of the file. In the example above, `"foo"` is used meaning that `debputy` + would detect `debian/package.foo`. Note that this value must be unique across all packager provided files known + by `debputy` and all loaded plugins. + + * `installed-path` (required): A format string describing where the file should be installed. This is + `"/usr/share/foo/{name}.conf"` from the example above and leads to `debian/package.foo` being installed + as `/usr/share/foo/package.conf`. + + The following placeholders are supported: + + * `{name}` - The name in the name segment (defaulting the package name if no name segment is given) + * `{priority}` / `{priority:02}` - The priority of the file. Only provided priorities are used (that + is, `default-priority` is provided). The latter variant ensuring that the priority takes at least + two characters and the `0` character is left-padded for priorities that takes less than two + characters. + * `{owning_package}` - The name of the package. Should only be used when `{name}` alone is insufficient. + If you do not want the "name" segment in the first place, set `allow-name-segment` to `false` instead. + + The path is always interpreted as relative to the binary package root. + + * `default-mode` (optional): If provided, it must be an octal mode (such as `"0755"`), which defines the mode + that `debputy` will use by default for this kind of file. Note that the mode must be provided as a string. + + * `default-priority` (optional): If provided, it must be an integer declaring the default priority of the file, + which will be a part of the filename. The `installed-path` will be required to have the `{priority}` or + `{priority:02}` placeholder. This attribute is useful for directories where the files are read in "sorted" + and there is a convention of naming files like `20-foo.conf` to ensure files are processed in the correct + order. + + * `allow-name-segment` (optional): If provided, it must be a boolean (defaults to `true`), which determines + whether `debputy` should allow a name segment for the file. + + * `allow-architecture-segment` (optional): If provided, it must be a boolean (defaults to `false`), which determines + whether `debputy` should allow an architecture restriction for the file. + + * `reference-documentation` (optional): If provided, the following keys can be used: + + * `description` (optional): If provided, it is used as a description for the file if the user requests + documentation about the file. + + * `format-documentation-uris` (optional): If provided, it must be a list of URIs that describes the format + of the file. `http`, `https` and `man` URIs are recommended. + + +### Python-based packager provided files (Category 1 tools) [NOT RECOMMENDED] + +**This section uses a Python-based API, which is not recommended at this time as the logistics are not finished** + +With the Python based method, the plugin JSON file should contain a reference to the python module. A minimal +example is: + +```json +{ + "api-compat-level": 1, + "plugin-initializer": "initialize_my_plugin" +} +``` +(This file should be saved as `debian/package.debputy-plugins/my-plugin.json`.) + +The python module file should then provide the `initialize_my_plugin` function, which could look something like this: + +```python +from debputy.plugin.api import DebputyPluginInitializer + +def initialize_my_plugin(api: DebputyPluginInitializer): + api.packager_provided_file( + "foo", # This is the "foo" in "debian/pkg.foo" + "/usr/share/foo/{name}.conf", # This is the directory to install them at. + ) +``` +(This file would be saved as `debian/package.debputy-plugins/my_plugin.py` assuming `my-plugin.json` was +used for the metadata file) + +This example code would make `debputy` install `debian/my-pkg.foo` as `/usr/share/foo/my-pkg.conf` provided the +plugin is loaded. Please review the API docs for the full details of options. + +This can be done via the interactive python shell with: + +```python +import sys +sys.path.insert(0, "/usr/share/dh-debputy/") +from debputy.plugin.api import DebputyPluginInitializer +help(DebputyPluginInitializer.packager_provided_file) +``` + +### Testing your plugin + +If you are the type that like to provide tests for your code, the following `py.test` snippet can get you started: + +```python +from debputy.plugin.api.test_api import initialize_plugin_under_test + + +def test_packager_provided_files(): + plugin = initialize_plugin_under_test() + ppf_by_stem = plugin.packager_provided_files_by_stem() + assert ppf_by_stem.keys() == {'foo'} + foo_file = ppf_by_stem['foo'] + + assert foo_file.stem == 'foo' + + # Note, the discard part is the installed into directory, and it is skipped because `debputy` + # normalize the directory as an implementation detail and the test would depend on said detail + # for no good reason in this case. If your case have the variable in the directory part, tweak + # the test as necessary. + _, basename = foo_file.compute_dest("my-package") + assert basename == 'my-package.conf' + # Test other things you might have configured: + # assert foo_file.default_mode == 0o755 # ... if the file is to be executable + # assert foo_file.default_priority == 20 # ... if the file has priority + # ... +``` +(This file would be saved as `debian/package.debputy-plugins/my_plugin_check.py` assuming `my-plugin.json` was +used for the metadata file) + +This test works the same regardless of whether the JSON-based or Python-based method was chosen. + +## Step 4: Migrate metadata detection (Category 3 tools) [NOT RECOMMENDED] + +*This step only applies if the tool in question generates substvars, maintscripts or triggers based on +certain paths being present or having certain content like `dh_installtmpfiles` does. However, +this section does **NOT** apply to service management tools (such as `dh_installsystemd`). If not, please +skip this section as it is not relevant to your case.* + +For dealing with substvars, maintscripts and triggers, the plugin will need to register a function that +can perform the detection. The `debputy` API refers to it as a "detector" and functionally it behaves like +a "callback" or "hook". The "detector" will be run once per package that it applies to with some context and +is expected to register the changes it wants. + +A short example is: + +```python +from debputy.plugin.api import ( + DebputyPluginInitializer, + VirtualPath, + BinaryCtrlAccessor, + PackageProcessingContext, +) + + +def initialize_my_plugin(api: DebputyPluginInitializer): + # ... remember to preserve any existing code here that you may have had from previous steps. + api.metadata_or_maintscript_detector( + "foo-detector", # This is an ID of the detector. It is part of the plugins API and should not change. + # Packagers see it if it triggers an error and will also be able to disable by this ID. + detect_foo_files, # This is the detector (hook) itself. + ) + + +def detect_foo_files(fs_root: VirtualPath, + ctrl: BinaryCtrlAccessor, + context: PackageProcessingContext, + ) -> None: + # If for some reason the hook should not apply to all packages, and `metadata_or_maintscript_detector` does not + # provide a filter for it, then you just do an `if <should not apply>: return` + if not context.binary_package.is_arch_all: + # For some reason, our hook only applies to arch:all packages. + return + + foo_dir = fs_root.lookup("usr/share/foo") + if not foo_dir: + return + + conf_files = [path.absolute for path in foo_dir.iterdir if path.is_file and path.name.endswith(".conf")] + if not conf_files: + return + ctrl.substvars.add_dependency("misc:Depends", "foo-utils") + conf_files_escaped = ctrl.maintscript.escape_shell_words(*conf_files) + # With multi-line snippets, consider: + # + # snippet = textwrap.dedent("""\ + # ... content here using {var} + # """).format(var=value) + # + # (As that tends to result in more readable snippets, when the dedent happens before formatting) + snippet = f"foo-analyze --install {conf_files_escaped}" + ctrl.maintscript.on_configure(snippet) +``` +(This file would be saved as `debian/package.debputy-plugins/my_plugin.py`) + +This code would register the `detect_foo_files` function as a metadata hook. It would be run for all regular `deb` +packages processed by `debputy` (`udeb` requires opt-in, auto-generated packages such as `-dbgsym` cannot be +targeted). + +The hook conditionally generates a dependency (via the `${misc:Depends}` substvar) on `foo-utils` and a `postinst` +snippet to be run when the package is configured. + +An important thing to note is that `debputy` have *NOT* materialized the package anywhere. Instead, `debputy` +provides an in-memory view of the file system (`fs_root`) and related path metadata that the plugin should base its +analysis of. The in-memory view of the file system can have virtual paths that are not backed by any real +path on the file system. This commonly happens for directories and symlinks - and during tests, also for files. + + +In addition to the python code above, remember that the plugin JSON file should contain a reference to the python +module. A minimal example for this is: + +```json +{ + "api-compat-level": 1, + "plugin-initializer": "initialize_my_plugin" +} +``` +(This file should be saved into `debian/package.debputy-plugins/my-plugin.json` assuming `my_plugin.py` was +used for the module file) + + +If you are the type that like to provide tests for your code, the following `py.test` snippet can get you started: + +```python +from debputy.plugin.api.test_api import initialize_plugin_under_test, build_virtual_file_system, \ + package_metadata_context + + +def test_packager_provided_files(): + plugin = initialize_plugin_under_test() + detector_id = 'foo-detector' + + fs_root = build_virtual_file_system([ + '/usr/share/foo/foo.conf' # Creates a virtual (no-content) file. + # Use virtual_path_def(..., fs_path="/path") if your detector needs to read the file + # NB: You have to create that "/path" yourself. + ]) + + metadata = plugin.run_metadata_detector( + detector_id, + fs_root, + # Test with an arch:any package. The test framework will supply a minimum number of fields + # (e.g., "Package") so you do not *have* to provide them if you do not need them. + # That is also why providing `Architecture` alone works here. + context=package_metadata_context(package_fields={'Architecture': 'any'}) + ) + # Per definition of our detector, there should be no dependency added (even though the file is there) + assert 'misc:Depends' not in metadata.substvars + # Nor should any maintscripts have been added + assert metadata.maintscripts() == [] + + metadata = plugin.run_metadata_detector( + detector_id, + fs_root, + # This time, we test with an arch:all package + context=package_metadata_context(package_fields={'Architecture': 'all'}) + ) + + assert metadata.substvars['misc:Depends'] == 'foo-utils' + + # You could also have added `maintscript='postinst'` to filter by which script it was added to. + snippets = metadata.maintscripts() + # There should be exactly one snippet + assert len(snippets) == 1 + snippet = snippets[0] + # And we can verify that the snippet is as expected. + assert snippet.maintscript == 'postinst' + assert snippet.registration_method == 'on_configure' + assert 'foo-analyze --install /usr/share/foo/foo.conf' in snippet.plugin_provided_script +``` +(This file should be saved into `debian/package.debputy-plugins/my_plugin_check.json` assuming `my_plugin.py` was +used for the module file) + +This test works the same regardless of whether the JSON-based or Python-based method was chosen. + +## Step 4: Have your package provide `debputy-plugin-X` + +All third-party `debputy` plugins are loaded by adding a build dependency on `debputy-plugin-X`, +where `X` is the basename of the plugin JSON file. Accordingly, any package providing a `debputy` plugin +must either be named `debputy-plugin-X` or provide `debputy-plugin-X` (where `X` is replaced with the concrete +plugin name). + +## Step 5: Running the tests + +To run the tests, you have two options: + + 1) Add `python3-pytest <!nocheck>` to the `Build-Depends` in `debian/control`. This will cause + `dh_installdebputy` to run the tests when the package is built. The tests will be skipped + if `DEB_BUILD_OPTIONS` contains `nocheck` per Debian Policy. You will also need to have + the `debputy` command in PATH. This generally happens as a side effect of the + `dh-sequence-installdebputy` build dependency. + + 2) Add `autopkgtest-pkg-debputy` to the `Testsuite` field in `debian/control`. This will cause + the Debian CI framework (via the `autodep8` command) to generate an autopkgtest that will + run the plugin tests against the installed plugin. + +Using both options where possible is generally preferable. + +If your upstream uses a Python test framework that auto-detects tests such as `py.test`, you may +find that it picks up the `debputy` plugin or its tests. If this is causing you issues, please have +a look at the `dh_installdebputy` manpage, which have a section dedicated to how to resolve these +issues. + +## Side note: Python byte-compilation + +When you install a `debputy` plugin into `/usr/share/debputy/debputy/plugins`, then `debputy` will +manage the Python byte-compilation for you. + +## Closing + +You should now either have done all the basic steps of migrating the debhelper tool to `debputy` +or discovered some feature that the guide did not cover. In the latter case, please have a look +at the [debputy issue tracker] and consider whether you should file a feature request for it. + +[how-to guide]: https://documentation.divio.com/how-to-guides/ +[debputy issue tracker]: https://salsa.debian.org/debian/debputy/-/issues |