diff options
Diffstat (limited to 'docs')
| -rw-r--r-- | docs/IMPLEMENTATION-DECISIONS.md | 206 |
1 files changed, 206 insertions, 0 deletions
diff --git a/docs/IMPLEMENTATION-DECISIONS.md b/docs/IMPLEMENTATION-DECISIONS.md new file mode 100644 index 0000000..a5fad33 --- /dev/null +++ b/docs/IMPLEMENTATION-DECISIONS.md @@ -0,0 +1,206 @@ +# Implementation decisions for debputy + +This document logs important decisions taken during the design of `debputy` along with the +rationale and alternatives considered at the time. This tech note collects decisions, analysis, +and trade-offs made in the implementation of the system that may be of future interest. It also +collects a list of intended future work. The historical background here may be useful for +understanding design and implementation decisions, but would clutter other documents and distract +from the details of the system as implemented. + +## Border between "installation" and "transformation" + +In `debputy`, a contributor can request certain actions to be performed such as `install foo into pkg` +or `ensure bar in pkg is a symlink to baz`. While the former is clearly an installation rule, is the +latter an installation rule or a transformation rule? + +Answering this was important to ensure that actions were placed where people would expect them or would +find it logical to look for them. This is complicated by the fact that `install` (the command line tool) +can perform mode and ownership transformation, create directories, whereas `dh_install` deals only with +installing (copying) paths into packages and mode/ownership changes is related to a separate helper. + +The considered options were: + +### Install upstream bits and then apply packaging modification (chosen) + +In this line of thinking, the logic conceptually boils down to the following rule of thumb: + + > If a path does not come from upstream, then it is a transform. + +Expanding a bit, anything that would install paths from upstream (usually `debian/tmp/...`) into a +package is considered part of `installation`. If further mutations are needed (such as, `create an +empty dir at X as integration point`), they are transformations. + +All path metadata modifications (owner, group or mode) are considered transformations. Even in the +case, where the transformation is "disabling" a built-in normalization. The logic here is that the +packager's transform rule is undoing a built-in transformation rule. + +This option was chosen because it fit the perceived idea of how a packager views their own work +per the following 4-step list: + + 1. Do any upstream build required. + 2. Install files to build the initial trees for each Debian package. + 3. Transform those trees for any additional fixes required. + 4. Turn those trees into debs. + +Note: The `debhelper` stack has all transformations (according to this definition) under its +installation phase as defined by `dh`'s `install` target. Concretely, the `dh install` target covers +`dh_installdirs`, `dh_link` and `dh_fixperms`. However, it is less important what `debhelper` is +doing as long as the definition is simple and not counter-intuitive to packagers. + +### Define the structural and then apply non-structural modifications + +Another proposal was to see the `file layout` phase as anything that did structural changes to the +content of the package. By the end of the `file layout` phase, all paths would be present where +they were expected. So any mutation by the packager that changed the deb structurally would be a +part of the `file layout` phase. + +Note file compression (and therefore renaming of files) could occur after `file layout` when this +model was discussed. + +The primary advantage was that it works without having an upstream build system. However, even +native packages tend to have an "upstream-like" build system, so it is not as much of an advantage +in practice. + +Note this definition is not a 1:1 match with debhelper either. As an example, file mode +modification would be a transformation in this definition, whereas `debhelper` has it under +`dh install`. + +## Stateless vs. Stateful installation rules + +A key concept in packaging is to "install" paths provided by upstream's build system into one or +more packages. In source packages producing multiple binary packages, the packager will want to +divide the content across multiple packages and `debputy` should facilitate this in the best +possible fashion. + +There were two "schools of thought" considered here, which is easiest to illustrate with the +following example: + + Assume that the upstream build system provides 4 programs in `debian/tmp/usr/bin`. One of + these (`foo`) would have to be installed into the package `pkg` and the other would be more + special purpose and go into `foo-utils`. + + +For a "stateless" ruleset, the packager would have to specify the request as: + + * install `usr/bin/foo` into `pkg` + * install `usr/bin/*` except `usr/bin/foo` into `pkg-utils` + +Whereas with a "stateful" ruleset, the packager would have to specify the request as: + + 1. install `usr/bin/foo` into `pkg` + 2. install `usr/bin/*` into `pkg-utils` + - Could be read as "install everything remaining in `usr/bin` into `pkg-utils`". + + +### Stateful installation rules (chosen) + +The chosen model ended up being "stateful" patterns. + +Pros: + + 1. Stateful rules provides a "natural" way to say "install FILE1 in DIR into A, + FILE2 from DIR into B, and the rest of DIR into C" without having to accumulating + "excludes" or replacing a glob with "subglobs" to avoid double matching. + + 2. There is a "natural" way to define that something should *not* be installed + via the `discard` rule, which interfaces nicely with the `dh_missing`-like + behaviour (detecting things that might have been overlooked). + + 3. Avoids the complexity of having a glob expansion with a "per-rule" `exclude`, + where the `exclude` itself contains globs (`usr/lib/*` except `usr/bin/*.la`). + +Cons: + 1. Stateful parsing requires `debputy` to track what has already been matched. + 2. Rules cannot be interpreted in isolation nor out of order. + 3. Naming does not (always) imply the "destructiveness" or state of the action. + - The `install` term is commonly understood to have `copy` semantics rather + than `move` semantics. + 4. It is a step away from default `debhelper` mechanics and might cause + surprises for people assuming `debhelper` semantics. + + +The 1st con would have applied anyway, as to avoid accidental RC bugs the +contributor is required to explicitly list multiple packages for any install +rule that would install the same path into two distinct packages or to provide +`dh_missing` functionality. Therefore, the tracking would have existed in +some form regardless. + +The 2nd con can be mitigated by leveraging the tracking to report if the +rules appear to run in opposite order. + +The 3rd con is partly mitigated by using `discard` rather than `exclude` (which +was the original name). Additionally, the mitigation for the 2nd con generally +covers the most common cases as well. The only "surprising" case if you have +one tool path you want installed into two packages at the same time, where you +use two matches and the second one is a glob. However, the use-case is rare and +was considered an acceptable risk given its probability. + +The 4th con is less of a problem when migrating from `debhelper` to `debputy`. +Any `debhelper` based package will not have (unintentional) overlapping matches +causing file conflicts. There might be some benign double matching that the +packager will have to clean up post migration, because `debhelper` is more +forgiving. Migration from `debputy` to `debhelper` might be more difficult but +not a goal for `debputy`, so it was not considered relevant. + +Prior art: `dh-exec` supports a similar feature via `=> usr/bin/foo`. + + +### Stateless installation rules + +Pros: + + 1. It matches the default helper, so it requires less cognitive effort for + people migrating. + + 2. The `install` term would effectively have `copy` semantics. + + 3. In theory, `debputy` could do with simpler tracking mechanics. + - In practice, the tracked used for the error reporting required was 80% + of the complexity. This severely limits any practical benefit. + + +Cons: + + 1. No obvious way to deliberately ignore content that are not of a glob + exclude. + - While the `usr/bin/* except <matches>` could work, the default is "new appearances" + gets installed rather than aborting the built with a "there is a new tool for you + to consider". Alternatives such as including a stand-alone `exclude` or `discard` + rule would imply stateful parsing, but would not actually be stateful for `install` + and therefore being a potential source of confusion. Therefore, such a feature + would have to require a separate configuration next to installations. + + 2. Install rules with globs would have to accumulate excludes or degenerate to the "magic + sub-matching globs" to avoid overlaps. The latter is the pattern supported by debhelper. + +# Plugin integration + +Looking at `debhelper`, one of its major sources of success is that "anyone" could extend it +to solve their specific need and that it was easy to do so. When looking at the debhelper +extensions, it seems a vast majority of Debian packages do the debhelper extension "on the +side" (such as a bundle it inside an existing `-dev` package). Having package dedicated +to the debhelper tooling does happen but seems to be very rare. + +With this in mind, using python's `entry_points` API was ruled out. It would require packagers +to do a Python project inside their existing package with double build-systems, which basically +no existing package helper does well (CDBS a possible exception but CDBS is generally frowned +upon by the general Debian contributor population). + +Instead, a "drop a .json file here" approach was chosen instead to get a more "light-weight" +integration up and running. When designing it, the following things were important: + + * It should be possible to extract the metadata of the plugin *without* running any code from it + as running code could "taint" the process and break "list all plugins" features. + (This ruled out loading python code directly) + + * Simple features would ideally not require code at all. Packager provided files as an example + can basically be done as a configuration rather than code. This means that `debputy` can provide + automated plugin upgrades from the current format to a future one if needed be. + + * Being able to re-use the declarative parser to handle the error messages and data normalization + (this implies `JSON`, `YAML` or similar formats that is easily parsed in to mappings and lists). + + * It is important that there is a plugin API compat level that enables us to change the format or + API between `debputy` and the plugins if we learn that the current API is inadequate. + +At the time of writing, the plugin integration is still in development. What is important can change +as we get actual users. |