diff options
99 files changed, 9382 insertions, 1284 deletions
diff --git a/GETTING-STARTED-WITH-dh-debputy.md b/GETTING-STARTED-WITH-dh-debputy.md index 7b6796b..5825b49 100644 --- a/GETTING-STARTED-WITH-dh-debputy.md +++ b/GETTING-STARTED-WITH-dh-debputy.md @@ -44,10 +44,17 @@ your packaging. At the time of writing, your options are: converted at this time. Note that this mode does *not* interact well with most third-party `dh` addons. You are recommended to start with source packages without third-party `dh` addons. + 3) `full`: This mode exists but is beyond the scope of this guide. In this mode, `dh` and `debian/rules` + are removed from the build process. If `full` integration is your goal, using `dh-sequence-zz-debputy` + is a good starting point, as any actions needed for `dh-sequence-zz-debputy` will also be needed for + the `full` integration mode. + Since you can always migrate from "less integrated" to "more integrated", you are recommended to start with `dh-sequence-zz-debputy-rrr` first. If that works, you can re-run the migration with `dh-sequence-zz-debputy` as the target to see if further integration seems feasible / desirable. +For more details on integration modes, please see [INTEGRATION-MODES.md](INTEGRATION-MODES.md) + Note: More options may appear in the future. ## Step 2: Have `debputy` convert relevant `debhelper` files diff --git a/INTEGRATION-MODES.md b/INTEGRATION-MODES.md new file mode 100644 index 0000000..1094e67 --- /dev/null +++ b/INTEGRATION-MODES.md @@ -0,0 +1,149 @@ +debputy - Integration modes +=========================== + +_This is [reference documentation] and is primarily useful if you want to know more on integration modes_ +_If you want to migrate to one of these integration modes, then [GETTING-STARTED-WITH-dh-debputy.md](GETTING-STARTED-WITH-dh-debputy.md) might be better._ + +<!-- To writers and reviewers: Check the documentation against https://documentation.divio.com/ --> + + +The debputy tool is a Debian package builder, and it has multiple levels of +"integration" with the package build. Each integration mode has different +pros and cons, which will be covered in this document. + +The integration modes are: + + * `dh-sequence-zz-debputy-rrr` + * `dh-sequence-zz-debputy` (or `dh-sequence-debputy`) + * `full` + +The integration modes that start with `dh-sequence-` are named such because +they leverage `dh` and its add-on system. The `dh-sequence-zz-` is a trick +to convince `dh` to load the add-ons last when they are activated via +`Build-Depends`, which is part of `debputy`'s strategy for maximizing +compatibility with other `dh` add-ons. + +Integration mode - `dh-sequence-zz-debputy-rrr` +----------------------------------------------- + +This integration mode is a minimal integration mode aimed exactly at removing the +(implicit) requirement for `fakeroot` in packages that need static ownership in +the `.deb`. + +It trades many of `debputy` features for compatibility with `debhelper` and ease +of transitions. + +This integration mode is relevant for you when: + + * You want to get rid of the implicit `fakeroot` requirement, and you need static + ownership, OR + * You want to transition to `debputy` in the long term, but more involved integration + modes do not support what you need, OR + * The mode has a particular feature you want that `debhelper` does not provide. + + +Pros: + + * You can use `debputy` to assign static ownerships without needing `fakeroot`, + which is not possible with `debhelper`. + * You get maximum compatibility with existing `dh` add-ons. + * Migration is generally possible with minimal or small changes. + * It is accessible in `bookworm-backports` + +Cons: + + * Many of `debputy`'s features cannot be used. + * Most limitations of `debhelper` still applies (though these limitations are the + status quo, so the package would have a solution to them if needed). + * You still longer a turning complete configuration language for your package helper + (`debian/rules`) with poor introspection. + +To migrate, please use: + + debputy migrate-from-dh --migration-target dh-sequence-zz-debputy-rrr + +Note: The `debputy migrate-from-dh` command accepts a `--no-act --acceptable-migration-issues=ALL`, +if you want to see how involved the migration will be. + +For documentation, please see: + * [GETTING-STARTED-WITH-dh-debputy.md] for a more detailed migration guide (how-to guide). + * https://wiki.debian.org/BuildingWithoutFakeroot + + +Integration mode - `dh-sequence-zz-debputy` +------------------------------------------- + +This integration mode is a more involved integration mode of `debputy` that partly leverages +`dh`. In this mode, `debputy` will take over all the logic of installing files into the +respective package staging directories (`debian/<pkg>`). Roughly speaking, the original +`debhelper` runs until `dh_auto_install` and then `debputy` takes over. + +This integration mode is relevant when: + + * You want to migrate to the `full` integration mode, but you would like to split the migration + in two, OR + * You want to use more of `debputy`'s features and do not use any unsupported `dh` add-ons without + wanting to migrate the build part. + +Pros: + + * You can use the most features of `debputy`. Only the build and environment related ones are + not accessible. + * It is accessible in `bookworm-backports` + +Cons: + + * Almost all `dh` add-ons will stop working since they rely on being able to see the content + of `debian/<pkg>`. Since `debputy` will populate *and* assemble the `.deb`, there is never + a window for the affected add-on to work. Any features provided by these add-ons would have + to be provided by a `debputy` plugin (or `debputy` itself). + * Your only `debhelper` limitations is `dh` notorious lack of proper multi-build support. + * You still longer a turning complete configuration language for your package helper + (`debian/rules`) with poor introspection. + +To migrate, please use: + + debputy migrate-from-dh --migration-target dh-sequence-zz-debputy + +Note: The `debputy migrate-from-dh` command accepts a `--no-act --acceptable-migration-issues=ALL`, +if you want to see how involved the migration will be. It will also detect possible incompatible +`dh` add-ons if you are concerned about whether your package can be supported. + +For documentation, please see: + * [GETTING-STARTED-WITH-dh-debputy.md] for a more detailed migration guide (how-to guide). + + +Integration mode - `full` +------------------------- + +This is the integration mode that `debputy` is about. In the `full` integration mode, `debputy` +replaces `dh` as the package helper. It even removes `debian/rules` replacing it with `dpkg`'s +new `Build-Driver` feature. + +Pros: + + * You can use all features from `debputy` including its native multi-build support. + * You can still leverage `debhelper` build systems (anything integrating with the `dh_auto_*` + tools.) via the `debhelper` build system. + * You no longer have a turning complete configuration language for your package helper + (`debian/rules`) with poor introspection. + +Cons: + + * (Temporary) Incomplete `debputy migrate-from-dh` support + * It is a new ecosystem missing a lot of the third-party features you would find for `dh`. + Only `debhelper` build systems (`dh_auto_*`) can be reused. + * It requires Debian `trixie` or later due to `Build-Driver` (`dpkg-dev`) + * You no longer have the flexibility of a turning complete configuration language for your + package helper, which some people might miss. :) + +To migrate, please use: + + debputy migrate-from-dh --migration-target full + +Note: The `debputy migrate-from-dh` command accepts a `--no-act --acceptable-migration-issues=ALL`, +if you want to see how involved the migration will be. It will also detect possible incompatible +`dh` add-ons if you are concerned about whether your package can be supported. + + +[reference documentation]: https://documentation.divio.com/reference/ diff --git a/MANIFEST-FORMAT.md b/MANIFEST-FORMAT.md index d1474bf..aeb737f 100644 --- a/MANIFEST-FORMAT.md +++ b/MANIFEST-FORMAT.md @@ -1,6 +1,6 @@ # The debputy manifest format -_This is [reference guide] and is primarily useful if you have an idea of what you are looking for._ +_This is [reference documentation] and is primarily useful if you have an idea of what you are looking for._ _If you are new to `debputy`, maybe you want to read [GETTING-STARTED-WITH-dh-debputy.md](GETTING-STARTED-WITH-dh-debputy.md) first._ <!-- To writers and reviewers: Check the documentation against https://documentation.divio.com/ --> @@ -542,6 +542,60 @@ Limitations: `{{PACKAGE}}` cannot be used when defining a variable. This restriction may be lifted in the future. +# Build environment (`build-environment`) + +Define the environment variables used in all build commands. + +The environment definition can be used to tweak the environment variables used by the +build commands. An example: + + environment: + set: + ENV_VAR: foo + ANOTHER_ENV_VAR: bar + +The environment definition has multiple attributes for setting environment variables +which determines when the definition is applied. The resulting environment is the +result of the following order of operations. + + 1. The environment `debputy` received from its parent process. + 2. Apply all the variable definitions from `set` (if the attribute is present) + 3. Apply all computed variables (such as variables from `dpkg-buildflags`). + 4. Apply all the variable definitions from `override` (if the attribute is present) + 5. Remove all variables listed in `unset` (if the attribute is present). + +Accordingly, both `override` and `unset` will overrule any computed variables while +`set` will be overruled by any computed variables. + +Note that these variables are not available via manifest substitution (they are only +visible to build commands). They are only available to build commands. + +The `build-environment` attribute is a mapping and has the following attributes: + + - `set` (optional): Mapping of string + + A mapping of environment variables to be set. + + Note these environment variables are set before computed variables (such + as `dpkg-buildflags`) are provided. They can affect the content of the + computed variables, but they cannot overrule them. If you need to overrule + a computed variable, please use `override` instead. + + - `override` (optional): Mapping of string + + A mapping of environment variables to set. + + Similar to `set`, but it can overrule computed variables like those from + `dpkg-buildflags`. + + - `unset` (optional): List of string + + A list of environment variables to unset. + + Any environment variable named here will be unset. No warnings or errors + will be raised if a given variable was not set. + + # Installations For source packages building a single binary, the `dh_auto_install` from debhelper will default to @@ -625,7 +679,7 @@ you could do: installations: - install: sources: - # By-pass automatic discard of `libfoo.la` - no globs *cannot* be used! + # By-pass automatic discard of `libfoo.la` - globs *cannot* be used! - "usr/lib/libfoo.la" - "usr/lib/libfoo*.so*" into: libfoo1 @@ -755,8 +809,8 @@ alternative name. ## Install examples (`install-examples`) -This install rule resemble that of `dh_installexamples`. It is a shorthand over the generic ` -install` rule with the following key features: +This install rule resemble that of `dh_installexamples`. It is a shorthand over the generic +`install` rule with the following key features: 1) It pre-defines the `dest-dir` that respects the "main documentation package" recommendation from Debian Policy. The `install-examples` will use the `examples` subdir for the package documentation @@ -15,10 +15,12 @@ install: $(DESTDIR)/$(DEBPUTY_INSTALLED_ROOT_DIR) \ $(DESTDIR)/$(DEBPUTY_INSTALLED_PLUGIN_ROOT_DIR)/debputy \ $(DESTDIR)/usr/share/perl5/Debian/Debhelper/Sequence \ + $(DESTDIR)/usr/share/perl5/Dpkg/BuildDriver \ $(DESTDIR)/usr/share/man/man1 install -m0755 -t $(DESTDIR)/usr/bin dh_debputy dh_installdebputy assets/debputy install -m0755 -t $(DESTDIR)/$(DEBPUTY_INSTALLED_ROOT_DIR) deb_packer.py deb_materialization.py install -m0644 -t $(DESTDIR)/usr/share/perl5/Debian/Debhelper/Sequence lib/Debian/Debhelper/Sequence/*.pm + install -m0644 -t $(DESTDIR)/usr/share/perl5/Dpkg/BuildDriver lib/Dpkg/BuildDriver/*.pm cp -a --reflink=auto src/debputy $(DESTDIR)/$(DEBPUTY_INSTALLED_ROOT_DIR) cp -a --reflink=auto debputy $(DESTDIR)/$(DEBPUTY_INSTALLED_PLUGIN_ROOT_DIR) sed -i "s/^__version__ =.*/__version__ = '$$(dpkg-parsechangelog -SVersion)'/; s/^__release_commit__ =.*/__release_commit__ = 'N\\/A'/;" \ diff --git a/debian/changelog b/debian/changelog index 4125e3e..2ff5421 100644 --- a/debian/changelog +++ b/debian/changelog @@ -1,3 +1,291 @@ +debputy (0.1.48-0.0~progress7.99u1) graograman-backports; urgency=medium + + * Uploading to graograman-updates, remaining changes: + - Updating maintainer field. + - Updating uploaders field. + - Updating bugs field. + - Updating vcs fields. + - Updating source format. + * Merging upstream version 0.1.48. + * Merging debian version 0.1.48. + + -- Daniel Baumann <daniel.baumann@progress-linux.org> Mon, 26 Aug 2024 12:23:26 +0200 + +debputy (0.1.48) unstable; urgency=medium + + * INTEGRATION-MODES.md: Correct `bullseye` that should have been `bookworm` + * Also set PKG_CONFIG in the `perl-build` build system like with + is done in the `perl-makemaker` build system. + * Remove lintian-overrides since `debputy` no longer uses `dh` + * d/control: Fix typo in description + * debputy.pod: Document security aspects of `debputy lsp server` + and `debputy lint`. + * debputy: Fix inaccurate directory listing in an error message. + * debputy: Ensure `u+rwX` as min mode when installing paths into the + packages. This change prevent crashes when upstream has marked + paths as read-only (such as `0444`). + Thanks to Andrey Rakhmatullin <wrar@debian.org> (Closes: #1078906) + Note: Paths can still get their user write bit removed via + transformations or built-in rules (such as files in `etc/sudoers.d` + which must be `0440`) + * debputy: Require directory mode to always have `u+rwx`. Previously, + only the `u+rx` mode was required. There is no known use-case for + directories to not have a read, write and execute for the owner. + + -- Niels Thykier <niels@thykier.net> Sun, 18 Aug 2024 09:47:40 +0000 + +debputy (0.1.47-0.0~progress7.99u1) graograman-backports; urgency=medium + + * Uploading to graograman-updates, remaining changes: + - Updating maintainer field. + - Updating uploaders field. + - Updating bugs field. + - Updating vcs fields. + - Updating source format. + * Merging upstream version 0.1.47. + * Merging debian version 0.1.47. + + -- Daniel Baumann <daniel.baumann@progress-linux.org> Mon, 26 Aug 2024 12:23:12 +0200 + +debputy (0.1.47) unstable; urgency=medium + + * LSP/Lint: + - Fix missing textwrap.dedent for hover docs + - Tweak wording for undocumented fields + - Tweak wording for the `important` value from `Priority` + - Tweak rendering of header of hover docs for known value + - In hover docs, recognize `debputy-plugin-X` similar to `dh-sequence-Y` + - Make a d/changelog diagnostic message a bit more precise + + * migrate-from-dh: + - Add remark about https://bugs.debian.org/1067653 in relevant + warnings. + + * Expand documentation on integration modes (Closes: debputy#117) + * debputy: Fix a warning from auto-detecting when the source + package has no `debhelper` compat level. + * debputy: Restore log output from `pygls` (`debputy lsp server`) + + + -- Niels Thykier <niels@thykier.net> Tue, 13 Aug 2024 12:44:52 +0000 + +debputy (0.1.46-0.0~progress7.99u1) graograman-backports; urgency=medium + + * Uploading to graograman-updates, remaining changes: + - Updating maintainer field. + - Updating uploaders field. + - Updating bugs field. + - Updating vcs fields. + - Updating source format. + * Merging upstream version 0.1.46. + * Merging debian version 0.1.46. + + -- Daniel Baumann <daniel.baumann@progress-linux.org> Mon, 26 Aug 2024 12:22:59 +0200 + +debputy (0.1.46) unstable; urgency=medium + + * LSP: + - Improve wording of and fix typo in hover docs. + - Fix a typo of `hyphen-` that would distrupt "name to section" + checks for `hyphen-` packages. + + [ Niels Thykier ] + * LSP: Fix typo of `hyphen-` and improve wording `Multi-Arch` field + * debputy: Hoist `nocheck` and `nodoc` into `DEB_BUILD_OPTIONS` + Thanks to Chris Hofstaedtler <zeha@debian.org> + * Skip `debian/foo.1` for packager provided files typo checks. + Thanks to Chris Hofstaedtler <zeha@debian.org> + * Set `PKG_CONFIG` ENV var for `perl-makemaker` + * Fix bug in initialization of `EnvironmentModification` (only affects + packages in `full` integration mode). + * Fix bug in escape_util and have it use more readable output by default + * Fix bug where pre-compressed manpage would be corrupted. + Thanks to Andrey Rakhmatullin <wrar@debian.org> (Closes: #1078448) + + [ Guillem Jover ] + * LSP: Fix typos in several descriptions + + + -- Niels Thykier <niels@thykier.net> Sat, 10 Aug 2024 18:16:23 +0000 + +debputy (0.1.45-0.0~progress7.99u1) graograman-backports; urgency=medium + + * Uploading to graograman-updates, remaining changes: + - Updating maintainer field. + - Updating uploaders field. + - Updating bugs field. + - Updating vcs fields. + - Updating source format. + * Merging upstream version 0.1.45. + * Merging debian version 0.1.45. + + -- Daniel Baumann <daniel.baumann@progress-linux.org> Mon, 26 Aug 2024 12:22:45 +0200 + +debputy (0.1.45) unstable; urgency=medium + + * Manifest: + - Add new `default-build-environment`, `build-environments` and `build-rules` + for the new `full` integration mode where `debhelper` is no longer driving + the build. + - Port all `debhelper` provided build systems to have an native `debputy` + counter part with a few tweaks. + + + * LSP/Lint: + - Add `xpm` as a known extension for packager files + - Add `Build-Driver` as a known field for `debian/control`. + - Add `Files-Excluded` and `Files-Included` as known fields for DEP-5 + - Quote values in completation items for the manifest. This means that + the value for `manifest-version` is now properly YAML quoted. + + * debputy: Remove `--spellcheck` from "random" subcommands + * debputy: Fix missing warning with `--debug` (it was supposed to be omitted, + but never was due to exception flow). + * debputy reformat: Fix bug where orphaned packages would have a style by + default + * doc: Document decision on plugin module import vs. initialization design + * LSP/Lint: Fix crash with certain field being duplicated + * MANIFEST-FORMAT.md: Fix wording + * Add initial minimal `Build-Driver` support meaning that `debputy` can now use + `dpkg-buildpackage`'s new `Build-Driver` support to drive the build without + relying on `dh`. This requires bleeding edge features in `dpkg-buildpackage` + and `debputy`. + * Avoid stacktrace for CTRL + C during `run_command` + * Improve error tracking related to plugins + * Add remark about `env:` manifest variables sometimes respecting + `build-environment`. + + -- Niels Thykier <niels@thykier.net> Sun, 04 Aug 2024 12:37:21 +0000 + +debputy (0.1.44-0.0~progress7.99u1) graograman-backports; urgency=medium + + * Uploading to graograman-updates, remaining changes: + - Updating maintainer field. + - Updating uploaders field. + - Updating bugs field. + - Updating vcs fields. + - Updating source format. + * Merging upstream version 0.1.44. + * Merging debian version 0.1.44. + + -- Daniel Baumann <daniel.baumann@progress-linux.org> Mon, 26 Aug 2024 12:22:30 +0200 + +debputy (0.1.44) unstable; urgency=medium + + * LSP: + - Provide hover docs for packages in relationship fields. Due to + implementation limitations, OR relations (`foo | bar`) do not + work. The hover doc relies on the system's APT cache with all + the limitations from that (like the data not matching the target + distribution) + - Fix wording in the synopsis hover doc + + * debputy reformat: If possible provide the reformatting tool when the + style cannot be applied. + * debputy: Remove `printf` debug when log level is changed + * debputy: Prefer using line numbers in error messages when available + * editor-config: Improve the LSP config for `eglot` + * debputy: Improve mode correction for `*.pm` files. + Thanks to Russ Allbery <rra@debian.org> (Closes: #1076346) + + -- Niels Thykier <niels@thykier.net> Mon, 15 Jul 2024 11:24:40 +0000 + +debputy (0.1.43-0.0~progress7.99u1) graograman-backports; urgency=medium + + * Uploading to graograman-updates, remaining changes: + - Updating maintainer field. + - Updating uploaders field. + - Updating bugs field. + - Updating vcs fields. + - Updating source format. + * Merging upstream version 0.1.43. + * Merging debian version 0.1.43. + + -- Daniel Baumann <daniel.baumann@progress-linux.org> Mon, 26 Aug 2024 12:22:17 +0200 + +debputy (0.1.43) unstable; urgency=medium + + * debputy: Only warn about missing `-dev` SO symlinks for libraries + in common lib paths (such as usr/lib/${MA}) + * debputy: Fix bug where `DH_VERBOSE` would fail to print all + commands + * debputy: Distinguish between `--debug` and `DH_VERBOSE` in terms + of verbosity. + * debputy: Restore `Multi-Arch: same` field in dbgsym packages. + Thanks to Russ Allbery <rra@debian.org> + + -- Niels Thykier <niels@thykier.net> Sun, 07 Jul 2024 07:18:46 +0000 + +debputy (0.1.42-0.0~progress7.99u1) graograman-backports; urgency=medium + + * Uploading to graograman-updates, remaining changes: + - Updating maintainer field. + - Updating uploaders field. + - Updating bugs field. + - Updating vcs fields. + - Updating source format. + * Merging upstream version 0.1.42. + * Merging debian version 0.1.42. + + -- Daniel Baumann <daniel.baumann@progress-linux.org> Mon, 26 Aug 2024 12:22:01 +0200 + +debputy (0.1.42) unstable; urgency=medium + + * LSP/Lint: + - Fix exception on some syntax errors + - Add `X?-Ruby-Versions` as a known (obsolete) field + - Include the final line of context (`debputy lint`-only) + - Syntax check dependency relation fields + - Detect incorrect version operators in `Provides` + - Fix crash on package stanzas without `Package` field + - Flag usage of `|` in relationship fields that does not + support it + - Fix invalid ranges for some checks when using field comments + + -- Niels Thykier <niels@thykier.net> Sat, 06 Jul 2024 18:50:17 +0000 + +debputy (0.1.41-0.0~progress7.99u1) graograman-backports; urgency=medium + + * Uploading to graograman-updates, remaining changes: + - Updating maintainer field. + - Updating uploaders field. + - Updating bugs field. + - Updating vcs fields. + - Updating source format. + * Merging upstream version 0.1.41. + * Merging debian version 0.1.41. + + -- Daniel Baumann <daniel.baumann@progress-linux.org> Mon, 26 Aug 2024 12:21:37 +0200 + +debputy (0.1.41) unstable; urgency=medium + + * LSP/Lint: + - Fix typo in diagnostic message + - Reduce the number of false positive pkgfile typo warnings. + Thanks to Otto Kekäläinen <otto@debian.org> (Closes: debputy#109) + - Avoid recommending name segment for packaging files that do not + support that. Note all `debhelper` files currently "support" + name segments as far as `debputy` can tell, since `dh_assistant` + does not track this feature. + * debputy reformat: Fix error in instructions + * debputy-documentation.json: Add `debian/lintian-brush.conf` as + known file. + + -- Niels Thykier <niels@thykier.net> Tue, 02 Jul 2024 20:52:51 +0000 + +debputy (0.1.40-0.0~progress7.99u1) graograman-backports; urgency=medium + + * Uploading to graograman-updates, remaining changes: + - Updating maintainer field. + - Updating uploaders field. + - Updating bugs field. + - Updating vcs fields. + - Updating source format. + * Merging upstream version 0.1.40. + * Merging debian version 0.1.40. + + -- Daniel Baumann <daniel.baumann@progress-linux.org> Mon, 01 Jul 2024 20:12:35 +0200 + debputy (0.1.40) unstable; urgency=medium * Manifest changes: diff --git a/debian/control b/debian/control index 47dab2d..5c8f27d 100644 --- a/debian/control +++ b/debian/control @@ -77,9 +77,15 @@ Description: Manifest style Debian-based package builder (debhelper integration) Package builder that provides a declarative manifest for building Debian-based packages. . - This version integrates with the debhelper sequencer dh and will replace - several of debhelper's tools that are covered by debputy. - . The debputy package builder aims to reduce cognitive load for the packager and provide better introspection to packagers, linters and the Debian janitor. + . + This version integrates with the debhelper sequencer dh. Multiple integration + modes are possible with dh. The most trivial integration mode is + dh-sequence-zz-debputy-rrr, which is a mostly standard dh sequence with very + few changes aimed entirely at removing the need for fakeroot when assembling + the .deb. + . + For more information on all integration modes and what they do, please see + https://salsa.debian.org/debian/debputy/-/blob/main/INTEGRATION-MODES.md diff --git a/debian/rules b/debian/rules index 1221eb2..424cabe 100755 --- a/debian/rules +++ b/debian/rules @@ -1,11 +1,6 @@ #!/usr/bin/make -f -export PERL5LIB=$(CURDIR)/lib -export PATH:=$(CURDIR):$(PATH) export DEBPUTY_CMD=$(CURDIR)/debputy.sh %: - dh $@ --with debputy - -override_dh_debputy: - dh_debputy --plugin self-hosting-plugins/debputy-self-hosting.json + $(DEBPUTY_CMD) --plugin self-hosting-plugins/debputy-self-hosting.json internal-command dpkg-build-driver-run-task $@ diff --git a/debian/source/lintian-overrides b/debian/source/lintian-overrides deleted file mode 100644 index 43506f5..0000000 --- a/debian/source/lintian-overrides +++ /dev/null @@ -1,5 +0,0 @@ -# https://salsa.debian.org/debian/debputy/-/issues/77 -missing-build-dependency-for-dh-addon debputy (does not satisfy dh-debputy:any) [debian/rules] -missing-build-dependency-for-dh_-command dh_debputy (does not satisfy dh-debputy:any) [debian/rules] -# https://bugs.debian.org/1067653 substvar hints are no longer universally applicable -debhelper-but-no-misc-depends dh-debputy diff --git a/debputy.pod b/debputy.pod index 3ce9b90..7d08813 100644 --- a/debputy.pod +++ b/debputy.pod @@ -136,9 +136,9 @@ For packages that does not have the B<X-Style> field, B<debputy> will result to can derive a style that all parties would agree too (or the team style for packaging teams), then that style will be used. -At the time of introduction, the B<black> style is similar to that of B<wrap-and-sort -astkb>, since -that was one of the most common style according to L<https://bugs.debian.org/895570>, But the style is -expected to evolve over time and the two styles may diverge over time. +The B<black> style started as similar to that of B<wrap-and-sort -ast>, since that was one of the most +common styles according to L<https://bugs.debian.org/895570>, but the style is expected to evolve over +time and the two styles may diverge over time. The command accepts the following options: @@ -208,6 +208,12 @@ files. This command is useful for CI or for when you cannot use the language ser the same diagnostics as B<debputy lsp server> would but without requiring an LSP capable editor as intermediate. The output is only intended for human consumption. Machine readable is not a goal at this time. +The B<debputy lint> command is a form of static analysis and will not load nor execute code from the +code it is scanning. It is a security bug for B<debputy lint> to violate this principle directly +or indirectly. Therefore, B<debputy> can only provide diagnostics from libraries and tools that takes +this principle seriously. It also means that B<debputy> will likely have false-positives for self-hosting +code, since it is not allowed to load the self-hosted code. + Note that at the time of writing, the B<debputy.manifest> file is only B<partially> supported. If you have F<debian/debputy.manifest>, please also run B<debputy check-manifest> to get more thorough checks for that file for now. The B<lint> command will inform you about this issue in the output if a @@ -335,6 +341,21 @@ B<debputy lsp server> will suffice (using the stdio transport). See B<debputy ls for other integration options such as TCP (B<--tcp>) or websocket (B<--ws>) plus related supporting options. +The B<debputy lsp server> command provides several features including a form of static analysis in the +form of "as-you-type" diagnostics. For the diagnostics, is B<debputy lsp server> not allowed load nor +execute code from the code it is scanning. It is a security bug for B<debputy lsp server> to violate +this principle directly or indirectly. Therefore, B<debputy> can only provide diagnostics from libraries +and tools that takes this principle seriously. It also means that B<debputy> will likely have +false-positives for self-hosting code, since it is not allowed to load the self-hosted code. + +This security principle also applies to hover docs, completion suggestions and other trivial code editing +or viewing features. However, it is not universal, since certain LSP features are deliberately designed +to run the code you are viewing. As an example, B<debputy lsp server> can provide a "code lens" (LSP term) +for building the current package. On activation of the code lens, B<debputy> will trigger code from the +package to be run and that is expected. The critical points here are that the user most explicitly +trigger the feature and it must use terms commonly associated with running code such as B<build>, +B<run> or B<execute> (non-exhaustive list). + If you need to debug an issue with the language server, the TCP integration (B<--tcp>) can be quite helpful. In this mode, you run B<debputy lsp server --tcp> in a terminal before starting your editor. This means you have direct and unfiltered access to the B<debputy> command and its output. diff --git a/debputy/plugins/debputy-documentation.json b/debputy/plugins/debputy-documentation.json index 4017e2c..2cfde53 100644 --- a/debputy/plugins/debputy-documentation.json +++ b/debputy/plugins/debputy-documentation.json @@ -46,6 +46,13 @@ "file-categories": ["maint-config"] }, { + "path": "debian/lintian-brush.conf", + "documentation-uris": [ + "man:lintian-brush.conf(5)" + ], + "file-categories": ["maint-config"] + }, + { "path": "debian/source/options", "documentation-uris": [ "man:dpkg-source(1)" diff --git a/docs/IMPLEMENTATION-DECISIONS.md b/docs/IMPLEMENTATION-DECISIONS.md index d3faef6..b238d27 100644 --- a/docs/IMPLEMENTATION-DECISIONS.md +++ b/docs/IMPLEMENTATION-DECISIONS.md @@ -204,3 +204,32 @@ integration up and running. When designing it, the following things were import At the time of writing, the plugin integration is still in development. What is important can change as we get actual users. + +# Plugin module import vs. initialization design + +When loading the python code for plugins, there were multiple ways to initialize the plugin: + + 1. The plugin could have an initialization function that is called by `debputy` + + 2. The plugin could use `@some_feature` annotations on types to be registered. This is similar + in spirit to the `dh` add-on system where module load is considered the initialization event. + +The 1. option was chosen because it is more reliable at directing the control flow and enables +the plugin module to be importable by non-`debputy` code. The latter might not seem directly +useful, but code like `py.test` attempts to import everything it can (even in `debian/` by +default) which could cause unwanted breakage for plugin providers by simply adding a +Python-based plugin. + +With the module autoloads mechanism, `debputy` would have to assume everything imported is +part of the plugin. This can cause issues (misattribution of errors) when a plugin loads +code or definitions from another plugin in addition to the `py.test` problem above. Safeguards +could be added, but the solutions found would either break the "py.test can import the module"- +property (see above) or be "safety is opt-in rather than always on/opt-out" by having an explicit +"This is plugin X code coming now" - the absence of a marker then causing misattribution). + +If these problems could be solved, the annotation based loading could be considered. + +Note: This is not to say that `@feature` cannot be used at all in the plugin code. The annotation +can be used to add metadata to things that simplify the logic required by the initialization +function. The annotation would have to be stateless and cannot make assumptions about which +plugin is being loaded while it is run. diff --git a/lib/Dpkg/BuildDriver/Debputy.pm b/lib/Dpkg/BuildDriver/Debputy.pm new file mode 100644 index 0000000..957f608 --- /dev/null +++ b/lib/Dpkg/BuildDriver/Debputy.pm @@ -0,0 +1,44 @@ +package Dpkg::BuildDriver::Debputy; +use strict; +use warnings FATAL => 'all'; +use Dpkg::ErrorHandling; + +sub _run_cmd { + my @cmd = @_; + printcmd(@cmd); + system @cmd and subprocerr("@cmd"); +} + +sub new { + my ($this, %opts) = @_; + my $class = ref($this) || $this; + my $self = bless({ + 'ctrl' => $opts{ctrl}, + 'debputy_cmd' => 'debputy', + }, $class); + return $self; +} + + +sub pre_check { + my ($self) = @_; + my $ctrl_src = $self->{'ctrl'}->get_source(); + my $debputy_self_hosting_cmd = './debputy.sh'; + if ($ctrl_src->{"Source"} eq 'debputy' and -f -x $debputy_self_hosting_cmd) { + $self->{'debputy_cmd'} = $debputy_self_hosting_cmd; + notice("Detected this is a self-hosting build of debputy. Using \"${debputy_self_hosting_cmd}\" to self-host."); + } + return; +} + +sub need_build_task { + return 0; +} + +sub run_task { + my ($self, $task) = @_; + _run_cmd($self->{'debputy_cmd'}, 'internal-command', 'dpkg-build-driver-run-task', $task); + return; +} + +1; diff --git a/src/debputy/_manifest_constants.py b/src/debputy/_manifest_constants.py index 3ed992b..974ef7b 100644 --- a/src/debputy/_manifest_constants.py +++ b/src/debputy/_manifest_constants.py @@ -8,6 +8,8 @@ assert DEFAULT_MANIFEST_VERSION in SUPPORTED_MANIFEST_VERSIONS MK_MANIFEST_VERSION = "manifest-version" MK_PACKAGES = "packages" +MK_BUILDS = "builds" + MK_INSTALLATIONS = "installations" MK_INSTALLATIONS_INSTALL = "install" MK_INSTALLATIONS_MULTI_DEST_INSTALL = "multi-dest-install" diff --git a/src/debputy/analysis/debian_dir.py b/src/debputy/analysis/debian_dir.py index 1e88b14..8ada2a0 100644 --- a/src/debputy/analysis/debian_dir.py +++ b/src/debputy/analysis/debian_dir.py @@ -16,6 +16,7 @@ from typing import ( Iterator, TypedDict, NotRequired, + Container, ) from debputy.analysis import REFERENCE_DATA_TABLE @@ -23,6 +24,7 @@ from debputy.analysis.analysis_util import flatten_ppfs from debputy.dh.dh_assistant import ( resolve_active_and_inactive_dh_commands, read_dh_addon_sequences, + extract_dh_compat_level, ) from debputy.packager_provided_files import ( PackagerProvidedFile, @@ -105,7 +107,7 @@ def scan_debian_dir( or "zz_debputy" in dh_sequences or "zz-debputy-rrr" in dh_sequences ) - dh_compat_level, dh_assistant_exit_code = _extract_dh_compat_level() + dh_compat_level, dh_assistant_exit_code = extract_dh_compat_level() dh_issues = [] static_packaging_files = { @@ -128,6 +130,7 @@ def scan_debian_dir( binary_packages, allow_fuzzy_matches=True, detect_typos=True, + ignore_paths=static_packaging_files, ) ) ) @@ -139,7 +142,7 @@ def scan_debian_dir( all_dh_ppfs, dh_issues, dh_assistant_exit_code, - ) = _resolve_debhelper_config_files( + ) = resolve_debhelper_config_files( debian_dir, binary_packages, debputy_plugin_metadata, @@ -147,6 +150,7 @@ def scan_debian_dir( dh_sequences, dh_compat_level, uses_dh_sequencer, + ignore_paths=static_packaging_files, ) else: @@ -285,7 +289,7 @@ def _kpf_install_pattern( return ppkpf.info.get("install_pattern") -def _resolve_debhelper_config_files( +def resolve_debhelper_config_files( debian_dir: VirtualPath, binary_packages: Mapping[str, BinaryPackage], debputy_plugin_metadata: DebputyPluginMetadata, @@ -293,6 +297,7 @@ def _resolve_debhelper_config_files( dh_rules_addons: AbstractSet[str], dh_compat_level: int, saw_dh: bool, + ignore_paths: Container[str] = frozenset(), ) -> Tuple[List[PackagerProvidedFile], Optional[object], int]: dh_ppfs = {} commands, exit_code = _relevant_dh_commands(dh_rules_addons) @@ -434,6 +439,7 @@ def _resolve_debhelper_config_files( binary_packages, allow_fuzzy_matches=True, detect_typos=True, + ignore_paths=ignore_paths, ) ) ) @@ -532,27 +538,6 @@ def _merge_ppfs( _merge_list(details, "documentation-uris", documentation_uris) -def _extract_dh_compat_level() -> Tuple[Optional[int], int]: - try: - output = subprocess.check_output( - ["dh_assistant", "active-compat-level"], - stderr=subprocess.DEVNULL, - ) - except (FileNotFoundError, subprocess.CalledProcessError) as e: - exit_code = 127 - if isinstance(e, subprocess.CalledProcessError): - exit_code = e.returncode - return None, exit_code - else: - data = json.loads(output) - active_compat_level = data.get("active-compat-level") - exit_code = 0 - if not isinstance(active_compat_level, int) or active_compat_level < 1: - active_compat_level = None - exit_code = 255 - return active_compat_level, exit_code - - def _relevant_dh_commands(dh_rules_addons: Iterable[str]) -> Tuple[List[str], int]: cmd = ["dh_assistant", "list-commands", "--output-format=json"] if dh_rules_addons: diff --git a/src/debputy/build_support/__init__.py b/src/debputy/build_support/__init__.py new file mode 100644 index 0000000..8123659 --- /dev/null +++ b/src/debputy/build_support/__init__.py @@ -0,0 +1,7 @@ +from debputy.build_support.build_logic import perform_builds +from debputy.build_support.clean_logic import perform_clean + +__all__ = [ + "perform_clean", + "perform_builds", +] diff --git a/src/debputy/build_support/build_context.py b/src/debputy/build_support/build_context.py new file mode 100644 index 0000000..cdaf4b4 --- /dev/null +++ b/src/debputy/build_support/build_context.py @@ -0,0 +1,95 @@ +from typing import Mapping, Optional + +from debputy.architecture_support import DpkgArchitectureBuildProcessValuesTable +from debputy.commands.debputy_cmd.context import CommandContext +from debputy.manifest_conditions import _run_build_time_tests + + +class BuildContext: + @staticmethod + def from_command_context( + cmd_context: CommandContext, + ) -> "BuildContext": + return BuildContextImpl(cmd_context) + + @property + def deb_build_options(self) -> Mapping[str, Optional[str]]: + raise NotImplementedError + + def parallelization_limit(self, *, support_zero_as_unlimited: bool = False) -> int: + """Parallelization limit of the build + + This is accessor that reads the `parallel` option from `DEB_BUILD_OPTIONS` with relevant + fallback behavior. + + :param support_zero_as_unlimited: The debhelper framework allowed `0` to mean unlimited + in some build systems. If the build system supports this, it should set this option + to True, which will allow `0` as a possible return value. WHen this option is False + (which is the default), `0` will be remapped to a high number to preserve the effect + in spirit (said fallback number is also from `debhelper`). + """ + limit = self.deb_build_options.get("parallel") + if limit is None: + return 1 + try: + v = int(limit) + except ValueError: + return 1 + if v == 0 and not support_zero_as_unlimited: + # debhelper allowed "0" to be used as unlimited in some cases. Preserve that feature + # for callers that are prepared for it. For everyone else, remap 0 to an obscene number + # that de facto has the same behaviour + # + # The number is taken out of `cmake.pm` from `debhelper` to be "Bug compatible" with + # debhelper on the fallback as well. + return 999 + return v + + @property + def is_terse_build(self) -> bool: + """Whether the build is terse + + This is a shorthand for testing for `terse` in DEB_BUILD_OPTIONS + """ + return "terse" in self.deb_build_options + + @property + def is_cross_compiling(self) -> bool: + """Whether the build is considered a cross build + + Note: Do **not** use this as indicator for whether tests should run. Use `should_run_tests` instead. + To the naive eye, they seem like they overlap in functionality, but they do not. There are cross + builds where tests can be run. Additionally, there are non-cross-builds where tests should be + skipped. + """ + return self.dpkg_architecture_variables.is_cross_compiling + + def cross_tool(self, command: str) -> str: + if not self.is_cross_compiling: + return command + cross_prefix = self.dpkg_architecture_variables["DEB_HOST_GNU_TYPE"] + return f"{cross_prefix}-{command}" + + @property + def dpkg_architecture_variables(self) -> DpkgArchitectureBuildProcessValuesTable: + raise NotImplementedError + + @property + def should_run_tests(self) -> bool: + return _run_build_time_tests(self.deb_build_options) + + +class BuildContextImpl(BuildContext): + def __init__( + self, + cmd_context: CommandContext, + ) -> None: + self._cmd_context = cmd_context + + @property + def deb_build_options(self) -> Mapping[str, Optional[str]]: + return self._cmd_context.deb_build_options + + @property + def dpkg_architecture_variables(self) -> DpkgArchitectureBuildProcessValuesTable: + return self._cmd_context.dpkg_architecture_variables() diff --git a/src/debputy/build_support/build_logic.py b/src/debputy/build_support/build_logic.py new file mode 100644 index 0000000..ee247e7 --- /dev/null +++ b/src/debputy/build_support/build_logic.py @@ -0,0 +1,193 @@ +import collections +import contextlib +import os +from typing import ( + Iterator, + Mapping, + List, + Dict, + Optional, +) + +from debputy.build_support.build_context import BuildContext +from debputy.build_support.buildsystem_detection import ( + auto_detect_buildsystem, +) +from debputy.commands.debputy_cmd.context import CommandContext +from debputy.highlevel_manifest import HighLevelManifest +from debputy.manifest_parser.base_types import BuildEnvironmentDefinition +from debputy.plugin.debputy.to_be_api_types import BuildRule +from debputy.util import ( + _error, + _info, + _non_verbose_info, +) + + +@contextlib.contextmanager +def in_build_env(build_env: BuildEnvironmentDefinition): + remove_unnecessary_env() + # Should possibly be per build + with _setup_build_env(build_env): + yield + + +def _set_stem_if_absent(stems: List[Optional[str]], idx: int, stem: str) -> None: + if stems[idx] is None: + stems[idx] = stem + + +def assign_stems( + build_rules: List[BuildRule], + manifest: HighLevelManifest, +) -> None: + if not build_rules: + return + if len(build_rules) == 1: + build_rules[0].auto_generated_stem = "" + return + + debs = {p.name for p in manifest.all_packages if p.package_type == "deb"} + udebs = {p.name for p in manifest.all_packages if p.package_type == "udeb"} + deb_only_builds: List[int] = [] + udeb_only_builds: List[int] = [] + by_name_only_builds: Dict[str, List[int]] = collections.defaultdict(list) + stems = [rule.name for rule in build_rules] + reserved_stems = set(n for n in stems if n is not None) + + for idx, rule in enumerate(build_rules): + stem = stems[idx] + if stem is not None: + continue + pkg_names = {p.name for p in rule.for_packages} + if pkg_names == debs: + deb_only_builds.append(idx) + elif pkg_names == udebs: + udeb_only_builds.append(idx) + + if len(pkg_names) == 1: + pkg_name = next(iter(pkg_names)) + by_name_only_builds[pkg_name].append(idx) + + if "deb" not in reserved_stems and len(deb_only_builds) == 1: + _set_stem_if_absent(stems, deb_only_builds[0], "deb") + + if "udeb" not in reserved_stems and len(udeb_only_builds) == 1: + _set_stem_if_absent(stems, udeb_only_builds[0], "udeb") + + for pkg, idxs in by_name_only_builds.items(): + if len(idxs) != 1 or pkg in reserved_stems: + continue + _set_stem_if_absent(stems, idxs[0], pkg) + + for idx, rule in enumerate(build_rules): + stem = stems[idx] + if stem is None: + stem = f"bno_{idx}" + rule.auto_generated_stem = stem + _info(f"Assigned {rule.auto_generated_stem} [{stem}] to step {idx}") + + +def perform_builds( + context: CommandContext, + manifest: HighLevelManifest, +) -> None: + build_rules = manifest.build_rules + if build_rules is not None: + if not build_rules: + # Defined but empty disables the auto-detected build system + return + active_packages = frozenset(manifest.active_packages) + condition_context = manifest.source_condition_context + build_context = BuildContext.from_command_context(context) + assign_stems(build_rules, manifest) + for step_no, build_rule in enumerate(build_rules): + step_ref = ( + f"step {step_no} [{build_rule.auto_generated_stem}]" + if build_rule.name is None + else f"step {step_no} [{build_rule.name}]" + ) + if build_rule.for_packages.isdisjoint(active_packages): + _info( + f"Skipping build for {step_ref}: None of the relevant packages are being built" + ) + continue + manifest_condition = build_rule.manifest_condition + if manifest_condition is not None and not manifest_condition.evaluate( + condition_context + ): + _info( + f"Skipping build for {step_ref}: The condition clause evaluated to false" + ) + continue + _info(f"Starting build for {step_ref}.") + with in_build_env(build_rule.environment): + try: + build_rule.run_build(build_context, manifest) + except (RuntimeError, AttributeError) as e: + if context.parsed_args.debug_mode: + raise e + _error( + f"An error occurred during build/install at {step_ref} (defined at {build_rule.attribute_path.path}): {str(e)}" + ) + _info(f"Completed build for {step_ref}.") + + else: + build_system = auto_detect_buildsystem(manifest) + if build_system: + _info(f"Auto-detected build system: {build_system.__class__.__name__}") + build_context = BuildContext.from_command_context(context) + with in_build_env(build_system.environment): + with in_build_env(build_system.environment): + build_system.run_build( + build_context, + manifest, + ) + + _non_verbose_info("Upstream builds completed successfully") + else: + _info("No build system was detected from the current plugin set.") + + +def remove_unnecessary_env() -> None: + vs = [ + "XDG_CACHE_HOME", + "XDG_CONFIG_DIRS", + "XDG_CONFIG_HOME", + "XDG_DATA_HOME", + "XDG_DATA_DIRS", + "XDG_RUNTIME_DIR", + ] + for v in vs: + if v in os.environ: + del os.environ[v] + + # FIXME: Add custom HOME + XDG_RUNTIME_DIR + + +@contextlib.contextmanager +def _setup_build_env(build_env: BuildEnvironmentDefinition) -> Iterator[None]: + env_backup = dict(os.environ) + env = dict(env_backup) + had_delta = False + build_env.update_env(env) + if env != env_backup: + _set_env(env) + had_delta = True + _info("Updated environment to match build") + yield + if had_delta or env != env_backup: + _set_env(env_backup) + + +def _set_env(desired_env: Mapping[str, str]) -> None: + os_env = os.environ + for key in os_env.keys() | desired_env.keys(): + desired_value = desired_env.get(key) + if desired_value is None: + try: + del os_env[key] + except KeyError: + pass + else: + os_env[key] = desired_value diff --git a/src/debputy/build_support/buildsystem_detection.py b/src/debputy/build_support/buildsystem_detection.py new file mode 100644 index 0000000..47415fd --- /dev/null +++ b/src/debputy/build_support/buildsystem_detection.py @@ -0,0 +1,112 @@ +from typing import ( + Optional, +) + +from debputy.exceptions import ( + DebputyPluginRuntimeError, + PluginBaseError, +) +from debputy.filesystem_scan import FSRootDir, FSROOverlay +from debputy.highlevel_manifest import HighLevelManifest +from debputy.manifest_parser.base_types import BuildEnvironmentDefinition +from debputy.manifest_parser.util import AttributePath +from debputy.plugin.debputy.to_be_api_types import ( + BuildSystemRule, +) +from debputy.plugin.plugin_state import run_in_context_of_plugin_wrap_errors +from debputy.util import ( + _error, + _debug_log, +) + + +def default_build_environment_only( + manifest: HighLevelManifest, +) -> BuildEnvironmentDefinition: + build_envs = manifest.build_environments + if build_envs.environments: + _error( + 'When automatic build system detection is used, the manifest cannot use "build-environments"' + ) + build_env = build_envs.default_environment + assert build_env is not None + return build_env + + +def auto_detect_buildsystem( + manifest: HighLevelManifest, +) -> Optional[BuildSystemRule]: + auto_detectable_build_systems = ( + manifest.plugin_provided_feature_set.auto_detectable_build_systems + ) + excludes = set() + options = [] + _debug_log("Auto-detecting build systems.") + source_root = FSROOverlay.create_root_dir("", ".") + for ppadbs in auto_detectable_build_systems.values(): + detected = ppadbs.detector(source_root) + if not isinstance(detected, bool): + _error( + f'The auto-detector for the build system {ppadbs.manifest_keyword} returned a "non-bool"' + f" ({detected!r}), which could be a bug in the plugin or the plugin relying on a newer" + " version of `debputy` that changed the auto-detection protocol." + ) + if not detected: + _debug_log( + f"Skipping build system {ppadbs.manifest_keyword}: Detector returned False!" + ) + continue + _debug_log( + f"Considering build system {ppadbs.manifest_keyword} as its Detector returned True!" + ) + if ppadbs.auto_detection_shadow_build_systems: + names = ", ".join( + sorted(x for x in ppadbs.auto_detection_shadow_build_systems) + ) + _debug_log(f"Build system {ppadbs.manifest_keyword} excludes: {names}!") + excludes.update(ppadbs.auto_detection_shadow_build_systems) + options.append(ppadbs) + + if not options: + _debug_log("Zero candidates; continuing without a build system") + return None + + if excludes: + names = ", ".join(sorted(x for x in excludes)) + _debug_log(f"The following build systems have been excluded: {names}!") + remaining_options = [o for o in options if o.manifest_keyword not in excludes] + else: + remaining_options = options + + if len(remaining_options) > 1: + names = ", ".join(o.manifest_keyword for o in remaining_options) + # TODO: This means adding an auto-detectable build system to an existing plugin causes FTBFS + # We need a better way of handling this. Probably the build systems should include + # a grace timer based on d/changelog. Anything before the changelog date is in + # "grace mode" and will not conflict with a build system that is. If all choices + # are in "grace mode", "oldest one" wins. + _error( + f"Multiple build systems match, please pick one explicitly (under `builds:`): {names}" + ) + + if not remaining_options: + names = ", ".join(o.build_system_rule_type.__name__ for o in options) + # TODO: Detect at registration time + _error( + f"Multiple build systems matched but they all shadowed each other: {names}." + f" There is a bug in at least one of them!" + ) + + chosen_build_system = remaining_options[0] + environment = default_build_environment_only(manifest) + bs = run_in_context_of_plugin_wrap_errors( + chosen_build_system.plugin_metadata.plugin_name, + chosen_build_system.constructor, + { + "environment": environment, + }, + AttributePath.builtin_path(), + manifest, + ) + bs.auto_generated_stem = "" + return bs diff --git a/src/debputy/build_support/clean_logic.py b/src/debputy/build_support/clean_logic.py new file mode 100644 index 0000000..13347b0 --- /dev/null +++ b/src/debputy/build_support/clean_logic.py @@ -0,0 +1,233 @@ +import os.path +from typing import ( + Set, + cast, + List, +) + +from debputy.build_support.build_context import BuildContext +from debputy.build_support.build_logic import ( + in_build_env, + assign_stems, +) +from debputy.build_support.buildsystem_detection import auto_detect_buildsystem +from debputy.commands.debputy_cmd.context import CommandContext +from debputy.highlevel_manifest import HighLevelManifest +from debputy.plugin.debputy.to_be_api_types import BuildSystemRule, CleanHelper +from debputy.util import _info, print_command, _error, _debug_log, _warn +from debputy.util import ( + run_build_system_command, +) + +_REMOVE_DIRS = frozenset( + [ + "__pycache__", + "autom4te.cache", + ] +) +_IGNORE_DIRS = frozenset( + [ + ".git", + ".svn", + ".bzr", + ".hg", + "CVS", + ".pc", + "_darcs", + ] +) +DELETE_FILE_EXT = ( + "~", + ".orig", + ".rej", + ".bak", +) +DELETE_FILE_BASENAMES = { + "DEADJOE", + ".SUMS", + "TAGS", +} + + +def _debhelper_left_overs() -> bool: + if os.path.lexists("debian/.debhelper") or os.path.lexists( + "debian/debhelper-build-stamp" + ): + return True + with os.scandir(".") as root_dir: + for child in root_dir: + if child.is_file(follow_symlinks=False) and ( + child.name.endswith(".debhelper.log") + or child.name.endswith(".debhelper") + ): + return True + return False + + +class CleanHelperImpl(CleanHelper): + + def __init__(self) -> None: + self.files_to_remove: Set[str] = set() + self.dirs_to_remove: Set[str] = set() + + def schedule_removal_of_files(self, *args: str) -> None: + self.files_to_remove.update(args) + + def schedule_removal_of_directories(self, *args: str) -> None: + if any(p == "/" for p in args): + raise ValueError("Refusing to delete '/'") + self.dirs_to_remove.update(args) + + +def _scan_for_standard_removals(clean_helper: CleanHelperImpl) -> None: + remove_files = clean_helper.files_to_remove + remove_dirs = clean_helper.dirs_to_remove + with os.scandir(".") as root_dir: + for child in root_dir: + if child.is_file(follow_symlinks=False) and child.name.endswith("-stamp"): + remove_files.add(child.path) + for current_dir, subdirs, files in os.walk("."): + for remove_dir in [d for d in subdirs if d in _REMOVE_DIRS]: + path = os.path.join(current_dir, remove_dir) + remove_dirs.add(path) + subdirs.remove(remove_dir) + for skip_dir in [d for d in subdirs if d in _IGNORE_DIRS]: + subdirs.remove(skip_dir) + + for basename in files: + if ( + basename.endswith(DELETE_FILE_EXT) + or basename in DELETE_FILE_BASENAMES + or (basename.startswith("#") and basename.endswith("#")) + ): + path = os.path.join(current_dir, basename) + remove_files.add(path) + + +def perform_clean( + context: CommandContext, + manifest: HighLevelManifest, +) -> None: + clean_helper = CleanHelperImpl() + + build_rules = manifest.build_rules + if build_rules is not None: + if not build_rules: + # Defined but empty disables the auto-detected build system + return + active_packages = frozenset(manifest.active_packages) + condition_context = manifest.source_condition_context + build_context = BuildContext.from_command_context(context) + assign_stems(build_rules, manifest) + for step_no, build_rule in enumerate(build_rules): + step_ref = ( + f"step {step_no} [{build_rule.auto_generated_stem}]" + if build_rule.name is None + else f"step {step_no} [{build_rule.name}]" + ) + if not build_rule.is_buildsystem: + _debug_log(f"Skipping clean for {step_ref}: Not a build system") + continue + build_system_rule: BuildSystemRule = cast("BuildSystemRule", build_rule) + if build_system_rule.for_packages.isdisjoint(active_packages): + _info( + f"Skipping build for {step_ref}: None of the relevant packages are being built" + ) + continue + manifest_condition = build_system_rule.manifest_condition + if manifest_condition is not None and not manifest_condition.evaluate( + condition_context + ): + _info( + f"Skipping clean for {step_ref}: The condition clause evaluated to false" + ) + continue + _info(f"Starting clean for {step_ref}.") + with in_build_env(build_rule.environment): + try: + build_system_rule.run_clean( + build_context, + manifest, + clean_helper, + ) + except (RuntimeError, AttributeError) as e: + if context.parsed_args.debug_mode: + raise e + _error( + f"An error occurred during clean at {step_ref} (defined at {build_rule.attribute_path.path}): {str(e)}" + ) + _info(f"Completed clean for {step_ref}.") + else: + build_system = auto_detect_buildsystem(manifest) + if build_system: + _info(f"Auto-detected build system: {build_system.__class__.__name__}") + build_context = BuildContext.from_command_context(context) + with in_build_env(build_system.environment): + build_system.run_clean( + build_context, + manifest, + clean_helper, + ) + else: + _info("No build system was detected from the current plugin set.") + + dh_autoreconf_used = os.path.lexists("debian/autoreconf.before") + debhelper_used = False + + if dh_autoreconf_used or _debhelper_left_overs(): + debhelper_used = True + + _scan_for_standard_removals(clean_helper) + + for package in manifest.all_packages: + package_staging_dir = os.path.join("debian", package.name) + if os.path.lexists(package_staging_dir): + clean_helper.schedule_removal_of_directories(package_staging_dir) + + remove_files = clean_helper.files_to_remove + remove_dirs = clean_helper.dirs_to_remove + if remove_files: + print_command("rm", "-f", *remove_files) + _remove_files_if_exists(*remove_files) + if remove_dirs: + run_build_system_command("rm", "-fr", *remove_dirs) + + if debhelper_used: + _info( + "Noted traces of debhelper commands being used; invoking dh_clean to clean up after them" + ) + if dh_autoreconf_used: + run_build_system_command("dh_autoreconf_clean") + run_build_system_command("dh_clean") + + try: + run_build_system_command("dpkg-buildtree", "clean") + except FileNotFoundError: + _warn("The dpkg-buildtree command is not present. Emulating it") + # This is from the manpage of dpkg-buildtree for 1.22.11. + _remove_files_if_exists( + "debian/files", + "debian/files.new", + "debian/substvars", + "debian/substvars.new", + ) + run_build_system_command("rm", "-fr", "debian/tmp") + # Remove debian/.debputy as a separate step. While `rm -fr` should process things in order, + # it will continue on error, which could cause our manifests of things to delete to be deleted + # while leaving things half-removed unless we do this extra step. + run_build_system_command("rm", "-fr", "debian/.debputy") + + +def _remove_files_if_exists(*args: str) -> None: + for path in args: + try: + os.unlink(path) + except FileNotFoundError: + continue + except OSError as e: + if os.path.isdir(path): + _error( + f"Failed to remove {path}: It is a directory, but it should have been a non-directory." + " Please verify everything is as expected and, if it is, remove it manually." + ) + _error(f"Failed to remove {path}: {str(e)}") diff --git a/src/debputy/builtin_manifest_rules.py b/src/debputy/builtin_manifest_rules.py index c8e6557..e31a50e 100644 --- a/src/debputy/builtin_manifest_rules.py +++ b/src/debputy/builtin_manifest_rules.py @@ -17,7 +17,7 @@ from debputy.path_matcher import ( ) from debputy.substitution import Substitution from debputy.types import VP -from debputy.util import _normalize_path, perl_module_dirs +from debputy.util import _normalize_path, resolve_perl_config # Imported from dh_fixperms _PERMISSION_NORMALIZATION_SOURCE_DEFINITION = "permission normalization" @@ -218,20 +218,19 @@ def builtin_mode_normalization_rules( OctalMode(0o0644), ) + perl_config_data = resolve_perl_config(dpkg_architecture_variables, dctrl_bin) + yield from ( ( BasenameGlobMatch( "*.pm", - only_when_in_directory=perl_dir, + only_when_in_directory=_normalize_path(perl_dir), path_type=PathType.FILE, recursive_match=True, ), - SymbolicMode.parse_filesystem_mode( - "a-x", - attribute_path['"*.pm'], - ), + _STD_FILE_MODE, ) - for perl_dir in perl_module_dirs(dpkg_architecture_variables, dctrl_bin) + for perl_dir in (perl_config_data.vendorlib, perl_config_data.vendorarch) ) yield ( @@ -241,10 +240,7 @@ def builtin_mode_normalization_rules( path_type=PathType.FILE, recursive_match=True, ), - SymbolicMode.parse_filesystem_mode( - "a-w", - attribute_path['"*.ali"'], - ), + OctalMode(0o444), ) yield ( diff --git a/src/debputy/commands/debputy_cmd/__main__.py b/src/debputy/commands/debputy_cmd/__main__.py index 731576e..58a6fb4 100644 --- a/src/debputy/commands/debputy_cmd/__main__.py +++ b/src/debputy/commands/debputy_cmd/__main__.py @@ -22,15 +22,17 @@ from typing import ( ) from debputy import DEBPUTY_ROOT_DIR, DEBPUTY_PLUGIN_ROOT_DIR +from debputy._deb_options_profiles import DebBuildOptionsAndProfiles from debputy.analysis import REFERENCE_DATA_TABLE from debputy.analysis.debian_dir import scan_debian_dir +from debputy.build_support import perform_clean, perform_builds from debputy.commands.debputy_cmd.context import ( CommandContext, add_arg, ROOT_COMMAND, CommandArg, ) -from debputy.commands.debputy_cmd.output import _stream_to_pager +from debputy.commands.debputy_cmd.output import _stream_to_pager, _output_styling from debputy.dh_migration.migrators import MIGRATORS from debputy.exceptions import ( DebputyRuntimeError, @@ -40,10 +42,15 @@ from debputy.exceptions import ( UnhandledOrUnexpectedErrorFromPluginError, SymlinkLoopError, ) +from debputy.highlevel_manifest import HighLevelManifest from debputy.package_build.assemble_deb import ( assemble_debs, ) -from debputy.plugin.api.spec import INTEGRATION_MODE_DH_DEBPUTY_RRR +from debputy.plugin.api.spec import ( + INTEGRATION_MODE_DH_DEBPUTY_RRR, + DebputyIntegrationMode, + INTEGRATION_MODE_FULL, +) try: from argcomplete import autocomplete @@ -92,6 +99,9 @@ from debputy.util import ( escape_shell, program_name, integrated_with_debhelper, + PRINT_BUILD_SYSTEM_COMMAND, + PRINT_COMMAND, + change_log_level, ) @@ -253,6 +263,18 @@ def _add_packages_args(parser: argparse.ArgumentParser) -> None: ) +def _build_subcommand_log_level(context: CommandContext) -> int: + parsed_args = context.parsed_args + log_level: Optional[int] = None + if os.environ.get("DH_VERBOSE", "") != "": + log_level = PRINT_COMMAND + if parsed_args.debug_mode: + log_level = logging.INFO + if log_level is not None: + change_log_level(log_level) + return PRINT_BUILD_SYSTEM_COMMAND + + internal_commands = ROOT_COMMAND.add_dispatching_subcommand( "internal-command", dest="internal_command", @@ -628,10 +650,71 @@ def _run_tests_for_plugin(context: CommandContext) -> None: @internal_commands.register_subcommand( + "dpkg-build-driver-run-task", + help_description="[Internal command] Perform a given Dpkg::BuildDriver task (Not stable API)", + requested_plugins_only=True, + default_log_level=_build_subcommand_log_level, + argparser=[ + add_arg( + "task_name", + metavar="task-name", + choices=[ + "clean", + "build", + "build-arch", + "build-indep", + "binary", + "binary-arch", + "binary-indep", + ], + help="The task to run", + ), + add_arg( + "output", + nargs="?", + default="..", + metavar="output", + help="Where to place the resulting packages. Should be a directory", + ), + ], +) +def _dpkg_build_driver_integration(context: CommandContext) -> None: + parsed_args = context.parsed_args + log_level = context.set_log_level_for_build_subcommand() + task_name = parsed_args.task_name + + if task_name.endswith("-indep"): + context.package_set = "indep" + elif task_name.endswith("arch"): + context.package_set = "arch" + + manifest = context.parse_manifest() + + plugins = context.load_plugins().plugin_data + for plugin in plugins.values(): + if not plugin.is_bundled: + _info(f"Loaded plugin {plugin.plugin_name}") + if task_name == "clean": + perform_clean(context, manifest) + elif task_name in ("build", "build-indep", "build-arch"): + perform_builds(context, manifest) + elif task_name in ("binary", "binary-indep", "binary-arch"): + perform_builds(context, manifest) + assemble( + context, + manifest, + INTEGRATION_MODE_FULL, + debug_materialization=log_level is not None, + ) + else: + _error(f"Unsupported Dpkg::BuildDriver task: {task_name}.") + + +@internal_commands.register_subcommand( "dh-integration-generate-debs", help_description="[Internal command] Generate .deb/.udebs packages from debian/<pkg> (Not stable API)", requested_plugins_only=True, - default_log_level=logging.WARN, + default_log_level=_build_subcommand_log_level, argparser=[ _add_packages_args, add_arg( @@ -651,7 +734,7 @@ def _run_tests_for_plugin(context: CommandContext) -> None: ) def _dh_integration_generate_debs(context: CommandContext) -> None: integrated_with_debhelper() - parsed_args = context.parsed_args + log_level = context.set_log_level_for_build_subcommand() integration_mode = context.resolve_integration_mode() is_dh_rrr_only_mode = integration_mode == INTEGRATION_MODE_DH_DEBPUTY_RRR if is_dh_rrr_only_mode: @@ -662,9 +745,6 @@ def _dh_integration_generate_debs(context: CommandContext) -> None: _error( f"Plugins are not supported in the zz-debputy-rrr sequence. Detected plugins: {plugin_names}" ) - debug_materialization = ( - os.environ.get("DH_VERBOSE", "") != "" or parsed_args.debug_mode - ) plugins = context.load_plugins().plugin_data for plugin in plugins.values(): @@ -672,13 +752,26 @@ def _dh_integration_generate_debs(context: CommandContext) -> None: _info(f"Loaded plugin {plugin.plugin_name}") manifest = context.parse_manifest() - package_data_table = manifest.perform_installations( - enable_manifest_installation_feature=not is_dh_rrr_only_mode + assemble( + context, + manifest, + integration_mode, + debug_materialization=log_level is not None, ) + + +def assemble( + context: CommandContext, + manifest: HighLevelManifest, + integration_mode: DebputyIntegrationMode, + *, + debug_materialization: bool = False, +) -> None: source_fs = FSROOverlay.create_root_dir("..", ".") source_version = manifest.source_version() is_native = "-" not in source_version - + is_dh_rrr_only_mode = integration_mode == INTEGRATION_MODE_DH_DEBPUTY_RRR + package_data_table = manifest.perform_installations(integration_mode) if not is_dh_rrr_only_mode: for dctrl_bin in manifest.active_packages: package = dctrl_bin.name @@ -696,7 +789,7 @@ def _dh_integration_generate_debs(context: CommandContext) -> None: fs_root, dctrl_data.substvars, ) - if "nostrip" not in manifest.build_env.deb_build_options: + if "nostrip" not in manifest.deb_options_and_profiles.deb_build_options: dbgsym_ids = relocate_dwarves_into_dbgsym_packages( dctrl_bin, fs_root, @@ -708,7 +801,7 @@ def _dh_integration_generate_debs(context: CommandContext) -> None: dctrl_bin, fs_root, is_native, - manifest.build_env, + manifest.deb_options_and_profiles, ) if not is_native: install_upstream_changelog( @@ -895,10 +988,10 @@ def _json_output(data: Any) -> None: ], ) def _migrate_from_dh(context: CommandContext) -> None: + context.must_be_called_in_source_root() parsed_args = context.parsed_args - resolved_migration_target = _check_migration_target( - context.debian_dir, + context, parsed_args.migration_target, ) context.debputy_integration_mode = resolved_migration_target @@ -909,6 +1002,7 @@ def _migrate_from_dh(context: CommandContext) -> None: ) ) migrate_from_dh( + _output_styling(context.parsed_args, sys.stdout), manifest, acceptable_migration_issues, parsed_args.destructive, diff --git a/src/debputy/commands/debputy_cmd/context.py b/src/debputy/commands/debputy_cmd/context.py index 0c184c7..d0a6cf7 100644 --- a/src/debputy/commands/debputy_cmd/context.py +++ b/src/debputy/commands/debputy_cmd/context.py @@ -15,6 +15,7 @@ from typing import ( Callable, Dict, TYPE_CHECKING, + Literal, ) from debian.debian_support import DpkgArchTable @@ -45,7 +46,15 @@ from debputy.substitution import ( SubstitutionImpl, NULL_SUBSTITUTION, ) -from debputy.util import _error, PKGNAME_REGEX, resolve_source_date_epoch, setup_logging +from debputy.util import ( + _error, + PKGNAME_REGEX, + resolve_source_date_epoch, + setup_logging, + PRINT_COMMAND, + change_log_level, + _warn, +) if TYPE_CHECKING: from argparse import _SubParsersAction @@ -81,6 +90,22 @@ class Command: requested_plugins_only: bool = False +def _host_dpo_to_dbo(opt_and_profiles: "DebBuildOptionsAndProfiles", v: str) -> bool: + + if ( + v in opt_and_profiles.deb_build_profiles + and v not in opt_and_profiles.deb_build_options + ): + val = os.environ.get("DEB_BUILD_OPTIONS", "") + " " + v + _warn( + f'Copying "{v}" into DEB_BUILD_OPTIONS: It was in DEB_BUILD_PROFILES but not in DEB_BUILD_OPTIONS' + ) + os.environ["DEB_BUILD_OPTIONS"] = val.lstrip() + # Note: It will not be immediately visible since `DebBuildOptionsAndProfiles` caches the result + return True + return False + + class CommandContext: def __init__( self, @@ -110,6 +135,19 @@ class CommandContext: Mapping[str, "BinaryPackage"], ] ] = None + self._package_set: Literal["both", "arch", "indep"] = "both" + + @property + def package_set(self) -> Literal["both", "arch", "indep"]: + return self._package_set + + @package_set.setter + def package_set(self, new_value: Literal["both", "arch", "indep"]) -> None: + if self._dctrl_parser is not None: + raise TypeError( + "package_set cannot be redefined once the debian/control parser has been initialized" + ) + self._package_set = new_value @property def debian_dir(self) -> VirtualPath: @@ -132,12 +170,21 @@ class CommandContext: if hasattr(self.parsed_args, "packages"): packages = self.parsed_args.packages + instance = DebBuildOptionsAndProfiles(environ=os.environ) + + dirty = _host_dpo_to_dbo(instance, "nodoc") + dirty = _host_dpo_to_dbo(instance, "nocheck") or dirty + + if dirty: + instance = DebBuildOptionsAndProfiles(environ=os.environ) + parser = DctrlParser( packages, # -p/--package set(), # -N/--no-package - False, # -i - False, # -a - build_env=DebBuildOptionsAndProfiles.instance(), + # binary-indep and binary-indep (dpkg BuildDriver integration only) + self._package_set == "indep", + self._package_set == "arch", + deb_options_and_profiles=instance, dpkg_architecture_variables=dpkg_architecture_table(), dpkg_arch_query_table=DpkgArchTable.load_arch_table(), ) @@ -152,6 +199,9 @@ class CommandContext: _, binary_package_table = self._parse_dctrl() return binary_package_table + def dpkg_architecture_variables(self) -> DpkgArchitectureBuildProcessValuesTable: + return self.dctrl_parser.dpkg_architecture_variables + def requested_plugins(self) -> Sequence[str]: if self._requested_plugins is None: self._requested_plugins = self._resolve_requested_plugins() @@ -162,7 +212,7 @@ class CommandContext: @property def deb_build_options_and_profiles(self) -> "DebBuildOptionsAndProfiles": - return self.dctrl_parser.build_env + return self.dctrl_parser.deb_options_and_profiles @property def deb_build_options(self) -> Mapping[str, Optional[str]]: @@ -292,20 +342,37 @@ class CommandContext: debian_control = self.debian_dir.get("control") return debian_control is not None - def resolve_integration_mode(self) -> DebputyIntegrationMode: + def resolve_integration_mode( + self, + require_integration: bool = True, + ) -> DebputyIntegrationMode: integration_mode = self.debputy_integration_mode if integration_mode is None: r = read_dh_addon_sequences(self.debian_dir) bd_sequences, dr_sequences, _ = r all_sequences = bd_sequences | dr_sequences - integration_mode = determine_debputy_integration_mode(all_sequences) - if integration_mode is None: + integration_mode = determine_debputy_integration_mode( + self.source_package().fields, + all_sequences, + ) + if integration_mode is None and not require_integration: _error( "Cannot resolve the integration mode expected for this package. Is this package using `debputy`?" ) self.debputy_integration_mode = integration_mode return integration_mode + def set_log_level_for_build_subcommand(self) -> Optional[int]: + parsed_args = self.parsed_args + log_level: Optional[int] = None + if os.environ.get("DH_VERBOSE", "") != "": + log_level = PRINT_COMMAND + if parsed_args.debug_mode or os.environ.get("DEBPUTY_DEBUG", "") != "": + log_level = logging.DEBUG + if log_level is not None: + change_log_level(log_level) + return log_level + def manifest_parser( self, *, @@ -320,7 +387,6 @@ class CommandContext: manifest_path = self.parsed_args.debputy_manifest if manifest_path is None: manifest_path = os.path.join(self.debian_dir.fs_path, "debputy.manifest") - debian_dir = self.debian_dir return YAMLManifestParser( manifest_path, source_package, @@ -328,7 +394,7 @@ class CommandContext: substitution, dctrl_parser.dpkg_architecture_variables, dctrl_parser.dpkg_arch_query_table, - dctrl_parser.build_env, + dctrl_parser.deb_options_and_profiles, self.load_plugins(), self.resolve_integration_mode(), debian_dir=self.debian_dir, @@ -420,7 +486,7 @@ class GenericSubCommand(SubcommandBase): require_substitution: bool = True, requested_plugins_only: bool = False, log_only_to_stderr: bool = False, - default_log_level: int = logging.INFO, + default_log_level: Union[int, Callable[[CommandContext], int]] = logging.INFO, ) -> None: super().__init__(name, aliases=aliases, help_description=help_description) self._handler = handler @@ -452,7 +518,18 @@ class GenericSubCommand(SubcommandBase): ) if self._log_only_to_stderr: setup_logging(reconfigure_logging=True, log_only_to_stderr=True) - logging.getLogger().setLevel(self._default_log_level) + + default_log_level = self._default_log_level + if isinstance(default_log_level, int): + level = default_log_level + else: + assert callable(default_log_level) + level = default_log_level(context) + change_log_level(level) + if level > logging.DEBUG and ( + context.parsed_args.debug_mode or os.environ.get("DEBPUTY_DEBUG", "") != "" + ): + change_log_level(logging.DEBUG) return self._handler(context) @@ -494,7 +571,7 @@ class DispatchingCommandMixin(CommandBase): require_substitution: bool = True, requested_plugins_only: bool = False, log_only_to_stderr: bool = False, - default_log_level: int = logging.INFO, + default_log_level: Union[int, Callable[[CommandContext], int]] = logging.INFO, ) -> Callable[[CommandHandler], GenericSubCommand]: if isinstance(name, str): cmd_name = name diff --git a/src/debputy/commands/debputy_cmd/lint_and_lsp_cmds.py b/src/debputy/commands/debputy_cmd/lint_and_lsp_cmds.py index 46b536b..a03126b 100644 --- a/src/debputy/commands/debputy_cmd/lint_and_lsp_cmds.py +++ b/src/debputy/commands/debputy_cmd/lint_and_lsp_cmds.py @@ -22,21 +22,22 @@ _EDITOR_SNIPPETS = { (add-to-list 'auto-mode-alist '("/debian/debputy.manifest\\'" . yaml-mode)) ;; Inform eglot about the debputy LSP (with-eval-after-load 'eglot - (add-to-list 'eglot-server-programs - '(debian-control-mode . ("debputy" "lsp" "server" "--ignore-language-ids"))) - (add-to-list 'eglot-server-programs - '(debian-changelog-mode . ("debputy" "lsp" "server" "--ignore-language-ids"))) - (add-to-list 'eglot-server-programs - '(debian-copyright-mode . ("debputy" "lsp" "server" "--ignore-language-ids"))) - ;; Requires elpa-dpkg-dev-el (>= 37.12) - (add-to-list 'eglot-server-programs - '(debian-autopkgtest-control-mode . ("debputy" "lsp" "server" "--ignore-language-ids"))) - ;; The debian/rules file uses the qmake mode. - (add-to-list 'eglot-server-programs - '(makefile-gmake-mode . ("debputy" "lsp" "server" "--ignore-language-ids"))) - (add-to-list 'eglot-server-programs - '(yaml-mode . ("debputy" "lsp" "server" "--ignore-language-ids"))) - ) + (add-to-list 'eglot-server-programs + '( + ( + ;; Requires elpa-dpkg-dev-el (>= 37.12) + (debian-autopkgtest-control-mode :language-id "debian/tests/control") + ;; Requires elpa-dpkg-dev-el + (debian-control-mode :language-id "debian/control") + (debian-changelog-mode :language-id "debian/changelog") + (debian-copyright-mode :language-id "debian/copyright") + ;; No language id for these atm. + makefile-gmake-mode + ;; Requires elpa-yaml-mode + yaml-mode + ) + . ("debputy" "lsp" "server") + ))) ;; Auto-start eglot for the relevant modes. (add-hook 'debian-control-mode-hook 'eglot-ensure) @@ -182,7 +183,7 @@ def lsp_server_cmd(context: CommandContext) -> None: debputy_language_server.dctrl_parser = context.dctrl_parser debputy_language_server.trust_language_ids = parsed_args.trust_language_ids - debputy_language_server.finish_initialization() + debputy_language_server.finish_startup_initialization() if parsed_args.tcp and parsed_args.ws: _error("Sorry, --tcp and --ws are mutually exclusive") @@ -266,7 +267,6 @@ def lsp_describe_features(context: CommandContext) -> None: "--spellcheck", dest="spellcheck", action="store_true", - shared=True, help="Enable spellchecking", ), add_arg( diff --git a/src/debputy/commands/debputy_cmd/output.py b/src/debputy/commands/debputy_cmd/output.py index 5159980..334eab4 100644 --- a/src/debputy/commands/debputy_cmd/output.py +++ b/src/debputy/commands/debputy_cmd/output.py @@ -192,6 +192,9 @@ class OutputStylingBase: def render_url(self, link_url: str) -> str: return link_url + def bts(self, bugno) -> str: + return f"https://bugs.debian.org/{bugno}" + class ANSIOutputStylingBase(OutputStylingBase): def __init__( @@ -260,6 +263,25 @@ class ANSIOutputStylingBase(OutputStylingBase): link_url = f"https://manpages.debian.org/{page}.{section}" return URL_START + f"{link_url}\a{link_text}" + URL_END + def bts(self, bugno) -> str: + if not self._support_clickable_urls: + return super().bts(bugno) + return self.render_url(f"https://bugs.debian.org/{bugno}") + + +def no_fancy_output( + stream: IO[str] = None, + output_format: str = str, + optimize_for_screen_reader: bool = False, +) -> OutputStylingBase: + if stream is None: + stream = sys.stdout + return OutputStylingBase( + stream, + output_format, + optimize_for_screen_reader=optimize_for_screen_reader, + ) + def _output_styling( parsed_args: argparse.Namespace, @@ -270,9 +292,12 @@ def _output_styling( output_format = "text" optimize_for_screen_reader = os.environ.get("OPTIMIZE_FOR_SCREEN_READER", "") != "" if not stream.isatty(): - return OutputStylingBase( - stream, output_format, optimize_for_screen_reader=optimize_for_screen_reader + return no_fancy_output( + stream, + output_format, + optimize_for_screen_reader=optimize_for_screen_reader, ) + return ANSIOutputStylingBase( stream, output_format, optimize_for_screen_reader=optimize_for_screen_reader ) diff --git a/src/debputy/commands/debputy_cmd/plugin_cmds.py b/src/debputy/commands/debputy_cmd/plugin_cmds.py index 83bb88f..9721702 100644 --- a/src/debputy/commands/debputy_cmd/plugin_cmds.py +++ b/src/debputy/commands/debputy_cmd/plugin_cmds.py @@ -2,6 +2,7 @@ import argparse import operator import os import sys +import textwrap from itertools import chain from typing import ( Sequence, @@ -28,7 +29,7 @@ from debputy.commands.debputy_cmd.output import ( ) from debputy.exceptions import DebputySubstitutionError from debputy.filesystem_scan import build_virtual_fs -from debputy.manifest_parser.base_types import TypeMapping +from debputy.manifest_parser.tagging_types import TypeMapping from debputy.manifest_parser.declarative_parser import ( BASIC_SIMPLE_TYPES, ) @@ -45,13 +46,15 @@ from debputy.plugin.api.impl_types import ( PackagerProvidedFileClassSpec, PluginProvidedManifestVariable, DispatchingParserBase, - SUPPORTED_DISPATCHABLE_TABLE_PARSERS, - OPARSER_MANIFEST_ROOT, PluginProvidedDiscardRule, AutomaticDiscardRuleExample, MetadataOrMaintscriptDetector, PluginProvidedTypeMapping, ) +from debputy.plugin.api.parser_tables import ( + SUPPORTED_DISPATCHABLE_TABLE_PARSERS, + OPARSER_MANIFEST_ROOT, +) from debputy.plugin.api.spec import ( TypeMappingExample, ) @@ -538,7 +541,16 @@ def _plugin_cmd_show_manifest_variables(context: CommandContext) -> None: variable_value=None, is_context_specific_variable=False, is_documentation_placeholder=True, - variable_reference_documentation=f'Environment variable "{env_var}"', + variable_reference_documentation=textwrap.dedent( + f"""\ + Environment variable "{env_var}" + + Note that uses beneath `builds:` may use the environment variable defined by + `build-environment:` (depends on whether the rule uses eager or lazy + substitution) while uses outside `builds:` will generally not use a definition + from `build-environment:`. + """ + ), ) else: variable = variables.get(variable_name) diff --git a/src/debputy/deb_packaging_support.py b/src/debputy/deb_packaging_support.py index 897c2b6..92f57a2 100644 --- a/src/debputy/deb_packaging_support.py +++ b/src/debputy/deb_packaging_support.py @@ -81,7 +81,7 @@ from debputy.util import ( _error, ensure_dir, assume_not_none, - perl_module_dirs, + resolve_perl_config, perlxs_api_dependency, detect_fakeroot, grouper, @@ -186,11 +186,11 @@ def handle_perl_code( fs_root: FSPath, substvars: FlushableSubstvars, ) -> None: - known_perl_inc_dirs = perl_module_dirs(dpkg_architecture_variables, dctrl_bin) + perl_config_data = resolve_perl_config(dpkg_architecture_variables, dctrl_bin) detected_dep_requirements = 0 # MakeMaker always makes lib and share dirs, but typically only one directory is actually used. - for perl_inc_dir in known_perl_inc_dirs: + for perl_inc_dir in (perl_config_data.vendorarch, perl_config_data.vendorlib): p = fs_root.lookup(perl_inc_dir) if p and p.is_dir: p.prune_if_empty_dir() @@ -198,8 +198,8 @@ def handle_perl_code( # FIXME: 80% of this belongs in a metadata detector, but that requires us to expose .walk() in the public API, # which will not be today. for d, pm_mode in [ - (known_perl_inc_dirs.vendorlib, PERL_DEP_INDEP_PM_MODULE), - (known_perl_inc_dirs.vendorarch, PERL_DEP_ARCH_PM_MODULE), + (perl_config_data.vendorlib, PERL_DEP_INDEP_PM_MODULE), + (perl_config_data.vendorarch, PERL_DEP_ARCH_PM_MODULE), ]: inc_dir = fs_root.lookup(d) if not inc_dir: @@ -1337,6 +1337,8 @@ def _generate_dbgsym_control_file_if_relevant( component = section.split("/", 1)[1] + "/" if multi_arch != "same": extra_params.append("-UMulti-Arch") + else: + extra_params.append(f"-DMulti-Arch={multi_arch}") extra_params.append("-UReplaces") extra_params.append("-UBreaks") dbgsym_control_dir = os.path.join(dbgsym_root_dir, "DEBIAN") diff --git a/src/debputy/dh/dh_assistant.py b/src/debputy/dh/dh_assistant.py index ba8c14f..836b1b5 100644 --- a/src/debputy/dh/dh_assistant.py +++ b/src/debputy/dh/dh_assistant.py @@ -7,7 +7,6 @@ from typing import Iterable, FrozenSet, Optional, List, Union, Mapping, Any, Set from debian.deb822 import Deb822 from debputy.plugin.api import VirtualPath -from debputy.util import _info _FIND_DH_WITH = re.compile(r"--with(?:\s+|=)(\S+)") _DEP_REGEX = re.compile("^([a-z0-9][-+.a-z0-9]+)", re.ASCII) @@ -123,3 +122,25 @@ def read_dh_addon_sequences( extract_dh_addons_from_control(source_paragraph, bd_sequences) return bd_sequences, dr_sequences, saw_dh return None + + +def extract_dh_compat_level(*, cwd=None) -> Tuple[Optional[int], int]: + try: + output = subprocess.check_output( + ["dh_assistant", "active-compat-level"], + stderr=subprocess.DEVNULL, + cwd=cwd, + ) + except (FileNotFoundError, subprocess.CalledProcessError) as e: + exit_code = 127 + if isinstance(e, subprocess.CalledProcessError): + exit_code = e.returncode + return None, exit_code + else: + data = json.loads(output) + active_compat_level = data.get("active-compat-level") + exit_code = 0 + if not isinstance(active_compat_level, int) or active_compat_level < 1: + active_compat_level = None + exit_code = 255 + return active_compat_level, exit_code diff --git a/src/debputy/dh_migration/migration.py b/src/debputy/dh_migration/migration.py index f7b7d9e..7b06135 100644 --- a/src/debputy/dh_migration/migration.py +++ b/src/debputy/dh_migration/migration.py @@ -3,10 +3,12 @@ import os import re import subprocess from itertools import chain -from typing import Optional, List, Callable, Set, Container +from typing import Optional, List, Callable, Set, Container, Mapping, FrozenSet from debian.deb822 import Deb822 +from debputy.commands.debputy_cmd.context import CommandContext +from debputy.commands.debputy_cmd.output import OutputStylingBase from debputy.dh.debhelper_emulation import CannotEmulateExecutableDHConfigFile from debputy.dh_migration.migrators import MIGRATORS from debputy.dh_migration.migrators_impl import ( @@ -24,11 +26,28 @@ from debputy.highlevel_manifest import HighLevelManifest from debputy.integration_detection import determine_debputy_integration_mode from debputy.manifest_parser.exceptions import ManifestParseException from debputy.plugin.api import VirtualPath -from debputy.plugin.api.spec import DebputyIntegrationMode +from debputy.plugin.api.spec import DebputyIntegrationMode, INTEGRATION_MODE_FULL from debputy.util import _error, _warn, _info, escape_shell, assume_not_none +SUPPORTED_MIGRATIONS: Mapping[ + DebputyIntegrationMode, FrozenSet[DebputyIntegrationMode] +] = { + INTEGRATION_MODE_FULL: frozenset([INTEGRATION_MODE_FULL]), + INTEGRATION_MODE_DH_DEBPUTY: frozenset( + [INTEGRATION_MODE_DH_DEBPUTY, INTEGRATION_MODE_FULL] + ), + INTEGRATION_MODE_DH_DEBPUTY_RRR: frozenset( + [ + INTEGRATION_MODE_DH_DEBPUTY_RRR, + INTEGRATION_MODE_DH_DEBPUTY, + INTEGRATION_MODE_FULL, + ] + ), +} + def _print_migration_summary( + fo: OutputStylingBase, migrations: List[FeatureMigration], compat: int, min_compat_level: int, @@ -42,9 +61,11 @@ def _print_migration_summary( continue underline = "-" * len(migration.tagline) if migration.warnings: + if warning_count: + _warn("") _warn(f"Summary for migration: {migration.tagline}") - _warn(f"-----------------------{underline}") - _warn(" /!\\ ATTENTION /!\\") + if not fo.optimize_for_screen_reader: + _warn(f"-----------------------{underline}") warning_count += len(migration.warnings) for warning in migration.warnings: _warn(f" * {warning}") @@ -53,7 +74,8 @@ def _print_migration_summary( if warning_count: _warn("") _warn("Supported debhelper compat check") - _warn("--------------------------------") + if not fo.optimize_for_screen_reader: + _warn("--------------------------------") warning_count += 1 _warn( f"The migration tool assumes debhelper compat {min_compat_level}+ semantics, but this package" @@ -68,7 +90,8 @@ def _print_migration_summary( if warning_count: _warn("") _warn("Missing debputy plugin check") - _warn("----------------------------") + if not fo.optimize_for_screen_reader: + _warn("----------------------------") _warn( f"The migration tool could not read d/control and therefore cannot tell if all the required" f" plugins have been requested. Please ensure that the package Build-Depends on: {needed_plugins}" @@ -83,7 +106,8 @@ def _print_migration_summary( if warning_count: _warn("") _warn("Missing debputy plugin check") - _warn("----------------------------") + if not fo.optimize_for_screen_reader: + _warn("----------------------------") _warn( f"The migration tool asserted that the following `debputy` plugins would be required, which" f" are not explicitly requested. Please add the following to Build-Depends: {needed_plugins}" @@ -143,22 +167,33 @@ def _requested_debputy_plugins(debian_dir: VirtualPath) -> Optional[Set[str]]: def _check_migration_target( - debian_dir: VirtualPath, + context: CommandContext, migration_target: Optional[DebputyIntegrationMode], ) -> DebputyIntegrationMode: - r = read_dh_addon_sequences(debian_dir) - if r is None and migration_target is None: - _error("debian/control is missing and no migration target was provided") - bd_sequences, dr_sequences, _ = r - all_sequences = bd_sequences | dr_sequences + r = read_dh_addon_sequences(context.debian_dir) + if r is not None: + bd_sequences, dr_sequences, _ = r + all_sequences = bd_sequences | dr_sequences + detected_migration_target = determine_debputy_integration_mode( + context.source_package().fields, + all_sequences, + ) + else: + detected_migration_target = None - detected_migration_target = determine_debputy_integration_mode(all_sequences) + if migration_target is not None and detected_migration_target is not None: + supported_migrations = SUPPORTED_MIGRATIONS.get( + detected_migration_target, + frozenset([detected_migration_target]), + ) - if ( - migration_target == INTEGRATION_MODE_DH_DEBPUTY_RRR - and detected_migration_target == INTEGRATION_MODE_DH_DEBPUTY - ): - _error("Cannot migrate from (zz-)debputy to zz-debputy-rrr") + if ( + migration_target != detected_migration_target + and migration_target not in supported_migrations + ): + _error( + f"Cannot migrate from {detected_migration_target} to {migration_target}" + ) if migration_target is not None: resolved_migration_target = migration_target @@ -179,6 +214,7 @@ def _check_migration_target( def migrate_from_dh( + fo: OutputStylingBase, manifest: HighLevelManifest, acceptable_migration_issues: AcceptableMigrationIssues, permit_destructive_changes: Optional[bool], @@ -197,7 +233,7 @@ def migrate_from_dh( try: for migrator in MIGRATORS[migration_target]: - feature_migration = FeatureMigration(migrator.__name__) + feature_migration = FeatureMigration(migrator.__name__, fo) migrator( debian_dir, manifest, @@ -263,7 +299,12 @@ def migrate_from_dh( ) _print_migration_summary( - migrations, compat, min_compat, required_plugins, requested_plugins + fo, + migrations, + compat, + min_compat, + required_plugins, + requested_plugins, ) migration_count = sum((m.performed_changes for m in migrations), 0) diff --git a/src/debputy/dh_migration/migrators.py b/src/debputy/dh_migration/migrators.py index 8eff679..3c2d6e0 100644 --- a/src/debputy/dh_migration/migrators.py +++ b/src/debputy/dh_migration/migrators.py @@ -1,5 +1,6 @@ -from typing import Callable, List, Mapping +from typing import Callable, List, Mapping, Protocol, Optional +from debputy.commands.debputy_cmd.output import OutputStylingBase from debputy.dh_migration.migrators_impl import ( migrate_links_files, migrate_maintscript, @@ -13,7 +14,7 @@ from debputy.dh_migration.migrators_impl import ( migrate_lintian_overrides_files, detect_unsupported_zz_debputy_features, detect_pam_files, - detect_dh_addons, + detect_dh_addons_with_zz_integration, migrate_not_installed_file, migrate_installman_file, migrate_bash_completion, @@ -21,6 +22,7 @@ from debputy.dh_migration.migrators_impl import ( migrate_dh_installsystemd_files, detect_obsolete_substvars, detect_dh_addons_zz_debputy_rrr, + detect_dh_addons_with_full_integration, ) from debputy.dh_migration.models import AcceptableMigrationIssues, FeatureMigration from debputy.highlevel_manifest import HighLevelManifest @@ -29,13 +31,42 @@ from debputy.plugin.api.spec import ( DebputyIntegrationMode, INTEGRATION_MODE_DH_DEBPUTY_RRR, INTEGRATION_MODE_DH_DEBPUTY, + INTEGRATION_MODE_FULL, ) Migrator = Callable[ - [VirtualPath, HighLevelManifest, AcceptableMigrationIssues, FeatureMigration, str], + [ + VirtualPath, + HighLevelManifest, + AcceptableMigrationIssues, + FeatureMigration, + DebputyIntegrationMode, + ], None, ] +_DH_DEBPUTY_MIGRATORS = [ + detect_unsupported_zz_debputy_features, + detect_pam_files, + migrate_dh_hook_targets, + migrate_dh_installsystemd_files, + migrate_install_file, + migrate_installdocs_file, + migrate_installexamples_file, + migrate_installman_file, + migrate_installinfo_file, + migrate_misspelled_readme_debian_files, + migrate_doc_base_files, + migrate_links_files, + migrate_maintscript, + migrate_tmpfile, + migrate_lintian_overrides_files, + migrate_bash_completion, + detect_obsolete_substvars, + # not-installed should go last, so its rules appear after other installations + # It is not perfect, but it is a start. + migrate_not_installed_file, +] MIGRATORS: Mapping[DebputyIntegrationMode, List[Migrator]] = { INTEGRATION_MODE_DH_DEBPUTY_RRR: [ @@ -45,26 +76,12 @@ MIGRATORS: Mapping[DebputyIntegrationMode, List[Migrator]] = { detect_obsolete_substvars, ], INTEGRATION_MODE_DH_DEBPUTY: [ - detect_unsupported_zz_debputy_features, - detect_pam_files, - migrate_dh_hook_targets, - migrate_dh_installsystemd_files, - migrate_install_file, - migrate_installdocs_file, - migrate_installexamples_file, - migrate_installman_file, - migrate_installinfo_file, - migrate_misspelled_readme_debian_files, - migrate_doc_base_files, - migrate_links_files, - migrate_maintscript, - migrate_tmpfile, - migrate_lintian_overrides_files, - migrate_bash_completion, - detect_dh_addons, - detect_obsolete_substvars, - # not-installed should go last, so its rules appear after other installations - # It is not perfect, but it is a start. - migrate_not_installed_file, + *_DH_DEBPUTY_MIGRATORS, + detect_dh_addons_with_zz_integration, + ], + INTEGRATION_MODE_FULL: [ + *_DH_DEBPUTY_MIGRATORS, + detect_dh_addons_with_full_integration, ], } +del _DH_DEBPUTY_MIGRATORS diff --git a/src/debputy/dh_migration/migrators_impl.py b/src/debputy/dh_migration/migrators_impl.py index 91ea8cd..ca44b68 100644 --- a/src/debputy/dh_migration/migrators_impl.py +++ b/src/debputy/dh_migration/migrators_impl.py @@ -17,12 +17,14 @@ from typing import ( Callable, TypeVar, Dict, + Container, ) from debian.deb822 import Deb822 from debputy import DEBPUTY_DOC_ROOT_DIR from debputy.architecture_support import dpkg_architecture_table +from debputy.commands.debputy_cmd.output import OutputStylingBase from debputy.deb_packaging_support import dpkg_field_list_pkg_dep from debputy.dh.debhelper_emulation import ( dhe_filedoublearray, @@ -51,6 +53,8 @@ from debputy.plugin.api import VirtualPath from debputy.plugin.api.spec import ( INTEGRATION_MODE_DH_DEBPUTY_RRR, INTEGRATION_MODE_DH_DEBPUTY, + DebputyIntegrationMode, + INTEGRATION_MODE_FULL, ) from debputy.util import ( _error, @@ -61,8 +65,15 @@ from debputy.util import ( has_glob_magic, ) + +class ContainsEverything: + + def __contains__(self, item: str) -> bool: + return True + + # Align with debputy.py -DH_COMMANDS_REPLACED = { +DH_COMMANDS_REPLACED: Mapping[DebputyIntegrationMode, Container[str]] = { INTEGRATION_MODE_DH_DEBPUTY_RRR: frozenset( { "dh_fixperms", @@ -124,6 +135,7 @@ DH_COMMANDS_REPLACED = { "dh_builddeb", } ), + INTEGRATION_MODE_FULL: ContainsEverything(), } _GS_DOC = f"{DEBPUTY_DOC_ROOT_DIR}/GETTING-STARTED-WITH-dh-debputy.md" @@ -375,7 +387,7 @@ def migrate_bash_completion( manifest: HighLevelManifest, acceptable_migration_issues: AcceptableMigrationIssues, feature_migration: FeatureMigration, - _migration_target: str, + _migration_target: DebputyIntegrationMode, ) -> None: feature_migration.tagline = "dh_bash-completion files" is_single_binary = sum(1 for _ in manifest.all_packages) == 1 @@ -466,7 +478,7 @@ def migrate_dh_installsystemd_files( manifest: HighLevelManifest, _acceptable_migration_issues: AcceptableMigrationIssues, feature_migration: FeatureMigration, - _migration_target: str, + _migration_target: DebputyIntegrationMode, ) -> None: feature_migration.tagline = "dh_installsystemd files" for dctrl_bin in manifest.all_packages: @@ -499,7 +511,7 @@ def migrate_maintscript( manifest: HighLevelManifest, acceptable_migration_issues: AcceptableMigrationIssues, feature_migration: FeatureMigration, - _migration_target: str, + _migration_target: DebputyIntegrationMode, ) -> None: feature_migration.tagline = "dh_installdeb files" mutable_manifest = assume_not_none(manifest.mutable_manifest) @@ -608,7 +620,7 @@ def migrate_install_file( manifest: HighLevelManifest, acceptable_migration_issues: AcceptableMigrationIssues, feature_migration: FeatureMigration, - _migration_target: str, + _migration_target: DebputyIntegrationMode, ) -> None: feature_migration.tagline = "dh_install config files" mutable_manifest = assume_not_none(manifest.mutable_manifest) @@ -799,7 +811,7 @@ def migrate_installdocs_file( manifest: HighLevelManifest, acceptable_migration_issues: AcceptableMigrationIssues, feature_migration: FeatureMigration, - _migration_target: str, + _migration_target: DebputyIntegrationMode, ) -> None: feature_migration.tagline = "dh_installdocs config files" mutable_manifest = assume_not_none(manifest.mutable_manifest) @@ -846,7 +858,7 @@ def migrate_installexamples_file( manifest: HighLevelManifest, acceptable_migration_issues: AcceptableMigrationIssues, feature_migration: FeatureMigration, - _migration_target: str, + _migration_target: DebputyIntegrationMode, ) -> None: feature_migration.tagline = "dh_installexamples config files" mutable_manifest = assume_not_none(manifest.mutable_manifest) @@ -900,7 +912,7 @@ def migrate_installinfo_file( manifest: HighLevelManifest, acceptable_migration_issues: AcceptableMigrationIssues, feature_migration: FeatureMigration, - _migration_target: str, + _migration_target: DebputyIntegrationMode, ) -> None: feature_migration.tagline = "dh_installinfo config files" mutable_manifest = assume_not_none(manifest.mutable_manifest) @@ -975,7 +987,7 @@ def migrate_installman_file( manifest: HighLevelManifest, acceptable_migration_issues: AcceptableMigrationIssues, feature_migration: FeatureMigration, - _migration_target: str, + _migration_target: DebputyIntegrationMode, ) -> None: feature_migration.tagline = "dh_installman config files" mutable_manifest = assume_not_none(manifest.mutable_manifest) @@ -1095,7 +1107,7 @@ def migrate_not_installed_file( manifest: HighLevelManifest, acceptable_migration_issues: AcceptableMigrationIssues, feature_migration: FeatureMigration, - _migration_target: str, + _migration_target: DebputyIntegrationMode, ) -> None: feature_migration.tagline = "dh_missing's not-installed config file" mutable_manifest = assume_not_none(manifest.mutable_manifest) @@ -1135,7 +1147,7 @@ def detect_pam_files( manifest: HighLevelManifest, _acceptable_migration_issues: AcceptableMigrationIssues, feature_migration: FeatureMigration, - _migration_target: str, + _migration_target: DebputyIntegrationMode, ) -> None: feature_migration.tagline = "detect dh_installpam files (min dh compat)" for dctrl_bin in manifest.all_packages: @@ -1150,7 +1162,7 @@ def migrate_tmpfile( manifest: HighLevelManifest, _acceptable_migration_issues: AcceptableMigrationIssues, feature_migration: FeatureMigration, - _migration_target: str, + _migration_target: DebputyIntegrationMode, ) -> None: feature_migration.tagline = "dh_installtmpfiles config files" for dctrl_bin in manifest.all_packages: @@ -1174,7 +1186,7 @@ def migrate_lintian_overrides_files( manifest: HighLevelManifest, acceptable_migration_issues: AcceptableMigrationIssues, feature_migration: FeatureMigration, - _migration_target: str, + _migration_target: DebputyIntegrationMode, ) -> None: feature_migration.tagline = "dh_lintian config files" for dctrl_bin in manifest.all_packages: @@ -1198,7 +1210,7 @@ def migrate_links_files( manifest: HighLevelManifest, acceptable_migration_issues: AcceptableMigrationIssues, feature_migration: FeatureMigration, - _migration_target: str, + _migration_target: DebputyIntegrationMode, ) -> None: feature_migration.tagline = "dh_link files" mutable_manifest = assume_not_none(manifest.mutable_manifest) @@ -1272,7 +1284,7 @@ def migrate_misspelled_readme_debian_files( manifest: HighLevelManifest, acceptable_migration_issues: AcceptableMigrationIssues, feature_migration: FeatureMigration, - _migration_target: str, + _migration_target: DebputyIntegrationMode, ) -> None: feature_migration.tagline = "misspelled README.Debian files" for dctrl_bin in manifest.all_packages: @@ -1304,7 +1316,7 @@ def migrate_doc_base_files( manifest: HighLevelManifest, _: AcceptableMigrationIssues, feature_migration: FeatureMigration, - _migration_target: str, + _migration_target: DebputyIntegrationMode, ) -> None: feature_migration.tagline = "doc-base files" # ignore the dh_make ".EX" file if one should still be present. The dh_installdocs tool ignores it too. @@ -1355,7 +1367,7 @@ def migrate_dh_hook_targets( _: HighLevelManifest, acceptable_migration_issues: AcceptableMigrationIssues, feature_migration: FeatureMigration, - migration_target: str, + migration_target: DebputyIntegrationMode, ) -> None: feature_migration.tagline = "dh hook targets" source_root = os.path.dirname(debian_dir.fs_path) @@ -1407,7 +1419,7 @@ def detect_unsupported_zz_debputy_features( manifest: HighLevelManifest, acceptable_migration_issues: AcceptableMigrationIssues, feature_migration: FeatureMigration, - _migration_target: str, + _migration_target: DebputyIntegrationMode, ) -> None: feature_migration.tagline = "Known unsupported features" @@ -1426,7 +1438,7 @@ def detect_obsolete_substvars( _manifest: HighLevelManifest, _acceptable_migration_issues: AcceptableMigrationIssues, feature_migration: FeatureMigration, - _migration_target: str, + _migration_target: DebputyIntegrationMode, ) -> None: feature_migration.tagline = ( "Check for obsolete ${foo:var} variables in debian/control" @@ -1489,16 +1501,18 @@ def detect_obsolete_substvars( seen_obsolete_relationship_substvars.update(obsolete_substvars_in_field) package = p.get("Package", "(Missing package name!?)") + fo = feature_migration.fo if obsolete_fields: fields = ", ".join(obsolete_fields) feature_migration.warn( f"The following relationship fields can be removed from {package}: {fields}." - f" (The content in them would be applied automatically.)" + f" (The content in them would be applied automatically. Note: {fo.bts('1067653')})" ) if seen_obsolete_relationship_substvars: v = ", ".join(sorted(seen_obsolete_relationship_substvars)) feature_migration.warn( f"The following relationship substitution variables can be removed from {package}: {v}" + f" (Note: {fo.bts('1067653')})" ) @@ -1507,7 +1521,7 @@ def detect_dh_addons_zz_debputy_rrr( _manifest: HighLevelManifest, _acceptable_migration_issues: AcceptableMigrationIssues, feature_migration: FeatureMigration, - _migration_target: str, + _migration_target: DebputyIntegrationMode, ) -> None: feature_migration.tagline = "Check for dh-sequence-addons" r = read_dh_addon_sequences(debian_dir) @@ -1527,12 +1541,31 @@ def detect_dh_addons_zz_debputy_rrr( feature_migration.warn("Missing Build-Depends on dh-sequence-zz-debputy-rrr") -def detect_dh_addons( +def detect_dh_addons_with_full_integration( + _debian_dir: VirtualPath, + _manifest: HighLevelManifest, + _acceptable_migration_issues: AcceptableMigrationIssues, + feature_migration: FeatureMigration, + _migration_target: DebputyIntegrationMode, +) -> None: + feature_migration.tagline = "Check for dh-sequence-addons and Build-Depends" + feature_migration.warn( + "TODO: Not implemented: Please remove any dh-sequence Build-Dependency" + ) + feature_migration.warn( + "TODO: Not implemented: Please ensure there is a Build-Dependency on `debputy (>= 0.1.45~)" + ) + feature_migration.warn( + "TODO: Not implemented: Please ensure there is a Build-Dependency on `dpkg-dev (>= 1.22.7~)" + ) + + +def detect_dh_addons_with_zz_integration( debian_dir: VirtualPath, _manifest: HighLevelManifest, acceptable_migration_issues: AcceptableMigrationIssues, feature_migration: FeatureMigration, - _migration_target: str, + _migration_target: DebputyIntegrationMode, ) -> None: feature_migration.tagline = "Check for dh-sequence-addons" r = read_dh_addon_sequences(debian_dir) @@ -1544,6 +1577,8 @@ def detect_dh_addons( ) return + assert _migration_target != INTEGRATION_MODE_FULL + bd_sequences, dr_sequences, _ = r remaining_sequences = bd_sequences | dr_sequences diff --git a/src/debputy/dh_migration/models.py b/src/debputy/dh_migration/models.py index ace4185..c093633 100644 --- a/src/debputy/dh_migration/models.py +++ b/src/debputy/dh_migration/models.py @@ -3,6 +3,7 @@ import re from typing import Sequence, Optional, FrozenSet, Tuple, List, cast from debputy.architecture_support import DpkgArchitectureBuildProcessValuesTable +from debputy.commands.debputy_cmd.output import OutputStylingBase from debputy.highlevel_manifest import MutableYAMLManifest from debputy.substitution import Substitution @@ -38,6 +39,7 @@ class ConflictingChange(RuntimeError): @dataclasses.dataclass(slots=True) class FeatureMigration: tagline: str + fo: OutputStylingBase successful_manifest_changes: int = 0 already_present: int = 0 warnings: List[str] = dataclasses.field(default_factory=list) diff --git a/src/debputy/exceptions.py b/src/debputy/exceptions.py index a445997..b3ff7d5 100644 --- a/src/debputy/exceptions.py +++ b/src/debputy/exceptions.py @@ -10,6 +10,10 @@ class DebputyRuntimeError(RuntimeError): return cast("str", self.args[0]) +class DebputyBuildStepError(DebputyRuntimeError): + pass + + class DebputySubstitutionError(DebputyRuntimeError): pass @@ -64,6 +68,10 @@ class PluginInitializationError(PluginBaseError): pass +class PluginIncorrectRegistrationError(PluginInitializationError): + pass + + class PluginMetadataError(PluginBaseError): pass diff --git a/src/debputy/filesystem_scan.py b/src/debputy/filesystem_scan.py index 7b20040..29d37bd 100644 --- a/src/debputy/filesystem_scan.py +++ b/src/debputy/filesystem_scan.py @@ -57,6 +57,7 @@ from debputy.util import ( escape_shell, assume_not_none, _normalize_path, + _debug_log, ) BY_BASENAME = operator.attrgetter("name") @@ -668,7 +669,7 @@ class FSPath(VirtualPathBase, ABC): @mode.setter def mode(self, new_mode: int) -> None: self._rw_check() - min_bit = 0o500 if self.is_dir else 0o400 + min_bit = 0o700 if self.is_dir else 0o400 if (new_mode & min_bit) != min_bit: omode = oct(new_mode)[2:] omin = oct(min_bit)[2:] @@ -680,6 +681,22 @@ class FSPath(VirtualPathBase, ABC): ) self._mode = new_mode + def _ensure_min_mode(self) -> None: + min_bit = 0o700 if self.is_dir else 0o600 + if self.has_fs_path and (self.mode & 0o600) != 0o600: + try: + fs_path = self.fs_path + except TestPathWithNonExistentFSPathError: + pass + else: + st = os.stat(fs_path) + new_fs_mode = stat.S_IMODE(st.st_mode) | min_bit + _debug_log( + f"Applying chmod {oct(min_bit)[2:]} {fs_path} ({self.path}) to avoid problems down the line" + ) + os.chmod(fs_path, new_fs_mode) + self.mode |= min_bit + @property def mtime(self) -> float: mtime = self._mtime @@ -1243,6 +1260,7 @@ class VirtualDirectoryFSPath(VirtualPathWithReference): ) self._reference_path = reference_path assert reference_path is None or reference_path.is_dir + self._ensure_min_mode() @property def is_dir(self) -> bool: @@ -1326,6 +1344,7 @@ class FSBackedFilePath(VirtualPathWithReference): assert ( not replaceable_inline or "debputy/scratch-dir/" in fs_path ), f"{fs_path} should not be inline-replaceable -- {self.path}" + self._ensure_min_mode() @property def is_dir(self) -> bool: diff --git a/src/debputy/highlevel_manifest.py b/src/debputy/highlevel_manifest.py index 9bdc225..ef47d45 100644 --- a/src/debputy/highlevel_manifest.py +++ b/src/debputy/highlevel_manifest.py @@ -49,7 +49,11 @@ from .maintscript_snippet import ( MaintscriptSnippetContainer, ) from .manifest_conditions import ConditionContext -from .manifest_parser.base_types import FileSystemMatchRule, FileSystemExactMatchRule +from .manifest_parser.base_types import ( + FileSystemMatchRule, + FileSystemExactMatchRule, + BuildEnvironments, +) from .manifest_parser.util import AttributePath from .packager_provided_files import PackagerProvidedFile from .packages import BinaryPackage, SourcePackage @@ -59,8 +63,15 @@ from .plugin.api.impl_types import ( PackageProcessingContextProvider, PackageDataTable, ) -from .plugin.api.spec import FlushableSubstvars, VirtualPath +from .plugin.api.spec import ( + FlushableSubstvars, + VirtualPath, + DebputyIntegrationMode, + INTEGRATION_MODE_DH_DEBPUTY_RRR, +) from .plugin.debputy.binary_package_rules import ServiceRule +from .plugin.debputy.to_be_api_types import BuildRule +from .plugin.plugin_state import run_in_context_of_plugin from .substitution import Substitution from .transformation_rules import ( TransformationRule, @@ -1036,7 +1047,9 @@ def _install_everything_from_source_dir_if_present( ) -> None: attribute_path = AttributePath.builtin_path()[f"installing {source_dir.fs_path}"] pkg_set = frozenset([dctrl_bin]) - install_rule = InstallRule.install_dest( + install_rule = run_in_context_of_plugin( + "debputy", + InstallRule.install_dest, [FileSystemMatchRule.from_path_match("*", attribute_path, substitution)], None, pkg_set, @@ -1086,6 +1099,8 @@ class HighLevelManifest: dpkg_architecture_variables: DpkgArchitectureBuildProcessValuesTable, dpkg_arch_query_table: DpkgArchTable, build_env: DebBuildOptionsAndProfiles, + build_environments: BuildEnvironments, + build_rules: Optional[List[BuildRule]], plugin_provided_feature_set: PluginProvidedFeatureSet, debian_dir: VirtualPath, ) -> None: @@ -1100,8 +1115,17 @@ class HighLevelManifest: self._dpkg_arch_query_table = dpkg_arch_query_table self._build_env = build_env self._used_for: Set[str] = set() + self.build_environments = build_environments + self.build_rules = build_rules self._plugin_provided_feature_set = plugin_provided_feature_set self._debian_dir = debian_dir + self._source_condition_context = ConditionContext( + binary_package=None, + substitution=self.substitution, + deb_options_and_profiles=self._build_env, + dpkg_architecture_variables=self._dpkg_architecture_variables, + dpkg_arch_query_table=self._dpkg_arch_query_table, + ) def source_version(self, include_binnmu_version: bool = True) -> str: # TODO: There should an easier way to determine the source version; really. @@ -1116,6 +1140,10 @@ class HighLevelManifest: raise AssertionError(f"Could not resolve {version_var}") from e @property + def source_condition_context(self) -> ConditionContext: + return self._source_condition_context + + @property def debian_dir(self) -> VirtualPath: return self._debian_dir @@ -1124,7 +1152,7 @@ class HighLevelManifest: return self._dpkg_architecture_variables @property - def build_env(self) -> DebBuildOptionsAndProfiles: + def deb_options_and_profiles(self) -> DebBuildOptionsAndProfiles: return self._build_env @property @@ -1162,12 +1190,15 @@ class HighLevelManifest: def perform_installations( self, + integration_mode: DebputyIntegrationMode, *, install_request_context: Optional[InstallSearchDirContext] = None, - enable_manifest_installation_feature: bool = True, ) -> PackageDataTable: package_data_dict = {} package_data_table = PackageDataTable(package_data_dict) + enable_manifest_installation_feature = ( + integration_mode != INTEGRATION_MODE_DH_DEBPUTY_RRR + ) if install_request_context is None: @functools.lru_cache(None) @@ -1178,6 +1209,7 @@ class HighLevelManifest: source_root_dir = _as_path(".") into = frozenset(self._binary_packages.values()) default_search_dirs = [dtmp_dir] + # TODO: In integration-mode full use build systems to define the per_package_search_dirs per_package_search_dirs = { t.binary_package: [_as_path(f.match_rule.path) for f in t.search_dirs] for t in self.package_transformations.values() @@ -1270,13 +1302,7 @@ class HighLevelManifest: ] path_matcher = SourcePathMatcher(discard_rules) - source_condition_context = ConditionContext( - binary_package=None, - substitution=self.substitution, - build_env=self._build_env, - dpkg_architecture_variables=self._dpkg_architecture_variables, - dpkg_arch_query_table=self._dpkg_arch_query_table, - ) + source_condition_context = self._source_condition_context for dctrl_bin in self.active_packages: package = dctrl_bin.name @@ -1416,23 +1442,14 @@ class HighLevelManifest: self, binary_package: Optional[Union[BinaryPackage, str]] ) -> ConditionContext: if binary_package is None: - return ConditionContext( - binary_package=None, - substitution=self.substitution, - build_env=self._build_env, - dpkg_architecture_variables=self._dpkg_architecture_variables, - dpkg_arch_query_table=self._dpkg_arch_query_table, - ) + return self._source_condition_context if not isinstance(binary_package, str): binary_package = binary_package.name package_transformation = self.package_transformations[binary_package] - return ConditionContext( + return self._source_condition_context.replace( binary_package=package_transformation.binary_package, substitution=package_transformation.substitution, - build_env=self._build_env, - dpkg_architecture_variables=self._dpkg_architecture_variables, - dpkg_arch_query_table=self._dpkg_arch_query_table, ) def apply_fs_transformations( @@ -1452,7 +1469,7 @@ class HighLevelManifest: condition_context = ConditionContext( binary_package=package_transformation.binary_package, substitution=package_transformation.substitution, - build_env=self._build_env, + deb_options_and_profiles=self._build_env, dpkg_architecture_variables=self._dpkg_architecture_variables, dpkg_arch_query_table=self._dpkg_arch_query_table, ) @@ -1466,7 +1483,7 @@ class HighLevelManifest: norm_mode_transformation_rule = ModeNormalizationTransformationRule(norm_rules) norm_mode_transformation_rule.transform_file_system(fs_root, condition_context) for transformation in package_transformation.transformations: - transformation.transform_file_system(fs_root, condition_context) + transformation.run_transform_file_system(fs_root, condition_context) interpreter_normalization = NormalizeShebangLineTransformation() interpreter_normalization.transform_file_system(fs_root, condition_context) diff --git a/src/debputy/highlevel_manifest_parser.py b/src/debputy/highlevel_manifest_parser.py index dd97d58..18d9fa7 100644 --- a/src/debputy/highlevel_manifest_parser.py +++ b/src/debputy/highlevel_manifest_parser.py @@ -43,20 +43,22 @@ from ._deb_options_profiles import DebBuildOptionsAndProfiles from .architecture_support import DpkgArchitectureBuildProcessValuesTable from .filesystem_scan import FSROOverlay from .installations import InstallRule, PPFInstallRule +from .manifest_parser.base_types import BuildEnvironments, BuildEnvironmentDefinition from .manifest_parser.exceptions import ManifestParseException from .manifest_parser.parser_data import ParserContextData from .manifest_parser.util import AttributePath from .packager_provided_files import detect_all_packager_provided_files from .plugin.api import VirtualPath +from .plugin.api.feature_set import PluginProvidedFeatureSet from .plugin.api.impl_types import ( TP, TTP, DispatchingTableParser, - OPARSER_MANIFEST_ROOT, PackageContextData, ) -from .plugin.api.feature_set import PluginProvidedFeatureSet +from debputy.plugin.api.parser_tables import OPARSER_MANIFEST_ROOT from .plugin.api.spec import DebputyIntegrationMode +from .plugin.debputy.to_be_api_types import BuildRule from .yaml import YAMLError, MANIFEST_YAML try: @@ -131,11 +133,19 @@ class HighLevelManifestParser(ParserContextData): self._substitution = substitution self._dpkg_architecture_variables = dpkg_architecture_variables self._dpkg_arch_query_table = dpkg_arch_query_table - self._build_env = build_env + self._deb_options_and_profiles = build_env self._package_state_stack: List[PackageTransformationDefinition] = [] self._plugin_provided_feature_set = plugin_provided_feature_set self._debputy_integration_mode = debputy_integration_mode self._declared_variables = {} + self._used_named_envs = set() + self._build_environments: Optional[BuildEnvironments] = BuildEnvironments( + {}, + None, + ) + self._has_set_default_build_environment = False + self._read_build_environment = False + self._build_rules: Optional[List[BuildRule]] = None if isinstance(debian_dir, str): debian_dir = FSROOverlay.create_root_dir("debian", debian_dir) @@ -202,10 +212,21 @@ class HighLevelManifestParser(ParserContextData): return self._dpkg_arch_query_table @property - def build_env(self) -> DebBuildOptionsAndProfiles: - return self._build_env + def deb_options_and_profiles(self) -> DebBuildOptionsAndProfiles: + return self._deb_options_and_profiles + + def _self_check(self) -> None: + unused_envs = ( + self._build_environments.environments.keys() - self._used_named_envs + ) + if unused_envs: + unused_env_names = ", ".join(unused_envs) + raise ManifestParseException( + f"The following named environments were never referenced: {unused_env_names}" + ) def build_manifest(self) -> HighLevelManifest: + self._self_check() if self._used: raise TypeError("build_manifest can only be called once!") self._used = True @@ -214,7 +235,7 @@ class HighLevelManifestParser(ParserContextData): if not self.substitution.is_used(var): raise ManifestParseException( f'The variable "{var}" is unused. Either use it or remove it.' - f" The variable was declared at {attribute_path.path}." + f" The variable was declared at {attribute_path.path_key_lc}." ) if isinstance(self, YAMLManifestParser) and self._mutable_yaml_manifest is None: self._mutable_yaml_manifest = MutableYAMLManifest.empty_manifest() @@ -240,6 +261,8 @@ class HighLevelManifestParser(ParserContextData): ppf_result.reserved_only ) self._transform_dpkg_maintscript_helpers_to_snippets() + build_environments = self.build_environments() + assert build_environments is not None return HighLevelManifest( self.manifest_path, @@ -251,7 +274,9 @@ class HighLevelManifestParser(ParserContextData): self._package_states, self._dpkg_architecture_variables, self._dpkg_arch_query_table, - self._build_env, + self._deb_options_and_profiles, + build_environments, + self._build_rules, self._plugin_provided_feature_set, self._debian_dir, ) @@ -325,6 +350,69 @@ class HighLevelManifestParser(ParserContextData): def debputy_integration_mode(self, new_value: DebputyIntegrationMode) -> None: self._debputy_integration_mode = new_value + def _register_build_environment( + self, + name: Optional[str], + build_environment: BuildEnvironmentDefinition, + attribute_path: AttributePath, + is_default: bool = False, + ) -> None: + assert not self._read_build_environment + + # TODO: Reference the paths of the original environments for the error messages where that is relevant. + if is_default: + if self._has_set_default_build_environment: + raise ManifestParseException( + f"There cannot be multiple default environments and" + f" therefore {attribute_path.path} cannot be a default environment" + ) + self._has_set_default_build_environment = True + self._build_environments.default_environment = build_environment + if name is None: + return + elif name is None: + raise ManifestParseException( + f"Useless environment defined at {attribute_path.path}. It is neither the" + " default environment nor does it have a name (so no rules can reference it" + " explicitly)" + ) + + if name in self._build_environments.environments: + raise ManifestParseException( + f'The environment defined at {attribute_path.path} reuse the name "{name}".' + " The environment name must be unique." + ) + self._build_environments.environments[name] = build_environment + + def resolve_build_environment( + self, + name: Optional[str], + attribute_path: AttributePath, + ) -> BuildEnvironmentDefinition: + if name is None: + return self.build_environments().default_environment + try: + env = self.build_environments().environments[name] + except KeyError: + raise ManifestParseException( + f'The environment "{name}" requested at {attribute_path.path} was not' + f" defined in the `build-environments`" + ) + else: + self._used_named_envs.add(name) + return env + + def build_environments(self) -> BuildEnvironments: + v = self._build_environments + if ( + not self._read_build_environment + and not self._build_environments.environments + and self._build_environments.default_environment is None + ): + self._build_environments.default_environment = BuildEnvironmentDefinition() + self._read_build_environment = True + return v + def _transform_dpkg_maintscript_helpers_to_snippets(self) -> None: package_state = self.current_binary_package_state for dmh in package_state.dpkg_maintscript_helper_snippets: @@ -451,7 +539,7 @@ class YAMLManifestParser(HighLevelManifestParser): return v def from_yaml_dict(self, yaml_data: object) -> "HighLevelManifest": - attribute_path = AttributePath.root_path() + attribute_path = AttributePath.root_path(yaml_data) parser_generator = self._plugin_provided_feature_set.manifest_parser_generator dispatchable_object_parsers = parser_generator.dispatchable_object_parsers manifest_root_parser = dispatchable_object_parsers[OPARSER_MANIFEST_ROOT] @@ -504,6 +592,7 @@ class YAMLManifestParser(HighLevelManifestParser): ) if service_rules: package_state.requested_service_rules.extend(service_rules) + self._build_rules = parsed_data.get("builds") return self.build_manifest() diff --git a/src/debputy/installations.py b/src/debputy/installations.py index b781757..1ed89ca 100644 --- a/src/debputy/installations.py +++ b/src/debputy/installations.py @@ -31,10 +31,11 @@ from debputy.manifest_conditions import ( from debputy.manifest_parser.base_types import ( FileSystemMatchRule, FileSystemExactMatchRule, - DebputyDispatchableType, ) +from debputy.manifest_parser.tagging_types import DebputyDispatchableType from debputy.packages import BinaryPackage from debputy.path_matcher import MatchRule, ExactFileSystemPath, MATCH_ANYTHING +from debputy.plugin.plugin_state import run_in_context_of_plugin from debputy.substitution import Substitution from debputy.util import _error, _warn @@ -585,6 +586,7 @@ class InstallRule(DebputyDispatchableType): *, match_filter: Optional[Callable[["VirtualPath"], bool]] = None, ) -> None: + super().__init__() self._condition = condition self._definition_source = definition_source self._match_filter = match_filter @@ -1019,13 +1021,16 @@ class PPFInstallRule(InstallRule): "_into", ) + # noinspection PyMissingConstructor def __init__( self, into: BinaryPackage, substitution: Substitution, ppfs: Sequence["PackagerProvidedFile"], ) -> None: - super().__init__( + run_in_context_of_plugin( + "debputy", + super().__init__, None, "<built-in; PPF install rule>", ) @@ -1142,8 +1147,8 @@ class DiscardRule(InstallRule): if s.search_dir.fs_path in matches ) if len(limit_to) != len(search_dirs): - matches.difference(s.search_dir.fs_path for s in search_dirs) - paths = ":".join(matches) + m = matches.difference(s.search_dir.fs_path for s in search_dirs) + paths = ":".join(m) _error( f"The discard rule defined at {self._definition_source} mentions the following" f" search directories that were not known to debputy: {paths}." diff --git a/src/debputy/integration_detection.py b/src/debputy/integration_detection.py index f412268..cc19057 100644 --- a/src/debputy/integration_detection.py +++ b/src/debputy/integration_detection.py @@ -1,16 +1,21 @@ -from typing import Container, Optional +from typing import Container, Optional, Mapping from debputy.plugin.api.spec import ( DebputyIntegrationMode, INTEGRATION_MODE_DH_DEBPUTY_RRR, INTEGRATION_MODE_DH_DEBPUTY, + INTEGRATION_MODE_FULL, ) def determine_debputy_integration_mode( + source_fields: Mapping[str, str], all_sequences: Container[str], ) -> Optional[DebputyIntegrationMode]: + if source_fields.get("Build-Driver", "").lower() == "debputy": + return INTEGRATION_MODE_FULL + has_zz_debputy = "zz-debputy" in all_sequences or "debputy" in all_sequences has_zz_debputy_rrr = "zz-debputy-rrr" in all_sequences has_any_existing = has_zz_debputy or has_zz_debputy_rrr @@ -18,4 +23,7 @@ def determine_debputy_integration_mode( return INTEGRATION_MODE_DH_DEBPUTY_RRR if has_any_existing: return INTEGRATION_MODE_DH_DEBPUTY + if source_fields.get("Source", "") == "debputy": + # Self-hosting. We cannot set the Build-Driver field since that creates a self-circular dependency loop + return INTEGRATION_MODE_FULL return None diff --git a/src/debputy/linting/lint_impl.py b/src/debputy/linting/lint_impl.py index b424460..0f37dce 100644 --- a/src/debputy/linting/lint_impl.py +++ b/src/debputy/linting/lint_impl.py @@ -6,18 +6,6 @@ import sys import textwrap from typing import Optional, List, Union, NoReturn, Mapping -from debputy.lsprotocol.types import ( - CodeAction, - Command, - CodeActionParams, - CodeActionContext, - TextDocumentIdentifier, - TextEdit, - Position, - DiagnosticSeverity, - Diagnostic, -) - from debputy.commands.debputy_cmd.context import CommandContext from debputy.commands.debputy_cmd.output import _output_styling, OutputStylingBase from debputy.filesystem_scan import FSROOverlay @@ -45,13 +33,13 @@ from debputy.lsp.lsp_debian_tests_control import ( _lint_debian_tests_control, _reformat_debian_tests_control, ) +from debputy.lsp.maint_prefs import ( + MaintainerPreferenceTable, + EffectiveFormattingPreference, + determine_effective_preference, +) from debputy.lsp.quickfixes import provide_standard_quickfixes_from_diagnostics from debputy.lsp.spellchecking import disable_spellchecking -from debputy.lsp.style_prefs import ( - StylePreferenceTable, - EffectivePreference, - determine_effective_style, -) from debputy.lsp.text_edit import ( get_well_formatted_edit, merge_sort_text_edits, @@ -59,6 +47,17 @@ from debputy.lsp.text_edit import ( OverLappingTextEditException, ) from debputy.lsp.vendoring._deb822_repro import Deb822FileElement +from debputy.lsprotocol.types import ( + CodeAction, + Command, + CodeActionParams, + CodeActionContext, + TextDocumentIdentifier, + TextEdit, + Position, + DiagnosticSeverity, + Diagnostic, +) from debputy.packages import SourcePackage, BinaryPackage from debputy.plugin.api import VirtualPath from debputy.plugin.api.feature_set import PluginProvidedFeatureSet @@ -87,20 +86,21 @@ REFORMAT_FORMATS = { @dataclasses.dataclass(slots=True) class LintContext: plugin_feature_set: PluginProvidedFeatureSet - style_preference_table: StylePreferenceTable + maint_preference_table: MaintainerPreferenceTable source_root: Optional[VirtualPath] debian_dir: Optional[VirtualPath] parsed_deb822_file_content: Optional[Deb822FileElement] = None source_package: Optional[SourcePackage] = None binary_packages: Optional[Mapping[str, BinaryPackage]] = None - effective_preference: Optional[EffectivePreference] = None + effective_preference: Optional[EffectiveFormattingPreference] = None + style_tool: Optional[str] = None unsupported_preference_reason: Optional[str] = None salsa_ci: Optional[CommentedMap] = None def state_for(self, path: str, content: str, lines: List[str]) -> LintStateImpl: return LintStateImpl( self.plugin_feature_set, - self.style_preference_table, + self.maint_preference_table, self.source_root, self.debian_dir, path, @@ -119,7 +119,7 @@ def gather_lint_info(context: CommandContext) -> LintContext: debian_dir = None lint_context = LintContext( context.load_plugins(), - StylePreferenceTable.load_styles(), + MaintainerPreferenceTable.load_preferences(), source_root, debian_dir, ) @@ -147,12 +147,13 @@ def gather_lint_info(context: CommandContext) -> LintContext: except YAMLError: break if source_package is not None or salsa_ci_map is not None: - pref, pref_reason = determine_effective_style( - lint_context.style_preference_table, + pref, tool, pref_reason = determine_effective_preference( + lint_context.maint_preference_table, source_package, salsa_ci_map, ) lint_context.effective_preference = pref + lint_context.style_tool = tool lint_context.unsupported_preference_reason = pref_reason return lint_context @@ -232,14 +233,12 @@ def perform_reformat( named_style: Optional[str] = None, ) -> None: parsed_args = context.parsed_args - if not parsed_args.spellcheck: - disable_spellchecking() fo = _output_styling(context.parsed_args, sys.stdout) lint_context = gather_lint_info(context) if named_style is not None: - style = lint_context.style_preference_table.named_styles.get(named_style) + style = lint_context.maint_preference_table.named_styles.get(named_style) if style is None: - styles = ", ".join(lint_context.style_preference_table.named_styles) + styles = ", ".join(lint_context.maint_preference_table.named_styles) _error(f'There is no style named "{style}". Options include: {styles}') if ( lint_context.effective_preference is not None @@ -257,10 +256,15 @@ def perform_reformat( "While `debputy` could identify a formatting for this package, it does not support it." ) _warn(f"{lint_context.unsupported_preference_reason}") + if lint_context.style_tool is not None: + _info( + f"The following tool might be able to apply the style: {lint_context.style_tool}" + ) if parsed_args.supported_style_required: _error( "Sorry; `debputy` does not support the style. Use --unknown-or-unsupported-style-is-ok to make" - " this a non-error." + " this a non-error (note that `debputy` will not reformat the packaging in this case; just not" + " exit with an error code)." ) else: print( @@ -276,7 +280,7 @@ def perform_reformat( https://salsa.debian.org/debian/debputy-pre-commit-hooks * If you use the Debian Salsa CI pipeline, then you can set SALSA_CI_DISABLE_WRAP_AND_SORT - to a truth value and `debputy` will pick up the configuration from there. + to a "no" or 0 and `debputy` will pick up the configuration from there. - Note: The option must be in `.gitlab-ci.yml` or `debian/salsa-ci.yml` to work. The Salsa CI pipeline will use `wrap-and-sort` while `debputy` uses its own emulation of `wrap-and-sort` (`debputy` also needs to apply the style via `debputy lsp server`). @@ -293,6 +297,11 @@ def perform_reformat( ) ) if parsed_args.supported_style_required: + if lint_context.style_tool is not None: + _error( + "Sorry, `debputy reformat` does not support the packaging style. However, the" + f" formatting is supposedly handled by: {lint_context.style_tool}" + ) _error( "Sorry; `debputy` does not know which style to use for this package. Please either set a" "style or use --unknown-or-unsupported-style-is-ok to make this a non-error" diff --git a/src/debputy/linting/lint_report_junit.py b/src/debputy/linting/lint_report_junit.py index 2ad993a..4df6879 100644 --- a/src/debputy/linting/lint_report_junit.py +++ b/src/debputy/linting/lint_report_junit.py @@ -69,9 +69,10 @@ class JunitLintReport(LintReport): range_desc = "entire file" else: range_desc = str(diagnostic.range) + code = f" [{diagnostic.code}]" if diagnostic.code else "" output = textwrap.dedent( f"""\ - {filename} [{severity}] ({range_desc}): {diagnostic.message} + {filename}{code} ({severity}) {range_desc}: {diagnostic.message} """ ) case.add_failure_info( diff --git a/src/debputy/linting/lint_util.py b/src/debputy/linting/lint_util.py index ddce7c2..feca476 100644 --- a/src/debputy/linting/lint_util.py +++ b/src/debputy/linting/lint_util.py @@ -41,9 +41,9 @@ from debputy.util import _warn if TYPE_CHECKING: from debputy.lsp.text_util import LintCapablePositionCodec - from debputy.lsp.style_prefs import ( - StylePreferenceTable, - EffectivePreference, + from debputy.lsp.maint_prefs import ( + MaintainerPreferenceTable, + EffectiveFormattingPreference, ) @@ -56,9 +56,14 @@ class DebputyMetadata: debputy_integration_mode: Optional[DebputyIntegrationMode] @classmethod - def from_data(cls, dh_sequencer_data: DhSequencerData) -> typing.Self: + def from_data( + cls, + source_fields: Mapping[str, str], + dh_sequencer_data: DhSequencerData, + ) -> typing.Self: integration_mode = determine_debputy_integration_mode( - dh_sequencer_data.sequences + source_fields, + dh_sequencer_data.sequences, ) return cls(integration_mode) @@ -110,16 +115,21 @@ class LintState: raise NotImplementedError @property - def style_preference_table(self) -> "StylePreferenceTable": + def maint_preference_table(self) -> "MaintainerPreferenceTable": raise NotImplementedError @property - def effective_preference(self) -> Optional["EffectivePreference"]: + def effective_preference(self) -> Optional["EffectiveFormattingPreference"]: raise NotImplementedError @property def debputy_metadata(self) -> DebputyMetadata: - return DebputyMetadata.from_data(self.dh_sequencer_data) + src_pkg = self.source_package + src_fields = src_pkg.fields if src_pkg else {} + return DebputyMetadata.from_data( + src_fields, + self.dh_sequencer_data, + ) @property def dh_sequencer_data(self) -> DhSequencerData: @@ -129,7 +139,7 @@ class LintState: @dataclasses.dataclass(slots=True) class LintStateImpl(LintState): plugin_feature_set: PluginProvidedFeatureSet = dataclasses.field(repr=False) - style_preference_table: "StylePreferenceTable" = dataclasses.field(repr=False) + maint_preference_table: "MaintainerPreferenceTable" = dataclasses.field(repr=False) source_root: Optional[VirtualPathBase] debian_dir: Optional[VirtualPathBase] path: str @@ -137,7 +147,7 @@ class LintStateImpl(LintState): lines: List[str] source_package: Optional[SourcePackage] = None binary_packages: Optional[Mapping[str, BinaryPackage]] = None - effective_preference: Optional["EffectivePreference"] = None + effective_preference: Optional["EffectiveFormattingPreference"] = None _parsed_cache: Optional[Deb822FileElement] = None _dh_sequencer_cache: Optional[DhSequencerData] = None @@ -421,8 +431,11 @@ class TermLintReport(LintReport): if diagnostic_result.result_state == LintDiagnosticResultState.FIXABLE: has_fixit = " [Correctable via --auto-fix]" + + code = f"[{diagnostic.code}]: " if diagnostic.code else "" + msg = f"{code}{diagnostic.message}" print( - f"{tag}: File: {filename}:{start_line+1}:{start_position}:{end_line+1}:{end_position}: {diagnostic.message}{has_fixit}", + f"{tag}: File: {filename}:{start_line+1}:{start_position}:{end_line+1}:{end_position}: {msg}{has_fixit}", ) if diagnostic_result.missing_severity: _warn( @@ -440,13 +453,9 @@ class TermLintReport(LintReport): ) return lines_to_print = _lines_to_print(diagnostic.range) - if lines_to_print == 1: - line = _highlight_range(fo, lines[start_line], start_line, diagnostic.range) - print(f" {start_line+1:{line_no_width}}: {line}") - else: - for line_no in range(start_line, end_line): - line = _highlight_range(fo, lines[line_no], line_no, diagnostic.range) - print(f" {line_no+1:{line_no_width}}: {line}") + for line_no in range(start_line, start_line + lines_to_print): + line = _highlight_range(fo, lines[line_no], line_no, diagnostic.range) + print(f" {line_no+1:{line_no_width}}: {line}") class LinterPositionCodec: diff --git a/src/debputy/lsp/apt_cache.py b/src/debputy/lsp/apt_cache.py new file mode 100644 index 0000000..45988a7 --- /dev/null +++ b/src/debputy/lsp/apt_cache.py @@ -0,0 +1,167 @@ +import asyncio +import dataclasses +import subprocess +import sys +from collections import defaultdict +from typing import Literal, Optional, Sequence, Iterable, Mapping + +from debian.deb822 import Deb822 +from debian.debian_support import Version + +AptCacheState = Literal[ + "not-loaded", + "loading", + "loaded", + "failed", + "tooling-not-available", + "empty-cache", +] + + +@dataclasses.dataclass(slots=True) +class PackageInformation: + name: str + architecture: str + version: Version + multi_arch: str + # suites: Sequence[Tuple[str, ...]] + synopsis: str + section: str + provides: Optional[str] + upstream_homepage: Optional[str] + + +@dataclasses.dataclass(slots=True, frozen=True) +class PackageLookup: + name: str + package: Optional[PackageInformation] + provided_by: Sequence[PackageInformation] + + +class AptCache: + + def __init__(self) -> None: + self._state: AptCacheState = "not-loaded" + self._load_error: Optional[str] = None + self._lookups: Mapping[str, PackageLookup] = {} + + @property + def state(self) -> AptCacheState: + return self._state + + @property + def load_error(self) -> Optional[str]: + return self._load_error + + def lookup(self, name: str) -> Optional[PackageLookup]: + return self._lookups.get(name) + + async def load(self) -> None: + if self._state in ("loading", "loaded"): + raise RuntimeError(f"Already {self._state}") + self._load_error = None + self._state = "loading" + try: + files_raw = subprocess.check_output( + [ + "apt-get", + "indextargets", + "--format", + "$(IDENTIFIER)\x1f$(FILENAME)", + ] + ).decode("utf-8") + except FileNotFoundError: + self._state = "tooling-not-available" + self._load_error = "apt-get not available in PATH" + return + except subprocess.CalledProcessError as e: + self._state = "failed" + self._load_error = f"apt-get exited with {e.returncode}" + return + packages = {} + for raw_file_line in files_raw.split("\n"): + if not raw_file_line or raw_file_line.isspace(): + continue + identifier, filename = raw_file_line.split("\x1f") + if identifier not in ("Packages",): + continue + try: + for package_info in parse_apt_file(filename): + # Let other computations happen if needed. + await asyncio.sleep(0) + existing = packages.get(package_info.name) + if existing and package_info.version < existing.version: + continue + packages[package_info.name] = package_info + except FileNotFoundError: + self._state = "tooling-not-available" + self._load_error = "/usr/lib/apt/apt-helper not available" + return + except (AttributeError, RuntimeError, IndexError) as e: + self._state = "failed" + self._load_error = str(e) + return + provides = defaultdict(list) + for package_info in packages.values(): + if not package_info.provides: + continue + # Some packages (`debhelper`) provides the same package multiple times (`debhelper-compat`). + # Normalize that into one. + deps = { + clause.split("(")[0].strip() + for clause in package_info.provides.split(",") + } + for dep in sorted(deps): + provides[dep].append(package_info) + + self._lookups = { + name: PackageLookup( + name, + packages.get(name), + tuple(provides.get(name, [])), + ) + for name in packages.keys() | provides.keys() + } + self._state = "loaded" + + +def parse_apt_file(filename: str) -> Iterable[PackageInformation]: + proc = subprocess.Popen( + ["/usr/lib/apt/apt-helper", "cat-file", filename], + stdin=subprocess.DEVNULL, + stdout=subprocess.PIPE, + ) + with proc: + for stanza in Deb822.iter_paragraphs(proc.stdout): + pkg_info = stanza_to_package_info(stanza) + if pkg_info is not None: + yield pkg_info + + +def stanza_to_package_info(stanza: Deb822) -> Optional[PackageInformation]: + try: + name = stanza["Package"] + architecture = sys.intern(stanza["Architecture"]) + version = Version(stanza["Version"]) + multi_arch = sys.intern(stanza.get("Multi-Arch", "no")) + synopsis = stanza["Description"] + section = sys.intern(stanza["Section"]) + provides = stanza.get("Provides") + homepage = stanza.get("Homepage") + except KeyError: + return None + if "\n" in synopsis: + # "Modern" Packages files do not have the full description. But in case we see a (very old one) + # have consistent behavior with the modern ones. + synopsis = synopsis.split("\n")[0] + + return PackageInformation( + name, + architecture, + version, + multi_arch, + synopsis, + section, + provides, + homepage, + ) diff --git a/src/debputy/lsp/debputy_ls.py b/src/debputy/lsp/debputy_ls.py index eb4162f..a1475b2 100644 --- a/src/debputy/lsp/debputy_ls.py +++ b/src/debputy/lsp/debputy_ls.py @@ -1,5 +1,6 @@ import dataclasses import os +import time from typing import ( Optional, List, @@ -17,19 +18,19 @@ from debputy.dh.dh_assistant import ( DhSequencerData, extract_dh_addons_from_control, ) -from debputy.lsprotocol.types import MarkupKind - from debputy.filesystem_scan import FSROOverlay, VirtualPathBase from debputy.linting.lint_util import ( LintState, ) -from debputy.lsp.style_prefs import ( - StylePreferenceTable, +from debputy.lsp.apt_cache import AptCache +from debputy.lsp.maint_prefs import ( + MaintainerPreferenceTable, MaintainerPreference, - determine_effective_style, + determine_effective_preference, ) from debputy.lsp.text_util import LintCapablePositionCodec from debputy.lsp.vendoring._deb822_repro import Deb822FileElement, parse_deb822_file +from debputy.lsprotocol.types import MarkupKind from debputy.packages import ( SourcePackage, BinaryPackage, @@ -301,16 +302,16 @@ class LSProvidedLintState(LintState): salsa_ci = self._resolve_salsa_ci() if source_package is None and salsa_ci is None: return None - style, _ = determine_effective_style( - self.style_preference_table, + style, _, _ = determine_effective_preference( + self.maint_preference_table, source_package, salsa_ci, ) return style @property - def style_preference_table(self) -> StylePreferenceTable: - return self._ls.style_preferences + def maint_preference_table(self) -> MaintainerPreferenceTable: + return self._ls.maint_preferences @property def salsa_ci(self) -> Optional[CommentedMap]: @@ -360,20 +361,42 @@ class DebputyLanguageServer(LanguageServer): self._plugin_feature_set: Optional[PluginProvidedFeatureSet] = None self._trust_language_ids: Optional[bool] = None self._finished_initialization = False - self.style_preferences = StylePreferenceTable({}, {}) + self.maint_preferences = MaintainerPreferenceTable({}, {}) + self.apt_cache = AptCache() + self.background_tasks = set() - def finish_initialization(self) -> None: + def finish_startup_initialization(self) -> None: if self._finished_initialization: return assert self._dctrl_parser is not None assert self._plugin_feature_set is not None assert self._trust_language_ids is not None - self.style_preferences = self.style_preferences.load_styles() + self.maint_preferences = self.maint_preferences.load_preferences() _info( - f"Loaded style preferences: {len(self.style_preferences.maintainer_preferences)} unique maintainer preferences recorded" + f"Loaded style preferences: {len(self.maint_preferences.maintainer_preferences)} unique maintainer preferences recorded" ) self._finished_initialization = True + async def on_initialize(self) -> None: + task = self.loop.create_task(self._load_apt_cache(), name="Index apt cache") + self.background_tasks.add(task) + task.add_done_callback(self.background_tasks.discard) + + def shutdown(self) -> None: + for task in self.background_tasks: + _info(f"Cancelling task: {task.get_name()}") + self.loop.call_soon_threadsafe(task.cancel) + return super().shutdown() + + async def _load_apt_cache(self) -> None: + _info("Starting load of apt cache data") + start = time.time() + await self.apt_cache.load() + end = time.time() + _info( + f"Loading apt cache finished after {end-start} seconds and is now in state {self.apt_cache.state}" + ) + @property def plugin_feature_set(self) -> PluginProvidedFeatureSet: res = self._plugin_feature_set diff --git a/src/debputy/lsp/lsp_debian_changelog.py b/src/debputy/lsp/lsp_debian_changelog.py index bde15b5..e3d2154 100644 --- a/src/debputy/lsp/lsp_debian_changelog.py +++ b/src/debputy/lsp/lsp_debian_changelog.py @@ -142,7 +142,7 @@ def _check_footer_line( if day_name not in _KNOWN_WEEK_DAYS: yield Diagnostic( position_codec.range_to_client_units(lines, day_name_range_server_units), - "Expected a three letter date here (Mon, Tue, ..., Sun).", + "Expected a three letter date here using US English format (Mon, Tue, ..., Sun).", severity=DiagnosticSeverity.Error, source="debputy", ) diff --git a/src/debputy/lsp/lsp_debian_control.py b/src/debputy/lsp/lsp_debian_control.py index 5bb6265..e33292d 100644 --- a/src/debputy/lsp/lsp_debian_control.py +++ b/src/debputy/lsp/lsp_debian_control.py @@ -2,6 +2,7 @@ import dataclasses import os.path import re import textwrap +from itertools import chain from typing import ( Union, Sequence, @@ -13,8 +14,11 @@ from typing import ( Iterable, ) -from debputy.analysis.debian_dir import scan_debian_dir +from debputy.analysis.analysis_util import flatten_ppfs +from debputy.analysis.debian_dir import resolve_debhelper_config_files +from debputy.dh.dh_assistant import extract_dh_compat_level from debputy.linting.lint_util import LintState +from debputy.lsp.apt_cache import PackageLookup from debputy.lsp.debputy_ls import DebputyLanguageServer from debputy.lsp.diagnostics import DiagnosticData from debputy.lsp.lsp_debian_control_reference_data import ( @@ -25,6 +29,7 @@ from debputy.lsp.lsp_debian_control_reference_data import ( package_name_to_section, all_package_relationship_fields, extract_first_value_and_position, + all_source_relationship_fields, ) from debputy.lsp.lsp_features import ( lint_diagnostics, @@ -92,7 +97,12 @@ from debputy.lsprotocol.types import ( InlayHint, InlayHintLabelPart, ) -from debputy.util import detect_possible_typo +from debputy.packager_provided_files import ( + PackagerProvidedFile, + detect_all_packager_provided_files, +) +from debputy.plugin.api.impl import plugin_metadata_for_debputys_own_plugin +from debputy.util import detect_possible_typo, PKGNAME_REGEX, _info try: from debputy.lsp.vendoring._deb822_repro.locatable import ( @@ -304,58 +314,12 @@ def _debian_control_hover( return deb822_hover(ls, params, _DCTRL_FILE_METADATA, custom_handler=_custom_hover) -def _custom_hover( - server_position: Position, - _current_field: Optional[str], +def _custom_hover_description( + _ls: "DebputyLanguageServer", + _known_field: DctrlKnownField, + line: str, _word_at_position: str, - known_field: Optional[DctrlKnownField], - in_value: bool, - _doc: "TextDocument", - lines: List[str], ) -> Optional[Union[Hover, str]]: - if not in_value: - return None - - line_no = server_position.line - line = lines[line_no] - substvar_search_ref = server_position.character - substvar = "" - try: - if line and line[substvar_search_ref] in ("$", "{"): - substvar_search_ref += 2 - substvar_start = line.rindex("${", 0, substvar_search_ref) - substvar_end = line.index("}", substvar_start) - if server_position.character <= substvar_end: - substvar = line[substvar_start : substvar_end + 1] - except (ValueError, IndexError): - pass - - if substvar == "${}" or _SUBSTVAR_RE.fullmatch(substvar): - substvar_md = _SUBSTVARS_DOC.get(substvar) - - computed_doc = "" - for_field = relationship_substvar_for_field(substvar) - if for_field: - # Leading empty line is intentional! - computed_doc = textwrap.dedent( - f""" - This substvar is a relationship substvar for the field {for_field}. - Relationship substvars are automatically added in the field they - are named after in `debhelper-compat (= 14)` or later, or with - `debputy` (any integration mode after 0.1.21). - """ - ) - - if substvar_md is None: - doc = f"No documentation for {substvar}.\n" - md_fields = "" - else: - doc = substvar_md.description - md_fields = "\n" + substvar_md.render_metadata_fields() - return f"# Substvar `{substvar}`\n\n{doc}{computed_doc}{md_fields}" - - if known_field is None or known_field.name != "Description": - return None if line[0].isspace(): return None try: @@ -398,7 +362,7 @@ def _custom_hover( package name already and it generally does not help the user understand what they are looking at. * In many situations, the user will only see the package name - and its synopsis. The synopsis must standalone. + and its synopsis. The synopsis must be able to stand alone. **Example renderings in various terminal UIs**: ``` @@ -443,6 +407,269 @@ def _custom_hover( ) +def _render_package_lookup( + package_lookup: PackageLookup, + known_field: DctrlKnownField, +) -> str: + name = package_lookup.name + provider = package_lookup.package + if package_lookup.package is None and len(package_lookup.provided_by) == 1: + provider = package_lookup.provided_by[0] + + if provider: + segments = [ + f"# {name} ({provider.version}, {provider.architecture}) ", + "", + ] + + if ( + _is_bd_field(known_field) + and name.startswith("dh-sequence-") + and len(name) > 12 + ): + sequence = name[12:] + segments.append( + f"This build-dependency will activate the `dh` sequence called `{sequence}`." + ) + segments.append("") + + elif ( + known_field.name == "Build-Depends" + and name.startswith("debputy-plugin-") + and len(name) > 15 + ): + plugin_name = name[15:] + segments.append( + f"This build-dependency will activate the `debputy` plugin called `{plugin_name}`." + ) + segments.append("") + + segments.extend( + [ + f"Synopsis: {provider.synopsis}", + f"Multi-Arch: {provider.multi_arch}", + f"Section: {provider.section}", + ] + ) + if provider.upstream_homepage is not None: + segments.append(f"Upstream homepage: {provider.upstream_homepage}") + segments.append("") + segments.append( + "Data is from the system's APT cache, which may not match the target distribution." + ) + return "\n".join(segments) + + segments = [ + f"# {name} [virtual]", + "", + "The package {name} is a virtual package provided by one of:", + ] + segments.extend(f" * {p.name}" for p in package_lookup.provided_by) + segments.append("") + segments.append( + "Data is from the system's APT cache, which may not match the target distribution." + ) + return "\n".join(segments) + + +def _disclaimer(is_empty: bool) -> str: + if is_empty: + return textwrap.dedent( + """\ + The system's APT cache is empty, so it was not possible to verify that the + package exist. +""" + ) + return textwrap.dedent( + """\ + The package is not known by the APT cache on this system, so there may be typo + or the package may not be available in the version of your distribution. +""" + ) + + +def _render_package_by_name( + name: str, known_field: DctrlKnownField, is_empty: bool +) -> Optional[str]: + if _is_bd_field(known_field) and name.startswith("dh-sequence-") and len(name) > 12: + sequence = name[12:] + return ( + textwrap.dedent( + f"""\ + # {name} + + This build-dependency will activate the `dh` sequence called `{sequence}`. + + """ + ) + + _disclaimer(is_empty) + ) + if ( + known_field.name == "Build-Depends" + and name.startswith("debputy-plugin-") + and len(name) > 15 + ): + plugin_name = name[15:] + return ( + textwrap.dedent( + f"""\ + # {name} + + This build-dependency will activate the `debputy` plugin called `{plugin_name}`. + + """ + ) + + _disclaimer(is_empty) + ) + return ( + textwrap.dedent( + f"""\ + # {name} + + """ + ) + + _disclaimer(is_empty) + ) + + +def _is_bd_field(known_field: DctrlKnownField) -> bool: + return known_field.name in ( + "Build-Depends", + "Build-Depends-Arch", + "Build-Depends-Indep", + ) + + +def _custom_hover_relationship_field( + ls: "DebputyLanguageServer", + known_field: DctrlKnownField, + _line: str, + word_at_position: str, +) -> Optional[Union[Hover, str]]: + apt_cache = ls.apt_cache + state = apt_cache.state + is_empty = False + _info(f"Rel field: {known_field.name} - {word_at_position} - {state}") + if "|" in word_at_position: + return textwrap.dedent( + f"""\ + Sorry, no hover docs for OR relations at the moment. + + The relation being matched: `{word_at_position}` + + The code is missing logic to determine which side of the OR the lookup is happening. + """ + ) + match = next(iter(PKGNAME_REGEX.finditer(word_at_position)), None) + if match is None: + return + package = match.group() + if state == "empty-cache": + state = "loaded" + is_empty = True + if state == "loaded": + result = apt_cache.lookup(package) + if result is None: + return _render_package_by_name( + package, + known_field, + is_empty=is_empty, + ) + return _render_package_lookup(result, known_field) + + if state in ( + "not-loaded", + "failed", + "tooling-not-available", + ): + details = apt_cache.load_error if apt_cache.load_error else "N/A" + return textwrap.dedent( + f"""\ + Sorry, the APT cache data is not available due to an error or missing tool. + + Details: {details} + """ + ) + + if state == "empty-cache": + return f"Cannot lookup {package}: APT cache data was empty" + + if state == "loading": + return f"Cannot lookup {package}: APT cache data is still being indexed. Please try again in a moment." + return None + + +_CUSTOM_FIELD_HOVER = { + field: _custom_hover_relationship_field + for field in chain( + all_package_relationship_fields().values(), + all_source_relationship_fields().values(), + ) + if field != "Provides" +} + +_CUSTOM_FIELD_HOVER["Description"] = _custom_hover_description + + +def _custom_hover( + ls: "DebputyLanguageServer", + server_position: Position, + _current_field: Optional[str], + word_at_position: str, + known_field: Optional[DctrlKnownField], + in_value: bool, + _doc: "TextDocument", + lines: List[str], +) -> Optional[Union[Hover, str]]: + if not in_value: + return None + + line_no = server_position.line + line = lines[line_no] + substvar_search_ref = server_position.character + substvar = "" + try: + if line and line[substvar_search_ref] in ("$", "{"): + substvar_search_ref += 2 + substvar_start = line.rindex("${", 0, substvar_search_ref) + substvar_end = line.index("}", substvar_start) + if server_position.character <= substvar_end: + substvar = line[substvar_start : substvar_end + 1] + except (ValueError, IndexError): + pass + + if substvar == "${}" or _SUBSTVAR_RE.fullmatch(substvar): + substvar_md = _SUBSTVARS_DOC.get(substvar) + + computed_doc = "" + for_field = relationship_substvar_for_field(substvar) + if for_field: + # Leading empty line is intentional! + computed_doc = textwrap.dedent( + f""" + This substvar is a relationship substvar for the field {for_field}. + Relationship substvars are automatically added in the field they + are named after in `debhelper-compat (= 14)` or later, or with + `debputy` (any integration mode after 0.1.21). + """ + ) + + if substvar_md is None: + doc = f"No documentation for {substvar}.\n" + md_fields = "" + else: + doc = substvar_md.description + md_fields = "\n" + substvar_md.render_metadata_fields() + return f"# Substvar `{substvar}`\n\n{doc}{computed_doc}{md_fields}" + + if known_field is None: + return None + dispatch = _CUSTOM_FIELD_HOVER.get(known_field.name) + if dispatch is None: + return None + return dispatch(ls, known_field, line, word_at_position) + + @lsp_completer(_LANGUAGE_IDS) def _debian_control_completions( ls: "DebputyLanguageServer", @@ -479,7 +706,7 @@ def _doc_inlay_hint( stanza_range = stanza.range_in_parent() if stanza_no < 1: continue - pkg_kvpair = stanza.get_kvpair_element("Package", use_get=True) + pkg_kvpair = stanza.get_kvpair_element(("Package", 0), use_get=True) if pkg_kvpair is None: continue @@ -580,7 +807,7 @@ def _binary_package_checks( ) -> None: package_name = stanza.get("Package", "") source_section = source_stanza.get("Section") - section_kvpair = stanza.get_kvpair_element("Section", use_get=True) + section_kvpair = stanza.get_kvpair_element(("Section", 0), use_get=True) section: Optional[str] = None if section_kvpair is not None: section, section_range = extract_first_value_and_position( @@ -708,9 +935,9 @@ def _diagnostics_for_paragraph( normalized_field_name_lc = normalize_dctrl_field_name(field_name_lc) known_field = known_fields.get(normalized_field_name_lc) field_value = stanza[field_name] - kvpair_position = kvpair.position_in_parent().relative_to(stanza_position) + kvpair_range_te = kvpair.range_in_parent().relative_to(stanza_position) field_range_te = kvpair.field_token.range_in_parent().relative_to( - kvpair_position + kvpair_range_te.start_pos ) field_position_te = field_range_te.start_pos field_range_server_units = te_range_to_lsp(field_range_te) @@ -790,7 +1017,7 @@ def _diagnostics_for_paragraph( kvpair, stanza, stanza_position, - kvpair_position, + kvpair_range_te, lint_state, field_name_typo_reported=field_name_typo_detected, ) @@ -1029,7 +1256,7 @@ def _package_range_of_stanza( binary_stanzas: List[Tuple[Deb822ParagraphElement, TEPosition]], ) -> Iterable[Tuple[str, Optional[str], Range]]: for stanza, stanza_position in binary_stanzas: - kvpair = stanza.get_kvpair_element("Package") + kvpair = stanza.get_kvpair_element(("Package", 0), use_get=True) if kvpair is None: continue representation_field_range = kvpair.range_in_parent().relative_to( @@ -1042,43 +1269,111 @@ def _package_range_of_stanza( yield stanza["Package"], stanza.get("Architecture"), representation_field_range -def _detect_misspelled_packaging_files( +def _packaging_files( lint_state: LintState, - binary_stanzas_w_pos: List[Tuple[Deb822ParagraphElement, TEPosition]], - diagnostics: List[Diagnostic], -) -> None: +) -> Iterable[PackagerProvidedFile]: + source_root = lint_state.source_root debian_dir = lint_state.debian_dir binary_packages = lint_state.binary_packages - if debian_dir is None or binary_packages is None: + if ( + source_root is None + or not source_root.has_fs_path + or debian_dir is None + or binary_packages is None + ): return dh_sequencer_data = lint_state.dh_sequencer_data - - all_pkg_file_data, _, _, _ = scan_debian_dir( - lint_state.plugin_feature_set, - binary_packages, - debian_dir, - uses_dh_sequencer=dh_sequencer_data.uses_dh_sequencer, - dh_sequences=dh_sequencer_data.sequences, + dh_sequences = dh_sequencer_data.sequences + is_debputy_package = ( + "debputy" in dh_sequences + or "zz-debputy" in dh_sequences + or "zz_debputy" in dh_sequences + or "zz-debputy-rrr" in dh_sequences ) + feature_set = lint_state.plugin_feature_set + known_packaging_files = feature_set.known_packaging_files + static_packaging_files = { + kpf.detection_value: kpf + for kpf in known_packaging_files.values() + if kpf.detection_method == "path" + } + ignored_path = set(static_packaging_files) + + if is_debputy_package: + all_debputy_ppfs = list( + flatten_ppfs( + detect_all_packager_provided_files( + feature_set.packager_provided_files, + debian_dir, + binary_packages, + allow_fuzzy_matches=True, + detect_typos=True, + ignore_paths=ignored_path, + ) + ) + ) + for ppf in all_debputy_ppfs: + if ppf.path.path in ignored_path: + continue + ignored_path.add(ppf.path.path) + yield ppf + + # FIXME: This should read the editor data, but dh_assistant does not support that. + dh_compat_level, _ = extract_dh_compat_level(cwd=source_root.fs_path) + if dh_compat_level is not None: + debputy_plugin_metadata = plugin_metadata_for_debputys_own_plugin() + dh_pkgfile_docs = { + kpf.detection_value: kpf + for kpf in known_packaging_files.values() + if kpf.detection_method == "dh.pkgfile" + } + ( + all_dh_ppfs, + _, + _, + ) = resolve_debhelper_config_files( + debian_dir, + binary_packages, + debputy_plugin_metadata, + dh_pkgfile_docs, + dh_sequences, + dh_compat_level, + saw_dh=dh_sequencer_data.uses_dh_sequencer, + ignore_paths=ignored_path, + ) + for ppf in all_dh_ppfs: + if ppf.path.path in ignored_path: + continue + ignored_path.add(ppf.path.path) + yield ppf + + +def _detect_misspelled_packaging_files( + lint_state: LintState, + binary_stanzas_w_pos: List[Tuple[Deb822ParagraphElement, TEPosition]], + diagnostics: List[Diagnostic], +) -> None: stanza_ranges = { p: (a, r) for p, a, r in _package_range_of_stanza(lint_state, binary_stanzas_w_pos) } - - for pkg_file_data in all_pkg_file_data: - binary_package = pkg_file_data.get("binary-package") - explicit_package = pkg_file_data.get("pkgfile-explicit-package-name", True) - name_segment = pkg_file_data.get("pkgfile-name-segment") - stem = pkg_file_data.get("pkgfile-stem") + for ppf in _packaging_files(lint_state): + binary_package = ppf.package_name + explicit_package = ppf.uses_explicit_package_name + name_segment = ppf.name_segment is not None + stem = ppf.definition.stem if binary_package is None or stem is None: continue - declared_arch, diag_range = stanza_ranges.get(binary_package) + res = stanza_ranges.get(binary_package) + if res is None: + continue + declared_arch, diag_range = res if diag_range is None: continue - path = pkg_file_data["path"] - likely_typo_of = pkg_file_data.get("likely-typo-of") - arch_restriction = pkg_file_data.get("pkgfile-architecture-restriction") + path = ppf.path.path + likely_typo_of = ppf.expected_path + arch_restriction = ppf.architecture_restriction if likely_typo_of is not None: # Handles arch_restriction == 'all' at the same time due to how # the `likely-typo-of` is created @@ -1120,7 +1415,7 @@ def _detect_misspelled_packaging_files( ) ) - if not pkg_file_data.get("pkgfile-is-active-in-build", True): + if not ppf.definition.has_active_command: diagnostics.append( Diagnostic( diag_range, @@ -1137,14 +1432,20 @@ def _detect_misspelled_packaging_files( if not explicit_package and name_segment is not None: basename = os.path.basename(path) + if basename == ppf.definition.stem: + continue alt_name = f"{binary_package}.{stem}" if arch_restriction is not None: alt_name = f"{alt_name}.{arch_restriction}" + if ppf.definition.allow_name_segment: + or_alt_name = f' (or maybe "debian/{binary_package}.{basename}")' + else: + or_alt_name = "" diagnostics.append( Diagnostic( diag_range, - f'Possible typo in "{path}". Consider renaming the file to "debian/{binary_package}.{basename}"' - f' or "debian/{alt_name} if it is intended for {binary_package}', + f'Possible typo in "{path}". Consider renaming the file to "debian/{alt_name}"' + f"{or_alt_name} if it is intended for {binary_package}", severity=DiagnosticSeverity.Warning, source="debputy", data=DiagnosticData( diff --git a/src/debputy/lsp/lsp_debian_control_reference_data.py b/src/debputy/lsp/lsp_debian_control_reference_data.py index 1e32d3c..85a04a3 100644 --- a/src/debputy/lsp/lsp_debian_control_reference_data.py +++ b/src/debputy/lsp/lsp_debian_control_reference_data.py @@ -49,6 +49,8 @@ from debputy.lsp.lsp_reference_keyword import ( ALL_PUBLIC_NAMED_STYLES, Keyword, allowed_values, + format_comp_item_synopsis_doc, + UsageHint, ) from debputy.lsp.quickfixes import ( propose_correct_text_quick_fix, @@ -104,7 +106,7 @@ except ImportError: if TYPE_CHECKING: - from debputy.lsp.style_prefs import EffectivePreference + from debputy.lsp.maint_prefs import EffectiveFormattingPreference F = TypeVar("F", bound="Deb822KnownField") @@ -161,6 +163,7 @@ CustomFieldCheck = Callable[ Deb822FileElement, Deb822KeyValuePairElement, "TERange", + "TERange", Deb822ParagraphElement, "TEPosition", LintState, @@ -299,7 +302,8 @@ ALL_SECTIONS = allowed_values( ALL_PRIORITIES = allowed_values( Keyword( "required", - synopsis_doc="[RARE]: Package is Essential or an Essential package needs it (and is not a library)", + usage_hint="rare", + synopsis_doc="Package is Essential or an Essential package needs it (and is not a library)", hover_text=textwrap.dedent( """\ The package is necessary for the proper functioning of the system (read: dpkg needs it). @@ -313,7 +317,8 @@ ALL_PRIORITIES = allowed_values( ), Keyword( "important", - synopsis_doc="[RARE]: Bare minimum of bare minimum of commonly-expected and necessary tools", + usage_hint="rare", + synopsis_doc="Bare minimum of commonly-expected and necessary tools", hover_text=textwrap.dedent( """\ The *important* packages are a bare minimum of commonly-expected and necessary tools. @@ -327,7 +332,8 @@ ALL_PRIORITIES = allowed_values( ), Keyword( "standard", - synopsis_doc="[RARE]: If your distribution installer would install this by default (not for libraries)", + usage_hint="rare", + synopsis_doc="If your distribution installer would install this by default (not for libraries)", hover_text=textwrap.dedent( """\ These packages provide a reasonable small but not too limited character-mode system. This is @@ -454,7 +460,8 @@ def _sv_field_validation( _known_field: "F", _deb822_file: Deb822FileElement, kvpair: Deb822KeyValuePairElement, - _field_range: "TERange", + _kvpair_range: "TERange", + _field_name_range_te: "TERange", _stanza: Deb822ParagraphElement, stanza_position: "TEPosition", lint_state: LintState, @@ -511,12 +518,13 @@ def _dctrl_ma_field_validation( _known_field: "F", _deb822_file: Deb822FileElement, _kvpair: Deb822KeyValuePairElement, - _field_range: "TERange", + _kvpair_range: "TERange", + _field_name_range: "TERange", stanza: Deb822ParagraphElement, stanza_position: "TEPosition", lint_state: LintState, ) -> Iterable[Diagnostic]: - ma_kvpair = stanza.get_kvpair_element("Multi-Arch", use_get=True) + ma_kvpair = stanza.get_kvpair_element(("Multi-Arch", 0), use_get=True) arch = stanza.get("Architecture", "any") if arch == "all" and ma_kvpair is not None: ma_value, ma_value_range = extract_first_value_and_position( @@ -537,14 +545,15 @@ def _udeb_only_field_validation( known_field: "F", _deb822_file: Deb822FileElement, _kvpair: Deb822KeyValuePairElement, - field_range_te: "TERange", + _kvpair_range: "TERange", + field_name_range: "TERange", stanza: Deb822ParagraphElement, _stanza_position: "TEPosition", lint_state: LintState, ) -> Iterable[Diagnostic]: package_type = stanza.get("Package-Type") if package_type != "udeb": - field_range_server_units = te_range_to_lsp(field_range_te) + field_range_server_units = te_range_to_lsp(field_name_range) field_range = lint_state.position_codec.range_to_client_units( lint_state.lines, field_range_server_units, @@ -584,14 +593,15 @@ def _arch_not_all_only_field_validation( known_field: "F", _deb822_file: Deb822FileElement, _kvpair: Deb822KeyValuePairElement, - field_range_te: "TERange", + _kvpair_range_te: "TERange", + field_name_range_te: "TERange", stanza: Deb822ParagraphElement, _stanza_position: "TEPosition", lint_state: LintState, ) -> Iterable[Diagnostic]: architecture = stanza.get("Architecture") if architecture == "all": - field_range_server_units = te_range_to_lsp(field_range_te) + field_range_server_units = te_range_to_lsp(field_name_range_te) field_range = lint_state.position_codec.range_to_client_units( lint_state.lines, field_range_server_units, @@ -604,7 +614,7 @@ def _arch_not_all_only_field_validation( ) -def _span_to_client_range( +def _single_line_span_to_client_range( span: Tuple[int, int], relative_to: "TEPosition", lint_state: LintState, @@ -655,7 +665,7 @@ def _check_synopsis( # TODO: Handle ${...} expansion if starts_with_article: yield Diagnostic( - _span_to_client_range( + _single_line_span_to_client_range( starts_with_article.span(1), synopsis_range_te.start_pos, lint_state, @@ -668,7 +678,7 @@ def _check_synopsis( # Policy says `certainly under 80 characters.`, so exactly 80 characters is considered bad too. span = synopsis_offset + 79, len(synopsis_text_with_leading_space) yield Diagnostic( - _span_to_client_range( + _single_line_span_to_client_range( span, synopsis_range_te.start_pos, lint_state, @@ -681,7 +691,7 @@ def _check_synopsis( synopsis_text_with_leading_space ): yield Diagnostic( - _span_to_client_range( + _single_line_span_to_client_range( template_match.span(1), synopsis_range_te.start_pos, lint_state, @@ -694,7 +704,7 @@ def _check_synopsis( synopsis_text_with_leading_space ): yield Diagnostic( - _span_to_client_range( + _single_line_span_to_client_range( too_short_match.span(1), synopsis_range_te.start_pos, lint_state, @@ -709,7 +719,8 @@ def dctrl_description_validator( _known_field: "F", _deb822_file: Deb822FileElement, kvpair: Deb822KeyValuePairElement, - field_range_te: "TERange", + kvpair_range_te: "TERange", + _field_name_range: "TERange", stanza: Deb822ParagraphElement, _stanza_position: "TEPosition", lint_state: LintState, @@ -720,11 +731,11 @@ def dctrl_description_validator( package = stanza.get("Package") synopsis_value_line = value_lines[0] value_range_te = kvpair.value_element.range_in_parent().relative_to( - field_range_te.start_pos + kvpair_range_te.start_pos ) if synopsis_value_line.continuation_line_token is None: field_name_range_te = kvpair.field_token.range_in_parent().relative_to( - field_range_te.start_pos + kvpair_range_te.start_pos ) synopsis_range_te = synopsis_value_line.range_in_parent().relative_to( value_range_te.start_pos @@ -748,14 +759,15 @@ def _each_value_match_regex_validation( _known_field: "F", _deb822_file: Deb822FileElement, kvpair: Deb822KeyValuePairElement, - field_range_te: "TERange", + kvpair_range_te: "TERange", + _field_name_range_te: "TERange", _stanza: Deb822ParagraphElement, _stanza_position: "TEPosition", lint_state: LintState, ) -> Iterable[Diagnostic]: value_element_pos = kvpair.value_element.position_in_parent().relative_to( - field_range_te.start_pos + kvpair_range_te.start_pos ) for value_ref in kvpair.interpret_as( LIST_SPACE_SEPARATED_INTERPRETATION @@ -787,6 +799,277 @@ def _each_value_match_regex_validation( return _validator +_DEP_OR_RELATION = re.compile(r"[|]") +_DEP_RELATION_CLAUSE = re.compile( + r""" + ^ + \s* + (?P<name_arch_qual>[-+.a-zA-Z0-9${}:]{2,}) + \s* + (?: [(] \s* (?P<operator>>>|>=|=|<=|<<) \s* (?P<version> [^)]+) \s* [)] \s* )? + (?: \[ (?P<arch_restriction> [\s!\w\-]+) ] \s*)? + (?: < (?P<build_profile_restriction> .+ ) > \s*)? + ((?P<garbage>\S.*)\s*)? + $ +""", + re.VERBOSE | re.MULTILINE, +) + + +def _span_to_te_range( + text: str, + start_pos: int, + end_pos: int, +) -> TERange: + prefix = text[0:start_pos] + prefix_plus_text = text[0:end_pos] + + start_line = prefix.count("\n") + if start_line: + start_newline_offset = prefix.rindex("\n") + # +1 to skip past the newline + start_cursor_pos = start_pos - (start_newline_offset + 1) + else: + start_cursor_pos = start_pos + + end_line = prefix_plus_text.count("\n") + if end_line == start_line: + end_cursor_pos = start_cursor_pos + (end_pos - start_pos) + else: + end_newline_offset = prefix_plus_text.rindex("\n") + end_cursor_pos = end_pos - (end_newline_offset + 1) + + return TERange( + TEPosition( + start_line, + start_cursor_pos, + ), + TEPosition( + end_line, + end_cursor_pos, + ), + ) + + +def _split_w_spans( + v: str, + sep: str, + *, + offset: int = 0, +) -> Sequence[Tuple[str, int, int]]: + separator_size = len(sep) + parts = v.split(sep) + for part in parts: + size = len(part) + end_offset = offset + size + yield part, offset, end_offset + offset = end_offset + separator_size + + +_COLLAPSE_WHITESPACE = re.compile(r"\s+") + + +def _cleanup_rel(rel: str) -> str: + return _COLLAPSE_WHITESPACE.sub(" ", rel.strip()) + + +def _text_to_te_position(text: str) -> "TEPosition": + newlines = text.count("\n") + if not newlines: + return TEPosition( + newlines, + len(text), + ) + last_newline_offset = text.rindex("\n") + line_offset = len(text) - (last_newline_offset + 1) + return TEPosition( + newlines, + line_offset, + ) + + +def _dctrl_validate_dep( + known_field: "F", + _deb822_file: Deb822FileElement, + kvpair: Deb822KeyValuePairElement, + kvpair_range_te: "TERange", + _field_name_range: "TERange", + _stanza: Deb822ParagraphElement, + _stanza_position: "TEPosition", + lint_state: LintState, +) -> Iterable[Diagnostic]: + value_element_pos = kvpair.value_element.position_in_parent().relative_to( + kvpair_range_te.start_pos + ) + raw_value_with_comments = kvpair.value_element.convert_to_text() + raw_value_masked_comments = "".join( + (line if not line.startswith("#") else (" " * (len(line) - 1)) + "\n") + for line in raw_value_with_comments.splitlines(keepends=True) + ) + if isinstance(known_field, DctrlRelationshipKnownField): + version_operators = known_field.allowed_version_operators + supports_or_relation = known_field.supports_or_relation + else: + version_operators = frozenset() + supports_or_relation = True + + for rel, rel_offset, rel_end_offset in _split_w_spans( + raw_value_masked_comments, "," + ): + seen_relation = False + for or_rel, offset, end_offset in _split_w_spans(rel, "|", offset=rel_offset): + if or_rel.isspace(): + continue + if seen_relation and not supports_or_relation: + separator_range_te = TERange( + _text_to_te_position(raw_value_masked_comments[: offset - 1]), + _text_to_te_position(raw_value_masked_comments[:offset]), + ).relative_to(value_element_pos) + separator_range = lint_state.position_codec.range_to_client_units( + lint_state.lines, + te_range_to_lsp(separator_range_te), + ) + yield Diagnostic( + lint_state.position_codec.range_to_client_units( + lint_state.lines, + separator_range, + ), + f'The field {known_field.name} does not support "|" (OR) in relations.', + DiagnosticSeverity.Error, + source="debputy", + ) + seen_relation = True + m = _DEP_RELATION_CLAUSE.fullmatch(or_rel) + + if m is not None: + garbage = m.group("garbage") + version_operator = m.group("operator") + if ( + version_operators + and version_operator is not None + and version_operator not in version_operators + ): + operator_span = m.span("operator") + v_start_offset = offset + operator_span[0] + v_end_offset = offset + operator_span[1] + version_problem_range_te = TERange( + _text_to_te_position( + raw_value_masked_comments[:v_start_offset] + ), + _text_to_te_position(raw_value_masked_comments[:v_end_offset]), + ).relative_to(value_element_pos) + + version_problem_range = ( + lint_state.position_codec.range_to_client_units( + lint_state.lines, + te_range_to_lsp(version_problem_range_te), + ) + ) + sorted_version_operators = sorted(version_operators) + yield Diagnostic( + lint_state.position_codec.range_to_client_units( + lint_state.lines, + version_problem_range, + ), + f'The version operator "{version_operator}" is not allowed in {known_field.name}', + DiagnosticSeverity.Error, + source="debputy", + data=DiagnosticData( + quickfixes=[ + propose_correct_text_quick_fix(n) + for n in sorted_version_operators + ] + ), + ) + else: + garbage = None + + if m is not None and not garbage: + continue + if m is not None: + garbage_span = m.span("garbage") + garbage_start, garbage_end = garbage_span + error_start_offset = offset + garbage_start + error_end_offset = offset + garbage_end + garbage_part = raw_value_masked_comments[ + error_start_offset:error_end_offset + ] + else: + garbage_part = None + error_start_offset = offset + error_end_offset = end_offset + + problem_range_te = TERange( + _text_to_te_position(raw_value_masked_comments[:error_start_offset]), + _text_to_te_position(raw_value_masked_comments[:error_end_offset]), + ).relative_to(value_element_pos) + + problem_range = lint_state.position_codec.range_to_client_units( + lint_state.lines, + te_range_to_lsp(problem_range_te), + ) + if garbage_part is not None: + if _DEP_RELATION_CLAUSE.fullmatch(garbage_part) is not None: + msg = ( + "Trailing data after a relationship that might be a second relationship." + " Is a separator missing before this part?" + ) + else: + msg = "Parse error of the relationship. Either a syntax error or a missing separator somewhere." + yield Diagnostic( + lint_state.position_codec.range_to_client_units( + lint_state.lines, + problem_range, + ), + msg, + DiagnosticSeverity.Error, + source="debputy", + ) + else: + dep = _cleanup_rel( + raw_value_masked_comments[error_start_offset:error_end_offset] + ) + yield Diagnostic( + lint_state.position_codec.range_to_client_units( + lint_state.lines, + problem_range, + ), + f'Could not parse "{dep}" as a dependency relation.', + DiagnosticSeverity.Error, + source="debputy", + ) + + +def _rrr_build_driver_mismatch( + _known_field: "F", + _deb822_file: Deb822FileElement, + _kvpair: Deb822KeyValuePairElement, + kvpair_range_te: "TERange", + _field_name_range: "TERange", + stanza: Deb822ParagraphElement, + _stanza_position: "TEPosition", + lint_state: LintState, +) -> Iterable[Diagnostic]: + dr = stanza.get("Build-Driver", "debian-rules") + if dr != "debian-rules": + yield Diagnostic( + lint_state.position_codec.range_to_client_units( + lint_state.lines, + te_range_to_lsp(kvpair_range_te), + ), + f"The Rules-Requires-Root field is irrelevant for the `Build-Driver` `{dr}`.", + DiagnosticSeverity.Warning, + source="debputy", + data=DiagnosticData( + quickfixes=[ + propose_remove_range_quick_fix( + proposed_title="Remove Rules-Requires-Root" + ) + ] + ), + ) + + class Dep5Matcher(BasenameGlobMatch): def __init__(self, basename_glob: str) -> None: super().__init__( @@ -882,7 +1165,8 @@ def _dep5_files_check( known_field: "F", _deb822_file: Deb822FileElement, kvpair: Deb822KeyValuePairElement, - field_range_te: "TERange", + kvpair_range_te: "TERange", + _field_name_range: "TERange", _stanza: Deb822ParagraphElement, _stanza_position: "TEPosition", lint_state: LintState, @@ -890,7 +1174,7 @@ def _dep5_files_check( interpreter = known_field.field_value_class.interpreter() assert interpreter is not None full_value_range = kvpair.value_element.range_in_parent().relative_to( - field_range_te.start_pos + kvpair_range_te.start_pos ) values_with_ranges = [] for value_ref in kvpair.interpret_as(interpreter).iter_value_references(): @@ -916,7 +1200,8 @@ def _combined_custom_field_check(*checks: CustomFieldCheck) -> CustomFieldCheck: known_field: "F", deb822_file: Deb822FileElement, kvpair: Deb822KeyValuePairElement, - field_range_te: "TERange", + kvpair_range_te: "TERange", + field_name_range_te: "TERange", stanza: Deb822ParagraphElement, stanza_position: "TEPosition", lint_state: LintState, @@ -926,7 +1211,8 @@ def _combined_custom_field_check(*checks: CustomFieldCheck) -> CustomFieldCheck: known_field, deb822_file, kvpair, - field_range_te, + kvpair_range_te, + field_name_range_te, stanza, stanza_position, lint_state, @@ -1082,7 +1368,7 @@ _PKGNAME_VS_SECTION_RULES = [ ), _package_name_section_rule( "localization", - lambda n: n.startswith("hypen-"), + lambda n: n.startswith("hyphen-"), confirm_re=re.compile(r"^hyphen-[a-z]{2}(?:-[a-z]{2})?$"), ), _package_name_section_rule( @@ -1215,6 +1501,7 @@ class Deb822KnownField: ) # One-line description for space-constrained docs (such as completion docs) synopsis_doc: Optional[str] = None + usage_hint: Optional[UsageHint] = None hover_text: Optional[str] = None spellcheck_value: bool = False is_stanza_name: bool = False @@ -1274,7 +1561,11 @@ class Deb822KnownField: insert_text=complete_as, deprecated=is_deprecated, tags=tags, - detail=self.synopsis_doc, + detail=format_comp_item_synopsis_doc( + self.usage_hint, + self.synopsis_doc, + is_deprecated, + ), documentation=doc, ) @@ -1400,7 +1691,13 @@ class Deb822KnownField: keyword.value, insert_text=keyword.value, sort_text=keyword.sort_text, - detail=keyword.synopsis_doc, + detail=format_comp_item_synopsis_doc( + keyword.usage_hint, + keyword.synopsis_doc, + keyword.is_deprecated, + ), + deprecated=keyword.is_deprecated, + tags=[CompletionItemTag.Deprecated] if keyword.is_deprecated else None, documentation=( MarkupContent(value=keyword.hover_text, kind=markdown_kind) if keyword.hover_text @@ -1437,18 +1734,18 @@ class Deb822KnownField: kvpair: Deb822KeyValuePairElement, stanza: Deb822ParagraphElement, stanza_position: "TEPosition", - kvpair_position: "TEPosition", + kvpair_range_te: "TERange", lint_state: LintState, *, field_name_typo_reported: bool = False, ) -> Iterable[Diagnostic]: field_name_token = kvpair.field_token - field_range_te = kvpair.field_token.range_in_parent().relative_to( - kvpair_position + field_name_range_te = kvpair.field_token.range_in_parent().relative_to( + kvpair_range_te.start_pos ) yield from self._diagnostics_for_field_name( field_name_token, - field_range_te, + field_name_range_te, field_name_typo_reported, lint_state, ) @@ -1457,16 +1754,19 @@ class Deb822KnownField: self, deb822_file, kvpair, - field_range_te, + kvpair_range_te, + field_name_range_te, stanza, stanza_position, lint_state, ) - yield from self._dep5_file_list_diagnostics(kvpair, kvpair_position, lint_state) + yield from self._dep5_file_list_diagnostics( + kvpair, kvpair_range_te.start_pos, lint_state + ) if not self.spellcheck_value: yield from self._known_value_diagnostics( kvpair, - kvpair_position, + kvpair_range_te.start_pos, lint_state, ) @@ -1725,7 +2025,7 @@ class Deb822KnownField: def reformat_field( self, - effective_preference: "EffectivePreference", + effective_preference: "EffectiveFormattingPreference", stanza_range: TERange, kvpair: Deb822KeyValuePairElement, formatter: FormatterCallback, @@ -1748,7 +2048,7 @@ class DctrlLikeKnownField(Deb822KnownField): def reformat_field( self, - effective_preference: "EffectivePreference", + effective_preference: "EffectiveFormattingPreference", stanza_range: TERange, kvpair: Deb822KeyValuePairElement, formatter: FormatterCallback, @@ -1757,7 +2057,7 @@ class DctrlLikeKnownField(Deb822KnownField): ) -> Iterable[TextEdit]: interpretation = self.field_value_class.interpreter() if ( - not effective_preference.formatting_deb822_normalize_field_content + not effective_preference.deb822_normalize_field_content or interpretation is None ): yield from super(DctrlLikeKnownField, self).reformat_field( @@ -1879,7 +2179,7 @@ class DctrlKnownField(DctrlLikeKnownField): def reformat_field( self, - effective_preference: "EffectivePreference", + effective_preference: "EffectiveFormattingPreference", stanza_range: TERange, kvpair: Deb822KeyValuePairElement, formatter: FormatterCallback, @@ -1888,7 +2188,7 @@ class DctrlKnownField(DctrlLikeKnownField): ) -> Iterable[TextEdit]: if ( self.name == "Architecture" - and effective_preference.formatting_deb822_normalize_field_content + and effective_preference.deb822_normalize_field_content ): interpretation = self.field_value_class.interpreter() assert interpretation is not None @@ -1930,6 +2230,16 @@ class DctrlKnownField(DctrlLikeKnownField): return self.is_relationship_field or self.name == "Uploaders" +@dataclasses.dataclass(slots=True, frozen=True) +class DctrlRelationshipKnownField(DctrlKnownField): + allowed_version_operators: FrozenSet[str] = frozenset() + supports_or_relation: bool = True + + @property + def is_relationship_field(self) -> bool: + return True + + SOURCE_FIELDS = _fields( DctrlKnownField( "Source", @@ -2053,6 +2363,47 @@ SOURCE_FIELDS = _fields( ), ), DctrlKnownField( + "Build-Driver", + FieldValueClass.SINGLE_VALUE, + default_value="debian-rules", + known_values=allowed_values( + Keyword( + "debian-rules", + synopsis_doc="Build via `debian/rules`", + hover_text=textwrap.dedent( + """\ + Use the `debian/rules` interface for building packages. + + This is the historical default and the interface that Debian Packages have used for + decades to build debs. + """ + ), + ), + Keyword( + "debputy", + synopsis_doc="Build with `debputy`", + hover_text=textwrap.dedent( + """\ + Use the `debputy` interface for building the package. + + This is provides the "full" integration mode with `debputy` where all parts of the + package build is handled by `debputy`. + + This *may* make any `debhelper` build-dependency redundant depending on which build + system is used. Some build systems (such as `autoconf` still use `debhelper` based tools). + """ + ), + ), + ), + synopsis_doc="Which implementation dpkg should use for the build", + hover_text=textwrap.dedent( + """\ + The name of the build driver that dpkg (`dpkg-buildpackage`) will use for assembling the + package. + """ + ), + ), + DctrlKnownField( "Vcs-Browser", FieldValueClass.SINGLE_VALUE, synopsis_doc="URL for browsers to interact with packaging VCS", @@ -2169,7 +2520,7 @@ SOURCE_FIELDS = _fields( deprecated_with_no_replacement=True, default_value="no", known_values=allowed_values("yes", "no"), - synopsis_doc="**Obsolete**: Old ACL mechanism for Debian Maintainers", + synopsis_doc="Old ACL mechanism for Debian Maintainers", hover_text=textwrap.dedent( """\ Obsolete field @@ -2186,6 +2537,7 @@ SOURCE_FIELDS = _fields( DctrlKnownField( "Build-Depends", FieldValueClass.COMMA_SEPARATED_LIST, + custom_field_check=_dctrl_validate_dep, synopsis_doc="Dependencies requires for clean and full build actions", hover_text=textwrap.dedent( """\ @@ -2196,6 +2548,7 @@ SOURCE_FIELDS = _fields( DctrlKnownField( "Build-Depends-Arch", FieldValueClass.COMMA_SEPARATED_LIST, + custom_field_check=_dctrl_validate_dep, synopsis_doc="Dependencies requires for arch:any action (build-arch/binary-arch)", hover_text=textwrap.dedent( """\ @@ -2212,6 +2565,7 @@ SOURCE_FIELDS = _fields( DctrlKnownField( "Build-Depends-Indep", FieldValueClass.COMMA_SEPARATED_LIST, + custom_field_check=_dctrl_validate_dep, synopsis_doc="Dependencies requires for arch:all action (build-indep/binary-indep)", hover_text=textwrap.dedent( """\ @@ -2225,9 +2579,11 @@ SOURCE_FIELDS = _fields( """ ), ), - DctrlKnownField( + DctrlRelationshipKnownField( "Build-Conflicts", FieldValueClass.COMMA_SEPARATED_LIST, + supports_or_relation=False, + custom_field_check=_dctrl_validate_dep, synopsis_doc="Package versions that will break the build or the clean target (use sparingly)", hover_text=textwrap.dedent( """\ @@ -2240,9 +2596,11 @@ SOURCE_FIELDS = _fields( """ ), ), - DctrlKnownField( + DctrlRelationshipKnownField( "Build-Conflicts-Arch", FieldValueClass.COMMA_SEPARATED_LIST, + supports_or_relation=False, + custom_field_check=_dctrl_validate_dep, synopsis_doc="Package versions that will break an arch:any build (use sparingly)", hover_text=textwrap.dedent( """\ @@ -2255,9 +2613,11 @@ SOURCE_FIELDS = _fields( """ ), ), - DctrlKnownField( + DctrlRelationshipKnownField( "Build-Conflicts-Indep", FieldValueClass.COMMA_SEPARATED_LIST, + supports_or_relation=False, + custom_field_check=_dctrl_validate_dep, synopsis_doc="Package versions that will break an arch:all build (use sparingly)", hover_text=textwrap.dedent( """\ @@ -2307,6 +2667,7 @@ SOURCE_FIELDS = _fields( "Rules-Requires-Root", FieldValueClass.SPACE_SEPARATED_LIST, unknown_value_diagnostic_severity=None, + custom_field_check=_rrr_build_driver_mismatch, known_values=allowed_values( Keyword( "no", @@ -2369,6 +2730,9 @@ SOURCE_FIELDS = _fields( ` Build-Depends` on `dpkg-build-api (>= 1)` or later, the default is `no`. Otherwise, the default is `binary-target` + This field is only relevant when when the `Build-Driver` is `debian-rules` (which it is by + default). + Note it is **not** possible to require running the package as "true root". """ ), @@ -2405,7 +2769,7 @@ SOURCE_FIELDS = _fields( "X-Python-Version", FieldValueClass.COMMA_SEPARATED_LIST, replaced_by="X-Python3-Version", - synopsis_doc="**Obsolete**: Supported Python2 versions (`dh-python` specific)", + synopsis_doc="Supported Python2 versions (`dh-python` specific)", hover_text=textwrap.dedent( """\ Obsolete field for declaring the supported Python2 versions @@ -2467,6 +2831,17 @@ SOURCE_FIELDS = _fields( ), ), DctrlKnownField( + "XS-Ruby-Versions", + FieldValueClass.FREE_TEXT_FIELD, + deprecated_with_no_replacement=True, + synopsis_doc="Obsolete", + hover_text=textwrap.dedent( + """\ + Obsolete according to https://bugs.debian.org/1075762 + """ + ), + ), + DctrlKnownField( "Description", FieldValueClass.FREE_TEXT_FIELD, spellcheck_value=True, @@ -2622,10 +2997,10 @@ BINARY_FIELDS = _fields( (APT and dpkg) will refuse to uninstall it without some very insisting force options and warnings. * Other packages are not required to declare explicit dependencies on essential packages as a - side-effect of the above except as to ensure a that the given essential package is upgraded + side-effect of the above except as to ensure that the given essential package is upgraded to a given minimum version. - * Once installed, essential packages function must at all time no matter where dpkg is in its + * Once installed, essential packages must function at all times no matter where dpkg is in its installation or upgrade process. During bootstrapping or installation, this requirement is relaxed. """ @@ -2729,6 +3104,7 @@ BINARY_FIELDS = _fields( DctrlKnownField( "Depends", FieldValueClass.COMMA_SEPARATED_LIST, + custom_field_check=_dctrl_validate_dep, synopsis_doc="Dependencies required to install and use this package", hover_text=textwrap.dedent( """\ @@ -2758,6 +3134,7 @@ BINARY_FIELDS = _fields( DctrlKnownField( "Recommends", FieldValueClass.COMMA_SEPARATED_LIST, + custom_field_check=_dctrl_validate_dep, synopsis_doc="Optional dependencies **most** people should have", hover_text=textwrap.dedent( """\ @@ -2782,6 +3159,7 @@ BINARY_FIELDS = _fields( DctrlKnownField( "Suggests", FieldValueClass.COMMA_SEPARATED_LIST, + custom_field_check=_dctrl_validate_dep, synopsis_doc="Optional dependencies that some people might want", hover_text=textwrap.dedent( """\ @@ -2799,6 +3177,7 @@ BINARY_FIELDS = _fields( DctrlKnownField( "Enhances", FieldValueClass.COMMA_SEPARATED_LIST, + custom_field_check=_dctrl_validate_dep, synopsis_doc="Packages enhanced by installing this package", hover_text=textwrap.dedent( """\ @@ -2814,9 +3193,12 @@ BINARY_FIELDS = _fields( """ ), ), - DctrlKnownField( + DctrlRelationshipKnownField( "Provides", FieldValueClass.COMMA_SEPARATED_LIST, + custom_field_check=_dctrl_validate_dep, + supports_or_relation=False, + allowed_version_operators=frozenset(["="]), synopsis_doc="Additional packages/versions this package dependency-wise satisfy", hover_text=textwrap.dedent( """\ @@ -2857,9 +3239,11 @@ BINARY_FIELDS = _fields( """ ), ), - DctrlKnownField( + DctrlRelationshipKnownField( "Conflicts", FieldValueClass.COMMA_SEPARATED_LIST, + custom_field_check=_dctrl_validate_dep, + supports_or_relation=False, synopsis_doc="Packages that this package is not co-installable with", hover_text=textwrap.dedent( """\ @@ -2886,9 +3270,11 @@ BINARY_FIELDS = _fields( """ ), ), - DctrlKnownField( + DctrlRelationshipKnownField( "Breaks", FieldValueClass.COMMA_SEPARATED_LIST, + custom_field_check=_dctrl_validate_dep, + supports_or_relation=False, synopsis_doc="Package/versions that does not work with this package", hover_text=textwrap.dedent( """\ @@ -2912,9 +3298,10 @@ BINARY_FIELDS = _fields( """ ), ), - DctrlKnownField( + DctrlRelationshipKnownField( "Replaces", FieldValueClass.COMMA_SEPARATED_LIST, + custom_field_check=_dctrl_validate_dep, synopsis_doc="This package replaces content from these packages/versions", hover_text=textwrap.dedent( """\ @@ -2953,7 +3340,7 @@ BINARY_FIELDS = _fields( *advised to leave it at its default until you have a working basic package or lots of time to understand* *this topic.* - Declare that the package will only built when the given build-profiles are satisfied. + Declare that the package will only be built when the given build-profiles are satisfied. This field is primarily used in combination with build profiles inside the build dependency related fields to reduce the number of build dependencies required during bootstrapping of a new architecture. @@ -3086,7 +3473,8 @@ BINARY_FIELDS = _fields( "allowed", # Never show `allowed` first, it is the absolute least likely candidate. sort_text="zzzz-allowed", - synopsis_doc="[RARE]: Consumer decides whether it is `same` and `foreign`", + usage_hint="rare", + synopsis_doc="Consumer decides whether it is `same` and `foreign`", hover_text=textwrap.dedent( """\ **Advanced and very rare value**. This value is exceedingly rare to the point that less @@ -3141,8 +3529,9 @@ BINARY_FIELDS = _fields( * If you have an architecture dependent package, where everything is installed in `/usr/lib/${DEB_HOST_MULTIARCH}` (plus a bit of standard documentation in `/usr/share/doc`), then - you *probably* want `Multi-Arch: same`. Note that `debputy` automatically detects the most common - variants of this case and sets the field for you. + you *probably* want `Multi-Arch: same`. Note that when using `debputy` as the build helper, `debputy` + will automatically detect the most common variants of this case and sets the field for you when + relevant. * If none of the above applies, then omit the field unless you know what you are doing or you are receiving advice from a Multi-Arch expert. @@ -3168,11 +3557,11 @@ BINARY_FIELDS = _fields( solely providing data or binaries that have "Multi-Arch neutral interfaces". Sadly, describing a "Multi-Arch neutral interface" is hard and often only done by Multi-Arch experts on a case-by-case basis. Among other, scripts despite being the same on all architectures can still have a "non-neutral" - "Multi-Arch" interface if their output is architecture dependent or if they dependencies force them + "Multi-Arch" interface if their output is architecture dependent or if their dependencies force them out of the `foreign` role. The dependency issue usually happens when depending indirectly on an `Multi-Arch: allowed` package. - Some programs are have "Multi-Arch dependent interfaces" and are not safe to declare as + Some programs have "Multi-Arch dependent interfaces" and are not safe to declare as `Multi-Arch: foreign`. The name `foreign` refers to the fact that the package can satisfy relations for native *and foreign* architectures at the same time. @@ -3189,7 +3578,7 @@ BINARY_FIELDS = _fields( Note: This value **cannot** be used with `Architecture: all`. - * `allowed` - **Advanced value**. This value is for a complex use-case that most people does not + * `allowed` - **Advanced value**. This value is for a complex use-case that most people do not need. Consider it only if none of the other values seem to do the trick. The package is **NOT** co-installable with itself but can satisfy Multi-Arch foreign and Multi-Arch same @@ -3416,6 +3805,17 @@ BINARY_FIELDS = _fields( """ ), ), + DctrlKnownField( + "XB-Ruby-Versions", + FieldValueClass.FREE_TEXT_FIELD, + deprecated_with_no_replacement=True, + synopsis_doc="Obsolete", + hover_text=textwrap.dedent( + """\ + Obsolete according to https://bugs.debian.org/1075762 + """ + ), + ), ) _DEP5_HEADER_FIELDS = _fields( Deb822KnownField( @@ -3598,6 +3998,57 @@ _DEP5_HEADER_FIELDS = _fields( """ ), ), + Deb822KnownField( + "Files-Excluded", + FieldValueClass.FREE_TEXT_FIELD, + hover_text=textwrap.dedent( + """\ + Remove the listed files from the tarball when repacking (commonly via uscan). This can be useful when the + listed files are non-free but not necessary for the Debian package. In this case, the upstream version of + the package should generally end with `~dfsg` or `+dfsg` (to mark the content changed due to the + Debian Free Software Guidelines). The exclusion can also be useful to remove large files or directories + that are not used by Debian or pre-built binaries. In this case, `~ds` or `+ds` should be added to the + version instead of `~dfsg` or `+dfsg` for "Debian Source" to mark it as altered by Debian. If both reasons + are used, the `~dfsg` or `+dfsg` version is used as that is the more important reason for the repacking. + + Example: + ``` + Files-Excluded: exclude-this + exclude-dir + */exclude-dir + .* + */js/jquery.js + ``` + + The `Files-Included` field can be used to "re-include" files matched by `Files-Excluded`. + + It is also possible to exclude files in specific "upstream components" for source packages with multiple + upstream tarballs. This is done by adding a field called `Files-Excluded-<component>`. The `<component>` + part should then match the component name exactly (case sensitive). + + Defined by: mk-origtargz (usually used via uscan) + """ + ), + ), + Deb822KnownField( + "Files-Included", + FieldValueClass.FREE_TEXT_FIELD, + hover_text=textwrap.dedent( + """\ + Re-include files that were marked for exclusion by `Files-Excluded`. This can be useful for "exclude + everything except X" style semantics where `Files-Excluded` has a very broad pattern and + `Files-Included` then marks a few exceptions. + + It is also possible to re-include files in specific "upstream components" for source packages with multiple + upstream tarballs. This is done by adding a field called `Files-Include-<component>` which is then used + in tandem with `Files-Exclude-<component>`. The `<component>` part should then match the component name + exactly (case sensitive). + + + Defined by: mk-origtargz (usually used via uscan) + """ + ), + ), ) _DEP5_FILES_FIELDS = _fields( Deb822KnownField( @@ -3905,7 +4356,8 @@ _DTESTSCTRL_FIELDS = _fields( FieldValueClass.COMMA_SEPARATED_LIST, default_value="@", synopsis_doc="Dependencies for running the tests", - hover_text="""\ + hover_text=textwrap.dedent( + """\ Declares that the specified packages must be installed for the test to go ahead. This supports all features of dpkg dependencies, including the architecture qualifiers (see @@ -3938,7 +4390,8 @@ _DTESTSCTRL_FIELDS = _fields( the source tree's Build-Dependencies are *not* necessarily installed, and if you specify any Depends, no binary packages from the source are installed unless explicitly requested. - """, + """ + ), ), DTestsCtrlKnownField( "Features", @@ -4357,7 +4810,7 @@ class StanzaMetadata(Mapping[str, F], Generic[F], ABC): def reformat_stanza( self, - effective_preference: "EffectivePreference", + effective_preference: "EffectiveFormattingPreference", stanza: Deb822ParagraphElement, stanza_range: TERange, formatter: FormatterCallback, @@ -4365,7 +4818,7 @@ class StanzaMetadata(Mapping[str, F], Generic[F], ABC): lines: List[str], ) -> Iterable[TextEdit]: for known_field in self.stanza_fields.values(): - kvpair = stanza.get_kvpair_element(known_field.name, use_get=True) + kvpair = stanza.get_kvpair_element((known_field.name, 0), use_get=True) if kvpair is None: continue yield from known_field.reformat_field( @@ -4437,7 +4890,7 @@ class Deb822FileMetadata(Generic[S]): def reformat( self, - effective_preference: "EffectivePreference", + effective_preference: "EffectiveFormattingPreference", deb822_file: Deb822FileElement, formatter: FormatterCallback, _content: str, @@ -4549,7 +5002,7 @@ class DctrlFileMetadata(Deb822FileMetadata[DctrlStanzaMetadata]): def reformat( self, - effective_preference: "EffectivePreference", + effective_preference: "EffectiveFormattingPreference", deb822_file: Deb822FileElement, formatter: FormatterCallback, content: str, @@ -4568,7 +5021,7 @@ class DctrlFileMetadata(Deb822FileMetadata[DctrlStanzaMetadata]): ) if ( - not effective_preference.formatting_deb822_normalize_stanza_order + not effective_preference.deb822_normalize_stanza_order or deb822_file.find_first_error_element() is not None ): return edits diff --git a/src/debputy/lsp/lsp_debian_copyright.py b/src/debputy/lsp/lsp_debian_copyright.py index 9cbac26..e5bcbff 100644 --- a/src/debputy/lsp/lsp_debian_copyright.py +++ b/src/debputy/lsp/lsp_debian_copyright.py @@ -184,9 +184,9 @@ def _diagnostics_for_paragraph( normalized_field_name_lc = normalize_dctrl_field_name(field_name_lc) known_field = known_fields.get(normalized_field_name_lc) field_value = stanza[field_name] - kvpair_position = kvpair.position_in_parent().relative_to(stanza_position) + kvpair_range_te = kvpair.range_in_parent().relative_to(stanza_position) field_range_te = kvpair.field_token.range_in_parent().relative_to( - kvpair_position + kvpair_range_te.start_pos ) field_position_te = field_range_te.start_pos field_range_server_units = te_range_to_lsp(field_range_te) @@ -268,7 +268,7 @@ def _diagnostics_for_paragraph( kvpair, stanza, stanza_position, - kvpair_position, + kvpair_range_te, lint_state, field_name_typo_reported=field_name_typo_detected, ) diff --git a/src/debputy/lsp/lsp_debian_debputy_manifest.py b/src/debputy/lsp/lsp_debian_debputy_manifest.py index 15e9aa6..8c7aeac 100644 --- a/src/debputy/lsp/lsp_debian_debputy_manifest.py +++ b/src/debputy/lsp/lsp_debian_debputy_manifest.py @@ -47,7 +47,7 @@ from debputy.lsprotocol.types import ( DiagnosticRelatedInformation, Location, ) -from debputy.manifest_parser.base_types import DebputyDispatchableType +from debputy.manifest_parser.tagging_types import DebputyDispatchableType from debputy.manifest_parser.declarative_parser import ( AttributeDescription, ParserGenerator, @@ -57,7 +57,6 @@ from debputy.manifest_parser.declarative_parser import DeclarativeMappingInputPa from debputy.manifest_parser.util import AttributePath from debputy.plugin.api.impl import plugin_metadata_for_debputys_own_plugin from debputy.plugin.api.impl_types import ( - OPARSER_MANIFEST_ROOT, DeclarativeInputParser, DispatchingParserBase, DebputyPluginMetadata, @@ -65,6 +64,7 @@ from debputy.plugin.api.impl_types import ( InPackageContextParser, DeclarativeValuelessKeywordInputParser, ) +from debputy.plugin.api.parser_tables import OPARSER_MANIFEST_ROOT from debputy.plugin.api.spec import DebputyIntegrationMode from debputy.plugin.debputy.private_api import Capability, load_libcap from debputy.util import _info, detect_possible_typo @@ -742,6 +742,22 @@ def _insert_snippet(lines: List[str], server_position: Position) -> bool: return True +def _maybe_quote(v: str) -> str: + if v and v[0].isdigit(): + try: + float(v) + return f"'{v}'" + except ValueError: + pass + return v + + +def _complete_value(v: Any) -> str: + if isinstance(v, str): + return _maybe_quote(v) + return str(v) + + @lsp_completer(_LANGUAGE_IDS) def debputy_manifest_completer( ls: "DebputyLanguageServer", @@ -752,7 +768,6 @@ def debputy_manifest_completer( server_position = doc.position_codec.position_from_client_units( lines, params.position ) - attribute_root_path = AttributePath.root_path() orig_line = lines[server_position.line].rstrip() has_colon = ":" in orig_line added_key = _insert_snippet(lines, server_position) @@ -791,6 +806,7 @@ def debputy_manifest_completer( context = lines[server_position.line].replace("\n", "\\n") _info(f"Completion failed: parse error: Line in question: {context}") return None + attribute_root_path = AttributePath.root_path(content) m = _trace_cursor(content, attribute_root_path, server_position) if m is None: @@ -820,7 +836,9 @@ def debputy_manifest_completer( if isinstance(parser, DispatchingParserBase): if matched_key: items = [ - CompletionItem(k if has_colon else f"{k}:") + CompletionItem( + _maybe_quote(k) if has_colon else f"{_maybe_quote(k)}:" + ) for k in parser.registered_keywords() if k not in parent and not isinstance( @@ -830,7 +848,7 @@ def debputy_manifest_completer( ] else: items = [ - CompletionItem(k) + CompletionItem(_maybe_quote(k)) for k in parser.registered_keywords() if k not in parent and isinstance( @@ -842,7 +860,9 @@ def debputy_manifest_completer( binary_packages = ls.lint_state(doc).binary_packages if binary_packages is not None: items = [ - CompletionItem(p if has_colon else f"{p}:") + CompletionItem( + _maybe_quote(p) if has_colon else f"{_maybe_quote(p)}:" + ) for p in binary_packages if p not in parent ] @@ -858,7 +878,9 @@ def debputy_manifest_completer( locked.add(attr_name) break items = [ - CompletionItem(k if has_colon else f"{k}:") + CompletionItem( + _maybe_quote(k) if has_colon else f"{_maybe_quote(k)}:" + ) for k in parser.manifest_attributes if k not in locked ] @@ -913,7 +935,7 @@ def _completion_from_attr( _info(f"Already filled: {matched} is one of {valid_values}") return None if valid_values: - return [CompletionItem(x) for x in valid_values] + return [CompletionItem(_complete_value(x)) for x in valid_values] return None @@ -925,13 +947,13 @@ def debputy_manifest_hover( doc = ls.workspace.get_text_document(params.text_document.uri) lines = doc.lines position_codec = doc.position_codec - attribute_root_path = AttributePath.root_path() server_position = position_codec.position_from_client_units(lines, params.position) try: content = MANIFEST_YAML.load("".join(lines)) except YAMLError: return None + attribute_root_path = AttributePath.root_path(content) m = _trace_cursor(content, attribute_root_path, server_position) if m is None: _info("No match") diff --git a/src/debputy/lsp/lsp_debian_tests_control.py b/src/debputy/lsp/lsp_debian_tests_control.py index f188762..ff0a209 100644 --- a/src/debputy/lsp/lsp_debian_tests_control.py +++ b/src/debputy/lsp/lsp_debian_tests_control.py @@ -199,9 +199,9 @@ def _diagnostics_for_paragraph( normalized_field_name_lc = normalize_dctrl_field_name(field_name_lc) known_field = known_fields.get(normalized_field_name_lc) field_value = stanza[field_name] - kvpair_position = kvpair.position_in_parent().relative_to(stanza_position) + kvpair_range_te = kvpair.range_in_parent().relative_to(stanza_position) field_range_te = kvpair.field_token.range_in_parent().relative_to( - kvpair_position + kvpair_range_te.start_pos ) field_position_te = field_range_te.start_pos field_range_server_units = te_range_to_lsp(field_range_te) @@ -268,7 +268,7 @@ def _diagnostics_for_paragraph( kvpair, stanza, stanza_position, - kvpair_position, + kvpair_range_te, lint_state, field_name_typo_reported=field_name_typo_detected, ) diff --git a/src/debputy/lsp/lsp_dispatch.py b/src/debputy/lsp/lsp_dispatch.py index 27170d0..b5e61dc 100644 --- a/src/debputy/lsp/lsp_dispatch.py +++ b/src/debputy/lsp/lsp_dispatch.py @@ -54,6 +54,8 @@ from debputy.lsprotocol.types import ( TEXT_DOCUMENT_WILL_SAVE_WAIT_UNTIL, WillSaveTextDocumentParams, TEXT_DOCUMENT_FORMATTING, + INITIALIZE, + InitializeParams, ) _DOCUMENT_VERSION_TABLE: Dict[str, int] = {} @@ -94,6 +96,14 @@ def is_doc_at_version(uri: str, version: int) -> bool: return dv == version +@DEBPUTY_LANGUAGE_SERVER.feature(INITIALIZE) +async def _on_initialize( + ls: "DebputyLanguageServer", + _: InitializeParams, +) -> None: + await ls.on_initialize() + + @DEBPUTY_LANGUAGE_SERVER.feature(TEXT_DOCUMENT_DID_OPEN) async def _open_document( ls: "DebputyLanguageServer", diff --git a/src/debputy/lsp/lsp_generic_deb822.py b/src/debputy/lsp/lsp_generic_deb822.py index 895a3e0..cde6e8b 100644 --- a/src/debputy/lsp/lsp_generic_deb822.py +++ b/src/debputy/lsp/lsp_generic_deb822.py @@ -311,6 +311,7 @@ def deb822_hover( custom_handler: Optional[ Callable[ [ + "DebputyLanguageServer", Position, Optional[str], str, @@ -351,6 +352,7 @@ def deb822_hover( hover_text = None if custom_handler is not None: res = custom_handler( + ls, server_pos, current_field, word_at_position, @@ -378,11 +380,17 @@ def deb822_hover( return None hover_text = keyword.hover_text if hover_text is not None: - hover_text = f"# Value `{keyword.value}` (Field: {known_field.name})\n\n{hover_text}" + header = "`{VALUE}` (Field: {FIELD_NAME})".format( + VALUE=keyword.value, + FIELD_NAME=known_field.name, + ) + hover_text = f"# {header})\n\n{hover_text}" else: hover_text = known_field.hover_text if hover_text is None: - hover_text = f"The field {current_field} had no documentation." + hover_text = ( + f"No documentation is available for the field {current_field}." + ) hover_text = f"# {known_field.name}\n\n{hover_text}" if hover_text is None: diff --git a/src/debputy/lsp/lsp_generic_yaml.py b/src/debputy/lsp/lsp_generic_yaml.py index 94267f7..5e67428 100644 --- a/src/debputy/lsp/lsp_generic_yaml.py +++ b/src/debputy/lsp/lsp_generic_yaml.py @@ -1,6 +1,6 @@ from typing import Union, Any, Optional, List, Tuple -from debputy.manifest_parser.base_types import DebputyDispatchableType +from debputy.manifest_parser.tagging_types import DebputyDispatchableType from debputy.manifest_parser.declarative_parser import DeclarativeMappingInputParser from debputy.manifest_parser.parser_doc import ( render_rule, diff --git a/src/debputy/lsp/lsp_reference_keyword.py b/src/debputy/lsp/lsp_reference_keyword.py index 4046aaa..0e16ac1 100644 --- a/src/debputy/lsp/lsp_reference_keyword.py +++ b/src/debputy/lsp/lsp_reference_keyword.py @@ -1,10 +1,23 @@ import dataclasses import textwrap -from typing import Optional, Union, Mapping, Sequence, Callable, Iterable +from typing import Optional, Union, Mapping, Sequence, Callable, Iterable, Literal from debputy.lsp.vendoring._deb822_repro import Deb822ParagraphElement +UsageHint = Literal["rare",] + + +def format_comp_item_synopsis_doc( + usage_hint: Optional[UsageHint], synopsis_doc: str, is_deprecated: bool +) -> str: + if is_deprecated: + return f"[OBSOLETE]: {synopsis_doc}" + if usage_hint is not None: + return f"[{usage_hint.upper()}]: {synopsis_doc}" + return synopsis_doc + + @dataclasses.dataclass(slots=True, frozen=True) class Keyword: value: str @@ -14,6 +27,7 @@ class Keyword: replaced_by: Optional[str] = None is_exclusive: bool = False sort_text: Optional[str] = None + usage_hint: Optional[UsageHint] = None can_complete_keyword_in_stanza: Optional[ Callable[[Iterable[Deb822ParagraphElement]], bool] ] = None @@ -22,6 +36,10 @@ class Keyword: value in `Architecture` of `debian/control` cannot be used with any other architecture. """ + @property + def is_deprecated(self) -> bool: + return self.is_obsolete or self.replaced_by is not None + def is_keyword_valid_completion_in_stanza( self, stanza_parts: Sequence[Deb822ParagraphElement], diff --git a/src/debputy/lsp/lsp_self_check.py b/src/debputy/lsp/lsp_self_check.py index aa28a56..ec0c7e7 100644 --- a/src/debputy/lsp/lsp_self_check.py +++ b/src/debputy/lsp/lsp_self_check.py @@ -110,7 +110,7 @@ def spell_checking() -> bool: @lsp_generic_check( - feature_name="Extra dh support", + feature_name="extra dh support", problem="Missing dependencies", how_to_fix="Run `apt satisfy debhelper (>= 13.16~)` to enable this feature", ) @@ -135,6 +135,30 @@ def check_dh_version() -> bool: return Version(parts[0]) >= Version("13.16~") +@lsp_generic_check( + feature_name="apt cache packages", + problem="Missing apt or empty apt cache", + how_to_fix="", +) +def check_apt_cache() -> bool: + try: + output = subprocess.check_output( + [ + "apt-get", + "indextargets", + "--format", + "$(IDENTIFIER)", + ] + ).decode("utf-8") + except (FileNotFoundError, subprocess.CalledProcessError): + return False + for line in output.splitlines(): + if line.strip() == "Packages": + return True + + return False + + def assert_can_start_lsp() -> None: for self_check in LSP_CHECKS: if self_check.is_mandatory and not self_check.test(): diff --git a/src/debputy/lsp/style-preferences.yaml b/src/debputy/lsp/maint-preferences.yaml index 982f242..982f242 100644 --- a/src/debputy/lsp/style-preferences.yaml +++ b/src/debputy/lsp/maint-preferences.yaml diff --git a/src/debputy/lsp/style_prefs.py b/src/debputy/lsp/maint_prefs.py index 1bcd800..4cc70d5 100644 --- a/src/debputy/lsp/style_prefs.py +++ b/src/debputy/lsp/maint_prefs.py @@ -23,46 +23,46 @@ from debputy.lsp.lsp_reference_keyword import ALL_PUBLIC_NAMED_STYLES from debputy.lsp.vendoring._deb822_repro.types import FormatterCallback from debputy.lsp.vendoring.wrap_and_sort import wrap_and_sort_formatter from debputy.packages import SourcePackage -from debputy.util import _error, _info +from debputy.util import _error from debputy.yaml import MANIFEST_YAML from debputy.yaml.compat import CommentedMap PT = TypeVar("PT", bool, str, int) -BUILTIN_STYLES = os.path.join(os.path.dirname(__file__), "style-preferences.yaml") +BUILTIN_STYLES = os.path.join(os.path.dirname(__file__), "maint-preferences.yaml") -_NORMALISE_FIELD_CONTENT_KEY = ["formatting", "deb822", "normalize-field-content"] +_NORMALISE_FIELD_CONTENT_KEY = ["deb822", "normalize-field-content"] _UPLOADER_SPLIT_RE = re.compile(r"(?<=>)\s*,") _WAS_OPTIONS = { - "-a": ("formatting_deb822_always_wrap", True), - "--always-wrap": ("formatting_deb822_always_wrap", True), - "-s": ("formatting_deb822_short_indent", True), - "--short-indent": ("formatting_deb822_short_indent", True), - "-t": ("formatting_deb822_trailing_separator", True), - "--trailing-separator": ("formatting_deb822_trailing_separator", True), + "-a": ("deb822_always_wrap", True), + "--always-wrap": ("deb822_always_wrap", True), + "-s": ("deb822_short_indent", True), + "--short-indent": ("deb822_short_indent", True), + "-t": ("deb822_trailing_separator", True), + "--trailing-separator": ("deb822_trailing_separator", True), # Noise option for us; we do not accept `--no-keep-first` though "-k": (None, True), "--keep-first": (None, True), "--no-keep-first": ("DISABLE_NORMALIZE_STANZA_ORDER", True), - "-b": ("formatting_deb822_normalize_stanza_order", True), - "--sort-binary-packages": ("formatting_deb822_normalize_stanza_order", True), + "-b": ("deb822_normalize_stanza_order", True), + "--sort-binary-packages": ("deb822_normalize_stanza_order", True), } _WAS_DEFAULTS = { - "formatting_deb822_always_wrap": False, - "formatting_deb822_short_indent": False, - "formatting_deb822_trailing_separator": False, - "formatting_deb822_normalize_stanza_order": False, - "formatting_deb822_normalize_field_content": True, + "deb822_always_wrap": False, + "deb822_short_indent": False, + "deb822_trailing_separator": False, + "deb822_normalize_stanza_order": False, + "deb822_normalize_field_content": True, } @dataclasses.dataclass(slots=True, frozen=True, kw_only=True) class PreferenceOption(Generic[PT]): key: Union[str, List[str]] - expected_type: Type[PT] + expected_type: Union[Type[PT], Callable[[Any], Optional[str]]] description: str default_value: Optional[Union[PT, Callable[[CommentedMap], Optional[PT]]]] = None @@ -88,11 +88,17 @@ class PreferenceOption(Generic[PT]): if callable(default_value): return default_value(data) return default_value - if isinstance(v, self.expected_type): + val_issue: Optional[str] = None + expected_type = self.expected_type + if not isinstance(expected_type, type) and callable(self.expected_type): + val_issue = self.expected_type(v) + elif not isinstance(v, self.expected_type): + val_issue = f"It should have been a {self.expected_type} but it was not" + + if val_issue is None: return v raise ValueError( - f'The value "{self.name}" for key {key} in file "{filename}" should have been a' - f" {self.expected_type} but it was not" + f'The value "{self.name}" for key {key} in file "{filename}" was incorrect: {val_issue}' ) @@ -108,7 +114,7 @@ def _false_when_formatting_content(m: CommentedMap) -> Optional[bool]: return m.mlget(_NORMALISE_FIELD_CONTENT_KEY, list_ok=True, default=False) is True -OPTIONS: List[PreferenceOption] = [ +MAINT_OPTIONS: List[PreferenceOption] = [ PreferenceOption( key="canonical-name", expected_type=str, @@ -139,7 +145,25 @@ OPTIONS: List[PreferenceOption] = [ ), ), PreferenceOption( - key=["formatting", "deb822", "short-indent"], + key="formatting", + expected_type=lambda x: ( + None + if isinstance(x, EffectiveFormattingPreference) + else "It should have been a EffectiveFormattingPreference but it was not" + ), + default_value=None, + description=textwrap.dedent( + """\ + The formatting preference of the maintainer. Can either be a string for a named style or an inline + style. + """ + ), + ), +] + +FORMATTING_OPTIONS = [ + PreferenceOption( + key=["deb822", "short-indent"], expected_type=bool, description=textwrap.dedent( """\ @@ -175,7 +199,7 @@ OPTIONS: List[PreferenceOption] = [ ), ), PreferenceOption( - key=["formatting", "deb822", "always-wrap"], + key=["deb822", "always-wrap"], expected_type=bool, description=textwrap.dedent( """\ @@ -210,7 +234,7 @@ OPTIONS: List[PreferenceOption] = [ ), ), PreferenceOption( - key=["formatting", "deb822", "trailing-separator"], + key=["deb822", "trailing-separator"], expected_type=bool, default_value=False, description=textwrap.dedent( @@ -241,7 +265,7 @@ OPTIONS: List[PreferenceOption] = [ ), ), PreferenceOption( - key=["formatting", "deb822", "max-line-length"], + key=["deb822", "max-line-length"], expected_type=int, default_value=79, description=textwrap.dedent( @@ -297,7 +321,7 @@ OPTIONS: List[PreferenceOption] = [ ), ), PreferenceOption( - key=["formatting", "deb822", "normalize-field-order"], + key=["deb822", "normalize-field-order"], expected_type=bool, default_value=False, description=textwrap.dedent( @@ -332,7 +356,7 @@ OPTIONS: List[PreferenceOption] = [ ), ), PreferenceOption( - key=["formatting", "deb822", "normalize-stanza-order"], + key=["deb822", "normalize-stanza-order"], expected_type=bool, default_value=False, description=textwrap.dedent( @@ -385,43 +409,43 @@ OPTIONS: List[PreferenceOption] = [ @dataclasses.dataclass(slots=True, frozen=True) -class EffectivePreference: - formatting_deb822_short_indent: Optional[bool] = None - formatting_deb822_always_wrap: Optional[bool] = None - formatting_deb822_trailing_separator: bool = False - formatting_deb822_normalize_field_content: bool = False - formatting_deb822_normalize_field_order: bool = False - formatting_deb822_normalize_stanza_order: bool = False - formatting_deb822_max_line_length: int = 79 +class EffectiveFormattingPreference: + deb822_short_indent: Optional[bool] = None + deb822_always_wrap: Optional[bool] = None + deb822_trailing_separator: bool = False + deb822_normalize_field_content: bool = False + deb822_normalize_field_order: bool = False + deb822_normalize_stanza_order: bool = False + deb822_max_line_length: int = 79 @classmethod def from_file( cls, filename: str, key: str, - stylees: CommentedMap, + styles: CommentedMap, ) -> Self: attr = {} - for option in OPTIONS: + for option in FORMATTING_OPTIONS: if not hasattr(cls, option.attribute_name): continue - value = option.extract_value(filename, key, stylees) + value = option.extract_value(filename, key, styles) attr[option.attribute_name] = value return cls(**attr) # type: ignore @classmethod def aligned_preference( cls, - a: Optional["EffectivePreference"], - b: Optional["EffectivePreference"], - ) -> Optional["EffectivePreference"]: + a: Optional["EffectiveFormattingPreference"], + b: Optional["EffectiveFormattingPreference"], + ) -> Optional["EffectiveFormattingPreference"]: if a is None or b is None: return None - for option in OPTIONS: + for option in MAINT_OPTIONS: attr_name = option.attribute_name - if not hasattr(EffectivePreference, attr_name): + if not hasattr(EffectiveFormattingPreference, attr_name): continue a_value = getattr(a, attr_name) b_value = getattr(b, attr_name) @@ -430,14 +454,12 @@ class EffectivePreference: return a def deb822_formatter(self) -> FormatterCallback: - line_length = self.formatting_deb822_max_line_length + line_length = self.deb822_max_line_length return wrap_and_sort_formatter( - 1 if self.formatting_deb822_short_indent else "FIELD_NAME_LENGTH", - trailing_separator=self.formatting_deb822_trailing_separator, - immediate_empty_line=self.formatting_deb822_short_indent or False, - max_line_length_one_liner=( - 0 if self.formatting_deb822_always_wrap else line_length - ), + 1 if self.deb822_short_indent else "FIELD_NAME_LENGTH", + trailing_separator=self.deb822_trailing_separator, + immediate_empty_line=self.deb822_short_indent or False, + max_line_length_one_liner=(0 if self.deb822_always_wrap else line_length), ) def replace(self, /, **changes: Any) -> Self: @@ -445,32 +467,41 @@ class EffectivePreference: @dataclasses.dataclass(slots=True, frozen=True) -class MaintainerPreference(EffectivePreference): +class MaintainerPreference: canonical_name: Optional[str] = None is_packaging_team: bool = False + formatting: Optional[EffectiveFormattingPreference] = None - def as_effective_pref(self) -> EffectivePreference: - fields = { - k: v - for k, v in dataclasses.asdict(self).items() - if hasattr(EffectivePreference, k) - } - return EffectivePreference(**fields) + @classmethod + def from_file( + cls, + filename: str, + key: str, + styles: CommentedMap, + ) -> Self: + attr = {} + + for option in MAINT_OPTIONS: + if not hasattr(cls, option.attribute_name): + continue + value = option.extract_value(filename, key, styles) + attr[option.attribute_name] = value + return cls(**attr) # type: ignore -class StylePreferenceTable: +class MaintainerPreferenceTable: def __init__( self, - named_styles: Mapping[str, EffectivePreference], + named_styles: Mapping[str, EffectiveFormattingPreference], maintainer_preferences: Mapping[str, MaintainerPreference], ) -> None: self._named_styles = named_styles self._maintainer_preferences = maintainer_preferences @classmethod - def load_styles(cls) -> Self: - named_styles: Dict[str, EffectivePreference] = {} + def load_preferences(cls) -> Self: + named_styles: Dict[str, EffectiveFormattingPreference] = {} maintainer_preferences: Dict[str, MaintainerPreference] = {} with open(BUILTIN_STYLES) as fd: parse_file(named_styles, maintainer_preferences, BUILTIN_STYLES, fd) @@ -488,7 +519,7 @@ class StylePreferenceTable: return cls(named_styles, maintainer_preferences) @property - def named_styles(self) -> Mapping[str, EffectivePreference]: + def named_styles(self) -> Mapping[str, EffectiveFormattingPreference]: return self._named_styles @property @@ -497,7 +528,7 @@ class StylePreferenceTable: def parse_file( - named_styles: Dict[str, EffectivePreference], + named_styles: Dict[str, EffectiveFormattingPreference], maintainer_preferences: Dict[str, MaintainerPreference], filename: str, fd, @@ -520,38 +551,45 @@ def parse_file( named_styles_raw = {} for style_name, content in named_styles_raw.items(): - wrapped_style = CommentedMap({"formatting": content}) - style = EffectivePreference.from_file( + style = EffectiveFormattingPreference.from_file( filename, style_name, - wrapped_style, + content, ) named_styles[style_name] = style - for maintainer_email, maintainer_styles in maintainer_rules.items(): - if not isinstance(maintainer_styles, CommentedMap): + for maintainer_email, maintainer_pref in maintainer_rules.items(): + if not isinstance(maintainer_pref, CommentedMap): line_no = maintainer_rules.lc.key(maintainer_email).line raise ValueError( f'The value for maintainer "{maintainer_email}" should have been a mapping,' f' but it is not. The problem entry is at line {line_no} in "{filename}"' ) - formatting = maintainer_styles.get("formatting") + formatting = maintainer_pref.get("formatting") if isinstance(formatting, str): try: - style = named_styles_raw[formatting] + style = named_styles[formatting] except KeyError: line_no = maintainer_rules.lc.key(maintainer_email).line raise ValueError( f'The maintainer "{maintainer_email}" requested the named style "{formatting}",' f' but said style was not defined {filename}. The problem entry is at line {line_no} in "{filename}"' ) from None - maintainer_styles["formatting"] = style - maintainer_preferences[maintainer_email] = MaintainerPreference.from_file( + maintainer_pref["formatting"] = style + elif formatting is not None: + maintainer_pref["formatting"] = EffectiveFormattingPreference.from_file( + filename, + "formatting", + formatting, + ) + mp = MaintainerPreference.from_file( filename, maintainer_email, - maintainer_styles, + maintainer_pref, ) + maintainer_preferences[maintainer_email] = mp + @functools.lru_cache(64) def extract_maint_email(maint: str) -> str: @@ -565,16 +603,16 @@ def extract_maint_email(maint: str) -> str: return maint[idx + 1 : -1] -def determine_effective_style( - style_preference_table: StylePreferenceTable, +def determine_effective_preference( + maint_preference_table: MaintainerPreferenceTable, source_package: Optional[SourcePackage], salsa_ci: Optional[CommentedMap], -) -> Tuple[Optional[EffectivePreference], Optional[str]]: +) -> Tuple[Optional[EffectiveFormattingPreference], Optional[str], Optional[str]]: style = source_package.fields.get("X-Style") if source_package is not None else None if style is not None: if style not in ALL_PUBLIC_NAMED_STYLES: - return None, "X-Style contained an unknown/unsupported style" - return style_preference_table.named_styles.get(style), None + return None, None, "X-Style contained an unknown/unsupported style" + return maint_preference_table.named_styles.get(style), "debputy reformat", None if salsa_ci: disable_wrap_and_sort = salsa_ci.mlget( @@ -600,53 +638,69 @@ def determine_effective_style( if wrap_and_sort_options is None: wrap_and_sort_options = "" elif not isinstance(wrap_and_sort_options, str): - return None, "The salsa-ci had a non-string option for wrap-and-sort" + return ( + None, + None, + "The salsa-ci had a non-string option for wrap-and-sort", + ) detected_style = parse_salsa_ci_wrap_and_sort_args(wrap_and_sort_options) + tool_w_args = f"wrap-and-sort {wrap_and_sort_options}".strip() if detected_style is None: msg = "One or more of the wrap-and-sort options in the salsa-ci file was not supported" else: msg = None - return detected_style, msg - + return detected_style, tool_w_args, msg if source_package is None: - return None, None + return None, None, None maint = source_package.fields.get("Maintainer") if maint is None: - return None, None + return None, None, None maint_email = extract_maint_email(maint) - maint_style = style_preference_table.maintainer_preferences.get(maint_email) + maint_pref = maint_preference_table.maintainer_preferences.get(maint_email) # Special-case "@packages.debian.org" when missing, since they are likely to be "ad-hoc" # teams that will not be registered. In that case, we fall back to looking at the uploader # preferences as-if the maintainer had not been listed at all. - if maint_style is None and not maint_email.endswith("@packages.debian.org"): - return None, None - if maint_style is not None and maint_style.is_packaging_team: + if maint_pref is None and not maint_email.endswith("@packages.debian.org"): + return None, None, None + if maint_pref is not None and maint_pref.is_packaging_team: # When the maintainer is registered as a packaging team, then we assume the packaging # team's style applies unconditionally. - return maint_style.as_effective_pref(), None + effective = maint_pref.formatting + tool_w_args = _guess_tool_from_style(maint_preference_table, effective) + return effective, tool_w_args, None uploaders = source_package.fields.get("Uploaders") if uploaders is None: - detected_style = ( - maint_style.as_effective_pref() if maint_style is not None else None - ) - return detected_style, None - all_styles: List[Optional[EffectivePreference]] = [] - if maint_style is not None: - all_styles.append(maint_style) + detected_style = maint_pref.formatting if maint_pref is not None else None + tool_w_args = _guess_tool_from_style(maint_preference_table, detected_style) + return detected_style, tool_w_args, None + all_styles: List[Optional[EffectiveFormattingPreference]] = [] + if maint_pref is not None: + all_styles.append(maint_pref.formatting) for uploader in _UPLOADER_SPLIT_RE.split(uploaders): uploader_email = extract_maint_email(uploader) - uploader_style = style_preference_table.maintainer_preferences.get( + uploader_pref = maint_preference_table.maintainer_preferences.get( uploader_email ) - all_styles.append(uploader_style) + all_styles.append(uploader_pref.formatting if uploader_pref else None) if not all_styles: - return None, None - r = functools.reduce(EffectivePreference.aligned_preference, all_styles) - if isinstance(r, MaintainerPreference): - return r.as_effective_pref(), None - return r, None + return None, None, None + r = functools.reduce(EffectiveFormattingPreference.aligned_preference, all_styles) + assert not isinstance(r, MaintainerPreference) + tool_w_args = _guess_tool_from_style(maint_preference_table, r) + return r, tool_w_args, None + + +def _guess_tool_from_style( + maint_preference_table: MaintainerPreferenceTable, + pref: Optional[EffectiveFormattingPreference], +) -> Optional[str]: + if pref is None: + return None + if maint_preference_table.named_styles["black"] == pref: + return "debputy reformat" + return None def _split_options(args: Iterable[str]) -> Iterable[str]: @@ -662,7 +716,9 @@ def _split_options(args: Iterable[str]) -> Iterable[str]: @functools.lru_cache -def parse_salsa_ci_wrap_and_sort_args(args: str) -> Optional[EffectivePreference]: +def parse_salsa_ci_wrap_and_sort_args( + args: str, +) -> Optional[EffectiveFormattingPreference]: options = dict(_WAS_DEFAULTS) for arg in _split_options(args.split()): v = _WAS_OPTIONS.get(arg) @@ -674,6 +730,6 @@ def parse_salsa_ci_wrap_and_sort_args(args: str) -> Optional[EffectivePreference options[varname] = value if "DISABLE_NORMALIZE_STANZA_ORDER" in options: del options["DISABLE_NORMALIZE_STANZA_ORDER"] - options["formatting_deb822_normalize_stanza_order"] = False + options["deb822_normalize_stanza_order"] = False - return EffectivePreference(**options) # type: ignore + return EffectiveFormattingPreference(**options) # type: ignore diff --git a/src/debputy/maintscript_snippet.py b/src/debputy/maintscript_snippet.py index ca81ca5..58a6bba 100644 --- a/src/debputy/maintscript_snippet.py +++ b/src/debputy/maintscript_snippet.py @@ -1,7 +1,7 @@ import dataclasses from typing import Sequence, Optional, List, Literal, Iterable, Dict, Self -from debputy.manifest_parser.base_types import DebputyDispatchableType +from debputy.manifest_parser.tagging_types import DebputyDispatchableType from debputy.manifest_parser.util import AttributePath STD_CONTROL_SCRIPTS = frozenset( diff --git a/src/debputy/manifest_conditions.py b/src/debputy/manifest_conditions.py index 0f5c298..3e97b00 100644 --- a/src/debputy/manifest_conditions.py +++ b/src/debputy/manifest_conditions.py @@ -1,12 +1,12 @@ import dataclasses from enum import Enum -from typing import List, Callable, Optional, Sequence +from typing import List, Callable, Optional, Sequence, Any, Self, Mapping from debian.debian_support import DpkgArchTable from debputy._deb_options_profiles import DebBuildOptionsAndProfiles from debputy.architecture_support import DpkgArchitectureBuildProcessValuesTable -from debputy.manifest_parser.base_types import DebputyDispatchableType +from debputy.manifest_parser.tagging_types import DebputyDispatchableType from debputy.packages import BinaryPackage from debputy.substitution import Substitution from debputy.util import active_profiles_match @@ -15,11 +15,14 @@ from debputy.util import active_profiles_match @dataclasses.dataclass(slots=True, frozen=True) class ConditionContext: binary_package: Optional[BinaryPackage] - build_env: DebBuildOptionsAndProfiles + deb_options_and_profiles: DebBuildOptionsAndProfiles substitution: Substitution dpkg_architecture_variables: DpkgArchitectureBuildProcessValuesTable dpkg_arch_query_table: DpkgArchTable + def replace(self, /, **changes: Any) -> "Self": + return dataclasses.replace(self, **changes) + class ManifestCondition(DebputyDispatchableType): __slots__ = () @@ -72,6 +75,7 @@ class NegatedManifestCondition(ManifestCondition): __slots__ = ("_condition",) def __init__(self, condition: ManifestCondition) -> None: + super().__init__() self._condition = condition def negated(self) -> "ManifestCondition": @@ -107,6 +111,7 @@ class ManifestConditionGroup(ManifestCondition): match_type: _ConditionGroupMatchType, conditions: Sequence[ManifestCondition], ) -> None: + super().__init__() self.match_type = match_type self._conditions = conditions @@ -132,6 +137,7 @@ class ArchMatchManifestConditionBase(ManifestCondition): __slots__ = ("_arch_spec", "_is_negated") def __init__(self, arch_spec: List[str], *, is_negated: bool = False) -> None: + super().__init__() self._arch_spec = arch_spec self._is_negated = is_negated @@ -177,6 +183,7 @@ class BuildProfileMatch(ManifestCondition): __slots__ = ("_profile_spec", "_is_negated") def __init__(self, profile_spec: str, *, is_negated: bool = False) -> None: + super().__init__() self._profile_spec = profile_spec self._is_negated = is_negated @@ -190,7 +197,7 @@ class BuildProfileMatch(ManifestCondition): def evaluate(self, context: ConditionContext) -> bool: match = active_profiles_match( - self._profile_spec, context.build_env.deb_build_profiles + self._profile_spec, context.deb_options_and_profiles.deb_build_profiles ) return not match if self._is_negated else match @@ -211,7 +218,14 @@ def _can_run_built_binaries(context: ConditionContext) -> bool: if not context.dpkg_architecture_variables.is_cross_compiling: return True # User / Builder asserted that we could even though we are cross-compiling, so we have to assume it is true - return "crossbuildcanrunhostbinaries" in context.build_env.deb_build_options + return ( + "crossbuildcanrunhostbinaries" + in context.deb_options_and_profiles.deb_build_options + ) + + +def _run_build_time_tests(deb_build_options: Mapping[str, Optional[str]]) -> bool: + return "nocheck" not in deb_build_options _IS_CROSS_BUILDING = _SingletonCondition( @@ -226,12 +240,12 @@ _CAN_EXECUTE_COMPILED_BINARIES = _SingletonCondition( _RUN_BUILD_TIME_TESTS = _SingletonCondition( "Run build time tests", - lambda c: "nocheck" not in c.build_env.deb_build_options, + lambda c: _run_build_time_tests(c.deb_options_and_profiles.deb_build_options), ) _BUILD_DOCS_BDO = _SingletonCondition( "Build docs (nodocs not in DEB_BUILD_OPTIONS)", - lambda c: "nodocs" not in c.build_env.deb_build_options, + lambda c: "nodocs" not in c.deb_options_and_profiles.deb_build_options, ) diff --git a/src/debputy/manifest_parser/base_types.py b/src/debputy/manifest_parser/base_types.py index 865e320..106c30e 100644 --- a/src/debputy/manifest_parser/base_types.py +++ b/src/debputy/manifest_parser/base_types.py @@ -1,9 +1,8 @@ import dataclasses import os +import subprocess from functools import lru_cache from typing import ( - TypedDict, - NotRequired, Sequence, Optional, Union, @@ -12,12 +11,14 @@ from typing import ( Mapping, Iterable, TYPE_CHECKING, - Callable, - Type, - Generic, + Dict, + MutableMapping, + NotRequired, ) +from debputy.manifest_conditions import ManifestCondition from debputy.manifest_parser.exceptions import ManifestParseException +from debputy.manifest_parser.tagging_types import DebputyParsedContent from debputy.manifest_parser.util import ( AttributePath, _SymbolicModeSegment, @@ -25,37 +26,20 @@ from debputy.manifest_parser.util import ( ) from debputy.path_matcher import MatchRule, ExactFileSystemPath from debputy.substitution import Substitution -from debputy.types import S -from debputy.util import _normalize_path, T +from debputy.util import _normalize_path, _error, _warn, _debug_log if TYPE_CHECKING: - from debputy.manifest_conditions import ManifestCondition from debputy.manifest_parser.parser_data import ParserContextData -class DebputyParsedContent(TypedDict): - pass - - -class DebputyDispatchableType: - __slots__ = () - - -class DebputyParsedContentStandardConditional(DebputyParsedContent): - when: NotRequired["ManifestCondition"] - - @dataclasses.dataclass(slots=True, frozen=True) class OwnershipDefinition: entity_name: str entity_id: int -@dataclasses.dataclass -class TypeMapping(Generic[S, T]): - target_type: Type[T] - source_type: Type[S] - mapper: Callable[[S, AttributePath, Optional["ParserContextData"]], T] +class DebputyParsedContentStandardConditional(DebputyParsedContent): + when: NotRequired[ManifestCondition] ROOT_DEFINITION = OwnershipDefinition("root", 0) @@ -438,3 +422,76 @@ class FileSystemExactMatchRule(FileSystemMatchRule): class FileSystemExactNonDirMatchRule(FileSystemExactMatchRule): pass + + +class BuildEnvironmentDefinition: + + def dpkg_buildflags_env( + self, + env: Mapping[str, str], + definition_source: Optional[str], + ) -> Dict[str, str]: + dpkg_env = {} + try: + bf_output = subprocess.check_output(["dpkg-buildflags"], env=env) + except FileNotFoundError: + if definition_source is None: + _error( + "The dpkg-buildflags command was not available and is necessary to set the relevant" + "env variables by default." + ) + _error( + "The dpkg-buildflags command was not available and is necessary to set the relevant" + f"env variables for the environment defined at {definition_source}." + ) + except subprocess.CalledProcessError as e: + if definition_source is None: + _error( + f"The dpkg-buildflags command failed with exit code {e.returncode}. Please review the output from" + f" dpkg-buildflags above to resolve the issue." + ) + _error( + f"The dpkg-buildflags command failed with exit code {e.returncode}. Please review the output from" + f" dpkg-buildflags above to resolve the issue. The environment definition that triggered this call" + f" was {definition_source}" + ) + else: + warned = False + for line in bf_output.decode("utf-8").splitlines(keepends=False): + if "=" not in line or line.startswith("="): + if not warned: + _warn( + f"Unexpected output from dpkg-buildflags (not a K=V line): {line}" + ) + continue + k, v = line.split("=", 1) + if k.strip() != k: + if not warned: + _warn( + f'Unexpected output from dpkg-buildflags (Key had spaces): "{line}"' + ) + continue + dpkg_env[k] = v + + return dpkg_env + + def log_computed_env(self, source: str, computed_env: Mapping[str, str]) -> None: + _debug_log(f"Computed environment variables from {source}") + for k, v in computed_env.items(): + _debug_log(f" {k}={v}") + + def update_env(self, env: MutableMapping[str, str]) -> None: + dpkg_env = self.dpkg_buildflags_env(env, None) + self.log_computed_env("dpkg-buildflags", dpkg_env) + env.update(dpkg_env) + + +class BuildEnvironments: + + def __init__( + self, + environments: Dict[str, BuildEnvironmentDefinition], + default_environment: Optional[BuildEnvironmentDefinition], + ) -> None: + self.environments = environments + self.default_environment = default_environment diff --git a/src/debputy/manifest_parser/declarative_parser.py b/src/debputy/manifest_parser/declarative_parser.py index 4a32368..2c350a0 100644 --- a/src/debputy/manifest_parser/declarative_parser.py +++ b/src/debputy/manifest_parser/declarative_parser.py @@ -1,5 +1,6 @@ import collections import dataclasses +import typing from typing import ( Any, Callable, @@ -16,7 +17,6 @@ from typing import ( Mapping, Optional, cast, - is_typeddict, Type, Union, List, @@ -28,13 +28,7 @@ from typing import ( Container, ) -from debputy.manifest_parser.base_types import ( - DebputyParsedContent, - FileSystemMatchRule, - FileSystemExactMatchRule, - DebputyDispatchableType, - TypeMapping, -) +from debputy.manifest_parser.base_types import FileSystemMatchRule from debputy.manifest_parser.exceptions import ( ManifestParseException, ) @@ -43,7 +37,20 @@ from debputy.manifest_parser.mapper_code import ( wrap_into_list, map_each_element, ) +from debputy.manifest_parser.parse_hints import ( + ConditionalRequired, + DebputyParseHint, + TargetAttribute, + ManifestAttribute, + ConflictWithSourceAttribute, + NotPathHint, +) from debputy.manifest_parser.parser_data import ParserContextData +from debputy.manifest_parser.tagging_types import ( + DebputyParsedContent, + DebputyDispatchableType, + TypeMapping, +) from debputy.manifest_parser.util import ( AttributePath, unpack_type, @@ -53,8 +60,6 @@ from debputy.manifest_parser.util import ( from debputy.plugin.api.impl_types import ( DeclarativeInputParser, TD, - _ALL_PACKAGE_TYPES, - resolve_package_type_selectors, ListWrappedDeclarativeInputParser, DispatchingObjectParser, DispatchingTableParser, @@ -64,8 +69,11 @@ from debputy.plugin.api.impl_types import ( ) from debputy.plugin.api.spec import ( ParserDocumentation, - PackageTypeSelector, DebputyIntegrationMode, + StandardParserAttributeDocumentation, + undocumented_attr, + ParserAttributeDocumentation, + reference_documentation, ) from debputy.util import _info, _warn, assume_not_none @@ -349,16 +357,16 @@ class DeclarativeMappingInputParser(DeclarativeInputParser[TD], Generic[TD, SF]) if unused_keys: k = ", ".join(unused_keys) raise ManifestParseException( - f'Unknown keys "{unknown_keys}" at {path.path}". Keys that could be used here are: {k}.{doc_ref}' + f'Unknown keys "{unknown_keys}" at {path.path_container_lc}". Keys that could be used here are: {k}.{doc_ref}' ) raise ManifestParseException( - f'Unknown keys "{unknown_keys}" at {path.path}". Please remove them.{doc_ref}' + f'Unknown keys "{unknown_keys}" at {path.path_container_lc}". Please remove them.{doc_ref}' ) missing_keys = self.input_time_required_parameters - value.keys() if missing_keys: required = ", ".join(repr(k) for k in sorted(missing_keys)) raise ManifestParseException( - f"The following keys were required but not present at {path.path}: {required}{doc_ref}" + f"The following keys were required but not present at {path.path_container_lc}: {required}{doc_ref}" ) for maybe_required in self.all_parameters - value.keys(): attr = self.manifest_attributes[maybe_required] @@ -371,14 +379,14 @@ class DeclarativeMappingInputParser(DeclarativeInputParser[TD], Generic[TD, SF]) ): reason = attr.conditional_required.reason raise ManifestParseException( - f'Missing the *conditionally* required attribute "{maybe_required}" at {path.path}. {reason}{doc_ref}' + f'Missing the *conditionally* required attribute "{maybe_required}" at {path.path_container_lc}. {reason}{doc_ref}' ) for keyset in self.at_least_one_of: matched_keys = value.keys() & keyset if not matched_keys: conditionally_required = ", ".join(repr(k) for k in sorted(keyset)) raise ManifestParseException( - f"At least one of the following keys must be present at {path.path}:" + f"At least one of the following keys must be present at {path.path_container_lc}:" f" {conditionally_required}{doc_ref}" ) for group in self.mutually_exclusive_attributes: @@ -386,7 +394,7 @@ class DeclarativeMappingInputParser(DeclarativeInputParser[TD], Generic[TD, SF]) if len(matched) > 1: ck = ", ".join(repr(k) for k in sorted(matched)) raise ManifestParseException( - f"Could not parse {path.path}: The following attributes are" + f"Could not parse {path.path_container_lc}: The following attributes are" f" mutually exclusive: {ck}{doc_ref}" ) @@ -478,242 +486,6 @@ class DeclarativeMappingInputParser(DeclarativeInputParser[TD], Generic[TD, SF]) return self._per_attribute_conflicts_cache -class DebputyParseHint: - @classmethod - def target_attribute(cls, target_attribute: str) -> "DebputyParseHint": - """Define this source attribute to have a different target attribute name - - As an example: - - >>> class SourceType(TypedDict): - ... source: Annotated[NotRequired[str], DebputyParseHint.target_attribute("sources")] - ... sources: NotRequired[List[str]] - >>> class TargetType(TypedDict): - ... sources: List[str] - >>> pg = ParserGenerator() - >>> parser = pg.generate_parser(TargetType, source_content=SourceType) - - In this example, the user can provide either `source` or `sources` and the parser will - map them to the `sources` attribute in the `TargetType`. Note this example relies on - the builtin mapping of `str` to `List[str]` to align the types between `source` (from - SourceType) and `sources` (from TargetType). - - The following rules apply: - - * All source attributes that map to the same target attribute will be mutually exclusive - (that is, the user cannot give `source` *and* `sources` as input). - * When the target attribute is required, the source attributes are conditionally - mandatory requiring the user to provide exactly one of them. - * When multiple source attributes point to a single target attribute, none of the source - attributes can be Required. - * The annotation can only be used for the source type specification and the source type - specification must be different from the target type specification. - - The `target_attribute` annotation can be used without having multiple source attributes. This - can be useful if the source attribute name is not valid as a python variable identifier to - rename it to a valid python identifier. - - :param target_attribute: The attribute name in the target content - :return: The annotation. - """ - return TargetAttribute(target_attribute) - - @classmethod - def conflicts_with_source_attributes( - cls, - *conflicting_source_attributes: str, - ) -> "DebputyParseHint": - """Declare a conflict with one or more source attributes - - Example: - - >>> class SourceType(TypedDict): - ... source: Annotated[NotRequired[str], DebputyParseHint.target_attribute("sources")] - ... sources: NotRequired[List[str]] - ... into_dir: NotRequired[str] - ... renamed_to: Annotated[ - ... NotRequired[str], - ... DebputyParseHint.conflicts_with_source_attributes("sources", "into_dir") - ... ] - >>> class TargetType(TypedDict): - ... sources: List[str] - ... into_dir: NotRequired[str] - ... renamed_to: NotRequired[str] - >>> pg = ParserGenerator() - >>> parser = pg.generate_parser(TargetType, source_content=SourceType) - - In this example, if the user was to provide `renamed_to` with `sources` or `into_dir` the parser would report - an error. However, the parser will allow `renamed_to` with `source` as the conflict is considered only for - the input source. That is, it is irrelevant that `sources` and `source´ happens to "map" to the same target - attribute. - - The following rules apply: - * It is not possible for a target attribute to declare conflicts unless the target type spec is reused as - source type spec. - * All attributes involved in a conflict must be NotRequired. If any of the attributes are Required, then - the parser generator will reject the input. - * All attributes listed in the conflict must be valid attributes in the source type spec. - - Note you do not have to specify conflicts between two attributes with the same target attribute name. The - `target_attribute` annotation will handle that for you. - - :param conflicting_source_attributes: All source attributes that cannot be used with this attribute. - :return: The annotation. - """ - if len(conflicting_source_attributes) < 1: - raise ValueError( - "DebputyParseHint.conflicts_with_source_attributes requires at least one attribute as input" - ) - return ConflictWithSourceAttribute(frozenset(conflicting_source_attributes)) - - @classmethod - def required_when_single_binary( - cls, - *, - package_type: PackageTypeSelector = _ALL_PACKAGE_TYPES, - ) -> "DebputyParseHint": - """Declare a source attribute as required when the source package produces exactly one binary package - - The attribute in question must always be declared as `NotRequired` in the TypedDict and this condition - can only be used for source attributes. - """ - resolved_package_types = resolve_package_type_selectors(package_type) - reason = "The field is required for source packages producing exactly one binary package" - if resolved_package_types != _ALL_PACKAGE_TYPES: - types = ", ".join(sorted(resolved_package_types)) - reason += f" of type {types}" - return ConditionalRequired( - reason, - lambda c: len( - [ - p - for p in c.binary_packages.values() - if p.package_type in package_type - ] - ) - == 1, - ) - return ConditionalRequired( - reason, - lambda c: c.is_single_binary_package, - ) - - @classmethod - def required_when_multi_binary( - cls, - *, - package_type: PackageTypeSelector = _ALL_PACKAGE_TYPES, - ) -> "DebputyParseHint": - """Declare a source attribute as required when the source package produces two or more binary package - - The attribute in question must always be declared as `NotRequired` in the TypedDict and this condition - can only be used for source attributes. - """ - resolved_package_types = resolve_package_type_selectors(package_type) - reason = "The field is required for source packages producing two or more binary packages" - if resolved_package_types != _ALL_PACKAGE_TYPES: - types = ", ".join(sorted(resolved_package_types)) - reason = ( - "The field is required for source packages producing not producing exactly one binary packages" - f" of type {types}" - ) - return ConditionalRequired( - reason, - lambda c: len( - [ - p - for p in c.binary_packages.values() - if p.package_type in package_type - ] - ) - != 1, - ) - return ConditionalRequired( - reason, - lambda c: not c.is_single_binary_package, - ) - - @classmethod - def manifest_attribute(cls, attribute: str) -> "DebputyParseHint": - """Declare what the attribute name (as written in the manifest) should be - - By default, debputy will do an attribute normalizing that will take valid python identifiers such - as `dest_dir` and remap it to the manifest variant (such as `dest-dir`) automatically. If you have - a special case, where this built-in normalization is insufficient or the python name is considerably - different from what the user would write in the manifest, you can use this parse hint to set the - name that the user would have to write in the manifest for this attribute. - - >>> class SourceType(TypedDict): - ... source: List[FileSystemMatchRule] - ... # Use "as" in the manifest because "as_" was not pretty enough - ... install_as: Annotated[NotRequired[FileSystemExactMatchRule], DebputyParseHint.manifest_attribute("as")] - - In this example, we use the parse hint to use "as" as the name in the manifest, because we cannot - use "as" a valid python identifier (it is a keyword). While debputy would map `as_` to `as` for us, - we have chosen to use `install_as` as a python identifier. - """ - return ManifestAttribute(attribute) - - @classmethod - def not_path_error_hint(cls) -> "DebputyParseHint": - """Mark this attribute as not a "path hint" when it comes to reporting errors - - By default, `debputy` will pick up attributes that uses path names (FileSystemMatchRule) as - candidates for parse error hints (the little "<Search for: VALUE>" in error messages). - - Most rules only have one active path-based attribute and paths tends to be unique enough - that it helps people spot the issue faster. However, in rare cases, you can have multiple - attributes that fit the bill. In this case, this hint can be used to "hide" the suboptimal - choice. As an example: - - >>> class SourceType(TypedDict): - ... source: List[FileSystemMatchRule] - ... install_as: Annotated[NotRequired[FileSystemExactMatchRule], DebputyParseHint.not_path_error_hint()] - - In this case, without the hint, `debputy` might pick up `install_as` as the attribute to - use as hint for error reporting. However, here we have decided that we never want `install_as` - leaving `source` as the only option. - - Generally, this type hint must be placed on the **source** format. Any source attribute matching - the parsed format will be ignored. - - Mind the asymmetry: The annotation is placed in the **source** format while `debputy` looks at - the type of the target attribute to determine if it counts as path. - """ - return NOT_PATH_HINT - - -@dataclasses.dataclass(frozen=True, slots=True) -class TargetAttribute(DebputyParseHint): - attribute: str - - -@dataclasses.dataclass(frozen=True, slots=True) -class ConflictWithSourceAttribute(DebputyParseHint): - conflicting_attributes: FrozenSet[str] - - -@dataclasses.dataclass(frozen=True, slots=True) -class ConditionalRequired(DebputyParseHint): - reason: str - condition: Callable[["ParserContextData"], bool] - - def condition_applies(self, context: "ParserContextData") -> bool: - return self.condition(context) - - -@dataclasses.dataclass(frozen=True, slots=True) -class ManifestAttribute(DebputyParseHint): - attribute: str - - -class NotPathHint(DebputyParseHint): - pass - - -NOT_PATH_HINT = NotPathHint() - - def _is_path_attribute_candidate( source_attribute: AttributeDescription, target_attribute: AttributeDescription ) -> bool: @@ -730,6 +502,16 @@ def _is_path_attribute_candidate( return isinstance(match_type, type) and issubclass(match_type, FileSystemMatchRule) +if typing.is_typeddict(DebputyParsedContent): + is_typeddict = typing.is_typeddict +else: + + def is_typeddict(t: Any) -> bool: + if typing.is_typeddict(t): + return True + return isinstance(t, type) and issubclass(t, DebputyParsedContent) + + class ParserGenerator: def __init__(self) -> None: self._registered_types: Dict[Any, TypeMapping[Any, Any]] = {} @@ -811,6 +593,9 @@ class ParserGenerator: expected_debputy_integration_mode: Optional[ Container[DebputyIntegrationMode] ] = None, + automatic_docs: Optional[ + Mapping[Type[Any], Sequence[StandardParserAttributeDocumentation]] + ] = None, ) -> DeclarativeInputParser[TD]: """Derive a parser from a TypedDict @@ -978,7 +763,7 @@ class ParserGenerator: f"Unsupported parsed_content descriptor: {parsed_content.__qualname__}." ' Only "TypedDict"-based types and a subset of "DebputyDispatchableType" are supported.' ) - if is_list_wrapped: + if is_list_wrapped and source_content is not None: if get_origin(source_content) != list: raise ValueError( "If the parsed_content is a List type, then source_format must be a List type as well." @@ -1133,10 +918,15 @@ class ParserGenerator: parsed_alt_form.type_validator.combine_mapper(bridge_mapper) ) - _verify_inline_reference_documentation( - source_content_attributes, - inline_reference_documentation, - parsed_alt_form is not None, + inline_reference_documentation = ( + _verify_and_auto_correct_inline_reference_documentation( + parsed_content, + source_typed_dict, + source_content_attributes, + inline_reference_documentation, + parsed_alt_form is not None, + automatic_docs, + ) ) if non_mapping_source_only: parser = DeclarativeNonMappingInputParser( @@ -1700,45 +1490,133 @@ class ParserGenerator: return orig_td -def _verify_inline_reference_documentation( +def _sort_key(attr: StandardParserAttributeDocumentation) -> Any: + key = next(iter(attr.attributes)) + return attr.sort_category, key + + +def _apply_std_docs( + std_doc_table: Optional[ + Mapping[Type[Any], Sequence[StandardParserAttributeDocumentation]] + ], + source_format_typed_dict: Type[Any], + attribute_docs: Optional[Sequence[ParserAttributeDocumentation]], +) -> Optional[Sequence[ParserAttributeDocumentation]]: + if std_doc_table is None or not std_doc_table: + return attribute_docs + + has_docs_for = set() + if attribute_docs: + for attribute_doc in attribute_docs: + has_docs_for.update(attribute_doc.attributes) + + base_seen = set() + std_docs_used = [] + + remaining_bases = set(getattr(source_format_typed_dict, "__orig_bases__", [])) + base_seen.update(remaining_bases) + while remaining_bases: + base = remaining_bases.pop() + new_bases_to_check = { + x for x in getattr(base, "__orig_bases__", []) if x not in base_seen + } + remaining_bases.update(new_bases_to_check) + base_seen.update(new_bases_to_check) + std_docs = std_doc_table.get(base) + if std_docs: + for std_doc in std_docs: + if any(a in has_docs_for for a in std_doc.attributes): + # If there is any overlap, do not add the docs + continue + has_docs_for.update(std_doc.attributes) + std_docs_used.append(std_doc) + + if not std_docs_used: + return attribute_docs + docs = sorted(std_docs_used, key=_sort_key) + if attribute_docs: + # Plugin provided attributes first + c = list(attribute_docs) + c.extend(docs) + docs = c + return tuple(docs) + + +def _verify_and_auto_correct_inline_reference_documentation( + parsed_content: Type[TD], + source_typed_dict: Type[Any], source_content_attributes: Mapping[str, AttributeDescription], inline_reference_documentation: Optional[ParserDocumentation], has_alt_form: bool, -) -> None: - if inline_reference_documentation is None: - return - attribute_doc = inline_reference_documentation.attribute_doc - if attribute_doc: + automatic_docs: Optional[ + Mapping[Type[Any], Sequence[StandardParserAttributeDocumentation]] + ] = None, +) -> Optional[ParserDocumentation]: + orig_attribute_docs = ( + inline_reference_documentation.attribute_doc + if inline_reference_documentation + else None + ) + attribute_docs = _apply_std_docs( + automatic_docs, + source_typed_dict, + orig_attribute_docs, + ) + if inline_reference_documentation is None and attribute_docs is None: + return None + changes = {} + if attribute_docs: seen = set() - for attr_doc in attribute_doc: + had_any_custom_docs = False + for attr_doc in attribute_docs: + if not isinstance(attr_doc, StandardParserAttributeDocumentation): + had_any_custom_docs = True for attr_name in attr_doc.attributes: attr = source_content_attributes.get(attr_name) if attr is None: raise ValueError( - f'The inline_reference_documentation references an attribute "{attr_name}", which does not' - f" exist in the source format." + f"The inline_reference_documentation for the source format of {parsed_content.__qualname__}" + f' references an attribute "{attr_name}", which does not exist in the source format.' ) if attr_name in seen: raise ValueError( - f'The inline_reference_documentation has documentation for "{attr_name}" twice,' - f" which is not supported. Please document it at most once" + f"The inline_reference_documentation for the source format of {parsed_content.__qualname__}" + f' has documentation for "{attr_name}" twice, which is not supported.' + f" Please document it at most once" ) seen.add(attr_name) - undocumented = source_content_attributes.keys() - seen if undocumented: - undocumented_attrs = ", ".join(undocumented) - raise ValueError( - "The following attributes were not documented. If this is deliberate, then please" - ' declare each them as undocumented (via undocumented_attr("foo")):' - f" {undocumented_attrs}" - ) + if had_any_custom_docs: + undocumented_attrs = ", ".join(undocumented) + raise ValueError( + f"The following attributes were not documented for the source format of" + f" {parsed_content.__qualname__}. If this is deliberate, then please" + ' declare each them as undocumented (via undocumented_attr("foo")):' + f" {undocumented_attrs}" + ) + combined_docs = list(attribute_docs) + combined_docs.extend(undocumented_attr(a) for a in sorted(undocumented)) + attribute_docs = combined_docs + + if attribute_docs and orig_attribute_docs != attribute_docs: + assert attribute_docs is not None + changes["attribute_doc"] = tuple(attribute_docs) - if inline_reference_documentation.alt_parser_description and not has_alt_form: + if ( + inline_reference_documentation is not None + and inline_reference_documentation.alt_parser_description + and not has_alt_form + ): raise ValueError( "The inline_reference_documentation had documentation for an non-mapping format," " but the source format does not have a non-mapping format." ) + if changes: + if inline_reference_documentation is None: + inline_reference_documentation = reference_documentation() + return inline_reference_documentation.replace(**changes) + return inline_reference_documentation def _check_conflicts( diff --git a/src/debputy/manifest_parser/exceptions.py b/src/debputy/manifest_parser/exceptions.py index 671ec1b..f058458 100644 --- a/src/debputy/manifest_parser/exceptions.py +++ b/src/debputy/manifest_parser/exceptions.py @@ -1,9 +1,17 @@ from debputy.exceptions import DebputyRuntimeError -class ManifestParseException(DebputyRuntimeError): +class ManifestException(DebputyRuntimeError): + pass + + +class ManifestParseException(ManifestException): pass class ManifestTypeException(ManifestParseException): pass + + +class ManifestInvalidUserDataException(ManifestException): + pass diff --git a/src/debputy/manifest_parser/mapper_code.py b/src/debputy/manifest_parser/mapper_code.py index d7a08c3..f206af9 100644 --- a/src/debputy/manifest_parser/mapper_code.py +++ b/src/debputy/manifest_parser/mapper_code.py @@ -4,22 +4,25 @@ from typing import ( Union, List, Callable, + TYPE_CHECKING, ) from debputy.manifest_parser.exceptions import ManifestTypeException -from debputy.manifest_parser.parser_data import ParserContextData -from debputy.manifest_parser.util import AttributePath from debputy.packages import BinaryPackage from debputy.util import assume_not_none +if TYPE_CHECKING: + from debputy.manifest_parser.util import AttributePath + from debputy.manifest_parser.parser_data import ParserContextData + S = TypeVar("S") T = TypeVar("T") def type_mapper_str2package( raw_package_name: str, - ap: AttributePath, - opc: Optional[ParserContextData], + ap: "AttributePath", + opc: Optional["ParserContextData"], ) -> BinaryPackage: pc = assume_not_none(opc) if "{{" in raw_package_name: @@ -50,7 +53,7 @@ def type_mapper_str2package( def wrap_into_list( x: T, - _ap: AttributePath, + _ap: "AttributePath", _pc: Optional["ParserContextData"], ) -> List[T]: return [x] @@ -58,18 +61,18 @@ def wrap_into_list( def normalize_into_list( x: Union[T, List[T]], - _ap: AttributePath, + _ap: "AttributePath", _pc: Optional["ParserContextData"], ) -> List[T]: return x if isinstance(x, list) else [x] def map_each_element( - mapper: Callable[[S, AttributePath, Optional["ParserContextData"]], T], -) -> Callable[[List[S], AttributePath, Optional["ParserContextData"]], List[T]]: + mapper: Callable[[S, "AttributePath", Optional["ParserContextData"]], T], +) -> Callable[[List[S], "AttributePath", Optional["ParserContextData"]], List[T]]: def _generated_mapper( xs: List[S], - ap: AttributePath, + ap: "AttributePath", pc: Optional["ParserContextData"], ) -> List[T]: return [mapper(s, ap[i], pc) for i, s in enumerate(xs)] diff --git a/src/debputy/manifest_parser/parse_hints.py b/src/debputy/manifest_parser/parse_hints.py new file mode 100644 index 0000000..30b8aca --- /dev/null +++ b/src/debputy/manifest_parser/parse_hints.py @@ -0,0 +1,259 @@ +import dataclasses +from typing import ( + NotRequired, + TypedDict, + TYPE_CHECKING, + Callable, + FrozenSet, + Annotated, + List, +) + +from debputy.manifest_parser.util import ( + resolve_package_type_selectors, + _ALL_PACKAGE_TYPES, +) +from debputy.plugin.api.spec import PackageTypeSelector + +if TYPE_CHECKING: + from debputy.manifest_parser.parser_data import ParserContextData + + +class DebputyParseHint: + @classmethod + def target_attribute(cls, target_attribute: str) -> "DebputyParseHint": + """Define this source attribute to have a different target attribute name + + As an example: + + >>> from debputy.manifest_parser.declarative_parser import ParserGenerator + >>> class SourceType(TypedDict): + ... source: Annotated[NotRequired[str], DebputyParseHint.target_attribute("sources")] + ... sources: NotRequired[List[str]] + >>> class TargetType(TypedDict): + ... sources: List[str] + >>> pg = ParserGenerator() + >>> parser = pg.generate_parser(TargetType, source_content=SourceType) + + In this example, the user can provide either `source` or `sources` and the parser will + map them to the `sources` attribute in the `TargetType`. Note this example relies on + the builtin mapping of `str` to `List[str]` to align the types between `source` (from + SourceType) and `sources` (from TargetType). + + The following rules apply: + + * All source attributes that map to the same target attribute will be mutually exclusive + (that is, the user cannot give `source` *and* `sources` as input). + * When the target attribute is required, the source attributes are conditionally + mandatory requiring the user to provide exactly one of them. + * When multiple source attributes point to a single target attribute, none of the source + attributes can be Required. + * The annotation can only be used for the source type specification and the source type + specification must be different from the target type specification. + + The `target_attribute` annotation can be used without having multiple source attributes. This + can be useful if the source attribute name is not valid as a python variable identifier to + rename it to a valid python identifier. + + :param target_attribute: The attribute name in the target content + :return: The annotation. + """ + return TargetAttribute(target_attribute) + + @classmethod + def conflicts_with_source_attributes( + cls, + *conflicting_source_attributes: str, + ) -> "DebputyParseHint": + """Declare a conflict with one or more source attributes + + Example: + + >>> from debputy.manifest_parser.declarative_parser import ParserGenerator + >>> class SourceType(TypedDict): + ... source: Annotated[NotRequired[str], DebputyParseHint.target_attribute("sources")] + ... sources: NotRequired[List[str]] + ... into_dir: NotRequired[str] + ... renamed_to: Annotated[ + ... NotRequired[str], + ... DebputyParseHint.conflicts_with_source_attributes("sources", "into_dir") + ... ] + >>> class TargetType(TypedDict): + ... sources: List[str] + ... into_dir: NotRequired[str] + ... renamed_to: NotRequired[str] + >>> pg = ParserGenerator() + >>> parser = pg.generate_parser(TargetType, source_content=SourceType) + + In this example, if the user was to provide `renamed_to` with `sources` or `into_dir` the parser would report + an error. However, the parser will allow `renamed_to` with `source` as the conflict is considered only for + the input source. That is, it is irrelevant that `sources` and `source´ happens to "map" to the same target + attribute. + + The following rules apply: + * It is not possible for a target attribute to declare conflicts unless the target type spec is reused as + source type spec. + * All attributes involved in a conflict must be NotRequired. If any of the attributes are Required, then + the parser generator will reject the input. + * All attributes listed in the conflict must be valid attributes in the source type spec. + + Note you do not have to specify conflicts between two attributes with the same target attribute name. The + `target_attribute` annotation will handle that for you. + + :param conflicting_source_attributes: All source attributes that cannot be used with this attribute. + :return: The annotation. + """ + if len(conflicting_source_attributes) < 1: + raise ValueError( + "DebputyParseHint.conflicts_with_source_attributes requires at least one attribute as input" + ) + return ConflictWithSourceAttribute(frozenset(conflicting_source_attributes)) + + @classmethod + def required_when_single_binary( + cls, + *, + package_type: PackageTypeSelector = _ALL_PACKAGE_TYPES, + ) -> "DebputyParseHint": + """Declare a source attribute as required when the source package produces exactly one binary package + + The attribute in question must always be declared as `NotRequired` in the TypedDict and this condition + can only be used for source attributes. + """ + resolved_package_types = resolve_package_type_selectors(package_type) + reason = "The field is required for source packages producing exactly one binary package" + if resolved_package_types != _ALL_PACKAGE_TYPES: + types = ", ".join(sorted(resolved_package_types)) + reason += f" of type {types}" + return ConditionalRequired( + reason, + lambda c: len( + [ + p + for p in c.binary_packages.values() + if p.package_type in package_type + ] + ) + == 1, + ) + return ConditionalRequired( + reason, + lambda c: c.is_single_binary_package, + ) + + @classmethod + def required_when_multi_binary( + cls, + *, + package_type: PackageTypeSelector = _ALL_PACKAGE_TYPES, + ) -> "DebputyParseHint": + """Declare a source attribute as required when the source package produces two or more binary package + + The attribute in question must always be declared as `NotRequired` in the TypedDict and this condition + can only be used for source attributes. + """ + resolved_package_types = resolve_package_type_selectors(package_type) + reason = "The field is required for source packages producing two or more binary packages" + if resolved_package_types != _ALL_PACKAGE_TYPES: + types = ", ".join(sorted(resolved_package_types)) + reason = ( + "The field is required for source packages producing not producing exactly one binary packages" + f" of type {types}" + ) + return ConditionalRequired( + reason, + lambda c: len( + [ + p + for p in c.binary_packages.values() + if p.package_type in package_type + ] + ) + != 1, + ) + return ConditionalRequired( + reason, + lambda c: not c.is_single_binary_package, + ) + + @classmethod + def manifest_attribute(cls, attribute: str) -> "DebputyParseHint": + """Declare what the attribute name (as written in the manifest) should be + + By default, debputy will do an attribute normalizing that will take valid python identifiers such + as `dest_dir` and remap it to the manifest variant (such as `dest-dir`) automatically. If you have + a special case, where this built-in normalization is insufficient or the python name is considerably + different from what the user would write in the manifest, you can use this parse hint to set the + name that the user would have to write in the manifest for this attribute. + + >>> from debputy.manifest_parser.base_types import FileSystemMatchRule, FileSystemExactMatchRule + >>> class SourceType(TypedDict): + ... source: List[FileSystemMatchRule] + ... # Use "as" in the manifest because "as_" was not pretty enough + ... install_as: Annotated[NotRequired[FileSystemExactMatchRule], DebputyParseHint.manifest_attribute("as")] + + In this example, we use the parse hint to use "as" as the name in the manifest, because we cannot + use "as" a valid python identifier (it is a keyword). While debputy would map `as_` to `as` for us, + we have chosen to use `install_as` as a python identifier. + """ + return ManifestAttribute(attribute) + + @classmethod + def not_path_error_hint(cls) -> "DebputyParseHint": + """Mark this attribute as not a "path hint" when it comes to reporting errors + + By default, `debputy` will pick up attributes that uses path names (FileSystemMatchRule) as + candidates for parse error hints (the little "<Search for: VALUE>" in error messages). + + Most rules only have one active path-based attribute and paths tends to be unique enough + that it helps people spot the issue faster. However, in rare cases, you can have multiple + attributes that fit the bill. In this case, this hint can be used to "hide" the suboptimal + choice. As an example: + + >>> from debputy.manifest_parser.base_types import FileSystemMatchRule, FileSystemExactMatchRule + >>> class SourceType(TypedDict): + ... source: List[FileSystemMatchRule] + ... install_as: Annotated[NotRequired[FileSystemExactMatchRule], DebputyParseHint.not_path_error_hint()] + + In this case, without the hint, `debputy` might pick up `install_as` as the attribute to + use as hint for error reporting. However, here we have decided that we never want `install_as` + leaving `source` as the only option. + + Generally, this type hint must be placed on the **source** format. Any source attribute matching + the parsed format will be ignored. + + Mind the asymmetry: The annotation is placed in the **source** format while `debputy` looks at + the type of the target attribute to determine if it counts as path. + """ + return NOT_PATH_HINT + + +@dataclasses.dataclass(frozen=True, slots=True) +class TargetAttribute(DebputyParseHint): + attribute: str + + +@dataclasses.dataclass(frozen=True, slots=True) +class ConflictWithSourceAttribute(DebputyParseHint): + conflicting_attributes: FrozenSet[str] + + +@dataclasses.dataclass(frozen=True, slots=True) +class ConditionalRequired(DebputyParseHint): + reason: str + condition: Callable[["ParserContextData"], bool] + + def condition_applies(self, context: "ParserContextData") -> bool: + return self.condition(context) + + +@dataclasses.dataclass(frozen=True, slots=True) +class ManifestAttribute(DebputyParseHint): + attribute: str + + +class NotPathHint(DebputyParseHint): + pass + + +NOT_PATH_HINT = NotPathHint() diff --git a/src/debputy/manifest_parser/parser_data.py b/src/debputy/manifest_parser/parser_data.py index 30d9ce0..acc5c67 100644 --- a/src/debputy/manifest_parser/parser_data.py +++ b/src/debputy/manifest_parser/parser_data.py @@ -11,12 +11,15 @@ from debian.debian_support import DpkgArchTable from debputy._deb_options_profiles import DebBuildOptionsAndProfiles from debputy.architecture_support import DpkgArchitectureBuildProcessValuesTable +from debputy.manifest_parser.base_types import BuildEnvironmentDefinition from debputy.manifest_parser.exceptions import ManifestParseException from debputy.manifest_parser.util import AttributePath -from debputy.packages import BinaryPackage -from debputy.plugin.api.impl_types import ( +from debputy.manifest_parser.util import ( _ALL_PACKAGE_TYPES, resolve_package_type_selectors, +) +from debputy.packages import BinaryPackage +from debputy.plugin.api.impl_types import ( TP, DispatchingTableParser, TTP, @@ -101,7 +104,7 @@ class ParserContextData: raise NotImplementedError @property - def build_env(self) -> DebBuildOptionsAndProfiles: + def deb_options_and_profiles(self) -> DebBuildOptionsAndProfiles: raise NotImplementedError @contextlib.contextmanager @@ -129,3 +132,8 @@ class ParserContextData: @property def debputy_integration_mode(self) -> DebputyIntegrationMode: raise NotImplementedError + + def resolve_build_environment( + self, name: Optional[str], attribute_path: AttributePath + ) -> BuildEnvironmentDefinition: + raise NotImplementedError diff --git a/src/debputy/manifest_parser/tagging_types.py b/src/debputy/manifest_parser/tagging_types.py new file mode 100644 index 0000000..83030f0 --- /dev/null +++ b/src/debputy/manifest_parser/tagging_types.py @@ -0,0 +1,36 @@ +import dataclasses +from typing import ( + TypedDict, + TYPE_CHECKING, + Generic, + Type, + Callable, + Optional, +) + +from debputy.plugin.plugin_state import current_debputy_plugin_required +from debputy.types import S +from debputy.util import T + +if TYPE_CHECKING: + from debputy.manifest_parser.parser_data import ParserContextData + + from debputy.manifest_parser.util import AttributePath + + +class DebputyParsedContent(TypedDict): + pass + + +class DebputyDispatchableType: + __slots__ = ("_debputy_plugin",) + + def __init__(self) -> None: + self._debputy_plugin = current_debputy_plugin_required() + + +@dataclasses.dataclass +class TypeMapping(Generic[S, T]): + target_type: Type[T] + source_type: Type[S] + mapper: Callable[[S, "AttributePath", Optional["ParserContextData"]], T] diff --git a/src/debputy/manifest_parser/util.py b/src/debputy/manifest_parser/util.py index a9cbbe8..4e8fd7c 100644 --- a/src/debputy/manifest_parser/util.py +++ b/src/debputy/manifest_parser/util.py @@ -14,13 +14,18 @@ from typing import ( TYPE_CHECKING, Iterable, Container, + Literal, + FrozenSet, + cast, ) +from debputy.yaml.compat import CommentedBase + from debputy.manifest_parser.exceptions import ManifestParseException if TYPE_CHECKING: from debputy.manifest_parser.parser_data import ParserContextData - from debputy.plugin.api.spec import DebputyIntegrationMode + from debputy.plugin.api.spec import DebputyIntegrationMode, PackageTypeSelector MP = TypeVar("MP", bound="DebputyParseHint") @@ -28,26 +33,48 @@ StrOrInt = Union[str, int] AttributePathAliasMapping = Mapping[ StrOrInt, Tuple[StrOrInt, Optional["AttributePathAliasMapping"]] ] +LineReportKind = Literal["key", "value", "container"] + + +_PACKAGE_TYPE_DEB_ONLY = frozenset(["deb"]) +_ALL_PACKAGE_TYPES = frozenset(["deb", "udeb"]) + + +def resolve_package_type_selectors( + package_type: "PackageTypeSelector", +) -> FrozenSet[str]: + if package_type is _ALL_PACKAGE_TYPES or package_type is _PACKAGE_TYPE_DEB_ONLY: + return cast("FrozenSet[str]", package_type) + if isinstance(package_type, str): + return ( + _PACKAGE_TYPE_DEB_ONLY + if package_type == "deb" + else frozenset([package_type]) + ) + else: + return frozenset(package_type) -class AttributePath(object): - __slots__ = ("parent", "name", "alias_mapping", "path_hint") +class AttributePath: + __slots__ = ("parent", "container", "name", "alias_mapping", "path_hint") def __init__( self, parent: Optional["AttributePath"], key: Optional[Union[str, int]], *, + container: Optional[Any] = None, alias_mapping: Optional[AttributePathAliasMapping] = None, ) -> None: self.parent = parent + self.container = container self.name = key self.path_hint: Optional[str] = None self.alias_mapping = alias_mapping @classmethod - def root_path(cls) -> "AttributePath": - return AttributePath(None, None) + def root_path(cls, container: Optional[Any]) -> "AttributePath": + return AttributePath(None, None, container=container) @classmethod def builtin_path(cls) -> "AttributePath": @@ -70,8 +97,29 @@ class AttributePath(object): segments.reverse() yield from (s.name for s in segments) - @property - def path(self) -> str: + def _resolve_path(self, report_kind: LineReportKind) -> str: + parent = self.parent + key = self.name + if report_kind == "container": + key = parent.name if parent else None + parent = parent.parent if parent else None + container = parent.container if parent is not None else None + + if isinstance(container, CommentedBase): + lc = container.lc + try: + if isinstance(key, str): + if report_kind == "key": + lc_data = lc.key(key) + else: + lc_data = lc.value(key) + else: + lc_data = lc.item(key) + except (AttributeError, RuntimeError, LookupError, TypeError): + lc_data = None + else: + lc_data = None + segments = list(self._iter_path()) segments.reverse() parts: List[str] = [] @@ -88,12 +136,31 @@ class AttributePath(object): if parts: parts.append(".") parts.append(k) - if path_hint: + + if lc_data is not None: + line_pos, col = lc_data + # Translate 0-based (index) to 1-based (line number) + line_pos += 1 + parts.append(f" [Line {line_pos} column {col}]") + + elif path_hint: parts.append(f" <Search for: {path_hint}>") if not parts: return "document root" return "".join(parts) + @property + def path_container_lc(self) -> str: + return self._resolve_path("container") + + @property + def path_key_lc(self) -> str: + return self._resolve_path("key") + + @property + def path(self) -> str: + return self._resolve_path("value") + def __str__(self) -> str: return self.path @@ -106,9 +173,25 @@ class AttributePath(object): if item == "": # Support `sources[0]` mapping to `source` by `sources -> source` and `0 -> ""`. return AttributePath( - self.parent, self.name, alias_mapping=alias_mapping + self.parent, + self.name, + alias_mapping=alias_mapping, + container=self.container, ) - return AttributePath(self, item, alias_mapping=alias_mapping) + container = self.container + if container is not None: + try: + child_container = self.container[item] + except (AttributeError, RuntimeError, LookupError, TypeError): + child_container = None + else: + child_container = None + return AttributePath( + self, + item, + alias_mapping=alias_mapping, + container=child_container, + ) def _iter_path(self) -> Iterator["AttributePath"]: current = self diff --git a/src/debputy/package_build/assemble_deb.py b/src/debputy/package_build/assemble_deb.py index fd92f37..21fe0d6 100644 --- a/src/debputy/package_build/assemble_deb.py +++ b/src/debputy/package_build/assemble_deb.py @@ -95,8 +95,8 @@ def assemble_debs( package_metadata_context = dctrl_data.package_metadata_context if ( dbgsym_package_name in package_data_table - or "noautodbgsym" in manifest.build_env.deb_build_options - or "noddebs" in manifest.build_env.deb_build_options + or "noautodbgsym" in manifest.deb_options_and_profiles.deb_build_options + or "noddebs" in manifest.deb_options_and_profiles.deb_build_options ): # Discard the dbgsym part if it conflicts with a real package, or # we were asked not to build it. diff --git a/src/debputy/packager_provided_files.py b/src/debputy/packager_provided_files.py index 8e0e0af..95d1b17 100644 --- a/src/debputy/packager_provided_files.py +++ b/src/debputy/packager_provided_files.py @@ -1,6 +1,6 @@ import collections import dataclasses -from typing import Mapping, Iterable, Dict, List, Optional, Tuple, Sequence +from typing import Mapping, Iterable, Dict, List, Optional, Tuple, Sequence, Container from debputy.packages import BinaryPackage from debputy.plugin.api import VirtualPath @@ -10,7 +10,11 @@ from debputy.util import _error, CAN_DETECT_TYPOS, detect_possible_typo _KNOWN_NON_PPFS = frozenset( { + # Some of these overlap with the _KNOWN_NON_TYPO_EXTENSIONS below + # This one is a quicker check. The _KNOWN_NON_TYPO_EXTENSIONS is a general (but more + # expensive check). "gbp.conf", # Typo matches with `gbp.config` (dh_installdebconf) in two edits steps + "salsa-ci.yml", # Typo matches with `salsa-ci.wm` (dh_installwm) in two edits steps # No reason to check any of these as they are never PPFs "clean", "control", @@ -21,6 +25,21 @@ _KNOWN_NON_PPFS = frozenset( } ) +_KNOWN_NON_TYPO_EXTENSIONS = frozenset( + { + "conf", + "sh", + "yml", + "yaml", + "json", + "bash", + "pl", + "py", + # Fairly common image format in older packages + "xpm", + } +) + @dataclasses.dataclass(frozen=True, slots=True) class PackagerProvidedFile: @@ -107,6 +126,7 @@ def _find_definition( basename: str, *, period2stems: Optional[Mapping[int, Sequence[str]]] = None, + had_arch: bool = False, ) -> Tuple[Optional[str], Optional[PackagerProvidedFileClassSpec], Optional[str]]: for stem, install_as_name, period_count in _iterate_stem_splits(basename): definition = packager_provided_files.get(stem) @@ -118,6 +138,12 @@ def _find_definition( if not stems: continue + # If the stem is also the extension and a known one at that, then + # we do not consider it a typo match (to avoid false positives). + # + # We also ignore "foo.1" since manpages are kind of common. + if not had_arch and (stem in _KNOWN_NON_TYPO_EXTENSIONS or stem.isdigit()): + continue matches = detect_possible_typo(stem, stems) if matches is not None and len(matches) == 1: definition = packager_provided_files[matches[0]] @@ -252,6 +278,7 @@ def _split_path( packager_provided_files, basename, period2stems=period2stems, + had_arch=bool(arch_restriction), ) if definition is None: continue @@ -365,10 +392,19 @@ def detect_all_packager_provided_files( *, allow_fuzzy_matches: bool = False, detect_typos: bool = False, + ignore_paths: Container[str] = frozenset(), ) -> Dict[str, PerPackagePackagerProvidedResult]: - main_binary_package = [ - p.name for p in binary_packages.values() if p.is_main_package - ][0] + main_packages = [p.name for p in binary_packages.values() if p.is_main_package] + if not main_packages: + assert allow_fuzzy_matches + main_binary_package = next( + iter(p.name for p in binary_packages.values() if "Package" in p.fields), + None, + ) + if main_binary_package is None: + return {} + else: + main_binary_package = main_packages[0] provided_files: Dict[str, Dict[Tuple[str, str], PackagerProvidedFile]] = { n: {} for n in binary_packages } @@ -381,7 +417,7 @@ def detect_all_packager_provided_files( for entry in debian_dir.iterdir: if entry.is_dir: continue - if entry.name in _KNOWN_NON_PPFS: + if entry.path in ignore_paths or entry.name in _KNOWN_NON_PPFS: continue matching_ppfs = _split_path( packager_provided_files, diff --git a/src/debputy/packages.py b/src/debputy/packages.py index 3a6ee16..0a3876a 100644 --- a/src/debputy/packages.py +++ b/src/debputy/packages.py @@ -44,15 +44,15 @@ class DctrlParser: DpkgArchitectureBuildProcessValuesTable ] = None, dpkg_arch_query_table: Optional[DpkgArchTable] = None, - build_env: Optional[DebBuildOptionsAndProfiles] = None, + deb_options_and_profiles: Optional[DebBuildOptionsAndProfiles] = None, ignore_errors: bool = False, ) -> None: if dpkg_architecture_variables is None: dpkg_architecture_variables = dpkg_architecture_table() if dpkg_arch_query_table is None: dpkg_arch_query_table = DpkgArchTable.load_arch_table() - if build_env is None: - build_env = DebBuildOptionsAndProfiles.instance() + if deb_options_and_profiles is None: + deb_options_and_profiles = DebBuildOptionsAndProfiles.instance() # If no selection option is set, then all packages are acted on (except the # excluded ones) @@ -66,7 +66,7 @@ class DctrlParser: self.select_arch_any = select_arch_any self.dpkg_architecture_variables = dpkg_architecture_variables self.dpkg_arch_query_table = dpkg_arch_query_table - self.build_env = build_env + self.deb_options_and_profiles = deb_options_and_profiles self.ignore_errors = ignore_errors @overload @@ -138,7 +138,7 @@ class DctrlParser: self.select_arch_any, self.dpkg_architecture_variables, self.dpkg_arch_query_table, - self.build_env, + self.deb_options_and_profiles, i, ) ) diff --git a/src/debputy/packaging/makeshlibs.py b/src/debputy/packaging/makeshlibs.py index 127a64d..a020545 100644 --- a/src/debputy/packaging/makeshlibs.py +++ b/src/debputy/packaging/makeshlibs.py @@ -200,12 +200,19 @@ def generate_shlib_dirs( ) -> None: dir_scanned: Dict[str, Dict[str, Set[str]]] = {} dirs: Dict[str, str] = {} + warn_dirs = { + "/usr/lib", + "/lib", + f"/usr/lib/{pkg.deb_multiarch}", + f"/lib/{pkg.deb_multiarch}", + } for soname_info in soname_info_list: elf_binary = soname_info.path p = assume_not_none(elf_binary.parent_dir) - matches = dir_scanned.get(p.absolute) - materialized_dir = dirs.get(p.absolute) + abs_parent_path = p.absolute + matches = dir_scanned.get(abs_parent_path) + materialized_dir = dirs.get(abs_parent_path) if matches is None: matches = collections.defaultdict(set) for child in p.iterdir: @@ -216,17 +223,18 @@ def generate_shlib_dirs( # The shlib symlinks (we are interested in) are relative to the same folder continue matches[target].add(child.name) - dir_scanned[p.absolute] = matches + dir_scanned[abs_parent_path] = matches symlinks = matches.get(elf_binary.name) if not symlinks: - _warn( - f"Could not find any SO symlinks pointing to {elf_binary.absolute} in {pkg.name} !?" - ) + if abs_parent_path in warn_dirs: + _warn( + f"Could not find any SO symlinks pointing to {elf_binary.absolute} in {pkg.name} !?" + ) continue if materialized_dir is None: materialized_dir = tempfile.mkdtemp(prefix=f"{pkg.name}_", dir=root_dir) materialized_dirs.append(materialized_dir) - dirs[p.absolute] = materialized_dir + dirs[abs_parent_path] = materialized_dir os.symlink(elf_binary.fs_path, os.path.join(materialized_dir, elf_binary.name)) for link in symlinks: diff --git a/src/debputy/path_matcher.py b/src/debputy/path_matcher.py index 2917b14..a7b8356 100644 --- a/src/debputy/path_matcher.py +++ b/src/debputy/path_matcher.py @@ -161,7 +161,7 @@ class MatchRule: _error( f'The pattern "{path_or_glob}" (defined in {definition_source}) looks like it contains a' f' brace expansion (such as "{{a,b}}" or "{{a..b}}"). Brace expansions are not supported.' - " If you wanted to match the literal path a brace in it, please use a substitution to insert" + " If you wanted to match the literal path with a brace in it, please use a substitution to insert" f' the opening brace. As an example: "{replacement}"' ) diff --git a/src/debputy/plugin/api/feature_set.py b/src/debputy/plugin/api/feature_set.py index a56f37b..30d79be 100644 --- a/src/debputy/plugin/api/feature_set.py +++ b/src/debputy/plugin/api/feature_set.py @@ -1,29 +1,24 @@ import dataclasses -import textwrap -from typing import Dict, List, Tuple, Sequence, Any +from typing import Dict, List, Tuple, Sequence, Any, Optional, Type -from debputy import DEBPUTY_DOC_ROOT_DIR from debputy.manifest_parser.declarative_parser import ParserGenerator -from debputy.plugin.api import reference_documentation from debputy.plugin.api.impl_types import ( DebputyPluginMetadata, PackagerProvidedFileClassSpec, MetadataOrMaintscriptDetector, - TTP, - DispatchingTableParser, - TP, - SUPPORTED_DISPATCHABLE_TABLE_PARSERS, - DispatchingObjectParser, - SUPPORTED_DISPATCHABLE_OBJECT_PARSERS, PluginProvidedManifestVariable, PluginProvidedPackageProcessor, PluginProvidedDiscardRule, ServiceManagerDetails, PluginProvidedKnownPackagingFile, PluginProvidedTypeMapping, - OPARSER_PACKAGES, - OPARSER_PACKAGES_ROOT, + PluginProvidedBuildSystemAutoDetection, +) +from debputy.plugin.api.parser_tables import ( + SUPPORTED_DISPATCHABLE_OBJECT_PARSERS, + SUPPORTED_DISPATCHABLE_TABLE_PARSERS, ) +from debputy.plugin.debputy.to_be_api_types import BuildSystemRule def _initialize_parser_generator() -> ParserGenerator: @@ -70,6 +65,9 @@ class PluginProvidedFeatureSet: manifest_parser_generator: ParserGenerator = dataclasses.field( default_factory=_initialize_parser_generator ) + auto_detectable_build_systems: Dict[ + Type[BuildSystemRule], PluginProvidedBuildSystemAutoDetection + ] = dataclasses.field(default_factory=dict) def package_processors_in_order(self) -> Sequence[PluginProvidedPackageProcessor]: order = [] diff --git a/src/debputy/plugin/api/impl.py b/src/debputy/plugin/api/impl.py index 6369663..c2f03d0 100644 --- a/src/debputy/plugin/api/impl.py +++ b/src/debputy/plugin/api/impl.py @@ -31,7 +31,8 @@ from typing import ( Any, Literal, Container, - get_args, + TYPE_CHECKING, + is_typeddict, ) from debputy import DEBPUTY_DOC_ROOT_DIR @@ -43,16 +44,18 @@ from debputy.exceptions import ( PluginInitializationError, PluginAPIViolationError, PluginNotFoundError, + PluginIncorrectRegistrationError, ) from debputy.maintscript_snippet import ( STD_CONTROL_SCRIPTS, MaintscriptSnippetContainer, MaintscriptSnippet, ) -from debputy.manifest_parser.base_types import TypeMapping from debputy.manifest_parser.exceptions import ManifestParseException from debputy.manifest_parser.parser_data import ParserContextData +from debputy.manifest_parser.tagging_types import TypeMapping from debputy.manifest_parser.util import AttributePath +from debputy.manifest_parser.util import resolve_package_type_selectors from debputy.plugin.api.feature_set import PluginProvidedFeatureSet from debputy.plugin.api.impl_types import ( DebputyPluginMetadata, @@ -70,11 +73,12 @@ from debputy.plugin.api.impl_types import ( AutomaticDiscardRuleExample, PPFFormatParam, ServiceManagerDetails, - resolve_package_type_selectors, KnownPackagingFileInfo, PluginProvidedKnownPackagingFile, InstallPatternDHCompatRule, PluginProvidedTypeMapping, + PluginProvidedBuildSystemAutoDetection, + BSR, ) from debputy.plugin.api.plugin_parser import ( PLUGIN_METADATA_PARSER, @@ -108,6 +112,21 @@ from debputy.plugin.api.spec import ( packager_provided_file_reference_documentation, TypeMappingDocumentation, DebputyIntegrationMode, + reference_documentation, + _DEBPUTY_DISPATCH_METADATA_ATTR_NAME, + BuildSystemManifestRuleMetadata, +) +from debputy.plugin.api.std_docs import _STD_ATTR_DOCS +from debputy.plugin.debputy.to_be_api_types import ( + BuildSystemRule, + BuildRuleParsedFormat, + BSPF, + debputy_build_system, +) +from debputy.plugin.plugin_state import ( + run_in_context_of_plugin, + run_in_context_of_plugin_wrap_errors, + wrap_plugin_code, ) from debputy.substitution import ( Substitution, @@ -123,6 +142,9 @@ from debputy.util import ( _warn, ) +if TYPE_CHECKING: + from debputy.highlevel_manifest import HighLevelManifest + PLUGIN_TEST_SUFFIX = re.compile(r"_(?:t|test|check)(?:_([a-z0-9_]+))?[.]py$") @@ -362,7 +384,7 @@ class DebputyPluginInitializerProvider(DebputyPluginInitializer): all_detectors[self._plugin_name].append( MetadataOrMaintscriptDetector( detector_id=auto_detector_id, - detector=auto_detector, + detector=wrap_plugin_code(self._plugin_name, auto_detector), plugin_metadata=self._plugin_metadata, applies_to_package_types=package_types, enabled=True, @@ -575,7 +597,7 @@ class DebputyPluginInitializerProvider(DebputyPluginInitializer): package_processors[processor_key] = PluginProvidedPackageProcessor( processor_id, resolve_package_type_selectors(package_type), - processor, + wrap_plugin_code(self._plugin_name, processor), frozenset(dependencies), self._plugin_metadata, ) @@ -704,8 +726,8 @@ class DebputyPluginInitializerProvider(DebputyPluginInitializer): ) service_managers[service_manager] = ServiceManagerDetails( service_manager, - detector, - integrator, + wrap_plugin_code(self._plugin_name, detector), + wrap_plugin_code(self._plugin_name, integrator), self._plugin_metadata, ) @@ -776,7 +798,7 @@ class DebputyPluginInitializerProvider(DebputyPluginInitializer): dispatching_parser = parser_generator.dispatchable_table_parsers[rule_type] dispatching_parser.register_keyword( rule_name, - handler, + wrap_plugin_code(self._plugin_name, handler), self._plugin_metadata, inline_reference_documentation=inline_reference_documentation, ) @@ -820,6 +842,10 @@ class DebputyPluginInitializerProvider(DebputyPluginInitializer): ) parent_dispatcher = dispatchable_object_parsers[rule_type] child_dispatcher = dispatchable_object_parsers[object_parser_key] + + if on_end_parse_step is not None: + on_end_parse_step = wrap_plugin_code(self._plugin_name, on_end_parse_step) + parent_dispatcher.register_child_parser( rule_name, child_dispatcher, @@ -838,7 +864,7 @@ class DebputyPluginInitializerProvider(DebputyPluginInitializer): def pluggable_manifest_rule( self, rule_type: Union[TTP, str], - rule_name: Union[str, List[str]], + rule_name: Union[str, Sequence[str]], parsed_format: Type[PF], handler: DIPHandler, *, @@ -847,8 +873,15 @@ class DebputyPluginInitializerProvider(DebputyPluginInitializer): expected_debputy_integration_mode: Optional[ Container[DebputyIntegrationMode] ] = None, + apply_standard_attribute_documentation: bool = False, ) -> None: + # When unrestricted this, consider which types will be unrestricted self._restricted_api() + if apply_standard_attribute_documentation and sys.version_info < (3, 12): + _error( + f"The plugin {self._plugin_metadata.plugin_name} requires python 3.12 due to" + f" its use of apply_standard_attribute_documentation" + ) feature_set = self._feature_set parser_generator = feature_set.manifest_parser_generator if isinstance(rule_type, str): @@ -870,16 +903,22 @@ class DebputyPluginInitializerProvider(DebputyPluginInitializer): ) dispatching_parser = parser_generator.dispatchable_table_parsers[rule_type] + if apply_standard_attribute_documentation: + docs = _STD_ATTR_DOCS + else: + docs = None + parser = feature_set.manifest_parser_generator.generate_parser( parsed_format, source_content=source_format, inline_reference_documentation=inline_reference_documentation, expected_debputy_integration_mode=expected_debputy_integration_mode, + automatic_docs=docs, ) dispatching_parser.register_parser( rule_name, parser, - handler, + wrap_plugin_code(self._plugin_name, handler), self._plugin_metadata, ) @@ -890,6 +929,108 @@ class DebputyPluginInitializerProvider(DebputyPluginInitializer): self._unloaders.append(_unload) + def register_build_system( + self, + build_system_definition: type[BSPF], + ) -> None: + self._restricted_api() + if not is_typeddict(build_system_definition): + raise PluginInitializationError( + f"Expected build_system_definition to be a subclass of {BuildRuleParsedFormat.__name__}," + f" but got {build_system_definition.__name__} instead" + ) + metadata = getattr( + build_system_definition, + _DEBPUTY_DISPATCH_METADATA_ATTR_NAME, + None, + ) + if not isinstance(metadata, BuildSystemManifestRuleMetadata): + raise PluginIncorrectRegistrationError( + f"The {build_system_definition.__qualname__} type should have been annotated with" + f" @{debputy_build_system.__name__}." + ) + assert len(metadata.manifest_keywords) == 1 + build_system_impl = metadata.build_system_impl + assert build_system_impl is not None + manifest_keyword = next(iter(metadata.manifest_keywords)) + self.pluggable_manifest_rule( + metadata.dispatched_type, + metadata.manifest_keywords, + build_system_definition, + # pluggable_manifest_rule does the wrapping + metadata.unwrapped_constructor, + source_format=metadata.source_format, + ) + self._auto_detectable_build_system( + manifest_keyword, + build_system_impl, + constructor=wrap_plugin_code( + self._plugin_name, + build_system_impl, + ), + shadowing_build_systems_when_active=metadata.auto_detection_shadow_build_systems, + ) + + def _auto_detectable_build_system( + self, + manifest_keyword: str, + rule_type: type[BSR], + *, + shadowing_build_systems_when_active: FrozenSet[str] = frozenset(), + constructor: Optional[ + Callable[[BuildRuleParsedFormat, AttributePath, "HighLevelManifest"], BSR] + ] = None, + ) -> None: + self._restricted_api() + feature_set = self._feature_set + existing = feature_set.auto_detectable_build_systems.get(rule_type) + if existing is not None: + bs_name = rule_type.__class__.__name__ + if existing.plugin_metadata.plugin_name == self._plugin_name: + message = ( + f"Bug in the plugin {self._plugin_name}: It tried to register the" + f' auto-detection of the build system "{bs_name}" twice.' + ) + else: + message = ( + f"The plugins {existing.plugin_metadata.plugin_name} and {self._plugin_name}" + f' both tried to provide auto-detection of the build system "{bs_name}"' + ) + raise PluginConflictError( + message, existing.plugin_metadata, self._plugin_metadata + ) + + if constructor is None: + + def impl( + attributes: BuildRuleParsedFormat, + attribute_path: AttributePath, + manifest: "HighLevelManifest", + ) -> BSR: + return rule_type(attributes, attribute_path, manifest) + + else: + impl = constructor + + feature_set.auto_detectable_build_systems[rule_type] = ( + PluginProvidedBuildSystemAutoDetection( + manifest_keyword, + rule_type, + wrap_plugin_code(self._plugin_name, rule_type.auto_detect_build_system), + impl, + shadowing_build_systems_when_active, + self._plugin_metadata, + ) + ) + + def _unload() -> None: + try: + del feature_set.auto_detectable_build_systems[rule_type] + except KeyError: + pass + + self._unloaders.append(_unload) + def known_packaging_files( self, packaging_file_details: KnownPackagingFileInfo, @@ -981,6 +1122,7 @@ class DebputyPluginInitializerProvider(DebputyPluginInitializer): message, existing.plugin_metadata, self._plugin_metadata ) parser_generator = self._feature_set.manifest_parser_generator + # TODO: Wrap the mapper in the plugin context mapped_types[target_type] = PluginProvidedTypeMapping( type_mapping, reference_documentation, self._plugin_metadata ) @@ -1437,6 +1579,10 @@ def load_plugin_features( if plugin_metadata.plugin_name not in unloadable_plugins: raise if debug_mode: + _warn( + f"The optional plugin {plugin_metadata.plugin_name} failed during load. Re-raising due" + f" to --debug/-d." + ) raise try: api.unload_plugin() @@ -1448,11 +1594,6 @@ def load_plugin_features( ) raise e from None else: - if debug_mode: - _warn( - f"The optional plugin {plugin_metadata.plugin_name} failed during load. Re-raising due" - f" to --debug/-d." - ) _warn( f"The optional plugin {plugin_metadata.plugin_name} failed during load. The plugin was" f" deactivated. Use debug mode (--debug) to show the stacktrace (the warning will become an error)" @@ -1635,7 +1776,7 @@ def _resolve_module_initializer( ) sys.modules[module_name] = mod try: - loader.exec_module(mod) + run_in_context_of_plugin(plugin_name, loader.exec_module, mod) except (Exception, GeneratorExit) as e: raise PluginInitializationError( f"Failed to load {plugin_name} (path: {module_fs_path})." @@ -1645,7 +1786,9 @@ def _resolve_module_initializer( if module is None: try: - module = importlib.import_module(module_name) + module = run_in_context_of_plugin( + plugin_name, importlib.import_module, module_name + ) except ModuleNotFoundError as e: if module_fs_path is None: raise PluginMetadataError( @@ -1660,7 +1803,12 @@ def _resolve_module_initializer( f' explicit "module" definition in {json_file_path}.' ) from e - plugin_initializer = getattr(module, plugin_initializer_name) + plugin_initializer = run_in_context_of_plugin_wrap_errors( + plugin_name, + getattr, + module, + plugin_initializer_name, + ) if plugin_initializer is None: raise PluginMetadataError( @@ -1867,7 +2015,7 @@ def parse_json_plugin_desc( f" clash with the bundled plugin of same name." ) - attribute_path = AttributePath.root_path() + attribute_path = AttributePath.root_path(raw) try: plugin_json_metadata = PLUGIN_METADATA_PARSER.parse_input( diff --git a/src/debputy/plugin/api/impl_types.py b/src/debputy/plugin/api/impl_types.py index 77e96ea..85beaf8 100644 --- a/src/debputy/plugin/api/impl_types.py +++ b/src/debputy/plugin/api/impl_types.py @@ -1,6 +1,5 @@ import dataclasses import os.path -import textwrap from typing import ( Optional, Callable, @@ -24,22 +23,21 @@ from typing import ( Set, Iterator, Container, + Protocol, ) from weakref import ref -from debputy import DEBPUTY_DOC_ROOT_DIR from debputy.exceptions import ( DebputyFSIsROError, PluginAPIViolationError, PluginConflictError, UnhandledOrUnexpectedErrorFromPluginError, + PluginBaseError, + PluginInitializationError, ) from debputy.filesystem_scan import as_path_def -from debputy.installations import InstallRule -from debputy.maintscript_snippet import DpkgMaintscriptHelperCommand -from debputy.manifest_conditions import ManifestCondition -from debputy.manifest_parser.base_types import DebputyParsedContent, TypeMapping from debputy.manifest_parser.exceptions import ManifestParseException +from debputy.manifest_parser.tagging_types import DebputyParsedContent, TypeMapping from debputy.manifest_parser.util import AttributePath, check_integration_mode from debputy.packages import BinaryPackage from debputy.plugin.api import ( @@ -62,15 +60,16 @@ from debputy.plugin.api.spec import ( TypeMappingDocumentation, DebputyIntegrationMode, ) +from debputy.plugin.plugin_state import ( + run_in_context_of_plugin, +) from debputy.substitution import VariableContext -from debputy.transformation_rules import TransformationRule from debputy.util import _normalize_path, package_cross_check_precheck if TYPE_CHECKING: from debputy.plugin.api.spec import ( ServiceDetector, ServiceIntegrator, - PackageTypeSelector, ) from debputy.manifest_parser.parser_data import ParserContextData from debputy.highlevel_manifest import ( @@ -78,10 +77,10 @@ if TYPE_CHECKING: PackageTransformationDefinition, BinaryPackageData, ) - - -_PACKAGE_TYPE_DEB_ONLY = frozenset(["deb"]) -_ALL_PACKAGE_TYPES = frozenset(["deb", "udeb"]) + from debputy.plugin.debputy.to_be_api_types import ( + BuildSystemRule, + BuildRuleParsedFormat, + ) TD = TypeVar("TD", bound="Union[DebputyParsedContent, List[DebputyParsedContent]]") @@ -89,26 +88,12 @@ PF = TypeVar("PF") SF = TypeVar("SF") TP = TypeVar("TP") TTP = Type[TP] +BSR = TypeVar("BSR", bound="BuildSystemRule") DIPKWHandler = Callable[[str, AttributePath, "ParserContextData"], TP] DIPHandler = Callable[[str, PF, AttributePath, "ParserContextData"], TP] -def resolve_package_type_selectors( - package_type: "PackageTypeSelector", -) -> FrozenSet[str]: - if package_type is _ALL_PACKAGE_TYPES or package_type is _PACKAGE_TYPE_DEB_ONLY: - return cast("FrozenSet[str]", package_type) - if isinstance(package_type, str): - return ( - _PACKAGE_TYPE_DEB_ONLY - if package_type == "deb" - else frozenset([package_type]) - ) - else: - return frozenset(package_type) - - @dataclasses.dataclass(slots=True) class DebputyPluginMetadata: plugin_name: str @@ -143,7 +128,17 @@ class DebputyPluginMetadata: def load_plugin(self) -> None: plugin_loader = self.plugin_loader assert plugin_loader is not None - self.plugin_initializer = plugin_loader() + try: + self.plugin_initializer = run_in_context_of_plugin( + self.plugin_name, + plugin_loader, + ) + except PluginBaseError: + raise + except Exception as e: + raise PluginInitializationError( + f"Initialization of {self.plugin_name} failed due to its initializer raising an exception" + ) from e assert self.plugin_initializer is not None @@ -270,12 +265,10 @@ class MetadataOrMaintscriptDetector: " this stage (file system layout is committed and the attempted changes" " would be lost)." ) from e - except (ChildProcessError, RuntimeError, AttributeError) as e: - nv = f"{self.plugin_metadata.plugin_name}" - raise UnhandledOrUnexpectedErrorFromPluginError( - f"The plugin {nv} threw an unhandled or unexpected exception from its metadata" - f" detector with id {self.detector_id}." - ) from e + except UnhandledOrUnexpectedErrorFromPluginError as e: + e.add_note( + f"The exception was raised by the detector with the ID: {self.detector_id}" + ) class DeclarativeInputParser(Generic[TD]): @@ -587,11 +580,11 @@ class DispatchingObjectParser( ) if not isinstance(orig_value, dict): raise ManifestParseException( - f"The attribute {attribute_path.path} must be a non-empty mapping.{doc_ref}" + f"The attribute {attribute_path.path_container_lc} must be a non-empty mapping.{doc_ref}" ) if not orig_value: raise ManifestParseException( - f"The attribute {attribute_path.path} must be a non-empty mapping.{doc_ref}" + f"The attribute {attribute_path.path_container_lc} must be a non-empty mapping.{doc_ref}" ) result = {} unknown_keys = orig_value.keys() - self._parsers.keys() @@ -605,7 +598,7 @@ class DispatchingObjectParser( ) remaining_valid_attribute_names = ", ".join(remaining_valid_attributes) raise ManifestParseException( - f'The attribute "{first_key}" is not applicable at {attribute_path.path}(with the current set' + f'The attribute "{first_key}" is not applicable at {attribute_path.path} (with the current set' " of plugins). Possible attributes available (and not already used) are:" f" {remaining_valid_attribute_names}.{doc_ref}" ) @@ -615,7 +608,10 @@ class DispatchingObjectParser( if value is None: if isinstance(provided_parser.parser, DispatchingObjectParser): provided_parser.handler( - key, {}, attribute_path[key], parser_context + key, + {}, + attribute_path[key], + parser_context, ) continue value_path = attribute_path[key] @@ -675,7 +671,7 @@ class InPackageContextParser( ) if not isinstance(orig_value, dict) or not orig_value: raise ManifestParseException( - f"The attribute {attribute_path.path} must be a non-empty mapping.{doc_ref}" + f"The attribute {attribute_path.path_container_lc} must be a non-empty mapping.{doc_ref}" ) delegate = self.delegate result = {} @@ -774,64 +770,6 @@ class DeclarativeValuelessKeywordInputParser(DeclarativeInputParser[None]): ) -SUPPORTED_DISPATCHABLE_TABLE_PARSERS = { - InstallRule: "installations", - TransformationRule: "packages.{{PACKAGE}}.transformations", - DpkgMaintscriptHelperCommand: "packages.{{PACKAGE}}.conffile-management", - ManifestCondition: "*.when", -} - -OPARSER_MANIFEST_ROOT = "<ROOT>" -OPARSER_PACKAGES_ROOT = "packages" -OPARSER_PACKAGES = "packages.{{PACKAGE}}" -OPARSER_MANIFEST_DEFINITIONS = "definitions" - -SUPPORTED_DISPATCHABLE_OBJECT_PARSERS = { - OPARSER_MANIFEST_ROOT: reference_documentation( - reference_documentation_url=f"{DEBPUTY_DOC_ROOT_DIR}/MANIFEST-FORMAT.md", - ), - OPARSER_MANIFEST_DEFINITIONS: reference_documentation( - title="Packager provided definitions", - description="Reusable packager provided definitions such as manifest variables.", - reference_documentation_url=f"{DEBPUTY_DOC_ROOT_DIR}/MANIFEST-FORMAT.md#packager-provided-definitions", - ), - OPARSER_PACKAGES: reference_documentation( - title="Binary package rules", - description=textwrap.dedent( - """\ - Inside the manifest, the `packages` mapping can be used to define requests for the binary packages - you want `debputy` to produce. Each key inside `packages` must be the name of a binary package - defined in `debian/control`. The value is a dictionary defining which features that `debputy` - should apply to that binary package. An example could be: - - packages: - foo: - transformations: - - create-symlink: - path: usr/share/foo/my-first-symlink - target: /usr/share/bar/symlink-target - - create-symlink: - path: usr/lib/{{DEB_HOST_MULTIARCH}}/my-second-symlink - target: /usr/lib/{{DEB_HOST_MULTIARCH}}/baz/symlink-target - bar: - transformations: - - create-directories: - - some/empty/directory.d - - another/empty/integration-point.d - - create-directories: - path: a/third-empty/directory.d - owner: www-data - group: www-data - - In this case, `debputy` will create some symlinks inside the `foo` package and some directories for - the `bar` package. The following subsections define the keys you can use under each binary package. - """ - ), - reference_documentation_url=f"{DEBPUTY_DOC_ROOT_DIR}/MANIFEST-FORMAT.md#binary-package-rules", - ), -} - - @dataclasses.dataclass(slots=True) class PluginProvidedManifestVariable: plugin_metadata: DebputyPluginMetadata @@ -1214,6 +1152,11 @@ class PluginProvidedKnownPackagingFile: plugin_metadata: DebputyPluginMetadata +class BuildSystemAutoDetector(Protocol): + + def __call__(self, source_root: VirtualPath, *args: Any, **kwargs: Any) -> bool: ... + + @dataclasses.dataclass(slots=True, frozen=True) class PluginProvidedTypeMapping: mapped_type: TypeMapping[Any, Any] @@ -1221,6 +1164,19 @@ class PluginProvidedTypeMapping: plugin_metadata: DebputyPluginMetadata +@dataclasses.dataclass(slots=True, frozen=True) +class PluginProvidedBuildSystemAutoDetection(Generic[BSR]): + manifest_keyword: str + build_system_rule_type: Type[BSR] + detector: BuildSystemAutoDetector + constructor: Callable[ + ["BuildRuleParsedFormat", AttributePath, "HighLevelManifest"], + BSR, + ] + auto_detection_shadow_build_systems: FrozenSet[str] + plugin_metadata: DebputyPluginMetadata + + class PackageDataTable: def __init__(self, package_data_table: Mapping[str, "BinaryPackageData"]) -> None: self._package_data_table = package_data_table diff --git a/src/debputy/plugin/api/parser_tables.py b/src/debputy/plugin/api/parser_tables.py new file mode 100644 index 0000000..37d3e37 --- /dev/null +++ b/src/debputy/plugin/api/parser_tables.py @@ -0,0 +1,67 @@ +import textwrap + +from debputy import DEBPUTY_DOC_ROOT_DIR +from debputy.installations import InstallRule +from debputy.maintscript_snippet import DpkgMaintscriptHelperCommand +from debputy.manifest_conditions import ManifestCondition +from debputy.plugin.api import reference_documentation +from debputy.plugin.debputy.to_be_api_types import BuildRule +from debputy.transformation_rules import TransformationRule + +SUPPORTED_DISPATCHABLE_TABLE_PARSERS = { + InstallRule: "installations", + TransformationRule: "packages.{{PACKAGE}}.transformations", + DpkgMaintscriptHelperCommand: "packages.{{PACKAGE}}.conffile-management", + ManifestCondition: "*.when", + BuildRule: "builds", +} + +OPARSER_MANIFEST_ROOT = "<ROOT>" +OPARSER_PACKAGES_ROOT = "packages" +OPARSER_PACKAGES = "packages.{{PACKAGE}}" +OPARSER_MANIFEST_DEFINITIONS = "definitions" + +SUPPORTED_DISPATCHABLE_OBJECT_PARSERS = { + OPARSER_MANIFEST_ROOT: reference_documentation( + reference_documentation_url=f"{DEBPUTY_DOC_ROOT_DIR}/MANIFEST-FORMAT.md", + ), + OPARSER_MANIFEST_DEFINITIONS: reference_documentation( + title="Packager provided definitions", + description="Reusable packager provided definitions such as manifest variables.", + reference_documentation_url=f"{DEBPUTY_DOC_ROOT_DIR}/MANIFEST-FORMAT.md#packager-provided-definitions", + ), + OPARSER_PACKAGES: reference_documentation( + title="Binary package rules", + description=textwrap.dedent( + """\ + Inside the manifest, the `packages` mapping can be used to define requests for the binary packages + you want `debputy` to produce. Each key inside `packages` must be the name of a binary package + defined in `debian/control`. The value is a dictionary defining which features that `debputy` + should apply to that binary package. An example could be: + + packages: + foo: + transformations: + - create-symlink: + path: usr/share/foo/my-first-symlink + target: /usr/share/bar/symlink-target + - create-symlink: + path: usr/lib/{{DEB_HOST_MULTIARCH}}/my-second-symlink + target: /usr/lib/{{DEB_HOST_MULTIARCH}}/baz/symlink-target + bar: + transformations: + - create-directories: + - some/empty/directory.d + - another/empty/integration-point.d + - create-directories: + path: a/third-empty/directory.d + owner: www-data + group: www-data + + In this case, `debputy` will create some symlinks inside the `foo` package and some directories for + the `bar` package. The following subsections define the keys you can use under each binary package. + """ + ), + reference_documentation_url=f"{DEBPUTY_DOC_ROOT_DIR}/MANIFEST-FORMAT.md#binary-package-rules", + ), +} diff --git a/src/debputy/plugin/api/plugin_parser.py b/src/debputy/plugin/api/plugin_parser.py index dd5c0d0..0e7954b 100644 --- a/src/debputy/plugin/api/plugin_parser.py +++ b/src/debputy/plugin/api/plugin_parser.py @@ -1,10 +1,10 @@ from typing import NotRequired, List, Any, TypedDict -from debputy.manifest_parser.base_types import ( +from debputy.manifest_parser.tagging_types import ( DebputyParsedContent, - OctalMode, TypeMapping, ) +from debputy.manifest_parser.base_types import OctalMode from debputy.manifest_parser.declarative_parser import ParserGenerator from debputy.plugin.api.impl_types import KnownPackagingFileInfo diff --git a/src/debputy/plugin/api/spec.py b/src/debputy/plugin/api/spec.py index b7f19c0..30308f9 100644 --- a/src/debputy/plugin/api/spec.py +++ b/src/debputy/plugin/api/spec.py @@ -25,6 +25,7 @@ from typing import ( Tuple, get_args, Container, + final, ) from debian.substvars import Substvars @@ -32,17 +33,23 @@ from debian.substvars import Substvars from debputy import util from debputy.exceptions import TestPathWithNonExistentFSPathError, PureVirtualPathError from debputy.interpreter import Interpreter, extract_shebang_interpreter_from_file +from debputy.manifest_parser.tagging_types import DebputyDispatchableType from debputy.manifest_parser.util import parse_symbolic_mode from debputy.packages import BinaryPackage from debputy.types import S if TYPE_CHECKING: + from debputy.plugin.debputy.to_be_api_types import BuildRule, BSR, BuildSystemRule + from debputy.plugin.api.impl_types import DIPHandler from debputy.manifest_parser.base_types import ( StaticFileSystemOwner, StaticFileSystemGroup, ) +DP = TypeVar("DP", bound=DebputyDispatchableType) + + PluginInitializationEntryPoint = Callable[["DebputyPluginInitializer"], None] MetadataAutoDetector = Callable[ ["VirtualPath", "BinaryCtrlAccessor", "PackageProcessingContext"], None @@ -86,17 +93,20 @@ DebputyIntegrationMode = Literal[ "dh-sequence-zz-debputy-rrr", ] +INTEGRATION_MODE_FULL: DebputyIntegrationMode = "full" INTEGRATION_MODE_DH_DEBPUTY_RRR: DebputyIntegrationMode = "dh-sequence-zz-debputy-rrr" INTEGRATION_MODE_DH_DEBPUTY: DebputyIntegrationMode = "dh-sequence-zz-debputy" ALL_DEBPUTY_INTEGRATION_MODES: FrozenSet[DebputyIntegrationMode] = frozenset( get_args(DebputyIntegrationMode) ) +_DEBPUTY_DISPATCH_METADATA_ATTR_NAME = "_debputy_dispatch_metadata" + def only_integrations( *integrations: DebputyIntegrationMode, ) -> Container[DebputyIntegrationMode]: - return frozenset(*integrations) + return frozenset(integrations) def not_integrations( @@ -212,6 +222,27 @@ class PathDef: materialized_content: Optional[str] = None +@dataclasses.dataclass(slots=True, frozen=True) +class DispatchablePluggableManifestRuleMetadata(Generic[DP]): + """NOT PUBLIC API (used internally by part of the public API)""" + + manifest_keywords: Sequence[str] + dispatched_type: Type[DP] + unwrapped_constructor: "DIPHandler" + expected_debputy_integration_mode: Optional[Container[DebputyIntegrationMode]] = ( + None + ) + online_reference_documentation: Optional["ParserDocumentation"] = None + apply_standard_attribute_documentation: bool = False + source_format: Optional[Any] = None + + +@dataclasses.dataclass(slots=True, frozen=True) +class BuildSystemManifestRuleMetadata(DispatchablePluggableManifestRuleMetadata): + build_system_impl: Optional[Type["BuildSystemRule"]] = (None,) + auto_detection_shadow_build_systems: FrozenSet[str] = frozenset() + + def virtual_path_def( path_name: str, /, @@ -1507,6 +1538,16 @@ class ParserAttributeDocumentation: attributes: FrozenSet[str] description: Optional[str] + @property + def is_hidden(self) -> bool: + return False + + +@final +@dataclasses.dataclass(slots=True, frozen=True) +class StandardParserAttributeDocumentation(ParserAttributeDocumentation): + sort_category: int = 0 + def undocumented_attr(attr: str) -> ParserAttributeDocumentation: """Describe an attribute as undocumented @@ -1514,6 +1555,8 @@ def undocumented_attr(attr: str) -> ParserAttributeDocumentation: If you for some reason do not want to document a particular attribute, you can mark it as undocumented. This is required if you are only documenting a subset of the attributes, because `debputy` assumes any omission to be a mistake. + + :param attr: Name of the attribute """ return ParserAttributeDocumentation( frozenset({attr}), diff --git a/src/debputy/plugin/api/std_docs.py b/src/debputy/plugin/api/std_docs.py new file mode 100644 index 0000000..f07c307 --- /dev/null +++ b/src/debputy/plugin/api/std_docs.py @@ -0,0 +1,142 @@ +import textwrap +from typing import Type, Sequence, Mapping, Container, Iterable, Any + +from debputy.manifest_parser.base_types import DebputyParsedContentStandardConditional +from debputy.manifest_parser.tagging_types import DebputyParsedContent +from debputy.plugin.api.spec import ( + ParserAttributeDocumentation, + StandardParserAttributeDocumentation, +) +from debputy.plugin.debputy.to_be_api_types import ( + OptionalInstallDirectly, + OptionalInSourceBuild, + OptionalBuildDirectory, + BuildRuleParsedFormat, +) + +_STD_ATTR_DOCS: Mapping[ + Type[DebputyParsedContent], + Sequence[ParserAttributeDocumentation], +] = { + BuildRuleParsedFormat: [ + StandardParserAttributeDocumentation( + frozenset(["name"]), + textwrap.dedent( + """\ + The name of the build step. + + The name is used for multiple things, such as: + 1) If you ever need to reference the build elsewhere, the name will be used. + 2) When `debputy` references the build in log output and error, it will use the name. + 3) It is used as defaults for when `debputy` derives build and `DESTDIR` directories + for the build. + """ + ), + # Put in top, + sort_category=-1000, + ), + StandardParserAttributeDocumentation( + frozenset(["for_packages"]), + textwrap.dedent( + """\ + Which package or packages this build step applies to. + + Either a package name or a list of package names. + """ + ), + ), + StandardParserAttributeDocumentation( + frozenset(["environment"]), + textwrap.dedent( + """\ + Specify that this build step uses the named environment + + If omitted, the default environment will be used. If no default environment is present, + then this option is mandatory. + """ + ), + ), + ], + OptionalBuildDirectory: [ + StandardParserAttributeDocumentation( + frozenset(["build_directory"]), + textwrap.dedent( + """\ + The build directory to use for the build. + + By default, `debputy` will derive a build directory automatically if the build system needs + it. However, it can be useful if you need to reference the directory name from other parts + of the manifest or want a "better" name than `debputy` comes up with. + """ + ), + ), + ], + OptionalInSourceBuild: [ + StandardParserAttributeDocumentation( + frozenset(["perform_in_source_build"]), + textwrap.dedent( + """\ + Whether the build system should use "in source" or "out of source" build. + + This is mostly useful for forcing "in source" builds for build systems that default to + "out of source" builds like `autoconf`. + + The default depends on the build system and the value of the `build-directory` attribute + (if supported by the build system). + """ + ), + # Late + sort_category=500, + ), + ], + OptionalInstallDirectly: [ + StandardParserAttributeDocumentation( + frozenset(["install_directly_to_package"]), + textwrap.dedent( + """\ + Whether the build system should install all upstream content directly into the package. + + This option is mostly useful for disabling said behavior by setting the attribute to `false`. + The attribute conditionally defaults to `true` when the build only applies to one package. + If explicitly set to `true`, then this build step must apply to exactly one package (usually + implying that `for` is set to that package when the source builds multiple packages). + + When `true`, this behaves similar to `dh_auto_install --destdir=debian/PACKAGE`. + """ + ), + ), + ], + DebputyParsedContentStandardConditional: [ + StandardParserAttributeDocumentation( + frozenset(["when"]), + textwrap.dedent( + """\ + A condition as defined in [Conditional rules]({MANIFEST_FORMAT_DOC}#Conditional rules). + + The conditional will disable the entire rule when the conditional evaluates to false. + """ + ), + # Last + sort_category=9999, + ), + ], +} + + +def docs_from( + *ts: Any, + exclude_attributes: Container[str] = frozenset(), +) -> Iterable[ParserAttributeDocumentation]: + """Provide standard attribute documentation from existing types + + This is a work-around for `apply_standard_attribute_documentation` requiring python3.12. + If you can assume python3.12, use `apply_standard_attribute_documentation` instead. + """ + for t in ts: + attrs = _STD_ATTR_DOCS.get(t) + if attrs is None: + raise ValueError(f"No standard documentation for {str(t)}") + for attr in attrs: + if any(a in exclude_attributes for a in attrs): + continue + yield attr diff --git a/src/debputy/plugin/debputy/binary_package_rules.py b/src/debputy/plugin/debputy/binary_package_rules.py index 98da763..45547b9 100644 --- a/src/debputy/plugin/debputy/binary_package_rules.py +++ b/src/debputy/plugin/debputy/binary_package_rules.py @@ -17,14 +17,10 @@ from typing import ( from debputy import DEBPUTY_DOC_ROOT_DIR from debputy.maintscript_snippet import DpkgMaintscriptHelperCommand, MaintscriptSnippet -from debputy.manifest_parser.base_types import ( - DebputyParsedContent, - FileSystemExactMatchRule, -) -from debputy.manifest_parser.declarative_parser import ( - DebputyParseHint, - ParserGenerator, -) +from debputy.manifest_parser.base_types import FileSystemExactMatchRule +from debputy.manifest_parser.tagging_types import DebputyParsedContent +from debputy.manifest_parser.parse_hints import DebputyParseHint +from debputy.manifest_parser.declarative_parser import ParserGenerator from debputy.manifest_parser.exceptions import ManifestParseException from debputy.manifest_parser.parser_data import ParserContextData from debputy.manifest_parser.util import AttributePath @@ -34,7 +30,7 @@ from debputy.plugin.api.impl import ( DebputyPluginInitializerProvider, ServiceDefinitionImpl, ) -from debputy.plugin.api.impl_types import OPARSER_PACKAGES +from debputy.plugin.api.parser_tables import OPARSER_PACKAGES from debputy.plugin.api.spec import ( ServiceUpgradeRule, ServiceDefinition, diff --git a/src/debputy/plugin/debputy/build_system_rules.py b/src/debputy/plugin/debputy/build_system_rules.py new file mode 100644 index 0000000..6d70d23 --- /dev/null +++ b/src/debputy/plugin/debputy/build_system_rules.py @@ -0,0 +1,2321 @@ +import dataclasses +import json +import os +import subprocess +import textwrap +from typing import ( + NotRequired, + TypedDict, + Self, + cast, + Dict, + Mapping, + Sequence, + MutableMapping, + Iterable, + Container, + List, + Tuple, + Union, + Optional, + TYPE_CHECKING, + Literal, +) + +from debian.debian_support import Version + +from debputy import DEBPUTY_DOC_ROOT_DIR +from debputy._manifest_constants import MK_BUILDS +from debputy.manifest_parser.base_types import ( + BuildEnvironmentDefinition, + DebputyParsedContentStandardConditional, + FileSystemExactMatchRule, +) +from debputy.manifest_parser.exceptions import ( + ManifestParseException, + ManifestInvalidUserDataException, +) +from debputy.manifest_parser.parser_data import ParserContextData +from debputy.manifest_parser.util import AttributePath +from debputy.plugin.api import reference_documentation +from debputy.plugin.api.impl import ( + DebputyPluginInitializerProvider, +) +from debputy.plugin.api.parser_tables import OPARSER_MANIFEST_ROOT +from debputy.plugin.api.spec import ( + documented_attr, + INTEGRATION_MODE_FULL, + only_integrations, + VirtualPath, +) +from debputy.plugin.api.std_docs import docs_from +from debputy.plugin.debputy.to_be_api_types import ( + BuildRule, + StepBasedBuildSystemRule, + OptionalInstallDirectly, + BuildSystemCharacteristics, + OptionalBuildDirectory, + OptionalInSourceBuild, + MakefileSupport, + BuildRuleParsedFormat, + debputy_build_system, + CleanHelper, + NinjaBuildSupport, +) +from debputy.types import EnvironmentModification +from debputy.util import ( + _warn, + run_build_system_command, + _error, + PerlConfigVars, + resolve_perl_config, + generated_content_dir, +) + +if TYPE_CHECKING: + from debputy.build_support.build_context import BuildContext + from debputy.highlevel_manifest import HighLevelManifest + + +PERL_CMD = "perl" + + +def register_build_system_rules(api: DebputyPluginInitializerProvider) -> None: + register_build_keywords(api) + register_build_rules(api) + + +def register_build_keywords(api: DebputyPluginInitializerProvider) -> None: + + api.pluggable_manifest_rule( + OPARSER_MANIFEST_ROOT, + "build-environments", + List[NamedEnvironmentSourceFormat], + _parse_build_environments, + expected_debputy_integration_mode=only_integrations(INTEGRATION_MODE_FULL), + inline_reference_documentation=reference_documentation( + title="Build Environments (`build-environments`)", + description=textwrap.dedent( + """\ + Define named environments to set the environment for any build commands that needs + a non-default environment. + + The environment definitions can be used to tweak the environment variables used by the + build commands. An example: + + build-environments: + - name: custom-env + set: + ENV_VAR: foo + ANOTHER_ENV_VAR: bar + builds: + - autoconf: + environment: custom-env + + The environment definition has multiple attributes for setting environment variables + which determines when the definition is applied. The resulting environment is the + result of the following order of operations. + + 1. The environment `debputy` received from its parent process. + 2. Apply all the variable definitions from `set` (if the attribute is present) + 3. Apply all computed variables (such as variables from `dpkg-buildflags`). + 4. Apply all the variable definitions from `override` (if the attribute is present) + 5. Remove all variables listed in `unset` (if the attribute is present). + + Accordingly, both `override` and `unset` will overrule any computed variables while + `set` will be overruled by any computed variables. + + Note that these variables are not available via manifest substitution (they are only + visible to build commands). They are only available to build commands. + """ + ), + attributes=[ + documented_attr( + "name", + textwrap.dedent( + """\ + The name of the environment + + The name is used to reference the environment from build rules. + """ + ), + ), + documented_attr( + "set", + textwrap.dedent( + """\ + A mapping of environment variables to be set. + + Note these environment variables are set before computed variables (such + as `dpkg-buildflags`) are provided. They can affect the content of the + computed variables, but they cannot overrule them. If you need to overrule + a computed variable, please use `override` instead. + """ + ), + ), + documented_attr( + "override", + textwrap.dedent( + """\ + A mapping of environment variables to set. + + Similar to `set`, but it can overrule computed variables like those from + `dpkg-buildflags`. + """ + ), + ), + documented_attr( + "unset", + textwrap.dedent( + """\ + A list of environment variables to unset. + + Any environment variable named here will be unset. No warnings or errors + will be raised if a given variable was not set. + """ + ), + ), + ], + reference_documentation_url=f"{DEBPUTY_DOC_ROOT_DIR}/MANIFEST-FORMAT.md#build-environment-build-environment", + ), + ) + api.pluggable_manifest_rule( + OPARSER_MANIFEST_ROOT, + "default-build-environment", + EnvironmentSourceFormat, + _parse_default_environment, + expected_debputy_integration_mode=only_integrations(INTEGRATION_MODE_FULL), + inline_reference_documentation=reference_documentation( + title="Default Build Environment (`default-build-environment`)", + description=textwrap.dedent( + """\ + Define the environment variables used in all build commands that uses the default + environment. + + The environment definition can be used to tweak the environment variables used by the + build commands. An example: + + default-build-environment: + set: + ENV_VAR: foo + ANOTHER_ENV_VAR: bar + + The environment definition has multiple attributes for setting environment variables + which determines when the definition is applied. The resulting environment is the + result of the following order of operations. + + 1. The environment `debputy` received from its parent process. + 2. Apply all the variable definitions from `set` (if the attribute is present) + 3. Apply all computed variables (such as variables from `dpkg-buildflags`). + 4. Apply all the variable definitions from `override` (if the attribute is present) + 5. Remove all variables listed in `unset` (if the attribute is present). + + Accordingly, both `override` and `unset` will overrule any computed variables while + `set` will be overruled by any computed variables. + + Note that these variables are not available via manifest substitution (they are only + visible to build commands). They are only available to build commands. + """ + ), + attributes=[ + documented_attr( + "set", + textwrap.dedent( + """\ + A mapping of environment variables to be set. + + Note these environment variables are set before computed variables (such + as `dpkg-buildflags`) are provided. They can affect the content of the + computed variables, but they cannot overrule them. If you need to overrule + a computed variable, please use `override` instead. + """ + ), + ), + documented_attr( + "override", + textwrap.dedent( + """\ + A mapping of environment variables to set. + + Similar to `set`, but it can overrule computed variables like those from + `dpkg-buildflags`. + """ + ), + ), + documented_attr( + "unset", + textwrap.dedent( + """\ + A list of environment variables to unset. + + Any environment variable named here will be unset. No warnings or errors + will be raised if a given variable was not set. + """ + ), + ), + ], + reference_documentation_url=f"{DEBPUTY_DOC_ROOT_DIR}/MANIFEST-FORMAT.md#build-environment-build-environment", + ), + ) + api.pluggable_manifest_rule( + OPARSER_MANIFEST_ROOT, + MK_BUILDS, + List[BuildRule], + _handle_build_rules, + expected_debputy_integration_mode=only_integrations(INTEGRATION_MODE_FULL), + inline_reference_documentation=reference_documentation( + title="Build rules", + description=textwrap.dedent( + """\ + Define how to build the upstream part of the package. Usually this is done via "build systems", + which also defines the clean rules. + """ + ), + ), + ) + + +def register_build_rules(api: DebputyPluginInitializerProvider) -> None: + api.register_build_system(ParsedAutoconfBuildRuleDefinition) + api.register_build_system(ParsedMakeBuildRuleDefinition) + + api.register_build_system(ParsedPerlBuildBuildRuleDefinition) + api.register_build_system(ParsedPerlMakeMakerBuildRuleDefinition) + api.register_build_system(ParsedDebhelperBuildRuleDefinition) + + api.register_build_system(ParsedCMakeBuildRuleDefinition) + api.register_build_system(ParsedMesonBuildRuleDefinition) + + api.register_build_system(ParsedQmakeBuildRuleDefinition) + api.register_build_system(ParsedQmake6BuildRuleDefinition) + + +class EnvironmentSourceFormat(TypedDict): + set: NotRequired[Dict[str, str]] + override: NotRequired[Dict[str, str]] + unset: NotRequired[List[str]] + + +class NamedEnvironmentSourceFormat(EnvironmentSourceFormat): + name: str + + +_READ_ONLY_ENV_VARS = { + "DEB_CHECK_COMMAND": None, + "DEB_SIGN_KEYID": None, + "DEB_SIGN_KEYFILE": None, + "DEB_BUILD_OPTIONS": "DEB_BUILD_MAINT_OPTIONS", + "DEB_BUILD_PROFILES": None, + "DEB_RULES_REQUIRES_ROOT": None, + "DEB_GAIN_ROOT_COMMAND": None, + "DH_EXTRA_ADDONS": None, + "DH_NO_ACT": None, +} + + +def _check_variables( + env_vars: Iterable[str], + attribute_path: AttributePath, +) -> None: + for env_var in env_vars: + if env_var not in _READ_ONLY_ENV_VARS: + continue + alt = _READ_ONLY_ENV_VARS.get(env_var) + var_path = attribute_path[env_var].path_key_lc + if alt is None: + raise ManifestParseException( + f"The variable {env_var} cannot be modified by the manifest. This restriction is generally" + f" because the build should not touch those variables or changing them have no effect" + f" (since the consumer will not see the change). The problematic definition was {var_path}" + ) + else: + raise ManifestParseException( + f"The variable {env_var} cannot be modified by the manifest. This restriction is generally" + f" because the build should not touch those variables or changing them have no effect" + f" (since the consumer will not see the change). Depending on what you are trying to" + f' accomplish, the variable "{alt}" might be a suitable alternative.' + f" The problematic definition was {var_path}" + ) + + +def _no_overlap( + lhs: Iterable[Union[str, Tuple[int, str]]], + rhs: Container[str], + lhs_key: str, + rhs_key: str, + redundant_key: str, + attribute_path: AttributePath, +) -> None: + for kt in lhs: + if isinstance(kt, tuple): + lhs_path_key, var = kt + else: + lhs_path_key = var = kt + if var not in rhs: + continue + lhs_path = attribute_path[lhs_key][lhs_path_key].path_key_lc + rhs_path = attribute_path[rhs_key][var].path_key_lc + r_path = lhs_path if redundant_key == rhs_key else rhs_path + raise ManifestParseException( + f"The environment variable {var} was declared in {lhs_path} and {rhs_path}." + f" Due to how the variables are applied, the definition in {r_path} is redundant" + f" and can effectively be removed. Please review the manifest and remove one of" + f" the two definitions." + ) + + +@dataclasses.dataclass(slots=True, frozen=True) +class ManifestProvidedBuildEnvironment(BuildEnvironmentDefinition): + + name: str + is_default: bool + attribute_path: AttributePath + parser_context: ParserContextData + + set_vars: Mapping[str, str] + override_vars: Mapping[str, str] + unset_vars: Sequence[str] + + @classmethod + def from_environment_definition( + cls, + env: EnvironmentSourceFormat, + attribute_path: AttributePath, + parser_context: ParserContextData, + is_default: bool = False, + ) -> Self: + reference_name: Optional[str] + if is_default: + name = "default-env" + reference_name = None + else: + named_env = cast("NamedEnvironmentSourceFormat", env) + name = named_env["name"] + reference_name = name + + set_vars = env.get("set", {}) + override_vars = env.get("override", {}) + unset_vars = env.get("unset", []) + _check_variables(set_vars, attribute_path["set"]) + _check_variables(override_vars, attribute_path["override"]) + _check_variables(unset_vars, attribute_path["unset"]) + + if not set_vars and not override_vars and not unset_vars: + raise ManifestParseException( + f"The environment definition {attribute_path.path_key_lc} was empty. Please provide" + " some content or delete the definition." + ) + + _no_overlap( + enumerate(unset_vars), + set_vars, + "unset", + "set", + "set", + attribute_path, + ) + _no_overlap( + enumerate(unset_vars), + override_vars, + "unset", + "override", + "override", + attribute_path, + ) + _no_overlap( + override_vars, + set_vars, + "override", + "set", + "set", + attribute_path, + ) + + r = cls( + name, + is_default, + attribute_path, + parser_context, + set_vars, + override_vars, + unset_vars, + ) + parser_context._register_build_environment( + reference_name, + r, + attribute_path, + is_default, + ) + + return r + + def update_env(self, env: MutableMapping[str, str]) -> None: + if set_vars := self.set_vars: + env.update(set_vars) + dpkg_env = self.dpkg_buildflags_env(env, self.attribute_path.path_key_lc) + self.log_computed_env(f"dpkg-buildflags [{self.name}]", dpkg_env) + if overlapping_env := dpkg_env.keys() & set_vars.keys(): + for var in overlapping_env: + key_lc = self.attribute_path["set"][var].path_key_lc + _warn( + f'The variable "{var}" defined at {key_lc} is shadowed by a computed variable.' + f" If the manifest definition is more important, please define it via `override` rather than" + f" `set`." + ) + env.update(dpkg_env) + if override_vars := self.override_vars: + env.update(override_vars) + if unset_vars := self.unset_vars: + for var in unset_vars: + try: + del env[var] + except KeyError: + pass + + +_MAKE_DEFAULT_TOOLS = [ + ("CC", "gcc"), + ("CXX", "g++"), + ("PKG_CONFIG", "pkg-config"), +] + + +class MakefileBuildSystemRule(StepBasedBuildSystemRule): + + __slots__ = ("_make_support", "_build_target", "_install_target", "_directory") + + def __init__( + self, + attributes: "ParsedMakeBuildRuleDefinition", + attribute_path: AttributePath, + parser_context: Union[ParserContextData, "HighLevelManifest"], + ) -> None: + super().__init__(attributes, attribute_path, parser_context) + directory = attributes.get("directory") + if directory is not None: + self._directory = directory.match_rule.path + else: + self._directory = None + self._make_support = MakefileSupport.from_build_system(self) + self._build_target = attributes.get("build_target") + self._test_target = attributes.get("test_target") + self._install_target = attributes.get("install_target") + + @classmethod + def auto_detect_build_system( + cls, + source_root: VirtualPath, + *args, + **kwargs, + ) -> bool: + return any(p in source_root for p in ("Makefile", "makefile", "GNUmakefile")) + + @classmethod + def characteristics(cls) -> BuildSystemCharacteristics: + return BuildSystemCharacteristics( + out_of_source_builds="not-supported", + ) + + def configure_impl( + self, + context: "BuildContext", + manifest: "HighLevelManifest", + **kwargs, + ) -> None: + # No configure step + pass + + def build_impl( + self, + context: "BuildContext", + manifest: "HighLevelManifest", + **kwargs, + ) -> None: + extra_vars = [] + build_target = self._build_target + if build_target is not None: + extra_vars.append(build_target) + if context.is_cross_compiling: + for envvar, tool in _MAKE_DEFAULT_TOOLS: + cross_tool = os.environ.get(envvar) + if cross_tool is None: + cross_tool = context.cross_tool(tool) + extra_vars.append(f"{envvar}={cross_tool}") + self._make_support.run_make( + context, + *extra_vars, + "INSTALL=install --strip-program=true", + directory=self._directory, + ) + + def test_impl( + self, + context: "BuildContext", + manifest: "HighLevelManifest", + **kwargs, + ) -> None: + self._run_make_maybe_explicit_target( + context, + self._test_target, + ["test", "check"], + ) + + def install_impl( + self, + context: "BuildContext", + manifest: "HighLevelManifest", + dest_dir: str, + **kwargs, + ) -> None: + self._run_make_maybe_explicit_target( + context, + self._install_target, + ["install"], + f"DESTDIR={dest_dir}", + "AM_UPDATE_INFO_DIR=no", + "INSTALL=install --strip-program=true", + ) + + def _run_make_maybe_explicit_target( + self, + context: "BuildContext", + provided_target: Optional[str], + fallback_targets: Sequence[str], + *make_args: str, + ) -> None: + make_support = self._make_support + if provided_target is not None: + make_support.run_make( + context, + provided_target, + *make_args, + directory=self._directory, + ) + else: + make_support.run_first_existing_target_if_any( + context, + fallback_targets, + *make_args, + directory=self._directory, + ) + + def clean_impl( + self, + context: "BuildContext", + manifest: "HighLevelManifest", + clean_helper: "CleanHelper", + **kwargs, + ) -> None: + self._make_support.run_first_existing_target_if_any( + context, + ["distclean", "realclean", "clean"], + ) + + +class PerlBuildBuildSystemRule(StepBasedBuildSystemRule): + + __slots__ = "configure_args" + + def __init__( + self, + attributes: "ParsedPerlBuildBuildRuleDefinition", + attribute_path: AttributePath, + parser_context: Union[ParserContextData, "HighLevelManifest"], + ) -> None: + super().__init__(attributes, attribute_path, parser_context) + self.configure_args = attributes.get("configure_args", []) + + @classmethod + def auto_detect_build_system( + cls, + source_root: VirtualPath, + *args, + **kwargs, + ) -> bool: + return "Build.PL" in source_root + + @classmethod + def characteristics(cls) -> BuildSystemCharacteristics: + return BuildSystemCharacteristics( + out_of_source_builds="not-supported", + ) + + @staticmethod + def _perl_cross_build_env( + context: "BuildContext", + ) -> Tuple[PerlConfigVars, Optional[EnvironmentModification]]: + perl_config_data = resolve_perl_config( + context.dpkg_architecture_variables, + None, + ) + if context.is_cross_compiling: + perl5lib_dir = perl_config_data.cross_inc_dir + if perl5lib_dir is not None: + env_perl5lib = os.environ.get("PERL5LIB") + if env_perl5lib is not None: + perl5lib_dir = ( + perl5lib_dir + perl_config_data.path_sep + env_perl5lib + ) + env_mod = EnvironmentModification( + replacements=(("PERL5LIB", perl5lib_dir),), + ) + return perl_config_data, env_mod + return perl_config_data, None + + def configure_impl( + self, + context: "BuildContext", + manifest: "HighLevelManifest", + **kwargs, + ) -> None: + perl_config_data, cross_env_mod = self._perl_cross_build_env(context) + configure_env = EnvironmentModification( + replacements=( + ("PERL_MM_USE_DEFAULT", "1"), + ("PKG_CONFIG", context.cross_tool("pkg-config")), + ) + ) + if cross_env_mod is not None: + configure_env = configure_env.combine(cross_env_mod) + + configure_cmd = [ + PERL_CMD, + "Build.PL", + "--installdirs", + "vendor", + ] + cflags = os.environ.get("CFLAGS", "") + cppflags = os.environ.get("CPPFLAGS", "") + ldflags = os.environ.get("LDFLAGS", "") + + if cflags != "" or cppflags != "": + configure_cmd.append("--config") + combined = f"{cflags} {cppflags}".strip() + configure_cmd.append(f"optimize={combined}") + + if ldflags != "" or cflags != "" or context.is_cross_compiling: + configure_cmd.append("--config") + combined = f"{perl_config_data.ld} {cflags} {ldflags}".strip() + configure_cmd.append(f"ld={combined}") + if self.configure_args: + substitution = self.substitution + attr_path = self.attribute_path["configure_args"] + configure_cmd.extend( + substitution.substitute(v, attr_path[i].path) + for i, v in enumerate(self.configure_args) + ) + run_build_system_command(*configure_cmd, env_mod=configure_env) + + def build_impl( + self, + context: "BuildContext", + manifest: "HighLevelManifest", + **kwargs, + ) -> None: + _, cross_env_mod = self._perl_cross_build_env(context) + run_build_system_command(PERL_CMD, "Build", env_mod=cross_env_mod) + + def test_impl( + self, + context: "BuildContext", + manifest: "HighLevelManifest", + **kwargs, + ) -> None: + _, cross_env_mod = self._perl_cross_build_env(context) + run_build_system_command( + PERL_CMD, + "Build", + "test", + "--verbose", + "1", + env_mod=cross_env_mod, + ) + + def install_impl( + self, + context: "BuildContext", + manifest: "HighLevelManifest", + dest_dir: str, + **kwargs, + ) -> None: + _, cross_env_mod = self._perl_cross_build_env(context) + run_build_system_command( + PERL_CMD, + "Build", + "install", + "--destdir", + dest_dir, + "--create_packlist", + "0", + env_mod=cross_env_mod, + ) + + def clean_impl( + self, + context: "BuildContext", + manifest: "HighLevelManifest", + clean_helper: "CleanHelper", + **kwargs, + ) -> None: + _, cross_env_mod = self._perl_cross_build_env(context) + if os.path.lexists("Build"): + run_build_system_command( + PERL_CMD, + "Build", + "realclean", + "--allow_mb_mismatch", + "1", + env_mod=cross_env_mod, + ) + + +class PerlMakeMakerBuildSystemRule(StepBasedBuildSystemRule): + + __slots__ = ("configure_args", "_make_support") + + def __init__( + self, + attributes: "ParsedPerlBuildBuildRuleDefinition", + attribute_path: AttributePath, + parser_context: Union[ParserContextData, "HighLevelManifest"], + ) -> None: + super().__init__(attributes, attribute_path, parser_context) + self.configure_args = attributes.get("configure_args", []) + self._make_support = MakefileSupport.from_build_system(self) + + @classmethod + def auto_detect_build_system( + cls, + source_root: VirtualPath, + *args, + **kwargs, + ) -> bool: + return "Makefile.PL" in source_root + + @classmethod + def characteristics(cls) -> BuildSystemCharacteristics: + return BuildSystemCharacteristics( + out_of_source_builds="not-supported", + ) + + def configure_impl( + self, + context: "BuildContext", + manifest: "HighLevelManifest", + **kwargs, + ) -> None: + configure_env = EnvironmentModification( + replacements=( + ("PERL_MM_USE_DEFAULT", "1"), + ("PERL_AUTOINSTALL", "--skipdeps"), + ("PKG_CONFIG", context.cross_tool("pkg-config")), + ) + ) + perl_args = [] + mm_args = ["INSTALLDIRS=vendor"] + if "CFLAGS" in os.environ: + mm_args.append( + f"OPTIMIZE={os.environ['CFLAGS']} {os.environ['CPPFLAGS']}".rstrip() + ) + + perl_config_data = resolve_perl_config( + context.dpkg_architecture_variables, + None, + ) + + if "LDFLAGS" in os.environ: + mm_args.append( + f"LD={perl_config_data.ld} {os.environ['CFLAGS']} {os.environ['LDFLAGS']}" + ) + + if context.is_cross_compiling: + perl5lib_dir = perl_config_data.cross_inc_dir + if perl5lib_dir is not None: + perl_args.append(f"-I{perl5lib_dir}") + + if self.configure_args: + substitution = self.substitution + attr_path = self.attribute_path["configure_args"] + mm_args.extend( + substitution.substitute(v, attr_path[i].path) + for i, v in enumerate(self.configure_args) + ) + run_build_system_command( + PERL_CMD, + *perl_args, + "Makefile.PL", + *mm_args, + env_mod=configure_env, + ) + + def build_impl( + self, + context: "BuildContext", + manifest: "HighLevelManifest", + **kwargs, + ) -> None: + self._make_support.run_make(context) + + def test_impl( + self, + context: "BuildContext", + manifest: "HighLevelManifest", + **kwargs, + ) -> None: + self._make_support.run_first_existing_target_if_any( + context, + ["check", "test"], + "TEST_VERBOSE=1", + ) + + def install_impl( + self, + context: "BuildContext", + manifest: "HighLevelManifest", + dest_dir: str, + **kwargs, + ) -> None: + is_mm_makefile = False + with open("Makefile", "rb") as fd: + for line in fd: + if b"generated automatically by MakeMaker" in line: + is_mm_makefile = True + break + + install_args = [f"DESTDIR={dest_dir}"] + + # Special case for Makefile.PL that uses + # Module::Build::Compat. PREFIX should not be passed + # for those; it already installs into /usr by default. + if is_mm_makefile: + install_args.append("PREFIX=/usr") + + self._make_support.run_first_existing_target_if_any( + context, + ["install"], + *install_args, + ) + + def clean_impl( + self, + context: "BuildContext", + manifest: "HighLevelManifest", + clean_helper: "CleanHelper", + **kwargs, + ) -> None: + self._make_support.run_first_existing_target_if_any( + context, + ["distclean", "realclean", "clean"], + ) + + +class DebhelperBuildSystemRule(StepBasedBuildSystemRule): + + __slots__ = ("configure_args", "dh_build_system") + + def __init__( + self, + parsed_data: "ParsedDebhelperBuildRuleDefinition", + attribute_path: AttributePath, + parser_context: Union[ParserContextData, "HighLevelManifest"], + ) -> None: + super().__init__(parsed_data, attribute_path, parser_context) + self.configure_args = parsed_data.get("configure_args", []) + self.dh_build_system = parsed_data.get("dh_build_system") + + @classmethod + def auto_detect_build_system( + cls, + source_root: VirtualPath, + *args, + **kwargs, + ) -> bool: + try: + v = subprocess.check_output( + ["dh_assistant", "which-build-system"], + # Packages without `debhelper-compat` will trigger an error, which will just be noise + stderr=subprocess.DEVNULL, + cwd=source_root.fs_path, + ) + except subprocess.CalledProcessError: + return False + else: + d = json.loads(v) + build_system = d.get("build-system") + return build_system is not None + + @classmethod + def characteristics(cls) -> BuildSystemCharacteristics: + return BuildSystemCharacteristics( + out_of_source_builds="supported-but-not-default", + ) + + def before_first_impl_step( + self, *, stage: Literal["build", "clean"], **kwargs + ) -> None: + dh_build_system = self.dh_build_system + if dh_build_system is None: + return + try: + subprocess.check_call( + ["dh_assistant", "which-build-system", f"-S{dh_build_system}"] + ) + except FileNotFoundError: + _error( + "The debhelper build system assumes `dh_assistant` is available (`debhelper (>= 13.5~)`)" + ) + except subprocess.SubprocessError: + raise ManifestInvalidUserDataException( + f'The debhelper build system "{dh_build_system}" does not seem to' + f" be available according to" + f" `dh_assistant which-build-system -S{dh_build_system}`" + ) from None + + def _default_options(self) -> List[str]: + default_options = [] + if self.dh_build_system is not None: + default_options.append(f"-S{self.dh_build_system}") + if self.build_directory is not None: + default_options.append(f"-B{self.build_directory}") + + return default_options + + def configure_impl( + self, + context: "BuildContext", + manifest: "HighLevelManifest", + **kwargs, + ) -> None: + if ( + os.path.lexists("configure.ac") or os.path.lexists("configure.in") + ) and not os.path.lexists("debian/autoreconf.before"): + run_build_system_command("dh_update_autotools_config") + run_build_system_command("dh_autoreconf") + + default_options = self._default_options() + configure_args = default_options.copy() + if self.configure_args: + configure_args.append("--") + substitution = self.substitution + attr_path = self.attribute_path["configure_args"] + configure_args.extend( + substitution.substitute(v, attr_path[i].path) + for i, v in enumerate(self.configure_args) + ) + run_build_system_command("dh_auto_configure", *configure_args) + + def build_impl( + self, + context: "BuildContext", + manifest: "HighLevelManifest", + **kwargs, + ) -> None: + default_options = self._default_options() + run_build_system_command("dh_auto_build", *default_options) + + def test_impl( + self, + context: "BuildContext", + manifest: "HighLevelManifest", + **kwargs, + ) -> None: + default_options = self._default_options() + run_build_system_command("dh_auto_test", *default_options) + + def install_impl( + self, + context: "BuildContext", + manifest: "HighLevelManifest", + dest_dir: str, + **kwargs, + ) -> None: + default_options = self._default_options() + run_build_system_command( + "dh_auto_install", + *default_options, + f"--destdir={dest_dir}", + ) + + def clean_impl( + self, + context: "BuildContext", + manifest: "HighLevelManifest", + clean_helper: "CleanHelper", + **kwargs, + ) -> None: + default_options = self._default_options() + run_build_system_command("dh_auto_clean", *default_options) + # The "global" clean logic takes care of `dh_autoreconf_clean` and `dh_clean` + + +class AutoconfBuildSystemRule(StepBasedBuildSystemRule): + + __slots__ = ("configure_args", "_make_support") + + def __init__( + self, + parsed_data: "ParsedAutoconfBuildRuleDefinition", + attribute_path: AttributePath, + parser_context: Union[ParserContextData, "HighLevelManifest"], + ) -> None: + super().__init__(parsed_data, attribute_path, parser_context) + configure_args = [a for a in parsed_data.get("configure_args", [])] + self.configure_args = configure_args + self._make_support = MakefileSupport.from_build_system(self) + + @classmethod + def characteristics(cls) -> BuildSystemCharacteristics: + return BuildSystemCharacteristics( + out_of_source_builds="supported-and-default", + ) + + @classmethod + def auto_detect_build_system( + cls, + source_root: VirtualPath, + *args, + **kwargs, + ) -> bool: + if "configure.ac" in source_root: + return True + configure_in = source_root.get("configure.in") + if configure_in is not None and configure_in.is_file: + with configure_in.open(byte_io=True, buffering=4096) as fd: + for no, line in enumerate(fd): + if no > 100: + break + if b"AC_INIT" in line or b"AC_PREREQ" in line: + return True + configure = source_root.get("configure") + if configure is None or not configure.is_executable or not configure.is_file: + return False + with configure.open(byte_io=True, buffering=4096) as fd: + for no, line in enumerate(fd): + if no > 10: + break + if b"GNU Autoconf" in line: + return True + return False + + def configure_impl( + self, + context: "BuildContext", + manifest: "HighLevelManifest", + **kwargs, + ) -> None: + if ( + os.path.lexists("configure.ac") or os.path.lexists("configure.in") + ) and not os.path.lexists("debian/autoreconf.before"): + run_build_system_command("dh_update_autotools_config") + run_build_system_command("dh_autoreconf") + + dpkg_architecture_variables = context.dpkg_architecture_variables + multi_arch = dpkg_architecture_variables.current_host_multiarch + silent_rules = ( + "--enable-silent-rules" + if context.is_terse_build + else "--disable-silent-rules" + ) + + configure_args = [ + f"--build={dpkg_architecture_variables['DEB_BUILD_GNU_TYPE']}", + "--prefix=/usr", + "--includedir=${prefix}/include", + "--mandir=${prefix}/share/man", + "--infodir=${prefix}/share/info", + "--sysconfdir=/etc", + "--localstatedir=/var", + "--disable-option-checking", + silent_rules, + f"--libdir=${{prefix}}/{multi_arch}", + "--runstatedir=/run", + "--disable-maintainer-mode", + "--disable-dependency-tracking", + ] + if dpkg_architecture_variables.is_cross_compiling: + configure_args.append( + f"--host={dpkg_architecture_variables['DEB_HOST_GNU_TYPE']}" + ) + if self.configure_args: + substitution = self.substitution + attr_path = self.attribute_path["configure_args"] + configure_args.extend( + substitution.substitute(v, attr_path[i].path) + for i, v in enumerate(self.configure_args) + ) + self.ensure_build_dir_exists() + configure_script = self.relative_from_builddir_to_source("configure") + with self.dump_logs_on_error("config.log"): + run_build_system_command( + configure_script, + *configure_args, + cwd=self.build_directory, + ) + + def build_impl( + self, + context: "BuildContext", + manifest: "HighLevelManifest", + **kwargs, + ) -> None: + self._make_support.run_make(context) + + def test_impl( + self, + context: "BuildContext", + manifest: "HighLevelManifest", + **kwargs, + ) -> None: + limit = context.parallelization_limit(support_zero_as_unlimited=True) + testsuite_flags = [f"-j{limit}"] if limit else ["-j"] + + if not context.is_terse_build: + testsuite_flags.append("--verbose") + self._make_support.run_first_existing_target_if_any( + context, + # Order is deliberately inverse compared to debhelper (#924052) + ["check", "test"], + f"TESTSUITEFLAGS={' '.join(testsuite_flags)}", + "VERBOSE=1", + ) + + def install_impl( + self, + context: "BuildContext", + manifest: "HighLevelManifest", + dest_dir: str, + **kwargs, + ) -> None: + enable_parallelization = not os.path.lexists(self.build_dir_path("libtool")) + self._make_support.run_first_existing_target_if_any( + context, + ["install"], + f"DESTDIR={dest_dir}", + "AM_UPDATE_INFO_DIR=no", + enable_parallelization=enable_parallelization, + ) + + def clean_impl( + self, + context: "BuildContext", + manifest: "HighLevelManifest", + clean_helper: "CleanHelper", + **kwargs, + ) -> None: + if self.out_of_source_build: + return + self._make_support.run_first_existing_target_if_any( + context, + ["distclean", "realclean", "clean"], + ) + # The "global" clean logic takes care of `dh_autoreconf_clean` and `dh_clean` + + +class CMakeBuildSystemRule(StepBasedBuildSystemRule): + + __slots__ = ( + "configure_args", + "target_build_system", + "_make_support", + "_ninja_support", + ) + + def __init__( + self, + parsed_data: "ParsedCMakeBuildRuleDefinition", + attribute_path: AttributePath, + parser_context: Union[ParserContextData, "HighLevelManifest"], + ) -> None: + super().__init__(parsed_data, attribute_path, parser_context) + configure_args = [a for a in parsed_data.get("configure_args", [])] + self.configure_args = configure_args + self.target_build_system: Literal["make", "ninja"] = parsed_data.get( + "target_build_system", "make" + ) + self._make_support = MakefileSupport.from_build_system(self) + self._ninja_support = NinjaBuildSupport.from_build_system(self) + + @classmethod + def characteristics(cls) -> BuildSystemCharacteristics: + return BuildSystemCharacteristics( + out_of_source_builds="required", + ) + + @classmethod + def auto_detect_build_system( + cls, + source_root: VirtualPath, + *args, + **kwargs, + ) -> bool: + return "CMakeLists.txt" in source_root + + @staticmethod + def _default_cmake_env( + build_context: "BuildContext", + ) -> EnvironmentModification: + replacements = {} + if "DEB_PYTHON_INSTALL_LAYOUT" not in os.environ: + replacements["DEB_PYTHON_INSTALL_LAYOUT"] = "deb" + if "PKG_CONFIG" not in os.environ: + replacements["PKG_CONFIG"] = build_context.cross_tool("pkg-config") + return EnvironmentModification( + replacements=tuple((k, v) for k, v in replacements.items()) + ) + + @classmethod + def cmake_generator(cls, target_build_system: Literal["make", "ninja"]) -> str: + cmake_generators = { + "make": "Unix Makefiles", + "ninja": "Ninja", + } + return cmake_generators[target_build_system] + + @staticmethod + def _compiler_and_cross_flags( + context: "BuildContext", + cmake_flags: List[str], + ) -> None: + + if "CC" in os.environ: + cmake_flags.append(f"-DCMAKE_C_COMPILER={os.environ['CC']}") + elif context.is_cross_compiling: + cmake_flags.append(f"-DCMAKE_C_COMPILER={context.cross_tool('gcc')}") + + if "CXX" in os.environ: + cmake_flags.append(f"-DCMAKE_CXX_COMPILER={os.environ['CXX']}") + elif context.is_cross_compiling: + cmake_flags.append(f"-DCMAKE_CXX_COMPILER={context.cross_tool('g++')}") + + if context.is_cross_compiling: + deb_host2cmake_system = { + "linux": "Linux", + "kfreebsd": "kFreeBSD", + "hurd": "GNU", + } + + gnu_cpu2system_processor = { + "arm": "armv7l", + "misp64el": "mips64", + "powerpc64le": "ppc64le", + } + dpkg_architecture_variables = context.dpkg_architecture_variables + + try: + system_name = deb_host2cmake_system[ + dpkg_architecture_variables["DEB_HOST_ARCH_OS"] + ] + except KeyError as e: + name = e.args[0] + _error( + f"Cannot cross-compile via cmake: Missing CMAKE_SYSTEM_NAME for the DEB_HOST_ARCH_OS {name}" + ) + + gnu_cpu = dpkg_architecture_variables["DEB_HOST_GNU_CPU"] + system_processor = gnu_cpu2system_processor.get(gnu_cpu, gnu_cpu) + + cmake_flags.append(f"-DCMAKE_SYSTEM_NAME={system_name}") + cmake_flags.append(f"-DCMAKE_SYSTEM_PROCESSOR={system_processor}") + + pkg_config = context.cross_tool("pkg-config") + # Historical uses. Current versions of cmake uses the env variable instead. + cmake_flags.append(f"-DPKG_CONFIG_EXECUTABLE=/usr/bin/{pkg_config}") + cmake_flags.append(f"-DPKGCONFIG_EXECUTABLE=/usr/bin/{pkg_config}") + cmake_flags.append( + f"-DQMAKE_EXECUTABLE=/usr/bin/{context.cross_tool('qmake')}" + ) + + def configure_impl( + self, + context: "BuildContext", + manifest: "HighLevelManifest", + **kwargs, + ) -> None: + cmake_flags = [ + "-DCMAKE_INSTALL_PREFIX=/usr", + "-DCMAKE_BUILD_TYPE=None", + "-DCMAKE_INSTALL_SYSCONFDIR=/etc", + "-DCMAKE_INSTALL_LOCALSTATEDIR=/var", + "-DCMAKE_EXPORT_NO_PACKAGE_REGISTRY=ON", + "-DCMAKE_FIND_USE_PACKAGE_REGISTRY=OFF", + "-DCMAKE_FIND_PACKAGE_NO_PACKAGE_REGISTRY=ON", + "-DFETCHCONTENT_FULLY_DISCONNECTED=ON", + "-DCMAKE_INSTALL_RUNSTATEDIR=/run", + "-DCMAKE_SKIP_INSTALL_ALL_DEPENDENCY=ON", + "-DCMAKE_BUILD_RPATH_USE_ORIGIN=ON", + f"-G{self.cmake_generator(self.target_build_system)}", + ] + if not context.is_terse_build: + cmake_flags.append("-DCMAKE_VERBOSE_MAKEFILE=ON") + + self._compiler_and_cross_flags(context, cmake_flags) + + if self.configure_args: + substitution = self.substitution + attr_path = self.attribute_path["configure_args"] + cmake_flags.extend( + substitution.substitute(v, attr_path[i].path) + for i, v in enumerate(self.configure_args) + ) + + env_mod = self._default_cmake_env(context) + if "CPPFLAGS" in os.environ: + # CMake doesn't respect CPPFLAGS, see #653916. + cppflags = os.environ["CPPFLAGS"] + cflags = os.environ.get("CFLAGS", "") + f" {cppflags}".lstrip() + cxxflags = os.environ.get("CXXFLAGS", "") + f" {cppflags}".lstrip() + env_mod = env_mod.combine( + # The debhelper build system never showed this delta, so people might find it annoying. + EnvironmentModification( + replacements=( + ("CFLAGS", cflags), + ("CXXFLAGS", cxxflags), + ) + ) + ) + if "ASMFLAGS" not in os.environ and "ASFLAGS" in os.environ: + env_mod = env_mod.combine( + # The debhelper build system never showed this delta, so people might find it annoying. + EnvironmentModification( + replacements=(("ASMFLAGS", os.environ["ASFLAGS"]),), + ) + ) + self.ensure_build_dir_exists() + source_dir_from_build_dir = self.relative_from_builddir_to_source() + + with self.dump_logs_on_error( + "CMakeCache.txt", + "CMakeFiles/CMakeOutput.log", + "CMakeFiles/CMakeError.log", + ): + run_build_system_command( + "cmake", + *cmake_flags, + source_dir_from_build_dir, + cwd=self.build_directory, + env_mod=env_mod, + ) + + def build_impl( + self, + context: "BuildContext", + manifest: "HighLevelManifest", + **kwargs, + ) -> None: + if self.target_build_system == "make": + make_flags = [] + if not context.is_terse_build: + make_flags.append("VERBOSE=1") + self._make_support.run_make(context, *make_flags) + else: + self._ninja_support.run_ninja_build(context) + + def test_impl( + self, + context: "BuildContext", + manifest: "HighLevelManifest", + **kwargs, + ) -> None: + env_mod = EnvironmentModification( + replacements=(("CTEST_OUTPUT_ON_FAILURE", "1"),), + ) + if self.target_build_system == "make": + # Unlike make, CTest does not have "unlimited parallel" setting (-j implies + # -j1). Therefore, we do not set "allow zero as unlimited" here. + make_flags = [f"ARGS+=-j{context.parallelization_limit()}"] + if not context.is_terse_build: + make_flags.append("ARGS+=--verbose") + self._make_support.run_first_existing_target_if_any( + context, + ["check", "test"], + *make_flags, + env_mod=env_mod, + ) + else: + self._ninja_support.run_ninja_test(context, env_mod=env_mod) + + limit = context.parallelization_limit(support_zero_as_unlimited=True) + testsuite_flags = [f"-j{limit}"] if limit else ["-j"] + + if not context.is_terse_build: + testsuite_flags.append("--verbose") + self._make_support.run_first_existing_target_if_any( + context, + # Order is deliberately inverse compared to debhelper (#924052) + ["check", "test"], + f"TESTSUITEFLAGS={' '.join(testsuite_flags)}", + "VERBOSE=1", + ) + + def install_impl( + self, + context: "BuildContext", + manifest: "HighLevelManifest", + dest_dir: str, + **kwargs, + ) -> None: + env_mod = EnvironmentModification( + replacements=( + ("LC_ALL", "C.UTF-8"), + ("DESTDIR", dest_dir), + ) + ).combine(self._default_cmake_env(context)) + run_build_system_command( + "cmake", + "--install", + self.build_directory, + env_mod=env_mod, + ) + + def clean_impl( + self, + context: "BuildContext", + manifest: "HighLevelManifest", + clean_helper: "CleanHelper", + **kwargs, + ) -> None: + if self.out_of_source_build: + return + if self.target_build_system == "make": + # Keep it here in case we change the `required` "out of source" to "supported-default" + self._make_support.run_first_existing_target_if_any( + context, + ["distclean", "realclean", "clean"], + ) + else: + self._ninja_support.run_ninja_clean(context) + + +class MesonBuildSystemRule(StepBasedBuildSystemRule): + + __slots__ = ( + "configure_args", + "_ninja_support", + ) + + def __init__( + self, + parsed_data: "ParsedMesonBuildRuleDefinition", + attribute_path: AttributePath, + parser_context: Union[ParserContextData, "HighLevelManifest"], + ) -> None: + super().__init__(parsed_data, attribute_path, parser_context) + configure_args = [a for a in parsed_data.get("configure_args", [])] + self.configure_args = configure_args + self._ninja_support = NinjaBuildSupport.from_build_system(self) + + @classmethod + def characteristics(cls) -> BuildSystemCharacteristics: + return BuildSystemCharacteristics( + out_of_source_builds="required", + ) + + @classmethod + def auto_detect_build_system( + cls, + source_root: VirtualPath, + *args, + **kwargs, + ) -> bool: + return "meson.build" in source_root + + @staticmethod + def _default_meson_env() -> EnvironmentModification: + replacements = { + "LC_ALL": "C.UTF-8", + } + if "DEB_PYTHON_INSTALL_LAYOUT" not in os.environ: + replacements["DEB_PYTHON_INSTALL_LAYOUT"] = "deb" + return EnvironmentModification( + replacements=tuple((k, v) for k, v in replacements.items()) + ) + + @classmethod + def cmake_generator(cls, target_build_system: Literal["make", "ninja"]) -> str: + cmake_generators = { + "make": "Unix Makefiles", + "ninja": "Ninja", + } + return cmake_generators[target_build_system] + + @staticmethod + def _cross_flags( + context: "BuildContext", + meson_flags: List[str], + ) -> None: + if not context.is_cross_compiling: + return + # Needs a cross-file http://mesonbuild.com/Cross-compilation.html + cross_files_dir = os.path.abspath( + generated_content_dir( + subdir_key="meson-cross-files", + ) + ) + host_arch = context.dpkg_architecture_variables.current_host_arch + cross_file = os.path.join(cross_files_dir, f"meson-cross-file-{host_arch}.conf") + if not os.path.isfile(cross_file): + env = os.environ + if env.get("LC_ALL") != "C.UTF-8": + env = dict(env) + env["LC_ALL"] = "C.UTF-8" + else: + env = None + subprocess.check_call( + [ + "/usr/share/meson/debcrossgen", + f"--arch={host_arch}", + f"-o{cross_file}", + ], + stdout=subprocess.DEVNULL, + env=env, + ) + + meson_flags.append("--cross-file") + meson_flags.append(cross_file) + + def configure_impl( + self, + context: "BuildContext", + manifest: "HighLevelManifest", + **kwargs, + ) -> None: + meson_version = Version( + subprocess.check_output( + ["meson", "--version"], + encoding="utf-8", + ).strip() + ) + dpkg_architecture_variables = context.dpkg_architecture_variables + + meson_flags = [ + "--wrap-mode=odownload", + "--buildtype=plain", + "--sysconfdir=/etc", + "--localstatedir=/var", + f"--libdir=lib/{dpkg_architecture_variables.current_host_multiarch}", + "--auto-features=enabled", + ] + if meson_version >= Version("1.2.0"): + # There was a behaviour change in Meson 1.2.0: previously + # byte-compilation wasn't supported, but since 1.2.0 it is on by + # default. We can only use this option to turn it off in versions + # where the option exists. + meson_flags.append("-Dpython.bytecompile=-1") + + self._cross_flags(context, meson_flags) + + if self.configure_args: + substitution = self.substitution + attr_path = self.attribute_path["configure_args"] + meson_flags.extend( + substitution.substitute(v, attr_path[i].path) + for i, v in enumerate(self.configure_args) + ) + + env_mod = self._default_meson_env() + + self.ensure_build_dir_exists() + source_dir_from_build_dir = self.relative_from_builddir_to_source() + + with self.dump_logs_on_error("meson-logs/meson-log.txt"): + run_build_system_command( + "meson", + "setup", + source_dir_from_build_dir, + *meson_flags, + cwd=self.build_directory, + env_mod=env_mod, + ) + + def build_impl( + self, + context: "BuildContext", + manifest: "HighLevelManifest", + **kwargs, + ) -> None: + self._ninja_support.run_ninja_build(context) + + def test_impl( + self, + context: "BuildContext", + manifest: "HighLevelManifest", + **kwargs, + ) -> None: + env_mod = EnvironmentModification( + replacements=(("MESON_TESTTHREDS", f"{context.parallelization_limit()}"),), + ).combine(self._default_meson_env()) + with self.dump_logs_on_error("meson-logs/testlog.txt"): + run_build_system_command( + "meson", + "test", + env_mod=env_mod, + cwd=self.build_directory, + ) + + def install_impl( + self, + context: "BuildContext", + manifest: "HighLevelManifest", + dest_dir: str, + **kwargs, + ) -> None: + run_build_system_command( + "meson", + "install", + "--destdir", + dest_dir, + env_mod=self._default_meson_env(), + ) + + def clean_impl( + self, + context: "BuildContext", + manifest: "HighLevelManifest", + clean_helper: "CleanHelper", + **kwargs, + ) -> None: + # `debputy` will handle all the cleanup for us by virtue of "out of source build" + assert self.out_of_source_build + + +def _add_qmake_flag(options: List[str], envvar: str, *, include_cppflags: bool) -> None: + value = os.environ.get(envvar) + if value is None: + return + if include_cppflags: + cppflags = os.environ.get("CPPFLAGS") + if cppflags: + value = f"{value} {cppflags}" + + options.append(f"QMAKE_{envvar}_RELEASE={value}") + options.append(f"QMAKE_{envvar}_DEBUG={value}") + + +class ParsedGenericQmakeBuildRuleDefinition( + OptionalInstallDirectly, + OptionalInSourceBuild, + OptionalBuildDirectory, +): + configure_args: NotRequired[List[str]] + + +class AbstractQmakeBuildSystemRule(StepBasedBuildSystemRule): + + __slots__ = ("configure_args", "_make_support") + + def __init__( + self, + parsed_data: "ParsedGenericQmakeBuildRuleDefinition", + attribute_path: AttributePath, + parser_context: Union[ParserContextData, "HighLevelManifest"], + ) -> None: + super().__init__(parsed_data, attribute_path, parser_context) + configure_args = [a for a in parsed_data.get("configure_args", [])] + self.configure_args = configure_args + self._make_support = MakefileSupport.from_build_system(self) + + @classmethod + def characteristics(cls) -> BuildSystemCharacteristics: + return BuildSystemCharacteristics( + out_of_source_builds="supported-and-default", + ) + + @classmethod + def auto_detect_build_system( + cls, + source_root: VirtualPath, + *args, + **kwargs, + ) -> bool: + return any(p.name.endswith(".pro") for p in source_root.iterdir) + + @classmethod + def os_mkspec_mapping(cls) -> Mapping[str, str]: + return { + "linux": "linux-g++", + "kfreebsd": "gnukfreebsd-g++", + "hurd": "hurd-g++", + } + + def qmake_command(self) -> str: + raise NotImplementedError + + def configure_impl( + self, + context: "BuildContext", + manifest: "HighLevelManifest", + **kwargs, + ) -> None: + + configure_args = [ + "-makefile", + ] + qmake_cmd = context.cross_tool(self.qmake_command()) + + if context.is_cross_compiling: + host_os = context.dpkg_architecture_variables["DEB_HOST_ARCH_OS"] + os2mkspec = self.os_mkspec_mapping() + try: + spec = os2mkspec[host_os] + except KeyError: + _error( + f'Sorry, `debputy` cannot cross build this package for "{host_os}".' + f' Missing a "DEB OS -> qmake -spec <VALUE>" mapping.' + ) + configure_args.append("-spec") + configure_args.append(spec) + + _add_qmake_flag(configure_args, "CFLAGS", include_cppflags=True) + _add_qmake_flag(configure_args, "CXXFLAGS", include_cppflags=True) + _add_qmake_flag(configure_args, "LDFLAGS", include_cppflags=False) + + configure_args.append("QMAKE_STRIP=:") + configure_args.append("PREFIX=/usr") + + if self.configure_args: + substitution = self.substitution + attr_path = self.attribute_path["configure_args"] + configure_args.extend( + substitution.substitute(v, attr_path[i].path) + for i, v in enumerate(self.configure_args) + ) + + self.ensure_build_dir_exists() + if not self.out_of_source_build: + configure_args.append(self.relative_from_builddir_to_source()) + + with self.dump_logs_on_error("config.log"): + run_build_system_command( + qmake_cmd, + *configure_args, + cwd=self.build_directory, + ) + + def build_impl( + self, + context: "BuildContext", + manifest: "HighLevelManifest", + **kwargs, + ) -> None: + self._make_support.run_make(context) + + def test_impl( + self, + context: "BuildContext", + manifest: "HighLevelManifest", + **kwargs, + ) -> None: + limit = context.parallelization_limit(support_zero_as_unlimited=True) + testsuite_flags = [f"-j{limit}"] if limit else ["-j"] + + if not context.is_terse_build: + testsuite_flags.append("--verbose") + self._make_support.run_first_existing_target_if_any( + context, + # Order is deliberately inverse compared to debhelper (#924052) + ["check", "test"], + f"TESTSUITEFLAGS={' '.join(testsuite_flags)}", + "VERBOSE=1", + ) + + def install_impl( + self, + context: "BuildContext", + manifest: "HighLevelManifest", + dest_dir: str, + **kwargs, + ) -> None: + enable_parallelization = not os.path.lexists(self.build_dir_path("libtool")) + self._make_support.run_first_existing_target_if_any( + context, + ["install"], + f"DESTDIR={dest_dir}", + "AM_UPDATE_INFO_DIR=no", + enable_parallelization=enable_parallelization, + ) + + def clean_impl( + self, + context: "BuildContext", + manifest: "HighLevelManifest", + clean_helper: "CleanHelper", + **kwargs, + ) -> None: + if self.out_of_source_build: + return + self._make_support.run_first_existing_target_if_any( + context, + ["distclean", "realclean", "clean"], + ) + + +class QmakeBuildSystemRule(AbstractQmakeBuildSystemRule): + + def qmake_command(self) -> str: + return "qmake" + + +class Qmake6BuildSystemRule(AbstractQmakeBuildSystemRule): + + def qmake_command(self) -> str: + return "qmake6" + + +@debputy_build_system( + "make", + MakefileBuildSystemRule, + auto_detection_shadows_build_systems="debhelper", + online_reference_documentation=reference_documentation( + title="Make Build System", + description=textwrap.dedent( + "" + """\ + Run a plain `make` file with nothing else. + + This build system will attempt to use `make` to leverage instructions + in a makefile (such as, `Makefile` or `GNUMakefile`). + + By default, the makefile build system assumes it should use "in-source" + build semantics. If needed be, an explicit `build-directory` can be + provided if the `Makefile` is not in the source folder but instead in + some other directory. + """ + ), + attributes=[ + documented_attr( + "directory", + textwrap.dedent( + """\ + The directory from which to run make if it is not the source root + + This works like using `make -C DIRECTORY ...` (or `cd DIRECTORY && make ...`). + """ + ), + ), + documented_attr( + "build_target", + textwrap.dedent( + """\ + The target name to use for the "build" step. + + If omitted, `make` will be run without any explicit target leaving it to decide + the default. + """ + ), + ), + documented_attr( + "test_target", + textwrap.dedent( + """\ + The target name to use for the "test" step. + + If omitted, `make check` or `make test` will be used if it looks like `make` + will accept one of those targets. Otherwise, the step will be skipped. + """ + ), + ), + documented_attr( + "install_target", + textwrap.dedent( + """\ + The target name to use for the "install" step. + + If omitted, `make install` will be used if it looks like `make` will accept that target. + Otherwise, the step will be skipped. + """ + ), + ), + *docs_from( + DebputyParsedContentStandardConditional, + OptionalInstallDirectly, + BuildRuleParsedFormat, + ), + ], + ), +) +class ParsedMakeBuildRuleDefinition( + OptionalInstallDirectly, +): + directory: NotRequired[FileSystemExactMatchRule] + build_target: NotRequired[str] + test_target: NotRequired[str] + install_target: NotRequired[str] + + +@debputy_build_system( + "autoconf", + AutoconfBuildSystemRule, + auto_detection_shadows_build_systems=["debhelper", "make"], + online_reference_documentation=reference_documentation( + title="Autoconf Build System", + description=textwrap.dedent( + """\ + Run an autoconf-based build system as the upstream build system. + + This build rule will attempt to use autoreconf to update the `configure` + script before running the `configure` script if needed. Otherwise, it + follows the classic `./configure && make && make install` pattern. + + The build rule uses "out of source" builds by default since it is easier + and more reliable for clean and makes it easier to support multiple + builds (that is, two or more build systems for the same source package). + This is in contract to `debhelper`, which defaults to "in source" builds + for `autoconf`. If you need that behavior, please set + `perform-in-source-build: true`. + """ + ), + attributes=[ + documented_attr( + "configure_args", + textwrap.dedent( + """\ + Arguments to be passed to the `configure` script. + """ + ), + ), + *docs_from( + DebputyParsedContentStandardConditional, + OptionalInstallDirectly, + OptionalInSourceBuild, + OptionalBuildDirectory, + BuildRuleParsedFormat, + ), + ], + ), +) +class ParsedAutoconfBuildRuleDefinition( + OptionalInstallDirectly, + OptionalInSourceBuild, + OptionalBuildDirectory, +): + configure_args: NotRequired[List[str]] + + +@debputy_build_system( + "cmake", + CMakeBuildSystemRule, + auto_detection_shadows_build_systems=["debhelper", "make"], + online_reference_documentation=reference_documentation( + title="CMake Build System", + description=textwrap.dedent( + """\ + Run an cmake-based build system as the upstream build system. + + The build rule uses "out of source" builds. + """ + ), + attributes=[ + documented_attr( + "configure_args", + textwrap.dedent( + """\ + Arguments to be passed to the `cmake` command. + """ + ), + ), + *docs_from( + DebputyParsedContentStandardConditional, + OptionalInstallDirectly, + OptionalBuildDirectory, + BuildRuleParsedFormat, + ), + ], + ), +) +class ParsedCMakeBuildRuleDefinition( + OptionalInstallDirectly, + OptionalBuildDirectory, +): + configure_args: NotRequired[List[str]] + target_build_system: Literal["make", "ninja"] + + +@debputy_build_system( + "meson", + MesonBuildSystemRule, + auto_detection_shadows_build_systems=["debhelper", "make"], + online_reference_documentation=reference_documentation( + title="Meson Build System", + description=textwrap.dedent( + """\ + Run an meson-based build system as the upstream build system. + + The build rule uses "out of source" builds. + """ + ), + attributes=[ + documented_attr( + "configure_args", + textwrap.dedent( + """\ + Arguments to be passed to the `meson` command. + """ + ), + ), + *docs_from( + DebputyParsedContentStandardConditional, + OptionalInstallDirectly, + OptionalBuildDirectory, + BuildRuleParsedFormat, + ), + ], + ), +) +class ParsedMesonBuildRuleDefinition( + OptionalInstallDirectly, + OptionalBuildDirectory, +): + configure_args: NotRequired[List[str]] + + +@debputy_build_system( + "perl-build", + PerlBuildBuildSystemRule, + auto_detection_shadows_build_systems=[ + "debhelper", + "make", + "perl-makemaker", + ], + online_reference_documentation=reference_documentation( + title='Perl "Build.PL" Build System', + description=textwrap.dedent( + """\ + Build using the `Build.PL` Build system used by some Perl packages. + + This build rule will attempt to use the `Build.PL` script to build the + upstream code. + """ + ), + attributes=[ + documented_attr( + "configure_args", + textwrap.dedent( + """\ + Arguments to be passed to the `Build.PL` script. + """ + ), + ), + *docs_from( + DebputyParsedContentStandardConditional, + OptionalInstallDirectly, + BuildRuleParsedFormat, + ), + ], + ), +) +class ParsedPerlBuildBuildRuleDefinition( + OptionalInstallDirectly, +): + configure_args: NotRequired[List[str]] + + +@debputy_build_system( + "debhelper", + DebhelperBuildSystemRule, + online_reference_documentation=reference_documentation( + title="Debhelper Build System", + description=textwrap.dedent( + """\ + Delegate to a debhelper provided build system + + This build rule will attempt to use the `dh_auto_*` tools to build the + upstream code. By default, `dh_auto_*` will use auto-detection to determine + which build system they will use. This can be overridden by the + `dh-build-system` attribute. + """ + ), + attributes=[ + documented_attr( + "dh_build_system", + textwrap.dedent( + """\ + Which debhelper build system to use. This attribute is passed to + the `dh_auto_*` commands as the `-S` parameter, so any value valid + for that will be accepted. + + Note that many debhelper build systems require extra build + dependencies before they can be used. Please consult the documentation + of the relevant debhelper build system for details. + """ + ), + ), + documented_attr( + "configure_args", + textwrap.dedent( + """\ + Arguments to be passed to underlying configuration command + (via `dh_auto_configure -- <configure-args`). + """ + ), + ), + *docs_from( + DebputyParsedContentStandardConditional, + OptionalInstallDirectly, + OptionalBuildDirectory, + BuildRuleParsedFormat, + ), + ], + ), +) +class ParsedDebhelperBuildRuleDefinition( + OptionalInstallDirectly, + OptionalBuildDirectory, +): + configure_args: NotRequired[List[str]] + dh_build_system: NotRequired[str] + + +@debputy_build_system( + "perl-makemaker", + PerlMakeMakerBuildSystemRule, + auto_detection_shadows_build_systems=[ + "debhelper", + "make", + ], + online_reference_documentation=reference_documentation( + title='Perl "MakeMaker" Build System', + description=textwrap.dedent( + """\ + Build using the "MakeMaker" Build system used by some Perl packages. + + This build rule will attempt to use the `Makefile.PL` script to build the + upstream code. + """ + ), + attributes=[ + documented_attr( + "configure_args", + textwrap.dedent( + """\ + Arguments to be passed to the `Makefile.PL` script. + """ + ), + ), + *docs_from( + DebputyParsedContentStandardConditional, + OptionalInstallDirectly, + BuildRuleParsedFormat, + ), + ], + ), +) +class ParsedPerlMakeMakerBuildRuleDefinition( + OptionalInstallDirectly, +): + configure_args: NotRequired[List[str]] + + +@debputy_build_system( + "qmake", + QmakeBuildSystemRule, + auto_detection_shadows_build_systems=[ + "debhelper", + "make", + # Open question, should this shadow "qmake6" and later? + ], + online_reference_documentation=reference_documentation( + title='QT "qmake" Build System', + description=textwrap.dedent( + """\ + Build using the "qmake" by QT. + """ + ), + attributes=[ + documented_attr( + "configure_args", + textwrap.dedent( + """\ + Arguments to be passed to the `qmake` command. + """ + ), + ), + *docs_from( + DebputyParsedContentStandardConditional, + OptionalInstallDirectly, + OptionalInSourceBuild, + OptionalBuildDirectory, + BuildRuleParsedFormat, + ), + ], + ), +) +class ParsedQmakeBuildRuleDefinition(ParsedGenericQmakeBuildRuleDefinition): + pass + + +@debputy_build_system( + "qmake6", + Qmake6BuildSystemRule, + auto_detection_shadows_build_systems=[ + "debhelper", + "make", + ], + online_reference_documentation=reference_documentation( + title='QT "qmake6" Build System', + description=textwrap.dedent( + """\ + Build using the "qmake6" from the `qmake6` package. This is like the `qmake` system + but is specifically for QT6. + """ + ), + attributes=[ + documented_attr( + "configure_args", + textwrap.dedent( + """\ + Arguments to be passed to the `qmake6` command. + """ + ), + ), + *docs_from( + DebputyParsedContentStandardConditional, + OptionalInstallDirectly, + OptionalInSourceBuild, + OptionalBuildDirectory, + BuildRuleParsedFormat, + ), + ], + ), +) +class ParsedQmake6BuildRuleDefinition(ParsedGenericQmakeBuildRuleDefinition): + pass + + +def _parse_default_environment( + _name: str, + parsed_data: EnvironmentSourceFormat, + attribute_path: AttributePath, + parser_context: ParserContextData, +) -> ManifestProvidedBuildEnvironment: + return ManifestProvidedBuildEnvironment.from_environment_definition( + parsed_data, + attribute_path, + parser_context, + is_default=True, + ) + + +def _parse_build_environments( + _name: str, + parsed_data: List[NamedEnvironmentSourceFormat], + attribute_path: AttributePath, + parser_context: ParserContextData, +) -> List[ManifestProvidedBuildEnvironment]: + return [ + ManifestProvidedBuildEnvironment.from_environment_definition( + value, + attribute_path[idx], + parser_context, + is_default=False, + ) + for idx, value in enumerate(parsed_data) + ] + + +def _handle_build_rules( + _name: str, + parsed_data: List[BuildRule], + _attribute_path: AttributePath, + _parser_context: ParserContextData, +) -> List[BuildRule]: + return parsed_data diff --git a/src/debputy/plugin/debputy/manifest_root_rules.py b/src/debputy/plugin/debputy/manifest_root_rules.py index 80a4799..f539243 100644 --- a/src/debputy/plugin/debputy/manifest_root_rules.py +++ b/src/debputy/plugin/debputy/manifest_root_rules.py @@ -12,18 +12,22 @@ from debputy._manifest_constants import ( ) from debputy.exceptions import DebputySubstitutionError from debputy.installations import InstallRule -from debputy.manifest_parser.base_types import DebputyParsedContent +from debputy.manifest_parser.tagging_types import DebputyParsedContent from debputy.manifest_parser.exceptions import ManifestParseException from debputy.manifest_parser.parser_data import ParserContextData from debputy.manifest_parser.util import AttributePath from debputy.plugin.api import reference_documentation from debputy.plugin.api.impl import DebputyPluginInitializerProvider -from debputy.plugin.api.impl_types import ( +from debputy.plugin.api.parser_tables import ( OPARSER_MANIFEST_ROOT, OPARSER_MANIFEST_DEFINITIONS, OPARSER_PACKAGES, ) -from debputy.plugin.api.spec import not_integrations, INTEGRATION_MODE_DH_DEBPUTY_RRR +from debputy.plugin.api.spec import ( + not_integrations, + INTEGRATION_MODE_DH_DEBPUTY_RRR, +) +from debputy.plugin.debputy.build_system_rules import register_build_system_rules from debputy.substitution import VariableNameState, SUBST_VAR_RE if TYPE_CHECKING: @@ -166,6 +170,8 @@ def register_manifest_root_rules(api: DebputyPluginInitializerProvider) -> None: nested_in_package_context=True, ) + register_build_system_rules(api) + class ManifestVersionFormat(DebputyParsedContent): manifest_version: ManifestVersion @@ -212,13 +218,13 @@ def _handle_manifest_variables( key_path = variables_path[key] if not SUBST_VAR_RE.match("{{" + key + "}}"): raise ManifestParseException( - f"The variable at {key_path.path} has an invalid name and therefore cannot" + f"The variable at {key_path.path_key_lc} has an invalid name and therefore cannot" " be used." ) if substitution.variable_state(key) != VariableNameState.UNDEFINED: raise ManifestParseException( f'The variable "{key}" is already reserved/defined. Error triggered by' - f" {key_path.path}." + f" {key_path.path_key_lc}." ) try: value = substitution.substitute(value_raw, key_path.path) diff --git a/src/debputy/plugin/debputy/package_processors.py b/src/debputy/plugin/debputy/package_processors.py index 1d19b66..0099c3b 100644 --- a/src/debputy/plugin/debputy/package_processors.py +++ b/src/debputy/plugin/debputy/package_processors.py @@ -8,7 +8,15 @@ from contextlib import ExitStack from typing import Optional, Iterator, IO, Any, List, Dict, Callable, Union from debputy.plugin.api import VirtualPath -from debputy.util import _error, xargs, escape_shell, _info, assume_not_none +from debputy.util import ( + _error, + xargs, + escape_shell, + _info, + assume_not_none, + print_command, + _debug_log, +) @contextlib.contextmanager @@ -119,8 +127,27 @@ def process_manpages(fs_root: VirtualPath, _unused1: Any, _unused2: Any) -> None for manpage in manpages: dest_name = manpage if dest_name.endswith(".gz"): - dest_name = dest_name[:-3] - os.rename(f"{dest_name}.encoded", manpage) + encoded_name = dest_name[:-3] + ".encoded" + with open(dest_name, "wb") as out: + _debug_log( + f"Recompressing {dest_name} via gzip -9nc {escape_shell(encoded_name)}" + ) + try: + subprocess.check_call( + [ + "gzip", + "-9nc", + encoded_name, + ], + stdin=subprocess.DEVNULL, + stdout=out, + ) + except subprocess.CalledProcessError: + _error( + f"The command {escape_shell('gzip', '-nc', f'{encoded_name}')} > {dest_name} failed!" + ) + else: + os.rename(f"{dest_name}.encoded", manpage) def _filter_compress_paths() -> Callable[[VirtualPath], Iterator[VirtualPath]]: diff --git a/src/debputy/plugin/debputy/private_api.py b/src/debputy/plugin/debputy/private_api.py index d042378..75081a4 100644 --- a/src/debputy/plugin/debputy/private_api.py +++ b/src/debputy/plugin/debputy/private_api.py @@ -43,9 +43,11 @@ from debputy.manifest_conditions import ( BuildProfileMatch, SourceContextArchMatchManifestCondition, ) -from debputy.manifest_parser.base_types import ( +from debputy.manifest_parser.tagging_types import ( DebputyParsedContent, - DebputyParsedContentStandardConditional, + TypeMapping, +) +from debputy.manifest_parser.base_types import ( FileSystemMode, StaticFileSystemOwner, StaticFileSystemGroup, @@ -53,11 +55,12 @@ from debputy.manifest_parser.base_types import ( FileSystemExactMatchRule, FileSystemMatchRule, SymbolicMode, - TypeMapping, OctalMode, FileSystemExactNonDirMatchRule, + BuildEnvironmentDefinition, + DebputyParsedContentStandardConditional, ) -from debputy.manifest_parser.declarative_parser import DebputyParseHint +from debputy.manifest_parser.parse_hints import DebputyParseHint from debputy.manifest_parser.exceptions import ManifestParseException from debputy.manifest_parser.mapper_code import type_mapper_str2package from debputy.manifest_parser.parser_data import ParserContextData @@ -79,7 +82,9 @@ from debputy.plugin.api.spec import ( not_integrations, INTEGRATION_MODE_DH_DEBPUTY_RRR, ) +from debputy.plugin.api.std_docs import docs_from from debputy.plugin.debputy.binary_package_rules import register_binary_package_rules +from debputy.plugin.debputy.build_system_rules import register_build_system_rules from debputy.plugin.debputy.discard_rules import ( _debputy_discard_pyc_files, _debputy_prune_la_files, @@ -587,6 +592,16 @@ def register_type_mappings(api: DebputyPluginInitializerProvider) -> None: ], ), ) + api.register_mapped_type( + TypeMapping( + BuildEnvironmentDefinition, + str, + lambda v, ap, pc: pc.resolve_build_environment(v, ap), + ), + reference_documentation=type_mapping_reference_documentation( + description="Reference to an build environment defined in `build-environments`", + ), + ) def register_service_managers( @@ -897,14 +912,7 @@ def register_install_rules(api: DebputyPluginInitializerProvider) -> None: """ ), ), - documented_attr( - "when", - textwrap.dedent( - """\ - A condition as defined in [Conditional rules]({MANIFEST_FORMAT_DOC}#Conditional rules). - """ - ), - ), + *docs_from(DebputyParsedContentStandardConditional), ], reference_documentation_url=_manifest_format_doc("generic-install-install"), ), @@ -1193,14 +1201,7 @@ def register_install_rules(api: DebputyPluginInitializerProvider) -> None: """ ), ), - documented_attr( - "when", - textwrap.dedent( - """\ - A condition as defined in [Conditional rules]({MANIFEST_FORMAT_DOC}#Conditional rules). - """ - ), - ), + *docs_from(DebputyParsedContentStandardConditional), ], reference_documentation_url=_manifest_format_doc( "install-manpages-install-man" @@ -1355,14 +1356,7 @@ def register_install_rules(api: DebputyPluginInitializerProvider) -> None: """ ), ), - documented_attr( - "when", - textwrap.dedent( - """\ - A condition as defined in [Conditional rules]({MANIFEST_FORMAT_DOC}#Conditional rules). - """ - ), - ), + *docs_from(DebputyParsedContentStandardConditional), ], reference_documentation_url=_manifest_format_doc("generic-install-install"), ), @@ -1408,14 +1402,7 @@ def register_transformation_rules(api: DebputyPluginInitializerProvider) -> None """ ), ), - documented_attr( - "when", - textwrap.dedent( - """\ - A condition as defined in [Conditional rules]({MANIFEST_FORMAT_DOC}#Conditional rules). - """ - ), - ), + *docs_from(DebputyParsedContentStandardConditional), ], reference_documentation_url=_manifest_format_doc( "move-transformation-rule-move" @@ -1564,14 +1551,7 @@ def register_transformation_rules(api: DebputyPluginInitializerProvider) -> None """ ), ), - documented_attr( - "when", - textwrap.dedent( - """\ - A condition as defined in [Conditional rules]({MANIFEST_FORMAT_DOC}#Conditional rules). - """ - ), - ), + *docs_from(DebputyParsedContentStandardConditional), ], reference_documentation_url=_manifest_format_doc( "create-symlinks-transformation-rule-create-symlink" @@ -1680,14 +1660,7 @@ def register_transformation_rules(api: DebputyPluginInitializerProvider) -> None """ ), ), - documented_attr( - "when", - textwrap.dedent( - """\ - A condition as defined in [Conditional rules]({MANIFEST_FORMAT_DOC}#Conditional rules). - """ - ), - ), + *docs_from(DebputyParsedContentStandardConditional), ], reference_documentation_url=_manifest_format_doc( "change-path-ownergroup-or-mode-path-metadata" @@ -1768,14 +1741,7 @@ def register_transformation_rules(api: DebputyPluginInitializerProvider) -> None """ ), ), - documented_attr( - "when", - textwrap.dedent( - """\ - A condition as defined in [Conditional rules]({MANIFEST_FORMAT_DOC}#Conditional rules). - """ - ), - ), + *docs_from(DebputyParsedContentStandardConditional), ], reference_documentation_url=_manifest_format_doc( "create-directories-transformation-rule-directories" diff --git a/src/debputy/plugin/debputy/to_be_api_types.py b/src/debputy/plugin/debputy/to_be_api_types.py new file mode 100644 index 0000000..720d1d2 --- /dev/null +++ b/src/debputy/plugin/debputy/to_be_api_types.py @@ -0,0 +1,1031 @@ +import contextlib +import dataclasses +import os.path +import subprocess +from typing import ( + Optional, + FrozenSet, + final, + TYPE_CHECKING, + Union, + Annotated, + List, + NotRequired, + Literal, + Any, + Type, + TypeVar, + Self, + Sequence, + Callable, + Container, + Iterable, + is_typeddict, +) + +from debputy.exceptions import PluginAPIViolationError, PluginInitializationError +from debputy.manifest_conditions import ManifestCondition +from debputy.manifest_parser.base_types import ( + BuildEnvironmentDefinition, + DebputyParsedContentStandardConditional, + FileSystemExactMatchRule, +) +from debputy.manifest_parser.exceptions import ( + ManifestParseException, + ManifestInvalidUserDataException, +) +from debputy.manifest_parser.parse_hints import DebputyParseHint +from debputy.manifest_parser.parser_data import ParserContextData +from debputy.manifest_parser.tagging_types import DebputyDispatchableType +from debputy.manifest_parser.util import AttributePath +from debputy.packages import BinaryPackage +from debputy.plugin.api.spec import ( + ParserDocumentation, + DebputyIntegrationMode, + BuildSystemManifestRuleMetadata, + _DEBPUTY_DISPATCH_METADATA_ATTR_NAME, + VirtualPath, +) +from debputy.plugin.plugin_state import run_in_context_of_plugin +from debputy.substitution import Substitution +from debputy.types import EnvironmentModification +from debputy.util import run_build_system_command, _debug_log, _info, _warn + +if TYPE_CHECKING: + from debputy.build_support.build_context import BuildContext + from debputy.highlevel_manifest import HighLevelManifest + from debputy.plugin.api.impl_types import DIPHandler + + +AT = TypeVar("AT") +BSR = TypeVar("BSR", bound="BuildSystemRule") +BSPF = TypeVar("BSPF", bound="BuildRuleDefinitionBase") + + +@dataclasses.dataclass(slots=True, frozen=True) +class BuildSystemCharacteristics: + out_of_source_builds: Literal[ + "required", + "supported-and-default", + "supported-but-not-default", + "not-supported", + ] + + +class CleanHelper: + def schedule_removal_of_files(self, *args: str) -> None: + """Schedule removal of these files + + This will remove the provided files in bulk. The files are not guaranteed + to be deleted in any particular order. If anything needs urgent removal, + `os.unlink` can be used directly. + + Note: Symlinks will **not** be followed. If a symlink and target must + be deleted, ensure both are passed. + + + :param args: Path names to remove. Each must be removable with + `os.unlink` + """ + raise NotImplementedError + + def schedule_removal_of_directories(self, *args: str) -> None: + """Schedule removal of these directories + + This will remove the provided dirs in bulk. The dirs are not guaranteed + to be deleted in any particular order. If anything needs urgent removal, + then it can be done directly instead of passing it to this method. + + If anything needs urgent removal, then it can be removed immediately. + + :param args: Path names to remove. + """ + raise NotImplementedError + + +class BuildRuleParsedFormat(DebputyParsedContentStandardConditional): + name: NotRequired[str] + for_packages: NotRequired[ + Annotated[ + Union[BinaryPackage, List[BinaryPackage]], + DebputyParseHint.manifest_attribute("for"), + ] + ] + environment: NotRequired[BuildEnvironmentDefinition] + + +class OptionalBuildDirectory(BuildRuleParsedFormat): + build_directory: NotRequired[FileSystemExactMatchRule] + + +class OptionalInSourceBuild(BuildRuleParsedFormat): + perform_in_source_build: NotRequired[bool] + + +class OptionalInstallDirectly(BuildRuleParsedFormat): + install_directly_to_package: NotRequired[bool] + + +BuildSystemDefinition = Union[ + BuildRuleParsedFormat, + OptionalBuildDirectory, + OptionalInSourceBuild, + OptionalInstallDirectly, +] + + +class BuildRule(DebputyDispatchableType): + __slots__ = ( + "_auto_generated_stem", + "_name", + "_for_packages", + "_manifest_condition", + "_attribute_path", + "_environment", + "_substitution", + ) + + def __init__( + self, + attributes: BuildRuleParsedFormat, + attribute_path: AttributePath, + parser_context: Union[ParserContextData, "HighLevelManifest"], + ) -> None: + super().__init__() + + self._name = attributes.get("name") + for_packages = attributes.get("for_packages") + + if for_packages is None: + if isinstance(parser_context, ParserContextData): + all_binaries = parser_context.binary_packages.values() + else: + all_binaries = parser_context.all_packages + self._for_packages = frozenset(all_binaries) + else: + self._for_packages = frozenset( + for_packages if isinstance(for_packages, list) else [for_packages] + ) + self._manifest_condition = attributes.get("when") + self._attribute_path = attribute_path + self._substitution = parser_context.substitution + self._auto_generated_stem: Optional[str] = None + environment = attributes.get("environment") + if environment is None: + assert isinstance(parser_context, ParserContextData) + self._environment = parser_context.resolve_build_environment( + None, + attribute_path, + ) + else: + self._environment = environment + + @final + @property + def name(self) -> Optional[str]: + return self._name + + @final + @property + def attribute_path(self) -> AttributePath: + return self._attribute_path + + @final + @property + def manifest_condition(self) -> Optional[ManifestCondition]: + return self._manifest_condition + + @final + @property + def for_packages(self) -> FrozenSet[BinaryPackage]: + return self._for_packages + + @final + @property + def substitution(self) -> Substitution: + return self._substitution + + @final + @property + def environment(self) -> BuildEnvironmentDefinition: + return self._environment + + @final + @property + def auto_generated_stem(self) -> str: + stem = self._auto_generated_stem + if stem is None: + raise AssertionError( + "The auto-generated-stem is not available at this time" + ) + return stem + + @final + @auto_generated_stem.setter + def auto_generated_stem(self, value: str) -> None: + if self._auto_generated_stem is not None: + raise AssertionError("The auto-generated-stem should only be set once") + assert value is not None + self._auto_generated_stem = value + + @final + def run_build( + self, + context: "BuildContext", + manifest: "HighLevelManifest", + **kwargs, + ) -> None: + run_in_context_of_plugin( + self._debputy_plugin, + self.perform_build, + context, + manifest, + **kwargs, + ) + + def perform_build( + self, + context: "BuildContext", + manifest: "HighLevelManifest", + **kwargs, + ) -> None: + raise NotImplementedError + + @property + def is_buildsystem(self) -> bool: + return False + + @property + def name_or_tag(self) -> str: + name = self.name + if name is None: + return self.auto_generated_stem + return name + + +def _is_type_or_none(v: Optional[Any], expected_type: Type[AT]) -> Optional[AT]: + if isinstance(v, expected_type): + return v + return None + + +class BuildSystemRule(BuildRule): + + __slots__ = ( + "_build_directory", + "source_directory", + "install_directly_to_package", + "perform_in_source_build", + ) + + def __init__( + self, + attributes: BuildSystemDefinition, + attribute_path: AttributePath, + parser_context: Union[ParserContextData, "HighLevelManifest"], + ) -> None: + super().__init__(attributes, attribute_path, parser_context) + build_directory = _is_type_or_none( + attributes.get("build_directory"), FileSystemExactMatchRule + ) + if build_directory is not None: + self._build_directory = build_directory.match_rule.path + else: + self._build_directory = None + self.source_directory = "." + self.install_directly_to_package = False + self.perform_in_source_build = _is_type_or_none( + attributes.get("perform_in_source_build"), bool + ) + install_directly_to_package = _is_type_or_none( + attributes.get("install_directly_to_package"), bool + ) + if install_directly_to_package is None: + self.install_directly_to_package = len(self.for_packages) == 1 + elif install_directly_to_package and len(self.for_packages) > 1: + idtp_path = attribute_path["install_directly_to_package"].path + raise ManifestParseException( + f'The attribute "install-directly-to-package" ({idtp_path}) cannot' + " be true when the build system applies to multiple packages." + ) + else: + self.install_directly_to_package = install_directly_to_package + + @classmethod + def auto_detect_build_system( + cls, + source_root: VirtualPath, + *args, + **kwargs, + ) -> bool: + """Check if the build system apply automatically. + + This class method is called when the manifest does not declare any build rules at + all. + + :param source_root: The source root (the directory containing `debian/`). Usually, + the detection code would look at this for files related to the upstream build system. + :param args: For future compat, new arguments might appear as positional arguments. + :param kwargs: For future compat, new arguments might appear as keyword argument. + :return: True if the build system can be used, False when it would not be useful + to use the build system (at least with all defaults). + Note: Be sure to use proper `bool` return values. The calling code does an + `isinstance` check to ensure that the version of `debputy` supports the + auto-detector (in case the return type is ever expanded in the future). + """ + return False + + @property + def out_of_source_build(self) -> bool: + build_directory = self.build_directory + return build_directory != self.source_directory + + @property + def build_directory(self) -> str: + directory = self._build_directory + if directory is None: + return self.source_directory + return directory + + @contextlib.contextmanager + def dump_logs_on_error(self, *logs: str) -> None: + """Context manager that will dump logs to stdout on error + + :param logs: The logs to be dumped. Relative path names are assumed to be relative to + the build directory. + """ + try: + yield + except (Exception, KeyboardInterrupt, SystemExit): + _warn( + "Error occurred, attempting to provide relevant logs as requested by the build system provider" + ) + found_any = False + for log in logs: + if not os.path.isabs(log): + log = self.build_dir_path(log) + if not os.path.isfile(log): + _info( + f'Would have pushed "{log}" to stdout, but it does not exist.' + ) + continue + subprocess.run(["tail", "-v", "-n", "+0", log]) + found_any = True + if not found_any: + _warn( + f"None of the logs provided were available (relative to build directory): {', '.join(logs)}" + ) + raise + + @final + def run_clean( + self, + context: "BuildContext", + manifest: "HighLevelManifest", + clean_helper: CleanHelper, + **kwargs, + ) -> None: + run_in_context_of_plugin( + self._debputy_plugin, + self.perform_clean, + context, + manifest, + clean_helper, + **kwargs, + ) + + def perform_clean( + self, + context: "BuildContext", + manifest: "HighLevelManifest", + clean_helper: CleanHelper, + **kwargs, + ) -> None: + raise NotImplementedError + + def ensure_build_dir_exists(self) -> None: + build_dir = self.build_directory + source_dir = self.source_directory + if build_dir == source_dir: + return + os.makedirs(build_dir, mode=0o755, exist_ok=True) + + def build_dir_path(self, /, path: str = "") -> str: + build_dir = self.build_directory + if path == "": + return build_dir + return os.path.join(build_dir, path) + + def relative_from_builddir_to_source( + self, + path_in_source_dir: Optional[str] = None, + ) -> str: + build_dir = self.build_directory + source_dir = self.source_directory + if build_dir == source_dir: + return path_in_source_dir + return os.path.relpath(os.path.join(source_dir, path_in_source_dir), build_dir) + + @final + @property + def is_buildsystem(self) -> bool: + return True + + +class StepBasedBuildSystemRule(BuildSystemRule): + + @classmethod + def characteristics(cls) -> BuildSystemCharacteristics: + raise NotImplementedError + + @final + def perform_clean( + self, + context: "BuildContext", + manifest: "HighLevelManifest", + clean_helper: CleanHelper, + **kwargs, + ) -> None: + self._check_characteristics() + self.before_first_impl_step(stage="clean") + self.clean_impl(context, manifest, clean_helper, **kwargs) + if self.out_of_source_build: + build_directory = self.build_directory + assert build_directory is not None + if os.path.lexists(build_directory): + clean_helper.schedule_removal_of_directories(build_directory) + dest_dir = self.resolve_dest_dir() + if not isinstance(dest_dir, BinaryPackage): + clean_helper.schedule_removal_of_directories(dest_dir) + + @final + def perform_build( + self, + context: "BuildContext", + manifest: "HighLevelManifest", + **kwargs, + ) -> None: + self._check_characteristics() + self.before_first_impl_step(stage="build") + self.configure_impl(context, manifest, **kwargs) + self.build_impl(context, manifest, **kwargs) + if context.should_run_tests: + self.test_impl(context, manifest, **kwargs) + dest_dir = self.resolve_dest_dir() + if isinstance(dest_dir, BinaryPackage): + dest_dir = f"debian/{dest_dir.name}" + # Make it absolute for everyone (that worked for debhelper). + # At least autoconf's "make install" requires an absolute path, so making is + # relative would have at least one known issue. + abs_dest_dir = os.path.abspath(dest_dir) + self.install_impl(context, manifest, abs_dest_dir, **kwargs) + + def before_first_impl_step( + self, + /, + stage: Literal["build", "clean"], + **kwargs, + ) -> None: + """Called before any `*_impl` method is called. + + This can be used to validate input against data that is not available statically + (that is, it will be checked during build but not in static checks). An example + is that the `debhelper` build system uses this to validate the provided `dh-build-system` + to ensure that `debhelper` knows about the build system. This check cannot be done + statically since the build system is only required to be available in a chroot build + and not on the host system. + + The method can also be used to compute common state for all the `*_impl` methods that + is awkward to do in `__init__`. Note there is no data sharing between the different + stages. This has to do with how `debputy` will be called (usually `clean` followed by + a source package assembly in `dpkg` and then `build`). + + The check is done both on build and on clean before the relevant implementation methods + are invoked. + + Any exception will abort the build. Prefer to raise ManifestInvalidUserDataException + exceptions for issues related to incorrect data. + + The method is not invoked if the steps are skipped, which can happen with build profiles + or arch:any vs. arch:all builds. + + :param stage: A discriminator variable to determine which kind of steps will be invoked + after this method returns. For state initialization, this can be useful if the state + is somewhat expensive and not needed for `clean`. + """ + pass + + def configure_impl( + self, + context: "BuildContext", + manifest: "HighLevelManifest", + **kwargs, + ) -> None: + """Called to handle the "configure" and "build" part of the build + + This is basically a mix of `dh_auto_configure` and `dh_auto_build` from `debhelper`. + If the upstream build also runs test as a part of the build, this method should + check `context.should_run_tests` and pass the relevant flags to disable tests when + `context.should_run_tests` is false. + """ + raise NotImplementedError + + def build_impl( + self, + context: "BuildContext", + manifest: "HighLevelManifest", + **kwargs, + ) -> None: + """Called to handle the "configure" and "build" part of the build + + This is basically a mix of `dh_auto_configure` and `dh_auto_build` from `debhelper`. + If the upstream build also runs test as a part of the build, this method should + check `context.should_run_tests` and pass the relevant flags to disable tests when + `context.should_run_tests` is false. + """ + raise NotImplementedError + + def test_impl( + self, + context: "BuildContext", + manifest: "HighLevelManifest", + **kwargs, + ) -> None: + """Called to handle the "test" part of the build + + This is basically `dh_auto_test` from `debhelper`. + + Note: This will be skipped when `context.should_run_tests` is False. Therefore, the + method can assume that when invoked then tests must be run. + + It is always run after `configure_and_build_impl`. + """ + raise NotImplementedError + + def install_impl( + self, + context: "BuildContext", + manifest: "HighLevelManifest", + dest_dir: str, + **kwargs, + ) -> None: + """Called to handle the "install" part of the build + + This is basically `dh_auto_install` from `debhelper`. + + The `dest_dir` attribute is what the upstream should install its data into. It + follows the `DESTDIR` convention from autoconf/make. The `dest_dir` should not + be second-guessed since `debputy` will provide automatically as a search path + for installation rules when relevant. + + It is always run after `configure_and_build_impl` and, if relevant, `test_impl`. + """ + raise NotImplementedError + + def clean_impl( + self, + context: "BuildContext", + manifest: "HighLevelManifest", + clean_helper: "CleanHelper", + **kwargs, + ) -> None: + """Called to handle the "clean" part of the build + + This is basically `dh_auto_clean` from `debhelper`. + + For out-of-source builds, `debputy` will remove the build directory for you + if it exists (when this method returns). This method is only "in-source" cleaning + or for "dirty" state left outside the designated build directory. + + Note that state *cannot* be shared between `clean` and other steps due to limitations + of how the Debian build system works in general. + """ + raise NotImplementedError + + def _check_characteristics(self) -> None: + characteristics = self.characteristics() + + _debug_log(f"Characteristics for {self.name_or_tag} {self.__class__.__name__} ") + + if self.out_of_source_build and self.perform_in_source_build: + raise ManifestInvalidUserDataException( + f"Cannot use 'build-directory' with 'perform-in-source-build' at {self.attribute_path.path}" + ) + if ( + characteristics.out_of_source_builds == "required" + and self.perform_in_source_build + ): + path = self.attribute_path["perform_in_source_build"].path_key_lc + + # FIXME: How do I determine the faulty plugin from here. + raise PluginAPIViolationError( + f"The build system {self.__class__.__qualname__} had an perform-in-source-build attribute, but claims" + f" it requires out of source builds. Please file a bug against the provider asking them not to use" + f' "{OptionalInSourceBuild.__name__}" as base for their build system definition or tweak' + f" the characteristics of the build system as the current combination is inconsistent." + f" The offending definition is at {path}." + ) + + if ( + characteristics.out_of_source_builds + in ("required", "supported-and-default") + and not self.out_of_source_build + ): + + if not self.perform_in_source_build: + self._build_directory = self._pick_build_dir() + else: + assert characteristics.out_of_source_builds != "required" + elif ( + characteristics.out_of_source_builds == "not-supported" + and self.out_of_source_build + ): + path = self.attribute_path["build_directory"].path_key_lc + + # FIXME: How do I determine the faulty plugin from here. + raise PluginAPIViolationError( + f"The build system {self.__class__.__qualname__} had a build-directory attribute, but claims it does" + f" not support out of source builds. Please file a bug against the provider asking them not to use" + f' "{OptionalBuildDirectory.__name__}" as base for their build system definition or tweak' + f" the characteristics of the build system as the current combination is inconsistent." + f" The offending definition is at {path}." + ) + + def _pick_build_dir(self) -> str: + tag = self.name if self.name is not None else self.auto_generated_stem + if tag == "": + return "_build" + return f"_build-{tag}" + + @final + def resolve_dest_dir(self) -> Union[str, BinaryPackage]: + auto_generated_stem = self.auto_generated_stem + if self.install_directly_to_package: + assert len(self.for_packages) == 1 + return next(iter(self.for_packages)) + if auto_generated_stem == "": + return "debian/tmp" + return f"debian/tmp-{auto_generated_stem}" + + +# Using the same logic as debhelper for the same reasons. +def _make_target_exists(make_cmd: str, target: str, *, directory: str = ".") -> bool: + cmd = [ + make_cmd, + "-s", + "-n", + "--no-print-directory", + ] + if directory and directory != ".": + cmd.append("-C") + cmd.append(directory) + cmd.append(target) + env = dict(os.environ) + env["LC_ALL"] = "C.UTF-8" + try: + res = subprocess.run( + cmd, + stdin=subprocess.DEVNULL, + stdout=subprocess.PIPE, + stderr=subprocess.STDOUT, + env=env, + restore_signals=True, + ) + except FileNotFoundError: + return False + + options = ( + f"*** No rule to make target '{target}", + f"*** No rule to make target `{target}", + ) + + stdout = res.stdout.decode("utf-8") + return not any(o in stdout for o in options) + + +def _find_first_existing_make_target( + make_cmd: str, + targets: Sequence[str], + *, + directory: str = ".", +) -> Optional[str]: + for target in targets: + if _make_target_exists(make_cmd, target, directory=directory): + return target + return None + + +_UNSET = object() + + +class NinjaBuildSupport: + __slots__ = ("_provided_ninja_program", "_build_system_rule") + + def __init__( + self, + provided_ninja_program: str, + build_system_rule: BuildSystemRule, + ) -> None: + self._provided_ninja_program = provided_ninja_program + self._build_system_rule = build_system_rule + + @classmethod + def from_build_system( + cls, + build_system: BuildSystemRule, + *, + ninja_program: Optional[str] = None, + ) -> Self: + if ninja_program is None: + ninja_program = "ninja" + return cls(ninja_program, build_system) + + @property + def _directory(self) -> str: + return self._build_system_rule.build_directory + + def _pick_directory( + self, arg: Union[Optional[str], _UNSET] = _UNSET + ) -> Optional[str]: + if arg is _UNSET: + return self._directory + return arg + + def run_ninja_build( + self, + build_context: "BuildContext", + *ninja_args: str, + directory: Union[Optional[str], _UNSET] = _UNSET, + env_mod: Optional[EnvironmentModification] = None, + enable_parallelization: bool = True, + ) -> None: + extra_ninja_args = [] + if not build_context.is_terse_build: + extra_ninja_args.append("-v") + self._run_ninja( + build_context, + *extra_ninja_args, + *ninja_args, + env_mod=env_mod, + directory=directory, + enable_parallelization=enable_parallelization, + ) + + def run_ninja_test( + self, + build_context: "BuildContext", + *ninja_args: str, + directory: Union[Optional[str], _UNSET] = _UNSET, + env_mod: Optional[EnvironmentModification] = None, + enable_parallelization: bool = True, + ) -> None: + self._run_ninja( + build_context, + "test", + *ninja_args, + env_mod=env_mod, + directory=directory, + enable_parallelization=enable_parallelization, + ) + + def run_ninja_install( + self, + build_context: "BuildContext", + dest_dir: str, + *ninja_args: str, + directory: Union[Optional[str], _UNSET] = _UNSET, + env_mod: Optional[EnvironmentModification] = None, + # debhelper never had parallel installs, so we do not have it either for now. + enable_parallelization: bool = False, + ) -> None: + install_env_mod = EnvironmentModification(replacements=(("DESTDIR", dest_dir),)) + if env_mod is not None: + install_env_mod = install_env_mod.combine(env_mod) + self._run_ninja( + build_context, + "install", + *ninja_args, + directory=directory, + env_mod=install_env_mod, + enable_parallelization=enable_parallelization, + ) + + def run_ninja_clean( + self, + build_context: "BuildContext", + *ninja_args: str, + directory: Union[Optional[str], _UNSET] = _UNSET, + env_mod: Optional[EnvironmentModification] = None, + enable_parallelization: bool = True, + ) -> None: + self._run_ninja( + build_context, + "clean", + *ninja_args, + env_mod=env_mod, + directory=directory, + enable_parallelization=enable_parallelization, + ) + + def _run_ninja( + self, + build_context: "BuildContext", + *ninja_args: str, + directory: Union[Optional[str], _UNSET] = _UNSET, + env_mod: Optional[EnvironmentModification] = None, + enable_parallelization: bool = True, + ) -> None: + extra_ninja_args = [] + limit = ( + build_context.parallelization_limit(support_zero_as_unlimited=True) + if enable_parallelization + else 1 + ) + extra_ninja_args.append(f"-j{limit}") + ninja_env_mod = EnvironmentModification(replacements=(("LC_ALL", "C.UTF-8"),)) + if env_mod is not None: + ninja_env_mod = ninja_env_mod.combine(env_mod) + run_build_system_command( + self._provided_ninja_program, + *extra_ninja_args, + *ninja_args, + cwd=self._pick_directory(directory), + env_mod=ninja_env_mod, + ) + + +class MakefileSupport: + + __slots__ = ("_provided_make_program", "_build_system_rule") + + def __init__( + self, + make_program: str, + build_system_rule: BuildSystemRule, + ) -> None: + self._provided_make_program = make_program + self._build_system_rule = build_system_rule + + @classmethod + def from_build_system( + cls, + build_system: BuildSystemRule, + *, + make_program: Optional[str] = None, + ) -> Self: + if make_program is None: + make_program = os.environ.get("MAKE", "make") + return cls(make_program, build_system) + + @property + def _directory(self) -> str: + return self._build_system_rule.build_directory + + @property + def _make_program(self) -> str: + make_program = self._provided_make_program + if self._provided_make_program is None: + return os.environ.get("MAKE", "make") + return make_program + + def _pick_directory( + self, arg: Union[Optional[str], _UNSET] = _UNSET + ) -> Optional[str]: + if arg is _UNSET: + return self._directory + return arg + + def find_first_existing_make_target( + self, + targets: Sequence[str], + *, + directory: Union[Optional[str], _UNSET] = _UNSET, + ) -> Optional[str]: + for target in targets: + if self.make_target_exists(target, directory=directory): + return target + return None + + def make_target_exists( + self, + target: str, + *, + directory: Union[Optional[str], _UNSET] = _UNSET, + ) -> bool: + return _make_target_exists( + self._make_program, + target, + directory=self._pick_directory(directory), + ) + + def run_first_existing_target_if_any( + self, + build_context: "BuildContext", + targets: Sequence[str], + *make_args: str, + enable_parallelization: bool = True, + directory: Union[Optional[str], _UNSET] = _UNSET, + env_mod: Optional[EnvironmentModification] = None, + ) -> bool: + target = self.find_first_existing_make_target(targets, directory=directory) + if target is None: + return False + + self.run_make( + build_context, + target, + *make_args, + enable_parallelization=enable_parallelization, + directory=directory, + env_mod=env_mod, + ) + return True + + def run_make( + self, + build_context: "BuildContext", + *make_args: str, + enable_parallelization: bool = True, + directory: Union[Optional[str], _UNSET] = _UNSET, + env_mod: Optional[EnvironmentModification] = None, + ) -> None: + limit = ( + build_context.parallelization_limit(support_zero_as_unlimited=True) + if enable_parallelization + else 1 + ) + extra_make_args = [f"-j{limit}"] if limit else ["-j"] + run_build_system_command( + self._make_program, + *extra_make_args, + *make_args, + cwd=self._pick_directory(directory), + env_mod=env_mod, + ) + + +def debputy_build_system( + # For future self: Before you get ideas about making manifest_keyword accept a list, + # remember it has consequences for shadowing_build_systems_when_active. + manifest_keyword: str, + provider: Type[BSR], + *, + expected_debputy_integration_mode: Optional[ + Container[DebputyIntegrationMode] + ] = None, + auto_detection_shadows_build_systems: Optional[ + Union[str, Iterable[str]] + ] = frozenset(), + online_reference_documentation: Optional[ParserDocumentation] = None, + apply_standard_attribute_documentation: bool = False, + source_format: Optional[Any] = None, +) -> Callable[[Type[BSPF]], Type[BSPF]]: + if not isinstance(provider, type) or not issubclass(provider, BuildSystemRule): + raise PluginInitializationError( + f"The provider for @{debputy_build_system.__name__} must be subclass of {BuildSystemRule.__name__}goes on the TypedDict that defines the parsed" + f" variant of the manifest definition. Not the build system implementation class." + ) + + def _constructor_wrapper( + _rule_used: str, + *args, + **kwargs, + ) -> BSR: + return provider(*args, **kwargs) + + if isinstance(auto_detection_shadows_build_systems, str): + shadows = frozenset([auto_detection_shadows_build_systems]) + else: + shadows = frozenset(auto_detection_shadows_build_systems) + + metadata = BuildSystemManifestRuleMetadata( + (manifest_keyword,), + BuildRule, + _constructor_wrapper, + expected_debputy_integration_mode=expected_debputy_integration_mode, + source_format=source_format, + online_reference_documentation=online_reference_documentation, + apply_standard_attribute_documentation=apply_standard_attribute_documentation, + auto_detection_shadow_build_systems=shadows, + build_system_impl=provider, + ) + + def _decorator_impl(pf_cls: Type[BSPF]) -> Type[BSPF]: + if isinstance(pf_cls, type) and issubclass(pf_cls, BuildSystemRule): + raise PluginInitializationError( + f"The @{debputy_build_system.__name__} annotation goes on the TypedDict that defines the parsed" + f" variant of the manifest definition. Not the build system implementation class." + ) + + # TODO: In python3.12 we can check more than just `is_typeddict`. In python3.11, woe is us and + # is_typeddict is the only thing that reliably works (cpython#103699) + if not is_typeddict(pf_cls): + raise PluginInitializationError( + f"Expected annotated class to be a subclass of {BuildRuleParsedFormat.__name__}," + f" but got {pf_cls.__name__} instead" + ) + + setattr(pf_cls, _DEBPUTY_DISPATCH_METADATA_ATTR_NAME, metadata) + return pf_cls + + return _decorator_impl diff --git a/src/debputy/plugin/plugin_state.py b/src/debputy/plugin/plugin_state.py new file mode 100644 index 0000000..ef4dabb --- /dev/null +++ b/src/debputy/plugin/plugin_state.py @@ -0,0 +1,113 @@ +import contextvars +import functools +import inspect +from contextvars import ContextVar +from typing import Optional, Callable, ParamSpec, TypeVar, NoReturn, Union + +from debputy.exceptions import ( + UnhandledOrUnexpectedErrorFromPluginError, + DebputyRuntimeError, +) +from debputy.util import _debug_log, _is_debug_log_enabled + +_current_debputy_plugin_cxt_var: ContextVar[Optional[str]] = ContextVar( + "current_debputy_plugin", + default=None, +) + +P = ParamSpec("P") +R = TypeVar("R") + + +def current_debputy_plugin_if_present() -> Optional[str]: + return _current_debputy_plugin_cxt_var.get() + + +def current_debputy_plugin_required() -> str: + v = current_debputy_plugin_if_present() + if v is None: + raise AssertionError( + "current_debputy_plugin_required() was called, but no plugin was set." + ) + return v + + +def wrap_plugin_code( + plugin_name: str, + func: Callable[P, R], + *, + non_debputy_exception_handling: Union[bool, Callable[[Exception], NoReturn]] = True, +) -> Callable[P, R]: + if isinstance(non_debputy_exception_handling, bool): + + runner = run_in_context_of_plugin + if non_debputy_exception_handling: + runner = run_in_context_of_plugin_wrap_errors + + def _wrapper(*args: P.args, **kwargs: P.kwargs) -> None: + return runner(plugin_name, func, *args, **kwargs) + + functools.update_wrapper(_wrapper, func) + return _wrapper + + def _wrapper(*args: P.args, **kwargs: P.kwargs) -> None: + try: + return run_in_context_of_plugin(plugin_name, func, *args, **kwargs) + except DebputyRuntimeError: + raise + except Exception as e: + non_debputy_exception_handling(e) + + functools.update_wrapper(_wrapper, func) + return _wrapper + + +def run_in_context_of_plugin( + plugin: str, + func: Callable[P, R], + *args: P.args, + **kwargs: P.kwargs, +) -> R: + context = contextvars.copy_context() + if _is_debug_log_enabled(): + call_stack = inspect.stack() + caller: str = "[N/A]" + for frame in call_stack: + if frame.filename != __file__: + try: + fname = frame.frame.f_code.co_qualname + except AttributeError: + fname = None + if fname is None: + fname = frame.function + caller = f"{frame.filename}:{frame.lineno} ({fname})" + break + # Do not keep the reference longer than necessary + del call_stack + _debug_log( + f"Switching plugin context to {plugin} at {caller} (from context: {current_debputy_plugin_if_present()})" + ) + # Wish we could just do a regular set without wrapping it in `context.run` + context.run(_current_debputy_plugin_cxt_var.set, plugin) + return context.run(func, *args, **kwargs) + + +def run_in_context_of_plugin_wrap_errors( + plugin: str, + func: Callable[P, R], + *args: P.args, + **kwargs: P.kwargs, +) -> R: + try: + return run_in_context_of_plugin(plugin, func, *args, **kwargs) + except DebputyRuntimeError: + raise + except Exception as e: + if plugin != "debputy": + raise UnhandledOrUnexpectedErrorFromPluginError( + f"{func.__qualname__} from the plugin {plugin} raised exception that was not expected here." + ) from e + else: + raise AssertionError( + "Bug in the `debputy` plugin: Unhandled exception." + ) from e diff --git a/src/debputy/transformation_rules.py b/src/debputy/transformation_rules.py index c7f8a2a..6e96c64 100644 --- a/src/debputy/transformation_rules.py +++ b/src/debputy/transformation_rules.py @@ -11,6 +11,7 @@ from typing import ( Dict, TypeVar, cast, + final, ) from debputy.exceptions import ( @@ -27,12 +28,15 @@ from debputy.manifest_parser.base_types import ( FileSystemMode, StaticFileSystemOwner, StaticFileSystemGroup, - DebputyDispatchableType, ) +from debputy.manifest_parser.tagging_types import DebputyDispatchableType from debputy.manifest_parser.util import AttributePath from debputy.path_matcher import MatchRule from debputy.plugin.api import VirtualPath from debputy.plugin.debputy.types import DebputyCapability +from debputy.plugin.plugin_state import ( + run_in_context_of_plugin_wrap_errors, +) from debputy.util import _warn @@ -59,10 +63,26 @@ class PreProvidedExclusion: class TransformationRule(DebputyDispatchableType): + __slots__ = () + @final + def run_transform_file_system( + self, + fs_root: FSPath, + condition_context: ConditionContext, + ) -> None: + run_in_context_of_plugin_wrap_errors( + self._debputy_plugin, + self.transform_file_system, + fs_root, + condition_context, + ) + def transform_file_system( - self, fs_root: FSPath, condition_context: ConditionContext + self, + fs_root: FSPath, + condition_context: ConditionContext, ) -> None: raise NotImplementedError @@ -134,6 +154,7 @@ class RemoveTransformationRule(TransformationRule): keep_empty_parent_dirs: bool, definition_source: AttributePath, ) -> None: + super().__init__() self._match_rules = match_rules self._keep_empty_parent_dirs = keep_empty_parent_dirs self._definition_source = definition_source.path @@ -180,6 +201,7 @@ class MoveTransformationRule(TransformationRule): definition_source: AttributePath, condition: Optional[ManifestCondition], ) -> None: + super().__init__() self._match_rule = match_rule self._dest_path = dest_path self._dest_is_dir = dest_is_dir @@ -283,6 +305,7 @@ class CreateSymlinkPathTransformationRule(TransformationRule): definition_source: AttributePath, condition: Optional[ManifestCondition], ) -> None: + super().__init__() self._link_target = link_target self._link_dest = link_dest self._replacement_rule = replacement_rule @@ -550,6 +573,9 @@ class ModeNormalizationTransformationRule(TransformationRule): self, normalizations: Sequence[Tuple[MatchRule, FileSystemMode]], ) -> None: + # A bit of a hack since it is initialized outside `debputy`. It probably should not + # be a "TransformationRule" (hindsight and all) + run_in_context_of_plugin_wrap_errors("debputy", super().__init__) self._normalizations = normalizations def transform_file_system( @@ -575,6 +601,12 @@ class ModeNormalizationTransformationRule(TransformationRule): class NormalizeShebangLineTransformation(TransformationRule): + + def __init__(self) -> None: + # A bit of a hack since it is initialized outside `debputy`. It probably should not + # be a "TransformationRule" (hindsight and all) + run_in_context_of_plugin_wrap_errors("debputy", super().__init__) + def transform_file_system( self, fs_root: VirtualPath, diff --git a/src/debputy/types.py b/src/debputy/types.py index 05e68c9..5f0bde0 100644 --- a/src/debputy/types.py +++ b/src/debputy/types.py @@ -1,9 +1,116 @@ -from typing import TypeVar, TYPE_CHECKING +import dataclasses +from typing import ( + TypeVar, + TYPE_CHECKING, + Sequence, + Tuple, + Mapping, + Dict, + Optional, + TypedDict, + NotRequired, + List, + MutableMapping, +) if TYPE_CHECKING: from debputy.plugin.api import VirtualPath from debputy.filesystem_scan import FSPath + VP = TypeVar("VP", VirtualPath, FSPath) + S = TypeVar("S", str, bytes) +else: + VP = TypeVar("VP", "VirtualPath", "FSPath") + S = TypeVar("S", str, bytes) -VP = TypeVar("VP", "VirtualPath", "FSPath") -S = TypeVar("S", str, bytes) + +class EnvironmentModificationSerialized(TypedDict): + replacements: NotRequired[Dict[str, str]] + removals: NotRequired[List[str]] + + +@dataclasses.dataclass(slots=True, frozen=True) +class EnvironmentModification: + replacements: Sequence[Tuple[str, str]] = tuple() + removals: Sequence[str] = tuple() + + def __bool__(self) -> bool: + return not self.removals and not self.replacements + + def combine( + self, + other: "Optional[EnvironmentModification]", + ) -> "EnvironmentModification": + if not other: + return self + existing_replacements = {k: v for k, v in self.replacements} + extra_replacements = { + k: v + for k, v in other.replacements + if k not in existing_replacements or existing_replacements[k] != v + } + seen_removals = set(self.removals) + extra_removals = [r for r in other.removals if r not in seen_removals] + + if not extra_replacements and isinstance(self.replacements, tuple): + new_replacements = self.replacements + else: + new_replacements = [] + for k, v in existing_replacements: + if k not in extra_replacements: + new_replacements.append((k, v)) + + for k, v in other.replacements: + if k in extra_replacements: + new_replacements.append((k, v)) + + new_replacements = tuple(new_replacements) + + if not extra_removals and isinstance(self.removals, tuple): + new_removals = self.removals + else: + new_removals = list(self.removals) + new_removals.extend(extra_removals) + new_removals = tuple(new_removals) + + if self.replacements is new_replacements and self.removals is new_removals: + return self + + return EnvironmentModification( + new_replacements, + new_removals, + ) + + def update_inplace(self, env: MutableMapping[str, str]) -> None: + for k, v in self.replacements: + existing_value = env.get(k) + if v == existing_value: + continue + env[k] = v + + for k in self.removals: + if k not in env: + continue + del env[k] + + def compute_env(self, base_env: Mapping[str, str]) -> Mapping[str, str]: + updated_env: Optional[Dict[str, str]] = None + for k, v in self.replacements: + existing_value = base_env.get(k) + if v == existing_value: + continue + + if updated_env is None: + updated_env = dict(base_env) + updated_env[k] = v + + for k in self.removals: + if k not in base_env: + continue + if updated_env is None: + updated_env = dict(base_env) + del updated_env[k] + + if updated_env is not None: + return updated_env + return base_env diff --git a/src/debputy/util.py b/src/debputy/util.py index 860280e..76f4076 100644 --- a/src/debputy/util.py +++ b/src/debputy/util.py @@ -34,7 +34,7 @@ from debian.deb822 import Deb822 from debputy.architecture_support import DpkgArchitectureBuildProcessValuesTable from debputy.exceptions import DebputySubstitutionError - +from debputy.types import EnvironmentModification try: from Levenshtein import distance @@ -99,12 +99,19 @@ POSTINST_DEFAULT_CONDITION = ( _SPACE_RE = re.compile(r"\s") +_WORD_EQUAL = re.compile(r"^-*[\w_\-]+=") _DOUBLE_ESCAPEES = re.compile(r'([\n`$"\\])') -_REGULAR_ESCAPEES = re.compile(r'([\s!"$()*+#;<>?@\[\]\\`|~])') +_REGULAR_ESCAPEES = re.compile(r"""([\s!"$()*+#;<>?@'\[\]\\`|~])""") _PROFILE_GROUP_SPLIT = re.compile(r">\s+<") _DEFAULT_LOGGER: Optional[logging.Logger] = None _STDOUT_HANDLER: Optional[logging.StreamHandler[Any]] = None _STDERR_HANDLER: Optional[logging.StreamHandler[Any]] = None +PRINT_COMMAND = logging.INFO + 3 +PRINT_BUILD_SYSTEM_COMMAND = PRINT_COMMAND + 3 + +# Map them back to `INFO`. The names must be unique so the prefix is stripped. +logging.addLevelName(PRINT_COMMAND, "__INFO") +logging.addLevelName(PRINT_BUILD_SYSTEM_COMMAND, "_INFO") def assume_not_none(x: Optional[T]) -> T: @@ -115,6 +122,13 @@ def assume_not_none(x: Optional[T]) -> T: return x +def _non_verbose_info(msg: str) -> None: + global _DEFAULT_LOGGER + logger = _DEFAULT_LOGGER + if logger is not None: + logger.log(PRINT_BUILD_SYSTEM_COMMAND, msg) + + def _info(msg: str) -> None: global _DEFAULT_LOGGER logger = _DEFAULT_LOGGER @@ -123,6 +137,20 @@ def _info(msg: str) -> None: # No fallback print for info +def _is_debug_log_enabled() -> bool: + global _DEFAULT_LOGGER + logger = _DEFAULT_LOGGER + return logger is not None and logger.isEnabledFor(logging.DEBUG) + + +def _debug_log(msg: str) -> None: + global _DEFAULT_LOGGER + logger = _DEFAULT_LOGGER + if logger: + logger.debug(msg) + # No fallback print for info + + def _error(msg: str, *, prog: Optional[str] = None) -> "NoReturn": global _DEFAULT_LOGGER logger = _DEFAULT_LOGGER @@ -215,7 +243,13 @@ def _backslash_escape(m: re.Match[str]) -> str: def _escape_shell_word(w: str) -> str: - if _SPACE_RE.match(w): + if _SPACE_RE.search(w): + if "=" in w and (m := _WORD_EQUAL.search(w)) is not None: + s = m.span(0) + assert s[0] == 0 + prefix = w[0 : s[1]] + escaped_value = _DOUBLE_ESCAPEES.sub(_backslash_escape, w[s[1] :]) + return f'{prefix}"{escaped_value}"' w = _DOUBLE_ESCAPEES.sub(_backslash_escape, w) return f'"{w}"' return _REGULAR_ESCAPEES.sub(_backslash_escape, w) @@ -225,9 +259,88 @@ def escape_shell(*args: str) -> str: return " ".join(_escape_shell_word(w) for w in args) -def print_command(*args: str, print_at_log_level: int = logging.INFO) -> None: - if logging.getLogger().isEnabledFor(print_at_log_level): - print(f" {escape_shell(*args)}") +def render_command( + *args: str, + cwd: Optional[str] = None, + env_mod: Optional[EnvironmentModification] = None, +) -> str: + env_mod_prefix = "" + if env_mod: + env_mod_parts = [] + if bool(env_mod.removals): + env_mod_parts.append("env") + if cwd is not None: + env_mod_parts.append(f"--chdir={escape_shell(cwd)}") + env_mod_parts.extend(f"--unset={escape_shell(v)}" for v in env_mod.removals) + env_mod_parts.extend( + f"{escape_shell(k)}={escape_shell(v)}" for k, v in env_mod.replacements + ) + + chdir_prefix = "" + if cwd is not None and cwd != ".": + chdir_prefix = f"cd {escape_shell(cwd)} && " + return f"{chdir_prefix}{env_mod_prefix}{escape_shell(*args)}" + + +def print_command( + *args: str, + cwd: Optional[str] = None, + env_mod: Optional[EnvironmentModification] = None, + print_at_log_level: int = PRINT_COMMAND, +) -> None: + if _DEFAULT_LOGGER is None or not _DEFAULT_LOGGER.isEnabledFor(print_at_log_level): + return + + rendered_cmd = render_command( + *args, + cwd=cwd, + env_mod=env_mod, + ) + print(f" {rendered_cmd}") + + +def run_command( + *args: str, + cwd: Optional[str] = None, + env: Optional[Mapping[str, str]] = None, + env_mod: Optional[EnvironmentModification] = None, + print_at_log_level: int = PRINT_COMMAND, +) -> None: + print_command( + *args, + cwd=cwd, + env_mod=env_mod, + print_at_log_level=print_at_log_level, + ) + if env_mod: + if env is None: + env = os.environ + env = env_mod.compute_env(env) + if env is os.environ: + env = None + try: + subprocess.check_call(args, cwd=cwd, env=env) + # At least "clean_logic.py" relies on catching FileNotFoundError + except KeyboardInterrupt: + _error(f"Interrupted (SIGINT) while running {escape_shell(*args)}") + except subprocess.CalledProcessError as e: + _error(f"The command {escape_shell(*args)} failed with status: {e.returncode}") + + +def run_build_system_command( + *args: str, + cwd: Optional[str] = None, + env: Optional[Mapping[str, str]] = None, + env_mod: Optional[EnvironmentModification] = None, + print_at_log_level: int = PRINT_BUILD_SYSTEM_COMMAND, +) -> None: + run_command( + *args, + cwd=cwd, + env=env, + env_mod=env_mod, + print_at_log_level=print_at_log_level, + ) def debian_policy_normalize_symlink_target( @@ -397,7 +510,7 @@ def integrated_with_debhelper() -> None: _DH_INTEGRATION_MODE = True -def scratch_dir() -> str: +def scratch_dir(*, create_if_not_exists: bool = True) -> str: global _SCRATCH_DIR if _SCRATCH_DIR is not None: return _SCRATCH_DIR @@ -410,9 +523,10 @@ def scratch_dir() -> str: is_debputy_dir = False else: _SCRATCH_DIR = debputy_scratch_dir - ensure_dir(_SCRATCH_DIR) - if is_debputy_dir: - Path("debian/.debputy/.gitignore").write_text("*\n") + if create_if_not_exists: + ensure_dir(_SCRATCH_DIR) + if is_debputy_dir: + Path("debian/.debputy/.gitignore").write_text("*\n") return _SCRATCH_DIR @@ -454,9 +568,11 @@ def generated_content_dir( return directory -PerlIncDir = collections.namedtuple("PerlIncDir", ["vendorlib", "vendorarch"]) +PerlConfigVars = collections.namedtuple( + "PerlIncDir", ["vendorlib", "vendorarch", "cross_inc_dir", "ld", "path_sep"] +) PerlConfigData = collections.namedtuple("PerlConfigData", ["version", "debian_abi"]) -_PERL_MODULE_DIRS: Dict[str, PerlIncDir] = {} +_PERL_MODULE_DIRS: Dict[str, PerlConfigVars] = {} @functools.lru_cache(1) @@ -489,42 +605,56 @@ def perlxs_api_dependency() -> str: return f"perlapi-{config.version}" -def perl_module_dirs( +def resolve_perl_config( dpkg_architecture_variables: DpkgArchitectureBuildProcessValuesTable, - dctrl_bin: "BinaryPackage", -) -> PerlIncDir: + dctrl_bin: Optional["BinaryPackage"], +) -> PerlConfigVars: global _PERL_MODULE_DIRS - arch = ( - dctrl_bin.resolved_architecture - if dpkg_architecture_variables.is_cross_compiling - else "_default_" - ) - module_dir = _PERL_MODULE_DIRS.get(arch) - if module_dir is None: + if dpkg_architecture_variables.is_cross_compiling: + arch = ( + dctrl_bin.resolved_architecture + if dctrl_bin is not None + else dpkg_architecture_variables.current_host_arch + ) + else: + arch = "_build_arch_" + config_vars = _PERL_MODULE_DIRS.get(arch) + if config_vars is None: cmd = ["perl"] if dpkg_architecture_variables.is_cross_compiling: version = _perl_version() - inc_dir = f"/usr/lib/{dctrl_bin.deb_multiarch}/perl/cross-config-{version}" + cross_inc_dir = ( + f"/usr/lib/{dctrl_bin.deb_multiarch}/perl/cross-config-{version}" + ) # FIXME: This should not fallback to "build-arch" but on the other hand, we use the perl module dirs # for every package at the moment. So mandating correct perl dirs implies mandating perl-xs-dev in # cross builds... meh. - if os.path.exists(os.path.join(inc_dir, "Config.pm")): - cmd.append(f"-I{inc_dir}") + if os.path.exists(os.path.join(cross_inc_dir, "Config.pm")): + cmd.append(f"-I{cross_inc_dir}") + else: + cross_inc_dir = None cmd.extend( - ["-MConfig", "-e", 'print "$Config{vendorlib}\n$Config{vendorarch}\n"'] + [ + "-MConfig", + "-e", + 'print "$Config{vendorlib}\n$Config{vendorarch}\n$Config{ld}\n$Config{path_sep}\n"', + ] ) output = subprocess.check_output(cmd).decode("utf-8").splitlines(keepends=False) - if len(output) != 2: + if len(output) != 4: raise ValueError( "Internal error: Unable to determine the perl include directories:" f" Raw output from perl snippet: {output}" ) - module_dir = PerlIncDir( - vendorlib=_normalize_path(output[0]), - vendorarch=_normalize_path(output[1]), + config_vars = PerlConfigVars( + vendorlib="/" + _normalize_path(output[0], with_prefix=False), + vendorarch="/" + _normalize_path(output[1], with_prefix=False), + cross_inc_dir=cross_inc_dir, + ld=output[2], + path_sep=output[3], ) - _PERL_MODULE_DIRS[arch] = module_dir - return module_dir + _PERL_MODULE_DIRS[arch] = config_vars + return config_vars @functools.lru_cache(1) @@ -728,6 +858,20 @@ def package_cross_check_precheck( return a_may_see_b, b_may_see_a +def change_log_level( + log_level: int, +) -> None: + if _DEFAULT_LOGGER is not None: + _DEFAULT_LOGGER.setLevel(log_level) + logging.getLogger("").setLevel(log_level) + + +def current_log_level() -> Optional[int]: + if _DEFAULT_LOGGER is not None: + return _DEFAULT_LOGGER.level + return None + + def setup_logging( *, log_only_to_stderr: bool = False, @@ -740,13 +884,20 @@ def setup_logging( " Use reconfigure_logging=True if you need to reconfigure it" ) stdout_color, stderr_color, bad_request = _check_color() + colors: Optional[Dict[str, str]] = None if stdout_color or stderr_color: try: import colorlog + except ImportError: stdout_color = False stderr_color = False + else: + colors = dict(colorlog.default_log_colors) + # Add our custom levels. + colors["_INFO"] = colors["INFO"] + colors["__INFO"] = colors["INFO"] if log_only_to_stderr: stdout = sys.stderr @@ -777,7 +928,12 @@ def setup_logging( if stdout_color: stdout_handler = colorlog.StreamHandler(stdout) stdout_handler.setFormatter( - colorlog.ColoredFormatter(color_format, style="{", force_color=True) + colorlog.ColoredFormatter( + color_format, + style="{", + force_color=True, + log_colors=colors, + ) ) logger = colorlog.getLogger() if existing_stdout_handler is not None: @@ -796,7 +952,12 @@ def setup_logging( if stderr_color: stderr_handler = colorlog.StreamHandler(sys.stderr) stderr_handler.setFormatter( - colorlog.ColoredFormatter(color_format, style="{", force_color=True) + colorlog.ColoredFormatter( + color_format, + style="{", + force_color=True, + log_colors=colors, + ) ) logger = logging.getLogger() if existing_stderr_handler is not None: @@ -823,6 +984,7 @@ def setup_logging( *args: Any, **kwargs: Any ) -> logging.LogRecord: # pragma: no cover record = old_factory(*args, **kwargs) + record.levelname = record.levelname.lstrip("_") record.levelnamelower = record.levelname.lower() return record diff --git a/tests/lint_tests/lint_tutil.py b/tests/lint_tests/lint_tutil.py index 267f669..c16fde3 100644 --- a/tests/lint_tests/lint_tutil.py +++ b/tests/lint_tests/lint_tutil.py @@ -1,5 +1,5 @@ import collections -from typing import List, Optional, Mapping, Any, Callable +from typing import List, Optional, Mapping, Any, Callable, Sequence import pytest @@ -9,11 +9,14 @@ from debputy.linting.lint_util import ( LintStateImpl, LintState, ) -from debputy.lsp.style_prefs import StylePreferenceTable, EffectivePreference +from debputy.lsp.maint_prefs import ( + MaintainerPreferenceTable, + EffectiveFormattingPreference, +) from debputy.packages import DctrlParser from debputy.plugin.api.feature_set import PluginProvidedFeatureSet -from debputy.lsprotocol.types import Diagnostic, DiagnosticSeverity +from debputy.lsprotocol.types import Diagnostic, DiagnosticSeverity, Range try: @@ -42,8 +45,8 @@ class LintWrapper: self.path = path self._dctrl_parser = dctrl_parser self.source_root: Optional[VirtualPathBase] = None - self.lint_style_preference_table = StylePreferenceTable({}, {}) - self.effective_preference: Optional[EffectivePreference] = None + self.lint_maint_preference_table = MaintainerPreferenceTable({}, {}) + self.effective_preference: Optional[EffectiveFormattingPreference] = None def __call__(self, lines: List[str]) -> Optional[List["Diagnostic"]]: source_package = None @@ -59,7 +62,7 @@ class LintWrapper: debian_dir = source_root.get("debian") if source_root is not None else None state = LintStateImpl( self._debputy_plugin_feature_set, - self.lint_style_preference_table, + self.lint_maint_preference_table, source_root, debian_dir, self.path, @@ -108,3 +111,18 @@ def group_diagnostics_by_severity( by_severity[severity].append(diagnostic) return by_severity + + +def diag_range_to_text(lines: Sequence[str], range_: "Range") -> str: + parts = [] + for line_no in range(range_.start.line, range_.end.line + 1): + line = lines[line_no] + chunk = line + if line_no == range_.start.line and line_no == range_.end.line: + chunk = line[range_.start.character : range_.end.character] + elif line_no == range_.start.line: + chunk = line[range_.start.character :] + elif line_no == range_.end.line: + chunk = line[: range_.end.character] + parts.append(chunk) + return "".join(parts) diff --git a/tests/lint_tests/test_lint_dctrl.py b/tests/lint_tests/test_lint_dctrl.py index 745c323..229acc1 100644 --- a/tests/lint_tests/test_lint_dctrl.py +++ b/tests/lint_tests/test_lint_dctrl.py @@ -13,6 +13,7 @@ from lint_tests.lint_tutil import ( group_diagnostics_by_severity, requires_levenshtein, LintWrapper, + diag_range_to_text, ) from debputy.lsprotocol.types import Diagnostic, DiagnosticSeverity @@ -732,7 +733,63 @@ def test_dctrl_lint_ambiguous_pkgfile(line_linter: LintWrapper) -> None: # FIXME: This relies on "cwd" being a valid debian directory using debhelper. Fix and # remove the `build_time_only` restriction - line_linter.source_root = build_virtual_file_system(["./debian/bar.install"]) + line_linter.source_root = build_virtual_file_system( + [ + virtual_path_def(".", fs_path="."), + "./debian/bar.service", + ] + ) + + diagnostics = line_linter(lines) + print(diagnostics) + assert diagnostics and len(diagnostics) == 1 + issue = diagnostics[0] + + msg = ( + 'Possible typo in "./debian/bar.service". Consider renaming the file to "debian/foo.service"' + ' (or maybe "debian/foo.bar.service") if it is intended for foo' + ) + assert issue.message == msg + assert f"{issue.range}" == "7:0-8:0" + assert issue.severity == DiagnosticSeverity.Warning + diag_data = issue.data + assert isinstance(diag_data, dict) + assert diag_data.get("report_for_related_file") in ( + "./debian/bar.service", + "debian/bar.service", + ) + + +@build_time_only +def test_dctrl_lint_ambiguous_pkgfile_no_name_segment(line_linter: LintWrapper) -> None: + lines = textwrap.dedent( + f"""\ + Source: foo + Section: devel + Priority: optional + Standards-Version: {CURRENT_STANDARDS_VERSION} + Maintainer: Jane Developer <jane@example.com> + Build-Depends: debhelper-compat (= 13), dh-sequence-zz-debputy, + + Package: foo + Architecture: all + Depends: bar, baz + Description: some short synopsis + A very interesting description + with a valid synopsis + . + Just so be clear, this is for a test. + """ + ).splitlines(keepends=True) + + # FIXME: This relies on "cwd" being a valid debian directory using debhelper. Fix and + # remove the `build_time_only` restriction + line_linter.source_root = build_virtual_file_system( + [ + virtual_path_def(".", fs_path="."), + "./debian/bar.alternatives", + ] + ) diagnostics = line_linter(lines) print(diagnostics) @@ -740,8 +797,8 @@ def test_dctrl_lint_ambiguous_pkgfile(line_linter: LintWrapper) -> None: issue = diagnostics[0] msg = ( - 'Possible typo in "./debian/bar.install". Consider renaming the file to "debian/foo.bar.install"' - ' or "debian/foo.install if it is intended for foo' + 'Possible typo in "./debian/bar.alternatives". Consider renaming the file to "debian/foo.alternatives"' + " if it is intended for foo" ) assert issue.message == msg assert f"{issue.range}" == "7:0-8:0" @@ -749,8 +806,8 @@ def test_dctrl_lint_ambiguous_pkgfile(line_linter: LintWrapper) -> None: diag_data = issue.data assert isinstance(diag_data, dict) assert diag_data.get("report_for_related_file") in ( - "./debian/bar.install", - "debian/bar.install", + "./debian/bar.alternatives", + "debian/bar.alternatives", ) @@ -779,7 +836,12 @@ def test_dctrl_lint_stem_typo_pkgfile(line_linter: LintWrapper) -> None: # FIXME: This relies on "cwd" being a valid debian directory using debhelper. Fix and # remove the `build_time_only` restriction - line_linter.source_root = build_virtual_file_system(["./debian/foo.intsall"]) + line_linter.source_root = build_virtual_file_system( + [ + virtual_path_def(".", fs_path="."), + "./debian/foo.intsall", + ] + ) diagnostics = line_linter(lines) print(diagnostics) @@ -827,6 +889,7 @@ def test_dctrl_lint_stem_inactive_pkgfile_fp(line_linter: LintWrapper) -> None: # load the `zz-debputy` sequence. line_linter.source_root = build_virtual_file_system( [ + virtual_path_def(".", fs_path="."), "./debian/foo.install", virtual_path_def( "./debian/rules", @@ -846,3 +909,372 @@ def test_dctrl_lint_stem_inactive_pkgfile_fp(line_linter: LintWrapper) -> None: print(diagnostics) # We should not emit diagnostics when the package is not using dh! assert not diagnostics + + +@requires_levenshtein +@build_time_only +def test_dctrl_lint_stem_typo_pkgfile_ignored_exts_or_files( + line_linter: LintWrapper, +) -> None: + lines = textwrap.dedent( + f"""\ + Source: foo + Section: devel + Priority: optional + Standards-Version: {CURRENT_STANDARDS_VERSION} + Maintainer: Jane Developer <jane@example.com> + Build-Depends: debhelper-compat (= 13) + + Package: foo + Architecture: all + Depends: bar, baz + Description: some short synopsis + A very interesting description + with a valid synopsis + . + Just so be clear, this is for a test. + """ + ).splitlines(keepends=True) + + # FIXME: This relies on "cwd" being a valid debian directory using debhelper. Fix and + # remove the `build_time_only` restriction + line_linter.source_root = build_virtual_file_system( + [ + virtual_path_def(".", fs_path="."), + "debian/salsa-ci.yml", + "debian/gbp.conf", + "debian/foo.conf", + "debian/foo.sh", + "debian/foo.yml", + # One wrong one to ensure the test works. + "debian/foo.intsall", + ] + ) + + diagnostics = line_linter(lines) + print(diagnostics) + assert diagnostics and len(diagnostics) == 1 + issue = diagnostics[0] + + msg = 'The file "./debian/foo.intsall" is likely a typo of "./debian/foo.install"' + assert issue.message == msg + assert f"{issue.range}" == "7:0-8:0" + assert issue.severity == DiagnosticSeverity.Warning + diag_data = issue.data + assert isinstance(diag_data, dict) + assert diag_data.get("report_for_related_file") in ( + "./debian/foo.intsall", + "debian/foo.intsall", + ) + + +def test_dctrl_lint_dep_field_missing_sep( + line_linter: LintWrapper, +) -> None: + lines = textwrap.dedent( + f"""\ + Source: foo + Section: devel + Priority: optional + Standards-Version: {CURRENT_STANDARDS_VERSION} + Maintainer: Jane Developer <jane@example.com> + Build-Depends: debhelper-compat (= 13) + + Package: foo + Architecture: all + Depends: bar, baz + # Missing separator between baz and libfubar1 + libfubar1, + Description: some short synopsis + A very interesting description + with a valid synopsis + . + Just so be clear, this is for a test. + """ + ).splitlines(keepends=True) + + diagnostics = line_linter(lines) + print(diagnostics) + assert diagnostics and len(diagnostics) == 1 + issue = diagnostics[0] + msg = ( + "Trailing data after a relationship that might be a second relationship." + " Is a separator missing before this part?" + ) + problem_text = diag_range_to_text(lines, issue.range) + assert issue.message == msg + assert problem_text == "libfubar1" + assert f"{issue.range}" == "11:1-11:10" + assert issue.severity == DiagnosticSeverity.Error + + +def test_dctrl_lint_dep_field_missing_sep_or_syntax_error( + line_linter: LintWrapper, +) -> None: + lines = textwrap.dedent( + f"""\ + Source: foo + Section: devel + Priority: optional + Standards-Version: {CURRENT_STANDARDS_VERSION} + Maintainer: Jane Developer <jane@example.com> + Build-Depends: debhelper-compat (= 13) + + Package: foo + Architecture: all + Depends: bar, baz + # Missing separator between baz and libfubar1 + _libfubar1, + Description: some short synopsis + A very interesting description + with a valid synopsis + . + Just so be clear, this is for a test. + """ + ).splitlines(keepends=True) + + diagnostics = line_linter(lines) + print(diagnostics) + assert diagnostics and len(diagnostics) == 1 + issue = diagnostics[0] + msg = "Parse error of the relationship. Either a syntax error or a missing separator somewhere." + problem_text = diag_range_to_text(lines, issue.range) + assert issue.message == msg + assert problem_text == "_libfubar1" + assert f"{issue.range}" == "11:1-11:11" + assert issue.severity == DiagnosticSeverity.Error + + +def test_dctrl_lint_dep_field_completely_busted( + line_linter: LintWrapper, +) -> None: + lines = textwrap.dedent( + f"""\ + Source: foo + Section: devel + Priority: optional + Standards-Version: {CURRENT_STANDARDS_VERSION} + Maintainer: Jane Developer <jane@example.com> + Build-Depends: debhelper-compat (= 13) + + Package: foo + Architecture: all + Depends: bar, baz, _asd + # This is just busted + _libfubar1, + Description: some short synopsis + A very interesting description + with a valid synopsis + . + Just so be clear, this is for a test. + """ + ).splitlines(keepends=True) + + diagnostics = line_linter(lines) + print(diagnostics) + assert diagnostics and len(diagnostics) == 1 + issue = diagnostics[0] + msg = 'Could not parse "_asd _libfubar1" as a dependency relation.' + problem_text = diag_range_to_text(lines, issue.range) + expected_problem_text = "\n".join((" _asd", "# This is just busted", " _libfubar1")) + assert issue.message == msg + assert problem_text == expected_problem_text + assert f"{issue.range}" == "9:18-11:11" + assert issue.severity == DiagnosticSeverity.Error + + +def test_dctrl_lint_dep_field_completely_busted_first_line( + line_linter: LintWrapper, +) -> None: + lines = textwrap.dedent( + f"""\ + Source: foo + Section: devel + Priority: optional + Standards-Version: {CURRENT_STANDARDS_VERSION} + Maintainer: Jane Developer <jane@example.com> + Build-Depends: debhelper-compat (= 13) + + Package: foo + Architecture: all + # A wild field comment appeared! + Depends: _bar, + asd, + # This is fine (but the _bar part is not) + libfubar1, + Description: some short synopsis + A very interesting description + with a valid synopsis + . + Just so be clear, this is for a test. + """ + ).splitlines(keepends=True) + + diagnostics = line_linter(lines) + print(diagnostics) + assert diagnostics and len(diagnostics) == 1 + issue = diagnostics[0] + msg = 'Could not parse "_bar" as a dependency relation.' + problem_text = diag_range_to_text(lines, issue.range) + assert issue.message == msg + assert problem_text == " _bar" + assert f"{issue.range}" == "10:8-10:13" + assert issue.severity == DiagnosticSeverity.Error + + +def test_dctrl_lint_dep_field_restricted_operator( + line_linter: LintWrapper, +) -> None: + lines = textwrap.dedent( + f"""\ + Source: foo + Section: devel + Priority: optional + Standards-Version: {CURRENT_STANDARDS_VERSION} + Maintainer: Jane Developer <jane@example.com> + Build-Depends: debhelper-compat (= 13) + + Package: foo + Architecture: all + # Some random field comment + Provides: bar (>= 2), + bar + # Inline comment to spice up things + (<= 1), + # This one is valid + fubar (= 2), + Description: some short synopsis + A very interesting description + with a valid synopsis + . + Just so be clear, this is for a test. + """ + ).splitlines(keepends=True) + + diagnostics = line_linter(lines) + print(diagnostics) + assert diagnostics and len(diagnostics) == 2 + first_issue, second_issue = diagnostics + + msg = 'The version operator ">=" is not allowed in Provides' + problem_text = diag_range_to_text(lines, first_issue.range) + assert first_issue.message == msg + assert problem_text == ">=" + assert f"{first_issue.range}" == "10:15-10:17" + assert first_issue.severity == DiagnosticSeverity.Error + + msg = 'The version operator "<=" is not allowed in Provides' + problem_text = diag_range_to_text(lines, second_issue.range) + assert second_issue.message == msg + assert problem_text == "<=" + assert f"{second_issue.range}" == "13:2-13:4" + assert second_issue.severity == DiagnosticSeverity.Error + + +def test_dctrl_lint_dep_field_restricted_or_relations( + line_linter: LintWrapper, +) -> None: + lines = textwrap.dedent( + f"""\ + Source: foo + Section: devel + Priority: optional + Standards-Version: {CURRENT_STANDARDS_VERSION} + Maintainer: Jane Developer <jane@example.com> + Build-Depends: debhelper-compat (= 13) + + Package: foo + Architecture: all + Depends: pkg-a + | pkg-b + # What goes in Depends do not always work in Provides + Provides: foo-a + | foo-b + Description: some short synopsis + A very interesting description + with a valid synopsis + . + Just so be clear, this is for a test. + """ + ).splitlines(keepends=True) + + diagnostics = line_linter(lines) + print(diagnostics) + assert diagnostics and len(diagnostics) == 1 + issue = diagnostics[0] + + msg = 'The field Provides does not support "|" (OR) in relations.' + problem_text = diag_range_to_text(lines, issue.range) + assert issue.message == msg + assert problem_text == "|" + assert f"{issue.range}" == "13:1-13:2" + assert issue.severity == DiagnosticSeverity.Error + + +def test_dctrl_duplicate_key(line_linter: LintWrapper) -> None: + lines = textwrap.dedent( + f"""\ + Source: jquery-tablesorter + Section: javascript + Priority: optional + Maintainer: Debian Javascript Maintainers <pkg-javascript-devel@lists.alioth.de\ + bian.org> + Uploaders: Paul Gevers <elbrus@debian.org> + Build-Depends: + debhelper-compat (=13), + grunt, + libjs-qunit, + node-grunt-contrib-clean, + node-grunt-contrib-copy, + node-grunt-contrib-uglify, + node-grunt-contrib-concat, + Standards-Version: {CURRENT_STANDARDS_VERSION} + Homepage: https://github.com/Mottie/tablesorter + Vcs-Git: https://salsa.debian.org/js-team/jquery-tablesorter.git + Vcs-Browser: https://salsa.debian.org/js-team/jquery-tablesorter + Rules-Requires-Root: no + + Package: libjs-jquery-tablesorter + Architecture: all + Multi-Arch: foreign + Depends: + ${{misc:Depends}}, + libjs-jquery, + libjs-jquery-metadata, + Recommends: javascript-common + Multi-Arch: foreign + Description: jQuery flexible client-side table sorting plugin + Tablesorter is a jQuery plugin for turning a standard HTML table with THEAD + and TBODY tags into a sortable table without page refreshes. Tablesorter can + successfully parse and sort many types of data including linked data in a + cell. It has many useful features including: + . + * Multi-column alphanumeric sorting and filtering. + * Multi-tbody sorting + * Supports Bootstrap v2-4. + * Parsers for sorting text, alphanumeric text, URIs, integers, currency, + floats, IP addresses, dates (ISO, long and short formats) and time. + Add your own easily. + * Inline editing + * Support for ROWSPAN and COLSPAN on TH elements. + * Support secondary "hidden" sorting (e.g., maintain alphabetical sort when + sorting on other criteria). + * Extensibility via widget system. + * Cross-browser: IE 6.0+, FF 2+, Safari 2.0+, Opera 9.0+, Chrome 5.0+. + + """ + ).splitlines(keepends=True) + + diagnostics = line_linter(lines) + assert len(diagnostics) == 1 + + issue = diagnostics[0] + + msg = ( + "The Multi-Arch field name was used multiple times in this stanza." + " Please ensure the field is only used once per stanza. Note that Multi-Arch and" + " X[BCS]-Multi-Arch are considered the same field." + ) + assert issue.message == msg + assert f"{issue.range}" == "27:0-27:10" + assert issue.severity == DiagnosticSeverity.Error diff --git a/tests/lsp_tests/test_lsp_debputy_manifest_completer.py b/tests/lsp_tests/test_lsp_debputy_manifest_completer.py index f052164..92056ed 100644 --- a/tests/lsp_tests/test_lsp_debputy_manifest_completer.py +++ b/tests/lsp_tests/test_lsp_debputy_manifest_completer.py @@ -200,7 +200,7 @@ def test_basic_debputy_completer_manifest_variable_value( ) assert isinstance(completions, list) keywords = {m.label for m in completions} - assert "0.1" in keywords + assert "'0.1'" in keywords cursor_pos = put_doc_with_cursor( ls, @@ -219,7 +219,7 @@ def test_basic_debputy_completer_manifest_variable_value( ) assert isinstance(completions, list) keywords = {m.label for m in completions} - assert "0.1" in keywords + assert "'0.1'" in keywords def test_basic_debputy_completer_install_rule_dispatch_key( diff --git a/tests/test_debputy_plugin.py b/tests/test_debputy_plugin.py index dc60597..8a4cc59 100644 --- a/tests/test_debputy_plugin.py +++ b/tests/test_debputy_plugin.py @@ -19,6 +19,16 @@ from debputy.plugin.api.test_api import ( from debputy.plugin.api.test_api import manifest_variable_resolution_context from debputy.plugin.api.test_api.test_impl import initialize_plugin_under_test_preloaded from debputy.plugin.api.test_api.test_spec import DetectedService +from debputy.plugin.debputy.build_system_rules import ( + AutoconfBuildSystemRule, + MakefileBuildSystemRule, + PerlBuildBuildSystemRule, + PerlMakeMakerBuildSystemRule, + QmakeBuildSystemRule, + Qmake6BuildSystemRule, + CMakeBuildSystemRule, + MesonBuildSystemRule, +) from debputy.plugin.debputy.debputy_plugin import initialize_debputy_features from debputy.plugin.debputy.private_api import load_libcap from debputy.plugin.debputy.service_management import SystemdServiceContext @@ -1252,3 +1262,129 @@ def test_auto_depends_solink() -> None: context=context_too_many_matches, ) assert "misc:Depends" not in sodep_metadata.substvars + + +@pytest.mark.parametrize( + "filename,expected,mode,content", + [ + ("configure.ac", True, 0o0644, None), + ("configure.in", True, 0o0644, "AC_INIT"), + ("configure.in", True, 0o0644, "AC_PREREQ"), + ("configure.in", False, 0o0644, "None of the above"), + ("configure", True, 0o0755, "GNU Autoconf"), + ("configure", False, 0o0644, "GNU Autoconf"), + ("configure", False, 0o0755, "No magic keyword"), + ("random-file", False, 0o0644, "No configure at all"), + ], +) +def test_auto_detect_autoconf_build_system( + filename: str, + expected: bool, + mode: int, + content: Optional[str], +) -> None: + fs_root = build_virtual_file_system( + [virtual_path_def(filename, mode=mode, content=content)] + ) + detected = AutoconfBuildSystemRule.auto_detect_build_system(fs_root) + assert detected == expected + + +@pytest.mark.parametrize( + "filename,expected", + [ + ("GNUmakefile", True), + ("Makefile", True), + ("makefile", True), + ("random-file", False), + ], +) +def test_auto_detect_make_build_system( + filename: str, + expected: bool, +) -> None: + fs_root = build_virtual_file_system([filename]) + detected = MakefileBuildSystemRule.auto_detect_build_system(fs_root) + assert detected == expected + + +@pytest.mark.parametrize( + "filename,expected", + [ + ("Build.PL", True), + ("random-file", False), + ], +) +def test_auto_detect_perl_build_build_system( + filename: str, + expected: bool, +) -> None: + fs_root = build_virtual_file_system([filename]) + detected = PerlBuildBuildSystemRule.auto_detect_build_system(fs_root) + assert detected == expected + + +@pytest.mark.parametrize( + "filename,expected", + [ + ("Makefile.PL", True), + ("random-file", False), + ], +) +def test_auto_detect_perl_makemaker_build_system( + filename: str, + expected: bool, +) -> None: + fs_root = build_virtual_file_system([filename]) + detected = PerlMakeMakerBuildSystemRule.auto_detect_build_system(fs_root) + assert detected == expected + + +@pytest.mark.parametrize( + "filename,expected", + [ + ("foo.pro", True), + ("random-file", False), + ], +) +def test_auto_detect_qmake_build_systems( + filename: str, + expected: bool, +) -> None: + fs_root = build_virtual_file_system([filename]) + detected_qmake = QmakeBuildSystemRule.auto_detect_build_system(fs_root) + detected_qmake6 = Qmake6BuildSystemRule.auto_detect_build_system(fs_root) + assert detected_qmake == expected + assert detected_qmake6 == expected + + +@pytest.mark.parametrize( + "filename,expected", + [ + ("CMakeLists.txt", True), + ("random-file", False), + ], +) +def test_auto_detect_cmake_build_systems( + filename: str, + expected: bool, +) -> None: + fs_root = build_virtual_file_system([filename]) + detected = CMakeBuildSystemRule.auto_detect_build_system(fs_root) + assert detected == expected + + +@pytest.mark.parametrize( + "filename,expected", + [ + ("meson.build", True), + ("random-file", False), + ], +) +def test_auto_detect_meson_build_systems( + filename: str, + expected: bool, +) -> None: + fs_root = build_virtual_file_system([filename]) + detected = MesonBuildSystemRule.auto_detect_build_system(fs_root) + assert detected == expected diff --git a/tests/test_declarative_parser.py b/tests/test_declarative_parser.py index 26291dd..d52f1c3 100644 --- a/tests/test_declarative_parser.py +++ b/tests/test_declarative_parser.py @@ -10,11 +10,12 @@ from typing import ( import pytest from debputy.highlevel_manifest import PackageTransformationDefinition -from debputy.manifest_parser.base_types import DebputyParsedContent, TypeMapping -from debputy.manifest_parser.declarative_parser import ( - DebputyParseHint, - ParserGenerator, +from debputy.manifest_parser.tagging_types import ( + DebputyParsedContent, + TypeMapping, ) +from debputy.manifest_parser.parse_hints import DebputyParseHint +from debputy.manifest_parser.declarative_parser import ParserGenerator from debputy.manifest_parser.mapper_code import type_mapper_str2package from debputy.manifest_parser.parser_data import ParserContextData from debputy.manifest_parser.util import AttributePath diff --git a/tests/test_fs_metadata.py b/tests/test_fs_metadata.py index 7dd3d55..3cbbb03 100644 --- a/tests/test_fs_metadata.py +++ b/tests/test_fs_metadata.py @@ -132,7 +132,9 @@ def test_mtime_clamp_and_builtin_dir_mode( verify_paths(intermediate_manifest, path_defs) -def test_transformations_create_symlink(manifest_parser_pkg_foo): +def test_transformations_create_symlink( + manifest_parser_pkg_foo: YAMLManifestParser, +) -> None: content = textwrap.dedent( """\ manifest-version: '0.1' @@ -178,7 +180,9 @@ def test_transformations_create_symlink(manifest_parser_pkg_foo): verify_paths(intermediate_manifest, expected_results) -def test_transformations_create_symlink_replace_success(manifest_parser_pkg_foo): +def test_transformations_create_symlink_replace_success( + manifest_parser_pkg_foo: YAMLManifestParser, +) -> None: content = textwrap.dedent( """\ manifest-version: '0.1' @@ -230,8 +234,10 @@ def test_transformations_create_symlink_replace_success(manifest_parser_pkg_foo) ], ) def test_transformations_create_symlink_replace_failure( - manifest_parser_pkg_foo, replacement_rule, reason -): + manifest_parser_pkg_foo: YAMLManifestParser, + replacement_rule: str, + reason: str, +) -> None: content = textwrap.dedent( f"""\ manifest-version: '0.1' @@ -257,14 +263,15 @@ def test_transformations_create_symlink_replace_failure( f"Refusing to replace ./usr/share/foo with a symlink; {reason} and the active" f" replacement-rule was {replacement_rule}. You can set the replacement-rule to" ' "discard-existing", if you are not interested in the contents of ./usr/share/foo. This error' - " was triggered by packages.foo.transformations[0].create-symlink <Search for: usr/share/foo>." + # Ideally, this would be reported for line 5. + " was triggered by packages.foo.transformations[0].create-symlink [Line 6 column 18]." ) assert e_info.value.args[0] == msg def test_transformations_create_symlink_replace_with_explicit_remove( - manifest_parser_pkg_foo, -): + manifest_parser_pkg_foo: YAMLManifestParser, +) -> None: content = textwrap.dedent( """\ manifest-version: '0.1' @@ -299,8 +306,8 @@ def test_transformations_create_symlink_replace_with_explicit_remove( def test_transformations_create_symlink_replace_with_replacement_rule( - manifest_parser_pkg_foo, -): + manifest_parser_pkg_foo: YAMLManifestParser, +) -> None: content = textwrap.dedent( """\ manifest-version: '0.1' @@ -335,7 +342,9 @@ def test_transformations_create_symlink_replace_with_replacement_rule( verify_paths(intermediate_manifest, expected_results) -def test_transformations_path_metadata(manifest_parser_pkg_foo): +def test_transformations_path_metadata( + manifest_parser_pkg_foo: YAMLManifestParser, +) -> None: content = textwrap.dedent( """\ manifest-version: '0.1' @@ -382,7 +391,9 @@ def test_transformations_path_metadata(manifest_parser_pkg_foo): verify_paths(intermediate_manifest, expected_results) -def test_transformations_directories(manifest_parser_pkg_foo): +def test_transformations_directories( + manifest_parser_pkg_foo: YAMLManifestParser, +) -> None: content = textwrap.dedent( """\ manifest-version: '0.1' @@ -453,7 +464,9 @@ def test_transformations_directories(manifest_parser_pkg_foo): verify_paths(intermediate_manifest, expected_results) -def test_transformation_remove(manifest_parser_pkg_foo): +def test_transformation_remove( + manifest_parser_pkg_foo: YAMLManifestParser, +) -> None: content = textwrap.dedent( """\ manifest-version: '0.1' @@ -489,7 +502,9 @@ def test_transformation_remove(manifest_parser_pkg_foo): verify_paths(intermediate_manifest, expected_results) -def test_transformation_remove_keep_empty(manifest_parser_pkg_foo): +def test_transformation_remove_keep_empty( + manifest_parser_pkg_foo: YAMLManifestParser, +) -> None: content = textwrap.dedent( """\ manifest-version: '0.1' @@ -529,7 +544,9 @@ def test_transformation_remove_keep_empty(manifest_parser_pkg_foo): verify_paths(intermediate_manifest, expected_results) -def test_transformation_remove_glob(manifest_parser_pkg_foo): +def test_transformation_remove_glob( + manifest_parser_pkg_foo: YAMLManifestParser, +) -> None: content = textwrap.dedent( """\ manifest-version: '0.1' @@ -581,7 +598,9 @@ def test_transformation_remove_glob(manifest_parser_pkg_foo): verify_paths(intermediate_manifest, expected_results) -def test_transformation_remove_no_match(manifest_parser_pkg_foo): +def test_transformation_remove_no_match( + manifest_parser_pkg_foo: YAMLManifestParser, +) -> None: content = textwrap.dedent( """\ manifest-version: '0.1' @@ -612,13 +631,15 @@ def test_transformation_remove_no_match(manifest_parser_pkg_foo): manifest.apply_to_binary_staging_directory("foo", fs_root, claim_mtime_to) expected = ( 'The match rule "./some/non-existing-path" in transformation' - ' "packages.foo.transformations[0].remove <Search for: some/non-existing-path>" did not match any paths. Either' + ' "packages.foo.transformations[0].remove [Line 5 column 18]" did not match any paths. Either' " the definition is redundant (and can be omitted) or the match rule is incorrect." ) assert expected == e_info.value.args[0] -def test_transformation_move_basic(manifest_parser_pkg_foo): +def test_transformation_move_basic( + manifest_parser_pkg_foo: YAMLManifestParser, +) -> None: content = textwrap.dedent( """\ manifest-version: '0.1' @@ -682,7 +703,9 @@ def test_transformation_move_basic(manifest_parser_pkg_foo): verify_paths(intermediate_manifest, expected_results) -def test_transformation_move_no_match(manifest_parser_pkg_foo): +def test_transformation_move_no_match( + manifest_parser_pkg_foo: YAMLManifestParser, +) -> None: content = textwrap.dedent( """\ manifest-version: '0.1' @@ -715,13 +738,15 @@ def test_transformation_move_no_match(manifest_parser_pkg_foo): manifest.apply_to_binary_staging_directory("foo", fs_root, claim_mtime_to) expected = ( 'The match rule "./some/non-existing-path" in transformation' - ' "packages.foo.transformations[0].move <Search for: some/non-existing-path>" did not match any paths. Either' + ' "packages.foo.transformations[0].move [Line 6 column 12]" did not match any paths. Either' " the definition is redundant (and can be omitted) or the match rule is incorrect." ) assert expected == e_info.value.args[0] -def test_builtin_mode_normalization(manifest_parser_pkg_foo): +def test_builtin_mode_normalization_shell_scripts( + manifest_parser_pkg_foo: YAMLManifestParser, +) -> None: manifest = manifest_parser_pkg_foo.build_manifest() claim_mtime_to = 255 sh_script_content = "#!/bin/sh" @@ -773,3 +798,53 @@ def test_builtin_mode_normalization(manifest_parser_pkg_foo): print(intermediate_manifest) verify_paths(intermediate_manifest, expected_results) + + +def test_builtin_mode_normalization( + manifest_parser_pkg_foo: YAMLManifestParser, +) -> None: + manifest = manifest_parser_pkg_foo.build_manifest() + claim_mtime_to = 255 + + paths = [ + virtual_path_def("usr/", mode=0o755, mtime=10, fs_path="/nowhere/usr"), + virtual_path_def( + "usr/share/", mode=0o755, mtime=10, fs_path="/nowhere/usr/share" + ), + virtual_path_def( + "usr/share/perl5/", mode=0o755, mtime=10, fs_path="/nowhere/usr/share/perl5" + ), + virtual_path_def( + "usr/share/perl5/Foo.pm", + # #1076346 + mode=0o444, + mtime=10, + fs_path="/nowhere/Foo.pm", + ), + virtual_path_def( + "usr/share/perl5/Bar.pm", + mode=0o755, + mtime=10, + fs_path="/nowhere/Bar.pm", + ), + ] + + fs_root = build_virtual_fs(paths, read_write_fs=True) + assert [p.name for p in manifest.all_packages] == ["foo"] + + expected_results = [ + ("usr/", Expected(mode=0o755, mtime=10)), + ("usr/share/", Expected(mode=0o755, mtime=10)), + ("usr/share/perl5/", Expected(mode=0o755, mtime=10)), + ("usr/share/perl5/Bar.pm", Expected(mode=0o644, mtime=10)), + ("usr/share/perl5/Foo.pm", Expected(mode=0o644, mtime=10)), + ] + assert [p.name for p in manifest.all_packages] == ["foo"] + + intermediate_manifest = manifest.apply_to_binary_staging_directory( + "foo", fs_root, claim_mtime_to + ) + + print(intermediate_manifest) + + verify_paths(intermediate_manifest, expected_results) diff --git a/tests/test_install_rules.py b/tests/test_install_rules.py index a361864..ecc49fd 100644 --- a/tests/test_install_rules.py +++ b/tests/test_install_rules.py @@ -9,6 +9,7 @@ from debputy.installations import ( SearchDir, ) from debputy.plugin.api import virtual_path_def +from debputy.plugin.api.spec import INTEGRATION_MODE_DH_DEBPUTY from debputy.plugin.api.test_api import build_virtual_file_system @@ -118,13 +119,14 @@ def test_install_rules(manifest_parser_pkg_foo) -> None: all_pkgs = frozenset(manifest.all_packages) result = manifest.perform_installations( + INTEGRATION_MODE_DH_DEBPUTY, install_request_context=InstallSearchDirContext( [ SearchDir(debian_tmp_dir, all_pkgs), SearchDir(debian_source_root_dir, all_pkgs), ], [debian_tmp_dir], - ) + ), ) assert "foo" in result foo_fs_root = result["foo"].fs_root @@ -207,12 +209,13 @@ def test_multi_dest_install_rules(manifest_parser_pkg_foo) -> None: all_pkgs = frozenset(manifest.all_packages) result = manifest.perform_installations( + INTEGRATION_MODE_DH_DEBPUTY, install_request_context=InstallSearchDirContext( [ SearchDir(debian_source_root_dir, all_pkgs), ], [], - ) + ), ) assert "foo" in result foo_fs_root = result["foo"].fs_root @@ -300,13 +303,14 @@ def test_install_rules_with_glob(manifest_parser_pkg_foo) -> None: all_pkgs = frozenset(manifest.all_packages) result = manifest.perform_installations( + INTEGRATION_MODE_DH_DEBPUTY, install_request_context=InstallSearchDirContext( [ SearchDir(debian_tmp_dir, all_pkgs), SearchDir(debian_source_root_dir, all_pkgs), ], [debian_tmp_dir], - ) + ), ) assert "foo" in result foo_fs_root = result["foo"].fs_root @@ -365,13 +369,14 @@ def test_install_rules_auto_discard_rules_dir(manifest_parser_pkg_foo) -> None: all_pkgs = frozenset(manifest.all_packages) result = manifest.perform_installations( + INTEGRATION_MODE_DH_DEBPUTY, install_request_context=InstallSearchDirContext( [ SearchDir(debian_tmp_dir, all_pkgs), SearchDir(debian_source_root_dir, all_pkgs), ], [debian_tmp_dir], - ) + ), ) assert "foo" in result foo_fs_root = result["foo"].fs_root @@ -424,13 +429,14 @@ def test_install_rules_auto_discard_rules_glob(manifest_parser_pkg_foo) -> None: all_pkgs = frozenset(manifest.all_packages) result = manifest.perform_installations( + INTEGRATION_MODE_DH_DEBPUTY, install_request_context=InstallSearchDirContext( [ SearchDir(debian_tmp_dir, all_pkgs), SearchDir(debian_source_root_dir, all_pkgs), ], [debian_tmp_dir], - ) + ), ) assert "foo" in result foo_fs_root = result["foo"].fs_root @@ -490,13 +496,14 @@ def test_install_rules_auto_discard_rules_overruled_by_explicit_install_rule( all_pkgs = frozenset(manifest.all_packages) result = manifest.perform_installations( + INTEGRATION_MODE_DH_DEBPUTY, install_request_context=InstallSearchDirContext( [ SearchDir(debian_tmp_dir, all_pkgs), SearchDir(debian_source_root_dir, all_pkgs), ], [debian_tmp_dir], - ) + ), ) assert "foo" in result foo_fs_root = result["foo"].fs_root @@ -559,13 +566,14 @@ def test_install_rules_install_as_with_var(manifest_parser_pkg_foo) -> None: all_pkgs = frozenset(manifest.all_packages) result = manifest.perform_installations( + INTEGRATION_MODE_DH_DEBPUTY, install_request_context=InstallSearchDirContext( [ SearchDir(debian_tmp_dir, all_pkgs), SearchDir(debian_source_root_dir, all_pkgs), ], [debian_tmp_dir], - ) + ), ) assert "foo" in result foo_fs_root = result["foo"].fs_root @@ -610,17 +618,18 @@ def test_install_rules_no_matches(manifest_parser_pkg_foo) -> None: with pytest.raises(NoMatchForInstallPatternError) as e_info: manifest.perform_installations( + INTEGRATION_MODE_DH_DEBPUTY, install_request_context=InstallSearchDirContext( [ SearchDir(debian_tmp_dir, all_pkgs), SearchDir(debian_source_root_dir, all_pkgs), ], [debian_tmp_dir], - ) + ), ) expected_msg = ( "There were no matches for build/private-arch-tool in /nowhere/debian/tmp, /nowhere" - " (definition: installations[0].install <Search for: build/private-arch-tool>)." + " (definition: installations[0].install [Line 5 column 6])." " Match rule: ./build/private-arch-tool (the exact path / no globbing)" ) assert e_info.value.message == expected_msg @@ -719,6 +728,7 @@ def test_install_rules_per_package_search_dirs(manifest_parser_pkg_foo_w_udeb) - all_udeb_pkgs = frozenset({p for p in all_pkgs if p.is_udeb}) result = manifest.perform_installations( + INTEGRATION_MODE_DH_DEBPUTY, install_request_context=InstallSearchDirContext( [ SearchDir(debian_tmp_deb_dir, all_deb_pkgs), @@ -726,7 +736,7 @@ def test_install_rules_per_package_search_dirs(manifest_parser_pkg_foo_w_udeb) - SearchDir(debian_source_root_dir, all_pkgs), ], [debian_tmp_deb_dir], - ) + ), ) for pkg, ptype in [ ("foo", "deb"), @@ -820,12 +830,13 @@ def test_install_rules_multi_into(manifest_parser_pkg_foo_w_udeb) -> None: all_pkgs = frozenset(manifest.all_packages) result = manifest.perform_installations( + INTEGRATION_MODE_DH_DEBPUTY, install_request_context=InstallSearchDirContext( [ SearchDir(debian_source_root_dir, all_pkgs), ], [], - ) + ), ) for pkg in ["foo", "foo-udeb"]: assert pkg in result @@ -939,6 +950,7 @@ def test_auto_install_d_pkg(manifest_parser_pkg_foo_w_udeb) -> None: all_pkgs = frozenset(manifest.all_packages) result = manifest.perform_installations( + INTEGRATION_MODE_DH_DEBPUTY, install_request_context=InstallSearchDirContext( [ SearchDir(debian_source_root_dir, all_pkgs), @@ -948,7 +960,7 @@ def test_auto_install_d_pkg(manifest_parser_pkg_foo_w_udeb) -> None: "foo": debian_foo_dir, "foo-udeb": debian_foo_udeb_dir, }, - ) + ), ) for pkg in ["foo", "foo-udeb"]: assert pkg in result @@ -1024,12 +1036,13 @@ def test_install_doc_rules_ignore_udeb(manifest_parser_pkg_foo_w_udeb) -> None: all_pkgs = frozenset(manifest.all_packages) result = manifest.perform_installations( + INTEGRATION_MODE_DH_DEBPUTY, install_request_context=InstallSearchDirContext( [ SearchDir(debian_source_root_dir, all_pkgs), ], [], - ) + ), ) assert "foo" in result foo_fs_root = result["foo"].fs_root diff --git a/tests/test_migrations.py b/tests/test_migrations.py index f53c716..0704c60 100644 --- a/tests/test_migrations.py +++ b/tests/test_migrations.py @@ -4,6 +4,7 @@ from typing import Callable, Optional, List, Tuple, Sequence import pytest +from debputy.commands.debputy_cmd.output import no_fancy_output from debputy.dh_migration.migrators import Migrator from debputy.dh_migration.migrators_impl import ( migrate_tmpfile, @@ -15,7 +16,7 @@ from debputy.dh_migration.migrators_impl import ( migrate_install_file, migrate_maintscript, migrate_links_files, - detect_dh_addons, + detect_dh_addons_with_zz_integration, migrate_not_installed_file, migrate_installman_file, migrate_bash_completion, @@ -99,7 +100,7 @@ def run_migrator( *, migration_target=INTEGRATION_MODE_DH_DEBPUTY, ) -> FeatureMigration: - feature_migration = FeatureMigration(migrator.__name__) + feature_migration = FeatureMigration(migrator.__name__, no_fancy_output()) migrator( path, manifest, @@ -1308,7 +1309,7 @@ def test_detect_obsolete_substvars( ) msg = ( "The following relationship substitution variables can be removed from foo:" - " ${misc:Depends}, ${shlibs:Depends}, ${so:Depends}" + " ${misc:Depends}, ${shlibs:Depends}, ${so:Depends} (Note: https://bugs.debian.org/1067653)" ) assert migration.anything_to_do assert migration.warnings == [msg] @@ -1356,7 +1357,7 @@ def test_detect_obsolete_substvars_remove_field( ) msg = ( "The following relationship fields can be removed from foo: Pre-Depends." - " (The content in them would be applied automatically.)" + " (The content in them would be applied automatically. Note: https://bugs.debian.org/1067653)" ) assert migration.anything_to_do assert migration.warnings == [msg] @@ -1406,7 +1407,7 @@ def test_detect_obsolete_substvars_remove_field_essential( ) msg = ( "The following relationship fields can be removed from foo: Pre-Depends." - " (The content in them would be applied automatically.)" + " (The content in them would be applied automatically. Note: https://bugs.debian.org/1067653)" ) assert migration.anything_to_do assert migration.warnings == [msg] @@ -1465,7 +1466,7 @@ def test_detect_dh_addons( accept_no_migration_issues: AcceptableMigrationIssues, accept_any_migration_issues: AcceptableMigrationIssues, ) -> None: - migrator = detect_dh_addons + migrator = detect_dh_addons_with_zz_integration empty_fs = build_virtual_file_system([DEBIAN_DIR_ENTRY]) dctrl_no_addons_content = textwrap.dedent( diff --git a/tests/test_parser.py b/tests/test_parser.py index 4aee024..0b1ed56 100644 --- a/tests/test_parser.py +++ b/tests/test_parser.py @@ -124,7 +124,10 @@ def test_parsing_variables_reserved(manifest_parser_pkg_foo, varname): with pytest.raises(ManifestParseException) as e_info: manifest_parser_pkg_foo.parse_manifest(fd=content) - msg = f'The variable "{varname}" is already reserved/defined. Error triggered by definitions.variables.{varname}.' + msg = ( + f'The variable "{varname}" is already reserved/defined.' + f" Error triggered by definitions.variables.{varname} [Line 4 column 4]." + ) assert normalize_doc_link(e_info.value.args[0]) == msg @@ -163,7 +166,7 @@ def test_parsing_variables_unused(manifest_parser_pkg_foo): msg = ( 'The variable "UNUSED" is unused. Either use it or remove it.' - " The variable was declared at definitions.variables.UNUSED." + " The variable was declared at definitions.variables.UNUSED [Line 4 column 4]." ) assert normalize_doc_link(e_info.value.args[0]) == msg @@ -181,7 +184,7 @@ def test_parsing_package_foo_empty(manifest_parser_pkg_foo): manifest_parser_pkg_foo.parse_manifest(fd=content) msg = ( - "The attribute packages.foo must be a non-empty mapping. Please see" + "The attribute packages.foo [Line 3 column 4] must be a non-empty mapping. Please see" " {{DEBPUTY_DOC_ROOT_DIR}}/MANIFEST-FORMAT.md#binary-package-rules for the documentation." ) assert normalize_doc_link(e_info.value.args[0]) == msg @@ -238,8 +241,8 @@ def test_create_symlinks_missing_path(manifest_parser_pkg_foo): manifest_parser_pkg_foo.parse_manifest(fd=content) msg = ( - "The following keys were required but not present at packages.foo.transformations[0].create-symlink: 'path'" - " (Documentation: " + "The following keys were required but not present at packages.foo.transformations[0].create-symlink" + " [Line 5 column 12]: 'path' (Documentation: " "{{DEBPUTY_DOC_ROOT_DIR}}/MANIFEST-FORMAT.md#create-symlinks-transformation-rule-create-symlink)" ) assert normalize_doc_link(e_info.value.args[0]) == msg @@ -263,7 +266,7 @@ def test_create_symlinks_unknown_replacement_rule(manifest_parser_pkg_foo): manifest_parser_pkg_foo.parse_manifest(fd=content) msg = ( - 'The attribute "packages.foo.transformations[0].create-symlink.replacement-rule <Search for: usr/share/foo>"' + 'The attribute "packages.foo.transformations[0].create-symlink.replacement-rule [Line 8 column 32]"' " did not have a valid structure/type: Value (golf) must be one of the following literal values:" ' "error-if-exists", "error-if-directory", "abort-on-non-empty-directory", "discard-existing"' ) @@ -286,8 +289,8 @@ def test_create_symlinks_missing_target(manifest_parser_pkg_foo): manifest_parser_pkg_foo.parse_manifest(fd=content) msg = ( - "The following keys were required but not present at packages.foo.transformations[0].create-symlink: 'target'" - " (Documentation: " + "The following keys were required but not present at packages.foo.transformations[0].create-symlink" + " [Line 5 column 12]: 'target' (Documentation: " "{{DEBPUTY_DOC_ROOT_DIR}}/MANIFEST-FORMAT.md#create-symlinks-transformation-rule-create-symlink)" ) assert normalize_doc_link(e_info.value.args[0]) == msg @@ -310,7 +313,7 @@ def test_create_symlinks_not_normalized_path(manifest_parser_pkg_foo): manifest_parser_pkg_foo.parse_manifest(fd=content) expected = ( - 'The path "../bar" provided in packages.foo.transformations[0].create-symlink.path <Search for: ../bar>' + 'The path "../bar" provided in packages.foo.transformations[0].create-symlink.path [Line 6 column 20]' ' should be relative to the root of the package and not use any ".." or "." segments.' ) assert e_info.value.args[0] == expected @@ -332,7 +335,7 @@ def test_unresolvable_subst_in_source_context(manifest_parser_pkg_foo): expected = ( "The variable {{PACKAGE}} is not available while processing installations[0].install.as" - " <Search for: foo.sh>." + " [Line 5 column 7]." ) assert e_info.value.args[0] == expected @@ -397,7 +400,7 @@ def test_yaml_octal_mode_int(manifest_parser_pkg_foo): manifest_parser_pkg_foo.parse_manifest(fd=content) msg = ( - 'The attribute "packages.foo.transformations[0].path-metadata.mode <Search for: usr/share/bar>" did not' + 'The attribute "packages.foo.transformations[0].path-metadata.mode [Line 7 column 20]" did not' " have a valid structure/type: The attribute must be a FileSystemMode (string)" ) @@ -486,3 +489,86 @@ def test_yaml_clean_after_removal_unsafe_path( else: with pytest.raises(ManifestParseException) as e_info: manifest_parser_pkg_foo.parse_manifest(fd=content) + + +def test_yaml_build_environment_default( + manifest_parser_pkg_foo: YAMLManifestParser, +) -> None: + content = textwrap.dedent( + ( + """\ + + manifest-version: '0.1' + default-build-environment: + set: + FOO: "bar" + builds: + # FIXME: we should not require an empty dict here + - autoconf: {} + """ + ) + ) + manifest = manifest_parser_pkg_foo.parse_manifest(fd=content) + envs = manifest.build_environments + assert not envs.environments + base_env = {} + envs.default_environment.update_env(base_env) + assert "FOO" in base_env + build_rule = manifest.build_rules[0] + assert build_rule.environment is envs.default_environment + + +def test_yaml_build_environments_no_default( + manifest_parser_pkg_foo: YAMLManifestParser, +) -> None: + content = textwrap.dedent( + ( + f"""\ + + manifest-version: '0.1' + build-environments: + - name: custom-env + set: + FOO: "bar" + builds: + - autoconf: + environment: custom-env + """ + ) + ) + manifest = manifest_parser_pkg_foo.parse_manifest(fd=content) + envs = manifest.build_environments + assert "custom-env" in envs.environments + custom_env = envs.environments["custom-env"] + assert envs.default_environment is None + base_env = {} + custom_env.update_env(base_env) + assert "FOO" in base_env + build_rule = manifest.build_rules[0] + assert build_rule.environment is custom_env + + +def test_yaml_build_environments_no_default_error( + manifest_parser_pkg_foo: YAMLManifestParser, +) -> None: + content = textwrap.dedent( + ( + """\ + + manifest-version: '0.1' + build-environments: + - name: custom-env + set: + FOO: "bar" + builds: + # FIXME: we should not require an empty dict here + - autoconf: {} + """ + ) + ) + with pytest.raises(ManifestParseException) as e_info: + manifest_parser_pkg_foo.parse_manifest(fd=content) + + expected_msg = "The following named environments were never referenced: custom-env" + msg = e_info.value.args[0] + assert msg == expected_msg diff --git a/tests/test_style.py b/tests/test_style.py index ef6ddc4..8f5b6ca 100644 --- a/tests/test_style.py +++ b/tests/test_style.py @@ -4,51 +4,57 @@ import pytest from debian.deb822 import Deb822 from debputy.yaml.compat import CommentedMap -from debputy.lsp.style_prefs import ( - StylePreferenceTable, - determine_effective_style, - EffectivePreference, +from debputy.lsp.maint_prefs import ( + MaintainerPreferenceTable, + determine_effective_preference, + EffectiveFormattingPreference, _WAS_DEFAULTS, ) from debputy.packages import SourcePackage def test_load_styles() -> None: - styles = StylePreferenceTable.load_styles() + styles = MaintainerPreferenceTable.load_preferences() assert "niels@thykier.net" in styles.maintainer_preferences - nt_style = styles.maintainer_preferences["niels@thykier.net"] + nt_maint_pref = styles.maintainer_preferences["niels@thykier.net"] # Note this is data dependent; if it fails because the style changes, update the test - assert nt_style.canonical_name == "Niels Thykier" - assert not nt_style.is_packaging_team - assert nt_style.formatting_deb822_normalize_field_content - assert nt_style.formatting_deb822_short_indent - assert nt_style.formatting_deb822_always_wrap - assert nt_style.formatting_deb822_trailing_separator - assert nt_style.formatting_deb822_max_line_length == 79 - assert not nt_style.formatting_deb822_normalize_stanza_order + assert nt_maint_pref.canonical_name == "Niels Thykier" + assert not nt_maint_pref.is_packaging_team + black_style = styles.named_styles["black"] + nt_style = nt_maint_pref.formatting + assert nt_style is not None + assert black_style == black_style - # TODO: Not implemented yet - assert not nt_style.formatting_deb822_normalize_field_order + +def test_load_no_styles() -> None: + styles = MaintainerPreferenceTable.load_preferences() + assert "packages@qa.debian.org" in styles.maintainer_preferences + qa_maint_pref = styles.maintainer_preferences["packages@qa.debian.org"] + assert qa_maint_pref.canonical_name == "Debian QA Group" + assert qa_maint_pref.is_packaging_team + # Orphaned packages do not have a predefined style, since Debian (nor Debian QA) have + # one well-defined style. + assert qa_maint_pref.formatting is None def test_load_named_styles() -> None: - styles = StylePreferenceTable.load_styles() + styles = MaintainerPreferenceTable.load_preferences() assert "black" in styles.named_styles black_style = styles.named_styles["black"] # Note this is data dependent; if it fails because the style changes, update the test - assert black_style.formatting_deb822_normalize_field_content - assert black_style.formatting_deb822_short_indent - assert black_style.formatting_deb822_always_wrap - assert black_style.formatting_deb822_trailing_separator - assert black_style.formatting_deb822_max_line_length == 79 - assert not black_style.formatting_deb822_normalize_stanza_order + assert black_style.deb822_normalize_field_content + assert black_style.deb822_short_indent + assert black_style.deb822_always_wrap + assert black_style.deb822_trailing_separator + assert black_style.deb822_max_line_length == 79 + assert not black_style.deb822_normalize_stanza_order # TODO: Not implemented yet - assert not black_style.formatting_deb822_normalize_field_order + assert not black_style.deb822_normalize_field_order def test_compat_styles() -> None: - styles = StylePreferenceTable.load_styles() + styles = MaintainerPreferenceTable.load_preferences() # Data dependent; if it breaks, provide a stubbed style preference table assert "niels@thykier.net" in styles.maintainer_preferences @@ -56,11 +62,11 @@ def test_compat_styles() -> None: assert "random-package@packages.debian.org" not in styles.maintainer_preferences assert "random@example.org" not in styles.maintainer_preferences - nt_pref = styles.maintainer_preferences["niels@thykier.net"].as_effective_pref() - zeha_pref = styles.maintainer_preferences["zeha@debian.org"].as_effective_pref() + nt_style = styles.maintainer_preferences["niels@thykier.net"].formatting + zeha_style = styles.maintainer_preferences["zeha@debian.org"].formatting # Data dependency - assert nt_pref == zeha_pref + assert nt_style == zeha_style fields = Deb822( { @@ -71,30 +77,33 @@ def test_compat_styles() -> None: ) src = SourcePackage(fields) - effective_style, _ = determine_effective_style(styles, src, None) - assert effective_style == nt_pref + effective_style, tool, _ = determine_effective_preference(styles, src, None) + assert effective_style == nt_style + assert tool == "debputy reformat" fields["Uploaders"] = ( "Niels Thykier <niels@thykier.net>, Chris Hofstaedtler <zeha@debian.org>" ) src = SourcePackage(fields) - effective_style, _ = determine_effective_style(styles, src, None) - assert effective_style == nt_pref - assert effective_style == zeha_pref + effective_style, tool, _ = determine_effective_preference(styles, src, None) + assert effective_style == nt_style + assert effective_style == zeha_style + assert tool == "debputy reformat" fields["Uploaders"] = ( "Niels Thykier <niels@thykier.net>, Chris Hofstaedtler <zeha@debian.org>, Random Developer <random@example.org>" ) src = SourcePackage(fields) - effective_style, _ = determine_effective_style(styles, src, None) + effective_style, tool, _ = determine_effective_preference(styles, src, None) assert effective_style is None + assert tool is None @pytest.mark.xfail def test_compat_styles_team_maint() -> None: - styles = StylePreferenceTable.load_styles() + styles = MaintainerPreferenceTable.load_preferences() fields = Deb822( { "Package": "foo", @@ -108,12 +117,13 @@ def test_compat_styles_team_maint() -> None: assert "random@example.org" not in styles.maintainer_preferences team_style = styles.maintainer_preferences["team@lists.debian.org"] assert team_style.is_packaging_team - effective_style, _ = determine_effective_style(styles, src, None) - assert effective_style == team_style.as_effective_pref() + effective_style, tool, _ = determine_effective_preference(styles, src, None) + assert effective_style == team_style.formatting + assert tool is None def test_x_style() -> None: - styles = StylePreferenceTable.load_styles() + styles = MaintainerPreferenceTable.load_preferences() fields = Deb822( { "Package": "foo", @@ -125,12 +135,13 @@ def test_x_style() -> None: assert "random@example.org" not in styles.maintainer_preferences assert "black" in styles.named_styles black_style = styles.named_styles["black"] - effective_style, _ = determine_effective_style(styles, src, None) + effective_style, tool, _ = determine_effective_preference(styles, src, None) assert effective_style == black_style + assert tool == "debputy reformat" def test_was_from_salsa_ci_style() -> None: - styles = StylePreferenceTable.load_styles() + styles = MaintainerPreferenceTable.load_preferences() fields = Deb822( { "Package": "foo", @@ -139,20 +150,23 @@ def test_was_from_salsa_ci_style() -> None: ) src = SourcePackage(fields) assert "random@example.org" not in styles.maintainer_preferences - effective_style, _ = determine_effective_style(styles, src, None) + effective_style, tool, _ = determine_effective_preference(styles, src, None) assert effective_style is None + assert tool is None salsa_ci = CommentedMap( {"variables": CommentedMap({"SALSA_CI_DISABLE_WRAP_AND_SORT": "yes"})} ) - effective_style, _ = determine_effective_style(styles, src, salsa_ci) + effective_style, tool, _ = determine_effective_preference(styles, src, salsa_ci) assert effective_style is None + assert tool is None salsa_ci = CommentedMap( {"variables": CommentedMap({"SALSA_CI_DISABLE_WRAP_AND_SORT": "no"})} ) - effective_style, _ = determine_effective_style(styles, src, salsa_ci) - was_style = EffectivePreference(**_WAS_DEFAULTS) + effective_style, tool, _ = determine_effective_preference(styles, src, salsa_ci) + was_style = EffectiveFormattingPreference(**_WAS_DEFAULTS) assert effective_style == was_style + assert tool == "wrap-and-sort" @pytest.mark.parametrize( @@ -161,45 +175,46 @@ def test_was_from_salsa_ci_style() -> None: ( "-a", { - "formatting_deb822_always_wrap": True, + "deb822_always_wrap": True, }, ), ( "-sa", { - "formatting_deb822_always_wrap": True, - "formatting_deb822_short_indent": True, + "deb822_always_wrap": True, + "deb822_short_indent": True, }, ), ( "-sa --keep-first", { - "formatting_deb822_always_wrap": True, - "formatting_deb822_short_indent": True, + "deb822_always_wrap": True, + "deb822_short_indent": True, }, ), ( "-sab --keep-first", { - "formatting_deb822_always_wrap": True, - "formatting_deb822_short_indent": True, - "formatting_deb822_normalize_stanza_order": True, + "deb822_always_wrap": True, + "deb822_short_indent": True, + "deb822_normalize_stanza_order": True, }, ), ( "-sab --no-keep-first", { - "formatting_deb822_always_wrap": True, - "formatting_deb822_short_indent": True, - "formatting_deb822_normalize_stanza_order": False, + "deb822_always_wrap": True, + "deb822_short_indent": True, + "deb822_normalize_stanza_order": False, }, ), ], ) def test_was_from_salsa_ci_style_args( - was_args: str, style_delta: Optional[Mapping[str, Any]] + was_args: str, + style_delta: Optional[Mapping[str, Any]], ) -> None: - styles = StylePreferenceTable.load_styles() + styles = MaintainerPreferenceTable.load_preferences() fields = Deb822( { "Package": "foo", @@ -218,12 +233,14 @@ def test_was_from_salsa_ci_style_args( ) } ) - effective_style, _ = determine_effective_style(styles, src, salsa_ci) + effective_style, tool, _ = determine_effective_preference(styles, src, salsa_ci) if style_delta is None: assert effective_style is None + assert tool is None else: - was_style = EffectivePreference(**_WAS_DEFAULTS).replace( + was_style = EffectiveFormattingPreference(**_WAS_DEFAULTS).replace( **style_delta, ) assert effective_style == was_style + assert tool == f"wrap-and-sort {was_args}".strip() diff --git a/tests/test_substitute.py b/tests/test_substitute.py index a83cc7f..81eb2e0 100644 --- a/tests/test_substitute.py +++ b/tests/test_substitute.py @@ -1,6 +1,7 @@ import pytest from debputy.architecture_support import faked_arch_table +from debputy.commands.debputy_cmd.output import no_fancy_output from debputy.dh_migration.models import ( DHMigrationSubstitution, AcceptableMigrationIssues, @@ -55,7 +56,7 @@ def test_substitution_match(debputy_plugin_feature_set, value, expected) -> None def test_migrate_substitution() -> None: - feature_migration = FeatureMigration("test migration") + feature_migration = FeatureMigration("test migration", no_fancy_output()) subst = DHMigrationSubstitution( MOCK_DPKG_ARCH_TABLE, AcceptableMigrationIssues(frozenset()), diff --git a/tests/test_utils.py b/tests/test_utils.py new file mode 100644 index 0000000..e1dc87e --- /dev/null +++ b/tests/test_utils.py @@ -0,0 +1,20 @@ +from typing import Sequence, Union + +import pytest + +from debputy.util import escape_shell + + +@pytest.mark.parametrize( + "arg,expected", + [ + ("foo bar", '"foo bar"'), + ("a'b", r"""a\'b"""), + ("foo=bar and baz", 'foo="bar and baz"'), + ("--foo=bar and baz", '--foo="bar and baz"'), + ("--foo with spaces=bar and baz", '"--foo with spaces=bar and baz"'), + ], +) +def test_symlink_normalization(arg: Union[str, Sequence[str]], expected: str) -> None: + actual = escape_shell(arg) if isinstance(arg, str) else escape_shell(*arg) + assert actual == expected |