diff options
Diffstat (limited to 'taskcluster/docs/partner-repacks.rst')
| -rw-r--r-- | taskcluster/docs/partner-repacks.rst | 256 |
1 files changed, 256 insertions, 0 deletions
diff --git a/taskcluster/docs/partner-repacks.rst b/taskcluster/docs/partner-repacks.rst new file mode 100644 index 0000000000..236b7babd4 --- /dev/null +++ b/taskcluster/docs/partner-repacks.rst @@ -0,0 +1,256 @@ +Partner repacks +=============== +.. _partner repacks: + +We create slightly-modified Firefox releases for some extra audiences + +* EME-free builds, which disable DRM plugins by default +* Funnelcake builds, which are used for Mozilla experiments +* partner builds, which customize Firefox for external partners + +We use the phrase "partner repacks" to refer to all these builds because they +use the same process of repacking regular Firefox releases with additional files. +The specific differences depend on the type of build. + +We produce partner repacks for some beta builds, and for release builds, as part of the release +automation. We don't produce any files to update these builds as they are handled automatically +(see updates_). + +We also produce :ref:`partner attribution` builds, which are Firefox Windows installers with a cohort identifier +added. + +Parameters & Scheduling +----------------------- + +Partner repacks have a number of parameters which control how they work: + +* ``release_enable_emefree`` +* ``release_enable_partner_repack`` +* ``release_partner_config`` +* ``release_partner_build_number`` +* ``release_partners`` + +We split the repacks into two 'paths', EME-free and everything else, to retain some +flexibility over enabling/disabling them separately. This costs us some duplication of the kinds +in the repacking stack. The two enable parameters are booleans to turn these two paths +on/off. We set them in shipit's `is_partner_enabled() <https://github.com/mozilla-releng/shipit/blob/main/api/src/shipit_api/admin/release.py#L93>`_ when starting a +release. They're both true for Firefox betas >= b8 and releases, but otherwise disabled. + +``release_partner_config`` is a dictionary of configuration data which drives the task generation +logic. It's usually looked up during the release promotion action task, using the Github +GraphQL API in the `get_partner_config_by_url() +<python/taskgraph.util.html#taskgraph.util.partners.get_partner_config_by_url>`_ function, with the +url defined in `taskcluster/ci/config.yml <https://searchfox +.org/mozilla-release/search?q=regexp%3A^partner+path%3Aconfig.yml&redirect=true>`_. + +``release_partner_build_number`` is an integer used to create unique upload paths in the firefox +candidates directory, while ``release_partners`` is a list of partners that should be +repacked (i.e. a subset of the whole config). Both are intended for use when respinning a few partners after +the regular Firefox has shipped. More information on that can be found in the +`RelEng Docs <https://moz-releng-docs.readthedocs.io/en/latest/procedures/misc-operations/off-cycle-partner-repacks-and-funnelcake.html>`_. + +Most of the machine time for generating partner repacks takes place in the `promote` phase of the +automation, or `promote_rc` in the case of X.0 release candidates. The EME-free builds are copied into the +Firefox releases directory in the `push` phase, along with the regular bits. + + +Configuration +------------- + +We need some configuration to know *what* to repack, and *how* to do that. The *what* is defined by +default.xml manifests, as used with the `repo <https://gerrit.googlesource.com/git-repo>`_ tool +for git. The `default.xml for EME-free <https://github +.com/mozilla-partners/mozilla-EME-free-manifest/blob/master/default.xml>`_ illustrates this:: + + <?xml version="1.0" ?> + <manifest> + <remote fetch="git@github.com:mozilla-partners/" name="mozilla-partners"/> + <remote fetch="git@github.com:mozilla/" name="mozilla"/> + + <project name="repack-scripts" path="scripts" remote="mozilla-partners" revision="master"/> + <project name="build-tools" path="scripts/tools" remote="mozilla" revision="master"/> + <project name="mozilla-EME-free" path="partners/mozilla-EME-free" remote="mozilla-partners" revision="master"/> + </manifest> + +The repack-scripts and build-tools repos are found in all manifests, and then there is a list of +partner repositories which contain the *how* configuration. Some of these repos are not publicly +visible. + +A partner repository may contain multiple configurations inside the ``desktop`` directory. Each +subdirectory must contain a ``repack.cfg`` and a ``distribution`` directory, the latter +containing the customizations needed. Here's `EME-free's repack.cfg <https://github.com/mozilla-partners/mozilla-EME-free/blob/master/desktop/mozilla-EME-free/repack.cfg>`_:: + + aus="mozilla-EMEfree" + dist_id="mozilla-EMEfree" + dist_version="1.0" + linux-i686=false + linux-x86_64=false + locales="ach af an ar" # truncated for display here + mac=true + win32=true + win64=true + output_dir="%(platform)s-EME-free/%(locale)s" + + # Upload params + upload_to_candidates=true + +Note the list of locales and boolean toggles for enabling platforms. The ``output_dir`` and +``upload_to_candidates`` parameters are only present for repacks which are uploaded into the +`candidates directory <https://archive.mozilla.org/pub/firefox/candidates/>`_. + +All customizations will be placed in the ``distribution`` directory at the root of the Firefox +install directory, or in the case of OS X in ``Firefox.app/Contents/Resources/distribution/``. A +``distribution.ini`` file is the minimal requirement, here's an example from `EME-free +<https://github.com/mozilla-partners/mozilla-EME-free/blob/master/desktop/mozilla-EME-free/distribution +/distribution.ini>`_:: + + # Partner Distribution Configuration File + # Author: Mozilla + # Date: 2015-03-27 + + [Global] + id=mozilla-EMEfree + version=1.0 + about=Mozilla Firefox EME-free + + [Preferences] + media.eme.enabled=false + app.partner.mozilla-EMEfree="mozilla-EMEfree" + +Extensions and other customizations might also be included in repacks. + + +Repacking process +----------------- + +The stack of tasks to create partner repacks is broadly similar to localised nightlies and +regular releases. The basic form is + +* partner repack - insert the customisations into the the regular builds +* signing - sign the internals which will become the installer (Mac only) +* repackage - create the "installer" (Mac and Windows) +* chunking dummy - a linux only bridge to ... +* repackage signing - sign the "installers" (mainly Windows) +* beetmover - move the files to a partner-specific destination +* beetmover checksums - possibly beetmove the checksums from previous step + +Some key divergences are: + +* all intermediate artifacts are uploaded with a ``releng/partner`` prefix +* we don't insert any binaries on Windows so no need for internal signing +* there's no need to create any complete mar files at the repackage step +* we support both public and private destinations in beetmover +* we only need beetmover checksums for EME-free builds + + +Partner repack +^^^^^^^^^^^^^^ + +* kinds: ``release-partner-repack`` ``release-eme-free-repack`` +* platforms: Typically all (but depends on what's enabled by partner configuration) +* upstreams: ``build-signing`` ``l10n-signing`` + +There is one task per platform in this step, calling out to `scripts/desktop_partner_repacks.py +<https://hg.mozilla.org/mozilla-central/file/default/testing/mozharness/scripts +/desktop_partner_repacks.py>`_ in mozharness to prepare an environment and then perform the repacks. +The actual repacking is done by `python/mozrelease/mozrelease/partner_repack.py +<https://hg.mozilla.org/mozilla-central/file/default/python/mozrelease/mozrelease/partner_repack.py>`_. + +It takes as input the build-signing and l10n-signing artifacts, which are all zip/tar.gz/tar.bz2 +archives, simplifying the repack process by avoiding dmg and exe. Windows produces ``target.zip`` +& ``setup.exe``, Mac is ``target.tar.gz``, Linux is the final product ``target.tar.bz2`` +(beetmover handles pretty naming as usual). + +Signing +^^^^^^^ + +* kinds: ``release-partner-repack-notarization-part-1`` ``release-partner-repack-notarization-poller`` ``release-partner-repack-signing`` +* platforms: Mac +* upstreams: ``release-partner-repack`` ``release-eme-free-repack`` + +We chunk the single partner repack task out to a signing task with 5 artifacts each. For +example, EME-free will become 19 tasks. We collect the target.tar.gz from the +upstream, and return a signed target.tar.gz. We use a ``target.dmg`` artifact for +nightlies/regular releases, but this is converted to ``target.tar.gz`` by the signing +scriptworker before sending it to the signing server, so partners are equivalent. The ``part-1`` task +uploads the binaries to apple, while the ``poller`` task waits for their approval, then +``release-partner-repack-signing`` staples on the notarization ticket. + +Repackage +^^^^^^^^^ + +* kinds: ``release-partner-repack-repackage`` ``release-eme-free-repack-repackage`` +* platforms: Mac & Windows +* upstreams: + + * Mac: ``release-partner-signing`` ``release-eme-free-signing`` + * Windows: ``release-partner-repack`` ``release-eme-free-repack`` + +Mac has a repackage job for each of the signing tasks. Windows repackages are chunked here to +the same granularity as mac. Takes ``target.zip`` & ``setup.exe`` to produce ``target.exe`` on +Windows, and ``target.tar.gz`` to produce ``target.dmg`` on Mac. There's no need to produce any +complete.mar files here like regular release bits do because we can reuse those. + +Chunking dummy +^^^^^^^^^^^^^^ + +* kinds: ``release-partner-repack-chunking-dummy`` +* platforms: Linux +* upstreams: ``release-partner-repack`` + +We're need Linux chunked at the next step so this dummy takes care of that for the relatively simple path +Linux follows. One task per sub config+locale combination, the same as Windows and Mac. This doesn't need to +exist for EME-free because we don't need to create Linux builds there. + +Repackage Signing +^^^^^^^^^^^^^^^^^ + +* kinds: ``release-partner-repack-repackage-signing`` ``release-eme-free-repack-repackage-signing`` +* platforms: All +* upstreams: + + * Mac & Windows: ``release-partner-repackage`` ``release-eme-free-repackage`` + * Linux: ``release-partner-repack-chunking-dummy`` + +This step GPG signs all platforms, and authenticode signs the Windows installer. + +Beetmover +^^^^^^^^^ + +* kinds: ``release-partner-repack-beetmover`` ``release-eme-free-repack-beetmover`` +* platforms: All +* upstreams: ``release-partner-repack-repackage-signing`` ``release-eme-free-repack-repackage-signing`` + +Moves and renames the artifacts to their public location in the `candidates directory +<https://archive.mozilla.org/pub/firefox/candidates/>`_, or a private S3 bucket. Each task will +have the ``project:releng:beetmover:action:push-to-partner`` scope, with public uploads having +``project:releng:beetmover:bucket:release`` and private uploads using +``project:releng:beetmover:bucket:partner``. The ``upload_to_candidates`` key in the partner config +controls the second scope. There's a separate partner code path in `beetmoverscript <https://github.com/mozilla-releng/scriptworker-scripts/tree/master/beetmoverscript>`_. + +Beetmover checksums +^^^^^^^^^^^^^^^^^^^ + +* kinds: ``release-eme-free-repack-beetmover-checksums`` +* platforms: Mac & Windows +* upstreams: ``release-eme-free-repack-repackage-beetmover`` + +The EME-free builds should be present in our SHA256SUMS file and friends (`e.g. <https://archive +.mozilla.org/pub/firefox/releases/61.0/SHA256SUMS>`_) so we beetmove the target.checksums from +the beetmover tasks into the candidates directory. They get picked up by the +``release-generate-checksums`` kind. + +.. _updates: + +Updates +------- + +It's very rare to need to update a partner repack differently from the original +release build but we retain that capability. A partner build with distribution name ``foo``, +based on a release Firefox build, will query for an update on the ``release-cck-foo`` channel. If +the update server `Balrog <http://mozilla-balrog.readthedocs.io/en/latest/>`_ finds no rule for +that channel it will fallback to the ``release`` channel. The update files for the regular releases do not +modify the ``distribution/`` directory, so the customizations are not modified. + +`Bug 1430254 <https://bugzilla.mozilla.org/show_bug.cgi?id=1430254>`_ is an example of an exception to this +logic. |