diff options
Diffstat (limited to 'taskcluster/docs/partials.rst')
| -rw-r--r-- | taskcluster/docs/partials.rst | 123 |
1 files changed, 123 insertions, 0 deletions
diff --git a/taskcluster/docs/partials.rst b/taskcluster/docs/partials.rst new file mode 100644 index 0000000000..c0aeca1c3c --- /dev/null +++ b/taskcluster/docs/partials.rst @@ -0,0 +1,123 @@ +Partial Update Generation +========================= + +Overview +-------- + +Windows, Mac and Linux releases have partial updates, to reduce +the file size end-users have to download in order to receive new +versions. These are created using a docker image, some Python, +``mbsdiff``, and the tools in ``tools/update-packaging`` + +The task has been called 'Funsize' for quite some time. This might +make sense depending on what brands of chocolate bar are available +near you. + +How the Task Works +------------------ + +Funsize uses a docker image that's built in-tree, named funsize-update-generator. +The image contains some Python to examine the task definition and determine +what needs to be done, but it downloads tools like ``mar`` and ``mbsdiff`` +from either locations specified in the task definition, or default mozilla-central +locations. + +The 'extra' section of the task definition contains most of the payload, under +the 'funsize' key. In here is a list of partials that this specific task will +generate, and each entry includes the earlier (or 'from') version, and the most +recent (or 'to') version, which for most releases will likely be a taskcluster +artifact. + +.. code-block:: json + + { + "to_mar": "https://tc.net/api/queue/v1/task/EWtBFqVuT-WqG3tGLxWhmA/artifacts/public/build/ach/target.complete.mar", + "product": "Firefox", + "dest_mar": "target-60.0b8.partial.mar", + "locale": "ach", + "from_mar": "http://archive.mozilla.org/pub/firefox/candidates/60.0b8-candidates/build1/update/linux-i686/ach/firefox-60.0b8.complete.mar", + "update_number": 2, + "platform": "linux32", + "previousVersion": "60.0b8", + "previousBuildNumber": "1", + "branch": "mozilla-beta" + } + +The 'update number' indicates how many released versions there are between 'to' and the current 'from'. +For example, if we are building a partial update for the current nightly from the previous one, the update +number will be 1. For the release before that, it will be 2. This lets us use generic output artifact +names that we can rename in the later ``beetmover`` tasks. + +Inside the task, for each partial it has been told to generate, it will download, unpack and virus +scan the 'from_mar' and 'to_mar', download the tools, and run ``make_incremental_update.sh`` from +``tools/update-packaging``. + +If a scope is given for a set of temporary S3 credentials, the task will use a caching script, +to allow reuse of the diffs made for larger files. Some of the larger files are not localised, +and this allows us to save a lot of compute time. + +For Releases +------------ + +Partials are made as part of the ``promote`` task group. The previous +versions used to create the update are specified in ship-it by +Release Management. + +Nightly Partials +---------------- + +Since nightly releases don't appear in ship-it, the partials to create +are determined in the decision task. This was controversial, and so here +are the assumptions and reasons, so that when an alternative solution is +discovered, we can assess it in context: + +1. Balrog is the source of truth for previous nightly releases. +2. Re-running a task should produce the same results. +3. A task's input and output should be specified in the definition. +4. A task transform should avoid external dependencies. This is to + increase the number of scenarios in which 'mach taskgraph' works. +5. A task graph doesn't explicitly know that it's intended for nightlies, + only that specific tasks are only present for nightly. +6. The decision task is explicitly told that its target is nightly + using the target-tasks-method argument. + +a. From 2 and 3, this means that the partials task itself cannot query + balrog for the history, as it may get different results when re-run, + and hides the inputs and outputs from the task definition. +b. From 4, anything run by 'mach taskgraph' is an inappropriate place + to query Balrog, even if it results in a repeatable task graph. +c. Since these restrictions don't apply to the decision task, and given + 6, we can query Balrog in the decision task if the target-tasks-method + given contains 'nightly', such as 'nightly_desktop' or 'nightly_linux' + +Using the decision task involves making fewer, larger queries to Balrog, +and storing the results for task graph regeneration and later audit. At +the moment this data is stored in the ``parameters`` under the label +``release_history``, since the parameters are an existing method for +passing data to the task transforms, but a case could be made +for adding a separate store, as it's a significantly larger number of +records than anything else in the parameters. + +Nightly Partials and Beetmover +------------------------------ + +A release for a specific platform and locale may not have a history of +prior releases that can be used to build partial updates. This could be +for a variety of reasons, such as a new locale, or a hiatus in nightly +releases creating too long a gap in the history. + +This means that the ``partials`` and ``partials-signing`` tasks may have +nothing to do for a platform and locale. If this is true, then the tasks +are filtered out in the ``transform``. + +This does mean that the downstream task, ``beetmover-repackage`` can not +rely on the ``partials-signing`` task existing. It depends on both the +``partials-signing`` and ``repackage-signing`` task, and chooses which +to depend on in the transform. + +If there is a history in the ``parameters`` ``release_history`` section +then ``beetmover-repackage`` will depend on ``partials-signing``. +Otherwise, it will depend on ``repackage-signing``. + +This is not ideal, as it results in unclear logic in the task graph +generation. It will be improved. |