Adding upstream version 1:115.7.0.upstream/1%115.7.0upstream

Signed-off-by: Daniel Baumann <daniel.baumann@progress-linux.org>

4 files changed, 555 insertions, 0 deletions

diff --git a/toolkit/mozapps/update/docs/BackgroundUpdates.rst b/toolkit/mozapps/update/docs/BackgroundUpdates.rst
new file mode 100644
index 0000000000..c692f4958a
--- /dev/null
+++ b/toolkit/mozapps/update/docs/BackgroundUpdates.rst
@@ -0,0 +1,224 @@
+==================
+Background Updates
+==================
+
+The purpose of the background update system is to perform application updates
+during times when Firefox is not running. It was originally implemented in `bug
+1689520 <https://bugzilla.mozilla.org/show_bug.cgi?id=1689520>`__.
+
+The system has three main tasks it needs to handle:
+
+1. :ref:`Determining whether background updates are possible <background-updates-determining>`
+
+2. :ref:`Scheduling background tasks <background-updates-scheduling>`
+
+3. :ref:`Checking for updates <background-updates-checking>`
+
+Architecturally, the background task is an instance of Firefox running in a
+special background mode, not a separate tool. This allows it to leverage
+existing functionality in Firefox, including the existing update code, but also
+keep acceptable performance characteristics for a background task by controlling
+and limiting the parts of Firefox that are loaded.
+
+Everything in this document applies only to Microsoft Windows systems. In the
+future, we would like to extend background update support to macOS (see `bug
+1653435 <https://bugzilla.mozilla.org/show_bug.cgi?id=1653435>`__), however
+support for Linux and other Unix variants is not planned due to the variation in
+OS-level scheduling affordances across distributions/configurations.
+
+Lifecycle
+=========
+
+When background updates are possible, the background update task will be invoked
+every 7 hours (by default).  The first invocation initiates an update download
+which proceeds after the task exits using Windows BITS.  The second invocation
+prepares and stages the update.  The third invocation installs the update as it
+starts up, and then checks for a newer update, possibly initiating another
+update download.  The cycle then continues.  If the user launches Firefox at any
+point in this process, it will take over.  If the background update task is
+invoked while Firefox proper is running, the task exits without doing any work.
+In the future, the second invocation will stage and then restart to finish
+installing the update, rather than waiting for the third invocation (see `bug
+1704855 <https://bugzilla.mozilla.org/show_bug.cgi?id=1704855>`__).
+
+.. _background-updates-determining:
+
+Determining whether background updates are possible
+===================================================
+
+Configuration
+-------------
+
+Updating Firefox, by definition, is an operation that applies to a Firefox
+installation. However, Firefox configuration is generally done via preference
+values and other files which are stored in a Firefox profile, and in general
+profiles do not correspond 1:1 with installations. This raises the question of
+how the configuration for something like the background updater should be
+managed. We deal with this question in two different ways.
+
+There are two main preferences specifically relevant to updates. Those
+are ``app.update.auto``, which controls whether updates should be
+downloaded automatically at all, even if Firefox is running, and
+``app.update.background.enabled``, to specifically control whether to
+use the background update system. We store these preferences in the
+update root directory, which is located in a per-installation location
+outside of any profile. Any profile loaded in that installation can
+observe and control these settings.
+
+But there are some other pieces of state which absolutely must come from a
+profile, such as the telemetry client ID and logging level settings (see
+`BackgroundTasksUtils.jsm <https://searchfox.org/mozilla-central/source/toolkit/components/backgroundtasks/BackgroundTasksUtils.jsm>`__).
+
+This means that, in addition to our per-installation prefs, we also need
+to be able to identify and load a profile. To do that, we leverage `the profile
+service <https://searchfox.org/mozilla-central/source/toolkit/profile/nsIToolkitProfileService.idl>`__
+to determine what the default profile for the installation would be if we were
+running a normal browser session, and the background updater always uses it.
+
+Criteria
+--------
+
+The default profile must satisfy several conditions in order for background
+updates to be scheduled. None of these confounding factors are present in fully
+default configurations, but some are relatively common. See
+`BackgroundUpdate.REASON <https://searchfox.org/mozilla-central/search?q=symbol:BackgroundUpdate%23REASON>`__
+for all the details.
+
+In order for the background task to be scheduled:
+
+-  The per-installation ``app.update.background.enabled`` pref must be
+   true
+
+-  The per-installation ``app.update.auto`` pref must be true (the
+   default)
+
+-  The installation must have been created by an installer executable and not by
+   manually extracting an archive file
+
+-  The current OS user must be capable of updating the installation based on its
+   file system permissions, either by having permission to write to application
+   files directly or by using the Mozilla Maintenance Service (which also
+   requires that it be installed and enabled, as it is by default)
+
+-  BITS must be enabled via ``app.update.BITS.enabled`` (the default)
+
+-  Firefox proxy server settings must not be configured (the default)
+
+-  ``app.update.langpack.enabled`` must be false, or otherwise there must be no
+   langpacks installed. Background tasks cannot update addons such as langpacks,
+   because they are installed into a profile, and langpacks that are not
+   precisely matched with the version of Firefox that is installed can cause
+   YSOD failures (see `bug 1647443 <https://bugzilla.mozilla.org/show_bug.cgi?id=1647443>`__),
+   so background updating in the presence of langpacks is too risky.
+
+If any per-installation prefs are changed while the default profile is not
+running, the background update task will witness the changed prefs during its
+next scheduled run, and exit if appropriate. The background task will not be
+unscheduled at that point; that is delayed until a browser session is run with
+the default profile (it should be possible for the background update task to
+unschedule itself, but currently we prefer the simplicity of handling all
+scheduling tasks from a single location).
+
+In the extremely unusual case when prefs belonging to the default profile are
+modified outside of Firefox (with a text editor, say), then the
+background task will generally pick up those changes with no action needed,
+because it will fish the changed settings directly from the profile.
+
+.. _background-updates-scheduling:
+
+Scheduling background tasks
+===========================
+
+We use OS-level scheduling mechanisms to schedule the command ``firefox
+--backgroundtask backgroundupdate`` to run on a particular cadence. This cadence
+is controlled by the ``app.update.background.interval`` preference, which
+defaults to 7 hours.
+
+On Windows, we use the `Task Scheduler
+API <https://docs.microsoft.com/en-us/windows/win32/taskschd/task-scheduler-start-page>`__;
+on macOS this will use
+`launchd <https://developer.apple.com/library/archive/documentation/MacOSX/Conceptual/BPSystemStartup/Chapters/CreatingLaunchdJobs.html>`__.
+For platform-specific scheduling details, see the
+`TaskScheduler.jsm <https://searchfox.org/mozilla-central/source/toolkit/components/taskscheduler/TaskScheduler.jsm>`__
+module.
+
+These background tasks are scheduled per OS user and run with that user’s
+permissions. No additional privileges are requested or needed, regardless of the
+user account's status, because we have already verified that either the user has
+all the permissions they need or that the Maintenance Service can be used.
+
+Scheduling is done from within Firefox (or a background task) itself. To
+reduce shared state, only the *default* Firefox profile will interact
+with the OS-level task scheduling mechanism.
+
+.. _background-updates-checking:
+
+Checking for updates
+====================
+
+After verifying all the preconditions and exiting immediately if any do not
+hold, the ``backgroundupdate`` task then verifies that it is the only Firefox
+instance running (as determined by a multi-instance lock, see `bug
+1553982 <https://bugzilla.mozilla.org/show_bug.cgi?id=1553982>`__), since
+otherwise it would be unsafe to continue performing any update work.
+
+The task then fishes configuration settings from the default profile, namely:
+
+-  A subset of update specific preferences, such as ``app.update.log``
+
+-  Data reporting preferences, to ensure the task respects the user’s choices
+
+-  The (legacy) Telemetry client ID, so that background update Telemetry
+   can be correlated with other Firefox Telemetry
+
+The background task creates a distinct profile for itself to load, because a
+profile must be present in order for most of the Firefox code that it relies on
+to function.  This distinct profile is non-ephemeral, i.e., persistent, but not
+visible to users: see `bug 1775132
+<https://bugzilla.mozilla.org/show_bug.cgi?id=1775132>`__
+
+After setting up this profile and reading all the configuration we need
+into it, the regular
+`UpdateService.jsm <https://searchfox.org/mozilla-central/source/toolkit/mozapps/update/UpdateService.jsm>`__
+check process is initiated. To the greatest extent possible, this process is
+identical to what happens during any regular browsing session.
+
+Specific topics
+===============
+
+User interface
+--------------
+
+The background update task must not produce any user-visible interface. If it
+did, whatever appeared would be \*disembodied\*, unconnected to any usage of
+Firefox itself and appearing to a user as a weird, scary popup that came out of
+nowhere. To this end, we disable all UI within the updater when invoking
+from a background task. See `bug
+1696276 <https://bugzilla.mozilla.org/show_bug.cgi?id=1696276>`__.
+
+This point also means that we cannot prompt for user elevation (on Windows this
+would mean a UAC prompt) from within the task, so we have to make very sure that
+we will be able to perform an update without needing to elevate. By default on
+Windows we are able to do this because of the presence of the Maintenance
+Service, but it may be disabled or not installed, so we still have to check.
+
+Staging
+-------
+
+The background update task will follow the update staging setting in the user’s
+default profile. The default setting is to enable staging, so most users will
+have it. In the future, background update tasks will recognize when an update
+has been staged and try to restart to finalize the staged update (see `bug
+1704855 <https://bugzilla.mozilla.org/show_bug.cgi?id=1704855>`__). Background
+tasks cannot finalize a staged update in all cases however; for one example, see
+`bug 1695797 <https://bugzilla.mozilla.org/show_bug.cgi?id=1695797>`__, where we
+ensure that background tasks do not finalize a staged update while other
+instances of the application are running.
+
+Staging is enabled by default because it provides a marked improvement in
+startup time for a browsing session. Without staging, browser startup following
+retrieving an update would be blocked on extracting the update archive and
+patching each individual application file. Staging does all of that in advance,
+so that all that needs to be done to complete an update (and therefore all that
+needs to be done during the startup path), is to move the already patched (that
+is, staged) files into place, a much faster and less resource intensive job.
diff --git a/toolkit/mozapps/update/docs/MaintenanceServiceTests.rst b/toolkit/mozapps/update/docs/MaintenanceServiceTests.rst
new file mode 100644
index 0000000000..b954b572f8
--- /dev/null
+++ b/toolkit/mozapps/update/docs/MaintenanceServiceTests.rst
@@ -0,0 +1,103 @@
+Maintenance Service Tests
+=========================
+
+The automated tests for the Mozilla Maintenance Service are a bit tricky. They
+are located in ``toolkit/mozapps/update/tests/unit_service_updater/`` and they
+allow for automated testing of application update using the Service.
+
+In automation, everything gets signed and the tests properly check that things
+like certificate verification work. If, however, you want to run the tests
+locally, the MAR and the updater binary will not be signed. Thus, the
+verification of those certificates will fail and the tests will not run
+properly.
+
+We don't want these tests to just always fail if someone runs large amounts of
+tests locally. To avoid this, the tests basically just unconditionally pass if
+you run them locally and don't take the time to set them up properly.
+
+If you want them to actually run locally, you will need to set up your
+environment properly.
+
+Setting Up to Run the Tests Locally
+-----------------------------------
+
+In order to run the service tests locally, we have to bypass much of the
+certificate verification. Thus, this method may not be helpful if you need to
+test that feature in particular.
+
+Add Fallback Key to Registry
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+First, you will need to add the fallback key to the registry. Normally, the
+Firefox installer writes some certificate information to a registry key in an
+installation-specific place. In testing, however, we can't get the permissions
+to write this key, nor can we have the test environment predict every possible
+installation directory that we might test with. To get around this problem, if
+the Service can't find the installation-specific key, it will check a static
+fallback location.
+
+The easiest way to correctly set up the fallback key is to copy the text below
+into a ``.reg`` file and then double click it in the file browser to merge it
+into the registry.
+
+.. code::
+
+   Windows Registry Editor Version 5.00
+
+   [HKEY_LOCAL_MACHINE\SOFTWARE\Mozilla\MaintenanceService\3932ecacee736d366d6436db0f55bce4]
+
+   [HKEY_LOCAL_MACHINE\SOFTWARE\Mozilla\MaintenanceService\3932ecacee736d366d6436db0f55bce4\0]
+   "issuer"="DigiCert SHA2 Assured ID Code Signing CA"
+   "name"="Mozilla Corporation"
+
+   [HKEY_LOCAL_MACHINE\SOFTWARE\Mozilla\MaintenanceService\3932ecacee736d366d6436db0f55bce4\1]
+   "issuer"="DigiCert Assured ID Code Signing CA-1"
+   "name"="Mozilla Corporation"
+
+   [HKEY_LOCAL_MACHINE\SOFTWARE\Mozilla\MaintenanceService\3932ecacee736d366d6436db0f55bce4\2]
+   "issuer"="Mozilla Fake CA"
+   "name"="Mozilla Fake SPC"
+
+Build without Certificate Verification
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+To disable certificate verification, add this build flag to your ``mozconfig``
+file:
+
+.. code::
+
+   ac_add_options --enable-unverified-updates
+
+You will need to rebuild for this to take effect.
+
+Copy the Maintenance Service Binary
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+This step will assume that you already have the Maintenance Service installed.
+
+First, move the existing Maintenance Service binary out of the way. It will
+initially be located at
+``C:\Program Files (x86)\Mozilla Maintenance Service\maintenanceservice.exe``.
+An easy way to do this is to append ``.bak`` to its name. You should probably
+restore your original Maintenance Service binary when you are done testing.
+
+Now, copy the Maintenance Service binary that you built into that directory.
+It will be located at ``<obj directory>\dist\bin\maintenanceservice.exe``.
+
+If you make changes to the Maintenance Service and rebuild, you will have to
+repeat this step.
+
+Running the Tests
+-----------------
+
+You should now be ready run a service test:
+
+.. code::
+
+   ./mach test toolkit/mozapps/update/tests/unit_service_updater/<test>
+
+Or run all of them:
+
+.. code::
+
+   ./mach test toolkit/mozapps/update/tests/unit_service_updater
diff --git a/toolkit/mozapps/update/docs/SettingUpAnUpdateServer.rst b/toolkit/mozapps/update/docs/SettingUpAnUpdateServer.rst
new file mode 100644
index 0000000000..cecc81a1d9
--- /dev/null
+++ b/toolkit/mozapps/update/docs/SettingUpAnUpdateServer.rst
@@ -0,0 +1,218 @@
+Setting Up An Update Server
+===========================
+
+The goal of this document is to provide instructions for installing a
+locally-served Firefox update.
+
+Obtaining an update MAR
+-----------------------
+
+Updates are served as MAR files. There are two common ways to obtain a
+MAR to use: download a prebuilt one, or build one yourself.
+
+Downloading a MAR
+~~~~~~~~~~~~~~~~~
+
+Prebuilt Nightly MARs can be found
+`here <https://archive.mozilla.org/pub/firefox/nightly/>`__ on
+archive.mozilla.org. Be sure that you use the one that matches your
+machine's configuration. For example, if you want the Nightly MAR from
+2019-09-17 for a 64 bit Windows machine, you probably want the MAR
+located at
+https://archive.mozilla.org/pub/firefox/nightly/2019/09/2019-09-17-09-36-29-mozilla-central/firefox-71.0a1.en-US.win64.complete.mar.
+
+Prebuilt MARs for release and beta can be found
+`here <https://archive.mozilla.org/pub/firefox/releases/>`__. Beta
+builds are those with a ``b`` in the version string. After locating the
+desired version, the MARs will be in the ``update`` directory. You want
+to use the MAR labelled ``complete``, not a partial MAR. Here is an
+example of an appropriate MAR file to use:
+https://archive.mozilla.org/pub/firefox/releases/69.0b9/update/win64/en-US/firefox-69.0b9.complete.mar.
+
+Building a MAR
+~~~~~~~~~~~~~~
+
+Building a MAR locally is more complicated. Part of the problem is that
+MARs are signed by Mozilla and so you cannot really build an "official"
+MAR yourself. This is a security measure designed to prevent anyone from
+serving malicious updates. If you want to use a locally-built MAR, the
+copy of Firefox being updated will need to be built to allow un-signed
+MARs. See :ref:`Building Firefox <Firefox Contributors' Quick Reference>`
+for more information on building Firefox locally. In order to use a locally
+built MAR, you will need to put this line in the mozconfig file in root of the
+build directory (create it if it does not exist):
+
+.. code::
+
+   ac_add_options --enable-unverified-updates
+
+Firefox should otherwise be built normally. After building, you may want
+to copy the installation of Firefox elsewhere. If you update the
+installation without moving it, attempts at further incremental builds
+will not work properly, and a clobber will be needed when building next.
+To move the installation, first call ``./mach package``, then copy
+``<obj dir>/dist/firefox`` elsewhere. The copied directory will be your
+install directory.
+
+If you are running Windows and want the `Mozilla Maintenance
+Service <https://support.mozilla.org/en-US/kb/what-mozilla-maintenance-service>`__
+to be used, there are a few additional steps to be taken here. First,
+the maintenance service needs to be "installed". Most likely, a
+different maintenance service is already installed, probably at
+``C:\Program Files (x86)\Mozilla Maintenance Service\maintenanceservice.exe``.
+Backup that file to another location and replace it with
+``<obj dir>/dist/bin/maintenanceservice.exe``. Don't forget to restore
+the backup when you are done. Next, you will need to change the
+permissions on the Firefox install directory that you created. Both that
+directory and its parent directory should have permissions preventing
+the current user from writing to it.
+
+Now that you have a build of Firefox capable of using a locally-built
+MAR, it's time to build the MAR. First, build Firefox the way you want
+it to be after updating. If you want it to be the same before and after
+updating, this step is unnecessary and you can use the same build that
+you used to create the installation. Then run these commands,
+substituting ``<obj dir>``, ``<MAR output path>``, ``<version>`` and
+``<channel>`` appropriately:
+
+.. code:: bash
+
+   $ ./mach package
+   $ touch "<obj dir>/dist/firefox/precomplete"
+   $ MAR="<obj dir>/dist/host/bin/mar.exe" MOZ_PRODUCT_VERSION=<version> MAR_CHANNEL_ID=<channel> ./tools/update-packaging/make_full_update.sh <MAR output path> "<obj dir>/dist/firefox"
+
+For macOS you should use these commands:
+
+.. code:: bash
+
+   $ ./mach package
+   $ touch "<obj dir>/dist/firefox/Firefox.app/Contents/Resources/precomplete"
+   $ MAR="<obj dir>/dist/host/bin/mar.exe" MOZ_PRODUCT_VERSION=<version> MAR_CHANNEL_ID=<channel> ./tools/update-packaging/make_full_update.sh <MAR output path> "<obj dir>/dist/firefox/Firefox.app"
+
+For a local build, ``<channel>`` can be ``default``, and ``<version>``
+can be the value from ``browser/config/version.txt`` (or something
+arbitrarily large like ``2000.0a1``).
+
+.. container:: blockIndicator note
+
+   Note: It can be a bit tricky to get the ``make_full_update.sh``
+   script to accept paths with spaces.
+
+Serving the update
+------------------
+
+Preparing the update files
+~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+First, create the directory that updates will be served from and put the
+MAR file in it. Then, create a file within called ``update.xml`` with
+these contents, replacing ``<mar name>``, ``<hash>`` and ``<size>`` with
+the MAR's filename, its sha512 hash, and its file size in bytes.
+
+::
+
+   <?xml version="1.0" encoding="UTF-8"?>
+   <updates>
+       <update type="minor" displayVersion="2000.0a1" appVersion="2000.0a1" platformVersion="2000.0a1" buildID="21181002100236">
+           <patch type="complete" URL="http://127.0.0.1:8000/<mar name>" hashFunction="sha512" hashValue="<hash>" size="<size>"/>
+       </update>
+   </updates>
+
+If you've downloaded the MAR you're using, you'll find the sha512 value
+in a file called SHA512SUMS in the root of the release directory on
+archive.mozilla.org for a release or beta build (you'll have to search
+it for the file name of your MAR, since it includes the sha512 for every
+file that's part of that release), and for a nightly build you'll find a
+file with a .checksums extension adjacent to your MAR that contains that
+information (for instance, for the MAR file at
+https://archive.mozilla.org/pub/firefox/nightly/2019/09/2019-09-17-09-36-29-mozilla-central/firefox-71.0a1.en-US.win64.complete.mar,
+the file
+https://archive.mozilla.org/pub/firefox/nightly/2019/09/2019-09-17-09-36-29-mozilla-central/firefox-71.0a1.en-US.win64.checksums
+contains the sha512 for that file as well as for all the other win64
+files that are part of that nightly release).
+
+If you've built your own MAR, you can obtain its sha512 checksum by
+running the following command, which should work in Linux, macOS, or
+Windows in the MozillaBuild environment:
+
+.. code::
+
+   shasum --algorithm 512 <filename>
+
+On Windows, you can get the exact file size in bytes for your MAR by
+right clicking on it in the file explorer and selecting Properties.
+You'll find the correct size in bytes at the end of the line that begins
+"Size", **not** the one that begins "Size on disk". Be sure to remove
+the commas when you paste this number into the XML file.
+
+On macOS, you can get the exact size of your MAR by running the command:
+
+.. code::
+
+   stat -f%z <filename>
+
+Or on Linux, the same command would be:
+
+.. code::
+
+   stat --format "%s" <filename>
+
+Starting your update server
+~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Now, start an update server to serve the update files on port 8000. An
+easy way to do this is with Python. Remember to navigate to the correct
+directory before starting the server. This is the Python2 command:
+
+.. code:: bash
+
+   $ python -m SimpleHTTPServer 8000
+
+or, this is the Python3 command:
+
+.. code:: bash
+
+   $ python3 -m http.server 8000
+
+.. container:: blockIndicator note
+
+   If you aren't sure that you started the server correctly, try using a
+   web browser to navigate to ``http://127.0.0.1:8000/update.xml`` and
+   make sure that you get the XML file you created earlier.
+
+Installing the update
+---------------------
+
+You may want to start by deleting any pending updates to ensure that no
+previously found updates interfere with installing the desired update.
+You can use this command with Firefox's browser console to determine the
+update directory:
+
+.. code::
+
+   ChromeUtils.importESModule("resource://gre/modules/FileUtils.sys.mjs").FileUtils.getDir("UpdRootD", [], false).path
+
+Once you have determined the update directory, close Firefox, browse to
+the directory and remove the subdirectory called ``updates``.
+
+| Next, you need to change the update URL to point to the local XML
+  file. This can be done most reliably with an enterprise policy. The
+  policy file location depends on the operating system you are using.
+| Windows/Linux: ``<install dir>/distribution/policies.json``
+| macOS: ``<install dir>/Contents/Resources/distribution/policies.json``
+| Create the ``distribution`` directory, if necessary, and put this in
+  ``policies.json``:
+
+::
+
+   {
+     "policies": {
+       "AppUpdateURL": "http://127.0.0.1:8000/update.xml"
+     }
+   }
+
+Now you are ready to update! Launch Firefox out of its installation
+directory and navigate to the Update section ``about:preferences``. You
+should see it downloading the update to the update directory. Since the
+transfer is entirely local this should finish quickly, and a "Restart to
+Update" button should appear. Click it to restart and apply the update.
diff --git a/toolkit/mozapps/update/docs/index.rst b/toolkit/mozapps/update/docs/index.rst
new file mode 100644
index 0000000000..d6cc9f5e64
--- /dev/null
+++ b/toolkit/mozapps/update/docs/index.rst
@@ -0,0 +1,10 @@
+==================
+Application Update
+==================
+
+.. toctree::
+   :maxdepth: 1
+
+   BackgroundUpdates
+   MaintenanceServiceTests
+   SettingUpAnUpdateServer