10 files changed, 631 insertions, 0 deletions
diff --git a/tools/tryselect/docs/selectors/again.rst b/tools/tryselect/docs/selectors/again.rst
new file mode 100644
index 0000000000..da592b7a34
--- /dev/null
+++ b/tools/tryselect/docs/selectors/again.rst
@@ -0,0 +1,36 @@
+Again Selector
+==============
+
+When you push to try, the computed ``try_task_config.json`` is saved in a
+history file under ``~/.mozbuild/srcdirs/<srcdir hash>/history`` (note: the
+``syntax`` selector does not use ``try_task_config.json`` yet so does not save
+any history). You can then use the ``again`` selector to re-push any of your
+previously generated task configs.
+
+In the simple case, you can re-run your last try push with:
+
+.. code-block:: shell
+
+    $ mach try again
+
+If you want to re-push a task config a little further down the history stack,
+first you need to figure out its index with:
+
+.. code-block:: shell
+
+    $ mach try again --list
+
+Then run:
+
+.. code-block:: shell
+
+    $ mach try again --index <index>
+
+Note that index ``0`` is always the most recent ``try_task_config.json`` in the
+history. You can clear your history with:
+
+.. code-block:: shell
+
+    $ mach try again --purge
+
+Only the 10 most recent pushes will be saved in your history.
diff --git a/tools/tryselect/docs/selectors/auto.rst b/tools/tryselect/docs/selectors/auto.rst
new file mode 100644
index 0000000000..39ab5b7450
--- /dev/null
+++ b/tools/tryselect/docs/selectors/auto.rst
@@ -0,0 +1,24 @@
+Auto Selector
+=============
+
+This selector automatically determines the most efficient set of tests and
+tasks to run against your push. It accomplishes this via a combination of
+machine learning and manual heuristics. The tasks selected here should match
+pretty closely to what would be scheduled if your patch were pushed to
+autoland.
+
+It is the officially recommended selector to use when you are unsure of which
+tasks should run on your push.
+
+To use:
+
+.. code-block::
+
+   $ mach try auto
+
+Unlike other try selectors, tasks are not chosen locally. Rather they will be
+computed by the decision task.
+
+Like most other selectors, ``mach try auto`` supports many of the standard
+templates such as ``--artifact`` or ``--env``. See ``mach try auto --help`` for
+the full list of supported templates.
diff --git a/tools/tryselect/docs/selectors/chooser.rst b/tools/tryselect/docs/selectors/chooser.rst
new file mode 100644
index 0000000000..6c59d54009
--- /dev/null
+++ b/tools/tryselect/docs/selectors/chooser.rst
@@ -0,0 +1,32 @@
+Chooser Selector
+================
+
+When pushing to try, there are a very large amount of builds and tests to choose from. Often too
+many to remember, making it easy to forget a set of tasks which should otherwise have been run.
+
+This selector allows you to select tasks from a web interface that lists all the possible build and
+test tasks and allows you to select them from a list. It is similar in concept to the old `try
+syntax chooser`_ page, except that the values are dynamically generated using the :ref:`taskgraph<TaskCluster Task-Graph Generation>` as an
+input. This ensures that it will never be out of date.
+
+To use:
+
+.. code-block:: shell
+
+    $ mach try chooser
+
+This will spin up a local web server (using Flask) which serves the chooser app. After making your
+selection, simply press ``Push`` and the rest will be handled from there. No need to copy/paste any
+syntax strings or the like.
+
+You can run:
+
+.. code-block:: shell
+
+    $ mach try chooser --full
+
+To generate the interface using the full :ref:`taskgraph<TaskCluster Task-Graph Generation>` instead. This will include tasks that don't run
+on mozilla-central.
+
+
+.. _try syntax chooser: https://mozilla-releng.net/trychooser
diff --git a/tools/tryselect/docs/selectors/empty.rst b/tools/tryselect/docs/selectors/empty.rst
new file mode 100644
index 0000000000..4eb6a03fd8
--- /dev/null
+++ b/tools/tryselect/docs/selectors/empty.rst
@@ -0,0 +1,22 @@
+Empty Selector
+==============
+
+The ``mach try empty`` subcommand is very simple, it won't schedule any additional tasks. You'll
+still see lint tasks and python-unittest tasks if applicable, this is due to a configuration option
+in taskcluster.
+
+Other than those, your try run will be empty. You can use treeherder's ``Add new jobs`` feature to
+selectively add additional tasks after the fact.
+
+.. note::
+
+    To use ``Add new jobs`` you'll need to be logged in and have commit access level 1, just as if
+    you were pushing to try.
+
+To do this:
+
+    1. Click the drop-down arrow at the top right of your commit.
+    2. Select ``Add new jobs`` (it may take a couple seconds to load).
+    3. Choose your desired tasks by clicking them one at a time.
+    4. At the top of your commit, select ``Trigger New Jobs``.
+
diff --git a/tools/tryselect/docs/selectors/fuzzy.rst b/tools/tryselect/docs/selectors/fuzzy.rst
new file mode 100644
index 0000000000..d50f801eb0
--- /dev/null
+++ b/tools/tryselect/docs/selectors/fuzzy.rst
@@ -0,0 +1,371 @@
+Fuzzy Selector
+==============
+
+The fuzzy selector uses a tool called `fzf`_. It allows you to filter down all of the task labels
+from a terminal based UI and an intelligent fuzzy finding algorithm. If the ``fzf`` binary is not
+installed, you'll be prompted to bootstrap it on first run.
+
+
+Understanding the Interface
+---------------------------
+
+When you run ``mach try fuzzy`` an interface similar to the one below will open. This is `fzf`_.
+
+.. image:: fzf.png
+
+There's a lot to unpack here, so let's examine each component a bit more closely.
+
+   A. The set of tasks that match the currently typed-out query. In the above image only tasks that
+   match the query ``'linux64 mochibrochr`` are displayed.
+
+   B. The set of selected tasks. These are the tasks that will be scheduled once you hit ``Enter``.
+   In other words, if the task you want does not appear here, *it won't be scheduled*.
+
+   C. Count information of the form ``x/y (z)``, where ``x`` is the number of tasks that match the
+   current query, ``y`` is the total number of tasks and ``z`` is the number of tasks you have
+   selected.
+
+   D. The input bar for entering queries. As you type you'll notice the list of tasks in ``A``
+   starts to update immediately. In the image above, the query ``'linux64 mochibrochr`` is entered.
+   Correspondingly only tasks matching that query are displayed.
+
+In general terms, you first find tasks on the left. Then you move them over to the right by
+selecting them. Once you are satisfied with your selection, press ``Enter`` to push to try.
+
+
+Selecting Tasks
+---------------
+
+There are few ways you can select tasks. If you are feeling a bit overwhelmed, it might be best to
+stick with the mouse to start:
+
+   1. Enter a query (e.g ``mochitest``) to reduce the task list a little.
+   2. Scroll up and look for the task(s) you want.
+   3. ``Right-Click`` as many tasks as desired to select them.
+   4. Optionally delete your query, go back to step 1) and repeat.
+   5. Press ``Enter`` to push (or ``Esc`` to cancel).
+
+.. note::
+
+   Dependencies are automatically filled in, so you can select a test task without needing
+   to select the build it depends on.
+
+As you ``Right-Click``, notice that a little arrow appears to the left of the task label. This
+indicates that the task is selected and exists in the preview pane to the right.
+
+Once you are a bit more comfortable with the interface, using the keyboard is much better at quickly
+selecting tasks. Here are the main shortcuts you'll need:
+
+.. code-block:: text
+
+    Ctrl-K / Up    => Move cursor up
+    Ctrl-J / Down  => Move cursor down
+    Tab            => Select task + move cursor down
+    Shift-Tab      => Select task + move cursor up
+    Ctrl-A         => Select all currently filtered tasks
+    Ctrl-T         => Toggle select all currently filtered tasks
+    Ctrl-D         => De-select all selected tasks (both filtered and not)
+    Alt-Bspace     => Clear query from input bar
+    Enter          => Accept selection and exit
+    Ctrl-C / Esc   => Cancel selection and exit
+    ?              => Toggle preview pane
+
+
+The process for selecting tasks is otherwise the same as for a mouse. A particularly fast and
+powerful way to select tasks is to:
+
+.. code-block:: text
+
+    Write a precise query => Ctrl-A => Alt-Bspace => Repeat
+
+As before, when you are satisfied with your selection press ``Enter`` and all the tasks in the
+preview pane will be pushed to try. If you change your mind you can press ``Esc`` or ``Ctrl-C`` to
+exit the interface without pushing anything.
+
+.. note::
+
+   Initially ``fzf`` will automatically select whichever task is under your cursor. This is a
+   convenience feature for the case where you are only selecting a single task. This feature will be
+   turned off as soon as you *lock in* a selection with ``Right-Click``, ``Tab`` or ``Ctrl-A``.
+
+
+Writing Queries
+---------------
+
+Queries are built from a series of terms, each separated by a space. Terms are logically joined by
+the AND operator. For example:
+
+.. code-block:: text
+
+    'windows 'mochitest
+
+This query has two terms, and is the equivalent of saying: Give me all the tasks that match both the
+term ``'windows'`` and the term ``'mochitest'``. In other words, this query matches all Windows
+mochitest tasks.
+
+The single quote prefix before each term tells ``fzf`` to use exact substring matches, so only tasks
+that contain both the literal string ``windows`` AND the literal string ``mochitest`` will be
+matched.
+
+Another thing to note is that the order of the terms makes no difference, so ``'windows 'mochitest``
+and ``'mochitest 'windows`` are equivalent.
+
+
+Fuzzy terms
+~~~~~~~~~~~
+
+If a term is *not* prefixed with a single quote, that makes it a fuzzy term. This means the
+characters in the term need to show up in order, but not in sequence. E.g the fuzzy term ``max``
+would match the string ``mozilla firefox`` (as first there is an ``m``, then an ``a`` and finally an
+``x``), but not the string ``firefox by mozilla`` (since the ``x`` is now out of order). Here's a
+less contrived example:
+
+.. code-block:: text
+
+    wndws mchtst
+
+Like the query above, this one would also select all Windows mochitest tasks. But it will
+additionally select:
+
+.. code-block:: text
+
+    test-macosx1014-64-shippable/opt-talos-sessionrestore-many-windows-e10s
+
+This is because both sequences of letters (``wndws`` and ``mchtst``) independently appear in order
+somewhere in this string (remember the order of the terms makes no difference).
+
+At first fuzzy terms may not seem very useful, but they are actually extremely powerful! Let's use
+the term from the interface image above, ``'linux64 mochibrochr``, as an example. First, just notice
+how in the image ``fzf`` highlights the characters that constitute the match in green. Next, notice
+how typing ``mochibrochr`` can quickly get us all mochitest browser-chrome tasks. The power of fuzzy
+terms is that you don't need to memorize the exact task labels you are looking for. Just start
+typing something you think is vaguely correct and chances are you'll see the task you're looking for.
+
+
+Term Modifiers
+~~~~~~~~~~~~~~
+
+The following modifiers can be applied to a search term:
+
+.. code-block:: text
+
+    'word    => exact match (line must contain the literal string "word")
+    ^word    => exact prefix match (line must start with literal "word")
+    word$    => exact suffix match (line must end with literal "word")
+    !word    => exact negation match (line must not contain literal "word")
+    'a | 'b  => OR operator (joins two exact match operators together)
+
+For example:
+
+.. code-block:: text
+
+    ^start 'exact | 'other !ignore fuzzy end$
+
+would match the string:
+
+.. code-block:: text
+
+    starting to bake isn't exactly fun, but pizza is yummy in the end
+
+.. note::
+
+    The best way to learn how to write queries is to run ``mach try fuzzy --no-push`` and play
+    around with all of these modifiers!
+
+
+Specifying Queries on the Command Line
+--------------------------------------
+
+Sometimes it's more convenient to skip the interactive interface and specify a query on the command
+line with ``-q/--query``. This is equivalent to opening the interface then typing:
+``<query><ctrl-a><enter>``.
+
+For example:
+
+.. code-block:: shell
+
+    # selects all mochitest tasks
+    $ mach try fuzzy --query "mochitest"
+
+You can pass in multiple queries at once and the results of each will be joined together:
+
+.. code-block:: shell
+
+    # selects all mochitest and reftest tasks
+    $ mach try fuzzy -q "mochitest" -q "reftest"
+
+If instead you want the intersection of queries, you can pass in ``-x/--and``:
+
+.. code-block:: shell
+
+    # selects all windows mochitest tasks
+    $ mach try fuzzy --and -q "mochitest" -q "windows"
+
+
+Modifying Presets
+~~~~~~~~~~~~~~~~~
+
+:doc:`Presets <../presets>` make it easy to run a pre-determined set of tasks. But sometimes you
+might not want to run that set exactly as is, you may only want to use the preset as a starting
+point then add or remove tasks as needed. This can be accomplished with ``-q/--query`` or
+``-i/--interactive``.
+
+Here are some examples of adding tasks to a preset:
+
+.. code-block:: shell
+
+    # selects all perf tasks plus all mochitest-chrome tasks
+    $ mach try fuzzy --preset perf -q "mochitest-chrome"
+
+    # adds tasks to the perf preset interactively
+    $ mach try fuzzy --preset perf -i
+
+Similarly, ``-x/--and`` can be used to filter down a preset by taking the intersection of the two
+sets:
+
+.. code-block:: shell
+
+    # limits perf tasks to windows only
+    $ mach try fuzzy --preset perf -xq "windows"
+
+    # limits perf tasks interactively
+    $ mach try fuzzy --preset perf -xi
+
+
+Shell Conflicts
+~~~~~~~~~~~~~~~
+
+Unfortunately ``fzf``'s query language uses some characters (namely ``'``, ``!`` and ``$``) that can
+interfere with your shell when using ``-q/--query``. Below are some tips for how to type out a query
+on the command line.
+
+The ``!`` character is typically used for history expansion. If you don't use this feature, the
+easiest way to specify queries on the command line is to disable it:
+
+.. code-block:: shell
+
+    # bash
+    $ set +H
+    $ ./mach try fuzzy -q "'foo !bar"
+
+    # zsh
+    $ setopt no_banghist
+    $ ./mach try fuzzy -q "'foo !bar"
+
+If using ``bash``, add ``set +H`` to your ``~/.bashrc``, ``~/.bash_profile`` or equivalent. If using
+``zsh``, add ``setopt no_banghist`` to your ``~/.zshrc`` or equivalent.
+
+If you don't want to disable history expansion, you can escape your queries like this:
+
+.. code-block:: shell
+
+    # bash
+    $ ./mach try fuzzy -q $'\'foo !bar'
+
+    # zsh
+    $ ./mach try fuzzy -q "'foo \!bar"
+
+
+The third option is to use ``-e/--exact`` which reverses the behaviour of the ``'`` character (see
+:ref:`additional-arguments` for more details). Using this flag means you won't need to escape the
+``'`` character as often and allows you to run your queries like this:
+
+.. code-block:: shell
+
+    # bash and zsh
+    $ ./mach try fuzzy -eq 'foo !bar'
+
+This method is only useful if you find you almost always prefix terms with ``'`` (and rarely use
+fuzzy terms). Otherwise as soon as you want to use a fuzzy match you'll run into the same problem as
+before.
+
+.. note:: All the examples in these three approaches will select the same set of tasks.
+
+If you use ``fish`` shell, you won't need to escape ``!``, however you will need to escape ``$``:
+
+.. code-block:: shell
+
+    # fish
+    $ ./mach try fuzzy -q "'foo !bar baz\$"
+
+
+Test Paths
+----------
+
+One or more paths to a file or directory may be specified as positional arguments. When
+specifying paths, the list of available tasks to choose from is filtered down such that
+only suites that have tests in a specified path can be selected. Notably, only the first
+chunk of each suite/platform appears. When the tasks are scheduled, only tests that live
+under one of the specified paths will be run.
+
+.. note::
+
+    When using paths, be aware that all tests under the specified paths will run in the
+    same chunk. This might produce a different ordering from what gets run on production
+    branches, and may yield different results.
+
+    For suites that restart the browser between each manifest (like mochitest), this
+    shouldn't be as big of a concern.
+
+Paths can be used with the interactive ``fzf`` window, or using the ``-q/--query`` argument.
+For example, running:
+
+.. code-block:: shell
+
+    $ mach try fuzzy layout/reftests/reftest-sanity -q "!pgo !cov !asan 'linux64"
+
+Would produce the following ``try_task_config.json``:
+
+.. code-block:: json
+
+    {
+      "env":{
+        "MOZHARNESS_TEST_PATHS":"{\"reftest\":\"layout/reftests/reftest-sanity\"}"
+      },
+      "tasks":[
+        "test-linux64-qr/debug-reftest-e10s-1",
+        "test-linux64-qr/opt-reftest-e10s-1",
+        "test-linux64/debug-reftest-e10s-1",
+        "test-linux64/debug-reftest-no-accel-e10s-1",
+        "test-linux64/opt-reftest-e10s-1",
+        "test-linux64/opt-reftest-no-accel-e10s-1",
+      ]
+    }
+
+Inside of these tasks, the reftest harness will only run tests that live under
+``layout/reftests/reftest-sanity``.
+
+
+.. _additional-arguments:
+
+Additional Arguments
+--------------------
+
+There are a few additional command line arguments you may wish to use:
+
+``-e/--exact``
+By default, ``fzf`` treats terms as a fuzzy match and prefixing a term with ``'`` turns it into an exact
+match. If passing in ``--exact``, this behaviour is reversed. Non-prefixed terms become exact, and a
+``'`` prefix makes a term fuzzy.
+
+``--full``
+By default, only target tasks (e.g tasks that would normally run on mozilla-central)
+are generated. Passing in ``--full`` allows you to select from all tasks. This is useful for
+things like nightly or release tasks.
+
+``-u/--update``
+Update the bootstrapped ``fzf`` binary to the latest version.
+
+For a full list of command line arguments, run:
+
+.. code-block:: shell
+
+    $ mach try fuzzy --help
+
+For more information on using ``fzf``, run:
+
+.. code-block:: shell
+
+    $ man fzf
+
+.. _fzf: https://github.com/junegunn/fzf
diff --git a/tools/tryselect/docs/selectors/fzf.png b/tools/tryselect/docs/selectors/fzf.png
new file mode 100644
index 0000000000..a64f4b04f3
--- /dev/null
+++ b/tools/tryselect/docs/selectors/fzf.png
diff --git a/tools/tryselect/docs/selectors/index.rst b/tools/tryselect/docs/selectors/index.rst
new file mode 100644
index 0000000000..40624fc910
--- /dev/null
+++ b/tools/tryselect/docs/selectors/index.rst
@@ -0,0 +1,43 @@
+Selectors
+=========
+
+These are the currently implemented try selectors:
+
+* :doc:`auto <auto>`: Have tasks chosen for you automatically.
+* :doc:`fuzzy <fuzzy>`: Select tasks using a fuzzy finding algorithm and
+  a terminal interface.
+* :doc:`chooser <chooser>`: Select tasks using a web interface.
+* :doc:`again <again>`: Re-run a previous ``try_task_config.json`` based
+  push.
+* :doc:`empty <empty>`: Don't select any tasks. Taskcluster will still run
+  some tasks automatically (like lint and python unittest tasks). Further tasks
+  can be chosen with treeherder's ``Add New Jobs`` feature.
+* :doc:`syntax <syntax>`: Select tasks using classic try syntax.
+* :doc:`release <release>`: Prepare a tree for doing a staging release.
+* :doc:`scriptworker <scriptworker>`: Run scriptworker tasks against a recent release.
+
+You can run them with:
+
+.. code-block:: shell
+
+    $ mach try <selector>
+
+See selector specific options by running:
+
+.. code-block:: shell
+
+    $ mach try <selector> --help
+
+.. toctree::
+  :caption: Available Selectors
+  :maxdepth: 1
+  :hidden:
+
+  Auto <auto>
+  Fuzzy <fuzzy>
+  Chooser <chooser>
+  Again <again>
+  Empty <empty>
+  Syntax <syntax>
+  Release <release>
+  Scriptworker <scriptworker>
diff --git a/tools/tryselect/docs/selectors/release.rst b/tools/tryselect/docs/selectors/release.rst
new file mode 100644
index 0000000000..946266249b
--- /dev/null
+++ b/tools/tryselect/docs/selectors/release.rst
@@ -0,0 +1,31 @@
+Release Selector
+================
+
+This command configures the tree in preparation for doing a staging release,
+and pushes the result to try. The changes that that are made include:
+
+- Updating the version number.
+- Applying the migrations that are done as part of merge day.
+- Disabling repacking most locales. (This can be disabled by passing ``--no-limit-locales``).
+
+For staging a beta release, run the following (with an appropriate version number):
+
+.. code-block:: shell
+
+   $ mach try release --version 64.0b5 --migration central-to-beta
+
+For staging a final release (rc or patch), run the following (with an appropriate version number)
+
+.. code-block:: shell
+
+   $ mach try release --version 64.0 --migration central-to-beta --migration beta-to-release
+
+Once the decision task is on the push is complete, you can start the release
+through `staging ship-it instance <https://shipit.staging.mozilla-releng.net/new>`_\ [#shipit]_.
+
+.. note::
+
+   If pushing from beta or release, the corresponding migration should not be
+   passed, as they have already been applied.
+
+.. [#shipit] This is only available to release engineering and release management (as of 2018-10-15).
diff --git a/tools/tryselect/docs/selectors/scriptworker.rst b/tools/tryselect/docs/selectors/scriptworker.rst
new file mode 100644
index 0000000000..a3cba08cbe
--- /dev/null
+++ b/tools/tryselect/docs/selectors/scriptworker.rst
@@ -0,0 +1,31 @@
+Scriptworker Selector
+=====================
+
+This command runs a selection of scriptworker tasks against builds from a
+recent release.  This is aimed at release engineering, to test changes to
+scriptworker implementations. It currently requires being connected to
+Mozilla's internal datacenter VPN with access to shipit\ [#shipit]_.
+
+There are a number of preset groups of tasks to run. To run a particular
+set of tasks, pass the name of the set to ``mach try scriptworker``:
+
+.. code-block:: shell
+
+   $ mach try scriptworker linux-signing
+
+To get the list of task sets, along with the list of tasks they will run:
+
+.. code-block:: shell
+
+   $ mach try scriptworker list
+
+The selector defaults to using tasks from the most recent beta, to use tasks
+from a different release, pass ``--release-type <release-type>``:
+
+.. code-block:: shell
+
+   $ mach try scriptworker --release-type release linux-signing
+
+
+.. [#shipit] The shipit API is not currently publicly available, and is used
+   to find the release graph to pull previous tasks from.
diff --git a/tools/tryselect/docs/selectors/syntax.rst b/tools/tryselect/docs/selectors/syntax.rst
new file mode 100644
index 0000000000..109b034c7b
--- /dev/null
+++ b/tools/tryselect/docs/selectors/syntax.rst
@@ -0,0 +1,41 @@
+Syntax Selector
+===============
+
+.. warning::
+
+    Try syntax is antiquated and hard to understand. If you aren't already
+    familiar with try syntax, you might want to use the ``fuzzy`` selector
+    instead.
+
+Try syntax is a command line string that goes into the commit message. Using
+``mach try syntax`` will automatically create a temporary commit with your
+chosen syntax and then delete it again after pushing to try.
+
+Try syntax can contain all kinds of different options parsed by various
+places in various repos, but the majority are parsed by `try_option_syntax.py`_.
+The most common arguments include:
+
+    * ``-b/--build`` - One of ``d``, ``o`` or ``do``. This is the build type,
+      either opt, debug or both (required).
+    * ``-p/--platform`` - The platforms you want to build and/or run tests on
+      (required).
+    * ``-u/--unittests`` - The test tasks you want to run (optional).
+    * ``-t/--talos`` - The talos tasks you want to run (optional).
+
+Here are some examples:
+
+.. code-block:: shell
+
+    $ mach try syntax -b do -p linux,macosx64 -u mochitest-e10s-1,crashtest -t none
+    $ mach try syntax -b d -p win64 -u all
+    $ mach try syntax -b o -p linux64
+
+Unfortunately, knowing the magic strings that make it work can be a bit of a
+guessing game. If you are unsure of what string will get your task to run, try
+using :doc:`mach try fuzzy <fuzzy>` instead.
+
+While using ``mach try syntax -b do -p all -u all -t all`` will work, heavy use
+of ``all`` is discouraged as it consumes a lot of unnecessary resources (some of
+which are hardware constrained).
+
+..  _try_option_syntax.py: https://searchfox.org/mozilla-central/source/taskcluster/taskgraph/try_option_syntax.py