diff options
Diffstat (limited to 'tools/tryselect/docs/selectors')
-rw-r--r-- | tools/tryselect/docs/selectors/again.rst | 36 | ||||
-rw-r--r-- | tools/tryselect/docs/selectors/auto.rst | 24 | ||||
-rw-r--r-- | tools/tryselect/docs/selectors/chooser.rst | 32 | ||||
-rw-r--r-- | tools/tryselect/docs/selectors/compare.rst | 17 | ||||
-rw-r--r-- | tools/tryselect/docs/selectors/empty.rst | 21 | ||||
-rw-r--r-- | tools/tryselect/docs/selectors/fuzzy.rst | 371 | ||||
-rw-r--r-- | tools/tryselect/docs/selectors/fzf.png | bin | 0 -> 44467 bytes | |||
-rw-r--r-- | tools/tryselect/docs/selectors/index.rst | 44 | ||||
-rw-r--r-- | tools/tryselect/docs/selectors/release.rst | 31 | ||||
-rw-r--r-- | tools/tryselect/docs/selectors/scriptworker.rst | 31 | ||||
-rw-r--r-- | tools/tryselect/docs/selectors/syntax.rst | 41 |
11 files changed, 648 insertions, 0 deletions
diff --git a/tools/tryselect/docs/selectors/again.rst b/tools/tryselect/docs/selectors/again.rst new file mode 100644 index 0000000000..da592b7a34 --- /dev/null +++ b/tools/tryselect/docs/selectors/again.rst @@ -0,0 +1,36 @@ +Again Selector +============== + +When you push to try, the computed ``try_task_config.json`` is saved in a +history file under ``~/.mozbuild/srcdirs/<srcdir hash>/history`` (note: the +``syntax`` selector does not use ``try_task_config.json`` yet so does not save +any history). You can then use the ``again`` selector to re-push any of your +previously generated task configs. + +In the simple case, you can re-run your last try push with: + +.. code-block:: shell + + $ mach try again + +If you want to re-push a task config a little further down the history stack, +first you need to figure out its index with: + +.. code-block:: shell + + $ mach try again --list + +Then run: + +.. code-block:: shell + + $ mach try again --index <index> + +Note that index ``0`` is always the most recent ``try_task_config.json`` in the +history. You can clear your history with: + +.. code-block:: shell + + $ mach try again --purge + +Only the 10 most recent pushes will be saved in your history. diff --git a/tools/tryselect/docs/selectors/auto.rst b/tools/tryselect/docs/selectors/auto.rst new file mode 100644 index 0000000000..39ab5b7450 --- /dev/null +++ b/tools/tryselect/docs/selectors/auto.rst @@ -0,0 +1,24 @@ +Auto Selector +============= + +This selector automatically determines the most efficient set of tests and +tasks to run against your push. It accomplishes this via a combination of +machine learning and manual heuristics. The tasks selected here should match +pretty closely to what would be scheduled if your patch were pushed to +autoland. + +It is the officially recommended selector to use when you are unsure of which +tasks should run on your push. + +To use: + +.. code-block:: + + $ mach try auto + +Unlike other try selectors, tasks are not chosen locally. Rather they will be +computed by the decision task. + +Like most other selectors, ``mach try auto`` supports many of the standard +templates such as ``--artifact`` or ``--env``. See ``mach try auto --help`` for +the full list of supported templates. diff --git a/tools/tryselect/docs/selectors/chooser.rst b/tools/tryselect/docs/selectors/chooser.rst new file mode 100644 index 0000000000..6c59d54009 --- /dev/null +++ b/tools/tryselect/docs/selectors/chooser.rst @@ -0,0 +1,32 @@ +Chooser Selector +================ + +When pushing to try, there are a very large amount of builds and tests to choose from. Often too +many to remember, making it easy to forget a set of tasks which should otherwise have been run. + +This selector allows you to select tasks from a web interface that lists all the possible build and +test tasks and allows you to select them from a list. It is similar in concept to the old `try +syntax chooser`_ page, except that the values are dynamically generated using the :ref:`taskgraph<TaskCluster Task-Graph Generation>` as an +input. This ensures that it will never be out of date. + +To use: + +.. code-block:: shell + + $ mach try chooser + +This will spin up a local web server (using Flask) which serves the chooser app. After making your +selection, simply press ``Push`` and the rest will be handled from there. No need to copy/paste any +syntax strings or the like. + +You can run: + +.. code-block:: shell + + $ mach try chooser --full + +To generate the interface using the full :ref:`taskgraph<TaskCluster Task-Graph Generation>` instead. This will include tasks that don't run +on mozilla-central. + + +.. _try syntax chooser: https://mozilla-releng.net/trychooser diff --git a/tools/tryselect/docs/selectors/compare.rst b/tools/tryselect/docs/selectors/compare.rst new file mode 100644 index 0000000000..a87b263030 --- /dev/null +++ b/tools/tryselect/docs/selectors/compare.rst @@ -0,0 +1,17 @@ +Compare Selector +================ + +When this command runs it pushes two identical try jobs to treeherder. The first +job is on the current commit you are on, and the second one is a commit +specified in the command line arguments. This selector is aimed at helping +engineers test performance enhancements or resolve performance regressions. + +Currently the only way you can select jobs is through fuzzy but we are +planning on expanding to other choosing frameworks also. + +You pass the commit you want to compare against as a commit hash as either +``-cc`` or ``--compare-commit``, an example is show below + +.. code-block:: shell + + $ mach try compare --compare-commit <commit-hash> diff --git a/tools/tryselect/docs/selectors/empty.rst b/tools/tryselect/docs/selectors/empty.rst new file mode 100644 index 0000000000..c3ea61b9ce --- /dev/null +++ b/tools/tryselect/docs/selectors/empty.rst @@ -0,0 +1,21 @@ +Empty Selector +============== + +The ``mach try empty`` subcommand is very simple, it won't schedule any additional tasks. You'll +still see lint tasks and python-unittest tasks if applicable, this is due to a configuration option +in taskcluster. + +Other than those, your try run will be empty. You can use treeherder's ``Add new jobs`` feature to +selectively add additional tasks after the fact. + +.. note:: + + To use ``Add new jobs`` you'll need to be logged in and have commit access level 1, just as if + you were pushing to try. + +To do this: + + 1. Click the drop-down arrow at the top right of your commit. + 2. Select ``Add new jobs`` (it may take a couple seconds to load). + 3. Choose your desired tasks by clicking them one at a time. + 4. At the top of your commit, select ``Trigger New Jobs``. diff --git a/tools/tryselect/docs/selectors/fuzzy.rst b/tools/tryselect/docs/selectors/fuzzy.rst new file mode 100644 index 0000000000..d50f801eb0 --- /dev/null +++ b/tools/tryselect/docs/selectors/fuzzy.rst @@ -0,0 +1,371 @@ +Fuzzy Selector +============== + +The fuzzy selector uses a tool called `fzf`_. It allows you to filter down all of the task labels +from a terminal based UI and an intelligent fuzzy finding algorithm. If the ``fzf`` binary is not +installed, you'll be prompted to bootstrap it on first run. + + +Understanding the Interface +--------------------------- + +When you run ``mach try fuzzy`` an interface similar to the one below will open. This is `fzf`_. + +.. image:: fzf.png + +There's a lot to unpack here, so let's examine each component a bit more closely. + + A. The set of tasks that match the currently typed-out query. In the above image only tasks that + match the query ``'linux64 mochibrochr`` are displayed. + + B. The set of selected tasks. These are the tasks that will be scheduled once you hit ``Enter``. + In other words, if the task you want does not appear here, *it won't be scheduled*. + + C. Count information of the form ``x/y (z)``, where ``x`` is the number of tasks that match the + current query, ``y`` is the total number of tasks and ``z`` is the number of tasks you have + selected. + + D. The input bar for entering queries. As you type you'll notice the list of tasks in ``A`` + starts to update immediately. In the image above, the query ``'linux64 mochibrochr`` is entered. + Correspondingly only tasks matching that query are displayed. + +In general terms, you first find tasks on the left. Then you move them over to the right by +selecting them. Once you are satisfied with your selection, press ``Enter`` to push to try. + + +Selecting Tasks +--------------- + +There are few ways you can select tasks. If you are feeling a bit overwhelmed, it might be best to +stick with the mouse to start: + + 1. Enter a query (e.g ``mochitest``) to reduce the task list a little. + 2. Scroll up and look for the task(s) you want. + 3. ``Right-Click`` as many tasks as desired to select them. + 4. Optionally delete your query, go back to step 1) and repeat. + 5. Press ``Enter`` to push (or ``Esc`` to cancel). + +.. note:: + + Dependencies are automatically filled in, so you can select a test task without needing + to select the build it depends on. + +As you ``Right-Click``, notice that a little arrow appears to the left of the task label. This +indicates that the task is selected and exists in the preview pane to the right. + +Once you are a bit more comfortable with the interface, using the keyboard is much better at quickly +selecting tasks. Here are the main shortcuts you'll need: + +.. code-block:: text + + Ctrl-K / Up => Move cursor up + Ctrl-J / Down => Move cursor down + Tab => Select task + move cursor down + Shift-Tab => Select task + move cursor up + Ctrl-A => Select all currently filtered tasks + Ctrl-T => Toggle select all currently filtered tasks + Ctrl-D => De-select all selected tasks (both filtered and not) + Alt-Bspace => Clear query from input bar + Enter => Accept selection and exit + Ctrl-C / Esc => Cancel selection and exit + ? => Toggle preview pane + + +The process for selecting tasks is otherwise the same as for a mouse. A particularly fast and +powerful way to select tasks is to: + +.. code-block:: text + + Write a precise query => Ctrl-A => Alt-Bspace => Repeat + +As before, when you are satisfied with your selection press ``Enter`` and all the tasks in the +preview pane will be pushed to try. If you change your mind you can press ``Esc`` or ``Ctrl-C`` to +exit the interface without pushing anything. + +.. note:: + + Initially ``fzf`` will automatically select whichever task is under your cursor. This is a + convenience feature for the case where you are only selecting a single task. This feature will be + turned off as soon as you *lock in* a selection with ``Right-Click``, ``Tab`` or ``Ctrl-A``. + + +Writing Queries +--------------- + +Queries are built from a series of terms, each separated by a space. Terms are logically joined by +the AND operator. For example: + +.. code-block:: text + + 'windows 'mochitest + +This query has two terms, and is the equivalent of saying: Give me all the tasks that match both the +term ``'windows'`` and the term ``'mochitest'``. In other words, this query matches all Windows +mochitest tasks. + +The single quote prefix before each term tells ``fzf`` to use exact substring matches, so only tasks +that contain both the literal string ``windows`` AND the literal string ``mochitest`` will be +matched. + +Another thing to note is that the order of the terms makes no difference, so ``'windows 'mochitest`` +and ``'mochitest 'windows`` are equivalent. + + +Fuzzy terms +~~~~~~~~~~~ + +If a term is *not* prefixed with a single quote, that makes it a fuzzy term. This means the +characters in the term need to show up in order, but not in sequence. E.g the fuzzy term ``max`` +would match the string ``mozilla firefox`` (as first there is an ``m``, then an ``a`` and finally an +``x``), but not the string ``firefox by mozilla`` (since the ``x`` is now out of order). Here's a +less contrived example: + +.. code-block:: text + + wndws mchtst + +Like the query above, this one would also select all Windows mochitest tasks. But it will +additionally select: + +.. code-block:: text + + test-macosx1014-64-shippable/opt-talos-sessionrestore-many-windows-e10s + +This is because both sequences of letters (``wndws`` and ``mchtst``) independently appear in order +somewhere in this string (remember the order of the terms makes no difference). + +At first fuzzy terms may not seem very useful, but they are actually extremely powerful! Let's use +the term from the interface image above, ``'linux64 mochibrochr``, as an example. First, just notice +how in the image ``fzf`` highlights the characters that constitute the match in green. Next, notice +how typing ``mochibrochr`` can quickly get us all mochitest browser-chrome tasks. The power of fuzzy +terms is that you don't need to memorize the exact task labels you are looking for. Just start +typing something you think is vaguely correct and chances are you'll see the task you're looking for. + + +Term Modifiers +~~~~~~~~~~~~~~ + +The following modifiers can be applied to a search term: + +.. code-block:: text + + 'word => exact match (line must contain the literal string "word") + ^word => exact prefix match (line must start with literal "word") + word$ => exact suffix match (line must end with literal "word") + !word => exact negation match (line must not contain literal "word") + 'a | 'b => OR operator (joins two exact match operators together) + +For example: + +.. code-block:: text + + ^start 'exact | 'other !ignore fuzzy end$ + +would match the string: + +.. code-block:: text + + starting to bake isn't exactly fun, but pizza is yummy in the end + +.. note:: + + The best way to learn how to write queries is to run ``mach try fuzzy --no-push`` and play + around with all of these modifiers! + + +Specifying Queries on the Command Line +-------------------------------------- + +Sometimes it's more convenient to skip the interactive interface and specify a query on the command +line with ``-q/--query``. This is equivalent to opening the interface then typing: +``<query><ctrl-a><enter>``. + +For example: + +.. code-block:: shell + + # selects all mochitest tasks + $ mach try fuzzy --query "mochitest" + +You can pass in multiple queries at once and the results of each will be joined together: + +.. code-block:: shell + + # selects all mochitest and reftest tasks + $ mach try fuzzy -q "mochitest" -q "reftest" + +If instead you want the intersection of queries, you can pass in ``-x/--and``: + +.. code-block:: shell + + # selects all windows mochitest tasks + $ mach try fuzzy --and -q "mochitest" -q "windows" + + +Modifying Presets +~~~~~~~~~~~~~~~~~ + +:doc:`Presets <../presets>` make it easy to run a pre-determined set of tasks. But sometimes you +might not want to run that set exactly as is, you may only want to use the preset as a starting +point then add or remove tasks as needed. This can be accomplished with ``-q/--query`` or +``-i/--interactive``. + +Here are some examples of adding tasks to a preset: + +.. code-block:: shell + + # selects all perf tasks plus all mochitest-chrome tasks + $ mach try fuzzy --preset perf -q "mochitest-chrome" + + # adds tasks to the perf preset interactively + $ mach try fuzzy --preset perf -i + +Similarly, ``-x/--and`` can be used to filter down a preset by taking the intersection of the two +sets: + +.. code-block:: shell + + # limits perf tasks to windows only + $ mach try fuzzy --preset perf -xq "windows" + + # limits perf tasks interactively + $ mach try fuzzy --preset perf -xi + + +Shell Conflicts +~~~~~~~~~~~~~~~ + +Unfortunately ``fzf``'s query language uses some characters (namely ``'``, ``!`` and ``$``) that can +interfere with your shell when using ``-q/--query``. Below are some tips for how to type out a query +on the command line. + +The ``!`` character is typically used for history expansion. If you don't use this feature, the +easiest way to specify queries on the command line is to disable it: + +.. code-block:: shell + + # bash + $ set +H + $ ./mach try fuzzy -q "'foo !bar" + + # zsh + $ setopt no_banghist + $ ./mach try fuzzy -q "'foo !bar" + +If using ``bash``, add ``set +H`` to your ``~/.bashrc``, ``~/.bash_profile`` or equivalent. If using +``zsh``, add ``setopt no_banghist`` to your ``~/.zshrc`` or equivalent. + +If you don't want to disable history expansion, you can escape your queries like this: + +.. code-block:: shell + + # bash + $ ./mach try fuzzy -q $'\'foo !bar' + + # zsh + $ ./mach try fuzzy -q "'foo \!bar" + + +The third option is to use ``-e/--exact`` which reverses the behaviour of the ``'`` character (see +:ref:`additional-arguments` for more details). Using this flag means you won't need to escape the +``'`` character as often and allows you to run your queries like this: + +.. code-block:: shell + + # bash and zsh + $ ./mach try fuzzy -eq 'foo !bar' + +This method is only useful if you find you almost always prefix terms with ``'`` (and rarely use +fuzzy terms). Otherwise as soon as you want to use a fuzzy match you'll run into the same problem as +before. + +.. note:: All the examples in these three approaches will select the same set of tasks. + +If you use ``fish`` shell, you won't need to escape ``!``, however you will need to escape ``$``: + +.. code-block:: shell + + # fish + $ ./mach try fuzzy -q "'foo !bar baz\$" + + +Test Paths +---------- + +One or more paths to a file or directory may be specified as positional arguments. When +specifying paths, the list of available tasks to choose from is filtered down such that +only suites that have tests in a specified path can be selected. Notably, only the first +chunk of each suite/platform appears. When the tasks are scheduled, only tests that live +under one of the specified paths will be run. + +.. note:: + + When using paths, be aware that all tests under the specified paths will run in the + same chunk. This might produce a different ordering from what gets run on production + branches, and may yield different results. + + For suites that restart the browser between each manifest (like mochitest), this + shouldn't be as big of a concern. + +Paths can be used with the interactive ``fzf`` window, or using the ``-q/--query`` argument. +For example, running: + +.. code-block:: shell + + $ mach try fuzzy layout/reftests/reftest-sanity -q "!pgo !cov !asan 'linux64" + +Would produce the following ``try_task_config.json``: + +.. code-block:: json + + { + "env":{ + "MOZHARNESS_TEST_PATHS":"{\"reftest\":\"layout/reftests/reftest-sanity\"}" + }, + "tasks":[ + "test-linux64-qr/debug-reftest-e10s-1", + "test-linux64-qr/opt-reftest-e10s-1", + "test-linux64/debug-reftest-e10s-1", + "test-linux64/debug-reftest-no-accel-e10s-1", + "test-linux64/opt-reftest-e10s-1", + "test-linux64/opt-reftest-no-accel-e10s-1", + ] + } + +Inside of these tasks, the reftest harness will only run tests that live under +``layout/reftests/reftest-sanity``. + + +.. _additional-arguments: + +Additional Arguments +-------------------- + +There are a few additional command line arguments you may wish to use: + +``-e/--exact`` +By default, ``fzf`` treats terms as a fuzzy match and prefixing a term with ``'`` turns it into an exact +match. If passing in ``--exact``, this behaviour is reversed. Non-prefixed terms become exact, and a +``'`` prefix makes a term fuzzy. + +``--full`` +By default, only target tasks (e.g tasks that would normally run on mozilla-central) +are generated. Passing in ``--full`` allows you to select from all tasks. This is useful for +things like nightly or release tasks. + +``-u/--update`` +Update the bootstrapped ``fzf`` binary to the latest version. + +For a full list of command line arguments, run: + +.. code-block:: shell + + $ mach try fuzzy --help + +For more information on using ``fzf``, run: + +.. code-block:: shell + + $ man fzf + +.. _fzf: https://github.com/junegunn/fzf diff --git a/tools/tryselect/docs/selectors/fzf.png b/tools/tryselect/docs/selectors/fzf.png Binary files differnew file mode 100644 index 0000000000..a64f4b04f3 --- /dev/null +++ b/tools/tryselect/docs/selectors/fzf.png diff --git a/tools/tryselect/docs/selectors/index.rst b/tools/tryselect/docs/selectors/index.rst new file mode 100644 index 0000000000..be618202d6 --- /dev/null +++ b/tools/tryselect/docs/selectors/index.rst @@ -0,0 +1,44 @@ +Selectors +========= + +These are the currently implemented try selectors: + +* :doc:`auto <auto>`: Have tasks chosen for you automatically. +* :doc:`fuzzy <fuzzy>`: Select tasks using a fuzzy finding algorithm and + a terminal interface. +* :doc:`chooser <chooser>`: Select tasks using a web interface. +* :doc:`again <again>`: Re-run a previous ``try_task_config.json`` based + push. +* :doc:`empty <empty>`: Don't select any tasks. Taskcluster will still run + some tasks automatically (like lint and python unittest tasks). Further tasks + can be chosen with treeherder's ``Add New Jobs`` feature. +* :doc:`syntax <syntax>`: Select tasks using classic try syntax. +* :doc:`release <release>`: Prepare a tree for doing a staging release. +* :doc:`scriptworker <scriptworker>`: Run scriptworker tasks against a recent release. +* :doc:`compare <compare>`: Push two identical try jobs, one on your current commit and another of your choice + +You can run them with: + +.. code-block:: shell + + $ mach try <selector> + +See selector specific options by running: + +.. code-block:: shell + + $ mach try <selector> --help + +.. toctree:: + :caption: Available Selectors + :maxdepth: 1 + :hidden: + + Auto <auto> + Fuzzy <fuzzy> + Chooser <chooser> + Again <again> + Empty <empty> + Syntax <syntax> + Release <release> + Scriptworker <scriptworker> diff --git a/tools/tryselect/docs/selectors/release.rst b/tools/tryselect/docs/selectors/release.rst new file mode 100644 index 0000000000..946266249b --- /dev/null +++ b/tools/tryselect/docs/selectors/release.rst @@ -0,0 +1,31 @@ +Release Selector +================ + +This command configures the tree in preparation for doing a staging release, +and pushes the result to try. The changes that that are made include: + +- Updating the version number. +- Applying the migrations that are done as part of merge day. +- Disabling repacking most locales. (This can be disabled by passing ``--no-limit-locales``). + +For staging a beta release, run the following (with an appropriate version number): + +.. code-block:: shell + + $ mach try release --version 64.0b5 --migration central-to-beta + +For staging a final release (rc or patch), run the following (with an appropriate version number) + +.. code-block:: shell + + $ mach try release --version 64.0 --migration central-to-beta --migration beta-to-release + +Once the decision task is on the push is complete, you can start the release +through `staging ship-it instance <https://shipit.staging.mozilla-releng.net/new>`_\ [#shipit]_. + +.. note:: + + If pushing from beta or release, the corresponding migration should not be + passed, as they have already been applied. + +.. [#shipit] This is only available to release engineering and release management (as of 2018-10-15). diff --git a/tools/tryselect/docs/selectors/scriptworker.rst b/tools/tryselect/docs/selectors/scriptworker.rst new file mode 100644 index 0000000000..a3cba08cbe --- /dev/null +++ b/tools/tryselect/docs/selectors/scriptworker.rst @@ -0,0 +1,31 @@ +Scriptworker Selector +===================== + +This command runs a selection of scriptworker tasks against builds from a +recent release. This is aimed at release engineering, to test changes to +scriptworker implementations. It currently requires being connected to +Mozilla's internal datacenter VPN with access to shipit\ [#shipit]_. + +There are a number of preset groups of tasks to run. To run a particular +set of tasks, pass the name of the set to ``mach try scriptworker``: + +.. code-block:: shell + + $ mach try scriptworker linux-signing + +To get the list of task sets, along with the list of tasks they will run: + +.. code-block:: shell + + $ mach try scriptworker list + +The selector defaults to using tasks from the most recent beta, to use tasks +from a different release, pass ``--release-type <release-type>``: + +.. code-block:: shell + + $ mach try scriptworker --release-type release linux-signing + + +.. [#shipit] The shipit API is not currently publicly available, and is used + to find the release graph to pull previous tasks from. diff --git a/tools/tryselect/docs/selectors/syntax.rst b/tools/tryselect/docs/selectors/syntax.rst new file mode 100644 index 0000000000..b64bb65ab7 --- /dev/null +++ b/tools/tryselect/docs/selectors/syntax.rst @@ -0,0 +1,41 @@ +Syntax Selector +=============== + +.. warning:: + + Try syntax is antiquated and hard to understand. If you aren't already + familiar with try syntax, you might want to use the ``fuzzy`` selector + instead. + +Try syntax is a command line string that goes into the commit message. Using +``mach try syntax`` will automatically create a temporary commit with your +chosen syntax and then delete it again after pushing to try. + +Try syntax can contain all kinds of different options parsed by various +places in various repos, but the majority are parsed by `try_option_syntax.py`_. +The most common arguments include: + + * ``-b/--build`` - One of ``d``, ``o`` or ``do``. This is the build type, + either opt, debug or both (required). + * ``-p/--platform`` - The platforms you want to build and/or run tests on + (required). + * ``-u/--unittests`` - The test tasks you want to run (optional). + * ``-t/--talos`` - The talos tasks you want to run (optional). + +Here are some examples: + +.. code-block:: shell + + $ mach try syntax -b do -p linux,macosx64 -u mochitest-e10s-1,crashtest -t none + $ mach try syntax -b d -p win64 -u all + $ mach try syntax -b o -p linux64 + +Unfortunately, knowing the magic strings that make it work can be a bit of a +guessing game. If you are unsure of what string will get your task to run, try +using :doc:`mach try fuzzy <fuzzy>` instead. + +While using ``mach try syntax -b do -p all -u all -t all`` will work, heavy use +of ``all`` is discouraged as it consumes a lot of unnecessary resources (some of +which are hardware constrained). + +.. _try_option_syntax.py: https://searchfox.org/mozilla-central/source/taskcluster/gecko_taskgraph/try_option_syntax.py |