diff options
Diffstat (limited to 'tools/tryselect/docs')
-rw-r--r-- | tools/tryselect/docs/configuration.rst | 30 | ||||
-rw-r--r-- | tools/tryselect/docs/img/add-new-jobs.png | bin | 0 -> 33561 bytes | |||
-rw-r--r-- | tools/tryselect/docs/img/phab-treeherder-link.png | bin | 0 -> 35922 bytes | |||
-rw-r--r-- | tools/tryselect/docs/index.rst | 92 | ||||
-rw-r--r-- | tools/tryselect/docs/presets.rst | 85 | ||||
-rw-r--r-- | tools/tryselect/docs/selectors/again.rst | 36 | ||||
-rw-r--r-- | tools/tryselect/docs/selectors/auto.rst | 24 | ||||
-rw-r--r-- | tools/tryselect/docs/selectors/chooser.rst | 32 | ||||
-rw-r--r-- | tools/tryselect/docs/selectors/compare.rst | 17 | ||||
-rw-r--r-- | tools/tryselect/docs/selectors/empty.rst | 21 | ||||
-rw-r--r-- | tools/tryselect/docs/selectors/fuzzy.rst | 371 | ||||
-rw-r--r-- | tools/tryselect/docs/selectors/fzf.png | bin | 0 -> 44467 bytes | |||
-rw-r--r-- | tools/tryselect/docs/selectors/index.rst | 44 | ||||
-rw-r--r-- | tools/tryselect/docs/selectors/release.rst | 31 | ||||
-rw-r--r-- | tools/tryselect/docs/selectors/scriptworker.rst | 31 | ||||
-rw-r--r-- | tools/tryselect/docs/selectors/syntax.rst | 41 | ||||
-rw-r--r-- | tools/tryselect/docs/tasks.rst | 152 |
17 files changed, 1007 insertions, 0 deletions
diff --git a/tools/tryselect/docs/configuration.rst b/tools/tryselect/docs/configuration.rst new file mode 100644 index 0000000000..6743d5f385 --- /dev/null +++ b/tools/tryselect/docs/configuration.rst @@ -0,0 +1,30 @@ +Configuring Try +=============== + + +Getting Level 1 Commit Access +----------------------------- + +In order to push to try, `Level 1 Commit Access`_ is required. Please see `Becoming a Mozilla +Committer`_ for more information on how to obtain this. + + +Configuring Version Control +--------------------------- + +After you have level 1 access, you'll need to do a little bit of setup before you can push. Both +``hg`` and ``git`` are supported, move on to the appropriate section below. + + +Configuring Try with Mercurial / Git +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +The recommended way to push to try is via the ``mach try`` command. This requires the +``push-to-try`` extension which can be installed by running: + +.. code-block:: shell + + $ mach vcs-setup + +.. _Level 1 Commit Access: https://www.mozilla.org/en-US/about/governance/policies/commit/access-policy/ +.. _Becoming a Mozilla Committer: https://www.mozilla.org/en-US/about/governance/policies/commit/ diff --git a/tools/tryselect/docs/img/add-new-jobs.png b/tools/tryselect/docs/img/add-new-jobs.png Binary files differnew file mode 100644 index 0000000000..513565cb14 --- /dev/null +++ b/tools/tryselect/docs/img/add-new-jobs.png diff --git a/tools/tryselect/docs/img/phab-treeherder-link.png b/tools/tryselect/docs/img/phab-treeherder-link.png Binary files differnew file mode 100644 index 0000000000..52b58b6231 --- /dev/null +++ b/tools/tryselect/docs/img/phab-treeherder-link.png diff --git a/tools/tryselect/docs/index.rst b/tools/tryselect/docs/index.rst new file mode 100644 index 0000000000..8978a72eaf --- /dev/null +++ b/tools/tryselect/docs/index.rst @@ -0,0 +1,92 @@ +Pushing to Try +============== + +"Pushing to Try" allows developers to build and test their changes on Mozilla's automation servers +without requiring their code to be reviewed and landed. + +First, :doc:`ensure that you can push to Try <configuration>`. +Try knows how to run tasks that are defined in-tree, +such as ``build-linux64/opt`` (build Firefox for Linux). To manually select some tasks for +Try to process, run the following command: + +.. code-block:: shell + + ./mach try fuzzy + +After submitting your requested tasks, you'll be given a link to your "push" in Treeherder. +It may take a few minutes for your push to appear in Treeherder! Be patient, and it will automatically +update when Try begins processing your work. + +Another very useful Try command is ``./mach try auto``, which will automatically select the tasks +that are mostly likely to be affected by your changes. +See the :doc:`selectors page <selectors/index>` to view all the other ways to select which tasks to push. + +It is possible to set environment variables, notably :doc:`MOZ_LOG </xpcom/logging>`, when pushing to +try: + +.. code-block:: shell + + ./mach try fuzzy --env MOZ_LOG=cubeb:4 + +Resolving "<Try build> is damaged and can't be opened" error +------------------------------------------------------------ + +To run a try build on macOS, you need to get around Apple's restrictions on downloaded applications. + +These restrictions differ based on your hardware: Apple Silicon machines (M1 etc.) are much stricter. + +For Apple Silicon machines, you will need to download the target.dmg artifact from the +"repackage-macosx64-shippable/opt" job. +This is a universal build (i.e. it contains both x86_64 and arm64 code), and it is signed but not notarized. +You can trigger this job using ``./mach try fuzzy --full``. + +On Intel Macs, you can run unsigned builds, once you get around the quarantining (see below), +so you can just download the "target.dmg" from a regular opt build. + +Regardless of hardware, you need to make sure that there is no quarantining attribute on +the downloaded dmg file before you attempt to run it: +Apple automatically quarantines apps that are downloaded with a browser from an untrusted +location. This "quarantine status" can be cleared by doing ``xattr -c <Try build>`` after +downloading. You can avoid this "quarantine status" by downloading the build from the command +line instead, such as by using ``curl``: + +.. code-block:: shell + + curl -L <artifact-url> -o <file-name> + +.. _attach-job-review: + +Adding Try jobs to a Phabricator patch +-------------------------------------- + +For every patch submitted for review in Phabricator, a new Try run is automatically created. +A link called ``Treeherder Jobs`` can be found in the ``Diff Detail`` section of the review in +Phabricator. + +.. image:: img/phab-treeherder-link.png + +This run is created for static analysis, linting and other tasks. Attaching new jobs to the run is +easy and doesn't require more actions from the developer. +Click on the down-arrow to access the actions menu, select the relevant jobs +and, click on ``Trigger X new jobs`` (located on the top of the job). + +.. image:: img/add-new-jobs.png + +Table of Contents +----------------- + +.. toctree:: + :maxdepth: 2 + + configuration + selectors/index + presets + tasks + + +Indices and tables +------------------ + +* :ref:`genindex` +* :ref:`modindex` +* :ref:`search` diff --git a/tools/tryselect/docs/presets.rst b/tools/tryselect/docs/presets.rst new file mode 100644 index 0000000000..a3368cf8b5 --- /dev/null +++ b/tools/tryselect/docs/presets.rst @@ -0,0 +1,85 @@ +Presets +======= + +Some selectors, such as ``fuzzy`` and ``syntax``, allow saving and loading presets from a file. This is a +good way to reuse a selection, either at a later date or by sharing with others. Look for a +'preset' section in ``mach <selector> --help`` to determine whether the selector supports this +functionality. + +Using Presets +------------- + +To save a preset, run: + +.. code-block:: shell + + $ mach try <selector> --save <name> <args> + +For example, to save a preset that selects all Windows mochitests: + +.. code-block:: shell + + $ mach try fuzzy --save all-windows-mochitests --query "'win 'mochitest" + preset saved, run with: --preset=all-windows-mochitests + +Then run that saved preset like this: + +.. code-block:: shell + + $ mach try --preset all-windows-mochitests + +To see a list of all available presets run: + +.. code-block:: shell + + $ mach try --list-presets + + +Editing and Sharing Presets +--------------------------- + +Presets can be defined in one of two places, in your home directory or in a file checked into +mozilla-central. + +Local Presets +~~~~~~~~~~~~~ + +These are defined in your ``$MOZBUILD_STATE_DIR``, typically ``~/.mozbuild/try_presets.yml``. +Presets defined here are your own personal collection of presets. You can modify them by running: + +.. code-block:: shell + + $ ./mach try --edit-presets + + +Shared Presets +~~~~~~~~~~~~~~ + +You can also check presets into mozilla-central in `tools/tryselect/try_presets.yml`_. These presets +will be available to all users of ``mach try``, so please be mindful when editing this file. Make +sure the name of the preset is scoped appropriately (i.e doesn't contain any team or module specific +terminology). It is good practice to prefix the preset name with the name of the team or module that +will get the most use out of it. + + +Preset Format +~~~~~~~~~~~~~ + +Presets are simple key/value objects, with the name as the key and a metadata object as the value. +For example, the preset saved above would look something like this in ``try_presets.yml``: + +.. code-block:: yaml + + all-windows-mochitests: + selector: fuzzy + description: >- + Runs all windows mochitests. + query: + - "'win 'mochitest" + +The ``selector`` key (required) allows ``mach try`` to determine which subcommand to dispatch to. +The ``description`` key (optional in user presets but required for shared presets) is a human +readable string describing what the preset selects and when to use it. All other values in the +preset are forwarded to the specified selector as is. + +.. _tools/tryselect/try_presets.yml: https://searchfox.org/mozilla-central/source/tools/tryselect/try_presets.yml diff --git a/tools/tryselect/docs/selectors/again.rst b/tools/tryselect/docs/selectors/again.rst new file mode 100644 index 0000000000..da592b7a34 --- /dev/null +++ b/tools/tryselect/docs/selectors/again.rst @@ -0,0 +1,36 @@ +Again Selector +============== + +When you push to try, the computed ``try_task_config.json`` is saved in a +history file under ``~/.mozbuild/srcdirs/<srcdir hash>/history`` (note: the +``syntax`` selector does not use ``try_task_config.json`` yet so does not save +any history). You can then use the ``again`` selector to re-push any of your +previously generated task configs. + +In the simple case, you can re-run your last try push with: + +.. code-block:: shell + + $ mach try again + +If you want to re-push a task config a little further down the history stack, +first you need to figure out its index with: + +.. code-block:: shell + + $ mach try again --list + +Then run: + +.. code-block:: shell + + $ mach try again --index <index> + +Note that index ``0`` is always the most recent ``try_task_config.json`` in the +history. You can clear your history with: + +.. code-block:: shell + + $ mach try again --purge + +Only the 10 most recent pushes will be saved in your history. diff --git a/tools/tryselect/docs/selectors/auto.rst b/tools/tryselect/docs/selectors/auto.rst new file mode 100644 index 0000000000..c5f6d4df5e --- /dev/null +++ b/tools/tryselect/docs/selectors/auto.rst @@ -0,0 +1,24 @@ +Auto Selector +============= + +This selector automatically determines the most efficient set of tests and +tasks to run against your push. It accomplishes this via a combination of +machine learning and manual heuristics. The tasks selected here should match +pretty closely to what would be scheduled if your patch were pushed to +autoland. + +It is the officially recommended selector to use when you are unsure of which +tasks should run on your push. + +To use: + +.. code-block:: bash + + $ mach try auto + +Unlike other try selectors, tasks are not chosen locally. Rather they will be +computed by the decision task. + +Like most other selectors, ``mach try auto`` supports many of the standard +templates such as ``--artifact`` or ``--env``. See ``mach try auto --help`` for +the full list of supported templates. diff --git a/tools/tryselect/docs/selectors/chooser.rst b/tools/tryselect/docs/selectors/chooser.rst new file mode 100644 index 0000000000..6c59d54009 --- /dev/null +++ b/tools/tryselect/docs/selectors/chooser.rst @@ -0,0 +1,32 @@ +Chooser Selector +================ + +When pushing to try, there are a very large amount of builds and tests to choose from. Often too +many to remember, making it easy to forget a set of tasks which should otherwise have been run. + +This selector allows you to select tasks from a web interface that lists all the possible build and +test tasks and allows you to select them from a list. It is similar in concept to the old `try +syntax chooser`_ page, except that the values are dynamically generated using the :ref:`taskgraph<TaskCluster Task-Graph Generation>` as an +input. This ensures that it will never be out of date. + +To use: + +.. code-block:: shell + + $ mach try chooser + +This will spin up a local web server (using Flask) which serves the chooser app. After making your +selection, simply press ``Push`` and the rest will be handled from there. No need to copy/paste any +syntax strings or the like. + +You can run: + +.. code-block:: shell + + $ mach try chooser --full + +To generate the interface using the full :ref:`taskgraph<TaskCluster Task-Graph Generation>` instead. This will include tasks that don't run +on mozilla-central. + + +.. _try syntax chooser: https://mozilla-releng.net/trychooser diff --git a/tools/tryselect/docs/selectors/compare.rst b/tools/tryselect/docs/selectors/compare.rst new file mode 100644 index 0000000000..a87b263030 --- /dev/null +++ b/tools/tryselect/docs/selectors/compare.rst @@ -0,0 +1,17 @@ +Compare Selector +================ + +When this command runs it pushes two identical try jobs to treeherder. The first +job is on the current commit you are on, and the second one is a commit +specified in the command line arguments. This selector is aimed at helping +engineers test performance enhancements or resolve performance regressions. + +Currently the only way you can select jobs is through fuzzy but we are +planning on expanding to other choosing frameworks also. + +You pass the commit you want to compare against as a commit hash as either +``-cc`` or ``--compare-commit``, an example is show below + +.. code-block:: shell + + $ mach try compare --compare-commit <commit-hash> diff --git a/tools/tryselect/docs/selectors/empty.rst b/tools/tryselect/docs/selectors/empty.rst new file mode 100644 index 0000000000..c3ea61b9ce --- /dev/null +++ b/tools/tryselect/docs/selectors/empty.rst @@ -0,0 +1,21 @@ +Empty Selector +============== + +The ``mach try empty`` subcommand is very simple, it won't schedule any additional tasks. You'll +still see lint tasks and python-unittest tasks if applicable, this is due to a configuration option +in taskcluster. + +Other than those, your try run will be empty. You can use treeherder's ``Add new jobs`` feature to +selectively add additional tasks after the fact. + +.. note:: + + To use ``Add new jobs`` you'll need to be logged in and have commit access level 1, just as if + you were pushing to try. + +To do this: + + 1. Click the drop-down arrow at the top right of your commit. + 2. Select ``Add new jobs`` (it may take a couple seconds to load). + 3. Choose your desired tasks by clicking them one at a time. + 4. At the top of your commit, select ``Trigger New Jobs``. diff --git a/tools/tryselect/docs/selectors/fuzzy.rst b/tools/tryselect/docs/selectors/fuzzy.rst new file mode 100644 index 0000000000..d50f801eb0 --- /dev/null +++ b/tools/tryselect/docs/selectors/fuzzy.rst @@ -0,0 +1,371 @@ +Fuzzy Selector +============== + +The fuzzy selector uses a tool called `fzf`_. It allows you to filter down all of the task labels +from a terminal based UI and an intelligent fuzzy finding algorithm. If the ``fzf`` binary is not +installed, you'll be prompted to bootstrap it on first run. + + +Understanding the Interface +--------------------------- + +When you run ``mach try fuzzy`` an interface similar to the one below will open. This is `fzf`_. + +.. image:: fzf.png + +There's a lot to unpack here, so let's examine each component a bit more closely. + + A. The set of tasks that match the currently typed-out query. In the above image only tasks that + match the query ``'linux64 mochibrochr`` are displayed. + + B. The set of selected tasks. These are the tasks that will be scheduled once you hit ``Enter``. + In other words, if the task you want does not appear here, *it won't be scheduled*. + + C. Count information of the form ``x/y (z)``, where ``x`` is the number of tasks that match the + current query, ``y`` is the total number of tasks and ``z`` is the number of tasks you have + selected. + + D. The input bar for entering queries. As you type you'll notice the list of tasks in ``A`` + starts to update immediately. In the image above, the query ``'linux64 mochibrochr`` is entered. + Correspondingly only tasks matching that query are displayed. + +In general terms, you first find tasks on the left. Then you move them over to the right by +selecting them. Once you are satisfied with your selection, press ``Enter`` to push to try. + + +Selecting Tasks +--------------- + +There are few ways you can select tasks. If you are feeling a bit overwhelmed, it might be best to +stick with the mouse to start: + + 1. Enter a query (e.g ``mochitest``) to reduce the task list a little. + 2. Scroll up and look for the task(s) you want. + 3. ``Right-Click`` as many tasks as desired to select them. + 4. Optionally delete your query, go back to step 1) and repeat. + 5. Press ``Enter`` to push (or ``Esc`` to cancel). + +.. note:: + + Dependencies are automatically filled in, so you can select a test task without needing + to select the build it depends on. + +As you ``Right-Click``, notice that a little arrow appears to the left of the task label. This +indicates that the task is selected and exists in the preview pane to the right. + +Once you are a bit more comfortable with the interface, using the keyboard is much better at quickly +selecting tasks. Here are the main shortcuts you'll need: + +.. code-block:: text + + Ctrl-K / Up => Move cursor up + Ctrl-J / Down => Move cursor down + Tab => Select task + move cursor down + Shift-Tab => Select task + move cursor up + Ctrl-A => Select all currently filtered tasks + Ctrl-T => Toggle select all currently filtered tasks + Ctrl-D => De-select all selected tasks (both filtered and not) + Alt-Bspace => Clear query from input bar + Enter => Accept selection and exit + Ctrl-C / Esc => Cancel selection and exit + ? => Toggle preview pane + + +The process for selecting tasks is otherwise the same as for a mouse. A particularly fast and +powerful way to select tasks is to: + +.. code-block:: text + + Write a precise query => Ctrl-A => Alt-Bspace => Repeat + +As before, when you are satisfied with your selection press ``Enter`` and all the tasks in the +preview pane will be pushed to try. If you change your mind you can press ``Esc`` or ``Ctrl-C`` to +exit the interface without pushing anything. + +.. note:: + + Initially ``fzf`` will automatically select whichever task is under your cursor. This is a + convenience feature for the case where you are only selecting a single task. This feature will be + turned off as soon as you *lock in* a selection with ``Right-Click``, ``Tab`` or ``Ctrl-A``. + + +Writing Queries +--------------- + +Queries are built from a series of terms, each separated by a space. Terms are logically joined by +the AND operator. For example: + +.. code-block:: text + + 'windows 'mochitest + +This query has two terms, and is the equivalent of saying: Give me all the tasks that match both the +term ``'windows'`` and the term ``'mochitest'``. In other words, this query matches all Windows +mochitest tasks. + +The single quote prefix before each term tells ``fzf`` to use exact substring matches, so only tasks +that contain both the literal string ``windows`` AND the literal string ``mochitest`` will be +matched. + +Another thing to note is that the order of the terms makes no difference, so ``'windows 'mochitest`` +and ``'mochitest 'windows`` are equivalent. + + +Fuzzy terms +~~~~~~~~~~~ + +If a term is *not* prefixed with a single quote, that makes it a fuzzy term. This means the +characters in the term need to show up in order, but not in sequence. E.g the fuzzy term ``max`` +would match the string ``mozilla firefox`` (as first there is an ``m``, then an ``a`` and finally an +``x``), but not the string ``firefox by mozilla`` (since the ``x`` is now out of order). Here's a +less contrived example: + +.. code-block:: text + + wndws mchtst + +Like the query above, this one would also select all Windows mochitest tasks. But it will +additionally select: + +.. code-block:: text + + test-macosx1014-64-shippable/opt-talos-sessionrestore-many-windows-e10s + +This is because both sequences of letters (``wndws`` and ``mchtst``) independently appear in order +somewhere in this string (remember the order of the terms makes no difference). + +At first fuzzy terms may not seem very useful, but they are actually extremely powerful! Let's use +the term from the interface image above, ``'linux64 mochibrochr``, as an example. First, just notice +how in the image ``fzf`` highlights the characters that constitute the match in green. Next, notice +how typing ``mochibrochr`` can quickly get us all mochitest browser-chrome tasks. The power of fuzzy +terms is that you don't need to memorize the exact task labels you are looking for. Just start +typing something you think is vaguely correct and chances are you'll see the task you're looking for. + + +Term Modifiers +~~~~~~~~~~~~~~ + +The following modifiers can be applied to a search term: + +.. code-block:: text + + 'word => exact match (line must contain the literal string "word") + ^word => exact prefix match (line must start with literal "word") + word$ => exact suffix match (line must end with literal "word") + !word => exact negation match (line must not contain literal "word") + 'a | 'b => OR operator (joins two exact match operators together) + +For example: + +.. code-block:: text + + ^start 'exact | 'other !ignore fuzzy end$ + +would match the string: + +.. code-block:: text + + starting to bake isn't exactly fun, but pizza is yummy in the end + +.. note:: + + The best way to learn how to write queries is to run ``mach try fuzzy --no-push`` and play + around with all of these modifiers! + + +Specifying Queries on the Command Line +-------------------------------------- + +Sometimes it's more convenient to skip the interactive interface and specify a query on the command +line with ``-q/--query``. This is equivalent to opening the interface then typing: +``<query><ctrl-a><enter>``. + +For example: + +.. code-block:: shell + + # selects all mochitest tasks + $ mach try fuzzy --query "mochitest" + +You can pass in multiple queries at once and the results of each will be joined together: + +.. code-block:: shell + + # selects all mochitest and reftest tasks + $ mach try fuzzy -q "mochitest" -q "reftest" + +If instead you want the intersection of queries, you can pass in ``-x/--and``: + +.. code-block:: shell + + # selects all windows mochitest tasks + $ mach try fuzzy --and -q "mochitest" -q "windows" + + +Modifying Presets +~~~~~~~~~~~~~~~~~ + +:doc:`Presets <../presets>` make it easy to run a pre-determined set of tasks. But sometimes you +might not want to run that set exactly as is, you may only want to use the preset as a starting +point then add or remove tasks as needed. This can be accomplished with ``-q/--query`` or +``-i/--interactive``. + +Here are some examples of adding tasks to a preset: + +.. code-block:: shell + + # selects all perf tasks plus all mochitest-chrome tasks + $ mach try fuzzy --preset perf -q "mochitest-chrome" + + # adds tasks to the perf preset interactively + $ mach try fuzzy --preset perf -i + +Similarly, ``-x/--and`` can be used to filter down a preset by taking the intersection of the two +sets: + +.. code-block:: shell + + # limits perf tasks to windows only + $ mach try fuzzy --preset perf -xq "windows" + + # limits perf tasks interactively + $ mach try fuzzy --preset perf -xi + + +Shell Conflicts +~~~~~~~~~~~~~~~ + +Unfortunately ``fzf``'s query language uses some characters (namely ``'``, ``!`` and ``$``) that can +interfere with your shell when using ``-q/--query``. Below are some tips for how to type out a query +on the command line. + +The ``!`` character is typically used for history expansion. If you don't use this feature, the +easiest way to specify queries on the command line is to disable it: + +.. code-block:: shell + + # bash + $ set +H + $ ./mach try fuzzy -q "'foo !bar" + + # zsh + $ setopt no_banghist + $ ./mach try fuzzy -q "'foo !bar" + +If using ``bash``, add ``set +H`` to your ``~/.bashrc``, ``~/.bash_profile`` or equivalent. If using +``zsh``, add ``setopt no_banghist`` to your ``~/.zshrc`` or equivalent. + +If you don't want to disable history expansion, you can escape your queries like this: + +.. code-block:: shell + + # bash + $ ./mach try fuzzy -q $'\'foo !bar' + + # zsh + $ ./mach try fuzzy -q "'foo \!bar" + + +The third option is to use ``-e/--exact`` which reverses the behaviour of the ``'`` character (see +:ref:`additional-arguments` for more details). Using this flag means you won't need to escape the +``'`` character as often and allows you to run your queries like this: + +.. code-block:: shell + + # bash and zsh + $ ./mach try fuzzy -eq 'foo !bar' + +This method is only useful if you find you almost always prefix terms with ``'`` (and rarely use +fuzzy terms). Otherwise as soon as you want to use a fuzzy match you'll run into the same problem as +before. + +.. note:: All the examples in these three approaches will select the same set of tasks. + +If you use ``fish`` shell, you won't need to escape ``!``, however you will need to escape ``$``: + +.. code-block:: shell + + # fish + $ ./mach try fuzzy -q "'foo !bar baz\$" + + +Test Paths +---------- + +One or more paths to a file or directory may be specified as positional arguments. When +specifying paths, the list of available tasks to choose from is filtered down such that +only suites that have tests in a specified path can be selected. Notably, only the first +chunk of each suite/platform appears. When the tasks are scheduled, only tests that live +under one of the specified paths will be run. + +.. note:: + + When using paths, be aware that all tests under the specified paths will run in the + same chunk. This might produce a different ordering from what gets run on production + branches, and may yield different results. + + For suites that restart the browser between each manifest (like mochitest), this + shouldn't be as big of a concern. + +Paths can be used with the interactive ``fzf`` window, or using the ``-q/--query`` argument. +For example, running: + +.. code-block:: shell + + $ mach try fuzzy layout/reftests/reftest-sanity -q "!pgo !cov !asan 'linux64" + +Would produce the following ``try_task_config.json``: + +.. code-block:: json + + { + "env":{ + "MOZHARNESS_TEST_PATHS":"{\"reftest\":\"layout/reftests/reftest-sanity\"}" + }, + "tasks":[ + "test-linux64-qr/debug-reftest-e10s-1", + "test-linux64-qr/opt-reftest-e10s-1", + "test-linux64/debug-reftest-e10s-1", + "test-linux64/debug-reftest-no-accel-e10s-1", + "test-linux64/opt-reftest-e10s-1", + "test-linux64/opt-reftest-no-accel-e10s-1", + ] + } + +Inside of these tasks, the reftest harness will only run tests that live under +``layout/reftests/reftest-sanity``. + + +.. _additional-arguments: + +Additional Arguments +-------------------- + +There are a few additional command line arguments you may wish to use: + +``-e/--exact`` +By default, ``fzf`` treats terms as a fuzzy match and prefixing a term with ``'`` turns it into an exact +match. If passing in ``--exact``, this behaviour is reversed. Non-prefixed terms become exact, and a +``'`` prefix makes a term fuzzy. + +``--full`` +By default, only target tasks (e.g tasks that would normally run on mozilla-central) +are generated. Passing in ``--full`` allows you to select from all tasks. This is useful for +things like nightly or release tasks. + +``-u/--update`` +Update the bootstrapped ``fzf`` binary to the latest version. + +For a full list of command line arguments, run: + +.. code-block:: shell + + $ mach try fuzzy --help + +For more information on using ``fzf``, run: + +.. code-block:: shell + + $ man fzf + +.. _fzf: https://github.com/junegunn/fzf diff --git a/tools/tryselect/docs/selectors/fzf.png b/tools/tryselect/docs/selectors/fzf.png Binary files differnew file mode 100644 index 0000000000..a64f4b04f3 --- /dev/null +++ b/tools/tryselect/docs/selectors/fzf.png diff --git a/tools/tryselect/docs/selectors/index.rst b/tools/tryselect/docs/selectors/index.rst new file mode 100644 index 0000000000..be618202d6 --- /dev/null +++ b/tools/tryselect/docs/selectors/index.rst @@ -0,0 +1,44 @@ +Selectors +========= + +These are the currently implemented try selectors: + +* :doc:`auto <auto>`: Have tasks chosen for you automatically. +* :doc:`fuzzy <fuzzy>`: Select tasks using a fuzzy finding algorithm and + a terminal interface. +* :doc:`chooser <chooser>`: Select tasks using a web interface. +* :doc:`again <again>`: Re-run a previous ``try_task_config.json`` based + push. +* :doc:`empty <empty>`: Don't select any tasks. Taskcluster will still run + some tasks automatically (like lint and python unittest tasks). Further tasks + can be chosen with treeherder's ``Add New Jobs`` feature. +* :doc:`syntax <syntax>`: Select tasks using classic try syntax. +* :doc:`release <release>`: Prepare a tree for doing a staging release. +* :doc:`scriptworker <scriptworker>`: Run scriptworker tasks against a recent release. +* :doc:`compare <compare>`: Push two identical try jobs, one on your current commit and another of your choice + +You can run them with: + +.. code-block:: shell + + $ mach try <selector> + +See selector specific options by running: + +.. code-block:: shell + + $ mach try <selector> --help + +.. toctree:: + :caption: Available Selectors + :maxdepth: 1 + :hidden: + + Auto <auto> + Fuzzy <fuzzy> + Chooser <chooser> + Again <again> + Empty <empty> + Syntax <syntax> + Release <release> + Scriptworker <scriptworker> diff --git a/tools/tryselect/docs/selectors/release.rst b/tools/tryselect/docs/selectors/release.rst new file mode 100644 index 0000000000..946266249b --- /dev/null +++ b/tools/tryselect/docs/selectors/release.rst @@ -0,0 +1,31 @@ +Release Selector +================ + +This command configures the tree in preparation for doing a staging release, +and pushes the result to try. The changes that that are made include: + +- Updating the version number. +- Applying the migrations that are done as part of merge day. +- Disabling repacking most locales. (This can be disabled by passing ``--no-limit-locales``). + +For staging a beta release, run the following (with an appropriate version number): + +.. code-block:: shell + + $ mach try release --version 64.0b5 --migration central-to-beta + +For staging a final release (rc or patch), run the following (with an appropriate version number) + +.. code-block:: shell + + $ mach try release --version 64.0 --migration central-to-beta --migration beta-to-release + +Once the decision task is on the push is complete, you can start the release +through `staging ship-it instance <https://shipit.staging.mozilla-releng.net/new>`_\ [#shipit]_. + +.. note:: + + If pushing from beta or release, the corresponding migration should not be + passed, as they have already been applied. + +.. [#shipit] This is only available to release engineering and release management (as of 2018-10-15). diff --git a/tools/tryselect/docs/selectors/scriptworker.rst b/tools/tryselect/docs/selectors/scriptworker.rst new file mode 100644 index 0000000000..a3cba08cbe --- /dev/null +++ b/tools/tryselect/docs/selectors/scriptworker.rst @@ -0,0 +1,31 @@ +Scriptworker Selector +===================== + +This command runs a selection of scriptworker tasks against builds from a +recent release. This is aimed at release engineering, to test changes to +scriptworker implementations. It currently requires being connected to +Mozilla's internal datacenter VPN with access to shipit\ [#shipit]_. + +There are a number of preset groups of tasks to run. To run a particular +set of tasks, pass the name of the set to ``mach try scriptworker``: + +.. code-block:: shell + + $ mach try scriptworker linux-signing + +To get the list of task sets, along with the list of tasks they will run: + +.. code-block:: shell + + $ mach try scriptworker list + +The selector defaults to using tasks from the most recent beta, to use tasks +from a different release, pass ``--release-type <release-type>``: + +.. code-block:: shell + + $ mach try scriptworker --release-type release linux-signing + + +.. [#shipit] The shipit API is not currently publicly available, and is used + to find the release graph to pull previous tasks from. diff --git a/tools/tryselect/docs/selectors/syntax.rst b/tools/tryselect/docs/selectors/syntax.rst new file mode 100644 index 0000000000..b64bb65ab7 --- /dev/null +++ b/tools/tryselect/docs/selectors/syntax.rst @@ -0,0 +1,41 @@ +Syntax Selector +=============== + +.. warning:: + + Try syntax is antiquated and hard to understand. If you aren't already + familiar with try syntax, you might want to use the ``fuzzy`` selector + instead. + +Try syntax is a command line string that goes into the commit message. Using +``mach try syntax`` will automatically create a temporary commit with your +chosen syntax and then delete it again after pushing to try. + +Try syntax can contain all kinds of different options parsed by various +places in various repos, but the majority are parsed by `try_option_syntax.py`_. +The most common arguments include: + + * ``-b/--build`` - One of ``d``, ``o`` or ``do``. This is the build type, + either opt, debug or both (required). + * ``-p/--platform`` - The platforms you want to build and/or run tests on + (required). + * ``-u/--unittests`` - The test tasks you want to run (optional). + * ``-t/--talos`` - The talos tasks you want to run (optional). + +Here are some examples: + +.. code-block:: shell + + $ mach try syntax -b do -p linux,macosx64 -u mochitest-e10s-1,crashtest -t none + $ mach try syntax -b d -p win64 -u all + $ mach try syntax -b o -p linux64 + +Unfortunately, knowing the magic strings that make it work can be a bit of a +guessing game. If you are unsure of what string will get your task to run, try +using :doc:`mach try fuzzy <fuzzy>` instead. + +While using ``mach try syntax -b do -p all -u all -t all`` will work, heavy use +of ``all`` is discouraged as it consumes a lot of unnecessary resources (some of +which are hardware constrained). + +.. _try_option_syntax.py: https://searchfox.org/mozilla-central/source/taskcluster/gecko_taskgraph/try_option_syntax.py diff --git a/tools/tryselect/docs/tasks.rst b/tools/tryselect/docs/tasks.rst new file mode 100644 index 0000000000..61de9ec9ed --- /dev/null +++ b/tools/tryselect/docs/tasks.rst @@ -0,0 +1,152 @@ +Task Generation +=============== + +Many selectors (including ``chooser``, ``coverage`` and ``fuzzy``) source their available tasks +directly from the :ref:`taskgraph <TaskCluster Task-Graph Generation>` module by building the taskgraph +locally. This means that the list of available tasks will never be stale. While this is very +powerful, it comes with a large enough performance cost to get annoying (around twenty seconds). + +The result of the taskgraph generation will be cached, so this penalty will only be incurred +whenever a file in the ``/taskcluster`` directory is modified. Unfortunately this directory changes +pretty frequently, so developers can expect to rebuild the cache each time they pull in +``mozilla-central``. Developers who regularly work on ``/taskcluster`` can expect to rebuild even +more frequently. + + +Configuring Watchman +-------------------- + +It's possible to bypass this penalty completely by using the file watching service `watchman`_. If +you use the ``fsmonitor`` mercurial extension, you already have ``watchman`` installed. + +.. note:: + + If you aren't using `fsmonitor`_ but end up installng watchman anyway, you + might as well enable it for a faster Mercurial experience. + +Otherwise, `install watchman`_. If using Linux you'll likely run into the `inotify limits`_ outlined +on that page due to the size of ``mozilla-central``. You can `read this page`_ for more information +on how to bump the limits permanently. + +Next run the following commands: + +.. code-block:: shell + + $ cd path/to/mozilla-central + $ watchman watch . + $ watchman -j < tools/tryselect/watchman.json + +You should see output like: + +.. code-block:: json + + { + "triggerid": "rebuild-taskgraph-cache", + "disposition": "created", + "version": "20200920.192359.0" + } + +That's it. Now anytime a file under ``/taskcluster`` is modified (either by your editor, or by +updating version control), the taskgraph cache will be rebuilt in the background, allowing you to +skip the wait the next time you run ``mach try``. + +.. note:: + + Watchman triggers are persistent and don't need to be added more than once. + See `Managing Triggers`_ for how to remove a trigger. + +You can test that everything is working by running these commands: + +.. code-block:: shell + + $ statedir=`mach python -c "from mach.util import get_state_dir; print(get_state_dir(specific_to_topsrcdir=True))"` + $ rm -rf $statedir/cache/taskgraph + $ touch taskcluster/mach_commands.py + # wait a minute for generation to trigger and finish + $ ls $statedir/cache/taskgraph + +If the ``target_task_set`` file exists, you are good to go. If not you can look at the ``watchman`` +log to see if there were any errors. This typically lives somewhere like +``/usr/local/var/run/watchman/$USER-state/log``. In this case please file a bug under ``Firefox +Build System :: Try`` and include the relevant portion of the log. + + +Running Watchman on Startup +~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Watchman is both a client and a service all in one. When running a ``watchman`` command, the client +binary will start the service in the background if it isn't running. This means on reboot the +service won't be running and you'll need to start the service each time by invoking the client +binary (e.g by running ``watchman version``). + +If you'd like this to happen automatically, you can use your favourite platform specific way of +running commands at startup (``crontab``, ``rc.local``, etc.). Watchman stores separate state for +each user, so be sure you run the command as the user that set up the triggers. + +Setting up a systemd Service +++++++++++++++++++++++++++++ + +If ``systemd`` is an option you can create a service: + +.. code-block:: ini + + [Unit] + Description=Watchman for %i + After=network.target + + [Service] + Type=simple + User=%i + ExecStart=/usr/local/bin/watchman --log-level 1 watch-list -f + ExecStop=/usr/local/bin/watchman shutdown-server + + [Install] + WantedBy=multi-user.target + +Save this to a file called ``/etc/systemd/system/watchman@.service``. Then run: + +.. code-block:: shell + + $ sudo systemctl enable watchman@$USER.service + $ sudo systemctl start watchman@$USER.service + +The next time you reboot, the watchman service should start automatically. + + +Managing Triggers +~~~~~~~~~~~~~~~~~ + +When adding a trigger watchman writes it to disk. Typically it'll be a path similar to +``/usr/local/var/run/watchman/$USER-state/state``. While editing that file by hand would work, the +watchman binary provides an interface for managing your triggers. + +To see all directories you are currently watching: + +.. code-block:: shell + + $ watchman watch-list + +To view triggers that are active in a specified watch: + +.. code-block:: shell + + $ watchman trigger-list <path> + +To delete a trigger from a specified watch: + +.. code-block:: shell + + $ watchman trigger-del <path> <name> + +In the above two examples, replace ``<path>`` with the path of the watch, presumably +``mozilla-central``. Using ``.`` works as well if that is already your working directory. For more +information on managing triggers and a reference of other commands, see the `official docs`_. + + +.. _watchman: https://facebook.github.io/watchman/ +.. _fsmonitor: https://www.mercurial-scm.org/wiki/FsMonitorExtension +.. _install watchman: https://facebook.github.io/watchman/docs/install.html +.. _inotify limits: https://facebook.github.io/watchman/docs/install.html#linux-inotify-limits +.. _read this page: https://github.com/guard/listen/wiki/Increasing-the-amount-of-inotify-watchers +.. _this hint: https://github.com/facebook/watchman/commit/2985377eaf8c8538b28fae9add061b67991a87c2 +.. _official docs: https://facebook.github.io/watchman/docs/cmd/trigger.html |