diff options
Diffstat (limited to 'python/mach/docs')
| -rw-r--r-- | python/mach/docs/commands.rst | 129 | ||||
| -rw-r--r-- | python/mach/docs/driver.rst | 32 | ||||
| -rw-r--r-- | python/mach/docs/faq.rst | 152 | ||||
| -rw-r--r-- | python/mach/docs/index.rst | 89 | ||||
| -rw-r--r-- | python/mach/docs/logging.rst | 100 | ||||
| -rw-r--r-- | python/mach/docs/metrics.md | 55 | ||||
| -rw-r--r-- | python/mach/docs/settings.rst | 138 | ||||
| -rw-r--r-- | python/mach/docs/telemetry.rst | 37 | ||||
| -rw-r--r-- | python/mach/docs/usage.rst | 150 | ||||
| -rw-r--r-- | python/mach/docs/windows-usage-outside-mozillabuild.rst | 124 |
10 files changed, 1006 insertions, 0 deletions
diff --git a/python/mach/docs/commands.rst b/python/mach/docs/commands.rst new file mode 100644 index 0000000000..7547193000 --- /dev/null +++ b/python/mach/docs/commands.rst @@ -0,0 +1,129 @@ +.. _mach_commands: + +===================== +Implementing Commands +===================== + +Mach commands are defined via Python decorators. + +All the relevant decorators are defined in the *mach.decorators* module. +The important decorators are as follows: + +:py:func:`Command <mach.decorators.Command>` + A function decorator that denotes that the function should be called when + the specified command is requested. The decorator takes a command name + as its first argument and a number of additional arguments to + configure the behavior of the command. The decorated function must take a + ``command_context`` argument as its first. + ``command_context`` is a properly configured instance of a ``MozbuildObject`` + subclass, meaning it can be used for accessing things like the current config + and running processes. + +:py:func:`CommandArgument <mach.decorators.CommandArgument>` + A function decorator that defines an argument to the command. Its + arguments are essentially proxied to ArgumentParser.add_argument() + +:py:func:`SubCommand <mach.decorators.SubCommand>` + A function decorator that denotes that the function should be a + sub-command to an existing ``@Command``. The decorator takes the + parent command name as its first argument and the sub-command name + as its second argument. + + ``@CommandArgument`` can be used on ``@SubCommand`` instances just + like they can on ``@Command`` instances. + + +Here is a complete example: + +.. code-block:: python + + from mach.decorators import ( + CommandArgument, + Command, + ) + + @Command('doit', help='Do ALL OF THE THINGS.') + @CommandArgument('--force', '-f', action='store_true', + help='Force doing it.') + def doit(command_context, force=False): + # Do stuff here. + +When the module is loaded, the decorators tell mach about all handlers. +When mach runs, it takes the assembled metadata from these handlers and +hooks it up to the command line driver. Under the hood, arguments passed +to the decorators are being used to help mach parse command arguments, +formulate arguments to the methods, etc. See the documentation in the +:py:mod:`mach.base` module for more. + +The Python modules defining mach commands do not need to live inside the +main mach source tree. + +Conditionally Filtering Commands +================================ + +Sometimes it might only make sense to run a command given a certain +context. For example, running tests only makes sense if the product +they are testing has been built, and said build is available. To make +sure a command is only runnable from within a correct context, you can +define a series of conditions on the +:py:func:`Command <mach.decorators.Command>` decorator. + +A condition is simply a function that takes an instance of the +:py:func:`mozbuild.base.MachCommandBase` class as an argument, and +returns ``True`` or ``False``. If any of the conditions defined on a +command return ``False``, the command will not be runnable. The +docstring of a condition function is used in error messages, to explain +why the command cannot currently be run. + +Here is an example: + +.. code-block:: python + + from mach.decorators import ( + Command, + ) + + def build_available(cls): + """The build needs to be available.""" + return cls.build_path is not None + + @Command('run_tests', conditions=[build_available]) + def run_tests(command_context): + # Do stuff here. + +By default all commands without any conditions applied will be runnable, +but it is possible to change this behaviour by setting +``require_conditions`` to ``True``: + +.. code-block:: python + + m = mach.main.Mach() + m.require_conditions = True + +Minimizing Code in Commands +=========================== + +Mach command modules, classes, and methods work best when they are +minimal dispatchers. The reason is import bloat. Currently, the mach +core needs to import every Python file potentially containing mach +commands for every command invocation. If you have dozens of commands or +commands in modules that import a lot of Python code, these imports +could slow mach down and waste memory. + +It is thus recommended that mach modules, classes, and methods do as +little work as possible. Ideally the module should only import from +the :py:mod:`mach` package. If you need external modules, you should +import them from within the command method. + +To keep code size small, the body of a command method should be limited +to: + +1. Obtaining user input (parsing arguments, prompting, etc) +2. Calling into some other Python package +3. Formatting output + +Of course, these recommendations can be ignored if you want to risk +slower performance. + +In the future, the mach driver may cache the dispatching information or +have it intelligently loaded to facilitate lazy loading. diff --git a/python/mach/docs/driver.rst b/python/mach/docs/driver.rst new file mode 100644 index 0000000000..8a2a99a2f5 --- /dev/null +++ b/python/mach/docs/driver.rst @@ -0,0 +1,32 @@ +.. _mach_driver: + +======= +Drivers +======= + +Entry Points +============ + +It is possible to use setuptools' entry points to load commands +directly from python packages. A mach entry point is a function which +returns a list of files or directories containing mach command +providers. e.g.: + +.. code-block:: python + + def list_providers(): + providers = [] + here = os.path.abspath(os.path.dirname(__file__)) + for p in os.listdir(here): + if p.endswith('.py'): + providers.append(os.path.join(here, p)) + return providers + +See http://pythonhosted.org/setuptools/setuptools.html#dynamic-discovery-of-services-and-plugins +for more information on creating an entry point. To search for entry +point plugins, you can call +:py:meth:`mach.main.Mach.load_commands_from_entry_point`. e.g.: + +.. code-block:: python + + mach.load_commands_from_entry_point("mach.external.providers") diff --git a/python/mach/docs/faq.rst b/python/mach/docs/faq.rst new file mode 100644 index 0000000000..a640f83e87 --- /dev/null +++ b/python/mach/docs/faq.rst @@ -0,0 +1,152 @@ +.. _mach_faq: + +========================== +Frequently Asked Questions +========================== + +How do I report bugs? +--------------------- + +Bugs against the ``mach`` core can be filed in Bugzilla in the `Firefox +Build System::Mach +Core <https://bugzilla.mozilla.org/enter_bug.cgi?product=Firefox%20Build%20System&component=Mach%20Core>`__ component. + +.. note:: + + Most ``mach`` bugs are bugs in individual commands, not bugs in the core + ``mach`` code. Bugs for individual commands should be filed against the + component that command is related to. For example, bugs in the + *build* command should be filed against *Firefox Build System :: + General*. Bugs against testing commands should be filed somewhere in + the *Testing* product. + +How do I debug a command failing with a Python exception? +--------------------------------------------------------- + +You can run a command and break into ``pdb``, the Python debugger, +when the command is invoked with: + +.. code-block:: shell + + ./mach --debug-command FAILING-COMMAND ARGS ... + +How do I debug ``mach`` itself? +------------------------------- + +If you are editing the mach code, or other Python modules you can +open the terminal and start debugging with pdb with the following: + +.. code-block:: shell + + python3 -m pdb ./mach <command> + +How do I debug ``pytest`` tests? +-------------------------------- + +First, before debugging, run ``./mach python-test`` once to ensure that +the testing virtualenv is up-to-date: + +.. code-block:: shell + + ./mach python-test path/to/test.py + +Then, using the testing virtualenv, debug the test file: + +.. code-block:: shell + + <objdir>/_virtualenvs/python-test/bin/python -m pdb path/to/test.py + +How do I profile a slow command? +-------------------------------- + +To diagnose bottlenecks, you can collect a performance profile: + +.. code-block:: shell + + ./mach --profile-command SLOW-COMMAND ARGS ... + +Then, you can visualize ``mach_profile_SLOW-COMMAND.cProfile`` using +`snakeviz <https://jiffyclub.github.io/snakeviz/>`__: + +.. code-block:: shell + + # If you don't have snakeviz installed yet: + python3 -m pip install snakeviz + python3 -m snakeviz mach_profile_SLOW-COMMAND.cProfile + +How do I profile ``mach`` itself? +--------------------------------- + +Since ``--profile-command`` only profiles commands, you'll need to invoke ``cProfile`` +directly to profile ``mach`` itself: + +.. code-block:: shell + + python3 -m cProfile -o mach.cProfile ./mach ... + python3 -m snakeviz mach.cProfile + +Is ``mach`` a build system? +--------------------------- + +No. ``mach`` is just a generic command dispatching tool that happens to have +a few commands that interact with the real build system. Historically, +``mach`` *was* born to become a better interface to the build system. +However, its potential beyond just build system interaction was quickly +realized and ``mach`` grew to fit those needs. + +How do I add features to ``mach``? +---------------------------------- +If you would like to add a new feature to ``mach`` that cannot be implemented as +a ``mach`` command, the first step is to file a bug in the +``Firefox Build System :: Mach Core`` component. + +Should I implement X as a ``mach`` command? +------------------------------------------- + +There are no hard or fast rules. Generally speaking, if you have some +piece of functionality or action that is useful to multiple people +(especially if it results in productivity wins), then you should +consider implementing a ``mach`` command for it. + +Some other cases where you should consider implementing something as a +``mach`` command: + +- When your tool is a random script in the tree. Random scripts are + hard to find and may not conform to coding conventions or best + practices. ``Mach`` provides a framework in which your tool can live that + will put it in a better position to succeed than if it were on its + own. +- When the alternative is a ``make`` target. The build team generally does + not like one-off ``make`` targets that aren't part of building (read: + compiling) the tree. This includes things related to testing and + packaging. These weigh down ``Makefiles`` and add to the burden of + maintaining the build system. Instead, you are encouraged to + implement ancillary functionality in Python. If you do implement something + in Python, hooking it up to ``mach`` is often trivial. + +How do I use 3rd-party Python packages in my ``mach`` command? +-------------------------------------------------------------- + +See :ref:`Using third-party Python packages`. + +How does ``mach`` fit into the modules system? +---------------------------------------------- + +Mozilla operates with a `modules governance +system <https://www.mozilla.org/about/governance/policies/module-ownership/>`__ where +there are different components with different owners. There is not +currently a ``mach`` module. There may or may never be one; currently ``mach`` +is owned by the build team. + +Even if a ``mach`` module were established, ``mach`` command modules would +likely never belong to it. Instead, ``mach`` command modules are owned by the +team/module that owns the system they interact with. In other words, ``mach`` +is not a power play to consolidate authority for tooling. Instead, it aims to +expose that tooling through a common, shared interface. + + +Who do I contact for help or to report issues? +---------------------------------------------- + +You can ask questions in +`#build <https://chat.mozilla.org/#/room/#build:mozilla.org>`__. diff --git a/python/mach/docs/index.rst b/python/mach/docs/index.rst new file mode 100644 index 0000000000..752fe93219 --- /dev/null +++ b/python/mach/docs/index.rst @@ -0,0 +1,89 @@ +==== +Mach +==== + +Mach (German for *do*) is a generic command dispatcher for the command +line. + +To use mach, you install the mach core (a Python package), create an +executable *driver* script (named whatever you want), and write mach +commands. When the *driver* is executed, mach dispatches to the +requested command handler automatically. + +.. raw:: html + + <h2>Features</h2> + +---- + +On a high level, mach is similar to using argparse with subparsers (for +command handling). When you dig deeper, mach offers a number of +additional features: + +Distributed command definitions + With optparse/argparse, you have to define your commands on a central + parser instance. With mach, you annotate your command methods with + decorators and mach finds and dispatches to them automatically. + +Command categories + Mach commands can be grouped into categories when displayed in help. + This is currently not possible with argparse. + +Logging management + Mach provides a facility for logging (both classical text and + structured) that is available to any command handler. + +Settings files + Mach provides a facility for reading settings from an ini-like file + format. + +.. raw:: html + + <h2>Components</h2> + +---- + +Mach is conceptually composed of the following components: + +core + The mach core is the core code powering mach. This is a Python package + that contains all the business logic that makes mach work. The mach + core is common to all mach deployments. + +commands + These are what mach dispatches to. Commands are simply Python methods + registered as command names. The set of commands is unique to the + environment mach is deployed in. + +driver + The *driver* is the entry-point to mach. It is simply an executable + script that loads the mach core, tells it where commands can be found, + then asks the mach core to handle the current request. The driver is + unique to the deployed environment. But, it's usually based on an + example from this source tree. + +.. raw:: html + + <h2> Project State</h2> + +---- + +mach was originally written as a command dispatching framework to aid +Firefox development. While the code is mostly generic, there are still +some pieces that closely tie it to Mozilla/Firefox. The goal is for +these to eventually be removed and replaced with generic features so +mach is suitable for anybody to use. Until then, mach may not be the +best fit for you. + +.. toctree:: + :maxdepth: 1 + :hidden: + + usage + commands + driver + logging + settings + telemetry + windows-usage-outside-mozillabuild + faq diff --git a/python/mach/docs/logging.rst b/python/mach/docs/logging.rst new file mode 100644 index 0000000000..ff245cf032 --- /dev/null +++ b/python/mach/docs/logging.rst @@ -0,0 +1,100 @@ +.. _mach_logging: + +======= +Logging +======= + +Mach configures a built-in logging facility so commands can easily log +data. + +What sets the logging facility apart from most loggers you've seen is +that it encourages structured logging. Instead of conventional logging +where simple strings are logged, the internal logging mechanism logs all +events with the following pieces of information: + +* A string *action* +* A dict of log message fields +* A formatting string + +Essentially, instead of assembling a human-readable string at +logging-time, you create an object holding all the pieces of data that +will constitute your logged event. For each unique type of logged event, +you assign an *action* name. + +Depending on how logging is configured, your logged event could get +written a couple of different ways. + +JSON Logging +============ + +Where machines are the intended target of the logging data, a JSON +logger is configured. The JSON logger assembles an array consisting of +the following elements: + +* Decimal wall clock time in seconds since UNIX epoch +* String *action* of message +* Object with structured message data + +The JSON-serialized array is written to a configured file handle. +Consumers of this logging stream can just perform a readline() then feed +that into a JSON deserializer to reconstruct the original logged +message. They can key off the *action* element to determine how to +process individual events. There is no need to invent a parser. +Convenient, isn't it? + +Logging for Humans +================== + +Where humans are the intended consumer of a log message, the structured +log message are converted to more human-friendly form. This is done by +utilizing the *formatting* string provided at log time. The logger +simply calls the *format* method of the formatting string, passing the +dict containing the message's fields. + +When *mach* is used in a terminal that supports it, the logging facility +also supports terminal features such as colorization. This is done +automatically in the logging layer - there is no need to control this at +logging time. + +In addition, messages intended for humans typically prepends every line +with the time passed since the application started. + +Logging HOWTO +============= + +Structured logging piggybacks on top of Python's built-in logging +infrastructure provided by the *logging* package. We accomplish this by +taking advantage of *logging.Logger.log()*'s *extra* argument. To this +argument, we pass a dict with the fields *action* and *params*. These +are the string *action* and dict of message fields, respectively. The +formatting string is passed as the *msg* argument, like normal. + +If you were logging to a logger directly, you would do something like: + +.. code-block:: python + + logger.log(logging.INFO, 'My name is {name}', + extra={'action': 'my_name', 'params': {'name': 'Gregory'}}) + +The JSON logging would produce something like:: + + [1339985554.306338, "my_name", {"name": "Gregory"}] + +Human logging would produce something like:: + + 0.52 My name is Gregory + +Since there is a lot of complexity using logger.log directly, it is +recommended to go through a wrapping layer that hides part of the +complexity for you. The easiest way to do this is by utilizing the +LoggingMixin: + +.. code-block:: python + + import logging + from mach.mixin.logging import LoggingMixin + + class MyClass(LoggingMixin): + def foo(self): + self.log(logging.INFO, 'foo_start', {'bar': True}, + 'Foo performed. Bar: {bar}') diff --git a/python/mach/docs/metrics.md b/python/mach/docs/metrics.md new file mode 100644 index 0000000000..8c826f54a9 --- /dev/null +++ b/python/mach/docs/metrics.md @@ -0,0 +1,55 @@ +<!-- AUTOGENERATED BY glean_parser. DO NOT EDIT. --> + +# Metrics +This document enumerates the metrics collected by this project using the [Glean SDK](https://mozilla.github.io/glean/book/index.html). +This project may depend on other projects which also collect metrics. +This means you might have to go searching through the dependency tree to get a full picture of everything collected by this project. + +# Pings + + - [usage](#usage) + + +## usage + +Sent when the mach invocation is completed (regardless of result). Contains information about the mach invocation that was made, its result, and some details about the current environment and hardware. + + +This ping includes the [client id](https://mozilla.github.io/glean/book/user/pings/index.html#the-client_info-section). + +**Data reviews for this ping:** + +- <https://bugzilla.mozilla.org/show_bug.cgi?id=1291053#c34> + +**Bugs related to this ping:** + +- <https://bugzilla.mozilla.org/show_bug.cgi?id=1291053> + +The following metrics are added to the ping: + +| Name | Type | Description | Data reviews | Extras | Expiration | [Data Sensitivity](https://wiki.mozilla.org/Firefox/Data_Collection) | +| --- | --- | --- | --- | --- | --- | --- | +| mach.argv |[string_list](https://mozilla.github.io/glean/book/user/metrics/string_list.html) |Parameters provided to mach. Absolute paths are sanitized to be relative to one of a few key base paths, such as the "$topsrcdir", "$topobjdir", or "$HOME". For example: "/home/mozilla/dev/firefox/python/mozbuild" would be replaced with "$topsrcdir/python/mozbuild". If a valid replacement base path cannot be found, the path is replaced with "<path omitted>". |[1](https://bugzilla.mozilla.org/show_bug.cgi?id=1291053#c34)||never | | +| mach.command |[string](https://mozilla.github.io/glean/book/user/metrics/string.html) |The name of the mach command that was invoked, such as "build", "doc", or "try". |[1](https://bugzilla.mozilla.org/show_bug.cgi?id=1291053#c34)||never | | +| mach.duration |[timespan](https://mozilla.github.io/glean/book/user/metrics/timespan.html) |How long it took for the command to complete. |[1](https://bugzilla.mozilla.org/show_bug.cgi?id=1291053#c34)||never | | +| mach.success |[boolean](https://mozilla.github.io/glean/book/user/metrics/boolean.html) |True if the mach invocation succeeded. |[1](https://bugzilla.mozilla.org/show_bug.cgi?id=1291053#c34)||never | | +| mach.system.cpu_brand |[string](https://mozilla.github.io/glean/book/user/metrics/string.html) |CPU brand string from CPUID. |[1](https://bugzilla.mozilla.org/show_bug.cgi?id=1291053#c34)||never | | +| mach.system.distro |[string](https://mozilla.github.io/glean/book/user/metrics/string.html) |The name of the operating system distribution. |[1](https://bugzilla.mozilla.org/show_bug.cgi?id=1655845#c3)||never | | +| mach.system.distro_version |[string](https://mozilla.github.io/glean/book/user/metrics/string.html) |The high-level OS version. |[1](https://bugzilla.mozilla.org/show_bug.cgi?id=1655845#c3)||never | | +| mach.system.logical_cores |[counter](https://mozilla.github.io/glean/book/user/metrics/counter.html) |Number of logical CPU cores present. |[1](https://bugzilla.mozilla.org/show_bug.cgi?id=1291053#c34)||never | | +| mach.system.memory |[memory_distribution](https://mozilla.github.io/glean/book/user/metrics/memory_distribution.html) |Amount of system memory. |[1](https://bugzilla.mozilla.org/show_bug.cgi?id=1291053#c34)||never | | +| mach.system.physical_cores |[counter](https://mozilla.github.io/glean/book/user/metrics/counter.html) |Number of physical CPU cores present. |[1](https://bugzilla.mozilla.org/show_bug.cgi?id=1291053#c34)||never | | +| mozbuild.artifact |[boolean](https://mozilla.github.io/glean/book/user/metrics/boolean.html) |True if `--enable-artifact-builds`. |[1](https://bugzilla.mozilla.org/show_bug.cgi?id=1291053#c34)||never | | +| mozbuild.ccache |[boolean](https://mozilla.github.io/glean/book/user/metrics/boolean.html) |True if `--with-ccache`. |[1](https://bugzilla.mozilla.org/show_bug.cgi?id=1291053#c34)||never | | +| mozbuild.clobber |[boolean](https://mozilla.github.io/glean/book/user/metrics/boolean.html) |True if the build was a clobber/full build. |[1](https://bugzilla.mozilla.org/show_bug.cgi?id=1526072#c15)||never | | +| mozbuild.compiler |[string](https://mozilla.github.io/glean/book/user/metrics/string.html) |The compiler type in use (CC_TYPE), such as "clang" or "gcc". |[1](https://bugzilla.mozilla.org/show_bug.cgi?id=1291053#c34)||never | | +| mozbuild.debug |[boolean](https://mozilla.github.io/glean/book/user/metrics/boolean.html) |True if `--enable-debug`. |[1](https://bugzilla.mozilla.org/show_bug.cgi?id=1291053#c34)||never | | +| mozbuild.icecream |[boolean](https://mozilla.github.io/glean/book/user/metrics/boolean.html) |True if icecream in use. |[1](https://bugzilla.mozilla.org/show_bug.cgi?id=1291053#c34)||never | | +| mozbuild.opt |[boolean](https://mozilla.github.io/glean/book/user/metrics/boolean.html) |True if `--enable-optimize`. |[1](https://bugzilla.mozilla.org/show_bug.cgi?id=1291053#c34)||never | | +| mozbuild.project |[string](https://mozilla.github.io/glean/book/user/metrics/string.html) |The project being built. |[1](https://bugzilla.mozilla.org/show_bug.cgi?id=1654084#c2)||never | | +| mozbuild.sccache |[boolean](https://mozilla.github.io/glean/book/user/metrics/boolean.html) |True if ccache in use is sccache. |[1](https://bugzilla.mozilla.org/show_bug.cgi?id=1291053#c34)||never | | + + +Data categories are [defined here](https://wiki.mozilla.org/Firefox/Data_Collection). + +<!-- AUTOGENERATED BY glean_parser. DO NOT EDIT. --> diff --git a/python/mach/docs/settings.rst b/python/mach/docs/settings.rst new file mode 100644 index 0000000000..4daba37472 --- /dev/null +++ b/python/mach/docs/settings.rst @@ -0,0 +1,138 @@ +.. _mach_settings: + +======== +Settings +======== + +Mach can read settings in from a set of configuration files. These +configuration files are either named ``machrc`` or ``.machrc`` and +are specified by the bootstrap script. In mozilla-central, these files +can live in ``~/.mozbuild`` and/or ``topsrcdir``. + +Settings can be specified anywhere, and used both by mach core or +individual commands. + + +Core Settings +============= + +These settings are implemented by mach core. + +* alias - Create a command alias. This is useful if you want to alias a command to something else, optionally including some defaults. It can either be used to create an entire new command, or provide defaults for an existing one. For example: + +.. parsed-literal:: + + [alias] + mochitest = mochitest -f browser + browser-test = mochitest -f browser + + +Defining Settings +================= + +Settings need to be explicitly defined, along with their type, +otherwise mach will throw when trying to access them. + +To define settings, use the :func:`~decorators.SettingsProvider` +decorator in an existing mach command module. E.g: + +.. code-block:: python + + from mach.decorators import SettingsProvider + from mozbuild.base import MachCommandBase + + @SettingsProvider + class ArbitraryClassName(MachCommandBase): + config_settings = [ + ('foo.bar', 'string', "A helpful description"), + ('foo.baz', 'int', "Another description", 0, {'choices': set([0,1,2])}), + ] + +``@SettingsProvider``'s must specify a variable called ``config_settings`` +that returns a list of tuples. Alternatively, it can specify a function +called ``config_settings`` that returns a list of tuples. + +Each tuple is of the form: + +.. code-block:: python + + ('<section>.<option>', '<type>', '<description>', default, extra) + +``type`` is a string and can be one of: +string, boolean, int, pos_int, path + +``description`` is a string explaining how to define the settings and +where they get used. Descriptions should ideally be multi-line paragraphs +where the first line acts as a short description. + +``default`` is optional, and provides a default value in case none was +specified by any of the configuration files. + +``extra`` is also optional and is a dict containing additional key/value +pairs to add to the setting's metadata. The following keys may be specified +in the ``extra`` dict: + + * ``choices`` - A set of allowed values for the setting. + +Wildcards +--------- + +Sometimes a section should allow arbitrarily defined options from the user, such +as the ``alias`` section mentioned above. To define a section like this, use ``*`` +as the option name. For example: + +.. parsed-literal:: + + ('foo.*', 'string', 'desc') + +This allows configuration files like this: + +.. parsed-literal:: + + [foo] + arbitrary1 = some string + arbitrary2 = some other string + + +Finding Settings +================ + +You can see which settings are available as well as their description and +expected values by running: + +.. parsed-literal:: + + ./mach settings # or + ./mach settings --list + + +Accessing Settings +================== + +Now that the settings are defined and documented, they're accessible from +individual mach commands from the mach command context. +For example: + +.. code-block:: python + + from mach.decorators import ( + Command, + SettingsProvider, + ) + from mozbuild.base import MachCommandBase + + @SettingsProvider + class ExampleSettings(object): + config_settings = [ + ('a.b', 'string', 'desc', 'default'), + ('foo.bar', 'string', 'desc',), + ('foo.baz', 'int', 'desc', 0, {'choices': set([0,1,2])}), + ] + + @Command('command', category='misc', + description='Prints a setting') + def command(command_context): + settings = command_context._mach_context.settings + print(settings.a.b) + for option in settings.foo: + print(settings.foo[option]) diff --git a/python/mach/docs/telemetry.rst b/python/mach/docs/telemetry.rst new file mode 100644 index 0000000000..2d185a970e --- /dev/null +++ b/python/mach/docs/telemetry.rst @@ -0,0 +1,37 @@ +.. _mach_telemetry: + +============== +Mach Telemetry +============== + +`Glean <https://mozilla.github.io/glean/>`_ is used to collect telemetry, and uses the metrics +defined in the ``metrics.yaml`` files in-tree. +These files are all documented in a single :ref:`generated file here<metrics>`. + +.. toctree:: + :maxdepth: 1 + + metrics + +Adding Metrics to a new Command +=============================== + +If you would like to submit telemetry metrics from your mach ``@Command``, you should take two steps: + +#. Parameterize your ``@Command`` annotation with ``metrics_path``. +#. Use the ``command_context.metrics`` handle provided by ``MachCommandBase`` + +For example:: + + METRICS_PATH = os.path.abspath(os.path.join(__file__, '..', '..', 'metrics.yaml')) + + @Command('custom-command', metrics_path=METRICS_PATH) + def custom_command(command_context): + command_context.metrics.custom.foo.set('bar') + +Updating Generated Metrics Docs +=============================== + +When a ``metrics.yaml`` is added/changed/removed, :ref:`the metrics document<metrics>` will need to be updated:: + + ./mach doc mach-telemetry diff --git a/python/mach/docs/usage.rst b/python/mach/docs/usage.rst new file mode 100644 index 0000000000..a32b35395c --- /dev/null +++ b/python/mach/docs/usage.rst @@ -0,0 +1,150 @@ +.. _mach_usage: + +========== +User Guide +========== + +Mach is the central entry point for most operations that can be performed in +mozilla-central. + + +Command Help +------------ + +To see an overview of all the available commands, run: + +.. code-block:: shell + + $ ./mach help + +For more detailed information on a specific command, run: + +.. code-block:: shell + + $ ./mach help <command> + +If a command has subcommands listed, you can see more details on the subcommand +by running: + +.. code-block:: shell + + $ ./mach help <command> <subcommand> + +Alternatively, you can pass ``-h/--help``. For example, all of the +following are valid: + +.. code-block:: shell + + $ ./mach help try + $ ./mach help try fuzzy + $ ./mach try -h + $ ./mach try fuzzy --help + + +Tab Completion +-------------- + +There are commands built-in to ``mach`` that can generate a fast tab completion +script for various shells. Supported shells are currently ``bash``, ``zsh`` and +``fish``. These generated scripts will slowly become out of date over time, so +you may want to create a cron task to periodically re-generate them. + +See below for installation instructions: + +Bash +~~~~ + +.. code-block:: shell + + $ mach mach-completion bash -f _mach + $ sudo mv _mach /etc/bash_completion.d + +Bash (homebrew) +~~~~~~~~~~~~~~~ + +.. code-block:: shell + + $ mach mach-completion bash -f $(brew --prefix)/etc/bash_completion.d/mach.bash-completion + +Zsh +~~~ + +.. code-block:: shell + + $ mkdir ~/.zfunc + $ mach mach-completion zsh -f ~/.zfunc/_mach + +then edit ~/.zshrc and add: + +.. code-block:: shell + + fpath+=~/.zfunc + autoload -U compinit && compinit + +You can use any directory of your choosing. + +Zsh (oh-my-zsh) +~~~~~~~~~~~~~~~ + +.. code-block:: shell + + $ mkdir $ZSH/plugins/mach + $ mach mach-completion zsh -f $ZSH/plugins/mach/_mach + +then edit ~/.zshrc and add 'mach' to your enabled plugins: + +.. code-block:: shell + + plugins(mach ...) + +Zsh (prezto) +~~~~~~~~~~~~ + +.. code-block:: shell + + $ mach mach-completion zsh -f ~/.zprezto/modules/completion/external/src/_mach + +Fish +~~~~ + +.. code-block:: shell + + $ ./mach mach-completion fish -f ~/.config/fish/completions/mach.fish + +Fish (homebrew) +~~~~~~~~~~~~~~~ + +.. code-block:: shell + + $ ./mach mach-completion fish -f (brew --prefix)/share/fish/vendor_completions.d/mach.fish + + +User Settings +------------- + +Some mach commands can read configuration from a ``machrc`` file. The default +location for this file is ``~/.mozbuild/machrc`` (you'll need to create it). +This can also be set to a different location by setting the ``MACHRC`` +environment variable. + +For a list of all the available settings, run: + +.. code-block:: shell + + $ ./mach settings + +The settings file follows the ``ini`` format, e.g: + +.. code-block:: ini + + [alias] + eslint = lint -l eslint + + [build] + telemetry = true + + [try] + default = fuzzy + + +.. _bash completion: https://searchfox.org/mozilla-central/source/python/mach/bash-completion.sh diff --git a/python/mach/docs/windows-usage-outside-mozillabuild.rst b/python/mach/docs/windows-usage-outside-mozillabuild.rst new file mode 100644 index 0000000000..6a034fd384 --- /dev/null +++ b/python/mach/docs/windows-usage-outside-mozillabuild.rst @@ -0,0 +1,124 @@ +========================================== +Using Mach on Windows Outside MozillaBuild +========================================== + +.. note:: + + These docs still require that you've followed the :ref:`Building Firefox On Windows` guide. + +`MozillaBuild <https://wiki.mozilla.org/MozillaBuild>`__ is required to build +Firefox on Windows, because it provides necessary unix-y tools such as ``sh`` and ``awk``. + +Traditionally, to interact with Mach and the Firefox Build System, Windows +developers would have to do so from within the MozillaBuild shell. This could be +disadvantageous for two main reasons: + +1. The MozillaBuild environment is unix-y and based on ``bash``, which may be unfamiliar + for developers used to the Windows Command Prompt or Powershell. +2. There have been long-standing stability issues with MozillaBuild - this is due to + the fragile interface point between the underlying "MSYS" tools and "native Windows" + binaries. + +It is now (experimentally!) possible to invoke Mach directly from other command line +environments, such as Powershell, Command Prompt, or even a developer-managed MSYS2 +environment. Windows Terminal should work as well, for those on the "cutting edge". + +.. note:: + + If you're using a Cygwin-based environment such as MSYS2, it'll probably be + best to use the Windows-native version of Python (as described below) instead of a Python + distribution provided by the environment's package manager. Otherwise you'll likely run into + compatibility issues: + + * Cygwin/MSYS Python will run into compatibility issues with Mach due to its unexpected Unix-y + conventions despite Mach assuming it's on a "Windows" platform. Additionally, there may + be performance issues. + * MinGW Python will encounter issues building native packages because they'll expect the + MSVC toolchain. + +.. warning:: + + This is only recommended for more advanced Windows developers: this work is experimental + and may run into unexpected failures! + +Following are steps for preparing Windows-native (Command Prompt/Powershell) usage of Mach: + +1. Install Python +~~~~~~~~~~~~~~~~~ + +Download Python from the `the official website <https://www.python.org/downloads/windows/>`__. + +.. note:: + + To avoid Mach compatibility issues with recent Python releases, it's recommended to install + the 2nd-most recent "major version". For example, at time of writing, the current modern Python + version is 3.10.1, so a safe version to install would be the most recent 3.9 release. + +You'll want to download the "Windows installer (64-bit)" associated with the release you've chosen. +During installation, ensure that you check the "Add Python 3.x to PATH" option, otherwise you might +`encounter issues running Mercurial <https://bz.mercurial-scm.org/show_bug.cgi?id=6635>`__. + +.. note:: + + Due to issues with Python DLL import failures with pip-installed binaries, it's not + recommended to use the Windows Store release of Python. + +2. Modify your PATH +~~~~~~~~~~~~~~~~~~~ + +The Python "user site-packages directory" needs to be added to your ``PATH`` so that packages +installed via ``pip install --user`` (such as ``hg``) can be invoked from the command-line. + +1. From the Start menu, go to the Control Panel entry for "Edit environment variables + for your account". +2. Double-click the ``Path`` row in the top list of variables. Click "New" to add a new item to + the list. +3. In a Command Prompt window, resolve the Python directory with the command + ``python -c "import site; import os; print(os.path.abspath(os.path.join(site.getusersitepackages(), '..', 'Scripts')))"``. +4. Paste the output into the new item entry in the "Edit environment variable" window. +5. Click "New" again, and add the ``bin`` folder of MozillaBuild: probably ``C:\mozilla-build\bin``. +6. Click "OK". + +3. Install Version Control System +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +If you're using Mercurial, you'll need to install it to your Windows-native Python: + +.. code-block:: shell + + pip3 install --user mercurial windows-curses + +If you're using Git with Cinnabar, follow its `setup instructions <https://github.com/glandium/git-cinnabar#setup>`__. + +4. Set Powershell Execution Policy +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +If you're using Powershell, Windows will raise an error by default when you try to invoke +``.\mach.ps1``: + +.. code:: + + .\mach : File <topsrcdir>\mach.ps1 cannot be loaded because running scripts is disabled on this system. For + more information, see about_Execution_Policies at https:/go.microsoft.com/fwlink/?LinkID=135170. + At line:1 char:1 + +To work around this: + +1. From the Start menu, type in "Powershell", then right-click on the best match and click + "Run as administrator" +2. Run the command ``Set-ExecutionPolicy RemoteSigned`` +3. Close the Administrator Powershell window, and open a regular Powershell window +4. Go to your Firefox checkout (likely ``C:\mozilla-source\mozilla-unified``) +5. Test the new execution policy by running ``.\mach bootstrap``. If it doesn't immediately fail + with the error about "Execution Policies", then the problem is resolved. + +Success! +~~~~~~~~ + +At this point, you should be able to invoke Mach and manage your version control system outside +of MozillaBuild. + +.. tip:: + + `See here <https://crisal.io/words/2022/11/22/msys2-firefox-development.html>`__ for a detailed guide on + installing and customizing a development environment with MSYS2, zsh, and Windows Terminal. |