diff options
Diffstat (limited to 'docs/docsite/rst/dev_guide/testing_running_locally.rst')
| -rw-r--r-- | docs/docsite/rst/dev_guide/testing_running_locally.rst | 382 |
1 files changed, 382 insertions, 0 deletions
diff --git a/docs/docsite/rst/dev_guide/testing_running_locally.rst b/docs/docsite/rst/dev_guide/testing_running_locally.rst new file mode 100644 index 0000000..0d03189 --- /dev/null +++ b/docs/docsite/rst/dev_guide/testing_running_locally.rst @@ -0,0 +1,382 @@ +:orphan: + +.. _testing_running_locally: + +******************************* +Testing Ansible and Collections +******************************* + +This document describes how to run tests using ``ansible-test``. + +.. contents:: + :local: + +Setup +===== + +Before running ``ansible-test``, set up your environment for :ref:`testing_an_ansible_collection` or +:ref:`testing_ansible_core`, depending on which scenario applies to you. + +.. warning:: + + If you use ``git`` for version control, make sure the files you are working with are not ignored by ``git``. + If they are, ``ansible-test`` will ignore them as well. + +.. _testing_an_ansible_collection: + +Testing an Ansible Collection +----------------------------- + +If you are testing an Ansible Collection, you need a copy of the collection, preferably a git clone. +For example, to work with the ``community.windows`` collection, follow these steps: + +1. Clone the collection you want to test into a valid collection root: + + .. code-block:: shell + + git clone https://github.com/ansible-collections/community.windows ~/dev/ansible_collections/community/windows + + .. important:: + + The path must end with ``/ansible_collections/{collection_namespace}/{collection_name}`` where + ``{collection_namespace}`` is the namespace of the collection and ``{collection_name}`` is the collection name. + +2. Clone any collections on which the collection depends: + + .. code-block:: shell + + git clone https://github.com/ansible-collections/ansible.windows ~/dev/ansible_collections/ansible/windows + + .. important:: + + If your collection has any dependencies on other collections, they must be in the same collection root, since + ``ansible-test`` will not use your configured collection roots (or other Ansible configuration). + + .. note:: + + See the collection's ``galaxy.yml`` for a list of possible dependencies. + +3. Switch to the directory where the collection to test resides: + + .. code-block:: shell + + cd ~/dev/ansible_collections/community/windows + +.. _testing_ansible_core: + +Testing ``ansible-core`` +------------------------ + +If you are testing ``ansible-core`` itself, you need a copy of the ``ansible-core`` source code, preferably a git clone. +Having an installed copy of ``ansible-core`` is not sufficient or required. +For example, to work with the ``ansible-core`` source cloned from GitHub, follow these steps: + +1. Clone the ``ansible-core`` repository: + + .. code-block:: shell + + git clone https://github.com/ansible/ansible ~/dev/ansible + +2. Switch to the directory where the ``ansible-core`` source resides: + + .. code-block:: shell + + cd ~/dev/ansible + +3. Add ``ansible-core`` programs to your ``PATH``: + + .. code-block:: shell + + source hacking/env-setup + + .. note:: + + You can skip this step if you only need to run ``ansible-test``, and not other ``ansible-core`` programs. + In that case, simply run ``bin/ansible-test`` from the root of the ``ansible-core`` source. + + .. caution:: + + If you have an installed version of ``ansible-core`` and are trying to run ``ansible-test`` from your ``PATH``, + make sure the program found by your shell is the one from the ``ansible-core`` source: + + .. code-block:: shell + + which ansible-test + +Commands +======== + +The most commonly used test commands are: + +* ``ansible-test sanity`` - Run sanity tests (mostly linters and static analysis). +* ``ansible-test integration`` - Run integration tests. +* ``ansible-test units`` - Run unit tests. + +Run ``ansible-test --help`` to see a complete list of available commands. + +.. note:: + + For detailed help on a specific command, add the ``--help`` option after the command. + +Environments +============ + +Most ``ansible-test`` commands support running in one or more isolated test environments to simplify testing. + +Containers +---------- + +Containers are recommended for running sanity, unit and integration tests, since they provide consistent environments. +Unit tests will be run with network isolation, which avoids unintentional dependencies on network resources. + +The ``--docker`` option runs tests in a container using either Docker or Podman. + +.. note:: + + If both Docker and Podman are installed, Docker will be used. + To override this, set the environment variable ``ANSIBLE_TEST_PREFER_PODMAN`` to any non-empty value. + +Choosing a container +^^^^^^^^^^^^^^^^^^^^ + +Without an additional argument, the ``--docker`` option uses the ``default`` container. +To use another container, specify it immediately after the ``--docker`` option. + +.. note:: + + The ``default`` container is recommended for all sanity and unit tests. + +To see the list of supported containers, use the ``--help`` option with the ``ansible-test`` command you want to use. + +.. note:: + + The list of available containers is dependent on the ``ansible-test`` command you are using. + +You can also specify your own container. +When doing so, you will need to indicate the Python version in the container with the ``--python`` option. + +Custom containers +""""""""""""""""" + +When building custom containers, keep in mind the following requirements: + +* The ``USER`` should be ``root``. +* Use an ``init`` process, such as ``systemd``. +* Include ``sshd`` and accept connections on the default port of ``22``. +* Include a POSIX compatible ``sh`` shell which can be found on ``PATH``. +* Include a ``sleep`` utility which runs as a subprocess. +* Include a supported version of Python. +* Avoid using the ``VOLUME`` statement. + +Docker and SELinux +^^^^^^^^^^^^^^^^^^ + +Using Docker on a host with SELinux may require setting the system in permissive mode. +Consider using Podman instead. + +Docker Desktop with WSL2 +^^^^^^^^^^^^^^^^^^^^^^^^ + +These instructions explain how to use ``ansible-test`` with WSL2 and Docker Desktop *without* ``systemd`` support. + +.. note:: + + If your WSL2 environment includes ``systemd`` support, these steps are not required. + +.. _configuration_requirements: + +Configuration requirements +"""""""""""""""""""""""""" + +1. Open Docker Desktop and go to the **Settings** screen. +2. On the the **General** tab: + + a. Uncheck the **Start Docker Desktop when you log in** checkbox. + b. Check the **Use the WSL 2 based engine** checkbox. + +3. On the **Resources** tab under the **WSL Integration** section: + + a. Enable distros you want to use under the **Enable integration with additional distros** section. + +4. Click **Apply and restart** if changes were made. + +.. _setup_instructions: + +Setup instructions +"""""""""""""""""" + +.. note:: + + If all WSL instances have been stopped, these changes will need to be re-applied. + +1. Verify Docker Desktop is properly configured (see :ref:`configuration_requirements`). +2. Quit Docker Desktop if it is running: + + a. Right click the **Docker Desktop** taskbar icon. + b. Click the **Quit Docker Desktop** option. + +3. Stop any running WSL instances with the command: + + .. code-block:: shell + + wsl --shutdown + +4. Verify all WSL instances have stopped with the command: + + .. code-block:: shell + + wsl -l -v + +5. Start a WSL instance and perform the following steps as ``root``: + + a. Verify the ``systemd`` subsystem is not registered: + + a. Check for the ``systemd`` cgroup hierarchy with the following command: + + .. code-block:: shell + + grep systemd /proc/self/cgroup + + b. If any matches are found, re-check the :ref:`configuration_requirements` and follow the + :ref:`setup_instructions` again. + + b. Mount the ``systemd`` cgroup hierarchy with the following commands: + + .. code-block:: shell + + mkdir /sys/fs/cgroup/systemd + mount cgroup -t cgroup /sys/fs/cgroup/systemd -o none,name=systemd,xattr + +6. Start Docker Desktop. + +You should now be able to use ``ansible-test`` with the ``--docker`` option. + +.. _linux_cgroup_configuration: + +Linux cgroup configuration +^^^^^^^^^^^^^^^^^^^^^^^^^^ + +.. note:: + + These changes will need to be re-applied each time the container host is booted. + +For certain container hosts and container combinations, additional setup on the container host may be required. +In these situations ``ansible-test`` will report an error and provide additional instructions to run as ``root``: + +.. code-block:: shell + + mkdir /sys/fs/cgroup/systemd + mount cgroup -t cgroup /sys/fs/cgroup/systemd -o none,name=systemd,xattr + +If you are using rootless Podman, an additional command must be run, also as ``root``. +Make sure to substitute your user and group for ``{user}`` and ``{group}`` respectively: + +.. code-block:: shell + + chown -R {user}:{group} /sys/fs/cgroup/systemd + +Podman +"""""" + +When using Podman, you may need to stop existing Podman processes after following the :ref:`linux_cgroup_configuration` +instructions. Otherwise Podman may be unable to see the new mount point. + +You can check to see if Podman is running by looking for ``podman`` and ``catatonit`` processes. + +Remote virtual machines +----------------------- + +Remote virtual machines are recommended for running integration tests not suitable for execution in containers. + +The ``--remote`` option runs tests in a cloud hosted ephemeral virtual machine. + +.. note:: + + An API key is required to use this feature, unless running under an approved Azure Pipelines organization. + +To see the list of supported systems, use the ``--help`` option with the ``ansible-test`` command you want to use. + +.. note:: + + The list of available systems is dependent on the ``ansible-test`` command you are using. + +Python virtual environments +--------------------------- + +Python virtual environments provide a simple way to achieve isolation from the system and user Python environments. +They are recommended for unit and integration tests when the ``--docker`` and ``--remote`` options cannot be used. + +The ``--venv`` option runs tests in a virtual environment managed by ``ansible-test``. +Requirements are automatically installed before tests are run. + +Composite environment arguments +------------------------------- + +The environment arguments covered in this document are sufficient for most use cases. +However, some scenarios may require the additional flexibility offered by composite environment arguments. + +The ``--controller`` and ``--target`` options are alternatives to the ``--docker``, ``--remote`` and ``--venv`` options. + +.. note:: + + When using the ``shell`` command, the ``--target`` option is replaced by three platform specific options. + +Add the ``--help`` option to your ``ansible-test`` command to learn more about the composite environment arguments. + +Additional Requirements +======================= + +Some ``ansible-test`` commands have additional requirements. +You can use the ``--requirements`` option to automatically install them. + +.. note:: + + When using a test environment managed by ``ansible-test`` the ``--requirements`` option is usually unnecessary. + +Environment variables +===================== + +When using environment variables to manipulate tests there some limitations to keep in mind. Environment variables are: + +* Not propagated from the host to the test environment when using the ``--docker`` or ``--remote`` options. +* Not exposed to the test environment unless enabled in ``test/lib/ansible_test/_internal/util.py`` in the ``common_environment`` function. + + Example: ``ANSIBLE_KEEP_REMOTE_FILES=1`` can be set when running ``ansible-test integration --venv``. However, using the ``--docker`` option would + require running ``ansible-test shell`` to gain access to the Docker environment. Once at the shell prompt, the environment variable could be set + and the tests executed. This is useful for debugging tests inside a container by following the + :ref:`debugging_modules` instructions. + +Interactive shell +================= + +Use the ``ansible-test shell`` command to get an interactive shell in the same environment used to run tests. Examples: + +* ``ansible-test shell --docker`` - Open a shell in the default docker container. +* ``ansible-test shell --venv --python 3.10`` - Open a shell in a Python 3.10 virtual environment. + +Code coverage +============= + +Code coverage reports make it easy to identify untested code for which more tests should +be written. Online reports are available but only cover the ``devel`` branch (see +:ref:`developing_testing`). For new code local reports are needed. + +Add the ``--coverage`` option to any test command to collect code coverage data. If you +aren't using the ``--venv`` or ``--docker`` options which create an isolated python +environment then you may have to use the ``--requirements`` option to ensure that the +correct version of the coverage module is installed: + +.. code-block:: shell + + ansible-test coverage erase + ansible-test units --coverage apt + ansible-test integration --coverage aws_lambda + ansible-test coverage html + +Reports can be generated in several different formats: + +* ``ansible-test coverage report`` - Console report. +* ``ansible-test coverage html`` - HTML report. +* ``ansible-test coverage xml`` - XML report. + +To clear data between test runs, use the ``ansible-test coverage erase`` command. |