diff options
Diffstat (limited to 'docs/docsite/rst/community/collection_contributors')
12 files changed, 1521 insertions, 0 deletions
diff --git a/docs/docsite/rst/community/collection_contributors/collection_integration_about.rst b/docs/docsite/rst/community/collection_contributors/collection_integration_about.rst new file mode 100644 index 0000000..d011fdb --- /dev/null +++ b/docs/docsite/rst/community/collection_contributors/collection_integration_about.rst @@ -0,0 +1,155 @@ +.. _collection_integration_tests_about: + +Understanding integration tests +================================= + +.. note:: + + Some collections do not have integration tests. + +Integration tests are functional tests of modules and plugins. +With integration tests, we check if a module or plugin satisfies its functional requirements. Simply put, we check that features work as expected and users get the outcome described in the module or plugin documentation. + +There are :ref:`two kinds of integration tests <collections_adding_integration_test>` used in collections: + +* integration tests that use Ansible roles +* integration tests that use ``runme.sh``. + +This section focuses on integration tests that use Ansible roles. + +Integration tests check modules with playbooks that invoke those modules. The tests pass standalone parameters and their combinations, check what the module or plugin reports with the :ref:`assert <ansible_collections.ansible.builtin.assert_module>` module, and the actual state of the system after each task. + +Integration test example +------------------------- + +Let's say we want to test the ``postgresql_user`` module invoked with the ``name`` parameter. We expect that the module will both create a user based on the provided value of the ``name`` parameter and will report that the system state has changed. We cannot rely on only what the module reports. To be sure that the user has been created, we query our database with another module to see if the user exists. + +.. code-block:: yaml + + - name: Create PostgreSQL user and store module's output to the result variable + community.postgresql.postgresql_user: + name: test_user + register: result + + - name: Check the module returns what we expect + assert: + that: + - result is changed + + - name: Check actual system state with another module, in other words, that the user exists + community.postgresql.postgresql_query: + query: SELECT * FROM pg_authid WHERE rolename = 'test_user' + register: query_result + + - name: We expect it returns one row, check it + assert: + that: + - query_result.rowcount == 1 + + +Details about integration tests +-------------------------------- + +The basic entity of an Ansible integration test is a ``target``. The target is an :ref:`Ansible role <playbooks_reuse_roles>` stored in the ``tests/integration/targets`` directory of the collection repository. The target role contains everything that is needed to test a module. + +The names of targets contain the module or plugin name that they test. Target names that start with ``setup_`` are usually executed as dependencies before module and plugin targets start execution. See :ref:`collection_creating_integration_tests` for details. + +To run integration tests, we use the ``ansible-test`` utility that is included in the ``ansible-core`` and ``ansible`` packages. See :ref:`collection_run_integration_tests` for details. After you finish your integration tests, see to :ref:`collection_quickstart` to learn how to submit a pull request. + +.. _collection_integration_prepare: + +Preparing for integration tests for collections +================================================= + +To prepare for developing integration tests: + +#. :ref:`Set up your local environment <collection_prepare_environment>`. + +#. Determine if integration tests already exist. + + .. code-block:: bash + + ansible-test integration --list-targets + + +If a collection already has integration tests, they are stored in ``tests/integration/targets/*`` subdirectories of the collection repository. + +If you use ``bash`` and the ``argcomplete`` package is installed with ``pip`` on your system, you can also get a full target list. + +.. code-block:: shell + + ansible-test integration <tab><tab> + +Alternately, you can check if the ``tests/integration/targets`` directory contains a corresponding directory with the same name as the module. For example, the tests for the ``postgresql_user`` module of the ``community.postgresql`` collection are stored in the ``tests/integration/targets/postgresql_user`` directory of the collection repository. If there is no corresponding target there, then that module does not have integration tests. In this case, consider adding integration tests for the module. See :ref:`collection_creating_integration_tests` for details. + + +.. _collection_integration_recommendations: + +Recommendations on coverage +=========================== + +Bugfixes +-------- + +Before fixing code, create a test case in an :ref:`appropriate test target<collection_integration_prepare>` that reproduces the bug provided by the issue reporter and described in the ``Steps to Reproduce`` issue section. :ref:`Run <collection_run_integration_tests>` the tests. + +If you failed to reproduce the bug, ask the reporter to provide additional information. The issue may be related to environment settings. Sometimes specific environment issues cannot be reproduced in integration tests, in that case, manual testing by issue reporter or other interested users is required. + +Refactoring code +---------------- + +When refactoring code, always check that related options are covered in a :ref:`corresponding test target<collection_integration_prepare>`. Do not assume if the test target exists, everything is covered. + +.. _collections_recommendation_modules: + +Covering modules / new features +------------------------------- + +When covering a module, cover all its options separately and their meaningful combinations. Every possible use of the module should be tested against: + +- Idempotency - Does rerunning a task report no changes? +- Check-mode - Does dry-running a task behave the same as a real run? Does it not make any changes? +- Return values - Does the module return values consistently under different conditions? + +Each test action has to be tested at least the following times: + +- Perform an action in check-mode if supported. This should indicate a change. +- Check with another module that the changes have ``not`` been actually made. +- Perform the action for real. This should indicate a change. +- Check with another module that the changes have been actually made. +- Perform the action again in check-mode. This should indicate ``no`` change. +- Perform the action again for real. This should indicate ``no`` change. + +To check a task: + +1. Register the outcome of the task as a variable, for example, ``register: result``. Using the :ref:`assert <ansible_collections.ansible.builtin.assert_module>` module, check: + + #. If ``- result is changed`` or not. + #. Expected return values. + +2. If the module changes the system state, check the actual system state using at least one other module. For example, if the module changes a file, we can check that the file has been changed by checking its checksum with the :ref:`stat <ansible_collections.ansible.builtin.stat_module>` module before and after the test tasks. +3. Run the same task with ``check_mode: true`` if check-mode is supported by the module. Check with other modules that the actual system state has not been changed. +4. Cover cases when the module must fail. Use the ``ignore_errors: true`` option and check the returned message with the ``assert`` module. + +Example: + +.. code-block:: yaml + + - name: Task to fail + abstract_module: + ... + register: result + + - name: Check the task fails and its error message + assert: + that: + - result is failed + - result.msg == 'Message we expect' + +Here is a summary: + +- Cover options and their sensible combinations. +- Check returned values. +- Cover check-mode if supported. +- Check a system state using other modules. +- Check when a module must fail and error messages. diff --git a/docs/docsite/rst/community/collection_contributors/collection_integration_add.rst b/docs/docsite/rst/community/collection_contributors/collection_integration_add.rst new file mode 100644 index 0000000..a4c8a10 --- /dev/null +++ b/docs/docsite/rst/community/collection_contributors/collection_integration_add.rst @@ -0,0 +1,250 @@ +.. _collection_creating_integration_tests: + +Creating new integration tests +================================= + +This section covers the following cases: + +- There are no integration tests for a collection or group of modules in a collection at all. +- You are adding a new module and you want to include integration tests. +- You want to add integration tests for a module that already exists without integration tests. + +In other words, there are currently no tests for a module regardless of whether the module exists or not. + +If the module already has tests, see :ref:`collection_updating_integration_tests`. + +Simplified example +-------------------- + +Here is a simplified abstract example. + +Let's say we are going to add integration tests to a new module in the ``community.abstract`` collection which interacts with some service. + +We :ref:`checked<collection_integration_prepare>` and determined that there are no integration tests at all. + +We should basically do the following: + +1. Install and run the service with a ``setup`` target. +2. Create a test target. +3. Add integration tests for the module. +4. :ref:`Run the tests<collection_run_integration_tests>`. +5. Fix the code and tests as needed, run the tests again, and repeat the cycle until they pass. + +.. note:: + + You can reuse the ``setup`` target when implementing other targets that also use the same service. + +1. Clone the collection to the ``~/ansible_collections/community.abstract`` directory on your local machine. + +2. From the ``~/ansible_collections/community.abstract`` directory, create directories for the ``setup`` target: + +.. code-block:: bash + + mkdir -p tests/integration/targets/setup_abstract_service/tasks + +3. Write all the tasks needed to prepare the environment, install, and run the service. + +For simplicity, let's imagine that the service is available in the native distribution repositories and no sophisticated environment configuration is required. + +Add the following tasks to the ``tests/integration/targets/setup_abstract_service/tasks/main.yml`` file to install and run the service: + +.. code-block:: yaml + + - name: Install abstract service + package: + name: abstract_service + + - name: Run the service + systemd: + name: abstract_service + state: started + +This is a very simplified example. + +4. Add the target for the module you are testing. + +Let's say the module is called ``abstract_service_info``. Create the following directory structure in the target: + +.. code-block:: bash + + mkdir -p tests/integration/targets/abstract_service_info/tasks + mkdir -p tests/integration/targets/abstract_service_info/meta + +Add all of the needed subdirectories. For example, if you are going to use defaults and files, add the ``defaults`` and ``files`` directories, and so on. The approach is the same as when you are creating a role. + +5. To make the ``setup_abstract_service`` target run before the module's target, add the following lines to the ``tests/integration/targets/abstract_service_info/meta/main.yml`` file. + +.. code-block:: yaml + + dependencies: + - setup_abstract_service + +6. Start with writing a single stand-alone task to check that your module can interact with the service. + +We assume that the ``abstract_service_info`` module fetches some information from the ``abstract_service`` and that it has two connection parameters. + +Among other fields, it returns a field called ``version`` containing a service version. + +Add the following to ``tests/integration/targets/abstract_service_info/tasks/main.yml``: + +.. code-block:: yaml + + - name: Fetch info from abstract service + abstract_service_info: + host: 127.0.0.1 # We assume the service accepts local connection by default + port: 1234 # We assume that the service is listening to this port by default + register: result # This variable will contain the returned JSON including the server version + + - name: Test the output + assert: + that: + - result.version == '1.0.0' # Check version field contains what we expect + +7. :ref:`Run the tests<collection_run_integration_tests>` with the ``-vvv`` argument. + +If there are any issues with connectivity (for example, the service is not accepting connections) or with the code, the play will fail. + +Examine the output to see at which step the failure occurred. Investigate the reason, fix it, and run again. Repeat the cycle until the test passes. + +8. If the test succeeds, write more tests. Refer to the :ref:`Recommendations on coverage<collection_integration_recommendations>` section for details. + +``community.postgresql`` example +-------------------------------- + +Here is a real example of writing integration tests from scratch for the ``community.postgresql.postgresql_info`` module. + +For the sake of simplicity, we will create very basic tests which we will run using the Ubuntu 20.04 test container. + +We use ``Linux`` as a work environment and have ``git`` and ``docker`` installed and running. + +We also installed ``ansible-core``. + +1. Create the following directories in your home directory: + +.. code-block:: bash + + mkdir -p ~/ansible_collections/community + +2. Fork the `collection repository <https://github.com/ansible-collections/community.postgresql>`_ through the GitHub web interface. + +3. Clone the forked repository from your profile to the created path: + +.. code-block:: bash + + git clone https://github.com/YOURACC/community.postgresql.git ~/ansible_collections/community/postgresql + +If you prefer to use the SSH protocol: + +.. code-block:: bash + + git clone git@github.com:YOURACC/community.postgresql.git ~/ansible_collections/community/postgresql + +4. Go to the cloned repository: + +.. code-block:: bash + + cd ~/ansible_collections/community/postgresql + +5. Be sure you are in the default branch: + +.. code-block:: bash + + git status + +6. Checkout a test branch: + +.. code-block:: bash + + git checkout -b postgresql_info_tests + +7. Since we already have tests for the ``postgresql_info`` module, we will run the following command: + +.. code-block:: bash + + rm -rf tests/integration/targets/* + +With all of the targets now removed, the current state is as if we do not have any integration tests for the ``community.postgresql`` collection at all. We can now start writing integration tests from scratch. + +8. We will start with creating a ``setup`` target that will install all required packages and will launch PostgreSQL. Create the following directories: + +.. code-block:: bash + + mkdir -p tests/integration/targets/setup_postgresql_db/tasks + +9. Create the ``tests/integration/targets/setup_postgresql_db/tasks/main.yml`` file and add the following tasks to it: + +.. code-block:: yaml + + - name: Install required packages + package: + name: + - apt-utils + - postgresql + - postgresql-common + - python3-psycopg2 + + - name: Initialize PostgreSQL + shell: . /usr/share/postgresql-common/maintscripts-functions && set_system_locale && /usr/bin/pg_createcluster -u postgres 12 main + args: + creates: /etc/postgresql/12/ + + - name: Start PostgreSQL service + ansible.builtin.service: + name: postgresql + state: started + +That is enough for our very basic example. + +10. Then, create the following directories for the ``postgresql_info`` target: + +.. code-block:: bash + + mkdir -p tests/integration/targets/postgresql_info/tasks tests/integration/targets/postgresql_info/meta + +11. To make the ``setup_postgresql_db`` target run before the ``postgresql_info`` target as a dependency, create the ``tests/integration/targets/postgresql_info/meta/main.yml`` file and add the following code to it: + +.. code-block:: yaml + + dependencies: + - setup_postgresql_db + +12. Now we are ready to add our first test task for the ``postgresql_info`` module. Create the ``tests/integration/targets/postgresql_info/tasks/main.yml`` file and add the following code to it: + +.. code-block:: yaml + + - name: Test postgresql_info module + become: true + become_user: postgres + community.postgresql.postgresql_info: + login_user: postgres + login_db: postgres + register: result + + - name: Check the module returns what we expect + assert: + that: + - result is not changed + - result.version.major == 12 + - result.version.minor == 8 + +In the first task, we run the ``postgresql_info`` module to fetch information from the database we installed and launched with the ``setup_postgresql_db`` target. We are saving the values returned by the module into the ``result`` variable. + +In the second task, we check the ``result`` variable, which is what the first task returned, with the ``assert`` module. We expect that, among other things, the result has the version and reports that the system state has not been changed. + +13. Run the tests in the Ubuntu 20.04 docker container: + +.. code-block:: bash + + ansible-test integration postgresql_info --docker ubuntu2004 -vvv + +The tests should pass. If we look at the output, we should see something like the following: + +.. code-block:: shell + + TASK [postgresql_info : Check the module returns what we expect] *************** + ok: [testhost] => { + "changed": false, + "msg": "All assertions passed" + } + +If your tests fail when you are working on your project, examine the output to see at which step the failure occurred. Investigate the reason, fix it, and run again. Repeat the cycle until the test passes. If the test succeeds, write more tests. Refer to the :ref:`Recommendations on coverage<collection_integration_recommendations>` section for details. diff --git a/docs/docsite/rst/community/collection_contributors/collection_integration_running.rst b/docs/docsite/rst/community/collection_contributors/collection_integration_running.rst new file mode 100644 index 0000000..64e610d --- /dev/null +++ b/docs/docsite/rst/community/collection_contributors/collection_integration_running.rst @@ -0,0 +1,32 @@ +.. _collection_run_integration_tests: + +Running integration tests +============================ + +In the following examples, we will use ``Docker`` to run integration tests locally. Ensure you :ref:`collection_prepare_environment` first. + +We assume that you are in the ``~/ansible_collections/NAMESPACE/COLLECTION`` directory. + +After you change the tests, you can run them with the following command: + +.. code-block:: text + + ansible-test integration <target_name> --docker <distro> + +The ``target_name`` is a test role directory containing the tests. For example, if the test files you changed are stored in the ``tests/integration/targets/postgresql_info/`` directory and you want to use the ``fedora34`` container image, then the command will be: + +.. code-block:: bash + + ansible-test integration postgresql_info --docker fedora34 + +You can use the ``-vv`` or ``-vvv`` argument if you need more detailed output. + +In the examples above, the ``fedora34`` test image will be automatically downloaded and used to create and run a test container. + +See the :ref:`list of supported container images <test_container_images>`. + +In some cases, for example, for platform-independent tests, the ``default`` test image is required. Use the ``--docker default`` or just ``--docker`` option without specifying a distribution in this case. + +.. note:: + + If you have any difficulties with writing or running integration tests or you are not sure if the case can be covered, submit your pull request without the tests. Other contributors can help you with them later if needed. diff --git a/docs/docsite/rst/community/collection_contributors/collection_integration_tests.rst b/docs/docsite/rst/community/collection_contributors/collection_integration_tests.rst new file mode 100644 index 0000000..9eef463 --- /dev/null +++ b/docs/docsite/rst/community/collection_contributors/collection_integration_tests.rst @@ -0,0 +1,36 @@ +.. _collection_integration_tests: + + +***************************************** +Adding integration tests to a collection +***************************************** + +This section describes the steps to add integration tests to a collection and how to run them locally using the ``ansible-test`` command. + +.. toctree:: + :maxdepth: 1 + + collection_integration_about + collection_integration_updating + collection_integration_running + collection_integration_add + + +.. seealso:: + + :ref:`testing_units_modules` + Unit testing Ansible modules + `pytest <https://docs.pytest.org/en/latest/>`_ + Pytest framework documentation + :ref:`developing_testing` + Ansible Testing Guide + :ref:`collection_unit_tests` + Unit testing for collections + :ref:`testing_integration` + Integration tests guide + :ref:`testing_collections` + Testing collections + :ref:`testing_resource_modules` + Resource module integration tests + :ref:`collection_pr_test` + How to test a pull request locally diff --git a/docs/docsite/rst/community/collection_contributors/collection_integration_updating.rst b/docs/docsite/rst/community/collection_contributors/collection_integration_updating.rst new file mode 100644 index 0000000..46d18c0 --- /dev/null +++ b/docs/docsite/rst/community/collection_contributors/collection_integration_updating.rst @@ -0,0 +1,169 @@ +.. _collection_updating_integration_tests: + +Adding to an existing integration test +======================================= + +The test tasks are stored in the ``tests/integration/targets/<target_name>/tasks`` directory. + +The ``main.yml`` file holds test tasks and includes other test files. +Look for a suitable test file to integrate your tests or create and include or import a separate test file. +You can use one of the existing test files as a draft. + +When fixing a bug +----------------- + +When fixing a bug: + +1. :ref:`Determine if integration tests for the module exist<collection_integration_prepare>`. If they do not, see :ref:`collection_creating_integration_tests` section. +2. Add a task that reproduces the bug to an appropriate file within the ``tests/integration/targets/<target_name>/tasks`` directory. +3. :ref:`Run the tests<collection_run_integration_tests>`. The newly added task should fail. +4. If they do not fail, re-check if your environment or test task satisfies the conditions described in the ``Steps to Reproduce`` section of the issue. +5. If you reproduce the bug and tests fail, change the code. +6. :ref:`Run the tests<collection_run_integration_tests>` again. +7. If they fail, repeat steps 5-6 until the tests pass. + +Here is an example. + +Let's say someone reported an issue in the ``community.postgresql`` collection that when users pass a name containing underscores to the ``postgresql_user`` module, the module fails. + +We cloned the collection repository to the ``~/ansible_collections/community/postgresql`` directory and :ref:`prepared our environment <collection_prepare_environment>`. From the collection's root directory, we run ``ansible-test integration --list-targets`` and it shows a target called ``postgresql_user``. It means that we already have tests for the module. + +We start with reproducing the bug. + +First, we look into the ``tests/integration/targets/postgresql_user/tasks/main.yml`` file. In this particular case, the file imports other files from the ``tasks`` directory. The ``postgresql_user_general.yml`` looks like an appropriate one to add our tests. + +.. code-block:: yaml + + # General tests: + - import_tasks: postgresql_user_general.yml + when: postgres_version_resp.stdout is version('9.4', '>=') + +We will add the following code to the file. + +.. code-block:: yaml + + # https://github.com/ansible-collections/community.postgresql/issues/NUM + - name: Test user name containing underscore + community.postgresql.postgresql_user: + name: underscored_user + register: result + + - name: Check the module returns what we expect + assert: + that: + - result is changed + + - name: Query the database if the user exists + community.postgresql.postgresql_query: + query: SELECT * FROM pg_authid WHERE rolename = 'underscored_user' + register: result + + - name: Check the database returns one row + assert: + that: + - result.query_result.rowcount == 1 + +When we :ref:`run the tests<collection_run_integration_tests>` with ``postgresql_user`` as a test target, this task must fail. + +Now that we have our failing test; we will fix the bug and run the same tests again. Once the tests pass, we will consider the bug fixed and will submit a pull request. + +When adding a new feature +------------------------- + +.. note:: + + The process described in this section also applies when you want to add integration tests to a feature that already exists, but is missing integration tests. + +If you have not already implemented the new feature, you can start by writing the integration tests for it. They will not work as the code does not yet exist, but they can help you improve your implementation design before you start writing any code. + +When adding new features, the process of adding tests consists of the following steps: + +1. :ref:`Determine if integration tests for the module exists<collection_integration_prepare>`. If they do not, see :ref:`collection_creating_integration_tests`. +2. Find an appropriate file for your tests within the ``tests/integration/targets/<target_name>/tasks`` directory. +3. Cover your feature with tests. Refer to the :ref:`Recommendations on coverage<collection_integration_recommendations>` section for details. +4. :ref:`Run the tests<collection_run_integration_tests>`. +5. If they fail, see the test output for details. Fix your code or tests and run the tests again. +6. Repeat steps 4-5 until the tests pass. + +Here is an example. + +Let's say we decided to add a new option called ``add_attribute`` to the ``postgresql_user`` module of the ``community.postgresql`` collection. + +The option is boolean. If set to ``yes``, it adds an additional attribute to a database user. + +We cloned the collection repository to the ``~/ansible_collections/community/postgresql`` directory and :ref:`prepared our environment<collection_integration_prepare>`. From the collection's root directory, we run ``ansible-test integration --list-targets`` and it shows a target called ``postgresql_user``. Therefore, we already have some tests for the module. + +First, we look at the ``tests/integration/targets/<target_name>/tasks/main.yml`` file. In this particular case, the file imports other files from the ``tasks`` directory. The ``postgresql_user_general.yml`` file looks like an appropriate one to add our tests. + +.. code-block:: yaml + + # General tests: + - import_tasks: postgresql_user_general.yml + when: postgres_version_resp.stdout is version('9.4', '>=') + +We will add the following code to the file. + +.. code-block:: yaml + + # https://github.com/ansible-collections/community.postgresql/issues/NUM + # We should also run the same tasks with check_mode: yes. We omit it here for simplicity. + - name: Test for new_option, create new user WITHOUT the attribute + community.postgresql.postgresql_user: + name: test_user + register: result + + - name: Check the module returns what we expect + assert: + that: + - result is changed + + - name: Query the database if the user exists but does not have the attribute (it is NULL) + community.postgresql.postgresql_query: + query: SELECT * FROM pg_authid WHERE rolename = 'test_user' AND attribute = NULL + register: result + + - name: Check the database returns one row + assert: + that: + - result.query_result.rowcount == 1 + + - name: Test for new_option, create new user WITH the attribute + community.postgresql.postgresql_user: + name: test_user + register: result + + - name: Check the module returns what we expect + assert: + that: + - result is changed + + - name: Query the database if the user has the attribute (it is TRUE) + community.postgresql.postgresql_query: + query: SELECT * FROM pg_authid WHERE rolename = 'test_user' AND attribute = 't' + register: result + + - name: Check the database returns one row + assert: + that: + - result.query_result.rowcount == 1 + +Then we :ref:`run the tests<collection_run_integration_tests>` with ``postgresql_user`` passed as a test target. + +In reality, we would alternate the tasks above with the same tasks run with the ``check_mode: yes`` option to be sure our option works as expected in check-mode as well. See :ref:`Recommendations on coverage<collection_integration_recommendations>` for details. + +If we expect a task to fail, we use the ``ignore_errors: true`` option and check that the task actually failed and returned the message we expect: + +.. code-block:: yaml + + - name: Test for fail_when_true option + community.postgresql.postgresql_user: + name: test_user + fail_when_true: true + register: result + ignore_errors: true + + - name: Check the module fails and returns message we expect + assert: + that: + - result is failed + - result.msg == 'The message we expect' diff --git a/docs/docsite/rst/community/collection_contributors/collection_release_with_branches.rst b/docs/docsite/rst/community/collection_contributors/collection_release_with_branches.rst new file mode 100644 index 0000000..f8ba7ed --- /dev/null +++ b/docs/docsite/rst/community/collection_contributors/collection_release_with_branches.rst @@ -0,0 +1,341 @@ +.. _collection_release_with_branches: + +Releasing collections with release branches +============================================ + +Collections MUST follow the `semantic versioning <https://semver.org/>`_ rules. See :ref:`releasing_collections` for high-level details. + +.. contents:: + :local: + + +Release planning and announcement +---------------------------------- + +#. Announce your intention to release the collection in a corresponding pinned release issue/community pinboard of the collection and in the ``#ansible-community`` `Matrix/IRC channel <https://docs.ansible.com/ansible/devel/community/communication.html#real-time-chat>`_. Repeat the announcement in any other dedicated channels if they exist. + +#. Ensure all the other repository maintainers are informed about the time of the following release. + + +Releasing major collection versions +------------------------------------- + +The new version is assumed to be ``X.0.0``. + +1. Make sure that ``galaxy.yml`` contains the correct version number ``X.0.0``. If that is not the case, create a PR to update it. This will make sanity tests fail for all deprecations that have to be removed in ``X.0.0``, so this is potentially a lot of work and should have been done weeks before the major release. + +2. Check the collection for deprecations that are planned for removal in the major release that were not reported by the sanity tests. Use past changelogs or run ``grep -r `X.0.0` plugins/`` in the repository. + +3. If you are going to release the ``community.general`` and ``community.network`` collections, create a new ``backport-X`` label in the corresponding repositories. Copy the styles and descriptions from the corresponding existing labels. + +4. Ensure you are in a default branch in your local fork. These examples use ``main``. + + .. code-block:: bash + + git status + git checkout main # if needed + + +5. Update your local fork: + + .. code-block:: bash + + git pull --rebase upstream main + + +Creating the release branch +^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +1. Create a branch ``stable-X``. Replace ``X`` with a correct number and push it to the **upstream** repository, NOT to the ``origin``.: + + .. code-block:: bash + + git branch stable-X main + git push upstream stable-X + + +2. Create and checkout to another branch from the ``main`` branch: + + .. code-block:: bash + + git checkout -b update_repo + + +3. Update the version in ``galaxy.yml`` in the branch to the next **expected** version, for example, ``X.1.0``. + + +Creating the changelogs +^^^^^^^^^^^^^^^^^^^^^^^^ + +1. Replace ``changelogs/changelog.yml`` with: + + .. code-block:: yaml + + ancestor: X.0.0 + releases: {} + + +2. Remove all changelog fragments from ``changelogs/fragments/``. Removing the changelog fragments ensures that every major release has a changelog describing changes since the last major release. + +3. Add and commit all the changes made. Push the branch to the ``origin`` repository. + +4. Create a pull request in the collection repository. If CI tests pass, merge the pull request since the ``main`` branch is expecting changes for the next minor/major versions + +5. Switch to the ``stable-X`` branch. + +6. In the ``stable-X`` branch, verify that ``galaxy.yml`` contains the correct version number ``X.0.0``. + +7. In the ``stable-X`` branch, ensure that ``changelogs/changelog.yml`` contains a correct ancestor's version: + + .. code-block:: yaml + + ancestor: X-1.0.0 + releases: {} + + +8. In the ``stable-X`` branch, add a changelog fragment ``changelogs/fragments/X.0.0.yml`` with the content: + + .. code-block:: yaml + + release_summary: |- + Write some text here that should appear as the release summary for this version. + The format is reStructuredText, but not a list as for regular changelog fragments. + This text will be inserted into the changelog. + + + For example: + + .. code-block:: yaml + + release_summary: This is release 2.0.0 of ``community.foo``, released on YYYY-MM-DD. + + +9. In the ``stable-X`` branch, generate the changelogs: + + .. code-block:: bash + + antsibull-changelog release --cummulative-release + + +10. In the ``stable-X`` branch, verify that the ``CHANGELOG.rst`` looks as expected. + +11. In the ``stable-X`` branch, update ``README.md`` so that the changelog link points to ``/tree/stable-X/`` and no longer to ``/tree/main/``, and change badges respectively, for example, in case of AZP, add ``?branchName=stable-X`` to the AZP CI badge (https://dev.azure.com/ansible/community.xxx/_apis/build/status/CI?branchName=stable-X). + +12. In the ``stable-X`` branch, add, commit, and push changes to ``README.md``, ``CHANGELOG.rst`` and ``changelogs/changelog.yaml``, and potentially deleted/archived fragments to the **upstream** repository, NOT to the ``origin``. + + +Publishing the collection +^^^^^^^^^^^^^^^^^^^^^^^^^^ + +1. In the ``stable-X`` branch, add an annotated tag to the last commit with the collection version ``X.0.0``. Pushing this tag to the ``upstream`` repository will make Zuul publish the collection on `Ansible Galaxy <https://galaxy.ansible.com/>`_. + + .. code-block:: bash + + git tag -n # see current tags and their comments + git tag -a NEW_VERSION -m "comment here" # the comment can be, for example, "community.foo: 2.0.0" + git push upstream NEW_VERSION + + +2. If the collection uses `Zuul <https://github.com/ansible/zuul-config/blob/master/README.rst>`_ for publishing its releases, wait until the new version is published on the collection's `Ansible Galaxy <https://galaxy.ansible.com/>`_ page. It will appear in a list of tarballs available to download. + +3. If the release tarball did not appear within several hours after pushing the tag, try to re-tag the release commit and push the tag again. In the ``stable-X`` branch being at the release commit: + + .. code-block:: bash + + git tag --delete NEW_VERSION + git push upstream :NEW_VERSION + git tag -a NEW_VERSION -m "comment here" # the comment can be, for example, "community.foo: 2.0.0" + git push upstream NEW_VERSION + + +4. Add a GitHub release for the new tag. The title should be the version and content, such as - ``See https://github.com/ansible-collections/community.xxx/blob/stable-X/CHANGELOG.rst for all changes``. + +5. Announce the release through the `Bullhorn Newsletter <https://github.com/ansible/community/wiki/News#the-bullhorn>`_. + +6. Announce the release in the pinned release issue/community pinboard of the collection and in the ``#ansible-community`` `Matrix/Libera.Chat IRC channel <https://docs.ansible.com/ansible/devel/community/communication.html#real-time-chat>`_. + +7. In the ``stable-X`` branch, update the version in ``galaxy.yml`` to the next **expected** version, for example, ``X.1.0``. Add, commit and push to the **upstream** repository. + + +Releasing minor collection versions +------------------------------------- + +The new version is assumed to be ``X.Y.0``. All changes that should go into it are expected to be previously backported from the default branch to the ``stable-X`` branch. + +Creating the changelogs +^^^^^^^^^^^^^^^^^^^^^^^^ + +1. In the ``stable-X`` branch, make sure that ``galaxy.yml`` contains the correct version number ``X.Y.0``. If not, update it. + +2. In the ``stable-X`` branch, add a changelog fragment ``changelogs/fragments/X.Y.0.yml`` with content: + + .. code-block:: yaml + + release_summary: |- + Write some text here that should appear as the release summary for this version. + The format is reStructuredText, but not a list as for regular changelog fragments. + This text will be inserted into the changelog. + + +3. In the ``stable-X`` branch, run: + + .. code-block:: bash + + antsibull-changelog release + + +4. In the ``stable-X`` branch, verify that ``CHANGELOG.rst`` looks as expected. + +5. In the ``stable-X`` branch, add, commit, and push changes to ``CHANGELOG.rst`` and ``changelogs/changelog.yaml``, and potentially deleted/archived fragments to the **upstream** repository, NOT to the origin. + + +Publishing the collection +^^^^^^^^^^^^^^^^^^^^^^^^^^ + +1. In the ``stable-X`` branch, add an annotated tag to the last commit with the collection version ``X.Y.0``. Pushing this tag to the ``upstream`` repository will make Zuul publish the collection on `Ansible Galaxy <https://galaxy.ansible.com/>`_. + + .. code-block:: bash + + git tag -n # see current tags and their comments + git tag -a NEW_VERSION -m "comment here" # the comment can be, for example, "community.foo: 2.1.0" + git push upstream NEW_VERSION + + +2. Wait until the new version is published on the collection's `Ansible Galaxy <https://galaxy.ansible.com/>`_ page. The published version will appear in a list of tarballs available to download. + +3. Add a GitHub release for the new tag. The title should be the version and content, such as - ``See https://github.com/ansible-collections/community.xxx/blob/stable-X/CHANGELOG.rst for all changes``. + +4. Announce the release through the `Bullhorn Newsletter <https://github.com/ansible/community/wiki/News#the-bullhorn>`_. + +5. Announce the release in the pinned release issue/community pinboard of the collection and in the ``#ansible-community`` `Matrix/IRC channel <https://docs.ansible.com/ansible/devel/community/communication.html#real-time-chat>`_. Additionally, you can announce it using GitHub's Releases system. + +6. In the ``stable-X`` branch, update the version in ``galaxy.yml`` to the next **expected** version, for example, if you have released ``X.1.0``, the next expected version could be ``X.2.0``. Add, commit and push to the **upstream** repository. + +7. Checkout to the ``main`` branch. + +8. In the ``main`` branch: + + #. If more minor versions are released before the next major version, update the version in ``galaxy.yml`` to ``X.(Y+1).0`` as well. Create a dedicated pull request and merge. + + #. If the next version will be a new major version, create a pull request where you update the version in ``galaxy.yml`` to ``(X+1).0.0``. Note that the sanity tests will most likely fail since there will be deprecations with removal scheduled for ``(X+1).0.0``, which are flagged by the tests. + + For every such deprecation, decide: + + * Whether to remove them now. For example you remove the complete ``modules/plugins`` or you remove redirects. + * Whether to add ignore entries to the corresponding ``tests/sanity/ignore-*.txt`` file and create issues, for example for removed features in ``modules/plugins``. + + Once the CI tests pass, merge the pull request. Make sure that this pull request is merged not too much later after the release + for ``version_added`` sanity tests not to expect the wrong version for the new feature pull request. + +.. note:: + + It makes sense to already do some removals in the days before the release. These removals must happen in the main branch and must not be backported. + + +Releasing patch versions +------------------------- + +The new version is assumed to be ``X.Y.Z``, and the previous patch version is assumed to be ``X.Y.z`` with ``z < Z``. ``z`` is frequently``0`` since patch releases are uncommon. + +Releasing when more minor versions are expected +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +1. Checkout the ``X.Y.z`` tag. + +2. Update ``galaxy.yml`` so that the version is ``X.Y.Z``. Add and commit. + +3. Cherry-pick all changes from ``stable-X`` that were added after ``X.Y.z`` and should go into ``X.Y.Z``. + +4. Add a changelog fragment ``changelogs/fragments/X.Y.Z.yml`` with content: + + .. code-block:: yaml + + release_summary: |- + Write some text here that should appear as the release summary for this version. + The format is reStructuredText but not a list as for regular changelog fragments. + This text will be inserted into the changelog. + + Add to git and commit. + +5. Generate the changelogs. + +.. code-block:: bash + + antsibull-changelog release + +6. Verify that ``CHANGELOG.rst`` looks as expected. + +7. Add and commit changes to ``CHANGELOG.rst`` and ``changelogs/changelog.yaml``, and potentially deleted/archived fragments. + +**Publishing the collection** + + +1. Add an annotated tag to the last commit with the collection version ``X.Y.Z``. Pushing this tag to the ``upstream`` repository will make Zuul publish the collection on `Ansible Galaxy <https://galaxy.ansible.com/>`_. + + .. code-block:: bash + + git tag -n # see current tags and their comments + git tag -a NEW_VERSION -m "comment here" # the comment can be, for example, "community.foo: 2.1.1" + git push upstream NEW_VERSION + + +2. Wait until the new version is published on the collection's `Ansible Galaxy <https://galaxy.ansible.com/>`_ page. It will appear in a list of tarballs available to download. + +3. Add a GitHub release for the new tag. The title should be the version and content, such as - ``See https://github.com/ansible-collections/community.xxx/blob/stable-X/CHANGELOG.rst for all changes``. + + .. note:: + + The data for this release is only contained in a tag, and not in a branch, in particular not in ``stable-X``. + This is deliberate, since the next minor release ``X.(Y+1).0`` already contains the changes for ``X.Y.Z`` as well, since these were cherry-picked from ``stable-X``. + + +4. Announce the release through the `Bullhorn Newsletter <https://github.com/ansible/community/wiki/News#the-bullhorn>`_. + +5. Announce the release in the pinned release issue/community pinboard of the collection and in the ``#ansible-community`` `Matrix/IRC channel <https://docs.ansible.com/ansible/devel/community/communication.html#real-time-chat>`. + + +Releasing when no more minor versions are expected +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +1. In the ``stable-X`` branch, make sure that ``galaxy.yml`` contains the correct version number ``X.Y.Z``. If not, update it! + +2. In the ``stable-X`` branch, add a changelog fragment ``changelogs/fragments/X.Y.Z.yml`` with content: + + .. code-block:: yaml + + release_summary: |- + Write some text here that should appear as the release summary for this version. + The format is reStructuredText, but not a list as for regular changelog fragments. + This text will be inserted into the changelog. + + +3. Generate the changelogs in the ``stable-X`` branch. + + .. code-block:: bash + + antsibull-changelog release + + +4. In the ``stable-X`` branch, verify that ``CHANGELOG.rst`` looks as expected. + +5. In the ``stable-X`` branch, add, commit, and push changes to ``CHANGELOG.rst`` and ``changelogs/changelog.yaml``, and potentially deleted/archived fragments to the **upstream** repository, NOT to the origin. + +**Publishing the collection** + + +1. In the ``stable-X`` branch, add an annotated tag to the last commit with the collection version ``X.Y.Z``. Pushing this tag to the ``upstream`` repository will make Zuul publish the collection on `Ansible Galaxy <https://galaxy.ansible.com/>`_. + + .. code-block:: bash + + git tag -n # see current tags and their comments + git tag -a NEW_VERSION -m "comment here" # the comment can be, for example, "community.foo: 2.1.1" + git push upstream NEW_VERSION + + +2. Wait until the new version is published on the collection's `Ansible Galaxy <https://galaxy.ansible.com/>`_ page. It will appear in a list of tarballs available to download. + +3. Add a GitHub release for the new tag. Title should be the version and content, such as: ``See https://github.com/ansible-collections/community.xxx/blob/stable-X/CHANGELOG.rst for all changes``. + +4. Announce the release through the `Bullhorn Newsletter <https://github.com/ansible/community/wiki/News#the-bullhorn>`_. + +5. Announce the release in the pinned issue/community pinboard of the collection and in the ``#ansible-community`` `Matrix/IRC channel <https://docs.ansible.com/ansible/devel/community/communication.html#real-time-chat>`_. diff --git a/docs/docsite/rst/community/collection_contributors/collection_release_without_branches.rst b/docs/docsite/rst/community/collection_contributors/collection_release_without_branches.rst new file mode 100644 index 0000000..4cef33f --- /dev/null +++ b/docs/docsite/rst/community/collection_contributors/collection_release_without_branches.rst @@ -0,0 +1,115 @@ + +.. _collection_release_without_branches: + +Releasing collections without release branches +=============================================== + +Since no release branches are used, this section does not distinguish between releasing a major, minor, or patch version. + +.. contents:: + :local: + +Release planning and announcement +---------------------------------- + +#. Examine the collection to determine if there are merged changes to release. + +#. According to the changes made, choose an appropriate release version number. Keep in mind that the collections must follow the `semantic versioning <https://semver.org/>`_ rules. See :ref:`collection_versioning_and_deprecation` for details. + +#. Announce your intention to release the collection in a corresponding pinned release issue or community pinboard of the collection and in the ``community`` :ref:`Matrix/IRC channel <communication_irc>`. + +Creating the release branch +---------------------------- + +1. Ensure you are in a default branch in your local fork. We use ``main`` in the following examples. + + .. code:: bash + + git status + git checkout main # if needed + + +2. Update your local fork: + + .. code:: bash + + git pull --rebase upstream main + + +3. Checkout a new release branch from the default branch: + + .. code:: bash + + git checkout -b release_branch + +4. Ensure the ``galaxy.yml`` contains the correct release version number. + + +Generating the changelog +------------------------- + +1. Add a changelog fragment ``changelogs/fragments/X.Y.Z.yml`` with content: + + .. code:: yaml + + release_summary: |- + Write some text here that should appear as the release summary for this version. + The format is reStructuredText, but not a list as for regular changelog fragments. + This text will be inserted into the changelog. + + For example: + + .. code:: yaml + + release_summary: |- + This is the minor release of the ``community.mysql`` collection. + This changelog contains all changes to the modules and plugins in this collection + that have been made after the previous release. + + +2. If the content was recently moved from another collection (for example, migrating a module from one collection to another), ensure you have all related changelog fragments in the ``changelogs/fragments`` directory. If not, copy them previously. + +3. Run ``antsibull-changelog release --reload-plugins`` . This package should previously be installed with ``pip install antsibull-changelog``. + +4. Verify that the ``CHANGELOG.rst`` looks as expected. + +5. Commit and push changes to the ``CHANGELOG.rst`` and ``changelogs/changelog.yaml``, and potentially deleted/archived fragments to the ``origin`` repository's ``release_branch``. + + .. code:: bash + + git commit -a -m "Release VERSION commit" + git push origin release_branch + + +6. Create a pull request in the collection repository. If CI tests pass, merge it. + +7. Checkout the default branch and pull the changes: + + .. code:: bash + + git checkout main + git pull --rebase upstream main + + +Publish the collection +----------------------------------- + +1. Add an annotated tag to the release commit with the collection version. Pushing this tag to the ``upstream`` repository will make Zuul publish the collection on `Ansible Galaxy <https://galaxy.ansible.com/>`_. + + .. code:: bash + + git tag -n # see current tags and their comments + git tag -a NEW_VERSION -m "comment here" # the comment can be, for example, "community.postgresql: 1.2.0" + git push upstream NEW_VERSION + + + +2. Wait until the new version is published on the collection's `Ansible Galaxy <https://galaxy.ansible.com/>`_ page. It will appear in a list of tarballs available to download. + +3. Update the version in the ``galaxy.yml`` file to the next **expected** version. Add, commit, and push to the ``upstream``'s default branch. + +4. Add a GitHub release for the new tag. Title should be the version and content ``See https://github.com/ansible-collections/community.xxx/blob/main/CHANGELOG.rst for all changes``. + +5. Announce the release through the `Bullhorn Newsletter issue <https://github.com/ansible/community/wiki/News#the-bullhorn>`_. + +6. Announce the release in the pinned release issue/community pinboard of the collection mentioned in step 3 and in the ``community`` :ref:`Matrix/IRC channel <communication_irc>`. diff --git a/docs/docsite/rst/community/collection_contributors/collection_releasing.rst b/docs/docsite/rst/community/collection_contributors/collection_releasing.rst new file mode 100644 index 0000000..481208b --- /dev/null +++ b/docs/docsite/rst/community/collection_contributors/collection_releasing.rst @@ -0,0 +1,106 @@ + +.. _releasing_collections: +.. _Releasing: + +Releasing collections +====================== + +Collection maintainers release all supported stable versions of the collections regularly, +provided that there have been enough changes merged to release. + + +.. contents:: + :local: + +Preparing to release a collection +-------------------------------------------- + + +The collections under the `ansible-collections organization <https://github.com/ansible-collections>`_ follow `semantic versioning <https://semver.org/>`_ when releasing. See :ref:`collection_versioning_and_deprecation` for details. + +To prepare for a release, a collection must have: + +* A publicly available policy of releasing, versioning, and deprecation. This can be, for example, written in its README or in a dedicated pinned issue. +* A pinned issue when its release managers inform the community about planned or completed releases. This can be combined with the release policy issue mentioned above. +* A :ref:`changelog <collection_changelogs>`. +* Releases of the collection tagged in the collection's repository. +* CI pipelines up and running. This can be implemented by using GitHub Actions, Azure Pipelines, Zuul. +* All CI tests running against a commit that releases the collection. If they do not pass, the collection MUST NOT be released. + +See :ref:`including_collection_ansible` if you plan on adding a new collection to the Ansible package. + +.. note:: + + Your collection must pass ``ansible-test sanity`` tests. See :ref:`testing_collections` for details. + + +.. _collection_versioning_and_deprecation: + +Collection versioning and deprecation +-------------------------------------- + +.. note:: + + Collections MUST adhere to `semantic versioning <https://semver.org/>`_. + +To preserve backward compatibility for users, every Ansible minor version series (5.1.x, 5.2.x, and so on) will keep the major version of a collection constant. For example, if Ansible 5.0.0 includes ``community.general`` 4.0.2, then each Ansible 5.X.x release will include the latest ``community.general`` 4.y.z release available at build time. Ansible 5.x.x will **never** include a ``community.general`` 5.y.x release, even if it is available. Major collection version changes will be included in the next Ansible major release (6.0.0 in this case). +Ensure that the current major release of your collection included in 6.0.0 receives at least bugfixes as long as new Ansible 6.X.X releases are produced. + +Since new minor releases are included, you can include new features, modules and plugins. You must make sure that you do not break backwards compatibility. See `semantic versioning <https://semver.org/>`_. for more details. This means in particular: + +* You can fix bugs in **patch** releases but not add new features or deprecate things. +* You can add new features and deprecate things in **minor** releases, but not remove things or change behavior of existing features. +* You can only remove things or make breaking changes in **major** releases. + +Ensure that if a deprecation is added in a collection version that is included in 5.x.y, the removal itself will only happen in a collection version included in 7.0.0 or later. +Ensure that the policy of releasing, versioning, and deprecation is announced to contributors and users in some way. For an example of how to do this, see `the announcement in community.general <https://github.com/ansible-collections/community.general/issues/582>`_. You could also do this in the collection README file. + +.. _collection_changelog: + +Collection changelogs +---------------------- + +Collections MUST include a changelog. To give a consistent feel for changelogs across collections and ensure changelogs exist for collections included in the ``ansible`` package, we suggest you use `antsibull-changelog <https://github.com/ansible-community/antsibull-changelog>`_ to maintain and generate this. + +Before releasing, verify the following for your changelogs: + +* All merged pull requests since the last release, except ones related to documentation and new modules/plugins, have :ref:`changelog fragments <collection_changelog_fragments>`. +* New module and plugin pull requests, except jinja2 test and filter plugins, do **not** need a changelog fragment, they are auto-detected by the changelog generator by their ``version_added`` value. +* All the fragments follow the :ref:`changelog entry format <collection_changelogs_how_to_format>`. + +Options for releasing a collection +----------------------------------- + +There are several approaches on how to release a collection. If you are not aware of which approach to use, ask in the ``#ansible-community`` IRC channel or the ``community`` Matrix channel. + +This section assumes that publishing the collection is done with `Zuul <https://github.com/ansible/project-config>`_ and that `antsibull-changelog <https://github.com/ansible-community/antsibull-changelog>`_ is used for the changelog. + +Releasing without release branches +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Use releasing without release branches when: + +* There are no prior major releases of the collection. +* There are no breaking changes introduced since the ``1.0.0`` release of the collection. + +See :ref:`collection_release_without_branches` for details. + +When there is a need to introduce breaking changes, you can switch to the next approach. + +Hybrid approach +^^^^^^^^^^^^^^^^^^^^^ + +In this approach, releases for the current major version are made from the ``main`` branch, while new releases for older major versions are made from release branches for these versions. + +Releasing with release branches +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Use releasing with release branches when breaking changes have been introduced. This approach is usually only used by the large community collections, ``community.general`` and ``community.network``. + +See :ref:`collection_release_with_branches` for details. + +.. toctree:: + :maxdepth: 1 + + collection_release_without_branches + collection_release_with_branches diff --git a/docs/docsite/rst/community/collection_contributors/collection_reviewing.rst b/docs/docsite/rst/community/collection_contributors/collection_reviewing.rst new file mode 100644 index 0000000..0eb203b --- /dev/null +++ b/docs/docsite/rst/community/collection_contributors/collection_reviewing.rst @@ -0,0 +1,74 @@ +.. _review_checklist: + +Review checklist for collection PRs +==================================== + +Use this section as a checklist reminder of items to review when you review a collection PR. + +Reviewing bug reports +---------------------- + +When users report bugs, verify the behavior reported. Remember always to be kind with your feedback. + +* Did the user make a mistake in the code they put in the Steps to Reproduce issue section? We often see user errors reported as bugs. +* Did the user assume an unexpected behavior? Ensure that the related documentation is clear. If not, the issue is useful to help us improve documentation. +* Is there a minimal reproducer? If not, ask the reporter to reduce the complexity to help pinpoint the issue. +* Is the issue a consequence of a misconfigured environment? +* If it seems to be a real bug, does the behaviour still exist in the most recent release or the development branch? +* Reproduce the bug, or if you do not have a suitable infrastructure, ask other contributors to reproduce the bug. + + +Reviewing suggested changes +--------------------------- + +When reviewing PRs, verify that the suggested changes do not: + +* Unnecessarily break backward compatibility. +* Bring more harm than value. +* Introduce non-idempotent solutions. +* Duplicate already existing features (inside or outside the collection). +* Violate the :ref:`Ansible development conventions <module_conventions>`. + + +Other standards to check for in a PR include: + +* A pull request MUST NOT contain a mix of bug fixes and new features that are not tightly related. If yes, ask the author to split the pull request into separate PRs. +* If the pull request is not a documentation fix, it must include a :ref:`changelog fragment <collection_changelog_fragments>`. Check the format carefully as follows: + + * New modules and plugins (that are not jinja2 filter and test plugins) do not need changelog fragments. + * For jinja2 filter and test plugins, check out the `special syntax for changelog fragments <https://github.com/ansible-community/antsibull-changelog/blob/main/docs/changelogs.rst#adding-new-roles-playbooks-test-and-filter-plugins>`_. + * The changelog content contains useful information for end users of the collection. + +* If new files are added with the pull request, they follow the `licensing rules <https://github.com/ansible-collections/overview/blob/main/collection_requirements.rst#licensing>`_. +* The changes follow the :ref:`Ansible documentation standards <developing_modules_documenting>` and the :ref:`style_guide`. +* The changes follow the :ref:`Development conventions <developing_modules_best_practices>`. +* If a new plugin is added, it is one of the `allowed plugin types <https://github.com/ansible-collections/overview/blob/main/collection_requirements.rst#modules-plugins>`_. +* Documentation, examples, and return sections use FQCNs for the ``M(..)`` :ref:`format macros <module_documents_linking>` when referring to modules. +* Modules and plugins from ansible-core use ``ansible.builtin.`` as an FQCN prefix when mentioned. +* When a new option, module, plugin, or return value is added, the corresponding documentation or return sections use ``version_added:`` containing the *collection* version in which they will be first released. + + * This is typically the next minor release, sometimes the next major release. For example: if 2.7.5 is the current release, the next minor release will be 2.8.0, and the next major release will be 3.0.0). + +* FQCNs are used for ``extends_documentation_fragment:``, unless the author is referring to doc_fragments from ansible-core. +* New features have corresponding examples in the :ref:`examples_block`. +* Return values are documented in the :ref:`return_block`. + + +Review tests in the PR +---------------------- +Review the following if tests are applicable and possible to implement for the changes included in the PR: + + +* Where applicable, the pull request has :ref:`testing_integration` and :ref:`testing_units`. +* All changes are covered. For example, a bug case or a new option separately and in sensible combinations with other options. +* Integration tests cover ``check_mode`` if supported. +* Integration tests check the actual state of the system, not only what the module reports. For example, if the module actually changes a file, check that the file was changed by using the ``ansible.builtin.stat`` module.. +* Integration tests check return values, if applicable. + + +Review for merge commits and breaking changes +--------------------------------------------- + +* The pull request does not contain merge commits. See the GitHub warnings at the bottom of the pull request. If merge commits are present, ask the author to rebase the pull request branch. +* If the pull request contains breaking changes, ask the author and the collection maintainers if it really is needed, and if there is a way not to introduce breaking changes. If breaking changes are present, they MUST only appear in the next major release and MUST NOT appear in a minor or patch release. The only exception is breaking changes caused by security fixes that are absolutely necessary to fix the security issue. + diff --git a/docs/docsite/rst/community/collection_contributors/collection_test_pr_locally.rst b/docs/docsite/rst/community/collection_contributors/collection_test_pr_locally.rst new file mode 100644 index 0000000..7b01e2d --- /dev/null +++ b/docs/docsite/rst/community/collection_contributors/collection_test_pr_locally.rst @@ -0,0 +1,67 @@ +.. _collection_pr_test: + +**************************** +How to test a collection PR +**************************** + +Reviewers and issue authors can verify a PR fixes the reported bug by testing the PR locally. + +.. contents:: + :local: + +.. _collection_prepare_environment: + +Prepare your environment +======================== + +We assume that you use Linux as a work environment (you can use a virtual machine as well) and have ``git`` installed. + + +1. :ref:`Install Ansible <installation_guide>` or ansible-core. + +2. Create the following directories in your home directory: + + .. code:: bash + + mkdir -p ~/ansible_collections/NAMESPACE/COLLECTION_NAME + + For example, if the collection is ``community.general``: + + .. code:: bash + + mkdir -p ~/ansible_collections/community/general + + If the collection is ``ansible.posix``: + + .. code:: bash + + mkdir -p ~/ansible_collections/ansible/posix + + +3. Clone the forked repository from the author profile to the created path: + + .. code:: bash + + git clone https://github.com/AUTHOR_ACC/COLLECTION_REPO.git ~/ansible_collections/NAMESPACE/COLLECTION_NAME + +4. Go to the cloned repository. + + .. code:: bash + + cd ~/ansible_collections/NAMESPACE/COLLECTION_NAME + +5. Checkout the PR branch (it can be retrieved from the PR's page): + + .. code:: bash + + git checkout pr_branch + + +Test the Pull Request +===================== + +1. Include `~/ansible_collections` in `COLLECTIONS_PATHS`. See :ref:`COLLECTIONS_PATHS` for details. + +2. Run your playbook using the PR branch and verify the PR fixed the bug. + +3. Give feedback on the pull request or the linked issue(s). diff --git a/docs/docsite/rst/community/collection_contributors/collection_unit_tests.rst b/docs/docsite/rst/community/collection_contributors/collection_unit_tests.rst new file mode 100644 index 0000000..e357358 --- /dev/null +++ b/docs/docsite/rst/community/collection_contributors/collection_unit_tests.rst @@ -0,0 +1,162 @@ + +.. _collection_unit_tests: + +****************************** +Add unit tests to a collection +****************************** + +This section describes all of the steps needed to add unit tests to a collection and how to run them locally using the ``ansible-test`` command. + +See :ref:`testing_units_modules` for more details. + +.. contents:: + :local: + +Understanding the purpose of unit tests +======================================== + +Unit tests ensure that a section of code (known as a ``unit``) meets its design requirements and behaves as intended. Some collections do not have unit tests but it does not mean they are not needed. + + +A ``unit`` is a function or method of a class used in a module or plugin. Unit tests verify that a function with a certain input returns the expected output. + +Unit tests should also verify when a function raises or handles exceptions. + +Ansible uses `pytest <https://docs.pytest.org/en/latest/>`_ as a testing framework. + +See :ref:`testing_units_modules` for complete details. + +Inclusion in the Ansible package `requires integration and/or unit tests <https://github.com/ansible-collections/overview/blob/main/collection_requirements.rst#requirements-for-collections-to-be-included-in-the-ansible-package>`_ You should have tests for your collection as well as for individual modules and plugins to make your code more reliable To learn how to get started with integration tests, see :ref:`collection_integration_tests`. + + +See :ref:`collection_prepare_local` to prepare your environment. + +.. _collection_unit_test_required: + +Determine if unit tests exist +============================= + +Ansible collection unit tests are located in the ``tests/units`` directory. + +The structure of the unit tests matches the structure of the code base, so the tests can reside in the ``tests/units/plugins/modules/`` and ``tests/units/plugins/module_utils`` directories. There can be sub-directories, if modules are organized by module groups. + +If you are adding unit tests for ``my_module`` for example, check to see if the tests already exist in the collection source tree with the path ``tests/units/plugins/modules/test_my_module.py``. + +Example of unit tests +===================== + +Let's assume that the following function is in ``my_module`` : + +.. code:: python + + def convert_to_supported(val): + """Convert unsupported types to appropriate.""" + if isinstance(val, decimal.Decimal): + return float(val) + + if isinstance(val, datetime.timedelta): + return str(val) + + if val == 42: + raise ValueError("This number is just too cool for us ;)") + + return val + +Unit tests for this function should, at a minimum, check the following: + +* If the function gets a ``Decimal`` argument, it returns a corresponding ``float`` value. +* If the function gets a ``timedelta`` argument, it returns a corresponding ``str`` value. +* If the function gets ``42`` as an argument, it raises a ``ValueError``. +* If the function gets an argument of any other type, it does nothing and returns the same value. + +To write these unit tests in collection is called ``community.mycollection``: + +1. If you already have your local environment :ref:`prepared <collection_prepare_local>`, go to the collection root directory. + + .. code:: bash + + cd ~/ansible_collection/community/mycollection + +2. Create a test file for ``my_module``. If the path does not exist, create it. + + .. code:: bash + + touch tests/units/plugins/modules/test_my_module.py + +3. Add the following code to the file: + + .. code:: python + + # -*- coding: utf-8 -*- + + from __future__ import (absolute_import, division, print_function) + __metaclass__ = type + + from datetime import timedelta + from decimal import Decimal + + import pytest + + from ansible_collections.community.mycollection.plugins.modules.my_module import ( + convert_to_supported, + ) + + # We use the @pytest.mark.parametrize decorator to parametrize the function + # https://docs.pytest.org/en/latest/how-to/parametrize.html + # Simply put, the first element of each tuple will be passed to + # the test_convert_to_supported function as the test_input argument + # and the second element of each tuple will be passed as + # the expected argument. + # In the function's body, we use the assert statement to check + # if the convert_to_supported function given the test_input, + # returns what we expect. + @pytest.mark.parametrize('test_input, expected', [ + (timedelta(0, 43200), '12:00:00'), + (Decimal('1.01'), 1.01), + ('string', 'string'), + (None, None), + (1, 1), + ]) + def test_convert_to_supported(test_input, expected): + assert convert_to_supported(test_input) == expected + + def test_convert_to_supported_exception(): + with pytest.raises(ValueError, match=r"too cool"): + convert_to_supported(42) + + See :ref:`testing_units_modules` for examples on how to mock ``AnsibleModule`` objects, monkeypatch methods (``module.fail_json``, ``module.exit_json``), emulate API responses, and more. + +4. Run the tests using docker: + + .. code:: bash + + ansible-test units tests/unit/plugins/modules/test_my_module.py --docker + + +.. _collection_recommendation_unit: + +Recommendations on coverage +=========================== + +Use the following tips to organize your code and test coverage: + +* Make your functions simple. Small functions that do one thing with no or minimal side effects are easier to test. +* Test all possible behaviors of a function including exception related ones such as raising, catching and handling exceptions. +* When a function invokes the ``module.fail_json`` method, passed messages should also be checked. + +.. seealso:: + + :ref:`testing_units_modules` + Unit testing Ansible modules + :ref:`developing_testing` + Ansible Testing Guide + :ref:`collection_integration_tests` + Integration testing for collections + :ref:`testing_integration` + Integration tests guide + :ref:`testing_collections` + Testing collections + :ref:`testing_resource_modules` + Resource module integration tests + :ref:`collection_pr_test` + How to test a pull request locally diff --git a/docs/docsite/rst/community/collection_contributors/test_index.rst b/docs/docsite/rst/community/collection_contributors/test_index.rst new file mode 100644 index 0000000..19a6142 --- /dev/null +++ b/docs/docsite/rst/community/collection_contributors/test_index.rst @@ -0,0 +1,14 @@ +.. _testing_collections_guide: + +********************************************** +Testing Collection Contributions +********************************************** + +This section focuses on the different tests a contributor should run on their collection PR. + +.. toctree:: + :maxdepth: 1 + + collection_test_pr_locally + collection_unit_tests + collection_integration_tests |