diff options
author | Daniel Baumann <daniel.baumann@progress-linux.org> | 2024-04-13 12:04:41 +0000 |
---|---|---|
committer | Daniel Baumann <daniel.baumann@progress-linux.org> | 2024-04-13 12:04:41 +0000 |
commit | 975f66f2eebe9dadba04f275774d4ab83f74cf25 (patch) | |
tree | 89bd26a93aaae6a25749145b7e4bca4a1e75b2be /ansible_collections/community/hashi_vault/docs/docsite | |
parent | Initial commit. (diff) | |
download | ansible-975f66f2eebe9dadba04f275774d4ab83f74cf25.tar.xz ansible-975f66f2eebe9dadba04f275774d4ab83f74cf25.zip |
Adding upstream version 7.7.0+dfsg.upstream/7.7.0+dfsg
Signed-off-by: Daniel Baumann <daniel.baumann@progress-linux.org>
Diffstat (limited to 'ansible_collections/community/hashi_vault/docs/docsite')
10 files changed, 1838 insertions, 0 deletions
diff --git a/ansible_collections/community/hashi_vault/docs/docsite/extra-docs.yml b/ansible_collections/community/hashi_vault/docs/docsite/extra-docs.yml new file mode 100644 index 000000000..82e329fcf --- /dev/null +++ b/ansible_collections/community/hashi_vault/docs/docsite/extra-docs.yml @@ -0,0 +1,14 @@ +--- +sections: + - title: Changelog + toctree: + - CHANGELOG + - title: Guides + toctree: + - filter_guide + - user_guide + - migration_hashi_vault_lookup + - about_hashi_vault_lookup + - lookup_guide + - contributor_guide + - localenv_developer_guide diff --git a/ansible_collections/community/hashi_vault/docs/docsite/links.yml b/ansible_collections/community/hashi_vault/docs/docsite/links.yml new file mode 100644 index 000000000..a0d100880 --- /dev/null +++ b/ansible_collections/community/hashi_vault/docs/docsite/links.yml @@ -0,0 +1,41 @@ +--- +# based on https://github.com/ansible-collections/collection_template/blob/main/docs/docsite/links.yml +# +# This will make sure that plugin and module documentation gets Edit on GitHub links +# that allow users to directly create a PR for this plugin or module in GitHub's UI. +# Remove this section if the collection repository is not on GitHub, or if you do not want this +# functionality for your collection. +edit_on_github: + repository: ansible-collections/community.hashi_vault + branch: main + # If your collection root (the directory containing galaxy.yml) does not coincide with your + # repository's root, you have to specify the path to the collection root here. For example, + # if the collection root is in a subdirectory ansible_collections/community/REPO_NAME + # in your repository, you have to set path_prefix to 'ansible_collections/community/REPO_NAME'. + path_prefix: '' + +# Here you can add arbitrary extra links. Please keep the number of links down to a +# minimum! Also please keep the description short, since this will be the text put on +# a button. +# +# Also note that some links are automatically added from information in galaxy.yml. +# The following are automatically added: +# 1. A link to the issue tracker (if `issues` is specified); +# 2. A link to the homepage (if `homepage` is specified and does not equal the +# `documentation` or `repository` link); +# 3. A link to the collection's repository (if `repository` is specified). + +extra_links: + - description: Discussion, Q&A, troubleshooting + url: https://github.com/ansible-collections/community.hashi_vault/discussions + +# Specify communication channels for your collection. We suggest to not specify more +# than one place for communication per communication tool to avoid confusion. +communication: + matrix_rooms: + - topic: General usage and support questions + room: '#users:ansible.im' + irc_channels: + - topic: General usage and support questions + network: Libera + channel: '#ansible' diff --git a/ansible_collections/community/hashi_vault/docs/docsite/rst/CHANGELOG.rst b/ansible_collections/community/hashi_vault/docs/docsite/rst/CHANGELOG.rst new file mode 100644 index 000000000..5223d4a97 --- /dev/null +++ b/ansible_collections/community/hashi_vault/docs/docsite/rst/CHANGELOG.rst @@ -0,0 +1,604 @@ +=================================== +community.hashi_vault Release Notes +=================================== + +.. contents:: Topics + + +v4.2.1 +====== + +Release Summary +--------------- + +This patch version updates the documentation for the ``vault_kv2_write`` module. There are no functional changes. + +v4.2.0 +====== + +Release Summary +--------------- + +This release contains a new module for KVv2 writes, and a new warning for duplicated term string options in the ``hashi_vault`` lookup. + +Deprecated Features +------------------- + +- hashi_vault lookup - in ``v5.0.0`` duplicate term string options will raise an exception instead of showing a warning (https://github.com/ansible-collections/community.hashi_vault/issues/356). + +Bugfixes +-------- + +- hashi_vault lookup - a term string with duplicate options would silently use the last value. The lookup now shows a warning on option duplication (https://github.com/ansible-collections/community.hashi_vault/issues/349). + +New Modules +----------- + +- vault_kv2_write - Perform a write operation against a KVv2 secret in HashiCorp Vault + +v4.1.0 +====== + +Release Summary +--------------- + +This release brings new generic ``vault_list`` plugins from a new contributor! +There are also some deprecation notices for the next major version, and some updates to documentation attributes. + +Deprecated Features +------------------- + +- ansible-core - support for ``ansible-core`` versions ``2.11`` and ``2.12`` will be dropped in collection version ``5.0.0``, making ``2.13`` the minimum supported version of ``ansible-core`` (https://github.com/ansible-collections/community.hashi_vault/issues/340). +- hvac - the minimum version of ``hvac`` to be supported in collection version ``5.0.0`` will be at least ``1.0.2``; this minimum may be raised before ``5.0.0`` is released, so please subscribe to the linked issue and look out for new notices in the changelog (https://github.com/ansible-collections/community.hashi_vault/issues/324). + +New Plugins +----------- + +Lookup +~~~~~~ + +- vault_list - Perform a list operation against HashiCorp Vault + +New Modules +----------- + +- vault_list - Perform a list operation against HashiCorp Vault + +v4.0.0 +====== + +Release Summary +--------------- + +The next major version of the collection includes previously announced breaking changes to some default values, and improvements to module documentation with attributes that describe the use of action groups and check mode support. + +Minor Changes +------------- + +- modules - all modules now document their action group and support for check mode in their attributes documentation (https://github.com/ansible-collections/community.hashi_vault/issues/197). + +Breaking Changes / Porting Guide +-------------------------------- + +- auth - the default value for ``token_validate`` has changed from ``true`` to ``false``, as previously announced (https://github.com/ansible-collections/community.hashi_vault/issues/248). +- vault_kv2_get lookup - as previously announced, the default value for ``engine_mount_point`` in the ``vault_kv2_get`` lookup has changed from ``kv`` to ``secret`` (https://github.com/ansible-collections/community.hashi_vault/issues/279). + +v3.4.0 +====== + +Release Summary +--------------- + +This release includes a new module, fixes (another) ``requests`` header issue, and updates some inaccurate documentation. +This is the last planned release before v4.0.0. + +Minor Changes +------------- + +- vault_pki_generate_certificate - the documentation has been updated to match the argspec for the default values of options ``alt_names``, ``ip_sans``, ``other_sans``, and ``uri_sans`` (https://github.com/ansible-collections/community.hashi_vault/pull/318). + +Bugfixes +-------- + +- connection options - the ``namespace`` connection option will be forced into a string to ensure cmpatibility with recent ``requests`` versions (https://github.com/ansible-collections/community.hashi_vault/issues/309). + +New Modules +----------- + +- vault_kv2_delete - Delete one or more versions of a secret from HashiCorp Vault's KV version 2 secret store + +v3.3.1 +====== + +Release Summary +--------------- + +No functional changes in this release, this provides updated filter documentation for the public docsite. + +v3.3.0 +====== + +Release Summary +--------------- + +With the release of ``hvac`` version ``1.0.0``, we needed to update ``vault_token_create``'s support for orphan tokens. +The collection's changelog is now viewable in the Ansible documentation site. + +Minor Changes +------------- + +- vault_token_create - creation or orphan tokens uses ``hvac``'s new v1 method for creating orphans, or falls back to the v0 method if needed (https://github.com/ansible-collections/community.hashi_vault/issues/301). + +v3.2.0 +====== + +Release Summary +--------------- + +This release brings support for the ``azure`` auth method, adds ``412`` to the default list of HTTP status codes to be retried, and fixes a bug that causes failures in token auth with ``requests>=2.28.0``. + +Minor Changes +------------- + +- community.hashi_vault collection - add support for ``azure`` auth method, for Azure service principal, managed identity, or plain JWT access token (https://github.com/ansible-collections/community.hashi_vault/issues/293). +- community.hashi_vault retries - `HTTP status code 412 <https://www.vaultproject.io/api-docs#412>`__ has been added to the default list of codes to be retried, for the new `Server Side Consistent Token feature <https://www.vaultproject.io/docs/faq/ssct#q-is-there-anything-else-i-need-to-consider-to-achieve-consistency-besides-upgrading-to-vault-1-10>`__ in Vault Enterprise (https://github.com/ansible-collections/community.hashi_vault/issues/290). + +Bugfixes +-------- + +- community.hashi_vault plugins - tokens will be cast to a string type before being sent to ``hvac`` to prevent errors in ``requests`` when values are ``AnsibleUnsafe`` (https://github.com/ansible-collections/community.hashi_vault/issues/289). +- modules - fix a "variable used before assignment" that cannot be reached but causes sanity test failures (https://github.com/ansible-collections/community.hashi_vault/issues/296). + +v3.1.0 +====== + +Release Summary +--------------- + +A default value that was set incorrectly will be corrected in ``4.0.0``. +A deprecation warning will be shown until then if the value is not specified explicitly. +This version also includes some fixes and improvements to the licensing in the collection, which does not affect any functionality. + +Deprecated Features +------------------- + +- vault_kv2_get lookup - the ``engine_mount_point option`` in the ``vault_kv2_get`` lookup only will change its default from ``kv`` to ``secret`` in community.hashi_vault version 4.0.0 (https://github.com/ansible-collections/community.hashi_vault/issues/279). + +Bugfixes +-------- + +- Add SPDX license headers to individual files (https://github.com/ansible-collections/community.hashi_vault/pull/282). +- Add missing ``BSD-2-Clause.txt`` file for BSD licensed content (https://github.com/ansible-collections/community.hashi_vault/issues/275). +- Use the correct GPL license for plugin_utils (https://github.com/ansible-collections/community.hashi_vault/issues/276). + +v3.0.0 +====== + +Release Summary +--------------- + +Version 3.0.0 of ``community.hashi_vault`` drops support for Ansible 2.9 and ansible-base 2.10. +Several deprecated features have been removed. See the changelog for the full list. + +Deprecated Features +------------------- + +- token_validate options - the shared auth option ``token_validate`` will change its default from ``true`` to ``false`` in community.hashi_vault version 4.0.0. The ``vault_login`` lookup and module will keep the default value of ``true`` (https://github.com/ansible-collections/community.hashi_vault/issues/248). + +Removed Features (previously deprecated) +---------------------------------------- + +- aws_iam auth - the deprecated alias ``aws_iam_login`` for the ``aws_iam`` value of the ``auth_method`` option has been removed (https://github.com/ansible-collections/community.hashi_vault/issues/194). +- community.hashi_vault collection - support for Ansible 2.9 and ansible-base 2.10 has been removed (https://github.com/ansible-collections/community.hashi_vault/issues/189). +- hashi_vault lookup - the deprecated ``[lookup_hashi_vault]`` INI config section has been removed in favor of the collection-wide ``[hashi_vault_collection]`` section (https://github.com/ansible-collections/community.hashi_vault/issues/179). + +v2.5.0 +====== + +Release Summary +--------------- + +This release finally contains dedicated KV plugins and modules, and an exciting new lookup to help use plugin values in module calls. +With that, we also have a guide in the collection docsite for migrating away from the ``hashi_vault`` lookup toward dedicated content. +We are also announcing that the ``token_validate`` option will change its default value in version 4.0.0. +This is the last planned release before 3.0.0. See the porting guide for breaking changes and removed features in the next version. + +Minor Changes +------------- + +- vault_login module & lookup - no friendly error message was given when ``hvac`` was missing (https://github.com/ansible-collections/community.hashi_vault/issues/257). +- vault_pki_certificate - add ``vault_pki_certificate`` to the ``community.hashi_vault.vault`` action group (https://github.com/ansible-collections/community.hashi_vault/issues/251). +- vault_read module & lookup - no friendly error message was given when ``hvac`` was missing (https://github.com/ansible-collections/community.hashi_vault/issues/257). +- vault_token_create - add ``vault_token_create`` to the ``community.hashi_vault.vault`` action group (https://github.com/ansible-collections/community.hashi_vault/issues/251). +- vault_token_create module & lookup - no friendly error message was given when ``hvac`` was missing (https://github.com/ansible-collections/community.hashi_vault/issues/257). +- vault_write - add ``vault_write`` to the ``community.hashi_vault.vault`` action group (https://github.com/ansible-collections/community.hashi_vault/issues/251). + +Deprecated Features +------------------- + +- token_validate options - the shared auth option ``token_validate`` will change its default from ``True`` to ``False`` in community.hashi_vault version 4.0.0. The ``vault_login`` lookup and module will keep the default value of ``True`` (https://github.com/ansible-collections/community.hashi_vault/issues/248). + +New Plugins +----------- + +Lookup +~~~~~~ + +- vault_ansible_settings - Returns plugin settings (options) +- vault_kv1_get - Get a secret from HashiCorp Vault's KV version 1 secret store +- vault_kv2_get - Get a secret from HashiCorp Vault's KV version 2 secret store + +New Modules +----------- + +- vault_kv1_get - Get a secret from HashiCorp Vault's KV version 1 secret store +- vault_kv2_get - Get a secret from HashiCorp Vault's KV version 2 secret store + +v2.4.0 +====== + +Release Summary +--------------- + +Our first content for writing to Vault is now live. + +New Plugins +----------- + +Lookup +~~~~~~ + +- vault_write - Perform a write operation against HashiCorp Vault + +New Modules +----------- + +- vault_write - Perform a write operation against HashiCorp Vault + +v2.3.0 +====== + +Release Summary +--------------- + +This release contains new plugins and modules for creating tokens and for generating certificates with Vault's PKI secrets engine. + +New Plugins +----------- + +Lookup +~~~~~~ + +- vault_token_create - Create a HashiCorp Vault token + +New Modules +----------- + +- vault_pki_generate_certificate - Generates a new set of credentials (private key and certificate) using HashiCorp Vault PKI +- vault_token_create - Create a HashiCorp Vault token + +v2.2.0 +====== + +Release Summary +--------------- + +This release contains a new lookup/module combo for logging in to Vault, and includes our first filter plugin. + +Minor Changes +------------- + +- The Filter guide has been added to the collection's docsite. + +New Plugins +----------- + +Filter +~~~~~~ + +- vault_login_token - Extracts the client token from a Vault login response + +Lookup +~~~~~~ + +- vault_login - Perform a login operation against HashiCorp Vault + +New Modules +----------- + +- vault_login - Perform a login operation against HashiCorp Vault + +v2.1.0 +====== + +Release Summary +--------------- + +The most important change in this release is renaming the ``aws_iam_login`` auth method to ``aws_iam`` and deprecating the old name. This release also announces the deprecation of Ansible 2.9 and ansible-base 2.10 support in 3.0.0. + +Deprecated Features +------------------- + +- Support for Ansible 2.9 and ansible-base 2.10 is deprecated, and will be removed in the next major release (community.hashi_vault 3.0.0) next spring (https://github.com/ansible-community/community-topics/issues/50, https://github.com/ansible-collections/community.hashi_vault/issues/189). +- aws_iam_login auth method - the ``aws_iam_login`` method has been renamed to ``aws_iam``. The old name will be removed in collection version ``3.0.0``. Until then both names will work, and a warning will be displayed when using the old name (https://github.com/ansible-collections/community.hashi_vault/pull/193). + +Removed Features (previously deprecated) +---------------------------------------- + +- the "legacy" integration test setup has been removed; this does not affect end users and is only relevant to contributors (https://github.com/ansible-collections/community.hashi_vault/pull/191). + +v2.0.0 +====== + +Release Summary +--------------- + +Version 2.0.0 of the collection drops support for Python 2 & Python 3.5, making Python 3.6 the minimum supported version. +Some deprecated features and settings have been removed as well. + +Breaking Changes / Porting Guide +-------------------------------- + +- connection options - there is no longer a default value for the ``url`` option (the Vault address), so a value must be supplied (https://github.com/ansible-collections/community.hashi_vault/issues/83). + +Removed Features (previously deprecated) +---------------------------------------- + +- drop support for Python 2 and Python 3.5 (https://github.com/ansible-collections/community.hashi_vault/issues/81). +- support for the following deprecated environment variables has been removed: ``VAULT_AUTH_METHOD``, ``VAULT_TOKEN_PATH``, ``VAULT_TOKEN_FILE``, ``VAULT_ROLE_ID``, ``VAULT_SECRET_ID`` (https://github.com/ansible-collections/community.hashi_vault/pull/173). + +v1.5.0 +====== + +Release Summary +--------------- + +This release includes a new action group for use with ``module_defaults``, and additional ways of specifying the ``mount_point`` option for plugins. +This will be the last ``1.x`` release. + +Minor Changes +------------- + +- add the ``community.hashi_vault.vault`` action group (https://github.com/ansible-collections/community.hashi_vault/pull/172). +- auth methods - Add support for configuring the ``mount_point`` auth method option in plugins via the ``ANSIBLE_HASHI_VAULT_MOUNT_POINT`` environment variable, ``ansible_hashi_vault_mount_point`` ansible variable, or ``mount_point`` INI section (https://github.com/ansible-collections/community.hashi_vault/pull/171). + +v1.4.1 +====== + +Release Summary +--------------- + +This release contains a bugfix for ``aws_iam_login`` authentication. + +Bugfixes +-------- + +- aws_iam_login auth method - fix incorrect use of ``boto3``/``botocore`` that prevented proper loading of AWS IAM role credentials (https://github.com/ansible-collections/community.hashi_vault/issues/167). + +v1.4.0 +====== + +Release Summary +--------------- + +This release includes bugfixes, a new auth method (``cert``), and the first new content since the collection's formation, the ``vault_read`` module and lookup plugin. +We're also announcing the deprecation of the ``[lookup_hashi_vault]`` INI section (which will continue working up until its removal only for the ``hashi_vault`` lookup), to be replaced by the ``[hashi_vault_collection]`` section that will apply to all plugins in the collection. + +Minor Changes +------------- + +- community.hashi_vault collection - add cert auth method (https://github.com/ansible-collections/community.hashi_vault/pull/159). + +Deprecated Features +------------------- + +- lookup hashi_vault - the ``[lookup_hashi_vault]`` section in the ``ansible.cfg`` file is deprecated and will be removed in collection version ``3.0.0``. Instead, the section ``[hashi_vault_collection]`` can be used, which will apply to all plugins in the collection going forward (https://github.com/ansible-collections/community.hashi_vault/pull/144). + +Bugfixes +-------- + +- aws_iam_login auth - the ``aws_security_token`` option was not used, causing assumed role credentials to fail (https://github.com/ansible-collections/community.hashi_vault/issues/160). +- hashi_vault collection - a fallback import supporting the ``retries`` option for ``urllib3`` via ``requests.packages.urllib3`` was not correctly formed (https://github.com/ansible-collections/community.hashi_vault/issues/116). +- hashi_vault collection - unhandled exception with ``token`` auth when ``token_file`` exists but is a directory (https://github.com/ansible-collections/community.hashi_vault/issues/152). + +New Plugins +----------- + +Lookup +~~~~~~ + +- vault_read - Perform a read operation against HashiCorp Vault + +New Modules +----------- + +- vault_read - Perform a read operation against HashiCorp Vault + +v1.3.2 +====== + +Release Summary +--------------- + +This release adds requirements detection support for Ansible Execution Environments. It also updates and adds new guides in our `collection docsite <https://docs.ansible.com/ansible/devel/collections/community/hashi_vault>`_. +This release also announces the dropping of Python 3.5 support in version ``2.0.0`` of the collection, alongside the previous announcement dropping Python 2.x in ``2.0.0``. + +Minor Changes +------------- + +- hashi_vault collection - add ``execution-environment.yml`` and a python requirements file to better support ``ansible-builder`` (https://github.com/ansible-collections/community.hashi_vault/pull/105). + +Deprecated Features +------------------- + +- hashi_vault collection - support for Python 3.5 will be dropped in version ``2.0.0`` of ``community.hashi_vault`` (https://github.com/ansible-collections/community.hashi_vault/issues/81). + +v1.3.1 +====== + +Release Summary +--------------- + +This release fixes an error in the documentation. No functionality is changed so it's not necessary to upgrade from ``1.3.0``. + +v1.3.0 +====== + +Release Summary +--------------- + +This release adds two connection-based options for controlling timeouts and retrying failed Vault requests. + +Minor Changes +------------- + +- hashi_vault lookup - add ``retries`` and ``retry_action`` to enable built-in retry on failure (https://github.com/ansible-collections/community.hashi_vault/pull/71). +- hashi_vault lookup - add ``timeout`` option to control connection timeouts (https://github.com/ansible-collections/community.hashi_vault/pull/100). + +v1.2.0 +====== + +Release Summary +--------------- + +This release brings several new ways of accessing options, like using Ansible vars, and addng new environment variables and INI config entries. +A special ``none`` auth type is also added, for working with certain Vault Agent configurations. +This release also announces the deprecation of Python 2 support in version ``2.0.0`` of the collection. + +Minor Changes +------------- + +- hashi_vault lookup - add ``ANSIBLE_HASHI_VAULT_CA_CERT`` env var (with ``VAULT_CACERT`` low-precedence fallback) for ``ca_cert`` option (https://github.com/ansible-collections/community.hashi_vault/pull/97). +- hashi_vault lookup - add ``ANSIBLE_HASHI_VAULT_PASSWORD`` env var and ``ansible_hashi_vault_password`` ansible var for ``password`` option (https://github.com/ansible-collections/community.hashi_vault/pull/96). +- hashi_vault lookup - add ``ANSIBLE_HASHI_VAULT_USERNAME`` env var and ``ansible_hashi_vault_username`` ansible var for ``username`` option (https://github.com/ansible-collections/community.hashi_vault/pull/96). +- hashi_vault lookup - add ``ansible_hashi_vault_auth_method`` Ansible vars entry to the ``proxies`` option (https://github.com/ansible-collections/community.hashi_vault/pull/86). +- hashi_vault lookup - add ``ansible_hashi_vault_ca_cert`` ansible var for ``ca_cert`` option (https://github.com/ansible-collections/community.hashi_vault/pull/97). +- hashi_vault lookup - add ``ansible_hashi_vault_namespace`` Ansible vars entry to the ``namespace`` option (https://github.com/ansible-collections/community.hashi_vault/pull/86). +- hashi_vault lookup - add ``ansible_hashi_vault_proxies`` Ansible vars entry to the ``proxies`` option (https://github.com/ansible-collections/community.hashi_vault/pull/86). +- hashi_vault lookup - add ``ansible_hashi_vault_role_id`` Ansible vars entry to the ``proxies`` option (https://github.com/ansible-collections/community.hashi_vault/pull/86). +- hashi_vault lookup - add ``ansible_hashi_vault_secret_id`` Ansible vars entry to the ``proxies`` option (https://github.com/ansible-collections/community.hashi_vault/pull/86). +- hashi_vault lookup - add ``ansible_hashi_vault_token_file`` Ansible vars entry to the ``token_file`` option (https://github.com/ansible-collections/community.hashi_vault/pull/95). +- hashi_vault lookup - add ``ansible_hashi_vault_token_path`` Ansible vars entry to the ``token_path`` option (https://github.com/ansible-collections/community.hashi_vault/pull/95). +- hashi_vault lookup - add ``ansible_hashi_vault_token_validate`` Ansible vars entry to the ``proxies`` option (https://github.com/ansible-collections/community.hashi_vault/pull/86). +- hashi_vault lookup - add ``ansible_hashi_vault_token`` Ansible vars entry to the ``proxies`` option (https://github.com/ansible-collections/community.hashi_vault/pull/86). +- hashi_vault lookup - add ``ansible_hashi_vault_url`` and ``ansible_hashi_vault_addr`` Ansible vars entries to the ``url`` option (https://github.com/ansible-collections/community.hashi_vault/pull/86). +- hashi_vault lookup - add ``ansible_hashi_vault_validate_certs`` Ansible vars entry to the ``validate_certs`` option (https://github.com/ansible-collections/community.hashi_vault/pull/95). +- hashi_vault lookup - add ``ca_cert`` INI config file key ``ca_cert`` option (https://github.com/ansible-collections/community.hashi_vault/pull/97). +- hashi_vault lookup - add ``none`` auth type which allows for passive auth via a Vault agent (https://github.com/ansible-collections/community.hashi_vault/pull/80). + +Deprecated Features +------------------- + +- hashi_vault collection - support for Python 2 will be dropped in version ``2.0.0`` of ``community.hashi_vault`` (https://github.com/ansible-collections/community.hashi_vault/issues/81). + +v1.1.3 +====== + +Release Summary +--------------- + +This release fixes a bug with ``userpass`` authentication and ``hvac`` versions 0.9.6 and higher. + +Bugfixes +-------- + +- hashi_vault - userpass authentication did not work with hvac 0.9.6 or higher (https://github.com/ansible-collections/community.hashi_vault/pull/68). + +v1.1.2 +====== + +Release Summary +--------------- + +This release contains the same functionality as 1.1.1. The only change is to mark some code as internal to the collection. If you are already using 1.1.1 as an end user you do not need to update. + +v1.1.1 +====== + +Release Summary +--------------- + +This bugfix release restores the use of the ``VAULT_ADDR`` environment variable for setting the ``url`` option. +See the PR linked from the changelog entry for details and workarounds if you cannot upgrade. + +Bugfixes +-------- + +- hashi_vault - restore use of ``VAULT_ADDR`` environment variable as a low preference env var (https://github.com/ansible-collections/community.hashi_vault/pull/61). + +v1.1.0 +====== + +Release Summary +--------------- + +This release contains a new ``proxies`` option for the ``hashi_vault`` lookup. + +Minor Changes +------------- + +- hashi_vault - add ``proxies`` option (https://github.com/ansible-collections/community.hashi_vault/pull/50). + +v1.0.0 +====== + +Release Summary +--------------- + +Our first major release contains a single breaking change that will affect only a small subset of users. No functionality is removed. See the details in the changelog to determine if you're affected and if so how to transition to remediate. + +Breaking Changes / Porting Guide +-------------------------------- + +- hashi_vault - the ``VAULT_ADDR`` environment variable is now checked last for the ``url`` parameter. For details on which use cases are impacted, see (https://github.com/ansible-collections/community.hashi_vault/issues/8). + +v0.2.0 +====== + +Release Summary +--------------- + +Several backwards-compatible bugfixes and enhancements in this release. +Some environment variables are deprecated and have standardized replacements. + +Minor Changes +------------- + +- Add optional ``aws_iam_server_id`` parameter as the value for ``X-Vault-AWS-IAM-Server-ID`` header (https://github.com/ansible-collections/community.hashi_vault/pull/27). +- hashi_vault - ``ANSIBLE_HASHI_VAULT_ADDR`` environment variable added for option ``url`` (https://github.com/ansible-collections/community.hashi_vault/issues/8). +- hashi_vault - ``ANSIBLE_HASHI_VAULT_AUTH_METHOD`` environment variable added for option ``auth_method`` (https://github.com/ansible-collections/community.hashi_vault/issues/17). +- hashi_vault - ``ANSIBLE_HASHI_VAULT_ROLE_ID`` environment variable added for option ``role_id`` (https://github.com/ansible-collections/community.hashi_vault/issues/20). +- hashi_vault - ``ANSIBLE_HASHI_VAULT_SECRET_ID`` environment variable added for option ``secret_id`` (https://github.com/ansible-collections/community.hashi_vault/issues/20). +- hashi_vault - ``ANSIBLE_HASHI_VAULT_TOKEN_FILE`` environment variable added for option ``token_file`` (https://github.com/ansible-collections/community.hashi_vault/issues/15). +- hashi_vault - ``ANSIBLE_HASHI_VAULT_TOKEN_PATH`` environment variable added for option ``token_path`` (https://github.com/ansible-collections/community.hashi_vault/issues/15). +- hashi_vault - ``namespace`` parameter can be specified in INI or via env vars ``ANSIBLE_HASHI_VAULT_NAMESPACE`` (new) and ``VAULT_NAMESPACE`` (lower preference) (https://github.com/ansible-collections/community.hashi_vault/issues/14). +- hashi_vault - ``token`` parameter can now be specified via ``ANSIBLE_HASHI_VAULT_TOKEN`` as well as via ``VAULT_TOKEN`` (the latter with lower preference) (https://github.com/ansible-collections/community.hashi_vault/issues/16). +- hashi_vault - add ``token_validate`` option to control token validation (https://github.com/ansible-collections/community.hashi_vault/pull/24). +- hashi_vault - uses new AppRole method in hvac 0.10.6 with fallback to deprecated method with warning (https://github.com/ansible-collections/community.hashi_vault/pull/33). + +Deprecated Features +------------------- + +- hashi_vault - ``VAULT_ADDR`` environment variable for option ``url`` will have its precedence lowered in 1.0.0; use ``ANSIBLE_HASHI_VAULT_ADDR`` to intentionally override a config value (https://github.com/ansible-collections/community.hashi_vault/issues/8). +- hashi_vault - ``VAULT_AUTH_METHOD`` environment variable for option ``auth_method`` will be removed in 2.0.0, use ``ANSIBLE_HASHI_VAULT_AUTH_METHOD`` instead (https://github.com/ansible-collections/community.hashi_vault/issues/17). +- hashi_vault - ``VAULT_ROLE_ID`` environment variable for option ``role_id`` will be removed in 2.0.0, use ``ANSIBLE_HASHI_VAULT_ROLE_ID`` instead (https://github.com/ansible-collections/community.hashi_vault/issues/20). +- hashi_vault - ``VAULT_SECRET_ID`` environment variable for option ``secret_id`` will be removed in 2.0.0, use ``ANSIBLE_HASHI_VAULT_SECRET_ID`` instead (https://github.com/ansible-collections/community.hashi_vault/issues/20). +- hashi_vault - ``VAULT_TOKEN_FILE`` environment variable for option ``token_file`` will be removed in 2.0.0, use ``ANSIBLE_HASHI_VAULT_TOKEN_FILE`` instead (https://github.com/ansible-collections/community.hashi_vault/issues/15). +- hashi_vault - ``VAULT_TOKEN_PATH`` environment variable for option ``token_path`` will be removed in 2.0.0, use ``ANSIBLE_HASHI_VAULT_TOKEN_PATH`` instead (https://github.com/ansible-collections/community.hashi_vault/issues/15). + +Bugfixes +-------- + +- hashi_vault - ``mount_point`` parameter did not work with ``aws_iam_login`` auth method (https://github.com/ansible-collections/community.hashi_vault/issues/7) +- hashi_vault - fallback logic for handling deprecated style of auth in hvac was not implemented correctly (https://github.com/ansible-collections/community.hashi_vault/pull/33). +- hashi_vault - parameter ``mount_point`` does not work with JWT auth (https://github.com/ansible-collections/community.hashi_vault/issues/29). +- hashi_vault - tokens without ``lookup-self`` ability can't be used because of validation (https://github.com/ansible-collections/community.hashi_vault/issues/18). + +v0.1.0 +====== + +Release Summary +--------------- + +Our first release matches the ``hashi_vault`` lookup functionality provided by ``community.general`` version ``1.3.0``. + diff --git a/ansible_collections/community/hashi_vault/docs/docsite/rst/about_hashi_vault_lookup.rst b/ansible_collections/community/hashi_vault/docs/docsite/rst/about_hashi_vault_lookup.rst new file mode 100644 index 000000000..84bb0faec --- /dev/null +++ b/ansible_collections/community/hashi_vault/docs/docsite/rst/about_hashi_vault_lookup.rst @@ -0,0 +1,90 @@ +.. _ansible_collections.community.hashi_vault.docsite.about_hashi_vault_lookup: + +******************************** +About the ``hashi_vault`` lookup +******************************** + +This page explains the past, present, and future of the ``hashi_vault`` :ref:`lookup plugin <ansible_collections.community.hashi_vault.hashi_vault_lookup>`. + +The ``hashi_vault`` lookup is the oldest Vault-related content in Ansible. It was included in pre-collections Ansible (<2.10). As a result, it's the most used plugin for Vault, and the one most people are familiar with. + +At this time, we recommend using newer content in the collection, and we believe all use cases for ``hashi_vault`` have been covered by newer plugins. To understand the history, continue reading this document. For help with migration, :ref:`the hashi_vault migration guide <ansible_collections.community.hashi_vault.docsite.migration_hashi_vault_lookup>` has you covered. + +.. contents:: + :local: + :depth: 2 + +Synopsis +======== + +The short summary is: + +* The ``hashi_vault`` lookup does several jobs and uses some patterns that we would like to change, but are well-entrenched. +* The ``community.hashi_vault`` collection is developing and releasing new plugins and modules that are more tightly-scoped and will offer individual coverage for many use cases that the ``hashi_vault`` lookup has been used for. +* At this time, there are no plans to deprecate the ``hashi_vault`` lookup, but it is also unlikely that it will receive new features specific to that lookup (improvements in shared code like new auth methods are included automatically). +* As more plugins are released in the collection, we will be adding specific migration guidance to this page with examples. + +The long story +============== + +``hashi_vault`` lookup considerations +------------------------------------- + +Due to the history of the ``hashi_vault`` lookup plugin, it does many jobs. It is versatile, but sometimes unintuitive. + +The ``hashi_vault`` lookup plugin performs three main tasks: + +* authentication, taking parameters for various login types, performing a login, and acquiring a token with which it can make additional calls to Vault. +* a generic read operation, which allows it to read any kind of Vault path, without having to be written with that type of path in mind. +* transforming responses that look like ``kv2`` responses into simpler responses that resemble those from ``kv1``. + +Reading secrets is the most common use case, with the ``kv`` (key/value) store built into Vault as by far the most common secret store. Most implementations use v2 of the ``kv`` store. To make reading v2 ``kv`` secrets easy, the lookup plugin assumes that you're probably trying to read a ``kv`` secret, and tries to infer if the response is from ``kv2``, because the responses from version 2 include metadata and have the secret value additionally wrapped in another structure. The lookup plugin seeks to make ``kv2`` responses look more like responses from version 1. + +Since the ``kv`` store has one or more key/value pairs in each secret, the lookup also supports a non-standard suffix in its path that can be used to access a value belonging to one specific key, via the ``:keyname`` syntax. While this is useful to provide a compact way to access a single secret value (admittedly a very common use case), it complicates the implementation and leads to bad habits. + +For example, it became common to see people use many lookup invocations with the same path, each with a different ``:keyname``, to access multiple values within a single secret, but this is quite wasteful, as it does a separate login and secret lookup, all to return the same value, and the key dereferencing is done client side. Further, dereferencing can be done directly in Jinja where it's more clear what's going on, using the ``.key`` or ``['key']`` syntax. + +One last idiosyncrasy of the plugin is its support for supplying all of its parameters in the term string. This looks compact, but it greatly complicates the processing of plugin options. At the time that this lookup was created, many other lookups allowed options to be supplied in the term string, but it has since been considered an anti-pattern, and has been deprecated/removed from core plugins. + +Another downside of this is that it prevents us from effectively re-using the authentication token in cases when multiple term strings are supplied, directly or via ``with_community.hashi_vault.hashi_vault``, and as a result this type of usage results in a new login for each term. In newer lookups, we can take advantage of a single login to perform multiple operations. + +All of these considerations make sense in context, but it somewhat muddles the purpose of the lookup: + +* If a response from a completely different endpoint ended up looking like a ``kv2`` response, it would return an unexpected result. +* If you try to give the path of a ``kv2`` secret directly, it will not work unless you insert a ``/data/`` component into the path, in order to match the *API path* rather than the path people are usually familiar with. +* If you want the metadata returned along with a ``kv2`` response, you cannot get it. +* Other features of ``kv2`` like secret versioning cannot directly be used, unless you modify the URL, which is error prone and unintuitive. +* Getting access to the token created by the internal login, in order to re-use it, is not possible. + +How we are addressing the considerations +---------------------------------------- + +The built-in authentication support will be kept, and in fact it has been moved to shared utilities within the collection, so that all plguins and modules can share the functionality, and work consistently. That makes it easier to test new and existing auth methods, easier to add new ones (which automaticallly become part of all existing content), and easier to add new content, because authentication does not need to be reimplemented. + +In addition, it is now possible to perform a login directly and return the token, for general re-use, via the ``community.hashi_vault.vault_login`` :ref:`module <ansible_collections.community.hashi_vault.vault_login_module>` and :ref:`lookup plugin <ansible_collections.community.hashi_vault.vault_login_lookup>`. + +Generic read (not ``kv`` specific) is still important functionality, so we have the ``community.hashi_vault.vault_read`` :ref:`module <ansible_collections.community.hashi_vault.vault_read_module>` and :ref:`lookup plugin <ansible_collections.community.hashi_vault.vault_read_lookup>` to provide that without trying to infer whether the response is from a specific backend. + +Since reading from the ``kv`` store is by far the most common use case, we have dedicated content for that: + +* ``community.hashi_vault.vault_kv1_get`` :ref:`module <ansible_collections.community.hashi_vault.vault_kv1_get_module>` +* ``community.hashi_vault.vault_kv2_get`` :ref:`module <ansible_collections.community.hashi_vault.vault_kv2_get_module>` +* ``community.hashi_vault.vault_kv1_get`` :ref:`lookup <ansible_collections.community.hashi_vault.vault_kv1_get_lookup>` +* ``community.hashi_vault.vault_kv2_get`` :ref:`lookup <ansible_collections.community.hashi_vault.vault_kv2_get_lookup>` + +The dictionary dereferencing via ``:keyname`` syntax *will not be supported* in other content. That will be achieved in Jinja via: + +* dot syntax ``.keyname`` +* lookup syntax ``['keyname']`` +* specialized filters in some circumstances, such as the ``vault_login_token`` :ref:`filter <ansible_collections.community.hashi_vault.docsite.filter_guide.vault_login_token>`. + +Parameters via term string *will not be supported* in other lookups. Its use is discouraged by core developers, and steps have already been taken in core to remove the functionality where it still exists, however it will remain in the ``hashi_vault`` plugin for backwards compatibility and because it is likely to still be in use in a lot of places. + +The future of the ``hashi_vault`` lookup +---------------------------------------- + +There are no plans currently to deprecate or remove the ``hashi_vault`` plugin. It is likely that it will stay indefinitely, for backwards compatibility and because so much functionality has been moved to shared code that very little maintenance is required to keep it. This decision may be revisited if circumstances change. + +That being said, we will encourage the use of newer content that has functionality with a tighter scope and is expected to receive updates and enchancements as appropriate. + +New features and functionality are unlikely to be added or accepted in the ``hashi_vault`` lookup, except for the ones that come for "free", like new auth methods (these require no code changes to the plugin itself). diff --git a/ansible_collections/community/hashi_vault/docs/docsite/rst/contributor_guide.rst b/ansible_collections/community/hashi_vault/docs/docsite/rst/contributor_guide.rst new file mode 100644 index 000000000..ed5259fcd --- /dev/null +++ b/ansible_collections/community/hashi_vault/docs/docsite/rst/contributor_guide.rst @@ -0,0 +1,299 @@ +.. _ansible_collections.community.hashi_vault.docsite.contributor_guide: + +***************** +Contributor guide +***************** + +This guide aims to help anyone who wishes to contribute to the ``community.hashi_vault`` collection. + +.. note:: + + This guide can be improved with your help! Open a `GitHub issue in the repository <https://github.com/ansible-collections/community.hashi_vault/issues>`_ or contribute directly by following the instructions below. + + +.. contents:: + :local: + :depth: 3 + + +Quick start +=========== + +#. Log into your GitHub account. +#. Fork the `ansible-collections/community.hashi_vault repository <https://github.com/ansible-collections/community.hashi_vault>`_ by clicking the **Fork** button in the upper right corner. This will create a fork in your own account. +#. Clone the repository locally, following :ref:`the example instructions here <hacking_collections>` (but replace ``general`` with ``hashi_vault``). **Pay special attention to the local path structure** of the cloned repository as described in those instructions (for example ``ansible_collections/community/hashi_vault``). +#. As mentioned on that page, commit your changes to a branch, push them to your fork, and create a pull request (GitHub will automatically prompt you to do so when you look at your repository). +#. :ref:`See the guidance on Changelogs <community_changelogs>` and include a :ref:`changelog fragment <changelogs_how_to>` if appropriate. + +Contributing documentation +========================== + +Additions to the collection documentation are very welcome! We have three primary types of documentation, each with their own syntax and rules. + +README and other markdown files +------------------------------- + +Markdown files (those with the extension ``.md``) can be found in several directories within the repository. These files are primarily aimed at developers and those browsing the repository, to explain or give context to the other files nearby. + +The main exception to the above is the ``README.md`` in the repository root. This file is more important because it provides introductory information and links for anyone browsing the repository, both on GitHub and on the collection's `Ansible Galaxy page <https://galaxy.ansible.com/community/hashi_vault>`_. + +Markdown files can be previewed natively in GitHub, so they are easy to validate by reviewers, and there are no specific tests that need to run against them. + +Your IDE or code editor may also be able to preview these files. For example `Visual Studio Code has built-in markdown preview <https://code.visualstudio.com/docs/languages/markdown#_markdown-preview>`_. + +Module and plugin documentation +------------------------------- + +This type of documentation gets generated from structured YAML, inside of a Python string. It is included in the same code that it's documenting, or in a separate Python file, such as a doc fragment. Please see the :ref:`module format and documentation guidance <developing_modules_documenting>` for more information. + +This type of documentation is highly structured and tested with ``ansible-test sanity``. Full instructions are available on the :ref:`testing module documentation <testing_module_documentation>` page. + +Additionally, the docsite build on pull requests (or built locally) will include module and plugin documentation as well. See the next section for details. + +Collection docsite +------------------ + +The collection docsite is what you are reading now. It is written in reStructuredText (RST) format and published on the :ref:`ansible_documentation` site. This is where we have long-form documentation that doesn't fit into the other two categories. + +If you are considering adding an entirely new document here it may be best to open a GitHub issue first to discuss the idea and how best to organize it. + +Refer to the :ref:`Ansible style guide <style_guide>` for all submissions to the collection docsite. + +RST files for the docsite are in the ``docs/docsite/rst/`` directory. Some submissions may also require edits to ``docs/docsite/extra-docs.yml``. + +When a pull request is submitted which changes the collection's documentation, a new docsite will be generated and published to a temporary site, and a bot will post a comment on the PR with a link. This will let you see the rendered docs to help with spotting formatting errors. + +It's also possible to build the docs locally, by installing some extra Python requirements and running the build script: + +.. code-block:: shell-session + + $ pushd docs/preview + $ pip install -r requirements.txt + $ ./build.sh + +You can then find the generated HTML in ``docs/preview/build/html`` and can open them locally in your browser. + +Running tests locally +===================== + +If you're making anything more than very small or one-time changes, run the tests locally to avoid having to push a commit for each thing, and waiting for the CI to run tests. + +First, :ref:`review the guidance on testing collections <testing_collections>`, as it applies to this collection as well. + +Integration Tests +----------------- + +Unlike other collections, we require an `integration_config.yml <https://docs.ansible.com/ansible/latest/dev_guide/testing_integration.html#integration-config-yml>`_ file for properly running integration tests, as the tests require external dependencies (like a Vault server) and they need to know where to find those dependencies. + +If you have contributed to this collection or to the ``hashi_vault`` lookup plugin in the past, you might remember that the integration tests used to download, extract, and run a Vault server during the course of the tests, by default. This *legacy mode* is **no longer available**. + + +.. _ansible_collections.community.hashi_vault.docsite.contributor_guide.localenv_docker: + +Docker Compose localenv +^^^^^^^^^^^^^^^^^^^^^^^ + +The recommended way to run the tests has Vault and other dependencies running in their own containers, set up via docker-compose, and the integration tests run in their own container separately. + +We have a pre-defined "localenv" setup role for this purpose. + +Usage +""""" + +For ease of typing / length of commands, we'll enter the role directory first: + +.. code-block:: shell-session + + $ pushd tests/integration/targets/setup_localenv_docker + +This localenv has both Ansible collection and Python requirements, so let's get those out of the way: + +.. code-block:: shell-session + + $ pip install -r files/requirements/requirements.txt -c files/requirements/constraints.txt + $ ansible-galaxy collection install -r files/requirements/requirements.yml + +To set up your docker-compose environment with all the defaults: + +.. code-block:: shell-session + + $ ./setup.sh + +The setup script does the following: + +#. Template a ``docker-compose.yml`` for the project. +#. Generate a private key and self-signed certificate for Vault. +#. Template a Vault config file. +#. Bring down the existing compose project. +#. Bring up the compose project as defined by the vars (specified or defaults). +#. Template an ``integration_config.yml`` file that has all the right settings for integration tests to connect. +#. Copy the integration config to the correct location *if there isn't already one there* (it won't overwrite, in case you had customized changes). + +With your containers running, you can now run the tests in docker (after returning back to the collection root): + +.. code-block:: shell-session + + $ popd + $ ansible-test integration --docker default --docker-network hashi_vault_default -v + +The ``--docker-network`` part is important, because it ensures that the Ansible test container is in the same network as the dependency containers, that way the test container can reach them by their container names. The network name, ``hashi_vault_default`` comes from the default docker-compose project name used by this role (``hashi_vault``). See the :ref:`customization section <ansible_collections.community.hashi_vault.docsite.contributor_guide.localenv_docker_customization>` for more information. + +Running ``setup.sh`` again can be used to re-deploy the containers, or if you prefer you can use the generated ``files/.output/<project_name>/docker-compose.yml`` directly with local tools. + +If running again, remember to manually copy the contents of newly generated ``files/.output/integration_config.yml`` to the integration root, or delete the file in the root before re-running setup so that it copies the file automatically. + +.. _ansible_collections.community.hashi_vault.docsite.contributor_guide.localenv_docker_customization: + +Customization +""""""""""""" + +``setup.sh`` passes any additional params you send it to the ``ansible-playbook`` command it calls, so you can customize variables with the standard ``--extra-vars`` (or ``-e``) option. There are many advanced scenarios possible, but a few things you might want to override: + +* ``vault_version`` -- can target any version of Vault for which a docker container exists (this is the container's tag), defaults to ``latest`` +* ``docker_compose`` (defaults to ``clean`` but could be set to ``up``, ``down``, or ``none``) + * ``up`` -- similar to running ``docker-compose up`` (no op if the project is running as it should) + * ``down`` -- similar to ``docker-compose down`` (destroys the project) + * ``clean`` -- (default) similar to ``docker-compose down`` followed by ``docker-compose up`` + * ``none`` -- does the other tasks, including templating, but does not bring the project up or down. With this option, the ``community.docker`` collection is not required. +* ``vault_crypto_force`` -- by default this is ``false`` so if the cert and key exist they won't be regenerated. Setting to ``true`` will overwrite them. +* ``vault_port_http``, ``vault_port_https``, ``proxy_port`` -- all of the ports are exposed to the host, so if you already have any of the default ports in use on your host, you may need to override these. +* ``vault_container_name``, ``proxy_container_name`` -- these are the names for their respective containers, which will also be the DNS names used within the container network. In case you have the default names in use you may need to override these. +* ``docker_compose_project_name`` -- unlikely to need to be changed, but it affects the name of the docker network which will be needed for your ``ansible-test`` invocation, so it's worth mentioning. For example, if you set this to ``ansible_hashi_vault`` then the docker network name will be ``ansible_hashi_vault_default``. + +.. _ansible_collections.community.hashi_vault.docsite.contributor_guide.contributing_auth_methods: + +Contributing auth methods +========================= + +In this collection, auth methods are shared among all plugins and modules rather than being re-implemented in each one. This saves the effort of re-inventing the wheel, prevents test bloat by having to test functionality across auth methods, and provides a consistent experience. + +File location & scope +--------------------- + +Auth methods are implemented as classes in ``module_utils``, in a file named ``plugins/module_utils/_auth_method_<method_name>.py``. The leading underscore indicates that the module util is private to the collection and that it is not intended to be used outside the collection; this lets us make changes as needed without needing to release a new major version, and clearly indicates to would-be downstream users that they should not rely on these utils outside content within the collection. + +In addition, all auth method module utils within the collection must contain a comment explaining this, such as: + +.. code-block:: python + + # FOR INTERNAL COLLECTION USE ONLY + # The interfaces in this file are meant for use within the community.hashi_vault collection + # and may not remain stable to outside uses. Changes may be made in ANY release, even a bugfix release. + # See also: https://github.com/ansible/community/issues/539#issuecomment-780839686 + # Please open an issue if you have questions about this. + +It is best to look at `existing auth methods <https://github.com/ansible-collections/community.hashi_vault/tree/main/plugins/module_utils>`_ to get a feel for how they are implemented. + +Class anatomy +------------- + +Each auth method class should be named ``HashiVaultAuthMethod<MethodName>`` and inherit from ``HashiVaultAuthMethodBase``. + +The base class provides some common functionality, like standardizing a way to emit warnings and providing a common function for validating required options. + +An auth method must run the base class's ``__init__`` function. + +It must implement two methods: + +* ``validate()`` -- this method does everything it can to ensure that the requirements are met for performing authentication with this particular auth method. This may include checking for required options, validating the values of those options, pulling in additional information and context from the environment, preparing that information for use by ``authenticate()``, etc. Generally speaking, it should not contact Vault, and should minimize reliance on external sources and services, but that is a guideline and the details will depend on the specifics of the auth method in question. ``validate()`` raises an exception if validation fails. If it succeeds, nothing is returned. +* ``authenticate(client, use_token=False)`` -- this method performs the actual authentication, and it returns the API result of the authentication (which will include the token, lease information, etc.). The HVAC client object is passed in, as well an optional parameter ``use_token`` which specifies whether the client should have its token field set to the result of authentication (typically this is desired). + +The auth method class should also contain two fields: + +* ``NAME`` -- the name of the auth method. +* ``OPTIONS`` -- a list containing the name of every option that may be used by the auth method, including optional options; this list should not include the ``auth_method`` option. + +Raising exceptions and warnings +------------------------------- + +Because auth methods are shared among both Ansible modules and Ansible plugins, any exceptions raised must be applicable to both. Standard Python exceptions like ``KeyError`` can be raised if they appropriate. + +In situations where you would normally raise ``AnsibleError`` (in plugins), or call ``module.fail_json()`` (in modules), you may raise ``HashiVaultValueError`` with your error message. Plugins and modules in this collection should expect this type and act accordingly. + +Similarly for warnings, because plugins and modules implement warnings differently, module util code that needs to warn takes a warning callback, and this is true for auth methods as well. + +The base class provides a ``warn()`` method that handles calling the callback specified at class init, so a simple ``self.warn()`` can be used in auth method code. + +Accessing options +----------------- + +Because auth methods are shared among both Ansible modules and Ansible plugins, which do not access options in the same way, this collection implements a class called ``HashiVaultOptionAdapter``. This class provides a standard interface for accessing option values in code that must work in both plugins and modules. + +It implements the following methods: + +* ``get_option(key)`` -- gets the option with the specified name. Raises ``KeyError`` if the option is not present. +* ``get_option_default(key, default=None)`` -- gets the option with the specified name. If it's not present, returns the value of ``default``. +* ``set_option(key, value)`` -- sets the value of the specified option ``key`` to ``value``. +* ``set_option_default(key, default=None)`` -- returns the value of the option ``key``. If the key is not present, sets its value to ``default`` and returns that value. +* ``has_option(key)`` -- returns ``True`` if the specified option *is present* (``None`` value counts as present). +* ``set_options(**kwargs)`` -- sets options to the key/value pairs specified in ``kwargs``. +* ``get_options(*args)`` -- returns a dict of the option names specified in ``args``. +* ``get_filtered_options(filter, *args)`` -- returns a dict of the option names specified in ``args``, if the callable ``filter`` (which has ``key`` and ``value`` passed into it) returns ``True`` for the given key/value pair. +* ``get_filled_options(*args)`` -- returns a dict of the option names specified in ``*args`` that are not ``None``. + +The authenticator +----------------- + +The ``HashiVaultAuthenticator`` class is how most of the content in the collection will handle authentication, rather than having to directly references each individual auth method. As a result, ``_authenticator.py`` needs to be modified for every new auth method, because it imports and directly references each class. See `the implementation of this class <https://github.com/ansible-collections/community.hashi_vault/blob/main/plugins/module_utils/_authenticator.py>`_ to find the places that need to be modified. + +Auth method options and documentation +------------------------------------- + +Because auth methods are shared among collection content, their options are documented in doc_fragment plugins. Because many options end up being shared among many auth methods (for example ``role_id``, ``username``, ``password``), we do not have a separate doc fragment for each auth method, as this would end up with duplicated option documentation. + +Instead, all of the options for auth methods are in ``plugins/doc_fragments/auth.py``. + +This contains the standard ``DOCUMENTATION`` field, as well as a ``PLUGINS`` field. The reason for this split is that there are certain parts of the documentation that are only applicable to plugins; namely the ``env``, ``ini``, and ``vars`` entries. + +``DOCUMENTATION`` should contains all fields common to both, like ``description``, ``type``, ``version_added``, ``required``, etc., while anything plugin-specific goes in ``PLUGINS``. + +Since plugins and modules will reference the doc fragments, it's not usually required to modify the docstrings in the content directly; if it seems necessary, please raise an issue to discuss. + +Wherever possible, we should provide ``env``, ``ini``, and ``vars`` alternatives for specifying options, to give maximum flexibility for plugins. Occasionally, these won't make sense, like providing ``token`` (a sensitive value) in ``ini``. + +When deciding to implement new options for an auth method, consider whether existing options can or should be reused. If a new option is needed, consider scoping its name to the auth method, in order to differentiate it from current or future option names that could be confusing in another context. + +For example ``cert_auth_public_key`` and ``cert_auth_private_key`` were named that way to prevent them being confused with other certificate options that relate to the Vault connection, or other contexts where specific plugins or modules might need key pairs. + +Testing auth methods +-------------------- + +Because auth methods are shared across the collection, we want them to be very well tested. Auth methods have both unit and integration tests, and the combination of those should give us high confidence that the methods work as intended. + +Unit tests +^^^^^^^^^^ + +Unit tests allow us to check some of the functionality that is difficult to test effectively in integration tests, like checking that every possible combination of options behaves as it should, or simulating conditions that we can't easily reproduce. The coverage of various scenarios should be extensive, and the details of which, or how complex they are, will depend on the intricacies of the auth method itself. Looking at existing examples is highly recommended. + +A pytest fixture is provided to load fixtures files that contain sample Vault API responses. Using these allows for mocking of the HVAC authentication calls within the unit tests. + +Integration tests +^^^^^^^^^^^^^^^^^ + +Our integration tests provide a running Vault server, and with that we can set up any auth methods we want (in theory). In practice, auth methods often require external services to integrate with. When possible, we should consider setting up such external services so that we can create a meaningful, real life integration and test it. + +Often however, this is not possible, or difficult. We must consider that tests are not only run in CI, but should be able to be run locally as well. + +Mocking integrations +"""""""""""""""""""" + +We have implemented `MMock (Monster Mock) <https://github.com/jmartin82/mmock>`_ in our integration test setup to help with this. This server is setup to proxy its requests to the test Vault server, but you can write configurations that allow it to return different data for specific requests. By carefully constructing these responses, we can simulate the Vault API's response to login requests for specific auth methods, and also simulate its failures. With that, we can then run integration tests that hopefully provide us some assurance that we are implementing it correctly. + +Testing plugin and module Usage +""""""""""""""""""""""""""""""" + +Auth methods are usable from modules and plugins, so integration tests for an auth method must test it via both plugins and modules. + +We provide custom modules and plugins specifically for testing auth methods within the integration tests. These are simplified implementations but they use the common code that should be used by all content, and they can be set to return some useful information about the login process. See the existing tests for details. + +Test coverage +^^^^^^^^^^^^^ + +In CI, we use CodeCov to track coverage. We also set some specific "tags" in coverage, and one of those is to tag individual auth methods as targets for integration tests. This happens automatically in CI, however new auth methods need an entry into ``codecov.yml`` that maps the coverage flag to the file where the auth method is implemented. For example: + +.. code:: yaml + + flags: + target_auth_aws_iam: + paths: + - plugins/module_utils/_auth_method_aws_iam.py diff --git a/ansible_collections/community/hashi_vault/docs/docsite/rst/filter_guide.rst b/ansible_collections/community/hashi_vault/docs/docsite/rst/filter_guide.rst new file mode 100644 index 000000000..8cee2b987 --- /dev/null +++ b/ansible_collections/community/hashi_vault/docs/docsite/rst/filter_guide.rst @@ -0,0 +1,139 @@ +.. _ansible_collections.community.hashi_vault.docsite.filter_guide: + +Filter guide +============ + +.. note:: + + Filter Plugins are now included with other :ref:`plugin documentation <plugins_in_community.hashi_vault>`. + + +.. contents:: Filters + +.. _ansible_collections.community.hashi_vault.docsite.filter_guide.vault_login_token: + +``vault_login_token`` filter +---------------------------- + +.. versionadded:: 2.2.0 + +The ``vault_login_token`` filter extracts the token value from the structure returned by a Vault token creation operation, such as those returned by the ``community.hashi_vault.vault_login`` :ref:`module <ansible_collections.community.hashi_vault.vault_login_module>` or :ref:`lookup plugin <ansible_collections.community.hashi_vault.vault_login_lookup>`, or the ``community.hashi_vault.vault_token_create`` :ref:`module <ansible_collections.community.hashi_vault.vault_token_create_module>` or :ref:`lookup plugin <ansible_collections.community.hashi_vault.vault_token_create_lookup>`. + +The filter takes an optional parameter ``optional_field`` with defaults to ``login``. If this field exists in the input dictionary, then the value of that field is taken the be the login response, rather than the input dictionary itself. + +The purpose of this is primarily to deal with the difference between the output of lookup plugins (which return the login response directly) and modules, which return the login response in a ``login`` field in its return. + +Here is a sample login response: + +.. code-block:: json + + { + "auth": { + "accessor": "mQewzgKRx5Yui1h1eMemJlMu", + "client_token": "s.drgLxu6ZtttSVn5Zkoy0huMR", + "entity_id": "8a74ffd3-f71b-8ebe-7942-610428051ea9", + "lease_duration": 3600, + "metadata": { + "username": "testuser" + }, + "orphan": true, + "policies": [ + "alt-policy", + "default", + "userpass-policy" + ], + "renewable": true, + "token_policies": [ + "alt-policy", + "default", + "userpass-policy" + ], + "token_type": "service" + }, + "data": null, + "lease_duration": 0, + "lease_id": "", + "renewable": false, + "request_id": "511e8fba-83f0-4b7e-95ea-770aa19c1957", + "warnings": null, + "wrap_info": null + } + +The token that we want to extract is in ``auth.client_token``. + +Here's an example usage with the ``vault_login`` module and lookup. + +.. code-block:: yaml+jinja + + - name: Set defaults + vars: + ansible_hashi_vault_url: https://vault:9801/ + ansible_hashi_vault_auth_method: userpass + ansible_hashi_vault_username: user + ansible_hashi_vault_password: "{{ lookup('env', 'MY_SECRET_PASSWORD') }}" + module_defaults: + community.hashi_vault.vault_login: + url: '{{ ansible_hashi_vault_url }}' + auth_method: '{{ ansible_hashi_vault_auth_method }}' + username: '{{ ansible_hashi_vault_username }}' + password: '{{ ansible_hashi_vault_password }}' + block: + - name: Perform a login with a lookup and display the token + vars: + login_response: "{{ lookup('community.hashi_vault.vault_login') }}" + debug: + msg: "The token is {{ login_response | community.hashi_vault.vault_login_token }}" + + - name: Perform a login with a module + community.hashi_vault.vault_login: + register: login_response + + - name: Display the token + debug: + msg: "The token is {{ login_response | community.hashi_vault.vault_login_token }}" + +Which produces: + +.. code-block:: ansible-output + + TASK [Perform a login with a lookup and display the token] ******************************** + ok: [localhost] => { + "msg": "s.drgLxu6ZtttSVn5Zkoy0huMR" + } + + TASK [Perform a login with a module] ***************************************************** + ok: [localhost] => {"changed": true, "login": {"auth": { "accessor": "mQewzgKRx5Yui1h1eMemJlMu", + "client_token": "s.drgLxu6ZtttSVn5Zkoy0huMR", "entity_id": "8a74ffd3-f71b-8ebe-7942-610428051ea9", + "lease_duration": 3600, "metadata": {"username": "testuser"}, "orphan": true, "policies": + ["alt-policy", "default", "userpass-policy"], "renewable": true, "token_policies": ["alt-policy", + "default", "userpass-policy"], "token_type": "service"}, "data": null, "lease_duration": 0, + "lease_id": "", "renewable": false, "request_id": "511e8fba-83f0-4b7e-95ea-770aa19c1957", + "warnings": null, "wrap_info": null}} + } + + TASK [Display the token] ***************************************************************** + ok: [localhost] => { + "msg": "s.drgLxu6ZtttSVn5Zkoy0huMR" + } + +This filter is the equivalent of reading into the dictionary directly, but it has the advantages of providing semantic meaning and automatically working against the differing output of modules and lookups. + +.. code-block:: yaml+jinja + + --- + lookup_token: "{{ lookup_login_response['auth']['client_token'] }}" + module_token: "{{ module_login_response['login']['auth']['client_token'] }}" + +The ``optional_field`` can be changed in case you've put the raw login response in some other structure, but you could also dereference that directly instead. + +.. code-block:: yaml+jinja + + --- + my_data: + something: somedata + vault_login: "{{ lookup_login_response }}" + + token_from_param: "{{ my_data | community.hashi_vault.vault_login_token(optional_field='vault_login') }}" + token_from_deref: "{{ my_data['vault_login'] | community.hashi_vault.vault_login_token }}" + # if the optional field doesn't exist, the dictionary itself is still checked + unused_optional: "{{ my_data['vault_login'] | community.hashi_vault.vault_login_token(optional_field='missing') }}" diff --git a/ansible_collections/community/hashi_vault/docs/docsite/rst/localenv_developer_guide.rst b/ansible_collections/community/hashi_vault/docs/docsite/rst/localenv_developer_guide.rst new file mode 100644 index 000000000..0b111d564 --- /dev/null +++ b/ansible_collections/community/hashi_vault/docs/docsite/rst/localenv_developer_guide.rst @@ -0,0 +1,93 @@ +.. _ansible_collections.community.hashi_vault.docsite.localenv_developer_guide: + +************************ +localenv developer guide +************************ + +A "localenv" role in this collection sets up the external dependencies required to run the integration tests. The idea is to provide a pre-packaged way for a contributor to set up their local environment in a consistent, repeatable way. + +.. note:: + + This guide is a work-in-progress and is **very** light on details. For the time being, it's best to open an issue in the repository to discuss it if you're thinking of a new localenv. Looking at ``setup_localenv_docker`` should also be helpful as it's the most complete one to date. + + +.. contents:: + :local: + :depth: 2 + + +Required external dependencies +============================== + +HashiCorp Vault +--------------- + +A Vault server is required for the integration tests. Using `Vault Dev Server Mode <https://www.vaultproject.io/docs/concepts/dev-server>`_ is recommended as it's the easiest and fastest way to get a server started. + +A unencrypted (plain HTTP) listener is *required* for our purposes as most of the tests will expect to connect that way. + +To run the tests that deal specifically with TLS/HTTPS access, you must start the Vault server with a TLS enabled listener. The TLS address:port, and the CA cert (or the cert itself if self-signed) must be supplied. + +The **root token** of the Vault server is needed, as the integration tests make changes to Vault's configuration, and expect to have that token available to do so. It's possible to let Vault generate the token on startup and then retrieve it but it may be easiest to pre-generate one and pass it into Vault, via the ``-dev-root-token-id`` option or ``VAULT_DEV_ROOT_TOKEN_ID`` environment variable (see `Dev Options <https://www.vaultproject.io/docs/commands/server#dev-options>`_). + +Relevant ``integration_config.yml`` variables +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "var", "example", "description" + :widths: 15, 20, 65 + + "``vault_test_server_http``", "``http://myvault:8200``", "The full HTTP URL of your Vault test server." + "``vault_test_server_https``", "``https://myvault:8300``", "The full HTTPS URL of your Vault test server." + "``vault_dev_root_token_id``", "``3ee9a1f7-f115-4f7c-90a3-d3c73361bcb5``", "The root token used to authenticate to Vault." + "``vault_version``", "``1.7.3``", "The version of Vault in use (usually this is written by a localenv, so a value set manually is not used anywhere)." + "``vault_cert_content``", "``-----BEGIN CERTIFICATE-----<snip>``", "The public cert of the CA that signed the cert used for Vault's TLS listener (or the cert itself if self-signed)." + + +Proxy server +------------ + +A proxy server is used to test the proxy connectivity options. + +In theory any proxy supporting http/s targets could be used for this purpose, but `tinyproxy <https://github.com/tinyproxy/tinyproxy>`_ is recommended for being, well.. tiny, as well as easy to configure and run, and available in package managers and containers. + +Relevant ``integration_config.yml`` variables +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "var", "example", "description" + :widths: 15, 20, 65 + + "``vault_proxy_server``", "``http://proxy:8080``", "The full HTTP URL of your proxy server." + + +MMock server +------------ + +`MMock (short for Monster Mock) <https://github.com/jmartin82/mmock>`_ is an HTTP server designed for mocking HTTP responses. It can also transparently proxy through to a real server. We use it to proxy our test Vault server while intercepting certain API calls to Vault and returning mocked responses. + +This is useful for Vault integrations that are more difficult to set up in our CI environment. + +For example, we use this for testing the ``aws_iam`` auth method, since we don't have an AWS account we can use and configure and connect to from our GitHub CI. + +For these integration tests, all Vault interactions are directed to MMock rather than directly to Vault, and we pre-configure MMock to respond to the relevant calls in a way that models a real Vault server's success and failure responses. + +Relevant ``integration_config.yml`` variables +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "var", "example", "description" + :widths: 15, 20, 65 + + "``vault_mmock_server_http``", "``http://mmock:8900``", "The full HTTP URL of the MMock server." + + +localenv role conventions +========================= + +* Use ``files/.output`` to hold generated artifacts. +* Anything generated should be in a ``.gitignore``; conversely anything not in a ``.gitignore`` should not be overwritten or modified by this process. That is, there should be no changes to git status that arise from this. +* Consider providing a ``setup.sh`` to avoid having to manually run ``ansible-`` commands. It should ideally operate correctly regardless of the current working directory. +* Generate a usable ``integration_config.yml`` that allows for using the result of the localenv. Generate it within the role output, not outside the role. Copy it to the right location, but do not overwrite an existing one. +* If the role has external dependencies, try to codify those in file(s) that can be used by the right tool, like ``requirements.yml`` for ``ansible-galaxy``, etc. +* localenv roles are meant to run **outside** of the ``ansible-test`` environment, but they can make (re)use of other roles. diff --git a/ansible_collections/community/hashi_vault/docs/docsite/rst/lookup_guide.rst b/ansible_collections/community/hashi_vault/docs/docsite/rst/lookup_guide.rst new file mode 100644 index 000000000..da73854b4 --- /dev/null +++ b/ansible_collections/community/hashi_vault/docs/docsite/rst/lookup_guide.rst @@ -0,0 +1,99 @@ +.. _ansible_collections.community.hashi_vault.docsite.lookup_guide: + +************ +Lookup guide +************ + +This guide is not a comprehensive listing of included lookup plugins and how to use them, rather it is intended to explain the role of the lookup plugins in ``community.hashi_vault`` and how they are they used, especially when compared to modules of the same name. + +For information about the ``hashi_vault`` lookup specifically, see :ref:`this page that covers it in detail <ansible_collections.community.hashi_vault.docsite.about_hashi_vault_lookup>`. + +.. contents:: + :local: + :depth: 2 + + +Lookups and writes +================== + +Most Ansible lookups perform read-only, non-destructive operations. They are run in templating, they generally *return* values, and they **do not run differently in check mode** (that is they do the same thing they would in normal mode, even if that means changing something). However, some lookups do change state, sometimes by performing write operations. For example, the ``password`` :ref:`lookup <ansible_collections.ansible.builtin.password_lookup>` writes a generated password to a file, to act as a sort of cache, and the ``pipe`` :ref:`lookup <ansible_collections.ansible.builtin.pipe_lookup>` runs an arbitrary shell command so it could easily write or change state. + +Writes in Vault +--------------- + +Operations that perform writes in Vault are not limited to the obvious ones such as writing a secret value, creating a policy, or enabling a new auth method. + +Any operation that creates a token for example, such as any login operation, is also a write; tokens use storage in Vault and having too many active tokens is a common cause of performance problems. + +Additionally, some values in Vault can only be "read" at the moment of their creation, and so the only way to retrieve such a value, is to get it as a response from the "write" that created it. A common example is AppRole secret IDs. + +The way this relates to Ansible and this collection, is that we may have lookup plugins that either unintuitively perform writes (like ``vault_login``), or appear inappropriate to exist as lookups in the first place, like the planned ``vault_write`` lookup. + +The reason for this is that the we often consider these operations to be logical "read" operations, like performing a login, and want to use their results in other expressions. + +Something like ``vault_write`` does not always fit that description, because you could use it in a way that is clearly an explicit write, for example you could create a new policy with the lookup. But there are times it may be appropriate to use it in lookup semantics, like when "retrieving" (really creating) a new secret ID for an approle. + +When considering built-in support for auth methods, any auth method other than ``token`` or ``none`` makes every lookup, even ``vault_read``, into something that's changing state and performing a write within Vault. This actually applies to many modules too, even when using check mode. + +How to reason about when to use lookups +--------------------------------------- + +Because there is potential for writes in any lookup, it is very important to carefully consider when you are using a lookup vs. a module/other plugin. Check mode has no effect on lookups, so there is potential to perform many writes within your check mode run, but maybe sometimes you want that, for example if you're performing a ``vault_login`` via lookup to retrieve a token to use in your module calls, you may want that to still happen in check mode so that your module calls can properly check the things they need to. + +Some modules that are read focused, like the ``vault_read`` module, when used with auth other than ``token`` or ``none``, will still perform an internal login even in check mode, so this is still another consideration. + +Lookups and lazy templating +--------------------------- + +The capacity for lookups to perform writes or change state is exacerbated by Ansible's "lazy" templating, if not used carefully. + +Consider the following example: + +.. code-block:: yaml+jinja + + - vars: + token: "{{ lookup('community.hashi_vault.vault_login', auth_method='userpass', username='user', password='pass') | community.hashi_vault.vault_login_token }}" + secret: "{{ lookup('community.hashi_vault.vault_read', 'secrets/data/my-secret', token=token) }}" + value_a: "{{ secret.data.data.a }}" + value_b: "{{ secret.data.data.b }}" + ansible.builtin.debug: + msg: "Secret value A is '{{ value_a }}' while value B is '{{ value_b }}'." + +Since templating is recursive and evaluated lazily, this will unfortunately *not* result in a single login, reusing the token to perform a single secret read, which is then used is dictionary lookups. + +Instead, evaluation of ``value_a`` and ``value_b`` will *each* cause separate evaluation of ``secret``, so that lookup will be performed twice, and *each of those lookups* will cause a separate evaluation of ``token``, which will perform two separate logins, resulting in two tokens being created, and two reads of the exact same secret being performed. + +If you combine this with loops, or reusing vars over multiple tasks, you can very quickly multiply the number of requests being made to Vault, and in the case of writes, the number of objects being created. + +Tasks can be better for this, since they execute when encountered without being accidentally repeated, and the values they return are static. + +.. code-block:: yaml+jinja + + - name: login + community.hashi_vault.vault_login: + auth_method: userpass + username: user + password: pass + register: login + + - name: get secret + community.hashi_vault.vault_read: + token: '{{ login | community.hashi_vault.vault_login_token }}' + path: 'secrets/data/my-secret' + register: secret + + - vars: + value_a: "{{ secret.data.data.data.a }}" + value_b: "{{ secret.data.data.data.b }}" + ansible.builtin.debug: + msg: "Secret value A is '{{ value_a }}' while value B is '{{ value_b }}'." + +This example will do a single login and secret lookup, even though it is more verbose. It also means the ``secret`` and ``login`` variables can be re-used in more tasks without performing additional requests to Vault. + +Another thing to consider in both of the examples is that tasks run *per host*, so you may be multiplying the requests yet again. + +In the lookup example, those requests all happen on the controller, and in the module example, they happen on the remote host unless the play or task is targeted locally. + +In both cases, you may *want* to make those requests per host, because some of the variables involved in the lookups may rely on per-host values, like differing authentication, different secret paths, even different Vault servers altogether, or in the case of certain access restrictions, you may need the remote host to make the connection rather than the controller. + +But if all of your secret access is intended to be from the controller, and the requests do not depend on host-level variables, you can potentially cut your requests by a lot, by using ``run_once``, or making Vault calls in a separate play that only targets ``localhost`` and using ``ansible.builtin.set_fact``, or via other methods. diff --git a/ansible_collections/community/hashi_vault/docs/docsite/rst/migration_hashi_vault_lookup.rst b/ansible_collections/community/hashi_vault/docs/docsite/rst/migration_hashi_vault_lookup.rst new file mode 100644 index 000000000..9c206bcde --- /dev/null +++ b/ansible_collections/community/hashi_vault/docs/docsite/rst/migration_hashi_vault_lookup.rst @@ -0,0 +1,369 @@ +.. _ansible_collections.community.hashi_vault.docsite.migration_hashi_vault_lookup: + +***************************************** +Migrating from the ``hashi_vault`` lookup +***************************************** + +This is a guide for migrating from the ``hashi_vault`` :ref:`lookup plugin <ansible_collections.community.hashi_vault.hashi_vault_lookup>` to newer content in this collection. + +To understand why, please see :ref:`this page describing the plugin's history and future <ansible_collections.community.hashi_vault.docsite.about_hashi_vault_lookup>`. + +.. contents:: + :local: + :depth: 2 + +A note about lookups vs. modules +================================ + +Since the ``hashi_vault`` plugin is a lookup, it is often most straightforward to replace its use with other lookups. There was no module option available previously, however there is now. + +Although it may be more involved, consider each use case to determine if a module is more appropriate. + +For more information, see the :ref:`lookup guide <ansible_collections.community.hashi_vault.docsite.lookup_guide>`. + +General changes +=============== + +This section will cover some general differences not related to specific scenarios. + +Options: direct vs. term string +------------------------------- + +For a long time, the ``hashi_vault`` lookup took all of its options as ``name=value`` strings inside the term string, so you would do a lookup with a single string that looked something like ``secret/data/path auth_method=userpass username=my_user password=somepass``. + +This way of passing options is discouraged, and ``hashi_vault`` was updated (before this collection existed) to support passing options as individual keyword arguments. The term string method was kept for backward compatibility. + +.. note:: + + None of the other lookups in this collection will support the old style term string syntax, so changing to direct options is highly recommended. + +If your existing lookups use options in the term string, you may want to first change to direct use of options before trying to change the plugin, **especially if you intend to continue using lookups instead of modules**. + +Examples of the term string style: + +.. code-block:: yaml+jinja + + - name: Term string style + vars: + user: my_user + pass: '{{ my_secret_password }}' + mount: secret + relpath: path + ansible.builtin.debug: + msg: + - "Static: {{ lookup('community.hashi_vault.hashi_vault', 'secret/data/path auth_method=userpass username=my_user password=somepass') }}" + - "Variables: {{ lookup('community.hashi_vault.hashi_vault', mount ~ '/data/' ~ path ~ ' auth_method=userpass username=' ~ user ~ ' password=' ~ pass) }}" + # note these necessary but easy to miss spaces ^ ^ + +And the same lookups converted to direct options: + +.. code-block:: yaml+jinja + + - name: Direct option style + vars: + user: my_user + pass: '{{ my_secret_password }}' + mount: secret + relpath: path + ansible.builtin.debug: + msg: + - "Static: {{ lookup('community.hashi_vault.hashi_vault', 'secret/data/path', auth_method='userpass', username='my_user', password='somepass') }}" + - "Variables: {{ lookup('community.hashi_vault.hashi_vault', mount ~ '/data/' ~ path, auth_method='userpass', username=user, password=pass) }}" + + +Key dereferencing +----------------- + +For these examples we will assume our result dictionary has this structure: + +.. code-block:: yaml + + key_1: value1 + 'key-2': 2 + 'key three': three + + +``hashi_vault`` also supported a dictionary dereferencing syntax with colon ``:``, so it was common to see this: + +.. code-block:: yaml+jinja + + - ansible.builtin.debug: + msg: + - "KV1 (key1): {{ lookup('community.hashi_vault.hashi_vault', 'kv1_mount/path/to/secret:key_1') }}" + - "KV2 (key1): {{ lookup('community.hashi_vault.hashi_vault', 'kv2_mount/data/path/to/secret:key_1') }}" + +With the above syntax, only the *value* of ``key_1`` is returned. Note that ``key three`` could not have been retrieved this way, because the space was the delimiter for the term string options. + +.. note:: + + The colon ``:`` syntax is not supported in any other lookups in the collection, and its use is discouraged. + +**Colon** ``:`` **use does not correspond to any server-side filtering or other optimization**, so other than compact syntax there is there no advantage to using it. + +The colon ``:`` syntax could always have been replaced by directly dereferencing in the Jinja2 template. Direct dereferencing can be done with the Jinja2 dot ``.`` syntax (which has restrictions on the key names) or via square brackets ``[]``, like so (KV version does not matter): + +.. code-block:: yaml+jinja + + - vars: + k1: key_1 + k2: key-2 + k3: key three + ansible.builtin.debug: + msg: + - "KV1 (key1, dot): {{ lookup('community.hashi_vault.hashi_vault', 'kv1_mount/path/to/secret').key_1 }}" + - "KV1 (key1, [ ]): {{ lookup('community.hashi_vault.hashi_vault', 'kv1_mount/path/to/secret')['key_1'] }}" + - "KV1 (var1, [ ]): {{ lookup('community.hashi_vault.hashi_vault', 'kv1_mount/path/to/secret')[k1] }}" + - "KV1 (key2, [ ]): {{ lookup('community.hashi_vault.hashi_vault', 'kv1_mount/path/to/secret')['key-2'] }}" + - "KV1 (var2, [ ]): {{ lookup('community.hashi_vault.hashi_vault', 'kv1_mount/path/to/secret')[k2] }}" + - "KV1 (key3, [ ]): {{ lookup('community.hashi_vault.hashi_vault', 'kv1_mount/path/to/secret')['key three'] }}" + - "KV1 (var3, [ ]): {{ lookup('community.hashi_vault.hashi_vault', 'kv1_mount/path/to/secret')[k3] }}" + +Note that only ``key_1`` could use the dot ``.`` syntax because the allowed characters for that are limited to those allowed for Python symbols. Variables also cannot be used with dot ``.`` access. + +Furthermore, the colon ``:`` syntax encouraged multiple lookups to the same secret only for the purpose of getting different keys, leading to multiple identical requests to Vault. **The above example also suffers from this**. + +A more DRY approach might look like this: + +.. code-block:: yaml+jinja + + - vars: + secret: "{{ lookup('community.hashi_vault.hashi_vault', 'kv1_mount/path/to/secret') }}" + k1: key_1 + k2: key-2 + k3: key three + ansible.builtin.debug: + msg: + - "KV1 (key1, dot): {{ secret.key_1 }}" + - "KV1 (key1, [ ]): {{ secret['key_1'] }}" + - "KV1 (var1, [ ]): {{ secret[k1] }}" + - "KV1 (key2, [ ]): {{ secret['key-2'] }}" + - "KV1 (var2, [ ]): {{ secret[k2] }}" + - "KV1 (key3, [ ]): {{ secret['key three'] }}" + - "KV1 (var3, [ ]): {{ secret[k3] }}" + +This looks a lot better, and it is from a readability perspective, but **in fact it will operate exactly the same way**, making a new request on every reference to ``secret``. This is due to lazy template evaluation in Ansible, and is discussed in more detail in the :ref:`lookup guide <ansible_collections.community.hashi_vault.docsite.lookup_guide>`. This can be remedied by either using ``ansible.builtin.set_fact`` to set the ``secret`` variable, or by using a module to do the read. + +If you have extensive use of the colon ``:`` syntax, updating it before moving onto other plugins is recommended. + +Return format +------------- + +.. note:: + + The ``return_format`` option will not be supported in other plugins. It is recommended to replace it with Jinja2 if you are using it currently. + +The ``hashi_vault`` lookup takes a ``return_format`` option that defaults to ``dict``. The lookup always looks for a ``data`` field (see the :ref:`KV response details <ansible_collections.community.hashi_vault.docsite.migration_hashi_vault_lookup.kv_response>` for more information), and that is what is returned by default. + +The ``raw`` value for ``return_format`` gives the raw API response from the request. This can be used to get the metadata from a KV2 request for example, which is usually stripped off, or it can be used to read from a non-KV path whose response happens to look like a KV response (with one or more ``data`` structures), and gets interpreted as one as a result. + +For reading non-KV paths :ref:`other options are available <ansible_collections.community.hashi_vault.docsite.migration_hashi_vault_lookup.non_kv_replacements>`. + +For getting access to KV2 metadata, see the section on :ref:`KV replacements <ansible_collections.community.hashi_vault.docsite.migration_hashi_vault_lookup.kv_replacements>`. + +The ``return_format`` option can also be set to ``values`` to return a list of the dictionary's values. + +This can be replaced with Jinja2. We will use our example secret again: + +.. code-block:: yaml + + key_1: value1 + 'key-2': 2 + 'key three': three + +And look at uses with ``return_format``: + +.. code-block:: yaml+jinja + + # show a list of values, ['value1', 2, 'three'] + - ansible.builtin.debug: + msg: + - "KV1: {{ lookup('community.hashi_vault.hashi_vault', 'kv1_mount/path/to/secret', return_format='values') }}" + + # run debug once for each value + - ansible.builtin.debug: + msg: "{{ item }}" + loop: "{{ query('community.hashi_vault.hashi_vault', 'kv1_mount/path/to/secret', return_format='values') }}" + +We can do the same with Jinja2: + +.. code-block:: yaml+jinja + + # show a list of values + - ansible.builtin.debug: + msg: + - "KV1: {{ lookup('community.hashi_vault.hashi_vault', 'kv1_mount/path/to/secret').values() | list }}" + + # run debug once for each value + - ansible.builtin.debug: + msg: "{{ item }}" + loop: "{{ lookup('community.hashi_vault.hashi_vault', 'kv1_mount/path/to/secret').values() | list }}" + + +Vault KV reads +============== + +The most common use for the ``hashi_vault`` lookup is reading secrets from the KV secret store. + +.. code-block:: yaml+jinja + + - ansible.builtin.debug: + msg: + - "KV1: {{ lookup('community.hashi_vault.hashi_vault', 'kv1_mount/path/to/secret') }}" + - "KV2: {{ lookup('community.hashi_vault.hashi_vault', 'kv2_mount/data/path/to/secret') }}" + +The return value of both of those is the dictionary of the key/value pairs in the secret, with no additional information from the API response, nor the metadata (in the case of KV2). + +.. _ansible_collections.community.hashi_vault.docsite.migration_hashi_vault_lookup.kv_response: + +KV1 and KV2 response structure +------------------------------ + +Under the hood, the return format of version 1 and version 2 of the KV store differs. + +Here is a sample KV1 response: + +.. code-block:: json + + { + "auth": null, + "data": { + "Key1": "val1", + "Key2": "val2" + }, + "lease_duration": 2764800, + "lease_id": "", + "renewable": false, + "request_id": "e26a7521-e512-82f1-3998-7cc494f14e86", + "warnings": null, + "wrap_info": null + } + +And a sample KV2 response: + +.. code-block:: json + + { + "auth": null, + "data": { + "data": { + "Key1": "val1", + "Key2": "val2" + }, + "metadata": { + "created_time": "2022-04-21T15:56:58.8525402Z", + "custom_metadata": null, + "deletion_time": "", + "destroyed": false, + "version": 2 + } + }, + "lease_duration": 0, + "lease_id": "", + "renewable": false, + "request_id": "15538d55-0ad9-1c39-2f4b-dcbb982f13cc", + "warnings": null, + "wrap_info": null + } + +The ``hashi_vault`` lookup traditionally returned the ``data`` field of whatever it was reading, and then later the plugin was updated to its current behavior, where it looks for the nested ``data.data`` structure, and if found, it returns only the inner ``data``. This aims to always return the secret data from KV1 and KV2 in a consistent format, but it means any additional information from KV2's metadata could not be accessed. + +KV1 and KV2 API paths +--------------------- + +KV1's API path had the secret paths directly concatenated to the mount point. So for example, if a KV1 engine is mounted at ``kv/v/1`` (mount paths can contain ``/``), and a secret was created in that store at ``app/deploy_key``, the path would be ``kv/v/1/app/deploy_key``. + +In KV2, there are separate paths that deal with the data and the metadata of a secret, so an additional ``/data/`` or ``/metadata/`` component needs to be inserted between the mount and the path. + +For example with a KV2 store mounted at ``kv/v/2``, and a secret at ``app/deploy_key``, the path to read the secret data is ``kv/v/2/data/app/deploy_key``. For metadata operations it would be ``kv/v/2/metadata/app/deploy_key``. + +Since ``hashi_vault`` does a generic read to an API path, anyone using it must know to insert those into the path, which causes a lot of confusion. + +KV2 secret vesions +------------------ + +Since KV2 is a versioned secret store, multiple versions of the same secret usually exist. There was no dedicated way to get anything but the latest secret (default) with the ``hashi_vault`` lookup, but docs suggested that ``?version=2`` could be added to the path to get secret version 2. This did work but it directly modified the API path, so it was not considered a stable option. The dedicated KV2 content in the collection supports this as a first class option. + + +.. _ansible_collections.community.hashi_vault.docsite.migration_hashi_vault_lookup.kv_replacements: + +KV get replacements +------------------- + +As of collection version 2.5.0, the ``vault_kv1_get`` and ``vault_kv2_get`` lookups and modules were added: + + * ``vault_kv1_get`` :ref:`lookup <ansible_collections.community.hashi_vault.vault_kv1_get_lookup>` + * ``vault_kv2_get`` :ref:`lookup <ansible_collections.community.hashi_vault.vault_kv2_get_lookup>` + * ``vault_kv1_get`` :ref:`module <ansible_collections.community.hashi_vault.vault_kv1_get_module>` + * ``vault_kv2_get`` :ref:`module <ansible_collections.community.hashi_vault.vault_kv2_get_module>` + +These dedicated plugins clearly separate KV1 and KV2 operations. This ensures their behavior is clear and predictable. + +As it relates to API paths, these plugins take the approach of most Vault client libraries, and recommended by HashiCorp, which is to accept the mount point as an option (``engine_mount_point``), separate from the path to be read. This ensures a proper path will be constructed internally, and does not require the caller to insert ``/data/`` on KV2. + +For return values, the KV plugins no longer return a direct secret. Instead, the return values from KV1 and KV2, and both the module and lookup forms, have been unified to give easy access to the secret, the full API response, and other parts of the response discretely. + +The return values are covered directly in the documentation for each plugin in the return and examples sections. + +Examples +-------- + +Here are some before and after KV examples. + +We will go back to our sample secret: + +.. code-block:: yaml + + key_1: value1 + 'key-2': 2 + 'key three': three + +And some usage: + +.. code-block:: yaml+jinja + + - name: Reading secrets with hashi_vault and colon dereferencing + ansible.builtin.debug: + msg: + - "KV1 (key1): {{ lookup('community.hashi_vault.hashi_vault', 'kv1_mount/path/to/secret:key_1') }}" + - "KV2 (key1): {{ lookup('community.hashi_vault.hashi_vault', 'kv2_mount/data/path/to/secret:key_1') }}" + + - name: Replacing the above + ansible.builtin.debug: + msg: + - "KV1 (key1): {{ lookup('community.hashi_vault.vault_kv1_get', 'path/to/secret', engine_mount_point='kv1_mount').secret.key_1 }}" + - "KV2 (key1): {{ lookup('community.hashi_vault.vault_kv2_get', 'path/to/secret', engine_mount_point='kv2_mount').secret.key_1 }}" + + - name: Reading secret version 7 (old) + ansible.builtin.debug: + msg: + - "KV2 (v7): {{ lookup('community.hashi_vault.hashi_vault', 'kv2_mount/data/path/to/secret?version=7') }}" + + - name: Reading secret version 7 (new) + ansible.builtin.debug: + msg: + - "KV2 (v7): {{ lookup('community.hashi_vault.vault_kv2_get', 'path/to/secret', engine_mount_point='kv2_mount', version=7).secret }}" + + - name: Reading KV2 metadata (old) + ansible.builtin.debug: + msg: + - "KV2 (metadata): {{ lookup('community.hashi_vault.hashi_vault', 'kv2_mount/data/path/to/secret', return_format='raw').data.metadata }}" + + - name: Reading KV2 metadata (new) + ansible.builtin.debug: + msg: + - "KV2 (metadata): {{ lookup('community.hashi_vault.vault_kv2_get', 'path/to/secret', engine_mount_point='kv2_mount').metadata }}" + + +.. _ansible_collections.community.hashi_vault.docsite.migration_hashi_vault_lookup.non_kv_replacements: + +General reads (non-KV) +====================== + +Since the ``hashi_vault`` lookup does a generic read internally, it can be used to read other paths that are not KV-specifc, for example reading from a cubbyhole or retrieving an AppRole's role ID. + +More specific-purpose content is expected in the future, for example plugins for retrieving a role ID, but for anything not covered right now, we have the ``vault_read`` lookup and module: + + * ``vault_read`` :ref:`lookup <ansible_collections.community.hashi_vault.vault_read_lookup>` + * ``vault_read`` :ref:`module <ansible_collections.community.hashi_vault.vault_read_module>` + +These always do a direct read, and return a raw result, without trying to do any additional interpretation of the response. See their documentation for examples. diff --git a/ansible_collections/community/hashi_vault/docs/docsite/rst/user_guide.rst b/ansible_collections/community/hashi_vault/docs/docsite/rst/user_guide.rst new file mode 100644 index 000000000..a3f417800 --- /dev/null +++ b/ansible_collections/community/hashi_vault/docs/docsite/rst/user_guide.rst @@ -0,0 +1,90 @@ +.. _ansible_collections.community.hashi_vault.docsite.user_guide: + +********** +User guide +********** + +The `community.hashi_vault collection <https://galaxy.ansible.com/community/hashi_vault>`_ offers Ansible content for working with `HashiCorp Vault <https://www.vaultproject.io/>`_. + +.. note:: + + This guide is a work-in-progress and should not be considered complete. Use it in conjunction with plugin documentation. + +.. contents:: + :local: + :depth: 1 + + +.. _ansible_collections.community.hashi_vault.docsite.user_guide.requirements: + +Requirements +============ + +The content in ``community.hashi_vault`` requires the `hvac <https://hvac.readthedocs.io/en/stable/>`_ library. + +.. code-block:: shell-session + + $ pip install hvac + +``hvac`` version specifics +-------------------------- + +In general, we recommend using the latest version of ``hvac`` that is supported for your given Python version because that is what we test against. Where possible we will try to list version-specific restrictions here, but this list may not be exhaustive. + +* ``hvac`` 0.7.0+ (for Azure auth and namespace support) +* ``hvac`` 0.9.6+ (to avoid most deprecation warnings) +* ``hvac`` 0.10.5+ (for JWT auth) +* ``hvac`` 0.10.6+ (to avoid deprecation warning for AppRole) +* ``hvac`` 0.10.12+ (for cert auth) + +Other requirements +------------------ + +* ``boto3`` (only if loading credentials from a boto session, for example using an AWS profile or IAM role credentials) +* ``azure-identity`` (only if using a service principal or managed identity) + +Retrying failed requests +======================== + +Via the ``retries`` parameter, you can control what happens when a request to Vault fails, and automatically retry certain requests. Retries are based on the `urllib3 Retry class <https://urllib3.readthedocs.io/en/latest/reference/urllib3.util.html#urllib3.util.Retry>`_ and so all of its options are supported. + +Retries are disabled by default. + +In ``community.hashi_vault`` you can specify the ``retries`` parameter in two ways: + +* Set a positive number (integer), where ``0`` disables retries and any positive number sets the number of tries, with the rest of the retry parameters using the collection defaults. +* Set a dictionary, where you can set any field that the ``Retry`` class can be initialized with, in order to fully customize your retry experience. + + +About the collection defaults +----------------------------- + +The collection uses its own set of recommended defaults for retries, including which HTTP status codes to retry, which HTTP methods are subject to retries, and the backoff factor used. **These defaults are subject to change at any time (in any release) and won't be considered breaking changes.** By setting ``retries`` to a number you are opting in to trust the defaults in the collection. To enable retries with full control over its behavior, be sure to specify a dictionary. + +Current Defaults (always check the source code to confirm the defaults in your specific collection version): + +.. code-block:: yaml + + status_forcelist: + # https://www.vaultproject.io/api#http-status-codes + # 429 is usually a "too many requests" status, but in Vault it's the default health status response for standby nodes. + - 412 # Precondition failed. Returned on Enterprise when a request can't be processed yet due to some missing eventually consistent data. Should be retried, perhaps with a little backoff. + - 500 # Internal server error. An internal error has occurred, try again later. If the error persists, report a bug. + - 502 # A request to Vault required Vault making a request to a third party; the third party responded with an error of some kind. + - 503 # Vault is down for maintenance or is currently sealed. Try again later. + allowed_methods: null # None allows retries on all methods, including those which may not be considered idempotent, like POST + backoff_factor: 0.3 + +Any of the ``Retry`` class's parameters that are not specified in the collection defaults or in your custom dictionary, are initialized using the class's defaults, with one exception: the ``raise_on_status`` parameter is always set to ``false`` unless you explicitly added it your custom dictionary. The reason is that this lets our error handling look for the expected ``hvac`` exceptions, instead of the ``Retry``-specfic exceptions. It is recommended that you don't override this as it may cause unexpected error messages on common failures if they are retried. + +Controlling retry warnings +-------------------------- + +By default, if a retry is performed, a warning will be emitted that shows how many retries are remaining. This can be controlled with the ``retry_action`` option which defaults to ``warn``. It is recommended to keep this enabled unless you have other processes that will be thrown off by the warning output. + +A note about timeouts +--------------------- + +Consider setting the ``timeout`` option appropriately when using retries, as a connection timeout doesn't count toward time between retries (backoff). A long timeout can cause very long delays for a connection that isn't going to recover, multiplied by number of retries. + +However, also consider the type of request being made, and the auth method in use. Because Vault auth methods may have their own dependencies on other systems (an LDAP server, a cloud provider like AWS, a required MFA prompt that depends on a human to respond), the time to complete a request could be quite long, and setting a timeout too short will prevent an otherwise successful request from completing. |