Adding upstream version 5.3.0.upstream/5.3.0upstream

Signed-off-by: Daniel Baumann <daniel.baumann@progress-linux.org>

10 files changed, 1317 insertions, 0 deletions

diff --git a/doc/development/tutorials/autodoc_ext.rst b/doc/development/tutorials/autodoc_ext.rst
new file mode 100644
index 0000000..8de2e4d
--- /dev/null
+++ b/doc/development/tutorials/autodoc_ext.rst
@@ -0,0 +1,140 @@
+.. _autodoc_ext_tutorial:
+
+Developing autodoc extension for IntEnum
+========================================
+
+The objective of this tutorial is to create an extension that adds
+support for new type for autodoc. This autodoc extension will format
+the ``IntEnum`` class from Python standard library. (module ``enum``)
+
+Overview
+--------
+
+We want the extension that will create auto-documentation for IntEnum.
+``IntEnum`` is the integer enum class from standard library ``enum`` module.
+
+Currently this class has no special auto documentation behavior.
+
+We want to add following to autodoc:
+
+* A new ``autointenum`` directive that will document the ``IntEnum`` class.
+* The generated documentation will have all the enum possible values
+  with names.
+* The ``autointenum`` directive will have an option ``:hex:`` which will
+  cause the integers be printed in hexadecimal form.
+
+
+Prerequisites
+-------------
+
+We need the same setup as in :doc:`the previous extensions <todo>`. This time,
+we will be putting out extension in a file called :file:`autodoc_intenum.py`.
+The :file:`my_enums.py` will contain the sample enums we will document.
+
+Here is an example of the folder structure you might obtain:
+
+.. code-block:: text
+
+      └── source
+          ├── _ext
+          │   └── autodoc_intenum.py
+          ├── conf.py
+          ├── index.rst
+          └── my_enums.py
+
+
+Writing the extension
+---------------------
+
+Start with ``setup`` function for the extension.
+
+.. literalinclude:: examples/autodoc_intenum.py
+   :language: python
+   :linenos:
+   :pyobject: setup
+
+
+The :meth:`~Sphinx.setup_extension` method will pull the autodoc extension
+because our new extension depends on autodoc. :meth:`~Sphinx.add_autodocumenter`
+is the method that registers our new auto documenter class.
+
+We want to import certain objects from the autodoc extension:
+
+.. literalinclude:: examples/autodoc_intenum.py
+   :language: python
+   :linenos:
+   :lines: 1-7
+
+
+There are several different documenter classes such as ``MethodDocumenter``
+or ``AttributeDocumenter`` available in the autodoc extension but
+our new class is the subclass of ``ClassDocumenter`` which a
+documenter class used by autodoc to document classes.
+
+This is the definition of our new the auto-documenter class:
+
+.. literalinclude:: examples/autodoc_intenum.py
+   :language: python
+   :linenos:
+   :pyobject: IntEnumDocumenter
+
+
+Important attributes of the new class:
+
+**objtype**
+    This attribute determines the ``auto`` directive name. In
+    this case the auto directive will be ``autointenum``.
+
+**directivetype**
+    This attribute sets the generated directive name. In
+    this example the generated directive will be ``.. :py:class::``.
+
+**priority**
+    the larger the number the higher is the priority. We want our
+    documenter be higher priority than the parent.
+
+**option_spec**
+    option specifications. We copy the parent class options and
+    add a new option *hex*.
+
+
+Overridden members:
+
+**can_document_member**
+    This member is important to override. It should
+    return *True* when the passed object can be documented by this class.
+
+**add_directive_header**
+    This method generates the directive header. We add
+    **:final:** directive option. Remember to call **super** or no directive
+    will be generated.
+
+**add_content**
+    This method generates the body of the class documentation.
+    After calling the super method we generate lines for enum description.
+
+
+Using the extension
+-------------------
+
+You can now use the new autodoc directive to document any ``IntEnum``.
+
+For example, you have the following ``IntEnum``:
+
+.. code-block:: python
+   :caption: my_enums.py
+
+   class Colors(IntEnum):
+       """Colors enumerator"""
+       NONE = 0
+       RED = 1
+       GREEN = 2
+       BLUE = 3
+
+
+This will be the documentation file with auto-documentation directive:
+
+.. code-block:: rst
+   :caption: index.rst
+
+   .. autointenum:: my_enums.Colors
diff --git a/doc/development/tutorials/examples/README.rst b/doc/development/tutorials/examples/README.rst
new file mode 100644
index 0000000..2b9c01b
--- /dev/null
+++ b/doc/development/tutorials/examples/README.rst
@@ -0,0 +1,11 @@
+:orphan:
+
+Tutorial examples
+=================
+
+This directory contains a number of examples used in the tutorials. These are
+intended to be increasingly complex to demonstrate the various features of
+Sphinx, but should aim to be as complicated as necessary but no more.
+Individual sections are referenced by line numbers, meaning if you make changes
+to the source files, you should update the references in the documentation
+accordingly.
diff --git a/doc/development/tutorials/examples/autodoc_intenum.py b/doc/development/tutorials/examples/autodoc_intenum.py
new file mode 100644
index 0000000..a23f9ce
--- /dev/null
+++ b/doc/development/tutorials/examples/autodoc_intenum.py
@@ -0,0 +1,54 @@
+from enum import IntEnum
+from typing import Any, Optional
+
+from docutils.statemachine import StringList
+
+from sphinx.application import Sphinx
+from sphinx.ext.autodoc import ClassDocumenter, bool_option
+
+
+class IntEnumDocumenter(ClassDocumenter):
+    objtype = 'intenum'
+    directivetype = ClassDocumenter.objtype
+    priority = 10 + ClassDocumenter.priority
+    option_spec = dict(ClassDocumenter.option_spec)
+    option_spec['hex'] = bool_option
+
+    @classmethod
+    def can_document_member(cls,
+                            member: Any, membername: str,
+                            isattr: bool, parent: Any) -> bool:
+        try:
+            return issubclass(member, IntEnum)
+        except TypeError:
+            return False
+
+    def add_directive_header(self, sig: str) -> None:
+        super().add_directive_header(sig)
+        self.add_line('   :final:', self.get_sourcename())
+
+    def add_content(self,
+                    more_content: Optional[StringList],
+                    no_docstring: bool = False
+                    ) -> None:
+
+        super().add_content(more_content, no_docstring)
+
+        source_name = self.get_sourcename()
+        enum_object: IntEnum = self.object
+        use_hex = self.options.hex
+        self.add_line('', source_name)
+
+        for the_member_name, enum_member in enum_object.__members__.items():
+            the_member_value = enum_member.value
+            if use_hex:
+                the_member_value = hex(the_member_value)
+
+            self.add_line(
+                f"**{the_member_name}**: {the_member_value}", source_name)
+            self.add_line('', source_name)
+
+
+def setup(app: Sphinx) -> None:
+    app.setup_extension('sphinx.ext.autodoc')  # Require autodoc extension
+    app.add_autodocumenter(IntEnumDocumenter)
diff --git a/doc/development/tutorials/examples/helloworld.py b/doc/development/tutorials/examples/helloworld.py
new file mode 100644
index 0000000..d6d81fd
--- /dev/null
+++ b/doc/development/tutorials/examples/helloworld.py
@@ -0,0 +1,19 @@
+from docutils import nodes
+from docutils.parsers.rst import Directive
+
+
+class HelloWorld(Directive):
+
+    def run(self):
+        paragraph_node = nodes.paragraph(text='Hello World!')
+        return [paragraph_node]
+
+
+def setup(app):
+    app.add_directive("helloworld", HelloWorld)
+
+    return {
+        'version': '0.1',
+        'parallel_read_safe': True,
+        'parallel_write_safe': True,
+    }
diff --git a/doc/development/tutorials/examples/recipe.py b/doc/development/tutorials/examples/recipe.py
new file mode 100644
index 0000000..8dc53fd
--- /dev/null
+++ b/doc/development/tutorials/examples/recipe.py
@@ -0,0 +1,159 @@
+from collections import defaultdict
+
+from docutils.parsers.rst import directives
+
+from sphinx import addnodes
+from sphinx.directives import ObjectDescription
+from sphinx.domains import Domain, Index
+from sphinx.roles import XRefRole
+from sphinx.util.nodes import make_refnode
+
+
+class RecipeDirective(ObjectDescription):
+    """A custom directive that describes a recipe."""
+
+    has_content = True
+    required_arguments = 1
+    option_spec = {
+        'contains': directives.unchanged_required,
+    }
+
+    def handle_signature(self, sig, signode):
+        signode += addnodes.desc_name(text=sig)
+        return sig
+
+    def add_target_and_index(self, name_cls, sig, signode):
+        signode['ids'].append('recipe' + '-' + sig)
+        if 'contains' in self.options:
+            ingredients = [
+                x.strip() for x in self.options.get('contains').split(',')]
+
+            recipes = self.env.get_domain('recipe')
+            recipes.add_recipe(sig, ingredients)
+
+
+class IngredientIndex(Index):
+    """A custom index that creates an ingredient matrix."""
+
+    name = 'ingredient'
+    localname = 'Ingredient Index'
+    shortname = 'Ingredient'
+
+    def generate(self, docnames=None):
+        content = defaultdict(list)
+
+        recipes = {name: (dispname, typ, docname, anchor)
+                   for name, dispname, typ, docname, anchor, _
+                   in self.domain.get_objects()}
+        recipe_ingredients = self.domain.data['recipe_ingredients']
+        ingredient_recipes = defaultdict(list)
+
+        # flip from recipe_ingredients to ingredient_recipes
+        for recipe_name, ingredients in recipe_ingredients.items():
+            for ingredient in ingredients:
+                ingredient_recipes[ingredient].append(recipe_name)
+
+        # convert the mapping of ingredient to recipes to produce the expected
+        # output, shown below, using the ingredient name as a key to group
+        #
+        # name, subtype, docname, anchor, extra, qualifier, description
+        for ingredient, recipe_names in ingredient_recipes.items():
+            for recipe_name in recipe_names:
+                dispname, typ, docname, anchor = recipes[recipe_name]
+                content[ingredient].append(
+                    (dispname, 0, docname, anchor, docname, '', typ))
+
+        # convert the dict to the sorted list of tuples expected
+        content = sorted(content.items())
+
+        return content, True
+
+
+class RecipeIndex(Index):
+    """A custom index that creates an recipe matrix."""
+
+    name = 'recipe'
+    localname = 'Recipe Index'
+    shortname = 'Recipe'
+
+    def generate(self, docnames=None):
+        content = defaultdict(list)
+
+        # sort the list of recipes in alphabetical order
+        recipes = self.domain.get_objects()
+        recipes = sorted(recipes, key=lambda recipe: recipe[0])
+
+        # generate the expected output, shown below, from the above using the
+        # first letter of the recipe as a key to group thing
+        #
+        # name, subtype, docname, anchor, extra, qualifier, description
+        for _name, dispname, typ, docname, anchor, _priority in recipes:
+            content[dispname[0].lower()].append(
+                (dispname, 0, docname, anchor, docname, '', typ))
+
+        # convert the dict to the sorted list of tuples expected
+        content = sorted(content.items())
+
+        return content, True
+
+
+class RecipeDomain(Domain):
+
+    name = 'recipe'
+    label = 'Recipe Sample'
+    roles = {
+        'ref': XRefRole()
+    }
+    directives = {
+        'recipe': RecipeDirective,
+    }
+    indices = {
+        RecipeIndex,
+        IngredientIndex
+    }
+    initial_data = {
+        'recipes': [],  # object list
+        'recipe_ingredients': {},  # name -> object
+    }
+
+    def get_full_qualified_name(self, node):
+        return '{}.{}'.format('recipe', node.arguments[0])
+
+    def get_objects(self):
+        yield from self.data['recipes']
+
+    def resolve_xref(self, env, fromdocname, builder, typ, target, node,
+                     contnode):
+        match = [(docname, anchor)
+                 for name, sig, typ, docname, anchor, prio
+                 in self.get_objects() if sig == target]
+
+        if len(match) > 0:
+            todocname = match[0][0]
+            targ = match[0][1]
+
+            return make_refnode(builder, fromdocname, todocname, targ,
+                                contnode, targ)
+        else:
+            print('Awww, found nothing')
+            return None
+
+    def add_recipe(self, signature, ingredients):
+        """Add a new recipe to the domain."""
+        name = '{}.{}'.format('recipe', signature)
+        anchor = 'recipe-{}'.format(signature)
+
+        self.data['recipe_ingredients'][name] = ingredients
+        # name, dispname, type, docname, anchor, priority
+        self.data['recipes'].append(
+            (name, signature, 'Recipe', self.env.docname, anchor, 0))
+
+
+def setup(app):
+    app.add_domain(RecipeDomain)
+
+    return {
+        'version': '0.1',
+        'parallel_read_safe': True,
+        'parallel_write_safe': True,
+    }
diff --git a/doc/development/tutorials/examples/todo.py b/doc/development/tutorials/examples/todo.py
new file mode 100644
index 0000000..15368f4
--- /dev/null
+++ b/doc/development/tutorials/examples/todo.py
@@ -0,0 +1,135 @@
+from docutils import nodes
+from docutils.parsers.rst import Directive
+
+from sphinx.locale import _
+from sphinx.util.docutils import SphinxDirective
+
+
+class todo(nodes.Admonition, nodes.Element):
+    pass
+
+
+class todolist(nodes.General, nodes.Element):
+    pass
+
+
+def visit_todo_node(self, node):
+    self.visit_admonition(node)
+
+
+def depart_todo_node(self, node):
+    self.depart_admonition(node)
+
+
+class TodolistDirective(Directive):
+
+    def run(self):
+        return [todolist('')]
+
+
+class TodoDirective(SphinxDirective):
+
+    # this enables content in the directive
+    has_content = True
+
+    def run(self):
+        targetid = 'todo-%d' % self.env.new_serialno('todo')
+        targetnode = nodes.target('', '', ids=[targetid])
+
+        todo_node = todo('\n'.join(self.content))
+        todo_node += nodes.title(_('Todo'), _('Todo'))
+        self.state.nested_parse(self.content, self.content_offset, todo_node)
+
+        if not hasattr(self.env, 'todo_all_todos'):
+            self.env.todo_all_todos = []
+
+        self.env.todo_all_todos.append({
+            'docname': self.env.docname,
+            'lineno': self.lineno,
+            'todo': todo_node.deepcopy(),
+            'target': targetnode,
+        })
+
+        return [targetnode, todo_node]
+
+
+def purge_todos(app, env, docname):
+    if not hasattr(env, 'todo_all_todos'):
+        return
+
+    env.todo_all_todos = [todo for todo in env.todo_all_todos
+                          if todo['docname'] != docname]
+
+
+def merge_todos(app, env, docnames, other):
+    if not hasattr(env, 'todo_all_todos'):
+        env.todo_all_todos = []
+    if hasattr(other, 'todo_all_todos'):
+        env.todo_all_todos.extend(other.todo_all_todos)
+
+
+def process_todo_nodes(app, doctree, fromdocname):
+    if not app.config.todo_include_todos:
+        for node in doctree.findall(todo):
+            node.parent.remove(node)
+
+    # Replace all todolist nodes with a list of the collected todos.
+    # Augment each todo with a backlink to the original location.
+    env = app.builder.env
+
+    if not hasattr(env, 'todo_all_todos'):
+        env.todo_all_todos = []
+
+    for node in doctree.findall(todolist):
+        if not app.config.todo_include_todos:
+            node.replace_self([])
+            continue
+
+        content = []
+
+        for todo_info in env.todo_all_todos:
+            para = nodes.paragraph()
+            filename = env.doc2path(todo_info['docname'], base=None)
+            description = (
+                _('(The original entry is located in %s, line %d and can be found ') %
+                (filename, todo_info['lineno']))
+            para += nodes.Text(description)
+
+            # Create a reference
+            newnode = nodes.reference('', '')
+            innernode = nodes.emphasis(_('here'), _('here'))
+            newnode['refdocname'] = todo_info['docname']
+            newnode['refuri'] = app.builder.get_relative_uri(
+                fromdocname, todo_info['docname'])
+            newnode['refuri'] += '#' + todo_info['target']['refid']
+            newnode.append(innernode)
+            para += newnode
+            para += nodes.Text('.)')
+
+            # Insert into the todolist
+            content.append(todo_info['todo'])
+            content.append(para)
+
+        node.replace_self(content)
+
+
+def setup(app):
+    app.add_config_value('todo_include_todos', False, 'html')
+
+    app.add_node(todolist)
+    app.add_node(todo,
+                 html=(visit_todo_node, depart_todo_node),
+                 latex=(visit_todo_node, depart_todo_node),
+                 text=(visit_todo_node, depart_todo_node))
+
+    app.add_directive('todo', TodoDirective)
+    app.add_directive('todolist', TodolistDirective)
+    app.connect('doctree-resolved', process_todo_nodes)
+    app.connect('env-purge-doc', purge_todos)
+    app.connect('env-merge-info', merge_todos)
+
+    return {
+        'version': '0.1',
+        'parallel_read_safe': True,
+        'parallel_write_safe': True,
+    }
diff --git a/doc/development/tutorials/helloworld.rst b/doc/development/tutorials/helloworld.rst
new file mode 100644
index 0000000..8940e3d
--- /dev/null
+++ b/doc/development/tutorials/helloworld.rst
@@ -0,0 +1,189 @@
+Developing a "Hello world" extension
+====================================
+
+The objective of this tutorial is to create a very basic extension that adds a
+new directive. This directive will output a paragraph containing "hello world".
+
+Only basic information is provided in this tutorial. For more information, refer
+to the :doc:`other tutorials <index>` that go into more details.
+
+.. warning::
+
+   For this extension, you will need some basic understanding of docutils_
+   and Python.
+
+
+Overview
+--------
+
+We want the extension to add the following to Sphinx:
+
+* A ``helloworld`` directive, that will simply output the text "hello world".
+
+
+Prerequisites
+-------------
+
+We will not be distributing this plugin via `PyPI`_ and will instead include it
+as part of an existing project. This means you will need to use an existing
+project or create a new one using :program:`sphinx-quickstart`.
+
+We assume you are using separate source (:file:`source`) and build
+(:file:`build`) folders. Your extension file could be in any folder of your
+project. In our case, let's do the following:
+
+#. Create an :file:`_ext` folder in :file:`source`
+#. Create a new Python file in the :file:`_ext` folder called
+   :file:`helloworld.py`
+
+Here is an example of the folder structure you might obtain:
+
+.. code-block:: text
+
+      └── source
+          ├── _ext
+          │   └── helloworld.py
+          ├── _static
+          ├── conf.py
+          ├── somefolder
+          ├── index.rst
+          ├── somefile.rst
+          └── someotherfile.rst
+
+
+Writing the extension
+---------------------
+
+Open :file:`helloworld.py` and paste the following code in it:
+
+.. literalinclude:: examples/helloworld.py
+   :language: python
+   :linenos:
+
+Some essential things are happening in this example, and you will see them for
+all directives.
+
+.. rubric:: The directive class
+
+Our new directive is declared in the ``HelloWorld`` class.
+
+.. literalinclude:: examples/helloworld.py
+   :language: python
+   :linenos:
+   :lines: 5-9
+
+This class extends the docutils_' ``Directive`` class. All extensions that
+create directives should extend this class.
+
+.. seealso::
+
+   `The docutils documentation on creating directives <docutils directives_>`_
+
+This class contains a ``run`` method.  This method is a requirement and it is
+part of every directive.  It contains the main logic of the directive and it
+returns a list of docutils nodes to be processed by Sphinx. These nodes are
+docutils' way of representing the content of a document. There are many types of
+nodes available: text, paragraph, reference, table, etc.
+
+.. seealso::
+
+   `The docutils documentation on nodes <docutils nodes_>`_
+
+The ``nodes.paragraph`` class creates a new paragraph node. A paragraph
+node typically contains some text that we can set during instantiation using
+the ``text`` parameter.
+
+.. rubric:: The ``setup`` function
+
+.. currentmodule:: sphinx.application
+
+This function is a requirement. We use it to plug our new directive into
+Sphinx.
+
+.. literalinclude:: examples/helloworld.py
+   :language: python
+   :linenos:
+   :lines: 12-
+
+The simplest thing you can do is to call the :meth:`~Sphinx.add_directive` method,
+which is what we've done here. For this particular call, the first argument is
+the name of the directive itself as used in a reST file. In this case, we would
+use ``helloworld``. For example:
+
+.. code-block:: rst
+
+   Some intro text here...
+
+   .. helloworld::
+
+   Some more text here...
+
+We also return the :ref:`extension metadata <ext-metadata>` that indicates the
+version of our extension, along with the fact that it is safe to use the
+extension for both parallel reading and writing.
+
+
+Using the extension
+-------------------
+
+The extension has to be declared in your :file:`conf.py` file to make Sphinx
+aware of it. There are two steps necessary here:
+
+#. Add the :file:`_ext` directory to the `Python path`_ using
+   ``sys.path.append``. This should be placed at the top of the file.
+
+#. Update or create the :confval:`extensions` list and add the extension file
+   name to the list
+
+For example:
+
+.. code-block:: python
+
+   import os
+   import sys
+
+   sys.path.append(os.path.abspath("./_ext"))
+
+   extensions = ['helloworld']
+
+.. tip::
+
+   We're not distributing this extension as a `Python package`_, we need to
+   modify the `Python path`_ so Sphinx can find our extension. This is why we
+   need the call to ``sys.path.append``.
+
+You can now use the extension in a file. For example:
+
+.. code-block:: rst
+
+   Some intro text here...
+
+   .. helloworld::
+
+   Some more text here...
+
+The sample above would generate:
+
+.. code-block:: text
+
+   Some intro text here...
+
+   Hello World!
+
+   Some more text here...
+
+
+Further reading
+---------------
+
+This is the very basic principle of an extension that creates a new directive.
+
+For a more advanced example, refer to :doc:`todo`.
+
+
+.. _docutils: https://docutils.sourceforge.io/
+.. _docutils directives: https://docutils.sourceforge.io/docs/howto/rst-directives.html
+.. _docutils nodes: https://docutils.sourceforge.io/docs/ref/doctree.html
+.. _PyPI: https://pypi.org/
+.. _Python package: https://packaging.python.org/
+.. _Python path: https://docs.python.org/3/using/cmdline.html#envvar-PYTHONPATH
diff --git a/doc/development/tutorials/index.rst b/doc/development/tutorials/index.rst
new file mode 100644
index 0000000..a7eee48
--- /dev/null
+++ b/doc/development/tutorials/index.rst
@@ -0,0 +1,17 @@
+.. _extension-tutorials-index:
+
+Extension tutorials
+===================
+
+Refer to the following tutorials to get started with extension development.
+
+
+.. toctree::
+   :caption: Directive tutorials
+   :maxdepth: 1
+
+   helloworld
+   todo
+   recipe
+   autodoc_ext
+
diff --git a/doc/development/tutorials/recipe.rst b/doc/development/tutorials/recipe.rst
new file mode 100644
index 0000000..0e96a45
--- /dev/null
+++ b/doc/development/tutorials/recipe.rst
@@ -0,0 +1,226 @@
+Developing a "recipe" extension
+===============================
+
+The objective of this tutorial is to illustrate roles, directives and domains.
+Once complete, we will be able to use this extension to describe a recipe and
+reference that recipe from elsewhere in our documentation.
+
+.. note::
+
+   This tutorial is based on a guide first published on `opensource.com`_ and
+   is provided here with the original author's permission.
+
+   .. _opensource.com: https://opensource.com/article/18/11/building-custom-workflows-sphinx
+
+
+Overview
+--------
+
+We want the extension to add the following to Sphinx:
+
+* A ``recipe`` :term:`directive`, containing some content describing the recipe
+  steps, along with a ``:contains:`` option highlighting the main ingredients
+  of the recipe.
+
+* A ``ref`` :term:`role`, which provides a cross-reference to the recipe
+  itself.
+
+* A ``recipe`` :term:`domain`, which allows us to tie together the above role
+  and domain, along with things like indices.
+
+For that, we will need to add the following elements to Sphinx:
+
+* A new directive called ``recipe``
+
+* New indexes to allow us to reference ingredient and recipes
+
+* A new domain called ``recipe``, which will contain the ``recipe`` directive
+  and ``ref`` role
+
+
+Prerequisites
+-------------
+
+We need the same setup as in :doc:`the previous extensions <todo>`. This time,
+we will be putting out extension in a file called :file:`recipe.py`.
+
+Here is an example of the folder structure you might obtain:
+
+.. code-block:: text
+
+      └── source
+          ├── _ext
+          │   └── recipe.py
+          ├── conf.py
+          └── index.rst
+
+
+Writing the extension
+---------------------
+
+Open :file:`recipe.py` and paste the following code in it, all of which we will
+explain in detail shortly:
+
+.. literalinclude:: examples/recipe.py
+   :language: python
+   :linenos:
+
+Let's look at each piece of this extension step-by-step to explain what's going
+on.
+
+.. rubric:: The directive class
+
+The first thing to examine is the ``RecipeDirective`` directive:
+
+.. literalinclude:: examples/recipe.py
+   :language: python
+   :linenos:
+   :pyobject: RecipeDirective
+
+Unlike :doc:`helloworld` and :doc:`todo`, this directive doesn't derive from
+:class:`docutils.parsers.rst.Directive` and doesn't define a ``run`` method.
+Instead, it derives from :class:`sphinx.directives.ObjectDescription` and
+defines  ``handle_signature`` and ``add_target_and_index`` methods. This is
+because ``ObjectDescription`` is a special-purpose directive that's intended
+for describing things like classes, functions, or, in our case, recipes. More
+specifically, ``handle_signature`` implements parsing the signature of the
+directive and passes on the object's name and type to its superclass, while
+``add_taget_and_index`` adds a target (to link to) and an entry to the index
+for this node.
+
+We also see that this directive defines ``has_content``, ``required_arguments``
+and ``option_spec``. Unlike the ``TodoDirective`` directive added in the
+:doc:`previous tutorial <todo>`, this directive takes a single argument, the
+recipe name, and an option, ``contains``, in addition to the nested
+reStructuredText in the body.
+
+.. rubric:: The index classes
+
+.. currentmodule:: sphinx.domains
+
+.. todo:: Add brief overview of indices
+
+.. literalinclude:: examples/recipe.py
+   :language: python
+   :linenos:
+   :pyobject: IngredientIndex
+
+.. literalinclude:: examples/recipe.py
+   :language: python
+   :linenos:
+   :pyobject: RecipeIndex
+
+Both ``IngredientIndex`` and ``RecipeIndex`` are derived from :class:`Index`.
+They implement custom logic to generate a tuple of values that define the
+index. Note that ``RecipeIndex`` is a simple index that has only one entry.
+Extending it to cover more object types is not yet part of the code.
+
+Both indices use the method :meth:`Index.generate` to do their work. This
+method combines the information from our domain, sorts it, and returns it in a
+list structure that will be accepted by Sphinx. This might look complicated but
+all it really is is a list of tuples like ``('tomato', 'TomatoSoup', 'test',
+'rec-TomatoSoup',...)``. Refer to the :doc:`domain API guide
+</extdev/domainapi>` for more information on this API.
+
+These index pages can be referred by combination of domain name and its
+``name`` using :rst:role:`ref` role.  For example, ``RecipeIndex`` can be
+referred by ``:ref:`recipe-recipe```.
+
+.. rubric:: The domain
+
+A Sphinx domain is a specialized container that ties together roles,
+directives, and indices, among other things. Let's look at the domain we're
+creating here.
+
+.. literalinclude:: examples/recipe.py
+   :language: python
+   :linenos:
+   :pyobject: RecipeDomain
+
+There are some interesting things to note about this ``recipe`` domain and domains
+in general. Firstly, we actually register our directives, roles and indices
+here, via the ``directives``, ``roles`` and ``indices`` attributes, rather than
+via calls later on in ``setup``. We can also note that we aren't actually
+defining a custom role and are instead reusing the
+:class:`sphinx.roles.XRefRole` role and defining the
+:class:`sphinx.domains.Domain.resolve_xref` method. This method takes two
+arguments, ``typ`` and ``target``, which refer to the cross-reference type and
+its target name. We'll use ``target`` to resolve our destination from our
+domain's ``recipes`` because we currently have only one type of node.
+
+Moving on, we can see that we've defined ``initial_data``. The values defined in
+``initial_data`` will be copied to ``env.domaindata[domain_name]`` as the
+initial data of the domain, and domain instances can access it via
+``self.data``. We see that we have defined two items in ``initial_data``:
+``recipes`` and ``recipe2ingredient``. These contain a list of all objects
+defined (i.e. all recipes) and a hash that maps a canonical ingredient name to
+the list of objects. The way we name objects is common across our extension and
+is defined in the ``get_full_qualified_name`` method. For each object created,
+the canonical name is ``recipe.<recipename>``, where ``<recipename>`` is the
+name the documentation writer gives the object (a recipe). This enables the
+extension to use different object types that share the same name. Having a
+canonical name and central place for our objects is a huge advantage. Both our
+indices and our cross-referencing code use this feature.
+
+.. rubric:: The ``setup`` function
+
+.. currentmodule:: sphinx.application
+
+:doc:`As always <todo>`, the ``setup`` function is a requirement and is used to
+hook the various parts of our extension into Sphinx. Let's look at the
+``setup`` function for this extension.
+
+.. literalinclude:: examples/recipe.py
+   :language: python
+   :linenos:
+   :pyobject: setup
+
+This looks a little different to what we're used to seeing. There are no calls
+to :meth:`~Sphinx.add_directive` or even :meth:`~Sphinx.add_role`. Instead, we
+have a single call to :meth:`~Sphinx.add_domain` followed by some
+initialization of the :ref:`standard domain <domains-std>`. This is because we
+had already registered our directives, roles and indexes as part of the
+directive itself.
+
+
+Using the extension
+-------------------
+
+You can now use the extension throughout your project. For example:
+
+.. code-block:: rst
+   :caption: index.rst
+
+   Joe's Recipes
+   =============
+
+   Below are a collection of my favourite recipes. I highly recommend the
+   :recipe:ref:`TomatoSoup` recipe in particular!
+
+   .. toctree::
+
+      tomato-soup
+
+.. code-block:: rst
+   :caption: tomato-soup.rst
+
+   The recipe contains `tomato` and `cilantro`.
+
+   .. recipe:recipe:: TomatoSoup
+      :contains: tomato, cilantro, salt, pepper
+
+      This recipe is a tasty tomato soup, combine all ingredients
+      and cook.
+
+The important things to note are the use of the ``:recipe:ref:`` role to
+cross-reference the recipe actually defined elsewhere (using the
+``:recipe:recipe:`` directive.
+
+
+Further reading
+---------------
+
+For more information, refer to the `docutils`_ documentation and
+:doc:`/extdev/index`.
+
+.. _docutils: https://docutils.sourceforge.io/docs/
diff --git a/doc/development/tutorials/todo.rst b/doc/development/tutorials/todo.rst
new file mode 100644
index 0000000..c0a5c0d
--- /dev/null
+++ b/doc/development/tutorials/todo.rst
@@ -0,0 +1,367 @@
+Developing a "TODO" extension
+=============================
+
+The objective of this tutorial is to create a more comprehensive extension than
+that created in :doc:`helloworld`. Whereas that guide just covered writing a
+custom :term:`directive`, this guide adds multiple directives, along with custom
+nodes, additional config values and custom event handlers. To this end, we will
+cover a ``todo`` extension that adds capabilities to include todo entries in the
+documentation, and to collect these in a central place. This is similar the
+``sphinxext.todo`` extension distributed with Sphinx.
+
+
+Overview
+--------
+
+.. note::
+   To understand the design of this extension, refer to
+   :ref:`important-objects` and :ref:`build-phases`.
+
+We want the extension to add the following to Sphinx:
+
+* A ``todo`` directive, containing some content that is marked with "TODO" and
+  only shown in the output if a new config value is set. Todo entries should not
+  be in the output by default.
+
+* A ``todolist`` directive that creates a list of all todo entries throughout
+  the documentation.
+
+For that, we will need to add the following elements to Sphinx:
+
+* New directives, called ``todo`` and ``todolist``.
+
+* New document tree nodes to represent these directives, conventionally also
+  called ``todo`` and ``todolist``.  We wouldn't need new nodes if the new
+  directives only produced some content representable by existing nodes.
+
+* A new config value ``todo_include_todos`` (config value names should start
+  with the extension name, in order to stay unique) that controls whether todo
+  entries make it into the output.
+
+* New event handlers: one for the :event:`doctree-resolved` event, to
+  replace the todo and todolist nodes, one for :event:`env-merge-info`
+  to merge intermediate results from parallel builds, and one for
+  :event:`env-purge-doc` (the reason for that will be covered later).
+
+
+Prerequisites
+-------------
+
+As with :doc:`helloworld`, we will not be distributing this plugin via PyPI so
+once again we need a Sphinx project to call this from. You can use an existing
+project or create a new one using :program:`sphinx-quickstart`.
+
+We assume you are using separate source (:file:`source`) and build
+(:file:`build`) folders. Your extension file could be in any folder of your
+project. In our case, let's do the following:
+
+#. Create an :file:`_ext` folder in :file:`source`
+#. Create a new Python file in the :file:`_ext` folder called :file:`todo.py`
+
+Here is an example of the folder structure you might obtain:
+
+.. code-block:: text
+
+      └── source
+          ├── _ext
+          │   └── todo.py
+          ├── _static
+          ├── conf.py
+          ├── somefolder
+          ├── index.rst
+          ├── somefile.rst
+          └── someotherfile.rst
+
+
+Writing the extension
+---------------------
+
+Open :file:`todo.py` and paste the following code in it, all of which we will
+explain in detail shortly:
+
+.. literalinclude:: examples/todo.py
+   :language: python
+   :linenos:
+
+This is far more extensive extension than the one detailed in :doc:`helloworld`,
+however, we will will look at each piece step-by-step to explain what's
+happening.
+
+.. rubric:: The node classes
+
+Let's start with the node classes:
+
+.. literalinclude:: examples/todo.py
+   :language: python
+   :linenos:
+   :lines: 8-21
+
+Node classes usually don't have to do anything except inherit from the standard
+docutils classes defined in :mod:`docutils.nodes`.  ``todo`` inherits from
+``Admonition`` because it should be handled like a note or warning, ``todolist``
+is just a "general" node.
+
+.. note::
+
+   Many extensions will not have to create their own node classes and work fine
+   with the nodes already provided by `docutils
+   <https://docutils.sourceforge.io/docs/ref/doctree.html>`__ and :ref:`Sphinx
+   <nodes>`.
+
+.. attention::
+
+   It is important to know that while you can extend Sphinx without
+   leaving your ``conf.py``, if you declare an inherited node right
+   there, you'll hit an unobvious :py:class:`PickleError`. So if
+   something goes wrong, please make sure that you put inherited nodes
+   into a separate Python module.
+
+   For more details, see:
+
+   - https://github.com/sphinx-doc/sphinx/issues/6751
+   - https://github.com/sphinx-doc/sphinx/issues/1493
+   - https://github.com/sphinx-doc/sphinx/issues/1424
+
+.. rubric:: The directive classes
+
+A directive class is a class deriving usually from
+:class:`docutils.parsers.rst.Directive`. The directive interface is also
+covered in detail in the `docutils documentation`_; the important thing is that
+the class should have attributes that configure the allowed markup, and a
+``run`` method that returns a list of nodes.
+
+Looking first at the ``TodolistDirective`` directive:
+
+.. literalinclude:: examples/todo.py
+   :language: python
+   :linenos:
+   :lines: 24-27
+
+It's very simple, creating and returning an instance of our ``todolist`` node
+class.  The ``TodolistDirective`` directive itself has neither content nor
+arguments that need to be handled. That brings us to the ``TodoDirective``
+directive:
+
+.. literalinclude:: examples/todo.py
+   :language: python
+   :linenos:
+   :lines: 30-53
+
+Several important things are covered here. First, as you can see, we're now
+subclassing the :class:`~sphinx.util.docutils.SphinxDirective` helper class
+instead of the usual :class:`~docutils.parsers.rst.Directive` class. This
+gives us access to the :ref:`build environment instance <important-objects>`
+using the ``self.env`` property. Without this, we'd have to use the rather
+convoluted ``self.state.document.settings.env``. Then, to act as a link target
+(from ``TodolistDirective``), the ``TodoDirective`` directive needs to return a
+target node in addition to the ``todo`` node.  The target ID (in HTML, this will
+be the anchor name) is generated by using ``env.new_serialno`` which returns a
+new unique integer on each call and therefore leads to unique target names. The
+target node is instantiated without any text (the first two arguments).
+
+On creating admonition node, the content body of the directive are parsed using
+``self.state.nested_parse``.  The first argument gives the content body, and
+the second one gives content offset.  The third argument gives the parent node
+of parsed result, in our case the ``todo`` node. Following this, the ``todo``
+node is added to the environment.  This is needed to be able to create a list of
+all todo entries throughout the documentation, in the place where the author
+puts a ``todolist`` directive.  For this case, the environment attribute
+``todo_all_todos`` is used (again, the name should be unique, so it is prefixed
+by the extension name).  It does not exist when a new environment is created, so
+the directive must check and create it if necessary.  Various information about
+the todo entry's location are stored along with a copy of the node.
+
+In the last line, the nodes that should be put into the doctree are returned:
+the target node and the admonition node.
+
+The node structure that the directive returns looks like this::
+
+   +--------------------+
+   | target node        |
+   +--------------------+
+   +--------------------+
+   | todo node          |
+   +--------------------+
+     \__+--------------------+
+        | admonition title   |
+        +--------------------+
+        | paragraph          |
+        +--------------------+
+        | ...                |
+        +--------------------+
+
+.. rubric:: The event handlers
+
+Event handlers are one of Sphinx's most powerful features, providing a way to
+do hook into any part of the documentation process. There are many events
+provided by Sphinx itself, as detailed in :ref:`the API guide <events>`, and
+we're going to use a subset of them here.
+
+Let's look at the event handlers used in the above example.  First, the one for
+the :event:`env-purge-doc` event:
+
+.. literalinclude:: examples/todo.py
+   :language: python
+   :linenos:
+   :lines: 56-61
+
+Since we store information from source files in the environment, which is
+persistent, it may become out of date when the source file changes.  Therefore,
+before each source file is read, the environment's records of it are cleared,
+and the :event:`env-purge-doc` event gives extensions a chance to do the same.
+Here we clear out all todos whose docname matches the given one from the
+``todo_all_todos`` list.  If there are todos left in the document, they will be
+added again during parsing.
+
+The next handler, for the :event:`env-merge-info` event, is used
+during parallel builds. As during parallel builds all threads have
+their own ``env``, there's multiple ``todo_all_todos`` lists that need
+to be merged:
+
+.. literalinclude:: examples/todo.py
+   :language: python
+   :linenos:
+   :lines: 64-68
+
+
+The other handler belongs to the :event:`doctree-resolved` event:
+
+.. literalinclude:: examples/todo.py
+   :language: python
+   :linenos:
+   :lines: 71-113
+
+The :event:`doctree-resolved` event is emitted at the end of :ref:`phase 3
+(resolving) <build-phases>` and allows custom resolving to be done. The handler
+we have written for this event is a bit more involved. If the
+``todo_include_todos`` config value (which we'll describe shortly) is false,
+all ``todo`` and ``todolist`` nodes are removed from the documents. If not,
+``todo`` nodes just stay where and how they are.  ``todolist`` nodes are
+replaced by a list of todo entries, complete with backlinks to the location
+where they come from.  The list items are composed of the nodes from the
+``todo`` entry and docutils nodes created on the fly: a paragraph for each
+entry, containing text that gives the location, and a link (reference node
+containing an italic node) with the backreference. The reference URI is built
+by :meth:`sphinx.builders.Builder.get_relative_uri` which creates a suitable
+URI depending on the used builder, and appending the todo node's (the target's)
+ID as the anchor name.
+
+.. rubric:: The ``setup`` function
+
+.. currentmodule:: sphinx.application
+
+As noted :doc:`previously <helloworld>`, the ``setup`` function is a requirement
+and is used to plug directives into Sphinx. However, we also use it to hook up
+the other parts of our extension. Let's look at our ``setup`` function:
+
+.. literalinclude:: examples/todo.py
+   :language: python
+   :linenos:
+   :lines: 116-
+
+The calls in this function refer to the classes and functions we added earlier.
+What the individual calls do is the following:
+
+* :meth:`~Sphinx.add_config_value` lets Sphinx know that it should recognize the
+  new *config value* ``todo_include_todos``, whose default value should be
+  ``False`` (this also tells Sphinx that it is a boolean value).
+
+  If the third argument was ``'html'``, HTML documents would be full rebuild if the
+  config value changed its value.  This is needed for config values that
+  influence reading (build :ref:`phase 1 (reading) <build-phases>`).
+
+* :meth:`~Sphinx.add_node` adds a new *node class* to the build system.  It also
+  can specify visitor functions for each supported output format.  These visitor
+  functions are needed when the new nodes stay until :ref:`phase 4 (writing)
+  <build-phases>`. Since the ``todolist`` node is always replaced in
+  :ref:`phase 3 (resolving) <build-phases>`, it doesn't need any.
+
+* :meth:`~Sphinx.add_directive` adds a new *directive*, given by name and class.
+
+* Finally, :meth:`~Sphinx.connect` adds an *event handler* to the event whose
+  name is given by the first argument.  The event handler function is called
+  with several arguments which are documented with the event.
+
+With this, our extension is complete.
+
+
+Using the extension
+-------------------
+
+As before, we need to enable the extension by declaring it in our
+:file:`conf.py` file. There are two steps necessary here:
+
+#. Add the :file:`_ext` directory to the `Python path`_ using
+   ``sys.path.append``. This should be placed at the top of the file.
+
+#. Update or create the :confval:`extensions` list and add the extension file
+   name to the list
+
+In addition, we may wish to set the ``todo_include_todos`` config value. As
+noted above, this defaults to ``False`` but we can set it explicitly.
+
+For example:
+
+.. code-block:: python
+
+   import os
+   import sys
+
+   sys.path.append(os.path.abspath("./_ext"))
+
+   extensions = ['todo']
+
+   todo_include_todos = False
+
+You can now use the extension throughout your project. For example:
+
+.. code-block:: rst
+   :caption: index.rst
+
+   Hello, world
+   ============
+
+   .. toctree::
+      somefile.rst
+      someotherfile.rst
+
+   Hello world. Below is the list of TODOs.
+
+   .. todolist::
+
+.. code-block:: rst
+   :caption: somefile.rst
+
+   foo
+   ===
+
+   Some intro text here...
+
+   .. todo:: Fix this
+
+.. code-block:: rst
+   :caption: someotherfile.rst
+
+   bar
+   ===
+
+   Some more text here...
+
+   .. todo:: Fix that
+
+Because we have configured ``todo_include_todos`` to ``False``, we won't
+actually see anything rendered for the ``todo`` and ``todolist`` directives.
+However, if we toggle this to true, we will see the output described
+previously.
+
+
+Further reading
+---------------
+
+For more information, refer to the `docutils`_ documentation and
+:doc:`/extdev/index`.
+
+
+.. _docutils: https://docutils.sourceforge.io/docs/
+.. _Python path: https://docs.python.org/3/using/cmdline.html#envvar-PYTHONPATH
+.. _docutils documentation: https://docutils.sourceforge.io/docs/ref/rst/directives.html