Adding upstream version 18.2.2.upstream/18.2.2

Signed-off-by: Daniel Baumann <daniel.baumann@progress-linux.org>

7 files changed, 2060 insertions, 0 deletions

diff --git a/doc/start/ceph.conf b/doc/start/ceph.conf
new file mode 100644
index 000000000..f3d558eb0
--- /dev/null
+++ b/doc/start/ceph.conf
@@ -0,0 +1,3 @@
+[global]
+	# list your monitors here
+	mon host = {mon-host-1}, {mon-host-2}
diff --git a/doc/start/documenting-ceph.rst b/doc/start/documenting-ceph.rst
new file mode 100644
index 000000000..02d4dccc4
--- /dev/null
+++ b/doc/start/documenting-ceph.rst
@@ -0,0 +1,1085 @@
+.. _documenting_ceph:
+
+==================
+ Documenting Ceph
+==================
+
+You can help the Ceph project by contributing to the documentation.  Even
+small contributions help the Ceph project. 
+
+The easiest way to suggest a correction to the documentation is to send an
+email to `ceph-users@ceph.io`. Include the string "ATTN: DOCS" or
+"Attention: Docs" or "Attention: Documentation" in the subject line.  In
+the body of the email, include the text to be corrected (so that I can find
+it in the repo) and include your correction.
+
+Another way to suggest a documentation correction is to make a pull request.
+The instructions for making a pull request against the Ceph documentation are
+in the section :ref:`making_contributions`.
+
+If this is your first time making an improvement to the documentation or
+if you have noticed a small mistake (such as a spelling error or a typo),
+it will be easier to send an email than to make a pull request. You will
+be credited for the improvement unless you instruct Ceph Upstream
+Documentation not to credit you.
+
+Location of the Documentation in the Repository
+===============================================
+
+The Ceph documentation source is in the ``ceph/doc`` directory of the Ceph
+repository. Python Sphinx renders the source into HTML and manpages. 
+
+Viewing Old Ceph Documentation
+==============================
+The https://docs.ceph.com link displays the latest release branch by default
+(for example, if "Quincy" is the most recent release, then by default
+https://docs.ceph.com displays the documentation for Quincy), but you can view
+the documentation for older versions of Ceph (for example, ``pacific``) by
+replacing the version name in the url (for example, ``quincy`` in
+`https://docs.ceph.com/en/pacific <https://docs.ceph.com/en/quincy>`_) with the
+branch name you prefer (for example, ``pacific``, to create a URL that reads
+`https://docs.ceph.com/en/pacific/ <https://docs.ceph.com/en/pacific/>`_).
+
+.. _making_contributions:
+
+Making Contributions
+====================
+
+Making a documentation contribution involves the same basic procedure as making
+a code contribution, with one exception: you must build documentation source
+instead of compiling program source. This sequence (the sequence of building
+the documentation source) includes the following steps:
+
+#. `Get the Source`_
+#. `Select a Branch`_
+#. `Make a Change`_
+#. `Build the Source`_
+#. `Commit the Change`_
+#. `Push the Change`_
+#. `Make a Pull Request`_
+#. `Notify Us`_
+
+Get the Source
+--------------
+
+The source of the Ceph documentation is a collection of ReStructured Text files
+that are in the Ceph repository in the ``ceph/doc`` directory. For details
+on GitHub and Ceph, see :ref:`Get Involved`.
+
+Use the `Fork and Pull`_ approach to make documentation contributions. To do
+this, you must:
+
+#. Install git locally. In Debian or Ubuntu, run the following command:
+
+   .. prompt:: bash $
+
+	sudo apt-get install git
+
+   In Fedora, run the following command:
+
+   .. prompt:: bash $
+
+	sudo yum install git
+
+   In CentOS/RHEL, run the following command:
+
+   .. prompt:: bash $
+
+	sudo yum install git
+
+#. Make sure that your ``.gitconfig`` file has been configured to include your
+   name and email address:
+
+   .. code-block:: ini
+
+	[user]
+	   email = {your-email-address}
+	   name = {your-name}
+
+   For example:
+
+   .. prompt:: bash $
+
+	git config --global user.name "John Doe"
+	git config --global user.email johndoe@example.com
+
+
+#. Create a  `github`_ account (if you don't have one).
+
+#. Fork the Ceph project. See https://github.com/ceph/ceph.
+
+#. Clone your fork of the Ceph project to your local host. This creates what is
+   known as a "local working copy".
+
+The Ceph documentation is organized by component:
+
+- **Ceph Storage Cluster:** The Ceph Storage Cluster documentation is 
+  in the ``doc/rados`` directory.
+  
+- **Ceph Block Device:** The Ceph Block Device documentation is in 
+  the ``doc/rbd`` directory.
+  
+- **Ceph Object Storage:** The Ceph Object Storage documentation is in 
+  the ``doc/radosgw`` directory.
+
+- **Ceph File System:** The Ceph File System documentation is in the 
+  ``doc/cephfs`` directory.
+  
+- **Installation (Quick):** Quick start documentation is in the
+  ``doc/start`` directory.
+  
+- **Installation (Manual):** Documentaton concerning the manual installation of
+  Ceph is in the ``doc/install`` directory.
+  
+- **Manpage:** Manpage source is in the ``doc/man`` directory.
+
+- **Developer:** Developer documentation is in the ``doc/dev`` 
+  directory.
+
+- **Images:** Images including JPEG and PNG files are stored in the
+  ``doc/images`` directory.
+
+
+Select a Branch
+---------------
+
+When you make small changes to the documentation, such as fixing typographical
+errors or clarifying explanations, use the ``main`` branch (default). You
+should also use the ``main`` branch when making contributions to features that
+are in the current release. ``main`` is the most commonly used branch. :
+
+.. prompt:: bash $
+
+	git checkout main
+
+When you make changes to documentation that affect an upcoming release, use 
+the ``next`` branch. ``next`` is the second most commonly used branch. :
+
+.. prompt:: bash $
+
+	git checkout next
+
+When you are making substantial contributions such as new features that are not
+yet in the current release; if your contribution is related to an issue with a
+tracker ID; or, if you want to see your documentation rendered on the Ceph.com
+website before it gets merged into the ``main`` branch, you should create a
+branch. To distinguish branches that include only documentation updates, we
+prepend them with ``wip-doc`` by convention, following the form
+``wip-doc-{your-branch-name}``. If the branch relates to an issue filed in
+http://tracker.ceph.com/issues, the branch name incorporates the issue number.
+For example, if a documentation branch is a fix for issue #4000, the branch name
+should be ``wip-doc-4000`` by convention and the relevant tracker URL will be
+http://tracker.ceph.com/issues/4000.
+
+.. note:: Please do not mingle documentation contributions and source code
+   contributions in a single commit. When you keep documentation
+   commits separate from source code commits, it simplifies the review
+   process. We highly recommend that any pull request that adds a feature or
+   a configuration option should also include a documentation commit that
+   describes the changes.
+
+Before you create your branch name, ensure that it doesn't already exist in the
+local or remote repository. :
+
+.. prompt:: bash $
+
+	git branch -a | grep wip-doc-{your-branch-name}
+
+If it doesn't exist, create your branch:
+
+.. prompt:: bash $
+
+	git checkout -b wip-doc-{your-branch-name}
+
+
+Make a Change
+-------------
+
+Modifying a document involves opening a reStructuredText file, changing
+its contents, and saving the changes. See `Documentation Style Guide`_ for
+details on syntax requirements.
+
+Adding a document involves creating a new reStructuredText file within the
+``doc`` directory tree with a ``*.rst``
+extension. You must also include a reference to the document: a hyperlink
+or a table of contents entry. The ``index.rst`` file of a top-level directory
+usually contains a TOC, where you can add the new file name. All documents must
+have a title. See `Headings`_ for details.
+
+Your new document doesn't get tracked by ``git`` automatically. When you want 
+to add the document to the repository,  you must use ``git add 
+{path-to-filename}``. For example, from the top level  directory of the
+repository, adding an ``example.rst`` file to the ``rados`` subdirectory would
+look like this:
+
+.. prompt:: bash $
+
+	git add doc/rados/example.rst
+
+Deleting a document involves removing it from the repository with ``git rm
+{path-to-filename}``. For example:
+
+.. prompt:: bash $
+
+	git rm doc/rados/example.rst
+
+You must also remove any reference to a deleted document from other documents.
+
+
+Build the Source
+----------------
+
+To build the documentation, navigate to the ``ceph`` repository directory:
+
+
+.. prompt:: bash $
+
+	cd ceph
+
+.. note::
+   The directory that contains ``build-doc`` and ``serve-doc`` must be included
+   in the ``PATH`` environment variable in order for these commands to work.
+
+
+To build the documentation on Debian/Ubuntu, Fedora, or CentOS/RHEL, execute:
+
+.. prompt:: bash $
+
+	admin/build-doc
+
+To scan for the reachability of external links, execute:
+
+.. prompt:: bash $
+
+	admin/build-doc linkcheck
+
+Executing ``admin/build-doc`` will create a ``build-doc`` directory under
+``ceph``.  You may need to create a directory under ``ceph/build-doc`` for
+output of Javadoc files:
+
+.. prompt:: bash $
+
+	mkdir -p output/html/api/libcephfs-java/javadoc
+
+The build script ``build-doc`` will produce an output of errors and warnings.
+You MUST fix errors in documents you modified before committing a change, and
+you SHOULD fix warnings that are related to syntax you modified.
+
+.. important:: You must validate ALL HYPERLINKS. If a hyperlink is broken,
+   it automatically breaks the build!
+
+Once you build the documentation set, you may start an HTTP server at
+``http://localhost:8080/`` to view it:
+
+.. prompt:: bash $
+
+	admin/serve-doc
+
+You can also navigate to ``build-doc/output`` to inspect the built documents.
+There should be an ``html`` directory and a ``man`` directory containing
+documentation in HTML and manpage formats respectively.
+
+Build the Source (First Time)
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Ceph uses Python Sphinx, which is generally distribution agnostic. The first
+time you build Ceph documentation, it will generate a doxygen XML tree, which
+is a bit time consuming.
+
+Python Sphinx does have some dependencies that vary across distributions. The
+first time you build the documentation, the script will notify you if you do not
+have the dependencies installed. To run Sphinx and build documentation successfully,
+the following packages are required:
+
+.. raw:: html
+
+	<style type="text/css">div.body h3{margin:5px 0px 0px 0px;}</style>
+	<table cellpadding="10"><colgroup><col width="30%"><col width="30%"><col width="30%"></colgroup><tbody valign="top"><tr><td><h3>Debian/Ubuntu</h3>
+
+- gcc
+- python3-dev
+- python3-pip
+- python3-sphinx
+- python3-venv
+- libxml2-dev
+- libxslt1-dev
+- doxygen
+- graphviz
+- ant
+- ditaa
+
+.. raw:: html
+
+	</td><td><h3>Fedora</h3>
+
+- gcc
+- python-devel
+- python-pip
+- python-docutils
+- python-jinja2
+- python-pygments
+- python-sphinx
+- libxml2-devel
+- libxslt1-devel
+- doxygen
+- graphviz
+- ant
+- ditaa
+
+.. raw:: html
+
+	</td><td><h3>CentOS/RHEL</h3>
+
+- gcc
+- python-devel
+- python-pip
+- python-docutils
+- python-jinja2
+- python-pygments
+- python-sphinx
+- libxml2-dev
+- libxslt1-dev
+- doxygen
+- graphviz
+- ant
+
+.. raw:: html
+
+	</td></tr></tbody></table>
+
+
+Install each dependency that is not installed on your host. For Debian/Ubuntu
+distributions, execute the following:
+
+.. prompt:: bash $
+
+	sudo apt-get install gcc python-dev python3-pip libxml2-dev libxslt-dev doxygen graphviz ant ditaa
+	sudo apt-get install python3-sphinx python3-venv
+
+For Fedora distributions, execute the following:
+
+.. prompt:: bash $
+
+   sudo yum install gcc python-devel python-pip libxml2-devel libxslt-devel doxygen graphviz ant
+   sudo pip install html2text
+   sudo yum install python-jinja2 python-pygments python-docutils python-sphinx
+   sudo yum install jericho-html ditaa
+
+For CentOS/RHEL distributions, it is recommended to have ``epel`` (Extra
+Packages for Enterprise Linux) repository as it provides some extra packages
+which are not available in the default repository. To install ``epel``, execute
+the following:
+
+.. prompt:: bash $
+
+        sudo yum install -y https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm
+
+For CentOS/RHEL distributions, execute the following:
+
+.. prompt:: bash $
+
+	sudo yum install gcc python-devel python-pip libxml2-devel libxslt-devel doxygen graphviz ant
+	sudo pip install html2text
+
+For CentOS/RHEL distributions, the remaining python packages are not available
+in the default and ``epel`` repositories. So, use http://rpmfind.net/ to find
+the packages. Then, download them from a mirror and install them. For example:
+
+.. prompt:: bash $
+
+	wget http://rpmfind.net/linux/centos/7/os/x86_64/Packages/python-jinja2-2.7.2-2.el7.noarch.rpm
+	sudo yum install python-jinja2-2.7.2-2.el7.noarch.rpm
+	wget http://rpmfind.net/linux/centos/7/os/x86_64/Packages/python-pygments-1.4-9.el7.noarch.rpm
+	sudo yum install python-pygments-1.4-9.el7.noarch.rpm
+	wget http://rpmfind.net/linux/centos/7/os/x86_64/Packages/python-docutils-0.11-0.2.20130715svn7687.el7.noarch.rpm
+	sudo yum install python-docutils-0.11-0.2.20130715svn7687.el7.noarch.rpm
+	wget http://rpmfind.net/linux/centos/7/os/x86_64/Packages/python-sphinx-1.1.3-11.el7.noarch.rpm
+	sudo yum install python-sphinx-1.1.3-11.el7.noarch.rpm
+
+Ceph documentation makes extensive use of `ditaa`_, which is not presently built
+for CentOS/RHEL7. You must install ``ditaa`` if you are making changes to
+``ditaa`` diagrams so that you can verify that they render properly before you
+commit new or modified ``ditaa`` diagrams. You may retrieve compatible required
+packages for CentOS/RHEL distributions and install them manually. To run
+``ditaa`` on CentOS/RHEL7, following dependencies are required:
+
+- jericho-html
+- jai-imageio-core
+- batik
+
+Use http://rpmfind.net/ to find compatible ``ditaa`` and the dependencies.
+Then, download them from a mirror and install them. For example:
+
+.. prompt:: bash $
+
+	wget http://rpmfind.net/linux/fedora/linux/releases/22/Everything/x86_64/os/Packages/j/jericho-html-3.3-4.fc22.noarch.rpm
+	sudo yum install jericho-html-3.3-4.fc22.noarch.rpm
+	wget http://rpmfind.net/linux/centos/7/os/x86_64/Packages/jai-imageio-core-1.2-0.14.20100217cvs.el7.noarch.rpm
+	sudo yum install jai-imageio-core-1.2-0.14.20100217cvs.el7.noarch.rpm
+	wget http://rpmfind.net/linux/centos/7/os/x86_64/Packages/batik-1.8-0.12.svn1230816.el7.noarch.rpm
+	sudo yum install batik-1.8-0.12.svn1230816.el7.noarch.rpm
+	wget http://rpmfind.net/linux/fedora/linux/releases/22/Everything/x86_64/os/Packages/d/ditaa-0.9-13.r74.fc21.noarch.rpm
+	sudo yum install ditaa-0.9-13.r74.fc21.noarch.rpm
+
+Once you have installed all these packages, build the documentation by following
+the steps given in `Build the Source`_.
+
+
+Commit the Change
+-----------------
+
+Ceph documentation commits are simple, but follow a strict convention:
+
+- A commit SHOULD have 1 file per commit (it simplifies rollback). You MAY
+  commit multiple files with related changes. Unrelated changes SHOULD NOT
+  be put into the same commit.
+- A commit MUST have a comment.
+- A commit comment MUST be prepended with ``doc:``. (strict)
+- The comment summary MUST be one line only. (strict)
+- Additional comments MAY follow a blank line after the summary, 
+  but should be terse.
+- A commit MAY include ``Fixes: https://tracker.ceph.com/issues/{bug number}``.
+- Commits MUST include ``Signed-off-by: Firstname Lastname <email>``. (strict)
+
+.. tip:: Follow the foregoing convention particularly where it says 
+   ``(strict)`` or you will be asked to modify your commit to comply with 
+   this convention.
+
+The following is a common commit comment (preferred):: 
+
+	doc: Fixes a spelling error and a broken hyperlink.
+	
+	Signed-off-by: John Doe <john.doe@gmail.com>
+
+
+The following comment includes a reference to a bug. :: 
+
+	doc: Fixes a spelling error and a broken hyperlink.
+
+	Fixes: https://tracker.ceph.com/issues/1234
+	
+	Signed-off-by: John Doe <john.doe@gmail.com>
+
+
+The following comment includes a terse sentence following the comment summary.
+There is a carriage return between the summary line and the description:: 
+
+	doc: Added mon setting to monitor config reference
+	
+	Describes 'mon setting', which is a new setting added
+	to config_opts.h.
+	
+	Signed-off-by: John Doe <john.doe@gmail.com>
+
+
+To commit changes, execute the following:
+
+.. prompt:: bash $
+
+	git commit -a
+	
+
+An easy way to manage your documentation commits is to use visual tools for
+``git``. For example, ``gitk`` provides a graphical interface for viewing the
+repository history, and ``git-gui`` provides a graphical interface for viewing
+your uncommitted changes, staging them for commit, committing the changes and
+pushing them to your forked Ceph repository.
+
+
+For Debian/Ubuntu, execute:
+
+.. prompt:: bash $
+
+	sudo apt-get install gitk git-gui
+
+For Fedora/CentOS/RHEL, execute:
+
+.. prompt:: bash $
+
+	sudo yum install gitk git-gui
+
+Then, execute:
+
+.. prompt:: bash $
+
+	cd {git-ceph-repo-path}
+	gitk
+	
+Finally, select **File->Start git gui** to activate the graphical user interface.
+
+
+Push the Change
+---------------
+
+Once you have one or more commits, you must push them from the local copy of the
+repository to ``github``. A graphical tool like ``git-gui`` provides a user
+interface for pushing to the repository. If you created a branch previously:
+
+.. prompt:: bash $
+
+	git push origin wip-doc-{your-branch-name}
+
+Otherwise:
+
+.. prompt:: bash $
+
+	git push
+
+
+Make a Pull Request
+-------------------
+
+As noted earlier, you can make documentation contributions using the `Fork and
+Pull`_ approach.
+
+
+Squash Extraneous Commits
+-------------------------
+Each pull request ought to be associated with only a single commit. If you have
+made more than one commit to the feature branch that you are working in, you
+will need to "squash" the multiple commits. "Squashing" is the colloquial term
+for a particular kind of "interactive rebase". Squashing can be done in a great
+number of ways, but the example here will deal with a situation in which there
+are three commits and the changes in all three of the commits are kept. The three
+commits will be squashed into a single commit.
+
+#. Make the commits that you will later squash.
+
+   #. Make the first commit.
+   
+      ::
+   
+         doc/glossary: improve "CephX" entry
+   
+         Improve the glossary entry for "CephX".
+   
+         Signed-off-by: Zac Dover <zac.dover@proton.me>
+   
+         # Please enter the commit message for your changes. Lines starting
+         # with '#' will be ignored, and an empty message aborts the commit.
+         #
+         # On branch wip-doc-2023-03-28-glossary-cephx
+         # Changes to be committed:
+         #       modified:   glossary.rst
+         #
+   
+   #. Make the second commit.
+   
+      ::
+   
+         doc/glossary: add link to architecture doc
+         
+         Add a link to a section in the architecture document, which link
+         will be used in the process of improving the "CephX" glossary entry.
+         
+         Signed-off-by: Zac Dover <zac.dover@proton.me>
+      
+            # Please enter the commit message for your changes. Lines starting
+            # with '#' will be ignored, and an empty message aborts the commit.
+            #
+            # On branch wip-doc-2023-03-28-glossary-cephx
+            # Your branch is up to date with 'origin/wip-doc-2023-03-28-glossary-cephx'.
+            #
+            # Changes to be committed:
+            #       modified:   architecture.rst
+      
+   #. Make the third commit.
+   
+      ::
+      
+         doc/glossary: link to Arch doc in "CephX" glossary
+         
+         Link to the Architecture document from the "CephX" entry in the
+         Glossary.
+         
+         Signed-off-by: Zac Dover <zac.dover@proton.me>
+         
+         # Please enter the commit message for your changes. Lines starting
+         # with '#' will be ignored, and an empty message aborts the commit.
+         #
+         # On branch wip-doc-2023-03-28-glossary-cephx
+         # Your branch is up to date with 'origin/wip-doc-2023-03-28-glossary-cephx'.
+         #
+         # Changes to be committed:
+         #       modified:   glossary.rst
+
+#. There are now three commits in the feature branch. We will now begin the
+   process of squashing them into a single commit. 
+   
+   #. Run the command ``git rebase -i main``, which rebases the current branch 
+      (the feature branch) against the ``main`` branch:
+
+      .. prompt:: bash
+   
+         git rebase -i main
+   
+   #. A list of the commits that have been made to the feature branch now
+      appear, and looks like this:
+
+      ::
+      
+         pick d395e500883 doc/glossary: improve "CephX" entry
+         pick b34986e2922 doc/glossary: add link to architecture doc
+         pick 74d0719735c doc/glossary: link to Arch doc in "CephX" glossary
+         
+         # Rebase 0793495b9d1..74d0719735c onto 0793495b9d1 (3 commands)
+         #
+         # Commands:
+         # p, pick <commit> = use commit
+         # r, reword <commit> = use commit, but edit the commit message
+         # e, edit <commit> = use commit, but stop for amending
+         # s, squash <commit> = use commit, but meld into previous commit
+         # f, fixup [-C | -c] <commit> = like "squash" but keep only the previous
+         #                    commit's log message, unless -C is used, in which case
+         #                    keep only this commit's message; -c is same as -C but
+         #                    opens the editor
+         # x, exec <command> = run command (the rest of the line) using shell
+         # b, break = stop here (continue rebase later with 'git rebase --continue')
+         # d, drop <commit> = remove commit
+         # l, label <label> = label current HEAD with a name
+         # t, reset <label> = reset HEAD to a label
+         # m, merge [-C <commit> | -c <commit>] <label> [# <oneline>]
+         #         create a merge commit using the original merge commit's
+         #         message (or the oneline, if no original merge commit was
+         #         specified); use -c <commit> to reword the commit message
+         # u, update-ref <ref> = track a placeholder for the <ref> to be updated
+         #                       to this position in the new commits. The <ref> is
+         #                       updated at the end of the rebase
+         #
+         # These lines can be re-ordered; they are executed from top to bottom.
+         #
+         # If you remove a line here THAT COMMIT WILL BE LOST.
+
+      Find the part of the screen that says "pick". This is the part that you will 
+      alter. There are three commits that are currently labeled "pick". We will
+      choose one of them to remain labeled "pick", and we will label the other two
+      commits "squash".
+
+#. Label two of the three commits ``squash``:
+
+   ::
+
+      pick d395e500883 doc/glossary: improve "CephX" entry
+      squash b34986e2922 doc/glossary: add link to architecture doc
+      squash 74d0719735c doc/glossary: link to Arch doc in "CephX" glossary
+      
+      # Rebase 0793495b9d1..74d0719735c onto 0793495b9d1 (3 commands)
+      #
+      # Commands:
+      # p, pick <commit> = use commit
+      # r, reword <commit> = use commit, but edit the commit message
+      # e, edit <commit> = use commit, but stop for amending
+      # s, squash <commit> = use commit, but meld into previous commit
+      # f, fixup [-C | -c] <commit> = like "squash" but keep only the previous
+      #                    commit's log message, unless -C is used, in which case
+      #                    keep only this commit's message; -c is same as -C but
+      #                    opens the editor
+      # x, exec <command> = run command (the rest of the line) using shell
+      # b, break = stop here (continue rebase later with 'git rebase --continue')
+      # d, drop <commit> = remove commit
+      # l, label <label> = label current HEAD with a name
+      # t, reset <label> = reset HEAD to a label
+      # m, merge [-C <commit> | -c <commit>] <label> [# <oneline>]
+      #         create a merge commit using the original merge commit's
+      #         message (or the oneline, if no original merge commit was
+      #         specified); use -c <commit> to reword the commit message
+      # u, update-ref <ref> = track a placeholder for the <ref> to be updated
+      #                       to this position in the new commits. The <ref> is
+      #                       updated at the end of the rebase
+      #
+      # These lines can be re-ordered; they are executed from top to bottom.
+      #
+      # If you remove a line here THAT COMMIT WILL BE LOST.
+
+#. Now we create a commit message that applies to all the commits that have
+   been squashed together:
+
+   #. When you save and close the list of commits that you have designated for
+      squashing, a list of all three commit messages appears, and it looks
+      like this:
+
+      ::
+      
+         # This is a combination of 3 commits.
+         # This is the 1st commit message:
+      
+         doc/glossary: improve "CephX" entry
+      
+         Improve the glossary entry for "CephX".
+      
+         Signed-off-by: Zac Dover <zac.dover@proton.me>
+      
+         # This is the commit message #2:
+      
+         doc/glossary: add link to architecture doc
+      
+         Add a link to a section in the architecture document, which link
+         will be used in the process of improving the "CephX" glossary entry.
+      
+         Signed-off-by: Zac Dover <zac.dover@proton.me>
+      
+         # This is the commit message #3:
+      
+         doc/glossary: link to Arch doc in "CephX" glossary
+      
+         Link to the Architecture document from the "CephX" entry in the
+         Glossary.
+      
+         Signed-off-by: Zac Dover <zac.dover@proton.me>
+      
+         # Please enter the commit message for your changes. Lines starting
+         # with '#' will be ignored, and an empty message aborts the commit.
+         #
+         # Date:      Tue Mar 28 18:42:11 2023 +1000
+         #
+         # interactive rebase in progress; onto 0793495b9d1
+         # Last commands done (3 commands done):
+         #    squash b34986e2922 doc/glossary: add link to architecture doc
+         #    squash 74d0719735c doc/glossary: link to Arch doc in "CephX" glossary
+         # No commands remaining.
+         # You are currently rebasing branch 'wip-doc-2023-03-28-glossary-cephx' on '0793495b9d1'.
+         #
+         # Changes to be committed:
+         #       modified:   doc/architecture.rst
+         #       modified:   doc/glossary.rst
+      
+   #. The commit messages have been revised into the simpler form presented here:   
+            
+      ::
+      
+         doc/glossary: improve "CephX" entry
+      
+         Improve the glossary entry for "CephX".
+      
+         Signed-off-by: Zac Dover <zac.dover@proton.me>
+      
+         # Please enter the commit message for your changes. Lines starting
+         # with '#' will be ignored, and an empty message aborts the commit.
+         #
+         # Date:      Tue Mar 28 18:42:11 2023 +1000
+         #
+         # interactive rebase in progress; onto 0793495b9d1
+         # Last commands done (3 commands done):
+         #    squash b34986e2922 doc/glossary: add link to architecture doc
+         #    squash 74d0719735c doc/glossary: link to Arch doc in "CephX" glossary
+         # No commands remaining.
+         # You are currently rebasing branch 'wip-doc-2023-03-28-glossary-cephx' on '0793495b9d1'.
+         #
+         # Changes to be committed:
+         #       modified:   doc/architecture.rst
+         #       modified:   doc/glossary.rst
+
+#. Force push the squashed commit from your local working copy to the remote
+   upstream branch. The force push is necessary because the newly squashed commit
+   does not have an ancestor in the remote. If that confuses you, just run this 
+   command and don't think too much about it:
+
+   .. prompt:: bash $  
+
+      git push -f
+   
+   ::
+
+      Enumerating objects: 9, done.
+      Counting objects: 100% (9/9), done.
+      Delta compression using up to 8 threads
+      Compressing objects: 100% (5/5), done.
+      Writing objects: 100% (5/5), 722 bytes | 722.00 KiB/s, done.
+      Total 5 (delta 4), reused 0 (delta 0), pack-reused 0
+      remote: Resolving deltas: 100% (4/4), completed with 4 local objects.
+      To github.com:zdover23/ceph.git
+       + b34986e2922...02e3a5cb763 wip-doc-2023-03-28-glossary-cephx -> wip-doc-2023-03-28-glossary-cephx (forced update)
+
+
+
+
+
+Notify Us
+---------
+
+If some time has passed and the pull request that you raised has not been
+reviewed, contact the component lead and ask what's taking so long. See
+:ref:`ctl` for a list of component leads.
+
+Documentation Style Guide
+=========================
+
+One objective of the Ceph documentation project is to ensure the readability of
+the documentation in both native reStructuredText format and its rendered
+formats such as HTML. Navigate to your Ceph repository and view a document in
+its native format. You may notice that it is generally as legible in a terminal
+as it is in its rendered HTML format. Additionally, you may also notice that
+diagrams in ``ditaa`` format also render reasonably well in text mode. :
+
+.. prompt:: bash $
+
+	less doc/architecture.rst
+
+Review the following style guides to maintain this consistency.
+
+
+Headings
+--------
+
+#. **Document Titles:** Document titles use the ``=`` character overline and 
+   underline with a leading and trailing space on the title text line. 
+   See `Document Title`_ for details.
+
+#. **Section Titles:** Section tiles use the ``=`` character underline with no
+   leading or trailing spaces for text. Two carriage returns should precede a 
+   section title (unless an inline reference precedes it). See `Sections`_ for
+   details.
+
+#. **Subsection Titles:** Subsection titles use the ``_`` character underline 
+   with no leading or trailing spaces for text.  Two carriage returns should 
+   precede a subsection title (unless an inline reference precedes it).
+
+
+Text Body
+---------
+
+As a general rule, we prefer text to wrap at column 80 so that it is legible in
+a command line interface without leading or trailing white space. Where
+possible, we prefer to maintain this convention with text, lists, literal text
+(exceptions allowed), tables, and ``ditaa`` graphics.
+
+#. **Paragraphs**: Paragraphs have a leading and a trailing carriage return, 
+   and should be 80 characters wide or less so that the documentation can be 
+   read in native format in a command line terminal.
+
+#. **Literal Text:** To create an example of literal text (e.g., command line
+   usage), terminate the preceding paragraph with ``::`` or enter a carriage
+   return to create an empty line after the preceding paragraph; then, enter
+   ``::`` on a separate line followed by another empty line. Then, begin the
+   literal text with tab indentation (preferred) or space indentation of 3 
+   characters.
+
+#. **Indented Text:** Indented text such as bullet points 
+   (e.g., ``- some text``) may span multiple lines. The text of subsequent
+   lines should begin at the same character position as the text of the
+   indented text (less numbers, bullets, etc.).
+
+   Indented text may include literal text examples. Whereas, text indentation
+   should be done with spaces, literal text examples should be indented with
+   tabs. This convention enables you to add an additional indented paragraph
+   following a literal example by leaving a blank line and beginning the
+   subsequent paragraph with space indentation.
+
+#. **Numbered Lists:** Numbered lists should use autonumbering by starting
+   a numbered indent with ``#.`` instead of the actual number so that
+   numbered paragraphs can be repositioned without requiring manual 
+   renumbering.
+
+#. **Code Examples:** Ceph supports the use of the 
+   ``.. code-block::<language>`` role, so that you can add highlighting to 
+   source examples. This is preferred for source code. However, use of this 
+   tag will cause autonumbering to restart at 1 if it is used as an example 
+   within a numbered list. See `Showing code examples`_ for details.
+
+
+Paragraph Level Markup
+----------------------
+
+The Ceph project uses `paragraph level markup`_ to highlight points.
+
+#. **Tip:** Use the ``.. tip::`` directive to provide additional information
+   that assists the reader or steers the reader away from trouble.
+
+#. **Note**: Use the ``.. note::`` directive to highlight an important point.
+
+#. **Important:** Use the ``.. important::`` directive to highlight important
+   requirements or caveats (e.g., anything that could lead to data loss). Use
+   this directive sparingly, because it renders in red.
+
+#. **Version Added:** Use the ``.. versionadded::`` directive for new features
+   or configuration settings so that users know the minimum release for using
+   a feature.
+   
+#. **Version Changed:** Use the ``.. versionchanged::`` directive for changes
+   in usage or configuration settings.
+
+#. **Deprecated:** Use the ``.. deprecated::`` directive when CLI usage, 
+   a feature or a configuration setting is no longer preferred or will be 
+   discontinued.
+
+#. **Topic:** Use the ``.. topic::`` directive to encapsulate text that is
+   outside the main flow of the document. See the `topic directive`_ for
+   additional details.
+
+
+Table of Contents (TOC) and Hyperlinks
+---------------------------------------
+
+The documents in the Ceph documentation suite follow certain conventions that
+are explained in this section.
+
+Every document (every ``.rst`` file) in the Sphinx-controlled Ceph
+documentation suite must be linked either (1) from another document in the
+documentation suite or (2) from a table of contents (TOC). If any document in
+the documentation suite is not linked in this way, the ``build-doc`` script
+generates warnings when it tries to build the documentation. 
+
+The Ceph project uses the ``.. toctree::`` directive. See `The TOC tree`_ for
+details. When rendering a table of contents (TOC), specify the ``:maxdepth:``
+parameter so that the rendered TOC is not too long.
+
+Use the ``:ref:`` syntax where a link target contains a specific unique
+identifier (for example, ``.. _unique-target-id:``). A link to the section
+designated by ``.. _unique-target-id:`` looks like this:
+``:ref:`unique-target-id```. If this convention is followed, the links within
+the ``.rst`` source files will work even if the source files are moved within
+the ``ceph/doc`` directory. See `Cross referencing arbitrary locations`_ for
+details.
+
+.. _start_external_hyperlink_example:
+
+External Hyperlink Example
+~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+It is also possible to create a link to a section of the documentation and to
+have custom text appear in the body of the link. This is useful when it is more
+important to preserve the text of the sentence containing the link than it is
+to refer explicitly to the title of the section being linked to.
+
+For example, RST that links to the Sphinx Python Document Generator homepage
+and generates a sentence reading "Click here to learn more about Python
+Sphinx." looks like this: 
+
+::
+
+    ``Click `here <https://www.sphinx-doc.org>`_ to learn more about Python
+    Sphinx.`` 
+
+And here it is, rendered:
+
+Click `here <https://www.sphinx-doc.org>`_ to learn more about Python Sphinx. 
+
+Pay special attention to the underscore after the backtick. If you forget to
+include it and this is your first day working with RST, there's a chance that
+you'll spend all day wondering what went wrong without realizing that you
+omitted that underscore. Also, pay special attention to the space between the
+substitution text (in this case, "here") and the less-than bracket that sets
+the explicit link apart from the substition text. The link will not render
+properly without this space.
+
+Linking Customs
+~~~~~~~~~~~~~~~
+
+By a custom established when Ceph was still being developed by Inktank,
+contributors to the documentation of the Ceph project preferred to use the
+convention of putting ``.. _Link Text: ../path`` links at the bottom of the
+document and linking to them using references of the form ``:ref:`path```. This
+convention was preferred because it made the documents more readable in a
+command line interface. As of 2023, though, we have no preference for one over
+the other. Use whichever convention makes the text easier to read.
+
+Using a part of a sentence as a hyperlink, `like this <docs.ceph.com>`_, is
+discouraged. The convention of writing "See X" is preferred. Here are some
+preferred formulations:
+
+#. For more information, see `docs.ceph.com <docs.ceph.com>`_.
+
+#. See `docs.ceph.com <docs.ceph.com>`_.
+
+
+Quirks of ReStructured Text
+---------------------------
+
+External Links
+~~~~~~~~~~~~~~
+
+.. _external_link_with_inline_text:
+
+Use the formula immediately below to render links that direct the reader to
+addresses external to the Ceph documentation:
+
+::
+
+   `inline text <http:www.foo.com>`_
+
+.. note:: Do not fail to include the space between the inline text and the
+   less-than sign. 
+   
+   Do not fail to include the underscore after the final backtick.
+
+   To link to addresses that are external to the Ceph documentation, include a
+   space between the inline text and the angle bracket that precedes the
+   external address. This is precisely the opposite of the convention for
+   inline text that links to a location inside the Ceph documentation. See
+   :ref:`here <internal_link_with_inline_text>` for an exemplar of this
+   convention.
+
+   If this seems inconsistent and confusing to you, then you're right. It is
+   inconsistent and confusing.
+
+See also ":ref:`External Hyperlink Example<start_external_hyperlink_example>`".
+
+Internal Links
+~~~~~~~~~~~~~~
+
+To link to a section in the Ceph documentation, you must (1) define a target
+link before the section and then (2) link to that target from another location
+in the documentation. Here are the formulas for targets and links to those
+targets:
+
+Target::
+
+   .. _target:
+
+   Title of Targeted Section
+   =========================
+
+   Lorem ipsum...
+
+Link to target::
+
+   :ref:`target`
+
+.. _internal_link_with_inline_text:
+
+Link to target with inline text::
+
+   :ref:`inline text<target>`
+
+.. note:: 
+
+   There is no space between "inline text" and the angle bracket that
+   immediately follows it. This is precisely the opposite of :ref:`the
+   convention for inline text that links to a location outside of the Ceph
+   documentation<external_link_with_inline_text>`. If this seems inconsistent
+   and confusing to you, then you're right. It is inconsistent and confusing.
+
+Escaping Bold Characters within Words
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+This section explains how to make certain letters within a word bold while
+leaving the other letters in the word regular (non-bold). 
+
+The following single-line paragraph provides an example of this:
+
+**C**\eph **F**\ile **S**\ystem.
+
+In ReStructured Text, the following formula will not work:
+
+::
+
+   **C**eph **F**ile **S**ystem
+
+The bolded notation must be turned off by means of the escape character (\\), as shown here:
+
+::
+
+   **C**\eph **F**\ile **S**\ystem
+
+.. _Python Sphinx: https://www.sphinx-doc.org
+.. _restructuredText: http://docutils.sourceforge.net/rst.html
+.. _Fork and Pull: https://help.github.com/articles/using-pull-requests
+.. _github: http://github.com
+.. _ditaa: http://ditaa.sourceforge.net/
+.. _Document Title: http://docutils.sourceforge.net/docs/user/rst/quickstart.html#document-title-subtitle
+.. _Sections: http://docutils.sourceforge.net/docs/user/rst/quickstart.html#sections
+.. _Cross referencing arbitrary locations: http://www.sphinx-doc.org/en/master/usage/restructuredtext/roles.html#role-ref
+.. _The TOC tree: http://sphinx-doc.org/markup/toctree.html
+.. _Showing code examples: http://sphinx-doc.org/markup/code.html
+.. _paragraph level markup: http://sphinx-doc.org/markup/para.html
+.. _topic directive: http://docutils.sourceforge.net/docs/ref/rst/directives.html#topic
diff --git a/doc/start/get-involved.rst b/doc/start/get-involved.rst
new file mode 100644
index 000000000..4f5277e37
--- /dev/null
+++ b/doc/start/get-involved.rst
@@ -0,0 +1,99 @@
+.. _Get Involved:
+
+=====================================
+ Get Involved in the Ceph Community!
+=====================================
+
+These are exciting times in the Ceph community! Get involved!
+
++----------------------+-------------------------------------------------+-----------------------------------------------+
+|Channel               | Description                                     | Contact Info                                  |
++======================+=================================================+===============================================+
+| **Blog**             | Check the Ceph Blog_ periodically to keep track | http://ceph.com/community/blog/               |
+|                      | of Ceph progress and important announcements.   |                                               |
++----------------------+-------------------------------------------------+-----------------------------------------------+
+| **Planet Ceph**      | Check the blog aggregation on Planet Ceph for   | https://old.ceph.com/category/planet/         |
+|                      | interesting stories, information and            |                                               |
+|                      | experiences from the community. **NOTE: NO      |                                               |
+|                      | longer updated as of 2023.**                    |                                               |
++----------------------+-------------------------------------------------+-----------------------------------------------+
+| **Wiki**             | Check the Ceph Wiki is a source for more        | http://wiki.ceph.com/                         |
+|                      | community and development related topics. You   |                                               |
+|                      | can find there information about blueprints,    |                                               |
+|                      | meetups, the Ceph Developer Summits and more.   |                                               |
++----------------------+-------------------------------------------------+-----------------------------------------------+
+| **IRC**              | As you delve into Ceph, you may have questions  |                                               |
+|                      | or feedback for the Ceph development team. Ceph | - **Domain:**                                 |
+|                      | developers are often available on the ``#ceph`` |     ``irc.oftc.net``                          |
+|                      | IRC channel particularly during daytime hours   | - **Channels:**                               |
+|                      | in the US Pacific Standard Time zone.           |     ``#ceph``,                                |
+|                      | While ``#ceph`` is a good starting point for    |     ``#ceph-devel``,                          |
+|                      | cluster operators and users, there is also      |     ``#ceph-dashboard``,                      |
+|                      | ``#ceph-devel``, ``#ceph-dashboard`` and        |     ``#cephfs``                               |
+|                      | ``#cephfs`` dedicated for Ceph developers.      |                                               |
++----------------------+-------------------------------------------------+-----------------------------------------------+
+| **User List**        | Ask and answer user-related questions by        |                                               |
+|                      | subscribing to the email list at                | - `User Subscribe`_                           |
+|                      | ceph-users@ceph.io. You can opt out of the email| - `User Unsubscribe`_                         |
+|                      | list at any time by unsubscribing. A simple     | - `User Archives`_                            |
+|                      | email is all it takes!                          |                                               |
++----------------------+-------------------------------------------------+-----------------------------------------------+
+| **Devel List**       | Keep in touch with developer activity by        |                                               |
+|                      | subscribing to the email list at dev@ceph.io.   | - `Devel Subscribe`_                          |
+|                      | You can opt out of the email list at any time by| - `Devel Unsubscribe`_                        |
+|                      | unsubscribing. A simple email is all it takes!  | - `Devel Archives`_                           |
++----------------------+-------------------------------------------------+-----------------------------------------------+
+| **Kernel Client**    | Linux kernel-related traffic, including kernel  | - `Kernel Client Subscribe`_                  |
+|                      | patches and discussion of implementation details| - `Kernel Client Unsubscribe`_                |
+|                      | for the kernel client code.                     | - `Kernel Client Archives`_                   |
++----------------------+-------------------------------------------------+-----------------------------------------------+
+| **Commit List**      | Subscribe to ceph-commit@ceph.com to get        |                                               |
+|                      | commit notifications via email. You can opt out | - `Commit Subscribe`_                         |
+|                      | of the email list at any time by unsubscribing. | - `Commit Unsubscribe`_                       |
+|                      | A simple email is all it takes!                 | - `Mailing list archives`_                    |
++----------------------+-------------------------------------------------+-----------------------------------------------+
+| **QA List**          | For Quality Assurance (QA) related activities   |                                               |
+|                      | subscribe to this list. You can opt out         | - `QA Subscribe`_                             |
+|                      | of the email list at any time by unsubscribing. | - `QA Unsubscribe`_                           |
+|                      | A simple email is all it takes!                 | - `Mailing list archives`_                    |
++----------------------+-------------------------------------------------+-----------------------------------------------+
+| **Community List**   | For all discussions related to the Ceph User    |                                               |
+|                      | Committee and other community topics. You can   | - `Community Subscribe`_                      |
+|                      | opt out of the email list at any time by        | - `Community Unsubscribe`_                    |
+|                      | unsubscribing. A simple email is all it takes!  | - `Mailing list archives`_                    |
++----------------------+-------------------------------------------------+-----------------------------------------------+
+| **Bug Tracker**      | You can help keep Ceph production worthy by     | http://tracker.ceph.com/projects/ceph         |
+|                      | filing and tracking bugs, and providing feature |                                               |
+|                      | requests using the Bug Tracker_.                |                                               |
++----------------------+-------------------------------------------------+-----------------------------------------------+
+| **Source Code**      | If you would like to participate in             |                                               |
+|                      | development, bug fixing, or if you just want    | - http://github.com/ceph/ceph                 |
+|                      | the very latest code for Ceph, you can get it   | - http://download.ceph.com/tarballs/          |
+|                      | at http://github.com. See `Ceph Source Code`_   |                                               |
+|                      | for details on cloning from github.             |                                               |
++----------------------+-------------------------------------------------+-----------------------------------------------+
+| **Ceph Calendar**    | Learn about upcoming Ceph events.               | https://ceph.io/contribute/                   |
++----------------------+-------------------------------------------------+-----------------------------------------------+
+
+
+
+.. _Devel Subscribe: mailto:dev-request@ceph.io?body=subscribe
+.. _Devel Unsubscribe: mailto:dev-request@ceph.io?body=unsubscribe
+.. _Kernel Client Subscribe: mailto:majordomo@vger.kernel.org?body=subscribe+ceph-devel
+.. _Kernel Client Unsubscribe: mailto:majordomo@vger.kernel.org?body=unsubscribe+ceph-devel
+.. _User Subscribe: mailto:ceph-users-request@ceph.io?body=subscribe
+.. _User Unsubscribe: mailto:ceph-users-request@ceph.io?body=unsubscribe
+.. _Community Subscribe: mailto:ceph-community-join@lists.ceph.com
+.. _Community Unsubscribe: mailto:ceph-community-leave@lists.ceph.com
+.. _Commit Subscribe: mailto:ceph-commit-join@lists.ceph.com
+.. _Commit Unsubscribe: mailto:ceph-commit-leave@lists.ceph.com
+.. _QA Subscribe: mailto:ceph-qa-join@lists.ceph.com
+.. _QA Unsubscribe: mailto:ceph-qa-leave@lists.ceph.com
+.. _Devel Archives: https://lists.ceph.io/hyperkitty/list/dev@ceph.io/
+.. _User Archives: https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/
+.. _Kernel Client Archives: https://www.spinics.net/lists/ceph-devel/
+.. _Mailing list archives: http://lists.ceph.com/
+.. _Blog: http://ceph.com/community/blog/
+.. _Tracker: http://tracker.ceph.com/
+.. _Ceph Source Code: http://github.com/ceph/ceph
+
diff --git a/doc/start/hardware-recommendations.rst b/doc/start/hardware-recommendations.rst
new file mode 100644
index 000000000..a63b5a457
--- /dev/null
+++ b/doc/start/hardware-recommendations.rst
@@ -0,0 +1,623 @@
+.. _hardware-recommendations:
+
+==========================
+ hardware recommendations
+==========================
+
+Ceph is designed to run on commodity hardware, which makes building and
+maintaining petabyte-scale data clusters flexible and economically feasible. 
+When planning your cluster's hardware, you will need to balance a number 
+of considerations, including failure domains, cost, and performance.
+Hardware planning should include distributing Ceph daemons and 
+other processes that use Ceph across many hosts. Generally, we recommend 
+running Ceph daemons of a specific type on a host configured for that type 
+of daemon. We recommend using separate hosts for processes that utilize your 
+data cluster (e.g., OpenStack, CloudStack, Kubernetes, etc).
+
+The requirements of one Ceph cluster are not the same as the requirements of
+another, but below are some general guidelines. 
+
+.. tip:: check out the `ceph blog`_ too.
+
+CPU
+===
+
+CephFS Metadata Servers (MDS) are CPU-intensive. They are
+are single-threaded and perform best with CPUs with a high clock rate (GHz). MDS
+servers do not need a large number of CPU cores unless they are also hosting other
+services, such as SSD OSDs for the CephFS metadata pool.
+OSD nodes need enough processing power to run the RADOS service, to calculate data
+placement with CRUSH, to replicate data, and to maintain their own copies of the
+cluster map.
+
+With earlier releases of Ceph, we would make hardware recommendations based on
+the number of cores per OSD, but this cores-per-osd metric is no longer as
+useful a metric as the number of cycles per IOP and the number of IOPS per OSD.
+For example, with NVMe OSD drives, Ceph can easily utilize five or six cores on real
+clusters and up to about fourteen cores on single OSDs in isolation. So cores
+per OSD are no longer as pressing a concern as they were. When selecting
+hardware, select for IOPS per core.
+
+.. tip:: When we speak of CPU _cores_, we mean _threads_ when hyperthreading
+	 is enabled.  Hyperthreading is usually beneficial for Ceph servers.
+
+Monitor nodes and Manager nodes do not have heavy CPU demands and require only
+modest processors. if your hosts will run CPU-intensive processes in
+addition to Ceph daemons, make sure that you have enough processing power to
+run both the CPU-intensive processes and the Ceph daemons. (OpenStack Nova is
+one example of a CPU-intensive process.) We recommend that you run
+non-Ceph CPU-intensive processes on separate hosts (that is, on hosts that are
+not your Monitor and Manager nodes) in order to avoid resource contention.
+If your cluster deployes the Ceph Object Gateway, RGW daemons may co-reside
+with your Mon and Manager services if the nodes have sufficient resources.
+
+RAM
+===
+
+Generally, more RAM is better.  Monitor / Manager nodes for a modest cluster
+might do fine with 64GB; for a larger cluster with hundreds of OSDs 128GB
+is advised.
+
+.. tip:: when we speak of RAM and storage requirements, we often describe
+	 the needs of a single daemon of a given type.  A given server as
+	 a whole will thus need at least the sum of the needs of the
+	 daemons that it hosts as well as resources for logs and other operating
+	 system components.  Keep in mind that a server's need for RAM
+	 and storage will be greater at startup and when components
+	 fail or are added and the cluster rebalances.  In other words,
+	 allow headroom past what you might see used during a calm period
+	 on a small initial cluster footprint.
+
+There is an :confval:`osd_memory_target` setting for BlueStore OSDs that
+defaults to 4GB.  Factor in a prudent margin for the operating system and
+administrative tasks (like monitoring and metrics) as well as increased
+consumption during recovery:  provisioning ~8GB *per BlueStore OSD* is thus
+advised.
+
+Monitors and managers (ceph-mon and ceph-mgr)
+---------------------------------------------
+
+Monitor and manager daemon memory usage scales with the size of the
+cluster.  Note that at boot-time and during topology changes and recovery these
+daemons will need more RAM than they do during steady-state operation, so plan
+for peak usage. For very small clusters, 32 GB suffices. For clusters of up to,
+say, 300 OSDs go with 64GB. For clusters built with (or which will grow to)
+even more OSDs you should provision 128GB. You may also want to consider
+tuning the following settings:
+
+* :confval:`mon_osd_cache_size`
+* :confval:`rocksdb_cache_size`
+
+
+Metadata servers (ceph-mds)
+---------------------------
+
+CephFS metadata daemon memory utilization depends on the configured size of
+its cache. We recommend 1 GB as a minimum for most systems.  See
+:confval:`mds_cache_memory_limit`.
+
+
+Memory
+======
+
+Bluestore uses its own memory to cache data rather than relying on the
+operating system's page cache. In Bluestore you can adjust the amount of memory
+that the OSD attempts to consume by changing the :confval:`osd_memory_target`
+configuration option.
+
+- Setting the :confval:`osd_memory_target` below 2GB is not
+  recommended. Ceph may fail to keep the memory consumption under 2GB and 
+  extremely slow performance is likely.
+
+- Setting the memory target between 2GB and 4GB typically works but may result
+  in degraded performance: metadata may need to be read from disk during IO
+  unless the active data set is relatively small.
+
+- 4GB is the current default value for :confval:`osd_memory_target` This default
+  was chosen for typical use cases, and is intended to balance RAM cost and
+  OSD performance.
+
+- Setting the :confval:`osd_memory_target` higher than 4GB can improve
+  performance when there many (small) objects or when large (256GB/OSD 
+  or more) data sets are processed.  This is especially true with fast
+  NVMe OSDs.
+
+.. important:: OSD memory management is "best effort". Although the OSD may
+   unmap memory to allow the kernel to reclaim it, there is no guarantee that
+   the kernel will actually reclaim freed memory within a specific time
+   frame. This applies especially in older versions of Ceph, where transparent
+   huge pages can prevent the kernel from reclaiming memory that was freed from
+   fragmented huge pages. Modern versions of Ceph disable transparent huge
+   pages at the application level to avoid this, but that does not
+   guarantee that the kernel will immediately reclaim unmapped memory. The OSD
+   may still at times exceed its memory target. We recommend budgeting 
+   at least 20% extra memory on your system to prevent OSDs from going OOM
+   (**O**\ut **O**\f **M**\emory) during temporary spikes or due to delay in
+   the kernel reclaiming freed pages. That 20% value might be more or less than
+   needed, depending on the exact configuration of the system.
+
+.. tip:: Configuring the operating system with swap to provide additional
+	 virtual memory for daemons is not advised for modern systems.  Doing
+	 may result in lower performance, and your Ceph cluster may well be
+	 happier with a daemon that crashes vs one that slows to a crawl.
+
+When using the legacy FileStore back end, the OS page cache was used for caching
+data, so tuning was not normally needed. When using the legacy FileStore backend,
+the OSD memory consumption was related to the number of PGs per daemon in the
+system.
+
+
+Data Storage
+============
+
+Plan your data storage configuration carefully. There are significant cost and
+performance tradeoffs to consider when planning for data storage. Simultaneous
+OS operations and simultaneous requests from multiple daemons for read and
+write operations against a single drive can impact performance.
+
+OSDs require substantial storage drive space for RADOS data. We recommend a
+minimum drive size of 1 terabyte. OSD drives much smaller than one terabyte
+use a significant fraction of their capacity for metadata, and drives smaller
+than 100 gigabytes will not be effective at all.
+
+It is *strongly* suggested that (enterprise-class) SSDs are provisioned for, at a
+minimum, Ceph Monitor and Ceph Manager hosts, as well as CephFS Metadata Server
+metadata pools and Ceph Object Gateway (RGW) index pools, even if HDDs are to
+be provisioned for bulk OSD data.
+
+To get the best performance out of Ceph, provision the following on separate
+drives:
+
+* The operating systems
+* OSD data
+* BlueStore WAL+DB
+
+For more
+information on how to effectively use a mix of fast drives and slow drives in
+your Ceph cluster, see the `block and block.db`_ section of the Bluestore
+Configuration Reference.
+
+Hard Disk Drives
+----------------
+
+Consider carefully the cost-per-gigabyte advantage
+of larger disks. We recommend dividing the price of the disk drive by the
+number of gigabytes to arrive at a cost per gigabyte, because larger drives may
+have a significant impact on the cost-per-gigabyte. For example, a 1 terabyte
+hard disk priced at $75.00 has a cost of $0.07 per gigabyte (i.e., $75 / 1024 =
+0.0732). By contrast, a 3 terabyte disk priced at $150.00 has a cost of $0.05
+per gigabyte (i.e., $150 / 3072 = 0.0488). In the foregoing example, using the
+1 terabyte disks would generally increase the cost per gigabyte by
+40%--rendering your cluster substantially less cost efficient.
+
+.. tip:: Hosting multiple OSDs on a single SAS / SATA HDD
+   is **NOT** a good idea.
+
+.. tip:: Hosting an OSD with monitor, manager, or MDS data on a single 
+   drive is also **NOT** a good idea.
+
+.. tip:: With spinning disks, the SATA and SAS interface increasingly
+   becomes a bottleneck at larger capacities. See also the `Storage Networking 
+   Industry Association's Total Cost of Ownership calculator`_.
+
+
+Storage drives are subject to limitations on seek time, access time, read and
+write times, as well as total throughput. These physical limitations affect
+overall system performance--especially during recovery. We recommend using a
+dedicated (ideally mirrored) drive for the operating system and software, and
+one drive for each Ceph OSD Daemon you run on the host.
+Many "slow OSD" issues (when they are not attributable to hardware failure)
+arise from running an operating system and multiple OSDs on the same drive.
+Also be aware that today's 22TB HDD uses the same SATA interface as a
+3TB HDD from ten years ago: more than seven times the data to squeeze
+through the same same interface.  For this reason, when using HDDs for
+OSDs, drives larger than 8TB may be best suited for storage of large
+files / objects that are not at all performance-sensitive.
+
+
+Solid State Drives
+------------------
+
+Ceph performance is much improved when using solid-state drives (SSDs). This
+reduces random access time and reduces latency while increasing throughput. 
+
+SSDs cost more per gigabyte than do HDDs but SSDs often offer
+access times that are, at a minimum, 100 times faster than HDDs.
+SSDs avoid hotspot issues and bottleneck issues within busy clusters, and
+they may offer better economics when TCO is evaluated holistically. Notably,
+the amortized drive cost for a given number of IOPS is much lower with SSDs
+than with HDDs.  SSDs do not suffer rotational or seek latency and in addition
+to improved client performance, they substantially improve the speed and
+client impact of cluster changes including rebalancing when OSDs or Monitors
+are added, removed, or fail.
+
+SSDs do not have moving mechanical parts, so they are not subject
+to many of the limitations of HDDs.  SSDs do have significant
+limitations though. When evaluating SSDs, it is important to consider the
+performance of sequential and random reads and writes.
+
+.. important:: We recommend exploring the use of SSDs to improve performance. 
+   However, before making a significant investment in SSDs, we **strongly
+   recommend** reviewing the performance metrics of an SSD and testing the
+   SSD in a test configuration in order to gauge performance. 
+
+Relatively inexpensive SSDs may appeal to your sense of economy. Use caution.
+Acceptable IOPS are not the only factor to consider when selecting SSDs for
+use with Ceph. Bargain SSDs are often a false economy: they may experience
+"cliffing", which means that after an initial burst, sustained performance
+once a limited cache is filled declines considerably.  Consider also durability:
+a drive rated for 0.3 Drive Writes Per Day (DWPD or equivalent) may be fine for
+OSDs dedicated to certain types of sequentially-written read-mostly data, but
+are not a good choice for Ceph Monitor duty.  Enterprise-class SSDs are best
+for Ceph:  they almost always feature power less protection (PLP) and do
+not suffer the dramatic cliffing that client (desktop) models may experience.
+
+When using a single (or mirrored pair) SSD for both operating system boot
+and Ceph Monitor / Manager purposes, a minimum capacity of 256GB is advised
+and at least 480GB is recommended. A drive model rated at 1+ DWPD (or the
+equivalent in TBW (TeraBytes Written) is suggested.  However, for a given write
+workload, a larger drive than technically required will provide more endurance
+because it effectively has greater overprovsioning. We stress that
+enterprise-class drives are best for production use, as they feature power
+loss protection and increased durability compared to client (desktop) SKUs
+that are intended for much lighter and intermittent duty cycles.
+
+SSDs were historically been cost prohibitive for object storage, but
+QLC SSDs are closing the gap, offering greater density with lower power
+consumption and less power spent on cooling. Also, HDD OSDs may see a
+significant write latency improvement by offloading WAL+DB onto an SSD.
+Many Ceph OSD deployments do not require an SSD with greater endurance than
+1 DWPD (aka "read-optimized").  "Mixed-use" SSDs in the 3 DWPD class are
+often overkill for this purpose and cost signficantly more.
+
+To get a better sense of the factors that determine the total cost of storage,
+you might use the `Storage Networking Industry Association's Total Cost of
+Ownership calculator`_
+
+Partition Alignment
+~~~~~~~~~~~~~~~~~~~
+
+When using SSDs with Ceph, make sure that your partitions are properly aligned.
+Improperly aligned partitions suffer slower data transfer speeds than do
+properly aligned partitions. For more information about proper partition
+alignment and example commands that show how to align partitions properly, see
+`Werner Fischer's blog post on partition alignment`_.
+
+CephFS Metadata Segregation
+~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+One way that Ceph accelerates CephFS file system performance is by separating
+the storage of CephFS metadata from the storage of the CephFS file contents.
+Ceph provides a default ``metadata`` pool for CephFS metadata. You will never
+have to manually create a pool for CephFS metadata, but you can create a CRUSH map
+hierarchy for your CephFS metadata pool that includes only SSD storage media.
+See :ref:`CRUSH Device Class<crush-map-device-class>` for details.
+
+
+Controllers
+-----------
+
+Disk controllers (HBAs) can have a significant impact on write throughput.
+Carefully consider your selection of HBAs to ensure that they do not create a
+performance bottleneck. Notably, RAID-mode (IR) HBAs may exhibit higher latency
+than simpler "JBOD" (IT) mode HBAs. The RAID SoC, write cache, and battery
+backup can substantially increase hardware and maintenance costs. Many RAID
+HBAs can be configured with an IT-mode "personality" or "JBOD mode" for
+streamlined operation.
+
+You do not need an RoC (RAID-capable) HBA. ZFS or Linux MD software mirroring
+serve well for boot volume durability.  When using SAS or SATA data drives,
+forgoing HBA RAID capabilities can reduce the gap between HDD and SSD
+media cost.  Moreover, when using NVMe SSDs, you do not need *any* HBA.  This
+additionally reduces the HDD vs SSD cost gap when the system as a whole is
+considered. The initial cost of a fancy RAID HBA plus onboard cache plus
+battery backup (BBU or supercapacitor) can easily exceed more than 1000 US
+dollars even after discounts - a sum that goes a log way toward SSD cost parity.
+An HBA-free system may also cost hundreds of US dollars less every year if one
+purchases an annual maintenance contract or extended warranty.
+
+.. tip:: The `Ceph blog`_ is often an excellent source of information on Ceph
+   performance issues. See `Ceph Write Throughput 1`_ and `Ceph Write 
+   Throughput 2`_ for additional details.
+
+
+Benchmarking
+------------
+
+BlueStore opens storage devices with ``O_DIRECT`` and issues ``fsync()``
+frequently to ensure that data is safely persisted to media. You can evaluate a
+drive's low-level write performance using ``fio``. For example, 4kB random write
+performance is measured as follows:
+
+.. code-block:: console
+
+  # fio --name=/dev/sdX --ioengine=libaio --direct=1 --fsync=1 --readwrite=randwrite --blocksize=4k --runtime=300
+
+Write Caches
+------------
+
+Enterprise SSDs and HDDs normally include power loss protection features which
+ensure data durability when power is lost while operating, and
+use multi-level caches to speed up direct or synchronous writes.  These devices
+can be toggled between two caching modes -- a volatile cache flushed to
+persistent media with fsync, or a non-volatile cache written synchronously.
+
+These two modes are selected by either "enabling" or "disabling" the write
+(volatile) cache.  When the volatile cache is enabled, Linux uses a device in
+"write back" mode, and when disabled, it uses "write through".
+
+The default configuration (usually: caching is enabled) may not be optimal, and
+OSD performance may be dramatically increased in terms of increased IOPS and
+decreased commit latency by disabling this write cache.
+
+Users are therefore encouraged to benchmark their devices with ``fio`` as
+described earlier and persist the optimal cache configuration for their
+devices.
+
+The cache configuration can be queried with ``hdparm``, ``sdparm``,
+``smartctl`` or by reading the values in ``/sys/class/scsi_disk/*/cache_type``,
+for example:
+
+.. code-block:: console
+
+  # hdparm -W /dev/sda
+
+  /dev/sda:
+   write-caching =  1 (on)
+
+  # sdparm --get WCE /dev/sda
+      /dev/sda: ATA       TOSHIBA MG07ACA1  0101
+  WCE           1  [cha: y]
+  # smartctl -g wcache /dev/sda
+  smartctl 7.1 2020-04-05 r5049 [x86_64-linux-4.18.0-305.19.1.el8_4.x86_64] (local build)
+  Copyright (C) 2002-19, Bruce Allen, Christian Franke, www.smartmontools.org
+
+  Write cache is:   Enabled
+
+  # cat /sys/class/scsi_disk/0\:0\:0\:0/cache_type
+  write back
+
+The write cache can be disabled with those same tools:
+
+.. code-block:: console
+
+  # hdparm -W0 /dev/sda
+
+  /dev/sda:
+   setting drive write-caching to 0 (off)
+   write-caching =  0 (off)
+
+  # sdparm --clear WCE /dev/sda
+      /dev/sda: ATA       TOSHIBA MG07ACA1  0101
+  # smartctl -s wcache,off /dev/sda
+  smartctl 7.1 2020-04-05 r5049 [x86_64-linux-4.18.0-305.19.1.el8_4.x86_64] (local build)
+  Copyright (C) 2002-19, Bruce Allen, Christian Franke, www.smartmontools.org
+
+  === START OF ENABLE/DISABLE COMMANDS SECTION ===
+  Write cache disabled
+
+In most cases, disabling this cache  using ``hdparm``, ``sdparm``, or ``smartctl``
+results in the cache_type changing automatically to "write through". If this is
+not the case, you can try setting it directly as follows. (Users should ensure
+that setting cache_type also correctly persists the caching mode of the device
+until the next reboot as some drives require this to be repeated at every boot):
+
+.. code-block:: console
+
+  # echo "write through" > /sys/class/scsi_disk/0\:0\:0\:0/cache_type
+
+  # hdparm -W /dev/sda
+
+  /dev/sda:
+   write-caching =  0 (off)
+
+.. tip:: This udev rule (tested on CentOS 8) will set all SATA/SAS device cache_types to "write
+  through":
+
+  .. code-block:: console
+
+    # cat /etc/udev/rules.d/99-ceph-write-through.rules
+    ACTION=="add", SUBSYSTEM=="scsi_disk", ATTR{cache_type}:="write through"
+
+.. tip:: This udev rule (tested on CentOS 7) will set all SATA/SAS device cache_types to "write
+  through":
+
+  .. code-block:: console
+
+    # cat /etc/udev/rules.d/99-ceph-write-through-el7.rules
+    ACTION=="add", SUBSYSTEM=="scsi_disk", RUN+="/bin/sh -c 'echo write through > /sys/class/scsi_disk/$kernel/cache_type'"
+
+.. tip:: The ``sdparm`` utility can be used to view/change the volatile write
+  cache on several devices at once:
+
+  .. code-block:: console
+
+    # sdparm --get WCE /dev/sd*
+        /dev/sda: ATA       TOSHIBA MG07ACA1  0101
+    WCE           0  [cha: y]
+        /dev/sdb: ATA       TOSHIBA MG07ACA1  0101
+    WCE           0  [cha: y]
+    # sdparm --clear WCE /dev/sd*
+        /dev/sda: ATA       TOSHIBA MG07ACA1  0101
+        /dev/sdb: ATA       TOSHIBA MG07ACA1  0101
+
+Additional Considerations
+-------------------------
+
+Ceph operators typically provision  multiple OSDs per host, but you should
+ensure that the aggregate throughput of your OSD drives doesn't exceed the
+network bandwidth required to service a client's read and write operations.
+You should also each host's percentage of the cluster's overall capacity. If
+the percentage located on a particular host is large and the host fails, it
+can lead to problems such as recovery causing OSDs to exceed the ``full ratio``,
+which in turn causes Ceph to halt operations to prevent data loss.
+
+When you run multiple OSDs per host, you also need to ensure that the kernel
+is up to date. See `OS Recommendations`_ for notes on ``glibc`` and
+``syncfs(2)`` to ensure that your hardware performs as expected when running
+multiple OSDs per host.
+
+
+Networks
+========
+
+Provision at least 10 Gb/s networking in your datacenter, both among Ceph
+hosts and between clients and your Ceph cluster.  Network link active/active
+bonding across separate network switches is strongly recommended both for
+increased throughput and for tolerance of network failures and maintenance.
+Take care that your bonding hash policy distributes traffic across links.
+
+Speed
+-----
+
+It takes three hours to replicate 1 TB of data across a 1 Gb/s network and it
+takes thirty hours to replicate 10 TB across a 1 Gb/s network. But it takes only
+twenty minutes to replicate 1 TB across a 10 Gb/s network, and it takes
+only one hour to replicate 10 TB across a 10 Gb/s network.
+
+Note that a 40 Gb/s network link is effectively four 10 Gb/s channels in
+parallel, and that a 100Gb/s network link is effectively four 25 Gb/s channels
+in parallel.  Thus, and perhaps somewhat counterintuitively, an individual
+packet on a 25 Gb/s network has slightly lower latency compared to a 40 Gb/s
+network.
+
+
+Cost
+----
+
+The larger the Ceph cluster, the more common OSD failures will be.
+The faster that a placement group (PG) can recover from a degraded state to
+an ``active + clean`` state, the better. Notably, fast recovery minimizes
+the likelihood of multiple, overlapping failures that can cause data to become
+temporarily unavailable or even lost. Of course, when provisioning your
+network, you will have to balance price against performance. 
+
+Some deployment tools employ VLANs to make hardware and network cabling more
+manageable. VLANs that use the 802.1q protocol require VLAN-capable NICs and
+switches. The added expense of this hardware may be offset by the operational
+cost savings on network setup and maintenance. When using VLANs to handle VM
+traffic between the cluster and compute stacks (e.g., OpenStack, CloudStack,
+etc.), there is additional value in using 10 Gb/s Ethernet or better; 40 Gb/s or
+increasingly 25/50/100 Gb/s networking as of 2022 is common for production clusters.
+
+Top-of-rack (TOR) switches also need fast and redundant uplinks to
+core / spine network switches or routers, often at least 40 Gb/s.
+
+
+Baseboard Management Controller (BMC)
+-------------------------------------
+
+Your server chassis should have a Baseboard Management Controller (BMC).
+Well-known examples are iDRAC (Dell), CIMC (Cisco UCS), and iLO (HPE).
+Administration and deployment tools may also use BMCs extensively, especially
+via IPMI or Redfish, so consider the cost/benefit tradeoff of an out-of-band
+network for security and administration.  Hypervisor SSH access, VM image uploads,
+OS image installs, management sockets, etc. can impose significant loads on a network.
+Running multiple networks may seem like overkill, but each traffic path represents
+a potential capacity, throughput and/or performance bottleneck that you should
+carefully consider before deploying a large scale data cluster.
+
+Additionally BMCs as of 2023 rarely sport network connections faster than 1 Gb/s,
+so dedicated and inexpensive 1 Gb/s switches for BMC administrative traffic
+may reduce costs by wasting fewer expenive ports on faster host switches.
+ 
+
+Failure Domains
+===============
+
+A failure domain can be thought of as any component loss that prevents access to
+one or more OSDs or other Ceph daemons. These could be a stopped daemon on a host;
+a storage drive failure, an OS crash, a malfunctioning NIC, a failed power supply,
+a network outage, a power outage, and so forth. When planning your hardware
+deployment, you must balance the risk of reducing costs by placing too many
+responsibilities into too few failure domains against the added costs of
+isolating every potential failure domain.
+
+
+Minimum Hardware Recommendations
+================================
+
+Ceph can run on inexpensive commodity hardware. Small production clusters
+and development clusters can run successfully with modest hardware.  As
+we noted above: when we speak of CPU _cores_, we mean _threads_ when
+hyperthreading (HT) is enabled.  Each modern physical x64 CPU core typically
+provides two logical CPU threads; other CPU architectures may vary.
+
+Take care that there are many factors that influence resource choices.  The
+minimum resources that suffice for one purpose will not necessarily suffice for
+another.  A sandbox cluster with one OSD built on a laptop with VirtualBox or on
+a trio of Raspberry PIs will get by with fewer resources than a production
+deployment with a thousand OSDs serving five thousand of RBD clients.  The
+classic Fisher Price PXL 2000 captures video, as does an IMAX or RED camera.
+One would not expect the former to do the job of the latter.  We especially
+cannot stress enough the criticality of using enterprise-quality storage
+media for production workloads.
+
+Additional insights into resource planning for production clusters are
+found above and elsewhere within this documentation.
+
++--------------+----------------+-----------------------------------------+
+|  Process     | Criteria       | Bare Minimum and Recommended            |
++==============+================+=========================================+
+| ``ceph-osd`` | Processor      | - 1 core minimum, 2 recommended         |
+|              |                | - 1 core per 200-500 MB/s throughput    |
+|              |                | - 1 core per 1000-3000 IOPS             |
+|              |                |                                         |
+|              |                | * Results are before replication.       |
+|              |                | * Results may vary across CPU and drive |
+|              |                |   models and Ceph configuration:        |
+|              |                |   (erasure coding, compression, etc)    |
+|              |                | * ARM processors specifically may       |
+|              |                |   require more cores for performance.   |
+|              |                | * SSD OSDs, especially NVMe, will       |
+|              |                |   benefit from additional cores per OSD.|
+|              |                | * Actual performance depends on many    |
+|              |                |   factors including drives, net, and    |
+|              |                |   client throughput and latency.        |
+|              |                |   Benchmarking is highly recommended.   |
+|              +----------------+-----------------------------------------+
+|              | RAM            | - 4GB+ per daemon (more is better)      |
+|              |                | - 2-4GB may function but may be slow    |
+|              |                | - Less than 2GB is not recommended      |
+|              +----------------+-----------------------------------------+
+|              | Storage Drives |  1x storage drive per OSD               |
+|              +----------------+-----------------------------------------+
+|              | DB/WAL         |  1x SSD partion per HDD OSD             |
+|              | (optional)     |  4-5x HDD OSDs per DB/WAL SATA SSD      |
+|              |                |  <= 10 HDD OSDss per DB/WAL NVMe SSD    |
+|              +----------------+-----------------------------------------+
+|              | Network        |  1x 1Gb/s (bonded 10+ Gb/s recommended) |
++--------------+----------------+-----------------------------------------+
+| ``ceph-mon`` | Processor      | - 2 cores minimum                       |
+|              +----------------+-----------------------------------------+
+|              | RAM            |  5GB+ per daemon (large / production    |
+|              |                |  clusters need more)                    |
+|              +----------------+-----------------------------------------+
+|              | Storage        |  100 GB per daemon, SSD is recommended  |
+|              +----------------+-----------------------------------------+
+|              | Network        |  1x 1Gb/s (10+ Gb/s recommended)        |
++--------------+----------------+-----------------------------------------+
+| ``ceph-mds`` | Processor      | - 2 cores minimum                       |
+|              +----------------+-----------------------------------------+
+|              | RAM            |  2GB+ per daemon (more for production)  |
+|              +----------------+-----------------------------------------+
+|              | Disk Space     |  1 GB per daemon                        |
+|              +----------------+-----------------------------------------+
+|              | Network        |  1x 1Gb/s (10+ Gb/s recommended)        |
++--------------+----------------+-----------------------------------------+
+
+.. tip:: If you are running an OSD node with a single storage drive, create a
+   partition for your OSD that is separate from the partition
+   containing the OS. We recommend separate drives for the
+   OS and for OSD storage.
+
+
+
+.. _block and block.db: https://docs.ceph.com/en/latest/rados/configuration/bluestore-config-ref/#block-and-block-db
+.. _Ceph blog: https://ceph.com/community/blog/
+.. _Ceph Write Throughput 1: http://ceph.com/community/ceph-performance-part-1-disk-controller-write-throughput/
+.. _Ceph Write Throughput 2: http://ceph.com/community/ceph-performance-part-2-write-throughput-without-ssd-journals/
+.. _Mapping Pools to Different Types of OSDs: ../../rados/operations/crush-map#placing-different-pools-on-different-osds
+.. _OS Recommendations: ../os-recommendations
+.. _Storage Networking Industry Association's Total Cost of Ownership calculator: https://www.snia.org/forums/cmsi/programs/TCOcalc
+.. _Werner Fischer's blog post on partition alignment: https://www.thomas-krenn.com/en/wiki/Partition_Alignment_detailed_explanation
diff --git a/doc/start/intro.rst b/doc/start/intro.rst
new file mode 100644
index 000000000..3a50a8733
--- /dev/null
+++ b/doc/start/intro.rst
@@ -0,0 +1,99 @@
+===============
+ Intro to Ceph
+===============
+
+Ceph can be used to provide :term:`Ceph Object Storage` to :term:`Cloud
+Platforms` and Ceph can be used to provide :term:`Ceph Block Device` services
+to :term:`Cloud Platforms`. Ceph can be used to deploy a :term:`Ceph File
+System`.  All :term:`Ceph Storage Cluster` deployments begin with setting up
+each :term:`Ceph Node` and then setting up the network. 
+
+A Ceph Storage Cluster requires the following: at least one Ceph Monitor and at
+least one Ceph Manager, and at least as many Ceph OSDs as there are copies of
+an object stored on the Ceph cluster (for example, if three copies of a given
+object are stored on the Ceph cluster, then at least three OSDs must exist in
+that Ceph cluster). 
+
+The Ceph Metadata Server is necessary to run Ceph File System clients.
+
+.. note::
+
+   It is a best practice to have a Ceph Manager for each Monitor, but it is not
+   necessary. 
+
+.. ditaa::
+
+            +---------------+ +------------+ +------------+ +---------------+
+            |      OSDs     | | Monitors   | |  Managers  | |      MDSs     |
+            +---------------+ +------------+ +------------+ +---------------+
+
+- **Monitors**: A :term:`Ceph Monitor` (``ceph-mon``) maintains maps
+  of the cluster state, including the monitor map, manager map, the
+  OSD map, the MDS map, and the CRUSH map.  These maps are critical 
+  cluster state required for Ceph daemons to coordinate with each other.  
+  Monitors are also responsible for managing authentication between 
+  daemons and clients.  At least three monitors are normally required 
+  for redundancy and high availability.
+
+- **Managers**: A :term:`Ceph Manager` daemon (``ceph-mgr``) is
+  responsible for keeping track of runtime metrics and the current
+  state of the Ceph cluster, including storage utilization, current
+  performance metrics, and system load.  The Ceph Manager daemons also
+  host python-based modules to manage and expose Ceph cluster
+  information, including a web-based :ref:`mgr-dashboard` and
+  `REST API`_.  At least two managers are normally required for high
+  availability.
+
+- **Ceph OSDs**: An Object Storage Daemon (:term:`Ceph OSD`,
+  ``ceph-osd``) stores data, handles data replication, recovery,
+  rebalancing, and provides some monitoring information to Ceph
+  Monitors and Managers by checking other Ceph OSD Daemons for a
+  heartbeat. At least three Ceph OSDs are normally required for 
+  redundancy and high availability.
+
+- **MDSs**: A :term:`Ceph Metadata Server` (MDS, ``ceph-mds``) stores
+  metadata on behalf of the :term:`Ceph File System` (i.e., Ceph Block
+  Devices and Ceph Object Storage do not use MDS). Ceph Metadata
+  Servers allow POSIX file system users to execute basic commands (like
+  ``ls``, ``find``, etc.) without placing an enormous burden on the
+  Ceph Storage Cluster.
+
+Ceph stores data as objects within logical storage pools. Using the
+:term:`CRUSH` algorithm, Ceph calculates which placement group (PG) should
+contain the object, and which OSD should store the placement group.  The
+CRUSH algorithm enables the Ceph Storage Cluster to scale, rebalance, and
+recover dynamically.
+
+.. _REST API: ../../mgr/restful
+
+.. container:: columns-2
+
+   .. container:: column
+
+      .. raw:: html
+
+          <h3>Recommendations</h3>
+
+      To begin using Ceph in production, you should review our hardware
+      recommendations and operating system recommendations.
+
+      .. toctree::
+         :maxdepth: 2
+
+         Hardware Recommendations <hardware-recommendations>
+         OS Recommendations <os-recommendations>
+
+   .. container:: column
+
+      .. raw:: html
+
+          <h3>Get Involved</h3>
+
+      You can avail yourself of help or contribute documentation, source
+      code or bugs by getting involved in the Ceph community.
+
+      .. toctree::
+         :maxdepth: 2
+
+         get-involved
+         documenting-ceph
diff --git a/doc/start/os-recommendations.rst b/doc/start/os-recommendations.rst
new file mode 100644
index 000000000..81906569e
--- /dev/null
+++ b/doc/start/os-recommendations.rst
@@ -0,0 +1,82 @@
+====================
+ OS Recommendations
+====================
+
+Ceph Dependencies
+=================
+
+As a general rule, we recommend deploying Ceph on newer releases of Linux. 
+We also recommend deploying on releases with long-term support.
+
+Linux Kernel
+------------
+
+- **Ceph Kernel Client**
+
+  If you are using the kernel client to map RBD block devices or mount
+  CephFS, the general advice is to use a "stable" or "longterm
+  maintenance" kernel series provided by either http://kernel.org or
+  your Linux distribution on any client hosts.
+
+  For RBD, if you choose to *track* long-term kernels, we recommend
+  *at least* 4.19-based "longterm maintenance" kernel series.  If you can
+  use a newer "stable" or "longterm maintenance" kernel series, do it.
+
+  For CephFS, see the section about `Mounting CephFS using Kernel Driver`_
+  for kernel version guidance.
+
+  Older kernel client versions may not support your `CRUSH tunables`_ profile
+  or other newer features of the Ceph cluster, requiring the storage cluster to
+  be configured with those features disabled. For RBD, a kernel of version 5.3
+  or CentOS 8.2 is the minimum necessary for reasonable support for RBD image
+  features.
+
+
+Platforms
+=========
+
+The chart below shows which Linux platforms Ceph provides packages for, and
+which platforms Ceph has been tested on. 
+
+Ceph does not require a specific Linux distribution. Ceph can run on any
+distribution that includes a supported kernel and supported system startup
+framework, for example ``sysvinit`` or ``systemd``. Ceph is sometimes ported to
+non-Linux systems but these are not supported by the core Ceph effort.
+
+
++---------------+---------------+-----------------+------------------+------------------+
+|               | Reef (18.2.z) | Quincy (17.2.z) | Pacific (16.2.z) | Octopus (15.2.z) |
++===============+===============+=================+==================+==================+
+| Centos 7      |               |                 |         A        |      B           |
++---------------+---------------+-----------------+------------------+------------------+
+| Centos 8      |    A          |     A           |         A        |      A           |
++---------------+---------------+-----------------+------------------+------------------+
+| Centos 9      |    A          |                 |                  |                  |
++---------------+---------------+-----------------+------------------+------------------+
+| Debian 10     |    C          |                 |         C        |      C           |
++---------------+---------------+-----------------+------------------+------------------+
+| Debian 11     |    C          |     C           |         C        |                  |
++---------------+---------------+-----------------+------------------+------------------+
+| OpenSUSE 15.2 |    C          |                 |         C        |      C           |
++---------------+---------------+-----------------+------------------+------------------+
+| OpenSUSE 15.3 |    C          |     C           |                  |                  |
++---------------+---------------+-----------------+------------------+------------------+
+| Ubuntu 18.04  |               |                 |         C        |      C           |
++---------------+---------------+-----------------+------------------+------------------+
+| Ubuntu 20.04  |    A          |     A           |         A        |      A           |
++---------------+---------------+-----------------+------------------+------------------+
+| Ubuntu 22.04  |    A          |                 |                  |                  |
++---------------+---------------+-----------------+------------------+------------------+
+
+- **A**: Ceph provides packages and has done comprehensive tests on the software in them.
+- **B**: Ceph provides packages and has done basic tests on the software in them.
+- **C**: Ceph provides packages only. No tests have been done on these releases.
+
+.. note::
+   **For Centos 7 Users** 
+   
+   ``Btrfs`` is no longer tested on Centos 7 in the Octopus release. We recommend using ``bluestore`` instead.
+
+.. _CRUSH Tunables: ../../rados/operations/crush-map#tunables
+
+.. _Mounting CephFS using Kernel Driver: ../../cephfs/mount-using-kernel-driver#which-kernel-version
diff --git a/doc/start/quick-rbd.rst b/doc/start/quick-rbd.rst
new file mode 100644
index 000000000..c1cf77098
--- /dev/null
+++ b/doc/start/quick-rbd.rst
@@ -0,0 +1,69 @@
+==========================
+ Block Device Quick Start
+==========================
+
+Ensure your :term:`Ceph Storage Cluster` is in an ``active + clean`` state
+before working with the :term:`Ceph Block Device`.
+
+.. note:: The Ceph Block Device is also known as :term:`RBD` or :term:`RADOS`
+   Block Device.
+
+
+.. ditaa::
+
+           /------------------\         /----------------\
+           |    Admin Node    |         |   ceph-client  |
+           |                  +-------->+ cCCC           |
+           |    ceph-deploy   |         |      ceph      |
+           \------------------/         \----------------/
+
+
+You may use a virtual machine for your ``ceph-client`` node, but do not 
+execute the following procedures on the same physical node as your Ceph 
+Storage Cluster nodes (unless you use a VM). See `FAQ`_ for details.
+
+Create a Block Device Pool
+==========================
+
+#. On the admin node, use the ``ceph`` tool to `create a pool`_
+   (we recommend the name 'rbd').
+
+#. On the admin node, use the ``rbd`` tool to initialize the pool for use by RBD::
+
+        rbd pool init <pool-name>
+
+Configure a Block Device
+========================
+
+#. On the ``ceph-client`` node, create a block device image. :: 
+
+	rbd create foo --size 4096 --image-feature layering [-m {mon-IP}] [-k /path/to/ceph.client.admin.keyring] [-p {pool-name}]
+
+#. On the ``ceph-client`` node, map the image to a block device. :: 
+
+	sudo rbd map foo --name client.admin [-m {mon-IP}] [-k /path/to/ceph.client.admin.keyring] [-p {pool-name}]
+	
+#. Use the block device by creating a file system on the ``ceph-client`` 
+   node. :: 
+
+	sudo mkfs.ext4 -m0 /dev/rbd/{pool-name}/foo
+	
+	This may take a few moments.
+	
+#. Mount the file system on the ``ceph-client`` node. ::
+
+	sudo mkdir /mnt/ceph-block-device
+	sudo mount /dev/rbd/{pool-name}/foo /mnt/ceph-block-device
+	cd /mnt/ceph-block-device
+
+#. Optionally configure the block device to be automatically mapped and mounted
+   at boot (and unmounted/unmapped at shutdown) - see the `rbdmap manpage`_.
+
+
+See `block devices`_ for additional details.
+
+.. _create a pool: ../../rados/operations/pools/#create-a-pool
+.. _block devices: ../../rbd
+.. _FAQ: http://wiki.ceph.com/How_Can_I_Give_Ceph_a_Try
+.. _OS Recommendations: ../os-recommendations
+.. _rbdmap manpage: ../../man/8/rbdmap