1 files changed, 237 insertions, 0 deletions
diff --git a/doc/cephadm/services/mon.rst b/doc/cephadm/services/mon.rst
new file mode 100644
index 000000000..389dc450e
--- /dev/null
+++ b/doc/cephadm/services/mon.rst
@@ -0,0 +1,237 @@
+===========
+MON Service
+===========
+
+.. _deploy_additional_monitors:
+
+Deploying additional monitors 
+=============================
+
+A typical Ceph cluster has three or five monitor daemons that are spread
+across different hosts.  We recommend deploying five monitors if there are
+five or more nodes in your cluster.
+
+.. _CIDR: https://en.wikipedia.org/wiki/Classless_Inter-Domain_Routing#CIDR_notation
+
+Ceph deploys monitor daemons automatically as the cluster grows and Ceph
+scales back monitor daemons automatically as the cluster shrinks. The
+smooth execution of this automatic growing and shrinking depends upon
+proper subnet configuration.
+
+The cephadm bootstrap procedure assigns the first monitor daemon in the
+cluster to a particular subnet. ``cephadm`` designates that subnet as the
+default subnet of the cluster. New monitor daemons will be assigned by
+default to that subnet unless cephadm is instructed to do otherwise. 
+
+If all of the ceph monitor daemons in your cluster are in the same subnet,
+manual administration of the ceph monitor daemons is not necessary.
+``cephadm`` will automatically add up to five monitors to the subnet, as
+needed, as new hosts are added to the cluster.
+
+By default, cephadm will deploy 5 daemons on arbitrary hosts. See
+:ref:`orchestrator-cli-placement-spec` for details of specifying
+the placement of daemons.
+
+Designating a Particular Subnet for Monitors
+--------------------------------------------
+
+To designate a particular IP subnet for use by ceph monitor daemons, use a
+command of the following form, including the subnet's address in `CIDR`_
+format (e.g., ``10.1.2.0/24``):
+
+  .. prompt:: bash #
+
+     ceph config set mon public_network *<mon-cidr-network>*
+
+  For example:
+
+  .. prompt:: bash #
+
+     ceph config set mon public_network 10.1.2.0/24
+
+Cephadm deploys new monitor daemons only on hosts that have IP addresses in
+the designated subnet.
+
+You can also specify two public networks by using a list of networks:
+
+  .. prompt:: bash #
+
+     ceph config set mon public_network *<mon-cidr-network1>,<mon-cidr-network2>*
+
+  For example:
+
+  .. prompt:: bash #
+
+     ceph config set mon public_network 10.1.2.0/24,192.168.0.1/24
+
+
+Deploying Monitors on a Particular Network 
+------------------------------------------
+
+You can explicitly specify the IP address or CIDR network for each monitor and
+control where each monitor is placed.  To disable automated monitor deployment,
+run this command:
+
+  .. prompt:: bash #
+
+    ceph orch apply mon --unmanaged
+
+  To deploy each additional monitor:
+
+  .. prompt:: bash #
+
+    ceph orch daemon add mon *<host1:ip-or-network1>
+
+  For example, to deploy a second monitor on ``newhost1`` using an IP
+  address ``10.1.2.123`` and a third monitor on ``newhost2`` in
+  network ``10.1.2.0/24``, run the following commands:
+
+  .. prompt:: bash #
+
+    ceph orch apply mon --unmanaged
+    ceph orch daemon add mon newhost1:10.1.2.123
+    ceph orch daemon add mon newhost2:10.1.2.0/24
+
+  Now, enable automatic placement of Daemons
+
+  .. prompt:: bash #
+
+    ceph orch apply mon --placement="newhost1,newhost2,newhost3" --dry-run
+
+  See :ref:`orchestrator-cli-placement-spec` for details of specifying
+  the placement of daemons.
+
+  Finally apply this new placement by dropping ``--dry-run``
+
+  .. prompt:: bash #
+
+    ceph orch apply mon --placement="newhost1,newhost2,newhost3"
+
+
+Moving Monitors to a Different Network
+--------------------------------------
+
+To move Monitors to a new network, deploy new monitors on the new network and
+subsequently remove monitors from the old network. It is not advised to
+modify and inject the ``monmap`` manually.
+
+First, disable the automated placement of daemons:
+
+  .. prompt:: bash #
+
+    ceph orch apply mon --unmanaged
+
+To deploy each additional monitor:
+
+  .. prompt:: bash #
+
+    ceph orch daemon add mon *<newhost1:ip-or-network1>*
+
+For example, to deploy a second monitor on ``newhost1`` using an IP
+address ``10.1.2.123`` and a third monitor on ``newhost2`` in
+network ``10.1.2.0/24``, run the following commands:
+
+  .. prompt:: bash #
+
+    ceph orch apply mon --unmanaged
+    ceph orch daemon add mon newhost1:10.1.2.123
+    ceph orch daemon add mon newhost2:10.1.2.0/24
+
+  Subsequently remove monitors from the old network:
+
+  .. prompt:: bash #
+
+    ceph orch daemon rm *mon.<oldhost1>*
+
+  Update the ``public_network``:
+
+  .. prompt:: bash #
+
+     ceph config set mon public_network *<mon-cidr-network>*
+
+  For example:
+
+  .. prompt:: bash #
+
+     ceph config set mon public_network 10.1.2.0/24
+
+  Now, enable automatic placement of Daemons
+
+  .. prompt:: bash #
+
+    ceph orch apply mon --placement="newhost1,newhost2,newhost3" --dry-run
+
+  See :ref:`orchestrator-cli-placement-spec` for details of specifying
+  the placement of daemons.
+
+  Finally apply this new placement by dropping ``--dry-run``
+
+  .. prompt:: bash #
+
+    ceph orch apply mon --placement="newhost1,newhost2,newhost3" 
+
+
+Setting Crush Locations for Monitors
+------------------------------------
+
+Cephadm supports setting CRUSH locations for mon daemons
+using the mon service spec. The CRUSH locations are set
+by hostname. When cephadm deploys a mon on a host that matches
+a hostname specified in the CRUSH locations, it will add
+``--set-crush-location <CRUSH-location>`` where the CRUSH location
+is the first entry in the list of CRUSH locations for that
+host. If multiple CRUSH locations are set for one host, cephadm
+will attempt to set the additional locations using the
+"ceph mon set_location" command.
+
+.. note::
+
+   Setting the CRUSH location in the spec is the recommended way of
+   replacing tiebreaker mon daemons, as they require having a location
+   set when they are added.
+
+ .. note::
+
+   Tiebreaker mon daemons are a part of stretch mode clusters. For more
+   info on stretch mode clusters see :ref:`stretch_mode`
+
+Example syntax for setting the CRUSH locations:
+
+.. code-block:: yaml
+
+    service_type: mon
+    service_name: mon
+    placement:
+      count: 5
+    spec:
+      crush_locations:
+        host1:
+        - datacenter=a
+        host2:
+        - datacenter=b
+        - rack=2
+        host3:
+        - datacenter=a
+
+.. note::
+
+   Sometimes, based on the timing of mon daemons being admitted to the mon
+   quorum, cephadm may fail to set the CRUSH location for some mon daemons
+   when multiple locations are specified. In this case, the recommended
+   action is to re-apply the same mon spec to retrigger the service action.
+
+.. note::
+
+   Mon daemons will only get the ``--set-crush-location`` flag set when cephadm
+   actually deploys them. This means if a spec is applied that includes a CRUSH
+   location for a mon that is already deployed, the flag may not be set until
+   a redeploy command is issued for that mon daemon.
+
+
+Further Reading
+===============
+
+* :ref:`rados-operations`
+* :ref:`rados-troubleshooting-mon`
+* :ref:`cephadm-restore-quorum`
+