summaryrefslogtreecommitdiffstats
path: root/doc
diff options
context:
space:
mode:
authorDaniel Baumann <daniel.baumann@progress-linux.org>2024-04-17 06:48:59 +0000
committerDaniel Baumann <daniel.baumann@progress-linux.org>2024-04-17 06:48:59 +0000
commitd835b2cae8abc71958b69362162e6a70c3d7ef63 (patch)
tree81052e3d2ce3e1bcda085f73d925e9d6257dec15 /doc
parentInitial commit. (diff)
downloadcrmsh-d835b2cae8abc71958b69362162e6a70c3d7ef63.tar.xz
crmsh-d835b2cae8abc71958b69362162e6a70c3d7ef63.zip
Adding upstream version 4.6.0.upstream/4.6.0upstream
Signed-off-by: Daniel Baumann <daniel.baumann@progress-linux.org>
Diffstat (limited to '')
-rw-r--r--doc/bootstrap-howto.md206
-rw-r--r--doc/bootstrap-todo.md56
-rw-r--r--doc/crm.8.adoc5102
-rw-r--r--doc/crmsh_crm_report.8.adoc15
-rw-r--r--doc/development.md314
-rw-r--r--doc/profiles.adoc47
-rw-r--r--doc/releasing-a-new-version.md199
-rw-r--r--doc/sort-doc.py82
-rw-r--r--doc/website-v1/404.adoc9
-rw-r--r--doc/website-v1/Makefile145
-rw-r--r--doc/website-v1/about.adoc19
-rw-r--r--doc/website-v1/configuration.adoc132
-rw-r--r--doc/website-v1/crm.conf601
-rw-r--r--doc/website-v1/crmold.conf602
-rw-r--r--doc/website-v1/css/crm.css570
-rw-r--r--doc/website-v1/css/font-awesome.css1338
-rw-r--r--doc/website-v1/css/font-awesome.min.css4
-rw-r--r--doc/website-v1/development.adoc74
-rw-r--r--doc/website-v1/documentation.adoc42
-rw-r--r--doc/website-v1/download.adoc40
-rw-r--r--doc/website-v1/faq.adoc60
-rw-r--r--doc/website-v1/fonts/FontAwesome.otfbin0 -> 62856 bytes
-rwxr-xr-xdoc/website-v1/fonts/fontawesome-webfont.eotbin0 -> 38205 bytes
-rwxr-xr-xdoc/website-v1/fonts/fontawesome-webfont.svg414
-rwxr-xr-xdoc/website-v1/fonts/fontawesome-webfont.ttfbin0 -> 80652 bytes
-rwxr-xr-xdoc/website-v1/fonts/fontawesome-webfont.woffbin0 -> 44432 bytes
-rw-r--r--doc/website-v1/history-guide.adoc275
-rw-r--r--doc/website-v1/img/history-guide/sample-cluster.conf.pngbin0 -> 10009 bytes
-rw-r--r--doc/website-v1/img/history-guide/smallapache-start.pngbin0 -> 1146 bytes
-rw-r--r--doc/website-v1/img/icons/README5
-rw-r--r--doc/website-v1/img/icons/callouts/1.pngbin0 -> 329 bytes
-rw-r--r--doc/website-v1/img/icons/callouts/10.pngbin0 -> 361 bytes
-rw-r--r--doc/website-v1/img/icons/callouts/11.pngbin0 -> 565 bytes
-rw-r--r--doc/website-v1/img/icons/callouts/12.pngbin0 -> 617 bytes
-rw-r--r--doc/website-v1/img/icons/callouts/13.pngbin0 -> 623 bytes
-rw-r--r--doc/website-v1/img/icons/callouts/14.pngbin0 -> 411 bytes
-rw-r--r--doc/website-v1/img/icons/callouts/15.pngbin0 -> 640 bytes
-rw-r--r--doc/website-v1/img/icons/callouts/2.pngbin0 -> 353 bytes
-rw-r--r--doc/website-v1/img/icons/callouts/3.pngbin0 -> 350 bytes
-rw-r--r--doc/website-v1/img/icons/callouts/4.pngbin0 -> 345 bytes
-rw-r--r--doc/website-v1/img/icons/callouts/5.pngbin0 -> 348 bytes
-rw-r--r--doc/website-v1/img/icons/callouts/6.pngbin0 -> 355 bytes
-rw-r--r--doc/website-v1/img/icons/callouts/7.pngbin0 -> 344 bytes
-rw-r--r--doc/website-v1/img/icons/callouts/8.pngbin0 -> 357 bytes
-rw-r--r--doc/website-v1/img/icons/callouts/9.pngbin0 -> 357 bytes
-rw-r--r--doc/website-v1/img/icons/caution.pngbin0 -> 2734 bytes
-rw-r--r--doc/website-v1/img/icons/example.pngbin0 -> 2599 bytes
-rw-r--r--doc/website-v1/img/icons/home.pngbin0 -> 1340 bytes
-rw-r--r--doc/website-v1/img/icons/important.pngbin0 -> 2980 bytes
-rw-r--r--doc/website-v1/img/icons/next.pngbin0 -> 1302 bytes
-rw-r--r--doc/website-v1/img/icons/note.pngbin0 -> 2494 bytes
-rw-r--r--doc/website-v1/img/icons/prev.pngbin0 -> 1348 bytes
-rw-r--r--doc/website-v1/img/icons/tip.pngbin0 -> 2718 bytes
-rw-r--r--doc/website-v1/img/icons/up.pngbin0 -> 1320 bytes
-rw-r--r--doc/website-v1/img/icons/warning.pngbin0 -> 3214 bytes
-rw-r--r--doc/website-v1/img/laptop.pngbin0 -> 2569 bytes
-rw-r--r--doc/website-v1/img/loader.gifbin0 -> 2545 bytes
-rw-r--r--doc/website-v1/img/servers.gifbin0 -> 4513 bytes
-rw-r--r--doc/website-v1/include/history-guide/basic-transition.typescript22
-rw-r--r--doc/website-v1/include/history-guide/diff.typescript11
-rw-r--r--doc/website-v1/include/history-guide/info.typescript16
-rw-r--r--doc/website-v1/include/history-guide/nfs-probe-err.typescript20
-rw-r--r--doc/website-v1/include/history-guide/resource-trace.typescript7
-rw-r--r--doc/website-v1/include/history-guide/resource.typescript6
-rw-r--r--doc/website-v1/include/history-guide/sample-cluster.conf.crm54
-rw-r--r--doc/website-v1/include/history-guide/status-probe-fail.typescript15
-rw-r--r--doc/website-v1/include/history-guide/stonith-corosync-stopped.typescript8
-rw-r--r--doc/website-v1/include/history-guide/transition-log.typescript13
-rw-r--r--doc/website-v1/index.adoc25
-rw-r--r--doc/website-v1/installation.adoc4
-rw-r--r--doc/website-v1/make-news.py136
-rw-r--r--doc/website-v1/man-1.2.adoc3437
-rw-r--r--doc/website-v1/man-2.0.adoc5048
-rw-r--r--doc/website-v1/man-3.adoc5309
-rw-r--r--doc/website-v1/man-4.3.adoc5160
-rw-r--r--doc/website-v1/news.adoc26
-rw-r--r--doc/website-v1/news/2014-06-30-release-2_1.adoc93
-rw-r--r--doc/website-v1/news/2014-10-28-release-2_1_1.adoc58
-rw-r--r--doc/website-v1/news/2015-01-26-release-2_1_2.adoc69
-rw-r--r--doc/website-v1/news/2015-04-10-release-2_1_3.adoc68
-rw-r--r--doc/website-v1/news/2015-05-13-release-2_1_4.adoc126
-rw-r--r--doc/website-v1/news/2015-05-25-getting-started-jp.adoc17
-rw-r--r--doc/website-v1/news/2016-01-12-release-2_1_5.adoc56
-rw-r--r--doc/website-v1/news/2016-01-15-release-2_2_0.adoc210
-rw-r--r--doc/website-v1/news/2016-04-28-release-2_2_1.adoc73
-rw-r--r--doc/website-v1/news/2016-08-12-release-2_3_0.adoc76
-rw-r--r--doc/website-v1/news/2016-09-01-release-2_1_7.adoc46
-rw-r--r--doc/website-v1/news/2016-09-02-release-2_3_1.adoc33
-rw-r--r--doc/website-v1/news/2016-09-05-release-2_2_2.adoc36
-rw-r--r--doc/website-v1/news/2017-01-31-release-3_0_0.adoc48
-rw-r--r--doc/website-v1/news/2021-06-17-release-4_3_1.adoc55
-rw-r--r--doc/website-v1/postprocess.py141
-rw-r--r--doc/website-v1/rsctest-guide.adoc238
-rw-r--r--doc/website-v1/scripts.adoc660
-rw-r--r--doc/website-v1/start-guide.adoc208
95 files changed, 31979 insertions, 0 deletions
diff --git a/doc/bootstrap-howto.md b/doc/bootstrap-howto.md
new file mode 100644
index 0000000..dac3eec
--- /dev/null
+++ b/doc/bootstrap-howto.md
@@ -0,0 +1,206 @@
+# How to use the Bootstrap commands
+
+## Introduction
+
+`crmsh` includes a set of cluster bootstrapping commands, both for
+setting up an initial cluster, adding and removing nodes in the
+cluster and for setting up geo clusters including arbitrators.
+
+This document is a simplified guide to using these commands. There are
+a lot of optional features that won't be fully covered by this guide,
+but it should serve as a basic introduction to the bootstrap commands.
+
+*Note:* These commands currently work correctly only on SUSE Linux
+ Enterprise or openSUSE, and only if the `csync2` command is installed
+ on all cluster nodes. For users of other distributions, please see
+ the documentation included with your operating system.
+
+## Commands
+
+First, here is the list of commands and a brief description of each
+one. Each command is available in the `crm cluster` namespace, so to
+run the `init` command, either call `crm cluster init` from the shell
+command line or navigate to the `cluster` level in the interactive
+`crm` shell and call the `init` command directly:
+
+* `init` - Initialize a new cluster from scratch.
+* `add` - Add a node to the current cluster.
+* `join` - Add the current node to a cluster.
+* `remove` - Remove a node from the cluster.
+* `geo-init` - Create a new geo cluster with the current cluster as its first member.
+* `geo-init-arbitrator` - Make the current node a geo cluster arbitrator.
+* `geo-join` - Join the current cluster to an existing geo cluster.
+
+## Initializing a basic cluster
+
+For the full documentation of the `init` command, see
+`crm help cluster init` in the interactive shell, or refer to the
+online documentation at [crmsh.github.io](https://crmsh.github.io/).
+
+### Using csync2 to synchronize configuration files
+
+By default, the bootstrap commands make some assumptions about the
+configuration to apply in order to simplify the process. One such
+assumption is that the `csync2` command is installed and available for
+use to synchronize the cluster configuration files across the
+cluster. When initializing the basic cluster, `init` will configure
+SSH access to all cluster nodes, open the necessary ports in the
+firewall, and configure `csync2` so that configuration files can be
+mirrored securely across the cluster.
+
+### Configuring SBD
+
+`SBD` is the storage based fencing mechanism recommended for use with
+pacemaker. Using a storage based fencing method simplifies
+configuration, as access to external hardware such as a lights-out
+device or UPS doesn't have to be configured, and nodes can self-fence
+if they detect reduced connectivity and loss of quorum.
+
+`init` will optionally configure SBD for you. To do this, pass the
+device to use as the SBD shared storage device using the
+`--sbd-device=<device>` argument. It is also possible to configure
+both SBD and a shared storage device formatted with the OCFS2 file
+system, using the `--partition-device=<device>` argument. To use this
+option, enable the `ocfs2` template using `-t ocfs2`.
+
+### Basic Example
+
+This command line when run on the uninitialized node `alice` will
+configure and start a basic cluster on the three nodes `alice`, `bob`
+and `carol`.
+
+```
+init --name basic-cluster --nodes "alice bob carol"
+```
+
+The `--name` argument is optional for regular clusters, but required
+when configuring a geo cluster.
+
+### Non-interactive configuration
+
+To run the initialization steps non-interactively, pass the `-y` or
+`--yes` flag to `init`. The default option will be chosen wherever the
+command would otherwise have prompted for user input. If no default
+option is available and user input is required, the command will
+abort.
+
+### Configuring shared storage
+
+To configure shared storage using the `init` command, make sure that
+you have a storage device accessible from all cluster nodes. This can
+be an iSCSI device provided by a SAN, or a shared storage volume as
+provided by your virtualization platform. To partition this device
+into two volumes, one for use by SBD and one as a shared OCFS2 device,
+use a command line like the following:
+
+```
+init --name storage-cluster --nodes "alice bob carol" -t ocfs2 -p /dev/shared
+```
+
+### Configuring an administration IP
+
+To immediately configure a virtual IP address resource in the cluster,
+use the `-A` argument: `init -A 1.2.3.4`.
+
+The common use case for this virtual IP is to have a single point of
+entry to [Hawk](https://hawk-ui.github.io), the cluster web
+interface. It is also useful as a first example of a cluster
+resource.
+
+## Adding a new node to a cluster
+
+There are two commands for adding a node to a cluster. When running
+the command from one of the existing cluster nodes, use the `add`
+form. For example, if there is an existing cluster consisting of the
+nodes `alice` and `bob`, the following command will add `carol` as the
+third node in the cluster when run from `alice` or `bob`:
+
+```
+alice# crm cluster add carol
+```
+
+It is also possible to add `carol` to the cluster from `carol`
+directly, using the `join` form:
+
+```
+carol# crm cluster join -c alice
+```
+
+Note that `join` takes an argument `-c <node>`.
+
+## Removing a node from a cluster
+
+To remove a node from the cluster, run
+
+```
+crm cluster remove <node>
+```
+
+To remove the last node in a cluster (thereby destroying the cluster),
+it is required to pass the `--force` flag to `crm`:
+
+```
+crm --force cluster remove $(hostname)
+```
+
+## Creating a geo cluster
+
+Once you have a cluster up and running and you made sure to give it a
+sensible name using `--name` (or by editing `corosync.conf` on all
+cluster nodes and restarting the cluster), you can turn that cluster
+into the first member in a geo cluster. Geo clusters are managed by
+the `booth` daemon, so to use these commands, `booth` needs to be
+installed on all cluster nodes.
+
+The `geo-init` command takes as its arguments a complete description
+of the geo cluster. This is because `booth` does not share its
+configuration across the cluster, instead each cluster node in each
+cluster needs to have a copy of the `booth` configuration.
+
+As an example, we will configure a geo cluster consisting of five
+nodes in total: The nodes `alice` and `bob` are members of the
+`amsterdam` cluster. `carol` and `dave` are members of the `london`
+cluster. Finally, `eve` is the arbitrator node located at a third
+site. The `amsterdam` cluster is identified by the virtual IP
+`192.168.100.8`, while the `london` cluster is identified by the
+virtual IP `192.168.100.9`.
+
+The `geo-init` command will configure these virtual IPs in each
+cluster, so there is no need to configure them before-hand.
+
+This geo cluster will share a single ticket, called `mcguffin`.
+
+To create this configuration, run
+
+```
+crm cluster geo-init \
+ --arbitrator eve \
+ --tickets mcguffin \
+ --clusters "amsterdam=192.168.100.8 london=192.168.100.9"
+```
+
+This will configure both the required cluster resources and the booth
+daemon itself in the initial cluster.
+
+## Adding an arbitrator to a geo cluster
+
+This example uses the same basic setup as the `geo-init` example.
+
+To configure the arbitrator `eve`, run the `geo-init-arbitrator`
+command on `eve`, passing the cluster IP of the existing `amsterdam`
+geo cluster member:
+
+```
+crm cluster geo-init-arbitrator \
+ --cluster-node 192.168.100.8
+```
+
+## Adding a second cluster to a geo cluster
+
+To add the `london` cluster to the existing geo cluster described in
+the previous two sections, run the `geo-join` command from one of the
+nodes in the cluster:
+
+```
+crm cluster geo-join --cluster-node 192.168.100.8
+```
diff --git a/doc/bootstrap-todo.md b/doc/bootstrap-todo.md
new file mode 100644
index 0000000..3db7136
--- /dev/null
+++ b/doc/bootstrap-todo.md
@@ -0,0 +1,56 @@
+# Bootstrap TODO
+
+(inherited from the bootstrap project)
+
+## Unclear Responsibility
+
+These may be in purview of ha-cluster-bootstrap, or may be in appliance image:
+
+* install debuginfo packages
+* enable coredumps
+
+
+## General / Random
+
+* csync2_remote assumes there's only one group in csync2.cfg, or, more to
+ the point, will only add new hosts to the first group.
+* Likewise, ssh_merge grabs all hosts regardless of what group they're in
+ (although this is probably fine)
+* get rid of curses junk in log file (fix ENV term)
+* Multi-device SBD (use multiple -s args)
+* Start new node on standby
+ # crm configure node node-1 attributes standby="on"
+ # crm node clearstate node-1 (requires confirmation)
+ - start new node, unbelievably it works!
+ # crm node online node-1
+* don't error to log if log not started
+* is "partx -a <device>" any sort of sane replacement for partprobe?
+* Use ssh-copy-id instead of manual fiddling with authorized_keys?
+
+
+## STONITH Config
+
+* See https://bugzilla.novell.com/show_bug.cgi?id=722405 for stonith timeout suggestions
+
+
+## Template Mode
+
+Generally specific to OCFS2 template ATM, as that's the only one extant.
+
+* Very long path to partition (/dev/disk/by-path/... for iSCSI) means we
+ can't determine paths to new partitions, thanks to bnc#722959. Unclear
+ if this will be fixed for SP2.
+* /dev/disk/by-id/dm-name paths are unreliable (at least for determining
+ partitions after carving the device up).
+* Probably need to prompt user for new partitions after carving, should
+ they not be found (FFS).
+* ocfs2 template is not the same as Hawk's. Consider enhancing
+ ha-cluster-bootstrap so it uses Hawk's templates directly rather than using
+ its own.
+* Ensure required RPMs are installed when running template (they're just
+ Recommends in the spec)
+* Specifying sbd without ocfs2 partition may be incompatible with ocfs2
+ template (need to test)
+* block device size " blockdev --getsz" etc. (when making OCFS2 partition
+ with "-T vmstore")
+
diff --git a/doc/crm.8.adoc b/doc/crm.8.adoc
new file mode 100644
index 0000000..da9fb38
--- /dev/null
+++ b/doc/crm.8.adoc
@@ -0,0 +1,5102 @@
+:man source: crm
+:man version: 4.6.0
+:man manual: crmsh documentation
+
+crm(8)
+======
+
+NAME
+----
+crm - Pacemaker command line interface for configuration and management
+
+
+SYNOPSIS
+--------
+*crm* [OPTIONS] [SUBCOMMAND ARGS...]
+
+
+[[topics_Description,Program description]]
+DESCRIPTION
+-----------
+The `crm` shell is a command-line based cluster configuration and
+management tool. Its goal is to assist as much as possible with the
+configuration and maintenance of Pacemaker-based High Availability
+clusters.
+
+For more information on Pacemaker itself, see http://clusterlabs.org/.
+
+`crm` works both as a command-line tool to be called directly from the
+system shell, and as an interactive shell with extensive tab
+completion and help.
+
+The primary focus of the `crm` shell is to provide a simplified and
+consistent interface to Pacemaker, but it also provides tools for
+managing the creation and configuration of High Availability clusters
+from scratch. To learn more about this aspect of `crm`, see the
+`cluster` section below.
+
+The `crm` shell can be used to manage every aspect of configuring and
+maintaining a cluster. It provides a simplified line-based syntax on
+top of the XML configuration format used by Pacemaker, commands for
+starting and stopping resources, tools for exploring the history of a
+cluster including log scraping and a set of cluster scripts useful for
+automating the setup and installation of services on the cluster
+nodes.
+
+The `crm` shell is line oriented: every command must start and finish
+on the same line. It is possible to use a continuation character (+\+)
+to write one command in two or more lines. The continuation character
+is commonly used when displaying configurations.
+
+[[topics_CommandLine,Command line options]]
+OPTIONS
+-------
+*-f, --file*='FILE'::
+ Load commands from the given file. If a dash +-+ is used in place
+ of a file name, `crm` will read commands from the shell standard
+ input (`stdin`).
+
+*-c, --cib*='CIB'::
+ Start the session using the given shadow CIB file.
+ Equivalent to +cib use <CIB>+.
+
+*-D, --display=*'OUTPUT_TYPE'::
+ Choose one of the output options: +plain+, +color-always+, +color+,
+ or +uppercase+. The default is +color+ if the terminal emulation
+ supports colors. Otherwise, +plain+ is used.
+
+*-F, --force*::
+ Make `crm` proceed with applying changes where it would normally
+ ask the user to confirm before proceeding. This option is mainly
+ useful in scripts, and should be used with care.
+
+*-w, --wait*::
+ Make `crm` wait for the cluster transition to finish (for the
+ changes to take effect) after each processed line.
+
+*-H, --history*='DIR|FILE|SESSION'::
+ A directory or file containing a cluster report to load
+ into the `history` commands, or the name of a previously
+ saved history session.
+
+*-h, --help*::
+ Print help page.
+
+*--version*::
+ Print crmsh version and build information (Mercurial Hg changeset
+ hash).
+
+*-d, --debug*::
+ Print verbose debugging information.
+
+*-R, --regression-tests*::
+ Enables extra verbose trace logging used by the regression
+ tests. Logs all external calls made by crmsh.
+
+*--scriptdir*='DIR'::
+ Extra directory where crm looks for cluster scripts, or a list of
+ directories separated by semi-colons (e.g. +/dir1;/dir2;etc.+).
+
+*-o, --opt*='OPTION=VALUE'::
+ Set crmsh option temporarily. If the options are saved using
+ +options save+ then the value passed here will also be saved.
+ Multiple options can be set by using +-o+ multiple times.
+
+[[topics_Introduction,Introduction]]
+== Introduction
+
+This section of the user guide covers general topics about the user
+interface and describes some of the features of `crmsh` in detail.
+
+[[topics_Introduction_Interface,User interface]]
+=== User interface
+
+The main purpose of `crmsh` is to provide a simple yet powerful
+interface to the cluster stack. There are two main modes of operation
+with the user interface of `crmsh`:
+
+* Command line (single-shot) use - Use `crm` as a regular UNIX command
+ from your usual shell. `crm` has full bash completion built in, so
+ using it in this manner should be as comfortable and familiar as
+ using any other command-line tool.
+
+* Interactive mode - By calling `crm` without arguments, or by calling
+ it with only a sublevel as argument, `crm` enters the interactive
+ mode. In this mode, it acts as its own command shell, which
+ remembers which sublevel you are currently in and allows for rapid
+ and convenient execution of multiple commands within the same
+ sublevel. This mode also has full tab completion, as well as
+ built-in interactive help and syntax highlighting.
+
+Here are a few examples of using `crm` both as a command-line tool and
+as an interactive shell:
+
+.Command line (one-shot) use:
+........
+# crm resource stop www_app
+........
+
+.Interactive use:
+........
+# crm
+crm(live)# resource
+crm(live)resource# unmanage tetris_1
+crm(live)resource# up
+crm(live)# node standby node4
+........
+
+.Cluster configuration:
+........
+# crm configure<<EOF
+ #
+ # resources
+ #
+ primitive disk0 iscsi \
+ params portal=192.168.2.108:3260 target=iqn.2008-07.com.suse:disk0
+ primitive fs0 Filesystem \
+ params device=/dev/disk/by-label/disk0 directory=/disk0 fstype=ext3
+ primitive internal_ip IPaddr params ip=192.168.1.101
+ primitive apache apache \
+ params configfile=/disk0/etc/apache2/site0.conf
+ primitive apcfence stonith:apcsmart \
+ params ttydev=/dev/ttyS0 hostlist="node1 node2" \
+ op start timeout=60s
+ primitive pingd ocf:pacemaker:ping \
+ params name=pingd dampen=5s multiplier=100 host_list="r1 r2"
+ #
+ # monitor apache and the UPS
+ #
+ monitor apache 60s:30s
+ monitor apcfence 120m:60s
+ #
+ # cluster layout
+ #
+ group internal_www \
+ disk0 fs0 internal_ip apache
+ clone fence apcfence \
+ meta globally-unique=false clone-max=2 clone-node-max=1
+ clone conn pingd \
+ meta globally-unique=false clone-max=2 clone-node-max=1
+ location node_pref internal_www \
+ rule 50: #uname eq node1 \
+ rule pingd: defined pingd
+ #
+ # cluster properties
+ #
+ property stonith-enabled=true
+ commit
+EOF
+........
+
+The `crm` interface is hierarchical, with commands organized into
+separate levels by functionality. To list the available levels and
+commands, either execute +help <level>+, or, if at the top level of
+the shell, simply typing `help` will provide an overview of all
+available levels and commands.
+
+The +(live)+ string in the `crm` prompt signifies that the current CIB
+in use is the cluster live configuration. It is also possible to
+work with so-called <<topics_Features_Shadows,shadow CIBs>>. These are separate, inactive
+configurations stored in files, that can be applied and thereby
+replace the live configuration at any time.
+
+[[topics_Introduction_Completion,Tab completion]]
+=== Tab completion
+
+The `crm` makes extensive use of tab completion. The completion
+is both static (i.e. for `crm` commands) and dynamic. The latter
+takes into account the current status of the cluster or
+information from installed resource agents. Sometimes, completion
+may also be used to get short help on resource parameters. Here
+are a few examples:
+
+...............
+crm(live)resource# <TAB><TAB>
+ban demote maintenance param scores trace
+cd failcount manage promote secret unmanage
+cleanup help meta quit start untrace
+clear locate move refresh status up
+constraints ls operations restart stop utilization
+
+crm(live)configure# primitive fence-1 <TAB><TAB>
+lsb: ocf: service: stonith: systemd:
+
+crm(live)configure# primitive fence-1 stonith:<TAB><TAB>
+apcmaster external/ippower9258 fence_legacy
+apcmastersnmp external/kdumpcheck ibmhmc
+apcsmart external/libvirt ipmilan
+
+crm(live)configure# primitive fence-1 stonith:ipmilan params <TAB><TAB>
+auth= hostname= ipaddr= login= password= port= priv=
+
+crm(live)configure# primitive fence-1 stonith:ipmilan params auth=<TAB><TAB>
+auth* (string)
+ The authorization type of the IPMI session ("none", "straight", "md2", or "md5")
+...............
+
+`crmsh` also comes with bash completion usable directly from the
+system shell. This should be installed automatically with the command
+itself.
+
+[[topics_Introduction_Shorthand,Shorthand syntax]]
+=== Shorthand syntax
+
+When using the `crm` shell to manage clusters, you will end up typing
+a lot of commands many times over. Clear command names like
++configure+ help in understanding and learning to use the cluster
+shell, but is easy to misspell and is tedious to type repeatedly. The
+interactive mode and tab completion both help with this, but the `crm`
+shell also has the ability to understand a variety of shorthand
+aliases for all of the commands.
+
+For example, instead of typing `crm status`, you can type `crm st` or
+`crm stat`. Instead of `crm configure` you can type `crm cfg` or even
+`crm cf`. `crm resource` can be shorted as `crm rsc`, and so on.
+
+The exact list of accepted aliases is too long to print in full, but
+experimentation and typos should help in discovering more of them.
+
+[[topics_Features,Features]]
+== Features
+
+The feature set of crmsh covers a wide range of functionality, and
+understanding how and when to use the various features of the shell
+can be difficult. This section of the guide describes some of the
+features and use cases of `crmsh` in more depth. The intention is to
+provide a deeper understanding of these features, but also to serve as
+a guide to using them.
+
+[[topics_Features_Shadows,Shadow CIB usage]]
+=== Shadow CIB usage
+
+A Shadow CIB is a normal cluster configuration stored in a file.
+They may be manipulated in much the same way as the _live_ CIB, with
+the key difference that changes to a shadow CIB have no effect on the
+actual cluster resources. An administrator may choose to apply any of
+them to the cluster, thus replacing the running configuration with the
+one found in the shadow CIB.
+
+The `crm` prompt always contains the name of the configuration which
+is currently in use, or the string _live_ if using the live cluster
+configuration.
+
+When editing the configuration in the `configure` level, no changes
+are actually applied until the `commit` command is executed. It is
+possible to start editing a configuration as usual, but instead of
+committing the changes to the active CIB, save them to a shadow CIB.
+
+The following example `configure` session demonstrates how this can be
+done:
+...............
+crm(live)configure# cib new test-2
+INFO: test-2 shadow CIB created
+crm(test-2)configure# commit
+...............
+
+[[topics_Features_Checks,Configuration semantic checks]]
+=== Configuration semantic checks
+
+Resource definitions may be checked against the meta-data
+provided with the resource agents. These checks are currently
+carried out:
+
+- are required parameters set
+- existence of defined parameters
+- timeout values for operations
+
+The parameter checks are obvious and need no further explanation.
+Failures in these checks are treated as configuration errors.
+
+The timeouts for operations should be at least as long as those
+recommended in the meta-data. Too short timeout values are a
+common mistake in cluster configurations and, even worse, they
+often slip through if cluster testing was not thorough. Though
+operation timeouts issues are treated as warnings, make sure that
+the timeouts are usable in your environment. Note also that the
+values given are just _advisory minimum_---your resources may
+require longer timeouts.
+
+User may tune the frequency of checks and the treatment of errors
+by the <<cmdhelp_options_check-frequency,`check-frequency`>> and
+<<cmdhelp_options_check-mode,`check-mode`>> preferences.
+
+Note that if the +check-frequency+ is set to +always+ and the
++check-mode+ to +strict+, errors are not tolerated and such
+configuration cannot be saved.
+
+[[topics_Features_Templates,Configuration templates]]
+=== Configuration templates
+
+.Deprecation note
+****************************
+Configuration templates have been deprecated in favor of the more
+capable `cluster scripts`. To learn how to use cluster scripts, see
+the dedicated documentation on the `crmsh` website at
+http://crmsh.github.io/, or in the <<cmdhelp_script,Script section>>.
+****************************
+
+Configuration templates are ready made configurations created by
+cluster experts. They are designed in such a way so that users
+may generate valid cluster configurations with minimum effort.
+If you are new to Pacemaker, templates may be the best way to
+start.
+
+We will show here how to create a simple yet functional Apache
+configuration:
+...............
+# crm configure
+crm(live)configure# template
+crm(live)configure template# list templates
+apache filesystem virtual-ip
+crm(live)configure template# new web <TAB><TAB>
+apache filesystem virtual-ip
+crm(live)configure template# new web apache
+INFO: pulling in template apache
+INFO: pulling in template virtual-ip
+crm(live)configure template# list
+web2-d web2 vip2 web3 vip web
+...............
+
+We enter the `template` level from `configure`. Use the `list`
+command to show templates available on the system. The `new`
+command creates a configuration from the +apache+ template. You
+can use tab completion to pick templates. Note that the apache
+template depends on a virtual IP address which is automatically
+pulled along. The `list` command shows the just created +web+
+configuration, among other configurations (I hope that you,
+unlike me, will use more sensible and descriptive names).
+
+The `show` command, which displays the resulting configuration,
+may be used to get an idea about the minimum required changes
+which have to be done. All +ERROR+ messages show the line numbers
+in which the respective parameters are to be defined:
+...............
+crm(live)configure template# show
+ERROR: 23: required parameter ip not set
+ERROR: 61: required parameter id not set
+ERROR: 65: required parameter configfile not set
+crm(live)configure template# edit
+...............
+
+The `edit` command invokes the preferred text editor with the
++web+ configuration. At the top of the file, the user is advised
+how to make changes. A good template should require from the user
+to specify only parameters. For example, the +web+ configuration
+we created above has the following required and optional
+parameters (all parameter lines start with +%%+):
+...............
+$ grep -n ^%% ~/.crmconf/web
+23:%% ip
+31:%% netmask
+35:%% lvs_support
+61:%% id
+65:%% configfile
+71:%% options
+76:%% envfiles
+...............
+
+These lines are the only ones that should be modified. Simply
+append the parameter value at the end of the line. For instance,
+after editing this template, the result could look like this (we
+used tabs instead of spaces to make the values stand out):
+...............
+$ grep -n ^%% ~/.crmconf/web
+23:%% ip 192.168.1.101
+31:%% netmask
+35:%% lvs_support
+61:%% id websvc
+65:%% configfile /etc/apache2/httpd.conf
+71:%% options
+76:%% envfiles
+...............
+
+As you can see, the parameter line format is very simple:
+...............
+%% <name> <value>
+...............
+
+After editing the file, use `show` again to display the
+configuration:
+...............
+crm(live)configure template# show
+primitive virtual-ip IPaddr \
+ params ip=192.168.1.101
+primitive apache apache \
+ params configfile="/etc/apache2/httpd.conf"
+monitor apache 120s:60s
+group websvc \
+ apache virtual-ip
+...............
+
+The target resource of the apache template is a group which we
+named +websvc+ in this sample session.
+
+This configuration looks exactly as you could type it at the
+`configure` level. The point of templates is to save you some
+typing. It is important, however, to understand the configuration
+produced.
+
+Finally, the configuration may be applied to the current
+crm configuration (note how the configuration changed slightly,
+though it is still equivalent, after being digested at the
+`configure` level):
+...............
+crm(live)configure template# apply
+crm(live)configure template# cd ..
+crm(live)configure# show
+node xen-b
+node xen-c
+primitive apache apache \
+ params configfile="/etc/apache2/httpd.conf" \
+ op monitor interval=120s timeout=60s
+primitive virtual-ip IPaddr \
+ params ip=192.168.1.101
+group websvc apache virtual-ip
+...............
+
+Note that this still does not commit the configuration to the CIB
+which is used in the shell, either the running one (+live+) or
+some shadow CIB. For that you still need to execute the `commit`
+command.
+
+To complete our example, we should also define the preferred node
+to run the service:
+
+...............
+crm(live)configure# location websvc-pref websvc 100: xen-b
+...............
+
+If you are not happy with some resource names which are provided
+by default, you can rename them now:
+
+...............
+crm(live)configure# rename virtual-ip intranet-ip
+crm(live)configure# show
+node xen-b
+node xen-c
+primitive apache apache \
+ params configfile="/etc/apache2/httpd.conf" \
+ op monitor interval=120s timeout=60s
+primitive intranet-ip IPaddr \
+ params ip=192.168.1.101
+group websvc apache intranet-ip
+location websvc-pref websvc 100: xen-b
+...............
+
+To summarize, working with templates typically consists of the
+following steps:
+
+- `new`: create a new configuration from templates
+- `edit`: define parameters, at least the required ones
+- `show`: see if the configuration is valid
+- `apply`: apply the configuration to the `configure` level
+
+[[topics_Features_Testing,Resource testing]]
+=== Resource testing
+
+The amount of detail in a cluster makes all configurations prone
+to errors. By far the largest number of issues in a cluster is
+due to bad resource configuration. The shell can help quickly
+diagnose such problems. And considerably reduce your keyboard
+wear.
+
+Let's say that we entered the following configuration:
+...............
+node xen-b
+node xen-c
+node xen-d
+primitive fencer stonith:external/libvirt \
+ params hypervisor_uri="qemu+tcp://10.2.13.1/system" \
+ hostlist="xen-b xen-c xen-d" \
+ op monitor interval=2h
+primitive svc Xinetd \
+ params service=systat \
+ op monitor interval=30s
+primitive intranet-ip IPaddr2 \
+ params ip=10.2.13.100 \
+ op monitor interval=30s
+primitive apache apache \
+ params configfile="/etc/apache2/httpd.conf" \
+ op monitor interval=120s timeout=60s
+group websvc apache intranet-ip
+location websvc-pref websvc 100: xen-b
+...............
+
+Before typing `commit` to submit the configuration to the cib we
+can make sure that all resources are usable on all nodes:
+...............
+crm(live)configure# rsctest websvc svc fencer
+...............
+
+It is important that resources being tested are not running on
+any nodes. Otherwise, the `rsctest` command will refuse to do
+anything. Of course, if the current configuration resides in a
+CIB shadow, then a `commit` is irrelevant. The point being that
+resources are not running on any node.
+
+.Note on stopping all resources
+****************************
+Alternatively to not committing a configuration, it is also
+possible to tell Pacemaker not to start any resources:
+
+...............
+crm(live)configure# property stop-all-resources=yes
+...............
+Almost none---resources of class stonith are still started. But
+shell is not as strict when it comes to stonith resources.
+****************************
+
+Order of resources is significant insofar that a resource depends
+on all resources to its left. In most configurations, it's
+probably practical to test resources in several runs, based on
+their dependencies.
+
+Apart from groups, `crm` does not interpret constraints and
+therefore knows nothing about resource dependencies. It also
+doesn't know if a resource can run on a node at all in case of an
+asymmetric cluster. It is up to the user to specify a list of
+eligible nodes if a resource is not meant to run on every node.
+
+[[topics_Features_Security,Access Control Lists (ACL)]]
+=== Access Control Lists (ACL)
+
+.Note on ACLs in Pacemaker 1.1.12
+****************************
+The support for ACLs has been revised in Pacemaker version 1.1.12 and
+up. Depending on which version you are using, the information in this
+section may no longer be accurate. Look for the `acl_target`
+configuration element for more details on the new syntax.
+****************************
+
+By default, the users from the +haclient+ group have full access
+to the cluster (or, more precisely, to the CIB). Access control
+lists allow for finer access control to the cluster.
+
+Access control lists consist of an ordered set of access rules.
+Each rule allows read or write access or denies access
+completely. Rules are typically combined to produce a specific
+role. Then, users may be assigned a role.
+
+For instance, this is a role which defines a set of rules
+allowing management of a single resource:
+
+...............
+role bigdb_admin \
+ write meta:bigdb:target-role \
+ write meta:bigdb:is-managed \
+ write location:bigdb \
+ read ref:bigdb
+...............
+
+The first two rules allow modifying the +target-role+ and
++is-managed+ meta attributes which effectively enables users in
+this role to stop/start and manage/unmanage the resource. The
+constraints write access rule allows moving the resource around.
+Finally, the user is granted read access to the resource
+definition.
+
+For proper operation of all Pacemaker programs, it is advisable
+to add the following role to all users:
+
+...............
+role read_all \
+ read cib
+...............
+
+For finer grained read access try with the rules listed in the
+following role:
+
+...............
+role basic_read \
+ read node attribute:uname \
+ read node attribute:type \
+ read property \
+ read status
+...............
+
+It is however possible that some Pacemaker programs (e.g.
+`ptest`) may not function correctly if the whole CIB is not
+readable.
+
+Some of the ACL rules in the examples above are expanded by the
+shell to XPath specifications. For instance,
++meta:bigdb:target-role+ expands to:
+
+........
+//primitive[@id='bigdb']/meta_attributes/nvpair[@name='target-role']
+........
+
+You can see the expansion by showing XML:
+
+...............
+crm(live) configure# show xml bigdb_admin
+...
+<acls>
+ <acl_role id="bigdb_admin">
+ <write id="bigdb_admin-write"
+ xpath="//primitive[@id='bigdb']/meta_attributes/nvpair[@name='target-role']"/>
+...............
+
+Many different XPath expressions can have equal meaning. For
+instance, the following two are equal, but only the first one is
+going to be recognized as shortcut:
+
+...............
+//primitive[@id='bigdb']/meta_attributes/nvpair[@name='target-role']
+//resources/primitive[@id='bigdb']/meta_attributes/nvpair[@name='target-role']
+...............
+
+XPath is a powerful language, but you should try to keep your ACL
+xpaths simple and the builtin shortcuts should be used whenever
+possible.
+
+[[topics_Features_Resourcesets,Syntax: Resource sets]]
+=== Syntax: Resource sets
+
+Using resource sets can be a bit confusing unless one knows the
+details of the implementation in Pacemaker as well as how to interpret
+the syntax provided by `crmsh`.
+
+Three different types of resource sets are provided by `crmsh`, and
+each one implies different values for the two resource set attributes,
++sequential+ and +require-all+.
+
++sequential+::
+ If false, the resources in the set do not depend on each other
+ internally. Setting +sequential+ to +true+ implies a strict order of
+ dependency within the set.
+
++require-all+::
+ If false, only one resource in the set is required to fulfil the
+ requirements of the set. The set of A, B and C with +require-all+
+ set to +false+ is be read as "A OR B OR C" when its dependencies
+ are resolved.
+
+The three types of resource sets modify the attributes in the
+following way:
+
+1. Implicit sets (no brackets). +sequential=true+, +require-all=true+
+2. Parenthesis set (+(+ ... +)+). +sequential=false+, +require-all=true+
+3. Bracket set (+[+ ... +]+). +sequential=false+, +require-all=false+
+
+To create a set with the properties +sequential=true+ and
++require-all=false+, explicitly set +sequential+ in a bracketed set,
++[ A B C sequential=true ]+.
+
+To create multiple sets with both +sequential+ and +require-all+ set to
+true, explicitly set +sequential+ in a parenthesis set:
++A B ( C D sequential=true )+.
+
+[[topics_Features_AttributeListReferences,Syntax: Attribute list references]]
+=== Syntax: Attribute list references
+
+Attribute lists are used to set attributes and parameters for
+resources, constraints and property definitions. For example, to set
+the virtual IP used by an +IPAddr2+ resource the attribute +ip+ can be
+set in an attribute list for that resource.
+
+Attribute lists can have identifiers that name them, and other
+resources can reuse the same attribute list by referring to that name
+using an +$id-ref+. For example, the following statement defines a
+simple dummy resource with an attribute list which sets the parameter
++state+ to the value 1 and sets the identifier for the attribute list
+to +on-state+:
+
+..............
+primitive dummy-1 Dummy params $id=on-state state=1
+..............
+
+To refer to this attribute list from a different resource, refer to
+the +on-state+ name using an id-ref:
+
+..............
+primitive dummy-2 Dummy params $id-ref=on-state
+..............
+
+The resource +dummy-2+ will now also have the parameter +state+ set to the value 1.
+
+[[topics_Features_AttributeReferences,Syntax: Attribute references]]
+=== Syntax: Attribute references
+
+In some cases, referencing complete attribute lists is too
+coarse-grained, for example if two different parameters with different
+names should have the same value set. Instead of having to copy the
+value in multiple places, it is possible to create references to
+individual attributes in attribute lists.
+
+To name an attribute in order to be able to refer to it later, prefix
+the attribute name with a +$+ character (as seen above with the
+special names +$id+ and +$id-ref+:
+
+............
+primitive dummy-1 Dummy params $state=1
+............
+
+The identifier +state+ can now be used to refer to this attribute from other
+primitives, using the +@<id>+ syntax:
+
+............
+primitive dummy-2 Dummy params @state
+............
+
+In some cases, using the attribute name as the identifier doesn't work
+due to name clashes. In this case, the syntax +$<id>:<name>=<value>+
+can be used to give the attribute a different identifier:
+
+............
+primitive dummy-1 params $dummy-state-on:state=1
+primitive dummy-2 params @dummy-state-on
+............
+
+There is also the possibility that two resources both use the same
+attribute value but with different names. For example, a web server
+may have a parameter +server_ip+ for setting the IP address where it
+listens for incoming requests, and a virtual IP resource may have a
+parameter called +ip+ which sets the IP address it creates. To
+configure these two resources with an IP without repeating the value,
+the reference can be given a name using the syntax +@<id>:<name>+.
+
+Example:
+............
+primitive virtual-ip IPaddr2 params $vip:ip=192.168.1.100
+primitive webserver apache params @vip:server_ip
+............
+
+[[topics_Syntax_RuleExpressions,Syntax: Rule expressions]]
+=== Syntax: Rule expressions
+
+Many of the configuration commands in `crmsh` now support the use of
+_rule expressions_, which can influence what attributes apply to a
+resource or under which conditions a constraint is applied, depending
+on changing conditions like date, time, the value of attributes and
+more.
+
+Here is an example of a simple rule expression used to apply a
+a different resource parameter on the node named `node1`:
+
+..............
+primitive my_resource Special \
+ params 2: rule #uname eq node1 interface=eth1 \
+ params 1: interface=eth0
+..............
+
+This primitive resource has two lists of parameters with descending
+priority. The parameter list with the highest priority is applied
+first, but only if the rule expressions for that parameter list all
+apply. In this case, the rule `#uname eq node1` limits the parameter
+list so that it is only applied on `node1`.
+
+Note that rule expressions are not terminated and are immediately
+followed by the data to which the rule is applied. In this case, the
+name-value pair `interface=eth1`.
+
+Rule expressions can contain multiple expressions connected using the
+boolean operator `or` and `and`. The full syntax for rule expressions
+is listed below.
+
+..............
+rules ::
+ rule [id_spec] [$role=<role>] <score>: <expression>
+ [rule [id_spec] [$role=<role>] <score>: <expression> ...]
+
+id_spec :: $id=<id> | $id-ref=<id>
+score :: <number> | <attribute> | [-]inf
+expression :: <simple_exp> [<bool_op> <simple_exp> ...]
+bool_op :: or | and
+simple_exp :: <attribute> [type:]<binary_op> <value>
+ | <unary_op> <attribute>
+ | date <date_expr>
+type :: <string> | <version> | <number>
+binary_op :: lt | gt | lte | gte | eq | ne
+unary_op :: defined | not_defined
+
+date_expr :: lt <end>
+ | gt <start>
+ | in start=<start> end=<end>
+ | in start=<start> <duration>
+ | spec <date_spec>
+duration|date_spec ::
+ hours=<value>
+ | monthdays=<value>
+ | weekdays=<value>
+ | yearsdays=<value>
+ | months=<value>
+ | weeks=<value>
+ | years=<value>
+ | weekyears=<value>
+ | moon=<value>
+..............
+
+[[topics_Lifetime,Lifetime parameter format]]
+== Lifetime parameter format
+
+Lifetimes can be specified in the ISO 8601 time format or the ISO 8601
+duration format. To distinguish between months and minutes, use the PT
+prefix before specifying minutes. The duration format is one of
++PnYnMnDTnHnMnS+, +PnW+, +P<date>T<time>+.
+
+P = duration. Y = year. M = month. W = week. D = day. T = time. H =
+hour. M = minute. S = second.
+
+Examples:
+.................
+PT5M = 5 minutes later.
+3D = 3 days later.
+PT1H = 1 hour later.
+.................
+
+The cluster checks lifetimes at an interval defined by the
+cluster-recheck-interval property (default 15 minutes).
+
+
+[[topics_Reference,Command reference]]
+== Command reference
+
+The commands are structured to be compatible with the shell command
+line. Sometimes, the underlying Pacemaker grammar uses characters that
+have special meaning in bash, that will need to be quoted. This
+includes the hash or pound sign (`#`), single and double quotes, and
+any significant whitespace.
+
+Whitespace is also significant when assigning values, meaning that
++key=value+ is different from +key = value+.
+
+Commands can be referenced using short-hand as long as the short-hand
+is unique. This can be either a prefix of the command name or a prefix
+string of characters found in the name.
+
+For example, +status+ can be abbreviated as +st+ or +su+, and
++configure+ as +conf+ or +cfg+.
+
+The syntax for the commands is given below in an informal, BNF-like
+grammar.
+
+* `<value>` denotes a string.
+* `[value]` means that the construct is optional.
+* The ellipsis (`...`) signifies that the previous construct may be
+ repeated.
+* `first|second` means either first or second.
+* The rest are literals (strings, `:`, `=`).
+
+[[cmdhelp_root_status,Cluster status]]
+=== `status`
+
+Show cluster status. The status is displayed by `crm_mon`. Supply
+additional arguments for more information or different format.
+See `crm_mon(8)` for more details.
+
+Example:
+...............
+status
+status simple
+status full
+...............
+
+Usage:
+...............
+status [<option> ...]
+
+option :: full
+ | bynode
+ | inactive
+ | ops
+ | timing
+ | failcounts
+ | verbose
+ | quiet
+ | html
+ | xml
+ | simple
+ | tickets
+ | noheaders
+ | detail
+ | brief
+...............
+
+[[cmdhelp_root_verify,Verify cluster status]]
+=== `verify`
+
+Performs basic checks for the cluster configuration and
+current status, reporting potential issues.
+
+See `crm_verify(8)` and `crm_simulate(8)` for more details.
+
+Example:
+...............
+verify
+verify scores
+...............
+
+Usage:
+...............
+verify [scores]
+...............
+
+[[cmdhelp_cluster,Cluster setup and management]]
+=== `cluster` - Cluster setup and management
+
+Whole-cluster configuration management with High Availability
+awareness.
+
+The commands on the cluster level allows configuration and
+modification of the underlying cluster infrastructure, and also
+supplies tools to do whole-cluster systems management.
+
+These commands enable easy installation and maintenance of a HA
+cluster, by providing support for package installation, configuration
+of the cluster messaging layer, file system setup and more.
+
+[[cmdhelp_cluster_copy,Copy file to other cluster nodes]]
+==== `copy`
+
+Copy file to other cluster nodes.
+
+Copies the given file to all other nodes unless given a
+list of nodes to copy to as argument.
+
+Usage:
+...............
+copy <filename> [nodes ...]
+...............
+
+Example:
+...............
+copy /etc/motd
+...............
+
+[[cmdhelp_cluster_diff,Diff file across cluster]]
+==== `diff`
+
+Displays the difference, if any, between a given file
+on different nodes. If the second argument is `--checksum`,
+a checksum of the file will be calculated and displayed for
+each node.
+
+Usage:
+...............
+diff <file> [--checksum] [nodes...]
+...............
+
+Example:
+...............
+diff /etc/crm/crm.conf node2
+diff /etc/resolv.conf --checksum
+...............
+
+[[cmdhelp_cluster_disable,Disable cluster services,From Code]]
+==== `disable`
+See "crm cluster help disable" or "crm cluster disable --help"
+
+[[cmdhelp_cluster_enable,Enable cluster services,From Code]]
+==== `enable`
+See "crm cluster help enable" or "crm cluster enable --help
+
+[[cmdhelp_cluster_geo_init,Configure cluster as geo cluster,From Code]]
+==== `geo-init`
+See "crm cluster help geo_init" or "crm cluster geo_init --help"
+
+[[cmdhelp_cluster_geo_init_arbitrator,Initialize node as geo cluster arbitrator,From Code]]
+==== `geo-init-arbitrator`
+See "crm cluster help geo_init_arbitrator" or "crm cluster geo_init_arbitrator --help"
+
+[[cmdhelp_cluster_geo_join,Join cluster to existing geo cluster,From Code]]
+==== `geo-join`
+See "crm cluster help geo_join" or "crm cluster geo_join --help"
+
+[[cmdhelp_cluster_health,Cluster health check]]
+==== `health`
+
+Runs a larger set of tests and queries on all nodes in the cluster to
+verify the general system health and detect potential problems.
+
+Usage:
+...............
+health
+...............
+
+[[cmdhelp_cluster_init,Initializes a new HA cluster,From Code]]
+==== `init`
+See "crm cluster help init" or "crm cluster init --help"
+
+[[cmdhelp_cluster_join,Join existing cluster,From Code]]
+==== `join`
+See "crm cluster help join" or "crm cluster join --help"
+
+[[cmdhelp_cluster_remove,Remove node(s) from the cluster,From Code]]
+==== `remove`
+See "crm cluster help remove" or "crm cluster remove --help"
+
+[[cmdhelp_cluster_crash_test,Cluster crash test tool set,From Code]]
+==== `crash_test`
+See "crm cluster help crash_test" or "crm cluster crash_test --help"
+
+[[cmdhelp_cluster_restart,Restart cluster services,From Code]]
+==== `restart`
+See "crm cluster help restart" or "crm cluster restart --help"
+
+[[cmdhelp_cluster_rename,Rename the cluster]]
+==== `rename`
+
+Rename the cluster name
+
+Usage:
+...............
+rename <new_cluster_name>
+...............
+
+
+[[cmdhelp_cluster_run,Execute an arbitrary command on all nodes/specific node]]
+==== `run`
+
+This command takes a shell statement as argument, executes that
+statement on all nodes in the cluster or a specific node,
+and reports the result.
+
+Usage:
+...............
+run <command> [node ...]
+...............
+
+Example:
+...............
+run "cat /proc/uptime"
+run "ls" node1 node2
+...............
+
+[[cmdhelp_cluster_start,Start cluster services,From Code]]
+==== `start`
+See "crm cluster help start" or "crm cluster start --help"
+
+[[cmdhelp_cluster_status,Cluster status check]]
+==== `status`
+
+Reports the status for the cluster messaging layer on the local
+node.
+
+Usage:
+...............
+status
+...............
+
+[[cmdhelp_cluster_stop,Stop cluster services,From Code]]
+==== `stop`
+See "crm cluster help stop" or "crm cluster stop --help"
+
+[[cmdhelp_cluster_wait_for_startup,Wait for cluster to start]]
+==== `wait_for_startup`
+
+Mostly useful in scripts or automated workflows, this command will
+attempt to connect to the local cluster node repeatedly. The command
+will keep trying until the cluster node responds, or the `timeout`
+elapses. The timeout can be changed by supplying a value in seconds as
+an argument.
+
+Usage:
+........
+wait_for_startup
+........
+
+[[cmdhelp_script,Cluster script management]]
+=== `script` - Cluster script management
+
+A big part of the configuration and management of a cluster is
+collecting information about all cluster nodes and deploying changes
+to those nodes. Often, just performing the same procedure on all nodes
+will encounter problems, due to subtle differences in the
+configuration.
+
+For example, when configuring a cluster for the first time, the
+software needs to be installed and configured on all nodes before the
+cluster software can be launched and configured using `crmsh`. This
+process is cumbersome and error-prone, and the goal is for scripts to
+make this process easier.
+
+Scripts are implemented using the python `parallax` package which
+provides a thin wrapper on top of SSH. This allows the scripts to
+function through the usual SSH channels used for system maintenance,
+requiring no additional software to be installed or maintained.
+
+[[cmdhelp_script_json,JSON API for cluster scripts]]
+==== `json`
+
+This command provides a JSON API for the cluster scripts, intended for
+use in user interface tools that want to interact with the cluster via
+scripts.
+
+The command takes a single argument, which should be a JSON array with
+the first member identifying the command to perform.
+
+The output is line-based: Commands that return multiple results will
+return them line-by-line, ending with a terminator value: "end".
+
+When providing parameter values to this command, they should be
+provided as nested objects, so +virtual-ip:ip=192.168.0.5+ on the
+command line becomes the JSON object
++{"virtual-ip":{"ip":"192.168.0.5"}}+.
+
+API:
+........
+["list"]
+=> [{name, shortdesc, category}]
+
+["show", <name>]
+=> [{name, shortdesc, longdesc, category, <<steps>>}]
+
+<<steps>> := [{name, shortdesc], longdesc, required, parameters, steps}]
+
+<<params>> := [{name, shortdesc, longdesc, required, unique, advanced,
+ type, value, example}]
+
+["verify", <name>, <<values>>]
+=> [{shortdesc, longdesc, text, nodes}]
+
+["run", <name>, <<values>>]
+=> [{shortdesc, rc, output|error}]
+........
+
+
+[[cmdhelp_script_list,List available scripts]]
+==== `list`
+
+Lists the available scripts, sorted by category. Scripts that have the
+special `Script` category are hidden by default, since they are mainly
+used by other scripts or commands. To also show these, pass `all` as
+argument.
+
+To get a flat list of script names, not sorted by category, pass
+`names` as an extra argument.
+
+Usage:
+............
+list [all] [names]
+............
+
+Example:
+............
+list
+list all names
+............
+
+[[cmdhelp_script_run,Run the script]]
+==== `run`
+
+Given a list of parameter values, this command will execute the
+actions specified by the cluster script. The format for the parameter
+values is the same as for the `verify` command.
+
+Can optionally take at least two parameters:
+* `nodes=<nodes>`: List of nodes that the script runs over
+* `dry_run=yes|no`: If set, the script will not perform any modifications.
+
+Additional parameters may be available depending on the script.
+
+Use the `show` command to see what parameters are available.
+
+Usage:
+.............
+run <script> [args...]
+.............
+
+Example:
+.............
+run apache install=true
+run sbd id=sbd-1 node=node1 sbd_device=/dev/disk/by-uuid/F00D-CAFE
+.............
+
+[[cmdhelp_script_show,Describe the script]]
+==== `show`
+
+Prints a description and short summary of the script, with
+descriptions of the accepted parameters.
+
+Advanced parameters are hidden by default. To show the complete list
+of parameters accepted by the script, pass `all` as argument.
+
+Usage:
+............
+show <script> [all]
+............
+
+Example:
+............
+show virtual-ip
+............
+
+[[cmdhelp_script_verify,Verify the script]]
+==== `verify`
+
+Checks the given parameter values, and returns a list
+of actions that will be executed when running the script
+if provided the same list of parameter values.
+
+Usage:
+............
+verify <script> [args...]
+............
+
+Example:
+............
+verify sbd id=sbd-1 node=node1 sbd_device=/dev/disk/by-uuid/F00D-CAFE
+............
+
+[[cmdhelp_corosync,Corosync management]]
+=== `corosync` - Corosync management
+
+Corosync is the underlying messaging layer for most HA clusters.
+This level provides commands for editing and managing the corosync
+configuration.
+
+[[cmdhelp_corosync_add-node,Add a corosync node]]
+==== `add-node`
+
+Adds a node to the corosync configuration. This is used with the `udpu`
+type configuration in corosync.
+
+A nodeid for the added node is generated automatically.
+
+Note that this command assumes that only a single ring is used, and
+sets only the address for ring0.
+
+Usage:
+.........
+add-node <addr> [name]
+.........
+
+[[cmdhelp_corosync_del-node,Remove a corosync node]]
+==== `del-node`
+
+Removes a node from the corosync configuration. The argument given is
+the `ring0_addr` address set in the configuration file.
+
+Usage:
+.........
+del-node <addr>
+.........
+
+[[cmdhelp_corosync_diff,Diffs the corosync configuration]]
+==== `diff`
+
+Diffs the corosync configurations on different nodes. If no nodes are
+given as arguments, the corosync configurations on all nodes in the
+cluster are compared.
+
+`diff` takes an option argument `--checksum`, to display a checksum
+for each file instead of calculating a diff.
+
+Usage:
+.........
+diff [--checksum] [node...]
+.........
+
+[[cmdhelp_corosync_edit,Edit the corosync configuration]]
+==== `edit`
+
+Opens the Corosync configuration file in an editor.
+
+Usage:
+.........
+edit
+.........
+
+[[cmdhelp_corosync_get,Get a corosync configuration value]]
+==== `get`
+
+Returns the value configured in `corosync.conf`, which is not
+necessarily the value used in the running configuration. See `reload`
+for telling corosync about configuration changes.
+
+The argument is the complete dot-separated path to the value.
+
+If there are multiple values configured with the same path, the
+command returns all values for that path. For example, to get all
+configured `ring0_addr` values, use this command:
+
+Example:
+........
+get nodelist.node.ring0_addr
+........
+
+[[cmdhelp_corosync_log,Show the corosync log file]]
+==== `log`
+
+Opens the log file specified in the corosync configuration file. If no
+log file is configured, this command returns an error.
+
+The pager used can be configured either using the PAGER
+environment variable or in `crm.conf`.
+
+Usage:
+.........
+log
+.........
+
+[[cmdhelp_corosync_pull,Pulls the corosync configuration]]
+==== `pull`
+
+Gets the corosync configuration from another node and copies
+it to this node.
+
+Usage:
+.........
+pull <node>
+.........
+
+[[cmdhelp_corosync_push,Push the corosync configuration]]
+==== `push`
+
+Pushes the corosync configuration file on this node to
+the list of nodes provided. If no target nodes are given,
+the configuration is pushed to all other nodes in the cluster.
+
+It is recommended to use `csync2` to distribute the cluster
+configuration files rather than relying on this command.
+
+Usage:
+.........
+push [node] ...
+.........
+
+Example:
+.........
+push node-2 node-3
+.........
+
+[[cmdhelp_corosync_reload,Reload the corosync configuration]]
+==== `reload`
+
+Tells all instances of corosync in this cluster to reload
+`corosync.conf`.
+
+After pushing a new configuration to all cluster nodes, call this
+command to make corosync use the new configuration.
+
+Usage:
+.........
+reload
+.........
+
+[[cmdhelp_corosync_set,Set a corosync configuration value]]
+==== `set`
+
+Sets the value identified by the given path. If the value does not
+exist in the configuration file, it will be added. However, if the
+section containing the value does not exist, the command will fail.
+
+Usage:
+.........
+set quorum.expected_votes 2
+.........
+
+[[cmdhelp_corosync_show,Display the corosync configuration]]
+==== `show`
+
+Displays the corosync configuration on the current node.
+
+.........
+show
+.........
+
+[[cmdhelp_corosync_status,Display the corosync status]]
+==== `status`
+
+Displays the corosync ring status(default), also can display quorum/qdevice/qnetd status.
+
+Usage:
+.........
+status [ring|quorum|qdevice|qnetd]
+.........
+
+[[cmdhelp_cib,CIB shadow management]]
+=== `cib` - CIB shadow management
+
+This level is for management of shadow CIBs. It is available both
+at the top level and the `configure` level.
+
+All the commands are implemented using `cib_shadow(8)` and the
+`CIB_shadow` environment variable. The user prompt always
+includes the name of the currently active shadow or the live CIB.
+
+[[cmdhelp_cib_cibstatus,CIB status management and editing]]
+==== `cibstatus`
+
+Enter edit and manage the CIB status section level. See the
+<<cmdhelp_cibstatus,CIB status management section>>.
+
+[[cmdhelp_cib_commit,copy a shadow CIB to the cluster]]
+==== `commit`
+
+Apply a shadow CIB to the cluster. If the shadow name is omitted
+then the current shadow CIB is applied.
+
+Temporary shadow CIBs are removed automatically on commit.
+
+Usage:
+...............
+commit [<cib>]
+...............
+
+[[cmdhelp_cib_delete,delete a shadow CIB]]
+==== `delete`
+
+Delete an existing shadow CIB.
+
+Usage:
+...............
+delete <cib>
+...............
+
+[[cmdhelp_cib_diff,diff between the shadow CIB and the live CIB]]
+==== `diff`
+
+Print differences between the current cluster configuration and
+the active shadow CIB.
+
+Usage:
+...............
+diff
+...............
+
+[[cmdhelp_cib_import,import a CIB or PE input file to a shadow]]
+==== `import`
+
+At times it may be useful to create a shadow file from the
+existing CIB. The CIB may be specified as file or as a PE input
+file number. The shell will look up files in the local directory
+first and then in the PE directory (typically `/var/lib/pengine`).
+Once the CIB file is found, it is copied to a shadow and this
+shadow is immediately available for use at both `configure` and
+`cibstatus` levels.
+
+If the shadow name is omitted then the target shadow is named
+after the input CIB file.
+
+Note that there are often more than one PE input file, so you may
+need to specify the full name.
+
+Usage:
+...............
+import {<file>|<number>} [<shadow>]
+...............
+Examples:
+...............
+import pe-warn-2222
+import 2289 issue2
+...............
+
+[[cmdhelp_cib_list,list all shadow CIBs]]
+==== `list`
+
+List existing shadow CIBs.
+
+Usage:
+...............
+list
+...............
+
+[[cmdhelp_cib_new,create a new shadow CIB]]
+==== `new`
+
+Create a new shadow CIB. The live cluster configuration and
+status is copied to the shadow CIB.
+
+If the name of the shadow is omitted, we create a temporary CIB
+shadow. It is useful if multiple level sessions are desired
+without affecting the cluster. A temporary CIB shadow is short
+lived and will be removed either on `commit` or on program exit.
+Note that if the temporary shadow is not committed all changes in
+the temporary shadow are lost.
+
+Specify `withstatus` if you want to edit the status section of
+the shadow CIB (see the <<cmdhelp_cibstatus,cibstatus section>>).
+Add `force` to force overwriting the existing shadow CIB.
+
+To start with an empty configuration that is not copied from the live
+CIB, specify the `empty` keyword. (This also allows a shadow CIB to be
+created in case no cluster is running.)
+
+Usage:
+...............
+new [<cib>] [withstatus] [force] [empty]
+...............
+
+[[cmdhelp_cib_reset,copy live cib to a shadow CIB]]
+==== `reset`
+
+Copy the current cluster configuration into the shadow CIB.
+
+Usage:
+...............
+reset <cib>
+...............
+
+[[cmdhelp_cib_use,change working CIB]]
+==== `use`
+
+Choose a CIB source. If you want to edit the status from the
+shadow CIB specify `withstatus` (see <<cmdhelp_cibstatus,`cibstatus`>>).
+Leave out the CIB name to switch to the running CIB.
+
+Usage:
+...............
+use [<cib>] [withstatus]
+...............
+
+[[cmdhelp_ra,Resource Agents (RA) lists and documentation]]
+=== `ra` - Resource Agents (RA) lists and documentation
+
+This level contains commands which show various information about
+the installed resource agents. It is available both at the top
+level and at the `configure` level.
+
+[[cmdhelp_ra_classes,list classes and providers]]
+==== `classes`
+
+Print all resource agents' classes and, where appropriate, a list
+of available providers.
+
+Usage:
+...............
+classes
+...............
+
+[[cmdhelp_ra_info,show meta data for a RA]]
+==== `info` (`meta`)
+
+Show the meta-data of a resource agent type. This is where users
+can find information on how to use a resource agent. It is also
+possible to get information from some programs: `pengine`,
+`crmd`, `cib`, and `stonithd`. Just specify the program name
+instead of an RA.
+
+Usage:
+...............
+info [<class>:[<provider>:]]<type>
+info <type> <class> [<provider>] (obsolete)
+...............
+Example:
+...............
+info apache
+info ocf:pacemaker:Dummy
+info stonith:ipmilan
+info pengine
+...............
+
+[[cmdhelp_ra_list,list RA for a class (and provider)]]
+==== `list`
+
+List available resource agents for the given class. If the class
+is `ocf`, supply a provider to get agents which are available
+only from that provider.
+
+Usage:
+...............
+list <class> [<provider>]
+...............
+Example:
+...............
+list ocf pacemaker
+...............
+
+[[cmdhelp_ra_providers,show providers for a RA and a class]]
+==== `providers`
+
+List providers for a resource agent type. The class parameter
+defaults to `ocf`.
+
+Usage:
+...............
+providers <type> [<class>]
+...............
+Example:
+...............
+providers apache
+...............
+
+[[cmdhelp_ra_validate,validate parameters for RA]]
+==== `validate`
+
+If the resource agent supports the `validate-all` action, this calls
+the action with the given parameters, printing any warnings or errors
+reported by the agent.
+
+Usage:
+................
+validate <agent> [<key>=<value> ...]
+................
+
+[[cmdhelp_resource,Resource management]]
+=== `resource` - Resource management
+
+At this level resources may be managed.
+
+All (or almost all) commands are implemented with the CRM tools
+such as `crm_resource(8)`.
+
+[[cmdhelp_resource_ban,ban a resource from a node]]
+==== `ban`
+
+Ban a resource from running on a certain node. If no node is given
+as argument, the resource is banned from the current location.
+
+See `move` for details on other arguments.
+
+Usage:
+...............
+ban <rsc> [<node>] [<lifetime>] [force]
+...............
+
+[[cmdhelp_resource_cleanup,cleanup resource status]]
+==== `cleanup`
+
+If resource has any past failures, clear its history and fail
+count. Typically done after the resource has temporarily
+failed.
+
+If a node is omitted, cleanup on all nodes.
+
++(Pacemaker 1.1.14)+ Pass force to cleanup the resource itself,
+otherwise the cleanup command will apply to the parent resource (if
+any).
+
+Usage:
+...............
+cleanup [<rsc>] [<node>] [force]
+...............
+
+[[cmdhelp_resource_clear,Clear any relocation constraint]]
+==== `clear` (`unmove`, `unmigrate`, `unban`)
+
+Remove any relocation constraint created by
+the `move`, `migrate` or `ban` command.
+
+Usage:
+...............
+clear <rsc>
+unmigrate <rsc>
+unban <rsc>
+...............
+
+[[cmdhelp_resource_constraints,Show constraints affecting a resource]]
+==== `constraints`
+
+Display the location and colocation constraints affecting the
+resource.
+
+Usage:
+................
+constraints <rsc>
+................
+
+[[cmdhelp_resource_demote,demote a promotable resource]]
+==== `demote`
+
+Demote a promotable resource using the `target-role`
+attribute.
+
+Usage:
+...............
+demote <rsc>
+...............
+
+[[cmdhelp_resource_failcount,manage failcounts]]
+==== `failcount`
+
+Show/edit/delete the failcount of a resource.
+When `set` a non-zero value, `operation` and `interval` should be
+provided when multiple operation failcount entries exist.
+`interval` is a value in seconds.
+
+Usage:
+...............
+failcount <rsc> set <node> <value> [operation] [interval]
+failcount <rsc> delete <node>
+failcount <rsc> show <node>
+...............
+Example:
+...............
+failcount fs_0 delete node2
+...............
+
+[[cmdhelp_resource_locate,show the location of resources]]
+==== `locate`
+
+Show the current location of one or more resources.
+
+Usage:
+...............
+locate [<rsc> ...]
+...............
+
+[[cmdhelp_resource_maintenance,Enable/disable per-resource maintenance mode]]
+==== `maintenance`
+
+Enables or disables the per-resource maintenance mode. When this mode
+is enabled, no monitor operations will be triggered for the resource.
+`maintenance` attribute conflicts with the `is-managed`. When setting
+the `maintenance` attribute, the user is proposed to remove the
+`is-managed` attribute if it exists.
+
+Usage:
+..................
+maintenance <resource> [on|off|true|false]
+..................
+
+Example:
+..................
+maintenance rsc1
+maintenance rsc2 off
+..................
+
+[[cmdhelp_resource_manage,put a resource into managed mode]]
+==== `manage`
+
+Manage a resource using the `is-managed` attribute. If there
+are multiple meta attributes sets, the attribute is set in all of
+them. If the resource is a clone, all `is-managed` attributes are
+removed from the children resources.
+`is-managed` attribute conflicts with the `maintenance`. When setting
+the `is-managed` attribute, the user is proposed to remove the
+`maintenance` attribute if it exists.
+
+For details on group management see <<cmdhelp_options_manage-children,`options manage-children`>>.
+
+Usage:
+...............
+manage <rsc>
+...............
+
+[[cmdhelp_resource_meta,manage a meta attribute]]
+==== `meta`
+
+Show/edit/delete a meta attribute of a resource. Currently, all
+meta attributes of a resource may be managed with other commands
+such as `resource stop`.
+
+Usage:
+...............
+meta <rsc> set <attr> <value>
+meta <rsc> delete <attr>
+meta <rsc> show <attr>
+...............
+Example:
+...............
+meta ip_0 set target-role stopped
+...............
+
+[[cmdhelp_resource_move,Move a resource to another node]]
+==== `move` (`migrate`)
+
+Move a resource away from its current location.
+
+If the destination node is left out, the resource is migrated by
+creating a constraint which prevents it from running on the current
+node. For this type of constraint to be created, the +force+ argument
+is required.
+
+A lifetime may be given for the constraint. Once it expires, the
+location constraint will no longer be active.
+
+Usage:
+...............
+move <rsc> [<node>] [<lifetime>] [force]
+...............
+
+[[cmdhelp_resource_operations,Show active resource operations]]
+==== `operations`
+
+Show active operations, optionally filtered by resource and node.
+
+Usage:
+................
+operations [<rsc>] [<node>]
+................
+
+[[cmdhelp_resource_param,manage a parameter of a resource]]
+==== `param`
+
+Show/edit/delete a parameter of a resource.
+
+Usage:
+...............
+param <rsc> set <param> <value>
+param <rsc> delete <param>
+param <rsc> show <param>
+...............
+Example:
+...............
+param ip_0 show ip
+...............
+
+[[cmdhelp_resource_promote,promote a promotable resource]]
+==== `promote`
+
+Promote a promotable resource using the `target-role`
+attribute.
+
+Usage:
+...............
+promote <rsc>
+...............
+
+[[cmdhelp_resource_refresh,Recheck current resource status and drop failure history]]
+==== `refresh`
+
+Delete resource's history (including failures) so its current state is rechecked.
+
+Usage:
+...............
+refresh [<rsc>] [<node>] [force]
+...............
+
+[[cmdhelp_resource_restart,restart resources]]
+==== `restart`
+
+Restart one or more resources. This is essentially a shortcut for
+resource stop followed by a start. The shell is first going to wait
+for the stop to finish, that is for all resources to really stop, and
+only then to order the start action. Due to this command
+entailing a whole set of operations, informational messages are
+printed to let the user see some progress.
+
+For details on group management see
+<<cmdhelp_options_manage-children,`options manage-children`>>.
+
+Usage:
+...............
+restart <rsc> [<rsc> ...]
+...............
+Example:
+...............
+# crm resource restart g_webserver
+INFO: ordering g_webserver to stop
+waiting for stop to finish .... done
+INFO: ordering g_webserver to start
+#
+...............
+
+[[cmdhelp_resource_scores,Display resource scores]]
+==== `scores`
+
+Display the allocation scores for all resources.
+
+Usage:
+................
+scores
+................
+
+[[cmdhelp_resource_secret,manage sensitive parameters]]
+==== `secret`
+
+Sensitive parameters can be kept in local files rather than CIB
+in order to prevent accidental data exposure. Use the `secret`
+command to manage such parameters. `stash` and `unstash` move the
+value from the CIB and back to the CIB respectively. The `set`
+subcommand sets the parameter to the provided value. `delete`
+removes the parameter completely. `show` displays the value of
+the parameter from the local file. Use `check` to verify if the
+local file content is valid.
+
+Usage:
+...............
+secret <rsc> set <param> <value>
+secret <rsc> stash <param>
+secret <rsc> unstash <param>
+secret <rsc> delete <param>
+secret <rsc> show <param>
+secret <rsc> check <param>
+...............
+Example:
+...............
+secret fence_1 show password
+secret fence_1 stash password
+secret fence_1 set password secret_value
+...............
+
+[[cmdhelp_resource_start,start resources]]
+==== `start`
+
+Start one or more resources by setting the `target-role` attribute. If
+there are multiple meta attributes sets, the attribute is set in all
+of them. If the resource is a clone, all `target-role` attributes are
+removed from the children resources.
+
+For details on group management see
+<<cmdhelp_options_manage-children,`options manage-children`>>.
+
+Usage:
+...............
+start <rsc> [<rsc> ...]
+...............
+
+[[cmdhelp_resource_status,show status of resources]]
+==== `status` (`show`, `list`)
+
+Print resource status. More than one resource can be shown at once. If
+the resource parameter is left out, the status of all resources is
+printed.
+
+Usage:
+...............
+status [<rsc> ...]
+...............
+
+[[cmdhelp_resource_stop,stop resources]]
+==== `stop`
+
+Stop one or more resources using the `target-role` attribute. If there
+are multiple meta attributes sets, the attribute is set in all of
+them. If the resource is a clone, all `target-role` attributes are
+removed from the children resources.
+
+For details on group management see
+<<cmdhelp_options_manage-children,`options manage-children`>>.
+
+Usage:
+...............
+stop <rsc> [<rsc> ...]
+...............
+
+[[cmdhelp_resource_trace,start RA tracing]]
+==== `trace`
+
+Start tracing RA for the given operation. When `[<log-dir>]`
+is not specified the trace files are stored in `$HA_VARLIB/trace_ra`.
+If the operation to be traced is monitor, note that the number
+of trace files can grow very quickly.
+
+If no operation name is given, crmsh will attempt to trace all
+operations for the RA. This includes any configured operations, start
+and stop as well as promote/demote for multistate resources.
+
+To trace the probe operation which exists for all resources, either
+set a trace for `monitor` with interval `0`, or use `probe` as the
+operation name.
+
+Note: RA tracing is only supported by OCF resource agents;
+The pacemaker-execd daemon does not log recurring monitor operations
+unless an error occurred.
+
+Usage:
+...............
+trace <rsc> [<op> [<interval>] [<log-dir>]]
+...............
+Example:
+...............
+trace fs start
+trace webserver
+trace webserver probe
+trace fs monitor 0 /var/log/foo/bar
+...............
+
+[[cmdhelp_resource_unmanage,put a resource into unmanaged mode]]
+==== `unmanage`
+
+Unmanage a resource using the `is-managed` attribute. If there
+are multiple meta attributes sets, the attribute is set in all of
+them. If the resource is a clone, all `is-managed` attributes are
+removed from the children resources.
+
+For details on group management see <<cmdhelp_options_manage-children,`options manage-children`>>.
+
+Usage:
+...............
+unmanage <rsc>
+...............
+
+[[cmdhelp_resource_untrace,stop RA tracing]]
+==== `untrace`
+
+Stop tracing RA for the given operation. If no operation name is
+given, crmsh will attempt to stop tracing all operations in resource.
+
+Usage:
+...............
+untrace <rsc> [<op> [<interval>] ]
+...............
+Example:
+...............
+untrace fs start
+untrace webserver
+...............
+
+[[cmdhelp_resource_utilization,manage a utilization attribute]]
+==== `utilization`
+
+Show/edit/delete a utilization attribute of a resource. These
+attributes describe hardware requirements. By setting the
+`placement-strategy` cluster property appropriately, it is
+possible then to distribute resources based on resource
+requirements and node size. See also <<cmdhelp_node_utilization,node utilization attributes>>.
+
+Usage:
+...............
+utilization <rsc> set <attr> <value>
+utilization <rsc> delete <attr>
+utilization <rsc> show <attr>
+...............
+Example:
+...............
+utilization xen1 set memory 4096
+...............
+
+[[cmdhelp_node,Node management]]
+=== `node` - Node management
+
+Node management and status commands.
+
+[[cmdhelp_node_attribute,manage attributes]]
+==== `attribute`
+
+Edit node attributes. This kind of attribute should refer to
+relatively static properties, such as memory size.
+
+Usage:
+...............
+attribute <node> set <attr> <value>
+attribute <node> delete <attr>
+attribute <node> show <attr>
+...............
+Example:
+...............
+attribute node_1 set memory_size 4096
+...............
+
+[[cmdhelp_node_clearstate,Clear node state]]
+==== `clearstate`
+
+Resets and clears the state of the specified node. This node is
+afterwards assumed clean and offline. This command can be used to
+manually confirm that a node has been fenced (e.g., powered off).
+
+Be careful! This can cause data corruption if you confirm that a node is
+down that is, in fact, not cleanly down - the cluster will proceed as if
+the fence had succeeded, possibly starting resources multiple times.
+
+Usage:
+...............
+clearstate <node>
+...............
+
+[[cmdhelp_node_delete,delete node (deprecated)]]
+==== `delete`
+
+Remove a node from cluster.
+
+If the node is still listed as active and a member of our
+partition we refuse to remove it. With the global force option
+(`-F`) we will try to delete the node anyway.
+
+Usage:
+...............
+delete <node>
+...............
+
+.Deprecation note
+*****
+This command is deprecated and in favor of `crm cluster remove [-F] -c <node>`,
+which will adjust the related cluster configurations and clean up the leftover
+(eg. stopping the cluster services) on the removed node.
+*****
+
+[[cmdhelp_node_fence,fence node]]
+
+==== `fence`
+
+Make CRM fence a node. This functionality depends on stonith
+resources capable of fencing the specified node. No such stonith
+resources, no fencing will happen.
+
+Usage:
+...............
+fence <node>
+...............
+
+[[cmdhelp_node_maintenance,put node into maintenance mode]]
+==== `maintenance`
+
+Set the node status to maintenance. This is equivalent to the
+cluster-wide `maintenance-mode` property but puts just one node
+into the maintenance mode. If there are maintained resources on
+the node, the user will be proposed to remove the maintenance
+property from them.
+
+The node parameter defaults to the node where the command is run.
+
+Usage:
+...............
+maintenance [<node>]
+...............
+
+[[cmdhelp_node_online,set node online,From Code]]
+==== `online`
+See "crm node help online" or "crm node online --help"
+
+[[cmdhelp_node_ready,put node into ready mode]]
+==== `ready`
+
+Set the node's maintenance status to `off`. The node should be
+now again fully operational and capable of running resource
+operations.
+
+The node parameter defaults to the node where the command is run.
+
+Usage:
+...............
+ready [<node>]
+...............
+
+[[cmdhelp_node_server,show node hostname or server address]]
+==== `server`
+
+Remote nodes may have a configured server address which should
+be used when contacting the node. This command prints the
+server address if configured, else the node name.
+
+If no parameter is given, the addresses or names for all nodes
+are printed.
+
+Usage:
+...............
+server [<node> ...]
+...............
+
+[[cmdhelp_node_show,show node]]
+==== `show`
+
+Show a node definition. If the node parameter is omitted, then all
+nodes are shown.
+
+Usage:
+...............
+show [<node>]
+...............
+
+[[cmdhelp_node_standby,put node into standby,From Code]]
+==== `standby`
+See "crm node help standby" or "crm node standby --help"
+
+[[cmdhelp_node_status-attr,manage status attributes]]
+==== `status-attr`
+
+Edit node attributes which are in the CIB status section, i.e.,
+attributes which hold properties of a more volatile nature. One
+typical example is attribute generated by the `pingd` utility.
+
+Usage:
+...............
+status-attr <node> set <attr> <value>
+status-attr <node> delete <attr>
+status-attr <node> show <attr>
+...............
+Example:
+...............
+status-attr node_1 show pingd
+...............
+
+[[cmdhelp_node_utilization,manage utilization attributes]]
+==== `utilization`
+
+Edit node utilization attributes. These attributes describe
+hardware characteristics as integer numbers such as memory size
+or the number of CPUs. By setting the `placement-strategy`
+cluster property appropriately, it is possible then to distribute
+resources based on resource requirements and node size. See also
+<<cmdhelp_resource_utilization,resource utilization attributes>>.
+
+Usage:
+...............
+utilization <node> set <attr> <value>
+utilization <node> delete <attr>
+utilization <node> show <attr>
+...............
+Examples:
+...............
+utilization node_1 set memory 16384
+utilization node_1 show cpu
+...............
+
+[[cmdhelp_site,GEO clustering site support]]
+=== `site` - GEO clustering site support
+
+A cluster may consist of two or more subclusters in different and
+distant locations. This set of commands supports such setups.
+
+[[cmdhelp_site_ticket,manage site tickets]]
+==== `ticket`
+
+Tickets are cluster-wide attributes. They can be managed at the
+site where this command is executed.
+
+It is then possible to constrain resources depending on the
+ticket availability (see the <<cmdhelp_configure_rsc_ticket,`rsc_ticket`>> command
+for more details).
+
+Usage:
+...............
+ticket {grant|revoke|standby|activate|show|time|delete} <ticket>
+...............
+Example:
+...............
+ticket grant ticket1
+...............
+
+[[cmdhelp_options,User preferences]]
+=== `options` - User preferences
+
+The user may set various options for the crm shell itself.
+
+[[cmdhelp_options_add-quotes,add quotes around parameters containing spaces]]
+==== `add-quotes`
+
+The shell (as in `/bin/sh`) parser strips quotes from the command
+line. This may sometimes make it really difficult to type values
+which contain white space. One typical example is the configure
+filter command. The crm shell will supply extra quotes around
+arguments which contain white space. The default is `yes`.
+
+.Note on quotes use
+****************************
+Adding quotes around arguments automatically has been introduced
+with version 1.2.2 and it is technically a regression. Being a
+regression is the only reason the `add-quotes` option exists. If
+you have custom shell scripts which would break, just set the
+`add-quotes` option to `no`.
+
+For instance, with adding quotes enabled, it is possible to do
+the following:
+...............
+# crm configure primitive d1 Dummy \
+ meta description="some description here"
+# crm configure filter 'sed "s/hostlist=./&node-c /"' fencing
+...............
+****************************
+
+[[cmdhelp_options_check-frequency,when to perform semantic check]]
+==== `check-frequency`
+
+Semantic check of the CIB or elements modified or created may be
+done on every configuration change (`always`), when verifying
+(`on-verify`) or `never`. It is by default set to `always`.
+Experts may want to change the setting to `on-verify`.
+
+The checks require that resource agents are present. If they are
+not installed at the configuration time set this preference to
+`never`.
+
+See <<topics_Features_Checks,Configuration semantic checks>> for more details.
+
+[[cmdhelp_options_check-mode,how to treat semantic errors]]
+==== `check-mode`
+
+Semantic check of the CIB or elements modified or created may be
+done in the `strict` mode or in the `relaxed` mode. In the former
+certain problems are treated as configuration errors. In the
+`relaxed` mode all are treated as warnings. The default is `strict`.
+
+See <<topics_Features_Checks,Configuration semantic checks>> for more details.
+
+[[cmdhelp_options_colorscheme,set colors for output]]
+==== `colorscheme`
+
+With `output` set to `color`, a comma separated list of colors
+from this option are used to emphasize:
+
+- keywords
+- object ids
+- attribute names
+- attribute values
+- scores
+- resource references
+
+`crm` can show colors only if there is curses support for python
+installed (usually provided by the `python-curses` package). The
+colors are whatever is available in your terminal. Use `normal`
+if you want to keep the default foreground color.
+
+This user preference defaults to
+`yellow,normal,cyan,red,green,magenta` which is good for
+terminals with dark background. You may want to change the color
+scheme and save it in the preferences file for other color
+setups.
+
+Example:
+...............
+colorscheme yellow,normal,blue,red,green,magenta
+...............
+
+[[cmdhelp_options_editor,set preferred editor program]]
+==== `editor`
+
+The `edit` command invokes an editor. Use this to specify your
+preferred editor program. If not set, it will default to either
+the value of the `EDITOR` environment variable or to one of the
+standard UNIX editors (`vi`,`emacs`,`nano`).
+
+Usage:
+...............
+editor program
+...............
+Example:
+...............
+editor vim
+...............
+
+[[cmdhelp_options_manage-children,how to handle children resource attributes]]
+==== `manage-children`
+
+Some resource management commands, such as `resource stop`, when
+the target resource is a group, may not always produce desired
+result. Each element, group and the primitive members, can have a
+meta-attribute and those attributes may end up with conflicting
+values. Consider the following construct:
+...............
+crm(live)# configure show svc fs virtual-ip
+primitive fs Filesystem \
+ params device="/dev/drbd0" directory="/srv/nfs" fstype=ext3 \
+ op monitor interval=10s \
+ meta target-role=Started
+primitive virtual-ip IPaddr2 \
+ params ip=10.2.13.110 iflabel=1 \
+ op monitor interval=10s \
+ op start interval=0 \
+ meta target-role=Started
+group svc fs virtual-ip \
+ meta target-role=Stopped
+...............
+
+Even though the element +svc+ should be stopped, the group is
+actually running because all its members have the +target-role+
+set to +Started+:
+...............
+crm(live)# resource show svc
+resource svc is running on: xen-f
+...............
+
+Hence, if the user invokes +resource stop svc+ the intention is
+not clear. This preference gives the user an opportunity to
+better control what happens if attributes of group members have
+values which are in conflict with the same attribute of the group
+itself.
+
+Possible values are +ask+ (the default), +always+, and +never+.
+If set to +always+, the crm shell removes all children attributes
+which have values different from the parent. If set to +never+,
+all children attributes are left intact. Finally, if set to
++ask+, the user will be asked for each member what is to be done.
+
+[[cmdhelp_options_output,set output type]]
+==== `output`
+
+`crm` can adorn configurations in two ways: in color (similar to
+for instance the `ls --color` command) and by showing keywords in
+upper case. Possible values are `plain`, `color-always`, `color`,
+and 'uppercase'. It is possible to combine `uppercase` with one
+of the color values in order to get an upper case xmass tree. Just
+set this option to `color,uppercase` or `color-always,uppercase`.
+In case you need color codes in pipes, `color-always` forces color
+codes even in case the terminal is not a tty (just like `ls
+--color=always`).
+
+[[cmdhelp_options_pager,set preferred pager program]]
+==== `pager`
+
+The `view` command displays text through a pager. Use this to
+specify your preferred pager program. If not set, it will default
+to either the value of the `PAGER` environment variable or to one
+of the standard UNIX system pagers (`less`,`more`,`pg`).
+
+[[cmdhelp_options_reset,reset user preferences to factory defaults]]
+==== `reset`
+
+This command resets all user options to the defaults. If used as
+a single-shot command, the rc file (+$HOME/.config/crm/rc+) is
+reset to the defaults too.
+
+[[cmdhelp_options_save,save the user preferences to the rc file]]
+==== `save`
+
+Save current settings to the rc file (+$HOME/.config/crm/rc+). On
+further `crm` runs, the rc file is automatically read and parsed.
+
+[[cmdhelp_options_set,Set the value of a given option]]
+==== `set`
+
+Sets the value of an option. Takes the fully qualified
+name of the option as argument, as displayed by +show all+.
+
+The modified option value is stored in the user-local
+configuration file, usually found in +~/.config/crm/crm.conf+.
+
+Usage:
+........
+set <option> <value>
+........
+
+Example:
+........
+set color.warn "magenta bold"
+set editor nano
+........
+
+[[cmdhelp_options_show,show current user preference]]
+==== `show`
+
+Display all current settings.
+
+Given an option name as argument, `show` will display only the value
+of that argument.
+
+Given +all+ as argument, `show` displays all available user options.
+
+Usage:
+........
+show [all|<option>]
+........
+
+Example:
+........
+show
+show skill-level
+show all
+........
+
+[[cmdhelp_options_skill-level,set skill level]]
+==== `skill-level`
+
+Based on the skill-level setting, the user is allowed to use only
+a subset of commands. There are three levels: operator,
+administrator, and expert. The operator level allows only
+commands at the `resource` and `node` levels, but not editing
+or deleting resources. The administrator may do that and may also
+configure the cluster at the `configure` level and manage the
+shadow CIBs. The expert may do all.
+
+Usage:
+...............
+skill-level <level>
+
+level :: operator | administrator | expert
+...............
+
+.Note on security
+****************************
+The `skill-level` option is advisory only. There is nothing
+stopping any users change their skill level (see
+<<topics_Features_Security,Access Control Lists (ACL)>> on how to enforce
+access control).
+****************************
+
+[[cmdhelp_options_sort-elements,sort CIB elements]]
+==== `sort-elements`
+
+`crm` by default sorts CIB elements. If you want them appear in
+the order they were created, set this option to `no`.
+
+Usage:
+...............
+sort-elements {yes|no}
+...............
+Example:
+...............
+sort-elements no
+...............
+
+[[cmdhelp_options_user,set the cluster user]]
+==== `user`
+
+Sufficient privileges are necessary in order to manage a
+cluster: programs such as `crm_verify` or `crm_resource` and,
+ultimately, `cibadmin` have to be run either as `root` or as the
+CRM owner user (typically `hacluster`). You don't have to worry
+about that if you run `crm` as `root`. A more secure way is to
+run the program with your usual privileges, set this option to
+the appropriate user (such as `hacluster`), and setup the
+`sudoers` file.
+
+Usage:
+...............
+user system-user
+...............
+Example:
+...............
+user hacluster
+...............
+
+[[cmdhelp_options_wait,synchronous operation]]
+==== `wait`
+
+In normal operation, `crm` runs a command and gets back
+immediately to process other commands or get input from the user.
+With this option set to `yes` it will wait for the started
+transition to finish. In interactive mode dots are printed to
+indicate progress.
+
+Usage:
+...............
+wait {yes|no}
+...............
+Example:
+...............
+wait yes
+...............
+
+[[cmdhelp_configure,CIB configuration]]
+=== `configure` - CIB configuration
+
+This level enables all CIB object definition commands.
+
+The configuration may be logically divided into four parts:
+nodes, resources, constraints, and (cluster) properties and
+attributes. Each of these commands support one or more basic CIB
+objects.
+
+Nodes and attributes describing nodes are managed using the
+`node` command.
+
+Commands for resources are:
+
+- `primitive`
+- `monitor`
+- `group`
+- `clone` (promotable clones)
+- `ms`/`master` (master-slave) (deprecated)
+
+In order to streamline large configurations, it is possible to
+define a template which can later be referenced in primitives:
+
+- `rsc_template`
+
+In that case the primitive inherits all attributes defined in the
+template.
+
+There are three types of constraints:
+
+- `location`
+- `colocation`
+- `order`
+
+It is possible to define fencing order (stonith resource
+priorities):
+
+- `fencing_topology`
+
+Finally, there are the cluster properties, resource meta-attributes
+defaults, and operations defaults. All are just a set of attributes.
+These attributes are managed by the following commands:
+
+- `property`
+- `rsc_defaults`
+- `op_defaults`
+
+In addition to the cluster configuration, the Access Control
+Lists (ACL) can be setup to allow access to parts of the CIB for
+users other than +root+ and +hacluster+. The following commands
+manage ACL:
+
+- `user`
+- `role`
+
+In Pacemaker 1.1.12 and up, this command replaces the `user` command
+for handling ACLs:
+
+- `acl_target`
+
+The changes are applied to the current CIB only on ending the
+configuration session or using the `commit` command.
+
+Comments start with +#+ in the first line. The comments are tied
+to the element which follows. If the element moves, its comments
+will follow.
+
+[[cmdhelp_configure_acl_target,Define target access rights]]
+==== `acl_target`
+
+Defines an ACL target.
+
+Usage:
+................
+acl_target <tid> [<role> ...]
+................
+Example:
+................
+acl_target joe resource_admin constraint_editor
+................
+
+[[cmdhelp_configure_alert,Event-driven alerts]]
+==== `alert`
+
+.Version note
+****************************
+This feature is only available
+in Pacemaker 1.1.15+.
+****************************
+
+Event-driven alerts enables calling scripts whenever interesting
+events occur in the cluster (nodes joining or leaving, resources
+starting or stopping, etc.).
+
+The +path+ is an arbitrary file path to an alert script. Existing
+external scripts used with ClusterMon resources can be used as alert
+scripts, since the interface is compatible.
+
+Each alert may have a number of recipients configured. These will be
+passed to the script as arguments. The first recipient will also be
+passed as the +CRM_alert_recipient+ environment variable, for
+compatibility with existing scripts that only support one recipient.
+
+The available meta-attributes are +timeout+ (default 30s) and
++timestamp-format+ (default `"%H:%M:%S.%06N"`).
+
+Some configurations may require each recipient to be delimited by
+brackets, to avoid ambiguity. In the example +alert-2+ below, the meta
+attribute for `timeout` is defined after the recipient, so the
+brackets are used to ensure that the meta attribute is set for the
+alert and not just the recipient. This can be avoided by setting any
+alert attributes before defining the recipients.
+
+Usage:
+...............
+alert <id> <path> \
+ [attributes <nvpair> ...] \
+ [meta <nvpair> ...] \
+ [select [nodes | fencing | resources | attributes '{' <attribute> ... '}' ] ...] \
+ [to [{] <recipient>
+ [attributes <nvpair> ...] \
+ [meta <nvpair> ...] [}] \
+ ...]
+...............
+
+Example:
+...............
+alert alert-1 /srv/pacemaker/pcmk_alert_sample.sh \
+ to /var/log/cluster-alerts.log
+
+alert alert-2 /srv/pacemaker/example_alert.sh \
+ meta timeout=60s \
+ to { /var/log/cluster-alerts.log }
+
+alert alert-3 /srv/pacemaker/example_alert.sh \
+ select fencing \
+ to { /var/log/fencing-alerts.log }
+
+...............
+
+[[cmdhelp_configure_bundle,Container bundle]]
+==== `bundle`
+
+A bundle is a single resource specifying the settings, networking
+requirements, and storage requirements for any number of containers
+generated from the same container image.
+
+Pacemaker bundles support Docker (since version 1.1.17) and rkt (since
+version 1.1.18) container technologies.
+
+A bundle must contain exactly one +docker+ or +rkt+ element.
+
+The bundle definition may contain a reference to a primitive
+resource which defining the resource running inside the
+container.
+
+Example:
+...............
+
+primitive httpd-apache ocf:heartbeat:apache
+
+bundle httpd \
+ docker image=pcmk:httpd replicas=3 \
+ network ip-range-start=10.10.10.123 host-netmask=24 \
+ port-mapping port=80 \
+ storage \
+ storage-mapping target-dir=/var/www/html source-dir=/srv/www options=rw \
+ primitive httpd-apache
+
+...............
+
+[[cmdhelp_configure_cib,CIB shadow management]]
+==== `cib`
+
+This level is for management of shadow CIBs. It is available at
+the `configure` level to enable saving intermediate changes to a
+shadow CIB instead of to the live cluster. This short excerpt
+shows how:
+...............
+crm(live)configure# cib new test-2
+INFO: test-2 shadow CIB created
+crm(test-2)configure# commit
+...............
+Note how the current CIB in the prompt changed from +live+ to
++test-2+ after issuing the `cib new` command. See also the
+<<cmdhelp_cib,CIB shadow management>> for more information.
+
+[[cmdhelp_configure_cibstatus,CIB status management and editing]]
+==== `cibstatus`
+
+Enter edit and manage the CIB status section level. See the
+<<cmdhelp_cibstatus,CIB status management section>>.
+
+[[cmdhelp_configure_clone,define a clone]]
+==== `clone`
+
+The `clone` command creates a resource clone. It may contain a
+single primitive resource or one group of resources.
+
++Promotable clones+ are clone resources with the +promotable=true+ meta-attribute for the given promotable resources.
+It's used to deprecate the master-slave resources.
+
+Usage:
+...............
+clone <name> <rsc>
+ [description=<description>]
+ [meta <attr_list>]
+ [params <attr_list>]
+
+attr_list :: [$id=<id>] <attr>=<val> [<attr>=<val>...] | $id-ref=<id>
+...............
+Example:
+...............
+clone cl_fence apc_1 \
+ meta clone-node-max=1 globally-unique=false
+
+clone disk1 drbd1 \
+ meta promotable=true notify=true globally-unique=false
+...............
+
+[[cmdhelp_configure_colocation,colocate resources]]
+==== `colocation` (`collocation`)
+
+This constraint expresses the placement relation between two
+or more resources. If there are more than two resources, then the
+constraint is called a resource set.
+
+The score is used to indicate the priority of the constraint. A
+positive score indicates that the resources should run on the same
+node. A negative score that they should not run on the same
+node. Values of positive or negative +infinity+ indicate a mandatory
+constraint.
+
+In the two resource form, the cluster will place +<with-rsc>+ first,
+and then decide where to put the +<rsc>+ resource.
+
+Collocation resource sets have an extra attribute (+sequential+)
+to allow for sets of resources which don't depend on each other
+in terms of state. The shell syntax for such sets is to put
+resources in parentheses.
+
+Sets cannot be nested.
+
+The optional +node-attribute+ can be used to colocate resources on a
+set of nodes and not necessarily on the same node. For example, by
+setting a node attribute +color+ on all nodes and setting the
++node-attribute+ value to +color+ as well, the colocated resources
+will be placed on any node that has the same color.
+
+For more details on how to configure resource sets, see
+<<topics_Features_Resourcesets,`Syntax: Resource sets`>>.
+
+Usage:
+...............
+colocation <id> <score>: <rsc>[:<role>] <with-rsc>[:<role>]
+ [node-attribute=<node_attr>]
+
+colocation <id> <score>: <resource_sets>
+ [node-attribute=<node_attr>]
+
+resource_sets :: <resource_set> [<resource_set> ...]
+
+resource_set :: ["("|"["] <rsc>[:<role>] [<rsc>[:<role>] ...] \
+ [<attributes>] [")"|"]"]
+
+attributes :: [require-all=(true|false)] [sequential=(true|false)]
+
+...............
+Example:
+...............
+colocation never_put_apache_with_dummy -inf: apache dummy
+colocation c1 inf: A ( B C )
+...............
+
+[[cmdhelp_configure_commit,commit the changes to the CIB]]
+==== `commit`
+
+Commit the current configuration to the CIB in use. As noted
+elsewhere, commands in a configure session don't have immediate
+effect on the CIB. All changes are applied at one point in time,
+either using `commit` or when the user leaves the configure
+level. In case the CIB in use changed in the meantime, presumably
+by somebody else, the crm shell will refuse to apply the changes.
+
+If you know that it's fine to still apply them, add +force+ to the
+command line.
+
+To disable CIB patching and apply the changes by replacing the CIB
+completely, add +replace+ to the command line. Note that this can lead
+to previous changes being overwritten if some other process
+concurrently modifies the CIB.
+
+Usage:
+...............
+commit [force] [replace]
+...............
+
+[[cmdhelp_configure_default-timeouts,set timeouts for operations to minimums from the meta-data]]
+==== `default-timeouts`
+
+This command takes the timeouts from the actions section of the
+resource agent meta-data and sets them for the operations of the
+primitive.
+
+Usage:
+...............
+default-timeouts <id> [<id>...]
+...............
+
+.Note on `default-timeouts`
+****************************
+The use of this command is discouraged in favor of manually
+determining the best timeouts required for the particular
+configuration. Relying on the resource agent to supply appropriate
+timeouts can cause the resource to fail at the worst possible moment.
+
+Appropriate timeouts for resource actions are context-sensitive, and
+should be carefully considered with the whole configuration in mind.
+****************************
+
+[[cmdhelp_configure_delete,delete CIB objects]]
+==== `delete`
+
+Delete one or more objects. If an object to be deleted belongs to
+a container object, such as a group, and it is the only resource
+in that container, then the container is deleted as well. Any
+related constraints are removed as well.
+
+If the object is a started resource, it will not be deleted unless the
++--force+ flag is passed to the command, or the +force+ option is set.
+
+Usage:
+...............
+delete [--force] <id> [<id>...]
+...............
+
+[[cmdhelp_configure_edit,edit CIB objects]]
+==== `edit`
+
+This command invokes the editor with the object description. As
+with the `show` command, the user may choose to edit all objects
+or a set of objects.
+
+If the user insists, he or she may edit the XML edition of the
+object. If you do that, don't modify any id attributes.
+
+Usage:
+...............
+edit [xml] [<id> ...]
+edit [xml] changed
+...............
+
+.Note on renaming element ids
+****************************
+The edit command sometimes cannot properly handle modifying
+element ids. In particular for elements which belong to group or
+ms resources. Group and ms resources themselves also cannot be
+renamed. Please use the `rename` command instead.
+****************************
+
+[[cmdhelp_configure_erase,erase the CIB]]
+==== `erase`
+
+.Deprecation note
+****************************
+`crm configure erase` is deprecated.
+The replacement could be `crm cluster remove [node]`
+****************************
+
+The `erase` clears all configuration. Apart from nodes. To remove
+nodes, you have to specify an additional keyword `nodes`.
+
+Note that removing nodes from the live cluster may have some
+strange/interesting/unwelcome effects.
+
+Usage:
+...............
+erase [nodes]
+...............
+
+[[cmdhelp_configure_fencing_topology,node fencing order]]
+==== `fencing_topology`
+
+If multiple fencing (stonith) devices are available capable of
+fencing a node, their order may be specified by +fencing_topology+.
+The order is specified per node.
+
+Stonith resources can be separated by +,+ in which case all of
+them need to succeed. If they fail, the next stonith resource (or
+set of resources) is used. In other words, use comma to separate
+resources which all need to succeed and whitespace for serial
+order. It is not allowed to use whitespace around comma.
+
+If the node is left out, the order is used for all nodes.
+That should reduce the configuration size in some stonith setups.
+
+From Pacemaker version 1.1.14, it is possible to use a node attribute
+as the +target+ in a fencing topology. The syntax for this usage is
+described below.
+
+From Pacemaker version 1.1.14, it is also possible to use regular
+expression patterns as the +target+ in a fencing topology. The configured
+fencing sequence then applies to all devices matching the pattern.
+
+Usage:
+...............
+fencing_topology <stonith_resources> [<stonith_resources> ...]
+fencing_topology <fencing_order> [<fencing_order> ...]
+
+fencing_order :: <target> <stonith_resources> [<stonith_resources> ...]
+
+stonith_resources :: <rsc>[,<rsc>...]
+target :: <node>: | attr:<node-attribute>=<value> | pattern:<pattern>
+...............
+Example:
+...............
+# Only kill the power if poison-pill fails
+fencing_topology poison-pill power
+
+# As above for node-a, but a different strategy for node-b
+fencing_topology \
+ node-a: poison-pill power \
+ node-b: ipmi serial
+
+# Fencing anything on rack 1 requires fencing via both APC 1 and 2,
+# to defeat the redundancy provided by two separate UPS units.
+fencing_topology attr:rack=1 apc01,apc02
+
+# Fencing for all machines named green.* is done using the pear
+# fencing device first, while all machines named red.* are fenced
+# using the apple fencing device first.
+fencing_topology \
+ pattern:green.* pear apple \
+ pattern:red.* apple pear
+...............
+
+[[cmdhelp_configure_filter,filter CIB objects]]
+==== `filter`
+
+This command filters the given CIB elements through an external
+program. The program should accept input on `stdin` and send
+output to `stdout` (the standard UNIX filter conventions). As
+with the `show` command, the user may choose to filter all or
+just a subset of elements.
+
+It is possible to filter the XML representation of objects, but
+probably not as useful as the configuration language. The
+presentation is somewhat different from what would be displayed
+by the `show` command---each element is shown on a single line,
+i.e., there are no backslashes and no other embellishments.
+
+Don't forget to put quotes around the filter if it contains
+spaces.
+
+Usage:
+...............
+filter <prog> [xml] [<id> ...]
+filter <prog> [xml] changed
+...............
+Examples:
+...............
+filter "sed '/^primitive/s/target-role=[^ ]*//'"
+# crm configure filter "sed '/^primitive/s/target-role=[^ ]*//'"
+crm configure <<END
+ filter "sed '/threshold=\"1\"/s/=\"1\"/=\"0\"/g'"
+END
+...............
+
+.Note on quotation marks
+**************************
+Filter commands which feature a blend of quotation marks can be
+difficult to get right, especially when used directly from bash, since
+bash does its own quotation parsing. In these cases, it can be easier
+to supply the filter command as standard input. See the last example
+above.
+**************************
+
+[[cmdhelp_configure_get_property,Get property value]]
+==== `get-property`
+
+Show the value of the given property. If the value is not set, the
+command will print the default value for the property, if known.
+
+If no property name is passed to the command, the list of known
+cluster properties is printed.
+
+If the property is set multiple times, for example using multiple
+property sets with different rule expressions, the output of this
+command is undefined.
+
+Pass the argument +-t+ or +--true+ to `get-property` to translate
+the argument value into +true+ or +false+. If the value is not
+set, the command will print +false+.
+
+Usage:
+...............
+get-property [-t|--true] [<name>]
+...............
+
+Example:
+...............
+get-property stonith-enabled
+get-property -t maintenance-mode
+...............
+
+[[cmdhelp_configure_graph,generate a directed graph]]
+==== `graph`
+
+Create a graphviz graphical layout from the current cluster
+configuration.
+
+Currently, only `dot` (directed graph) is supported. It is
+essentially a visualization of resource ordering.
+
+The graph may be saved to a file which can be used as source for
+various graphviz tools (by default it is displayed in the user's
+X11 session). Optionally, by specifying the format, one can also
+produce an image instead.
+
+For more or different graphviz attributes, it is possible to save
+the default set of attributes to an ini file. If this file exists
+it will always override the builtin settings. The +exportsettings+
+subcommand also prints the location of the ini file.
+
+Usage:
+...............
+graph [<gtype> [<file> [<img_format>]]]
+graph exportsettings
+
+gtype :: dot
+img_format :: `dot` output format (see the +-T+ option)
+...............
+Example:
+...............
+graph dot
+graph dot clu1.conf.dot
+graph dot clu1.conf.svg svg
+...............
+
+[[cmdhelp_configure_group,define a group]]
+==== `group`
+
+The `group` command creates a group of resources. This can be useful
+when resources depend on other resources and require that those
+resources start in order on the same node. A common use of resource
+groups is to ensure that a server and a virtual IP are located
+together, and that the virtual IP is started before the server.
+
+Grouped resources are started in the order they appear in the group,
+and stopped in the reverse order. If a resource in the group cannot
+run anywhere, resources following it in the group will not start.
+
+`group` can be passed the "container" meta attribute, to indicate that
+it is to be used to group VM resources monitored using Nagios. The
+resource referred to by the container attribute must be of type
+`ocf:heartbeat:Xen`, `ocf:heartbeat:VirtualDomain` or `ocf:heartbeat:lxc`.
+
+Usage:
+...............
+group <name> <rsc> [<rsc>...]
+ [description=<description>]
+ [meta attr_list]
+ [params attr_list]
+
+attr_list :: [$id=<id>] <attr>=<val> [<attr>=<val>...] | $id-ref=<id>
+...............
+Example:
+...............
+group internal_www disk0 fs0 internal_ip apache \
+ meta target_role=stopped
+
+group vm-and-services vm vm-sshd meta container="vm"
+...............
+
+[[cmdhelp_configure_load,import the CIB from a file]]
+==== `load`
+
+Load a part of configuration (or all of it) from a local file or
+a network URL. The +replace+ method replaces the current
+configuration with the one from the source. The +update+ method
+tries to import the contents into the current configuration. The
++push+ method imports the contents into the current configuration
+and removes any lines that are not present in the given
+configuration.
+The file may be a CLI file or an XML file.
+
+If the URL is `-`, the configuration is read from standard input.
+
+Usage:
+...............
+load [xml] <method> URL
+
+method :: replace | update | push
+...............
+Example:
+...............
+load xml update myfirstcib.xml
+load xml replace http://storage.big.com/cibs/bigcib.xml
+load xml push smallcib.xml
+...............
+
+[[cmdhelp_configure_location,a location preference]]
+==== `location`
+
+`location` defines the preference of nodes for the given
+resource. The location constraints consist of one or more rules
+which specify a score to be awarded if the rule matches.
+
+The resource referenced by the location constraint can be one of the
+following:
+
+* Plain resource reference: +location loc1 webserver 100: node1+
+* Resource set in curly brackets: +location loc1 { virtual-ip webserver } 100: node1+
+* Tag containing resource ids: +location loc1 tag1 100: node1+
+* Resource pattern: +location loc1 /web.*/ 100: node1+
+
+The +resource-discovery+ attribute allows probes to be selectively
+enabled or disabled per resource and node.
+
+The syntax for resource sets is described in detail for
+<<cmdhelp_configure_colocation,`colocation`>>.
+
+For more details on how to configure resource sets, see
+<<topics_Features_Resourcesets,`Syntax: Resource sets`>>.
+
+For more information on rule expressions, see
+<<topics_Syntax_RuleExpressions,Syntax: Rule expressions>>.
+
+Usage:
+...............
+location <id> <rsc> [<attributes>] {<node_pref>|<rules>}
+
+rsc :: /<rsc-pattern>/
+ | { resource_sets }
+ | <rsc>
+
+attributes :: role=<role> | resource-discovery=always|never|exclusive
+
+node_pref :: <score>: <node>
+
+rules ::
+ rule [id_spec] [$role=<role>] <score>: <expression>
+ [rule [id_spec] [$role=<role>] <score>: <expression> ...]
+
+id_spec :: $id=<id> | $id-ref=<id>
+score :: <number> | <attribute> | [-]inf
+expression :: <simple_exp> [<bool_op> <simple_exp> ...]
+bool_op :: or | and
+simple_exp :: <attribute> [type:]<binary_op> <value>
+ | <unary_op> <attribute>
+ | date <date_expr>
+type :: string | version | number
+binary_op :: lt | gt | lte | gte | eq | ne
+unary_op :: defined | not_defined
+
+date_expr :: lt <end>
+ | gt <start>
+ | in start=<start> end=<end>
+ | in start=<start> <duration>
+ | spec <date_spec>
+duration|date_spec ::
+ hours=<value>
+ | monthdays=<value>
+ | weekdays=<value>
+ | yearsdays=<value>
+ | months=<value>
+ | weeks=<value>
+ | years=<value>
+ | weekyears=<value>
+ | moon=<value>
+...............
+Examples:
+...............
+location conn_1 internal_www 100: node1
+
+location conn_1 internal_www \
+ rule 50: #uname eq node1 \
+ rule pingd: defined pingd
+
+location conn_2 dummy_float \
+ rule -inf: not_defined pingd or pingd number:lte 0
+
+# never probe for rsc1 on node1
+location no-probe rsc1 resource-discovery=never -inf: node1
+...............
+
+[[cmdhelp_configure_modgroup,modify group]]
+==== `modgroup`
+
+Add or remove primitives in a group. The `add` subcommand appends
+the new group member by default. Should it go elsewhere, there
+are `after` and `before` clauses.
+
+Usage:
+...............
+modgroup <id> add <id> [after <id>|before <id>]
+modgroup <id> remove <id>
+...............
+Examples:
+...............
+modgroup share1 add storage2 before share1-fs
+...............
+
+[[cmdhelp_configure_monitor,add monitor operation to a primitive]]
+==== `monitor`
+
+Monitor is by far the most common operation. It is possible to
+add it without editing the whole resource. Also, long primitive
+definitions may be a bit uncluttered. In order to make this
+command as concise as possible, less common operation attributes
+are not available. If you need them, then use the `op` part of
+the `primitive` command.
+
+Usage:
+...............
+monitor <rsc>[:<role>] <interval>[:<timeout>]
+...............
+Example:
+...............
+monitor apcfence 60m:60s
+...............
+
+Note that after executing the command, the monitor operation may
+be shown as part of the primitive definition.
+
+[[cmdhelp_configure_ms,define a master-slave resource (deprecated)]]
+==== `ms` (`master`)
+
+The `ms` command creates a master/slave resource type. It may contain a
+single primitive resource or one group of resources.
+
+Usage:
+...............
+ms <name> <rsc>
+ [description=<description>]
+ [meta attr_list]
+ [params attr_list]
+
+attr_list :: [$id=<id>] <attr>=<val> [<attr>=<val>...] | $id-ref=<id>
+...............
+Example:
+...............
+ms disk1 drbd1 \
+ meta notify=true globally-unique=false
+...............
+
+.Note on `ms` deprecated
+****************************
+From Pacemaker-2.0, the resource type referred to as "master/slave",
+"stateful", or "multi-state" is no longer a separate resource type,
+but a variation of clone now referred to as a "promotable clone".
+For backward compatibility, above configurations are also accepted.
+...............
+clone disk1 drbd1 \
+ meta promotable=true notify=true globally-unique=false
+...............
+****************************
+
+.Note on `id-ref` usage
+****************************
+Instance or meta attributes (`params` and `meta`) may contain
+a reference to another set of attributes. In that case, no other
+attributes are allowed. Since attribute sets' ids, though they do
+exist, are not shown in the `crm`, it is also possible to
+reference an object instead of an attribute set. `crm` will
+automatically replace such a reference with the right id:
+
+...............
+crm(live)configure# primitive a2 www-2 meta $id-ref=a1
+crm(live)configure# show a2
+primitive a2 apache \
+ meta $id-ref=a1-meta_attributes
+ [...]
+...............
+It is advisable to give meaningful names to attribute sets which
+are going to be referenced.
+****************************
+
+[[cmdhelp_configure_node,define a cluster node]]
+==== `node`
+
+The node command describes a cluster node. Nodes in the CIB are
+commonly created automatically by the CRM. Hence, you should not
+need to deal with nodes unless you also want to define node
+attributes. Note that it is also possible to manage node
+attributes at the `node` level.
+
+Usage:
+...............
+node [$id=<id>] <uname>[:<type>]
+ [description=<description>]
+ [attributes [$id=<id>] [<score>:] [rule...]
+ <param>=<value> [<param>=<value>...]] | $id-ref=<ref>
+ [utilization [$id=<id>] [<score>:] [rule...]
+ <param>=<value> [<param>=<value>...]] | $id-ref=<ref>
+
+type :: normal | member | ping | remote
+...............
+Example:
+...............
+node node1
+node big_node attributes memory=64
+...............
+
+[[cmdhelp_configure_op_defaults,set resource operations defaults]]
+==== `op_defaults`
+
+Set defaults for the operations meta attributes.
+
+For more information on rule expressions, see
+<<topics_Syntax_RuleExpressions,Syntax: Rule expressions>>.
+
+Usage:
+...............
+op_defaults [$id=<set_id>] [rule ...] <option>=<value> [<option>=<value> ...]
+...............
+Example:
+...............
+op_defaults record-pending=true
+...............
+
+[[cmdhelp_configure_order,order resources]]
+==== `order`
+
+This constraint expresses the order of actions on two resources
+or more resources. If there are more than two resources, then the
+constraint is called a resource set.
+
+Ordered resource sets have an extra attribute to allow for sets
+of resources whose actions may run in parallel. The shell syntax
+for such sets is to put resources in parentheses.
+
+If the subsequent resource can start or promote after any one of the
+resources in a set has done, enclose the set in brackets (+[+ and +]+).
+
+Sets cannot be nested.
+
+Three strings are reserved to specify a kind of order constraint:
++Mandatory+, +Optional+, and +Serialize+. It is preferred to use
+one of these settings instead of score. Previous versions mapped
+scores +0+ and +inf+ to keywords +advisory+ and +mandatory+.
+That is still valid but deprecated.
+
+For more details on how to configure resource sets, see
+<<topics_Features_Resourcesets,`Syntax: Resource sets`>>.
+
+Usage:
+...............
+order <id> [kind:] first then [symmetrical=<bool>]
+
+order <id> [kind:] resource_sets [symmetrical=<bool>]
+
+kind :: Mandatory | Optional | Serialize
+
+first :: <rsc>[:<action>]
+
+then :: <rsc>[:<action>]
+
+resource_sets :: resource_set [resource_set ...]
+
+resource_set :: ["["|"("] <rsc>[:<action>] [<rsc>[:<action>] ...] \
+ [attributes] ["]"|")"]
+
+attributes :: [require-all=(true|false)] [sequential=(true|false)]
+
+...............
+Example:
+...............
+order o-1 Mandatory: apache:start ip_1
+order o-2 Serialize: A ( B C )
+order o-4 first-resource then-resource
+...............
+
+[[cmdhelp_configure_primitive,define a resource]]
+==== `primitive`
+
+The primitive command describes a resource. It may be referenced
+only once in group, or clone objects. If it's not
+referenced, then it is placed as a single resource in the CIB.
+
+Operations may be specified anonymously, as a group or by reference:
+
+* "Anonymous", as a list of +op+ specifications. Use this
+ method if you don't need to reference the set of operations
+ elsewhere. This is the most common way to define operations.
+
+* If reusing operation sets is desired, use the +operations+ keyword
+ along with an id to give the operations set a name. Use the
+ +operations+ keyword and an id-ref value set to the id of another
+ operations set, to apply the same set of operations to this
+ primitive.
+
+Operation attributes which are not recognized are saved as
+instance attributes of that operation. A typical example is
++OCF_CHECK_LEVEL+.
+
+For multistate resources, roles are specified as +role=<role>+.
+The +Master/Slave+ resources are deprecated and replaced by +Promoted/Unpromoted+ promotable resources if desired.
+
+A template may be defined for resources which are of the same
+type and which share most of the configuration. See
+<<cmdhelp_configure_rsc_template,`rsc_template`>> for more information.
+
+Attributes containing time values, such as the +interval+ attribute on
+operations, are configured either as a plain number, which is
+interpreted as a time in seconds, or using one of the following
+suffixes:
+
+* +s+, +sec+ - time in seconds (same as no suffix)
+* +ms+, +msec+ - time in milliseconds
+* +us+, +usec+ - time in microseconds
+* +m+, +min+ - time in minutes
+* +h+, +hr+ - time in hours
+
+Usage:
+...............
+primitive <rsc> {[<class>:[<provider>:]]<type>|@<template>}
+ [description=<description>]
+ [[params] attr_list]
+ [meta attr_list]
+ [utilization attr_list]
+ [operations id_spec]
+ [op op_type [<attribute>=<value>...]
+ [[op_params] attr_list]
+ [op_meta attr_list] ...]
+
+attr_list :: [$id=<id>] [<score>:] [rule...]
+ <attr>=<val> [<attr>=<val>...]] | $id-ref=<id>
+id_spec :: $id=<id> | $id-ref=<id>
+op_type :: start | stop | monitor
+...............
+Example:
+...............
+primitive apcfence stonith:apcsmart \
+ params ttydev=/dev/ttyS0 hostlist="node1 node2" \
+ op start timeout=60s \
+ op monitor interval=30m timeout=60s
+
+primitive www8 apache \
+ configfile=/etc/apache/www8.conf \
+ operations $id-ref=apache_ops
+
+primitive db0 mysql \
+ params config=/etc/mysql/db0.conf \
+ op monitor interval=60s \
+ op monitor interval=300s OCF_CHECK_LEVEL=10
+
+primitive r0 ocf:linbit:drbd \
+ params drbd_resource=r0 \
+ op monitor role=Promoted interval=60s \
+ op monitor role=Unpromoted interval=300s
+
+primitive xen0 @vm_scheme1 xmfile=/etc/xen/vm/xen0
+
+primitive mySpecialRsc Special \
+ params 3: rule #uname eq node1 interface=eth1 \
+ params 2: rule #uname eq node2 interface=eth2 port=8888 \
+ params 1: interface=eth0 port=9999
+
+primitive A ocf:pacemaker:Dummy \
+ op start \
+ op_meta 2: rule #ra-version version:gt 1.0 timeout=120s \
+ op_meta 1: timeout=60s
+...............
+
+[[cmdhelp_configure_property,set a cluster property]]
+==== `property`
+
+Set cluster configuration properties. To list the
+available cluster configuration properties, use the
+<<cmdhelp_ra_info,`ra info`>> command with +pengine+, +crmd+,
++cib+ and +stonithd+ as arguments.
+When setting the +maintenance-mode+ property, it will
+inform the user if there are nodes or resources that
+have the +maintenance+ property.
+
+For more information on rule expressions, see
+<<topics_Syntax_RuleExpressions,Syntax: Rule expressions>>.
+
+Usage:
+...............
+property [<set_id>:] [rule ...] <option>=<value> [<option>=<value> ...]
+...............
+Example:
+...............
+property stonith-enabled=true
+property rule date spec years=2014 stonith-enabled=false
+...............
+
+[[cmdhelp_configure_ptest,show cluster actions if changes were committed]]
+==== `ptest` (`simulate`)
+
+Show PE (Policy Engine) motions using `ptest(8)` or
+`crm_simulate(8)`.
+
+A CIB is constructed using the current user edited configuration
+and the status from the running CIB. The resulting CIB is run
+through `ptest` (or `crm_simulate`) to show changes which would
+happen if the configuration is committed.
+
+The status section may be loaded from another source and modified
+using the <<cmdhelp_cibstatus,`cibstatus`>> level commands. In that case, the
+`ptest` command will issue a message informing the user that the
+Policy Engine graph is not calculated based on the current status
+section and therefore won't show what would happen to the
+running but some imaginary cluster.
+
+If you have graphviz installed and X11 session, `dotty(1)` is run
+to display the changes graphically.
+
+Add a string of +v+ characters to increase verbosity. `ptest`
+can also show allocation scores. +utilization+ turns on
+information about the remaining capacity of nodes. With the
++actions+ option, `ptest` will print all resource actions.
+
+The `ptest` program has been replaced by `crm_simulate` in newer
+Pacemaker versions. In some installations both could be
+installed. Use `simulate` to enfore using `crm_simulate`.
+
+Usage:
+...............
+ptest [nograph] [v...] [scores] [actions] [utilization]
+...............
+Examples:
+...............
+ptest scores
+ptest vvvvv
+simulate actions
+...............
+
+[[cmdhelp_configure_refresh,refresh from CIB]]
+==== `refresh`
+
+Refresh the internal structures from the CIB. All changes made
+during this session are lost.
+
+Usage:
+...............
+refresh
+...............
+
+[[cmdhelp_configure_rename,rename a CIB object]]
+==== `rename`
+
+Rename an object. It is recommended to use this command to rename
+a resource, because it will take care of updating all related
+constraints and a parent resource. Changing ids with the edit
+command won't have the same effect.
+
+If you want to rename a resource, it must be in the stopped state.
+
+Usage:
+...............
+rename <old_id> <new_id>
+...............
+
+[[cmdhelp_configure_role,define role access rights]]
+==== `role`
+
+An ACL role is a set of rules which describe access rights to
+CIB. Rules consist of an access right +read+, +write+, or +deny+
+and a specification denoting part of the configuration to which
+the access right applies. The specification can be an XPath or a
+combination of tag and id references. If an attribute is
+appended, then the specification applies only to that attribute
+of the matching element.
+
+There is a number of shortcuts for XPath specifications. The
++meta+, +params+, and +utilization+ shortcuts reference resource
+meta attributes, parameters, and utilization respectively. The
+`location` may be used to specify location constraints most of
+the time to allow resource `move` and `unmove` commands. The
+`property` references cluster properties. The `node` allows
+reading node attributes. +nodeattr+ and +nodeutil+ reference node
+attributes and node capacity (utilization). The `status` shortcut
+references the whole status section of the CIB. Read access to
+status is necessary for various monitoring tools such as
+`crm_mon(8)` (aka `crm status`).
+
+For more information on rule expressions, see
+<<topics_Syntax_RuleExpressions,Syntax: Rule expressions>>.
+
+Usage:
+...............
+role <role-id> rule [rule ...]
+
+rule :: acl-right cib-spec [attribute:<attribute>]
+
+acl-right :: read | write | deny
+
+cib-spec :: xpath-spec | tag-ref-spec
+xpath-spec :: xpath:<xpath> | shortcut
+tag-ref-spec :: tag:<tag> | ref:<id> | tag:<tag> ref:<id>
+
+shortcut :: meta:<rsc>[:<attr>]
+ params:<rsc>[:<attr>]
+ utilization:<rsc>
+ location:<rsc>
+ property[:<attr>]
+ node[:<node>]
+ nodeattr[:<attr>]
+ nodeutil[:<node>]
+ status
+...............
+Example:
+...............
+role app1_admin \
+ write meta:app1:target-role \
+ write meta:app1:is-managed \
+ write location:app1 \
+ read ref:app1
+...............
+
+[[cmdhelp_configure_rsc_defaults,set resource defaults]]
+==== `rsc_defaults`
+
+Set defaults for the resource meta attributes.
+
+For more information on rule expressions, see
+<<topics_Syntax_RuleExpressions,Syntax: Rule expressions>>.
+
+Usage:
+...............
+rsc_defaults [<set_id>:] [rule ...] <option>=<value> [<option>=<value> ...]
+...............
+Example:
+...............
+rsc_defaults failure-timeout=3m
+...............
+
+[[cmdhelp_configure_rsc_template,define a resource template]]
+==== `rsc_template`
+
+The `rsc_template` command creates a resource template. It may be
+referenced in primitives. It is used to reduce large
+configurations with many similar resources.
+
+Usage:
+...............
+rsc_template <name> [<class>:[<provider>:]]<type>
+ [description=<description>]
+ [params attr_list]
+ [meta attr_list]
+ [utilization attr_list]
+ [operations id_spec]
+ [op op_type [<attribute>=<value>...] ...]
+
+attr_list :: [$id=<id>] <attr>=<val> [<attr>=<val>...] | $id-ref=<id>
+id_spec :: $id=<id> | $id-ref=<id>
+op_type :: start | stop | monitor
+...............
+Example:
+...............
+rsc_template public_vm Xen \
+ op start timeout=300s \
+ op stop timeout=300s \
+ op monitor interval=30s timeout=60s \
+ op migrate_from timeout=600s \
+ op migrate_to timeout=600s
+primitive xen0 @public_vm \
+ params xmfile=/etc/xen/xen0
+primitive xen1 @public_vm \
+ params xmfile=/etc/xen/xen1
+...............
+
+[[cmdhelp_configure_rsc_ticket,resources ticket dependency]]
+==== `rsc_ticket`
+
+This constraint expresses dependency of resources on cluster-wide
+attributes, also known as tickets. Tickets are mainly used in
+geo-clusters, which consist of multiple sites. A ticket may be
+granted to a site, thus allowing resources to run there.
+
+The +loss-policy+ attribute specifies what happens to the
+resource (or resources) if the ticket is revoked. The default is
+either +stop+ or +demote+ depending on whether a resource is
+multi-state.
+
+See also the <<cmdhelp_site_ticket,`site`>> set of commands.
+
+Usage:
+...............
+rsc_ticket <id> <ticket_id>: <rsc>[:<role>] [<rsc>[:<role>] ...]
+ [loss-policy=<loss_policy_action>]
+
+loss_policy_action :: stop | demote | fence | freeze
+...............
+Example:
+...............
+rsc_ticket ticket-A_public-ip ticket-A: public-ip
+rsc_ticket ticket-A_bigdb ticket-A: bigdb loss-policy=fence
+rsc_ticket ticket-B_storage ticket-B: drbd-a:Promoted drbd-b:Promoted
+...............
+
+
+[[cmdhelp_configure_rsctest,test resources as currently configured]]
+==== `rsctest`
+
+Test resources with current resource configuration. If no nodes
+are specified, tests are run on all known nodes.
+
+The order of resources is significant: it is assumed that later
+resources depend on earlier ones.
+
+If a resource is multi-state, it is assumed that the role on
+which later resources depend is master.
+
+Tests are run sequentially to prevent running the same resource
+on two or more nodes. Tests are carried out only if none of the
+specified nodes currently run any of the specified resources.
+However, it won't verify whether resources run on the other
+nodes.
+
+Superuser privileges are obviously required: either run this as
+root or setup the `sudoers` file appropriately.
+
+Note that resource testing may take some time.
+
+Usage:
+...............
+rsctest <rsc_id> [<rsc_id> ...] [<node_id> ...]
+...............
+Examples:
+...............
+rsctest my_ip websvc
+rsctest websvc nodeB
+...............
+
+[[cmdhelp_configure_save,save the CIB to a file]]
+==== `save`
+
+Save the current configuration to a file. Optionally, as XML. Use
++-+ instead of file name to write the output to `stdout`.
+
+The `save` command accepts the same selection arguments as the `show`
+command. See the <<cmdhelp_configure_show,help section>> for `show`
+for more details.
+
+Usage:
+...............
+save [xml] [<id> | type:<type | tag:<tag> |
+ related:<obj> | changed ...] <file>
+...............
+Example:
+...............
+save myfirstcib.txt
+save web-server server-config.txt
+...............
+
+[[cmdhelp_configure_schema,set or display current CIB RNG schema]]
+==== `schema`
+
+CIB's content is validated by a RNG schema. Pacemaker supports
+several, depending on version. At least the following schemas are
+accepted by `crmsh`:
+
+* +pacemaker-1.0+
+* +pacemaker-1.1+
+* +pacemaker-1.2+
+* +pacemaker-1.3+
+* +pacemaker-2.0+
+
+Use this command to display or switch to another RNG schema.
+
+Usage:
+...............
+schema [<schema>]
+...............
+Example:
+...............
+schema pacemaker-1.1
+...............
+
+[[cmdhelp_configure_set,set an attribute value]]
+==== `set`
+
+Set the value of a configured attribute. The attribute must
+configured previously, and can be an agent parameter, meta attribute,
+utilization value or operation value.
+
+The first argument to the command is a path to an attribute.
+This is a dot-separated sequence beginning with the name of
+the resource or object, and ending with the name of the attribute to
+set. To set operation value, `op_type` should be specified; when multi
+operations exist like multi monitors, `interval` should be specified.
+
+Usage:
+...............
+set <path> <value>
+
+path:: id.[op_type.][interval.]name
+...............
+Examples:
+...............
+set vip1.ip 192.168.20.5
+set vm-a.force_stop 1
+set vip1.monitor.on-fail ignore
+set drbd.monitor.10s.interval 20s
+...............
+
+[[cmdhelp_configure_show,display CIB objects]]
+==== `show`
+
+The `show` command displays CIB objects. Without any argument, it
+displays all objects in the CIB, but the set of objects displayed by
+`show` can be limited to only objects with the given IDs or by using
+one or more of the special prefixes described below.
+
+The XML representation for the objects can be displayed by passing
++xml+ as the first argument.
+
+To show one or more specific objects, pass the object IDs as
+arguments.
+
+To show all objects of a certain type, use the +type:+ prefix.
+
+To show all objects in a tag, use the +tag:+ prefix.
+
+To show all constraints related to a primitive, or
+to show all objects of a certain RA type, use the +related:+ prefix.
+
+To show all modified objects, pass the argument +changed+.
+
+The prefixes can be used together on a single command line. For
+example, to show both the tag itself and the objects tagged by it the
+following combination can be used: +show tag:my-tag my-tag+.
+
+To refine a selection of objects using multiple modifiers, the keywords
++and+ and +or+ can be used. For example, to select all primitives tagged
++foo+, the following combination can be used:
++show type:primitive and tag:foo+.
+
+To hide values when displaying the configuration, use the
++obscure:<glob>+ argument. This can be useful when sending the
+configuration over a public channel, to avoid exposing potentially
+sensitive information. The +<glob>+ argument is a bash-style pattern
+matching attribute keys.
+
+In +/etc/crm/crm.conf+, +obscure_pattern+ option is the persisent configuration of CLI.
+Example, for the high security concern,
+...............
+[core]
+obscure_pattern = passw* | ip
+...............
+Which makes +crm configure show+ is equal to
+...............
+node-1:~ # crm configure show obscure:passw* obscure:ip
+node 1084783297: node1
+primitive fence_device stonith:fence_ilo5 \
+ params password="******"
+primitive ip IPaddr2 \
+ params ip="******"
+...............
+The default suggestion is +passw*+
+If you don't want to obscure, change the value to blank.
+
+Usage:
+...............
+show [xml] [<id>
+ | changed
+ | type:<type>
+ | tag:<id>
+ | related:<obj>
+ | obscure:<glob>
+ ...]
+
+type :: node | primitive | group | clone | ms | rsc_template
+ | location | colocation | order
+ | rsc_ticket
+ | property | rsc_defaults | op_defaults
+ | fencing_topology
+ | role | user | acl_target
+ | tag
+...............
+
+Example:
+...............
+show webapp
+show type:primitive
+show xml tag:db tag:fs
+show related:webapp
+show related:IPaddr2
+show related:ipad
+show related:ocf:heartbeat:Dummy
+show related:ocf:heartbeat:dum
+show related:ocf
+show related:heartbeat
+show related:pacemaker
+show related:suse
+show related:stonith
+show type:primitive obscure:passwd
+...............
+
+[[cmdhelp_configure_tag,Define resource tags]]
+==== `tag`
+
+Define a resource tag. A tag is an id referring to one or more
+resources, without implying any constraints between the tagged
+resources. This can be useful for grouping conceptually related
+resources.
+
+Usage:
+...............
+tag <tag-name>: <rsc> [<rsc> ...]
+tag <tag-name> <rsc> [<rsc> ...]
+...............
+Example:
+...............
+tag web: p-webserver p-vip
+tag ips server-vip admin-vip
+...............
+
+[[cmdhelp_configure_template,edit and import a configuration from a template]]
+==== `template`
+
+The specified template is loaded into the editor. It's up to the
+user to make a good CRM configuration out of it. See also the
+<<cmdhelp_template,template section>>.
+
+Usage:
+...............
+template [xml] url
+...............
+Example:
+...............
+template two-apaches.txt
+...............
+
+[[cmdhelp_configure_upgrade,upgrade the CIB]]
+==== `upgrade`
+
+Attempts to upgrade the CIB to validate with the current
+version. Commonly, this is required if the error
+`CIB not supported` occurs. It typically means that the
+active CIB version is coming from an older release.
+
+As a safety precaution, the force argument is required if the
++validation-with+ attribute is set to anything other than
++0.6+. Thus in most cases, it is required.
+
+Usage:
+...............
+upgrade [force]
+...............
+
+Example:
+...............
+upgrade force
+...............
+
+[[cmdhelp_configure_user,define user access rights]]
+==== `user`
+
+Users which normally cannot view or manage cluster configuration
+can be allowed access to parts of the CIB. The access is defined
+by a set of +read+, +write+, and +deny+ rules as in role
+definitions or by referencing roles. The latter is considered
+best practice.
+
+For more information on rule expressions, see
+<<topics_Syntax_RuleExpressions,Syntax: Rule expressions>>.
+
+Usage:
+...............
+user <uid> {roles|rules}
+
+roles :: role:<role-ref> [role:<role-ref> ...]
+rules :: rule [rule ...]
+...............
+Example:
+...............
+user joe \
+ role:app1_admin \
+ role:read_all
+...............
+
+[[cmdhelp_configure_validate_all,call agent validate-all for resource]]
+==== `validate-all`
+
+Call the `validate-all` action for the resource, if possible.
+
+Limitations:
+
+* The resource agent must implement the `validate-all` action.
+* The current user must be root.
+* The primitive resource must not use nvpair references.
+
+Usage:
+...............
+validate-all <rsc>
+...............
+
+
+[[cmdhelp_configure_verify,verify the CIB with crm_verify]]
+==== `verify`
+
+Verify the contents of the CIB which would be committed.
+
+Usage:
+...............
+verify
+...............
+
+[[cmdhelp_configure_xml,raw xml]]
+==== `xml`
+
+Even though we promissed no xml, it may happen, but hopefully
+very very seldom, that an element from the CIB cannot be rendered
+in the configuration language. In that case, the element will be
+shown as raw xml, prefixed by this command. That element can then
+be edited like any other. If the shell finds out that after the
+change it can digest it, then it is going to be converted into
+the normal configuration language. Otherwise, there is no need to
+use `xml` for configuration.
+
+Usage:
+...............
+xml <xml>
+...............
+
+[[cmdhelp_template,edit and import a configuration from a template]]
+=== `template` - Import configuration from templates
+
+User may be assisted in the cluster configuration by templates
+prepared in advance. Templates consist of a typical ready
+configuration which may be edited to suit particular user needs.
+
+This command enters a template level where additional commands
+for configuration/template management are available.
+
+[[cmdhelp_template_apply,process and apply the current configuration to the current CIB]]
+==== `apply`
+
+Copy the current or given configuration to the current CIB. By
+default, the CIB is replaced, unless the method is set to
+"update".
+
+Usage:
+...............
+apply [<method>] [<config>]
+
+method :: replace | update
+...............
+
+[[cmdhelp_template_delete,delete a configuration]]
+==== `delete`
+
+Remove a configuration. The loaded (active) configuration may be
+removed by force.
+
+Usage:
+...............
+delete <config> [force]
+...............
+
+[[cmdhelp_template_edit,edit a configuration]]
+==== `edit`
+
+Edit current or given configuration using your favourite editor.
+
+Usage:
+...............
+edit [<config>]
+...............
+
+[[cmdhelp_template_list,list configurations/templates]]
+==== `list`
+
+When called with no argument, lists existing templates and
+configurations.
+
+Given the argument +templates+, lists the available templates.
+
+Given the argument +configs+, lists the available configurations.
+
+Usage:
+...............
+list [templates|configs]
+...............
+
+[[cmdhelp_template_load,load a configuration]]
+==== `load`
+
+Load an existing configuration. Further `edit`, `show`, and
+`apply` commands will refer to this configuration.
+
+Usage:
+...............
+load <config>
+...............
+
+[[cmdhelp_template_new,create a new configuration from templates]]
+==== `new`
+
+Create a new configuration from one or more templates. Note that
+configurations and templates are kept in different places, so it
+is possible to have a configuration name equal a template name.
+
+If you already know which parameters are required, you can set
+them directly on the command line.
+
+The parameter name +id+ is set by default to the name of the
+configuration.
+
+If no parameters are being set and you don't want a particular name
+for your configuration, you can call this command with a template name
+as the only parameter. A unique configuration name based on the
+template name will be generated.
+
+Usage:
+...............
+new [<config>] <template> [<template> ...] [params name=value ...]
+...............
+
+Example:
+...............
+new vip virtual-ip
+new bigfs ocfs2 params device=/dev/sdx8 directory=/bigfs
+new apache
+...............
+
+[[cmdhelp_template_show,show the processed configuration]]
+==== `show`
+
+Process the current or given configuration and display the result.
+
+Usage:
+...............
+show [<config>]
+...............
+
+[[cmdhelp_cibstatus,CIB status management and editing]]
+=== `cibstatus` - CIB status management and editing
+
+The `status` section of the CIB keeps the current status of nodes
+and resources. It is modified _only_ on events, i.e. when some
+resource operation is run or node status changes. For obvious
+reasons, the CRM has no user interface with which it is possible
+to affect the status section. From the user's point of view, the
+status section is essentially a read-only part of the CIB. The
+current status is never even written to disk, though it is
+available in the PE (Policy Engine) input files which represent
+the history of cluster motions. The current status may be read
+using the +cibadmin -Q+ command.
+
+It may sometimes be of interest to see how status changes would
+affect the Policy Engine. The set of `cibstatus` level commands
+allow the user to load status sections from various sources and
+then insert or modify resource operations or change nodes' state.
+
+The effect of those changes may then be observed by running the
+<<cmdhelp_configure_ptest,`ptest`>> command at the `configure` level
+or `simulate` and `run` commands at this level. The `ptest`
+runs with the user edited CIB whereas the latter two commands
+run with the CIB which was loaded along with the status section.
+
+The `simulate` and `run` commands as well as all status
+modification commands are implemented using `crm_simulate(8)`.
+
+[[cmdhelp_cibstatus_load,load the CIB status section]]
+==== `load`
+
+Load a status section from a file, a shadow CIB, or the running
+cluster. By default, the current (+live+) status section is
+modified. Note that if the +live+ status section is modified it
+is not going to be updated if the cluster status changes, because
+that would overwrite the user changes. To make `crm` drop changes
+and resume use of the running cluster status, run +load live+.
+
+All CIB shadow configurations contain the status section which is
+a snapshot of the status section taken at the time the shadow was
+created. Obviously, this status section doesn't have much to do
+with the running cluster status, unless the shadow CIB has just
+been created. Therefore, the `ptest` command by default uses the
+running cluster status section.
+
+Usage:
+...............
+load {<file>|shadow:<cib>|live}
+...............
+Example:
+...............
+load bug-12299.xml
+load shadow:test1
+...............
+
+[[cmdhelp_cibstatus_node,change node status]]
+==== `node`
+
+Change the node status. It is possible to throw a node out of
+the cluster, make it a member, or set its state to unclean.
+
++online+:: Set the +node_state+ `crmd` attribute to +online+
+and the +expected+ and +join+ attributes to +member+. The effect
+is that the node becomes a cluster member.
+
++offline+:: Set the +node_state+ `crmd` attribute to +offline+
+and the +expected+ attribute to empty. This makes the node
+cleanly removed from the cluster.
+
++unclean+:: Set the +node_state+ `crmd` attribute to +offline+
+and the +expected+ attribute to +member+. In this case the node
+has unexpectedly disappeared.
+
+Usage:
+...............
+node <node> {online|offline|unclean}
+...............
+Example:
+...............
+node xen-b unclean
+...............
+
+[[cmdhelp_cibstatus_op,edit outcome of a resource operation]]
+==== `op`
+
+Edit the outcome of a resource operation. This way you can
+tell CRM that it ran an operation and that the resource agent
+returned certain exit code. It is also possible to change the
+operation's status. In case the operation status is set to
+something other than +done+, the exit code is effectively
+ignored.
+
+Usage:
+...............
+op <operation> <resource> <exit_code> [<op_status>] [<node>]
+
+operation :: probe | monitor[:<n>] | start | stop |
+ promote | demote | notify | migrate_to | migrate_from
+exit_code :: <rc> | success | generic | args |
+ unimplemented | perm | installed | configured | not_running |
+ master | failed_master
+op_status :: pending | done | cancelled | timeout | notsupported | error
+
+n :: the monitor interval in seconds; if omitted, the first
+ recurring operation is referenced
+rc :: numeric exit code in range 0..9
+...............
+Example:
+...............
+op start d1 xen-b generic
+op start d1 xen-b 1
+op monitor d1 xen-b not_running
+op stop d1 xen-b 0 timeout
+...............
+
+[[cmdhelp_cibstatus_origin,display origin of the CIB status section]]
+==== `origin`
+
+Show the origin of the status section currently in use. This
+essentially shows the latest `load` argument.
+
+Usage:
+...............
+origin
+...............
+
+[[cmdhelp_cibstatus_quorum,set the quorum]]
+==== `quorum`
+
+Set the quorum value.
+
+Usage:
+...............
+quorum <bool>
+...............
+Example:
+...............
+quorum false
+...............
+
+[[cmdhelp_cibstatus_run,run policy engine]]
+==== `run`
+
+Run the policy engine with the edited status section.
+
+Add a string of +v+ characters to increase verbosity. Specify
++scores+ to see allocation scores also. +utilization+ turns on
+information about the remaining capacity of nodes.
+
+If you have graphviz installed and X11 session, `dotty(1)` is run
+to display the changes graphically.
+
+Usage:
+...............
+run [nograph] [v...] [scores] [utilization]
+...............
+Example:
+...............
+run
+...............
+
+[[cmdhelp_cibstatus_save,save the CIB status section]]
+==== `save`
+
+The current internal status section with whatever modifications
+were performed can be saved to a file or shadow CIB.
+
+If the file exists and contains a complete CIB, only the status
+section is going to be replaced and the rest of the CIB will
+remain intact. Otherwise, the current user edited configuration
+is saved along with the status section.
+
+Note that all modifications are saved in the source file as soon
+as they are run.
+
+Usage:
+...............
+save [<file>|shadow:<cib>]
+...............
+Example:
+...............
+save bug-12299.xml
+...............
+
+[[cmdhelp_cibstatus_show,show CIB status section]]
+==== `show`
+
+Show the current status section in the XML format. Brace yourself
+for some unreadable output. Add +changed+ option to get a human
+readable output of all changes.
+
+Usage:
+...............
+show [changed]
+...............
+
+[[cmdhelp_cibstatus_simulate,simulate cluster transition]]
+==== `simulate`
+
+Run the policy engine with the edited status section and simulate
+the transition.
+
+Add a string of +v+ characters to increase verbosity. Specify
++scores+ to see allocation scores also. +utilization+ turns on
+information about the remaining capacity of nodes.
+
+If you have graphviz installed and X11 session, `dotty(1)` is run
+to display the changes graphically.
+
+Usage:
+...............
+simulate [nograph] [v...] [scores] [utilization]
+...............
+Example:
+...............
+simulate
+...............
+
+[[cmdhelp_cibstatus_ticket,manage tickets]]
+==== `ticket`
+
+Modify the ticket status. Tickets can be granted and revoked.
+Granted tickets could be activated or put in standby.
+
+Usage:
+...............
+ticket <ticket> {grant|revoke|activate|standby}
+...............
+Example:
+...............
+ticket ticketA grant
+...............
+
+[[cmdhelp_assist,Configuration assistant]]
+=== `assist` - Configuration assistant
+
+The `assist` sublevel is a collection of helper
+commands that create or modify resources and
+constraints, to simplify the creation of certain
+configurations.
+
+For more information on individual commands, see
+the help text for those commands.
+
+[[cmdhelp_assist_template,Create template for primitives]]
+==== `template`
+
+This command takes a list of primitives as argument, and creates a new
+`rsc_template` for these primitives. It can only do this if the
+primitives do not already share a template and are of the same type.
+
+Usage:
+........
+template primitive-1 primitive-2 primitive-3
+........
+
+[[cmdhelp_assist_weak-bond,Create a weak bond between resources]]
+==== `weak-bond`
+
+A colocation between a group of resources says that the resources
+should be located together, but it also means that those resources are
+dependent on each other. If one of the resources fails, the others
+will be restarted.
+
+If this is not desired, it is possible to circumvent: By placing the
+resources in a non-sequential set and colocating the set with a dummy
+resource which is not monitored, the resources will be placed together
+but will have no further dependency on each other.
+
+This command creates both the constraint and the dummy resource needed
+for such a colocation.
+
+Usage:
+........
+weak-bond resource-1 resource-2
+........
+
+[[cmdhelp_maintenance,Maintenance mode commands]]
+=== `maintenance` - Maintenance mode commands
+
+Maintenance mode commands are commands that manipulate resources
+directly without going through the cluster infrastructure. Therefore,
+it is essential to ensure that the cluster does not attempt to monitor
+or manipulate the resources while these commands are being executed.
+
+To ensure this, these commands require that maintenance mode is set
+either for the particular resource, or for the whole cluster.
+
+[[cmdhelp_maintenance_action,Invoke a resource action]]
+==== `action`
+
+Invokes the given action for the resource. This is
+done directly via the resource agent, so the command must
+be issued while the cluster or the resource is in
+maintenance mode.
+
+Unless the action is `start` or `monitor`, the action must be invoked
+on the same node as where the resource is running. If the resource is
+running on multiple nodes, the command will fail.
+
+To use SSH for executing resource actions on multiple nodes, append
+`ssh` after the action name. This requires SSH access to be configured
+between the nodes and the parallax python package to be installed.
+
+Usage:
+...............
+action <rsc> <action>
+action <rsc> <action> ssh
+...............
+Example:
+...............
+action webserver reload
+action webserver monitor ssh
+...............
+
+[[cmdhelp_maintenance_off,Disable maintenance mode]]
+==== `off`
+
+Disables maintenances mode, either for the whole cluster
+or for the given resource.
+
+Usage:
+...............
+off
+off <rsc>
+...............
+Example:
+...............
+off rsc1
+...............
+
+[[cmdhelp_maintenance_on,Enable maintenance mode]]
+==== `on`
+
+Enables maintenances mode, either for the whole cluster
+or for the given resource.
+
+Usage:
+...............
+on
+on <rsc>
+...............
+Example:
+...............
+on rsc1
+...............
+
+[[cmdhelp_history,Cluster history]]
+=== `history` - Cluster history
+
+Examining Pacemaker's history is a particularly involved task. The
+number of subsystems to be considered, the complexity of the
+configuration, and the set of various information sources, most of
+which are not exactly human readable, keep analyzing resource or node
+problems accessible to only the most knowledgeable. Or, depending on
+the point of view, to the most persistent. The following set of
+commands has been devised in hope to make cluster history more
+accessible.
+
+Of course, looking at _all_ history could be time consuming regardless
+of how good the tools at hand are. Therefore, one should first say
+which period he or she wants to analyze. If not otherwise specified,
+the last hour is considered. Logs and other relevant information is
+collected using `crm report`. Since this process takes some time and
+we always need fresh logs, information is refreshed in a much faster
+way using the python parallax module. If +python-parallax+ is not
+found on the system, examining a live cluster is still possible --
+though not as comfortable.
+
+Apart from examining a live cluster, events may be retrieved from a
+report generated by `crm report` (see also the +-H+ option). In that
+case we assume that the period stretching the whole report needs to be
+investigated. Of course, it is still possible to further reduce the
+time range.
+
+If you have discovered an issue that you want to show someone else,
+you can use the `session pack` command to save the current session as
+a tarball, similar to those generated by `crm report`.
+
+In order to minimize the size of the tarball, and to make it easier
+for others to find the interesting events, it is recommended to limit
+the time frame which the saved session covers. This can be done using
+the `timeframe` command (example below).
+
+It is also possible to name the saved session using the `session save`
+command.
+
+Example:
+...............
+crm(live)history# limit "Jul 18 12:00" "Jul 18 12:30"
+crm(live)history# session save strange_restart
+crm(live)history# session pack
+Report saved in .../strange_restart.tar.bz2
+crm(live)history#
+...............
+
+[[cmdhelp_history_detail,set the level of detail shown]]
+==== `detail`
+
+How much detail to show from the logs. Valid detail levels are either
+`0` or `1`, where `1` is the highest detail level. The default detail
+level is `0`.
+
+Usage:
+...............
+detail <detail_level>
+
+detail_level :: small integer (defaults to 0)
+...............
+Example:
+...............
+detail 1
+...............
+
+[[cmdhelp_history_diff,cluster states/transitions difference]]
+==== `diff`
+
+A transition represents a change in cluster configuration or
+state. Use `diff` to see what has changed between two
+transitions.
+
+If you want to specify the current cluster configuration and
+status, use the string +live+.
+
+Normally, the first transition specified should be the one which
+is older, but we are not going to enforce that.
+
+Note that a single configuration update may result in more than
+one transition.
+
+Usage:
+...............
+diff <pe> <pe> [status] [html]
+
+pe :: <number>|<index>|<file>|live
+...............
+Examples:
+...............
+diff 2066 2067
+diff pe-input-2080.bz2 live status
+...............
+
+[[cmdhelp_history_events,Show events in log]]
+==== `events`
+
+By analysing the log output and looking for particular
+patterns, the `events` command helps sifting through
+the logs to find when particular events like resources
+changing state or node failure may have occurred.
+
+This can be used to generate a combined list of events
+from all nodes.
+
+Usage:
+...............
+events
+...............
+
+Example:
+...............
+events
+...............
+
+[[cmdhelp_history_exclude,exclude log messages]]
+==== `exclude`
+
+If a log is infested with irrelevant messages, those messages may
+be excluded by specifying a regular expression. The regular
+expressions used are Python extended. This command is additive.
+To drop all regular expressions, use +exclude clear+. Run
+`exclude` only to see the current list of regular expressions.
+Excludes are saved along with the history sessions.
+
+Usage:
+...............
+exclude [<regex>|clear]
+...............
+Example:
+...............
+exclude kernel.*ocfs2
+...............
+
+[[cmdhelp_history_graph,generate a directed graph from the PE file]]
+==== `graph`
+
+Create a graphviz graphical layout from the PE file (the
+transition). Every transition contains the cluster configuration
+which was active at the time. See also <<cmdhelp_configure_graph,generate a directed graph
+from configuration>>.
+
+Usage:
+...............
+graph <pe> [<gtype> [<file> [<img_format>]]]
+
+gtype :: dot
+img_format :: `dot` output format (see the +-T+ option)
+...............
+Example:
+...............
+graph -1
+graph 322 dot clu1.conf.dot
+graph 322 dot clu1.conf.svg svg
+...............
+
+[[cmdhelp_history_info,Cluster information summary]]
+==== `info`
+
+The `info` command provides a summary of the information source, which
+can be either a live cluster snapshot or a previously generated
+report.
+
+Usage:
+...............
+info
+...............
+Example:
+...............
+info
+...............
+
+[[cmdhelp_history_latest,show latest news from the cluster]]
+==== `latest`
+
+The `latest` command shows a bit of recent history, more
+precisely whatever happened since the last cluster change (the
+latest transition). If the transition is running, the shell will
+first wait until it finishes.
+
+Usage:
+...............
+latest
+...............
+Example:
+...............
+latest
+...............
+
+[[cmdhelp_history_limit,limit timeframe to be examined]]
+==== `limit` (`timeframe`)
+
+This command can be used to modify the time span to examine. All
+history commands look at events within a certain time span.
+
+For the `live` source, the default time span is the _last hour_.
+
+There is no time span limit for the `crm report` source.
+
+The time period is parsed by the `dateutil` python module. It
+covers a wide range of date formats. For instance:
+
+- 3:00 (today at 3am)
+- 15:00 (today at 3pm)
+- 2010/9/1 2pm (September 1st 2010 at 2pm)
+
+For more examples of valid time/date statements, please refer to the
+`python-dateutil` documentation:
+
+- https://dateutil.readthedocs.org/[dateutil.readthedocs.org]
+
+If the dateutil module is not available, then the time is parsed using
+strptime and only the kind as printed by `date(1)` is allowed:
+
+- Tue Sep 15 20:46:27 CEST 2010
+
+Usage:
+...............
+limit [<from_time>] [<to_time>]
+...............
+Examples:
+...............
+limit 10:15
+limit 15h22m 16h
+limit "Sun 5 20:46" "Sun 5 22:00"
+...............
+
+[[cmdhelp_history_log,log content]]
+==== `log`
+
+Show messages logged on one or more nodes. Leaving out a node
+name produces combined logs of all nodes. Messages are sorted by
+time and, if the terminal emulations supports it, displayed in
+different colours depending on the node to allow for easier
+reading.
+
+The sorting key is the timestamp as written by syslog which
+normally has the maximum resolution of one second. Obviously,
+messages generated by events which share the same timestamp may
+not be sorted in the same way as they happened. Such close events
+may actually happen fairly often.
+
+Usage:
+...............
+log [<node> [<node> ...] ]
+...............
+Example:
+...............
+log node-a
+...............
+
+[[cmdhelp_history_node,node events]]
+==== `node`
+
+Show important events that happened on a node. Important events
+are node lost and join, standby and online, and fence. Use either
+node names or extended regular expressions.
+
+Usage:
+...............
+node <node> [<node> ...]
+...............
+Example:
+...............
+node node1
+...............
+
+[[cmdhelp_history_peinputs,list or get PE input files]]
+==== `peinputs`
+
+Every event in the cluster results in generating one or more
+Policy Engine (PE) files. These files describe future motions of
+resources. The files are listed as full paths in the current
+report directory. Add +v+ to also see the creation time stamps.
+
+Usage:
+...............
+peinputs [{<range>|<number>} ...] [v]
+
+range :: <n1>:<n2>
+...............
+Example:
+...............
+peinputs
+peinputs 440:444 446
+peinputs v
+...............
+
+[[cmdhelp_history_refresh,refresh live report]]
+==== `refresh`
+
+This command makes sense only for the +live+ source and makes
+`crm` collect the latest logs and other relevant information from
+the logs. If you want to make a completely new report, specify
++force+.
+
+Usage:
+...............
+refresh [force]
+...............
+
+[[cmdhelp_history_resource,resource events]]
+==== `resource`
+
+Show actions and any failures that happened on all specified
+resources on all nodes. Normally, one gives resource names as
+arguments, but it is also possible to use extended regular
+expressions. Note that neither groups nor clones or master/slave
+names are ever logged. The resource command is going to expand
+all of these appropriately, so that clone instances or resources
+which are part of a group are shown.
+
+Usage:
+...............
+resource <rsc> [<rsc> ...]
+...............
+Example:
+...............
+resource bigdb public_ip
+resource my_.*_db2
+resource ping_clone
+...............
+
+[[cmdhelp_history_session,manage history sessions]]
+==== `session`
+
+Sometimes you may want to get back to examining a particular
+history period or bug report. In order to make that easier, the
+current settings can be saved and later retrieved.
+
+If the current history being examined is coming from a live
+cluster the logs, PE inputs, and other files are saved too,
+because they may disappear from nodes. For the existing reports
+coming from `crm report`, only the directory location is saved
+(not to waste space).
+
+A history session may also be packed into a tarball which can
+then be sent to support.
+
+Leave out subcommand to see the current session.
+
+Usage:
+...............
+session [{save|load|delete} <name> | pack [<name>] | update | list]
+...............
+Examples:
+...............
+session save bnc966622
+session load rsclost-2
+session list
+...............
+
+[[cmdhelp_history_setnodes,set the list of cluster nodes]]
+==== `setnodes`
+
+In case the host this program runs on is not part of the cluster,
+it is necessary to set the list of nodes.
+
+Usage:
+...............
+setnodes node <node> [<node> ...]
+...............
+Example:
+...............
+setnodes node_a node_b
+...............
+
+[[cmdhelp_history_show,show status or configuration of the PE input file]]
+==== `show`
+
+Every transition is saved as a PE file. Use this command to
+render that PE file either as configuration or status. The
+configuration output is the same as `crm configure show`.
+
+Usage:
+...............
+show <pe> [status]
+
+pe :: <number>|<index>|<file>|live
+...............
+Examples:
+...............
+show 2066
+show pe-input-2080.bz2 status
+...............
+
+[[cmdhelp_history_source,set source to be examined]]
+==== `source`
+
+Events to be examined can come from the current cluster or from a
+`crm report` report. This command sets the source. `source live`
+sets source to the running cluster and system logs. If no source
+is specified, the current source information is printed.
+
+In case a report source is specified as a file reference, the file
+is going to be unpacked in place where it resides. This directory
+is not removed on exit.
+
+Usage:
+...............
+source [<dir>|<file>|live]
+...............
+Examples:
+...............
+source live
+source /tmp/customer_case_22.tar.bz2
+source /tmp/customer_case_22
+source
+...............
+
+[[cmdhelp_history_transition,show transition]]
+==== `transition`
+
+This command will print actions planned by the PE and run
+graphviz (`dotty`) to display a graphical representation of the
+transition. Of course, for the latter an X11 session is required.
+This command invokes `ptest(8)` in background.
+
+The +showdot+ subcommand runs graphviz (`dotty`) to display a
+graphical representation of the +.dot+ file which has been
+included in the report. Essentially, it shows the calculation
+produced by `pengine` which is installed on the node where the
+report was produced. In optimal case this output should not
+differ from the one produced by the locally installed `pengine`.
+
+The `log` subcommand shows the full log for the duration of the
+transition.
+
+A transition can also be saved to a CIB shadow for further
+analysis or use with `cib` or `configure` commands (use the
+`save` subcommand). The shadow file name defaults to the name of
+the PE input file.
+
+If the PE input file number is not provided, it defaults to the
+last one, i.e. the last transition. The last transition can also
+be referenced with number 0. If the number is negative, then the
+corresponding transition relative to the last one is chosen.
+
+If there are warning and error PE input files or different nodes
+were the DC in the observed timeframe, it may happen that PE
+input file numbers collide. In that case provide some unique part
+of the path to the file.
+
+After the `ptest` output, logs about events that happened during
+the transition are printed.
+
+The `tags` subcommand scans the logs for the transition and return a
+list of key events during that transition. For example, the tag
++error+ will be returned if there are any errors logged during the
+transition.
+
+Usage:
+...............
+transition [<number>|<index>|<file>] [nograph] [v...] [scores] [actions] [utilization]
+transition showdot [<number>|<index>|<file>]
+transition log [<number>|<index>|<file>]
+transition save [<number>|<index>|<file> [name]]
+transition tags [<number>|<index>|<file>]
+...............
+Examples:
+...............
+transition
+transition 444
+transition -1
+transition pe-error-3.bz2
+transition node-a/pengine/pe-input-2.bz2
+transition showdot 444
+transition log
+transition save 0 enigma-22
+...............
+
+[[cmdhelp_history_transitions,List transitions]]
+==== `transitions`
+
+A transition represents a change in cluster configuration or
+state. This command lists the transitions in the current timeframe.
+
+Usage:
+...............
+transitions
+...............
+Example:
+...............
+transitions
+...............
+
+
+[[cmdhelp_history_wdiff,cluster states/transitions difference]]
+==== `wdiff`
+
+A transition represents a change in cluster configuration or
+state. Use `wdiff` to see what has changed between two
+transitions as word differences on a line-by-line basis.
+
+If you want to specify the current cluster configuration and
+status, use the string +live+.
+
+Normally, the first transition specified should be the one which
+is older, but we are not going to enforce that.
+
+Note that a single configuration update may result in more than
+one transition.
+
+Usage:
+...............
+wdiff <pe> <pe> [status]
+
+pe :: <number>|<index>|<file>|live
+...............
+Examples:
+...............
+wdiff 2066 2067
+wdiff pe-input-2080.bz2 live status
+...............
+
+[[cmdhelp_root_report,Create cluster status report,From Code]]
+=== `report`
+See "crm help report" or "crm report --help"
+
+=== `end` (`cd`, `up`)
+
+The `end` command ends the current level and the user moves to
+the parent level. This command is available everywhere.
+
+Usage:
+...............
+end
+...............
+
+=== `help`
+
+The `help` command prints help for the current level or for the
+specified topic (command). This command is available everywhere.
+
+Usage:
+...............
+help [<topic>]
+...............
+
+=== `quit` (`exit`, `bye`)
+
+Leave the program.
+
+FILES
+---
+include::profiles.adoc[]
+
+BUGS
+----
+Even though all sensible configurations (and most of those that
+are not) are going to be supported by the crm shell, I suspect
+that it may still happen that certain XML constructs may confuse
+the tool. When that happens, please file a bug report.
+
+The crm shell will not try to update the objects it does not
+understand. Of course, it is always possible to edit such objects
+in the XML format.
+
+AUTHORS
+-------
+Dejan Muhamedagic, <dejan@suse.de>
+Kristoffer Gronlund <kgronlund@suse.com>
+and many OTHERS
+
+SEE ALSO
+--------
+crm_resource(8), crm_attribute(8), crm_mon(8), cib_shadow(8),
+ptest(8), dotty(1), crm_simulate(8), cibadmin(8)
+
+
+COPYING
+-------
+Copyright \(C) 2008-2013 Dejan Muhamedagic.
+Copyright \(C) 2013 Kristoffer Gronlund.
+
+Free use of this software is granted under the terms of the GNU General Public License (GPL).
+
+//////////////////////
+ vim:ts=4:sw=4:expandtab:
+//////////////////////
diff --git a/doc/crmsh_crm_report.8.adoc b/doc/crmsh_crm_report.8.adoc
new file mode 100644
index 0000000..9e5e91c
--- /dev/null
+++ b/doc/crmsh_crm_report.8.adoc
@@ -0,0 +1,15 @@
+:man source: crmsh_crm_report
+:man version: 4.6.0
+:man manual: crmsh documentation
+
+crmsh_crm_report(8)
+==================
+
+NAME
+----
+crmsh_crm_report - create report for CRM based clusters (Pacemaker)
+
+
+SEE ALSO
+--------
+See "crm help report" or "crm report --help"
diff --git a/doc/development.md b/doc/development.md
new file mode 100644
index 0000000..2406b4d
--- /dev/null
+++ b/doc/development.md
@@ -0,0 +1,314 @@
+# Notes for developers and contributors
+
+This is mostly a list of notes that Dejan prepared for me when I
+started working on crmsh (me being Kristoffer). I've decided to update
+it at least enough to not be completely outdated, so the information
+here should be mostly up-to-date for crmsh 2.1.
+
+## data-manifest
+
+This file contains a list of all shared data files to install.
+
+Whenever a file that is to be installed to `/usr/share/crmsh` is added,
+for example a cluster script or crmsh template, the `data-manifest`
+file needs to be regenerated, by running `./update-data-manifest.sh`.
+
+## Website
+
+To build the website, you will need **Asciidoc**, **Pygments** plus
+two special lexers for Pygments installed as a separate module. This
+module is included in the source tree for crmsh under `contrib`. To
+install the module and build the website, do the following:
+
+```
+cd contrib
+sudo python setup.py install
+cd ..
+cd doc/website-v1
+make
+```
+
+If everything worked out as it should, the website should now be
+generated in `doc/website-v1/gen`.
+
+## Test suite
+
+There are two separate test suites for crmsh:
+
+* `test/unittests` - These are unit tests that test small pieces of
+ code or functionality. To run these tests, run the `test/run` script
+ from the project root.
+
+* `test/testcases` - These are larger integration tests which require
+ a Pacemaker installation on the machine where the tests are to
+ run. Usually, we run these tests using the OBS and the `osc` command
+ line tool:
+
+ 1. Check out the crmsh python package to a directory (usually
+ `~/build-service/network:ha-clustering:Factory/crmsh`)
+
+ 2. Replace the tarball for crmsh in the OBS project with an archive
+ built from the current source tree. Replace the version number with
+ whatever version is the current one on OBS:
+
+ git archive --format=tar --prefix=crmsh-2.3.0+git.1470991992.7deaa3a/ -o <tmpdir>/crmsh-2.3.0+git.1470991992.7deaa3a.tar HEAD
+ bzip2 <tmpdir>/crmsh-2.3.0+git.1470991992.7deaa3a.tar
+ cp <tmpdir>/crmsh-2.3.0+git.1470991992.7deaa3a.tar.bz2 ~/build-service/network:ha-clustering:Factory/crmsh/crmsh-2.3.0+git.1470991992.7deaa3a.tar.bz2
+
+ 3. Build the rpm package for crmsh with the `with_regression_tests`
+ flag set to 1:
+
+ cd ~/build-service/network:ha-clustering:Factory/crmsh
+ osc build -d --no-verify --release=1 --define with_regression_tests 1 openSUSE_Tumbleweed x86_64 crmsh.spec
+
+To simplify this process, there is a utility called `obs` which can be
+downloaded here: https://github.com/krig/obs-scripts
+
+Using the `obs` script, the above is reduced to calling `obs test
+factory`, given an appropriate `obs.conf` file. See the README in the
+obs-scripts project for more details on using `obs`.
+
+## Modules
+
+This is the list of all modules including short descriptions.
+
+- `crm`
+
+ The program. Tries to detect incompatible python versions or a
+ missing crmsh module, and report an understandable error message
+ in either case.
+
+- `crmsh/main.py`
+
+ This is where execution really starts. Verifies the environment
+ and detects the pacemaker version.
+
+- `crmsh/config.py`
+
+ Reads the `crm.conf` configuration file and tries to detect basic
+ information about where pacemaker is located etc. Some magic is
+ used to generate an object hierarchy based on the configuration,
+ so that the rest of the code can access configuration variables
+ directly.
+
+- `crmsh/constants.py`
+
+ Various hard-coded constants. Many of these should probably be
+ read from pacemaker metadata for better compatibility across
+ different versions.
+
+- `crmsh/ui_*.py`
+
+ The UI context (`ui_context.py`) parses the input command and
+ keeps track of which is the current level in the UI. `ui_root.py`
+ is the root of the UI hierarchy.
+
+- `crmsh/help.py`
+
+ Reads help from a text file and presents parts of it in
+ response to the help command. The text file has special
+ anchors to demarcate help topics and command help text.
+
+- `doc/crm.8.adoc`
+
+ Online help in asciidoc format. Several help topics (search
+ for +[[topic_+) and command reference (search for
+ +[[cmdhelp_+). Every user interface change needs to be
+ reflected here. _Actually, every user interface change has to
+ start here_. A source for the +crm(8)+ man page too.
+
+- `crmsh/cibconfig.py`
+
+ Configuration (CIB) manager. Implements the configure level.
+ The bigest and the most complex part. There are three major
+ classes:
+
+ - +CibFactory+: operations on the CIB or parts of it.
+
+ - +CibObject+: every CIB element is implemented in a
+ subclass of +CibObject+. The configuration consists of a
+ set of +CibObject+ instances (subclassed, e.g. +CibNode+ or
+ +CibPrimitive+).
+
+ - +CibObjectSet+: enables operations on sets of CIB
+ elements. Two subclasses with CLI and XML presentations
+ of cib elements. Most operations are going via these
+ subclasses (+show+, +edit+, +save+, +filter+).
+
+- `crmsh/scripts.py`
+
+ Implements the cluster scripts. Reads multiple kinds of script
+ definition languages including the XML wizard format used by
+ Hawk.
+
+- `crmsh/handles.py`
+
+ A primitive handlebar-style templating language used in cluster
+ scripts.
+
+- `crmsh/idmgmt.py`
+
+ CIB id management. Guarantees that all ids are unique.
+ A helper for CibFactory.
+
+- `crmsh/parse.py`
+
+ Parses CLI -> XML.
+
+- `crmsh/cliformat.py`
+
+ Parses XML -> CLI.
+
+ Not as cleanly separated as the CLI parser, mostly a set of
+ functions called from `cibconfig.py`.
+
+- `crmsh/clidisplay.py`, `crmsh/term.py`
+
+ Applies colors to terminal output.
+
+- `crmsh/crm_gv.py`
+
+ Interface to GraphViz. Generates graph specs for dotty(1).
+
+- `crmsh/cibstatus.py`
+
+ CIB status section editor and manipulator (cibstatus
+ level). Interface to crm_simulate.
+
+- `crmsh/ra.py`
+
+ Resource agents interface.
+
+- `crmsh/rsctest.py`
+
+ Resource tester (configure rsctest command).
+
+- `crmsh/history.py`
+
+ Cluster history. Interface to logs and other artifacts left
+ on disk by the cluster.
+
+- `crmsh/log_patterns.py`, `log_patterns_118.py`
+
+ Pacemaker subsystems' log patterns. For versions earlier than
+ 1.1.8 and the latter.
+
+- `crmsh/schema.py`, `pacemaker.py`
+
+ Support for pacemaker RNG schema.
+
+- `crmsh/cache.py`
+
+ A very rudimentary cache implementation. Used to cache
+ results of expensive operations (i.e. ra meta).
+
+- `crmsh/crm_pssh.py`
+
+ Interface to the parallax library for remote SSH commands.
+
+- `crmsh/corosync.py`
+
+ Parse and edit the `corosync.conf` configuration file.
+
+- `crmsh/msg.py`
+
+ Messages for users. Can count lines and include line
+ numbers. Needs refinement.
+
+- `crmsh/utils.py`
+
+ A bag of useful functions. Needs more order.
+
+- `crmsh/xmlutil.py`
+
+ A bag of useful XML functions. Needs more order.
+
+## Code improvements / TODO
+
+These are some thoughts on how to improve maintainability and
+make crmsh nicer. Mostly for people looking at the code, the
+users shouldn't notice much (or any) difference.
+
+Everybody's invited to comment and make further suggestions, in
+particular experienced pythonistas.
+
+### Syntax bug with automatic constraint handling
+
+See issue on github https://github.com/ClusterLabs/crmsh/issues/140 .
+
+The problem is the sequence of modifications: crmsh tries to be too
+smart and changes the constraint which refers to all members of the
+group so that it now refers to the group. But when the group is
+then deleted, the constraint is also deleted.
+
+### Rewrite the hb_report script completely in Python
+
+Right now, the `hb_report` script is written in bash. This means it
+has some duplicated code, for example finding pacemaker binaries,
+with crmsh. It also means that it can be difficult to debug and
+maintain. It would be better if it was completely implemented in
+Python.
+
+### Python 3 compatibility
+
+The code is currently only compatible with Python 2.6+. We will need
+to port crmsh to Python 3 eventually. The best solution for this is
+probably using the six python library which enables code which is
+both Python 2 and Python 3-compatible.
+
+### Validate more using pacemaker schema
+
+- We have the pacemaker CIB schema available (see schema.py),
+however using it is difficult and so it is not used in enough
+places.
+
+### Investigate switching to python-prompt-toolkit
+
+Either switch crmsh over to using the prompt toolkit for
+implementing the interactive mode, or at least look at it
+to see what ideas we can lift.
+
+https://github.com/jonathanslenders/python-prompt-toolkit
+
+### History transition should be able to save the graph to a file
+
+See https://github.com/ClusterLabs/crmsh/issues/98
+
+### Add support for ordering attribute on resource sets
+
+See https://github.com/ClusterLabs/crmsh/issues/84
+
+### Better version detection
+
+Be better at detecting and handling the Pacemaker version.
+Ensure backwards compatibility, for example with old vs.
+new ACL command syntax.
+
+### Syntax highlighting
+
+- syntax highlighting is done before producing output, which
+ is basically wrong and makes code convoluted; it further
+ makes extra processing more difficult
+
+- use a python library (pygments seems to be the best
+ candidate); that should also allow other output formats
+ (not only terminal)
+
+- how to extend pygments to understand a new language? it'd
+ be good to be able to get this _without_ pushing the parser
+ upstream (that would take _long_ to propagate to
+ distributions)
+
+### CibFactory is huge
+
+- this is a single central CIB class, it'd be good to have it
+ split into several smaller classes (how?)
+
+### The element create/update procedure is complex
+
+- not sure how to improve this
+
+### Bad namespace separation
+
+- xmlutil and utils are just a loose collection of functions,
+need to be organized better (get rid of 'from xyz import *')
diff --git a/doc/profiles.adoc b/doc/profiles.adoc
new file mode 100644
index 0000000..7201a81
--- /dev/null
+++ b/doc/profiles.adoc
@@ -0,0 +1,47 @@
+=== /etc/crm/profiles.yml
+
+==== Purpose
+
+YAML file `/etc/crm/profiles.yml` contains Corosync, SBD and Pacemaker parameters for different platforms.
+
+crmsh bootstrap detects system environment and load the corresponding parameters predefined in this file.
+
+==== Syntax
+
+............
+profile_name:
+ key_name: value
+............
+
+The valid profile names are:
+"microsoft-azure", "google-cloud-platform", "amazon-web-services", "s390", "default"
+
+`key_name` is a known Corosync, SBD, or Pacemaker parameters, like
+`corosync.totem.token` or `sbd.watchdog_timeout`.
+
+More details about the parameter definitions please refer to the man page of corosync.conf(5), sbd(8).
+
+Example
+............
+default:
+ corosync.totem.crypto_hash: sha1
+ corosync.totem.crypto_cipher: aes256
+ corosync.totem.token: 5000
+ corosync.totem.join: 60
+ corosync.totem.max_messages: 20
+ corosync.totem.token_retransmits_before_loss_const: 10
+ sbd.watchdog_timeout: 15
+
+microsoft-azure:
+ corosync.totem.token: 30000
+ sbd.watchdog_timeout: 60
+............
+
+==== How the content of the file is interpreted
+
+The profiles has the following properties:
+
+* Profiles are only loaded on bootstrap init node.
+* The "default" profile is loaded in the beginning.
+* Specific profiles will override the corresponding values in the "default" profile (if the specific environment is detected).
+* Users could customize the "default" profile for their needs. For example, those on-premise environments which is not defined yet.
diff --git a/doc/releasing-a-new-version.md b/doc/releasing-a-new-version.md
new file mode 100644
index 0000000..8141f87
--- /dev/null
+++ b/doc/releasing-a-new-version.md
@@ -0,0 +1,199 @@
+# Releasing a new version
+
+A guide to releasing new versions of crmsh.
+
+## Version scheme
+
+We follow a somewhat loose version of Semantic Versioning, with a
+three-part version number:
+
+ <major>.<minor>.<patch>
+
+The major version number is increased rarely, arbitrarily and
+indicates big changes to the shell. Moving from Python 2 to Python 3
+was such a change, for example. It does not indicate breaking
+changes: We try not to make breaking changes at all. If there is a
+breaking change, it is hopefully a mistake that would be fixed with a
+patch. If not, it should be noted very clearly and probably only
+released with a major version number change.
+
+The minor version number indicates new features and bugfixes, but
+hopefully no breaking changes.
+
+The patch version number indicates bugfixes only, and no breaking
+changes.
+
+## Steps
+
+1. Updating the changelog
+
+In `/ChangeLog`, there is a curated list of changes included in this
+release. This log should be updated based on the git history. Remove
+any updates that are tagged `dev:` or `test:` since these are internal
+changes, and clean up the changelog in any other way you might want.
+
+To get the list of changes since the last release, you can use `git
+log` with a custom format. This example gets the changes between 3.0
+and 3.0.1, filtering out any changes tagged `dev:` or `test:`:
+
+ PAGER=cat git log --format="- %s" 3.0.0..3.0.1 | \
+ grep -Ev -- '-[ ](dev|test):.*'
+
+2. Tagging the release
+
+Using `git tag` you can see the list of existing tags. Depending
+on the version being released, you will want to tag the current commit
+with that release. Make it a signed tag:
+
+ git tag -s -u <user@example.com> 4.1.0
+
+In the tag message I usually just put
+
+ Release 4.1.0
+
+(of course, change `4.1.0` to whatever release it is you are tagging.
+
+Remember to push the tag to the Github repository. Assuming that the
+github repository is `origin`, this command should do the trick:
+
+ git push --tags origin
+
+
+3. Release email
+
+Send a release email to users@clusterlabs.org. Here is the template
+that I usually follow with release emails:
+
+```
+Hello everyone!
+
+I'm happy to announce the release of crmsh version <VERSION>.
+
+<DESCRIPTION: some notes about the release>
+
+There are some other changes in this release as well, see the
+ChangeLog for the complete list of changes:
+
+* https://github.com/ClusterLabs/crmsh/blob/<VERSION>/ChangeLog
+
+The source code can be downloaded from Github:
+
+* https://github.com/ClusterLabs/crmsh/releases/tag/<VERSION>
+
+Packaged versions of crmsh should be available shortly from your
+distribution of choice. Development packages for openSUSE Tumbleweed
+are available from the Open Build System, here:
+
+* http://download.opensuse.org/repositories/network:/ha-clustering:/Factory/
+
+Archives of the tagged release:
+
+* https://github.com/ClusterLabs/crmsh/archive/<VERSION>.tar.gz
+* https://github.com/ClusterLabs/crmsh/archive/<VERSION>.zip
+
+As usual, a huge thank you to all contributors and users of crmsh!
+
+```
+
+4. Website update
+
+The crmsh website is hosted on Github as a github page. The URL to the
+website repository is
+
+ https://github.com/crmsh/crmsh.github.io
+
+The website contents themselves (the sources) are found in the regular
+crmsh repository, in the `/doc/website-v1` folder. There is a
+`Makefile` in this folder which can be used to regenerate the
+website.
+
+Doing this requires `asciidoc` and `Pygments` to be installed, as well
+as the custom Pygments filter `ansiclr`.
+
+`ansiclr` is found in the `/contrib/pygments_crmsh_lexers` folder, and
+can be installed by running `python setup.py install` in the
+`/contrib` folder.
+
+**Note: ansiclr seems to be broken at the moment. Just ignore
+it. Everything should still work except some highlighting.**
+
+To create the news update, copy a previous update (found in
+`/doc/website-v1/news`), rename it to an appropriate name based on the
+current date, and replace the contents based on the announcement
+email.
+
+Remember to update the title, author and date information at the top
+of the news entry, to ensure that it appears correctly on the site.
+
+To generate the site including the new entry, run
+
+ make
+
+The new site should now sit in `/doc/website-v1/gen`. To update the
+site, using rsync should work:
+
+ rsync -Pavz doc/website-v1/gen/ <path-to-website-checkout>/crmsh.github.io/
+
+5. Update `network:ha-clustering:Factory`
+
+On the Open Build Service, the project
+`network:ha-clustering:Factory/crmsh` is used as the development
+project for openSUSE Tumbleweed. This project mirrors the state of the
+`master` branch in crmsh, but for policy reasons it is not
+automatically updated.
+
+The following steps assumes that you are a maintainer of
+`network:ha-clustering:Factory`. If not, you can still make the update
+but you will have to branch the `crmsh` package, make the update there
+and then submit an update request using `osc submit`. Then a
+maintainer will have to review your submission.
+
+To update the package and submit to `openSUSE:Factory`, the following
+steps will do the trick. First, check out a local copy of the crmsh
+project:
+
+ osc co network:ha-clustering:Factory crmsh
+ cd network:ha-clustering:Factory/crmsh
+
+If you already have a copy, make sure it is up to date:
+
+ osc update
+
+Update the `_service` file so that the version number reflects the
+latest version of the `master` branch in git.
+
+Pull in the latest changes from git:
+
+ osc service dr
+
+This will update the spec file and the changes file. Clean up the
+changes file if you want (not strictly neccessary), and remove any old
+tarball still in the package directory, and then add/remove the
+changes to osc:
+
+ osc ar
+
+Check that everything looks good, and then commit:
+
+ osc diff
+ osc commit
+
+Once the package has built successfully on OBS, you can submit to
+`openSUSE:Factory`:
+
+ osc submit
+
+6. Update `network:ha-clustering:Stable`
+
+If this is a minor release for the latest major release, or a new
+major release, we should update the version of crmsh hosted at
+`network:ha-clustering:Stable` on OBS.
+
+Since `master` is probably up to date with this latest version, doing
+so should be as simple as submitting crmsh from
+`network:ha-clustering:Factory` to `network:ha-clustering:Stable`.
+
+Once the release is tagged, the announcement email is sent, the
+website updated and the packages updated, the release is done.
+
+Congratulations!
diff --git a/doc/sort-doc.py b/doc/sort-doc.py
new file mode 100644
index 0000000..87c35a2
--- /dev/null
+++ b/doc/sort-doc.py
@@ -0,0 +1,82 @@
+# Tool to sort the documentation alphabetically
+# Makes a lot of assumptions about the structure of the document it edits
+# Looks for special markers that indicate structure
+
+# prints output to stdout
+
+# print lines until in a cmdhelp_<section>
+# collect all cmdhelp_<section>_<subsection> subsections
+# sort and print
+
+import sys
+import re
+
+
+class Sorter(object):
+ def __init__(self):
+ self.current_section = None
+ self.current_subsection = None
+ self.subsections = []
+ self.re_section = re.compile(r'^\[\[cmdhelp_([^_,]+),')
+ self.re_subsection = re.compile(r'^\[\[cmdhelp_([^_]+)_([^,]+),')
+
+ def beginsection(self, line):
+ m = self.re_section.match(line)
+ name = m.group(1)
+ self.current_section = [name, line]
+ self.current_subsection = None
+ self.subsections = []
+ return self.insection
+
+ def insection(self, line):
+ if line.startswith('[[cmdhelp_%s_' % (self.current_section[0])):
+ return self.beginsubsection(line)
+ elif line.startswith('[['):
+ self.finishsection()
+ return self.preprint(line)
+ else:
+ self.current_section[1] += line
+ return self.insection
+
+ def beginsubsection(self, line):
+ m = self.re_subsection.match(line)
+ name = m.group(2)
+ self.current_subsection = [name, line]
+ return self.insubsection
+
+ def insubsection(self, line):
+ if line.startswith('[['):
+ self.subsections.append(self.current_subsection)
+ self.current_subsection = None
+ return self.insection(line)
+ self.current_subsection[1] += line
+ return self.insubsection
+
+ def finishsection(self):
+ if self.current_section:
+ print self.current_section[1],
+ for name, text in sorted(self.subsections, key=lambda x: x[0]):
+ print text,
+ self.current_section = None
+ self.subsections = []
+
+ def preprint(self, line):
+ if self.re_section.match(line):
+ return self.beginsection(line)
+ print line,
+ return self.preprint
+
+ def run(self, lines):
+ action = self.preprint
+ for line in lines:
+ prevaction = action
+ action = action(line)
+ if action is None:
+ print prevaction
+ print self.current_section
+ print self.current_subsection
+ sys.exit(1)
+ if self.current_section:
+ self.finishsection()
+
+Sorter().run(open(sys.argv[1]).readlines())
diff --git a/doc/website-v1/404.adoc b/doc/website-v1/404.adoc
new file mode 100644
index 0000000..926d803
--- /dev/null
+++ b/doc/website-v1/404.adoc
@@ -0,0 +1,9 @@
+404: Page not found
+===================
+
+Apologies, but there is nothing here!
+
+The page you are looking for may have moved.
+
+* link:/documentation[Documentation]
+* link:/faq[Frequently Asked Questions]
diff --git a/doc/website-v1/Makefile b/doc/website-v1/Makefile
new file mode 100644
index 0000000..8ea2f46
--- /dev/null
+++ b/doc/website-v1/Makefile
@@ -0,0 +1,145 @@
+ASCIIDOC := asciidoc
+CRMCONF := crm.conf
+SRC := faq.adoc documentation.adoc development.adoc installation.adoc \
+ configuration.adoc about.adoc rsctest-guide.adoc download.adoc \
+ history-guide.adoc start-guide.adoc man-1.2.adoc scripts.adoc man-2.0.adoc man-3.adoc man-4.3.adoc
+HISTORY_LISTINGS = include/history-guide/nfs-probe-err.typescript \
+ include/history-guide/sample-cluster.conf.crm \
+ include/history-guide/status-probe-fail.typescript \
+ include/history-guide/resource-trace.typescript \
+ include/history-guide/stonith-corosync-stopped.typescript \
+ include/history-guide/basic-transition.typescript \
+ include/history-guide/diff.typescript \
+ include/history-guide/info.typescript \
+ include/history-guide/resource.typescript \
+ include/history-guide/transition-log.typescript
+TGT := $(patsubst %.adoc,gen/%/index.html,$(SRC))
+CSS := css/crm.css css/font-awesome.min.css
+CSS := $(patsubst %,gen/%,$(CSS))
+ICONS := \
+ img/icons/caution.png \
+ img/icons/example.png \
+ img/icons/home.png \
+ img/icons/important.png \
+ img/icons/next.png \
+ img/icons/note.png \
+ img/icons/prev.png \
+ img/icons/tip.png \
+ img/icons/up.png \
+ img/icons/warning.png \
+ img/icons/callouts/10.png \
+ img/icons/callouts/11.png \
+ img/icons/callouts/12.png \
+ img/icons/callouts/13.png \
+ img/icons/callouts/14.png \
+ img/icons/callouts/15.png \
+ img/icons/callouts/1.png \
+ img/icons/callouts/2.png \
+ img/icons/callouts/3.png \
+ img/icons/callouts/4.png \
+ img/icons/callouts/5.png \
+ img/icons/callouts/6.png \
+ img/icons/callouts/7.png \
+ img/icons/callouts/8.png \
+ img/icons/callouts/9.png
+IMG := $(ICONS) img/loader.gif img/laptop.png img/servers.gif \
+ img/history-guide/sample-cluster.conf.png \
+ img/history-guide/smallapache-start.png
+IMG := $(patsubst %,gen/%,$(IMG))
+FONTS := fonts/FontAwesome.otf fonts/fontawesome-webfont.eot \
+ fonts/fontawesome-webfont.svg fonts/fontawesome-webfont.ttf \
+ fonts/fontawesome-webfont.woff
+FONTS := $(patsubst %,gen/%,$(FONTS))
+WATCHDIR := watchdir
+XDGOPEN := xdg-open
+NEWS := $(wildcard news/*.adoc)
+NEWSDOC := $(patsubst %.adoc,gen/%/index.html,$(NEWS))
+
+.PHONY: all clean deploy open
+
+all: site
+
+gen/index.html: index.adoc $(CRMCONF)
+ @mkdir -p $(dir $@)
+ @$(ASCIIDOC) --unsafe -b html5 -a icons -a iconsdir=/img/icons -f $(CRMCONF) -o $@ $<
+ @python ./postprocess.py -o $@ $<
+
+gen/%/index.html: %.adoc $(CRMCONF)
+ @mkdir -p $(dir $@)
+ @$(ASCIIDOC) --unsafe -b html5 -a icons -a iconsdir=/img/icons -f $(CRMCONF) -o $@ $<
+ @python ./postprocess.py -o $@ $<
+
+gen/history-guide/index.html: $(HISTORY_LISTINGS)
+
+gen/man/index.html: ../crm.8.adoc $(CRMCONF)
+ @mkdir -p $(dir $@)
+ @$(ASCIIDOC) --unsafe -b html5 -f $(CRMCONF) -o $@ $<
+ @python ./postprocess.py -o $@ $<
+
+gen/404.html: 404.adoc $(CRMCONF)
+ @mkdir -p $(dir $@)
+ @$(ASCIIDOC) --unsafe -b html5 -f $(CRMCONF) -o $@ $<
+ @python ./postprocess.py -o $@ $<
+
+news.adoc: $(NEWS) $(CRMCONF)
+ @echo "news:" $(NEWS)
+ python ./make-news.py $@ $(NEWS)
+
+gen/news/index.html: news.adoc
+ @mkdir -p $(dir $@)
+ $(ASCIIDOC) --unsafe -b html5 -f $(CRMCONF) -o $@ $<
+ @python ./postprocess.py -o $@ $<
+
+gen/css/%.css: css/%.css
+ @mkdir -p gen/css
+ @cp -r $< $@
+ @echo "+ $@"
+
+gen/js/%.js: js/%.js
+ @mkdir -p gen/js
+ @cp -r $< $@
+ @echo "+ $@"
+
+gen/img/icons/callouts/%: img/icons/callouts/%
+ @mkdir -p gen/img/icons/callouts
+ @cp -r $< $@
+ @echo "+ $@"
+
+gen/img/icons/%: img/icons/%
+ @mkdir -p gen/img/icons
+ @cp -r $< $@
+ @echo "+ $@"
+
+gen/img/history-guide/%: img/history-guide/%
+ @mkdir -p gen/img/history-guide
+ @cp -r $< $@
+ @echo "+ $@"
+
+gen/img/%: img/%
+ @mkdir -p gen/img
+ @cp -r $< $@
+ @echo "+ $@"
+
+gen/fonts/%: fonts/%
+ @mkdir -p gen/fonts
+ @cp -r $< $@
+ @echo "+ $@"
+
+gen/atom.xml: $(NEWSDOC)
+ @echo "atom:" $(NEWSDOC)
+ python ./make-news.py gen/atom.xml $(NEWS)
+
+site: gen/atom.xml gen/index.html gen/404.html gen/news/index.html gen/man/index.html $(TGT) $(CSS) $(IMG) $(FONTS) $(NEWSDOC)
+ @which dos2unix >/dev/null && find gen -name "*.html" -type f -exec dos2unix {} \;
+
+deploy: site
+ @echo "TODO: CVS upload"
+
+open: site
+ @$(XDGOPEN) gen/index.html
+
+watch:
+ @$(WATCHDIR) --verbose --cmd "make" . css img fonts
+
+clean:
+ -@$(RM) -rf gen/* news.adoc
diff --git a/doc/website-v1/about.adoc b/doc/website-v1/about.adoc
new file mode 100644
index 0000000..2656625
--- /dev/null
+++ b/doc/website-v1/about.adoc
@@ -0,0 +1,19 @@
+= About =
+
+== Authors ==
+
+include::../../AUTHORS[]
+
+== Site ==
+
+This site was generated from http://asciidoc.org[AsciiDoc] sources.
+
+The CSS for this site started as a clone of the +bare+ theme by https://github.com/rtomayko/adoc-themes[Ryan Tomayko].
+
+Fonts used are https://www.google.com/fonts/specimen/Open+Sans[Open Sans] and http://fontawesome.io[Font Awesome].
+
+== License ==
+
+`crmsh` is licensed under the GNU General Public License (GPL).
+
+For more information, see https://gnu.org/licenses/gpl.html
diff --git a/doc/website-v1/configuration.adoc b/doc/website-v1/configuration.adoc
new file mode 100644
index 0000000..fb48c93
--- /dev/null
+++ b/doc/website-v1/configuration.adoc
@@ -0,0 +1,132 @@
+= Configuration =
+
+.Version information
+NOTE: This section applies to `crmsh 2.0+` only.
+
+
+`crm` can be configured using both a system-wide configuration file,
+and a per-user configuration file. The values set in the user-local
+file take precedence over the system-wide settings.
+
+The global configuration file is usually installed at
+`/etc/crm/crm.conf`, and the user-local configuration file at
+`~/.config/crm/crm.conf`.
+
+
+== Upgrading from crm 1.x to 2.x ==
+
+The configuration file format and location changed significantly going
+from crm 1.x to 2.x. If `crm` cannot find a user-local configuration
+file when starting up, it will look for an old-style configuration
+file at `~/.crm.rc`. If this file exists, `crm` will prompt the user
+asking if the old-style configuration should be automatically
+converted to a new-style configuration file.
+
+
+== Format description ==
+
+The `settings` file consists of sections introduced by a `[section]`
+header, and followed by `name=value` pairs.
+
+Leading whitespace is stripped from values.
+
+Values can contain format strings referring to other values in the
+same section.
+
+Lines starting with `#` or `;` are interpreted as comments.
+
+Values starting with `$` are interpreted as environment variable
+references, and the value will be retrieved from the named environment
+variable if set.
+
+== Example configuration ==
+
+The example configuration below lists all available options and their
+default values.
+
+----------------------
+[core]
+editor = $EDITOR
+pager = $PAGER
+user =
+skill_level = expert
+sort_elements = yes
+check_frequency = always
+check_mode = strict
+wait = no
+add_quotes = yes
+manage_children = ask
+force = no
+debug = no
+ptest = ptest, crm_simulate
+dotty = dotty
+dot = dot
+
+[path]
+sharedir = /usr/share/crmsh
+cache = /var/cache/crm
+crm_config = /var/lib/pacemaker/cib
+crm_daemon_dir = /usr/lib64/pacemaker
+crm_daemon_user = hacluster
+ocf_root = /usr/lib/ocf
+crm_dtd_dir = /usr/share/pacemaker
+pe_state_dir = /var/lib/pacemaker/pengine
+heartbeat_dir = /var/lib/heartbeat
+hb_delnode = /usr/share/heartbeat/hb_delnode
+nagios_plugins = /usr/lib64/nagios/plugins
+
+[color]
+style = color
+error = red bold
+ok = green bold
+warn = yellow bold
+info = cyan
+help_keyword = blue bold underline
+help_header = normal bold
+help_topic = yellow bold
+help_block = cyan
+keyword = yellow
+identifier = normal
+attr_name = cyan
+attr_value = red
+resource_reference = green
+id_reference = green
+score = magenta
+ticket = magenta
+----------------------
+
+
+== Loading and saving options ==
+
+Options are loaded from the global configuration file first, and the
+user-local file second. This means that the user-local options take
+precedence over the global configuration.
+
+When changing an option using the `options` sublevel, the
+configuration file is written to disk with the new value.
+
+== Syntax highlighting ==
+
+By default, `crm` will try to syntax highlight its output when
+connected to a TTY. To disable this behavior, set the configuration
+value `style = none` in the `[color]` section.
+
+The available color choices may depend on the terminal used, but
+normally include the following:
+
+----
+black blue green cyan red magenta yellow white
+----
+
+Colors can be combined with styles:
+
+----
+bold blink dim reverse underline normal
+----
+
+== Setting options from the interactive shell ==
+
+Options can be set directly from the interactive shell using the
+`options` sublevel. These options will be written to the per-user
+configuration file. Note that changing an option in this way may erase
+comments added to the configuration file.
diff --git a/doc/website-v1/crm.conf b/doc/website-v1/crm.conf
new file mode 100644
index 0000000..d1502cf
--- /dev/null
+++ b/doc/website-v1/crm.conf
@@ -0,0 +1,601 @@
+#
+# html5.conf
+#
+# Asciidoc configuration file.
+# html5 backend.
+#
+
+[miscellaneous]
+outfilesuffix=.html
+
+[attributes]
+basebackend=html
+basebackend-html=
+basebackend-html5=
+b
+[replacements2]
+# Line break.
+(?m)^(.*)\s\+$=\1<br>
+
+[replacements]
+ifdef::asciidoc7compatible[]
+# Superscripts.
+\^(.+?)\^=<sup>\1</sup>
+# Subscripts.
+~(.+?)~=<sub>\1</sub>
+endif::asciidoc7compatible[]
+
+[ruler-blockmacro]
+<hr>
+
+[pagebreak-blockmacro]
+<div style="page-break-after:always"></div>
+
+[blockdef-pass]
+asciimath-style=template="asciimathblock",subs=()
+latexmath-style=template="latexmathblock",subs=()
+
+[macros]
+(?u)^(?P<name>audio|video)::(?P<target>\S*?)(\[(?P<attrlist>.*?)\])$=#
+# math macros.
+# Special characters are escaped in HTML math markup.
+(?su)[\\]?(?P<name>asciimath|latexmath):(?P<subslist>\S*?)\[(?P<passtext>.*?)(?<!\\)\]=[specialcharacters]
+(?u)^(?P<name>asciimath|latexmath)::(?P<subslist>\S*?)(\[(?P<passtext>.*?)\])$=#[specialcharacters]
+
+[asciimath-inlinemacro]
+`{passtext}`
+
+[asciimath-blockmacro]
+<div class="mathblock{role? {role}}{unbreakable-option? unbreakable}"{id? id="{id}"}>
+<div class="content">
+<div class="title">{title}</div>
+`{passtext}`
+</div></div>
+
+[asciimathblock]
+<div class="mathblock{role? {role}}{unbreakable-option? unbreakable}"{id? id="{id}"}>
+<div class="content">
+<div class="title">{title}</div>
+`|`
+</div></div>
+
+[latexmath-inlinemacro]
+{passtext}
+
+[latexmath-blockmacro]
+<div class="mathblock{role? {role}}{unbreakable-option? unbreakable}"{id? id="{id}"}>
+<div class="content">
+<div class="title">{title}</div>
+{passtext}
+</div></div>
+
+[latexmathblock]
+<div class="mathblock{role? {role}}{unbreakable-option? unbreakable}"{id? id="{id}"}>
+<div class="content">
+<div class="title">{title}</div>
+|
+</div></div>
+
+[image-inlinemacro]
+<span class="image{role? {role}}">
+<a class="image" href="{link}">
+{data-uri%}<img src="{imagesdir=}{imagesdir?/}{target}" alt="{alt={target}}"{width? width="{width}"}{height? height="{height}"}{title? title="{title}"}>
+{data-uri#}<img alt="{alt={target}}"{width? width="{width}"}{height? height="{height}"}{title? title="{title}"}
+{data-uri#}{sys:"{python}" -u -c "import mimetypes,base64,sys; print 'src=\"data:'+mimetypes.guess_type(r'{target}')[0]+';base64,'; base64.encode(sys.stdin,sys.stdout)" < "{eval:os.path.join(r"{indir={outdir}}",r"{imagesdir=}",r"{target}")}"}">
+{link#}</a>
+</span>
+
+[image-blockmacro]
+<div class="imageblock{style? {style}}{role? {role}}{unbreakable-option? unbreakable}"{id? id="{id}"}{align? style="text-align:{align};"}{float? style="float:{float};"}>
+<div class="content">
+<a class="image" href="{link}">
+{data-uri%}<img src="{imagesdir=}{imagesdir?/}{target}" alt="{alt={target}}"{width? width="{width}"}{height? height="{height}"}>
+{data-uri#}<img alt="{alt={target}}"{width? width="{width}"}{height? height="{height}"}
+{data-uri#}{sys:"{python}" -u -c "import mimetypes,base64,sys; print 'src=\"data:'+mimetypes.guess_type(r'{target}')[0]+';base64,'; base64.encode(sys.stdin,sys.stdout)" < "{eval:os.path.join(r"{indir={outdir}}",r"{imagesdir=}",r"{target}")}"}">
+{link#}</a>
+</div>
+<div class="title">{caption={figure-caption} {counter:figure-number}. }{title}</div>
+</div>
+
+[audio-blockmacro]
+<div class="audioblock{role? {role}}{unbreakable-option? unbreakable}"{id? id="{id}"}>
+<div class="title">{caption=}{title}</div>
+<div class="content">
+<audio src="{imagesdir=}{imagesdir?/}{target}"{autoplay-option? autoplay}{nocontrols-option! controls}{loop-option? loop}>
+Your browser does not support the audio tag.
+</audio>
+</div></div>
+
+[video-blockmacro]
+<div class="videoblock{role? {role}}{unbreakable-option? unbreakable}"{id? id="{id}"}>
+<div class="title">{caption=}{title}</div>
+<div class="content">
+<video src="{imagesdir=}{imagesdir?/}{target}"{width? width="{width}"}{height? height="{height}"}{poster? poster="{poster}"}{autoplay-option? autoplay}{nocontrols-option! controls}{loop-option? loop}>
+Your browser does not support the video tag.
+</video>
+</div></div>
+
+[unfloat-blockmacro]
+<div style="clear:both;"></div>
+
+[toc-blockmacro]
+template::[toc]
+
+[indexterm-inlinemacro]
+# Index term.
+{empty}
+
+[indexterm2-inlinemacro]
+# Index term.
+# Single entry index term that is visible in the primary text flow.
+{1}
+
+[footnote-inlinemacro]
+# footnote:[<text>].
+<span class="footnote"><br>[{0}]<br></span>
+
+[footnoteref-inlinemacro]
+# footnoteref:[<id>], create reference to footnote.
+{2%}<span class="footnoteref"><br><a href="#_footnote_{1}">[{1}]</a><br></span>
+# footnoteref:[<id>,<text>], create footnote with ID.
+{2#}<span class="footnote" id="_footnote_{1}"><br>[{2}]<br></span>
+
+[callout-inlinemacro]
+ifndef::icons[]
+<b>&lt;{index}&gt;</b>
+endif::icons[]
+ifdef::icons[]
+ifndef::data-uri[]
+<img src="{icon={iconsdir}/callouts/{index}.png}" alt="{index}">
+endif::data-uri[]
+ifdef::data-uri[]
+<img alt="{index}" src="data:image/png;base64,
+{sys:"{python}" -u -c "import base64,sys; base64.encode(sys.stdin,sys.stdout)" < "{eval:os.path.join(r"{indir={outdir}}",r"{icon={iconsdir}/callouts/{index}.png}")}"}">
+endif::data-uri[]
+endif::icons[]
+
+# Comment line macros.
+[comment-inlinemacro]
+{showcomments#}<br><span class="comment">{passtext}</span><br>
+
+[comment-blockmacro]
+{showcomments#}<p><span class="comment">{passtext}</span></p>
+
+[literal-inlinemacro]
+# Inline literal.
+<span class="monospaced">{passtext}</span>
+
+# List tags.
+[listtags-bulleted]
+list=<div class="ulist{style? {style}}{compact-option? compact}{role? {role}}"{id? id="{id}"}>{title?<div class="title">{title}</div>}<ul>|</ul></div>
+item=<li>|</li>
+text=<p>|</p>
+
+[listtags-numbered]
+# The start attribute is not valid XHTML 1.1 but all browsers support it.
+list=<div class="olist{style? {style}}{compact-option? compact}{role? {role}}"{id? id="{id}"}>{title?<div class="title">{title}</div>}<ol class="{style}"{start? start="{start}"}>|</ol></div>
+item=<li>|</li>
+text=<p>|</p>
+
+[listtags-labeled]
+list=<div class="dlist{compact-option? compact}{role? {role}}"{id? id="{id}"}>{title?<div class="title">{title}</div>}<dl>|</dl></div>
+entry=
+label=
+term=<dt class="hdlist1{strong-option? strong}">|</dt>
+item=<dd>|</dd>
+text=<p>|</p>
+
+[listtags-horizontal]
+list=<div class="hdlist{compact-option? compact}{role? {role}}"{id? id="{id}"}>{title?<div class="title">{title}</div>}<table>{labelwidth?<col width="{labelwidth}%">}{itemwidth?<col width="{itemwidth}%">}|</table></div>
+label=<td class="hdlist1{strong-option? strong}">|</td>
+term=|<br>
+entry=<tr>|</tr>
+item=<td class="hdlist2">|</td>
+text=<p style="margin-top: 0;">|</p>
+
+[listtags-qanda]
+list=<div class="qlist{style? {style}}{role? {role}}"{id? id="{id}"}>{title?<div class="title">{title}</div>}<ol>|</ol></div>
+entry=<li>|</li>
+label=
+term=<p><em>|</em></p>
+item=
+text=<p>|</p>
+
+[listtags-callout]
+ifndef::icons[]
+list=<div class="colist{style? {style}}{role? {role}}"{id? id="{id}"}>{title?<div class="title">{title}</div>}<ol>|</ol></div>
+item=<li>|</li>
+text=<p>|</p>
+endif::icons[]
+ifdef::icons[]
+list=<div class="colist{style? {style}}{role? {role}}"{id? id="{id}"}>{title?<div class="title">{title}</div>}<table>|</table></div>
+ifndef::data-uri[]
+item=<tr><td><img src="{iconsdir}/callouts/{listindex}.png" alt="{listindex}"></td><td>|</td></tr>
+endif::data-uri[]
+ifdef::data-uri[]
+item=<tr><td><img alt="{listindex}" src="data:image/png;base64, {sys:"{python}" -u -c "import base64,sys; base64.encode(sys.stdin,sys.stdout)" < "{eval:os.path.join(r"{indir={outdir}}",r"{icon={iconsdir}/callouts/{listindex}.png}")}"}"></td><td>|</td></tr>
+endif::data-uri[]
+text=|
+endif::icons[]
+
+[listtags-glossary]
+list=<div class="dlist{style? {style}}{role? {role}}"{id? id="{id}"}>{title?<div class="title">{title}</div>}<dl>|</dl></div>
+label=
+entry=
+term=<dt>|</dt>
+item=<dd>|</dd>
+text=<p>|</p>
+
+[listtags-bibliography]
+list=<div class="ulist{style? {style}}{role? {role}}"{id? id="{id}"}>{title?<div class="title">{title}</div>}<ul>|</ul></div>
+item=<li>|</li>
+text=<p>|</p>
+
+[tags]
+# Quoted text.
+emphasis=<em>{1?<span class="{1}">}|{1?</span>}</em>
+strong=<strong>{1?<span class="{1}">}|{1?</span>}</strong>
+monospaced=<span class="monospaced{1? {1}}">|</span>
+singlequoted={lsquo}{1?<span class="{1}">}|{1?</span>}{rsquo}
+doublequoted={ldquo}{1?<span class="{1}">}|{1?</span>}{rdquo}
+unquoted={1?<span class="{1}">}|{1?</span>}
+superscript=<sup>{1?<span class="{1}">}|{1?</span>}</sup>
+subscript=<sub>{1?<span class="{1}">}|{1?</span>}</sub>
+
+ifdef::deprecated-quotes[]
+# Override with deprecated quote attributes.
+emphasis={role?<span class="{role}">}<em{1,2,3? style="}{1?color:{1};}{2?background-color:{2};}{3?font-size:{3}em;}{1,2,3?"}>|</em>{role?</span>}
+strong={role?<span class="{role}">}<strong{1,2,3? style="}{1?color:{1};}{2?background-color:{2};}{3?font-size:{3}em;}{1,2,3?"}>|</strong>{role?</span>}
+monospaced=<span class="monospaced{role? {role}}"{1,2,3? style="}{1?color:{1};}{2?background-color:{2};}{3?font-size:{3}em;}{1,2,3?"}>|</span>
+singlequoted={role?<span class="{role}">}{1,2,3?<span style="}{1?color:{1};}{2?background-color:{2};}{3?font-size:{3}em;}{1,2,3?">}{amp}#8216;|{amp}#8217;{1,2,3?</span>}{role?</span>}
+doublequoted={role?<span class="{role}">}{1,2,3?<span style="}{1?color:{1};}{2?background-color:{2};}{3?font-size:{3}em;}{1,2,3?">}{amp}#8220;|{amp}#8221;{1,2,3?</span>}{role?</span>}
+unquoted={role?<span class="{role}">}{1,2,3?<span style="{1?color:{1};}{2?background-color:{2};}{3?font-size:{3}em;}">}|{1,2,3?</span>}{role?</span>}
+superscript={role?<span class="{role}">}<sup{1,2,3? style="}{1?color:{1};}{2?background-color:{2};}{3?font-size:{3}em;}{1,2,3?"}>|</sup>{role?</span>}
+subscript={role?<span class="{role}">}<sub{1,2,3? style="}{1?color:{1};}{2?background-color:{2};}{3?font-size:{3}em;}{1,2,3?"}>|</sub>{role?</span>}
+endif::deprecated-quotes[]
+
+# Inline macros
+[http-inlinemacro]
+<a href="{name}:{target}">{0={name}:{target}}</a>
+[https-inlinemacro]
+<a href="{name}:{target}">{0={name}:{target}}</a>
+[ftp-inlinemacro]
+<a href="{name}:{target}">{0={name}:{target}}</a>
+[file-inlinemacro]
+<a href="{name}:{target}">{0={name}:{target}}</a>
+[irc-inlinemacro]
+<a href="{name}:{target}">{0={name}:{target}}</a>
+[mailto-inlinemacro]
+<a href="mailto:{target}">{0={target}}</a>
+[link-inlinemacro]
+<a href="{target}">{0={target}}</a>
+[callto-inlinemacro]
+<a href="{name}:{target}">{0={target}}</a>
+# anchor:id[text]
+[anchor-inlinemacro]
+<a id="{target}"></a>
+# [[id,text]]
+[anchor2-inlinemacro]
+<a id="{1}"></a>
+# [[[id]]]
+[anchor3-inlinemacro]
+<a id="{1}"></a>[{1}]
+# xref:id[text]
+[xref-inlinemacro]
+<a href="#{target}">{0=[{target}]}</a>
+# <<id,text>>
+[xref2-inlinemacro]
+<a href="#{1}">{2=[{1}]}</a>
+
+# Special word substitution.
+[emphasizedwords]
+<em>{words}</em>
+[monospacedwords]
+<span class="monospaced">{words}</span>
+[strongwords]
+<strong>{words}</strong>
+
+# Paragraph substitution.
+[paragraph]
+<div class="paragraph{role? {role}}{unbreakable-option? unbreakable}"{id? id="{id}"}>{title?<div class="title">{title}</div>}<p>
+|
+</p></div>
+
+[admonitionparagraph]
+template::[admonitionblock]
+
+# Delimited blocks.
+[listingblock]
+<div class="listingblock{role? {role}}{unbreakable-option? unbreakable}"{id? id="{id}"}>
+<div class="title">{caption=}{title}</div>
+<div class="content monospaced">
+<pre>
+|
+</pre>
+</div></div>
+
+[literalblock]
+<div class="literalblock{role? {role}}{unbreakable-option? unbreakable}"{id? id="{id}"}>
+<div class="title">{title}</div>
+<div class="content monospaced">
+<pre>
+|
+</pre>
+</div></div>
+
+[sidebarblock]
+<div class="sidebarblock{role? {role}}{unbreakable-option? unbreakable}"{id? id="{id}"}>
+<div class="content">
+<div class="title">{title}</div>
+|
+</div></div>
+
+[openblock]
+<div class="openblock{role? {role}}{unbreakable-option? unbreakable}"{id? id="{id}"}>
+<div class="title">{title}</div>
+<div class="content">
+|
+</div></div>
+
+[partintroblock]
+template::[openblock]
+
+[abstractblock]
+template::[quoteblock]
+
+[quoteblock]
+<div class="quoteblock{role? {role}}{unbreakable-option? unbreakable}"{id? id="{id}"}>
+<div class="title">{title}</div>
+<div class="content">
+|
+</div>
+<div class="attribution">
+<em>{citetitle}</em>{attribution?<br>}
+&#8212; {attribution}
+</div></div>
+
+[verseblock]
+<div class="verseblock{role? {role}}{unbreakable-option? unbreakable}"{id? id="{id}"}>
+<div class="title">{title}</div>
+<pre class="content">
+|
+</pre>
+<div class="attribution">
+<em>{citetitle}</em>{attribution?<br>}
+&#8212; {attribution}
+</div></div>
+
+[exampleblock]
+<div class="exampleblock{role? {role}}{unbreakable-option? unbreakable}"{id? id="{id}"}>
+<div class="title">{caption={example-caption} {counter:example-number}. }{title}</div>
+<div class="content">
+|
+</div></div>
+
+[admonitionblock]
+<div class="admonitionblock{role? {role}}{unbreakable-option? unbreakable}"{id? id="{id}"}>
+<table><tr>
+<td class="icon">
+{data-uri%}{icons#}<img src="{icon={iconsdir}/{name}.png}" alt="{caption}">
+{data-uri#}{icons#}<img alt="{caption}" src="data:image/png;base64,
+{data-uri#}{icons#}{sys:"{python}" -u -c "import base64,sys; base64.encode(sys.stdin,sys.stdout)" < "{eval:os.path.join(r"{indir={outdir}}",r"{icon={iconsdir}/{name}.png}")}"}">
+{icons%}<div class="title">{caption}</div>
+</td>
+<td class="content">
+<div class="title">{title}</div>
+|
+</td>
+</tr></table>
+</div>
+
+# Tables.
+[tabletags-default]
+colspec=<col{autowidth-option! style="width:{colpcwidth}%;"}>
+bodyrow=<tr>|</tr>
+headdata=<th class="tableblock halign-{halign=left} valign-{valign=top}" {colspan@1::colspan="{colspan}" }{rowspan@1::rowspan="{rowspan}" }>|</th>
+bodydata=<td class="tableblock halign-{halign=left} valign-{valign=top}" {colspan@1::colspan="{colspan}" }{rowspan@1::rowspan="{rowspan}" }>|</td>
+paragraph=<p class="tableblock">|</p>
+
+[tabletags-header]
+paragraph=<p class="tableblock header">|</p>
+
+[tabletags-emphasis]
+paragraph=<p class="tableblock"><em>|</em></p>
+
+[tabletags-strong]
+paragraph=<p class="tableblock"><strong>|</strong></p>
+
+[tabletags-monospaced]
+paragraph=<p class="tableblock monospaced">|</p>
+
+[tabletags-verse]
+bodydata=<td class="tableblock halign-{halign=left} valign-{valign=top}" {colspan@1::colspan="{colspan}" }{rowspan@1::rowspan="{rowspan}" }><div class="verse">|</div></td>
+paragraph=
+
+[tabletags-literal]
+bodydata=<td class="tableblock halign-{halign=left} valign-{valign=top}" {colspan@1::colspan="{colspan}" }{rowspan@1::rowspan="{rowspan}" }><div class="literal monospaced"><pre>|</pre></div></td>
+paragraph=
+
+[tabletags-asciidoc]
+bodydata=<td class="tableblock halign-{halign=left} valign-{valign=top}" {colspan@1::colspan="{colspan}" }{rowspan@1::rowspan="{rowspan}" }><div>|</div></td>
+paragraph=
+
+[table]
+<table class="tableblock frame-{frame=all} grid-{grid=all}{role? {role}}{unbreakable-option? unbreakable}"{id? id="{id}"}
+style="
+margin-left:{align@left:0}{align@center|right:auto}; margin-right:{align@left|center:auto}{align@right:0};
+float:{float};
+{autowidth-option%}width:{tablepcwidth}%;
+{autowidth-option#}{width#style=width:{tablepcwidth}%;}
+">
+<caption class="title">{caption={table-caption} {counter:table-number}. }{title}</caption>
+{colspecs}
+{headrows#}<thead>
+{headrows}
+{headrows#}</thead>
+{footrows#}<tfoot>
+{footrows}
+{footrows#}</tfoot>
+<tbody>
+{bodyrows}
+</tbody>
+</table>
+
+#--------------------------------------------------------------------
+# Deprecated old table definitions.
+#
+
+[miscellaneous]
+# Screen width in pixels.
+pagewidth=800
+pageunits=px
+
+[old_tabledef-default]
+template=old_table
+colspec=<col style="width:{colwidth}{pageunits};" />
+bodyrow=<tr>|</tr>
+headdata=<th class="tableblock halign-{colalign=left}">|</th>
+footdata=<td class="tableblock halign-{colalign=left}">|</td>
+bodydata=<td class="tableblock halign-{colalign=left}">|</td>
+
+[old_table]
+<table class="tableblock frame-{frame=all} grid-{grid=all}"{id? id="{id}"}>
+<caption class="title">{caption={table-caption}}{title}</caption>
+{colspecs}
+{headrows#}<thead>
+{headrows}
+{headrows#}</thead>
+{footrows#}<tfoot>
+{footrows}
+{footrows#}</tfoot>
+<tbody style="vertical-align:top;">
+{bodyrows}
+</tbody>
+</table>
+
+# End of deprecated old table definitions.
+#--------------------------------------------------------------------
+
+[floatingtitle]
+<h{level@0:1}{level@1:2}{level@2:3}{level@3:4}{level@4:5}{id? id="{id}"} class="float">{title}</h{level@0:1}{level@1:2}{level@2:3}{level@3:4}{level@4:5}>
+
+[preamble]
+# Untitled elements between header and first section title.
+<div id="preamble">
+<div class="sectionbody">
+|
+</div>
+</div>
+
+# Document sections.
+[sect0]
+<h1{id? id="{id}"}>{title}</h1>
+|
+
+[sect1]
+<div class="sect1{style? {style}}{role? {role}}">
+<h2{id? id="{id}"}>{numbered?{sectnum} }{title}</h2>
+<div class="sectionbody">
+|
+</div>
+</div>
+
+[sect2]
+<div class="sect2{style? {style}}{role? {role}}">
+<h3{id? id="{id}"}>{numbered?{sectnum} }{title}</h3>
+|
+</div>
+
+[sect3]
+<div class="sect3{style? {style}}{role? {role}}">
+<h4{id? id="{id}"}>{numbered?{sectnum} }{title}</h4>
+|
+</div>
+
+[sect4]
+<div class="sect4{style? {style}}{role? {role}}">
+<h5{id? id="{id}"}>{title}</h5>
+|
+</div>
+
+[appendix]
+<div class="sect1{style? {style}}{role? {role}}">
+<h2{id? id="{id}"}>{numbered?{sectnum} }{appendix-caption} {counter:appendix-number:A}: {title}</h2>
+<div class="sectionbody">
+|
+</div>
+</div>
+
+[toc]
+<div id="toc">
+ <div id="toctitle">{toc-title}</div>
+ <noscript><p><b>JavaScript must be enabled in your browser to display the table of contents.</b></p></noscript>
+</div>
+
+[header]
+<!DOCTYPE html>
+<html lang="{lang=en}">
+<head>
+<meta http-equiv="Content-Type" content="text/html; charset={encoding}">
+<meta name="generator" content="AsciiDoc {asciidoc-version}">
+<meta name="description" content="{description}">
+<meta name="keywords" content="{keywords}">
+<title>crmsh - {title}</title>
+{title%}<title>crmsh - {doctitle=}</title>
+<link rel="stylesheet" href="/css/font-awesome.min.css">
+<link rel="stylesheet" href="/css/crm.css" type="text/css">
+<link href='//fonts.googleapis.com/css?family=Open+Sans:400,700' rel='stylesheet' type='text/css'>
+<link href="/atom.xml" type="application/atom+xml" rel="alternate" title="crmsh atom feed">
+</head>
+<body>
+<div id="header">
+<h1>
+<a href="/"><span class="fa-stack">
+ <i class="fa fa-square fa-stack-2x"></i>
+ <i class="fa fa-terminal fa-stack-1x fa-inverse"></i>
+</span>crmsh</a>
+</h1>
+
+<div id="topbar-small">
+<ul>
+<li><a href="/news" title="News"><i class="fa fa-rss fa-2x"></i></a></li>
+<li><a href="/documentation" title="Documentation"><i class="fa fa-book fa-2x"></i></a></li>
+<li><a href="/download" title="Download"><i class="fa fa-download fa-2x"></i></a></li>
+<li><a href="/development" title="Development"><i class="fa fa-code-fork fa-2x"></i></a></li>
+<li><a href="/about" title="About"><i class="fa fa-question fa-2x"></i></a></li>
+</ul>
+</div>
+
+<div id="topbar">
+<ul>
+<li><a href="/news">News</a></li>
+<li><a href="/documentation">Documentation</a></li>
+<li><a href="/download">Download</a></li>
+<li><a href="/development">Development</a></li>
+<li><a href="/about">About</a></li>
+</ul>
+</div>
+</div>
+<!--TOC-->
+<div id="container">
+<div id="content">
+<h1>{doctitle}</h1>
+
+[footer]
+</div>
+</div>
+<div id="footer">
+<div id="footer-text">
+</div>
+</div>
+
+<a href="https://github.com/ClusterLabs/crmsh"><img style="position: absolute; top: 0; right: 0; border: 0;" src="https://camo.githubusercontent.com/a6677b08c955af8400f44c6298f40e7d19cc5b2d/68747470733a2f2f73332e616d617a6f6e6177732e636f6d2f6769746875622f726962626f6e732f666f726b6d655f72696768745f677261795f3664366436642e706e67" alt="Fork me on GitHub" data-canonical-src="https://s3.amazonaws.com/github/ribbons/forkme_right_gray_6d6d6d.png"></a>
+
+</body>
+</html>
+
+ifdef::doctype-manpage[]
+[synopsis]
+template::[sect1]
+endif::doctype-manpage[]
+
diff --git a/doc/website-v1/crmold.conf b/doc/website-v1/crmold.conf
new file mode 100644
index 0000000..271d88d
--- /dev/null
+++ b/doc/website-v1/crmold.conf
@@ -0,0 +1,602 @@
+#
+# html5.conf
+#
+# Asciidoc configuration file.
+# html5 backend.
+#
+
+[miscellaneous]
+outfilesuffix=.html
+
+[attributes]
+basebackend=html
+basebackend-html=
+basebackend-html5=
+b
+[replacements2]
+# Line break.
+(?m)^(.*)\s\+$=\1<br>
+
+[replacements]
+ifdef::asciidoc7compatible[]
+# Superscripts.
+\^(.+?)\^=<sup>\1</sup>
+# Subscripts.
+~(.+?)~=<sub>\1</sub>
+endif::asciidoc7compatible[]
+
+[ruler-blockmacro]
+<hr>
+
+[pagebreak-blockmacro]
+<div style="page-break-after:always"></div>
+
+[blockdef-pass]
+asciimath-style=template="asciimathblock",subs=()
+latexmath-style=template="latexmathblock",subs=()
+
+[macros]
+(?u)^(?P<name>audio|video)::(?P<target>\S*?)(\[(?P<attrlist>.*?)\])$=#
+# math macros.
+# Special characters are escaped in HTML math markup.
+(?su)[\\]?(?P<name>asciimath|latexmath):(?P<subslist>\S*?)\[(?P<passtext>.*?)(?<!\\)\]=[specialcharacters]
+(?u)^(?P<name>asciimath|latexmath)::(?P<subslist>\S*?)(\[(?P<passtext>.*?)\])$=#[specialcharacters]
+
+[asciimath-inlinemacro]
+`{passtext}`
+
+[asciimath-blockmacro]
+<div class="mathblock{role? {role}}{unbreakable-option? unbreakable}"{id? id="{id}"}>
+<div class="content">
+<div class="title">{title}</div>
+`{passtext}`
+</div></div>
+
+[asciimathblock]
+<div class="mathblock{role? {role}}{unbreakable-option? unbreakable}"{id? id="{id}"}>
+<div class="content">
+<div class="title">{title}</div>
+`|`
+</div></div>
+
+[latexmath-inlinemacro]
+{passtext}
+
+[latexmath-blockmacro]
+<div class="mathblock{role? {role}}{unbreakable-option? unbreakable}"{id? id="{id}"}>
+<div class="content">
+<div class="title">{title}</div>
+{passtext}
+</div></div>
+
+[latexmathblock]
+<div class="mathblock{role? {role}}{unbreakable-option? unbreakable}"{id? id="{id}"}>
+<div class="content">
+<div class="title">{title}</div>
+|
+</div></div>
+
+[image-inlinemacro]
+<span class="image{role? {role}}">
+<a class="image" href="{link}">
+{data-uri%}<img src="{imagesdir=}{imagesdir?/}{target}" alt="{alt={target}}"{width? width="{width}"}{height? height="{height}"}{title? title="{title}"}>
+{data-uri#}<img alt="{alt={target}}"{width? width="{width}"}{height? height="{height}"}{title? title="{title}"}
+{data-uri#}{sys:"{python}" -u -c "import mimetypes,base64,sys; print 'src=\"data:'+mimetypes.guess_type(r'{target}')[0]+';base64,'; base64.encode(sys.stdin,sys.stdout)" < "{eval:os.path.join(r"{indir={outdir}}",r"{imagesdir=}",r"{target}")}"}">
+{link#}</a>
+</span>
+
+[image-blockmacro]
+<div class="imageblock{style? {style}}{role? {role}}{unbreakable-option? unbreakable}"{id? id="{id}"}{align? style="text-align:{align};"}{float? style="float:{float};"}>
+<div class="content">
+<a class="image" href="{link}">
+{data-uri%}<img src="{imagesdir=}{imagesdir?/}{target}" alt="{alt={target}}"{width? width="{width}"}{height? height="{height}"}>
+{data-uri#}<img alt="{alt={target}}"{width? width="{width}"}{height? height="{height}"}
+{data-uri#}{sys:"{python}" -u -c "import mimetypes,base64,sys; print 'src=\"data:'+mimetypes.guess_type(r'{target}')[0]+';base64,'; base64.encode(sys.stdin,sys.stdout)" < "{eval:os.path.join(r"{indir={outdir}}",r"{imagesdir=}",r"{target}")}"}">
+{link#}</a>
+</div>
+<div class="title">{caption={figure-caption} {counter:figure-number}. }{title}</div>
+</div>
+
+[audio-blockmacro]
+<div class="audioblock{role? {role}}{unbreakable-option? unbreakable}"{id? id="{id}"}>
+<div class="title">{caption=}{title}</div>
+<div class="content">
+<audio src="{imagesdir=}{imagesdir?/}{target}"{autoplay-option? autoplay}{nocontrols-option! controls}{loop-option? loop}>
+Your browser does not support the audio tag.
+</audio>
+</div></div>
+
+[video-blockmacro]
+<div class="videoblock{role? {role}}{unbreakable-option? unbreakable}"{id? id="{id}"}>
+<div class="title">{caption=}{title}</div>
+<div class="content">
+<video src="{imagesdir=}{imagesdir?/}{target}"{width? width="{width}"}{height? height="{height}"}{poster? poster="{poster}"}{autoplay-option? autoplay}{nocontrols-option! controls}{loop-option? loop}>
+Your browser does not support the video tag.
+</video>
+</div></div>
+
+[unfloat-blockmacro]
+<div style="clear:both;"></div>
+
+[toc-blockmacro]
+template::[toc]
+
+[indexterm-inlinemacro]
+# Index term.
+{empty}
+
+[indexterm2-inlinemacro]
+# Index term.
+# Single entry index term that is visible in the primary text flow.
+{1}
+
+[footnote-inlinemacro]
+# footnote:[<text>].
+<span class="footnote"><br>[{0}]<br></span>
+
+[footnoteref-inlinemacro]
+# footnoteref:[<id>], create reference to footnote.
+{2%}<span class="footnoteref"><br><a href="#_footnote_{1}">[{1}]</a><br></span>
+# footnoteref:[<id>,<text>], create footnote with ID.
+{2#}<span class="footnote" id="_footnote_{1}"><br>[{2}]<br></span>
+
+[callout-inlinemacro]
+ifndef::icons[]
+<b>&lt;{index}&gt;</b>
+endif::icons[]
+ifdef::icons[]
+ifndef::data-uri[]
+<img src="{icon={iconsdir}/callouts/{index}.png}" alt="{index}">
+endif::data-uri[]
+ifdef::data-uri[]
+<img alt="{index}" src="data:image/png;base64,
+{sys:"{python}" -u -c "import base64,sys; base64.encode(sys.stdin,sys.stdout)" < "{eval:os.path.join(r"{indir={outdir}}",r"{icon={iconsdir}/callouts/{index}.png}")}"}">
+endif::data-uri[]
+endif::icons[]
+
+# Comment line macros.
+[comment-inlinemacro]
+{showcomments#}<br><span class="comment">{passtext}</span><br>
+
+[comment-blockmacro]
+{showcomments#}<p><span class="comment">{passtext}</span></p>
+
+[literal-inlinemacro]
+# Inline literal.
+<span class="monospaced">{passtext}</span>
+
+# List tags.
+[listtags-bulleted]
+list=<div class="ulist{style? {style}}{compact-option? compact}{role? {role}}"{id? id="{id}"}>{title?<div class="title">{title}</div>}<ul>|</ul></div>
+item=<li>|</li>
+text=<p>|</p>
+
+[listtags-numbered]
+# The start attribute is not valid XHTML 1.1 but all browsers support it.
+list=<div class="olist{style? {style}}{compact-option? compact}{role? {role}}"{id? id="{id}"}>{title?<div class="title">{title}</div>}<ol class="{style}"{start? start="{start}"}>|</ol></div>
+item=<li>|</li>
+text=<p>|</p>
+
+[listtags-labeled]
+list=<div class="dlist{compact-option? compact}{role? {role}}"{id? id="{id}"}>{title?<div class="title">{title}</div>}<dl>|</dl></div>
+entry=
+label=
+term=<dt class="hdlist1{strong-option? strong}">|</dt>
+item=<dd>|</dd>
+text=<p>|</p>
+
+[listtags-horizontal]
+list=<div class="hdlist{compact-option? compact}{role? {role}}"{id? id="{id}"}>{title?<div class="title">{title}</div>}<table>{labelwidth?<col width="{labelwidth}%">}{itemwidth?<col width="{itemwidth}%">}|</table></div>
+label=<td class="hdlist1{strong-option? strong}">|</td>
+term=|<br>
+entry=<tr>|</tr>
+item=<td class="hdlist2">|</td>
+text=<p style="margin-top: 0;">|</p>
+
+[listtags-qanda]
+list=<div class="qlist{style? {style}}{role? {role}}"{id? id="{id}"}>{title?<div class="title">{title}</div>}<ol>|</ol></div>
+entry=<li>|</li>
+label=
+term=<p><em>|</em></p>
+item=
+text=<p>|</p>
+
+[listtags-callout]
+ifndef::icons[]
+list=<div class="colist{style? {style}}{role? {role}}"{id? id="{id}"}>{title?<div class="title">{title}</div>}<ol>|</ol></div>
+item=<li>|</li>
+text=<p>|</p>
+endif::icons[]
+ifdef::icons[]
+list=<div class="colist{style? {style}}{role? {role}}"{id? id="{id}"}>{title?<div class="title">{title}</div>}<table>|</table></div>
+ifndef::data-uri[]
+item=<tr><td><img src="{iconsdir}/callouts/{listindex}.png" alt="{listindex}"></td><td>|</td></tr>
+endif::data-uri[]
+ifdef::data-uri[]
+item=<tr><td><img alt="{listindex}" src="data:image/png;base64, {sys:"{python}" -u -c "import base64,sys; base64.encode(sys.stdin,sys.stdout)" < "{eval:os.path.join(r"{indir={outdir}}",r"{icon={iconsdir}/callouts/{listindex}.png}")}"}"></td><td>|</td></tr>
+endif::data-uri[]
+text=|
+endif::icons[]
+
+[listtags-glossary]
+list=<div class="dlist{style? {style}}{role? {role}}"{id? id="{id}"}>{title?<div class="title">{title}</div>}<dl>|</dl></div>
+label=
+entry=
+term=<dt>|</dt>
+item=<dd>|</dd>
+text=<p>|</p>
+
+[listtags-bibliography]
+list=<div class="ulist{style? {style}}{role? {role}}"{id? id="{id}"}>{title?<div class="title">{title}</div>}<ul>|</ul></div>
+item=<li>|</li>
+text=<p>|</p>
+
+[tags]
+# Quoted text.
+emphasis=<em>{1?<span class="{1}">}|{1?</span>}</em>
+strong=<strong>{1?<span class="{1}">}|{1?</span>}</strong>
+monospaced=<span class="monospaced{1? {1}}">|</span>
+singlequoted={lsquo}{1?<span class="{1}">}|{1?</span>}{rsquo}
+doublequoted={ldquo}{1?<span class="{1}">}|{1?</span>}{rdquo}
+unquoted={1?<span class="{1}">}|{1?</span>}
+superscript=<sup>{1?<span class="{1}">}|{1?</span>}</sup>
+subscript=<sub>{1?<span class="{1}">}|{1?</span>}</sub>
+
+ifdef::deprecated-quotes[]
+# Override with deprecated quote attributes.
+emphasis={role?<span class="{role}">}<em{1,2,3? style="}{1?color:{1};}{2?background-color:{2};}{3?font-size:{3}em;}{1,2,3?"}>|</em>{role?</span>}
+strong={role?<span class="{role}">}<strong{1,2,3? style="}{1?color:{1};}{2?background-color:{2};}{3?font-size:{3}em;}{1,2,3?"}>|</strong>{role?</span>}
+monospaced=<span class="monospaced{role? {role}}"{1,2,3? style="}{1?color:{1};}{2?background-color:{2};}{3?font-size:{3}em;}{1,2,3?"}>|</span>
+singlequoted={role?<span class="{role}">}{1,2,3?<span style="}{1?color:{1};}{2?background-color:{2};}{3?font-size:{3}em;}{1,2,3?">}{amp}#8216;|{amp}#8217;{1,2,3?</span>}{role?</span>}
+doublequoted={role?<span class="{role}">}{1,2,3?<span style="}{1?color:{1};}{2?background-color:{2};}{3?font-size:{3}em;}{1,2,3?">}{amp}#8220;|{amp}#8221;{1,2,3?</span>}{role?</span>}
+unquoted={role?<span class="{role}">}{1,2,3?<span style="{1?color:{1};}{2?background-color:{2};}{3?font-size:{3}em;}">}|{1,2,3?</span>}{role?</span>}
+superscript={role?<span class="{role}">}<sup{1,2,3? style="}{1?color:{1};}{2?background-color:{2};}{3?font-size:{3}em;}{1,2,3?"}>|</sup>{role?</span>}
+subscript={role?<span class="{role}">}<sub{1,2,3? style="}{1?color:{1};}{2?background-color:{2};}{3?font-size:{3}em;}{1,2,3?"}>|</sub>{role?</span>}
+endif::deprecated-quotes[]
+
+# Inline macros
+[http-inlinemacro]
+<a href="{name}:{target}">{0={name}:{target}}</a>
+[https-inlinemacro]
+<a href="{name}:{target}">{0={name}:{target}}</a>
+[ftp-inlinemacro]
+<a href="{name}:{target}">{0={name}:{target}}</a>
+[file-inlinemacro]
+<a href="{name}:{target}">{0={name}:{target}}</a>
+[irc-inlinemacro]
+<a href="{name}:{target}">{0={name}:{target}}</a>
+[mailto-inlinemacro]
+<a href="mailto:{target}">{0={target}}</a>
+[link-inlinemacro]
+<a href="{target}">{0={target}}</a>
+[callto-inlinemacro]
+<a href="{name}:{target}">{0={target}}</a>
+# anchor:id[text]
+[anchor-inlinemacro]
+<a id="{target}"></a>
+# [[id,text]]
+[anchor2-inlinemacro]
+<a id="{1}"></a>
+# [[[id]]]
+[anchor3-inlinemacro]
+<a id="{1}"></a>[{1}]
+# xref:id[text]
+[xref-inlinemacro]
+<a href="#{target}">{0=[{target}]}</a>
+# <<id,text>>
+[xref2-inlinemacro]
+<a href="#{1}">{2=[{1}]}</a>
+
+# Special word substitution.
+[emphasizedwords]
+<em>{words}</em>
+[monospacedwords]
+<span class="monospaced">{words}</span>
+[strongwords]
+<strong>{words}</strong>
+
+# Paragraph substitution.
+[paragraph]
+<div class="paragraph{role? {role}}{unbreakable-option? unbreakable}"{id? id="{id}"}>{title?<div class="title">{title}</div>}<p>
+|
+</p></div>
+
+[admonitionparagraph]
+template::[admonitionblock]
+
+# Delimited blocks.
+[listingblock]
+<div class="listingblock{role? {role}}{unbreakable-option? unbreakable}"{id? id="{id}"}>
+<div class="title">{caption=}{title}</div>
+<div class="content monospaced">
+<pre>
+|
+</pre>
+</div></div>
+
+[literalblock]
+<div class="literalblock{role? {role}}{unbreakable-option? unbreakable}"{id? id="{id}"}>
+<div class="title">{title}</div>
+<div class="content monospaced">
+<pre>
+|
+</pre>
+</div></div>
+
+[sidebarblock]
+<div class="sidebarblock{role? {role}}{unbreakable-option? unbreakable}"{id? id="{id}"}>
+<div class="content">
+<div class="title">{title}</div>
+|
+</div></div>
+
+[openblock]
+<div class="openblock{role? {role}}{unbreakable-option? unbreakable}"{id? id="{id}"}>
+<div class="title">{title}</div>
+<div class="content">
+|
+</div></div>
+
+[partintroblock]
+template::[openblock]
+
+[abstractblock]
+template::[quoteblock]
+
+[quoteblock]
+<div class="quoteblock{role? {role}}{unbreakable-option? unbreakable}"{id? id="{id}"}>
+<div class="title">{title}</div>
+<div class="content">
+|
+</div>
+<div class="attribution">
+<em>{citetitle}</em>{attribution?<br>}
+&#8212; {attribution}
+</div></div>
+
+[verseblock]
+<div class="verseblock{role? {role}}{unbreakable-option? unbreakable}"{id? id="{id}"}>
+<div class="title">{title}</div>
+<pre class="content">
+|
+</pre>
+<div class="attribution">
+<em>{citetitle}</em>{attribution?<br>}
+&#8212; {attribution}
+</div></div>
+
+[exampleblock]
+<div class="exampleblock{role? {role}}{unbreakable-option? unbreakable}"{id? id="{id}"}>
+<div class="title">{caption={example-caption} {counter:example-number}. }{title}</div>
+<div class="content">
+|
+</div></div>
+
+[admonitionblock]
+<div class="admonitionblock{role? {role}}{unbreakable-option? unbreakable}"{id? id="{id}"}>
+<table><tr>
+<td class="icon">
+{data-uri%}{icons#}<img src="{icon={iconsdir}/{name}.png}" alt="{caption}">
+{data-uri#}{icons#}<img alt="{caption}" src="data:image/png;base64,
+{data-uri#}{icons#}{sys:"{python}" -u -c "import base64,sys; base64.encode(sys.stdin,sys.stdout)" < "{eval:os.path.join(r"{indir={outdir}}",r"{icon={iconsdir}/{name}.png}")}"}">
+{icons%}<div class="title">{caption}</div>
+</td>
+<td class="content">
+<div class="title">{title}</div>
+|
+</td>
+</tr></table>
+</div>
+
+# Tables.
+[tabletags-default]
+colspec=<col{autowidth-option! style="width:{colpcwidth}%;"}>
+bodyrow=<tr>|</tr>
+headdata=<th class="tableblock halign-{halign=left} valign-{valign=top}" {colspan@1::colspan="{colspan}" }{rowspan@1::rowspan="{rowspan}" }>|</th>
+bodydata=<td class="tableblock halign-{halign=left} valign-{valign=top}" {colspan@1::colspan="{colspan}" }{rowspan@1::rowspan="{rowspan}" }>|</td>
+paragraph=<p class="tableblock">|</p>
+
+[tabletags-header]
+paragraph=<p class="tableblock header">|</p>
+
+[tabletags-emphasis]
+paragraph=<p class="tableblock"><em>|</em></p>
+
+[tabletags-strong]
+paragraph=<p class="tableblock"><strong>|</strong></p>
+
+[tabletags-monospaced]
+paragraph=<p class="tableblock monospaced">|</p>
+
+[tabletags-verse]
+bodydata=<td class="tableblock halign-{halign=left} valign-{valign=top}" {colspan@1::colspan="{colspan}" }{rowspan@1::rowspan="{rowspan}" }><div class="verse">|</div></td>
+paragraph=
+
+[tabletags-literal]
+bodydata=<td class="tableblock halign-{halign=left} valign-{valign=top}" {colspan@1::colspan="{colspan}" }{rowspan@1::rowspan="{rowspan}" }><div class="literal monospaced"><pre>|</pre></div></td>
+paragraph=
+
+[tabletags-asciidoc]
+bodydata=<td class="tableblock halign-{halign=left} valign-{valign=top}" {colspan@1::colspan="{colspan}" }{rowspan@1::rowspan="{rowspan}" }><div>|</div></td>
+paragraph=
+
+[table]
+<table class="tableblock frame-{frame=all} grid-{grid=all}{role? {role}}{unbreakable-option? unbreakable}"{id? id="{id}"}
+style="
+margin-left:{align@left:0}{align@center|right:auto}; margin-right:{align@left|center:auto}{align@right:0};
+float:{float};
+{autowidth-option%}width:{tablepcwidth}%;
+{autowidth-option#}{width#style=width:{tablepcwidth}%;}
+">
+<caption class="title">{caption={table-caption} {counter:table-number}. }{title}</caption>
+{colspecs}
+{headrows#}<thead>
+{headrows}
+{headrows#}</thead>
+{footrows#}<tfoot>
+{footrows}
+{footrows#}</tfoot>
+<tbody>
+{bodyrows}
+</tbody>
+</table>
+
+#--------------------------------------------------------------------
+# Deprecated old table definitions.
+#
+
+[miscellaneous]
+# Screen width in pixels.
+pagewidth=800
+pageunits=px
+
+[old_tabledef-default]
+template=old_table
+colspec=<col style="width:{colwidth}{pageunits};" />
+bodyrow=<tr>|</tr>
+headdata=<th class="tableblock halign-{colalign=left}">|</th>
+footdata=<td class="tableblock halign-{colalign=left}">|</td>
+bodydata=<td class="tableblock halign-{colalign=left}">|</td>
+
+[old_table]
+<table class="tableblock frame-{frame=all} grid-{grid=all}"{id? id="{id}"}>
+<caption class="title">{caption={table-caption}}{title}</caption>
+{colspecs}
+{headrows#}<thead>
+{headrows}
+{headrows#}</thead>
+{footrows#}<tfoot>
+{footrows}
+{footrows#}</tfoot>
+<tbody style="vertical-align:top;">
+{bodyrows}
+</tbody>
+</table>
+
+# End of deprecated old table definitions.
+#--------------------------------------------------------------------
+
+[floatingtitle]
+<h{level@0:1}{level@1:2}{level@2:3}{level@3:4}{level@4:5}{id? id="{id}"} class="float">{title}</h{level@0:1}{level@1:2}{level@2:3}{level@3:4}{level@4:5}>
+
+[preamble]
+# Untitled elements between header and first section title.
+<div id="preamble">
+<div class="sectionbody">
+|
+</div>
+</div>
+
+# Document sections.
+[sect0]
+<h1{id? id="{id}"}>{title}</h1>
+|
+
+[sect1]
+<div class="sect1{style? {style}}{role? {role}}">
+<h2{id? id="{id}"}>{numbered?{sectnum} }{title}</h2>
+<div class="sectionbody">
+|
+</div>
+</div>
+
+[sect2]
+<div class="sect2{style? {style}}{role? {role}}">
+<h3{id? id="{id}"}>{numbered?{sectnum} }{title}</h3>
+|
+</div>
+
+[sect3]
+<div class="sect3{style? {style}}{role? {role}}">
+<h4{id? id="{id}"}>{numbered?{sectnum} }{title}</h4>
+|
+</div>
+
+[sect4]
+<div class="sect4{style? {style}}{role? {role}}">
+<h5{id? id="{id}"}>{title}</h5>
+|
+</div>
+
+[appendix]
+<div class="sect1{style? {style}}{role? {role}}">
+<h2{id? id="{id}"}>{numbered?{sectnum} }{appendix-caption} {counter:appendix-number:A}: {title}</h2>
+<div class="sectionbody">
+|
+</div>
+</div>
+
+[toc]
+<div id="toc">
+ <div id="toctitle">{toc-title}</div>
+ <noscript><p><b>JavaScript must be enabled in your browser to display the table of contents.</b></p></noscript>
+</div>
+
+[header]
+<!DOCTYPE html>
+<html lang="{lang=en}">
+<head>
+<meta http-equiv="Content-Type" content="text/html; charset={encoding}">
+<meta name="generator" content="AsciiDoc {asciidoc-version}">
+<meta name="description" content="{description}">
+<meta name="keywords" content="{keywords}">
+<title>crmsh - {title}</title>
+{title%}<title>crmsh - {doctitle=}</title>
+<link rel="stylesheet" href="http://crmsh.nongnu.org/css/font-awesome.min.css">
+<link rel="stylesheet" href="http://crmsh.nongnu.org/css/crm.css" type="text/css">
+<link href='http://fonts.googleapis.com/css?family=Open+Sans:400,700|Ubuntu+Mono' rel='stylesheet' type='text/css'>
+<link href="http://crmsh.github.io/atom.xml" type="application/atom+xml" rel="alternate" title="crmsh atom feed">
+<style>
+\#movenotice {
+ width: 600px;
+ margin-top: 1em;
+ margin-bottom: 1em;
+ margin-left: auto;
+ margin-right: auto;
+ font-size: 100%;
+ padding: 4px;
+ border: 2px dashed red;
+}
+</style>
+</head>
+<body>
+<div id="header">
+<h1><a href="http://crmsh.github.io/index.html"><span class="fa-stack">
+ <i class="fa fa-square fa-stack-2x"></i>
+ <i class="fa fa-terminal fa-stack-1x fa-inverse"></i>
+</span>crmsh</a></h1>
+<div id="topbar">
+<ul>
+<li><a href="http://crmsh.github.io/news">News</a></li>
+<li><a href="http://crmsh.github.io/documentation">Documentation</a></li>
+<li><a href="http://crmsh.github.io/development">Development</a></li>
+<li><a href="http://crmsh.github.io/about">About</a></li>
+</ul>
+</div>
+</div>
+<!--TOC-->
+<div id="container">
+<div id="content">
+
+<div id="movenotice">We have moved! The website for crmsh is now <a href="http://crmsh.github.io">http://crmsh.github.io</a>.</div>
+
+<h1>{doctitle}</h1>
+
+[footer]
+</div>
+</div>
+<div id="footer">
+<div id="footer-text">
+</div>
+</div>
+
+<a href="https://github.com/crmsh/crmsh"><img style="position: absolute; top: 0; right: 0; border: 0;" src="https://camo.githubusercontent.com/652c5b9acfaddf3a9c326fa6bde407b87f7be0f4/68747470733a2f2f73332e616d617a6f6e6177732e636f6d2f6769746875622f726962626f6e732f666f726b6d655f72696768745f6f72616e67655f6666373630302e706e67" alt="Fork me on GitHub" data-canonical-src="https://s3.amazonaws.com/github/ribbons/forkme_right_orange_ff7600.png"></a>
+
+</body>
+</html>
+
+ifdef::doctype-manpage[]
+[synopsis]
+template::[sect1]
+endif::doctype-manpage[]
+
diff --git a/doc/website-v1/css/crm.css b/doc/website-v1/css/crm.css
new file mode 100644
index 0000000..9fdfba2
--- /dev/null
+++ b/doc/website-v1/css/crm.css
@@ -0,0 +1,570 @@
+/* ---------------------------------------------------------------------------
+ Based on
+ Bare AsciiDoc styles
+ Ryan Tomayko <r@tomayko.com>
+
+ Heavily modified by
+ Kristoffer Gronlund <kgronlund@suse.com>
+ --------------------------------------------------------------------------- */
+
+/* Fonts */
+
+body {
+ font-family:'Open Sans', 'lucida grande',verdana,helvetica,arial,sans-serif;
+ font-size: 16px;
+ line-height: 21px;
+}
+
+.small {
+ font-size: 12px; /* 75% of the baseline */
+}
+
+.large {
+ font-size: 20px; /* 125% of the baseline */
+}
+
+em {
+ font-style:italic;
+}
+
+strong {
+ font-weight:bold;
+}
+
+.monospaced {
+ font-family: 'Menlo', 'Monaco', 'Consolas', 'lucida console', 'bitstream vera sans mono', 'courier new', monospace;
+}
+
+dt {
+ font-weight:normal;
+}
+
+h1, h2, h3, h4, h5 {
+ font-family:'Open Sans', 'lucida grande',verdana,helvetica,arial,sans-serif;
+ font-weight:normal;
+}
+
+h1 {
+ font-size:2.6rem;
+ line-height:1.428;
+}
+
+h2 {
+ font-size:2rem;
+ line-height:1.36363636; /* repeating, of course */
+}
+
+h3 {
+ font-size:1.6rem;
+ line-height:1.1;
+}
+
+h4 {
+ font-weight: bold;
+ font-size:1.3rem;
+ line-height:1.538;
+}
+
+h5 {
+ font-size:1.2rem;
+ font-style:italic;
+ line-height:1.538;
+}
+
+pre {
+ font-family:'Menlo', 'Monaco', consolas, 'lucida console', 'bitstream vera sans mono', 'courier new', monospace;
+ font-size: 14px;
+}
+
+#header h1 {
+ font-size: 42px;
+ margin: 0px;
+ display: inline;
+}
+
+#topbar {
+ display: inline;
+ font-size: 18px;
+}
+
+#topbar-small {
+ display: none;
+ font-size: 14px;
+}
+
+
+
+/* Style */
+
+body {
+ margin: 0px 0px;
+ padding: 0px;
+ width: 100%;
+ color:#333;
+ background: #fff;
+}
+
+.monospaced {
+ color: #211;
+ background-color: #fafaf8;
+}
+
+p {
+ margin-bottom: 1.3636rem;
+}
+
+ul, ol, dl {
+ margin-top: 1rem;
+ margin-bottom: 2rem;
+}
+
+ul p {
+ margin: 10px 0;
+}
+
+dl {
+ margin-left:40px
+}
+
+dt {
+ color:#000;
+}
+
+h1, h2, h3, h4, h5 {
+ color:#000;
+}
+
+h2, h3, h4, h5 {
+ padding-bottom: 0.333rem;
+ border-bottom: 1px solid #eee;
+}
+
+h1 {
+ margin:0px;
+ margin-top: 48px;
+}
+
+h2 {
+ margin-top: 36px;
+ margin-bottom: 1.5rem;
+}
+
+h3 {
+ margin: 0px;
+ margin-top: 30px;
+}
+
+pre {
+ color: #211;
+ overflow-x: auto;
+}
+
+#header {
+ background: #ecf0f1;
+ padding-left: 24px;
+ padding-top: 4px;
+ padding-bottom: 0px;
+ border-bottom: 2px solid #efefea;
+ width: 100%;
+ height: 86px;
+ overflow: hidden;
+}
+
+#header a {
+ text-decoration: none;
+ color: #34495e;
+}
+
+#header a:hover {
+ color:#ee3300;
+}
+
+#topbar ul {
+ list-style: none;
+ display: inline;
+}
+
+#topbar li {
+ list-style: none;
+ display: inline;
+ padding-right: 1rem;
+}
+
+#topbar-small ul {
+ list-style: none;
+ display: inline;
+ padding-left: 0.5rem;
+}
+
+#topbar-small li {
+ list-style: none;
+ display: inline;
+ padding-right: 0.5rem;
+}
+
+
+#container {
+ max-width: 720px;
+ margin-left: 240px;
+ padding-left: 8px;
+ text-align:left;
+}
+
+#author {
+ color:#999;
+}
+
+a {
+ text-decoration: none;
+ color:#419eda;
+}
+
+a:active {
+ color:#6ec654;
+}
+
+a:hover {
+ color:#ee3300;
+ text-decoration: underline;
+}
+
+
+#content {
+}
+
+h1 {
+ margin-left: auto;
+ margin-right: auto;
+ width: 551px;
+ text-align: center;
+ margin-bottom: 1.5rem;
+}
+
+.frontpage-image {
+ margin-left: auto;
+ margin-right: auto;
+ width: 551px;
+}
+
+.title, .sidebar-title {
+ font-weight:normal;
+ color:#000;
+ margin-bottom:0;
+}
+
+div.content {
+ margin: 8px;
+ padding: 0;
+}
+
+div.admonitionblock .title {
+ font-weight:bold;
+}
+
+div.admonitionblock {
+ margin:30px 0px;
+ color:#555;
+}
+
+div.admonitionblock td.icon {
+ width:30px;
+ padding-right:20px;
+ padding-left:20px;
+ text-transform:uppercase;
+ font-weight:bold;
+ color:#888;
+}
+
+div.listingblock .content {
+ border-left:4px solid #419eda;
+ padding:8px;
+
+ background: #faf7f8;
+
+ background-image: -moz-linear-gradient(left, right,
+ from(#faf7f8),
+ to(#ffffff));
+
+ background-image: -webkit-gradient(linear, left top, right bottom,
+ color-stop(0.00, #faf7f8),
+ color-stop(1.00, #ffffff));
+
+}
+
+div.listingblock .content pre {
+ margin:0;
+}
+
+div.literalblock .content {
+ margin-left: 20px;
+}
+
+div.verseblock .content {
+ white-space:pre
+}
+
+div.sidebarblock {
+ margin-top: 1.5rem;
+ margin-bottom: 2rem;
+}
+
+div.sidebarblock > div.content {
+ border-left:4px solid #ee3300;
+ background: #faf7f8;
+ padding:0 10px;
+ color:#222;
+ font-size: 14px;
+ line-height:18px;
+ max-width: 720px;
+
+ background-image: -moz-linear-gradient(left, right,
+ from(#faf7f8),
+ to(#ffffff));
+
+ background-image: -webkit-gradient(linear, left top, right bottom,
+ color-stop(0.00, #faf7f8),
+ color-stop(1.00, #ffffff));
+
+}
+
+div.sidebarblock .title {
+ margin:10px 0;
+ font-weight:bold;
+ font-size: 14px;
+ color:#442;
+}
+
+.quoteblock-content {
+ font-style:italic;
+ color:#444;
+ margin-left:40px;
+}
+
+.quoteblock-content .attribution {
+ font-style:normal;
+ text-align:right;
+ color:#000;
+}
+
+.exampleblock-content *:first-child { margin-top:0 }
+.exampleblock-content {
+ border-left:2px solid silver;
+ padding-left:8px;
+}
+
+#footnotes {
+ text-align:left;
+}
+
+#footnotes hr {
+ height: 1px;
+ color: #ccc;
+ width: 80%;
+}
+
+#footer {
+ font-size: 12px;
+ color:#888;
+ margin-top:40px;
+ text-align: right;
+}
+
+.nav {
+ margin-bottom: 0;
+ padding-left: 0;
+ list-style: none;
+}
+
+.nav li {
+ line-height: 4rem;
+}
+
+.nav a {
+ font-size: 20px;
+ text-decoration: none;
+}
+
+.feedEkList .newsItem {
+ list-style-type: none;
+}
+
+.feedEkList .itemTitle {
+ font-size: large;
+}
+
+.feedEkList .itemDate {
+ font-size: smaller;
+}
+
+.feedEkList .itemContent {
+}
+
+@media screen {
+ #toc {
+ position: fixed;
+ top: 120px;
+ left: 4px;
+ margin: 0px;
+ font-size: 12px;
+ line-height: 1.2em;
+ }
+
+ #toc a .monospaced {
+ color:#419eda;
+ }
+
+ #toc a {
+ text-decoration: none;
+ }
+
+ #toc .toclevel1 {
+ padding: 1px;
+ margin-top: 8px;
+ font-size: 14px;
+ line-height: 16px;
+ }
+
+ #toc .toclevel2 {
+ margin-left: 8px;
+ padding-left: 4px;
+ font-size: 12px;
+ line-height: 16px;
+ }
+
+ #toc .toclevel3 {
+ margin-left: 24px;
+ padding-left: 4px;
+ font-size: 11px;
+ line-height: 15px;
+ font-weight: bold;
+ }
+
+ #toctitle {
+ margin:20px 0;
+
+ }
+}
+
+@media screen and (max-width: 900px) {
+ #toc {
+ display: none;
+ }
+
+ #container {
+ max-width: 720px;
+ margin: 0px auto;
+ text-align:left;
+ padding-right: 8px;
+ }
+
+ #topbar {
+ display: none;
+ }
+
+ #topbar-small {
+ display: inline;
+ }
+}
+
+@media screen and (min-width: 900px) and (max-width: 1280px) {
+ #toc {
+ position: absolute;
+ overflow: hidden;
+ top: 120px;
+ left: 4px;
+ max-width: 240px;
+ }
+}
+
+@media screen and (min-width: 1280px) {
+ #toc {
+ position: absolute;
+ overflow: hidden;
+ top: 120px;
+ left: 4px;
+ max-width: 240px;
+ }
+
+ #container {
+ max-width: 960px;
+ }
+}
+
+@media screen and (min-width: 1500px) {
+ #container {
+ margin-left: auto;
+ margin-right: auto;
+ }
+}
+
+/* pygments highlighting */
+
+.hll { background-color: #ffffcc }
+.c { color: #999988; font-style: italic } /* Comment */
+.err { color: #a61717; background-color: #e3d2d2 } /* Error */
+.k { color: #000000; font-weight: bold } /* Keyword */
+.o { color: #000000; font-weight: bold } /* Operator */
+.cm { color: #999988; font-style: italic } /* Comment.Multiline */
+.cp { color: #999999; font-weight: bold; font-style: italic } /* Comment.Preproc */
+.c1 { color: #999988; font-style: italic } /* Comment.Single */
+.cs { color: #999999; font-weight: bold; font-style: italic } /* Comment.Special */
+.gd { color: #000000; background-color: #ffdddd } /* Generic.Deleted */
+.ge { color: #000000; font-style: italic } /* Generic.Emph */
+.gr { color: #aa0000 } /* Generic.Error */
+.gh { color: #999999 } /* Generic.Heading */
+.gi { color: #000000; background-color: #ddffdd } /* Generic.Inserted */
+.go { color: #888888 } /* Generic.Output */
+.gp { color: #555555 } /* Generic.Prompt */
+.gs { font-weight: bold } /* Generic.Strong */
+.gu { color: #aaaaaa } /* Generic.Subheading */
+.gt { color: #aa0000 } /* Generic.Traceback */
+.kc { color: #000000; font-weight: bold } /* Keyword.Constant */
+.kd { color: #000000; font-weight: bold } /* Keyword.Declaration */
+.kn { color: #000000; font-weight: bold } /* Keyword.Namespace */
+.kp { color: #000000; font-weight: bold } /* Keyword.Pseudo */
+.kr { color: #000000; font-weight: bold } /* Keyword.Reserved */
+.kt { color: #445588; font-weight: bold } /* Keyword.Type */
+.m { color: #009999 } /* Literal.Number */
+.s { color: #d01040 } /* Literal.String */
+.na { color: #008080 } /* Name.Attribute */
+.nb { color: #0086B3 } /* Name.Builtin */
+.nc { color: #445588; font-weight: bold } /* Name.Class */
+.no { color: #008080 } /* Name.Constant */
+.nd { color: #3c5d5d; font-weight: bold } /* Name.Decorator */
+.ni { color: #800080 } /* Name.Entity */
+.ne { color: #990000; font-weight: bold } /* Name.Exception */
+.nf { color: #990000; font-weight: bold } /* Name.Function */
+.nl { color: #990000; font-weight: bold } /* Name.Label */
+.nn { color: #555555 } /* Name.Namespace */
+.nt { color: #000080 } /* Name.Tag */
+.nv { color: #008080 } /* Name.Variable */
+.ow { color: #000000; font-weight: bold } /* Operator.Word */
+.w { color: #bbbbbb } /* Text.Whitespace */
+.mf { color: #009999 } /* Literal.Number.Float */
+.mh { color: #009999 } /* Literal.Number.Hex */
+.mi { color: #009999 } /* Literal.Number.Integer */
+.mo { color: #009999 } /* Literal.Number.Oct */
+.sb { color: #d01040 } /* Literal.String.Backtick */
+.sc { color: #d01040 } /* Literal.String.Char */
+.sd { color: #d01040 } /* Literal.String.Doc */
+.s2 { color: #d01040 } /* Literal.String.Double */
+.se { color: #d01040 } /* Literal.String.Escape */
+.sh { color: #d01040 } /* Literal.String.Heredoc */
+.si { color: #d01040 } /* Literal.String.Interpol */
+.sx { color: #d01040 } /* Literal.String.Other */
+.sr { color: #009926 } /* Literal.String.Regex */
+.s1 { color: #d01040 } /* Literal.String.Single */
+.ss { color: #990073 } /* Literal.String.Symbol */
+.bp { color: #999999 } /* Name.Builtin.Pseudo */
+.vc { color: #008080 } /* Name.Variable.Class */
+.vg { color: #008080 } /* Name.Variable.Global */
+.vi { color: #008080 } /* Name.Variable.Instance */
+.il { color: #009999 } /* Literal.Number.Integer.Long */
+.highlight .-Color-Black { color: #000000 } /* Color.Black */
+.highlight .-Color-Blue { color: #0000c0 } /* Color.Blue */
+.highlight .-Color-Cyan { color: #008080 } /* Color.Cyan */
+.highlight .-Color-Green { color: #008000 } /* Color.Green */
+.highlight .-Color-Magenta { color: #c000c0 } /* Color.Magenta */
+.highlight .-Color-Red { color: #c00000 } /* Color.Red */
+.highlight .-Color-White { color: #c0c0c0 } /* Color.White */
+.highlight .-Color-Yellow { color: #808000 } /* Color.Yellow */
diff --git a/doc/website-v1/css/font-awesome.css b/doc/website-v1/css/font-awesome.css
new file mode 100644
index 0000000..048cff9
--- /dev/null
+++ b/doc/website-v1/css/font-awesome.css
@@ -0,0 +1,1338 @@
+/*!
+ * Font Awesome 4.0.3 by @davegandy - http://fontawesome.io - @fontawesome
+ * License - http://fontawesome.io/license (Font: SIL OFL 1.1, CSS: MIT License)
+ */
+/* FONT PATH
+ * -------------------------- */
+@font-face {
+ font-family: 'FontAwesome';
+ src: url('../fonts/fontawesome-webfont.eot?v=4.0.3');
+ src: url('../fonts/fontawesome-webfont.eot?#iefix&v=4.0.3') format('embedded-opentype'), url('../fonts/fontawesome-webfont.woff?v=4.0.3') format('woff'), url('../fonts/fontawesome-webfont.ttf?v=4.0.3') format('truetype'), url('../fonts/fontawesome-webfont.svg?v=4.0.3#fontawesomeregular') format('svg');
+ font-weight: normal;
+ font-style: normal;
+}
+.fa {
+ display: inline-block;
+ font-family: FontAwesome;
+ font-style: normal;
+ font-weight: normal;
+ line-height: 1;
+ -webkit-font-smoothing: antialiased;
+ -moz-osx-font-smoothing: grayscale;
+}
+/* makes the font 33% larger relative to the icon container */
+.fa-lg {
+ font-size: 1.3333333333333333em;
+ line-height: 0.75em;
+ vertical-align: -15%;
+}
+.fa-2x {
+ font-size: 2em;
+}
+.fa-3x {
+ font-size: 3em;
+}
+.fa-4x {
+ font-size: 4em;
+}
+.fa-5x {
+ font-size: 5em;
+}
+.fa-fw {
+ width: 1.2857142857142858em;
+ text-align: center;
+}
+.fa-ul {
+ padding-left: 0;
+ margin-left: 2.142857142857143em;
+ list-style-type: none;
+}
+.fa-ul > li {
+ position: relative;
+}
+.fa-li {
+ position: absolute;
+ left: -2.142857142857143em;
+ width: 2.142857142857143em;
+ top: 0.14285714285714285em;
+ text-align: center;
+}
+.fa-li.fa-lg {
+ left: -1.8571428571428572em;
+}
+.fa-border {
+ padding: .2em .25em .15em;
+ border: solid 0.08em #eeeeee;
+ border-radius: .1em;
+}
+.pull-right {
+ float: right;
+}
+.pull-left {
+ float: left;
+}
+.fa.pull-left {
+ margin-right: .3em;
+}
+.fa.pull-right {
+ margin-left: .3em;
+}
+.fa-spin {
+ -webkit-animation: spin 2s infinite linear;
+ -moz-animation: spin 2s infinite linear;
+ -o-animation: spin 2s infinite linear;
+ animation: spin 2s infinite linear;
+}
+@-moz-keyframes spin {
+ 0% {
+ -moz-transform: rotate(0deg);
+ }
+ 100% {
+ -moz-transform: rotate(359deg);
+ }
+}
+@-webkit-keyframes spin {
+ 0% {
+ -webkit-transform: rotate(0deg);
+ }
+ 100% {
+ -webkit-transform: rotate(359deg);
+ }
+}
+@-o-keyframes spin {
+ 0% {
+ -o-transform: rotate(0deg);
+ }
+ 100% {
+ -o-transform: rotate(359deg);
+ }
+}
+@-ms-keyframes spin {
+ 0% {
+ -ms-transform: rotate(0deg);
+ }
+ 100% {
+ -ms-transform: rotate(359deg);
+ }
+}
+@keyframes spin {
+ 0% {
+ transform: rotate(0deg);
+ }
+ 100% {
+ transform: rotate(359deg);
+ }
+}
+.fa-rotate-90 {
+ filter: progid:DXImageTransform.Microsoft.BasicImage(rotation=1);
+ -webkit-transform: rotate(90deg);
+ -moz-transform: rotate(90deg);
+ -ms-transform: rotate(90deg);
+ -o-transform: rotate(90deg);
+ transform: rotate(90deg);
+}
+.fa-rotate-180 {
+ filter: progid:DXImageTransform.Microsoft.BasicImage(rotation=2);
+ -webkit-transform: rotate(180deg);
+ -moz-transform: rotate(180deg);
+ -ms-transform: rotate(180deg);
+ -o-transform: rotate(180deg);
+ transform: rotate(180deg);
+}
+.fa-rotate-270 {
+ filter: progid:DXImageTransform.Microsoft.BasicImage(rotation=3);
+ -webkit-transform: rotate(270deg);
+ -moz-transform: rotate(270deg);
+ -ms-transform: rotate(270deg);
+ -o-transform: rotate(270deg);
+ transform: rotate(270deg);
+}
+.fa-flip-horizontal {
+ filter: progid:DXImageTransform.Microsoft.BasicImage(rotation=0, mirror=1);
+ -webkit-transform: scale(-1, 1);
+ -moz-transform: scale(-1, 1);
+ -ms-transform: scale(-1, 1);
+ -o-transform: scale(-1, 1);
+ transform: scale(-1, 1);
+}
+.fa-flip-vertical {
+ filter: progid:DXImageTransform.Microsoft.BasicImage(rotation=2, mirror=1);
+ -webkit-transform: scale(1, -1);
+ -moz-transform: scale(1, -1);
+ -ms-transform: scale(1, -1);
+ -o-transform: scale(1, -1);
+ transform: scale(1, -1);
+}
+.fa-stack {
+ position: relative;
+ display: inline-block;
+ width: 2em;
+ height: 2em;
+ line-height: 2em;
+ vertical-align: middle;
+}
+.fa-stack-1x,
+.fa-stack-2x {
+ position: absolute;
+ left: 0;
+ width: 100%;
+ text-align: center;
+}
+.fa-stack-1x {
+ line-height: inherit;
+}
+.fa-stack-2x {
+ font-size: 2em;
+}
+.fa-inverse {
+ color: #ffffff;
+}
+/* Font Awesome uses the Unicode Private Use Area (PUA) to ensure screen
+ readers do not read off random characters that represent icons */
+.fa-glass:before {
+ content: "\f000";
+}
+.fa-music:before {
+ content: "\f001";
+}
+.fa-search:before {
+ content: "\f002";
+}
+.fa-envelope-o:before {
+ content: "\f003";
+}
+.fa-heart:before {
+ content: "\f004";
+}
+.fa-star:before {
+ content: "\f005";
+}
+.fa-star-o:before {
+ content: "\f006";
+}
+.fa-user:before {
+ content: "\f007";
+}
+.fa-film:before {
+ content: "\f008";
+}
+.fa-th-large:before {
+ content: "\f009";
+}
+.fa-th:before {
+ content: "\f00a";
+}
+.fa-th-list:before {
+ content: "\f00b";
+}
+.fa-check:before {
+ content: "\f00c";
+}
+.fa-times:before {
+ content: "\f00d";
+}
+.fa-search-plus:before {
+ content: "\f00e";
+}
+.fa-search-minus:before {
+ content: "\f010";
+}
+.fa-power-off:before {
+ content: "\f011";
+}
+.fa-signal:before {
+ content: "\f012";
+}
+.fa-gear:before,
+.fa-cog:before {
+ content: "\f013";
+}
+.fa-trash-o:before {
+ content: "\f014";
+}
+.fa-home:before {
+ content: "\f015";
+}
+.fa-file-o:before {
+ content: "\f016";
+}
+.fa-clock-o:before {
+ content: "\f017";
+}
+.fa-road:before {
+ content: "\f018";
+}
+.fa-download:before {
+ content: "\f019";
+}
+.fa-arrow-circle-o-down:before {
+ content: "\f01a";
+}
+.fa-arrow-circle-o-up:before {
+ content: "\f01b";
+}
+.fa-inbox:before {
+ content: "\f01c";
+}
+.fa-play-circle-o:before {
+ content: "\f01d";
+}
+.fa-rotate-right:before,
+.fa-repeat:before {
+ content: "\f01e";
+}
+.fa-refresh:before {
+ content: "\f021";
+}
+.fa-list-alt:before {
+ content: "\f022";
+}
+.fa-lock:before {
+ content: "\f023";
+}
+.fa-flag:before {
+ content: "\f024";
+}
+.fa-headphones:before {
+ content: "\f025";
+}
+.fa-volume-off:before {
+ content: "\f026";
+}
+.fa-volume-down:before {
+ content: "\f027";
+}
+.fa-volume-up:before {
+ content: "\f028";
+}
+.fa-qrcode:before {
+ content: "\f029";
+}
+.fa-barcode:before {
+ content: "\f02a";
+}
+.fa-tag:before {
+ content: "\f02b";
+}
+.fa-tags:before {
+ content: "\f02c";
+}
+.fa-book:before {
+ content: "\f02d";
+}
+.fa-bookmark:before {
+ content: "\f02e";
+}
+.fa-print:before {
+ content: "\f02f";
+}
+.fa-camera:before {
+ content: "\f030";
+}
+.fa-font:before {
+ content: "\f031";
+}
+.fa-bold:before {
+ content: "\f032";
+}
+.fa-italic:before {
+ content: "\f033";
+}
+.fa-text-height:before {
+ content: "\f034";
+}
+.fa-text-width:before {
+ content: "\f035";
+}
+.fa-align-left:before {
+ content: "\f036";
+}
+.fa-align-center:before {
+ content: "\f037";
+}
+.fa-align-right:before {
+ content: "\f038";
+}
+.fa-align-justify:before {
+ content: "\f039";
+}
+.fa-list:before {
+ content: "\f03a";
+}
+.fa-dedent:before,
+.fa-outdent:before {
+ content: "\f03b";
+}
+.fa-indent:before {
+ content: "\f03c";
+}
+.fa-video-camera:before {
+ content: "\f03d";
+}
+.fa-picture-o:before {
+ content: "\f03e";
+}
+.fa-pencil:before {
+ content: "\f040";
+}
+.fa-map-marker:before {
+ content: "\f041";
+}
+.fa-adjust:before {
+ content: "\f042";
+}
+.fa-tint:before {
+ content: "\f043";
+}
+.fa-edit:before,
+.fa-pencil-square-o:before {
+ content: "\f044";
+}
+.fa-share-square-o:before {
+ content: "\f045";
+}
+.fa-check-square-o:before {
+ content: "\f046";
+}
+.fa-arrows:before {
+ content: "\f047";
+}
+.fa-step-backward:before {
+ content: "\f048";
+}
+.fa-fast-backward:before {
+ content: "\f049";
+}
+.fa-backward:before {
+ content: "\f04a";
+}
+.fa-play:before {
+ content: "\f04b";
+}
+.fa-pause:before {
+ content: "\f04c";
+}
+.fa-stop:before {
+ content: "\f04d";
+}
+.fa-forward:before {
+ content: "\f04e";
+}
+.fa-fast-forward:before {
+ content: "\f050";
+}
+.fa-step-forward:before {
+ content: "\f051";
+}
+.fa-eject:before {
+ content: "\f052";
+}
+.fa-chevron-left:before {
+ content: "\f053";
+}
+.fa-chevron-right:before {
+ content: "\f054";
+}
+.fa-plus-circle:before {
+ content: "\f055";
+}
+.fa-minus-circle:before {
+ content: "\f056";
+}
+.fa-times-circle:before {
+ content: "\f057";
+}
+.fa-check-circle:before {
+ content: "\f058";
+}
+.fa-question-circle:before {
+ content: "\f059";
+}
+.fa-info-circle:before {
+ content: "\f05a";
+}
+.fa-crosshairs:before {
+ content: "\f05b";
+}
+.fa-times-circle-o:before {
+ content: "\f05c";
+}
+.fa-check-circle-o:before {
+ content: "\f05d";
+}
+.fa-ban:before {
+ content: "\f05e";
+}
+.fa-arrow-left:before {
+ content: "\f060";
+}
+.fa-arrow-right:before {
+ content: "\f061";
+}
+.fa-arrow-up:before {
+ content: "\f062";
+}
+.fa-arrow-down:before {
+ content: "\f063";
+}
+.fa-mail-forward:before,
+.fa-share:before {
+ content: "\f064";
+}
+.fa-expand:before {
+ content: "\f065";
+}
+.fa-compress:before {
+ content: "\f066";
+}
+.fa-plus:before {
+ content: "\f067";
+}
+.fa-minus:before {
+ content: "\f068";
+}
+.fa-asterisk:before {
+ content: "\f069";
+}
+.fa-exclamation-circle:before {
+ content: "\f06a";
+}
+.fa-gift:before {
+ content: "\f06b";
+}
+.fa-leaf:before {
+ content: "\f06c";
+}
+.fa-fire:before {
+ content: "\f06d";
+}
+.fa-eye:before {
+ content: "\f06e";
+}
+.fa-eye-slash:before {
+ content: "\f070";
+}
+.fa-warning:before,
+.fa-exclamation-triangle:before {
+ content: "\f071";
+}
+.fa-plane:before {
+ content: "\f072";
+}
+.fa-calendar:before {
+ content: "\f073";
+}
+.fa-random:before {
+ content: "\f074";
+}
+.fa-comment:before {
+ content: "\f075";
+}
+.fa-magnet:before {
+ content: "\f076";
+}
+.fa-chevron-up:before {
+ content: "\f077";
+}
+.fa-chevron-down:before {
+ content: "\f078";
+}
+.fa-retweet:before {
+ content: "\f079";
+}
+.fa-shopping-cart:before {
+ content: "\f07a";
+}
+.fa-folder:before {
+ content: "\f07b";
+}
+.fa-folder-open:before {
+ content: "\f07c";
+}
+.fa-arrows-v:before {
+ content: "\f07d";
+}
+.fa-arrows-h:before {
+ content: "\f07e";
+}
+.fa-bar-chart-o:before {
+ content: "\f080";
+}
+.fa-twitter-square:before {
+ content: "\f081";
+}
+.fa-facebook-square:before {
+ content: "\f082";
+}
+.fa-camera-retro:before {
+ content: "\f083";
+}
+.fa-key:before {
+ content: "\f084";
+}
+.fa-gears:before,
+.fa-cogs:before {
+ content: "\f085";
+}
+.fa-comments:before {
+ content: "\f086";
+}
+.fa-thumbs-o-up:before {
+ content: "\f087";
+}
+.fa-thumbs-o-down:before {
+ content: "\f088";
+}
+.fa-star-half:before {
+ content: "\f089";
+}
+.fa-heart-o:before {
+ content: "\f08a";
+}
+.fa-sign-out:before {
+ content: "\f08b";
+}
+.fa-linkedin-square:before {
+ content: "\f08c";
+}
+.fa-thumb-tack:before {
+ content: "\f08d";
+}
+.fa-external-link:before {
+ content: "\f08e";
+}
+.fa-sign-in:before {
+ content: "\f090";
+}
+.fa-trophy:before {
+ content: "\f091";
+}
+.fa-github-square:before {
+ content: "\f092";
+}
+.fa-upload:before {
+ content: "\f093";
+}
+.fa-lemon-o:before {
+ content: "\f094";
+}
+.fa-phone:before {
+ content: "\f095";
+}
+.fa-square-o:before {
+ content: "\f096";
+}
+.fa-bookmark-o:before {
+ content: "\f097";
+}
+.fa-phone-square:before {
+ content: "\f098";
+}
+.fa-twitter:before {
+ content: "\f099";
+}
+.fa-facebook:before {
+ content: "\f09a";
+}
+.fa-github:before {
+ content: "\f09b";
+}
+.fa-unlock:before {
+ content: "\f09c";
+}
+.fa-credit-card:before {
+ content: "\f09d";
+}
+.fa-rss:before {
+ content: "\f09e";
+}
+.fa-hdd-o:before {
+ content: "\f0a0";
+}
+.fa-bullhorn:before {
+ content: "\f0a1";
+}
+.fa-bell:before {
+ content: "\f0f3";
+}
+.fa-certificate:before {
+ content: "\f0a3";
+}
+.fa-hand-o-right:before {
+ content: "\f0a4";
+}
+.fa-hand-o-left:before {
+ content: "\f0a5";
+}
+.fa-hand-o-up:before {
+ content: "\f0a6";
+}
+.fa-hand-o-down:before {
+ content: "\f0a7";
+}
+.fa-arrow-circle-left:before {
+ content: "\f0a8";
+}
+.fa-arrow-circle-right:before {
+ content: "\f0a9";
+}
+.fa-arrow-circle-up:before {
+ content: "\f0aa";
+}
+.fa-arrow-circle-down:before {
+ content: "\f0ab";
+}
+.fa-globe:before {
+ content: "\f0ac";
+}
+.fa-wrench:before {
+ content: "\f0ad";
+}
+.fa-tasks:before {
+ content: "\f0ae";
+}
+.fa-filter:before {
+ content: "\f0b0";
+}
+.fa-briefcase:before {
+ content: "\f0b1";
+}
+.fa-arrows-alt:before {
+ content: "\f0b2";
+}
+.fa-group:before,
+.fa-users:before {
+ content: "\f0c0";
+}
+.fa-chain:before,
+.fa-link:before {
+ content: "\f0c1";
+}
+.fa-cloud:before {
+ content: "\f0c2";
+}
+.fa-flask:before {
+ content: "\f0c3";
+}
+.fa-cut:before,
+.fa-scissors:before {
+ content: "\f0c4";
+}
+.fa-copy:before,
+.fa-files-o:before {
+ content: "\f0c5";
+}
+.fa-paperclip:before {
+ content: "\f0c6";
+}
+.fa-save:before,
+.fa-floppy-o:before {
+ content: "\f0c7";
+}
+.fa-square:before {
+ content: "\f0c8";
+}
+.fa-bars:before {
+ content: "\f0c9";
+}
+.fa-list-ul:before {
+ content: "\f0ca";
+}
+.fa-list-ol:before {
+ content: "\f0cb";
+}
+.fa-strikethrough:before {
+ content: "\f0cc";
+}
+.fa-underline:before {
+ content: "\f0cd";
+}
+.fa-table:before {
+ content: "\f0ce";
+}
+.fa-magic:before {
+ content: "\f0d0";
+}
+.fa-truck:before {
+ content: "\f0d1";
+}
+.fa-pinterest:before {
+ content: "\f0d2";
+}
+.fa-pinterest-square:before {
+ content: "\f0d3";
+}
+.fa-google-plus-square:before {
+ content: "\f0d4";
+}
+.fa-google-plus:before {
+ content: "\f0d5";
+}
+.fa-money:before {
+ content: "\f0d6";
+}
+.fa-caret-down:before {
+ content: "\f0d7";
+}
+.fa-caret-up:before {
+ content: "\f0d8";
+}
+.fa-caret-left:before {
+ content: "\f0d9";
+}
+.fa-caret-right:before {
+ content: "\f0da";
+}
+.fa-columns:before {
+ content: "\f0db";
+}
+.fa-unsorted:before,
+.fa-sort:before {
+ content: "\f0dc";
+}
+.fa-sort-down:before,
+.fa-sort-asc:before {
+ content: "\f0dd";
+}
+.fa-sort-up:before,
+.fa-sort-desc:before {
+ content: "\f0de";
+}
+.fa-envelope:before {
+ content: "\f0e0";
+}
+.fa-linkedin:before {
+ content: "\f0e1";
+}
+.fa-rotate-left:before,
+.fa-undo:before {
+ content: "\f0e2";
+}
+.fa-legal:before,
+.fa-gavel:before {
+ content: "\f0e3";
+}
+.fa-dashboard:before,
+.fa-tachometer:before {
+ content: "\f0e4";
+}
+.fa-comment-o:before {
+ content: "\f0e5";
+}
+.fa-comments-o:before {
+ content: "\f0e6";
+}
+.fa-flash:before,
+.fa-bolt:before {
+ content: "\f0e7";
+}
+.fa-sitemap:before {
+ content: "\f0e8";
+}
+.fa-umbrella:before {
+ content: "\f0e9";
+}
+.fa-paste:before,
+.fa-clipboard:before {
+ content: "\f0ea";
+}
+.fa-lightbulb-o:before {
+ content: "\f0eb";
+}
+.fa-exchange:before {
+ content: "\f0ec";
+}
+.fa-cloud-download:before {
+ content: "\f0ed";
+}
+.fa-cloud-upload:before {
+ content: "\f0ee";
+}
+.fa-user-md:before {
+ content: "\f0f0";
+}
+.fa-stethoscope:before {
+ content: "\f0f1";
+}
+.fa-suitcase:before {
+ content: "\f0f2";
+}
+.fa-bell-o:before {
+ content: "\f0a2";
+}
+.fa-coffee:before {
+ content: "\f0f4";
+}
+.fa-cutlery:before {
+ content: "\f0f5";
+}
+.fa-file-text-o:before {
+ content: "\f0f6";
+}
+.fa-building-o:before {
+ content: "\f0f7";
+}
+.fa-hospital-o:before {
+ content: "\f0f8";
+}
+.fa-ambulance:before {
+ content: "\f0f9";
+}
+.fa-medkit:before {
+ content: "\f0fa";
+}
+.fa-fighter-jet:before {
+ content: "\f0fb";
+}
+.fa-beer:before {
+ content: "\f0fc";
+}
+.fa-h-square:before {
+ content: "\f0fd";
+}
+.fa-plus-square:before {
+ content: "\f0fe";
+}
+.fa-angle-double-left:before {
+ content: "\f100";
+}
+.fa-angle-double-right:before {
+ content: "\f101";
+}
+.fa-angle-double-up:before {
+ content: "\f102";
+}
+.fa-angle-double-down:before {
+ content: "\f103";
+}
+.fa-angle-left:before {
+ content: "\f104";
+}
+.fa-angle-right:before {
+ content: "\f105";
+}
+.fa-angle-up:before {
+ content: "\f106";
+}
+.fa-angle-down:before {
+ content: "\f107";
+}
+.fa-desktop:before {
+ content: "\f108";
+}
+.fa-laptop:before {
+ content: "\f109";
+}
+.fa-tablet:before {
+ content: "\f10a";
+}
+.fa-mobile-phone:before,
+.fa-mobile:before {
+ content: "\f10b";
+}
+.fa-circle-o:before {
+ content: "\f10c";
+}
+.fa-quote-left:before {
+ content: "\f10d";
+}
+.fa-quote-right:before {
+ content: "\f10e";
+}
+.fa-spinner:before {
+ content: "\f110";
+}
+.fa-circle:before {
+ content: "\f111";
+}
+.fa-mail-reply:before,
+.fa-reply:before {
+ content: "\f112";
+}
+.fa-github-alt:before {
+ content: "\f113";
+}
+.fa-folder-o:before {
+ content: "\f114";
+}
+.fa-folder-open-o:before {
+ content: "\f115";
+}
+.fa-smile-o:before {
+ content: "\f118";
+}
+.fa-frown-o:before {
+ content: "\f119";
+}
+.fa-meh-o:before {
+ content: "\f11a";
+}
+.fa-gamepad:before {
+ content: "\f11b";
+}
+.fa-keyboard-o:before {
+ content: "\f11c";
+}
+.fa-flag-o:before {
+ content: "\f11d";
+}
+.fa-flag-checkered:before {
+ content: "\f11e";
+}
+.fa-terminal:before {
+ content: "\f120";
+}
+.fa-code:before {
+ content: "\f121";
+}
+.fa-reply-all:before {
+ content: "\f122";
+}
+.fa-mail-reply-all:before {
+ content: "\f122";
+}
+.fa-star-half-empty:before,
+.fa-star-half-full:before,
+.fa-star-half-o:before {
+ content: "\f123";
+}
+.fa-location-arrow:before {
+ content: "\f124";
+}
+.fa-crop:before {
+ content: "\f125";
+}
+.fa-code-fork:before {
+ content: "\f126";
+}
+.fa-unlink:before,
+.fa-chain-broken:before {
+ content: "\f127";
+}
+.fa-question:before {
+ content: "\f128";
+}
+.fa-info:before {
+ content: "\f129";
+}
+.fa-exclamation:before {
+ content: "\f12a";
+}
+.fa-superscript:before {
+ content: "\f12b";
+}
+.fa-subscript:before {
+ content: "\f12c";
+}
+.fa-eraser:before {
+ content: "\f12d";
+}
+.fa-puzzle-piece:before {
+ content: "\f12e";
+}
+.fa-microphone:before {
+ content: "\f130";
+}
+.fa-microphone-slash:before {
+ content: "\f131";
+}
+.fa-shield:before {
+ content: "\f132";
+}
+.fa-calendar-o:before {
+ content: "\f133";
+}
+.fa-fire-extinguisher:before {
+ content: "\f134";
+}
+.fa-rocket:before {
+ content: "\f135";
+}
+.fa-maxcdn:before {
+ content: "\f136";
+}
+.fa-chevron-circle-left:before {
+ content: "\f137";
+}
+.fa-chevron-circle-right:before {
+ content: "\f138";
+}
+.fa-chevron-circle-up:before {
+ content: "\f139";
+}
+.fa-chevron-circle-down:before {
+ content: "\f13a";
+}
+.fa-html5:before {
+ content: "\f13b";
+}
+.fa-css3:before {
+ content: "\f13c";
+}
+.fa-anchor:before {
+ content: "\f13d";
+}
+.fa-unlock-alt:before {
+ content: "\f13e";
+}
+.fa-bullseye:before {
+ content: "\f140";
+}
+.fa-ellipsis-h:before {
+ content: "\f141";
+}
+.fa-ellipsis-v:before {
+ content: "\f142";
+}
+.fa-rss-square:before {
+ content: "\f143";
+}
+.fa-play-circle:before {
+ content: "\f144";
+}
+.fa-ticket:before {
+ content: "\f145";
+}
+.fa-minus-square:before {
+ content: "\f146";
+}
+.fa-minus-square-o:before {
+ content: "\f147";
+}
+.fa-level-up:before {
+ content: "\f148";
+}
+.fa-level-down:before {
+ content: "\f149";
+}
+.fa-check-square:before {
+ content: "\f14a";
+}
+.fa-pencil-square:before {
+ content: "\f14b";
+}
+.fa-external-link-square:before {
+ content: "\f14c";
+}
+.fa-share-square:before {
+ content: "\f14d";
+}
+.fa-compass:before {
+ content: "\f14e";
+}
+.fa-toggle-down:before,
+.fa-caret-square-o-down:before {
+ content: "\f150";
+}
+.fa-toggle-up:before,
+.fa-caret-square-o-up:before {
+ content: "\f151";
+}
+.fa-toggle-right:before,
+.fa-caret-square-o-right:before {
+ content: "\f152";
+}
+.fa-euro:before,
+.fa-eur:before {
+ content: "\f153";
+}
+.fa-gbp:before {
+ content: "\f154";
+}
+.fa-dollar:before,
+.fa-usd:before {
+ content: "\f155";
+}
+.fa-rupee:before,
+.fa-inr:before {
+ content: "\f156";
+}
+.fa-cny:before,
+.fa-rmb:before,
+.fa-yen:before,
+.fa-jpy:before {
+ content: "\f157";
+}
+.fa-ruble:before,
+.fa-rouble:before,
+.fa-rub:before {
+ content: "\f158";
+}
+.fa-won:before,
+.fa-krw:before {
+ content: "\f159";
+}
+.fa-bitcoin:before,
+.fa-btc:before {
+ content: "\f15a";
+}
+.fa-file:before {
+ content: "\f15b";
+}
+.fa-file-text:before {
+ content: "\f15c";
+}
+.fa-sort-alpha-asc:before {
+ content: "\f15d";
+}
+.fa-sort-alpha-desc:before {
+ content: "\f15e";
+}
+.fa-sort-amount-asc:before {
+ content: "\f160";
+}
+.fa-sort-amount-desc:before {
+ content: "\f161";
+}
+.fa-sort-numeric-asc:before {
+ content: "\f162";
+}
+.fa-sort-numeric-desc:before {
+ content: "\f163";
+}
+.fa-thumbs-up:before {
+ content: "\f164";
+}
+.fa-thumbs-down:before {
+ content: "\f165";
+}
+.fa-youtube-square:before {
+ content: "\f166";
+}
+.fa-youtube:before {
+ content: "\f167";
+}
+.fa-xing:before {
+ content: "\f168";
+}
+.fa-xing-square:before {
+ content: "\f169";
+}
+.fa-youtube-play:before {
+ content: "\f16a";
+}
+.fa-dropbox:before {
+ content: "\f16b";
+}
+.fa-stack-overflow:before {
+ content: "\f16c";
+}
+.fa-instagram:before {
+ content: "\f16d";
+}
+.fa-flickr:before {
+ content: "\f16e";
+}
+.fa-adn:before {
+ content: "\f170";
+}
+.fa-bitbucket:before {
+ content: "\f171";
+}
+.fa-bitbucket-square:before {
+ content: "\f172";
+}
+.fa-tumblr:before {
+ content: "\f173";
+}
+.fa-tumblr-square:before {
+ content: "\f174";
+}
+.fa-long-arrow-down:before {
+ content: "\f175";
+}
+.fa-long-arrow-up:before {
+ content: "\f176";
+}
+.fa-long-arrow-left:before {
+ content: "\f177";
+}
+.fa-long-arrow-right:before {
+ content: "\f178";
+}
+.fa-apple:before {
+ content: "\f179";
+}
+.fa-windows:before {
+ content: "\f17a";
+}
+.fa-android:before {
+ content: "\f17b";
+}
+.fa-linux:before {
+ content: "\f17c";
+}
+.fa-dribbble:before {
+ content: "\f17d";
+}
+.fa-skype:before {
+ content: "\f17e";
+}
+.fa-foursquare:before {
+ content: "\f180";
+}
+.fa-trello:before {
+ content: "\f181";
+}
+.fa-female:before {
+ content: "\f182";
+}
+.fa-male:before {
+ content: "\f183";
+}
+.fa-gittip:before {
+ content: "\f184";
+}
+.fa-sun-o:before {
+ content: "\f185";
+}
+.fa-moon-o:before {
+ content: "\f186";
+}
+.fa-archive:before {
+ content: "\f187";
+}
+.fa-bug:before {
+ content: "\f188";
+}
+.fa-vk:before {
+ content: "\f189";
+}
+.fa-weibo:before {
+ content: "\f18a";
+}
+.fa-renren:before {
+ content: "\f18b";
+}
+.fa-pagelines:before {
+ content: "\f18c";
+}
+.fa-stack-exchange:before {
+ content: "\f18d";
+}
+.fa-arrow-circle-o-right:before {
+ content: "\f18e";
+}
+.fa-arrow-circle-o-left:before {
+ content: "\f190";
+}
+.fa-toggle-left:before,
+.fa-caret-square-o-left:before {
+ content: "\f191";
+}
+.fa-dot-circle-o:before {
+ content: "\f192";
+}
+.fa-wheelchair:before {
+ content: "\f193";
+}
+.fa-vimeo-square:before {
+ content: "\f194";
+}
+.fa-turkish-lira:before,
+.fa-try:before {
+ content: "\f195";
+}
+.fa-plus-square-o:before {
+ content: "\f196";
+}
diff --git a/doc/website-v1/css/font-awesome.min.css b/doc/website-v1/css/font-awesome.min.css
new file mode 100644
index 0000000..449d6ac
--- /dev/null
+++ b/doc/website-v1/css/font-awesome.min.css
@@ -0,0 +1,4 @@
+/*!
+ * Font Awesome 4.0.3 by @davegandy - http://fontawesome.io - @fontawesome
+ * License - http://fontawesome.io/license (Font: SIL OFL 1.1, CSS: MIT License)
+ */@font-face{font-family:'FontAwesome';src:url('../fonts/fontawesome-webfont.eot?v=4.0.3');src:url('../fonts/fontawesome-webfont.eot?#iefix&v=4.0.3') format('embedded-opentype'),url('../fonts/fontawesome-webfont.woff?v=4.0.3') format('woff'),url('../fonts/fontawesome-webfont.ttf?v=4.0.3') format('truetype'),url('../fonts/fontawesome-webfont.svg?v=4.0.3#fontawesomeregular') format('svg');font-weight:normal;font-style:normal}.fa{display:inline-block;font-family:FontAwesome;font-style:normal;font-weight:normal;line-height:1;-webkit-font-smoothing:antialiased;-moz-osx-font-smoothing:grayscale}.fa-lg{font-size:1.3333333333333333em;line-height:.75em;vertical-align:-15%}.fa-2x{font-size:2em}.fa-3x{font-size:3em}.fa-4x{font-size:4em}.fa-5x{font-size:5em}.fa-fw{width:1.2857142857142858em;text-align:center}.fa-ul{padding-left:0;margin-left:2.142857142857143em;list-style-type:none}.fa-ul>li{position:relative}.fa-li{position:absolute;left:-2.142857142857143em;width:2.142857142857143em;top:.14285714285714285em;text-align:center}.fa-li.fa-lg{left:-1.8571428571428572em}.fa-border{padding:.2em .25em .15em;border:solid .08em #eee;border-radius:.1em}.pull-right{float:right}.pull-left{float:left}.fa.pull-left{margin-right:.3em}.fa.pull-right{margin-left:.3em}.fa-spin{-webkit-animation:spin 2s infinite linear;-moz-animation:spin 2s infinite linear;-o-animation:spin 2s infinite linear;animation:spin 2s infinite linear}@-moz-keyframes spin{0%{-moz-transform:rotate(0deg)}100%{-moz-transform:rotate(359deg)}}@-webkit-keyframes spin{0%{-webkit-transform:rotate(0deg)}100%{-webkit-transform:rotate(359deg)}}@-o-keyframes spin{0%{-o-transform:rotate(0deg)}100%{-o-transform:rotate(359deg)}}@-ms-keyframes spin{0%{-ms-transform:rotate(0deg)}100%{-ms-transform:rotate(359deg)}}@keyframes spin{0%{transform:rotate(0deg)}100%{transform:rotate(359deg)}}.fa-rotate-90{filter:progid:DXImageTransform.Microsoft.BasicImage(rotation=1);-webkit-transform:rotate(90deg);-moz-transform:rotate(90deg);-ms-transform:rotate(90deg);-o-transform:rotate(90deg);transform:rotate(90deg)}.fa-rotate-180{filter:progid:DXImageTransform.Microsoft.BasicImage(rotation=2);-webkit-transform:rotate(180deg);-moz-transform:rotate(180deg);-ms-transform:rotate(180deg);-o-transform:rotate(180deg);transform:rotate(180deg)}.fa-rotate-270{filter:progid:DXImageTransform.Microsoft.BasicImage(rotation=3);-webkit-transform:rotate(270deg);-moz-transform:rotate(270deg);-ms-transform:rotate(270deg);-o-transform:rotate(270deg);transform:rotate(270deg)}.fa-flip-horizontal{filter:progid:DXImageTransform.Microsoft.BasicImage(rotation=0,mirror=1);-webkit-transform:scale(-1,1);-moz-transform:scale(-1,1);-ms-transform:scale(-1,1);-o-transform:scale(-1,1);transform:scale(-1,1)}.fa-flip-vertical{filter:progid:DXImageTransform.Microsoft.BasicImage(rotation=2,mirror=1);-webkit-transform:scale(1,-1);-moz-transform:scale(1,-1);-ms-transform:scale(1,-1);-o-transform:scale(1,-1);transform:scale(1,-1)}.fa-stack{position:relative;display:inline-block;width:2em;height:2em;line-height:2em;vertical-align:middle}.fa-stack-1x,.fa-stack-2x{position:absolute;left:0;width:100%;text-align:center}.fa-stack-1x{line-height:inherit}.fa-stack-2x{font-size:2em}.fa-inverse{color:#fff}.fa-glass:before{content:"\f000"}.fa-music:before{content:"\f001"}.fa-search:before{content:"\f002"}.fa-envelope-o:before{content:"\f003"}.fa-heart:before{content:"\f004"}.fa-star:before{content:"\f005"}.fa-star-o:before{content:"\f006"}.fa-user:before{content:"\f007"}.fa-film:before{content:"\f008"}.fa-th-large:before{content:"\f009"}.fa-th:before{content:"\f00a"}.fa-th-list:before{content:"\f00b"}.fa-check:before{content:"\f00c"}.fa-times:before{content:"\f00d"}.fa-search-plus:before{content:"\f00e"}.fa-search-minus:before{content:"\f010"}.fa-power-off:before{content:"\f011"}.fa-signal:before{content:"\f012"}.fa-gear:before,.fa-cog:before{content:"\f013"}.fa-trash-o:before{content:"\f014"}.fa-home:before{content:"\f015"}.fa-file-o:before{content:"\f016"}.fa-clock-o:before{content:"\f017"}.fa-road:before{content:"\f018"}.fa-download:before{content:"\f019"}.fa-arrow-circle-o-down:before{content:"\f01a"}.fa-arrow-circle-o-up:before{content:"\f01b"}.fa-inbox:before{content:"\f01c"}.fa-play-circle-o:before{content:"\f01d"}.fa-rotate-right:before,.fa-repeat:before{content:"\f01e"}.fa-refresh:before{content:"\f021"}.fa-list-alt:before{content:"\f022"}.fa-lock:before{content:"\f023"}.fa-flag:before{content:"\f024"}.fa-headphones:before{content:"\f025"}.fa-volume-off:before{content:"\f026"}.fa-volume-down:before{content:"\f027"}.fa-volume-up:before{content:"\f028"}.fa-qrcode:before{content:"\f029"}.fa-barcode:before{content:"\f02a"}.fa-tag:before{content:"\f02b"}.fa-tags:before{content:"\f02c"}.fa-book:before{content:"\f02d"}.fa-bookmark:before{content:"\f02e"}.fa-print:before{content:"\f02f"}.fa-camera:before{content:"\f030"}.fa-font:before{content:"\f031"}.fa-bold:before{content:"\f032"}.fa-italic:before{content:"\f033"}.fa-text-height:before{content:"\f034"}.fa-text-width:before{content:"\f035"}.fa-align-left:before{content:"\f036"}.fa-align-center:before{content:"\f037"}.fa-align-right:before{content:"\f038"}.fa-align-justify:before{content:"\f039"}.fa-list:before{content:"\f03a"}.fa-dedent:before,.fa-outdent:before{content:"\f03b"}.fa-indent:before{content:"\f03c"}.fa-video-camera:before{content:"\f03d"}.fa-picture-o:before{content:"\f03e"}.fa-pencil:before{content:"\f040"}.fa-map-marker:before{content:"\f041"}.fa-adjust:before{content:"\f042"}.fa-tint:before{content:"\f043"}.fa-edit:before,.fa-pencil-square-o:before{content:"\f044"}.fa-share-square-o:before{content:"\f045"}.fa-check-square-o:before{content:"\f046"}.fa-arrows:before{content:"\f047"}.fa-step-backward:before{content:"\f048"}.fa-fast-backward:before{content:"\f049"}.fa-backward:before{content:"\f04a"}.fa-play:before{content:"\f04b"}.fa-pause:before{content:"\f04c"}.fa-stop:before{content:"\f04d"}.fa-forward:before{content:"\f04e"}.fa-fast-forward:before{content:"\f050"}.fa-step-forward:before{content:"\f051"}.fa-eject:before{content:"\f052"}.fa-chevron-left:before{content:"\f053"}.fa-chevron-right:before{content:"\f054"}.fa-plus-circle:before{content:"\f055"}.fa-minus-circle:before{content:"\f056"}.fa-times-circle:before{content:"\f057"}.fa-check-circle:before{content:"\f058"}.fa-question-circle:before{content:"\f059"}.fa-info-circle:before{content:"\f05a"}.fa-crosshairs:before{content:"\f05b"}.fa-times-circle-o:before{content:"\f05c"}.fa-check-circle-o:before{content:"\f05d"}.fa-ban:before{content:"\f05e"}.fa-arrow-left:before{content:"\f060"}.fa-arrow-right:before{content:"\f061"}.fa-arrow-up:before{content:"\f062"}.fa-arrow-down:before{content:"\f063"}.fa-mail-forward:before,.fa-share:before{content:"\f064"}.fa-expand:before{content:"\f065"}.fa-compress:before{content:"\f066"}.fa-plus:before{content:"\f067"}.fa-minus:before{content:"\f068"}.fa-asterisk:before{content:"\f069"}.fa-exclamation-circle:before{content:"\f06a"}.fa-gift:before{content:"\f06b"}.fa-leaf:before{content:"\f06c"}.fa-fire:before{content:"\f06d"}.fa-eye:before{content:"\f06e"}.fa-eye-slash:before{content:"\f070"}.fa-warning:before,.fa-exclamation-triangle:before{content:"\f071"}.fa-plane:before{content:"\f072"}.fa-calendar:before{content:"\f073"}.fa-random:before{content:"\f074"}.fa-comment:before{content:"\f075"}.fa-magnet:before{content:"\f076"}.fa-chevron-up:before{content:"\f077"}.fa-chevron-down:before{content:"\f078"}.fa-retweet:before{content:"\f079"}.fa-shopping-cart:before{content:"\f07a"}.fa-folder:before{content:"\f07b"}.fa-folder-open:before{content:"\f07c"}.fa-arrows-v:before{content:"\f07d"}.fa-arrows-h:before{content:"\f07e"}.fa-bar-chart-o:before{content:"\f080"}.fa-twitter-square:before{content:"\f081"}.fa-facebook-square:before{content:"\f082"}.fa-camera-retro:before{content:"\f083"}.fa-key:before{content:"\f084"}.fa-gears:before,.fa-cogs:before{content:"\f085"}.fa-comments:before{content:"\f086"}.fa-thumbs-o-up:before{content:"\f087"}.fa-thumbs-o-down:before{content:"\f088"}.fa-star-half:before{content:"\f089"}.fa-heart-o:before{content:"\f08a"}.fa-sign-out:before{content:"\f08b"}.fa-linkedin-square:before{content:"\f08c"}.fa-thumb-tack:before{content:"\f08d"}.fa-external-link:before{content:"\f08e"}.fa-sign-in:before{content:"\f090"}.fa-trophy:before{content:"\f091"}.fa-github-square:before{content:"\f092"}.fa-upload:before{content:"\f093"}.fa-lemon-o:before{content:"\f094"}.fa-phone:before{content:"\f095"}.fa-square-o:before{content:"\f096"}.fa-bookmark-o:before{content:"\f097"}.fa-phone-square:before{content:"\f098"}.fa-twitter:before{content:"\f099"}.fa-facebook:before{content:"\f09a"}.fa-github:before{content:"\f09b"}.fa-unlock:before{content:"\f09c"}.fa-credit-card:before{content:"\f09d"}.fa-rss:before{content:"\f09e"}.fa-hdd-o:before{content:"\f0a0"}.fa-bullhorn:before{content:"\f0a1"}.fa-bell:before{content:"\f0f3"}.fa-certificate:before{content:"\f0a3"}.fa-hand-o-right:before{content:"\f0a4"}.fa-hand-o-left:before{content:"\f0a5"}.fa-hand-o-up:before{content:"\f0a6"}.fa-hand-o-down:before{content:"\f0a7"}.fa-arrow-circle-left:before{content:"\f0a8"}.fa-arrow-circle-right:before{content:"\f0a9"}.fa-arrow-circle-up:before{content:"\f0aa"}.fa-arrow-circle-down:before{content:"\f0ab"}.fa-globe:before{content:"\f0ac"}.fa-wrench:before{content:"\f0ad"}.fa-tasks:before{content:"\f0ae"}.fa-filter:before{content:"\f0b0"}.fa-briefcase:before{content:"\f0b1"}.fa-arrows-alt:before{content:"\f0b2"}.fa-group:before,.fa-users:before{content:"\f0c0"}.fa-chain:before,.fa-link:before{content:"\f0c1"}.fa-cloud:before{content:"\f0c2"}.fa-flask:before{content:"\f0c3"}.fa-cut:before,.fa-scissors:before{content:"\f0c4"}.fa-copy:before,.fa-files-o:before{content:"\f0c5"}.fa-paperclip:before{content:"\f0c6"}.fa-save:before,.fa-floppy-o:before{content:"\f0c7"}.fa-square:before{content:"\f0c8"}.fa-bars:before{content:"\f0c9"}.fa-list-ul:before{content:"\f0ca"}.fa-list-ol:before{content:"\f0cb"}.fa-strikethrough:before{content:"\f0cc"}.fa-underline:before{content:"\f0cd"}.fa-table:before{content:"\f0ce"}.fa-magic:before{content:"\f0d0"}.fa-truck:before{content:"\f0d1"}.fa-pinterest:before{content:"\f0d2"}.fa-pinterest-square:before{content:"\f0d3"}.fa-google-plus-square:before{content:"\f0d4"}.fa-google-plus:before{content:"\f0d5"}.fa-money:before{content:"\f0d6"}.fa-caret-down:before{content:"\f0d7"}.fa-caret-up:before{content:"\f0d8"}.fa-caret-left:before{content:"\f0d9"}.fa-caret-right:before{content:"\f0da"}.fa-columns:before{content:"\f0db"}.fa-unsorted:before,.fa-sort:before{content:"\f0dc"}.fa-sort-down:before,.fa-sort-asc:before{content:"\f0dd"}.fa-sort-up:before,.fa-sort-desc:before{content:"\f0de"}.fa-envelope:before{content:"\f0e0"}.fa-linkedin:before{content:"\f0e1"}.fa-rotate-left:before,.fa-undo:before{content:"\f0e2"}.fa-legal:before,.fa-gavel:before{content:"\f0e3"}.fa-dashboard:before,.fa-tachometer:before{content:"\f0e4"}.fa-comment-o:before{content:"\f0e5"}.fa-comments-o:before{content:"\f0e6"}.fa-flash:before,.fa-bolt:before{content:"\f0e7"}.fa-sitemap:before{content:"\f0e8"}.fa-umbrella:before{content:"\f0e9"}.fa-paste:before,.fa-clipboard:before{content:"\f0ea"}.fa-lightbulb-o:before{content:"\f0eb"}.fa-exchange:before{content:"\f0ec"}.fa-cloud-download:before{content:"\f0ed"}.fa-cloud-upload:before{content:"\f0ee"}.fa-user-md:before{content:"\f0f0"}.fa-stethoscope:before{content:"\f0f1"}.fa-suitcase:before{content:"\f0f2"}.fa-bell-o:before{content:"\f0a2"}.fa-coffee:before{content:"\f0f4"}.fa-cutlery:before{content:"\f0f5"}.fa-file-text-o:before{content:"\f0f6"}.fa-building-o:before{content:"\f0f7"}.fa-hospital-o:before{content:"\f0f8"}.fa-ambulance:before{content:"\f0f9"}.fa-medkit:before{content:"\f0fa"}.fa-fighter-jet:before{content:"\f0fb"}.fa-beer:before{content:"\f0fc"}.fa-h-square:before{content:"\f0fd"}.fa-plus-square:before{content:"\f0fe"}.fa-angle-double-left:before{content:"\f100"}.fa-angle-double-right:before{content:"\f101"}.fa-angle-double-up:before{content:"\f102"}.fa-angle-double-down:before{content:"\f103"}.fa-angle-left:before{content:"\f104"}.fa-angle-right:before{content:"\f105"}.fa-angle-up:before{content:"\f106"}.fa-angle-down:before{content:"\f107"}.fa-desktop:before{content:"\f108"}.fa-laptop:before{content:"\f109"}.fa-tablet:before{content:"\f10a"}.fa-mobile-phone:before,.fa-mobile:before{content:"\f10b"}.fa-circle-o:before{content:"\f10c"}.fa-quote-left:before{content:"\f10d"}.fa-quote-right:before{content:"\f10e"}.fa-spinner:before{content:"\f110"}.fa-circle:before{content:"\f111"}.fa-mail-reply:before,.fa-reply:before{content:"\f112"}.fa-github-alt:before{content:"\f113"}.fa-folder-o:before{content:"\f114"}.fa-folder-open-o:before{content:"\f115"}.fa-smile-o:before{content:"\f118"}.fa-frown-o:before{content:"\f119"}.fa-meh-o:before{content:"\f11a"}.fa-gamepad:before{content:"\f11b"}.fa-keyboard-o:before{content:"\f11c"}.fa-flag-o:before{content:"\f11d"}.fa-flag-checkered:before{content:"\f11e"}.fa-terminal:before{content:"\f120"}.fa-code:before{content:"\f121"}.fa-reply-all:before{content:"\f122"}.fa-mail-reply-all:before{content:"\f122"}.fa-star-half-empty:before,.fa-star-half-full:before,.fa-star-half-o:before{content:"\f123"}.fa-location-arrow:before{content:"\f124"}.fa-crop:before{content:"\f125"}.fa-code-fork:before{content:"\f126"}.fa-unlink:before,.fa-chain-broken:before{content:"\f127"}.fa-question:before{content:"\f128"}.fa-info:before{content:"\f129"}.fa-exclamation:before{content:"\f12a"}.fa-superscript:before{content:"\f12b"}.fa-subscript:before{content:"\f12c"}.fa-eraser:before{content:"\f12d"}.fa-puzzle-piece:before{content:"\f12e"}.fa-microphone:before{content:"\f130"}.fa-microphone-slash:before{content:"\f131"}.fa-shield:before{content:"\f132"}.fa-calendar-o:before{content:"\f133"}.fa-fire-extinguisher:before{content:"\f134"}.fa-rocket:before{content:"\f135"}.fa-maxcdn:before{content:"\f136"}.fa-chevron-circle-left:before{content:"\f137"}.fa-chevron-circle-right:before{content:"\f138"}.fa-chevron-circle-up:before{content:"\f139"}.fa-chevron-circle-down:before{content:"\f13a"}.fa-html5:before{content:"\f13b"}.fa-css3:before{content:"\f13c"}.fa-anchor:before{content:"\f13d"}.fa-unlock-alt:before{content:"\f13e"}.fa-bullseye:before{content:"\f140"}.fa-ellipsis-h:before{content:"\f141"}.fa-ellipsis-v:before{content:"\f142"}.fa-rss-square:before{content:"\f143"}.fa-play-circle:before{content:"\f144"}.fa-ticket:before{content:"\f145"}.fa-minus-square:before{content:"\f146"}.fa-minus-square-o:before{content:"\f147"}.fa-level-up:before{content:"\f148"}.fa-level-down:before{content:"\f149"}.fa-check-square:before{content:"\f14a"}.fa-pencil-square:before{content:"\f14b"}.fa-external-link-square:before{content:"\f14c"}.fa-share-square:before{content:"\f14d"}.fa-compass:before{content:"\f14e"}.fa-toggle-down:before,.fa-caret-square-o-down:before{content:"\f150"}.fa-toggle-up:before,.fa-caret-square-o-up:before{content:"\f151"}.fa-toggle-right:before,.fa-caret-square-o-right:before{content:"\f152"}.fa-euro:before,.fa-eur:before{content:"\f153"}.fa-gbp:before{content:"\f154"}.fa-dollar:before,.fa-usd:before{content:"\f155"}.fa-rupee:before,.fa-inr:before{content:"\f156"}.fa-cny:before,.fa-rmb:before,.fa-yen:before,.fa-jpy:before{content:"\f157"}.fa-ruble:before,.fa-rouble:before,.fa-rub:before{content:"\f158"}.fa-won:before,.fa-krw:before{content:"\f159"}.fa-bitcoin:before,.fa-btc:before{content:"\f15a"}.fa-file:before{content:"\f15b"}.fa-file-text:before{content:"\f15c"}.fa-sort-alpha-asc:before{content:"\f15d"}.fa-sort-alpha-desc:before{content:"\f15e"}.fa-sort-amount-asc:before{content:"\f160"}.fa-sort-amount-desc:before{content:"\f161"}.fa-sort-numeric-asc:before{content:"\f162"}.fa-sort-numeric-desc:before{content:"\f163"}.fa-thumbs-up:before{content:"\f164"}.fa-thumbs-down:before{content:"\f165"}.fa-youtube-square:before{content:"\f166"}.fa-youtube:before{content:"\f167"}.fa-xing:before{content:"\f168"}.fa-xing-square:before{content:"\f169"}.fa-youtube-play:before{content:"\f16a"}.fa-dropbox:before{content:"\f16b"}.fa-stack-overflow:before{content:"\f16c"}.fa-instagram:before{content:"\f16d"}.fa-flickr:before{content:"\f16e"}.fa-adn:before{content:"\f170"}.fa-bitbucket:before{content:"\f171"}.fa-bitbucket-square:before{content:"\f172"}.fa-tumblr:before{content:"\f173"}.fa-tumblr-square:before{content:"\f174"}.fa-long-arrow-down:before{content:"\f175"}.fa-long-arrow-up:before{content:"\f176"}.fa-long-arrow-left:before{content:"\f177"}.fa-long-arrow-right:before{content:"\f178"}.fa-apple:before{content:"\f179"}.fa-windows:before{content:"\f17a"}.fa-android:before{content:"\f17b"}.fa-linux:before{content:"\f17c"}.fa-dribbble:before{content:"\f17d"}.fa-skype:before{content:"\f17e"}.fa-foursquare:before{content:"\f180"}.fa-trello:before{content:"\f181"}.fa-female:before{content:"\f182"}.fa-male:before{content:"\f183"}.fa-gittip:before{content:"\f184"}.fa-sun-o:before{content:"\f185"}.fa-moon-o:before{content:"\f186"}.fa-archive:before{content:"\f187"}.fa-bug:before{content:"\f188"}.fa-vk:before{content:"\f189"}.fa-weibo:before{content:"\f18a"}.fa-renren:before{content:"\f18b"}.fa-pagelines:before{content:"\f18c"}.fa-stack-exchange:before{content:"\f18d"}.fa-arrow-circle-o-right:before{content:"\f18e"}.fa-arrow-circle-o-left:before{content:"\f190"}.fa-toggle-left:before,.fa-caret-square-o-left:before{content:"\f191"}.fa-dot-circle-o:before{content:"\f192"}.fa-wheelchair:before{content:"\f193"}.fa-vimeo-square:before{content:"\f194"}.fa-turkish-lira:before,.fa-try:before{content:"\f195"}.fa-plus-square-o:before{content:"\f196"} \ No newline at end of file
diff --git a/doc/website-v1/development.adoc b/doc/website-v1/development.adoc
new file mode 100644
index 0000000..a334a6b
--- /dev/null
+++ b/doc/website-v1/development.adoc
@@ -0,0 +1,74 @@
+= Development =
+
+== Tools ==
+
+++++
+<ul class="nav">
+<li><a href="https://github.com/ClusterLabs/crmsh"><i class="fa fa-code-fork fa-3x fa-fw"></i> Source Repository</a></li>
+<li><a href="http://clusterlabs.org/mailman/listinfo/users"><i class="fa fa-envelope fa-3x fa-fw"></i> Mailing List</a></li>
+<li><a href="https://github.com/ClusterLabs/crmsh/issues"><i class="fa fa-bug fa-3x fa-fw"></i> Issue Tracker</a></li>
+<li><a href="irc://freenode.net/#clusterlabs"><i class="fa fa-comments fa-3x fa-fw"></i> IRC: #clusterlabs on Freenode</a></li>
+<li><a href="https://github.com/ClusterLabs/crmsh/commits/master.atom"><i class="fa fa-rss fa-3x fa-fw"></i> Atom feed</a></li>
+</ul>
+++++
+
+== Source Code ==
+
+The source code for `crmsh` is kept in a
+http://git-scm.com/[git] repository
+hosted at https://github.com[github]. Use +git+ to get a working copy:
+
+----
+git clone https://github.com/ClusterLabs/crmsh.git
+----
+
+Dependencies
+~~~~~~~~~~~~
+
+Building and installing crmsh requires Python version 2.6 and up (but not 3, yet).
+
+Additionally, the following Python modules are needed:
+
+* `lxml`
+* `PyYAML`
+* `setuptools`
+* `parallax`
+* `python-dateutil`
+
+Building
+~~~~~~~~
+
+`crmsh` uses the autotools suite to manage the build process.
+
+----
+./autogen.sh
+./configure
+make
+make install
+----
+
+=== Tests ===
+
+The unit tests for `crmsh` require +nose+ to run. On most distributions, this can be installed
+by installing the package +python-nose+, or using +pip+.
+
+To run the unit test suite, go to the source code directory of `crmsh`
+and call:
+
+----
+./test/run
+----
+
+`crmsh` also comes with a comprehensive regression test suite. The regression tests need
+to run after installation, on a system which has both crmsh and pacemaker installed. You
+will also need to install +pacemaker+ and the +cluster-glue+ development headers.
+
+* link:http://hg.linux-ha.org/glue[cluster-glue]
+* link:https://github.com/ClusterLabs/pacemaker[pacemaker]
+
+To execute the tests, call:
+
+----
+/usr/share/crmsh/tests/regression.sh
+cat crmtestout/regression.out
+----
diff --git a/doc/website-v1/documentation.adoc b/doc/website-v1/documentation.adoc
new file mode 100644
index 0000000..dce5a0d
--- /dev/null
+++ b/doc/website-v1/documentation.adoc
@@ -0,0 +1,42 @@
+= Documentation =
+
+The main documentation for `crmsh` comes in the form of the
+`manual`, which is the same help as found using the `help`
+command in the interactive shell.
+
+Additionally, there are a couple of guides and other documents
+that will hopefully make using the shell as easy as possible.
+
+== Manual ==
+
+* link:/man[Manual (Development)]
+* link:/man-4.3[Manual (v4.3.x)]
+* link:/man-3[Manual (v3.x)]
+* link:/man-2.0[Manual (v2.x)]
+* link:/man-1.2[Manual (v1.2.x)]
+
+== Guides ==
+
+* link:/start-guide[Getting Started]
+* link:/history-guide[History Guide]
+* link:/rsctest-guide[Resource Testing Guide]
+* link:/configuration[Configuration]
+* link:/scripts[Cluster scripts]
+* link:/faq[Frequently Asked Questions]
+
+== Translations ==
+
+* https://blog.3ware.co.jp/2015/05/crmsh-getting-started/[Getting Started (Japanese)]
+
+== External documentation ==
+
+The SUSE
+https://www.suse.com/documentation/sle_ha/book_sleha/?page=/documentation/sle_ha/book_sleha/data/book_sleha.html[High
+Availability Guide] provides a guide to
+installing and configuring a complete cluster solution including both
+the `crm` shell and Hawk, the web GUI which uses the `crm` shell as
+its backend.
+
+For more information on Pacemaker in general, see the
+http://clusterlabs.org/doc/[Pacemaker documentation] at `clusterlabs.org`.
+
diff --git a/doc/website-v1/download.adoc b/doc/website-v1/download.adoc
new file mode 100644
index 0000000..8d81153
--- /dev/null
+++ b/doc/website-v1/download.adoc
@@ -0,0 +1,40 @@
+= Download =
+
+The easiest way to install `crmsh` is via the package manager of your distribution.
+
+== SLES / openSUSE ==
+
+`crmsh` is commercially supported on SLE via the https://www.suse.com/products/highavailability/[SUSE Linux Enterprise High Availability Extension]. It is also available for openSUSE with the package name `crmsh`. Development packages can be downloaded from the OBS:
+
+* https://build.opensuse.org/package/show/network:ha-clustering:Stable/crmsh[Stable version]
+* https://build.opensuse.org/package/show/network:ha-clustering:Factory/crmsh[Development version]
+
+== Red Hat / CentOS / Fedora ==
+
+We try to build Red Hat / CentOS / Fedora-compatible RPM packages on the OBS (see above).
+
+=== CentOS 7 ===
+
+---
+dnf config-manager --add-repo http://download.opensuse.org/repositories/network:/ha-clustering:/Stable/CentOS_CentOS-7/network:ha-clustering:Stable.repo
+---
+
+== Debian ==
+
+The versions of `crmsh` and `pacemaker` currently available in the latest Debian release are quite old. Newer packages are available via the Debian-HA team https://wiki.debian.org/Debian-HA[wiki] and the distribution packages will hopefully be updated soon.
+
+== Ubuntu ==
+
+Packages for `crmsh` are available from the https://launchpad.net/ubuntu/+source/crmsh[Launchpad].
+
+== Gentoo ==
+
+A fairly up-to-date version is available https://packages.gentoo.org/packages/sys-cluster/crmsh[here].
+
+== Arch ==
+
+`crmsh` is available via the https://aur.archlinux.org/packages/ha-pacemaker-crmsh/[AUR]. Unfortunately the package seems somewhat out of date.
+
+== Source Packages ==
+
+Releases are available as `.tar.gz` or `.zip` archives via https://github.com/ClusterLabs/crmsh/releases[Github].
diff --git a/doc/website-v1/faq.adoc b/doc/website-v1/faq.adoc
new file mode 100644
index 0000000..c9c5d02
--- /dev/null
+++ b/doc/website-v1/faq.adoc
@@ -0,0 +1,60 @@
+= Frequently Asked Questions
+
+== What is the crm shell?
+
+The `crm` shell is a command-line interface to the Pacemaker cluster
+resource management stack. If that doesn't make any sense to you, the
+easiest way to get up to speed is to go to the
+http://clusterlabs.org/[Pacemaker] website and read more about what it
+does there.
+
+The `crm` shell provides a simpler interface to configuring Pacemaker
+than manipulating the XML of the CIB (Cluster Information Base)
+directly. With its command-line style interface, changes to the
+cluster can be performed quickly and painlessly. It also works as a
+scripting tool, allowing more complicated changes to be applied to the
+cluster.
+
+The `crm` shell also functions as a management console, providing a
+unified interface to a number of other auxilliary tools related to
+Pacemaker and cluster management.
+
+== What distributions does the shell run on?
+
+Many distributions provide packages for the `crm` shell in their
+respective package repositories. The best and safest way to obtain the
+`crm` shell is via the distribution repositories, so look there first.
+
+The intention is for the `crm` shell to work well on all the major
+distributions. Pre-built packages are provided for the
+following distros:
+
+ * openSUSE
+ * Fedora
+ * CentOS
+ * Red Hat Linux
+
+More information can be found on the
+link:/documentation#_installation[Documentation] page.
+
+== Didn't crm use to be part of Pacemaker?
+
+Yes, initially, the `crm` shell was distributed as part of the
+Pacemaker project. It was split into its own, separate project in
+2011.
+
+A common misconception is that `crm` has been replaced by `pcs`
+(available at https://github.com/feist/pcs[github.com/feist/pcs]). `pcs`
+is an alternative command line interface similar to `crm`. Both
+projects are being actively developed, with slightly different
+goals. Our recommendation is to use whatever shell your distribution
+of choice comes with and supports, unless you have a particular
+preference or are on a distribution which doesn't bundle either. In
+that case, we are obviously biased towards one of the available
+choices. ;)
+
+== Command-line is well and good, but is there a web interface?
+
+Yes! Take a look at https://github.com/ClusterLabs/hawk[Hawk].
+
+Hawk uses the `crm` shell as its backend to interact with the cluster.
diff --git a/doc/website-v1/fonts/FontAwesome.otf b/doc/website-v1/fonts/FontAwesome.otf
new file mode 100644
index 0000000..8b0f54e
--- /dev/null
+++ b/doc/website-v1/fonts/FontAwesome.otf
Binary files differ
diff --git a/doc/website-v1/fonts/fontawesome-webfont.eot b/doc/website-v1/fonts/fontawesome-webfont.eot
new file mode 100755
index 0000000..7c79c6a
--- /dev/null
+++ b/doc/website-v1/fonts/fontawesome-webfont.eot
Binary files differ
diff --git a/doc/website-v1/fonts/fontawesome-webfont.svg b/doc/website-v1/fonts/fontawesome-webfont.svg
new file mode 100755
index 0000000..45fdf33
--- /dev/null
+++ b/doc/website-v1/fonts/fontawesome-webfont.svg
@@ -0,0 +1,414 @@
+<?xml version="1.0" standalone="no"?>
+<!DOCTYPE svg PUBLIC "-//W3C//DTD SVG 1.1//EN" "http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd" >
+<svg xmlns="http://www.w3.org/2000/svg">
+<metadata></metadata>
+<defs>
+<font id="fontawesomeregular" horiz-adv-x="1536" >
+<font-face units-per-em="1792" ascent="1536" descent="-256" />
+<missing-glyph horiz-adv-x="448" />
+<glyph unicode=" " horiz-adv-x="448" />
+<glyph unicode="&#x09;" horiz-adv-x="448" />
+<glyph unicode="&#xa0;" horiz-adv-x="448" />
+<glyph unicode="&#xa8;" horiz-adv-x="1792" />
+<glyph unicode="&#xa9;" horiz-adv-x="1792" />
+<glyph unicode="&#xae;" horiz-adv-x="1792" />
+<glyph unicode="&#xb4;" horiz-adv-x="1792" />
+<glyph unicode="&#xc6;" horiz-adv-x="1792" />
+<glyph unicode="&#x2000;" horiz-adv-x="768" />
+<glyph unicode="&#x2001;" />
+<glyph unicode="&#x2002;" horiz-adv-x="768" />
+<glyph unicode="&#x2003;" />
+<glyph unicode="&#x2004;" horiz-adv-x="512" />
+<glyph unicode="&#x2005;" horiz-adv-x="384" />
+<glyph unicode="&#x2006;" horiz-adv-x="256" />
+<glyph unicode="&#x2007;" horiz-adv-x="256" />
+<glyph unicode="&#x2008;" horiz-adv-x="192" />
+<glyph unicode="&#x2009;" horiz-adv-x="307" />
+<glyph unicode="&#x200a;" horiz-adv-x="85" />
+<glyph unicode="&#x202f;" horiz-adv-x="307" />
+<glyph unicode="&#x205f;" horiz-adv-x="384" />
+<glyph unicode="&#x2122;" horiz-adv-x="1792" />
+<glyph unicode="&#x221e;" horiz-adv-x="1792" />
+<glyph unicode="&#x2260;" horiz-adv-x="1792" />
+<glyph unicode="&#xe000;" horiz-adv-x="500" d="M0 0z" />
+<glyph unicode="&#xf000;" horiz-adv-x="1792" d="M1699 1350q0 -35 -43 -78l-632 -632v-768h320q26 0 45 -19t19 -45t-19 -45t-45 -19h-896q-26 0 -45 19t-19 45t19 45t45 19h320v768l-632 632q-43 43 -43 78q0 23 18 36.5t38 17.5t43 4h1408q23 0 43 -4t38 -17.5t18 -36.5z" />
+<glyph unicode="&#xf001;" d="M1536 1312v-1120q0 -50 -34 -89t-86 -60.5t-103.5 -32t-96.5 -10.5t-96.5 10.5t-103.5 32t-86 60.5t-34 89t34 89t86 60.5t103.5 32t96.5 10.5q105 0 192 -39v537l-768 -237v-709q0 -50 -34 -89t-86 -60.5t-103.5 -32t-96.5 -10.5t-96.5 10.5t-103.5 32t-86 60.5t-34 89 t34 89t86 60.5t103.5 32t96.5 10.5q105 0 192 -39v967q0 31 19 56.5t49 35.5l832 256q12 4 28 4q40 0 68 -28t28 -68z" />
+<glyph unicode="&#xf002;" horiz-adv-x="1664" d="M1152 704q0 185 -131.5 316.5t-316.5 131.5t-316.5 -131.5t-131.5 -316.5t131.5 -316.5t316.5 -131.5t316.5 131.5t131.5 316.5zM1664 -128q0 -52 -38 -90t-90 -38q-54 0 -90 38l-343 342q-179 -124 -399 -124q-143 0 -273.5 55.5t-225 150t-150 225t-55.5 273.5 t55.5 273.5t150 225t225 150t273.5 55.5t273.5 -55.5t225 -150t150 -225t55.5 -273.5q0 -220 -124 -399l343 -343q37 -37 37 -90z" />
+<glyph unicode="&#xf003;" horiz-adv-x="1792" d="M1664 32v768q-32 -36 -69 -66q-268 -206 -426 -338q-51 -43 -83 -67t-86.5 -48.5t-102.5 -24.5h-1h-1q-48 0 -102.5 24.5t-86.5 48.5t-83 67q-158 132 -426 338q-37 30 -69 66v-768q0 -13 9.5 -22.5t22.5 -9.5h1472q13 0 22.5 9.5t9.5 22.5zM1664 1083v11v13.5t-0.5 13 t-3 12.5t-5.5 9t-9 7.5t-14 2.5h-1472q-13 0 -22.5 -9.5t-9.5 -22.5q0 -168 147 -284q193 -152 401 -317q6 -5 35 -29.5t46 -37.5t44.5 -31.5t50.5 -27.5t43 -9h1h1q20 0 43 9t50.5 27.5t44.5 31.5t46 37.5t35 29.5q208 165 401 317q54 43 100.5 115.5t46.5 131.5z M1792 1120v-1088q0 -66 -47 -113t-113 -47h-1472q-66 0 -113 47t-47 113v1088q0 66 47 113t113 47h1472q66 0 113 -47t47 -113z" />
+<glyph unicode="&#xf004;" horiz-adv-x="1792" d="M896 -128q-26 0 -44 18l-624 602q-10 8 -27.5 26t-55.5 65.5t-68 97.5t-53.5 121t-23.5 138q0 220 127 344t351 124q62 0 126.5 -21.5t120 -58t95.5 -68.5t76 -68q36 36 76 68t95.5 68.5t120 58t126.5 21.5q224 0 351 -124t127 -344q0 -221 -229 -450l-623 -600 q-18 -18 -44 -18z" />
+<glyph unicode="&#xf005;" horiz-adv-x="1664" d="M1664 889q0 -22 -26 -48l-363 -354l86 -500q1 -7 1 -20q0 -21 -10.5 -35.5t-30.5 -14.5q-19 0 -40 12l-449 236l-449 -236q-22 -12 -40 -12q-21 0 -31.5 14.5t-10.5 35.5q0 6 2 20l86 500l-364 354q-25 27 -25 48q0 37 56 46l502 73l225 455q19 41 49 41t49 -41l225 -455 l502 -73q56 -9 56 -46z" />
+<glyph unicode="&#xf006;" horiz-adv-x="1664" d="M1137 532l306 297l-422 62l-189 382l-189 -382l-422 -62l306 -297l-73 -421l378 199l377 -199zM1664 889q0 -22 -26 -48l-363 -354l86 -500q1 -7 1 -20q0 -50 -41 -50q-19 0 -40 12l-449 236l-449 -236q-22 -12 -40 -12q-21 0 -31.5 14.5t-10.5 35.5q0 6 2 20l86 500 l-364 354q-25 27 -25 48q0 37 56 46l502 73l225 455q19 41 49 41t49 -41l225 -455l502 -73q56 -9 56 -46z" />
+<glyph unicode="&#xf007;" horiz-adv-x="1408" d="M1408 131q0 -120 -73 -189.5t-194 -69.5h-874q-121 0 -194 69.5t-73 189.5q0 53 3.5 103.5t14 109t26.5 108.5t43 97.5t62 81t85.5 53.5t111.5 20q9 0 42 -21.5t74.5 -48t108 -48t133.5 -21.5t133.5 21.5t108 48t74.5 48t42 21.5q61 0 111.5 -20t85.5 -53.5t62 -81 t43 -97.5t26.5 -108.5t14 -109t3.5 -103.5zM1088 1024q0 -159 -112.5 -271.5t-271.5 -112.5t-271.5 112.5t-112.5 271.5t112.5 271.5t271.5 112.5t271.5 -112.5t112.5 -271.5z" />
+<glyph unicode="&#xf008;" horiz-adv-x="1920" d="M384 -64v128q0 26 -19 45t-45 19h-128q-26 0 -45 -19t-19 -45v-128q0 -26 19 -45t45 -19h128q26 0 45 19t19 45zM384 320v128q0 26 -19 45t-45 19h-128q-26 0 -45 -19t-19 -45v-128q0 -26 19 -45t45 -19h128q26 0 45 19t19 45zM384 704v128q0 26 -19 45t-45 19h-128 q-26 0 -45 -19t-19 -45v-128q0 -26 19 -45t45 -19h128q26 0 45 19t19 45zM1408 -64v512q0 26 -19 45t-45 19h-768q-26 0 -45 -19t-19 -45v-512q0 -26 19 -45t45 -19h768q26 0 45 19t19 45zM384 1088v128q0 26 -19 45t-45 19h-128q-26 0 -45 -19t-19 -45v-128q0 -26 19 -45 t45 -19h128q26 0 45 19t19 45zM1792 -64v128q0 26 -19 45t-45 19h-128q-26 0 -45 -19t-19 -45v-128q0 -26 19 -45t45 -19h128q26 0 45 19t19 45zM1408 704v512q0 26 -19 45t-45 19h-768q-26 0 -45 -19t-19 -45v-512q0 -26 19 -45t45 -19h768q26 0 45 19t19 45zM1792 320v128 q0 26 -19 45t-45 19h-128q-26 0 -45 -19t-19 -45v-128q0 -26 19 -45t45 -19h128q26 0 45 19t19 45zM1792 704v128q0 26 -19 45t-45 19h-128q-26 0 -45 -19t-19 -45v-128q0 -26 19 -45t45 -19h128q26 0 45 19t19 45zM1792 1088v128q0 26 -19 45t-45 19h-128q-26 0 -45 -19 t-19 -45v-128q0 -26 19 -45t45 -19h128q26 0 45 19t19 45zM1920 1248v-1344q0 -66 -47 -113t-113 -47h-1600q-66 0 -113 47t-47 113v1344q0 66 47 113t113 47h1600q66 0 113 -47t47 -113z" />
+<glyph unicode="&#xf009;" horiz-adv-x="1664" d="M768 512v-384q0 -52 -38 -90t-90 -38h-512q-52 0 -90 38t-38 90v384q0 52 38 90t90 38h512q52 0 90 -38t38 -90zM768 1280v-384q0 -52 -38 -90t-90 -38h-512q-52 0 -90 38t-38 90v384q0 52 38 90t90 38h512q52 0 90 -38t38 -90zM1664 512v-384q0 -52 -38 -90t-90 -38 h-512q-52 0 -90 38t-38 90v384q0 52 38 90t90 38h512q52 0 90 -38t38 -90zM1664 1280v-384q0 -52 -38 -90t-90 -38h-512q-52 0 -90 38t-38 90v384q0 52 38 90t90 38h512q52 0 90 -38t38 -90z" />
+<glyph unicode="&#xf00a;" horiz-adv-x="1792" d="M512 288v-192q0 -40 -28 -68t-68 -28h-320q-40 0 -68 28t-28 68v192q0 40 28 68t68 28h320q40 0 68 -28t28 -68zM512 800v-192q0 -40 -28 -68t-68 -28h-320q-40 0 -68 28t-28 68v192q0 40 28 68t68 28h320q40 0 68 -28t28 -68zM1152 288v-192q0 -40 -28 -68t-68 -28h-320 q-40 0 -68 28t-28 68v192q0 40 28 68t68 28h320q40 0 68 -28t28 -68zM512 1312v-192q0 -40 -28 -68t-68 -28h-320q-40 0 -68 28t-28 68v192q0 40 28 68t68 28h320q40 0 68 -28t28 -68zM1152 800v-192q0 -40 -28 -68t-68 -28h-320q-40 0 -68 28t-28 68v192q0 40 28 68t68 28 h320q40 0 68 -28t28 -68zM1792 288v-192q0 -40 -28 -68t-68 -28h-320q-40 0 -68 28t-28 68v192q0 40 28 68t68 28h320q40 0 68 -28t28 -68zM1152 1312v-192q0 -40 -28 -68t-68 -28h-320q-40 0 -68 28t-28 68v192q0 40 28 68t68 28h320q40 0 68 -28t28 -68zM1792 800v-192 q0 -40 -28 -68t-68 -28h-320q-40 0 -68 28t-28 68v192q0 40 28 68t68 28h320q40 0 68 -28t28 -68zM1792 1312v-192q0 -40 -28 -68t-68 -28h-320q-40 0 -68 28t-28 68v192q0 40 28 68t68 28h320q40 0 68 -28t28 -68z" />
+<glyph unicode="&#xf00b;" horiz-adv-x="1792" d="M512 288v-192q0 -40 -28 -68t-68 -28h-320q-40 0 -68 28t-28 68v192q0 40 28 68t68 28h320q40 0 68 -28t28 -68zM512 800v-192q0 -40 -28 -68t-68 -28h-320q-40 0 -68 28t-28 68v192q0 40 28 68t68 28h320q40 0 68 -28t28 -68zM1792 288v-192q0 -40 -28 -68t-68 -28h-960 q-40 0 -68 28t-28 68v192q0 40 28 68t68 28h960q40 0 68 -28t28 -68zM512 1312v-192q0 -40 -28 -68t-68 -28h-320q-40 0 -68 28t-28 68v192q0 40 28 68t68 28h320q40 0 68 -28t28 -68zM1792 800v-192q0 -40 -28 -68t-68 -28h-960q-40 0 -68 28t-28 68v192q0 40 28 68t68 28 h960q40 0 68 -28t28 -68zM1792 1312v-192q0 -40 -28 -68t-68 -28h-960q-40 0 -68 28t-28 68v192q0 40 28 68t68 28h960q40 0 68 -28t28 -68z" />
+<glyph unicode="&#xf00c;" horiz-adv-x="1792" d="M1671 970q0 -40 -28 -68l-724 -724l-136 -136q-28 -28 -68 -28t-68 28l-136 136l-362 362q-28 28 -28 68t28 68l136 136q28 28 68 28t68 -28l294 -295l656 657q28 28 68 28t68 -28l136 -136q28 -28 28 -68z" />
+<glyph unicode="&#xf00d;" horiz-adv-x="1408" d="M1298 214q0 -40 -28 -68l-136 -136q-28 -28 -68 -28t-68 28l-294 294l-294 -294q-28 -28 -68 -28t-68 28l-136 136q-28 28 -28 68t28 68l294 294l-294 294q-28 28 -28 68t28 68l136 136q28 28 68 28t68 -28l294 -294l294 294q28 28 68 28t68 -28l136 -136q28 -28 28 -68 t-28 -68l-294 -294l294 -294q28 -28 28 -68z" />
+<glyph unicode="&#xf00e;" horiz-adv-x="1664" d="M1024 736v-64q0 -13 -9.5 -22.5t-22.5 -9.5h-224v-224q0 -13 -9.5 -22.5t-22.5 -9.5h-64q-13 0 -22.5 9.5t-9.5 22.5v224h-224q-13 0 -22.5 9.5t-9.5 22.5v64q0 13 9.5 22.5t22.5 9.5h224v224q0 13 9.5 22.5t22.5 9.5h64q13 0 22.5 -9.5t9.5 -22.5v-224h224 q13 0 22.5 -9.5t9.5 -22.5zM1152 704q0 185 -131.5 316.5t-316.5 131.5t-316.5 -131.5t-131.5 -316.5t131.5 -316.5t316.5 -131.5t316.5 131.5t131.5 316.5zM1664 -128q0 -53 -37.5 -90.5t-90.5 -37.5q-54 0 -90 38l-343 342q-179 -124 -399 -124q-143 0 -273.5 55.5 t-225 150t-150 225t-55.5 273.5t55.5 273.5t150 225t225 150t273.5 55.5t273.5 -55.5t225 -150t150 -225t55.5 -273.5q0 -220 -124 -399l343 -343q37 -37 37 -90z" />
+<glyph unicode="&#xf010;" horiz-adv-x="1664" d="M1024 736v-64q0 -13 -9.5 -22.5t-22.5 -9.5h-576q-13 0 -22.5 9.5t-9.5 22.5v64q0 13 9.5 22.5t22.5 9.5h576q13 0 22.5 -9.5t9.5 -22.5zM1152 704q0 185 -131.5 316.5t-316.5 131.5t-316.5 -131.5t-131.5 -316.5t131.5 -316.5t316.5 -131.5t316.5 131.5t131.5 316.5z M1664 -128q0 -53 -37.5 -90.5t-90.5 -37.5q-54 0 -90 38l-343 342q-179 -124 -399 -124q-143 0 -273.5 55.5t-225 150t-150 225t-55.5 273.5t55.5 273.5t150 225t225 150t273.5 55.5t273.5 -55.5t225 -150t150 -225t55.5 -273.5q0 -220 -124 -399l343 -343q37 -37 37 -90z " />
+<glyph unicode="&#xf011;" d="M1536 640q0 -156 -61 -298t-164 -245t-245 -164t-298 -61t-298 61t-245 164t-164 245t-61 298q0 182 80.5 343t226.5 270q43 32 95.5 25t83.5 -50q32 -42 24.5 -94.5t-49.5 -84.5q-98 -74 -151.5 -181t-53.5 -228q0 -104 40.5 -198.5t109.5 -163.5t163.5 -109.5 t198.5 -40.5t198.5 40.5t163.5 109.5t109.5 163.5t40.5 198.5q0 121 -53.5 228t-151.5 181q-42 32 -49.5 84.5t24.5 94.5q31 43 84 50t95 -25q146 -109 226.5 -270t80.5 -343zM896 1408v-640q0 -52 -38 -90t-90 -38t-90 38t-38 90v640q0 52 38 90t90 38t90 -38t38 -90z" />
+<glyph unicode="&#xf012;" horiz-adv-x="1792" d="M256 96v-192q0 -14 -9 -23t-23 -9h-192q-14 0 -23 9t-9 23v192q0 14 9 23t23 9h192q14 0 23 -9t9 -23zM640 224v-320q0 -14 -9 -23t-23 -9h-192q-14 0 -23 9t-9 23v320q0 14 9 23t23 9h192q14 0 23 -9t9 -23zM1024 480v-576q0 -14 -9 -23t-23 -9h-192q-14 0 -23 9t-9 23 v576q0 14 9 23t23 9h192q14 0 23 -9t9 -23zM1408 864v-960q0 -14 -9 -23t-23 -9h-192q-14 0 -23 9t-9 23v960q0 14 9 23t23 9h192q14 0 23 -9t9 -23zM1792 1376v-1472q0 -14 -9 -23t-23 -9h-192q-14 0 -23 9t-9 23v1472q0 14 9 23t23 9h192q14 0 23 -9t9 -23z" />
+<glyph unicode="&#xf013;" d="M1024 640q0 106 -75 181t-181 75t-181 -75t-75 -181t75 -181t181 -75t181 75t75 181zM1536 749v-222q0 -12 -8 -23t-20 -13l-185 -28q-19 -54 -39 -91q35 -50 107 -138q10 -12 10 -25t-9 -23q-27 -37 -99 -108t-94 -71q-12 0 -26 9l-138 108q-44 -23 -91 -38 q-16 -136 -29 -186q-7 -28 -36 -28h-222q-14 0 -24.5 8.5t-11.5 21.5l-28 184q-49 16 -90 37l-141 -107q-10 -9 -25 -9q-14 0 -25 11q-126 114 -165 168q-7 10 -7 23q0 12 8 23q15 21 51 66.5t54 70.5q-27 50 -41 99l-183 27q-13 2 -21 12.5t-8 23.5v222q0 12 8 23t19 13 l186 28q14 46 39 92q-40 57 -107 138q-10 12 -10 24q0 10 9 23q26 36 98.5 107.5t94.5 71.5q13 0 26 -10l138 -107q44 23 91 38q16 136 29 186q7 28 36 28h222q14 0 24.5 -8.5t11.5 -21.5l28 -184q49 -16 90 -37l142 107q9 9 24 9q13 0 25 -10q129 -119 165 -170q7 -8 7 -22 q0 -12 -8 -23q-15 -21 -51 -66.5t-54 -70.5q26 -50 41 -98l183 -28q13 -2 21 -12.5t8 -23.5z" />
+<glyph unicode="&#xf014;" horiz-adv-x="1408" d="M512 800v-576q0 -14 -9 -23t-23 -9h-64q-14 0 -23 9t-9 23v576q0 14 9 23t23 9h64q14 0 23 -9t9 -23zM768 800v-576q0 -14 -9 -23t-23 -9h-64q-14 0 -23 9t-9 23v576q0 14 9 23t23 9h64q14 0 23 -9t9 -23zM1024 800v-576q0 -14 -9 -23t-23 -9h-64q-14 0 -23 9t-9 23v576 q0 14 9 23t23 9h64q14 0 23 -9t9 -23zM1152 76v948h-896v-948q0 -22 7 -40.5t14.5 -27t10.5 -8.5h832q3 0 10.5 8.5t14.5 27t7 40.5zM480 1152h448l-48 117q-7 9 -17 11h-317q-10 -2 -17 -11zM1408 1120v-64q0 -14 -9 -23t-23 -9h-96v-948q0 -83 -47 -143.5t-113 -60.5h-832 q-66 0 -113 58.5t-47 141.5v952h-96q-14 0 -23 9t-9 23v64q0 14 9 23t23 9h309l70 167q15 37 54 63t79 26h320q40 0 79 -26t54 -63l70 -167h309q14 0 23 -9t9 -23z" />
+<glyph unicode="&#xf015;" horiz-adv-x="1664" d="M1408 544v-480q0 -26 -19 -45t-45 -19h-384v384h-256v-384h-384q-26 0 -45 19t-19 45v480q0 1 0.5 3t0.5 3l575 474l575 -474q1 -2 1 -6zM1631 613l-62 -74q-8 -9 -21 -11h-3q-13 0 -21 7l-692 577l-692 -577q-12 -8 -24 -7q-13 2 -21 11l-62 74q-8 10 -7 23.5t11 21.5 l719 599q32 26 76 26t76 -26l244 -204v195q0 14 9 23t23 9h192q14 0 23 -9t9 -23v-408l219 -182q10 -8 11 -21.5t-7 -23.5z" />
+<glyph unicode="&#xf016;" horiz-adv-x="1280" d="M128 0h1024v768h-416q-40 0 -68 28t-28 68v416h-512v-1280zM768 896h376q-10 29 -22 41l-313 313q-12 12 -41 22v-376zM1280 864v-896q0 -40 -28 -68t-68 -28h-1088q-40 0 -68 28t-28 68v1344q0 40 28 68t68 28h640q40 0 88 -20t76 -48l312 -312q28 -28 48 -76t20 -88z " />
+<glyph unicode="&#xf017;" d="M896 992v-448q0 -14 -9 -23t-23 -9h-320q-14 0 -23 9t-9 23v64q0 14 9 23t23 9h224v352q0 14 9 23t23 9h64q14 0 23 -9t9 -23zM1312 640q0 148 -73 273t-198 198t-273 73t-273 -73t-198 -198t-73 -273t73 -273t198 -198t273 -73t273 73t198 198t73 273zM1536 640 q0 -209 -103 -385.5t-279.5 -279.5t-385.5 -103t-385.5 103t-279.5 279.5t-103 385.5t103 385.5t279.5 279.5t385.5 103t385.5 -103t279.5 -279.5t103 -385.5z" />
+<glyph unicode="&#xf018;" horiz-adv-x="1920" d="M1111 540v4l-24 320q-1 13 -11 22.5t-23 9.5h-186q-13 0 -23 -9.5t-11 -22.5l-24 -320v-4q-1 -12 8 -20t21 -8h244q12 0 21 8t8 20zM1870 73q0 -73 -46 -73h-704q13 0 22 9.5t8 22.5l-20 256q-1 13 -11 22.5t-23 9.5h-272q-13 0 -23 -9.5t-11 -22.5l-20 -256 q-1 -13 8 -22.5t22 -9.5h-704q-46 0 -46 73q0 54 26 116l417 1044q8 19 26 33t38 14h339q-13 0 -23 -9.5t-11 -22.5l-15 -192q-1 -14 8 -23t22 -9h166q13 0 22 9t8 23l-15 192q-1 13 -11 22.5t-23 9.5h339q20 0 38 -14t26 -33l417 -1044q26 -62 26 -116z" />
+<glyph unicode="&#xf019;" horiz-adv-x="1664" d="M1280 192q0 26 -19 45t-45 19t-45 -19t-19 -45t19 -45t45 -19t45 19t19 45zM1536 192q0 26 -19 45t-45 19t-45 -19t-19 -45t19 -45t45 -19t45 19t19 45zM1664 416v-320q0 -40 -28 -68t-68 -28h-1472q-40 0 -68 28t-28 68v320q0 40 28 68t68 28h465l135 -136 q58 -56 136 -56t136 56l136 136h464q40 0 68 -28t28 -68zM1339 985q17 -41 -14 -70l-448 -448q-18 -19 -45 -19t-45 19l-448 448q-31 29 -14 70q17 39 59 39h256v448q0 26 19 45t45 19h256q26 0 45 -19t19 -45v-448h256q42 0 59 -39z" />
+<glyph unicode="&#xf01a;" d="M1120 608q0 -12 -10 -24l-319 -319q-11 -9 -23 -9t-23 9l-320 320q-15 16 -7 35q8 20 30 20h192v352q0 14 9 23t23 9h192q14 0 23 -9t9 -23v-352h192q14 0 23 -9t9 -23zM768 1184q-148 0 -273 -73t-198 -198t-73 -273t73 -273t198 -198t273 -73t273 73t198 198t73 273 t-73 273t-198 198t-273 73zM1536 640q0 -209 -103 -385.5t-279.5 -279.5t-385.5 -103t-385.5 103t-279.5 279.5t-103 385.5t103 385.5t279.5 279.5t385.5 103t385.5 -103t279.5 -279.5t103 -385.5z" />
+<glyph unicode="&#xf01b;" d="M1118 660q-8 -20 -30 -20h-192v-352q0 -14 -9 -23t-23 -9h-192q-14 0 -23 9t-9 23v352h-192q-14 0 -23 9t-9 23q0 12 10 24l319 319q11 9 23 9t23 -9l320 -320q15 -16 7 -35zM768 1184q-148 0 -273 -73t-198 -198t-73 -273t73 -273t198 -198t273 -73t273 73t198 198 t73 273t-73 273t-198 198t-273 73zM1536 640q0 -209 -103 -385.5t-279.5 -279.5t-385.5 -103t-385.5 103t-279.5 279.5t-103 385.5t103 385.5t279.5 279.5t385.5 103t385.5 -103t279.5 -279.5t103 -385.5z" />
+<glyph unicode="&#xf01c;" d="M1023 576h316q-1 3 -2.5 8t-2.5 8l-212 496h-708l-212 -496q-1 -2 -2.5 -8t-2.5 -8h316l95 -192h320zM1536 546v-482q0 -26 -19 -45t-45 -19h-1408q-26 0 -45 19t-19 45v482q0 62 25 123l238 552q10 25 36.5 42t52.5 17h832q26 0 52.5 -17t36.5 -42l238 -552 q25 -61 25 -123z" />
+<glyph unicode="&#xf01d;" d="M1184 640q0 -37 -32 -55l-544 -320q-15 -9 -32 -9q-16 0 -32 8q-32 19 -32 56v640q0 37 32 56q33 18 64 -1l544 -320q32 -18 32 -55zM1312 640q0 148 -73 273t-198 198t-273 73t-273 -73t-198 -198t-73 -273t73 -273t198 -198t273 -73t273 73t198 198t73 273zM1536 640 q0 -209 -103 -385.5t-279.5 -279.5t-385.5 -103t-385.5 103t-279.5 279.5t-103 385.5t103 385.5t279.5 279.5t385.5 103t385.5 -103t279.5 -279.5t103 -385.5z" />
+<glyph unicode="&#xf01e;" d="M1536 1280v-448q0 -26 -19 -45t-45 -19h-448q-42 0 -59 40q-17 39 14 69l138 138q-148 137 -349 137q-104 0 -198.5 -40.5t-163.5 -109.5t-109.5 -163.5t-40.5 -198.5t40.5 -198.5t109.5 -163.5t163.5 -109.5t198.5 -40.5q119 0 225 52t179 147q7 10 23 12q14 0 25 -9 l137 -138q9 -8 9.5 -20.5t-7.5 -22.5q-109 -132 -264 -204.5t-327 -72.5q-156 0 -298 61t-245 164t-164 245t-61 298t61 298t164 245t245 164t298 61q147 0 284.5 -55.5t244.5 -156.5l130 129q29 31 70 14q39 -17 39 -59z" />
+<glyph unicode="&#xf021;" d="M1511 480q0 -5 -1 -7q-64 -268 -268 -434.5t-478 -166.5q-146 0 -282.5 55t-243.5 157l-129 -129q-19 -19 -45 -19t-45 19t-19 45v448q0 26 19 45t45 19h448q26 0 45 -19t19 -45t-19 -45l-137 -137q71 -66 161 -102t187 -36q134 0 250 65t186 179q11 17 53 117 q8 23 30 23h192q13 0 22.5 -9.5t9.5 -22.5zM1536 1280v-448q0 -26 -19 -45t-45 -19h-448q-26 0 -45 19t-19 45t19 45l138 138q-148 137 -349 137q-134 0 -250 -65t-186 -179q-11 -17 -53 -117q-8 -23 -30 -23h-199q-13 0 -22.5 9.5t-9.5 22.5v7q65 268 270 434.5t480 166.5 q146 0 284 -55.5t245 -156.5l130 129q19 19 45 19t45 -19t19 -45z" />
+<glyph unicode="&#xf022;" horiz-adv-x="1792" d="M384 352v-64q0 -13 -9.5 -22.5t-22.5 -9.5h-64q-13 0 -22.5 9.5t-9.5 22.5v64q0 13 9.5 22.5t22.5 9.5h64q13 0 22.5 -9.5t9.5 -22.5zM384 608v-64q0 -13 -9.5 -22.5t-22.5 -9.5h-64q-13 0 -22.5 9.5t-9.5 22.5v64q0 13 9.5 22.5t22.5 9.5h64q13 0 22.5 -9.5t9.5 -22.5z M384 864v-64q0 -13 -9.5 -22.5t-22.5 -9.5h-64q-13 0 -22.5 9.5t-9.5 22.5v64q0 13 9.5 22.5t22.5 9.5h64q13 0 22.5 -9.5t9.5 -22.5zM1536 352v-64q0 -13 -9.5 -22.5t-22.5 -9.5h-960q-13 0 -22.5 9.5t-9.5 22.5v64q0 13 9.5 22.5t22.5 9.5h960q13 0 22.5 -9.5t9.5 -22.5z M1536 608v-64q0 -13 -9.5 -22.5t-22.5 -9.5h-960q-13 0 -22.5 9.5t-9.5 22.5v64q0 13 9.5 22.5t22.5 9.5h960q13 0 22.5 -9.5t9.5 -22.5zM1536 864v-64q0 -13 -9.5 -22.5t-22.5 -9.5h-960q-13 0 -22.5 9.5t-9.5 22.5v64q0 13 9.5 22.5t22.5 9.5h960q13 0 22.5 -9.5 t9.5 -22.5zM1664 160v832q0 13 -9.5 22.5t-22.5 9.5h-1472q-13 0 -22.5 -9.5t-9.5 -22.5v-832q0 -13 9.5 -22.5t22.5 -9.5h1472q13 0 22.5 9.5t9.5 22.5zM1792 1248v-1088q0 -66 -47 -113t-113 -47h-1472q-66 0 -113 47t-47 113v1088q0 66 47 113t113 47h1472q66 0 113 -47 t47 -113z" />
+<glyph unicode="&#xf023;" horiz-adv-x="1152" d="M320 768h512v192q0 106 -75 181t-181 75t-181 -75t-75 -181v-192zM1152 672v-576q0 -40 -28 -68t-68 -28h-960q-40 0 -68 28t-28 68v576q0 40 28 68t68 28h32v192q0 184 132 316t316 132t316 -132t132 -316v-192h32q40 0 68 -28t28 -68z" />
+<glyph unicode="&#xf024;" horiz-adv-x="1792" d="M320 1280q0 -72 -64 -110v-1266q0 -13 -9.5 -22.5t-22.5 -9.5h-64q-13 0 -22.5 9.5t-9.5 22.5v1266q-64 38 -64 110q0 53 37.5 90.5t90.5 37.5t90.5 -37.5t37.5 -90.5zM1792 1216v-763q0 -25 -12.5 -38.5t-39.5 -27.5q-215 -116 -369 -116q-61 0 -123.5 22t-108.5 48 t-115.5 48t-142.5 22q-192 0 -464 -146q-17 -9 -33 -9q-26 0 -45 19t-19 45v742q0 32 31 55q21 14 79 43q236 120 421 120q107 0 200 -29t219 -88q38 -19 88 -19q54 0 117.5 21t110 47t88 47t54.5 21q26 0 45 -19t19 -45z" />
+<glyph unicode="&#xf025;" horiz-adv-x="1664" d="M1664 650q0 -166 -60 -314l-20 -49l-185 -33q-22 -83 -90.5 -136.5t-156.5 -53.5v-32q0 -14 -9 -23t-23 -9h-64q-14 0 -23 9t-9 23v576q0 14 9 23t23 9h64q14 0 23 -9t9 -23v-32q71 0 130 -35.5t93 -95.5l68 12q29 95 29 193q0 148 -88 279t-236.5 209t-315.5 78 t-315.5 -78t-236.5 -209t-88 -279q0 -98 29 -193l68 -12q34 60 93 95.5t130 35.5v32q0 14 9 23t23 9h64q14 0 23 -9t9 -23v-576q0 -14 -9 -23t-23 -9h-64q-14 0 -23 9t-9 23v32q-88 0 -156.5 53.5t-90.5 136.5l-185 33l-20 49q-60 148 -60 314q0 151 67 291t179 242.5 t266 163.5t320 61t320 -61t266 -163.5t179 -242.5t67 -291z" />
+<glyph unicode="&#xf026;" horiz-adv-x="768" d="M768 1184v-1088q0 -26 -19 -45t-45 -19t-45 19l-333 333h-262q-26 0 -45 19t-19 45v384q0 26 19 45t45 19h262l333 333q19 19 45 19t45 -19t19 -45z" />
+<glyph unicode="&#xf027;" horiz-adv-x="1152" d="M768 1184v-1088q0 -26 -19 -45t-45 -19t-45 19l-333 333h-262q-26 0 -45 19t-19 45v384q0 26 19 45t45 19h262l333 333q19 19 45 19t45 -19t19 -45zM1152 640q0 -76 -42.5 -141.5t-112.5 -93.5q-10 -5 -25 -5q-26 0 -45 18.5t-19 45.5q0 21 12 35.5t29 25t34 23t29 35.5 t12 57t-12 57t-29 35.5t-34 23t-29 25t-12 35.5q0 27 19 45.5t45 18.5q15 0 25 -5q70 -27 112.5 -93t42.5 -142z" />
+<glyph unicode="&#xf028;" horiz-adv-x="1664" d="M768 1184v-1088q0 -26 -19 -45t-45 -19t-45 19l-333 333h-262q-26 0 -45 19t-19 45v384q0 26 19 45t45 19h262l333 333q19 19 45 19t45 -19t19 -45zM1152 640q0 -76 -42.5 -141.5t-112.5 -93.5q-10 -5 -25 -5q-26 0 -45 18.5t-19 45.5q0 21 12 35.5t29 25t34 23t29 35.5 t12 57t-12 57t-29 35.5t-34 23t-29 25t-12 35.5q0 27 19 45.5t45 18.5q15 0 25 -5q70 -27 112.5 -93t42.5 -142zM1408 640q0 -153 -85 -282.5t-225 -188.5q-13 -5 -25 -5q-27 0 -46 19t-19 45q0 39 39 59q56 29 76 44q74 54 115.5 135.5t41.5 173.5t-41.5 173.5 t-115.5 135.5q-20 15 -76 44q-39 20 -39 59q0 26 19 45t45 19q13 0 26 -5q140 -59 225 -188.5t85 -282.5zM1664 640q0 -230 -127 -422.5t-338 -283.5q-13 -5 -26 -5q-26 0 -45 19t-19 45q0 36 39 59q7 4 22.5 10.5t22.5 10.5q46 25 82 51q123 91 192 227t69 289t-69 289 t-192 227q-36 26 -82 51q-7 4 -22.5 10.5t-22.5 10.5q-39 23 -39 59q0 26 19 45t45 19q13 0 26 -5q211 -91 338 -283.5t127 -422.5z" />
+<glyph unicode="&#xf029;" horiz-adv-x="1408" d="M384 384v-128h-128v128h128zM384 1152v-128h-128v128h128zM1152 1152v-128h-128v128h128zM128 129h384v383h-384v-383zM128 896h384v384h-384v-384zM896 896h384v384h-384v-384zM640 640v-640h-640v640h640zM1152 128v-128h-128v128h128zM1408 128v-128h-128v128h128z M1408 640v-384h-384v128h-128v-384h-128v640h384v-128h128v128h128zM640 1408v-640h-640v640h640zM1408 1408v-640h-640v640h640z" />
+<glyph unicode="&#xf02a;" horiz-adv-x="1792" d="M63 0h-63v1408h63v-1408zM126 1h-32v1407h32v-1407zM220 1h-31v1407h31v-1407zM377 1h-31v1407h31v-1407zM534 1h-62v1407h62v-1407zM660 1h-31v1407h31v-1407zM723 1h-31v1407h31v-1407zM786 1h-31v1407h31v-1407zM943 1h-63v1407h63v-1407zM1100 1h-63v1407h63v-1407z M1226 1h-63v1407h63v-1407zM1352 1h-63v1407h63v-1407zM1446 1h-63v1407h63v-1407zM1635 1h-94v1407h94v-1407zM1698 1h-32v1407h32v-1407zM1792 0h-63v1408h63v-1408z" />
+<glyph unicode="&#xf02b;" d="M448 1088q0 53 -37.5 90.5t-90.5 37.5t-90.5 -37.5t-37.5 -90.5t37.5 -90.5t90.5 -37.5t90.5 37.5t37.5 90.5zM1515 512q0 -53 -37 -90l-491 -492q-39 -37 -91 -37q-53 0 -90 37l-715 716q-38 37 -64.5 101t-26.5 117v416q0 52 38 90t90 38h416q53 0 117 -26.5t102 -64.5 l715 -714q37 -39 37 -91z" />
+<glyph unicode="&#xf02c;" horiz-adv-x="1920" d="M448 1088q0 53 -37.5 90.5t-90.5 37.5t-90.5 -37.5t-37.5 -90.5t37.5 -90.5t90.5 -37.5t90.5 37.5t37.5 90.5zM1515 512q0 -53 -37 -90l-491 -492q-39 -37 -91 -37q-53 0 -90 37l-715 716q-38 37 -64.5 101t-26.5 117v416q0 52 38 90t90 38h416q53 0 117 -26.5t102 -64.5 l715 -714q37 -39 37 -91zM1899 512q0 -53 -37 -90l-491 -492q-39 -37 -91 -37q-36 0 -59 14t-53 45l470 470q37 37 37 90q0 52 -37 91l-715 714q-38 38 -102 64.5t-117 26.5h224q53 0 117 -26.5t102 -64.5l715 -714q37 -39 37 -91z" />
+<glyph unicode="&#xf02d;" horiz-adv-x="1664" d="M1639 1058q40 -57 18 -129l-275 -906q-19 -64 -76.5 -107.5t-122.5 -43.5h-923q-77 0 -148.5 53.5t-99.5 131.5q-24 67 -2 127q0 4 3 27t4 37q1 8 -3 21.5t-3 19.5q2 11 8 21t16.5 23.5t16.5 23.5q23 38 45 91.5t30 91.5q3 10 0.5 30t-0.5 28q3 11 17 28t17 23 q21 36 42 92t25 90q1 9 -2.5 32t0.5 28q4 13 22 30.5t22 22.5q19 26 42.5 84.5t27.5 96.5q1 8 -3 25.5t-2 26.5q2 8 9 18t18 23t17 21q8 12 16.5 30.5t15 35t16 36t19.5 32t26.5 23.5t36 11.5t47.5 -5.5l-1 -3q38 9 51 9h761q74 0 114 -56t18 -130l-274 -906 q-36 -119 -71.5 -153.5t-128.5 -34.5h-869q-27 0 -38 -15q-11 -16 -1 -43q24 -70 144 -70h923q29 0 56 15.5t35 41.5l300 987q7 22 5 57q38 -15 59 -43zM575 1056q-4 -13 2 -22.5t20 -9.5h608q13 0 25.5 9.5t16.5 22.5l21 64q4 13 -2 22.5t-20 9.5h-608q-13 0 -25.5 -9.5 t-16.5 -22.5zM492 800q-4 -13 2 -22.5t20 -9.5h608q13 0 25.5 9.5t16.5 22.5l21 64q4 13 -2 22.5t-20 9.5h-608q-13 0 -25.5 -9.5t-16.5 -22.5z" />
+<glyph unicode="&#xf02e;" horiz-adv-x="1280" d="M1164 1408q23 0 44 -9q33 -13 52.5 -41t19.5 -62v-1289q0 -34 -19.5 -62t-52.5 -41q-19 -8 -44 -8q-48 0 -83 32l-441 424l-441 -424q-36 -33 -83 -33q-23 0 -44 9q-33 13 -52.5 41t-19.5 62v1289q0 34 19.5 62t52.5 41q21 9 44 9h1048z" />
+<glyph unicode="&#xf02f;" horiz-adv-x="1664" d="M384 0h896v256h-896v-256zM384 640h896v384h-160q-40 0 -68 28t-28 68v160h-640v-640zM1536 576q0 26 -19 45t-45 19t-45 -19t-19 -45t19 -45t45 -19t45 19t19 45zM1664 576v-416q0 -13 -9.5 -22.5t-22.5 -9.5h-224v-160q0 -40 -28 -68t-68 -28h-960q-40 0 -68 28t-28 68 v160h-224q-13 0 -22.5 9.5t-9.5 22.5v416q0 79 56.5 135.5t135.5 56.5h64v544q0 40 28 68t68 28h672q40 0 88 -20t76 -48l152 -152q28 -28 48 -76t20 -88v-256h64q79 0 135.5 -56.5t56.5 -135.5z" />
+<glyph unicode="&#xf030;" horiz-adv-x="1920" d="M960 864q119 0 203.5 -84.5t84.5 -203.5t-84.5 -203.5t-203.5 -84.5t-203.5 84.5t-84.5 203.5t84.5 203.5t203.5 84.5zM1664 1280q106 0 181 -75t75 -181v-896q0 -106 -75 -181t-181 -75h-1408q-106 0 -181 75t-75 181v896q0 106 75 181t181 75h224l51 136 q19 49 69.5 84.5t103.5 35.5h512q53 0 103.5 -35.5t69.5 -84.5l51 -136h224zM960 128q185 0 316.5 131.5t131.5 316.5t-131.5 316.5t-316.5 131.5t-316.5 -131.5t-131.5 -316.5t131.5 -316.5t316.5 -131.5z" />
+<glyph unicode="&#xf031;" horiz-adv-x="1664" d="M725 977l-170 -450q73 -1 153.5 -2t119 -1.5t52.5 -0.5l29 2q-32 95 -92 241q-53 132 -92 211zM21 -128h-21l2 79q22 7 80 18q89 16 110 31q20 16 48 68l237 616l280 724h75h53l11 -21l205 -480q103 -242 124 -297q39 -102 96 -235q26 -58 65 -164q24 -67 65 -149 q22 -49 35 -57q22 -19 69 -23q47 -6 103 -27q6 -39 6 -57q0 -14 -1 -26q-80 0 -192 8q-93 8 -189 8q-79 0 -135 -2l-200 -11l-58 -2q0 45 4 78l131 28q56 13 68 23q12 12 12 27t-6 32l-47 114l-92 228l-450 2q-29 -65 -104 -274q-23 -64 -23 -84q0 -31 17 -43 q26 -21 103 -32q3 0 13.5 -2t30 -5t40.5 -6q1 -28 1 -58q0 -17 -2 -27q-66 0 -349 20l-48 -8q-81 -14 -167 -14z" />
+<glyph unicode="&#xf032;" horiz-adv-x="1408" d="M555 15q76 -32 140 -32q131 0 216 41t122 113q38 70 38 181q0 114 -41 180q-58 94 -141 126q-80 32 -247 32q-74 0 -101 -10v-144l-1 -173l3 -270q0 -15 12 -44zM541 761q43 -7 109 -7q175 0 264 65t89 224q0 112 -85 187q-84 75 -255 75q-52 0 -130 -13q0 -44 2 -77 q7 -122 6 -279l-1 -98q0 -43 1 -77zM0 -128l2 94q45 9 68 12q77 12 123 31q17 27 21 51q9 66 9 194l-2 497q-5 256 -9 404q-1 87 -11 109q-1 4 -12 12q-18 12 -69 15q-30 2 -114 13l-4 83l260 6l380 13l45 1q5 0 14 0.5t14 0.5q1 0 21.5 -0.5t40.5 -0.5h74q88 0 191 -27 q43 -13 96 -39q57 -29 102 -76q44 -47 65 -104t21 -122q0 -70 -32 -128t-95 -105q-26 -20 -150 -77q177 -41 267 -146q92 -106 92 -236q0 -76 -29 -161q-21 -62 -71 -117q-66 -72 -140 -108q-73 -36 -203 -60q-82 -15 -198 -11l-197 4q-84 2 -298 -11q-33 -3 -272 -11z" />
+<glyph unicode="&#xf033;" horiz-adv-x="1024" d="M0 -126l17 85q4 1 77 20q76 19 116 39q29 37 41 101l27 139l56 268l12 64q8 44 17 84.5t16 67t12.5 46.5t9 30.5t3.5 11.5l29 157l16 63l22 135l8 50v38q-41 22 -144 28q-28 2 -38 4l19 103l317 -14q39 -2 73 -2q66 0 214 9q33 2 68 4.5t36 2.5q-2 -19 -6 -38 q-7 -29 -13 -51q-55 -19 -109 -31q-64 -16 -101 -31q-12 -31 -24 -88q-9 -44 -13 -82q-44 -199 -66 -306l-61 -311l-38 -158l-43 -235l-12 -45q-2 -7 1 -27q64 -15 119 -21q36 -5 66 -10q-1 -29 -7 -58q-7 -31 -9 -41q-18 0 -23 -1q-24 -2 -42 -2q-9 0 -28 3q-19 4 -145 17 l-198 2q-41 1 -174 -11q-74 -7 -98 -9z" />
+<glyph unicode="&#xf034;" horiz-adv-x="1792" d="M81 1407l54 -27q20 -5 211 -5h130l19 3l115 1l215 -1h293l34 -2q14 -1 28 7t21 16l7 8l42 1q15 0 28 -1v-104.5t1 -131.5l1 -100l-1 -58q0 -32 -4 -51q-39 -15 -68 -18q-25 43 -54 128q-8 24 -15.5 62.5t-11.5 65.5t-6 29q-13 15 -27 19q-7 2 -42.5 2t-103.5 -1t-111 -1 q-34 0 -67 -5q-10 -97 -8 -136l1 -152v-332l3 -359l-1 -147q-1 -46 11 -85q49 -25 89 -32q2 0 18 -5t44 -13t43 -12q30 -8 50 -18q5 -45 5 -50q0 -10 -3 -29q-14 -1 -34 -1q-110 0 -187 10q-72 8 -238 8q-88 0 -233 -14q-48 -4 -70 -4q-2 22 -2 26l-1 26v9q21 33 79 49 q139 38 159 50q9 21 12 56q8 192 6 433l-5 428q-1 62 -0.5 118.5t0.5 102.5t-2 57t-6 15q-6 5 -14 6q-38 6 -148 6q-43 0 -100 -13.5t-73 -24.5q-13 -9 -22 -33t-22 -75t-24 -84q-6 -19 -19.5 -32t-20.5 -13q-44 27 -56 44v297v86zM1744 128q33 0 42 -18.5t-11 -44.5 l-126 -162q-20 -26 -49 -26t-49 26l-126 162q-20 26 -11 44.5t42 18.5h80v1024h-80q-33 0 -42 18.5t11 44.5l126 162q20 26 49 26t49 -26l126 -162q20 -26 11 -44.5t-42 -18.5h-80v-1024h80z" />
+<glyph unicode="&#xf035;" d="M81 1407l54 -27q20 -5 211 -5h130l19 3l115 1l446 -1h318l34 -2q14 -1 28 7t21 16l7 8l42 1q15 0 28 -1v-104.5t1 -131.5l1 -100l-1 -58q0 -32 -4 -51q-39 -15 -68 -18q-25 43 -54 128q-8 24 -15.5 62.5t-11.5 65.5t-6 29q-13 15 -27 19q-7 2 -58.5 2t-138.5 -1t-128 -1 q-94 0 -127 -5q-10 -97 -8 -136l1 -152v52l3 -359l-1 -147q-1 -46 11 -85q49 -25 89 -32q2 0 18 -5t44 -13t43 -12q30 -8 50 -18q5 -45 5 -50q0 -10 -3 -29q-14 -1 -34 -1q-110 0 -187 10q-72 8 -238 8q-82 0 -233 -13q-45 -5 -70 -5q-2 22 -2 26l-1 26v9q21 33 79 49 q139 38 159 50q9 21 12 56q6 137 6 433l-5 44q0 265 -2 278q-2 11 -6 15q-6 5 -14 6q-38 6 -148 6q-50 0 -168.5 -14t-132.5 -24q-13 -9 -22 -33t-22 -75t-24 -84q-6 -19 -19.5 -32t-20.5 -13q-44 27 -56 44v297v86zM1505 113q26 -20 26 -49t-26 -49l-162 -126 q-26 -20 -44.5 -11t-18.5 42v80h-1024v-80q0 -33 -18.5 -42t-44.5 11l-162 126q-26 20 -26 49t26 49l162 126q26 20 44.5 11t18.5 -42v-80h1024v80q0 33 18.5 42t44.5 -11z" />
+<glyph unicode="&#xf036;" horiz-adv-x="1792" d="M1792 192v-128q0 -26 -19 -45t-45 -19h-1664q-26 0 -45 19t-19 45v128q0 26 19 45t45 19h1664q26 0 45 -19t19 -45zM1408 576v-128q0 -26 -19 -45t-45 -19h-1280q-26 0 -45 19t-19 45v128q0 26 19 45t45 19h1280q26 0 45 -19t19 -45zM1664 960v-128q0 -26 -19 -45 t-45 -19h-1536q-26 0 -45 19t-19 45v128q0 26 19 45t45 19h1536q26 0 45 -19t19 -45zM1280 1344v-128q0 -26 -19 -45t-45 -19h-1152q-26 0 -45 19t-19 45v128q0 26 19 45t45 19h1152q26 0 45 -19t19 -45z" />
+<glyph unicode="&#xf037;" horiz-adv-x="1792" d="M1792 192v-128q0 -26 -19 -45t-45 -19h-1664q-26 0 -45 19t-19 45v128q0 26 19 45t45 19h1664q26 0 45 -19t19 -45zM1408 576v-128q0 -26 -19 -45t-45 -19h-896q-26 0 -45 19t-19 45v128q0 26 19 45t45 19h896q26 0 45 -19t19 -45zM1664 960v-128q0 -26 -19 -45t-45 -19 h-1408q-26 0 -45 19t-19 45v128q0 26 19 45t45 19h1408q26 0 45 -19t19 -45zM1280 1344v-128q0 -26 -19 -45t-45 -19h-640q-26 0 -45 19t-19 45v128q0 26 19 45t45 19h640q26 0 45 -19t19 -45z" />
+<glyph unicode="&#xf038;" horiz-adv-x="1792" d="M1792 192v-128q0 -26 -19 -45t-45 -19h-1664q-26 0 -45 19t-19 45v128q0 26 19 45t45 19h1664q26 0 45 -19t19 -45zM1792 576v-128q0 -26 -19 -45t-45 -19h-1280q-26 0 -45 19t-19 45v128q0 26 19 45t45 19h1280q26 0 45 -19t19 -45zM1792 960v-128q0 -26 -19 -45 t-45 -19h-1536q-26 0 -45 19t-19 45v128q0 26 19 45t45 19h1536q26 0 45 -19t19 -45zM1792 1344v-128q0 -26 -19 -45t-45 -19h-1152q-26 0 -45 19t-19 45v128q0 26 19 45t45 19h1152q26 0 45 -19t19 -45z" />
+<glyph unicode="&#xf039;" horiz-adv-x="1792" d="M1792 192v-128q0 -26 -19 -45t-45 -19h-1664q-26 0 -45 19t-19 45v128q0 26 19 45t45 19h1664q26 0 45 -19t19 -45zM1792 576v-128q0 -26 -19 -45t-45 -19h-1664q-26 0 -45 19t-19 45v128q0 26 19 45t45 19h1664q26 0 45 -19t19 -45zM1792 960v-128q0 -26 -19 -45 t-45 -19h-1664q-26 0 -45 19t-19 45v128q0 26 19 45t45 19h1664q26 0 45 -19t19 -45zM1792 1344v-128q0 -26 -19 -45t-45 -19h-1664q-26 0 -45 19t-19 45v128q0 26 19 45t45 19h1664q26 0 45 -19t19 -45z" />
+<glyph unicode="&#xf03a;" horiz-adv-x="1792" d="M256 224v-192q0 -13 -9.5 -22.5t-22.5 -9.5h-192q-13 0 -22.5 9.5t-9.5 22.5v192q0 13 9.5 22.5t22.5 9.5h192q13 0 22.5 -9.5t9.5 -22.5zM256 608v-192q0 -13 -9.5 -22.5t-22.5 -9.5h-192q-13 0 -22.5 9.5t-9.5 22.5v192q0 13 9.5 22.5t22.5 9.5h192q13 0 22.5 -9.5 t9.5 -22.5zM256 992v-192q0 -13 -9.5 -22.5t-22.5 -9.5h-192q-13 0 -22.5 9.5t-9.5 22.5v192q0 13 9.5 22.5t22.5 9.5h192q13 0 22.5 -9.5t9.5 -22.5zM1792 224v-192q0 -13 -9.5 -22.5t-22.5 -9.5h-1344q-13 0 -22.5 9.5t-9.5 22.5v192q0 13 9.5 22.5t22.5 9.5h1344 q13 0 22.5 -9.5t9.5 -22.5zM256 1376v-192q0 -13 -9.5 -22.5t-22.5 -9.5h-192q-13 0 -22.5 9.5t-9.5 22.5v192q0 13 9.5 22.5t22.5 9.5h192q13 0 22.5 -9.5t9.5 -22.5zM1792 608v-192q0 -13 -9.5 -22.5t-22.5 -9.5h-1344q-13 0 -22.5 9.5t-9.5 22.5v192q0 13 9.5 22.5 t22.5 9.5h1344q13 0 22.5 -9.5t9.5 -22.5zM1792 992v-192q0 -13 -9.5 -22.5t-22.5 -9.5h-1344q-13 0 -22.5 9.5t-9.5 22.5v192q0 13 9.5 22.5t22.5 9.5h1344q13 0 22.5 -9.5t9.5 -22.5zM1792 1376v-192q0 -13 -9.5 -22.5t-22.5 -9.5h-1344q-13 0 -22.5 9.5t-9.5 22.5v192 q0 13 9.5 22.5t22.5 9.5h1344q13 0 22.5 -9.5t9.5 -22.5z" />
+<glyph unicode="&#xf03b;" horiz-adv-x="1792" d="M384 992v-576q0 -13 -9.5 -22.5t-22.5 -9.5q-14 0 -23 9l-288 288q-9 9 -9 23t9 23l288 288q9 9 23 9q13 0 22.5 -9.5t9.5 -22.5zM1792 224v-192q0 -13 -9.5 -22.5t-22.5 -9.5h-1728q-13 0 -22.5 9.5t-9.5 22.5v192q0 13 9.5 22.5t22.5 9.5h1728q13 0 22.5 -9.5 t9.5 -22.5zM1792 608v-192q0 -13 -9.5 -22.5t-22.5 -9.5h-1088q-13 0 -22.5 9.5t-9.5 22.5v192q0 13 9.5 22.5t22.5 9.5h1088q13 0 22.5 -9.5t9.5 -22.5zM1792 992v-192q0 -13 -9.5 -22.5t-22.5 -9.5h-1088q-13 0 -22.5 9.5t-9.5 22.5v192q0 13 9.5 22.5t22.5 9.5h1088 q13 0 22.5 -9.5t9.5 -22.5zM1792 1376v-192q0 -13 -9.5 -22.5t-22.5 -9.5h-1728q-13 0 -22.5 9.5t-9.5 22.5v192q0 13 9.5 22.5t22.5 9.5h1728q13 0 22.5 -9.5t9.5 -22.5z" />
+<glyph unicode="&#xf03c;" horiz-adv-x="1792" d="M352 704q0 -14 -9 -23l-288 -288q-9 -9 -23 -9q-13 0 -22.5 9.5t-9.5 22.5v576q0 13 9.5 22.5t22.5 9.5q14 0 23 -9l288 -288q9 -9 9 -23zM1792 224v-192q0 -13 -9.5 -22.5t-22.5 -9.5h-1728q-13 0 -22.5 9.5t-9.5 22.5v192q0 13 9.5 22.5t22.5 9.5h1728q13 0 22.5 -9.5 t9.5 -22.5zM1792 608v-192q0 -13 -9.5 -22.5t-22.5 -9.5h-1088q-13 0 -22.5 9.5t-9.5 22.5v192q0 13 9.5 22.5t22.5 9.5h1088q13 0 22.5 -9.5t9.5 -22.5zM1792 992v-192q0 -13 -9.5 -22.5t-22.5 -9.5h-1088q-13 0 -22.5 9.5t-9.5 22.5v192q0 13 9.5 22.5t22.5 9.5h1088 q13 0 22.5 -9.5t9.5 -22.5zM1792 1376v-192q0 -13 -9.5 -22.5t-22.5 -9.5h-1728q-13 0 -22.5 9.5t-9.5 22.5v192q0 13 9.5 22.5t22.5 9.5h1728q13 0 22.5 -9.5t9.5 -22.5z" />
+<glyph unicode="&#xf03d;" horiz-adv-x="1792" d="M1792 1184v-1088q0 -42 -39 -59q-13 -5 -25 -5q-27 0 -45 19l-403 403v-166q0 -119 -84.5 -203.5t-203.5 -84.5h-704q-119 0 -203.5 84.5t-84.5 203.5v704q0 119 84.5 203.5t203.5 84.5h704q119 0 203.5 -84.5t84.5 -203.5v-165l403 402q18 19 45 19q12 0 25 -5 q39 -17 39 -59z" />
+<glyph unicode="&#xf03e;" horiz-adv-x="1920" d="M640 960q0 -80 -56 -136t-136 -56t-136 56t-56 136t56 136t136 56t136 -56t56 -136zM1664 576v-448h-1408v192l320 320l160 -160l512 512zM1760 1280h-1600q-13 0 -22.5 -9.5t-9.5 -22.5v-1216q0 -13 9.5 -22.5t22.5 -9.5h1600q13 0 22.5 9.5t9.5 22.5v1216 q0 13 -9.5 22.5t-22.5 9.5zM1920 1248v-1216q0 -66 -47 -113t-113 -47h-1600q-66 0 -113 47t-47 113v1216q0 66 47 113t113 47h1600q66 0 113 -47t47 -113z" />
+<glyph unicode="&#xf040;" d="M363 0l91 91l-235 235l-91 -91v-107h128v-128h107zM886 928q0 22 -22 22q-10 0 -17 -7l-542 -542q-7 -7 -7 -17q0 -22 22 -22q10 0 17 7l542 542q7 7 7 17zM832 1120l416 -416l-832 -832h-416v416zM1515 1024q0 -53 -37 -90l-166 -166l-416 416l166 165q36 38 90 38 q53 0 91 -38l235 -234q37 -39 37 -91z" />
+<glyph unicode="&#xf041;" horiz-adv-x="1024" d="M768 896q0 106 -75 181t-181 75t-181 -75t-75 -181t75 -181t181 -75t181 75t75 181zM1024 896q0 -109 -33 -179l-364 -774q-16 -33 -47.5 -52t-67.5 -19t-67.5 19t-46.5 52l-365 774q-33 70 -33 179q0 212 150 362t362 150t362 -150t150 -362z" />
+<glyph unicode="&#xf042;" d="M768 96v1088q-148 0 -273 -73t-198 -198t-73 -273t73 -273t198 -198t273 -73zM1536 640q0 -209 -103 -385.5t-279.5 -279.5t-385.5 -103t-385.5 103t-279.5 279.5t-103 385.5t103 385.5t279.5 279.5t385.5 103t385.5 -103t279.5 -279.5t103 -385.5z" />
+<glyph unicode="&#xf043;" horiz-adv-x="1024" d="M512 384q0 36 -20 69q-1 1 -15.5 22.5t-25.5 38t-25 44t-21 50.5q-4 16 -21 16t-21 -16q-7 -23 -21 -50.5t-25 -44t-25.5 -38t-15.5 -22.5q-20 -33 -20 -69q0 -53 37.5 -90.5t90.5 -37.5t90.5 37.5t37.5 90.5zM1024 512q0 -212 -150 -362t-362 -150t-362 150t-150 362 q0 145 81 275q6 9 62.5 90.5t101 151t99.5 178t83 201.5q9 30 34 47t51 17t51.5 -17t33.5 -47q28 -93 83 -201.5t99.5 -178t101 -151t62.5 -90.5q81 -127 81 -275z" />
+<glyph unicode="&#xf044;" horiz-adv-x="1792" d="M888 352l116 116l-152 152l-116 -116v-56h96v-96h56zM1328 1072q-16 16 -33 -1l-350 -350q-17 -17 -1 -33t33 1l350 350q17 17 1 33zM1408 478v-190q0 -119 -84.5 -203.5t-203.5 -84.5h-832q-119 0 -203.5 84.5t-84.5 203.5v832q0 119 84.5 203.5t203.5 84.5h832 q63 0 117 -25q15 -7 18 -23q3 -17 -9 -29l-49 -49q-14 -14 -32 -8q-23 6 -45 6h-832q-66 0 -113 -47t-47 -113v-832q0 -66 47 -113t113 -47h832q66 0 113 47t47 113v126q0 13 9 22l64 64q15 15 35 7t20 -29zM1312 1216l288 -288l-672 -672h-288v288zM1756 1084l-92 -92 l-288 288l92 92q28 28 68 28t68 -28l152 -152q28 -28 28 -68t-28 -68z" />
+<glyph unicode="&#xf045;" horiz-adv-x="1664" d="M1408 547v-259q0 -119 -84.5 -203.5t-203.5 -84.5h-832q-119 0 -203.5 84.5t-84.5 203.5v832q0 119 84.5 203.5t203.5 84.5h255v0q13 0 22.5 -9.5t9.5 -22.5q0 -27 -26 -32q-77 -26 -133 -60q-10 -4 -16 -4h-112q-66 0 -113 -47t-47 -113v-832q0 -66 47 -113t113 -47h832 q66 0 113 47t47 113v214q0 19 18 29q28 13 54 37q16 16 35 8q21 -9 21 -29zM1645 1043l-384 -384q-18 -19 -45 -19q-12 0 -25 5q-39 17 -39 59v192h-160q-323 0 -438 -131q-119 -137 -74 -473q3 -23 -20 -34q-8 -2 -12 -2q-16 0 -26 13q-10 14 -21 31t-39.5 68.5t-49.5 99.5 t-38.5 114t-17.5 122q0 49 3.5 91t14 90t28 88t47 81.5t68.5 74t94.5 61.5t124.5 48.5t159.5 30.5t196.5 11h160v192q0 42 39 59q13 5 25 5q26 0 45 -19l384 -384q19 -19 19 -45t-19 -45z" />
+<glyph unicode="&#xf046;" horiz-adv-x="1664" d="M1408 606v-318q0 -119 -84.5 -203.5t-203.5 -84.5h-832q-119 0 -203.5 84.5t-84.5 203.5v832q0 119 84.5 203.5t203.5 84.5h832q63 0 117 -25q15 -7 18 -23q3 -17 -9 -29l-49 -49q-10 -10 -23 -10q-3 0 -9 2q-23 6 -45 6h-832q-66 0 -113 -47t-47 -113v-832 q0 -66 47 -113t113 -47h832q66 0 113 47t47 113v254q0 13 9 22l64 64q10 10 23 10q6 0 12 -3q20 -8 20 -29zM1639 1095l-814 -814q-24 -24 -57 -24t-57 24l-430 430q-24 24 -24 57t24 57l110 110q24 24 57 24t57 -24l263 -263l647 647q24 24 57 24t57 -24l110 -110 q24 -24 24 -57t-24 -57z" />
+<glyph unicode="&#xf047;" horiz-adv-x="1792" d="M1792 640q0 -26 -19 -45l-256 -256q-19 -19 -45 -19t-45 19t-19 45v128h-384v-384h128q26 0 45 -19t19 -45t-19 -45l-256 -256q-19 -19 -45 -19t-45 19l-256 256q-19 19 -19 45t19 45t45 19h128v384h-384v-128q0 -26 -19 -45t-45 -19t-45 19l-256 256q-19 19 -19 45 t19 45l256 256q19 19 45 19t45 -19t19 -45v-128h384v384h-128q-26 0 -45 19t-19 45t19 45l256 256q19 19 45 19t45 -19l256 -256q19 -19 19 -45t-19 -45t-45 -19h-128v-384h384v128q0 26 19 45t45 19t45 -19l256 -256q19 -19 19 -45z" />
+<glyph unicode="&#xf048;" horiz-adv-x="1024" d="M979 1395q19 19 32 13t13 -32v-1472q0 -26 -13 -32t-32 13l-710 710q-9 9 -13 19v-678q0 -26 -19 -45t-45 -19h-128q-26 0 -45 19t-19 45v1408q0 26 19 45t45 19h128q26 0 45 -19t19 -45v-678q4 11 13 19z" />
+<glyph unicode="&#xf049;" horiz-adv-x="1792" d="M1747 1395q19 19 32 13t13 -32v-1472q0 -26 -13 -32t-32 13l-710 710q-9 9 -13 19v-710q0 -26 -13 -32t-32 13l-710 710q-9 9 -13 19v-678q0 -26 -19 -45t-45 -19h-128q-26 0 -45 19t-19 45v1408q0 26 19 45t45 19h128q26 0 45 -19t19 -45v-678q4 11 13 19l710 710 q19 19 32 13t13 -32v-710q4 11 13 19z" />
+<glyph unicode="&#xf04a;" horiz-adv-x="1664" d="M1619 1395q19 19 32 13t13 -32v-1472q0 -26 -13 -32t-32 13l-710 710q-8 9 -13 19v-710q0 -26 -13 -32t-32 13l-710 710q-19 19 -19 45t19 45l710 710q19 19 32 13t13 -32v-710q5 11 13 19z" />
+<glyph unicode="&#xf04b;" horiz-adv-x="1408" d="M1384 609l-1328 -738q-23 -13 -39.5 -3t-16.5 36v1472q0 26 16.5 36t39.5 -3l1328 -738q23 -13 23 -31t-23 -31z" />
+<glyph unicode="&#xf04c;" d="M1536 1344v-1408q0 -26 -19 -45t-45 -19h-512q-26 0 -45 19t-19 45v1408q0 26 19 45t45 19h512q26 0 45 -19t19 -45zM640 1344v-1408q0 -26 -19 -45t-45 -19h-512q-26 0 -45 19t-19 45v1408q0 26 19 45t45 19h512q26 0 45 -19t19 -45z" />
+<glyph unicode="&#xf04d;" d="M1536 1344v-1408q0 -26 -19 -45t-45 -19h-1408q-26 0 -45 19t-19 45v1408q0 26 19 45t45 19h1408q26 0 45 -19t19 -45z" />
+<glyph unicode="&#xf04e;" horiz-adv-x="1664" d="M45 -115q-19 -19 -32 -13t-13 32v1472q0 26 13 32t32 -13l710 -710q8 -8 13 -19v710q0 26 13 32t32 -13l710 -710q19 -19 19 -45t-19 -45l-710 -710q-19 -19 -32 -13t-13 32v710q-5 -10 -13 -19z" />
+<glyph unicode="&#xf050;" horiz-adv-x="1792" d="M45 -115q-19 -19 -32 -13t-13 32v1472q0 26 13 32t32 -13l710 -710q8 -8 13 -19v710q0 26 13 32t32 -13l710 -710q8 -8 13 -19v678q0 26 19 45t45 19h128q26 0 45 -19t19 -45v-1408q0 -26 -19 -45t-45 -19h-128q-26 0 -45 19t-19 45v678q-5 -10 -13 -19l-710 -710 q-19 -19 -32 -13t-13 32v710q-5 -10 -13 -19z" />
+<glyph unicode="&#xf051;" horiz-adv-x="1024" d="M45 -115q-19 -19 -32 -13t-13 32v1472q0 26 13 32t32 -13l710 -710q8 -8 13 -19v678q0 26 19 45t45 19h128q26 0 45 -19t19 -45v-1408q0 -26 -19 -45t-45 -19h-128q-26 0 -45 19t-19 45v678q-5 -10 -13 -19z" />
+<glyph unicode="&#xf052;" horiz-adv-x="1538" d="M14 557l710 710q19 19 45 19t45 -19l710 -710q19 -19 13 -32t-32 -13h-1472q-26 0 -32 13t13 32zM1473 0h-1408q-26 0 -45 19t-19 45v256q0 26 19 45t45 19h1408q26 0 45 -19t19 -45v-256q0 -26 -19 -45t-45 -19z" />
+<glyph unicode="&#xf053;" horiz-adv-x="1152" d="M742 -37l-652 651q-37 37 -37 90.5t37 90.5l652 651q37 37 90.5 37t90.5 -37l75 -75q37 -37 37 -90.5t-37 -90.5l-486 -486l486 -485q37 -38 37 -91t-37 -90l-75 -75q-37 -37 -90.5 -37t-90.5 37z" />
+<glyph unicode="&#xf054;" horiz-adv-x="1152" d="M1099 704q0 -52 -37 -91l-652 -651q-37 -37 -90 -37t-90 37l-76 75q-37 39 -37 91q0 53 37 90l486 486l-486 485q-37 39 -37 91q0 53 37 90l76 75q36 38 90 38t90 -38l652 -651q37 -37 37 -90z" />
+<glyph unicode="&#xf055;" d="M1216 576v128q0 26 -19 45t-45 19h-256v256q0 26 -19 45t-45 19h-128q-26 0 -45 -19t-19 -45v-256h-256q-26 0 -45 -19t-19 -45v-128q0 -26 19 -45t45 -19h256v-256q0 -26 19 -45t45 -19h128q26 0 45 19t19 45v256h256q26 0 45 19t19 45zM1536 640q0 -209 -103 -385.5 t-279.5 -279.5t-385.5 -103t-385.5 103t-279.5 279.5t-103 385.5t103 385.5t279.5 279.5t385.5 103t385.5 -103t279.5 -279.5t103 -385.5z" />
+<glyph unicode="&#xf056;" d="M1216 576v128q0 26 -19 45t-45 19h-768q-26 0 -45 -19t-19 -45v-128q0 -26 19 -45t45 -19h768q26 0 45 19t19 45zM1536 640q0 -209 -103 -385.5t-279.5 -279.5t-385.5 -103t-385.5 103t-279.5 279.5t-103 385.5t103 385.5t279.5 279.5t385.5 103t385.5 -103t279.5 -279.5 t103 -385.5z" />
+<glyph unicode="&#xf057;" d="M1149 414q0 26 -19 45l-181 181l181 181q19 19 19 45q0 27 -19 46l-90 90q-19 19 -46 19q-26 0 -45 -19l-181 -181l-181 181q-19 19 -45 19q-27 0 -46 -19l-90 -90q-19 -19 -19 -46q0 -26 19 -45l181 -181l-181 -181q-19 -19 -19 -45q0 -27 19 -46l90 -90q19 -19 46 -19 q26 0 45 19l181 181l181 -181q19 -19 45 -19q27 0 46 19l90 90q19 19 19 46zM1536 640q0 -209 -103 -385.5t-279.5 -279.5t-385.5 -103t-385.5 103t-279.5 279.5t-103 385.5t103 385.5t279.5 279.5t385.5 103t385.5 -103t279.5 -279.5t103 -385.5z" />
+<glyph unicode="&#xf058;" d="M1284 802q0 28 -18 46l-91 90q-19 19 -45 19t-45 -19l-408 -407l-226 226q-19 19 -45 19t-45 -19l-91 -90q-18 -18 -18 -46q0 -27 18 -45l362 -362q19 -19 45 -19q27 0 46 19l543 543q18 18 18 45zM1536 640q0 -209 -103 -385.5t-279.5 -279.5t-385.5 -103t-385.5 103 t-279.5 279.5t-103 385.5t103 385.5t279.5 279.5t385.5 103t385.5 -103t279.5 -279.5t103 -385.5z" />
+<glyph unicode="&#xf059;" d="M896 160v192q0 14 -9 23t-23 9h-192q-14 0 -23 -9t-9 -23v-192q0 -14 9 -23t23 -9h192q14 0 23 9t9 23zM1152 832q0 88 -55.5 163t-138.5 116t-170 41q-243 0 -371 -213q-15 -24 8 -42l132 -100q7 -6 19 -6q16 0 25 12q53 68 86 92q34 24 86 24q48 0 85.5 -26t37.5 -59 q0 -38 -20 -61t-68 -45q-63 -28 -115.5 -86.5t-52.5 -125.5v-36q0 -14 9 -23t23 -9h192q14 0 23 9t9 23q0 19 21.5 49.5t54.5 49.5q32 18 49 28.5t46 35t44.5 48t28 60.5t12.5 81zM1536 640q0 -209 -103 -385.5t-279.5 -279.5t-385.5 -103t-385.5 103t-279.5 279.5 t-103 385.5t103 385.5t279.5 279.5t385.5 103t385.5 -103t279.5 -279.5t103 -385.5z" />
+<glyph unicode="&#xf05a;" d="M1024 160v160q0 14 -9 23t-23 9h-96v512q0 14 -9 23t-23 9h-320q-14 0 -23 -9t-9 -23v-160q0 -14 9 -23t23 -9h96v-320h-96q-14 0 -23 -9t-9 -23v-160q0 -14 9 -23t23 -9h448q14 0 23 9t9 23zM896 1056v160q0 14 -9 23t-23 9h-192q-14 0 -23 -9t-9 -23v-160q0 -14 9 -23 t23 -9h192q14 0 23 9t9 23zM1536 640q0 -209 -103 -385.5t-279.5 -279.5t-385.5 -103t-385.5 103t-279.5 279.5t-103 385.5t103 385.5t279.5 279.5t385.5 103t385.5 -103t279.5 -279.5t103 -385.5z" />
+<glyph unicode="&#xf05b;" d="M1197 512h-109q-26 0 -45 19t-19 45v128q0 26 19 45t45 19h109q-32 108 -112.5 188.5t-188.5 112.5v-109q0 -26 -19 -45t-45 -19h-128q-26 0 -45 19t-19 45v109q-108 -32 -188.5 -112.5t-112.5 -188.5h109q26 0 45 -19t19 -45v-128q0 -26 -19 -45t-45 -19h-109 q32 -108 112.5 -188.5t188.5 -112.5v109q0 26 19 45t45 19h128q26 0 45 -19t19 -45v-109q108 32 188.5 112.5t112.5 188.5zM1536 704v-128q0 -26 -19 -45t-45 -19h-143q-37 -161 -154.5 -278.5t-278.5 -154.5v-143q0 -26 -19 -45t-45 -19h-128q-26 0 -45 19t-19 45v143 q-161 37 -278.5 154.5t-154.5 278.5h-143q-26 0 -45 19t-19 45v128q0 26 19 45t45 19h143q37 161 154.5 278.5t278.5 154.5v143q0 26 19 45t45 19h128q26 0 45 -19t19 -45v-143q161 -37 278.5 -154.5t154.5 -278.5h143q26 0 45 -19t19 -45z" />
+<glyph unicode="&#xf05c;" d="M1097 457l-146 -146q-10 -10 -23 -10t-23 10l-137 137l-137 -137q-10 -10 -23 -10t-23 10l-146 146q-10 10 -10 23t10 23l137 137l-137 137q-10 10 -10 23t10 23l146 146q10 10 23 10t23 -10l137 -137l137 137q10 10 23 10t23 -10l146 -146q10 -10 10 -23t-10 -23 l-137 -137l137 -137q10 -10 10 -23t-10 -23zM1312 640q0 148 -73 273t-198 198t-273 73t-273 -73t-198 -198t-73 -273t73 -273t198 -198t273 -73t273 73t198 198t73 273zM1536 640q0 -209 -103 -385.5t-279.5 -279.5t-385.5 -103t-385.5 103t-279.5 279.5t-103 385.5 t103 385.5t279.5 279.5t385.5 103t385.5 -103t279.5 -279.5t103 -385.5z" />
+<glyph unicode="&#xf05d;" d="M1171 723l-422 -422q-19 -19 -45 -19t-45 19l-294 294q-19 19 -19 45t19 45l102 102q19 19 45 19t45 -19l147 -147l275 275q19 19 45 19t45 -19l102 -102q19 -19 19 -45t-19 -45zM1312 640q0 148 -73 273t-198 198t-273 73t-273 -73t-198 -198t-73 -273t73 -273t198 -198 t273 -73t273 73t198 198t73 273zM1536 640q0 -209 -103 -385.5t-279.5 -279.5t-385.5 -103t-385.5 103t-279.5 279.5t-103 385.5t103 385.5t279.5 279.5t385.5 103t385.5 -103t279.5 -279.5t103 -385.5z" />
+<glyph unicode="&#xf05e;" d="M1312 643q0 161 -87 295l-754 -753q137 -89 297 -89q111 0 211.5 43.5t173.5 116.5t116 174.5t43 212.5zM313 344l755 754q-135 91 -300 91q-148 0 -273 -73t-198 -199t-73 -274q0 -162 89 -299zM1536 643q0 -157 -61 -300t-163.5 -246t-245 -164t-298.5 -61t-298.5 61 t-245 164t-163.5 246t-61 300t61 299.5t163.5 245.5t245 164t298.5 61t298.5 -61t245 -164t163.5 -245.5t61 -299.5z" />
+<glyph unicode="&#xf060;" d="M1536 640v-128q0 -53 -32.5 -90.5t-84.5 -37.5h-704l293 -294q38 -36 38 -90t-38 -90l-75 -76q-37 -37 -90 -37q-52 0 -91 37l-651 652q-37 37 -37 90q0 52 37 91l651 650q38 38 91 38q52 0 90 -38l75 -74q38 -38 38 -91t-38 -91l-293 -293h704q52 0 84.5 -37.5 t32.5 -90.5z" />
+<glyph unicode="&#xf061;" d="M1472 576q0 -54 -37 -91l-651 -651q-39 -37 -91 -37q-51 0 -90 37l-75 75q-38 38 -38 91t38 91l293 293h-704q-52 0 -84.5 37.5t-32.5 90.5v128q0 53 32.5 90.5t84.5 37.5h704l-293 294q-38 36 -38 90t38 90l75 75q38 38 90 38q53 0 91 -38l651 -651q37 -35 37 -90z" />
+<glyph unicode="&#xf062;" horiz-adv-x="1664" d="M1611 565q0 -51 -37 -90l-75 -75q-38 -38 -91 -38q-54 0 -90 38l-294 293v-704q0 -52 -37.5 -84.5t-90.5 -32.5h-128q-53 0 -90.5 32.5t-37.5 84.5v704l-294 -293q-36 -38 -90 -38t-90 38l-75 75q-38 38 -38 90q0 53 38 91l651 651q35 37 90 37q54 0 91 -37l651 -651 q37 -39 37 -91z" />
+<glyph unicode="&#xf063;" horiz-adv-x="1664" d="M1611 704q0 -53 -37 -90l-651 -652q-39 -37 -91 -37q-53 0 -90 37l-651 652q-38 36 -38 90q0 53 38 91l74 75q39 37 91 37q53 0 90 -37l294 -294v704q0 52 38 90t90 38h128q52 0 90 -38t38 -90v-704l294 294q37 37 90 37q52 0 91 -37l75 -75q37 -39 37 -91z" />
+<glyph unicode="&#xf064;" horiz-adv-x="1792" d="M1792 896q0 -26 -19 -45l-512 -512q-19 -19 -45 -19t-45 19t-19 45v256h-224q-98 0 -175.5 -6t-154 -21.5t-133 -42.5t-105.5 -69.5t-80 -101t-48.5 -138.5t-17.5 -181q0 -55 5 -123q0 -6 2.5 -23.5t2.5 -26.5q0 -15 -8.5 -25t-23.5 -10q-16 0 -28 17q-7 9 -13 22 t-13.5 30t-10.5 24q-127 285 -127 451q0 199 53 333q162 403 875 403h224v256q0 26 19 45t45 19t45 -19l512 -512q19 -19 19 -45z" />
+<glyph unicode="&#xf065;" d="M755 480q0 -13 -10 -23l-332 -332l144 -144q19 -19 19 -45t-19 -45t-45 -19h-448q-26 0 -45 19t-19 45v448q0 26 19 45t45 19t45 -19l144 -144l332 332q10 10 23 10t23 -10l114 -114q10 -10 10 -23zM1536 1344v-448q0 -26 -19 -45t-45 -19t-45 19l-144 144l-332 -332 q-10 -10 -23 -10t-23 10l-114 114q-10 10 -10 23t10 23l332 332l-144 144q-19 19 -19 45t19 45t45 19h448q26 0 45 -19t19 -45z" />
+<glyph unicode="&#xf066;" d="M768 576v-448q0 -26 -19 -45t-45 -19t-45 19l-144 144l-332 -332q-10 -10 -23 -10t-23 10l-114 114q-10 10 -10 23t10 23l332 332l-144 144q-19 19 -19 45t19 45t45 19h448q26 0 45 -19t19 -45zM1523 1248q0 -13 -10 -23l-332 -332l144 -144q19 -19 19 -45t-19 -45 t-45 -19h-448q-26 0 -45 19t-19 45v448q0 26 19 45t45 19t45 -19l144 -144l332 332q10 10 23 10t23 -10l114 -114q10 -10 10 -23z" />
+<glyph unicode="&#xf067;" horiz-adv-x="1408" d="M1408 800v-192q0 -40 -28 -68t-68 -28h-416v-416q0 -40 -28 -68t-68 -28h-192q-40 0 -68 28t-28 68v416h-416q-40 0 -68 28t-28 68v192q0 40 28 68t68 28h416v416q0 40 28 68t68 28h192q40 0 68 -28t28 -68v-416h416q40 0 68 -28t28 -68z" />
+<glyph unicode="&#xf068;" horiz-adv-x="1408" d="M1408 800v-192q0 -40 -28 -68t-68 -28h-1216q-40 0 -68 28t-28 68v192q0 40 28 68t68 28h1216q40 0 68 -28t28 -68z" />
+<glyph unicode="&#xf069;" horiz-adv-x="1664" d="M1482 486q46 -26 59.5 -77.5t-12.5 -97.5l-64 -110q-26 -46 -77.5 -59.5t-97.5 12.5l-266 153v-307q0 -52 -38 -90t-90 -38h-128q-52 0 -90 38t-38 90v307l-266 -153q-46 -26 -97.5 -12.5t-77.5 59.5l-64 110q-26 46 -12.5 97.5t59.5 77.5l266 154l-266 154 q-46 26 -59.5 77.5t12.5 97.5l64 110q26 46 77.5 59.5t97.5 -12.5l266 -153v307q0 52 38 90t90 38h128q52 0 90 -38t38 -90v-307l266 153q46 26 97.5 12.5t77.5 -59.5l64 -110q26 -46 12.5 -97.5t-59.5 -77.5l-266 -154z" />
+<glyph unicode="&#xf06a;" d="M768 1408q209 0 385.5 -103t279.5 -279.5t103 -385.5t-103 -385.5t-279.5 -279.5t-385.5 -103t-385.5 103t-279.5 279.5t-103 385.5t103 385.5t279.5 279.5t385.5 103zM896 161v190q0 14 -9 23.5t-22 9.5h-192q-13 0 -23 -10t-10 -23v-190q0 -13 10 -23t23 -10h192 q13 0 22 9.5t9 23.5zM894 505l18 621q0 12 -10 18q-10 8 -24 8h-220q-14 0 -24 -8q-10 -6 -10 -18l17 -621q0 -10 10 -17.5t24 -7.5h185q14 0 23.5 7.5t10.5 17.5z" />
+<glyph unicode="&#xf06b;" d="M928 180v56v468v192h-320v-192v-468v-56q0 -25 18 -38.5t46 -13.5h192q28 0 46 13.5t18 38.5zM472 1024h195l-126 161q-26 31 -69 31q-40 0 -68 -28t-28 -68t28 -68t68 -28zM1160 1120q0 40 -28 68t-68 28q-43 0 -69 -31l-125 -161h194q40 0 68 28t28 68zM1536 864v-320 q0 -14 -9 -23t-23 -9h-96v-416q0 -40 -28 -68t-68 -28h-1088q-40 0 -68 28t-28 68v416h-96q-14 0 -23 9t-9 23v320q0 14 9 23t23 9h440q-93 0 -158.5 65.5t-65.5 158.5t65.5 158.5t158.5 65.5q107 0 168 -77l128 -165l128 165q61 77 168 77q93 0 158.5 -65.5t65.5 -158.5 t-65.5 -158.5t-158.5 -65.5h440q14 0 23 -9t9 -23z" />
+<glyph unicode="&#xf06c;" horiz-adv-x="1792" d="M1280 832q0 26 -19 45t-45 19q-172 0 -318 -49.5t-259.5 -134t-235.5 -219.5q-19 -21 -19 -45q0 -26 19 -45t45 -19q24 0 45 19q27 24 74 71t67 66q137 124 268.5 176t313.5 52q26 0 45 19t19 45zM1792 1030q0 -95 -20 -193q-46 -224 -184.5 -383t-357.5 -268 q-214 -108 -438 -108q-148 0 -286 47q-15 5 -88 42t-96 37q-16 0 -39.5 -32t-45 -70t-52.5 -70t-60 -32q-30 0 -51 11t-31 24t-27 42q-2 4 -6 11t-5.5 10t-3 9.5t-1.5 13.5q0 35 31 73.5t68 65.5t68 56t31 48q0 4 -14 38t-16 44q-9 51 -9 104q0 115 43.5 220t119 184.5 t170.5 139t204 95.5q55 18 145 25.5t179.5 9t178.5 6t163.5 24t113.5 56.5l29.5 29.5t29.5 28t27 20t36.5 16t43.5 4.5q39 0 70.5 -46t47.5 -112t24 -124t8 -96z" />
+<glyph unicode="&#xf06d;" horiz-adv-x="1408" d="M1408 -160v-64q0 -13 -9.5 -22.5t-22.5 -9.5h-1344q-13 0 -22.5 9.5t-9.5 22.5v64q0 13 9.5 22.5t22.5 9.5h1344q13 0 22.5 -9.5t9.5 -22.5zM1152 896q0 -78 -24.5 -144t-64 -112.5t-87.5 -88t-96 -77.5t-87.5 -72t-64 -81.5t-24.5 -96.5q0 -96 67 -224l-4 1l1 -1 q-90 41 -160 83t-138.5 100t-113.5 122.5t-72.5 150.5t-27.5 184q0 78 24.5 144t64 112.5t87.5 88t96 77.5t87.5 72t64 81.5t24.5 96.5q0 94 -66 224l3 -1l-1 1q90 -41 160 -83t138.5 -100t113.5 -122.5t72.5 -150.5t27.5 -184z" />
+<glyph unicode="&#xf06e;" horiz-adv-x="1792" d="M1664 576q-152 236 -381 353q61 -104 61 -225q0 -185 -131.5 -316.5t-316.5 -131.5t-316.5 131.5t-131.5 316.5q0 121 61 225q-229 -117 -381 -353q133 -205 333.5 -326.5t434.5 -121.5t434.5 121.5t333.5 326.5zM944 960q0 20 -14 34t-34 14q-125 0 -214.5 -89.5 t-89.5 -214.5q0 -20 14 -34t34 -14t34 14t14 34q0 86 61 147t147 61q20 0 34 14t14 34zM1792 576q0 -34 -20 -69q-140 -230 -376.5 -368.5t-499.5 -138.5t-499.5 139t-376.5 368q-20 35 -20 69t20 69q140 229 376.5 368t499.5 139t499.5 -139t376.5 -368q20 -35 20 -69z" />
+<glyph unicode="&#xf070;" horiz-adv-x="1792" d="M555 201l78 141q-87 63 -136 159t-49 203q0 121 61 225q-229 -117 -381 -353q167 -258 427 -375zM944 960q0 20 -14 34t-34 14q-125 0 -214.5 -89.5t-89.5 -214.5q0 -20 14 -34t34 -14t34 14t14 34q0 86 61 147t147 61q20 0 34 14t14 34zM1307 1151q0 -7 -1 -9 q-105 -188 -315 -566t-316 -567l-49 -89q-10 -16 -28 -16q-12 0 -134 70q-16 10 -16 28q0 12 44 87q-143 65 -263.5 173t-208.5 245q-20 31 -20 69t20 69q153 235 380 371t496 136q89 0 180 -17l54 97q10 16 28 16q5 0 18 -6t31 -15.5t33 -18.5t31.5 -18.5t19.5 -11.5 q16 -10 16 -27zM1344 704q0 -139 -79 -253.5t-209 -164.5l280 502q8 -45 8 -84zM1792 576q0 -35 -20 -69q-39 -64 -109 -145q-150 -172 -347.5 -267t-419.5 -95l74 132q212 18 392.5 137t301.5 307q-115 179 -282 294l63 112q95 -64 182.5 -153t144.5 -184q20 -34 20 -69z " />
+<glyph unicode="&#xf071;" horiz-adv-x="1792" d="M1024 161v190q0 14 -9.5 23.5t-22.5 9.5h-192q-13 0 -22.5 -9.5t-9.5 -23.5v-190q0 -14 9.5 -23.5t22.5 -9.5h192q13 0 22.5 9.5t9.5 23.5zM1022 535l18 459q0 12 -10 19q-13 11 -24 11h-220q-11 0 -24 -11q-10 -7 -10 -21l17 -457q0 -10 10 -16.5t24 -6.5h185 q14 0 23.5 6.5t10.5 16.5zM1008 1469l768 -1408q35 -63 -2 -126q-17 -29 -46.5 -46t-63.5 -17h-1536q-34 0 -63.5 17t-46.5 46q-37 63 -2 126l768 1408q17 31 47 49t65 18t65 -18t47 -49z" />
+<glyph unicode="&#xf072;" horiz-adv-x="1408" d="M1376 1376q44 -52 12 -148t-108 -172l-161 -161l160 -696q5 -19 -12 -33l-128 -96q-7 -6 -19 -6q-4 0 -7 1q-15 3 -21 16l-279 508l-259 -259l53 -194q5 -17 -8 -31l-96 -96q-9 -9 -23 -9h-2q-15 2 -24 13l-189 252l-252 189q-11 7 -13 23q-1 13 9 25l96 97q9 9 23 9 q6 0 8 -1l194 -53l259 259l-508 279q-14 8 -17 24q-2 16 9 27l128 128q14 13 30 8l665 -159l160 160q76 76 172 108t148 -12z" />
+<glyph unicode="&#xf073;" horiz-adv-x="1664" d="M128 -128h288v288h-288v-288zM480 -128h320v288h-320v-288zM128 224h288v320h-288v-320zM480 224h320v320h-320v-320zM128 608h288v288h-288v-288zM864 -128h320v288h-320v-288zM480 608h320v288h-320v-288zM1248 -128h288v288h-288v-288zM864 224h320v320h-320v-320z M512 1088v288q0 13 -9.5 22.5t-22.5 9.5h-64q-13 0 -22.5 -9.5t-9.5 -22.5v-288q0 -13 9.5 -22.5t22.5 -9.5h64q13 0 22.5 9.5t9.5 22.5zM1248 224h288v320h-288v-320zM864 608h320v288h-320v-288zM1248 608h288v288h-288v-288zM1280 1088v288q0 13 -9.5 22.5t-22.5 9.5h-64 q-13 0 -22.5 -9.5t-9.5 -22.5v-288q0 -13 9.5 -22.5t22.5 -9.5h64q13 0 22.5 9.5t9.5 22.5zM1664 1152v-1280q0 -52 -38 -90t-90 -38h-1408q-52 0 -90 38t-38 90v1280q0 52 38 90t90 38h128v96q0 66 47 113t113 47h64q66 0 113 -47t47 -113v-96h384v96q0 66 47 113t113 47 h64q66 0 113 -47t47 -113v-96h128q52 0 90 -38t38 -90z" />
+<glyph unicode="&#xf074;" horiz-adv-x="1792" d="M666 1055q-60 -92 -137 -273q-22 45 -37 72.5t-40.5 63.5t-51 56.5t-63 35t-81.5 14.5h-224q-14 0 -23 9t-9 23v192q0 14 9 23t23 9h224q250 0 410 -225zM1792 256q0 -14 -9 -23l-320 -320q-9 -9 -23 -9q-13 0 -22.5 9.5t-9.5 22.5v192q-32 0 -85 -0.5t-81 -1t-73 1 t-71 5t-64 10.5t-63 18.5t-58 28.5t-59 40t-55 53.5t-56 69.5q59 93 136 273q22 -45 37 -72.5t40.5 -63.5t51 -56.5t63 -35t81.5 -14.5h256v192q0 14 9 23t23 9q12 0 24 -10l319 -319q9 -9 9 -23zM1792 1152q0 -14 -9 -23l-320 -320q-9 -9 -23 -9q-13 0 -22.5 9.5t-9.5 22.5 v192h-256q-48 0 -87 -15t-69 -45t-51 -61.5t-45 -77.5q-32 -62 -78 -171q-29 -66 -49.5 -111t-54 -105t-64 -100t-74 -83t-90 -68.5t-106.5 -42t-128 -16.5h-224q-14 0 -23 9t-9 23v192q0 14 9 23t23 9h224q48 0 87 15t69 45t51 61.5t45 77.5q32 62 78 171q29 66 49.5 111 t54 105t64 100t74 83t90 68.5t106.5 42t128 16.5h256v192q0 14 9 23t23 9q12 0 24 -10l319 -319q9 -9 9 -23z" />
+<glyph unicode="&#xf075;" horiz-adv-x="1792" d="M1792 640q0 -174 -120 -321.5t-326 -233t-450 -85.5q-70 0 -145 8q-198 -175 -460 -242q-49 -14 -114 -22q-17 -2 -30.5 9t-17.5 29v1q-3 4 -0.5 12t2 10t4.5 9.5l6 9t7 8.5t8 9q7 8 31 34.5t34.5 38t31 39.5t32.5 51t27 59t26 76q-157 89 -247.5 220t-90.5 281 q0 130 71 248.5t191 204.5t286 136.5t348 50.5q244 0 450 -85.5t326 -233t120 -321.5z" />
+<glyph unicode="&#xf076;" d="M1536 704v-128q0 -201 -98.5 -362t-274 -251.5t-395.5 -90.5t-395.5 90.5t-274 251.5t-98.5 362v128q0 26 19 45t45 19h384q26 0 45 -19t19 -45v-128q0 -52 23.5 -90t53.5 -57t71 -30t64 -13t44 -2t44 2t64 13t71 30t53.5 57t23.5 90v128q0 26 19 45t45 19h384 q26 0 45 -19t19 -45zM512 1344v-384q0 -26 -19 -45t-45 -19h-384q-26 0 -45 19t-19 45v384q0 26 19 45t45 19h384q26 0 45 -19t19 -45zM1536 1344v-384q0 -26 -19 -45t-45 -19h-384q-26 0 -45 19t-19 45v384q0 26 19 45t45 19h384q26 0 45 -19t19 -45z" />
+<glyph unicode="&#xf077;" horiz-adv-x="1664" d="M1611 320q0 -53 -37 -90l-75 -75q-38 -38 -91 -38q-54 0 -90 38l-486 485l-486 -485q-36 -38 -90 -38t-90 38l-75 75q-38 36 -38 90q0 53 38 91l651 651q37 37 90 37q52 0 91 -37l650 -651q38 -38 38 -91z" />
+<glyph unicode="&#xf078;" horiz-adv-x="1664" d="M1611 832q0 -53 -37 -90l-651 -651q-38 -38 -91 -38q-54 0 -90 38l-651 651q-38 36 -38 90q0 53 38 91l74 75q39 37 91 37q53 0 90 -37l486 -486l486 486q37 37 90 37q52 0 91 -37l75 -75q37 -39 37 -91z" />
+<glyph unicode="&#xf079;" horiz-adv-x="1920" d="M1280 32q0 -13 -9.5 -22.5t-22.5 -9.5h-960q-8 0 -13.5 2t-9 7t-5.5 8t-3 11.5t-1 11.5v13v11v160v416h-192q-26 0 -45 19t-19 45q0 24 15 41l320 384q19 22 49 22t49 -22l320 -384q15 -17 15 -41q0 -26 -19 -45t-45 -19h-192v-384h576q16 0 25 -11l160 -192q7 -11 7 -21 zM1920 448q0 -24 -15 -41l-320 -384q-20 -23 -49 -23t-49 23l-320 384q-15 17 -15 41q0 26 19 45t45 19h192v384h-576q-16 0 -25 12l-160 192q-7 9 -7 20q0 13 9.5 22.5t22.5 9.5h960q8 0 13.5 -2t9 -7t5.5 -8t3 -11.5t1 -11.5v-13v-11v-160v-416h192q26 0 45 -19t19 -45z " />
+<glyph unicode="&#xf07a;" horiz-adv-x="1664" d="M640 0q0 -53 -37.5 -90.5t-90.5 -37.5t-90.5 37.5t-37.5 90.5t37.5 90.5t90.5 37.5t90.5 -37.5t37.5 -90.5zM1536 0q0 -53 -37.5 -90.5t-90.5 -37.5t-90.5 37.5t-37.5 90.5t37.5 90.5t90.5 37.5t90.5 -37.5t37.5 -90.5zM1664 1088v-512q0 -24 -16 -42.5t-41 -21.5 l-1044 -122q1 -7 4.5 -21.5t6 -26.5t2.5 -22q0 -16 -24 -64h920q26 0 45 -19t19 -45t-19 -45t-45 -19h-1024q-26 0 -45 19t-19 45q0 14 11 39.5t29.5 59.5t20.5 38l-177 823h-204q-26 0 -45 19t-19 45t19 45t45 19h256q16 0 28.5 -6.5t20 -15.5t13 -24.5t7.5 -26.5 t5.5 -29.5t4.5 -25.5h1201q26 0 45 -19t19 -45z" />
+<glyph unicode="&#xf07b;" horiz-adv-x="1664" d="M1664 928v-704q0 -92 -66 -158t-158 -66h-1216q-92 0 -158 66t-66 158v960q0 92 66 158t158 66h320q92 0 158 -66t66 -158v-32h672q92 0 158 -66t66 -158z" />
+<glyph unicode="&#xf07c;" horiz-adv-x="1920" d="M1879 584q0 -31 -31 -66l-336 -396q-43 -51 -120.5 -86.5t-143.5 -35.5h-1088q-34 0 -60.5 13t-26.5 43q0 31 31 66l336 396q43 51 120.5 86.5t143.5 35.5h1088q34 0 60.5 -13t26.5 -43zM1536 928v-160h-832q-94 0 -197 -47.5t-164 -119.5l-337 -396l-5 -6q0 4 -0.5 12.5 t-0.5 12.5v960q0 92 66 158t158 66h320q92 0 158 -66t66 -158v-32h544q92 0 158 -66t66 -158z" />
+<glyph unicode="&#xf07d;" horiz-adv-x="768" d="M704 1216q0 -26 -19 -45t-45 -19h-128v-1024h128q26 0 45 -19t19 -45t-19 -45l-256 -256q-19 -19 -45 -19t-45 19l-256 256q-19 19 -19 45t19 45t45 19h128v1024h-128q-26 0 -45 19t-19 45t19 45l256 256q19 19 45 19t45 -19l256 -256q19 -19 19 -45z" />
+<glyph unicode="&#xf07e;" horiz-adv-x="1792" d="M1792 640q0 -26 -19 -45l-256 -256q-19 -19 -45 -19t-45 19t-19 45v128h-1024v-128q0 -26 -19 -45t-45 -19t-45 19l-256 256q-19 19 -19 45t19 45l256 256q19 19 45 19t45 -19t19 -45v-128h1024v128q0 26 19 45t45 19t45 -19l256 -256q19 -19 19 -45z" />
+<glyph unicode="&#xf080;" horiz-adv-x="1920" d="M512 512v-384h-256v384h256zM896 1024v-896h-256v896h256zM1280 768v-640h-256v640h256zM1664 1152v-1024h-256v1024h256zM1792 32v1216q0 13 -9.5 22.5t-22.5 9.5h-1600q-13 0 -22.5 -9.5t-9.5 -22.5v-1216q0 -13 9.5 -22.5t22.5 -9.5h1600q13 0 22.5 9.5t9.5 22.5z M1920 1248v-1216q0 -66 -47 -113t-113 -47h-1600q-66 0 -113 47t-47 113v1216q0 66 47 113t113 47h1600q66 0 113 -47t47 -113z" />
+<glyph unicode="&#xf081;" d="M1280 926q-56 -25 -121 -34q68 40 93 117q-65 -38 -134 -51q-61 66 -153 66q-87 0 -148.5 -61.5t-61.5 -148.5q0 -29 5 -48q-129 7 -242 65t-192 155q-29 -50 -29 -106q0 -114 91 -175q-47 1 -100 26v-2q0 -75 50 -133.5t123 -72.5q-29 -8 -51 -8q-13 0 -39 4 q21 -63 74.5 -104t121.5 -42q-116 -90 -261 -90q-26 0 -50 3q148 -94 322 -94q112 0 210 35.5t168 95t120.5 137t75 162t24.5 168.5q0 18 -1 27q63 45 105 109zM1536 1120v-960q0 -119 -84.5 -203.5t-203.5 -84.5h-960q-119 0 -203.5 84.5t-84.5 203.5v960q0 119 84.5 203.5 t203.5 84.5h960q119 0 203.5 -84.5t84.5 -203.5z" />
+<glyph unicode="&#xf082;" d="M1307 618l23 219h-198v109q0 49 15.5 68.5t71.5 19.5h110v219h-175q-152 0 -218 -72t-66 -213v-131h-131v-219h131v-635h262v635h175zM1536 1120v-960q0 -119 -84.5 -203.5t-203.5 -84.5h-960q-119 0 -203.5 84.5t-84.5 203.5v960q0 119 84.5 203.5t203.5 84.5h960 q119 0 203.5 -84.5t84.5 -203.5z" />
+<glyph unicode="&#xf083;" horiz-adv-x="1792" d="M928 704q0 14 -9 23t-23 9q-66 0 -113 -47t-47 -113q0 -14 9 -23t23 -9t23 9t9 23q0 40 28 68t68 28q14 0 23 9t9 23zM1152 574q0 -106 -75 -181t-181 -75t-181 75t-75 181t75 181t181 75t181 -75t75 -181zM128 0h1536v128h-1536v-128zM1280 574q0 159 -112.5 271.5 t-271.5 112.5t-271.5 -112.5t-112.5 -271.5t112.5 -271.5t271.5 -112.5t271.5 112.5t112.5 271.5zM256 1216h384v128h-384v-128zM128 1024h1536v118v138h-828l-64 -128h-644v-128zM1792 1280v-1280q0 -53 -37.5 -90.5t-90.5 -37.5h-1536q-53 0 -90.5 37.5t-37.5 90.5v1280 q0 53 37.5 90.5t90.5 37.5h1536q53 0 90.5 -37.5t37.5 -90.5z" />
+<glyph unicode="&#xf084;" horiz-adv-x="1792" d="M832 1024q0 80 -56 136t-136 56t-136 -56t-56 -136q0 -42 19 -83q-41 19 -83 19q-80 0 -136 -56t-56 -136t56 -136t136 -56t136 56t56 136q0 42 -19 83q41 -19 83 -19q80 0 136 56t56 136zM1683 320q0 -17 -49 -66t-66 -49q-9 0 -28.5 16t-36.5 33t-38.5 40t-24.5 26 l-96 -96l220 -220q28 -28 28 -68q0 -42 -39 -81t-81 -39q-40 0 -68 28l-671 671q-176 -131 -365 -131q-163 0 -265.5 102.5t-102.5 265.5q0 160 95 313t248 248t313 95q163 0 265.5 -102.5t102.5 -265.5q0 -189 -131 -365l355 -355l96 96q-3 3 -26 24.5t-40 38.5t-33 36.5 t-16 28.5q0 17 49 66t66 49q13 0 23 -10q6 -6 46 -44.5t82 -79.5t86.5 -86t73 -78t28.5 -41z" />
+<glyph unicode="&#xf085;" horiz-adv-x="1920" d="M896 640q0 106 -75 181t-181 75t-181 -75t-75 -181t75 -181t181 -75t181 75t75 181zM1664 128q0 52 -38 90t-90 38t-90 -38t-38 -90q0 -53 37.5 -90.5t90.5 -37.5t90.5 37.5t37.5 90.5zM1664 1152q0 52 -38 90t-90 38t-90 -38t-38 -90q0 -53 37.5 -90.5t90.5 -37.5 t90.5 37.5t37.5 90.5zM1280 731v-185q0 -10 -7 -19.5t-16 -10.5l-155 -24q-11 -35 -32 -76q34 -48 90 -115q7 -10 7 -20q0 -12 -7 -19q-23 -30 -82.5 -89.5t-78.5 -59.5q-11 0 -21 7l-115 90q-37 -19 -77 -31q-11 -108 -23 -155q-7 -24 -30 -24h-186q-11 0 -20 7.5t-10 17.5 l-23 153q-34 10 -75 31l-118 -89q-7 -7 -20 -7q-11 0 -21 8q-144 133 -144 160q0 9 7 19q10 14 41 53t47 61q-23 44 -35 82l-152 24q-10 1 -17 9.5t-7 19.5v185q0 10 7 19.5t16 10.5l155 24q11 35 32 76q-34 48 -90 115q-7 11 -7 20q0 12 7 20q22 30 82 89t79 59q11 0 21 -7 l115 -90q34 18 77 32q11 108 23 154q7 24 30 24h186q11 0 20 -7.5t10 -17.5l23 -153q34 -10 75 -31l118 89q8 7 20 7q11 0 21 -8q144 -133 144 -160q0 -9 -7 -19q-12 -16 -42 -54t-45 -60q23 -48 34 -82l152 -23q10 -2 17 -10.5t7 -19.5zM1920 198v-140q0 -16 -149 -31 q-12 -27 -30 -52q51 -113 51 -138q0 -4 -4 -7q-122 -71 -124 -71q-8 0 -46 47t-52 68q-20 -2 -30 -2t-30 2q-14 -21 -52 -68t-46 -47q-2 0 -124 71q-4 3 -4 7q0 25 51 138q-18 25 -30 52q-149 15 -149 31v140q0 16 149 31q13 29 30 52q-51 113 -51 138q0 4 4 7q4 2 35 20 t59 34t30 16q8 0 46 -46.5t52 -67.5q20 2 30 2t30 -2q51 71 92 112l6 2q4 0 124 -70q4 -3 4 -7q0 -25 -51 -138q17 -23 30 -52q149 -15 149 -31zM1920 1222v-140q0 -16 -149 -31q-12 -27 -30 -52q51 -113 51 -138q0 -4 -4 -7q-122 -71 -124 -71q-8 0 -46 47t-52 68 q-20 -2 -30 -2t-30 2q-14 -21 -52 -68t-46 -47q-2 0 -124 71q-4 3 -4 7q0 25 51 138q-18 25 -30 52q-149 15 -149 31v140q0 16 149 31q13 29 30 52q-51 113 -51 138q0 4 4 7q4 2 35 20t59 34t30 16q8 0 46 -46.5t52 -67.5q20 2 30 2t30 -2q51 71 92 112l6 2q4 0 124 -70 q4 -3 4 -7q0 -25 -51 -138q17 -23 30 -52q149 -15 149 -31z" />
+<glyph unicode="&#xf086;" horiz-adv-x="1792" d="M1408 768q0 -139 -94 -257t-256.5 -186.5t-353.5 -68.5q-86 0 -176 16q-124 -88 -278 -128q-36 -9 -86 -16h-3q-11 0 -20.5 8t-11.5 21q-1 3 -1 6.5t0.5 6.5t2 6l2.5 5t3.5 5.5t4 5t4.5 5t4 4.5q5 6 23 25t26 29.5t22.5 29t25 38.5t20.5 44q-124 72 -195 177t-71 224 q0 139 94 257t256.5 186.5t353.5 68.5t353.5 -68.5t256.5 -186.5t94 -257zM1792 512q0 -120 -71 -224.5t-195 -176.5q10 -24 20.5 -44t25 -38.5t22.5 -29t26 -29.5t23 -25q1 -1 4 -4.5t4.5 -5t4 -5t3.5 -5.5l2.5 -5t2 -6t0.5 -6.5t-1 -6.5q-3 -14 -13 -22t-22 -7 q-50 7 -86 16q-154 40 -278 128q-90 -16 -176 -16q-271 0 -472 132q58 -4 88 -4q161 0 309 45t264 129q125 92 192 212t67 254q0 77 -23 152q129 -71 204 -178t75 -230z" />
+<glyph unicode="&#xf087;" d="M256 192q0 26 -19 45t-45 19t-45 -19t-19 -45t19 -45t45 -19t45 19t19 45zM1408 768q0 51 -39 89.5t-89 38.5h-352q0 58 48 159.5t48 160.5q0 98 -32 145t-128 47q-26 -26 -38 -85t-30.5 -125.5t-59.5 -109.5q-22 -23 -77 -91q-4 -5 -23 -30t-31.5 -41t-34.5 -42.5 t-40 -44t-38.5 -35.5t-40 -27t-35.5 -9h-32v-640h32q13 0 31.5 -3t33 -6.5t38 -11t35 -11.5t35.5 -12.5t29 -10.5q211 -73 342 -73h121q192 0 192 167q0 26 -5 56q30 16 47.5 52.5t17.5 73.5t-18 69q53 50 53 119q0 25 -10 55.5t-25 47.5q32 1 53.5 47t21.5 81zM1536 769 q0 -89 -49 -163q9 -33 9 -69q0 -77 -38 -144q3 -21 3 -43q0 -101 -60 -178q1 -139 -85 -219.5t-227 -80.5h-36h-93q-96 0 -189.5 22.5t-216.5 65.5q-116 40 -138 40h-288q-53 0 -90.5 37.5t-37.5 90.5v640q0 53 37.5 90.5t90.5 37.5h274q36 24 137 155q58 75 107 128 q24 25 35.5 85.5t30.5 126.5t62 108q39 37 90 37q84 0 151 -32.5t102 -101.5t35 -186q0 -93 -48 -192h176q104 0 180 -76t76 -179z" />
+<glyph unicode="&#xf088;" d="M256 1088q0 26 -19 45t-45 19t-45 -19t-19 -45t19 -45t45 -19t45 19t19 45zM1408 512q0 35 -21.5 81t-53.5 47q15 17 25 47.5t10 55.5q0 69 -53 119q18 32 18 69t-17.5 73.5t-47.5 52.5q5 30 5 56q0 85 -49 126t-136 41h-128q-131 0 -342 -73q-5 -2 -29 -10.5 t-35.5 -12.5t-35 -11.5t-38 -11t-33 -6.5t-31.5 -3h-32v-640h32q16 0 35.5 -9t40 -27t38.5 -35.5t40 -44t34.5 -42.5t31.5 -41t23 -30q55 -68 77 -91q41 -43 59.5 -109.5t30.5 -125.5t38 -85q96 0 128 47t32 145q0 59 -48 160.5t-48 159.5h352q50 0 89 38.5t39 89.5z M1536 511q0 -103 -76 -179t-180 -76h-176q48 -99 48 -192q0 -118 -35 -186q-35 -69 -102 -101.5t-151 -32.5q-51 0 -90 37q-34 33 -54 82t-25.5 90.5t-17.5 84.5t-31 64q-48 50 -107 127q-101 131 -137 155h-274q-53 0 -90.5 37.5t-37.5 90.5v640q0 53 37.5 90.5t90.5 37.5 h288q22 0 138 40q128 44 223 66t200 22h112q140 0 226.5 -79t85.5 -216v-5q60 -77 60 -178q0 -22 -3 -43q38 -67 38 -144q0 -36 -9 -69q49 -74 49 -163z" />
+<glyph unicode="&#xf089;" horiz-adv-x="896" d="M832 1504v-1339l-449 -236q-22 -12 -40 -12q-21 0 -31.5 14.5t-10.5 35.5q0 6 2 20l86 500l-364 354q-25 27 -25 48q0 37 56 46l502 73l225 455q19 41 49 41z" />
+<glyph unicode="&#xf08a;" horiz-adv-x="1792" d="M1664 940q0 81 -21.5 143t-55 98.5t-81.5 59.5t-94 31t-98 8t-112 -25.5t-110.5 -64t-86.5 -72t-60 -61.5q-18 -22 -49 -22t-49 22q-24 28 -60 61.5t-86.5 72t-110.5 64t-112 25.5t-98 -8t-94 -31t-81.5 -59.5t-55 -98.5t-21.5 -143q0 -168 187 -355l581 -560l580 559 q188 188 188 356zM1792 940q0 -221 -229 -450l-623 -600q-18 -18 -44 -18t-44 18l-624 602q-10 8 -27.5 26t-55.5 65.5t-68 97.5t-53.5 121t-23.5 138q0 220 127 344t351 124q62 0 126.5 -21.5t120 -58t95.5 -68.5t76 -68q36 36 76 68t95.5 68.5t120 58t126.5 21.5 q224 0 351 -124t127 -344z" />
+<glyph unicode="&#xf08b;" horiz-adv-x="1664" d="M640 96q0 -4 1 -20t0.5 -26.5t-3 -23.5t-10 -19.5t-20.5 -6.5h-320q-119 0 -203.5 84.5t-84.5 203.5v704q0 119 84.5 203.5t203.5 84.5h320q13 0 22.5 -9.5t9.5 -22.5q0 -4 1 -20t0.5 -26.5t-3 -23.5t-10 -19.5t-20.5 -6.5h-320q-66 0 -113 -47t-47 -113v-704 q0 -66 47 -113t113 -47h288h11h13t11.5 -1t11.5 -3t8 -5.5t7 -9t2 -13.5zM1568 640q0 -26 -19 -45l-544 -544q-19 -19 -45 -19t-45 19t-19 45v288h-448q-26 0 -45 19t-19 45v384q0 26 19 45t45 19h448v288q0 26 19 45t45 19t45 -19l544 -544q19 -19 19 -45z" />
+<glyph unicode="&#xf08c;" d="M237 122h231v694h-231v-694zM483 1030q-1 52 -36 86t-93 34t-94.5 -34t-36.5 -86q0 -51 35.5 -85.5t92.5 -34.5h1q59 0 95 34.5t36 85.5zM1068 122h231v398q0 154 -73 233t-193 79q-136 0 -209 -117h2v101h-231q3 -66 0 -694h231v388q0 38 7 56q15 35 45 59.5t74 24.5 q116 0 116 -157v-371zM1536 1120v-960q0 -119 -84.5 -203.5t-203.5 -84.5h-960q-119 0 -203.5 84.5t-84.5 203.5v960q0 119 84.5 203.5t203.5 84.5h960q119 0 203.5 -84.5t84.5 -203.5z" />
+<glyph unicode="&#xf08d;" horiz-adv-x="1152" d="M480 672v448q0 14 -9 23t-23 9t-23 -9t-9 -23v-448q0 -14 9 -23t23 -9t23 9t9 23zM1152 320q0 -26 -19 -45t-45 -19h-429l-51 -483q-2 -12 -10.5 -20.5t-20.5 -8.5h-1q-27 0 -32 27l-76 485h-404q-26 0 -45 19t-19 45q0 123 78.5 221.5t177.5 98.5v512q-52 0 -90 38 t-38 90t38 90t90 38h640q52 0 90 -38t38 -90t-38 -90t-90 -38v-512q99 0 177.5 -98.5t78.5 -221.5z" />
+<glyph unicode="&#xf08e;" horiz-adv-x="1792" d="M1408 608v-320q0 -119 -84.5 -203.5t-203.5 -84.5h-832q-119 0 -203.5 84.5t-84.5 203.5v832q0 119 84.5 203.5t203.5 84.5h704q14 0 23 -9t9 -23v-64q0 -14 -9 -23t-23 -9h-704q-66 0 -113 -47t-47 -113v-832q0 -66 47 -113t113 -47h832q66 0 113 47t47 113v320 q0 14 9 23t23 9h64q14 0 23 -9t9 -23zM1792 1472v-512q0 -26 -19 -45t-45 -19t-45 19l-176 176l-652 -652q-10 -10 -23 -10t-23 10l-114 114q-10 10 -10 23t10 23l652 652l-176 176q-19 19 -19 45t19 45t45 19h512q26 0 45 -19t19 -45z" />
+<glyph unicode="&#xf090;" d="M1184 640q0 -26 -19 -45l-544 -544q-19 -19 -45 -19t-45 19t-19 45v288h-448q-26 0 -45 19t-19 45v384q0 26 19 45t45 19h448v288q0 26 19 45t45 19t45 -19l544 -544q19 -19 19 -45zM1536 992v-704q0 -119 -84.5 -203.5t-203.5 -84.5h-320q-13 0 -22.5 9.5t-9.5 22.5 q0 4 -1 20t-0.5 26.5t3 23.5t10 19.5t20.5 6.5h320q66 0 113 47t47 113v704q0 66 -47 113t-113 47h-288h-11h-13t-11.5 1t-11.5 3t-8 5.5t-7 9t-2 13.5q0 4 -1 20t-0.5 26.5t3 23.5t10 19.5t20.5 6.5h320q119 0 203.5 -84.5t84.5 -203.5z" />
+<glyph unicode="&#xf091;" horiz-adv-x="1664" d="M458 653q-74 162 -74 371h-256v-96q0 -78 94.5 -162t235.5 -113zM1536 928v96h-256q0 -209 -74 -371q141 29 235.5 113t94.5 162zM1664 1056v-128q0 -71 -41.5 -143t-112 -130t-173 -97.5t-215.5 -44.5q-42 -54 -95 -95q-38 -34 -52.5 -72.5t-14.5 -89.5q0 -54 30.5 -91 t97.5 -37q75 0 133.5 -45.5t58.5 -114.5v-64q0 -14 -9 -23t-23 -9h-832q-14 0 -23 9t-9 23v64q0 69 58.5 114.5t133.5 45.5q67 0 97.5 37t30.5 91q0 51 -14.5 89.5t-52.5 72.5q-53 41 -95 95q-113 5 -215.5 44.5t-173 97.5t-112 130t-41.5 143v128q0 40 28 68t68 28h288v96 q0 66 47 113t113 47h576q66 0 113 -47t47 -113v-96h288q40 0 68 -28t28 -68z" />
+<glyph unicode="&#xf092;" d="M394 184q-8 -9 -20 3q-13 11 -4 19q8 9 20 -3q12 -11 4 -19zM352 245q9 -12 0 -19q-8 -6 -17 7t0 18q9 7 17 -6zM291 305q-5 -7 -13 -2q-10 5 -7 12q3 5 13 2q10 -5 7 -12zM322 271q-6 -7 -16 3q-9 11 -2 16q6 6 16 -3q9 -11 2 -16zM451 159q-4 -12 -19 -6q-17 4 -13 15 t19 7q16 -5 13 -16zM514 154q0 -11 -16 -11q-17 -2 -17 11q0 11 16 11q17 2 17 -11zM572 164q2 -10 -14 -14t-18 8t14 15q16 2 18 -9zM1536 1120v-960q0 -119 -84.5 -203.5t-203.5 -84.5h-224q-16 0 -24.5 1t-19.5 5t-16 14.5t-5 27.5v239q0 97 -52 142q57 6 102.5 18t94 39 t81 66.5t53 105t20.5 150.5q0 121 -79 206q37 91 -8 204q-28 9 -81 -11t-92 -44l-38 -24q-93 26 -192 26t-192 -26q-16 11 -42.5 27t-83.5 38.5t-86 13.5q-44 -113 -7 -204q-79 -85 -79 -206q0 -85 20.5 -150t52.5 -105t80.5 -67t94 -39t102.5 -18q-40 -36 -49 -103 q-21 -10 -45 -15t-57 -5t-65.5 21.5t-55.5 62.5q-19 32 -48.5 52t-49.5 24l-20 3q-21 0 -29 -4.5t-5 -11.5t9 -14t13 -12l7 -5q22 -10 43.5 -38t31.5 -51l10 -23q13 -38 44 -61.5t67 -30t69.5 -7t55.5 3.5l23 4q0 -38 0.5 -103t0.5 -68q0 -22 -11 -33.5t-22 -13t-33 -1.5 h-224q-119 0 -203.5 84.5t-84.5 203.5v960q0 119 84.5 203.5t203.5 84.5h960q119 0 203.5 -84.5t84.5 -203.5z" />
+<glyph unicode="&#xf093;" horiz-adv-x="1664" d="M1280 64q0 26 -19 45t-45 19t-45 -19t-19 -45t19 -45t45 -19t45 19t19 45zM1536 64q0 26 -19 45t-45 19t-45 -19t-19 -45t19 -45t45 -19t45 19t19 45zM1664 288v-320q0 -40 -28 -68t-68 -28h-1472q-40 0 -68 28t-28 68v320q0 40 28 68t68 28h427q21 -56 70.5 -92 t110.5 -36h256q61 0 110.5 36t70.5 92h427q40 0 68 -28t28 -68zM1339 936q-17 -40 -59 -40h-256v-448q0 -26 -19 -45t-45 -19h-256q-26 0 -45 19t-19 45v448h-256q-42 0 -59 40q-17 39 14 69l448 448q18 19 45 19t45 -19l448 -448q31 -30 14 -69z" />
+<glyph unicode="&#xf094;" d="M1407 710q0 44 -7 113.5t-18 96.5q-12 30 -17 44t-9 36.5t-4 48.5q0 23 5 68.5t5 67.5q0 37 -10 55q-4 1 -13 1q-19 0 -58 -4.5t-59 -4.5q-60 0 -176 24t-175 24q-43 0 -94.5 -11.5t-85 -23.5t-89.5 -34q-137 -54 -202 -103q-96 -73 -159.5 -189.5t-88 -236t-24.5 -248.5 q0 -40 12.5 -120t12.5 -121q0 -23 -11 -66.5t-11 -65.5t12 -36.5t34 -14.5q24 0 72.5 11t73.5 11q57 0 169.5 -15.5t169.5 -15.5q181 0 284 36q129 45 235.5 152.5t166 245.5t59.5 275zM1535 712q0 -165 -70 -327.5t-196 -288t-281 -180.5q-124 -44 -326 -44 q-57 0 -170 14.5t-169 14.5q-24 0 -72.5 -14.5t-73.5 -14.5q-73 0 -123.5 55.5t-50.5 128.5q0 24 11 68t11 67q0 40 -12.5 120.5t-12.5 121.5q0 111 18 217.5t54.5 209.5t100.5 194t150 156q78 59 232 120q194 78 316 78q60 0 175.5 -24t173.5 -24q19 0 57 5t58 5 q81 0 118 -50.5t37 -134.5q0 -23 -5 -68t-5 -68q0 -10 1 -18.5t3 -17t4 -13.5t6.5 -16t6.5 -17q16 -40 25 -118.5t9 -136.5z" />
+<glyph unicode="&#xf095;" horiz-adv-x="1408" d="M1408 296q0 -27 -10 -70.5t-21 -68.5q-21 -50 -122 -106q-94 -51 -186 -51q-27 0 -52.5 3.5t-57.5 12.5t-47.5 14.5t-55.5 20.5t-49 18q-98 35 -175 83q-128 79 -264.5 215.5t-215.5 264.5q-48 77 -83 175q-3 9 -18 49t-20.5 55.5t-14.5 47.5t-12.5 57.5t-3.5 52.5 q0 92 51 186q56 101 106 122q25 11 68.5 21t70.5 10q14 0 21 -3q18 -6 53 -76q11 -19 30 -54t35 -63.5t31 -53.5q3 -4 17.5 -25t21.5 -35.5t7 -28.5q0 -20 -28.5 -50t-62 -55t-62 -53t-28.5 -46q0 -9 5 -22.5t8.5 -20.5t14 -24t11.5 -19q76 -137 174 -235t235 -174 q2 -1 19 -11.5t24 -14t20.5 -8.5t22.5 -5q18 0 46 28.5t53 62t55 62t50 28.5q14 0 28.5 -7t35.5 -21.5t25 -17.5q25 -15 53.5 -31t63.5 -35t54 -30q70 -35 76 -53q3 -7 3 -21z" />
+<glyph unicode="&#xf096;" horiz-adv-x="1408" d="M1120 1280h-832q-66 0 -113 -47t-47 -113v-832q0 -66 47 -113t113 -47h832q66 0 113 47t47 113v832q0 66 -47 113t-113 47zM1408 1120v-832q0 -119 -84.5 -203.5t-203.5 -84.5h-832q-119 0 -203.5 84.5t-84.5 203.5v832q0 119 84.5 203.5t203.5 84.5h832 q119 0 203.5 -84.5t84.5 -203.5z" />
+<glyph unicode="&#xf097;" horiz-adv-x="1280" d="M1152 1280h-1024v-1242l423 406l89 85l89 -85l423 -406v1242zM1164 1408q23 0 44 -9q33 -13 52.5 -41t19.5 -62v-1289q0 -34 -19.5 -62t-52.5 -41q-19 -8 -44 -8q-48 0 -83 32l-441 424l-441 -424q-36 -33 -83 -33q-23 0 -44 9q-33 13 -52.5 41t-19.5 62v1289 q0 34 19.5 62t52.5 41q21 9 44 9h1048z" />
+<glyph unicode="&#xf098;" d="M1280 343q0 11 -2 16q-3 8 -38.5 29.5t-88.5 49.5l-53 29q-5 3 -19 13t-25 15t-21 5q-18 0 -47 -32.5t-57 -65.5t-44 -33q-7 0 -16.5 3.5t-15.5 6.5t-17 9.5t-14 8.5q-99 55 -170.5 126.5t-126.5 170.5q-2 3 -8.5 14t-9.5 17t-6.5 15.5t-3.5 16.5q0 13 20.5 33.5t45 38.5 t45 39.5t20.5 36.5q0 10 -5 21t-15 25t-13 19q-3 6 -15 28.5t-25 45.5t-26.5 47.5t-25 40.5t-16.5 18t-16 2q-48 0 -101 -22q-46 -21 -80 -94.5t-34 -130.5q0 -16 2.5 -34t5 -30.5t9 -33t10 -29.5t12.5 -33t11 -30q60 -164 216.5 -320.5t320.5 -216.5q6 -2 30 -11t33 -12.5 t29.5 -10t33 -9t30.5 -5t34 -2.5q57 0 130.5 34t94.5 80q22 53 22 101zM1536 1120v-960q0 -119 -84.5 -203.5t-203.5 -84.5h-960q-119 0 -203.5 84.5t-84.5 203.5v960q0 119 84.5 203.5t203.5 84.5h960q119 0 203.5 -84.5t84.5 -203.5z" />
+<glyph unicode="&#xf099;" horiz-adv-x="1664" d="M1620 1128q-67 -98 -162 -167q1 -14 1 -42q0 -130 -38 -259.5t-115.5 -248.5t-184.5 -210.5t-258 -146t-323 -54.5q-271 0 -496 145q35 -4 78 -4q225 0 401 138q-105 2 -188 64.5t-114 159.5q33 -5 61 -5q43 0 85 11q-112 23 -185.5 111.5t-73.5 205.5v4q68 -38 146 -41 q-66 44 -105 115t-39 154q0 88 44 163q121 -149 294.5 -238.5t371.5 -99.5q-8 38 -8 74q0 134 94.5 228.5t228.5 94.5q140 0 236 -102q109 21 205 78q-37 -115 -142 -178q93 10 186 50z" />
+<glyph unicode="&#xf09a;" horiz-adv-x="768" d="M511 980h257l-30 -284h-227v-824h-341v824h-170v284h170v171q0 182 86 275.5t283 93.5h227v-284h-142q-39 0 -62.5 -6.5t-34 -23.5t-13.5 -34.5t-3 -49.5v-142z" />
+<glyph unicode="&#xf09b;" d="M1536 640q0 -251 -146.5 -451.5t-378.5 -277.5q-27 -5 -39.5 7t-12.5 30v211q0 97 -52 142q57 6 102.5 18t94 39t81 66.5t53 105t20.5 150.5q0 121 -79 206q37 91 -8 204q-28 9 -81 -11t-92 -44l-38 -24q-93 26 -192 26t-192 -26q-16 11 -42.5 27t-83.5 38.5t-86 13.5 q-44 -113 -7 -204q-79 -85 -79 -206q0 -85 20.5 -150t52.5 -105t80.5 -67t94 -39t102.5 -18q-40 -36 -49 -103q-21 -10 -45 -15t-57 -5t-65.5 21.5t-55.5 62.5q-19 32 -48.5 52t-49.5 24l-20 3q-21 0 -29 -4.5t-5 -11.5t9 -14t13 -12l7 -5q22 -10 43.5 -38t31.5 -51l10 -23 q13 -38 44 -61.5t67 -30t69.5 -7t55.5 3.5l23 4q0 -38 0.5 -89t0.5 -54q0 -18 -13 -30t-40 -7q-232 77 -378.5 277.5t-146.5 451.5q0 209 103 385.5t279.5 279.5t385.5 103t385.5 -103t279.5 -279.5t103 -385.5z" />
+<glyph unicode="&#xf09c;" horiz-adv-x="1664" d="M1664 960v-256q0 -26 -19 -45t-45 -19h-64q-26 0 -45 19t-19 45v256q0 106 -75 181t-181 75t-181 -75t-75 -181v-192h96q40 0 68 -28t28 -68v-576q0 -40 -28 -68t-68 -28h-960q-40 0 -68 28t-28 68v576q0 40 28 68t68 28h672v192q0 185 131.5 316.5t316.5 131.5 t316.5 -131.5t131.5 -316.5z" />
+<glyph unicode="&#xf09d;" horiz-adv-x="1920" d="M1760 1408q66 0 113 -47t47 -113v-1216q0 -66 -47 -113t-113 -47h-1600q-66 0 -113 47t-47 113v1216q0 66 47 113t113 47h1600zM160 1280q-13 0 -22.5 -9.5t-9.5 -22.5v-224h1664v224q0 13 -9.5 22.5t-22.5 9.5h-1600zM1760 0q13 0 22.5 9.5t9.5 22.5v608h-1664v-608 q0 -13 9.5 -22.5t22.5 -9.5h1600zM256 128v128h256v-128h-256zM640 128v128h384v-128h-384z" />
+<glyph unicode="&#xf09e;" horiz-adv-x="1408" d="M384 192q0 -80 -56 -136t-136 -56t-136 56t-56 136t56 136t136 56t136 -56t56 -136zM896 69q2 -28 -17 -48q-18 -21 -47 -21h-135q-25 0 -43 16.5t-20 41.5q-22 229 -184.5 391.5t-391.5 184.5q-25 2 -41.5 20t-16.5 43v135q0 29 21 47q17 17 43 17h5q160 -13 306 -80.5 t259 -181.5q114 -113 181.5 -259t80.5 -306zM1408 67q2 -27 -18 -47q-18 -20 -46 -20h-143q-26 0 -44.5 17.5t-19.5 42.5q-12 215 -101 408.5t-231.5 336t-336 231.5t-408.5 102q-25 1 -42.5 19.5t-17.5 43.5v143q0 28 20 46q18 18 44 18h3q262 -13 501.5 -120t425.5 -294 q187 -186 294 -425.5t120 -501.5z" />
+<glyph unicode="&#xf0a0;" d="M1040 320q0 -33 -23.5 -56.5t-56.5 -23.5t-56.5 23.5t-23.5 56.5t23.5 56.5t56.5 23.5t56.5 -23.5t23.5 -56.5zM1296 320q0 -33 -23.5 -56.5t-56.5 -23.5t-56.5 23.5t-23.5 56.5t23.5 56.5t56.5 23.5t56.5 -23.5t23.5 -56.5zM1408 160v320q0 13 -9.5 22.5t-22.5 9.5 h-1216q-13 0 -22.5 -9.5t-9.5 -22.5v-320q0 -13 9.5 -22.5t22.5 -9.5h1216q13 0 22.5 9.5t9.5 22.5zM178 640h1180l-157 482q-4 13 -16 21.5t-26 8.5h-782q-14 0 -26 -8.5t-16 -21.5zM1536 480v-320q0 -66 -47 -113t-113 -47h-1216q-66 0 -113 47t-47 113v320q0 25 16 75 l197 606q17 53 63 86t101 33h782q55 0 101 -33t63 -86l197 -606q16 -50 16 -75z" />
+<glyph unicode="&#xf0a1;" horiz-adv-x="1792" d="M1664 896q53 0 90.5 -37.5t37.5 -90.5t-37.5 -90.5t-90.5 -37.5v-384q0 -52 -38 -90t-90 -38q-417 347 -812 380q-58 -19 -91 -66t-31 -100.5t40 -92.5q-20 -33 -23 -65.5t6 -58t33.5 -55t48 -50t61.5 -50.5q-29 -58 -111.5 -83t-168.5 -11.5t-132 55.5q-7 23 -29.5 87.5 t-32 94.5t-23 89t-15 101t3.5 98.5t22 110.5h-122q-66 0 -113 47t-47 113v192q0 66 47 113t113 47h480q435 0 896 384q52 0 90 -38t38 -90v-384zM1536 292v954q-394 -302 -768 -343v-270q377 -42 768 -341z" />
+<glyph unicode="&#xf0a2;" horiz-adv-x="1664" d="M848 -160q0 16 -16 16q-59 0 -101.5 42.5t-42.5 101.5q0 16 -16 16t-16 -16q0 -73 51.5 -124.5t124.5 -51.5q16 0 16 16zM183 128h1298q-164 181 -246.5 411.5t-82.5 484.5q0 256 -320 256t-320 -256q0 -254 -82.5 -484.5t-246.5 -411.5zM1664 128q0 -52 -38 -90t-90 -38 h-448q0 -106 -75 -181t-181 -75t-181 75t-75 181h-448q-52 0 -90 38t-38 90q190 161 287 397.5t97 498.5q0 165 96 262t264 117q-8 18 -8 37q0 40 28 68t68 28t68 -28t28 -68q0 -19 -8 -37q168 -20 264 -117t96 -262q0 -262 97 -498.5t287 -397.5z" />
+<glyph unicode="&#xf0a3;" d="M1376 640l138 -135q30 -28 20 -70q-12 -41 -52 -51l-188 -48l53 -186q12 -41 -19 -70q-29 -31 -70 -19l-186 53l-48 -188q-10 -40 -51 -52q-12 -2 -19 -2q-31 0 -51 22l-135 138l-135 -138q-28 -30 -70 -20q-41 11 -51 52l-48 188l-186 -53q-41 -12 -70 19q-31 29 -19 70 l53 186l-188 48q-40 10 -52 51q-10 42 20 70l138 135l-138 135q-30 28 -20 70q12 41 52 51l188 48l-53 186q-12 41 19 70q29 31 70 19l186 -53l48 188q10 41 51 51q41 12 70 -19l135 -139l135 139q29 30 70 19q41 -10 51 -51l48 -188l186 53q41 12 70 -19q31 -29 19 -70 l-53 -186l188 -48q40 -10 52 -51q10 -42 -20 -70z" />
+<glyph unicode="&#xf0a4;" horiz-adv-x="1792" d="M256 192q0 26 -19 45t-45 19t-45 -19t-19 -45t19 -45t45 -19t45 19t19 45zM1664 768q0 51 -39 89.5t-89 38.5h-576q0 20 15 48.5t33 55t33 68t15 84.5q0 67 -44.5 97.5t-115.5 30.5q-24 0 -90 -139q-24 -44 -37 -65q-40 -64 -112 -145q-71 -81 -101 -106 q-69 -57 -140 -57h-32v-640h32q72 0 167 -32t193.5 -64t179.5 -32q189 0 189 167q0 26 -5 56q30 16 47.5 52.5t17.5 73.5t-18 69q53 50 53 119q0 25 -10 55.5t-25 47.5h331q52 0 90 38t38 90zM1792 769q0 -105 -75.5 -181t-180.5 -76h-169q-4 -62 -37 -119q3 -21 3 -43 q0 -101 -60 -178q1 -139 -85 -219.5t-227 -80.5q-133 0 -322 69q-164 59 -223 59h-288q-53 0 -90.5 37.5t-37.5 90.5v640q0 53 37.5 90.5t90.5 37.5h288q10 0 21.5 4.5t23.5 14t22.5 18t24 22.5t20.5 21.5t19 21.5t14 17q65 74 100 129q13 21 33 62t37 72t40.5 63t55 49.5 t69.5 17.5q125 0 206.5 -67t81.5 -189q0 -68 -22 -128h374q104 0 180 -76t76 -179z" />
+<glyph unicode="&#xf0a5;" horiz-adv-x="1792" d="M1376 128h32v640h-32q-35 0 -67.5 12t-62.5 37t-50 46t-49 54q-2 3 -3.5 4.5t-4 4.5t-4.5 5q-72 81 -112 145q-14 22 -38 68q-1 3 -10.5 22.5t-18.5 36t-20 35.5t-21.5 30.5t-18.5 11.5q-71 0 -115.5 -30.5t-44.5 -97.5q0 -43 15 -84.5t33 -68t33 -55t15 -48.5h-576 q-50 0 -89 -38.5t-39 -89.5q0 -52 38 -90t90 -38h331q-15 -17 -25 -47.5t-10 -55.5q0 -69 53 -119q-18 -32 -18 -69t17.5 -73.5t47.5 -52.5q-4 -24 -4 -56q0 -85 48.5 -126t135.5 -41q84 0 183 32t194 64t167 32zM1664 192q0 26 -19 45t-45 19t-45 -19t-19 -45t19 -45 t45 -19t45 19t19 45zM1792 768v-640q0 -53 -37.5 -90.5t-90.5 -37.5h-288q-59 0 -223 -59q-190 -69 -317 -69q-142 0 -230 77.5t-87 217.5l1 5q-61 76 -61 178q0 22 3 43q-33 57 -37 119h-169q-105 0 -180.5 76t-75.5 181q0 103 76 179t180 76h374q-22 60 -22 128 q0 122 81.5 189t206.5 67q38 0 69.5 -17.5t55 -49.5t40.5 -63t37 -72t33 -62q35 -55 100 -129q2 -3 14 -17t19 -21.5t20.5 -21.5t24 -22.5t22.5 -18t23.5 -14t21.5 -4.5h288q53 0 90.5 -37.5t37.5 -90.5z" />
+<glyph unicode="&#xf0a6;" d="M1280 -64q0 26 -19 45t-45 19t-45 -19t-19 -45t19 -45t45 -19t45 19t19 45zM1408 700q0 189 -167 189q-26 0 -56 -5q-16 30 -52.5 47.5t-73.5 17.5t-69 -18q-50 53 -119 53q-25 0 -55.5 -10t-47.5 -25v331q0 52 -38 90t-90 38q-51 0 -89.5 -39t-38.5 -89v-576 q-20 0 -48.5 15t-55 33t-68 33t-84.5 15q-67 0 -97.5 -44.5t-30.5 -115.5q0 -24 139 -90q44 -24 65 -37q64 -40 145 -112q81 -71 106 -101q57 -69 57 -140v-32h640v32q0 72 32 167t64 193.5t32 179.5zM1536 705q0 -133 -69 -322q-59 -164 -59 -223v-288q0 -53 -37.5 -90.5 t-90.5 -37.5h-640q-53 0 -90.5 37.5t-37.5 90.5v288q0 10 -4.5 21.5t-14 23.5t-18 22.5t-22.5 24t-21.5 20.5t-21.5 19t-17 14q-74 65 -129 100q-21 13 -62 33t-72 37t-63 40.5t-49.5 55t-17.5 69.5q0 125 67 206.5t189 81.5q68 0 128 -22v374q0 104 76 180t179 76 q105 0 181 -75.5t76 -180.5v-169q62 -4 119 -37q21 3 43 3q101 0 178 -60q139 1 219.5 -85t80.5 -227z" />
+<glyph unicode="&#xf0a7;" d="M1408 576q0 84 -32 183t-64 194t-32 167v32h-640v-32q0 -35 -12 -67.5t-37 -62.5t-46 -50t-54 -49q-9 -8 -14 -12q-81 -72 -145 -112q-22 -14 -68 -38q-3 -1 -22.5 -10.5t-36 -18.5t-35.5 -20t-30.5 -21.5t-11.5 -18.5q0 -71 30.5 -115.5t97.5 -44.5q43 0 84.5 15t68 33 t55 33t48.5 15v-576q0 -50 38.5 -89t89.5 -39q52 0 90 38t38 90v331q46 -35 103 -35q69 0 119 53q32 -18 69 -18t73.5 17.5t52.5 47.5q24 -4 56 -4q85 0 126 48.5t41 135.5zM1280 1344q0 26 -19 45t-45 19t-45 -19t-19 -45t19 -45t45 -19t45 19t19 45zM1536 580 q0 -142 -77.5 -230t-217.5 -87l-5 1q-76 -61 -178 -61q-22 0 -43 3q-54 -30 -119 -37v-169q0 -105 -76 -180.5t-181 -75.5q-103 0 -179 76t-76 180v374q-54 -22 -128 -22q-121 0 -188.5 81.5t-67.5 206.5q0 38 17.5 69.5t49.5 55t63 40.5t72 37t62 33q55 35 129 100 q3 2 17 14t21.5 19t21.5 20.5t22.5 24t18 22.5t14 23.5t4.5 21.5v288q0 53 37.5 90.5t90.5 37.5h640q53 0 90.5 -37.5t37.5 -90.5v-288q0 -59 59 -223q69 -190 69 -317z" />
+<glyph unicode="&#xf0a8;" d="M1280 576v128q0 26 -19 45t-45 19h-502l189 189q19 19 19 45t-19 45l-91 91q-18 18 -45 18t-45 -18l-362 -362l-91 -91q-18 -18 -18 -45t18 -45l91 -91l362 -362q18 -18 45 -18t45 18l91 91q18 18 18 45t-18 45l-189 189h502q26 0 45 19t19 45zM1536 640 q0 -209 -103 -385.5t-279.5 -279.5t-385.5 -103t-385.5 103t-279.5 279.5t-103 385.5t103 385.5t279.5 279.5t385.5 103t385.5 -103t279.5 -279.5t103 -385.5z" />
+<glyph unicode="&#xf0a9;" d="M1285 640q0 27 -18 45l-91 91l-362 362q-18 18 -45 18t-45 -18l-91 -91q-18 -18 -18 -45t18 -45l189 -189h-502q-26 0 -45 -19t-19 -45v-128q0 -26 19 -45t45 -19h502l-189 -189q-19 -19 -19 -45t19 -45l91 -91q18 -18 45 -18t45 18l362 362l91 91q18 18 18 45zM1536 640 q0 -209 -103 -385.5t-279.5 -279.5t-385.5 -103t-385.5 103t-279.5 279.5t-103 385.5t103 385.5t279.5 279.5t385.5 103t385.5 -103t279.5 -279.5t103 -385.5z" />
+<glyph unicode="&#xf0aa;" d="M1284 641q0 27 -18 45l-362 362l-91 91q-18 18 -45 18t-45 -18l-91 -91l-362 -362q-18 -18 -18 -45t18 -45l91 -91q18 -18 45 -18t45 18l189 189v-502q0 -26 19 -45t45 -19h128q26 0 45 19t19 45v502l189 -189q19 -19 45 -19t45 19l91 91q18 18 18 45zM1536 640 q0 -209 -103 -385.5t-279.5 -279.5t-385.5 -103t-385.5 103t-279.5 279.5t-103 385.5t103 385.5t279.5 279.5t385.5 103t385.5 -103t279.5 -279.5t103 -385.5z" />
+<glyph unicode="&#xf0ab;" d="M1284 639q0 27 -18 45l-91 91q-18 18 -45 18t-45 -18l-189 -189v502q0 26 -19 45t-45 19h-128q-26 0 -45 -19t-19 -45v-502l-189 189q-19 19 -45 19t-45 -19l-91 -91q-18 -18 -18 -45t18 -45l362 -362l91 -91q18 -18 45 -18t45 18l91 91l362 362q18 18 18 45zM1536 640 q0 -209 -103 -385.5t-279.5 -279.5t-385.5 -103t-385.5 103t-279.5 279.5t-103 385.5t103 385.5t279.5 279.5t385.5 103t385.5 -103t279.5 -279.5t103 -385.5z" />
+<glyph unicode="&#xf0ac;" d="M768 1408q209 0 385.5 -103t279.5 -279.5t103 -385.5t-103 -385.5t-279.5 -279.5t-385.5 -103t-385.5 103t-279.5 279.5t-103 385.5t103 385.5t279.5 279.5t385.5 103zM1042 887q-2 -1 -9.5 -9.5t-13.5 -9.5q2 0 4.5 5t5 11t3.5 7q6 7 22 15q14 6 52 12q34 8 51 -11 q-2 2 9.5 13t14.5 12q3 2 15 4.5t15 7.5l2 22q-12 -1 -17.5 7t-6.5 21q0 -2 -6 -8q0 7 -4.5 8t-11.5 -1t-9 -1q-10 3 -15 7.5t-8 16.5t-4 15q-2 5 -9.5 10.5t-9.5 10.5q-1 2 -2.5 5.5t-3 6.5t-4 5.5t-5.5 2.5t-7 -5t-7.5 -10t-4.5 -5q-3 2 -6 1.5t-4.5 -1t-4.5 -3t-5 -3.5 q-3 -2 -8.5 -3t-8.5 -2q15 5 -1 11q-10 4 -16 3q9 4 7.5 12t-8.5 14h5q-1 4 -8.5 8.5t-17.5 8.5t-13 6q-8 5 -34 9.5t-33 0.5q-5 -6 -4.5 -10.5t4 -14t3.5 -12.5q1 -6 -5.5 -13t-6.5 -12q0 -7 14 -15.5t10 -21.5q-3 -8 -16 -16t-16 -12q-5 -8 -1.5 -18.5t10.5 -16.5 q2 -2 1.5 -4t-3.5 -4.5t-5.5 -4t-6.5 -3.5l-3 -2q-11 -5 -20.5 6t-13.5 26q-7 25 -16 30q-23 8 -29 -1q-5 13 -41 26q-25 9 -58 4q6 1 0 15q-7 15 -19 12q3 6 4 17.5t1 13.5q3 13 12 23q1 1 7 8.5t9.5 13.5t0.5 6q35 -4 50 11q5 5 11.5 17t10.5 17q9 6 14 5.5t14.5 -5.5 t14.5 -5q14 -1 15.5 11t-7.5 20q12 -1 3 17q-5 7 -8 9q-12 4 -27 -5q-8 -4 2 -8q-1 1 -9.5 -10.5t-16.5 -17.5t-16 5q-1 1 -5.5 13.5t-9.5 13.5q-8 0 -16 -15q3 8 -11 15t-24 8q19 12 -8 27q-7 4 -20.5 5t-19.5 -4q-5 -7 -5.5 -11.5t5 -8t10.5 -5.5t11.5 -4t8.5 -3 q14 -10 8 -14q-2 -1 -8.5 -3.5t-11.5 -4.5t-6 -4q-3 -4 0 -14t-2 -14q-5 5 -9 17.5t-7 16.5q7 -9 -25 -6l-10 1q-4 0 -16 -2t-20.5 -1t-13.5 8q-4 8 0 20q1 4 4 2q-4 3 -11 9.5t-10 8.5q-46 -15 -94 -41q6 -1 12 1q5 2 13 6.5t10 5.5q34 14 42 7l5 5q14 -16 20 -25 q-7 4 -30 1q-20 -6 -22 -12q7 -12 5 -18q-4 3 -11.5 10t-14.5 11t-15 5q-16 0 -22 -1q-146 -80 -235 -222q7 -7 12 -8q4 -1 5 -9t2.5 -11t11.5 3q9 -8 3 -19q1 1 44 -27q19 -17 21 -21q3 -11 -10 -18q-1 2 -9 9t-9 4q-3 -5 0.5 -18.5t10.5 -12.5q-7 0 -9.5 -16t-2.5 -35.5 t-1 -23.5l2 -1q-3 -12 5.5 -34.5t21.5 -19.5q-13 -3 20 -43q6 -8 8 -9q3 -2 12 -7.5t15 -10t10 -10.5q4 -5 10 -22.5t14 -23.5q-2 -6 9.5 -20t10.5 -23q-1 0 -2.5 -1t-2.5 -1q3 -7 15.5 -14t15.5 -13q1 -3 2 -10t3 -11t8 -2q2 20 -24 62q-15 25 -17 29q-3 5 -5.5 15.5 t-4.5 14.5q2 0 6 -1.5t8.5 -3.5t7.5 -4t2 -3q-3 -7 2 -17.5t12 -18.5t17 -19t12 -13q6 -6 14 -19.5t0 -13.5q9 0 20 -10t17 -20q5 -8 8 -26t5 -24q2 -7 8.5 -13.5t12.5 -9.5l16 -8t13 -7q5 -2 18.5 -10.5t21.5 -11.5q10 -4 16 -4t14.5 2.5t13.5 3.5q15 2 29 -15t21 -21 q36 -19 55 -11q-2 -1 0.5 -7.5t8 -15.5t9 -14.5t5.5 -8.5q5 -6 18 -15t18 -15q6 4 7 9q-3 -8 7 -20t18 -10q14 3 14 32q-31 -15 -49 18q0 1 -2.5 5.5t-4 8.5t-2.5 8.5t0 7.5t5 3q9 0 10 3.5t-2 12.5t-4 13q-1 8 -11 20t-12 15q-5 -9 -16 -8t-16 9q0 -1 -1.5 -5.5t-1.5 -6.5 q-13 0 -15 1q1 3 2.5 17.5t3.5 22.5q1 4 5.5 12t7.5 14.5t4 12.5t-4.5 9.5t-17.5 2.5q-19 -1 -26 -20q-1 -3 -3 -10.5t-5 -11.5t-9 -7q-7 -3 -24 -2t-24 5q-13 8 -22.5 29t-9.5 37q0 10 2.5 26.5t3 25t-5.5 24.5q3 2 9 9.5t10 10.5q2 1 4.5 1.5t4.5 0t4 1.5t3 6q-1 1 -4 3 q-3 3 -4 3q7 -3 28.5 1.5t27.5 -1.5q15 -11 22 2q0 1 -2.5 9.5t-0.5 13.5q5 -27 29 -9q3 -3 15.5 -5t17.5 -5q3 -2 7 -5.5t5.5 -4.5t5 0.5t8.5 6.5q10 -14 12 -24q11 -40 19 -44q7 -3 11 -2t4.5 9.5t0 14t-1.5 12.5l-1 8v18l-1 8q-15 3 -18.5 12t1.5 18.5t15 18.5q1 1 8 3.5 t15.5 6.5t12.5 8q21 19 15 35q7 0 11 9q-1 0 -5 3t-7.5 5t-4.5 2q9 5 2 16q5 3 7.5 11t7.5 10q9 -12 21 -2q7 8 1 16q5 7 20.5 10.5t18.5 9.5q7 -2 8 2t1 12t3 12q4 5 15 9t13 5l17 11q3 4 0 4q18 -2 31 11q10 11 -6 20q3 6 -3 9.5t-15 5.5q3 1 11.5 0.5t10.5 1.5 q15 10 -7 16q-17 5 -43 -12zM879 10q206 36 351 189q-3 3 -12.5 4.5t-12.5 3.5q-18 7 -24 8q1 7 -2.5 13t-8 9t-12.5 8t-11 7q-2 2 -7 6t-7 5.5t-7.5 4.5t-8.5 2t-10 -1l-3 -1q-3 -1 -5.5 -2.5t-5.5 -3t-4 -3t0 -2.5q-21 17 -36 22q-5 1 -11 5.5t-10.5 7t-10 1.5t-11.5 -7 q-5 -5 -6 -15t-2 -13q-7 5 0 17.5t2 18.5q-3 6 -10.5 4.5t-12 -4.5t-11.5 -8.5t-9 -6.5t-8.5 -5.5t-8.5 -7.5q-3 -4 -6 -12t-5 -11q-2 4 -11.5 6.5t-9.5 5.5q2 -10 4 -35t5 -38q7 -31 -12 -48q-27 -25 -29 -40q-4 -22 12 -26q0 -7 -8 -20.5t-7 -21.5q0 -6 2 -16z" />
+<glyph unicode="&#xf0ad;" horiz-adv-x="1664" d="M384 64q0 26 -19 45t-45 19t-45 -19t-19 -45t19 -45t45 -19t45 19t19 45zM1028 484l-682 -682q-37 -37 -90 -37q-52 0 -91 37l-106 108q-38 36 -38 90q0 53 38 91l681 681q39 -98 114.5 -173.5t173.5 -114.5zM1662 919q0 -39 -23 -106q-47 -134 -164.5 -217.5 t-258.5 -83.5q-185 0 -316.5 131.5t-131.5 316.5t131.5 316.5t316.5 131.5q58 0 121.5 -16.5t107.5 -46.5q16 -11 16 -28t-16 -28l-293 -169v-224l193 -107q5 3 79 48.5t135.5 81t70.5 35.5q15 0 23.5 -10t8.5 -25z" />
+<glyph unicode="&#xf0ae;" horiz-adv-x="1792" d="M1024 128h640v128h-640v-128zM640 640h1024v128h-1024v-128zM1280 1152h384v128h-384v-128zM1792 320v-256q0 -26 -19 -45t-45 -19h-1664q-26 0 -45 19t-19 45v256q0 26 19 45t45 19h1664q26 0 45 -19t19 -45zM1792 832v-256q0 -26 -19 -45t-45 -19h-1664q-26 0 -45 19 t-19 45v256q0 26 19 45t45 19h1664q26 0 45 -19t19 -45zM1792 1344v-256q0 -26 -19 -45t-45 -19h-1664q-26 0 -45 19t-19 45v256q0 26 19 45t45 19h1664q26 0 45 -19t19 -45z" />
+<glyph unicode="&#xf0b0;" horiz-adv-x="1408" d="M1403 1241q17 -41 -14 -70l-493 -493v-742q0 -42 -39 -59q-13 -5 -25 -5q-27 0 -45 19l-256 256q-19 19 -19 45v486l-493 493q-31 29 -14 70q17 39 59 39h1280q42 0 59 -39z" />
+<glyph unicode="&#xf0b1;" horiz-adv-x="1792" d="M640 1280h512v128h-512v-128zM1792 640v-480q0 -66 -47 -113t-113 -47h-1472q-66 0 -113 47t-47 113v480h672v-160q0 -26 19 -45t45 -19h320q26 0 45 19t19 45v160h672zM1024 640v-128h-256v128h256zM1792 1120v-384h-1792v384q0 66 47 113t113 47h352v160q0 40 28 68 t68 28h576q40 0 68 -28t28 -68v-160h352q66 0 113 -47t47 -113z" />
+<glyph unicode="&#xf0b2;" d="M1283 995l-355 -355l355 -355l144 144q29 31 70 14q39 -17 39 -59v-448q0 -26 -19 -45t-45 -19h-448q-42 0 -59 40q-17 39 14 69l144 144l-355 355l-355 -355l144 -144q31 -30 14 -69q-17 -40 -59 -40h-448q-26 0 -45 19t-19 45v448q0 42 40 59q39 17 69 -14l144 -144 l355 355l-355 355l-144 -144q-19 -19 -45 -19q-12 0 -24 5q-40 17 -40 59v448q0 26 19 45t45 19h448q42 0 59 -40q17 -39 -14 -69l-144 -144l355 -355l355 355l-144 144q-31 30 -14 69q17 40 59 40h448q26 0 45 -19t19 -45v-448q0 -42 -39 -59q-13 -5 -25 -5q-26 0 -45 19z " />
+<glyph unicode="&#xf0c0;" horiz-adv-x="1920" d="M593 640q-162 -5 -265 -128h-134q-82 0 -138 40.5t-56 118.5q0 353 124 353q6 0 43.5 -21t97.5 -42.5t119 -21.5q67 0 133 23q-5 -37 -5 -66q0 -139 81 -256zM1664 3q0 -120 -73 -189.5t-194 -69.5h-874q-121 0 -194 69.5t-73 189.5q0 53 3.5 103.5t14 109t26.5 108.5 t43 97.5t62 81t85.5 53.5t111.5 20q10 0 43 -21.5t73 -48t107 -48t135 -21.5t135 21.5t107 48t73 48t43 21.5q61 0 111.5 -20t85.5 -53.5t62 -81t43 -97.5t26.5 -108.5t14 -109t3.5 -103.5zM640 1280q0 -106 -75 -181t-181 -75t-181 75t-75 181t75 181t181 75t181 -75 t75 -181zM1344 896q0 -159 -112.5 -271.5t-271.5 -112.5t-271.5 112.5t-112.5 271.5t112.5 271.5t271.5 112.5t271.5 -112.5t112.5 -271.5zM1920 671q0 -78 -56 -118.5t-138 -40.5h-134q-103 123 -265 128q81 117 81 256q0 29 -5 66q66 -23 133 -23q59 0 119 21.5t97.5 42.5 t43.5 21q124 0 124 -353zM1792 1280q0 -106 -75 -181t-181 -75t-181 75t-75 181t75 181t181 75t181 -75t75 -181z" />
+<glyph unicode="&#xf0c1;" horiz-adv-x="1664" d="M1456 320q0 40 -28 68l-208 208q-28 28 -68 28q-42 0 -72 -32q3 -3 19 -18.5t21.5 -21.5t15 -19t13 -25.5t3.5 -27.5q0 -40 -28 -68t-68 -28q-15 0 -27.5 3.5t-25.5 13t-19 15t-21.5 21.5t-18.5 19q-33 -31 -33 -73q0 -40 28 -68l206 -207q27 -27 68 -27q40 0 68 26 l147 146q28 28 28 67zM753 1025q0 40 -28 68l-206 207q-28 28 -68 28q-39 0 -68 -27l-147 -146q-28 -28 -28 -67q0 -40 28 -68l208 -208q27 -27 68 -27q42 0 72 31q-3 3 -19 18.5t-21.5 21.5t-15 19t-13 25.5t-3.5 27.5q0 40 28 68t68 28q15 0 27.5 -3.5t25.5 -13t19 -15 t21.5 -21.5t18.5 -19q33 31 33 73zM1648 320q0 -120 -85 -203l-147 -146q-83 -83 -203 -83q-121 0 -204 85l-206 207q-83 83 -83 203q0 123 88 209l-88 88q-86 -88 -208 -88q-120 0 -204 84l-208 208q-84 84 -84 204t85 203l147 146q83 83 203 83q121 0 204 -85l206 -207 q83 -83 83 -203q0 -123 -88 -209l88 -88q86 88 208 88q120 0 204 -84l208 -208q84 -84 84 -204z" />
+<glyph unicode="&#xf0c2;" horiz-adv-x="1920" d="M1920 384q0 -159 -112.5 -271.5t-271.5 -112.5h-1088q-185 0 -316.5 131.5t-131.5 316.5q0 132 71 241.5t187 163.5q-2 28 -2 43q0 212 150 362t362 150q158 0 286.5 -88t187.5 -230q70 62 166 62q106 0 181 -75t75 -181q0 -75 -41 -138q129 -30 213 -134.5t84 -239.5z " />
+<glyph unicode="&#xf0c3;" horiz-adv-x="1664" d="M1527 88q56 -89 21.5 -152.5t-140.5 -63.5h-1152q-106 0 -140.5 63.5t21.5 152.5l503 793v399h-64q-26 0 -45 19t-19 45t19 45t45 19h512q26 0 45 -19t19 -45t-19 -45t-45 -19h-64v-399zM748 813l-272 -429h712l-272 429l-20 31v37v399h-128v-399v-37z" />
+<glyph unicode="&#xf0c4;" horiz-adv-x="1792" d="M960 640q26 0 45 -19t19 -45t-19 -45t-45 -19t-45 19t-19 45t19 45t45 19zM1260 576l507 -398q28 -20 25 -56q-5 -35 -35 -51l-128 -64q-13 -7 -29 -7q-17 0 -31 8l-690 387l-110 -66q-8 -4 -12 -5q14 -49 10 -97q-7 -77 -56 -147.5t-132 -123.5q-132 -84 -277 -84 q-136 0 -222 78q-90 84 -79 207q7 76 56 147t131 124q132 84 278 84q83 0 151 -31q9 13 22 22l122 73l-122 73q-13 9 -22 22q-68 -31 -151 -31q-146 0 -278 84q-82 53 -131 124t-56 147q-5 59 15.5 113t63.5 93q85 79 222 79q145 0 277 -84q83 -52 132 -123t56 -148 q4 -48 -10 -97q4 -1 12 -5l110 -66l690 387q14 8 31 8q16 0 29 -7l128 -64q30 -16 35 -51q3 -36 -25 -56zM579 836q46 42 21 108t-106 117q-92 59 -192 59q-74 0 -113 -36q-46 -42 -21 -108t106 -117q92 -59 192 -59q74 0 113 36zM494 91q81 51 106 117t-21 108 q-39 36 -113 36q-100 0 -192 -59q-81 -51 -106 -117t21 -108q39 -36 113 -36q100 0 192 59zM672 704l96 -58v11q0 36 33 56l14 8l-79 47l-26 -26q-3 -3 -10 -11t-12 -12q-2 -2 -4 -3.5t-3 -2.5zM896 480l96 -32l736 576l-128 64l-768 -431v-113l-160 -96l9 -8q2 -2 7 -6 q4 -4 11 -12t11 -12l26 -26zM1600 64l128 64l-520 408l-177 -138q-2 -3 -13 -7z" />
+<glyph unicode="&#xf0c5;" horiz-adv-x="1792" d="M1696 1152q40 0 68 -28t28 -68v-1216q0 -40 -28 -68t-68 -28h-960q-40 0 -68 28t-28 68v288h-544q-40 0 -68 28t-28 68v672q0 40 20 88t48 76l408 408q28 28 76 48t88 20h416q40 0 68 -28t28 -68v-328q68 40 128 40h416zM1152 939l-299 -299h299v299zM512 1323l-299 -299 h299v299zM708 676l316 316v416h-384v-416q0 -40 -28 -68t-68 -28h-416v-640h512v256q0 40 20 88t48 76zM1664 -128v1152h-384v-416q0 -40 -28 -68t-68 -28h-416v-640h896z" />
+<glyph unicode="&#xf0c6;" horiz-adv-x="1408" d="M1404 151q0 -117 -79 -196t-196 -79q-135 0 -235 100l-777 776q-113 115 -113 271q0 159 110 270t269 111q158 0 273 -113l605 -606q10 -10 10 -22q0 -16 -30.5 -46.5t-46.5 -30.5q-13 0 -23 10l-606 607q-79 77 -181 77q-106 0 -179 -75t-73 -181q0 -105 76 -181 l776 -777q63 -63 145 -63q64 0 106 42t42 106q0 82 -63 145l-581 581q-26 24 -60 24q-29 0 -48 -19t-19 -48q0 -32 25 -59l410 -410q10 -10 10 -22q0 -16 -31 -47t-47 -31q-12 0 -22 10l-410 410q-63 61 -63 149q0 82 57 139t139 57q88 0 149 -63l581 -581q100 -98 100 -235 z" />
+<glyph unicode="&#xf0c7;" d="M384 0h768v384h-768v-384zM1280 0h128v896q0 14 -10 38.5t-20 34.5l-281 281q-10 10 -34 20t-39 10v-416q0 -40 -28 -68t-68 -28h-576q-40 0 -68 28t-28 68v416h-128v-1280h128v416q0 40 28 68t68 28h832q40 0 68 -28t28 -68v-416zM896 928v320q0 13 -9.5 22.5t-22.5 9.5 h-192q-13 0 -22.5 -9.5t-9.5 -22.5v-320q0 -13 9.5 -22.5t22.5 -9.5h192q13 0 22.5 9.5t9.5 22.5zM1536 896v-928q0 -40 -28 -68t-68 -28h-1344q-40 0 -68 28t-28 68v1344q0 40 28 68t68 28h928q40 0 88 -20t76 -48l280 -280q28 -28 48 -76t20 -88z" />
+<glyph unicode="&#xf0c8;" d="M1536 1120v-960q0 -119 -84.5 -203.5t-203.5 -84.5h-960q-119 0 -203.5 84.5t-84.5 203.5v960q0 119 84.5 203.5t203.5 84.5h960q119 0 203.5 -84.5t84.5 -203.5z" />
+<glyph unicode="&#xf0c9;" d="M1536 192v-128q0 -26 -19 -45t-45 -19h-1408q-26 0 -45 19t-19 45v128q0 26 19 45t45 19h1408q26 0 45 -19t19 -45zM1536 704v-128q0 -26 -19 -45t-45 -19h-1408q-26 0 -45 19t-19 45v128q0 26 19 45t45 19h1408q26 0 45 -19t19 -45zM1536 1216v-128q0 -26 -19 -45 t-45 -19h-1408q-26 0 -45 19t-19 45v128q0 26 19 45t45 19h1408q26 0 45 -19t19 -45z" />
+<glyph unicode="&#xf0ca;" horiz-adv-x="1792" d="M384 128q0 -80 -56 -136t-136 -56t-136 56t-56 136t56 136t136 56t136 -56t56 -136zM384 640q0 -80 -56 -136t-136 -56t-136 56t-56 136t56 136t136 56t136 -56t56 -136zM1792 224v-192q0 -13 -9.5 -22.5t-22.5 -9.5h-1216q-13 0 -22.5 9.5t-9.5 22.5v192q0 13 9.5 22.5 t22.5 9.5h1216q13 0 22.5 -9.5t9.5 -22.5zM384 1152q0 -80 -56 -136t-136 -56t-136 56t-56 136t56 136t136 56t136 -56t56 -136zM1792 736v-192q0 -13 -9.5 -22.5t-22.5 -9.5h-1216q-13 0 -22.5 9.5t-9.5 22.5v192q0 13 9.5 22.5t22.5 9.5h1216q13 0 22.5 -9.5t9.5 -22.5z M1792 1248v-192q0 -13 -9.5 -22.5t-22.5 -9.5h-1216q-13 0 -22.5 9.5t-9.5 22.5v192q0 13 9.5 22.5t22.5 9.5h1216q13 0 22.5 -9.5t9.5 -22.5z" />
+<glyph unicode="&#xf0cb;" horiz-adv-x="1792" d="M381 -84q0 -80 -54.5 -126t-135.5 -46q-106 0 -172 66l57 88q49 -45 106 -45q29 0 50.5 14.5t21.5 42.5q0 64 -105 56l-26 56q8 10 32.5 43.5t42.5 54t37 38.5v1q-16 0 -48.5 -1t-48.5 -1v-53h-106v152h333v-88l-95 -115q51 -12 81 -49t30 -88zM383 543v-159h-362 q-6 36 -6 54q0 51 23.5 93t56.5 68t66 47.5t56.5 43.5t23.5 45q0 25 -14.5 38.5t-39.5 13.5q-46 0 -81 -58l-85 59q24 51 71.5 79.5t105.5 28.5q73 0 123 -41.5t50 -112.5q0 -50 -34 -91.5t-75 -64.5t-75.5 -50.5t-35.5 -52.5h127v60h105zM1792 224v-192q0 -13 -9.5 -22.5 t-22.5 -9.5h-1216q-13 0 -22.5 9.5t-9.5 22.5v192q0 14 9 23t23 9h1216q13 0 22.5 -9.5t9.5 -22.5zM384 1123v-99h-335v99h107q0 41 0.5 122t0.5 121v12h-2q-8 -17 -50 -54l-71 76l136 127h106v-404h108zM1792 736v-192q0 -13 -9.5 -22.5t-22.5 -9.5h-1216q-13 0 -22.5 9.5 t-9.5 22.5v192q0 14 9 23t23 9h1216q13 0 22.5 -9.5t9.5 -22.5zM1792 1248v-192q0 -13 -9.5 -22.5t-22.5 -9.5h-1216q-13 0 -22.5 9.5t-9.5 22.5v192q0 13 9.5 22.5t22.5 9.5h1216q13 0 22.5 -9.5t9.5 -22.5z" />
+<glyph unicode="&#xf0cc;" horiz-adv-x="1792" d="M1760 640q14 0 23 -9t9 -23v-64q0 -14 -9 -23t-23 -9h-1728q-14 0 -23 9t-9 23v64q0 14 9 23t23 9h1728zM483 704q-28 35 -51 80q-48 97 -48 188q0 181 134 309q133 127 393 127q50 0 167 -19q66 -12 177 -48q10 -38 21 -118q14 -123 14 -183q0 -18 -5 -45l-12 -3l-84 6 l-14 2q-50 149 -103 205q-88 91 -210 91q-114 0 -182 -59q-67 -58 -67 -146q0 -73 66 -140t279 -129q69 -20 173 -66q58 -28 95 -52h-743zM990 448h411q7 -39 7 -92q0 -111 -41 -212q-23 -55 -71 -104q-37 -35 -109 -81q-80 -48 -153 -66q-80 -21 -203 -21q-114 0 -195 23 l-140 40q-57 16 -72 28q-8 8 -8 22v13q0 108 -2 156q-1 30 0 68l2 37v44l102 2q15 -34 30 -71t22.5 -56t12.5 -27q35 -57 80 -94q43 -36 105 -57q59 -22 132 -22q64 0 139 27q77 26 122 86q47 61 47 129q0 84 -81 157q-34 29 -137 71z" />
+<glyph unicode="&#xf0cd;" d="M48 1313q-37 2 -45 4l-3 88q13 1 40 1q60 0 112 -4q132 -7 166 -7q86 0 168 3q116 4 146 5q56 0 86 2l-1 -14l2 -64v-9q-60 -9 -124 -9q-60 0 -79 -25q-13 -14 -13 -132q0 -13 0.5 -32.5t0.5 -25.5l1 -229l14 -280q6 -124 51 -202q35 -59 96 -92q88 -47 177 -47 q104 0 191 28q56 18 99 51q48 36 65 64q36 56 53 114q21 73 21 229q0 79 -3.5 128t-11 122.5t-13.5 159.5l-4 59q-5 67 -24 88q-34 35 -77 34l-100 -2l-14 3l2 86h84l205 -10q76 -3 196 10l18 -2q6 -38 6 -51q0 -7 -4 -31q-45 -12 -84 -13q-73 -11 -79 -17q-15 -15 -15 -41 q0 -7 1.5 -27t1.5 -31q8 -19 22 -396q6 -195 -15 -304q-15 -76 -41 -122q-38 -65 -112 -123q-75 -57 -182 -89q-109 -33 -255 -33q-167 0 -284 46q-119 47 -179 122q-61 76 -83 195q-16 80 -16 237v333q0 188 -17 213q-25 36 -147 39zM1536 -96v64q0 14 -9 23t-23 9h-1472 q-14 0 -23 -9t-9 -23v-64q0 -14 9 -23t23 -9h1472q14 0 23 9t9 23z" />
+<glyph unicode="&#xf0ce;" horiz-adv-x="1664" d="M512 160v192q0 14 -9 23t-23 9h-320q-14 0 -23 -9t-9 -23v-192q0 -14 9 -23t23 -9h320q14 0 23 9t9 23zM512 544v192q0 14 -9 23t-23 9h-320q-14 0 -23 -9t-9 -23v-192q0 -14 9 -23t23 -9h320q14 0 23 9t9 23zM1024 160v192q0 14 -9 23t-23 9h-320q-14 0 -23 -9t-9 -23 v-192q0 -14 9 -23t23 -9h320q14 0 23 9t9 23zM512 928v192q0 14 -9 23t-23 9h-320q-14 0 -23 -9t-9 -23v-192q0 -14 9 -23t23 -9h320q14 0 23 9t9 23zM1024 544v192q0 14 -9 23t-23 9h-320q-14 0 -23 -9t-9 -23v-192q0 -14 9 -23t23 -9h320q14 0 23 9t9 23zM1536 160v192 q0 14 -9 23t-23 9h-320q-14 0 -23 -9t-9 -23v-192q0 -14 9 -23t23 -9h320q14 0 23 9t9 23zM1024 928v192q0 14 -9 23t-23 9h-320q-14 0 -23 -9t-9 -23v-192q0 -14 9 -23t23 -9h320q14 0 23 9t9 23zM1536 544v192q0 14 -9 23t-23 9h-320q-14 0 -23 -9t-9 -23v-192 q0 -14 9 -23t23 -9h320q14 0 23 9t9 23zM1536 928v192q0 14 -9 23t-23 9h-320q-14 0 -23 -9t-9 -23v-192q0 -14 9 -23t23 -9h320q14 0 23 9t9 23zM1664 1248v-1088q0 -66 -47 -113t-113 -47h-1344q-66 0 -113 47t-47 113v1088q0 66 47 113t113 47h1344q66 0 113 -47t47 -113 z" />
+<glyph unicode="&#xf0d0;" horiz-adv-x="1664" d="M1190 955l293 293l-107 107l-293 -293zM1637 1248q0 -27 -18 -45l-1286 -1286q-18 -18 -45 -18t-45 18l-198 198q-18 18 -18 45t18 45l1286 1286q18 18 45 18t45 -18l198 -198q18 -18 18 -45zM286 1438l98 -30l-98 -30l-30 -98l-30 98l-98 30l98 30l30 98zM636 1276 l196 -60l-196 -60l-60 -196l-60 196l-196 60l196 60l60 196zM1566 798l98 -30l-98 -30l-30 -98l-30 98l-98 30l98 30l30 98zM926 1438l98 -30l-98 -30l-30 -98l-30 98l-98 30l98 30l30 98z" />
+<glyph unicode="&#xf0d1;" horiz-adv-x="1792" d="M640 128q0 52 -38 90t-90 38t-90 -38t-38 -90t38 -90t90 -38t90 38t38 90zM256 640h384v256h-158q-13 0 -22 -9l-195 -195q-9 -9 -9 -22v-30zM1536 128q0 52 -38 90t-90 38t-90 -38t-38 -90t38 -90t90 -38t90 38t38 90zM1792 1216v-1024q0 -15 -4 -26.5t-13.5 -18.5 t-16.5 -11.5t-23.5 -6t-22.5 -2t-25.5 0t-22.5 0.5q0 -106 -75 -181t-181 -75t-181 75t-75 181h-384q0 -106 -75 -181t-181 -75t-181 75t-75 181h-64q-3 0 -22.5 -0.5t-25.5 0t-22.5 2t-23.5 6t-16.5 11.5t-13.5 18.5t-4 26.5q0 26 19 45t45 19v320q0 8 -0.5 35t0 38 t2.5 34.5t6.5 37t14 30.5t22.5 30l198 198q19 19 50.5 32t58.5 13h160v192q0 26 19 45t45 19h1024q26 0 45 -19t19 -45z" />
+<glyph unicode="&#xf0d2;" d="M1536 640q0 -209 -103 -385.5t-279.5 -279.5t-385.5 -103q-111 0 -218 32q59 93 78 164q9 34 54 211q20 -39 73 -67.5t114 -28.5q121 0 216 68.5t147 188.5t52 270q0 114 -59.5 214t-172.5 163t-255 63q-105 0 -196 -29t-154.5 -77t-109 -110.5t-67 -129.5t-21.5 -134 q0 -104 40 -183t117 -111q30 -12 38 20q2 7 8 31t8 30q6 23 -11 43q-51 61 -51 151q0 151 104.5 259.5t273.5 108.5q151 0 235.5 -82t84.5 -213q0 -170 -68.5 -289t-175.5 -119q-61 0 -98 43.5t-23 104.5q8 35 26.5 93.5t30 103t11.5 75.5q0 50 -27 83t-77 33 q-62 0 -105 -57t-43 -142q0 -73 25 -122l-99 -418q-17 -70 -13 -177q-206 91 -333 281t-127 423q0 209 103 385.5t279.5 279.5t385.5 103t385.5 -103t279.5 -279.5t103 -385.5z" />
+<glyph unicode="&#xf0d3;" d="M1248 1408q119 0 203.5 -84.5t84.5 -203.5v-960q0 -119 -84.5 -203.5t-203.5 -84.5h-725q85 122 108 210q9 34 53 209q21 -39 73.5 -67t112.5 -28q181 0 295.5 147.5t114.5 373.5q0 84 -35 162.5t-96.5 139t-152.5 97t-197 36.5q-104 0 -194.5 -28.5t-153 -76.5 t-107.5 -109.5t-66.5 -128t-21.5 -132.5q0 -102 39.5 -180t116.5 -110q13 -5 23.5 0t14.5 19q10 44 15 61q6 23 -11 42q-50 62 -50 150q0 150 103.5 256.5t270.5 106.5q149 0 232.5 -81t83.5 -210q0 -168 -67.5 -286t-173.5 -118q-60 0 -97 43.5t-23 103.5q8 34 26.5 92.5 t29.5 102t11 74.5q0 49 -26.5 81.5t-75.5 32.5q-61 0 -103.5 -56.5t-42.5 -139.5q0 -72 24 -121l-98 -414q-24 -100 -7 -254h-183q-119 0 -203.5 84.5t-84.5 203.5v960q0 119 84.5 203.5t203.5 84.5h960z" />
+<glyph unicode="&#xf0d4;" d="M678 -57q0 -38 -10 -71h-380q-95 0 -171.5 56.5t-103.5 147.5q24 45 69 77.5t100 49.5t107 24t107 7q32 0 49 -2q6 -4 30.5 -21t33 -23t31 -23t32 -25.5t27.5 -25.5t26.5 -29.5t21 -30.5t17.5 -34.5t9.5 -36t4.5 -40.5zM385 294q-234 -7 -385 -85v433q103 -118 273 -118 q32 0 70 5q-21 -61 -21 -86q0 -67 63 -149zM558 805q0 -100 -43.5 -160.5t-140.5 -60.5q-51 0 -97 26t-78 67.5t-56 93.5t-35.5 104t-11.5 99q0 96 51.5 165t144.5 69q66 0 119 -41t84 -104t47 -130t16 -128zM1536 896v-736q0 -119 -84.5 -203.5t-203.5 -84.5h-468 q39 73 39 157q0 66 -22 122.5t-55.5 93t-72 71t-72 59.5t-55.5 54.5t-22 59.5q0 36 23 68t56 61.5t65.5 64.5t55.5 93t23 131t-26.5 145.5t-75.5 118.5q-6 6 -14 11t-12.5 7.5t-10 9.5t-10.5 17h135l135 64h-437q-138 0 -244.5 -38.5t-182.5 -133.5q0 126 81 213t207 87h960 q119 0 203.5 -84.5t84.5 -203.5v-96h-256v256h-128v-256h-256v-128h256v-256h128v256h256z" />
+<glyph unicode="&#xf0d5;" horiz-adv-x="1664" d="M876 71q0 21 -4.5 40.5t-9.5 36t-17.5 34.5t-21 30.5t-26.5 29.5t-27.5 25.5t-32 25.5t-31 23t-33 23t-30.5 21q-17 2 -50 2q-54 0 -106 -7t-108 -25t-98 -46t-69 -75t-27 -107q0 -68 35.5 -121.5t93 -84t120.5 -45.5t127 -15q59 0 112.5 12.5t100.5 39t74.5 73.5 t27.5 110zM756 933q0 60 -16.5 127.5t-47 130.5t-84 104t-119.5 41q-93 0 -144 -69t-51 -165q0 -47 11.5 -99t35.5 -104t56 -93.5t78 -67.5t97 -26q97 0 140.5 60.5t43.5 160.5zM625 1408h437l-135 -79h-135q71 -45 110 -126t39 -169q0 -74 -23 -131.5t-56 -92.5t-66 -64.5 t-56 -61t-23 -67.5q0 -26 16.5 -51t43 -48t58.5 -48t64 -55.5t58.5 -66t43 -85t16.5 -106.5q0 -160 -140 -282q-152 -131 -420 -131q-59 0 -119.5 10t-122 33.5t-108.5 58t-77 89t-30 121.5q0 61 37 135q32 64 96 110.5t145 71t155 36t150 13.5q-64 83 -64 149q0 12 2 23.5 t5 19.5t8 21.5t7 21.5q-40 -5 -70 -5q-149 0 -255.5 98t-106.5 246q0 140 95 250.5t234 141.5q94 20 187 20zM1664 1152v-128h-256v-256h-128v256h-256v128h256v256h128v-256h256z" />
+<glyph unicode="&#xf0d6;" horiz-adv-x="1920" d="M768 384h384v96h-128v448h-114l-148 -137l77 -80q42 37 55 57h2v-288h-128v-96zM1280 640q0 -70 -21 -142t-59.5 -134t-101.5 -101t-138 -39t-138 39t-101.5 101t-59.5 134t-21 142t21 142t59.5 134t101.5 101t138 39t138 -39t101.5 -101t59.5 -134t21 -142zM1792 384 v512q-106 0 -181 75t-75 181h-1152q0 -106 -75 -181t-181 -75v-512q106 0 181 -75t75 -181h1152q0 106 75 181t181 75zM1920 1216v-1152q0 -26 -19 -45t-45 -19h-1792q-26 0 -45 19t-19 45v1152q0 26 19 45t45 19h1792q26 0 45 -19t19 -45z" />
+<glyph unicode="&#xf0d7;" horiz-adv-x="1024" d="M1024 832q0 -26 -19 -45l-448 -448q-19 -19 -45 -19t-45 19l-448 448q-19 19 -19 45t19 45t45 19h896q26 0 45 -19t19 -45z" />
+<glyph unicode="&#xf0d8;" horiz-adv-x="1024" d="M1024 320q0 -26 -19 -45t-45 -19h-896q-26 0 -45 19t-19 45t19 45l448 448q19 19 45 19t45 -19l448 -448q19 -19 19 -45z" />
+<glyph unicode="&#xf0d9;" horiz-adv-x="640" d="M640 1088v-896q0 -26 -19 -45t-45 -19t-45 19l-448 448q-19 19 -19 45t19 45l448 448q19 19 45 19t45 -19t19 -45z" />
+<glyph unicode="&#xf0da;" horiz-adv-x="640" d="M576 640q0 -26 -19 -45l-448 -448q-19 -19 -45 -19t-45 19t-19 45v896q0 26 19 45t45 19t45 -19l448 -448q19 -19 19 -45z" />
+<glyph unicode="&#xf0db;" horiz-adv-x="1664" d="M160 0h608v1152h-640v-1120q0 -13 9.5 -22.5t22.5 -9.5zM1536 32v1120h-640v-1152h608q13 0 22.5 9.5t9.5 22.5zM1664 1248v-1216q0 -66 -47 -113t-113 -47h-1344q-66 0 -113 47t-47 113v1216q0 66 47 113t113 47h1344q66 0 113 -47t47 -113z" />
+<glyph unicode="&#xf0dc;" horiz-adv-x="1024" d="M1024 448q0 -26 -19 -45l-448 -448q-19 -19 -45 -19t-45 19l-448 448q-19 19 -19 45t19 45t45 19h896q26 0 45 -19t19 -45zM1024 832q0 -26 -19 -45t-45 -19h-896q-26 0 -45 19t-19 45t19 45l448 448q19 19 45 19t45 -19l448 -448q19 -19 19 -45z" />
+<glyph unicode="&#xf0dd;" horiz-adv-x="1024" d="M1024 448q0 -26 -19 -45l-448 -448q-19 -19 -45 -19t-45 19l-448 448q-19 19 -19 45t19 45t45 19h896q26 0 45 -19t19 -45z" />
+<glyph unicode="&#xf0de;" horiz-adv-x="1024" d="M1024 832q0 -26 -19 -45t-45 -19h-896q-26 0 -45 19t-19 45t19 45l448 448q19 19 45 19t45 -19l448 -448q19 -19 19 -45z" />
+<glyph unicode="&#xf0e0;" horiz-adv-x="1792" d="M1792 826v-794q0 -66 -47 -113t-113 -47h-1472q-66 0 -113 47t-47 113v794q44 -49 101 -87q362 -246 497 -345q57 -42 92.5 -65.5t94.5 -48t110 -24.5h1h1q51 0 110 24.5t94.5 48t92.5 65.5q170 123 498 345q57 39 100 87zM1792 1120q0 -79 -49 -151t-122 -123 q-376 -261 -468 -325q-10 -7 -42.5 -30.5t-54 -38t-52 -32.5t-57.5 -27t-50 -9h-1h-1q-23 0 -50 9t-57.5 27t-52 32.5t-54 38t-42.5 30.5q-91 64 -262 182.5t-205 142.5q-62 42 -117 115.5t-55 136.5q0 78 41.5 130t118.5 52h1472q65 0 112.5 -47t47.5 -113z" />
+<glyph unicode="&#xf0e1;" d="M349 911v-991h-330v991h330zM370 1217q1 -73 -50.5 -122t-135.5 -49h-2q-82 0 -132 49t-50 122q0 74 51.5 122.5t134.5 48.5t133 -48.5t51 -122.5zM1536 488v-568h-329v530q0 105 -40.5 164.5t-126.5 59.5q-63 0 -105.5 -34.5t-63.5 -85.5q-11 -30 -11 -81v-553h-329 q2 399 2 647t-1 296l-1 48h329v-144h-2q20 32 41 56t56.5 52t87 43.5t114.5 15.5q171 0 275 -113.5t104 -332.5z" />
+<glyph unicode="&#xf0e2;" d="M1536 640q0 -156 -61 -298t-164 -245t-245 -164t-298 -61q-172 0 -327 72.5t-264 204.5q-7 10 -6.5 22.5t8.5 20.5l137 138q10 9 25 9q16 -2 23 -12q73 -95 179 -147t225 -52q104 0 198.5 40.5t163.5 109.5t109.5 163.5t40.5 198.5t-40.5 198.5t-109.5 163.5 t-163.5 109.5t-198.5 40.5q-98 0 -188 -35.5t-160 -101.5l137 -138q31 -30 14 -69q-17 -40 -59 -40h-448q-26 0 -45 19t-19 45v448q0 42 40 59q39 17 69 -14l130 -129q107 101 244.5 156.5t284.5 55.5q156 0 298 -61t245 -164t164 -245t61 -298z" />
+<glyph unicode="&#xf0e3;" horiz-adv-x="1792" d="M1771 0q0 -53 -37 -90l-107 -108q-39 -37 -91 -37q-53 0 -90 37l-363 364q-38 36 -38 90q0 53 43 96l-256 256l-126 -126q-14 -14 -34 -14t-34 14q2 -2 12.5 -12t12.5 -13t10 -11.5t10 -13.5t6 -13.5t5.5 -16.5t1.5 -18q0 -38 -28 -68q-3 -3 -16.5 -18t-19 -20.5 t-18.5 -16.5t-22 -15.5t-22 -9t-26 -4.5q-40 0 -68 28l-408 408q-28 28 -28 68q0 13 4.5 26t9 22t15.5 22t16.5 18.5t20.5 19t18 16.5q30 28 68 28q10 0 18 -1.5t16.5 -5.5t13.5 -6t13.5 -10t11.5 -10t13 -12.5t12 -12.5q-14 14 -14 34t14 34l348 348q14 14 34 14t34 -14 q-2 2 -12.5 12t-12.5 13t-10 11.5t-10 13.5t-6 13.5t-5.5 16.5t-1.5 18q0 38 28 68q3 3 16.5 18t19 20.5t18.5 16.5t22 15.5t22 9t26 4.5q40 0 68 -28l408 -408q28 -28 28 -68q0 -13 -4.5 -26t-9 -22t-15.5 -22t-16.5 -18.5t-20.5 -19t-18 -16.5q-30 -28 -68 -28 q-10 0 -18 1.5t-16.5 5.5t-13.5 6t-13.5 10t-11.5 10t-13 12.5t-12 12.5q14 -14 14 -34t-14 -34l-126 -126l256 -256q43 43 96 43q52 0 91 -37l363 -363q37 -39 37 -91z" />
+<glyph unicode="&#xf0e4;" horiz-adv-x="1792" d="M384 384q0 53 -37.5 90.5t-90.5 37.5t-90.5 -37.5t-37.5 -90.5t37.5 -90.5t90.5 -37.5t90.5 37.5t37.5 90.5zM576 832q0 53 -37.5 90.5t-90.5 37.5t-90.5 -37.5t-37.5 -90.5t37.5 -90.5t90.5 -37.5t90.5 37.5t37.5 90.5zM1004 351l101 382q6 26 -7.5 48.5t-38.5 29.5 t-48 -6.5t-30 -39.5l-101 -382q-60 -5 -107 -43.5t-63 -98.5q-20 -77 20 -146t117 -89t146 20t89 117q16 60 -6 117t-72 91zM1664 384q0 53 -37.5 90.5t-90.5 37.5t-90.5 -37.5t-37.5 -90.5t37.5 -90.5t90.5 -37.5t90.5 37.5t37.5 90.5zM1024 1024q0 53 -37.5 90.5 t-90.5 37.5t-90.5 -37.5t-37.5 -90.5t37.5 -90.5t90.5 -37.5t90.5 37.5t37.5 90.5zM1472 832q0 53 -37.5 90.5t-90.5 37.5t-90.5 -37.5t-37.5 -90.5t37.5 -90.5t90.5 -37.5t90.5 37.5t37.5 90.5zM1792 384q0 -261 -141 -483q-19 -29 -54 -29h-1402q-35 0 -54 29 q-141 221 -141 483q0 182 71 348t191 286t286 191t348 71t348 -71t286 -191t191 -286t71 -348z" />
+<glyph unicode="&#xf0e5;" horiz-adv-x="1792" d="M896 1152q-204 0 -381.5 -69.5t-282 -187.5t-104.5 -255q0 -112 71.5 -213.5t201.5 -175.5l87 -50l-27 -96q-24 -91 -70 -172q152 63 275 171l43 38l57 -6q69 -8 130 -8q204 0 381.5 69.5t282 187.5t104.5 255t-104.5 255t-282 187.5t-381.5 69.5zM1792 640 q0 -174 -120 -321.5t-326 -233t-450 -85.5q-70 0 -145 8q-198 -175 -460 -242q-49 -14 -114 -22h-5q-15 0 -27 10.5t-16 27.5v1q-3 4 -0.5 12t2 10t4.5 9.5l6 9t7 8.5t8 9q7 8 31 34.5t34.5 38t31 39.5t32.5 51t27 59t26 76q-157 89 -247.5 220t-90.5 281q0 174 120 321.5 t326 233t450 85.5t450 -85.5t326 -233t120 -321.5z" />
+<glyph unicode="&#xf0e6;" horiz-adv-x="1792" d="M704 1152q-153 0 -286 -52t-211.5 -141t-78.5 -191q0 -82 53 -158t149 -132l97 -56l-35 -84q34 20 62 39l44 31l53 -10q78 -14 153 -14q153 0 286 52t211.5 141t78.5 191t-78.5 191t-211.5 141t-286 52zM704 1280q191 0 353.5 -68.5t256.5 -186.5t94 -257t-94 -257 t-256.5 -186.5t-353.5 -68.5q-86 0 -176 16q-124 -88 -278 -128q-36 -9 -86 -16h-3q-11 0 -20.5 8t-11.5 21q-1 3 -1 6.5t0.5 6.5t2 6l2.5 5t3.5 5.5t4 5t4.5 5t4 4.5q5 6 23 25t26 29.5t22.5 29t25 38.5t20.5 44q-124 72 -195 177t-71 224q0 139 94 257t256.5 186.5 t353.5 68.5zM1526 111q10 -24 20.5 -44t25 -38.5t22.5 -29t26 -29.5t23 -25q1 -1 4 -4.5t4.5 -5t4 -5t3.5 -5.5l2.5 -5t2 -6t0.5 -6.5t-1 -6.5q-3 -14 -13 -22t-22 -7q-50 7 -86 16q-154 40 -278 128q-90 -16 -176 -16q-271 0 -472 132q58 -4 88 -4q161 0 309 45t264 129 q125 92 192 212t67 254q0 77 -23 152q129 -71 204 -178t75 -230q0 -120 -71 -224.5t-195 -176.5z" />
+<glyph unicode="&#xf0e7;" horiz-adv-x="896" d="M885 970q18 -20 7 -44l-540 -1157q-13 -25 -42 -25q-4 0 -14 2q-17 5 -25.5 19t-4.5 30l197 808l-406 -101q-4 -1 -12 -1q-18 0 -31 11q-18 15 -13 39l201 825q4 14 16 23t28 9h328q19 0 32 -12.5t13 -29.5q0 -8 -5 -18l-171 -463l396 98q8 2 12 2q19 0 34 -15z" />
+<glyph unicode="&#xf0e8;" horiz-adv-x="1792" d="M1792 288v-320q0 -40 -28 -68t-68 -28h-320q-40 0 -68 28t-28 68v320q0 40 28 68t68 28h96v192h-512v-192h96q40 0 68 -28t28 -68v-320q0 -40 -28 -68t-68 -28h-320q-40 0 -68 28t-28 68v320q0 40 28 68t68 28h96v192h-512v-192h96q40 0 68 -28t28 -68v-320 q0 -40 -28 -68t-68 -28h-320q-40 0 -68 28t-28 68v320q0 40 28 68t68 28h96v192q0 52 38 90t90 38h512v192h-96q-40 0 -68 28t-28 68v320q0 40 28 68t68 28h320q40 0 68 -28t28 -68v-320q0 -40 -28 -68t-68 -28h-96v-192h512q52 0 90 -38t38 -90v-192h96q40 0 68 -28t28 -68 z" />
+<glyph unicode="&#xf0e9;" horiz-adv-x="1664" d="M896 708v-580q0 -104 -76 -180t-180 -76t-180 76t-76 180q0 26 19 45t45 19t45 -19t19 -45q0 -50 39 -89t89 -39t89 39t39 89v580q33 11 64 11t64 -11zM1664 681q0 -13 -9.5 -22.5t-22.5 -9.5q-11 0 -23 10q-49 46 -93 69t-102 23q-68 0 -128 -37t-103 -97 q-7 -10 -17.5 -28t-14.5 -24q-11 -17 -28 -17q-18 0 -29 17q-4 6 -14.5 24t-17.5 28q-43 60 -102.5 97t-127.5 37t-127.5 -37t-102.5 -97q-7 -10 -17.5 -28t-14.5 -24q-11 -17 -29 -17q-17 0 -28 17q-4 6 -14.5 24t-17.5 28q-43 60 -103 97t-128 37q-58 0 -102 -23t-93 -69 q-12 -10 -23 -10q-13 0 -22.5 9.5t-9.5 22.5q0 5 1 7q45 183 172.5 319.5t298 204.5t360.5 68q140 0 274.5 -40t246.5 -113.5t194.5 -187t115.5 -251.5q1 -2 1 -7zM896 1408v-98q-42 2 -64 2t-64 -2v98q0 26 19 45t45 19t45 -19t19 -45z" />
+<glyph unicode="&#xf0ea;" horiz-adv-x="1792" d="M768 -128h896v640h-416q-40 0 -68 28t-28 68v416h-384v-1152zM1024 1312v64q0 13 -9.5 22.5t-22.5 9.5h-704q-13 0 -22.5 -9.5t-9.5 -22.5v-64q0 -13 9.5 -22.5t22.5 -9.5h704q13 0 22.5 9.5t9.5 22.5zM1280 640h299l-299 299v-299zM1792 512v-672q0 -40 -28 -68t-68 -28 h-960q-40 0 -68 28t-28 68v160h-544q-40 0 -68 28t-28 68v1344q0 40 28 68t68 28h1088q40 0 68 -28t28 -68v-328q21 -13 36 -28l408 -408q28 -28 48 -76t20 -88z" />
+<glyph unicode="&#xf0eb;" horiz-adv-x="1024" d="M736 960q0 -13 -9.5 -22.5t-22.5 -9.5t-22.5 9.5t-9.5 22.5q0 46 -54 71t-106 25q-13 0 -22.5 9.5t-9.5 22.5t9.5 22.5t22.5 9.5q50 0 99.5 -16t87 -54t37.5 -90zM896 960q0 72 -34.5 134t-90 101.5t-123 62t-136.5 22.5t-136.5 -22.5t-123 -62t-90 -101.5t-34.5 -134 q0 -101 68 -180q10 -11 30.5 -33t30.5 -33q128 -153 141 -298h228q13 145 141 298q10 11 30.5 33t30.5 33q68 79 68 180zM1024 960q0 -155 -103 -268q-45 -49 -74.5 -87t-59.5 -95.5t-34 -107.5q47 -28 47 -82q0 -37 -25 -64q25 -27 25 -64q0 -52 -45 -81q13 -23 13 -47 q0 -46 -31.5 -71t-77.5 -25q-20 -44 -60 -70t-87 -26t-87 26t-60 70q-46 0 -77.5 25t-31.5 71q0 24 13 47q-45 29 -45 81q0 37 25 64q-25 27 -25 64q0 54 47 82q-4 50 -34 107.5t-59.5 95.5t-74.5 87q-103 113 -103 268q0 99 44.5 184.5t117 142t164 89t186.5 32.5 t186.5 -32.5t164 -89t117 -142t44.5 -184.5z" />
+<glyph unicode="&#xf0ec;" horiz-adv-x="1792" d="M1792 352v-192q0 -13 -9.5 -22.5t-22.5 -9.5h-1376v-192q0 -13 -9.5 -22.5t-22.5 -9.5q-12 0 -24 10l-319 320q-9 9 -9 22q0 14 9 23l320 320q9 9 23 9q13 0 22.5 -9.5t9.5 -22.5v-192h1376q13 0 22.5 -9.5t9.5 -22.5zM1792 896q0 -14 -9 -23l-320 -320q-9 -9 -23 -9 q-13 0 -22.5 9.5t-9.5 22.5v192h-1376q-13 0 -22.5 9.5t-9.5 22.5v192q0 13 9.5 22.5t22.5 9.5h1376v192q0 14 9 23t23 9q12 0 24 -10l319 -319q9 -9 9 -23z" />
+<glyph unicode="&#xf0ed;" horiz-adv-x="1920" d="M1280 608q0 14 -9 23t-23 9h-224v352q0 13 -9.5 22.5t-22.5 9.5h-192q-13 0 -22.5 -9.5t-9.5 -22.5v-352h-224q-13 0 -22.5 -9.5t-9.5 -22.5q0 -14 9 -23l352 -352q9 -9 23 -9t23 9l351 351q10 12 10 24zM1920 384q0 -159 -112.5 -271.5t-271.5 -112.5h-1088 q-185 0 -316.5 131.5t-131.5 316.5q0 130 70 240t188 165q-2 30 -2 43q0 212 150 362t362 150q156 0 285.5 -87t188.5 -231q71 62 166 62q106 0 181 -75t75 -181q0 -76 -41 -138q130 -31 213.5 -135.5t83.5 -238.5z" />
+<glyph unicode="&#xf0ee;" horiz-adv-x="1920" d="M1280 672q0 14 -9 23l-352 352q-9 9 -23 9t-23 -9l-351 -351q-10 -12 -10 -24q0 -14 9 -23t23 -9h224v-352q0 -13 9.5 -22.5t22.5 -9.5h192q13 0 22.5 9.5t9.5 22.5v352h224q13 0 22.5 9.5t9.5 22.5zM1920 384q0 -159 -112.5 -271.5t-271.5 -112.5h-1088 q-185 0 -316.5 131.5t-131.5 316.5q0 130 70 240t188 165q-2 30 -2 43q0 212 150 362t362 150q156 0 285.5 -87t188.5 -231q71 62 166 62q106 0 181 -75t75 -181q0 -76 -41 -138q130 -31 213.5 -135.5t83.5 -238.5z" />
+<glyph unicode="&#xf0f0;" horiz-adv-x="1408" d="M384 192q0 -26 -19 -45t-45 -19t-45 19t-19 45t19 45t45 19t45 -19t19 -45zM1408 131q0 -121 -73 -190t-194 -69h-874q-121 0 -194 69t-73 190q0 68 5.5 131t24 138t47.5 132.5t81 103t120 60.5q-22 -52 -22 -120v-203q-58 -20 -93 -70t-35 -111q0 -80 56 -136t136 -56 t136 56t56 136q0 61 -35.5 111t-92.5 70v203q0 62 25 93q132 -104 295 -104t295 104q25 -31 25 -93v-64q-106 0 -181 -75t-75 -181v-89q-32 -29 -32 -71q0 -40 28 -68t68 -28t68 28t28 68q0 42 -32 71v89q0 52 38 90t90 38t90 -38t38 -90v-89q-32 -29 -32 -71q0 -40 28 -68 t68 -28t68 28t28 68q0 42 -32 71v89q0 68 -34.5 127.5t-93.5 93.5q0 10 0.5 42.5t0 48t-2.5 41.5t-7 47t-13 40q68 -15 120 -60.5t81 -103t47.5 -132.5t24 -138t5.5 -131zM1088 1024q0 -159 -112.5 -271.5t-271.5 -112.5t-271.5 112.5t-112.5 271.5t112.5 271.5t271.5 112.5 t271.5 -112.5t112.5 -271.5z" />
+<glyph unicode="&#xf0f1;" horiz-adv-x="1408" d="M1280 832q0 26 -19 45t-45 19t-45 -19t-19 -45t19 -45t45 -19t45 19t19 45zM1408 832q0 -62 -35.5 -111t-92.5 -70v-395q0 -159 -131.5 -271.5t-316.5 -112.5t-316.5 112.5t-131.5 271.5v132q-164 20 -274 128t-110 252v512q0 26 19 45t45 19q6 0 16 -2q17 30 47 48 t65 18q53 0 90.5 -37.5t37.5 -90.5t-37.5 -90.5t-90.5 -37.5q-33 0 -64 18v-402q0 -106 94 -181t226 -75t226 75t94 181v402q-31 -18 -64 -18q-53 0 -90.5 37.5t-37.5 90.5t37.5 90.5t90.5 37.5q35 0 65 -18t47 -48q10 2 16 2q26 0 45 -19t19 -45v-512q0 -144 -110 -252 t-274 -128v-132q0 -106 94 -181t226 -75t226 75t94 181v395q-57 21 -92.5 70t-35.5 111q0 80 56 136t136 56t136 -56t56 -136z" />
+<glyph unicode="&#xf0f2;" horiz-adv-x="1792" d="M640 1152h512v128h-512v-128zM288 1152v-1280h-64q-92 0 -158 66t-66 158v832q0 92 66 158t158 66h64zM1408 1152v-1280h-1024v1280h128v160q0 40 28 68t68 28h576q40 0 68 -28t28 -68v-160h128zM1792 928v-832q0 -92 -66 -158t-158 -66h-64v1280h64q92 0 158 -66 t66 -158z" />
+<glyph unicode="&#xf0f3;" horiz-adv-x="1664" d="M848 -160q0 16 -16 16q-59 0 -101.5 42.5t-42.5 101.5q0 16 -16 16t-16 -16q0 -73 51.5 -124.5t124.5 -51.5q16 0 16 16zM1664 128q0 -52 -38 -90t-90 -38h-448q0 -106 -75 -181t-181 -75t-181 75t-75 181h-448q-52 0 -90 38t-38 90q190 161 287 397.5t97 498.5 q0 165 96 262t264 117q-8 18 -8 37q0 40 28 68t68 28t68 -28t28 -68q0 -19 -8 -37q168 -20 264 -117t96 -262q0 -262 97 -498.5t287 -397.5z" />
+<glyph unicode="&#xf0f4;" horiz-adv-x="1920" d="M1664 896q0 80 -56 136t-136 56h-64v-384h64q80 0 136 56t56 136zM0 128h1792q0 -106 -75 -181t-181 -75h-1280q-106 0 -181 75t-75 181zM1856 896q0 -159 -112.5 -271.5t-271.5 -112.5h-64v-32q0 -92 -66 -158t-158 -66h-704q-92 0 -158 66t-66 158v736q0 26 19 45 t45 19h1152q159 0 271.5 -112.5t112.5 -271.5z" />
+<glyph unicode="&#xf0f5;" horiz-adv-x="1408" d="M640 1472v-640q0 -61 -35.5 -111t-92.5 -70v-779q0 -52 -38 -90t-90 -38h-128q-52 0 -90 38t-38 90v779q-57 20 -92.5 70t-35.5 111v640q0 26 19 45t45 19t45 -19t19 -45v-416q0 -26 19 -45t45 -19t45 19t19 45v416q0 26 19 45t45 19t45 -19t19 -45v-416q0 -26 19 -45 t45 -19t45 19t19 45v416q0 26 19 45t45 19t45 -19t19 -45zM1408 1472v-1600q0 -52 -38 -90t-90 -38h-128q-52 0 -90 38t-38 90v512h-224q-13 0 -22.5 9.5t-9.5 22.5v800q0 132 94 226t226 94h256q26 0 45 -19t19 -45z" />
+<glyph unicode="&#xf0f6;" horiz-adv-x="1280" d="M1024 352v-64q0 -14 -9 -23t-23 -9h-704q-14 0 -23 9t-9 23v64q0 14 9 23t23 9h704q14 0 23 -9t9 -23zM1024 608v-64q0 -14 -9 -23t-23 -9h-704q-14 0 -23 9t-9 23v64q0 14 9 23t23 9h704q14 0 23 -9t9 -23zM128 0h1024v768h-416q-40 0 -68 28t-28 68v416h-512v-1280z M768 896h376q-10 29 -22 41l-313 313q-12 12 -41 22v-376zM1280 864v-896q0 -40 -28 -68t-68 -28h-1088q-40 0 -68 28t-28 68v1344q0 40 28 68t68 28h640q40 0 88 -20t76 -48l312 -312q28 -28 48 -76t20 -88z" />
+<glyph unicode="&#xf0f7;" horiz-adv-x="1408" d="M384 224v-64q0 -13 -9.5 -22.5t-22.5 -9.5h-64q-13 0 -22.5 9.5t-9.5 22.5v64q0 13 9.5 22.5t22.5 9.5h64q13 0 22.5 -9.5t9.5 -22.5zM384 480v-64q0 -13 -9.5 -22.5t-22.5 -9.5h-64q-13 0 -22.5 9.5t-9.5 22.5v64q0 13 9.5 22.5t22.5 9.5h64q13 0 22.5 -9.5t9.5 -22.5z M640 480v-64q0 -13 -9.5 -22.5t-22.5 -9.5h-64q-13 0 -22.5 9.5t-9.5 22.5v64q0 13 9.5 22.5t22.5 9.5h64q13 0 22.5 -9.5t9.5 -22.5zM384 736v-64q0 -13 -9.5 -22.5t-22.5 -9.5h-64q-13 0 -22.5 9.5t-9.5 22.5v64q0 13 9.5 22.5t22.5 9.5h64q13 0 22.5 -9.5t9.5 -22.5z M1152 224v-64q0 -13 -9.5 -22.5t-22.5 -9.5h-64q-13 0 -22.5 9.5t-9.5 22.5v64q0 13 9.5 22.5t22.5 9.5h64q13 0 22.5 -9.5t9.5 -22.5zM896 480v-64q0 -13 -9.5 -22.5t-22.5 -9.5h-64q-13 0 -22.5 9.5t-9.5 22.5v64q0 13 9.5 22.5t22.5 9.5h64q13 0 22.5 -9.5t9.5 -22.5z M640 736v-64q0 -13 -9.5 -22.5t-22.5 -9.5h-64q-13 0 -22.5 9.5t-9.5 22.5v64q0 13 9.5 22.5t22.5 9.5h64q13 0 22.5 -9.5t9.5 -22.5zM384 992v-64q0 -13 -9.5 -22.5t-22.5 -9.5h-64q-13 0 -22.5 9.5t-9.5 22.5v64q0 13 9.5 22.5t22.5 9.5h64q13 0 22.5 -9.5t9.5 -22.5z M1152 480v-64q0 -13 -9.5 -22.5t-22.5 -9.5h-64q-13 0 -22.5 9.5t-9.5 22.5v64q0 13 9.5 22.5t22.5 9.5h64q13 0 22.5 -9.5t9.5 -22.5zM896 736v-64q0 -13 -9.5 -22.5t-22.5 -9.5h-64q-13 0 -22.5 9.5t-9.5 22.5v64q0 13 9.5 22.5t22.5 9.5h64q13 0 22.5 -9.5t9.5 -22.5z M640 992v-64q0 -13 -9.5 -22.5t-22.5 -9.5h-64q-13 0 -22.5 9.5t-9.5 22.5v64q0 13 9.5 22.5t22.5 9.5h64q13 0 22.5 -9.5t9.5 -22.5zM384 1248v-64q0 -13 -9.5 -22.5t-22.5 -9.5h-64q-13 0 -22.5 9.5t-9.5 22.5v64q0 13 9.5 22.5t22.5 9.5h64q13 0 22.5 -9.5t9.5 -22.5z M1152 736v-64q0 -13 -9.5 -22.5t-22.5 -9.5h-64q-13 0 -22.5 9.5t-9.5 22.5v64q0 13 9.5 22.5t22.5 9.5h64q13 0 22.5 -9.5t9.5 -22.5zM896 992v-64q0 -13 -9.5 -22.5t-22.5 -9.5h-64q-13 0 -22.5 9.5t-9.5 22.5v64q0 13 9.5 22.5t22.5 9.5h64q13 0 22.5 -9.5t9.5 -22.5z M640 1248v-64q0 -13 -9.5 -22.5t-22.5 -9.5h-64q-13 0 -22.5 9.5t-9.5 22.5v64q0 13 9.5 22.5t22.5 9.5h64q13 0 22.5 -9.5t9.5 -22.5zM1152 992v-64q0 -13 -9.5 -22.5t-22.5 -9.5h-64q-13 0 -22.5 9.5t-9.5 22.5v64q0 13 9.5 22.5t22.5 9.5h64q13 0 22.5 -9.5t9.5 -22.5z M896 1248v-64q0 -13 -9.5 -22.5t-22.5 -9.5h-64q-13 0 -22.5 9.5t-9.5 22.5v64q0 13 9.5 22.5t22.5 9.5h64q13 0 22.5 -9.5t9.5 -22.5zM1152 1248v-64q0 -13 -9.5 -22.5t-22.5 -9.5h-64q-13 0 -22.5 9.5t-9.5 22.5v64q0 13 9.5 22.5t22.5 9.5h64q13 0 22.5 -9.5t9.5 -22.5z M896 -128h384v1536h-1152v-1536h384v224q0 13 9.5 22.5t22.5 9.5h320q13 0 22.5 -9.5t9.5 -22.5v-224zM1408 1472v-1664q0 -26 -19 -45t-45 -19h-1280q-26 0 -45 19t-19 45v1664q0 26 19 45t45 19h1280q26 0 45 -19t19 -45z" />
+<glyph unicode="&#xf0f8;" horiz-adv-x="1408" d="M384 224v-64q0 -13 -9.5 -22.5t-22.5 -9.5h-64q-13 0 -22.5 9.5t-9.5 22.5v64q0 13 9.5 22.5t22.5 9.5h64q13 0 22.5 -9.5t9.5 -22.5zM384 480v-64q0 -13 -9.5 -22.5t-22.5 -9.5h-64q-13 0 -22.5 9.5t-9.5 22.5v64q0 13 9.5 22.5t22.5 9.5h64q13 0 22.5 -9.5t9.5 -22.5z M640 480v-64q0 -13 -9.5 -22.5t-22.5 -9.5h-64q-13 0 -22.5 9.5t-9.5 22.5v64q0 13 9.5 22.5t22.5 9.5h64q13 0 22.5 -9.5t9.5 -22.5zM384 736v-64q0 -13 -9.5 -22.5t-22.5 -9.5h-64q-13 0 -22.5 9.5t-9.5 22.5v64q0 13 9.5 22.5t22.5 9.5h64q13 0 22.5 -9.5t9.5 -22.5z M1152 224v-64q0 -13 -9.5 -22.5t-22.5 -9.5h-64q-13 0 -22.5 9.5t-9.5 22.5v64q0 13 9.5 22.5t22.5 9.5h64q13 0 22.5 -9.5t9.5 -22.5zM896 480v-64q0 -13 -9.5 -22.5t-22.5 -9.5h-64q-13 0 -22.5 9.5t-9.5 22.5v64q0 13 9.5 22.5t22.5 9.5h64q13 0 22.5 -9.5t9.5 -22.5z M640 736v-64q0 -13 -9.5 -22.5t-22.5 -9.5h-64q-13 0 -22.5 9.5t-9.5 22.5v64q0 13 9.5 22.5t22.5 9.5h64q13 0 22.5 -9.5t9.5 -22.5zM1152 480v-64q0 -13 -9.5 -22.5t-22.5 -9.5h-64q-13 0 -22.5 9.5t-9.5 22.5v64q0 13 9.5 22.5t22.5 9.5h64q13 0 22.5 -9.5t9.5 -22.5z M896 736v-64q0 -13 -9.5 -22.5t-22.5 -9.5h-64q-13 0 -22.5 9.5t-9.5 22.5v64q0 13 9.5 22.5t22.5 9.5h64q13 0 22.5 -9.5t9.5 -22.5zM1152 736v-64q0 -13 -9.5 -22.5t-22.5 -9.5h-64q-13 0 -22.5 9.5t-9.5 22.5v64q0 13 9.5 22.5t22.5 9.5h64q13 0 22.5 -9.5t9.5 -22.5z M896 -128h384v1152h-256v-32q0 -40 -28 -68t-68 -28h-448q-40 0 -68 28t-28 68v32h-256v-1152h384v224q0 13 9.5 22.5t22.5 9.5h320q13 0 22.5 -9.5t9.5 -22.5v-224zM896 1056v320q0 13 -9.5 22.5t-22.5 9.5h-64q-13 0 -22.5 -9.5t-9.5 -22.5v-96h-128v96q0 13 -9.5 22.5 t-22.5 9.5h-64q-13 0 -22.5 -9.5t-9.5 -22.5v-320q0 -13 9.5 -22.5t22.5 -9.5h64q13 0 22.5 9.5t9.5 22.5v96h128v-96q0 -13 9.5 -22.5t22.5 -9.5h64q13 0 22.5 9.5t9.5 22.5zM1408 1088v-1280q0 -26 -19 -45t-45 -19h-1280q-26 0 -45 19t-19 45v1280q0 26 19 45t45 19h320 v288q0 40 28 68t68 28h448q40 0 68 -28t28 -68v-288h320q26 0 45 -19t19 -45z" />
+<glyph unicode="&#xf0f9;" horiz-adv-x="1920" d="M640 128q0 53 -37.5 90.5t-90.5 37.5t-90.5 -37.5t-37.5 -90.5t37.5 -90.5t90.5 -37.5t90.5 37.5t37.5 90.5zM256 640h384v256h-158q-14 -2 -22 -9l-195 -195q-7 -12 -9 -22v-30zM1536 128q0 53 -37.5 90.5t-90.5 37.5t-90.5 -37.5t-37.5 -90.5t37.5 -90.5t90.5 -37.5 t90.5 37.5t37.5 90.5zM1664 800v192q0 14 -9 23t-23 9h-224v224q0 14 -9 23t-23 9h-192q-14 0 -23 -9t-9 -23v-224h-224q-14 0 -23 -9t-9 -23v-192q0 -14 9 -23t23 -9h224v-224q0 -14 9 -23t23 -9h192q14 0 23 9t9 23v224h224q14 0 23 9t9 23zM1920 1344v-1152 q0 -26 -19 -45t-45 -19h-192q0 -106 -75 -181t-181 -75t-181 75t-75 181h-384q0 -106 -75 -181t-181 -75t-181 75t-75 181h-128q-26 0 -45 19t-19 45t19 45t45 19v416q0 26 13 58t32 51l198 198q19 19 51 32t58 13h160v320q0 26 19 45t45 19h1152q26 0 45 -19t19 -45z" />
+<glyph unicode="&#xf0fa;" horiz-adv-x="1792" d="M1280 416v192q0 14 -9 23t-23 9h-224v224q0 14 -9 23t-23 9h-192q-14 0 -23 -9t-9 -23v-224h-224q-14 0 -23 -9t-9 -23v-192q0 -14 9 -23t23 -9h224v-224q0 -14 9 -23t23 -9h192q14 0 23 9t9 23v224h224q14 0 23 9t9 23zM640 1152h512v128h-512v-128zM256 1152v-1280h-32 q-92 0 -158 66t-66 158v832q0 92 66 158t158 66h32zM1440 1152v-1280h-1088v1280h160v160q0 40 28 68t68 28h576q40 0 68 -28t28 -68v-160h160zM1792 928v-832q0 -92 -66 -158t-158 -66h-32v1280h32q92 0 158 -66t66 -158z" />
+<glyph unicode="&#xf0fb;" horiz-adv-x="1920" d="M1920 576q-1 -32 -288 -96l-352 -32l-224 -64h-64l-293 -352h69q26 0 45 -4.5t19 -11.5t-19 -11.5t-45 -4.5h-96h-160h-64v32h64v416h-160l-192 -224h-96l-32 32v192h32v32h128v8l-192 24v128l192 24v8h-128v32h-32v192l32 32h96l192 -224h160v416h-64v32h64h160h96 q26 0 45 -4.5t19 -11.5t-19 -11.5t-45 -4.5h-69l293 -352h64l224 -64l352 -32q261 -58 287 -93z" />
+<glyph unicode="&#xf0fc;" horiz-adv-x="1664" d="M640 640v384h-256v-256q0 -53 37.5 -90.5t90.5 -37.5h128zM1664 192v-192h-1152v192l128 192h-128q-159 0 -271.5 112.5t-112.5 271.5v320l-64 64l32 128h480l32 128h960l32 -192l-64 -32v-800z" />
+<glyph unicode="&#xf0fd;" d="M1280 192v896q0 26 -19 45t-45 19h-128q-26 0 -45 -19t-19 -45v-320h-512v320q0 26 -19 45t-45 19h-128q-26 0 -45 -19t-19 -45v-896q0 -26 19 -45t45 -19h128q26 0 45 19t19 45v320h512v-320q0 -26 19 -45t45 -19h128q26 0 45 19t19 45zM1536 1120v-960 q0 -119 -84.5 -203.5t-203.5 -84.5h-960q-119 0 -203.5 84.5t-84.5 203.5v960q0 119 84.5 203.5t203.5 84.5h960q119 0 203.5 -84.5t84.5 -203.5z" />
+<glyph unicode="&#xf0fe;" d="M1280 576v128q0 26 -19 45t-45 19h-320v320q0 26 -19 45t-45 19h-128q-26 0 -45 -19t-19 -45v-320h-320q-26 0 -45 -19t-19 -45v-128q0 -26 19 -45t45 -19h320v-320q0 -26 19 -45t45 -19h128q26 0 45 19t19 45v320h320q26 0 45 19t19 45zM1536 1120v-960 q0 -119 -84.5 -203.5t-203.5 -84.5h-960q-119 0 -203.5 84.5t-84.5 203.5v960q0 119 84.5 203.5t203.5 84.5h960q119 0 203.5 -84.5t84.5 -203.5z" />
+<glyph unicode="&#xf100;" horiz-adv-x="1024" d="M627 160q0 -13 -10 -23l-50 -50q-10 -10 -23 -10t-23 10l-466 466q-10 10 -10 23t10 23l466 466q10 10 23 10t23 -10l50 -50q10 -10 10 -23t-10 -23l-393 -393l393 -393q10 -10 10 -23zM1011 160q0 -13 -10 -23l-50 -50q-10 -10 -23 -10t-23 10l-466 466q-10 10 -10 23 t10 23l466 466q10 10 23 10t23 -10l50 -50q10 -10 10 -23t-10 -23l-393 -393l393 -393q10 -10 10 -23z" />
+<glyph unicode="&#xf101;" horiz-adv-x="1024" d="M595 576q0 -13 -10 -23l-466 -466q-10 -10 -23 -10t-23 10l-50 50q-10 10 -10 23t10 23l393 393l-393 393q-10 10 -10 23t10 23l50 50q10 10 23 10t23 -10l466 -466q10 -10 10 -23zM979 576q0 -13 -10 -23l-466 -466q-10 -10 -23 -10t-23 10l-50 50q-10 10 -10 23t10 23 l393 393l-393 393q-10 10 -10 23t10 23l50 50q10 10 23 10t23 -10l466 -466q10 -10 10 -23z" />
+<glyph unicode="&#xf102;" horiz-adv-x="1152" d="M1075 224q0 -13 -10 -23l-50 -50q-10 -10 -23 -10t-23 10l-393 393l-393 -393q-10 -10 -23 -10t-23 10l-50 50q-10 10 -10 23t10 23l466 466q10 10 23 10t23 -10l466 -466q10 -10 10 -23zM1075 608q0 -13 -10 -23l-50 -50q-10 -10 -23 -10t-23 10l-393 393l-393 -393 q-10 -10 -23 -10t-23 10l-50 50q-10 10 -10 23t10 23l466 466q10 10 23 10t23 -10l466 -466q10 -10 10 -23z" />
+<glyph unicode="&#xf103;" horiz-adv-x="1152" d="M1075 672q0 -13 -10 -23l-466 -466q-10 -10 -23 -10t-23 10l-466 466q-10 10 -10 23t10 23l50 50q10 10 23 10t23 -10l393 -393l393 393q10 10 23 10t23 -10l50 -50q10 -10 10 -23zM1075 1056q0 -13 -10 -23l-466 -466q-10 -10 -23 -10t-23 10l-466 466q-10 10 -10 23 t10 23l50 50q10 10 23 10t23 -10l393 -393l393 393q10 10 23 10t23 -10l50 -50q10 -10 10 -23z" />
+<glyph unicode="&#xf104;" horiz-adv-x="640" d="M627 992q0 -13 -10 -23l-393 -393l393 -393q10 -10 10 -23t-10 -23l-50 -50q-10 -10 -23 -10t-23 10l-466 466q-10 10 -10 23t10 23l466 466q10 10 23 10t23 -10l50 -50q10 -10 10 -23z" />
+<glyph unicode="&#xf105;" horiz-adv-x="640" d="M595 576q0 -13 -10 -23l-466 -466q-10 -10 -23 -10t-23 10l-50 50q-10 10 -10 23t10 23l393 393l-393 393q-10 10 -10 23t10 23l50 50q10 10 23 10t23 -10l466 -466q10 -10 10 -23z" />
+<glyph unicode="&#xf106;" horiz-adv-x="1152" d="M1075 352q0 -13 -10 -23l-50 -50q-10 -10 -23 -10t-23 10l-393 393l-393 -393q-10 -10 -23 -10t-23 10l-50 50q-10 10 -10 23t10 23l466 466q10 10 23 10t23 -10l466 -466q10 -10 10 -23z" />
+<glyph unicode="&#xf107;" horiz-adv-x="1152" d="M1075 800q0 -13 -10 -23l-466 -466q-10 -10 -23 -10t-23 10l-466 466q-10 10 -10 23t10 23l50 50q10 10 23 10t23 -10l393 -393l393 393q10 10 23 10t23 -10l50 -50q10 -10 10 -23z" />
+<glyph unicode="&#xf108;" horiz-adv-x="1920" d="M1792 544v832q0 13 -9.5 22.5t-22.5 9.5h-1600q-13 0 -22.5 -9.5t-9.5 -22.5v-832q0 -13 9.5 -22.5t22.5 -9.5h1600q13 0 22.5 9.5t9.5 22.5zM1920 1376v-1088q0 -66 -47 -113t-113 -47h-544q0 -37 16 -77.5t32 -71t16 -43.5q0 -26 -19 -45t-45 -19h-512q-26 0 -45 19 t-19 45q0 14 16 44t32 70t16 78h-544q-66 0 -113 47t-47 113v1088q0 66 47 113t113 47h1600q66 0 113 -47t47 -113z" />
+<glyph unicode="&#xf109;" horiz-adv-x="1920" d="M416 256q-66 0 -113 47t-47 113v704q0 66 47 113t113 47h1088q66 0 113 -47t47 -113v-704q0 -66 -47 -113t-113 -47h-1088zM384 1120v-704q0 -13 9.5 -22.5t22.5 -9.5h1088q13 0 22.5 9.5t9.5 22.5v704q0 13 -9.5 22.5t-22.5 9.5h-1088q-13 0 -22.5 -9.5t-9.5 -22.5z M1760 192h160v-96q0 -40 -47 -68t-113 -28h-1600q-66 0 -113 28t-47 68v96h160h1600zM1040 96q16 0 16 16t-16 16h-160q-16 0 -16 -16t16 -16h160z" />
+<glyph unicode="&#xf10a;" horiz-adv-x="1152" d="M640 128q0 26 -19 45t-45 19t-45 -19t-19 -45t19 -45t45 -19t45 19t19 45zM1024 288v960q0 13 -9.5 22.5t-22.5 9.5h-832q-13 0 -22.5 -9.5t-9.5 -22.5v-960q0 -13 9.5 -22.5t22.5 -9.5h832q13 0 22.5 9.5t9.5 22.5zM1152 1248v-1088q0 -66 -47 -113t-113 -47h-832 q-66 0 -113 47t-47 113v1088q0 66 47 113t113 47h832q66 0 113 -47t47 -113z" />
+<glyph unicode="&#xf10b;" horiz-adv-x="768" d="M464 128q0 33 -23.5 56.5t-56.5 23.5t-56.5 -23.5t-23.5 -56.5t23.5 -56.5t56.5 -23.5t56.5 23.5t23.5 56.5zM672 288v704q0 13 -9.5 22.5t-22.5 9.5h-512q-13 0 -22.5 -9.5t-9.5 -22.5v-704q0 -13 9.5 -22.5t22.5 -9.5h512q13 0 22.5 9.5t9.5 22.5zM480 1136 q0 16 -16 16h-160q-16 0 -16 -16t16 -16h160q16 0 16 16zM768 1152v-1024q0 -52 -38 -90t-90 -38h-512q-52 0 -90 38t-38 90v1024q0 52 38 90t90 38h512q52 0 90 -38t38 -90z" />
+<glyph unicode="&#xf10c;" d="M768 1184q-148 0 -273 -73t-198 -198t-73 -273t73 -273t198 -198t273 -73t273 73t198 198t73 273t-73 273t-198 198t-273 73zM1536 640q0 -209 -103 -385.5t-279.5 -279.5t-385.5 -103t-385.5 103t-279.5 279.5t-103 385.5t103 385.5t279.5 279.5t385.5 103t385.5 -103 t279.5 -279.5t103 -385.5z" />
+<glyph unicode="&#xf10d;" horiz-adv-x="1664" d="M768 576v-384q0 -80 -56 -136t-136 -56h-384q-80 0 -136 56t-56 136v704q0 104 40.5 198.5t109.5 163.5t163.5 109.5t198.5 40.5h64q26 0 45 -19t19 -45v-128q0 -26 -19 -45t-45 -19h-64q-106 0 -181 -75t-75 -181v-32q0 -40 28 -68t68 -28h224q80 0 136 -56t56 -136z M1664 576v-384q0 -80 -56 -136t-136 -56h-384q-80 0 -136 56t-56 136v704q0 104 40.5 198.5t109.5 163.5t163.5 109.5t198.5 40.5h64q26 0 45 -19t19 -45v-128q0 -26 -19 -45t-45 -19h-64q-106 0 -181 -75t-75 -181v-32q0 -40 28 -68t68 -28h224q80 0 136 -56t56 -136z" />
+<glyph unicode="&#xf10e;" horiz-adv-x="1664" d="M768 1216v-704q0 -104 -40.5 -198.5t-109.5 -163.5t-163.5 -109.5t-198.5 -40.5h-64q-26 0 -45 19t-19 45v128q0 26 19 45t45 19h64q106 0 181 75t75 181v32q0 40 -28 68t-68 28h-224q-80 0 -136 56t-56 136v384q0 80 56 136t136 56h384q80 0 136 -56t56 -136zM1664 1216 v-704q0 -104 -40.5 -198.5t-109.5 -163.5t-163.5 -109.5t-198.5 -40.5h-64q-26 0 -45 19t-19 45v128q0 26 19 45t45 19h64q106 0 181 75t75 181v32q0 40 -28 68t-68 28h-224q-80 0 -136 56t-56 136v384q0 80 56 136t136 56h384q80 0 136 -56t56 -136z" />
+<glyph unicode="&#xf110;" horiz-adv-x="1568" d="M496 192q0 -60 -42.5 -102t-101.5 -42q-60 0 -102 42t-42 102t42 102t102 42q59 0 101.5 -42t42.5 -102zM928 0q0 -53 -37.5 -90.5t-90.5 -37.5t-90.5 37.5t-37.5 90.5t37.5 90.5t90.5 37.5t90.5 -37.5t37.5 -90.5zM320 640q0 -66 -47 -113t-113 -47t-113 47t-47 113 t47 113t113 47t113 -47t47 -113zM1360 192q0 -46 -33 -79t-79 -33t-79 33t-33 79t33 79t79 33t79 -33t33 -79zM528 1088q0 -73 -51.5 -124.5t-124.5 -51.5t-124.5 51.5t-51.5 124.5t51.5 124.5t124.5 51.5t124.5 -51.5t51.5 -124.5zM992 1280q0 -80 -56 -136t-136 -56 t-136 56t-56 136t56 136t136 56t136 -56t56 -136zM1536 640q0 -40 -28 -68t-68 -28t-68 28t-28 68t28 68t68 28t68 -28t28 -68zM1328 1088q0 -33 -23.5 -56.5t-56.5 -23.5t-56.5 23.5t-23.5 56.5t23.5 56.5t56.5 23.5t56.5 -23.5t23.5 -56.5z" />
+<glyph unicode="&#xf111;" d="M1536 640q0 -209 -103 -385.5t-279.5 -279.5t-385.5 -103t-385.5 103t-279.5 279.5t-103 385.5t103 385.5t279.5 279.5t385.5 103t385.5 -103t279.5 -279.5t103 -385.5z" />
+<glyph unicode="&#xf112;" horiz-adv-x="1792" d="M1792 416q0 -166 -127 -451q-3 -7 -10.5 -24t-13.5 -30t-13 -22q-12 -17 -28 -17q-15 0 -23.5 10t-8.5 25q0 9 2.5 26.5t2.5 23.5q5 68 5 123q0 101 -17.5 181t-48.5 138.5t-80 101t-105.5 69.5t-133 42.5t-154 21.5t-175.5 6h-224v-256q0 -26 -19 -45t-45 -19t-45 19 l-512 512q-19 19 -19 45t19 45l512 512q19 19 45 19t45 -19t19 -45v-256h224q713 0 875 -403q53 -134 53 -333z" />
+<glyph unicode="&#xf113;" horiz-adv-x="1664" d="M640 320q0 -40 -12.5 -82t-43 -76t-72.5 -34t-72.5 34t-43 76t-12.5 82t12.5 82t43 76t72.5 34t72.5 -34t43 -76t12.5 -82zM1280 320q0 -40 -12.5 -82t-43 -76t-72.5 -34t-72.5 34t-43 76t-12.5 82t12.5 82t43 76t72.5 34t72.5 -34t43 -76t12.5 -82zM1440 320 q0 120 -69 204t-187 84q-41 0 -195 -21q-71 -11 -157 -11t-157 11q-152 21 -195 21q-118 0 -187 -84t-69 -204q0 -88 32 -153.5t81 -103t122 -60t140 -29.5t149 -7h168q82 0 149 7t140 29.5t122 60t81 103t32 153.5zM1664 496q0 -207 -61 -331q-38 -77 -105.5 -133t-141 -86 t-170 -47.5t-171.5 -22t-167 -4.5q-78 0 -142 3t-147.5 12.5t-152.5 30t-137 51.5t-121 81t-86 115q-62 123 -62 331q0 237 136 396q-27 82 -27 170q0 116 51 218q108 0 190 -39.5t189 -123.5q147 35 309 35q148 0 280 -32q105 82 187 121t189 39q51 -102 51 -218 q0 -87 -27 -168q136 -160 136 -398z" />
+<glyph unicode="&#xf114;" horiz-adv-x="1664" d="M1536 224v704q0 40 -28 68t-68 28h-704q-40 0 -68 28t-28 68v64q0 40 -28 68t-68 28h-320q-40 0 -68 -28t-28 -68v-960q0 -40 28 -68t68 -28h1216q40 0 68 28t28 68zM1664 928v-704q0 -92 -66 -158t-158 -66h-1216q-92 0 -158 66t-66 158v960q0 92 66 158t158 66h320 q92 0 158 -66t66 -158v-32h672q92 0 158 -66t66 -158z" />
+<glyph unicode="&#xf115;" horiz-adv-x="1920" d="M1781 605q0 35 -53 35h-1088q-40 0 -85.5 -21.5t-71.5 -52.5l-294 -363q-18 -24 -18 -40q0 -35 53 -35h1088q40 0 86 22t71 53l294 363q18 22 18 39zM640 768h768v160q0 40 -28 68t-68 28h-576q-40 0 -68 28t-28 68v64q0 40 -28 68t-68 28h-320q-40 0 -68 -28t-28 -68 v-853l256 315q44 53 116 87.5t140 34.5zM1909 605q0 -62 -46 -120l-295 -363q-43 -53 -116 -87.5t-140 -34.5h-1088q-92 0 -158 66t-66 158v960q0 92 66 158t158 66h320q92 0 158 -66t66 -158v-32h544q92 0 158 -66t66 -158v-160h192q54 0 99 -24.5t67 -70.5q15 -32 15 -68z " />
+<glyph unicode="&#xf116;" horiz-adv-x="1792" />
+<glyph unicode="&#xf117;" horiz-adv-x="1792" />
+<glyph unicode="&#xf118;" d="M1134 461q-37 -121 -138 -195t-228 -74t-228 74t-138 195q-8 25 4 48.5t38 31.5q25 8 48.5 -4t31.5 -38q25 -80 92.5 -129.5t151.5 -49.5t151.5 49.5t92.5 129.5q8 26 32 38t49 4t37 -31.5t4 -48.5zM640 896q0 -53 -37.5 -90.5t-90.5 -37.5t-90.5 37.5t-37.5 90.5 t37.5 90.5t90.5 37.5t90.5 -37.5t37.5 -90.5zM1152 896q0 -53 -37.5 -90.5t-90.5 -37.5t-90.5 37.5t-37.5 90.5t37.5 90.5t90.5 37.5t90.5 -37.5t37.5 -90.5zM1408 640q0 130 -51 248.5t-136.5 204t-204 136.5t-248.5 51t-248.5 -51t-204 -136.5t-136.5 -204t-51 -248.5 t51 -248.5t136.5 -204t204 -136.5t248.5 -51t248.5 51t204 136.5t136.5 204t51 248.5zM1536 640q0 -209 -103 -385.5t-279.5 -279.5t-385.5 -103t-385.5 103t-279.5 279.5t-103 385.5t103 385.5t279.5 279.5t385.5 103t385.5 -103t279.5 -279.5t103 -385.5z" />
+<glyph unicode="&#xf119;" d="M1134 307q8 -25 -4 -48.5t-37 -31.5t-49 4t-32 38q-25 80 -92.5 129.5t-151.5 49.5t-151.5 -49.5t-92.5 -129.5q-8 -26 -31.5 -38t-48.5 -4q-26 8 -38 31.5t-4 48.5q37 121 138 195t228 74t228 -74t138 -195zM640 896q0 -53 -37.5 -90.5t-90.5 -37.5t-90.5 37.5 t-37.5 90.5t37.5 90.5t90.5 37.5t90.5 -37.5t37.5 -90.5zM1152 896q0 -53 -37.5 -90.5t-90.5 -37.5t-90.5 37.5t-37.5 90.5t37.5 90.5t90.5 37.5t90.5 -37.5t37.5 -90.5zM1408 640q0 130 -51 248.5t-136.5 204t-204 136.5t-248.5 51t-248.5 -51t-204 -136.5t-136.5 -204 t-51 -248.5t51 -248.5t136.5 -204t204 -136.5t248.5 -51t248.5 51t204 136.5t136.5 204t51 248.5zM1536 640q0 -209 -103 -385.5t-279.5 -279.5t-385.5 -103t-385.5 103t-279.5 279.5t-103 385.5t103 385.5t279.5 279.5t385.5 103t385.5 -103t279.5 -279.5t103 -385.5z" />
+<glyph unicode="&#xf11a;" d="M1152 448q0 -26 -19 -45t-45 -19h-640q-26 0 -45 19t-19 45t19 45t45 19h640q26 0 45 -19t19 -45zM640 896q0 -53 -37.5 -90.5t-90.5 -37.5t-90.5 37.5t-37.5 90.5t37.5 90.5t90.5 37.5t90.5 -37.5t37.5 -90.5zM1152 896q0 -53 -37.5 -90.5t-90.5 -37.5t-90.5 37.5 t-37.5 90.5t37.5 90.5t90.5 37.5t90.5 -37.5t37.5 -90.5zM1408 640q0 130 -51 248.5t-136.5 204t-204 136.5t-248.5 51t-248.5 -51t-204 -136.5t-136.5 -204t-51 -248.5t51 -248.5t136.5 -204t204 -136.5t248.5 -51t248.5 51t204 136.5t136.5 204t51 248.5zM1536 640 q0 -209 -103 -385.5t-279.5 -279.5t-385.5 -103t-385.5 103t-279.5 279.5t-103 385.5t103 385.5t279.5 279.5t385.5 103t385.5 -103t279.5 -279.5t103 -385.5z" />
+<glyph unicode="&#xf11b;" horiz-adv-x="1920" d="M832 448v128q0 14 -9 23t-23 9h-192v192q0 14 -9 23t-23 9h-128q-14 0 -23 -9t-9 -23v-192h-192q-14 0 -23 -9t-9 -23v-128q0 -14 9 -23t23 -9h192v-192q0 -14 9 -23t23 -9h128q14 0 23 9t9 23v192h192q14 0 23 9t9 23zM1408 384q0 53 -37.5 90.5t-90.5 37.5t-90.5 -37.5 t-37.5 -90.5t37.5 -90.5t90.5 -37.5t90.5 37.5t37.5 90.5zM1664 640q0 53 -37.5 90.5t-90.5 37.5t-90.5 -37.5t-37.5 -90.5t37.5 -90.5t90.5 -37.5t90.5 37.5t37.5 90.5zM1920 512q0 -212 -150 -362t-362 -150q-192 0 -338 128h-220q-146 -128 -338 -128q-212 0 -362 150 t-150 362t150 362t362 150h896q212 0 362 -150t150 -362z" />
+<glyph unicode="&#xf11c;" horiz-adv-x="1920" d="M384 368v-96q0 -16 -16 -16h-96q-16 0 -16 16v96q0 16 16 16h96q16 0 16 -16zM512 624v-96q0 -16 -16 -16h-224q-16 0 -16 16v96q0 16 16 16h224q16 0 16 -16zM384 880v-96q0 -16 -16 -16h-96q-16 0 -16 16v96q0 16 16 16h96q16 0 16 -16zM1408 368v-96q0 -16 -16 -16 h-864q-16 0 -16 16v96q0 16 16 16h864q16 0 16 -16zM768 624v-96q0 -16 -16 -16h-96q-16 0 -16 16v96q0 16 16 16h96q16 0 16 -16zM640 880v-96q0 -16 -16 -16h-96q-16 0 -16 16v96q0 16 16 16h96q16 0 16 -16zM1024 624v-96q0 -16 -16 -16h-96q-16 0 -16 16v96q0 16 16 16 h96q16 0 16 -16zM896 880v-96q0 -16 -16 -16h-96q-16 0 -16 16v96q0 16 16 16h96q16 0 16 -16zM1280 624v-96q0 -16 -16 -16h-96q-16 0 -16 16v96q0 16 16 16h96q16 0 16 -16zM1664 368v-96q0 -16 -16 -16h-96q-16 0 -16 16v96q0 16 16 16h96q16 0 16 -16zM1152 880v-96 q0 -16 -16 -16h-96q-16 0 -16 16v96q0 16 16 16h96q16 0 16 -16zM1408 880v-96q0 -16 -16 -16h-96q-16 0 -16 16v96q0 16 16 16h96q16 0 16 -16zM1664 880v-352q0 -16 -16 -16h-224q-16 0 -16 16v96q0 16 16 16h112v240q0 16 16 16h96q16 0 16 -16zM1792 128v896h-1664v-896 h1664zM1920 1024v-896q0 -53 -37.5 -90.5t-90.5 -37.5h-1664q-53 0 -90.5 37.5t-37.5 90.5v896q0 53 37.5 90.5t90.5 37.5h1664q53 0 90.5 -37.5t37.5 -90.5z" />
+<glyph unicode="&#xf11d;" horiz-adv-x="1792" d="M1664 491v616q-169 -91 -306 -91q-82 0 -145 32q-100 49 -184 76.5t-178 27.5q-173 0 -403 -127v-599q245 113 433 113q55 0 103.5 -7.5t98 -26t77 -31t82.5 -39.5l28 -14q44 -22 101 -22q120 0 293 92zM320 1280q0 -35 -17.5 -64t-46.5 -46v-1266q0 -14 -9 -23t-23 -9 h-64q-14 0 -23 9t-9 23v1266q-29 17 -46.5 46t-17.5 64q0 53 37.5 90.5t90.5 37.5t90.5 -37.5t37.5 -90.5zM1792 1216v-763q0 -39 -35 -57q-10 -5 -17 -9q-218 -116 -369 -116q-88 0 -158 35l-28 14q-64 33 -99 48t-91 29t-114 14q-102 0 -235.5 -44t-228.5 -102 q-15 -9 -33 -9q-16 0 -32 8q-32 19 -32 56v742q0 35 31 55q35 21 78.5 42.5t114 52t152.5 49.5t155 19q112 0 209 -31t209 -86q38 -19 89 -19q122 0 310 112q22 12 31 17q31 16 62 -2q31 -20 31 -55z" />
+<glyph unicode="&#xf11e;" horiz-adv-x="1792" d="M832 536v192q-181 -16 -384 -117v-185q205 96 384 110zM832 954v197q-172 -8 -384 -126v-189q215 111 384 118zM1664 491v184q-235 -116 -384 -71v224q-20 6 -39 15q-5 3 -33 17t-34.5 17t-31.5 15t-34.5 15.5t-32.5 13t-36 12.5t-35 8.5t-39.5 7.5t-39.5 4t-44 2 q-23 0 -49 -3v-222h19q102 0 192.5 -29t197.5 -82q19 -9 39 -15v-188q42 -17 91 -17q120 0 293 92zM1664 918v189q-169 -91 -306 -91q-45 0 -78 8v-196q148 -42 384 90zM320 1280q0 -35 -17.5 -64t-46.5 -46v-1266q0 -14 -9 -23t-23 -9h-64q-14 0 -23 9t-9 23v1266 q-29 17 -46.5 46t-17.5 64q0 53 37.5 90.5t90.5 37.5t90.5 -37.5t37.5 -90.5zM1792 1216v-763q0 -39 -35 -57q-10 -5 -17 -9q-218 -116 -369 -116q-88 0 -158 35l-28 14q-64 33 -99 48t-91 29t-114 14q-102 0 -235.5 -44t-228.5 -102q-15 -9 -33 -9q-16 0 -32 8 q-32 19 -32 56v742q0 35 31 55q35 21 78.5 42.5t114 52t152.5 49.5t155 19q112 0 209 -31t209 -86q38 -19 89 -19q122 0 310 112q22 12 31 17q31 16 62 -2q31 -20 31 -55z" />
+<glyph unicode="&#xf120;" horiz-adv-x="1664" d="M585 553l-466 -466q-10 -10 -23 -10t-23 10l-50 50q-10 10 -10 23t10 23l393 393l-393 393q-10 10 -10 23t10 23l50 50q10 10 23 10t23 -10l466 -466q10 -10 10 -23t-10 -23zM1664 96v-64q0 -14 -9 -23t-23 -9h-960q-14 0 -23 9t-9 23v64q0 14 9 23t23 9h960q14 0 23 -9 t9 -23z" />
+<glyph unicode="&#xf121;" horiz-adv-x="1920" d="M617 137l-50 -50q-10 -10 -23 -10t-23 10l-466 466q-10 10 -10 23t10 23l466 466q10 10 23 10t23 -10l50 -50q10 -10 10 -23t-10 -23l-393 -393l393 -393q10 -10 10 -23t-10 -23zM1208 1204l-373 -1291q-4 -13 -15.5 -19.5t-23.5 -2.5l-62 17q-13 4 -19.5 15.5t-2.5 24.5 l373 1291q4 13 15.5 19.5t23.5 2.5l62 -17q13 -4 19.5 -15.5t2.5 -24.5zM1865 553l-466 -466q-10 -10 -23 -10t-23 10l-50 50q-10 10 -10 23t10 23l393 393l-393 393q-10 10 -10 23t10 23l50 50q10 10 23 10t23 -10l466 -466q10 -10 10 -23t-10 -23z" />
+<glyph unicode="&#xf122;" horiz-adv-x="1792" d="M640 454v-70q0 -42 -39 -59q-13 -5 -25 -5q-27 0 -45 19l-512 512q-19 19 -19 45t19 45l512 512q29 31 70 14q39 -17 39 -59v-69l-397 -398q-19 -19 -19 -45t19 -45zM1792 416q0 -58 -17 -133.5t-38.5 -138t-48 -125t-40.5 -90.5l-20 -40q-8 -17 -28 -17q-6 0 -9 1 q-25 8 -23 34q43 400 -106 565q-64 71 -170.5 110.5t-267.5 52.5v-251q0 -42 -39 -59q-13 -5 -25 -5q-27 0 -45 19l-512 512q-19 19 -19 45t19 45l512 512q29 31 70 14q39 -17 39 -59v-262q411 -28 599 -221q169 -173 169 -509z" />
+<glyph unicode="&#xf123;" horiz-adv-x="1664" d="M1186 579l257 250l-356 52l-66 10l-30 60l-159 322v-963l59 -31l318 -168l-60 355l-12 66zM1638 841l-363 -354l86 -500q5 -33 -6 -51.5t-34 -18.5q-17 0 -40 12l-449 236l-449 -236q-23 -12 -40 -12q-23 0 -34 18.5t-6 51.5l86 500l-364 354q-32 32 -23 59.5t54 34.5 l502 73l225 455q20 41 49 41q28 0 49 -41l225 -455l502 -73q45 -7 54 -34.5t-24 -59.5z" />
+<glyph unicode="&#xf124;" horiz-adv-x="1408" d="M1401 1187l-640 -1280q-17 -35 -57 -35q-5 0 -15 2q-22 5 -35.5 22.5t-13.5 39.5v576h-576q-22 0 -39.5 13.5t-22.5 35.5t4 42t29 30l1280 640q13 7 29 7q27 0 45 -19q15 -14 18.5 -34.5t-6.5 -39.5z" />
+<glyph unicode="&#xf125;" horiz-adv-x="1664" d="M557 256h595v595zM512 301l595 595h-595v-595zM1664 224v-192q0 -14 -9 -23t-23 -9h-224v-224q0 -14 -9 -23t-23 -9h-192q-14 0 -23 9t-9 23v224h-864q-14 0 -23 9t-9 23v864h-224q-14 0 -23 9t-9 23v192q0 14 9 23t23 9h224v224q0 14 9 23t23 9h192q14 0 23 -9t9 -23 v-224h851l246 247q10 9 23 9t23 -9q9 -10 9 -23t-9 -23l-247 -246v-851h224q14 0 23 -9t9 -23z" />
+<glyph unicode="&#xf126;" horiz-adv-x="1024" d="M288 64q0 40 -28 68t-68 28t-68 -28t-28 -68t28 -68t68 -28t68 28t28 68zM288 1216q0 40 -28 68t-68 28t-68 -28t-28 -68t28 -68t68 -28t68 28t28 68zM928 1088q0 40 -28 68t-68 28t-68 -28t-28 -68t28 -68t68 -28t68 28t28 68zM1024 1088q0 -52 -26 -96.5t-70 -69.5 q-2 -287 -226 -414q-68 -38 -203 -81q-128 -40 -169.5 -71t-41.5 -100v-26q44 -25 70 -69.5t26 -96.5q0 -80 -56 -136t-136 -56t-136 56t-56 136q0 52 26 96.5t70 69.5v820q-44 25 -70 69.5t-26 96.5q0 80 56 136t136 56t136 -56t56 -136q0 -52 -26 -96.5t-70 -69.5v-497 q54 26 154 57q55 17 87.5 29.5t70.5 31t59 39.5t40.5 51t28 69.5t8.5 91.5q-44 25 -70 69.5t-26 96.5q0 80 56 136t136 56t136 -56t56 -136z" />
+<glyph unicode="&#xf127;" horiz-adv-x="1664" d="M439 265l-256 -256q-10 -9 -23 -9q-12 0 -23 9q-9 10 -9 23t9 23l256 256q10 9 23 9t23 -9q9 -10 9 -23t-9 -23zM608 224v-320q0 -14 -9 -23t-23 -9t-23 9t-9 23v320q0 14 9 23t23 9t23 -9t9 -23zM384 448q0 -14 -9 -23t-23 -9h-320q-14 0 -23 9t-9 23t9 23t23 9h320 q14 0 23 -9t9 -23zM1648 320q0 -120 -85 -203l-147 -146q-83 -83 -203 -83q-121 0 -204 85l-334 335q-21 21 -42 56l239 18l273 -274q27 -27 68 -27.5t68 26.5l147 146q28 28 28 67q0 40 -28 68l-274 275l18 239q35 -21 56 -42l336 -336q84 -86 84 -204zM1031 1044l-239 -18 l-273 274q-28 28 -68 28q-39 0 -68 -27l-147 -146q-28 -28 -28 -67q0 -40 28 -68l274 -274l-18 -240q-35 21 -56 42l-336 336q-84 86 -84 204q0 120 85 203l147 146q83 83 203 83q121 0 204 -85l334 -335q21 -21 42 -56zM1664 960q0 -14 -9 -23t-23 -9h-320q-14 0 -23 9 t-9 23t9 23t23 9h320q14 0 23 -9t9 -23zM1120 1504v-320q0 -14 -9 -23t-23 -9t-23 9t-9 23v320q0 14 9 23t23 9t23 -9t9 -23zM1527 1353l-256 -256q-11 -9 -23 -9t-23 9q-9 10 -9 23t9 23l256 256q10 9 23 9t23 -9q9 -10 9 -23t-9 -23z" />
+<glyph unicode="&#xf128;" horiz-adv-x="1024" d="M704 280v-240q0 -16 -12 -28t-28 -12h-240q-16 0 -28 12t-12 28v240q0 16 12 28t28 12h240q16 0 28 -12t12 -28zM1020 880q0 -54 -15.5 -101t-35 -76.5t-55 -59.5t-57.5 -43.5t-61 -35.5q-41 -23 -68.5 -65t-27.5 -67q0 -17 -12 -32.5t-28 -15.5h-240q-15 0 -25.5 18.5 t-10.5 37.5v45q0 83 65 156.5t143 108.5q59 27 84 56t25 76q0 42 -46.5 74t-107.5 32q-65 0 -108 -29q-35 -25 -107 -115q-13 -16 -31 -16q-12 0 -25 8l-164 125q-13 10 -15.5 25t5.5 28q160 266 464 266q80 0 161 -31t146 -83t106 -127.5t41 -158.5z" />
+<glyph unicode="&#xf129;" horiz-adv-x="640" d="M640 192v-128q0 -26 -19 -45t-45 -19h-512q-26 0 -45 19t-19 45v128q0 26 19 45t45 19h64v384h-64q-26 0 -45 19t-19 45v128q0 26 19 45t45 19h384q26 0 45 -19t19 -45v-576h64q26 0 45 -19t19 -45zM512 1344v-192q0 -26 -19 -45t-45 -19h-256q-26 0 -45 19t-19 45v192 q0 26 19 45t45 19h256q26 0 45 -19t19 -45z" />
+<glyph unicode="&#xf12a;" horiz-adv-x="640" d="M512 288v-224q0 -26 -19 -45t-45 -19h-256q-26 0 -45 19t-19 45v224q0 26 19 45t45 19h256q26 0 45 -19t19 -45zM542 1344l-28 -768q-1 -26 -20.5 -45t-45.5 -19h-256q-26 0 -45.5 19t-20.5 45l-28 768q-1 26 17.5 45t44.5 19h320q26 0 44.5 -19t17.5 -45z" />
+<glyph unicode="&#xf12b;" d="M897 167v-167h-248l-159 252l-24 42q-8 9 -11 21h-3l-9 -21q-10 -20 -25 -44l-155 -250h-258v167h128l197 291l-185 272h-137v168h276l139 -228q2 -4 23 -42q8 -9 11 -21h3q3 9 11 21l25 42l140 228h257v-168h-125l-184 -267l204 -296h109zM1534 846v-206h-514l-3 27 q-4 28 -4 46q0 64 26 117t65 86.5t84 65t84 54.5t65 54t26 64q0 38 -29.5 62.5t-70.5 24.5q-51 0 -97 -39q-14 -11 -36 -38l-105 92q26 37 63 66q83 65 188 65q110 0 178 -59.5t68 -158.5q0 -56 -24.5 -103t-62 -76.5t-81.5 -58.5t-82 -50.5t-65.5 -51.5t-30.5 -63h232v80 h126z" />
+<glyph unicode="&#xf12c;" d="M897 167v-167h-248l-159 252l-24 42q-8 9 -11 21h-3l-9 -21q-10 -20 -25 -44l-155 -250h-258v167h128l197 291l-185 272h-137v168h276l139 -228q2 -4 23 -42q8 -9 11 -21h3q3 9 11 21l25 42l140 228h257v-168h-125l-184 -267l204 -296h109zM1536 -50v-206h-514l-4 27 q-3 45 -3 46q0 64 26 117t65 86.5t84 65t84 54.5t65 54t26 64q0 38 -29.5 62.5t-70.5 24.5q-51 0 -97 -39q-14 -11 -36 -38l-105 92q26 37 63 66q80 65 188 65q110 0 178 -59.5t68 -158.5q0 -66 -34.5 -118.5t-84 -86t-99.5 -62.5t-87 -63t-41 -73h232v80h126z" />
+<glyph unicode="&#xf12d;" horiz-adv-x="1920" d="M896 128l336 384h-768l-336 -384h768zM1909 1205q15 -34 9.5 -71.5t-30.5 -65.5l-896 -1024q-38 -44 -96 -44h-768q-38 0 -69.5 20.5t-47.5 54.5q-15 34 -9.5 71.5t30.5 65.5l896 1024q38 44 96 44h768q38 0 69.5 -20.5t47.5 -54.5z" />
+<glyph unicode="&#xf12e;" horiz-adv-x="1664" d="M1664 438q0 -81 -44.5 -135t-123.5 -54q-41 0 -77.5 17.5t-59 38t-56.5 38t-71 17.5q-110 0 -110 -124q0 -39 16 -115t15 -115v-5q-22 0 -33 -1q-34 -3 -97.5 -11.5t-115.5 -13.5t-98 -5q-61 0 -103 26.5t-42 83.5q0 37 17.5 71t38 56.5t38 59t17.5 77.5q0 79 -54 123.5 t-135 44.5q-84 0 -143 -45.5t-59 -127.5q0 -43 15 -83t33.5 -64.5t33.5 -53t15 -50.5q0 -45 -46 -89q-37 -35 -117 -35q-95 0 -245 24q-9 2 -27.5 4t-27.5 4l-13 2q-1 0 -3 1q-2 0 -2 1v1024q2 -1 17.5 -3.5t34 -5t21.5 -3.5q150 -24 245 -24q80 0 117 35q46 44 46 89 q0 22 -15 50.5t-33.5 53t-33.5 64.5t-15 83q0 82 59 127.5t144 45.5q80 0 134 -44.5t54 -123.5q0 -41 -17.5 -77.5t-38 -59t-38 -56.5t-17.5 -71q0 -57 42 -83.5t103 -26.5q64 0 180 15t163 17v-2q-1 -2 -3.5 -17.5t-5 -34t-3.5 -21.5q-24 -150 -24 -245q0 -80 35 -117 q44 -46 89 -46q22 0 50.5 15t53 33.5t64.5 33.5t83 15q82 0 127.5 -59t45.5 -143z" />
+<glyph unicode="&#xf130;" horiz-adv-x="1152" d="M1152 832v-128q0 -221 -147.5 -384.5t-364.5 -187.5v-132h256q26 0 45 -19t19 -45t-19 -45t-45 -19h-640q-26 0 -45 19t-19 45t19 45t45 19h256v132q-217 24 -364.5 187.5t-147.5 384.5v128q0 26 19 45t45 19t45 -19t19 -45v-128q0 -185 131.5 -316.5t316.5 -131.5 t316.5 131.5t131.5 316.5v128q0 26 19 45t45 19t45 -19t19 -45zM896 1216v-512q0 -132 -94 -226t-226 -94t-226 94t-94 226v512q0 132 94 226t226 94t226 -94t94 -226z" />
+<glyph unicode="&#xf131;" horiz-adv-x="1408" d="M271 591l-101 -101q-42 103 -42 214v128q0 26 19 45t45 19t45 -19t19 -45v-128q0 -53 15 -113zM1385 1193l-361 -361v-128q0 -132 -94 -226t-226 -94q-55 0 -109 19l-96 -96q97 -51 205 -51q185 0 316.5 131.5t131.5 316.5v128q0 26 19 45t45 19t45 -19t19 -45v-128 q0 -221 -147.5 -384.5t-364.5 -187.5v-132h256q26 0 45 -19t19 -45t-19 -45t-45 -19h-640q-26 0 -45 19t-19 45t19 45t45 19h256v132q-125 13 -235 81l-254 -254q-10 -10 -23 -10t-23 10l-82 82q-10 10 -10 23t10 23l1234 1234q10 10 23 10t23 -10l82 -82q10 -10 10 -23 t-10 -23zM1005 1325l-621 -621v512q0 132 94 226t226 94q102 0 184.5 -59t116.5 -152z" />
+<glyph unicode="&#xf132;" horiz-adv-x="1280" d="M1088 576v640h-448v-1137q119 63 213 137q235 184 235 360zM1280 1344v-768q0 -86 -33.5 -170.5t-83 -150t-118 -127.5t-126.5 -103t-121 -77.5t-89.5 -49.5t-42.5 -20q-12 -6 -26 -6t-26 6q-16 7 -42.5 20t-89.5 49.5t-121 77.5t-126.5 103t-118 127.5t-83 150 t-33.5 170.5v768q0 26 19 45t45 19h1152q26 0 45 -19t19 -45z" />
+<glyph unicode="&#xf133;" horiz-adv-x="1664" d="M128 -128h1408v1024h-1408v-1024zM512 1088v288q0 14 -9 23t-23 9h-64q-14 0 -23 -9t-9 -23v-288q0 -14 9 -23t23 -9h64q14 0 23 9t9 23zM1280 1088v288q0 14 -9 23t-23 9h-64q-14 0 -23 -9t-9 -23v-288q0 -14 9 -23t23 -9h64q14 0 23 9t9 23zM1664 1152v-1280 q0 -52 -38 -90t-90 -38h-1408q-52 0 -90 38t-38 90v1280q0 52 38 90t90 38h128v96q0 66 47 113t113 47h64q66 0 113 -47t47 -113v-96h384v96q0 66 47 113t113 47h64q66 0 113 -47t47 -113v-96h128q52 0 90 -38t38 -90z" />
+<glyph unicode="&#xf134;" horiz-adv-x="1408" d="M512 1344q0 26 -19 45t-45 19t-45 -19t-19 -45t19 -45t45 -19t45 19t19 45zM1408 1376v-320q0 -16 -12 -25q-8 -7 -20 -7q-4 0 -7 1l-448 96q-11 2 -18 11t-7 20h-256v-102q111 -23 183.5 -111t72.5 -203v-800q0 -26 -19 -45t-45 -19h-512q-26 0 -45 19t-19 45v800 q0 106 62.5 190.5t161.5 114.5v111h-32q-59 0 -115 -23.5t-91.5 -53t-66 -66.5t-40.5 -53.5t-14 -24.5q-17 -35 -57 -35q-16 0 -29 7q-23 12 -31.5 37t3.5 49q5 10 14.5 26t37.5 53.5t60.5 70t85 67t108.5 52.5q-25 42 -25 86q0 66 47 113t113 47t113 -47t47 -113 q0 -33 -14 -64h302q0 11 7 20t18 11l448 96q3 1 7 1q12 0 20 -7q12 -9 12 -25z" />
+<glyph unicode="&#xf135;" horiz-adv-x="1664" d="M1440 1088q0 40 -28 68t-68 28t-68 -28t-28 -68t28 -68t68 -28t68 28t28 68zM1664 1376q0 -249 -75.5 -430.5t-253.5 -360.5q-81 -80 -195 -176l-20 -379q-2 -16 -16 -26l-384 -224q-7 -4 -16 -4q-12 0 -23 9l-64 64q-13 14 -8 32l85 276l-281 281l-276 -85q-3 -1 -9 -1 q-14 0 -23 9l-64 64q-17 19 -5 39l224 384q10 14 26 16l379 20q96 114 176 195q188 187 358 258t431 71q14 0 24 -9.5t10 -22.5z" />
+<glyph unicode="&#xf136;" horiz-adv-x="1792" d="M1745 763l-164 -763h-334l178 832q13 56 -15 88q-27 33 -83 33h-169l-204 -953h-334l204 953h-286l-204 -953h-334l204 953l-153 327h1276q101 0 189.5 -40.5t147.5 -113.5q60 -73 81 -168.5t0 -194.5z" />
+<glyph unicode="&#xf137;" d="M909 141l102 102q19 19 19 45t-19 45l-307 307l307 307q19 19 19 45t-19 45l-102 102q-19 19 -45 19t-45 -19l-454 -454q-19 -19 -19 -45t19 -45l454 -454q19 -19 45 -19t45 19zM1536 640q0 -209 -103 -385.5t-279.5 -279.5t-385.5 -103t-385.5 103t-279.5 279.5 t-103 385.5t103 385.5t279.5 279.5t385.5 103t385.5 -103t279.5 -279.5t103 -385.5z" />
+<glyph unicode="&#xf138;" d="M717 141l454 454q19 19 19 45t-19 45l-454 454q-19 19 -45 19t-45 -19l-102 -102q-19 -19 -19 -45t19 -45l307 -307l-307 -307q-19 -19 -19 -45t19 -45l102 -102q19 -19 45 -19t45 19zM1536 640q0 -209 -103 -385.5t-279.5 -279.5t-385.5 -103t-385.5 103t-279.5 279.5 t-103 385.5t103 385.5t279.5 279.5t385.5 103t385.5 -103t279.5 -279.5t103 -385.5z" />
+<glyph unicode="&#xf139;" d="M1165 397l102 102q19 19 19 45t-19 45l-454 454q-19 19 -45 19t-45 -19l-454 -454q-19 -19 -19 -45t19 -45l102 -102q19 -19 45 -19t45 19l307 307l307 -307q19 -19 45 -19t45 19zM1536 640q0 -209 -103 -385.5t-279.5 -279.5t-385.5 -103t-385.5 103t-279.5 279.5 t-103 385.5t103 385.5t279.5 279.5t385.5 103t385.5 -103t279.5 -279.5t103 -385.5z" />
+<glyph unicode="&#xf13a;" d="M813 237l454 454q19 19 19 45t-19 45l-102 102q-19 19 -45 19t-45 -19l-307 -307l-307 307q-19 19 -45 19t-45 -19l-102 -102q-19 -19 -19 -45t19 -45l454 -454q19 -19 45 -19t45 19zM1536 640q0 -209 -103 -385.5t-279.5 -279.5t-385.5 -103t-385.5 103t-279.5 279.5 t-103 385.5t103 385.5t279.5 279.5t385.5 103t385.5 -103t279.5 -279.5t103 -385.5z" />
+<glyph unicode="&#xf13b;" horiz-adv-x="1408" d="M1130 939l16 175h-884l47 -534h612l-22 -228l-197 -53l-196 53l-13 140h-175l22 -278l362 -100h4v1l359 99l50 544h-644l-15 181h674zM0 1408h1408l-128 -1438l-578 -162l-574 162z" />
+<glyph unicode="&#xf13c;" horiz-adv-x="1792" d="M275 1408h1505l-266 -1333l-804 -267l-698 267l71 356h297l-29 -147l422 -161l486 161l68 339h-1208l58 297h1209l38 191h-1208z" />
+<glyph unicode="&#xf13d;" horiz-adv-x="1792" d="M960 1280q0 26 -19 45t-45 19t-45 -19t-19 -45t19 -45t45 -19t45 19t19 45zM1792 352v-352q0 -22 -20 -30q-8 -2 -12 -2q-13 0 -23 9l-93 93q-119 -143 -318.5 -226.5t-429.5 -83.5t-429.5 83.5t-318.5 226.5l-93 -93q-9 -9 -23 -9q-4 0 -12 2q-20 8 -20 30v352 q0 14 9 23t23 9h352q22 0 30 -20q8 -19 -7 -35l-100 -100q67 -91 189.5 -153.5t271.5 -82.5v647h-192q-26 0 -45 19t-19 45v128q0 26 19 45t45 19h192v163q-58 34 -93 92.5t-35 128.5q0 106 75 181t181 75t181 -75t75 -181q0 -70 -35 -128.5t-93 -92.5v-163h192q26 0 45 -19 t19 -45v-128q0 -26 -19 -45t-45 -19h-192v-647q149 20 271.5 82.5t189.5 153.5l-100 100q-15 16 -7 35q8 20 30 20h352q14 0 23 -9t9 -23z" />
+<glyph unicode="&#xf13e;" horiz-adv-x="1152" d="M1056 768q40 0 68 -28t28 -68v-576q0 -40 -28 -68t-68 -28h-960q-40 0 -68 28t-28 68v576q0 40 28 68t68 28h32v320q0 185 131.5 316.5t316.5 131.5t316.5 -131.5t131.5 -316.5q0 -26 -19 -45t-45 -19h-64q-26 0 -45 19t-19 45q0 106 -75 181t-181 75t-181 -75t-75 -181 v-320h736z" />
+<glyph unicode="&#xf140;" d="M1024 640q0 -106 -75 -181t-181 -75t-181 75t-75 181t75 181t181 75t181 -75t75 -181zM1152 640q0 159 -112.5 271.5t-271.5 112.5t-271.5 -112.5t-112.5 -271.5t112.5 -271.5t271.5 -112.5t271.5 112.5t112.5 271.5zM1280 640q0 -212 -150 -362t-362 -150t-362 150 t-150 362t150 362t362 150t362 -150t150 -362zM1408 640q0 130 -51 248.5t-136.5 204t-204 136.5t-248.5 51t-248.5 -51t-204 -136.5t-136.5 -204t-51 -248.5t51 -248.5t136.5 -204t204 -136.5t248.5 -51t248.5 51t204 136.5t136.5 204t51 248.5zM1536 640 q0 -209 -103 -385.5t-279.5 -279.5t-385.5 -103t-385.5 103t-279.5 279.5t-103 385.5t103 385.5t279.5 279.5t385.5 103t385.5 -103t279.5 -279.5t103 -385.5z" />
+<glyph unicode="&#xf141;" horiz-adv-x="1408" d="M384 800v-192q0 -40 -28 -68t-68 -28h-192q-40 0 -68 28t-28 68v192q0 40 28 68t68 28h192q40 0 68 -28t28 -68zM896 800v-192q0 -40 -28 -68t-68 -28h-192q-40 0 -68 28t-28 68v192q0 40 28 68t68 28h192q40 0 68 -28t28 -68zM1408 800v-192q0 -40 -28 -68t-68 -28h-192 q-40 0 -68 28t-28 68v192q0 40 28 68t68 28h192q40 0 68 -28t28 -68z" />
+<glyph unicode="&#xf142;" horiz-adv-x="384" d="M384 288v-192q0 -40 -28 -68t-68 -28h-192q-40 0 -68 28t-28 68v192q0 40 28 68t68 28h192q40 0 68 -28t28 -68zM384 800v-192q0 -40 -28 -68t-68 -28h-192q-40 0 -68 28t-28 68v192q0 40 28 68t68 28h192q40 0 68 -28t28 -68zM384 1312v-192q0 -40 -28 -68t-68 -28h-192 q-40 0 -68 28t-28 68v192q0 40 28 68t68 28h192q40 0 68 -28t28 -68z" />
+<glyph unicode="&#xf143;" d="M512 256q0 53 -37.5 90.5t-90.5 37.5t-90.5 -37.5t-37.5 -90.5t37.5 -90.5t90.5 -37.5t90.5 37.5t37.5 90.5zM863 162q-13 232 -177 396t-396 177q-14 1 -24 -9t-10 -23v-128q0 -13 8.5 -22t21.5 -10q154 -11 264 -121t121 -264q1 -13 10 -21.5t22 -8.5h128q13 0 23 10 t9 24zM1247 161q-5 154 -56 297.5t-139.5 260t-205 205t-260 139.5t-297.5 56q-14 1 -23 -9q-10 -10 -10 -23v-128q0 -13 9 -22t22 -10q204 -7 378 -111.5t278.5 -278.5t111.5 -378q1 -13 10 -22t22 -9h128q13 0 23 10q11 9 9 23zM1536 1120v-960q0 -119 -84.5 -203.5 t-203.5 -84.5h-960q-119 0 -203.5 84.5t-84.5 203.5v960q0 119 84.5 203.5t203.5 84.5h960q119 0 203.5 -84.5t84.5 -203.5z" />
+<glyph unicode="&#xf144;" d="M768 1408q209 0 385.5 -103t279.5 -279.5t103 -385.5t-103 -385.5t-279.5 -279.5t-385.5 -103t-385.5 103t-279.5 279.5t-103 385.5t103 385.5t279.5 279.5t385.5 103zM1152 585q32 18 32 55t-32 55l-544 320q-31 19 -64 1q-32 -19 -32 -56v-640q0 -37 32 -56 q16 -8 32 -8q17 0 32 9z" />
+<glyph unicode="&#xf145;" horiz-adv-x="1792" d="M1024 1084l316 -316l-572 -572l-316 316zM813 105l618 618q19 19 19 45t-19 45l-362 362q-18 18 -45 18t-45 -18l-618 -618q-19 -19 -19 -45t19 -45l362 -362q18 -18 45 -18t45 18zM1702 742l-907 -908q-37 -37 -90.5 -37t-90.5 37l-126 126q56 56 56 136t-56 136 t-136 56t-136 -56l-125 126q-37 37 -37 90.5t37 90.5l907 906q37 37 90.5 37t90.5 -37l125 -125q-56 -56 -56 -136t56 -136t136 -56t136 56l126 -125q37 -37 37 -90.5t-37 -90.5z" />
+<glyph unicode="&#xf146;" d="M1280 576v128q0 26 -19 45t-45 19h-896q-26 0 -45 -19t-19 -45v-128q0 -26 19 -45t45 -19h896q26 0 45 19t19 45zM1536 1120v-960q0 -119 -84.5 -203.5t-203.5 -84.5h-960q-119 0 -203.5 84.5t-84.5 203.5v960q0 119 84.5 203.5t203.5 84.5h960q119 0 203.5 -84.5 t84.5 -203.5z" />
+<glyph unicode="&#xf147;" horiz-adv-x="1408" d="M1152 736v-64q0 -14 -9 -23t-23 -9h-832q-14 0 -23 9t-9 23v64q0 14 9 23t23 9h832q14 0 23 -9t9 -23zM1280 288v832q0 66 -47 113t-113 47h-832q-66 0 -113 -47t-47 -113v-832q0 -66 47 -113t113 -47h832q66 0 113 47t47 113zM1408 1120v-832q0 -119 -84.5 -203.5 t-203.5 -84.5h-832q-119 0 -203.5 84.5t-84.5 203.5v832q0 119 84.5 203.5t203.5 84.5h832q119 0 203.5 -84.5t84.5 -203.5z" />
+<glyph unicode="&#xf148;" horiz-adv-x="1024" d="M1018 933q-18 -37 -58 -37h-192v-864q0 -14 -9 -23t-23 -9h-704q-21 0 -29 18q-8 20 4 35l160 192q9 11 25 11h320v640h-192q-40 0 -58 37q-17 37 9 68l320 384q18 22 49 22t49 -22l320 -384q27 -32 9 -68z" />
+<glyph unicode="&#xf149;" horiz-adv-x="1024" d="M32 1280h704q13 0 22.5 -9.5t9.5 -23.5v-863h192q40 0 58 -37t-9 -69l-320 -384q-18 -22 -49 -22t-49 22l-320 384q-26 31 -9 69q18 37 58 37h192v640h-320q-14 0 -25 11l-160 192q-13 14 -4 34q9 19 29 19z" />
+<glyph unicode="&#xf14a;" d="M685 237l614 614q19 19 19 45t-19 45l-102 102q-19 19 -45 19t-45 -19l-467 -467l-211 211q-19 19 -45 19t-45 -19l-102 -102q-19 -19 -19 -45t19 -45l358 -358q19 -19 45 -19t45 19zM1536 1120v-960q0 -119 -84.5 -203.5t-203.5 -84.5h-960q-119 0 -203.5 84.5 t-84.5 203.5v960q0 119 84.5 203.5t203.5 84.5h960q119 0 203.5 -84.5t84.5 -203.5z" />
+<glyph unicode="&#xf14b;" d="M404 428l152 -152l-52 -52h-56v96h-96v56zM818 818q14 -13 -3 -30l-291 -291q-17 -17 -30 -3q-14 13 3 30l291 291q17 17 30 3zM544 128l544 544l-288 288l-544 -544v-288h288zM1152 736l92 92q28 28 28 68t-28 68l-152 152q-28 28 -68 28t-68 -28l-92 -92zM1536 1120 v-960q0 -119 -84.5 -203.5t-203.5 -84.5h-960q-119 0 -203.5 84.5t-84.5 203.5v960q0 119 84.5 203.5t203.5 84.5h960q119 0 203.5 -84.5t84.5 -203.5z" />
+<glyph unicode="&#xf14c;" d="M1280 608v480q0 26 -19 45t-45 19h-480q-42 0 -59 -39q-17 -41 14 -70l144 -144l-534 -534q-19 -19 -19 -45t19 -45l102 -102q19 -19 45 -19t45 19l534 534l144 -144q18 -19 45 -19q12 0 25 5q39 17 39 59zM1536 1120v-960q0 -119 -84.5 -203.5t-203.5 -84.5h-960 q-119 0 -203.5 84.5t-84.5 203.5v960q0 119 84.5 203.5t203.5 84.5h960q119 0 203.5 -84.5t84.5 -203.5z" />
+<glyph unicode="&#xf14d;" d="M1005 435l352 352q19 19 19 45t-19 45l-352 352q-30 31 -69 14q-40 -17 -40 -59v-160q-119 0 -216 -19.5t-162.5 -51t-114 -79t-76.5 -95.5t-44.5 -109t-21.5 -111.5t-5 -110.5q0 -181 167 -404q10 -12 25 -12q7 0 13 3q22 9 19 33q-44 354 62 473q46 52 130 75.5 t224 23.5v-160q0 -42 40 -59q12 -5 24 -5q26 0 45 19zM1536 1120v-960q0 -119 -84.5 -203.5t-203.5 -84.5h-960q-119 0 -203.5 84.5t-84.5 203.5v960q0 119 84.5 203.5t203.5 84.5h960q119 0 203.5 -84.5t84.5 -203.5z" />
+<glyph unicode="&#xf14e;" d="M640 448l256 128l-256 128v-256zM1024 1039v-542l-512 -256v542zM1312 640q0 148 -73 273t-198 198t-273 73t-273 -73t-198 -198t-73 -273t73 -273t198 -198t273 -73t273 73t198 198t73 273zM1536 640q0 -209 -103 -385.5t-279.5 -279.5t-385.5 -103t-385.5 103 t-279.5 279.5t-103 385.5t103 385.5t279.5 279.5t385.5 103t385.5 -103t279.5 -279.5t103 -385.5z" />
+<glyph unicode="&#xf150;" d="M1145 861q18 -35 -5 -66l-320 -448q-19 -27 -52 -27t-52 27l-320 448q-23 31 -5 66q17 35 57 35h640q40 0 57 -35zM1280 160v960q0 13 -9.5 22.5t-22.5 9.5h-960q-13 0 -22.5 -9.5t-9.5 -22.5v-960q0 -13 9.5 -22.5t22.5 -9.5h960q13 0 22.5 9.5t9.5 22.5zM1536 1120 v-960q0 -119 -84.5 -203.5t-203.5 -84.5h-960q-119 0 -203.5 84.5t-84.5 203.5v960q0 119 84.5 203.5t203.5 84.5h960q119 0 203.5 -84.5t84.5 -203.5z" />
+<glyph unicode="&#xf151;" d="M1145 419q-17 -35 -57 -35h-640q-40 0 -57 35q-18 35 5 66l320 448q19 27 52 27t52 -27l320 -448q23 -31 5 -66zM1280 160v960q0 13 -9.5 22.5t-22.5 9.5h-960q-13 0 -22.5 -9.5t-9.5 -22.5v-960q0 -13 9.5 -22.5t22.5 -9.5h960q13 0 22.5 9.5t9.5 22.5zM1536 1120v-960 q0 -119 -84.5 -203.5t-203.5 -84.5h-960q-119 0 -203.5 84.5t-84.5 203.5v960q0 119 84.5 203.5t203.5 84.5h960q119 0 203.5 -84.5t84.5 -203.5z" />
+<glyph unicode="&#xf152;" d="M1088 640q0 -33 -27 -52l-448 -320q-31 -23 -66 -5q-35 17 -35 57v640q0 40 35 57q35 18 66 -5l448 -320q27 -19 27 -52zM1280 160v960q0 14 -9 23t-23 9h-960q-14 0 -23 -9t-9 -23v-960q0 -14 9 -23t23 -9h960q14 0 23 9t9 23zM1536 1120v-960q0 -119 -84.5 -203.5 t-203.5 -84.5h-960q-119 0 -203.5 84.5t-84.5 203.5v960q0 119 84.5 203.5t203.5 84.5h960q119 0 203.5 -84.5t84.5 -203.5z" />
+<glyph unicode="&#xf153;" horiz-adv-x="1024" d="M976 229l35 -159q3 -12 -3 -22.5t-17 -14.5l-5 -1q-4 -2 -10.5 -3.5t-16 -4.5t-21.5 -5.5t-25.5 -5t-30 -5t-33.5 -4.5t-36.5 -3t-38.5 -1q-234 0 -409 130.5t-238 351.5h-95q-13 0 -22.5 9.5t-9.5 22.5v113q0 13 9.5 22.5t22.5 9.5h66q-2 57 1 105h-67q-14 0 -23 9 t-9 23v114q0 14 9 23t23 9h98q67 210 243.5 338t400.5 128q102 0 194 -23q11 -3 20 -15q6 -11 3 -24l-43 -159q-3 -13 -14 -19.5t-24 -2.5l-4 1q-4 1 -11.5 2.5l-17.5 3.5t-22.5 3.5t-26 3t-29 2.5t-29.5 1q-126 0 -226 -64t-150 -176h468q16 0 25 -12q10 -12 7 -26 l-24 -114q-5 -26 -32 -26h-488q-3 -37 0 -105h459q15 0 25 -12q9 -12 6 -27l-24 -112q-2 -11 -11 -18.5t-20 -7.5h-387q48 -117 149.5 -185.5t228.5 -68.5q18 0 36 1.5t33.5 3.5t29.5 4.5t24.5 5t18.5 4.5l12 3l5 2q13 5 26 -2q12 -7 15 -21z" />
+<glyph unicode="&#xf154;" horiz-adv-x="1024" d="M1020 399v-367q0 -14 -9 -23t-23 -9h-956q-14 0 -23 9t-9 23v150q0 13 9.5 22.5t22.5 9.5h97v383h-95q-14 0 -23 9.5t-9 22.5v131q0 14 9 23t23 9h95v223q0 171 123.5 282t314.5 111q185 0 335 -125q9 -8 10 -20.5t-7 -22.5l-103 -127q-9 -11 -22 -12q-13 -2 -23 7 q-5 5 -26 19t-69 32t-93 18q-85 0 -137 -47t-52 -123v-215h305q13 0 22.5 -9t9.5 -23v-131q0 -13 -9.5 -22.5t-22.5 -9.5h-305v-379h414v181q0 13 9 22.5t23 9.5h162q14 0 23 -9.5t9 -22.5z" />
+<glyph unicode="&#xf155;" horiz-adv-x="1024" d="M978 351q0 -153 -99.5 -263.5t-258.5 -136.5v-175q0 -14 -9 -23t-23 -9h-135q-13 0 -22.5 9.5t-9.5 22.5v175q-66 9 -127.5 31t-101.5 44.5t-74 48t-46.5 37.5t-17.5 18q-17 21 -2 41l103 135q7 10 23 12q15 2 24 -9l2 -2q113 -99 243 -125q37 -8 74 -8q81 0 142.5 43 t61.5 122q0 28 -15 53t-33.5 42t-58.5 37.5t-66 32t-80 32.5q-39 16 -61.5 25t-61.5 26.5t-62.5 31t-56.5 35.5t-53.5 42.5t-43.5 49t-35.5 58t-21 66.5t-8.5 78q0 138 98 242t255 134v180q0 13 9.5 22.5t22.5 9.5h135q14 0 23 -9t9 -23v-176q57 -6 110.5 -23t87 -33.5 t63.5 -37.5t39 -29t15 -14q17 -18 5 -38l-81 -146q-8 -15 -23 -16q-14 -3 -27 7q-3 3 -14.5 12t-39 26.5t-58.5 32t-74.5 26t-85.5 11.5q-95 0 -155 -43t-60 -111q0 -26 8.5 -48t29.5 -41.5t39.5 -33t56 -31t60.5 -27t70 -27.5q53 -20 81 -31.5t76 -35t75.5 -42.5t62 -50 t53 -63.5t31.5 -76.5t13 -94z" />
+<glyph unicode="&#xf156;" horiz-adv-x="898" d="M898 1066v-102q0 -14 -9 -23t-23 -9h-168q-23 -144 -129 -234t-276 -110q167 -178 459 -536q14 -16 4 -34q-8 -18 -29 -18h-195q-16 0 -25 12q-306 367 -498 571q-9 9 -9 22v127q0 13 9.5 22.5t22.5 9.5h112q132 0 212.5 43t102.5 125h-427q-14 0 -23 9t-9 23v102 q0 14 9 23t23 9h413q-57 113 -268 113h-145q-13 0 -22.5 9.5t-9.5 22.5v133q0 14 9 23t23 9h832q14 0 23 -9t9 -23v-102q0 -14 -9 -23t-23 -9h-233q47 -61 64 -144h171q14 0 23 -9t9 -23z" />
+<glyph unicode="&#xf157;" horiz-adv-x="1027" d="M603 0h-172q-13 0 -22.5 9t-9.5 23v330h-288q-13 0 -22.5 9t-9.5 23v103q0 13 9.5 22.5t22.5 9.5h288v85h-288q-13 0 -22.5 9t-9.5 23v104q0 13 9.5 22.5t22.5 9.5h214l-321 578q-8 16 0 32q10 16 28 16h194q19 0 29 -18l215 -425q19 -38 56 -125q10 24 30.5 68t27.5 61 l191 420q8 19 29 19h191q17 0 27 -16q9 -14 1 -31l-313 -579h215q13 0 22.5 -9.5t9.5 -22.5v-104q0 -14 -9.5 -23t-22.5 -9h-290v-85h290q13 0 22.5 -9.5t9.5 -22.5v-103q0 -14 -9.5 -23t-22.5 -9h-290v-330q0 -13 -9.5 -22.5t-22.5 -9.5z" />
+<glyph unicode="&#xf158;" horiz-adv-x="1280" d="M1043 971q0 100 -65 162t-171 62h-320v-448h320q106 0 171 62t65 162zM1280 971q0 -193 -126.5 -315t-326.5 -122h-340v-118h505q14 0 23 -9t9 -23v-128q0 -14 -9 -23t-23 -9h-505v-192q0 -14 -9.5 -23t-22.5 -9h-167q-14 0 -23 9t-9 23v192h-224q-14 0 -23 9t-9 23v128 q0 14 9 23t23 9h224v118h-224q-14 0 -23 9t-9 23v149q0 13 9 22.5t23 9.5h224v629q0 14 9 23t23 9h539q200 0 326.5 -122t126.5 -315z" />
+<glyph unicode="&#xf159;" horiz-adv-x="1792" d="M514 341l81 299h-159l75 -300q1 -1 1 -3t1 -3q0 1 0.5 3.5t0.5 3.5zM630 768l35 128h-292l32 -128h225zM822 768h139l-35 128h-70zM1271 340l78 300h-162l81 -299q0 -1 0.5 -3.5t1.5 -3.5q0 1 0.5 3t0.5 3zM1382 768l33 128h-297l34 -128h230zM1792 736v-64q0 -14 -9 -23 t-23 -9h-213l-164 -616q-7 -24 -31 -24h-159q-24 0 -31 24l-166 616h-209l-167 -616q-7 -24 -31 -24h-159q-11 0 -19.5 7t-10.5 17l-160 616h-208q-14 0 -23 9t-9 23v64q0 14 9 23t23 9h175l-33 128h-142q-14 0 -23 9t-9 23v64q0 14 9 23t23 9h109l-89 344q-5 15 5 28 q10 12 26 12h137q26 0 31 -24l90 -360h359l97 360q7 24 31 24h126q24 0 31 -24l98 -360h365l93 360q5 24 31 24h137q16 0 26 -12q10 -13 5 -28l-91 -344h111q14 0 23 -9t9 -23v-64q0 -14 -9 -23t-23 -9h-145l-34 -128h179q14 0 23 -9t9 -23z" />
+<glyph unicode="&#xf15a;" horiz-adv-x="1280" d="M1167 896q18 -182 -131 -258q117 -28 175 -103t45 -214q-7 -71 -32.5 -125t-64.5 -89t-97 -58.5t-121.5 -34.5t-145.5 -15v-255h-154v251q-80 0 -122 1v-252h-154v255q-18 0 -54 0.5t-55 0.5h-200l31 183h111q50 0 58 51v402h16q-6 1 -16 1v287q-13 68 -89 68h-111v164 l212 -1q64 0 97 1v252h154v-247q82 2 122 2v245h154v-252q79 -7 140 -22.5t113 -45t82.5 -78t36.5 -114.5zM952 351q0 36 -15 64t-37 46t-57.5 30.5t-65.5 18.5t-74 9t-69 3t-64.5 -1t-47.5 -1v-338q8 0 37 -0.5t48 -0.5t53 1.5t58.5 4t57 8.5t55.5 14t47.5 21t39.5 30 t24.5 40t9.5 51zM881 827q0 33 -12.5 58.5t-30.5 42t-48 28t-55 16.5t-61.5 8t-58 2.5t-54 -1t-39.5 -0.5v-307q5 0 34.5 -0.5t46.5 0t50 2t55 5.5t51.5 11t48.5 18.5t37 27t27 38.5t9 51z" />
+<glyph unicode="&#xf15b;" horiz-adv-x="1280" d="M1280 768v-800q0 -40 -28 -68t-68 -28h-1088q-40 0 -68 28t-28 68v1344q0 40 28 68t68 28h544v-544q0 -40 28 -68t68 -28h544zM1277 896h-509v509q82 -15 132 -65l312 -312q50 -50 65 -132z" />
+<glyph unicode="&#xf15c;" horiz-adv-x="1280" d="M1024 160v64q0 14 -9 23t-23 9h-704q-14 0 -23 -9t-9 -23v-64q0 -14 9 -23t23 -9h704q14 0 23 9t9 23zM1024 416v64q0 14 -9 23t-23 9h-704q-14 0 -23 -9t-9 -23v-64q0 -14 9 -23t23 -9h704q14 0 23 9t9 23zM1280 768v-800q0 -40 -28 -68t-68 -28h-1088q-40 0 -68 28 t-28 68v1344q0 40 28 68t68 28h544v-544q0 -40 28 -68t68 -28h544zM1277 896h-509v509q82 -15 132 -65l312 -312q50 -50 65 -132z" />
+<glyph unicode="&#xf15d;" horiz-adv-x="1664" d="M1191 1128h177l-72 218l-12 47q-2 16 -2 20h-4l-3 -20q0 -1 -3.5 -18t-7.5 -29zM736 96q0 -12 -10 -24l-319 -319q-10 -9 -23 -9q-12 0 -23 9l-320 320q-15 16 -7 35q8 20 30 20h192v1376q0 14 9 23t23 9h192q14 0 23 -9t9 -23v-1376h192q14 0 23 -9t9 -23zM1572 -23 v-233h-584v90l369 529q12 18 21 27l11 9v3q-2 0 -6.5 -0.5t-7.5 -0.5q-12 -3 -30 -3h-232v-115h-120v229h567v-89l-369 -530q-6 -8 -21 -26l-11 -11v-2l14 2q9 2 30 2h248v119h121zM1661 874v-106h-288v106h75l-47 144h-243l-47 -144h75v-106h-287v106h70l230 662h162 l230 -662h70z" />
+<glyph unicode="&#xf15e;" horiz-adv-x="1664" d="M1191 104h177l-72 218l-12 47q-2 16 -2 20h-4l-3 -20q0 -1 -3.5 -18t-7.5 -29zM736 96q0 -12 -10 -24l-319 -319q-10 -9 -23 -9q-12 0 -23 9l-320 320q-15 16 -7 35q8 20 30 20h192v1376q0 14 9 23t23 9h192q14 0 23 -9t9 -23v-1376h192q14 0 23 -9t9 -23zM1661 -150 v-106h-288v106h75l-47 144h-243l-47 -144h75v-106h-287v106h70l230 662h162l230 -662h70zM1572 1001v-233h-584v90l369 529q12 18 21 27l11 9v3q-2 0 -6.5 -0.5t-7.5 -0.5q-12 -3 -30 -3h-232v-115h-120v229h567v-89l-369 -530q-6 -8 -21 -26l-11 -10v-3l14 3q9 1 30 1h248 v119h121z" />
+<glyph unicode="&#xf160;" horiz-adv-x="1792" d="M736 96q0 -12 -10 -24l-319 -319q-10 -9 -23 -9q-12 0 -23 9l-320 320q-15 16 -7 35q8 20 30 20h192v1376q0 14 9 23t23 9h192q14 0 23 -9t9 -23v-1376h192q14 0 23 -9t9 -23zM1792 -32v-192q0 -14 -9 -23t-23 -9h-832q-14 0 -23 9t-9 23v192q0 14 9 23t23 9h832 q14 0 23 -9t9 -23zM1600 480v-192q0 -14 -9 -23t-23 -9h-640q-14 0 -23 9t-9 23v192q0 14 9 23t23 9h640q14 0 23 -9t9 -23zM1408 992v-192q0 -14 -9 -23t-23 -9h-448q-14 0 -23 9t-9 23v192q0 14 9 23t23 9h448q14 0 23 -9t9 -23zM1216 1504v-192q0 -14 -9 -23t-23 -9h-256 q-14 0 -23 9t-9 23v192q0 14 9 23t23 9h256q14 0 23 -9t9 -23z" />
+<glyph unicode="&#xf161;" horiz-adv-x="1792" d="M1216 -32v-192q0 -14 -9 -23t-23 -9h-256q-14 0 -23 9t-9 23v192q0 14 9 23t23 9h256q14 0 23 -9t9 -23zM736 96q0 -12 -10 -24l-319 -319q-10 -9 -23 -9q-12 0 -23 9l-320 320q-15 16 -7 35q8 20 30 20h192v1376q0 14 9 23t23 9h192q14 0 23 -9t9 -23v-1376h192 q14 0 23 -9t9 -23zM1408 480v-192q0 -14 -9 -23t-23 -9h-448q-14 0 -23 9t-9 23v192q0 14 9 23t23 9h448q14 0 23 -9t9 -23zM1600 992v-192q0 -14 -9 -23t-23 -9h-640q-14 0 -23 9t-9 23v192q0 14 9 23t23 9h640q14 0 23 -9t9 -23zM1792 1504v-192q0 -14 -9 -23t-23 -9h-832 q-14 0 -23 9t-9 23v192q0 14 9 23t23 9h832q14 0 23 -9t9 -23z" />
+<glyph unicode="&#xf162;" d="M1346 223q0 63 -44 116t-103 53q-52 0 -83 -37t-31 -94t36.5 -95t104.5 -38q50 0 85 27t35 68zM736 96q0 -12 -10 -24l-319 -319q-10 -9 -23 -9q-12 0 -23 9l-320 320q-15 16 -7 35q8 20 30 20h192v1376q0 14 9 23t23 9h192q14 0 23 -9t9 -23v-1376h192q14 0 23 -9t9 -23 zM1486 165q0 -62 -13 -121.5t-41 -114t-68 -95.5t-98.5 -65.5t-127.5 -24.5q-62 0 -108 16q-24 8 -42 15l39 113q15 -7 31 -11q37 -13 75 -13q84 0 134.5 58.5t66.5 145.5h-2q-21 -23 -61.5 -37t-84.5 -14q-106 0 -173 71.5t-67 172.5q0 105 72 178t181 73q123 0 205 -94.5 t82 -252.5zM1456 882v-114h-469v114h167v432q0 7 0.5 19t0.5 17v16h-2l-7 -12q-8 -13 -26 -31l-62 -58l-82 86l192 185h123v-654h165z" />
+<glyph unicode="&#xf163;" d="M1346 1247q0 63 -44 116t-103 53q-52 0 -83 -37t-31 -94t36.5 -95t104.5 -38q50 0 85 27t35 68zM736 96q0 -12 -10 -24l-319 -319q-10 -9 -23 -9q-12 0 -23 9l-320 320q-15 16 -7 35q8 20 30 20h192v1376q0 14 9 23t23 9h192q14 0 23 -9t9 -23v-1376h192q14 0 23 -9 t9 -23zM1456 -142v-114h-469v114h167v432q0 7 0.5 19t0.5 17v16h-2l-7 -12q-8 -13 -26 -31l-62 -58l-82 86l192 185h123v-654h165zM1486 1189q0 -62 -13 -121.5t-41 -114t-68 -95.5t-98.5 -65.5t-127.5 -24.5q-62 0 -108 16q-24 8 -42 15l39 113q15 -7 31 -11q37 -13 75 -13 q84 0 134.5 58.5t66.5 145.5h-2q-21 -23 -61.5 -37t-84.5 -14q-106 0 -173 71.5t-67 172.5q0 105 72 178t181 73q123 0 205 -94.5t82 -252.5z" />
+<glyph unicode="&#xf164;" horiz-adv-x="1664" d="M256 192q0 26 -19 45t-45 19q-27 0 -45.5 -19t-18.5 -45q0 -27 18.5 -45.5t45.5 -18.5q26 0 45 18.5t19 45.5zM416 704v-640q0 -26 -19 -45t-45 -19h-288q-26 0 -45 19t-19 45v640q0 26 19 45t45 19h288q26 0 45 -19t19 -45zM1600 704q0 -86 -55 -149q15 -44 15 -76 q3 -76 -43 -137q17 -56 0 -117q-15 -57 -54 -94q9 -112 -49 -181q-64 -76 -197 -78h-36h-76h-17q-66 0 -144 15.5t-121.5 29t-120.5 39.5q-123 43 -158 44q-26 1 -45 19.5t-19 44.5v641q0 25 18 43.5t43 20.5q24 2 76 59t101 121q68 87 101 120q18 18 31 48t17.5 48.5 t13.5 60.5q7 39 12.5 61t19.5 52t34 50q19 19 45 19q46 0 82.5 -10.5t60 -26t40 -40.5t24 -45t12 -50t5 -45t0.5 -39q0 -38 -9.5 -76t-19 -60t-27.5 -56q-3 -6 -10 -18t-11 -22t-8 -24h277q78 0 135 -57t57 -135z" />
+<glyph unicode="&#xf165;" horiz-adv-x="1664" d="M256 960q0 -26 -19 -45t-45 -19q-27 0 -45.5 19t-18.5 45q0 27 18.5 45.5t45.5 18.5q26 0 45 -18.5t19 -45.5zM416 448v640q0 26 -19 45t-45 19h-288q-26 0 -45 -19t-19 -45v-640q0 -26 19 -45t45 -19h288q26 0 45 19t19 45zM1545 597q55 -61 55 -149q-1 -78 -57.5 -135 t-134.5 -57h-277q4 -14 8 -24t11 -22t10 -18q18 -37 27 -57t19 -58.5t10 -76.5q0 -24 -0.5 -39t-5 -45t-12 -50t-24 -45t-40 -40.5t-60 -26t-82.5 -10.5q-26 0 -45 19q-20 20 -34 50t-19.5 52t-12.5 61q-9 42 -13.5 60.5t-17.5 48.5t-31 48q-33 33 -101 120q-49 64 -101 121 t-76 59q-25 2 -43 20.5t-18 43.5v641q0 26 19 44.5t45 19.5q35 1 158 44q77 26 120.5 39.5t121.5 29t144 15.5h17h76h36q133 -2 197 -78q58 -69 49 -181q39 -37 54 -94q17 -61 0 -117q46 -61 43 -137q0 -32 -15 -76z" />
+<glyph unicode="&#xf166;" d="M919 233v157q0 50 -29 50q-17 0 -33 -16v-224q16 -16 33 -16q29 0 29 49zM1103 355h66v34q0 51 -33 51t-33 -51v-34zM532 621v-70h-80v-423h-74v423h-78v70h232zM733 495v-367h-67v40q-39 -45 -76 -45q-33 0 -42 28q-6 16 -6 54v290h66v-270q0 -24 1 -26q1 -15 15 -15 q20 0 42 31v280h67zM985 384v-146q0 -52 -7 -73q-12 -42 -53 -42q-35 0 -68 41v-36h-67v493h67v-161q32 40 68 40q41 0 53 -42q7 -21 7 -74zM1236 255v-9q0 -29 -2 -43q-3 -22 -15 -40q-27 -40 -80 -40q-52 0 -81 38q-21 27 -21 86v129q0 59 20 86q29 38 80 38t78 -38 q21 -28 21 -86v-76h-133v-65q0 -51 34 -51q24 0 30 26q0 1 0.5 7t0.5 16.5v21.5h68zM785 1079v-156q0 -51 -32 -51t-32 51v156q0 52 32 52t32 -52zM1318 366q0 177 -19 260q-10 44 -43 73.5t-76 34.5q-136 15 -412 15q-275 0 -411 -15q-44 -5 -76.5 -34.5t-42.5 -73.5 q-20 -87 -20 -260q0 -176 20 -260q10 -43 42.5 -73t75.5 -35q137 -15 412 -15t412 15q43 5 75.5 35t42.5 73q20 84 20 260zM563 1017l90 296h-75l-51 -195l-53 195h-78l24 -69t23 -69q35 -103 46 -158v-201h74v201zM852 936v130q0 58 -21 87q-29 38 -78 38q-51 0 -78 -38 q-21 -29 -21 -87v-130q0 -58 21 -87q27 -38 78 -38q49 0 78 38q21 27 21 87zM1033 816h67v370h-67v-283q-22 -31 -42 -31q-15 0 -16 16q-1 2 -1 26v272h-67v-293q0 -37 6 -55q11 -27 43 -27q36 0 77 45v-40zM1536 1120v-960q0 -119 -84.5 -203.5t-203.5 -84.5h-960 q-119 0 -203.5 84.5t-84.5 203.5v960q0 119 84.5 203.5t203.5 84.5h960q119 0 203.5 -84.5t84.5 -203.5z" />
+<glyph unicode="&#xf167;" d="M971 292v-211q0 -67 -39 -67q-23 0 -45 22v301q22 22 45 22q39 0 39 -67zM1309 291v-46h-90v46q0 68 45 68t45 -68zM343 509h107v94h-312v-94h105v-569h100v569zM631 -60h89v494h-89v-378q-30 -42 -57 -42q-18 0 -21 21q-1 3 -1 35v364h-89v-391q0 -49 8 -73 q12 -37 58 -37q48 0 102 61v-54zM1060 88v197q0 73 -9 99q-17 56 -71 56q-50 0 -93 -54v217h-89v-663h89v48q45 -55 93 -55q54 0 71 55q9 27 9 100zM1398 98v13h-91q0 -51 -2 -61q-7 -36 -40 -36q-46 0 -46 69v87h179v103q0 79 -27 116q-39 51 -106 51q-68 0 -107 -51 q-28 -37 -28 -116v-173q0 -79 29 -116q39 -51 108 -51q72 0 108 53q18 27 21 54q2 9 2 58zM790 1011v210q0 69 -43 69t-43 -69v-210q0 -70 43 -70t43 70zM1509 260q0 -234 -26 -350q-14 -59 -58 -99t-102 -46q-184 -21 -555 -21t-555 21q-58 6 -102.5 46t-57.5 99 q-26 112 -26 350q0 234 26 350q14 59 58 99t103 47q183 20 554 20t555 -20q58 -7 102.5 -47t57.5 -99q26 -112 26 -350zM511 1536h102l-121 -399v-271h-100v271q-14 74 -61 212q-37 103 -65 187h106l71 -263zM881 1203v-175q0 -81 -28 -118q-37 -51 -106 -51q-67 0 -105 51 q-28 38 -28 118v175q0 80 28 117q38 51 105 51q69 0 106 -51q28 -37 28 -117zM1216 1365v-499h-91v55q-53 -62 -103 -62q-46 0 -59 37q-8 24 -8 75v394h91v-367q0 -33 1 -35q3 -22 21 -22q27 0 57 43v381h91z" />
+<glyph unicode="&#xf168;" horiz-adv-x="1408" d="M597 869q-10 -18 -257 -456q-27 -46 -65 -46h-239q-21 0 -31 17t0 36l253 448q1 0 0 1l-161 279q-12 22 -1 37q9 15 32 15h239q40 0 66 -45zM1403 1511q11 -16 0 -37l-528 -934v-1l336 -615q11 -20 1 -37q-10 -15 -32 -15h-239q-42 0 -66 45l-339 622q18 32 531 942 q25 45 64 45h241q22 0 31 -15z" />
+<glyph unicode="&#xf169;" d="M685 771q0 1 -126 222q-21 34 -52 34h-184q-18 0 -26 -11q-7 -12 1 -29l125 -216v-1l-196 -346q-9 -14 0 -28q8 -13 24 -13h185q31 0 50 36zM1309 1268q-7 12 -24 12h-187q-30 0 -49 -35l-411 -729q1 -2 262 -481q20 -35 52 -35h184q18 0 25 12q8 13 -1 28l-260 476v1 l409 723q8 16 0 28zM1536 1120v-960q0 -119 -84.5 -203.5t-203.5 -84.5h-960q-119 0 -203.5 84.5t-84.5 203.5v960q0 119 84.5 203.5t203.5 84.5h960q119 0 203.5 -84.5t84.5 -203.5z" />
+<glyph unicode="&#xf16a;" horiz-adv-x="1792" d="M1280 640q0 37 -30 54l-512 320q-31 20 -65 2q-33 -18 -33 -56v-640q0 -38 33 -56q16 -8 31 -8q20 0 34 10l512 320q30 17 30 54zM1792 640q0 -96 -1 -150t-8.5 -136.5t-22.5 -147.5q-16 -73 -69 -123t-124 -58q-222 -25 -671 -25t-671 25q-71 8 -124.5 58t-69.5 123 q-14 65 -21.5 147.5t-8.5 136.5t-1 150t1 150t8.5 136.5t22.5 147.5q16 73 69 123t124 58q222 25 671 25t671 -25q71 -8 124.5 -58t69.5 -123q14 -65 21.5 -147.5t8.5 -136.5t1 -150z" />
+<glyph unicode="&#xf16b;" horiz-adv-x="1792" d="M402 829l494 -305l-342 -285l-490 319zM1388 274v-108l-490 -293v-1l-1 1l-1 -1v1l-489 293v108l147 -96l342 284v2l1 -1l1 1v-2l343 -284zM554 1418l342 -285l-494 -304l-338 270zM1390 829l338 -271l-489 -319l-343 285zM1239 1418l489 -319l-338 -270l-494 304z" />
+<glyph unicode="&#xf16c;" horiz-adv-x="1408" d="M928 135v-151l-707 -1v151zM1169 481v-701l-1 -35v-1h-1132l-35 1h-1v736h121v-618h928v618h120zM241 393l704 -65l-13 -150l-705 65zM309 709l683 -183l-39 -146l-683 183zM472 1058l609 -360l-77 -130l-609 360zM832 1389l398 -585l-124 -85l-399 584zM1285 1536 l121 -697l-149 -26l-121 697z" />
+<glyph unicode="&#xf16d;" d="M1362 110v648h-135q20 -63 20 -131q0 -126 -64 -232.5t-174 -168.5t-240 -62q-197 0 -337 135.5t-140 327.5q0 68 20 131h-141v-648q0 -26 17.5 -43.5t43.5 -17.5h1069q25 0 43 17.5t18 43.5zM1078 643q0 124 -90.5 211.5t-218.5 87.5q-127 0 -217.5 -87.5t-90.5 -211.5 t90.5 -211.5t217.5 -87.5q128 0 218.5 87.5t90.5 211.5zM1362 1003v165q0 28 -20 48.5t-49 20.5h-174q-29 0 -49 -20.5t-20 -48.5v-165q0 -29 20 -49t49 -20h174q29 0 49 20t20 49zM1536 1211v-1142q0 -81 -58 -139t-139 -58h-1142q-81 0 -139 58t-58 139v1142q0 81 58 139 t139 58h1142q81 0 139 -58t58 -139z" />
+<glyph unicode="&#xf16e;" d="M1248 1408q119 0 203.5 -84.5t84.5 -203.5v-960q0 -119 -84.5 -203.5t-203.5 -84.5h-960q-119 0 -203.5 84.5t-84.5 203.5v960q0 119 84.5 203.5t203.5 84.5h960zM698 640q0 88 -62 150t-150 62t-150 -62t-62 -150t62 -150t150 -62t150 62t62 150zM1262 640q0 88 -62 150 t-150 62t-150 -62t-62 -150t62 -150t150 -62t150 62t62 150z" />
+<glyph unicode="&#xf170;" d="M768 914l201 -306h-402zM1133 384h94l-459 691l-459 -691h94l104 160h522zM1536 640q0 -209 -103 -385.5t-279.5 -279.5t-385.5 -103t-385.5 103t-279.5 279.5t-103 385.5t103 385.5t279.5 279.5t385.5 103t385.5 -103t279.5 -279.5t103 -385.5z" />
+<glyph unicode="&#xf171;" horiz-adv-x="1408" d="M815 677q8 -63 -50.5 -101t-111.5 -6q-39 17 -53.5 58t-0.5 82t52 58q36 18 72.5 12t64 -35.5t27.5 -67.5zM926 698q-14 107 -113 164t-197 13q-63 -28 -100.5 -88.5t-34.5 -129.5q4 -91 77.5 -155t165.5 -56q91 8 152 84t50 168zM1165 1240q-20 27 -56 44.5t-58 22 t-71 12.5q-291 47 -566 -2q-43 -7 -66 -12t-55 -22t-50 -43q30 -28 76 -45.5t73.5 -22t87.5 -11.5q228 -29 448 -1q63 8 89.5 12t72.5 21.5t75 46.5zM1222 205q-8 -26 -15.5 -76.5t-14 -84t-28.5 -70t-58 -56.5q-86 -48 -189.5 -71.5t-202 -22t-201.5 18.5q-46 8 -81.5 18 t-76.5 27t-73 43.5t-52 61.5q-25 96 -57 292l6 16l18 9q223 -148 506.5 -148t507.5 148q21 -6 24 -23t-5 -45t-8 -37zM1403 1166q-26 -167 -111 -655q-5 -30 -27 -56t-43.5 -40t-54.5 -31q-252 -126 -610 -88q-248 27 -394 139q-15 12 -25.5 26.5t-17 35t-9 34t-6 39.5 t-5.5 35q-9 50 -26.5 150t-28 161.5t-23.5 147.5t-22 158q3 26 17.5 48.5t31.5 37.5t45 30t46 22.5t48 18.5q125 46 313 64q379 37 676 -50q155 -46 215 -122q16 -20 16.5 -51t-5.5 -54z" />
+<glyph unicode="&#xf172;" d="M848 666q0 43 -41 66t-77 1q-43 -20 -42.5 -72.5t43.5 -70.5q39 -23 81 4t36 72zM928 682q8 -66 -36 -121t-110 -61t-119 40t-56 113q-2 49 25.5 93t72.5 64q70 31 141.5 -10t81.5 -118zM1100 1073q-20 -21 -53.5 -34t-53 -16t-63.5 -8q-155 -20 -324 0q-44 6 -63 9.5 t-52.5 16t-54.5 32.5q13 19 36 31t40 15.5t47 8.5q198 35 408 1q33 -5 51 -8.5t43 -16t39 -31.5zM1142 327q0 7 5.5 26.5t3 32t-17.5 16.5q-161 -106 -365 -106t-366 106l-12 -6l-5 -12q26 -154 41 -210q47 -81 204 -108q249 -46 428 53q34 19 49 51.5t22.5 85.5t12.5 71z M1272 1020q9 53 -8 75q-43 55 -155 88q-216 63 -487 36q-132 -12 -226 -46q-38 -15 -59.5 -25t-47 -34t-29.5 -54q8 -68 19 -138t29 -171t24 -137q1 -5 5 -31t7 -36t12 -27t22 -28q105 -80 284 -100q259 -28 440 63q24 13 39.5 23t31 29t19.5 40q48 267 80 473zM1536 1120 v-960q0 -119 -84.5 -203.5t-203.5 -84.5h-960q-119 0 -203.5 84.5t-84.5 203.5v960q0 119 84.5 203.5t203.5 84.5h960q119 0 203.5 -84.5t84.5 -203.5z" />
+<glyph unicode="&#xf173;" horiz-adv-x="1024" d="M390 1408h219v-388h364v-241h-364v-394q0 -136 14 -172q13 -37 52 -60q50 -31 117 -31q117 0 232 76v-242q-102 -48 -178 -65q-77 -19 -173 -19q-105 0 -186 27q-78 25 -138 75q-58 51 -79 105q-22 54 -22 161v539h-170v217q91 30 155 84q64 55 103 132q39 78 54 196z " />
+<glyph unicode="&#xf174;" d="M1123 127v181q-88 -56 -174 -56q-51 0 -88 23q-29 17 -39 45q-11 30 -11 129v295h274v181h-274v291h-164q-11 -90 -40 -147t-78 -99q-48 -40 -116 -63v-163h127v-404q0 -78 17 -121q17 -42 59 -78q43 -37 104 -57q62 -20 140 -20q67 0 129 14q57 13 134 49zM1536 1120 v-960q0 -119 -84.5 -203.5t-203.5 -84.5h-960q-119 0 -203.5 84.5t-84.5 203.5v960q0 119 84.5 203.5t203.5 84.5h960q119 0 203.5 -84.5t84.5 -203.5z" />
+<glyph unicode="&#xf175;" horiz-adv-x="768" d="M765 237q8 -19 -5 -35l-350 -384q-10 -10 -23 -10q-14 0 -24 10l-355 384q-13 16 -5 35q9 19 29 19h224v1248q0 14 9 23t23 9h192q14 0 23 -9t9 -23v-1248h224q21 0 29 -19z" />
+<glyph unicode="&#xf176;" horiz-adv-x="768" d="M765 1043q-9 -19 -29 -19h-224v-1248q0 -14 -9 -23t-23 -9h-192q-14 0 -23 9t-9 23v1248h-224q-21 0 -29 19t5 35l350 384q10 10 23 10q14 0 24 -10l355 -384q13 -16 5 -35z" />
+<glyph unicode="&#xf177;" horiz-adv-x="1792" d="M1792 736v-192q0 -14 -9 -23t-23 -9h-1248v-224q0 -21 -19 -29t-35 5l-384 350q-10 10 -10 23q0 14 10 24l384 354q16 14 35 6q19 -9 19 -29v-224h1248q14 0 23 -9t9 -23z" />
+<glyph unicode="&#xf178;" horiz-adv-x="1792" d="M1728 643q0 -14 -10 -24l-384 -354q-16 -14 -35 -6q-19 9 -19 29v224h-1248q-14 0 -23 9t-9 23v192q0 14 9 23t23 9h1248v224q0 21 19 29t35 -5l384 -350q10 -10 10 -23z" />
+<glyph unicode="&#xf179;" horiz-adv-x="1408" d="M1393 321q-39 -125 -123 -250q-129 -196 -257 -196q-49 0 -140 32q-86 32 -151 32q-61 0 -142 -33q-81 -34 -132 -34q-152 0 -301 259q-147 261 -147 503q0 228 113 374q112 144 284 144q72 0 177 -30q104 -30 138 -30q45 0 143 34q102 34 173 34q119 0 213 -65 q52 -36 104 -100q-79 -67 -114 -118q-65 -94 -65 -207q0 -124 69 -223t158 -126zM1017 1494q0 -61 -29 -136q-30 -75 -93 -138q-54 -54 -108 -72q-37 -11 -104 -17q3 149 78 257q74 107 250 148q1 -3 2.5 -11t2.5 -11q0 -4 0.5 -10t0.5 -10z" />
+<glyph unicode="&#xf17a;" horiz-adv-x="1664" d="M682 530v-651l-682 94v557h682zM682 1273v-659h-682v565zM1664 530v-786l-907 125v661h907zM1664 1408v-794h-907v669z" />
+<glyph unicode="&#xf17b;" horiz-adv-x="1408" d="M493 1053q16 0 27.5 11.5t11.5 27.5t-11.5 27.5t-27.5 11.5t-27 -11.5t-11 -27.5t11 -27.5t27 -11.5zM915 1053q16 0 27 11.5t11 27.5t-11 27.5t-27 11.5t-27.5 -11.5t-11.5 -27.5t11.5 -27.5t27.5 -11.5zM103 869q42 0 72 -30t30 -72v-430q0 -43 -29.5 -73t-72.5 -30 t-73 30t-30 73v430q0 42 30 72t73 30zM1163 850v-666q0 -46 -32 -78t-77 -32h-75v-227q0 -43 -30 -73t-73 -30t-73 30t-30 73v227h-138v-227q0 -43 -30 -73t-73 -30q-42 0 -72 30t-30 73l-1 227h-74q-46 0 -78 32t-32 78v666h918zM931 1255q107 -55 171 -153.5t64 -215.5 h-925q0 117 64 215.5t172 153.5l-71 131q-7 13 5 20q13 6 20 -6l72 -132q95 42 201 42t201 -42l72 132q7 12 20 6q12 -7 5 -20zM1408 767v-430q0 -43 -30 -73t-73 -30q-42 0 -72 30t-30 73v430q0 43 30 72.5t72 29.5q43 0 73 -29.5t30 -72.5z" />
+<glyph unicode="&#xf17c;" d="M663 1125q-11 -1 -15.5 -10.5t-8.5 -9.5q-5 -1 -5 5q0 12 19 15h10zM750 1111q-4 -1 -11.5 6.5t-17.5 4.5q24 11 32 -2q3 -6 -3 -9zM399 684q-4 1 -6 -3t-4.5 -12.5t-5.5 -13.5t-10 -13q-7 -10 -1 -12q4 -1 12.5 7t12.5 18q1 3 2 7t2 6t1.5 4.5t0.5 4v3t-1 2.5t-3 2z M1254 325q0 18 -55 42q4 15 7.5 27.5t5 26t3 21.5t0.5 22.5t-1 19.5t-3.5 22t-4 20.5t-5 25t-5.5 26.5q-10 48 -47 103t-72 75q24 -20 57 -83q87 -162 54 -278q-11 -40 -50 -42q-31 -4 -38.5 18.5t-8 83.5t-11.5 107q-9 39 -19.5 69t-19.5 45.5t-15.5 24.5t-13 15t-7.5 7 q-14 62 -31 103t-29.5 56t-23.5 33t-15 40q-4 21 6 53.5t4.5 49.5t-44.5 25q-15 3 -44.5 18t-35.5 16q-8 1 -11 26t8 51t36 27q37 3 51 -30t4 -58q-11 -19 -2 -26.5t30 -0.5q13 4 13 36v37q-5 30 -13.5 50t-21 30.5t-23.5 15t-27 7.5q-107 -8 -89 -134q0 -15 -1 -15 q-9 9 -29.5 10.5t-33 -0.5t-15.5 5q1 57 -16 90t-45 34q-27 1 -41.5 -27.5t-16.5 -59.5q-1 -15 3.5 -37t13 -37.5t15.5 -13.5q10 3 16 14q4 9 -7 8q-7 0 -15.5 14.5t-9.5 33.5q-1 22 9 37t34 14q17 0 27 -21t9.5 -39t-1.5 -22q-22 -15 -31 -29q-8 -12 -27.5 -23.5 t-20.5 -12.5q-13 -14 -15.5 -27t7.5 -18q14 -8 25 -19.5t16 -19t18.5 -13t35.5 -6.5q47 -2 102 15q2 1 23 7t34.5 10.5t29.5 13t21 17.5q9 14 20 8q5 -3 6.5 -8.5t-3 -12t-16.5 -9.5q-20 -6 -56.5 -21.5t-45.5 -19.5q-44 -19 -70 -23q-25 -5 -79 2q-10 2 -9 -2t17 -19 q25 -23 67 -22q17 1 36 7t36 14t33.5 17.5t30 17t24.5 12t17.5 2.5t8.5 -11q0 -2 -1 -4.5t-4 -5t-6 -4.5t-8.5 -5t-9 -4.5t-10 -5t-9.5 -4.5q-28 -14 -67.5 -44t-66.5 -43t-49 -1q-21 11 -63 73q-22 31 -25 22q-1 -3 -1 -10q0 -25 -15 -56.5t-29.5 -55.5t-21 -58t11.5 -63 q-23 -6 -62.5 -90t-47.5 -141q-2 -18 -1.5 -69t-5.5 -59q-8 -24 -29 -3q-32 31 -36 94q-2 28 4 56q4 19 -1 18l-4 -5q-36 -65 10 -166q5 -12 25 -28t24 -20q20 -23 104 -90.5t93 -76.5q16 -15 17.5 -38t-14 -43t-45.5 -23q8 -15 29 -44.5t28 -54t7 -70.5q46 24 7 92 q-4 8 -10.5 16t-9.5 12t-2 6q3 5 13 9.5t20 -2.5q46 -52 166 -36q133 15 177 87q23 38 34 30q12 -6 10 -52q-1 -25 -23 -92q-9 -23 -6 -37.5t24 -15.5q3 19 14.5 77t13.5 90q2 21 -6.5 73.5t-7.5 97t23 70.5q15 18 51 18q1 37 34.5 53t72.5 10.5t60 -22.5zM626 1152 q3 17 -2.5 30t-11.5 15q-9 2 -9 -7q2 -5 5 -6q10 0 7 -15q-3 -20 8 -20q3 0 3 3zM1045 955q-2 8 -6.5 11.5t-13 5t-14.5 5.5q-5 3 -9.5 8t-7 8t-5.5 6.5t-4 4t-4 -1.5q-14 -16 7 -43.5t39 -31.5q9 -1 14.5 8t3.5 20zM867 1168q0 11 -5 19.5t-11 12.5t-9 3q-14 -1 -7 -7l4 -2 q14 -4 18 -31q0 -3 8 2zM921 1401q0 2 -2.5 5t-9 7t-9.5 6q-15 15 -24 15q-9 -1 -11.5 -7.5t-1 -13t-0.5 -12.5q-1 -4 -6 -10.5t-6 -9t3 -8.5q4 -3 8 0t11 9t15 9q1 1 9 1t15 2t9 7zM1486 60q20 -12 31 -24.5t12 -24t-2.5 -22.5t-15.5 -22t-23.5 -19.5t-30 -18.5 t-31.5 -16.5t-32 -15.5t-27 -13q-38 -19 -85.5 -56t-75.5 -64q-17 -16 -68 -19.5t-89 14.5q-18 9 -29.5 23.5t-16.5 25.5t-22 19.5t-47 9.5q-44 1 -130 1q-19 0 -57 -1.5t-58 -2.5q-44 -1 -79.5 -15t-53.5 -30t-43.5 -28.5t-53.5 -11.5q-29 1 -111 31t-146 43q-19 4 -51 9.5 t-50 9t-39.5 9.5t-33.5 14.5t-17 19.5q-10 23 7 66.5t18 54.5q1 16 -4 40t-10 42.5t-4.5 36.5t10.5 27q14 12 57 14t60 12q30 18 42 35t12 51q21 -73 -32 -106q-32 -20 -83 -15q-34 3 -43 -10q-13 -15 5 -57q2 -6 8 -18t8.5 -18t4.5 -17t1 -22q0 -15 -17 -49t-14 -48 q3 -17 37 -26q20 -6 84.5 -18.5t99.5 -20.5q24 -6 74 -22t82.5 -23t55.5 -4q43 6 64.5 28t23 48t-7.5 58.5t-19 52t-20 36.5q-121 190 -169 242q-68 74 -113 40q-11 -9 -15 15q-3 16 -2 38q1 29 10 52t24 47t22 42q8 21 26.5 72t29.5 78t30 61t39 54q110 143 124 195 q-12 112 -16 310q-2 90 24 151.5t106 104.5q39 21 104 21q53 1 106 -13.5t89 -41.5q57 -42 91.5 -121.5t29.5 -147.5q-5 -95 30 -214q34 -113 133 -218q55 -59 99.5 -163t59.5 -191q8 -49 5 -84.5t-12 -55.5t-20 -22q-10 -2 -23.5 -19t-27 -35.5t-40.5 -33.5t-61 -14 q-18 1 -31.5 5t-22.5 13.5t-13.5 15.5t-11.5 20.5t-9 19.5q-22 37 -41 30t-28 -49t7 -97q20 -70 1 -195q-10 -65 18 -100.5t73 -33t85 35.5q59 49 89.5 66.5t103.5 42.5q53 18 77 36.5t18.5 34.5t-25 28.5t-51.5 23.5q-33 11 -49.5 48t-15 72.5t15.5 47.5q1 -31 8 -56.5 t14.5 -40.5t20.5 -28.5t21 -19t21.5 -13t16.5 -9.5z" />
+<glyph unicode="&#xf17d;" d="M1024 36q-42 241 -140 498h-2l-2 -1q-16 -6 -43 -16.5t-101 -49t-137 -82t-131 -114.5t-103 -148l-15 11q184 -150 418 -150q132 0 256 52zM839 643q-21 49 -53 111q-311 -93 -673 -93q-1 -7 -1 -21q0 -124 44 -236.5t124 -201.5q50 89 123.5 166.5t142.5 124.5t130.5 81 t99.5 48l37 13q4 1 13 3.5t13 4.5zM732 855q-120 213 -244 378q-138 -65 -234 -186t-128 -272q302 0 606 80zM1416 536q-210 60 -409 29q87 -239 128 -469q111 75 185 189.5t96 250.5zM611 1277q-1 0 -2 -1q1 1 2 1zM1201 1132q-185 164 -433 164q-76 0 -155 -19 q131 -170 246 -382q69 26 130 60.5t96.5 61.5t65.5 57t37.5 40.5zM1424 647q-3 232 -149 410l-1 -1q-9 -12 -19 -24.5t-43.5 -44.5t-71 -60.5t-100 -65t-131.5 -64.5q25 -53 44 -95q2 -6 6.5 -17.5t7.5 -16.5q36 5 74.5 7t73.5 2t69 -1.5t64 -4t56.5 -5.5t48 -6.5t36.5 -6 t25 -4.5zM1536 640q0 -209 -103 -385.5t-279.5 -279.5t-385.5 -103t-385.5 103t-279.5 279.5t-103 385.5t103 385.5t279.5 279.5t385.5 103t385.5 -103t279.5 -279.5t103 -385.5z" />
+<glyph unicode="&#xf17e;" d="M1173 473q0 50 -19.5 91.5t-48.5 68.5t-73 49t-82.5 34t-87.5 23l-104 24q-30 7 -44 10.5t-35 11.5t-30 16t-16.5 21t-7.5 30q0 77 144 77q43 0 77 -12t54 -28.5t38 -33.5t40 -29t48 -12q47 0 75.5 32t28.5 77q0 55 -56 99.5t-142 67.5t-182 23q-68 0 -132 -15.5 t-119.5 -47t-89 -87t-33.5 -128.5q0 -61 19 -106.5t56 -75.5t80 -48.5t103 -32.5l146 -36q90 -22 112 -36q32 -20 32 -60q0 -39 -40 -64.5t-105 -25.5q-51 0 -91.5 16t-65 38.5t-45.5 45t-46 38.5t-54 16q-50 0 -75.5 -30t-25.5 -75q0 -92 122 -157.5t291 -65.5 q73 0 140 18.5t122.5 53.5t88.5 93.5t33 131.5zM1536 256q0 -159 -112.5 -271.5t-271.5 -112.5q-130 0 -234 80q-77 -16 -150 -16q-143 0 -273.5 55.5t-225 150t-150 225t-55.5 273.5q0 73 16 150q-80 104 -80 234q0 159 112.5 271.5t271.5 112.5q130 0 234 -80 q77 16 150 16q143 0 273.5 -55.5t225 -150t150 -225t55.5 -273.5q0 -73 -16 -150q80 -104 80 -234z" />
+<glyph unicode="&#xf180;" horiz-adv-x="1664" d="M1483 512l-587 -587q-52 -53 -127.5 -53t-128.5 53l-587 587q-53 53 -53 128t53 128l587 587q53 53 128 53t128 -53l265 -265l-398 -399l-188 188q-42 42 -99 42q-59 0 -100 -41l-120 -121q-42 -40 -42 -99q0 -58 42 -100l406 -408q30 -28 67 -37l6 -4h28q60 0 99 41 l619 619l2 -3q53 -53 53 -128t-53 -128zM1406 1138l120 -120q14 -15 14 -36t-14 -36l-730 -730q-17 -15 -37 -15v0q-4 0 -6 1q-18 2 -30 14l-407 408q-14 15 -14 36t14 35l121 120q13 15 35 15t36 -15l252 -252l574 575q15 15 36 15t36 -15z" />
+<glyph unicode="&#xf181;" d="M704 192v1024q0 14 -9 23t-23 9h-480q-14 0 -23 -9t-9 -23v-1024q0 -14 9 -23t23 -9h480q14 0 23 9t9 23zM1376 576v640q0 14 -9 23t-23 9h-480q-14 0 -23 -9t-9 -23v-640q0 -14 9 -23t23 -9h480q14 0 23 9t9 23zM1536 1344v-1408q0 -26 -19 -45t-45 -19h-1408 q-26 0 -45 19t-19 45v1408q0 26 19 45t45 19h1408q26 0 45 -19t19 -45z" />
+<glyph unicode="&#xf182;" horiz-adv-x="1280" d="M1280 480q0 -40 -28 -68t-68 -28q-51 0 -80 43l-227 341h-45v-132l247 -411q9 -15 9 -33q0 -26 -19 -45t-45 -19h-192v-272q0 -46 -33 -79t-79 -33h-160q-46 0 -79 33t-33 79v272h-192q-26 0 -45 19t-19 45q0 18 9 33l247 411v132h-45l-227 -341q-29 -43 -80 -43 q-40 0 -68 28t-28 68q0 29 16 53l256 384q73 107 176 107h384q103 0 176 -107l256 -384q16 -24 16 -53zM864 1280q0 -93 -65.5 -158.5t-158.5 -65.5t-158.5 65.5t-65.5 158.5t65.5 158.5t158.5 65.5t158.5 -65.5t65.5 -158.5z" />
+<glyph unicode="&#xf183;" horiz-adv-x="1024" d="M1024 832v-416q0 -40 -28 -68t-68 -28t-68 28t-28 68v352h-64v-912q0 -46 -33 -79t-79 -33t-79 33t-33 79v464h-64v-464q0 -46 -33 -79t-79 -33t-79 33t-33 79v912h-64v-352q0 -40 -28 -68t-68 -28t-68 28t-28 68v416q0 80 56 136t136 56h640q80 0 136 -56t56 -136z M736 1280q0 -93 -65.5 -158.5t-158.5 -65.5t-158.5 65.5t-65.5 158.5t65.5 158.5t158.5 65.5t158.5 -65.5t65.5 -158.5z" />
+<glyph unicode="&#xf184;" d="M773 234l350 473q16 22 24.5 59t-6 85t-61.5 79q-40 26 -83 25.5t-73.5 -17.5t-54.5 -45q-36 -40 -96 -40q-59 0 -95 40q-24 28 -54.5 45t-73.5 17.5t-84 -25.5q-46 -31 -60.5 -79t-6 -85t24.5 -59zM1536 640q0 -209 -103 -385.5t-279.5 -279.5t-385.5 -103t-385.5 103 t-279.5 279.5t-103 385.5t103 385.5t279.5 279.5t385.5 103t385.5 -103t279.5 -279.5t103 -385.5z" />
+<glyph unicode="&#xf185;" horiz-adv-x="1792" d="M1472 640q0 117 -45.5 223.5t-123 184t-184 123t-223.5 45.5t-223.5 -45.5t-184 -123t-123 -184t-45.5 -223.5t45.5 -223.5t123 -184t184 -123t223.5 -45.5t223.5 45.5t184 123t123 184t45.5 223.5zM1748 363q-4 -15 -20 -20l-292 -96v-306q0 -16 -13 -26q-15 -10 -29 -4 l-292 94l-180 -248q-10 -13 -26 -13t-26 13l-180 248l-292 -94q-14 -6 -29 4q-13 10 -13 26v306l-292 96q-16 5 -20 20q-5 17 4 29l180 248l-180 248q-9 13 -4 29q4 15 20 20l292 96v306q0 16 13 26q15 10 29 4l292 -94l180 248q9 12 26 12t26 -12l180 -248l292 94 q14 6 29 -4q13 -10 13 -26v-306l292 -96q16 -5 20 -20q5 -16 -4 -29l-180 -248l180 -248q9 -12 4 -29z" />
+<glyph unicode="&#xf186;" d="M1262 233q-54 -9 -110 -9q-182 0 -337 90t-245 245t-90 337q0 192 104 357q-201 -60 -328.5 -229t-127.5 -384q0 -130 51 -248.5t136.5 -204t204 -136.5t248.5 -51q144 0 273.5 61.5t220.5 171.5zM1465 318q-94 -203 -283.5 -324.5t-413.5 -121.5q-156 0 -298 61 t-245 164t-164 245t-61 298q0 153 57.5 292.5t156 241.5t235.5 164.5t290 68.5q44 2 61 -39q18 -41 -15 -72q-86 -78 -131.5 -181.5t-45.5 -218.5q0 -148 73 -273t198 -198t273 -73q118 0 228 51q41 18 72 -13q14 -14 17.5 -34t-4.5 -38z" />
+<glyph unicode="&#xf187;" horiz-adv-x="1792" d="M1088 704q0 26 -19 45t-45 19h-256q-26 0 -45 -19t-19 -45t19 -45t45 -19h256q26 0 45 19t19 45zM1664 896v-960q0 -26 -19 -45t-45 -19h-1408q-26 0 -45 19t-19 45v960q0 26 19 45t45 19h1408q26 0 45 -19t19 -45zM1728 1344v-256q0 -26 -19 -45t-45 -19h-1536 q-26 0 -45 19t-19 45v256q0 26 19 45t45 19h1536q26 0 45 -19t19 -45z" />
+<glyph unicode="&#xf188;" horiz-adv-x="1664" d="M1632 576q0 -26 -19 -45t-45 -19h-224q0 -171 -67 -290l208 -209q19 -19 19 -45t-19 -45q-18 -19 -45 -19t-45 19l-198 197q-5 -5 -15 -13t-42 -28.5t-65 -36.5t-82 -29t-97 -13v896h-128v-896q-51 0 -101.5 13.5t-87 33t-66 39t-43.5 32.5l-15 14l-183 -207 q-20 -21 -48 -21q-24 0 -43 16q-19 18 -20.5 44.5t15.5 46.5l202 227q-58 114 -58 274h-224q-26 0 -45 19t-19 45t19 45t45 19h224v294l-173 173q-19 19 -19 45t19 45t45 19t45 -19l173 -173h844l173 173q19 19 45 19t45 -19t19 -45t-19 -45l-173 -173v-294h224q26 0 45 -19 t19 -45zM1152 1152h-640q0 133 93.5 226.5t226.5 93.5t226.5 -93.5t93.5 -226.5z" />
+<glyph unicode="&#xf189;" horiz-adv-x="1920" d="M1917 1016q23 -64 -150 -294q-24 -32 -65 -85q-78 -100 -90 -131q-17 -41 14 -81q17 -21 81 -82h1l1 -1l1 -1l2 -2q141 -131 191 -221q3 -5 6.5 -12.5t7 -26.5t-0.5 -34t-25 -27.5t-59 -12.5l-256 -4q-24 -5 -56 5t-52 22l-20 12q-30 21 -70 64t-68.5 77.5t-61 58 t-56.5 15.5q-3 -1 -8 -3.5t-17 -14.5t-21.5 -29.5t-17 -52t-6.5 -77.5q0 -15 -3.5 -27.5t-7.5 -18.5l-4 -5q-18 -19 -53 -22h-115q-71 -4 -146 16.5t-131.5 53t-103 66t-70.5 57.5l-25 24q-10 10 -27.5 30t-71.5 91t-106 151t-122.5 211t-130.5 272q-6 16 -6 27t3 16l4 6 q15 19 57 19l274 2q12 -2 23 -6.5t16 -8.5l5 -3q16 -11 24 -32q20 -50 46 -103.5t41 -81.5l16 -29q29 -60 56 -104t48.5 -68.5t41.5 -38.5t34 -14t27 5q2 1 5 5t12 22t13.5 47t9.5 81t0 125q-2 40 -9 73t-14 46l-6 12q-25 34 -85 43q-13 2 5 24q17 19 38 30q53 26 239 24 q82 -1 135 -13q20 -5 33.5 -13.5t20.5 -24t10.5 -32t3.5 -45.5t-1 -55t-2.5 -70.5t-1.5 -82.5q0 -11 -1 -42t-0.5 -48t3.5 -40.5t11.5 -39t22.5 -24.5q8 -2 17 -4t26 11t38 34.5t52 67t68 107.5q60 104 107 225q4 10 10 17.5t11 10.5l4 3l5 2.5t13 3t20 0.5l288 2 q39 5 64 -2.5t31 -16.5z" />
+<glyph unicode="&#xf18a;" horiz-adv-x="1792" d="M675 252q21 34 11 69t-45 50q-34 14 -73 1t-60 -46q-22 -34 -13 -68.5t43 -50.5t74.5 -2.5t62.5 47.5zM769 373q8 13 3.5 26.5t-17.5 18.5q-14 5 -28.5 -0.5t-21.5 -18.5q-17 -31 13 -45q14 -5 29 0.5t22 18.5zM943 266q-45 -102 -158 -150t-224 -12 q-107 34 -147.5 126.5t6.5 187.5q47 93 151.5 139t210.5 19q111 -29 158.5 -119.5t2.5 -190.5zM1255 426q-9 96 -89 170t-208.5 109t-274.5 21q-223 -23 -369.5 -141.5t-132.5 -264.5q9 -96 89 -170t208.5 -109t274.5 -21q223 23 369.5 141.5t132.5 264.5zM1563 422 q0 -68 -37 -139.5t-109 -137t-168.5 -117.5t-226 -83t-270.5 -31t-275 33.5t-240.5 93t-171.5 151t-65 199.5q0 115 69.5 245t197.5 258q169 169 341.5 236t246.5 -7q65 -64 20 -209q-4 -14 -1 -20t10 -7t14.5 0.5t13.5 3.5l6 2q139 59 246 59t153 -61q45 -63 0 -178 q-2 -13 -4.5 -20t4.5 -12.5t12 -7.5t17 -6q57 -18 103 -47t80 -81.5t34 -116.5zM1489 1046q42 -47 54.5 -108.5t-6.5 -117.5q-8 -23 -29.5 -34t-44.5 -4q-23 8 -34 29.5t-4 44.5q20 63 -24 111t-107 35q-24 -5 -45 8t-25 37q-5 24 8 44.5t37 25.5q60 13 119 -5.5t101 -65.5z M1670 1209q87 -96 112.5 -222.5t-13.5 -241.5q-9 -27 -34 -40t-52 -4t-40 34t-5 52q28 82 10 172t-80 158q-62 69 -148 95.5t-173 8.5q-28 -6 -52 9.5t-30 43.5t9.5 51.5t43.5 29.5q123 26 244 -11.5t208 -134.5z" />
+<glyph unicode="&#xf18b;" d="M1133 -34q-171 -94 -368 -94q-196 0 -367 94q138 87 235.5 211t131.5 268q35 -144 132.5 -268t235.5 -211zM638 1394v-485q0 -252 -126.5 -459.5t-330.5 -306.5q-181 215 -181 495q0 187 83.5 349.5t229.5 269.5t325 137zM1536 638q0 -280 -181 -495 q-204 99 -330.5 306.5t-126.5 459.5v485q179 -30 325 -137t229.5 -269.5t83.5 -349.5z" />
+<glyph unicode="&#xf18c;" horiz-adv-x="1408" d="M1402 433q-32 -80 -76 -138t-91 -88.5t-99 -46.5t-101.5 -14.5t-96.5 8.5t-86.5 22t-69.5 27.5t-46 22.5l-17 10q-113 -228 -289.5 -359.5t-384.5 -132.5q-19 0 -32 13t-13 32t13 31.5t32 12.5q173 1 322.5 107.5t251.5 294.5q-36 -14 -72 -23t-83 -13t-91 2.5t-93 28.5 t-92 59t-84.5 100t-74.5 146q114 47 214 57t167.5 -7.5t124.5 -56.5t88.5 -77t56.5 -82q53 131 79 291q-7 -1 -18 -2.5t-46.5 -2.5t-69.5 0.5t-81.5 10t-88.5 23t-84 42.5t-75 65t-54.5 94.5t-28.5 127.5q70 28 133.5 36.5t112.5 -1t92 -30t73.5 -50t56 -61t42 -63t27.5 -56 t16 -39.5l4 -16q12 122 12 195q-8 6 -21.5 16t-49 44.5t-63.5 71.5t-54 93t-33 112.5t12 127t70 138.5q73 -25 127.5 -61.5t84.5 -76.5t48 -85t20.5 -89t-0.5 -85.5t-13 -76.5t-19 -62t-17 -42l-7 -15q1 -5 1 -50.5t-1 -71.5q3 7 10 18.5t30.5 43t50.5 58t71 55.5t91.5 44.5 t112 14.5t132.5 -24q-2 -78 -21.5 -141.5t-50 -104.5t-69.5 -71.5t-81.5 -45.5t-84.5 -24t-80 -9.5t-67.5 1t-46.5 4.5l-17 3q-23 -147 -73 -283q6 7 18 18.5t49.5 41t77.5 52.5t99.5 42t117.5 20t129 -23.5t137 -77.5z" />
+<glyph unicode="&#xf18d;" horiz-adv-x="1280" d="M1259 283v-66q0 -85 -57.5 -144.5t-138.5 -59.5h-57l-260 -269v269h-529q-81 0 -138.5 59.5t-57.5 144.5v66h1238zM1259 609v-255h-1238v255h1238zM1259 937v-255h-1238v255h1238zM1259 1077v-67h-1238v67q0 84 57.5 143.5t138.5 59.5h846q81 0 138.5 -59.5t57.5 -143.5z " />
+<glyph unicode="&#xf18e;" d="M1152 640q0 -14 -9 -23l-320 -320q-9 -9 -23 -9q-13 0 -22.5 9.5t-9.5 22.5v192h-352q-13 0 -22.5 9.5t-9.5 22.5v192q0 13 9.5 22.5t22.5 9.5h352v192q0 14 9 23t23 9q12 0 24 -10l319 -319q9 -9 9 -23zM1312 640q0 148 -73 273t-198 198t-273 73t-273 -73t-198 -198 t-73 -273t73 -273t198 -198t273 -73t273 73t198 198t73 273zM1536 640q0 -209 -103 -385.5t-279.5 -279.5t-385.5 -103t-385.5 103t-279.5 279.5t-103 385.5t103 385.5t279.5 279.5t385.5 103t385.5 -103t279.5 -279.5t103 -385.5z" />
+<glyph unicode="&#xf190;" d="M1152 736v-192q0 -13 -9.5 -22.5t-22.5 -9.5h-352v-192q0 -14 -9 -23t-23 -9q-12 0 -24 10l-319 319q-9 9 -9 23t9 23l320 320q9 9 23 9q13 0 22.5 -9.5t9.5 -22.5v-192h352q13 0 22.5 -9.5t9.5 -22.5zM1312 640q0 148 -73 273t-198 198t-273 73t-273 -73t-198 -198 t-73 -273t73 -273t198 -198t273 -73t273 73t198 198t73 273zM1536 640q0 -209 -103 -385.5t-279.5 -279.5t-385.5 -103t-385.5 103t-279.5 279.5t-103 385.5t103 385.5t279.5 279.5t385.5 103t385.5 -103t279.5 -279.5t103 -385.5z" />
+<glyph unicode="&#xf191;" d="M1024 960v-640q0 -26 -19 -45t-45 -19q-20 0 -37 12l-448 320q-27 19 -27 52t27 52l448 320q17 12 37 12q26 0 45 -19t19 -45zM1280 160v960q0 13 -9.5 22.5t-22.5 9.5h-960q-13 0 -22.5 -9.5t-9.5 -22.5v-960q0 -13 9.5 -22.5t22.5 -9.5h960q13 0 22.5 9.5t9.5 22.5z M1536 1120v-960q0 -119 -84.5 -203.5t-203.5 -84.5h-960q-119 0 -203.5 84.5t-84.5 203.5v960q0 119 84.5 203.5t203.5 84.5h960q119 0 203.5 -84.5t84.5 -203.5z" />
+<glyph unicode="&#xf192;" d="M1024 640q0 -106 -75 -181t-181 -75t-181 75t-75 181t75 181t181 75t181 -75t75 -181zM768 1184q-148 0 -273 -73t-198 -198t-73 -273t73 -273t198 -198t273 -73t273 73t198 198t73 273t-73 273t-198 198t-273 73zM1536 640q0 -209 -103 -385.5t-279.5 -279.5 t-385.5 -103t-385.5 103t-279.5 279.5t-103 385.5t103 385.5t279.5 279.5t385.5 103t385.5 -103t279.5 -279.5t103 -385.5z" />
+<glyph unicode="&#xf193;" horiz-adv-x="1664" d="M1023 349l102 -204q-58 -179 -210 -290t-339 -111q-156 0 -288.5 77.5t-210 210t-77.5 288.5q0 181 104.5 330t274.5 211l17 -131q-122 -54 -195 -165.5t-73 -244.5q0 -185 131.5 -316.5t316.5 -131.5q126 0 232.5 65t165 175.5t49.5 236.5zM1571 249l58 -114l-256 -128 q-13 -7 -29 -7q-40 0 -57 35l-239 477h-472q-24 0 -42.5 16.5t-21.5 40.5l-96 779q-2 16 6 42q14 51 57 82.5t97 31.5q66 0 113 -47t47 -113q0 -69 -52 -117.5t-120 -41.5l37 -289h423v-128h-407l16 -128h455q40 0 57 -35l228 -455z" />
+<glyph unicode="&#xf194;" d="M1254 899q16 85 -21 132q-52 65 -187 45q-17 -3 -41 -12.5t-57.5 -30.5t-64.5 -48.5t-59.5 -70t-44.5 -91.5q80 7 113.5 -16t26.5 -99q-5 -52 -52 -143q-43 -78 -71 -99q-44 -32 -87 14q-23 24 -37.5 64.5t-19 73t-10 84t-8.5 71.5q-23 129 -34 164q-12 37 -35.5 69 t-50.5 40q-57 16 -127 -25q-54 -32 -136.5 -106t-122.5 -102v-7q16 -8 25.5 -26t21.5 -20q21 -3 54.5 8.5t58 10.5t41.5 -30q11 -18 18.5 -38.5t15 -48t12.5 -40.5q17 -46 53 -187q36 -146 57 -197q42 -99 103 -125q43 -12 85 -1.5t76 31.5q131 77 250 237 q104 139 172.5 292.5t82.5 226.5zM1536 1120v-960q0 -119 -84.5 -203.5t-203.5 -84.5h-960q-119 0 -203.5 84.5t-84.5 203.5v960q0 119 84.5 203.5t203.5 84.5h960q119 0 203.5 -84.5t84.5 -203.5z" />
+<glyph unicode="&#xf195;" horiz-adv-x="1152" d="M1152 704q0 -191 -94.5 -353t-256.5 -256.5t-353 -94.5h-160q-14 0 -23 9t-9 23v611l-215 -66q-3 -1 -9 -1q-10 0 -19 6q-13 10 -13 26v128q0 23 23 31l233 71v93l-215 -66q-3 -1 -9 -1q-10 0 -19 6q-13 10 -13 26v128q0 23 23 31l233 71v250q0 14 9 23t23 9h160 q14 0 23 -9t9 -23v-181l375 116q15 5 28 -5t13 -26v-128q0 -23 -23 -31l-393 -121v-93l375 116q15 5 28 -5t13 -26v-128q0 -23 -23 -31l-393 -121v-487q188 13 318 151t130 328q0 14 9 23t23 9h160q14 0 23 -9t9 -23z" />
+<glyph unicode="&#xf196;" horiz-adv-x="1408" d="M1152 736v-64q0 -14 -9 -23t-23 -9h-352v-352q0 -14 -9 -23t-23 -9h-64q-14 0 -23 9t-9 23v352h-352q-14 0 -23 9t-9 23v64q0 14 9 23t23 9h352v352q0 14 9 23t23 9h64q14 0 23 -9t9 -23v-352h352q14 0 23 -9t9 -23zM1280 288v832q0 66 -47 113t-113 47h-832 q-66 0 -113 -47t-47 -113v-832q0 -66 47 -113t113 -47h832q66 0 113 47t47 113zM1408 1120v-832q0 -119 -84.5 -203.5t-203.5 -84.5h-832q-119 0 -203.5 84.5t-84.5 203.5v832q0 119 84.5 203.5t203.5 84.5h832q119 0 203.5 -84.5t84.5 -203.5z" />
+<glyph unicode="&#xf197;" horiz-adv-x="1792" />
+<glyph unicode="&#xf198;" horiz-adv-x="1792" />
+<glyph unicode="&#xf199;" horiz-adv-x="1792" />
+<glyph unicode="&#xf19a;" horiz-adv-x="1792" />
+<glyph unicode="&#xf19b;" horiz-adv-x="1792" />
+<glyph unicode="&#xf19c;" horiz-adv-x="1792" />
+<glyph unicode="&#xf19d;" horiz-adv-x="1792" />
+<glyph unicode="&#xf19e;" horiz-adv-x="1792" />
+<glyph unicode="&#xf500;" horiz-adv-x="1792" />
+</font>
+</defs></svg> \ No newline at end of file
diff --git a/doc/website-v1/fonts/fontawesome-webfont.ttf b/doc/website-v1/fonts/fontawesome-webfont.ttf
new file mode 100755
index 0000000..e89738d
--- /dev/null
+++ b/doc/website-v1/fonts/fontawesome-webfont.ttf
Binary files differ
diff --git a/doc/website-v1/fonts/fontawesome-webfont.woff b/doc/website-v1/fonts/fontawesome-webfont.woff
new file mode 100755
index 0000000..8c1748a
--- /dev/null
+++ b/doc/website-v1/fonts/fontawesome-webfont.woff
Binary files differ
diff --git a/doc/website-v1/history-guide.adoc b/doc/website-v1/history-guide.adoc
new file mode 100644
index 0000000..a3dd9c6
--- /dev/null
+++ b/doc/website-v1/history-guide.adoc
@@ -0,0 +1,275 @@
+= Cluster history =
+:source-highlighter: pygments
+
+This guide should help administrators and consultants tackle
+issues in Pacemaker cluster installations. We concentrate
+on troubleshooting and analysis methods with the crmsh history.
+
+Cluster leaves numerous traces behind, more than any other
+system. The logs and the rest are spread among all cluster nodes
+and multiple directories. The obvious difficulty is to show that
+information in a consolidated manner. This is where crmsh
+history helps.
+
+Hopefully, the guide will help you investigate your
+specific issue with more efficiency and less effort.
+
+== Sample cluster
+
+In <<Listing 1>> a modestly complex sample cluster is shown with
+which we can experiment and break in some hopefully instructive
+ways.
+
+NOTE: We won't be going into how to setup nodes or configure the
+ cluster. For that, please refer to the
+ link:/start-guide[Getting Started] document.
+
+[source,crmsh]
+[caption="Listing 1: "]
+.Sample cluster configuration[[Listing 1]]
+-----------------
+include::include/history-guide/sample-cluster.conf.crm[]
+-----------------
+
+If you're new to clusters, that configuration may look
+overwhelming. A graphical presentation in <<Image 1>> of the
+essential elements and relations between them is easier on the eye
+(and the mind).
+
+[caption="Image 1: "]
+.Sample cluster configuration as a graph[[Image 1]]
+image::/img/history-guide/sample-cluster.conf.png[link="/img/history-guide/sample-cluster.conf.png"]
+
+As homework, try to match the two cluster representations.
+
+== Quick (& dirty) start
+
+For the impatient, we give here a few examples of history use.
+
+Most of the time you will be dealing with various resource
+(a.k.a. application) induced phenomena. For instance, while
+preparing this document we noticed that a probe failed repeatedly
+on a node which wasn't even running the resource <<Listing 2>>.
+
+[source,ansiclr]
+[caption="Listing 2: "]
+.crm status output[[Listing 2]]
+-----------------
+include::include/history-guide/status-probe-fail.typescript[]
+-----------------
+
+The history +resource+ command shows log messages relevant to the
+supplied resource <<Listing 3>>.
+
+[source,ansiclr]
+[caption="Listing 3: "]
+.Logs on failed +nfs-server+ probe operation[[Listing 3]]
+-----------------
+include::include/history-guide/nfs-probe-err.typescript[]
+-----------------
+
+<1> NFS server error message.
+<2> Warning about a non-existing user id.
+
+NOTE: Messages logged by resource agents are always tagged with
+ 'type(ID)' (in <<Listing 3>>: +nfsserver(nfs-server)+).
+ +
+ Everything dumped to +stderr/stdout+ (in <<Listing 3>>:
+ +id: rpcuser: no such user+) is captured and subsequently
+ logged by +lrmd+. The +stdout+ output is at the 'info'
+ severity which is by default _not_ logged by pacemaker
+ since version 1.1.12.
+
+At the very top we see error message reporting that the
+NFS server is running, but some other stuff, apparently
+unexpectedly, is not. However, we know that it cannot be
+running on the 'c' node as it is already running on the 'a' node.
+Not being able to figure out what is going on, we had to turn on
+tracing of the resource agent. <<Listing 4>> shows how to do
+that.
+
+[source,ansiclr]
+[caption="Listing 4: "]
+.Set `nfs-server` probe operation resource tracing[[Listing 4]]
+-----------------
+include::include/history-guide/resource-trace.typescript[]
+-----------------
+
+Trace of the +nfsserver+ RA revealed that the +nfs-server+ init
+script (used internally by the resource agent) _always_ exits
+with success for status. That was actually due to the recent port
+to systemd and erroneous interpretation of `systemctl status`
+semantics: it always exits with success (due to some paradigm
+shift, we guess). FYI, `systemctl is-active` should be used
+instead and it does report a service status as expected.
+
+As a bonus, a minor issue about a non-existing user id +rpcuser+
+is also revealed.
+
+NOTE: Messages in the crm history log output are colored
+ depending on the originating host.
+
+The rest of this document gives more details about crmsh history.
+If you're more of a just-try-it-out person, enter +crm history+
+and experiment. With +history+ commands you cannot really break
+anything (fingers crossed).
+
+== Introduction to crmsh `history`
+
+The history crmsh feature, as the name suggests, deals with the
+past. It was conceived as a facility to bring to the fore all
+trails pacemaker cluster leaves behind which are relevant to a
+particular resource, node, or event. It is used in the first
+place as a troubleshooting tool, but it can also be helpful in
+studying pacemaker clusters.
+
+To begin, we run the `info` command which gives an overview, as
+shown in <<Listing 5>>.
+
+[source,ansiclr]
+[caption="Listing 5: "]
+.Basic history information[[Listing 5]]
+-----------------
+include::include/history-guide/info.typescript[]
+-----------------
+
+The `timeframe` command limits the observed period and helps
+focus on the events of interest. Here we wanted to look at the
+10 minute period. Two transitions were executed during this time.
+
+== Transitions
+
+Transitions are basic units capturing cluster movements
+(resource operations and node events). A transition
+consists of a set of actions to reach a desired cluster
+status as specified in the cluster configuration by the
+user.
+
+Every configuration or status change results in a transition.
+
+Every transition is also a CIB, which is how cluster
+configuration and status are stored. Transitions are saved
+to files, the so called PE (Policy Engine) inputs.
+
+In <<Listing 6>> we show how to display transitions.
+The listing is annotated to explain the output in more detail.
+
+
+[source,ansiclr]
+[caption="Listing 6: "]
+.Viewing transitions[[Listing 6]]
+-----------------
+include::include/history-guide/basic-transition.typescript[]
+-----------------
+
+<1> The transition command without arguments displays the latest
+transition.
+<2> Graph of transition actions is provided by `graphviz`. See
+<<Image 2>>.
+<3> Output of `crm_simulate` with irrelevant stuff edited out.
+`crm_simulate` was formerly known as `ptest`.
+<4> Transition summary followed by selection of log messages.
+History weeds out messages which are of lesser importance. See
+<<Listing 8>> if you want to see what history has been hiding
+from you here.
+
+Incidentally, if you wonder why all transitions in these examples
+are green, that is not because they were green in any sense of
+the color, but just due to that being color of node 'c': as
+chance would have it, 'c' was calling shots at the time (being
+Designated Coordinator or DC). That is also why all `crmd` and
+`pengine` messages are coming from 'c'.
+
+NOTE: Transitions are the basis of pacemaker operation, make sure
+that you understand them.
+
+What you cannot see in the listing is a graph generated and shown
+in a separate window in your X11 display. <<Image 2>> may not be
+very involved, but we reckon it's as good a start as starts go.
+
+[caption="Image 2: "]
+.Graph for transition 1907[[Image 2]]
+image::/img/history-guide/smallapache-start.png[link="/img/history-guide/smallapache-start.png"]
+
+It may sometimes be useful to see what changed between two
+transitions. History `diff` command is in action in <<Listing 7>>.
+
+[source,ansiclr]
+[caption="Listing 7: "]
+.Viewing transitions[[Listing 7]]
+-----------------
+include::include/history-guide/diff.typescript[]
+-----------------
+
+<1> Configuration diff between two last transitions. Transitions
+may be referenced with indexes starting at 0 and going backwards.
+<2> Status diff between two last transitions.
+
+Whereas configuration diff is (hopefully) obvious, status diff
+needs some explanation: the status section of the PE inputs
+(transitions) always lags behind the configuration. This is
+because at the time the transition is saved to a file, the
+actions of that transition are yet to be executed. So, the status
+section of transition _N_ corresponds to the configuration _N-1_.
+
+[source,ansiclr]
+[caption="Listing 8: "]
+.Full transition log[[Listing 8]]
+-----------------
+include::include/history-guide/transition-log.typescript[]
+-----------------
+
+== Resource and node events
+
+Apart from transitions, events such as resource start or stop are
+what we usually want to examine. In our extremely exciting
+example of apache resource restart, the history `resource`
+command picks the most interesting resource related messages as
+shown in <<Listing 9>>. Again, history shows only the most
+important log parts.
+
+NOTE: If you want to see more detail (which may not always be
+ recommendable), then use the history `detail` command to
+ increase the level of detail displayed.
+
+[source,ansiclr]
+[caption="Listing 9: "]
+.Resource related messages[[Listing 9]]
+-----------------
+include::include/history-guide/resource.typescript[]
+-----------------
+
+Node related events are node start and stop (cluster-wise),
+node membership changes, and stonith events (aka node fence).
+We'll refrain from showing examples of the history `node`
+command--it is analogue to the `resource` command.
+
+== Viewing logs
+
+History `log` command, unsurprisingly, displays logs. The
+messages from various nodes are weaved and shown in different
+colors for the sake of easier viewing. Unlike other history
+commands, `log` shows all messages captured in the report. If you
+find some of them irrelevant they can be filtered out:
+the `exclude` command takes extended regular expressions and it
+is additive. We usually set the exclude expression to at least
+`ssh|systemd|kernel`. Use `exclude clear` to remove all
+expressions. And don't forget the `timeframe` command that
+imposes a time window on the report.
+
+== External reports, configurations, and graphs
+
+The information source history works with is `hb_report`
+generated report. Even when examining live cluster, `hb_report` is
+run behind the scene to collect the data before presenting it to
+the user. Well, at least to generate the first report: there is a
+special procedure for log refreshing and collecting new PE
+inputs, which runs much faster than creating a report from
+scratch. However, juggling with multiple sources, appending logs,
+moving time windows, may not always be foolproof, and if
+the source gets borked you can always ask for a brand new report
+with `refresh force`.
+
+Analyzing reports from external source is no different from what
+we've seen so far. In fact, there's a `source` command which
+tells history where to look for data.
diff --git a/doc/website-v1/img/history-guide/sample-cluster.conf.png b/doc/website-v1/img/history-guide/sample-cluster.conf.png
new file mode 100644
index 0000000..0863923
--- /dev/null
+++ b/doc/website-v1/img/history-guide/sample-cluster.conf.png
Binary files differ
diff --git a/doc/website-v1/img/history-guide/smallapache-start.png b/doc/website-v1/img/history-guide/smallapache-start.png
new file mode 100644
index 0000000..47853c9
--- /dev/null
+++ b/doc/website-v1/img/history-guide/smallapache-start.png
Binary files differ
diff --git a/doc/website-v1/img/icons/README b/doc/website-v1/img/icons/README
new file mode 100644
index 0000000..f12b2a7
--- /dev/null
+++ b/doc/website-v1/img/icons/README
@@ -0,0 +1,5 @@
+Replaced the plain DocBook XSL admonition icons with Jimmac's DocBook
+icons (http://jimmac.musichall.cz/ikony.php3). I dropped transparency
+from the Jimmac icons to get round MS IE and FOP PNG incompatibilies.
+
+Stuart Rackham
diff --git a/doc/website-v1/img/icons/callouts/1.png b/doc/website-v1/img/icons/callouts/1.png
new file mode 100644
index 0000000..7d47343
--- /dev/null
+++ b/doc/website-v1/img/icons/callouts/1.png
Binary files differ
diff --git a/doc/website-v1/img/icons/callouts/10.png b/doc/website-v1/img/icons/callouts/10.png
new file mode 100644
index 0000000..997bbc8
--- /dev/null
+++ b/doc/website-v1/img/icons/callouts/10.png
Binary files differ
diff --git a/doc/website-v1/img/icons/callouts/11.png b/doc/website-v1/img/icons/callouts/11.png
new file mode 100644
index 0000000..ce47dac
--- /dev/null
+++ b/doc/website-v1/img/icons/callouts/11.png
Binary files differ
diff --git a/doc/website-v1/img/icons/callouts/12.png b/doc/website-v1/img/icons/callouts/12.png
new file mode 100644
index 0000000..31daf4e
--- /dev/null
+++ b/doc/website-v1/img/icons/callouts/12.png
Binary files differ
diff --git a/doc/website-v1/img/icons/callouts/13.png b/doc/website-v1/img/icons/callouts/13.png
new file mode 100644
index 0000000..14021a8
--- /dev/null
+++ b/doc/website-v1/img/icons/callouts/13.png
Binary files differ
diff --git a/doc/website-v1/img/icons/callouts/14.png b/doc/website-v1/img/icons/callouts/14.png
new file mode 100644
index 0000000..64014b7
--- /dev/null
+++ b/doc/website-v1/img/icons/callouts/14.png
Binary files differ
diff --git a/doc/website-v1/img/icons/callouts/15.png b/doc/website-v1/img/icons/callouts/15.png
new file mode 100644
index 0000000..0d65765
--- /dev/null
+++ b/doc/website-v1/img/icons/callouts/15.png
Binary files differ
diff --git a/doc/website-v1/img/icons/callouts/2.png b/doc/website-v1/img/icons/callouts/2.png
new file mode 100644
index 0000000..5d09341
--- /dev/null
+++ b/doc/website-v1/img/icons/callouts/2.png
Binary files differ
diff --git a/doc/website-v1/img/icons/callouts/3.png b/doc/website-v1/img/icons/callouts/3.png
new file mode 100644
index 0000000..ef7b700
--- /dev/null
+++ b/doc/website-v1/img/icons/callouts/3.png
Binary files differ
diff --git a/doc/website-v1/img/icons/callouts/4.png b/doc/website-v1/img/icons/callouts/4.png
new file mode 100644
index 0000000..adb8364
--- /dev/null
+++ b/doc/website-v1/img/icons/callouts/4.png
Binary files differ
diff --git a/doc/website-v1/img/icons/callouts/5.png b/doc/website-v1/img/icons/callouts/5.png
new file mode 100644
index 0000000..4d7eb46
--- /dev/null
+++ b/doc/website-v1/img/icons/callouts/5.png
Binary files differ
diff --git a/doc/website-v1/img/icons/callouts/6.png b/doc/website-v1/img/icons/callouts/6.png
new file mode 100644
index 0000000..0ba694a
--- /dev/null
+++ b/doc/website-v1/img/icons/callouts/6.png
Binary files differ
diff --git a/doc/website-v1/img/icons/callouts/7.png b/doc/website-v1/img/icons/callouts/7.png
new file mode 100644
index 0000000..472e96f
--- /dev/null
+++ b/doc/website-v1/img/icons/callouts/7.png
Binary files differ
diff --git a/doc/website-v1/img/icons/callouts/8.png b/doc/website-v1/img/icons/callouts/8.png
new file mode 100644
index 0000000..5e60973
--- /dev/null
+++ b/doc/website-v1/img/icons/callouts/8.png
Binary files differ
diff --git a/doc/website-v1/img/icons/callouts/9.png b/doc/website-v1/img/icons/callouts/9.png
new file mode 100644
index 0000000..a0676d2
--- /dev/null
+++ b/doc/website-v1/img/icons/callouts/9.png
Binary files differ
diff --git a/doc/website-v1/img/icons/caution.png b/doc/website-v1/img/icons/caution.png
new file mode 100644
index 0000000..9a8c515
--- /dev/null
+++ b/doc/website-v1/img/icons/caution.png
Binary files differ
diff --git a/doc/website-v1/img/icons/example.png b/doc/website-v1/img/icons/example.png
new file mode 100644
index 0000000..1199e86
--- /dev/null
+++ b/doc/website-v1/img/icons/example.png
Binary files differ
diff --git a/doc/website-v1/img/icons/home.png b/doc/website-v1/img/icons/home.png
new file mode 100644
index 0000000..37a5231
--- /dev/null
+++ b/doc/website-v1/img/icons/home.png
Binary files differ
diff --git a/doc/website-v1/img/icons/important.png b/doc/website-v1/img/icons/important.png
new file mode 100644
index 0000000..be685cc
--- /dev/null
+++ b/doc/website-v1/img/icons/important.png
Binary files differ
diff --git a/doc/website-v1/img/icons/next.png b/doc/website-v1/img/icons/next.png
new file mode 100644
index 0000000..64e126b
--- /dev/null
+++ b/doc/website-v1/img/icons/next.png
Binary files differ
diff --git a/doc/website-v1/img/icons/note.png b/doc/website-v1/img/icons/note.png
new file mode 100644
index 0000000..7c1f3e2
--- /dev/null
+++ b/doc/website-v1/img/icons/note.png
Binary files differ
diff --git a/doc/website-v1/img/icons/prev.png b/doc/website-v1/img/icons/prev.png
new file mode 100644
index 0000000..3e8f12f
--- /dev/null
+++ b/doc/website-v1/img/icons/prev.png
Binary files differ
diff --git a/doc/website-v1/img/icons/tip.png b/doc/website-v1/img/icons/tip.png
new file mode 100644
index 0000000..f087c73
--- /dev/null
+++ b/doc/website-v1/img/icons/tip.png
Binary files differ
diff --git a/doc/website-v1/img/icons/up.png b/doc/website-v1/img/icons/up.png
new file mode 100644
index 0000000..2db1ce6
--- /dev/null
+++ b/doc/website-v1/img/icons/up.png
Binary files differ
diff --git a/doc/website-v1/img/icons/warning.png b/doc/website-v1/img/icons/warning.png
new file mode 100644
index 0000000..d41edb9
--- /dev/null
+++ b/doc/website-v1/img/icons/warning.png
Binary files differ
diff --git a/doc/website-v1/img/laptop.png b/doc/website-v1/img/laptop.png
new file mode 100644
index 0000000..2f831ba
--- /dev/null
+++ b/doc/website-v1/img/laptop.png
Binary files differ
diff --git a/doc/website-v1/img/loader.gif b/doc/website-v1/img/loader.gif
new file mode 100644
index 0000000..b2cfedb
--- /dev/null
+++ b/doc/website-v1/img/loader.gif
Binary files differ
diff --git a/doc/website-v1/img/servers.gif b/doc/website-v1/img/servers.gif
new file mode 100644
index 0000000..20afdcb
--- /dev/null
+++ b/doc/website-v1/img/servers.gif
Binary files differ
diff --git a/doc/website-v1/include/history-guide/basic-transition.typescript b/doc/website-v1/include/history-guide/basic-transition.typescript
new file mode 100644
index 0000000..a5a0a31
--- /dev/null
+++ b/doc/website-v1/include/history-guide/basic-transition.typescript
@@ -0,0 +1,22 @@
+crm(live)history# transition <1>
+INFO: running ptest with /var/cache/crm/history/live/sle12-c/pengine/pe-input-1907.bz2
+INFO: starting dotty to show transition graph <2>
+Current cluster status: <3>
+Online: [ sle12-a sle12-c ]
+ s-libvirt (stonith:external/libvirt): Started sle12-c
+ ...
+ small-apache (ocf::heartbeat:apache): Stopped
+Transition Summary:
+ * Start small-apache (sle12-a)
+Executing cluster transition:
+ * Resource action: small-apache start on sle12-a
+Revised cluster status:
+Online: [ sle12-a sle12-c ]
+ s-libvirt (stonith:external/libvirt): Started sle12-c
+ ...
+ small-apache (ocf::heartbeat:apache): Started sle12-a
+
+Transition sle12-c:pe-input-1907 (20:30:14 - 20:30:15): <4>
+ total 1 actions: 1 Complete
+Apr 15 20:30:14 sle12-c crmd[1136]: notice: te_rsc_command: Initiating action 60: start small-apache_start_0 on sle12-a
+Apr 15 20:30:14 sle12-a apache(small-apache)[1586]: INFO: AH00558: httpd2: Could not reliably determine the server's fully qualified domain name, using 10.2.12.51. Set the 'ServerName' directive globally to suppress this message
diff --git a/doc/website-v1/include/history-guide/diff.typescript b/doc/website-v1/include/history-guide/diff.typescript
new file mode 100644
index 0000000..129febc
--- /dev/null
+++ b/doc/website-v1/include/history-guide/diff.typescript
@@ -0,0 +1,11 @@
+crm(live)history# diff -1 0 <1>
+--- -1
++++ 0
+@@ -11 +11 @@
+-primitive small-apache apache params configfile="/etc/apache2/small.conf" meta target-role=Stopped
++primitive small-apache apache params configfile="/etc/apache2/small.conf" meta target-role=Started
+crm(live)history# diff -1 0 status <2>
+--- -1
++++ 0
+@@ -15 +14,0 @@
+- small-apache (ocf::heartbeat:apache): Started sle12-a
diff --git a/doc/website-v1/include/history-guide/info.typescript b/doc/website-v1/include/history-guide/info.typescript
new file mode 100644
index 0000000..d7aae8d
--- /dev/null
+++ b/doc/website-v1/include/history-guide/info.typescript
@@ -0,0 +1,16 @@
+# crm history
+crm(live)history# timeframe "Apr 15 20:25" "Apr 15 20:35"
+crm(live)history# info
+Source: live
+Created on: Thu Apr 16 11:32:36 CEST 2015
+By: report -Z -Q -f Wed Apr 15 20:25:00 2015 -t 2015-04-15 20:35:00 /var/cache/crm/history/live
+Period: 2015-04-15 20:25:00 - 2015-04-15 20:35:00
+Nodes: sle12-a sle12-c
+Groups: nfs-srv nfs-disk
+Resources: s-libvirt p_drbd_nfs nfs-vg fs1 virtual-ip nfs-server websrv websrv-ip small-apache
+Transitions: 1906 1907
+crm(live)history# peinputs v
+Date Start End Filename Client User Origin
+==== ===== === ======== ====== ==== ======
+2015-04-15 20:29:59 20:30:01 pe-input-1906 no-client no-user no-origin
+2015-04-15 20:30:14 20:30:15 pe-input-1907 no-client no-user no-origin
diff --git a/doc/website-v1/include/history-guide/nfs-probe-err.typescript b/doc/website-v1/include/history-guide/nfs-probe-err.typescript
new file mode 100644
index 0000000..ca34ba5
--- /dev/null
+++ b/doc/website-v1/include/history-guide/nfs-probe-err.typescript
@@ -0,0 +1,20 @@
+# crm history resource nfs-server
+INFO: fetching new logs, please wait ...
+Dec 16 11:53:23 sle12-c nfsserver(nfs-server)[14911]: <1> ERROR: NFS server is up, but the locking daemons are down
+Dec 16 11:53:23 sle12-c crmd[2823]: notice: te_rsc_command: Initiating action 54: stop nfs-server_stop_0 on sle12-a
+Dec 16 11:53:23 sle12-c crmd[2823]: notice: te_rsc_command: Initiating action 3: stop nfs-server_stop_0 on sle12-c (local)
+Dec 16 11:53:23 sle12-c nfsserver(nfs-server)[14944]: INFO: Stopping NFS server ...
+Dec 16 11:53:23 sle12-c nfsserver(nfs-server)[14944]: INFO: Stopping sm-notify
+Dec 16 11:53:23 sle12-c nfsserver(nfs-server)[14944]: INFO: Stopping rpc.statd
+Dec 16 11:53:23 sle12-c nfsserver(nfs-server)[14944]: INFO: NFS server stopped
+Dec 16 11:53:23 sle12-c crmd[2823]: notice: te_rsc_command: Initiating action 55: start nfs-server_start_0 on sle12-a
+Dec 16 11:53:23 sle12-a nfsserver(nfs-server)[23255]: INFO: Stopping NFS server ...
+Dec 16 11:53:23 sle12-a nfsserver(nfs-server)[23255]: INFO: Stopping sm-notify
+Dec 16 11:53:23 sle12-a nfsserver(nfs-server)[23255]: INFO: Stopping rpc.statd
+Dec 16 11:53:23 sle12-a nfsserver(nfs-server)[23255]: INFO: NFS server stopped
+Dec 16 11:53:23 sle12-a nfsserver(nfs-server)[23320]: INFO: Starting NFS server ...
+Dec 16 11:53:23 sle12-a nfsserver(nfs-server)[23320]: INFO: Starting rpc.statd.
+Dec 16 11:53:24 sle12-a nfsserver(nfs-server)[23320]: INFO: executing sm-notify
+Dec 16 11:53:24 sle12-a nfsserver(nfs-server)[23320]: INFO: NFS server started
+Dec 16 11:53:24 sle12-a lrmd[6904]: <2> notice: operation_finished: nfs-server_start_0:23320:stderr [ id: rpcuser: no such user ]
+Dec 16 11:53:24 sle12-a lrmd[6904]: message repeated 3 times: [ notice: operation_finished: nfs-server_start_0:23320:stderr [ id: rpcuser: no such user ]]
diff --git a/doc/website-v1/include/history-guide/resource-trace.typescript b/doc/website-v1/include/history-guide/resource-trace.typescript
new file mode 100644
index 0000000..e66ff7c
--- /dev/null
+++ b/doc/website-v1/include/history-guide/resource-trace.typescript
@@ -0,0 +1,7 @@
+# crm resource trace nfs-server monitor 0
+INFO: Trace for nfs-server:monitor is written to /var/lib/heartbeat/trace_ra/
+INFO: Trace set, restart nfs-server to trace non-monitor operations
+# crm resource cleanup nfs-server
+Cleaning up nfs-server on sle12-a
+Cleaning up nfs-server on sle12-c
+Waiting for 2 replies from the CRMd.. OK
diff --git a/doc/website-v1/include/history-guide/resource.typescript b/doc/website-v1/include/history-guide/resource.typescript
new file mode 100644
index 0000000..90f0265
--- /dev/null
+++ b/doc/website-v1/include/history-guide/resource.typescript
@@ -0,0 +1,6 @@
+crm(live)history# resource small-apache
+Apr 15 20:29:59 sle12-c crmd[1136]: notice: te_rsc_command: Initiating action 60: stop small-apache_stop_0 on sle12-a
+Apr 15 20:29:59 sle12-a apache(small-apache)[1366]: INFO: Attempting graceful stop of apache PID 9155
+Apr 15 20:30:01 sle12-a apache(small-apache)[1366]: INFO: apache stopped.
+Apr 15 20:30:14 sle12-a apache(small-apache)[1586]: INFO: AH00558: httpd2: Could not reliably determine the server's fully qualified domain name, using 10.2.12.51. Set the 'ServerName' directive globally to suppress this message
+Apr 15 20:30:14 sle12-c crmd[1136]: notice: te_rsc_command: Initiating action 60: start small-apache_start_0 on sle12-a
diff --git a/doc/website-v1/include/history-guide/sample-cluster.conf.crm b/doc/website-v1/include/history-guide/sample-cluster.conf.crm
new file mode 100644
index 0000000..8b44663
--- /dev/null
+++ b/doc/website-v1/include/history-guide/sample-cluster.conf.crm
@@ -0,0 +1,54 @@
+node 167906357: sle12-c
+node 167906355: sle12-a
+primitive s-libvirt stonith:external/libvirt \
+ params hostlist="sle12-a sle12-c" hypervisor_uri="qemu+ssh://hex-10.suse.de/system?keyfile=/root/.ssh/xen" reset_method=reboot \
+ op monitor interval=5m timeout=60s
+primitive p_drbd_nfs ocf:linbit:drbd \
+ params drbd_resource=nfs \
+ op monitor interval=15 role=Master \
+ op monitor interval=30 role=Slave \
+ op start interval=0 timeout=300 \
+ op stop interval=0 timeout=120
+primitive nfs-vg LVM \
+ params volgrpname=nfs-vg
+primitive fs1 Filesystem \
+ params device="/dev/nfs-vg/fs1" directory="/srv/nfs" fstype=ext3 \
+ op monitor interval=30s
+primitive virtual-ip IPaddr2 \
+ params ip=10.2.12.100
+primitive nfs-server nfsserver \
+ params nfs_shared_infodir="/srv/nfs/state" nfs_ip=10.2.12.100 \
+ op monitor interval=30s
+primitive websrv apache \
+ params configfile="/etc/apache2/httpd.conf" \
+ op monitor interval=30
+primitive websrv-ip IPaddr2 \
+ params ip=10.2.12.101
+primitive small-apache apache \
+ params configfile="/etc/apache2/small.conf"
+group nfs-disk nfs-vg fs1
+group nfs-srv virtual-ip nfs-server
+ms ms_drbd_nfs p_drbd_nfs \
+ meta notify=true clone-max=2
+location nfs-pref virtual-ip 100: sle12-a
+location websrv-pref websrv 100: sle12-c
+colocation vg-with-drbd inf: nfs-vg ms_drbd_nfs:Master
+colocation c-nfs inf: nfs-srv nfs-disk
+colocation c-websrv inf: websrv websrv-ip
+colocation small-apache-with-virtual-ip inf: small-apache virtual-ip
+# need fs1 for the NFS server
+order o-nfs inf: nfs-disk nfs-srv
+# websrv serves requests at IP websrv-ip
+order o-websrv inf: websrv-ip websrv
+# small apache serves requests at IP virtual-ip
+order virtual-ip-before-small-apache inf: virtual-ip small-apache
+# drbd device is the nfs-vg PV
+order drbd-before-nfs-vg inf: ms_drbd_nfs:promote nfs-vg:start
+property cib-bootstrap-options: \
+ dc-version=1.1.12-ad083a8 \
+ cluster-infrastructure=corosync \
+ cluster-name=sle12-test3l-public \
+ no-quorum-policy=ignore \
+ last-lrm-refresh=1429192263
+op_defaults op-options: \
+ timeout=120s
diff --git a/doc/website-v1/include/history-guide/status-probe-fail.typescript b/doc/website-v1/include/history-guide/status-probe-fail.typescript
new file mode 100644
index 0000000..d1024e8
--- /dev/null
+++ b/doc/website-v1/include/history-guide/status-probe-fail.typescript
@@ -0,0 +1,15 @@
+# crm status
+Last updated: Tue Dec 16 11:57:04 2014
+Last change: Tue Dec 16 11:53:22 2014
+Stack: corosync
+Current DC: sle12-c (167906357) - partition with quorum
+Version: 1.1.12-ad083a8
+2 Nodes configured
+10 Resources configured
+Online: [ sle12-a sle12-c ]
+[...]
+ nfs-server (ocf::heartbeat:nfsserver): Started sle12-a
+[...]
+Failed actions:
+ nfs-server_monitor_0 on sle12-c 'unknown error' (1): call=298, status=complete,
+ last-rc-change='Tue Dec 16 11:53:23 2014', queued=0ms, exec=135ms
diff --git a/doc/website-v1/include/history-guide/stonith-corosync-stopped.typescript b/doc/website-v1/include/history-guide/stonith-corosync-stopped.typescript
new file mode 100644
index 0000000..1bca5ac
--- /dev/null
+++ b/doc/website-v1/include/history-guide/stonith-corosync-stopped.typescript
@@ -0,0 +1,8 @@
+# crm history node sle12-c
+INFO: fetching new logs, please wait ...
+Dec 19 14:36:18 sle12-c corosync[29551]: [MAIN ] Corosync Cluster Engine ('2.3.3'): started and ready to provide service.
+Dec 19 14:36:19 sle12-c corosync[29545]: Starting Corosync Cluster Engine (corosync): [ OK ]
+Dec 19 14:36:20 sle12-a pengine[6906]: warning: pe_fence_node: Node sle12-c will be fenced because our peer process is no longer available
+Dec 19 14:36:20 sle12-a pengine[6906]: warning: stage6: Scheduling Node sle12-c for STONITH
+Dec 19 14:36:20 sle12-a crmd[6907]: notice: te_fence_node: Executing reboot fencing operation (65) on sle12-c (timeout=60000)
+Dec 19 14:36:20 sle12-a crmd[6907]: notice: peer_update_callback: Node return implies stonith of sle12-c (action 65) completed
diff --git a/doc/website-v1/include/history-guide/transition-log.typescript b/doc/website-v1/include/history-guide/transition-log.typescript
new file mode 100644
index 0000000..eb689ec
--- /dev/null
+++ b/doc/website-v1/include/history-guide/transition-log.typescript
@@ -0,0 +1,13 @@
+crm(live)history# transition log
+INFO: retrieving information from cluster nodes, please wait ...
+Apr 15 20:30:14 sle12-c crmd[1136]: notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_FSA_INTERNAL origin=abort_transition_graph ]
+Apr 15 20:30:14 sle12-c stonithd[1132]: notice: unpack_config: On loss of CCM Quorum: Ignore
+Apr 15 20:30:14 sle12-c pengine[1135]: notice: unpack_config: On loss of CCM Quorum: Ignore
+Apr 15 20:30:14 sle12-c pengine[1135]: notice: LogActions: Start small-apache#011(sle12-a)
+Apr 15 20:30:14 sle12-c crmd[1136]: notice: do_te_invoke: Processing graph 123 (ref=pe_calc-dc-1429122614-234) derived from /var/lib/pacemaker/pengine/pe-input-1907.bz2
+Apr 15 20:30:14 sle12-c crmd[1136]: notice: te_rsc_command: Initiating action 60: start small-apache_start_0 on sle12-a
+Apr 15 20:30:14 sle12-c pengine[1135]: notice: process_pe_message: Calculated Transition 123: /var/lib/pacemaker/pengine/pe-input-1907.bz2
+Apr 15 20:30:14 sle12-a stonithd[1160]: notice: unpack_config: On loss of CCM Quorum: Ignore
+Apr 15 20:30:14 sle12-a apache(small-apache)[1586]: INFO: AH00558: httpd2: Could not reliably determine the server's fully qualified domain name, using 10.2.12.51. Set the 'ServerName' directive globally to suppress this message
+Apr 15 20:30:14 sle12-a crmd[1164]: notice: process_lrm_event: Operation small-apache_start_0: ok (node=sle12-a, call=69, rc=0, cib-update=48, confirmed=true)
+Apr 15 20:30:15 sle12-c crmd[1136]: notice: run_graph: Transition 123 (Complete=1, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-1907.bz2): Complete
diff --git a/doc/website-v1/index.adoc b/doc/website-v1/index.adoc
new file mode 100644
index 0000000..1ebd635
--- /dev/null
+++ b/doc/website-v1/index.adoc
@@ -0,0 +1,25 @@
+The CRM Shell
+=============
+
+++++
+<div class="frontpage-image">
+<br><br>
+<img src="/img/laptop.png">
+<br><br><br>
+</div>
+++++
+
+*`crmsh` is a cluster management shell* for the Pacemaker High Availability stack.
+
+Configure, manage and troubleshoot clusters from the command line,
+with full tab completion and extensive help. `crmsh` also provides
+advanced features like low-level cluster configuration, cluster scripting,
+package management, and history exploration tools giving you a complete
+insight into the state of your cluster.
+
+* https://github.com/ClusterLabs/crmsh/[Source Code]
+* http://crmsh.github.io/man-4.3/[Reference Manual (v4.3.1)]
+* http://crmsh.github.io/man-3/[Reference Manual (v3.0.0)]
+* http://crmsh.github.io/man-2.0/[Reference Manual (v2.3.2)]
+* https://build.opensuse.org/package/show/network:ha-clustering:Stable/crmsh[Packages]
+* http://clusterlabs.org[Cluster Labs]
diff --git a/doc/website-v1/installation.adoc b/doc/website-v1/installation.adoc
new file mode 100644
index 0000000..5a027a0
--- /dev/null
+++ b/doc/website-v1/installation.adoc
@@ -0,0 +1,4 @@
+Installation
+============
+
+See link:/download[Download].
diff --git a/doc/website-v1/make-news.py b/doc/website-v1/make-news.py
new file mode 100644
index 0000000..f3c9073
--- /dev/null
+++ b/doc/website-v1/make-news.py
@@ -0,0 +1,136 @@
+#!/usr/bin/env python
+"""
+Output a combined news.adoc document
+Also write an Atom feed document
+"""
+
+import os
+import sys
+import hashlib
+import datetime
+import time
+
+OUTPUT_HEADER = """= News
+
+"""
+OUTPUT_FOOTER = """
+link:https://savannah.nongnu.org/news/?group_id=10890[Old News Archive]
+"""
+
+ATOM_TEMPLATE = """<?xml version="1.0" encoding="utf-8"?>
+<feed xmlns="http://www.w3.org/2005/Atom">
+<title>crmsh</title>
+<subtitle>Cluster manager shell news</subtitle>
+<link href="http://crmsh.github.io/atom.xml" rel="self" />
+<link href="http://crmsh.github.io/" />
+<id>%(id)s</id>
+<updated>%(updated)s</updated>
+%(entries)s
+</feed>
+"""
+
+ATOM_NAME = "gen/atom.xml"
+
+root_id = "tag:crmsh.github.io,2014:/atom"
+
+def escape(s):
+ s = s.replace('&', '&amp;')
+ s = s.replace('<', '&lt;')
+ s = s.replace('>', '&gt;')
+ s = s.replace('"', "&quot;")
+ return s
+
+class Entry(object):
+ def __init__(self, fname):
+ self.filename = fname
+ self.name = os.path.splitext(os.path.basename(fname))[0]
+ with open(fname) as f:
+ self.title = f.readline().strip()
+ f.readline()
+ l = f.readline()
+ while l.startswith(':'):
+ k, v = l[1:].split(':', 1)
+ k = k.lower()
+ v = v.strip()
+ setattr(self, k, v)
+ l = f.readline()
+ self.content = l + f.read()
+ if not hasattr(self, 'author'):
+ raise ValueError("Missing author")
+ if not hasattr(self, 'email'):
+ raise ValueError("Missing email")
+ if not hasattr(self, 'date'):
+ raise ValueError("Missing date")
+
+ def atom_id(self):
+ return root_id + '::' + hashlib.sha1(self.filename).hexdigest()
+
+ def atom_date(self):
+ return self.date.replace(' ', 'T') + ':00' + time.tzname[0]
+
+ def date_obj(self):
+ from dateutil import parser
+ return (parser.parse(self.date))
+
+ def atom_content(self):
+ return escape('<pre>\n' + self.content + '\n</pre>\n')
+
+ def atom(self):
+ data = {'title': self.title,
+ 'id': self.atom_id(),
+ 'updated': self.atom_date(),
+ 'name': self.name,
+ 'content': self.atom_content(),
+ 'author': self.author,
+ 'email': self.email}
+ return """<entry>
+<title>%(title)s</title>
+<id>%(id)s</id>
+<updated>%(updated)s</updated>
+<link>http://crmsh.github.io/news/%(name)s</link>
+<content type="html">
+%(content)s
+</content>
+<author>
+<name>%(author)s</name>
+<email>%(email)s</email>
+</author>
+</entry>
+""" % data
+
+
+def sort_entries(entries):
+ return list(reversed(sorted(entries, key=lambda e: e.date_obj())))
+
+
+def make_atom():
+ inputs = sort_entries([Entry(f) for f in sys.argv[2:]])
+ with open(ATOM_NAME, 'w') as output:
+ output.write(ATOM_TEMPLATE % {
+ 'id': root_id,
+ 'updated': inputs[0].atom_date(),
+ 'entries': '\n'.join(f.atom() for f in inputs)
+ })
+
+
+def main():
+ # TODO: sort by date
+ inputs = sort_entries([Entry(f) for f in sys.argv[2:]])
+ with open(sys.argv[1], 'w') as output:
+ output.write(OUTPUT_HEADER)
+ e = inputs[0]
+ output.write("link:/news/%s[%s]\n\n" % (e.name, e.date))
+ output.write(":leveloffset: 1\n\n")
+ output.write("include::%s[]\n\n" % (e.filename))
+ output.write(":leveloffset: 0\n\n")
+
+ output.write("''''\n")
+ for e in inputs[1:]:
+ output.write("* link:/news/%s[%s %s]\n" % (e.name, e.date, e.title))
+ output.write(OUTPUT_FOOTER)
+
+if __name__ == "__main__":
+ if sys.argv[1] == ATOM_NAME:
+ make_atom()
+ else:
+ main()
diff --git a/doc/website-v1/man-1.2.adoc b/doc/website-v1/man-1.2.adoc
new file mode 100644
index 0000000..d945719
--- /dev/null
+++ b/doc/website-v1/man-1.2.adoc
@@ -0,0 +1,3437 @@
+:man source: crm
+:man version: 1.2.6
+:man manual: crmsh documentation
+
+crm(8)
+======
+
+NOTE: This is the documentation for stable release 1.2.6 of `crmsh`.
+
+
+NAME
+----
+crm - Pacemaker command line interface for configuration and management
+
+
+SYNOPSIS
+--------
+*crm* [-D output_type] [-f file] [-c cib] [-H hist_src] [-hFRDw] [--version] [args]
+
+
+[[topics_Description,Program description]]
+DESCRIPTION
+-----------
+Pacemaker configuration is stored in a CIB file (Cluster
+Information Base). The CIB is a set of instructions coded in XML.
+Editing the CIB is a challenge, not only due to its complexity
+and a wide variety of options, but also because XML is more
+computer than user friendly. The `crm` shell alleviates this
+issue significantly by introducing small and simple configuration
+language. The CIB is translated into this language on the fly.
+
+`crm` is also a management tool. For management tasks it relies
+almost exclusively on other command line tools, such as
+`crm_resource(8)` or `crm_attribute(8)`. Use of these programs
+is, however, plagued by the notorious weakness common to all UNIX
+tools: a multitude of options, necessary for operation and yet
+very hard to remember. `crm` tries to present a consistent
+interface to the user and to hide the arcane detail.
+
+It may be used either as an interactive shell or for single
+commands directly on the shell's command line. It is also
+possible to feed it a set of commands from standard input or a
+file, thus turning it into a scripting tool. Templates with ready
+made configurations may help newbies learn about the cluster
+configuration or facilitate testing procedures.
+
+The `crm` shell is line oriented: every command must start and
+finish on the same line. It is possible to use a continuation
+character (`\`) to write one command in two or more lines. The
+continuation character is commonly used when displaying
+configurations.
+
+OPTIONS
+-------
+*-f, --file*='FILE'::
+ Load commands from the given file. If the file is `-` then
+ use terminal `stdin`.
+
+*-c, --cib*='CIB'::
+ Start the session with the given shadow CIB file.
+ Equivalent to `cib use`.
+
+*-D, --display=*'OUTPUT_TYPE'::
+ Choose one of the output options: `plain`, `color`, or
+ `uppercase`. The default is `color` if the terminal emulation
+ supports colors. Otherwise, `plain` is used.
+
+*-F, --force*::
+ Make `crm` proceed with doing changes even though it would
+ normally ask user to confirm some of them. Mostly useful in
+ scripts.
+
+*-w, --wait*::
+ Make `crm` wait for the cluster transition to finish (for the
+ changes to take effect) after each processed line.
+
+*-H, --history*='DIR|FILE'::
+ The `history` commands can examine either live cluster
+ (default) or a report generated by `hb_report`. Use this
+ option to specify a directory or file containing the report.
+
+*-h, --help*::
+ Print help page.
+
+*--version*::
+ Print crmsh version and build information (Mercurial Hg
+ changeset hash).
+
+*-R, --regression-tests*::
+ Run in the regression test mode. Used mainly by the
+ regression testing suite.
+
+*-d, --debug*::
+ Print some debug information. Used by developers. [Not yet
+ refined enough to print useful information for other users.]
+
+[[topics_Introduction,Introduction to the user interface]]
+== Introduction to the user interface
+
+Arguably the most important aspect of `crm` is the user
+interface. We begin with an informal introduction so that the
+reader may get acquainted with it and get a general feeling of
+the tool. It is probably best just to give some examples:
+
+1. Command line (one-shot) use:
+
+ # crm resource stop www_app
+
+2. Interactive use:
+
+ # crm
+ crm(live)# resource
+ crm(live)resource# unmanage tetris_1
+ crm(live)resource# end
+ crm(live)# node standby node4
+
+3. Cluster configuration:
+
+ # crm configure<<EOF
+ #
+ # resources
+ #
+ primitive disk0 iscsi \
+ params portal=192.168.2.108:3260 target=iqn.2008-07.com.suse:disk0
+ primitive fs0 Filesystem \
+ params device=/dev/disk/by-label/disk0 directory=/disk0 fstype=ext3
+ primitive internal_ip IPaddr params ip=192.168.1.101
+ primitive apache apache \
+ params configfile=/disk0/etc/apache2/site0.conf
+ primitive apcfence stonith:apcsmart \
+ params ttydev=/dev/ttyS0 hostlist="node1 node2" \
+ op start timeout=60s
+ primitive pingd pingd \
+ params name=pingd dampen=5s multiplier=100 host_list="r1 r2"
+ #
+ # monitor apache and the UPS
+ #
+ monitor apache 60s:30s
+ monitor apcfence 120m:60s
+ #
+ # cluster layout
+ #
+ group internal_www \
+ disk0 fs0 internal_ip apache
+ clone fence apcfence \
+ meta globally-unique=false clone-max=2 clone-node-max=1
+ clone conn pingd \
+ meta globally-unique=false clone-max=2 clone-node-max=1
+ location node_pref internal_www \
+ rule 50: #uname eq node1 \
+ rule pingd: defined pingd
+ #
+ # cluster properties
+ #
+ property stonith-enabled=true
+ commit
+ EOF
+
+If you've ever done a CRM style configuration, you should be able
+to understand the above examples without much difficulties. The
+shell should provide a means to manage the cluster efficiently or
+put together a configuration in a concise manner.
+
+The `(live)` string in the prompt signifies that the current CIB
+in use is the cluster live configuration. It is also possible to
+work with the so-called shadow CIBs, i.e. configurations which
+are stored in files and aren't active, but may be applied at any
+time to the cluster.
+
+Since the CIB is hierarchical such is the interface too. There
+are several levels and entering each of them enables the user to
+use a certain set of commands.
+
+[[topics_Shadows,Shadow CIB usage]]
+== Shadow CIB usage
+
+Shadow CIB is a normal cluster configuration stored in a file.
+They may be manipulated in the same way like the _live_ CIB, but
+these changes have no effect on the cluster resources. The
+administrator may choose to apply any of them to the cluster,
+thus replacing the running configuration with the one which is in
+the shadow CIB. The `crm` prompt always contains the name of the
+configuration which is currently in use or string _live_ if we
+are using the current cluster configuration.
+
+At the configure level no changes take place before the `commit`
+command. Sometimes though, the administrator may start working
+with the running configuration, but change mind and instead of
+committing the changes to the cluster save them to a shadow CIB.
+This short `configure` session excerpt shows how:
+...............
+ crm(live)configure# cib new test-2
+ INFO: test-2 shadow CIB created
+ crm(test-2)configure# commit
+...............
+
+[[topics_Templates,Configuration templates]]
+== Configuration templates
+
+Configuration templates are ready made configurations created by
+cluster experts. They are designed in such a way so that users
+may generate valid cluster configurations with minimum effort.
+If you are new to Pacemaker, templates may be the best way to
+start.
+
+We will show here how to create a simple yet functional Apache
+configuration:
+...............
+ # crm configure
+ crm(live)configure# template
+ crm(live)configure template# list templates
+ apache filesystem virtual-ip
+ crm(live)configure template# new web <TAB><TAB>
+ apache filesystem virtual-ip
+ crm(live)configure template# new web apache
+ INFO: pulling in template apache
+ INFO: pulling in template virtual-ip
+ crm(live)configure template# list
+ web2-d web2 vip2 web3 vip web
+...............
+
+We enter the `template` level from `configure`. Use the `list`
+command to show templates available on the system. The `new`
+command creates a configuration from the `apache` template. You
+can use tab completion to pick templates. Note that the apache
+template depends on a virtual IP address which is automatically
+pulled along. The `list` command shows the just created `web`
+configuration, among other configurations (I hope that you,
+unlike me, will use more sensible and descriptive names).
+
+The `show` command, which displays the resulting configuration,
+may be used to get an idea about the minimum required changes
+which have to be done. All `ERROR` messages show the line numbers
+in which the respective parameters are to be defined:
+...............
+ crm(live)configure template# show
+ ERROR: 23: required parameter ip not set
+ ERROR: 61: required parameter id not set
+ ERROR: 65: required parameter configfile not set
+ crm(live)configure template# edit
+...............
+
+The `edit` command invokes the preferred text editor with the
+`web` configuration. At the top of the file, the user is advised
+how to make changes. A good template should require from the user
+to specify only parameters. For example, the `web` configuration
+we created above has the following required and optional
+parameters (all parameter lines start with `%%`):
+...............
+ $ grep -n ^%% ~/.crmconf/web
+ 23:%% ip
+ 31:%% netmask
+ 35:%% lvs_support
+ 61:%% id
+ 65:%% configfile
+ 71:%% options
+ 76:%% envfiles
+...............
+
+These lines are the only ones that should be modified. Simply
+append the parameter value at the end of the line. For instance,
+after editing this template, the result could look like this (we
+used tabs instead of spaces to make the values stand out):
+...............
+ $ grep -n ^%% ~/.crmconf/web
+ 23:%% ip 192.168.1.101
+ 31:%% netmask
+ 35:%% lvs_support
+ 61:%% id websvc
+ 65:%% configfile /etc/apache2/httpd.conf
+ 71:%% options
+ 76:%% envfiles
+...............
+
+As you can see, the parameter line format is very simple:
+...............
+ %% <name> <value>
+...............
+
+After editing the file, use `show` again to display the
+configuration:
+...............
+ crm(live)configure template# show
+ primitive virtual-ip ocf:heartbeat:IPaddr \
+ params ip="192.168.1.101"
+ primitive apache ocf:heartbeat:apache \
+ params configfile="/etc/apache2/httpd.conf"
+ monitor apache 120s:60s
+ group websvc \
+ apache virtual-ip
+...............
+
+The target resource of the apache template is a group which we
+named `websvc` in this sample session.
+
+This configuration looks exactly as you could type it at the
+`configure` level. The point of templates is to save you some
+typing. It is important, however, to understand the configuration
+produced.
+
+Finally, the configuration may be applied to the current
+crm configuration (note how the configuration changed slightly,
+though it is still equivalent, after being digested at the
+`configure` level):
+...............
+ crm(live)configure template# apply
+ crm(live)configure template# cd ..
+ crm(live)configure# show
+ node xen-b
+ node xen-c
+ primitive apache ocf:heartbeat:apache \
+ params configfile="/etc/apache2/httpd.conf" \
+ op monitor interval="120s" timeout="60s"
+ primitive virtual-ip ocf:heartbeat:IPaddr \
+ params ip="192.168.1.101"
+ group websvc apache virtual-ip
+...............
+
+Note that this still does not commit the configuration to the CIB
+which is used in the shell, either the running one (`live`) or
+some shadow CIB. For that you still need to execute the `commit`
+command.
+
+To complete our example, we should also define the preferred node
+to run the service:
+...............
+ crm(live)configure# location websvc-pref websvc 100: xen-b
+...............
+
+If you are not happy with some resource names which are provided
+by default, you can rename them now:
+...............
+ crm(live)configure# rename virtual-ip intranet-ip
+ crm(live)configure# show
+ node xen-b
+ node xen-c
+ primitive apache ocf:heartbeat:apache \
+ params configfile="/etc/apache2/httpd.conf" \
+ op monitor interval="120s" timeout="60s"
+ primitive intranet-ip ocf:heartbeat:IPaddr \
+ params ip="192.168.1.101"
+ group websvc apache intranet-ip
+ location websvc-pref websvc 100: xen-b
+...............
+
+To summarize, working with templates typically consists of the
+following steps:
+
+- `new`: create a new configuration from templates
+- `edit`: define parameters, at least the required ones
+- `show`: see if the configuration is valid
+- `apply`: apply the configuration to the `configure` level
+
+[[topics_Testing,Resource testing]]
+== Resource testing
+
+The amount of detail in a cluster makes all configurations prone
+to errors. By far the largest number of issues in a cluster is
+due to bad resource configuration. The shell can help quickly
+diagnose such problems. And considerably reduce your keyboard
+wear.
+
+Let's say that we entered the following configuration:
+...............
+ node xen-b
+ node xen-c
+ node xen-d
+ primitive fencer stonith:external/libvirt \
+ params hypervisor_uri="qemu+tcp://10.2.13.1/system" \
+ hostlist="xen-b xen-c xen-d" \
+ op monitor interval="2h"
+ primitive svc ocf:heartbeat:Xinetd \
+ params service="systat" \
+ op monitor interval="30s"
+ primitive intranet-ip ocf:heartbeat:IPaddr2 \
+ params ip="10.2.13.100" \
+ op monitor interval="30s"
+ primitive apache ocf:heartbeat:apache \
+ params configfile="/etc/apache2/httpd.conf" \
+ op monitor interval="120s" timeout="60s"
+ group websvc apache intranet-ip
+ location websvc-pref websvc 100: xen-b
+...............
+
+Before typing `commit` to submit the configuration to the cib we
+can make sure that all resources are usable on all nodes:
+...............
+ crm(live)configure# rsctest websvc svc fencer
+...............
+
+It is important that resources being tested are not running on
+any nodes. Otherwise, the `rsctest` command will refuse to do
+anything. Of course, if the current configuration resides in a
+CIB shadow, then a `commit` is irrelevant. The point being that
+resources are not running on any node.
+
+.Note on stopping all resources
+****************************
+Alternatively to not committing a configuration, it is also
+possible to tell Pacemaker not to start any resources:
+
+...............
+ crm(live)configure# property stop-all-resources="yes"
+...............
+Almost none---resources of class stonith are still started. But
+shell is not as strict when it comes to stonith resources.
+****************************
+
+Order of resources is significant insofar that a resource depends
+on all resources to its left. In most configurations, it's
+probably practical to test resources in several runs, based on
+their dependencies.
+
+Apart from groups, `crm` does not interpret constraints and
+therefore knows nothing about resource dependencies. It also
+doesn't know if a resource can run on a node at all in case of an
+asymmetric cluster. It is up to the user to specify a list of
+eligible nodes if a resource is not meant to run on every node.
+
+[[topics_Completion,Tab completion]]
+== Tab completion
+
+The `crm` makes extensive use of tab completion. The completion
+is both static (i.e. for `crm` commands) and dynamic. The latter
+takes into account the current status of the cluster or
+information from installed resource agents. Sometimes, completion
+may also be used to get short help on resource parameters. Here a
+few examples:
+...............
+ crm(live)# resource
+ crm(live)resource# <TAB><TAB>
+ bye failcount move restart unmigrate
+ cd help param show unmove
+ cleanup list promote start up
+ demote manage quit status utilization
+ end meta refresh stop
+ exit migrate reprobe unmanage
+ crm(live)resource# end
+ crm(live)# configure
+ crm(live)configure# primitive fence-1 <TAB><TAB>
+ heartbeat: lsb: ocf: stonith:
+ crm(live)configure# primitive fence-1 stonith:<TAB><TAB>
+ apcmaster external/ippower9258 fence_legacy
+ apcmastersnmp external/kdumpcheck ibmhmc
+ apcsmart external/libvirt ipmilan
+ baytech external/nut meatware
+ bladehpi external/rackpdu null
+ cyclades external/riloe nw_rpc100s
+ drac3 external/sbd rcd_serial
+ external/drac5 external/ssh rps10
+ external/dracmc-telnet external/ssh-bad ssh
+ external/hmchttp external/ssh-slow suicide
+ external/ibmrsa external/vmware wti_mpc
+ external/ibmrsa-telnet external/xen0 wti_nps
+ external/ipmi external/xen0-ha
+ crm(live)configure# primitive fence-1 stonith:ipmilan params <TAB><TAB>
+ auth= hostname= ipaddr= login= password= port= priv=
+ crm(live)configure# primitive fence-1 stonith:ipmilan params auth=<TAB><TAB>
+ auth* (string)
+ The authorization type of the IPMI session ("none", "straight", "md2", or "md5")
+ crm(live)configure# primitive fence-1 stonith:ipmilan params auth=
+...............
+
+[[topics_Checks,Configuration semantic checks]]
+== Configuration semantic checks
+
+Resource definitions may be checked against the meta-data
+provided with the resource agents. These checks are currently
+carried out:
+
+- are required parameters set
+- existence of defined parameters
+- timeout values for operations
+
+The parameter checks are obvious and need no further explanation.
+Failures in these checks are treated as configuration errors.
+
+The timeouts for operations should be at least as long as those
+recommended in the meta-data. Too short timeout values are a
+common mistake in cluster configurations and, even worse, they
+often slip through if cluster testing was not thorough. Though
+operation timeouts issues are treated as warnings, make sure that
+the timeouts are usable in your environment. Note also that the
+values given are just _advisory minimum_---your resources may
+require longer timeouts.
+
+User may tune the frequency of checks and the treatment of errors
+by the <<cmdhelp_options_check-frequency,`check-frequency`>> and
+<<cmdhelp_options_check-mode,`check-mode`>> preferences.
+
+Note that if the `check-frequency` is set to `always` and the
+`check-mode` to `strict`, errors are not tolerated and such
+configuration cannot be saved.
+
+[[topics_Security,Access Control Lists (ACL)]]
+== Access Control Lists (ACL)
+
+By default, the users from the `haclient` group have full access
+to the cluster (or, more precisely, to the CIB). Access control
+lists allow for finer access control to the cluster.
+
+Access control lists consist of an ordered set of access rules.
+Each rule allows read or write access or denies access
+completely. Rules are typically combined to produce a specific
+role. Then, users may be assigned a role.
+
+For instance, this is a role which defines a set of rules
+allowing management of a single resource:
+
+...............
+ role bigdb_admin \
+ write meta:bigdb:target-role \
+ write meta:bigdb:is-managed \
+ write location:bigdb \
+ read ref:bigdb
+...............
+
+The first two rules allow modifying the `target-role` and
+`is-managed` meta attributes which effectively enables users in
+this role to stop/start and manage/unmanage the resource. The
+constraints write access rule allows moving the resource around.
+Finally, the user is granted read access to the resource
+definition.
+
+For proper operation of all Pacemaker programs, it is advisable
+to add the following role to all users:
+
+...............
+ role read_all \
+ read cib
+...............
+
+For finer grained read access try with the rules listed in the
+following role:
+
+...............
+ role basic_read \
+ read node attribute:uname \
+ read node attribute:type \
+ read property \
+ read status
+...............
+
+It is however possible that some Pacemaker programs (e.g.
+`ptest`) may not function correctly if the whole CIB is not
+readable.
+
+Some of the ACL rules in the examples above are expanded by the
+shell to XPath specifications. For instance,
+`meta:bigdb:target-role` is a shortcut for
+`//primitive[@id='bigdb']/meta_attributes/nvpair[@name='target-role']`.
+You can see the expansion by showing XML:
+
+...............
+ crm(live) configure# show xml bigdb_admin
+ ...
+ <acls>
+ <acl_role id="bigdb_admin">
+ <write id="bigdb_admin-write"
+ xpath="//primitive[@id='bigdb']/meta_attributes/nvpair[@name='target-role']"/>
+...............
+
+Many different XPath expressions can have equal meaning. For
+instance, the following two are equal, but only the first one is
+going to be recognized as shortcut:
+
+...............
+ //primitive[@id='bigdb']/meta_attributes/nvpair[@name='target-role']
+ //resources/primitive[@id='bigdb']/meta_attributes/nvpair[@name='target-role']
+...............
+
+XPath is a powerful language, but you should try to keep your ACL
+xpaths simple and the builtin shortcuts should be used whenever
+possible.
+
+[[topics_Reference,Command reference]]
+== Command reference
+
+We define a small and simple language. Most commands consist of
+just a list of simple tokens. The only complex constructs are
+found at the `configure` level.
+
+The syntax is described in a somewhat informal manner: `<>`
+denotes a string, `[]` means that the construct is optional, the
+ellipsis (`...`) signifies that the previous construct may be
+repeated, `|` means pick one of many, and the rest are literals
+(strings, `:`, `=`).
+
+=== `status`
+
+Show cluster status. The status is displayed by `crm_mon`. Supply
+additional arguments for more information or different format.
+See `crm_mon(8)` for more details.
+
+Usage:
+...............
+ status [<option> ...]
+
+ option :: bynode | inactive | ops | timing | failcounts
+...............
+
+[[cmdhelp_cib,CIB shadow management]]
+=== `cib` (shadow CIBs)
+
+This level is for management of shadow CIBs. It is available both
+at the top level and the `configure` level.
+
+All the commands are implemented using `cib_shadow(8)` and the
+`CIB_shadow` environment variable. The user prompt always
+includes the name of the currently active shadow or the live CIB.
+
+[[cmdhelp_cib_new,create a new shadow CIB]]
+==== `new`
+
+Create a new shadow CIB. The live cluster configuration and
+status is copied to the shadow CIB. Specify `withstatus` if you
+want to edit the status section of the shadow CIB (see the
+<<cmdhelp_cibstatus,cibstatus section>>). Add `force` to force overwriting the
+existing shadow CIB.
+
+To start with an empty configuration that is not copied from the live
+CIB, specify the `empty` keyword. (This also allows a shadow CIB to be
+created in case no cluster is running.)
+
+Usage:
+...............
+ new <cib> [withstatus] [force] [empty]
+...............
+
+[[cmdhelp_cib_delete,delete a shadow CIB]]
+==== `delete`
+
+Delete an existing shadow CIB.
+
+Usage:
+...............
+ delete <cib>
+...............
+
+[[cmdhelp_cib_reset,copy live cib to a shadow CIB]]
+==== `reset`
+
+Copy the current cluster configuration into the shadow CIB.
+
+Usage:
+...............
+ reset <cib>
+...............
+
+[[cmdhelp_cib_commit,copy a shadow CIB to the cluster]]
+==== `commit`
+
+Apply a shadow CIB to the cluster.
+
+Usage:
+...............
+ commit <cib>
+...............
+
+[[cmdhelp_cib_use,change working CIB]]
+==== `use`
+
+Choose a CIB source. If you want to edit the status from the
+shadow CIB specify `withstatus` (see <<cmdhelp_cibstatus,`cibstatus`>>).
+Leave out the CIB name to switch to the running CIB.
+
+Usage:
+...............
+ use [<cib>] [withstatus]
+...............
+
+[[cmdhelp_cib_diff,diff between the shadow CIB and the live CIB]]
+==== `diff`
+
+Print differences between the current cluster configuration and
+the active shadow CIB.
+
+Usage:
+...............
+ diff
+...............
+
+[[cmdhelp_cib_list,list all shadow CIBs]]
+==== `list`
+
+List existing shadow CIBs.
+
+Usage:
+...............
+ list
+...............
+
+[[cmdhelp_cib_import,import a CIB or PE input file to a shadow]]
+==== `import`
+
+At times it may be useful to create a shadow file from the
+existing CIB. The CIB may be specified as file or as a PE input
+file number. The shell will look up files in the local directory
+first and then in the PE directory (typically `/var/lib/pengine`).
+Once the CIB file is found, it is copied to a shadow and this
+shadow is immediately available for use at both `configure` and
+`cibstatus` levels.
+
+If the shadow name is omitted then the target shadow is named
+after the input CIB file.
+
+Note that there are often more than one PE input file, so you may
+need to specify the full name.
+
+Usage:
+...............
+ import {<file>|<number>} [<shadow>]
+...............
+Examples:
+...............
+ import pe-warn-2222
+ import 2289 issue2
+...............
+
+[[cmdhelp_cib_cibstatus,CIB status management and editing]]
+==== `cibstatus`
+
+Enter edit and manage the CIB status section level. See the
+<<cmdhelp_cibstatus,CIB status management section>>.
+
+[[cmdhelp_ra,Resource Agents (RA) lists and documentation]]
+=== `ra`
+
+This level contains commands which show various information about
+the installed resource agents. It is available both at the top
+level and at the `configure` level.
+
+[[cmdhelp_ra_classes,list classes and providers]]
+==== `classes`
+
+Print all resource agents' classes and, where appropriate, a list
+of available providers.
+
+Usage:
+...............
+ classes
+...............
+
+[[cmdhelp_ra_list,list RA for a class (and provider)]]
+==== `list`
+
+List available resource agents for the given class. If the class
+is `ocf`, supply a provider to get agents which are available
+only from that provider.
+
+Usage:
+...............
+ list <class> [<provider>]
+...............
+Example:
+...............
+ list ocf pacemaker
+...............
+
+[[cmdhelp_ra_meta,show meta data for a RA]]
+==== `meta` (`info`)
+
+Show the meta-data of a resource agent type. This is where users
+can find information on how to use a resource agent. It is also
+possible to get information from some programs: `pengine`,
+`crmd`, `cib`, and `stonithd`. Just specify the program name
+instead of an RA.
+
+Usage:
+...............
+ info [<class>:[<provider>:]]<type>
+ info <type> <class> [<provider>] (obsolete)
+...............
+Example:
+...............
+ info apache
+ info ocf:pacemaker:Dummy
+ info stonith:ipmilan
+ info pengine
+...............
+
+[[cmdhelp_ra_providers,show providers for a RA and a class]]
+==== `providers`
+
+List providers for a resource agent type. The class parameter
+defaults to `ocf`.
+
+Usage:
+...............
+ providers <type> [<class>]
+...............
+Example:
+...............
+ providers apache
+...............
+
+[[cmdhelp_resource,Resource management]]
+=== `resource`
+
+At this level resources may be managed.
+
+All (or almost all) commands are implemented with the CRM tools
+such as `crm_resource(8)`.
+
+[[cmdhelp_resource_status,show status of resources]]
+==== `status` (`show`, `list`)
+
+Print resource status. If the resource parameter is left out
+status of all resources is printed.
+
+Usage:
+...............
+ status [<rsc>]
+...............
+
+[[cmdhelp_resource_start,start a resource]]
+==== `start`
+
+Start a resource by setting the `target-role` attribute. If there
+are multiple meta attributes sets, the attribute is set in all of
+them. If the resource is a clone, all `target-role` attributes
+are removed from the children resources.
+
+For details on group management see <<cmdhelp_options_manage-children,`options manage-children`>>.
+
+Usage:
+...............
+ start <rsc>
+...............
+
+[[cmdhelp_resource_stop,stop a resource]]
+==== `stop`
+
+Stop a resource using the `target-role` attribute. If there
+are multiple meta attributes sets, the attribute is set in all of
+them. If the resource is a clone, all `target-role` attributes
+are removed from the children resources.
+
+For details on group management see <<cmdhelp_options_manage-children,`options manage-children`>>.
+
+Usage:
+...............
+ stop <rsc>
+...............
+
+[[cmdhelp_resource_restart,restart a resource]]
+==== `restart`
+
+Restart a resource. This is essentially a shortcut for resource
+stop followed by a start. The shell is first going to wait for
+the stop to finish, that is for all resources to really stop, and
+only then to order the start action. Due to this command
+entailing a whole set of operations, informational messages are
+printed to let the user see some progress.
+
+For details on group management see <<cmdhelp_options_manage-children,`options manage-children`>>.
+
+Usage:
+...............
+ restart <rsc>
+...............
+Example:
+...............
+ # crm resource restart g_webserver
+ INFO: ordering g_webserver to stop
+ waiting for stop to finish .... done
+ INFO: ordering g_webserver to start
+ #
+...............
+
+[[cmdhelp_resource_promote,promote a master-slave resource]]
+==== `promote`
+
+Promote a master-slave resource using the `target-role`
+attribute.
+
+Usage:
+...............
+ promote <rsc>
+...............
+
+[[cmdhelp_resource_demote,demote a master-slave resource]]
+==== `demote`
+
+Demote a master-slave resource using the `target-role`
+attribute.
+
+Usage:
+...............
+ demote <rsc>
+...............
+
+[[cmdhelp_resource_manage,put a resource into managed mode]]
+==== `manage`
+
+Manage a resource using the `is-managed` attribute. If there
+are multiple meta attributes sets, the attribute is set in all of
+them. If the resource is a clone, all `is-managed` attributes are
+removed from the children resources.
+
+For details on group management see <<cmdhelp_options_manage-children,`options manage-children`>>.
+
+Usage:
+...............
+ manage <rsc>
+...............
+
+[[cmdhelp_resource_unmanage,put a resource into unmanaged mode]]
+==== `unmanage`
+
+Unmanage a resource using the `is-managed` attribute. If there
+are multiple meta attributes sets, the attribute is set in all of
+them. If the resource is a clone, all `is-managed` attributes are
+removed from the children resources.
+
+For details on group management see <<cmdhelp_options_manage-children,`options manage-children`>>.
+
+Usage:
+...............
+ unmanage <rsc>
+...............
+
+[[cmdhelp_resource_migrate,migrate a resource to another node]]
+==== `migrate` (`move`)
+
+Migrate a resource to a different node. If node is left out, the
+resource is migrated by creating a constraint which prevents it from
+running on the current node. Additionally, you may specify a
+lifetime for the constraint---once it expires, the location
+constraint will no longer be active.
+
+Usage:
+...............
+ migrate <rsc> [<node>] [<lifetime>] [force]
+...............
+
+[[cmdhelp_resource_unmigrate,unmigrate a resource to another node]]
+==== `unmigrate` (`unmove`)
+
+Remove the constraint generated by the previous migrate command.
+
+Usage:
+...............
+ unmigrate <rsc>
+...............
+
+[[cmdhelp_resource_param,manage a parameter of a resource]]
+==== `param`
+
+Show/edit/delete a parameter of a resource.
+
+Usage:
+...............
+ param <rsc> set <param> <value>
+ param <rsc> delete <param>
+ param <rsc> show <param>
+...............
+Example:
+...............
+ param ip_0 show ip
+...............
+
+[[cmdhelp_resource_secret,manage sensitive parameters]]
+==== `secret`
+
+Sensitive parameters can be kept in local files rather than CIB
+in order to prevent accidental data exposure. Use the `secret`
+command to manage such parameters. `stash` and `unstash` move the
+value from the CIB and back to the CIB respectively. The `set`
+subcommand sets the parameter to the provided value. `delete`
+removes the parameter completely. `show` displays the value of
+the parameter from the local file. Use `check` to verify if the
+local file content is valid.
+
+Usage:
+...............
+ secret <rsc> set <param> <value>
+ secret <rsc> stash <param>
+ secret <rsc> unstash <param>
+ secret <rsc> delete <param>
+ secret <rsc> show <param>
+ secret <rsc> check <param>
+...............
+Example:
+...............
+ secret fence_1 show password
+ secret fence_1 stash password
+ secret fence_1 set password secret_value
+...............
+
+[[cmdhelp_resource_meta,manage a meta attribute]]
+==== `meta`
+
+Show/edit/delete a meta attribute of a resource. Currently, all
+meta attributes of a resource may be managed with other commands
+such as `resource stop`.
+
+Usage:
+...............
+ meta <rsc> set <attr> <value>
+ meta <rsc> delete <attr>
+ meta <rsc> show <attr>
+...............
+Example:
+...............
+ meta ip_0 set target-role stopped
+...............
+
+[[cmdhelp_resource_utilization,manage a utilization attribute]]
+==== `utilization`
+
+Show/edit/delete a utilization attribute of a resource. These
+attributes describe hardware requirements. By setting the
+`placement-strategy` cluster property appropriately, it is
+possible then to distribute resources based on resource
+requirements and node size. See also <<cmdhelp_node_utilization,node utilization attributes>>.
+
+Usage:
+...............
+ utilization <rsc> set <attr> <value>
+ utilization <rsc> delete <attr>
+ utilization <rsc> show <attr>
+...............
+Example:
+...............
+ utilization xen1 set memory 4096
+...............
+
+[[cmdhelp_resource_failcount,manage failcounts]]
+==== `failcount`
+
+Show/edit/delete the failcount of a resource.
+
+Usage:
+...............
+ failcount <rsc> set <node> <value>
+ failcount <rsc> delete <node>
+ failcount <rsc> show <node>
+...............
+Example:
+...............
+ failcount fs_0 delete node2
+...............
+
+[[cmdhelp_resource_cleanup,cleanup resource status]]
+==== `cleanup`
+
+Cleanup resource status. Typically done after the resource has
+temporarily failed. If a node is omitted, cleanup on all nodes.
+If there are many nodes, the command may take a while.
+
+Usage:
+...............
+ cleanup <rsc> [<node>]
+...............
+
+[[cmdhelp_resource_refresh,refresh CIB from the LRM status]]
+==== `refresh`
+
+Refresh CIB from the LRM status.
+
+Usage:
+...............
+ refresh [<node>]
+...............
+
+[[cmdhelp_resource_reprobe,probe for resources not started by the CRM]]
+==== `reprobe`
+
+Probe for resources not started by the CRM.
+
+Usage:
+...............
+ reprobe [<node>]
+...............
+
+[[cmdhelp_resource_trace,start RA tracing]]
+==== `trace`
+
+Start tracing RA for the given operation. The trace files are
+stored in `$HA_VARLIB/trace_ra`. If the operation to be traced is
+monitor, note that the number of trace files can grow very
+quickly.
+
+Usage:
+...............
+ trace <rsc> <op> [<interval>]
+...............
+Example:
+...............
+ trace fs start
+...............
+
+[[cmdhelp_resource_untrace,stop RA tracing]]
+==== `untrace`
+
+Stop tracing RA for the given operation.
+
+Usage:
+...............
+ untrace <rsc> <op> [<interval>]
+...............
+Example:
+...............
+ untrace fs start
+...............
+
+[[cmdhelp_node,Nodes management]]
+=== `node`
+
+Node management and status commands.
+
+[[cmdhelp_node_status,show nodes' status as XML]]
+==== `status`
+
+Show nodes' status as XML. If the node parameter is omitted then
+all nodes are shown.
+
+Usage:
+...............
+ status [<node>]
+...............
+
+[[cmdhelp_node_show,show node]]
+==== `show`
+
+Show a node definition. If the node parameter is omitted then all
+nodes are shown.
+
+Usage:
+...............
+ show [<node>]
+...............
+
+[[cmdhelp_node_standby,put node into standby]]
+==== `standby`
+
+Set a node to standby status. The node parameter defaults to the
+node where the command is run. Additionally, you may specify a
+lifetime for the standby---if set to `reboot`, the node will be
+back online once it reboots. `forever` will keep the node in
+standby after reboot.
+
+Usage:
+...............
+ standby [<node>] [<lifetime>]
+
+ lifetime :: reboot | forever
+...............
+
+[[cmdhelp_node_online,set node online]]
+==== `online`
+
+Set a node to online status. The node parameter
+defaults to the node where the command is run.
+
+Usage:
+...............
+ online [<node>]
+...............
+
+[[cmdhelp_node_maintenance,put node into maintenance mode]]
+==== `maintenance`
+
+Set the node status to maintenance. This is equivalent to the
+cluster-wide `maintenance-mode` property but puts just one node
+into the maintenance mode. The node parameter defaults to the
+node where the command is run.
+
+Usage:
+...............
+ maintenance [<node>]
+...............
+
+[[cmdhelp_node_ready,put node into ready mode]]
+==== `ready`
+
+Set the node's maintenance status to `off`. The node should be
+now again fully operational and capable of running resource
+operations.
+
+Usage:
+...............
+ ready [<node>]
+...............
+
+[[cmdhelp_node_fence,fence node]]
+==== `fence`
+
+Make CRM fence a node. This functionality depends on stonith
+resources capable of fencing the specified node. No such stonith
+resources, no fencing will happen.
+
+Usage:
+...............
+ fence <node>
+...............
+
+[[cmdhelp_node_clearstate,Clear node state]]
+==== `clearnodestate`
+
+Resets and clears the state of the specified node. This node is
+afterwards assumed clean and offline. This command can be used to
+manually confirm that a node has been fenced (e.g., powered off).
+
+Be careful! This can cause data corruption if you confirm that a node is
+down that is, in fact, not cleanly down - the cluster will proceed as if
+the fence had succeeded, possibly starting resources multiple times.
+
+Usage:
+...............
+ clearstate <node>
+...............
+
+[[cmdhelp_node_delete,delete node]]
+==== `delete`
+
+Delete a node. This command will remove the node from the CIB
+and, in case the cluster stack is running, use the appropriate
+program (`crm_node` or `hb_delnode`) to remove the node from the
+membership.
+
+If the node is still listed as active and a member of our
+partition we refuse to remove it. With the global force option
+(`-F`) we will try to delete the node anyway.
+
+Usage:
+...............
+ delete <node>
+...............
+
+[[cmdhelp_node_attribute,manage attributes]]
+==== `attribute`
+
+Edit node attributes. This kind of attribute should refer to
+relatively static properties, such as memory size.
+
+Usage:
+...............
+ attribute <node> set <attr> <value>
+ attribute <node> delete <attr>
+ attribute <node> show <attr>
+...............
+Example:
+...............
+ attribute node_1 set memory_size 4096
+...............
+
+[[cmdhelp_node_utilization,manage utilization attributes]]
+==== `utilization`
+
+Edit node utilization attributes. These attributes describe
+hardware characteristics as integer numbers such as memory size
+or the number of CPUs. By setting the `placement-strategy`
+cluster property appropriately, it is possible then to distribute
+resources based on resource requirements and node size. See also
+<<cmdhelp_resource_utilization,resource utilization attributes>>.
+
+Usage:
+...............
+ utilization <node> set <attr> <value>
+ utilization <node> delete <attr>
+ utilization <node> show <attr>
+...............
+Examples:
+...............
+ utilization node_1 set memory 16384
+ utilization node_1 show cpu
+...............
+
+[[cmdhelp_node_status-attr,manage status attributes]]
+==== `status-attr`
+
+Edit node attributes which are in the CIB status section, i.e.
+attributes which hold properties of a more volatile nature. One
+typical example is attribute generated by the `pingd` utility.
+
+Usage:
+...............
+ status-attr <node> set <attr> <value>
+ status-attr <node> delete <attr>
+ status-attr <node> show <attr>
+...............
+Example:
+...............
+ status-attr node_1 show pingd
+...............
+
+[[cmdhelp_site,site support]]
+=== `site`
+
+A cluster may consist of two or more subclusters in different and
+distant locations. This set of commands supports such setups.
+
+[[cmdhelp_site_ticket,manage site tickets]]
+==== `ticket`
+
+Tickets are cluster-wide attributes. They can be managed at the
+site where this command is executed.
+
+It is then possible to constrain resources depending on the
+ticket availability (see the <<cmdhelp_configure_rsc_ticket,`rsc_ticket`>> command
+for more details).
+
+Usage:
+...............
+ ticket {grant|revoke|standby|activate|show|time|delete} <ticket>
+...............
+Example:
+...............
+ ticket grant ticket1
+...............
+
+[[cmdhelp_options,user preferences]]
+=== `options`
+
+The user may set various options for the crm shell itself.
+
+[[cmdhelp_options_skill-level,set skill level]]
+==== `skill-level`
+
+Based on the skill-level setting, the user is allowed to use only
+a subset of commands. There are three levels: operator,
+administrator, and expert. The operator level allows only
+commands at the `resource` and `node` levels, but not editing
+or deleting resources. The administrator may do that and may also
+configure the cluster at the `configure` level and manage the
+shadow CIBs. The expert may do all.
+
+Usage:
+...............
+ skill-level <level>
+
+ level :: operator | administrator | expert
+...............
+
+.Note on security
+****************************
+The `skill-level` option is advisory only. There is nothing
+stopping any users change their skill level (see
+<<topics_Security,Access Control Lists (ACL)>> on how to enforce
+access control).
+****************************
+
+[[cmdhelp_options_user,set the cluster user]]
+==== `user`
+
+Sufficient privileges are necessary in order to manage a
+cluster: programs such as `crm_verify` or `crm_resource` and,
+ultimately, `cibadmin` have to be run either as `root` or as the
+CRM owner user (typically `hacluster`). You don't have to worry
+about that if you run `crm` as `root`. A more secure way is to
+run the program with your usual privileges, set this option to
+the appropriate user (such as `hacluster`), and setup the
+`sudoers` file.
+
+Usage:
+...............
+ user system-user
+...............
+Example:
+...............
+ user hacluster
+...............
+
+[[cmdhelp_options_editor,set preferred editor program]]
+==== `editor`
+
+The `edit` command invokes an editor. Use this to specify your
+preferred editor program. If not set, it will default to either
+the value of the `EDITOR` environment variable or to one of the
+standard UNIX editors (`vi`,`emacs`,`nano`).
+
+Usage:
+...............
+ editor program
+...............
+Example:
+...............
+ editor vim
+...............
+
+[[cmdhelp_options_pager,set preferred pager program]]
+==== `pager`
+
+The `view` command displays text through a pager. Use this to
+specify your preferred pager program. If not set, it will default
+to either the value of the `PAGER` environment variable or to one
+of the standard UNIX system pagers (`less`,`more`,`pg`).
+
+[[cmdhelp_options_sort-elements,sort CIB elements]]
+==== `sort-elements`
+
+`crm` by default sorts CIB elements. If you want them appear in
+the order they were created, set this option to `no`.
+
+Usage:
+...............
+ sort-elements {yes|no}
+...............
+Example:
+...............
+ sort-elements no
+...............
+
+[[cmdhelp_options_wait,synchronous operation]]
+==== `wait`
+
+In normal operation, `crm` runs a command and gets back
+immediately to process other commands or get input from the user.
+With this option set to `yes` it will wait for the started
+transition to finish. In interactive mode dots are printed to
+indicate progress.
+
+Usage:
+...............
+ wait {yes|no}
+...............
+Example:
+...............
+ wait yes
+...............
+
+[[cmdhelp_options_output,set output type]]
+==== `output`
+
+`crm` can adorn configurations in two ways: in color (similar to
+for instance the `ls --color` command) and by showing keywords in
+upper case. Possible values are `plain`, `color`, and
+'uppercase'. It is possible to combine the latter two in order to
+get an upper case xmass tree. Just set this option to
+`color,uppercase`.
+
+[[cmdhelp_options_colorscheme,set colors for output]]
+==== `colorscheme`
+
+With `output` set to `color`, a comma separated list of colors
+from this option are used to emphasize:
+
+- keywords
+- object ids
+- attribute names
+- attribute values
+- scores
+- resource references
+
+`crm` can show colors only if there is curses support for python
+installed (usually provided by the `python-curses` package). The
+colors are whatever is available in your terminal. Use `normal`
+if you want to keep the default foreground color.
+
+This user preference defaults to
+`yellow,normal,cyan,red,green,magenta` which is good for
+terminals with dark background. You may want to change the color
+scheme and save it in the preferences file for other color
+setups.
+
+Example:
+...............
+ colorscheme yellow,normal,blue,red,green,magenta
+...............
+
+[[cmdhelp_options_check-frequency,when to perform semantic check]]
+==== `check-frequency`
+
+Semantic check of the CIB or elements modified or created may be
+done on every configuration change (`always`), when verifying
+(`on-verify`) or `never`. It is by default set to `always`.
+Experts may want to change the setting to `on-verify`.
+
+The checks require that resource agents are present. If they are
+not installed at the configuration time set this preference to
+`never`.
+
+See <<topics_Checks,Configuration semantic checks>> for more details.
+
+[[cmdhelp_options_check-mode,how to treat semantic errors]]
+==== `check-mode`
+
+Semantic check of the CIB or elements modified or created may be
+done in the `strict` mode or in the `relaxed` mode. In the former
+certain problems are treated as configuration errors. In the
+`relaxed` mode all are treated as warnings. The default is `strict`.
+
+See <<topics_Checks,Configuration semantic checks>> for more details.
+
+[[cmdhelp_options_add-quotes,add quotes around parameters containing spaces]]
+==== `add-quotes`
+
+The shell (as in `/bin/sh`) parser strips quotes from the command
+line. This may sometimes make it really difficult to type values
+which contain white space. One typical example is the configure
+filter command. The crm shell will supply extra quotes around
+arguments which contain white space. The default is `yes`.
+
+.Note on quotes use
+****************************
+Adding quotes around arguments automatically has been introduced
+with version 1.2.2 and it is technically a regression. Being a
+regression is the only reason the `add-quotes` option exists. If
+you have custom shell scripts which would break, just set the
+`add-quotes` option to `no`.
+
+For instance, with adding quotes enabled, it is possible to do
+the following:
+...............
+ # crm configure primitive d1 ocf:heartbeat:Dummy meta description="some description here"
+ # crm configure filter 'sed "s/hostlist=./&node-c /"' fencing
+...............
+****************************
+
+[[cmdhelp_options_manage-children,how to handle children resource attributes]]
+==== `manage-children`
+
+Some resource management commands, such as `resource stop`, when
+the target resource is a group, may not always produce desired
+result. Each element, group and the primitive members, can have a
+meta attribute and those attributes may end up with conflicting
+values. Consider the following construct:
+...............
+ crm(live)# configure show svc fs virtual-ip
+ primitive fs ocf:heartbeat:Filesystem \
+ params device="/dev/drbd0" directory="/srv/nfs" fstype="ext3" \
+ op monitor interval="10s" \
+ meta target-role="Started"
+ primitive virtual-ip ocf:heartbeat:IPaddr2 \
+ params ip="10.2.13.110" iflabel="1" \
+ op monitor interval="10s" \
+ op start interval="0" \
+ meta target-role="Started"
+ group svc fs virtual-ip \
+ meta target-role="Stopped"
+...............
+
+Even though the element `svc` should be stopped, the group is
+actually running because all its members have the `target-role`
+set to `Started`:
+...............
+ crm(live)# resource show svc
+ resource svc is running on: xen-f
+...............
+
+Hence, if the user invokes `resource stop svc` the intention is
+not clear. This preference gives the user an opportunity to
+better control what happens if attributes of group members have
+values which are in conflict with the same attribute of the group
+itself.
+
+Possible values are `ask` (the default), `always`, and `never`.
+If set to `always`, the crm shell removes all children attributes
+which have values different from the parent. If set to `never`,
+all children attributes are left intact. Finally, if set to
+`ask`, the user will be asked for each member what is to be done.
+
+[[cmdhelp_options_show,show current user preference]]
+==== `show`
+
+Display all current settings.
+
+[[cmdhelp_options_save,save the user preferences to the rc file]]
+==== `save`
+
+Save current settings to the rc file (`$HOME/.config/crm/rc`). On
+further `crm` runs, the rc file is automatically read and parsed.
+
+[[cmdhelp_options_reset,reset user preferences to factory defaults]]
+==== `reset`
+
+This command resets all user options to the defaults. If used as
+a single-shot command, the rc file (`$HOME/.config/crm/rc`) is
+reset to the defaults too.
+
+[[cmdhelp_configure,CIB configuration]]
+=== `configure`
+
+This level enables all CIB object definition commands.
+
+The configuration may be logically divided into four parts:
+nodes, resources, constraints, and (cluster) properties and
+attributes. Each of these commands support one or more basic CIB
+objects.
+
+Nodes and attributes describing nodes are managed using the
+`node` command.
+
+Commands for resources are:
+
+- `primitive`
+- `monitor`
+- `group`
+- `clone`
+- `ms`/`master` (master-slave)
+
+In order to streamline large configurations, it is possible to
+define a template which can later be referenced in primitives:
+
+- `rsc_template`
+
+In that case the primitive inherits all attributes defined in the
+template.
+
+There are three types of constraints:
+
+- `location`
+- `colocation`
+- `order`
+
+It is possible to define fencing order (stonith resource
+priorities):
+
+- `fencing_topology`
+
+Finally, there are the cluster properties, resource meta
+attributes defaults, and operations defaults. All are just a set
+of attributes. These attributes are managed by the following
+commands:
+
+- `property`
+- `rsc_defaults`
+- `op_defaults`
+
+In addition to the cluster configuration, the Access Control
+Lists (ACL) can be setup to allow access to parts of the CIB for
+users other than `root` and `hacluster`. The following commands
+manage ACL:
+
+- `user`
+- `role`
+
+The changes are applied to the current CIB only on ending the
+configuration session or using the `commit` command.
+
+Comments start with `#` in the first line. The comments are tied
+to the element which follows. If the element moves, its comments
+will follow.
+
+[[cmdhelp_configure_node,define a cluster node]]
+==== `node`
+
+The node command describes a cluster node. Nodes in the CIB are
+commonly created automatically by the CRM. Hence, you should not
+need to deal with nodes unless you also want to define node
+attributes. Note that it is also possible to manage node
+attributes at the `node` level.
+
+Usage:
+...............
+ node <uname>[:<type>]
+ [attributes <param>=<value> [<param>=<value>...]]
+ [utilization <param>=<value> [<param>=<value>...]]
+
+ type :: normal | member | ping
+...............
+Example:
+...............
+ node node1
+ node big_node attributes memory=64
+...............
+
+[[cmdhelp_configure_primitive,define a resource]]
+==== `primitive`
+
+The primitive command describes a resource. It may be referenced
+only once in group, clone, or master-slave objects. If it's not
+referenced, then it is placed as a single resource in the CIB.
+
+Operations may be specified in three ways. "Anonymous" as a
+simple list of "op" specifications. Use that if you don't want to
+reference the set of operations elsewhere. That's by far the most
+common way to define operations. If reusing operation sets is
+desired, use the "operations" keyword along with the id to give
+the operations set a name and the id-ref to reference another set
+of operations.
+
+Operation's attributes which are not recognized are saved as
+instance attributes of that operation. A typical example is
+`OCF_CHECK_LEVEL`.
+
+For multistate resources, roles are specified as `role=<role>`.
+
+A template may be defined for resources which are of the same
+type and which share most of the configuration. See
+<<cmdhelp_configure_rsc_template,`rsc_template`>> for more information.
+
+Usage:
+...............
+ primitive <rsc> {[<class>:[<provider>:]]<type>|@<template>}
+ [params attr_list]
+ [meta attr_list]
+ [utilization attr_list]
+ [operations id_spec]
+ [op op_type [<attribute>=<value>...] ...]
+
+ attr_list :: [$id=<id>] <attr>=<val> [<attr>=<val>...] | $id-ref=<id>
+ id_spec :: $id=<id> | $id-ref=<id>
+ op_type :: start | stop | monitor
+...............
+Example:
+...............
+ primitive apcfence stonith:apcsmart \
+ params ttydev=/dev/ttyS0 hostlist="node1 node2" \
+ op start timeout=60s \
+ op monitor interval=30m timeout=60s
+
+ primitive www8 apache \
+ params configfile=/etc/apache/www8.conf \
+ operations $id-ref=apache_ops
+
+ primitive db0 mysql \
+ params config=/etc/mysql/db0.conf \
+ op monitor interval=60s \
+ op monitor interval=300s OCF_CHECK_LEVEL=10
+
+ primitive r0 ocf:linbit:drbd \
+ params drbd_resource=r0 \
+ op monitor role=Master interval=60s \
+ op monitor role=Slave interval=300s
+
+ primitive xen0 @vm_scheme1 \
+ params xmfile=/etc/xen/vm/xen0
+...............
+
+[[cmdhelp_configure_monitor,add monitor operation to a primitive]]
+==== `monitor`
+
+Monitor is by far the most common operation. It is possible to
+add it without editing the whole resource. Also, long primitive
+definitions may be a bit uncluttered. In order to make this
+command as concise as possible, less common operation attributes
+are not available. If you need them, then use the `op` part of
+the `primitive` command.
+
+Usage:
+...............
+ monitor <rsc>[:<role>] <interval>[:<timeout>]
+...............
+Example:
+...............
+ monitor apcfence 60m:60s
+...............
+
+Note that after executing the command, the monitor operation may
+be shown as part of the primitive definition.
+
+[[cmdhelp_configure_group,define a group]]
+==== `group`
+
+The `group` command creates a group of resources. This can be useful
+when resources depend on other resources and require that those
+resources start in order on the same node. A common use of resource
+groups is to ensure that a server and a virtual IP are located
+together, and that the virtual IP is started before the server.
+
+Grouped resources are started in the order they appear in the group,
+and stopped in the reverse order. If a resource in the group cannot
+run anywhere, resources following it in the group will not start.
+
+`group` can be passed the "container" meta attribute, to indicate that
+it is to be used to group VM resources monitored using Nagios. The
+resource referred to by the container attribute must be of type
+`ocf:heartbeat:Xen`, `ocf:heartbeat:VirtualDomain` or `ocf:heartbeat:lxc`.
+
+Usage:
+...............
+ group <name> <rsc> [<rsc>...]
+ [meta attr_list]
+ [params attr_list]
+
+ attr_list :: [$id=<id>] <attr>=<val> [<attr>=<val>...] | $id-ref=<id>
+...............
+Example:
+...............
+ group internal_www disk0 fs0 internal_ip apache \
+ meta target_role=stopped
+
+ group vm-and-services vm vm-sshd meta container="vm"
+...............
+
+[[cmdhelp_configure_clone,define a clone]]
+==== `clone`
+
+The `clone` command creates a resource clone. It may contain a
+single primitive resource or one group of resources.
+
+Usage:
+...............
+ clone <name> <rsc>
+ [meta attr_list]
+ [params attr_list]
+
+ attr_list :: [$id=<id>] <attr>=<val> [<attr>=<val>...] | $id-ref=<id>
+...............
+Example:
+...............
+ clone cl_fence apc_1 \
+ meta clone-node-max=1 globally-unique=false
+...............
+
+[[cmdhelp_configure_ms,define a master-slave resource]]
+==== `ms` (`master`)
+
+The `ms` command creates a master/slave resource type. It may contain a
+single primitive resource or one group of resources.
+
+Usage:
+...............
+ ms <name> <rsc>
+ [meta attr_list]
+ [params attr_list]
+
+ attr_list :: [$id=<id>] <attr>=<val> [<attr>=<val>...] | $id-ref=<id>
+...............
+Example:
+...............
+ ms disk1 drbd1 \
+ meta notify=true globally-unique=false
+...............
+
+.Note on `id-ref` usage
+****************************
+Instance or meta attributes (`params` and `meta`) may contain
+a reference to another set of attributes. In that case, no other
+attributes are allowed. Since attribute sets' ids, though they do
+exist, are not shown in the `crm`, it is also possible to
+reference an object instead of an attribute set. `crm` will
+automatically replace such a reference with the right id:
+
+...............
+ crm(live)configure# primitive a2 www-2 meta $id-ref=a1
+ crm(live)configure# show a2
+ primitive a2 ocf:heartbeat:apache \
+ meta $id-ref="a1-meta_attributes"
+ [...]
+...............
+It is advisable to give meaningful names to attribute sets which
+are going to be referenced.
+****************************
+
+[[cmdhelp_configure_rsc_template,define a resource template]]
+==== `rsc_template`
+
+The `rsc_template` command creates a resource template. It may be
+referenced in primitives. It is used to reduce large
+configurations with many similar resources.
+
+Usage:
+...............
+ rsc_template <name> [<class>:[<provider>:]]<type>
+ [params attr_list]
+ [meta attr_list]
+ [utilization attr_list]
+ [operations id_spec]
+ [op op_type [<attribute>=<value>...] ...]
+
+ attr_list :: [$id=<id>] <attr>=<val> [<attr>=<val>...] | $id-ref=<id>
+ id_spec :: $id=<id> | $id-ref=<id>
+ op_type :: start | stop | monitor
+...............
+Example:
+...............
+ rsc_template public_vm ocf:heartbeat:Xen \
+ op start timeout=300s \
+ op stop timeout=300s \
+ op monitor interval=30s timeout=60s \
+ op migrate_from timeout=600s \
+ op migrate_to timeout=600s
+ primitive xen0 @public_vm \
+ params xmfile=/etc/xen/xen0
+ primitive xen1 @public_vm \
+ params xmfile=/etc/xen/xen1
+...............
+
+[[cmdhelp_configure_location,a location preference]]
+==== `location`
+
+`location` defines the preference of nodes for the given
+resource. The location constraints consist of one or more rules
+which specify a score to be awarded if the rule matches.
+
+Usage:
+...............
+ location <id> <rsc> {node_pref|rules}
+
+ node_pref :: <score>: <node>
+
+ rules ::
+ rule [id_spec] [$role=<role>] <score>: <expression>
+ [rule [id_spec] [$role=<role>] <score>: <expression> ...]
+
+ id_spec :: $id=<id> | $id-ref=<id>
+ score :: <number> | <attribute> | [-]inf
+ expression :: <simple_exp> [bool_op <simple_exp> ...]
+ bool_op :: or | and
+ simple_exp :: <attribute> [type:]<binary_op> <value>
+ | <unary_op> <attribute>
+ | date <date_expr>
+ type :: string | version | number
+ binary_op :: lt | gt | lte | gte | eq | ne
+ unary_op :: defined | not_defined
+
+ date_expr :: lt <end>
+ | gt <start>
+ | in_range start=<start> end=<end>
+ | in_range start=<start> <duration>
+ | date_spec <date_spec>
+ duration|date_spec ::
+ hours=<value>
+ | monthdays=<value>
+ | weekdays=<value>
+ | yearsdays=<value>
+ | months=<value>
+ | weeks=<value>
+ | years=<value>
+ | weekyears=<value>
+ | moon=<value>
+...............
+Examples:
+...............
+ location conn_1 internal_www 100: node1
+
+ location conn_1 internal_www \
+ rule 50: #uname eq node1 \
+ rule pingd: defined pingd
+
+ location conn_2 dummy_float \
+ rule -inf: not_defined pingd or pingd number:lte 0
+...............
+
+[[cmdhelp_configure_colocation,colocate resources]]
+==== `colocation` (`collocation`)
+
+This constraint expresses the placement relation between two
+or more resources. If there are more than two resources, then the
+constraint is called a resource set.
+
+The score is used to indicate the priority of the constraint. A
+positive score indicates that the resources should run on the same
+node. A negative score that they should not run on the same
+node. Values of positive or negative `infinity` indicate a mandatory
+constraint.
+
+In the two resource form, the cluster will place `<with-rsc>` first,
+and then decide where to put the `<rsc>` resource.
+
+Collocation resource sets have an extra attribute (`sequential`)
+to allow for sets of resources which don't depend on each other
+in terms of state. The shell syntax for such sets is to put
+resources in parentheses.
+
+Sets cannot be nested.
+
+The optional `node-attribute` references an attribute in nodes'
+instance attributes.
+
+Usage:
+...............
+ colocation <id> <score>: <rsc>[:<role>] <with-rsc>[:<role>]
+ [node-attribute=<node_attr>]
+
+ colocation <id> <score>: <rsc>[:<role>] <rsc>[:<role>] ...
+ [node-attribute=<node_attr>]
+...............
+Example:
+...............
+ colocation never_put_apache_with_dummy -inf: apache dummy
+ colocation c1 inf: A ( B C )
+...............
+
+[[cmdhelp_configure_order,order resources]]
+==== `order`
+
+This constraint expresses the order of actions on two resources
+or more resources. If there are more than two resources, then the
+constraint is called a resource set.
+
+Ordered resource sets have an extra attribute to allow for sets
+of resources whose actions may run in parallel. The shell syntax
+for such sets is to put resources in parentheses.
+
+If the subsequent resource can start or promote after any one of the
+resources in a set has done, enclose the set in brackets (`[` and `]`).
+
+Sets cannot be nested.
+
+Three strings are reserved to specify a kind of order constraint:
+`Mandatory`, `Optional`, and `Serialize`. It is preferred to use
+one of these settings instead of score. Previous versions mapped
+scores `0` and `inf` to keywords `advisory` and `mandatory`.
+That is still valid but deprecated.
+
+.Note on resource sets' XML attributes
+****************************
+The XML attribute `require-all` controls whether all resources in
+a set are, well, required. The bracketed sets actually have this
+attribute as well as `sequential` set to `false`. If you need a
+different combination, for whatever reason, just set one of the
+attributes within the set. Something like this:
+
+...............
+ crm(live)configure# order o1 Mandatory: [ A B sequential=true ] C
+...............
+It is up to you to find out whether such a combination makes
+sense.
+****************************
+
+Usage:
+...............
+ order <id> {kind|<score>}: <rsc>[:<action>] <rsc>[:<action>] ...
+ [symmetrical=<bool>]
+
+ kind :: Mandatory | Optional | Serialize
+...............
+Example:
+...............
+ order c_apache_1 Mandatory: apache:start ip_1
+ order o1 Serialize: A ( B C )
+ order order_2 Mandatory: [ A B ] C
+...............
+
+[[cmdhelp_configure_rsc_ticket,resources ticket dependency]]
+==== `rsc_ticket`
+
+This constraint expresses dependency of resources on cluster-wide
+attributes, also known as tickets. Tickets are mainly used in
+geo-clusters, which consist of multiple sites. A ticket may be
+granted to a site, thus allowing resources to run there.
+
+The `loss-policy` attribute specifies what happens to the
+resource (or resources) if the ticket is revoked. The default is
+either `stop` or `demote` depending on whether a resource is
+multi-state.
+
+See also the <<cmdhelp_site_ticket,`site`>> set of commands.
+
+Usage:
+...............
+ rsc_ticket <id> <ticket_id>: <rsc>[:<role>] [<rsc>[:<role>] ...]
+ [loss-policy=<loss_policy_action>]
+
+ loss_policy_action :: stop | demote | fence | freeze
+...............
+Example:
+...............
+ rsc_ticket ticket-A_public-ip ticket-A: public-ip
+ rsc_ticket ticket-A_bigdb ticket-A: bigdb loss-policy=fence
+ rsc_ticket ticket-B_storage ticket-B: drbd-a:Master drbd-b:Master
+...............
+
+
+[[cmdhelp_configure_property,set a cluster property]]
+==== `property`
+
+Set the cluster (`crm_config`) options.
+
+Usage:
+...............
+ property [$id=<set_id>] <option>=<value> [<option>=<value> ...]
+...............
+Example:
+...............
+ property stonith-enabled=true
+...............
+
+[[cmdhelp_configure_rsc_defaults,set resource defaults]]
+==== `rsc_defaults`
+
+Set defaults for the resource meta attributes.
+
+Usage:
+...............
+ rsc_defaults [$id=<set_id>] <option>=<value> [<option>=<value> ...]
+...............
+Example:
+...............
+ rsc_defaults failure-timeout=3m
+...............
+
+[[cmdhelp_configure_fencing_topology,node fencing order]]
+==== `fencing_topology`
+
+If multiple fencing (stonith) devices are available capable of
+fencing a node, their order may be specified by `fencing_topology`.
+The order is specified per node.
+
+Stonith resources can be separated by `,` in which case all of
+them need to succeed. If they fail, the next stonith resource (or
+set of resources) is used. In other words, use comma to separate
+resources which all need to succeed and whitespace for serial
+order. It is not allowed to use whitespace around comma.
+
+If the node is left out, the order is used for all nodes.
+That should reduce the configuration size in some stonith setups.
+
+Usage:
+...............
+ fencing_topology stonith_resources [stonith_resources ...]
+ fencing_topology fencing_order [fencing_order ...]
+
+ fencing_order :: <node>: stonith_resources [stonith_resources ...]
+
+ stonith_resources :: <rsc>[,<rsc>...]
+...............
+Example:
+...............
+ fencing_topology poison-pill power
+ fencing_topology \
+ node-a: poison-pill power
+ node-b: ipmi serial
+...............
+
+[[cmdhelp_configure_role,define role access rights]]
+==== `role`
+
+An ACL role is a set of rules which describe access rights to
+CIB. Rules consist of an access right `read`, `write`, or `deny`
+and a specification denoting part of the configuration to which
+the access right applies. The specification can be an XPath or a
+combination of tag and id references. If an attribute is
+appended, then the specification applies only to that attribute
+of the matching element.
+
+There is a number of shortcuts for XPath specifications. The
+`meta,` `params`, and `utilization` shortcuts reference resource
+meta attributes, parameters, and utilization respectively. The
+`location` may be used to specify location constraints most of
+the time to allow resource `move` and `unmove` commands. The
+`property` references cluster properties. The `node` allows
+reading node attributes. `nodeattr` and `nodeutil` reference node
+attributes and node capacity (utilization). The `status` shortcut
+references the whole status section of the CIB. Read access to
+status is necessary for various monitoring tools such as
+`crm_mon(8)` (aka `crm status`).
+
+Usage:
+...............
+ role <role-id> rule [rule ...]
+
+ rule :: acl-right cib-spec [attribute:<attribute>]
+
+ acl-right :: read | write | deny
+
+ cib-spec :: xpath-spec | tag-ref-spec
+ xpath-spec :: xpath:<xpath> | shortcut
+ tag-ref-spec :: tag:<tag> | ref:<id> | tag:<tag> ref:<id>
+
+ shortcut :: meta:<rsc>[:<attr>]
+ params:<rsc>[:<attr>]
+ utilization:<rsc>
+ location:<rsc>
+ property[:<attr>]
+ node[:<node>]
+ nodeattr[:<attr>]
+ nodeutil[:<node>]
+ status
+...............
+Example:
+...............
+ role app1_admin \
+ write meta:app1:target-role \
+ write meta:app1:is-managed \
+ write location:app1 \
+ read ref:app1
+...............
+
+[[cmdhelp_configure_user,define user access rights]]
+==== `user`
+
+Users which normally cannot view or manage cluster configuration
+can be allowed access to parts of the CIB. The access is defined
+by a set of `read`, `write`, and `deny` rules as in role
+definitions or by referencing roles. The latter is considered
+best practice.
+
+Usage:
+...............
+ user <uid> {roles|rules}
+
+ roles :: role:<role-ref> [role:<role-ref> ...]
+ rules :: rule [rule ...]
+...............
+Example:
+...............
+ user joe \
+ role:app1_admin \
+ role:read_all
+...............
+
+[[cmdhelp_configure_op_defaults,set resource operations defaults]]
+==== `op_defaults`
+
+Set defaults for the operations meta attributes.
+
+Usage:
+...............
+ op_defaults [$id=<set_id>] <option>=<value> [<option>=<value> ...]
+...............
+Example:
+...............
+ op_defaults record-pending=true
+...............
+
+[[cmdhelp_configure_schema,set or display current CIB RNG schema]]
+==== `schema`
+
+CIB's content is validated by a RNG schema. Pacemaker supports
+several, depending on version. Currently supported schemas are
+`pacemaker-1.0`, `pacemaker-1.1`, and `pacemaker-1.2`.
+
+Use this command to display or switch to another RNG schema.
+
+Usage:
+...............
+ schema [<schema>]
+...............
+Example:
+...............
+ schema pacemaker-1.1
+...............
+
+[[cmdhelp_configure_show,display CIB objects]]
+==== `show`
+
+The `show` command displays objects. It may display all objects
+or a set of objects. The user may also choose to see only objects
+which were changed.
+Optionally, the XML code may be displayed instead of the CLI
+representation.
+
+Usage:
+...............
+ show [xml] [<id> ...]
+ show [xml] changed
+...............
+
+[[cmdhelp_configure_edit,edit CIB objects]]
+==== `edit`
+
+This command invokes the editor with the object description. As
+with the `show` command, the user may choose to edit all objects
+or a set of objects.
+
+If the user insists, he or she may edit the XML edition of the
+object. If you do that, don't modify any id attributes.
+
+Usage:
+...............
+ edit [xml] [<id> ...]
+ edit [xml] changed
+...............
+
+.Note on renaming element ids
+****************************
+The edit command sometimes cannot properly handle modifying
+element ids. In particular for elements which belong to group or
+ms resources. Group and ms resources themselves also cannot be
+renamed. Please use the `rename` command instead.
+****************************
+
+[[cmdhelp_configure_filter,filter CIB objects]]
+==== `filter`
+
+This command filters the given CIB elements through an external
+program. The program should accept input on `stdin` and send
+output to `stdout` (the standard UNIX filter conventions). As
+with the `show` command, the user may choose to filter all or
+just a subset of elements.
+
+It is possible to filter the XML representation of objects, but
+probably not as useful as the configuration language. The
+presentation is somewhat different from what would be displayed
+by the `show` command---each element is shown on a single line,
+i.e. there are no backslashes and no other embelishments.
+
+Don't forget to put quotes around the filter if it contains
+spaces.
+
+Usage:
+...............
+ filter <prog> [xml] [<id> ...]
+ filter <prog> [xml] changed
+...............
+Examples:
+...............
+ filter "sed '/^primitive/s/target-role=[^ ]*//'"
+ # crm configure filter "sed '/^primitive/s/target-role=[^ ]*//'"
+...............
+
+[[cmdhelp_configure_delete,delete CIB objects]]
+==== `delete`
+
+Delete one or more objects. If an object to be deleted belongs to
+a container object, such as a group, and it is the only resource
+in that container, then the container is deleted as well. Any
+related constraints are removed as well.
+
+Usage:
+...............
+ delete <id> [<id>...]
+...............
+
+[[cmdhelp_configure_default-timeouts,set timeouts for operations to minimums from the meta-data]]
+==== `default-timeouts`
+
+This command takes the timeouts from the actions section of the
+resource agent meta-data and sets them for the operations of the
+primitive.
+
+Usage:
+...............
+ default-timeouts <id> [<id>...]
+...............
+
+.Note on `default-timeouts`
+****************************
+You may be happy using this, but your applications may not. And
+it will tell you so at the worst possible moment. You have been
+warned.
+****************************
+
+[[cmdhelp_configure_rename,rename a CIB object]]
+==== `rename`
+
+Rename an object. It is recommended to use this command to rename
+a resource, because it will take care of updating all related
+constraints and a parent resource. Changing ids with the edit
+command won't have the same effect.
+
+If you want to rename a resource, it must be in the stopped state.
+
+Usage:
+...............
+ rename <old_id> <new_id>
+...............
+
+[[cmdhelp_configure_modgroup,modify group]]
+==== `modgroup`
+
+Add or remove primitives in a group. The `add` subcommand appends
+the new group member by default. Should it go elsewhere, there
+are `after` and `before` clauses.
+
+Usage:
+...............
+ modgroup <id> add <id> [after <id>|before <id>]
+ modgroup <id> remove <id>
+...............
+Examples:
+...............
+ modgroup share1 add storage2 before share1-fs
+...............
+
+[[cmdhelp_configure_refresh,refresh from CIB]]
+==== `refresh`
+
+Refresh the internal structures from the CIB. All changes made
+during this session are lost.
+
+Usage:
+...............
+ refresh
+...............
+
+[[cmdhelp_configure_erase,erase the CIB]]
+==== `erase`
+
+The `erase` clears all configuration. Apart from nodes. To remove
+nodes, you have to specify an additional keyword `nodes`.
+
+Note that removing nodes from the live cluster may have some
+strange/interesting/unwelcome effects.
+
+Usage:
+...............
+ erase [nodes]
+...............
+
+[[cmdhelp_configure_ptest,show cluster actions if changes were committed]]
+==== `ptest` (`simulate`)
+
+Show PE (Policy Engine) motions using `ptest(8)` or
+`crm_simulate(8)`.
+
+A CIB is constructed using the current user edited configuration
+and the status from the running CIB. The resulting CIB is run
+through `ptest` (or `crm_simulate`) to show changes which would
+happen if the configuration is committed.
+
+The status section may be loaded from another source and modified
+using the <<cmdhelp_cibstatus,`cibstatus`>> level commands. In that case, the
+`ptest` command will issue a message informing the user that the
+Policy Engine graph is not calculated based on the current status
+section and therefore won't show what would happen to the
+running but some imaginary cluster.
+
+If you have graphviz installed and X11 session, `dotty(1)` is run
+to display the changes graphically.
+
+Add a string of `v` characters to increase verbosity. `ptest`
+can also show allocation scores. `utilization` turns on
+information about the remaining capacity of nodes. With the
+`actions` option, `ptest` will print all resource actions.
+
+The `ptest` program has been replaced by `crm_simulate` in newer
+Pacemaker versions. In some installations both could be
+installed. Use `simulate` to enfore using `crm_simulate`.
+
+Usage:
+...............
+ ptest [nograph] [v...] [scores] [actions] [utilization]
+...............
+Examples:
+...............
+ ptest scores
+ ptest vvvvv
+ simulate actions
+...............
+
+[[cmdhelp_configure_rsctest,test resources as currently configured]]
+==== `rsctest`
+
+Test resources with current resource configuration. If no nodes
+are specified, tests are run on all known nodes.
+
+The order of resources is significant: it is assumed that later
+resources depend on earlier ones.
+
+If a resource is multi-state, it is assumed that the role on
+which later resources depend is master.
+
+Tests are run sequentially to prevent running the same resource
+on two or more nodes. Tests are carried out only if none of the
+specified nodes currently run any of the specified resources.
+However, it won't verify whether resources run on the other
+nodes.
+
+Superuser privileges are obviously required: either run this as
+root or setup the `sudoers` file appropriately.
+
+Note that resource testing may take some time.
+
+Usage:
+...............
+ rsctest <rsc_id> [<rsc_id> ...] [<node_id> ...]
+...............
+Examples:
+...............
+ rsctest my_ip websvc
+ rsctest websvc nodeB
+...............
+
+[[cmdhelp_configure_cib,CIB shadow management]]
+=== `cib` (shadow CIBs)
+
+This level is for management of shadow CIBs. It is available at
+the `configure` level to enable saving intermediate changes to a
+shadow CIB instead of to the live cluster. This short excerpt
+shows how:
+...............
+ crm(live)configure# cib new test-2
+ INFO: test-2 shadow CIB created
+ crm(test-2)configure# commit
+...............
+Note how the current CIB in the prompt changed from `live` to
+`test-2` after issuing the `cib new` command. See also the
+<<cmdhelp_cib,CIB shadow management>> for more information.
+
+[[cmdhelp_configure_cibstatus,CIB status management and editing]]
+==== `cibstatus`
+
+Enter edit and manage the CIB status section level. See the
+<<cmdhelp_cibstatus,CIB status management section>>.
+
+[[cmdhelp_configure_template,edit and import a configuration from a template]]
+==== `template`
+
+The specified template is loaded into the editor. It's up to the
+user to make a good CRM configuration out of it. See also the
+<<cmdhelp_template,template section>>.
+
+Usage:
+...............
+ template [xml] url
+...............
+Example:
+...............
+ template two-apaches.txt
+...............
+
+[[cmdhelp_configure_commit,commit the changes to the CIB]]
+==== `commit`
+
+Commit the current configuration to the CIB in use. As noted
+elsewhere, commands in a configure session don't have immediate
+effect on the CIB. All changes are applied at one point in time,
+either using `commit` or when the user leaves the configure
+level. In case the CIB in use changed in the meantime, presumably
+by somebody else, the crm shell will refuse to apply the changes.
+If you know that it's fine to still apply them add `force`.
+
+Usage:
+...............
+ commit [force]
+...............
+
+[[cmdhelp_configure_verify,verify the CIB with crm_verify]]
+==== `verify`
+
+Verify the contents of the CIB which would be committed.
+
+Usage:
+...............
+ verify
+...............
+
+[[cmdhelp_configure_upgrade,upgrade the CIB to version 1.0]]
+==== `upgrade`
+
+If you get the `CIB not supported` error, which typically means
+that the current CIB version is coming from the older release,
+you may try to upgrade it to the latest revision. The command
+to perform the upgrade is:
+...............
+ # cibadmin --upgrade --force
+...............
+
+If we don't recognize the current CIB as the old one, but you're
+sure that it is, you may force the command.
+
+Usage:
+...............
+ upgrade [force]
+...............
+
+[[cmdhelp_configure_save,save the CIB to a file]]
+==== `save`
+
+Save the current configuration to a file. Optionally, as XML. Use
+`-` instead of file name to write the output to `stdout`.
+
+Usage:
+...............
+ save [xml] <file>
+...............
+Example:
+...............
+ save myfirstcib.txt
+...............
+
+[[cmdhelp_configure_load,import the CIB from a file]]
+==== `load`
+
+Load a part of configuration (or all of it) from a local file or
+a network URL. The `replace` method replaces the current
+configuration with the one from the source. The `update` tries to
+import the contents into the current configuration.
+The file may be a CLI file or an XML file.
+
+Usage:
+...............
+ load [xml] <method> URL
+
+ method :: replace | update
+...............
+Example:
+...............
+ load xml update myfirstcib.xml
+ load xml replace http://storage.big.com/cibs/bigcib.xml
+...............
+
+[[cmdhelp_configure_graph,generate a directed graph]]
+==== `graph`
+
+Create a graphviz graphical layout from the current cluster
+configuration.
+
+Currently, only `dot` (directed graph) is supported. It is
+essentially a visualization of resource ordering.
+
+The graph may be saved to a file which can be used as source for
+various graphviz tools (by default it is displayed in the user's
+X11 session). Optionally, by specifying the format, one can also
+produce an image instead.
+
+For more or different graphviz attributes, it is possible to save
+the default set of attributes to an ini file. If this file exists
+it will always override the builtin settings. The `exportsettings`
+subcommand also prints the location of the ini file.
+
+Usage:
+...............
+ graph [<gtype> [<file> [<img_format>]]]
+ graph exportsettings
+
+ gtype :: dot
+ img_format :: `dot` output format (see the `-T` option)
+...............
+Example:
+...............
+ graph dot
+ graph dot clu1.conf.dot
+ graph dot clu1.conf.svg svg
+...............
+
+[[cmdhelp_configure_xml,raw xml]]
+==== `xml`
+
+Even though we promissed no xml, it may happen, but hopefully
+very very seldom, that an element from the CIB cannot be rendered
+in the configuration language. In that case, the element will be
+shown as raw xml, prefixed by this command. That element can then
+be edited like any other. If the shell finds out that after the
+change it can digest it, then it is going to be converted into
+the normal configuration language. Otherwise, there is no need to
+use `xml` for configuration.
+
+Usage:
+...............
+ xml <xml>
+...............
+
+[[cmdhelp_template,edit and import a configuration from a template]]
+=== `template`
+
+User may be assisted in the cluster configuration by templates
+prepared in advance. Templates consist of a typical ready
+configuration which may be edited to suit particular user needs.
+
+This command enters a template level where additional commands
+for configuration/template management are available.
+
+[[cmdhelp_template_new,create a new configuration from templates]]
+==== `new`
+
+Create a new configuration from one or more templates. Note that
+configurations and templates are kept in different places, so it
+is possible to have a configuration name equal a template name.
+
+If you already know which parameters are required, you can set
+them directly on the command line.
+
+The parameter name `id` is set by default to the name of the
+configuration.
+
+Usage:
+...............
+ new <config> <template> [<template> ...] [params name=value ...]"
+...............
+Examples:
+...............
+ new vip virtual-ip
+ new bigfs ocfs2 params device=/dev/sdx8 directory=/bigfs
+...............
+
+[[cmdhelp_template_load,load a configuration]]
+==== `load`
+
+Load an existing configuration. Further `edit`, `show`, and
+`apply` commands will refer to this configuration.
+
+Usage:
+...............
+ load <config>
+...............
+
+[[cmdhelp_template_edit,edit a configuration]]
+==== `edit`
+
+Edit current or given configuration using your favourite editor.
+
+Usage:
+...............
+ edit [<config>]
+...............
+
+[[cmdhelp_template_delete,delete a configuration]]
+==== `delete`
+
+Remove a configuration. The loaded (active) configuration may be
+removed by force.
+
+Usage:
+...............
+ delete <config> [force]
+...............
+
+[[cmdhelp_template_list,list configurations/templates]]
+==== `list`
+
+List existing configurations or templates.
+
+Usage:
+...............
+ list [templates]
+...............
+
+[[cmdhelp_template_apply,process and apply the current configuration to the current CIB]]
+==== `apply`
+
+Copy the current or given configuration to the current CIB. By
+default, the CIB is replaced, unless the method is set to
+"update".
+
+Usage:
+...............
+ apply [<method>] [<config>]
+
+ method :: replace | update
+...............
+
+[[cmdhelp_template_show,show the processed configuration]]
+==== `show`
+
+Process the current or given configuration and display the result.
+
+Usage:
+...............
+ show [<config>]
+...............
+
+[[cmdhelp_cibstatus,CIB status management and editing]]
+=== `cibstatus`
+
+The `status` section of the CIB keeps the current status of nodes
+and resources. It is modified _only_ on events, i.e. when some
+resource operation is run or node status changes. For obvious
+reasons, the CRM has no user interface with which it is possible
+to affect the status section. From the user's point of view, the
+status section is essentially a read-only part of the CIB. The
+current status is never even written to disk, though it is
+available in the PE (Policy Engine) input files which represent
+the history of cluster motions. The current status may be read
+using the `cibadmin -Q` command.
+
+It may sometimes be of interest to see how status changes would
+affect the Policy Engine. The set of `cibstatus` level commands
+allow the user to load status sections from various sources and
+then insert or modify resource operations or change nodes' state.
+
+The effect of those changes may then be observed by running the
+<<cmdhelp_configure_ptest,`ptest`>> command at the `configure` level
+or `simulate` and `run` commands at this level. The `ptest`
+runs with the user edited CIB whereas the latter two commands
+run with the CIB which was loaded along with the status section.
+
+The `simulate` and `run` commands as well as all status
+modification commands are implemented using `crm_simulate(8)`.
+
+[[cmdhelp_cibstatus_load,load the CIB status section]]
+==== `load`
+
+Load a status section from a file, a shadow CIB, or the running
+cluster. By default, the current (`live`) status section is
+modified. Note that if the `live` status section is modified it
+is not going to be updated if the cluster status changes, because
+that would overwrite the user changes. To make `crm` drop changes
+and resume use of the running cluster status, run `load live`.
+
+All CIB shadow configurations contain the status section which is
+a snapshot of the status section taken at the time the shadow was
+created. Obviously, this status section doesn't have much to do
+with the running cluster status, unless the shadow CIB has just
+been created. Therefore, the `ptest` command by default uses the
+running cluster status section.
+
+Usage:
+...............
+ load {<file>|shadow:<cib>|live}
+...............
+Example:
+...............
+ load bug-12299.xml
+ load shadow:test1
+...............
+
+[[cmdhelp_cibstatus_save,save the CIB status section]]
+==== `save`
+
+The current internal status section with whatever modifications
+were performed can be saved to a file or shadow CIB.
+
+If the file exists and contains a complete CIB, only the status
+section is going to be replaced and the rest of the CIB will
+remain intact. Otherwise, the current user edited configuration
+is saved along with the status section.
+
+Note that all modifications are saved in the source file as soon
+as they are run.
+
+Usage:
+...............
+ save [<file>|shadow:<cib>]
+...............
+Example:
+...............
+ save bug-12299.xml
+...............
+
+[[cmdhelp_cibstatus_origin,display origin of the CIB status section]]
+==== `origin`
+
+Show the origin of the status section currently in use. This
+essentially shows the latest `load` argument.
+
+Usage:
+...............
+ origin
+...............
+
+[[cmdhelp_cibstatus_show,show CIB status section]]
+==== `show`
+
+Show the current status section in the XML format. Brace yourself
+for some unreadable output. Add `changed` option to get a human
+readable output of all changes.
+
+Usage:
+...............
+ show [changed]
+...............
+
+[[cmdhelp_cibstatus_node,change node status]]
+==== `node`
+
+Change the node status. It is possible to throw a node out of
+the cluster, make it a member, or set its state to unclean.
+
+`online`:: Set the `node_state` `crmd` attribute to `online`
+and the `expected` and `join` attributes to `member`. The effect
+is that the node becomes a cluster member.
+
+`offline`:: Set the `node_state` `crmd` attribute to `offline`
+and the `expected` attribute to empty. This makes the node
+cleanly removed from the cluster.
+
+`unclean`:: Set the `node_state` `crmd` attribute to `offline`
+and the `expected` attribute to `member`. In this case the node
+has unexpectedly disappeared.
+
+Usage:
+...............
+ node <node> {online|offline|unclean}
+...............
+Example:
+...............
+ node xen-b unclean
+...............
+
+[[cmdhelp_cibstatus_op,edit outcome of a resource operation]]
+==== `op`
+
+Edit the outcome of a resource operation. This way you can
+tell CRM that it ran an operation and that the resource agent
+returned certain exit code. It is also possible to change the
+operation's status. In case the operation status is set to
+something other than `done`, the exit code is effectively
+ignored.
+
+Usage:
+...............
+ op <operation> <resource> <exit_code> [<op_status>] [<node>]
+
+ operation :: probe | monitor[:<n>] | start | stop |
+ promote | demote | notify | migrate_to | migrate_from
+ exit_code :: <rc> | success | generic | args |
+ unimplemented | perm | installed | configured | not_running |
+ master | failed_master
+ op_status :: pending | done | cancelled | timeout | notsupported | error
+
+ n :: the monitor interval in seconds; if omitted, the first
+ recurring operation is referenced
+ rc :: numeric exit code in range 0..9
+...............
+Example:
+...............
+ op start d1 xen-b generic
+ op start d1 xen-b 1
+ op monitor d1 xen-b not_running
+ op stop d1 xen-b 0 timeout
+...............
+
+[[cmdhelp_cibstatus_quorum,set the quorum]]
+==== `quorum`
+
+Set the quorum value.
+
+Usage:
+...............
+ quorum <bool>
+...............
+Example:
+...............
+ quorum false
+...............
+
+[[cmdhelp_cibstatus_ticket,manage tickets]]
+==== `ticket`
+
+Modify the ticket status. Tickets can be granted and revoked.
+Granted tickets could be activated or put in standby.
+
+Usage:
+...............
+ ticket <ticket> {grant|revoke|activate|standby}
+...............
+Example:
+...............
+ ticket ticketA grant
+...............
+
+[[cmdhelp_cibstatus_run,run policy engine]]
+==== `run`
+
+Run the policy engine with the edited status section.
+
+Add a string of `v` characters to increase verbosity. Specify
+`scores` to see allocation scores also. `utilization` turns on
+information about the remaining capacity of nodes.
+
+If you have graphviz installed and X11 session, `dotty(1)` is run
+to display the changes graphically.
+
+Usage:
+...............
+ run [nograph] [v...] [scores] [utilization]
+...............
+Example:
+...............
+ run
+...............
+
+[[cmdhelp_cibstatus_simulate,simulate cluster transition]]
+==== `simulate`
+
+Run the policy engine with the edited status section and simulate
+the transition.
+
+Add a string of `v` characters to increase verbosity. Specify
+`scores` to see allocation scores also. `utilization` turns on
+information about the remaining capacity of nodes.
+
+If you have graphviz installed and X11 session, `dotty(1)` is run
+to display the changes graphically.
+
+Usage:
+...............
+ simulate [nograph] [v...] [scores] [utilization]
+...............
+Example:
+...............
+ simulate
+...............
+
+[[cmdhelp_history,cluster history]]
+=== `history`
+
+Examining Pacemaker's history is a particularly involved task.
+The number of subsystems to be considered, the complexity of the
+configuration, and the set of various information sources, most
+of which are not exactly human readable, keep analyzing resource
+or node problems accessible to only the most knowledgeable. Or,
+depending on the point of view, to the most persistent. The
+following set of commands has been devised in hope to make
+cluster history more accessible.
+
+Of course, looking at _all_ history could be time consuming
+regardless of how good tools at hand are. Therefore, one should
+first say which period he or she wants to analyze. If not
+otherwise specified, the last hour is considered. Logs and other
+relevant information is collected using `hb_report`. Since this
+process takes some time and we always need fresh logs,
+information is refreshed in a much faster way using `pssh(1)`. If
+`python-pssh` is not found on the system, examining live cluster
+is still possible though not as comfortable.
+
+Apart from examining live cluster, events may be retrieved from a
+report generated by `hb_report` (see also the `-H` option). In
+that case we assume that the period stretching the whole report
+needs to be investigated. Of course, it is still possible to
+further reduce the time range.
+
+If you think you may have found a bug or just need clarification
+from developers or your support, the `session pack` command can
+help create a report. This is an example:
+...............
+ crm(live)history# limit "Jul 18 12:00" "Jul 18 12:30"
+ crm(live)history# session save strange_restart
+ crm(live)history# session pack
+ Report saved in .../strange_restart.tar.bz2
+ crm(live)history#
+...............
+In order to reduce report size and allow developers to
+concentrate on the issue, you should beforehand limit the time
+frame. Giving a meaningful session name helps too.
+
+==== `info`
+
+The `info` command shows most important information about the
+cluster.
+
+Usage:
+...............
+ info
+...............
+Example:
+...............
+ info
+...............
+
+[[cmdhelp_history_latest,show latest news from the cluster]]
+==== `latest`
+
+The `latest` command shows a bit of recent history, more
+precisely whatever happened since the last cluster change (the
+latest transition). If the transition is running, the shell will
+first wait until it finishes.
+
+Usage:
+...............
+ latest
+...............
+Example:
+...............
+ latest
+...............
+
+[[cmdhelp_history_limit,limit timeframe to be examined]]
+==== `limit` (`timeframe`)
+
+All history commands look at events within certain period. It
+defaults to the last hour for the live cluster source. There is
+no limit for the `hb_report` source. Use this command to set the
+timeframe.
+
+The time period is parsed by the dateutil python module. It
+covers wide range of date formats. For instance:
+
+- 3:00 (today at 3am)
+- 15:00 (today at 3pm)
+- 2010/9/1 2pm (September 1st 2010 at 2pm)
+
+We won't bother to give definition of the time specification in
+usage below. Either use common sense or read the
+http://labix.org/python-dateutil[dateutil] documentation.
+
+If dateutil is not available, then the time is parsed using
+strptime and only the kind as printed by `date(1)` is allowed:
+
+- Tue Sep 15 20:46:27 CEST 2010
+
+Usage:
+...............
+ limit [<from_time> [<to_time>]]
+...............
+Examples:
+...............
+ limit 10:15
+ limit 15h22m 16h
+ limit "Sun 5 20:46" "Sun 5 22:00"
+...............
+
+[[cmdhelp_history_source,set source to be examined]]
+==== `source`
+
+Events to be examined can come from the current cluster or from a
+`hb_report` report. This command sets the source. `source live`
+sets source to the running cluster and system logs. If no source
+is specified, the current source information is printed.
+
+In case a report source is specified as a file reference, the file
+is going to be unpacked in place where it resides. This directory
+is not removed on exit.
+
+Usage:
+...............
+ source [<dir>|<file>|live]
+...............
+Examples:
+...............
+ source live
+ source /tmp/customer_case_22.tar.bz2
+ source /tmp/customer_case_22
+ source
+...............
+
+[[cmdhelp_history_refresh,refresh live report]]
+==== `refresh`
+
+This command makes sense only for the `live` source and makes
+`crm` collect the latest logs and other relevant information from
+the logs. If you want to make a completely new report, specify
+`force`.
+
+Usage:
+...............
+ refresh [force]
+...............
+
+[[cmdhelp_history_detail,set the level of detail shown]]
+==== `detail`
+
+How much detail to show from the logs.
+
+Usage:
+...............
+ detail <detail_level>
+
+ detail_level :: small integer (defaults to 0)
+...............
+Example:
+...............
+ detail 1
+...............
+
+[[cmdhelp_history_setnodes,set the list of cluster nodes]]
+==== `setnodes`
+
+In case the host this program runs on is not part of the cluster,
+it is necessary to set the list of nodes.
+
+Usage:
+...............
+ setnodes node <node> [<node> ...]
+...............
+Example:
+...............
+ setnodes node_a node_b
+...............
+
+[[cmdhelp_history_resource,resource events]]
+==== `resource`
+
+Show actions and any failures that happened on all specified
+resources on all nodes. Normally, one gives resource names as
+arguments, but it is also possible to use extended regular
+expressions. Note that neither groups nor clones or master/slave
+names are ever logged. The resource command is going to expand
+all of these appropriately, so that clone instances or resources
+which are part of a group are shown.
+
+Usage:
+...............
+ resource <rsc> [<rsc> ...]
+...............
+Example:
+...............
+ resource bigdb public_ip
+ resource my_.*_db2
+ resource ping_clone
+...............
+
+[[cmdhelp_history_node,node events]]
+==== `node`
+
+Show important events that happened on a node. Important events
+are node lost and join, standby and online, and fence. Use either
+node names or extended regular expressions.
+
+Usage:
+...............
+ node <node> [<node> ...]
+...............
+Example:
+...............
+ node node1
+...............
+
+[[cmdhelp_history_log,log content]]
+==== `log`
+
+Show messages logged on one or more nodes. Leaving out a node
+name produces combined logs of all nodes. Messages are sorted by
+time and, if the terminal emulations supports it, displayed in
+different colours depending on the node to allow for easier
+reading.
+
+The sorting key is the timestamp as written by syslog which
+normally has the maximum resolution of one second. Obviously,
+messages generated by events which share the same timestamp may
+not be sorted in the same way as they happened. Such close events
+may actually happen fairly often.
+
+Usage:
+...............
+ log [<node>]
+...............
+Example:
+...............
+ log node-a
+...............
+
+[[cmdhelp_history_exclude,exclude log messages]]
+==== `exclude`
+
+If a log is infested with irrelevant messages, those messages may
+be excluded by specifying a regular expression. The regular
+expressions used are Python extended. This command is additive.
+To drop all regular expressions, use `exclude clear`. Run
+`exclude` only to see the current list of regular expressions.
+Excludes are saved along with the history sessions.
+
+Usage:
+...............
+ exclude [<regex>|clear]
+...............
+Example:
+...............
+ exclude kernel.*ocfs2
+...............
+
+[[cmdhelp_history_peinputs,list or get PE input files]]
+==== `peinputs`
+
+Every event in the cluster results in generating one or more
+Policy Engine (PE) files. These files describe future motions of
+resources. The files are listed as full paths in the current
+report directory. Add `v` to also see the creation time stamps.
+
+Usage:
+...............
+ peinputs [{<range>|<number>} ...] [v]
+
+ range :: <n1>:<n2>
+...............
+Example:
+...............
+ peinputs
+ peinputs 440:444 446
+ peinputs v
+...............
+
+[[cmdhelp_history_transition,show transition]]
+==== `transition`
+
+This command will print actions planned by the PE and run
+graphviz (`dotty`) to display a graphical representation of the
+transition. Of course, for the latter an X11 session is required.
+This command invokes `ptest(8)` in background.
+
+The `showdot` subcommand runs graphviz (`dotty`) to display a
+graphical representation of the `.dot` file which has been
+included in the report. Essentially, it shows the calculation
+produced by `pengine` which is installed on the node where the
+report was produced. In optimal case this output should not
+differ from the one produced by the locally installed `pengine`.
+
+The `log` subcommand shows the full log for the duration of the
+transition.
+
+A transition can also be saved to a CIB shadow for further
+analysis or use with `cib` or `configure` commands (use the
+`save` subcommand). The shadow file name defaults to the name of
+the PE input file.
+
+If the PE input file number is not provided, it defaults to the
+last one, i.e. the last transition. The last transition can also
+be referenced with number 0. If the number is negative, then the
+corresponding transition relative to the last one is chosen.
+
+If there are warning and error PE input files or different nodes
+were the DC in the observed timeframe, it may happen that PE
+input file numbers collide. In that case provide some unique part
+of the path to the file.
+
+After the `ptest` output, logs about events that happened during
+the transition are printed.
+
+Usage:
+...............
+ transition [<number>|<index>|<file>] [nograph] [v...] [scores] [actions] [utilization]
+ transition showdot [<number>|<index>|<file>]
+ transition log [<number>|<index>|<file>]
+ transition save [<number>|<index>|<file> [name]]
+...............
+Examples:
+...............
+ transition
+ transition 444
+ transition -1
+ transition pe-error-3.bz2
+ transition node-a/pengine/pe-input-2.bz2
+ transition showdot 444
+ transition log
+ transition save 0 enigma-22
+...............
+
+[[cmdhelp_history_show,show status or configuration of the PE input file]]
+==== `show`
+
+Every transition is saved as a PE file. Use this command to
+render that PE file either as configuration or status. The
+configuration output is the same as `crm configure show`.
+
+Usage:
+...............
+ show <pe> [status]
+
+ pe :: <number>|<index>|<file>|live
+...............
+Examples:
+...............
+ show 2066
+ show pe-input-2080.bz2 status
+...............
+
+[[cmdhelp_history_graph,generate a directed graph from the PE file]]
+==== `graph`
+
+Create a graphviz graphical layout from the PE file (the
+transition). Every transition contains the cluster configuration
+which was active at the time. See also <<cmdhelp_configure_graph,generate a directed graph
+from configuration>>.
+
+Usage:
+...............
+ graph <pe> [<gtype> [<file> [<img_format>]]]
+
+ gtype :: dot
+ img_format :: `dot` output format (see the `-T` option)
+...............
+Example:
+...............
+ graph -1
+ graph 322 dot clu1.conf.dot
+ graph 322 dot clu1.conf.svg svg
+...............
+
+[[cmdhelp_history_diff,cluster states/transitions difference]]
+==== `diff`
+
+A transition represents a change in cluster configuration or
+state. Use `diff` to see what has changed between two
+transitions.
+
+If you want to specify the current cluster configuration and
+status, use the string `live`.
+
+Normally, the first transition specified should be the one which
+is older, but we are not going to enforce that.
+
+Note that a single configuration update may result in more than
+one transition.
+
+Usage:
+...............
+ diff <pe> <pe> [status] [html]
+
+ pe :: <number>|<index>|<file>|live
+...............
+Examples:
+...............
+ diff 2066 2067
+ diff pe-input-2080.bz2 live status
+...............
+
+[[cmdhelp_history_session,manage history sessions]]
+==== `session`
+
+Sometimes you may want to get back to examining a particular
+history period or bug report. In order to make that easier, the
+current settings can be saved and later retrieved.
+
+If the current history being examined is coming from a live
+cluster the logs, PE inputs, and other files are saved too,
+because they may disappear from nodes. For the existing reports
+coming from `hb_report`, only the directory location is saved
+(not to waste space).
+
+A history session may also be packed into a tarball which can
+then be sent to support.
+
+Leave out subcommand to see the current session.
+
+Usage:
+...............
+ session [{save|load|delete} <name> | pack [<name>] | update | list]
+...............
+Examples:
+...............
+ session save bnc966622
+ session load rsclost-2
+ session list
+...............
+
+=== `end` (`cd`, `up`)
+
+The `end` command ends the current level and the user moves to
+the parent level. This command is available everywhere.
+
+Usage:
+...............
+ end
+...............
+
+=== `help`
+
+The `help` command prints help for the current level or for the
+specified topic (command). This command is available everywhere.
+
+Usage:
+...............
+ help [<topic>]
+...............
+
+=== `quit` (`exit`, `bye`)
+
+Leave the program.
+
+BUGS
+----
+Even though all sensible configurations (and most of those that
+are not) are going to be supported by the crm shell, I suspect
+that it may still happen that certain XML constructs may confuse
+the tool. When that happens, please file a bug report.
+
+The crm shell will not try to update the objects it does not
+understand. Of course, it is always possible to edit such objects
+in the XML format.
+
+AUTHOR
+------
+Dejan Muhamedagic, <dejan@suse.de>
+and many OTHERS
+
+SEE ALSO
+--------
+crm_resource(8), crm_attribute(8), crm_mon(8), cib_shadow(8),
+ptest(8), dotty(1), crm_simulate(8), cibadmin(8)
+
+
+COPYING
+-------
+Copyright \(C) 2008-2011 Dejan Muhamedagic. Free use of this
+software is granted under the terms of the GNU General Public License (GPL).
+
+//////////////////////
+ vim:ts=4:sw=4:expandtab:
+//////////////////////
diff --git a/doc/website-v1/man-2.0.adoc b/doc/website-v1/man-2.0.adoc
new file mode 100644
index 0000000..a2127d4
--- /dev/null
+++ b/doc/website-v1/man-2.0.adoc
@@ -0,0 +1,5048 @@
+:man source: crm
+:man version: 2.3.2
+:man manual: crmsh documentation
+
+crm(8)
+======
+
+NAME
+----
+crm - Pacemaker command line interface for configuration and management
+
+
+SYNOPSIS
+--------
+*crm* [OPTIONS] [SUBCOMMAND ARGS...]
+
+
+[[topics_Description,Program description]]
+DESCRIPTION
+-----------
+The `crm` shell is a command-line based cluster configuration and
+management tool. Its goal is to assist as much as possible with the
+configuration and maintenance of Pacemaker-based High Availability
+clusters.
+
+For more information on Pacemaker itself, see http://clusterlabs.org/.
+
+`crm` works both as a command-line tool to be called directly from the
+system shell, and as an interactive shell with extensive tab
+completion and help.
+
+The primary focus of the `crm` shell is to provide a simplified and
+consistent interface to Pacemaker, but it also provides tools for
+managing the creation and configuration of High Availability clusters
+from scratch. To learn more about this aspect of `crm`, see the
+`cluster` section below.
+
+The `crm` shell can be used to manage every aspect of configuring and
+maintaining a cluster. It provides a simplified line-based syntax on
+top of the XML configuration format used by Pacemaker, commands for
+starting and stopping resources, tools for exploring the history of a
+cluster including log scraping and a set of cluster scripts useful for
+automating the setup and installation of services on the cluster
+nodes.
+
+The `crm` shell is line oriented: every command must start and finish
+on the same line. It is possible to use a continuation character (+\+)
+to write one command in two or more lines. The continuation character
+is commonly used when displaying configurations.
+
+[[topics_CommandLine,Command line options]]
+OPTIONS
+-------
+*-f, --file*='FILE'::
+ Load commands from the given file. If a dash +-+ is used in place
+ of a file name, `crm` will read commands from the shell standard
+ input (`stdin`).
+
+*-c, --cib*='CIB'::
+ Start the session using the given shadow CIB file.
+ Equivalent to +cib use <CIB>+.
+
+*-D, --display=*'OUTPUT_TYPE'::
+ Choose one of the output options: +plain+, +color-always+, +color+,
+ or +uppercase+. The default is +color+ if the terminal emulation
+ supports colors. Otherwise, +plain+ is used.
+
+*-F, --force*::
+ Make `crm` proceed with applying changes where it would normally
+ ask the user to confirm before proceeding. This option is mainly
+ useful in scripts, and should be used with care.
+
+*-w, --wait*::
+ Make `crm` wait for the cluster transition to finish (for the
+ changes to take effect) after each processed line.
+
+*-H, --history*='DIR|FILE|SESSION'::
+ A directory or file containing a cluster report to load
+ into the `history` commands, or the name of a previously
+ saved history session.
+
+*-h, --help*::
+ Print help page.
+
+*--version*::
+ Print crmsh version and build information (Mercurial Hg changeset
+ hash).
+
+*-d, --debug*::
+ Print verbose debugging information.
+
+*-R, --regression-tests*::
+ Enables extra verbose trace logging used by the regression
+ tests. Logs all external calls made by crmsh.
+
+*--scriptdir*='DIR'::
+ Extra directory where crm looks for cluster scripts, or a list of
+ directories separated by semi-colons (e.g. +/dir1;/dir2;etc.+).
+
+*-o, --opt*='OPTION=VALUE'::
+ Set crmsh option temporarily. If the options are saved using
+ +options save+ then the value passed here will also be saved.
+ Multiple options can be set by using +-o+ multiple times.
+
+[[topics_Introduction,Introduction]]
+== Introduction
+
+This section of the user guide covers general topics about the user
+interface and describes some of the features of `crmsh` in detail.
+
+[[topics_Introduction_Interface,User interface]]
+=== User interface
+
+The main purpose of `crmsh` is to provide a simple yet powerful
+interface to the cluster stack. There are two main modes of operation
+with the user interface of `crmsh`:
+
+* Command line (single-shot) use - Use `crm` as a regular UNIX command
+ from your usual shell. `crm` has full bash completion built in, so
+ using it in this manner should be as comfortable and familiar as
+ using any other command-line tool.
+
+* Interactive mode - By calling `crm` without arguments, or by calling
+ it with only a sublevel as argument, `crm` enters the interactive
+ mode. In this mode, it acts as its own command shell, which
+ remembers which sublevel you are currently in and allows for rapid
+ and convenient execution of multiple commands within the same
+ sublevel. This mode also has full tab completion, as well as
+ built-in interactive help and syntax highlighting.
+
+Here are a few examples of using `crm` both as a command-line tool and
+as an interactive shell:
+
+.Command line (one-shot) use:
+........
+# crm resource stop www_app
+........
+
+.Interactive use:
+........
+# crm
+crm(live)# resource
+crm(live)resource# unmanage tetris_1
+crm(live)resource# up
+crm(live)# node standby node4
+........
+
+.Cluster configuration:
+........
+# crm configure<<EOF
+ #
+ # resources
+ #
+ primitive disk0 iscsi \
+ params portal=192.168.2.108:3260 target=iqn.2008-07.com.suse:disk0
+ primitive fs0 Filesystem \
+ params device=/dev/disk/by-label/disk0 directory=/disk0 fstype=ext3
+ primitive internal_ip IPaddr params ip=192.168.1.101
+ primitive apache apache \
+ params configfile=/disk0/etc/apache2/site0.conf
+ primitive apcfence stonith:apcsmart \
+ params ttydev=/dev/ttyS0 hostlist="node1 node2" \
+ op start timeout=60s
+ primitive pingd pingd \
+ params name=pingd dampen=5s multiplier=100 host_list="r1 r2"
+ #
+ # monitor apache and the UPS
+ #
+ monitor apache 60s:30s
+ monitor apcfence 120m:60s
+ #
+ # cluster layout
+ #
+ group internal_www \
+ disk0 fs0 internal_ip apache
+ clone fence apcfence \
+ meta globally-unique=false clone-max=2 clone-node-max=1
+ clone conn pingd \
+ meta globally-unique=false clone-max=2 clone-node-max=1
+ location node_pref internal_www \
+ rule 50: #uname eq node1 \
+ rule pingd: defined pingd
+ #
+ # cluster properties
+ #
+ property stonith-enabled=true
+ commit
+EOF
+........
+
+The `crm` interface is hierarchical, with commands organized into
+separate levels by functionality. To list the available levels and
+commands, either execute +help <level>+, or, if at the top level of
+the shell, simply typing `help` will provide an overview of all
+available levels and commands.
+
+The +(live)+ string in the `crm` prompt signifies that the current CIB
+in use is the cluster live configuration. It is also possible to
+work with so-called <<topics_Features_Shadows,shadow CIBs>>. These are separate, inactive
+configurations stored in files, that can be applied and thereby
+replace the live configuration at any time.
+
+[[topics_Introduction_Completion,Tab completion]]
+=== Tab completion
+
+The `crm` makes extensive use of tab completion. The completion
+is both static (i.e. for `crm` commands) and dynamic. The latter
+takes into account the current status of the cluster or
+information from installed resource agents. Sometimes, completion
+may also be used to get short help on resource parameters. Here
+are a few examples:
+
+...............
+crm(live)resource# <TAB><TAB>
+bye failcount move restart unmigrate
+cd help param show unmove
+cleanup list promote start up
+demote manage quit status utilization
+end meta refresh stop
+exit migrate reprobe unmanage
+
+crm(live)configure# primitive fence-1 <TAB><TAB>
+heartbeat: lsb: ocf: stonith:
+
+crm(live)configure# primitive fence-1 stonith:<TAB><TAB>
+apcmaster external/ippower9258 fence_legacy
+apcmastersnmp external/kdumpcheck ibmhmc
+apcsmart external/libvirt ipmilan
+
+crm(live)configure# primitive fence-1 stonith:ipmilan params <TAB><TAB>
+auth= hostname= ipaddr= login= password= port= priv=
+
+crm(live)configure# primitive fence-1 stonith:ipmilan params auth=<TAB><TAB>
+auth* (string)
+ The authorization type of the IPMI session ("none", "straight", "md2", or "md5")
+...............
+
+`crmsh` also comes with bash completion usable directly from the
+system shell. This should be installed automatically with the command
+itself.
+
+[[topics_Introduction_Shorthand,Shorthand syntax]]
+=== Shorthand syntax
+
+When using the `crm` shell to manage clusters, you will end up typing
+a lot of commands many times over. Clear command names like
++configure+ help in understanding and learning to use the cluster
+shell, but is easy to misspell and is tedious to type repeatedly. The
+interactive mode and tab completion both help with this, but the `crm`
+shell also has the ability to understand a variety of shorthand
+aliases for all of the commands.
+
+For example, instead of typing `crm status`, you can type `crm st` or
+`crm stat`. Instead of `crm configure` you can type `crm cfg` or even
+`crm cf`. `crm resource` can be shorted as `crm rsc`, and so on.
+
+The exact list of accepted aliases is too long to print in full, but
+experimentation and typos should help in discovering more of them.
+
+[[topics_Features,Features]]
+== Features
+
+The feature set of crmsh covers a wide range of functionality, and
+understanding how and when to use the various features of the shell
+can be difficult. This section of the guide describes some of the
+features and use cases of `crmsh` in more depth. The intention is to
+provide a deeper understanding of these features, but also to serve as
+a guide to using them.
+
+[[topics_Features_Shadows,Shadow CIB usage]]
+=== Shadow CIB usage
+
+A Shadow CIB is a normal cluster configuration stored in a file.
+They may be manipulated in much the same way as the _live_ CIB, with
+the key difference that changes to a shadow CIB have no effect on the
+actual cluster resources. An administrator may choose to apply any of
+them to the cluster, thus replacing the running configuration with the
+one found in the shadow CIB.
+
+The `crm` prompt always contains the name of the configuration which
+is currently in use, or the string _live_ if using the live cluster
+configuration.
+
+When editing the configuration in the `configure` level, no changes
+are actually applied until the `commit` command is executed. It is
+possible to start editing a configuration as usual, but instead of
+committing the changes to the active CIB, save them to a shadow CIB.
+
+The following example `configure` session demonstrates how this can be
+done:
+...............
+crm(live)configure# cib new test-2
+INFO: test-2 shadow CIB created
+crm(test-2)configure# commit
+...............
+
+[[topics_Features_Checks,Configuration semantic checks]]
+=== Configuration semantic checks
+
+Resource definitions may be checked against the meta-data
+provided with the resource agents. These checks are currently
+carried out:
+
+- are required parameters set
+- existence of defined parameters
+- timeout values for operations
+
+The parameter checks are obvious and need no further explanation.
+Failures in these checks are treated as configuration errors.
+
+The timeouts for operations should be at least as long as those
+recommended in the meta-data. Too short timeout values are a
+common mistake in cluster configurations and, even worse, they
+often slip through if cluster testing was not thorough. Though
+operation timeouts issues are treated as warnings, make sure that
+the timeouts are usable in your environment. Note also that the
+values given are just _advisory minimum_---your resources may
+require longer timeouts.
+
+User may tune the frequency of checks and the treatment of errors
+by the <<cmdhelp_options_check-frequency,`check-frequency`>> and
+<<cmdhelp_options_check-mode,`check-mode`>> preferences.
+
+Note that if the +check-frequency+ is set to +always+ and the
++check-mode+ to +strict+, errors are not tolerated and such
+configuration cannot be saved.
+
+[[topics_Features_Templates,Configuration templates]]
+=== Configuration templates
+
+.Deprecation note
+****************************
+Configuration templates have been deprecated in favor of the more
+capable `cluster scripts`. To learn how to use cluster scripts, see
+the dedicated documentation on the `crmsh` website at
+http://crmsh.github.io/, or in the <<cmdhelp_script,Script section>>.
+****************************
+
+Configuration templates are ready made configurations created by
+cluster experts. They are designed in such a way so that users
+may generate valid cluster configurations with minimum effort.
+If you are new to Pacemaker, templates may be the best way to
+start.
+
+We will show here how to create a simple yet functional Apache
+configuration:
+...............
+# crm configure
+crm(live)configure# template
+crm(live)configure template# list templates
+apache filesystem virtual-ip
+crm(live)configure template# new web <TAB><TAB>
+apache filesystem virtual-ip
+crm(live)configure template# new web apache
+INFO: pulling in template apache
+INFO: pulling in template virtual-ip
+crm(live)configure template# list
+web2-d web2 vip2 web3 vip web
+...............
+
+We enter the `template` level from `configure`. Use the `list`
+command to show templates available on the system. The `new`
+command creates a configuration from the +apache+ template. You
+can use tab completion to pick templates. Note that the apache
+template depends on a virtual IP address which is automatically
+pulled along. The `list` command shows the just created +web+
+configuration, among other configurations (I hope that you,
+unlike me, will use more sensible and descriptive names).
+
+The `show` command, which displays the resulting configuration,
+may be used to get an idea about the minimum required changes
+which have to be done. All +ERROR+ messages show the line numbers
+in which the respective parameters are to be defined:
+...............
+crm(live)configure template# show
+ERROR: 23: required parameter ip not set
+ERROR: 61: required parameter id not set
+ERROR: 65: required parameter configfile not set
+crm(live)configure template# edit
+...............
+
+The `edit` command invokes the preferred text editor with the
++web+ configuration. At the top of the file, the user is advised
+how to make changes. A good template should require from the user
+to specify only parameters. For example, the +web+ configuration
+we created above has the following required and optional
+parameters (all parameter lines start with +%%+):
+...............
+$ grep -n ^%% ~/.crmconf/web
+23:%% ip
+31:%% netmask
+35:%% lvs_support
+61:%% id
+65:%% configfile
+71:%% options
+76:%% envfiles
+...............
+
+These lines are the only ones that should be modified. Simply
+append the parameter value at the end of the line. For instance,
+after editing this template, the result could look like this (we
+used tabs instead of spaces to make the values stand out):
+...............
+$ grep -n ^%% ~/.crmconf/web
+23:%% ip 192.168.1.101
+31:%% netmask
+35:%% lvs_support
+61:%% id websvc
+65:%% configfile /etc/apache2/httpd.conf
+71:%% options
+76:%% envfiles
+...............
+
+As you can see, the parameter line format is very simple:
+...............
+%% <name> <value>
+...............
+
+After editing the file, use `show` again to display the
+configuration:
+...............
+crm(live)configure template# show
+primitive virtual-ip IPaddr \
+ params ip=192.168.1.101
+primitive apache apache \
+ params configfile="/etc/apache2/httpd.conf"
+monitor apache 120s:60s
+group websvc \
+ apache virtual-ip
+...............
+
+The target resource of the apache template is a group which we
+named +websvc+ in this sample session.
+
+This configuration looks exactly as you could type it at the
+`configure` level. The point of templates is to save you some
+typing. It is important, however, to understand the configuration
+produced.
+
+Finally, the configuration may be applied to the current
+crm configuration (note how the configuration changed slightly,
+though it is still equivalent, after being digested at the
+`configure` level):
+...............
+crm(live)configure template# apply
+crm(live)configure template# cd ..
+crm(live)configure# show
+node xen-b
+node xen-c
+primitive apache apache \
+ params configfile="/etc/apache2/httpd.conf" \
+ op monitor interval=120s timeout=60s
+primitive virtual-ip IPaddr \
+ params ip=192.168.1.101
+group websvc apache virtual-ip
+...............
+
+Note that this still does not commit the configuration to the CIB
+which is used in the shell, either the running one (+live+) or
+some shadow CIB. For that you still need to execute the `commit`
+command.
+
+To complete our example, we should also define the preferred node
+to run the service:
+
+...............
+crm(live)configure# location websvc-pref websvc 100: xen-b
+...............
+
+If you are not happy with some resource names which are provided
+by default, you can rename them now:
+
+...............
+crm(live)configure# rename virtual-ip intranet-ip
+crm(live)configure# show
+node xen-b
+node xen-c
+primitive apache apache \
+ params configfile="/etc/apache2/httpd.conf" \
+ op monitor interval=120s timeout=60s
+primitive intranet-ip IPaddr \
+ params ip=192.168.1.101
+group websvc apache intranet-ip
+location websvc-pref websvc 100: xen-b
+...............
+
+To summarize, working with templates typically consists of the
+following steps:
+
+- `new`: create a new configuration from templates
+- `edit`: define parameters, at least the required ones
+- `show`: see if the configuration is valid
+- `apply`: apply the configuration to the `configure` level
+
+[[topics_Features_Testing,Resource testing]]
+=== Resource testing
+
+The amount of detail in a cluster makes all configurations prone
+to errors. By far the largest number of issues in a cluster is
+due to bad resource configuration. The shell can help quickly
+diagnose such problems. And considerably reduce your keyboard
+wear.
+
+Let's say that we entered the following configuration:
+...............
+node xen-b
+node xen-c
+node xen-d
+primitive fencer stonith:external/libvirt \
+ params hypervisor_uri="qemu+tcp://10.2.13.1/system" \
+ hostlist="xen-b xen-c xen-d" \
+ op monitor interval=2h
+primitive svc Xinetd \
+ params service=systat \
+ op monitor interval=30s
+primitive intranet-ip IPaddr2 \
+ params ip=10.2.13.100 \
+ op monitor interval=30s
+primitive apache apache \
+ params configfile="/etc/apache2/httpd.conf" \
+ op monitor interval=120s timeout=60s
+group websvc apache intranet-ip
+location websvc-pref websvc 100: xen-b
+...............
+
+Before typing `commit` to submit the configuration to the cib we
+can make sure that all resources are usable on all nodes:
+...............
+crm(live)configure# rsctest websvc svc fencer
+...............
+
+It is important that resources being tested are not running on
+any nodes. Otherwise, the `rsctest` command will refuse to do
+anything. Of course, if the current configuration resides in a
+CIB shadow, then a `commit` is irrelevant. The point being that
+resources are not running on any node.
+
+.Note on stopping all resources
+****************************
+Alternatively to not committing a configuration, it is also
+possible to tell Pacemaker not to start any resources:
+
+...............
+crm(live)configure# property stop-all-resources=yes
+...............
+Almost none---resources of class stonith are still started. But
+shell is not as strict when it comes to stonith resources.
+****************************
+
+Order of resources is significant insofar that a resource depends
+on all resources to its left. In most configurations, it's
+probably practical to test resources in several runs, based on
+their dependencies.
+
+Apart from groups, `crm` does not interpret constraints and
+therefore knows nothing about resource dependencies. It also
+doesn't know if a resource can run on a node at all in case of an
+asymmetric cluster. It is up to the user to specify a list of
+eligible nodes if a resource is not meant to run on every node.
+
+[[topics_Features_Security,Access Control Lists (ACL)]]
+=== Access Control Lists (ACL)
+
+.Note on ACLs in Pacemaker 1.1.12
+****************************
+The support for ACLs has been revised in Pacemaker version 1.1.12 and
+up. Depending on which version you are using, the information in this
+section may no longer be accurate. Look for the `acl_target`
+configuration element for more details on the new syntax.
+****************************
+
+By default, the users from the +haclient+ group have full access
+to the cluster (or, more precisely, to the CIB). Access control
+lists allow for finer access control to the cluster.
+
+Access control lists consist of an ordered set of access rules.
+Each rule allows read or write access or denies access
+completely. Rules are typically combined to produce a specific
+role. Then, users may be assigned a role.
+
+For instance, this is a role which defines a set of rules
+allowing management of a single resource:
+
+...............
+role bigdb_admin \
+ write meta:bigdb:target-role \
+ write meta:bigdb:is-managed \
+ write location:bigdb \
+ read ref:bigdb
+...............
+
+The first two rules allow modifying the +target-role+ and
++is-managed+ meta attributes which effectively enables users in
+this role to stop/start and manage/unmanage the resource. The
+constraints write access rule allows moving the resource around.
+Finally, the user is granted read access to the resource
+definition.
+
+For proper operation of all Pacemaker programs, it is advisable
+to add the following role to all users:
+
+...............
+role read_all \
+ read cib
+...............
+
+For finer grained read access try with the rules listed in the
+following role:
+
+...............
+role basic_read \
+ read node attribute:uname \
+ read node attribute:type \
+ read property \
+ read status
+...............
+
+It is however possible that some Pacemaker programs (e.g.
+`ptest`) may not function correctly if the whole CIB is not
+readable.
+
+Some of the ACL rules in the examples above are expanded by the
+shell to XPath specifications. For instance,
++meta:bigdb:target-role+ expands to:
+
+........
+//primitive[@id='bigdb']/meta_attributes/nvpair[@name='target-role']
+........
+
+You can see the expansion by showing XML:
+
+...............
+crm(live) configure# show xml bigdb_admin
+...
+<acls>
+ <acl_role id="bigdb_admin">
+ <write id="bigdb_admin-write"
+ xpath="//primitive[@id='bigdb']/meta_attributes/nvpair[@name='target-role']"/>
+...............
+
+Many different XPath expressions can have equal meaning. For
+instance, the following two are equal, but only the first one is
+going to be recognized as shortcut:
+
+...............
+//primitive[@id='bigdb']/meta_attributes/nvpair[@name='target-role']
+//resources/primitive[@id='bigdb']/meta_attributes/nvpair[@name='target-role']
+...............
+
+XPath is a powerful language, but you should try to keep your ACL
+xpaths simple and the builtin shortcuts should be used whenever
+possible.
+
+[[topics_Features_Resourcesets,Syntax: Resource sets]]
+=== Syntax: Resource sets
+
+Using resource sets can be a bit confusing unless one knows the
+details of the implementation in Pacemaker as well as how to interpret
+the syntax provided by `crmsh`.
+
+Three different types of resource sets are provided by `crmsh`, and
+each one implies different values for the two resource set attributes,
++sequential+ and +require-all+.
+
++sequential+::
+ If false, the resources in the set do not depend on each other
+ internally. Setting +sequential+ to +true+ implies a strict order of
+ dependency within the set.
+
++require-all+::
+ If false, only one resource in the set is required to fulfil the
+ requirements of the set. The set of A, B and C with +require-all+
+ set to +false+ is be read as "A OR B OR C" when its dependencies
+ are resolved.
+
+The three types of resource sets modify the attributes in the
+following way:
+
+1. Implicit sets (no brackets). +sequential=true+, +require-all=true+
+2. Parenthesis set (+(+ ... +)+). +sequential=false+, +require-all=true+
+3. Bracket set (+[+ ... +]+). +sequential=false+, +require-all=false+
+
+To create a set with the properties +sequential=true+ and
++require-all=false+, explicitly set +sequential+ in a bracketed set,
++[ A B C sequential=true ]+.
+
+To create multiple sets with both +sequential+ and +require-all+ set to
+true, explicitly set +sequential+ in a parenthesis set:
++A B ( C D sequential=true )+.
+
+[[topics_Features_AttributeListReferences,Syntax: Attribute list references]]
+=== Syntax: Attribute list references
+
+Attribute lists are used to set attributes and parameters for
+resources, constraints and property definitions. For example, to set
+the virtual IP used by an +IPAddr2+ resource the attribute +ip+ can be
+set in an attribute list for that resource.
+
+Attribute lists can have identifiers that name them, and other
+resources can reuse the same attribute list by referring to that name
+using an +$id-ref+. For example, the following statement defines a
+simple dummy resource with an attribute list which sets the parameter
++state+ to the value 1 and sets the identifier for the attribute list
+to +on-state+:
+
+..............
+primitive dummy-1 Dummy params $id=on-state state=1
+..............
+
+To refer to this attribute list from a different resource, refer to
+the +on-state+ name using an id-ref:
+
+..............
+primitive dummy-2 Dummy params $id-ref=on-state
+..............
+
+The resource +dummy-2+ will now also have the parameter +state+ set to the value 1.
+
+[[topics_Features_AttributeReferences,Syntax: Attribute references]]
+=== Syntax: Attribute references
+
+In some cases, referencing complete attribute lists is too
+coarse-grained, for example if two different parameters with different
+names should have the same value set. Instead of having to copy the
+value in multiple places, it is possible to create references to
+individual attributes in attribute lists.
+
+To name an attribute in order to be able to refer to it later, prefix
+the attribute name with a +$+ character (as seen above with the
+special names +$id+ and +$id-ref+:
+
+............
+primitive dummy-1 Dummy params $state=1
+............
+
+The identifier +state+ can now be used to refer to this attribute from other
+primitives, using the +@<id>+ syntax:
+
+............
+primitive dummy-2 Dummy params @state
+............
+
+In some cases, using the attribute name as the identifier doesn't work
+due to name clashes. In this case, the syntax +$<id>:<name>=<value>+
+can be used to give the attribute a different identifier:
+
+............
+primitive dummy-1 params $dummy-state-on:state=1
+primitive dummy-2 params @dummy-state-on
+............
+
+There is also the possibility that two resources both use the same
+attribute value but with different names. For example, a web server
+may have a parameter +server_ip+ for setting the IP address where it
+listens for incoming requests, and a virtual IP resource may have a
+parameter called +ip+ which sets the IP address it creates. To
+configure these two resources with an IP without repeating the value,
+the reference can be given a name using the syntax +@<id>:<name>+.
+
+Example:
+............
+primitive virtual-ip IPaddr2 params $vip:ip=192.168.1.100
+primitive webserver apache params @vip:server_ip
+............
+
+[[topics_Syntax_RuleExpressions,Syntax: Rule expressions]]
+=== Syntax: Rule expressions
+
+Many of the configuration commands in `crmsh` now support the use of
+_rule expressions_, which can influence what attributes apply to a
+resource or under which conditions a constraint is applied, depending
+on changing conditions like date, time, the value of attributes and
+more.
+
+Here is an example of a simple rule expression used to apply a
+a different resource parameter on the node named `node1`:
+
+..............
+primitive my_resource Special \
+ params 2: rule #uname eq node1 interface=eth1 \
+ params 1: interface=eth0
+..............
+
+This primitive resource has two lists of parameters with descending
+priority. The parameter list with the highest priority is applied
+first, but only if the rule expressions for that parameter list all
+apply. In this case, the rule `#uname eq node1` limits the parameter
+list so that it is only applied on `node1`.
+
+Note that rule expressions are not terminated and are immediately
+followed by the data to which the rule is applied. In this case, the
+name-value pair `interface=eth1`.
+
+Rule expressions can contain multiple expressions connected using the
+boolean operator `or` and `and`. The full syntax for rule expressions
+is listed below.
+
+..............
+rules ::
+ rule [id_spec] [$role=<role>] <score>: <expression>
+ [rule [id_spec] [$role=<role>] <score>: <expression> ...]
+
+id_spec :: $id=<id> | $id-ref=<id>
+score :: <number> | <attribute> | [-]inf
+expression :: <simple_exp> [<bool_op> <simple_exp> ...]
+bool_op :: or | and
+simple_exp :: <attribute> [type:]<binary_op> <value>
+ | <unary_op> <attribute>
+ | date <date_expr>
+type :: <string> | <version> | <number>
+binary_op :: lt | gt | lte | gte | eq | ne
+unary_op :: defined | not_defined
+
+date_expr :: lt <end>
+ | gt <start>
+ | in start=<start> end=<end>
+ | in start=<start> <duration>
+ | spec <date_spec>
+duration|date_spec ::
+ hours=<value>
+ | monthdays=<value>
+ | weekdays=<value>
+ | yearsdays=<value>
+ | months=<value>
+ | weeks=<value>
+ | years=<value>
+ | weekyears=<value>
+ | moon=<value>
+..............
+
+[[topics_Reference,Command reference]]
+== Command reference
+
+The commands are structured to be compatible with the shell command
+line. Sometimes, the underlying Pacemaker grammar uses characters that
+have special meaning in bash, that will need to be quoted. This
+includes the hash or pound sign (`#`), single and double quotes, and
+any significant whitespace.
+
+Whitespace is also significant when assigning values, meaning that
++key=value+ is different from +key = value+.
+
+Commands can be referenced using short-hand as long as the short-hand
+is unique. This can be either a prefix of the command name or a prefix
+string of characters found in the name.
+
+For example, +status+ can be abbreviated as +st+ or +su+, and
++configure+ as +conf+ or +cfg+.
+
+The syntax for the commands is given below in an informal, BNF-like
+grammar.
+
+* `<value>` denotes a string.
+* `[value]` means that the construct is optional.
+* The ellipsis (`...`) signifies that the previous construct may be
+ repeated.
+* `first|second` means either first or second.
+* The rest are literals (strings, `:`, `=`).
+
+[[cmdhelp_root_status,Cluster status]]
+=== `status`
+
+Show cluster status. The status is displayed by `crm_mon`. Supply
+additional arguments for more information or different format.
+See `crm_mon(8)` for more details.
+
+Example:
+...............
+status
+status simple
+status full
+...............
+
+Usage:
+...............
+status [<option> ...]
+
+option :: full
+ | bynode
+ | inactive
+ | ops
+ | timing
+ | failcounts
+ | verbose
+ | quiet
+ | html
+ | xml
+ | simple
+ | tickets
+ | noheaders
+ | detail
+ | brief
+...............
+
+[[cmdhelp_root_verify,Verify cluster status]]
+=== `verify`
+
+Performs basic checks for the cluster configuration and
+current status, reporting potential issues.
+
+See `crm_verify(8)` and `crm_simulate(8)` for more details.
+
+Example:
+...............
+verify
+verify scores
+...............
+
+Usage:
+...............
+verify [scores]
+...............
+
+
+[[cmdhelp_cluster,Cluster setup and management]]
+=== `cluster` - Cluster setup and management
+
+Whole-cluster configuration management with High Availability
+awareness.
+
+The commands on the cluster level allows configuration and
+modification of the underlying cluster infrastructure, and also
+supplies tools to do whole-cluster systems management.
+
+These commands enable easy installation and maintenance of a HA
+cluster, by providing support for package installation, configuration
+of the cluster messaging layer, file system setup and more.
+
+[[cmdhelp_cluster_add,Add a new node to the cluster]]
+==== `add`
+
+This command simplifies the process of adding a new node to a running
+cluster. The new node will be installed and configured with the
+packages and configuration files needed to run the cluster
+resources. If a cluster file system is used, the new node will be set
+up to host the file system.
+
+This command should be executed from a node already in the cluster.
+
+Usage:
+...............
+add <node>
+...............
+
+[[cmdhelp_cluster_copy,Copy file to other cluster nodes]]
+==== `copy`
+
+Copy file to other cluster nodes.
+
+Copies the given file to all other nodes unless given a
+list of nodes to copy to as argument.
+
+Usage:
+...............
+copy <filename> [nodes ...]
+...............
+
+Example:
+...............
+copy /etc/motd
+...............
+
+[[cmdhelp_cluster_diff,Diff file across cluster]]
+==== `diff`
+
+Displays the difference, if any, between a given file
+on different nodes. If the second argument is `--checksum`,
+a checksum of the file will be calculated and displayed for
+each node.
+
+Usage:
+...............
+diff <file> [--checksum] [nodes...]
+...............
+
+Example:
+...............
+diff /etc/crm/crm.conf node2
+diff /etc/resolv.conf --checksum
+...............
+
+[[cmdhelp_cluster_health,Cluster health check]]
+==== `health`
+
+Runs a larger set of tests and queries on all nodes in the cluster to
+verify the general system health and detect potential problems.
+
+Usage:
+...............
+health
+...............
+
+[[cmdhelp_cluster_init,Initializes a new HA cluster]]
+==== `init`
+
+Installs and configures a basic HA cluster on a set of nodes.
+
+Usage:
+........
+init node1 node2 node3
+init --dry-run node1 node2 node3
+........
+
+[[cmdhelp_cluster_remove,Remove a node from the cluster]]
+==== `remove`
+
+This command simplifies the process of removing a node from the
+cluster, moving any resources hosted by that node to other nodes.
+
+Usage:
+...............
+remove <node>
+...............
+
+[[cmdhelp_cluster_run,Execute an arbitrary command on all nodes]]
+==== `run`
+
+This command takes a shell statement as argument, executes that
+statement on all nodes in the cluster, and reports the result.
+
+Usage:
+...............
+run <command>
+...............
+
+Example:
+...............
+run "cat /proc/uptime"
+...............
+
+[[cmdhelp_cluster_start,Start cluster services]]
+==== `start`
+
+Starts the cluster-related system services on this node.
+
+Usage:
+.........
+start
+.........
+
+[[cmdhelp_cluster_status,Cluster status check]]
+==== `status`
+
+Reports the status for the cluster messaging layer on the local
+node.
+
+Usage:
+...............
+status
+...............
+
+[[cmdhelp_cluster_stop,Stop cluster services]]
+==== `stop`
+
+Stops the cluster-related system services on this node.
+
+Usage:
+.........
+stop
+.........
+
+[[cmdhelp_cluster_wait_for_startup,Wait for cluster to start]]
+==== `wait_for_startup`
+
+Mostly useful in scripts or automated workflows, this command will
+attempt to connect to the local cluster node repeatedly. The command
+will keep trying until the cluster node responds, or the `timeout`
+elapses. The timeout can be changed by supplying a value in seconds as
+an argument.
+
+Usage:
+........
+wait_for_startup
+........
+
+[[cmdhelp_script,Cluster script management]]
+=== `script` - Cluster script management
+
+A big part of the configuration and management of a cluster is
+collecting information about all cluster nodes and deploying changes
+to those nodes. Often, just performing the same procedure on all nodes
+will encounter problems, due to subtle differences in the
+configuration.
+
+For example, when configuring a cluster for the first time, the
+software needs to be installed and configured on all nodes before the
+cluster software can be launched and configured using `crmsh`. This
+process is cumbersome and error-prone, and the goal is for scripts to
+make this process easier.
+
+Scripts are implemented using the python `parallax` package which
+provides a thin wrapper on top of SSH. This allows the scripts to
+function through the usual SSH channels used for system maintenance,
+requiring no additional software to be installed or maintained.
+
+[[cmdhelp_script_json,JSON API for cluster scripts]]
+==== `json`
+
+This command provides a JSON API for the cluster scripts, intended for
+use in user interface tools that want to interact with the cluster via
+scripts.
+
+The command takes a single argument, which should be a JSON array with
+the first member identifying the command to perform.
+
+The output is line-based: Commands that return multiple results will
+return them line-by-line, ending with a terminator value: "end".
+
+When providing parameter values to this command, they should be
+provided as nested objects, so +virtual-ip:ip=192.168.0.5+ on the
+command line becomes the JSON object
++{"virtual-ip":{"ip":"192.168.0.5"}}+.
+
+API:
+........
+["list"]
+=> [{name, shortdesc, category}]
+
+["show", <name>]
+=> [{name, shortdesc, longdesc, category, <<steps>>}]
+
+<<steps>> := [{name, shortdesc], longdesc, required, parameters, steps}]
+
+<<params>> := [{name, shortdesc, longdesc, required, unique, advanced,
+ type, value, example}]
+
+["verify", <name>, <<values>>]
+=> [{shortdesc, longdesc, text, nodes}]
+
+["run", <name>, <<values>>]
+=> [{shortdesc, rc, output|error}]
+........
+
+
+[[cmdhelp_script_list,List available scripts]]
+==== `list`
+
+Lists the available scripts, sorted by category. Scripts that have the
+special `Script` category are hidden by default, since they are mainly
+used by other scripts or commands. To also show these, pass `all` as
+argument.
+
+To get a flat list of script names, not sorted by category, pass
+`names` as an extra argument.
+
+Usage:
+............
+list [all] [names]
+............
+
+Example:
+............
+list
+list all names
+............
+
+[[cmdhelp_script_run,Run the script]]
+==== `run`
+
+Given a list of parameter values, this command will execute the
+actions specified by the cluster script. The format for the parameter
+values is the same as for the `verify` command.
+
+Can optionally take at least two parameters:
+* `nodes=<nodes>`: List of nodes that the script runs over
+* `dry_run=yes|no`: If set, the script will not perform any modifications.
+
+Additional parameters may be available depending on the script.
+
+Use the `show` command to see what parameters are available.
+
+Usage:
+.............
+run <script> [args...]
+.............
+
+Example:
+.............
+run apache install=true
+run sbd id=sbd-1 node=node1 sbd_device=/dev/disk/by-uuid/F00D-CAFE
+.............
+
+[[cmdhelp_script_show,Describe the script]]
+==== `show`
+
+Prints a description and short summary of the script, with
+descriptions of the accepted parameters.
+
+Advanced parameters are hidden by default. To show the complete list
+of parameters accepted by the script, pass `all` as argument.
+
+Usage:
+............
+show <script> [all]
+............
+
+Example:
+............
+show virtual-ip
+............
+
+[[cmdhelp_script_verify,Verify the script]]
+==== `verify`
+
+Checks the given parameter values, and returns a list
+of actions that will be executed when running the script
+if provided the same list of parameter values.
+
+Usage:
+............
+verify <script> [args...]
+............
+
+Example:
+............
+verify sbd id=sbd-1 node=node1 sbd_device=/dev/disk/by-uuid/F00D-CAFE
+............
+
+[[cmdhelp_corosync,Corosync management]]
+=== `corosync` - Corosync management
+
+Corosync is the underlying messaging layer for most HA clusters.
+This level provides commands for editing and managing the corosync
+configuration.
+
+[[cmdhelp_corosync_add-node,Add a corosync node]]
+==== `add-node`
+
+Adds a node to the corosync configuration. This is used with the `udpu`
+type configuration in corosync.
+
+A nodeid for the added node is generated automatically.
+
+Note that this command assumes that only a single ring is used, and
+sets only the address for ring0.
+
+Usage:
+.........
+add-node <addr> [name]
+.........
+
+[[cmdhelp_corosync_del-node,Remove a corosync node]]
+==== `del-node`
+
+Removes a node from the corosync configuration. The argument given is
+the `ring0_addr` address set in the configuration file.
+
+Usage:
+.........
+del-node <addr>
+.........
+
+[[cmdhelp_corosync_diff,Diffs the corosync configuration]]
+==== `diff`
+
+Diffs the corosync configurations on different nodes. If no nodes are
+given as arguments, the corosync configurations on all nodes in the
+cluster are compared.
+
+`diff` takes an option argument `--checksum`, to display a checksum
+for each file instead of calculating a diff.
+
+Usage:
+.........
+diff [--checksum] [node...]
+.........
+
+[[cmdhelp_corosync_edit,Edit the corosync configuration]]
+==== `edit`
+
+Opens the Corosync configuration file in an editor.
+
+Usage:
+.........
+edit
+.........
+
+[[cmdhelp_corosync_get,Get a corosync configuration value]]
+==== `get`
+
+Returns the value configured in `corosync.conf`, which is not
+necessarily the value used in the running configuration. See `reload`
+for telling corosync about configuration changes.
+
+The argument is the complete dot-separated path to the value.
+
+If there are multiple values configured with the same path, the
+command returns all values for that path. For example, to get all
+configured `ring0_addr` values, use this command:
+
+Example:
+........
+get nodelist.node.ring0_addr
+........
+
+[[cmdhelp_corosync_log,Show the corosync log file]]
+==== `log`
+
+Opens the log file specified in the corosync configuration file. If no
+log file is configured, this command returns an error.
+
+The pager used can be configured either using the PAGER
+environment variable or in `crm.conf`.
+
+Usage:
+.........
+log
+.........
+
+[[cmdhelp_corosync_pull,Pulls the corosync configuration]]
+==== `pull`
+
+Gets the corosync configuration from another node and copies
+it to this node.
+
+Usage:
+.........
+pull <node>
+.........
+
+[[cmdhelp_corosync_push,Push the corosync configuration]]
+==== `push`
+
+Pushes the corosync configuration file on this node to
+the list of nodes provided. If no target nodes are given,
+the configuration is pushed to all other nodes in the cluster.
+
+It is recommended to use `csync2` to distribute the cluster
+configuration files rather than relying on this command.
+
+Usage:
+.........
+push [node] ...
+.........
+
+Example:
+.........
+push node-2 node-3
+.........
+
+[[cmdhelp_corosync_reload,Reload the corosync configuration]]
+==== `reload`
+
+Tells all instances of corosync in this cluster to reload
+`corosync.conf`.
+
+After pushing a new configuration to all cluster nodes, call this
+command to make corosync use the new configuration.
+
+Usage:
+.........
+reload
+.........
+
+[[cmdhelp_corosync_set,Set a corosync configuration value]]
+==== `set`
+
+Sets the value identified by the given path. If the value does not
+exist in the configuration file, it will be added. However, if the
+section containing the value does not exist, the command will fail.
+
+Usage:
+.........
+set quorum.expected_votes 2
+.........
+
+[[cmdhelp_corosync_show,Display the corosync configuration]]
+==== `show`
+
+Displays the corosync configuration on the current node.
+
+.........
+show
+.........
+
+[[cmdhelp_corosync_status,Display the corosync status]]
+==== `status`
+
+Displays the status of Corosync, including the votequorum state.
+
+Usage:
+.........
+status
+.........
+
+[[cmdhelp_cib,CIB shadow management]]
+=== `cib` - CIB shadow management
+
+This level is for management of shadow CIBs. It is available both
+at the top level and the `configure` level.
+
+All the commands are implemented using `cib_shadow(8)` and the
+`CIB_shadow` environment variable. The user prompt always
+includes the name of the currently active shadow or the live CIB.
+
+[[cmdhelp_cib_cibstatus,CIB status management and editing]]
+==== `cibstatus`
+
+Enter edit and manage the CIB status section level. See the
+<<cmdhelp_cibstatus,CIB status management section>>.
+
+[[cmdhelp_cib_commit,copy a shadow CIB to the cluster]]
+==== `commit`
+
+Apply a shadow CIB to the cluster. If the shadow name is omitted
+then the current shadow CIB is applied.
+
+Temporary shadow CIBs are removed automatically on commit.
+
+Usage:
+...............
+commit [<cib>]
+...............
+
+[[cmdhelp_cib_delete,delete a shadow CIB]]
+==== `delete`
+
+Delete an existing shadow CIB.
+
+Usage:
+...............
+delete <cib>
+...............
+
+[[cmdhelp_cib_diff,diff between the shadow CIB and the live CIB]]
+==== `diff`
+
+Print differences between the current cluster configuration and
+the active shadow CIB.
+
+Usage:
+...............
+diff
+...............
+
+[[cmdhelp_cib_import,import a CIB or PE input file to a shadow]]
+==== `import`
+
+At times it may be useful to create a shadow file from the
+existing CIB. The CIB may be specified as file or as a PE input
+file number. The shell will look up files in the local directory
+first and then in the PE directory (typically `/var/lib/pengine`).
+Once the CIB file is found, it is copied to a shadow and this
+shadow is immediately available for use at both `configure` and
+`cibstatus` levels.
+
+If the shadow name is omitted then the target shadow is named
+after the input CIB file.
+
+Note that there are often more than one PE input file, so you may
+need to specify the full name.
+
+Usage:
+...............
+import {<file>|<number>} [<shadow>]
+...............
+Examples:
+...............
+import pe-warn-2222
+import 2289 issue2
+...............
+
+[[cmdhelp_cib_list,list all shadow CIBs]]
+==== `list`
+
+List existing shadow CIBs.
+
+Usage:
+...............
+list
+...............
+
+[[cmdhelp_cib_new,create a new shadow CIB]]
+==== `new`
+
+Create a new shadow CIB. The live cluster configuration and
+status is copied to the shadow CIB.
+
+If the name of the shadow is omitted, we create a temporary CIB
+shadow. It is useful if multiple level sessions are desired
+without affecting the cluster. A temporary CIB shadow is short
+lived and will be removed either on `commit` or on program exit.
+Note that if the temporary shadow is not committed all changes in
+the temporary shadow are lost.
+
+Specify `withstatus` if you want to edit the status section of
+the shadow CIB (see the <<cmdhelp_cibstatus,cibstatus section>>).
+Add `force` to force overwriting the existing shadow CIB.
+
+To start with an empty configuration that is not copied from the live
+CIB, specify the `empty` keyword. (This also allows a shadow CIB to be
+created in case no cluster is running.)
+
+Usage:
+...............
+new [<cib>] [withstatus] [force] [empty]
+...............
+
+[[cmdhelp_cib_reset,copy live cib to a shadow CIB]]
+==== `reset`
+
+Copy the current cluster configuration into the shadow CIB.
+
+Usage:
+...............
+reset <cib>
+...............
+
+[[cmdhelp_cib_use,change working CIB]]
+==== `use`
+
+Choose a CIB source. If you want to edit the status from the
+shadow CIB specify `withstatus` (see <<cmdhelp_cibstatus,`cibstatus`>>).
+Leave out the CIB name to switch to the running CIB.
+
+Usage:
+...............
+use [<cib>] [withstatus]
+...............
+
+[[cmdhelp_ra,Resource Agents (RA) lists and documentation]]
+=== `ra` - Resource Agents (RA) lists and documentation
+
+This level contains commands which show various information about
+the installed resource agents. It is available both at the top
+level and at the `configure` level.
+
+[[cmdhelp_ra_classes,list classes and providers]]
+==== `classes`
+
+Print all resource agents' classes and, where appropriate, a list
+of available providers.
+
+Usage:
+...............
+classes
+...............
+
+[[cmdhelp_ra_info,show meta data for a RA]]
+==== `info` (`meta`)
+
+Show the meta-data of a resource agent type. This is where users
+can find information on how to use a resource agent. It is also
+possible to get information from some programs: `pengine`,
+`crmd`, `cib`, and `stonithd`. Just specify the program name
+instead of an RA.
+
+Usage:
+...............
+info [<class>:[<provider>:]]<type>
+info <type> <class> [<provider>] (obsolete)
+...............
+Example:
+...............
+info apache
+info ocf:pacemaker:Dummy
+info stonith:ipmilan
+info pengine
+...............
+
+[[cmdhelp_ra_list,list RA for a class (and provider)]]
+==== `list`
+
+List available resource agents for the given class. If the class
+is `ocf`, supply a provider to get agents which are available
+only from that provider.
+
+Usage:
+...............
+list <class> [<provider>]
+...............
+Example:
+...............
+list ocf pacemaker
+...............
+
+[[cmdhelp_ra_providers,show providers for a RA and a class]]
+==== `providers`
+
+List providers for a resource agent type. The class parameter
+defaults to `ocf`.
+
+Usage:
+...............
+providers <type> [<class>]
+...............
+Example:
+...............
+providers apache
+...............
+
+[[cmdhelp_ra_validate,validate parameters for RA]]
+==== `validate`
+
+If the resource agent supports the `validate-all` action, this calls
+the action with the given parameters, printing any warnings or errors
+reported by the agent.
+
+Usage:
+................
+validate <agent> [<key>=<value> ...]
+................
+
+[[cmdhelp_resource,Resource management]]
+=== `resource` - Resource management
+
+At this level resources may be managed.
+
+All (or almost all) commands are implemented with the CRM tools
+such as `crm_resource(8)`.
+
+[[cmdhelp_resource_ban,ban a resource from a node]]
+==== `ban`
+
+Ban a resource from running on a certain node. If no node is given
+as argument, the resource is banned from the current location.
+
+See `move` for details on other arguments.
+
+Usage:
+...............
+ban <rsc> [<node>] [<lifetime>] [force]
+...............
+
+[[cmdhelp_resource_cleanup,cleanup resource status]]
+==== `cleanup`
+
+Cleanup resource status. Typically done after the resource has
+temporarily failed. If a node is omitted, cleanup on all nodes.
+If there are many nodes, the command may take a while.
+
++(Pacemaker 1.1.14)+ Pass force to cleanup the resource itself,
+otherwise the cleanup command will apply to the parent resource (if
+any).
+
+Usage:
+...............
+cleanup <rsc> [<node>] [force]
+...............
+
+[[cmdhelp_resource_clear,Clear any relocation constraint]]
+==== `clear` (`unmove`, `unmigrate`, `unban`)
+
+Remove any relocation constraint created by
+the `move`, `migrate` or `ban` command.
+
+Usage:
+...............
+clear <rsc>
+unmigrate <rsc>
+unban <rsc>
+...............
+
+[[cmdhelp_resource_constraints,Show constraints affecting a resource]]
+==== `constraints`
+
+Display the location and colocation constraints affecting the
+resource.
+
+Usage:
+................
+constraints <rsc>
+................
+
+[[cmdhelp_resource_demote,demote a master-slave resource]]
+==== `demote`
+
+Demote a master-slave resource using the `target-role`
+attribute.
+
+Usage:
+...............
+demote <rsc>
+...............
+
+[[cmdhelp_resource_failcount,manage failcounts]]
+==== `failcount`
+
+Show/edit/delete the failcount of a resource.
+
+Usage:
+...............
+failcount <rsc> set <node> <value>
+failcount <rsc> delete <node>
+failcount <rsc> show <node>
+...............
+Example:
+...............
+failcount fs_0 delete node2
+...............
+
+[[cmdhelp_resource_locate,show the location of resources]]
+==== `locate`
+
+Show the current location of one or more resources.
+
+Usage:
+...............
+locate [<rsc> ...]
+...............
+
+[[cmdhelp_resource_maintenance,Enable/disable per-resource maintenance mode]]
+==== `maintenance`
+
+Enables or disables the per-resource maintenance mode. When this mode
+is enabled, no monitor operations will be triggered for the resource.
+`maintenance` attribute conflicts with the `is-managed`. When setting
+the `maintenance` attribute, the user is proposed to remove the
+`is-managed` attribute if it exists.
+
+Usage:
+..................
+maintenance <resource> [on|off|true|false]
+..................
+
+Example:
+..................
+maintenance rsc1
+maintenance rsc2 off
+..................
+
+[[cmdhelp_resource_manage,put a resource into managed mode]]
+==== `manage`
+
+Manage a resource using the `is-managed` attribute. If there
+are multiple meta attributes sets, the attribute is set in all of
+them. If the resource is a clone, all `is-managed` attributes are
+removed from the children resources.
+`is-managed` attribute conflicts with the `maintenance`. When setting
+the `is-managed` attribute, the user is proposed to remove the
+`maintenance` attribute if it exists.
+
+For details on group management see <<cmdhelp_options_manage-children,`options manage-children`>>.
+
+Usage:
+...............
+manage <rsc>
+...............
+
+[[cmdhelp_resource_meta,manage a meta attribute]]
+==== `meta`
+
+Show/edit/delete a meta attribute of a resource. Currently, all
+meta attributes of a resource may be managed with other commands
+such as `resource stop`.
+
+Usage:
+...............
+meta <rsc> set <attr> <value>
+meta <rsc> delete <attr>
+meta <rsc> show <attr>
+...............
+Example:
+...............
+meta ip_0 set target-role stopped
+...............
+
+[[cmdhelp_resource_move,Move a resource to another node]]
+==== `move` (`migrate`)
+
+Move a resource away from its current location.
+
+If the destination node is left out, the resource is migrated by
+creating a constraint which prevents it from running on the current
+node. For this type of constraint to be created, the +force+ argument
+is required.
+
+A lifetime may be given for the constraint. Once it expires, the
+location constraint will no longer be active.
+
+Usage:
+...............
+move <rsc> [<node>] [<lifetime>] [force]
+...............
+
+[[cmdhelp_resource_operations,Show active resource operations]]
+==== `operations`
+
+Show active operations, optionally filtered by resource and node.
+
+Usage:
+................
+operations [<rsc>] [<node>]
+................
+
+[[cmdhelp_resource_param,manage a parameter of a resource]]
+==== `param`
+
+Show/edit/delete a parameter of a resource.
+
+Usage:
+...............
+param <rsc> set <param> <value>
+param <rsc> delete <param>
+param <rsc> show <param>
+...............
+Example:
+...............
+param ip_0 show ip
+...............
+
+[[cmdhelp_resource_promote,promote a master-slave resource]]
+==== `promote`
+
+Promote a master-slave resource using the `target-role`
+attribute.
+
+Usage:
+...............
+promote <rsc>
+...............
+
+[[cmdhelp_resource_refresh,refresh CIB from the LRM status]]
+==== `refresh`
+
+Refresh CIB from the LRM status.
+
+.Note
+****************************
+`refresh` has been deprecated and is now
+an alias for `cleanup`.
+****************************
+
+Usage:
+...............
+refresh [<node>]
+...............
+
+[[cmdhelp_resource_reprobe,probe for resources not started by the CRM]]
+==== `reprobe`
+
+Probe for resources not started by the CRM.
+
+.Note
+****************************
+`reprobe` has been deprecated and is now
+an alias for `cleanup`.
+****************************
+
+Usage:
+...............
+reprobe [<node>]
+...............
+
+[[cmdhelp_resource_restart,restart resources]]
+==== `restart`
+
+Restart one or more resources. This is essentially a shortcut for
+resource stop followed by a start. The shell is first going to wait
+for the stop to finish, that is for all resources to really stop, and
+only then to order the start action. Due to this command
+entailing a whole set of operations, informational messages are
+printed to let the user see some progress.
+
+For details on group management see
+<<cmdhelp_options_manage-children,`options manage-children`>>.
+
+Usage:
+...............
+restart <rsc> [<rsc> ...]
+...............
+Example:
+...............
+# crm resource restart g_webserver
+INFO: ordering g_webserver to stop
+waiting for stop to finish .... done
+INFO: ordering g_webserver to start
+#
+...............
+
+[[cmdhelp_resource_scores,Display resource scores]]
+==== `scores`
+
+Display the allocation scores for all resources.
+
+Usage:
+................
+scores
+................
+
+[[cmdhelp_resource_secret,manage sensitive parameters]]
+==== `secret`
+
+Sensitive parameters can be kept in local files rather than CIB
+in order to prevent accidental data exposure. Use the `secret`
+command to manage such parameters. `stash` and `unstash` move the
+value from the CIB and back to the CIB respectively. The `set`
+subcommand sets the parameter to the provided value. `delete`
+removes the parameter completely. `show` displays the value of
+the parameter from the local file. Use `check` to verify if the
+local file content is valid.
+
+Usage:
+...............
+secret <rsc> set <param> <value>
+secret <rsc> stash <param>
+secret <rsc> unstash <param>
+secret <rsc> delete <param>
+secret <rsc> show <param>
+secret <rsc> check <param>
+...............
+Example:
+...............
+secret fence_1 show password
+secret fence_1 stash password
+secret fence_1 set password secret_value
+...............
+
+[[cmdhelp_resource_start,start resources]]
+==== `start`
+
+Start one or more resources by setting the `target-role` attribute. If
+there are multiple meta attributes sets, the attribute is set in all
+of them. If the resource is a clone, all `target-role` attributes are
+removed from the children resources.
+
+For details on group management see
+<<cmdhelp_options_manage-children,`options manage-children`>>.
+
+Usage:
+...............
+start <rsc> [<rsc> ...]
+...............
+
+[[cmdhelp_resource_status,show status of resources]]
+==== `status` (`show`, `list`)
+
+Print resource status. More than one resource can be shown at once. If
+the resource parameter is left out, the status of all resources is
+printed.
+
+Usage:
+...............
+status [<rsc> ...]
+...............
+
+[[cmdhelp_resource_stop,stop resources]]
+==== `stop`
+
+Stop one or more resources using the `target-role` attribute. If there
+are multiple meta attributes sets, the attribute is set in all of
+them. If the resource is a clone, all `target-role` attributes are
+removed from the children resources.
+
+For details on group management see
+<<cmdhelp_options_manage-children,`options manage-children`>>.
+
+Usage:
+...............
+stop <rsc> [<rsc> ...]
+...............
+
+[[cmdhelp_resource_trace,start RA tracing]]
+==== `trace`
+
+Start tracing RA for the given operation. The trace files are
+stored in `$HA_VARLIB/trace_ra`. If the operation to be traced is
+monitor, note that the number of trace files can grow very
+quickly.
+
+If no operation name is given, crmsh will attempt to trace all
+operations for the RA. This includes any configured operations, start
+and stop as well as promote/demote for multistate resources.
+
+To trace the probe operation which exists for all resources, either
+set a trace for `monitor` with interval `0`, or use `probe` as the
+operation name.
+
+Usage:
+...............
+trace <rsc> [<op> [<interval>] ]
+...............
+Example:
+...............
+trace fs start
+trace webserver
+trace webserver probe
+trace fs monitor 0
+...............
+
+[[cmdhelp_resource_unmanage,put a resource into unmanaged mode]]
+==== `unmanage`
+
+Unmanage a resource using the `is-managed` attribute. If there
+are multiple meta attributes sets, the attribute is set in all of
+them. If the resource is a clone, all `is-managed` attributes are
+removed from the children resources.
+
+For details on group management see <<cmdhelp_options_manage-children,`options manage-children`>>.
+
+Usage:
+...............
+unmanage <rsc>
+...............
+
+[[cmdhelp_resource_untrace,stop RA tracing]]
+==== `untrace`
+
+Stop tracing RA for the given operation. If no operation name is
+given, crmsh will attempt to stop tracing all operations in resource.
+
+Usage:
+...............
+untrace <rsc> [<op> [<interval>] ]
+...............
+Example:
+...............
+untrace fs start
+untrace webserver
+...............
+
+[[cmdhelp_resource_utilization,manage a utilization attribute]]
+==== `utilization`
+
+Show/edit/delete a utilization attribute of a resource. These
+attributes describe hardware requirements. By setting the
+`placement-strategy` cluster property appropriately, it is
+possible then to distribute resources based on resource
+requirements and node size. See also <<cmdhelp_node_utilization,node utilization attributes>>.
+
+Usage:
+...............
+utilization <rsc> set <attr> <value>
+utilization <rsc> delete <attr>
+utilization <rsc> show <attr>
+...............
+Example:
+...............
+utilization xen1 set memory 4096
+...............
+
+[[cmdhelp_node,Node management]]
+=== `node` - Node management
+
+Node management and status commands.
+
+[[cmdhelp_node_attribute,manage attributes]]
+==== `attribute`
+
+Edit node attributes. This kind of attribute should refer to
+relatively static properties, such as memory size.
+
+Usage:
+...............
+attribute <node> set <attr> <value>
+attribute <node> delete <attr>
+attribute <node> show <attr>
+...............
+Example:
+...............
+attribute node_1 set memory_size 4096
+...............
+
+[[cmdhelp_node_clearstate,Clear node state]]
+==== `clearstate`
+
+Resets and clears the state of the specified node. This node is
+afterwards assumed clean and offline. This command can be used to
+manually confirm that a node has been fenced (e.g., powered off).
+
+Be careful! This can cause data corruption if you confirm that a node is
+down that is, in fact, not cleanly down - the cluster will proceed as if
+the fence had succeeded, possibly starting resources multiple times.
+
+Usage:
+...............
+clearstate <node>
+...............
+
+[[cmdhelp_node_delete,delete node]]
+==== `delete`
+
+Delete a node. This command will remove the node from the CIB
+and, in case the cluster stack is running, use the appropriate
+program (`crm_node` or `hb_delnode`) to remove the node from the
+membership.
+
+If the node is still listed as active and a member of our
+partition we refuse to remove it. With the global force option
+(`-F`) we will try to delete the node anyway.
+
+Usage:
+...............
+delete <node>
+...............
+
+[[cmdhelp_node_fence,fence node]]
+==== `fence`
+
+Make CRM fence a node. This functionality depends on stonith
+resources capable of fencing the specified node. No such stonith
+resources, no fencing will happen.
+
+Usage:
+...............
+fence <node>
+...............
+
+[[cmdhelp_node_maintenance,put node into maintenance mode]]
+==== `maintenance`
+
+Set the node status to maintenance. This is equivalent to the
+cluster-wide `maintenance-mode` property but puts just one node
+into the maintenance mode. If there are maintenaned resources on
+the node, the user will be proposed to remove the maintenance
+property from them.
+
+The node parameter defaults to the node where the command is run.
+
+Usage:
+...............
+maintenance [<node>]
+...............
+
+[[cmdhelp_node_online,set node online]]
+==== `online`
+
+Set a node to online status.
+
+The node parameter defaults to the node where the command is run.
+
+Usage:
+...............
+online [<node>]
+...............
+
+[[cmdhelp_node_ready,put node into ready mode]]
+==== `ready`
+
+Set the node's maintenance status to `off`. The node should be
+now again fully operational and capable of running resource
+operations.
+
+The node parameter defaults to the node where the command is run.
+
+Usage:
+...............
+ready [<node>]
+...............
+
+[[cmdhelp_node_server,show node hostname or server address]]
+==== `server`
+
+Remote nodes may have a configured server address which should
+be used when contacting the node. This command prints the
+server address if configured, else the node name.
+
+If no parameter is given, the adresses or names for all nodes
+are printed.
+
+Usage:
+...............
+server [<node> ...]
+...............
+
+[[cmdhelp_node_show,show node]]
+==== `show`
+
+Show a node definition. If the node parameter is omitted then all
+nodes are shown.
+
+Usage:
+...............
+show [<node>]
+...............
+
+[[cmdhelp_node_standby,put node into standby]]
+==== `standby`
+
+Set a node to standby status. The node parameter defaults to the
+node where the command is run.
+
+Additionally, you may specify a lifetime for the standby---if set to
+`reboot`, the node will be back online once it reboots. `forever` will
+keep the node in standby after reboot. The life time defaults to
+`forever`.
+
+Usage:
+...............
+standby [<node>] [<lifetime>]
+
+lifetime :: reboot | forever
+...............
+
+Example:
+...............
+standby bob reboot
+...............
+
+
+[[cmdhelp_node_status,show nodes' status as XML]]
+==== `status`
+
+Show nodes' status as XML. If the node parameter is omitted then
+all nodes are shown.
+
+Usage:
+...............
+status [<node>]
+...............
+
+[[cmdhelp_node_status-attr,manage status attributes]]
+==== `status-attr`
+
+Edit node attributes which are in the CIB status section, i.e.
+attributes which hold properties of a more volatile nature. One
+typical example is attribute generated by the `pingd` utility.
+
+Usage:
+...............
+status-attr <node> set <attr> <value>
+status-attr <node> delete <attr>
+status-attr <node> show <attr>
+...............
+Example:
+...............
+status-attr node_1 show pingd
+...............
+
+[[cmdhelp_node_utilization,manage utilization attributes]]
+==== `utilization`
+
+Edit node utilization attributes. These attributes describe
+hardware characteristics as integer numbers such as memory size
+or the number of CPUs. By setting the `placement-strategy`
+cluster property appropriately, it is possible then to distribute
+resources based on resource requirements and node size. See also
+<<cmdhelp_resource_utilization,resource utilization attributes>>.
+
+Usage:
+...............
+utilization <node> set <attr> <value>
+utilization <node> delete <attr>
+utilization <node> show <attr>
+...............
+Examples:
+...............
+utilization node_1 set memory 16384
+utilization node_1 show cpu
+...............
+
+[[cmdhelp_site,GEO clustering site support]]
+=== `site` - GEO clustering site support
+
+A cluster may consist of two or more subclusters in different and
+distant locations. This set of commands supports such setups.
+
+[[cmdhelp_site_ticket,manage site tickets]]
+==== `ticket`
+
+Tickets are cluster-wide attributes. They can be managed at the
+site where this command is executed.
+
+It is then possible to constrain resources depending on the
+ticket availability (see the <<cmdhelp_configure_rsc_ticket,`rsc_ticket`>> command
+for more details).
+
+Usage:
+...............
+ticket {grant|revoke|standby|activate|show|time|delete} <ticket>
+...............
+Example:
+...............
+ticket grant ticket1
+...............
+
+[[cmdhelp_options,User preferences]]
+=== `options` - User preferences
+
+The user may set various options for the crm shell itself.
+
+[[cmdhelp_options_add-quotes,add quotes around parameters containing spaces]]
+==== `add-quotes`
+
+The shell (as in `/bin/sh`) parser strips quotes from the command
+line. This may sometimes make it really difficult to type values
+which contain white space. One typical example is the configure
+filter command. The crm shell will supply extra quotes around
+arguments which contain white space. The default is `yes`.
+
+.Note on quotes use
+****************************
+Adding quotes around arguments automatically has been introduced
+with version 1.2.2 and it is technically a regression. Being a
+regression is the only reason the `add-quotes` option exists. If
+you have custom shell scripts which would break, just set the
+`add-quotes` option to `no`.
+
+For instance, with adding quotes enabled, it is possible to do
+the following:
+...............
+# crm configure primitive d1 Dummy \
+ meta description="some description here"
+# crm configure filter 'sed "s/hostlist=./&node-c /"' fencing
+...............
+****************************
+
+[[cmdhelp_options_check-frequency,when to perform semantic check]]
+==== `check-frequency`
+
+Semantic check of the CIB or elements modified or created may be
+done on every configuration change (`always`), when verifying
+(`on-verify`) or `never`. It is by default set to `always`.
+Experts may want to change the setting to `on-verify`.
+
+The checks require that resource agents are present. If they are
+not installed at the configuration time set this preference to
+`never`.
+
+See <<topics_Features_Checks,Configuration semantic checks>> for more details.
+
+[[cmdhelp_options_check-mode,how to treat semantic errors]]
+==== `check-mode`
+
+Semantic check of the CIB or elements modified or created may be
+done in the `strict` mode or in the `relaxed` mode. In the former
+certain problems are treated as configuration errors. In the
+`relaxed` mode all are treated as warnings. The default is `strict`.
+
+See <<topics_Features_Checks,Configuration semantic checks>> for more details.
+
+[[cmdhelp_options_colorscheme,set colors for output]]
+==== `colorscheme`
+
+With `output` set to `color`, a comma separated list of colors
+from this option are used to emphasize:
+
+- keywords
+- object ids
+- attribute names
+- attribute values
+- scores
+- resource references
+
+`crm` can show colors only if there is curses support for python
+installed (usually provided by the `python-curses` package). The
+colors are whatever is available in your terminal. Use `normal`
+if you want to keep the default foreground color.
+
+This user preference defaults to
+`yellow,normal,cyan,red,green,magenta` which is good for
+terminals with dark background. You may want to change the color
+scheme and save it in the preferences file for other color
+setups.
+
+Example:
+...............
+colorscheme yellow,normal,blue,red,green,magenta
+...............
+
+[[cmdhelp_options_editor,set preferred editor program]]
+==== `editor`
+
+The `edit` command invokes an editor. Use this to specify your
+preferred editor program. If not set, it will default to either
+the value of the `EDITOR` environment variable or to one of the
+standard UNIX editors (`vi`,`emacs`,`nano`).
+
+Usage:
+...............
+editor program
+...............
+Example:
+...............
+editor vim
+...............
+
+[[cmdhelp_options_manage-children,how to handle children resource attributes]]
+==== `manage-children`
+
+Some resource management commands, such as `resource stop`, when
+the target resource is a group, may not always produce desired
+result. Each element, group and the primitive members, can have a
+meta attribute and those attributes may end up with conflicting
+values. Consider the following construct:
+...............
+crm(live)# configure show svc fs virtual-ip
+primitive fs Filesystem \
+ params device="/dev/drbd0" directory="/srv/nfs" fstype=ext3 \
+ op monitor interval=10s \
+ meta target-role=Started
+primitive virtual-ip IPaddr2 \
+ params ip=10.2.13.110 iflabel=1 \
+ op monitor interval=10s \
+ op start interval=0 \
+ meta target-role=Started
+group svc fs virtual-ip \
+ meta target-role=Stopped
+...............
+
+Even though the element +svc+ should be stopped, the group is
+actually running because all its members have the +target-role+
+set to +Started+:
+...............
+crm(live)# resource show svc
+resource svc is running on: xen-f
+...............
+
+Hence, if the user invokes +resource stop svc+ the intention is
+not clear. This preference gives the user an opportunity to
+better control what happens if attributes of group members have
+values which are in conflict with the same attribute of the group
+itself.
+
+Possible values are +ask+ (the default), +always+, and +never+.
+If set to +always+, the crm shell removes all children attributes
+which have values different from the parent. If set to +never+,
+all children attributes are left intact. Finally, if set to
++ask+, the user will be asked for each member what is to be done.
+
+[[cmdhelp_options_output,set output type]]
+==== `output`
+
+`crm` can adorn configurations in two ways: in color (similar to
+for instance the `ls --color` command) and by showing keywords in
+upper case. Possible values are `plain`, `color-always`, `color`,
+and 'uppercase'. It is possible to combine `uppercase` with one
+of the color values in order to get an upper case xmass tree. Just
+set this option to `color,uppercase` or `color-always,uppercase`.
+In case you need color codes in pipes, `color-always` forces color
+codes even in case the terminal is not a tty (just like `ls
+--color=always`).
+
+[[cmdhelp_options_pager,set preferred pager program]]
+==== `pager`
+
+The `view` command displays text through a pager. Use this to
+specify your preferred pager program. If not set, it will default
+to either the value of the `PAGER` environment variable or to one
+of the standard UNIX system pagers (`less`,`more`,`pg`).
+
+[[cmdhelp_options_reset,reset user preferences to factory defaults]]
+==== `reset`
+
+This command resets all user options to the defaults. If used as
+a single-shot command, the rc file (+$HOME/.config/crm/rc+) is
+reset to the defaults too.
+
+[[cmdhelp_options_save,save the user preferences to the rc file]]
+==== `save`
+
+Save current settings to the rc file (+$HOME/.config/crm/rc+). On
+further `crm` runs, the rc file is automatically read and parsed.
+
+[[cmdhelp_options_set,Set the value of a given option]]
+==== `set`
+
+Sets the value of an option. Takes the fully qualified
+name of the option as argument, as displayed by +show all+.
+
+The modified option value is stored in the user-local
+configuration file, usually found in +~/.config/crm/crm.conf+.
+
+Usage:
+........
+set <option> <value>
+........
+
+Example:
+........
+set color.warn "magenta bold"
+set editor nano
+........
+
+[[cmdhelp_options_show,show current user preference]]
+==== `show`
+
+Display all current settings.
+
+Given an option name as argument, `show` will display only the value
+of that argument.
+
+Given +all+ as argument, `show` displays all available user options.
+
+Usage:
+........
+show [all|<option>]
+........
+
+Example:
+........
+show
+show skill-level
+show all
+........
+
+[[cmdhelp_options_skill-level,set skill level]]
+==== `skill-level`
+
+Based on the skill-level setting, the user is allowed to use only
+a subset of commands. There are three levels: operator,
+administrator, and expert. The operator level allows only
+commands at the `resource` and `node` levels, but not editing
+or deleting resources. The administrator may do that and may also
+configure the cluster at the `configure` level and manage the
+shadow CIBs. The expert may do all.
+
+Usage:
+...............
+skill-level <level>
+
+level :: operator | administrator | expert
+...............
+
+.Note on security
+****************************
+The `skill-level` option is advisory only. There is nothing
+stopping any users change their skill level (see
+<<topics_Features_Security,Access Control Lists (ACL)>> on how to enforce
+access control).
+****************************
+
+[[cmdhelp_options_sort-elements,sort CIB elements]]
+==== `sort-elements`
+
+`crm` by default sorts CIB elements. If you want them appear in
+the order they were created, set this option to `no`.
+
+Usage:
+...............
+sort-elements {yes|no}
+...............
+Example:
+...............
+sort-elements no
+...............
+
+[[cmdhelp_options_user,set the cluster user]]
+==== `user`
+
+Sufficient privileges are necessary in order to manage a
+cluster: programs such as `crm_verify` or `crm_resource` and,
+ultimately, `cibadmin` have to be run either as `root` or as the
+CRM owner user (typically `hacluster`). You don't have to worry
+about that if you run `crm` as `root`. A more secure way is to
+run the program with your usual privileges, set this option to
+the appropriate user (such as `hacluster`), and setup the
+`sudoers` file.
+
+Usage:
+...............
+user system-user
+...............
+Example:
+...............
+user hacluster
+...............
+
+[[cmdhelp_options_wait,synchronous operation]]
+==== `wait`
+
+In normal operation, `crm` runs a command and gets back
+immediately to process other commands or get input from the user.
+With this option set to `yes` it will wait for the started
+transition to finish. In interactive mode dots are printed to
+indicate progress.
+
+Usage:
+...............
+wait {yes|no}
+...............
+Example:
+...............
+wait yes
+...............
+
+[[cmdhelp_configure,CIB configuration]]
+=== `configure` - CIB configuration
+
+This level enables all CIB object definition commands.
+
+The configuration may be logically divided into four parts:
+nodes, resources, constraints, and (cluster) properties and
+attributes. Each of these commands support one or more basic CIB
+objects.
+
+Nodes and attributes describing nodes are managed using the
+`node` command.
+
+Commands for resources are:
+
+- `primitive`
+- `monitor`
+- `group`
+- `clone`
+- `ms`/`master` (master-slave)
+
+In order to streamline large configurations, it is possible to
+define a template which can later be referenced in primitives:
+
+- `rsc_template`
+
+In that case the primitive inherits all attributes defined in the
+template.
+
+There are three types of constraints:
+
+- `location`
+- `colocation`
+- `order`
+
+It is possible to define fencing order (stonith resource
+priorities):
+
+- `fencing_topology`
+
+Finally, there are the cluster properties, resource meta
+attributes defaults, and operations defaults. All are just a set
+of attributes. These attributes are managed by the following
+commands:
+
+- `property`
+- `rsc_defaults`
+- `op_defaults`
+
+In addition to the cluster configuration, the Access Control
+Lists (ACL) can be setup to allow access to parts of the CIB for
+users other than +root+ and +hacluster+. The following commands
+manage ACL:
+
+- `user`
+- `role`
+
+In Pacemaker 1.1.12 and up, this command replaces the `user` command
+for handling ACLs:
+
+- `acl_target`
+
+The changes are applied to the current CIB only on ending the
+configuration session or using the `commit` command.
+
+Comments start with +#+ in the first line. The comments are tied
+to the element which follows. If the element moves, its comments
+will follow.
+
+[[cmdhelp_configure_acl_target,Define target access rights]]
+==== `acl_target`
+
+Defines an ACL target.
+
+Usage:
+................
+acl_target <tid> [<role> ...]
+................
+Example:
+................
+acl_target joe resource_admin constraint_editor
+................
+
+[[cmdhelp_configure_alert,Event-driven alerts]]
+==== `alert`
+
+.Version note
+****************************
+This feature is only available
+in Pacemaker 1.1.15+.
+****************************
+
+Event-driven alerts enables calling scripts whenever interesting
+events occur in the cluster (nodes joining or leaving, resources
+starting or stopping, etc.).
+
+The +path+ is an arbitrary file path to an alert script. Existing
+external scripts used with ClusterMon resources can be used as alert
+scripts, since the interface is compatible.
+
+Each alert may have a number of receipients configured. These will be
+passed to the script as arguments. The first recipient will also be
+passed as the +CRM_alert_recipient+ environment variable, for
+compatibility with existing scripts that only support one recipient.
+
+The available meta attributes are +timeout+ (default 30s) and
++timestamp-format+ (default `"%H:%M:%S.%06N"`).
+
+Some configurations may require each recipient to be delimited by
+brackets, to avoid ambiguity. In the example +alert-2+ below, the meta
+attribute for `timeout` is defined after the recipient, so the
+brackets are used to ensure that the meta attribute is set for the
+alert and not just the recipient. This can be avoided by setting any
+alert attributes before defining the recipients.
+
+Usage:
+...............
+alert <id> <path> \
+ [attributes <nvpair> ...] \
+ [meta <nvpair> ...] \
+ [to [{] <recipient>
+ [attributes <nvpair> ...] \
+ [meta <nvpair> ...] [}] \
+ ...]
+...............
+
+Example:
+...............
+alert alert-1 /srv/pacemaker/pcmk_alert_sample.sh \
+ to /var/log/cluster-alerts.log
+
+alert alert-2 /srv/pacemaker/example_alert.sh \
+ meta timeout=60s \
+ to { /var/log/cluster-alerts.log }
+...............
+
+[[cmdhelp_configure_cib,CIB shadow management]]
+==== `cib`
+
+This level is for management of shadow CIBs. It is available at
+the `configure` level to enable saving intermediate changes to a
+shadow CIB instead of to the live cluster. This short excerpt
+shows how:
+...............
+crm(live)configure# cib new test-2
+INFO: test-2 shadow CIB created
+crm(test-2)configure# commit
+...............
+Note how the current CIB in the prompt changed from +live+ to
++test-2+ after issuing the `cib new` command. See also the
+<<cmdhelp_cib,CIB shadow management>> for more information.
+
+[[cmdhelp_configure_cibstatus,CIB status management and editing]]
+==== `cibstatus`
+
+Enter edit and manage the CIB status section level. See the
+<<cmdhelp_cibstatus,CIB status management section>>.
+
+[[cmdhelp_configure_clone,define a clone]]
+==== `clone`
+
+The `clone` command creates a resource clone. It may contain a
+single primitive resource or one group of resources.
+
+Usage:
+...............
+clone <name> <rsc>
+ [description=<description>]
+ [meta <attr_list>]
+ [params <attr_list>]
+
+attr_list :: [$id=<id>] <attr>=<val> [<attr>=<val>...] | $id-ref=<id>
+...............
+Example:
+...............
+clone cl_fence apc_1 \
+ meta clone-node-max=1 globally-unique=false
+...............
+
+[[cmdhelp_configure_colocation,colocate resources]]
+==== `colocation` (`collocation`)
+
+This constraint expresses the placement relation between two
+or more resources. If there are more than two resources, then the
+constraint is called a resource set.
+
+The score is used to indicate the priority of the constraint. A
+positive score indicates that the resources should run on the same
+node. A negative score that they should not run on the same
+node. Values of positive or negative +infinity+ indicate a mandatory
+constraint.
+
+In the two resource form, the cluster will place +<with-rsc>+ first,
+and then decide where to put the +<rsc>+ resource.
+
+Collocation resource sets have an extra attribute (+sequential+)
+to allow for sets of resources which don't depend on each other
+in terms of state. The shell syntax for such sets is to put
+resources in parentheses.
+
+Sets cannot be nested.
+
+The optional +node-attribute+ can be used to colocate resources on a
+set of nodes and not necessarily on the same node. For example, by
+setting a node attribute +color+ on all nodes and setting the
++node-attribute+ value to +color+ as well, the colocated resources
+will be placed on any node that has the same color.
+
+For more details on how to configure resource sets, see
+<<topics_Features_Resourcesets,`Syntax: Resource sets`>>.
+
+Usage:
+...............
+colocation <id> <score>: <rsc>[:<role>] <with-rsc>[:<role>]
+ [node-attribute=<node_attr>]
+
+colocation <id> <score>: <resource_sets>
+ [node-attribute=<node_attr>]
+
+resource_sets :: <resource_set> [<resource_set> ...]
+
+resource_set :: ["("|"["] <rsc>[:<role>] [<rsc>[:<role>] ...] \
+ [<attributes>] [")"|"]"]
+
+attributes :: [require-all=(true|false)] [sequential=(true|false)]
+
+...............
+Example:
+...............
+colocation never_put_apache_with_dummy -inf: apache dummy
+colocation c1 inf: A ( B C )
+...............
+
+[[cmdhelp_configure_commit,commit the changes to the CIB]]
+==== `commit`
+
+Commit the current configuration to the CIB in use. As noted
+elsewhere, commands in a configure session don't have immediate
+effect on the CIB. All changes are applied at one point in time,
+either using `commit` or when the user leaves the configure
+level. In case the CIB in use changed in the meantime, presumably
+by somebody else, the crm shell will refuse to apply the changes.
+
+If you know that it's fine to still apply them, add +force+ to the
+command line.
+
+To disable CIB patching and apply the changes by replacing the CIB
+completely, add +replace+ to the command line. Note that this can lead
+to previous changes being overwritten if some other process
+concurrently modifies the CIB.
+
+Usage:
+...............
+commit [force] [replace]
+...............
+
+[[cmdhelp_configure_default-timeouts,set timeouts for operations to minimums from the meta-data]]
+==== `default-timeouts`
+
+This command takes the timeouts from the actions section of the
+resource agent meta-data and sets them for the operations of the
+primitive.
+
+Usage:
+...............
+default-timeouts <id> [<id>...]
+...............
+
+.Note on `default-timeouts`
+****************************
+The use of this command is discouraged in favor of manually
+determining the best timeouts required for the particular
+configuration. Relying on the resource agent to supply appropriate
+timeouts can cause the resource to fail at the worst possible moment.
+
+Appropriate timeouts for resource actions are context-sensitive, and
+should be carefully considered with the whole configuration in mind.
+****************************
+
+[[cmdhelp_configure_delete,delete CIB objects]]
+==== `delete`
+
+Delete one or more objects. If an object to be deleted belongs to
+a container object, such as a group, and it is the only resource
+in that container, then the container is deleted as well. Any
+related constraints are removed as well.
+
+If the object is a started resource, it will not be deleted unless the
++--force+ flag is passed to the command, or the +force+ option is set.
+
+Usage:
+...............
+delete [--force] <id> [<id>...]
+...............
+
+[[cmdhelp_configure_edit,edit CIB objects]]
+==== `edit`
+
+This command invokes the editor with the object description. As
+with the `show` command, the user may choose to edit all objects
+or a set of objects.
+
+If the user insists, he or she may edit the XML edition of the
+object. If you do that, don't modify any id attributes.
+
+Usage:
+...............
+edit [xml] [<id> ...]
+edit [xml] changed
+...............
+
+.Note on renaming element ids
+****************************
+The edit command sometimes cannot properly handle modifying
+element ids. In particular for elements which belong to group or
+ms resources. Group and ms resources themselves also cannot be
+renamed. Please use the `rename` command instead.
+****************************
+
+[[cmdhelp_configure_erase,erase the CIB]]
+==== `erase`
+
+The `erase` clears all configuration. Apart from nodes. To remove
+nodes, you have to specify an additional keyword `nodes`.
+
+Note that removing nodes from the live cluster may have some
+strange/interesting/unwelcome effects.
+
+Usage:
+...............
+erase [nodes]
+...............
+
+[[cmdhelp_configure_fencing_topology,node fencing order]]
+==== `fencing_topology`
+
+If multiple fencing (stonith) devices are available capable of
+fencing a node, their order may be specified by +fencing_topology+.
+The order is specified per node.
+
+Stonith resources can be separated by +,+ in which case all of
+them need to succeed. If they fail, the next stonith resource (or
+set of resources) is used. In other words, use comma to separate
+resources which all need to succeed and whitespace for serial
+order. It is not allowed to use whitespace around comma.
+
+If the node is left out, the order is used for all nodes.
+That should reduce the configuration size in some stonith setups.
+
+From Pacemaker version 1.1.14, it is possible to use a node attribute
+as the +target+ in a fencing topology. The syntax for this usage is
+described below.
+
+From Pacemaker version 1.1.14, it is also possible to use regular
+expression patterns as the +target+ in a fencing topology. The configured
+fencing sequence then applies to all devices matching the pattern.
+
+Usage:
+...............
+fencing_topology <stonith_resources> [<stonith_resources> ...]
+fencing_topology <fencing_order> [<fencing_order> ...]
+
+fencing_order :: <target> <stonith_resources> [<stonith_resources> ...]
+
+stonith_resources :: <rsc>[,<rsc>...]
+target :: <node>: | attr:<node-attribute>=<value> | pattern:<pattern>
+...............
+Example:
+...............
+# Only kill the power if poison-pill fails
+fencing_topology poison-pill power
+
+# As above for node-a, but a different strategy for node-b
+fencing_topology \
+ node-a: poison-pill power \
+ node-b: ipmi serial
+
+# Fencing anything on rack 1 requires fencing via both APC 1 and 2,
+# to defeat the redundancy provided by two separate UPS units.
+fencing_topology attr:rack=1 apc01,apc02
+
+# Fencing for all machines named green.* is done using the pear
+# fencing device first, while all machines named red.* are fenced
+# using the apple fencing device first.
+fencing_topology \
+ pattern:green.* pear apple \
+ pattern:red.* apple pear
+...............
+
+[[cmdhelp_configure_filter,filter CIB objects]]
+==== `filter`
+
+This command filters the given CIB elements through an external
+program. The program should accept input on `stdin` and send
+output to `stdout` (the standard UNIX filter conventions). As
+with the `show` command, the user may choose to filter all or
+just a subset of elements.
+
+It is possible to filter the XML representation of objects, but
+probably not as useful as the configuration language. The
+presentation is somewhat different from what would be displayed
+by the `show` command---each element is shown on a single line,
+i.e. there are no backslashes and no other embelishments.
+
+Don't forget to put quotes around the filter if it contains
+spaces.
+
+Usage:
+...............
+filter <prog> [xml] [<id> ...]
+filter <prog> [xml] changed
+...............
+Examples:
+...............
+filter "sed '/^primitive/s/target-role=[^ ]*//'"
+# crm configure filter "sed '/^primitive/s/target-role=[^ ]*//'"
+crm configure <<END
+ filter "sed '/threshold=\"1\"/s/=\"1\"/=\"0\"/g'"
+END
+...............
+
+.Note on quotation marks
+**************************
+Filter commands which feature a blend of quotation marks can be
+difficult to get right, especially when used directly from bash, since
+bash does its own quotation parsing. In these cases, it can be easier
+to supply the filter command as standard input. See the last example
+above.
+**************************
+
+[[cmdhelp_configure_get_property,Get property value]]
+==== `get-property`
+
+Show the value of the given property. If the value is not set, the
+command will print the default value for the property, if known.
+
+If no property name is passed to the command, the list of known
+cluster properties is printed.
+
+If the property is set multiple times, for example using multiple
+property sets with different rule expressions, the output of this
+command is undefined.
+
+Pass the argument +-t+ or +--true+ to `get-property` to translate
+the argument value into +true+ or +false+. If the value is not
+set, the command will print +false+.
+
+Usage:
+...............
+get-property [-t|--true] [<name>]
+...............
+
+Example:
+...............
+get-property stonith-enabled
+get-property -t maintenance-mode
+...............
+
+[[cmdhelp_configure_graph,generate a directed graph]]
+==== `graph`
+
+Create a graphviz graphical layout from the current cluster
+configuration.
+
+Currently, only `dot` (directed graph) is supported. It is
+essentially a visualization of resource ordering.
+
+The graph may be saved to a file which can be used as source for
+various graphviz tools (by default it is displayed in the user's
+X11 session). Optionally, by specifying the format, one can also
+produce an image instead.
+
+For more or different graphviz attributes, it is possible to save
+the default set of attributes to an ini file. If this file exists
+it will always override the builtin settings. The +exportsettings+
+subcommand also prints the location of the ini file.
+
+Usage:
+...............
+graph [<gtype> [<file> [<img_format>]]]
+graph exportsettings
+
+gtype :: dot
+img_format :: `dot` output format (see the +-T+ option)
+...............
+Example:
+...............
+graph dot
+graph dot clu1.conf.dot
+graph dot clu1.conf.svg svg
+...............
+
+[[cmdhelp_configure_group,define a group]]
+==== `group`
+
+The `group` command creates a group of resources. This can be useful
+when resources depend on other resources and require that those
+resources start in order on the same node. A common use of resource
+groups is to ensure that a server and a virtual IP are located
+together, and that the virtual IP is started before the server.
+
+Grouped resources are started in the order they appear in the group,
+and stopped in the reverse order. If a resource in the group cannot
+run anywhere, resources following it in the group will not start.
+
+`group` can be passed the "container" meta attribute, to indicate that
+it is to be used to group VM resources monitored using Nagios. The
+resource referred to by the container attribute must be of type
+`ocf:heartbeat:Xen`, `ocf:heartbeat:VirtualDomain` or `ocf:heartbeat:lxc`.
+
+Usage:
+...............
+group <name> <rsc> [<rsc>...]
+ [description=<description>]
+ [meta attr_list]
+ [params attr_list]
+
+attr_list :: [$id=<id>] <attr>=<val> [<attr>=<val>...] | $id-ref=<id>
+...............
+Example:
+...............
+group internal_www disk0 fs0 internal_ip apache \
+ meta target_role=stopped
+
+group vm-and-services vm vm-sshd meta container="vm"
+...............
+
+[[cmdhelp_configure_load,import the CIB from a file]]
+==== `load`
+
+Load a part of configuration (or all of it) from a local file or
+a network URL. The +replace+ method replaces the current
+configuration with the one from the source. The +update+ method
+tries to import the contents into the current configuration. The
++push+ method imports the contents into the current configuration
+and removes any lines that are not present in the given
+configuration.
+The file may be a CLI file or an XML file.
+
+If the URL is `-`, the configuration is read from standard input.
+
+Usage:
+...............
+load [xml] <method> URL
+
+method :: replace | update | push
+...............
+Example:
+...............
+load xml update myfirstcib.xml
+load xml replace http://storage.big.com/cibs/bigcib.xml
+load xml push smallcib.xml
+...............
+
+[[cmdhelp_configure_location,a location preference]]
+==== `location`
+
+`location` defines the preference of nodes for the given
+resource. The location constraints consist of one or more rules
+which specify a score to be awarded if the rule matches.
+
+The resource referenced by the location constraint can be one of the
+following:
+
+* Plain resource reference: +location loc1 webserver 100: node1+
+* Resource set in curly brackets: +location loc1 { virtual-ip webserver } 100: node1+
+* Tag containing resource ids: +location loc1 tag1 100: node1+
+* Resource pattern: +location loc1 /web.*/ 100: node1+
+
+The +resource-discovery+ attribute allows probes to be selectively
+enabled or disabled per resource and node.
+
+The syntax for resource sets is described in detail for
+<<cmdhelp_configure_colocation,`colocation`>>.
+
+For more details on how to configure resource sets, see
+<<topics_Features_Resourcesets,`Syntax: Resource sets`>>.
+
+For more information on rule expressions, see
+<<topics_Syntax_RuleExpressions,Syntax: Rule expressions>>.
+
+Usage:
+...............
+location <id> <rsc> [<attributes>] {<node_pref>|<rules>}
+
+rsc :: /<rsc-pattern>/
+ | { resource_sets }
+ | <rsc>
+
+attributes :: role=<role> | resource-discovery=always|never|exclusive
+
+node_pref :: <score>: <node>
+
+rules ::
+ rule [id_spec] [$role=<role>] <score>: <expression>
+ [rule [id_spec] [$role=<role>] <score>: <expression> ...]
+
+id_spec :: $id=<id> | $id-ref=<id>
+score :: <number> | <attribute> | [-]inf
+expression :: <simple_exp> [<bool_op> <simple_exp> ...]
+bool_op :: or | and
+simple_exp :: <attribute> [type:]<binary_op> <value>
+ | <unary_op> <attribute>
+ | date <date_expr>
+type :: string | version | number
+binary_op :: lt | gt | lte | gte | eq | ne
+unary_op :: defined | not_defined
+
+date_expr :: lt <end>
+ | gt <start>
+ | in start=<start> end=<end>
+ | in start=<start> <duration>
+ | spec <date_spec>
+duration|date_spec ::
+ hours=<value>
+ | monthdays=<value>
+ | weekdays=<value>
+ | yearsdays=<value>
+ | months=<value>
+ | weeks=<value>
+ | years=<value>
+ | weekyears=<value>
+ | moon=<value>
+...............
+Examples:
+...............
+location conn_1 internal_www 100: node1
+
+location conn_1 internal_www \
+ rule 50: #uname eq node1 \
+ rule pingd: defined pingd
+
+location conn_2 dummy_float \
+ rule -inf: not_defined pingd or pingd number:lte 0
+
+# never probe for rsc1 on node1
+location no-probe rsc1 resource-discovery=never -inf: node1
+...............
+
+[[cmdhelp_configure_modgroup,modify group]]
+==== `modgroup`
+
+Add or remove primitives in a group. The `add` subcommand appends
+the new group member by default. Should it go elsewhere, there
+are `after` and `before` clauses.
+
+Usage:
+...............
+modgroup <id> add <id> [after <id>|before <id>]
+modgroup <id> remove <id>
+...............
+Examples:
+...............
+modgroup share1 add storage2 before share1-fs
+...............
+
+[[cmdhelp_configure_monitor,add monitor operation to a primitive]]
+==== `monitor`
+
+Monitor is by far the most common operation. It is possible to
+add it without editing the whole resource. Also, long primitive
+definitions may be a bit uncluttered. In order to make this
+command as concise as possible, less common operation attributes
+are not available. If you need them, then use the `op` part of
+the `primitive` command.
+
+Usage:
+...............
+monitor <rsc>[:<role>] <interval>[:<timeout>]
+...............
+Example:
+...............
+monitor apcfence 60m:60s
+...............
+
+Note that after executing the command, the monitor operation may
+be shown as part of the primitive definition.
+
+[[cmdhelp_configure_ms,define a master-slave resource]]
+==== `ms` (`master`)
+
+The `ms` command creates a master/slave resource type. It may contain a
+single primitive resource or one group of resources.
+
+Usage:
+...............
+ms <name> <rsc>
+ [description=<description>]
+ [meta attr_list]
+ [params attr_list]
+
+attr_list :: [$id=<id>] <attr>=<val> [<attr>=<val>...] | $id-ref=<id>
+...............
+Example:
+...............
+ms disk1 drbd1 \
+ meta notify=true globally-unique=false
+...............
+
+.Note on `id-ref` usage
+****************************
+Instance or meta attributes (`params` and `meta`) may contain
+a reference to another set of attributes. In that case, no other
+attributes are allowed. Since attribute sets' ids, though they do
+exist, are not shown in the `crm`, it is also possible to
+reference an object instead of an attribute set. `crm` will
+automatically replace such a reference with the right id:
+
+...............
+crm(live)configure# primitive a2 www-2 meta $id-ref=a1
+crm(live)configure# show a2
+primitive a2 apache \
+ meta $id-ref=a1-meta_attributes
+ [...]
+...............
+It is advisable to give meaningful names to attribute sets which
+are going to be referenced.
+****************************
+
+[[cmdhelp_configure_node,define a cluster node]]
+==== `node`
+
+The node command describes a cluster node. Nodes in the CIB are
+commonly created automatically by the CRM. Hence, you should not
+need to deal with nodes unless you also want to define node
+attributes. Note that it is also possible to manage node
+attributes at the `node` level.
+
+Usage:
+...............
+node [$id=<id>] <uname>[:<type>]
+ [description=<description>]
+ [attributes [$id=<id>] [<score>:] [rule...]
+ <param>=<value> [<param>=<value>...]] | $id-ref=<ref>
+ [utilization [$id=<id>] [<score>:] [rule...]
+ <param>=<value> [<param>=<value>...]] | $id-ref=<ref>
+
+type :: normal | member | ping | remote
+...............
+Example:
+...............
+node node1
+node big_node attributes memory=64
+...............
+
+[[cmdhelp_configure_op_defaults,set resource operations defaults]]
+==== `op_defaults`
+
+Set defaults for the operations meta attributes.
+
+For more information on rule expressions, see
+<<topics_Syntax_RuleExpressions,Syntax: Rule expressions>>.
+
+Usage:
+...............
+op_defaults [$id=<set_id>] [rule ...] <option>=<value> [<option>=<value> ...]
+...............
+Example:
+...............
+op_defaults record-pending=true
+...............
+
+[[cmdhelp_configure_order,order resources]]
+==== `order`
+
+This constraint expresses the order of actions on two resources
+or more resources. If there are more than two resources, then the
+constraint is called a resource set.
+
+Ordered resource sets have an extra attribute to allow for sets
+of resources whose actions may run in parallel. The shell syntax
+for such sets is to put resources in parentheses.
+
+If the subsequent resource can start or promote after any one of the
+resources in a set has done, enclose the set in brackets (+[+ and +]+).
+
+Sets cannot be nested.
+
+Three strings are reserved to specify a kind of order constraint:
++Mandatory+, +Optional+, and +Serialize+. It is preferred to use
+one of these settings instead of score. Previous versions mapped
+scores +0+ and +inf+ to keywords +advisory+ and +mandatory+.
+That is still valid but deprecated.
+
+For more details on how to configure resource sets, see
+<<topics_Features_Resourcesets,`Syntax: Resource sets`>>.
+
+Usage:
+...............
+order <id> [{kind|<score>}:] first then [symmetrical=<bool>]
+
+order <id> [{kind|<score>}:] resource_sets [symmetrical=<bool>]
+
+kind :: Mandatory | Optional | Serialize
+
+first :: <rsc>[:<action>]
+
+then :: <rsc>[:<action>]
+
+resource_sets :: resource_set [resource_set ...]
+
+resource_set :: ["["|"("] <rsc>[:<action>] [<rsc>[:<action>] ...] \
+ [attributes] ["]"|")"]
+
+attributes :: [require-all=(true|false)] [sequential=(true|false)]
+
+...............
+Example:
+...............
+order o-1 Mandatory: apache:start ip_1
+order o-2 Serialize: A ( B C )
+order o-3 inf: [ A B ] C
+order o-4 first-resource then-resource
+...............
+
+[[cmdhelp_configure_primitive,define a resource]]
+==== `primitive`
+
+The primitive command describes a resource. It may be referenced
+only once in group, clone, or master-slave objects. If it's not
+referenced, then it is placed as a single resource in the CIB.
+
+Operations may be specified anonymously, as a group or by reference:
+
+* "Anonymous", as a list of +op+ specifications. Use this
+ method if you don't need to reference the set of operations
+ elsewhere. This is the most common way to define operations.
+
+* If reusing operation sets is desired, use the +operations+ keyword
+ along with an id to give the operations set a name. Use the
+ +operations+ keyword and an id-ref value set to the id of another
+ operations set, to apply the same set of operations to this
+ primitive.
+
+Operation attributes which are not recognized are saved as
+instance attributes of that operation. A typical example is
++OCF_CHECK_LEVEL+.
+
+For multistate resources, roles are specified as +role=<role>+.
+
+A template may be defined for resources which are of the same
+type and which share most of the configuration. See
+<<cmdhelp_configure_rsc_template,`rsc_template`>> for more information.
+
+Attributes containing time values, such as the +interval+ attribute on
+operations, are configured either as a plain number, which is
+interpreted as a time in seconds, or using one of the following
+suffixes:
+
+* +s+, +sec+ - time in seconds (same as no suffix)
+* +ms+, +msec+ - time in milliseconds
+* +us+, +usec+ - time in microseconds
+* +m+, +min+ - time in minutes
+* +h+, +hr+ - time in hours
+
+Usage:
+...............
+primitive <rsc> {[<class>:[<provider>:]]<type>|@<template>}
+ [description=<description>]
+ [[params] attr_list]
+ [meta attr_list]
+ [utilization attr_list]
+ [operations id_spec]
+ [op op_type [<attribute>=<value>...] ...]
+
+attr_list :: [$id=<id>] [<score>:] [rule...]
+ <attr>=<val> [<attr>=<val>...]] | $id-ref=<id>
+id_spec :: $id=<id> | $id-ref=<id>
+op_type :: start | stop | monitor
+...............
+Example:
+...............
+primitive apcfence stonith:apcsmart \
+ params ttydev=/dev/ttyS0 hostlist="node1 node2" \
+ op start timeout=60s \
+ op monitor interval=30m timeout=60s
+
+primitive www8 apache \
+ configfile=/etc/apache/www8.conf \
+ operations $id-ref=apache_ops
+
+primitive db0 mysql \
+ params config=/etc/mysql/db0.conf \
+ op monitor interval=60s \
+ op monitor interval=300s OCF_CHECK_LEVEL=10
+
+primitive r0 ocf:linbit:drbd \
+ params drbd_resource=r0 \
+ op monitor role=Master interval=60s \
+ op monitor role=Slave interval=300s
+
+primitive xen0 @vm_scheme1 xmfile=/etc/xen/vm/xen0
+
+primitive mySpecialRsc Special \
+ params 3: rule #uname eq node1 interface=eth1 \
+ params 2: rule #uname eq node2 interface=eth2 port=8888 \
+ params 1: interface=eth0 port=9999
+
+...............
+
+[[cmdhelp_configure_property,set a cluster property]]
+==== `property`
+
+Set cluster configuration properties. To list the
+available cluster configuration properties, use the
+<<cmdhelp_ra_info,`ra info`>> command with +pengine+, +crmd+,
++cib+ and +stonithd+ as arguments.
+When setting the +maintenance-mode+ property, it will
+inform the user if there are nodes or resources that
+have the +maintenance+ property.
+
+For more information on rule expressions, see
+<<topics_Syntax_RuleExpressions,Syntax: Rule expressions>>.
+
+Usage:
+...............
+property [<set_id>:] [rule ...] <option>=<value> [<option>=<value> ...]
+...............
+Example:
+...............
+property stonith-enabled=true
+property rule date spec years=2014 stonith-enabled=false
+...............
+
+[[cmdhelp_configure_ptest,show cluster actions if changes were committed]]
+==== `ptest` (`simulate`)
+
+Show PE (Policy Engine) motions using `ptest(8)` or
+`crm_simulate(8)`.
+
+A CIB is constructed using the current user edited configuration
+and the status from the running CIB. The resulting CIB is run
+through `ptest` (or `crm_simulate`) to show changes which would
+happen if the configuration is committed.
+
+The status section may be loaded from another source and modified
+using the <<cmdhelp_cibstatus,`cibstatus`>> level commands. In that case, the
+`ptest` command will issue a message informing the user that the
+Policy Engine graph is not calculated based on the current status
+section and therefore won't show what would happen to the
+running but some imaginary cluster.
+
+If you have graphviz installed and X11 session, `dotty(1)` is run
+to display the changes graphically.
+
+Add a string of +v+ characters to increase verbosity. `ptest`
+can also show allocation scores. +utilization+ turns on
+information about the remaining capacity of nodes. With the
++actions+ option, `ptest` will print all resource actions.
+
+The `ptest` program has been replaced by `crm_simulate` in newer
+Pacemaker versions. In some installations both could be
+installed. Use `simulate` to enfore using `crm_simulate`.
+
+Usage:
+...............
+ptest [nograph] [v...] [scores] [actions] [utilization]
+...............
+Examples:
+...............
+ptest scores
+ptest vvvvv
+simulate actions
+...............
+
+[[cmdhelp_configure_refresh,refresh from CIB]]
+==== `refresh`
+
+Refresh the internal structures from the CIB. All changes made
+during this session are lost.
+
+Usage:
+...............
+refresh
+...............
+
+[[cmdhelp_configure_rename,rename a CIB object]]
+==== `rename`
+
+Rename an object. It is recommended to use this command to rename
+a resource, because it will take care of updating all related
+constraints and a parent resource. Changing ids with the edit
+command won't have the same effect.
+
+If you want to rename a resource, it must be in the stopped state.
+
+Usage:
+...............
+rename <old_id> <new_id>
+...............
+
+[[cmdhelp_configure_role,define role access rights]]
+==== `role`
+
+An ACL role is a set of rules which describe access rights to
+CIB. Rules consist of an access right +read+, +write+, or +deny+
+and a specification denoting part of the configuration to which
+the access right applies. The specification can be an XPath or a
+combination of tag and id references. If an attribute is
+appended, then the specification applies only to that attribute
+of the matching element.
+
+There is a number of shortcuts for XPath specifications. The
++meta+, +params+, and +utilization+ shortcuts reference resource
+meta attributes, parameters, and utilization respectively. The
+`location` may be used to specify location constraints most of
+the time to allow resource `move` and `unmove` commands. The
+`property` references cluster properties. The `node` allows
+reading node attributes. +nodeattr+ and +nodeutil+ reference node
+attributes and node capacity (utilization). The `status` shortcut
+references the whole status section of the CIB. Read access to
+status is necessary for various monitoring tools such as
+`crm_mon(8)` (aka `crm status`).
+
+For more information on rule expressions, see
+<<topics_Syntax_RuleExpressions,Syntax: Rule expressions>>.
+
+Usage:
+...............
+role <role-id> rule [rule ...]
+
+rule :: acl-right cib-spec [attribute:<attribute>]
+
+acl-right :: read | write | deny
+
+cib-spec :: xpath-spec | tag-ref-spec
+xpath-spec :: xpath:<xpath> | shortcut
+tag-ref-spec :: tag:<tag> | ref:<id> | tag:<tag> ref:<id>
+
+shortcut :: meta:<rsc>[:<attr>]
+ params:<rsc>[:<attr>]
+ utilization:<rsc>
+ location:<rsc>
+ property[:<attr>]
+ node[:<node>]
+ nodeattr[:<attr>]
+ nodeutil[:<node>]
+ status
+...............
+Example:
+...............
+role app1_admin \
+ write meta:app1:target-role \
+ write meta:app1:is-managed \
+ write location:app1 \
+ read ref:app1
+...............
+
+[[cmdhelp_configure_rsc_defaults,set resource defaults]]
+==== `rsc_defaults`
+
+Set defaults for the resource meta attributes.
+
+For more information on rule expressions, see
+<<topics_Syntax_RuleExpressions,Syntax: Rule expressions>>.
+
+Usage:
+...............
+rsc_defaults [<set_id>:] [rule ...] <option>=<value> [<option>=<value> ...]
+...............
+Example:
+...............
+rsc_defaults failure-timeout=3m
+...............
+
+[[cmdhelp_configure_rsc_template,define a resource template]]
+==== `rsc_template`
+
+The `rsc_template` command creates a resource template. It may be
+referenced in primitives. It is used to reduce large
+configurations with many similar resources.
+
+Usage:
+...............
+rsc_template <name> [<class>:[<provider>:]]<type>
+ [description=<description>]
+ [params attr_list]
+ [meta attr_list]
+ [utilization attr_list]
+ [operations id_spec]
+ [op op_type [<attribute>=<value>...] ...]
+
+attr_list :: [$id=<id>] <attr>=<val> [<attr>=<val>...] | $id-ref=<id>
+id_spec :: $id=<id> | $id-ref=<id>
+op_type :: start | stop | monitor
+...............
+Example:
+...............
+rsc_template public_vm Xen \
+ op start timeout=300s \
+ op stop timeout=300s \
+ op monitor interval=30s timeout=60s \
+ op migrate_from timeout=600s \
+ op migrate_to timeout=600s
+primitive xen0 @public_vm \
+ params xmfile=/etc/xen/xen0
+primitive xen1 @public_vm \
+ params xmfile=/etc/xen/xen1
+...............
+
+[[cmdhelp_configure_rsc_ticket,resources ticket dependency]]
+==== `rsc_ticket`
+
+This constraint expresses dependency of resources on cluster-wide
+attributes, also known as tickets. Tickets are mainly used in
+geo-clusters, which consist of multiple sites. A ticket may be
+granted to a site, thus allowing resources to run there.
+
+The +loss-policy+ attribute specifies what happens to the
+resource (or resources) if the ticket is revoked. The default is
+either +stop+ or +demote+ depending on whether a resource is
+multi-state.
+
+See also the <<cmdhelp_site_ticket,`site`>> set of commands.
+
+Usage:
+...............
+rsc_ticket <id> <ticket_id>: <rsc>[:<role>] [<rsc>[:<role>] ...]
+ [loss-policy=<loss_policy_action>]
+
+loss_policy_action :: stop | demote | fence | freeze
+...............
+Example:
+...............
+rsc_ticket ticket-A_public-ip ticket-A: public-ip
+rsc_ticket ticket-A_bigdb ticket-A: bigdb loss-policy=fence
+rsc_ticket ticket-B_storage ticket-B: drbd-a:Master drbd-b:Master
+...............
+
+
+[[cmdhelp_configure_rsctest,test resources as currently configured]]
+==== `rsctest`
+
+Test resources with current resource configuration. If no nodes
+are specified, tests are run on all known nodes.
+
+The order of resources is significant: it is assumed that later
+resources depend on earlier ones.
+
+If a resource is multi-state, it is assumed that the role on
+which later resources depend is master.
+
+Tests are run sequentially to prevent running the same resource
+on two or more nodes. Tests are carried out only if none of the
+specified nodes currently run any of the specified resources.
+However, it won't verify whether resources run on the other
+nodes.
+
+Superuser privileges are obviously required: either run this as
+root or setup the `sudoers` file appropriately.
+
+Note that resource testing may take some time.
+
+Usage:
+...............
+rsctest <rsc_id> [<rsc_id> ...] [<node_id> ...]
+...............
+Examples:
+...............
+rsctest my_ip websvc
+rsctest websvc nodeB
+...............
+
+[[cmdhelp_configure_save,save the CIB to a file]]
+==== `save`
+
+Save the current configuration to a file. Optionally, as XML. Use
++-+ instead of file name to write the output to `stdout`.
+
+The `save` command accepts the same selection arguments as the `show`
+command. See the <<cmdhelp_configure_show,help section>> for `show`
+for more details.
+
+Usage:
+...............
+save [xml] [<id> | type:<type | tag:<tag> |
+ related:<obj> | changed ...] <file>
+...............
+Example:
+...............
+save myfirstcib.txt
+save web-server server-config.txt
+...............
+
+[[cmdhelp_configure_schema,set or display current CIB RNG schema]]
+==== `schema`
+
+CIB's content is validated by a RNG schema. Pacemaker supports
+several, depending on version. At least the following schemas are
+accepted by `crmsh`:
+
+* +pacemaker-1.0+
+* +pacemaker-1.1+
+* +pacemaker-1.2+
+* +pacemaker-1.3+
+* +pacemaker-2.0+
+
+Use this command to display or switch to another RNG schema.
+
+Usage:
+...............
+schema [<schema>]
+...............
+Example:
+...............
+schema pacemaker-1.1
+...............
+
+[[cmdhelp_configure_set,set an attribute value]]
+==== `set`
+
+Set the value of a configured attribute. The attribute must
+have a value configured previously, and can be an agent
+parameter, meta attribute or utilization value.
+
+The first argument to the command is a path to an attribute.
+This is a dot-separated sequence beginning with the name of
+the resource, and ending with the name of the attribute to
+set.
+
+Usage:
+...............
+set <path> <value>
+...............
+Examples:
+...............
+set vip1.ip 192.168.20.5
+set vm-a.force_stop 1
+...............
+
+[[cmdhelp_configure_show,display CIB objects]]
+==== `show`
+
+The `show` command displays CIB objects. Without any argument, it
+displays all objects in the CIB, but the set of objects displayed by
+`show` can be limited to only objects with the given IDs or by using
+one or more of the special prefixes described below.
+
+The XML representation for the objects can be displayed by passing
++xml+ as the first argument.
+
+To show one or more specific objects, pass the object IDs as
+arguments.
+
+To show all objects of a certain type, use the +type:+ prefix.
+
+To show all objects in a tag, use the +tag:+ prefix.
+
+To show all constraints related to a primitive, use the +related:+ prefix.
+
+To show all modified objects, pass the argument +changed+.
+
+The prefixes can be used together on a single command line. For
+example, to show both the tag itself and the objects tagged by it the
+following combination can be used: +show tag:my-tag my-tag+.
+
+To refine a selection of objects using multiple modifiers, the keywords
++and+ and +or+ can be used. For example, to select all primitives tagged
++foo+, the following combination can be used:
++show type:primitive and tag:foo+.
+
+To hide values when displaying the configuration, use the
++obscure:<glob>+ argument. This can be useful when sending the
+configuration over a public channel, to avoid exposing potentially
+sensitive information. The +<glob>+ argument is a bash-style pattern
+matching attribute keys.
+
+Usage:
+...............
+show [xml] [<id>
+ | changed
+ | type:<type>
+ | tag:<id>
+ | related:<obj>
+ | obscure:<glob>
+ ...]
+
+type :: node | primitive | group | clone | ms | rsc_template
+ | location | colocation | order
+ | rsc_ticket
+ | property | rsc_defaults | op_defaults
+ | fencing_topology
+ | role | user | acl_target
+ | tag
+...............
+
+Example:
+...............
+show webapp
+show type:primitive
+show xml tag:db tag:fs
+show related:webapp
+show type:primitive obscure:passwd
+...............
+
+[[cmdhelp_configure_tag,Define resource tags]]
+==== `tag`
+
+Define a resource tag. A tag is an id referring to one or more
+resources, without implying any constraints between the tagged
+resources. This can be useful for grouping conceptually related
+resources.
+
+Usage:
+...............
+tag <tag-name>: <rsc> [<rsc> ...]
+tag <tag-name> <rsc> [<rsc> ...]
+...............
+Example:
+...............
+tag web: p-webserver p-vip
+tag ips server-vip admin-vip
+...............
+
+[[cmdhelp_configure_template,edit and import a configuration from a template]]
+==== `template`
+
+The specified template is loaded into the editor. It's up to the
+user to make a good CRM configuration out of it. See also the
+<<cmdhelp_template,template section>>.
+
+Usage:
+...............
+template [xml] url
+...............
+Example:
+...............
+template two-apaches.txt
+...............
+
+[[cmdhelp_configure_upgrade,upgrade the CIB]]
+==== `upgrade`
+
+Attempts to upgrade the CIB to validate with the current
+version. Commonly, this is required if the error
+`CIB not supported` occurs. It typically means that the
+active CIB version is coming from an older release.
+
+As a safety precaution, the force argument is required if the
++validation-with+ attribute is set to anything other than
++0.6+. Thus in most cases, it is required.
+
+Usage:
+...............
+upgrade [force]
+...............
+
+Example:
+...............
+upgrade force
+...............
+
+[[cmdhelp_configure_user,define user access rights]]
+==== `user`
+
+Users which normally cannot view or manage cluster configuration
+can be allowed access to parts of the CIB. The access is defined
+by a set of +read+, +write+, and +deny+ rules as in role
+definitions or by referencing roles. The latter is considered
+best practice.
+
+For more information on rule expressions, see
+<<topics_Syntax_RuleExpressions,Syntax: Rule expressions>>.
+
+Usage:
+...............
+user <uid> {roles|rules}
+
+roles :: role:<role-ref> [role:<role-ref> ...]
+rules :: rule [rule ...]
+...............
+Example:
+...............
+user joe \
+ role:app1_admin \
+ role:read_all
+...............
+
+[[cmdhelp_configure_validate_all,call agent validate-all for resource]]
+==== `validate-all`
+
+Call the `validate-all` action for the resource, if possible.
+
+Limitations:
+
+* The resource agent must implement the `validate-all` action.
+* The current user must be root.
+* The primitive resource must not use nvpair references.
+
+Usage:
+...............
+validate-all <rsc>
+...............
+
+
+[[cmdhelp_configure_verify,verify the CIB with crm_verify]]
+==== `verify`
+
+Verify the contents of the CIB which would be committed.
+
+Usage:
+...............
+verify
+...............
+
+[[cmdhelp_configure_xml,raw xml]]
+==== `xml`
+
+Even though we promissed no xml, it may happen, but hopefully
+very very seldom, that an element from the CIB cannot be rendered
+in the configuration language. In that case, the element will be
+shown as raw xml, prefixed by this command. That element can then
+be edited like any other. If the shell finds out that after the
+change it can digest it, then it is going to be converted into
+the normal configuration language. Otherwise, there is no need to
+use `xml` for configuration.
+
+Usage:
+...............
+xml <xml>
+...............
+
+[[cmdhelp_template,edit and import a configuration from a template]]
+=== `template` - Import configuration from templates
+
+User may be assisted in the cluster configuration by templates
+prepared in advance. Templates consist of a typical ready
+configuration which may be edited to suit particular user needs.
+
+This command enters a template level where additional commands
+for configuration/template management are available.
+
+[[cmdhelp_template_apply,process and apply the current configuration to the current CIB]]
+==== `apply`
+
+Copy the current or given configuration to the current CIB. By
+default, the CIB is replaced, unless the method is set to
+"update".
+
+Usage:
+...............
+apply [<method>] [<config>]
+
+method :: replace | update
+...............
+
+[[cmdhelp_template_delete,delete a configuration]]
+==== `delete`
+
+Remove a configuration. The loaded (active) configuration may be
+removed by force.
+
+Usage:
+...............
+delete <config> [force]
+...............
+
+[[cmdhelp_template_edit,edit a configuration]]
+==== `edit`
+
+Edit current or given configuration using your favourite editor.
+
+Usage:
+...............
+edit [<config>]
+...............
+
+[[cmdhelp_template_list,list configurations/templates]]
+==== `list`
+
+When called with no argument, lists existing templates and
+configurations.
+
+Given the argument +templates+, lists the available templates.
+
+Given the argument +configs+, lists the available configurations.
+
+Usage:
+...............
+list [templates|configs]
+...............
+
+[[cmdhelp_template_load,load a configuration]]
+==== `load`
+
+Load an existing configuration. Further `edit`, `show`, and
+`apply` commands will refer to this configuration.
+
+Usage:
+...............
+load <config>
+...............
+
+[[cmdhelp_template_new,create a new configuration from templates]]
+==== `new`
+
+Create a new configuration from one or more templates. Note that
+configurations and templates are kept in different places, so it
+is possible to have a configuration name equal a template name.
+
+If you already know which parameters are required, you can set
+them directly on the command line.
+
+The parameter name +id+ is set by default to the name of the
+configuration.
+
+If no parameters are being set and you don't want a particular name
+for your configuration, you can call this command with a template name
+as the only parameter. A unique configuration name based on the
+template name will be generated.
+
+Usage:
+...............
+new [<config>] <template> [<template> ...] [params name=value ...]
+...............
+
+Example:
+...............
+new vip virtual-ip
+new bigfs ocfs2 params device=/dev/sdx8 directory=/bigfs
+new apache
+...............
+
+[[cmdhelp_template_show,show the processed configuration]]
+==== `show`
+
+Process the current or given configuration and display the result.
+
+Usage:
+...............
+show [<config>]
+...............
+
+[[cmdhelp_cibstatus,CIB status management and editing]]
+=== `cibstatus` - CIB status management and editing
+
+The `status` section of the CIB keeps the current status of nodes
+and resources. It is modified _only_ on events, i.e. when some
+resource operation is run or node status changes. For obvious
+reasons, the CRM has no user interface with which it is possible
+to affect the status section. From the user's point of view, the
+status section is essentially a read-only part of the CIB. The
+current status is never even written to disk, though it is
+available in the PE (Policy Engine) input files which represent
+the history of cluster motions. The current status may be read
+using the +cibadmin -Q+ command.
+
+It may sometimes be of interest to see how status changes would
+affect the Policy Engine. The set of `cibstatus` level commands
+allow the user to load status sections from various sources and
+then insert or modify resource operations or change nodes' state.
+
+The effect of those changes may then be observed by running the
+<<cmdhelp_configure_ptest,`ptest`>> command at the `configure` level
+or `simulate` and `run` commands at this level. The `ptest`
+runs with the user edited CIB whereas the latter two commands
+run with the CIB which was loaded along with the status section.
+
+The `simulate` and `run` commands as well as all status
+modification commands are implemented using `crm_simulate(8)`.
+
+[[cmdhelp_cibstatus_load,load the CIB status section]]
+==== `load`
+
+Load a status section from a file, a shadow CIB, or the running
+cluster. By default, the current (+live+) status section is
+modified. Note that if the +live+ status section is modified it
+is not going to be updated if the cluster status changes, because
+that would overwrite the user changes. To make `crm` drop changes
+and resume use of the running cluster status, run +load live+.
+
+All CIB shadow configurations contain the status section which is
+a snapshot of the status section taken at the time the shadow was
+created. Obviously, this status section doesn't have much to do
+with the running cluster status, unless the shadow CIB has just
+been created. Therefore, the `ptest` command by default uses the
+running cluster status section.
+
+Usage:
+...............
+load {<file>|shadow:<cib>|live}
+...............
+Example:
+...............
+load bug-12299.xml
+load shadow:test1
+...............
+
+[[cmdhelp_cibstatus_node,change node status]]
+==== `node`
+
+Change the node status. It is possible to throw a node out of
+the cluster, make it a member, or set its state to unclean.
+
++online+:: Set the +node_state+ `crmd` attribute to +online+
+and the +expected+ and +join+ attributes to +member+. The effect
+is that the node becomes a cluster member.
+
++offline+:: Set the +node_state+ `crmd` attribute to +offline+
+and the +expected+ attribute to empty. This makes the node
+cleanly removed from the cluster.
+
++unclean+:: Set the +node_state+ `crmd` attribute to +offline+
+and the +expected+ attribute to +member+. In this case the node
+has unexpectedly disappeared.
+
+Usage:
+...............
+node <node> {online|offline|unclean}
+...............
+Example:
+...............
+node xen-b unclean
+...............
+
+[[cmdhelp_cibstatus_op,edit outcome of a resource operation]]
+==== `op`
+
+Edit the outcome of a resource operation. This way you can
+tell CRM that it ran an operation and that the resource agent
+returned certain exit code. It is also possible to change the
+operation's status. In case the operation status is set to
+something other than +done+, the exit code is effectively
+ignored.
+
+Usage:
+...............
+op <operation> <resource> <exit_code> [<op_status>] [<node>]
+
+operation :: probe | monitor[:<n>] | start | stop |
+ promote | demote | notify | migrate_to | migrate_from
+exit_code :: <rc> | success | generic | args |
+ unimplemented | perm | installed | configured | not_running |
+ master | failed_master
+op_status :: pending | done | cancelled | timeout | notsupported | error
+
+n :: the monitor interval in seconds; if omitted, the first
+ recurring operation is referenced
+rc :: numeric exit code in range 0..9
+...............
+Example:
+...............
+op start d1 xen-b generic
+op start d1 xen-b 1
+op monitor d1 xen-b not_running
+op stop d1 xen-b 0 timeout
+...............
+
+[[cmdhelp_cibstatus_origin,display origin of the CIB status section]]
+==== `origin`
+
+Show the origin of the status section currently in use. This
+essentially shows the latest `load` argument.
+
+Usage:
+...............
+origin
+...............
+
+[[cmdhelp_cibstatus_quorum,set the quorum]]
+==== `quorum`
+
+Set the quorum value.
+
+Usage:
+...............
+quorum <bool>
+...............
+Example:
+...............
+quorum false
+...............
+
+[[cmdhelp_cibstatus_run,run policy engine]]
+==== `run`
+
+Run the policy engine with the edited status section.
+
+Add a string of +v+ characters to increase verbosity. Specify
++scores+ to see allocation scores also. +utilization+ turns on
+information about the remaining capacity of nodes.
+
+If you have graphviz installed and X11 session, `dotty(1)` is run
+to display the changes graphically.
+
+Usage:
+...............
+run [nograph] [v...] [scores] [utilization]
+...............
+Example:
+...............
+run
+...............
+
+[[cmdhelp_cibstatus_save,save the CIB status section]]
+==== `save`
+
+The current internal status section with whatever modifications
+were performed can be saved to a file or shadow CIB.
+
+If the file exists and contains a complete CIB, only the status
+section is going to be replaced and the rest of the CIB will
+remain intact. Otherwise, the current user edited configuration
+is saved along with the status section.
+
+Note that all modifications are saved in the source file as soon
+as they are run.
+
+Usage:
+...............
+save [<file>|shadow:<cib>]
+...............
+Example:
+...............
+save bug-12299.xml
+...............
+
+[[cmdhelp_cibstatus_show,show CIB status section]]
+==== `show`
+
+Show the current status section in the XML format. Brace yourself
+for some unreadable output. Add +changed+ option to get a human
+readable output of all changes.
+
+Usage:
+...............
+show [changed]
+...............
+
+[[cmdhelp_cibstatus_simulate,simulate cluster transition]]
+==== `simulate`
+
+Run the policy engine with the edited status section and simulate
+the transition.
+
+Add a string of +v+ characters to increase verbosity. Specify
++scores+ to see allocation scores also. +utilization+ turns on
+information about the remaining capacity of nodes.
+
+If you have graphviz installed and X11 session, `dotty(1)` is run
+to display the changes graphically.
+
+Usage:
+...............
+simulate [nograph] [v...] [scores] [utilization]
+...............
+Example:
+...............
+simulate
+...............
+
+[[cmdhelp_cibstatus_ticket,manage tickets]]
+==== `ticket`
+
+Modify the ticket status. Tickets can be granted and revoked.
+Granted tickets could be activated or put in standby.
+
+Usage:
+...............
+ticket <ticket> {grant|revoke|activate|standby}
+...............
+Example:
+...............
+ticket ticketA grant
+...............
+
+[[cmdhelp_assist,Configuration assistant]]
+=== `assist` - Configuration assistant
+
+The `assist` sublevel is a collection of helper
+commands that create or modify resources and
+constraints, to simplify the creation of certain
+configurations.
+
+For more information on individual commands, see
+the help text for those commands.
+
+[[cmdhelp_assist_template,Create template for primitives]]
+==== `template`
+
+This command takes a list of primitives as argument, and creates a new
+`rsc_template` for these primitives. It can only do this if the
+primitives do not already share a template and are of the same type.
+
+Usage:
+........
+template primitive-1 primitive-2 primitive-3
+........
+
+[[cmdhelp_assist_weak-bond,Create a weak bond between resources]]
+==== `weak-bond`
+
+A colocation between a group of resources says that the resources
+should be located together, but it also means that those resources are
+dependent on each other. If one of the resources fails, the others
+will be restarted.
+
+If this is not desired, it is possible to circumvent: By placing the
+resources in a non-sequential set and colocating the set with a dummy
+resource which is not monitored, the resources will be placed together
+but will have no further dependency on each other.
+
+This command creates both the constraint and the dummy resource needed
+for such a colocation.
+
+Usage:
+........
+weak-bond resource-1 resource-2
+........
+
+[[cmdhelp_maintenance,Maintenance mode commands]]
+=== `maintenance` - Maintenance mode commands
+
+Maintenance mode commands are commands that manipulate resources
+directly without going through the cluster infrastructure. Therefore,
+it is essential to ensure that the cluster does not attempt to monitor
+or manipulate the resources while these commands are being executed.
+
+To ensure this, these commands require that maintenance mode is set
+either for the particular resource, or for the whole cluster.
+
+[[cmdhelp_maintenance_action,Invoke a resource action]]
+==== `action`
+
+Invokes the given action for the resource. This is
+done directly via the resource agent, so the command must
+be issued while the cluster or the resource is in
+maintenance mode.
+
+Unless the action is `start` or `monitor`, the action must be invoked
+on the same node as where the resource is running. If the resource is
+running on multiple nodes, the command will fail.
+
+To use SSH for executing resource actions on multiple nodes, append
+`ssh` after the action name. This requires SSH access to be configured
+between the nodes and the parallax python package to be installed.
+
+Usage:
+...............
+action <rsc> <action>
+action <rsc> <action> ssh
+...............
+Example:
+...............
+action webserver reload
+action webserver monitor ssh
+...............
+
+[[cmdhelp_maintenance_off,Disable maintenance mode]]
+==== `off`
+
+Disables maintenances mode, either for the whole cluster
+or for the given resource.
+
+Usage:
+...............
+off
+off <rsc>
+...............
+Example:
+...............
+off rsc1
+...............
+
+[[cmdhelp_maintenance_on,Enable maintenance mode]]
+==== `on`
+
+Enables maintenances mode, either for the whole cluster
+or for the given resource.
+
+Usage:
+...............
+on
+on <rsc>
+...............
+Example:
+...............
+on rsc1
+...............
+
+[[cmdhelp_history,Cluster history]]
+=== `history` - Cluster history
+
+Examining Pacemaker's history is a particularly involved task. The
+number of subsystems to be considered, the complexity of the
+configuration, and the set of various information sources, most of
+which are not exactly human readable, keep analyzing resource or node
+problems accessible to only the most knowledgeable. Or, depending on
+the point of view, to the most persistent. The following set of
+commands has been devised in hope to make cluster history more
+accessible.
+
+Of course, looking at _all_ history could be time consuming regardless
+of how good the tools at hand are. Therefore, one should first say
+which period he or she wants to analyze. If not otherwise specified,
+the last hour is considered. Logs and other relevant information is
+collected using `crm report`. Since this process takes some time and
+we always need fresh logs, information is refreshed in a much faster
+way using the python parallax module. If +python-parallax+ is not
+found on the system, examining a live cluster is still possible --
+though not as comfortable.
+
+Apart from examining a live cluster, events may be retrieved from a
+report generated by `crm report` (see also the +-H+ option). In that
+case we assume that the period stretching the whole report needs to be
+investigated. Of course, it is still possible to further reduce the
+time range.
+
+If you have discovered an issue that you want to show someone else,
+you can use the `session pack` command to save the current session as
+a tarball, similar to those generated by `crm report`.
+
+In order to minimize the size of the tarball, and to make it easier
+for others to find the interesting events, it is recommended to limit
+the time frame which the saved session covers. This can be done using
+the `timeframe` command (example below).
+
+It is also possible to name the saved session using the `session save`
+command.
+
+Example:
+...............
+crm(live)history# limit "Jul 18 12:00" "Jul 18 12:30"
+crm(live)history# session save strange_restart
+crm(live)history# session pack
+Report saved in .../strange_restart.tar.bz2
+crm(live)history#
+...............
+
+[[cmdhelp_history_detail,set the level of detail shown]]
+==== `detail`
+
+How much detail to show from the logs. Valid detail levels are either
+`0` or `1`, where `1` is the highest detail level. The default detail
+level is `0`.
+
+Usage:
+...............
+detail <detail_level>
+
+detail_level :: small integer (defaults to 0)
+...............
+Example:
+...............
+detail 1
+...............
+
+[[cmdhelp_history_diff,cluster states/transitions difference]]
+==== `diff`
+
+A transition represents a change in cluster configuration or
+state. Use `diff` to see what has changed between two
+transitions.
+
+If you want to specify the current cluster configuration and
+status, use the string +live+.
+
+Normally, the first transition specified should be the one which
+is older, but we are not going to enforce that.
+
+Note that a single configuration update may result in more than
+one transition.
+
+Usage:
+...............
+diff <pe> <pe> [status] [html]
+
+pe :: <number>|<index>|<file>|live
+...............
+Examples:
+...............
+diff 2066 2067
+diff pe-input-2080.bz2 live status
+...............
+
+[[cmdhelp_history_events,Show events in log]]
+==== `events`
+
+By analysing the log output and looking for particular
+patterns, the `events` command helps sifting through
+the logs to find when particular events like resources
+changing state or node failure may have occurred.
+
+This can be used to generate a combined list of events
+from all nodes.
+
+Usage:
+...............
+events
+...............
+
+Example:
+...............
+events
+...............
+
+[[cmdhelp_history_exclude,exclude log messages]]
+==== `exclude`
+
+If a log is infested with irrelevant messages, those messages may
+be excluded by specifying a regular expression. The regular
+expressions used are Python extended. This command is additive.
+To drop all regular expressions, use +exclude clear+. Run
+`exclude` only to see the current list of regular expressions.
+Excludes are saved along with the history sessions.
+
+Usage:
+...............
+exclude [<regex>|clear]
+...............
+Example:
+...............
+exclude kernel.*ocfs2
+...............
+
+[[cmdhelp_history_graph,generate a directed graph from the PE file]]
+==== `graph`
+
+Create a graphviz graphical layout from the PE file (the
+transition). Every transition contains the cluster configuration
+which was active at the time. See also <<cmdhelp_configure_graph,generate a directed graph
+from configuration>>.
+
+Usage:
+...............
+graph <pe> [<gtype> [<file> [<img_format>]]]
+
+gtype :: dot
+img_format :: `dot` output format (see the +-T+ option)
+...............
+Example:
+...............
+graph -1
+graph 322 dot clu1.conf.dot
+graph 322 dot clu1.conf.svg svg
+...............
+
+[[cmdhelp_history_info,Cluster information summary]]
+==== `info`
+
+The `info` command provides a summary of the information source, which
+can be either a live cluster snapshot or a previously generated
+report.
+
+Usage:
+...............
+info
+...............
+Example:
+...............
+info
+...............
+
+[[cmdhelp_history_latest,show latest news from the cluster]]
+==== `latest`
+
+The `latest` command shows a bit of recent history, more
+precisely whatever happened since the last cluster change (the
+latest transition). If the transition is running, the shell will
+first wait until it finishes.
+
+Usage:
+...............
+latest
+...............
+Example:
+...............
+latest
+...............
+
+[[cmdhelp_history_limit,limit timeframe to be examined]]
+==== `limit` (`timeframe`)
+
+This command can be used to modify the time span to examine. All
+history commands look at events within a certain time span.
+
+For the `live` source, the default time span is the _last hour_.
+
+There is no time span limit for the `hb_report` source.
+
+The time period is parsed by the `dateutil` python module. It
+covers a wide range of date formats. For instance:
+
+- 3:00 (today at 3am)
+- 15:00 (today at 3pm)
+- 2010/9/1 2pm (September 1st 2010 at 2pm)
+
+For more examples of valid time/date statements, please refer to the
+`python-dateutil` documentation:
+
+- https://dateutil.readthedocs.org/[dateutil.readthedocs.org]
+
+If the dateutil module is not available, then the time is parsed using
+strptime and only the kind as printed by `date(1)` is allowed:
+
+- Tue Sep 15 20:46:27 CEST 2010
+
+Usage:
+...............
+limit [<from_time>] [<to_time>]
+...............
+Examples:
+...............
+limit 10:15
+limit 15h22m 16h
+limit "Sun 5 20:46" "Sun 5 22:00"
+...............
+
+[[cmdhelp_history_log,log content]]
+==== `log`
+
+Show messages logged on one or more nodes. Leaving out a node
+name produces combined logs of all nodes. Messages are sorted by
+time and, if the terminal emulations supports it, displayed in
+different colours depending on the node to allow for easier
+reading.
+
+The sorting key is the timestamp as written by syslog which
+normally has the maximum resolution of one second. Obviously,
+messages generated by events which share the same timestamp may
+not be sorted in the same way as they happened. Such close events
+may actually happen fairly often.
+
+Usage:
+...............
+log [<node> [<node> ...] ]
+...............
+Example:
+...............
+log node-a
+...............
+
+[[cmdhelp_history_node,node events]]
+==== `node`
+
+Show important events that happened on a node. Important events
+are node lost and join, standby and online, and fence. Use either
+node names or extended regular expressions.
+
+Usage:
+...............
+node <node> [<node> ...]
+...............
+Example:
+...............
+node node1
+...............
+
+[[cmdhelp_history_peinputs,list or get PE input files]]
+==== `peinputs`
+
+Every event in the cluster results in generating one or more
+Policy Engine (PE) files. These files describe future motions of
+resources. The files are listed as full paths in the current
+report directory. Add +v+ to also see the creation time stamps.
+
+Usage:
+...............
+peinputs [{<range>|<number>} ...] [v]
+
+range :: <n1>:<n2>
+...............
+Example:
+...............
+peinputs
+peinputs 440:444 446
+peinputs v
+...............
+
+[[cmdhelp_history_refresh,refresh live report]]
+==== `refresh`
+
+This command makes sense only for the +live+ source and makes
+`crm` collect the latest logs and other relevant information from
+the logs. If you want to make a completely new report, specify
++force+.
+
+Usage:
+...............
+refresh [force]
+...............
+
+[[cmdhelp_history_resource,resource events]]
+==== `resource`
+
+Show actions and any failures that happened on all specified
+resources on all nodes. Normally, one gives resource names as
+arguments, but it is also possible to use extended regular
+expressions. Note that neither groups nor clones or master/slave
+names are ever logged. The resource command is going to expand
+all of these appropriately, so that clone instances or resources
+which are part of a group are shown.
+
+Usage:
+...............
+resource <rsc> [<rsc> ...]
+...............
+Example:
+...............
+resource bigdb public_ip
+resource my_.*_db2
+resource ping_clone
+...............
+
+[[cmdhelp_history_session,manage history sessions]]
+==== `session`
+
+Sometimes you may want to get back to examining a particular
+history period or bug report. In order to make that easier, the
+current settings can be saved and later retrieved.
+
+If the current history being examined is coming from a live
+cluster the logs, PE inputs, and other files are saved too,
+because they may disappear from nodes. For the existing reports
+coming from `hb_report`, only the directory location is saved
+(not to waste space).
+
+A history session may also be packed into a tarball which can
+then be sent to support.
+
+Leave out subcommand to see the current session.
+
+Usage:
+...............
+session [{save|load|delete} <name> | pack [<name>] | update | list]
+...............
+Examples:
+...............
+session save bnc966622
+session load rsclost-2
+session list
+...............
+
+[[cmdhelp_history_setnodes,set the list of cluster nodes]]
+==== `setnodes`
+
+In case the host this program runs on is not part of the cluster,
+it is necessary to set the list of nodes.
+
+Usage:
+...............
+setnodes node <node> [<node> ...]
+...............
+Example:
+...............
+setnodes node_a node_b
+...............
+
+[[cmdhelp_history_show,show status or configuration of the PE input file]]
+==== `show`
+
+Every transition is saved as a PE file. Use this command to
+render that PE file either as configuration or status. The
+configuration output is the same as `crm configure show`.
+
+Usage:
+...............
+show <pe> [status]
+
+pe :: <number>|<index>|<file>|live
+...............
+Examples:
+...............
+show 2066
+show pe-input-2080.bz2 status
+...............
+
+[[cmdhelp_history_source,set source to be examined]]
+==== `source`
+
+Events to be examined can come from the current cluster or from a
+`hb_report` report. This command sets the source. `source live`
+sets source to the running cluster and system logs. If no source
+is specified, the current source information is printed.
+
+In case a report source is specified as a file reference, the file
+is going to be unpacked in place where it resides. This directory
+is not removed on exit.
+
+Usage:
+...............
+source [<dir>|<file>|live]
+...............
+Examples:
+...............
+source live
+source /tmp/customer_case_22.tar.bz2
+source /tmp/customer_case_22
+source
+...............
+
+[[cmdhelp_history_transition,show transition]]
+==== `transition`
+
+This command will print actions planned by the PE and run
+graphviz (`dotty`) to display a graphical representation of the
+transition. Of course, for the latter an X11 session is required.
+This command invokes `ptest(8)` in background.
+
+The +showdot+ subcommand runs graphviz (`dotty`) to display a
+graphical representation of the +.dot+ file which has been
+included in the report. Essentially, it shows the calculation
+produced by `pengine` which is installed on the node where the
+report was produced. In optimal case this output should not
+differ from the one produced by the locally installed `pengine`.
+
+The `log` subcommand shows the full log for the duration of the
+transition.
+
+A transition can also be saved to a CIB shadow for further
+analysis or use with `cib` or `configure` commands (use the
+`save` subcommand). The shadow file name defaults to the name of
+the PE input file.
+
+If the PE input file number is not provided, it defaults to the
+last one, i.e. the last transition. The last transition can also
+be referenced with number 0. If the number is negative, then the
+corresponding transition relative to the last one is chosen.
+
+If there are warning and error PE input files or different nodes
+were the DC in the observed timeframe, it may happen that PE
+input file numbers collide. In that case provide some unique part
+of the path to the file.
+
+After the `ptest` output, logs about events that happened during
+the transition are printed.
+
+The `tags` subcommand scans the logs for the transition and return a
+list of key events during that transition. For example, the tag
++error+ will be returned if there are any errors logged during the
+transition.
+
+Usage:
+...............
+transition [<number>|<index>|<file>] [nograph] [v...] [scores] [actions] [utilization]
+transition showdot [<number>|<index>|<file>]
+transition log [<number>|<index>|<file>]
+transition save [<number>|<index>|<file> [name]]
+transition tags [<number>|<index>|<file>]
+...............
+Examples:
+...............
+transition
+transition 444
+transition -1
+transition pe-error-3.bz2
+transition node-a/pengine/pe-input-2.bz2
+transition showdot 444
+transition log
+transition save 0 enigma-22
+...............
+
+[[cmdhelp_history_transitions,List transitions]]
+==== `transitions`
+
+A transition represents a change in cluster configuration or
+state. This command lists the transitions in the current timeframe.
+
+Usage:
+...............
+transitions
+...............
+Example:
+...............
+transitions
+...............
+
+
+[[cmdhelp_history_wdiff,cluster states/transitions difference]]
+==== `wdiff`
+
+A transition represents a change in cluster configuration or
+state. Use `wdiff` to see what has changed between two
+transitions as word differences on a line-by-line basis.
+
+If you want to specify the current cluster configuration and
+status, use the string +live+.
+
+Normally, the first transition specified should be the one which
+is older, but we are not going to enforce that.
+
+Note that a single configuration update may result in more than
+one transition.
+
+Usage:
+...............
+wdiff <pe> <pe> [status]
+
+pe :: <number>|<index>|<file>|live
+...............
+Examples:
+...............
+wdiff 2066 2067
+wdiff pe-input-2080.bz2 live status
+...............
+
+[[cmdhelp_root_report,Create cluster status report]]
+=== `report`
+
+Interface to a tool for creating a cluster report. A report is an
+archive containing log files, configuration files, system information
+and other relevant data for a given time period. This is a useful tool
+for collecting data to attach to bug reports, or for detecting the
+root cause of errors resulting in resource failover, for example.
+
+See `crmsh_hb_report(8)` for more details on arguments,
+or call `crm report -h`
+
+Usage:
+...............
+report -f {time|"cts:"testnum} [-t time] [-u user] [-l file]
+ [-n nodes] [-E files] [-p patt] [-L patt] [-e prog]
+ [-MSDZAVsvhd] [dest]
+...............
+
+Examples:
+...............
+report -f 2pm report_1
+report -f "2007/9/5 12:30" -t "2007/9/5 14:00" report_2
+report -f 1:00 -t 3:00 -l /var/log/cluster/ha-debug report_3
+report -f "09sep07 2:00" -u hbadmin report_4
+report -f 18:00 -p "usern.*" -p "admin.*" report_5
+report -f cts:133 ctstest_133
+...............
+
+=== `end` (`cd`, `up`)
+
+The `end` command ends the current level and the user moves to
+the parent level. This command is available everywhere.
+
+Usage:
+...............
+end
+...............
+
+=== `help`
+
+The `help` command prints help for the current level or for the
+specified topic (command). This command is available everywhere.
+
+Usage:
+...............
+help [<topic>]
+...............
+
+=== `quit` (`exit`, `bye`)
+
+Leave the program.
+
+BUGS
+----
+Even though all sensible configurations (and most of those that
+are not) are going to be supported by the crm shell, I suspect
+that it may still happen that certain XML constructs may confuse
+the tool. When that happens, please file a bug report.
+
+The crm shell will not try to update the objects it does not
+understand. Of course, it is always possible to edit such objects
+in the XML format.
+
+AUTHORS
+-------
+Dejan Muhamedagic, <dejan@suse.de>
+Kristoffer Gronlund <kgronlund@suse.com>
+and many OTHERS
+
+SEE ALSO
+--------
+crm_resource(8), crm_attribute(8), crm_mon(8), cib_shadow(8),
+ptest(8), dotty(1), crm_simulate(8), cibadmin(8)
+
+
+COPYING
+-------
+Copyright \(C) 2008-2013 Dejan Muhamedagic.
+Copyright \(C) 2013 Kristoffer Gronlund.
+
+Free use of this software is granted under the terms of the GNU General Public License (GPL).
+
+//////////////////////
+ vim:ts=4:sw=4:expandtab:
+//////////////////////
diff --git a/doc/website-v1/man-3.adoc b/doc/website-v1/man-3.adoc
new file mode 100644
index 0000000..e4411cc
--- /dev/null
+++ b/doc/website-v1/man-3.adoc
@@ -0,0 +1,5309 @@
+:man source: crm
+:man version: 2.3.0
+:man manual: crmsh documentation
+
+crm(8)
+======
+
+NAME
+----
+crm - Pacemaker command line interface for configuration and management
+
+
+SYNOPSIS
+--------
+*crm* [OPTIONS] [SUBCOMMAND ARGS...]
+
+
+[[topics_Description,Program description]]
+DESCRIPTION
+-----------
+The `crm` shell is a command-line based cluster configuration and
+management tool. Its goal is to assist as much as possible with the
+configuration and maintenance of Pacemaker-based High Availability
+clusters.
+
+For more information on Pacemaker itself, see http://clusterlabs.org/.
+
+`crm` works both as a command-line tool to be called directly from the
+system shell, and as an interactive shell with extensive tab
+completion and help.
+
+The primary focus of the `crm` shell is to provide a simplified and
+consistent interface to Pacemaker, but it also provides tools for
+managing the creation and configuration of High Availability clusters
+from scratch. To learn more about this aspect of `crm`, see the
+`cluster` section below.
+
+The `crm` shell can be used to manage every aspect of configuring and
+maintaining a cluster. It provides a simplified line-based syntax on
+top of the XML configuration format used by Pacemaker, commands for
+starting and stopping resources, tools for exploring the history of a
+cluster including log scraping and a set of cluster scripts useful for
+automating the setup and installation of services on the cluster
+nodes.
+
+The `crm` shell is line oriented: every command must start and finish
+on the same line. It is possible to use a continuation character (+\+)
+to write one command in two or more lines. The continuation character
+is commonly used when displaying configurations.
+
+[[topics_CommandLine,Command line options]]
+OPTIONS
+-------
+*-f, --file*='FILE'::
+ Load commands from the given file. If a dash +-+ is used in place
+ of a file name, `crm` will read commands from the shell standard
+ input (`stdin`).
+
+*-c, --cib*='CIB'::
+ Start the session using the given shadow CIB file.
+ Equivalent to +cib use <CIB>+.
+
+*-D, --display=*'OUTPUT_TYPE'::
+ Choose one of the output options: +plain+, +color-always+, +color+,
+ or +uppercase+. The default is +color+ if the terminal emulation
+ supports colors. Otherwise, +plain+ is used.
+
+*-F, --force*::
+ Make `crm` proceed with applying changes where it would normally
+ ask the user to confirm before proceeding. This option is mainly
+ useful in scripts, and should be used with care.
+
+*-w, --wait*::
+ Make `crm` wait for the cluster transition to finish (for the
+ changes to take effect) after each processed line.
+
+*-H, --history*='DIR|FILE|SESSION'::
+ A directory or file containing a cluster report to load
+ into the `history` commands, or the name of a previously
+ saved history session.
+
+*-h, --help*::
+ Print help page.
+
+*--version*::
+ Print crmsh version and build information (Mercurial Hg changeset
+ hash).
+
+*-d, --debug*::
+ Print verbose debugging information.
+
+*-R, --regression-tests*::
+ Enables extra verbose trace logging used by the regression
+ tests. Logs all external calls made by crmsh.
+
+*--scriptdir*='DIR'::
+ Extra directory where crm looks for cluster scripts, or a list of
+ directories separated by semi-colons (e.g. +/dir1;/dir2;etc.+).
+
+*-o, --opt*='OPTION=VALUE'::
+ Set crmsh option temporarily. If the options are saved using
+ +options save+ then the value passed here will also be saved.
+ Multiple options can be set by using +-o+ multiple times.
+
+[[topics_Introduction,Introduction]]
+== Introduction
+
+This section of the user guide covers general topics about the user
+interface and describes some of the features of `crmsh` in detail.
+
+[[topics_Introduction_Interface,User interface]]
+=== User interface
+
+The main purpose of `crmsh` is to provide a simple yet powerful
+interface to the cluster stack. There are two main modes of operation
+with the user interface of `crmsh`:
+
+* Command line (single-shot) use - Use `crm` as a regular UNIX command
+ from your usual shell. `crm` has full bash completion built in, so
+ using it in this manner should be as comfortable and familiar as
+ using any other command-line tool.
+
+* Interactive mode - By calling `crm` without arguments, or by calling
+ it with only a sublevel as argument, `crm` enters the interactive
+ mode. In this mode, it acts as its own command shell, which
+ remembers which sublevel you are currently in and allows for rapid
+ and convenient execution of multiple commands within the same
+ sublevel. This mode also has full tab completion, as well as
+ built-in interactive help and syntax highlighting.
+
+Here are a few examples of using `crm` both as a command-line tool and
+as an interactive shell:
+
+.Command line (one-shot) use:
+........
+# crm resource stop www_app
+........
+
+.Interactive use:
+........
+# crm
+crm(live)# resource
+crm(live)resource# unmanage tetris_1
+crm(live)resource# up
+crm(live)# node standby node4
+........
+
+.Cluster configuration:
+........
+# crm configure<<EOF
+ #
+ # resources
+ #
+ primitive disk0 iscsi \
+ params portal=192.168.2.108:3260 target=iqn.2008-07.com.suse:disk0
+ primitive fs0 Filesystem \
+ params device=/dev/disk/by-label/disk0 directory=/disk0 fstype=ext3
+ primitive internal_ip IPaddr params ip=192.168.1.101
+ primitive apache apache \
+ params configfile=/disk0/etc/apache2/site0.conf
+ primitive apcfence stonith:apcsmart \
+ params ttydev=/dev/ttyS0 hostlist="node1 node2" \
+ op start timeout=60s
+ primitive pingd pingd \
+ params name=pingd dampen=5s multiplier=100 host_list="r1 r2"
+ #
+ # monitor apache and the UPS
+ #
+ monitor apache 60s:30s
+ monitor apcfence 120m:60s
+ #
+ # cluster layout
+ #
+ group internal_www \
+ disk0 fs0 internal_ip apache
+ clone fence apcfence \
+ meta globally-unique=false clone-max=2 clone-node-max=1
+ clone conn pingd \
+ meta globally-unique=false clone-max=2 clone-node-max=1
+ location node_pref internal_www \
+ rule 50: #uname eq node1 \
+ rule pingd: defined pingd
+ #
+ # cluster properties
+ #
+ property stonith-enabled=true
+ commit
+EOF
+........
+
+The `crm` interface is hierarchical, with commands organized into
+separate levels by functionality. To list the available levels and
+commands, either execute +help <level>+, or, if at the top level of
+the shell, simply typing `help` will provide an overview of all
+available levels and commands.
+
+The +(live)+ string in the `crm` prompt signifies that the current CIB
+in use is the cluster live configuration. It is also possible to
+work with so-called <<topics_Features_Shadows,shadow CIBs>>. These are separate, inactive
+configurations stored in files, that can be applied and thereby
+replace the live configuration at any time.
+
+[[topics_Introduction_Completion,Tab completion]]
+=== Tab completion
+
+The `crm` makes extensive use of tab completion. The completion
+is both static (i.e. for `crm` commands) and dynamic. The latter
+takes into account the current status of the cluster or
+information from installed resource agents. Sometimes, completion
+may also be used to get short help on resource parameters. Here
+are a few examples:
+
+...............
+crm(live)resource# <TAB><TAB>
+bye failcount move restart unmigrate
+cd help param show unmove
+cleanup list promote start up
+demote manage quit status utilization
+end meta refresh stop
+exit migrate reprobe unmanage
+
+crm(live)configure# primitive fence-1 <TAB><TAB>
+heartbeat: lsb: ocf: stonith:
+
+crm(live)configure# primitive fence-1 stonith:<TAB><TAB>
+apcmaster external/ippower9258 fence_legacy
+apcmastersnmp external/kdumpcheck ibmhmc
+apcsmart external/libvirt ipmilan
+
+crm(live)configure# primitive fence-1 stonith:ipmilan params <TAB><TAB>
+auth= hostname= ipaddr= login= password= port= priv=
+
+crm(live)configure# primitive fence-1 stonith:ipmilan params auth=<TAB><TAB>
+auth* (string)
+ The authorization type of the IPMI session ("none", "straight", "md2", or "md5")
+...............
+
+`crmsh` also comes with bash completion usable directly from the
+system shell. This should be installed automatically with the command
+itself.
+
+[[topics_Introduction_Shorthand,Shorthand syntax]]
+=== Shorthand syntax
+
+When using the `crm` shell to manage clusters, you will end up typing
+a lot of commands many times over. Clear command names like
++configure+ help in understanding and learning to use the cluster
+shell, but is easy to misspell and is tedious to type repeatedly. The
+interactive mode and tab completion both help with this, but the `crm`
+shell also has the ability to understand a variety of shorthand
+aliases for all of the commands.
+
+For example, instead of typing `crm status`, you can type `crm st` or
+`crm stat`. Instead of `crm configure` you can type `crm cfg` or even
+`crm cf`. `crm resource` can be shorted as `crm rsc`, and so on.
+
+The exact list of accepted aliases is too long to print in full, but
+experimentation and typos should help in discovering more of them.
+
+[[topics_Features,Features]]
+== Features
+
+The feature set of crmsh covers a wide range of functionality, and
+understanding how and when to use the various features of the shell
+can be difficult. This section of the guide describes some of the
+features and use cases of `crmsh` in more depth. The intention is to
+provide a deeper understanding of these features, but also to serve as
+a guide to using them.
+
+[[topics_Features_Shadows,Shadow CIB usage]]
+=== Shadow CIB usage
+
+A Shadow CIB is a normal cluster configuration stored in a file.
+They may be manipulated in much the same way as the _live_ CIB, with
+the key difference that changes to a shadow CIB have no effect on the
+actual cluster resources. An administrator may choose to apply any of
+them to the cluster, thus replacing the running configuration with the
+one found in the shadow CIB.
+
+The `crm` prompt always contains the name of the configuration which
+is currently in use, or the string _live_ if using the live cluster
+configuration.
+
+When editing the configuration in the `configure` level, no changes
+are actually applied until the `commit` command is executed. It is
+possible to start editing a configuration as usual, but instead of
+committing the changes to the active CIB, save them to a shadow CIB.
+
+The following example `configure` session demonstrates how this can be
+done:
+...............
+crm(live)configure# cib new test-2
+INFO: test-2 shadow CIB created
+crm(test-2)configure# commit
+...............
+
+[[topics_Features_Checks,Configuration semantic checks]]
+=== Configuration semantic checks
+
+Resource definitions may be checked against the meta-data
+provided with the resource agents. These checks are currently
+carried out:
+
+- are required parameters set
+- existence of defined parameters
+- timeout values for operations
+
+The parameter checks are obvious and need no further explanation.
+Failures in these checks are treated as configuration errors.
+
+The timeouts for operations should be at least as long as those
+recommended in the meta-data. Too short timeout values are a
+common mistake in cluster configurations and, even worse, they
+often slip through if cluster testing was not thorough. Though
+operation timeouts issues are treated as warnings, make sure that
+the timeouts are usable in your environment. Note also that the
+values given are just _advisory minimum_---your resources may
+require longer timeouts.
+
+User may tune the frequency of checks and the treatment of errors
+by the <<cmdhelp_options_check-frequency,`check-frequency`>> and
+<<cmdhelp_options_check-mode,`check-mode`>> preferences.
+
+Note that if the +check-frequency+ is set to +always+ and the
++check-mode+ to +strict+, errors are not tolerated and such
+configuration cannot be saved.
+
+[[topics_Features_Templates,Configuration templates]]
+=== Configuration templates
+
+.Deprecation note
+****************************
+Configuration templates have been deprecated in favor of the more
+capable `cluster scripts`. To learn how to use cluster scripts, see
+the dedicated documentation on the `crmsh` website at
+http://crmsh.github.io/, or in the <<cmdhelp_script,Script section>>.
+****************************
+
+Configuration templates are ready made configurations created by
+cluster experts. They are designed in such a way so that users
+may generate valid cluster configurations with minimum effort.
+If you are new to Pacemaker, templates may be the best way to
+start.
+
+We will show here how to create a simple yet functional Apache
+configuration:
+...............
+# crm configure
+crm(live)configure# template
+crm(live)configure template# list templates
+apache filesystem virtual-ip
+crm(live)configure template# new web <TAB><TAB>
+apache filesystem virtual-ip
+crm(live)configure template# new web apache
+INFO: pulling in template apache
+INFO: pulling in template virtual-ip
+crm(live)configure template# list
+web2-d web2 vip2 web3 vip web
+...............
+
+We enter the `template` level from `configure`. Use the `list`
+command to show templates available on the system. The `new`
+command creates a configuration from the +apache+ template. You
+can use tab completion to pick templates. Note that the apache
+template depends on a virtual IP address which is automatically
+pulled along. The `list` command shows the just created +web+
+configuration, among other configurations (I hope that you,
+unlike me, will use more sensible and descriptive names).
+
+The `show` command, which displays the resulting configuration,
+may be used to get an idea about the minimum required changes
+which have to be done. All +ERROR+ messages show the line numbers
+in which the respective parameters are to be defined:
+...............
+crm(live)configure template# show
+ERROR: 23: required parameter ip not set
+ERROR: 61: required parameter id not set
+ERROR: 65: required parameter configfile not set
+crm(live)configure template# edit
+...............
+
+The `edit` command invokes the preferred text editor with the
++web+ configuration. At the top of the file, the user is advised
+how to make changes. A good template should require from the user
+to specify only parameters. For example, the +web+ configuration
+we created above has the following required and optional
+parameters (all parameter lines start with +%%+):
+...............
+$ grep -n ^%% ~/.crmconf/web
+23:%% ip
+31:%% netmask
+35:%% lvs_support
+61:%% id
+65:%% configfile
+71:%% options
+76:%% envfiles
+...............
+
+These lines are the only ones that should be modified. Simply
+append the parameter value at the end of the line. For instance,
+after editing this template, the result could look like this (we
+used tabs instead of spaces to make the values stand out):
+...............
+$ grep -n ^%% ~/.crmconf/web
+23:%% ip 192.168.1.101
+31:%% netmask
+35:%% lvs_support
+61:%% id websvc
+65:%% configfile /etc/apache2/httpd.conf
+71:%% options
+76:%% envfiles
+...............
+
+As you can see, the parameter line format is very simple:
+...............
+%% <name> <value>
+...............
+
+After editing the file, use `show` again to display the
+configuration:
+...............
+crm(live)configure template# show
+primitive virtual-ip IPaddr \
+ params ip=192.168.1.101
+primitive apache apache \
+ params configfile="/etc/apache2/httpd.conf"
+monitor apache 120s:60s
+group websvc \
+ apache virtual-ip
+...............
+
+The target resource of the apache template is a group which we
+named +websvc+ in this sample session.
+
+This configuration looks exactly as you could type it at the
+`configure` level. The point of templates is to save you some
+typing. It is important, however, to understand the configuration
+produced.
+
+Finally, the configuration may be applied to the current
+crm configuration (note how the configuration changed slightly,
+though it is still equivalent, after being digested at the
+`configure` level):
+...............
+crm(live)configure template# apply
+crm(live)configure template# cd ..
+crm(live)configure# show
+node xen-b
+node xen-c
+primitive apache apache \
+ params configfile="/etc/apache2/httpd.conf" \
+ op monitor interval=120s timeout=60s
+primitive virtual-ip IPaddr \
+ params ip=192.168.1.101
+group websvc apache virtual-ip
+...............
+
+Note that this still does not commit the configuration to the CIB
+which is used in the shell, either the running one (+live+) or
+some shadow CIB. For that you still need to execute the `commit`
+command.
+
+To complete our example, we should also define the preferred node
+to run the service:
+
+...............
+crm(live)configure# location websvc-pref websvc 100: xen-b
+...............
+
+If you are not happy with some resource names which are provided
+by default, you can rename them now:
+
+...............
+crm(live)configure# rename virtual-ip intranet-ip
+crm(live)configure# show
+node xen-b
+node xen-c
+primitive apache apache \
+ params configfile="/etc/apache2/httpd.conf" \
+ op monitor interval=120s timeout=60s
+primitive intranet-ip IPaddr \
+ params ip=192.168.1.101
+group websvc apache intranet-ip
+location websvc-pref websvc 100: xen-b
+...............
+
+To summarize, working with templates typically consists of the
+following steps:
+
+- `new`: create a new configuration from templates
+- `edit`: define parameters, at least the required ones
+- `show`: see if the configuration is valid
+- `apply`: apply the configuration to the `configure` level
+
+[[topics_Features_Testing,Resource testing]]
+=== Resource testing
+
+The amount of detail in a cluster makes all configurations prone
+to errors. By far the largest number of issues in a cluster is
+due to bad resource configuration. The shell can help quickly
+diagnose such problems. And considerably reduce your keyboard
+wear.
+
+Let's say that we entered the following configuration:
+...............
+node xen-b
+node xen-c
+node xen-d
+primitive fencer stonith:external/libvirt \
+ params hypervisor_uri="qemu+tcp://10.2.13.1/system" \
+ hostlist="xen-b xen-c xen-d" \
+ op monitor interval=2h
+primitive svc Xinetd \
+ params service=systat \
+ op monitor interval=30s
+primitive intranet-ip IPaddr2 \
+ params ip=10.2.13.100 \
+ op monitor interval=30s
+primitive apache apache \
+ params configfile="/etc/apache2/httpd.conf" \
+ op monitor interval=120s timeout=60s
+group websvc apache intranet-ip
+location websvc-pref websvc 100: xen-b
+...............
+
+Before typing `commit` to submit the configuration to the cib we
+can make sure that all resources are usable on all nodes:
+...............
+crm(live)configure# rsctest websvc svc fencer
+...............
+
+It is important that resources being tested are not running on
+any nodes. Otherwise, the `rsctest` command will refuse to do
+anything. Of course, if the current configuration resides in a
+CIB shadow, then a `commit` is irrelevant. The point being that
+resources are not running on any node.
+
+.Note on stopping all resources
+****************************
+Alternatively to not committing a configuration, it is also
+possible to tell Pacemaker not to start any resources:
+
+...............
+crm(live)configure# property stop-all-resources=yes
+...............
+Almost none---resources of class stonith are still started. But
+shell is not as strict when it comes to stonith resources.
+****************************
+
+Order of resources is significant insofar that a resource depends
+on all resources to its left. In most configurations, it's
+probably practical to test resources in several runs, based on
+their dependencies.
+
+Apart from groups, `crm` does not interpret constraints and
+therefore knows nothing about resource dependencies. It also
+doesn't know if a resource can run on a node at all in case of an
+asymmetric cluster. It is up to the user to specify a list of
+eligible nodes if a resource is not meant to run on every node.
+
+[[topics_Features_Security,Access Control Lists (ACL)]]
+=== Access Control Lists (ACL)
+
+.Note on ACLs in Pacemaker 1.1.12
+****************************
+The support for ACLs has been revised in Pacemaker version 1.1.12 and
+up. Depending on which version you are using, the information in this
+section may no longer be accurate. Look for the `acl_target`
+configuration element for more details on the new syntax.
+****************************
+
+By default, the users from the +haclient+ group have full access
+to the cluster (or, more precisely, to the CIB). Access control
+lists allow for finer access control to the cluster.
+
+Access control lists consist of an ordered set of access rules.
+Each rule allows read or write access or denies access
+completely. Rules are typically combined to produce a specific
+role. Then, users may be assigned a role.
+
+For instance, this is a role which defines a set of rules
+allowing management of a single resource:
+
+...............
+role bigdb_admin \
+ write meta:bigdb:target-role \
+ write meta:bigdb:is-managed \
+ write location:bigdb \
+ read ref:bigdb
+...............
+
+The first two rules allow modifying the +target-role+ and
++is-managed+ meta attributes which effectively enables users in
+this role to stop/start and manage/unmanage the resource. The
+constraints write access rule allows moving the resource around.
+Finally, the user is granted read access to the resource
+definition.
+
+For proper operation of all Pacemaker programs, it is advisable
+to add the following role to all users:
+
+...............
+role read_all \
+ read cib
+...............
+
+For finer grained read access try with the rules listed in the
+following role:
+
+...............
+role basic_read \
+ read node attribute:uname \
+ read node attribute:type \
+ read property \
+ read status
+...............
+
+It is however possible that some Pacemaker programs (e.g.
+`ptest`) may not function correctly if the whole CIB is not
+readable.
+
+Some of the ACL rules in the examples above are expanded by the
+shell to XPath specifications. For instance,
++meta:bigdb:target-role+ expands to:
+
+........
+//primitive[@id='bigdb']/meta_attributes/nvpair[@name='target-role']
+........
+
+You can see the expansion by showing XML:
+
+...............
+crm(live) configure# show xml bigdb_admin
+...
+<acls>
+ <acl_role id="bigdb_admin">
+ <write id="bigdb_admin-write"
+ xpath="//primitive[@id='bigdb']/meta_attributes/nvpair[@name='target-role']"/>
+...............
+
+Many different XPath expressions can have equal meaning. For
+instance, the following two are equal, but only the first one is
+going to be recognized as shortcut:
+
+...............
+//primitive[@id='bigdb']/meta_attributes/nvpair[@name='target-role']
+//resources/primitive[@id='bigdb']/meta_attributes/nvpair[@name='target-role']
+...............
+
+XPath is a powerful language, but you should try to keep your ACL
+xpaths simple and the builtin shortcuts should be used whenever
+possible.
+
+[[topics_Features_Resourcesets,Syntax: Resource sets]]
+=== Syntax: Resource sets
+
+Using resource sets can be a bit confusing unless one knows the
+details of the implementation in Pacemaker as well as how to interpret
+the syntax provided by `crmsh`.
+
+Three different types of resource sets are provided by `crmsh`, and
+each one implies different values for the two resource set attributes,
++sequential+ and +require-all+.
+
++sequential+::
+ If false, the resources in the set do not depend on each other
+ internally. Setting +sequential+ to +true+ implies a strict order of
+ dependency within the set.
+
++require-all+::
+ If false, only one resource in the set is required to fulfil the
+ requirements of the set. The set of A, B and C with +require-all+
+ set to +false+ is be read as "A OR B OR C" when its dependencies
+ are resolved.
+
+The three types of resource sets modify the attributes in the
+following way:
+
+1. Implicit sets (no brackets). +sequential=true+, +require-all=true+
+2. Parenthesis set (+(+ ... +)+). +sequential=false+, +require-all=true+
+3. Bracket set (+[+ ... +]+). +sequential=false+, +require-all=false+
+
+To create a set with the properties +sequential=true+ and
++require-all=false+, explicitly set +sequential+ in a bracketed set,
++[ A B C sequential=true ]+.
+
+To create multiple sets with both +sequential+ and +require-all+ set to
+true, explicitly set +sequential+ in a parenthesis set:
++A B ( C D sequential=true )+.
+
+[[topics_Features_AttributeListReferences,Syntax: Attribute list references]]
+=== Syntax: Attribute list references
+
+Attribute lists are used to set attributes and parameters for
+resources, constraints and property definitions. For example, to set
+the virtual IP used by an +IPAddr2+ resource the attribute +ip+ can be
+set in an attribute list for that resource.
+
+Attribute lists can have identifiers that name them, and other
+resources can reuse the same attribute list by referring to that name
+using an +$id-ref+. For example, the following statement defines a
+simple dummy resource with an attribute list which sets the parameter
++state+ to the value 1 and sets the identifier for the attribute list
+to +on-state+:
+
+..............
+primitive dummy-1 Dummy params $id=on-state state=1
+..............
+
+To refer to this attribute list from a different resource, refer to
+the +on-state+ name using an id-ref:
+
+..............
+primitive dummy-2 Dummy params $id-ref=on-state
+..............
+
+The resource +dummy-2+ will now also have the parameter +state+ set to the value 1.
+
+[[topics_Features_AttributeReferences,Syntax: Attribute references]]
+=== Syntax: Attribute references
+
+In some cases, referencing complete attribute lists is too
+coarse-grained, for example if two different parameters with different
+names should have the same value set. Instead of having to copy the
+value in multiple places, it is possible to create references to
+individual attributes in attribute lists.
+
+To name an attribute in order to be able to refer to it later, prefix
+the attribute name with a +$+ character (as seen above with the
+special names +$id+ and +$id-ref+:
+
+............
+primitive dummy-1 Dummy params $state=1
+............
+
+The identifier +state+ can now be used to refer to this attribute from other
+primitives, using the +@<id>+ syntax:
+
+............
+primitive dummy-2 Dummy params @state
+............
+
+In some cases, using the attribute name as the identifier doesn't work
+due to name clashes. In this case, the syntax +$<id>:<name>=<value>+
+can be used to give the attribute a different identifier:
+
+............
+primitive dummy-1 params $dummy-state-on:state=1
+primitive dummy-2 params @dummy-state-on
+............
+
+There is also the possibility that two resources both use the same
+attribute value but with different names. For example, a web server
+may have a parameter +server_ip+ for setting the IP address where it
+listens for incoming requests, and a virtual IP resource may have a
+parameter called +ip+ which sets the IP address it creates. To
+configure these two resources with an IP without repeating the value,
+the reference can be given a name using the syntax +@<id>:<name>+.
+
+Example:
+............
+primitive virtual-ip IPaddr2 params $vip:ip=192.168.1.100
+primitive webserver apache params @vip:server_ip
+............
+
+[[topics_Syntax_RuleExpressions,Syntax: Rule expressions]]
+=== Syntax: Rule expressions
+
+Many of the configuration commands in `crmsh` now support the use of
+_rule expressions_, which can influence what attributes apply to a
+resource or under which conditions a constraint is applied, depending
+on changing conditions like date, time, the value of attributes and
+more.
+
+Here is an example of a simple rule expression used to apply a
+a different resource parameter on the node named `node1`:
+
+..............
+primitive my_resource Special \
+ params 2: rule #uname eq node1 interface=eth1 \
+ params 1: interface=eth0
+..............
+
+This primitive resource has two lists of parameters with descending
+priority. The parameter list with the highest priority is applied
+first, but only if the rule expressions for that parameter list all
+apply. In this case, the rule `#uname eq node1` limits the parameter
+list so that it is only applied on `node1`.
+
+Note that rule expressions are not terminated and are immediately
+followed by the data to which the rule is applied. In this case, the
+name-value pair `interface=eth1`.
+
+Rule expressions can contain multiple expressions connected using the
+boolean operator `or` and `and`. The full syntax for rule expressions
+is listed below.
+
+..............
+rules ::
+ rule [id_spec] [$role=<role>] <score>: <expression>
+ [rule [id_spec] [$role=<role>] <score>: <expression> ...]
+
+id_spec :: $id=<id> | $id-ref=<id>
+score :: <number> | <attribute> | [-]inf
+expression :: <simple_exp> [<bool_op> <simple_exp> ...]
+bool_op :: or | and
+simple_exp :: <attribute> [type:]<binary_op> <value>
+ | <unary_op> <attribute>
+ | date <date_expr>
+type :: <string> | <version> | <number>
+binary_op :: lt | gt | lte | gte | eq | ne
+unary_op :: defined | not_defined
+
+date_expr :: lt <end>
+ | gt <start>
+ | in start=<start> end=<end>
+ | in start=<start> <duration>
+ | spec <date_spec>
+duration|date_spec ::
+ hours=<value>
+ | monthdays=<value>
+ | weekdays=<value>
+ | yearsdays=<value>
+ | months=<value>
+ | weeks=<value>
+ | years=<value>
+ | weekyears=<value>
+ | moon=<value>
+..............
+
+[[topics_Reference,Command reference]]
+== Command reference
+
+The commands are structured to be compatible with the shell command
+line. Sometimes, the underlying Pacemaker grammar uses characters that
+have special meaning in bash, that will need to be quoted. This
+includes the hash or pound sign (`#`), single and double quotes, and
+any significant whitespace.
+
+Whitespace is also significant when assigning values, meaning that
++key=value+ is different from +key = value+.
+
+Commands can be referenced using short-hand as long as the short-hand
+is unique. This can be either a prefix of the command name or a prefix
+string of characters found in the name.
+
+For example, +status+ can be abbreviated as +st+ or +su+, and
++configure+ as +conf+ or +cfg+.
+
+The syntax for the commands is given below in an informal, BNF-like
+grammar.
+
+* `<value>` denotes a string.
+* `[value]` means that the construct is optional.
+* The ellipsis (`...`) signifies that the previous construct may be
+ repeated.
+* `first|second` means either first or second.
+* The rest are literals (strings, `:`, `=`).
+
+[[cmdhelp_root_status,Cluster status]]
+=== `status`
+
+Show cluster status. The status is displayed by `crm_mon`. Supply
+additional arguments for more information or different format.
+See `crm_mon(8)` for more details.
+
+Example:
+...............
+status
+status simple
+status full
+...............
+
+Usage:
+...............
+status [<option> ...]
+
+option :: full
+ | bynode
+ | inactive
+ | ops
+ | timing
+ | failcounts
+ | verbose
+ | quiet
+ | html
+ | xml
+ | simple
+ | tickets
+ | noheaders
+ | detail
+ | brief
+...............
+
+[[cmdhelp_root_verify,Verify cluster status]]
+=== `verify`
+
+Performs basic checks for the cluster configuration and
+current status, reporting potential issues.
+
+See `crm_verify(8)` and `crm_simulate(8)` for more details.
+
+Example:
+...............
+verify
+verify scores
+...............
+
+Usage:
+...............
+verify [scores]
+...............
+
+
+[[cmdhelp_cluster,Cluster setup and management]]
+=== `cluster` - Cluster setup and management
+
+Whole-cluster configuration management with High Availability
+awareness.
+
+The commands on the cluster level allows configuration and
+modification of the underlying cluster infrastructure, and also
+supplies tools to do whole-cluster systems management.
+
+These commands enable easy installation and maintenance of a HA
+cluster, by providing support for package installation, configuration
+of the cluster messaging layer, file system setup and more.
+
+[[cmdhelp_cluster_add,Add a new node to the cluster]]
+==== `add`
+
+Add a new node to the cluster. The new node will be
+configured as a cluster member.
+
+Options:
+
+*-y, --yes*::
+ Answer "yes" to all prompts (use with caution)
+
+Usage:
+...............
+add [options] [<node> ...]
+...............
+
+[[cmdhelp_cluster_copy,Copy file to other cluster nodes]]
+==== `copy`
+
+Copy file to other cluster nodes.
+
+Copies the given file to all other nodes unless given a
+list of nodes to copy to as argument.
+
+Usage:
+...............
+copy <filename> [nodes ...]
+...............
+
+Example:
+...............
+copy /etc/motd
+...............
+
+[[cmdhelp_cluster_diff,Diff file across cluster]]
+==== `diff`
+
+Displays the difference, if any, between a given file
+on different nodes. If the second argument is `--checksum`,
+a checksum of the file will be calculated and displayed for
+each node.
+
+Usage:
+...............
+diff <file> [--checksum] [nodes...]
+...............
+
+Example:
+...............
+diff /etc/crm/crm.conf node2
+diff /etc/resolv.conf --checksum
+...............
+
+[[cmdhelp_cluster_geo_init,Configure cluster as geo cluster]]
+==== `geo-init`
+
+Create a new geo cluster with the current cluster as the
+first member. Pass the complete geo cluster topology as
+arguments to this command, and then use `geo-join` and
+`geo-init-arbitrator` to add the remaining members to
+the geo cluster.
+
+Options:
+
+*-q, --quiet*::
+ Be quiet (don't describe what's happening, just do it)
+
+*-y, --yes*::
+ Answer "yes" to all prompts (use with caution)
+
+*--arbitrator=IP*::
+ IP address of geo cluster arbitrator
+
+*--clusters=DESC*::
+ Cluster description (see details below)
+
+*--tickets=LIST*::
+ Tickets to create (space-separated)
+
+
+Cluster Description:
+
+This is a map of cluster names to IP addresses.
+Each IP address will be configured as a virtual IP
+representing that cluster in the geo cluster
+configuration.
+
+Example with two clusters named paris and amsterdam:
+
+............
+ --clusters "paris=192.168.10.10 amsterdam=192.168.10.11"
+............
+
+Name clusters using the +--name+ parameter to `init`.
+
+Usage:
+...............
+geo-init [options]
+...............
+
+
+[[cmdhelp_cluster_geo_init_arbitrator,Initialize node as geo cluster arbitrator]]
+==== `geo-init-arbitrator`
+
+Configure the current node as a geo arbitrator. The command
+requires an existing geo cluster or geo arbitrator from which
+to get the geo cluster configuration.
+
+Options:
+
+*--clusters=DESC*::
+ Cluster description (see +geo-init+ for details)
+
+*-c IP, --cluster-node=IP*::
+ IP address of an already-configured geo cluster
+
+Usage:
+...............
+geo-init-arbitrator [options]
+...............
+
+
+[[cmdhelp_cluster_geo_join,Join cluster to existing geo cluster]]
+==== `geo-join`
+
+This command should be run from one of the nodes in a cluster
+which is currently not a member of a geo cluster. The geo
+cluster configuration will be fetched from the provided node,
+and the cluster will be added to the geo cluster.
+
+Note that each cluster in a geo cluster needs to have a unique
+name set. The cluster name can be set using the `--name` argument
+to `init`, or by configuring corosync with the cluster name in
+an existing cluster.
+
+Options:
+
+*-c IP, --cluster-node=IP*::
+ IP address of an already-configured geo cluster or arbitrator
+
+Usage:
+...............
+geo-join [options]
+...............
+
+
+[[cmdhelp_cluster_health,Cluster health check]]
+==== `health`
+
+Runs a larger set of tests and queries on all nodes in the cluster to
+verify the general system health and detect potential problems.
+
+Usage:
+...............
+health
+...............
+
+[[cmdhelp_cluster_init,Initializes a new HA cluster]]
+==== `init`
+
+Initialize a cluster from scratch. This command configures
+a complete cluster, and can also add additional cluster
+nodes to the initial one-node cluster using the `--nodes`
+option.
+
+Options:
+
+*-q, --quiet*::
+ Be quiet (don't describe what's happening, just do it)
+
+*-y, --yes*::
+ Answer "yes" to all prompts (use with caution, this
+ is destructive, especially during the "storage" stage)
+
+*-t TEMPLATE, --template=TEMPLATE**::
+ Optionally configure cluster with template "name"
+ (currently only "ocfs2" is valid here)
+
+*-n NAME, --name=NAME*::
+ Set the name of the configured cluster.
+
+*-N NODES, --nodes=NODES*::
+ Additional nodes to add to the created cluster. May
+ include the current node, which will always be the
+ initial cluster node.
+
+*-w WATCHDOG, --watchdog=WATCHDOG*::
+ Use the given watchdog device.
+
+Network configuration:
+
+Options for configuring the network and messaging layer.
+
+*-i IF, --interface=IF*::
+ Bind to IP address on interface IF
+
+*-u, --unicast*::
+ Configure corosync to communicate over unicast (UDP),
+ and not multicast. Default is multicast unless an
+ environment where multicast cannot be used is
+ detected.
+
+*-A IP, --admin-ip=IP*::
+ Configure IP address as an administration virtual IP
+
+Storage configuration:
+
+Options for configuring shared storage.
+
+*-p DEVICE, --partition-device=DEVICE*::
+ Partition this shared storage device (only used in
+ "storage" stage)
+
+*-s DEVICE, --sbd-device=DEVICE*::
+ Block device to use for SBD fencing
+
+*-o DEVICE, --ocfs2-device=DEVICE*::
+ Block device to use for OCFS2 (only used in "vgfs"
+ stage)
+
+
+Stage can be one of:
+
+*ssh*::
+ Create SSH keys for passwordless SSH between cluster nodes
+
+*csync2*::
+ Configure csync2
+
+*corosync*::
+ Configure corosync
+
+*storage*::
+ Partition shared storage (ocfs2 template only)
+
+*sbd*::
+ Configure SBD (requires -s <dev>)
+
+*cluster*::
+ Bring the cluster online
+
+*vgfs*::
+ Create volume group and filesystem (ocfs2 template only, requires `-o <dev>`)
+
+*admin*::
+ Create administration virtual IP (optional)
+
+[NOTE]
+============
+- If stage is not specified, the script will run through each stage
+ in sequence, with prompts for required information.
+- If using the ocfs2 template, the storage stage will partition a block
+ device into two pieces, one for SBD, the remainder for OCFS2. This is
+ good for testing and demonstration, but not ideal for production.
+ To use storage you have already configured, pass -s and -o to specify
+ the block devices for SBD and OCFS2, and the automatic partitioning
+ will be skipped.
+============
+
+Usage:
+...............
+init [options] [STAGE]
+...............
+
+
+[[cmdhelp_cluster_join,Join existing cluster]]
+==== `join`
+
+Join the current node to an existing cluster. The
+current node cannot be a member of a cluster already.
+Pass any node in the existing cluster as the argument
+to the `-c` option.
+
+Options:
+
+*-q, --quiet*::
+ Be quiet (don't describe what's happening, just do it)
+
+*-y, --yes*::
+ Answer "yes" to all prompts (use with caution)
+
+*-w WATCHDOG, --watchdog=WATCHDOG*::
+ Use the given watchdog device
+
+Network configuration:
+
+Options for configuring the network and messaging layer.
+
+
+*-c HOST, --cluster-node=HOST*::
+ IP address or hostname of existing cluster node
+
+*-i IF, --interface=IF*::
+ Bind to IP address on interface IF
+
+
+Stage can be one of:
+
+*ssh*::
+ Obtain SSH keys from existing cluster node (requires -c <host>)
+
+*csync2*::
+ Configure csync2 (requires -c <host>)
+
+*ssh_merge*::
+ Merge root's SSH known_hosts across all nodes (csync2 must
+ already be configured).
+
+*cluster*::
+ Start the cluster on this node
+
+If stage is not specified, each stage will be invoked in sequence.
+
+Usage:
+...............
+join [options] [STAGE]
+...............
+
+
+[[cmdhelp_cluster_remove,Remove node(s) from the cluster]]
+==== `remove`
+
+Remove one or more nodes from the cluster.
+
+This command can remove the last node in the cluster,
+thus effectively removing the whole cluster. To remove
+the last node, pass `--force` argument to `crm` or set
+the `config.core.force` option.
+
+Options:
+
+*-q, --quiet*::
+ Be quiet (don't describe what's happening, just do it)
+
+*-y, --yes*::
+ Answer "yes" to all prompts (use with caution)
+
+*-c HOST, --cluster-node=HOST*::
+ IP address or hostname of cluster node which will be
+ removed from the cluster
+
+Usage:
+...............
+remove [options] [<node> ...]
+...............
+
+
+[[cmdhelp_cluster_run,Execute an arbitrary command on all nodes]]
+==== `run`
+
+This command takes a shell statement as argument, executes that
+statement on all nodes in the cluster, and reports the result.
+
+Usage:
+...............
+run <command>
+...............
+
+Example:
+...............
+run "cat /proc/uptime"
+...............
+
+[[cmdhelp_cluster_start,Start cluster services]]
+==== `start`
+
+Starts the cluster-related system services on this node.
+
+Usage:
+.........
+start
+.........
+
+[[cmdhelp_cluster_status,Cluster status check]]
+==== `status`
+
+Reports the status for the cluster messaging layer on the local
+node.
+
+Usage:
+...............
+status
+...............
+
+[[cmdhelp_cluster_stop,Stop cluster services]]
+==== `stop`
+
+Stops the cluster-related system services on this node.
+
+Usage:
+.........
+stop
+.........
+
+[[cmdhelp_cluster_wait_for_startup,Wait for cluster to start]]
+==== `wait_for_startup`
+
+Mostly useful in scripts or automated workflows, this command will
+attempt to connect to the local cluster node repeatedly. The command
+will keep trying until the cluster node responds, or the `timeout`
+elapses. The timeout can be changed by supplying a value in seconds as
+an argument.
+
+Usage:
+........
+wait_for_startup
+........
+
+[[cmdhelp_script,Cluster script management]]
+=== `script` - Cluster script management
+
+A big part of the configuration and management of a cluster is
+collecting information about all cluster nodes and deploying changes
+to those nodes. Often, just performing the same procedure on all nodes
+will encounter problems, due to subtle differences in the
+configuration.
+
+For example, when configuring a cluster for the first time, the
+software needs to be installed and configured on all nodes before the
+cluster software can be launched and configured using `crmsh`. This
+process is cumbersome and error-prone, and the goal is for scripts to
+make this process easier.
+
+Scripts are implemented using the python `parallax` package which
+provides a thin wrapper on top of SSH. This allows the scripts to
+function through the usual SSH channels used for system maintenance,
+requiring no additional software to be installed or maintained.
+
+[[cmdhelp_script_json,JSON API for cluster scripts]]
+==== `json`
+
+This command provides a JSON API for the cluster scripts, intended for
+use in user interface tools that want to interact with the cluster via
+scripts.
+
+The command takes a single argument, which should be a JSON array with
+the first member identifying the command to perform.
+
+The output is line-based: Commands that return multiple results will
+return them line-by-line, ending with a terminator value: "end".
+
+When providing parameter values to this command, they should be
+provided as nested objects, so +virtual-ip:ip=192.168.0.5+ on the
+command line becomes the JSON object
++{"virtual-ip":{"ip":"192.168.0.5"}}+.
+
+API:
+........
+["list"]
+=> [{name, shortdesc, category}]
+
+["show", <name>]
+=> [{name, shortdesc, longdesc, category, <<steps>>}]
+
+<<steps>> := [{name, shortdesc], longdesc, required, parameters, steps}]
+
+<<params>> := [{name, shortdesc, longdesc, required, unique, advanced,
+ type, value, example}]
+
+["verify", <name>, <<values>>]
+=> [{shortdesc, longdesc, text, nodes}]
+
+["run", <name>, <<values>>]
+=> [{shortdesc, rc, output|error}]
+........
+
+
+[[cmdhelp_script_list,List available scripts]]
+==== `list`
+
+Lists the available scripts, sorted by category. Scripts that have the
+special `Script` category are hidden by default, since they are mainly
+used by other scripts or commands. To also show these, pass `all` as
+argument.
+
+To get a flat list of script names, not sorted by category, pass
+`names` as an extra argument.
+
+Usage:
+............
+list [all] [names]
+............
+
+Example:
+............
+list
+list all names
+............
+
+[[cmdhelp_script_run,Run the script]]
+==== `run`
+
+Given a list of parameter values, this command will execute the
+actions specified by the cluster script. The format for the parameter
+values is the same as for the `verify` command.
+
+Can optionally take at least two parameters:
+* `nodes=<nodes>`: List of nodes that the script runs over
+* `dry_run=yes|no`: If set, the script will not perform any modifications.
+
+Additional parameters may be available depending on the script.
+
+Use the `show` command to see what parameters are available.
+
+Usage:
+.............
+run <script> [args...]
+.............
+
+Example:
+.............
+run apache install=true
+run sbd id=sbd-1 node=node1 sbd_device=/dev/disk/by-uuid/F00D-CAFE
+.............
+
+[[cmdhelp_script_show,Describe the script]]
+==== `show`
+
+Prints a description and short summary of the script, with
+descriptions of the accepted parameters.
+
+Advanced parameters are hidden by default. To show the complete list
+of parameters accepted by the script, pass `all` as argument.
+
+Usage:
+............
+show <script> [all]
+............
+
+Example:
+............
+show virtual-ip
+............
+
+[[cmdhelp_script_verify,Verify the script]]
+==== `verify`
+
+Checks the given parameter values, and returns a list
+of actions that will be executed when running the script
+if provided the same list of parameter values.
+
+Usage:
+............
+verify <script> [args...]
+............
+
+Example:
+............
+verify sbd id=sbd-1 node=node1 sbd_device=/dev/disk/by-uuid/F00D-CAFE
+............
+
+[[cmdhelp_corosync,Corosync management]]
+=== `corosync` - Corosync management
+
+Corosync is the underlying messaging layer for most HA clusters.
+This level provides commands for editing and managing the corosync
+configuration.
+
+[[cmdhelp_corosync_add-node,Add a corosync node]]
+==== `add-node`
+
+Adds a node to the corosync configuration. This is used with the `udpu`
+type configuration in corosync.
+
+A nodeid for the added node is generated automatically.
+
+Note that this command assumes that only a single ring is used, and
+sets only the address for ring0.
+
+Usage:
+.........
+add-node <addr> [name]
+.........
+
+[[cmdhelp_corosync_del-node,Remove a corosync node]]
+==== `del-node`
+
+Removes a node from the corosync configuration. The argument given is
+the `ring0_addr` address set in the configuration file.
+
+Usage:
+.........
+del-node <addr>
+.........
+
+[[cmdhelp_corosync_diff,Diffs the corosync configuration]]
+==== `diff`
+
+Diffs the corosync configurations on different nodes. If no nodes are
+given as arguments, the corosync configurations on all nodes in the
+cluster are compared.
+
+`diff` takes an option argument `--checksum`, to display a checksum
+for each file instead of calculating a diff.
+
+Usage:
+.........
+diff [--checksum] [node...]
+.........
+
+[[cmdhelp_corosync_edit,Edit the corosync configuration]]
+==== `edit`
+
+Opens the Corosync configuration file in an editor.
+
+Usage:
+.........
+edit
+.........
+
+[[cmdhelp_corosync_get,Get a corosync configuration value]]
+==== `get`
+
+Returns the value configured in `corosync.conf`, which is not
+necessarily the value used in the running configuration. See `reload`
+for telling corosync about configuration changes.
+
+The argument is the complete dot-separated path to the value.
+
+If there are multiple values configured with the same path, the
+command returns all values for that path. For example, to get all
+configured `ring0_addr` values, use this command:
+
+Example:
+........
+get nodelist.node.ring0_addr
+........
+
+[[cmdhelp_corosync_log,Show the corosync log file]]
+==== `log`
+
+Opens the log file specified in the corosync configuration file. If no
+log file is configured, this command returns an error.
+
+The pager used can be configured either using the PAGER
+environment variable or in `crm.conf`.
+
+Usage:
+.........
+log
+.........
+
+[[cmdhelp_corosync_pull,Pulls the corosync configuration]]
+==== `pull`
+
+Gets the corosync configuration from another node and copies
+it to this node.
+
+Usage:
+.........
+pull <node>
+.........
+
+[[cmdhelp_corosync_push,Push the corosync configuration]]
+==== `push`
+
+Pushes the corosync configuration file on this node to
+the list of nodes provided. If no target nodes are given,
+the configuration is pushed to all other nodes in the cluster.
+
+It is recommended to use `csync2` to distribute the cluster
+configuration files rather than relying on this command.
+
+Usage:
+.........
+push [node] ...
+.........
+
+Example:
+.........
+push node-2 node-3
+.........
+
+[[cmdhelp_corosync_reload,Reload the corosync configuration]]
+==== `reload`
+
+Tells all instances of corosync in this cluster to reload
+`corosync.conf`.
+
+After pushing a new configuration to all cluster nodes, call this
+command to make corosync use the new configuration.
+
+Usage:
+.........
+reload
+.........
+
+[[cmdhelp_corosync_set,Set a corosync configuration value]]
+==== `set`
+
+Sets the value identified by the given path. If the value does not
+exist in the configuration file, it will be added. However, if the
+section containing the value does not exist, the command will fail.
+
+Usage:
+.........
+set quorum.expected_votes 2
+.........
+
+[[cmdhelp_corosync_show,Display the corosync configuration]]
+==== `show`
+
+Displays the corosync configuration on the current node.
+
+.........
+show
+.........
+
+[[cmdhelp_corosync_status,Display the corosync status]]
+==== `status`
+
+Displays the status of Corosync, including the votequorum state.
+
+Usage:
+.........
+status
+.........
+
+[[cmdhelp_cib,CIB shadow management]]
+=== `cib` - CIB shadow management
+
+This level is for management of shadow CIBs. It is available both
+at the top level and the `configure` level.
+
+All the commands are implemented using `cib_shadow(8)` and the
+`CIB_shadow` environment variable. The user prompt always
+includes the name of the currently active shadow or the live CIB.
+
+[[cmdhelp_cib_cibstatus,CIB status management and editing]]
+==== `cibstatus`
+
+Enter edit and manage the CIB status section level. See the
+<<cmdhelp_cibstatus,CIB status management section>>.
+
+[[cmdhelp_cib_commit,copy a shadow CIB to the cluster]]
+==== `commit`
+
+Apply a shadow CIB to the cluster. If the shadow name is omitted
+then the current shadow CIB is applied.
+
+Temporary shadow CIBs are removed automatically on commit.
+
+Usage:
+...............
+commit [<cib>]
+...............
+
+[[cmdhelp_cib_delete,delete a shadow CIB]]
+==== `delete`
+
+Delete an existing shadow CIB.
+
+Usage:
+...............
+delete <cib>
+...............
+
+[[cmdhelp_cib_diff,diff between the shadow CIB and the live CIB]]
+==== `diff`
+
+Print differences between the current cluster configuration and
+the active shadow CIB.
+
+Usage:
+...............
+diff
+...............
+
+[[cmdhelp_cib_import,import a CIB or PE input file to a shadow]]
+==== `import`
+
+At times it may be useful to create a shadow file from the
+existing CIB. The CIB may be specified as file or as a PE input
+file number. The shell will look up files in the local directory
+first and then in the PE directory (typically `/var/lib/pengine`).
+Once the CIB file is found, it is copied to a shadow and this
+shadow is immediately available for use at both `configure` and
+`cibstatus` levels.
+
+If the shadow name is omitted then the target shadow is named
+after the input CIB file.
+
+Note that there are often more than one PE input file, so you may
+need to specify the full name.
+
+Usage:
+...............
+import {<file>|<number>} [<shadow>]
+...............
+Examples:
+...............
+import pe-warn-2222
+import 2289 issue2
+...............
+
+[[cmdhelp_cib_list,list all shadow CIBs]]
+==== `list`
+
+List existing shadow CIBs.
+
+Usage:
+...............
+list
+...............
+
+[[cmdhelp_cib_new,create a new shadow CIB]]
+==== `new`
+
+Create a new shadow CIB. The live cluster configuration and
+status is copied to the shadow CIB.
+
+If the name of the shadow is omitted, we create a temporary CIB
+shadow. It is useful if multiple level sessions are desired
+without affecting the cluster. A temporary CIB shadow is short
+lived and will be removed either on `commit` or on program exit.
+Note that if the temporary shadow is not committed all changes in
+the temporary shadow are lost.
+
+Specify `withstatus` if you want to edit the status section of
+the shadow CIB (see the <<cmdhelp_cibstatus,cibstatus section>>).
+Add `force` to force overwriting the existing shadow CIB.
+
+To start with an empty configuration that is not copied from the live
+CIB, specify the `empty` keyword. (This also allows a shadow CIB to be
+created in case no cluster is running.)
+
+Usage:
+...............
+new [<cib>] [withstatus] [force] [empty]
+...............
+
+[[cmdhelp_cib_reset,copy live cib to a shadow CIB]]
+==== `reset`
+
+Copy the current cluster configuration into the shadow CIB.
+
+Usage:
+...............
+reset <cib>
+...............
+
+[[cmdhelp_cib_use,change working CIB]]
+==== `use`
+
+Choose a CIB source. If you want to edit the status from the
+shadow CIB specify `withstatus` (see <<cmdhelp_cibstatus,`cibstatus`>>).
+Leave out the CIB name to switch to the running CIB.
+
+Usage:
+...............
+use [<cib>] [withstatus]
+...............
+
+[[cmdhelp_ra,Resource Agents (RA) lists and documentation]]
+=== `ra` - Resource Agents (RA) lists and documentation
+
+This level contains commands which show various information about
+the installed resource agents. It is available both at the top
+level and at the `configure` level.
+
+[[cmdhelp_ra_classes,list classes and providers]]
+==== `classes`
+
+Print all resource agents' classes and, where appropriate, a list
+of available providers.
+
+Usage:
+...............
+classes
+...............
+
+[[cmdhelp_ra_info,show meta data for a RA]]
+==== `info` (`meta`)
+
+Show the meta-data of a resource agent type. This is where users
+can find information on how to use a resource agent. It is also
+possible to get information from some programs: `pengine`,
+`crmd`, `cib`, and `stonithd`. Just specify the program name
+instead of an RA.
+
+Usage:
+...............
+info [<class>:[<provider>:]]<type>
+info <type> <class> [<provider>] (obsolete)
+...............
+Example:
+...............
+info apache
+info ocf:pacemaker:Dummy
+info stonith:ipmilan
+info pengine
+...............
+
+[[cmdhelp_ra_list,list RA for a class (and provider)]]
+==== `list`
+
+List available resource agents for the given class. If the class
+is `ocf`, supply a provider to get agents which are available
+only from that provider.
+
+Usage:
+...............
+list <class> [<provider>]
+...............
+Example:
+...............
+list ocf pacemaker
+...............
+
+[[cmdhelp_ra_providers,show providers for a RA and a class]]
+==== `providers`
+
+List providers for a resource agent type. The class parameter
+defaults to `ocf`.
+
+Usage:
+...............
+providers <type> [<class>]
+...............
+Example:
+...............
+providers apache
+...............
+
+[[cmdhelp_ra_validate,validate parameters for RA]]
+==== `validate`
+
+If the resource agent supports the `validate-all` action, this calls
+the action with the given parameters, printing any warnings or errors
+reported by the agent.
+
+Usage:
+................
+validate <agent> [<key>=<value> ...]
+................
+
+[[cmdhelp_resource,Resource management]]
+=== `resource` - Resource management
+
+At this level resources may be managed.
+
+All (or almost all) commands are implemented with the CRM tools
+such as `crm_resource(8)`.
+
+[[cmdhelp_resource_ban,ban a resource from a node]]
+==== `ban`
+
+Ban a resource from running on a certain node. If no node is given
+as argument, the resource is banned from the current location.
+
+See `move` for details on other arguments.
+
+Usage:
+...............
+ban <rsc> [<node>] [<lifetime>] [force]
+...............
+
+[[cmdhelp_resource_cleanup,cleanup resource status]]
+==== `cleanup`
+
+Cleanup resource status. Typically done after the resource has
+temporarily failed. If a node is omitted, cleanup on all nodes.
+If there are many nodes, the command may take a while.
+
++(Pacemaker 1.1.14)+ Pass force to cleanup the resource itself,
+otherwise the cleanup command will apply to the parent resource (if
+any).
+
+Usage:
+...............
+cleanup <rsc> [<node>] [force]
+...............
+
+[[cmdhelp_resource_clear,Clear any relocation constraint]]
+==== `clear` (`unmove`, `unmigrate`, `unban`)
+
+Remove any relocation constraint created by
+the `move`, `migrate` or `ban` command.
+
+Usage:
+...............
+clear <rsc>
+unmigrate <rsc>
+unban <rsc>
+...............
+
+[[cmdhelp_resource_constraints,Show constraints affecting a resource]]
+==== `constraints`
+
+Display the location and colocation constraints affecting the
+resource.
+
+Usage:
+................
+constraints <rsc>
+................
+
+[[cmdhelp_resource_demote,demote a master-slave resource]]
+==== `demote`
+
+Demote a master-slave resource using the `target-role`
+attribute.
+
+Usage:
+...............
+demote <rsc>
+...............
+
+[[cmdhelp_resource_failcount,manage failcounts]]
+==== `failcount`
+
+Show/edit/delete the failcount of a resource.
+
+Usage:
+...............
+failcount <rsc> set <node> <value>
+failcount <rsc> delete <node>
+failcount <rsc> show <node>
+...............
+Example:
+...............
+failcount fs_0 delete node2
+...............
+
+[[cmdhelp_resource_locate,show the location of resources]]
+==== `locate`
+
+Show the current location of one or more resources.
+
+Usage:
+...............
+locate [<rsc> ...]
+...............
+
+[[cmdhelp_resource_maintenance,Enable/disable per-resource maintenance mode]]
+==== `maintenance`
+
+Enables or disables the per-resource maintenance mode. When this mode
+is enabled, no monitor operations will be triggered for the resource.
+`maintenance` attribute conflicts with the `is-managed`. When setting
+the `maintenance` attribute, the user is proposed to remove the
+`is-managed` attribute if it exists.
+
+Usage:
+..................
+maintenance <resource> [on|off|true|false]
+..................
+
+Example:
+..................
+maintenance rsc1
+maintenance rsc2 off
+..................
+
+[[cmdhelp_resource_manage,put a resource into managed mode]]
+==== `manage`
+
+Manage a resource using the `is-managed` attribute. If there
+are multiple meta attributes sets, the attribute is set in all of
+them. If the resource is a clone, all `is-managed` attributes are
+removed from the children resources.
+`is-managed` attribute conflicts with the `maintenance`. When setting
+the `is-managed` attribute, the user is proposed to remove the
+`maintenance` attribute if it exists.
+
+For details on group management see <<cmdhelp_options_manage-children,`options manage-children`>>.
+
+Usage:
+...............
+manage <rsc>
+...............
+
+[[cmdhelp_resource_meta,manage a meta attribute]]
+==== `meta`
+
+Show/edit/delete a meta attribute of a resource. Currently, all
+meta attributes of a resource may be managed with other commands
+such as `resource stop`.
+
+Usage:
+...............
+meta <rsc> set <attr> <value>
+meta <rsc> delete <attr>
+meta <rsc> show <attr>
+...............
+Example:
+...............
+meta ip_0 set target-role stopped
+...............
+
+[[cmdhelp_resource_move,Move a resource to another node]]
+==== `move` (`migrate`)
+
+Move a resource away from its current location.
+
+If the destination node is left out, the resource is migrated by
+creating a constraint which prevents it from running on the current
+node. For this type of constraint to be created, the +force+ argument
+is required.
+
+A lifetime may be given for the constraint. Once it expires, the
+location constraint will no longer be active.
+
+Usage:
+...............
+move <rsc> [<node>] [<lifetime>] [force]
+...............
+
+[[cmdhelp_resource_operations,Show active resource operations]]
+==== `operations`
+
+Show active operations, optionally filtered by resource and node.
+
+Usage:
+................
+operations [<rsc>] [<node>]
+................
+
+[[cmdhelp_resource_param,manage a parameter of a resource]]
+==== `param`
+
+Show/edit/delete a parameter of a resource.
+
+Usage:
+...............
+param <rsc> set <param> <value>
+param <rsc> delete <param>
+param <rsc> show <param>
+...............
+Example:
+...............
+param ip_0 show ip
+...............
+
+[[cmdhelp_resource_promote,promote a master-slave resource]]
+==== `promote`
+
+Promote a master-slave resource using the `target-role`
+attribute.
+
+Usage:
+...............
+promote <rsc>
+...............
+
+[[cmdhelp_resource_refresh,refresh CIB from the LRM status]]
+==== `refresh`
+
+Refresh CIB from the LRM status.
+
+.Note
+****************************
+`refresh` has been deprecated and is now
+an alias for `cleanup`.
+****************************
+
+Usage:
+...............
+refresh [<node>]
+...............
+
+[[cmdhelp_resource_reprobe,probe for resources not started by the CRM]]
+==== `reprobe`
+
+Probe for resources not started by the CRM.
+
+.Note
+****************************
+`reprobe` has been deprecated and is now
+an alias for `cleanup`.
+****************************
+
+Usage:
+...............
+reprobe [<node>]
+...............
+
+[[cmdhelp_resource_restart,restart resources]]
+==== `restart`
+
+Restart one or more resources. This is essentially a shortcut for
+resource stop followed by a start. The shell is first going to wait
+for the stop to finish, that is for all resources to really stop, and
+only then to order the start action. Due to this command
+entailing a whole set of operations, informational messages are
+printed to let the user see some progress.
+
+For details on group management see
+<<cmdhelp_options_manage-children,`options manage-children`>>.
+
+Usage:
+...............
+restart <rsc> [<rsc> ...]
+...............
+Example:
+...............
+# crm resource restart g_webserver
+INFO: ordering g_webserver to stop
+waiting for stop to finish .... done
+INFO: ordering g_webserver to start
+#
+...............
+
+[[cmdhelp_resource_scores,Display resource scores]]
+==== `scores`
+
+Display the allocation scores for all resources.
+
+Usage:
+................
+scores
+................
+
+[[cmdhelp_resource_secret,manage sensitive parameters]]
+==== `secret`
+
+Sensitive parameters can be kept in local files rather than CIB
+in order to prevent accidental data exposure. Use the `secret`
+command to manage such parameters. `stash` and `unstash` move the
+value from the CIB and back to the CIB respectively. The `set`
+subcommand sets the parameter to the provided value. `delete`
+removes the parameter completely. `show` displays the value of
+the parameter from the local file. Use `check` to verify if the
+local file content is valid.
+
+Usage:
+...............
+secret <rsc> set <param> <value>
+secret <rsc> stash <param>
+secret <rsc> unstash <param>
+secret <rsc> delete <param>
+secret <rsc> show <param>
+secret <rsc> check <param>
+...............
+Example:
+...............
+secret fence_1 show password
+secret fence_1 stash password
+secret fence_1 set password secret_value
+...............
+
+[[cmdhelp_resource_start,start resources]]
+==== `start`
+
+Start one or more resources by setting the `target-role` attribute. If
+there are multiple meta attributes sets, the attribute is set in all
+of them. If the resource is a clone, all `target-role` attributes are
+removed from the children resources.
+
+For details on group management see
+<<cmdhelp_options_manage-children,`options manage-children`>>.
+
+Usage:
+...............
+start <rsc> [<rsc> ...]
+...............
+
+[[cmdhelp_resource_status,show status of resources]]
+==== `status` (`show`, `list`)
+
+Print resource status. More than one resource can be shown at once. If
+the resource parameter is left out, the status of all resources is
+printed.
+
+Usage:
+...............
+status [<rsc> ...]
+...............
+
+[[cmdhelp_resource_stop,stop resources]]
+==== `stop`
+
+Stop one or more resources using the `target-role` attribute. If there
+are multiple meta attributes sets, the attribute is set in all of
+them. If the resource is a clone, all `target-role` attributes are
+removed from the children resources.
+
+For details on group management see
+<<cmdhelp_options_manage-children,`options manage-children`>>.
+
+Usage:
+...............
+stop <rsc> [<rsc> ...]
+...............
+
+[[cmdhelp_resource_trace,start RA tracing]]
+==== `trace`
+
+Start tracing RA for the given operation. The trace files are
+stored in `$HA_VARLIB/trace_ra`. If the operation to be traced is
+monitor, note that the number of trace files can grow very
+quickly.
+
+If no operation name is given, crmsh will attempt to trace all
+operations for the RA. This includes any configured operations, start
+and stop as well as promote/demote for multistate resources.
+
+To trace the probe operation which exists for all resources, either
+set a trace for `monitor` with interval `0`, or use `probe` as the
+operation name.
+
+Usage:
+...............
+trace <rsc> [<op> [<interval>] ]
+...............
+Example:
+...............
+trace fs start
+trace webserver
+trace webserver probe
+trace fs monitor 0
+...............
+
+[[cmdhelp_resource_unmanage,put a resource into unmanaged mode]]
+==== `unmanage`
+
+Unmanage a resource using the `is-managed` attribute. If there
+are multiple meta attributes sets, the attribute is set in all of
+them. If the resource is a clone, all `is-managed` attributes are
+removed from the children resources.
+
+For details on group management see <<cmdhelp_options_manage-children,`options manage-children`>>.
+
+Usage:
+...............
+unmanage <rsc>
+...............
+
+[[cmdhelp_resource_untrace,stop RA tracing]]
+==== `untrace`
+
+Stop tracing RA for the given operation. If no operation name is
+given, crmsh will attempt to stop tracing all operations in resource.
+
+Usage:
+...............
+untrace <rsc> [<op> [<interval>] ]
+...............
+Example:
+...............
+untrace fs start
+untrace webserver
+...............
+
+[[cmdhelp_resource_utilization,manage a utilization attribute]]
+==== `utilization`
+
+Show/edit/delete a utilization attribute of a resource. These
+attributes describe hardware requirements. By setting the
+`placement-strategy` cluster property appropriately, it is
+possible then to distribute resources based on resource
+requirements and node size. See also <<cmdhelp_node_utilization,node utilization attributes>>.
+
+Usage:
+...............
+utilization <rsc> set <attr> <value>
+utilization <rsc> delete <attr>
+utilization <rsc> show <attr>
+...............
+Example:
+...............
+utilization xen1 set memory 4096
+...............
+
+[[cmdhelp_node,Node management]]
+=== `node` - Node management
+
+Node management and status commands.
+
+[[cmdhelp_node_attribute,manage attributes]]
+==== `attribute`
+
+Edit node attributes. This kind of attribute should refer to
+relatively static properties, such as memory size.
+
+Usage:
+...............
+attribute <node> set <attr> <value>
+attribute <node> delete <attr>
+attribute <node> show <attr>
+...............
+Example:
+...............
+attribute node_1 set memory_size 4096
+...............
+
+[[cmdhelp_node_clearstate,Clear node state]]
+==== `clearstate`
+
+Resets and clears the state of the specified node. This node is
+afterwards assumed clean and offline. This command can be used to
+manually confirm that a node has been fenced (e.g., powered off).
+
+Be careful! This can cause data corruption if you confirm that a node is
+down that is, in fact, not cleanly down - the cluster will proceed as if
+the fence had succeeded, possibly starting resources multiple times.
+
+Usage:
+...............
+clearstate <node>
+...............
+
+[[cmdhelp_node_delete,delete node]]
+==== `delete`
+
+Delete a node. This command will remove the node from the CIB
+and, in case the cluster stack is running, use the appropriate
+program (`crm_node` or `hb_delnode`) to remove the node from the
+membership.
+
+If the node is still listed as active and a member of our
+partition we refuse to remove it. With the global force option
+(`-F`) we will try to delete the node anyway.
+
+Usage:
+...............
+delete <node>
+...............
+
+[[cmdhelp_node_fence,fence node]]
+==== `fence`
+
+Make CRM fence a node. This functionality depends on stonith
+resources capable of fencing the specified node. No such stonith
+resources, no fencing will happen.
+
+Usage:
+...............
+fence <node>
+...............
+
+[[cmdhelp_node_maintenance,put node into maintenance mode]]
+==== `maintenance`
+
+Set the node status to maintenance. This is equivalent to the
+cluster-wide `maintenance-mode` property but puts just one node
+into the maintenance mode. If there are maintenaned resources on
+the node, the user will be proposed to remove the maintenance
+property from them.
+
+The node parameter defaults to the node where the command is run.
+
+Usage:
+...............
+maintenance [<node>]
+...............
+
+[[cmdhelp_node_online,set node online]]
+==== `online`
+
+Set a node to online status.
+
+The node parameter defaults to the node where the command is run.
+
+Usage:
+...............
+online [<node>]
+...............
+
+[[cmdhelp_node_ready,put node into ready mode]]
+==== `ready`
+
+Set the node's maintenance status to `off`. The node should be
+now again fully operational and capable of running resource
+operations.
+
+The node parameter defaults to the node where the command is run.
+
+Usage:
+...............
+ready [<node>]
+...............
+
+[[cmdhelp_node_server,show node hostname or server address]]
+==== `server`
+
+Remote nodes may have a configured server address which should
+be used when contacting the node. This command prints the
+server address if configured, else the node name.
+
+If no parameter is given, the addresses or names for all nodes
+are printed.
+
+Usage:
+...............
+server [<node> ...]
+...............
+
+[[cmdhelp_node_show,show node]]
+==== `show`
+
+Show a node definition. If the node parameter is omitted then all
+nodes are shown.
+
+Usage:
+...............
+show [<node>]
+...............
+
+[[cmdhelp_node_standby,put node into standby]]
+==== `standby`
+
+Set a node to standby status. The node parameter defaults to the
+node where the command is run.
+
+Additionally, you may specify a lifetime for the standby---if set to
+`reboot`, the node will be back online once it reboots. `forever` will
+keep the node in standby after reboot. The life time defaults to
+`forever`.
+
+Usage:
+...............
+standby [<node>] [<lifetime>]
+
+lifetime :: reboot | forever
+...............
+
+Example:
+...............
+standby bob reboot
+...............
+
+
+[[cmdhelp_node_status,show nodes' status as XML]]
+==== `status`
+
+Show nodes' status as XML. If the node parameter is omitted then
+all nodes are shown.
+
+Usage:
+...............
+status [<node>]
+...............
+
+[[cmdhelp_node_status-attr,manage status attributes]]
+==== `status-attr`
+
+Edit node attributes which are in the CIB status section, i.e.
+attributes which hold properties of a more volatile nature. One
+typical example is attribute generated by the `pingd` utility.
+
+Usage:
+...............
+status-attr <node> set <attr> <value>
+status-attr <node> delete <attr>
+status-attr <node> show <attr>
+...............
+Example:
+...............
+status-attr node_1 show pingd
+...............
+
+[[cmdhelp_node_utilization,manage utilization attributes]]
+==== `utilization`
+
+Edit node utilization attributes. These attributes describe
+hardware characteristics as integer numbers such as memory size
+or the number of CPUs. By setting the `placement-strategy`
+cluster property appropriately, it is possible then to distribute
+resources based on resource requirements and node size. See also
+<<cmdhelp_resource_utilization,resource utilization attributes>>.
+
+Usage:
+...............
+utilization <node> set <attr> <value>
+utilization <node> delete <attr>
+utilization <node> show <attr>
+...............
+Examples:
+...............
+utilization node_1 set memory 16384
+utilization node_1 show cpu
+...............
+
+[[cmdhelp_site,GEO clustering site support]]
+=== `site` - GEO clustering site support
+
+A cluster may consist of two or more subclusters in different and
+distant locations. This set of commands supports such setups.
+
+[[cmdhelp_site_ticket,manage site tickets]]
+==== `ticket`
+
+Tickets are cluster-wide attributes. They can be managed at the
+site where this command is executed.
+
+It is then possible to constrain resources depending on the
+ticket availability (see the <<cmdhelp_configure_rsc_ticket,`rsc_ticket`>> command
+for more details).
+
+Usage:
+...............
+ticket {grant|revoke|standby|activate|show|time|delete} <ticket>
+...............
+Example:
+...............
+ticket grant ticket1
+...............
+
+[[cmdhelp_options,User preferences]]
+=== `options` - User preferences
+
+The user may set various options for the crm shell itself.
+
+[[cmdhelp_options_add-quotes,add quotes around parameters containing spaces]]
+==== `add-quotes`
+
+The shell (as in `/bin/sh`) parser strips quotes from the command
+line. This may sometimes make it really difficult to type values
+which contain white space. One typical example is the configure
+filter command. The crm shell will supply extra quotes around
+arguments which contain white space. The default is `yes`.
+
+.Note on quotes use
+****************************
+Adding quotes around arguments automatically has been introduced
+with version 1.2.2 and it is technically a regression. Being a
+regression is the only reason the `add-quotes` option exists. If
+you have custom shell scripts which would break, just set the
+`add-quotes` option to `no`.
+
+For instance, with adding quotes enabled, it is possible to do
+the following:
+...............
+# crm configure primitive d1 Dummy \
+ meta description="some description here"
+# crm configure filter 'sed "s/hostlist=./&node-c /"' fencing
+...............
+****************************
+
+[[cmdhelp_options_check-frequency,when to perform semantic check]]
+==== `check-frequency`
+
+Semantic check of the CIB or elements modified or created may be
+done on every configuration change (`always`), when verifying
+(`on-verify`) or `never`. It is by default set to `always`.
+Experts may want to change the setting to `on-verify`.
+
+The checks require that resource agents are present. If they are
+not installed at the configuration time set this preference to
+`never`.
+
+See <<topics_Features_Checks,Configuration semantic checks>> for more details.
+
+[[cmdhelp_options_check-mode,how to treat semantic errors]]
+==== `check-mode`
+
+Semantic check of the CIB or elements modified or created may be
+done in the `strict` mode or in the `relaxed` mode. In the former
+certain problems are treated as configuration errors. In the
+`relaxed` mode all are treated as warnings. The default is `strict`.
+
+See <<topics_Features_Checks,Configuration semantic checks>> for more details.
+
+[[cmdhelp_options_colorscheme,set colors for output]]
+==== `colorscheme`
+
+With `output` set to `color`, a comma separated list of colors
+from this option are used to emphasize:
+
+- keywords
+- object ids
+- attribute names
+- attribute values
+- scores
+- resource references
+
+`crm` can show colors only if there is curses support for python
+installed (usually provided by the `python-curses` package). The
+colors are whatever is available in your terminal. Use `normal`
+if you want to keep the default foreground color.
+
+This user preference defaults to
+`yellow,normal,cyan,red,green,magenta` which is good for
+terminals with dark background. You may want to change the color
+scheme and save it in the preferences file for other color
+setups.
+
+Example:
+...............
+colorscheme yellow,normal,blue,red,green,magenta
+...............
+
+[[cmdhelp_options_editor,set preferred editor program]]
+==== `editor`
+
+The `edit` command invokes an editor. Use this to specify your
+preferred editor program. If not set, it will default to either
+the value of the `EDITOR` environment variable or to one of the
+standard UNIX editors (`vi`,`emacs`,`nano`).
+
+Usage:
+...............
+editor program
+...............
+Example:
+...............
+editor vim
+...............
+
+[[cmdhelp_options_manage-children,how to handle children resource attributes]]
+==== `manage-children`
+
+Some resource management commands, such as `resource stop`, when
+the target resource is a group, may not always produce desired
+result. Each element, group and the primitive members, can have a
+meta attribute and those attributes may end up with conflicting
+values. Consider the following construct:
+...............
+crm(live)# configure show svc fs virtual-ip
+primitive fs Filesystem \
+ params device="/dev/drbd0" directory="/srv/nfs" fstype=ext3 \
+ op monitor interval=10s \
+ meta target-role=Started
+primitive virtual-ip IPaddr2 \
+ params ip=10.2.13.110 iflabel=1 \
+ op monitor interval=10s \
+ op start interval=0 \
+ meta target-role=Started
+group svc fs virtual-ip \
+ meta target-role=Stopped
+...............
+
+Even though the element +svc+ should be stopped, the group is
+actually running because all its members have the +target-role+
+set to +Started+:
+...............
+crm(live)# resource show svc
+resource svc is running on: xen-f
+...............
+
+Hence, if the user invokes +resource stop svc+ the intention is
+not clear. This preference gives the user an opportunity to
+better control what happens if attributes of group members have
+values which are in conflict with the same attribute of the group
+itself.
+
+Possible values are +ask+ (the default), +always+, and +never+.
+If set to +always+, the crm shell removes all children attributes
+which have values different from the parent. If set to +never+,
+all children attributes are left intact. Finally, if set to
++ask+, the user will be asked for each member what is to be done.
+
+[[cmdhelp_options_output,set output type]]
+==== `output`
+
+`crm` can adorn configurations in two ways: in color (similar to
+for instance the `ls --color` command) and by showing keywords in
+upper case. Possible values are `plain`, `color-always`, `color`,
+and 'uppercase'. It is possible to combine `uppercase` with one
+of the color values in order to get an upper case xmass tree. Just
+set this option to `color,uppercase` or `color-always,uppercase`.
+In case you need color codes in pipes, `color-always` forces color
+codes even in case the terminal is not a tty (just like `ls
+--color=always`).
+
+[[cmdhelp_options_pager,set preferred pager program]]
+==== `pager`
+
+The `view` command displays text through a pager. Use this to
+specify your preferred pager program. If not set, it will default
+to either the value of the `PAGER` environment variable or to one
+of the standard UNIX system pagers (`less`,`more`,`pg`).
+
+[[cmdhelp_options_reset,reset user preferences to factory defaults]]
+==== `reset`
+
+This command resets all user options to the defaults. If used as
+a single-shot command, the rc file (+$HOME/.config/crm/rc+) is
+reset to the defaults too.
+
+[[cmdhelp_options_save,save the user preferences to the rc file]]
+==== `save`
+
+Save current settings to the rc file (+$HOME/.config/crm/rc+). On
+further `crm` runs, the rc file is automatically read and parsed.
+
+[[cmdhelp_options_set,Set the value of a given option]]
+==== `set`
+
+Sets the value of an option. Takes the fully qualified
+name of the option as argument, as displayed by +show all+.
+
+The modified option value is stored in the user-local
+configuration file, usually found in +~/.config/crm/crm.conf+.
+
+Usage:
+........
+set <option> <value>
+........
+
+Example:
+........
+set color.warn "magenta bold"
+set editor nano
+........
+
+[[cmdhelp_options_show,show current user preference]]
+==== `show`
+
+Display all current settings.
+
+Given an option name as argument, `show` will display only the value
+of that argument.
+
+Given +all+ as argument, `show` displays all available user options.
+
+Usage:
+........
+show [all|<option>]
+........
+
+Example:
+........
+show
+show skill-level
+show all
+........
+
+[[cmdhelp_options_skill-level,set skill level]]
+==== `skill-level`
+
+Based on the skill-level setting, the user is allowed to use only
+a subset of commands. There are three levels: operator,
+administrator, and expert. The operator level allows only
+commands at the `resource` and `node` levels, but not editing
+or deleting resources. The administrator may do that and may also
+configure the cluster at the `configure` level and manage the
+shadow CIBs. The expert may do all.
+
+Usage:
+...............
+skill-level <level>
+
+level :: operator | administrator | expert
+...............
+
+.Note on security
+****************************
+The `skill-level` option is advisory only. There is nothing
+stopping any users change their skill level (see
+<<topics_Features_Security,Access Control Lists (ACL)>> on how to enforce
+access control).
+****************************
+
+[[cmdhelp_options_sort-elements,sort CIB elements]]
+==== `sort-elements`
+
+`crm` by default sorts CIB elements. If you want them appear in
+the order they were created, set this option to `no`.
+
+Usage:
+...............
+sort-elements {yes|no}
+...............
+Example:
+...............
+sort-elements no
+...............
+
+[[cmdhelp_options_user,set the cluster user]]
+==== `user`
+
+Sufficient privileges are necessary in order to manage a
+cluster: programs such as `crm_verify` or `crm_resource` and,
+ultimately, `cibadmin` have to be run either as `root` or as the
+CRM owner user (typically `hacluster`). You don't have to worry
+about that if you run `crm` as `root`. A more secure way is to
+run the program with your usual privileges, set this option to
+the appropriate user (such as `hacluster`), and setup the
+`sudoers` file.
+
+Usage:
+...............
+user system-user
+...............
+Example:
+...............
+user hacluster
+...............
+
+[[cmdhelp_options_wait,synchronous operation]]
+==== `wait`
+
+In normal operation, `crm` runs a command and gets back
+immediately to process other commands or get input from the user.
+With this option set to `yes` it will wait for the started
+transition to finish. In interactive mode dots are printed to
+indicate progress.
+
+Usage:
+...............
+wait {yes|no}
+...............
+Example:
+...............
+wait yes
+...............
+
+[[cmdhelp_configure,CIB configuration]]
+=== `configure` - CIB configuration
+
+This level enables all CIB object definition commands.
+
+The configuration may be logically divided into four parts:
+nodes, resources, constraints, and (cluster) properties and
+attributes. Each of these commands support one or more basic CIB
+objects.
+
+Nodes and attributes describing nodes are managed using the
+`node` command.
+
+Commands for resources are:
+
+- `primitive`
+- `monitor`
+- `group`
+- `clone`
+- `ms`/`master` (master-slave)
+
+In order to streamline large configurations, it is possible to
+define a template which can later be referenced in primitives:
+
+- `rsc_template`
+
+In that case the primitive inherits all attributes defined in the
+template.
+
+There are three types of constraints:
+
+- `location`
+- `colocation`
+- `order`
+
+It is possible to define fencing order (stonith resource
+priorities):
+
+- `fencing_topology`
+
+Finally, there are the cluster properties, resource meta
+attributes defaults, and operations defaults. All are just a set
+of attributes. These attributes are managed by the following
+commands:
+
+- `property`
+- `rsc_defaults`
+- `op_defaults`
+
+In addition to the cluster configuration, the Access Control
+Lists (ACL) can be setup to allow access to parts of the CIB for
+users other than +root+ and +hacluster+. The following commands
+manage ACL:
+
+- `user`
+- `role`
+
+In Pacemaker 1.1.12 and up, this command replaces the `user` command
+for handling ACLs:
+
+- `acl_target`
+
+The changes are applied to the current CIB only on ending the
+configuration session or using the `commit` command.
+
+Comments start with +#+ in the first line. The comments are tied
+to the element which follows. If the element moves, its comments
+will follow.
+
+[[cmdhelp_configure_acl_target,Define target access rights]]
+==== `acl_target`
+
+Defines an ACL target.
+
+Usage:
+................
+acl_target <tid> [<role> ...]
+................
+Example:
+................
+acl_target joe resource_admin constraint_editor
+................
+
+[[cmdhelp_configure_alert,Event-driven alerts]]
+==== `alert`
+
+.Version note
+****************************
+This feature is only available
+in Pacemaker 1.1.15+.
+****************************
+
+Event-driven alerts enables calling scripts whenever interesting
+events occur in the cluster (nodes joining or leaving, resources
+starting or stopping, etc.).
+
+The +path+ is an arbitrary file path to an alert script. Existing
+external scripts used with ClusterMon resources can be used as alert
+scripts, since the interface is compatible.
+
+Each alert may have a number of receipients configured. These will be
+passed to the script as arguments. The first recipient will also be
+passed as the +CRM_alert_recipient+ environment variable, for
+compatibility with existing scripts that only support one recipient.
+
+The available meta attributes are +timeout+ (default 30s) and
++timestamp-format+ (default `"%H:%M:%S.%06N"`).
+
+Some configurations may require each recipient to be delimited by
+brackets, to avoid ambiguity. In the example +alert-2+ below, the meta
+attribute for `timeout` is defined after the recipient, so the
+brackets are used to ensure that the meta attribute is set for the
+alert and not just the recipient. This can be avoided by setting any
+alert attributes before defining the recipients.
+
+Usage:
+...............
+alert <id> <path> \
+ [attributes <nvpair> ...] \
+ [meta <nvpair> ...] \
+ [to [{] <recipient>
+ [attributes <nvpair> ...] \
+ [meta <nvpair> ...] [}] \
+ ...]
+...............
+
+Example:
+...............
+alert alert-1 /srv/pacemaker/pcmk_alert_sample.sh \
+ to /var/log/cluster-alerts.log
+
+alert alert-2 /srv/pacemaker/example_alert.sh \
+ meta timeout=60s \
+ to { /var/log/cluster-alerts.log }
+...............
+
+[[cmdhelp_configure_cib,CIB shadow management]]
+==== `cib`
+
+This level is for management of shadow CIBs. It is available at
+the `configure` level to enable saving intermediate changes to a
+shadow CIB instead of to the live cluster. This short excerpt
+shows how:
+...............
+crm(live)configure# cib new test-2
+INFO: test-2 shadow CIB created
+crm(test-2)configure# commit
+...............
+Note how the current CIB in the prompt changed from +live+ to
++test-2+ after issuing the `cib new` command. See also the
+<<cmdhelp_cib,CIB shadow management>> for more information.
+
+[[cmdhelp_configure_cibstatus,CIB status management and editing]]
+==== `cibstatus`
+
+Enter edit and manage the CIB status section level. See the
+<<cmdhelp_cibstatus,CIB status management section>>.
+
+[[cmdhelp_configure_clone,define a clone]]
+==== `clone`
+
+The `clone` command creates a resource clone. It may contain a
+single primitive resource or one group of resources.
+
+Usage:
+...............
+clone <name> <rsc>
+ [description=<description>]
+ [meta <attr_list>]
+ [params <attr_list>]
+
+attr_list :: [$id=<id>] <attr>=<val> [<attr>=<val>...] | $id-ref=<id>
+...............
+Example:
+...............
+clone cl_fence apc_1 \
+ meta clone-node-max=1 globally-unique=false
+...............
+
+[[cmdhelp_configure_colocation,colocate resources]]
+==== `colocation` (`collocation`)
+
+This constraint expresses the placement relation between two
+or more resources. If there are more than two resources, then the
+constraint is called a resource set.
+
+The score is used to indicate the priority of the constraint. A
+positive score indicates that the resources should run on the same
+node. A negative score that they should not run on the same
+node. Values of positive or negative +infinity+ indicate a mandatory
+constraint.
+
+In the two resource form, the cluster will place +<with-rsc>+ first,
+and then decide where to put the +<rsc>+ resource.
+
+Collocation resource sets have an extra attribute (+sequential+)
+to allow for sets of resources which don't depend on each other
+in terms of state. The shell syntax for such sets is to put
+resources in parentheses.
+
+Sets cannot be nested.
+
+The optional +node-attribute+ can be used to colocate resources on a
+set of nodes and not necessarily on the same node. For example, by
+setting a node attribute +color+ on all nodes and setting the
++node-attribute+ value to +color+ as well, the colocated resources
+will be placed on any node that has the same color.
+
+For more details on how to configure resource sets, see
+<<topics_Features_Resourcesets,`Syntax: Resource sets`>>.
+
+Usage:
+...............
+colocation <id> <score>: <rsc>[:<role>] <with-rsc>[:<role>]
+ [node-attribute=<node_attr>]
+
+colocation <id> <score>: <resource_sets>
+ [node-attribute=<node_attr>]
+
+resource_sets :: <resource_set> [<resource_set> ...]
+
+resource_set :: ["("|"["] <rsc>[:<role>] [<rsc>[:<role>] ...] \
+ [<attributes>] [")"|"]"]
+
+attributes :: [require-all=(true|false)] [sequential=(true|false)]
+
+...............
+Example:
+...............
+colocation never_put_apache_with_dummy -inf: apache dummy
+colocation c1 inf: A ( B C )
+...............
+
+[[cmdhelp_configure_commit,commit the changes to the CIB]]
+==== `commit`
+
+Commit the current configuration to the CIB in use. As noted
+elsewhere, commands in a configure session don't have immediate
+effect on the CIB. All changes are applied at one point in time,
+either using `commit` or when the user leaves the configure
+level. In case the CIB in use changed in the meantime, presumably
+by somebody else, the crm shell will refuse to apply the changes.
+
+If you know that it's fine to still apply them, add +force+ to the
+command line.
+
+To disable CIB patching and apply the changes by replacing the CIB
+completely, add +replace+ to the command line. Note that this can lead
+to previous changes being overwritten if some other process
+concurrently modifies the CIB.
+
+Usage:
+...............
+commit [force] [replace]
+...............
+
+[[cmdhelp_configure_default-timeouts,set timeouts for operations to minimums from the meta-data]]
+==== `default-timeouts`
+
+This command takes the timeouts from the actions section of the
+resource agent meta-data and sets them for the operations of the
+primitive.
+
+Usage:
+...............
+default-timeouts <id> [<id>...]
+...............
+
+.Note on `default-timeouts`
+****************************
+The use of this command is discouraged in favor of manually
+determining the best timeouts required for the particular
+configuration. Relying on the resource agent to supply appropriate
+timeouts can cause the resource to fail at the worst possible moment.
+
+Appropriate timeouts for resource actions are context-sensitive, and
+should be carefully considered with the whole configuration in mind.
+****************************
+
+[[cmdhelp_configure_delete,delete CIB objects]]
+==== `delete`
+
+Delete one or more objects. If an object to be deleted belongs to
+a container object, such as a group, and it is the only resource
+in that container, then the container is deleted as well. Any
+related constraints are removed as well.
+
+If the object is a started resource, it will not be deleted unless the
++--force+ flag is passed to the command, or the +force+ option is set.
+
+Usage:
+...............
+delete [--force] <id> [<id>...]
+...............
+
+[[cmdhelp_configure_edit,edit CIB objects]]
+==== `edit`
+
+This command invokes the editor with the object description. As
+with the `show` command, the user may choose to edit all objects
+or a set of objects.
+
+If the user insists, he or she may edit the XML edition of the
+object. If you do that, don't modify any id attributes.
+
+Usage:
+...............
+edit [xml] [<id> ...]
+edit [xml] changed
+...............
+
+.Note on renaming element ids
+****************************
+The edit command sometimes cannot properly handle modifying
+element ids. In particular for elements which belong to group or
+ms resources. Group and ms resources themselves also cannot be
+renamed. Please use the `rename` command instead.
+****************************
+
+[[cmdhelp_configure_erase,erase the CIB]]
+==== `erase`
+
+The `erase` clears all configuration. Apart from nodes. To remove
+nodes, you have to specify an additional keyword `nodes`.
+
+Note that removing nodes from the live cluster may have some
+strange/interesting/unwelcome effects.
+
+Usage:
+...............
+erase [nodes]
+...............
+
+[[cmdhelp_configure_fencing_topology,node fencing order]]
+==== `fencing_topology`
+
+If multiple fencing (stonith) devices are available capable of
+fencing a node, their order may be specified by +fencing_topology+.
+The order is specified per node.
+
+Stonith resources can be separated by +,+ in which case all of
+them need to succeed. If they fail, the next stonith resource (or
+set of resources) is used. In other words, use comma to separate
+resources which all need to succeed and whitespace for serial
+order. It is not allowed to use whitespace around comma.
+
+If the node is left out, the order is used for all nodes.
+That should reduce the configuration size in some stonith setups.
+
+From Pacemaker version 1.1.14, it is possible to use a node attribute
+as the +target+ in a fencing topology. The syntax for this usage is
+described below.
+
+From Pacemaker version 1.1.14, it is also possible to use regular
+expression patterns as the +target+ in a fencing topology. The configured
+fencing sequence then applies to all devices matching the pattern.
+
+Usage:
+...............
+fencing_topology <stonith_resources> [<stonith_resources> ...]
+fencing_topology <fencing_order> [<fencing_order> ...]
+
+fencing_order :: <target> <stonith_resources> [<stonith_resources> ...]
+
+stonith_resources :: <rsc>[,<rsc>...]
+target :: <node>: | attr:<node-attribute>=<value> | pattern:<pattern>
+...............
+Example:
+...............
+# Only kill the power if poison-pill fails
+fencing_topology poison-pill power
+
+# As above for node-a, but a different strategy for node-b
+fencing_topology \
+ node-a: poison-pill power \
+ node-b: ipmi serial
+
+# Fencing anything on rack 1 requires fencing via both APC 1 and 2,
+# to defeat the redundancy provided by two separate UPS units.
+fencing_topology attr:rack=1 apc01,apc02
+
+# Fencing for all machines named green.* is done using the pear
+# fencing device first, while all machines named red.* are fenced
+# using the apple fencing device first.
+fencing_topology \
+ pattern:green.* pear apple \
+ pattern:red.* apple pear
+...............
+
+[[cmdhelp_configure_filter,filter CIB objects]]
+==== `filter`
+
+This command filters the given CIB elements through an external
+program. The program should accept input on `stdin` and send
+output to `stdout` (the standard UNIX filter conventions). As
+with the `show` command, the user may choose to filter all or
+just a subset of elements.
+
+It is possible to filter the XML representation of objects, but
+probably not as useful as the configuration language. The
+presentation is somewhat different from what would be displayed
+by the `show` command---each element is shown on a single line,
+i.e. there are no backslashes and no other embelishments.
+
+Don't forget to put quotes around the filter if it contains
+spaces.
+
+Usage:
+...............
+filter <prog> [xml] [<id> ...]
+filter <prog> [xml] changed
+...............
+Examples:
+...............
+filter "sed '/^primitive/s/target-role=[^ ]*//'"
+# crm configure filter "sed '/^primitive/s/target-role=[^ ]*//'"
+crm configure <<END
+ filter "sed '/threshold=\"1\"/s/=\"1\"/=\"0\"/g'"
+END
+...............
+
+.Note on quotation marks
+**************************
+Filter commands which feature a blend of quotation marks can be
+difficult to get right, especially when used directly from bash, since
+bash does its own quotation parsing. In these cases, it can be easier
+to supply the filter command as standard input. See the last example
+above.
+**************************
+
+[[cmdhelp_configure_get_property,Get property value]]
+==== `get-property`
+
+Show the value of the given property. If the value is not set, the
+command will print the default value for the property, if known.
+
+If no property name is passed to the command, the list of known
+cluster properties is printed.
+
+If the property is set multiple times, for example using multiple
+property sets with different rule expressions, the output of this
+command is undefined.
+
+Pass the argument +-t+ or +--true+ to `get-property` to translate
+the argument value into +true+ or +false+. If the value is not
+set, the command will print +false+.
+
+Usage:
+...............
+get-property [-t|--true] [<name>]
+...............
+
+Example:
+...............
+get-property stonith-enabled
+get-property -t maintenance-mode
+...............
+
+[[cmdhelp_configure_graph,generate a directed graph]]
+==== `graph`
+
+Create a graphviz graphical layout from the current cluster
+configuration.
+
+Currently, only `dot` (directed graph) is supported. It is
+essentially a visualization of resource ordering.
+
+The graph may be saved to a file which can be used as source for
+various graphviz tools (by default it is displayed in the user's
+X11 session). Optionally, by specifying the format, one can also
+produce an image instead.
+
+For more or different graphviz attributes, it is possible to save
+the default set of attributes to an ini file. If this file exists
+it will always override the builtin settings. The +exportsettings+
+subcommand also prints the location of the ini file.
+
+Usage:
+...............
+graph [<gtype> [<file> [<img_format>]]]
+graph exportsettings
+
+gtype :: dot
+img_format :: `dot` output format (see the +-T+ option)
+...............
+Example:
+...............
+graph dot
+graph dot clu1.conf.dot
+graph dot clu1.conf.svg svg
+...............
+
+[[cmdhelp_configure_group,define a group]]
+==== `group`
+
+The `group` command creates a group of resources. This can be useful
+when resources depend on other resources and require that those
+resources start in order on the same node. A common use of resource
+groups is to ensure that a server and a virtual IP are located
+together, and that the virtual IP is started before the server.
+
+Grouped resources are started in the order they appear in the group,
+and stopped in the reverse order. If a resource in the group cannot
+run anywhere, resources following it in the group will not start.
+
+`group` can be passed the "container" meta attribute, to indicate that
+it is to be used to group VM resources monitored using Nagios. The
+resource referred to by the container attribute must be of type
+`ocf:heartbeat:Xen`, `ocf:heartbeat:VirtualDomain` or `ocf:heartbeat:lxc`.
+
+Usage:
+...............
+group <name> <rsc> [<rsc>...]
+ [description=<description>]
+ [meta attr_list]
+ [params attr_list]
+
+attr_list :: [$id=<id>] <attr>=<val> [<attr>=<val>...] | $id-ref=<id>
+...............
+Example:
+...............
+group internal_www disk0 fs0 internal_ip apache \
+ meta target_role=stopped
+
+group vm-and-services vm vm-sshd meta container="vm"
+...............
+
+[[cmdhelp_configure_load,import the CIB from a file]]
+==== `load`
+
+Load a part of configuration (or all of it) from a local file or
+a network URL. The +replace+ method replaces the current
+configuration with the one from the source. The +update+ method
+tries to import the contents into the current configuration. The
++push+ method imports the contents into the current configuration
+and removes any lines that are not present in the given
+configuration.
+The file may be a CLI file or an XML file.
+
+If the URL is `-`, the configuration is read from standard input.
+
+Usage:
+...............
+load [xml] <method> URL
+
+method :: replace | update | push
+...............
+Example:
+...............
+load xml update myfirstcib.xml
+load xml replace http://storage.big.com/cibs/bigcib.xml
+load xml push smallcib.xml
+...............
+
+[[cmdhelp_configure_location,a location preference]]
+==== `location`
+
+`location` defines the preference of nodes for the given
+resource. The location constraints consist of one or more rules
+which specify a score to be awarded if the rule matches.
+
+The resource referenced by the location constraint can be one of the
+following:
+
+* Plain resource reference: +location loc1 webserver 100: node1+
+* Resource set in curly brackets: +location loc1 { virtual-ip webserver } 100: node1+
+* Tag containing resource ids: +location loc1 tag1 100: node1+
+* Resource pattern: +location loc1 /web.*/ 100: node1+
+
+The +resource-discovery+ attribute allows probes to be selectively
+enabled or disabled per resource and node.
+
+The syntax for resource sets is described in detail for
+<<cmdhelp_configure_colocation,`colocation`>>.
+
+For more details on how to configure resource sets, see
+<<topics_Features_Resourcesets,`Syntax: Resource sets`>>.
+
+For more information on rule expressions, see
+<<topics_Syntax_RuleExpressions,Syntax: Rule expressions>>.
+
+Usage:
+...............
+location <id> <rsc> [<attributes>] {<node_pref>|<rules>}
+
+rsc :: /<rsc-pattern>/
+ | { resource_sets }
+ | <rsc>
+
+attributes :: role=<role> | resource-discovery=always|never|exclusive
+
+node_pref :: <score>: <node>
+
+rules ::
+ rule [id_spec] [$role=<role>] <score>: <expression>
+ [rule [id_spec] [$role=<role>] <score>: <expression> ...]
+
+id_spec :: $id=<id> | $id-ref=<id>
+score :: <number> | <attribute> | [-]inf
+expression :: <simple_exp> [<bool_op> <simple_exp> ...]
+bool_op :: or | and
+simple_exp :: <attribute> [type:]<binary_op> <value>
+ | <unary_op> <attribute>
+ | date <date_expr>
+type :: string | version | number
+binary_op :: lt | gt | lte | gte | eq | ne
+unary_op :: defined | not_defined
+
+date_expr :: lt <end>
+ | gt <start>
+ | in start=<start> end=<end>
+ | in start=<start> <duration>
+ | spec <date_spec>
+duration|date_spec ::
+ hours=<value>
+ | monthdays=<value>
+ | weekdays=<value>
+ | yearsdays=<value>
+ | months=<value>
+ | weeks=<value>
+ | years=<value>
+ | weekyears=<value>
+ | moon=<value>
+...............
+Examples:
+...............
+location conn_1 internal_www 100: node1
+
+location conn_1 internal_www \
+ rule 50: #uname eq node1 \
+ rule pingd: defined pingd
+
+location conn_2 dummy_float \
+ rule -inf: not_defined pingd or pingd number:lte 0
+
+# never probe for rsc1 on node1
+location no-probe rsc1 resource-discovery=never -inf: node1
+...............
+
+[[cmdhelp_configure_modgroup,modify group]]
+==== `modgroup`
+
+Add or remove primitives in a group. The `add` subcommand appends
+the new group member by default. Should it go elsewhere, there
+are `after` and `before` clauses.
+
+Usage:
+...............
+modgroup <id> add <id> [after <id>|before <id>]
+modgroup <id> remove <id>
+...............
+Examples:
+...............
+modgroup share1 add storage2 before share1-fs
+...............
+
+[[cmdhelp_configure_monitor,add monitor operation to a primitive]]
+==== `monitor`
+
+Monitor is by far the most common operation. It is possible to
+add it without editing the whole resource. Also, long primitive
+definitions may be a bit uncluttered. In order to make this
+command as concise as possible, less common operation attributes
+are not available. If you need them, then use the `op` part of
+the `primitive` command.
+
+Usage:
+...............
+monitor <rsc>[:<role>] <interval>[:<timeout>]
+...............
+Example:
+...............
+monitor apcfence 60m:60s
+...............
+
+Note that after executing the command, the monitor operation may
+be shown as part of the primitive definition.
+
+[[cmdhelp_configure_ms,define a master-slave resource]]
+==== `ms` (`master`)
+
+The `ms` command creates a master/slave resource type. It may contain a
+single primitive resource or one group of resources.
+
+Usage:
+...............
+ms <name> <rsc>
+ [description=<description>]
+ [meta attr_list]
+ [params attr_list]
+
+attr_list :: [$id=<id>] <attr>=<val> [<attr>=<val>...] | $id-ref=<id>
+...............
+Example:
+...............
+ms disk1 drbd1 \
+ meta notify=true globally-unique=false
+...............
+
+.Note on `id-ref` usage
+****************************
+Instance or meta attributes (`params` and `meta`) may contain
+a reference to another set of attributes. In that case, no other
+attributes are allowed. Since attribute sets' ids, though they do
+exist, are not shown in the `crm`, it is also possible to
+reference an object instead of an attribute set. `crm` will
+automatically replace such a reference with the right id:
+
+...............
+crm(live)configure# primitive a2 www-2 meta $id-ref=a1
+crm(live)configure# show a2
+primitive a2 apache \
+ meta $id-ref=a1-meta_attributes
+ [...]
+...............
+It is advisable to give meaningful names to attribute sets which
+are going to be referenced.
+****************************
+
+[[cmdhelp_configure_node,define a cluster node]]
+==== `node`
+
+The node command describes a cluster node. Nodes in the CIB are
+commonly created automatically by the CRM. Hence, you should not
+need to deal with nodes unless you also want to define node
+attributes. Note that it is also possible to manage node
+attributes at the `node` level.
+
+Usage:
+...............
+node [$id=<id>] <uname>[:<type>]
+ [description=<description>]
+ [attributes [$id=<id>] [<score>:] [rule...]
+ <param>=<value> [<param>=<value>...]] | $id-ref=<ref>
+ [utilization [$id=<id>] [<score>:] [rule...]
+ <param>=<value> [<param>=<value>...]] | $id-ref=<ref>
+
+type :: normal | member | ping | remote
+...............
+Example:
+...............
+node node1
+node big_node attributes memory=64
+...............
+
+[[cmdhelp_configure_op_defaults,set resource operations defaults]]
+==== `op_defaults`
+
+Set defaults for the operations meta attributes.
+
+For more information on rule expressions, see
+<<topics_Syntax_RuleExpressions,Syntax: Rule expressions>>.
+
+Usage:
+...............
+op_defaults [$id=<set_id>] [rule ...] <option>=<value> [<option>=<value> ...]
+...............
+Example:
+...............
+op_defaults record-pending=true
+...............
+
+[[cmdhelp_configure_order,order resources]]
+==== `order`
+
+This constraint expresses the order of actions on two resources
+or more resources. If there are more than two resources, then the
+constraint is called a resource set.
+
+Ordered resource sets have an extra attribute to allow for sets
+of resources whose actions may run in parallel. The shell syntax
+for such sets is to put resources in parentheses.
+
+If the subsequent resource can start or promote after any one of the
+resources in a set has done, enclose the set in brackets (+[+ and +]+).
+
+Sets cannot be nested.
+
+Three strings are reserved to specify a kind of order constraint:
++Mandatory+, +Optional+, and +Serialize+. It is preferred to use
+one of these settings instead of score. Previous versions mapped
+scores +0+ and +inf+ to keywords +advisory+ and +mandatory+.
+That is still valid but deprecated.
+
+For more details on how to configure resource sets, see
+<<topics_Features_Resourcesets,`Syntax: Resource sets`>>.
+
+Usage:
+...............
+order <id> [{kind|<score>}:] first then [symmetrical=<bool>]
+
+order <id> [{kind|<score>}:] resource_sets [symmetrical=<bool>]
+
+kind :: Mandatory | Optional | Serialize
+
+first :: <rsc>[:<action>]
+
+then :: <rsc>[:<action>]
+
+resource_sets :: resource_set [resource_set ...]
+
+resource_set :: ["["|"("] <rsc>[:<action>] [<rsc>[:<action>] ...] \
+ [attributes] ["]"|")"]
+
+attributes :: [require-all=(true|false)] [sequential=(true|false)]
+
+...............
+Example:
+...............
+order o-1 Mandatory: apache:start ip_1
+order o-2 Serialize: A ( B C )
+order o-3 inf: [ A B ] C
+order o-4 first-resource then-resource
+...............
+
+[[cmdhelp_configure_primitive,define a resource]]
+==== `primitive`
+
+The primitive command describes a resource. It may be referenced
+only once in group, clone, or master-slave objects. If it's not
+referenced, then it is placed as a single resource in the CIB.
+
+Operations may be specified anonymously, as a group or by reference:
+
+* "Anonymous", as a list of +op+ specifications. Use this
+ method if you don't need to reference the set of operations
+ elsewhere. This is the most common way to define operations.
+
+* If reusing operation sets is desired, use the +operations+ keyword
+ along with an id to give the operations set a name. Use the
+ +operations+ keyword and an id-ref value set to the id of another
+ operations set, to apply the same set of operations to this
+ primitive.
+
+Operation attributes which are not recognized are saved as
+instance attributes of that operation. A typical example is
++OCF_CHECK_LEVEL+.
+
+For multistate resources, roles are specified as +role=<role>+.
+
+A template may be defined for resources which are of the same
+type and which share most of the configuration. See
+<<cmdhelp_configure_rsc_template,`rsc_template`>> for more information.
+
+Attributes containing time values, such as the +interval+ attribute on
+operations, are configured either as a plain number, which is
+interpreted as a time in seconds, or using one of the following
+suffixes:
+
+* +s+, +sec+ - time in seconds (same as no suffix)
+* +ms+, +msec+ - time in milliseconds
+* +us+, +usec+ - time in microseconds
+* +m+, +min+ - time in minutes
+* +h+, +hr+ - time in hours
+
+Usage:
+...............
+primitive <rsc> {[<class>:[<provider>:]]<type>|@<template>}
+ [description=<description>]
+ [[params] attr_list]
+ [meta attr_list]
+ [utilization attr_list]
+ [operations id_spec]
+ [op op_type [<attribute>=<value>...] ...]
+
+attr_list :: [$id=<id>] [<score>:] [rule...]
+ <attr>=<val> [<attr>=<val>...]] | $id-ref=<id>
+id_spec :: $id=<id> | $id-ref=<id>
+op_type :: start | stop | monitor
+...............
+Example:
+...............
+primitive apcfence stonith:apcsmart \
+ params ttydev=/dev/ttyS0 hostlist="node1 node2" \
+ op start timeout=60s \
+ op monitor interval=30m timeout=60s
+
+primitive www8 apache \
+ configfile=/etc/apache/www8.conf \
+ operations $id-ref=apache_ops
+
+primitive db0 mysql \
+ params config=/etc/mysql/db0.conf \
+ op monitor interval=60s \
+ op monitor interval=300s OCF_CHECK_LEVEL=10
+
+primitive r0 ocf:linbit:drbd \
+ params drbd_resource=r0 \
+ op monitor role=Master interval=60s \
+ op monitor role=Slave interval=300s
+
+primitive xen0 @vm_scheme1 xmfile=/etc/xen/vm/xen0
+
+primitive mySpecialRsc Special \
+ params 3: rule #uname eq node1 interface=eth1 \
+ params 2: rule #uname eq node2 interface=eth2 port=8888 \
+ params 1: interface=eth0 port=9999
+
+...............
+
+[[cmdhelp_configure_property,set a cluster property]]
+==== `property`
+
+Set cluster configuration properties. To list the
+available cluster configuration properties, use the
+<<cmdhelp_ra_info,`ra info`>> command with +pengine+, +crmd+,
++cib+ and +stonithd+ as arguments.
+When setting the +maintenance-mode+ property, it will
+inform the user if there are nodes or resources that
+have the +maintenance+ property.
+
+For more information on rule expressions, see
+<<topics_Syntax_RuleExpressions,Syntax: Rule expressions>>.
+
+Usage:
+...............
+property [<set_id>:] [rule ...] <option>=<value> [<option>=<value> ...]
+...............
+Example:
+...............
+property stonith-enabled=true
+property rule date spec years=2014 stonith-enabled=false
+...............
+
+[[cmdhelp_configure_ptest,show cluster actions if changes were committed]]
+==== `ptest` (`simulate`)
+
+Show PE (Policy Engine) motions using `ptest(8)` or
+`crm_simulate(8)`.
+
+A CIB is constructed using the current user edited configuration
+and the status from the running CIB. The resulting CIB is run
+through `ptest` (or `crm_simulate`) to show changes which would
+happen if the configuration is committed.
+
+The status section may be loaded from another source and modified
+using the <<cmdhelp_cibstatus,`cibstatus`>> level commands. In that case, the
+`ptest` command will issue a message informing the user that the
+Policy Engine graph is not calculated based on the current status
+section and therefore won't show what would happen to the
+running but some imaginary cluster.
+
+If you have graphviz installed and X11 session, `dotty(1)` is run
+to display the changes graphically.
+
+Add a string of +v+ characters to increase verbosity. `ptest`
+can also show allocation scores. +utilization+ turns on
+information about the remaining capacity of nodes. With the
++actions+ option, `ptest` will print all resource actions.
+
+The `ptest` program has been replaced by `crm_simulate` in newer
+Pacemaker versions. In some installations both could be
+installed. Use `simulate` to enfore using `crm_simulate`.
+
+Usage:
+...............
+ptest [nograph] [v...] [scores] [actions] [utilization]
+...............
+Examples:
+...............
+ptest scores
+ptest vvvvv
+simulate actions
+...............
+
+[[cmdhelp_configure_refresh,refresh from CIB]]
+==== `refresh`
+
+Refresh the internal structures from the CIB. All changes made
+during this session are lost.
+
+Usage:
+...............
+refresh
+...............
+
+[[cmdhelp_configure_rename,rename a CIB object]]
+==== `rename`
+
+Rename an object. It is recommended to use this command to rename
+a resource, because it will take care of updating all related
+constraints and a parent resource. Changing ids with the edit
+command won't have the same effect.
+
+If you want to rename a resource, it must be in the stopped state.
+
+Usage:
+...............
+rename <old_id> <new_id>
+...............
+
+[[cmdhelp_configure_role,define role access rights]]
+==== `role`
+
+An ACL role is a set of rules which describe access rights to
+CIB. Rules consist of an access right +read+, +write+, or +deny+
+and a specification denoting part of the configuration to which
+the access right applies. The specification can be an XPath or a
+combination of tag and id references. If an attribute is
+appended, then the specification applies only to that attribute
+of the matching element.
+
+There is a number of shortcuts for XPath specifications. The
++meta+, +params+, and +utilization+ shortcuts reference resource
+meta attributes, parameters, and utilization respectively. The
+`location` may be used to specify location constraints most of
+the time to allow resource `move` and `unmove` commands. The
+`property` references cluster properties. The `node` allows
+reading node attributes. +nodeattr+ and +nodeutil+ reference node
+attributes and node capacity (utilization). The `status` shortcut
+references the whole status section of the CIB. Read access to
+status is necessary for various monitoring tools such as
+`crm_mon(8)` (aka `crm status`).
+
+For more information on rule expressions, see
+<<topics_Syntax_RuleExpressions,Syntax: Rule expressions>>.
+
+Usage:
+...............
+role <role-id> rule [rule ...]
+
+rule :: acl-right cib-spec [attribute:<attribute>]
+
+acl-right :: read | write | deny
+
+cib-spec :: xpath-spec | tag-ref-spec
+xpath-spec :: xpath:<xpath> | shortcut
+tag-ref-spec :: tag:<tag> | ref:<id> | tag:<tag> ref:<id>
+
+shortcut :: meta:<rsc>[:<attr>]
+ params:<rsc>[:<attr>]
+ utilization:<rsc>
+ location:<rsc>
+ property[:<attr>]
+ node[:<node>]
+ nodeattr[:<attr>]
+ nodeutil[:<node>]
+ status
+...............
+Example:
+...............
+role app1_admin \
+ write meta:app1:target-role \
+ write meta:app1:is-managed \
+ write location:app1 \
+ read ref:app1
+...............
+
+[[cmdhelp_configure_rsc_defaults,set resource defaults]]
+==== `rsc_defaults`
+
+Set defaults for the resource meta attributes.
+
+For more information on rule expressions, see
+<<topics_Syntax_RuleExpressions,Syntax: Rule expressions>>.
+
+Usage:
+...............
+rsc_defaults [<set_id>:] [rule ...] <option>=<value> [<option>=<value> ...]
+...............
+Example:
+...............
+rsc_defaults failure-timeout=3m
+...............
+
+[[cmdhelp_configure_rsc_template,define a resource template]]
+==== `rsc_template`
+
+The `rsc_template` command creates a resource template. It may be
+referenced in primitives. It is used to reduce large
+configurations with many similar resources.
+
+Usage:
+...............
+rsc_template <name> [<class>:[<provider>:]]<type>
+ [description=<description>]
+ [params attr_list]
+ [meta attr_list]
+ [utilization attr_list]
+ [operations id_spec]
+ [op op_type [<attribute>=<value>...] ...]
+
+attr_list :: [$id=<id>] <attr>=<val> [<attr>=<val>...] | $id-ref=<id>
+id_spec :: $id=<id> | $id-ref=<id>
+op_type :: start | stop | monitor
+...............
+Example:
+...............
+rsc_template public_vm Xen \
+ op start timeout=300s \
+ op stop timeout=300s \
+ op monitor interval=30s timeout=60s \
+ op migrate_from timeout=600s \
+ op migrate_to timeout=600s
+primitive xen0 @public_vm \
+ params xmfile=/etc/xen/xen0
+primitive xen1 @public_vm \
+ params xmfile=/etc/xen/xen1
+...............
+
+[[cmdhelp_configure_rsc_ticket,resources ticket dependency]]
+==== `rsc_ticket`
+
+This constraint expresses dependency of resources on cluster-wide
+attributes, also known as tickets. Tickets are mainly used in
+geo-clusters, which consist of multiple sites. A ticket may be
+granted to a site, thus allowing resources to run there.
+
+The +loss-policy+ attribute specifies what happens to the
+resource (or resources) if the ticket is revoked. The default is
+either +stop+ or +demote+ depending on whether a resource is
+multi-state.
+
+See also the <<cmdhelp_site_ticket,`site`>> set of commands.
+
+Usage:
+...............
+rsc_ticket <id> <ticket_id>: <rsc>[:<role>] [<rsc>[:<role>] ...]
+ [loss-policy=<loss_policy_action>]
+
+loss_policy_action :: stop | demote | fence | freeze
+...............
+Example:
+...............
+rsc_ticket ticket-A_public-ip ticket-A: public-ip
+rsc_ticket ticket-A_bigdb ticket-A: bigdb loss-policy=fence
+rsc_ticket ticket-B_storage ticket-B: drbd-a:Master drbd-b:Master
+...............
+
+
+[[cmdhelp_configure_rsctest,test resources as currently configured]]
+==== `rsctest`
+
+Test resources with current resource configuration. If no nodes
+are specified, tests are run on all known nodes.
+
+The order of resources is significant: it is assumed that later
+resources depend on earlier ones.
+
+If a resource is multi-state, it is assumed that the role on
+which later resources depend is master.
+
+Tests are run sequentially to prevent running the same resource
+on two or more nodes. Tests are carried out only if none of the
+specified nodes currently run any of the specified resources.
+However, it won't verify whether resources run on the other
+nodes.
+
+Superuser privileges are obviously required: either run this as
+root or setup the `sudoers` file appropriately.
+
+Note that resource testing may take some time.
+
+Usage:
+...............
+rsctest <rsc_id> [<rsc_id> ...] [<node_id> ...]
+...............
+Examples:
+...............
+rsctest my_ip websvc
+rsctest websvc nodeB
+...............
+
+[[cmdhelp_configure_save,save the CIB to a file]]
+==== `save`
+
+Save the current configuration to a file. Optionally, as XML. Use
++-+ instead of file name to write the output to `stdout`.
+
+The `save` command accepts the same selection arguments as the `show`
+command. See the <<cmdhelp_configure_show,help section>> for `show`
+for more details.
+
+Usage:
+...............
+save [xml] [<id> | type:<type | tag:<tag> |
+ related:<obj> | changed ...] <file>
+...............
+Example:
+...............
+save myfirstcib.txt
+save web-server server-config.txt
+...............
+
+[[cmdhelp_configure_schema,set or display current CIB RNG schema]]
+==== `schema`
+
+CIB's content is validated by a RNG schema. Pacemaker supports
+several, depending on version. At least the following schemas are
+accepted by `crmsh`:
+
+* +pacemaker-1.0+
+* +pacemaker-1.1+
+* +pacemaker-1.2+
+* +pacemaker-1.3+
+* +pacemaker-2.0+
+
+Use this command to display or switch to another RNG schema.
+
+Usage:
+...............
+schema [<schema>]
+...............
+Example:
+...............
+schema pacemaker-1.1
+...............
+
+[[cmdhelp_configure_set,set an attribute value]]
+==== `set`
+
+Set the value of a configured attribute. The attribute must
+have a value configured previously, and can be an agent
+parameter, meta attribute or utilization value.
+
+The first argument to the command is a path to an attribute.
+This is a dot-separated sequence beginning with the name of
+the resource, and ending with the name of the attribute to
+set.
+
+Usage:
+...............
+set <path> <value>
+...............
+Examples:
+...............
+set vip1.ip 192.168.20.5
+set vm-a.force_stop 1
+...............
+
+[[cmdhelp_configure_show,display CIB objects]]
+==== `show`
+
+The `show` command displays CIB objects. Without any argument, it
+displays all objects in the CIB, but the set of objects displayed by
+`show` can be limited to only objects with the given IDs or by using
+one or more of the special prefixes described below.
+
+The XML representation for the objects can be displayed by passing
++xml+ as the first argument.
+
+To show one or more specific objects, pass the object IDs as
+arguments.
+
+To show all objects of a certain type, use the +type:+ prefix.
+
+To show all objects in a tag, use the +tag:+ prefix.
+
+To show all constraints related to a primitive, use the +related:+ prefix.
+
+To show all modified objects, pass the argument +changed+.
+
+The prefixes can be used together on a single command line. For
+example, to show both the tag itself and the objects tagged by it the
+following combination can be used: +show tag:my-tag my-tag+.
+
+To refine a selection of objects using multiple modifiers, the keywords
++and+ and +or+ can be used. For example, to select all primitives tagged
++foo+, the following combination can be used:
++show type:primitive and tag:foo+.
+
+To hide values when displaying the configuration, use the
++obscure:<glob>+ argument. This can be useful when sending the
+configuration over a public channel, to avoid exposing potentially
+sensitive information. The +<glob>+ argument is a bash-style pattern
+matching attribute keys.
+
+Usage:
+...............
+show [xml] [<id>
+ | changed
+ | type:<type>
+ | tag:<id>
+ | related:<obj>
+ | obscure:<glob>
+ ...]
+
+type :: node | primitive | group | clone | ms | rsc_template
+ | location | colocation | order
+ | rsc_ticket
+ | property | rsc_defaults | op_defaults
+ | fencing_topology
+ | role | user | acl_target
+ | tag
+...............
+
+Example:
+...............
+show webapp
+show type:primitive
+show xml tag:db tag:fs
+show related:webapp
+show type:primitive obscure:passwd
+...............
+
+[[cmdhelp_configure_tag,Define resource tags]]
+==== `tag`
+
+Define a resource tag. A tag is an id referring to one or more
+resources, without implying any constraints between the tagged
+resources. This can be useful for grouping conceptually related
+resources.
+
+Usage:
+...............
+tag <tag-name>: <rsc> [<rsc> ...]
+tag <tag-name> <rsc> [<rsc> ...]
+...............
+Example:
+...............
+tag web: p-webserver p-vip
+tag ips server-vip admin-vip
+...............
+
+[[cmdhelp_configure_template,edit and import a configuration from a template]]
+==== `template`
+
+The specified template is loaded into the editor. It's up to the
+user to make a good CRM configuration out of it. See also the
+<<cmdhelp_template,template section>>.
+
+Usage:
+...............
+template [xml] url
+...............
+Example:
+...............
+template two-apaches.txt
+...............
+
+[[cmdhelp_configure_upgrade,upgrade the CIB]]
+==== `upgrade`
+
+Attempts to upgrade the CIB to validate with the current
+version. Commonly, this is required if the error
+`CIB not supported` occurs. It typically means that the
+active CIB version is coming from an older release.
+
+As a safety precaution, the force argument is required if the
++validation-with+ attribute is set to anything other than
++0.6+. Thus in most cases, it is required.
+
+Usage:
+...............
+upgrade [force]
+...............
+
+Example:
+...............
+upgrade force
+...............
+
+[[cmdhelp_configure_user,define user access rights]]
+==== `user`
+
+Users which normally cannot view or manage cluster configuration
+can be allowed access to parts of the CIB. The access is defined
+by a set of +read+, +write+, and +deny+ rules as in role
+definitions or by referencing roles. The latter is considered
+best practice.
+
+For more information on rule expressions, see
+<<topics_Syntax_RuleExpressions,Syntax: Rule expressions>>.
+
+Usage:
+...............
+user <uid> {roles|rules}
+
+roles :: role:<role-ref> [role:<role-ref> ...]
+rules :: rule [rule ...]
+...............
+Example:
+...............
+user joe \
+ role:app1_admin \
+ role:read_all
+...............
+
+[[cmdhelp_configure_validate_all,call agent validate-all for resource]]
+==== `validate-all`
+
+Call the `validate-all` action for the resource, if possible.
+
+Limitations:
+
+* The resource agent must implement the `validate-all` action.
+* The current user must be root.
+* The primitive resource must not use nvpair references.
+
+Usage:
+...............
+validate-all <rsc>
+...............
+
+
+[[cmdhelp_configure_verify,verify the CIB with crm_verify]]
+==== `verify`
+
+Verify the contents of the CIB which would be committed.
+
+Usage:
+...............
+verify
+...............
+
+[[cmdhelp_configure_xml,raw xml]]
+==== `xml`
+
+Even though we promissed no xml, it may happen, but hopefully
+very very seldom, that an element from the CIB cannot be rendered
+in the configuration language. In that case, the element will be
+shown as raw xml, prefixed by this command. That element can then
+be edited like any other. If the shell finds out that after the
+change it can digest it, then it is going to be converted into
+the normal configuration language. Otherwise, there is no need to
+use `xml` for configuration.
+
+Usage:
+...............
+xml <xml>
+...............
+
+[[cmdhelp_template,edit and import a configuration from a template]]
+=== `template` - Import configuration from templates
+
+User may be assisted in the cluster configuration by templates
+prepared in advance. Templates consist of a typical ready
+configuration which may be edited to suit particular user needs.
+
+This command enters a template level where additional commands
+for configuration/template management are available.
+
+[[cmdhelp_template_apply,process and apply the current configuration to the current CIB]]
+==== `apply`
+
+Copy the current or given configuration to the current CIB. By
+default, the CIB is replaced, unless the method is set to
+"update".
+
+Usage:
+...............
+apply [<method>] [<config>]
+
+method :: replace | update
+...............
+
+[[cmdhelp_template_delete,delete a configuration]]
+==== `delete`
+
+Remove a configuration. The loaded (active) configuration may be
+removed by force.
+
+Usage:
+...............
+delete <config> [force]
+...............
+
+[[cmdhelp_template_edit,edit a configuration]]
+==== `edit`
+
+Edit current or given configuration using your favourite editor.
+
+Usage:
+...............
+edit [<config>]
+...............
+
+[[cmdhelp_template_list,list configurations/templates]]
+==== `list`
+
+When called with no argument, lists existing templates and
+configurations.
+
+Given the argument +templates+, lists the available templates.
+
+Given the argument +configs+, lists the available configurations.
+
+Usage:
+...............
+list [templates|configs]
+...............
+
+[[cmdhelp_template_load,load a configuration]]
+==== `load`
+
+Load an existing configuration. Further `edit`, `show`, and
+`apply` commands will refer to this configuration.
+
+Usage:
+...............
+load <config>
+...............
+
+[[cmdhelp_template_new,create a new configuration from templates]]
+==== `new`
+
+Create a new configuration from one or more templates. Note that
+configurations and templates are kept in different places, so it
+is possible to have a configuration name equal a template name.
+
+If you already know which parameters are required, you can set
+them directly on the command line.
+
+The parameter name +id+ is set by default to the name of the
+configuration.
+
+If no parameters are being set and you don't want a particular name
+for your configuration, you can call this command with a template name
+as the only parameter. A unique configuration name based on the
+template name will be generated.
+
+Usage:
+...............
+new [<config>] <template> [<template> ...] [params name=value ...]
+...............
+
+Example:
+...............
+new vip virtual-ip
+new bigfs ocfs2 params device=/dev/sdx8 directory=/bigfs
+new apache
+...............
+
+[[cmdhelp_template_show,show the processed configuration]]
+==== `show`
+
+Process the current or given configuration and display the result.
+
+Usage:
+...............
+show [<config>]
+...............
+
+[[cmdhelp_cibstatus,CIB status management and editing]]
+=== `cibstatus` - CIB status management and editing
+
+The `status` section of the CIB keeps the current status of nodes
+and resources. It is modified _only_ on events, i.e. when some
+resource operation is run or node status changes. For obvious
+reasons, the CRM has no user interface with which it is possible
+to affect the status section. From the user's point of view, the
+status section is essentially a read-only part of the CIB. The
+current status is never even written to disk, though it is
+available in the PE (Policy Engine) input files which represent
+the history of cluster motions. The current status may be read
+using the +cibadmin -Q+ command.
+
+It may sometimes be of interest to see how status changes would
+affect the Policy Engine. The set of `cibstatus` level commands
+allow the user to load status sections from various sources and
+then insert or modify resource operations or change nodes' state.
+
+The effect of those changes may then be observed by running the
+<<cmdhelp_configure_ptest,`ptest`>> command at the `configure` level
+or `simulate` and `run` commands at this level. The `ptest`
+runs with the user edited CIB whereas the latter two commands
+run with the CIB which was loaded along with the status section.
+
+The `simulate` and `run` commands as well as all status
+modification commands are implemented using `crm_simulate(8)`.
+
+[[cmdhelp_cibstatus_load,load the CIB status section]]
+==== `load`
+
+Load a status section from a file, a shadow CIB, or the running
+cluster. By default, the current (+live+) status section is
+modified. Note that if the +live+ status section is modified it
+is not going to be updated if the cluster status changes, because
+that would overwrite the user changes. To make `crm` drop changes
+and resume use of the running cluster status, run +load live+.
+
+All CIB shadow configurations contain the status section which is
+a snapshot of the status section taken at the time the shadow was
+created. Obviously, this status section doesn't have much to do
+with the running cluster status, unless the shadow CIB has just
+been created. Therefore, the `ptest` command by default uses the
+running cluster status section.
+
+Usage:
+...............
+load {<file>|shadow:<cib>|live}
+...............
+Example:
+...............
+load bug-12299.xml
+load shadow:test1
+...............
+
+[[cmdhelp_cibstatus_node,change node status]]
+==== `node`
+
+Change the node status. It is possible to throw a node out of
+the cluster, make it a member, or set its state to unclean.
+
++online+:: Set the +node_state+ `crmd` attribute to +online+
+and the +expected+ and +join+ attributes to +member+. The effect
+is that the node becomes a cluster member.
+
++offline+:: Set the +node_state+ `crmd` attribute to +offline+
+and the +expected+ attribute to empty. This makes the node
+cleanly removed from the cluster.
+
++unclean+:: Set the +node_state+ `crmd` attribute to +offline+
+and the +expected+ attribute to +member+. In this case the node
+has unexpectedly disappeared.
+
+Usage:
+...............
+node <node> {online|offline|unclean}
+...............
+Example:
+...............
+node xen-b unclean
+...............
+
+[[cmdhelp_cibstatus_op,edit outcome of a resource operation]]
+==== `op`
+
+Edit the outcome of a resource operation. This way you can
+tell CRM that it ran an operation and that the resource agent
+returned certain exit code. It is also possible to change the
+operation's status. In case the operation status is set to
+something other than +done+, the exit code is effectively
+ignored.
+
+Usage:
+...............
+op <operation> <resource> <exit_code> [<op_status>] [<node>]
+
+operation :: probe | monitor[:<n>] | start | stop |
+ promote | demote | notify | migrate_to | migrate_from
+exit_code :: <rc> | success | generic | args |
+ unimplemented | perm | installed | configured | not_running |
+ master | failed_master
+op_status :: pending | done | cancelled | timeout | notsupported | error
+
+n :: the monitor interval in seconds; if omitted, the first
+ recurring operation is referenced
+rc :: numeric exit code in range 0..9
+...............
+Example:
+...............
+op start d1 xen-b generic
+op start d1 xen-b 1
+op monitor d1 xen-b not_running
+op stop d1 xen-b 0 timeout
+...............
+
+[[cmdhelp_cibstatus_origin,display origin of the CIB status section]]
+==== `origin`
+
+Show the origin of the status section currently in use. This
+essentially shows the latest `load` argument.
+
+Usage:
+...............
+origin
+...............
+
+[[cmdhelp_cibstatus_quorum,set the quorum]]
+==== `quorum`
+
+Set the quorum value.
+
+Usage:
+...............
+quorum <bool>
+...............
+Example:
+...............
+quorum false
+...............
+
+[[cmdhelp_cibstatus_run,run policy engine]]
+==== `run`
+
+Run the policy engine with the edited status section.
+
+Add a string of +v+ characters to increase verbosity. Specify
++scores+ to see allocation scores also. +utilization+ turns on
+information about the remaining capacity of nodes.
+
+If you have graphviz installed and X11 session, `dotty(1)` is run
+to display the changes graphically.
+
+Usage:
+...............
+run [nograph] [v...] [scores] [utilization]
+...............
+Example:
+...............
+run
+...............
+
+[[cmdhelp_cibstatus_save,save the CIB status section]]
+==== `save`
+
+The current internal status section with whatever modifications
+were performed can be saved to a file or shadow CIB.
+
+If the file exists and contains a complete CIB, only the status
+section is going to be replaced and the rest of the CIB will
+remain intact. Otherwise, the current user edited configuration
+is saved along with the status section.
+
+Note that all modifications are saved in the source file as soon
+as they are run.
+
+Usage:
+...............
+save [<file>|shadow:<cib>]
+...............
+Example:
+...............
+save bug-12299.xml
+...............
+
+[[cmdhelp_cibstatus_show,show CIB status section]]
+==== `show`
+
+Show the current status section in the XML format. Brace yourself
+for some unreadable output. Add +changed+ option to get a human
+readable output of all changes.
+
+Usage:
+...............
+show [changed]
+...............
+
+[[cmdhelp_cibstatus_simulate,simulate cluster transition]]
+==== `simulate`
+
+Run the policy engine with the edited status section and simulate
+the transition.
+
+Add a string of +v+ characters to increase verbosity. Specify
++scores+ to see allocation scores also. +utilization+ turns on
+information about the remaining capacity of nodes.
+
+If you have graphviz installed and X11 session, `dotty(1)` is run
+to display the changes graphically.
+
+Usage:
+...............
+simulate [nograph] [v...] [scores] [utilization]
+...............
+Example:
+...............
+simulate
+...............
+
+[[cmdhelp_cibstatus_ticket,manage tickets]]
+==== `ticket`
+
+Modify the ticket status. Tickets can be granted and revoked.
+Granted tickets could be activated or put in standby.
+
+Usage:
+...............
+ticket <ticket> {grant|revoke|activate|standby}
+...............
+Example:
+...............
+ticket ticketA grant
+...............
+
+[[cmdhelp_assist,Configuration assistant]]
+=== `assist` - Configuration assistant
+
+The `assist` sublevel is a collection of helper
+commands that create or modify resources and
+constraints, to simplify the creation of certain
+configurations.
+
+For more information on individual commands, see
+the help text for those commands.
+
+[[cmdhelp_assist_template,Create template for primitives]]
+==== `template`
+
+This command takes a list of primitives as argument, and creates a new
+`rsc_template` for these primitives. It can only do this if the
+primitives do not already share a template and are of the same type.
+
+Usage:
+........
+template primitive-1 primitive-2 primitive-3
+........
+
+[[cmdhelp_assist_weak-bond,Create a weak bond between resources]]
+==== `weak-bond`
+
+A colocation between a group of resources says that the resources
+should be located together, but it also means that those resources are
+dependent on each other. If one of the resources fails, the others
+will be restarted.
+
+If this is not desired, it is possible to circumvent: By placing the
+resources in a non-sequential set and colocating the set with a dummy
+resource which is not monitored, the resources will be placed together
+but will have no further dependency on each other.
+
+This command creates both the constraint and the dummy resource needed
+for such a colocation.
+
+Usage:
+........
+weak-bond resource-1 resource-2
+........
+
+[[cmdhelp_maintenance,Maintenance mode commands]]
+=== `maintenance` - Maintenance mode commands
+
+Maintenance mode commands are commands that manipulate resources
+directly without going through the cluster infrastructure. Therefore,
+it is essential to ensure that the cluster does not attempt to monitor
+or manipulate the resources while these commands are being executed.
+
+To ensure this, these commands require that maintenance mode is set
+either for the particular resource, or for the whole cluster.
+
+[[cmdhelp_maintenance_action,Invoke a resource action]]
+==== `action`
+
+Invokes the given action for the resource. This is
+done directly via the resource agent, so the command must
+be issued while the cluster or the resource is in
+maintenance mode.
+
+Unless the action is `start` or `monitor`, the action must be invoked
+on the same node as where the resource is running. If the resource is
+running on multiple nodes, the command will fail.
+
+To use SSH for executing resource actions on multiple nodes, append
+`ssh` after the action name. This requires SSH access to be configured
+between the nodes and the parallax python package to be installed.
+
+Usage:
+...............
+action <rsc> <action>
+action <rsc> <action> ssh
+...............
+Example:
+...............
+action webserver reload
+action webserver monitor ssh
+...............
+
+[[cmdhelp_maintenance_off,Disable maintenance mode]]
+==== `off`
+
+Disables maintenances mode, either for the whole cluster
+or for the given resource.
+
+Usage:
+...............
+off
+off <rsc>
+...............
+Example:
+...............
+off rsc1
+...............
+
+[[cmdhelp_maintenance_on,Enable maintenance mode]]
+==== `on`
+
+Enables maintenances mode, either for the whole cluster
+or for the given resource.
+
+Usage:
+...............
+on
+on <rsc>
+...............
+Example:
+...............
+on rsc1
+...............
+
+[[cmdhelp_history,Cluster history]]
+=== `history` - Cluster history
+
+Examining Pacemaker's history is a particularly involved task. The
+number of subsystems to be considered, the complexity of the
+configuration, and the set of various information sources, most of
+which are not exactly human readable, keep analyzing resource or node
+problems accessible to only the most knowledgeable. Or, depending on
+the point of view, to the most persistent. The following set of
+commands has been devised in hope to make cluster history more
+accessible.
+
+Of course, looking at _all_ history could be time consuming regardless
+of how good the tools at hand are. Therefore, one should first say
+which period he or she wants to analyze. If not otherwise specified,
+the last hour is considered. Logs and other relevant information is
+collected using `crm report`. Since this process takes some time and
+we always need fresh logs, information is refreshed in a much faster
+way using the python parallax module. If +python-parallax+ is not
+found on the system, examining a live cluster is still possible --
+though not as comfortable.
+
+Apart from examining a live cluster, events may be retrieved from a
+report generated by `crm report` (see also the +-H+ option). In that
+case we assume that the period stretching the whole report needs to be
+investigated. Of course, it is still possible to further reduce the
+time range.
+
+If you have discovered an issue that you want to show someone else,
+you can use the `session pack` command to save the current session as
+a tarball, similar to those generated by `crm report`.
+
+In order to minimize the size of the tarball, and to make it easier
+for others to find the interesting events, it is recommended to limit
+the time frame which the saved session covers. This can be done using
+the `timeframe` command (example below).
+
+It is also possible to name the saved session using the `session save`
+command.
+
+Example:
+...............
+crm(live)history# limit "Jul 18 12:00" "Jul 18 12:30"
+crm(live)history# session save strange_restart
+crm(live)history# session pack
+Report saved in .../strange_restart.tar.bz2
+crm(live)history#
+...............
+
+[[cmdhelp_history_detail,set the level of detail shown]]
+==== `detail`
+
+How much detail to show from the logs. Valid detail levels are either
+`0` or `1`, where `1` is the highest detail level. The default detail
+level is `0`.
+
+Usage:
+...............
+detail <detail_level>
+
+detail_level :: small integer (defaults to 0)
+...............
+Example:
+...............
+detail 1
+...............
+
+[[cmdhelp_history_diff,cluster states/transitions difference]]
+==== `diff`
+
+A transition represents a change in cluster configuration or
+state. Use `diff` to see what has changed between two
+transitions.
+
+If you want to specify the current cluster configuration and
+status, use the string +live+.
+
+Normally, the first transition specified should be the one which
+is older, but we are not going to enforce that.
+
+Note that a single configuration update may result in more than
+one transition.
+
+Usage:
+...............
+diff <pe> <pe> [status] [html]
+
+pe :: <number>|<index>|<file>|live
+...............
+Examples:
+...............
+diff 2066 2067
+diff pe-input-2080.bz2 live status
+...............
+
+[[cmdhelp_history_events,Show events in log]]
+==== `events`
+
+By analysing the log output and looking for particular
+patterns, the `events` command helps sifting through
+the logs to find when particular events like resources
+changing state or node failure may have occurred.
+
+This can be used to generate a combined list of events
+from all nodes.
+
+Usage:
+...............
+events
+...............
+
+Example:
+...............
+events
+...............
+
+[[cmdhelp_history_exclude,exclude log messages]]
+==== `exclude`
+
+If a log is infested with irrelevant messages, those messages may
+be excluded by specifying a regular expression. The regular
+expressions used are Python extended. This command is additive.
+To drop all regular expressions, use +exclude clear+. Run
+`exclude` only to see the current list of regular expressions.
+Excludes are saved along with the history sessions.
+
+Usage:
+...............
+exclude [<regex>|clear]
+...............
+Example:
+...............
+exclude kernel.*ocfs2
+...............
+
+[[cmdhelp_history_graph,generate a directed graph from the PE file]]
+==== `graph`
+
+Create a graphviz graphical layout from the PE file (the
+transition). Every transition contains the cluster configuration
+which was active at the time. See also <<cmdhelp_configure_graph,generate a directed graph
+from configuration>>.
+
+Usage:
+...............
+graph <pe> [<gtype> [<file> [<img_format>]]]
+
+gtype :: dot
+img_format :: `dot` output format (see the +-T+ option)
+...............
+Example:
+...............
+graph -1
+graph 322 dot clu1.conf.dot
+graph 322 dot clu1.conf.svg svg
+...............
+
+[[cmdhelp_history_info,Cluster information summary]]
+==== `info`
+
+The `info` command provides a summary of the information source, which
+can be either a live cluster snapshot or a previously generated
+report.
+
+Usage:
+...............
+info
+...............
+Example:
+...............
+info
+...............
+
+[[cmdhelp_history_latest,show latest news from the cluster]]
+==== `latest`
+
+The `latest` command shows a bit of recent history, more
+precisely whatever happened since the last cluster change (the
+latest transition). If the transition is running, the shell will
+first wait until it finishes.
+
+Usage:
+...............
+latest
+...............
+Example:
+...............
+latest
+...............
+
+[[cmdhelp_history_limit,limit timeframe to be examined]]
+==== `limit` (`timeframe`)
+
+This command can be used to modify the time span to examine. All
+history commands look at events within a certain time span.
+
+For the `live` source, the default time span is the _last hour_.
+
+There is no time span limit for the `hb_report` source.
+
+The time period is parsed by the `dateutil` python module. It
+covers a wide range of date formats. For instance:
+
+- 3:00 (today at 3am)
+- 15:00 (today at 3pm)
+- 2010/9/1 2pm (September 1st 2010 at 2pm)
+
+For more examples of valid time/date statements, please refer to the
+`python-dateutil` documentation:
+
+- https://dateutil.readthedocs.org/[dateutil.readthedocs.org]
+
+If the dateutil module is not available, then the time is parsed using
+strptime and only the kind as printed by `date(1)` is allowed:
+
+- Tue Sep 15 20:46:27 CEST 2010
+
+Usage:
+...............
+limit [<from_time>] [<to_time>]
+...............
+Examples:
+...............
+limit 10:15
+limit 15h22m 16h
+limit "Sun 5 20:46" "Sun 5 22:00"
+...............
+
+[[cmdhelp_history_log,log content]]
+==== `log`
+
+Show messages logged on one or more nodes. Leaving out a node
+name produces combined logs of all nodes. Messages are sorted by
+time and, if the terminal emulations supports it, displayed in
+different colours depending on the node to allow for easier
+reading.
+
+The sorting key is the timestamp as written by syslog which
+normally has the maximum resolution of one second. Obviously,
+messages generated by events which share the same timestamp may
+not be sorted in the same way as they happened. Such close events
+may actually happen fairly often.
+
+Usage:
+...............
+log [<node> [<node> ...] ]
+...............
+Example:
+...............
+log node-a
+...............
+
+[[cmdhelp_history_node,node events]]
+==== `node`
+
+Show important events that happened on a node. Important events
+are node lost and join, standby and online, and fence. Use either
+node names or extended regular expressions.
+
+Usage:
+...............
+node <node> [<node> ...]
+...............
+Example:
+...............
+node node1
+...............
+
+[[cmdhelp_history_peinputs,list or get PE input files]]
+==== `peinputs`
+
+Every event in the cluster results in generating one or more
+Policy Engine (PE) files. These files describe future motions of
+resources. The files are listed as full paths in the current
+report directory. Add +v+ to also see the creation time stamps.
+
+Usage:
+...............
+peinputs [{<range>|<number>} ...] [v]
+
+range :: <n1>:<n2>
+...............
+Example:
+...............
+peinputs
+peinputs 440:444 446
+peinputs v
+...............
+
+[[cmdhelp_history_refresh,refresh live report]]
+==== `refresh`
+
+This command makes sense only for the +live+ source and makes
+`crm` collect the latest logs and other relevant information from
+the logs. If you want to make a completely new report, specify
++force+.
+
+Usage:
+...............
+refresh [force]
+...............
+
+[[cmdhelp_history_resource,resource events]]
+==== `resource`
+
+Show actions and any failures that happened on all specified
+resources on all nodes. Normally, one gives resource names as
+arguments, but it is also possible to use extended regular
+expressions. Note that neither groups nor clones or master/slave
+names are ever logged. The resource command is going to expand
+all of these appropriately, so that clone instances or resources
+which are part of a group are shown.
+
+Usage:
+...............
+resource <rsc> [<rsc> ...]
+...............
+Example:
+...............
+resource bigdb public_ip
+resource my_.*_db2
+resource ping_clone
+...............
+
+[[cmdhelp_history_session,manage history sessions]]
+==== `session`
+
+Sometimes you may want to get back to examining a particular
+history period or bug report. In order to make that easier, the
+current settings can be saved and later retrieved.
+
+If the current history being examined is coming from a live
+cluster the logs, PE inputs, and other files are saved too,
+because they may disappear from nodes. For the existing reports
+coming from `hb_report`, only the directory location is saved
+(not to waste space).
+
+A history session may also be packed into a tarball which can
+then be sent to support.
+
+Leave out subcommand to see the current session.
+
+Usage:
+...............
+session [{save|load|delete} <name> | pack [<name>] | update | list]
+...............
+Examples:
+...............
+session save bnc966622
+session load rsclost-2
+session list
+...............
+
+[[cmdhelp_history_setnodes,set the list of cluster nodes]]
+==== `setnodes`
+
+In case the host this program runs on is not part of the cluster,
+it is necessary to set the list of nodes.
+
+Usage:
+...............
+setnodes node <node> [<node> ...]
+...............
+Example:
+...............
+setnodes node_a node_b
+...............
+
+[[cmdhelp_history_show,show status or configuration of the PE input file]]
+==== `show`
+
+Every transition is saved as a PE file. Use this command to
+render that PE file either as configuration or status. The
+configuration output is the same as `crm configure show`.
+
+Usage:
+...............
+show <pe> [status]
+
+pe :: <number>|<index>|<file>|live
+...............
+Examples:
+...............
+show 2066
+show pe-input-2080.bz2 status
+...............
+
+[[cmdhelp_history_source,set source to be examined]]
+==== `source`
+
+Events to be examined can come from the current cluster or from a
+`hb_report` report. This command sets the source. `source live`
+sets source to the running cluster and system logs. If no source
+is specified, the current source information is printed.
+
+In case a report source is specified as a file reference, the file
+is going to be unpacked in place where it resides. This directory
+is not removed on exit.
+
+Usage:
+...............
+source [<dir>|<file>|live]
+...............
+Examples:
+...............
+source live
+source /tmp/customer_case_22.tar.bz2
+source /tmp/customer_case_22
+source
+...............
+
+[[cmdhelp_history_transition,show transition]]
+==== `transition`
+
+This command will print actions planned by the PE and run
+graphviz (`dotty`) to display a graphical representation of the
+transition. Of course, for the latter an X11 session is required.
+This command invokes `ptest(8)` in background.
+
+The +showdot+ subcommand runs graphviz (`dotty`) to display a
+graphical representation of the +.dot+ file which has been
+included in the report. Essentially, it shows the calculation
+produced by `pengine` which is installed on the node where the
+report was produced. In optimal case this output should not
+differ from the one produced by the locally installed `pengine`.
+
+The `log` subcommand shows the full log for the duration of the
+transition.
+
+A transition can also be saved to a CIB shadow for further
+analysis or use with `cib` or `configure` commands (use the
+`save` subcommand). The shadow file name defaults to the name of
+the PE input file.
+
+If the PE input file number is not provided, it defaults to the
+last one, i.e. the last transition. The last transition can also
+be referenced with number 0. If the number is negative, then the
+corresponding transition relative to the last one is chosen.
+
+If there are warning and error PE input files or different nodes
+were the DC in the observed timeframe, it may happen that PE
+input file numbers collide. In that case provide some unique part
+of the path to the file.
+
+After the `ptest` output, logs about events that happened during
+the transition are printed.
+
+The `tags` subcommand scans the logs for the transition and return a
+list of key events during that transition. For example, the tag
++error+ will be returned if there are any errors logged during the
+transition.
+
+Usage:
+...............
+transition [<number>|<index>|<file>] [nograph] [v...] [scores] [actions] [utilization]
+transition showdot [<number>|<index>|<file>]
+transition log [<number>|<index>|<file>]
+transition save [<number>|<index>|<file> [name]]
+transition tags [<number>|<index>|<file>]
+...............
+Examples:
+...............
+transition
+transition 444
+transition -1
+transition pe-error-3.bz2
+transition node-a/pengine/pe-input-2.bz2
+transition showdot 444
+transition log
+transition save 0 enigma-22
+...............
+
+[[cmdhelp_history_transitions,List transitions]]
+==== `transitions`
+
+A transition represents a change in cluster configuration or
+state. This command lists the transitions in the current timeframe.
+
+Usage:
+...............
+transitions
+...............
+Example:
+...............
+transitions
+...............
+
+
+[[cmdhelp_history_wdiff,cluster states/transitions difference]]
+==== `wdiff`
+
+A transition represents a change in cluster configuration or
+state. Use `wdiff` to see what has changed between two
+transitions as word differences on a line-by-line basis.
+
+If you want to specify the current cluster configuration and
+status, use the string +live+.
+
+Normally, the first transition specified should be the one which
+is older, but we are not going to enforce that.
+
+Note that a single configuration update may result in more than
+one transition.
+
+Usage:
+...............
+wdiff <pe> <pe> [status]
+
+pe :: <number>|<index>|<file>|live
+...............
+Examples:
+...............
+wdiff 2066 2067
+wdiff pe-input-2080.bz2 live status
+...............
+
+[[cmdhelp_root_report,Create cluster status report]]
+=== `report`
+
+Interface to a tool for creating a cluster report. A report is an
+archive containing log files, configuration files, system information
+and other relevant data for a given time period. This is a useful tool
+for collecting data to attach to bug reports, or for detecting the
+root cause of errors resulting in resource failover, for example.
+
+See `crmsh_hb_report(8)` for more details on arguments,
+or call `crm report -h`
+
+Usage:
+...............
+report -f {time|"cts:"testnum} [-t time] [-u user] [-l file]
+ [-n nodes] [-E files] [-p patt] [-L patt] [-e prog]
+ [-MSDZAVsvhd] [dest]
+...............
+
+Examples:
+...............
+report -f 2pm report_1
+report -f "2007/9/5 12:30" -t "2007/9/5 14:00" report_2
+report -f 1:00 -t 3:00 -l /var/log/cluster/ha-debug report_3
+report -f "09sep07 2:00" -u hbadmin report_4
+report -f 18:00 -p "usern.*" -p "admin.*" report_5
+report -f cts:133 ctstest_133
+...............
+
+=== `end` (`cd`, `up`)
+
+The `end` command ends the current level and the user moves to
+the parent level. This command is available everywhere.
+
+Usage:
+...............
+end
+...............
+
+=== `help`
+
+The `help` command prints help for the current level or for the
+specified topic (command). This command is available everywhere.
+
+Usage:
+...............
+help [<topic>]
+...............
+
+=== `quit` (`exit`, `bye`)
+
+Leave the program.
+
+BUGS
+----
+Even though all sensible configurations (and most of those that
+are not) are going to be supported by the crm shell, I suspect
+that it may still happen that certain XML constructs may confuse
+the tool. When that happens, please file a bug report.
+
+The crm shell will not try to update the objects it does not
+understand. Of course, it is always possible to edit such objects
+in the XML format.
+
+AUTHORS
+-------
+Dejan Muhamedagic, <dejan@suse.de>
+Kristoffer Gronlund <kgronlund@suse.com>
+and many OTHERS
+
+SEE ALSO
+--------
+crm_resource(8), crm_attribute(8), crm_mon(8), cib_shadow(8),
+ptest(8), dotty(1), crm_simulate(8), cibadmin(8)
+
+
+COPYING
+-------
+Copyright \(C) 2008-2013 Dejan Muhamedagic.
+Copyright \(C) 2013 Kristoffer Gronlund.
+
+Free use of this software is granted under the terms of the GNU General Public License (GPL).
+
+//////////////////////
+ vim:ts=4:sw=4:expandtab:
+//////////////////////
diff --git a/doc/website-v1/man-4.3.adoc b/doc/website-v1/man-4.3.adoc
new file mode 100644
index 0000000..2b82298
--- /dev/null
+++ b/doc/website-v1/man-4.3.adoc
@@ -0,0 +1,5160 @@
+:man source: crm
+:man version: 4.0.0
+:man manual: crmsh documentation
+
+crm(8)
+======
+
+NAME
+----
+crm - Pacemaker command line interface for configuration and management
+
+
+SYNOPSIS
+--------
+*crm* [OPTIONS] [SUBCOMMAND ARGS...]
+
+
+[[topics_Description,Program description]]
+DESCRIPTION
+-----------
+The `crm` shell is a command-line based cluster configuration and
+management tool. Its goal is to assist as much as possible with the
+configuration and maintenance of Pacemaker-based High Availability
+clusters.
+
+For more information on Pacemaker itself, see http://clusterlabs.org/.
+
+`crm` works both as a command-line tool to be called directly from the
+system shell, and as an interactive shell with extensive tab
+completion and help.
+
+The primary focus of the `crm` shell is to provide a simplified and
+consistent interface to Pacemaker, but it also provides tools for
+managing the creation and configuration of High Availability clusters
+from scratch. To learn more about this aspect of `crm`, see the
+`cluster` section below.
+
+The `crm` shell can be used to manage every aspect of configuring and
+maintaining a cluster. It provides a simplified line-based syntax on
+top of the XML configuration format used by Pacemaker, commands for
+starting and stopping resources, tools for exploring the history of a
+cluster including log scraping and a set of cluster scripts useful for
+automating the setup and installation of services on the cluster
+nodes.
+
+The `crm` shell is line oriented: every command must start and finish
+on the same line. It is possible to use a continuation character (+\+)
+to write one command in two or more lines. The continuation character
+is commonly used when displaying configurations.
+
+[[topics_CommandLine,Command line options]]
+OPTIONS
+-------
+*-f, --file*='FILE'::
+ Load commands from the given file. If a dash +-+ is used in place
+ of a file name, `crm` will read commands from the shell standard
+ input (`stdin`).
+
+*-c, --cib*='CIB'::
+ Start the session using the given shadow CIB file.
+ Equivalent to +cib use <CIB>+.
+
+*-D, --display=*'OUTPUT_TYPE'::
+ Choose one of the output options: +plain+, +color-always+, +color+,
+ or +uppercase+. The default is +color+ if the terminal emulation
+ supports colors. Otherwise, +plain+ is used.
+
+*-F, --force*::
+ Make `crm` proceed with applying changes where it would normally
+ ask the user to confirm before proceeding. This option is mainly
+ useful in scripts, and should be used with care.
+
+*-w, --wait*::
+ Make `crm` wait for the cluster transition to finish (for the
+ changes to take effect) after each processed line.
+
+*-H, --history*='DIR|FILE|SESSION'::
+ A directory or file containing a cluster report to load
+ into the `history` commands, or the name of a previously
+ saved history session.
+
+*-h, --help*::
+ Print help page.
+
+*--version*::
+ Print crmsh version and build information (Mercurial Hg changeset
+ hash).
+
+*-d, --debug*::
+ Print verbose debugging information.
+
+*-R, --regression-tests*::
+ Enables extra verbose trace logging used by the regression
+ tests. Logs all external calls made by crmsh.
+
+*--scriptdir*='DIR'::
+ Extra directory where crm looks for cluster scripts, or a list of
+ directories separated by semi-colons (e.g. +/dir1;/dir2;etc.+).
+
+*-o, --opt*='OPTION=VALUE'::
+ Set crmsh option temporarily. If the options are saved using
+ +options save+ then the value passed here will also be saved.
+ Multiple options can be set by using +-o+ multiple times.
+
+[[topics_Introduction,Introduction]]
+== Introduction
+
+This section of the user guide covers general topics about the user
+interface and describes some of the features of `crmsh` in detail.
+
+[[topics_Introduction_Interface,User interface]]
+=== User interface
+
+The main purpose of `crmsh` is to provide a simple yet powerful
+interface to the cluster stack. There are two main modes of operation
+with the user interface of `crmsh`:
+
+* Command line (single-shot) use - Use `crm` as a regular UNIX command
+ from your usual shell. `crm` has full bash completion built in, so
+ using it in this manner should be as comfortable and familiar as
+ using any other command-line tool.
+
+* Interactive mode - By calling `crm` without arguments, or by calling
+ it with only a sublevel as argument, `crm` enters the interactive
+ mode. In this mode, it acts as its own command shell, which
+ remembers which sublevel you are currently in and allows for rapid
+ and convenient execution of multiple commands within the same
+ sublevel. This mode also has full tab completion, as well as
+ built-in interactive help and syntax highlighting.
+
+Here are a few examples of using `crm` both as a command-line tool and
+as an interactive shell:
+
+.Command line (one-shot) use:
+........
+# crm resource stop www_app
+........
+
+.Interactive use:
+........
+# crm
+crm(live)# resource
+crm(live)resource# unmanage tetris_1
+crm(live)resource# up
+crm(live)# node standby node4
+........
+
+.Cluster configuration:
+........
+# crm configure<<EOF
+ #
+ # resources
+ #
+ primitive disk0 iscsi \
+ params portal=192.168.2.108:3260 target=iqn.2008-07.com.suse:disk0
+ primitive fs0 Filesystem \
+ params device=/dev/disk/by-label/disk0 directory=/disk0 fstype=ext3
+ primitive internal_ip IPaddr params ip=192.168.1.101
+ primitive apache apache \
+ params configfile=/disk0/etc/apache2/site0.conf
+ primitive apcfence stonith:apcsmart \
+ params ttydev=/dev/ttyS0 hostlist="node1 node2" \
+ op start timeout=60s
+ primitive pingd pingd \
+ params name=pingd dampen=5s multiplier=100 host_list="r1 r2"
+ #
+ # monitor apache and the UPS
+ #
+ monitor apache 60s:30s
+ monitor apcfence 120m:60s
+ #
+ # cluster layout
+ #
+ group internal_www \
+ disk0 fs0 internal_ip apache
+ clone fence apcfence \
+ meta globally-unique=false clone-max=2 clone-node-max=1
+ clone conn pingd \
+ meta globally-unique=false clone-max=2 clone-node-max=1
+ location node_pref internal_www \
+ rule 50: #uname eq node1 \
+ rule pingd: defined pingd
+ #
+ # cluster properties
+ #
+ property stonith-enabled=true
+ commit
+EOF
+........
+
+The `crm` interface is hierarchical, with commands organized into
+separate levels by functionality. To list the available levels and
+commands, either execute +help <level>+, or, if at the top level of
+the shell, simply typing `help` will provide an overview of all
+available levels and commands.
+
+The +(live)+ string in the `crm` prompt signifies that the current CIB
+in use is the cluster live configuration. It is also possible to
+work with so-called <<topics_Features_Shadows,shadow CIBs>>. These are separate, inactive
+configurations stored in files, that can be applied and thereby
+replace the live configuration at any time.
+
+[[topics_Introduction_Completion,Tab completion]]
+=== Tab completion
+
+The `crm` makes extensive use of tab completion. The completion
+is both static (i.e. for `crm` commands) and dynamic. The latter
+takes into account the current status of the cluster or
+information from installed resource agents. Sometimes, completion
+may also be used to get short help on resource parameters. Here
+are a few examples:
+
+...............
+crm(live)resource# <TAB><TAB>
+ban demote maintenance param scores trace
+cd failcount manage promote secret unmanage
+cleanup help meta quit start untrace
+clear locate move refresh status up
+constraints ls operations restart stop utilization
+
+crm(live)configure# primitive fence-1 <TAB><TAB>
+lsb: ocf: service: stonith: systemd:
+
+crm(live)configure# primitive fence-1 stonith:<TAB><TAB>
+apcmaster external/ippower9258 fence_legacy
+apcmastersnmp external/kdumpcheck ibmhmc
+apcsmart external/libvirt ipmilan
+
+crm(live)configure# primitive fence-1 stonith:ipmilan params <TAB><TAB>
+auth= hostname= ipaddr= login= password= port= priv=
+
+crm(live)configure# primitive fence-1 stonith:ipmilan params auth=<TAB><TAB>
+auth* (string)
+ The authorization type of the IPMI session ("none", "straight", "md2", or "md5")
+...............
+
+`crmsh` also comes with bash completion usable directly from the
+system shell. This should be installed automatically with the command
+itself.
+
+[[topics_Introduction_Shorthand,Shorthand syntax]]
+=== Shorthand syntax
+
+When using the `crm` shell to manage clusters, you will end up typing
+a lot of commands many times over. Clear command names like
++configure+ help in understanding and learning to use the cluster
+shell, but is easy to misspell and is tedious to type repeatedly. The
+interactive mode and tab completion both help with this, but the `crm`
+shell also has the ability to understand a variety of shorthand
+aliases for all of the commands.
+
+For example, instead of typing `crm status`, you can type `crm st` or
+`crm stat`. Instead of `crm configure` you can type `crm cfg` or even
+`crm cf`. `crm resource` can be shorted as `crm rsc`, and so on.
+
+The exact list of accepted aliases is too long to print in full, but
+experimentation and typos should help in discovering more of them.
+
+[[topics_Features,Features]]
+== Features
+
+The feature set of crmsh covers a wide range of functionality, and
+understanding how and when to use the various features of the shell
+can be difficult. This section of the guide describes some of the
+features and use cases of `crmsh` in more depth. The intention is to
+provide a deeper understanding of these features, but also to serve as
+a guide to using them.
+
+[[topics_Features_Shadows,Shadow CIB usage]]
+=== Shadow CIB usage
+
+A Shadow CIB is a normal cluster configuration stored in a file.
+They may be manipulated in much the same way as the _live_ CIB, with
+the key difference that changes to a shadow CIB have no effect on the
+actual cluster resources. An administrator may choose to apply any of
+them to the cluster, thus replacing the running configuration with the
+one found in the shadow CIB.
+
+The `crm` prompt always contains the name of the configuration which
+is currently in use, or the string _live_ if using the live cluster
+configuration.
+
+When editing the configuration in the `configure` level, no changes
+are actually applied until the `commit` command is executed. It is
+possible to start editing a configuration as usual, but instead of
+committing the changes to the active CIB, save them to a shadow CIB.
+
+The following example `configure` session demonstrates how this can be
+done:
+...............
+crm(live)configure# cib new test-2
+INFO: test-2 shadow CIB created
+crm(test-2)configure# commit
+...............
+
+[[topics_Features_Checks,Configuration semantic checks]]
+=== Configuration semantic checks
+
+Resource definitions may be checked against the meta-data
+provided with the resource agents. These checks are currently
+carried out:
+
+- are required parameters set
+- existence of defined parameters
+- timeout values for operations
+
+The parameter checks are obvious and need no further explanation.
+Failures in these checks are treated as configuration errors.
+
+The timeouts for operations should be at least as long as those
+recommended in the meta-data. Too short timeout values are a
+common mistake in cluster configurations and, even worse, they
+often slip through if cluster testing was not thorough. Though
+operation timeouts issues are treated as warnings, make sure that
+the timeouts are usable in your environment. Note also that the
+values given are just _advisory minimum_---your resources may
+require longer timeouts.
+
+User may tune the frequency of checks and the treatment of errors
+by the <<cmdhelp_options_check-frequency,`check-frequency`>> and
+<<cmdhelp_options_check-mode,`check-mode`>> preferences.
+
+Note that if the +check-frequency+ is set to +always+ and the
++check-mode+ to +strict+, errors are not tolerated and such
+configuration cannot be saved.
+
+[[topics_Features_Templates,Configuration templates]]
+=== Configuration templates
+
+.Deprecation note
+****************************
+Configuration templates have been deprecated in favor of the more
+capable `cluster scripts`. To learn how to use cluster scripts, see
+the dedicated documentation on the `crmsh` website at
+http://crmsh.github.io/, or in the <<cmdhelp_script,Script section>>.
+****************************
+
+Configuration templates are ready made configurations created by
+cluster experts. They are designed in such a way so that users
+may generate valid cluster configurations with minimum effort.
+If you are new to Pacemaker, templates may be the best way to
+start.
+
+We will show here how to create a simple yet functional Apache
+configuration:
+...............
+# crm configure
+crm(live)configure# template
+crm(live)configure template# list templates
+apache filesystem virtual-ip
+crm(live)configure template# new web <TAB><TAB>
+apache filesystem virtual-ip
+crm(live)configure template# new web apache
+INFO: pulling in template apache
+INFO: pulling in template virtual-ip
+crm(live)configure template# list
+web2-d web2 vip2 web3 vip web
+...............
+
+We enter the `template` level from `configure`. Use the `list`
+command to show templates available on the system. The `new`
+command creates a configuration from the +apache+ template. You
+can use tab completion to pick templates. Note that the apache
+template depends on a virtual IP address which is automatically
+pulled along. The `list` command shows the just created +web+
+configuration, among other configurations (I hope that you,
+unlike me, will use more sensible and descriptive names).
+
+The `show` command, which displays the resulting configuration,
+may be used to get an idea about the minimum required changes
+which have to be done. All +ERROR+ messages show the line numbers
+in which the respective parameters are to be defined:
+...............
+crm(live)configure template# show
+ERROR: 23: required parameter ip not set
+ERROR: 61: required parameter id not set
+ERROR: 65: required parameter configfile not set
+crm(live)configure template# edit
+...............
+
+The `edit` command invokes the preferred text editor with the
++web+ configuration. At the top of the file, the user is advised
+how to make changes. A good template should require from the user
+to specify only parameters. For example, the +web+ configuration
+we created above has the following required and optional
+parameters (all parameter lines start with +%%+):
+...............
+$ grep -n ^%% ~/.crmconf/web
+23:%% ip
+31:%% netmask
+35:%% lvs_support
+61:%% id
+65:%% configfile
+71:%% options
+76:%% envfiles
+...............
+
+These lines are the only ones that should be modified. Simply
+append the parameter value at the end of the line. For instance,
+after editing this template, the result could look like this (we
+used tabs instead of spaces to make the values stand out):
+...............
+$ grep -n ^%% ~/.crmconf/web
+23:%% ip 192.168.1.101
+31:%% netmask
+35:%% lvs_support
+61:%% id websvc
+65:%% configfile /etc/apache2/httpd.conf
+71:%% options
+76:%% envfiles
+...............
+
+As you can see, the parameter line format is very simple:
+...............
+%% <name> <value>
+...............
+
+After editing the file, use `show` again to display the
+configuration:
+...............
+crm(live)configure template# show
+primitive virtual-ip IPaddr \
+ params ip=192.168.1.101
+primitive apache apache \
+ params configfile="/etc/apache2/httpd.conf"
+monitor apache 120s:60s
+group websvc \
+ apache virtual-ip
+...............
+
+The target resource of the apache template is a group which we
+named +websvc+ in this sample session.
+
+This configuration looks exactly as you could type it at the
+`configure` level. The point of templates is to save you some
+typing. It is important, however, to understand the configuration
+produced.
+
+Finally, the configuration may be applied to the current
+crm configuration (note how the configuration changed slightly,
+though it is still equivalent, after being digested at the
+`configure` level):
+...............
+crm(live)configure template# apply
+crm(live)configure template# cd ..
+crm(live)configure# show
+node xen-b
+node xen-c
+primitive apache apache \
+ params configfile="/etc/apache2/httpd.conf" \
+ op monitor interval=120s timeout=60s
+primitive virtual-ip IPaddr \
+ params ip=192.168.1.101
+group websvc apache virtual-ip
+...............
+
+Note that this still does not commit the configuration to the CIB
+which is used in the shell, either the running one (+live+) or
+some shadow CIB. For that you still need to execute the `commit`
+command.
+
+To complete our example, we should also define the preferred node
+to run the service:
+
+...............
+crm(live)configure# location websvc-pref websvc 100: xen-b
+...............
+
+If you are not happy with some resource names which are provided
+by default, you can rename them now:
+
+...............
+crm(live)configure# rename virtual-ip intranet-ip
+crm(live)configure# show
+node xen-b
+node xen-c
+primitive apache apache \
+ params configfile="/etc/apache2/httpd.conf" \
+ op monitor interval=120s timeout=60s
+primitive intranet-ip IPaddr \
+ params ip=192.168.1.101
+group websvc apache intranet-ip
+location websvc-pref websvc 100: xen-b
+...............
+
+To summarize, working with templates typically consists of the
+following steps:
+
+- `new`: create a new configuration from templates
+- `edit`: define parameters, at least the required ones
+- `show`: see if the configuration is valid
+- `apply`: apply the configuration to the `configure` level
+
+[[topics_Features_Testing,Resource testing]]
+=== Resource testing
+
+The amount of detail in a cluster makes all configurations prone
+to errors. By far the largest number of issues in a cluster is
+due to bad resource configuration. The shell can help quickly
+diagnose such problems. And considerably reduce your keyboard
+wear.
+
+Let's say that we entered the following configuration:
+...............
+node xen-b
+node xen-c
+node xen-d
+primitive fencer stonith:external/libvirt \
+ params hypervisor_uri="qemu+tcp://10.2.13.1/system" \
+ hostlist="xen-b xen-c xen-d" \
+ op monitor interval=2h
+primitive svc Xinetd \
+ params service=systat \
+ op monitor interval=30s
+primitive intranet-ip IPaddr2 \
+ params ip=10.2.13.100 \
+ op monitor interval=30s
+primitive apache apache \
+ params configfile="/etc/apache2/httpd.conf" \
+ op monitor interval=120s timeout=60s
+group websvc apache intranet-ip
+location websvc-pref websvc 100: xen-b
+...............
+
+Before typing `commit` to submit the configuration to the cib we
+can make sure that all resources are usable on all nodes:
+...............
+crm(live)configure# rsctest websvc svc fencer
+...............
+
+It is important that resources being tested are not running on
+any nodes. Otherwise, the `rsctest` command will refuse to do
+anything. Of course, if the current configuration resides in a
+CIB shadow, then a `commit` is irrelevant. The point being that
+resources are not running on any node.
+
+.Note on stopping all resources
+****************************
+Alternatively to not committing a configuration, it is also
+possible to tell Pacemaker not to start any resources:
+
+...............
+crm(live)configure# property stop-all-resources=yes
+...............
+Almost none---resources of class stonith are still started. But
+shell is not as strict when it comes to stonith resources.
+****************************
+
+Order of resources is significant insofar that a resource depends
+on all resources to its left. In most configurations, it's
+probably practical to test resources in several runs, based on
+their dependencies.
+
+Apart from groups, `crm` does not interpret constraints and
+therefore knows nothing about resource dependencies. It also
+doesn't know if a resource can run on a node at all in case of an
+asymmetric cluster. It is up to the user to specify a list of
+eligible nodes if a resource is not meant to run on every node.
+
+[[topics_Features_Security,Access Control Lists (ACL)]]
+=== Access Control Lists (ACL)
+
+.Note on ACLs in Pacemaker 1.1.12
+****************************
+The support for ACLs has been revised in Pacemaker version 1.1.12 and
+up. Depending on which version you are using, the information in this
+section may no longer be accurate. Look for the `acl_target`
+configuration element for more details on the new syntax.
+****************************
+
+By default, the users from the +haclient+ group have full access
+to the cluster (or, more precisely, to the CIB). Access control
+lists allow for finer access control to the cluster.
+
+Access control lists consist of an ordered set of access rules.
+Each rule allows read or write access or denies access
+completely. Rules are typically combined to produce a specific
+role. Then, users may be assigned a role.
+
+For instance, this is a role which defines a set of rules
+allowing management of a single resource:
+
+...............
+role bigdb_admin \
+ write meta:bigdb:target-role \
+ write meta:bigdb:is-managed \
+ write location:bigdb \
+ read ref:bigdb
+...............
+
+The first two rules allow modifying the +target-role+ and
++is-managed+ meta attributes which effectively enables users in
+this role to stop/start and manage/unmanage the resource. The
+constraints write access rule allows moving the resource around.
+Finally, the user is granted read access to the resource
+definition.
+
+For proper operation of all Pacemaker programs, it is advisable
+to add the following role to all users:
+
+...............
+role read_all \
+ read cib
+...............
+
+For finer grained read access try with the rules listed in the
+following role:
+
+...............
+role basic_read \
+ read node attribute:uname \
+ read node attribute:type \
+ read property \
+ read status
+...............
+
+It is however possible that some Pacemaker programs (e.g.
+`ptest`) may not function correctly if the whole CIB is not
+readable.
+
+Some of the ACL rules in the examples above are expanded by the
+shell to XPath specifications. For instance,
++meta:bigdb:target-role+ expands to:
+
+........
+//primitive[@id='bigdb']/meta_attributes/nvpair[@name='target-role']
+........
+
+You can see the expansion by showing XML:
+
+...............
+crm(live) configure# show xml bigdb_admin
+...
+<acls>
+ <acl_role id="bigdb_admin">
+ <write id="bigdb_admin-write"
+ xpath="//primitive[@id='bigdb']/meta_attributes/nvpair[@name='target-role']"/>
+...............
+
+Many different XPath expressions can have equal meaning. For
+instance, the following two are equal, but only the first one is
+going to be recognized as shortcut:
+
+...............
+//primitive[@id='bigdb']/meta_attributes/nvpair[@name='target-role']
+//resources/primitive[@id='bigdb']/meta_attributes/nvpair[@name='target-role']
+...............
+
+XPath is a powerful language, but you should try to keep your ACL
+xpaths simple and the builtin shortcuts should be used whenever
+possible.
+
+[[topics_Features_Resourcesets,Syntax: Resource sets]]
+=== Syntax: Resource sets
+
+Using resource sets can be a bit confusing unless one knows the
+details of the implementation in Pacemaker as well as how to interpret
+the syntax provided by `crmsh`.
+
+Three different types of resource sets are provided by `crmsh`, and
+each one implies different values for the two resource set attributes,
++sequential+ and +require-all+.
+
++sequential+::
+ If false, the resources in the set do not depend on each other
+ internally. Setting +sequential+ to +true+ implies a strict order of
+ dependency within the set.
+
++require-all+::
+ If false, only one resource in the set is required to fulfil the
+ requirements of the set. The set of A, B and C with +require-all+
+ set to +false+ is be read as "A OR B OR C" when its dependencies
+ are resolved.
+
+The three types of resource sets modify the attributes in the
+following way:
+
+1. Implicit sets (no brackets). +sequential=true+, +require-all=true+
+2. Parenthesis set (+(+ ... +)+). +sequential=false+, +require-all=true+
+3. Bracket set (+[+ ... +]+). +sequential=false+, +require-all=false+
+
+To create a set with the properties +sequential=true+ and
++require-all=false+, explicitly set +sequential+ in a bracketed set,
++[ A B C sequential=true ]+.
+
+To create multiple sets with both +sequential+ and +require-all+ set to
+true, explicitly set +sequential+ in a parenthesis set:
++A B ( C D sequential=true )+.
+
+[[topics_Features_AttributeListReferences,Syntax: Attribute list references]]
+=== Syntax: Attribute list references
+
+Attribute lists are used to set attributes and parameters for
+resources, constraints and property definitions. For example, to set
+the virtual IP used by an +IPAddr2+ resource the attribute +ip+ can be
+set in an attribute list for that resource.
+
+Attribute lists can have identifiers that name them, and other
+resources can reuse the same attribute list by referring to that name
+using an +$id-ref+. For example, the following statement defines a
+simple dummy resource with an attribute list which sets the parameter
++state+ to the value 1 and sets the identifier for the attribute list
+to +on-state+:
+
+..............
+primitive dummy-1 Dummy params $id=on-state state=1
+..............
+
+To refer to this attribute list from a different resource, refer to
+the +on-state+ name using an id-ref:
+
+..............
+primitive dummy-2 Dummy params $id-ref=on-state
+..............
+
+The resource +dummy-2+ will now also have the parameter +state+ set to the value 1.
+
+[[topics_Features_AttributeReferences,Syntax: Attribute references]]
+=== Syntax: Attribute references
+
+In some cases, referencing complete attribute lists is too
+coarse-grained, for example if two different parameters with different
+names should have the same value set. Instead of having to copy the
+value in multiple places, it is possible to create references to
+individual attributes in attribute lists.
+
+To name an attribute in order to be able to refer to it later, prefix
+the attribute name with a +$+ character (as seen above with the
+special names +$id+ and +$id-ref+:
+
+............
+primitive dummy-1 Dummy params $state=1
+............
+
+The identifier +state+ can now be used to refer to this attribute from other
+primitives, using the +@<id>+ syntax:
+
+............
+primitive dummy-2 Dummy params @state
+............
+
+In some cases, using the attribute name as the identifier doesn't work
+due to name clashes. In this case, the syntax +$<id>:<name>=<value>+
+can be used to give the attribute a different identifier:
+
+............
+primitive dummy-1 params $dummy-state-on:state=1
+primitive dummy-2 params @dummy-state-on
+............
+
+There is also the possibility that two resources both use the same
+attribute value but with different names. For example, a web server
+may have a parameter +server_ip+ for setting the IP address where it
+listens for incoming requests, and a virtual IP resource may have a
+parameter called +ip+ which sets the IP address it creates. To
+configure these two resources with an IP without repeating the value,
+the reference can be given a name using the syntax +@<id>:<name>+.
+
+Example:
+............
+primitive virtual-ip IPaddr2 params $vip:ip=192.168.1.100
+primitive webserver apache params @vip:server_ip
+............
+
+[[topics_Syntax_RuleExpressions,Syntax: Rule expressions]]
+=== Syntax: Rule expressions
+
+Many of the configuration commands in `crmsh` now support the use of
+_rule expressions_, which can influence what attributes apply to a
+resource or under which conditions a constraint is applied, depending
+on changing conditions like date, time, the value of attributes and
+more.
+
+Here is an example of a simple rule expression used to apply a
+a different resource parameter on the node named `node1`:
+
+..............
+primitive my_resource Special \
+ params 2: rule #uname eq node1 interface=eth1 \
+ params 1: interface=eth0
+..............
+
+This primitive resource has two lists of parameters with descending
+priority. The parameter list with the highest priority is applied
+first, but only if the rule expressions for that parameter list all
+apply. In this case, the rule `#uname eq node1` limits the parameter
+list so that it is only applied on `node1`.
+
+Note that rule expressions are not terminated and are immediately
+followed by the data to which the rule is applied. In this case, the
+name-value pair `interface=eth1`.
+
+Rule expressions can contain multiple expressions connected using the
+boolean operator `or` and `and`. The full syntax for rule expressions
+is listed below.
+
+..............
+rules ::
+ rule [id_spec] [$role=<role>] <score>: <expression>
+ [rule [id_spec] [$role=<role>] <score>: <expression> ...]
+
+id_spec :: $id=<id> | $id-ref=<id>
+score :: <number> | <attribute> | [-]inf
+expression :: <simple_exp> [<bool_op> <simple_exp> ...]
+bool_op :: or | and
+simple_exp :: <attribute> [type:]<binary_op> <value>
+ | <unary_op> <attribute>
+ | date <date_expr>
+type :: <string> | <version> | <number>
+binary_op :: lt | gt | lte | gte | eq | ne
+unary_op :: defined | not_defined
+
+date_expr :: lt <end>
+ | gt <start>
+ | in start=<start> end=<end>
+ | in start=<start> <duration>
+ | spec <date_spec>
+duration|date_spec ::
+ hours=<value>
+ | monthdays=<value>
+ | weekdays=<value>
+ | yearsdays=<value>
+ | months=<value>
+ | weeks=<value>
+ | years=<value>
+ | weekyears=<value>
+ | moon=<value>
+..............
+
+[[topics_Lifetime,Lifetime parameter format]]
+== Lifetime parameter format
+
+Lifetimes can be specified in the ISO 8601 time format or the ISO 8601
+duration format. To distinguish between months and minutes, use the PT
+prefix before specifying minutes. The duration format is one of
++PnYnMnDTnHnMnS+, +PnW+, +P<date>T<time>+.
+
+P = duration. Y = year. M = month. W = week. D = day. T = time. H =
+hour. M = minute. S = second.
+
+Examples:
+.................
+PT5M = 5 minutes later.
+3D = 3 days later.
+PT1H = 1 hour later.
+.................
+
+The cluster checks lifetimes at an interval defined by the
+cluster-recheck-interval property (default 15 minutes).
+
+
+[[topics_Reference,Command reference]]
+== Command reference
+
+The commands are structured to be compatible with the shell command
+line. Sometimes, the underlying Pacemaker grammar uses characters that
+have special meaning in bash, that will need to be quoted. This
+includes the hash or pound sign (`#`), single and double quotes, and
+any significant whitespace.
+
+Whitespace is also significant when assigning values, meaning that
++key=value+ is different from +key = value+.
+
+Commands can be referenced using short-hand as long as the short-hand
+is unique. This can be either a prefix of the command name or a prefix
+string of characters found in the name.
+
+For example, +status+ can be abbreviated as +st+ or +su+, and
++configure+ as +conf+ or +cfg+.
+
+The syntax for the commands is given below in an informal, BNF-like
+grammar.
+
+* `<value>` denotes a string.
+* `[value]` means that the construct is optional.
+* The ellipsis (`...`) signifies that the previous construct may be
+ repeated.
+* `first|second` means either first or second.
+* The rest are literals (strings, `:`, `=`).
+
+[[cmdhelp_root_status,Cluster status]]
+=== `status`
+
+Show cluster status. The status is displayed by `crm_mon`. Supply
+additional arguments for more information or different format.
+See `crm_mon(8)` for more details.
+
+Example:
+...............
+status
+status simple
+status full
+...............
+
+Usage:
+...............
+status [<option> ...]
+
+option :: full
+ | bynode
+ | inactive
+ | ops
+ | timing
+ | failcounts
+ | verbose
+ | quiet
+ | html
+ | xml
+ | simple
+ | tickets
+ | noheaders
+ | detail
+ | brief
+...............
+
+[[cmdhelp_root_verify,Verify cluster status]]
+=== `verify`
+
+Performs basic checks for the cluster configuration and
+current status, reporting potential issues.
+
+See `crm_verify(8)` and `crm_simulate(8)` for more details.
+
+Example:
+...............
+verify
+verify scores
+...............
+
+Usage:
+...............
+verify [scores]
+...............
+
+[[cmdhelp_cluster,Cluster setup and management]]
+=== `cluster` - Cluster setup and management
+
+Whole-cluster configuration management with High Availability
+awareness.
+
+The commands on the cluster level allows configuration and
+modification of the underlying cluster infrastructure, and also
+supplies tools to do whole-cluster systems management.
+
+These commands enable easy installation and maintenance of a HA
+cluster, by providing support for package installation, configuration
+of the cluster messaging layer, file system setup and more.
+
+[[cmdhelp_cluster_add,Add a new node to the cluster,From Code]]
+==== `add`
+See "crm cluster help add" or "crm cluster add --help"
+
+[[cmdhelp_cluster_copy,Copy file to other cluster nodes]]
+==== `copy`
+
+Copy file to other cluster nodes.
+
+Copies the given file to all other nodes unless given a
+list of nodes to copy to as argument.
+
+Usage:
+...............
+copy <filename> [nodes ...]
+...............
+
+Example:
+...............
+copy /etc/motd
+...............
+
+[[cmdhelp_cluster_diff,Diff file across cluster]]
+==== `diff`
+
+Displays the difference, if any, between a given file
+on different nodes. If the second argument is `--checksum`,
+a checksum of the file will be calculated and displayed for
+each node.
+
+Usage:
+...............
+diff <file> [--checksum] [nodes...]
+...............
+
+Example:
+...............
+diff /etc/crm/crm.conf node2
+diff /etc/resolv.conf --checksum
+...............
+
+[[cmdhelp_cluster_disable,Disable cluster services]]
+==== `disable`
+
+Disable the cluster-related system services on this node.
+
+Usage:
+...............
+disable
+...............
+
+[[cmdhelp_cluster_enable,Enable cluster services]]
+==== `enable`
+
+Enable the cluster-related system services on this node.
+
+Usage:
+...............
+enable
+...............
+
+[[cmdhelp_cluster_geo_init,Configure cluster as geo cluster,From Code]]
+==== `geo-init`
+See "crm cluster help geo_init" or "crm cluster geo_init --help"
+
+[[cmdhelp_cluster_geo_init_arbitrator,Initialize node as geo cluster arbitrator,From Code]]
+==== `geo-init-arbitrator`
+See "crm cluster help geo_init_arbitrator" or "crm cluster geo_init_arbitrator --help"
+
+[[cmdhelp_cluster_geo_join,Join cluster to existing geo cluster,From Code]]
+==== `geo-join`
+See "crm cluster help geo_join" or "crm cluster geo_join --help"
+
+[[cmdhelp_cluster_health,Cluster health check]]
+==== `health`
+
+Runs a larger set of tests and queries on all nodes in the cluster to
+verify the general system health and detect potential problems.
+
+Usage:
+...............
+health
+...............
+
+[[cmdhelp_cluster_init,Initializes a new HA cluster,From Code]]
+==== `init`
+See "crm cluster help init" or "crm cluster init --help"
+
+[[cmdhelp_cluster_join,Join existing cluster,From Code]]
+==== `join`
+See "crm cluster help join" or "crm cluster join --help"
+
+[[cmdhelp_cluster_remove,Remove node(s) from the cluster,From Code]]
+==== `remove`
+See "crm cluster help remove" or "crm cluster remove --help"
+
+[[cmdhelp_cluster_crash_test,Cluster crash test tool set,From Code]]
+==== `crash_test`
+See "crm cluster help crash_test" or "crm cluster crash_test --help"
+
+[[cmdhelp_cluster_restart,Restart cluster services]]
+==== `restart`
+
+Restarts the cluster-related system services on this node.
+
+Usage:
+.........
+restart
+.........
+
+[[cmdhelp_cluster_rename,Rename the cluster]]
+==== `rename`
+
+Rename the cluster name
+
+Usage:
+...............
+rename <new_cluster_name>
+...............
+
+
+[[cmdhelp_cluster_run,Execute an arbitrary command on all nodes/specific node]]
+==== `run`
+
+This command takes a shell statement as argument, executes that
+statement on all nodes in the cluster or a specific node,
+and reports the result.
+
+Usage:
+...............
+run <command> [node ...]
+...............
+
+Example:
+...............
+run "cat /proc/uptime"
+run "ls" node1 node2
+...............
+
+[[cmdhelp_cluster_start,Start cluster services]]
+==== `start`
+
+Starts the cluster-related system services on this node.
+
+Usage:
+.........
+start
+.........
+
+[[cmdhelp_cluster_status,Cluster status check]]
+==== `status`
+
+Reports the status for the cluster messaging layer on the local
+node.
+
+Usage:
+...............
+status
+...............
+
+[[cmdhelp_cluster_stop,Stop cluster services]]
+==== `stop`
+
+Stops the cluster-related system services on this node.
+
+Usage:
+.........
+stop
+.........
+
+[[cmdhelp_cluster_wait_for_startup,Wait for cluster to start]]
+==== `wait_for_startup`
+
+Mostly useful in scripts or automated workflows, this command will
+attempt to connect to the local cluster node repeatedly. The command
+will keep trying until the cluster node responds, or the `timeout`
+elapses. The timeout can be changed by supplying a value in seconds as
+an argument.
+
+Usage:
+........
+wait_for_startup
+........
+
+[[cmdhelp_script,Cluster script management]]
+=== `script` - Cluster script management
+
+A big part of the configuration and management of a cluster is
+collecting information about all cluster nodes and deploying changes
+to those nodes. Often, just performing the same procedure on all nodes
+will encounter problems, due to subtle differences in the
+configuration.
+
+For example, when configuring a cluster for the first time, the
+software needs to be installed and configured on all nodes before the
+cluster software can be launched and configured using `crmsh`. This
+process is cumbersome and error-prone, and the goal is for scripts to
+make this process easier.
+
+Scripts are implemented using the python `parallax` package which
+provides a thin wrapper on top of SSH. This allows the scripts to
+function through the usual SSH channels used for system maintenance,
+requiring no additional software to be installed or maintained.
+
+[[cmdhelp_script_json,JSON API for cluster scripts]]
+==== `json`
+
+This command provides a JSON API for the cluster scripts, intended for
+use in user interface tools that want to interact with the cluster via
+scripts.
+
+The command takes a single argument, which should be a JSON array with
+the first member identifying the command to perform.
+
+The output is line-based: Commands that return multiple results will
+return them line-by-line, ending with a terminator value: "end".
+
+When providing parameter values to this command, they should be
+provided as nested objects, so +virtual-ip:ip=192.168.0.5+ on the
+command line becomes the JSON object
++{"virtual-ip":{"ip":"192.168.0.5"}}+.
+
+API:
+........
+["list"]
+=> [{name, shortdesc, category}]
+
+["show", <name>]
+=> [{name, shortdesc, longdesc, category, <<steps>>}]
+
+<<steps>> := [{name, shortdesc], longdesc, required, parameters, steps}]
+
+<<params>> := [{name, shortdesc, longdesc, required, unique, advanced,
+ type, value, example}]
+
+["verify", <name>, <<values>>]
+=> [{shortdesc, longdesc, text, nodes}]
+
+["run", <name>, <<values>>]
+=> [{shortdesc, rc, output|error}]
+........
+
+
+[[cmdhelp_script_list,List available scripts]]
+==== `list`
+
+Lists the available scripts, sorted by category. Scripts that have the
+special `Script` category are hidden by default, since they are mainly
+used by other scripts or commands. To also show these, pass `all` as
+argument.
+
+To get a flat list of script names, not sorted by category, pass
+`names` as an extra argument.
+
+Usage:
+............
+list [all] [names]
+............
+
+Example:
+............
+list
+list all names
+............
+
+[[cmdhelp_script_run,Run the script]]
+==== `run`
+
+Given a list of parameter values, this command will execute the
+actions specified by the cluster script. The format for the parameter
+values is the same as for the `verify` command.
+
+Can optionally take at least two parameters:
+* `nodes=<nodes>`: List of nodes that the script runs over
+* `dry_run=yes|no`: If set, the script will not perform any modifications.
+
+Additional parameters may be available depending on the script.
+
+Use the `show` command to see what parameters are available.
+
+Usage:
+.............
+run <script> [args...]
+.............
+
+Example:
+.............
+run apache install=true
+run sbd id=sbd-1 node=node1 sbd_device=/dev/disk/by-uuid/F00D-CAFE
+.............
+
+[[cmdhelp_script_show,Describe the script]]
+==== `show`
+
+Prints a description and short summary of the script, with
+descriptions of the accepted parameters.
+
+Advanced parameters are hidden by default. To show the complete list
+of parameters accepted by the script, pass `all` as argument.
+
+Usage:
+............
+show <script> [all]
+............
+
+Example:
+............
+show virtual-ip
+............
+
+[[cmdhelp_script_verify,Verify the script]]
+==== `verify`
+
+Checks the given parameter values, and returns a list
+of actions that will be executed when running the script
+if provided the same list of parameter values.
+
+Usage:
+............
+verify <script> [args...]
+............
+
+Example:
+............
+verify sbd id=sbd-1 node=node1 sbd_device=/dev/disk/by-uuid/F00D-CAFE
+............
+
+[[cmdhelp_corosync,Corosync management]]
+=== `corosync` - Corosync management
+
+Corosync is the underlying messaging layer for most HA clusters.
+This level provides commands for editing and managing the corosync
+configuration.
+
+[[cmdhelp_corosync_add-node,Add a corosync node]]
+==== `add-node`
+
+Adds a node to the corosync configuration. This is used with the `udpu`
+type configuration in corosync.
+
+A nodeid for the added node is generated automatically.
+
+Note that this command assumes that only a single ring is used, and
+sets only the address for ring0.
+
+Usage:
+.........
+add-node <addr> [name]
+.........
+
+[[cmdhelp_corosync_del-node,Remove a corosync node]]
+==== `del-node`
+
+Removes a node from the corosync configuration. The argument given is
+the `ring0_addr` address set in the configuration file.
+
+Usage:
+.........
+del-node <addr>
+.........
+
+[[cmdhelp_corosync_diff,Diffs the corosync configuration]]
+==== `diff`
+
+Diffs the corosync configurations on different nodes. If no nodes are
+given as arguments, the corosync configurations on all nodes in the
+cluster are compared.
+
+`diff` takes an option argument `--checksum`, to display a checksum
+for each file instead of calculating a diff.
+
+Usage:
+.........
+diff [--checksum] [node...]
+.........
+
+[[cmdhelp_corosync_edit,Edit the corosync configuration]]
+==== `edit`
+
+Opens the Corosync configuration file in an editor.
+
+Usage:
+.........
+edit
+.........
+
+[[cmdhelp_corosync_get,Get a corosync configuration value]]
+==== `get`
+
+Returns the value configured in `corosync.conf`, which is not
+necessarily the value used in the running configuration. See `reload`
+for telling corosync about configuration changes.
+
+The argument is the complete dot-separated path to the value.
+
+If there are multiple values configured with the same path, the
+command returns all values for that path. For example, to get all
+configured `ring0_addr` values, use this command:
+
+Example:
+........
+get nodelist.node.ring0_addr
+........
+
+[[cmdhelp_corosync_log,Show the corosync log file]]
+==== `log`
+
+Opens the log file specified in the corosync configuration file. If no
+log file is configured, this command returns an error.
+
+The pager used can be configured either using the PAGER
+environment variable or in `crm.conf`.
+
+Usage:
+.........
+log
+.........
+
+[[cmdhelp_corosync_pull,Pulls the corosync configuration]]
+==== `pull`
+
+Gets the corosync configuration from another node and copies
+it to this node.
+
+Usage:
+.........
+pull <node>
+.........
+
+[[cmdhelp_corosync_push,Push the corosync configuration]]
+==== `push`
+
+Pushes the corosync configuration file on this node to
+the list of nodes provided. If no target nodes are given,
+the configuration is pushed to all other nodes in the cluster.
+
+It is recommended to use `csync2` to distribute the cluster
+configuration files rather than relying on this command.
+
+Usage:
+.........
+push [node] ...
+.........
+
+Example:
+.........
+push node-2 node-3
+.........
+
+[[cmdhelp_corosync_reload,Reload the corosync configuration]]
+==== `reload`
+
+Tells all instances of corosync in this cluster to reload
+`corosync.conf`.
+
+After pushing a new configuration to all cluster nodes, call this
+command to make corosync use the new configuration.
+
+Usage:
+.........
+reload
+.........
+
+[[cmdhelp_corosync_set,Set a corosync configuration value]]
+==== `set`
+
+Sets the value identified by the given path. If the value does not
+exist in the configuration file, it will be added. However, if the
+section containing the value does not exist, the command will fail.
+
+Usage:
+.........
+set quorum.expected_votes 2
+.........
+
+[[cmdhelp_corosync_show,Display the corosync configuration]]
+==== `show`
+
+Displays the corosync configuration on the current node.
+
+.........
+show
+.........
+
+[[cmdhelp_corosync_status,Display the corosync status]]
+==== `status`
+
+Displays the corosync ring status(default), also can display quorum/qdevice/qnetd status.
+
+Usage:
+.........
+status [ring|quorum|qdevice|qnetd]
+.........
+
+[[cmdhelp_cib,CIB shadow management]]
+=== `cib` - CIB shadow management
+
+This level is for management of shadow CIBs. It is available both
+at the top level and the `configure` level.
+
+All the commands are implemented using `cib_shadow(8)` and the
+`CIB_shadow` environment variable. The user prompt always
+includes the name of the currently active shadow or the live CIB.
+
+[[cmdhelp_cib_cibstatus,CIB status management and editing]]
+==== `cibstatus`
+
+Enter edit and manage the CIB status section level. See the
+<<cmdhelp_cibstatus,CIB status management section>>.
+
+[[cmdhelp_cib_commit,copy a shadow CIB to the cluster]]
+==== `commit`
+
+Apply a shadow CIB to the cluster. If the shadow name is omitted
+then the current shadow CIB is applied.
+
+Temporary shadow CIBs are removed automatically on commit.
+
+Usage:
+...............
+commit [<cib>]
+...............
+
+[[cmdhelp_cib_delete,delete a shadow CIB]]
+==== `delete`
+
+Delete an existing shadow CIB.
+
+Usage:
+...............
+delete <cib>
+...............
+
+[[cmdhelp_cib_diff,diff between the shadow CIB and the live CIB]]
+==== `diff`
+
+Print differences between the current cluster configuration and
+the active shadow CIB.
+
+Usage:
+...............
+diff
+...............
+
+[[cmdhelp_cib_import,import a CIB or PE input file to a shadow]]
+==== `import`
+
+At times it may be useful to create a shadow file from the
+existing CIB. The CIB may be specified as file or as a PE input
+file number. The shell will look up files in the local directory
+first and then in the PE directory (typically `/var/lib/pengine`).
+Once the CIB file is found, it is copied to a shadow and this
+shadow is immediately available for use at both `configure` and
+`cibstatus` levels.
+
+If the shadow name is omitted then the target shadow is named
+after the input CIB file.
+
+Note that there are often more than one PE input file, so you may
+need to specify the full name.
+
+Usage:
+...............
+import {<file>|<number>} [<shadow>]
+...............
+Examples:
+...............
+import pe-warn-2222
+import 2289 issue2
+...............
+
+[[cmdhelp_cib_list,list all shadow CIBs]]
+==== `list`
+
+List existing shadow CIBs.
+
+Usage:
+...............
+list
+...............
+
+[[cmdhelp_cib_new,create a new shadow CIB]]
+==== `new`
+
+Create a new shadow CIB. The live cluster configuration and
+status is copied to the shadow CIB.
+
+If the name of the shadow is omitted, we create a temporary CIB
+shadow. It is useful if multiple level sessions are desired
+without affecting the cluster. A temporary CIB shadow is short
+lived and will be removed either on `commit` or on program exit.
+Note that if the temporary shadow is not committed all changes in
+the temporary shadow are lost.
+
+Specify `withstatus` if you want to edit the status section of
+the shadow CIB (see the <<cmdhelp_cibstatus,cibstatus section>>).
+Add `force` to force overwriting the existing shadow CIB.
+
+To start with an empty configuration that is not copied from the live
+CIB, specify the `empty` keyword. (This also allows a shadow CIB to be
+created in case no cluster is running.)
+
+Usage:
+...............
+new [<cib>] [withstatus] [force] [empty]
+...............
+
+[[cmdhelp_cib_reset,copy live cib to a shadow CIB]]
+==== `reset`
+
+Copy the current cluster configuration into the shadow CIB.
+
+Usage:
+...............
+reset <cib>
+...............
+
+[[cmdhelp_cib_use,change working CIB]]
+==== `use`
+
+Choose a CIB source. If you want to edit the status from the
+shadow CIB specify `withstatus` (see <<cmdhelp_cibstatus,`cibstatus`>>).
+Leave out the CIB name to switch to the running CIB.
+
+Usage:
+...............
+use [<cib>] [withstatus]
+...............
+
+[[cmdhelp_ra,Resource Agents (RA) lists and documentation]]
+=== `ra` - Resource Agents (RA) lists and documentation
+
+This level contains commands which show various information about
+the installed resource agents. It is available both at the top
+level and at the `configure` level.
+
+[[cmdhelp_ra_classes,list classes and providers]]
+==== `classes`
+
+Print all resource agents' classes and, where appropriate, a list
+of available providers.
+
+Usage:
+...............
+classes
+...............
+
+[[cmdhelp_ra_info,show meta data for a RA]]
+==== `info` (`meta`)
+
+Show the meta-data of a resource agent type. This is where users
+can find information on how to use a resource agent. It is also
+possible to get information from some programs: `pengine`,
+`crmd`, `cib`, and `stonithd`. Just specify the program name
+instead of an RA.
+
+Usage:
+...............
+info [<class>:[<provider>:]]<type>
+info <type> <class> [<provider>] (obsolete)
+...............
+Example:
+...............
+info apache
+info ocf:pacemaker:Dummy
+info stonith:ipmilan
+info pengine
+...............
+
+[[cmdhelp_ra_list,list RA for a class (and provider)]]
+==== `list`
+
+List available resource agents for the given class. If the class
+is `ocf`, supply a provider to get agents which are available
+only from that provider.
+
+Usage:
+...............
+list <class> [<provider>]
+...............
+Example:
+...............
+list ocf pacemaker
+...............
+
+[[cmdhelp_ra_providers,show providers for a RA and a class]]
+==== `providers`
+
+List providers for a resource agent type. The class parameter
+defaults to `ocf`.
+
+Usage:
+...............
+providers <type> [<class>]
+...............
+Example:
+...............
+providers apache
+...............
+
+[[cmdhelp_ra_validate,validate parameters for RA]]
+==== `validate`
+
+If the resource agent supports the `validate-all` action, this calls
+the action with the given parameters, printing any warnings or errors
+reported by the agent.
+
+Usage:
+................
+validate <agent> [<key>=<value> ...]
+................
+
+[[cmdhelp_resource,Resource management]]
+=== `resource` - Resource management
+
+At this level resources may be managed.
+
+All (or almost all) commands are implemented with the CRM tools
+such as `crm_resource(8)`.
+
+[[cmdhelp_resource_ban,ban a resource from a node]]
+==== `ban`
+
+Ban a resource from running on a certain node. If no node is given
+as argument, the resource is banned from the current location.
+
+See `move` for details on other arguments.
+
+Usage:
+...............
+ban <rsc> [<node>] [<lifetime>] [force]
+...............
+
+[[cmdhelp_resource_cleanup,cleanup resource status]]
+==== `cleanup`
+
+If resource has any past failures, clear its history and fail
+count. Typically done after the resource has temporarily
+failed.
+
+If a node is omitted, cleanup on all nodes.
+
++(Pacemaker 1.1.14)+ Pass force to cleanup the resource itself,
+otherwise the cleanup command will apply to the parent resource (if
+any).
+
+Usage:
+...............
+cleanup [<rsc>] [<node>] [force]
+...............
+
+[[cmdhelp_resource_clear,Clear any relocation constraint]]
+==== `clear` (`unmove`, `unmigrate`, `unban`)
+
+Remove any relocation constraint created by
+the `move`, `migrate` or `ban` command.
+
+Usage:
+...............
+clear <rsc>
+unmigrate <rsc>
+unban <rsc>
+...............
+
+[[cmdhelp_resource_constraints,Show constraints affecting a resource]]
+==== `constraints`
+
+Display the location and colocation constraints affecting the
+resource.
+
+Usage:
+................
+constraints <rsc>
+................
+
+[[cmdhelp_resource_demote,demote a master-slave resource]]
+==== `demote`
+
+Demote a master-slave resource using the `target-role`
+attribute.
+
+Usage:
+...............
+demote <rsc>
+...............
+
+[[cmdhelp_resource_failcount,manage failcounts]]
+==== `failcount`
+
+Show/edit/delete the failcount of a resource.
+When `set` a non-zero value, `operation` and `interval` should be
+provided when multiple operation failcount entries exist.
+`interval` is a value in seconds.
+
+Usage:
+...............
+failcount <rsc> set <node> <value> [operation] [interval]
+failcount <rsc> delete <node>
+failcount <rsc> show <node>
+...............
+Example:
+...............
+failcount fs_0 delete node2
+...............
+
+[[cmdhelp_resource_locate,show the location of resources]]
+==== `locate`
+
+Show the current location of one or more resources.
+
+Usage:
+...............
+locate [<rsc> ...]
+...............
+
+[[cmdhelp_resource_maintenance,Enable/disable per-resource maintenance mode]]
+==== `maintenance`
+
+Enables or disables the per-resource maintenance mode. When this mode
+is enabled, no monitor operations will be triggered for the resource.
+`maintenance` attribute conflicts with the `is-managed`. When setting
+the `maintenance` attribute, the user is proposed to remove the
+`is-managed` attribute if it exists.
+
+Usage:
+..................
+maintenance <resource> [on|off|true|false]
+..................
+
+Example:
+..................
+maintenance rsc1
+maintenance rsc2 off
+..................
+
+[[cmdhelp_resource_manage,put a resource into managed mode]]
+==== `manage`
+
+Manage a resource using the `is-managed` attribute. If there
+are multiple meta attributes sets, the attribute is set in all of
+them. If the resource is a clone, all `is-managed` attributes are
+removed from the children resources.
+`is-managed` attribute conflicts with the `maintenance`. When setting
+the `is-managed` attribute, the user is proposed to remove the
+`maintenance` attribute if it exists.
+
+For details on group management see <<cmdhelp_options_manage-children,`options manage-children`>>.
+
+Usage:
+...............
+manage <rsc>
+...............
+
+[[cmdhelp_resource_meta,manage a meta attribute]]
+==== `meta`
+
+Show/edit/delete a meta attribute of a resource. Currently, all
+meta attributes of a resource may be managed with other commands
+such as `resource stop`.
+
+Usage:
+...............
+meta <rsc> set <attr> <value>
+meta <rsc> delete <attr>
+meta <rsc> show <attr>
+...............
+Example:
+...............
+meta ip_0 set target-role stopped
+...............
+
+[[cmdhelp_resource_move,Move a resource to another node]]
+==== `move` (`migrate`)
+
+Move a resource away from its current location.
+
+If the destination node is left out, the resource is migrated by
+creating a constraint which prevents it from running on the current
+node. For this type of constraint to be created, the +force+ argument
+is required.
+
+A lifetime may be given for the constraint. Once it expires, the
+location constraint will no longer be active.
+
+Usage:
+...............
+move <rsc> [<node>] [<lifetime>] [force]
+...............
+
+[[cmdhelp_resource_operations,Show active resource operations]]
+==== `operations`
+
+Show active operations, optionally filtered by resource and node.
+
+Usage:
+................
+operations [<rsc>] [<node>]
+................
+
+[[cmdhelp_resource_param,manage a parameter of a resource]]
+==== `param`
+
+Show/edit/delete a parameter of a resource.
+
+Usage:
+...............
+param <rsc> set <param> <value>
+param <rsc> delete <param>
+param <rsc> show <param>
+...............
+Example:
+...............
+param ip_0 show ip
+...............
+
+[[cmdhelp_resource_promote,promote a master-slave resource]]
+==== `promote`
+
+Promote a master-slave resource using the `target-role`
+attribute.
+
+Usage:
+...............
+promote <rsc>
+...............
+
+[[cmdhelp_resource_refresh,Recheck current resource status and drop failure history]]
+==== `refresh`
+
+Delete resource's history (including failures) so its current state is rechecked.
+
+Usage:
+...............
+refresh [<rsc>] [<node>] [force]
+...............
+
+[[cmdhelp_resource_restart,restart resources]]
+==== `restart`
+
+Restart one or more resources. This is essentially a shortcut for
+resource stop followed by a start. The shell is first going to wait
+for the stop to finish, that is for all resources to really stop, and
+only then to order the start action. Due to this command
+entailing a whole set of operations, informational messages are
+printed to let the user see some progress.
+
+For details on group management see
+<<cmdhelp_options_manage-children,`options manage-children`>>.
+
+Usage:
+...............
+restart <rsc> [<rsc> ...]
+...............
+Example:
+...............
+# crm resource restart g_webserver
+INFO: ordering g_webserver to stop
+waiting for stop to finish .... done
+INFO: ordering g_webserver to start
+#
+...............
+
+[[cmdhelp_resource_scores,Display resource scores]]
+==== `scores`
+
+Display the allocation scores for all resources.
+
+Usage:
+................
+scores
+................
+
+[[cmdhelp_resource_secret,manage sensitive parameters]]
+==== `secret`
+
+Sensitive parameters can be kept in local files rather than CIB
+in order to prevent accidental data exposure. Use the `secret`
+command to manage such parameters. `stash` and `unstash` move the
+value from the CIB and back to the CIB respectively. The `set`
+subcommand sets the parameter to the provided value. `delete`
+removes the parameter completely. `show` displays the value of
+the parameter from the local file. Use `check` to verify if the
+local file content is valid.
+
+Usage:
+...............
+secret <rsc> set <param> <value>
+secret <rsc> stash <param>
+secret <rsc> unstash <param>
+secret <rsc> delete <param>
+secret <rsc> show <param>
+secret <rsc> check <param>
+...............
+Example:
+...............
+secret fence_1 show password
+secret fence_1 stash password
+secret fence_1 set password secret_value
+...............
+
+[[cmdhelp_resource_start,start resources]]
+==== `start`
+
+Start one or more resources by setting the `target-role` attribute. If
+there are multiple meta attributes sets, the attribute is set in all
+of them. If the resource is a clone, all `target-role` attributes are
+removed from the children resources.
+
+For details on group management see
+<<cmdhelp_options_manage-children,`options manage-children`>>.
+
+Usage:
+...............
+start <rsc> [<rsc> ...]
+...............
+
+[[cmdhelp_resource_status,show status of resources]]
+==== `status` (`show`, `list`)
+
+Print resource status. More than one resource can be shown at once. If
+the resource parameter is left out, the status of all resources is
+printed.
+
+Usage:
+...............
+status [<rsc> ...]
+...............
+
+[[cmdhelp_resource_stop,stop resources]]
+==== `stop`
+
+Stop one or more resources using the `target-role` attribute. If there
+are multiple meta attributes sets, the attribute is set in all of
+them. If the resource is a clone, all `target-role` attributes are
+removed from the children resources.
+
+For details on group management see
+<<cmdhelp_options_manage-children,`options manage-children`>>.
+
+Usage:
+...............
+stop <rsc> [<rsc> ...]
+...............
+
+[[cmdhelp_resource_trace,start RA tracing]]
+==== `trace`
+
+Start tracing RA for the given operation. The trace files are
+stored in `$HA_VARLIB/trace_ra`. If the operation to be traced is
+monitor, note that the number of trace files can grow very
+quickly.
+
+If no operation name is given, crmsh will attempt to trace all
+operations for the RA. This includes any configured operations, start
+and stop as well as promote/demote for multistate resources.
+
+To trace the probe operation which exists for all resources, either
+set a trace for `monitor` with interval `0`, or use `probe` as the
+operation name.
+
+Note: RA tracing is only supported by OCF resource agents;
+The pacemaker-execd daemon does not log recurring monitor operations
+unless an error occurred.
+
+Usage:
+...............
+trace <rsc> [<op> [<interval>] ]
+...............
+Example:
+...............
+trace fs start
+trace webserver
+trace webserver probe
+trace fs monitor 0
+...............
+
+[[cmdhelp_resource_unmanage,put a resource into unmanaged mode]]
+==== `unmanage`
+
+Unmanage a resource using the `is-managed` attribute. If there
+are multiple meta attributes sets, the attribute is set in all of
+them. If the resource is a clone, all `is-managed` attributes are
+removed from the children resources.
+
+For details on group management see <<cmdhelp_options_manage-children,`options manage-children`>>.
+
+Usage:
+...............
+unmanage <rsc>
+...............
+
+[[cmdhelp_resource_untrace,stop RA tracing]]
+==== `untrace`
+
+Stop tracing RA for the given operation. If no operation name is
+given, crmsh will attempt to stop tracing all operations in resource.
+
+Usage:
+...............
+untrace <rsc> [<op> [<interval>] ]
+...............
+Example:
+...............
+untrace fs start
+untrace webserver
+...............
+
+[[cmdhelp_resource_utilization,manage a utilization attribute]]
+==== `utilization`
+
+Show/edit/delete a utilization attribute of a resource. These
+attributes describe hardware requirements. By setting the
+`placement-strategy` cluster property appropriately, it is
+possible then to distribute resources based on resource
+requirements and node size. See also <<cmdhelp_node_utilization,node utilization attributes>>.
+
+Usage:
+...............
+utilization <rsc> set <attr> <value>
+utilization <rsc> delete <attr>
+utilization <rsc> show <attr>
+...............
+Example:
+...............
+utilization xen1 set memory 4096
+...............
+
+[[cmdhelp_node,Node management]]
+=== `node` - Node management
+
+Node management and status commands.
+
+[[cmdhelp_node_attribute,manage attributes]]
+==== `attribute`
+
+Edit node attributes. This kind of attribute should refer to
+relatively static properties, such as memory size.
+
+Usage:
+...............
+attribute <node> set <attr> <value>
+attribute <node> delete <attr>
+attribute <node> show <attr>
+...............
+Example:
+...............
+attribute node_1 set memory_size 4096
+...............
+
+[[cmdhelp_node_clearstate,Clear node state]]
+==== `clearstate`
+
+Resets and clears the state of the specified node. This node is
+afterwards assumed clean and offline. This command can be used to
+manually confirm that a node has been fenced (e.g., powered off).
+
+Be careful! This can cause data corruption if you confirm that a node is
+down that is, in fact, not cleanly down - the cluster will proceed as if
+the fence had succeeded, possibly starting resources multiple times.
+
+Usage:
+...............
+clearstate <node>
+...............
+
+[[cmdhelp_node_delete,delete node]]
+==== `delete`
+
+Delete a node. This command will remove the node from the CIB
+and, in case the cluster stack is running, use the appropriate
+program (`crm_node` or `hb_delnode`) to remove the node from the
+membership.
+
+If the node is still listed as active and a member of our
+partition we refuse to remove it. With the global force option
+(`-F`) we will try to delete the node anyway.
+
+Usage:
+...............
+delete <node>
+...............
+
+[[cmdhelp_node_fence,fence node]]
+==== `fence`
+
+Make CRM fence a node. This functionality depends on stonith
+resources capable of fencing the specified node. No such stonith
+resources, no fencing will happen.
+
+Usage:
+...............
+fence <node>
+...............
+
+[[cmdhelp_node_maintenance,put node into maintenance mode]]
+==== `maintenance`
+
+Set the node status to maintenance. This is equivalent to the
+cluster-wide `maintenance-mode` property but puts just one node
+into the maintenance mode. If there are maintenaned resources on
+the node, the user will be proposed to remove the maintenance
+property from them.
+
+The node parameter defaults to the node where the command is run.
+
+Usage:
+...............
+maintenance [<node>]
+...............
+
+[[cmdhelp_node_online,set node online]]
+==== `online`
+
+Set a node to online status.
+
+The node parameter defaults to the node where the command is run.
+
+Usage:
+...............
+online [<node>]
+...............
+
+[[cmdhelp_node_ready,put node into ready mode]]
+==== `ready`
+
+Set the node's maintenance status to `off`. The node should be
+now again fully operational and capable of running resource
+operations.
+
+The node parameter defaults to the node where the command is run.
+
+Usage:
+...............
+ready [<node>]
+...............
+
+[[cmdhelp_node_server,show node hostname or server address]]
+==== `server`
+
+Remote nodes may have a configured server address which should
+be used when contacting the node. This command prints the
+server address if configured, else the node name.
+
+If no parameter is given, the addresses or names for all nodes
+are printed.
+
+Usage:
+...............
+server [<node> ...]
+...............
+
+[[cmdhelp_node_show,show node]]
+==== `show`
+
+Show a node definition. If the node parameter is omitted then all
+nodes are shown.
+
+Usage:
+...............
+show [<node>]
+...............
+
+[[cmdhelp_node_standby,put node into standby]]
+==== `standby`
+
+Set a node to standby status. The node parameter defaults to the
+node where the command is run.
+
+Additionally, you may specify a lifetime for the standby---if set to
+`reboot`, the node will be back online once it reboots. `forever` will
+keep the node in standby after reboot. The life time defaults to
+`forever`.
+
+Usage:
+...............
+standby [<node>] [<lifetime>]
+
+lifetime :: reboot | forever
+...............
+
+Example:
+...............
+standby bob reboot
+...............
+
+[[cmdhelp_node_status-attr,manage status attributes]]
+==== `status-attr`
+
+Edit node attributes which are in the CIB status section, i.e.
+attributes which hold properties of a more volatile nature. One
+typical example is attribute generated by the `pingd` utility.
+
+Usage:
+...............
+status-attr <node> set <attr> <value>
+status-attr <node> delete <attr>
+status-attr <node> show <attr>
+...............
+Example:
+...............
+status-attr node_1 show pingd
+...............
+
+[[cmdhelp_node_utilization,manage utilization attributes]]
+==== `utilization`
+
+Edit node utilization attributes. These attributes describe
+hardware characteristics as integer numbers such as memory size
+or the number of CPUs. By setting the `placement-strategy`
+cluster property appropriately, it is possible then to distribute
+resources based on resource requirements and node size. See also
+<<cmdhelp_resource_utilization,resource utilization attributes>>.
+
+Usage:
+...............
+utilization <node> set <attr> <value>
+utilization <node> delete <attr>
+utilization <node> show <attr>
+...............
+Examples:
+...............
+utilization node_1 set memory 16384
+utilization node_1 show cpu
+...............
+
+[[cmdhelp_site,GEO clustering site support]]
+=== `site` - GEO clustering site support
+
+A cluster may consist of two or more subclusters in different and
+distant locations. This set of commands supports such setups.
+
+[[cmdhelp_site_ticket,manage site tickets]]
+==== `ticket`
+
+Tickets are cluster-wide attributes. They can be managed at the
+site where this command is executed.
+
+It is then possible to constrain resources depending on the
+ticket availability (see the <<cmdhelp_configure_rsc_ticket,`rsc_ticket`>> command
+for more details).
+
+Usage:
+...............
+ticket {grant|revoke|standby|activate|show|time|delete} <ticket>
+...............
+Example:
+...............
+ticket grant ticket1
+...............
+
+[[cmdhelp_options,User preferences]]
+=== `options` - User preferences
+
+The user may set various options for the crm shell itself.
+
+[[cmdhelp_options_add-quotes,add quotes around parameters containing spaces]]
+==== `add-quotes`
+
+The shell (as in `/bin/sh`) parser strips quotes from the command
+line. This may sometimes make it really difficult to type values
+which contain white space. One typical example is the configure
+filter command. The crm shell will supply extra quotes around
+arguments which contain white space. The default is `yes`.
+
+.Note on quotes use
+****************************
+Adding quotes around arguments automatically has been introduced
+with version 1.2.2 and it is technically a regression. Being a
+regression is the only reason the `add-quotes` option exists. If
+you have custom shell scripts which would break, just set the
+`add-quotes` option to `no`.
+
+For instance, with adding quotes enabled, it is possible to do
+the following:
+...............
+# crm configure primitive d1 Dummy \
+ meta description="some description here"
+# crm configure filter 'sed "s/hostlist=./&node-c /"' fencing
+...............
+****************************
+
+[[cmdhelp_options_check-frequency,when to perform semantic check]]
+==== `check-frequency`
+
+Semantic check of the CIB or elements modified or created may be
+done on every configuration change (`always`), when verifying
+(`on-verify`) or `never`. It is by default set to `always`.
+Experts may want to change the setting to `on-verify`.
+
+The checks require that resource agents are present. If they are
+not installed at the configuration time set this preference to
+`never`.
+
+See <<topics_Features_Checks,Configuration semantic checks>> for more details.
+
+[[cmdhelp_options_check-mode,how to treat semantic errors]]
+==== `check-mode`
+
+Semantic check of the CIB or elements modified or created may be
+done in the `strict` mode or in the `relaxed` mode. In the former
+certain problems are treated as configuration errors. In the
+`relaxed` mode all are treated as warnings. The default is `strict`.
+
+See <<topics_Features_Checks,Configuration semantic checks>> for more details.
+
+[[cmdhelp_options_colorscheme,set colors for output]]
+==== `colorscheme`
+
+With `output` set to `color`, a comma separated list of colors
+from this option are used to emphasize:
+
+- keywords
+- object ids
+- attribute names
+- attribute values
+- scores
+- resource references
+
+`crm` can show colors only if there is curses support for python
+installed (usually provided by the `python-curses` package). The
+colors are whatever is available in your terminal. Use `normal`
+if you want to keep the default foreground color.
+
+This user preference defaults to
+`yellow,normal,cyan,red,green,magenta` which is good for
+terminals with dark background. You may want to change the color
+scheme and save it in the preferences file for other color
+setups.
+
+Example:
+...............
+colorscheme yellow,normal,blue,red,green,magenta
+...............
+
+[[cmdhelp_options_editor,set preferred editor program]]
+==== `editor`
+
+The `edit` command invokes an editor. Use this to specify your
+preferred editor program. If not set, it will default to either
+the value of the `EDITOR` environment variable or to one of the
+standard UNIX editors (`vi`,`emacs`,`nano`).
+
+Usage:
+...............
+editor program
+...............
+Example:
+...............
+editor vim
+...............
+
+[[cmdhelp_options_manage-children,how to handle children resource attributes]]
+==== `manage-children`
+
+Some resource management commands, such as `resource stop`, when
+the target resource is a group, may not always produce desired
+result. Each element, group and the primitive members, can have a
+meta attribute and those attributes may end up with conflicting
+values. Consider the following construct:
+...............
+crm(live)# configure show svc fs virtual-ip
+primitive fs Filesystem \
+ params device="/dev/drbd0" directory="/srv/nfs" fstype=ext3 \
+ op monitor interval=10s \
+ meta target-role=Started
+primitive virtual-ip IPaddr2 \
+ params ip=10.2.13.110 iflabel=1 \
+ op monitor interval=10s \
+ op start interval=0 \
+ meta target-role=Started
+group svc fs virtual-ip \
+ meta target-role=Stopped
+...............
+
+Even though the element +svc+ should be stopped, the group is
+actually running because all its members have the +target-role+
+set to +Started+:
+...............
+crm(live)# resource show svc
+resource svc is running on: xen-f
+...............
+
+Hence, if the user invokes +resource stop svc+ the intention is
+not clear. This preference gives the user an opportunity to
+better control what happens if attributes of group members have
+values which are in conflict with the same attribute of the group
+itself.
+
+Possible values are +ask+ (the default), +always+, and +never+.
+If set to +always+, the crm shell removes all children attributes
+which have values different from the parent. If set to +never+,
+all children attributes are left intact. Finally, if set to
++ask+, the user will be asked for each member what is to be done.
+
+[[cmdhelp_options_output,set output type]]
+==== `output`
+
+`crm` can adorn configurations in two ways: in color (similar to
+for instance the `ls --color` command) and by showing keywords in
+upper case. Possible values are `plain`, `color-always`, `color`,
+and 'uppercase'. It is possible to combine `uppercase` with one
+of the color values in order to get an upper case xmass tree. Just
+set this option to `color,uppercase` or `color-always,uppercase`.
+In case you need color codes in pipes, `color-always` forces color
+codes even in case the terminal is not a tty (just like `ls
+--color=always`).
+
+[[cmdhelp_options_pager,set preferred pager program]]
+==== `pager`
+
+The `view` command displays text through a pager. Use this to
+specify your preferred pager program. If not set, it will default
+to either the value of the `PAGER` environment variable or to one
+of the standard UNIX system pagers (`less`,`more`,`pg`).
+
+[[cmdhelp_options_reset,reset user preferences to factory defaults]]
+==== `reset`
+
+This command resets all user options to the defaults. If used as
+a single-shot command, the rc file (+$HOME/.config/crm/rc+) is
+reset to the defaults too.
+
+[[cmdhelp_options_save,save the user preferences to the rc file]]
+==== `save`
+
+Save current settings to the rc file (+$HOME/.config/crm/rc+). On
+further `crm` runs, the rc file is automatically read and parsed.
+
+[[cmdhelp_options_set,Set the value of a given option]]
+==== `set`
+
+Sets the value of an option. Takes the fully qualified
+name of the option as argument, as displayed by +show all+.
+
+The modified option value is stored in the user-local
+configuration file, usually found in +~/.config/crm/crm.conf+.
+
+Usage:
+........
+set <option> <value>
+........
+
+Example:
+........
+set color.warn "magenta bold"
+set editor nano
+........
+
+[[cmdhelp_options_show,show current user preference]]
+==== `show`
+
+Display all current settings.
+
+Given an option name as argument, `show` will display only the value
+of that argument.
+
+Given +all+ as argument, `show` displays all available user options.
+
+Usage:
+........
+show [all|<option>]
+........
+
+Example:
+........
+show
+show skill-level
+show all
+........
+
+[[cmdhelp_options_skill-level,set skill level]]
+==== `skill-level`
+
+Based on the skill-level setting, the user is allowed to use only
+a subset of commands. There are three levels: operator,
+administrator, and expert. The operator level allows only
+commands at the `resource` and `node` levels, but not editing
+or deleting resources. The administrator may do that and may also
+configure the cluster at the `configure` level and manage the
+shadow CIBs. The expert may do all.
+
+Usage:
+...............
+skill-level <level>
+
+level :: operator | administrator | expert
+...............
+
+.Note on security
+****************************
+The `skill-level` option is advisory only. There is nothing
+stopping any users change their skill level (see
+<<topics_Features_Security,Access Control Lists (ACL)>> on how to enforce
+access control).
+****************************
+
+[[cmdhelp_options_sort-elements,sort CIB elements]]
+==== `sort-elements`
+
+`crm` by default sorts CIB elements. If you want them appear in
+the order they were created, set this option to `no`.
+
+Usage:
+...............
+sort-elements {yes|no}
+...............
+Example:
+...............
+sort-elements no
+...............
+
+[[cmdhelp_options_user,set the cluster user]]
+==== `user`
+
+Sufficient privileges are necessary in order to manage a
+cluster: programs such as `crm_verify` or `crm_resource` and,
+ultimately, `cibadmin` have to be run either as `root` or as the
+CRM owner user (typically `hacluster`). You don't have to worry
+about that if you run `crm` as `root`. A more secure way is to
+run the program with your usual privileges, set this option to
+the appropriate user (such as `hacluster`), and setup the
+`sudoers` file.
+
+Usage:
+...............
+user system-user
+...............
+Example:
+...............
+user hacluster
+...............
+
+[[cmdhelp_options_wait,synchronous operation]]
+==== `wait`
+
+In normal operation, `crm` runs a command and gets back
+immediately to process other commands or get input from the user.
+With this option set to `yes` it will wait for the started
+transition to finish. In interactive mode dots are printed to
+indicate progress.
+
+Usage:
+...............
+wait {yes|no}
+...............
+Example:
+...............
+wait yes
+...............
+
+[[cmdhelp_configure,CIB configuration]]
+=== `configure` - CIB configuration
+
+This level enables all CIB object definition commands.
+
+The configuration may be logically divided into four parts:
+nodes, resources, constraints, and (cluster) properties and
+attributes. Each of these commands support one or more basic CIB
+objects.
+
+Nodes and attributes describing nodes are managed using the
+`node` command.
+
+Commands for resources are:
+
+- `primitive`
+- `monitor`
+- `group`
+- `clone`
+- `ms`/`master` (master-slave)
+
+In order to streamline large configurations, it is possible to
+define a template which can later be referenced in primitives:
+
+- `rsc_template`
+
+In that case the primitive inherits all attributes defined in the
+template.
+
+There are three types of constraints:
+
+- `location`
+- `colocation`
+- `order`
+
+It is possible to define fencing order (stonith resource
+priorities):
+
+- `fencing_topology`
+
+Finally, there are the cluster properties, resource meta
+attributes defaults, and operations defaults. All are just a set
+of attributes. These attributes are managed by the following
+commands:
+
+- `property`
+- `rsc_defaults`
+- `op_defaults`
+
+In addition to the cluster configuration, the Access Control
+Lists (ACL) can be setup to allow access to parts of the CIB for
+users other than +root+ and +hacluster+. The following commands
+manage ACL:
+
+- `user`
+- `role`
+
+In Pacemaker 1.1.12 and up, this command replaces the `user` command
+for handling ACLs:
+
+- `acl_target`
+
+The changes are applied to the current CIB only on ending the
+configuration session or using the `commit` command.
+
+Comments start with +#+ in the first line. The comments are tied
+to the element which follows. If the element moves, its comments
+will follow.
+
+[[cmdhelp_configure_acl_target,Define target access rights]]
+==== `acl_target`
+
+Defines an ACL target.
+
+Usage:
+................
+acl_target <tid> [<role> ...]
+................
+Example:
+................
+acl_target joe resource_admin constraint_editor
+................
+
+[[cmdhelp_configure_alert,Event-driven alerts]]
+==== `alert`
+
+.Version note
+****************************
+This feature is only available
+in Pacemaker 1.1.15+.
+****************************
+
+Event-driven alerts enables calling scripts whenever interesting
+events occur in the cluster (nodes joining or leaving, resources
+starting or stopping, etc.).
+
+The +path+ is an arbitrary file path to an alert script. Existing
+external scripts used with ClusterMon resources can be used as alert
+scripts, since the interface is compatible.
+
+Each alert may have a number of receipients configured. These will be
+passed to the script as arguments. The first recipient will also be
+passed as the +CRM_alert_recipient+ environment variable, for
+compatibility with existing scripts that only support one recipient.
+
+The available meta attributes are +timeout+ (default 30s) and
++timestamp-format+ (default `"%H:%M:%S.%06N"`).
+
+Some configurations may require each recipient to be delimited by
+brackets, to avoid ambiguity. In the example +alert-2+ below, the meta
+attribute for `timeout` is defined after the recipient, so the
+brackets are used to ensure that the meta attribute is set for the
+alert and not just the recipient. This can be avoided by setting any
+alert attributes before defining the recipients.
+
+Usage:
+...............
+alert <id> <path> \
+ [attributes <nvpair> ...] \
+ [meta <nvpair> ...] \
+ [select [nodes | fencing | resources | attributes '{' <attribute> ... '}' ] ...] \
+ [to [{] <recipient>
+ [attributes <nvpair> ...] \
+ [meta <nvpair> ...] [}] \
+ ...]
+...............
+
+Example:
+...............
+alert alert-1 /srv/pacemaker/pcmk_alert_sample.sh \
+ to /var/log/cluster-alerts.log
+
+alert alert-2 /srv/pacemaker/example_alert.sh \
+ meta timeout=60s \
+ to { /var/log/cluster-alerts.log }
+
+alert alert-3 /srv/pacemaker/example_alert.sh \
+ select fencing \
+ to { /var/log/fencing-alerts.log }
+
+...............
+
+[[cmdhelp_configure_bundle,Container bundle]]
+==== `bundle`
+
+A bundle is a single resource specifying the settings, networking
+requirements, and storage requirements for any number of containers
+generated from the same container image.
+
+Pacemaker bundles support Docker (since version 1.1.17) and rkt (since
+version 1.1.18) container technologies.
+
+A bundle must contain exactly one +docker+ or +rkt+ element.
+
+The bundle definition may contain a reference to a primitive
+resource which defining the resource running inside the
+container.
+
+Example:
+...............
+
+primitive httpd-apache ocf:heartbeat:apache
+
+bundle httpd \
+ docker image=pcmk:httpd replicas=3 \
+ network ip-range-start=10.10.10.123 host-netmask=24 \
+ port-mapping port=80 \
+ storage \
+ storage-mapping target-dir=/var/www/html source-dir=/srv/www options=rw \
+ primitive httpd-apache
+
+...............
+
+[[cmdhelp_configure_cib,CIB shadow management]]
+==== `cib`
+
+This level is for management of shadow CIBs. It is available at
+the `configure` level to enable saving intermediate changes to a
+shadow CIB instead of to the live cluster. This short excerpt
+shows how:
+...............
+crm(live)configure# cib new test-2
+INFO: test-2 shadow CIB created
+crm(test-2)configure# commit
+...............
+Note how the current CIB in the prompt changed from +live+ to
++test-2+ after issuing the `cib new` command. See also the
+<<cmdhelp_cib,CIB shadow management>> for more information.
+
+[[cmdhelp_configure_cibstatus,CIB status management and editing]]
+==== `cibstatus`
+
+Enter edit and manage the CIB status section level. See the
+<<cmdhelp_cibstatus,CIB status management section>>.
+
+[[cmdhelp_configure_clone,define a clone]]
+==== `clone`
+
+The `clone` command creates a resource clone. It may contain a
+single primitive resource or one group of resources.
+
+Usage:
+...............
+clone <name> <rsc>
+ [description=<description>]
+ [meta <attr_list>]
+ [params <attr_list>]
+
+attr_list :: [$id=<id>] <attr>=<val> [<attr>=<val>...] | $id-ref=<id>
+...............
+Example:
+...............
+clone cl_fence apc_1 \
+ meta clone-node-max=1 globally-unique=false
+
+clone disk1 drbd1 \
+ meta promotable=true notify=true globally-unique=false
+...............
+
+[[cmdhelp_configure_colocation,colocate resources]]
+==== `colocation` (`collocation`)
+
+This constraint expresses the placement relation between two
+or more resources. If there are more than two resources, then the
+constraint is called a resource set.
+
+The score is used to indicate the priority of the constraint. A
+positive score indicates that the resources should run on the same
+node. A negative score that they should not run on the same
+node. Values of positive or negative +infinity+ indicate a mandatory
+constraint.
+
+In the two resource form, the cluster will place +<with-rsc>+ first,
+and then decide where to put the +<rsc>+ resource.
+
+Collocation resource sets have an extra attribute (+sequential+)
+to allow for sets of resources which don't depend on each other
+in terms of state. The shell syntax for such sets is to put
+resources in parentheses.
+
+Sets cannot be nested.
+
+The optional +node-attribute+ can be used to colocate resources on a
+set of nodes and not necessarily on the same node. For example, by
+setting a node attribute +color+ on all nodes and setting the
++node-attribute+ value to +color+ as well, the colocated resources
+will be placed on any node that has the same color.
+
+For more details on how to configure resource sets, see
+<<topics_Features_Resourcesets,`Syntax: Resource sets`>>.
+
+Usage:
+...............
+colocation <id> <score>: <rsc>[:<role>] <with-rsc>[:<role>]
+ [node-attribute=<node_attr>]
+
+colocation <id> <score>: <resource_sets>
+ [node-attribute=<node_attr>]
+
+resource_sets :: <resource_set> [<resource_set> ...]
+
+resource_set :: ["("|"["] <rsc>[:<role>] [<rsc>[:<role>] ...] \
+ [<attributes>] [")"|"]"]
+
+attributes :: [require-all=(true|false)] [sequential=(true|false)]
+
+...............
+Example:
+...............
+colocation never_put_apache_with_dummy -inf: apache dummy
+colocation c1 inf: A ( B C )
+...............
+
+[[cmdhelp_configure_commit,commit the changes to the CIB]]
+==== `commit`
+
+Commit the current configuration to the CIB in use. As noted
+elsewhere, commands in a configure session don't have immediate
+effect on the CIB. All changes are applied at one point in time,
+either using `commit` or when the user leaves the configure
+level. In case the CIB in use changed in the meantime, presumably
+by somebody else, the crm shell will refuse to apply the changes.
+
+If you know that it's fine to still apply them, add +force+ to the
+command line.
+
+To disable CIB patching and apply the changes by replacing the CIB
+completely, add +replace+ to the command line. Note that this can lead
+to previous changes being overwritten if some other process
+concurrently modifies the CIB.
+
+Usage:
+...............
+commit [force] [replace]
+...............
+
+[[cmdhelp_configure_default-timeouts,set timeouts for operations to minimums from the meta-data]]
+==== `default-timeouts`
+
+This command takes the timeouts from the actions section of the
+resource agent meta-data and sets them for the operations of the
+primitive.
+
+Usage:
+...............
+default-timeouts <id> [<id>...]
+...............
+
+.Note on `default-timeouts`
+****************************
+The use of this command is discouraged in favor of manually
+determining the best timeouts required for the particular
+configuration. Relying on the resource agent to supply appropriate
+timeouts can cause the resource to fail at the worst possible moment.
+
+Appropriate timeouts for resource actions are context-sensitive, and
+should be carefully considered with the whole configuration in mind.
+****************************
+
+[[cmdhelp_configure_delete,delete CIB objects]]
+==== `delete`
+
+Delete one or more objects. If an object to be deleted belongs to
+a container object, such as a group, and it is the only resource
+in that container, then the container is deleted as well. Any
+related constraints are removed as well.
+
+If the object is a started resource, it will not be deleted unless the
++--force+ flag is passed to the command, or the +force+ option is set.
+
+Usage:
+...............
+delete [--force] <id> [<id>...]
+...............
+
+[[cmdhelp_configure_edit,edit CIB objects]]
+==== `edit`
+
+This command invokes the editor with the object description. As
+with the `show` command, the user may choose to edit all objects
+or a set of objects.
+
+If the user insists, he or she may edit the XML edition of the
+object. If you do that, don't modify any id attributes.
+
+Usage:
+...............
+edit [xml] [<id> ...]
+edit [xml] changed
+...............
+
+.Note on renaming element ids
+****************************
+The edit command sometimes cannot properly handle modifying
+element ids. In particular for elements which belong to group or
+ms resources. Group and ms resources themselves also cannot be
+renamed. Please use the `rename` command instead.
+****************************
+
+[[cmdhelp_configure_erase,erase the CIB]]
+==== `erase`
+
+The `erase` clears all configuration. Apart from nodes. To remove
+nodes, you have to specify an additional keyword `nodes`.
+
+Note that removing nodes from the live cluster may have some
+strange/interesting/unwelcome effects.
+
+Usage:
+...............
+erase [nodes]
+...............
+
+[[cmdhelp_configure_fencing_topology,node fencing order]]
+==== `fencing_topology`
+
+If multiple fencing (stonith) devices are available capable of
+fencing a node, their order may be specified by +fencing_topology+.
+The order is specified per node.
+
+Stonith resources can be separated by +,+ in which case all of
+them need to succeed. If they fail, the next stonith resource (or
+set of resources) is used. In other words, use comma to separate
+resources which all need to succeed and whitespace for serial
+order. It is not allowed to use whitespace around comma.
+
+If the node is left out, the order is used for all nodes.
+That should reduce the configuration size in some stonith setups.
+
+From Pacemaker version 1.1.14, it is possible to use a node attribute
+as the +target+ in a fencing topology. The syntax for this usage is
+described below.
+
+From Pacemaker version 1.1.14, it is also possible to use regular
+expression patterns as the +target+ in a fencing topology. The configured
+fencing sequence then applies to all devices matching the pattern.
+
+Usage:
+...............
+fencing_topology <stonith_resources> [<stonith_resources> ...]
+fencing_topology <fencing_order> [<fencing_order> ...]
+
+fencing_order :: <target> <stonith_resources> [<stonith_resources> ...]
+
+stonith_resources :: <rsc>[,<rsc>...]
+target :: <node>: | attr:<node-attribute>=<value> | pattern:<pattern>
+...............
+Example:
+...............
+# Only kill the power if poison-pill fails
+fencing_topology poison-pill power
+
+# As above for node-a, but a different strategy for node-b
+fencing_topology \
+ node-a: poison-pill power \
+ node-b: ipmi serial
+
+# Fencing anything on rack 1 requires fencing via both APC 1 and 2,
+# to defeat the redundancy provided by two separate UPS units.
+fencing_topology attr:rack=1 apc01,apc02
+
+# Fencing for all machines named green.* is done using the pear
+# fencing device first, while all machines named red.* are fenced
+# using the apple fencing device first.
+fencing_topology \
+ pattern:green.* pear apple \
+ pattern:red.* apple pear
+...............
+
+[[cmdhelp_configure_filter,filter CIB objects]]
+==== `filter`
+
+This command filters the given CIB elements through an external
+program. The program should accept input on `stdin` and send
+output to `stdout` (the standard UNIX filter conventions). As
+with the `show` command, the user may choose to filter all or
+just a subset of elements.
+
+It is possible to filter the XML representation of objects, but
+probably not as useful as the configuration language. The
+presentation is somewhat different from what would be displayed
+by the `show` command---each element is shown on a single line,
+i.e. there are no backslashes and no other embelishments.
+
+Don't forget to put quotes around the filter if it contains
+spaces.
+
+Usage:
+...............
+filter <prog> [xml] [<id> ...]
+filter <prog> [xml] changed
+...............
+Examples:
+...............
+filter "sed '/^primitive/s/target-role=[^ ]*//'"
+# crm configure filter "sed '/^primitive/s/target-role=[^ ]*//'"
+crm configure <<END
+ filter "sed '/threshold=\"1\"/s/=\"1\"/=\"0\"/g'"
+END
+...............
+
+.Note on quotation marks
+**************************
+Filter commands which feature a blend of quotation marks can be
+difficult to get right, especially when used directly from bash, since
+bash does its own quotation parsing. In these cases, it can be easier
+to supply the filter command as standard input. See the last example
+above.
+**************************
+
+[[cmdhelp_configure_get_property,Get property value]]
+==== `get-property`
+
+Show the value of the given property. If the value is not set, the
+command will print the default value for the property, if known.
+
+If no property name is passed to the command, the list of known
+cluster properties is printed.
+
+If the property is set multiple times, for example using multiple
+property sets with different rule expressions, the output of this
+command is undefined.
+
+Pass the argument +-t+ or +--true+ to `get-property` to translate
+the argument value into +true+ or +false+. If the value is not
+set, the command will print +false+.
+
+Usage:
+...............
+get-property [-t|--true] [<name>]
+...............
+
+Example:
+...............
+get-property stonith-enabled
+get-property -t maintenance-mode
+...............
+
+[[cmdhelp_configure_graph,generate a directed graph]]
+==== `graph`
+
+Create a graphviz graphical layout from the current cluster
+configuration.
+
+Currently, only `dot` (directed graph) is supported. It is
+essentially a visualization of resource ordering.
+
+The graph may be saved to a file which can be used as source for
+various graphviz tools (by default it is displayed in the user's
+X11 session). Optionally, by specifying the format, one can also
+produce an image instead.
+
+For more or different graphviz attributes, it is possible to save
+the default set of attributes to an ini file. If this file exists
+it will always override the builtin settings. The +exportsettings+
+subcommand also prints the location of the ini file.
+
+Usage:
+...............
+graph [<gtype> [<file> [<img_format>]]]
+graph exportsettings
+
+gtype :: dot
+img_format :: `dot` output format (see the +-T+ option)
+...............
+Example:
+...............
+graph dot
+graph dot clu1.conf.dot
+graph dot clu1.conf.svg svg
+...............
+
+[[cmdhelp_configure_group,define a group]]
+==== `group`
+
+The `group` command creates a group of resources. This can be useful
+when resources depend on other resources and require that those
+resources start in order on the same node. A common use of resource
+groups is to ensure that a server and a virtual IP are located
+together, and that the virtual IP is started before the server.
+
+Grouped resources are started in the order they appear in the group,
+and stopped in the reverse order. If a resource in the group cannot
+run anywhere, resources following it in the group will not start.
+
+`group` can be passed the "container" meta attribute, to indicate that
+it is to be used to group VM resources monitored using Nagios. The
+resource referred to by the container attribute must be of type
+`ocf:heartbeat:Xen`, `ocf:heartbeat:VirtualDomain` or `ocf:heartbeat:lxc`.
+
+Usage:
+...............
+group <name> <rsc> [<rsc>...]
+ [description=<description>]
+ [meta attr_list]
+ [params attr_list]
+
+attr_list :: [$id=<id>] <attr>=<val> [<attr>=<val>...] | $id-ref=<id>
+...............
+Example:
+...............
+group internal_www disk0 fs0 internal_ip apache \
+ meta target_role=stopped
+
+group vm-and-services vm vm-sshd meta container="vm"
+...............
+
+[[cmdhelp_configure_load,import the CIB from a file]]
+==== `load`
+
+Load a part of configuration (or all of it) from a local file or
+a network URL. The +replace+ method replaces the current
+configuration with the one from the source. The +update+ method
+tries to import the contents into the current configuration. The
++push+ method imports the contents into the current configuration
+and removes any lines that are not present in the given
+configuration.
+The file may be a CLI file or an XML file.
+
+If the URL is `-`, the configuration is read from standard input.
+
+Usage:
+...............
+load [xml] <method> URL
+
+method :: replace | update | push
+...............
+Example:
+...............
+load xml update myfirstcib.xml
+load xml replace http://storage.big.com/cibs/bigcib.xml
+load xml push smallcib.xml
+...............
+
+[[cmdhelp_configure_location,a location preference]]
+==== `location`
+
+`location` defines the preference of nodes for the given
+resource. The location constraints consist of one or more rules
+which specify a score to be awarded if the rule matches.
+
+The resource referenced by the location constraint can be one of the
+following:
+
+* Plain resource reference: +location loc1 webserver 100: node1+
+* Resource set in curly brackets: +location loc1 { virtual-ip webserver } 100: node1+
+* Tag containing resource ids: +location loc1 tag1 100: node1+
+* Resource pattern: +location loc1 /web.*/ 100: node1+
+
+The +resource-discovery+ attribute allows probes to be selectively
+enabled or disabled per resource and node.
+
+The syntax for resource sets is described in detail for
+<<cmdhelp_configure_colocation,`colocation`>>.
+
+For more details on how to configure resource sets, see
+<<topics_Features_Resourcesets,`Syntax: Resource sets`>>.
+
+For more information on rule expressions, see
+<<topics_Syntax_RuleExpressions,Syntax: Rule expressions>>.
+
+Usage:
+...............
+location <id> <rsc> [<attributes>] {<node_pref>|<rules>}
+
+rsc :: /<rsc-pattern>/
+ | { resource_sets }
+ | <rsc>
+
+attributes :: role=<role> | resource-discovery=always|never|exclusive
+
+node_pref :: <score>: <node>
+
+rules ::
+ rule [id_spec] [$role=<role>] <score>: <expression>
+ [rule [id_spec] [$role=<role>] <score>: <expression> ...]
+
+id_spec :: $id=<id> | $id-ref=<id>
+score :: <number> | <attribute> | [-]inf
+expression :: <simple_exp> [<bool_op> <simple_exp> ...]
+bool_op :: or | and
+simple_exp :: <attribute> [type:]<binary_op> <value>
+ | <unary_op> <attribute>
+ | date <date_expr>
+type :: string | version | number
+binary_op :: lt | gt | lte | gte | eq | ne
+unary_op :: defined | not_defined
+
+date_expr :: lt <end>
+ | gt <start>
+ | in start=<start> end=<end>
+ | in start=<start> <duration>
+ | spec <date_spec>
+duration|date_spec ::
+ hours=<value>
+ | monthdays=<value>
+ | weekdays=<value>
+ | yearsdays=<value>
+ | months=<value>
+ | weeks=<value>
+ | years=<value>
+ | weekyears=<value>
+ | moon=<value>
+...............
+Examples:
+...............
+location conn_1 internal_www 100: node1
+
+location conn_1 internal_www \
+ rule 50: #uname eq node1 \
+ rule pingd: defined pingd
+
+location conn_2 dummy_float \
+ rule -inf: not_defined pingd or pingd number:lte 0
+
+# never probe for rsc1 on node1
+location no-probe rsc1 resource-discovery=never -inf: node1
+...............
+
+[[cmdhelp_configure_modgroup,modify group]]
+==== `modgroup`
+
+Add or remove primitives in a group. The `add` subcommand appends
+the new group member by default. Should it go elsewhere, there
+are `after` and `before` clauses.
+
+Usage:
+...............
+modgroup <id> add <id> [after <id>|before <id>]
+modgroup <id> remove <id>
+...............
+Examples:
+...............
+modgroup share1 add storage2 before share1-fs
+...............
+
+[[cmdhelp_configure_monitor,add monitor operation to a primitive]]
+==== `monitor`
+
+Monitor is by far the most common operation. It is possible to
+add it without editing the whole resource. Also, long primitive
+definitions may be a bit uncluttered. In order to make this
+command as concise as possible, less common operation attributes
+are not available. If you need them, then use the `op` part of
+the `primitive` command.
+
+Usage:
+...............
+monitor <rsc>[:<role>] <interval>[:<timeout>]
+...............
+Example:
+...............
+monitor apcfence 60m:60s
+...............
+
+Note that after executing the command, the monitor operation may
+be shown as part of the primitive definition.
+
+[[cmdhelp_configure_ms,define a master-slave resource]]
+==== `ms` (`master`)
+
+The `ms` command creates a master/slave resource type. It may contain a
+single primitive resource or one group of resources.
+
+Usage:
+...............
+ms <name> <rsc>
+ [description=<description>]
+ [meta attr_list]
+ [params attr_list]
+
+attr_list :: [$id=<id>] <attr>=<val> [<attr>=<val>...] | $id-ref=<id>
+...............
+Example:
+...............
+ms disk1 drbd1 \
+ meta notify=true globally-unique=false
+...............
+
+.Note on `ms` deprecated
+****************************
+From Pacemaker-2.0, the resource type referred to as "master/slave",
+"stateful", or "multi-state" is no longer a separate resource type,
+but a variation of clone now referred to as a "promotable clone".
+For backward compatibility, above configurations are also accepted.
+...............
+clone disk1 drbd1 \
+ meta promotable=true notify=true globally-unique=false
+...............
+****************************
+
+.Note on `id-ref` usage
+****************************
+Instance or meta attributes (`params` and `meta`) may contain
+a reference to another set of attributes. In that case, no other
+attributes are allowed. Since attribute sets' ids, though they do
+exist, are not shown in the `crm`, it is also possible to
+reference an object instead of an attribute set. `crm` will
+automatically replace such a reference with the right id:
+
+...............
+crm(live)configure# primitive a2 www-2 meta $id-ref=a1
+crm(live)configure# show a2
+primitive a2 apache \
+ meta $id-ref=a1-meta_attributes
+ [...]
+...............
+It is advisable to give meaningful names to attribute sets which
+are going to be referenced.
+****************************
+
+[[cmdhelp_configure_node,define a cluster node]]
+==== `node`
+
+The node command describes a cluster node. Nodes in the CIB are
+commonly created automatically by the CRM. Hence, you should not
+need to deal with nodes unless you also want to define node
+attributes. Note that it is also possible to manage node
+attributes at the `node` level.
+
+Usage:
+...............
+node [$id=<id>] <uname>[:<type>]
+ [description=<description>]
+ [attributes [$id=<id>] [<score>:] [rule...]
+ <param>=<value> [<param>=<value>...]] | $id-ref=<ref>
+ [utilization [$id=<id>] [<score>:] [rule...]
+ <param>=<value> [<param>=<value>...]] | $id-ref=<ref>
+
+type :: normal | member | ping | remote
+...............
+Example:
+...............
+node node1
+node big_node attributes memory=64
+...............
+
+[[cmdhelp_configure_op_defaults,set resource operations defaults]]
+==== `op_defaults`
+
+Set defaults for the operations meta attributes.
+
+For more information on rule expressions, see
+<<topics_Syntax_RuleExpressions,Syntax: Rule expressions>>.
+
+Usage:
+...............
+op_defaults [$id=<set_id>] [rule ...] <option>=<value> [<option>=<value> ...]
+...............
+Example:
+...............
+op_defaults record-pending=true
+...............
+
+[[cmdhelp_configure_order,order resources]]
+==== `order`
+
+This constraint expresses the order of actions on two resources
+or more resources. If there are more than two resources, then the
+constraint is called a resource set.
+
+Ordered resource sets have an extra attribute to allow for sets
+of resources whose actions may run in parallel. The shell syntax
+for such sets is to put resources in parentheses.
+
+If the subsequent resource can start or promote after any one of the
+resources in a set has done, enclose the set in brackets (+[+ and +]+).
+
+Sets cannot be nested.
+
+Three strings are reserved to specify a kind of order constraint:
++Mandatory+, +Optional+, and +Serialize+. It is preferred to use
+one of these settings instead of score. Previous versions mapped
+scores +0+ and +inf+ to keywords +advisory+ and +mandatory+.
+That is still valid but deprecated.
+
+For more details on how to configure resource sets, see
+<<topics_Features_Resourcesets,`Syntax: Resource sets`>>.
+
+Usage:
+...............
+order <id> [kind:] first then [symmetrical=<bool>]
+
+order <id> [kind:] resource_sets [symmetrical=<bool>]
+
+kind :: Mandatory | Optional | Serialize
+
+first :: <rsc>[:<action>]
+
+then :: <rsc>[:<action>]
+
+resource_sets :: resource_set [resource_set ...]
+
+resource_set :: ["["|"("] <rsc>[:<action>] [<rsc>[:<action>] ...] \
+ [attributes] ["]"|")"]
+
+attributes :: [require-all=(true|false)] [sequential=(true|false)]
+
+...............
+Example:
+...............
+order o-1 Mandatory: apache:start ip_1
+order o-2 Serialize: A ( B C )
+order o-4 first-resource then-resource
+...............
+
+[[cmdhelp_configure_primitive,define a resource]]
+==== `primitive`
+
+The primitive command describes a resource. It may be referenced
+only once in group, clone, or master-slave objects. If it's not
+referenced, then it is placed as a single resource in the CIB.
+
+Operations may be specified anonymously, as a group or by reference:
+
+* "Anonymous", as a list of +op+ specifications. Use this
+ method if you don't need to reference the set of operations
+ elsewhere. This is the most common way to define operations.
+
+* If reusing operation sets is desired, use the +operations+ keyword
+ along with an id to give the operations set a name. Use the
+ +operations+ keyword and an id-ref value set to the id of another
+ operations set, to apply the same set of operations to this
+ primitive.
+
+Operation attributes which are not recognized are saved as
+instance attributes of that operation. A typical example is
++OCF_CHECK_LEVEL+.
+
+For multistate resources, roles are specified as +role=<role>+.
+
+A template may be defined for resources which are of the same
+type and which share most of the configuration. See
+<<cmdhelp_configure_rsc_template,`rsc_template`>> for more information.
+
+Attributes containing time values, such as the +interval+ attribute on
+operations, are configured either as a plain number, which is
+interpreted as a time in seconds, or using one of the following
+suffixes:
+
+* +s+, +sec+ - time in seconds (same as no suffix)
+* +ms+, +msec+ - time in milliseconds
+* +us+, +usec+ - time in microseconds
+* +m+, +min+ - time in minutes
+* +h+, +hr+ - time in hours
+
+Usage:
+...............
+primitive <rsc> {[<class>:[<provider>:]]<type>|@<template>}
+ [description=<description>]
+ [[params] attr_list]
+ [meta attr_list]
+ [utilization attr_list]
+ [operations id_spec]
+ [op op_type [<attribute>=<value>...]
+ [[op_params] attr_list]
+ [op_meta attr_list] ...]
+
+attr_list :: [$id=<id>] [<score>:] [rule...]
+ <attr>=<val> [<attr>=<val>...]] | $id-ref=<id>
+id_spec :: $id=<id> | $id-ref=<id>
+op_type :: start | stop | monitor
+...............
+Example:
+...............
+primitive apcfence stonith:apcsmart \
+ params ttydev=/dev/ttyS0 hostlist="node1 node2" \
+ op start timeout=60s \
+ op monitor interval=30m timeout=60s
+
+primitive www8 apache \
+ configfile=/etc/apache/www8.conf \
+ operations $id-ref=apache_ops
+
+primitive db0 mysql \
+ params config=/etc/mysql/db0.conf \
+ op monitor interval=60s \
+ op monitor interval=300s OCF_CHECK_LEVEL=10
+
+primitive r0 ocf:linbit:drbd \
+ params drbd_resource=r0 \
+ op monitor role=Master interval=60s \
+ op monitor role=Slave interval=300s
+
+primitive xen0 @vm_scheme1 xmfile=/etc/xen/vm/xen0
+
+primitive mySpecialRsc Special \
+ params 3: rule #uname eq node1 interface=eth1 \
+ params 2: rule #uname eq node2 interface=eth2 port=8888 \
+ params 1: interface=eth0 port=9999
+
+primitive A ocf:pacemaker:Dummy \
+ op start \
+ op_meta 2: rule #ra-version version:gt 1.0 timeout=120s \
+ op_meta 1: timeout=60s
+...............
+
+[[cmdhelp_configure_property,set a cluster property]]
+==== `property`
+
+Set cluster configuration properties. To list the
+available cluster configuration properties, use the
+<<cmdhelp_ra_info,`ra info`>> command with +pengine+, +crmd+,
++cib+ and +stonithd+ as arguments.
+When setting the +maintenance-mode+ property, it will
+inform the user if there are nodes or resources that
+have the +maintenance+ property.
+
+For more information on rule expressions, see
+<<topics_Syntax_RuleExpressions,Syntax: Rule expressions>>.
+
+Usage:
+...............
+property [<set_id>:] [rule ...] <option>=<value> [<option>=<value> ...]
+...............
+Example:
+...............
+property stonith-enabled=true
+property rule date spec years=2014 stonith-enabled=false
+...............
+
+[[cmdhelp_configure_ptest,show cluster actions if changes were committed]]
+==== `ptest` (`simulate`)
+
+Show PE (Policy Engine) motions using `ptest(8)` or
+`crm_simulate(8)`.
+
+A CIB is constructed using the current user edited configuration
+and the status from the running CIB. The resulting CIB is run
+through `ptest` (or `crm_simulate`) to show changes which would
+happen if the configuration is committed.
+
+The status section may be loaded from another source and modified
+using the <<cmdhelp_cibstatus,`cibstatus`>> level commands. In that case, the
+`ptest` command will issue a message informing the user that the
+Policy Engine graph is not calculated based on the current status
+section and therefore won't show what would happen to the
+running but some imaginary cluster.
+
+If you have graphviz installed and X11 session, `dotty(1)` is run
+to display the changes graphically.
+
+Add a string of +v+ characters to increase verbosity. `ptest`
+can also show allocation scores. +utilization+ turns on
+information about the remaining capacity of nodes. With the
++actions+ option, `ptest` will print all resource actions.
+
+The `ptest` program has been replaced by `crm_simulate` in newer
+Pacemaker versions. In some installations both could be
+installed. Use `simulate` to enfore using `crm_simulate`.
+
+Usage:
+...............
+ptest [nograph] [v...] [scores] [actions] [utilization]
+...............
+Examples:
+...............
+ptest scores
+ptest vvvvv
+simulate actions
+...............
+
+[[cmdhelp_configure_refresh,refresh from CIB]]
+==== `refresh`
+
+Refresh the internal structures from the CIB. All changes made
+during this session are lost.
+
+Usage:
+...............
+refresh
+...............
+
+[[cmdhelp_configure_rename,rename a CIB object]]
+==== `rename`
+
+Rename an object. It is recommended to use this command to rename
+a resource, because it will take care of updating all related
+constraints and a parent resource. Changing ids with the edit
+command won't have the same effect.
+
+If you want to rename a resource, it must be in the stopped state.
+
+Usage:
+...............
+rename <old_id> <new_id>
+...............
+
+[[cmdhelp_configure_role,define role access rights]]
+==== `role`
+
+An ACL role is a set of rules which describe access rights to
+CIB. Rules consist of an access right +read+, +write+, or +deny+
+and a specification denoting part of the configuration to which
+the access right applies. The specification can be an XPath or a
+combination of tag and id references. If an attribute is
+appended, then the specification applies only to that attribute
+of the matching element.
+
+There is a number of shortcuts for XPath specifications. The
++meta+, +params+, and +utilization+ shortcuts reference resource
+meta attributes, parameters, and utilization respectively. The
+`location` may be used to specify location constraints most of
+the time to allow resource `move` and `unmove` commands. The
+`property` references cluster properties. The `node` allows
+reading node attributes. +nodeattr+ and +nodeutil+ reference node
+attributes and node capacity (utilization). The `status` shortcut
+references the whole status section of the CIB. Read access to
+status is necessary for various monitoring tools such as
+`crm_mon(8)` (aka `crm status`).
+
+For more information on rule expressions, see
+<<topics_Syntax_RuleExpressions,Syntax: Rule expressions>>.
+
+Usage:
+...............
+role <role-id> rule [rule ...]
+
+rule :: acl-right cib-spec [attribute:<attribute>]
+
+acl-right :: read | write | deny
+
+cib-spec :: xpath-spec | tag-ref-spec
+xpath-spec :: xpath:<xpath> | shortcut
+tag-ref-spec :: tag:<tag> | ref:<id> | tag:<tag> ref:<id>
+
+shortcut :: meta:<rsc>[:<attr>]
+ params:<rsc>[:<attr>]
+ utilization:<rsc>
+ location:<rsc>
+ property[:<attr>]
+ node[:<node>]
+ nodeattr[:<attr>]
+ nodeutil[:<node>]
+ status
+...............
+Example:
+...............
+role app1_admin \
+ write meta:app1:target-role \
+ write meta:app1:is-managed \
+ write location:app1 \
+ read ref:app1
+...............
+
+[[cmdhelp_configure_rsc_defaults,set resource defaults]]
+==== `rsc_defaults`
+
+Set defaults for the resource meta attributes.
+
+For more information on rule expressions, see
+<<topics_Syntax_RuleExpressions,Syntax: Rule expressions>>.
+
+Usage:
+...............
+rsc_defaults [<set_id>:] [rule ...] <option>=<value> [<option>=<value> ...]
+...............
+Example:
+...............
+rsc_defaults failure-timeout=3m
+...............
+
+[[cmdhelp_configure_rsc_template,define a resource template]]
+==== `rsc_template`
+
+The `rsc_template` command creates a resource template. It may be
+referenced in primitives. It is used to reduce large
+configurations with many similar resources.
+
+Usage:
+...............
+rsc_template <name> [<class>:[<provider>:]]<type>
+ [description=<description>]
+ [params attr_list]
+ [meta attr_list]
+ [utilization attr_list]
+ [operations id_spec]
+ [op op_type [<attribute>=<value>...] ...]
+
+attr_list :: [$id=<id>] <attr>=<val> [<attr>=<val>...] | $id-ref=<id>
+id_spec :: $id=<id> | $id-ref=<id>
+op_type :: start | stop | monitor
+...............
+Example:
+...............
+rsc_template public_vm Xen \
+ op start timeout=300s \
+ op stop timeout=300s \
+ op monitor interval=30s timeout=60s \
+ op migrate_from timeout=600s \
+ op migrate_to timeout=600s
+primitive xen0 @public_vm \
+ params xmfile=/etc/xen/xen0
+primitive xen1 @public_vm \
+ params xmfile=/etc/xen/xen1
+...............
+
+[[cmdhelp_configure_rsc_ticket,resources ticket dependency]]
+==== `rsc_ticket`
+
+This constraint expresses dependency of resources on cluster-wide
+attributes, also known as tickets. Tickets are mainly used in
+geo-clusters, which consist of multiple sites. A ticket may be
+granted to a site, thus allowing resources to run there.
+
+The +loss-policy+ attribute specifies what happens to the
+resource (or resources) if the ticket is revoked. The default is
+either +stop+ or +demote+ depending on whether a resource is
+multi-state.
+
+See also the <<cmdhelp_site_ticket,`site`>> set of commands.
+
+Usage:
+...............
+rsc_ticket <id> <ticket_id>: <rsc>[:<role>] [<rsc>[:<role>] ...]
+ [loss-policy=<loss_policy_action>]
+
+loss_policy_action :: stop | demote | fence | freeze
+...............
+Example:
+...............
+rsc_ticket ticket-A_public-ip ticket-A: public-ip
+rsc_ticket ticket-A_bigdb ticket-A: bigdb loss-policy=fence
+rsc_ticket ticket-B_storage ticket-B: drbd-a:Master drbd-b:Master
+...............
+
+
+[[cmdhelp_configure_rsctest,test resources as currently configured]]
+==== `rsctest`
+
+Test resources with current resource configuration. If no nodes
+are specified, tests are run on all known nodes.
+
+The order of resources is significant: it is assumed that later
+resources depend on earlier ones.
+
+If a resource is multi-state, it is assumed that the role on
+which later resources depend is master.
+
+Tests are run sequentially to prevent running the same resource
+on two or more nodes. Tests are carried out only if none of the
+specified nodes currently run any of the specified resources.
+However, it won't verify whether resources run on the other
+nodes.
+
+Superuser privileges are obviously required: either run this as
+root or setup the `sudoers` file appropriately.
+
+Note that resource testing may take some time.
+
+Usage:
+...............
+rsctest <rsc_id> [<rsc_id> ...] [<node_id> ...]
+...............
+Examples:
+...............
+rsctest my_ip websvc
+rsctest websvc nodeB
+...............
+
+[[cmdhelp_configure_save,save the CIB to a file]]
+==== `save`
+
+Save the current configuration to a file. Optionally, as XML. Use
++-+ instead of file name to write the output to `stdout`.
+
+The `save` command accepts the same selection arguments as the `show`
+command. See the <<cmdhelp_configure_show,help section>> for `show`
+for more details.
+
+Usage:
+...............
+save [xml] [<id> | type:<type | tag:<tag> |
+ related:<obj> | changed ...] <file>
+...............
+Example:
+...............
+save myfirstcib.txt
+save web-server server-config.txt
+...............
+
+[[cmdhelp_configure_schema,set or display current CIB RNG schema]]
+==== `schema`
+
+CIB's content is validated by a RNG schema. Pacemaker supports
+several, depending on version. At least the following schemas are
+accepted by `crmsh`:
+
+* +pacemaker-1.0+
+* +pacemaker-1.1+
+* +pacemaker-1.2+
+* +pacemaker-1.3+
+* +pacemaker-2.0+
+
+Use this command to display or switch to another RNG schema.
+
+Usage:
+...............
+schema [<schema>]
+...............
+Example:
+...............
+schema pacemaker-1.1
+...............
+
+[[cmdhelp_configure_set,set an attribute value]]
+==== `set`
+
+Set the value of a configured attribute. The attribute must
+configured previously, and can be an agent parameter, meta attribute,
+utilization value or operation value.
+
+The first argument to the command is a path to an attribute.
+This is a dot-separated sequence beginning with the name of
+the resource or object, and ending with the name of the attribute to
+set. To set operation value, `op_type` should be specified; when multi
+operations exist like multi monitors, `interval` should be specified.
+
+Usage:
+...............
+set <path> <value>
+
+path:: id.[op_type.][interval.]name
+...............
+Examples:
+...............
+set vip1.ip 192.168.20.5
+set vm-a.force_stop 1
+set vip1.monitor.on-fail ignore
+set drbd.monitor.10s.interval 20s
+...............
+
+[[cmdhelp_configure_show,display CIB objects]]
+==== `show`
+
+The `show` command displays CIB objects. Without any argument, it
+displays all objects in the CIB, but the set of objects displayed by
+`show` can be limited to only objects with the given IDs or by using
+one or more of the special prefixes described below.
+
+The XML representation for the objects can be displayed by passing
++xml+ as the first argument.
+
+To show one or more specific objects, pass the object IDs as
+arguments.
+
+To show all objects of a certain type, use the +type:+ prefix.
+
+To show all objects in a tag, use the +tag:+ prefix.
+
+To show all constraints related to a primitive, use the +related:+ prefix.
+
+To show all modified objects, pass the argument +changed+.
+
+The prefixes can be used together on a single command line. For
+example, to show both the tag itself and the objects tagged by it the
+following combination can be used: +show tag:my-tag my-tag+.
+
+To refine a selection of objects using multiple modifiers, the keywords
++and+ and +or+ can be used. For example, to select all primitives tagged
++foo+, the following combination can be used:
++show type:primitive and tag:foo+.
+
+To hide values when displaying the configuration, use the
++obscure:<glob>+ argument. This can be useful when sending the
+configuration over a public channel, to avoid exposing potentially
+sensitive information. The +<glob>+ argument is a bash-style pattern
+matching attribute keys.
+
+In +/etc/crm/crm.conf+, +obscure_pattern+ option is the persisent configuration of CLI.
+Example, for the high security concern,
+...............
+[core]
+obscure_pattern = passw* | ip
+...............
+Which makes +crm configure show+ is equal to
+...............
+node-1:~ # crm configure show obscure:passw* obscure:ip
+node 1084783297: node1
+primitive fence_device stonith:fence_ilo5 \
+ params password="******"
+primitive ip IPaddr2 \
+ params ip="******"
+...............
+The default suggestion is +passw*+
+If you don't want to obscure, change the value to blank.
+
+Usage:
+...............
+show [xml] [<id>
+ | changed
+ | type:<type>
+ | tag:<id>
+ | related:<obj>
+ | obscure:<glob>
+ ...]
+
+type :: node | primitive | group | clone | ms | rsc_template
+ | location | colocation | order
+ | rsc_ticket
+ | property | rsc_defaults | op_defaults
+ | fencing_topology
+ | role | user | acl_target
+ | tag
+...............
+
+Example:
+...............
+show webapp
+show type:primitive
+show xml tag:db tag:fs
+show related:webapp
+show type:primitive obscure:passwd
+...............
+
+[[cmdhelp_configure_tag,Define resource tags]]
+==== `tag`
+
+Define a resource tag. A tag is an id referring to one or more
+resources, without implying any constraints between the tagged
+resources. This can be useful for grouping conceptually related
+resources.
+
+Usage:
+...............
+tag <tag-name>: <rsc> [<rsc> ...]
+tag <tag-name> <rsc> [<rsc> ...]
+...............
+Example:
+...............
+tag web: p-webserver p-vip
+tag ips server-vip admin-vip
+...............
+
+[[cmdhelp_configure_template,edit and import a configuration from a template]]
+==== `template`
+
+The specified template is loaded into the editor. It's up to the
+user to make a good CRM configuration out of it. See also the
+<<cmdhelp_template,template section>>.
+
+Usage:
+...............
+template [xml] url
+...............
+Example:
+...............
+template two-apaches.txt
+...............
+
+[[cmdhelp_configure_upgrade,upgrade the CIB]]
+==== `upgrade`
+
+Attempts to upgrade the CIB to validate with the current
+version. Commonly, this is required if the error
+`CIB not supported` occurs. It typically means that the
+active CIB version is coming from an older release.
+
+As a safety precaution, the force argument is required if the
++validation-with+ attribute is set to anything other than
++0.6+. Thus in most cases, it is required.
+
+Usage:
+...............
+upgrade [force]
+...............
+
+Example:
+...............
+upgrade force
+...............
+
+[[cmdhelp_configure_user,define user access rights]]
+==== `user`
+
+Users which normally cannot view or manage cluster configuration
+can be allowed access to parts of the CIB. The access is defined
+by a set of +read+, +write+, and +deny+ rules as in role
+definitions or by referencing roles. The latter is considered
+best practice.
+
+For more information on rule expressions, see
+<<topics_Syntax_RuleExpressions,Syntax: Rule expressions>>.
+
+Usage:
+...............
+user <uid> {roles|rules}
+
+roles :: role:<role-ref> [role:<role-ref> ...]
+rules :: rule [rule ...]
+...............
+Example:
+...............
+user joe \
+ role:app1_admin \
+ role:read_all
+...............
+
+[[cmdhelp_configure_validate_all,call agent validate-all for resource]]
+==== `validate-all`
+
+Call the `validate-all` action for the resource, if possible.
+
+Limitations:
+
+* The resource agent must implement the `validate-all` action.
+* The current user must be root.
+* The primitive resource must not use nvpair references.
+
+Usage:
+...............
+validate-all <rsc>
+...............
+
+
+[[cmdhelp_configure_verify,verify the CIB with crm_verify]]
+==== `verify`
+
+Verify the contents of the CIB which would be committed.
+
+Usage:
+...............
+verify
+...............
+
+[[cmdhelp_configure_xml,raw xml]]
+==== `xml`
+
+Even though we promissed no xml, it may happen, but hopefully
+very very seldom, that an element from the CIB cannot be rendered
+in the configuration language. In that case, the element will be
+shown as raw xml, prefixed by this command. That element can then
+be edited like any other. If the shell finds out that after the
+change it can digest it, then it is going to be converted into
+the normal configuration language. Otherwise, there is no need to
+use `xml` for configuration.
+
+Usage:
+...............
+xml <xml>
+...............
+
+[[cmdhelp_template,edit and import a configuration from a template]]
+=== `template` - Import configuration from templates
+
+User may be assisted in the cluster configuration by templates
+prepared in advance. Templates consist of a typical ready
+configuration which may be edited to suit particular user needs.
+
+This command enters a template level where additional commands
+for configuration/template management are available.
+
+[[cmdhelp_template_apply,process and apply the current configuration to the current CIB]]
+==== `apply`
+
+Copy the current or given configuration to the current CIB. By
+default, the CIB is replaced, unless the method is set to
+"update".
+
+Usage:
+...............
+apply [<method>] [<config>]
+
+method :: replace | update
+...............
+
+[[cmdhelp_template_delete,delete a configuration]]
+==== `delete`
+
+Remove a configuration. The loaded (active) configuration may be
+removed by force.
+
+Usage:
+...............
+delete <config> [force]
+...............
+
+[[cmdhelp_template_edit,edit a configuration]]
+==== `edit`
+
+Edit current or given configuration using your favourite editor.
+
+Usage:
+...............
+edit [<config>]
+...............
+
+[[cmdhelp_template_list,list configurations/templates]]
+==== `list`
+
+When called with no argument, lists existing templates and
+configurations.
+
+Given the argument +templates+, lists the available templates.
+
+Given the argument +configs+, lists the available configurations.
+
+Usage:
+...............
+list [templates|configs]
+...............
+
+[[cmdhelp_template_load,load a configuration]]
+==== `load`
+
+Load an existing configuration. Further `edit`, `show`, and
+`apply` commands will refer to this configuration.
+
+Usage:
+...............
+load <config>
+...............
+
+[[cmdhelp_template_new,create a new configuration from templates]]
+==== `new`
+
+Create a new configuration from one or more templates. Note that
+configurations and templates are kept in different places, so it
+is possible to have a configuration name equal a template name.
+
+If you already know which parameters are required, you can set
+them directly on the command line.
+
+The parameter name +id+ is set by default to the name of the
+configuration.
+
+If no parameters are being set and you don't want a particular name
+for your configuration, you can call this command with a template name
+as the only parameter. A unique configuration name based on the
+template name will be generated.
+
+Usage:
+...............
+new [<config>] <template> [<template> ...] [params name=value ...]
+...............
+
+Example:
+...............
+new vip virtual-ip
+new bigfs ocfs2 params device=/dev/sdx8 directory=/bigfs
+new apache
+...............
+
+[[cmdhelp_template_show,show the processed configuration]]
+==== `show`
+
+Process the current or given configuration and display the result.
+
+Usage:
+...............
+show [<config>]
+...............
+
+[[cmdhelp_cibstatus,CIB status management and editing]]
+=== `cibstatus` - CIB status management and editing
+
+The `status` section of the CIB keeps the current status of nodes
+and resources. It is modified _only_ on events, i.e. when some
+resource operation is run or node status changes. For obvious
+reasons, the CRM has no user interface with which it is possible
+to affect the status section. From the user's point of view, the
+status section is essentially a read-only part of the CIB. The
+current status is never even written to disk, though it is
+available in the PE (Policy Engine) input files which represent
+the history of cluster motions. The current status may be read
+using the +cibadmin -Q+ command.
+
+It may sometimes be of interest to see how status changes would
+affect the Policy Engine. The set of `cibstatus` level commands
+allow the user to load status sections from various sources and
+then insert or modify resource operations or change nodes' state.
+
+The effect of those changes may then be observed by running the
+<<cmdhelp_configure_ptest,`ptest`>> command at the `configure` level
+or `simulate` and `run` commands at this level. The `ptest`
+runs with the user edited CIB whereas the latter two commands
+run with the CIB which was loaded along with the status section.
+
+The `simulate` and `run` commands as well as all status
+modification commands are implemented using `crm_simulate(8)`.
+
+[[cmdhelp_cibstatus_load,load the CIB status section]]
+==== `load`
+
+Load a status section from a file, a shadow CIB, or the running
+cluster. By default, the current (+live+) status section is
+modified. Note that if the +live+ status section is modified it
+is not going to be updated if the cluster status changes, because
+that would overwrite the user changes. To make `crm` drop changes
+and resume use of the running cluster status, run +load live+.
+
+All CIB shadow configurations contain the status section which is
+a snapshot of the status section taken at the time the shadow was
+created. Obviously, this status section doesn't have much to do
+with the running cluster status, unless the shadow CIB has just
+been created. Therefore, the `ptest` command by default uses the
+running cluster status section.
+
+Usage:
+...............
+load {<file>|shadow:<cib>|live}
+...............
+Example:
+...............
+load bug-12299.xml
+load shadow:test1
+...............
+
+[[cmdhelp_cibstatus_node,change node status]]
+==== `node`
+
+Change the node status. It is possible to throw a node out of
+the cluster, make it a member, or set its state to unclean.
+
++online+:: Set the +node_state+ `crmd` attribute to +online+
+and the +expected+ and +join+ attributes to +member+. The effect
+is that the node becomes a cluster member.
+
++offline+:: Set the +node_state+ `crmd` attribute to +offline+
+and the +expected+ attribute to empty. This makes the node
+cleanly removed from the cluster.
+
++unclean+:: Set the +node_state+ `crmd` attribute to +offline+
+and the +expected+ attribute to +member+. In this case the node
+has unexpectedly disappeared.
+
+Usage:
+...............
+node <node> {online|offline|unclean}
+...............
+Example:
+...............
+node xen-b unclean
+...............
+
+[[cmdhelp_cibstatus_op,edit outcome of a resource operation]]
+==== `op`
+
+Edit the outcome of a resource operation. This way you can
+tell CRM that it ran an operation and that the resource agent
+returned certain exit code. It is also possible to change the
+operation's status. In case the operation status is set to
+something other than +done+, the exit code is effectively
+ignored.
+
+Usage:
+...............
+op <operation> <resource> <exit_code> [<op_status>] [<node>]
+
+operation :: probe | monitor[:<n>] | start | stop |
+ promote | demote | notify | migrate_to | migrate_from
+exit_code :: <rc> | success | generic | args |
+ unimplemented | perm | installed | configured | not_running |
+ master | failed_master
+op_status :: pending | done | cancelled | timeout | notsupported | error
+
+n :: the monitor interval in seconds; if omitted, the first
+ recurring operation is referenced
+rc :: numeric exit code in range 0..9
+...............
+Example:
+...............
+op start d1 xen-b generic
+op start d1 xen-b 1
+op monitor d1 xen-b not_running
+op stop d1 xen-b 0 timeout
+...............
+
+[[cmdhelp_cibstatus_origin,display origin of the CIB status section]]
+==== `origin`
+
+Show the origin of the status section currently in use. This
+essentially shows the latest `load` argument.
+
+Usage:
+...............
+origin
+...............
+
+[[cmdhelp_cibstatus_quorum,set the quorum]]
+==== `quorum`
+
+Set the quorum value.
+
+Usage:
+...............
+quorum <bool>
+...............
+Example:
+...............
+quorum false
+...............
+
+[[cmdhelp_cibstatus_run,run policy engine]]
+==== `run`
+
+Run the policy engine with the edited status section.
+
+Add a string of +v+ characters to increase verbosity. Specify
++scores+ to see allocation scores also. +utilization+ turns on
+information about the remaining capacity of nodes.
+
+If you have graphviz installed and X11 session, `dotty(1)` is run
+to display the changes graphically.
+
+Usage:
+...............
+run [nograph] [v...] [scores] [utilization]
+...............
+Example:
+...............
+run
+...............
+
+[[cmdhelp_cibstatus_save,save the CIB status section]]
+==== `save`
+
+The current internal status section with whatever modifications
+were performed can be saved to a file or shadow CIB.
+
+If the file exists and contains a complete CIB, only the status
+section is going to be replaced and the rest of the CIB will
+remain intact. Otherwise, the current user edited configuration
+is saved along with the status section.
+
+Note that all modifications are saved in the source file as soon
+as they are run.
+
+Usage:
+...............
+save [<file>|shadow:<cib>]
+...............
+Example:
+...............
+save bug-12299.xml
+...............
+
+[[cmdhelp_cibstatus_show,show CIB status section]]
+==== `show`
+
+Show the current status section in the XML format. Brace yourself
+for some unreadable output. Add +changed+ option to get a human
+readable output of all changes.
+
+Usage:
+...............
+show [changed]
+...............
+
+[[cmdhelp_cibstatus_simulate,simulate cluster transition]]
+==== `simulate`
+
+Run the policy engine with the edited status section and simulate
+the transition.
+
+Add a string of +v+ characters to increase verbosity. Specify
++scores+ to see allocation scores also. +utilization+ turns on
+information about the remaining capacity of nodes.
+
+If you have graphviz installed and X11 session, `dotty(1)` is run
+to display the changes graphically.
+
+Usage:
+...............
+simulate [nograph] [v...] [scores] [utilization]
+...............
+Example:
+...............
+simulate
+...............
+
+[[cmdhelp_cibstatus_ticket,manage tickets]]
+==== `ticket`
+
+Modify the ticket status. Tickets can be granted and revoked.
+Granted tickets could be activated or put in standby.
+
+Usage:
+...............
+ticket <ticket> {grant|revoke|activate|standby}
+...............
+Example:
+...............
+ticket ticketA grant
+...............
+
+[[cmdhelp_assist,Configuration assistant]]
+=== `assist` - Configuration assistant
+
+The `assist` sublevel is a collection of helper
+commands that create or modify resources and
+constraints, to simplify the creation of certain
+configurations.
+
+For more information on individual commands, see
+the help text for those commands.
+
+[[cmdhelp_assist_template,Create template for primitives]]
+==== `template`
+
+This command takes a list of primitives as argument, and creates a new
+`rsc_template` for these primitives. It can only do this if the
+primitives do not already share a template and are of the same type.
+
+Usage:
+........
+template primitive-1 primitive-2 primitive-3
+........
+
+[[cmdhelp_assist_weak-bond,Create a weak bond between resources]]
+==== `weak-bond`
+
+A colocation between a group of resources says that the resources
+should be located together, but it also means that those resources are
+dependent on each other. If one of the resources fails, the others
+will be restarted.
+
+If this is not desired, it is possible to circumvent: By placing the
+resources in a non-sequential set and colocating the set with a dummy
+resource which is not monitored, the resources will be placed together
+but will have no further dependency on each other.
+
+This command creates both the constraint and the dummy resource needed
+for such a colocation.
+
+Usage:
+........
+weak-bond resource-1 resource-2
+........
+
+[[cmdhelp_maintenance,Maintenance mode commands]]
+=== `maintenance` - Maintenance mode commands
+
+Maintenance mode commands are commands that manipulate resources
+directly without going through the cluster infrastructure. Therefore,
+it is essential to ensure that the cluster does not attempt to monitor
+or manipulate the resources while these commands are being executed.
+
+To ensure this, these commands require that maintenance mode is set
+either for the particular resource, or for the whole cluster.
+
+[[cmdhelp_maintenance_action,Invoke a resource action]]
+==== `action`
+
+Invokes the given action for the resource. This is
+done directly via the resource agent, so the command must
+be issued while the cluster or the resource is in
+maintenance mode.
+
+Unless the action is `start` or `monitor`, the action must be invoked
+on the same node as where the resource is running. If the resource is
+running on multiple nodes, the command will fail.
+
+To use SSH for executing resource actions on multiple nodes, append
+`ssh` after the action name. This requires SSH access to be configured
+between the nodes and the parallax python package to be installed.
+
+Usage:
+...............
+action <rsc> <action>
+action <rsc> <action> ssh
+...............
+Example:
+...............
+action webserver reload
+action webserver monitor ssh
+...............
+
+[[cmdhelp_maintenance_off,Disable maintenance mode]]
+==== `off`
+
+Disables maintenances mode, either for the whole cluster
+or for the given resource.
+
+Usage:
+...............
+off
+off <rsc>
+...............
+Example:
+...............
+off rsc1
+...............
+
+[[cmdhelp_maintenance_on,Enable maintenance mode]]
+==== `on`
+
+Enables maintenances mode, either for the whole cluster
+or for the given resource.
+
+Usage:
+...............
+on
+on <rsc>
+...............
+Example:
+...............
+on rsc1
+...............
+
+[[cmdhelp_history,Cluster history]]
+=== `history` - Cluster history
+
+Examining Pacemaker's history is a particularly involved task. The
+number of subsystems to be considered, the complexity of the
+configuration, and the set of various information sources, most of
+which are not exactly human readable, keep analyzing resource or node
+problems accessible to only the most knowledgeable. Or, depending on
+the point of view, to the most persistent. The following set of
+commands has been devised in hope to make cluster history more
+accessible.
+
+Of course, looking at _all_ history could be time consuming regardless
+of how good the tools at hand are. Therefore, one should first say
+which period he or she wants to analyze. If not otherwise specified,
+the last hour is considered. Logs and other relevant information is
+collected using `crm report`. Since this process takes some time and
+we always need fresh logs, information is refreshed in a much faster
+way using the python parallax module. If +python-parallax+ is not
+found on the system, examining a live cluster is still possible --
+though not as comfortable.
+
+Apart from examining a live cluster, events may be retrieved from a
+report generated by `crm report` (see also the +-H+ option). In that
+case we assume that the period stretching the whole report needs to be
+investigated. Of course, it is still possible to further reduce the
+time range.
+
+If you have discovered an issue that you want to show someone else,
+you can use the `session pack` command to save the current session as
+a tarball, similar to those generated by `crm report`.
+
+In order to minimize the size of the tarball, and to make it easier
+for others to find the interesting events, it is recommended to limit
+the time frame which the saved session covers. This can be done using
+the `timeframe` command (example below).
+
+It is also possible to name the saved session using the `session save`
+command.
+
+Example:
+...............
+crm(live)history# limit "Jul 18 12:00" "Jul 18 12:30"
+crm(live)history# session save strange_restart
+crm(live)history# session pack
+Report saved in .../strange_restart.tar.bz2
+crm(live)history#
+...............
+
+[[cmdhelp_history_detail,set the level of detail shown]]
+==== `detail`
+
+How much detail to show from the logs. Valid detail levels are either
+`0` or `1`, where `1` is the highest detail level. The default detail
+level is `0`.
+
+Usage:
+...............
+detail <detail_level>
+
+detail_level :: small integer (defaults to 0)
+...............
+Example:
+...............
+detail 1
+...............
+
+[[cmdhelp_history_diff,cluster states/transitions difference]]
+==== `diff`
+
+A transition represents a change in cluster configuration or
+state. Use `diff` to see what has changed between two
+transitions.
+
+If you want to specify the current cluster configuration and
+status, use the string +live+.
+
+Normally, the first transition specified should be the one which
+is older, but we are not going to enforce that.
+
+Note that a single configuration update may result in more than
+one transition.
+
+Usage:
+...............
+diff <pe> <pe> [status] [html]
+
+pe :: <number>|<index>|<file>|live
+...............
+Examples:
+...............
+diff 2066 2067
+diff pe-input-2080.bz2 live status
+...............
+
+[[cmdhelp_history_events,Show events in log]]
+==== `events`
+
+By analysing the log output and looking for particular
+patterns, the `events` command helps sifting through
+the logs to find when particular events like resources
+changing state or node failure may have occurred.
+
+This can be used to generate a combined list of events
+from all nodes.
+
+Usage:
+...............
+events
+...............
+
+Example:
+...............
+events
+...............
+
+[[cmdhelp_history_exclude,exclude log messages]]
+==== `exclude`
+
+If a log is infested with irrelevant messages, those messages may
+be excluded by specifying a regular expression. The regular
+expressions used are Python extended. This command is additive.
+To drop all regular expressions, use +exclude clear+. Run
+`exclude` only to see the current list of regular expressions.
+Excludes are saved along with the history sessions.
+
+Usage:
+...............
+exclude [<regex>|clear]
+...............
+Example:
+...............
+exclude kernel.*ocfs2
+...............
+
+[[cmdhelp_history_graph,generate a directed graph from the PE file]]
+==== `graph`
+
+Create a graphviz graphical layout from the PE file (the
+transition). Every transition contains the cluster configuration
+which was active at the time. See also <<cmdhelp_configure_graph,generate a directed graph
+from configuration>>.
+
+Usage:
+...............
+graph <pe> [<gtype> [<file> [<img_format>]]]
+
+gtype :: dot
+img_format :: `dot` output format (see the +-T+ option)
+...............
+Example:
+...............
+graph -1
+graph 322 dot clu1.conf.dot
+graph 322 dot clu1.conf.svg svg
+...............
+
+[[cmdhelp_history_info,Cluster information summary]]
+==== `info`
+
+The `info` command provides a summary of the information source, which
+can be either a live cluster snapshot or a previously generated
+report.
+
+Usage:
+...............
+info
+...............
+Example:
+...............
+info
+...............
+
+[[cmdhelp_history_latest,show latest news from the cluster]]
+==== `latest`
+
+The `latest` command shows a bit of recent history, more
+precisely whatever happened since the last cluster change (the
+latest transition). If the transition is running, the shell will
+first wait until it finishes.
+
+Usage:
+...............
+latest
+...............
+Example:
+...............
+latest
+...............
+
+[[cmdhelp_history_limit,limit timeframe to be examined]]
+==== `limit` (`timeframe`)
+
+This command can be used to modify the time span to examine. All
+history commands look at events within a certain time span.
+
+For the `live` source, the default time span is the _last hour_.
+
+There is no time span limit for the `hb_report` source.
+
+The time period is parsed by the `dateutil` python module. It
+covers a wide range of date formats. For instance:
+
+- 3:00 (today at 3am)
+- 15:00 (today at 3pm)
+- 2010/9/1 2pm (September 1st 2010 at 2pm)
+
+For more examples of valid time/date statements, please refer to the
+`python-dateutil` documentation:
+
+- https://dateutil.readthedocs.org/[dateutil.readthedocs.org]
+
+If the dateutil module is not available, then the time is parsed using
+strptime and only the kind as printed by `date(1)` is allowed:
+
+- Tue Sep 15 20:46:27 CEST 2010
+
+Usage:
+...............
+limit [<from_time>] [<to_time>]
+...............
+Examples:
+...............
+limit 10:15
+limit 15h22m 16h
+limit "Sun 5 20:46" "Sun 5 22:00"
+...............
+
+[[cmdhelp_history_log,log content]]
+==== `log`
+
+Show messages logged on one or more nodes. Leaving out a node
+name produces combined logs of all nodes. Messages are sorted by
+time and, if the terminal emulations supports it, displayed in
+different colours depending on the node to allow for easier
+reading.
+
+The sorting key is the timestamp as written by syslog which
+normally has the maximum resolution of one second. Obviously,
+messages generated by events which share the same timestamp may
+not be sorted in the same way as they happened. Such close events
+may actually happen fairly often.
+
+Usage:
+...............
+log [<node> [<node> ...] ]
+...............
+Example:
+...............
+log node-a
+...............
+
+[[cmdhelp_history_node,node events]]
+==== `node`
+
+Show important events that happened on a node. Important events
+are node lost and join, standby and online, and fence. Use either
+node names or extended regular expressions.
+
+Usage:
+...............
+node <node> [<node> ...]
+...............
+Example:
+...............
+node node1
+...............
+
+[[cmdhelp_history_peinputs,list or get PE input files]]
+==== `peinputs`
+
+Every event in the cluster results in generating one or more
+Policy Engine (PE) files. These files describe future motions of
+resources. The files are listed as full paths in the current
+report directory. Add +v+ to also see the creation time stamps.
+
+Usage:
+...............
+peinputs [{<range>|<number>} ...] [v]
+
+range :: <n1>:<n2>
+...............
+Example:
+...............
+peinputs
+peinputs 440:444 446
+peinputs v
+...............
+
+[[cmdhelp_history_refresh,refresh live report]]
+==== `refresh`
+
+This command makes sense only for the +live+ source and makes
+`crm` collect the latest logs and other relevant information from
+the logs. If you want to make a completely new report, specify
++force+.
+
+Usage:
+...............
+refresh [force]
+...............
+
+[[cmdhelp_history_resource,resource events]]
+==== `resource`
+
+Show actions and any failures that happened on all specified
+resources on all nodes. Normally, one gives resource names as
+arguments, but it is also possible to use extended regular
+expressions. Note that neither groups nor clones or master/slave
+names are ever logged. The resource command is going to expand
+all of these appropriately, so that clone instances or resources
+which are part of a group are shown.
+
+Usage:
+...............
+resource <rsc> [<rsc> ...]
+...............
+Example:
+...............
+resource bigdb public_ip
+resource my_.*_db2
+resource ping_clone
+...............
+
+[[cmdhelp_history_session,manage history sessions]]
+==== `session`
+
+Sometimes you may want to get back to examining a particular
+history period or bug report. In order to make that easier, the
+current settings can be saved and later retrieved.
+
+If the current history being examined is coming from a live
+cluster the logs, PE inputs, and other files are saved too,
+because they may disappear from nodes. For the existing reports
+coming from `hb_report`, only the directory location is saved
+(not to waste space).
+
+A history session may also be packed into a tarball which can
+then be sent to support.
+
+Leave out subcommand to see the current session.
+
+Usage:
+...............
+session [{save|load|delete} <name> | pack [<name>] | update | list]
+...............
+Examples:
+...............
+session save bnc966622
+session load rsclost-2
+session list
+...............
+
+[[cmdhelp_history_setnodes,set the list of cluster nodes]]
+==== `setnodes`
+
+In case the host this program runs on is not part of the cluster,
+it is necessary to set the list of nodes.
+
+Usage:
+...............
+setnodes node <node> [<node> ...]
+...............
+Example:
+...............
+setnodes node_a node_b
+...............
+
+[[cmdhelp_history_show,show status or configuration of the PE input file]]
+==== `show`
+
+Every transition is saved as a PE file. Use this command to
+render that PE file either as configuration or status. The
+configuration output is the same as `crm configure show`.
+
+Usage:
+...............
+show <pe> [status]
+
+pe :: <number>|<index>|<file>|live
+...............
+Examples:
+...............
+show 2066
+show pe-input-2080.bz2 status
+...............
+
+[[cmdhelp_history_source,set source to be examined]]
+==== `source`
+
+Events to be examined can come from the current cluster or from a
+`hb_report` report. This command sets the source. `source live`
+sets source to the running cluster and system logs. If no source
+is specified, the current source information is printed.
+
+In case a report source is specified as a file reference, the file
+is going to be unpacked in place where it resides. This directory
+is not removed on exit.
+
+Usage:
+...............
+source [<dir>|<file>|live]
+...............
+Examples:
+...............
+source live
+source /tmp/customer_case_22.tar.bz2
+source /tmp/customer_case_22
+source
+...............
+
+[[cmdhelp_history_transition,show transition]]
+==== `transition`
+
+This command will print actions planned by the PE and run
+graphviz (`dotty`) to display a graphical representation of the
+transition. Of course, for the latter an X11 session is required.
+This command invokes `ptest(8)` in background.
+
+The +showdot+ subcommand runs graphviz (`dotty`) to display a
+graphical representation of the +.dot+ file which has been
+included in the report. Essentially, it shows the calculation
+produced by `pengine` which is installed on the node where the
+report was produced. In optimal case this output should not
+differ from the one produced by the locally installed `pengine`.
+
+The `log` subcommand shows the full log for the duration of the
+transition.
+
+A transition can also be saved to a CIB shadow for further
+analysis or use with `cib` or `configure` commands (use the
+`save` subcommand). The shadow file name defaults to the name of
+the PE input file.
+
+If the PE input file number is not provided, it defaults to the
+last one, i.e. the last transition. The last transition can also
+be referenced with number 0. If the number is negative, then the
+corresponding transition relative to the last one is chosen.
+
+If there are warning and error PE input files or different nodes
+were the DC in the observed timeframe, it may happen that PE
+input file numbers collide. In that case provide some unique part
+of the path to the file.
+
+After the `ptest` output, logs about events that happened during
+the transition are printed.
+
+The `tags` subcommand scans the logs for the transition and return a
+list of key events during that transition. For example, the tag
++error+ will be returned if there are any errors logged during the
+transition.
+
+Usage:
+...............
+transition [<number>|<index>|<file>] [nograph] [v...] [scores] [actions] [utilization]
+transition showdot [<number>|<index>|<file>]
+transition log [<number>|<index>|<file>]
+transition save [<number>|<index>|<file> [name]]
+transition tags [<number>|<index>|<file>]
+...............
+Examples:
+...............
+transition
+transition 444
+transition -1
+transition pe-error-3.bz2
+transition node-a/pengine/pe-input-2.bz2
+transition showdot 444
+transition log
+transition save 0 enigma-22
+...............
+
+[[cmdhelp_history_transitions,List transitions]]
+==== `transitions`
+
+A transition represents a change in cluster configuration or
+state. This command lists the transitions in the current timeframe.
+
+Usage:
+...............
+transitions
+...............
+Example:
+...............
+transitions
+...............
+
+
+[[cmdhelp_history_wdiff,cluster states/transitions difference]]
+==== `wdiff`
+
+A transition represents a change in cluster configuration or
+state. Use `wdiff` to see what has changed between two
+transitions as word differences on a line-by-line basis.
+
+If you want to specify the current cluster configuration and
+status, use the string +live+.
+
+Normally, the first transition specified should be the one which
+is older, but we are not going to enforce that.
+
+Note that a single configuration update may result in more than
+one transition.
+
+Usage:
+...............
+wdiff <pe> <pe> [status]
+
+pe :: <number>|<index>|<file>|live
+...............
+Examples:
+...............
+wdiff 2066 2067
+wdiff pe-input-2080.bz2 live status
+...............
+
+[[cmdhelp_root_report,Create cluster status report]]
+=== `report`
+
+Interface to a tool for creating a cluster report. A report is an
+archive containing log files, configuration files, system information
+and other relevant data for a given time period. This is a useful tool
+for collecting data to attach to bug reports, or for detecting the
+root cause of errors resulting in resource failover, for example.
+
+See `crmsh_hb_report(8)` for more details on arguments,
+or call `crm report -h`
+
+Usage:
+...............
+report -f {time|"cts:"testnum} [-t time] [-u user] [-l file]
+ [-n nodes] [-E files] [-p patt] [-L patt] [-e prog]
+ [-MSDZAVsvhd] [dest]
+...............
+
+Examples:
+...............
+report -f 2pm report_1
+report -f "2007/9/5 12:30" -t "2007/9/5 14:00" report_2
+report -f 1:00 -t 3:00 -l /var/log/cluster/ha-debug report_3
+report -f "09sep07 2:00" -u hbadmin report_4
+report -f 18:00 -p "usern.*" -p "admin.*" report_5
+report -f cts:133 ctstest_133
+...............
+
+=== `end` (`cd`, `up`)
+
+The `end` command ends the current level and the user moves to
+the parent level. This command is available everywhere.
+
+Usage:
+...............
+end
+...............
+
+=== `help`
+
+The `help` command prints help for the current level or for the
+specified topic (command). This command is available everywhere.
+
+Usage:
+...............
+help [<topic>]
+...............
+
+=== `quit` (`exit`, `bye`)
+
+Leave the program.
+
+BUGS
+----
+Even though all sensible configurations (and most of those that
+are not) are going to be supported by the crm shell, I suspect
+that it may still happen that certain XML constructs may confuse
+the tool. When that happens, please file a bug report.
+
+The crm shell will not try to update the objects it does not
+understand. Of course, it is always possible to edit such objects
+in the XML format.
+
+AUTHORS
+-------
+Dejan Muhamedagic, <dejan@suse.de>
+Kristoffer Gronlund <kgronlund@suse.com>
+and many OTHERS
+
+SEE ALSO
+--------
+crm_resource(8), crm_attribute(8), crm_mon(8), cib_shadow(8),
+ptest(8), dotty(1), crm_simulate(8), cibadmin(8)
+
+
+COPYING
+-------
+Copyright \(C) 2008-2013 Dejan Muhamedagic.
+Copyright \(C) 2013 Kristoffer Gronlund.
+
+Free use of this software is granted under the terms of the GNU General Public License (GPL).
+
+//////////////////////
+ vim:ts=4:sw=4:expandtab:
+//////////////////////
diff --git a/doc/website-v1/news.adoc b/doc/website-v1/news.adoc
new file mode 100644
index 0000000..9cd0d6d
--- /dev/null
+++ b/doc/website-v1/news.adoc
@@ -0,0 +1,26 @@
+= News
+
+link:/news/2017-01-31-release-3_0_0[2017-01-31 10:00]
+
+:leveloffset: 1
+
+include::news/2017-01-31-release-3_0_0.adoc[]
+
+:leveloffset: 0
+
+''''
+* link:/news/2016-09-05-release-2_2_2[2016-09-05 19:00 Releasing crmsh version 2.2.2]
+* link:/news/2016-09-02-release-2_3_1[2016-09-02 10:00 Releasing crmsh version 2.3.1]
+* link:/news/2016-09-01-release-2_1_7[2016-09-01 09:00 Announcing crmsh stable release 2.1.7]
+* link:/news/2016-08-12-release-2_3_0[2016-08-12 10:30 Releasing crmsh version 2.3.0]
+* link:/news/2016-04-28-release-2_2_1[2016-04-28 01:00 crmsh 2.2.1 and 2.1.6 are released]
+* link:/news/2016-01-15-release-2_2_0[2016-01-15 15:00 crmsh 2.2.0 is released]
+* link:/news/2016-01-12-release-2_1_5[2016-01-12 10:00 Announcing crmsh stable release 2.1.5]
+* link:/news/2015-05-25-getting-started-jp[2015-05-25 13:30 Getting Started translated to Japanese]
+* link:/news/2015-05-13-release-2_1_4[2015-05-13 15:30 Announcing crmsh stable release 2.1.4]
+* link:/news/2015-04-10-release-2_1_3[2015-04-10 12:30 Announcing crmsh stable release 2.1.3]
+* link:/news/2015-01-26-release-2_1_2[2015-01-26 11:05 Announcing crmsh release 2.1.2]
+* link:/news/2014-10-28-release-2_1_1[2014-10-29 00:20 Announcing crmsh release 2.1.1]
+* link:/news/2014-06-30-release-2_1[2014-06-30 09:00 Announcing crmsh release 2.1]
+
+link:https://savannah.nongnu.org/news/?group_id=10890[Old News Archive]
diff --git a/doc/website-v1/news/2014-06-30-release-2_1.adoc b/doc/website-v1/news/2014-06-30-release-2_1.adoc
new file mode 100644
index 0000000..4c59a90
--- /dev/null
+++ b/doc/website-v1/news/2014-06-30-release-2_1.adoc
@@ -0,0 +1,93 @@
+Announcing crmsh release 2.1
+============================
+:Author: Kristoffer Gronlund
+:Email: kgronlund@suse.com
+:Date: 2014-06-30 09:00
+
+Today we are proud to announce the release of `crmsh` version 2.1!
+This version primarily fixes all known issues found since the release
+of `crmsh` 2.0 in April, but also has some major new features.
+
+A massive thank you to everyone who has helped out with bug fixes,
+comments and contributions for this release!
+
+For a complete list of changes since the previous version, please
+refer to the changelog:
+
+* https://github.com/crmsh/crmsh/blob/2.1.0/ChangeLog
+
+Packages for several popular Linux distributions can be downloaded
+from the Stable repository at the OBS:
+
+* http://download.opensuse.org/repositories/network:/ha-clustering:/Stable/
+
+Archives of the tagged release:
+
+* https://github.com/crmsh/crmsh/archive/2.1.0.tar.gz
+* https://github.com/crmsh/crmsh/archive/2.1.0.zip
+
+Here are some of the highlights of this release:
+
+== Rule expressions in attribute lists
+
+One of the biggest features in this release is full support for rule
+expressions wherever the XML syntax allows them.
+
+Here is an example of using rule expressions in an attribute list in
+order to set the virtual IP of an IPAddr2 resource to a different
+value on a specific node.
+
+----
+primitive vip-on-node1 IPAddr2 \
+ rule 10: #uname eq node1 ip=10.0.0.5 \
+ rule 1: ip=10.0.0.6
+----
+
+== Tags in the CIB
+
+A new feature added to Pacemaker recently is tags. This is a way
+to refer to multiple resources at once without creating any
+colocation or ordering relationship between them. For example, you
+could add all resources related to the database to a db tag, and
+then stop or start them all with a single command.
+
+----
+tag db drbd:Master fs sql-db
+----
+
+It is also possible to refer to tags in constraints.
+
+== Wildcards in show/edit
+
+The configure show and edit commands can now use glob-style
+wildcards to refer to multiple resources:
+
+----
+configure edit db-*
+----
+
+== Nvpair references
+
+Sometimes, different resources name the same parameters with different
+names. For example, an IPAddr2 may have an ip parameter that should be
+the same as a web servers server_ip parameter. By using nvpair
+references, it is possible to configure the ip in a single location.
+
+Note that this is a new feature in Pacemaker 1.1.12 and up.
+
+----
+primitive vip IPAddr2 params $my-ip:ip=192.168.0.1
+primitive www apache params @my-ip:server_ip
+----
+
+== New ACL syntax
+
+The support for Access Control Lists has been revised in Pacemaker
+1.1.12, and this release of crmsh supports the new syntax. Two new
+commands have been added: `acl_target` and `acl_group`. For more details,
+see the documentation.
+
+Thank you,
+
+Kristoffer and Dejan
+
diff --git a/doc/website-v1/news/2014-10-28-release-2_1_1.adoc b/doc/website-v1/news/2014-10-28-release-2_1_1.adoc
new file mode 100644
index 0000000..6b67f4f
--- /dev/null
+++ b/doc/website-v1/news/2014-10-28-release-2_1_1.adoc
@@ -0,0 +1,58 @@
+Announcing crmsh release 2.1.1
+==============================
+:Author: Kristoffer Gronlund
+:Email: kgronlund@suse.com
+:Date: 2014-10-29 00:20
+
+Today we are proud to announce the release of `crmsh` version 2.1.1!
+This version primarily fixes all known issues found since the release
+of `crmsh` 2.1 in June. We recommend that all users of crmsh upgrade
+to this version, especially if using Pacemaker 1.1.12 or newer.
+
+A massive thank you to everyone who has helped out with bug fixes,
+comments and contributions for this release!
+
+For a complete list of changes since the previous version, please
+refer to the changelog:
+
+* https://github.com/crmsh/crmsh/blob/2.1.1/ChangeLog
+
+Packages for several popular Linux distributions can be downloaded
+from the Stable repository at the OBS:
+
+* http://download.opensuse.org/repositories/network:/ha-clustering:/Stable/
+
+Archives of the tagged release:
+
+* https://github.com/crmsh/crmsh/archive/2.1.1.tar.gz
+* https://github.com/crmsh/crmsh/archive/2.1.1.zip
+
+Changes since the previous release:
+
+ - cibconfig: Clean up output from crm_verify (bnc#893138)
+ - high: constants: Add acl_target and acl_group to cib_cli_map (bnc#894041)
+ - high: parse: split shortcuts into valid rules
+ - medium: Handle broken CIB in find_objects
+ - high: scripts: Handle corosync.conf without nodelist in add-node (bnc#862577)
+ - medium: config: Assign default path in all cases
+ - high: cibconfig: Generate valid CLI syntax for attribute lists (bnc#897462)
+ - high: cibconfig: Add tag:<tag> to get all resources in tag
+ - doc: Documentation for show tag:<tag>
+ - low: report: Sort list of nodes
+ - high: parse: Allow empty attribute values in nvpairs (bnc#898625)
+ - high: cibconfig: Delay reinitialization after commit
+ - low: cibconfig: Improve wording of commit prompt
+ - low: cibconfig: Fix vim modeline
+ - high: report: Find nodes for any log type (boo#900654)
+ - high: hb_report: Collect logs from journald (boo#900654)
+ - high: cibconfig: Don't crash if given an invalid pattern (bnc#901714)
+ - high: xmlutil: Filter list of referenced resources (bnc#901714)
+ - medium: ui_resource: Only act on resources (#64)
+ - medium: ui_resource: Flatten, then filter (#64)
+ - high: ui_resource: Use correct name for error function (bnc#901453)
+ - high: ui_resource: resource trace failed if operation existed (bnc#901453)
+ - Improved test suite
+
+Thank you,
+
+Kristoffer and Dejan
diff --git a/doc/website-v1/news/2015-01-26-release-2_1_2.adoc b/doc/website-v1/news/2015-01-26-release-2_1_2.adoc
new file mode 100644
index 0000000..081bf1b
--- /dev/null
+++ b/doc/website-v1/news/2015-01-26-release-2_1_2.adoc
@@ -0,0 +1,69 @@
+Announcing crmsh release 2.1.2
+==============================
+:Author: Kristoffer Gronlund
+:Email: kgronlund@suse.com
+:Date: 2015-01-26 11:05
+
+Today we are proud to announce the release of `crmsh` version 2.1.2!
+This version primarily fixes all known issues found since the release
+of `crmsh` 2.1.1 in October. We recommend that all users of crmsh upgrade
+to this version, especially if using Pacemaker 1.1.12 or newer.
+
+A massive thank you to everyone who has helped out with bug fixes,
+comments and contributions for this release!
+
+For a complete list of changes since the previous version, please
+refer to the changelog:
+
+* https://github.com/crmsh/crmsh/blob/2.1.2/ChangeLog
+
+Packages for several popular Linux distributions can be downloaded
+from the Stable repository at the OBS:
+
+* http://download.opensuse.org/repositories/network:/ha-clustering:/Stable/
+
+Archives of the tagged release:
+
+* https://github.com/crmsh/crmsh/archive/2.1.2.tar.gz
+* https://github.com/crmsh/crmsh/archive/2.1.2.zip
+
+Changes since the previous release:
+
+ - medium: ui_resource: Set probe interval 0 if not set (bnc#905050)
+ - doc: Document probe op in resource trace (bnc#905050)
+ - high: config: Fix path to system-wide crm.conf (#67)
+ - medium: config: Fall back to /etc/crm/crmsh.conf (#67)
+ - low: cliformat: Colorize id: as identifier (boo#905338)
+ - medium: cibconfig: Don't bump epoch if stripping version
+ - medium: ui_context: Lazily import readline
+ - medium: config: Add core.ignore_missing_metadata (#68) (boo#905910)
+ - medium: cibconfig: Strip digest from v1 diffs (bnc#914098)
+ - medium: cibconfig: Detect v1 format and don't patch container changes (bnc#914098)
+ - high: xmlutil: Treat node type=member as normal (boo#904698)
+ - medium: xmlutil: Use idmgmt when creating new elements (bnc#901543)
+ - low: ui_resource: --reprobe and --refresh are deprecated (bnc#905092)
+ - doc: Document deprecation of refresh and reprobe (bnc#905092)
+ - medium: parse: Support resource-discovery in location constraints
+ - medium: Allow removing groups even if is_running (boo#905271)
+ - medium: cibconfig: Delete containers first in edits (boo#905268)
+ - medium: ui_history: Fix crash using empty object set
+ - Low: term: get rid of annying ^O in piped-to-less-R output
+ - medium: parse: Allow nvpair with no value using name= syntax (#71)
+ - medium: parse: Enable name[=value] for nvpair (#71)
+ - medium: utils: Check if path basename is less (#74)
+ - medium: utils: crm_daemon_dir is added to PATH in envsetup (#67)
+ - medium: cmd_status: Show pending if available, enable extra options
+ - high: utils: Locate binaries across sudo boundary (bnc#912483)
+ - Medium: history: match error/crit messages of pcmk 1.1.12
+ - low: ui_options: Add underscore aliases for legacy options
+ - medium: constants: Fix transition start detection
+ - medium: constants: Update transition regex (#77)
+ - medium: orderedset: Add OrderedSet type
+ - medium: cibconfig: Use orderedset to avoid reordering bugs (#79)
+ - low: xmlutil: logic bug in sanity_check_nvpairs
+ - medium: util: Don't fall back to current time
+ - medium: report: Fall back to end_ts = start_ts
+
+Thank you,
+
+Kristoffer and Dejan
diff --git a/doc/website-v1/news/2015-04-10-release-2_1_3.adoc b/doc/website-v1/news/2015-04-10-release-2_1_3.adoc
new file mode 100644
index 0000000..c186ff0
--- /dev/null
+++ b/doc/website-v1/news/2015-04-10-release-2_1_3.adoc
@@ -0,0 +1,68 @@
+Announcing crmsh stable release 2.1.3
+=====================================
+:Author: Kristoffer Gronlund
+:Email: kgronlund@suse.com
+:Date: 2015-04-10 12:30
+
+Today we are proud to announce the release of `crmsh` version 2.1.3!
+This version fixes all known issues found since the release of `crmsh`
+2.1.2 in January. We recommend that all users of crmsh upgrade
+to this version, especially if using Pacemaker 1.1.12 or newer.
+
+A massive thank you to everyone who has helped out with bug fixes,
+comments and contributions for this release!
+
+For a complete list of changes since the previous version, please
+refer to the changelog:
+
+* https://github.com/ClusterLabs/crmsh/blob/2.1.3/ChangeLog
+
+Packages for several popular Linux distributions can be downloaded
+from the Stable repository at the OBS:
+
+* http://download.opensuse.org/repositories/network:/ha-clustering:/Stable/
+
+Archives of the tagged release:
+
+* https://github.com/ClusterLabs/crmsh/archive/2.1.3.tar.gz
+* https://github.com/ClusterLabs/crmsh/archive/2.1.3.zip
+
+Changes since the previous release:
+
+ - medium: parse: nvpair attributes with no value = <nvpair name=".."/> (#71)
+ - doc: Add link to clusterlabs.org
+ - medium: report: Convert RE exception to simpler UI output
+ - medium: report: Include transitions with configuration changes (bnc#917131)
+ - medium: config: Fix case-sensitivity for booleans
+ - medium: ra: Handle non-OCF agent meta-data better
+ - Medium: cibconf: preserve cib user attributes
+ - low: cibconfig: Improved debug output when schema change fails
+ - medium: parse: Treat pacemaker-next schema as 2.0+
+ - medium: schema: Test if node type is optional via schema
+ - medium: schema: Remove extra debug output
+ - low: pacemaker: Remove debug output
+ - medium: cibconfig: If a change results in no diff, exit silently
+ - medium: cibconfig: Allow delete of objects that don't exist without returning error code
+ - medium: cibconfig: Allow removal of non-existing elements if --force is set
+ - low: allow (0,1) as option booleans
+ - low: allow pacemaker 1.0 version detection
+ - Low: hb_report: add -Q to usage
+ - Low: hb_report: add -X option for extra ssh options
+ - doc: Move the main crmsh repository to the ClusterLabs organization on github
+ - high: ui_configure: Remove acl_group command (bnc#921056)
+ - high: cibconfig: Don't delete valid tickets when removing referenced objects (bnc#922039)
+ - high: ui_context: Wait for DC after commit, not before (#85)
+ - medium: templates: Clearer descriptions for editing templates (boo#921028)
+ - high: cibconfig: Derive id for ops from referenced resource name (boo#921028)
+ - medium: ui_template: Always generate id unless explicitly defined (boo#921028)
+ - low: template: Add 'new <template>' shortcut
+ - medium: ui_template: Make new command more robust (bnc#924641)
+ - medium: parse: Disallow location rules without resources
+ - high: parse: Don't allow constraints without applicants
+ - medium: cliformat: Escape double-quotes in nvpair values
+ - low: hb_report: Use crmsh config to find pengine/cib dirs (bsc#926377)
+ - low: main: Catch any ValueErrors that may leak through
+
+Thank you,
+
+Kristoffer and Dejan
diff --git a/doc/website-v1/news/2015-05-13-release-2_1_4.adoc b/doc/website-v1/news/2015-05-13-release-2_1_4.adoc
new file mode 100644
index 0000000..31297cf
--- /dev/null
+++ b/doc/website-v1/news/2015-05-13-release-2_1_4.adoc
@@ -0,0 +1,126 @@
+Announcing crmsh stable release 2.1.4
+=====================================
+:Author: Kristoffer Gronlund
+:Email: kgronlund@suse.com
+:Date: 2015-05-13 15:30
+
+Today we are proud to announce the release of `crmsh` version 2.1.4!
+2.1.4 is a minor bug fix release with no major issues, so users
+already running 2.1.3 are mostly fine. Instead, the main reason
+for releasing 2.1.4 is as an excuse to talk about some other things
+that are happening with crmsh!
+
+The details for this release are available below.
+
+History Guide
+~~~~~~~~~~~~~
+
+Dejan has written a guide to using the crmsh history
+command. For those who are unfamiliar with the history explorer or
+want to know more about how to use it, this guide is a great
+introduction to what it does and how to use it.
+
+History is not a new crmsh feature, but, as we failed to
+advertise it and nothing works without proper marketing, it
+probably hasn't seen a very wide use. That's surely a pity and we
+hope that this gentle history guide is going to help.
+
+So, if you use crmsh and if you need help troubleshooting
+clusters (I surely do!), take a look here:
+
+http://crmsh.github.io/history-guide/
+
+FYI, the comprehensive crmsh help also has a short description of
+the feature:
+
+........
+crm history help
+........
+
+Goes without saying: all commands are described too.
+
+If you don't use crmsh, you'll still find a lot of useful
+information in the guide, so don't skip it.
+
+Hawk Presentation
+~~~~~~~~~~~~~~~~~
+
+I presented Hawk [1] and the History Explorer interface which
+builds upon the crmsh history feature at openSUSE conf in The Hague
+earlier this month. The video of that presentation is online here:
+
+++++++++++++
+<iframe width="420" height="315" src="https://www.youtube.com/embed/mngfxzXkFLw" frameborder="0" allowfullscreen></iframe>
+++++++++++++
+
+https://www.youtube.com/watch?v=mngfxzXkFLw
+
+[1]: https://github.com/ClusterLabs/hawk
+
+
+2.2.0 Development News
+~~~~~~~~~~~~~~~~~~~~~~~
+
+While 2.1.4 is the latest stable release, I am also working on releasing
+2.2.0 which will come with a bunch of new features. I'm still working
+on some of these and not everything is in the repository yet, so
+2.2.0 is probably at least a month or so away still. I was perhaps
+a bit optimistic when I tagged RC1 back in October last year. ;)
+
+However, right now I'd like to focus on one thing that is already in
+2.2.0 and which is available if you use the development packages from
+OBS: command shorthands. This makes crmsh a lot more convenient to use
+from the command line. Basically, you can use any unambiguous subset
+of a command name to refer to that command, and crmsh will figure out
+what you mean. This may sound confusing, so an example will help with
+explaining what I mean:
+
+This is one way of showing the current cluster configuration:
+
+........
+crm configure show
+........
+
+However, now you can shorten this to the following:
+
+........
+crm cfg show
+........
+
+Other examples of shorthand are `crm rsc stop r1` or `crm st`
+for status. And of course, tab completion in bash still works for
+the shorthand variants.
+
+The examples used here are not comprehensive. crmsh is pretty clever
+at figuring out which command was intended. Download the development
+release and try it out!
+
+2.1.4 Details
+~~~~~~~~~~~~~
+
+For a complete list of changes since the previous version, please
+refer to the changelog:
+
+* https://github.com/ClusterLabs/crmsh/blob/2.1.4/ChangeLog
+
+Packages for several popular Linux distributions can be downloaded
+from the Stable repository at the OBS:
+
+* http://download.opensuse.org/repositories/network:/ha-clustering:/Stable/
+
+Archives of the tagged release:
+
+* https://github.com/ClusterLabs/crmsh/archive/2.1.4.tar.gz
+* https://github.com/ClusterLabs/crmsh/archive/2.1.4.zip
+
+Changes since the previous release:
+
+- Medium: hb_report: use faster zypper interface if available
+- medium: ui_configure: Wait for DC when removing running resource
+- low: schema: Don't leak PacemakerError exceptions (#93)
+- parse: Don't require trailing colon in tag definitions
+- medium: utils: Allow 1/0 as boolean values for parameters
+
+Thank you,
+
+Kristoffer and Dejan
diff --git a/doc/website-v1/news/2015-05-25-getting-started-jp.adoc b/doc/website-v1/news/2015-05-25-getting-started-jp.adoc
new file mode 100644
index 0000000..c5c6759
--- /dev/null
+++ b/doc/website-v1/news/2015-05-25-getting-started-jp.adoc
@@ -0,0 +1,17 @@
+Getting Started translated to Japanese
+======================================
+:Author: Kristoffer Gronlund
+:Email: kgronlund@suse.com
+:Date: 2015-05-25 13:30
+
+Many thanks to Motoharu Kubo at 3ware for offering to translate the
+`crmsh` documentation to Japanese!
+
+The first document to be translated is the link:/start-guide/[Getting Started] guide,
+now available in Japanese at the following location:
+
+* https://blog.3ware.co.jp/2015/05/crmsh-getting-started/
+
+Thank you,
+Kristoffer and Dejan
+
diff --git a/doc/website-v1/news/2016-01-12-release-2_1_5.adoc b/doc/website-v1/news/2016-01-12-release-2_1_5.adoc
new file mode 100644
index 0000000..93a3242
--- /dev/null
+++ b/doc/website-v1/news/2016-01-12-release-2_1_5.adoc
@@ -0,0 +1,56 @@
+Announcing crmsh stable release 2.1.5
+=====================================
+:Author: Kristoffer Gronlund
+:Email: kgronlund@suse.com
+:Date: 2016-01-12 10:00
+
+Today we are proud to announce the release of `crmsh` version 2.1.5!
+This release mainly consists of bug fixes, as well as compatibility
+with Pacemaker 1.1.14.
+
+For a complete list of changes since the previous version, please
+refer to the changelog:
+
+* https://github.com/ClusterLabs/crmsh/blob/2.1.5/ChangeLog
+
+Packages for several popular Linux distributions can be downloaded
+from the Stable repository at the OBS:
+
+* http://download.opensuse.org/repositories/network:/ha-clustering:/Stable/
+
+Archives of the tagged release:
+
+* https://github.com/ClusterLabs/crmsh/archive/2.1.5.tar.gz
+* https://github.com/ClusterLabs/crmsh/archive/2.1.5.zip
+
+Changes since the previous release:
+
+- medium: report: Try to load source as session if possible (bsc#927407)
+- medium: crm_gv: Wrap non-identifier names in quotes (bsc#931837)
+- medium: crm_gv: Improved quoting of non-identifier node names (bsc#931837)
+- medium: crm_pkg: Fix cluster init bug on RH-based systems
+- medium: hb_report: Collect logs from pacemaker.log
+- medium: constants: Add 'provides' meta attribute (bsc#936587)
+- high: parse: Add attributes to terminator set (bsc#940920)
+- Medium: cibconfig: skip sanity check for properties other than cib-bootstrap-options
+- medium: config: Add report_tool_options (bsc#917638)
+- low: main: Bash completion didn't handle sudo correctly
+- high: report: New detection to fix missing transitions (bnc#917131)
+- medium: report: Add pacemaker.log to find_node_log list (bsc#941734)
+- high: hb_report: Prefer pacemaker.log if it exists (bsc#941681)
+- high: report: Output format from pacemaker has changed (bsc#941681)
+- high: report: Update transition edge regexes (bsc#942906)
+- medium: report: Reintroduce empty transition pruning (bsc#943291)
+- medium: log_patterns: Remove reference to function name in log patterns (bsc#942906)
+- low: hb_report: Collect libqb version (bsc#943327)
+- high: parse: Fix crash when referencing score types by name (bsc#940194)
+- low: constants: Add meta attributes for remote nodes
+- low: ui_history: Swap from and to times if to < from
+- high: cibconfig: Do not fail on unknown pacemaker schemas (bsc#946893)
+- high: log_patterns_118: Update the correct set of log patterns (bsc#942906)
+- high: xmlutil: Order is significant in resource_set (bsc#955434)
+- high: cibconfig: Fix XML import bug for cloned groups (bsc#959895)
+
+Thank you,
+
+Kristoffer and Dejan
diff --git a/doc/website-v1/news/2016-01-15-release-2_2_0.adoc b/doc/website-v1/news/2016-01-15-release-2_2_0.adoc
new file mode 100644
index 0000000..664526e
--- /dev/null
+++ b/doc/website-v1/news/2016-01-15-release-2_2_0.adoc
@@ -0,0 +1,210 @@
+crmsh 2.2.0 is released
+=======================
+:Author: Kristoffer Gronlund
+:Email: kgronlund@suse.com
+:Date: 2016-01-15 15:00
+
+In June of last year, I released Release Candidate 3 of crmsh 2.2.0,
+and I honestly expected to have the final version ready no more than a
+few weeks later. Well, it took around 6 months, but now it is finally
+here!
+
+The source code can be downloaded from Github:
+
+* https://github.com/ClusterLabs/crmsh/releases/tag/2.2.0
+
+Packages for several popular Linux distributions can be downloaded
+from the Stable repository at the OBS:
+
+* http://download.opensuse.org/repositories/network:/ha-clustering:/Stable/
+
+Archives of the tagged release:
+
+* https://github.com/ClusterLabs/crmsh/archive/2.2.0.tar.gz
+* https://github.com/ClusterLabs/crmsh/archive/2.2.0.zip
+
+There are many new features and quite a few bug fixes, but I'll try to
+describe some of the major changes below. For the full list of changes
+since version 2.1, see the ChangeLog. I am also including the list
+of changes since RC3 in the release notes below.
+
+* https://github.com/ClusterLabs/crmsh/blob/2.2.0/ChangeLog
+
+New features introduced in this version:
+
+* Support for many of the new features introduced in Pacemaker
+ 1.1.14, including improved support for Pacemaker Remote and
+ pattern-based fencing topology configurations.
+
+* The cluster scripts have been greatly improved and are now used as
+ the wizards backend for hawk. Together with the improvements in the
+ cluster scripts, the crmsh template sublevel has been deprecated,
+ and all of the templates are now available as cluster scripts. The
+ template sublevel will be removed in a future version.
+
+* The history explorer now replaces the combined log and event files
+ from hb_report using the commands crm history log and crm history
+ events. Additionally, the info summary now highlights transitions
+ that contain error logs or other potentially interesting events.
+
+* Improved verification before committing new changes, for example
+ crmsh now warns if colocation constraints are applied to group
+ children.
+
+* resource start / stop / restart now take multiple resource arguments.
+
+* Added a maintenance sublevel with commands intended to simplify the
+ use of maintenance mode.
+
+* Switched to using the python parallax library instead of pssh for
+ remote node communication. Parallax is a fork of pssh with
+ additional features added specifically for using it as a python
+ library rather than a command line tool. Packages for
+ python-parallax can be downloaded from the OBS, or it can be
+ installed from PyPI.
+
+Full list of changes since 2.2.0-rc3:
+
+- medium: history: Fix live report refresh (bsc#950422) (bsc#927414)
+- medium: history: Ignore central log
+- medium: cibconfig: Detect false container children
+- low: clidisplay: Avoid crash when colorizing None
+- medium: scripts: Load single file yml scripts
+- medium: scripts: Reformat scripts to simplified form
+- medium: ui_history: Add events command (bsc#952449)
+- low: hb_report: Drop function from event patterns
+- high: cibconfig: Preserve failure through edit (bsc#959965)
+- high: cibconfig: fail if new object already exists (bsc#959965)
+- medium: ui_cib: Call crm_shadow in batch mode to avoid spawning subshell (bsc#961392)
+- high: cibconfig: Fix XML import bug for cloned groups (bsc#959895)
+- high: ui_configure: Move validate-all validation to a separate command (bsc#956442)
+- high: scripts: Don't require scripts to be an array of one element
+- medium: scripts: Enable setting category in legacy wizards (bnc#957926)
+- high: scripts: Don't delete steps from upgraded wizards (bnc#957925)
+- high: ra: Only run validate-all if current user is root
+- high: cibconfig: Call validate-all action on agent in verify (bsc#956442)
+- high: script: Fix issues found in cluster scripts
+- high: ui_ra: Add ra validate command (bsc#956442)
+- low: resource: Fix unban alias for unmigrate
+- high: ui_resource: Add constraints and operations commands
+- high: ui_resource: Enable start/stop/status for multiple resources at once (bsc#952775)
+- high: scripts: Conservatively verify scripts that modify the CIB (bsc#951954)
+- high: xmlutil: Order is significant in resource_set (bsc#955434)
+- medium: scripts: Lower copy target to string
+- doc: configure load can read from stdin
+- medium: script: (filesystem) create stopped (bsc#952670)
+- medium: scripts: Check required parameters for optional sub-steps
+- high: scripts: Eval CIB text in correct scope (bsc#952600)
+- medium: utils: Fix python 2.6 compatibility
+- medium: ui_script: Tag legacy wizards as legacy in show (bsc#952226)
+- medium: scripts: No optional steps in legacy wizards (bsc#952226)
+- high: utils: Revised time zone handling (bsc#951759)
+- high: report: Fix syslog parser regexps (bsc#951759)
+- low: constants: Tweaked graph colours
+- high: scripts: Fix DRBD script resource reference (bsc#951028)
+- low: constants: Tweaked graph colors
+- medium: report: Make transitions without end stretch to 2525
+- high: utils: Handle time zones in parse_time (bsc#949511)
+- medium: hb_report: Remove reference to function name in event patterns (bsc#942906)
+- medium: ui_script: Optionally print common params
+- medium: cibconfig: Fix sanity check for attribute-based fencing topology (#110)
+- high: cibconfig: Fix bug with node/resource collision
+- high: scripts: Determine output format of script correctly (bsc#949980)
+- doc: add explanatory comments to fencing_topology
+- doc: add missing backslash in fencing_topology example
+- doc: add missing <> to fencing_topology syntax
+- low: don't use deprecated crm_attribute -U option
+- doc: resource-discovery for location constraints
+- high: utils: Fix cluster_copy_file error when nodes provided
+- low: xmlutil: More informative message when updating resource references after rename
+- doc: fix some command syntax grammar in the man page
+- high: cibconfig: Delete constraints before resources
+- high: cibconfig: Fix bug in is_edit_valid (bsc#948547)
+- medium: hb_report: Don't cat binary logs
+- high: cibconfig: Allow node/rsc id collision in _set_update (bsc#948547)
+- low: report: Silence tar warning on early stream close
+- high: cibconfig: Allow nodes and resources with the same ID (bsc#948547)
+- high: log_patterns_118: Update the correct set of log patterns (bsc#942906)
+- low: ui_resource: Silence spurious migration non-warning from pacemaker
+- medium: config: Always fall back to /usr/bin:/usr/sbin:/bin:/sbin for programs (bsc#947818)
+- medium: report: Enable opening .xz-compressed report tarballs
+- medium: cibconfig: Only warn for grouped children in colocations (bsc#927423)
+- medium: cibconfig: Allow order constraints on group children (bsc#927423)
+- medium: cibconfig: Warn if configuring constraint on child resource (bsc#927423) (#101)
+- high: ui_node: Show remote nodes in crm node list (bsc#877962)
+- high: config: Remove config.core.supported_schemas (bsc#946893)
+- medium: report: Mark transitions with errors with a star in info output (bsc#943470)
+- low: report: Remove first transition tag regex
+- medium: report: Add transition tags command (bsc#943470)
+- low: ui_history: Better error handling and documentation for the detail command
+- low: ui_history: Swap from and to times if to < from
+- medium: cibconfig: XML parser support for node-attr fencing topology
+- medium: parse: Updated syntax for fencing-topology target attribute
+- medium: parse: Add support for node attribute as fencing topology target
+- high: scripts: Add enum type to script values
+- low: scripts: [MailTo] install mailx package
+- low: scripts: Fix typo in email type verifier
+- high: script: Fix subscript agent reference bug
+- low: constants: Add meta attributes for remote nodes
+- medium: scripts: Fix typo in lvm script
+- high: scripts: Generate actions for includes if none are defined
+- low: scripts: [virtual-ip] make lvs_support an advanced parameter
+- medium: crm_pssh: Timeout is an int (bsc#943820)
+- medium: scripts: Add MailTo script
+- low: scripts: Improved script parameter validation
+- high: parse: Fix crash when referencing score types by name (bsc#940194)
+- doc: Clarify documentation for colocations using node-attribute
+- high: ui_script: Print cached errors in json run
+- medium: scripts: Use --no option over --force unless force: true is set in the script
+- medium: options: Add --no option
+- high: scripts: Default to passing --force to crm after all
+- high: scripts: Add force parameter to cib and crm actions, and don't pass --force by default
+- low: scripts: Make virtual IP optional [nfsserver]
+- medium: scripts: Ensure that the Filesystem resource exists [nfsserver] (bsc#898658)
+- medium: report: Reintroduce empty transition pruning (bsc#943291)
+- low: hb_report: Collect libqb version (bsc#943327)
+- medium: log_patterns: Remove reference to function name in log patterns (bsc#942906)
+- low: hb_report: Increase time to wait for the logmark
+- high: hb_report: Always prefer syslog if available (bsc#942906)
+- high: report: Update transition edge regexes (bsc#942906)
+- medium: scripts: Switch install default to false
+- low: scripts: Catch attempt to pass dict as parameter value
+- high: report: Output format from pacemaker has changed (bsc#941681)
+- high: hb_report: Prefer pacemaker.log if it exists (bsc#941681)
+- medium: report: Add pacemaker.log to find_node_log list (bsc#941734)
+- high: hb_report: Correct path to hb_report after move to subdirectory (bsc#936026)
+- low: main: Bash completion didn't handle sudo correctly
+- medium: config: Add report_tool_options (bsc#917638)
+- high: parse: Add attributes to terminator set (bsc#940920)
+- Medium: cibconfig: skip sanity check for properties other than cib-bootstrap-options
+- medium: ui_script: Fix bug in verify json encoding
+- low: ui_script: Check JSON command syntax
+- medium: ui_script: Add name to action output (fate#318211)
+- low: scripts: Preserve formatting of longdescs
+- low: scripts: Clearer shortdesc for filesystem
+- low: scripts: Fix formatting for SAP scripts
+- low: scripts: add missing type annotations to libvirt script
+- low: scripts: make overridden parameters non-advanced by default
+- low: scripts: Tweak description for libvirt
+- low: scripts: Strip shortdesc for scripts and params
+- low: scripts: Title and category for exportfs
+- high: ui_script: drop end sentinel from API output (fate#318211)
+- low: scripts: Fix possible reference error in agent include
+- low: scripts: Clearer error message
+- low: Remove build revision from version
+- low: Add HAProxy script to data manifest
+- medium: constants: Add 'provides' meta attribute (bsc#936587)
+- medium: scripts: Add HAProxy script
+- high: hb_report: find utility scripts after move (bsc#936026)
+- high: ui_report: Move hb_report to subdirectory (bsc#936026)
+- high: Makefile: Don't unstall hb_report using data-manifest (bsc#936026)
+- medium: report: Fall back to cluster-glue hb_report if necessary (bsc#936026)
+- medium: scripts: stop inserting comments as values
+- high: scripts: subscript values not required if subscript has no parameters / all defaults (fate#318211)
+- medium: scripts: Fix name override for subscripts (fate#318211)
+- low: scripts: Clean up generated CIB (fate#318211)
+
+As usual, a huge thank you to all contributors and users of crmsh!
+
+Cheers,
+Kristoffer
diff --git a/doc/website-v1/news/2016-04-28-release-2_2_1.adoc b/doc/website-v1/news/2016-04-28-release-2_2_1.adoc
new file mode 100644
index 0000000..e162b2f
--- /dev/null
+++ b/doc/website-v1/news/2016-04-28-release-2_2_1.adoc
@@ -0,0 +1,73 @@
+crmsh 2.2.1 and 2.1.6 are released
+==================================
+:Author: Kristoffer Gronlund
+:Email: kgronlund@suse.com
+:Date: 2016-04-28 01:00
+
+Today I'm announcing two new releases of crmsh: 2.2.1 and 2.1.6.
+
+Both are stable releases, and I would recommend all users to
+upgrade to crmsh 2.2.1 if they can.
+
+== crmsh 2.2.1
+
+Here is a brief list of the most significant changes in this release:
+
+* Rewritten history explorer internally to improve performance significantly
+* Allow configuring reload operation
+* Fix fencing for remote nodes
+* Recycle corosync node IDs when possible
+* Several bug fixes in crm report
+* Warn if generated report is empty
+* Improved SBD cluster script
+* Add push method for configure load
+* Fixed cluster init
+
+The source code can be downloaded from Github:
+
+* https://github.com/ClusterLabs/crmsh/releases/tag/2.2.1
+
+Packages for several popular Linux distributions can be downloaded
+from the Stable repository at the OBS:
+
+* http://download.opensuse.org/repositories/network:/ha-clustering:/Stable/
+
+Archives of the tagged release:
+
+* https://github.com/ClusterLabs/crmsh/archive/2.2.1.tar.gz
+* https://github.com/ClusterLabs/crmsh/archive/2.2.1.zip
+
+For the full list of changes since version 2.2.0, see the ChangeLog,
+available at:
+
+* https://github.com/ClusterLabs/crmsh/blob/2.2.1/ChangeLog
+
+== crmsh 2.1.6
+
+This is a bug fix release.
+Most fixes are minor or related to time handling in reports.
+
+For a complete list of changes since the previous version, please
+refer to the changelog:
+
+* https://github.com/ClusterLabs/crmsh/blob/2.1.6/ChangeLog
+
+Currently I'm not building binary releases for 2.1.6 as the stable
+series (at OBS) is at 2.2. This release is intended for users of
+the 2.1 series who have yet to migrate to 2.2.
+
+Archives of the tagged release:
+
+https://github.com/ClusterLabs/crmsh/archive/2.1.6.tar.gz
+https://github.com/ClusterLabs/crmsh/archive/2.1.6.zip
+
+For the full list of changes since version 2.1.5, see the ChangeLog,
+available at:
+
+* https://github.com/ClusterLabs/crmsh/blob/2.1.5/ChangeLog
+
+
+As usual, a huge thank you to all contributors and users of crmsh!
+
+Cheers,
+Kristoffer
diff --git a/doc/website-v1/news/2016-08-12-release-2_3_0.adoc b/doc/website-v1/news/2016-08-12-release-2_3_0.adoc
new file mode 100644
index 0000000..bb3efa9
--- /dev/null
+++ b/doc/website-v1/news/2016-08-12-release-2_3_0.adoc
@@ -0,0 +1,76 @@
+Releasing crmsh version 2.3.0
+=============================
+:Author: Kristoffer Gronlund
+:Email: kgronlund@suse.com
+:Date: 2016-08-12 10:30
+
+Hello everyone!
+
+I am proud to present crmsh version 2.3.0, the latest stable
+release. I would recommend all users to upgrade to 2.3.0 if they
+can.
+
+For this release, I would like to begin by highlighting the new
+contributors to crmsh since 2.2.0 was released in January:
+
+* Marc A. Smith added the new subcommand "configure load push", which
+ removes any configuration lines that aren't included in the cib
+ provided when pushing.
+
+* Andrei Maruha added an optional name parameter to the "corosync
+ add-node" command, and made the add-node command recycle old node
+ IDs if possible.
+
+* Kai Kang fixed a build system bug when removing generated docs,
+ causing issues with parallel make.
+
+* Daniel Hoffend contributed various fixes improving support for
+ building crmsh for Debian and Ubuntu.
+
+* Pedro Salgado fixed a bug in the graph rendering code in crmsh,
+ added a tox configuration file to make testing with multiple
+ versions of Python easy, and updated the Travis CI configuration to
+ use tox.
+
+* Nate Clark fixed a bug in the parser for fencing hierarchies.
+
+I would also like to thank all the other contributors, testers and
+users who have helped in making this release as stable and reliable as
+possible.
+
+Some of the other major features in 2.3.0 include:
+
+* Support for the new event-based alerts feature in Pacemaker 1.1.15
+
+* Greatly improved timezone handling in crm report and the history
+ explorer
+
+* Improvements to the cluster scripts / wizards, as well as new
+ wizards for LVM on DRBD, and NFS on LVM and DRBD and VMware/vCenter
+
+* Better support for fencing remote nodes
+
+The source code can be downloaded from Github:
+
+* https://github.com/ClusterLabs/crmsh/releases/tag/2.3.0
+
+Packages for several popular Linux distributions can be downloaded
+from the Stable repository at the OBS:
+
+* http://download.opensuse.org/repositories/network:/ha-clustering:/Stable/
+
+Archives of the tagged release:
+
+* https://github.com/ClusterLabs/crmsh/archive/2.3.0.tar.gz
+* https://github.com/ClusterLabs/crmsh/archive/2.3.0.zip
+
+For the full list of changes since version 2.3.0, see the ChangeLog,
+available at:
+
+* https://github.com/ClusterLabs/crmsh/blob/2.3.0/ChangeLog
+
+
+As usual, a huge thank you to all contributors and users of crmsh!
+
+Cheers,
+Kristoffer
diff --git a/doc/website-v1/news/2016-09-01-release-2_1_7.adoc b/doc/website-v1/news/2016-09-01-release-2_1_7.adoc
new file mode 100644
index 0000000..d1e477e
--- /dev/null
+++ b/doc/website-v1/news/2016-09-01-release-2_1_7.adoc
@@ -0,0 +1,46 @@
+Announcing crmsh stable release 2.1.7
+=====================================
+:Author: Kristoffer Gronlund
+:Email: kgronlund@suse.com
+:Date: 2016-09-01 09:00
+
+Today I are proud to announce the release of `crmsh` version 2.1.7!
+The major new thing in this release is a backports of the events-based
+alerts support from the 2.3 branch.
+
+Big thanks to Hideo Yamauchi for his patience and testing of the
+alerts backport.
+
+This time, the list of changes is small enough that I can add it right
+here:
+
+- high: parse: Backport of event-driven alerts parser (#150)
+- high: hb_report: Don't collect logs from journalctl if -M is set (bsc#990025)
+- high: hb_report: Skip lines without timestamps in log correctly (bsc#989810)
+- high: constants: Add maintenance to set of known attributes (bsc#981659)
+- high: utils: Avoid deadlock if DC changes during idle wait (bsc#978480)
+- medium: scripts: no-quorum-policy=ignore is deprecated (bsc#981056)
+- low: cibconfig: Don't mix up CLI name with XML tag
+
+You can also get the list of changes from the changelog:
+
+* https://github.com/ClusterLabs/crmsh/blob/2.1.7/ChangeLog
+
+Right now, I don't have a set of pre-built rpm packages for Linux
+distributions ready, but I am going to make this available soon. This
+is in particular for centOS 6.x which still relies on Python 2.6
+support which makes running the later releases there more
+difficult. These packages will most likely appear as a subrepository
+here (more details coming soon):
+
+* http://download.opensuse.org/repositories/network:/ha-clustering:/Stable/
+
+Archives of the tagged release:
+
+* https://github.com/ClusterLabs/crmsh/archive/2.1.7.tar.gz
+* https://github.com/ClusterLabs/crmsh/archive/2.1.7.zip
+
+
+Thank you,
+
+Kristoffer
diff --git a/doc/website-v1/news/2016-09-02-release-2_3_1.adoc b/doc/website-v1/news/2016-09-02-release-2_3_1.adoc
new file mode 100644
index 0000000..2a90bef
--- /dev/null
+++ b/doc/website-v1/news/2016-09-02-release-2_3_1.adoc
@@ -0,0 +1,33 @@
+Releasing crmsh version 2.3.1
+=============================
+:Author: Kristoffer Gronlund
+:Email: kgronlund@suse.com
+:Date: 2016-09-02 10:00
+
+Hello everyone!
+
+Today I am releasing crmsh version 2.3.1. The only change this time is
+to lower the Python version requirement from 2.7 to 2.6. This is so
+that crmsh remains compatible with centOS 6, where there is no
+standardized Python 2.7 version available. For users of other
+distributions where Python 2.7 is available, there are no other
+changes in this release and no need to upgrade.
+
+The source code can be downloaded from Github:
+
+* https://github.com/ClusterLabs/crmsh/releases/tag/2.3.1
+
+Packages for several popular Linux distributions can be downloaded
+from the Stable repository at the OBS:
+
+* http://download.opensuse.org/repositories/network:/ha-clustering:/Stable/
+
+Archives of the tagged release:
+
+* https://github.com/ClusterLabs/crmsh/archive/2.3.1.tar.gz
+* https://github.com/ClusterLabs/crmsh/archive/2.3.1.zip
+
+As usual, a huge thank you to all contributors and users of crmsh!
+
+Cheers,
+Kristoffer
diff --git a/doc/website-v1/news/2016-09-05-release-2_2_2.adoc b/doc/website-v1/news/2016-09-05-release-2_2_2.adoc
new file mode 100644
index 0000000..9816b3c
--- /dev/null
+++ b/doc/website-v1/news/2016-09-05-release-2_2_2.adoc
@@ -0,0 +1,36 @@
+Releasing crmsh version 2.2.2
+=============================
+:Author: Kristoffer Gronlund
+:Email: kgronlund@suse.com
+:Date: 2016-09-05 19:00
+
+Hello everyone!
+
+Today I am releasing crmsh version 2.2.2. The biggest change in this
+release is the backport of the support for event-based alerts from the
+2.3 branch. The full list of changes follows below:
+
+- high: parse: Backport of event-driven alerts parser (#150)
+- high: hb_report: Don't collect logs from journalctl if -M is set (bsc#990025)
+- high: hb_report: Skip lines without timestamps in log correctly (bsc#989810)
+- high: constants: Add maintenance to set of known attributes (bsc#981659)
+- high: utils: Avoid deadlock if DC changes during idle wait (bsc#978480)
+- medium: scripts: no-quorum-policy=ignore is deprecated (bsc#981056)
+- medium: tmpfiles: Create temporary directory if non-existing (bsc#981583)
+- medium: xmlutil: reduce unknown attribute to warning (bsc#981659)
+- medium: ui_resource: Add force argument to resource cleanup (bsc#979420)
+- parse: Use original _TARGET_RE
+
+The source code can be downloaded from Github:
+
+* https://github.com/ClusterLabs/crmsh/releases/tag/2.2.2
+
+Archives of the tagged release:
+
+* https://github.com/ClusterLabs/crmsh/archive/2.2.2.tar.gz
+* https://github.com/ClusterLabs/crmsh/archive/2.2.2.zip
+
+As usual, a huge thank you to all contributors and users of crmsh!
+
+Cheers,
+Kristoffer
diff --git a/doc/website-v1/news/2017-01-31-release-3_0_0.adoc b/doc/website-v1/news/2017-01-31-release-3_0_0.adoc
new file mode 100644
index 0000000..31eacea
--- /dev/null
+++ b/doc/website-v1/news/2017-01-31-release-3_0_0.adoc
@@ -0,0 +1,48 @@
+Releasing crmsh version 3.0.0
+=============================
+:Author: Kristoffer Gronlund
+:Email: kgronlund@suse.com
+:Date: 2017-01-31 10:00
+
+Hello everyone!
+
+I'm happy to announce the release of crmsh version 3.0.0 today. The
+main reason for the major version bump is because I have merged the
+sleha-bootstrap project with crmsh, replacing the cluster
+init/add/remove commands with the corresponding commands from
+sleha-bootstrap.
+
+At the moment, these commands are highly specific to SLE and openSUSE,
+unfortunately. I am working on making them as distribution agnostic as
+possible, but would appreciate help from users of other distributions
+in making them work as well on those platforms as they do on
+SLE/openSUSE.
+
+Briefly, the "cluster init" command configures a complete cluster from
+scratch, including optional configuration of fencing via SBD, shared
+storage using OCFS2, setting up the Hawk web interface etc.
+
+There are some other changes in this release as well, see the
+ChangeLog for the complete list of changes:
+
+* https://github.com/ClusterLabs/crmsh/blob/3.0.0/ChangeLog
+
+The source code can be downloaded from Github:
+
+* https://github.com/ClusterLabs/crmsh/releases/tag/3.0.0
+
+This version of crmsh will be available in openSUSE Tumbleweed as soon
+as possible, and packages for several popular Linux distributions are
+available from the Stable repository at the OBS:
+
+* http://download.opensuse.org/repositories/network:/ha-clustering:/Stable/
+
+Archives of the tagged release:
+
+* https://github.com/ClusterLabs/crmsh/archive/3.0.0.tar.gz
+* https://github.com/ClusterLabs/crmsh/archive/3.0.0.zip
+
+As usual, a huge thank you to all contributors and users of crmsh!
+
+Cheers,
+Kristoffer
diff --git a/doc/website-v1/news/2021-06-17-release-4_3_1.adoc b/doc/website-v1/news/2021-06-17-release-4_3_1.adoc
new file mode 100644
index 0000000..1b6a4a6
--- /dev/null
+++ b/doc/website-v1/news/2021-06-17-release-4_3_1.adoc
@@ -0,0 +1,55 @@
+Releasing crmsh version 4.3.1
+=============================
+:Author: Xin Liang
+:Email: XLiang@suse.com
+:Date: 2021-06-17 11:00
+
+Hello everyone!
+
+I'm happy to announce the release of crmsh version 4.3.1
+
+Major changes since 4.3.0
+
+Features:
+
+* Add "crm cluster crash_test" for cluster failure simulation (#825)
+
+* Add ocfs2.OCFS2Manager to manage ocfs2 stage process with cluster lvm2 (#798)
+
+* Support setup SBD via bootstrap "sbd" stage on an existing cluster (#744)
+
+* Enable configuring qdevice on interactive mode (#765)
+
+Fixes:
+
+* Adjust sbd watchdog timeout when using diskless SBD with qdevice (#818)
+
+* Not allow property setting with an empty value (#817)
+
+* Keep consistent for "help <sub-command>" and "<sub-command> -h" for those using argparse (#644)
+
+* Sync corosync.conf before finished joining (#775)
+
+* Adjust qdevice configure/remove process to avoid race condition due to quorum lost (#741)
+
+* Walk through hb_report process under hacluster (#742)
+
+There are some other changes in this release as well, see the
+ChangeLog for the complete list of changes:
+
+* https://github.com/ClusterLabs/crmsh/blob/master/ChangeLog
+
+The source code can be downloaded from Github:
+
+* https://github.com/ClusterLabs/crmsh/releases/tag/4.3.1
+
+Development packages for openSUSE Tumbleweed
+are available from the Open Build System, here:
+
+* https://build.opensuse.org/package/show/network:ha-clustering:Factory/crmsh
+
+As usual, a huge thank you to all contributors and users of crmsh!
+
+
+Regards,
+xin
diff --git a/doc/website-v1/postprocess.py b/doc/website-v1/postprocess.py
new file mode 100644
index 0000000..859abaa
--- /dev/null
+++ b/doc/website-v1/postprocess.py
@@ -0,0 +1,141 @@
+#!/usr/bin/env python
+# create a table of contents for pages that need it
+
+import sys
+import re
+import argparse
+
+TOC_PAGES = ['man/index.html',
+ 'man-4.3/index.html',
+ 'man-3/index.html',
+ 'man-2.0/index.html',
+ 'man-1.2/index.html']
+V2_PAGES = ['index.html']
+INSERT_AFTER = '<!--TOC-->'
+
+def read_toc_data(infile, debug):
+ topics_data = []
+ commands_data = []
+ f = open(infile)
+ for line in f:
+ if line.startswith('[['):
+ line = line[2:-3] # strip [[ and ]]\n
+ info, short_help = line.split(',', 1)
+ short_help = short_help.strip()
+ info_split = info.split('_')
+ if info_split[0] == 'topics':
+ if len(info_split) == 2:
+ topics_data.append((1, short_help, info))
+ elif len(info_split) >= 3:
+ topics_data.append((2, short_help, info))
+ elif info_split[0] == 'cmdhelp':
+ if len(info_split) == 2:
+ commands_data.append((2, info_split[1], info))
+ elif len(info_split) >= 3:
+ commands_data.append((3, '_'.join(info_split[2:]), info))
+ toc = ''
+ if len(topics_data) > 0 or len(commands_data) > 0:
+ toc = '<div id="toc">\n'
+ for depth, text, link in topics_data:
+ toc += '<div class="toclevel%s"><a href="#%s">%s</a></div>\n' % (
+ depth, link, text)
+ for depth, text, link in commands_data:
+ toc += '<div class="toclevel%s"><a href="#%s">%s</a></div>\n' % (
+ depth, link, text)
+ toc += '</div>\n'
+ return toc
+
+def generate_toc(infile, outfile, debug):
+
+ if debug:
+ print "Infile:", infile
+ toc = read_toc_data(infile, debug)
+ '''
+ toc_data = []
+ section = re.compile(r"<h(?P<depth>[0-9])( id=\"(?P<id>[^\"]+)\")?>(?P<text>.*)</h[0-9]>")
+ for line in f:
+ m = section.match(line)
+ if m:
+ if debug:
+ print "toc_data: %s" % str(((m.group('depth'), m.group('text'), m.group('id'))))
+ toc_data.append((m.group('depth'), m.group('text'), m.group('id')))
+
+ toc = ''
+ if len(toc_data) > 0:
+ toc = '<div id="toc">\n'
+ for depth, text, link in toc_data:
+ if depth >= 2 and link is not None:
+ toc += '<div class="toclevel%s"><a href="#%s">%s</a></div>\n' % (
+ int(depth) - 1, link, text)
+ toc += '</div>\n'
+'''
+
+ # Write TOC to outfile
+ if outfile:
+ if debug:
+ print "Writing TOC:"
+ print "----"
+ print toc
+ print "----"
+ print "Outfile:", outfile
+ fil = open(outfile)
+ f = fil.readlines()
+ fil.close()
+ f2 = open(outfile, 'w')
+ for line in f:
+ f2.write(line)
+ if toc and line.startswith(INSERT_AFTER):
+ f2.write(toc)
+ f2.close()
+
+def generate_v2(page, debug):
+ f = open(page).readlines()
+ toc_data = []
+ section = re.compile(r"<h(?P<depth>[0-9])( id=\"(?P<id>[^\"]+)\")?>(?P<text>.*)</h[0-9]>")
+ for line in f:
+ m = section.match(line)
+ if m:
+ if debug:
+ print "toc_data: %s" % str(((m.group('depth'), m.group('text'), m.group('id'))))
+ toc_data.append((m.group('depth'), m.group('text'), m.group('id')))
+
+ toc = ''
+ if len(toc_data) > 0:
+ toc = '<div id="toc">\n'
+ for depth, text, link in toc_data:
+ if depth >= 2 and link is not None:
+ toc += '<div class="toclevel%s"><a href="#%s">%s</a></div>\n' % (
+ int(depth) - 1, link, text)
+ toc += '</div>\n'
+ f2 = open(page, 'w')
+ for line in f:
+ f2.write(line)
+ if toc and line.startswith(INSERT_AFTER):
+ f2.write(toc)
+ f2.close()
+
+def main():
+ parser = argparse.ArgumentParser(description="Generate table of contents")
+ parser.add_argument('-d', '--debug', dest='debug', action='store_true',
+ help="Enable debug output")
+ parser.add_argument('-o', '--output', metavar='output', type=str,
+ help="File to inject TOC into")
+ parser.add_argument('input', metavar='input', type=str,
+ help="File to read TOC metadata from")
+ args = parser.parse_args()
+ debug = args.debug
+ outfile = args.output
+ infile = args.input
+ print "+ %s -> %s" % (infile, outfile)
+ gen = False
+ for tocpage in TOC_PAGES:
+ if not gen and outfile.endswith(tocpage):
+ generate_toc(infile, outfile, debug)
+ gen = True
+ for tocpage in V2_PAGES:
+ if not gen and outfile.endswith(tocpage):
+ generate_v2(outfile, debug)
+ gen = True
+
+if __name__ == "__main__":
+ main()
diff --git a/doc/website-v1/rsctest-guide.adoc b/doc/website-v1/rsctest-guide.adoc
new file mode 100644
index 0000000..2dcd865
--- /dev/null
+++ b/doc/website-v1/rsctest-guide.adoc
@@ -0,0 +1,238 @@
+= Resource testing =
+
+Never created a pacemaker cluster configuration before? Please
+read on.
+
+Ever created a pacemaker configuration without errors? All
+resources worked from the get go on all your nodes? Really? We
+want a photo of you!
+
+Seriously, it is so error prone to get a cluster resource
+definition right that I think I ever only managed to do it with
+`Dummy` resources. There are many intricate details that have to be
+just right, and all of them are stuffed in a single place as simple
+name-value attributes. Then there are multiple nodes, each node
+containing a complex system environment inevitably always in flux and
+changing (entropy anybody?).
+
+Now, once you defined your set of resources and are about to
+_commit_ the configuration (at that point it usually takes a
+deep breath to do so), be ready to meet an avalanche of error
+messages, not all of which are easy to understand or follow. Not
+to mention that you need to read the logs too. Even though we do
+have a link:history-tutorial.html[tool] to help with digging through
+the logs, it is going to be an interesting experience and not quite
+recommended if you're just starting with pacemaker clusters. Even the
+experts can save a lot of time and headaches by following the advice
+below.
+
+== Basic usage ==
+
+Enter resource testing. It is a special feature designed to help
+users find problems in resource configurations.
+
+The usage is very simple:
+
+----
+crm(live)configure# rsctest web-server
+Probing resources ..
+testing on xen-f: apache web-ip
+testing on xen-g: apache web-ip
+crm(live)configure#
+----
+
+What actually happened above and what is it good for? From the
+output we can infer that the `web-server` resource is actually a
+group comprising one apache web server and one IP address.
+Indeed:
+
+----
+crm(live)configure# show web-server
+group web-server apache web-ip \
+ meta target-role="Stopped"
+crm(live)configure#
+----
+
+The `rsctest` command first established that the resources are
+stopped on all nodes in the cluster. Then it tests the resources
+in the order defined by the resource group on all nodes. It does
+this by manually starting the resources, one by one, then running
+a "monitor" for each resource to make sure that the resources are
+healthy, and finally stopping the resources in reverse order.
+
+Since there is no additional output, the test passed. It looks
+like we have a properly defined web server group.
+
+== Reporting problems ==
+
+Now, the above run was not very interesting so let's spoil the
+idyll:
+
+----
+xen-f:~ # mv /etc/apache2/httpd.conf /tmp
+----
+
+We moved the apache configuration file away on node `xen-f`. The
+`apache` resource should fail now:
+
+----
+crm(live)configure# rsctest web-server
+Probing resources ..
+testing on xen-f: apache
+host xen-f (exit code 5)
+xen-f stderr:
+2013/10/17_16:51:26 ERROR: Configuration file /etc/apache2/httpd.conf not found!
+2013/10/17_16:51:26 ERROR: environment is invalid, resource considered stopped
+
+testing on xen-g: apache web-ip
+crm(live)configure#
+----
+
+As expected, `apache` failed to start on node `xen-f`. When the
+cluster resource manager runs an operation on a resource, all
+messages are logged (there is no terminal attached to the
+cluster, anyway). All one can see in the resource status is the type
+of the exit code. In this case, it is an installation problem.
+
+For instance, the output could look like this:
+
+----
+xen-f:~ # crm status
+Last updated: Thu Oct 17 19:21:44 2013
+Last change: Thu Oct 17 19:21:28 2013 by root via crm_resource on xen-f
+...
+Failed actions:
+ apache_start_0 on xen-f 'not installed' (5): call=2074, status=complete,
+last-rc-change='Thu Oct 17 19:21:31 2013', queued=164ms, exec=0ms
+----
+
+That does not look very informative. With `rsctest` we can
+immediately see what the problem is. It saves us prowling the
+logs looking for messages of the `apache` resource agent.
+
+Note that the IP address is not tested, because the resource it
+depends on could not be started.
+
+== What is tested? ==
+
+The start, monitor, and stop operations, in exactly that order,
+are tested for every resource specified. Note that normally the
+two latter operations should never fail if the resource agent is
+well implemented. The RA should under normal circumstances be
+able to stop or monitor a started resource. However, this is
+_not_ a replacement for resource agent testing. If that is what
+you are looking for, see
+http://www.linux-ha.org/doc/dev-guides/_testing_resource_agents.html[the
+RA testing chapter] of the RA development guide.
+
+== Protecting resources ==
+
+The `rsctest` command goes to great lengths to prevent starting a
+resource on more than one node at the same time. For some stuff
+that would actually mean data corruption and we certainly don't
+want that to happen.
+
+----
+xen-f:~ # (echo start web-server; echo show web-server) | crm -w resource
+resource web-server is running on: xen-g
+xen-f:~ # crm configure rsctest web-server
+Probing resources .WARNING: apache:probe: resource running at xen-g
+.WARNING: web-ip:probe: resource running at xen-g
+
+Stop all resources before testing!
+xen-f:~ # crm configure rsctest web-server xen-f
+Probing resources .WARNING: apache:probe: resource running at xen-g
+.WARNING: web-ip:probe: resource running at xen-g
+
+Stop all resources before testing!
+xen-f:~ #
+----
+
+As you can see, if `rsctest` finds any of the resources running
+on any node it refuses to run any tests.
+
+== Multi-state and clone resources ==
+
+Apart from groups, the `rsctest` can also handle the other two
+special kinds of resources. Let's take a look at one `drbd`-based
+configuration:
+
+----
+crm(live)configure# show ms_drbd_nfs drbd0-vg
+primitive drbd0-vg ocf:heartbeat:LVM \
+ params volgrpname="drbd0-vg"
+primitive p_drbd_nfs ocf:linbit:drbd \
+ meta target-role="Stopped" \
+ params drbd_resource="nfs" \
+ op monitor interval="15" role="Master" \
+ op monitor interval="30" role="Slave" \
+ op start interval="0" timeout="300" \
+ op stop interval="0" timeout="120"
+ms ms_drbd_nfs p_drbd_nfs \
+ meta notify="true" clone-max="2"
+crm(live)configure#
+----
+
+The `nfs` drbd resource contains a volume group `drbd0-vg`.
+
+----
+crm(live)configure# rsctest ms_drbd_nfs drbd0-vg
+Probing resources ..
+testing on xen-f: p_drbd_nfs drbd0-vg
+testing on xen-g: p_drbd_nfs drbd0-vg
+crm(live)configure#
+----
+
+For the multi-state (master-slave) resources, the involved
+resource motions are somewhat more complex: the resource is first
+started on both nodes and then promoted on the node where the
+next resource is to be tested (in this case the volume group).
+Then it gets demoted to slave and promoted on the other
+node to master so that the depending resources can be tested on
+that node too.
+
+Note that even though we asked for `ms_drbd_nfs` to be tested,
+there is `p_drbd_nfs` in the output which is the primitive
+encapsulated in the master-slave resource. You can specify either
+one.
+
+== Stonith resources ==
+
+The stonith resources are also special and need special
+treatment. What is tested is just the device status. Actually
+fencing nodes was deemed too drastic. Please use `node fence` to
+test the fencing device effectiveness. It also does not matter
+whether the stonith resource is "running" on any node: being
+started is just something that happens virtually in the
+`stonithd` process.
+
+== Summary ==
+
+- use `rsctest` to make sure that the resources can be started
+ correctly on all nodes
+
+- `rsctest` protects resources by making sure beforehand that
+ none of them is currently running on any of the cluster nodes
+
+- `rsctest` understands groups, master-slave (multi-state), and
+ clone resources, but nothing else of the configuration
+ (constraints or any other placement/order cluster configuration
+ elements)
+
+- it is up to the user to test resources only on nodes which are
+ really supposed to run them and in a proper order (if that
+ order is expressed via constraints)
+
+- `rsctest` cannot protect resources if they are running on
+ nodes which are not present in the cluster or from bad RA
+ implementations (but neither would a cluster resource manager)
+
+- `rsctest` was designed as a debugging and configuration aid, and is
+ not intended to provide full Resource Agent test coverage.
+
+== `crmsh` help and online resources (_sic!_) ==
+
+- link:crm.8.html#topics_Testing[`crm help Testing`]
+
+- link:crm.8.html#cmdhelp_configure_rsctest[`crm configure help
+rsctest`]
diff --git a/doc/website-v1/scripts.adoc b/doc/website-v1/scripts.adoc
new file mode 100644
index 0000000..7742729
--- /dev/null
+++ b/doc/website-v1/scripts.adoc
@@ -0,0 +1,660 @@
+= Cluster Scripts =
+:source-highlighter: pygments
+
+.Version information
+NOTE: This section applies to `crmsh 2.2+` only.
+
+== Introduction ==
+
+A big part of the configuration and management of a cluster is
+collecting information about all cluster nodes and deploying changes
+to those nodes. Often, just performing the same procedure on all nodes
+will encounter problems, due to subtle differences in the
+configuration.
+
+For example, when configuring a cluster for the first time, the
+software needs to be installed and configured on all nodes before the
+cluster software can be launched and configured using `crmsh`. This
+process is cumbersome and error-prone, and the goal is for scripts to
+make this process easier.
+
+Another important function of scripts is collecting information and
+reporting potential issues with the cluster. For example, software
+versions may differ between nodes, causing byzantine errors or random
+failure. `crmsh` comes packaged with a `health` script which will
+detect and warn about many of these types of problems.
+
+There are many tools for managing a collection of nodes, and scripts
+are not intended to replace these tools. Rather, they provide an
+integrated way to perform tasks across the cluster that would
+otherwise be tedious, repetitive and error-prone. The scripts
+functionality in the crm shell is mainly inspired by Ansible, a
+light-weight and efficient configuration management tool.
+
+Scripts are implemented using the python `parallax` package which
+provides a thin wrapper on top of SSH. This allows the scripts to
+function through the usual SSH channels used for system maintenance,
+requiring no additional software to be installed or maintained.
+
+For many scripts that only configure cluster resources or only perform
+changes on the local machine, the use of SSH is not necessary. These
+scripts can be used even if there is no way for `crmsh` to reach the
+other nodes other than through the cluster configuration.
+
+NOTE: The scripts functionality in `crmsh` has been greatly expanded
+and improved in `crmsh` 2.2. Many new scripts have been added, and in
+addition the scripts are now used as the backend for the wizards
+functionality in HAWK, the HA web interface. For more information, see
+https://github.com/ClusterLabs/hawk.
+
+== Usage ==
+
+Scripts are available through the `cluster` sub-level in the crm
+shell. Some scripts have custom commands linked to them for
+convenience, such as the `init`, `add` and `remove` commands available
+in the `cluster` sublevel, for creating new clusters, introducing new
+nodes into the cluster and for removing nodes from a running cluster.
+
+Other scripts can be accessed through the `script` sub-level.
+
+=== Common Parameters ===
+
+Which parameters a script accepts varies from script to
+script. However, there is a set of parameters that are common to all
+scripts. These parameters can be passed to any script.
+
+`nodes`::
+ List of nodes to execute the script for
+`dry_run`::
+ If set, simulate execution only
+ (default: no)
+`action`::
+ If set, only execute a single action (index, as returned by verify)
+`statefile`::
+ When single-stepping, the state is saved in the given file
+`user`::
+ Run script as the given user
+`sudo`::
+ If set, crm will prompt for a sudo password and use sudo when appropriate
+ (default: no)
+`port`::
+ Port to connect on
+`timeout`::
+ Execution timeout in seconds
+ (default: 600)
+
+=== List available scripts ===
+
+To list the available scripts, use the following command:
+
+.........
+# crm script
+list
+.........
+
+The available scripts are listed along with a short
+description. Optionally, the arguments +all+ or +names+ can be
+used. Without the +all+ flag, some scripts that are used by `crmsh` to
+implement certain commands are hidden from view. With the +names+
+flag, only a plain list of script names is printed.
+
+=== Script description ===
+
+To get more details about a script, run the `show` command. For
+example, to get more information about what the `virtual-ip` script does
+and what parameters it accepts, use the following command:
+
+.........
+# crm script
+show virtual-ip
+.........
+
+`show` will print a longer description of the script, along with a
+list of parameters divided into _steps_. Each script is divided into a
+series of steps which are performed in order. Some steps may not
+accept any parameters, but for those that do, the available parameters
+are listed here.
+
+By default, only a basic subset of the available parameters is printed
+in order to make the scripts easier to use. By passing `all` to the
+`show` command, the advanced parameters are also shown. In addition,
+there is a list of common parameters
+
+`show` will print a longer explanation for the script, along with
+a list of parameters, each parameter having a description, a note
+saying if it is an optional or required parameter, and if optional,
+what the default value is.
+
+=== Verifying parameters ===
+
+Since a script potentially performs a series of actions and may fail
+for various reasons at any point, it is advisable to review the
+actions that a script will perform before actually running it. To do
+this, the `verify` command can be used.
+
+Pass the parameters that you would pass to `run`, and `verify` will
+check that the parameter values are OK, as well as print the sequence
+of steps that will be performed given the particular parameter values
+given.
+
+The following is an example showing how to verify the creation of a
+Virtual IP resource, using the `virtual-ip` script:
+
+..........
+# crm script
+verify virtual-ip id=my-virtual-ip ip=192.168.0.10
+..........
+
+`crmsh` will print something similar to the following output:
+
+...........
+1. Configure cluster resources
+
+ primitive my-virtual-ip ocf:heartbeat:IPaddr2
+ ip="192.168.0.10"
+ op start timeout="20" op stop timeout="20"
+ op monitor interval="10" timeout="20"
+...........
+
+In this particular case, there is only a single step, and that step
+configures a primitive resource. Other scripts may configure multiple
+resources and constraints, or may perform multiple steps in sequence.
+
+=== Running a script ===
+
+To run a script, all required parameters and any optional parameters
+that should have values other than the default should be provided as
+`key=value` pairs on the command line.
+
+The following example shows how to create a Virtual IP resource using
+the `virtual-ip` script:
+
+........
+# crm script
+run virtual-ip id=my-virtual-ip ip=192.168.0.10
+........
+
+==== Single-stepping a script ====
+
+It is possible to run a script action-by-action, with manual intervention
+between actions. First of all, list the actions to perform given a
+certain set of parameter values:
+
+........
+crm script verify health
+........
+
+To execute a single action, two things need to be provided:
+
+1. The index of the action to execute (printed by `verify`)
+2. a file in which `crmsh` stores the state of execution.
+
+Note that it is entirely possible to run actions out-of-order, however
+this is unlikely to work in practice since actions often rely on the
+outcome of previous actions.
+
+The following command will execute the first action of the `health`
+script and store the output in a temporary file named `health.json`:
+
+........
+crm script run health action=1 statefile='health.json'
+........
+
+The statefile contains the script parameters and the output of
+previous steps, encoded as `json` data.
+
+To continue executing the next action in sequence, enter the next
+action index:
+
+........
+crm script run health action=2 statefile='health.json'
+........
+
+Note that the `dry_run` flag that can be used to do partial execution
+of scripts is not taken into consideration when single-stepping
+through a script.
+
+== Creating a script ==
+
+This section will describe how to create a new script, where to put
+the script to allow `crmsh` to find it, and how to test that the
+script works as intended.
+
+=== How scripts work, in detail ===
+
+NOTE: The implementation of cluster scripts was revised between
+`crmsh` 2.0 and `crmsh` 2.2. This section describes the revised
+cluster script format. The old format is still accepted by `crmsh`.
+
+A cluster script consists of four main sections:
+
+. The name and description of the script.
+. Any other scripts or agents included by this script, and any parameter value overrides to those provided by the included script.
+. A set of parameters accepted by the script itself, in addition to those accepted by any scripts or agents included in the script.
+. A sequence of actions which the script will perform.
+
+When the script runs, the actions defined in `main.yml` as described
+below are executed one at a time. Each action prescribes a
+modification that is applied to the cluster. Some actions work by
+calling out to scripts on each of the cluster nodes, and others apply
+only on the local node from which the script was executed.
+
+=== Actions ===
+
+Scripts perform actions that are classified into a few basic
+types. Each action is performed by calling out to a shell script,
+but the arguments and location of that script varies depending on the
+type.
+
+Here are the types of script actions that can be performed:
+
+cib::
+ * Applies a new CIB configuration to the cluster
+
+install::
+ * Ensures that the given list of packages is installed on all
+ cluster nodes using the system package manager.
+
+service::
+ * Manages system services using the system init tools. The argument
+ should be a space-separated list of <service>:<state> pairs.
+
+call::
+ * Run a shell command as specified in the action, either on the
+ local node on or all nodes.
+
+copy::
+ * Installs a file on the cluster nodes.
+ * Using a configuration template, install a file on the cluster
+ nodes.
+
+crm::
+ * Runs the given command using the `crm` shell. This can be used to
+ start and stop resources, for example.
+
+collect::
+ * Runs on all cluster nodes
+ * Gathers information about the nodes, both general information and
+ information specific to the script.
+
+validate::
+ * Runs on the local node
+ * Validate parameter values and node state based on collected
+ information. Can modify default values and report issues that
+ would prevent the script from applying successfully.
+
+apply::
+ * Runs on all or any cluster nodes
+ * Applies changes, returning information about the applied changes
+ to the local node.
+
+apply_local::
+ * Runs on the local node
+ * Applies changes to the cluster, where an action taken on a single
+ node affect the entire cluster. This includes updating the CIB in
+ Pacemaker, and also reloading the configuration for Corosync.
+
+report::
+ * Runs on the local node
+ * This is similar to the _apply_local_ action, with the difference
+ that the output of a Report action is not interpreted as JSON data
+ to be passed to the next action. Instead, the output is printed to
+ the screen.
+
+==== When expressions ====
+
+Actions can be made conditional on the value of script parameters using
+the +when:+ expression. This expression has two basic forms.
+
+The first form is in the form of the name of a script parameter. For
+example, given a boolean script parameter named +install+, an action
+can be made conditional on that parameter being true using the syntax
++when: install+.
+
+The second form is a more complex expression. All parameters are
+interpreted as either a string value or None if no value was provided.
+These can be compared to string literals using python-style
+comparators. For example, an action can be conditional on the string
+parameter +mode+ having the value +"advanced"+ using the following
+syntax: +when: mode == "advanced"+.
+
+=== Basic structure ===
+
+The crm shell looks for scripts in two primary locations: Included
+scripts are installed in the system-wide shared folder, usually
+`/usr/share/crmsh/scripts/`. Local and custom scripts are loaded from
+the user-local XDG_CONFIG folder, usually found at
+`~/.local/crm/scripts/`. These locations may differ depending on how
+the crm shell was installed and which system is used, but these are
+the locations used on most distributions.
+
+To create a new script, make a new folder in the user-local scripts
+folder and give it a unique name. In this example, we will call our
+new script `check-uptime`.
+
+........
+mkdir -p ~/.local/crm/scripts/check-uptime
+........
+
+In this directory, create a file called `main.yml`. This is a YAML
+document which describes the script, which parameters it requires, and
+what actions it will perform.
+
+YAML is a human-readable markup language which is designed to be easy
+to read and modify, while at the same time be compatible with JSON. To
+learn more, see http:://yaml.org/[yaml.org].
+
+Here is an example `main.yml` file which wraps the resource agent
+`ocf:heartbeat:IPaddr2`.
+
+[source,yaml]
+----
+# The version must be exactly 2.2, and must always be
+# specified in the script. If the version is missing or
+# is less than 2.2, the script is assumed to be a legacy
+# script (specified in the format used before crmsh 2.2).
+version: 2.2
+shortdesc: Virtual IP
+category: Basic
+include:
+ - agent: ocf:heartbeat:IPaddr2
+ name: virtual-ip
+ parameters:
+ - name: id
+ type: resource
+ required: true
+ - name: ip
+ type: ip_address
+ required: true
+ - name: cidr_netmask
+ type: integer
+ required: false
+ - name: broadcast
+ type: ip_address
+ required: false
+ ops: |
+ op start timeout="20" op stop timeout="20"
+ op monitor interval="10" timeout="20"
+actions:
+ - include: virtual-ip
+----
+
+For a bigger example, here is the `apache` agent which includes
+multiple optional steps, the optional installation of packages,
+defines multiple cluster resources and potentially calls bash commands
+on each of the cluster nodes.
+
+[source,yaml]
+----
+# Copyright (C) 2009 Dejan Muhamedagic
+# Copyright (C) 2015 Kristoffer Gronlund
+#
+# License: GNU General Public License (GPL)
+version: 2.2
+category: Server
+shortdesc: Apache Webserver
+longdesc: |
+ Configure a resource group containing a virtual IP address and
+ an instance of the Apache web server.
+
+ You can optionally configure a Filesystem resource which will be
+ mounted before the web server is started.
+
+ You can also optionally configure a database resource which will
+ be started before the web server but after mounting the optional
+ filesystem.
+include:
+ - agent: ocf:heartbeat:apache
+ name: apache
+ longdesc: |
+ The Apache configuration file specified here must be available via the
+ same path on all cluster nodes, and Apache must be configured with
+ mod_status enabled. If in doubt, try running Apache manually via
+ its init script first, and ensure http://localhost:80/server-status is
+ accessible.
+ ops: |
+ op start timeout="40"
+ op stop timeout="60"
+ op monitor interval="10" timeout="20"
+ - script: virtual-ip
+ shortdesc: The IP address configured here will start before the Apache instance.
+ parameters:
+ - name: id
+ value: "{{id}}-vip"
+ - script: filesystem
+ shortdesc: Optional filesystem mounted before the web server is started.
+ required: false
+ - script: database
+ shortdesc: Optional database started before the web server is started.
+ required: false
+parameters:
+ - name: install
+ type: boolean
+ shortdesc: Install and configure apache
+ value: false
+actions:
+ - install:
+ - apache2
+ shortdesc: Install the apache package
+ when: install
+ - service:
+ - apache: disable
+ shortdesc: Let cluster manage apache
+ when: install
+ - call: a2enmod status; true
+ shortdesc: Enable status module
+ when: install
+ - include: filesystem
+ - include: database
+ - include: virtual-ip
+ - include: apache
+ - cib: |
+ group g-{{id}}
+ {{filesystem:id}}
+ {{database:id}}
+ {{virtual-ip:id}}
+ {{id}}
+----
+
+The language for referring to parameter values in `cib` actions is
+described below.
+
+=== Command arguments ===
+
+The actions that accept a command as argument must not refer to
+commands written in python. They can be plain bash scripts or any
+other executable script as long as the nodes have the necessary
+dependencies installed. However, see below why implementing scripts in
+Python is easier.
+
+Actions report their progress either by returning JSON on standard
+output, or by returning a non-zero return value and printing an error
+message to standard error.
+
+Any JSON returned by an action will be available to the following
+steps in the script. When the script executes, it does so in a
+temporary folder created for that purpose. In that folder is a file
+named `script.input`, containing a JSON array with the output produced
+by previous steps.
+
+The first element in the array (the zeroth element, to be precise) is
+a dict containing the parameter values.
+
+The following elements are dicts with the hostname of each node as key
+and the output of the action generated by that node as value.
+
+In most cases, only local actions (`validate` and `apply_local`) will
+use the information in previous steps, but scripts are not limited in
+what they can do.
+
+With this knowledge, we can implement `fetch.py` and `report.py`.
+
+`fetch.py`:
+
+[source,python]
+----
+#!/usr/bin/python3
+import crm_script as crm
+try:
+ uptime = open('/proc/uptime').read().split()[0]
+ crm.exit_ok(uptime)
+except Exception as e:
+ crm.exit_fail("Couldn't open /proc/uptime: %s" % (e))
+----
+
+`report.py`:
+
+[source,python]
+----
+#!/usr/bin/python3
+import crm_script as crm
+show_all = crm.is_true(crm.param('show_all'))
+uptimes = list(crm.output(1).items())
+max_uptime = '', 0
+for host, uptime in uptimes:
+ if float(uptime) > max_uptime[1]:
+ max_uptime = host, float(uptime)
+if show_all:
+ print("Uptimes: %s" % (', '.join("%s: %s" % v for v in uptimes)))
+print("Longest uptime is %s seconds on host %s" % (max_uptime[1], max_uptime[0]))
+----
+
+See below for more details on the helper library `crm_script`.
+
+Save the scripts as executable files in the same directory as the
+`main.yml` file.
+
+Before running the script, it is possible to verify that the files are
+in a valid format and in the right location. Run the following
+command:
+
+........
+crm script verify check-uptime
+........
+
+If the verification is successful, try executing the script with the
+following command:
+
+........
+crm script run check-uptime
+........
+
+Example output:
+
+[source,bash]
+----
+# crm script run check-uptime
+INFO: Check uptime of nodes
+INFO: Nodes: ha-three, ha-one
+OK: Fetch uptimes
+OK: Report uptime
+Longest uptime is 161054.04 seconds on host ha-one
+----
+
+To see if the `show_all` parameter works as intended, run the
+following:
+
+........
+crm script run check-uptime show_all=yes
+........
+
+Example output:
+
+[source,bash]
+----
+# crm script run check-uptime show_all=yes
+INFO: Check uptime of nodes
+INFO: Nodes: ha-three, ha-one
+OK: Fetch uptimes
+OK: Report uptime
+Uptimes: ha-one: 161069.83, ha-three: 159950.38
+Longest uptime is 161069.83 seconds on host ha-one
+----
+
+=== Remote permissions ===
+
+Some scripts may require super-user access to remote or local
+nodes. It is recommended that this is handled through SSH certificates
+and agents, to facilitate password-less access to nodes.
+
+=== Running scripts without a cluster ===
+
+All cluster scripts can optionally take a `nodes` argument, which
+determines the nodes that the script will run on. This node list is
+not limited to nodes already in the cluster. It is even possible to
+execute cluster scripts before a cluster is set up, such as the
+`health` and `init` scripts used by the `cluster` sub-level.
+
+........
+crm script run health nodes=example1,example2
+........
+
+The list of nodes can be comma- or space-separated, but if the list
+contains spaces, the whole argument will have to be quoted:
+
+........
+crm script run health nodes="example1 example2"
+........
+
+=== Running in validate mode ===
+
+It may be desirable to do a dry-run of a script, to see if any
+problems are present that would make the script fail before trying to
+apply it. To do this, add the argument `dry_run=yes` to the invocation:
+
+.........
+crm script run health dry_run=yes
+.........
+
+The script execution will stop at the first `apply` action. Note that
+non-modifying steps that happen after the first `apply` action will
+not be performed in a dry run.
+
+=== Helper library ===
+
+When the script data is copied to each node, a small helper library is
+also passed along with the script. This library can be found in
+`utils/crm_script.py` in the source repository. This library helps
+with producing output in the correct format, parsing the
+`script.input` data provided to scripts, and more.
+
+.`crm_script` API
+`host()`::
+ Returns hostname of current node
+`get_input()`::
+ Returns the input data list. The first element in the list
+ is a dict of the script parameters. The rest are the output
+ from previous steps.
+`parameters()`::
+ Returns the script parameters as a dict.
+`param(name)`::
+ Returns the value of the named script parameter.
+`output(step_idx)`::
+ Returns the output of the given step, with the first step being step 1.
+`exit_ok(data)`::
+ Exits the step returning `data` as output.
+`exit_fail(msg)`::
+ Exits the step returning `msg` as error message.
+`is_true(value)`::
+ Converts a truth value from string to boolean.
+`call(cmd, shell=False)`::
+ Perform a system call. Returns `(rc, stdout, stderr)`.
+
+=== The handles language ===
+
+CIB configurations and commands can refer to the value of parameters
+in the text of the action. This is done using a custom language,
+similar to handlebars.
+
+The language accepts the following constructions:
+
+............
+{{name}} = Inserts the value of the parameter <name>
+{{script:name}} = Inserts the value of the parameter <name> from the
+ included script named <script>.
+{{#name}} ... {{/name}} = Inserts the text between the mustasches when
+ name is truthy.
+{{^name}} ... {{/name}} = Inserts the text between the mustasches when
+ name is falsy.
+............
+
diff --git a/doc/website-v1/start-guide.adoc b/doc/website-v1/start-guide.adoc
new file mode 100644
index 0000000..7ad6a82
--- /dev/null
+++ b/doc/website-v1/start-guide.adoc
@@ -0,0 +1,208 @@
+= Getting Started
+
+So, you've successfully installed `crmsh` on one or more machines, and
+now you want to configure a basic cluster. This guide is intended to
+provide step-by-step instructions for configuring Pacemaker
+with a single resource capable of failing over between a pair of
+nodes, and then builds on that base to cover some more advanced topics
+of cluster management.
+
+****
+Haven't installed yet? Please follow the
+link:/installation[installation instructions]
+before continuing this guide. Only `crmsh` and
+its dependencies need to be installed before
+following this guide.
+****
+
+Before continuing, make sure that this command executes successfully
+on all nodes, and returns a version number that is `3.0` or higher:
+
+........
+crm --version
+........
+
+****
+In crmsh 3, the cluster init commands were replaced by the SLE HA
+bootstrap scripts. These rely on `csync2` for configuration file
+management, so make sure that you have the `csync2` command
+installed before proceeding. This requirement may be removed in
+the future.
+****
+
+.Example cluster
+**************************
+
+These are the machines used as an example in this guide. Please
+replace the references to these names and IP addresses to the values
+appropriate for your cluster:
+
+
+[options="header,footer"]
+|=======================
+|Name |IP
+|alice |10.0.0.2
+|bob |10.0.0.3
+|=======================
+**************************
+
+
+== The cluster stack
+
+The composition of the GNU/Linux cluster stack has changed somewhat
+over the years. The stack described here is the currently most common
+variant, but there are other ways of configuring these tools.
+
+Simply put, a High Availability cluster is a set of machines (commonly
+referred to as *nodes*) with redundant capacity, such that if one or
+more of these machines experience failure of any kind, the other nodes
+in the cluster can take over the responsibilities previously handled
+by the failed node.
+
+The cluster stack is a set of programs running on all of these nodes,
+communicating with each other over the network to monitor each other
+and deciding where, when and how resources are stopped, started or
+reconfigured.
+
+The main component of the stack is *Pacemaker*, the software
+responsible for managing cluster resources, allocating them to cluster
+nodes according to the rules specified in the *CIB*.
+
+The CIB is an XML document maintained by Pacemaker, which describes
+all cluster resources, their configuration and the constraints that
+decide where and how they are managed. This document is not edited
+directly, and with the help of `crmsh` it is possible to avoid
+exposure to the underlying XML at all.
+
+Beneath Pacemaker in the stack sits *Corosync*, a cluster
+communication system. Corosync provides the communication capabilities
+and cluster membership functionality used by Pacemaker. Corosync is
+configured through the file `/etc/corosync/corosync.conf`. `crmsh`
+provides tools for configuring corosync similar to Pacemaker.
+
+Aside from these two components, the stack also consists of a
+collection of *Resource Agents*. These are basically scripts that wrap
+software that the cluster needs to manage, providing a unified
+interface to configuration, supervision and management of the
+software. For example, there are agents that handle virtual IP
+resources, web servers, databases and filesystems.
+
+`crmsh` is a command line tool which interfaces against all of these
+components, providing a unified interface for configuration and
+management of the whole cluster stack.
+
+== SSH
+
+`crmsh` runs as a command line tool on any one of the cluster
+nodes. In order for to to control all cluster nodes, it needs to be
+able to execute commands remotely. `crmsh` does this by invoking
+`ssh`.
+
+Configure `/etc/hosts` on each of the nodes so that the names of the
+other nodes map to the IP addresses of those nodes. For example in a
+cluster consisting of `alice` and `bob`, executing `ping bob` when
+logged in as root on `alice` should successfully locate `bob` on the
+network. Given the IP addresses of `alice` and `bob` above, the
+following should be entered into `/etc/hosts` on both nodes:
+
+........
+10.0.0.2 alice
+10.0.0.3 bob
+........
+
+== Install and configure
+
+To configure the basic cluster, we use the `cluster init` command
+provided by `crmsh`. This command has quite a few options for
+setting up the cluster, but we will use a fairly basic configuration.
+
+........
+crm cluster init --name demo-cluster --nodes "alice bob"
+........
+
+The initialization tool will now ask a series of questions about the
+configuration, and then proceed to configure and start the cluster
+on both nodes.
+
+== Check cluster status
+
+To see if Pacemaker is running, what nodes are part of the cluster and
+what resources are active, use the `status` command:
+
+.........
+crm status
+.........
+
+If this command fails or times out, there is some problem with
+Pacemaker or Corosync on the local machine. Perhaps some dependency is
+missing, a firewall is blocking cluster communication or some other
+unrelated problem has occurred. If this is the case, the `cluster
+health` command may be of use.
+
+== Cluster health check
+
+To check the health status of the machines in the cluster, use the
+following command:
+
+........
+crm cluster health
+........
+
+This command will perform multiple diagnostics on all nodes in the
+cluster, and return information about low disk space, communication
+issues or problems with mismatching software versions between nodes,
+for example.
+
+If no cluster has been configured or there is some fundamental problem
+with cluster communications, `crmsh` may be unable to figure out what
+nodes are part of the cluster. If this is the case, the list of nodes
+can be provided to the health command directly:
+
+........
+crm cluster health nodes=alice,bob
+........
+
+== Adding a resource
+
+To test the cluster and make sure it is working properly, we can
+configure a Dummy resource. The Dummy resource agent is a simple
+resource that doesn't actually manage any software. It exposes a
+single numerical parameter called `state` which can be used to test
+the basic functionality of the cluster before introducing the
+complexities of actual resources.
+
+To configure a Dummy resource, run the following command:
+
+........
+crm configure primitive p0 Dummy
+........
+
+This creates a new resource, gives it the name `p0` and sets the
+agent for the resource to be the `Dummy` agent.
+
+`crm status` should now show the `p0` resource as started on one
+of the cluster nodes:
+
+........
+# crm status
+Last updated: Wed Jul 2 21:49:26 2014
+Last change: Wed Jul 2 21:49:19 2014
+Stack: corosync
+Current DC: alice (2) - partition with quorum
+Version: 1.1.11-c3f1a7f
+2 Nodes configured
+1 Resources configured
+
+
+Online: [ alice bob ]
+
+ p0 (ocf::heartbeat:Dummy): Started alice
+........
+
+The resource can be stopped or started using the `resource start` and
+`resource stop` commands:
+
+........
+crm resource stop p0
+crm resource start p0
+........