diff options
Diffstat (limited to 'doc')
95 files changed, 31979 insertions, 0 deletions
diff --git a/doc/bootstrap-howto.md b/doc/bootstrap-howto.md new file mode 100644 index 0000000..dac3eec --- /dev/null +++ b/doc/bootstrap-howto.md @@ -0,0 +1,206 @@ +# How to use the Bootstrap commands + +## Introduction + +`crmsh` includes a set of cluster bootstrapping commands, both for +setting up an initial cluster, adding and removing nodes in the +cluster and for setting up geo clusters including arbitrators. + +This document is a simplified guide to using these commands. There are +a lot of optional features that won't be fully covered by this guide, +but it should serve as a basic introduction to the bootstrap commands. + +*Note:* These commands currently work correctly only on SUSE Linux + Enterprise or openSUSE, and only if the `csync2` command is installed + on all cluster nodes. For users of other distributions, please see + the documentation included with your operating system. + +## Commands + +First, here is the list of commands and a brief description of each +one. Each command is available in the `crm cluster` namespace, so to +run the `init` command, either call `crm cluster init` from the shell +command line or navigate to the `cluster` level in the interactive +`crm` shell and call the `init` command directly: + +* `init` - Initialize a new cluster from scratch. +* `add` - Add a node to the current cluster. +* `join` - Add the current node to a cluster. +* `remove` - Remove a node from the cluster. +* `geo-init` - Create a new geo cluster with the current cluster as its first member. +* `geo-init-arbitrator` - Make the current node a geo cluster arbitrator. +* `geo-join` - Join the current cluster to an existing geo cluster. + +## Initializing a basic cluster + +For the full documentation of the `init` command, see +`crm help cluster init` in the interactive shell, or refer to the +online documentation at [crmsh.github.io](https://crmsh.github.io/). + +### Using csync2 to synchronize configuration files + +By default, the bootstrap commands make some assumptions about the +configuration to apply in order to simplify the process. One such +assumption is that the `csync2` command is installed and available for +use to synchronize the cluster configuration files across the +cluster. When initializing the basic cluster, `init` will configure +SSH access to all cluster nodes, open the necessary ports in the +firewall, and configure `csync2` so that configuration files can be +mirrored securely across the cluster. + +### Configuring SBD + +`SBD` is the storage based fencing mechanism recommended for use with +pacemaker. Using a storage based fencing method simplifies +configuration, as access to external hardware such as a lights-out +device or UPS doesn't have to be configured, and nodes can self-fence +if they detect reduced connectivity and loss of quorum. + +`init` will optionally configure SBD for you. To do this, pass the +device to use as the SBD shared storage device using the +`--sbd-device=<device>` argument. It is also possible to configure +both SBD and a shared storage device formatted with the OCFS2 file +system, using the `--partition-device=<device>` argument. To use this +option, enable the `ocfs2` template using `-t ocfs2`. + +### Basic Example + +This command line when run on the uninitialized node `alice` will +configure and start a basic cluster on the three nodes `alice`, `bob` +and `carol`. + +``` +init --name basic-cluster --nodes "alice bob carol" +``` + +The `--name` argument is optional for regular clusters, but required +when configuring a geo cluster. + +### Non-interactive configuration + +To run the initialization steps non-interactively, pass the `-y` or +`--yes` flag to `init`. The default option will be chosen wherever the +command would otherwise have prompted for user input. If no default +option is available and user input is required, the command will +abort. + +### Configuring shared storage + +To configure shared storage using the `init` command, make sure that +you have a storage device accessible from all cluster nodes. This can +be an iSCSI device provided by a SAN, or a shared storage volume as +provided by your virtualization platform. To partition this device +into two volumes, one for use by SBD and one as a shared OCFS2 device, +use a command line like the following: + +``` +init --name storage-cluster --nodes "alice bob carol" -t ocfs2 -p /dev/shared +``` + +### Configuring an administration IP + +To immediately configure a virtual IP address resource in the cluster, +use the `-A` argument: `init -A 1.2.3.4`. + +The common use case for this virtual IP is to have a single point of +entry to [Hawk](https://hawk-ui.github.io), the cluster web +interface. It is also useful as a first example of a cluster +resource. + +## Adding a new node to a cluster + +There are two commands for adding a node to a cluster. When running +the command from one of the existing cluster nodes, use the `add` +form. For example, if there is an existing cluster consisting of the +nodes `alice` and `bob`, the following command will add `carol` as the +third node in the cluster when run from `alice` or `bob`: + +``` +alice# crm cluster add carol +``` + +It is also possible to add `carol` to the cluster from `carol` +directly, using the `join` form: + +``` +carol# crm cluster join -c alice +``` + +Note that `join` takes an argument `-c <node>`. + +## Removing a node from a cluster + +To remove a node from the cluster, run + +``` +crm cluster remove <node> +``` + +To remove the last node in a cluster (thereby destroying the cluster), +it is required to pass the `--force` flag to `crm`: + +``` +crm --force cluster remove $(hostname) +``` + +## Creating a geo cluster + +Once you have a cluster up and running and you made sure to give it a +sensible name using `--name` (or by editing `corosync.conf` on all +cluster nodes and restarting the cluster), you can turn that cluster +into the first member in a geo cluster. Geo clusters are managed by +the `booth` daemon, so to use these commands, `booth` needs to be +installed on all cluster nodes. + +The `geo-init` command takes as its arguments a complete description +of the geo cluster. This is because `booth` does not share its +configuration across the cluster, instead each cluster node in each +cluster needs to have a copy of the `booth` configuration. + +As an example, we will configure a geo cluster consisting of five +nodes in total: The nodes `alice` and `bob` are members of the +`amsterdam` cluster. `carol` and `dave` are members of the `london` +cluster. Finally, `eve` is the arbitrator node located at a third +site. The `amsterdam` cluster is identified by the virtual IP +`192.168.100.8`, while the `london` cluster is identified by the +virtual IP `192.168.100.9`. + +The `geo-init` command will configure these virtual IPs in each +cluster, so there is no need to configure them before-hand. + +This geo cluster will share a single ticket, called `mcguffin`. + +To create this configuration, run + +``` +crm cluster geo-init \ + --arbitrator eve \ + --tickets mcguffin \ + --clusters "amsterdam=192.168.100.8 london=192.168.100.9" +``` + +This will configure both the required cluster resources and the booth +daemon itself in the initial cluster. + +## Adding an arbitrator to a geo cluster + +This example uses the same basic setup as the `geo-init` example. + +To configure the arbitrator `eve`, run the `geo-init-arbitrator` +command on `eve`, passing the cluster IP of the existing `amsterdam` +geo cluster member: + +``` +crm cluster geo-init-arbitrator \ + --cluster-node 192.168.100.8 +``` + +## Adding a second cluster to a geo cluster + +To add the `london` cluster to the existing geo cluster described in +the previous two sections, run the `geo-join` command from one of the +nodes in the cluster: + +``` +crm cluster geo-join --cluster-node 192.168.100.8 +``` diff --git a/doc/bootstrap-todo.md b/doc/bootstrap-todo.md new file mode 100644 index 0000000..3db7136 --- /dev/null +++ b/doc/bootstrap-todo.md @@ -0,0 +1,56 @@ +# Bootstrap TODO + +(inherited from the bootstrap project) + +## Unclear Responsibility + +These may be in purview of ha-cluster-bootstrap, or may be in appliance image: + +* install debuginfo packages +* enable coredumps + + +## General / Random + +* csync2_remote assumes there's only one group in csync2.cfg, or, more to + the point, will only add new hosts to the first group. +* Likewise, ssh_merge grabs all hosts regardless of what group they're in + (although this is probably fine) +* get rid of curses junk in log file (fix ENV term) +* Multi-device SBD (use multiple -s args) +* Start new node on standby + # crm configure node node-1 attributes standby="on" + # crm node clearstate node-1 (requires confirmation) + - start new node, unbelievably it works! + # crm node online node-1 +* don't error to log if log not started +* is "partx -a <device>" any sort of sane replacement for partprobe? +* Use ssh-copy-id instead of manual fiddling with authorized_keys? + + +## STONITH Config + +* See https://bugzilla.novell.com/show_bug.cgi?id=722405 for stonith timeout suggestions + + +## Template Mode + +Generally specific to OCFS2 template ATM, as that's the only one extant. + +* Very long path to partition (/dev/disk/by-path/... for iSCSI) means we + can't determine paths to new partitions, thanks to bnc#722959. Unclear + if this will be fixed for SP2. +* /dev/disk/by-id/dm-name paths are unreliable (at least for determining + partitions after carving the device up). +* Probably need to prompt user for new partitions after carving, should + they not be found (FFS). +* ocfs2 template is not the same as Hawk's. Consider enhancing + ha-cluster-bootstrap so it uses Hawk's templates directly rather than using + its own. +* Ensure required RPMs are installed when running template (they're just + Recommends in the spec) +* Specifying sbd without ocfs2 partition may be incompatible with ocfs2 + template (need to test) +* block device size " blockdev --getsz" etc. (when making OCFS2 partition + with "-T vmstore") + diff --git a/doc/crm.8.adoc b/doc/crm.8.adoc new file mode 100644 index 0000000..da9fb38 --- /dev/null +++ b/doc/crm.8.adoc @@ -0,0 +1,5102 @@ +:man source: crm +:man version: 4.6.0 +:man manual: crmsh documentation + +crm(8) +====== + +NAME +---- +crm - Pacemaker command line interface for configuration and management + + +SYNOPSIS +-------- +*crm* [OPTIONS] [SUBCOMMAND ARGS...] + + +[[topics_Description,Program description]] +DESCRIPTION +----------- +The `crm` shell is a command-line based cluster configuration and +management tool. Its goal is to assist as much as possible with the +configuration and maintenance of Pacemaker-based High Availability +clusters. + +For more information on Pacemaker itself, see http://clusterlabs.org/. + +`crm` works both as a command-line tool to be called directly from the +system shell, and as an interactive shell with extensive tab +completion and help. + +The primary focus of the `crm` shell is to provide a simplified and +consistent interface to Pacemaker, but it also provides tools for +managing the creation and configuration of High Availability clusters +from scratch. To learn more about this aspect of `crm`, see the +`cluster` section below. + +The `crm` shell can be used to manage every aspect of configuring and +maintaining a cluster. It provides a simplified line-based syntax on +top of the XML configuration format used by Pacemaker, commands for +starting and stopping resources, tools for exploring the history of a +cluster including log scraping and a set of cluster scripts useful for +automating the setup and installation of services on the cluster +nodes. + +The `crm` shell is line oriented: every command must start and finish +on the same line. It is possible to use a continuation character (+\+) +to write one command in two or more lines. The continuation character +is commonly used when displaying configurations. + +[[topics_CommandLine,Command line options]] +OPTIONS +------- +*-f, --file*='FILE':: + Load commands from the given file. If a dash +-+ is used in place + of a file name, `crm` will read commands from the shell standard + input (`stdin`). + +*-c, --cib*='CIB':: + Start the session using the given shadow CIB file. + Equivalent to +cib use <CIB>+. + +*-D, --display=*'OUTPUT_TYPE':: + Choose one of the output options: +plain+, +color-always+, +color+, + or +uppercase+. The default is +color+ if the terminal emulation + supports colors. Otherwise, +plain+ is used. + +*-F, --force*:: + Make `crm` proceed with applying changes where it would normally + ask the user to confirm before proceeding. This option is mainly + useful in scripts, and should be used with care. + +*-w, --wait*:: + Make `crm` wait for the cluster transition to finish (for the + changes to take effect) after each processed line. + +*-H, --history*='DIR|FILE|SESSION':: + A directory or file containing a cluster report to load + into the `history` commands, or the name of a previously + saved history session. + +*-h, --help*:: + Print help page. + +*--version*:: + Print crmsh version and build information (Mercurial Hg changeset + hash). + +*-d, --debug*:: + Print verbose debugging information. + +*-R, --regression-tests*:: + Enables extra verbose trace logging used by the regression + tests. Logs all external calls made by crmsh. + +*--scriptdir*='DIR':: + Extra directory where crm looks for cluster scripts, or a list of + directories separated by semi-colons (e.g. +/dir1;/dir2;etc.+). + +*-o, --opt*='OPTION=VALUE':: + Set crmsh option temporarily. If the options are saved using + +options save+ then the value passed here will also be saved. + Multiple options can be set by using +-o+ multiple times. + +[[topics_Introduction,Introduction]] +== Introduction + +This section of the user guide covers general topics about the user +interface and describes some of the features of `crmsh` in detail. + +[[topics_Introduction_Interface,User interface]] +=== User interface + +The main purpose of `crmsh` is to provide a simple yet powerful +interface to the cluster stack. There are two main modes of operation +with the user interface of `crmsh`: + +* Command line (single-shot) use - Use `crm` as a regular UNIX command + from your usual shell. `crm` has full bash completion built in, so + using it in this manner should be as comfortable and familiar as + using any other command-line tool. + +* Interactive mode - By calling `crm` without arguments, or by calling + it with only a sublevel as argument, `crm` enters the interactive + mode. In this mode, it acts as its own command shell, which + remembers which sublevel you are currently in and allows for rapid + and convenient execution of multiple commands within the same + sublevel. This mode also has full tab completion, as well as + built-in interactive help and syntax highlighting. + +Here are a few examples of using `crm` both as a command-line tool and +as an interactive shell: + +.Command line (one-shot) use: +........ +# crm resource stop www_app +........ + +.Interactive use: +........ +# crm +crm(live)# resource +crm(live)resource# unmanage tetris_1 +crm(live)resource# up +crm(live)# node standby node4 +........ + +.Cluster configuration: +........ +# crm configure<<EOF + # + # resources + # + primitive disk0 iscsi \ + params portal=192.168.2.108:3260 target=iqn.2008-07.com.suse:disk0 + primitive fs0 Filesystem \ + params device=/dev/disk/by-label/disk0 directory=/disk0 fstype=ext3 + primitive internal_ip IPaddr params ip=192.168.1.101 + primitive apache apache \ + params configfile=/disk0/etc/apache2/site0.conf + primitive apcfence stonith:apcsmart \ + params ttydev=/dev/ttyS0 hostlist="node1 node2" \ + op start timeout=60s + primitive pingd ocf:pacemaker:ping \ + params name=pingd dampen=5s multiplier=100 host_list="r1 r2" + # + # monitor apache and the UPS + # + monitor apache 60s:30s + monitor apcfence 120m:60s + # + # cluster layout + # + group internal_www \ + disk0 fs0 internal_ip apache + clone fence apcfence \ + meta globally-unique=false clone-max=2 clone-node-max=1 + clone conn pingd \ + meta globally-unique=false clone-max=2 clone-node-max=1 + location node_pref internal_www \ + rule 50: #uname eq node1 \ + rule pingd: defined pingd + # + # cluster properties + # + property stonith-enabled=true + commit +EOF +........ + +The `crm` interface is hierarchical, with commands organized into +separate levels by functionality. To list the available levels and +commands, either execute +help <level>+, or, if at the top level of +the shell, simply typing `help` will provide an overview of all +available levels and commands. + +The +(live)+ string in the `crm` prompt signifies that the current CIB +in use is the cluster live configuration. It is also possible to +work with so-called <<topics_Features_Shadows,shadow CIBs>>. These are separate, inactive +configurations stored in files, that can be applied and thereby +replace the live configuration at any time. + +[[topics_Introduction_Completion,Tab completion]] +=== Tab completion + +The `crm` makes extensive use of tab completion. The completion +is both static (i.e. for `crm` commands) and dynamic. The latter +takes into account the current status of the cluster or +information from installed resource agents. Sometimes, completion +may also be used to get short help on resource parameters. Here +are a few examples: + +............... +crm(live)resource# <TAB><TAB> +ban demote maintenance param scores trace +cd failcount manage promote secret unmanage +cleanup help meta quit start untrace +clear locate move refresh status up +constraints ls operations restart stop utilization + +crm(live)configure# primitive fence-1 <TAB><TAB> +lsb: ocf: service: stonith: systemd: + +crm(live)configure# primitive fence-1 stonith:<TAB><TAB> +apcmaster external/ippower9258 fence_legacy +apcmastersnmp external/kdumpcheck ibmhmc +apcsmart external/libvirt ipmilan + +crm(live)configure# primitive fence-1 stonith:ipmilan params <TAB><TAB> +auth= hostname= ipaddr= login= password= port= priv= + +crm(live)configure# primitive fence-1 stonith:ipmilan params auth=<TAB><TAB> +auth* (string) + The authorization type of the IPMI session ("none", "straight", "md2", or "md5") +............... + +`crmsh` also comes with bash completion usable directly from the +system shell. This should be installed automatically with the command +itself. + +[[topics_Introduction_Shorthand,Shorthand syntax]] +=== Shorthand syntax + +When using the `crm` shell to manage clusters, you will end up typing +a lot of commands many times over. Clear command names like ++configure+ help in understanding and learning to use the cluster +shell, but is easy to misspell and is tedious to type repeatedly. The +interactive mode and tab completion both help with this, but the `crm` +shell also has the ability to understand a variety of shorthand +aliases for all of the commands. + +For example, instead of typing `crm status`, you can type `crm st` or +`crm stat`. Instead of `crm configure` you can type `crm cfg` or even +`crm cf`. `crm resource` can be shorted as `crm rsc`, and so on. + +The exact list of accepted aliases is too long to print in full, but +experimentation and typos should help in discovering more of them. + +[[topics_Features,Features]] +== Features + +The feature set of crmsh covers a wide range of functionality, and +understanding how and when to use the various features of the shell +can be difficult. This section of the guide describes some of the +features and use cases of `crmsh` in more depth. The intention is to +provide a deeper understanding of these features, but also to serve as +a guide to using them. + +[[topics_Features_Shadows,Shadow CIB usage]] +=== Shadow CIB usage + +A Shadow CIB is a normal cluster configuration stored in a file. +They may be manipulated in much the same way as the _live_ CIB, with +the key difference that changes to a shadow CIB have no effect on the +actual cluster resources. An administrator may choose to apply any of +them to the cluster, thus replacing the running configuration with the +one found in the shadow CIB. + +The `crm` prompt always contains the name of the configuration which +is currently in use, or the string _live_ if using the live cluster +configuration. + +When editing the configuration in the `configure` level, no changes +are actually applied until the `commit` command is executed. It is +possible to start editing a configuration as usual, but instead of +committing the changes to the active CIB, save them to a shadow CIB. + +The following example `configure` session demonstrates how this can be +done: +............... +crm(live)configure# cib new test-2 +INFO: test-2 shadow CIB created +crm(test-2)configure# commit +............... + +[[topics_Features_Checks,Configuration semantic checks]] +=== Configuration semantic checks + +Resource definitions may be checked against the meta-data +provided with the resource agents. These checks are currently +carried out: + +- are required parameters set +- existence of defined parameters +- timeout values for operations + +The parameter checks are obvious and need no further explanation. +Failures in these checks are treated as configuration errors. + +The timeouts for operations should be at least as long as those +recommended in the meta-data. Too short timeout values are a +common mistake in cluster configurations and, even worse, they +often slip through if cluster testing was not thorough. Though +operation timeouts issues are treated as warnings, make sure that +the timeouts are usable in your environment. Note also that the +values given are just _advisory minimum_---your resources may +require longer timeouts. + +User may tune the frequency of checks and the treatment of errors +by the <<cmdhelp_options_check-frequency,`check-frequency`>> and +<<cmdhelp_options_check-mode,`check-mode`>> preferences. + +Note that if the +check-frequency+ is set to +always+ and the ++check-mode+ to +strict+, errors are not tolerated and such +configuration cannot be saved. + +[[topics_Features_Templates,Configuration templates]] +=== Configuration templates + +.Deprecation note +**************************** +Configuration templates have been deprecated in favor of the more +capable `cluster scripts`. To learn how to use cluster scripts, see +the dedicated documentation on the `crmsh` website at +http://crmsh.github.io/, or in the <<cmdhelp_script,Script section>>. +**************************** + +Configuration templates are ready made configurations created by +cluster experts. They are designed in such a way so that users +may generate valid cluster configurations with minimum effort. +If you are new to Pacemaker, templates may be the best way to +start. + +We will show here how to create a simple yet functional Apache +configuration: +............... +# crm configure +crm(live)configure# template +crm(live)configure template# list templates +apache filesystem virtual-ip +crm(live)configure template# new web <TAB><TAB> +apache filesystem virtual-ip +crm(live)configure template# new web apache +INFO: pulling in template apache +INFO: pulling in template virtual-ip +crm(live)configure template# list +web2-d web2 vip2 web3 vip web +............... + +We enter the `template` level from `configure`. Use the `list` +command to show templates available on the system. The `new` +command creates a configuration from the +apache+ template. You +can use tab completion to pick templates. Note that the apache +template depends on a virtual IP address which is automatically +pulled along. The `list` command shows the just created +web+ +configuration, among other configurations (I hope that you, +unlike me, will use more sensible and descriptive names). + +The `show` command, which displays the resulting configuration, +may be used to get an idea about the minimum required changes +which have to be done. All +ERROR+ messages show the line numbers +in which the respective parameters are to be defined: +............... +crm(live)configure template# show +ERROR: 23: required parameter ip not set +ERROR: 61: required parameter id not set +ERROR: 65: required parameter configfile not set +crm(live)configure template# edit +............... + +The `edit` command invokes the preferred text editor with the ++web+ configuration. At the top of the file, the user is advised +how to make changes. A good template should require from the user +to specify only parameters. For example, the +web+ configuration +we created above has the following required and optional +parameters (all parameter lines start with +%%+): +............... +$ grep -n ^%% ~/.crmconf/web +23:%% ip +31:%% netmask +35:%% lvs_support +61:%% id +65:%% configfile +71:%% options +76:%% envfiles +............... + +These lines are the only ones that should be modified. Simply +append the parameter value at the end of the line. For instance, +after editing this template, the result could look like this (we +used tabs instead of spaces to make the values stand out): +............... +$ grep -n ^%% ~/.crmconf/web +23:%% ip 192.168.1.101 +31:%% netmask +35:%% lvs_support +61:%% id websvc +65:%% configfile /etc/apache2/httpd.conf +71:%% options +76:%% envfiles +............... + +As you can see, the parameter line format is very simple: +............... +%% <name> <value> +............... + +After editing the file, use `show` again to display the +configuration: +............... +crm(live)configure template# show +primitive virtual-ip IPaddr \ + params ip=192.168.1.101 +primitive apache apache \ + params configfile="/etc/apache2/httpd.conf" +monitor apache 120s:60s +group websvc \ + apache virtual-ip +............... + +The target resource of the apache template is a group which we +named +websvc+ in this sample session. + +This configuration looks exactly as you could type it at the +`configure` level. The point of templates is to save you some +typing. It is important, however, to understand the configuration +produced. + +Finally, the configuration may be applied to the current +crm configuration (note how the configuration changed slightly, +though it is still equivalent, after being digested at the +`configure` level): +............... +crm(live)configure template# apply +crm(live)configure template# cd .. +crm(live)configure# show +node xen-b +node xen-c +primitive apache apache \ + params configfile="/etc/apache2/httpd.conf" \ + op monitor interval=120s timeout=60s +primitive virtual-ip IPaddr \ + params ip=192.168.1.101 +group websvc apache virtual-ip +............... + +Note that this still does not commit the configuration to the CIB +which is used in the shell, either the running one (+live+) or +some shadow CIB. For that you still need to execute the `commit` +command. + +To complete our example, we should also define the preferred node +to run the service: + +............... +crm(live)configure# location websvc-pref websvc 100: xen-b +............... + +If you are not happy with some resource names which are provided +by default, you can rename them now: + +............... +crm(live)configure# rename virtual-ip intranet-ip +crm(live)configure# show +node xen-b +node xen-c +primitive apache apache \ + params configfile="/etc/apache2/httpd.conf" \ + op monitor interval=120s timeout=60s +primitive intranet-ip IPaddr \ + params ip=192.168.1.101 +group websvc apache intranet-ip +location websvc-pref websvc 100: xen-b +............... + +To summarize, working with templates typically consists of the +following steps: + +- `new`: create a new configuration from templates +- `edit`: define parameters, at least the required ones +- `show`: see if the configuration is valid +- `apply`: apply the configuration to the `configure` level + +[[topics_Features_Testing,Resource testing]] +=== Resource testing + +The amount of detail in a cluster makes all configurations prone +to errors. By far the largest number of issues in a cluster is +due to bad resource configuration. The shell can help quickly +diagnose such problems. And considerably reduce your keyboard +wear. + +Let's say that we entered the following configuration: +............... +node xen-b +node xen-c +node xen-d +primitive fencer stonith:external/libvirt \ + params hypervisor_uri="qemu+tcp://10.2.13.1/system" \ + hostlist="xen-b xen-c xen-d" \ + op monitor interval=2h +primitive svc Xinetd \ + params service=systat \ + op monitor interval=30s +primitive intranet-ip IPaddr2 \ + params ip=10.2.13.100 \ + op monitor interval=30s +primitive apache apache \ + params configfile="/etc/apache2/httpd.conf" \ + op monitor interval=120s timeout=60s +group websvc apache intranet-ip +location websvc-pref websvc 100: xen-b +............... + +Before typing `commit` to submit the configuration to the cib we +can make sure that all resources are usable on all nodes: +............... +crm(live)configure# rsctest websvc svc fencer +............... + +It is important that resources being tested are not running on +any nodes. Otherwise, the `rsctest` command will refuse to do +anything. Of course, if the current configuration resides in a +CIB shadow, then a `commit` is irrelevant. The point being that +resources are not running on any node. + +.Note on stopping all resources +**************************** +Alternatively to not committing a configuration, it is also +possible to tell Pacemaker not to start any resources: + +............... +crm(live)configure# property stop-all-resources=yes +............... +Almost none---resources of class stonith are still started. But +shell is not as strict when it comes to stonith resources. +**************************** + +Order of resources is significant insofar that a resource depends +on all resources to its left. In most configurations, it's +probably practical to test resources in several runs, based on +their dependencies. + +Apart from groups, `crm` does not interpret constraints and +therefore knows nothing about resource dependencies. It also +doesn't know if a resource can run on a node at all in case of an +asymmetric cluster. It is up to the user to specify a list of +eligible nodes if a resource is not meant to run on every node. + +[[topics_Features_Security,Access Control Lists (ACL)]] +=== Access Control Lists (ACL) + +.Note on ACLs in Pacemaker 1.1.12 +**************************** +The support for ACLs has been revised in Pacemaker version 1.1.12 and +up. Depending on which version you are using, the information in this +section may no longer be accurate. Look for the `acl_target` +configuration element for more details on the new syntax. +**************************** + +By default, the users from the +haclient+ group have full access +to the cluster (or, more precisely, to the CIB). Access control +lists allow for finer access control to the cluster. + +Access control lists consist of an ordered set of access rules. +Each rule allows read or write access or denies access +completely. Rules are typically combined to produce a specific +role. Then, users may be assigned a role. + +For instance, this is a role which defines a set of rules +allowing management of a single resource: + +............... +role bigdb_admin \ + write meta:bigdb:target-role \ + write meta:bigdb:is-managed \ + write location:bigdb \ + read ref:bigdb +............... + +The first two rules allow modifying the +target-role+ and ++is-managed+ meta attributes which effectively enables users in +this role to stop/start and manage/unmanage the resource. The +constraints write access rule allows moving the resource around. +Finally, the user is granted read access to the resource +definition. + +For proper operation of all Pacemaker programs, it is advisable +to add the following role to all users: + +............... +role read_all \ + read cib +............... + +For finer grained read access try with the rules listed in the +following role: + +............... +role basic_read \ + read node attribute:uname \ + read node attribute:type \ + read property \ + read status +............... + +It is however possible that some Pacemaker programs (e.g. +`ptest`) may not function correctly if the whole CIB is not +readable. + +Some of the ACL rules in the examples above are expanded by the +shell to XPath specifications. For instance, ++meta:bigdb:target-role+ expands to: + +........ +//primitive[@id='bigdb']/meta_attributes/nvpair[@name='target-role'] +........ + +You can see the expansion by showing XML: + +............... +crm(live) configure# show xml bigdb_admin +... +<acls> + <acl_role id="bigdb_admin"> + <write id="bigdb_admin-write" + xpath="//primitive[@id='bigdb']/meta_attributes/nvpair[@name='target-role']"/> +............... + +Many different XPath expressions can have equal meaning. For +instance, the following two are equal, but only the first one is +going to be recognized as shortcut: + +............... +//primitive[@id='bigdb']/meta_attributes/nvpair[@name='target-role'] +//resources/primitive[@id='bigdb']/meta_attributes/nvpair[@name='target-role'] +............... + +XPath is a powerful language, but you should try to keep your ACL +xpaths simple and the builtin shortcuts should be used whenever +possible. + +[[topics_Features_Resourcesets,Syntax: Resource sets]] +=== Syntax: Resource sets + +Using resource sets can be a bit confusing unless one knows the +details of the implementation in Pacemaker as well as how to interpret +the syntax provided by `crmsh`. + +Three different types of resource sets are provided by `crmsh`, and +each one implies different values for the two resource set attributes, ++sequential+ and +require-all+. + ++sequential+:: + If false, the resources in the set do not depend on each other + internally. Setting +sequential+ to +true+ implies a strict order of + dependency within the set. + ++require-all+:: + If false, only one resource in the set is required to fulfil the + requirements of the set. The set of A, B and C with +require-all+ + set to +false+ is be read as "A OR B OR C" when its dependencies + are resolved. + +The three types of resource sets modify the attributes in the +following way: + +1. Implicit sets (no brackets). +sequential=true+, +require-all=true+ +2. Parenthesis set (+(+ ... +)+). +sequential=false+, +require-all=true+ +3. Bracket set (+[+ ... +]+). +sequential=false+, +require-all=false+ + +To create a set with the properties +sequential=true+ and ++require-all=false+, explicitly set +sequential+ in a bracketed set, ++[ A B C sequential=true ]+. + +To create multiple sets with both +sequential+ and +require-all+ set to +true, explicitly set +sequential+ in a parenthesis set: ++A B ( C D sequential=true )+. + +[[topics_Features_AttributeListReferences,Syntax: Attribute list references]] +=== Syntax: Attribute list references + +Attribute lists are used to set attributes and parameters for +resources, constraints and property definitions. For example, to set +the virtual IP used by an +IPAddr2+ resource the attribute +ip+ can be +set in an attribute list for that resource. + +Attribute lists can have identifiers that name them, and other +resources can reuse the same attribute list by referring to that name +using an +$id-ref+. For example, the following statement defines a +simple dummy resource with an attribute list which sets the parameter ++state+ to the value 1 and sets the identifier for the attribute list +to +on-state+: + +.............. +primitive dummy-1 Dummy params $id=on-state state=1 +.............. + +To refer to this attribute list from a different resource, refer to +the +on-state+ name using an id-ref: + +.............. +primitive dummy-2 Dummy params $id-ref=on-state +.............. + +The resource +dummy-2+ will now also have the parameter +state+ set to the value 1. + +[[topics_Features_AttributeReferences,Syntax: Attribute references]] +=== Syntax: Attribute references + +In some cases, referencing complete attribute lists is too +coarse-grained, for example if two different parameters with different +names should have the same value set. Instead of having to copy the +value in multiple places, it is possible to create references to +individual attributes in attribute lists. + +To name an attribute in order to be able to refer to it later, prefix +the attribute name with a +$+ character (as seen above with the +special names +$id+ and +$id-ref+: + +............ +primitive dummy-1 Dummy params $state=1 +............ + +The identifier +state+ can now be used to refer to this attribute from other +primitives, using the +@<id>+ syntax: + +............ +primitive dummy-2 Dummy params @state +............ + +In some cases, using the attribute name as the identifier doesn't work +due to name clashes. In this case, the syntax +$<id>:<name>=<value>+ +can be used to give the attribute a different identifier: + +............ +primitive dummy-1 params $dummy-state-on:state=1 +primitive dummy-2 params @dummy-state-on +............ + +There is also the possibility that two resources both use the same +attribute value but with different names. For example, a web server +may have a parameter +server_ip+ for setting the IP address where it +listens for incoming requests, and a virtual IP resource may have a +parameter called +ip+ which sets the IP address it creates. To +configure these two resources with an IP without repeating the value, +the reference can be given a name using the syntax +@<id>:<name>+. + +Example: +............ +primitive virtual-ip IPaddr2 params $vip:ip=192.168.1.100 +primitive webserver apache params @vip:server_ip +............ + +[[topics_Syntax_RuleExpressions,Syntax: Rule expressions]] +=== Syntax: Rule expressions + +Many of the configuration commands in `crmsh` now support the use of +_rule expressions_, which can influence what attributes apply to a +resource or under which conditions a constraint is applied, depending +on changing conditions like date, time, the value of attributes and +more. + +Here is an example of a simple rule expression used to apply a +a different resource parameter on the node named `node1`: + +.............. +primitive my_resource Special \ + params 2: rule #uname eq node1 interface=eth1 \ + params 1: interface=eth0 +.............. + +This primitive resource has two lists of parameters with descending +priority. The parameter list with the highest priority is applied +first, but only if the rule expressions for that parameter list all +apply. In this case, the rule `#uname eq node1` limits the parameter +list so that it is only applied on `node1`. + +Note that rule expressions are not terminated and are immediately +followed by the data to which the rule is applied. In this case, the +name-value pair `interface=eth1`. + +Rule expressions can contain multiple expressions connected using the +boolean operator `or` and `and`. The full syntax for rule expressions +is listed below. + +.............. +rules :: + rule [id_spec] [$role=<role>] <score>: <expression> + [rule [id_spec] [$role=<role>] <score>: <expression> ...] + +id_spec :: $id=<id> | $id-ref=<id> +score :: <number> | <attribute> | [-]inf +expression :: <simple_exp> [<bool_op> <simple_exp> ...] +bool_op :: or | and +simple_exp :: <attribute> [type:]<binary_op> <value> + | <unary_op> <attribute> + | date <date_expr> +type :: <string> | <version> | <number> +binary_op :: lt | gt | lte | gte | eq | ne +unary_op :: defined | not_defined + +date_expr :: lt <end> + | gt <start> + | in start=<start> end=<end> + | in start=<start> <duration> + | spec <date_spec> +duration|date_spec :: + hours=<value> + | monthdays=<value> + | weekdays=<value> + | yearsdays=<value> + | months=<value> + | weeks=<value> + | years=<value> + | weekyears=<value> + | moon=<value> +.............. + +[[topics_Lifetime,Lifetime parameter format]] +== Lifetime parameter format + +Lifetimes can be specified in the ISO 8601 time format or the ISO 8601 +duration format. To distinguish between months and minutes, use the PT +prefix before specifying minutes. The duration format is one of ++PnYnMnDTnHnMnS+, +PnW+, +P<date>T<time>+. + +P = duration. Y = year. M = month. W = week. D = day. T = time. H = +hour. M = minute. S = second. + +Examples: +................. +PT5M = 5 minutes later. +3D = 3 days later. +PT1H = 1 hour later. +................. + +The cluster checks lifetimes at an interval defined by the +cluster-recheck-interval property (default 15 minutes). + + +[[topics_Reference,Command reference]] +== Command reference + +The commands are structured to be compatible with the shell command +line. Sometimes, the underlying Pacemaker grammar uses characters that +have special meaning in bash, that will need to be quoted. This +includes the hash or pound sign (`#`), single and double quotes, and +any significant whitespace. + +Whitespace is also significant when assigning values, meaning that ++key=value+ is different from +key = value+. + +Commands can be referenced using short-hand as long as the short-hand +is unique. This can be either a prefix of the command name or a prefix +string of characters found in the name. + +For example, +status+ can be abbreviated as +st+ or +su+, and ++configure+ as +conf+ or +cfg+. + +The syntax for the commands is given below in an informal, BNF-like +grammar. + +* `<value>` denotes a string. +* `[value]` means that the construct is optional. +* The ellipsis (`...`) signifies that the previous construct may be + repeated. +* `first|second` means either first or second. +* The rest are literals (strings, `:`, `=`). + +[[cmdhelp_root_status,Cluster status]] +=== `status` + +Show cluster status. The status is displayed by `crm_mon`. Supply +additional arguments for more information or different format. +See `crm_mon(8)` for more details. + +Example: +............... +status +status simple +status full +............... + +Usage: +............... +status [<option> ...] + +option :: full + | bynode + | inactive + | ops + | timing + | failcounts + | verbose + | quiet + | html + | xml + | simple + | tickets + | noheaders + | detail + | brief +............... + +[[cmdhelp_root_verify,Verify cluster status]] +=== `verify` + +Performs basic checks for the cluster configuration and +current status, reporting potential issues. + +See `crm_verify(8)` and `crm_simulate(8)` for more details. + +Example: +............... +verify +verify scores +............... + +Usage: +............... +verify [scores] +............... + +[[cmdhelp_cluster,Cluster setup and management]] +=== `cluster` - Cluster setup and management + +Whole-cluster configuration management with High Availability +awareness. + +The commands on the cluster level allows configuration and +modification of the underlying cluster infrastructure, and also +supplies tools to do whole-cluster systems management. + +These commands enable easy installation and maintenance of a HA +cluster, by providing support for package installation, configuration +of the cluster messaging layer, file system setup and more. + +[[cmdhelp_cluster_copy,Copy file to other cluster nodes]] +==== `copy` + +Copy file to other cluster nodes. + +Copies the given file to all other nodes unless given a +list of nodes to copy to as argument. + +Usage: +............... +copy <filename> [nodes ...] +............... + +Example: +............... +copy /etc/motd +............... + +[[cmdhelp_cluster_diff,Diff file across cluster]] +==== `diff` + +Displays the difference, if any, between a given file +on different nodes. If the second argument is `--checksum`, +a checksum of the file will be calculated and displayed for +each node. + +Usage: +............... +diff <file> [--checksum] [nodes...] +............... + +Example: +............... +diff /etc/crm/crm.conf node2 +diff /etc/resolv.conf --checksum +............... + +[[cmdhelp_cluster_disable,Disable cluster services,From Code]] +==== `disable` +See "crm cluster help disable" or "crm cluster disable --help" + +[[cmdhelp_cluster_enable,Enable cluster services,From Code]] +==== `enable` +See "crm cluster help enable" or "crm cluster enable --help + +[[cmdhelp_cluster_geo_init,Configure cluster as geo cluster,From Code]] +==== `geo-init` +See "crm cluster help geo_init" or "crm cluster geo_init --help" + +[[cmdhelp_cluster_geo_init_arbitrator,Initialize node as geo cluster arbitrator,From Code]] +==== `geo-init-arbitrator` +See "crm cluster help geo_init_arbitrator" or "crm cluster geo_init_arbitrator --help" + +[[cmdhelp_cluster_geo_join,Join cluster to existing geo cluster,From Code]] +==== `geo-join` +See "crm cluster help geo_join" or "crm cluster geo_join --help" + +[[cmdhelp_cluster_health,Cluster health check]] +==== `health` + +Runs a larger set of tests and queries on all nodes in the cluster to +verify the general system health and detect potential problems. + +Usage: +............... +health +............... + +[[cmdhelp_cluster_init,Initializes a new HA cluster,From Code]] +==== `init` +See "crm cluster help init" or "crm cluster init --help" + +[[cmdhelp_cluster_join,Join existing cluster,From Code]] +==== `join` +See "crm cluster help join" or "crm cluster join --help" + +[[cmdhelp_cluster_remove,Remove node(s) from the cluster,From Code]] +==== `remove` +See "crm cluster help remove" or "crm cluster remove --help" + +[[cmdhelp_cluster_crash_test,Cluster crash test tool set,From Code]] +==== `crash_test` +See "crm cluster help crash_test" or "crm cluster crash_test --help" + +[[cmdhelp_cluster_restart,Restart cluster services,From Code]] +==== `restart` +See "crm cluster help restart" or "crm cluster restart --help" + +[[cmdhelp_cluster_rename,Rename the cluster]] +==== `rename` + +Rename the cluster name + +Usage: +............... +rename <new_cluster_name> +............... + + +[[cmdhelp_cluster_run,Execute an arbitrary command on all nodes/specific node]] +==== `run` + +This command takes a shell statement as argument, executes that +statement on all nodes in the cluster or a specific node, +and reports the result. + +Usage: +............... +run <command> [node ...] +............... + +Example: +............... +run "cat /proc/uptime" +run "ls" node1 node2 +............... + +[[cmdhelp_cluster_start,Start cluster services,From Code]] +==== `start` +See "crm cluster help start" or "crm cluster start --help" + +[[cmdhelp_cluster_status,Cluster status check]] +==== `status` + +Reports the status for the cluster messaging layer on the local +node. + +Usage: +............... +status +............... + +[[cmdhelp_cluster_stop,Stop cluster services,From Code]] +==== `stop` +See "crm cluster help stop" or "crm cluster stop --help" + +[[cmdhelp_cluster_wait_for_startup,Wait for cluster to start]] +==== `wait_for_startup` + +Mostly useful in scripts or automated workflows, this command will +attempt to connect to the local cluster node repeatedly. The command +will keep trying until the cluster node responds, or the `timeout` +elapses. The timeout can be changed by supplying a value in seconds as +an argument. + +Usage: +........ +wait_for_startup +........ + +[[cmdhelp_script,Cluster script management]] +=== `script` - Cluster script management + +A big part of the configuration and management of a cluster is +collecting information about all cluster nodes and deploying changes +to those nodes. Often, just performing the same procedure on all nodes +will encounter problems, due to subtle differences in the +configuration. + +For example, when configuring a cluster for the first time, the +software needs to be installed and configured on all nodes before the +cluster software can be launched and configured using `crmsh`. This +process is cumbersome and error-prone, and the goal is for scripts to +make this process easier. + +Scripts are implemented using the python `parallax` package which +provides a thin wrapper on top of SSH. This allows the scripts to +function through the usual SSH channels used for system maintenance, +requiring no additional software to be installed or maintained. + +[[cmdhelp_script_json,JSON API for cluster scripts]] +==== `json` + +This command provides a JSON API for the cluster scripts, intended for +use in user interface tools that want to interact with the cluster via +scripts. + +The command takes a single argument, which should be a JSON array with +the first member identifying the command to perform. + +The output is line-based: Commands that return multiple results will +return them line-by-line, ending with a terminator value: "end". + +When providing parameter values to this command, they should be +provided as nested objects, so +virtual-ip:ip=192.168.0.5+ on the +command line becomes the JSON object ++{"virtual-ip":{"ip":"192.168.0.5"}}+. + +API: +........ +["list"] +=> [{name, shortdesc, category}] + +["show", <name>] +=> [{name, shortdesc, longdesc, category, <<steps>>}] + +<<steps>> := [{name, shortdesc], longdesc, required, parameters, steps}] + +<<params>> := [{name, shortdesc, longdesc, required, unique, advanced, + type, value, example}] + +["verify", <name>, <<values>>] +=> [{shortdesc, longdesc, text, nodes}] + +["run", <name>, <<values>>] +=> [{shortdesc, rc, output|error}] +........ + + +[[cmdhelp_script_list,List available scripts]] +==== `list` + +Lists the available scripts, sorted by category. Scripts that have the +special `Script` category are hidden by default, since they are mainly +used by other scripts or commands. To also show these, pass `all` as +argument. + +To get a flat list of script names, not sorted by category, pass +`names` as an extra argument. + +Usage: +............ +list [all] [names] +............ + +Example: +............ +list +list all names +............ + +[[cmdhelp_script_run,Run the script]] +==== `run` + +Given a list of parameter values, this command will execute the +actions specified by the cluster script. The format for the parameter +values is the same as for the `verify` command. + +Can optionally take at least two parameters: +* `nodes=<nodes>`: List of nodes that the script runs over +* `dry_run=yes|no`: If set, the script will not perform any modifications. + +Additional parameters may be available depending on the script. + +Use the `show` command to see what parameters are available. + +Usage: +............. +run <script> [args...] +............. + +Example: +............. +run apache install=true +run sbd id=sbd-1 node=node1 sbd_device=/dev/disk/by-uuid/F00D-CAFE +............. + +[[cmdhelp_script_show,Describe the script]] +==== `show` + +Prints a description and short summary of the script, with +descriptions of the accepted parameters. + +Advanced parameters are hidden by default. To show the complete list +of parameters accepted by the script, pass `all` as argument. + +Usage: +............ +show <script> [all] +............ + +Example: +............ +show virtual-ip +............ + +[[cmdhelp_script_verify,Verify the script]] +==== `verify` + +Checks the given parameter values, and returns a list +of actions that will be executed when running the script +if provided the same list of parameter values. + +Usage: +............ +verify <script> [args...] +............ + +Example: +............ +verify sbd id=sbd-1 node=node1 sbd_device=/dev/disk/by-uuid/F00D-CAFE +............ + +[[cmdhelp_corosync,Corosync management]] +=== `corosync` - Corosync management + +Corosync is the underlying messaging layer for most HA clusters. +This level provides commands for editing and managing the corosync +configuration. + +[[cmdhelp_corosync_add-node,Add a corosync node]] +==== `add-node` + +Adds a node to the corosync configuration. This is used with the `udpu` +type configuration in corosync. + +A nodeid for the added node is generated automatically. + +Note that this command assumes that only a single ring is used, and +sets only the address for ring0. + +Usage: +......... +add-node <addr> [name] +......... + +[[cmdhelp_corosync_del-node,Remove a corosync node]] +==== `del-node` + +Removes a node from the corosync configuration. The argument given is +the `ring0_addr` address set in the configuration file. + +Usage: +......... +del-node <addr> +......... + +[[cmdhelp_corosync_diff,Diffs the corosync configuration]] +==== `diff` + +Diffs the corosync configurations on different nodes. If no nodes are +given as arguments, the corosync configurations on all nodes in the +cluster are compared. + +`diff` takes an option argument `--checksum`, to display a checksum +for each file instead of calculating a diff. + +Usage: +......... +diff [--checksum] [node...] +......... + +[[cmdhelp_corosync_edit,Edit the corosync configuration]] +==== `edit` + +Opens the Corosync configuration file in an editor. + +Usage: +......... +edit +......... + +[[cmdhelp_corosync_get,Get a corosync configuration value]] +==== `get` + +Returns the value configured in `corosync.conf`, which is not +necessarily the value used in the running configuration. See `reload` +for telling corosync about configuration changes. + +The argument is the complete dot-separated path to the value. + +If there are multiple values configured with the same path, the +command returns all values for that path. For example, to get all +configured `ring0_addr` values, use this command: + +Example: +........ +get nodelist.node.ring0_addr +........ + +[[cmdhelp_corosync_log,Show the corosync log file]] +==== `log` + +Opens the log file specified in the corosync configuration file. If no +log file is configured, this command returns an error. + +The pager used can be configured either using the PAGER +environment variable or in `crm.conf`. + +Usage: +......... +log +......... + +[[cmdhelp_corosync_pull,Pulls the corosync configuration]] +==== `pull` + +Gets the corosync configuration from another node and copies +it to this node. + +Usage: +......... +pull <node> +......... + +[[cmdhelp_corosync_push,Push the corosync configuration]] +==== `push` + +Pushes the corosync configuration file on this node to +the list of nodes provided. If no target nodes are given, +the configuration is pushed to all other nodes in the cluster. + +It is recommended to use `csync2` to distribute the cluster +configuration files rather than relying on this command. + +Usage: +......... +push [node] ... +......... + +Example: +......... +push node-2 node-3 +......... + +[[cmdhelp_corosync_reload,Reload the corosync configuration]] +==== `reload` + +Tells all instances of corosync in this cluster to reload +`corosync.conf`. + +After pushing a new configuration to all cluster nodes, call this +command to make corosync use the new configuration. + +Usage: +......... +reload +......... + +[[cmdhelp_corosync_set,Set a corosync configuration value]] +==== `set` + +Sets the value identified by the given path. If the value does not +exist in the configuration file, it will be added. However, if the +section containing the value does not exist, the command will fail. + +Usage: +......... +set quorum.expected_votes 2 +......... + +[[cmdhelp_corosync_show,Display the corosync configuration]] +==== `show` + +Displays the corosync configuration on the current node. + +......... +show +......... + +[[cmdhelp_corosync_status,Display the corosync status]] +==== `status` + +Displays the corosync ring status(default), also can display quorum/qdevice/qnetd status. + +Usage: +......... +status [ring|quorum|qdevice|qnetd] +......... + +[[cmdhelp_cib,CIB shadow management]] +=== `cib` - CIB shadow management + +This level is for management of shadow CIBs. It is available both +at the top level and the `configure` level. + +All the commands are implemented using `cib_shadow(8)` and the +`CIB_shadow` environment variable. The user prompt always +includes the name of the currently active shadow or the live CIB. + +[[cmdhelp_cib_cibstatus,CIB status management and editing]] +==== `cibstatus` + +Enter edit and manage the CIB status section level. See the +<<cmdhelp_cibstatus,CIB status management section>>. + +[[cmdhelp_cib_commit,copy a shadow CIB to the cluster]] +==== `commit` + +Apply a shadow CIB to the cluster. If the shadow name is omitted +then the current shadow CIB is applied. + +Temporary shadow CIBs are removed automatically on commit. + +Usage: +............... +commit [<cib>] +............... + +[[cmdhelp_cib_delete,delete a shadow CIB]] +==== `delete` + +Delete an existing shadow CIB. + +Usage: +............... +delete <cib> +............... + +[[cmdhelp_cib_diff,diff between the shadow CIB and the live CIB]] +==== `diff` + +Print differences between the current cluster configuration and +the active shadow CIB. + +Usage: +............... +diff +............... + +[[cmdhelp_cib_import,import a CIB or PE input file to a shadow]] +==== `import` + +At times it may be useful to create a shadow file from the +existing CIB. The CIB may be specified as file or as a PE input +file number. The shell will look up files in the local directory +first and then in the PE directory (typically `/var/lib/pengine`). +Once the CIB file is found, it is copied to a shadow and this +shadow is immediately available for use at both `configure` and +`cibstatus` levels. + +If the shadow name is omitted then the target shadow is named +after the input CIB file. + +Note that there are often more than one PE input file, so you may +need to specify the full name. + +Usage: +............... +import {<file>|<number>} [<shadow>] +............... +Examples: +............... +import pe-warn-2222 +import 2289 issue2 +............... + +[[cmdhelp_cib_list,list all shadow CIBs]] +==== `list` + +List existing shadow CIBs. + +Usage: +............... +list +............... + +[[cmdhelp_cib_new,create a new shadow CIB]] +==== `new` + +Create a new shadow CIB. The live cluster configuration and +status is copied to the shadow CIB. + +If the name of the shadow is omitted, we create a temporary CIB +shadow. It is useful if multiple level sessions are desired +without affecting the cluster. A temporary CIB shadow is short +lived and will be removed either on `commit` or on program exit. +Note that if the temporary shadow is not committed all changes in +the temporary shadow are lost. + +Specify `withstatus` if you want to edit the status section of +the shadow CIB (see the <<cmdhelp_cibstatus,cibstatus section>>). +Add `force` to force overwriting the existing shadow CIB. + +To start with an empty configuration that is not copied from the live +CIB, specify the `empty` keyword. (This also allows a shadow CIB to be +created in case no cluster is running.) + +Usage: +............... +new [<cib>] [withstatus] [force] [empty] +............... + +[[cmdhelp_cib_reset,copy live cib to a shadow CIB]] +==== `reset` + +Copy the current cluster configuration into the shadow CIB. + +Usage: +............... +reset <cib> +............... + +[[cmdhelp_cib_use,change working CIB]] +==== `use` + +Choose a CIB source. If you want to edit the status from the +shadow CIB specify `withstatus` (see <<cmdhelp_cibstatus,`cibstatus`>>). +Leave out the CIB name to switch to the running CIB. + +Usage: +............... +use [<cib>] [withstatus] +............... + +[[cmdhelp_ra,Resource Agents (RA) lists and documentation]] +=== `ra` - Resource Agents (RA) lists and documentation + +This level contains commands which show various information about +the installed resource agents. It is available both at the top +level and at the `configure` level. + +[[cmdhelp_ra_classes,list classes and providers]] +==== `classes` + +Print all resource agents' classes and, where appropriate, a list +of available providers. + +Usage: +............... +classes +............... + +[[cmdhelp_ra_info,show meta data for a RA]] +==== `info` (`meta`) + +Show the meta-data of a resource agent type. This is where users +can find information on how to use a resource agent. It is also +possible to get information from some programs: `pengine`, +`crmd`, `cib`, and `stonithd`. Just specify the program name +instead of an RA. + +Usage: +............... +info [<class>:[<provider>:]]<type> +info <type> <class> [<provider>] (obsolete) +............... +Example: +............... +info apache +info ocf:pacemaker:Dummy +info stonith:ipmilan +info pengine +............... + +[[cmdhelp_ra_list,list RA for a class (and provider)]] +==== `list` + +List available resource agents for the given class. If the class +is `ocf`, supply a provider to get agents which are available +only from that provider. + +Usage: +............... +list <class> [<provider>] +............... +Example: +............... +list ocf pacemaker +............... + +[[cmdhelp_ra_providers,show providers for a RA and a class]] +==== `providers` + +List providers for a resource agent type. The class parameter +defaults to `ocf`. + +Usage: +............... +providers <type> [<class>] +............... +Example: +............... +providers apache +............... + +[[cmdhelp_ra_validate,validate parameters for RA]] +==== `validate` + +If the resource agent supports the `validate-all` action, this calls +the action with the given parameters, printing any warnings or errors +reported by the agent. + +Usage: +................ +validate <agent> [<key>=<value> ...] +................ + +[[cmdhelp_resource,Resource management]] +=== `resource` - Resource management + +At this level resources may be managed. + +All (or almost all) commands are implemented with the CRM tools +such as `crm_resource(8)`. + +[[cmdhelp_resource_ban,ban a resource from a node]] +==== `ban` + +Ban a resource from running on a certain node. If no node is given +as argument, the resource is banned from the current location. + +See `move` for details on other arguments. + +Usage: +............... +ban <rsc> [<node>] [<lifetime>] [force] +............... + +[[cmdhelp_resource_cleanup,cleanup resource status]] +==== `cleanup` + +If resource has any past failures, clear its history and fail +count. Typically done after the resource has temporarily +failed. + +If a node is omitted, cleanup on all nodes. + ++(Pacemaker 1.1.14)+ Pass force to cleanup the resource itself, +otherwise the cleanup command will apply to the parent resource (if +any). + +Usage: +............... +cleanup [<rsc>] [<node>] [force] +............... + +[[cmdhelp_resource_clear,Clear any relocation constraint]] +==== `clear` (`unmove`, `unmigrate`, `unban`) + +Remove any relocation constraint created by +the `move`, `migrate` or `ban` command. + +Usage: +............... +clear <rsc> +unmigrate <rsc> +unban <rsc> +............... + +[[cmdhelp_resource_constraints,Show constraints affecting a resource]] +==== `constraints` + +Display the location and colocation constraints affecting the +resource. + +Usage: +................ +constraints <rsc> +................ + +[[cmdhelp_resource_demote,demote a promotable resource]] +==== `demote` + +Demote a promotable resource using the `target-role` +attribute. + +Usage: +............... +demote <rsc> +............... + +[[cmdhelp_resource_failcount,manage failcounts]] +==== `failcount` + +Show/edit/delete the failcount of a resource. +When `set` a non-zero value, `operation` and `interval` should be +provided when multiple operation failcount entries exist. +`interval` is a value in seconds. + +Usage: +............... +failcount <rsc> set <node> <value> [operation] [interval] +failcount <rsc> delete <node> +failcount <rsc> show <node> +............... +Example: +............... +failcount fs_0 delete node2 +............... + +[[cmdhelp_resource_locate,show the location of resources]] +==== `locate` + +Show the current location of one or more resources. + +Usage: +............... +locate [<rsc> ...] +............... + +[[cmdhelp_resource_maintenance,Enable/disable per-resource maintenance mode]] +==== `maintenance` + +Enables or disables the per-resource maintenance mode. When this mode +is enabled, no monitor operations will be triggered for the resource. +`maintenance` attribute conflicts with the `is-managed`. When setting +the `maintenance` attribute, the user is proposed to remove the +`is-managed` attribute if it exists. + +Usage: +.................. +maintenance <resource> [on|off|true|false] +.................. + +Example: +.................. +maintenance rsc1 +maintenance rsc2 off +.................. + +[[cmdhelp_resource_manage,put a resource into managed mode]] +==== `manage` + +Manage a resource using the `is-managed` attribute. If there +are multiple meta attributes sets, the attribute is set in all of +them. If the resource is a clone, all `is-managed` attributes are +removed from the children resources. +`is-managed` attribute conflicts with the `maintenance`. When setting +the `is-managed` attribute, the user is proposed to remove the +`maintenance` attribute if it exists. + +For details on group management see <<cmdhelp_options_manage-children,`options manage-children`>>. + +Usage: +............... +manage <rsc> +............... + +[[cmdhelp_resource_meta,manage a meta attribute]] +==== `meta` + +Show/edit/delete a meta attribute of a resource. Currently, all +meta attributes of a resource may be managed with other commands +such as `resource stop`. + +Usage: +............... +meta <rsc> set <attr> <value> +meta <rsc> delete <attr> +meta <rsc> show <attr> +............... +Example: +............... +meta ip_0 set target-role stopped +............... + +[[cmdhelp_resource_move,Move a resource to another node]] +==== `move` (`migrate`) + +Move a resource away from its current location. + +If the destination node is left out, the resource is migrated by +creating a constraint which prevents it from running on the current +node. For this type of constraint to be created, the +force+ argument +is required. + +A lifetime may be given for the constraint. Once it expires, the +location constraint will no longer be active. + +Usage: +............... +move <rsc> [<node>] [<lifetime>] [force] +............... + +[[cmdhelp_resource_operations,Show active resource operations]] +==== `operations` + +Show active operations, optionally filtered by resource and node. + +Usage: +................ +operations [<rsc>] [<node>] +................ + +[[cmdhelp_resource_param,manage a parameter of a resource]] +==== `param` + +Show/edit/delete a parameter of a resource. + +Usage: +............... +param <rsc> set <param> <value> +param <rsc> delete <param> +param <rsc> show <param> +............... +Example: +............... +param ip_0 show ip +............... + +[[cmdhelp_resource_promote,promote a promotable resource]] +==== `promote` + +Promote a promotable resource using the `target-role` +attribute. + +Usage: +............... +promote <rsc> +............... + +[[cmdhelp_resource_refresh,Recheck current resource status and drop failure history]] +==== `refresh` + +Delete resource's history (including failures) so its current state is rechecked. + +Usage: +............... +refresh [<rsc>] [<node>] [force] +............... + +[[cmdhelp_resource_restart,restart resources]] +==== `restart` + +Restart one or more resources. This is essentially a shortcut for +resource stop followed by a start. The shell is first going to wait +for the stop to finish, that is for all resources to really stop, and +only then to order the start action. Due to this command +entailing a whole set of operations, informational messages are +printed to let the user see some progress. + +For details on group management see +<<cmdhelp_options_manage-children,`options manage-children`>>. + +Usage: +............... +restart <rsc> [<rsc> ...] +............... +Example: +............... +# crm resource restart g_webserver +INFO: ordering g_webserver to stop +waiting for stop to finish .... done +INFO: ordering g_webserver to start +# +............... + +[[cmdhelp_resource_scores,Display resource scores]] +==== `scores` + +Display the allocation scores for all resources. + +Usage: +................ +scores +................ + +[[cmdhelp_resource_secret,manage sensitive parameters]] +==== `secret` + +Sensitive parameters can be kept in local files rather than CIB +in order to prevent accidental data exposure. Use the `secret` +command to manage such parameters. `stash` and `unstash` move the +value from the CIB and back to the CIB respectively. The `set` +subcommand sets the parameter to the provided value. `delete` +removes the parameter completely. `show` displays the value of +the parameter from the local file. Use `check` to verify if the +local file content is valid. + +Usage: +............... +secret <rsc> set <param> <value> +secret <rsc> stash <param> +secret <rsc> unstash <param> +secret <rsc> delete <param> +secret <rsc> show <param> +secret <rsc> check <param> +............... +Example: +............... +secret fence_1 show password +secret fence_1 stash password +secret fence_1 set password secret_value +............... + +[[cmdhelp_resource_start,start resources]] +==== `start` + +Start one or more resources by setting the `target-role` attribute. If +there are multiple meta attributes sets, the attribute is set in all +of them. If the resource is a clone, all `target-role` attributes are +removed from the children resources. + +For details on group management see +<<cmdhelp_options_manage-children,`options manage-children`>>. + +Usage: +............... +start <rsc> [<rsc> ...] +............... + +[[cmdhelp_resource_status,show status of resources]] +==== `status` (`show`, `list`) + +Print resource status. More than one resource can be shown at once. If +the resource parameter is left out, the status of all resources is +printed. + +Usage: +............... +status [<rsc> ...] +............... + +[[cmdhelp_resource_stop,stop resources]] +==== `stop` + +Stop one or more resources using the `target-role` attribute. If there +are multiple meta attributes sets, the attribute is set in all of +them. If the resource is a clone, all `target-role` attributes are +removed from the children resources. + +For details on group management see +<<cmdhelp_options_manage-children,`options manage-children`>>. + +Usage: +............... +stop <rsc> [<rsc> ...] +............... + +[[cmdhelp_resource_trace,start RA tracing]] +==== `trace` + +Start tracing RA for the given operation. When `[<log-dir>]` +is not specified the trace files are stored in `$HA_VARLIB/trace_ra`. +If the operation to be traced is monitor, note that the number +of trace files can grow very quickly. + +If no operation name is given, crmsh will attempt to trace all +operations for the RA. This includes any configured operations, start +and stop as well as promote/demote for multistate resources. + +To trace the probe operation which exists for all resources, either +set a trace for `monitor` with interval `0`, or use `probe` as the +operation name. + +Note: RA tracing is only supported by OCF resource agents; +The pacemaker-execd daemon does not log recurring monitor operations +unless an error occurred. + +Usage: +............... +trace <rsc> [<op> [<interval>] [<log-dir>]] +............... +Example: +............... +trace fs start +trace webserver +trace webserver probe +trace fs monitor 0 /var/log/foo/bar +............... + +[[cmdhelp_resource_unmanage,put a resource into unmanaged mode]] +==== `unmanage` + +Unmanage a resource using the `is-managed` attribute. If there +are multiple meta attributes sets, the attribute is set in all of +them. If the resource is a clone, all `is-managed` attributes are +removed from the children resources. + +For details on group management see <<cmdhelp_options_manage-children,`options manage-children`>>. + +Usage: +............... +unmanage <rsc> +............... + +[[cmdhelp_resource_untrace,stop RA tracing]] +==== `untrace` + +Stop tracing RA for the given operation. If no operation name is +given, crmsh will attempt to stop tracing all operations in resource. + +Usage: +............... +untrace <rsc> [<op> [<interval>] ] +............... +Example: +............... +untrace fs start +untrace webserver +............... + +[[cmdhelp_resource_utilization,manage a utilization attribute]] +==== `utilization` + +Show/edit/delete a utilization attribute of a resource. These +attributes describe hardware requirements. By setting the +`placement-strategy` cluster property appropriately, it is +possible then to distribute resources based on resource +requirements and node size. See also <<cmdhelp_node_utilization,node utilization attributes>>. + +Usage: +............... +utilization <rsc> set <attr> <value> +utilization <rsc> delete <attr> +utilization <rsc> show <attr> +............... +Example: +............... +utilization xen1 set memory 4096 +............... + +[[cmdhelp_node,Node management]] +=== `node` - Node management + +Node management and status commands. + +[[cmdhelp_node_attribute,manage attributes]] +==== `attribute` + +Edit node attributes. This kind of attribute should refer to +relatively static properties, such as memory size. + +Usage: +............... +attribute <node> set <attr> <value> +attribute <node> delete <attr> +attribute <node> show <attr> +............... +Example: +............... +attribute node_1 set memory_size 4096 +............... + +[[cmdhelp_node_clearstate,Clear node state]] +==== `clearstate` + +Resets and clears the state of the specified node. This node is +afterwards assumed clean and offline. This command can be used to +manually confirm that a node has been fenced (e.g., powered off). + +Be careful! This can cause data corruption if you confirm that a node is +down that is, in fact, not cleanly down - the cluster will proceed as if +the fence had succeeded, possibly starting resources multiple times. + +Usage: +............... +clearstate <node> +............... + +[[cmdhelp_node_delete,delete node (deprecated)]] +==== `delete` + +Remove a node from cluster. + +If the node is still listed as active and a member of our +partition we refuse to remove it. With the global force option +(`-F`) we will try to delete the node anyway. + +Usage: +............... +delete <node> +............... + +.Deprecation note +***** +This command is deprecated and in favor of `crm cluster remove [-F] -c <node>`, +which will adjust the related cluster configurations and clean up the leftover +(eg. stopping the cluster services) on the removed node. +***** + +[[cmdhelp_node_fence,fence node]] + +==== `fence` + +Make CRM fence a node. This functionality depends on stonith +resources capable of fencing the specified node. No such stonith +resources, no fencing will happen. + +Usage: +............... +fence <node> +............... + +[[cmdhelp_node_maintenance,put node into maintenance mode]] +==== `maintenance` + +Set the node status to maintenance. This is equivalent to the +cluster-wide `maintenance-mode` property but puts just one node +into the maintenance mode. If there are maintained resources on +the node, the user will be proposed to remove the maintenance +property from them. + +The node parameter defaults to the node where the command is run. + +Usage: +............... +maintenance [<node>] +............... + +[[cmdhelp_node_online,set node online,From Code]] +==== `online` +See "crm node help online" or "crm node online --help" + +[[cmdhelp_node_ready,put node into ready mode]] +==== `ready` + +Set the node's maintenance status to `off`. The node should be +now again fully operational and capable of running resource +operations. + +The node parameter defaults to the node where the command is run. + +Usage: +............... +ready [<node>] +............... + +[[cmdhelp_node_server,show node hostname or server address]] +==== `server` + +Remote nodes may have a configured server address which should +be used when contacting the node. This command prints the +server address if configured, else the node name. + +If no parameter is given, the addresses or names for all nodes +are printed. + +Usage: +............... +server [<node> ...] +............... + +[[cmdhelp_node_show,show node]] +==== `show` + +Show a node definition. If the node parameter is omitted, then all +nodes are shown. + +Usage: +............... +show [<node>] +............... + +[[cmdhelp_node_standby,put node into standby,From Code]] +==== `standby` +See "crm node help standby" or "crm node standby --help" + +[[cmdhelp_node_status-attr,manage status attributes]] +==== `status-attr` + +Edit node attributes which are in the CIB status section, i.e., +attributes which hold properties of a more volatile nature. One +typical example is attribute generated by the `pingd` utility. + +Usage: +............... +status-attr <node> set <attr> <value> +status-attr <node> delete <attr> +status-attr <node> show <attr> +............... +Example: +............... +status-attr node_1 show pingd +............... + +[[cmdhelp_node_utilization,manage utilization attributes]] +==== `utilization` + +Edit node utilization attributes. These attributes describe +hardware characteristics as integer numbers such as memory size +or the number of CPUs. By setting the `placement-strategy` +cluster property appropriately, it is possible then to distribute +resources based on resource requirements and node size. See also +<<cmdhelp_resource_utilization,resource utilization attributes>>. + +Usage: +............... +utilization <node> set <attr> <value> +utilization <node> delete <attr> +utilization <node> show <attr> +............... +Examples: +............... +utilization node_1 set memory 16384 +utilization node_1 show cpu +............... + +[[cmdhelp_site,GEO clustering site support]] +=== `site` - GEO clustering site support + +A cluster may consist of two or more subclusters in different and +distant locations. This set of commands supports such setups. + +[[cmdhelp_site_ticket,manage site tickets]] +==== `ticket` + +Tickets are cluster-wide attributes. They can be managed at the +site where this command is executed. + +It is then possible to constrain resources depending on the +ticket availability (see the <<cmdhelp_configure_rsc_ticket,`rsc_ticket`>> command +for more details). + +Usage: +............... +ticket {grant|revoke|standby|activate|show|time|delete} <ticket> +............... +Example: +............... +ticket grant ticket1 +............... + +[[cmdhelp_options,User preferences]] +=== `options` - User preferences + +The user may set various options for the crm shell itself. + +[[cmdhelp_options_add-quotes,add quotes around parameters containing spaces]] +==== `add-quotes` + +The shell (as in `/bin/sh`) parser strips quotes from the command +line. This may sometimes make it really difficult to type values +which contain white space. One typical example is the configure +filter command. The crm shell will supply extra quotes around +arguments which contain white space. The default is `yes`. + +.Note on quotes use +**************************** +Adding quotes around arguments automatically has been introduced +with version 1.2.2 and it is technically a regression. Being a +regression is the only reason the `add-quotes` option exists. If +you have custom shell scripts which would break, just set the +`add-quotes` option to `no`. + +For instance, with adding quotes enabled, it is possible to do +the following: +............... +# crm configure primitive d1 Dummy \ + meta description="some description here" +# crm configure filter 'sed "s/hostlist=./&node-c /"' fencing +............... +**************************** + +[[cmdhelp_options_check-frequency,when to perform semantic check]] +==== `check-frequency` + +Semantic check of the CIB or elements modified or created may be +done on every configuration change (`always`), when verifying +(`on-verify`) or `never`. It is by default set to `always`. +Experts may want to change the setting to `on-verify`. + +The checks require that resource agents are present. If they are +not installed at the configuration time set this preference to +`never`. + +See <<topics_Features_Checks,Configuration semantic checks>> for more details. + +[[cmdhelp_options_check-mode,how to treat semantic errors]] +==== `check-mode` + +Semantic check of the CIB or elements modified or created may be +done in the `strict` mode or in the `relaxed` mode. In the former +certain problems are treated as configuration errors. In the +`relaxed` mode all are treated as warnings. The default is `strict`. + +See <<topics_Features_Checks,Configuration semantic checks>> for more details. + +[[cmdhelp_options_colorscheme,set colors for output]] +==== `colorscheme` + +With `output` set to `color`, a comma separated list of colors +from this option are used to emphasize: + +- keywords +- object ids +- attribute names +- attribute values +- scores +- resource references + +`crm` can show colors only if there is curses support for python +installed (usually provided by the `python-curses` package). The +colors are whatever is available in your terminal. Use `normal` +if you want to keep the default foreground color. + +This user preference defaults to +`yellow,normal,cyan,red,green,magenta` which is good for +terminals with dark background. You may want to change the color +scheme and save it in the preferences file for other color +setups. + +Example: +............... +colorscheme yellow,normal,blue,red,green,magenta +............... + +[[cmdhelp_options_editor,set preferred editor program]] +==== `editor` + +The `edit` command invokes an editor. Use this to specify your +preferred editor program. If not set, it will default to either +the value of the `EDITOR` environment variable or to one of the +standard UNIX editors (`vi`,`emacs`,`nano`). + +Usage: +............... +editor program +............... +Example: +............... +editor vim +............... + +[[cmdhelp_options_manage-children,how to handle children resource attributes]] +==== `manage-children` + +Some resource management commands, such as `resource stop`, when +the target resource is a group, may not always produce desired +result. Each element, group and the primitive members, can have a +meta-attribute and those attributes may end up with conflicting +values. Consider the following construct: +............... +crm(live)# configure show svc fs virtual-ip +primitive fs Filesystem \ + params device="/dev/drbd0" directory="/srv/nfs" fstype=ext3 \ + op monitor interval=10s \ + meta target-role=Started +primitive virtual-ip IPaddr2 \ + params ip=10.2.13.110 iflabel=1 \ + op monitor interval=10s \ + op start interval=0 \ + meta target-role=Started +group svc fs virtual-ip \ + meta target-role=Stopped +............... + +Even though the element +svc+ should be stopped, the group is +actually running because all its members have the +target-role+ +set to +Started+: +............... +crm(live)# resource show svc +resource svc is running on: xen-f +............... + +Hence, if the user invokes +resource stop svc+ the intention is +not clear. This preference gives the user an opportunity to +better control what happens if attributes of group members have +values which are in conflict with the same attribute of the group +itself. + +Possible values are +ask+ (the default), +always+, and +never+. +If set to +always+, the crm shell removes all children attributes +which have values different from the parent. If set to +never+, +all children attributes are left intact. Finally, if set to ++ask+, the user will be asked for each member what is to be done. + +[[cmdhelp_options_output,set output type]] +==== `output` + +`crm` can adorn configurations in two ways: in color (similar to +for instance the `ls --color` command) and by showing keywords in +upper case. Possible values are `plain`, `color-always`, `color`, +and 'uppercase'. It is possible to combine `uppercase` with one +of the color values in order to get an upper case xmass tree. Just +set this option to `color,uppercase` or `color-always,uppercase`. +In case you need color codes in pipes, `color-always` forces color +codes even in case the terminal is not a tty (just like `ls +--color=always`). + +[[cmdhelp_options_pager,set preferred pager program]] +==== `pager` + +The `view` command displays text through a pager. Use this to +specify your preferred pager program. If not set, it will default +to either the value of the `PAGER` environment variable or to one +of the standard UNIX system pagers (`less`,`more`,`pg`). + +[[cmdhelp_options_reset,reset user preferences to factory defaults]] +==== `reset` + +This command resets all user options to the defaults. If used as +a single-shot command, the rc file (+$HOME/.config/crm/rc+) is +reset to the defaults too. + +[[cmdhelp_options_save,save the user preferences to the rc file]] +==== `save` + +Save current settings to the rc file (+$HOME/.config/crm/rc+). On +further `crm` runs, the rc file is automatically read and parsed. + +[[cmdhelp_options_set,Set the value of a given option]] +==== `set` + +Sets the value of an option. Takes the fully qualified +name of the option as argument, as displayed by +show all+. + +The modified option value is stored in the user-local +configuration file, usually found in +~/.config/crm/crm.conf+. + +Usage: +........ +set <option> <value> +........ + +Example: +........ +set color.warn "magenta bold" +set editor nano +........ + +[[cmdhelp_options_show,show current user preference]] +==== `show` + +Display all current settings. + +Given an option name as argument, `show` will display only the value +of that argument. + +Given +all+ as argument, `show` displays all available user options. + +Usage: +........ +show [all|<option>] +........ + +Example: +........ +show +show skill-level +show all +........ + +[[cmdhelp_options_skill-level,set skill level]] +==== `skill-level` + +Based on the skill-level setting, the user is allowed to use only +a subset of commands. There are three levels: operator, +administrator, and expert. The operator level allows only +commands at the `resource` and `node` levels, but not editing +or deleting resources. The administrator may do that and may also +configure the cluster at the `configure` level and manage the +shadow CIBs. The expert may do all. + +Usage: +............... +skill-level <level> + +level :: operator | administrator | expert +............... + +.Note on security +**************************** +The `skill-level` option is advisory only. There is nothing +stopping any users change their skill level (see +<<topics_Features_Security,Access Control Lists (ACL)>> on how to enforce +access control). +**************************** + +[[cmdhelp_options_sort-elements,sort CIB elements]] +==== `sort-elements` + +`crm` by default sorts CIB elements. If you want them appear in +the order they were created, set this option to `no`. + +Usage: +............... +sort-elements {yes|no} +............... +Example: +............... +sort-elements no +............... + +[[cmdhelp_options_user,set the cluster user]] +==== `user` + +Sufficient privileges are necessary in order to manage a +cluster: programs such as `crm_verify` or `crm_resource` and, +ultimately, `cibadmin` have to be run either as `root` or as the +CRM owner user (typically `hacluster`). You don't have to worry +about that if you run `crm` as `root`. A more secure way is to +run the program with your usual privileges, set this option to +the appropriate user (such as `hacluster`), and setup the +`sudoers` file. + +Usage: +............... +user system-user +............... +Example: +............... +user hacluster +............... + +[[cmdhelp_options_wait,synchronous operation]] +==== `wait` + +In normal operation, `crm` runs a command and gets back +immediately to process other commands or get input from the user. +With this option set to `yes` it will wait for the started +transition to finish. In interactive mode dots are printed to +indicate progress. + +Usage: +............... +wait {yes|no} +............... +Example: +............... +wait yes +............... + +[[cmdhelp_configure,CIB configuration]] +=== `configure` - CIB configuration + +This level enables all CIB object definition commands. + +The configuration may be logically divided into four parts: +nodes, resources, constraints, and (cluster) properties and +attributes. Each of these commands support one or more basic CIB +objects. + +Nodes and attributes describing nodes are managed using the +`node` command. + +Commands for resources are: + +- `primitive` +- `monitor` +- `group` +- `clone` (promotable clones) +- `ms`/`master` (master-slave) (deprecated) + +In order to streamline large configurations, it is possible to +define a template which can later be referenced in primitives: + +- `rsc_template` + +In that case the primitive inherits all attributes defined in the +template. + +There are three types of constraints: + +- `location` +- `colocation` +- `order` + +It is possible to define fencing order (stonith resource +priorities): + +- `fencing_topology` + +Finally, there are the cluster properties, resource meta-attributes +defaults, and operations defaults. All are just a set of attributes. +These attributes are managed by the following commands: + +- `property` +- `rsc_defaults` +- `op_defaults` + +In addition to the cluster configuration, the Access Control +Lists (ACL) can be setup to allow access to parts of the CIB for +users other than +root+ and +hacluster+. The following commands +manage ACL: + +- `user` +- `role` + +In Pacemaker 1.1.12 and up, this command replaces the `user` command +for handling ACLs: + +- `acl_target` + +The changes are applied to the current CIB only on ending the +configuration session or using the `commit` command. + +Comments start with +#+ in the first line. The comments are tied +to the element which follows. If the element moves, its comments +will follow. + +[[cmdhelp_configure_acl_target,Define target access rights]] +==== `acl_target` + +Defines an ACL target. + +Usage: +................ +acl_target <tid> [<role> ...] +................ +Example: +................ +acl_target joe resource_admin constraint_editor +................ + +[[cmdhelp_configure_alert,Event-driven alerts]] +==== `alert` + +.Version note +**************************** +This feature is only available +in Pacemaker 1.1.15+. +**************************** + +Event-driven alerts enables calling scripts whenever interesting +events occur in the cluster (nodes joining or leaving, resources +starting or stopping, etc.). + +The +path+ is an arbitrary file path to an alert script. Existing +external scripts used with ClusterMon resources can be used as alert +scripts, since the interface is compatible. + +Each alert may have a number of recipients configured. These will be +passed to the script as arguments. The first recipient will also be +passed as the +CRM_alert_recipient+ environment variable, for +compatibility with existing scripts that only support one recipient. + +The available meta-attributes are +timeout+ (default 30s) and ++timestamp-format+ (default `"%H:%M:%S.%06N"`). + +Some configurations may require each recipient to be delimited by +brackets, to avoid ambiguity. In the example +alert-2+ below, the meta +attribute for `timeout` is defined after the recipient, so the +brackets are used to ensure that the meta attribute is set for the +alert and not just the recipient. This can be avoided by setting any +alert attributes before defining the recipients. + +Usage: +............... +alert <id> <path> \ + [attributes <nvpair> ...] \ + [meta <nvpair> ...] \ + [select [nodes | fencing | resources | attributes '{' <attribute> ... '}' ] ...] \ + [to [{] <recipient> + [attributes <nvpair> ...] \ + [meta <nvpair> ...] [}] \ + ...] +............... + +Example: +............... +alert alert-1 /srv/pacemaker/pcmk_alert_sample.sh \ + to /var/log/cluster-alerts.log + +alert alert-2 /srv/pacemaker/example_alert.sh \ + meta timeout=60s \ + to { /var/log/cluster-alerts.log } + +alert alert-3 /srv/pacemaker/example_alert.sh \ + select fencing \ + to { /var/log/fencing-alerts.log } + +............... + +[[cmdhelp_configure_bundle,Container bundle]] +==== `bundle` + +A bundle is a single resource specifying the settings, networking +requirements, and storage requirements for any number of containers +generated from the same container image. + +Pacemaker bundles support Docker (since version 1.1.17) and rkt (since +version 1.1.18) container technologies. + +A bundle must contain exactly one +docker+ or +rkt+ element. + +The bundle definition may contain a reference to a primitive +resource which defining the resource running inside the +container. + +Example: +............... + +primitive httpd-apache ocf:heartbeat:apache + +bundle httpd \ + docker image=pcmk:httpd replicas=3 \ + network ip-range-start=10.10.10.123 host-netmask=24 \ + port-mapping port=80 \ + storage \ + storage-mapping target-dir=/var/www/html source-dir=/srv/www options=rw \ + primitive httpd-apache + +............... + +[[cmdhelp_configure_cib,CIB shadow management]] +==== `cib` + +This level is for management of shadow CIBs. It is available at +the `configure` level to enable saving intermediate changes to a +shadow CIB instead of to the live cluster. This short excerpt +shows how: +............... +crm(live)configure# cib new test-2 +INFO: test-2 shadow CIB created +crm(test-2)configure# commit +............... +Note how the current CIB in the prompt changed from +live+ to ++test-2+ after issuing the `cib new` command. See also the +<<cmdhelp_cib,CIB shadow management>> for more information. + +[[cmdhelp_configure_cibstatus,CIB status management and editing]] +==== `cibstatus` + +Enter edit and manage the CIB status section level. See the +<<cmdhelp_cibstatus,CIB status management section>>. + +[[cmdhelp_configure_clone,define a clone]] +==== `clone` + +The `clone` command creates a resource clone. It may contain a +single primitive resource or one group of resources. + ++Promotable clones+ are clone resources with the +promotable=true+ meta-attribute for the given promotable resources. +It's used to deprecate the master-slave resources. + +Usage: +............... +clone <name> <rsc> + [description=<description>] + [meta <attr_list>] + [params <attr_list>] + +attr_list :: [$id=<id>] <attr>=<val> [<attr>=<val>...] | $id-ref=<id> +............... +Example: +............... +clone cl_fence apc_1 \ + meta clone-node-max=1 globally-unique=false + +clone disk1 drbd1 \ + meta promotable=true notify=true globally-unique=false +............... + +[[cmdhelp_configure_colocation,colocate resources]] +==== `colocation` (`collocation`) + +This constraint expresses the placement relation between two +or more resources. If there are more than two resources, then the +constraint is called a resource set. + +The score is used to indicate the priority of the constraint. A +positive score indicates that the resources should run on the same +node. A negative score that they should not run on the same +node. Values of positive or negative +infinity+ indicate a mandatory +constraint. + +In the two resource form, the cluster will place +<with-rsc>+ first, +and then decide where to put the +<rsc>+ resource. + +Collocation resource sets have an extra attribute (+sequential+) +to allow for sets of resources which don't depend on each other +in terms of state. The shell syntax for such sets is to put +resources in parentheses. + +Sets cannot be nested. + +The optional +node-attribute+ can be used to colocate resources on a +set of nodes and not necessarily on the same node. For example, by +setting a node attribute +color+ on all nodes and setting the ++node-attribute+ value to +color+ as well, the colocated resources +will be placed on any node that has the same color. + +For more details on how to configure resource sets, see +<<topics_Features_Resourcesets,`Syntax: Resource sets`>>. + +Usage: +............... +colocation <id> <score>: <rsc>[:<role>] <with-rsc>[:<role>] + [node-attribute=<node_attr>] + +colocation <id> <score>: <resource_sets> + [node-attribute=<node_attr>] + +resource_sets :: <resource_set> [<resource_set> ...] + +resource_set :: ["("|"["] <rsc>[:<role>] [<rsc>[:<role>] ...] \ + [<attributes>] [")"|"]"] + +attributes :: [require-all=(true|false)] [sequential=(true|false)] + +............... +Example: +............... +colocation never_put_apache_with_dummy -inf: apache dummy +colocation c1 inf: A ( B C ) +............... + +[[cmdhelp_configure_commit,commit the changes to the CIB]] +==== `commit` + +Commit the current configuration to the CIB in use. As noted +elsewhere, commands in a configure session don't have immediate +effect on the CIB. All changes are applied at one point in time, +either using `commit` or when the user leaves the configure +level. In case the CIB in use changed in the meantime, presumably +by somebody else, the crm shell will refuse to apply the changes. + +If you know that it's fine to still apply them, add +force+ to the +command line. + +To disable CIB patching and apply the changes by replacing the CIB +completely, add +replace+ to the command line. Note that this can lead +to previous changes being overwritten if some other process +concurrently modifies the CIB. + +Usage: +............... +commit [force] [replace] +............... + +[[cmdhelp_configure_default-timeouts,set timeouts for operations to minimums from the meta-data]] +==== `default-timeouts` + +This command takes the timeouts from the actions section of the +resource agent meta-data and sets them for the operations of the +primitive. + +Usage: +............... +default-timeouts <id> [<id>...] +............... + +.Note on `default-timeouts` +**************************** +The use of this command is discouraged in favor of manually +determining the best timeouts required for the particular +configuration. Relying on the resource agent to supply appropriate +timeouts can cause the resource to fail at the worst possible moment. + +Appropriate timeouts for resource actions are context-sensitive, and +should be carefully considered with the whole configuration in mind. +**************************** + +[[cmdhelp_configure_delete,delete CIB objects]] +==== `delete` + +Delete one or more objects. If an object to be deleted belongs to +a container object, such as a group, and it is the only resource +in that container, then the container is deleted as well. Any +related constraints are removed as well. + +If the object is a started resource, it will not be deleted unless the ++--force+ flag is passed to the command, or the +force+ option is set. + +Usage: +............... +delete [--force] <id> [<id>...] +............... + +[[cmdhelp_configure_edit,edit CIB objects]] +==== `edit` + +This command invokes the editor with the object description. As +with the `show` command, the user may choose to edit all objects +or a set of objects. + +If the user insists, he or she may edit the XML edition of the +object. If you do that, don't modify any id attributes. + +Usage: +............... +edit [xml] [<id> ...] +edit [xml] changed +............... + +.Note on renaming element ids +**************************** +The edit command sometimes cannot properly handle modifying +element ids. In particular for elements which belong to group or +ms resources. Group and ms resources themselves also cannot be +renamed. Please use the `rename` command instead. +**************************** + +[[cmdhelp_configure_erase,erase the CIB]] +==== `erase` + +.Deprecation note +**************************** +`crm configure erase` is deprecated. +The replacement could be `crm cluster remove [node]` +**************************** + +The `erase` clears all configuration. Apart from nodes. To remove +nodes, you have to specify an additional keyword `nodes`. + +Note that removing nodes from the live cluster may have some +strange/interesting/unwelcome effects. + +Usage: +............... +erase [nodes] +............... + +[[cmdhelp_configure_fencing_topology,node fencing order]] +==== `fencing_topology` + +If multiple fencing (stonith) devices are available capable of +fencing a node, their order may be specified by +fencing_topology+. +The order is specified per node. + +Stonith resources can be separated by +,+ in which case all of +them need to succeed. If they fail, the next stonith resource (or +set of resources) is used. In other words, use comma to separate +resources which all need to succeed and whitespace for serial +order. It is not allowed to use whitespace around comma. + +If the node is left out, the order is used for all nodes. +That should reduce the configuration size in some stonith setups. + +From Pacemaker version 1.1.14, it is possible to use a node attribute +as the +target+ in a fencing topology. The syntax for this usage is +described below. + +From Pacemaker version 1.1.14, it is also possible to use regular +expression patterns as the +target+ in a fencing topology. The configured +fencing sequence then applies to all devices matching the pattern. + +Usage: +............... +fencing_topology <stonith_resources> [<stonith_resources> ...] +fencing_topology <fencing_order> [<fencing_order> ...] + +fencing_order :: <target> <stonith_resources> [<stonith_resources> ...] + +stonith_resources :: <rsc>[,<rsc>...] +target :: <node>: | attr:<node-attribute>=<value> | pattern:<pattern> +............... +Example: +............... +# Only kill the power if poison-pill fails +fencing_topology poison-pill power + +# As above for node-a, but a different strategy for node-b +fencing_topology \ + node-a: poison-pill power \ + node-b: ipmi serial + +# Fencing anything on rack 1 requires fencing via both APC 1 and 2, +# to defeat the redundancy provided by two separate UPS units. +fencing_topology attr:rack=1 apc01,apc02 + +# Fencing for all machines named green.* is done using the pear +# fencing device first, while all machines named red.* are fenced +# using the apple fencing device first. +fencing_topology \ + pattern:green.* pear apple \ + pattern:red.* apple pear +............... + +[[cmdhelp_configure_filter,filter CIB objects]] +==== `filter` + +This command filters the given CIB elements through an external +program. The program should accept input on `stdin` and send +output to `stdout` (the standard UNIX filter conventions). As +with the `show` command, the user may choose to filter all or +just a subset of elements. + +It is possible to filter the XML representation of objects, but +probably not as useful as the configuration language. The +presentation is somewhat different from what would be displayed +by the `show` command---each element is shown on a single line, +i.e., there are no backslashes and no other embellishments. + +Don't forget to put quotes around the filter if it contains +spaces. + +Usage: +............... +filter <prog> [xml] [<id> ...] +filter <prog> [xml] changed +............... +Examples: +............... +filter "sed '/^primitive/s/target-role=[^ ]*//'" +# crm configure filter "sed '/^primitive/s/target-role=[^ ]*//'" +crm configure <<END + filter "sed '/threshold=\"1\"/s/=\"1\"/=\"0\"/g'" +END +............... + +.Note on quotation marks +************************** +Filter commands which feature a blend of quotation marks can be +difficult to get right, especially when used directly from bash, since +bash does its own quotation parsing. In these cases, it can be easier +to supply the filter command as standard input. See the last example +above. +************************** + +[[cmdhelp_configure_get_property,Get property value]] +==== `get-property` + +Show the value of the given property. If the value is not set, the +command will print the default value for the property, if known. + +If no property name is passed to the command, the list of known +cluster properties is printed. + +If the property is set multiple times, for example using multiple +property sets with different rule expressions, the output of this +command is undefined. + +Pass the argument +-t+ or +--true+ to `get-property` to translate +the argument value into +true+ or +false+. If the value is not +set, the command will print +false+. + +Usage: +............... +get-property [-t|--true] [<name>] +............... + +Example: +............... +get-property stonith-enabled +get-property -t maintenance-mode +............... + +[[cmdhelp_configure_graph,generate a directed graph]] +==== `graph` + +Create a graphviz graphical layout from the current cluster +configuration. + +Currently, only `dot` (directed graph) is supported. It is +essentially a visualization of resource ordering. + +The graph may be saved to a file which can be used as source for +various graphviz tools (by default it is displayed in the user's +X11 session). Optionally, by specifying the format, one can also +produce an image instead. + +For more or different graphviz attributes, it is possible to save +the default set of attributes to an ini file. If this file exists +it will always override the builtin settings. The +exportsettings+ +subcommand also prints the location of the ini file. + +Usage: +............... +graph [<gtype> [<file> [<img_format>]]] +graph exportsettings + +gtype :: dot +img_format :: `dot` output format (see the +-T+ option) +............... +Example: +............... +graph dot +graph dot clu1.conf.dot +graph dot clu1.conf.svg svg +............... + +[[cmdhelp_configure_group,define a group]] +==== `group` + +The `group` command creates a group of resources. This can be useful +when resources depend on other resources and require that those +resources start in order on the same node. A common use of resource +groups is to ensure that a server and a virtual IP are located +together, and that the virtual IP is started before the server. + +Grouped resources are started in the order they appear in the group, +and stopped in the reverse order. If a resource in the group cannot +run anywhere, resources following it in the group will not start. + +`group` can be passed the "container" meta attribute, to indicate that +it is to be used to group VM resources monitored using Nagios. The +resource referred to by the container attribute must be of type +`ocf:heartbeat:Xen`, `ocf:heartbeat:VirtualDomain` or `ocf:heartbeat:lxc`. + +Usage: +............... +group <name> <rsc> [<rsc>...] + [description=<description>] + [meta attr_list] + [params attr_list] + +attr_list :: [$id=<id>] <attr>=<val> [<attr>=<val>...] | $id-ref=<id> +............... +Example: +............... +group internal_www disk0 fs0 internal_ip apache \ + meta target_role=stopped + +group vm-and-services vm vm-sshd meta container="vm" +............... + +[[cmdhelp_configure_load,import the CIB from a file]] +==== `load` + +Load a part of configuration (or all of it) from a local file or +a network URL. The +replace+ method replaces the current +configuration with the one from the source. The +update+ method +tries to import the contents into the current configuration. The ++push+ method imports the contents into the current configuration +and removes any lines that are not present in the given +configuration. +The file may be a CLI file or an XML file. + +If the URL is `-`, the configuration is read from standard input. + +Usage: +............... +load [xml] <method> URL + +method :: replace | update | push +............... +Example: +............... +load xml update myfirstcib.xml +load xml replace http://storage.big.com/cibs/bigcib.xml +load xml push smallcib.xml +............... + +[[cmdhelp_configure_location,a location preference]] +==== `location` + +`location` defines the preference of nodes for the given +resource. The location constraints consist of one or more rules +which specify a score to be awarded if the rule matches. + +The resource referenced by the location constraint can be one of the +following: + +* Plain resource reference: +location loc1 webserver 100: node1+ +* Resource set in curly brackets: +location loc1 { virtual-ip webserver } 100: node1+ +* Tag containing resource ids: +location loc1 tag1 100: node1+ +* Resource pattern: +location loc1 /web.*/ 100: node1+ + +The +resource-discovery+ attribute allows probes to be selectively +enabled or disabled per resource and node. + +The syntax for resource sets is described in detail for +<<cmdhelp_configure_colocation,`colocation`>>. + +For more details on how to configure resource sets, see +<<topics_Features_Resourcesets,`Syntax: Resource sets`>>. + +For more information on rule expressions, see +<<topics_Syntax_RuleExpressions,Syntax: Rule expressions>>. + +Usage: +............... +location <id> <rsc> [<attributes>] {<node_pref>|<rules>} + +rsc :: /<rsc-pattern>/ + | { resource_sets } + | <rsc> + +attributes :: role=<role> | resource-discovery=always|never|exclusive + +node_pref :: <score>: <node> + +rules :: + rule [id_spec] [$role=<role>] <score>: <expression> + [rule [id_spec] [$role=<role>] <score>: <expression> ...] + +id_spec :: $id=<id> | $id-ref=<id> +score :: <number> | <attribute> | [-]inf +expression :: <simple_exp> [<bool_op> <simple_exp> ...] +bool_op :: or | and +simple_exp :: <attribute> [type:]<binary_op> <value> + | <unary_op> <attribute> + | date <date_expr> +type :: string | version | number +binary_op :: lt | gt | lte | gte | eq | ne +unary_op :: defined | not_defined + +date_expr :: lt <end> + | gt <start> + | in start=<start> end=<end> + | in start=<start> <duration> + | spec <date_spec> +duration|date_spec :: + hours=<value> + | monthdays=<value> + | weekdays=<value> + | yearsdays=<value> + | months=<value> + | weeks=<value> + | years=<value> + | weekyears=<value> + | moon=<value> +............... +Examples: +............... +location conn_1 internal_www 100: node1 + +location conn_1 internal_www \ + rule 50: #uname eq node1 \ + rule pingd: defined pingd + +location conn_2 dummy_float \ + rule -inf: not_defined pingd or pingd number:lte 0 + +# never probe for rsc1 on node1 +location no-probe rsc1 resource-discovery=never -inf: node1 +............... + +[[cmdhelp_configure_modgroup,modify group]] +==== `modgroup` + +Add or remove primitives in a group. The `add` subcommand appends +the new group member by default. Should it go elsewhere, there +are `after` and `before` clauses. + +Usage: +............... +modgroup <id> add <id> [after <id>|before <id>] +modgroup <id> remove <id> +............... +Examples: +............... +modgroup share1 add storage2 before share1-fs +............... + +[[cmdhelp_configure_monitor,add monitor operation to a primitive]] +==== `monitor` + +Monitor is by far the most common operation. It is possible to +add it without editing the whole resource. Also, long primitive +definitions may be a bit uncluttered. In order to make this +command as concise as possible, less common operation attributes +are not available. If you need them, then use the `op` part of +the `primitive` command. + +Usage: +............... +monitor <rsc>[:<role>] <interval>[:<timeout>] +............... +Example: +............... +monitor apcfence 60m:60s +............... + +Note that after executing the command, the monitor operation may +be shown as part of the primitive definition. + +[[cmdhelp_configure_ms,define a master-slave resource (deprecated)]] +==== `ms` (`master`) + +The `ms` command creates a master/slave resource type. It may contain a +single primitive resource or one group of resources. + +Usage: +............... +ms <name> <rsc> + [description=<description>] + [meta attr_list] + [params attr_list] + +attr_list :: [$id=<id>] <attr>=<val> [<attr>=<val>...] | $id-ref=<id> +............... +Example: +............... +ms disk1 drbd1 \ + meta notify=true globally-unique=false +............... + +.Note on `ms` deprecated +**************************** +From Pacemaker-2.0, the resource type referred to as "master/slave", +"stateful", or "multi-state" is no longer a separate resource type, +but a variation of clone now referred to as a "promotable clone". +For backward compatibility, above configurations are also accepted. +............... +clone disk1 drbd1 \ + meta promotable=true notify=true globally-unique=false +............... +**************************** + +.Note on `id-ref` usage +**************************** +Instance or meta attributes (`params` and `meta`) may contain +a reference to another set of attributes. In that case, no other +attributes are allowed. Since attribute sets' ids, though they do +exist, are not shown in the `crm`, it is also possible to +reference an object instead of an attribute set. `crm` will +automatically replace such a reference with the right id: + +............... +crm(live)configure# primitive a2 www-2 meta $id-ref=a1 +crm(live)configure# show a2 +primitive a2 apache \ + meta $id-ref=a1-meta_attributes + [...] +............... +It is advisable to give meaningful names to attribute sets which +are going to be referenced. +**************************** + +[[cmdhelp_configure_node,define a cluster node]] +==== `node` + +The node command describes a cluster node. Nodes in the CIB are +commonly created automatically by the CRM. Hence, you should not +need to deal with nodes unless you also want to define node +attributes. Note that it is also possible to manage node +attributes at the `node` level. + +Usage: +............... +node [$id=<id>] <uname>[:<type>] + [description=<description>] + [attributes [$id=<id>] [<score>:] [rule...] + <param>=<value> [<param>=<value>...]] | $id-ref=<ref> + [utilization [$id=<id>] [<score>:] [rule...] + <param>=<value> [<param>=<value>...]] | $id-ref=<ref> + +type :: normal | member | ping | remote +............... +Example: +............... +node node1 +node big_node attributes memory=64 +............... + +[[cmdhelp_configure_op_defaults,set resource operations defaults]] +==== `op_defaults` + +Set defaults for the operations meta attributes. + +For more information on rule expressions, see +<<topics_Syntax_RuleExpressions,Syntax: Rule expressions>>. + +Usage: +............... +op_defaults [$id=<set_id>] [rule ...] <option>=<value> [<option>=<value> ...] +............... +Example: +............... +op_defaults record-pending=true +............... + +[[cmdhelp_configure_order,order resources]] +==== `order` + +This constraint expresses the order of actions on two resources +or more resources. If there are more than two resources, then the +constraint is called a resource set. + +Ordered resource sets have an extra attribute to allow for sets +of resources whose actions may run in parallel. The shell syntax +for such sets is to put resources in parentheses. + +If the subsequent resource can start or promote after any one of the +resources in a set has done, enclose the set in brackets (+[+ and +]+). + +Sets cannot be nested. + +Three strings are reserved to specify a kind of order constraint: ++Mandatory+, +Optional+, and +Serialize+. It is preferred to use +one of these settings instead of score. Previous versions mapped +scores +0+ and +inf+ to keywords +advisory+ and +mandatory+. +That is still valid but deprecated. + +For more details on how to configure resource sets, see +<<topics_Features_Resourcesets,`Syntax: Resource sets`>>. + +Usage: +............... +order <id> [kind:] first then [symmetrical=<bool>] + +order <id> [kind:] resource_sets [symmetrical=<bool>] + +kind :: Mandatory | Optional | Serialize + +first :: <rsc>[:<action>] + +then :: <rsc>[:<action>] + +resource_sets :: resource_set [resource_set ...] + +resource_set :: ["["|"("] <rsc>[:<action>] [<rsc>[:<action>] ...] \ + [attributes] ["]"|")"] + +attributes :: [require-all=(true|false)] [sequential=(true|false)] + +............... +Example: +............... +order o-1 Mandatory: apache:start ip_1 +order o-2 Serialize: A ( B C ) +order o-4 first-resource then-resource +............... + +[[cmdhelp_configure_primitive,define a resource]] +==== `primitive` + +The primitive command describes a resource. It may be referenced +only once in group, or clone objects. If it's not +referenced, then it is placed as a single resource in the CIB. + +Operations may be specified anonymously, as a group or by reference: + +* "Anonymous", as a list of +op+ specifications. Use this + method if you don't need to reference the set of operations + elsewhere. This is the most common way to define operations. + +* If reusing operation sets is desired, use the +operations+ keyword + along with an id to give the operations set a name. Use the + +operations+ keyword and an id-ref value set to the id of another + operations set, to apply the same set of operations to this + primitive. + +Operation attributes which are not recognized are saved as +instance attributes of that operation. A typical example is ++OCF_CHECK_LEVEL+. + +For multistate resources, roles are specified as +role=<role>+. +The +Master/Slave+ resources are deprecated and replaced by +Promoted/Unpromoted+ promotable resources if desired. + +A template may be defined for resources which are of the same +type and which share most of the configuration. See +<<cmdhelp_configure_rsc_template,`rsc_template`>> for more information. + +Attributes containing time values, such as the +interval+ attribute on +operations, are configured either as a plain number, which is +interpreted as a time in seconds, or using one of the following +suffixes: + +* +s+, +sec+ - time in seconds (same as no suffix) +* +ms+, +msec+ - time in milliseconds +* +us+, +usec+ - time in microseconds +* +m+, +min+ - time in minutes +* +h+, +hr+ - time in hours + +Usage: +............... +primitive <rsc> {[<class>:[<provider>:]]<type>|@<template>} + [description=<description>] + [[params] attr_list] + [meta attr_list] + [utilization attr_list] + [operations id_spec] + [op op_type [<attribute>=<value>...] + [[op_params] attr_list] + [op_meta attr_list] ...] + +attr_list :: [$id=<id>] [<score>:] [rule...] + <attr>=<val> [<attr>=<val>...]] | $id-ref=<id> +id_spec :: $id=<id> | $id-ref=<id> +op_type :: start | stop | monitor +............... +Example: +............... +primitive apcfence stonith:apcsmart \ + params ttydev=/dev/ttyS0 hostlist="node1 node2" \ + op start timeout=60s \ + op monitor interval=30m timeout=60s + +primitive www8 apache \ + configfile=/etc/apache/www8.conf \ + operations $id-ref=apache_ops + +primitive db0 mysql \ + params config=/etc/mysql/db0.conf \ + op monitor interval=60s \ + op monitor interval=300s OCF_CHECK_LEVEL=10 + +primitive r0 ocf:linbit:drbd \ + params drbd_resource=r0 \ + op monitor role=Promoted interval=60s \ + op monitor role=Unpromoted interval=300s + +primitive xen0 @vm_scheme1 xmfile=/etc/xen/vm/xen0 + +primitive mySpecialRsc Special \ + params 3: rule #uname eq node1 interface=eth1 \ + params 2: rule #uname eq node2 interface=eth2 port=8888 \ + params 1: interface=eth0 port=9999 + +primitive A ocf:pacemaker:Dummy \ + op start \ + op_meta 2: rule #ra-version version:gt 1.0 timeout=120s \ + op_meta 1: timeout=60s +............... + +[[cmdhelp_configure_property,set a cluster property]] +==== `property` + +Set cluster configuration properties. To list the +available cluster configuration properties, use the +<<cmdhelp_ra_info,`ra info`>> command with +pengine+, +crmd+, ++cib+ and +stonithd+ as arguments. +When setting the +maintenance-mode+ property, it will +inform the user if there are nodes or resources that +have the +maintenance+ property. + +For more information on rule expressions, see +<<topics_Syntax_RuleExpressions,Syntax: Rule expressions>>. + +Usage: +............... +property [<set_id>:] [rule ...] <option>=<value> [<option>=<value> ...] +............... +Example: +............... +property stonith-enabled=true +property rule date spec years=2014 stonith-enabled=false +............... + +[[cmdhelp_configure_ptest,show cluster actions if changes were committed]] +==== `ptest` (`simulate`) + +Show PE (Policy Engine) motions using `ptest(8)` or +`crm_simulate(8)`. + +A CIB is constructed using the current user edited configuration +and the status from the running CIB. The resulting CIB is run +through `ptest` (or `crm_simulate`) to show changes which would +happen if the configuration is committed. + +The status section may be loaded from another source and modified +using the <<cmdhelp_cibstatus,`cibstatus`>> level commands. In that case, the +`ptest` command will issue a message informing the user that the +Policy Engine graph is not calculated based on the current status +section and therefore won't show what would happen to the +running but some imaginary cluster. + +If you have graphviz installed and X11 session, `dotty(1)` is run +to display the changes graphically. + +Add a string of +v+ characters to increase verbosity. `ptest` +can also show allocation scores. +utilization+ turns on +information about the remaining capacity of nodes. With the ++actions+ option, `ptest` will print all resource actions. + +The `ptest` program has been replaced by `crm_simulate` in newer +Pacemaker versions. In some installations both could be +installed. Use `simulate` to enfore using `crm_simulate`. + +Usage: +............... +ptest [nograph] [v...] [scores] [actions] [utilization] +............... +Examples: +............... +ptest scores +ptest vvvvv +simulate actions +............... + +[[cmdhelp_configure_refresh,refresh from CIB]] +==== `refresh` + +Refresh the internal structures from the CIB. All changes made +during this session are lost. + +Usage: +............... +refresh +............... + +[[cmdhelp_configure_rename,rename a CIB object]] +==== `rename` + +Rename an object. It is recommended to use this command to rename +a resource, because it will take care of updating all related +constraints and a parent resource. Changing ids with the edit +command won't have the same effect. + +If you want to rename a resource, it must be in the stopped state. + +Usage: +............... +rename <old_id> <new_id> +............... + +[[cmdhelp_configure_role,define role access rights]] +==== `role` + +An ACL role is a set of rules which describe access rights to +CIB. Rules consist of an access right +read+, +write+, or +deny+ +and a specification denoting part of the configuration to which +the access right applies. The specification can be an XPath or a +combination of tag and id references. If an attribute is +appended, then the specification applies only to that attribute +of the matching element. + +There is a number of shortcuts for XPath specifications. The ++meta+, +params+, and +utilization+ shortcuts reference resource +meta attributes, parameters, and utilization respectively. The +`location` may be used to specify location constraints most of +the time to allow resource `move` and `unmove` commands. The +`property` references cluster properties. The `node` allows +reading node attributes. +nodeattr+ and +nodeutil+ reference node +attributes and node capacity (utilization). The `status` shortcut +references the whole status section of the CIB. Read access to +status is necessary for various monitoring tools such as +`crm_mon(8)` (aka `crm status`). + +For more information on rule expressions, see +<<topics_Syntax_RuleExpressions,Syntax: Rule expressions>>. + +Usage: +............... +role <role-id> rule [rule ...] + +rule :: acl-right cib-spec [attribute:<attribute>] + +acl-right :: read | write | deny + +cib-spec :: xpath-spec | tag-ref-spec +xpath-spec :: xpath:<xpath> | shortcut +tag-ref-spec :: tag:<tag> | ref:<id> | tag:<tag> ref:<id> + +shortcut :: meta:<rsc>[:<attr>] + params:<rsc>[:<attr>] + utilization:<rsc> + location:<rsc> + property[:<attr>] + node[:<node>] + nodeattr[:<attr>] + nodeutil[:<node>] + status +............... +Example: +............... +role app1_admin \ + write meta:app1:target-role \ + write meta:app1:is-managed \ + write location:app1 \ + read ref:app1 +............... + +[[cmdhelp_configure_rsc_defaults,set resource defaults]] +==== `rsc_defaults` + +Set defaults for the resource meta attributes. + +For more information on rule expressions, see +<<topics_Syntax_RuleExpressions,Syntax: Rule expressions>>. + +Usage: +............... +rsc_defaults [<set_id>:] [rule ...] <option>=<value> [<option>=<value> ...] +............... +Example: +............... +rsc_defaults failure-timeout=3m +............... + +[[cmdhelp_configure_rsc_template,define a resource template]] +==== `rsc_template` + +The `rsc_template` command creates a resource template. It may be +referenced in primitives. It is used to reduce large +configurations with many similar resources. + +Usage: +............... +rsc_template <name> [<class>:[<provider>:]]<type> + [description=<description>] + [params attr_list] + [meta attr_list] + [utilization attr_list] + [operations id_spec] + [op op_type [<attribute>=<value>...] ...] + +attr_list :: [$id=<id>] <attr>=<val> [<attr>=<val>...] | $id-ref=<id> +id_spec :: $id=<id> | $id-ref=<id> +op_type :: start | stop | monitor +............... +Example: +............... +rsc_template public_vm Xen \ + op start timeout=300s \ + op stop timeout=300s \ + op monitor interval=30s timeout=60s \ + op migrate_from timeout=600s \ + op migrate_to timeout=600s +primitive xen0 @public_vm \ + params xmfile=/etc/xen/xen0 +primitive xen1 @public_vm \ + params xmfile=/etc/xen/xen1 +............... + +[[cmdhelp_configure_rsc_ticket,resources ticket dependency]] +==== `rsc_ticket` + +This constraint expresses dependency of resources on cluster-wide +attributes, also known as tickets. Tickets are mainly used in +geo-clusters, which consist of multiple sites. A ticket may be +granted to a site, thus allowing resources to run there. + +The +loss-policy+ attribute specifies what happens to the +resource (or resources) if the ticket is revoked. The default is +either +stop+ or +demote+ depending on whether a resource is +multi-state. + +See also the <<cmdhelp_site_ticket,`site`>> set of commands. + +Usage: +............... +rsc_ticket <id> <ticket_id>: <rsc>[:<role>] [<rsc>[:<role>] ...] + [loss-policy=<loss_policy_action>] + +loss_policy_action :: stop | demote | fence | freeze +............... +Example: +............... +rsc_ticket ticket-A_public-ip ticket-A: public-ip +rsc_ticket ticket-A_bigdb ticket-A: bigdb loss-policy=fence +rsc_ticket ticket-B_storage ticket-B: drbd-a:Promoted drbd-b:Promoted +............... + + +[[cmdhelp_configure_rsctest,test resources as currently configured]] +==== `rsctest` + +Test resources with current resource configuration. If no nodes +are specified, tests are run on all known nodes. + +The order of resources is significant: it is assumed that later +resources depend on earlier ones. + +If a resource is multi-state, it is assumed that the role on +which later resources depend is master. + +Tests are run sequentially to prevent running the same resource +on two or more nodes. Tests are carried out only if none of the +specified nodes currently run any of the specified resources. +However, it won't verify whether resources run on the other +nodes. + +Superuser privileges are obviously required: either run this as +root or setup the `sudoers` file appropriately. + +Note that resource testing may take some time. + +Usage: +............... +rsctest <rsc_id> [<rsc_id> ...] [<node_id> ...] +............... +Examples: +............... +rsctest my_ip websvc +rsctest websvc nodeB +............... + +[[cmdhelp_configure_save,save the CIB to a file]] +==== `save` + +Save the current configuration to a file. Optionally, as XML. Use ++-+ instead of file name to write the output to `stdout`. + +The `save` command accepts the same selection arguments as the `show` +command. See the <<cmdhelp_configure_show,help section>> for `show` +for more details. + +Usage: +............... +save [xml] [<id> | type:<type | tag:<tag> | + related:<obj> | changed ...] <file> +............... +Example: +............... +save myfirstcib.txt +save web-server server-config.txt +............... + +[[cmdhelp_configure_schema,set or display current CIB RNG schema]] +==== `schema` + +CIB's content is validated by a RNG schema. Pacemaker supports +several, depending on version. At least the following schemas are +accepted by `crmsh`: + +* +pacemaker-1.0+ +* +pacemaker-1.1+ +* +pacemaker-1.2+ +* +pacemaker-1.3+ +* +pacemaker-2.0+ + +Use this command to display or switch to another RNG schema. + +Usage: +............... +schema [<schema>] +............... +Example: +............... +schema pacemaker-1.1 +............... + +[[cmdhelp_configure_set,set an attribute value]] +==== `set` + +Set the value of a configured attribute. The attribute must +configured previously, and can be an agent parameter, meta attribute, +utilization value or operation value. + +The first argument to the command is a path to an attribute. +This is a dot-separated sequence beginning with the name of +the resource or object, and ending with the name of the attribute to +set. To set operation value, `op_type` should be specified; when multi +operations exist like multi monitors, `interval` should be specified. + +Usage: +............... +set <path> <value> + +path:: id.[op_type.][interval.]name +............... +Examples: +............... +set vip1.ip 192.168.20.5 +set vm-a.force_stop 1 +set vip1.monitor.on-fail ignore +set drbd.monitor.10s.interval 20s +............... + +[[cmdhelp_configure_show,display CIB objects]] +==== `show` + +The `show` command displays CIB objects. Without any argument, it +displays all objects in the CIB, but the set of objects displayed by +`show` can be limited to only objects with the given IDs or by using +one or more of the special prefixes described below. + +The XML representation for the objects can be displayed by passing ++xml+ as the first argument. + +To show one or more specific objects, pass the object IDs as +arguments. + +To show all objects of a certain type, use the +type:+ prefix. + +To show all objects in a tag, use the +tag:+ prefix. + +To show all constraints related to a primitive, or +to show all objects of a certain RA type, use the +related:+ prefix. + +To show all modified objects, pass the argument +changed+. + +The prefixes can be used together on a single command line. For +example, to show both the tag itself and the objects tagged by it the +following combination can be used: +show tag:my-tag my-tag+. + +To refine a selection of objects using multiple modifiers, the keywords ++and+ and +or+ can be used. For example, to select all primitives tagged ++foo+, the following combination can be used: ++show type:primitive and tag:foo+. + +To hide values when displaying the configuration, use the ++obscure:<glob>+ argument. This can be useful when sending the +configuration over a public channel, to avoid exposing potentially +sensitive information. The +<glob>+ argument is a bash-style pattern +matching attribute keys. + +In +/etc/crm/crm.conf+, +obscure_pattern+ option is the persisent configuration of CLI. +Example, for the high security concern, +............... +[core] +obscure_pattern = passw* | ip +............... +Which makes +crm configure show+ is equal to +............... +node-1:~ # crm configure show obscure:passw* obscure:ip +node 1084783297: node1 +primitive fence_device stonith:fence_ilo5 \ + params password="******" +primitive ip IPaddr2 \ + params ip="******" +............... +The default suggestion is +passw*+ +If you don't want to obscure, change the value to blank. + +Usage: +............... +show [xml] [<id> + | changed + | type:<type> + | tag:<id> + | related:<obj> + | obscure:<glob> + ...] + +type :: node | primitive | group | clone | ms | rsc_template + | location | colocation | order + | rsc_ticket + | property | rsc_defaults | op_defaults + | fencing_topology + | role | user | acl_target + | tag +............... + +Example: +............... +show webapp +show type:primitive +show xml tag:db tag:fs +show related:webapp +show related:IPaddr2 +show related:ipad +show related:ocf:heartbeat:Dummy +show related:ocf:heartbeat:dum +show related:ocf +show related:heartbeat +show related:pacemaker +show related:suse +show related:stonith +show type:primitive obscure:passwd +............... + +[[cmdhelp_configure_tag,Define resource tags]] +==== `tag` + +Define a resource tag. A tag is an id referring to one or more +resources, without implying any constraints between the tagged +resources. This can be useful for grouping conceptually related +resources. + +Usage: +............... +tag <tag-name>: <rsc> [<rsc> ...] +tag <tag-name> <rsc> [<rsc> ...] +............... +Example: +............... +tag web: p-webserver p-vip +tag ips server-vip admin-vip +............... + +[[cmdhelp_configure_template,edit and import a configuration from a template]] +==== `template` + +The specified template is loaded into the editor. It's up to the +user to make a good CRM configuration out of it. See also the +<<cmdhelp_template,template section>>. + +Usage: +............... +template [xml] url +............... +Example: +............... +template two-apaches.txt +............... + +[[cmdhelp_configure_upgrade,upgrade the CIB]] +==== `upgrade` + +Attempts to upgrade the CIB to validate with the current +version. Commonly, this is required if the error +`CIB not supported` occurs. It typically means that the +active CIB version is coming from an older release. + +As a safety precaution, the force argument is required if the ++validation-with+ attribute is set to anything other than ++0.6+. Thus in most cases, it is required. + +Usage: +............... +upgrade [force] +............... + +Example: +............... +upgrade force +............... + +[[cmdhelp_configure_user,define user access rights]] +==== `user` + +Users which normally cannot view or manage cluster configuration +can be allowed access to parts of the CIB. The access is defined +by a set of +read+, +write+, and +deny+ rules as in role +definitions or by referencing roles. The latter is considered +best practice. + +For more information on rule expressions, see +<<topics_Syntax_RuleExpressions,Syntax: Rule expressions>>. + +Usage: +............... +user <uid> {roles|rules} + +roles :: role:<role-ref> [role:<role-ref> ...] +rules :: rule [rule ...] +............... +Example: +............... +user joe \ + role:app1_admin \ + role:read_all +............... + +[[cmdhelp_configure_validate_all,call agent validate-all for resource]] +==== `validate-all` + +Call the `validate-all` action for the resource, if possible. + +Limitations: + +* The resource agent must implement the `validate-all` action. +* The current user must be root. +* The primitive resource must not use nvpair references. + +Usage: +............... +validate-all <rsc> +............... + + +[[cmdhelp_configure_verify,verify the CIB with crm_verify]] +==== `verify` + +Verify the contents of the CIB which would be committed. + +Usage: +............... +verify +............... + +[[cmdhelp_configure_xml,raw xml]] +==== `xml` + +Even though we promissed no xml, it may happen, but hopefully +very very seldom, that an element from the CIB cannot be rendered +in the configuration language. In that case, the element will be +shown as raw xml, prefixed by this command. That element can then +be edited like any other. If the shell finds out that after the +change it can digest it, then it is going to be converted into +the normal configuration language. Otherwise, there is no need to +use `xml` for configuration. + +Usage: +............... +xml <xml> +............... + +[[cmdhelp_template,edit and import a configuration from a template]] +=== `template` - Import configuration from templates + +User may be assisted in the cluster configuration by templates +prepared in advance. Templates consist of a typical ready +configuration which may be edited to suit particular user needs. + +This command enters a template level where additional commands +for configuration/template management are available. + +[[cmdhelp_template_apply,process and apply the current configuration to the current CIB]] +==== `apply` + +Copy the current or given configuration to the current CIB. By +default, the CIB is replaced, unless the method is set to +"update". + +Usage: +............... +apply [<method>] [<config>] + +method :: replace | update +............... + +[[cmdhelp_template_delete,delete a configuration]] +==== `delete` + +Remove a configuration. The loaded (active) configuration may be +removed by force. + +Usage: +............... +delete <config> [force] +............... + +[[cmdhelp_template_edit,edit a configuration]] +==== `edit` + +Edit current or given configuration using your favourite editor. + +Usage: +............... +edit [<config>] +............... + +[[cmdhelp_template_list,list configurations/templates]] +==== `list` + +When called with no argument, lists existing templates and +configurations. + +Given the argument +templates+, lists the available templates. + +Given the argument +configs+, lists the available configurations. + +Usage: +............... +list [templates|configs] +............... + +[[cmdhelp_template_load,load a configuration]] +==== `load` + +Load an existing configuration. Further `edit`, `show`, and +`apply` commands will refer to this configuration. + +Usage: +............... +load <config> +............... + +[[cmdhelp_template_new,create a new configuration from templates]] +==== `new` + +Create a new configuration from one or more templates. Note that +configurations and templates are kept in different places, so it +is possible to have a configuration name equal a template name. + +If you already know which parameters are required, you can set +them directly on the command line. + +The parameter name +id+ is set by default to the name of the +configuration. + +If no parameters are being set and you don't want a particular name +for your configuration, you can call this command with a template name +as the only parameter. A unique configuration name based on the +template name will be generated. + +Usage: +............... +new [<config>] <template> [<template> ...] [params name=value ...] +............... + +Example: +............... +new vip virtual-ip +new bigfs ocfs2 params device=/dev/sdx8 directory=/bigfs +new apache +............... + +[[cmdhelp_template_show,show the processed configuration]] +==== `show` + +Process the current or given configuration and display the result. + +Usage: +............... +show [<config>] +............... + +[[cmdhelp_cibstatus,CIB status management and editing]] +=== `cibstatus` - CIB status management and editing + +The `status` section of the CIB keeps the current status of nodes +and resources. It is modified _only_ on events, i.e. when some +resource operation is run or node status changes. For obvious +reasons, the CRM has no user interface with which it is possible +to affect the status section. From the user's point of view, the +status section is essentially a read-only part of the CIB. The +current status is never even written to disk, though it is +available in the PE (Policy Engine) input files which represent +the history of cluster motions. The current status may be read +using the +cibadmin -Q+ command. + +It may sometimes be of interest to see how status changes would +affect the Policy Engine. The set of `cibstatus` level commands +allow the user to load status sections from various sources and +then insert or modify resource operations or change nodes' state. + +The effect of those changes may then be observed by running the +<<cmdhelp_configure_ptest,`ptest`>> command at the `configure` level +or `simulate` and `run` commands at this level. The `ptest` +runs with the user edited CIB whereas the latter two commands +run with the CIB which was loaded along with the status section. + +The `simulate` and `run` commands as well as all status +modification commands are implemented using `crm_simulate(8)`. + +[[cmdhelp_cibstatus_load,load the CIB status section]] +==== `load` + +Load a status section from a file, a shadow CIB, or the running +cluster. By default, the current (+live+) status section is +modified. Note that if the +live+ status section is modified it +is not going to be updated if the cluster status changes, because +that would overwrite the user changes. To make `crm` drop changes +and resume use of the running cluster status, run +load live+. + +All CIB shadow configurations contain the status section which is +a snapshot of the status section taken at the time the shadow was +created. Obviously, this status section doesn't have much to do +with the running cluster status, unless the shadow CIB has just +been created. Therefore, the `ptest` command by default uses the +running cluster status section. + +Usage: +............... +load {<file>|shadow:<cib>|live} +............... +Example: +............... +load bug-12299.xml +load shadow:test1 +............... + +[[cmdhelp_cibstatus_node,change node status]] +==== `node` + +Change the node status. It is possible to throw a node out of +the cluster, make it a member, or set its state to unclean. + ++online+:: Set the +node_state+ `crmd` attribute to +online+ +and the +expected+ and +join+ attributes to +member+. The effect +is that the node becomes a cluster member. + ++offline+:: Set the +node_state+ `crmd` attribute to +offline+ +and the +expected+ attribute to empty. This makes the node +cleanly removed from the cluster. + ++unclean+:: Set the +node_state+ `crmd` attribute to +offline+ +and the +expected+ attribute to +member+. In this case the node +has unexpectedly disappeared. + +Usage: +............... +node <node> {online|offline|unclean} +............... +Example: +............... +node xen-b unclean +............... + +[[cmdhelp_cibstatus_op,edit outcome of a resource operation]] +==== `op` + +Edit the outcome of a resource operation. This way you can +tell CRM that it ran an operation and that the resource agent +returned certain exit code. It is also possible to change the +operation's status. In case the operation status is set to +something other than +done+, the exit code is effectively +ignored. + +Usage: +............... +op <operation> <resource> <exit_code> [<op_status>] [<node>] + +operation :: probe | monitor[:<n>] | start | stop | + promote | demote | notify | migrate_to | migrate_from +exit_code :: <rc> | success | generic | args | + unimplemented | perm | installed | configured | not_running | + master | failed_master +op_status :: pending | done | cancelled | timeout | notsupported | error + +n :: the monitor interval in seconds; if omitted, the first + recurring operation is referenced +rc :: numeric exit code in range 0..9 +............... +Example: +............... +op start d1 xen-b generic +op start d1 xen-b 1 +op monitor d1 xen-b not_running +op stop d1 xen-b 0 timeout +............... + +[[cmdhelp_cibstatus_origin,display origin of the CIB status section]] +==== `origin` + +Show the origin of the status section currently in use. This +essentially shows the latest `load` argument. + +Usage: +............... +origin +............... + +[[cmdhelp_cibstatus_quorum,set the quorum]] +==== `quorum` + +Set the quorum value. + +Usage: +............... +quorum <bool> +............... +Example: +............... +quorum false +............... + +[[cmdhelp_cibstatus_run,run policy engine]] +==== `run` + +Run the policy engine with the edited status section. + +Add a string of +v+ characters to increase verbosity. Specify ++scores+ to see allocation scores also. +utilization+ turns on +information about the remaining capacity of nodes. + +If you have graphviz installed and X11 session, `dotty(1)` is run +to display the changes graphically. + +Usage: +............... +run [nograph] [v...] [scores] [utilization] +............... +Example: +............... +run +............... + +[[cmdhelp_cibstatus_save,save the CIB status section]] +==== `save` + +The current internal status section with whatever modifications +were performed can be saved to a file or shadow CIB. + +If the file exists and contains a complete CIB, only the status +section is going to be replaced and the rest of the CIB will +remain intact. Otherwise, the current user edited configuration +is saved along with the status section. + +Note that all modifications are saved in the source file as soon +as they are run. + +Usage: +............... +save [<file>|shadow:<cib>] +............... +Example: +............... +save bug-12299.xml +............... + +[[cmdhelp_cibstatus_show,show CIB status section]] +==== `show` + +Show the current status section in the XML format. Brace yourself +for some unreadable output. Add +changed+ option to get a human +readable output of all changes. + +Usage: +............... +show [changed] +............... + +[[cmdhelp_cibstatus_simulate,simulate cluster transition]] +==== `simulate` + +Run the policy engine with the edited status section and simulate +the transition. + +Add a string of +v+ characters to increase verbosity. Specify ++scores+ to see allocation scores also. +utilization+ turns on +information about the remaining capacity of nodes. + +If you have graphviz installed and X11 session, `dotty(1)` is run +to display the changes graphically. + +Usage: +............... +simulate [nograph] [v...] [scores] [utilization] +............... +Example: +............... +simulate +............... + +[[cmdhelp_cibstatus_ticket,manage tickets]] +==== `ticket` + +Modify the ticket status. Tickets can be granted and revoked. +Granted tickets could be activated or put in standby. + +Usage: +............... +ticket <ticket> {grant|revoke|activate|standby} +............... +Example: +............... +ticket ticketA grant +............... + +[[cmdhelp_assist,Configuration assistant]] +=== `assist` - Configuration assistant + +The `assist` sublevel is a collection of helper +commands that create or modify resources and +constraints, to simplify the creation of certain +configurations. + +For more information on individual commands, see +the help text for those commands. + +[[cmdhelp_assist_template,Create template for primitives]] +==== `template` + +This command takes a list of primitives as argument, and creates a new +`rsc_template` for these primitives. It can only do this if the +primitives do not already share a template and are of the same type. + +Usage: +........ +template primitive-1 primitive-2 primitive-3 +........ + +[[cmdhelp_assist_weak-bond,Create a weak bond between resources]] +==== `weak-bond` + +A colocation between a group of resources says that the resources +should be located together, but it also means that those resources are +dependent on each other. If one of the resources fails, the others +will be restarted. + +If this is not desired, it is possible to circumvent: By placing the +resources in a non-sequential set and colocating the set with a dummy +resource which is not monitored, the resources will be placed together +but will have no further dependency on each other. + +This command creates both the constraint and the dummy resource needed +for such a colocation. + +Usage: +........ +weak-bond resource-1 resource-2 +........ + +[[cmdhelp_maintenance,Maintenance mode commands]] +=== `maintenance` - Maintenance mode commands + +Maintenance mode commands are commands that manipulate resources +directly without going through the cluster infrastructure. Therefore, +it is essential to ensure that the cluster does not attempt to monitor +or manipulate the resources while these commands are being executed. + +To ensure this, these commands require that maintenance mode is set +either for the particular resource, or for the whole cluster. + +[[cmdhelp_maintenance_action,Invoke a resource action]] +==== `action` + +Invokes the given action for the resource. This is +done directly via the resource agent, so the command must +be issued while the cluster or the resource is in +maintenance mode. + +Unless the action is `start` or `monitor`, the action must be invoked +on the same node as where the resource is running. If the resource is +running on multiple nodes, the command will fail. + +To use SSH for executing resource actions on multiple nodes, append +`ssh` after the action name. This requires SSH access to be configured +between the nodes and the parallax python package to be installed. + +Usage: +............... +action <rsc> <action> +action <rsc> <action> ssh +............... +Example: +............... +action webserver reload +action webserver monitor ssh +............... + +[[cmdhelp_maintenance_off,Disable maintenance mode]] +==== `off` + +Disables maintenances mode, either for the whole cluster +or for the given resource. + +Usage: +............... +off +off <rsc> +............... +Example: +............... +off rsc1 +............... + +[[cmdhelp_maintenance_on,Enable maintenance mode]] +==== `on` + +Enables maintenances mode, either for the whole cluster +or for the given resource. + +Usage: +............... +on +on <rsc> +............... +Example: +............... +on rsc1 +............... + +[[cmdhelp_history,Cluster history]] +=== `history` - Cluster history + +Examining Pacemaker's history is a particularly involved task. The +number of subsystems to be considered, the complexity of the +configuration, and the set of various information sources, most of +which are not exactly human readable, keep analyzing resource or node +problems accessible to only the most knowledgeable. Or, depending on +the point of view, to the most persistent. The following set of +commands has been devised in hope to make cluster history more +accessible. + +Of course, looking at _all_ history could be time consuming regardless +of how good the tools at hand are. Therefore, one should first say +which period he or she wants to analyze. If not otherwise specified, +the last hour is considered. Logs and other relevant information is +collected using `crm report`. Since this process takes some time and +we always need fresh logs, information is refreshed in a much faster +way using the python parallax module. If +python-parallax+ is not +found on the system, examining a live cluster is still possible -- +though not as comfortable. + +Apart from examining a live cluster, events may be retrieved from a +report generated by `crm report` (see also the +-H+ option). In that +case we assume that the period stretching the whole report needs to be +investigated. Of course, it is still possible to further reduce the +time range. + +If you have discovered an issue that you want to show someone else, +you can use the `session pack` command to save the current session as +a tarball, similar to those generated by `crm report`. + +In order to minimize the size of the tarball, and to make it easier +for others to find the interesting events, it is recommended to limit +the time frame which the saved session covers. This can be done using +the `timeframe` command (example below). + +It is also possible to name the saved session using the `session save` +command. + +Example: +............... +crm(live)history# limit "Jul 18 12:00" "Jul 18 12:30" +crm(live)history# session save strange_restart +crm(live)history# session pack +Report saved in .../strange_restart.tar.bz2 +crm(live)history# +............... + +[[cmdhelp_history_detail,set the level of detail shown]] +==== `detail` + +How much detail to show from the logs. Valid detail levels are either +`0` or `1`, where `1` is the highest detail level. The default detail +level is `0`. + +Usage: +............... +detail <detail_level> + +detail_level :: small integer (defaults to 0) +............... +Example: +............... +detail 1 +............... + +[[cmdhelp_history_diff,cluster states/transitions difference]] +==== `diff` + +A transition represents a change in cluster configuration or +state. Use `diff` to see what has changed between two +transitions. + +If you want to specify the current cluster configuration and +status, use the string +live+. + +Normally, the first transition specified should be the one which +is older, but we are not going to enforce that. + +Note that a single configuration update may result in more than +one transition. + +Usage: +............... +diff <pe> <pe> [status] [html] + +pe :: <number>|<index>|<file>|live +............... +Examples: +............... +diff 2066 2067 +diff pe-input-2080.bz2 live status +............... + +[[cmdhelp_history_events,Show events in log]] +==== `events` + +By analysing the log output and looking for particular +patterns, the `events` command helps sifting through +the logs to find when particular events like resources +changing state or node failure may have occurred. + +This can be used to generate a combined list of events +from all nodes. + +Usage: +............... +events +............... + +Example: +............... +events +............... + +[[cmdhelp_history_exclude,exclude log messages]] +==== `exclude` + +If a log is infested with irrelevant messages, those messages may +be excluded by specifying a regular expression. The regular +expressions used are Python extended. This command is additive. +To drop all regular expressions, use +exclude clear+. Run +`exclude` only to see the current list of regular expressions. +Excludes are saved along with the history sessions. + +Usage: +............... +exclude [<regex>|clear] +............... +Example: +............... +exclude kernel.*ocfs2 +............... + +[[cmdhelp_history_graph,generate a directed graph from the PE file]] +==== `graph` + +Create a graphviz graphical layout from the PE file (the +transition). Every transition contains the cluster configuration +which was active at the time. See also <<cmdhelp_configure_graph,generate a directed graph +from configuration>>. + +Usage: +............... +graph <pe> [<gtype> [<file> [<img_format>]]] + +gtype :: dot +img_format :: `dot` output format (see the +-T+ option) +............... +Example: +............... +graph -1 +graph 322 dot clu1.conf.dot +graph 322 dot clu1.conf.svg svg +............... + +[[cmdhelp_history_info,Cluster information summary]] +==== `info` + +The `info` command provides a summary of the information source, which +can be either a live cluster snapshot or a previously generated +report. + +Usage: +............... +info +............... +Example: +............... +info +............... + +[[cmdhelp_history_latest,show latest news from the cluster]] +==== `latest` + +The `latest` command shows a bit of recent history, more +precisely whatever happened since the last cluster change (the +latest transition). If the transition is running, the shell will +first wait until it finishes. + +Usage: +............... +latest +............... +Example: +............... +latest +............... + +[[cmdhelp_history_limit,limit timeframe to be examined]] +==== `limit` (`timeframe`) + +This command can be used to modify the time span to examine. All +history commands look at events within a certain time span. + +For the `live` source, the default time span is the _last hour_. + +There is no time span limit for the `crm report` source. + +The time period is parsed by the `dateutil` python module. It +covers a wide range of date formats. For instance: + +- 3:00 (today at 3am) +- 15:00 (today at 3pm) +- 2010/9/1 2pm (September 1st 2010 at 2pm) + +For more examples of valid time/date statements, please refer to the +`python-dateutil` documentation: + +- https://dateutil.readthedocs.org/[dateutil.readthedocs.org] + +If the dateutil module is not available, then the time is parsed using +strptime and only the kind as printed by `date(1)` is allowed: + +- Tue Sep 15 20:46:27 CEST 2010 + +Usage: +............... +limit [<from_time>] [<to_time>] +............... +Examples: +............... +limit 10:15 +limit 15h22m 16h +limit "Sun 5 20:46" "Sun 5 22:00" +............... + +[[cmdhelp_history_log,log content]] +==== `log` + +Show messages logged on one or more nodes. Leaving out a node +name produces combined logs of all nodes. Messages are sorted by +time and, if the terminal emulations supports it, displayed in +different colours depending on the node to allow for easier +reading. + +The sorting key is the timestamp as written by syslog which +normally has the maximum resolution of one second. Obviously, +messages generated by events which share the same timestamp may +not be sorted in the same way as they happened. Such close events +may actually happen fairly often. + +Usage: +............... +log [<node> [<node> ...] ] +............... +Example: +............... +log node-a +............... + +[[cmdhelp_history_node,node events]] +==== `node` + +Show important events that happened on a node. Important events +are node lost and join, standby and online, and fence. Use either +node names or extended regular expressions. + +Usage: +............... +node <node> [<node> ...] +............... +Example: +............... +node node1 +............... + +[[cmdhelp_history_peinputs,list or get PE input files]] +==== `peinputs` + +Every event in the cluster results in generating one or more +Policy Engine (PE) files. These files describe future motions of +resources. The files are listed as full paths in the current +report directory. Add +v+ to also see the creation time stamps. + +Usage: +............... +peinputs [{<range>|<number>} ...] [v] + +range :: <n1>:<n2> +............... +Example: +............... +peinputs +peinputs 440:444 446 +peinputs v +............... + +[[cmdhelp_history_refresh,refresh live report]] +==== `refresh` + +This command makes sense only for the +live+ source and makes +`crm` collect the latest logs and other relevant information from +the logs. If you want to make a completely new report, specify ++force+. + +Usage: +............... +refresh [force] +............... + +[[cmdhelp_history_resource,resource events]] +==== `resource` + +Show actions and any failures that happened on all specified +resources on all nodes. Normally, one gives resource names as +arguments, but it is also possible to use extended regular +expressions. Note that neither groups nor clones or master/slave +names are ever logged. The resource command is going to expand +all of these appropriately, so that clone instances or resources +which are part of a group are shown. + +Usage: +............... +resource <rsc> [<rsc> ...] +............... +Example: +............... +resource bigdb public_ip +resource my_.*_db2 +resource ping_clone +............... + +[[cmdhelp_history_session,manage history sessions]] +==== `session` + +Sometimes you may want to get back to examining a particular +history period or bug report. In order to make that easier, the +current settings can be saved and later retrieved. + +If the current history being examined is coming from a live +cluster the logs, PE inputs, and other files are saved too, +because they may disappear from nodes. For the existing reports +coming from `crm report`, only the directory location is saved +(not to waste space). + +A history session may also be packed into a tarball which can +then be sent to support. + +Leave out subcommand to see the current session. + +Usage: +............... +session [{save|load|delete} <name> | pack [<name>] | update | list] +............... +Examples: +............... +session save bnc966622 +session load rsclost-2 +session list +............... + +[[cmdhelp_history_setnodes,set the list of cluster nodes]] +==== `setnodes` + +In case the host this program runs on is not part of the cluster, +it is necessary to set the list of nodes. + +Usage: +............... +setnodes node <node> [<node> ...] +............... +Example: +............... +setnodes node_a node_b +............... + +[[cmdhelp_history_show,show status or configuration of the PE input file]] +==== `show` + +Every transition is saved as a PE file. Use this command to +render that PE file either as configuration or status. The +configuration output is the same as `crm configure show`. + +Usage: +............... +show <pe> [status] + +pe :: <number>|<index>|<file>|live +............... +Examples: +............... +show 2066 +show pe-input-2080.bz2 status +............... + +[[cmdhelp_history_source,set source to be examined]] +==== `source` + +Events to be examined can come from the current cluster or from a +`crm report` report. This command sets the source. `source live` +sets source to the running cluster and system logs. If no source +is specified, the current source information is printed. + +In case a report source is specified as a file reference, the file +is going to be unpacked in place where it resides. This directory +is not removed on exit. + +Usage: +............... +source [<dir>|<file>|live] +............... +Examples: +............... +source live +source /tmp/customer_case_22.tar.bz2 +source /tmp/customer_case_22 +source +............... + +[[cmdhelp_history_transition,show transition]] +==== `transition` + +This command will print actions planned by the PE and run +graphviz (`dotty`) to display a graphical representation of the +transition. Of course, for the latter an X11 session is required. +This command invokes `ptest(8)` in background. + +The +showdot+ subcommand runs graphviz (`dotty`) to display a +graphical representation of the +.dot+ file which has been +included in the report. Essentially, it shows the calculation +produced by `pengine` which is installed on the node where the +report was produced. In optimal case this output should not +differ from the one produced by the locally installed `pengine`. + +The `log` subcommand shows the full log for the duration of the +transition. + +A transition can also be saved to a CIB shadow for further +analysis or use with `cib` or `configure` commands (use the +`save` subcommand). The shadow file name defaults to the name of +the PE input file. + +If the PE input file number is not provided, it defaults to the +last one, i.e. the last transition. The last transition can also +be referenced with number 0. If the number is negative, then the +corresponding transition relative to the last one is chosen. + +If there are warning and error PE input files or different nodes +were the DC in the observed timeframe, it may happen that PE +input file numbers collide. In that case provide some unique part +of the path to the file. + +After the `ptest` output, logs about events that happened during +the transition are printed. + +The `tags` subcommand scans the logs for the transition and return a +list of key events during that transition. For example, the tag ++error+ will be returned if there are any errors logged during the +transition. + +Usage: +............... +transition [<number>|<index>|<file>] [nograph] [v...] [scores] [actions] [utilization] +transition showdot [<number>|<index>|<file>] +transition log [<number>|<index>|<file>] +transition save [<number>|<index>|<file> [name]] +transition tags [<number>|<index>|<file>] +............... +Examples: +............... +transition +transition 444 +transition -1 +transition pe-error-3.bz2 +transition node-a/pengine/pe-input-2.bz2 +transition showdot 444 +transition log +transition save 0 enigma-22 +............... + +[[cmdhelp_history_transitions,List transitions]] +==== `transitions` + +A transition represents a change in cluster configuration or +state. This command lists the transitions in the current timeframe. + +Usage: +............... +transitions +............... +Example: +............... +transitions +............... + + +[[cmdhelp_history_wdiff,cluster states/transitions difference]] +==== `wdiff` + +A transition represents a change in cluster configuration or +state. Use `wdiff` to see what has changed between two +transitions as word differences on a line-by-line basis. + +If you want to specify the current cluster configuration and +status, use the string +live+. + +Normally, the first transition specified should be the one which +is older, but we are not going to enforce that. + +Note that a single configuration update may result in more than +one transition. + +Usage: +............... +wdiff <pe> <pe> [status] + +pe :: <number>|<index>|<file>|live +............... +Examples: +............... +wdiff 2066 2067 +wdiff pe-input-2080.bz2 live status +............... + +[[cmdhelp_root_report,Create cluster status report,From Code]] +=== `report` +See "crm help report" or "crm report --help" + +=== `end` (`cd`, `up`) + +The `end` command ends the current level and the user moves to +the parent level. This command is available everywhere. + +Usage: +............... +end +............... + +=== `help` + +The `help` command prints help for the current level or for the +specified topic (command). This command is available everywhere. + +Usage: +............... +help [<topic>] +............... + +=== `quit` (`exit`, `bye`) + +Leave the program. + +FILES +--- +include::profiles.adoc[] + +BUGS +---- +Even though all sensible configurations (and most of those that +are not) are going to be supported by the crm shell, I suspect +that it may still happen that certain XML constructs may confuse +the tool. When that happens, please file a bug report. + +The crm shell will not try to update the objects it does not +understand. Of course, it is always possible to edit such objects +in the XML format. + +AUTHORS +------- +Dejan Muhamedagic, <dejan@suse.de> +Kristoffer Gronlund <kgronlund@suse.com> +and many OTHERS + +SEE ALSO +-------- +crm_resource(8), crm_attribute(8), crm_mon(8), cib_shadow(8), +ptest(8), dotty(1), crm_simulate(8), cibadmin(8) + + +COPYING +------- +Copyright \(C) 2008-2013 Dejan Muhamedagic. +Copyright \(C) 2013 Kristoffer Gronlund. + +Free use of this software is granted under the terms of the GNU General Public License (GPL). + +////////////////////// + vim:ts=4:sw=4:expandtab: +////////////////////// diff --git a/doc/crmsh_crm_report.8.adoc b/doc/crmsh_crm_report.8.adoc new file mode 100644 index 0000000..9e5e91c --- /dev/null +++ b/doc/crmsh_crm_report.8.adoc @@ -0,0 +1,15 @@ +:man source: crmsh_crm_report +:man version: 4.6.0 +:man manual: crmsh documentation + +crmsh_crm_report(8) +================== + +NAME +---- +crmsh_crm_report - create report for CRM based clusters (Pacemaker) + + +SEE ALSO +-------- +See "crm help report" or "crm report --help" diff --git a/doc/development.md b/doc/development.md new file mode 100644 index 0000000..2406b4d --- /dev/null +++ b/doc/development.md @@ -0,0 +1,314 @@ +# Notes for developers and contributors + +This is mostly a list of notes that Dejan prepared for me when I +started working on crmsh (me being Kristoffer). I've decided to update +it at least enough to not be completely outdated, so the information +here should be mostly up-to-date for crmsh 2.1. + +## data-manifest + +This file contains a list of all shared data files to install. + +Whenever a file that is to be installed to `/usr/share/crmsh` is added, +for example a cluster script or crmsh template, the `data-manifest` +file needs to be regenerated, by running `./update-data-manifest.sh`. + +## Website + +To build the website, you will need **Asciidoc**, **Pygments** plus +two special lexers for Pygments installed as a separate module. This +module is included in the source tree for crmsh under `contrib`. To +install the module and build the website, do the following: + +``` +cd contrib +sudo python setup.py install +cd .. +cd doc/website-v1 +make +``` + +If everything worked out as it should, the website should now be +generated in `doc/website-v1/gen`. + +## Test suite + +There are two separate test suites for crmsh: + +* `test/unittests` - These are unit tests that test small pieces of + code or functionality. To run these tests, run the `test/run` script + from the project root. + +* `test/testcases` - These are larger integration tests which require + a Pacemaker installation on the machine where the tests are to + run. Usually, we run these tests using the OBS and the `osc` command + line tool: + + 1. Check out the crmsh python package to a directory (usually + `~/build-service/network:ha-clustering:Factory/crmsh`) + + 2. Replace the tarball for crmsh in the OBS project with an archive + built from the current source tree. Replace the version number with + whatever version is the current one on OBS: + + git archive --format=tar --prefix=crmsh-2.3.0+git.1470991992.7deaa3a/ -o <tmpdir>/crmsh-2.3.0+git.1470991992.7deaa3a.tar HEAD + bzip2 <tmpdir>/crmsh-2.3.0+git.1470991992.7deaa3a.tar + cp <tmpdir>/crmsh-2.3.0+git.1470991992.7deaa3a.tar.bz2 ~/build-service/network:ha-clustering:Factory/crmsh/crmsh-2.3.0+git.1470991992.7deaa3a.tar.bz2 + + 3. Build the rpm package for crmsh with the `with_regression_tests` + flag set to 1: + + cd ~/build-service/network:ha-clustering:Factory/crmsh + osc build -d --no-verify --release=1 --define with_regression_tests 1 openSUSE_Tumbleweed x86_64 crmsh.spec + +To simplify this process, there is a utility called `obs` which can be +downloaded here: https://github.com/krig/obs-scripts + +Using the `obs` script, the above is reduced to calling `obs test +factory`, given an appropriate `obs.conf` file. See the README in the +obs-scripts project for more details on using `obs`. + +## Modules + +This is the list of all modules including short descriptions. + +- `crm` + + The program. Tries to detect incompatible python versions or a + missing crmsh module, and report an understandable error message + in either case. + +- `crmsh/main.py` + + This is where execution really starts. Verifies the environment + and detects the pacemaker version. + +- `crmsh/config.py` + + Reads the `crm.conf` configuration file and tries to detect basic + information about where pacemaker is located etc. Some magic is + used to generate an object hierarchy based on the configuration, + so that the rest of the code can access configuration variables + directly. + +- `crmsh/constants.py` + + Various hard-coded constants. Many of these should probably be + read from pacemaker metadata for better compatibility across + different versions. + +- `crmsh/ui_*.py` + + The UI context (`ui_context.py`) parses the input command and + keeps track of which is the current level in the UI. `ui_root.py` + is the root of the UI hierarchy. + +- `crmsh/help.py` + + Reads help from a text file and presents parts of it in + response to the help command. The text file has special + anchors to demarcate help topics and command help text. + +- `doc/crm.8.adoc` + + Online help in asciidoc format. Several help topics (search + for +[[topic_+) and command reference (search for + +[[cmdhelp_+). Every user interface change needs to be + reflected here. _Actually, every user interface change has to + start here_. A source for the +crm(8)+ man page too. + +- `crmsh/cibconfig.py` + + Configuration (CIB) manager. Implements the configure level. + The bigest and the most complex part. There are three major + classes: + + - +CibFactory+: operations on the CIB or parts of it. + + - +CibObject+: every CIB element is implemented in a + subclass of +CibObject+. The configuration consists of a + set of +CibObject+ instances (subclassed, e.g. +CibNode+ or + +CibPrimitive+). + + - +CibObjectSet+: enables operations on sets of CIB + elements. Two subclasses with CLI and XML presentations + of cib elements. Most operations are going via these + subclasses (+show+, +edit+, +save+, +filter+). + +- `crmsh/scripts.py` + + Implements the cluster scripts. Reads multiple kinds of script + definition languages including the XML wizard format used by + Hawk. + +- `crmsh/handles.py` + + A primitive handlebar-style templating language used in cluster + scripts. + +- `crmsh/idmgmt.py` + + CIB id management. Guarantees that all ids are unique. + A helper for CibFactory. + +- `crmsh/parse.py` + + Parses CLI -> XML. + +- `crmsh/cliformat.py` + + Parses XML -> CLI. + + Not as cleanly separated as the CLI parser, mostly a set of + functions called from `cibconfig.py`. + +- `crmsh/clidisplay.py`, `crmsh/term.py` + + Applies colors to terminal output. + +- `crmsh/crm_gv.py` + + Interface to GraphViz. Generates graph specs for dotty(1). + +- `crmsh/cibstatus.py` + + CIB status section editor and manipulator (cibstatus + level). Interface to crm_simulate. + +- `crmsh/ra.py` + + Resource agents interface. + +- `crmsh/rsctest.py` + + Resource tester (configure rsctest command). + +- `crmsh/history.py` + + Cluster history. Interface to logs and other artifacts left + on disk by the cluster. + +- `crmsh/log_patterns.py`, `log_patterns_118.py` + + Pacemaker subsystems' log patterns. For versions earlier than + 1.1.8 and the latter. + +- `crmsh/schema.py`, `pacemaker.py` + + Support for pacemaker RNG schema. + +- `crmsh/cache.py` + + A very rudimentary cache implementation. Used to cache + results of expensive operations (i.e. ra meta). + +- `crmsh/crm_pssh.py` + + Interface to the parallax library for remote SSH commands. + +- `crmsh/corosync.py` + + Parse and edit the `corosync.conf` configuration file. + +- `crmsh/msg.py` + + Messages for users. Can count lines and include line + numbers. Needs refinement. + +- `crmsh/utils.py` + + A bag of useful functions. Needs more order. + +- `crmsh/xmlutil.py` + + A bag of useful XML functions. Needs more order. + +## Code improvements / TODO + +These are some thoughts on how to improve maintainability and +make crmsh nicer. Mostly for people looking at the code, the +users shouldn't notice much (or any) difference. + +Everybody's invited to comment and make further suggestions, in +particular experienced pythonistas. + +### Syntax bug with automatic constraint handling + +See issue on github https://github.com/ClusterLabs/crmsh/issues/140 . + +The problem is the sequence of modifications: crmsh tries to be too +smart and changes the constraint which refers to all members of the +group so that it now refers to the group. But when the group is +then deleted, the constraint is also deleted. + +### Rewrite the hb_report script completely in Python + +Right now, the `hb_report` script is written in bash. This means it +has some duplicated code, for example finding pacemaker binaries, +with crmsh. It also means that it can be difficult to debug and +maintain. It would be better if it was completely implemented in +Python. + +### Python 3 compatibility + +The code is currently only compatible with Python 2.6+. We will need +to port crmsh to Python 3 eventually. The best solution for this is +probably using the six python library which enables code which is +both Python 2 and Python 3-compatible. + +### Validate more using pacemaker schema + +- We have the pacemaker CIB schema available (see schema.py), +however using it is difficult and so it is not used in enough +places. + +### Investigate switching to python-prompt-toolkit + +Either switch crmsh over to using the prompt toolkit for +implementing the interactive mode, or at least look at it +to see what ideas we can lift. + +https://github.com/jonathanslenders/python-prompt-toolkit + +### History transition should be able to save the graph to a file + +See https://github.com/ClusterLabs/crmsh/issues/98 + +### Add support for ordering attribute on resource sets + +See https://github.com/ClusterLabs/crmsh/issues/84 + +### Better version detection + +Be better at detecting and handling the Pacemaker version. +Ensure backwards compatibility, for example with old vs. +new ACL command syntax. + +### Syntax highlighting + +- syntax highlighting is done before producing output, which + is basically wrong and makes code convoluted; it further + makes extra processing more difficult + +- use a python library (pygments seems to be the best + candidate); that should also allow other output formats + (not only terminal) + +- how to extend pygments to understand a new language? it'd + be good to be able to get this _without_ pushing the parser + upstream (that would take _long_ to propagate to + distributions) + +### CibFactory is huge + +- this is a single central CIB class, it'd be good to have it + split into several smaller classes (how?) + +### The element create/update procedure is complex + +- not sure how to improve this + +### Bad namespace separation + +- xmlutil and utils are just a loose collection of functions, +need to be organized better (get rid of 'from xyz import *') diff --git a/doc/profiles.adoc b/doc/profiles.adoc new file mode 100644 index 0000000..7201a81 --- /dev/null +++ b/doc/profiles.adoc @@ -0,0 +1,47 @@ +=== /etc/crm/profiles.yml + +==== Purpose + +YAML file `/etc/crm/profiles.yml` contains Corosync, SBD and Pacemaker parameters for different platforms. + +crmsh bootstrap detects system environment and load the corresponding parameters predefined in this file. + +==== Syntax + +............ +profile_name: + key_name: value +............ + +The valid profile names are: +"microsoft-azure", "google-cloud-platform", "amazon-web-services", "s390", "default" + +`key_name` is a known Corosync, SBD, or Pacemaker parameters, like +`corosync.totem.token` or `sbd.watchdog_timeout`. + +More details about the parameter definitions please refer to the man page of corosync.conf(5), sbd(8). + +Example +............ +default: + corosync.totem.crypto_hash: sha1 + corosync.totem.crypto_cipher: aes256 + corosync.totem.token: 5000 + corosync.totem.join: 60 + corosync.totem.max_messages: 20 + corosync.totem.token_retransmits_before_loss_const: 10 + sbd.watchdog_timeout: 15 + +microsoft-azure: + corosync.totem.token: 30000 + sbd.watchdog_timeout: 60 +............ + +==== How the content of the file is interpreted + +The profiles has the following properties: + +* Profiles are only loaded on bootstrap init node. +* The "default" profile is loaded in the beginning. +* Specific profiles will override the corresponding values in the "default" profile (if the specific environment is detected). +* Users could customize the "default" profile for their needs. For example, those on-premise environments which is not defined yet. diff --git a/doc/releasing-a-new-version.md b/doc/releasing-a-new-version.md new file mode 100644 index 0000000..8141f87 --- /dev/null +++ b/doc/releasing-a-new-version.md @@ -0,0 +1,199 @@ +# Releasing a new version + +A guide to releasing new versions of crmsh. + +## Version scheme + +We follow a somewhat loose version of Semantic Versioning, with a +three-part version number: + + <major>.<minor>.<patch> + +The major version number is increased rarely, arbitrarily and +indicates big changes to the shell. Moving from Python 2 to Python 3 +was such a change, for example. It does not indicate breaking +changes: We try not to make breaking changes at all. If there is a +breaking change, it is hopefully a mistake that would be fixed with a +patch. If not, it should be noted very clearly and probably only +released with a major version number change. + +The minor version number indicates new features and bugfixes, but +hopefully no breaking changes. + +The patch version number indicates bugfixes only, and no breaking +changes. + +## Steps + +1. Updating the changelog + +In `/ChangeLog`, there is a curated list of changes included in this +release. This log should be updated based on the git history. Remove +any updates that are tagged `dev:` or `test:` since these are internal +changes, and clean up the changelog in any other way you might want. + +To get the list of changes since the last release, you can use `git +log` with a custom format. This example gets the changes between 3.0 +and 3.0.1, filtering out any changes tagged `dev:` or `test:`: + + PAGER=cat git log --format="- %s" 3.0.0..3.0.1 | \ + grep -Ev -- '-[ ](dev|test):.*' + +2. Tagging the release + +Using `git tag` you can see the list of existing tags. Depending +on the version being released, you will want to tag the current commit +with that release. Make it a signed tag: + + git tag -s -u <user@example.com> 4.1.0 + +In the tag message I usually just put + + Release 4.1.0 + +(of course, change `4.1.0` to whatever release it is you are tagging. + +Remember to push the tag to the Github repository. Assuming that the +github repository is `origin`, this command should do the trick: + + git push --tags origin + + +3. Release email + +Send a release email to users@clusterlabs.org. Here is the template +that I usually follow with release emails: + +``` +Hello everyone! + +I'm happy to announce the release of crmsh version <VERSION>. + +<DESCRIPTION: some notes about the release> + +There are some other changes in this release as well, see the +ChangeLog for the complete list of changes: + +* https://github.com/ClusterLabs/crmsh/blob/<VERSION>/ChangeLog + +The source code can be downloaded from Github: + +* https://github.com/ClusterLabs/crmsh/releases/tag/<VERSION> + +Packaged versions of crmsh should be available shortly from your +distribution of choice. Development packages for openSUSE Tumbleweed +are available from the Open Build System, here: + +* http://download.opensuse.org/repositories/network:/ha-clustering:/Factory/ + +Archives of the tagged release: + +* https://github.com/ClusterLabs/crmsh/archive/<VERSION>.tar.gz +* https://github.com/ClusterLabs/crmsh/archive/<VERSION>.zip + +As usual, a huge thank you to all contributors and users of crmsh! + +``` + +4. Website update + +The crmsh website is hosted on Github as a github page. The URL to the +website repository is + + https://github.com/crmsh/crmsh.github.io + +The website contents themselves (the sources) are found in the regular +crmsh repository, in the `/doc/website-v1` folder. There is a +`Makefile` in this folder which can be used to regenerate the +website. + +Doing this requires `asciidoc` and `Pygments` to be installed, as well +as the custom Pygments filter `ansiclr`. + +`ansiclr` is found in the `/contrib/pygments_crmsh_lexers` folder, and +can be installed by running `python setup.py install` in the +`/contrib` folder. + +**Note: ansiclr seems to be broken at the moment. Just ignore +it. Everything should still work except some highlighting.** + +To create the news update, copy a previous update (found in +`/doc/website-v1/news`), rename it to an appropriate name based on the +current date, and replace the contents based on the announcement +email. + +Remember to update the title, author and date information at the top +of the news entry, to ensure that it appears correctly on the site. + +To generate the site including the new entry, run + + make + +The new site should now sit in `/doc/website-v1/gen`. To update the +site, using rsync should work: + + rsync -Pavz doc/website-v1/gen/ <path-to-website-checkout>/crmsh.github.io/ + +5. Update `network:ha-clustering:Factory` + +On the Open Build Service, the project +`network:ha-clustering:Factory/crmsh` is used as the development +project for openSUSE Tumbleweed. This project mirrors the state of the +`master` branch in crmsh, but for policy reasons it is not +automatically updated. + +The following steps assumes that you are a maintainer of +`network:ha-clustering:Factory`. If not, you can still make the update +but you will have to branch the `crmsh` package, make the update there +and then submit an update request using `osc submit`. Then a +maintainer will have to review your submission. + +To update the package and submit to `openSUSE:Factory`, the following +steps will do the trick. First, check out a local copy of the crmsh +project: + + osc co network:ha-clustering:Factory crmsh + cd network:ha-clustering:Factory/crmsh + +If you already have a copy, make sure it is up to date: + + osc update + +Update the `_service` file so that the version number reflects the +latest version of the `master` branch in git. + +Pull in the latest changes from git: + + osc service dr + +This will update the spec file and the changes file. Clean up the +changes file if you want (not strictly neccessary), and remove any old +tarball still in the package directory, and then add/remove the +changes to osc: + + osc ar + +Check that everything looks good, and then commit: + + osc diff + osc commit + +Once the package has built successfully on OBS, you can submit to +`openSUSE:Factory`: + + osc submit + +6. Update `network:ha-clustering:Stable` + +If this is a minor release for the latest major release, or a new +major release, we should update the version of crmsh hosted at +`network:ha-clustering:Stable` on OBS. + +Since `master` is probably up to date with this latest version, doing +so should be as simple as submitting crmsh from +`network:ha-clustering:Factory` to `network:ha-clustering:Stable`. + +Once the release is tagged, the announcement email is sent, the +website updated and the packages updated, the release is done. + +Congratulations! diff --git a/doc/sort-doc.py b/doc/sort-doc.py new file mode 100644 index 0000000..87c35a2 --- /dev/null +++ b/doc/sort-doc.py @@ -0,0 +1,82 @@ +# Tool to sort the documentation alphabetically +# Makes a lot of assumptions about the structure of the document it edits +# Looks for special markers that indicate structure + +# prints output to stdout + +# print lines until in a cmdhelp_<section> +# collect all cmdhelp_<section>_<subsection> subsections +# sort and print + +import sys +import re + + +class Sorter(object): + def __init__(self): + self.current_section = None + self.current_subsection = None + self.subsections = [] + self.re_section = re.compile(r'^\[\[cmdhelp_([^_,]+),') + self.re_subsection = re.compile(r'^\[\[cmdhelp_([^_]+)_([^,]+),') + + def beginsection(self, line): + m = self.re_section.match(line) + name = m.group(1) + self.current_section = [name, line] + self.current_subsection = None + self.subsections = [] + return self.insection + + def insection(self, line): + if line.startswith('[[cmdhelp_%s_' % (self.current_section[0])): + return self.beginsubsection(line) + elif line.startswith('[['): + self.finishsection() + return self.preprint(line) + else: + self.current_section[1] += line + return self.insection + + def beginsubsection(self, line): + m = self.re_subsection.match(line) + name = m.group(2) + self.current_subsection = [name, line] + return self.insubsection + + def insubsection(self, line): + if line.startswith('[['): + self.subsections.append(self.current_subsection) + self.current_subsection = None + return self.insection(line) + self.current_subsection[1] += line + return self.insubsection + + def finishsection(self): + if self.current_section: + print self.current_section[1], + for name, text in sorted(self.subsections, key=lambda x: x[0]): + print text, + self.current_section = None + self.subsections = [] + + def preprint(self, line): + if self.re_section.match(line): + return self.beginsection(line) + print line, + return self.preprint + + def run(self, lines): + action = self.preprint + for line in lines: + prevaction = action + action = action(line) + if action is None: + print prevaction + print self.current_section + print self.current_subsection + sys.exit(1) + if self.current_section: + self.finishsection() + +Sorter().run(open(sys.argv[1]).readlines()) diff --git a/doc/website-v1/404.adoc b/doc/website-v1/404.adoc new file mode 100644 index 0000000..926d803 --- /dev/null +++ b/doc/website-v1/404.adoc @@ -0,0 +1,9 @@ +404: Page not found +=================== + +Apologies, but there is nothing here! + +The page you are looking for may have moved. + +* link:/documentation[Documentation] +* link:/faq[Frequently Asked Questions] diff --git a/doc/website-v1/Makefile b/doc/website-v1/Makefile new file mode 100644 index 0000000..8ea2f46 --- /dev/null +++ b/doc/website-v1/Makefile @@ -0,0 +1,145 @@ +ASCIIDOC := asciidoc +CRMCONF := crm.conf +SRC := faq.adoc documentation.adoc development.adoc installation.adoc \ + configuration.adoc about.adoc rsctest-guide.adoc download.adoc \ + history-guide.adoc start-guide.adoc man-1.2.adoc scripts.adoc man-2.0.adoc man-3.adoc man-4.3.adoc +HISTORY_LISTINGS = include/history-guide/nfs-probe-err.typescript \ + include/history-guide/sample-cluster.conf.crm \ + include/history-guide/status-probe-fail.typescript \ + include/history-guide/resource-trace.typescript \ + include/history-guide/stonith-corosync-stopped.typescript \ + include/history-guide/basic-transition.typescript \ + include/history-guide/diff.typescript \ + include/history-guide/info.typescript \ + include/history-guide/resource.typescript \ + include/history-guide/transition-log.typescript +TGT := $(patsubst %.adoc,gen/%/index.html,$(SRC)) +CSS := css/crm.css css/font-awesome.min.css +CSS := $(patsubst %,gen/%,$(CSS)) +ICONS := \ + img/icons/caution.png \ + img/icons/example.png \ + img/icons/home.png \ + img/icons/important.png \ + img/icons/next.png \ + img/icons/note.png \ + img/icons/prev.png \ + img/icons/tip.png \ + img/icons/up.png \ + img/icons/warning.png \ + img/icons/callouts/10.png \ + img/icons/callouts/11.png \ + img/icons/callouts/12.png \ + img/icons/callouts/13.png \ + img/icons/callouts/14.png \ + img/icons/callouts/15.png \ + img/icons/callouts/1.png \ + img/icons/callouts/2.png \ + img/icons/callouts/3.png \ + img/icons/callouts/4.png \ + img/icons/callouts/5.png \ + img/icons/callouts/6.png \ + img/icons/callouts/7.png \ + img/icons/callouts/8.png \ + img/icons/callouts/9.png +IMG := $(ICONS) img/loader.gif img/laptop.png img/servers.gif \ + img/history-guide/sample-cluster.conf.png \ + img/history-guide/smallapache-start.png +IMG := $(patsubst %,gen/%,$(IMG)) +FONTS := fonts/FontAwesome.otf fonts/fontawesome-webfont.eot \ + fonts/fontawesome-webfont.svg fonts/fontawesome-webfont.ttf \ + fonts/fontawesome-webfont.woff +FONTS := $(patsubst %,gen/%,$(FONTS)) +WATCHDIR := watchdir +XDGOPEN := xdg-open +NEWS := $(wildcard news/*.adoc) +NEWSDOC := $(patsubst %.adoc,gen/%/index.html,$(NEWS)) + +.PHONY: all clean deploy open + +all: site + +gen/index.html: index.adoc $(CRMCONF) + @mkdir -p $(dir $@) + @$(ASCIIDOC) --unsafe -b html5 -a icons -a iconsdir=/img/icons -f $(CRMCONF) -o $@ $< + @python ./postprocess.py -o $@ $< + +gen/%/index.html: %.adoc $(CRMCONF) + @mkdir -p $(dir $@) + @$(ASCIIDOC) --unsafe -b html5 -a icons -a iconsdir=/img/icons -f $(CRMCONF) -o $@ $< + @python ./postprocess.py -o $@ $< + +gen/history-guide/index.html: $(HISTORY_LISTINGS) + +gen/man/index.html: ../crm.8.adoc $(CRMCONF) + @mkdir -p $(dir $@) + @$(ASCIIDOC) --unsafe -b html5 -f $(CRMCONF) -o $@ $< + @python ./postprocess.py -o $@ $< + +gen/404.html: 404.adoc $(CRMCONF) + @mkdir -p $(dir $@) + @$(ASCIIDOC) --unsafe -b html5 -f $(CRMCONF) -o $@ $< + @python ./postprocess.py -o $@ $< + +news.adoc: $(NEWS) $(CRMCONF) + @echo "news:" $(NEWS) + python ./make-news.py $@ $(NEWS) + +gen/news/index.html: news.adoc + @mkdir -p $(dir $@) + $(ASCIIDOC) --unsafe -b html5 -f $(CRMCONF) -o $@ $< + @python ./postprocess.py -o $@ $< + +gen/css/%.css: css/%.css + @mkdir -p gen/css + @cp -r $< $@ + @echo "+ $@" + +gen/js/%.js: js/%.js + @mkdir -p gen/js + @cp -r $< $@ + @echo "+ $@" + +gen/img/icons/callouts/%: img/icons/callouts/% + @mkdir -p gen/img/icons/callouts + @cp -r $< $@ + @echo "+ $@" + +gen/img/icons/%: img/icons/% + @mkdir -p gen/img/icons + @cp -r $< $@ + @echo "+ $@" + +gen/img/history-guide/%: img/history-guide/% + @mkdir -p gen/img/history-guide + @cp -r $< $@ + @echo "+ $@" + +gen/img/%: img/% + @mkdir -p gen/img + @cp -r $< $@ + @echo "+ $@" + +gen/fonts/%: fonts/% + @mkdir -p gen/fonts + @cp -r $< $@ + @echo "+ $@" + +gen/atom.xml: $(NEWSDOC) + @echo "atom:" $(NEWSDOC) + python ./make-news.py gen/atom.xml $(NEWS) + +site: gen/atom.xml gen/index.html gen/404.html gen/news/index.html gen/man/index.html $(TGT) $(CSS) $(IMG) $(FONTS) $(NEWSDOC) + @which dos2unix >/dev/null && find gen -name "*.html" -type f -exec dos2unix {} \; + +deploy: site + @echo "TODO: CVS upload" + +open: site + @$(XDGOPEN) gen/index.html + +watch: + @$(WATCHDIR) --verbose --cmd "make" . css img fonts + +clean: + -@$(RM) -rf gen/* news.adoc diff --git a/doc/website-v1/about.adoc b/doc/website-v1/about.adoc new file mode 100644 index 0000000..2656625 --- /dev/null +++ b/doc/website-v1/about.adoc @@ -0,0 +1,19 @@ += About = + +== Authors == + +include::../../AUTHORS[] + +== Site == + +This site was generated from http://asciidoc.org[AsciiDoc] sources. + +The CSS for this site started as a clone of the +bare+ theme by https://github.com/rtomayko/adoc-themes[Ryan Tomayko]. + +Fonts used are https://www.google.com/fonts/specimen/Open+Sans[Open Sans] and http://fontawesome.io[Font Awesome]. + +== License == + +`crmsh` is licensed under the GNU General Public License (GPL). + +For more information, see https://gnu.org/licenses/gpl.html diff --git a/doc/website-v1/configuration.adoc b/doc/website-v1/configuration.adoc new file mode 100644 index 0000000..fb48c93 --- /dev/null +++ b/doc/website-v1/configuration.adoc @@ -0,0 +1,132 @@ += Configuration = + +.Version information +NOTE: This section applies to `crmsh 2.0+` only. + + +`crm` can be configured using both a system-wide configuration file, +and a per-user configuration file. The values set in the user-local +file take precedence over the system-wide settings. + +The global configuration file is usually installed at +`/etc/crm/crm.conf`, and the user-local configuration file at +`~/.config/crm/crm.conf`. + + +== Upgrading from crm 1.x to 2.x == + +The configuration file format and location changed significantly going +from crm 1.x to 2.x. If `crm` cannot find a user-local configuration +file when starting up, it will look for an old-style configuration +file at `~/.crm.rc`. If this file exists, `crm` will prompt the user +asking if the old-style configuration should be automatically +converted to a new-style configuration file. + + +== Format description == + +The `settings` file consists of sections introduced by a `[section]` +header, and followed by `name=value` pairs. + +Leading whitespace is stripped from values. + +Values can contain format strings referring to other values in the +same section. + +Lines starting with `#` or `;` are interpreted as comments. + +Values starting with `$` are interpreted as environment variable +references, and the value will be retrieved from the named environment +variable if set. + +== Example configuration == + +The example configuration below lists all available options and their +default values. + +---------------------- +[core] +editor = $EDITOR +pager = $PAGER +user = +skill_level = expert +sort_elements = yes +check_frequency = always +check_mode = strict +wait = no +add_quotes = yes +manage_children = ask +force = no +debug = no +ptest = ptest, crm_simulate +dotty = dotty +dot = dot + +[path] +sharedir = /usr/share/crmsh +cache = /var/cache/crm +crm_config = /var/lib/pacemaker/cib +crm_daemon_dir = /usr/lib64/pacemaker +crm_daemon_user = hacluster +ocf_root = /usr/lib/ocf +crm_dtd_dir = /usr/share/pacemaker +pe_state_dir = /var/lib/pacemaker/pengine +heartbeat_dir = /var/lib/heartbeat +hb_delnode = /usr/share/heartbeat/hb_delnode +nagios_plugins = /usr/lib64/nagios/plugins + +[color] +style = color +error = red bold +ok = green bold +warn = yellow bold +info = cyan +help_keyword = blue bold underline +help_header = normal bold +help_topic = yellow bold +help_block = cyan +keyword = yellow +identifier = normal +attr_name = cyan +attr_value = red +resource_reference = green +id_reference = green +score = magenta +ticket = magenta +---------------------- + + +== Loading and saving options == + +Options are loaded from the global configuration file first, and the +user-local file second. This means that the user-local options take +precedence over the global configuration. + +When changing an option using the `options` sublevel, the +configuration file is written to disk with the new value. + +== Syntax highlighting == + +By default, `crm` will try to syntax highlight its output when +connected to a TTY. To disable this behavior, set the configuration +value `style = none` in the `[color]` section. + +The available color choices may depend on the terminal used, but +normally include the following: + +---- +black blue green cyan red magenta yellow white +---- + +Colors can be combined with styles: + +---- +bold blink dim reverse underline normal +---- + +== Setting options from the interactive shell == + +Options can be set directly from the interactive shell using the +`options` sublevel. These options will be written to the per-user +configuration file. Note that changing an option in this way may erase +comments added to the configuration file. diff --git a/doc/website-v1/crm.conf b/doc/website-v1/crm.conf new file mode 100644 index 0000000..d1502cf --- /dev/null +++ b/doc/website-v1/crm.conf @@ -0,0 +1,601 @@ +# +# html5.conf +# +# Asciidoc configuration file. +# html5 backend. +# + +[miscellaneous] +outfilesuffix=.html + +[attributes] +basebackend=html +basebackend-html= +basebackend-html5= +b +[replacements2] +# Line break. +(?m)^(.*)\s\+$=\1<br> + +[replacements] +ifdef::asciidoc7compatible[] +# Superscripts. +\^(.+?)\^=<sup>\1</sup> +# Subscripts. +~(.+?)~=<sub>\1</sub> +endif::asciidoc7compatible[] + +[ruler-blockmacro] +<hr> + +[pagebreak-blockmacro] +<div style="page-break-after:always"></div> + +[blockdef-pass] +asciimath-style=template="asciimathblock",subs=() +latexmath-style=template="latexmathblock",subs=() + +[macros] +(?u)^(?P<name>audio|video)::(?P<target>\S*?)(\[(?P<attrlist>.*?)\])$=# +# math macros. +# Special characters are escaped in HTML math markup. +(?su)[\\]?(?P<name>asciimath|latexmath):(?P<subslist>\S*?)\[(?P<passtext>.*?)(?<!\\)\]=[specialcharacters] +(?u)^(?P<name>asciimath|latexmath)::(?P<subslist>\S*?)(\[(?P<passtext>.*?)\])$=#[specialcharacters] + +[asciimath-inlinemacro] +`{passtext}` + +[asciimath-blockmacro] +<div class="mathblock{role? {role}}{unbreakable-option? unbreakable}"{id? id="{id}"}> +<div class="content"> +<div class="title">{title}</div> +`{passtext}` +</div></div> + +[asciimathblock] +<div class="mathblock{role? {role}}{unbreakable-option? unbreakable}"{id? id="{id}"}> +<div class="content"> +<div class="title">{title}</div> +`|` +</div></div> + +[latexmath-inlinemacro] +{passtext} + +[latexmath-blockmacro] +<div class="mathblock{role? {role}}{unbreakable-option? unbreakable}"{id? id="{id}"}> +<div class="content"> +<div class="title">{title}</div> +{passtext} +</div></div> + +[latexmathblock] +<div class="mathblock{role? {role}}{unbreakable-option? unbreakable}"{id? id="{id}"}> +<div class="content"> +<div class="title">{title}</div> +| +</div></div> + +[image-inlinemacro] +<span class="image{role? {role}}"> +<a class="image" href="{link}"> +{data-uri%}<img src="{imagesdir=}{imagesdir?/}{target}" alt="{alt={target}}"{width? width="{width}"}{height? height="{height}"}{title? title="{title}"}> +{data-uri#}<img alt="{alt={target}}"{width? width="{width}"}{height? height="{height}"}{title? title="{title}"} +{data-uri#}{sys:"{python}" -u -c "import mimetypes,base64,sys; print 'src=\"data:'+mimetypes.guess_type(r'{target}')[0]+';base64,'; base64.encode(sys.stdin,sys.stdout)" < "{eval:os.path.join(r"{indir={outdir}}",r"{imagesdir=}",r"{target}")}"}"> +{link#}</a> +</span> + +[image-blockmacro] +<div class="imageblock{style? {style}}{role? {role}}{unbreakable-option? unbreakable}"{id? id="{id}"}{align? style="text-align:{align};"}{float? style="float:{float};"}> +<div class="content"> +<a class="image" href="{link}"> +{data-uri%}<img src="{imagesdir=}{imagesdir?/}{target}" alt="{alt={target}}"{width? width="{width}"}{height? height="{height}"}> +{data-uri#}<img alt="{alt={target}}"{width? width="{width}"}{height? height="{height}"} +{data-uri#}{sys:"{python}" -u -c "import mimetypes,base64,sys; print 'src=\"data:'+mimetypes.guess_type(r'{target}')[0]+';base64,'; base64.encode(sys.stdin,sys.stdout)" < "{eval:os.path.join(r"{indir={outdir}}",r"{imagesdir=}",r"{target}")}"}"> +{link#}</a> +</div> +<div class="title">{caption={figure-caption} {counter:figure-number}. }{title}</div> +</div> + +[audio-blockmacro] +<div class="audioblock{role? {role}}{unbreakable-option? unbreakable}"{id? id="{id}"}> +<div class="title">{caption=}{title}</div> +<div class="content"> +<audio src="{imagesdir=}{imagesdir?/}{target}"{autoplay-option? autoplay}{nocontrols-option! controls}{loop-option? loop}> +Your browser does not support the audio tag. +</audio> +</div></div> + +[video-blockmacro] +<div class="videoblock{role? {role}}{unbreakable-option? unbreakable}"{id? id="{id}"}> +<div class="title">{caption=}{title}</div> +<div class="content"> +<video src="{imagesdir=}{imagesdir?/}{target}"{width? width="{width}"}{height? height="{height}"}{poster? poster="{poster}"}{autoplay-option? autoplay}{nocontrols-option! controls}{loop-option? loop}> +Your browser does not support the video tag. +</video> +</div></div> + +[unfloat-blockmacro] +<div style="clear:both;"></div> + +[toc-blockmacro] +template::[toc] + +[indexterm-inlinemacro] +# Index term. +{empty} + +[indexterm2-inlinemacro] +# Index term. +# Single entry index term that is visible in the primary text flow. +{1} + +[footnote-inlinemacro] +# footnote:[<text>]. +<span class="footnote"><br>[{0}]<br></span> + +[footnoteref-inlinemacro] +# footnoteref:[<id>], create reference to footnote. +{2%}<span class="footnoteref"><br><a href="#_footnote_{1}">[{1}]</a><br></span> +# footnoteref:[<id>,<text>], create footnote with ID. +{2#}<span class="footnote" id="_footnote_{1}"><br>[{2}]<br></span> + +[callout-inlinemacro] +ifndef::icons[] +<b><{index}></b> +endif::icons[] +ifdef::icons[] +ifndef::data-uri[] +<img src="{icon={iconsdir}/callouts/{index}.png}" alt="{index}"> +endif::data-uri[] +ifdef::data-uri[] +<img alt="{index}" src="data:image/png;base64, +{sys:"{python}" -u -c "import base64,sys; base64.encode(sys.stdin,sys.stdout)" < "{eval:os.path.join(r"{indir={outdir}}",r"{icon={iconsdir}/callouts/{index}.png}")}"}"> +endif::data-uri[] +endif::icons[] + +# Comment line macros. +[comment-inlinemacro] +{showcomments#}<br><span class="comment">{passtext}</span><br> + +[comment-blockmacro] +{showcomments#}<p><span class="comment">{passtext}</span></p> + +[literal-inlinemacro] +# Inline literal. +<span class="monospaced">{passtext}</span> + +# List tags. +[listtags-bulleted] +list=<div class="ulist{style? {style}}{compact-option? compact}{role? {role}}"{id? id="{id}"}>{title?<div class="title">{title}</div>}<ul>|</ul></div> +item=<li>|</li> +text=<p>|</p> + +[listtags-numbered] +# The start attribute is not valid XHTML 1.1 but all browsers support it. +list=<div class="olist{style? {style}}{compact-option? compact}{role? {role}}"{id? id="{id}"}>{title?<div class="title">{title}</div>}<ol class="{style}"{start? start="{start}"}>|</ol></div> +item=<li>|</li> +text=<p>|</p> + +[listtags-labeled] +list=<div class="dlist{compact-option? compact}{role? {role}}"{id? id="{id}"}>{title?<div class="title">{title}</div>}<dl>|</dl></div> +entry= +label= +term=<dt class="hdlist1{strong-option? strong}">|</dt> +item=<dd>|</dd> +text=<p>|</p> + +[listtags-horizontal] +list=<div class="hdlist{compact-option? compact}{role? {role}}"{id? id="{id}"}>{title?<div class="title">{title}</div>}<table>{labelwidth?<col width="{labelwidth}%">}{itemwidth?<col width="{itemwidth}%">}|</table></div> +label=<td class="hdlist1{strong-option? strong}">|</td> +term=|<br> +entry=<tr>|</tr> +item=<td class="hdlist2">|</td> +text=<p style="margin-top: 0;">|</p> + +[listtags-qanda] +list=<div class="qlist{style? {style}}{role? {role}}"{id? id="{id}"}>{title?<div class="title">{title}</div>}<ol>|</ol></div> +entry=<li>|</li> +label= +term=<p><em>|</em></p> +item= +text=<p>|</p> + +[listtags-callout] +ifndef::icons[] +list=<div class="colist{style? {style}}{role? {role}}"{id? id="{id}"}>{title?<div class="title">{title}</div>}<ol>|</ol></div> +item=<li>|</li> +text=<p>|</p> +endif::icons[] +ifdef::icons[] +list=<div class="colist{style? {style}}{role? {role}}"{id? id="{id}"}>{title?<div class="title">{title}</div>}<table>|</table></div> +ifndef::data-uri[] +item=<tr><td><img src="{iconsdir}/callouts/{listindex}.png" alt="{listindex}"></td><td>|</td></tr> +endif::data-uri[] +ifdef::data-uri[] +item=<tr><td><img alt="{listindex}" src="data:image/png;base64, {sys:"{python}" -u -c "import base64,sys; base64.encode(sys.stdin,sys.stdout)" < "{eval:os.path.join(r"{indir={outdir}}",r"{icon={iconsdir}/callouts/{listindex}.png}")}"}"></td><td>|</td></tr> +endif::data-uri[] +text=| +endif::icons[] + +[listtags-glossary] +list=<div class="dlist{style? {style}}{role? {role}}"{id? id="{id}"}>{title?<div class="title">{title}</div>}<dl>|</dl></div> +label= +entry= +term=<dt>|</dt> +item=<dd>|</dd> +text=<p>|</p> + +[listtags-bibliography] +list=<div class="ulist{style? {style}}{role? {role}}"{id? id="{id}"}>{title?<div class="title">{title}</div>}<ul>|</ul></div> +item=<li>|</li> +text=<p>|</p> + +[tags] +# Quoted text. +emphasis=<em>{1?<span class="{1}">}|{1?</span>}</em> +strong=<strong>{1?<span class="{1}">}|{1?</span>}</strong> +monospaced=<span class="monospaced{1? {1}}">|</span> +singlequoted={lsquo}{1?<span class="{1}">}|{1?</span>}{rsquo} +doublequoted={ldquo}{1?<span class="{1}">}|{1?</span>}{rdquo} +unquoted={1?<span class="{1}">}|{1?</span>} +superscript=<sup>{1?<span class="{1}">}|{1?</span>}</sup> +subscript=<sub>{1?<span class="{1}">}|{1?</span>}</sub> + +ifdef::deprecated-quotes[] +# Override with deprecated quote attributes. +emphasis={role?<span class="{role}">}<em{1,2,3? style="}{1?color:{1};}{2?background-color:{2};}{3?font-size:{3}em;}{1,2,3?"}>|</em>{role?</span>} +strong={role?<span class="{role}">}<strong{1,2,3? style="}{1?color:{1};}{2?background-color:{2};}{3?font-size:{3}em;}{1,2,3?"}>|</strong>{role?</span>} +monospaced=<span class="monospaced{role? {role}}"{1,2,3? style="}{1?color:{1};}{2?background-color:{2};}{3?font-size:{3}em;}{1,2,3?"}>|</span> +singlequoted={role?<span class="{role}">}{1,2,3?<span style="}{1?color:{1};}{2?background-color:{2};}{3?font-size:{3}em;}{1,2,3?">}{amp}#8216;|{amp}#8217;{1,2,3?</span>}{role?</span>} +doublequoted={role?<span class="{role}">}{1,2,3?<span style="}{1?color:{1};}{2?background-color:{2};}{3?font-size:{3}em;}{1,2,3?">}{amp}#8220;|{amp}#8221;{1,2,3?</span>}{role?</span>} +unquoted={role?<span class="{role}">}{1,2,3?<span style="{1?color:{1};}{2?background-color:{2};}{3?font-size:{3}em;}">}|{1,2,3?</span>}{role?</span>} +superscript={role?<span class="{role}">}<sup{1,2,3? style="}{1?color:{1};}{2?background-color:{2};}{3?font-size:{3}em;}{1,2,3?"}>|</sup>{role?</span>} +subscript={role?<span class="{role}">}<sub{1,2,3? style="}{1?color:{1};}{2?background-color:{2};}{3?font-size:{3}em;}{1,2,3?"}>|</sub>{role?</span>} +endif::deprecated-quotes[] + +# Inline macros +[http-inlinemacro] +<a href="{name}:{target}">{0={name}:{target}}</a> +[https-inlinemacro] +<a href="{name}:{target}">{0={name}:{target}}</a> +[ftp-inlinemacro] +<a href="{name}:{target}">{0={name}:{target}}</a> +[file-inlinemacro] +<a href="{name}:{target}">{0={name}:{target}}</a> +[irc-inlinemacro] +<a href="{name}:{target}">{0={name}:{target}}</a> +[mailto-inlinemacro] +<a href="mailto:{target}">{0={target}}</a> +[link-inlinemacro] +<a href="{target}">{0={target}}</a> +[callto-inlinemacro] +<a href="{name}:{target}">{0={target}}</a> +# anchor:id[text] +[anchor-inlinemacro] +<a id="{target}"></a> +# [[id,text]] +[anchor2-inlinemacro] +<a id="{1}"></a> +# [[[id]]] +[anchor3-inlinemacro] +<a id="{1}"></a>[{1}] +# xref:id[text] +[xref-inlinemacro] +<a href="#{target}">{0=[{target}]}</a> +# <<id,text>> +[xref2-inlinemacro] +<a href="#{1}">{2=[{1}]}</a> + +# Special word substitution. +[emphasizedwords] +<em>{words}</em> +[monospacedwords] +<span class="monospaced">{words}</span> +[strongwords] +<strong>{words}</strong> + +# Paragraph substitution. +[paragraph] +<div class="paragraph{role? {role}}{unbreakable-option? unbreakable}"{id? id="{id}"}>{title?<div class="title">{title}</div>}<p> +| +</p></div> + +[admonitionparagraph] +template::[admonitionblock] + +# Delimited blocks. +[listingblock] +<div class="listingblock{role? {role}}{unbreakable-option? unbreakable}"{id? id="{id}"}> +<div class="title">{caption=}{title}</div> +<div class="content monospaced"> +<pre> +| +</pre> +</div></div> + +[literalblock] +<div class="literalblock{role? {role}}{unbreakable-option? unbreakable}"{id? id="{id}"}> +<div class="title">{title}</div> +<div class="content monospaced"> +<pre> +| +</pre> +</div></div> + +[sidebarblock] +<div class="sidebarblock{role? {role}}{unbreakable-option? unbreakable}"{id? id="{id}"}> +<div class="content"> +<div class="title">{title}</div> +| +</div></div> + +[openblock] +<div class="openblock{role? {role}}{unbreakable-option? unbreakable}"{id? id="{id}"}> +<div class="title">{title}</div> +<div class="content"> +| +</div></div> + +[partintroblock] +template::[openblock] + +[abstractblock] +template::[quoteblock] + +[quoteblock] +<div class="quoteblock{role? {role}}{unbreakable-option? unbreakable}"{id? id="{id}"}> +<div class="title">{title}</div> +<div class="content"> +| +</div> +<div class="attribution"> +<em>{citetitle}</em>{attribution?<br>} +— {attribution} +</div></div> + +[verseblock] +<div class="verseblock{role? {role}}{unbreakable-option? unbreakable}"{id? id="{id}"}> +<div class="title">{title}</div> +<pre class="content"> +| +</pre> +<div class="attribution"> +<em>{citetitle}</em>{attribution?<br>} +— {attribution} +</div></div> + +[exampleblock] +<div class="exampleblock{role? {role}}{unbreakable-option? unbreakable}"{id? id="{id}"}> +<div class="title">{caption={example-caption} {counter:example-number}. }{title}</div> +<div class="content"> +| +</div></div> + +[admonitionblock] +<div class="admonitionblock{role? {role}}{unbreakable-option? unbreakable}"{id? id="{id}"}> +<table><tr> +<td class="icon"> +{data-uri%}{icons#}<img src="{icon={iconsdir}/{name}.png}" alt="{caption}"> +{data-uri#}{icons#}<img alt="{caption}" src="data:image/png;base64, +{data-uri#}{icons#}{sys:"{python}" -u -c "import base64,sys; base64.encode(sys.stdin,sys.stdout)" < "{eval:os.path.join(r"{indir={outdir}}",r"{icon={iconsdir}/{name}.png}")}"}"> +{icons%}<div class="title">{caption}</div> +</td> +<td class="content"> +<div class="title">{title}</div> +| +</td> +</tr></table> +</div> + +# Tables. +[tabletags-default] +colspec=<col{autowidth-option! style="width:{colpcwidth}%;"}> +bodyrow=<tr>|</tr> +headdata=<th class="tableblock halign-{halign=left} valign-{valign=top}" {colspan@1::colspan="{colspan}" }{rowspan@1::rowspan="{rowspan}" }>|</th> +bodydata=<td class="tableblock halign-{halign=left} valign-{valign=top}" {colspan@1::colspan="{colspan}" }{rowspan@1::rowspan="{rowspan}" }>|</td> +paragraph=<p class="tableblock">|</p> + +[tabletags-header] +paragraph=<p class="tableblock header">|</p> + +[tabletags-emphasis] +paragraph=<p class="tableblock"><em>|</em></p> + +[tabletags-strong] +paragraph=<p class="tableblock"><strong>|</strong></p> + +[tabletags-monospaced] +paragraph=<p class="tableblock monospaced">|</p> + +[tabletags-verse] +bodydata=<td class="tableblock halign-{halign=left} valign-{valign=top}" {colspan@1::colspan="{colspan}" }{rowspan@1::rowspan="{rowspan}" }><div class="verse">|</div></td> +paragraph= + +[tabletags-literal] +bodydata=<td class="tableblock halign-{halign=left} valign-{valign=top}" {colspan@1::colspan="{colspan}" }{rowspan@1::rowspan="{rowspan}" }><div class="literal monospaced"><pre>|</pre></div></td> +paragraph= + +[tabletags-asciidoc] +bodydata=<td class="tableblock halign-{halign=left} valign-{valign=top}" {colspan@1::colspan="{colspan}" }{rowspan@1::rowspan="{rowspan}" }><div>|</div></td> +paragraph= + +[table] +<table class="tableblock frame-{frame=all} grid-{grid=all}{role? {role}}{unbreakable-option? unbreakable}"{id? id="{id}"} +style=" +margin-left:{align@left:0}{align@center|right:auto}; margin-right:{align@left|center:auto}{align@right:0}; +float:{float}; +{autowidth-option%}width:{tablepcwidth}%; +{autowidth-option#}{width#style=width:{tablepcwidth}%;} +"> +<caption class="title">{caption={table-caption} {counter:table-number}. }{title}</caption> +{colspecs} +{headrows#}<thead> +{headrows} +{headrows#}</thead> +{footrows#}<tfoot> +{footrows} +{footrows#}</tfoot> +<tbody> +{bodyrows} +</tbody> +</table> + +#-------------------------------------------------------------------- +# Deprecated old table definitions. +# + +[miscellaneous] +# Screen width in pixels. +pagewidth=800 +pageunits=px + +[old_tabledef-default] +template=old_table +colspec=<col style="width:{colwidth}{pageunits};" /> +bodyrow=<tr>|</tr> +headdata=<th class="tableblock halign-{colalign=left}">|</th> +footdata=<td class="tableblock halign-{colalign=left}">|</td> +bodydata=<td class="tableblock halign-{colalign=left}">|</td> + +[old_table] +<table class="tableblock frame-{frame=all} grid-{grid=all}"{id? id="{id}"}> +<caption class="title">{caption={table-caption}}{title}</caption> +{colspecs} +{headrows#}<thead> +{headrows} +{headrows#}</thead> +{footrows#}<tfoot> +{footrows} +{footrows#}</tfoot> +<tbody style="vertical-align:top;"> +{bodyrows} +</tbody> +</table> + +# End of deprecated old table definitions. +#-------------------------------------------------------------------- + +[floatingtitle] +<h{level@0:1}{level@1:2}{level@2:3}{level@3:4}{level@4:5}{id? id="{id}"} class="float">{title}</h{level@0:1}{level@1:2}{level@2:3}{level@3:4}{level@4:5}> + +[preamble] +# Untitled elements between header and first section title. +<div id="preamble"> +<div class="sectionbody"> +| +</div> +</div> + +# Document sections. +[sect0] +<h1{id? id="{id}"}>{title}</h1> +| + +[sect1] +<div class="sect1{style? {style}}{role? {role}}"> +<h2{id? id="{id}"}>{numbered?{sectnum} }{title}</h2> +<div class="sectionbody"> +| +</div> +</div> + +[sect2] +<div class="sect2{style? {style}}{role? {role}}"> +<h3{id? id="{id}"}>{numbered?{sectnum} }{title}</h3> +| +</div> + +[sect3] +<div class="sect3{style? {style}}{role? {role}}"> +<h4{id? id="{id}"}>{numbered?{sectnum} }{title}</h4> +| +</div> + +[sect4] +<div class="sect4{style? {style}}{role? {role}}"> +<h5{id? id="{id}"}>{title}</h5> +| +</div> + +[appendix] +<div class="sect1{style? {style}}{role? {role}}"> +<h2{id? id="{id}"}>{numbered?{sectnum} }{appendix-caption} {counter:appendix-number:A}: {title}</h2> +<div class="sectionbody"> +| +</div> +</div> + +[toc] +<div id="toc"> + <div id="toctitle">{toc-title}</div> + <noscript><p><b>JavaScript must be enabled in your browser to display the table of contents.</b></p></noscript> +</div> + +[header] +<!DOCTYPE html> +<html lang="{lang=en}"> +<head> +<meta http-equiv="Content-Type" content="text/html; charset={encoding}"> +<meta name="generator" content="AsciiDoc {asciidoc-version}"> +<meta name="description" content="{description}"> +<meta name="keywords" content="{keywords}"> +<title>crmsh - {title}</title> +{title%}<title>crmsh - {doctitle=}</title> +<link rel="stylesheet" href="/css/font-awesome.min.css"> +<link rel="stylesheet" href="/css/crm.css" type="text/css"> +<link href='//fonts.googleapis.com/css?family=Open+Sans:400,700' rel='stylesheet' type='text/css'> +<link href="/atom.xml" type="application/atom+xml" rel="alternate" title="crmsh atom feed"> +</head> +<body> +<div id="header"> +<h1> +<a href="/"><span class="fa-stack"> + <i class="fa fa-square fa-stack-2x"></i> + <i class="fa fa-terminal fa-stack-1x fa-inverse"></i> +</span>crmsh</a> +</h1> + +<div id="topbar-small"> +<ul> +<li><a href="/news" title="News"><i class="fa fa-rss fa-2x"></i></a></li> +<li><a href="/documentation" title="Documentation"><i class="fa fa-book fa-2x"></i></a></li> +<li><a href="/download" title="Download"><i class="fa fa-download fa-2x"></i></a></li> +<li><a href="/development" title="Development"><i class="fa fa-code-fork fa-2x"></i></a></li> +<li><a href="/about" title="About"><i class="fa fa-question fa-2x"></i></a></li> +</ul> +</div> + +<div id="topbar"> +<ul> +<li><a href="/news">News</a></li> +<li><a href="/documentation">Documentation</a></li> +<li><a href="/download">Download</a></li> +<li><a href="/development">Development</a></li> +<li><a href="/about">About</a></li> +</ul> +</div> +</div> +<!--TOC--> +<div id="container"> +<div id="content"> +<h1>{doctitle}</h1> + +[footer] +</div> +</div> +<div id="footer"> +<div id="footer-text"> +</div> +</div> + +<a href="https://github.com/ClusterLabs/crmsh"><img style="position: absolute; top: 0; right: 0; border: 0;" src="https://camo.githubusercontent.com/a6677b08c955af8400f44c6298f40e7d19cc5b2d/68747470733a2f2f73332e616d617a6f6e6177732e636f6d2f6769746875622f726962626f6e732f666f726b6d655f72696768745f677261795f3664366436642e706e67" alt="Fork me on GitHub" data-canonical-src="https://s3.amazonaws.com/github/ribbons/forkme_right_gray_6d6d6d.png"></a> + +</body> +</html> + +ifdef::doctype-manpage[] +[synopsis] +template::[sect1] +endif::doctype-manpage[] + diff --git a/doc/website-v1/crmold.conf b/doc/website-v1/crmold.conf new file mode 100644 index 0000000..271d88d --- /dev/null +++ b/doc/website-v1/crmold.conf @@ -0,0 +1,602 @@ +# +# html5.conf +# +# Asciidoc configuration file. +# html5 backend. +# + +[miscellaneous] +outfilesuffix=.html + +[attributes] +basebackend=html +basebackend-html= +basebackend-html5= +b +[replacements2] +# Line break. +(?m)^(.*)\s\+$=\1<br> + +[replacements] +ifdef::asciidoc7compatible[] +# Superscripts. +\^(.+?)\^=<sup>\1</sup> +# Subscripts. +~(.+?)~=<sub>\1</sub> +endif::asciidoc7compatible[] + +[ruler-blockmacro] +<hr> + +[pagebreak-blockmacro] +<div style="page-break-after:always"></div> + +[blockdef-pass] +asciimath-style=template="asciimathblock",subs=() +latexmath-style=template="latexmathblock",subs=() + +[macros] +(?u)^(?P<name>audio|video)::(?P<target>\S*?)(\[(?P<attrlist>.*?)\])$=# +# math macros. +# Special characters are escaped in HTML math markup. +(?su)[\\]?(?P<name>asciimath|latexmath):(?P<subslist>\S*?)\[(?P<passtext>.*?)(?<!\\)\]=[specialcharacters] +(?u)^(?P<name>asciimath|latexmath)::(?P<subslist>\S*?)(\[(?P<passtext>.*?)\])$=#[specialcharacters] + +[asciimath-inlinemacro] +`{passtext}` + +[asciimath-blockmacro] +<div class="mathblock{role? {role}}{unbreakable-option? unbreakable}"{id? id="{id}"}> +<div class="content"> +<div class="title">{title}</div> +`{passtext}` +</div></div> + +[asciimathblock] +<div class="mathblock{role? {role}}{unbreakable-option? unbreakable}"{id? id="{id}"}> +<div class="content"> +<div class="title">{title}</div> +`|` +</div></div> + +[latexmath-inlinemacro] +{passtext} + +[latexmath-blockmacro] +<div class="mathblock{role? {role}}{unbreakable-option? unbreakable}"{id? id="{id}"}> +<div class="content"> +<div class="title">{title}</div> +{passtext} +</div></div> + +[latexmathblock] +<div class="mathblock{role? {role}}{unbreakable-option? unbreakable}"{id? id="{id}"}> +<div class="content"> +<div class="title">{title}</div> +| +</div></div> + +[image-inlinemacro] +<span class="image{role? {role}}"> +<a class="image" href="{link}"> +{data-uri%}<img src="{imagesdir=}{imagesdir?/}{target}" alt="{alt={target}}"{width? width="{width}"}{height? height="{height}"}{title? title="{title}"}> +{data-uri#}<img alt="{alt={target}}"{width? width="{width}"}{height? height="{height}"}{title? title="{title}"} +{data-uri#}{sys:"{python}" -u -c "import mimetypes,base64,sys; print 'src=\"data:'+mimetypes.guess_type(r'{target}')[0]+';base64,'; base64.encode(sys.stdin,sys.stdout)" < "{eval:os.path.join(r"{indir={outdir}}",r"{imagesdir=}",r"{target}")}"}"> +{link#}</a> +</span> + +[image-blockmacro] +<div class="imageblock{style? {style}}{role? {role}}{unbreakable-option? unbreakable}"{id? id="{id}"}{align? style="text-align:{align};"}{float? style="float:{float};"}> +<div class="content"> +<a class="image" href="{link}"> +{data-uri%}<img src="{imagesdir=}{imagesdir?/}{target}" alt="{alt={target}}"{width? width="{width}"}{height? height="{height}"}> +{data-uri#}<img alt="{alt={target}}"{width? width="{width}"}{height? height="{height}"} +{data-uri#}{sys:"{python}" -u -c "import mimetypes,base64,sys; print 'src=\"data:'+mimetypes.guess_type(r'{target}')[0]+';base64,'; base64.encode(sys.stdin,sys.stdout)" < "{eval:os.path.join(r"{indir={outdir}}",r"{imagesdir=}",r"{target}")}"}"> +{link#}</a> +</div> +<div class="title">{caption={figure-caption} {counter:figure-number}. }{title}</div> +</div> + +[audio-blockmacro] +<div class="audioblock{role? {role}}{unbreakable-option? unbreakable}"{id? id="{id}"}> +<div class="title">{caption=}{title}</div> +<div class="content"> +<audio src="{imagesdir=}{imagesdir?/}{target}"{autoplay-option? autoplay}{nocontrols-option! controls}{loop-option? loop}> +Your browser does not support the audio tag. +</audio> +</div></div> + +[video-blockmacro] +<div class="videoblock{role? {role}}{unbreakable-option? unbreakable}"{id? id="{id}"}> +<div class="title">{caption=}{title}</div> +<div class="content"> +<video src="{imagesdir=}{imagesdir?/}{target}"{width? width="{width}"}{height? height="{height}"}{poster? poster="{poster}"}{autoplay-option? autoplay}{nocontrols-option! controls}{loop-option? loop}> +Your browser does not support the video tag. +</video> +</div></div> + +[unfloat-blockmacro] +<div style="clear:both;"></div> + +[toc-blockmacro] +template::[toc] + +[indexterm-inlinemacro] +# Index term. +{empty} + +[indexterm2-inlinemacro] +# Index term. +# Single entry index term that is visible in the primary text flow. +{1} + +[footnote-inlinemacro] +# footnote:[<text>]. +<span class="footnote"><br>[{0}]<br></span> + +[footnoteref-inlinemacro] +# footnoteref:[<id>], create reference to footnote. +{2%}<span class="footnoteref"><br><a href="#_footnote_{1}">[{1}]</a><br></span> +# footnoteref:[<id>,<text>], create footnote with ID. +{2#}<span class="footnote" id="_footnote_{1}"><br>[{2}]<br></span> + +[callout-inlinemacro] +ifndef::icons[] +<b><{index}></b> +endif::icons[] +ifdef::icons[] +ifndef::data-uri[] +<img src="{icon={iconsdir}/callouts/{index}.png}" alt="{index}"> +endif::data-uri[] +ifdef::data-uri[] +<img alt="{index}" src="data:image/png;base64, +{sys:"{python}" -u -c "import base64,sys; base64.encode(sys.stdin,sys.stdout)" < "{eval:os.path.join(r"{indir={outdir}}",r"{icon={iconsdir}/callouts/{index}.png}")}"}"> +endif::data-uri[] +endif::icons[] + +# Comment line macros. +[comment-inlinemacro] +{showcomments#}<br><span class="comment">{passtext}</span><br> + +[comment-blockmacro] +{showcomments#}<p><span class="comment">{passtext}</span></p> + +[literal-inlinemacro] +# Inline literal. +<span class="monospaced">{passtext}</span> + +# List tags. +[listtags-bulleted] +list=<div class="ulist{style? {style}}{compact-option? compact}{role? {role}}"{id? id="{id}"}>{title?<div class="title">{title}</div>}<ul>|</ul></div> +item=<li>|</li> +text=<p>|</p> + +[listtags-numbered] +# The start attribute is not valid XHTML 1.1 but all browsers support it. +list=<div class="olist{style? {style}}{compact-option? compact}{role? {role}}"{id? id="{id}"}>{title?<div class="title">{title}</div>}<ol class="{style}"{start? start="{start}"}>|</ol></div> +item=<li>|</li> +text=<p>|</p> + +[listtags-labeled] +list=<div class="dlist{compact-option? compact}{role? {role}}"{id? id="{id}"}>{title?<div class="title">{title}</div>}<dl>|</dl></div> +entry= +label= +term=<dt class="hdlist1{strong-option? strong}">|</dt> +item=<dd>|</dd> +text=<p>|</p> + +[listtags-horizontal] +list=<div class="hdlist{compact-option? compact}{role? {role}}"{id? id="{id}"}>{title?<div class="title">{title}</div>}<table>{labelwidth?<col width="{labelwidth}%">}{itemwidth?<col width="{itemwidth}%">}|</table></div> +label=<td class="hdlist1{strong-option? strong}">|</td> +term=|<br> +entry=<tr>|</tr> +item=<td class="hdlist2">|</td> +text=<p style="margin-top: 0;">|</p> + +[listtags-qanda] +list=<div class="qlist{style? {style}}{role? {role}}"{id? id="{id}"}>{title?<div class="title">{title}</div>}<ol>|</ol></div> +entry=<li>|</li> +label= +term=<p><em>|</em></p> +item= +text=<p>|</p> + +[listtags-callout] +ifndef::icons[] +list=<div class="colist{style? {style}}{role? {role}}"{id? id="{id}"}>{title?<div class="title">{title}</div>}<ol>|</ol></div> +item=<li>|</li> +text=<p>|</p> +endif::icons[] +ifdef::icons[] +list=<div class="colist{style? {style}}{role? {role}}"{id? id="{id}"}>{title?<div class="title">{title}</div>}<table>|</table></div> +ifndef::data-uri[] +item=<tr><td><img src="{iconsdir}/callouts/{listindex}.png" alt="{listindex}"></td><td>|</td></tr> +endif::data-uri[] +ifdef::data-uri[] +item=<tr><td><img alt="{listindex}" src="data:image/png;base64, {sys:"{python}" -u -c "import base64,sys; base64.encode(sys.stdin,sys.stdout)" < "{eval:os.path.join(r"{indir={outdir}}",r"{icon={iconsdir}/callouts/{listindex}.png}")}"}"></td><td>|</td></tr> +endif::data-uri[] +text=| +endif::icons[] + +[listtags-glossary] +list=<div class="dlist{style? {style}}{role? {role}}"{id? id="{id}"}>{title?<div class="title">{title}</div>}<dl>|</dl></div> +label= +entry= +term=<dt>|</dt> +item=<dd>|</dd> +text=<p>|</p> + +[listtags-bibliography] +list=<div class="ulist{style? {style}}{role? {role}}"{id? id="{id}"}>{title?<div class="title">{title}</div>}<ul>|</ul></div> +item=<li>|</li> +text=<p>|</p> + +[tags] +# Quoted text. +emphasis=<em>{1?<span class="{1}">}|{1?</span>}</em> +strong=<strong>{1?<span class="{1}">}|{1?</span>}</strong> +monospaced=<span class="monospaced{1? {1}}">|</span> +singlequoted={lsquo}{1?<span class="{1}">}|{1?</span>}{rsquo} +doublequoted={ldquo}{1?<span class="{1}">}|{1?</span>}{rdquo} +unquoted={1?<span class="{1}">}|{1?</span>} +superscript=<sup>{1?<span class="{1}">}|{1?</span>}</sup> +subscript=<sub>{1?<span class="{1}">}|{1?</span>}</sub> + +ifdef::deprecated-quotes[] +# Override with deprecated quote attributes. +emphasis={role?<span class="{role}">}<em{1,2,3? style="}{1?color:{1};}{2?background-color:{2};}{3?font-size:{3}em;}{1,2,3?"}>|</em>{role?</span>} +strong={role?<span class="{role}">}<strong{1,2,3? style="}{1?color:{1};}{2?background-color:{2};}{3?font-size:{3}em;}{1,2,3?"}>|</strong>{role?</span>} +monospaced=<span class="monospaced{role? {role}}"{1,2,3? style="}{1?color:{1};}{2?background-color:{2};}{3?font-size:{3}em;}{1,2,3?"}>|</span> +singlequoted={role?<span class="{role}">}{1,2,3?<span style="}{1?color:{1};}{2?background-color:{2};}{3?font-size:{3}em;}{1,2,3?">}{amp}#8216;|{amp}#8217;{1,2,3?</span>}{role?</span>} +doublequoted={role?<span class="{role}">}{1,2,3?<span style="}{1?color:{1};}{2?background-color:{2};}{3?font-size:{3}em;}{1,2,3?">}{amp}#8220;|{amp}#8221;{1,2,3?</span>}{role?</span>} +unquoted={role?<span class="{role}">}{1,2,3?<span style="{1?color:{1};}{2?background-color:{2};}{3?font-size:{3}em;}">}|{1,2,3?</span>}{role?</span>} +superscript={role?<span class="{role}">}<sup{1,2,3? style="}{1?color:{1};}{2?background-color:{2};}{3?font-size:{3}em;}{1,2,3?"}>|</sup>{role?</span>} +subscript={role?<span class="{role}">}<sub{1,2,3? style="}{1?color:{1};}{2?background-color:{2};}{3?font-size:{3}em;}{1,2,3?"}>|</sub>{role?</span>} +endif::deprecated-quotes[] + +# Inline macros +[http-inlinemacro] +<a href="{name}:{target}">{0={name}:{target}}</a> +[https-inlinemacro] +<a href="{name}:{target}">{0={name}:{target}}</a> +[ftp-inlinemacro] +<a href="{name}:{target}">{0={name}:{target}}</a> +[file-inlinemacro] +<a href="{name}:{target}">{0={name}:{target}}</a> +[irc-inlinemacro] +<a href="{name}:{target}">{0={name}:{target}}</a> +[mailto-inlinemacro] +<a href="mailto:{target}">{0={target}}</a> +[link-inlinemacro] +<a href="{target}">{0={target}}</a> +[callto-inlinemacro] +<a href="{name}:{target}">{0={target}}</a> +# anchor:id[text] +[anchor-inlinemacro] +<a id="{target}"></a> +# [[id,text]] +[anchor2-inlinemacro] +<a id="{1}"></a> +# [[[id]]] +[anchor3-inlinemacro] +<a id="{1}"></a>[{1}] +# xref:id[text] +[xref-inlinemacro] +<a href="#{target}">{0=[{target}]}</a> +# <<id,text>> +[xref2-inlinemacro] +<a href="#{1}">{2=[{1}]}</a> + +# Special word substitution. +[emphasizedwords] +<em>{words}</em> +[monospacedwords] +<span class="monospaced">{words}</span> +[strongwords] +<strong>{words}</strong> + +# Paragraph substitution. +[paragraph] +<div class="paragraph{role? {role}}{unbreakable-option? unbreakable}"{id? id="{id}"}>{title?<div class="title">{title}</div>}<p> +| +</p></div> + +[admonitionparagraph] +template::[admonitionblock] + +# Delimited blocks. +[listingblock] +<div class="listingblock{role? {role}}{unbreakable-option? unbreakable}"{id? id="{id}"}> +<div class="title">{caption=}{title}</div> +<div class="content monospaced"> +<pre> +| +</pre> +</div></div> + +[literalblock] +<div class="literalblock{role? {role}}{unbreakable-option? unbreakable}"{id? id="{id}"}> +<div class="title">{title}</div> +<div class="content monospaced"> +<pre> +| +</pre> +</div></div> + +[sidebarblock] +<div class="sidebarblock{role? {role}}{unbreakable-option? unbreakable}"{id? id="{id}"}> +<div class="content"> +<div class="title">{title}</div> +| +</div></div> + +[openblock] +<div class="openblock{role? {role}}{unbreakable-option? unbreakable}"{id? id="{id}"}> +<div class="title">{title}</div> +<div class="content"> +| +</div></div> + +[partintroblock] +template::[openblock] + +[abstractblock] +template::[quoteblock] + +[quoteblock] +<div class="quoteblock{role? {role}}{unbreakable-option? unbreakable}"{id? id="{id}"}> +<div class="title">{title}</div> +<div class="content"> +| +</div> +<div class="attribution"> +<em>{citetitle}</em>{attribution?<br>} +— {attribution} +</div></div> + +[verseblock] +<div class="verseblock{role? {role}}{unbreakable-option? unbreakable}"{id? id="{id}"}> +<div class="title">{title}</div> +<pre class="content"> +| +</pre> +<div class="attribution"> +<em>{citetitle}</em>{attribution?<br>} +— {attribution} +</div></div> + +[exampleblock] +<div class="exampleblock{role? {role}}{unbreakable-option? unbreakable}"{id? id="{id}"}> +<div class="title">{caption={example-caption} {counter:example-number}. }{title}</div> +<div class="content"> +| +</div></div> + +[admonitionblock] +<div class="admonitionblock{role? {role}}{unbreakable-option? unbreakable}"{id? id="{id}"}> +<table><tr> +<td class="icon"> +{data-uri%}{icons#}<img src="{icon={iconsdir}/{name}.png}" alt="{caption}"> +{data-uri#}{icons#}<img alt="{caption}" src="data:image/png;base64, +{data-uri#}{icons#}{sys:"{python}" -u -c "import base64,sys; base64.encode(sys.stdin,sys.stdout)" < "{eval:os.path.join(r"{indir={outdir}}",r"{icon={iconsdir}/{name}.png}")}"}"> +{icons%}<div class="title">{caption}</div> +</td> +<td class="content"> +<div class="title">{title}</div> +| +</td> +</tr></table> +</div> + +# Tables. +[tabletags-default] +colspec=<col{autowidth-option! style="width:{colpcwidth}%;"}> +bodyrow=<tr>|</tr> +headdata=<th class="tableblock halign-{halign=left} valign-{valign=top}" {colspan@1::colspan="{colspan}" }{rowspan@1::rowspan="{rowspan}" }>|</th> +bodydata=<td class="tableblock halign-{halign=left} valign-{valign=top}" {colspan@1::colspan="{colspan}" }{rowspan@1::rowspan="{rowspan}" }>|</td> +paragraph=<p class="tableblock">|</p> + +[tabletags-header] +paragraph=<p class="tableblock header">|</p> + +[tabletags-emphasis] +paragraph=<p class="tableblock"><em>|</em></p> + +[tabletags-strong] +paragraph=<p class="tableblock"><strong>|</strong></p> + +[tabletags-monospaced] +paragraph=<p class="tableblock monospaced">|</p> + +[tabletags-verse] +bodydata=<td class="tableblock halign-{halign=left} valign-{valign=top}" {colspan@1::colspan="{colspan}" }{rowspan@1::rowspan="{rowspan}" }><div class="verse">|</div></td> +paragraph= + +[tabletags-literal] +bodydata=<td class="tableblock halign-{halign=left} valign-{valign=top}" {colspan@1::colspan="{colspan}" }{rowspan@1::rowspan="{rowspan}" }><div class="literal monospaced"><pre>|</pre></div></td> +paragraph= + +[tabletags-asciidoc] +bodydata=<td class="tableblock halign-{halign=left} valign-{valign=top}" {colspan@1::colspan="{colspan}" }{rowspan@1::rowspan="{rowspan}" }><div>|</div></td> +paragraph= + +[table] +<table class="tableblock frame-{frame=all} grid-{grid=all}{role? {role}}{unbreakable-option? unbreakable}"{id? id="{id}"} +style=" +margin-left:{align@left:0}{align@center|right:auto}; margin-right:{align@left|center:auto}{align@right:0}; +float:{float}; +{autowidth-option%}width:{tablepcwidth}%; +{autowidth-option#}{width#style=width:{tablepcwidth}%;} +"> +<caption class="title">{caption={table-caption} {counter:table-number}. }{title}</caption> +{colspecs} +{headrows#}<thead> +{headrows} +{headrows#}</thead> +{footrows#}<tfoot> +{footrows} +{footrows#}</tfoot> +<tbody> +{bodyrows} +</tbody> +</table> + +#-------------------------------------------------------------------- +# Deprecated old table definitions. +# + +[miscellaneous] +# Screen width in pixels. +pagewidth=800 +pageunits=px + +[old_tabledef-default] +template=old_table +colspec=<col style="width:{colwidth}{pageunits};" /> +bodyrow=<tr>|</tr> +headdata=<th class="tableblock halign-{colalign=left}">|</th> +footdata=<td class="tableblock halign-{colalign=left}">|</td> +bodydata=<td class="tableblock halign-{colalign=left}">|</td> + +[old_table] +<table class="tableblock frame-{frame=all} grid-{grid=all}"{id? id="{id}"}> +<caption class="title">{caption={table-caption}}{title}</caption> +{colspecs} +{headrows#}<thead> +{headrows} +{headrows#}</thead> +{footrows#}<tfoot> +{footrows} +{footrows#}</tfoot> +<tbody style="vertical-align:top;"> +{bodyrows} +</tbody> +</table> + +# End of deprecated old table definitions. +#-------------------------------------------------------------------- + +[floatingtitle] +<h{level@0:1}{level@1:2}{level@2:3}{level@3:4}{level@4:5}{id? id="{id}"} class="float">{title}</h{level@0:1}{level@1:2}{level@2:3}{level@3:4}{level@4:5}> + +[preamble] +# Untitled elements between header and first section title. +<div id="preamble"> +<div class="sectionbody"> +| +</div> +</div> + +# Document sections. +[sect0] +<h1{id? id="{id}"}>{title}</h1> +| + +[sect1] +<div class="sect1{style? {style}}{role? {role}}"> +<h2{id? id="{id}"}>{numbered?{sectnum} }{title}</h2> +<div class="sectionbody"> +| +</div> +</div> + +[sect2] +<div class="sect2{style? {style}}{role? {role}}"> +<h3{id? id="{id}"}>{numbered?{sectnum} }{title}</h3> +| +</div> + +[sect3] +<div class="sect3{style? {style}}{role? {role}}"> +<h4{id? id="{id}"}>{numbered?{sectnum} }{title}</h4> +| +</div> + +[sect4] +<div class="sect4{style? {style}}{role? {role}}"> +<h5{id? id="{id}"}>{title}</h5> +| +</div> + +[appendix] +<div class="sect1{style? {style}}{role? {role}}"> +<h2{id? id="{id}"}>{numbered?{sectnum} }{appendix-caption} {counter:appendix-number:A}: {title}</h2> +<div class="sectionbody"> +| +</div> +</div> + +[toc] +<div id="toc"> + <div id="toctitle">{toc-title}</div> + <noscript><p><b>JavaScript must be enabled in your browser to display the table of contents.</b></p></noscript> +</div> + +[header] +<!DOCTYPE html> +<html lang="{lang=en}"> +<head> +<meta http-equiv="Content-Type" content="text/html; charset={encoding}"> +<meta name="generator" content="AsciiDoc {asciidoc-version}"> +<meta name="description" content="{description}"> +<meta name="keywords" content="{keywords}"> +<title>crmsh - {title}</title> +{title%}<title>crmsh - {doctitle=}</title> +<link rel="stylesheet" href="http://crmsh.nongnu.org/css/font-awesome.min.css"> +<link rel="stylesheet" href="http://crmsh.nongnu.org/css/crm.css" type="text/css"> +<link href='http://fonts.googleapis.com/css?family=Open+Sans:400,700|Ubuntu+Mono' rel='stylesheet' type='text/css'> +<link href="http://crmsh.github.io/atom.xml" type="application/atom+xml" rel="alternate" title="crmsh atom feed"> +<style> +\#movenotice { + width: 600px; + margin-top: 1em; + margin-bottom: 1em; + margin-left: auto; + margin-right: auto; + font-size: 100%; + padding: 4px; + border: 2px dashed red; +} +</style> +</head> +<body> +<div id="header"> +<h1><a href="http://crmsh.github.io/index.html"><span class="fa-stack"> + <i class="fa fa-square fa-stack-2x"></i> + <i class="fa fa-terminal fa-stack-1x fa-inverse"></i> +</span>crmsh</a></h1> +<div id="topbar"> +<ul> +<li><a href="http://crmsh.github.io/news">News</a></li> +<li><a href="http://crmsh.github.io/documentation">Documentation</a></li> +<li><a href="http://crmsh.github.io/development">Development</a></li> +<li><a href="http://crmsh.github.io/about">About</a></li> +</ul> +</div> +</div> +<!--TOC--> +<div id="container"> +<div id="content"> + +<div id="movenotice">We have moved! The website for crmsh is now <a href="http://crmsh.github.io">http://crmsh.github.io</a>.</div> + +<h1>{doctitle}</h1> + +[footer] +</div> +</div> +<div id="footer"> +<div id="footer-text"> +</div> +</div> + +<a href="https://github.com/crmsh/crmsh"><img style="position: absolute; top: 0; right: 0; border: 0;" src="https://camo.githubusercontent.com/652c5b9acfaddf3a9c326fa6bde407b87f7be0f4/68747470733a2f2f73332e616d617a6f6e6177732e636f6d2f6769746875622f726962626f6e732f666f726b6d655f72696768745f6f72616e67655f6666373630302e706e67" alt="Fork me on GitHub" data-canonical-src="https://s3.amazonaws.com/github/ribbons/forkme_right_orange_ff7600.png"></a> + +</body> +</html> + +ifdef::doctype-manpage[] +[synopsis] +template::[sect1] +endif::doctype-manpage[] + diff --git a/doc/website-v1/css/crm.css b/doc/website-v1/css/crm.css new file mode 100644 index 0000000..9fdfba2 --- /dev/null +++ b/doc/website-v1/css/crm.css @@ -0,0 +1,570 @@ +/* --------------------------------------------------------------------------- + Based on + Bare AsciiDoc styles + Ryan Tomayko <r@tomayko.com> + + Heavily modified by + Kristoffer Gronlund <kgronlund@suse.com> + --------------------------------------------------------------------------- */ + +/* Fonts */ + +body { + font-family:'Open Sans', 'lucida grande',verdana,helvetica,arial,sans-serif; + font-size: 16px; + line-height: 21px; +} + +.small { + font-size: 12px; /* 75% of the baseline */ +} + +.large { + font-size: 20px; /* 125% of the baseline */ +} + +em { + font-style:italic; +} + +strong { + font-weight:bold; +} + +.monospaced { + font-family: 'Menlo', 'Monaco', 'Consolas', 'lucida console', 'bitstream vera sans mono', 'courier new', monospace; +} + +dt { + font-weight:normal; +} + +h1, h2, h3, h4, h5 { + font-family:'Open Sans', 'lucida grande',verdana,helvetica,arial,sans-serif; + font-weight:normal; +} + +h1 { + font-size:2.6rem; + line-height:1.428; +} + +h2 { + font-size:2rem; + line-height:1.36363636; /* repeating, of course */ +} + +h3 { + font-size:1.6rem; + line-height:1.1; +} + +h4 { + font-weight: bold; + font-size:1.3rem; + line-height:1.538; +} + +h5 { + font-size:1.2rem; + font-style:italic; + line-height:1.538; +} + +pre { + font-family:'Menlo', 'Monaco', consolas, 'lucida console', 'bitstream vera sans mono', 'courier new', monospace; + font-size: 14px; +} + +#header h1 { + font-size: 42px; + margin: 0px; + display: inline; +} + +#topbar { + display: inline; + font-size: 18px; +} + +#topbar-small { + display: none; + font-size: 14px; +} + + + +/* Style */ + +body { + margin: 0px 0px; + padding: 0px; + width: 100%; + color:#333; + background: #fff; +} + +.monospaced { + color: #211; + background-color: #fafaf8; +} + +p { + margin-bottom: 1.3636rem; +} + +ul, ol, dl { + margin-top: 1rem; + margin-bottom: 2rem; +} + +ul p { + margin: 10px 0; +} + +dl { + margin-left:40px +} + +dt { + color:#000; +} + +h1, h2, h3, h4, h5 { + color:#000; +} + +h2, h3, h4, h5 { + padding-bottom: 0.333rem; + border-bottom: 1px solid #eee; +} + +h1 { + margin:0px; + margin-top: 48px; +} + +h2 { + margin-top: 36px; + margin-bottom: 1.5rem; +} + +h3 { + margin: 0px; + margin-top: 30px; +} + +pre { + color: #211; + overflow-x: auto; +} + +#header { + background: #ecf0f1; + padding-left: 24px; + padding-top: 4px; + padding-bottom: 0px; + border-bottom: 2px solid #efefea; + width: 100%; + height: 86px; + overflow: hidden; +} + +#header a { + text-decoration: none; + color: #34495e; +} + +#header a:hover { + color:#ee3300; +} + +#topbar ul { + list-style: none; + display: inline; +} + +#topbar li { + list-style: none; + display: inline; + padding-right: 1rem; +} + +#topbar-small ul { + list-style: none; + display: inline; + padding-left: 0.5rem; +} + +#topbar-small li { + list-style: none; + display: inline; + padding-right: 0.5rem; +} + + +#container { + max-width: 720px; + margin-left: 240px; + padding-left: 8px; + text-align:left; +} + +#author { + color:#999; +} + +a { + text-decoration: none; + color:#419eda; +} + +a:active { + color:#6ec654; +} + +a:hover { + color:#ee3300; + text-decoration: underline; +} + + +#content { +} + +h1 { + margin-left: auto; + margin-right: auto; + width: 551px; + text-align: center; + margin-bottom: 1.5rem; +} + +.frontpage-image { + margin-left: auto; + margin-right: auto; + width: 551px; +} + +.title, .sidebar-title { + font-weight:normal; + color:#000; + margin-bottom:0; +} + +div.content { + margin: 8px; + padding: 0; +} + +div.admonitionblock .title { + font-weight:bold; +} + +div.admonitionblock { + margin:30px 0px; + color:#555; +} + +div.admonitionblock td.icon { + width:30px; + padding-right:20px; + padding-left:20px; + text-transform:uppercase; + font-weight:bold; + color:#888; +} + +div.listingblock .content { + border-left:4px solid #419eda; + padding:8px; + + background: #faf7f8; + + background-image: -moz-linear-gradient(left, right, + from(#faf7f8), + to(#ffffff)); + + background-image: -webkit-gradient(linear, left top, right bottom, + color-stop(0.00, #faf7f8), + color-stop(1.00, #ffffff)); + +} + +div.listingblock .content pre { + margin:0; +} + +div.literalblock .content { + margin-left: 20px; +} + +div.verseblock .content { + white-space:pre +} + +div.sidebarblock { + margin-top: 1.5rem; + margin-bottom: 2rem; +} + +div.sidebarblock > div.content { + border-left:4px solid #ee3300; + background: #faf7f8; + padding:0 10px; + color:#222; + font-size: 14px; + line-height:18px; + max-width: 720px; + + background-image: -moz-linear-gradient(left, right, + from(#faf7f8), + to(#ffffff)); + + background-image: -webkit-gradient(linear, left top, right bottom, + color-stop(0.00, #faf7f8), + color-stop(1.00, #ffffff)); + +} + +div.sidebarblock .title { + margin:10px 0; + font-weight:bold; + font-size: 14px; + color:#442; +} + +.quoteblock-content { + font-style:italic; + color:#444; + margin-left:40px; +} + +.quoteblock-content .attribution { + font-style:normal; + text-align:right; + color:#000; +} + +.exampleblock-content *:first-child { margin-top:0 } +.exampleblock-content { + border-left:2px solid silver; + padding-left:8px; +} + +#footnotes { + text-align:left; +} + +#footnotes hr { + height: 1px; + color: #ccc; + width: 80%; +} + +#footer { + font-size: 12px; + color:#888; + margin-top:40px; + text-align: right; +} + +.nav { + margin-bottom: 0; + padding-left: 0; + list-style: none; +} + +.nav li { + line-height: 4rem; +} + +.nav a { + font-size: 20px; + text-decoration: none; +} + +.feedEkList .newsItem { + list-style-type: none; +} + +.feedEkList .itemTitle { + font-size: large; +} + +.feedEkList .itemDate { + font-size: smaller; +} + +.feedEkList .itemContent { +} + +@media screen { + #toc { + position: fixed; + top: 120px; + left: 4px; + margin: 0px; + font-size: 12px; + line-height: 1.2em; + } + + #toc a .monospaced { + color:#419eda; + } + + #toc a { + text-decoration: none; + } + + #toc .toclevel1 { + padding: 1px; + margin-top: 8px; + font-size: 14px; + line-height: 16px; + } + + #toc .toclevel2 { + margin-left: 8px; + padding-left: 4px; + font-size: 12px; + line-height: 16px; + } + + #toc .toclevel3 { + margin-left: 24px; + padding-left: 4px; + font-size: 11px; + line-height: 15px; + font-weight: bold; + } + + #toctitle { + margin:20px 0; + + } +} + +@media screen and (max-width: 900px) { + #toc { + display: none; + } + + #container { + max-width: 720px; + margin: 0px auto; + text-align:left; + padding-right: 8px; + } + + #topbar { + display: none; + } + + #topbar-small { + display: inline; + } +} + +@media screen and (min-width: 900px) and (max-width: 1280px) { + #toc { + position: absolute; + overflow: hidden; + top: 120px; + left: 4px; + max-width: 240px; + } +} + +@media screen and (min-width: 1280px) { + #toc { + position: absolute; + overflow: hidden; + top: 120px; + left: 4px; + max-width: 240px; + } + + #container { + max-width: 960px; + } +} + +@media screen and (min-width: 1500px) { + #container { + margin-left: auto; + margin-right: auto; + } +} + +/* pygments highlighting */ + +.hll { background-color: #ffffcc } +.c { color: #999988; font-style: italic } /* Comment */ +.err { color: #a61717; background-color: #e3d2d2 } /* Error */ +.k { color: #000000; font-weight: bold } /* Keyword */ +.o { color: #000000; font-weight: bold } /* Operator */ +.cm { color: #999988; font-style: italic } /* Comment.Multiline */ +.cp { color: #999999; font-weight: bold; font-style: italic } /* Comment.Preproc */ +.c1 { color: #999988; font-style: italic } /* Comment.Single */ +.cs { color: #999999; font-weight: bold; font-style: italic } /* Comment.Special */ +.gd { color: #000000; background-color: #ffdddd } /* Generic.Deleted */ +.ge { color: #000000; font-style: italic } /* Generic.Emph */ +.gr { color: #aa0000 } /* Generic.Error */ +.gh { color: #999999 } /* Generic.Heading */ +.gi { color: #000000; background-color: #ddffdd } /* Generic.Inserted */ +.go { color: #888888 } /* Generic.Output */ +.gp { color: #555555 } /* Generic.Prompt */ +.gs { font-weight: bold } /* Generic.Strong */ +.gu { color: #aaaaaa } /* Generic.Subheading */ +.gt { color: #aa0000 } /* Generic.Traceback */ +.kc { color: #000000; font-weight: bold } /* Keyword.Constant */ +.kd { color: #000000; font-weight: bold } /* Keyword.Declaration */ +.kn { color: #000000; font-weight: bold } /* Keyword.Namespace */ +.kp { color: #000000; font-weight: bold } /* Keyword.Pseudo */ +.kr { color: #000000; font-weight: bold } /* Keyword.Reserved */ +.kt { color: #445588; font-weight: bold } /* Keyword.Type */ +.m { color: #009999 } /* Literal.Number */ +.s { color: #d01040 } /* Literal.String */ +.na { color: #008080 } /* Name.Attribute */ +.nb { color: #0086B3 } /* Name.Builtin */ +.nc { color: #445588; font-weight: bold } /* Name.Class */ +.no { color: #008080 } /* Name.Constant */ +.nd { color: #3c5d5d; font-weight: bold } /* Name.Decorator */ +.ni { color: #800080 } /* Name.Entity */ +.ne { color: #990000; font-weight: bold } /* Name.Exception */ +.nf { color: #990000; font-weight: bold } /* Name.Function */ +.nl { color: #990000; font-weight: bold } /* Name.Label */ +.nn { color: #555555 } /* Name.Namespace */ +.nt { color: #000080 } /* Name.Tag */ +.nv { color: #008080 } /* Name.Variable */ +.ow { color: #000000; font-weight: bold } /* Operator.Word */ +.w { color: #bbbbbb } /* Text.Whitespace */ +.mf { color: #009999 } /* Literal.Number.Float */ +.mh { color: #009999 } /* Literal.Number.Hex */ +.mi { color: #009999 } /* Literal.Number.Integer */ +.mo { color: #009999 } /* Literal.Number.Oct */ +.sb { color: #d01040 } /* Literal.String.Backtick */ +.sc { color: #d01040 } /* Literal.String.Char */ +.sd { color: #d01040 } /* Literal.String.Doc */ +.s2 { color: #d01040 } /* Literal.String.Double */ +.se { color: #d01040 } /* Literal.String.Escape */ +.sh { color: #d01040 } /* Literal.String.Heredoc */ +.si { color: #d01040 } /* Literal.String.Interpol */ +.sx { color: #d01040 } /* Literal.String.Other */ +.sr { color: #009926 } /* Literal.String.Regex */ +.s1 { color: #d01040 } /* Literal.String.Single */ +.ss { color: #990073 } /* Literal.String.Symbol */ +.bp { color: #999999 } /* Name.Builtin.Pseudo */ +.vc { color: #008080 } /* Name.Variable.Class */ +.vg { color: #008080 } /* Name.Variable.Global */ +.vi { color: #008080 } /* Name.Variable.Instance */ +.il { color: #009999 } /* Literal.Number.Integer.Long */ +.highlight .-Color-Black { color: #000000 } /* Color.Black */ +.highlight .-Color-Blue { color: #0000c0 } /* Color.Blue */ +.highlight .-Color-Cyan { color: #008080 } /* Color.Cyan */ +.highlight .-Color-Green { color: #008000 } /* Color.Green */ +.highlight .-Color-Magenta { color: #c000c0 } /* Color.Magenta */ +.highlight .-Color-Red { color: #c00000 } /* Color.Red */ +.highlight .-Color-White { color: #c0c0c0 } /* Color.White */ +.highlight .-Color-Yellow { color: #808000 } /* Color.Yellow */ diff --git a/doc/website-v1/css/font-awesome.css b/doc/website-v1/css/font-awesome.css new file mode 100644 index 0000000..048cff9 --- /dev/null +++ b/doc/website-v1/css/font-awesome.css @@ -0,0 +1,1338 @@ +/*! + * Font Awesome 4.0.3 by @davegandy - http://fontawesome.io - @fontawesome + * License - http://fontawesome.io/license (Font: SIL OFL 1.1, CSS: MIT License) + */ +/* FONT PATH + * -------------------------- */ +@font-face { + font-family: 'FontAwesome'; + src: url('../fonts/fontawesome-webfont.eot?v=4.0.3'); + src: url('../fonts/fontawesome-webfont.eot?#iefix&v=4.0.3') format('embedded-opentype'), url('../fonts/fontawesome-webfont.woff?v=4.0.3') format('woff'), url('../fonts/fontawesome-webfont.ttf?v=4.0.3') format('truetype'), url('../fonts/fontawesome-webfont.svg?v=4.0.3#fontawesomeregular') format('svg'); + font-weight: normal; + font-style: normal; +} +.fa { + display: inline-block; + font-family: FontAwesome; + font-style: normal; + font-weight: normal; + line-height: 1; + -webkit-font-smoothing: antialiased; + -moz-osx-font-smoothing: grayscale; +} +/* makes the font 33% larger relative to the icon container */ +.fa-lg { + font-size: 1.3333333333333333em; + line-height: 0.75em; + vertical-align: -15%; +} +.fa-2x { + font-size: 2em; +} +.fa-3x { + font-size: 3em; +} +.fa-4x { + font-size: 4em; +} +.fa-5x { + font-size: 5em; +} +.fa-fw { + width: 1.2857142857142858em; + text-align: center; +} +.fa-ul { + padding-left: 0; + margin-left: 2.142857142857143em; + list-style-type: none; +} +.fa-ul > li { + position: relative; +} +.fa-li { + position: absolute; + left: -2.142857142857143em; + width: 2.142857142857143em; + top: 0.14285714285714285em; + text-align: center; +} +.fa-li.fa-lg { + left: -1.8571428571428572em; +} +.fa-border { + padding: .2em .25em .15em; + border: solid 0.08em #eeeeee; + border-radius: .1em; +} +.pull-right { + float: right; +} +.pull-left { + float: left; +} +.fa.pull-left { + margin-right: .3em; +} +.fa.pull-right { + margin-left: .3em; +} +.fa-spin { + -webkit-animation: spin 2s infinite linear; + -moz-animation: spin 2s infinite linear; + -o-animation: spin 2s infinite linear; + animation: spin 2s infinite linear; +} +@-moz-keyframes spin { + 0% { + -moz-transform: rotate(0deg); + } + 100% { + -moz-transform: rotate(359deg); + } +} +@-webkit-keyframes spin { + 0% { + -webkit-transform: rotate(0deg); + } + 100% { + -webkit-transform: rotate(359deg); + } +} +@-o-keyframes spin { + 0% { + -o-transform: rotate(0deg); + } + 100% { + -o-transform: rotate(359deg); + } +} +@-ms-keyframes spin { + 0% { + -ms-transform: rotate(0deg); + } + 100% { + -ms-transform: rotate(359deg); + } +} +@keyframes spin { + 0% { + transform: rotate(0deg); + } + 100% { + transform: rotate(359deg); + } +} +.fa-rotate-90 { + filter: progid:DXImageTransform.Microsoft.BasicImage(rotation=1); + -webkit-transform: rotate(90deg); + -moz-transform: rotate(90deg); + -ms-transform: rotate(90deg); + -o-transform: rotate(90deg); + transform: rotate(90deg); +} +.fa-rotate-180 { + filter: progid:DXImageTransform.Microsoft.BasicImage(rotation=2); + -webkit-transform: rotate(180deg); + -moz-transform: rotate(180deg); + -ms-transform: rotate(180deg); + -o-transform: rotate(180deg); + transform: rotate(180deg); +} +.fa-rotate-270 { + filter: progid:DXImageTransform.Microsoft.BasicImage(rotation=3); + -webkit-transform: rotate(270deg); + -moz-transform: rotate(270deg); + -ms-transform: rotate(270deg); + -o-transform: rotate(270deg); + transform: rotate(270deg); +} +.fa-flip-horizontal { + filter: progid:DXImageTransform.Microsoft.BasicImage(rotation=0, mirror=1); + -webkit-transform: scale(-1, 1); + -moz-transform: scale(-1, 1); + -ms-transform: scale(-1, 1); + -o-transform: scale(-1, 1); + transform: scale(-1, 1); +} +.fa-flip-vertical { + filter: progid:DXImageTransform.Microsoft.BasicImage(rotation=2, mirror=1); + -webkit-transform: scale(1, -1); + -moz-transform: scale(1, -1); + -ms-transform: scale(1, -1); + -o-transform: scale(1, -1); + transform: scale(1, -1); +} +.fa-stack { + position: relative; + display: inline-block; + width: 2em; + height: 2em; + line-height: 2em; + vertical-align: middle; +} +.fa-stack-1x, +.fa-stack-2x { + position: absolute; + left: 0; + width: 100%; + text-align: center; +} +.fa-stack-1x { + line-height: inherit; +} +.fa-stack-2x { + font-size: 2em; +} +.fa-inverse { + color: #ffffff; +} +/* Font Awesome uses the Unicode Private Use Area (PUA) to ensure screen + readers do not read off random characters that represent icons */ +.fa-glass:before { + content: "\f000"; +} +.fa-music:before { + content: "\f001"; +} +.fa-search:before { + content: "\f002"; +} +.fa-envelope-o:before { + content: "\f003"; +} +.fa-heart:before { + content: "\f004"; +} +.fa-star:before { + content: "\f005"; +} +.fa-star-o:before { + content: "\f006"; +} +.fa-user:before { + content: "\f007"; +} +.fa-film:before { + content: "\f008"; +} +.fa-th-large:before { + content: "\f009"; +} +.fa-th:before { + content: "\f00a"; +} +.fa-th-list:before { + content: "\f00b"; +} +.fa-check:before { + content: "\f00c"; +} +.fa-times:before { + content: "\f00d"; +} +.fa-search-plus:before { + content: "\f00e"; +} +.fa-search-minus:before { + content: "\f010"; +} +.fa-power-off:before { + content: "\f011"; +} +.fa-signal:before { + content: "\f012"; +} +.fa-gear:before, +.fa-cog:before { + content: "\f013"; +} +.fa-trash-o:before { + content: "\f014"; +} +.fa-home:before { + content: "\f015"; +} +.fa-file-o:before { + content: "\f016"; +} +.fa-clock-o:before { + content: "\f017"; +} +.fa-road:before { + content: "\f018"; +} +.fa-download:before { + content: "\f019"; +} +.fa-arrow-circle-o-down:before { + content: "\f01a"; +} +.fa-arrow-circle-o-up:before { + content: "\f01b"; +} +.fa-inbox:before { + content: "\f01c"; +} +.fa-play-circle-o:before { + content: "\f01d"; +} +.fa-rotate-right:before, +.fa-repeat:before { + content: "\f01e"; +} +.fa-refresh:before { + content: "\f021"; +} +.fa-list-alt:before { + content: "\f022"; +} +.fa-lock:before { + content: "\f023"; +} +.fa-flag:before { + content: "\f024"; +} +.fa-headphones:before { + content: "\f025"; +} +.fa-volume-off:before { + content: "\f026"; +} +.fa-volume-down:before { + content: "\f027"; +} +.fa-volume-up:before { + content: "\f028"; +} +.fa-qrcode:before { + content: "\f029"; +} +.fa-barcode:before { + content: "\f02a"; +} +.fa-tag:before { + content: "\f02b"; +} +.fa-tags:before { + content: "\f02c"; +} +.fa-book:before { + content: "\f02d"; +} +.fa-bookmark:before { + content: "\f02e"; +} +.fa-print:before { + content: "\f02f"; +} +.fa-camera:before { + content: "\f030"; +} +.fa-font:before { + content: "\f031"; +} +.fa-bold:before { + content: "\f032"; +} +.fa-italic:before { + content: "\f033"; +} +.fa-text-height:before { + content: "\f034"; +} +.fa-text-width:before { + content: "\f035"; +} +.fa-align-left:before { + content: "\f036"; +} +.fa-align-center:before { + content: "\f037"; +} +.fa-align-right:before { + content: "\f038"; +} +.fa-align-justify:before { + content: "\f039"; +} +.fa-list:before { + content: "\f03a"; +} +.fa-dedent:before, +.fa-outdent:before { + content: "\f03b"; +} +.fa-indent:before { + content: "\f03c"; +} +.fa-video-camera:before { + content: "\f03d"; +} +.fa-picture-o:before { + content: "\f03e"; +} +.fa-pencil:before { + content: "\f040"; +} +.fa-map-marker:before { + content: "\f041"; +} +.fa-adjust:before { + content: "\f042"; +} +.fa-tint:before { + content: "\f043"; +} +.fa-edit:before, +.fa-pencil-square-o:before { + content: "\f044"; +} +.fa-share-square-o:before { + content: "\f045"; +} +.fa-check-square-o:before { + content: "\f046"; +} +.fa-arrows:before { + content: "\f047"; +} +.fa-step-backward:before { + content: "\f048"; +} +.fa-fast-backward:before { + content: "\f049"; +} +.fa-backward:before { + content: "\f04a"; +} +.fa-play:before { + content: "\f04b"; +} +.fa-pause:before { + content: "\f04c"; +} +.fa-stop:before { + content: "\f04d"; +} +.fa-forward:before { + content: "\f04e"; +} +.fa-fast-forward:before { + content: "\f050"; +} +.fa-step-forward:before { + content: "\f051"; +} +.fa-eject:before { + content: "\f052"; +} +.fa-chevron-left:before { + content: "\f053"; +} +.fa-chevron-right:before { + content: "\f054"; +} +.fa-plus-circle:before { + content: "\f055"; +} +.fa-minus-circle:before { + content: "\f056"; +} +.fa-times-circle:before { + content: "\f057"; +} +.fa-check-circle:before { + content: "\f058"; +} +.fa-question-circle:before { + content: "\f059"; +} +.fa-info-circle:before { + content: "\f05a"; +} +.fa-crosshairs:before { + content: "\f05b"; +} +.fa-times-circle-o:before { + content: "\f05c"; +} +.fa-check-circle-o:before { + content: "\f05d"; +} +.fa-ban:before { + content: "\f05e"; +} +.fa-arrow-left:before { + content: "\f060"; +} +.fa-arrow-right:before { + content: "\f061"; +} +.fa-arrow-up:before { + content: "\f062"; +} +.fa-arrow-down:before { + content: "\f063"; +} +.fa-mail-forward:before, +.fa-share:before { + content: "\f064"; +} +.fa-expand:before { + content: "\f065"; +} +.fa-compress:before { + content: "\f066"; +} +.fa-plus:before { + content: "\f067"; +} +.fa-minus:before { + content: "\f068"; +} +.fa-asterisk:before { + content: "\f069"; +} +.fa-exclamation-circle:before { + content: "\f06a"; +} +.fa-gift:before { + content: "\f06b"; +} +.fa-leaf:before { + content: "\f06c"; +} +.fa-fire:before { + content: "\f06d"; +} +.fa-eye:before { + content: "\f06e"; +} +.fa-eye-slash:before { + content: "\f070"; +} +.fa-warning:before, +.fa-exclamation-triangle:before { + content: "\f071"; +} +.fa-plane:before { + content: "\f072"; +} +.fa-calendar:before { + content: "\f073"; +} +.fa-random:before { + content: "\f074"; +} +.fa-comment:before { + content: "\f075"; +} +.fa-magnet:before { + content: "\f076"; +} +.fa-chevron-up:before { + content: "\f077"; +} +.fa-chevron-down:before { + content: "\f078"; +} +.fa-retweet:before { + content: "\f079"; +} +.fa-shopping-cart:before { + content: "\f07a"; +} +.fa-folder:before { + content: "\f07b"; +} +.fa-folder-open:before { + content: "\f07c"; +} +.fa-arrows-v:before { + content: "\f07d"; +} +.fa-arrows-h:before { + content: "\f07e"; +} +.fa-bar-chart-o:before { + content: "\f080"; +} +.fa-twitter-square:before { + content: "\f081"; +} +.fa-facebook-square:before { + content: "\f082"; +} +.fa-camera-retro:before { + content: "\f083"; +} +.fa-key:before { + content: "\f084"; +} +.fa-gears:before, +.fa-cogs:before { + content: "\f085"; +} +.fa-comments:before { + content: "\f086"; +} +.fa-thumbs-o-up:before { + content: "\f087"; +} +.fa-thumbs-o-down:before { + content: "\f088"; +} +.fa-star-half:before { + content: "\f089"; +} +.fa-heart-o:before { + content: "\f08a"; +} +.fa-sign-out:before { + content: "\f08b"; +} +.fa-linkedin-square:before { + content: "\f08c"; +} +.fa-thumb-tack:before { + content: "\f08d"; +} +.fa-external-link:before { + content: "\f08e"; +} +.fa-sign-in:before { + content: "\f090"; +} +.fa-trophy:before { + content: "\f091"; +} +.fa-github-square:before { + content: "\f092"; +} +.fa-upload:before { + content: "\f093"; +} +.fa-lemon-o:before { + content: "\f094"; +} +.fa-phone:before { + content: "\f095"; +} +.fa-square-o:before { + content: "\f096"; +} +.fa-bookmark-o:before { + content: "\f097"; +} +.fa-phone-square:before { + content: "\f098"; +} +.fa-twitter:before { + content: "\f099"; +} +.fa-facebook:before { + content: "\f09a"; +} +.fa-github:before { + content: "\f09b"; +} +.fa-unlock:before { + content: "\f09c"; +} +.fa-credit-card:before { + content: "\f09d"; +} +.fa-rss:before { + content: "\f09e"; +} +.fa-hdd-o:before { + content: "\f0a0"; +} +.fa-bullhorn:before { + content: "\f0a1"; +} +.fa-bell:before { + content: "\f0f3"; +} +.fa-certificate:before { + content: "\f0a3"; +} +.fa-hand-o-right:before { + content: "\f0a4"; +} +.fa-hand-o-left:before { + content: "\f0a5"; +} +.fa-hand-o-up:before { + content: "\f0a6"; +} +.fa-hand-o-down:before { + content: "\f0a7"; +} +.fa-arrow-circle-left:before { + content: "\f0a8"; +} +.fa-arrow-circle-right:before { + content: "\f0a9"; +} +.fa-arrow-circle-up:before { + content: "\f0aa"; +} +.fa-arrow-circle-down:before { + content: "\f0ab"; +} +.fa-globe:before { + content: "\f0ac"; +} +.fa-wrench:before { + content: "\f0ad"; +} +.fa-tasks:before { + content: "\f0ae"; +} +.fa-filter:before { + content: "\f0b0"; +} +.fa-briefcase:before { + content: "\f0b1"; +} +.fa-arrows-alt:before { + content: "\f0b2"; +} +.fa-group:before, +.fa-users:before { + content: "\f0c0"; +} +.fa-chain:before, +.fa-link:before { + content: "\f0c1"; +} +.fa-cloud:before { + content: "\f0c2"; +} +.fa-flask:before { + content: "\f0c3"; +} +.fa-cut:before, +.fa-scissors:before { + content: "\f0c4"; +} +.fa-copy:before, +.fa-files-o:before { + content: "\f0c5"; +} +.fa-paperclip:before { + content: "\f0c6"; +} +.fa-save:before, +.fa-floppy-o:before { + content: "\f0c7"; +} +.fa-square:before { + content: "\f0c8"; +} +.fa-bars:before { + content: "\f0c9"; +} +.fa-list-ul:before { + content: "\f0ca"; +} +.fa-list-ol:before { + content: "\f0cb"; +} +.fa-strikethrough:before { + content: "\f0cc"; +} +.fa-underline:before { + content: "\f0cd"; +} +.fa-table:before { + content: "\f0ce"; +} +.fa-magic:before { + content: "\f0d0"; +} +.fa-truck:before { + content: "\f0d1"; +} +.fa-pinterest:before { + content: "\f0d2"; +} +.fa-pinterest-square:before { + content: "\f0d3"; +} +.fa-google-plus-square:before { + content: "\f0d4"; +} +.fa-google-plus:before { + content: "\f0d5"; +} +.fa-money:before { + content: "\f0d6"; +} +.fa-caret-down:before { + content: "\f0d7"; +} +.fa-caret-up:before { + content: "\f0d8"; +} +.fa-caret-left:before { + content: "\f0d9"; +} +.fa-caret-right:before { + content: "\f0da"; +} +.fa-columns:before { + content: "\f0db"; +} +.fa-unsorted:before, +.fa-sort:before { + content: "\f0dc"; +} +.fa-sort-down:before, +.fa-sort-asc:before { + content: "\f0dd"; +} +.fa-sort-up:before, +.fa-sort-desc:before { + content: "\f0de"; +} +.fa-envelope:before { + content: "\f0e0"; +} +.fa-linkedin:before { + content: "\f0e1"; +} +.fa-rotate-left:before, +.fa-undo:before { + content: "\f0e2"; +} +.fa-legal:before, +.fa-gavel:before { + content: "\f0e3"; +} +.fa-dashboard:before, +.fa-tachometer:before { + content: "\f0e4"; +} +.fa-comment-o:before { + content: "\f0e5"; +} +.fa-comments-o:before { + content: "\f0e6"; +} +.fa-flash:before, +.fa-bolt:before { + content: "\f0e7"; +} +.fa-sitemap:before { + content: "\f0e8"; +} +.fa-umbrella:before { + content: "\f0e9"; +} +.fa-paste:before, +.fa-clipboard:before { + content: "\f0ea"; +} +.fa-lightbulb-o:before { + content: "\f0eb"; +} +.fa-exchange:before { + content: "\f0ec"; +} +.fa-cloud-download:before { + content: "\f0ed"; +} +.fa-cloud-upload:before { + content: "\f0ee"; +} +.fa-user-md:before { + content: "\f0f0"; +} +.fa-stethoscope:before { + content: "\f0f1"; +} +.fa-suitcase:before { + content: "\f0f2"; +} +.fa-bell-o:before { + content: "\f0a2"; +} +.fa-coffee:before { + content: "\f0f4"; +} +.fa-cutlery:before { + content: "\f0f5"; +} +.fa-file-text-o:before { + content: "\f0f6"; +} +.fa-building-o:before { + content: "\f0f7"; +} +.fa-hospital-o:before { + content: "\f0f8"; +} +.fa-ambulance:before { + content: "\f0f9"; +} +.fa-medkit:before { + content: "\f0fa"; +} +.fa-fighter-jet:before { + content: "\f0fb"; +} +.fa-beer:before { + content: "\f0fc"; +} +.fa-h-square:before { + content: "\f0fd"; +} +.fa-plus-square:before { + content: "\f0fe"; +} +.fa-angle-double-left:before { + content: "\f100"; +} +.fa-angle-double-right:before { + content: "\f101"; +} +.fa-angle-double-up:before { + content: "\f102"; +} +.fa-angle-double-down:before { + content: "\f103"; +} +.fa-angle-left:before { + content: "\f104"; +} +.fa-angle-right:before { + content: "\f105"; +} +.fa-angle-up:before { + content: "\f106"; +} +.fa-angle-down:before { + content: "\f107"; +} +.fa-desktop:before { + content: "\f108"; +} +.fa-laptop:before { + content: "\f109"; +} +.fa-tablet:before { + content: "\f10a"; +} +.fa-mobile-phone:before, +.fa-mobile:before { + content: "\f10b"; +} +.fa-circle-o:before { + content: "\f10c"; +} +.fa-quote-left:before { + content: "\f10d"; +} +.fa-quote-right:before { + content: "\f10e"; +} +.fa-spinner:before { + content: "\f110"; +} +.fa-circle:before { + content: "\f111"; +} +.fa-mail-reply:before, +.fa-reply:before { + content: "\f112"; +} +.fa-github-alt:before { + content: "\f113"; +} +.fa-folder-o:before { + content: "\f114"; +} +.fa-folder-open-o:before { + content: "\f115"; +} +.fa-smile-o:before { + content: "\f118"; +} +.fa-frown-o:before { + content: "\f119"; +} +.fa-meh-o:before { + content: "\f11a"; +} +.fa-gamepad:before { + content: "\f11b"; +} +.fa-keyboard-o:before { + content: "\f11c"; +} +.fa-flag-o:before { + content: "\f11d"; +} +.fa-flag-checkered:before { + content: "\f11e"; +} +.fa-terminal:before { + content: "\f120"; +} +.fa-code:before { + content: "\f121"; +} +.fa-reply-all:before { + content: "\f122"; +} +.fa-mail-reply-all:before { + content: "\f122"; +} +.fa-star-half-empty:before, +.fa-star-half-full:before, +.fa-star-half-o:before { + content: "\f123"; +} +.fa-location-arrow:before { + content: "\f124"; +} +.fa-crop:before { + content: "\f125"; +} +.fa-code-fork:before { + content: "\f126"; +} +.fa-unlink:before, +.fa-chain-broken:before { + content: "\f127"; +} +.fa-question:before { + content: "\f128"; +} +.fa-info:before { + content: "\f129"; +} +.fa-exclamation:before { + content: "\f12a"; +} +.fa-superscript:before { + content: "\f12b"; +} +.fa-subscript:before { + content: "\f12c"; +} +.fa-eraser:before { + content: "\f12d"; +} +.fa-puzzle-piece:before { + content: "\f12e"; +} +.fa-microphone:before { + content: "\f130"; +} +.fa-microphone-slash:before { + content: "\f131"; +} +.fa-shield:before { + content: "\f132"; +} +.fa-calendar-o:before { + content: "\f133"; +} +.fa-fire-extinguisher:before { + content: "\f134"; +} +.fa-rocket:before { + content: "\f135"; +} +.fa-maxcdn:before { + content: "\f136"; +} +.fa-chevron-circle-left:before { + content: "\f137"; +} +.fa-chevron-circle-right:before { + content: "\f138"; +} +.fa-chevron-circle-up:before { + content: "\f139"; +} +.fa-chevron-circle-down:before { + content: "\f13a"; +} +.fa-html5:before { + content: "\f13b"; +} +.fa-css3:before { + content: "\f13c"; +} +.fa-anchor:before { + content: "\f13d"; +} +.fa-unlock-alt:before { + content: "\f13e"; +} +.fa-bullseye:before { + content: "\f140"; +} +.fa-ellipsis-h:before { + content: "\f141"; +} +.fa-ellipsis-v:before { + content: "\f142"; +} +.fa-rss-square:before { + content: "\f143"; +} +.fa-play-circle:before { + content: "\f144"; +} +.fa-ticket:before { + content: "\f145"; +} +.fa-minus-square:before { + content: "\f146"; +} +.fa-minus-square-o:before { + content: "\f147"; +} +.fa-level-up:before { + content: "\f148"; +} +.fa-level-down:before { + content: "\f149"; +} +.fa-check-square:before { + content: "\f14a"; +} +.fa-pencil-square:before { + content: "\f14b"; +} +.fa-external-link-square:before { + content: "\f14c"; +} +.fa-share-square:before { + content: "\f14d"; +} +.fa-compass:before { + content: "\f14e"; +} +.fa-toggle-down:before, +.fa-caret-square-o-down:before { + content: "\f150"; +} +.fa-toggle-up:before, +.fa-caret-square-o-up:before { + content: "\f151"; +} +.fa-toggle-right:before, +.fa-caret-square-o-right:before { + content: "\f152"; +} +.fa-euro:before, +.fa-eur:before { + content: "\f153"; +} +.fa-gbp:before { + content: "\f154"; +} +.fa-dollar:before, +.fa-usd:before { + content: "\f155"; +} +.fa-rupee:before, +.fa-inr:before { + content: "\f156"; +} +.fa-cny:before, +.fa-rmb:before, +.fa-yen:before, +.fa-jpy:before { + content: "\f157"; +} +.fa-ruble:before, +.fa-rouble:before, +.fa-rub:before { + content: "\f158"; +} +.fa-won:before, +.fa-krw:before { + content: "\f159"; +} +.fa-bitcoin:before, +.fa-btc:before { + content: "\f15a"; +} +.fa-file:before { + content: "\f15b"; +} +.fa-file-text:before { + content: "\f15c"; +} +.fa-sort-alpha-asc:before { + content: "\f15d"; +} +.fa-sort-alpha-desc:before { + content: "\f15e"; +} +.fa-sort-amount-asc:before { + content: "\f160"; +} +.fa-sort-amount-desc:before { + content: "\f161"; +} +.fa-sort-numeric-asc:before { + content: "\f162"; +} +.fa-sort-numeric-desc:before { + content: "\f163"; +} +.fa-thumbs-up:before { + content: "\f164"; +} +.fa-thumbs-down:before { + content: "\f165"; +} +.fa-youtube-square:before { + content: "\f166"; +} +.fa-youtube:before { + content: "\f167"; +} +.fa-xing:before { + content: "\f168"; +} +.fa-xing-square:before { + content: "\f169"; +} +.fa-youtube-play:before { + content: "\f16a"; +} +.fa-dropbox:before { + content: "\f16b"; +} +.fa-stack-overflow:before { + content: "\f16c"; +} +.fa-instagram:before { + content: "\f16d"; +} +.fa-flickr:before { + content: "\f16e"; +} +.fa-adn:before { + content: "\f170"; +} +.fa-bitbucket:before { + content: "\f171"; +} +.fa-bitbucket-square:before { + content: "\f172"; +} +.fa-tumblr:before { + content: "\f173"; +} +.fa-tumblr-square:before { + content: "\f174"; +} +.fa-long-arrow-down:before { + content: "\f175"; +} +.fa-long-arrow-up:before { + content: "\f176"; +} +.fa-long-arrow-left:before { + content: "\f177"; +} +.fa-long-arrow-right:before { + content: "\f178"; +} +.fa-apple:before { + content: "\f179"; +} +.fa-windows:before { + content: "\f17a"; +} +.fa-android:before { + content: "\f17b"; +} +.fa-linux:before { + content: "\f17c"; +} +.fa-dribbble:before { + content: "\f17d"; +} +.fa-skype:before { + content: "\f17e"; +} +.fa-foursquare:before { + content: "\f180"; +} +.fa-trello:before { + content: "\f181"; +} +.fa-female:before { + content: "\f182"; +} +.fa-male:before { + content: "\f183"; +} +.fa-gittip:before { + content: "\f184"; +} +.fa-sun-o:before { + content: "\f185"; +} +.fa-moon-o:before { + content: "\f186"; +} +.fa-archive:before { + content: "\f187"; +} +.fa-bug:before { + content: "\f188"; +} +.fa-vk:before { + content: "\f189"; +} +.fa-weibo:before { + content: "\f18a"; +} +.fa-renren:before { + content: "\f18b"; +} +.fa-pagelines:before { + content: "\f18c"; +} +.fa-stack-exchange:before { + content: "\f18d"; +} +.fa-arrow-circle-o-right:before { + content: "\f18e"; +} +.fa-arrow-circle-o-left:before { + content: "\f190"; +} +.fa-toggle-left:before, +.fa-caret-square-o-left:before { + content: "\f191"; +} +.fa-dot-circle-o:before { + content: "\f192"; +} +.fa-wheelchair:before { + content: "\f193"; +} +.fa-vimeo-square:before { + content: "\f194"; +} +.fa-turkish-lira:before, +.fa-try:before { + content: "\f195"; +} +.fa-plus-square-o:before { + content: "\f196"; +} diff --git a/doc/website-v1/css/font-awesome.min.css b/doc/website-v1/css/font-awesome.min.css new file mode 100644 index 0000000..449d6ac --- /dev/null +++ b/doc/website-v1/css/font-awesome.min.css @@ -0,0 +1,4 @@ +/*! + * Font Awesome 4.0.3 by @davegandy - http://fontawesome.io - @fontawesome + * License - http://fontawesome.io/license (Font: SIL OFL 1.1, CSS: MIT License) + */@font-face{font-family:'FontAwesome';src:url('../fonts/fontawesome-webfont.eot?v=4.0.3');src:url('../fonts/fontawesome-webfont.eot?#iefix&v=4.0.3') format('embedded-opentype'),url('../fonts/fontawesome-webfont.woff?v=4.0.3') format('woff'),url('../fonts/fontawesome-webfont.ttf?v=4.0.3') format('truetype'),url('../fonts/fontawesome-webfont.svg?v=4.0.3#fontawesomeregular') format('svg');font-weight:normal;font-style:normal}.fa{display:inline-block;font-family:FontAwesome;font-style:normal;font-weight:normal;line-height:1;-webkit-font-smoothing:antialiased;-moz-osx-font-smoothing:grayscale}.fa-lg{font-size:1.3333333333333333em;line-height:.75em;vertical-align:-15%}.fa-2x{font-size:2em}.fa-3x{font-size:3em}.fa-4x{font-size:4em}.fa-5x{font-size:5em}.fa-fw{width:1.2857142857142858em;text-align:center}.fa-ul{padding-left:0;margin-left:2.142857142857143em;list-style-type:none}.fa-ul>li{position:relative}.fa-li{position:absolute;left:-2.142857142857143em;width:2.142857142857143em;top:.14285714285714285em;text-align:center}.fa-li.fa-lg{left:-1.8571428571428572em}.fa-border{padding:.2em .25em .15em;border:solid .08em #eee;border-radius:.1em}.pull-right{float:right}.pull-left{float:left}.fa.pull-left{margin-right:.3em}.fa.pull-right{margin-left:.3em}.fa-spin{-webkit-animation:spin 2s infinite linear;-moz-animation:spin 2s infinite linear;-o-animation:spin 2s infinite linear;animation:spin 2s infinite linear}@-moz-keyframes spin{0%{-moz-transform:rotate(0deg)}100%{-moz-transform:rotate(359deg)}}@-webkit-keyframes spin{0%{-webkit-transform:rotate(0deg)}100%{-webkit-transform:rotate(359deg)}}@-o-keyframes spin{0%{-o-transform:rotate(0deg)}100%{-o-transform:rotate(359deg)}}@-ms-keyframes spin{0%{-ms-transform:rotate(0deg)}100%{-ms-transform:rotate(359deg)}}@keyframes spin{0%{transform:rotate(0deg)}100%{transform:rotate(359deg)}}.fa-rotate-90{filter:progid:DXImageTransform.Microsoft.BasicImage(rotation=1);-webkit-transform:rotate(90deg);-moz-transform:rotate(90deg);-ms-transform:rotate(90deg);-o-transform:rotate(90deg);transform:rotate(90deg)}.fa-rotate-180{filter:progid:DXImageTransform.Microsoft.BasicImage(rotation=2);-webkit-transform:rotate(180deg);-moz-transform:rotate(180deg);-ms-transform:rotate(180deg);-o-transform:rotate(180deg);transform:rotate(180deg)}.fa-rotate-270{filter:progid:DXImageTransform.Microsoft.BasicImage(rotation=3);-webkit-transform:rotate(270deg);-moz-transform:rotate(270deg);-ms-transform:rotate(270deg);-o-transform:rotate(270deg);transform:rotate(270deg)}.fa-flip-horizontal{filter:progid:DXImageTransform.Microsoft.BasicImage(rotation=0,mirror=1);-webkit-transform:scale(-1,1);-moz-transform:scale(-1,1);-ms-transform:scale(-1,1);-o-transform:scale(-1,1);transform:scale(-1,1)}.fa-flip-vertical{filter:progid:DXImageTransform.Microsoft.BasicImage(rotation=2,mirror=1);-webkit-transform:scale(1,-1);-moz-transform:scale(1,-1);-ms-transform:scale(1,-1);-o-transform:scale(1,-1);transform:scale(1,-1)}.fa-stack{position:relative;display:inline-block;width:2em;height:2em;line-height:2em;vertical-align:middle}.fa-stack-1x,.fa-stack-2x{position:absolute;left:0;width:100%;text-align:center}.fa-stack-1x{line-height:inherit}.fa-stack-2x{font-size:2em}.fa-inverse{color:#fff}.fa-glass:before{content:"\f000"}.fa-music:before{content:"\f001"}.fa-search:before{content:"\f002"}.fa-envelope-o:before{content:"\f003"}.fa-heart:before{content:"\f004"}.fa-star:before{content:"\f005"}.fa-star-o:before{content:"\f006"}.fa-user:before{content:"\f007"}.fa-film:before{content:"\f008"}.fa-th-large:before{content:"\f009"}.fa-th:before{content:"\f00a"}.fa-th-list:before{content:"\f00b"}.fa-check:before{content:"\f00c"}.fa-times:before{content:"\f00d"}.fa-search-plus:before{content:"\f00e"}.fa-search-minus:before{content:"\f010"}.fa-power-off:before{content:"\f011"}.fa-signal:before{content:"\f012"}.fa-gear:before,.fa-cog:before{content:"\f013"}.fa-trash-o:before{content:"\f014"}.fa-home:before{content:"\f015"}.fa-file-o:before{content:"\f016"}.fa-clock-o:before{content:"\f017"}.fa-road:before{content:"\f018"}.fa-download:before{content:"\f019"}.fa-arrow-circle-o-down:before{content:"\f01a"}.fa-arrow-circle-o-up:before{content:"\f01b"}.fa-inbox:before{content:"\f01c"}.fa-play-circle-o:before{content:"\f01d"}.fa-rotate-right:before,.fa-repeat:before{content:"\f01e"}.fa-refresh:before{content:"\f021"}.fa-list-alt:before{content:"\f022"}.fa-lock:before{content:"\f023"}.fa-flag:before{content:"\f024"}.fa-headphones:before{content:"\f025"}.fa-volume-off:before{content:"\f026"}.fa-volume-down:before{content:"\f027"}.fa-volume-up:before{content:"\f028"}.fa-qrcode:before{content:"\f029"}.fa-barcode:before{content:"\f02a"}.fa-tag:before{content:"\f02b"}.fa-tags:before{content:"\f02c"}.fa-book:before{content:"\f02d"}.fa-bookmark:before{content:"\f02e"}.fa-print:before{content:"\f02f"}.fa-camera:before{content:"\f030"}.fa-font:before{content:"\f031"}.fa-bold:before{content:"\f032"}.fa-italic:before{content:"\f033"}.fa-text-height:before{content:"\f034"}.fa-text-width:before{content:"\f035"}.fa-align-left:before{content:"\f036"}.fa-align-center:before{content:"\f037"}.fa-align-right:before{content:"\f038"}.fa-align-justify:before{content:"\f039"}.fa-list:before{content:"\f03a"}.fa-dedent:before,.fa-outdent:before{content:"\f03b"}.fa-indent:before{content:"\f03c"}.fa-video-camera:before{content:"\f03d"}.fa-picture-o:before{content:"\f03e"}.fa-pencil:before{content:"\f040"}.fa-map-marker:before{content:"\f041"}.fa-adjust:before{content:"\f042"}.fa-tint:before{content:"\f043"}.fa-edit:before,.fa-pencil-square-o:before{content:"\f044"}.fa-share-square-o:before{content:"\f045"}.fa-check-square-o:before{content:"\f046"}.fa-arrows:before{content:"\f047"}.fa-step-backward:before{content:"\f048"}.fa-fast-backward:before{content:"\f049"}.fa-backward:before{content:"\f04a"}.fa-play:before{content:"\f04b"}.fa-pause:before{content:"\f04c"}.fa-stop:before{content:"\f04d"}.fa-forward:before{content:"\f04e"}.fa-fast-forward:before{content:"\f050"}.fa-step-forward:before{content:"\f051"}.fa-eject:before{content:"\f052"}.fa-chevron-left:before{content:"\f053"}.fa-chevron-right:before{content:"\f054"}.fa-plus-circle:before{content:"\f055"}.fa-minus-circle:before{content:"\f056"}.fa-times-circle:before{content:"\f057"}.fa-check-circle:before{content:"\f058"}.fa-question-circle:before{content:"\f059"}.fa-info-circle:before{content:"\f05a"}.fa-crosshairs:before{content:"\f05b"}.fa-times-circle-o:before{content:"\f05c"}.fa-check-circle-o:before{content:"\f05d"}.fa-ban:before{content:"\f05e"}.fa-arrow-left:before{content:"\f060"}.fa-arrow-right:before{content:"\f061"}.fa-arrow-up:before{content:"\f062"}.fa-arrow-down:before{content:"\f063"}.fa-mail-forward:before,.fa-share:before{content:"\f064"}.fa-expand:before{content:"\f065"}.fa-compress:before{content:"\f066"}.fa-plus:before{content:"\f067"}.fa-minus:before{content:"\f068"}.fa-asterisk:before{content:"\f069"}.fa-exclamation-circle:before{content:"\f06a"}.fa-gift:before{content:"\f06b"}.fa-leaf:before{content:"\f06c"}.fa-fire:before{content:"\f06d"}.fa-eye:before{content:"\f06e"}.fa-eye-slash:before{content:"\f070"}.fa-warning:before,.fa-exclamation-triangle:before{content:"\f071"}.fa-plane:before{content:"\f072"}.fa-calendar:before{content:"\f073"}.fa-random:before{content:"\f074"}.fa-comment:before{content:"\f075"}.fa-magnet:before{content:"\f076"}.fa-chevron-up:before{content:"\f077"}.fa-chevron-down:before{content:"\f078"}.fa-retweet:before{content:"\f079"}.fa-shopping-cart:before{content:"\f07a"}.fa-folder:before{content:"\f07b"}.fa-folder-open:before{content:"\f07c"}.fa-arrows-v:before{content:"\f07d"}.fa-arrows-h:before{content:"\f07e"}.fa-bar-chart-o:before{content:"\f080"}.fa-twitter-square:before{content:"\f081"}.fa-facebook-square:before{content:"\f082"}.fa-camera-retro:before{content:"\f083"}.fa-key:before{content:"\f084"}.fa-gears:before,.fa-cogs:before{content:"\f085"}.fa-comments:before{content:"\f086"}.fa-thumbs-o-up:before{content:"\f087"}.fa-thumbs-o-down:before{content:"\f088"}.fa-star-half:before{content:"\f089"}.fa-heart-o:before{content:"\f08a"}.fa-sign-out:before{content:"\f08b"}.fa-linkedin-square:before{content:"\f08c"}.fa-thumb-tack:before{content:"\f08d"}.fa-external-link:before{content:"\f08e"}.fa-sign-in:before{content:"\f090"}.fa-trophy:before{content:"\f091"}.fa-github-square:before{content:"\f092"}.fa-upload:before{content:"\f093"}.fa-lemon-o:before{content:"\f094"}.fa-phone:before{content:"\f095"}.fa-square-o:before{content:"\f096"}.fa-bookmark-o:before{content:"\f097"}.fa-phone-square:before{content:"\f098"}.fa-twitter:before{content:"\f099"}.fa-facebook:before{content:"\f09a"}.fa-github:before{content:"\f09b"}.fa-unlock:before{content:"\f09c"}.fa-credit-card:before{content:"\f09d"}.fa-rss:before{content:"\f09e"}.fa-hdd-o:before{content:"\f0a0"}.fa-bullhorn:before{content:"\f0a1"}.fa-bell:before{content:"\f0f3"}.fa-certificate:before{content:"\f0a3"}.fa-hand-o-right:before{content:"\f0a4"}.fa-hand-o-left:before{content:"\f0a5"}.fa-hand-o-up:before{content:"\f0a6"}.fa-hand-o-down:before{content:"\f0a7"}.fa-arrow-circle-left:before{content:"\f0a8"}.fa-arrow-circle-right:before{content:"\f0a9"}.fa-arrow-circle-up:before{content:"\f0aa"}.fa-arrow-circle-down:before{content:"\f0ab"}.fa-globe:before{content:"\f0ac"}.fa-wrench:before{content:"\f0ad"}.fa-tasks:before{content:"\f0ae"}.fa-filter:before{content:"\f0b0"}.fa-briefcase:before{content:"\f0b1"}.fa-arrows-alt:before{content:"\f0b2"}.fa-group:before,.fa-users:before{content:"\f0c0"}.fa-chain:before,.fa-link:before{content:"\f0c1"}.fa-cloud:before{content:"\f0c2"}.fa-flask:before{content:"\f0c3"}.fa-cut:before,.fa-scissors:before{content:"\f0c4"}.fa-copy:before,.fa-files-o:before{content:"\f0c5"}.fa-paperclip:before{content:"\f0c6"}.fa-save:before,.fa-floppy-o:before{content:"\f0c7"}.fa-square:before{content:"\f0c8"}.fa-bars:before{content:"\f0c9"}.fa-list-ul:before{content:"\f0ca"}.fa-list-ol:before{content:"\f0cb"}.fa-strikethrough:before{content:"\f0cc"}.fa-underline:before{content:"\f0cd"}.fa-table:before{content:"\f0ce"}.fa-magic:before{content:"\f0d0"}.fa-truck:before{content:"\f0d1"}.fa-pinterest:before{content:"\f0d2"}.fa-pinterest-square:before{content:"\f0d3"}.fa-google-plus-square:before{content:"\f0d4"}.fa-google-plus:before{content:"\f0d5"}.fa-money:before{content:"\f0d6"}.fa-caret-down:before{content:"\f0d7"}.fa-caret-up:before{content:"\f0d8"}.fa-caret-left:before{content:"\f0d9"}.fa-caret-right:before{content:"\f0da"}.fa-columns:before{content:"\f0db"}.fa-unsorted:before,.fa-sort:before{content:"\f0dc"}.fa-sort-down:before,.fa-sort-asc:before{content:"\f0dd"}.fa-sort-up:before,.fa-sort-desc:before{content:"\f0de"}.fa-envelope:before{content:"\f0e0"}.fa-linkedin:before{content:"\f0e1"}.fa-rotate-left:before,.fa-undo:before{content:"\f0e2"}.fa-legal:before,.fa-gavel:before{content:"\f0e3"}.fa-dashboard:before,.fa-tachometer:before{content:"\f0e4"}.fa-comment-o:before{content:"\f0e5"}.fa-comments-o:before{content:"\f0e6"}.fa-flash:before,.fa-bolt:before{content:"\f0e7"}.fa-sitemap:before{content:"\f0e8"}.fa-umbrella:before{content:"\f0e9"}.fa-paste:before,.fa-clipboard:before{content:"\f0ea"}.fa-lightbulb-o:before{content:"\f0eb"}.fa-exchange:before{content:"\f0ec"}.fa-cloud-download:before{content:"\f0ed"}.fa-cloud-upload:before{content:"\f0ee"}.fa-user-md:before{content:"\f0f0"}.fa-stethoscope:before{content:"\f0f1"}.fa-suitcase:before{content:"\f0f2"}.fa-bell-o:before{content:"\f0a2"}.fa-coffee:before{content:"\f0f4"}.fa-cutlery:before{content:"\f0f5"}.fa-file-text-o:before{content:"\f0f6"}.fa-building-o:before{content:"\f0f7"}.fa-hospital-o:before{content:"\f0f8"}.fa-ambulance:before{content:"\f0f9"}.fa-medkit:before{content:"\f0fa"}.fa-fighter-jet:before{content:"\f0fb"}.fa-beer:before{content:"\f0fc"}.fa-h-square:before{content:"\f0fd"}.fa-plus-square:before{content:"\f0fe"}.fa-angle-double-left:before{content:"\f100"}.fa-angle-double-right:before{content:"\f101"}.fa-angle-double-up:before{content:"\f102"}.fa-angle-double-down:before{content:"\f103"}.fa-angle-left:before{content:"\f104"}.fa-angle-right:before{content:"\f105"}.fa-angle-up:before{content:"\f106"}.fa-angle-down:before{content:"\f107"}.fa-desktop:before{content:"\f108"}.fa-laptop:before{content:"\f109"}.fa-tablet:before{content:"\f10a"}.fa-mobile-phone:before,.fa-mobile:before{content:"\f10b"}.fa-circle-o:before{content:"\f10c"}.fa-quote-left:before{content:"\f10d"}.fa-quote-right:before{content:"\f10e"}.fa-spinner:before{content:"\f110"}.fa-circle:before{content:"\f111"}.fa-mail-reply:before,.fa-reply:before{content:"\f112"}.fa-github-alt:before{content:"\f113"}.fa-folder-o:before{content:"\f114"}.fa-folder-open-o:before{content:"\f115"}.fa-smile-o:before{content:"\f118"}.fa-frown-o:before{content:"\f119"}.fa-meh-o:before{content:"\f11a"}.fa-gamepad:before{content:"\f11b"}.fa-keyboard-o:before{content:"\f11c"}.fa-flag-o:before{content:"\f11d"}.fa-flag-checkered:before{content:"\f11e"}.fa-terminal:before{content:"\f120"}.fa-code:before{content:"\f121"}.fa-reply-all:before{content:"\f122"}.fa-mail-reply-all:before{content:"\f122"}.fa-star-half-empty:before,.fa-star-half-full:before,.fa-star-half-o:before{content:"\f123"}.fa-location-arrow:before{content:"\f124"}.fa-crop:before{content:"\f125"}.fa-code-fork:before{content:"\f126"}.fa-unlink:before,.fa-chain-broken:before{content:"\f127"}.fa-question:before{content:"\f128"}.fa-info:before{content:"\f129"}.fa-exclamation:before{content:"\f12a"}.fa-superscript:before{content:"\f12b"}.fa-subscript:before{content:"\f12c"}.fa-eraser:before{content:"\f12d"}.fa-puzzle-piece:before{content:"\f12e"}.fa-microphone:before{content:"\f130"}.fa-microphone-slash:before{content:"\f131"}.fa-shield:before{content:"\f132"}.fa-calendar-o:before{content:"\f133"}.fa-fire-extinguisher:before{content:"\f134"}.fa-rocket:before{content:"\f135"}.fa-maxcdn:before{content:"\f136"}.fa-chevron-circle-left:before{content:"\f137"}.fa-chevron-circle-right:before{content:"\f138"}.fa-chevron-circle-up:before{content:"\f139"}.fa-chevron-circle-down:before{content:"\f13a"}.fa-html5:before{content:"\f13b"}.fa-css3:before{content:"\f13c"}.fa-anchor:before{content:"\f13d"}.fa-unlock-alt:before{content:"\f13e"}.fa-bullseye:before{content:"\f140"}.fa-ellipsis-h:before{content:"\f141"}.fa-ellipsis-v:before{content:"\f142"}.fa-rss-square:before{content:"\f143"}.fa-play-circle:before{content:"\f144"}.fa-ticket:before{content:"\f145"}.fa-minus-square:before{content:"\f146"}.fa-minus-square-o:before{content:"\f147"}.fa-level-up:before{content:"\f148"}.fa-level-down:before{content:"\f149"}.fa-check-square:before{content:"\f14a"}.fa-pencil-square:before{content:"\f14b"}.fa-external-link-square:before{content:"\f14c"}.fa-share-square:before{content:"\f14d"}.fa-compass:before{content:"\f14e"}.fa-toggle-down:before,.fa-caret-square-o-down:before{content:"\f150"}.fa-toggle-up:before,.fa-caret-square-o-up:before{content:"\f151"}.fa-toggle-right:before,.fa-caret-square-o-right:before{content:"\f152"}.fa-euro:before,.fa-eur:before{content:"\f153"}.fa-gbp:before{content:"\f154"}.fa-dollar:before,.fa-usd:before{content:"\f155"}.fa-rupee:before,.fa-inr:before{content:"\f156"}.fa-cny:before,.fa-rmb:before,.fa-yen:before,.fa-jpy:before{content:"\f157"}.fa-ruble:before,.fa-rouble:before,.fa-rub:before{content:"\f158"}.fa-won:before,.fa-krw:before{content:"\f159"}.fa-bitcoin:before,.fa-btc:before{content:"\f15a"}.fa-file:before{content:"\f15b"}.fa-file-text:before{content:"\f15c"}.fa-sort-alpha-asc:before{content:"\f15d"}.fa-sort-alpha-desc:before{content:"\f15e"}.fa-sort-amount-asc:before{content:"\f160"}.fa-sort-amount-desc:before{content:"\f161"}.fa-sort-numeric-asc:before{content:"\f162"}.fa-sort-numeric-desc:before{content:"\f163"}.fa-thumbs-up:before{content:"\f164"}.fa-thumbs-down:before{content:"\f165"}.fa-youtube-square:before{content:"\f166"}.fa-youtube:before{content:"\f167"}.fa-xing:before{content:"\f168"}.fa-xing-square:before{content:"\f169"}.fa-youtube-play:before{content:"\f16a"}.fa-dropbox:before{content:"\f16b"}.fa-stack-overflow:before{content:"\f16c"}.fa-instagram:before{content:"\f16d"}.fa-flickr:before{content:"\f16e"}.fa-adn:before{content:"\f170"}.fa-bitbucket:before{content:"\f171"}.fa-bitbucket-square:before{content:"\f172"}.fa-tumblr:before{content:"\f173"}.fa-tumblr-square:before{content:"\f174"}.fa-long-arrow-down:before{content:"\f175"}.fa-long-arrow-up:before{content:"\f176"}.fa-long-arrow-left:before{content:"\f177"}.fa-long-arrow-right:before{content:"\f178"}.fa-apple:before{content:"\f179"}.fa-windows:before{content:"\f17a"}.fa-android:before{content:"\f17b"}.fa-linux:before{content:"\f17c"}.fa-dribbble:before{content:"\f17d"}.fa-skype:before{content:"\f17e"}.fa-foursquare:before{content:"\f180"}.fa-trello:before{content:"\f181"}.fa-female:before{content:"\f182"}.fa-male:before{content:"\f183"}.fa-gittip:before{content:"\f184"}.fa-sun-o:before{content:"\f185"}.fa-moon-o:before{content:"\f186"}.fa-archive:before{content:"\f187"}.fa-bug:before{content:"\f188"}.fa-vk:before{content:"\f189"}.fa-weibo:before{content:"\f18a"}.fa-renren:before{content:"\f18b"}.fa-pagelines:before{content:"\f18c"}.fa-stack-exchange:before{content:"\f18d"}.fa-arrow-circle-o-right:before{content:"\f18e"}.fa-arrow-circle-o-left:before{content:"\f190"}.fa-toggle-left:before,.fa-caret-square-o-left:before{content:"\f191"}.fa-dot-circle-o:before{content:"\f192"}.fa-wheelchair:before{content:"\f193"}.fa-vimeo-square:before{content:"\f194"}.fa-turkish-lira:before,.fa-try:before{content:"\f195"}.fa-plus-square-o:before{content:"\f196"}
\ No newline at end of file diff --git a/doc/website-v1/development.adoc b/doc/website-v1/development.adoc new file mode 100644 index 0000000..a334a6b --- /dev/null +++ b/doc/website-v1/development.adoc @@ -0,0 +1,74 @@ += Development = + +== Tools == + +++++ +<ul class="nav"> +<li><a href="https://github.com/ClusterLabs/crmsh"><i class="fa fa-code-fork fa-3x fa-fw"></i> Source Repository</a></li> +<li><a href="http://clusterlabs.org/mailman/listinfo/users"><i class="fa fa-envelope fa-3x fa-fw"></i> Mailing List</a></li> +<li><a href="https://github.com/ClusterLabs/crmsh/issues"><i class="fa fa-bug fa-3x fa-fw"></i> Issue Tracker</a></li> +<li><a href="irc://freenode.net/#clusterlabs"><i class="fa fa-comments fa-3x fa-fw"></i> IRC: #clusterlabs on Freenode</a></li> +<li><a href="https://github.com/ClusterLabs/crmsh/commits/master.atom"><i class="fa fa-rss fa-3x fa-fw"></i> Atom feed</a></li> +</ul> +++++ + +== Source Code == + +The source code for `crmsh` is kept in a +http://git-scm.com/[git] repository +hosted at https://github.com[github]. Use +git+ to get a working copy: + +---- +git clone https://github.com/ClusterLabs/crmsh.git +---- + +Dependencies +~~~~~~~~~~~~ + +Building and installing crmsh requires Python version 2.6 and up (but not 3, yet). + +Additionally, the following Python modules are needed: + +* `lxml` +* `PyYAML` +* `setuptools` +* `parallax` +* `python-dateutil` + +Building +~~~~~~~~ + +`crmsh` uses the autotools suite to manage the build process. + +---- +./autogen.sh +./configure +make +make install +---- + +=== Tests === + +The unit tests for `crmsh` require +nose+ to run. On most distributions, this can be installed +by installing the package +python-nose+, or using +pip+. + +To run the unit test suite, go to the source code directory of `crmsh` +and call: + +---- +./test/run +---- + +`crmsh` also comes with a comprehensive regression test suite. The regression tests need +to run after installation, on a system which has both crmsh and pacemaker installed. You +will also need to install +pacemaker+ and the +cluster-glue+ development headers. + +* link:http://hg.linux-ha.org/glue[cluster-glue] +* link:https://github.com/ClusterLabs/pacemaker[pacemaker] + +To execute the tests, call: + +---- +/usr/share/crmsh/tests/regression.sh +cat crmtestout/regression.out +---- diff --git a/doc/website-v1/documentation.adoc b/doc/website-v1/documentation.adoc new file mode 100644 index 0000000..dce5a0d --- /dev/null +++ b/doc/website-v1/documentation.adoc @@ -0,0 +1,42 @@ += Documentation = + +The main documentation for `crmsh` comes in the form of the +`manual`, which is the same help as found using the `help` +command in the interactive shell. + +Additionally, there are a couple of guides and other documents +that will hopefully make using the shell as easy as possible. + +== Manual == + +* link:/man[Manual (Development)] +* link:/man-4.3[Manual (v4.3.x)] +* link:/man-3[Manual (v3.x)] +* link:/man-2.0[Manual (v2.x)] +* link:/man-1.2[Manual (v1.2.x)] + +== Guides == + +* link:/start-guide[Getting Started] +* link:/history-guide[History Guide] +* link:/rsctest-guide[Resource Testing Guide] +* link:/configuration[Configuration] +* link:/scripts[Cluster scripts] +* link:/faq[Frequently Asked Questions] + +== Translations == + +* https://blog.3ware.co.jp/2015/05/crmsh-getting-started/[Getting Started (Japanese)] + +== External documentation == + +The SUSE +https://www.suse.com/documentation/sle_ha/book_sleha/?page=/documentation/sle_ha/book_sleha/data/book_sleha.html[High +Availability Guide] provides a guide to +installing and configuring a complete cluster solution including both +the `crm` shell and Hawk, the web GUI which uses the `crm` shell as +its backend. + +For more information on Pacemaker in general, see the +http://clusterlabs.org/doc/[Pacemaker documentation] at `clusterlabs.org`. + diff --git a/doc/website-v1/download.adoc b/doc/website-v1/download.adoc new file mode 100644 index 0000000..8d81153 --- /dev/null +++ b/doc/website-v1/download.adoc @@ -0,0 +1,40 @@ += Download = + +The easiest way to install `crmsh` is via the package manager of your distribution. + +== SLES / openSUSE == + +`crmsh` is commercially supported on SLE via the https://www.suse.com/products/highavailability/[SUSE Linux Enterprise High Availability Extension]. It is also available for openSUSE with the package name `crmsh`. Development packages can be downloaded from the OBS: + +* https://build.opensuse.org/package/show/network:ha-clustering:Stable/crmsh[Stable version] +* https://build.opensuse.org/package/show/network:ha-clustering:Factory/crmsh[Development version] + +== Red Hat / CentOS / Fedora == + +We try to build Red Hat / CentOS / Fedora-compatible RPM packages on the OBS (see above). + +=== CentOS 7 === + +--- +dnf config-manager --add-repo http://download.opensuse.org/repositories/network:/ha-clustering:/Stable/CentOS_CentOS-7/network:ha-clustering:Stable.repo +--- + +== Debian == + +The versions of `crmsh` and `pacemaker` currently available in the latest Debian release are quite old. Newer packages are available via the Debian-HA team https://wiki.debian.org/Debian-HA[wiki] and the distribution packages will hopefully be updated soon. + +== Ubuntu == + +Packages for `crmsh` are available from the https://launchpad.net/ubuntu/+source/crmsh[Launchpad]. + +== Gentoo == + +A fairly up-to-date version is available https://packages.gentoo.org/packages/sys-cluster/crmsh[here]. + +== Arch == + +`crmsh` is available via the https://aur.archlinux.org/packages/ha-pacemaker-crmsh/[AUR]. Unfortunately the package seems somewhat out of date. + +== Source Packages == + +Releases are available as `.tar.gz` or `.zip` archives via https://github.com/ClusterLabs/crmsh/releases[Github]. diff --git a/doc/website-v1/faq.adoc b/doc/website-v1/faq.adoc new file mode 100644 index 0000000..c9c5d02 --- /dev/null +++ b/doc/website-v1/faq.adoc @@ -0,0 +1,60 @@ += Frequently Asked Questions + +== What is the crm shell? + +The `crm` shell is a command-line interface to the Pacemaker cluster +resource management stack. If that doesn't make any sense to you, the +easiest way to get up to speed is to go to the +http://clusterlabs.org/[Pacemaker] website and read more about what it +does there. + +The `crm` shell provides a simpler interface to configuring Pacemaker +than manipulating the XML of the CIB (Cluster Information Base) +directly. With its command-line style interface, changes to the +cluster can be performed quickly and painlessly. It also works as a +scripting tool, allowing more complicated changes to be applied to the +cluster. + +The `crm` shell also functions as a management console, providing a +unified interface to a number of other auxilliary tools related to +Pacemaker and cluster management. + +== What distributions does the shell run on? + +Many distributions provide packages for the `crm` shell in their +respective package repositories. The best and safest way to obtain the +`crm` shell is via the distribution repositories, so look there first. + +The intention is for the `crm` shell to work well on all the major +distributions. Pre-built packages are provided for the +following distros: + + * openSUSE + * Fedora + * CentOS + * Red Hat Linux + +More information can be found on the +link:/documentation#_installation[Documentation] page. + +== Didn't crm use to be part of Pacemaker? + +Yes, initially, the `crm` shell was distributed as part of the +Pacemaker project. It was split into its own, separate project in +2011. + +A common misconception is that `crm` has been replaced by `pcs` +(available at https://github.com/feist/pcs[github.com/feist/pcs]). `pcs` +is an alternative command line interface similar to `crm`. Both +projects are being actively developed, with slightly different +goals. Our recommendation is to use whatever shell your distribution +of choice comes with and supports, unless you have a particular +preference or are on a distribution which doesn't bundle either. In +that case, we are obviously biased towards one of the available +choices. ;) + +== Command-line is well and good, but is there a web interface? + +Yes! Take a look at https://github.com/ClusterLabs/hawk[Hawk]. + +Hawk uses the `crm` shell as its backend to interact with the cluster. diff --git a/doc/website-v1/fonts/FontAwesome.otf b/doc/website-v1/fonts/FontAwesome.otf Binary files differnew file mode 100644 index 0000000..8b0f54e --- /dev/null +++ b/doc/website-v1/fonts/FontAwesome.otf diff --git a/doc/website-v1/fonts/fontawesome-webfont.eot b/doc/website-v1/fonts/fontawesome-webfont.eot Binary files differnew file mode 100755 index 0000000..7c79c6a --- /dev/null +++ b/doc/website-v1/fonts/fontawesome-webfont.eot diff --git a/doc/website-v1/fonts/fontawesome-webfont.svg b/doc/website-v1/fonts/fontawesome-webfont.svg new file mode 100755 index 0000000..45fdf33 --- /dev/null +++ b/doc/website-v1/fonts/fontawesome-webfont.svg @@ -0,0 +1,414 @@ +<?xml version="1.0" standalone="no"?> +<!DOCTYPE svg PUBLIC "-//W3C//DTD SVG 1.1//EN" "http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd" > +<svg xmlns="http://www.w3.org/2000/svg"> +<metadata></metadata> +<defs> +<font id="fontawesomeregular" horiz-adv-x="1536" > +<font-face units-per-em="1792" ascent="1536" descent="-256" /> +<missing-glyph horiz-adv-x="448" /> +<glyph unicode=" " horiz-adv-x="448" /> +<glyph unicode="	" horiz-adv-x="448" /> +<glyph unicode=" " horiz-adv-x="448" /> +<glyph unicode="¨" horiz-adv-x="1792" /> +<glyph unicode="©" horiz-adv-x="1792" /> +<glyph unicode="®" horiz-adv-x="1792" /> +<glyph unicode="´" horiz-adv-x="1792" /> +<glyph unicode="Æ" horiz-adv-x="1792" /> +<glyph unicode=" " horiz-adv-x="768" /> +<glyph unicode=" " /> +<glyph unicode=" " horiz-adv-x="768" /> +<glyph unicode=" " /> +<glyph unicode=" " horiz-adv-x="512" /> +<glyph unicode=" " horiz-adv-x="384" /> +<glyph unicode=" " horiz-adv-x="256" /> +<glyph unicode=" " horiz-adv-x="256" /> +<glyph unicode=" " horiz-adv-x="192" /> +<glyph unicode=" " horiz-adv-x="307" /> +<glyph unicode=" " horiz-adv-x="85" /> +<glyph unicode=" " horiz-adv-x="307" /> +<glyph unicode=" " horiz-adv-x="384" /> +<glyph unicode="™" horiz-adv-x="1792" /> +<glyph unicode="∞" horiz-adv-x="1792" /> +<glyph unicode="≠" horiz-adv-x="1792" /> +<glyph unicode="" horiz-adv-x="500" d="M0 0z" /> +<glyph unicode="" horiz-adv-x="1792" d="M1699 1350q0 -35 -43 -78l-632 -632v-768h320q26 0 45 -19t19 -45t-19 -45t-45 -19h-896q-26 0 -45 19t-19 45t19 45t45 19h320v768l-632 632q-43 43 -43 78q0 23 18 36.5t38 17.5t43 4h1408q23 0 43 -4t38 -17.5t18 -36.5z" /> +<glyph unicode="" d="M1536 1312v-1120q0 -50 -34 -89t-86 -60.5t-103.5 -32t-96.5 -10.5t-96.5 10.5t-103.5 32t-86 60.5t-34 89t34 89t86 60.5t103.5 32t96.5 10.5q105 0 192 -39v537l-768 -237v-709q0 -50 -34 -89t-86 -60.5t-103.5 -32t-96.5 -10.5t-96.5 10.5t-103.5 32t-86 60.5t-34 89 t34 89t86 60.5t103.5 32t96.5 10.5q105 0 192 -39v967q0 31 19 56.5t49 35.5l832 256q12 4 28 4q40 0 68 -28t28 -68z" /> +<glyph unicode="" horiz-adv-x="1664" d="M1152 704q0 185 -131.5 316.5t-316.5 131.5t-316.5 -131.5t-131.5 -316.5t131.5 -316.5t316.5 -131.5t316.5 131.5t131.5 316.5zM1664 -128q0 -52 -38 -90t-90 -38q-54 0 -90 38l-343 342q-179 -124 -399 -124q-143 0 -273.5 55.5t-225 150t-150 225t-55.5 273.5 t55.5 273.5t150 225t225 150t273.5 55.5t273.5 -55.5t225 -150t150 -225t55.5 -273.5q0 -220 -124 -399l343 -343q37 -37 37 -90z" /> +<glyph unicode="" horiz-adv-x="1792" d="M1664 32v768q-32 -36 -69 -66q-268 -206 -426 -338q-51 -43 -83 -67t-86.5 -48.5t-102.5 -24.5h-1h-1q-48 0 -102.5 24.5t-86.5 48.5t-83 67q-158 132 -426 338q-37 30 -69 66v-768q0 -13 9.5 -22.5t22.5 -9.5h1472q13 0 22.5 9.5t9.5 22.5zM1664 1083v11v13.5t-0.5 13 t-3 12.5t-5.5 9t-9 7.5t-14 2.5h-1472q-13 0 -22.5 -9.5t-9.5 -22.5q0 -168 147 -284q193 -152 401 -317q6 -5 35 -29.5t46 -37.5t44.5 -31.5t50.5 -27.5t43 -9h1h1q20 0 43 9t50.5 27.5t44.5 31.5t46 37.5t35 29.5q208 165 401 317q54 43 100.5 115.5t46.5 131.5z M1792 1120v-1088q0 -66 -47 -113t-113 -47h-1472q-66 0 -113 47t-47 113v1088q0 66 47 113t113 47h1472q66 0 113 -47t47 -113z" /> +<glyph unicode="" horiz-adv-x="1792" d="M896 -128q-26 0 -44 18l-624 602q-10 8 -27.5 26t-55.5 65.5t-68 97.5t-53.5 121t-23.5 138q0 220 127 344t351 124q62 0 126.5 -21.5t120 -58t95.5 -68.5t76 -68q36 36 76 68t95.5 68.5t120 58t126.5 21.5q224 0 351 -124t127 -344q0 -221 -229 -450l-623 -600 q-18 -18 -44 -18z" /> +<glyph unicode="" horiz-adv-x="1664" d="M1664 889q0 -22 -26 -48l-363 -354l86 -500q1 -7 1 -20q0 -21 -10.5 -35.5t-30.5 -14.5q-19 0 -40 12l-449 236l-449 -236q-22 -12 -40 -12q-21 0 -31.5 14.5t-10.5 35.5q0 6 2 20l86 500l-364 354q-25 27 -25 48q0 37 56 46l502 73l225 455q19 41 49 41t49 -41l225 -455 l502 -73q56 -9 56 -46z" /> +<glyph unicode="" horiz-adv-x="1664" d="M1137 532l306 297l-422 62l-189 382l-189 -382l-422 -62l306 -297l-73 -421l378 199l377 -199zM1664 889q0 -22 -26 -48l-363 -354l86 -500q1 -7 1 -20q0 -50 -41 -50q-19 0 -40 12l-449 236l-449 -236q-22 -12 -40 -12q-21 0 -31.5 14.5t-10.5 35.5q0 6 2 20l86 500 l-364 354q-25 27 -25 48q0 37 56 46l502 73l225 455q19 41 49 41t49 -41l225 -455l502 -73q56 -9 56 -46z" /> +<glyph unicode="" horiz-adv-x="1408" d="M1408 131q0 -120 -73 -189.5t-194 -69.5h-874q-121 0 -194 69.5t-73 189.5q0 53 3.5 103.5t14 109t26.5 108.5t43 97.5t62 81t85.5 53.5t111.5 20q9 0 42 -21.5t74.5 -48t108 -48t133.5 -21.5t133.5 21.5t108 48t74.5 48t42 21.5q61 0 111.5 -20t85.5 -53.5t62 -81 t43 -97.5t26.5 -108.5t14 -109t3.5 -103.5zM1088 1024q0 -159 -112.5 -271.5t-271.5 -112.5t-271.5 112.5t-112.5 271.5t112.5 271.5t271.5 112.5t271.5 -112.5t112.5 -271.5z" /> +<glyph unicode="" horiz-adv-x="1920" d="M384 -64v128q0 26 -19 45t-45 19h-128q-26 0 -45 -19t-19 -45v-128q0 -26 19 -45t45 -19h128q26 0 45 19t19 45zM384 320v128q0 26 -19 45t-45 19h-128q-26 0 -45 -19t-19 -45v-128q0 -26 19 -45t45 -19h128q26 0 45 19t19 45zM384 704v128q0 26 -19 45t-45 19h-128 q-26 0 -45 -19t-19 -45v-128q0 -26 19 -45t45 -19h128q26 0 45 19t19 45zM1408 -64v512q0 26 -19 45t-45 19h-768q-26 0 -45 -19t-19 -45v-512q0 -26 19 -45t45 -19h768q26 0 45 19t19 45zM384 1088v128q0 26 -19 45t-45 19h-128q-26 0 -45 -19t-19 -45v-128q0 -26 19 -45 t45 -19h128q26 0 45 19t19 45zM1792 -64v128q0 26 -19 45t-45 19h-128q-26 0 -45 -19t-19 -45v-128q0 -26 19 -45t45 -19h128q26 0 45 19t19 45zM1408 704v512q0 26 -19 45t-45 19h-768q-26 0 -45 -19t-19 -45v-512q0 -26 19 -45t45 -19h768q26 0 45 19t19 45zM1792 320v128 q0 26 -19 45t-45 19h-128q-26 0 -45 -19t-19 -45v-128q0 -26 19 -45t45 -19h128q26 0 45 19t19 45zM1792 704v128q0 26 -19 45t-45 19h-128q-26 0 -45 -19t-19 -45v-128q0 -26 19 -45t45 -19h128q26 0 45 19t19 45zM1792 1088v128q0 26 -19 45t-45 19h-128q-26 0 -45 -19 t-19 -45v-128q0 -26 19 -45t45 -19h128q26 0 45 19t19 45zM1920 1248v-1344q0 -66 -47 -113t-113 -47h-1600q-66 0 -113 47t-47 113v1344q0 66 47 113t113 47h1600q66 0 113 -47t47 -113z" /> +<glyph unicode="" horiz-adv-x="1664" d="M768 512v-384q0 -52 -38 -90t-90 -38h-512q-52 0 -90 38t-38 90v384q0 52 38 90t90 38h512q52 0 90 -38t38 -90zM768 1280v-384q0 -52 -38 -90t-90 -38h-512q-52 0 -90 38t-38 90v384q0 52 38 90t90 38h512q52 0 90 -38t38 -90zM1664 512v-384q0 -52 -38 -90t-90 -38 h-512q-52 0 -90 38t-38 90v384q0 52 38 90t90 38h512q52 0 90 -38t38 -90zM1664 1280v-384q0 -52 -38 -90t-90 -38h-512q-52 0 -90 38t-38 90v384q0 52 38 90t90 38h512q52 0 90 -38t38 -90z" /> +<glyph unicode="" horiz-adv-x="1792" d="M512 288v-192q0 -40 -28 -68t-68 -28h-320q-40 0 -68 28t-28 68v192q0 40 28 68t68 28h320q40 0 68 -28t28 -68zM512 800v-192q0 -40 -28 -68t-68 -28h-320q-40 0 -68 28t-28 68v192q0 40 28 68t68 28h320q40 0 68 -28t28 -68zM1152 288v-192q0 -40 -28 -68t-68 -28h-320 q-40 0 -68 28t-28 68v192q0 40 28 68t68 28h320q40 0 68 -28t28 -68zM512 1312v-192q0 -40 -28 -68t-68 -28h-320q-40 0 -68 28t-28 68v192q0 40 28 68t68 28h320q40 0 68 -28t28 -68zM1152 800v-192q0 -40 -28 -68t-68 -28h-320q-40 0 -68 28t-28 68v192q0 40 28 68t68 28 h320q40 0 68 -28t28 -68zM1792 288v-192q0 -40 -28 -68t-68 -28h-320q-40 0 -68 28t-28 68v192q0 40 28 68t68 28h320q40 0 68 -28t28 -68zM1152 1312v-192q0 -40 -28 -68t-68 -28h-320q-40 0 -68 28t-28 68v192q0 40 28 68t68 28h320q40 0 68 -28t28 -68zM1792 800v-192 q0 -40 -28 -68t-68 -28h-320q-40 0 -68 28t-28 68v192q0 40 28 68t68 28h320q40 0 68 -28t28 -68zM1792 1312v-192q0 -40 -28 -68t-68 -28h-320q-40 0 -68 28t-28 68v192q0 40 28 68t68 28h320q40 0 68 -28t28 -68z" /> +<glyph unicode="" horiz-adv-x="1792" d="M512 288v-192q0 -40 -28 -68t-68 -28h-320q-40 0 -68 28t-28 68v192q0 40 28 68t68 28h320q40 0 68 -28t28 -68zM512 800v-192q0 -40 -28 -68t-68 -28h-320q-40 0 -68 28t-28 68v192q0 40 28 68t68 28h320q40 0 68 -28t28 -68zM1792 288v-192q0 -40 -28 -68t-68 -28h-960 q-40 0 -68 28t-28 68v192q0 40 28 68t68 28h960q40 0 68 -28t28 -68zM512 1312v-192q0 -40 -28 -68t-68 -28h-320q-40 0 -68 28t-28 68v192q0 40 28 68t68 28h320q40 0 68 -28t28 -68zM1792 800v-192q0 -40 -28 -68t-68 -28h-960q-40 0 -68 28t-28 68v192q0 40 28 68t68 28 h960q40 0 68 -28t28 -68zM1792 1312v-192q0 -40 -28 -68t-68 -28h-960q-40 0 -68 28t-28 68v192q0 40 28 68t68 28h960q40 0 68 -28t28 -68z" /> +<glyph unicode="" horiz-adv-x="1792" d="M1671 970q0 -40 -28 -68l-724 -724l-136 -136q-28 -28 -68 -28t-68 28l-136 136l-362 362q-28 28 -28 68t28 68l136 136q28 28 68 28t68 -28l294 -295l656 657q28 28 68 28t68 -28l136 -136q28 -28 28 -68z" /> +<glyph unicode="" horiz-adv-x="1408" d="M1298 214q0 -40 -28 -68l-136 -136q-28 -28 -68 -28t-68 28l-294 294l-294 -294q-28 -28 -68 -28t-68 28l-136 136q-28 28 -28 68t28 68l294 294l-294 294q-28 28 -28 68t28 68l136 136q28 28 68 28t68 -28l294 -294l294 294q28 28 68 28t68 -28l136 -136q28 -28 28 -68 t-28 -68l-294 -294l294 -294q28 -28 28 -68z" /> +<glyph unicode="" horiz-adv-x="1664" d="M1024 736v-64q0 -13 -9.5 -22.5t-22.5 -9.5h-224v-224q0 -13 -9.5 -22.5t-22.5 -9.5h-64q-13 0 -22.5 9.5t-9.5 22.5v224h-224q-13 0 -22.5 9.5t-9.5 22.5v64q0 13 9.5 22.5t22.5 9.5h224v224q0 13 9.5 22.5t22.5 9.5h64q13 0 22.5 -9.5t9.5 -22.5v-224h224 q13 0 22.5 -9.5t9.5 -22.5zM1152 704q0 185 -131.5 316.5t-316.5 131.5t-316.5 -131.5t-131.5 -316.5t131.5 -316.5t316.5 -131.5t316.5 131.5t131.5 316.5zM1664 -128q0 -53 -37.5 -90.5t-90.5 -37.5q-54 0 -90 38l-343 342q-179 -124 -399 -124q-143 0 -273.5 55.5 t-225 150t-150 225t-55.5 273.5t55.5 273.5t150 225t225 150t273.5 55.5t273.5 -55.5t225 -150t150 -225t55.5 -273.5q0 -220 -124 -399l343 -343q37 -37 37 -90z" /> +<glyph unicode="" horiz-adv-x="1664" d="M1024 736v-64q0 -13 -9.5 -22.5t-22.5 -9.5h-576q-13 0 -22.5 9.5t-9.5 22.5v64q0 13 9.5 22.5t22.5 9.5h576q13 0 22.5 -9.5t9.5 -22.5zM1152 704q0 185 -131.5 316.5t-316.5 131.5t-316.5 -131.5t-131.5 -316.5t131.5 -316.5t316.5 -131.5t316.5 131.5t131.5 316.5z M1664 -128q0 -53 -37.5 -90.5t-90.5 -37.5q-54 0 -90 38l-343 342q-179 -124 -399 -124q-143 0 -273.5 55.5t-225 150t-150 225t-55.5 273.5t55.5 273.5t150 225t225 150t273.5 55.5t273.5 -55.5t225 -150t150 -225t55.5 -273.5q0 -220 -124 -399l343 -343q37 -37 37 -90z " /> +<glyph unicode="" d="M1536 640q0 -156 -61 -298t-164 -245t-245 -164t-298 -61t-298 61t-245 164t-164 245t-61 298q0 182 80.5 343t226.5 270q43 32 95.5 25t83.5 -50q32 -42 24.5 -94.5t-49.5 -84.5q-98 -74 -151.5 -181t-53.5 -228q0 -104 40.5 -198.5t109.5 -163.5t163.5 -109.5 t198.5 -40.5t198.5 40.5t163.5 109.5t109.5 163.5t40.5 198.5q0 121 -53.5 228t-151.5 181q-42 32 -49.5 84.5t24.5 94.5q31 43 84 50t95 -25q146 -109 226.5 -270t80.5 -343zM896 1408v-640q0 -52 -38 -90t-90 -38t-90 38t-38 90v640q0 52 38 90t90 38t90 -38t38 -90z" /> +<glyph unicode="" horiz-adv-x="1792" d="M256 96v-192q0 -14 -9 -23t-23 -9h-192q-14 0 -23 9t-9 23v192q0 14 9 23t23 9h192q14 0 23 -9t9 -23zM640 224v-320q0 -14 -9 -23t-23 -9h-192q-14 0 -23 9t-9 23v320q0 14 9 23t23 9h192q14 0 23 -9t9 -23zM1024 480v-576q0 -14 -9 -23t-23 -9h-192q-14 0 -23 9t-9 23 v576q0 14 9 23t23 9h192q14 0 23 -9t9 -23zM1408 864v-960q0 -14 -9 -23t-23 -9h-192q-14 0 -23 9t-9 23v960q0 14 9 23t23 9h192q14 0 23 -9t9 -23zM1792 1376v-1472q0 -14 -9 -23t-23 -9h-192q-14 0 -23 9t-9 23v1472q0 14 9 23t23 9h192q14 0 23 -9t9 -23z" /> +<glyph unicode="" d="M1024 640q0 106 -75 181t-181 75t-181 -75t-75 -181t75 -181t181 -75t181 75t75 181zM1536 749v-222q0 -12 -8 -23t-20 -13l-185 -28q-19 -54 -39 -91q35 -50 107 -138q10 -12 10 -25t-9 -23q-27 -37 -99 -108t-94 -71q-12 0 -26 9l-138 108q-44 -23 -91 -38 q-16 -136 -29 -186q-7 -28 -36 -28h-222q-14 0 -24.5 8.5t-11.5 21.5l-28 184q-49 16 -90 37l-141 -107q-10 -9 -25 -9q-14 0 -25 11q-126 114 -165 168q-7 10 -7 23q0 12 8 23q15 21 51 66.5t54 70.5q-27 50 -41 99l-183 27q-13 2 -21 12.5t-8 23.5v222q0 12 8 23t19 13 l186 28q14 46 39 92q-40 57 -107 138q-10 12 -10 24q0 10 9 23q26 36 98.5 107.5t94.5 71.5q13 0 26 -10l138 -107q44 23 91 38q16 136 29 186q7 28 36 28h222q14 0 24.5 -8.5t11.5 -21.5l28 -184q49 -16 90 -37l142 107q9 9 24 9q13 0 25 -10q129 -119 165 -170q7 -8 7 -22 q0 -12 -8 -23q-15 -21 -51 -66.5t-54 -70.5q26 -50 41 -98l183 -28q13 -2 21 -12.5t8 -23.5z" /> +<glyph unicode="" horiz-adv-x="1408" d="M512 800v-576q0 -14 -9 -23t-23 -9h-64q-14 0 -23 9t-9 23v576q0 14 9 23t23 9h64q14 0 23 -9t9 -23zM768 800v-576q0 -14 -9 -23t-23 -9h-64q-14 0 -23 9t-9 23v576q0 14 9 23t23 9h64q14 0 23 -9t9 -23zM1024 800v-576q0 -14 -9 -23t-23 -9h-64q-14 0 -23 9t-9 23v576 q0 14 9 23t23 9h64q14 0 23 -9t9 -23zM1152 76v948h-896v-948q0 -22 7 -40.5t14.5 -27t10.5 -8.5h832q3 0 10.5 8.5t14.5 27t7 40.5zM480 1152h448l-48 117q-7 9 -17 11h-317q-10 -2 -17 -11zM1408 1120v-64q0 -14 -9 -23t-23 -9h-96v-948q0 -83 -47 -143.5t-113 -60.5h-832 q-66 0 -113 58.5t-47 141.5v952h-96q-14 0 -23 9t-9 23v64q0 14 9 23t23 9h309l70 167q15 37 54 63t79 26h320q40 0 79 -26t54 -63l70 -167h309q14 0 23 -9t9 -23z" /> +<glyph unicode="" horiz-adv-x="1664" d="M1408 544v-480q0 -26 -19 -45t-45 -19h-384v384h-256v-384h-384q-26 0 -45 19t-19 45v480q0 1 0.5 3t0.5 3l575 474l575 -474q1 -2 1 -6zM1631 613l-62 -74q-8 -9 -21 -11h-3q-13 0 -21 7l-692 577l-692 -577q-12 -8 -24 -7q-13 2 -21 11l-62 74q-8 10 -7 23.5t11 21.5 l719 599q32 26 76 26t76 -26l244 -204v195q0 14 9 23t23 9h192q14 0 23 -9t9 -23v-408l219 -182q10 -8 11 -21.5t-7 -23.5z" /> +<glyph unicode="" horiz-adv-x="1280" d="M128 0h1024v768h-416q-40 0 -68 28t-28 68v416h-512v-1280zM768 896h376q-10 29 -22 41l-313 313q-12 12 -41 22v-376zM1280 864v-896q0 -40 -28 -68t-68 -28h-1088q-40 0 -68 28t-28 68v1344q0 40 28 68t68 28h640q40 0 88 -20t76 -48l312 -312q28 -28 48 -76t20 -88z " /> +<glyph unicode="" d="M896 992v-448q0 -14 -9 -23t-23 -9h-320q-14 0 -23 9t-9 23v64q0 14 9 23t23 9h224v352q0 14 9 23t23 9h64q14 0 23 -9t9 -23zM1312 640q0 148 -73 273t-198 198t-273 73t-273 -73t-198 -198t-73 -273t73 -273t198 -198t273 -73t273 73t198 198t73 273zM1536 640 q0 -209 -103 -385.5t-279.5 -279.5t-385.5 -103t-385.5 103t-279.5 279.5t-103 385.5t103 385.5t279.5 279.5t385.5 103t385.5 -103t279.5 -279.5t103 -385.5z" /> +<glyph unicode="" horiz-adv-x="1920" d="M1111 540v4l-24 320q-1 13 -11 22.5t-23 9.5h-186q-13 0 -23 -9.5t-11 -22.5l-24 -320v-4q-1 -12 8 -20t21 -8h244q12 0 21 8t8 20zM1870 73q0 -73 -46 -73h-704q13 0 22 9.5t8 22.5l-20 256q-1 13 -11 22.5t-23 9.5h-272q-13 0 -23 -9.5t-11 -22.5l-20 -256 q-1 -13 8 -22.5t22 -9.5h-704q-46 0 -46 73q0 54 26 116l417 1044q8 19 26 33t38 14h339q-13 0 -23 -9.5t-11 -22.5l-15 -192q-1 -14 8 -23t22 -9h166q13 0 22 9t8 23l-15 192q-1 13 -11 22.5t-23 9.5h339q20 0 38 -14t26 -33l417 -1044q26 -62 26 -116z" /> +<glyph unicode="" horiz-adv-x="1664" d="M1280 192q0 26 -19 45t-45 19t-45 -19t-19 -45t19 -45t45 -19t45 19t19 45zM1536 192q0 26 -19 45t-45 19t-45 -19t-19 -45t19 -45t45 -19t45 19t19 45zM1664 416v-320q0 -40 -28 -68t-68 -28h-1472q-40 0 -68 28t-28 68v320q0 40 28 68t68 28h465l135 -136 q58 -56 136 -56t136 56l136 136h464q40 0 68 -28t28 -68zM1339 985q17 -41 -14 -70l-448 -448q-18 -19 -45 -19t-45 19l-448 448q-31 29 -14 70q17 39 59 39h256v448q0 26 19 45t45 19h256q26 0 45 -19t19 -45v-448h256q42 0 59 -39z" /> +<glyph unicode="" d="M1120 608q0 -12 -10 -24l-319 -319q-11 -9 -23 -9t-23 9l-320 320q-15 16 -7 35q8 20 30 20h192v352q0 14 9 23t23 9h192q14 0 23 -9t9 -23v-352h192q14 0 23 -9t9 -23zM768 1184q-148 0 -273 -73t-198 -198t-73 -273t73 -273t198 -198t273 -73t273 73t198 198t73 273 t-73 273t-198 198t-273 73zM1536 640q0 -209 -103 -385.5t-279.5 -279.5t-385.5 -103t-385.5 103t-279.5 279.5t-103 385.5t103 385.5t279.5 279.5t385.5 103t385.5 -103t279.5 -279.5t103 -385.5z" /> +<glyph unicode="" d="M1118 660q-8 -20 -30 -20h-192v-352q0 -14 -9 -23t-23 -9h-192q-14 0 -23 9t-9 23v352h-192q-14 0 -23 9t-9 23q0 12 10 24l319 319q11 9 23 9t23 -9l320 -320q15 -16 7 -35zM768 1184q-148 0 -273 -73t-198 -198t-73 -273t73 -273t198 -198t273 -73t273 73t198 198 t73 273t-73 273t-198 198t-273 73zM1536 640q0 -209 -103 -385.5t-279.5 -279.5t-385.5 -103t-385.5 103t-279.5 279.5t-103 385.5t103 385.5t279.5 279.5t385.5 103t385.5 -103t279.5 -279.5t103 -385.5z" /> +<glyph unicode="" d="M1023 576h316q-1 3 -2.5 8t-2.5 8l-212 496h-708l-212 -496q-1 -2 -2.5 -8t-2.5 -8h316l95 -192h320zM1536 546v-482q0 -26 -19 -45t-45 -19h-1408q-26 0 -45 19t-19 45v482q0 62 25 123l238 552q10 25 36.5 42t52.5 17h832q26 0 52.5 -17t36.5 -42l238 -552 q25 -61 25 -123z" /> +<glyph unicode="" d="M1184 640q0 -37 -32 -55l-544 -320q-15 -9 -32 -9q-16 0 -32 8q-32 19 -32 56v640q0 37 32 56q33 18 64 -1l544 -320q32 -18 32 -55zM1312 640q0 148 -73 273t-198 198t-273 73t-273 -73t-198 -198t-73 -273t73 -273t198 -198t273 -73t273 73t198 198t73 273zM1536 640 q0 -209 -103 -385.5t-279.5 -279.5t-385.5 -103t-385.5 103t-279.5 279.5t-103 385.5t103 385.5t279.5 279.5t385.5 103t385.5 -103t279.5 -279.5t103 -385.5z" /> +<glyph unicode="" d="M1536 1280v-448q0 -26 -19 -45t-45 -19h-448q-42 0 -59 40q-17 39 14 69l138 138q-148 137 -349 137q-104 0 -198.5 -40.5t-163.5 -109.5t-109.5 -163.5t-40.5 -198.5t40.5 -198.5t109.5 -163.5t163.5 -109.5t198.5 -40.5q119 0 225 52t179 147q7 10 23 12q14 0 25 -9 l137 -138q9 -8 9.5 -20.5t-7.5 -22.5q-109 -132 -264 -204.5t-327 -72.5q-156 0 -298 61t-245 164t-164 245t-61 298t61 298t164 245t245 164t298 61q147 0 284.5 -55.5t244.5 -156.5l130 129q29 31 70 14q39 -17 39 -59z" /> +<glyph unicode="" d="M1511 480q0 -5 -1 -7q-64 -268 -268 -434.5t-478 -166.5q-146 0 -282.5 55t-243.5 157l-129 -129q-19 -19 -45 -19t-45 19t-19 45v448q0 26 19 45t45 19h448q26 0 45 -19t19 -45t-19 -45l-137 -137q71 -66 161 -102t187 -36q134 0 250 65t186 179q11 17 53 117 q8 23 30 23h192q13 0 22.5 -9.5t9.5 -22.5zM1536 1280v-448q0 -26 -19 -45t-45 -19h-448q-26 0 -45 19t-19 45t19 45l138 138q-148 137 -349 137q-134 0 -250 -65t-186 -179q-11 -17 -53 -117q-8 -23 -30 -23h-199q-13 0 -22.5 9.5t-9.5 22.5v7q65 268 270 434.5t480 166.5 q146 0 284 -55.5t245 -156.5l130 129q19 19 45 19t45 -19t19 -45z" /> +<glyph unicode="" horiz-adv-x="1792" d="M384 352v-64q0 -13 -9.5 -22.5t-22.5 -9.5h-64q-13 0 -22.5 9.5t-9.5 22.5v64q0 13 9.5 22.5t22.5 9.5h64q13 0 22.5 -9.5t9.5 -22.5zM384 608v-64q0 -13 -9.5 -22.5t-22.5 -9.5h-64q-13 0 -22.5 9.5t-9.5 22.5v64q0 13 9.5 22.5t22.5 9.5h64q13 0 22.5 -9.5t9.5 -22.5z M384 864v-64q0 -13 -9.5 -22.5t-22.5 -9.5h-64q-13 0 -22.5 9.5t-9.5 22.5v64q0 13 9.5 22.5t22.5 9.5h64q13 0 22.5 -9.5t9.5 -22.5zM1536 352v-64q0 -13 -9.5 -22.5t-22.5 -9.5h-960q-13 0 -22.5 9.5t-9.5 22.5v64q0 13 9.5 22.5t22.5 9.5h960q13 0 22.5 -9.5t9.5 -22.5z M1536 608v-64q0 -13 -9.5 -22.5t-22.5 -9.5h-960q-13 0 -22.5 9.5t-9.5 22.5v64q0 13 9.5 22.5t22.5 9.5h960q13 0 22.5 -9.5t9.5 -22.5zM1536 864v-64q0 -13 -9.5 -22.5t-22.5 -9.5h-960q-13 0 -22.5 9.5t-9.5 22.5v64q0 13 9.5 22.5t22.5 9.5h960q13 0 22.5 -9.5 t9.5 -22.5zM1664 160v832q0 13 -9.5 22.5t-22.5 9.5h-1472q-13 0 -22.5 -9.5t-9.5 -22.5v-832q0 -13 9.5 -22.5t22.5 -9.5h1472q13 0 22.5 9.5t9.5 22.5zM1792 1248v-1088q0 -66 -47 -113t-113 -47h-1472q-66 0 -113 47t-47 113v1088q0 66 47 113t113 47h1472q66 0 113 -47 t47 -113z" /> +<glyph unicode="" horiz-adv-x="1152" d="M320 768h512v192q0 106 -75 181t-181 75t-181 -75t-75 -181v-192zM1152 672v-576q0 -40 -28 -68t-68 -28h-960q-40 0 -68 28t-28 68v576q0 40 28 68t68 28h32v192q0 184 132 316t316 132t316 -132t132 -316v-192h32q40 0 68 -28t28 -68z" /> +<glyph unicode="" horiz-adv-x="1792" d="M320 1280q0 -72 -64 -110v-1266q0 -13 -9.5 -22.5t-22.5 -9.5h-64q-13 0 -22.5 9.5t-9.5 22.5v1266q-64 38 -64 110q0 53 37.5 90.5t90.5 37.5t90.5 -37.5t37.5 -90.5zM1792 1216v-763q0 -25 -12.5 -38.5t-39.5 -27.5q-215 -116 -369 -116q-61 0 -123.5 22t-108.5 48 t-115.5 48t-142.5 22q-192 0 -464 -146q-17 -9 -33 -9q-26 0 -45 19t-19 45v742q0 32 31 55q21 14 79 43q236 120 421 120q107 0 200 -29t219 -88q38 -19 88 -19q54 0 117.5 21t110 47t88 47t54.5 21q26 0 45 -19t19 -45z" /> +<glyph unicode="" horiz-adv-x="1664" d="M1664 650q0 -166 -60 -314l-20 -49l-185 -33q-22 -83 -90.5 -136.5t-156.5 -53.5v-32q0 -14 -9 -23t-23 -9h-64q-14 0 -23 9t-9 23v576q0 14 9 23t23 9h64q14 0 23 -9t9 -23v-32q71 0 130 -35.5t93 -95.5l68 12q29 95 29 193q0 148 -88 279t-236.5 209t-315.5 78 t-315.5 -78t-236.5 -209t-88 -279q0 -98 29 -193l68 -12q34 60 93 95.5t130 35.5v32q0 14 9 23t23 9h64q14 0 23 -9t9 -23v-576q0 -14 -9 -23t-23 -9h-64q-14 0 -23 9t-9 23v32q-88 0 -156.5 53.5t-90.5 136.5l-185 33l-20 49q-60 148 -60 314q0 151 67 291t179 242.5 t266 163.5t320 61t320 -61t266 -163.5t179 -242.5t67 -291z" /> +<glyph unicode="" horiz-adv-x="768" d="M768 1184v-1088q0 -26 -19 -45t-45 -19t-45 19l-333 333h-262q-26 0 -45 19t-19 45v384q0 26 19 45t45 19h262l333 333q19 19 45 19t45 -19t19 -45z" /> +<glyph unicode="" horiz-adv-x="1152" d="M768 1184v-1088q0 -26 -19 -45t-45 -19t-45 19l-333 333h-262q-26 0 -45 19t-19 45v384q0 26 19 45t45 19h262l333 333q19 19 45 19t45 -19t19 -45zM1152 640q0 -76 -42.5 -141.5t-112.5 -93.5q-10 -5 -25 -5q-26 0 -45 18.5t-19 45.5q0 21 12 35.5t29 25t34 23t29 35.5 t12 57t-12 57t-29 35.5t-34 23t-29 25t-12 35.5q0 27 19 45.5t45 18.5q15 0 25 -5q70 -27 112.5 -93t42.5 -142z" /> +<glyph unicode="" horiz-adv-x="1664" d="M768 1184v-1088q0 -26 -19 -45t-45 -19t-45 19l-333 333h-262q-26 0 -45 19t-19 45v384q0 26 19 45t45 19h262l333 333q19 19 45 19t45 -19t19 -45zM1152 640q0 -76 -42.5 -141.5t-112.5 -93.5q-10 -5 -25 -5q-26 0 -45 18.5t-19 45.5q0 21 12 35.5t29 25t34 23t29 35.5 t12 57t-12 57t-29 35.5t-34 23t-29 25t-12 35.5q0 27 19 45.5t45 18.5q15 0 25 -5q70 -27 112.5 -93t42.5 -142zM1408 640q0 -153 -85 -282.5t-225 -188.5q-13 -5 -25 -5q-27 0 -46 19t-19 45q0 39 39 59q56 29 76 44q74 54 115.5 135.5t41.5 173.5t-41.5 173.5 t-115.5 135.5q-20 15 -76 44q-39 20 -39 59q0 26 19 45t45 19q13 0 26 -5q140 -59 225 -188.5t85 -282.5zM1664 640q0 -230 -127 -422.5t-338 -283.5q-13 -5 -26 -5q-26 0 -45 19t-19 45q0 36 39 59q7 4 22.5 10.5t22.5 10.5q46 25 82 51q123 91 192 227t69 289t-69 289 t-192 227q-36 26 -82 51q-7 4 -22.5 10.5t-22.5 10.5q-39 23 -39 59q0 26 19 45t45 19q13 0 26 -5q211 -91 338 -283.5t127 -422.5z" /> +<glyph unicode="" horiz-adv-x="1408" d="M384 384v-128h-128v128h128zM384 1152v-128h-128v128h128zM1152 1152v-128h-128v128h128zM128 129h384v383h-384v-383zM128 896h384v384h-384v-384zM896 896h384v384h-384v-384zM640 640v-640h-640v640h640zM1152 128v-128h-128v128h128zM1408 128v-128h-128v128h128z M1408 640v-384h-384v128h-128v-384h-128v640h384v-128h128v128h128zM640 1408v-640h-640v640h640zM1408 1408v-640h-640v640h640z" /> +<glyph unicode="" horiz-adv-x="1792" d="M63 0h-63v1408h63v-1408zM126 1h-32v1407h32v-1407zM220 1h-31v1407h31v-1407zM377 1h-31v1407h31v-1407zM534 1h-62v1407h62v-1407zM660 1h-31v1407h31v-1407zM723 1h-31v1407h31v-1407zM786 1h-31v1407h31v-1407zM943 1h-63v1407h63v-1407zM1100 1h-63v1407h63v-1407z M1226 1h-63v1407h63v-1407zM1352 1h-63v1407h63v-1407zM1446 1h-63v1407h63v-1407zM1635 1h-94v1407h94v-1407zM1698 1h-32v1407h32v-1407zM1792 0h-63v1408h63v-1408z" /> +<glyph unicode="" d="M448 1088q0 53 -37.5 90.5t-90.5 37.5t-90.5 -37.5t-37.5 -90.5t37.5 -90.5t90.5 -37.5t90.5 37.5t37.5 90.5zM1515 512q0 -53 -37 -90l-491 -492q-39 -37 -91 -37q-53 0 -90 37l-715 716q-38 37 -64.5 101t-26.5 117v416q0 52 38 90t90 38h416q53 0 117 -26.5t102 -64.5 l715 -714q37 -39 37 -91z" /> +<glyph unicode="" horiz-adv-x="1920" d="M448 1088q0 53 -37.5 90.5t-90.5 37.5t-90.5 -37.5t-37.5 -90.5t37.5 -90.5t90.5 -37.5t90.5 37.5t37.5 90.5zM1515 512q0 -53 -37 -90l-491 -492q-39 -37 -91 -37q-53 0 -90 37l-715 716q-38 37 -64.5 101t-26.5 117v416q0 52 38 90t90 38h416q53 0 117 -26.5t102 -64.5 l715 -714q37 -39 37 -91zM1899 512q0 -53 -37 -90l-491 -492q-39 -37 -91 -37q-36 0 -59 14t-53 45l470 470q37 37 37 90q0 52 -37 91l-715 714q-38 38 -102 64.5t-117 26.5h224q53 0 117 -26.5t102 -64.5l715 -714q37 -39 37 -91z" /> +<glyph unicode="" horiz-adv-x="1664" d="M1639 1058q40 -57 18 -129l-275 -906q-19 -64 -76.5 -107.5t-122.5 -43.5h-923q-77 0 -148.5 53.5t-99.5 131.5q-24 67 -2 127q0 4 3 27t4 37q1 8 -3 21.5t-3 19.5q2 11 8 21t16.5 23.5t16.5 23.5q23 38 45 91.5t30 91.5q3 10 0.5 30t-0.5 28q3 11 17 28t17 23 q21 36 42 92t25 90q1 9 -2.5 32t0.5 28q4 13 22 30.5t22 22.5q19 26 42.5 84.5t27.5 96.5q1 8 -3 25.5t-2 26.5q2 8 9 18t18 23t17 21q8 12 16.5 30.5t15 35t16 36t19.5 32t26.5 23.5t36 11.5t47.5 -5.5l-1 -3q38 9 51 9h761q74 0 114 -56t18 -130l-274 -906 q-36 -119 -71.5 -153.5t-128.5 -34.5h-869q-27 0 -38 -15q-11 -16 -1 -43q24 -70 144 -70h923q29 0 56 15.5t35 41.5l300 987q7 22 5 57q38 -15 59 -43zM575 1056q-4 -13 2 -22.5t20 -9.5h608q13 0 25.5 9.5t16.5 22.5l21 64q4 13 -2 22.5t-20 9.5h-608q-13 0 -25.5 -9.5 t-16.5 -22.5zM492 800q-4 -13 2 -22.5t20 -9.5h608q13 0 25.5 9.5t16.5 22.5l21 64q4 13 -2 22.5t-20 9.5h-608q-13 0 -25.5 -9.5t-16.5 -22.5z" /> +<glyph unicode="" horiz-adv-x="1280" d="M1164 1408q23 0 44 -9q33 -13 52.5 -41t19.5 -62v-1289q0 -34 -19.5 -62t-52.5 -41q-19 -8 -44 -8q-48 0 -83 32l-441 424l-441 -424q-36 -33 -83 -33q-23 0 -44 9q-33 13 -52.5 41t-19.5 62v1289q0 34 19.5 62t52.5 41q21 9 44 9h1048z" /> +<glyph unicode="" horiz-adv-x="1664" d="M384 0h896v256h-896v-256zM384 640h896v384h-160q-40 0 -68 28t-28 68v160h-640v-640zM1536 576q0 26 -19 45t-45 19t-45 -19t-19 -45t19 -45t45 -19t45 19t19 45zM1664 576v-416q0 -13 -9.5 -22.5t-22.5 -9.5h-224v-160q0 -40 -28 -68t-68 -28h-960q-40 0 -68 28t-28 68 v160h-224q-13 0 -22.5 9.5t-9.5 22.5v416q0 79 56.5 135.5t135.5 56.5h64v544q0 40 28 68t68 28h672q40 0 88 -20t76 -48l152 -152q28 -28 48 -76t20 -88v-256h64q79 0 135.5 -56.5t56.5 -135.5z" /> +<glyph unicode="" horiz-adv-x="1920" d="M960 864q119 0 203.5 -84.5t84.5 -203.5t-84.5 -203.5t-203.5 -84.5t-203.5 84.5t-84.5 203.5t84.5 203.5t203.5 84.5zM1664 1280q106 0 181 -75t75 -181v-896q0 -106 -75 -181t-181 -75h-1408q-106 0 -181 75t-75 181v896q0 106 75 181t181 75h224l51 136 q19 49 69.5 84.5t103.5 35.5h512q53 0 103.5 -35.5t69.5 -84.5l51 -136h224zM960 128q185 0 316.5 131.5t131.5 316.5t-131.5 316.5t-316.5 131.5t-316.5 -131.5t-131.5 -316.5t131.5 -316.5t316.5 -131.5z" /> +<glyph unicode="" horiz-adv-x="1664" d="M725 977l-170 -450q73 -1 153.5 -2t119 -1.5t52.5 -0.5l29 2q-32 95 -92 241q-53 132 -92 211zM21 -128h-21l2 79q22 7 80 18q89 16 110 31q20 16 48 68l237 616l280 724h75h53l11 -21l205 -480q103 -242 124 -297q39 -102 96 -235q26 -58 65 -164q24 -67 65 -149 q22 -49 35 -57q22 -19 69 -23q47 -6 103 -27q6 -39 6 -57q0 -14 -1 -26q-80 0 -192 8q-93 8 -189 8q-79 0 -135 -2l-200 -11l-58 -2q0 45 4 78l131 28q56 13 68 23q12 12 12 27t-6 32l-47 114l-92 228l-450 2q-29 -65 -104 -274q-23 -64 -23 -84q0 -31 17 -43 q26 -21 103 -32q3 0 13.5 -2t30 -5t40.5 -6q1 -28 1 -58q0 -17 -2 -27q-66 0 -349 20l-48 -8q-81 -14 -167 -14z" /> +<glyph unicode="" horiz-adv-x="1408" d="M555 15q76 -32 140 -32q131 0 216 41t122 113q38 70 38 181q0 114 -41 180q-58 94 -141 126q-80 32 -247 32q-74 0 -101 -10v-144l-1 -173l3 -270q0 -15 12 -44zM541 761q43 -7 109 -7q175 0 264 65t89 224q0 112 -85 187q-84 75 -255 75q-52 0 -130 -13q0 -44 2 -77 q7 -122 6 -279l-1 -98q0 -43 1 -77zM0 -128l2 94q45 9 68 12q77 12 123 31q17 27 21 51q9 66 9 194l-2 497q-5 256 -9 404q-1 87 -11 109q-1 4 -12 12q-18 12 -69 15q-30 2 -114 13l-4 83l260 6l380 13l45 1q5 0 14 0.5t14 0.5q1 0 21.5 -0.5t40.5 -0.5h74q88 0 191 -27 q43 -13 96 -39q57 -29 102 -76q44 -47 65 -104t21 -122q0 -70 -32 -128t-95 -105q-26 -20 -150 -77q177 -41 267 -146q92 -106 92 -236q0 -76 -29 -161q-21 -62 -71 -117q-66 -72 -140 -108q-73 -36 -203 -60q-82 -15 -198 -11l-197 4q-84 2 -298 -11q-33 -3 -272 -11z" /> +<glyph unicode="" horiz-adv-x="1024" d="M0 -126l17 85q4 1 77 20q76 19 116 39q29 37 41 101l27 139l56 268l12 64q8 44 17 84.5t16 67t12.5 46.5t9 30.5t3.5 11.5l29 157l16 63l22 135l8 50v38q-41 22 -144 28q-28 2 -38 4l19 103l317 -14q39 -2 73 -2q66 0 214 9q33 2 68 4.5t36 2.5q-2 -19 -6 -38 q-7 -29 -13 -51q-55 -19 -109 -31q-64 -16 -101 -31q-12 -31 -24 -88q-9 -44 -13 -82q-44 -199 -66 -306l-61 -311l-38 -158l-43 -235l-12 -45q-2 -7 1 -27q64 -15 119 -21q36 -5 66 -10q-1 -29 -7 -58q-7 -31 -9 -41q-18 0 -23 -1q-24 -2 -42 -2q-9 0 -28 3q-19 4 -145 17 l-198 2q-41 1 -174 -11q-74 -7 -98 -9z" /> +<glyph unicode="" horiz-adv-x="1792" d="M81 1407l54 -27q20 -5 211 -5h130l19 3l115 1l215 -1h293l34 -2q14 -1 28 7t21 16l7 8l42 1q15 0 28 -1v-104.5t1 -131.5l1 -100l-1 -58q0 -32 -4 -51q-39 -15 -68 -18q-25 43 -54 128q-8 24 -15.5 62.5t-11.5 65.5t-6 29q-13 15 -27 19q-7 2 -42.5 2t-103.5 -1t-111 -1 q-34 0 -67 -5q-10 -97 -8 -136l1 -152v-332l3 -359l-1 -147q-1 -46 11 -85q49 -25 89 -32q2 0 18 -5t44 -13t43 -12q30 -8 50 -18q5 -45 5 -50q0 -10 -3 -29q-14 -1 -34 -1q-110 0 -187 10q-72 8 -238 8q-88 0 -233 -14q-48 -4 -70 -4q-2 22 -2 26l-1 26v9q21 33 79 49 q139 38 159 50q9 21 12 56q8 192 6 433l-5 428q-1 62 -0.5 118.5t0.5 102.5t-2 57t-6 15q-6 5 -14 6q-38 6 -148 6q-43 0 -100 -13.5t-73 -24.5q-13 -9 -22 -33t-22 -75t-24 -84q-6 -19 -19.5 -32t-20.5 -13q-44 27 -56 44v297v86zM1744 128q33 0 42 -18.5t-11 -44.5 l-126 -162q-20 -26 -49 -26t-49 26l-126 162q-20 26 -11 44.5t42 18.5h80v1024h-80q-33 0 -42 18.5t11 44.5l126 162q20 26 49 26t49 -26l126 -162q20 -26 11 -44.5t-42 -18.5h-80v-1024h80z" /> +<glyph unicode="" d="M81 1407l54 -27q20 -5 211 -5h130l19 3l115 1l446 -1h318l34 -2q14 -1 28 7t21 16l7 8l42 1q15 0 28 -1v-104.5t1 -131.5l1 -100l-1 -58q0 -32 -4 -51q-39 -15 -68 -18q-25 43 -54 128q-8 24 -15.5 62.5t-11.5 65.5t-6 29q-13 15 -27 19q-7 2 -58.5 2t-138.5 -1t-128 -1 q-94 0 -127 -5q-10 -97 -8 -136l1 -152v52l3 -359l-1 -147q-1 -46 11 -85q49 -25 89 -32q2 0 18 -5t44 -13t43 -12q30 -8 50 -18q5 -45 5 -50q0 -10 -3 -29q-14 -1 -34 -1q-110 0 -187 10q-72 8 -238 8q-82 0 -233 -13q-45 -5 -70 -5q-2 22 -2 26l-1 26v9q21 33 79 49 q139 38 159 50q9 21 12 56q6 137 6 433l-5 44q0 265 -2 278q-2 11 -6 15q-6 5 -14 6q-38 6 -148 6q-50 0 -168.5 -14t-132.5 -24q-13 -9 -22 -33t-22 -75t-24 -84q-6 -19 -19.5 -32t-20.5 -13q-44 27 -56 44v297v86zM1505 113q26 -20 26 -49t-26 -49l-162 -126 q-26 -20 -44.5 -11t-18.5 42v80h-1024v-80q0 -33 -18.5 -42t-44.5 11l-162 126q-26 20 -26 49t26 49l162 126q26 20 44.5 11t18.5 -42v-80h1024v80q0 33 18.5 42t44.5 -11z" /> +<glyph unicode="" horiz-adv-x="1792" d="M1792 192v-128q0 -26 -19 -45t-45 -19h-1664q-26 0 -45 19t-19 45v128q0 26 19 45t45 19h1664q26 0 45 -19t19 -45zM1408 576v-128q0 -26 -19 -45t-45 -19h-1280q-26 0 -45 19t-19 45v128q0 26 19 45t45 19h1280q26 0 45 -19t19 -45zM1664 960v-128q0 -26 -19 -45 t-45 -19h-1536q-26 0 -45 19t-19 45v128q0 26 19 45t45 19h1536q26 0 45 -19t19 -45zM1280 1344v-128q0 -26 -19 -45t-45 -19h-1152q-26 0 -45 19t-19 45v128q0 26 19 45t45 19h1152q26 0 45 -19t19 -45z" /> +<glyph unicode="" horiz-adv-x="1792" d="M1792 192v-128q0 -26 -19 -45t-45 -19h-1664q-26 0 -45 19t-19 45v128q0 26 19 45t45 19h1664q26 0 45 -19t19 -45zM1408 576v-128q0 -26 -19 -45t-45 -19h-896q-26 0 -45 19t-19 45v128q0 26 19 45t45 19h896q26 0 45 -19t19 -45zM1664 960v-128q0 -26 -19 -45t-45 -19 h-1408q-26 0 -45 19t-19 45v128q0 26 19 45t45 19h1408q26 0 45 -19t19 -45zM1280 1344v-128q0 -26 -19 -45t-45 -19h-640q-26 0 -45 19t-19 45v128q0 26 19 45t45 19h640q26 0 45 -19t19 -45z" /> +<glyph unicode="" horiz-adv-x="1792" d="M1792 192v-128q0 -26 -19 -45t-45 -19h-1664q-26 0 -45 19t-19 45v128q0 26 19 45t45 19h1664q26 0 45 -19t19 -45zM1792 576v-128q0 -26 -19 -45t-45 -19h-1280q-26 0 -45 19t-19 45v128q0 26 19 45t45 19h1280q26 0 45 -19t19 -45zM1792 960v-128q0 -26 -19 -45 t-45 -19h-1536q-26 0 -45 19t-19 45v128q0 26 19 45t45 19h1536q26 0 45 -19t19 -45zM1792 1344v-128q0 -26 -19 -45t-45 -19h-1152q-26 0 -45 19t-19 45v128q0 26 19 45t45 19h1152q26 0 45 -19t19 -45z" /> +<glyph unicode="" horiz-adv-x="1792" d="M1792 192v-128q0 -26 -19 -45t-45 -19h-1664q-26 0 -45 19t-19 45v128q0 26 19 45t45 19h1664q26 0 45 -19t19 -45zM1792 576v-128q0 -26 -19 -45t-45 -19h-1664q-26 0 -45 19t-19 45v128q0 26 19 45t45 19h1664q26 0 45 -19t19 -45zM1792 960v-128q0 -26 -19 -45 t-45 -19h-1664q-26 0 -45 19t-19 45v128q0 26 19 45t45 19h1664q26 0 45 -19t19 -45zM1792 1344v-128q0 -26 -19 -45t-45 -19h-1664q-26 0 -45 19t-19 45v128q0 26 19 45t45 19h1664q26 0 45 -19t19 -45z" /> +<glyph unicode="" horiz-adv-x="1792" d="M256 224v-192q0 -13 -9.5 -22.5t-22.5 -9.5h-192q-13 0 -22.5 9.5t-9.5 22.5v192q0 13 9.5 22.5t22.5 9.5h192q13 0 22.5 -9.5t9.5 -22.5zM256 608v-192q0 -13 -9.5 -22.5t-22.5 -9.5h-192q-13 0 -22.5 9.5t-9.5 22.5v192q0 13 9.5 22.5t22.5 9.5h192q13 0 22.5 -9.5 t9.5 -22.5zM256 992v-192q0 -13 -9.5 -22.5t-22.5 -9.5h-192q-13 0 -22.5 9.5t-9.5 22.5v192q0 13 9.5 22.5t22.5 9.5h192q13 0 22.5 -9.5t9.5 -22.5zM1792 224v-192q0 -13 -9.5 -22.5t-22.5 -9.5h-1344q-13 0 -22.5 9.5t-9.5 22.5v192q0 13 9.5 22.5t22.5 9.5h1344 q13 0 22.5 -9.5t9.5 -22.5zM256 1376v-192q0 -13 -9.5 -22.5t-22.5 -9.5h-192q-13 0 -22.5 9.5t-9.5 22.5v192q0 13 9.5 22.5t22.5 9.5h192q13 0 22.5 -9.5t9.5 -22.5zM1792 608v-192q0 -13 -9.5 -22.5t-22.5 -9.5h-1344q-13 0 -22.5 9.5t-9.5 22.5v192q0 13 9.5 22.5 t22.5 9.5h1344q13 0 22.5 -9.5t9.5 -22.5zM1792 992v-192q0 -13 -9.5 -22.5t-22.5 -9.5h-1344q-13 0 -22.5 9.5t-9.5 22.5v192q0 13 9.5 22.5t22.5 9.5h1344q13 0 22.5 -9.5t9.5 -22.5zM1792 1376v-192q0 -13 -9.5 -22.5t-22.5 -9.5h-1344q-13 0 -22.5 9.5t-9.5 22.5v192 q0 13 9.5 22.5t22.5 9.5h1344q13 0 22.5 -9.5t9.5 -22.5z" /> +<glyph unicode="" horiz-adv-x="1792" d="M384 992v-576q0 -13 -9.5 -22.5t-22.5 -9.5q-14 0 -23 9l-288 288q-9 9 -9 23t9 23l288 288q9 9 23 9q13 0 22.5 -9.5t9.5 -22.5zM1792 224v-192q0 -13 -9.5 -22.5t-22.5 -9.5h-1728q-13 0 -22.5 9.5t-9.5 22.5v192q0 13 9.5 22.5t22.5 9.5h1728q13 0 22.5 -9.5 t9.5 -22.5zM1792 608v-192q0 -13 -9.5 -22.5t-22.5 -9.5h-1088q-13 0 -22.5 9.5t-9.5 22.5v192q0 13 9.5 22.5t22.5 9.5h1088q13 0 22.5 -9.5t9.5 -22.5zM1792 992v-192q0 -13 -9.5 -22.5t-22.5 -9.5h-1088q-13 0 -22.5 9.5t-9.5 22.5v192q0 13 9.5 22.5t22.5 9.5h1088 q13 0 22.5 -9.5t9.5 -22.5zM1792 1376v-192q0 -13 -9.5 -22.5t-22.5 -9.5h-1728q-13 0 -22.5 9.5t-9.5 22.5v192q0 13 9.5 22.5t22.5 9.5h1728q13 0 22.5 -9.5t9.5 -22.5z" /> +<glyph unicode="" horiz-adv-x="1792" d="M352 704q0 -14 -9 -23l-288 -288q-9 -9 -23 -9q-13 0 -22.5 9.5t-9.5 22.5v576q0 13 9.5 22.5t22.5 9.5q14 0 23 -9l288 -288q9 -9 9 -23zM1792 224v-192q0 -13 -9.5 -22.5t-22.5 -9.5h-1728q-13 0 -22.5 9.5t-9.5 22.5v192q0 13 9.5 22.5t22.5 9.5h1728q13 0 22.5 -9.5 t9.5 -22.5zM1792 608v-192q0 -13 -9.5 -22.5t-22.5 -9.5h-1088q-13 0 -22.5 9.5t-9.5 22.5v192q0 13 9.5 22.5t22.5 9.5h1088q13 0 22.5 -9.5t9.5 -22.5zM1792 992v-192q0 -13 -9.5 -22.5t-22.5 -9.5h-1088q-13 0 -22.5 9.5t-9.5 22.5v192q0 13 9.5 22.5t22.5 9.5h1088 q13 0 22.5 -9.5t9.5 -22.5zM1792 1376v-192q0 -13 -9.5 -22.5t-22.5 -9.5h-1728q-13 0 -22.5 9.5t-9.5 22.5v192q0 13 9.5 22.5t22.5 9.5h1728q13 0 22.5 -9.5t9.5 -22.5z" /> +<glyph unicode="" horiz-adv-x="1792" d="M1792 1184v-1088q0 -42 -39 -59q-13 -5 -25 -5q-27 0 -45 19l-403 403v-166q0 -119 -84.5 -203.5t-203.5 -84.5h-704q-119 0 -203.5 84.5t-84.5 203.5v704q0 119 84.5 203.5t203.5 84.5h704q119 0 203.5 -84.5t84.5 -203.5v-165l403 402q18 19 45 19q12 0 25 -5 q39 -17 39 -59z" /> +<glyph unicode="" horiz-adv-x="1920" d="M640 960q0 -80 -56 -136t-136 -56t-136 56t-56 136t56 136t136 56t136 -56t56 -136zM1664 576v-448h-1408v192l320 320l160 -160l512 512zM1760 1280h-1600q-13 0 -22.5 -9.5t-9.5 -22.5v-1216q0 -13 9.5 -22.5t22.5 -9.5h1600q13 0 22.5 9.5t9.5 22.5v1216 q0 13 -9.5 22.5t-22.5 9.5zM1920 1248v-1216q0 -66 -47 -113t-113 -47h-1600q-66 0 -113 47t-47 113v1216q0 66 47 113t113 47h1600q66 0 113 -47t47 -113z" /> +<glyph unicode="" d="M363 0l91 91l-235 235l-91 -91v-107h128v-128h107zM886 928q0 22 -22 22q-10 0 -17 -7l-542 -542q-7 -7 -7 -17q0 -22 22 -22q10 0 17 7l542 542q7 7 7 17zM832 1120l416 -416l-832 -832h-416v416zM1515 1024q0 -53 -37 -90l-166 -166l-416 416l166 165q36 38 90 38 q53 0 91 -38l235 -234q37 -39 37 -91z" /> +<glyph unicode="" horiz-adv-x="1024" d="M768 896q0 106 -75 181t-181 75t-181 -75t-75 -181t75 -181t181 -75t181 75t75 181zM1024 896q0 -109 -33 -179l-364 -774q-16 -33 -47.5 -52t-67.5 -19t-67.5 19t-46.5 52l-365 774q-33 70 -33 179q0 212 150 362t362 150t362 -150t150 -362z" /> +<glyph unicode="" d="M768 96v1088q-148 0 -273 -73t-198 -198t-73 -273t73 -273t198 -198t273 -73zM1536 640q0 -209 -103 -385.5t-279.5 -279.5t-385.5 -103t-385.5 103t-279.5 279.5t-103 385.5t103 385.5t279.5 279.5t385.5 103t385.5 -103t279.5 -279.5t103 -385.5z" /> +<glyph unicode="" horiz-adv-x="1024" d="M512 384q0 36 -20 69q-1 1 -15.5 22.5t-25.5 38t-25 44t-21 50.5q-4 16 -21 16t-21 -16q-7 -23 -21 -50.5t-25 -44t-25.5 -38t-15.5 -22.5q-20 -33 -20 -69q0 -53 37.5 -90.5t90.5 -37.5t90.5 37.5t37.5 90.5zM1024 512q0 -212 -150 -362t-362 -150t-362 150t-150 362 q0 145 81 275q6 9 62.5 90.5t101 151t99.5 178t83 201.5q9 30 34 47t51 17t51.5 -17t33.5 -47q28 -93 83 -201.5t99.5 -178t101 -151t62.5 -90.5q81 -127 81 -275z" /> +<glyph unicode="" horiz-adv-x="1792" d="M888 352l116 116l-152 152l-116 -116v-56h96v-96h56zM1328 1072q-16 16 -33 -1l-350 -350q-17 -17 -1 -33t33 1l350 350q17 17 1 33zM1408 478v-190q0 -119 -84.5 -203.5t-203.5 -84.5h-832q-119 0 -203.5 84.5t-84.5 203.5v832q0 119 84.5 203.5t203.5 84.5h832 q63 0 117 -25q15 -7 18 -23q3 -17 -9 -29l-49 -49q-14 -14 -32 -8q-23 6 -45 6h-832q-66 0 -113 -47t-47 -113v-832q0 -66 47 -113t113 -47h832q66 0 113 47t47 113v126q0 13 9 22l64 64q15 15 35 7t20 -29zM1312 1216l288 -288l-672 -672h-288v288zM1756 1084l-92 -92 l-288 288l92 92q28 28 68 28t68 -28l152 -152q28 -28 28 -68t-28 -68z" /> +<glyph unicode="" horiz-adv-x="1664" d="M1408 547v-259q0 -119 -84.5 -203.5t-203.5 -84.5h-832q-119 0 -203.5 84.5t-84.5 203.5v832q0 119 84.5 203.5t203.5 84.5h255v0q13 0 22.5 -9.5t9.5 -22.5q0 -27 -26 -32q-77 -26 -133 -60q-10 -4 -16 -4h-112q-66 0 -113 -47t-47 -113v-832q0 -66 47 -113t113 -47h832 q66 0 113 47t47 113v214q0 19 18 29q28 13 54 37q16 16 35 8q21 -9 21 -29zM1645 1043l-384 -384q-18 -19 -45 -19q-12 0 -25 5q-39 17 -39 59v192h-160q-323 0 -438 -131q-119 -137 -74 -473q3 -23 -20 -34q-8 -2 -12 -2q-16 0 -26 13q-10 14 -21 31t-39.5 68.5t-49.5 99.5 t-38.5 114t-17.5 122q0 49 3.5 91t14 90t28 88t47 81.5t68.5 74t94.5 61.5t124.5 48.5t159.5 30.5t196.5 11h160v192q0 42 39 59q13 5 25 5q26 0 45 -19l384 -384q19 -19 19 -45t-19 -45z" /> +<glyph unicode="" horiz-adv-x="1664" d="M1408 606v-318q0 -119 -84.5 -203.5t-203.5 -84.5h-832q-119 0 -203.5 84.5t-84.5 203.5v832q0 119 84.5 203.5t203.5 84.5h832q63 0 117 -25q15 -7 18 -23q3 -17 -9 -29l-49 -49q-10 -10 -23 -10q-3 0 -9 2q-23 6 -45 6h-832q-66 0 -113 -47t-47 -113v-832 q0 -66 47 -113t113 -47h832q66 0 113 47t47 113v254q0 13 9 22l64 64q10 10 23 10q6 0 12 -3q20 -8 20 -29zM1639 1095l-814 -814q-24 -24 -57 -24t-57 24l-430 430q-24 24 -24 57t24 57l110 110q24 24 57 24t57 -24l263 -263l647 647q24 24 57 24t57 -24l110 -110 q24 -24 24 -57t-24 -57z" /> +<glyph unicode="" horiz-adv-x="1792" d="M1792 640q0 -26 -19 -45l-256 -256q-19 -19 -45 -19t-45 19t-19 45v128h-384v-384h128q26 0 45 -19t19 -45t-19 -45l-256 -256q-19 -19 -45 -19t-45 19l-256 256q-19 19 -19 45t19 45t45 19h128v384h-384v-128q0 -26 -19 -45t-45 -19t-45 19l-256 256q-19 19 -19 45 t19 45l256 256q19 19 45 19t45 -19t19 -45v-128h384v384h-128q-26 0 -45 19t-19 45t19 45l256 256q19 19 45 19t45 -19l256 -256q19 -19 19 -45t-19 -45t-45 -19h-128v-384h384v128q0 26 19 45t45 19t45 -19l256 -256q19 -19 19 -45z" /> +<glyph unicode="" horiz-adv-x="1024" d="M979 1395q19 19 32 13t13 -32v-1472q0 -26 -13 -32t-32 13l-710 710q-9 9 -13 19v-678q0 -26 -19 -45t-45 -19h-128q-26 0 -45 19t-19 45v1408q0 26 19 45t45 19h128q26 0 45 -19t19 -45v-678q4 11 13 19z" /> +<glyph unicode="" horiz-adv-x="1792" d="M1747 1395q19 19 32 13t13 -32v-1472q0 -26 -13 -32t-32 13l-710 710q-9 9 -13 19v-710q0 -26 -13 -32t-32 13l-710 710q-9 9 -13 19v-678q0 -26 -19 -45t-45 -19h-128q-26 0 -45 19t-19 45v1408q0 26 19 45t45 19h128q26 0 45 -19t19 -45v-678q4 11 13 19l710 710 q19 19 32 13t13 -32v-710q4 11 13 19z" /> +<glyph unicode="" horiz-adv-x="1664" d="M1619 1395q19 19 32 13t13 -32v-1472q0 -26 -13 -32t-32 13l-710 710q-8 9 -13 19v-710q0 -26 -13 -32t-32 13l-710 710q-19 19 -19 45t19 45l710 710q19 19 32 13t13 -32v-710q5 11 13 19z" /> +<glyph unicode="" horiz-adv-x="1408" d="M1384 609l-1328 -738q-23 -13 -39.5 -3t-16.5 36v1472q0 26 16.5 36t39.5 -3l1328 -738q23 -13 23 -31t-23 -31z" /> +<glyph unicode="" d="M1536 1344v-1408q0 -26 -19 -45t-45 -19h-512q-26 0 -45 19t-19 45v1408q0 26 19 45t45 19h512q26 0 45 -19t19 -45zM640 1344v-1408q0 -26 -19 -45t-45 -19h-512q-26 0 -45 19t-19 45v1408q0 26 19 45t45 19h512q26 0 45 -19t19 -45z" /> +<glyph unicode="" d="M1536 1344v-1408q0 -26 -19 -45t-45 -19h-1408q-26 0 -45 19t-19 45v1408q0 26 19 45t45 19h1408q26 0 45 -19t19 -45z" /> +<glyph unicode="" horiz-adv-x="1664" d="M45 -115q-19 -19 -32 -13t-13 32v1472q0 26 13 32t32 -13l710 -710q8 -8 13 -19v710q0 26 13 32t32 -13l710 -710q19 -19 19 -45t-19 -45l-710 -710q-19 -19 -32 -13t-13 32v710q-5 -10 -13 -19z" /> +<glyph unicode="" horiz-adv-x="1792" d="M45 -115q-19 -19 -32 -13t-13 32v1472q0 26 13 32t32 -13l710 -710q8 -8 13 -19v710q0 26 13 32t32 -13l710 -710q8 -8 13 -19v678q0 26 19 45t45 19h128q26 0 45 -19t19 -45v-1408q0 -26 -19 -45t-45 -19h-128q-26 0 -45 19t-19 45v678q-5 -10 -13 -19l-710 -710 q-19 -19 -32 -13t-13 32v710q-5 -10 -13 -19z" /> +<glyph unicode="" horiz-adv-x="1024" d="M45 -115q-19 -19 -32 -13t-13 32v1472q0 26 13 32t32 -13l710 -710q8 -8 13 -19v678q0 26 19 45t45 19h128q26 0 45 -19t19 -45v-1408q0 -26 -19 -45t-45 -19h-128q-26 0 -45 19t-19 45v678q-5 -10 -13 -19z" /> +<glyph unicode="" horiz-adv-x="1538" d="M14 557l710 710q19 19 45 19t45 -19l710 -710q19 -19 13 -32t-32 -13h-1472q-26 0 -32 13t13 32zM1473 0h-1408q-26 0 -45 19t-19 45v256q0 26 19 45t45 19h1408q26 0 45 -19t19 -45v-256q0 -26 -19 -45t-45 -19z" /> +<glyph unicode="" horiz-adv-x="1152" d="M742 -37l-652 651q-37 37 -37 90.5t37 90.5l652 651q37 37 90.5 37t90.5 -37l75 -75q37 -37 37 -90.5t-37 -90.5l-486 -486l486 -485q37 -38 37 -91t-37 -90l-75 -75q-37 -37 -90.5 -37t-90.5 37z" /> +<glyph unicode="" horiz-adv-x="1152" d="M1099 704q0 -52 -37 -91l-652 -651q-37 -37 -90 -37t-90 37l-76 75q-37 39 -37 91q0 53 37 90l486 486l-486 485q-37 39 -37 91q0 53 37 90l76 75q36 38 90 38t90 -38l652 -651q37 -37 37 -90z" /> +<glyph unicode="" d="M1216 576v128q0 26 -19 45t-45 19h-256v256q0 26 -19 45t-45 19h-128q-26 0 -45 -19t-19 -45v-256h-256q-26 0 -45 -19t-19 -45v-128q0 -26 19 -45t45 -19h256v-256q0 -26 19 -45t45 -19h128q26 0 45 19t19 45v256h256q26 0 45 19t19 45zM1536 640q0 -209 -103 -385.5 t-279.5 -279.5t-385.5 -103t-385.5 103t-279.5 279.5t-103 385.5t103 385.5t279.5 279.5t385.5 103t385.5 -103t279.5 -279.5t103 -385.5z" /> +<glyph unicode="" d="M1216 576v128q0 26 -19 45t-45 19h-768q-26 0 -45 -19t-19 -45v-128q0 -26 19 -45t45 -19h768q26 0 45 19t19 45zM1536 640q0 -209 -103 -385.5t-279.5 -279.5t-385.5 -103t-385.5 103t-279.5 279.5t-103 385.5t103 385.5t279.5 279.5t385.5 103t385.5 -103t279.5 -279.5 t103 -385.5z" /> +<glyph unicode="" d="M1149 414q0 26 -19 45l-181 181l181 181q19 19 19 45q0 27 -19 46l-90 90q-19 19 -46 19q-26 0 -45 -19l-181 -181l-181 181q-19 19 -45 19q-27 0 -46 -19l-90 -90q-19 -19 -19 -46q0 -26 19 -45l181 -181l-181 -181q-19 -19 -19 -45q0 -27 19 -46l90 -90q19 -19 46 -19 q26 0 45 19l181 181l181 -181q19 -19 45 -19q27 0 46 19l90 90q19 19 19 46zM1536 640q0 -209 -103 -385.5t-279.5 -279.5t-385.5 -103t-385.5 103t-279.5 279.5t-103 385.5t103 385.5t279.5 279.5t385.5 103t385.5 -103t279.5 -279.5t103 -385.5z" /> +<glyph unicode="" d="M1284 802q0 28 -18 46l-91 90q-19 19 -45 19t-45 -19l-408 -407l-226 226q-19 19 -45 19t-45 -19l-91 -90q-18 -18 -18 -46q0 -27 18 -45l362 -362q19 -19 45 -19q27 0 46 19l543 543q18 18 18 45zM1536 640q0 -209 -103 -385.5t-279.5 -279.5t-385.5 -103t-385.5 103 t-279.5 279.5t-103 385.5t103 385.5t279.5 279.5t385.5 103t385.5 -103t279.5 -279.5t103 -385.5z" /> +<glyph unicode="" d="M896 160v192q0 14 -9 23t-23 9h-192q-14 0 -23 -9t-9 -23v-192q0 -14 9 -23t23 -9h192q14 0 23 9t9 23zM1152 832q0 88 -55.5 163t-138.5 116t-170 41q-243 0 -371 -213q-15 -24 8 -42l132 -100q7 -6 19 -6q16 0 25 12q53 68 86 92q34 24 86 24q48 0 85.5 -26t37.5 -59 q0 -38 -20 -61t-68 -45q-63 -28 -115.5 -86.5t-52.5 -125.5v-36q0 -14 9 -23t23 -9h192q14 0 23 9t9 23q0 19 21.5 49.5t54.5 49.5q32 18 49 28.5t46 35t44.5 48t28 60.5t12.5 81zM1536 640q0 -209 -103 -385.5t-279.5 -279.5t-385.5 -103t-385.5 103t-279.5 279.5 t-103 385.5t103 385.5t279.5 279.5t385.5 103t385.5 -103t279.5 -279.5t103 -385.5z" /> +<glyph unicode="" d="M1024 160v160q0 14 -9 23t-23 9h-96v512q0 14 -9 23t-23 9h-320q-14 0 -23 -9t-9 -23v-160q0 -14 9 -23t23 -9h96v-320h-96q-14 0 -23 -9t-9 -23v-160q0 -14 9 -23t23 -9h448q14 0 23 9t9 23zM896 1056v160q0 14 -9 23t-23 9h-192q-14 0 -23 -9t-9 -23v-160q0 -14 9 -23 t23 -9h192q14 0 23 9t9 23zM1536 640q0 -209 -103 -385.5t-279.5 -279.5t-385.5 -103t-385.5 103t-279.5 279.5t-103 385.5t103 385.5t279.5 279.5t385.5 103t385.5 -103t279.5 -279.5t103 -385.5z" /> +<glyph unicode="" d="M1197 512h-109q-26 0 -45 19t-19 45v128q0 26 19 45t45 19h109q-32 108 -112.5 188.5t-188.5 112.5v-109q0 -26 -19 -45t-45 -19h-128q-26 0 -45 19t-19 45v109q-108 -32 -188.5 -112.5t-112.5 -188.5h109q26 0 45 -19t19 -45v-128q0 -26 -19 -45t-45 -19h-109 q32 -108 112.5 -188.5t188.5 -112.5v109q0 26 19 45t45 19h128q26 0 45 -19t19 -45v-109q108 32 188.5 112.5t112.5 188.5zM1536 704v-128q0 -26 -19 -45t-45 -19h-143q-37 -161 -154.5 -278.5t-278.5 -154.5v-143q0 -26 -19 -45t-45 -19h-128q-26 0 -45 19t-19 45v143 q-161 37 -278.5 154.5t-154.5 278.5h-143q-26 0 -45 19t-19 45v128q0 26 19 45t45 19h143q37 161 154.5 278.5t278.5 154.5v143q0 26 19 45t45 19h128q26 0 45 -19t19 -45v-143q161 -37 278.5 -154.5t154.5 -278.5h143q26 0 45 -19t19 -45z" /> +<glyph unicode="" d="M1097 457l-146 -146q-10 -10 -23 -10t-23 10l-137 137l-137 -137q-10 -10 -23 -10t-23 10l-146 146q-10 10 -10 23t10 23l137 137l-137 137q-10 10 -10 23t10 23l146 146q10 10 23 10t23 -10l137 -137l137 137q10 10 23 10t23 -10l146 -146q10 -10 10 -23t-10 -23 l-137 -137l137 -137q10 -10 10 -23t-10 -23zM1312 640q0 148 -73 273t-198 198t-273 73t-273 -73t-198 -198t-73 -273t73 -273t198 -198t273 -73t273 73t198 198t73 273zM1536 640q0 -209 -103 -385.5t-279.5 -279.5t-385.5 -103t-385.5 103t-279.5 279.5t-103 385.5 t103 385.5t279.5 279.5t385.5 103t385.5 -103t279.5 -279.5t103 -385.5z" /> +<glyph unicode="" d="M1171 723l-422 -422q-19 -19 -45 -19t-45 19l-294 294q-19 19 -19 45t19 45l102 102q19 19 45 19t45 -19l147 -147l275 275q19 19 45 19t45 -19l102 -102q19 -19 19 -45t-19 -45zM1312 640q0 148 -73 273t-198 198t-273 73t-273 -73t-198 -198t-73 -273t73 -273t198 -198 t273 -73t273 73t198 198t73 273zM1536 640q0 -209 -103 -385.5t-279.5 -279.5t-385.5 -103t-385.5 103t-279.5 279.5t-103 385.5t103 385.5t279.5 279.5t385.5 103t385.5 -103t279.5 -279.5t103 -385.5z" /> +<glyph unicode="" d="M1312 643q0 161 -87 295l-754 -753q137 -89 297 -89q111 0 211.5 43.5t173.5 116.5t116 174.5t43 212.5zM313 344l755 754q-135 91 -300 91q-148 0 -273 -73t-198 -199t-73 -274q0 -162 89 -299zM1536 643q0 -157 -61 -300t-163.5 -246t-245 -164t-298.5 -61t-298.5 61 t-245 164t-163.5 246t-61 300t61 299.5t163.5 245.5t245 164t298.5 61t298.5 -61t245 -164t163.5 -245.5t61 -299.5z" /> +<glyph unicode="" d="M1536 640v-128q0 -53 -32.5 -90.5t-84.5 -37.5h-704l293 -294q38 -36 38 -90t-38 -90l-75 -76q-37 -37 -90 -37q-52 0 -91 37l-651 652q-37 37 -37 90q0 52 37 91l651 650q38 38 91 38q52 0 90 -38l75 -74q38 -38 38 -91t-38 -91l-293 -293h704q52 0 84.5 -37.5 t32.5 -90.5z" /> +<glyph unicode="" d="M1472 576q0 -54 -37 -91l-651 -651q-39 -37 -91 -37q-51 0 -90 37l-75 75q-38 38 -38 91t38 91l293 293h-704q-52 0 -84.5 37.5t-32.5 90.5v128q0 53 32.5 90.5t84.5 37.5h704l-293 294q-38 36 -38 90t38 90l75 75q38 38 90 38q53 0 91 -38l651 -651q37 -35 37 -90z" /> +<glyph unicode="" horiz-adv-x="1664" d="M1611 565q0 -51 -37 -90l-75 -75q-38 -38 -91 -38q-54 0 -90 38l-294 293v-704q0 -52 -37.5 -84.5t-90.5 -32.5h-128q-53 0 -90.5 32.5t-37.5 84.5v704l-294 -293q-36 -38 -90 -38t-90 38l-75 75q-38 38 -38 90q0 53 38 91l651 651q35 37 90 37q54 0 91 -37l651 -651 q37 -39 37 -91z" /> +<glyph unicode="" horiz-adv-x="1664" d="M1611 704q0 -53 -37 -90l-651 -652q-39 -37 -91 -37q-53 0 -90 37l-651 652q-38 36 -38 90q0 53 38 91l74 75q39 37 91 37q53 0 90 -37l294 -294v704q0 52 38 90t90 38h128q52 0 90 -38t38 -90v-704l294 294q37 37 90 37q52 0 91 -37l75 -75q37 -39 37 -91z" /> +<glyph unicode="" horiz-adv-x="1792" d="M1792 896q0 -26 -19 -45l-512 -512q-19 -19 -45 -19t-45 19t-19 45v256h-224q-98 0 -175.5 -6t-154 -21.5t-133 -42.5t-105.5 -69.5t-80 -101t-48.5 -138.5t-17.5 -181q0 -55 5 -123q0 -6 2.5 -23.5t2.5 -26.5q0 -15 -8.5 -25t-23.5 -10q-16 0 -28 17q-7 9 -13 22 t-13.5 30t-10.5 24q-127 285 -127 451q0 199 53 333q162 403 875 403h224v256q0 26 19 45t45 19t45 -19l512 -512q19 -19 19 -45z" /> +<glyph unicode="" d="M755 480q0 -13 -10 -23l-332 -332l144 -144q19 -19 19 -45t-19 -45t-45 -19h-448q-26 0 -45 19t-19 45v448q0 26 19 45t45 19t45 -19l144 -144l332 332q10 10 23 10t23 -10l114 -114q10 -10 10 -23zM1536 1344v-448q0 -26 -19 -45t-45 -19t-45 19l-144 144l-332 -332 q-10 -10 -23 -10t-23 10l-114 114q-10 10 -10 23t10 23l332 332l-144 144q-19 19 -19 45t19 45t45 19h448q26 0 45 -19t19 -45z" /> +<glyph unicode="" d="M768 576v-448q0 -26 -19 -45t-45 -19t-45 19l-144 144l-332 -332q-10 -10 -23 -10t-23 10l-114 114q-10 10 -10 23t10 23l332 332l-144 144q-19 19 -19 45t19 45t45 19h448q26 0 45 -19t19 -45zM1523 1248q0 -13 -10 -23l-332 -332l144 -144q19 -19 19 -45t-19 -45 t-45 -19h-448q-26 0 -45 19t-19 45v448q0 26 19 45t45 19t45 -19l144 -144l332 332q10 10 23 10t23 -10l114 -114q10 -10 10 -23z" /> +<glyph unicode="" horiz-adv-x="1408" d="M1408 800v-192q0 -40 -28 -68t-68 -28h-416v-416q0 -40 -28 -68t-68 -28h-192q-40 0 -68 28t-28 68v416h-416q-40 0 -68 28t-28 68v192q0 40 28 68t68 28h416v416q0 40 28 68t68 28h192q40 0 68 -28t28 -68v-416h416q40 0 68 -28t28 -68z" /> +<glyph unicode="" horiz-adv-x="1408" d="M1408 800v-192q0 -40 -28 -68t-68 -28h-1216q-40 0 -68 28t-28 68v192q0 40 28 68t68 28h1216q40 0 68 -28t28 -68z" /> +<glyph unicode="" horiz-adv-x="1664" d="M1482 486q46 -26 59.5 -77.5t-12.5 -97.5l-64 -110q-26 -46 -77.5 -59.5t-97.5 12.5l-266 153v-307q0 -52 -38 -90t-90 -38h-128q-52 0 -90 38t-38 90v307l-266 -153q-46 -26 -97.5 -12.5t-77.5 59.5l-64 110q-26 46 -12.5 97.5t59.5 77.5l266 154l-266 154 q-46 26 -59.5 77.5t12.5 97.5l64 110q26 46 77.5 59.5t97.5 -12.5l266 -153v307q0 52 38 90t90 38h128q52 0 90 -38t38 -90v-307l266 153q46 26 97.5 12.5t77.5 -59.5l64 -110q26 -46 12.5 -97.5t-59.5 -77.5l-266 -154z" /> +<glyph unicode="" d="M768 1408q209 0 385.5 -103t279.5 -279.5t103 -385.5t-103 -385.5t-279.5 -279.5t-385.5 -103t-385.5 103t-279.5 279.5t-103 385.5t103 385.5t279.5 279.5t385.5 103zM896 161v190q0 14 -9 23.5t-22 9.5h-192q-13 0 -23 -10t-10 -23v-190q0 -13 10 -23t23 -10h192 q13 0 22 9.5t9 23.5zM894 505l18 621q0 12 -10 18q-10 8 -24 8h-220q-14 0 -24 -8q-10 -6 -10 -18l17 -621q0 -10 10 -17.5t24 -7.5h185q14 0 23.5 7.5t10.5 17.5z" /> +<glyph unicode="" d="M928 180v56v468v192h-320v-192v-468v-56q0 -25 18 -38.5t46 -13.5h192q28 0 46 13.5t18 38.5zM472 1024h195l-126 161q-26 31 -69 31q-40 0 -68 -28t-28 -68t28 -68t68 -28zM1160 1120q0 40 -28 68t-68 28q-43 0 -69 -31l-125 -161h194q40 0 68 28t28 68zM1536 864v-320 q0 -14 -9 -23t-23 -9h-96v-416q0 -40 -28 -68t-68 -28h-1088q-40 0 -68 28t-28 68v416h-96q-14 0 -23 9t-9 23v320q0 14 9 23t23 9h440q-93 0 -158.5 65.5t-65.5 158.5t65.5 158.5t158.5 65.5q107 0 168 -77l128 -165l128 165q61 77 168 77q93 0 158.5 -65.5t65.5 -158.5 t-65.5 -158.5t-158.5 -65.5h440q14 0 23 -9t9 -23z" /> +<glyph unicode="" horiz-adv-x="1792" d="M1280 832q0 26 -19 45t-45 19q-172 0 -318 -49.5t-259.5 -134t-235.5 -219.5q-19 -21 -19 -45q0 -26 19 -45t45 -19q24 0 45 19q27 24 74 71t67 66q137 124 268.5 176t313.5 52q26 0 45 19t19 45zM1792 1030q0 -95 -20 -193q-46 -224 -184.5 -383t-357.5 -268 q-214 -108 -438 -108q-148 0 -286 47q-15 5 -88 42t-96 37q-16 0 -39.5 -32t-45 -70t-52.5 -70t-60 -32q-30 0 -51 11t-31 24t-27 42q-2 4 -6 11t-5.5 10t-3 9.5t-1.5 13.5q0 35 31 73.5t68 65.5t68 56t31 48q0 4 -14 38t-16 44q-9 51 -9 104q0 115 43.5 220t119 184.5 t170.5 139t204 95.5q55 18 145 25.5t179.5 9t178.5 6t163.5 24t113.5 56.5l29.5 29.5t29.5 28t27 20t36.5 16t43.5 4.5q39 0 70.5 -46t47.5 -112t24 -124t8 -96z" /> +<glyph unicode="" horiz-adv-x="1408" d="M1408 -160v-64q0 -13 -9.5 -22.5t-22.5 -9.5h-1344q-13 0 -22.5 9.5t-9.5 22.5v64q0 13 9.5 22.5t22.5 9.5h1344q13 0 22.5 -9.5t9.5 -22.5zM1152 896q0 -78 -24.5 -144t-64 -112.5t-87.5 -88t-96 -77.5t-87.5 -72t-64 -81.5t-24.5 -96.5q0 -96 67 -224l-4 1l1 -1 q-90 41 -160 83t-138.5 100t-113.5 122.5t-72.5 150.5t-27.5 184q0 78 24.5 144t64 112.5t87.5 88t96 77.5t87.5 72t64 81.5t24.5 96.5q0 94 -66 224l3 -1l-1 1q90 -41 160 -83t138.5 -100t113.5 -122.5t72.5 -150.5t27.5 -184z" /> +<glyph unicode="" horiz-adv-x="1792" d="M1664 576q-152 236 -381 353q61 -104 61 -225q0 -185 -131.5 -316.5t-316.5 -131.5t-316.5 131.5t-131.5 316.5q0 121 61 225q-229 -117 -381 -353q133 -205 333.5 -326.5t434.5 -121.5t434.5 121.5t333.5 326.5zM944 960q0 20 -14 34t-34 14q-125 0 -214.5 -89.5 t-89.5 -214.5q0 -20 14 -34t34 -14t34 14t14 34q0 86 61 147t147 61q20 0 34 14t14 34zM1792 576q0 -34 -20 -69q-140 -230 -376.5 -368.5t-499.5 -138.5t-499.5 139t-376.5 368q-20 35 -20 69t20 69q140 229 376.5 368t499.5 139t499.5 -139t376.5 -368q20 -35 20 -69z" /> +<glyph unicode="" horiz-adv-x="1792" d="M555 201l78 141q-87 63 -136 159t-49 203q0 121 61 225q-229 -117 -381 -353q167 -258 427 -375zM944 960q0 20 -14 34t-34 14q-125 0 -214.5 -89.5t-89.5 -214.5q0 -20 14 -34t34 -14t34 14t14 34q0 86 61 147t147 61q20 0 34 14t14 34zM1307 1151q0 -7 -1 -9 q-105 -188 -315 -566t-316 -567l-49 -89q-10 -16 -28 -16q-12 0 -134 70q-16 10 -16 28q0 12 44 87q-143 65 -263.5 173t-208.5 245q-20 31 -20 69t20 69q153 235 380 371t496 136q89 0 180 -17l54 97q10 16 28 16q5 0 18 -6t31 -15.5t33 -18.5t31.5 -18.5t19.5 -11.5 q16 -10 16 -27zM1344 704q0 -139 -79 -253.5t-209 -164.5l280 502q8 -45 8 -84zM1792 576q0 -35 -20 -69q-39 -64 -109 -145q-150 -172 -347.5 -267t-419.5 -95l74 132q212 18 392.5 137t301.5 307q-115 179 -282 294l63 112q95 -64 182.5 -153t144.5 -184q20 -34 20 -69z " /> +<glyph unicode="" horiz-adv-x="1792" d="M1024 161v190q0 14 -9.5 23.5t-22.5 9.5h-192q-13 0 -22.5 -9.5t-9.5 -23.5v-190q0 -14 9.5 -23.5t22.5 -9.5h192q13 0 22.5 9.5t9.5 23.5zM1022 535l18 459q0 12 -10 19q-13 11 -24 11h-220q-11 0 -24 -11q-10 -7 -10 -21l17 -457q0 -10 10 -16.5t24 -6.5h185 q14 0 23.5 6.5t10.5 16.5zM1008 1469l768 -1408q35 -63 -2 -126q-17 -29 -46.5 -46t-63.5 -17h-1536q-34 0 -63.5 17t-46.5 46q-37 63 -2 126l768 1408q17 31 47 49t65 18t65 -18t47 -49z" /> +<glyph unicode="" horiz-adv-x="1408" d="M1376 1376q44 -52 12 -148t-108 -172l-161 -161l160 -696q5 -19 -12 -33l-128 -96q-7 -6 -19 -6q-4 0 -7 1q-15 3 -21 16l-279 508l-259 -259l53 -194q5 -17 -8 -31l-96 -96q-9 -9 -23 -9h-2q-15 2 -24 13l-189 252l-252 189q-11 7 -13 23q-1 13 9 25l96 97q9 9 23 9 q6 0 8 -1l194 -53l259 259l-508 279q-14 8 -17 24q-2 16 9 27l128 128q14 13 30 8l665 -159l160 160q76 76 172 108t148 -12z" /> +<glyph unicode="" horiz-adv-x="1664" d="M128 -128h288v288h-288v-288zM480 -128h320v288h-320v-288zM128 224h288v320h-288v-320zM480 224h320v320h-320v-320zM128 608h288v288h-288v-288zM864 -128h320v288h-320v-288zM480 608h320v288h-320v-288zM1248 -128h288v288h-288v-288zM864 224h320v320h-320v-320z M512 1088v288q0 13 -9.5 22.5t-22.5 9.5h-64q-13 0 -22.5 -9.5t-9.5 -22.5v-288q0 -13 9.5 -22.5t22.5 -9.5h64q13 0 22.5 9.5t9.5 22.5zM1248 224h288v320h-288v-320zM864 608h320v288h-320v-288zM1248 608h288v288h-288v-288zM1280 1088v288q0 13 -9.5 22.5t-22.5 9.5h-64 q-13 0 -22.5 -9.5t-9.5 -22.5v-288q0 -13 9.5 -22.5t22.5 -9.5h64q13 0 22.5 9.5t9.5 22.5zM1664 1152v-1280q0 -52 -38 -90t-90 -38h-1408q-52 0 -90 38t-38 90v1280q0 52 38 90t90 38h128v96q0 66 47 113t113 47h64q66 0 113 -47t47 -113v-96h384v96q0 66 47 113t113 47 h64q66 0 113 -47t47 -113v-96h128q52 0 90 -38t38 -90z" /> +<glyph unicode="" horiz-adv-x="1792" d="M666 1055q-60 -92 -137 -273q-22 45 -37 72.5t-40.5 63.5t-51 56.5t-63 35t-81.5 14.5h-224q-14 0 -23 9t-9 23v192q0 14 9 23t23 9h224q250 0 410 -225zM1792 256q0 -14 -9 -23l-320 -320q-9 -9 -23 -9q-13 0 -22.5 9.5t-9.5 22.5v192q-32 0 -85 -0.5t-81 -1t-73 1 t-71 5t-64 10.5t-63 18.5t-58 28.5t-59 40t-55 53.5t-56 69.5q59 93 136 273q22 -45 37 -72.5t40.5 -63.5t51 -56.5t63 -35t81.5 -14.5h256v192q0 14 9 23t23 9q12 0 24 -10l319 -319q9 -9 9 -23zM1792 1152q0 -14 -9 -23l-320 -320q-9 -9 -23 -9q-13 0 -22.5 9.5t-9.5 22.5 v192h-256q-48 0 -87 -15t-69 -45t-51 -61.5t-45 -77.5q-32 -62 -78 -171q-29 -66 -49.5 -111t-54 -105t-64 -100t-74 -83t-90 -68.5t-106.5 -42t-128 -16.5h-224q-14 0 -23 9t-9 23v192q0 14 9 23t23 9h224q48 0 87 15t69 45t51 61.5t45 77.5q32 62 78 171q29 66 49.5 111 t54 105t64 100t74 83t90 68.5t106.5 42t128 16.5h256v192q0 14 9 23t23 9q12 0 24 -10l319 -319q9 -9 9 -23z" /> +<glyph unicode="" horiz-adv-x="1792" d="M1792 640q0 -174 -120 -321.5t-326 -233t-450 -85.5q-70 0 -145 8q-198 -175 -460 -242q-49 -14 -114 -22q-17 -2 -30.5 9t-17.5 29v1q-3 4 -0.5 12t2 10t4.5 9.5l6 9t7 8.5t8 9q7 8 31 34.5t34.5 38t31 39.5t32.5 51t27 59t26 76q-157 89 -247.5 220t-90.5 281 q0 130 71 248.5t191 204.5t286 136.5t348 50.5q244 0 450 -85.5t326 -233t120 -321.5z" /> +<glyph unicode="" d="M1536 704v-128q0 -201 -98.5 -362t-274 -251.5t-395.5 -90.5t-395.5 90.5t-274 251.5t-98.5 362v128q0 26 19 45t45 19h384q26 0 45 -19t19 -45v-128q0 -52 23.5 -90t53.5 -57t71 -30t64 -13t44 -2t44 2t64 13t71 30t53.5 57t23.5 90v128q0 26 19 45t45 19h384 q26 0 45 -19t19 -45zM512 1344v-384q0 -26 -19 -45t-45 -19h-384q-26 0 -45 19t-19 45v384q0 26 19 45t45 19h384q26 0 45 -19t19 -45zM1536 1344v-384q0 -26 -19 -45t-45 -19h-384q-26 0 -45 19t-19 45v384q0 26 19 45t45 19h384q26 0 45 -19t19 -45z" /> +<glyph unicode="" horiz-adv-x="1664" d="M1611 320q0 -53 -37 -90l-75 -75q-38 -38 -91 -38q-54 0 -90 38l-486 485l-486 -485q-36 -38 -90 -38t-90 38l-75 75q-38 36 -38 90q0 53 38 91l651 651q37 37 90 37q52 0 91 -37l650 -651q38 -38 38 -91z" /> +<glyph unicode="" horiz-adv-x="1664" d="M1611 832q0 -53 -37 -90l-651 -651q-38 -38 -91 -38q-54 0 -90 38l-651 651q-38 36 -38 90q0 53 38 91l74 75q39 37 91 37q53 0 90 -37l486 -486l486 486q37 37 90 37q52 0 91 -37l75 -75q37 -39 37 -91z" /> +<glyph unicode="" horiz-adv-x="1920" d="M1280 32q0 -13 -9.5 -22.5t-22.5 -9.5h-960q-8 0 -13.5 2t-9 7t-5.5 8t-3 11.5t-1 11.5v13v11v160v416h-192q-26 0 -45 19t-19 45q0 24 15 41l320 384q19 22 49 22t49 -22l320 -384q15 -17 15 -41q0 -26 -19 -45t-45 -19h-192v-384h576q16 0 25 -11l160 -192q7 -11 7 -21 zM1920 448q0 -24 -15 -41l-320 -384q-20 -23 -49 -23t-49 23l-320 384q-15 17 -15 41q0 26 19 45t45 19h192v384h-576q-16 0 -25 12l-160 192q-7 9 -7 20q0 13 9.5 22.5t22.5 9.5h960q8 0 13.5 -2t9 -7t5.5 -8t3 -11.5t1 -11.5v-13v-11v-160v-416h192q26 0 45 -19t19 -45z " /> +<glyph unicode="" horiz-adv-x="1664" d="M640 0q0 -53 -37.5 -90.5t-90.5 -37.5t-90.5 37.5t-37.5 90.5t37.5 90.5t90.5 37.5t90.5 -37.5t37.5 -90.5zM1536 0q0 -53 -37.5 -90.5t-90.5 -37.5t-90.5 37.5t-37.5 90.5t37.5 90.5t90.5 37.5t90.5 -37.5t37.5 -90.5zM1664 1088v-512q0 -24 -16 -42.5t-41 -21.5 l-1044 -122q1 -7 4.5 -21.5t6 -26.5t2.5 -22q0 -16 -24 -64h920q26 0 45 -19t19 -45t-19 -45t-45 -19h-1024q-26 0 -45 19t-19 45q0 14 11 39.5t29.5 59.5t20.5 38l-177 823h-204q-26 0 -45 19t-19 45t19 45t45 19h256q16 0 28.5 -6.5t20 -15.5t13 -24.5t7.5 -26.5 t5.5 -29.5t4.5 -25.5h1201q26 0 45 -19t19 -45z" /> +<glyph unicode="" horiz-adv-x="1664" d="M1664 928v-704q0 -92 -66 -158t-158 -66h-1216q-92 0 -158 66t-66 158v960q0 92 66 158t158 66h320q92 0 158 -66t66 -158v-32h672q92 0 158 -66t66 -158z" /> +<glyph unicode="" horiz-adv-x="1920" d="M1879 584q0 -31 -31 -66l-336 -396q-43 -51 -120.5 -86.5t-143.5 -35.5h-1088q-34 0 -60.5 13t-26.5 43q0 31 31 66l336 396q43 51 120.5 86.5t143.5 35.5h1088q34 0 60.5 -13t26.5 -43zM1536 928v-160h-832q-94 0 -197 -47.5t-164 -119.5l-337 -396l-5 -6q0 4 -0.5 12.5 t-0.5 12.5v960q0 92 66 158t158 66h320q92 0 158 -66t66 -158v-32h544q92 0 158 -66t66 -158z" /> +<glyph unicode="" horiz-adv-x="768" d="M704 1216q0 -26 -19 -45t-45 -19h-128v-1024h128q26 0 45 -19t19 -45t-19 -45l-256 -256q-19 -19 -45 -19t-45 19l-256 256q-19 19 -19 45t19 45t45 19h128v1024h-128q-26 0 -45 19t-19 45t19 45l256 256q19 19 45 19t45 -19l256 -256q19 -19 19 -45z" /> +<glyph unicode="" horiz-adv-x="1792" d="M1792 640q0 -26 -19 -45l-256 -256q-19 -19 -45 -19t-45 19t-19 45v128h-1024v-128q0 -26 -19 -45t-45 -19t-45 19l-256 256q-19 19 -19 45t19 45l256 256q19 19 45 19t45 -19t19 -45v-128h1024v128q0 26 19 45t45 19t45 -19l256 -256q19 -19 19 -45z" /> +<glyph unicode="" horiz-adv-x="1920" d="M512 512v-384h-256v384h256zM896 1024v-896h-256v896h256zM1280 768v-640h-256v640h256zM1664 1152v-1024h-256v1024h256zM1792 32v1216q0 13 -9.5 22.5t-22.5 9.5h-1600q-13 0 -22.5 -9.5t-9.5 -22.5v-1216q0 -13 9.5 -22.5t22.5 -9.5h1600q13 0 22.5 9.5t9.5 22.5z M1920 1248v-1216q0 -66 -47 -113t-113 -47h-1600q-66 0 -113 47t-47 113v1216q0 66 47 113t113 47h1600q66 0 113 -47t47 -113z" /> +<glyph unicode="" d="M1280 926q-56 -25 -121 -34q68 40 93 117q-65 -38 -134 -51q-61 66 -153 66q-87 0 -148.5 -61.5t-61.5 -148.5q0 -29 5 -48q-129 7 -242 65t-192 155q-29 -50 -29 -106q0 -114 91 -175q-47 1 -100 26v-2q0 -75 50 -133.5t123 -72.5q-29 -8 -51 -8q-13 0 -39 4 q21 -63 74.5 -104t121.5 -42q-116 -90 -261 -90q-26 0 -50 3q148 -94 322 -94q112 0 210 35.5t168 95t120.5 137t75 162t24.5 168.5q0 18 -1 27q63 45 105 109zM1536 1120v-960q0 -119 -84.5 -203.5t-203.5 -84.5h-960q-119 0 -203.5 84.5t-84.5 203.5v960q0 119 84.5 203.5 t203.5 84.5h960q119 0 203.5 -84.5t84.5 -203.5z" /> +<glyph unicode="" d="M1307 618l23 219h-198v109q0 49 15.5 68.5t71.5 19.5h110v219h-175q-152 0 -218 -72t-66 -213v-131h-131v-219h131v-635h262v635h175zM1536 1120v-960q0 -119 -84.5 -203.5t-203.5 -84.5h-960q-119 0 -203.5 84.5t-84.5 203.5v960q0 119 84.5 203.5t203.5 84.5h960 q119 0 203.5 -84.5t84.5 -203.5z" /> +<glyph unicode="" horiz-adv-x="1792" d="M928 704q0 14 -9 23t-23 9q-66 0 -113 -47t-47 -113q0 -14 9 -23t23 -9t23 9t9 23q0 40 28 68t68 28q14 0 23 9t9 23zM1152 574q0 -106 -75 -181t-181 -75t-181 75t-75 181t75 181t181 75t181 -75t75 -181zM128 0h1536v128h-1536v-128zM1280 574q0 159 -112.5 271.5 t-271.5 112.5t-271.5 -112.5t-112.5 -271.5t112.5 -271.5t271.5 -112.5t271.5 112.5t112.5 271.5zM256 1216h384v128h-384v-128zM128 1024h1536v118v138h-828l-64 -128h-644v-128zM1792 1280v-1280q0 -53 -37.5 -90.5t-90.5 -37.5h-1536q-53 0 -90.5 37.5t-37.5 90.5v1280 q0 53 37.5 90.5t90.5 37.5h1536q53 0 90.5 -37.5t37.5 -90.5z" /> +<glyph unicode="" horiz-adv-x="1792" d="M832 1024q0 80 -56 136t-136 56t-136 -56t-56 -136q0 -42 19 -83q-41 19 -83 19q-80 0 -136 -56t-56 -136t56 -136t136 -56t136 56t56 136q0 42 -19 83q41 -19 83 -19q80 0 136 56t56 136zM1683 320q0 -17 -49 -66t-66 -49q-9 0 -28.5 16t-36.5 33t-38.5 40t-24.5 26 l-96 -96l220 -220q28 -28 28 -68q0 -42 -39 -81t-81 -39q-40 0 -68 28l-671 671q-176 -131 -365 -131q-163 0 -265.5 102.5t-102.5 265.5q0 160 95 313t248 248t313 95q163 0 265.5 -102.5t102.5 -265.5q0 -189 -131 -365l355 -355l96 96q-3 3 -26 24.5t-40 38.5t-33 36.5 t-16 28.5q0 17 49 66t66 49q13 0 23 -10q6 -6 46 -44.5t82 -79.5t86.5 -86t73 -78t28.5 -41z" /> +<glyph unicode="" horiz-adv-x="1920" d="M896 640q0 106 -75 181t-181 75t-181 -75t-75 -181t75 -181t181 -75t181 75t75 181zM1664 128q0 52 -38 90t-90 38t-90 -38t-38 -90q0 -53 37.5 -90.5t90.5 -37.5t90.5 37.5t37.5 90.5zM1664 1152q0 52 -38 90t-90 38t-90 -38t-38 -90q0 -53 37.5 -90.5t90.5 -37.5 t90.5 37.5t37.5 90.5zM1280 731v-185q0 -10 -7 -19.5t-16 -10.5l-155 -24q-11 -35 -32 -76q34 -48 90 -115q7 -10 7 -20q0 -12 -7 -19q-23 -30 -82.5 -89.5t-78.5 -59.5q-11 0 -21 7l-115 90q-37 -19 -77 -31q-11 -108 -23 -155q-7 -24 -30 -24h-186q-11 0 -20 7.5t-10 17.5 l-23 153q-34 10 -75 31l-118 -89q-7 -7 -20 -7q-11 0 -21 8q-144 133 -144 160q0 9 7 19q10 14 41 53t47 61q-23 44 -35 82l-152 24q-10 1 -17 9.5t-7 19.5v185q0 10 7 19.5t16 10.5l155 24q11 35 32 76q-34 48 -90 115q-7 11 -7 20q0 12 7 20q22 30 82 89t79 59q11 0 21 -7 l115 -90q34 18 77 32q11 108 23 154q7 24 30 24h186q11 0 20 -7.5t10 -17.5l23 -153q34 -10 75 -31l118 89q8 7 20 7q11 0 21 -8q144 -133 144 -160q0 -9 -7 -19q-12 -16 -42 -54t-45 -60q23 -48 34 -82l152 -23q10 -2 17 -10.5t7 -19.5zM1920 198v-140q0 -16 -149 -31 q-12 -27 -30 -52q51 -113 51 -138q0 -4 -4 -7q-122 -71 -124 -71q-8 0 -46 47t-52 68q-20 -2 -30 -2t-30 2q-14 -21 -52 -68t-46 -47q-2 0 -124 71q-4 3 -4 7q0 25 51 138q-18 25 -30 52q-149 15 -149 31v140q0 16 149 31q13 29 30 52q-51 113 -51 138q0 4 4 7q4 2 35 20 t59 34t30 16q8 0 46 -46.5t52 -67.5q20 2 30 2t30 -2q51 71 92 112l6 2q4 0 124 -70q4 -3 4 -7q0 -25 -51 -138q17 -23 30 -52q149 -15 149 -31zM1920 1222v-140q0 -16 -149 -31q-12 -27 -30 -52q51 -113 51 -138q0 -4 -4 -7q-122 -71 -124 -71q-8 0 -46 47t-52 68 q-20 -2 -30 -2t-30 2q-14 -21 -52 -68t-46 -47q-2 0 -124 71q-4 3 -4 7q0 25 51 138q-18 25 -30 52q-149 15 -149 31v140q0 16 149 31q13 29 30 52q-51 113 -51 138q0 4 4 7q4 2 35 20t59 34t30 16q8 0 46 -46.5t52 -67.5q20 2 30 2t30 -2q51 71 92 112l6 2q4 0 124 -70 q4 -3 4 -7q0 -25 -51 -138q17 -23 30 -52q149 -15 149 -31z" /> +<glyph unicode="" horiz-adv-x="1792" d="M1408 768q0 -139 -94 -257t-256.5 -186.5t-353.5 -68.5q-86 0 -176 16q-124 -88 -278 -128q-36 -9 -86 -16h-3q-11 0 -20.5 8t-11.5 21q-1 3 -1 6.5t0.5 6.5t2 6l2.5 5t3.5 5.5t4 5t4.5 5t4 4.5q5 6 23 25t26 29.5t22.5 29t25 38.5t20.5 44q-124 72 -195 177t-71 224 q0 139 94 257t256.5 186.5t353.5 68.5t353.5 -68.5t256.5 -186.5t94 -257zM1792 512q0 -120 -71 -224.5t-195 -176.5q10 -24 20.5 -44t25 -38.5t22.5 -29t26 -29.5t23 -25q1 -1 4 -4.5t4.5 -5t4 -5t3.5 -5.5l2.5 -5t2 -6t0.5 -6.5t-1 -6.5q-3 -14 -13 -22t-22 -7 q-50 7 -86 16q-154 40 -278 128q-90 -16 -176 -16q-271 0 -472 132q58 -4 88 -4q161 0 309 45t264 129q125 92 192 212t67 254q0 77 -23 152q129 -71 204 -178t75 -230z" /> +<glyph unicode="" d="M256 192q0 26 -19 45t-45 19t-45 -19t-19 -45t19 -45t45 -19t45 19t19 45zM1408 768q0 51 -39 89.5t-89 38.5h-352q0 58 48 159.5t48 160.5q0 98 -32 145t-128 47q-26 -26 -38 -85t-30.5 -125.5t-59.5 -109.5q-22 -23 -77 -91q-4 -5 -23 -30t-31.5 -41t-34.5 -42.5 t-40 -44t-38.5 -35.5t-40 -27t-35.5 -9h-32v-640h32q13 0 31.5 -3t33 -6.5t38 -11t35 -11.5t35.5 -12.5t29 -10.5q211 -73 342 -73h121q192 0 192 167q0 26 -5 56q30 16 47.5 52.5t17.5 73.5t-18 69q53 50 53 119q0 25 -10 55.5t-25 47.5q32 1 53.5 47t21.5 81zM1536 769 q0 -89 -49 -163q9 -33 9 -69q0 -77 -38 -144q3 -21 3 -43q0 -101 -60 -178q1 -139 -85 -219.5t-227 -80.5h-36h-93q-96 0 -189.5 22.5t-216.5 65.5q-116 40 -138 40h-288q-53 0 -90.5 37.5t-37.5 90.5v640q0 53 37.5 90.5t90.5 37.5h274q36 24 137 155q58 75 107 128 q24 25 35.5 85.5t30.5 126.5t62 108q39 37 90 37q84 0 151 -32.5t102 -101.5t35 -186q0 -93 -48 -192h176q104 0 180 -76t76 -179z" /> +<glyph unicode="" d="M256 1088q0 26 -19 45t-45 19t-45 -19t-19 -45t19 -45t45 -19t45 19t19 45zM1408 512q0 35 -21.5 81t-53.5 47q15 17 25 47.5t10 55.5q0 69 -53 119q18 32 18 69t-17.5 73.5t-47.5 52.5q5 30 5 56q0 85 -49 126t-136 41h-128q-131 0 -342 -73q-5 -2 -29 -10.5 t-35.5 -12.5t-35 -11.5t-38 -11t-33 -6.5t-31.5 -3h-32v-640h32q16 0 35.5 -9t40 -27t38.5 -35.5t40 -44t34.5 -42.5t31.5 -41t23 -30q55 -68 77 -91q41 -43 59.5 -109.5t30.5 -125.5t38 -85q96 0 128 47t32 145q0 59 -48 160.5t-48 159.5h352q50 0 89 38.5t39 89.5z M1536 511q0 -103 -76 -179t-180 -76h-176q48 -99 48 -192q0 -118 -35 -186q-35 -69 -102 -101.5t-151 -32.5q-51 0 -90 37q-34 33 -54 82t-25.5 90.5t-17.5 84.5t-31 64q-48 50 -107 127q-101 131 -137 155h-274q-53 0 -90.5 37.5t-37.5 90.5v640q0 53 37.5 90.5t90.5 37.5 h288q22 0 138 40q128 44 223 66t200 22h112q140 0 226.5 -79t85.5 -216v-5q60 -77 60 -178q0 -22 -3 -43q38 -67 38 -144q0 -36 -9 -69q49 -74 49 -163z" /> +<glyph unicode="" horiz-adv-x="896" d="M832 1504v-1339l-449 -236q-22 -12 -40 -12q-21 0 -31.5 14.5t-10.5 35.5q0 6 2 20l86 500l-364 354q-25 27 -25 48q0 37 56 46l502 73l225 455q19 41 49 41z" /> +<glyph unicode="" horiz-adv-x="1792" d="M1664 940q0 81 -21.5 143t-55 98.5t-81.5 59.5t-94 31t-98 8t-112 -25.5t-110.5 -64t-86.5 -72t-60 -61.5q-18 -22 -49 -22t-49 22q-24 28 -60 61.5t-86.5 72t-110.5 64t-112 25.5t-98 -8t-94 -31t-81.5 -59.5t-55 -98.5t-21.5 -143q0 -168 187 -355l581 -560l580 559 q188 188 188 356zM1792 940q0 -221 -229 -450l-623 -600q-18 -18 -44 -18t-44 18l-624 602q-10 8 -27.5 26t-55.5 65.5t-68 97.5t-53.5 121t-23.5 138q0 220 127 344t351 124q62 0 126.5 -21.5t120 -58t95.5 -68.5t76 -68q36 36 76 68t95.5 68.5t120 58t126.5 21.5 q224 0 351 -124t127 -344z" /> +<glyph unicode="" horiz-adv-x="1664" d="M640 96q0 -4 1 -20t0.5 -26.5t-3 -23.5t-10 -19.5t-20.5 -6.5h-320q-119 0 -203.5 84.5t-84.5 203.5v704q0 119 84.5 203.5t203.5 84.5h320q13 0 22.5 -9.5t9.5 -22.5q0 -4 1 -20t0.5 -26.5t-3 -23.5t-10 -19.5t-20.5 -6.5h-320q-66 0 -113 -47t-47 -113v-704 q0 -66 47 -113t113 -47h288h11h13t11.5 -1t11.5 -3t8 -5.5t7 -9t2 -13.5zM1568 640q0 -26 -19 -45l-544 -544q-19 -19 -45 -19t-45 19t-19 45v288h-448q-26 0 -45 19t-19 45v384q0 26 19 45t45 19h448v288q0 26 19 45t45 19t45 -19l544 -544q19 -19 19 -45z" /> +<glyph unicode="" d="M237 122h231v694h-231v-694zM483 1030q-1 52 -36 86t-93 34t-94.5 -34t-36.5 -86q0 -51 35.5 -85.5t92.5 -34.5h1q59 0 95 34.5t36 85.5zM1068 122h231v398q0 154 -73 233t-193 79q-136 0 -209 -117h2v101h-231q3 -66 0 -694h231v388q0 38 7 56q15 35 45 59.5t74 24.5 q116 0 116 -157v-371zM1536 1120v-960q0 -119 -84.5 -203.5t-203.5 -84.5h-960q-119 0 -203.5 84.5t-84.5 203.5v960q0 119 84.5 203.5t203.5 84.5h960q119 0 203.5 -84.5t84.5 -203.5z" /> +<glyph unicode="" horiz-adv-x="1152" d="M480 672v448q0 14 -9 23t-23 9t-23 -9t-9 -23v-448q0 -14 9 -23t23 -9t23 9t9 23zM1152 320q0 -26 -19 -45t-45 -19h-429l-51 -483q-2 -12 -10.5 -20.5t-20.5 -8.5h-1q-27 0 -32 27l-76 485h-404q-26 0 -45 19t-19 45q0 123 78.5 221.5t177.5 98.5v512q-52 0 -90 38 t-38 90t38 90t90 38h640q52 0 90 -38t38 -90t-38 -90t-90 -38v-512q99 0 177.5 -98.5t78.5 -221.5z" /> +<glyph unicode="" horiz-adv-x="1792" d="M1408 608v-320q0 -119 -84.5 -203.5t-203.5 -84.5h-832q-119 0 -203.5 84.5t-84.5 203.5v832q0 119 84.5 203.5t203.5 84.5h704q14 0 23 -9t9 -23v-64q0 -14 -9 -23t-23 -9h-704q-66 0 -113 -47t-47 -113v-832q0 -66 47 -113t113 -47h832q66 0 113 47t47 113v320 q0 14 9 23t23 9h64q14 0 23 -9t9 -23zM1792 1472v-512q0 -26 -19 -45t-45 -19t-45 19l-176 176l-652 -652q-10 -10 -23 -10t-23 10l-114 114q-10 10 -10 23t10 23l652 652l-176 176q-19 19 -19 45t19 45t45 19h512q26 0 45 -19t19 -45z" /> +<glyph unicode="" d="M1184 640q0 -26 -19 -45l-544 -544q-19 -19 -45 -19t-45 19t-19 45v288h-448q-26 0 -45 19t-19 45v384q0 26 19 45t45 19h448v288q0 26 19 45t45 19t45 -19l544 -544q19 -19 19 -45zM1536 992v-704q0 -119 -84.5 -203.5t-203.5 -84.5h-320q-13 0 -22.5 9.5t-9.5 22.5 q0 4 -1 20t-0.5 26.5t3 23.5t10 19.5t20.5 6.5h320q66 0 113 47t47 113v704q0 66 -47 113t-113 47h-288h-11h-13t-11.5 1t-11.5 3t-8 5.5t-7 9t-2 13.5q0 4 -1 20t-0.5 26.5t3 23.5t10 19.5t20.5 6.5h320q119 0 203.5 -84.5t84.5 -203.5z" /> +<glyph unicode="" horiz-adv-x="1664" d="M458 653q-74 162 -74 371h-256v-96q0 -78 94.5 -162t235.5 -113zM1536 928v96h-256q0 -209 -74 -371q141 29 235.5 113t94.5 162zM1664 1056v-128q0 -71 -41.5 -143t-112 -130t-173 -97.5t-215.5 -44.5q-42 -54 -95 -95q-38 -34 -52.5 -72.5t-14.5 -89.5q0 -54 30.5 -91 t97.5 -37q75 0 133.5 -45.5t58.5 -114.5v-64q0 -14 -9 -23t-23 -9h-832q-14 0 -23 9t-9 23v64q0 69 58.5 114.5t133.5 45.5q67 0 97.5 37t30.5 91q0 51 -14.5 89.5t-52.5 72.5q-53 41 -95 95q-113 5 -215.5 44.5t-173 97.5t-112 130t-41.5 143v128q0 40 28 68t68 28h288v96 q0 66 47 113t113 47h576q66 0 113 -47t47 -113v-96h288q40 0 68 -28t28 -68z" /> +<glyph unicode="" d="M394 184q-8 -9 -20 3q-13 11 -4 19q8 9 20 -3q12 -11 4 -19zM352 245q9 -12 0 -19q-8 -6 -17 7t0 18q9 7 17 -6zM291 305q-5 -7 -13 -2q-10 5 -7 12q3 5 13 2q10 -5 7 -12zM322 271q-6 -7 -16 3q-9 11 -2 16q6 6 16 -3q9 -11 2 -16zM451 159q-4 -12 -19 -6q-17 4 -13 15 t19 7q16 -5 13 -16zM514 154q0 -11 -16 -11q-17 -2 -17 11q0 11 16 11q17 2 17 -11zM572 164q2 -10 -14 -14t-18 8t14 15q16 2 18 -9zM1536 1120v-960q0 -119 -84.5 -203.5t-203.5 -84.5h-224q-16 0 -24.5 1t-19.5 5t-16 14.5t-5 27.5v239q0 97 -52 142q57 6 102.5 18t94 39 t81 66.5t53 105t20.5 150.5q0 121 -79 206q37 91 -8 204q-28 9 -81 -11t-92 -44l-38 -24q-93 26 -192 26t-192 -26q-16 11 -42.5 27t-83.5 38.5t-86 13.5q-44 -113 -7 -204q-79 -85 -79 -206q0 -85 20.5 -150t52.5 -105t80.5 -67t94 -39t102.5 -18q-40 -36 -49 -103 q-21 -10 -45 -15t-57 -5t-65.5 21.5t-55.5 62.5q-19 32 -48.5 52t-49.5 24l-20 3q-21 0 -29 -4.5t-5 -11.5t9 -14t13 -12l7 -5q22 -10 43.5 -38t31.5 -51l10 -23q13 -38 44 -61.5t67 -30t69.5 -7t55.5 3.5l23 4q0 -38 0.5 -103t0.5 -68q0 -22 -11 -33.5t-22 -13t-33 -1.5 h-224q-119 0 -203.5 84.5t-84.5 203.5v960q0 119 84.5 203.5t203.5 84.5h960q119 0 203.5 -84.5t84.5 -203.5z" /> +<glyph unicode="" horiz-adv-x="1664" d="M1280 64q0 26 -19 45t-45 19t-45 -19t-19 -45t19 -45t45 -19t45 19t19 45zM1536 64q0 26 -19 45t-45 19t-45 -19t-19 -45t19 -45t45 -19t45 19t19 45zM1664 288v-320q0 -40 -28 -68t-68 -28h-1472q-40 0 -68 28t-28 68v320q0 40 28 68t68 28h427q21 -56 70.5 -92 t110.5 -36h256q61 0 110.5 36t70.5 92h427q40 0 68 -28t28 -68zM1339 936q-17 -40 -59 -40h-256v-448q0 -26 -19 -45t-45 -19h-256q-26 0 -45 19t-19 45v448h-256q-42 0 -59 40q-17 39 14 69l448 448q18 19 45 19t45 -19l448 -448q31 -30 14 -69z" /> +<glyph unicode="" d="M1407 710q0 44 -7 113.5t-18 96.5q-12 30 -17 44t-9 36.5t-4 48.5q0 23 5 68.5t5 67.5q0 37 -10 55q-4 1 -13 1q-19 0 -58 -4.5t-59 -4.5q-60 0 -176 24t-175 24q-43 0 -94.5 -11.5t-85 -23.5t-89.5 -34q-137 -54 -202 -103q-96 -73 -159.5 -189.5t-88 -236t-24.5 -248.5 q0 -40 12.5 -120t12.5 -121q0 -23 -11 -66.5t-11 -65.5t12 -36.5t34 -14.5q24 0 72.5 11t73.5 11q57 0 169.5 -15.5t169.5 -15.5q181 0 284 36q129 45 235.5 152.5t166 245.5t59.5 275zM1535 712q0 -165 -70 -327.5t-196 -288t-281 -180.5q-124 -44 -326 -44 q-57 0 -170 14.5t-169 14.5q-24 0 -72.5 -14.5t-73.5 -14.5q-73 0 -123.5 55.5t-50.5 128.5q0 24 11 68t11 67q0 40 -12.5 120.5t-12.5 121.5q0 111 18 217.5t54.5 209.5t100.5 194t150 156q78 59 232 120q194 78 316 78q60 0 175.5 -24t173.5 -24q19 0 57 5t58 5 q81 0 118 -50.5t37 -134.5q0 -23 -5 -68t-5 -68q0 -10 1 -18.5t3 -17t4 -13.5t6.5 -16t6.5 -17q16 -40 25 -118.5t9 -136.5z" /> +<glyph unicode="" horiz-adv-x="1408" d="M1408 296q0 -27 -10 -70.5t-21 -68.5q-21 -50 -122 -106q-94 -51 -186 -51q-27 0 -52.5 3.5t-57.5 12.5t-47.5 14.5t-55.5 20.5t-49 18q-98 35 -175 83q-128 79 -264.5 215.5t-215.5 264.5q-48 77 -83 175q-3 9 -18 49t-20.5 55.5t-14.5 47.5t-12.5 57.5t-3.5 52.5 q0 92 51 186q56 101 106 122q25 11 68.5 21t70.5 10q14 0 21 -3q18 -6 53 -76q11 -19 30 -54t35 -63.5t31 -53.5q3 -4 17.5 -25t21.5 -35.5t7 -28.5q0 -20 -28.5 -50t-62 -55t-62 -53t-28.5 -46q0 -9 5 -22.5t8.5 -20.5t14 -24t11.5 -19q76 -137 174 -235t235 -174 q2 -1 19 -11.5t24 -14t20.5 -8.5t22.5 -5q18 0 46 28.5t53 62t55 62t50 28.5q14 0 28.5 -7t35.5 -21.5t25 -17.5q25 -15 53.5 -31t63.5 -35t54 -30q70 -35 76 -53q3 -7 3 -21z" /> +<glyph unicode="" horiz-adv-x="1408" d="M1120 1280h-832q-66 0 -113 -47t-47 -113v-832q0 -66 47 -113t113 -47h832q66 0 113 47t47 113v832q0 66 -47 113t-113 47zM1408 1120v-832q0 -119 -84.5 -203.5t-203.5 -84.5h-832q-119 0 -203.5 84.5t-84.5 203.5v832q0 119 84.5 203.5t203.5 84.5h832 q119 0 203.5 -84.5t84.5 -203.5z" /> +<glyph unicode="" horiz-adv-x="1280" d="M1152 1280h-1024v-1242l423 406l89 85l89 -85l423 -406v1242zM1164 1408q23 0 44 -9q33 -13 52.5 -41t19.5 -62v-1289q0 -34 -19.5 -62t-52.5 -41q-19 -8 -44 -8q-48 0 -83 32l-441 424l-441 -424q-36 -33 -83 -33q-23 0 -44 9q-33 13 -52.5 41t-19.5 62v1289 q0 34 19.5 62t52.5 41q21 9 44 9h1048z" /> +<glyph unicode="" d="M1280 343q0 11 -2 16q-3 8 -38.5 29.5t-88.5 49.5l-53 29q-5 3 -19 13t-25 15t-21 5q-18 0 -47 -32.5t-57 -65.5t-44 -33q-7 0 -16.5 3.5t-15.5 6.5t-17 9.5t-14 8.5q-99 55 -170.5 126.5t-126.5 170.5q-2 3 -8.5 14t-9.5 17t-6.5 15.5t-3.5 16.5q0 13 20.5 33.5t45 38.5 t45 39.5t20.5 36.5q0 10 -5 21t-15 25t-13 19q-3 6 -15 28.5t-25 45.5t-26.5 47.5t-25 40.5t-16.5 18t-16 2q-48 0 -101 -22q-46 -21 -80 -94.5t-34 -130.5q0 -16 2.5 -34t5 -30.5t9 -33t10 -29.5t12.5 -33t11 -30q60 -164 216.5 -320.5t320.5 -216.5q6 -2 30 -11t33 -12.5 t29.5 -10t33 -9t30.5 -5t34 -2.5q57 0 130.5 34t94.5 80q22 53 22 101zM1536 1120v-960q0 -119 -84.5 -203.5t-203.5 -84.5h-960q-119 0 -203.5 84.5t-84.5 203.5v960q0 119 84.5 203.5t203.5 84.5h960q119 0 203.5 -84.5t84.5 -203.5z" /> +<glyph unicode="" horiz-adv-x="1664" d="M1620 1128q-67 -98 -162 -167q1 -14 1 -42q0 -130 -38 -259.5t-115.5 -248.5t-184.5 -210.5t-258 -146t-323 -54.5q-271 0 -496 145q35 -4 78 -4q225 0 401 138q-105 2 -188 64.5t-114 159.5q33 -5 61 -5q43 0 85 11q-112 23 -185.5 111.5t-73.5 205.5v4q68 -38 146 -41 q-66 44 -105 115t-39 154q0 88 44 163q121 -149 294.5 -238.5t371.5 -99.5q-8 38 -8 74q0 134 94.5 228.5t228.5 94.5q140 0 236 -102q109 21 205 78q-37 -115 -142 -178q93 10 186 50z" /> +<glyph unicode="" horiz-adv-x="768" d="M511 980h257l-30 -284h-227v-824h-341v824h-170v284h170v171q0 182 86 275.5t283 93.5h227v-284h-142q-39 0 -62.5 -6.5t-34 -23.5t-13.5 -34.5t-3 -49.5v-142z" /> +<glyph unicode="" d="M1536 640q0 -251 -146.5 -451.5t-378.5 -277.5q-27 -5 -39.5 7t-12.5 30v211q0 97 -52 142q57 6 102.5 18t94 39t81 66.5t53 105t20.5 150.5q0 121 -79 206q37 91 -8 204q-28 9 -81 -11t-92 -44l-38 -24q-93 26 -192 26t-192 -26q-16 11 -42.5 27t-83.5 38.5t-86 13.5 q-44 -113 -7 -204q-79 -85 -79 -206q0 -85 20.5 -150t52.5 -105t80.5 -67t94 -39t102.5 -18q-40 -36 -49 -103q-21 -10 -45 -15t-57 -5t-65.5 21.5t-55.5 62.5q-19 32 -48.5 52t-49.5 24l-20 3q-21 0 -29 -4.5t-5 -11.5t9 -14t13 -12l7 -5q22 -10 43.5 -38t31.5 -51l10 -23 q13 -38 44 -61.5t67 -30t69.5 -7t55.5 3.5l23 4q0 -38 0.5 -89t0.5 -54q0 -18 -13 -30t-40 -7q-232 77 -378.5 277.5t-146.5 451.5q0 209 103 385.5t279.5 279.5t385.5 103t385.5 -103t279.5 -279.5t103 -385.5z" /> +<glyph unicode="" horiz-adv-x="1664" d="M1664 960v-256q0 -26 -19 -45t-45 -19h-64q-26 0 -45 19t-19 45v256q0 106 -75 181t-181 75t-181 -75t-75 -181v-192h96q40 0 68 -28t28 -68v-576q0 -40 -28 -68t-68 -28h-960q-40 0 -68 28t-28 68v576q0 40 28 68t68 28h672v192q0 185 131.5 316.5t316.5 131.5 t316.5 -131.5t131.5 -316.5z" /> +<glyph unicode="" horiz-adv-x="1920" d="M1760 1408q66 0 113 -47t47 -113v-1216q0 -66 -47 -113t-113 -47h-1600q-66 0 -113 47t-47 113v1216q0 66 47 113t113 47h1600zM160 1280q-13 0 -22.5 -9.5t-9.5 -22.5v-224h1664v224q0 13 -9.5 22.5t-22.5 9.5h-1600zM1760 0q13 0 22.5 9.5t9.5 22.5v608h-1664v-608 q0 -13 9.5 -22.5t22.5 -9.5h1600zM256 128v128h256v-128h-256zM640 128v128h384v-128h-384z" /> +<glyph unicode="" horiz-adv-x="1408" d="M384 192q0 -80 -56 -136t-136 -56t-136 56t-56 136t56 136t136 56t136 -56t56 -136zM896 69q2 -28 -17 -48q-18 -21 -47 -21h-135q-25 0 -43 16.5t-20 41.5q-22 229 -184.5 391.5t-391.5 184.5q-25 2 -41.5 20t-16.5 43v135q0 29 21 47q17 17 43 17h5q160 -13 306 -80.5 t259 -181.5q114 -113 181.5 -259t80.5 -306zM1408 67q2 -27 -18 -47q-18 -20 -46 -20h-143q-26 0 -44.5 17.5t-19.5 42.5q-12 215 -101 408.5t-231.5 336t-336 231.5t-408.5 102q-25 1 -42.5 19.5t-17.5 43.5v143q0 28 20 46q18 18 44 18h3q262 -13 501.5 -120t425.5 -294 q187 -186 294 -425.5t120 -501.5z" /> +<glyph unicode="" d="M1040 320q0 -33 -23.5 -56.5t-56.5 -23.5t-56.5 23.5t-23.5 56.5t23.5 56.5t56.5 23.5t56.5 -23.5t23.5 -56.5zM1296 320q0 -33 -23.5 -56.5t-56.5 -23.5t-56.5 23.5t-23.5 56.5t23.5 56.5t56.5 23.5t56.5 -23.5t23.5 -56.5zM1408 160v320q0 13 -9.5 22.5t-22.5 9.5 h-1216q-13 0 -22.5 -9.5t-9.5 -22.5v-320q0 -13 9.5 -22.5t22.5 -9.5h1216q13 0 22.5 9.5t9.5 22.5zM178 640h1180l-157 482q-4 13 -16 21.5t-26 8.5h-782q-14 0 -26 -8.5t-16 -21.5zM1536 480v-320q0 -66 -47 -113t-113 -47h-1216q-66 0 -113 47t-47 113v320q0 25 16 75 l197 606q17 53 63 86t101 33h782q55 0 101 -33t63 -86l197 -606q16 -50 16 -75z" /> +<glyph unicode="" horiz-adv-x="1792" d="M1664 896q53 0 90.5 -37.5t37.5 -90.5t-37.5 -90.5t-90.5 -37.5v-384q0 -52 -38 -90t-90 -38q-417 347 -812 380q-58 -19 -91 -66t-31 -100.5t40 -92.5q-20 -33 -23 -65.5t6 -58t33.5 -55t48 -50t61.5 -50.5q-29 -58 -111.5 -83t-168.5 -11.5t-132 55.5q-7 23 -29.5 87.5 t-32 94.5t-23 89t-15 101t3.5 98.5t22 110.5h-122q-66 0 -113 47t-47 113v192q0 66 47 113t113 47h480q435 0 896 384q52 0 90 -38t38 -90v-384zM1536 292v954q-394 -302 -768 -343v-270q377 -42 768 -341z" /> +<glyph unicode="" horiz-adv-x="1664" d="M848 -160q0 16 -16 16q-59 0 -101.5 42.5t-42.5 101.5q0 16 -16 16t-16 -16q0 -73 51.5 -124.5t124.5 -51.5q16 0 16 16zM183 128h1298q-164 181 -246.5 411.5t-82.5 484.5q0 256 -320 256t-320 -256q0 -254 -82.5 -484.5t-246.5 -411.5zM1664 128q0 -52 -38 -90t-90 -38 h-448q0 -106 -75 -181t-181 -75t-181 75t-75 181h-448q-52 0 -90 38t-38 90q190 161 287 397.5t97 498.5q0 165 96 262t264 117q-8 18 -8 37q0 40 28 68t68 28t68 -28t28 -68q0 -19 -8 -37q168 -20 264 -117t96 -262q0 -262 97 -498.5t287 -397.5z" /> +<glyph unicode="" d="M1376 640l138 -135q30 -28 20 -70q-12 -41 -52 -51l-188 -48l53 -186q12 -41 -19 -70q-29 -31 -70 -19l-186 53l-48 -188q-10 -40 -51 -52q-12 -2 -19 -2q-31 0 -51 22l-135 138l-135 -138q-28 -30 -70 -20q-41 11 -51 52l-48 188l-186 -53q-41 -12 -70 19q-31 29 -19 70 l53 186l-188 48q-40 10 -52 51q-10 42 20 70l138 135l-138 135q-30 28 -20 70q12 41 52 51l188 48l-53 186q-12 41 19 70q29 31 70 19l186 -53l48 188q10 41 51 51q41 12 70 -19l135 -139l135 139q29 30 70 19q41 -10 51 -51l48 -188l186 53q41 12 70 -19q31 -29 19 -70 l-53 -186l188 -48q40 -10 52 -51q10 -42 -20 -70z" /> +<glyph unicode="" horiz-adv-x="1792" d="M256 192q0 26 -19 45t-45 19t-45 -19t-19 -45t19 -45t45 -19t45 19t19 45zM1664 768q0 51 -39 89.5t-89 38.5h-576q0 20 15 48.5t33 55t33 68t15 84.5q0 67 -44.5 97.5t-115.5 30.5q-24 0 -90 -139q-24 -44 -37 -65q-40 -64 -112 -145q-71 -81 -101 -106 q-69 -57 -140 -57h-32v-640h32q72 0 167 -32t193.5 -64t179.5 -32q189 0 189 167q0 26 -5 56q30 16 47.5 52.5t17.5 73.5t-18 69q53 50 53 119q0 25 -10 55.5t-25 47.5h331q52 0 90 38t38 90zM1792 769q0 -105 -75.5 -181t-180.5 -76h-169q-4 -62 -37 -119q3 -21 3 -43 q0 -101 -60 -178q1 -139 -85 -219.5t-227 -80.5q-133 0 -322 69q-164 59 -223 59h-288q-53 0 -90.5 37.5t-37.5 90.5v640q0 53 37.5 90.5t90.5 37.5h288q10 0 21.5 4.5t23.5 14t22.5 18t24 22.5t20.5 21.5t19 21.5t14 17q65 74 100 129q13 21 33 62t37 72t40.5 63t55 49.5 t69.5 17.5q125 0 206.5 -67t81.5 -189q0 -68 -22 -128h374q104 0 180 -76t76 -179z" /> +<glyph unicode="" horiz-adv-x="1792" d="M1376 128h32v640h-32q-35 0 -67.5 12t-62.5 37t-50 46t-49 54q-2 3 -3.5 4.5t-4 4.5t-4.5 5q-72 81 -112 145q-14 22 -38 68q-1 3 -10.5 22.5t-18.5 36t-20 35.5t-21.5 30.5t-18.5 11.5q-71 0 -115.5 -30.5t-44.5 -97.5q0 -43 15 -84.5t33 -68t33 -55t15 -48.5h-576 q-50 0 -89 -38.5t-39 -89.5q0 -52 38 -90t90 -38h331q-15 -17 -25 -47.5t-10 -55.5q0 -69 53 -119q-18 -32 -18 -69t17.5 -73.5t47.5 -52.5q-4 -24 -4 -56q0 -85 48.5 -126t135.5 -41q84 0 183 32t194 64t167 32zM1664 192q0 26 -19 45t-45 19t-45 -19t-19 -45t19 -45 t45 -19t45 19t19 45zM1792 768v-640q0 -53 -37.5 -90.5t-90.5 -37.5h-288q-59 0 -223 -59q-190 -69 -317 -69q-142 0 -230 77.5t-87 217.5l1 5q-61 76 -61 178q0 22 3 43q-33 57 -37 119h-169q-105 0 -180.5 76t-75.5 181q0 103 76 179t180 76h374q-22 60 -22 128 q0 122 81.5 189t206.5 67q38 0 69.5 -17.5t55 -49.5t40.5 -63t37 -72t33 -62q35 -55 100 -129q2 -3 14 -17t19 -21.5t20.5 -21.5t24 -22.5t22.5 -18t23.5 -14t21.5 -4.5h288q53 0 90.5 -37.5t37.5 -90.5z" /> +<glyph unicode="" d="M1280 -64q0 26 -19 45t-45 19t-45 -19t-19 -45t19 -45t45 -19t45 19t19 45zM1408 700q0 189 -167 189q-26 0 -56 -5q-16 30 -52.5 47.5t-73.5 17.5t-69 -18q-50 53 -119 53q-25 0 -55.5 -10t-47.5 -25v331q0 52 -38 90t-90 38q-51 0 -89.5 -39t-38.5 -89v-576 q-20 0 -48.5 15t-55 33t-68 33t-84.5 15q-67 0 -97.5 -44.5t-30.5 -115.5q0 -24 139 -90q44 -24 65 -37q64 -40 145 -112q81 -71 106 -101q57 -69 57 -140v-32h640v32q0 72 32 167t64 193.5t32 179.5zM1536 705q0 -133 -69 -322q-59 -164 -59 -223v-288q0 -53 -37.5 -90.5 t-90.5 -37.5h-640q-53 0 -90.5 37.5t-37.5 90.5v288q0 10 -4.5 21.5t-14 23.5t-18 22.5t-22.5 24t-21.5 20.5t-21.5 19t-17 14q-74 65 -129 100q-21 13 -62 33t-72 37t-63 40.5t-49.5 55t-17.5 69.5q0 125 67 206.5t189 81.5q68 0 128 -22v374q0 104 76 180t179 76 q105 0 181 -75.5t76 -180.5v-169q62 -4 119 -37q21 3 43 3q101 0 178 -60q139 1 219.5 -85t80.5 -227z" /> +<glyph unicode="" d="M1408 576q0 84 -32 183t-64 194t-32 167v32h-640v-32q0 -35 -12 -67.5t-37 -62.5t-46 -50t-54 -49q-9 -8 -14 -12q-81 -72 -145 -112q-22 -14 -68 -38q-3 -1 -22.5 -10.5t-36 -18.5t-35.5 -20t-30.5 -21.5t-11.5 -18.5q0 -71 30.5 -115.5t97.5 -44.5q43 0 84.5 15t68 33 t55 33t48.5 15v-576q0 -50 38.5 -89t89.5 -39q52 0 90 38t38 90v331q46 -35 103 -35q69 0 119 53q32 -18 69 -18t73.5 17.5t52.5 47.5q24 -4 56 -4q85 0 126 48.5t41 135.5zM1280 1344q0 26 -19 45t-45 19t-45 -19t-19 -45t19 -45t45 -19t45 19t19 45zM1536 580 q0 -142 -77.5 -230t-217.5 -87l-5 1q-76 -61 -178 -61q-22 0 -43 3q-54 -30 -119 -37v-169q0 -105 -76 -180.5t-181 -75.5q-103 0 -179 76t-76 180v374q-54 -22 -128 -22q-121 0 -188.5 81.5t-67.5 206.5q0 38 17.5 69.5t49.5 55t63 40.5t72 37t62 33q55 35 129 100 q3 2 17 14t21.5 19t21.5 20.5t22.5 24t18 22.5t14 23.5t4.5 21.5v288q0 53 37.5 90.5t90.5 37.5h640q53 0 90.5 -37.5t37.5 -90.5v-288q0 -59 59 -223q69 -190 69 -317z" /> +<glyph unicode="" d="M1280 576v128q0 26 -19 45t-45 19h-502l189 189q19 19 19 45t-19 45l-91 91q-18 18 -45 18t-45 -18l-362 -362l-91 -91q-18 -18 -18 -45t18 -45l91 -91l362 -362q18 -18 45 -18t45 18l91 91q18 18 18 45t-18 45l-189 189h502q26 0 45 19t19 45zM1536 640 q0 -209 -103 -385.5t-279.5 -279.5t-385.5 -103t-385.5 103t-279.5 279.5t-103 385.5t103 385.5t279.5 279.5t385.5 103t385.5 -103t279.5 -279.5t103 -385.5z" /> +<glyph unicode="" d="M1285 640q0 27 -18 45l-91 91l-362 362q-18 18 -45 18t-45 -18l-91 -91q-18 -18 -18 -45t18 -45l189 -189h-502q-26 0 -45 -19t-19 -45v-128q0 -26 19 -45t45 -19h502l-189 -189q-19 -19 -19 -45t19 -45l91 -91q18 -18 45 -18t45 18l362 362l91 91q18 18 18 45zM1536 640 q0 -209 -103 -385.5t-279.5 -279.5t-385.5 -103t-385.5 103t-279.5 279.5t-103 385.5t103 385.5t279.5 279.5t385.5 103t385.5 -103t279.5 -279.5t103 -385.5z" /> +<glyph unicode="" d="M1284 641q0 27 -18 45l-362 362l-91 91q-18 18 -45 18t-45 -18l-91 -91l-362 -362q-18 -18 -18 -45t18 -45l91 -91q18 -18 45 -18t45 18l189 189v-502q0 -26 19 -45t45 -19h128q26 0 45 19t19 45v502l189 -189q19 -19 45 -19t45 19l91 91q18 18 18 45zM1536 640 q0 -209 -103 -385.5t-279.5 -279.5t-385.5 -103t-385.5 103t-279.5 279.5t-103 385.5t103 385.5t279.5 279.5t385.5 103t385.5 -103t279.5 -279.5t103 -385.5z" /> +<glyph unicode="" d="M1284 639q0 27 -18 45l-91 91q-18 18 -45 18t-45 -18l-189 -189v502q0 26 -19 45t-45 19h-128q-26 0 -45 -19t-19 -45v-502l-189 189q-19 19 -45 19t-45 -19l-91 -91q-18 -18 -18 -45t18 -45l362 -362l91 -91q18 -18 45 -18t45 18l91 91l362 362q18 18 18 45zM1536 640 q0 -209 -103 -385.5t-279.5 -279.5t-385.5 -103t-385.5 103t-279.5 279.5t-103 385.5t103 385.5t279.5 279.5t385.5 103t385.5 -103t279.5 -279.5t103 -385.5z" /> +<glyph unicode="" d="M768 1408q209 0 385.5 -103t279.5 -279.5t103 -385.5t-103 -385.5t-279.5 -279.5t-385.5 -103t-385.5 103t-279.5 279.5t-103 385.5t103 385.5t279.5 279.5t385.5 103zM1042 887q-2 -1 -9.5 -9.5t-13.5 -9.5q2 0 4.5 5t5 11t3.5 7q6 7 22 15q14 6 52 12q34 8 51 -11 q-2 2 9.5 13t14.5 12q3 2 15 4.5t15 7.5l2 22q-12 -1 -17.5 7t-6.5 21q0 -2 -6 -8q0 7 -4.5 8t-11.5 -1t-9 -1q-10 3 -15 7.5t-8 16.5t-4 15q-2 5 -9.5 10.5t-9.5 10.5q-1 2 -2.5 5.5t-3 6.5t-4 5.5t-5.5 2.5t-7 -5t-7.5 -10t-4.5 -5q-3 2 -6 1.5t-4.5 -1t-4.5 -3t-5 -3.5 q-3 -2 -8.5 -3t-8.5 -2q15 5 -1 11q-10 4 -16 3q9 4 7.5 12t-8.5 14h5q-1 4 -8.5 8.5t-17.5 8.5t-13 6q-8 5 -34 9.5t-33 0.5q-5 -6 -4.5 -10.5t4 -14t3.5 -12.5q1 -6 -5.5 -13t-6.5 -12q0 -7 14 -15.5t10 -21.5q-3 -8 -16 -16t-16 -12q-5 -8 -1.5 -18.5t10.5 -16.5 q2 -2 1.5 -4t-3.5 -4.5t-5.5 -4t-6.5 -3.5l-3 -2q-11 -5 -20.5 6t-13.5 26q-7 25 -16 30q-23 8 -29 -1q-5 13 -41 26q-25 9 -58 4q6 1 0 15q-7 15 -19 12q3 6 4 17.5t1 13.5q3 13 12 23q1 1 7 8.5t9.5 13.5t0.5 6q35 -4 50 11q5 5 11.5 17t10.5 17q9 6 14 5.5t14.5 -5.5 t14.5 -5q14 -1 15.5 11t-7.5 20q12 -1 3 17q-5 7 -8 9q-12 4 -27 -5q-8 -4 2 -8q-1 1 -9.5 -10.5t-16.5 -17.5t-16 5q-1 1 -5.5 13.5t-9.5 13.5q-8 0 -16 -15q3 8 -11 15t-24 8q19 12 -8 27q-7 4 -20.5 5t-19.5 -4q-5 -7 -5.5 -11.5t5 -8t10.5 -5.5t11.5 -4t8.5 -3 q14 -10 8 -14q-2 -1 -8.5 -3.5t-11.5 -4.5t-6 -4q-3 -4 0 -14t-2 -14q-5 5 -9 17.5t-7 16.5q7 -9 -25 -6l-10 1q-4 0 -16 -2t-20.5 -1t-13.5 8q-4 8 0 20q1 4 4 2q-4 3 -11 9.5t-10 8.5q-46 -15 -94 -41q6 -1 12 1q5 2 13 6.5t10 5.5q34 14 42 7l5 5q14 -16 20 -25 q-7 4 -30 1q-20 -6 -22 -12q7 -12 5 -18q-4 3 -11.5 10t-14.5 11t-15 5q-16 0 -22 -1q-146 -80 -235 -222q7 -7 12 -8q4 -1 5 -9t2.5 -11t11.5 3q9 -8 3 -19q1 1 44 -27q19 -17 21 -21q3 -11 -10 -18q-1 2 -9 9t-9 4q-3 -5 0.5 -18.5t10.5 -12.5q-7 0 -9.5 -16t-2.5 -35.5 t-1 -23.5l2 -1q-3 -12 5.5 -34.5t21.5 -19.5q-13 -3 20 -43q6 -8 8 -9q3 -2 12 -7.5t15 -10t10 -10.5q4 -5 10 -22.5t14 -23.5q-2 -6 9.5 -20t10.5 -23q-1 0 -2.5 -1t-2.5 -1q3 -7 15.5 -14t15.5 -13q1 -3 2 -10t3 -11t8 -2q2 20 -24 62q-15 25 -17 29q-3 5 -5.5 15.5 t-4.5 14.5q2 0 6 -1.5t8.5 -3.5t7.5 -4t2 -3q-3 -7 2 -17.5t12 -18.5t17 -19t12 -13q6 -6 14 -19.5t0 -13.5q9 0 20 -10t17 -20q5 -8 8 -26t5 -24q2 -7 8.5 -13.5t12.5 -9.5l16 -8t13 -7q5 -2 18.5 -10.5t21.5 -11.5q10 -4 16 -4t14.5 2.5t13.5 3.5q15 2 29 -15t21 -21 q36 -19 55 -11q-2 -1 0.5 -7.5t8 -15.5t9 -14.5t5.5 -8.5q5 -6 18 -15t18 -15q6 4 7 9q-3 -8 7 -20t18 -10q14 3 14 32q-31 -15 -49 18q0 1 -2.5 5.5t-4 8.5t-2.5 8.5t0 7.5t5 3q9 0 10 3.5t-2 12.5t-4 13q-1 8 -11 20t-12 15q-5 -9 -16 -8t-16 9q0 -1 -1.5 -5.5t-1.5 -6.5 q-13 0 -15 1q1 3 2.5 17.5t3.5 22.5q1 4 5.5 12t7.5 14.5t4 12.5t-4.5 9.5t-17.5 2.5q-19 -1 -26 -20q-1 -3 -3 -10.5t-5 -11.5t-9 -7q-7 -3 -24 -2t-24 5q-13 8 -22.5 29t-9.5 37q0 10 2.5 26.5t3 25t-5.5 24.5q3 2 9 9.5t10 10.5q2 1 4.5 1.5t4.5 0t4 1.5t3 6q-1 1 -4 3 q-3 3 -4 3q7 -3 28.5 1.5t27.5 -1.5q15 -11 22 2q0 1 -2.5 9.5t-0.5 13.5q5 -27 29 -9q3 -3 15.5 -5t17.5 -5q3 -2 7 -5.5t5.5 -4.5t5 0.5t8.5 6.5q10 -14 12 -24q11 -40 19 -44q7 -3 11 -2t4.5 9.5t0 14t-1.5 12.5l-1 8v18l-1 8q-15 3 -18.5 12t1.5 18.5t15 18.5q1 1 8 3.5 t15.5 6.5t12.5 8q21 19 15 35q7 0 11 9q-1 0 -5 3t-7.5 5t-4.5 2q9 5 2 16q5 3 7.5 11t7.5 10q9 -12 21 -2q7 8 1 16q5 7 20.5 10.5t18.5 9.5q7 -2 8 2t1 12t3 12q4 5 15 9t13 5l17 11q3 4 0 4q18 -2 31 11q10 11 -6 20q3 6 -3 9.5t-15 5.5q3 1 11.5 0.5t10.5 1.5 q15 10 -7 16q-17 5 -43 -12zM879 10q206 36 351 189q-3 3 -12.5 4.5t-12.5 3.5q-18 7 -24 8q1 7 -2.5 13t-8 9t-12.5 8t-11 7q-2 2 -7 6t-7 5.5t-7.5 4.5t-8.5 2t-10 -1l-3 -1q-3 -1 -5.5 -2.5t-5.5 -3t-4 -3t0 -2.5q-21 17 -36 22q-5 1 -11 5.5t-10.5 7t-10 1.5t-11.5 -7 q-5 -5 -6 -15t-2 -13q-7 5 0 17.5t2 18.5q-3 6 -10.5 4.5t-12 -4.5t-11.5 -8.5t-9 -6.5t-8.5 -5.5t-8.5 -7.5q-3 -4 -6 -12t-5 -11q-2 4 -11.5 6.5t-9.5 5.5q2 -10 4 -35t5 -38q7 -31 -12 -48q-27 -25 -29 -40q-4 -22 12 -26q0 -7 -8 -20.5t-7 -21.5q0 -6 2 -16z" /> +<glyph unicode="" horiz-adv-x="1664" d="M384 64q0 26 -19 45t-45 19t-45 -19t-19 -45t19 -45t45 -19t45 19t19 45zM1028 484l-682 -682q-37 -37 -90 -37q-52 0 -91 37l-106 108q-38 36 -38 90q0 53 38 91l681 681q39 -98 114.5 -173.5t173.5 -114.5zM1662 919q0 -39 -23 -106q-47 -134 -164.5 -217.5 t-258.5 -83.5q-185 0 -316.5 131.5t-131.5 316.5t131.5 316.5t316.5 131.5q58 0 121.5 -16.5t107.5 -46.5q16 -11 16 -28t-16 -28l-293 -169v-224l193 -107q5 3 79 48.5t135.5 81t70.5 35.5q15 0 23.5 -10t8.5 -25z" /> +<glyph unicode="" horiz-adv-x="1792" d="M1024 128h640v128h-640v-128zM640 640h1024v128h-1024v-128zM1280 1152h384v128h-384v-128zM1792 320v-256q0 -26 -19 -45t-45 -19h-1664q-26 0 -45 19t-19 45v256q0 26 19 45t45 19h1664q26 0 45 -19t19 -45zM1792 832v-256q0 -26 -19 -45t-45 -19h-1664q-26 0 -45 19 t-19 45v256q0 26 19 45t45 19h1664q26 0 45 -19t19 -45zM1792 1344v-256q0 -26 -19 -45t-45 -19h-1664q-26 0 -45 19t-19 45v256q0 26 19 45t45 19h1664q26 0 45 -19t19 -45z" /> +<glyph unicode="" horiz-adv-x="1408" d="M1403 1241q17 -41 -14 -70l-493 -493v-742q0 -42 -39 -59q-13 -5 -25 -5q-27 0 -45 19l-256 256q-19 19 -19 45v486l-493 493q-31 29 -14 70q17 39 59 39h1280q42 0 59 -39z" /> +<glyph unicode="" horiz-adv-x="1792" d="M640 1280h512v128h-512v-128zM1792 640v-480q0 -66 -47 -113t-113 -47h-1472q-66 0 -113 47t-47 113v480h672v-160q0 -26 19 -45t45 -19h320q26 0 45 19t19 45v160h672zM1024 640v-128h-256v128h256zM1792 1120v-384h-1792v384q0 66 47 113t113 47h352v160q0 40 28 68 t68 28h576q40 0 68 -28t28 -68v-160h352q66 0 113 -47t47 -113z" /> +<glyph unicode="" d="M1283 995l-355 -355l355 -355l144 144q29 31 70 14q39 -17 39 -59v-448q0 -26 -19 -45t-45 -19h-448q-42 0 -59 40q-17 39 14 69l144 144l-355 355l-355 -355l144 -144q31 -30 14 -69q-17 -40 -59 -40h-448q-26 0 -45 19t-19 45v448q0 42 40 59q39 17 69 -14l144 -144 l355 355l-355 355l-144 -144q-19 -19 -45 -19q-12 0 -24 5q-40 17 -40 59v448q0 26 19 45t45 19h448q42 0 59 -40q17 -39 -14 -69l-144 -144l355 -355l355 355l-144 144q-31 30 -14 69q17 40 59 40h448q26 0 45 -19t19 -45v-448q0 -42 -39 -59q-13 -5 -25 -5q-26 0 -45 19z " /> +<glyph unicode="" horiz-adv-x="1920" d="M593 640q-162 -5 -265 -128h-134q-82 0 -138 40.5t-56 118.5q0 353 124 353q6 0 43.5 -21t97.5 -42.5t119 -21.5q67 0 133 23q-5 -37 -5 -66q0 -139 81 -256zM1664 3q0 -120 -73 -189.5t-194 -69.5h-874q-121 0 -194 69.5t-73 189.5q0 53 3.5 103.5t14 109t26.5 108.5 t43 97.5t62 81t85.5 53.5t111.5 20q10 0 43 -21.5t73 -48t107 -48t135 -21.5t135 21.5t107 48t73 48t43 21.5q61 0 111.5 -20t85.5 -53.5t62 -81t43 -97.5t26.5 -108.5t14 -109t3.5 -103.5zM640 1280q0 -106 -75 -181t-181 -75t-181 75t-75 181t75 181t181 75t181 -75 t75 -181zM1344 896q0 -159 -112.5 -271.5t-271.5 -112.5t-271.5 112.5t-112.5 271.5t112.5 271.5t271.5 112.5t271.5 -112.5t112.5 -271.5zM1920 671q0 -78 -56 -118.5t-138 -40.5h-134q-103 123 -265 128q81 117 81 256q0 29 -5 66q66 -23 133 -23q59 0 119 21.5t97.5 42.5 t43.5 21q124 0 124 -353zM1792 1280q0 -106 -75 -181t-181 -75t-181 75t-75 181t75 181t181 75t181 -75t75 -181z" /> +<glyph unicode="" horiz-adv-x="1664" d="M1456 320q0 40 -28 68l-208 208q-28 28 -68 28q-42 0 -72 -32q3 -3 19 -18.5t21.5 -21.5t15 -19t13 -25.5t3.5 -27.5q0 -40 -28 -68t-68 -28q-15 0 -27.5 3.5t-25.5 13t-19 15t-21.5 21.5t-18.5 19q-33 -31 -33 -73q0 -40 28 -68l206 -207q27 -27 68 -27q40 0 68 26 l147 146q28 28 28 67zM753 1025q0 40 -28 68l-206 207q-28 28 -68 28q-39 0 -68 -27l-147 -146q-28 -28 -28 -67q0 -40 28 -68l208 -208q27 -27 68 -27q42 0 72 31q-3 3 -19 18.5t-21.5 21.5t-15 19t-13 25.5t-3.5 27.5q0 40 28 68t68 28q15 0 27.5 -3.5t25.5 -13t19 -15 t21.5 -21.5t18.5 -19q33 31 33 73zM1648 320q0 -120 -85 -203l-147 -146q-83 -83 -203 -83q-121 0 -204 85l-206 207q-83 83 -83 203q0 123 88 209l-88 88q-86 -88 -208 -88q-120 0 -204 84l-208 208q-84 84 -84 204t85 203l147 146q83 83 203 83q121 0 204 -85l206 -207 q83 -83 83 -203q0 -123 -88 -209l88 -88q86 88 208 88q120 0 204 -84l208 -208q84 -84 84 -204z" /> +<glyph unicode="" horiz-adv-x="1920" d="M1920 384q0 -159 -112.5 -271.5t-271.5 -112.5h-1088q-185 0 -316.5 131.5t-131.5 316.5q0 132 71 241.5t187 163.5q-2 28 -2 43q0 212 150 362t362 150q158 0 286.5 -88t187.5 -230q70 62 166 62q106 0 181 -75t75 -181q0 -75 -41 -138q129 -30 213 -134.5t84 -239.5z " /> +<glyph unicode="" horiz-adv-x="1664" d="M1527 88q56 -89 21.5 -152.5t-140.5 -63.5h-1152q-106 0 -140.5 63.5t21.5 152.5l503 793v399h-64q-26 0 -45 19t-19 45t19 45t45 19h512q26 0 45 -19t19 -45t-19 -45t-45 -19h-64v-399zM748 813l-272 -429h712l-272 429l-20 31v37v399h-128v-399v-37z" /> +<glyph unicode="" horiz-adv-x="1792" d="M960 640q26 0 45 -19t19 -45t-19 -45t-45 -19t-45 19t-19 45t19 45t45 19zM1260 576l507 -398q28 -20 25 -56q-5 -35 -35 -51l-128 -64q-13 -7 -29 -7q-17 0 -31 8l-690 387l-110 -66q-8 -4 -12 -5q14 -49 10 -97q-7 -77 -56 -147.5t-132 -123.5q-132 -84 -277 -84 q-136 0 -222 78q-90 84 -79 207q7 76 56 147t131 124q132 84 278 84q83 0 151 -31q9 13 22 22l122 73l-122 73q-13 9 -22 22q-68 -31 -151 -31q-146 0 -278 84q-82 53 -131 124t-56 147q-5 59 15.5 113t63.5 93q85 79 222 79q145 0 277 -84q83 -52 132 -123t56 -148 q4 -48 -10 -97q4 -1 12 -5l110 -66l690 387q14 8 31 8q16 0 29 -7l128 -64q30 -16 35 -51q3 -36 -25 -56zM579 836q46 42 21 108t-106 117q-92 59 -192 59q-74 0 -113 -36q-46 -42 -21 -108t106 -117q92 -59 192 -59q74 0 113 36zM494 91q81 51 106 117t-21 108 q-39 36 -113 36q-100 0 -192 -59q-81 -51 -106 -117t21 -108q39 -36 113 -36q100 0 192 59zM672 704l96 -58v11q0 36 33 56l14 8l-79 47l-26 -26q-3 -3 -10 -11t-12 -12q-2 -2 -4 -3.5t-3 -2.5zM896 480l96 -32l736 576l-128 64l-768 -431v-113l-160 -96l9 -8q2 -2 7 -6 q4 -4 11 -12t11 -12l26 -26zM1600 64l128 64l-520 408l-177 -138q-2 -3 -13 -7z" /> +<glyph unicode="" horiz-adv-x="1792" d="M1696 1152q40 0 68 -28t28 -68v-1216q0 -40 -28 -68t-68 -28h-960q-40 0 -68 28t-28 68v288h-544q-40 0 -68 28t-28 68v672q0 40 20 88t48 76l408 408q28 28 76 48t88 20h416q40 0 68 -28t28 -68v-328q68 40 128 40h416zM1152 939l-299 -299h299v299zM512 1323l-299 -299 h299v299zM708 676l316 316v416h-384v-416q0 -40 -28 -68t-68 -28h-416v-640h512v256q0 40 20 88t48 76zM1664 -128v1152h-384v-416q0 -40 -28 -68t-68 -28h-416v-640h896z" /> +<glyph unicode="" horiz-adv-x="1408" d="M1404 151q0 -117 -79 -196t-196 -79q-135 0 -235 100l-777 776q-113 115 -113 271q0 159 110 270t269 111q158 0 273 -113l605 -606q10 -10 10 -22q0 -16 -30.5 -46.5t-46.5 -30.5q-13 0 -23 10l-606 607q-79 77 -181 77q-106 0 -179 -75t-73 -181q0 -105 76 -181 l776 -777q63 -63 145 -63q64 0 106 42t42 106q0 82 -63 145l-581 581q-26 24 -60 24q-29 0 -48 -19t-19 -48q0 -32 25 -59l410 -410q10 -10 10 -22q0 -16 -31 -47t-47 -31q-12 0 -22 10l-410 410q-63 61 -63 149q0 82 57 139t139 57q88 0 149 -63l581 -581q100 -98 100 -235 z" /> +<glyph unicode="" d="M384 0h768v384h-768v-384zM1280 0h128v896q0 14 -10 38.5t-20 34.5l-281 281q-10 10 -34 20t-39 10v-416q0 -40 -28 -68t-68 -28h-576q-40 0 -68 28t-28 68v416h-128v-1280h128v416q0 40 28 68t68 28h832q40 0 68 -28t28 -68v-416zM896 928v320q0 13 -9.5 22.5t-22.5 9.5 h-192q-13 0 -22.5 -9.5t-9.5 -22.5v-320q0 -13 9.5 -22.5t22.5 -9.5h192q13 0 22.5 9.5t9.5 22.5zM1536 896v-928q0 -40 -28 -68t-68 -28h-1344q-40 0 -68 28t-28 68v1344q0 40 28 68t68 28h928q40 0 88 -20t76 -48l280 -280q28 -28 48 -76t20 -88z" /> +<glyph unicode="" d="M1536 1120v-960q0 -119 -84.5 -203.5t-203.5 -84.5h-960q-119 0 -203.5 84.5t-84.5 203.5v960q0 119 84.5 203.5t203.5 84.5h960q119 0 203.5 -84.5t84.5 -203.5z" /> +<glyph unicode="" d="M1536 192v-128q0 -26 -19 -45t-45 -19h-1408q-26 0 -45 19t-19 45v128q0 26 19 45t45 19h1408q26 0 45 -19t19 -45zM1536 704v-128q0 -26 -19 -45t-45 -19h-1408q-26 0 -45 19t-19 45v128q0 26 19 45t45 19h1408q26 0 45 -19t19 -45zM1536 1216v-128q0 -26 -19 -45 t-45 -19h-1408q-26 0 -45 19t-19 45v128q0 26 19 45t45 19h1408q26 0 45 -19t19 -45z" /> +<glyph unicode="" horiz-adv-x="1792" d="M384 128q0 -80 -56 -136t-136 -56t-136 56t-56 136t56 136t136 56t136 -56t56 -136zM384 640q0 -80 -56 -136t-136 -56t-136 56t-56 136t56 136t136 56t136 -56t56 -136zM1792 224v-192q0 -13 -9.5 -22.5t-22.5 -9.5h-1216q-13 0 -22.5 9.5t-9.5 22.5v192q0 13 9.5 22.5 t22.5 9.5h1216q13 0 22.5 -9.5t9.5 -22.5zM384 1152q0 -80 -56 -136t-136 -56t-136 56t-56 136t56 136t136 56t136 -56t56 -136zM1792 736v-192q0 -13 -9.5 -22.5t-22.5 -9.5h-1216q-13 0 -22.5 9.5t-9.5 22.5v192q0 13 9.5 22.5t22.5 9.5h1216q13 0 22.5 -9.5t9.5 -22.5z M1792 1248v-192q0 -13 -9.5 -22.5t-22.5 -9.5h-1216q-13 0 -22.5 9.5t-9.5 22.5v192q0 13 9.5 22.5t22.5 9.5h1216q13 0 22.5 -9.5t9.5 -22.5z" /> +<glyph unicode="" horiz-adv-x="1792" d="M381 -84q0 -80 -54.5 -126t-135.5 -46q-106 0 -172 66l57 88q49 -45 106 -45q29 0 50.5 14.5t21.5 42.5q0 64 -105 56l-26 56q8 10 32.5 43.5t42.5 54t37 38.5v1q-16 0 -48.5 -1t-48.5 -1v-53h-106v152h333v-88l-95 -115q51 -12 81 -49t30 -88zM383 543v-159h-362 q-6 36 -6 54q0 51 23.5 93t56.5 68t66 47.5t56.5 43.5t23.5 45q0 25 -14.5 38.5t-39.5 13.5q-46 0 -81 -58l-85 59q24 51 71.5 79.5t105.5 28.5q73 0 123 -41.5t50 -112.5q0 -50 -34 -91.5t-75 -64.5t-75.5 -50.5t-35.5 -52.5h127v60h105zM1792 224v-192q0 -13 -9.5 -22.5 t-22.5 -9.5h-1216q-13 0 -22.5 9.5t-9.5 22.5v192q0 14 9 23t23 9h1216q13 0 22.5 -9.5t9.5 -22.5zM384 1123v-99h-335v99h107q0 41 0.5 122t0.5 121v12h-2q-8 -17 -50 -54l-71 76l136 127h106v-404h108zM1792 736v-192q0 -13 -9.5 -22.5t-22.5 -9.5h-1216q-13 0 -22.5 9.5 t-9.5 22.5v192q0 14 9 23t23 9h1216q13 0 22.5 -9.5t9.5 -22.5zM1792 1248v-192q0 -13 -9.5 -22.5t-22.5 -9.5h-1216q-13 0 -22.5 9.5t-9.5 22.5v192q0 13 9.5 22.5t22.5 9.5h1216q13 0 22.5 -9.5t9.5 -22.5z" /> +<glyph unicode="" horiz-adv-x="1792" d="M1760 640q14 0 23 -9t9 -23v-64q0 -14 -9 -23t-23 -9h-1728q-14 0 -23 9t-9 23v64q0 14 9 23t23 9h1728zM483 704q-28 35 -51 80q-48 97 -48 188q0 181 134 309q133 127 393 127q50 0 167 -19q66 -12 177 -48q10 -38 21 -118q14 -123 14 -183q0 -18 -5 -45l-12 -3l-84 6 l-14 2q-50 149 -103 205q-88 91 -210 91q-114 0 -182 -59q-67 -58 -67 -146q0 -73 66 -140t279 -129q69 -20 173 -66q58 -28 95 -52h-743zM990 448h411q7 -39 7 -92q0 -111 -41 -212q-23 -55 -71 -104q-37 -35 -109 -81q-80 -48 -153 -66q-80 -21 -203 -21q-114 0 -195 23 l-140 40q-57 16 -72 28q-8 8 -8 22v13q0 108 -2 156q-1 30 0 68l2 37v44l102 2q15 -34 30 -71t22.5 -56t12.5 -27q35 -57 80 -94q43 -36 105 -57q59 -22 132 -22q64 0 139 27q77 26 122 86q47 61 47 129q0 84 -81 157q-34 29 -137 71z" /> +<glyph unicode="" d="M48 1313q-37 2 -45 4l-3 88q13 1 40 1q60 0 112 -4q132 -7 166 -7q86 0 168 3q116 4 146 5q56 0 86 2l-1 -14l2 -64v-9q-60 -9 -124 -9q-60 0 -79 -25q-13 -14 -13 -132q0 -13 0.5 -32.5t0.5 -25.5l1 -229l14 -280q6 -124 51 -202q35 -59 96 -92q88 -47 177 -47 q104 0 191 28q56 18 99 51q48 36 65 64q36 56 53 114q21 73 21 229q0 79 -3.5 128t-11 122.5t-13.5 159.5l-4 59q-5 67 -24 88q-34 35 -77 34l-100 -2l-14 3l2 86h84l205 -10q76 -3 196 10l18 -2q6 -38 6 -51q0 -7 -4 -31q-45 -12 -84 -13q-73 -11 -79 -17q-15 -15 -15 -41 q0 -7 1.5 -27t1.5 -31q8 -19 22 -396q6 -195 -15 -304q-15 -76 -41 -122q-38 -65 -112 -123q-75 -57 -182 -89q-109 -33 -255 -33q-167 0 -284 46q-119 47 -179 122q-61 76 -83 195q-16 80 -16 237v333q0 188 -17 213q-25 36 -147 39zM1536 -96v64q0 14 -9 23t-23 9h-1472 q-14 0 -23 -9t-9 -23v-64q0 -14 9 -23t23 -9h1472q14 0 23 9t9 23z" /> +<glyph unicode="" horiz-adv-x="1664" d="M512 160v192q0 14 -9 23t-23 9h-320q-14 0 -23 -9t-9 -23v-192q0 -14 9 -23t23 -9h320q14 0 23 9t9 23zM512 544v192q0 14 -9 23t-23 9h-320q-14 0 -23 -9t-9 -23v-192q0 -14 9 -23t23 -9h320q14 0 23 9t9 23zM1024 160v192q0 14 -9 23t-23 9h-320q-14 0 -23 -9t-9 -23 v-192q0 -14 9 -23t23 -9h320q14 0 23 9t9 23zM512 928v192q0 14 -9 23t-23 9h-320q-14 0 -23 -9t-9 -23v-192q0 -14 9 -23t23 -9h320q14 0 23 9t9 23zM1024 544v192q0 14 -9 23t-23 9h-320q-14 0 -23 -9t-9 -23v-192q0 -14 9 -23t23 -9h320q14 0 23 9t9 23zM1536 160v192 q0 14 -9 23t-23 9h-320q-14 0 -23 -9t-9 -23v-192q0 -14 9 -23t23 -9h320q14 0 23 9t9 23zM1024 928v192q0 14 -9 23t-23 9h-320q-14 0 -23 -9t-9 -23v-192q0 -14 9 -23t23 -9h320q14 0 23 9t9 23zM1536 544v192q0 14 -9 23t-23 9h-320q-14 0 -23 -9t-9 -23v-192 q0 -14 9 -23t23 -9h320q14 0 23 9t9 23zM1536 928v192q0 14 -9 23t-23 9h-320q-14 0 -23 -9t-9 -23v-192q0 -14 9 -23t23 -9h320q14 0 23 9t9 23zM1664 1248v-1088q0 -66 -47 -113t-113 -47h-1344q-66 0 -113 47t-47 113v1088q0 66 47 113t113 47h1344q66 0 113 -47t47 -113 z" /> +<glyph unicode="" horiz-adv-x="1664" d="M1190 955l293 293l-107 107l-293 -293zM1637 1248q0 -27 -18 -45l-1286 -1286q-18 -18 -45 -18t-45 18l-198 198q-18 18 -18 45t18 45l1286 1286q18 18 45 18t45 -18l198 -198q18 -18 18 -45zM286 1438l98 -30l-98 -30l-30 -98l-30 98l-98 30l98 30l30 98zM636 1276 l196 -60l-196 -60l-60 -196l-60 196l-196 60l196 60l60 196zM1566 798l98 -30l-98 -30l-30 -98l-30 98l-98 30l98 30l30 98zM926 1438l98 -30l-98 -30l-30 -98l-30 98l-98 30l98 30l30 98z" /> +<glyph unicode="" horiz-adv-x="1792" d="M640 128q0 52 -38 90t-90 38t-90 -38t-38 -90t38 -90t90 -38t90 38t38 90zM256 640h384v256h-158q-13 0 -22 -9l-195 -195q-9 -9 -9 -22v-30zM1536 128q0 52 -38 90t-90 38t-90 -38t-38 -90t38 -90t90 -38t90 38t38 90zM1792 1216v-1024q0 -15 -4 -26.5t-13.5 -18.5 t-16.5 -11.5t-23.5 -6t-22.5 -2t-25.5 0t-22.5 0.5q0 -106 -75 -181t-181 -75t-181 75t-75 181h-384q0 -106 -75 -181t-181 -75t-181 75t-75 181h-64q-3 0 -22.5 -0.5t-25.5 0t-22.5 2t-23.5 6t-16.5 11.5t-13.5 18.5t-4 26.5q0 26 19 45t45 19v320q0 8 -0.5 35t0 38 t2.5 34.5t6.5 37t14 30.5t22.5 30l198 198q19 19 50.5 32t58.5 13h160v192q0 26 19 45t45 19h1024q26 0 45 -19t19 -45z" /> +<glyph unicode="" d="M1536 640q0 -209 -103 -385.5t-279.5 -279.5t-385.5 -103q-111 0 -218 32q59 93 78 164q9 34 54 211q20 -39 73 -67.5t114 -28.5q121 0 216 68.5t147 188.5t52 270q0 114 -59.5 214t-172.5 163t-255 63q-105 0 -196 -29t-154.5 -77t-109 -110.5t-67 -129.5t-21.5 -134 q0 -104 40 -183t117 -111q30 -12 38 20q2 7 8 31t8 30q6 23 -11 43q-51 61 -51 151q0 151 104.5 259.5t273.5 108.5q151 0 235.5 -82t84.5 -213q0 -170 -68.5 -289t-175.5 -119q-61 0 -98 43.5t-23 104.5q8 35 26.5 93.5t30 103t11.5 75.5q0 50 -27 83t-77 33 q-62 0 -105 -57t-43 -142q0 -73 25 -122l-99 -418q-17 -70 -13 -177q-206 91 -333 281t-127 423q0 209 103 385.5t279.5 279.5t385.5 103t385.5 -103t279.5 -279.5t103 -385.5z" /> +<glyph unicode="" d="M1248 1408q119 0 203.5 -84.5t84.5 -203.5v-960q0 -119 -84.5 -203.5t-203.5 -84.5h-725q85 122 108 210q9 34 53 209q21 -39 73.5 -67t112.5 -28q181 0 295.5 147.5t114.5 373.5q0 84 -35 162.5t-96.5 139t-152.5 97t-197 36.5q-104 0 -194.5 -28.5t-153 -76.5 t-107.5 -109.5t-66.5 -128t-21.5 -132.5q0 -102 39.5 -180t116.5 -110q13 -5 23.5 0t14.5 19q10 44 15 61q6 23 -11 42q-50 62 -50 150q0 150 103.5 256.5t270.5 106.5q149 0 232.5 -81t83.5 -210q0 -168 -67.5 -286t-173.5 -118q-60 0 -97 43.5t-23 103.5q8 34 26.5 92.5 t29.5 102t11 74.5q0 49 -26.5 81.5t-75.5 32.5q-61 0 -103.5 -56.5t-42.5 -139.5q0 -72 24 -121l-98 -414q-24 -100 -7 -254h-183q-119 0 -203.5 84.5t-84.5 203.5v960q0 119 84.5 203.5t203.5 84.5h960z" /> +<glyph unicode="" d="M678 -57q0 -38 -10 -71h-380q-95 0 -171.5 56.5t-103.5 147.5q24 45 69 77.5t100 49.5t107 24t107 7q32 0 49 -2q6 -4 30.5 -21t33 -23t31 -23t32 -25.5t27.5 -25.5t26.5 -29.5t21 -30.5t17.5 -34.5t9.5 -36t4.5 -40.5zM385 294q-234 -7 -385 -85v433q103 -118 273 -118 q32 0 70 5q-21 -61 -21 -86q0 -67 63 -149zM558 805q0 -100 -43.5 -160.5t-140.5 -60.5q-51 0 -97 26t-78 67.5t-56 93.5t-35.5 104t-11.5 99q0 96 51.5 165t144.5 69q66 0 119 -41t84 -104t47 -130t16 -128zM1536 896v-736q0 -119 -84.5 -203.5t-203.5 -84.5h-468 q39 73 39 157q0 66 -22 122.5t-55.5 93t-72 71t-72 59.5t-55.5 54.5t-22 59.5q0 36 23 68t56 61.5t65.5 64.5t55.5 93t23 131t-26.5 145.5t-75.5 118.5q-6 6 -14 11t-12.5 7.5t-10 9.5t-10.5 17h135l135 64h-437q-138 0 -244.5 -38.5t-182.5 -133.5q0 126 81 213t207 87h960 q119 0 203.5 -84.5t84.5 -203.5v-96h-256v256h-128v-256h-256v-128h256v-256h128v256h256z" /> +<glyph unicode="" horiz-adv-x="1664" d="M876 71q0 21 -4.5 40.5t-9.5 36t-17.5 34.5t-21 30.5t-26.5 29.5t-27.5 25.5t-32 25.5t-31 23t-33 23t-30.5 21q-17 2 -50 2q-54 0 -106 -7t-108 -25t-98 -46t-69 -75t-27 -107q0 -68 35.5 -121.5t93 -84t120.5 -45.5t127 -15q59 0 112.5 12.5t100.5 39t74.5 73.5 t27.5 110zM756 933q0 60 -16.5 127.5t-47 130.5t-84 104t-119.5 41q-93 0 -144 -69t-51 -165q0 -47 11.5 -99t35.5 -104t56 -93.5t78 -67.5t97 -26q97 0 140.5 60.5t43.5 160.5zM625 1408h437l-135 -79h-135q71 -45 110 -126t39 -169q0 -74 -23 -131.5t-56 -92.5t-66 -64.5 t-56 -61t-23 -67.5q0 -26 16.5 -51t43 -48t58.5 -48t64 -55.5t58.5 -66t43 -85t16.5 -106.5q0 -160 -140 -282q-152 -131 -420 -131q-59 0 -119.5 10t-122 33.5t-108.5 58t-77 89t-30 121.5q0 61 37 135q32 64 96 110.5t145 71t155 36t150 13.5q-64 83 -64 149q0 12 2 23.5 t5 19.5t8 21.5t7 21.5q-40 -5 -70 -5q-149 0 -255.5 98t-106.5 246q0 140 95 250.5t234 141.5q94 20 187 20zM1664 1152v-128h-256v-256h-128v256h-256v128h256v256h128v-256h256z" /> +<glyph unicode="" horiz-adv-x="1920" d="M768 384h384v96h-128v448h-114l-148 -137l77 -80q42 37 55 57h2v-288h-128v-96zM1280 640q0 -70 -21 -142t-59.5 -134t-101.5 -101t-138 -39t-138 39t-101.5 101t-59.5 134t-21 142t21 142t59.5 134t101.5 101t138 39t138 -39t101.5 -101t59.5 -134t21 -142zM1792 384 v512q-106 0 -181 75t-75 181h-1152q0 -106 -75 -181t-181 -75v-512q106 0 181 -75t75 -181h1152q0 106 75 181t181 75zM1920 1216v-1152q0 -26 -19 -45t-45 -19h-1792q-26 0 -45 19t-19 45v1152q0 26 19 45t45 19h1792q26 0 45 -19t19 -45z" /> +<glyph unicode="" horiz-adv-x="1024" d="M1024 832q0 -26 -19 -45l-448 -448q-19 -19 -45 -19t-45 19l-448 448q-19 19 -19 45t19 45t45 19h896q26 0 45 -19t19 -45z" /> +<glyph unicode="" horiz-adv-x="1024" d="M1024 320q0 -26 -19 -45t-45 -19h-896q-26 0 -45 19t-19 45t19 45l448 448q19 19 45 19t45 -19l448 -448q19 -19 19 -45z" /> +<glyph unicode="" horiz-adv-x="640" d="M640 1088v-896q0 -26 -19 -45t-45 -19t-45 19l-448 448q-19 19 -19 45t19 45l448 448q19 19 45 19t45 -19t19 -45z" /> +<glyph unicode="" horiz-adv-x="640" d="M576 640q0 -26 -19 -45l-448 -448q-19 -19 -45 -19t-45 19t-19 45v896q0 26 19 45t45 19t45 -19l448 -448q19 -19 19 -45z" /> +<glyph unicode="" horiz-adv-x="1664" d="M160 0h608v1152h-640v-1120q0 -13 9.5 -22.5t22.5 -9.5zM1536 32v1120h-640v-1152h608q13 0 22.5 9.5t9.5 22.5zM1664 1248v-1216q0 -66 -47 -113t-113 -47h-1344q-66 0 -113 47t-47 113v1216q0 66 47 113t113 47h1344q66 0 113 -47t47 -113z" /> +<glyph unicode="" horiz-adv-x="1024" d="M1024 448q0 -26 -19 -45l-448 -448q-19 -19 -45 -19t-45 19l-448 448q-19 19 -19 45t19 45t45 19h896q26 0 45 -19t19 -45zM1024 832q0 -26 -19 -45t-45 -19h-896q-26 0 -45 19t-19 45t19 45l448 448q19 19 45 19t45 -19l448 -448q19 -19 19 -45z" /> +<glyph unicode="" horiz-adv-x="1024" d="M1024 448q0 -26 -19 -45l-448 -448q-19 -19 -45 -19t-45 19l-448 448q-19 19 -19 45t19 45t45 19h896q26 0 45 -19t19 -45z" /> +<glyph unicode="" horiz-adv-x="1024" d="M1024 832q0 -26 -19 -45t-45 -19h-896q-26 0 -45 19t-19 45t19 45l448 448q19 19 45 19t45 -19l448 -448q19 -19 19 -45z" /> +<glyph unicode="" horiz-adv-x="1792" d="M1792 826v-794q0 -66 -47 -113t-113 -47h-1472q-66 0 -113 47t-47 113v794q44 -49 101 -87q362 -246 497 -345q57 -42 92.5 -65.5t94.5 -48t110 -24.5h1h1q51 0 110 24.5t94.5 48t92.5 65.5q170 123 498 345q57 39 100 87zM1792 1120q0 -79 -49 -151t-122 -123 q-376 -261 -468 -325q-10 -7 -42.5 -30.5t-54 -38t-52 -32.5t-57.5 -27t-50 -9h-1h-1q-23 0 -50 9t-57.5 27t-52 32.5t-54 38t-42.5 30.5q-91 64 -262 182.5t-205 142.5q-62 42 -117 115.5t-55 136.5q0 78 41.5 130t118.5 52h1472q65 0 112.5 -47t47.5 -113z" /> +<glyph unicode="" d="M349 911v-991h-330v991h330zM370 1217q1 -73 -50.5 -122t-135.5 -49h-2q-82 0 -132 49t-50 122q0 74 51.5 122.5t134.5 48.5t133 -48.5t51 -122.5zM1536 488v-568h-329v530q0 105 -40.5 164.5t-126.5 59.5q-63 0 -105.5 -34.5t-63.5 -85.5q-11 -30 -11 -81v-553h-329 q2 399 2 647t-1 296l-1 48h329v-144h-2q20 32 41 56t56.5 52t87 43.5t114.5 15.5q171 0 275 -113.5t104 -332.5z" /> +<glyph unicode="" d="M1536 640q0 -156 -61 -298t-164 -245t-245 -164t-298 -61q-172 0 -327 72.5t-264 204.5q-7 10 -6.5 22.5t8.5 20.5l137 138q10 9 25 9q16 -2 23 -12q73 -95 179 -147t225 -52q104 0 198.5 40.5t163.5 109.5t109.5 163.5t40.5 198.5t-40.5 198.5t-109.5 163.5 t-163.5 109.5t-198.5 40.5q-98 0 -188 -35.5t-160 -101.5l137 -138q31 -30 14 -69q-17 -40 -59 -40h-448q-26 0 -45 19t-19 45v448q0 42 40 59q39 17 69 -14l130 -129q107 101 244.5 156.5t284.5 55.5q156 0 298 -61t245 -164t164 -245t61 -298z" /> +<glyph unicode="" horiz-adv-x="1792" d="M1771 0q0 -53 -37 -90l-107 -108q-39 -37 -91 -37q-53 0 -90 37l-363 364q-38 36 -38 90q0 53 43 96l-256 256l-126 -126q-14 -14 -34 -14t-34 14q2 -2 12.5 -12t12.5 -13t10 -11.5t10 -13.5t6 -13.5t5.5 -16.5t1.5 -18q0 -38 -28 -68q-3 -3 -16.5 -18t-19 -20.5 t-18.5 -16.5t-22 -15.5t-22 -9t-26 -4.5q-40 0 -68 28l-408 408q-28 28 -28 68q0 13 4.5 26t9 22t15.5 22t16.5 18.5t20.5 19t18 16.5q30 28 68 28q10 0 18 -1.5t16.5 -5.5t13.5 -6t13.5 -10t11.5 -10t13 -12.5t12 -12.5q-14 14 -14 34t14 34l348 348q14 14 34 14t34 -14 q-2 2 -12.5 12t-12.5 13t-10 11.5t-10 13.5t-6 13.5t-5.5 16.5t-1.5 18q0 38 28 68q3 3 16.5 18t19 20.5t18.5 16.5t22 15.5t22 9t26 4.5q40 0 68 -28l408 -408q28 -28 28 -68q0 -13 -4.5 -26t-9 -22t-15.5 -22t-16.5 -18.5t-20.5 -19t-18 -16.5q-30 -28 -68 -28 q-10 0 -18 1.5t-16.5 5.5t-13.5 6t-13.5 10t-11.5 10t-13 12.5t-12 12.5q14 -14 14 -34t-14 -34l-126 -126l256 -256q43 43 96 43q52 0 91 -37l363 -363q37 -39 37 -91z" /> +<glyph unicode="" horiz-adv-x="1792" d="M384 384q0 53 -37.5 90.5t-90.5 37.5t-90.5 -37.5t-37.5 -90.5t37.5 -90.5t90.5 -37.5t90.5 37.5t37.5 90.5zM576 832q0 53 -37.5 90.5t-90.5 37.5t-90.5 -37.5t-37.5 -90.5t37.5 -90.5t90.5 -37.5t90.5 37.5t37.5 90.5zM1004 351l101 382q6 26 -7.5 48.5t-38.5 29.5 t-48 -6.5t-30 -39.5l-101 -382q-60 -5 -107 -43.5t-63 -98.5q-20 -77 20 -146t117 -89t146 20t89 117q16 60 -6 117t-72 91zM1664 384q0 53 -37.5 90.5t-90.5 37.5t-90.5 -37.5t-37.5 -90.5t37.5 -90.5t90.5 -37.5t90.5 37.5t37.5 90.5zM1024 1024q0 53 -37.5 90.5 t-90.5 37.5t-90.5 -37.5t-37.5 -90.5t37.5 -90.5t90.5 -37.5t90.5 37.5t37.5 90.5zM1472 832q0 53 -37.5 90.5t-90.5 37.5t-90.5 -37.5t-37.5 -90.5t37.5 -90.5t90.5 -37.5t90.5 37.5t37.5 90.5zM1792 384q0 -261 -141 -483q-19 -29 -54 -29h-1402q-35 0 -54 29 q-141 221 -141 483q0 182 71 348t191 286t286 191t348 71t348 -71t286 -191t191 -286t71 -348z" /> +<glyph unicode="" horiz-adv-x="1792" d="M896 1152q-204 0 -381.5 -69.5t-282 -187.5t-104.5 -255q0 -112 71.5 -213.5t201.5 -175.5l87 -50l-27 -96q-24 -91 -70 -172q152 63 275 171l43 38l57 -6q69 -8 130 -8q204 0 381.5 69.5t282 187.5t104.5 255t-104.5 255t-282 187.5t-381.5 69.5zM1792 640 q0 -174 -120 -321.5t-326 -233t-450 -85.5q-70 0 -145 8q-198 -175 -460 -242q-49 -14 -114 -22h-5q-15 0 -27 10.5t-16 27.5v1q-3 4 -0.5 12t2 10t4.5 9.5l6 9t7 8.5t8 9q7 8 31 34.5t34.5 38t31 39.5t32.5 51t27 59t26 76q-157 89 -247.5 220t-90.5 281q0 174 120 321.5 t326 233t450 85.5t450 -85.5t326 -233t120 -321.5z" /> +<glyph unicode="" horiz-adv-x="1792" d="M704 1152q-153 0 -286 -52t-211.5 -141t-78.5 -191q0 -82 53 -158t149 -132l97 -56l-35 -84q34 20 62 39l44 31l53 -10q78 -14 153 -14q153 0 286 52t211.5 141t78.5 191t-78.5 191t-211.5 141t-286 52zM704 1280q191 0 353.5 -68.5t256.5 -186.5t94 -257t-94 -257 t-256.5 -186.5t-353.5 -68.5q-86 0 -176 16q-124 -88 -278 -128q-36 -9 -86 -16h-3q-11 0 -20.5 8t-11.5 21q-1 3 -1 6.5t0.5 6.5t2 6l2.5 5t3.5 5.5t4 5t4.5 5t4 4.5q5 6 23 25t26 29.5t22.5 29t25 38.5t20.5 44q-124 72 -195 177t-71 224q0 139 94 257t256.5 186.5 t353.5 68.5zM1526 111q10 -24 20.5 -44t25 -38.5t22.5 -29t26 -29.5t23 -25q1 -1 4 -4.5t4.5 -5t4 -5t3.5 -5.5l2.5 -5t2 -6t0.5 -6.5t-1 -6.5q-3 -14 -13 -22t-22 -7q-50 7 -86 16q-154 40 -278 128q-90 -16 -176 -16q-271 0 -472 132q58 -4 88 -4q161 0 309 45t264 129 q125 92 192 212t67 254q0 77 -23 152q129 -71 204 -178t75 -230q0 -120 -71 -224.5t-195 -176.5z" /> +<glyph unicode="" horiz-adv-x="896" d="M885 970q18 -20 7 -44l-540 -1157q-13 -25 -42 -25q-4 0 -14 2q-17 5 -25.5 19t-4.5 30l197 808l-406 -101q-4 -1 -12 -1q-18 0 -31 11q-18 15 -13 39l201 825q4 14 16 23t28 9h328q19 0 32 -12.5t13 -29.5q0 -8 -5 -18l-171 -463l396 98q8 2 12 2q19 0 34 -15z" /> +<glyph unicode="" horiz-adv-x="1792" d="M1792 288v-320q0 -40 -28 -68t-68 -28h-320q-40 0 -68 28t-28 68v320q0 40 28 68t68 28h96v192h-512v-192h96q40 0 68 -28t28 -68v-320q0 -40 -28 -68t-68 -28h-320q-40 0 -68 28t-28 68v320q0 40 28 68t68 28h96v192h-512v-192h96q40 0 68 -28t28 -68v-320 q0 -40 -28 -68t-68 -28h-320q-40 0 -68 28t-28 68v320q0 40 28 68t68 28h96v192q0 52 38 90t90 38h512v192h-96q-40 0 -68 28t-28 68v320q0 40 28 68t68 28h320q40 0 68 -28t28 -68v-320q0 -40 -28 -68t-68 -28h-96v-192h512q52 0 90 -38t38 -90v-192h96q40 0 68 -28t28 -68 z" /> +<glyph unicode="" horiz-adv-x="1664" d="M896 708v-580q0 -104 -76 -180t-180 -76t-180 76t-76 180q0 26 19 45t45 19t45 -19t19 -45q0 -50 39 -89t89 -39t89 39t39 89v580q33 11 64 11t64 -11zM1664 681q0 -13 -9.5 -22.5t-22.5 -9.5q-11 0 -23 10q-49 46 -93 69t-102 23q-68 0 -128 -37t-103 -97 q-7 -10 -17.5 -28t-14.5 -24q-11 -17 -28 -17q-18 0 -29 17q-4 6 -14.5 24t-17.5 28q-43 60 -102.5 97t-127.5 37t-127.5 -37t-102.5 -97q-7 -10 -17.5 -28t-14.5 -24q-11 -17 -29 -17q-17 0 -28 17q-4 6 -14.5 24t-17.5 28q-43 60 -103 97t-128 37q-58 0 -102 -23t-93 -69 q-12 -10 -23 -10q-13 0 -22.5 9.5t-9.5 22.5q0 5 1 7q45 183 172.5 319.5t298 204.5t360.5 68q140 0 274.5 -40t246.5 -113.5t194.5 -187t115.5 -251.5q1 -2 1 -7zM896 1408v-98q-42 2 -64 2t-64 -2v98q0 26 19 45t45 19t45 -19t19 -45z" /> +<glyph unicode="" horiz-adv-x="1792" d="M768 -128h896v640h-416q-40 0 -68 28t-28 68v416h-384v-1152zM1024 1312v64q0 13 -9.5 22.5t-22.5 9.5h-704q-13 0 -22.5 -9.5t-9.5 -22.5v-64q0 -13 9.5 -22.5t22.5 -9.5h704q13 0 22.5 9.5t9.5 22.5zM1280 640h299l-299 299v-299zM1792 512v-672q0 -40 -28 -68t-68 -28 h-960q-40 0 -68 28t-28 68v160h-544q-40 0 -68 28t-28 68v1344q0 40 28 68t68 28h1088q40 0 68 -28t28 -68v-328q21 -13 36 -28l408 -408q28 -28 48 -76t20 -88z" /> +<glyph unicode="" horiz-adv-x="1024" d="M736 960q0 -13 -9.5 -22.5t-22.5 -9.5t-22.5 9.5t-9.5 22.5q0 46 -54 71t-106 25q-13 0 -22.5 9.5t-9.5 22.5t9.5 22.5t22.5 9.5q50 0 99.5 -16t87 -54t37.5 -90zM896 960q0 72 -34.5 134t-90 101.5t-123 62t-136.5 22.5t-136.5 -22.5t-123 -62t-90 -101.5t-34.5 -134 q0 -101 68 -180q10 -11 30.5 -33t30.5 -33q128 -153 141 -298h228q13 145 141 298q10 11 30.5 33t30.5 33q68 79 68 180zM1024 960q0 -155 -103 -268q-45 -49 -74.5 -87t-59.5 -95.5t-34 -107.5q47 -28 47 -82q0 -37 -25 -64q25 -27 25 -64q0 -52 -45 -81q13 -23 13 -47 q0 -46 -31.5 -71t-77.5 -25q-20 -44 -60 -70t-87 -26t-87 26t-60 70q-46 0 -77.5 25t-31.5 71q0 24 13 47q-45 29 -45 81q0 37 25 64q-25 27 -25 64q0 54 47 82q-4 50 -34 107.5t-59.5 95.5t-74.5 87q-103 113 -103 268q0 99 44.5 184.5t117 142t164 89t186.5 32.5 t186.5 -32.5t164 -89t117 -142t44.5 -184.5z" /> +<glyph unicode="" horiz-adv-x="1792" d="M1792 352v-192q0 -13 -9.5 -22.5t-22.5 -9.5h-1376v-192q0 -13 -9.5 -22.5t-22.5 -9.5q-12 0 -24 10l-319 320q-9 9 -9 22q0 14 9 23l320 320q9 9 23 9q13 0 22.5 -9.5t9.5 -22.5v-192h1376q13 0 22.5 -9.5t9.5 -22.5zM1792 896q0 -14 -9 -23l-320 -320q-9 -9 -23 -9 q-13 0 -22.5 9.5t-9.5 22.5v192h-1376q-13 0 -22.5 9.5t-9.5 22.5v192q0 13 9.5 22.5t22.5 9.5h1376v192q0 14 9 23t23 9q12 0 24 -10l319 -319q9 -9 9 -23z" /> +<glyph unicode="" horiz-adv-x="1920" d="M1280 608q0 14 -9 23t-23 9h-224v352q0 13 -9.5 22.5t-22.5 9.5h-192q-13 0 -22.5 -9.5t-9.5 -22.5v-352h-224q-13 0 -22.5 -9.5t-9.5 -22.5q0 -14 9 -23l352 -352q9 -9 23 -9t23 9l351 351q10 12 10 24zM1920 384q0 -159 -112.5 -271.5t-271.5 -112.5h-1088 q-185 0 -316.5 131.5t-131.5 316.5q0 130 70 240t188 165q-2 30 -2 43q0 212 150 362t362 150q156 0 285.5 -87t188.5 -231q71 62 166 62q106 0 181 -75t75 -181q0 -76 -41 -138q130 -31 213.5 -135.5t83.5 -238.5z" /> +<glyph unicode="" horiz-adv-x="1920" d="M1280 672q0 14 -9 23l-352 352q-9 9 -23 9t-23 -9l-351 -351q-10 -12 -10 -24q0 -14 9 -23t23 -9h224v-352q0 -13 9.5 -22.5t22.5 -9.5h192q13 0 22.5 9.5t9.5 22.5v352h224q13 0 22.5 9.5t9.5 22.5zM1920 384q0 -159 -112.5 -271.5t-271.5 -112.5h-1088 q-185 0 -316.5 131.5t-131.5 316.5q0 130 70 240t188 165q-2 30 -2 43q0 212 150 362t362 150q156 0 285.5 -87t188.5 -231q71 62 166 62q106 0 181 -75t75 -181q0 -76 -41 -138q130 -31 213.5 -135.5t83.5 -238.5z" /> +<glyph unicode="" horiz-adv-x="1408" d="M384 192q0 -26 -19 -45t-45 -19t-45 19t-19 45t19 45t45 19t45 -19t19 -45zM1408 131q0 -121 -73 -190t-194 -69h-874q-121 0 -194 69t-73 190q0 68 5.5 131t24 138t47.5 132.5t81 103t120 60.5q-22 -52 -22 -120v-203q-58 -20 -93 -70t-35 -111q0 -80 56 -136t136 -56 t136 56t56 136q0 61 -35.5 111t-92.5 70v203q0 62 25 93q132 -104 295 -104t295 104q25 -31 25 -93v-64q-106 0 -181 -75t-75 -181v-89q-32 -29 -32 -71q0 -40 28 -68t68 -28t68 28t28 68q0 42 -32 71v89q0 52 38 90t90 38t90 -38t38 -90v-89q-32 -29 -32 -71q0 -40 28 -68 t68 -28t68 28t28 68q0 42 -32 71v89q0 68 -34.5 127.5t-93.5 93.5q0 10 0.5 42.5t0 48t-2.5 41.5t-7 47t-13 40q68 -15 120 -60.5t81 -103t47.5 -132.5t24 -138t5.5 -131zM1088 1024q0 -159 -112.5 -271.5t-271.5 -112.5t-271.5 112.5t-112.5 271.5t112.5 271.5t271.5 112.5 t271.5 -112.5t112.5 -271.5z" /> +<glyph unicode="" horiz-adv-x="1408" d="M1280 832q0 26 -19 45t-45 19t-45 -19t-19 -45t19 -45t45 -19t45 19t19 45zM1408 832q0 -62 -35.5 -111t-92.5 -70v-395q0 -159 -131.5 -271.5t-316.5 -112.5t-316.5 112.5t-131.5 271.5v132q-164 20 -274 128t-110 252v512q0 26 19 45t45 19q6 0 16 -2q17 30 47 48 t65 18q53 0 90.5 -37.5t37.5 -90.5t-37.5 -90.5t-90.5 -37.5q-33 0 -64 18v-402q0 -106 94 -181t226 -75t226 75t94 181v402q-31 -18 -64 -18q-53 0 -90.5 37.5t-37.5 90.5t37.5 90.5t90.5 37.5q35 0 65 -18t47 -48q10 2 16 2q26 0 45 -19t19 -45v-512q0 -144 -110 -252 t-274 -128v-132q0 -106 94 -181t226 -75t226 75t94 181v395q-57 21 -92.5 70t-35.5 111q0 80 56 136t136 56t136 -56t56 -136z" /> +<glyph unicode="" horiz-adv-x="1792" d="M640 1152h512v128h-512v-128zM288 1152v-1280h-64q-92 0 -158 66t-66 158v832q0 92 66 158t158 66h64zM1408 1152v-1280h-1024v1280h128v160q0 40 28 68t68 28h576q40 0 68 -28t28 -68v-160h128zM1792 928v-832q0 -92 -66 -158t-158 -66h-64v1280h64q92 0 158 -66 t66 -158z" /> +<glyph unicode="" horiz-adv-x="1664" d="M848 -160q0 16 -16 16q-59 0 -101.5 42.5t-42.5 101.5q0 16 -16 16t-16 -16q0 -73 51.5 -124.5t124.5 -51.5q16 0 16 16zM1664 128q0 -52 -38 -90t-90 -38h-448q0 -106 -75 -181t-181 -75t-181 75t-75 181h-448q-52 0 -90 38t-38 90q190 161 287 397.5t97 498.5 q0 165 96 262t264 117q-8 18 -8 37q0 40 28 68t68 28t68 -28t28 -68q0 -19 -8 -37q168 -20 264 -117t96 -262q0 -262 97 -498.5t287 -397.5z" /> +<glyph unicode="" horiz-adv-x="1920" d="M1664 896q0 80 -56 136t-136 56h-64v-384h64q80 0 136 56t56 136zM0 128h1792q0 -106 -75 -181t-181 -75h-1280q-106 0 -181 75t-75 181zM1856 896q0 -159 -112.5 -271.5t-271.5 -112.5h-64v-32q0 -92 -66 -158t-158 -66h-704q-92 0 -158 66t-66 158v736q0 26 19 45 t45 19h1152q159 0 271.5 -112.5t112.5 -271.5z" /> +<glyph unicode="" horiz-adv-x="1408" d="M640 1472v-640q0 -61 -35.5 -111t-92.5 -70v-779q0 -52 -38 -90t-90 -38h-128q-52 0 -90 38t-38 90v779q-57 20 -92.5 70t-35.5 111v640q0 26 19 45t45 19t45 -19t19 -45v-416q0 -26 19 -45t45 -19t45 19t19 45v416q0 26 19 45t45 19t45 -19t19 -45v-416q0 -26 19 -45 t45 -19t45 19t19 45v416q0 26 19 45t45 19t45 -19t19 -45zM1408 1472v-1600q0 -52 -38 -90t-90 -38h-128q-52 0 -90 38t-38 90v512h-224q-13 0 -22.5 9.5t-9.5 22.5v800q0 132 94 226t226 94h256q26 0 45 -19t19 -45z" /> +<glyph unicode="" horiz-adv-x="1280" d="M1024 352v-64q0 -14 -9 -23t-23 -9h-704q-14 0 -23 9t-9 23v64q0 14 9 23t23 9h704q14 0 23 -9t9 -23zM1024 608v-64q0 -14 -9 -23t-23 -9h-704q-14 0 -23 9t-9 23v64q0 14 9 23t23 9h704q14 0 23 -9t9 -23zM128 0h1024v768h-416q-40 0 -68 28t-28 68v416h-512v-1280z M768 896h376q-10 29 -22 41l-313 313q-12 12 -41 22v-376zM1280 864v-896q0 -40 -28 -68t-68 -28h-1088q-40 0 -68 28t-28 68v1344q0 40 28 68t68 28h640q40 0 88 -20t76 -48l312 -312q28 -28 48 -76t20 -88z" /> +<glyph unicode="" horiz-adv-x="1408" d="M384 224v-64q0 -13 -9.5 -22.5t-22.5 -9.5h-64q-13 0 -22.5 9.5t-9.5 22.5v64q0 13 9.5 22.5t22.5 9.5h64q13 0 22.5 -9.5t9.5 -22.5zM384 480v-64q0 -13 -9.5 -22.5t-22.5 -9.5h-64q-13 0 -22.5 9.5t-9.5 22.5v64q0 13 9.5 22.5t22.5 9.5h64q13 0 22.5 -9.5t9.5 -22.5z M640 480v-64q0 -13 -9.5 -22.5t-22.5 -9.5h-64q-13 0 -22.5 9.5t-9.5 22.5v64q0 13 9.5 22.5t22.5 9.5h64q13 0 22.5 -9.5t9.5 -22.5zM384 736v-64q0 -13 -9.5 -22.5t-22.5 -9.5h-64q-13 0 -22.5 9.5t-9.5 22.5v64q0 13 9.5 22.5t22.5 9.5h64q13 0 22.5 -9.5t9.5 -22.5z M1152 224v-64q0 -13 -9.5 -22.5t-22.5 -9.5h-64q-13 0 -22.5 9.5t-9.5 22.5v64q0 13 9.5 22.5t22.5 9.5h64q13 0 22.5 -9.5t9.5 -22.5zM896 480v-64q0 -13 -9.5 -22.5t-22.5 -9.5h-64q-13 0 -22.5 9.5t-9.5 22.5v64q0 13 9.5 22.5t22.5 9.5h64q13 0 22.5 -9.5t9.5 -22.5z M640 736v-64q0 -13 -9.5 -22.5t-22.5 -9.5h-64q-13 0 -22.5 9.5t-9.5 22.5v64q0 13 9.5 22.5t22.5 9.5h64q13 0 22.5 -9.5t9.5 -22.5zM384 992v-64q0 -13 -9.5 -22.5t-22.5 -9.5h-64q-13 0 -22.5 9.5t-9.5 22.5v64q0 13 9.5 22.5t22.5 9.5h64q13 0 22.5 -9.5t9.5 -22.5z M1152 480v-64q0 -13 -9.5 -22.5t-22.5 -9.5h-64q-13 0 -22.5 9.5t-9.5 22.5v64q0 13 9.5 22.5t22.5 9.5h64q13 0 22.5 -9.5t9.5 -22.5zM896 736v-64q0 -13 -9.5 -22.5t-22.5 -9.5h-64q-13 0 -22.5 9.5t-9.5 22.5v64q0 13 9.5 22.5t22.5 9.5h64q13 0 22.5 -9.5t9.5 -22.5z M640 992v-64q0 -13 -9.5 -22.5t-22.5 -9.5h-64q-13 0 -22.5 9.5t-9.5 22.5v64q0 13 9.5 22.5t22.5 9.5h64q13 0 22.5 -9.5t9.5 -22.5zM384 1248v-64q0 -13 -9.5 -22.5t-22.5 -9.5h-64q-13 0 -22.5 9.5t-9.5 22.5v64q0 13 9.5 22.5t22.5 9.5h64q13 0 22.5 -9.5t9.5 -22.5z M1152 736v-64q0 -13 -9.5 -22.5t-22.5 -9.5h-64q-13 0 -22.5 9.5t-9.5 22.5v64q0 13 9.5 22.5t22.5 9.5h64q13 0 22.5 -9.5t9.5 -22.5zM896 992v-64q0 -13 -9.5 -22.5t-22.5 -9.5h-64q-13 0 -22.5 9.5t-9.5 22.5v64q0 13 9.5 22.5t22.5 9.5h64q13 0 22.5 -9.5t9.5 -22.5z M640 1248v-64q0 -13 -9.5 -22.5t-22.5 -9.5h-64q-13 0 -22.5 9.5t-9.5 22.5v64q0 13 9.5 22.5t22.5 9.5h64q13 0 22.5 -9.5t9.5 -22.5zM1152 992v-64q0 -13 -9.5 -22.5t-22.5 -9.5h-64q-13 0 -22.5 9.5t-9.5 22.5v64q0 13 9.5 22.5t22.5 9.5h64q13 0 22.5 -9.5t9.5 -22.5z M896 1248v-64q0 -13 -9.5 -22.5t-22.5 -9.5h-64q-13 0 -22.5 9.5t-9.5 22.5v64q0 13 9.5 22.5t22.5 9.5h64q13 0 22.5 -9.5t9.5 -22.5zM1152 1248v-64q0 -13 -9.5 -22.5t-22.5 -9.5h-64q-13 0 -22.5 9.5t-9.5 22.5v64q0 13 9.5 22.5t22.5 9.5h64q13 0 22.5 -9.5t9.5 -22.5z M896 -128h384v1536h-1152v-1536h384v224q0 13 9.5 22.5t22.5 9.5h320q13 0 22.5 -9.5t9.5 -22.5v-224zM1408 1472v-1664q0 -26 -19 -45t-45 -19h-1280q-26 0 -45 19t-19 45v1664q0 26 19 45t45 19h1280q26 0 45 -19t19 -45z" /> +<glyph unicode="" horiz-adv-x="1408" d="M384 224v-64q0 -13 -9.5 -22.5t-22.5 -9.5h-64q-13 0 -22.5 9.5t-9.5 22.5v64q0 13 9.5 22.5t22.5 9.5h64q13 0 22.5 -9.5t9.5 -22.5zM384 480v-64q0 -13 -9.5 -22.5t-22.5 -9.5h-64q-13 0 -22.5 9.5t-9.5 22.5v64q0 13 9.5 22.5t22.5 9.5h64q13 0 22.5 -9.5t9.5 -22.5z M640 480v-64q0 -13 -9.5 -22.5t-22.5 -9.5h-64q-13 0 -22.5 9.5t-9.5 22.5v64q0 13 9.5 22.5t22.5 9.5h64q13 0 22.5 -9.5t9.5 -22.5zM384 736v-64q0 -13 -9.5 -22.5t-22.5 -9.5h-64q-13 0 -22.5 9.5t-9.5 22.5v64q0 13 9.5 22.5t22.5 9.5h64q13 0 22.5 -9.5t9.5 -22.5z M1152 224v-64q0 -13 -9.5 -22.5t-22.5 -9.5h-64q-13 0 -22.5 9.5t-9.5 22.5v64q0 13 9.5 22.5t22.5 9.5h64q13 0 22.5 -9.5t9.5 -22.5zM896 480v-64q0 -13 -9.5 -22.5t-22.5 -9.5h-64q-13 0 -22.5 9.5t-9.5 22.5v64q0 13 9.5 22.5t22.5 9.5h64q13 0 22.5 -9.5t9.5 -22.5z M640 736v-64q0 -13 -9.5 -22.5t-22.5 -9.5h-64q-13 0 -22.5 9.5t-9.5 22.5v64q0 13 9.5 22.5t22.5 9.5h64q13 0 22.5 -9.5t9.5 -22.5zM1152 480v-64q0 -13 -9.5 -22.5t-22.5 -9.5h-64q-13 0 -22.5 9.5t-9.5 22.5v64q0 13 9.5 22.5t22.5 9.5h64q13 0 22.5 -9.5t9.5 -22.5z M896 736v-64q0 -13 -9.5 -22.5t-22.5 -9.5h-64q-13 0 -22.5 9.5t-9.5 22.5v64q0 13 9.5 22.5t22.5 9.5h64q13 0 22.5 -9.5t9.5 -22.5zM1152 736v-64q0 -13 -9.5 -22.5t-22.5 -9.5h-64q-13 0 -22.5 9.5t-9.5 22.5v64q0 13 9.5 22.5t22.5 9.5h64q13 0 22.5 -9.5t9.5 -22.5z M896 -128h384v1152h-256v-32q0 -40 -28 -68t-68 -28h-448q-40 0 -68 28t-28 68v32h-256v-1152h384v224q0 13 9.5 22.5t22.5 9.5h320q13 0 22.5 -9.5t9.5 -22.5v-224zM896 1056v320q0 13 -9.5 22.5t-22.5 9.5h-64q-13 0 -22.5 -9.5t-9.5 -22.5v-96h-128v96q0 13 -9.5 22.5 t-22.5 9.5h-64q-13 0 -22.5 -9.5t-9.5 -22.5v-320q0 -13 9.5 -22.5t22.5 -9.5h64q13 0 22.5 9.5t9.5 22.5v96h128v-96q0 -13 9.5 -22.5t22.5 -9.5h64q13 0 22.5 9.5t9.5 22.5zM1408 1088v-1280q0 -26 -19 -45t-45 -19h-1280q-26 0 -45 19t-19 45v1280q0 26 19 45t45 19h320 v288q0 40 28 68t68 28h448q40 0 68 -28t28 -68v-288h320q26 0 45 -19t19 -45z" /> +<glyph unicode="" horiz-adv-x="1920" d="M640 128q0 53 -37.5 90.5t-90.5 37.5t-90.5 -37.5t-37.5 -90.5t37.5 -90.5t90.5 -37.5t90.5 37.5t37.5 90.5zM256 640h384v256h-158q-14 -2 -22 -9l-195 -195q-7 -12 -9 -22v-30zM1536 128q0 53 -37.5 90.5t-90.5 37.5t-90.5 -37.5t-37.5 -90.5t37.5 -90.5t90.5 -37.5 t90.5 37.5t37.5 90.5zM1664 800v192q0 14 -9 23t-23 9h-224v224q0 14 -9 23t-23 9h-192q-14 0 -23 -9t-9 -23v-224h-224q-14 0 -23 -9t-9 -23v-192q0 -14 9 -23t23 -9h224v-224q0 -14 9 -23t23 -9h192q14 0 23 9t9 23v224h224q14 0 23 9t9 23zM1920 1344v-1152 q0 -26 -19 -45t-45 -19h-192q0 -106 -75 -181t-181 -75t-181 75t-75 181h-384q0 -106 -75 -181t-181 -75t-181 75t-75 181h-128q-26 0 -45 19t-19 45t19 45t45 19v416q0 26 13 58t32 51l198 198q19 19 51 32t58 13h160v320q0 26 19 45t45 19h1152q26 0 45 -19t19 -45z" /> +<glyph unicode="" horiz-adv-x="1792" d="M1280 416v192q0 14 -9 23t-23 9h-224v224q0 14 -9 23t-23 9h-192q-14 0 -23 -9t-9 -23v-224h-224q-14 0 -23 -9t-9 -23v-192q0 -14 9 -23t23 -9h224v-224q0 -14 9 -23t23 -9h192q14 0 23 9t9 23v224h224q14 0 23 9t9 23zM640 1152h512v128h-512v-128zM256 1152v-1280h-32 q-92 0 -158 66t-66 158v832q0 92 66 158t158 66h32zM1440 1152v-1280h-1088v1280h160v160q0 40 28 68t68 28h576q40 0 68 -28t28 -68v-160h160zM1792 928v-832q0 -92 -66 -158t-158 -66h-32v1280h32q92 0 158 -66t66 -158z" /> +<glyph unicode="" horiz-adv-x="1920" d="M1920 576q-1 -32 -288 -96l-352 -32l-224 -64h-64l-293 -352h69q26 0 45 -4.5t19 -11.5t-19 -11.5t-45 -4.5h-96h-160h-64v32h64v416h-160l-192 -224h-96l-32 32v192h32v32h128v8l-192 24v128l192 24v8h-128v32h-32v192l32 32h96l192 -224h160v416h-64v32h64h160h96 q26 0 45 -4.5t19 -11.5t-19 -11.5t-45 -4.5h-69l293 -352h64l224 -64l352 -32q261 -58 287 -93z" /> +<glyph unicode="" horiz-adv-x="1664" d="M640 640v384h-256v-256q0 -53 37.5 -90.5t90.5 -37.5h128zM1664 192v-192h-1152v192l128 192h-128q-159 0 -271.5 112.5t-112.5 271.5v320l-64 64l32 128h480l32 128h960l32 -192l-64 -32v-800z" /> +<glyph unicode="" d="M1280 192v896q0 26 -19 45t-45 19h-128q-26 0 -45 -19t-19 -45v-320h-512v320q0 26 -19 45t-45 19h-128q-26 0 -45 -19t-19 -45v-896q0 -26 19 -45t45 -19h128q26 0 45 19t19 45v320h512v-320q0 -26 19 -45t45 -19h128q26 0 45 19t19 45zM1536 1120v-960 q0 -119 -84.5 -203.5t-203.5 -84.5h-960q-119 0 -203.5 84.5t-84.5 203.5v960q0 119 84.5 203.5t203.5 84.5h960q119 0 203.5 -84.5t84.5 -203.5z" /> +<glyph unicode="" d="M1280 576v128q0 26 -19 45t-45 19h-320v320q0 26 -19 45t-45 19h-128q-26 0 -45 -19t-19 -45v-320h-320q-26 0 -45 -19t-19 -45v-128q0 -26 19 -45t45 -19h320v-320q0 -26 19 -45t45 -19h128q26 0 45 19t19 45v320h320q26 0 45 19t19 45zM1536 1120v-960 q0 -119 -84.5 -203.5t-203.5 -84.5h-960q-119 0 -203.5 84.5t-84.5 203.5v960q0 119 84.5 203.5t203.5 84.5h960q119 0 203.5 -84.5t84.5 -203.5z" /> +<glyph unicode="" horiz-adv-x="1024" d="M627 160q0 -13 -10 -23l-50 -50q-10 -10 -23 -10t-23 10l-466 466q-10 10 -10 23t10 23l466 466q10 10 23 10t23 -10l50 -50q10 -10 10 -23t-10 -23l-393 -393l393 -393q10 -10 10 -23zM1011 160q0 -13 -10 -23l-50 -50q-10 -10 -23 -10t-23 10l-466 466q-10 10 -10 23 t10 23l466 466q10 10 23 10t23 -10l50 -50q10 -10 10 -23t-10 -23l-393 -393l393 -393q10 -10 10 -23z" /> +<glyph unicode="" horiz-adv-x="1024" d="M595 576q0 -13 -10 -23l-466 -466q-10 -10 -23 -10t-23 10l-50 50q-10 10 -10 23t10 23l393 393l-393 393q-10 10 -10 23t10 23l50 50q10 10 23 10t23 -10l466 -466q10 -10 10 -23zM979 576q0 -13 -10 -23l-466 -466q-10 -10 -23 -10t-23 10l-50 50q-10 10 -10 23t10 23 l393 393l-393 393q-10 10 -10 23t10 23l50 50q10 10 23 10t23 -10l466 -466q10 -10 10 -23z" /> +<glyph unicode="" horiz-adv-x="1152" d="M1075 224q0 -13 -10 -23l-50 -50q-10 -10 -23 -10t-23 10l-393 393l-393 -393q-10 -10 -23 -10t-23 10l-50 50q-10 10 -10 23t10 23l466 466q10 10 23 10t23 -10l466 -466q10 -10 10 -23zM1075 608q0 -13 -10 -23l-50 -50q-10 -10 -23 -10t-23 10l-393 393l-393 -393 q-10 -10 -23 -10t-23 10l-50 50q-10 10 -10 23t10 23l466 466q10 10 23 10t23 -10l466 -466q10 -10 10 -23z" /> +<glyph unicode="" horiz-adv-x="1152" d="M1075 672q0 -13 -10 -23l-466 -466q-10 -10 -23 -10t-23 10l-466 466q-10 10 -10 23t10 23l50 50q10 10 23 10t23 -10l393 -393l393 393q10 10 23 10t23 -10l50 -50q10 -10 10 -23zM1075 1056q0 -13 -10 -23l-466 -466q-10 -10 -23 -10t-23 10l-466 466q-10 10 -10 23 t10 23l50 50q10 10 23 10t23 -10l393 -393l393 393q10 10 23 10t23 -10l50 -50q10 -10 10 -23z" /> +<glyph unicode="" horiz-adv-x="640" d="M627 992q0 -13 -10 -23l-393 -393l393 -393q10 -10 10 -23t-10 -23l-50 -50q-10 -10 -23 -10t-23 10l-466 466q-10 10 -10 23t10 23l466 466q10 10 23 10t23 -10l50 -50q10 -10 10 -23z" /> +<glyph unicode="" horiz-adv-x="640" d="M595 576q0 -13 -10 -23l-466 -466q-10 -10 -23 -10t-23 10l-50 50q-10 10 -10 23t10 23l393 393l-393 393q-10 10 -10 23t10 23l50 50q10 10 23 10t23 -10l466 -466q10 -10 10 -23z" /> +<glyph unicode="" horiz-adv-x="1152" d="M1075 352q0 -13 -10 -23l-50 -50q-10 -10 -23 -10t-23 10l-393 393l-393 -393q-10 -10 -23 -10t-23 10l-50 50q-10 10 -10 23t10 23l466 466q10 10 23 10t23 -10l466 -466q10 -10 10 -23z" /> +<glyph unicode="" horiz-adv-x="1152" d="M1075 800q0 -13 -10 -23l-466 -466q-10 -10 -23 -10t-23 10l-466 466q-10 10 -10 23t10 23l50 50q10 10 23 10t23 -10l393 -393l393 393q10 10 23 10t23 -10l50 -50q10 -10 10 -23z" /> +<glyph unicode="" horiz-adv-x="1920" d="M1792 544v832q0 13 -9.5 22.5t-22.5 9.5h-1600q-13 0 -22.5 -9.5t-9.5 -22.5v-832q0 -13 9.5 -22.5t22.5 -9.5h1600q13 0 22.5 9.5t9.5 22.5zM1920 1376v-1088q0 -66 -47 -113t-113 -47h-544q0 -37 16 -77.5t32 -71t16 -43.5q0 -26 -19 -45t-45 -19h-512q-26 0 -45 19 t-19 45q0 14 16 44t32 70t16 78h-544q-66 0 -113 47t-47 113v1088q0 66 47 113t113 47h1600q66 0 113 -47t47 -113z" /> +<glyph unicode="" horiz-adv-x="1920" d="M416 256q-66 0 -113 47t-47 113v704q0 66 47 113t113 47h1088q66 0 113 -47t47 -113v-704q0 -66 -47 -113t-113 -47h-1088zM384 1120v-704q0 -13 9.5 -22.5t22.5 -9.5h1088q13 0 22.5 9.5t9.5 22.5v704q0 13 -9.5 22.5t-22.5 9.5h-1088q-13 0 -22.5 -9.5t-9.5 -22.5z M1760 192h160v-96q0 -40 -47 -68t-113 -28h-1600q-66 0 -113 28t-47 68v96h160h1600zM1040 96q16 0 16 16t-16 16h-160q-16 0 -16 -16t16 -16h160z" /> +<glyph unicode="" horiz-adv-x="1152" d="M640 128q0 26 -19 45t-45 19t-45 -19t-19 -45t19 -45t45 -19t45 19t19 45zM1024 288v960q0 13 -9.5 22.5t-22.5 9.5h-832q-13 0 -22.5 -9.5t-9.5 -22.5v-960q0 -13 9.5 -22.5t22.5 -9.5h832q13 0 22.5 9.5t9.5 22.5zM1152 1248v-1088q0 -66 -47 -113t-113 -47h-832 q-66 0 -113 47t-47 113v1088q0 66 47 113t113 47h832q66 0 113 -47t47 -113z" /> +<glyph unicode="" horiz-adv-x="768" d="M464 128q0 33 -23.5 56.5t-56.5 23.5t-56.5 -23.5t-23.5 -56.5t23.5 -56.5t56.5 -23.5t56.5 23.5t23.5 56.5zM672 288v704q0 13 -9.5 22.5t-22.5 9.5h-512q-13 0 -22.5 -9.5t-9.5 -22.5v-704q0 -13 9.5 -22.5t22.5 -9.5h512q13 0 22.5 9.5t9.5 22.5zM480 1136 q0 16 -16 16h-160q-16 0 -16 -16t16 -16h160q16 0 16 16zM768 1152v-1024q0 -52 -38 -90t-90 -38h-512q-52 0 -90 38t-38 90v1024q0 52 38 90t90 38h512q52 0 90 -38t38 -90z" /> +<glyph unicode="" d="M768 1184q-148 0 -273 -73t-198 -198t-73 -273t73 -273t198 -198t273 -73t273 73t198 198t73 273t-73 273t-198 198t-273 73zM1536 640q0 -209 -103 -385.5t-279.5 -279.5t-385.5 -103t-385.5 103t-279.5 279.5t-103 385.5t103 385.5t279.5 279.5t385.5 103t385.5 -103 t279.5 -279.5t103 -385.5z" /> +<glyph unicode="" horiz-adv-x="1664" d="M768 576v-384q0 -80 -56 -136t-136 -56h-384q-80 0 -136 56t-56 136v704q0 104 40.5 198.5t109.5 163.5t163.5 109.5t198.5 40.5h64q26 0 45 -19t19 -45v-128q0 -26 -19 -45t-45 -19h-64q-106 0 -181 -75t-75 -181v-32q0 -40 28 -68t68 -28h224q80 0 136 -56t56 -136z M1664 576v-384q0 -80 -56 -136t-136 -56h-384q-80 0 -136 56t-56 136v704q0 104 40.5 198.5t109.5 163.5t163.5 109.5t198.5 40.5h64q26 0 45 -19t19 -45v-128q0 -26 -19 -45t-45 -19h-64q-106 0 -181 -75t-75 -181v-32q0 -40 28 -68t68 -28h224q80 0 136 -56t56 -136z" /> +<glyph unicode="" horiz-adv-x="1664" d="M768 1216v-704q0 -104 -40.5 -198.5t-109.5 -163.5t-163.5 -109.5t-198.5 -40.5h-64q-26 0 -45 19t-19 45v128q0 26 19 45t45 19h64q106 0 181 75t75 181v32q0 40 -28 68t-68 28h-224q-80 0 -136 56t-56 136v384q0 80 56 136t136 56h384q80 0 136 -56t56 -136zM1664 1216 v-704q0 -104 -40.5 -198.5t-109.5 -163.5t-163.5 -109.5t-198.5 -40.5h-64q-26 0 -45 19t-19 45v128q0 26 19 45t45 19h64q106 0 181 75t75 181v32q0 40 -28 68t-68 28h-224q-80 0 -136 56t-56 136v384q0 80 56 136t136 56h384q80 0 136 -56t56 -136z" /> +<glyph unicode="" horiz-adv-x="1568" d="M496 192q0 -60 -42.5 -102t-101.5 -42q-60 0 -102 42t-42 102t42 102t102 42q59 0 101.5 -42t42.5 -102zM928 0q0 -53 -37.5 -90.5t-90.5 -37.5t-90.5 37.5t-37.5 90.5t37.5 90.5t90.5 37.5t90.5 -37.5t37.5 -90.5zM320 640q0 -66 -47 -113t-113 -47t-113 47t-47 113 t47 113t113 47t113 -47t47 -113zM1360 192q0 -46 -33 -79t-79 -33t-79 33t-33 79t33 79t79 33t79 -33t33 -79zM528 1088q0 -73 -51.5 -124.5t-124.5 -51.5t-124.5 51.5t-51.5 124.5t51.5 124.5t124.5 51.5t124.5 -51.5t51.5 -124.5zM992 1280q0 -80 -56 -136t-136 -56 t-136 56t-56 136t56 136t136 56t136 -56t56 -136zM1536 640q0 -40 -28 -68t-68 -28t-68 28t-28 68t28 68t68 28t68 -28t28 -68zM1328 1088q0 -33 -23.5 -56.5t-56.5 -23.5t-56.5 23.5t-23.5 56.5t23.5 56.5t56.5 23.5t56.5 -23.5t23.5 -56.5z" /> +<glyph unicode="" d="M1536 640q0 -209 -103 -385.5t-279.5 -279.5t-385.5 -103t-385.5 103t-279.5 279.5t-103 385.5t103 385.5t279.5 279.5t385.5 103t385.5 -103t279.5 -279.5t103 -385.5z" /> +<glyph unicode="" horiz-adv-x="1792" d="M1792 416q0 -166 -127 -451q-3 -7 -10.5 -24t-13.5 -30t-13 -22q-12 -17 -28 -17q-15 0 -23.5 10t-8.5 25q0 9 2.5 26.5t2.5 23.5q5 68 5 123q0 101 -17.5 181t-48.5 138.5t-80 101t-105.5 69.5t-133 42.5t-154 21.5t-175.5 6h-224v-256q0 -26 -19 -45t-45 -19t-45 19 l-512 512q-19 19 -19 45t19 45l512 512q19 19 45 19t45 -19t19 -45v-256h224q713 0 875 -403q53 -134 53 -333z" /> +<glyph unicode="" horiz-adv-x="1664" d="M640 320q0 -40 -12.5 -82t-43 -76t-72.5 -34t-72.5 34t-43 76t-12.5 82t12.5 82t43 76t72.5 34t72.5 -34t43 -76t12.5 -82zM1280 320q0 -40 -12.5 -82t-43 -76t-72.5 -34t-72.5 34t-43 76t-12.5 82t12.5 82t43 76t72.5 34t72.5 -34t43 -76t12.5 -82zM1440 320 q0 120 -69 204t-187 84q-41 0 -195 -21q-71 -11 -157 -11t-157 11q-152 21 -195 21q-118 0 -187 -84t-69 -204q0 -88 32 -153.5t81 -103t122 -60t140 -29.5t149 -7h168q82 0 149 7t140 29.5t122 60t81 103t32 153.5zM1664 496q0 -207 -61 -331q-38 -77 -105.5 -133t-141 -86 t-170 -47.5t-171.5 -22t-167 -4.5q-78 0 -142 3t-147.5 12.5t-152.5 30t-137 51.5t-121 81t-86 115q-62 123 -62 331q0 237 136 396q-27 82 -27 170q0 116 51 218q108 0 190 -39.5t189 -123.5q147 35 309 35q148 0 280 -32q105 82 187 121t189 39q51 -102 51 -218 q0 -87 -27 -168q136 -160 136 -398z" /> +<glyph unicode="" horiz-adv-x="1664" d="M1536 224v704q0 40 -28 68t-68 28h-704q-40 0 -68 28t-28 68v64q0 40 -28 68t-68 28h-320q-40 0 -68 -28t-28 -68v-960q0 -40 28 -68t68 -28h1216q40 0 68 28t28 68zM1664 928v-704q0 -92 -66 -158t-158 -66h-1216q-92 0 -158 66t-66 158v960q0 92 66 158t158 66h320 q92 0 158 -66t66 -158v-32h672q92 0 158 -66t66 -158z" /> +<glyph unicode="" horiz-adv-x="1920" d="M1781 605q0 35 -53 35h-1088q-40 0 -85.5 -21.5t-71.5 -52.5l-294 -363q-18 -24 -18 -40q0 -35 53 -35h1088q40 0 86 22t71 53l294 363q18 22 18 39zM640 768h768v160q0 40 -28 68t-68 28h-576q-40 0 -68 28t-28 68v64q0 40 -28 68t-68 28h-320q-40 0 -68 -28t-28 -68 v-853l256 315q44 53 116 87.5t140 34.5zM1909 605q0 -62 -46 -120l-295 -363q-43 -53 -116 -87.5t-140 -34.5h-1088q-92 0 -158 66t-66 158v960q0 92 66 158t158 66h320q92 0 158 -66t66 -158v-32h544q92 0 158 -66t66 -158v-160h192q54 0 99 -24.5t67 -70.5q15 -32 15 -68z " /> +<glyph unicode="" horiz-adv-x="1792" /> +<glyph unicode="" horiz-adv-x="1792" /> +<glyph unicode="" d="M1134 461q-37 -121 -138 -195t-228 -74t-228 74t-138 195q-8 25 4 48.5t38 31.5q25 8 48.5 -4t31.5 -38q25 -80 92.5 -129.5t151.5 -49.5t151.5 49.5t92.5 129.5q8 26 32 38t49 4t37 -31.5t4 -48.5zM640 896q0 -53 -37.5 -90.5t-90.5 -37.5t-90.5 37.5t-37.5 90.5 t37.5 90.5t90.5 37.5t90.5 -37.5t37.5 -90.5zM1152 896q0 -53 -37.5 -90.5t-90.5 -37.5t-90.5 37.5t-37.5 90.5t37.5 90.5t90.5 37.5t90.5 -37.5t37.5 -90.5zM1408 640q0 130 -51 248.5t-136.5 204t-204 136.5t-248.5 51t-248.5 -51t-204 -136.5t-136.5 -204t-51 -248.5 t51 -248.5t136.5 -204t204 -136.5t248.5 -51t248.5 51t204 136.5t136.5 204t51 248.5zM1536 640q0 -209 -103 -385.5t-279.5 -279.5t-385.5 -103t-385.5 103t-279.5 279.5t-103 385.5t103 385.5t279.5 279.5t385.5 103t385.5 -103t279.5 -279.5t103 -385.5z" /> +<glyph unicode="" d="M1134 307q8 -25 -4 -48.5t-37 -31.5t-49 4t-32 38q-25 80 -92.5 129.5t-151.5 49.5t-151.5 -49.5t-92.5 -129.5q-8 -26 -31.5 -38t-48.5 -4q-26 8 -38 31.5t-4 48.5q37 121 138 195t228 74t228 -74t138 -195zM640 896q0 -53 -37.5 -90.5t-90.5 -37.5t-90.5 37.5 t-37.5 90.5t37.5 90.5t90.5 37.5t90.5 -37.5t37.5 -90.5zM1152 896q0 -53 -37.5 -90.5t-90.5 -37.5t-90.5 37.5t-37.5 90.5t37.5 90.5t90.5 37.5t90.5 -37.5t37.5 -90.5zM1408 640q0 130 -51 248.5t-136.5 204t-204 136.5t-248.5 51t-248.5 -51t-204 -136.5t-136.5 -204 t-51 -248.5t51 -248.5t136.5 -204t204 -136.5t248.5 -51t248.5 51t204 136.5t136.5 204t51 248.5zM1536 640q0 -209 -103 -385.5t-279.5 -279.5t-385.5 -103t-385.5 103t-279.5 279.5t-103 385.5t103 385.5t279.5 279.5t385.5 103t385.5 -103t279.5 -279.5t103 -385.5z" /> +<glyph unicode="" d="M1152 448q0 -26 -19 -45t-45 -19h-640q-26 0 -45 19t-19 45t19 45t45 19h640q26 0 45 -19t19 -45zM640 896q0 -53 -37.5 -90.5t-90.5 -37.5t-90.5 37.5t-37.5 90.5t37.5 90.5t90.5 37.5t90.5 -37.5t37.5 -90.5zM1152 896q0 -53 -37.5 -90.5t-90.5 -37.5t-90.5 37.5 t-37.5 90.5t37.5 90.5t90.5 37.5t90.5 -37.5t37.5 -90.5zM1408 640q0 130 -51 248.5t-136.5 204t-204 136.5t-248.5 51t-248.5 -51t-204 -136.5t-136.5 -204t-51 -248.5t51 -248.5t136.5 -204t204 -136.5t248.5 -51t248.5 51t204 136.5t136.5 204t51 248.5zM1536 640 q0 -209 -103 -385.5t-279.5 -279.5t-385.5 -103t-385.5 103t-279.5 279.5t-103 385.5t103 385.5t279.5 279.5t385.5 103t385.5 -103t279.5 -279.5t103 -385.5z" /> +<glyph unicode="" horiz-adv-x="1920" d="M832 448v128q0 14 -9 23t-23 9h-192v192q0 14 -9 23t-23 9h-128q-14 0 -23 -9t-9 -23v-192h-192q-14 0 -23 -9t-9 -23v-128q0 -14 9 -23t23 -9h192v-192q0 -14 9 -23t23 -9h128q14 0 23 9t9 23v192h192q14 0 23 9t9 23zM1408 384q0 53 -37.5 90.5t-90.5 37.5t-90.5 -37.5 t-37.5 -90.5t37.5 -90.5t90.5 -37.5t90.5 37.5t37.5 90.5zM1664 640q0 53 -37.5 90.5t-90.5 37.5t-90.5 -37.5t-37.5 -90.5t37.5 -90.5t90.5 -37.5t90.5 37.5t37.5 90.5zM1920 512q0 -212 -150 -362t-362 -150q-192 0 -338 128h-220q-146 -128 -338 -128q-212 0 -362 150 t-150 362t150 362t362 150h896q212 0 362 -150t150 -362z" /> +<glyph unicode="" horiz-adv-x="1920" d="M384 368v-96q0 -16 -16 -16h-96q-16 0 -16 16v96q0 16 16 16h96q16 0 16 -16zM512 624v-96q0 -16 -16 -16h-224q-16 0 -16 16v96q0 16 16 16h224q16 0 16 -16zM384 880v-96q0 -16 -16 -16h-96q-16 0 -16 16v96q0 16 16 16h96q16 0 16 -16zM1408 368v-96q0 -16 -16 -16 h-864q-16 0 -16 16v96q0 16 16 16h864q16 0 16 -16zM768 624v-96q0 -16 -16 -16h-96q-16 0 -16 16v96q0 16 16 16h96q16 0 16 -16zM640 880v-96q0 -16 -16 -16h-96q-16 0 -16 16v96q0 16 16 16h96q16 0 16 -16zM1024 624v-96q0 -16 -16 -16h-96q-16 0 -16 16v96q0 16 16 16 h96q16 0 16 -16zM896 880v-96q0 -16 -16 -16h-96q-16 0 -16 16v96q0 16 16 16h96q16 0 16 -16zM1280 624v-96q0 -16 -16 -16h-96q-16 0 -16 16v96q0 16 16 16h96q16 0 16 -16zM1664 368v-96q0 -16 -16 -16h-96q-16 0 -16 16v96q0 16 16 16h96q16 0 16 -16zM1152 880v-96 q0 -16 -16 -16h-96q-16 0 -16 16v96q0 16 16 16h96q16 0 16 -16zM1408 880v-96q0 -16 -16 -16h-96q-16 0 -16 16v96q0 16 16 16h96q16 0 16 -16zM1664 880v-352q0 -16 -16 -16h-224q-16 0 -16 16v96q0 16 16 16h112v240q0 16 16 16h96q16 0 16 -16zM1792 128v896h-1664v-896 h1664zM1920 1024v-896q0 -53 -37.5 -90.5t-90.5 -37.5h-1664q-53 0 -90.5 37.5t-37.5 90.5v896q0 53 37.5 90.5t90.5 37.5h1664q53 0 90.5 -37.5t37.5 -90.5z" /> +<glyph unicode="" horiz-adv-x="1792" d="M1664 491v616q-169 -91 -306 -91q-82 0 -145 32q-100 49 -184 76.5t-178 27.5q-173 0 -403 -127v-599q245 113 433 113q55 0 103.5 -7.5t98 -26t77 -31t82.5 -39.5l28 -14q44 -22 101 -22q120 0 293 92zM320 1280q0 -35 -17.5 -64t-46.5 -46v-1266q0 -14 -9 -23t-23 -9 h-64q-14 0 -23 9t-9 23v1266q-29 17 -46.5 46t-17.5 64q0 53 37.5 90.5t90.5 37.5t90.5 -37.5t37.5 -90.5zM1792 1216v-763q0 -39 -35 -57q-10 -5 -17 -9q-218 -116 -369 -116q-88 0 -158 35l-28 14q-64 33 -99 48t-91 29t-114 14q-102 0 -235.5 -44t-228.5 -102 q-15 -9 -33 -9q-16 0 -32 8q-32 19 -32 56v742q0 35 31 55q35 21 78.5 42.5t114 52t152.5 49.5t155 19q112 0 209 -31t209 -86q38 -19 89 -19q122 0 310 112q22 12 31 17q31 16 62 -2q31 -20 31 -55z" /> +<glyph unicode="" horiz-adv-x="1792" d="M832 536v192q-181 -16 -384 -117v-185q205 96 384 110zM832 954v197q-172 -8 -384 -126v-189q215 111 384 118zM1664 491v184q-235 -116 -384 -71v224q-20 6 -39 15q-5 3 -33 17t-34.5 17t-31.5 15t-34.5 15.5t-32.5 13t-36 12.5t-35 8.5t-39.5 7.5t-39.5 4t-44 2 q-23 0 -49 -3v-222h19q102 0 192.5 -29t197.5 -82q19 -9 39 -15v-188q42 -17 91 -17q120 0 293 92zM1664 918v189q-169 -91 -306 -91q-45 0 -78 8v-196q148 -42 384 90zM320 1280q0 -35 -17.5 -64t-46.5 -46v-1266q0 -14 -9 -23t-23 -9h-64q-14 0 -23 9t-9 23v1266 q-29 17 -46.5 46t-17.5 64q0 53 37.5 90.5t90.5 37.5t90.5 -37.5t37.5 -90.5zM1792 1216v-763q0 -39 -35 -57q-10 -5 -17 -9q-218 -116 -369 -116q-88 0 -158 35l-28 14q-64 33 -99 48t-91 29t-114 14q-102 0 -235.5 -44t-228.5 -102q-15 -9 -33 -9q-16 0 -32 8 q-32 19 -32 56v742q0 35 31 55q35 21 78.5 42.5t114 52t152.5 49.5t155 19q112 0 209 -31t209 -86q38 -19 89 -19q122 0 310 112q22 12 31 17q31 16 62 -2q31 -20 31 -55z" /> +<glyph unicode="" horiz-adv-x="1664" d="M585 553l-466 -466q-10 -10 -23 -10t-23 10l-50 50q-10 10 -10 23t10 23l393 393l-393 393q-10 10 -10 23t10 23l50 50q10 10 23 10t23 -10l466 -466q10 -10 10 -23t-10 -23zM1664 96v-64q0 -14 -9 -23t-23 -9h-960q-14 0 -23 9t-9 23v64q0 14 9 23t23 9h960q14 0 23 -9 t9 -23z" /> +<glyph unicode="" horiz-adv-x="1920" d="M617 137l-50 -50q-10 -10 -23 -10t-23 10l-466 466q-10 10 -10 23t10 23l466 466q10 10 23 10t23 -10l50 -50q10 -10 10 -23t-10 -23l-393 -393l393 -393q10 -10 10 -23t-10 -23zM1208 1204l-373 -1291q-4 -13 -15.5 -19.5t-23.5 -2.5l-62 17q-13 4 -19.5 15.5t-2.5 24.5 l373 1291q4 13 15.5 19.5t23.5 2.5l62 -17q13 -4 19.5 -15.5t2.5 -24.5zM1865 553l-466 -466q-10 -10 -23 -10t-23 10l-50 50q-10 10 -10 23t10 23l393 393l-393 393q-10 10 -10 23t10 23l50 50q10 10 23 10t23 -10l466 -466q10 -10 10 -23t-10 -23z" /> +<glyph unicode="" horiz-adv-x="1792" d="M640 454v-70q0 -42 -39 -59q-13 -5 -25 -5q-27 0 -45 19l-512 512q-19 19 -19 45t19 45l512 512q29 31 70 14q39 -17 39 -59v-69l-397 -398q-19 -19 -19 -45t19 -45zM1792 416q0 -58 -17 -133.5t-38.5 -138t-48 -125t-40.5 -90.5l-20 -40q-8 -17 -28 -17q-6 0 -9 1 q-25 8 -23 34q43 400 -106 565q-64 71 -170.5 110.5t-267.5 52.5v-251q0 -42 -39 -59q-13 -5 -25 -5q-27 0 -45 19l-512 512q-19 19 -19 45t19 45l512 512q29 31 70 14q39 -17 39 -59v-262q411 -28 599 -221q169 -173 169 -509z" /> +<glyph unicode="" horiz-adv-x="1664" d="M1186 579l257 250l-356 52l-66 10l-30 60l-159 322v-963l59 -31l318 -168l-60 355l-12 66zM1638 841l-363 -354l86 -500q5 -33 -6 -51.5t-34 -18.5q-17 0 -40 12l-449 236l-449 -236q-23 -12 -40 -12q-23 0 -34 18.5t-6 51.5l86 500l-364 354q-32 32 -23 59.5t54 34.5 l502 73l225 455q20 41 49 41q28 0 49 -41l225 -455l502 -73q45 -7 54 -34.5t-24 -59.5z" /> +<glyph unicode="" horiz-adv-x="1408" d="M1401 1187l-640 -1280q-17 -35 -57 -35q-5 0 -15 2q-22 5 -35.5 22.5t-13.5 39.5v576h-576q-22 0 -39.5 13.5t-22.5 35.5t4 42t29 30l1280 640q13 7 29 7q27 0 45 -19q15 -14 18.5 -34.5t-6.5 -39.5z" /> +<glyph unicode="" horiz-adv-x="1664" d="M557 256h595v595zM512 301l595 595h-595v-595zM1664 224v-192q0 -14 -9 -23t-23 -9h-224v-224q0 -14 -9 -23t-23 -9h-192q-14 0 -23 9t-9 23v224h-864q-14 0 -23 9t-9 23v864h-224q-14 0 -23 9t-9 23v192q0 14 9 23t23 9h224v224q0 14 9 23t23 9h192q14 0 23 -9t9 -23 v-224h851l246 247q10 9 23 9t23 -9q9 -10 9 -23t-9 -23l-247 -246v-851h224q14 0 23 -9t9 -23z" /> +<glyph unicode="" horiz-adv-x="1024" d="M288 64q0 40 -28 68t-68 28t-68 -28t-28 -68t28 -68t68 -28t68 28t28 68zM288 1216q0 40 -28 68t-68 28t-68 -28t-28 -68t28 -68t68 -28t68 28t28 68zM928 1088q0 40 -28 68t-68 28t-68 -28t-28 -68t28 -68t68 -28t68 28t28 68zM1024 1088q0 -52 -26 -96.5t-70 -69.5 q-2 -287 -226 -414q-68 -38 -203 -81q-128 -40 -169.5 -71t-41.5 -100v-26q44 -25 70 -69.5t26 -96.5q0 -80 -56 -136t-136 -56t-136 56t-56 136q0 52 26 96.5t70 69.5v820q-44 25 -70 69.5t-26 96.5q0 80 56 136t136 56t136 -56t56 -136q0 -52 -26 -96.5t-70 -69.5v-497 q54 26 154 57q55 17 87.5 29.5t70.5 31t59 39.5t40.5 51t28 69.5t8.5 91.5q-44 25 -70 69.5t-26 96.5q0 80 56 136t136 56t136 -56t56 -136z" /> +<glyph unicode="" horiz-adv-x="1664" d="M439 265l-256 -256q-10 -9 -23 -9q-12 0 -23 9q-9 10 -9 23t9 23l256 256q10 9 23 9t23 -9q9 -10 9 -23t-9 -23zM608 224v-320q0 -14 -9 -23t-23 -9t-23 9t-9 23v320q0 14 9 23t23 9t23 -9t9 -23zM384 448q0 -14 -9 -23t-23 -9h-320q-14 0 -23 9t-9 23t9 23t23 9h320 q14 0 23 -9t9 -23zM1648 320q0 -120 -85 -203l-147 -146q-83 -83 -203 -83q-121 0 -204 85l-334 335q-21 21 -42 56l239 18l273 -274q27 -27 68 -27.5t68 26.5l147 146q28 28 28 67q0 40 -28 68l-274 275l18 239q35 -21 56 -42l336 -336q84 -86 84 -204zM1031 1044l-239 -18 l-273 274q-28 28 -68 28q-39 0 -68 -27l-147 -146q-28 -28 -28 -67q0 -40 28 -68l274 -274l-18 -240q-35 21 -56 42l-336 336q-84 86 -84 204q0 120 85 203l147 146q83 83 203 83q121 0 204 -85l334 -335q21 -21 42 -56zM1664 960q0 -14 -9 -23t-23 -9h-320q-14 0 -23 9 t-9 23t9 23t23 9h320q14 0 23 -9t9 -23zM1120 1504v-320q0 -14 -9 -23t-23 -9t-23 9t-9 23v320q0 14 9 23t23 9t23 -9t9 -23zM1527 1353l-256 -256q-11 -9 -23 -9t-23 9q-9 10 -9 23t9 23l256 256q10 9 23 9t23 -9q9 -10 9 -23t-9 -23z" /> +<glyph unicode="" horiz-adv-x="1024" d="M704 280v-240q0 -16 -12 -28t-28 -12h-240q-16 0 -28 12t-12 28v240q0 16 12 28t28 12h240q16 0 28 -12t12 -28zM1020 880q0 -54 -15.5 -101t-35 -76.5t-55 -59.5t-57.5 -43.5t-61 -35.5q-41 -23 -68.5 -65t-27.5 -67q0 -17 -12 -32.5t-28 -15.5h-240q-15 0 -25.5 18.5 t-10.5 37.5v45q0 83 65 156.5t143 108.5q59 27 84 56t25 76q0 42 -46.5 74t-107.5 32q-65 0 -108 -29q-35 -25 -107 -115q-13 -16 -31 -16q-12 0 -25 8l-164 125q-13 10 -15.5 25t5.5 28q160 266 464 266q80 0 161 -31t146 -83t106 -127.5t41 -158.5z" /> +<glyph unicode="" horiz-adv-x="640" d="M640 192v-128q0 -26 -19 -45t-45 -19h-512q-26 0 -45 19t-19 45v128q0 26 19 45t45 19h64v384h-64q-26 0 -45 19t-19 45v128q0 26 19 45t45 19h384q26 0 45 -19t19 -45v-576h64q26 0 45 -19t19 -45zM512 1344v-192q0 -26 -19 -45t-45 -19h-256q-26 0 -45 19t-19 45v192 q0 26 19 45t45 19h256q26 0 45 -19t19 -45z" /> +<glyph unicode="" horiz-adv-x="640" d="M512 288v-224q0 -26 -19 -45t-45 -19h-256q-26 0 -45 19t-19 45v224q0 26 19 45t45 19h256q26 0 45 -19t19 -45zM542 1344l-28 -768q-1 -26 -20.5 -45t-45.5 -19h-256q-26 0 -45.5 19t-20.5 45l-28 768q-1 26 17.5 45t44.5 19h320q26 0 44.5 -19t17.5 -45z" /> +<glyph unicode="" d="M897 167v-167h-248l-159 252l-24 42q-8 9 -11 21h-3l-9 -21q-10 -20 -25 -44l-155 -250h-258v167h128l197 291l-185 272h-137v168h276l139 -228q2 -4 23 -42q8 -9 11 -21h3q3 9 11 21l25 42l140 228h257v-168h-125l-184 -267l204 -296h109zM1534 846v-206h-514l-3 27 q-4 28 -4 46q0 64 26 117t65 86.5t84 65t84 54.5t65 54t26 64q0 38 -29.5 62.5t-70.5 24.5q-51 0 -97 -39q-14 -11 -36 -38l-105 92q26 37 63 66q83 65 188 65q110 0 178 -59.5t68 -158.5q0 -56 -24.5 -103t-62 -76.5t-81.5 -58.5t-82 -50.5t-65.5 -51.5t-30.5 -63h232v80 h126z" /> +<glyph unicode="" d="M897 167v-167h-248l-159 252l-24 42q-8 9 -11 21h-3l-9 -21q-10 -20 -25 -44l-155 -250h-258v167h128l197 291l-185 272h-137v168h276l139 -228q2 -4 23 -42q8 -9 11 -21h3q3 9 11 21l25 42l140 228h257v-168h-125l-184 -267l204 -296h109zM1536 -50v-206h-514l-4 27 q-3 45 -3 46q0 64 26 117t65 86.5t84 65t84 54.5t65 54t26 64q0 38 -29.5 62.5t-70.5 24.5q-51 0 -97 -39q-14 -11 -36 -38l-105 92q26 37 63 66q80 65 188 65q110 0 178 -59.5t68 -158.5q0 -66 -34.5 -118.5t-84 -86t-99.5 -62.5t-87 -63t-41 -73h232v80h126z" /> +<glyph unicode="" horiz-adv-x="1920" d="M896 128l336 384h-768l-336 -384h768zM1909 1205q15 -34 9.5 -71.5t-30.5 -65.5l-896 -1024q-38 -44 -96 -44h-768q-38 0 -69.5 20.5t-47.5 54.5q-15 34 -9.5 71.5t30.5 65.5l896 1024q38 44 96 44h768q38 0 69.5 -20.5t47.5 -54.5z" /> +<glyph unicode="" horiz-adv-x="1664" d="M1664 438q0 -81 -44.5 -135t-123.5 -54q-41 0 -77.5 17.5t-59 38t-56.5 38t-71 17.5q-110 0 -110 -124q0 -39 16 -115t15 -115v-5q-22 0 -33 -1q-34 -3 -97.5 -11.5t-115.5 -13.5t-98 -5q-61 0 -103 26.5t-42 83.5q0 37 17.5 71t38 56.5t38 59t17.5 77.5q0 79 -54 123.5 t-135 44.5q-84 0 -143 -45.5t-59 -127.5q0 -43 15 -83t33.5 -64.5t33.5 -53t15 -50.5q0 -45 -46 -89q-37 -35 -117 -35q-95 0 -245 24q-9 2 -27.5 4t-27.5 4l-13 2q-1 0 -3 1q-2 0 -2 1v1024q2 -1 17.5 -3.5t34 -5t21.5 -3.5q150 -24 245 -24q80 0 117 35q46 44 46 89 q0 22 -15 50.5t-33.5 53t-33.5 64.5t-15 83q0 82 59 127.5t144 45.5q80 0 134 -44.5t54 -123.5q0 -41 -17.5 -77.5t-38 -59t-38 -56.5t-17.5 -71q0 -57 42 -83.5t103 -26.5q64 0 180 15t163 17v-2q-1 -2 -3.5 -17.5t-5 -34t-3.5 -21.5q-24 -150 -24 -245q0 -80 35 -117 q44 -46 89 -46q22 0 50.5 15t53 33.5t64.5 33.5t83 15q82 0 127.5 -59t45.5 -143z" /> +<glyph unicode="" horiz-adv-x="1152" d="M1152 832v-128q0 -221 -147.5 -384.5t-364.5 -187.5v-132h256q26 0 45 -19t19 -45t-19 -45t-45 -19h-640q-26 0 -45 19t-19 45t19 45t45 19h256v132q-217 24 -364.5 187.5t-147.5 384.5v128q0 26 19 45t45 19t45 -19t19 -45v-128q0 -185 131.5 -316.5t316.5 -131.5 t316.5 131.5t131.5 316.5v128q0 26 19 45t45 19t45 -19t19 -45zM896 1216v-512q0 -132 -94 -226t-226 -94t-226 94t-94 226v512q0 132 94 226t226 94t226 -94t94 -226z" /> +<glyph unicode="" horiz-adv-x="1408" d="M271 591l-101 -101q-42 103 -42 214v128q0 26 19 45t45 19t45 -19t19 -45v-128q0 -53 15 -113zM1385 1193l-361 -361v-128q0 -132 -94 -226t-226 -94q-55 0 -109 19l-96 -96q97 -51 205 -51q185 0 316.5 131.5t131.5 316.5v128q0 26 19 45t45 19t45 -19t19 -45v-128 q0 -221 -147.5 -384.5t-364.5 -187.5v-132h256q26 0 45 -19t19 -45t-19 -45t-45 -19h-640q-26 0 -45 19t-19 45t19 45t45 19h256v132q-125 13 -235 81l-254 -254q-10 -10 -23 -10t-23 10l-82 82q-10 10 -10 23t10 23l1234 1234q10 10 23 10t23 -10l82 -82q10 -10 10 -23 t-10 -23zM1005 1325l-621 -621v512q0 132 94 226t226 94q102 0 184.5 -59t116.5 -152z" /> +<glyph unicode="" horiz-adv-x="1280" d="M1088 576v640h-448v-1137q119 63 213 137q235 184 235 360zM1280 1344v-768q0 -86 -33.5 -170.5t-83 -150t-118 -127.5t-126.5 -103t-121 -77.5t-89.5 -49.5t-42.5 -20q-12 -6 -26 -6t-26 6q-16 7 -42.5 20t-89.5 49.5t-121 77.5t-126.5 103t-118 127.5t-83 150 t-33.5 170.5v768q0 26 19 45t45 19h1152q26 0 45 -19t19 -45z" /> +<glyph unicode="" horiz-adv-x="1664" d="M128 -128h1408v1024h-1408v-1024zM512 1088v288q0 14 -9 23t-23 9h-64q-14 0 -23 -9t-9 -23v-288q0 -14 9 -23t23 -9h64q14 0 23 9t9 23zM1280 1088v288q0 14 -9 23t-23 9h-64q-14 0 -23 -9t-9 -23v-288q0 -14 9 -23t23 -9h64q14 0 23 9t9 23zM1664 1152v-1280 q0 -52 -38 -90t-90 -38h-1408q-52 0 -90 38t-38 90v1280q0 52 38 90t90 38h128v96q0 66 47 113t113 47h64q66 0 113 -47t47 -113v-96h384v96q0 66 47 113t113 47h64q66 0 113 -47t47 -113v-96h128q52 0 90 -38t38 -90z" /> +<glyph unicode="" horiz-adv-x="1408" d="M512 1344q0 26 -19 45t-45 19t-45 -19t-19 -45t19 -45t45 -19t45 19t19 45zM1408 1376v-320q0 -16 -12 -25q-8 -7 -20 -7q-4 0 -7 1l-448 96q-11 2 -18 11t-7 20h-256v-102q111 -23 183.5 -111t72.5 -203v-800q0 -26 -19 -45t-45 -19h-512q-26 0 -45 19t-19 45v800 q0 106 62.5 190.5t161.5 114.5v111h-32q-59 0 -115 -23.5t-91.5 -53t-66 -66.5t-40.5 -53.5t-14 -24.5q-17 -35 -57 -35q-16 0 -29 7q-23 12 -31.5 37t3.5 49q5 10 14.5 26t37.5 53.5t60.5 70t85 67t108.5 52.5q-25 42 -25 86q0 66 47 113t113 47t113 -47t47 -113 q0 -33 -14 -64h302q0 11 7 20t18 11l448 96q3 1 7 1q12 0 20 -7q12 -9 12 -25z" /> +<glyph unicode="" horiz-adv-x="1664" d="M1440 1088q0 40 -28 68t-68 28t-68 -28t-28 -68t28 -68t68 -28t68 28t28 68zM1664 1376q0 -249 -75.5 -430.5t-253.5 -360.5q-81 -80 -195 -176l-20 -379q-2 -16 -16 -26l-384 -224q-7 -4 -16 -4q-12 0 -23 9l-64 64q-13 14 -8 32l85 276l-281 281l-276 -85q-3 -1 -9 -1 q-14 0 -23 9l-64 64q-17 19 -5 39l224 384q10 14 26 16l379 20q96 114 176 195q188 187 358 258t431 71q14 0 24 -9.5t10 -22.5z" /> +<glyph unicode="" horiz-adv-x="1792" d="M1745 763l-164 -763h-334l178 832q13 56 -15 88q-27 33 -83 33h-169l-204 -953h-334l204 953h-286l-204 -953h-334l204 953l-153 327h1276q101 0 189.5 -40.5t147.5 -113.5q60 -73 81 -168.5t0 -194.5z" /> +<glyph unicode="" d="M909 141l102 102q19 19 19 45t-19 45l-307 307l307 307q19 19 19 45t-19 45l-102 102q-19 19 -45 19t-45 -19l-454 -454q-19 -19 -19 -45t19 -45l454 -454q19 -19 45 -19t45 19zM1536 640q0 -209 -103 -385.5t-279.5 -279.5t-385.5 -103t-385.5 103t-279.5 279.5 t-103 385.5t103 385.5t279.5 279.5t385.5 103t385.5 -103t279.5 -279.5t103 -385.5z" /> +<glyph unicode="" d="M717 141l454 454q19 19 19 45t-19 45l-454 454q-19 19 -45 19t-45 -19l-102 -102q-19 -19 -19 -45t19 -45l307 -307l-307 -307q-19 -19 -19 -45t19 -45l102 -102q19 -19 45 -19t45 19zM1536 640q0 -209 -103 -385.5t-279.5 -279.5t-385.5 -103t-385.5 103t-279.5 279.5 t-103 385.5t103 385.5t279.5 279.5t385.5 103t385.5 -103t279.5 -279.5t103 -385.5z" /> +<glyph unicode="" d="M1165 397l102 102q19 19 19 45t-19 45l-454 454q-19 19 -45 19t-45 -19l-454 -454q-19 -19 -19 -45t19 -45l102 -102q19 -19 45 -19t45 19l307 307l307 -307q19 -19 45 -19t45 19zM1536 640q0 -209 -103 -385.5t-279.5 -279.5t-385.5 -103t-385.5 103t-279.5 279.5 t-103 385.5t103 385.5t279.5 279.5t385.5 103t385.5 -103t279.5 -279.5t103 -385.5z" /> +<glyph unicode="" d="M813 237l454 454q19 19 19 45t-19 45l-102 102q-19 19 -45 19t-45 -19l-307 -307l-307 307q-19 19 -45 19t-45 -19l-102 -102q-19 -19 -19 -45t19 -45l454 -454q19 -19 45 -19t45 19zM1536 640q0 -209 -103 -385.5t-279.5 -279.5t-385.5 -103t-385.5 103t-279.5 279.5 t-103 385.5t103 385.5t279.5 279.5t385.5 103t385.5 -103t279.5 -279.5t103 -385.5z" /> +<glyph unicode="" horiz-adv-x="1408" d="M1130 939l16 175h-884l47 -534h612l-22 -228l-197 -53l-196 53l-13 140h-175l22 -278l362 -100h4v1l359 99l50 544h-644l-15 181h674zM0 1408h1408l-128 -1438l-578 -162l-574 162z" /> +<glyph unicode="" horiz-adv-x="1792" d="M275 1408h1505l-266 -1333l-804 -267l-698 267l71 356h297l-29 -147l422 -161l486 161l68 339h-1208l58 297h1209l38 191h-1208z" /> +<glyph unicode="" horiz-adv-x="1792" d="M960 1280q0 26 -19 45t-45 19t-45 -19t-19 -45t19 -45t45 -19t45 19t19 45zM1792 352v-352q0 -22 -20 -30q-8 -2 -12 -2q-13 0 -23 9l-93 93q-119 -143 -318.5 -226.5t-429.5 -83.5t-429.5 83.5t-318.5 226.5l-93 -93q-9 -9 -23 -9q-4 0 -12 2q-20 8 -20 30v352 q0 14 9 23t23 9h352q22 0 30 -20q8 -19 -7 -35l-100 -100q67 -91 189.5 -153.5t271.5 -82.5v647h-192q-26 0 -45 19t-19 45v128q0 26 19 45t45 19h192v163q-58 34 -93 92.5t-35 128.5q0 106 75 181t181 75t181 -75t75 -181q0 -70 -35 -128.5t-93 -92.5v-163h192q26 0 45 -19 t19 -45v-128q0 -26 -19 -45t-45 -19h-192v-647q149 20 271.5 82.5t189.5 153.5l-100 100q-15 16 -7 35q8 20 30 20h352q14 0 23 -9t9 -23z" /> +<glyph unicode="" horiz-adv-x="1152" d="M1056 768q40 0 68 -28t28 -68v-576q0 -40 -28 -68t-68 -28h-960q-40 0 -68 28t-28 68v576q0 40 28 68t68 28h32v320q0 185 131.5 316.5t316.5 131.5t316.5 -131.5t131.5 -316.5q0 -26 -19 -45t-45 -19h-64q-26 0 -45 19t-19 45q0 106 -75 181t-181 75t-181 -75t-75 -181 v-320h736z" /> +<glyph unicode="" d="M1024 640q0 -106 -75 -181t-181 -75t-181 75t-75 181t75 181t181 75t181 -75t75 -181zM1152 640q0 159 -112.5 271.5t-271.5 112.5t-271.5 -112.5t-112.5 -271.5t112.5 -271.5t271.5 -112.5t271.5 112.5t112.5 271.5zM1280 640q0 -212 -150 -362t-362 -150t-362 150 t-150 362t150 362t362 150t362 -150t150 -362zM1408 640q0 130 -51 248.5t-136.5 204t-204 136.5t-248.5 51t-248.5 -51t-204 -136.5t-136.5 -204t-51 -248.5t51 -248.5t136.5 -204t204 -136.5t248.5 -51t248.5 51t204 136.5t136.5 204t51 248.5zM1536 640 q0 -209 -103 -385.5t-279.5 -279.5t-385.5 -103t-385.5 103t-279.5 279.5t-103 385.5t103 385.5t279.5 279.5t385.5 103t385.5 -103t279.5 -279.5t103 -385.5z" /> +<glyph unicode="" horiz-adv-x="1408" d="M384 800v-192q0 -40 -28 -68t-68 -28h-192q-40 0 -68 28t-28 68v192q0 40 28 68t68 28h192q40 0 68 -28t28 -68zM896 800v-192q0 -40 -28 -68t-68 -28h-192q-40 0 -68 28t-28 68v192q0 40 28 68t68 28h192q40 0 68 -28t28 -68zM1408 800v-192q0 -40 -28 -68t-68 -28h-192 q-40 0 -68 28t-28 68v192q0 40 28 68t68 28h192q40 0 68 -28t28 -68z" /> +<glyph unicode="" horiz-adv-x="384" d="M384 288v-192q0 -40 -28 -68t-68 -28h-192q-40 0 -68 28t-28 68v192q0 40 28 68t68 28h192q40 0 68 -28t28 -68zM384 800v-192q0 -40 -28 -68t-68 -28h-192q-40 0 -68 28t-28 68v192q0 40 28 68t68 28h192q40 0 68 -28t28 -68zM384 1312v-192q0 -40 -28 -68t-68 -28h-192 q-40 0 -68 28t-28 68v192q0 40 28 68t68 28h192q40 0 68 -28t28 -68z" /> +<glyph unicode="" d="M512 256q0 53 -37.5 90.5t-90.5 37.5t-90.5 -37.5t-37.5 -90.5t37.5 -90.5t90.5 -37.5t90.5 37.5t37.5 90.5zM863 162q-13 232 -177 396t-396 177q-14 1 -24 -9t-10 -23v-128q0 -13 8.5 -22t21.5 -10q154 -11 264 -121t121 -264q1 -13 10 -21.5t22 -8.5h128q13 0 23 10 t9 24zM1247 161q-5 154 -56 297.5t-139.5 260t-205 205t-260 139.5t-297.5 56q-14 1 -23 -9q-10 -10 -10 -23v-128q0 -13 9 -22t22 -10q204 -7 378 -111.5t278.5 -278.5t111.5 -378q1 -13 10 -22t22 -9h128q13 0 23 10q11 9 9 23zM1536 1120v-960q0 -119 -84.5 -203.5 t-203.5 -84.5h-960q-119 0 -203.5 84.5t-84.5 203.5v960q0 119 84.5 203.5t203.5 84.5h960q119 0 203.5 -84.5t84.5 -203.5z" /> +<glyph unicode="" d="M768 1408q209 0 385.5 -103t279.5 -279.5t103 -385.5t-103 -385.5t-279.5 -279.5t-385.5 -103t-385.5 103t-279.5 279.5t-103 385.5t103 385.5t279.5 279.5t385.5 103zM1152 585q32 18 32 55t-32 55l-544 320q-31 19 -64 1q-32 -19 -32 -56v-640q0 -37 32 -56 q16 -8 32 -8q17 0 32 9z" /> +<glyph unicode="" horiz-adv-x="1792" d="M1024 1084l316 -316l-572 -572l-316 316zM813 105l618 618q19 19 19 45t-19 45l-362 362q-18 18 -45 18t-45 -18l-618 -618q-19 -19 -19 -45t19 -45l362 -362q18 -18 45 -18t45 18zM1702 742l-907 -908q-37 -37 -90.5 -37t-90.5 37l-126 126q56 56 56 136t-56 136 t-136 56t-136 -56l-125 126q-37 37 -37 90.5t37 90.5l907 906q37 37 90.5 37t90.5 -37l125 -125q-56 -56 -56 -136t56 -136t136 -56t136 56l126 -125q37 -37 37 -90.5t-37 -90.5z" /> +<glyph unicode="" d="M1280 576v128q0 26 -19 45t-45 19h-896q-26 0 -45 -19t-19 -45v-128q0 -26 19 -45t45 -19h896q26 0 45 19t19 45zM1536 1120v-960q0 -119 -84.5 -203.5t-203.5 -84.5h-960q-119 0 -203.5 84.5t-84.5 203.5v960q0 119 84.5 203.5t203.5 84.5h960q119 0 203.5 -84.5 t84.5 -203.5z" /> +<glyph unicode="" horiz-adv-x="1408" d="M1152 736v-64q0 -14 -9 -23t-23 -9h-832q-14 0 -23 9t-9 23v64q0 14 9 23t23 9h832q14 0 23 -9t9 -23zM1280 288v832q0 66 -47 113t-113 47h-832q-66 0 -113 -47t-47 -113v-832q0 -66 47 -113t113 -47h832q66 0 113 47t47 113zM1408 1120v-832q0 -119 -84.5 -203.5 t-203.5 -84.5h-832q-119 0 -203.5 84.5t-84.5 203.5v832q0 119 84.5 203.5t203.5 84.5h832q119 0 203.5 -84.5t84.5 -203.5z" /> +<glyph unicode="" horiz-adv-x="1024" d="M1018 933q-18 -37 -58 -37h-192v-864q0 -14 -9 -23t-23 -9h-704q-21 0 -29 18q-8 20 4 35l160 192q9 11 25 11h320v640h-192q-40 0 -58 37q-17 37 9 68l320 384q18 22 49 22t49 -22l320 -384q27 -32 9 -68z" /> +<glyph unicode="" horiz-adv-x="1024" d="M32 1280h704q13 0 22.5 -9.5t9.5 -23.5v-863h192q40 0 58 -37t-9 -69l-320 -384q-18 -22 -49 -22t-49 22l-320 384q-26 31 -9 69q18 37 58 37h192v640h-320q-14 0 -25 11l-160 192q-13 14 -4 34q9 19 29 19z" /> +<glyph unicode="" d="M685 237l614 614q19 19 19 45t-19 45l-102 102q-19 19 -45 19t-45 -19l-467 -467l-211 211q-19 19 -45 19t-45 -19l-102 -102q-19 -19 -19 -45t19 -45l358 -358q19 -19 45 -19t45 19zM1536 1120v-960q0 -119 -84.5 -203.5t-203.5 -84.5h-960q-119 0 -203.5 84.5 t-84.5 203.5v960q0 119 84.5 203.5t203.5 84.5h960q119 0 203.5 -84.5t84.5 -203.5z" /> +<glyph unicode="" d="M404 428l152 -152l-52 -52h-56v96h-96v56zM818 818q14 -13 -3 -30l-291 -291q-17 -17 -30 -3q-14 13 3 30l291 291q17 17 30 3zM544 128l544 544l-288 288l-544 -544v-288h288zM1152 736l92 92q28 28 28 68t-28 68l-152 152q-28 28 -68 28t-68 -28l-92 -92zM1536 1120 v-960q0 -119 -84.5 -203.5t-203.5 -84.5h-960q-119 0 -203.5 84.5t-84.5 203.5v960q0 119 84.5 203.5t203.5 84.5h960q119 0 203.5 -84.5t84.5 -203.5z" /> +<glyph unicode="" d="M1280 608v480q0 26 -19 45t-45 19h-480q-42 0 -59 -39q-17 -41 14 -70l144 -144l-534 -534q-19 -19 -19 -45t19 -45l102 -102q19 -19 45 -19t45 19l534 534l144 -144q18 -19 45 -19q12 0 25 5q39 17 39 59zM1536 1120v-960q0 -119 -84.5 -203.5t-203.5 -84.5h-960 q-119 0 -203.5 84.5t-84.5 203.5v960q0 119 84.5 203.5t203.5 84.5h960q119 0 203.5 -84.5t84.5 -203.5z" /> +<glyph unicode="" d="M1005 435l352 352q19 19 19 45t-19 45l-352 352q-30 31 -69 14q-40 -17 -40 -59v-160q-119 0 -216 -19.5t-162.5 -51t-114 -79t-76.5 -95.5t-44.5 -109t-21.5 -111.5t-5 -110.5q0 -181 167 -404q10 -12 25 -12q7 0 13 3q22 9 19 33q-44 354 62 473q46 52 130 75.5 t224 23.5v-160q0 -42 40 -59q12 -5 24 -5q26 0 45 19zM1536 1120v-960q0 -119 -84.5 -203.5t-203.5 -84.5h-960q-119 0 -203.5 84.5t-84.5 203.5v960q0 119 84.5 203.5t203.5 84.5h960q119 0 203.5 -84.5t84.5 -203.5z" /> +<glyph unicode="" d="M640 448l256 128l-256 128v-256zM1024 1039v-542l-512 -256v542zM1312 640q0 148 -73 273t-198 198t-273 73t-273 -73t-198 -198t-73 -273t73 -273t198 -198t273 -73t273 73t198 198t73 273zM1536 640q0 -209 -103 -385.5t-279.5 -279.5t-385.5 -103t-385.5 103 t-279.5 279.5t-103 385.5t103 385.5t279.5 279.5t385.5 103t385.5 -103t279.5 -279.5t103 -385.5z" /> +<glyph unicode="" d="M1145 861q18 -35 -5 -66l-320 -448q-19 -27 -52 -27t-52 27l-320 448q-23 31 -5 66q17 35 57 35h640q40 0 57 -35zM1280 160v960q0 13 -9.5 22.5t-22.5 9.5h-960q-13 0 -22.5 -9.5t-9.5 -22.5v-960q0 -13 9.5 -22.5t22.5 -9.5h960q13 0 22.5 9.5t9.5 22.5zM1536 1120 v-960q0 -119 -84.5 -203.5t-203.5 -84.5h-960q-119 0 -203.5 84.5t-84.5 203.5v960q0 119 84.5 203.5t203.5 84.5h960q119 0 203.5 -84.5t84.5 -203.5z" /> +<glyph unicode="" d="M1145 419q-17 -35 -57 -35h-640q-40 0 -57 35q-18 35 5 66l320 448q19 27 52 27t52 -27l320 -448q23 -31 5 -66zM1280 160v960q0 13 -9.5 22.5t-22.5 9.5h-960q-13 0 -22.5 -9.5t-9.5 -22.5v-960q0 -13 9.5 -22.5t22.5 -9.5h960q13 0 22.5 9.5t9.5 22.5zM1536 1120v-960 q0 -119 -84.5 -203.5t-203.5 -84.5h-960q-119 0 -203.5 84.5t-84.5 203.5v960q0 119 84.5 203.5t203.5 84.5h960q119 0 203.5 -84.5t84.5 -203.5z" /> +<glyph unicode="" d="M1088 640q0 -33 -27 -52l-448 -320q-31 -23 -66 -5q-35 17 -35 57v640q0 40 35 57q35 18 66 -5l448 -320q27 -19 27 -52zM1280 160v960q0 14 -9 23t-23 9h-960q-14 0 -23 -9t-9 -23v-960q0 -14 9 -23t23 -9h960q14 0 23 9t9 23zM1536 1120v-960q0 -119 -84.5 -203.5 t-203.5 -84.5h-960q-119 0 -203.5 84.5t-84.5 203.5v960q0 119 84.5 203.5t203.5 84.5h960q119 0 203.5 -84.5t84.5 -203.5z" /> +<glyph unicode="" horiz-adv-x="1024" d="M976 229l35 -159q3 -12 -3 -22.5t-17 -14.5l-5 -1q-4 -2 -10.5 -3.5t-16 -4.5t-21.5 -5.5t-25.5 -5t-30 -5t-33.5 -4.5t-36.5 -3t-38.5 -1q-234 0 -409 130.5t-238 351.5h-95q-13 0 -22.5 9.5t-9.5 22.5v113q0 13 9.5 22.5t22.5 9.5h66q-2 57 1 105h-67q-14 0 -23 9 t-9 23v114q0 14 9 23t23 9h98q67 210 243.5 338t400.5 128q102 0 194 -23q11 -3 20 -15q6 -11 3 -24l-43 -159q-3 -13 -14 -19.5t-24 -2.5l-4 1q-4 1 -11.5 2.5l-17.5 3.5t-22.5 3.5t-26 3t-29 2.5t-29.5 1q-126 0 -226 -64t-150 -176h468q16 0 25 -12q10 -12 7 -26 l-24 -114q-5 -26 -32 -26h-488q-3 -37 0 -105h459q15 0 25 -12q9 -12 6 -27l-24 -112q-2 -11 -11 -18.5t-20 -7.5h-387q48 -117 149.5 -185.5t228.5 -68.5q18 0 36 1.5t33.5 3.5t29.5 4.5t24.5 5t18.5 4.5l12 3l5 2q13 5 26 -2q12 -7 15 -21z" /> +<glyph unicode="" horiz-adv-x="1024" d="M1020 399v-367q0 -14 -9 -23t-23 -9h-956q-14 0 -23 9t-9 23v150q0 13 9.5 22.5t22.5 9.5h97v383h-95q-14 0 -23 9.5t-9 22.5v131q0 14 9 23t23 9h95v223q0 171 123.5 282t314.5 111q185 0 335 -125q9 -8 10 -20.5t-7 -22.5l-103 -127q-9 -11 -22 -12q-13 -2 -23 7 q-5 5 -26 19t-69 32t-93 18q-85 0 -137 -47t-52 -123v-215h305q13 0 22.5 -9t9.5 -23v-131q0 -13 -9.5 -22.5t-22.5 -9.5h-305v-379h414v181q0 13 9 22.5t23 9.5h162q14 0 23 -9.5t9 -22.5z" /> +<glyph unicode="" horiz-adv-x="1024" d="M978 351q0 -153 -99.5 -263.5t-258.5 -136.5v-175q0 -14 -9 -23t-23 -9h-135q-13 0 -22.5 9.5t-9.5 22.5v175q-66 9 -127.5 31t-101.5 44.5t-74 48t-46.5 37.5t-17.5 18q-17 21 -2 41l103 135q7 10 23 12q15 2 24 -9l2 -2q113 -99 243 -125q37 -8 74 -8q81 0 142.5 43 t61.5 122q0 28 -15 53t-33.5 42t-58.5 37.5t-66 32t-80 32.5q-39 16 -61.5 25t-61.5 26.5t-62.5 31t-56.5 35.5t-53.5 42.5t-43.5 49t-35.5 58t-21 66.5t-8.5 78q0 138 98 242t255 134v180q0 13 9.5 22.5t22.5 9.5h135q14 0 23 -9t9 -23v-176q57 -6 110.5 -23t87 -33.5 t63.5 -37.5t39 -29t15 -14q17 -18 5 -38l-81 -146q-8 -15 -23 -16q-14 -3 -27 7q-3 3 -14.5 12t-39 26.5t-58.5 32t-74.5 26t-85.5 11.5q-95 0 -155 -43t-60 -111q0 -26 8.5 -48t29.5 -41.5t39.5 -33t56 -31t60.5 -27t70 -27.5q53 -20 81 -31.5t76 -35t75.5 -42.5t62 -50 t53 -63.5t31.5 -76.5t13 -94z" /> +<glyph unicode="" horiz-adv-x="898" d="M898 1066v-102q0 -14 -9 -23t-23 -9h-168q-23 -144 -129 -234t-276 -110q167 -178 459 -536q14 -16 4 -34q-8 -18 -29 -18h-195q-16 0 -25 12q-306 367 -498 571q-9 9 -9 22v127q0 13 9.5 22.5t22.5 9.5h112q132 0 212.5 43t102.5 125h-427q-14 0 -23 9t-9 23v102 q0 14 9 23t23 9h413q-57 113 -268 113h-145q-13 0 -22.5 9.5t-9.5 22.5v133q0 14 9 23t23 9h832q14 0 23 -9t9 -23v-102q0 -14 -9 -23t-23 -9h-233q47 -61 64 -144h171q14 0 23 -9t9 -23z" /> +<glyph unicode="" horiz-adv-x="1027" d="M603 0h-172q-13 0 -22.5 9t-9.5 23v330h-288q-13 0 -22.5 9t-9.5 23v103q0 13 9.5 22.5t22.5 9.5h288v85h-288q-13 0 -22.5 9t-9.5 23v104q0 13 9.5 22.5t22.5 9.5h214l-321 578q-8 16 0 32q10 16 28 16h194q19 0 29 -18l215 -425q19 -38 56 -125q10 24 30.5 68t27.5 61 l191 420q8 19 29 19h191q17 0 27 -16q9 -14 1 -31l-313 -579h215q13 0 22.5 -9.5t9.5 -22.5v-104q0 -14 -9.5 -23t-22.5 -9h-290v-85h290q13 0 22.5 -9.5t9.5 -22.5v-103q0 -14 -9.5 -23t-22.5 -9h-290v-330q0 -13 -9.5 -22.5t-22.5 -9.5z" /> +<glyph unicode="" horiz-adv-x="1280" d="M1043 971q0 100 -65 162t-171 62h-320v-448h320q106 0 171 62t65 162zM1280 971q0 -193 -126.5 -315t-326.5 -122h-340v-118h505q14 0 23 -9t9 -23v-128q0 -14 -9 -23t-23 -9h-505v-192q0 -14 -9.5 -23t-22.5 -9h-167q-14 0 -23 9t-9 23v192h-224q-14 0 -23 9t-9 23v128 q0 14 9 23t23 9h224v118h-224q-14 0 -23 9t-9 23v149q0 13 9 22.5t23 9.5h224v629q0 14 9 23t23 9h539q200 0 326.5 -122t126.5 -315z" /> +<glyph unicode="" horiz-adv-x="1792" d="M514 341l81 299h-159l75 -300q1 -1 1 -3t1 -3q0 1 0.5 3.5t0.5 3.5zM630 768l35 128h-292l32 -128h225zM822 768h139l-35 128h-70zM1271 340l78 300h-162l81 -299q0 -1 0.5 -3.5t1.5 -3.5q0 1 0.5 3t0.5 3zM1382 768l33 128h-297l34 -128h230zM1792 736v-64q0 -14 -9 -23 t-23 -9h-213l-164 -616q-7 -24 -31 -24h-159q-24 0 -31 24l-166 616h-209l-167 -616q-7 -24 -31 -24h-159q-11 0 -19.5 7t-10.5 17l-160 616h-208q-14 0 -23 9t-9 23v64q0 14 9 23t23 9h175l-33 128h-142q-14 0 -23 9t-9 23v64q0 14 9 23t23 9h109l-89 344q-5 15 5 28 q10 12 26 12h137q26 0 31 -24l90 -360h359l97 360q7 24 31 24h126q24 0 31 -24l98 -360h365l93 360q5 24 31 24h137q16 0 26 -12q10 -13 5 -28l-91 -344h111q14 0 23 -9t9 -23v-64q0 -14 -9 -23t-23 -9h-145l-34 -128h179q14 0 23 -9t9 -23z" /> +<glyph unicode="" horiz-adv-x="1280" d="M1167 896q18 -182 -131 -258q117 -28 175 -103t45 -214q-7 -71 -32.5 -125t-64.5 -89t-97 -58.5t-121.5 -34.5t-145.5 -15v-255h-154v251q-80 0 -122 1v-252h-154v255q-18 0 -54 0.5t-55 0.5h-200l31 183h111q50 0 58 51v402h16q-6 1 -16 1v287q-13 68 -89 68h-111v164 l212 -1q64 0 97 1v252h154v-247q82 2 122 2v245h154v-252q79 -7 140 -22.5t113 -45t82.5 -78t36.5 -114.5zM952 351q0 36 -15 64t-37 46t-57.5 30.5t-65.5 18.5t-74 9t-69 3t-64.5 -1t-47.5 -1v-338q8 0 37 -0.5t48 -0.5t53 1.5t58.5 4t57 8.5t55.5 14t47.5 21t39.5 30 t24.5 40t9.5 51zM881 827q0 33 -12.5 58.5t-30.5 42t-48 28t-55 16.5t-61.5 8t-58 2.5t-54 -1t-39.5 -0.5v-307q5 0 34.5 -0.5t46.5 0t50 2t55 5.5t51.5 11t48.5 18.5t37 27t27 38.5t9 51z" /> +<glyph unicode="" horiz-adv-x="1280" d="M1280 768v-800q0 -40 -28 -68t-68 -28h-1088q-40 0 -68 28t-28 68v1344q0 40 28 68t68 28h544v-544q0 -40 28 -68t68 -28h544zM1277 896h-509v509q82 -15 132 -65l312 -312q50 -50 65 -132z" /> +<glyph unicode="" horiz-adv-x="1280" d="M1024 160v64q0 14 -9 23t-23 9h-704q-14 0 -23 -9t-9 -23v-64q0 -14 9 -23t23 -9h704q14 0 23 9t9 23zM1024 416v64q0 14 -9 23t-23 9h-704q-14 0 -23 -9t-9 -23v-64q0 -14 9 -23t23 -9h704q14 0 23 9t9 23zM1280 768v-800q0 -40 -28 -68t-68 -28h-1088q-40 0 -68 28 t-28 68v1344q0 40 28 68t68 28h544v-544q0 -40 28 -68t68 -28h544zM1277 896h-509v509q82 -15 132 -65l312 -312q50 -50 65 -132z" /> +<glyph unicode="" horiz-adv-x="1664" d="M1191 1128h177l-72 218l-12 47q-2 16 -2 20h-4l-3 -20q0 -1 -3.5 -18t-7.5 -29zM736 96q0 -12 -10 -24l-319 -319q-10 -9 -23 -9q-12 0 -23 9l-320 320q-15 16 -7 35q8 20 30 20h192v1376q0 14 9 23t23 9h192q14 0 23 -9t9 -23v-1376h192q14 0 23 -9t9 -23zM1572 -23 v-233h-584v90l369 529q12 18 21 27l11 9v3q-2 0 -6.5 -0.5t-7.5 -0.5q-12 -3 -30 -3h-232v-115h-120v229h567v-89l-369 -530q-6 -8 -21 -26l-11 -11v-2l14 2q9 2 30 2h248v119h121zM1661 874v-106h-288v106h75l-47 144h-243l-47 -144h75v-106h-287v106h70l230 662h162 l230 -662h70z" /> +<glyph unicode="" horiz-adv-x="1664" d="M1191 104h177l-72 218l-12 47q-2 16 -2 20h-4l-3 -20q0 -1 -3.5 -18t-7.5 -29zM736 96q0 -12 -10 -24l-319 -319q-10 -9 -23 -9q-12 0 -23 9l-320 320q-15 16 -7 35q8 20 30 20h192v1376q0 14 9 23t23 9h192q14 0 23 -9t9 -23v-1376h192q14 0 23 -9t9 -23zM1661 -150 v-106h-288v106h75l-47 144h-243l-47 -144h75v-106h-287v106h70l230 662h162l230 -662h70zM1572 1001v-233h-584v90l369 529q12 18 21 27l11 9v3q-2 0 -6.5 -0.5t-7.5 -0.5q-12 -3 -30 -3h-232v-115h-120v229h567v-89l-369 -530q-6 -8 -21 -26l-11 -10v-3l14 3q9 1 30 1h248 v119h121z" /> +<glyph unicode="" horiz-adv-x="1792" d="M736 96q0 -12 -10 -24l-319 -319q-10 -9 -23 -9q-12 0 -23 9l-320 320q-15 16 -7 35q8 20 30 20h192v1376q0 14 9 23t23 9h192q14 0 23 -9t9 -23v-1376h192q14 0 23 -9t9 -23zM1792 -32v-192q0 -14 -9 -23t-23 -9h-832q-14 0 -23 9t-9 23v192q0 14 9 23t23 9h832 q14 0 23 -9t9 -23zM1600 480v-192q0 -14 -9 -23t-23 -9h-640q-14 0 -23 9t-9 23v192q0 14 9 23t23 9h640q14 0 23 -9t9 -23zM1408 992v-192q0 -14 -9 -23t-23 -9h-448q-14 0 -23 9t-9 23v192q0 14 9 23t23 9h448q14 0 23 -9t9 -23zM1216 1504v-192q0 -14 -9 -23t-23 -9h-256 q-14 0 -23 9t-9 23v192q0 14 9 23t23 9h256q14 0 23 -9t9 -23z" /> +<glyph unicode="" horiz-adv-x="1792" d="M1216 -32v-192q0 -14 -9 -23t-23 -9h-256q-14 0 -23 9t-9 23v192q0 14 9 23t23 9h256q14 0 23 -9t9 -23zM736 96q0 -12 -10 -24l-319 -319q-10 -9 -23 -9q-12 0 -23 9l-320 320q-15 16 -7 35q8 20 30 20h192v1376q0 14 9 23t23 9h192q14 0 23 -9t9 -23v-1376h192 q14 0 23 -9t9 -23zM1408 480v-192q0 -14 -9 -23t-23 -9h-448q-14 0 -23 9t-9 23v192q0 14 9 23t23 9h448q14 0 23 -9t9 -23zM1600 992v-192q0 -14 -9 -23t-23 -9h-640q-14 0 -23 9t-9 23v192q0 14 9 23t23 9h640q14 0 23 -9t9 -23zM1792 1504v-192q0 -14 -9 -23t-23 -9h-832 q-14 0 -23 9t-9 23v192q0 14 9 23t23 9h832q14 0 23 -9t9 -23z" /> +<glyph unicode="" d="M1346 223q0 63 -44 116t-103 53q-52 0 -83 -37t-31 -94t36.5 -95t104.5 -38q50 0 85 27t35 68zM736 96q0 -12 -10 -24l-319 -319q-10 -9 -23 -9q-12 0 -23 9l-320 320q-15 16 -7 35q8 20 30 20h192v1376q0 14 9 23t23 9h192q14 0 23 -9t9 -23v-1376h192q14 0 23 -9t9 -23 zM1486 165q0 -62 -13 -121.5t-41 -114t-68 -95.5t-98.5 -65.5t-127.5 -24.5q-62 0 -108 16q-24 8 -42 15l39 113q15 -7 31 -11q37 -13 75 -13q84 0 134.5 58.5t66.5 145.5h-2q-21 -23 -61.5 -37t-84.5 -14q-106 0 -173 71.5t-67 172.5q0 105 72 178t181 73q123 0 205 -94.5 t82 -252.5zM1456 882v-114h-469v114h167v432q0 7 0.5 19t0.5 17v16h-2l-7 -12q-8 -13 -26 -31l-62 -58l-82 86l192 185h123v-654h165z" /> +<glyph unicode="" d="M1346 1247q0 63 -44 116t-103 53q-52 0 -83 -37t-31 -94t36.5 -95t104.5 -38q50 0 85 27t35 68zM736 96q0 -12 -10 -24l-319 -319q-10 -9 -23 -9q-12 0 -23 9l-320 320q-15 16 -7 35q8 20 30 20h192v1376q0 14 9 23t23 9h192q14 0 23 -9t9 -23v-1376h192q14 0 23 -9 t9 -23zM1456 -142v-114h-469v114h167v432q0 7 0.5 19t0.5 17v16h-2l-7 -12q-8 -13 -26 -31l-62 -58l-82 86l192 185h123v-654h165zM1486 1189q0 -62 -13 -121.5t-41 -114t-68 -95.5t-98.5 -65.5t-127.5 -24.5q-62 0 -108 16q-24 8 -42 15l39 113q15 -7 31 -11q37 -13 75 -13 q84 0 134.5 58.5t66.5 145.5h-2q-21 -23 -61.5 -37t-84.5 -14q-106 0 -173 71.5t-67 172.5q0 105 72 178t181 73q123 0 205 -94.5t82 -252.5z" /> +<glyph unicode="" horiz-adv-x="1664" d="M256 192q0 26 -19 45t-45 19q-27 0 -45.5 -19t-18.5 -45q0 -27 18.5 -45.5t45.5 -18.5q26 0 45 18.5t19 45.5zM416 704v-640q0 -26 -19 -45t-45 -19h-288q-26 0 -45 19t-19 45v640q0 26 19 45t45 19h288q26 0 45 -19t19 -45zM1600 704q0 -86 -55 -149q15 -44 15 -76 q3 -76 -43 -137q17 -56 0 -117q-15 -57 -54 -94q9 -112 -49 -181q-64 -76 -197 -78h-36h-76h-17q-66 0 -144 15.5t-121.5 29t-120.5 39.5q-123 43 -158 44q-26 1 -45 19.5t-19 44.5v641q0 25 18 43.5t43 20.5q24 2 76 59t101 121q68 87 101 120q18 18 31 48t17.5 48.5 t13.5 60.5q7 39 12.5 61t19.5 52t34 50q19 19 45 19q46 0 82.5 -10.5t60 -26t40 -40.5t24 -45t12 -50t5 -45t0.5 -39q0 -38 -9.5 -76t-19 -60t-27.5 -56q-3 -6 -10 -18t-11 -22t-8 -24h277q78 0 135 -57t57 -135z" /> +<glyph unicode="" horiz-adv-x="1664" d="M256 960q0 -26 -19 -45t-45 -19q-27 0 -45.5 19t-18.5 45q0 27 18.5 45.5t45.5 18.5q26 0 45 -18.5t19 -45.5zM416 448v640q0 26 -19 45t-45 19h-288q-26 0 -45 -19t-19 -45v-640q0 -26 19 -45t45 -19h288q26 0 45 19t19 45zM1545 597q55 -61 55 -149q-1 -78 -57.5 -135 t-134.5 -57h-277q4 -14 8 -24t11 -22t10 -18q18 -37 27 -57t19 -58.5t10 -76.5q0 -24 -0.5 -39t-5 -45t-12 -50t-24 -45t-40 -40.5t-60 -26t-82.5 -10.5q-26 0 -45 19q-20 20 -34 50t-19.5 52t-12.5 61q-9 42 -13.5 60.5t-17.5 48.5t-31 48q-33 33 -101 120q-49 64 -101 121 t-76 59q-25 2 -43 20.5t-18 43.5v641q0 26 19 44.5t45 19.5q35 1 158 44q77 26 120.5 39.5t121.5 29t144 15.5h17h76h36q133 -2 197 -78q58 -69 49 -181q39 -37 54 -94q17 -61 0 -117q46 -61 43 -137q0 -32 -15 -76z" /> +<glyph unicode="" d="M919 233v157q0 50 -29 50q-17 0 -33 -16v-224q16 -16 33 -16q29 0 29 49zM1103 355h66v34q0 51 -33 51t-33 -51v-34zM532 621v-70h-80v-423h-74v423h-78v70h232zM733 495v-367h-67v40q-39 -45 -76 -45q-33 0 -42 28q-6 16 -6 54v290h66v-270q0 -24 1 -26q1 -15 15 -15 q20 0 42 31v280h67zM985 384v-146q0 -52 -7 -73q-12 -42 -53 -42q-35 0 -68 41v-36h-67v493h67v-161q32 40 68 40q41 0 53 -42q7 -21 7 -74zM1236 255v-9q0 -29 -2 -43q-3 -22 -15 -40q-27 -40 -80 -40q-52 0 -81 38q-21 27 -21 86v129q0 59 20 86q29 38 80 38t78 -38 q21 -28 21 -86v-76h-133v-65q0 -51 34 -51q24 0 30 26q0 1 0.5 7t0.5 16.5v21.5h68zM785 1079v-156q0 -51 -32 -51t-32 51v156q0 52 32 52t32 -52zM1318 366q0 177 -19 260q-10 44 -43 73.5t-76 34.5q-136 15 -412 15q-275 0 -411 -15q-44 -5 -76.5 -34.5t-42.5 -73.5 q-20 -87 -20 -260q0 -176 20 -260q10 -43 42.5 -73t75.5 -35q137 -15 412 -15t412 15q43 5 75.5 35t42.5 73q20 84 20 260zM563 1017l90 296h-75l-51 -195l-53 195h-78l24 -69t23 -69q35 -103 46 -158v-201h74v201zM852 936v130q0 58 -21 87q-29 38 -78 38q-51 0 -78 -38 q-21 -29 -21 -87v-130q0 -58 21 -87q27 -38 78 -38q49 0 78 38q21 27 21 87zM1033 816h67v370h-67v-283q-22 -31 -42 -31q-15 0 -16 16q-1 2 -1 26v272h-67v-293q0 -37 6 -55q11 -27 43 -27q36 0 77 45v-40zM1536 1120v-960q0 -119 -84.5 -203.5t-203.5 -84.5h-960 q-119 0 -203.5 84.5t-84.5 203.5v960q0 119 84.5 203.5t203.5 84.5h960q119 0 203.5 -84.5t84.5 -203.5z" /> +<glyph unicode="" d="M971 292v-211q0 -67 -39 -67q-23 0 -45 22v301q22 22 45 22q39 0 39 -67zM1309 291v-46h-90v46q0 68 45 68t45 -68zM343 509h107v94h-312v-94h105v-569h100v569zM631 -60h89v494h-89v-378q-30 -42 -57 -42q-18 0 -21 21q-1 3 -1 35v364h-89v-391q0 -49 8 -73 q12 -37 58 -37q48 0 102 61v-54zM1060 88v197q0 73 -9 99q-17 56 -71 56q-50 0 -93 -54v217h-89v-663h89v48q45 -55 93 -55q54 0 71 55q9 27 9 100zM1398 98v13h-91q0 -51 -2 -61q-7 -36 -40 -36q-46 0 -46 69v87h179v103q0 79 -27 116q-39 51 -106 51q-68 0 -107 -51 q-28 -37 -28 -116v-173q0 -79 29 -116q39 -51 108 -51q72 0 108 53q18 27 21 54q2 9 2 58zM790 1011v210q0 69 -43 69t-43 -69v-210q0 -70 43 -70t43 70zM1509 260q0 -234 -26 -350q-14 -59 -58 -99t-102 -46q-184 -21 -555 -21t-555 21q-58 6 -102.5 46t-57.5 99 q-26 112 -26 350q0 234 26 350q14 59 58 99t103 47q183 20 554 20t555 -20q58 -7 102.5 -47t57.5 -99q26 -112 26 -350zM511 1536h102l-121 -399v-271h-100v271q-14 74 -61 212q-37 103 -65 187h106l71 -263zM881 1203v-175q0 -81 -28 -118q-37 -51 -106 -51q-67 0 -105 51 q-28 38 -28 118v175q0 80 28 117q38 51 105 51q69 0 106 -51q28 -37 28 -117zM1216 1365v-499h-91v55q-53 -62 -103 -62q-46 0 -59 37q-8 24 -8 75v394h91v-367q0 -33 1 -35q3 -22 21 -22q27 0 57 43v381h91z" /> +<glyph unicode="" horiz-adv-x="1408" d="M597 869q-10 -18 -257 -456q-27 -46 -65 -46h-239q-21 0 -31 17t0 36l253 448q1 0 0 1l-161 279q-12 22 -1 37q9 15 32 15h239q40 0 66 -45zM1403 1511q11 -16 0 -37l-528 -934v-1l336 -615q11 -20 1 -37q-10 -15 -32 -15h-239q-42 0 -66 45l-339 622q18 32 531 942 q25 45 64 45h241q22 0 31 -15z" /> +<glyph unicode="" d="M685 771q0 1 -126 222q-21 34 -52 34h-184q-18 0 -26 -11q-7 -12 1 -29l125 -216v-1l-196 -346q-9 -14 0 -28q8 -13 24 -13h185q31 0 50 36zM1309 1268q-7 12 -24 12h-187q-30 0 -49 -35l-411 -729q1 -2 262 -481q20 -35 52 -35h184q18 0 25 12q8 13 -1 28l-260 476v1 l409 723q8 16 0 28zM1536 1120v-960q0 -119 -84.5 -203.5t-203.5 -84.5h-960q-119 0 -203.5 84.5t-84.5 203.5v960q0 119 84.5 203.5t203.5 84.5h960q119 0 203.5 -84.5t84.5 -203.5z" /> +<glyph unicode="" horiz-adv-x="1792" d="M1280 640q0 37 -30 54l-512 320q-31 20 -65 2q-33 -18 -33 -56v-640q0 -38 33 -56q16 -8 31 -8q20 0 34 10l512 320q30 17 30 54zM1792 640q0 -96 -1 -150t-8.5 -136.5t-22.5 -147.5q-16 -73 -69 -123t-124 -58q-222 -25 -671 -25t-671 25q-71 8 -124.5 58t-69.5 123 q-14 65 -21.5 147.5t-8.5 136.5t-1 150t1 150t8.5 136.5t22.5 147.5q16 73 69 123t124 58q222 25 671 25t671 -25q71 -8 124.5 -58t69.5 -123q14 -65 21.5 -147.5t8.5 -136.5t1 -150z" /> +<glyph unicode="" horiz-adv-x="1792" d="M402 829l494 -305l-342 -285l-490 319zM1388 274v-108l-490 -293v-1l-1 1l-1 -1v1l-489 293v108l147 -96l342 284v2l1 -1l1 1v-2l343 -284zM554 1418l342 -285l-494 -304l-338 270zM1390 829l338 -271l-489 -319l-343 285zM1239 1418l489 -319l-338 -270l-494 304z" /> +<glyph unicode="" horiz-adv-x="1408" d="M928 135v-151l-707 -1v151zM1169 481v-701l-1 -35v-1h-1132l-35 1h-1v736h121v-618h928v618h120zM241 393l704 -65l-13 -150l-705 65zM309 709l683 -183l-39 -146l-683 183zM472 1058l609 -360l-77 -130l-609 360zM832 1389l398 -585l-124 -85l-399 584zM1285 1536 l121 -697l-149 -26l-121 697z" /> +<glyph unicode="" d="M1362 110v648h-135q20 -63 20 -131q0 -126 -64 -232.5t-174 -168.5t-240 -62q-197 0 -337 135.5t-140 327.5q0 68 20 131h-141v-648q0 -26 17.5 -43.5t43.5 -17.5h1069q25 0 43 17.5t18 43.5zM1078 643q0 124 -90.5 211.5t-218.5 87.5q-127 0 -217.5 -87.5t-90.5 -211.5 t90.5 -211.5t217.5 -87.5q128 0 218.5 87.5t90.5 211.5zM1362 1003v165q0 28 -20 48.5t-49 20.5h-174q-29 0 -49 -20.5t-20 -48.5v-165q0 -29 20 -49t49 -20h174q29 0 49 20t20 49zM1536 1211v-1142q0 -81 -58 -139t-139 -58h-1142q-81 0 -139 58t-58 139v1142q0 81 58 139 t139 58h1142q81 0 139 -58t58 -139z" /> +<glyph unicode="" d="M1248 1408q119 0 203.5 -84.5t84.5 -203.5v-960q0 -119 -84.5 -203.5t-203.5 -84.5h-960q-119 0 -203.5 84.5t-84.5 203.5v960q0 119 84.5 203.5t203.5 84.5h960zM698 640q0 88 -62 150t-150 62t-150 -62t-62 -150t62 -150t150 -62t150 62t62 150zM1262 640q0 88 -62 150 t-150 62t-150 -62t-62 -150t62 -150t150 -62t150 62t62 150z" /> +<glyph unicode="" d="M768 914l201 -306h-402zM1133 384h94l-459 691l-459 -691h94l104 160h522zM1536 640q0 -209 -103 -385.5t-279.5 -279.5t-385.5 -103t-385.5 103t-279.5 279.5t-103 385.5t103 385.5t279.5 279.5t385.5 103t385.5 -103t279.5 -279.5t103 -385.5z" /> +<glyph unicode="" horiz-adv-x="1408" d="M815 677q8 -63 -50.5 -101t-111.5 -6q-39 17 -53.5 58t-0.5 82t52 58q36 18 72.5 12t64 -35.5t27.5 -67.5zM926 698q-14 107 -113 164t-197 13q-63 -28 -100.5 -88.5t-34.5 -129.5q4 -91 77.5 -155t165.5 -56q91 8 152 84t50 168zM1165 1240q-20 27 -56 44.5t-58 22 t-71 12.5q-291 47 -566 -2q-43 -7 -66 -12t-55 -22t-50 -43q30 -28 76 -45.5t73.5 -22t87.5 -11.5q228 -29 448 -1q63 8 89.5 12t72.5 21.5t75 46.5zM1222 205q-8 -26 -15.5 -76.5t-14 -84t-28.5 -70t-58 -56.5q-86 -48 -189.5 -71.5t-202 -22t-201.5 18.5q-46 8 -81.5 18 t-76.5 27t-73 43.5t-52 61.5q-25 96 -57 292l6 16l18 9q223 -148 506.5 -148t507.5 148q21 -6 24 -23t-5 -45t-8 -37zM1403 1166q-26 -167 -111 -655q-5 -30 -27 -56t-43.5 -40t-54.5 -31q-252 -126 -610 -88q-248 27 -394 139q-15 12 -25.5 26.5t-17 35t-9 34t-6 39.5 t-5.5 35q-9 50 -26.5 150t-28 161.5t-23.5 147.5t-22 158q3 26 17.5 48.5t31.5 37.5t45 30t46 22.5t48 18.5q125 46 313 64q379 37 676 -50q155 -46 215 -122q16 -20 16.5 -51t-5.5 -54z" /> +<glyph unicode="" d="M848 666q0 43 -41 66t-77 1q-43 -20 -42.5 -72.5t43.5 -70.5q39 -23 81 4t36 72zM928 682q8 -66 -36 -121t-110 -61t-119 40t-56 113q-2 49 25.5 93t72.5 64q70 31 141.5 -10t81.5 -118zM1100 1073q-20 -21 -53.5 -34t-53 -16t-63.5 -8q-155 -20 -324 0q-44 6 -63 9.5 t-52.5 16t-54.5 32.5q13 19 36 31t40 15.5t47 8.5q198 35 408 1q33 -5 51 -8.5t43 -16t39 -31.5zM1142 327q0 7 5.5 26.5t3 32t-17.5 16.5q-161 -106 -365 -106t-366 106l-12 -6l-5 -12q26 -154 41 -210q47 -81 204 -108q249 -46 428 53q34 19 49 51.5t22.5 85.5t12.5 71z M1272 1020q9 53 -8 75q-43 55 -155 88q-216 63 -487 36q-132 -12 -226 -46q-38 -15 -59.5 -25t-47 -34t-29.5 -54q8 -68 19 -138t29 -171t24 -137q1 -5 5 -31t7 -36t12 -27t22 -28q105 -80 284 -100q259 -28 440 63q24 13 39.5 23t31 29t19.5 40q48 267 80 473zM1536 1120 v-960q0 -119 -84.5 -203.5t-203.5 -84.5h-960q-119 0 -203.5 84.5t-84.5 203.5v960q0 119 84.5 203.5t203.5 84.5h960q119 0 203.5 -84.5t84.5 -203.5z" /> +<glyph unicode="" horiz-adv-x="1024" d="M390 1408h219v-388h364v-241h-364v-394q0 -136 14 -172q13 -37 52 -60q50 -31 117 -31q117 0 232 76v-242q-102 -48 -178 -65q-77 -19 -173 -19q-105 0 -186 27q-78 25 -138 75q-58 51 -79 105q-22 54 -22 161v539h-170v217q91 30 155 84q64 55 103 132q39 78 54 196z " /> +<glyph unicode="" d="M1123 127v181q-88 -56 -174 -56q-51 0 -88 23q-29 17 -39 45q-11 30 -11 129v295h274v181h-274v291h-164q-11 -90 -40 -147t-78 -99q-48 -40 -116 -63v-163h127v-404q0 -78 17 -121q17 -42 59 -78q43 -37 104 -57q62 -20 140 -20q67 0 129 14q57 13 134 49zM1536 1120 v-960q0 -119 -84.5 -203.5t-203.5 -84.5h-960q-119 0 -203.5 84.5t-84.5 203.5v960q0 119 84.5 203.5t203.5 84.5h960q119 0 203.5 -84.5t84.5 -203.5z" /> +<glyph unicode="" horiz-adv-x="768" d="M765 237q8 -19 -5 -35l-350 -384q-10 -10 -23 -10q-14 0 -24 10l-355 384q-13 16 -5 35q9 19 29 19h224v1248q0 14 9 23t23 9h192q14 0 23 -9t9 -23v-1248h224q21 0 29 -19z" /> +<glyph unicode="" horiz-adv-x="768" d="M765 1043q-9 -19 -29 -19h-224v-1248q0 -14 -9 -23t-23 -9h-192q-14 0 -23 9t-9 23v1248h-224q-21 0 -29 19t5 35l350 384q10 10 23 10q14 0 24 -10l355 -384q13 -16 5 -35z" /> +<glyph unicode="" horiz-adv-x="1792" d="M1792 736v-192q0 -14 -9 -23t-23 -9h-1248v-224q0 -21 -19 -29t-35 5l-384 350q-10 10 -10 23q0 14 10 24l384 354q16 14 35 6q19 -9 19 -29v-224h1248q14 0 23 -9t9 -23z" /> +<glyph unicode="" horiz-adv-x="1792" d="M1728 643q0 -14 -10 -24l-384 -354q-16 -14 -35 -6q-19 9 -19 29v224h-1248q-14 0 -23 9t-9 23v192q0 14 9 23t23 9h1248v224q0 21 19 29t35 -5l384 -350q10 -10 10 -23z" /> +<glyph unicode="" horiz-adv-x="1408" d="M1393 321q-39 -125 -123 -250q-129 -196 -257 -196q-49 0 -140 32q-86 32 -151 32q-61 0 -142 -33q-81 -34 -132 -34q-152 0 -301 259q-147 261 -147 503q0 228 113 374q112 144 284 144q72 0 177 -30q104 -30 138 -30q45 0 143 34q102 34 173 34q119 0 213 -65 q52 -36 104 -100q-79 -67 -114 -118q-65 -94 -65 -207q0 -124 69 -223t158 -126zM1017 1494q0 -61 -29 -136q-30 -75 -93 -138q-54 -54 -108 -72q-37 -11 -104 -17q3 149 78 257q74 107 250 148q1 -3 2.5 -11t2.5 -11q0 -4 0.5 -10t0.5 -10z" /> +<glyph unicode="" horiz-adv-x="1664" d="M682 530v-651l-682 94v557h682zM682 1273v-659h-682v565zM1664 530v-786l-907 125v661h907zM1664 1408v-794h-907v669z" /> +<glyph unicode="" horiz-adv-x="1408" d="M493 1053q16 0 27.5 11.5t11.5 27.5t-11.5 27.5t-27.5 11.5t-27 -11.5t-11 -27.5t11 -27.5t27 -11.5zM915 1053q16 0 27 11.5t11 27.5t-11 27.5t-27 11.5t-27.5 -11.5t-11.5 -27.5t11.5 -27.5t27.5 -11.5zM103 869q42 0 72 -30t30 -72v-430q0 -43 -29.5 -73t-72.5 -30 t-73 30t-30 73v430q0 42 30 72t73 30zM1163 850v-666q0 -46 -32 -78t-77 -32h-75v-227q0 -43 -30 -73t-73 -30t-73 30t-30 73v227h-138v-227q0 -43 -30 -73t-73 -30q-42 0 -72 30t-30 73l-1 227h-74q-46 0 -78 32t-32 78v666h918zM931 1255q107 -55 171 -153.5t64 -215.5 h-925q0 117 64 215.5t172 153.5l-71 131q-7 13 5 20q13 6 20 -6l72 -132q95 42 201 42t201 -42l72 132q7 12 20 6q12 -7 5 -20zM1408 767v-430q0 -43 -30 -73t-73 -30q-42 0 -72 30t-30 73v430q0 43 30 72.5t72 29.5q43 0 73 -29.5t30 -72.5z" /> +<glyph unicode="" d="M663 1125q-11 -1 -15.5 -10.5t-8.5 -9.5q-5 -1 -5 5q0 12 19 15h10zM750 1111q-4 -1 -11.5 6.5t-17.5 4.5q24 11 32 -2q3 -6 -3 -9zM399 684q-4 1 -6 -3t-4.5 -12.5t-5.5 -13.5t-10 -13q-7 -10 -1 -12q4 -1 12.5 7t12.5 18q1 3 2 7t2 6t1.5 4.5t0.5 4v3t-1 2.5t-3 2z M1254 325q0 18 -55 42q4 15 7.5 27.5t5 26t3 21.5t0.5 22.5t-1 19.5t-3.5 22t-4 20.5t-5 25t-5.5 26.5q-10 48 -47 103t-72 75q24 -20 57 -83q87 -162 54 -278q-11 -40 -50 -42q-31 -4 -38.5 18.5t-8 83.5t-11.5 107q-9 39 -19.5 69t-19.5 45.5t-15.5 24.5t-13 15t-7.5 7 q-14 62 -31 103t-29.5 56t-23.5 33t-15 40q-4 21 6 53.5t4.5 49.5t-44.5 25q-15 3 -44.5 18t-35.5 16q-8 1 -11 26t8 51t36 27q37 3 51 -30t4 -58q-11 -19 -2 -26.5t30 -0.5q13 4 13 36v37q-5 30 -13.5 50t-21 30.5t-23.5 15t-27 7.5q-107 -8 -89 -134q0 -15 -1 -15 q-9 9 -29.5 10.5t-33 -0.5t-15.5 5q1 57 -16 90t-45 34q-27 1 -41.5 -27.5t-16.5 -59.5q-1 -15 3.5 -37t13 -37.5t15.5 -13.5q10 3 16 14q4 9 -7 8q-7 0 -15.5 14.5t-9.5 33.5q-1 22 9 37t34 14q17 0 27 -21t9.5 -39t-1.5 -22q-22 -15 -31 -29q-8 -12 -27.5 -23.5 t-20.5 -12.5q-13 -14 -15.5 -27t7.5 -18q14 -8 25 -19.5t16 -19t18.5 -13t35.5 -6.5q47 -2 102 15q2 1 23 7t34.5 10.5t29.5 13t21 17.5q9 14 20 8q5 -3 6.5 -8.5t-3 -12t-16.5 -9.5q-20 -6 -56.5 -21.5t-45.5 -19.5q-44 -19 -70 -23q-25 -5 -79 2q-10 2 -9 -2t17 -19 q25 -23 67 -22q17 1 36 7t36 14t33.5 17.5t30 17t24.5 12t17.5 2.5t8.5 -11q0 -2 -1 -4.5t-4 -5t-6 -4.5t-8.5 -5t-9 -4.5t-10 -5t-9.5 -4.5q-28 -14 -67.5 -44t-66.5 -43t-49 -1q-21 11 -63 73q-22 31 -25 22q-1 -3 -1 -10q0 -25 -15 -56.5t-29.5 -55.5t-21 -58t11.5 -63 q-23 -6 -62.5 -90t-47.5 -141q-2 -18 -1.5 -69t-5.5 -59q-8 -24 -29 -3q-32 31 -36 94q-2 28 4 56q4 19 -1 18l-4 -5q-36 -65 10 -166q5 -12 25 -28t24 -20q20 -23 104 -90.5t93 -76.5q16 -15 17.5 -38t-14 -43t-45.5 -23q8 -15 29 -44.5t28 -54t7 -70.5q46 24 7 92 q-4 8 -10.5 16t-9.5 12t-2 6q3 5 13 9.5t20 -2.5q46 -52 166 -36q133 15 177 87q23 38 34 30q12 -6 10 -52q-1 -25 -23 -92q-9 -23 -6 -37.5t24 -15.5q3 19 14.5 77t13.5 90q2 21 -6.5 73.5t-7.5 97t23 70.5q15 18 51 18q1 37 34.5 53t72.5 10.5t60 -22.5zM626 1152 q3 17 -2.5 30t-11.5 15q-9 2 -9 -7q2 -5 5 -6q10 0 7 -15q-3 -20 8 -20q3 0 3 3zM1045 955q-2 8 -6.5 11.5t-13 5t-14.5 5.5q-5 3 -9.5 8t-7 8t-5.5 6.5t-4 4t-4 -1.5q-14 -16 7 -43.5t39 -31.5q9 -1 14.5 8t3.5 20zM867 1168q0 11 -5 19.5t-11 12.5t-9 3q-14 -1 -7 -7l4 -2 q14 -4 18 -31q0 -3 8 2zM921 1401q0 2 -2.5 5t-9 7t-9.5 6q-15 15 -24 15q-9 -1 -11.5 -7.5t-1 -13t-0.5 -12.5q-1 -4 -6 -10.5t-6 -9t3 -8.5q4 -3 8 0t11 9t15 9q1 1 9 1t15 2t9 7zM1486 60q20 -12 31 -24.5t12 -24t-2.5 -22.5t-15.5 -22t-23.5 -19.5t-30 -18.5 t-31.5 -16.5t-32 -15.5t-27 -13q-38 -19 -85.5 -56t-75.5 -64q-17 -16 -68 -19.5t-89 14.5q-18 9 -29.5 23.5t-16.5 25.5t-22 19.5t-47 9.5q-44 1 -130 1q-19 0 -57 -1.5t-58 -2.5q-44 -1 -79.5 -15t-53.5 -30t-43.5 -28.5t-53.5 -11.5q-29 1 -111 31t-146 43q-19 4 -51 9.5 t-50 9t-39.5 9.5t-33.5 14.5t-17 19.5q-10 23 7 66.5t18 54.5q1 16 -4 40t-10 42.5t-4.5 36.5t10.5 27q14 12 57 14t60 12q30 18 42 35t12 51q21 -73 -32 -106q-32 -20 -83 -15q-34 3 -43 -10q-13 -15 5 -57q2 -6 8 -18t8.5 -18t4.5 -17t1 -22q0 -15 -17 -49t-14 -48 q3 -17 37 -26q20 -6 84.5 -18.5t99.5 -20.5q24 -6 74 -22t82.5 -23t55.5 -4q43 6 64.5 28t23 48t-7.5 58.5t-19 52t-20 36.5q-121 190 -169 242q-68 74 -113 40q-11 -9 -15 15q-3 16 -2 38q1 29 10 52t24 47t22 42q8 21 26.5 72t29.5 78t30 61t39 54q110 143 124 195 q-12 112 -16 310q-2 90 24 151.5t106 104.5q39 21 104 21q53 1 106 -13.5t89 -41.5q57 -42 91.5 -121.5t29.5 -147.5q-5 -95 30 -214q34 -113 133 -218q55 -59 99.5 -163t59.5 -191q8 -49 5 -84.5t-12 -55.5t-20 -22q-10 -2 -23.5 -19t-27 -35.5t-40.5 -33.5t-61 -14 q-18 1 -31.5 5t-22.5 13.5t-13.5 15.5t-11.5 20.5t-9 19.5q-22 37 -41 30t-28 -49t7 -97q20 -70 1 -195q-10 -65 18 -100.5t73 -33t85 35.5q59 49 89.5 66.5t103.5 42.5q53 18 77 36.5t18.5 34.5t-25 28.5t-51.5 23.5q-33 11 -49.5 48t-15 72.5t15.5 47.5q1 -31 8 -56.5 t14.5 -40.5t20.5 -28.5t21 -19t21.5 -13t16.5 -9.5z" /> +<glyph unicode="" d="M1024 36q-42 241 -140 498h-2l-2 -1q-16 -6 -43 -16.5t-101 -49t-137 -82t-131 -114.5t-103 -148l-15 11q184 -150 418 -150q132 0 256 52zM839 643q-21 49 -53 111q-311 -93 -673 -93q-1 -7 -1 -21q0 -124 44 -236.5t124 -201.5q50 89 123.5 166.5t142.5 124.5t130.5 81 t99.5 48l37 13q4 1 13 3.5t13 4.5zM732 855q-120 213 -244 378q-138 -65 -234 -186t-128 -272q302 0 606 80zM1416 536q-210 60 -409 29q87 -239 128 -469q111 75 185 189.5t96 250.5zM611 1277q-1 0 -2 -1q1 1 2 1zM1201 1132q-185 164 -433 164q-76 0 -155 -19 q131 -170 246 -382q69 26 130 60.5t96.5 61.5t65.5 57t37.5 40.5zM1424 647q-3 232 -149 410l-1 -1q-9 -12 -19 -24.5t-43.5 -44.5t-71 -60.5t-100 -65t-131.5 -64.5q25 -53 44 -95q2 -6 6.5 -17.5t7.5 -16.5q36 5 74.5 7t73.5 2t69 -1.5t64 -4t56.5 -5.5t48 -6.5t36.5 -6 t25 -4.5zM1536 640q0 -209 -103 -385.5t-279.5 -279.5t-385.5 -103t-385.5 103t-279.5 279.5t-103 385.5t103 385.5t279.5 279.5t385.5 103t385.5 -103t279.5 -279.5t103 -385.5z" /> +<glyph unicode="" d="M1173 473q0 50 -19.5 91.5t-48.5 68.5t-73 49t-82.5 34t-87.5 23l-104 24q-30 7 -44 10.5t-35 11.5t-30 16t-16.5 21t-7.5 30q0 77 144 77q43 0 77 -12t54 -28.5t38 -33.5t40 -29t48 -12q47 0 75.5 32t28.5 77q0 55 -56 99.5t-142 67.5t-182 23q-68 0 -132 -15.5 t-119.5 -47t-89 -87t-33.5 -128.5q0 -61 19 -106.5t56 -75.5t80 -48.5t103 -32.5l146 -36q90 -22 112 -36q32 -20 32 -60q0 -39 -40 -64.5t-105 -25.5q-51 0 -91.5 16t-65 38.5t-45.5 45t-46 38.5t-54 16q-50 0 -75.5 -30t-25.5 -75q0 -92 122 -157.5t291 -65.5 q73 0 140 18.5t122.5 53.5t88.5 93.5t33 131.5zM1536 256q0 -159 -112.5 -271.5t-271.5 -112.5q-130 0 -234 80q-77 -16 -150 -16q-143 0 -273.5 55.5t-225 150t-150 225t-55.5 273.5q0 73 16 150q-80 104 -80 234q0 159 112.5 271.5t271.5 112.5q130 0 234 -80 q77 16 150 16q143 0 273.5 -55.5t225 -150t150 -225t55.5 -273.5q0 -73 -16 -150q80 -104 80 -234z" /> +<glyph unicode="" horiz-adv-x="1664" d="M1483 512l-587 -587q-52 -53 -127.5 -53t-128.5 53l-587 587q-53 53 -53 128t53 128l587 587q53 53 128 53t128 -53l265 -265l-398 -399l-188 188q-42 42 -99 42q-59 0 -100 -41l-120 -121q-42 -40 -42 -99q0 -58 42 -100l406 -408q30 -28 67 -37l6 -4h28q60 0 99 41 l619 619l2 -3q53 -53 53 -128t-53 -128zM1406 1138l120 -120q14 -15 14 -36t-14 -36l-730 -730q-17 -15 -37 -15v0q-4 0 -6 1q-18 2 -30 14l-407 408q-14 15 -14 36t14 35l121 120q13 15 35 15t36 -15l252 -252l574 575q15 15 36 15t36 -15z" /> +<glyph unicode="" d="M704 192v1024q0 14 -9 23t-23 9h-480q-14 0 -23 -9t-9 -23v-1024q0 -14 9 -23t23 -9h480q14 0 23 9t9 23zM1376 576v640q0 14 -9 23t-23 9h-480q-14 0 -23 -9t-9 -23v-640q0 -14 9 -23t23 -9h480q14 0 23 9t9 23zM1536 1344v-1408q0 -26 -19 -45t-45 -19h-1408 q-26 0 -45 19t-19 45v1408q0 26 19 45t45 19h1408q26 0 45 -19t19 -45z" /> +<glyph unicode="" horiz-adv-x="1280" d="M1280 480q0 -40 -28 -68t-68 -28q-51 0 -80 43l-227 341h-45v-132l247 -411q9 -15 9 -33q0 -26 -19 -45t-45 -19h-192v-272q0 -46 -33 -79t-79 -33h-160q-46 0 -79 33t-33 79v272h-192q-26 0 -45 19t-19 45q0 18 9 33l247 411v132h-45l-227 -341q-29 -43 -80 -43 q-40 0 -68 28t-28 68q0 29 16 53l256 384q73 107 176 107h384q103 0 176 -107l256 -384q16 -24 16 -53zM864 1280q0 -93 -65.5 -158.5t-158.5 -65.5t-158.5 65.5t-65.5 158.5t65.5 158.5t158.5 65.5t158.5 -65.5t65.5 -158.5z" /> +<glyph unicode="" horiz-adv-x="1024" d="M1024 832v-416q0 -40 -28 -68t-68 -28t-68 28t-28 68v352h-64v-912q0 -46 -33 -79t-79 -33t-79 33t-33 79v464h-64v-464q0 -46 -33 -79t-79 -33t-79 33t-33 79v912h-64v-352q0 -40 -28 -68t-68 -28t-68 28t-28 68v416q0 80 56 136t136 56h640q80 0 136 -56t56 -136z M736 1280q0 -93 -65.5 -158.5t-158.5 -65.5t-158.5 65.5t-65.5 158.5t65.5 158.5t158.5 65.5t158.5 -65.5t65.5 -158.5z" /> +<glyph unicode="" d="M773 234l350 473q16 22 24.5 59t-6 85t-61.5 79q-40 26 -83 25.5t-73.5 -17.5t-54.5 -45q-36 -40 -96 -40q-59 0 -95 40q-24 28 -54.5 45t-73.5 17.5t-84 -25.5q-46 -31 -60.5 -79t-6 -85t24.5 -59zM1536 640q0 -209 -103 -385.5t-279.5 -279.5t-385.5 -103t-385.5 103 t-279.5 279.5t-103 385.5t103 385.5t279.5 279.5t385.5 103t385.5 -103t279.5 -279.5t103 -385.5z" /> +<glyph unicode="" horiz-adv-x="1792" d="M1472 640q0 117 -45.5 223.5t-123 184t-184 123t-223.5 45.5t-223.5 -45.5t-184 -123t-123 -184t-45.5 -223.5t45.5 -223.5t123 -184t184 -123t223.5 -45.5t223.5 45.5t184 123t123 184t45.5 223.5zM1748 363q-4 -15 -20 -20l-292 -96v-306q0 -16 -13 -26q-15 -10 -29 -4 l-292 94l-180 -248q-10 -13 -26 -13t-26 13l-180 248l-292 -94q-14 -6 -29 4q-13 10 -13 26v306l-292 96q-16 5 -20 20q-5 17 4 29l180 248l-180 248q-9 13 -4 29q4 15 20 20l292 96v306q0 16 13 26q15 10 29 4l292 -94l180 248q9 12 26 12t26 -12l180 -248l292 94 q14 6 29 -4q13 -10 13 -26v-306l292 -96q16 -5 20 -20q5 -16 -4 -29l-180 -248l180 -248q9 -12 4 -29z" /> +<glyph unicode="" d="M1262 233q-54 -9 -110 -9q-182 0 -337 90t-245 245t-90 337q0 192 104 357q-201 -60 -328.5 -229t-127.5 -384q0 -130 51 -248.5t136.5 -204t204 -136.5t248.5 -51q144 0 273.5 61.5t220.5 171.5zM1465 318q-94 -203 -283.5 -324.5t-413.5 -121.5q-156 0 -298 61 t-245 164t-164 245t-61 298q0 153 57.5 292.5t156 241.5t235.5 164.5t290 68.5q44 2 61 -39q18 -41 -15 -72q-86 -78 -131.5 -181.5t-45.5 -218.5q0 -148 73 -273t198 -198t273 -73q118 0 228 51q41 18 72 -13q14 -14 17.5 -34t-4.5 -38z" /> +<glyph unicode="" horiz-adv-x="1792" d="M1088 704q0 26 -19 45t-45 19h-256q-26 0 -45 -19t-19 -45t19 -45t45 -19h256q26 0 45 19t19 45zM1664 896v-960q0 -26 -19 -45t-45 -19h-1408q-26 0 -45 19t-19 45v960q0 26 19 45t45 19h1408q26 0 45 -19t19 -45zM1728 1344v-256q0 -26 -19 -45t-45 -19h-1536 q-26 0 -45 19t-19 45v256q0 26 19 45t45 19h1536q26 0 45 -19t19 -45z" /> +<glyph unicode="" horiz-adv-x="1664" d="M1632 576q0 -26 -19 -45t-45 -19h-224q0 -171 -67 -290l208 -209q19 -19 19 -45t-19 -45q-18 -19 -45 -19t-45 19l-198 197q-5 -5 -15 -13t-42 -28.5t-65 -36.5t-82 -29t-97 -13v896h-128v-896q-51 0 -101.5 13.5t-87 33t-66 39t-43.5 32.5l-15 14l-183 -207 q-20 -21 -48 -21q-24 0 -43 16q-19 18 -20.5 44.5t15.5 46.5l202 227q-58 114 -58 274h-224q-26 0 -45 19t-19 45t19 45t45 19h224v294l-173 173q-19 19 -19 45t19 45t45 19t45 -19l173 -173h844l173 173q19 19 45 19t45 -19t19 -45t-19 -45l-173 -173v-294h224q26 0 45 -19 t19 -45zM1152 1152h-640q0 133 93.5 226.5t226.5 93.5t226.5 -93.5t93.5 -226.5z" /> +<glyph unicode="" horiz-adv-x="1920" d="M1917 1016q23 -64 -150 -294q-24 -32 -65 -85q-78 -100 -90 -131q-17 -41 14 -81q17 -21 81 -82h1l1 -1l1 -1l2 -2q141 -131 191 -221q3 -5 6.5 -12.5t7 -26.5t-0.5 -34t-25 -27.5t-59 -12.5l-256 -4q-24 -5 -56 5t-52 22l-20 12q-30 21 -70 64t-68.5 77.5t-61 58 t-56.5 15.5q-3 -1 -8 -3.5t-17 -14.5t-21.5 -29.5t-17 -52t-6.5 -77.5q0 -15 -3.5 -27.5t-7.5 -18.5l-4 -5q-18 -19 -53 -22h-115q-71 -4 -146 16.5t-131.5 53t-103 66t-70.5 57.5l-25 24q-10 10 -27.5 30t-71.5 91t-106 151t-122.5 211t-130.5 272q-6 16 -6 27t3 16l4 6 q15 19 57 19l274 2q12 -2 23 -6.5t16 -8.5l5 -3q16 -11 24 -32q20 -50 46 -103.5t41 -81.5l16 -29q29 -60 56 -104t48.5 -68.5t41.5 -38.5t34 -14t27 5q2 1 5 5t12 22t13.5 47t9.5 81t0 125q-2 40 -9 73t-14 46l-6 12q-25 34 -85 43q-13 2 5 24q17 19 38 30q53 26 239 24 q82 -1 135 -13q20 -5 33.5 -13.5t20.5 -24t10.5 -32t3.5 -45.5t-1 -55t-2.5 -70.5t-1.5 -82.5q0 -11 -1 -42t-0.5 -48t3.5 -40.5t11.5 -39t22.5 -24.5q8 -2 17 -4t26 11t38 34.5t52 67t68 107.5q60 104 107 225q4 10 10 17.5t11 10.5l4 3l5 2.5t13 3t20 0.5l288 2 q39 5 64 -2.5t31 -16.5z" /> +<glyph unicode="" horiz-adv-x="1792" d="M675 252q21 34 11 69t-45 50q-34 14 -73 1t-60 -46q-22 -34 -13 -68.5t43 -50.5t74.5 -2.5t62.5 47.5zM769 373q8 13 3.5 26.5t-17.5 18.5q-14 5 -28.5 -0.5t-21.5 -18.5q-17 -31 13 -45q14 -5 29 0.5t22 18.5zM943 266q-45 -102 -158 -150t-224 -12 q-107 34 -147.5 126.5t6.5 187.5q47 93 151.5 139t210.5 19q111 -29 158.5 -119.5t2.5 -190.5zM1255 426q-9 96 -89 170t-208.5 109t-274.5 21q-223 -23 -369.5 -141.5t-132.5 -264.5q9 -96 89 -170t208.5 -109t274.5 -21q223 23 369.5 141.5t132.5 264.5zM1563 422 q0 -68 -37 -139.5t-109 -137t-168.5 -117.5t-226 -83t-270.5 -31t-275 33.5t-240.5 93t-171.5 151t-65 199.5q0 115 69.5 245t197.5 258q169 169 341.5 236t246.5 -7q65 -64 20 -209q-4 -14 -1 -20t10 -7t14.5 0.5t13.5 3.5l6 2q139 59 246 59t153 -61q45 -63 0 -178 q-2 -13 -4.5 -20t4.5 -12.5t12 -7.5t17 -6q57 -18 103 -47t80 -81.5t34 -116.5zM1489 1046q42 -47 54.5 -108.5t-6.5 -117.5q-8 -23 -29.5 -34t-44.5 -4q-23 8 -34 29.5t-4 44.5q20 63 -24 111t-107 35q-24 -5 -45 8t-25 37q-5 24 8 44.5t37 25.5q60 13 119 -5.5t101 -65.5z M1670 1209q87 -96 112.5 -222.5t-13.5 -241.5q-9 -27 -34 -40t-52 -4t-40 34t-5 52q28 82 10 172t-80 158q-62 69 -148 95.5t-173 8.5q-28 -6 -52 9.5t-30 43.5t9.5 51.5t43.5 29.5q123 26 244 -11.5t208 -134.5z" /> +<glyph unicode="" d="M1133 -34q-171 -94 -368 -94q-196 0 -367 94q138 87 235.5 211t131.5 268q35 -144 132.5 -268t235.5 -211zM638 1394v-485q0 -252 -126.5 -459.5t-330.5 -306.5q-181 215 -181 495q0 187 83.5 349.5t229.5 269.5t325 137zM1536 638q0 -280 -181 -495 q-204 99 -330.5 306.5t-126.5 459.5v485q179 -30 325 -137t229.5 -269.5t83.5 -349.5z" /> +<glyph unicode="" horiz-adv-x="1408" d="M1402 433q-32 -80 -76 -138t-91 -88.5t-99 -46.5t-101.5 -14.5t-96.5 8.5t-86.5 22t-69.5 27.5t-46 22.5l-17 10q-113 -228 -289.5 -359.5t-384.5 -132.5q-19 0 -32 13t-13 32t13 31.5t32 12.5q173 1 322.5 107.5t251.5 294.5q-36 -14 -72 -23t-83 -13t-91 2.5t-93 28.5 t-92 59t-84.5 100t-74.5 146q114 47 214 57t167.5 -7.5t124.5 -56.5t88.5 -77t56.5 -82q53 131 79 291q-7 -1 -18 -2.5t-46.5 -2.5t-69.5 0.5t-81.5 10t-88.5 23t-84 42.5t-75 65t-54.5 94.5t-28.5 127.5q70 28 133.5 36.5t112.5 -1t92 -30t73.5 -50t56 -61t42 -63t27.5 -56 t16 -39.5l4 -16q12 122 12 195q-8 6 -21.5 16t-49 44.5t-63.5 71.5t-54 93t-33 112.5t12 127t70 138.5q73 -25 127.5 -61.5t84.5 -76.5t48 -85t20.5 -89t-0.5 -85.5t-13 -76.5t-19 -62t-17 -42l-7 -15q1 -5 1 -50.5t-1 -71.5q3 7 10 18.5t30.5 43t50.5 58t71 55.5t91.5 44.5 t112 14.5t132.5 -24q-2 -78 -21.5 -141.5t-50 -104.5t-69.5 -71.5t-81.5 -45.5t-84.5 -24t-80 -9.5t-67.5 1t-46.5 4.5l-17 3q-23 -147 -73 -283q6 7 18 18.5t49.5 41t77.5 52.5t99.5 42t117.5 20t129 -23.5t137 -77.5z" /> +<glyph unicode="" horiz-adv-x="1280" d="M1259 283v-66q0 -85 -57.5 -144.5t-138.5 -59.5h-57l-260 -269v269h-529q-81 0 -138.5 59.5t-57.5 144.5v66h1238zM1259 609v-255h-1238v255h1238zM1259 937v-255h-1238v255h1238zM1259 1077v-67h-1238v67q0 84 57.5 143.5t138.5 59.5h846q81 0 138.5 -59.5t57.5 -143.5z " /> +<glyph unicode="" d="M1152 640q0 -14 -9 -23l-320 -320q-9 -9 -23 -9q-13 0 -22.5 9.5t-9.5 22.5v192h-352q-13 0 -22.5 9.5t-9.5 22.5v192q0 13 9.5 22.5t22.5 9.5h352v192q0 14 9 23t23 9q12 0 24 -10l319 -319q9 -9 9 -23zM1312 640q0 148 -73 273t-198 198t-273 73t-273 -73t-198 -198 t-73 -273t73 -273t198 -198t273 -73t273 73t198 198t73 273zM1536 640q0 -209 -103 -385.5t-279.5 -279.5t-385.5 -103t-385.5 103t-279.5 279.5t-103 385.5t103 385.5t279.5 279.5t385.5 103t385.5 -103t279.5 -279.5t103 -385.5z" /> +<glyph unicode="" d="M1152 736v-192q0 -13 -9.5 -22.5t-22.5 -9.5h-352v-192q0 -14 -9 -23t-23 -9q-12 0 -24 10l-319 319q-9 9 -9 23t9 23l320 320q9 9 23 9q13 0 22.5 -9.5t9.5 -22.5v-192h352q13 0 22.5 -9.5t9.5 -22.5zM1312 640q0 148 -73 273t-198 198t-273 73t-273 -73t-198 -198 t-73 -273t73 -273t198 -198t273 -73t273 73t198 198t73 273zM1536 640q0 -209 -103 -385.5t-279.5 -279.5t-385.5 -103t-385.5 103t-279.5 279.5t-103 385.5t103 385.5t279.5 279.5t385.5 103t385.5 -103t279.5 -279.5t103 -385.5z" /> +<glyph unicode="" d="M1024 960v-640q0 -26 -19 -45t-45 -19q-20 0 -37 12l-448 320q-27 19 -27 52t27 52l448 320q17 12 37 12q26 0 45 -19t19 -45zM1280 160v960q0 13 -9.5 22.5t-22.5 9.5h-960q-13 0 -22.5 -9.5t-9.5 -22.5v-960q0 -13 9.5 -22.5t22.5 -9.5h960q13 0 22.5 9.5t9.5 22.5z M1536 1120v-960q0 -119 -84.5 -203.5t-203.5 -84.5h-960q-119 0 -203.5 84.5t-84.5 203.5v960q0 119 84.5 203.5t203.5 84.5h960q119 0 203.5 -84.5t84.5 -203.5z" /> +<glyph unicode="" d="M1024 640q0 -106 -75 -181t-181 -75t-181 75t-75 181t75 181t181 75t181 -75t75 -181zM768 1184q-148 0 -273 -73t-198 -198t-73 -273t73 -273t198 -198t273 -73t273 73t198 198t73 273t-73 273t-198 198t-273 73zM1536 640q0 -209 -103 -385.5t-279.5 -279.5 t-385.5 -103t-385.5 103t-279.5 279.5t-103 385.5t103 385.5t279.5 279.5t385.5 103t385.5 -103t279.5 -279.5t103 -385.5z" /> +<glyph unicode="" horiz-adv-x="1664" d="M1023 349l102 -204q-58 -179 -210 -290t-339 -111q-156 0 -288.5 77.5t-210 210t-77.5 288.5q0 181 104.5 330t274.5 211l17 -131q-122 -54 -195 -165.5t-73 -244.5q0 -185 131.5 -316.5t316.5 -131.5q126 0 232.5 65t165 175.5t49.5 236.5zM1571 249l58 -114l-256 -128 q-13 -7 -29 -7q-40 0 -57 35l-239 477h-472q-24 0 -42.5 16.5t-21.5 40.5l-96 779q-2 16 6 42q14 51 57 82.5t97 31.5q66 0 113 -47t47 -113q0 -69 -52 -117.5t-120 -41.5l37 -289h423v-128h-407l16 -128h455q40 0 57 -35l228 -455z" /> +<glyph unicode="" d="M1254 899q16 85 -21 132q-52 65 -187 45q-17 -3 -41 -12.5t-57.5 -30.5t-64.5 -48.5t-59.5 -70t-44.5 -91.5q80 7 113.5 -16t26.5 -99q-5 -52 -52 -143q-43 -78 -71 -99q-44 -32 -87 14q-23 24 -37.5 64.5t-19 73t-10 84t-8.5 71.5q-23 129 -34 164q-12 37 -35.5 69 t-50.5 40q-57 16 -127 -25q-54 -32 -136.5 -106t-122.5 -102v-7q16 -8 25.5 -26t21.5 -20q21 -3 54.5 8.5t58 10.5t41.5 -30q11 -18 18.5 -38.5t15 -48t12.5 -40.5q17 -46 53 -187q36 -146 57 -197q42 -99 103 -125q43 -12 85 -1.5t76 31.5q131 77 250 237 q104 139 172.5 292.5t82.5 226.5zM1536 1120v-960q0 -119 -84.5 -203.5t-203.5 -84.5h-960q-119 0 -203.5 84.5t-84.5 203.5v960q0 119 84.5 203.5t203.5 84.5h960q119 0 203.5 -84.5t84.5 -203.5z" /> +<glyph unicode="" horiz-adv-x="1152" d="M1152 704q0 -191 -94.5 -353t-256.5 -256.5t-353 -94.5h-160q-14 0 -23 9t-9 23v611l-215 -66q-3 -1 -9 -1q-10 0 -19 6q-13 10 -13 26v128q0 23 23 31l233 71v93l-215 -66q-3 -1 -9 -1q-10 0 -19 6q-13 10 -13 26v128q0 23 23 31l233 71v250q0 14 9 23t23 9h160 q14 0 23 -9t9 -23v-181l375 116q15 5 28 -5t13 -26v-128q0 -23 -23 -31l-393 -121v-93l375 116q15 5 28 -5t13 -26v-128q0 -23 -23 -31l-393 -121v-487q188 13 318 151t130 328q0 14 9 23t23 9h160q14 0 23 -9t9 -23z" /> +<glyph unicode="" horiz-adv-x="1408" d="M1152 736v-64q0 -14 -9 -23t-23 -9h-352v-352q0 -14 -9 -23t-23 -9h-64q-14 0 -23 9t-9 23v352h-352q-14 0 -23 9t-9 23v64q0 14 9 23t23 9h352v352q0 14 9 23t23 9h64q14 0 23 -9t9 -23v-352h352q14 0 23 -9t9 -23zM1280 288v832q0 66 -47 113t-113 47h-832 q-66 0 -113 -47t-47 -113v-832q0 -66 47 -113t113 -47h832q66 0 113 47t47 113zM1408 1120v-832q0 -119 -84.5 -203.5t-203.5 -84.5h-832q-119 0 -203.5 84.5t-84.5 203.5v832q0 119 84.5 203.5t203.5 84.5h832q119 0 203.5 -84.5t84.5 -203.5z" /> +<glyph unicode="" horiz-adv-x="1792" /> +<glyph unicode="" horiz-adv-x="1792" /> +<glyph unicode="" horiz-adv-x="1792" /> +<glyph unicode="" horiz-adv-x="1792" /> +<glyph unicode="" horiz-adv-x="1792" /> +<glyph unicode="" horiz-adv-x="1792" /> +<glyph unicode="" horiz-adv-x="1792" /> +<glyph unicode="" horiz-adv-x="1792" /> +<glyph unicode="" horiz-adv-x="1792" /> +</font> +</defs></svg>
\ No newline at end of file diff --git a/doc/website-v1/fonts/fontawesome-webfont.ttf b/doc/website-v1/fonts/fontawesome-webfont.ttf Binary files differnew file mode 100755 index 0000000..e89738d --- /dev/null +++ b/doc/website-v1/fonts/fontawesome-webfont.ttf diff --git a/doc/website-v1/fonts/fontawesome-webfont.woff b/doc/website-v1/fonts/fontawesome-webfont.woff Binary files differnew file mode 100755 index 0000000..8c1748a --- /dev/null +++ b/doc/website-v1/fonts/fontawesome-webfont.woff diff --git a/doc/website-v1/history-guide.adoc b/doc/website-v1/history-guide.adoc new file mode 100644 index 0000000..a3dd9c6 --- /dev/null +++ b/doc/website-v1/history-guide.adoc @@ -0,0 +1,275 @@ += Cluster history = +:source-highlighter: pygments + +This guide should help administrators and consultants tackle +issues in Pacemaker cluster installations. We concentrate +on troubleshooting and analysis methods with the crmsh history. + +Cluster leaves numerous traces behind, more than any other +system. The logs and the rest are spread among all cluster nodes +and multiple directories. The obvious difficulty is to show that +information in a consolidated manner. This is where crmsh +history helps. + +Hopefully, the guide will help you investigate your +specific issue with more efficiency and less effort. + +== Sample cluster + +In <<Listing 1>> a modestly complex sample cluster is shown with +which we can experiment and break in some hopefully instructive +ways. + +NOTE: We won't be going into how to setup nodes or configure the + cluster. For that, please refer to the + link:/start-guide[Getting Started] document. + +[source,crmsh] +[caption="Listing 1: "] +.Sample cluster configuration[[Listing 1]] +----------------- +include::include/history-guide/sample-cluster.conf.crm[] +----------------- + +If you're new to clusters, that configuration may look +overwhelming. A graphical presentation in <<Image 1>> of the +essential elements and relations between them is easier on the eye +(and the mind). + +[caption="Image 1: "] +.Sample cluster configuration as a graph[[Image 1]] +image::/img/history-guide/sample-cluster.conf.png[link="/img/history-guide/sample-cluster.conf.png"] + +As homework, try to match the two cluster representations. + +== Quick (& dirty) start + +For the impatient, we give here a few examples of history use. + +Most of the time you will be dealing with various resource +(a.k.a. application) induced phenomena. For instance, while +preparing this document we noticed that a probe failed repeatedly +on a node which wasn't even running the resource <<Listing 2>>. + +[source,ansiclr] +[caption="Listing 2: "] +.crm status output[[Listing 2]] +----------------- +include::include/history-guide/status-probe-fail.typescript[] +----------------- + +The history +resource+ command shows log messages relevant to the +supplied resource <<Listing 3>>. + +[source,ansiclr] +[caption="Listing 3: "] +.Logs on failed +nfs-server+ probe operation[[Listing 3]] +----------------- +include::include/history-guide/nfs-probe-err.typescript[] +----------------- + +<1> NFS server error message. +<2> Warning about a non-existing user id. + +NOTE: Messages logged by resource agents are always tagged with + 'type(ID)' (in <<Listing 3>>: +nfsserver(nfs-server)+). + + + Everything dumped to +stderr/stdout+ (in <<Listing 3>>: + +id: rpcuser: no such user+) is captured and subsequently + logged by +lrmd+. The +stdout+ output is at the 'info' + severity which is by default _not_ logged by pacemaker + since version 1.1.12. + +At the very top we see error message reporting that the +NFS server is running, but some other stuff, apparently +unexpectedly, is not. However, we know that it cannot be +running on the 'c' node as it is already running on the 'a' node. +Not being able to figure out what is going on, we had to turn on +tracing of the resource agent. <<Listing 4>> shows how to do +that. + +[source,ansiclr] +[caption="Listing 4: "] +.Set `nfs-server` probe operation resource tracing[[Listing 4]] +----------------- +include::include/history-guide/resource-trace.typescript[] +----------------- + +Trace of the +nfsserver+ RA revealed that the +nfs-server+ init +script (used internally by the resource agent) _always_ exits +with success for status. That was actually due to the recent port +to systemd and erroneous interpretation of `systemctl status` +semantics: it always exits with success (due to some paradigm +shift, we guess). FYI, `systemctl is-active` should be used +instead and it does report a service status as expected. + +As a bonus, a minor issue about a non-existing user id +rpcuser+ +is also revealed. + +NOTE: Messages in the crm history log output are colored + depending on the originating host. + +The rest of this document gives more details about crmsh history. +If you're more of a just-try-it-out person, enter +crm history+ +and experiment. With +history+ commands you cannot really break +anything (fingers crossed). + +== Introduction to crmsh `history` + +The history crmsh feature, as the name suggests, deals with the +past. It was conceived as a facility to bring to the fore all +trails pacemaker cluster leaves behind which are relevant to a +particular resource, node, or event. It is used in the first +place as a troubleshooting tool, but it can also be helpful in +studying pacemaker clusters. + +To begin, we run the `info` command which gives an overview, as +shown in <<Listing 5>>. + +[source,ansiclr] +[caption="Listing 5: "] +.Basic history information[[Listing 5]] +----------------- +include::include/history-guide/info.typescript[] +----------------- + +The `timeframe` command limits the observed period and helps +focus on the events of interest. Here we wanted to look at the +10 minute period. Two transitions were executed during this time. + +== Transitions + +Transitions are basic units capturing cluster movements +(resource operations and node events). A transition +consists of a set of actions to reach a desired cluster +status as specified in the cluster configuration by the +user. + +Every configuration or status change results in a transition. + +Every transition is also a CIB, which is how cluster +configuration and status are stored. Transitions are saved +to files, the so called PE (Policy Engine) inputs. + +In <<Listing 6>> we show how to display transitions. +The listing is annotated to explain the output in more detail. + + +[source,ansiclr] +[caption="Listing 6: "] +.Viewing transitions[[Listing 6]] +----------------- +include::include/history-guide/basic-transition.typescript[] +----------------- + +<1> The transition command without arguments displays the latest +transition. +<2> Graph of transition actions is provided by `graphviz`. See +<<Image 2>>. +<3> Output of `crm_simulate` with irrelevant stuff edited out. +`crm_simulate` was formerly known as `ptest`. +<4> Transition summary followed by selection of log messages. +History weeds out messages which are of lesser importance. See +<<Listing 8>> if you want to see what history has been hiding +from you here. + +Incidentally, if you wonder why all transitions in these examples +are green, that is not because they were green in any sense of +the color, but just due to that being color of node 'c': as +chance would have it, 'c' was calling shots at the time (being +Designated Coordinator or DC). That is also why all `crmd` and +`pengine` messages are coming from 'c'. + +NOTE: Transitions are the basis of pacemaker operation, make sure +that you understand them. + +What you cannot see in the listing is a graph generated and shown +in a separate window in your X11 display. <<Image 2>> may not be +very involved, but we reckon it's as good a start as starts go. + +[caption="Image 2: "] +.Graph for transition 1907[[Image 2]] +image::/img/history-guide/smallapache-start.png[link="/img/history-guide/smallapache-start.png"] + +It may sometimes be useful to see what changed between two +transitions. History `diff` command is in action in <<Listing 7>>. + +[source,ansiclr] +[caption="Listing 7: "] +.Viewing transitions[[Listing 7]] +----------------- +include::include/history-guide/diff.typescript[] +----------------- + +<1> Configuration diff between two last transitions. Transitions +may be referenced with indexes starting at 0 and going backwards. +<2> Status diff between two last transitions. + +Whereas configuration diff is (hopefully) obvious, status diff +needs some explanation: the status section of the PE inputs +(transitions) always lags behind the configuration. This is +because at the time the transition is saved to a file, the +actions of that transition are yet to be executed. So, the status +section of transition _N_ corresponds to the configuration _N-1_. + +[source,ansiclr] +[caption="Listing 8: "] +.Full transition log[[Listing 8]] +----------------- +include::include/history-guide/transition-log.typescript[] +----------------- + +== Resource and node events + +Apart from transitions, events such as resource start or stop are +what we usually want to examine. In our extremely exciting +example of apache resource restart, the history `resource` +command picks the most interesting resource related messages as +shown in <<Listing 9>>. Again, history shows only the most +important log parts. + +NOTE: If you want to see more detail (which may not always be + recommendable), then use the history `detail` command to + increase the level of detail displayed. + +[source,ansiclr] +[caption="Listing 9: "] +.Resource related messages[[Listing 9]] +----------------- +include::include/history-guide/resource.typescript[] +----------------- + +Node related events are node start and stop (cluster-wise), +node membership changes, and stonith events (aka node fence). +We'll refrain from showing examples of the history `node` +command--it is analogue to the `resource` command. + +== Viewing logs + +History `log` command, unsurprisingly, displays logs. The +messages from various nodes are weaved and shown in different +colors for the sake of easier viewing. Unlike other history +commands, `log` shows all messages captured in the report. If you +find some of them irrelevant they can be filtered out: +the `exclude` command takes extended regular expressions and it +is additive. We usually set the exclude expression to at least +`ssh|systemd|kernel`. Use `exclude clear` to remove all +expressions. And don't forget the `timeframe` command that +imposes a time window on the report. + +== External reports, configurations, and graphs + +The information source history works with is `hb_report` +generated report. Even when examining live cluster, `hb_report` is +run behind the scene to collect the data before presenting it to +the user. Well, at least to generate the first report: there is a +special procedure for log refreshing and collecting new PE +inputs, which runs much faster than creating a report from +scratch. However, juggling with multiple sources, appending logs, +moving time windows, may not always be foolproof, and if +the source gets borked you can always ask for a brand new report +with `refresh force`. + +Analyzing reports from external source is no different from what +we've seen so far. In fact, there's a `source` command which +tells history where to look for data. diff --git a/doc/website-v1/img/history-guide/sample-cluster.conf.png b/doc/website-v1/img/history-guide/sample-cluster.conf.png Binary files differnew file mode 100644 index 0000000..0863923 --- /dev/null +++ b/doc/website-v1/img/history-guide/sample-cluster.conf.png diff --git a/doc/website-v1/img/history-guide/smallapache-start.png b/doc/website-v1/img/history-guide/smallapache-start.png Binary files differnew file mode 100644 index 0000000..47853c9 --- /dev/null +++ b/doc/website-v1/img/history-guide/smallapache-start.png diff --git a/doc/website-v1/img/icons/README b/doc/website-v1/img/icons/README new file mode 100644 index 0000000..f12b2a7 --- /dev/null +++ b/doc/website-v1/img/icons/README @@ -0,0 +1,5 @@ +Replaced the plain DocBook XSL admonition icons with Jimmac's DocBook +icons (http://jimmac.musichall.cz/ikony.php3). I dropped transparency +from the Jimmac icons to get round MS IE and FOP PNG incompatibilies. + +Stuart Rackham diff --git a/doc/website-v1/img/icons/callouts/1.png b/doc/website-v1/img/icons/callouts/1.png Binary files differnew file mode 100644 index 0000000..7d47343 --- /dev/null +++ b/doc/website-v1/img/icons/callouts/1.png diff --git a/doc/website-v1/img/icons/callouts/10.png b/doc/website-v1/img/icons/callouts/10.png Binary files differnew file mode 100644 index 0000000..997bbc8 --- /dev/null +++ b/doc/website-v1/img/icons/callouts/10.png diff --git a/doc/website-v1/img/icons/callouts/11.png b/doc/website-v1/img/icons/callouts/11.png Binary files differnew file mode 100644 index 0000000..ce47dac --- /dev/null +++ b/doc/website-v1/img/icons/callouts/11.png diff --git a/doc/website-v1/img/icons/callouts/12.png b/doc/website-v1/img/icons/callouts/12.png Binary files differnew file mode 100644 index 0000000..31daf4e --- /dev/null +++ b/doc/website-v1/img/icons/callouts/12.png diff --git a/doc/website-v1/img/icons/callouts/13.png b/doc/website-v1/img/icons/callouts/13.png Binary files differnew file mode 100644 index 0000000..14021a8 --- /dev/null +++ b/doc/website-v1/img/icons/callouts/13.png diff --git a/doc/website-v1/img/icons/callouts/14.png b/doc/website-v1/img/icons/callouts/14.png Binary files differnew file mode 100644 index 0000000..64014b7 --- /dev/null +++ b/doc/website-v1/img/icons/callouts/14.png diff --git a/doc/website-v1/img/icons/callouts/15.png b/doc/website-v1/img/icons/callouts/15.png Binary files differnew file mode 100644 index 0000000..0d65765 --- /dev/null +++ b/doc/website-v1/img/icons/callouts/15.png diff --git a/doc/website-v1/img/icons/callouts/2.png b/doc/website-v1/img/icons/callouts/2.png Binary files differnew file mode 100644 index 0000000..5d09341 --- /dev/null +++ b/doc/website-v1/img/icons/callouts/2.png diff --git a/doc/website-v1/img/icons/callouts/3.png b/doc/website-v1/img/icons/callouts/3.png Binary files differnew file mode 100644 index 0000000..ef7b700 --- /dev/null +++ b/doc/website-v1/img/icons/callouts/3.png diff --git a/doc/website-v1/img/icons/callouts/4.png b/doc/website-v1/img/icons/callouts/4.png Binary files differnew file mode 100644 index 0000000..adb8364 --- /dev/null +++ b/doc/website-v1/img/icons/callouts/4.png diff --git a/doc/website-v1/img/icons/callouts/5.png b/doc/website-v1/img/icons/callouts/5.png Binary files differnew file mode 100644 index 0000000..4d7eb46 --- /dev/null +++ b/doc/website-v1/img/icons/callouts/5.png diff --git a/doc/website-v1/img/icons/callouts/6.png b/doc/website-v1/img/icons/callouts/6.png Binary files differnew file mode 100644 index 0000000..0ba694a --- /dev/null +++ b/doc/website-v1/img/icons/callouts/6.png diff --git a/doc/website-v1/img/icons/callouts/7.png b/doc/website-v1/img/icons/callouts/7.png Binary files differnew file mode 100644 index 0000000..472e96f --- /dev/null +++ b/doc/website-v1/img/icons/callouts/7.png diff --git a/doc/website-v1/img/icons/callouts/8.png b/doc/website-v1/img/icons/callouts/8.png Binary files differnew file mode 100644 index 0000000..5e60973 --- /dev/null +++ b/doc/website-v1/img/icons/callouts/8.png diff --git a/doc/website-v1/img/icons/callouts/9.png b/doc/website-v1/img/icons/callouts/9.png Binary files differnew file mode 100644 index 0000000..a0676d2 --- /dev/null +++ b/doc/website-v1/img/icons/callouts/9.png diff --git a/doc/website-v1/img/icons/caution.png b/doc/website-v1/img/icons/caution.png Binary files differnew file mode 100644 index 0000000..9a8c515 --- /dev/null +++ b/doc/website-v1/img/icons/caution.png diff --git a/doc/website-v1/img/icons/example.png b/doc/website-v1/img/icons/example.png Binary files differnew file mode 100644 index 0000000..1199e86 --- /dev/null +++ b/doc/website-v1/img/icons/example.png diff --git a/doc/website-v1/img/icons/home.png b/doc/website-v1/img/icons/home.png Binary files differnew file mode 100644 index 0000000..37a5231 --- /dev/null +++ b/doc/website-v1/img/icons/home.png diff --git a/doc/website-v1/img/icons/important.png b/doc/website-v1/img/icons/important.png Binary files differnew file mode 100644 index 0000000..be685cc --- /dev/null +++ b/doc/website-v1/img/icons/important.png diff --git a/doc/website-v1/img/icons/next.png b/doc/website-v1/img/icons/next.png Binary files differnew file mode 100644 index 0000000..64e126b --- /dev/null +++ b/doc/website-v1/img/icons/next.png diff --git a/doc/website-v1/img/icons/note.png b/doc/website-v1/img/icons/note.png Binary files differnew file mode 100644 index 0000000..7c1f3e2 --- /dev/null +++ b/doc/website-v1/img/icons/note.png diff --git a/doc/website-v1/img/icons/prev.png b/doc/website-v1/img/icons/prev.png Binary files differnew file mode 100644 index 0000000..3e8f12f --- /dev/null +++ b/doc/website-v1/img/icons/prev.png diff --git a/doc/website-v1/img/icons/tip.png b/doc/website-v1/img/icons/tip.png Binary files differnew file mode 100644 index 0000000..f087c73 --- /dev/null +++ b/doc/website-v1/img/icons/tip.png diff --git a/doc/website-v1/img/icons/up.png b/doc/website-v1/img/icons/up.png Binary files differnew file mode 100644 index 0000000..2db1ce6 --- /dev/null +++ b/doc/website-v1/img/icons/up.png diff --git a/doc/website-v1/img/icons/warning.png b/doc/website-v1/img/icons/warning.png Binary files differnew file mode 100644 index 0000000..d41edb9 --- /dev/null +++ b/doc/website-v1/img/icons/warning.png diff --git a/doc/website-v1/img/laptop.png b/doc/website-v1/img/laptop.png Binary files differnew file mode 100644 index 0000000..2f831ba --- /dev/null +++ b/doc/website-v1/img/laptop.png diff --git a/doc/website-v1/img/loader.gif b/doc/website-v1/img/loader.gif Binary files differnew file mode 100644 index 0000000..b2cfedb --- /dev/null +++ b/doc/website-v1/img/loader.gif diff --git a/doc/website-v1/img/servers.gif b/doc/website-v1/img/servers.gif Binary files differnew file mode 100644 index 0000000..20afdcb --- /dev/null +++ b/doc/website-v1/img/servers.gif diff --git a/doc/website-v1/include/history-guide/basic-transition.typescript b/doc/website-v1/include/history-guide/basic-transition.typescript new file mode 100644 index 0000000..a5a0a31 --- /dev/null +++ b/doc/website-v1/include/history-guide/basic-transition.typescript @@ -0,0 +1,22 @@ +[32mcrm(live)history# [0;10mtransition <1> +[36mINFO[0;10m: running ptest with /var/cache/crm/history/live/sle12-c/pengine/pe-input-1907.bz2 +[36mINFO[0;10m: starting dotty to show transition graph <2> +Current cluster status: <3> +Online: [ sle12-a sle12-c ] + s-libvirt (stonith:external/libvirt): Started sle12-c + ... + small-apache (ocf::heartbeat:apache): Stopped +Transition Summary: + * Start small-apache (sle12-a) +Executing cluster transition: + * Resource action: small-apache start on sle12-a +Revised cluster status: +Online: [ sle12-a sle12-c ] + s-libvirt (stonith:external/libvirt): Started sle12-c + ... + small-apache (ocf::heartbeat:apache): Started sle12-a + +Transition sle12-c:pe-input-1907 (20:30:14 - 20:30:15): <4> + total 1 actions: 1 Complete +[32mApr 15 20:30:14 sle12-c crmd[1136]: notice: te_rsc_command: Initiating action 60: start small-apache_start_0 on sle12-a[0;10m +[0;10mApr 15 20:30:14 sle12-a apache(small-apache)[1586]: INFO: AH00558: httpd2: Could not reliably determine the server's fully qualified domain name, using 10.2.12.51. Set the 'ServerName' directive globally to suppress this message[0;10m diff --git a/doc/website-v1/include/history-guide/diff.typescript b/doc/website-v1/include/history-guide/diff.typescript new file mode 100644 index 0000000..129febc --- /dev/null +++ b/doc/website-v1/include/history-guide/diff.typescript @@ -0,0 +1,11 @@ +[32mcrm(live)history# [0;10mdiff -1 0 <1> +--- -1 ++++ 0 +@@ -11 +11 @@ +-[33mprimitive[0;10m [0;10msmall-apache[0;10m apache [33mparams[0;10m [36mconfigfile[0;10m=[31m"/etc/apache2/small.conf"[0;10m [33mmeta[0;10m [36mtarget-role[0;10m=[31mStopped[0;10m ++[33mprimitive[0;10m [0;10msmall-apache[0;10m apache [33mparams[0;10m [36mconfigfile[0;10m=[31m"/etc/apache2/small.conf"[0;10m [33mmeta[0;10m [36mtarget-role[0;10m=[31mStarted[0;10m +[32mcrm(live)history# [0;10mdiff -1 0 status <2> +--- -1 ++++ 0 +@@ -15 +14,0 @@ +- small-apache (ocf::heartbeat:apache): Started sle12-a diff --git a/doc/website-v1/include/history-guide/info.typescript b/doc/website-v1/include/history-guide/info.typescript new file mode 100644 index 0000000..d7aae8d --- /dev/null +++ b/doc/website-v1/include/history-guide/info.typescript @@ -0,0 +1,16 @@ +# crm history +[32mcrm(live)history# [0;10mtimeframe "Apr 15 20:25" "Apr 15 20:35" +[32mcrm(live)history# [0;10minfo +Source: live +Created on: Thu Apr 16 11:32:36 CEST 2015 +By: report -Z -Q -f Wed Apr 15 20:25:00 2015 -t 2015-04-15 20:35:00 /var/cache/crm/history/live +Period: 2015-04-15 20:25:00 - 2015-04-15 20:35:00 +Nodes: [0;10msle12-a[0;10m [32msle12-c[0;10m +Groups: nfs-srv nfs-disk +Resources: s-libvirt p_drbd_nfs nfs-vg fs1 virtual-ip nfs-server websrv websrv-ip small-apache +Transitions: [32m1906[0;10m [32m1907[0;10m +[32mcrm(live)history# [0;10mpeinputs v +Date Start End Filename Client User Origin +==== ===== === ======== ====== ==== ====== +2015-04-15 20:29:59 20:30:01 [32mpe-input-1906[0;10m no-client no-user no-origin +2015-04-15 20:30:14 20:30:15 [32mpe-input-1907[0;10m no-client no-user no-origin diff --git a/doc/website-v1/include/history-guide/nfs-probe-err.typescript b/doc/website-v1/include/history-guide/nfs-probe-err.typescript new file mode 100644 index 0000000..ca34ba5 --- /dev/null +++ b/doc/website-v1/include/history-guide/nfs-probe-err.typescript @@ -0,0 +1,20 @@ +# crm history resource nfs-server +[36mINFO[0;10m: fetching new logs, please wait ... +[32mDec 16 11:53:23 sle12-c nfsserver(nfs-server)[14911]: <1> ERROR: NFS server is up, but the locking daemons are down[0;10m +[32mDec 16 11:53:23 sle12-c crmd[2823]: notice: te_rsc_command: Initiating action 54: stop nfs-server_stop_0 on sle12-a[0;10m +[32mDec 16 11:53:23 sle12-c crmd[2823]: notice: te_rsc_command: Initiating action 3: stop nfs-server_stop_0 on sle12-c (local)[0;10m +[32mDec 16 11:53:23 sle12-c nfsserver(nfs-server)[14944]: INFO: Stopping NFS server ...[0;10m +[32mDec 16 11:53:23 sle12-c nfsserver(nfs-server)[14944]: INFO: Stopping sm-notify[0;10m +[32mDec 16 11:53:23 sle12-c nfsserver(nfs-server)[14944]: INFO: Stopping rpc.statd[0;10m +[32mDec 16 11:53:23 sle12-c nfsserver(nfs-server)[14944]: INFO: NFS server stopped[0;10m +[32mDec 16 11:53:23 sle12-c crmd[2823]: notice: te_rsc_command: Initiating action 55: start nfs-server_start_0 on sle12-a[0;10m +[0;10mDec 16 11:53:23 sle12-a nfsserver(nfs-server)[23255]: INFO: Stopping NFS server ...[0;10m +[0;10mDec 16 11:53:23 sle12-a nfsserver(nfs-server)[23255]: INFO: Stopping sm-notify[0;10m +[0;10mDec 16 11:53:23 sle12-a nfsserver(nfs-server)[23255]: INFO: Stopping rpc.statd[0;10m +[0;10mDec 16 11:53:23 sle12-a nfsserver(nfs-server)[23255]: INFO: NFS server stopped[0;10m +[0;10mDec 16 11:53:23 sle12-a nfsserver(nfs-server)[23320]: INFO: Starting NFS server ...[0;10m +[0;10mDec 16 11:53:23 sle12-a nfsserver(nfs-server)[23320]: INFO: Starting rpc.statd.[0;10m +[0;10mDec 16 11:53:24 sle12-a nfsserver(nfs-server)[23320]: INFO: executing sm-notify[0;10m +[0;10mDec 16 11:53:24 sle12-a nfsserver(nfs-server)[23320]: INFO: NFS server started[0;10m +[0;10mDec 16 11:53:24 sle12-a lrmd[6904]: <2> notice: operation_finished: nfs-server_start_0:23320:stderr [ id: rpcuser: no such user ][0;10m +[0;10mDec 16 11:53:24 sle12-a lrmd[6904]: message repeated 3 times: [ notice: operation_finished: nfs-server_start_0:23320:stderr [ id: rpcuser: no such user ]][0;10m diff --git a/doc/website-v1/include/history-guide/resource-trace.typescript b/doc/website-v1/include/history-guide/resource-trace.typescript new file mode 100644 index 0000000..e66ff7c --- /dev/null +++ b/doc/website-v1/include/history-guide/resource-trace.typescript @@ -0,0 +1,7 @@ +# crm resource trace nfs-server monitor 0 +[36mINFO[0;10m: Trace for nfs-server:monitor is written to /var/lib/heartbeat/trace_ra/ +[36mINFO[0;10m: Trace set, restart nfs-server to trace non-monitor operations +# crm resource cleanup nfs-server +Cleaning up nfs-server on sle12-a +Cleaning up nfs-server on sle12-c +Waiting for 2 replies from the CRMd.. OK diff --git a/doc/website-v1/include/history-guide/resource.typescript b/doc/website-v1/include/history-guide/resource.typescript new file mode 100644 index 0000000..90f0265 --- /dev/null +++ b/doc/website-v1/include/history-guide/resource.typescript @@ -0,0 +1,6 @@ +[32mcrm(live)history# [0;10mresource small-apache +[32mApr 15 20:29:59 sle12-c crmd[1136]: notice: te_rsc_command: Initiating action 60: stop small-apache_stop_0 on sle12-a[0;10m +[0;10mApr 15 20:29:59 sle12-a apache(small-apache)[1366]: INFO: Attempting graceful stop of apache PID 9155[0;10m +[0;10mApr 15 20:30:01 sle12-a apache(small-apache)[1366]: INFO: apache stopped.[0;10m +[0;10mApr 15 20:30:14 sle12-a apache(small-apache)[1586]: INFO: AH00558: httpd2: Could not reliably determine the server's fully qualified domain name, using 10.2.12.51. Set the 'ServerName' directive globally to suppress this message[0;10m +[32mApr 15 20:30:14 sle12-c crmd[1136]: notice: te_rsc_command: Initiating action 60: start small-apache_start_0 on sle12-a[0;10m diff --git a/doc/website-v1/include/history-guide/sample-cluster.conf.crm b/doc/website-v1/include/history-guide/sample-cluster.conf.crm new file mode 100644 index 0000000..8b44663 --- /dev/null +++ b/doc/website-v1/include/history-guide/sample-cluster.conf.crm @@ -0,0 +1,54 @@ +node 167906357: sle12-c +node 167906355: sle12-a +primitive s-libvirt stonith:external/libvirt \ + params hostlist="sle12-a sle12-c" hypervisor_uri="qemu+ssh://hex-10.suse.de/system?keyfile=/root/.ssh/xen" reset_method=reboot \ + op monitor interval=5m timeout=60s +primitive p_drbd_nfs ocf:linbit:drbd \ + params drbd_resource=nfs \ + op monitor interval=15 role=Master \ + op monitor interval=30 role=Slave \ + op start interval=0 timeout=300 \ + op stop interval=0 timeout=120 +primitive nfs-vg LVM \ + params volgrpname=nfs-vg +primitive fs1 Filesystem \ + params device="/dev/nfs-vg/fs1" directory="/srv/nfs" fstype=ext3 \ + op monitor interval=30s +primitive virtual-ip IPaddr2 \ + params ip=10.2.12.100 +primitive nfs-server nfsserver \ + params nfs_shared_infodir="/srv/nfs/state" nfs_ip=10.2.12.100 \ + op monitor interval=30s +primitive websrv apache \ + params configfile="/etc/apache2/httpd.conf" \ + op monitor interval=30 +primitive websrv-ip IPaddr2 \ + params ip=10.2.12.101 +primitive small-apache apache \ + params configfile="/etc/apache2/small.conf" +group nfs-disk nfs-vg fs1 +group nfs-srv virtual-ip nfs-server +ms ms_drbd_nfs p_drbd_nfs \ + meta notify=true clone-max=2 +location nfs-pref virtual-ip 100: sle12-a +location websrv-pref websrv 100: sle12-c +colocation vg-with-drbd inf: nfs-vg ms_drbd_nfs:Master +colocation c-nfs inf: nfs-srv nfs-disk +colocation c-websrv inf: websrv websrv-ip +colocation small-apache-with-virtual-ip inf: small-apache virtual-ip +# need fs1 for the NFS server +order o-nfs inf: nfs-disk nfs-srv +# websrv serves requests at IP websrv-ip +order o-websrv inf: websrv-ip websrv +# small apache serves requests at IP virtual-ip +order virtual-ip-before-small-apache inf: virtual-ip small-apache +# drbd device is the nfs-vg PV +order drbd-before-nfs-vg inf: ms_drbd_nfs:promote nfs-vg:start +property cib-bootstrap-options: \ + dc-version=1.1.12-ad083a8 \ + cluster-infrastructure=corosync \ + cluster-name=sle12-test3l-public \ + no-quorum-policy=ignore \ + last-lrm-refresh=1429192263 +op_defaults op-options: \ + timeout=120s diff --git a/doc/website-v1/include/history-guide/status-probe-fail.typescript b/doc/website-v1/include/history-guide/status-probe-fail.typescript new file mode 100644 index 0000000..d1024e8 --- /dev/null +++ b/doc/website-v1/include/history-guide/status-probe-fail.typescript @@ -0,0 +1,15 @@ +# crm status +Last updated: Tue Dec 16 11:57:04 2014 +Last change: Tue Dec 16 11:53:22 2014 +Stack: corosync +Current DC: sle12-c (167906357) - partition with quorum +Version: 1.1.12-ad083a8 +2 Nodes configured +10 Resources configured +Online: [ sle12-a sle12-c ] +[...] + nfs-server (ocf::heartbeat:nfsserver): Started sle12-a +[...] +Failed actions: + nfs-server_monitor_0 on sle12-c 'unknown error' (1): call=298, status=complete, + last-rc-change='Tue Dec 16 11:53:23 2014', queued=0ms, exec=135ms diff --git a/doc/website-v1/include/history-guide/stonith-corosync-stopped.typescript b/doc/website-v1/include/history-guide/stonith-corosync-stopped.typescript new file mode 100644 index 0000000..1bca5ac --- /dev/null +++ b/doc/website-v1/include/history-guide/stonith-corosync-stopped.typescript @@ -0,0 +1,8 @@ +# crm history node sle12-c +[36mINFO[0;10m: fetching new logs, please wait ... +[32mDec 19 14:36:18 sle12-c corosync[29551]: [MAIN ] Corosync Cluster Engine ('2.3.3'): started and ready to provide service.[0;10m +[32mDec 19 14:36:19 sle12-c corosync[29545]: Starting Corosync Cluster Engine (corosync): [ OK ][0;10m +[0;10mDec 19 14:36:20 sle12-a pengine[6906]: warning: pe_fence_node: Node sle12-c will be fenced because our peer process is no longer available[0;10m +[0;10mDec 19 14:36:20 sle12-a pengine[6906]: warning: stage6: Scheduling Node sle12-c for STONITH[0;10m +[0;10mDec 19 14:36:20 sle12-a crmd[6907]: notice: te_fence_node: Executing reboot fencing operation (65) on sle12-c (timeout=60000)[0;10m +[0;10mDec 19 14:36:20 sle12-a crmd[6907]: notice: peer_update_callback: Node return implies stonith of sle12-c (action 65) completed[0;10m diff --git a/doc/website-v1/include/history-guide/transition-log.typescript b/doc/website-v1/include/history-guide/transition-log.typescript new file mode 100644 index 0000000..eb689ec --- /dev/null +++ b/doc/website-v1/include/history-guide/transition-log.typescript @@ -0,0 +1,13 @@ +[32mcrm(live)history# [0;10mtransition log +[36mINFO[0;10m: retrieving information from cluster nodes, please wait ... +[32mApr 15 20:30:14 sle12-c crmd[1136]: notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_FSA_INTERNAL origin=abort_transition_graph ][0;10m +[32mApr 15 20:30:14 sle12-c stonithd[1132]: notice: unpack_config: On loss of CCM Quorum: Ignore[0;10m +[32mApr 15 20:30:14 sle12-c pengine[1135]: notice: unpack_config: On loss of CCM Quorum: Ignore[0;10m +[32mApr 15 20:30:14 sle12-c pengine[1135]: notice: LogActions: Start small-apache#011(sle12-a)[0;10m +[32mApr 15 20:30:14 sle12-c crmd[1136]: notice: do_te_invoke: Processing graph 123 (ref=pe_calc-dc-1429122614-234) derived from /var/lib/pacemaker/pengine/pe-input-1907.bz2[0;10m +[32mApr 15 20:30:14 sle12-c crmd[1136]: notice: te_rsc_command: Initiating action 60: start small-apache_start_0 on sle12-a[0;10m +[32mApr 15 20:30:14 sle12-c pengine[1135]: notice: process_pe_message: Calculated Transition 123: /var/lib/pacemaker/pengine/pe-input-1907.bz2[0;10m +[0;10mApr 15 20:30:14 sle12-a stonithd[1160]: notice: unpack_config: On loss of CCM Quorum: Ignore[0;10m +[0;10mApr 15 20:30:14 sle12-a apache(small-apache)[1586]: INFO: AH00558: httpd2: Could not reliably determine the server's fully qualified domain name, using 10.2.12.51. Set the 'ServerName' directive globally to suppress this message[0;10m +[0;10mApr 15 20:30:14 sle12-a crmd[1164]: notice: process_lrm_event: Operation small-apache_start_0: ok (node=sle12-a, call=69, rc=0, cib-update=48, confirmed=true)[0;10m +[32mApr 15 20:30:15 sle12-c crmd[1136]: notice: run_graph: Transition 123 (Complete=1, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-1907.bz2): Complete[0;10m diff --git a/doc/website-v1/index.adoc b/doc/website-v1/index.adoc new file mode 100644 index 0000000..1ebd635 --- /dev/null +++ b/doc/website-v1/index.adoc @@ -0,0 +1,25 @@ +The CRM Shell +============= + +++++ +<div class="frontpage-image"> +<br><br> +<img src="/img/laptop.png"> +<br><br><br> +</div> +++++ + +*`crmsh` is a cluster management shell* for the Pacemaker High Availability stack. + +Configure, manage and troubleshoot clusters from the command line, +with full tab completion and extensive help. `crmsh` also provides +advanced features like low-level cluster configuration, cluster scripting, +package management, and history exploration tools giving you a complete +insight into the state of your cluster. + +* https://github.com/ClusterLabs/crmsh/[Source Code] +* http://crmsh.github.io/man-4.3/[Reference Manual (v4.3.1)] +* http://crmsh.github.io/man-3/[Reference Manual (v3.0.0)] +* http://crmsh.github.io/man-2.0/[Reference Manual (v2.3.2)] +* https://build.opensuse.org/package/show/network:ha-clustering:Stable/crmsh[Packages] +* http://clusterlabs.org[Cluster Labs] diff --git a/doc/website-v1/installation.adoc b/doc/website-v1/installation.adoc new file mode 100644 index 0000000..5a027a0 --- /dev/null +++ b/doc/website-v1/installation.adoc @@ -0,0 +1,4 @@ +Installation +============ + +See link:/download[Download]. diff --git a/doc/website-v1/make-news.py b/doc/website-v1/make-news.py new file mode 100644 index 0000000..f3c9073 --- /dev/null +++ b/doc/website-v1/make-news.py @@ -0,0 +1,136 @@ +#!/usr/bin/env python +""" +Output a combined news.adoc document +Also write an Atom feed document +""" + +import os +import sys +import hashlib +import datetime +import time + +OUTPUT_HEADER = """= News + +""" +OUTPUT_FOOTER = """ +link:https://savannah.nongnu.org/news/?group_id=10890[Old News Archive] +""" + +ATOM_TEMPLATE = """<?xml version="1.0" encoding="utf-8"?> +<feed xmlns="http://www.w3.org/2005/Atom"> +<title>crmsh</title> +<subtitle>Cluster manager shell news</subtitle> +<link href="http://crmsh.github.io/atom.xml" rel="self" /> +<link href="http://crmsh.github.io/" /> +<id>%(id)s</id> +<updated>%(updated)s</updated> +%(entries)s +</feed> +""" + +ATOM_NAME = "gen/atom.xml" + +root_id = "tag:crmsh.github.io,2014:/atom" + +def escape(s): + s = s.replace('&', '&') + s = s.replace('<', '<') + s = s.replace('>', '>') + s = s.replace('"', """) + return s + +class Entry(object): + def __init__(self, fname): + self.filename = fname + self.name = os.path.splitext(os.path.basename(fname))[0] + with open(fname) as f: + self.title = f.readline().strip() + f.readline() + l = f.readline() + while l.startswith(':'): + k, v = l[1:].split(':', 1) + k = k.lower() + v = v.strip() + setattr(self, k, v) + l = f.readline() + self.content = l + f.read() + if not hasattr(self, 'author'): + raise ValueError("Missing author") + if not hasattr(self, 'email'): + raise ValueError("Missing email") + if not hasattr(self, 'date'): + raise ValueError("Missing date") + + def atom_id(self): + return root_id + '::' + hashlib.sha1(self.filename).hexdigest() + + def atom_date(self): + return self.date.replace(' ', 'T') + ':00' + time.tzname[0] + + def date_obj(self): + from dateutil import parser + return (parser.parse(self.date)) + + def atom_content(self): + return escape('<pre>\n' + self.content + '\n</pre>\n') + + def atom(self): + data = {'title': self.title, + 'id': self.atom_id(), + 'updated': self.atom_date(), + 'name': self.name, + 'content': self.atom_content(), + 'author': self.author, + 'email': self.email} + return """<entry> +<title>%(title)s</title> +<id>%(id)s</id> +<updated>%(updated)s</updated> +<link>http://crmsh.github.io/news/%(name)s</link> +<content type="html"> +%(content)s +</content> +<author> +<name>%(author)s</name> +<email>%(email)s</email> +</author> +</entry> +""" % data + + +def sort_entries(entries): + return list(reversed(sorted(entries, key=lambda e: e.date_obj()))) + + +def make_atom(): + inputs = sort_entries([Entry(f) for f in sys.argv[2:]]) + with open(ATOM_NAME, 'w') as output: + output.write(ATOM_TEMPLATE % { + 'id': root_id, + 'updated': inputs[0].atom_date(), + 'entries': '\n'.join(f.atom() for f in inputs) + }) + + +def main(): + # TODO: sort by date + inputs = sort_entries([Entry(f) for f in sys.argv[2:]]) + with open(sys.argv[1], 'w') as output: + output.write(OUTPUT_HEADER) + e = inputs[0] + output.write("link:/news/%s[%s]\n\n" % (e.name, e.date)) + output.write(":leveloffset: 1\n\n") + output.write("include::%s[]\n\n" % (e.filename)) + output.write(":leveloffset: 0\n\n") + + output.write("''''\n") + for e in inputs[1:]: + output.write("* link:/news/%s[%s %s]\n" % (e.name, e.date, e.title)) + output.write(OUTPUT_FOOTER) + +if __name__ == "__main__": + if sys.argv[1] == ATOM_NAME: + make_atom() + else: + main() diff --git a/doc/website-v1/man-1.2.adoc b/doc/website-v1/man-1.2.adoc new file mode 100644 index 0000000..d945719 --- /dev/null +++ b/doc/website-v1/man-1.2.adoc @@ -0,0 +1,3437 @@ +:man source: crm +:man version: 1.2.6 +:man manual: crmsh documentation + +crm(8) +====== + +NOTE: This is the documentation for stable release 1.2.6 of `crmsh`. + + +NAME +---- +crm - Pacemaker command line interface for configuration and management + + +SYNOPSIS +-------- +*crm* [-D output_type] [-f file] [-c cib] [-H hist_src] [-hFRDw] [--version] [args] + + +[[topics_Description,Program description]] +DESCRIPTION +----------- +Pacemaker configuration is stored in a CIB file (Cluster +Information Base). The CIB is a set of instructions coded in XML. +Editing the CIB is a challenge, not only due to its complexity +and a wide variety of options, but also because XML is more +computer than user friendly. The `crm` shell alleviates this +issue significantly by introducing small and simple configuration +language. The CIB is translated into this language on the fly. + +`crm` is also a management tool. For management tasks it relies +almost exclusively on other command line tools, such as +`crm_resource(8)` or `crm_attribute(8)`. Use of these programs +is, however, plagued by the notorious weakness common to all UNIX +tools: a multitude of options, necessary for operation and yet +very hard to remember. `crm` tries to present a consistent +interface to the user and to hide the arcane detail. + +It may be used either as an interactive shell or for single +commands directly on the shell's command line. It is also +possible to feed it a set of commands from standard input or a +file, thus turning it into a scripting tool. Templates with ready +made configurations may help newbies learn about the cluster +configuration or facilitate testing procedures. + +The `crm` shell is line oriented: every command must start and +finish on the same line. It is possible to use a continuation +character (`\`) to write one command in two or more lines. The +continuation character is commonly used when displaying +configurations. + +OPTIONS +------- +*-f, --file*='FILE':: + Load commands from the given file. If the file is `-` then + use terminal `stdin`. + +*-c, --cib*='CIB':: + Start the session with the given shadow CIB file. + Equivalent to `cib use`. + +*-D, --display=*'OUTPUT_TYPE':: + Choose one of the output options: `plain`, `color`, or + `uppercase`. The default is `color` if the terminal emulation + supports colors. Otherwise, `plain` is used. + +*-F, --force*:: + Make `crm` proceed with doing changes even though it would + normally ask user to confirm some of them. Mostly useful in + scripts. + +*-w, --wait*:: + Make `crm` wait for the cluster transition to finish (for the + changes to take effect) after each processed line. + +*-H, --history*='DIR|FILE':: + The `history` commands can examine either live cluster + (default) or a report generated by `hb_report`. Use this + option to specify a directory or file containing the report. + +*-h, --help*:: + Print help page. + +*--version*:: + Print crmsh version and build information (Mercurial Hg + changeset hash). + +*-R, --regression-tests*:: + Run in the regression test mode. Used mainly by the + regression testing suite. + +*-d, --debug*:: + Print some debug information. Used by developers. [Not yet + refined enough to print useful information for other users.] + +[[topics_Introduction,Introduction to the user interface]] +== Introduction to the user interface + +Arguably the most important aspect of `crm` is the user +interface. We begin with an informal introduction so that the +reader may get acquainted with it and get a general feeling of +the tool. It is probably best just to give some examples: + +1. Command line (one-shot) use: + + # crm resource stop www_app + +2. Interactive use: + + # crm + crm(live)# resource + crm(live)resource# unmanage tetris_1 + crm(live)resource# end + crm(live)# node standby node4 + +3. Cluster configuration: + + # crm configure<<EOF + # + # resources + # + primitive disk0 iscsi \ + params portal=192.168.2.108:3260 target=iqn.2008-07.com.suse:disk0 + primitive fs0 Filesystem \ + params device=/dev/disk/by-label/disk0 directory=/disk0 fstype=ext3 + primitive internal_ip IPaddr params ip=192.168.1.101 + primitive apache apache \ + params configfile=/disk0/etc/apache2/site0.conf + primitive apcfence stonith:apcsmart \ + params ttydev=/dev/ttyS0 hostlist="node1 node2" \ + op start timeout=60s + primitive pingd pingd \ + params name=pingd dampen=5s multiplier=100 host_list="r1 r2" + # + # monitor apache and the UPS + # + monitor apache 60s:30s + monitor apcfence 120m:60s + # + # cluster layout + # + group internal_www \ + disk0 fs0 internal_ip apache + clone fence apcfence \ + meta globally-unique=false clone-max=2 clone-node-max=1 + clone conn pingd \ + meta globally-unique=false clone-max=2 clone-node-max=1 + location node_pref internal_www \ + rule 50: #uname eq node1 \ + rule pingd: defined pingd + # + # cluster properties + # + property stonith-enabled=true + commit + EOF + +If you've ever done a CRM style configuration, you should be able +to understand the above examples without much difficulties. The +shell should provide a means to manage the cluster efficiently or +put together a configuration in a concise manner. + +The `(live)` string in the prompt signifies that the current CIB +in use is the cluster live configuration. It is also possible to +work with the so-called shadow CIBs, i.e. configurations which +are stored in files and aren't active, but may be applied at any +time to the cluster. + +Since the CIB is hierarchical such is the interface too. There +are several levels and entering each of them enables the user to +use a certain set of commands. + +[[topics_Shadows,Shadow CIB usage]] +== Shadow CIB usage + +Shadow CIB is a normal cluster configuration stored in a file. +They may be manipulated in the same way like the _live_ CIB, but +these changes have no effect on the cluster resources. The +administrator may choose to apply any of them to the cluster, +thus replacing the running configuration with the one which is in +the shadow CIB. The `crm` prompt always contains the name of the +configuration which is currently in use or string _live_ if we +are using the current cluster configuration. + +At the configure level no changes take place before the `commit` +command. Sometimes though, the administrator may start working +with the running configuration, but change mind and instead of +committing the changes to the cluster save them to a shadow CIB. +This short `configure` session excerpt shows how: +............... + crm(live)configure# cib new test-2 + INFO: test-2 shadow CIB created + crm(test-2)configure# commit +............... + +[[topics_Templates,Configuration templates]] +== Configuration templates + +Configuration templates are ready made configurations created by +cluster experts. They are designed in such a way so that users +may generate valid cluster configurations with minimum effort. +If you are new to Pacemaker, templates may be the best way to +start. + +We will show here how to create a simple yet functional Apache +configuration: +............... + # crm configure + crm(live)configure# template + crm(live)configure template# list templates + apache filesystem virtual-ip + crm(live)configure template# new web <TAB><TAB> + apache filesystem virtual-ip + crm(live)configure template# new web apache + INFO: pulling in template apache + INFO: pulling in template virtual-ip + crm(live)configure template# list + web2-d web2 vip2 web3 vip web +............... + +We enter the `template` level from `configure`. Use the `list` +command to show templates available on the system. The `new` +command creates a configuration from the `apache` template. You +can use tab completion to pick templates. Note that the apache +template depends on a virtual IP address which is automatically +pulled along. The `list` command shows the just created `web` +configuration, among other configurations (I hope that you, +unlike me, will use more sensible and descriptive names). + +The `show` command, which displays the resulting configuration, +may be used to get an idea about the minimum required changes +which have to be done. All `ERROR` messages show the line numbers +in which the respective parameters are to be defined: +............... + crm(live)configure template# show + ERROR: 23: required parameter ip not set + ERROR: 61: required parameter id not set + ERROR: 65: required parameter configfile not set + crm(live)configure template# edit +............... + +The `edit` command invokes the preferred text editor with the +`web` configuration. At the top of the file, the user is advised +how to make changes. A good template should require from the user +to specify only parameters. For example, the `web` configuration +we created above has the following required and optional +parameters (all parameter lines start with `%%`): +............... + $ grep -n ^%% ~/.crmconf/web + 23:%% ip + 31:%% netmask + 35:%% lvs_support + 61:%% id + 65:%% configfile + 71:%% options + 76:%% envfiles +............... + +These lines are the only ones that should be modified. Simply +append the parameter value at the end of the line. For instance, +after editing this template, the result could look like this (we +used tabs instead of spaces to make the values stand out): +............... + $ grep -n ^%% ~/.crmconf/web + 23:%% ip 192.168.1.101 + 31:%% netmask + 35:%% lvs_support + 61:%% id websvc + 65:%% configfile /etc/apache2/httpd.conf + 71:%% options + 76:%% envfiles +............... + +As you can see, the parameter line format is very simple: +............... + %% <name> <value> +............... + +After editing the file, use `show` again to display the +configuration: +............... + crm(live)configure template# show + primitive virtual-ip ocf:heartbeat:IPaddr \ + params ip="192.168.1.101" + primitive apache ocf:heartbeat:apache \ + params configfile="/etc/apache2/httpd.conf" + monitor apache 120s:60s + group websvc \ + apache virtual-ip +............... + +The target resource of the apache template is a group which we +named `websvc` in this sample session. + +This configuration looks exactly as you could type it at the +`configure` level. The point of templates is to save you some +typing. It is important, however, to understand the configuration +produced. + +Finally, the configuration may be applied to the current +crm configuration (note how the configuration changed slightly, +though it is still equivalent, after being digested at the +`configure` level): +............... + crm(live)configure template# apply + crm(live)configure template# cd .. + crm(live)configure# show + node xen-b + node xen-c + primitive apache ocf:heartbeat:apache \ + params configfile="/etc/apache2/httpd.conf" \ + op monitor interval="120s" timeout="60s" + primitive virtual-ip ocf:heartbeat:IPaddr \ + params ip="192.168.1.101" + group websvc apache virtual-ip +............... + +Note that this still does not commit the configuration to the CIB +which is used in the shell, either the running one (`live`) or +some shadow CIB. For that you still need to execute the `commit` +command. + +To complete our example, we should also define the preferred node +to run the service: +............... + crm(live)configure# location websvc-pref websvc 100: xen-b +............... + +If you are not happy with some resource names which are provided +by default, you can rename them now: +............... + crm(live)configure# rename virtual-ip intranet-ip + crm(live)configure# show + node xen-b + node xen-c + primitive apache ocf:heartbeat:apache \ + params configfile="/etc/apache2/httpd.conf" \ + op monitor interval="120s" timeout="60s" + primitive intranet-ip ocf:heartbeat:IPaddr \ + params ip="192.168.1.101" + group websvc apache intranet-ip + location websvc-pref websvc 100: xen-b +............... + +To summarize, working with templates typically consists of the +following steps: + +- `new`: create a new configuration from templates +- `edit`: define parameters, at least the required ones +- `show`: see if the configuration is valid +- `apply`: apply the configuration to the `configure` level + +[[topics_Testing,Resource testing]] +== Resource testing + +The amount of detail in a cluster makes all configurations prone +to errors. By far the largest number of issues in a cluster is +due to bad resource configuration. The shell can help quickly +diagnose such problems. And considerably reduce your keyboard +wear. + +Let's say that we entered the following configuration: +............... + node xen-b + node xen-c + node xen-d + primitive fencer stonith:external/libvirt \ + params hypervisor_uri="qemu+tcp://10.2.13.1/system" \ + hostlist="xen-b xen-c xen-d" \ + op monitor interval="2h" + primitive svc ocf:heartbeat:Xinetd \ + params service="systat" \ + op monitor interval="30s" + primitive intranet-ip ocf:heartbeat:IPaddr2 \ + params ip="10.2.13.100" \ + op monitor interval="30s" + primitive apache ocf:heartbeat:apache \ + params configfile="/etc/apache2/httpd.conf" \ + op monitor interval="120s" timeout="60s" + group websvc apache intranet-ip + location websvc-pref websvc 100: xen-b +............... + +Before typing `commit` to submit the configuration to the cib we +can make sure that all resources are usable on all nodes: +............... + crm(live)configure# rsctest websvc svc fencer +............... + +It is important that resources being tested are not running on +any nodes. Otherwise, the `rsctest` command will refuse to do +anything. Of course, if the current configuration resides in a +CIB shadow, then a `commit` is irrelevant. The point being that +resources are not running on any node. + +.Note on stopping all resources +**************************** +Alternatively to not committing a configuration, it is also +possible to tell Pacemaker not to start any resources: + +............... + crm(live)configure# property stop-all-resources="yes" +............... +Almost none---resources of class stonith are still started. But +shell is not as strict when it comes to stonith resources. +**************************** + +Order of resources is significant insofar that a resource depends +on all resources to its left. In most configurations, it's +probably practical to test resources in several runs, based on +their dependencies. + +Apart from groups, `crm` does not interpret constraints and +therefore knows nothing about resource dependencies. It also +doesn't know if a resource can run on a node at all in case of an +asymmetric cluster. It is up to the user to specify a list of +eligible nodes if a resource is not meant to run on every node. + +[[topics_Completion,Tab completion]] +== Tab completion + +The `crm` makes extensive use of tab completion. The completion +is both static (i.e. for `crm` commands) and dynamic. The latter +takes into account the current status of the cluster or +information from installed resource agents. Sometimes, completion +may also be used to get short help on resource parameters. Here a +few examples: +............... + crm(live)# resource + crm(live)resource# <TAB><TAB> + bye failcount move restart unmigrate + cd help param show unmove + cleanup list promote start up + demote manage quit status utilization + end meta refresh stop + exit migrate reprobe unmanage + crm(live)resource# end + crm(live)# configure + crm(live)configure# primitive fence-1 <TAB><TAB> + heartbeat: lsb: ocf: stonith: + crm(live)configure# primitive fence-1 stonith:<TAB><TAB> + apcmaster external/ippower9258 fence_legacy + apcmastersnmp external/kdumpcheck ibmhmc + apcsmart external/libvirt ipmilan + baytech external/nut meatware + bladehpi external/rackpdu null + cyclades external/riloe nw_rpc100s + drac3 external/sbd rcd_serial + external/drac5 external/ssh rps10 + external/dracmc-telnet external/ssh-bad ssh + external/hmchttp external/ssh-slow suicide + external/ibmrsa external/vmware wti_mpc + external/ibmrsa-telnet external/xen0 wti_nps + external/ipmi external/xen0-ha + crm(live)configure# primitive fence-1 stonith:ipmilan params <TAB><TAB> + auth= hostname= ipaddr= login= password= port= priv= + crm(live)configure# primitive fence-1 stonith:ipmilan params auth=<TAB><TAB> + auth* (string) + The authorization type of the IPMI session ("none", "straight", "md2", or "md5") + crm(live)configure# primitive fence-1 stonith:ipmilan params auth= +............... + +[[topics_Checks,Configuration semantic checks]] +== Configuration semantic checks + +Resource definitions may be checked against the meta-data +provided with the resource agents. These checks are currently +carried out: + +- are required parameters set +- existence of defined parameters +- timeout values for operations + +The parameter checks are obvious and need no further explanation. +Failures in these checks are treated as configuration errors. + +The timeouts for operations should be at least as long as those +recommended in the meta-data. Too short timeout values are a +common mistake in cluster configurations and, even worse, they +often slip through if cluster testing was not thorough. Though +operation timeouts issues are treated as warnings, make sure that +the timeouts are usable in your environment. Note also that the +values given are just _advisory minimum_---your resources may +require longer timeouts. + +User may tune the frequency of checks and the treatment of errors +by the <<cmdhelp_options_check-frequency,`check-frequency`>> and +<<cmdhelp_options_check-mode,`check-mode`>> preferences. + +Note that if the `check-frequency` is set to `always` and the +`check-mode` to `strict`, errors are not tolerated and such +configuration cannot be saved. + +[[topics_Security,Access Control Lists (ACL)]] +== Access Control Lists (ACL) + +By default, the users from the `haclient` group have full access +to the cluster (or, more precisely, to the CIB). Access control +lists allow for finer access control to the cluster. + +Access control lists consist of an ordered set of access rules. +Each rule allows read or write access or denies access +completely. Rules are typically combined to produce a specific +role. Then, users may be assigned a role. + +For instance, this is a role which defines a set of rules +allowing management of a single resource: + +............... + role bigdb_admin \ + write meta:bigdb:target-role \ + write meta:bigdb:is-managed \ + write location:bigdb \ + read ref:bigdb +............... + +The first two rules allow modifying the `target-role` and +`is-managed` meta attributes which effectively enables users in +this role to stop/start and manage/unmanage the resource. The +constraints write access rule allows moving the resource around. +Finally, the user is granted read access to the resource +definition. + +For proper operation of all Pacemaker programs, it is advisable +to add the following role to all users: + +............... + role read_all \ + read cib +............... + +For finer grained read access try with the rules listed in the +following role: + +............... + role basic_read \ + read node attribute:uname \ + read node attribute:type \ + read property \ + read status +............... + +It is however possible that some Pacemaker programs (e.g. +`ptest`) may not function correctly if the whole CIB is not +readable. + +Some of the ACL rules in the examples above are expanded by the +shell to XPath specifications. For instance, +`meta:bigdb:target-role` is a shortcut for +`//primitive[@id='bigdb']/meta_attributes/nvpair[@name='target-role']`. +You can see the expansion by showing XML: + +............... + crm(live) configure# show xml bigdb_admin + ... + <acls> + <acl_role id="bigdb_admin"> + <write id="bigdb_admin-write" + xpath="//primitive[@id='bigdb']/meta_attributes/nvpair[@name='target-role']"/> +............... + +Many different XPath expressions can have equal meaning. For +instance, the following two are equal, but only the first one is +going to be recognized as shortcut: + +............... + //primitive[@id='bigdb']/meta_attributes/nvpair[@name='target-role'] + //resources/primitive[@id='bigdb']/meta_attributes/nvpair[@name='target-role'] +............... + +XPath is a powerful language, but you should try to keep your ACL +xpaths simple and the builtin shortcuts should be used whenever +possible. + +[[topics_Reference,Command reference]] +== Command reference + +We define a small and simple language. Most commands consist of +just a list of simple tokens. The only complex constructs are +found at the `configure` level. + +The syntax is described in a somewhat informal manner: `<>` +denotes a string, `[]` means that the construct is optional, the +ellipsis (`...`) signifies that the previous construct may be +repeated, `|` means pick one of many, and the rest are literals +(strings, `:`, `=`). + +=== `status` + +Show cluster status. The status is displayed by `crm_mon`. Supply +additional arguments for more information or different format. +See `crm_mon(8)` for more details. + +Usage: +............... + status [<option> ...] + + option :: bynode | inactive | ops | timing | failcounts +............... + +[[cmdhelp_cib,CIB shadow management]] +=== `cib` (shadow CIBs) + +This level is for management of shadow CIBs. It is available both +at the top level and the `configure` level. + +All the commands are implemented using `cib_shadow(8)` and the +`CIB_shadow` environment variable. The user prompt always +includes the name of the currently active shadow or the live CIB. + +[[cmdhelp_cib_new,create a new shadow CIB]] +==== `new` + +Create a new shadow CIB. The live cluster configuration and +status is copied to the shadow CIB. Specify `withstatus` if you +want to edit the status section of the shadow CIB (see the +<<cmdhelp_cibstatus,cibstatus section>>). Add `force` to force overwriting the +existing shadow CIB. + +To start with an empty configuration that is not copied from the live +CIB, specify the `empty` keyword. (This also allows a shadow CIB to be +created in case no cluster is running.) + +Usage: +............... + new <cib> [withstatus] [force] [empty] +............... + +[[cmdhelp_cib_delete,delete a shadow CIB]] +==== `delete` + +Delete an existing shadow CIB. + +Usage: +............... + delete <cib> +............... + +[[cmdhelp_cib_reset,copy live cib to a shadow CIB]] +==== `reset` + +Copy the current cluster configuration into the shadow CIB. + +Usage: +............... + reset <cib> +............... + +[[cmdhelp_cib_commit,copy a shadow CIB to the cluster]] +==== `commit` + +Apply a shadow CIB to the cluster. + +Usage: +............... + commit <cib> +............... + +[[cmdhelp_cib_use,change working CIB]] +==== `use` + +Choose a CIB source. If you want to edit the status from the +shadow CIB specify `withstatus` (see <<cmdhelp_cibstatus,`cibstatus`>>). +Leave out the CIB name to switch to the running CIB. + +Usage: +............... + use [<cib>] [withstatus] +............... + +[[cmdhelp_cib_diff,diff between the shadow CIB and the live CIB]] +==== `diff` + +Print differences between the current cluster configuration and +the active shadow CIB. + +Usage: +............... + diff +............... + +[[cmdhelp_cib_list,list all shadow CIBs]] +==== `list` + +List existing shadow CIBs. + +Usage: +............... + list +............... + +[[cmdhelp_cib_import,import a CIB or PE input file to a shadow]] +==== `import` + +At times it may be useful to create a shadow file from the +existing CIB. The CIB may be specified as file or as a PE input +file number. The shell will look up files in the local directory +first and then in the PE directory (typically `/var/lib/pengine`). +Once the CIB file is found, it is copied to a shadow and this +shadow is immediately available for use at both `configure` and +`cibstatus` levels. + +If the shadow name is omitted then the target shadow is named +after the input CIB file. + +Note that there are often more than one PE input file, so you may +need to specify the full name. + +Usage: +............... + import {<file>|<number>} [<shadow>] +............... +Examples: +............... + import pe-warn-2222 + import 2289 issue2 +............... + +[[cmdhelp_cib_cibstatus,CIB status management and editing]] +==== `cibstatus` + +Enter edit and manage the CIB status section level. See the +<<cmdhelp_cibstatus,CIB status management section>>. + +[[cmdhelp_ra,Resource Agents (RA) lists and documentation]] +=== `ra` + +This level contains commands which show various information about +the installed resource agents. It is available both at the top +level and at the `configure` level. + +[[cmdhelp_ra_classes,list classes and providers]] +==== `classes` + +Print all resource agents' classes and, where appropriate, a list +of available providers. + +Usage: +............... + classes +............... + +[[cmdhelp_ra_list,list RA for a class (and provider)]] +==== `list` + +List available resource agents for the given class. If the class +is `ocf`, supply a provider to get agents which are available +only from that provider. + +Usage: +............... + list <class> [<provider>] +............... +Example: +............... + list ocf pacemaker +............... + +[[cmdhelp_ra_meta,show meta data for a RA]] +==== `meta` (`info`) + +Show the meta-data of a resource agent type. This is where users +can find information on how to use a resource agent. It is also +possible to get information from some programs: `pengine`, +`crmd`, `cib`, and `stonithd`. Just specify the program name +instead of an RA. + +Usage: +............... + info [<class>:[<provider>:]]<type> + info <type> <class> [<provider>] (obsolete) +............... +Example: +............... + info apache + info ocf:pacemaker:Dummy + info stonith:ipmilan + info pengine +............... + +[[cmdhelp_ra_providers,show providers for a RA and a class]] +==== `providers` + +List providers for a resource agent type. The class parameter +defaults to `ocf`. + +Usage: +............... + providers <type> [<class>] +............... +Example: +............... + providers apache +............... + +[[cmdhelp_resource,Resource management]] +=== `resource` + +At this level resources may be managed. + +All (or almost all) commands are implemented with the CRM tools +such as `crm_resource(8)`. + +[[cmdhelp_resource_status,show status of resources]] +==== `status` (`show`, `list`) + +Print resource status. If the resource parameter is left out +status of all resources is printed. + +Usage: +............... + status [<rsc>] +............... + +[[cmdhelp_resource_start,start a resource]] +==== `start` + +Start a resource by setting the `target-role` attribute. If there +are multiple meta attributes sets, the attribute is set in all of +them. If the resource is a clone, all `target-role` attributes +are removed from the children resources. + +For details on group management see <<cmdhelp_options_manage-children,`options manage-children`>>. + +Usage: +............... + start <rsc> +............... + +[[cmdhelp_resource_stop,stop a resource]] +==== `stop` + +Stop a resource using the `target-role` attribute. If there +are multiple meta attributes sets, the attribute is set in all of +them. If the resource is a clone, all `target-role` attributes +are removed from the children resources. + +For details on group management see <<cmdhelp_options_manage-children,`options manage-children`>>. + +Usage: +............... + stop <rsc> +............... + +[[cmdhelp_resource_restart,restart a resource]] +==== `restart` + +Restart a resource. This is essentially a shortcut for resource +stop followed by a start. The shell is first going to wait for +the stop to finish, that is for all resources to really stop, and +only then to order the start action. Due to this command +entailing a whole set of operations, informational messages are +printed to let the user see some progress. + +For details on group management see <<cmdhelp_options_manage-children,`options manage-children`>>. + +Usage: +............... + restart <rsc> +............... +Example: +............... + # crm resource restart g_webserver + INFO: ordering g_webserver to stop + waiting for stop to finish .... done + INFO: ordering g_webserver to start + # +............... + +[[cmdhelp_resource_promote,promote a master-slave resource]] +==== `promote` + +Promote a master-slave resource using the `target-role` +attribute. + +Usage: +............... + promote <rsc> +............... + +[[cmdhelp_resource_demote,demote a master-slave resource]] +==== `demote` + +Demote a master-slave resource using the `target-role` +attribute. + +Usage: +............... + demote <rsc> +............... + +[[cmdhelp_resource_manage,put a resource into managed mode]] +==== `manage` + +Manage a resource using the `is-managed` attribute. If there +are multiple meta attributes sets, the attribute is set in all of +them. If the resource is a clone, all `is-managed` attributes are +removed from the children resources. + +For details on group management see <<cmdhelp_options_manage-children,`options manage-children`>>. + +Usage: +............... + manage <rsc> +............... + +[[cmdhelp_resource_unmanage,put a resource into unmanaged mode]] +==== `unmanage` + +Unmanage a resource using the `is-managed` attribute. If there +are multiple meta attributes sets, the attribute is set in all of +them. If the resource is a clone, all `is-managed` attributes are +removed from the children resources. + +For details on group management see <<cmdhelp_options_manage-children,`options manage-children`>>. + +Usage: +............... + unmanage <rsc> +............... + +[[cmdhelp_resource_migrate,migrate a resource to another node]] +==== `migrate` (`move`) + +Migrate a resource to a different node. If node is left out, the +resource is migrated by creating a constraint which prevents it from +running on the current node. Additionally, you may specify a +lifetime for the constraint---once it expires, the location +constraint will no longer be active. + +Usage: +............... + migrate <rsc> [<node>] [<lifetime>] [force] +............... + +[[cmdhelp_resource_unmigrate,unmigrate a resource to another node]] +==== `unmigrate` (`unmove`) + +Remove the constraint generated by the previous migrate command. + +Usage: +............... + unmigrate <rsc> +............... + +[[cmdhelp_resource_param,manage a parameter of a resource]] +==== `param` + +Show/edit/delete a parameter of a resource. + +Usage: +............... + param <rsc> set <param> <value> + param <rsc> delete <param> + param <rsc> show <param> +............... +Example: +............... + param ip_0 show ip +............... + +[[cmdhelp_resource_secret,manage sensitive parameters]] +==== `secret` + +Sensitive parameters can be kept in local files rather than CIB +in order to prevent accidental data exposure. Use the `secret` +command to manage such parameters. `stash` and `unstash` move the +value from the CIB and back to the CIB respectively. The `set` +subcommand sets the parameter to the provided value. `delete` +removes the parameter completely. `show` displays the value of +the parameter from the local file. Use `check` to verify if the +local file content is valid. + +Usage: +............... + secret <rsc> set <param> <value> + secret <rsc> stash <param> + secret <rsc> unstash <param> + secret <rsc> delete <param> + secret <rsc> show <param> + secret <rsc> check <param> +............... +Example: +............... + secret fence_1 show password + secret fence_1 stash password + secret fence_1 set password secret_value +............... + +[[cmdhelp_resource_meta,manage a meta attribute]] +==== `meta` + +Show/edit/delete a meta attribute of a resource. Currently, all +meta attributes of a resource may be managed with other commands +such as `resource stop`. + +Usage: +............... + meta <rsc> set <attr> <value> + meta <rsc> delete <attr> + meta <rsc> show <attr> +............... +Example: +............... + meta ip_0 set target-role stopped +............... + +[[cmdhelp_resource_utilization,manage a utilization attribute]] +==== `utilization` + +Show/edit/delete a utilization attribute of a resource. These +attributes describe hardware requirements. By setting the +`placement-strategy` cluster property appropriately, it is +possible then to distribute resources based on resource +requirements and node size. See also <<cmdhelp_node_utilization,node utilization attributes>>. + +Usage: +............... + utilization <rsc> set <attr> <value> + utilization <rsc> delete <attr> + utilization <rsc> show <attr> +............... +Example: +............... + utilization xen1 set memory 4096 +............... + +[[cmdhelp_resource_failcount,manage failcounts]] +==== `failcount` + +Show/edit/delete the failcount of a resource. + +Usage: +............... + failcount <rsc> set <node> <value> + failcount <rsc> delete <node> + failcount <rsc> show <node> +............... +Example: +............... + failcount fs_0 delete node2 +............... + +[[cmdhelp_resource_cleanup,cleanup resource status]] +==== `cleanup` + +Cleanup resource status. Typically done after the resource has +temporarily failed. If a node is omitted, cleanup on all nodes. +If there are many nodes, the command may take a while. + +Usage: +............... + cleanup <rsc> [<node>] +............... + +[[cmdhelp_resource_refresh,refresh CIB from the LRM status]] +==== `refresh` + +Refresh CIB from the LRM status. + +Usage: +............... + refresh [<node>] +............... + +[[cmdhelp_resource_reprobe,probe for resources not started by the CRM]] +==== `reprobe` + +Probe for resources not started by the CRM. + +Usage: +............... + reprobe [<node>] +............... + +[[cmdhelp_resource_trace,start RA tracing]] +==== `trace` + +Start tracing RA for the given operation. The trace files are +stored in `$HA_VARLIB/trace_ra`. If the operation to be traced is +monitor, note that the number of trace files can grow very +quickly. + +Usage: +............... + trace <rsc> <op> [<interval>] +............... +Example: +............... + trace fs start +............... + +[[cmdhelp_resource_untrace,stop RA tracing]] +==== `untrace` + +Stop tracing RA for the given operation. + +Usage: +............... + untrace <rsc> <op> [<interval>] +............... +Example: +............... + untrace fs start +............... + +[[cmdhelp_node,Nodes management]] +=== `node` + +Node management and status commands. + +[[cmdhelp_node_status,show nodes' status as XML]] +==== `status` + +Show nodes' status as XML. If the node parameter is omitted then +all nodes are shown. + +Usage: +............... + status [<node>] +............... + +[[cmdhelp_node_show,show node]] +==== `show` + +Show a node definition. If the node parameter is omitted then all +nodes are shown. + +Usage: +............... + show [<node>] +............... + +[[cmdhelp_node_standby,put node into standby]] +==== `standby` + +Set a node to standby status. The node parameter defaults to the +node where the command is run. Additionally, you may specify a +lifetime for the standby---if set to `reboot`, the node will be +back online once it reboots. `forever` will keep the node in +standby after reboot. + +Usage: +............... + standby [<node>] [<lifetime>] + + lifetime :: reboot | forever +............... + +[[cmdhelp_node_online,set node online]] +==== `online` + +Set a node to online status. The node parameter +defaults to the node where the command is run. + +Usage: +............... + online [<node>] +............... + +[[cmdhelp_node_maintenance,put node into maintenance mode]] +==== `maintenance` + +Set the node status to maintenance. This is equivalent to the +cluster-wide `maintenance-mode` property but puts just one node +into the maintenance mode. The node parameter defaults to the +node where the command is run. + +Usage: +............... + maintenance [<node>] +............... + +[[cmdhelp_node_ready,put node into ready mode]] +==== `ready` + +Set the node's maintenance status to `off`. The node should be +now again fully operational and capable of running resource +operations. + +Usage: +............... + ready [<node>] +............... + +[[cmdhelp_node_fence,fence node]] +==== `fence` + +Make CRM fence a node. This functionality depends on stonith +resources capable of fencing the specified node. No such stonith +resources, no fencing will happen. + +Usage: +............... + fence <node> +............... + +[[cmdhelp_node_clearstate,Clear node state]] +==== `clearnodestate` + +Resets and clears the state of the specified node. This node is +afterwards assumed clean and offline. This command can be used to +manually confirm that a node has been fenced (e.g., powered off). + +Be careful! This can cause data corruption if you confirm that a node is +down that is, in fact, not cleanly down - the cluster will proceed as if +the fence had succeeded, possibly starting resources multiple times. + +Usage: +............... + clearstate <node> +............... + +[[cmdhelp_node_delete,delete node]] +==== `delete` + +Delete a node. This command will remove the node from the CIB +and, in case the cluster stack is running, use the appropriate +program (`crm_node` or `hb_delnode`) to remove the node from the +membership. + +If the node is still listed as active and a member of our +partition we refuse to remove it. With the global force option +(`-F`) we will try to delete the node anyway. + +Usage: +............... + delete <node> +............... + +[[cmdhelp_node_attribute,manage attributes]] +==== `attribute` + +Edit node attributes. This kind of attribute should refer to +relatively static properties, such as memory size. + +Usage: +............... + attribute <node> set <attr> <value> + attribute <node> delete <attr> + attribute <node> show <attr> +............... +Example: +............... + attribute node_1 set memory_size 4096 +............... + +[[cmdhelp_node_utilization,manage utilization attributes]] +==== `utilization` + +Edit node utilization attributes. These attributes describe +hardware characteristics as integer numbers such as memory size +or the number of CPUs. By setting the `placement-strategy` +cluster property appropriately, it is possible then to distribute +resources based on resource requirements and node size. See also +<<cmdhelp_resource_utilization,resource utilization attributes>>. + +Usage: +............... + utilization <node> set <attr> <value> + utilization <node> delete <attr> + utilization <node> show <attr> +............... +Examples: +............... + utilization node_1 set memory 16384 + utilization node_1 show cpu +............... + +[[cmdhelp_node_status-attr,manage status attributes]] +==== `status-attr` + +Edit node attributes which are in the CIB status section, i.e. +attributes which hold properties of a more volatile nature. One +typical example is attribute generated by the `pingd` utility. + +Usage: +............... + status-attr <node> set <attr> <value> + status-attr <node> delete <attr> + status-attr <node> show <attr> +............... +Example: +............... + status-attr node_1 show pingd +............... + +[[cmdhelp_site,site support]] +=== `site` + +A cluster may consist of two or more subclusters in different and +distant locations. This set of commands supports such setups. + +[[cmdhelp_site_ticket,manage site tickets]] +==== `ticket` + +Tickets are cluster-wide attributes. They can be managed at the +site where this command is executed. + +It is then possible to constrain resources depending on the +ticket availability (see the <<cmdhelp_configure_rsc_ticket,`rsc_ticket`>> command +for more details). + +Usage: +............... + ticket {grant|revoke|standby|activate|show|time|delete} <ticket> +............... +Example: +............... + ticket grant ticket1 +............... + +[[cmdhelp_options,user preferences]] +=== `options` + +The user may set various options for the crm shell itself. + +[[cmdhelp_options_skill-level,set skill level]] +==== `skill-level` + +Based on the skill-level setting, the user is allowed to use only +a subset of commands. There are three levels: operator, +administrator, and expert. The operator level allows only +commands at the `resource` and `node` levels, but not editing +or deleting resources. The administrator may do that and may also +configure the cluster at the `configure` level and manage the +shadow CIBs. The expert may do all. + +Usage: +............... + skill-level <level> + + level :: operator | administrator | expert +............... + +.Note on security +**************************** +The `skill-level` option is advisory only. There is nothing +stopping any users change their skill level (see +<<topics_Security,Access Control Lists (ACL)>> on how to enforce +access control). +**************************** + +[[cmdhelp_options_user,set the cluster user]] +==== `user` + +Sufficient privileges are necessary in order to manage a +cluster: programs such as `crm_verify` or `crm_resource` and, +ultimately, `cibadmin` have to be run either as `root` or as the +CRM owner user (typically `hacluster`). You don't have to worry +about that if you run `crm` as `root`. A more secure way is to +run the program with your usual privileges, set this option to +the appropriate user (such as `hacluster`), and setup the +`sudoers` file. + +Usage: +............... + user system-user +............... +Example: +............... + user hacluster +............... + +[[cmdhelp_options_editor,set preferred editor program]] +==== `editor` + +The `edit` command invokes an editor. Use this to specify your +preferred editor program. If not set, it will default to either +the value of the `EDITOR` environment variable or to one of the +standard UNIX editors (`vi`,`emacs`,`nano`). + +Usage: +............... + editor program +............... +Example: +............... + editor vim +............... + +[[cmdhelp_options_pager,set preferred pager program]] +==== `pager` + +The `view` command displays text through a pager. Use this to +specify your preferred pager program. If not set, it will default +to either the value of the `PAGER` environment variable or to one +of the standard UNIX system pagers (`less`,`more`,`pg`). + +[[cmdhelp_options_sort-elements,sort CIB elements]] +==== `sort-elements` + +`crm` by default sorts CIB elements. If you want them appear in +the order they were created, set this option to `no`. + +Usage: +............... + sort-elements {yes|no} +............... +Example: +............... + sort-elements no +............... + +[[cmdhelp_options_wait,synchronous operation]] +==== `wait` + +In normal operation, `crm` runs a command and gets back +immediately to process other commands or get input from the user. +With this option set to `yes` it will wait for the started +transition to finish. In interactive mode dots are printed to +indicate progress. + +Usage: +............... + wait {yes|no} +............... +Example: +............... + wait yes +............... + +[[cmdhelp_options_output,set output type]] +==== `output` + +`crm` can adorn configurations in two ways: in color (similar to +for instance the `ls --color` command) and by showing keywords in +upper case. Possible values are `plain`, `color`, and +'uppercase'. It is possible to combine the latter two in order to +get an upper case xmass tree. Just set this option to +`color,uppercase`. + +[[cmdhelp_options_colorscheme,set colors for output]] +==== `colorscheme` + +With `output` set to `color`, a comma separated list of colors +from this option are used to emphasize: + +- keywords +- object ids +- attribute names +- attribute values +- scores +- resource references + +`crm` can show colors only if there is curses support for python +installed (usually provided by the `python-curses` package). The +colors are whatever is available in your terminal. Use `normal` +if you want to keep the default foreground color. + +This user preference defaults to +`yellow,normal,cyan,red,green,magenta` which is good for +terminals with dark background. You may want to change the color +scheme and save it in the preferences file for other color +setups. + +Example: +............... + colorscheme yellow,normal,blue,red,green,magenta +............... + +[[cmdhelp_options_check-frequency,when to perform semantic check]] +==== `check-frequency` + +Semantic check of the CIB or elements modified or created may be +done on every configuration change (`always`), when verifying +(`on-verify`) or `never`. It is by default set to `always`. +Experts may want to change the setting to `on-verify`. + +The checks require that resource agents are present. If they are +not installed at the configuration time set this preference to +`never`. + +See <<topics_Checks,Configuration semantic checks>> for more details. + +[[cmdhelp_options_check-mode,how to treat semantic errors]] +==== `check-mode` + +Semantic check of the CIB or elements modified or created may be +done in the `strict` mode or in the `relaxed` mode. In the former +certain problems are treated as configuration errors. In the +`relaxed` mode all are treated as warnings. The default is `strict`. + +See <<topics_Checks,Configuration semantic checks>> for more details. + +[[cmdhelp_options_add-quotes,add quotes around parameters containing spaces]] +==== `add-quotes` + +The shell (as in `/bin/sh`) parser strips quotes from the command +line. This may sometimes make it really difficult to type values +which contain white space. One typical example is the configure +filter command. The crm shell will supply extra quotes around +arguments which contain white space. The default is `yes`. + +.Note on quotes use +**************************** +Adding quotes around arguments automatically has been introduced +with version 1.2.2 and it is technically a regression. Being a +regression is the only reason the `add-quotes` option exists. If +you have custom shell scripts which would break, just set the +`add-quotes` option to `no`. + +For instance, with adding quotes enabled, it is possible to do +the following: +............... + # crm configure primitive d1 ocf:heartbeat:Dummy meta description="some description here" + # crm configure filter 'sed "s/hostlist=./&node-c /"' fencing +............... +**************************** + +[[cmdhelp_options_manage-children,how to handle children resource attributes]] +==== `manage-children` + +Some resource management commands, such as `resource stop`, when +the target resource is a group, may not always produce desired +result. Each element, group and the primitive members, can have a +meta attribute and those attributes may end up with conflicting +values. Consider the following construct: +............... + crm(live)# configure show svc fs virtual-ip + primitive fs ocf:heartbeat:Filesystem \ + params device="/dev/drbd0" directory="/srv/nfs" fstype="ext3" \ + op monitor interval="10s" \ + meta target-role="Started" + primitive virtual-ip ocf:heartbeat:IPaddr2 \ + params ip="10.2.13.110" iflabel="1" \ + op monitor interval="10s" \ + op start interval="0" \ + meta target-role="Started" + group svc fs virtual-ip \ + meta target-role="Stopped" +............... + +Even though the element `svc` should be stopped, the group is +actually running because all its members have the `target-role` +set to `Started`: +............... + crm(live)# resource show svc + resource svc is running on: xen-f +............... + +Hence, if the user invokes `resource stop svc` the intention is +not clear. This preference gives the user an opportunity to +better control what happens if attributes of group members have +values which are in conflict with the same attribute of the group +itself. + +Possible values are `ask` (the default), `always`, and `never`. +If set to `always`, the crm shell removes all children attributes +which have values different from the parent. If set to `never`, +all children attributes are left intact. Finally, if set to +`ask`, the user will be asked for each member what is to be done. + +[[cmdhelp_options_show,show current user preference]] +==== `show` + +Display all current settings. + +[[cmdhelp_options_save,save the user preferences to the rc file]] +==== `save` + +Save current settings to the rc file (`$HOME/.config/crm/rc`). On +further `crm` runs, the rc file is automatically read and parsed. + +[[cmdhelp_options_reset,reset user preferences to factory defaults]] +==== `reset` + +This command resets all user options to the defaults. If used as +a single-shot command, the rc file (`$HOME/.config/crm/rc`) is +reset to the defaults too. + +[[cmdhelp_configure,CIB configuration]] +=== `configure` + +This level enables all CIB object definition commands. + +The configuration may be logically divided into four parts: +nodes, resources, constraints, and (cluster) properties and +attributes. Each of these commands support one or more basic CIB +objects. + +Nodes and attributes describing nodes are managed using the +`node` command. + +Commands for resources are: + +- `primitive` +- `monitor` +- `group` +- `clone` +- `ms`/`master` (master-slave) + +In order to streamline large configurations, it is possible to +define a template which can later be referenced in primitives: + +- `rsc_template` + +In that case the primitive inherits all attributes defined in the +template. + +There are three types of constraints: + +- `location` +- `colocation` +- `order` + +It is possible to define fencing order (stonith resource +priorities): + +- `fencing_topology` + +Finally, there are the cluster properties, resource meta +attributes defaults, and operations defaults. All are just a set +of attributes. These attributes are managed by the following +commands: + +- `property` +- `rsc_defaults` +- `op_defaults` + +In addition to the cluster configuration, the Access Control +Lists (ACL) can be setup to allow access to parts of the CIB for +users other than `root` and `hacluster`. The following commands +manage ACL: + +- `user` +- `role` + +The changes are applied to the current CIB only on ending the +configuration session or using the `commit` command. + +Comments start with `#` in the first line. The comments are tied +to the element which follows. If the element moves, its comments +will follow. + +[[cmdhelp_configure_node,define a cluster node]] +==== `node` + +The node command describes a cluster node. Nodes in the CIB are +commonly created automatically by the CRM. Hence, you should not +need to deal with nodes unless you also want to define node +attributes. Note that it is also possible to manage node +attributes at the `node` level. + +Usage: +............... + node <uname>[:<type>] + [attributes <param>=<value> [<param>=<value>...]] + [utilization <param>=<value> [<param>=<value>...]] + + type :: normal | member | ping +............... +Example: +............... + node node1 + node big_node attributes memory=64 +............... + +[[cmdhelp_configure_primitive,define a resource]] +==== `primitive` + +The primitive command describes a resource. It may be referenced +only once in group, clone, or master-slave objects. If it's not +referenced, then it is placed as a single resource in the CIB. + +Operations may be specified in three ways. "Anonymous" as a +simple list of "op" specifications. Use that if you don't want to +reference the set of operations elsewhere. That's by far the most +common way to define operations. If reusing operation sets is +desired, use the "operations" keyword along with the id to give +the operations set a name and the id-ref to reference another set +of operations. + +Operation's attributes which are not recognized are saved as +instance attributes of that operation. A typical example is +`OCF_CHECK_LEVEL`. + +For multistate resources, roles are specified as `role=<role>`. + +A template may be defined for resources which are of the same +type and which share most of the configuration. See +<<cmdhelp_configure_rsc_template,`rsc_template`>> for more information. + +Usage: +............... + primitive <rsc> {[<class>:[<provider>:]]<type>|@<template>} + [params attr_list] + [meta attr_list] + [utilization attr_list] + [operations id_spec] + [op op_type [<attribute>=<value>...] ...] + + attr_list :: [$id=<id>] <attr>=<val> [<attr>=<val>...] | $id-ref=<id> + id_spec :: $id=<id> | $id-ref=<id> + op_type :: start | stop | monitor +............... +Example: +............... + primitive apcfence stonith:apcsmart \ + params ttydev=/dev/ttyS0 hostlist="node1 node2" \ + op start timeout=60s \ + op monitor interval=30m timeout=60s + + primitive www8 apache \ + params configfile=/etc/apache/www8.conf \ + operations $id-ref=apache_ops + + primitive db0 mysql \ + params config=/etc/mysql/db0.conf \ + op monitor interval=60s \ + op monitor interval=300s OCF_CHECK_LEVEL=10 + + primitive r0 ocf:linbit:drbd \ + params drbd_resource=r0 \ + op monitor role=Master interval=60s \ + op monitor role=Slave interval=300s + + primitive xen0 @vm_scheme1 \ + params xmfile=/etc/xen/vm/xen0 +............... + +[[cmdhelp_configure_monitor,add monitor operation to a primitive]] +==== `monitor` + +Monitor is by far the most common operation. It is possible to +add it without editing the whole resource. Also, long primitive +definitions may be a bit uncluttered. In order to make this +command as concise as possible, less common operation attributes +are not available. If you need them, then use the `op` part of +the `primitive` command. + +Usage: +............... + monitor <rsc>[:<role>] <interval>[:<timeout>] +............... +Example: +............... + monitor apcfence 60m:60s +............... + +Note that after executing the command, the monitor operation may +be shown as part of the primitive definition. + +[[cmdhelp_configure_group,define a group]] +==== `group` + +The `group` command creates a group of resources. This can be useful +when resources depend on other resources and require that those +resources start in order on the same node. A common use of resource +groups is to ensure that a server and a virtual IP are located +together, and that the virtual IP is started before the server. + +Grouped resources are started in the order they appear in the group, +and stopped in the reverse order. If a resource in the group cannot +run anywhere, resources following it in the group will not start. + +`group` can be passed the "container" meta attribute, to indicate that +it is to be used to group VM resources monitored using Nagios. The +resource referred to by the container attribute must be of type +`ocf:heartbeat:Xen`, `ocf:heartbeat:VirtualDomain` or `ocf:heartbeat:lxc`. + +Usage: +............... + group <name> <rsc> [<rsc>...] + [meta attr_list] + [params attr_list] + + attr_list :: [$id=<id>] <attr>=<val> [<attr>=<val>...] | $id-ref=<id> +............... +Example: +............... + group internal_www disk0 fs0 internal_ip apache \ + meta target_role=stopped + + group vm-and-services vm vm-sshd meta container="vm" +............... + +[[cmdhelp_configure_clone,define a clone]] +==== `clone` + +The `clone` command creates a resource clone. It may contain a +single primitive resource or one group of resources. + +Usage: +............... + clone <name> <rsc> + [meta attr_list] + [params attr_list] + + attr_list :: [$id=<id>] <attr>=<val> [<attr>=<val>...] | $id-ref=<id> +............... +Example: +............... + clone cl_fence apc_1 \ + meta clone-node-max=1 globally-unique=false +............... + +[[cmdhelp_configure_ms,define a master-slave resource]] +==== `ms` (`master`) + +The `ms` command creates a master/slave resource type. It may contain a +single primitive resource or one group of resources. + +Usage: +............... + ms <name> <rsc> + [meta attr_list] + [params attr_list] + + attr_list :: [$id=<id>] <attr>=<val> [<attr>=<val>...] | $id-ref=<id> +............... +Example: +............... + ms disk1 drbd1 \ + meta notify=true globally-unique=false +............... + +.Note on `id-ref` usage +**************************** +Instance or meta attributes (`params` and `meta`) may contain +a reference to another set of attributes. In that case, no other +attributes are allowed. Since attribute sets' ids, though they do +exist, are not shown in the `crm`, it is also possible to +reference an object instead of an attribute set. `crm` will +automatically replace such a reference with the right id: + +............... + crm(live)configure# primitive a2 www-2 meta $id-ref=a1 + crm(live)configure# show a2 + primitive a2 ocf:heartbeat:apache \ + meta $id-ref="a1-meta_attributes" + [...] +............... +It is advisable to give meaningful names to attribute sets which +are going to be referenced. +**************************** + +[[cmdhelp_configure_rsc_template,define a resource template]] +==== `rsc_template` + +The `rsc_template` command creates a resource template. It may be +referenced in primitives. It is used to reduce large +configurations with many similar resources. + +Usage: +............... + rsc_template <name> [<class>:[<provider>:]]<type> + [params attr_list] + [meta attr_list] + [utilization attr_list] + [operations id_spec] + [op op_type [<attribute>=<value>...] ...] + + attr_list :: [$id=<id>] <attr>=<val> [<attr>=<val>...] | $id-ref=<id> + id_spec :: $id=<id> | $id-ref=<id> + op_type :: start | stop | monitor +............... +Example: +............... + rsc_template public_vm ocf:heartbeat:Xen \ + op start timeout=300s \ + op stop timeout=300s \ + op monitor interval=30s timeout=60s \ + op migrate_from timeout=600s \ + op migrate_to timeout=600s + primitive xen0 @public_vm \ + params xmfile=/etc/xen/xen0 + primitive xen1 @public_vm \ + params xmfile=/etc/xen/xen1 +............... + +[[cmdhelp_configure_location,a location preference]] +==== `location` + +`location` defines the preference of nodes for the given +resource. The location constraints consist of one or more rules +which specify a score to be awarded if the rule matches. + +Usage: +............... + location <id> <rsc> {node_pref|rules} + + node_pref :: <score>: <node> + + rules :: + rule [id_spec] [$role=<role>] <score>: <expression> + [rule [id_spec] [$role=<role>] <score>: <expression> ...] + + id_spec :: $id=<id> | $id-ref=<id> + score :: <number> | <attribute> | [-]inf + expression :: <simple_exp> [bool_op <simple_exp> ...] + bool_op :: or | and + simple_exp :: <attribute> [type:]<binary_op> <value> + | <unary_op> <attribute> + | date <date_expr> + type :: string | version | number + binary_op :: lt | gt | lte | gte | eq | ne + unary_op :: defined | not_defined + + date_expr :: lt <end> + | gt <start> + | in_range start=<start> end=<end> + | in_range start=<start> <duration> + | date_spec <date_spec> + duration|date_spec :: + hours=<value> + | monthdays=<value> + | weekdays=<value> + | yearsdays=<value> + | months=<value> + | weeks=<value> + | years=<value> + | weekyears=<value> + | moon=<value> +............... +Examples: +............... + location conn_1 internal_www 100: node1 + + location conn_1 internal_www \ + rule 50: #uname eq node1 \ + rule pingd: defined pingd + + location conn_2 dummy_float \ + rule -inf: not_defined pingd or pingd number:lte 0 +............... + +[[cmdhelp_configure_colocation,colocate resources]] +==== `colocation` (`collocation`) + +This constraint expresses the placement relation between two +or more resources. If there are more than two resources, then the +constraint is called a resource set. + +The score is used to indicate the priority of the constraint. A +positive score indicates that the resources should run on the same +node. A negative score that they should not run on the same +node. Values of positive or negative `infinity` indicate a mandatory +constraint. + +In the two resource form, the cluster will place `<with-rsc>` first, +and then decide where to put the `<rsc>` resource. + +Collocation resource sets have an extra attribute (`sequential`) +to allow for sets of resources which don't depend on each other +in terms of state. The shell syntax for such sets is to put +resources in parentheses. + +Sets cannot be nested. + +The optional `node-attribute` references an attribute in nodes' +instance attributes. + +Usage: +............... + colocation <id> <score>: <rsc>[:<role>] <with-rsc>[:<role>] + [node-attribute=<node_attr>] + + colocation <id> <score>: <rsc>[:<role>] <rsc>[:<role>] ... + [node-attribute=<node_attr>] +............... +Example: +............... + colocation never_put_apache_with_dummy -inf: apache dummy + colocation c1 inf: A ( B C ) +............... + +[[cmdhelp_configure_order,order resources]] +==== `order` + +This constraint expresses the order of actions on two resources +or more resources. If there are more than two resources, then the +constraint is called a resource set. + +Ordered resource sets have an extra attribute to allow for sets +of resources whose actions may run in parallel. The shell syntax +for such sets is to put resources in parentheses. + +If the subsequent resource can start or promote after any one of the +resources in a set has done, enclose the set in brackets (`[` and `]`). + +Sets cannot be nested. + +Three strings are reserved to specify a kind of order constraint: +`Mandatory`, `Optional`, and `Serialize`. It is preferred to use +one of these settings instead of score. Previous versions mapped +scores `0` and `inf` to keywords `advisory` and `mandatory`. +That is still valid but deprecated. + +.Note on resource sets' XML attributes +**************************** +The XML attribute `require-all` controls whether all resources in +a set are, well, required. The bracketed sets actually have this +attribute as well as `sequential` set to `false`. If you need a +different combination, for whatever reason, just set one of the +attributes within the set. Something like this: + +............... + crm(live)configure# order o1 Mandatory: [ A B sequential=true ] C +............... +It is up to you to find out whether such a combination makes +sense. +**************************** + +Usage: +............... + order <id> {kind|<score>}: <rsc>[:<action>] <rsc>[:<action>] ... + [symmetrical=<bool>] + + kind :: Mandatory | Optional | Serialize +............... +Example: +............... + order c_apache_1 Mandatory: apache:start ip_1 + order o1 Serialize: A ( B C ) + order order_2 Mandatory: [ A B ] C +............... + +[[cmdhelp_configure_rsc_ticket,resources ticket dependency]] +==== `rsc_ticket` + +This constraint expresses dependency of resources on cluster-wide +attributes, also known as tickets. Tickets are mainly used in +geo-clusters, which consist of multiple sites. A ticket may be +granted to a site, thus allowing resources to run there. + +The `loss-policy` attribute specifies what happens to the +resource (or resources) if the ticket is revoked. The default is +either `stop` or `demote` depending on whether a resource is +multi-state. + +See also the <<cmdhelp_site_ticket,`site`>> set of commands. + +Usage: +............... + rsc_ticket <id> <ticket_id>: <rsc>[:<role>] [<rsc>[:<role>] ...] + [loss-policy=<loss_policy_action>] + + loss_policy_action :: stop | demote | fence | freeze +............... +Example: +............... + rsc_ticket ticket-A_public-ip ticket-A: public-ip + rsc_ticket ticket-A_bigdb ticket-A: bigdb loss-policy=fence + rsc_ticket ticket-B_storage ticket-B: drbd-a:Master drbd-b:Master +............... + + +[[cmdhelp_configure_property,set a cluster property]] +==== `property` + +Set the cluster (`crm_config`) options. + +Usage: +............... + property [$id=<set_id>] <option>=<value> [<option>=<value> ...] +............... +Example: +............... + property stonith-enabled=true +............... + +[[cmdhelp_configure_rsc_defaults,set resource defaults]] +==== `rsc_defaults` + +Set defaults for the resource meta attributes. + +Usage: +............... + rsc_defaults [$id=<set_id>] <option>=<value> [<option>=<value> ...] +............... +Example: +............... + rsc_defaults failure-timeout=3m +............... + +[[cmdhelp_configure_fencing_topology,node fencing order]] +==== `fencing_topology` + +If multiple fencing (stonith) devices are available capable of +fencing a node, their order may be specified by `fencing_topology`. +The order is specified per node. + +Stonith resources can be separated by `,` in which case all of +them need to succeed. If they fail, the next stonith resource (or +set of resources) is used. In other words, use comma to separate +resources which all need to succeed and whitespace for serial +order. It is not allowed to use whitespace around comma. + +If the node is left out, the order is used for all nodes. +That should reduce the configuration size in some stonith setups. + +Usage: +............... + fencing_topology stonith_resources [stonith_resources ...] + fencing_topology fencing_order [fencing_order ...] + + fencing_order :: <node>: stonith_resources [stonith_resources ...] + + stonith_resources :: <rsc>[,<rsc>...] +............... +Example: +............... + fencing_topology poison-pill power + fencing_topology \ + node-a: poison-pill power + node-b: ipmi serial +............... + +[[cmdhelp_configure_role,define role access rights]] +==== `role` + +An ACL role is a set of rules which describe access rights to +CIB. Rules consist of an access right `read`, `write`, or `deny` +and a specification denoting part of the configuration to which +the access right applies. The specification can be an XPath or a +combination of tag and id references. If an attribute is +appended, then the specification applies only to that attribute +of the matching element. + +There is a number of shortcuts for XPath specifications. The +`meta,` `params`, and `utilization` shortcuts reference resource +meta attributes, parameters, and utilization respectively. The +`location` may be used to specify location constraints most of +the time to allow resource `move` and `unmove` commands. The +`property` references cluster properties. The `node` allows +reading node attributes. `nodeattr` and `nodeutil` reference node +attributes and node capacity (utilization). The `status` shortcut +references the whole status section of the CIB. Read access to +status is necessary for various monitoring tools such as +`crm_mon(8)` (aka `crm status`). + +Usage: +............... + role <role-id> rule [rule ...] + + rule :: acl-right cib-spec [attribute:<attribute>] + + acl-right :: read | write | deny + + cib-spec :: xpath-spec | tag-ref-spec + xpath-spec :: xpath:<xpath> | shortcut + tag-ref-spec :: tag:<tag> | ref:<id> | tag:<tag> ref:<id> + + shortcut :: meta:<rsc>[:<attr>] + params:<rsc>[:<attr>] + utilization:<rsc> + location:<rsc> + property[:<attr>] + node[:<node>] + nodeattr[:<attr>] + nodeutil[:<node>] + status +............... +Example: +............... + role app1_admin \ + write meta:app1:target-role \ + write meta:app1:is-managed \ + write location:app1 \ + read ref:app1 +............... + +[[cmdhelp_configure_user,define user access rights]] +==== `user` + +Users which normally cannot view or manage cluster configuration +can be allowed access to parts of the CIB. The access is defined +by a set of `read`, `write`, and `deny` rules as in role +definitions or by referencing roles. The latter is considered +best practice. + +Usage: +............... + user <uid> {roles|rules} + + roles :: role:<role-ref> [role:<role-ref> ...] + rules :: rule [rule ...] +............... +Example: +............... + user joe \ + role:app1_admin \ + role:read_all +............... + +[[cmdhelp_configure_op_defaults,set resource operations defaults]] +==== `op_defaults` + +Set defaults for the operations meta attributes. + +Usage: +............... + op_defaults [$id=<set_id>] <option>=<value> [<option>=<value> ...] +............... +Example: +............... + op_defaults record-pending=true +............... + +[[cmdhelp_configure_schema,set or display current CIB RNG schema]] +==== `schema` + +CIB's content is validated by a RNG schema. Pacemaker supports +several, depending on version. Currently supported schemas are +`pacemaker-1.0`, `pacemaker-1.1`, and `pacemaker-1.2`. + +Use this command to display or switch to another RNG schema. + +Usage: +............... + schema [<schema>] +............... +Example: +............... + schema pacemaker-1.1 +............... + +[[cmdhelp_configure_show,display CIB objects]] +==== `show` + +The `show` command displays objects. It may display all objects +or a set of objects. The user may also choose to see only objects +which were changed. +Optionally, the XML code may be displayed instead of the CLI +representation. + +Usage: +............... + show [xml] [<id> ...] + show [xml] changed +............... + +[[cmdhelp_configure_edit,edit CIB objects]] +==== `edit` + +This command invokes the editor with the object description. As +with the `show` command, the user may choose to edit all objects +or a set of objects. + +If the user insists, he or she may edit the XML edition of the +object. If you do that, don't modify any id attributes. + +Usage: +............... + edit [xml] [<id> ...] + edit [xml] changed +............... + +.Note on renaming element ids +**************************** +The edit command sometimes cannot properly handle modifying +element ids. In particular for elements which belong to group or +ms resources. Group and ms resources themselves also cannot be +renamed. Please use the `rename` command instead. +**************************** + +[[cmdhelp_configure_filter,filter CIB objects]] +==== `filter` + +This command filters the given CIB elements through an external +program. The program should accept input on `stdin` and send +output to `stdout` (the standard UNIX filter conventions). As +with the `show` command, the user may choose to filter all or +just a subset of elements. + +It is possible to filter the XML representation of objects, but +probably not as useful as the configuration language. The +presentation is somewhat different from what would be displayed +by the `show` command---each element is shown on a single line, +i.e. there are no backslashes and no other embelishments. + +Don't forget to put quotes around the filter if it contains +spaces. + +Usage: +............... + filter <prog> [xml] [<id> ...] + filter <prog> [xml] changed +............... +Examples: +............... + filter "sed '/^primitive/s/target-role=[^ ]*//'" + # crm configure filter "sed '/^primitive/s/target-role=[^ ]*//'" +............... + +[[cmdhelp_configure_delete,delete CIB objects]] +==== `delete` + +Delete one or more objects. If an object to be deleted belongs to +a container object, such as a group, and it is the only resource +in that container, then the container is deleted as well. Any +related constraints are removed as well. + +Usage: +............... + delete <id> [<id>...] +............... + +[[cmdhelp_configure_default-timeouts,set timeouts for operations to minimums from the meta-data]] +==== `default-timeouts` + +This command takes the timeouts from the actions section of the +resource agent meta-data and sets them for the operations of the +primitive. + +Usage: +............... + default-timeouts <id> [<id>...] +............... + +.Note on `default-timeouts` +**************************** +You may be happy using this, but your applications may not. And +it will tell you so at the worst possible moment. You have been +warned. +**************************** + +[[cmdhelp_configure_rename,rename a CIB object]] +==== `rename` + +Rename an object. It is recommended to use this command to rename +a resource, because it will take care of updating all related +constraints and a parent resource. Changing ids with the edit +command won't have the same effect. + +If you want to rename a resource, it must be in the stopped state. + +Usage: +............... + rename <old_id> <new_id> +............... + +[[cmdhelp_configure_modgroup,modify group]] +==== `modgroup` + +Add or remove primitives in a group. The `add` subcommand appends +the new group member by default. Should it go elsewhere, there +are `after` and `before` clauses. + +Usage: +............... + modgroup <id> add <id> [after <id>|before <id>] + modgroup <id> remove <id> +............... +Examples: +............... + modgroup share1 add storage2 before share1-fs +............... + +[[cmdhelp_configure_refresh,refresh from CIB]] +==== `refresh` + +Refresh the internal structures from the CIB. All changes made +during this session are lost. + +Usage: +............... + refresh +............... + +[[cmdhelp_configure_erase,erase the CIB]] +==== `erase` + +The `erase` clears all configuration. Apart from nodes. To remove +nodes, you have to specify an additional keyword `nodes`. + +Note that removing nodes from the live cluster may have some +strange/interesting/unwelcome effects. + +Usage: +............... + erase [nodes] +............... + +[[cmdhelp_configure_ptest,show cluster actions if changes were committed]] +==== `ptest` (`simulate`) + +Show PE (Policy Engine) motions using `ptest(8)` or +`crm_simulate(8)`. + +A CIB is constructed using the current user edited configuration +and the status from the running CIB. The resulting CIB is run +through `ptest` (or `crm_simulate`) to show changes which would +happen if the configuration is committed. + +The status section may be loaded from another source and modified +using the <<cmdhelp_cibstatus,`cibstatus`>> level commands. In that case, the +`ptest` command will issue a message informing the user that the +Policy Engine graph is not calculated based on the current status +section and therefore won't show what would happen to the +running but some imaginary cluster. + +If you have graphviz installed and X11 session, `dotty(1)` is run +to display the changes graphically. + +Add a string of `v` characters to increase verbosity. `ptest` +can also show allocation scores. `utilization` turns on +information about the remaining capacity of nodes. With the +`actions` option, `ptest` will print all resource actions. + +The `ptest` program has been replaced by `crm_simulate` in newer +Pacemaker versions. In some installations both could be +installed. Use `simulate` to enfore using `crm_simulate`. + +Usage: +............... + ptest [nograph] [v...] [scores] [actions] [utilization] +............... +Examples: +............... + ptest scores + ptest vvvvv + simulate actions +............... + +[[cmdhelp_configure_rsctest,test resources as currently configured]] +==== `rsctest` + +Test resources with current resource configuration. If no nodes +are specified, tests are run on all known nodes. + +The order of resources is significant: it is assumed that later +resources depend on earlier ones. + +If a resource is multi-state, it is assumed that the role on +which later resources depend is master. + +Tests are run sequentially to prevent running the same resource +on two or more nodes. Tests are carried out only if none of the +specified nodes currently run any of the specified resources. +However, it won't verify whether resources run on the other +nodes. + +Superuser privileges are obviously required: either run this as +root or setup the `sudoers` file appropriately. + +Note that resource testing may take some time. + +Usage: +............... + rsctest <rsc_id> [<rsc_id> ...] [<node_id> ...] +............... +Examples: +............... + rsctest my_ip websvc + rsctest websvc nodeB +............... + +[[cmdhelp_configure_cib,CIB shadow management]] +=== `cib` (shadow CIBs) + +This level is for management of shadow CIBs. It is available at +the `configure` level to enable saving intermediate changes to a +shadow CIB instead of to the live cluster. This short excerpt +shows how: +............... + crm(live)configure# cib new test-2 + INFO: test-2 shadow CIB created + crm(test-2)configure# commit +............... +Note how the current CIB in the prompt changed from `live` to +`test-2` after issuing the `cib new` command. See also the +<<cmdhelp_cib,CIB shadow management>> for more information. + +[[cmdhelp_configure_cibstatus,CIB status management and editing]] +==== `cibstatus` + +Enter edit and manage the CIB status section level. See the +<<cmdhelp_cibstatus,CIB status management section>>. + +[[cmdhelp_configure_template,edit and import a configuration from a template]] +==== `template` + +The specified template is loaded into the editor. It's up to the +user to make a good CRM configuration out of it. See also the +<<cmdhelp_template,template section>>. + +Usage: +............... + template [xml] url +............... +Example: +............... + template two-apaches.txt +............... + +[[cmdhelp_configure_commit,commit the changes to the CIB]] +==== `commit` + +Commit the current configuration to the CIB in use. As noted +elsewhere, commands in a configure session don't have immediate +effect on the CIB. All changes are applied at one point in time, +either using `commit` or when the user leaves the configure +level. In case the CIB in use changed in the meantime, presumably +by somebody else, the crm shell will refuse to apply the changes. +If you know that it's fine to still apply them add `force`. + +Usage: +............... + commit [force] +............... + +[[cmdhelp_configure_verify,verify the CIB with crm_verify]] +==== `verify` + +Verify the contents of the CIB which would be committed. + +Usage: +............... + verify +............... + +[[cmdhelp_configure_upgrade,upgrade the CIB to version 1.0]] +==== `upgrade` + +If you get the `CIB not supported` error, which typically means +that the current CIB version is coming from the older release, +you may try to upgrade it to the latest revision. The command +to perform the upgrade is: +............... + # cibadmin --upgrade --force +............... + +If we don't recognize the current CIB as the old one, but you're +sure that it is, you may force the command. + +Usage: +............... + upgrade [force] +............... + +[[cmdhelp_configure_save,save the CIB to a file]] +==== `save` + +Save the current configuration to a file. Optionally, as XML. Use +`-` instead of file name to write the output to `stdout`. + +Usage: +............... + save [xml] <file> +............... +Example: +............... + save myfirstcib.txt +............... + +[[cmdhelp_configure_load,import the CIB from a file]] +==== `load` + +Load a part of configuration (or all of it) from a local file or +a network URL. The `replace` method replaces the current +configuration with the one from the source. The `update` tries to +import the contents into the current configuration. +The file may be a CLI file or an XML file. + +Usage: +............... + load [xml] <method> URL + + method :: replace | update +............... +Example: +............... + load xml update myfirstcib.xml + load xml replace http://storage.big.com/cibs/bigcib.xml +............... + +[[cmdhelp_configure_graph,generate a directed graph]] +==== `graph` + +Create a graphviz graphical layout from the current cluster +configuration. + +Currently, only `dot` (directed graph) is supported. It is +essentially a visualization of resource ordering. + +The graph may be saved to a file which can be used as source for +various graphviz tools (by default it is displayed in the user's +X11 session). Optionally, by specifying the format, one can also +produce an image instead. + +For more or different graphviz attributes, it is possible to save +the default set of attributes to an ini file. If this file exists +it will always override the builtin settings. The `exportsettings` +subcommand also prints the location of the ini file. + +Usage: +............... + graph [<gtype> [<file> [<img_format>]]] + graph exportsettings + + gtype :: dot + img_format :: `dot` output format (see the `-T` option) +............... +Example: +............... + graph dot + graph dot clu1.conf.dot + graph dot clu1.conf.svg svg +............... + +[[cmdhelp_configure_xml,raw xml]] +==== `xml` + +Even though we promissed no xml, it may happen, but hopefully +very very seldom, that an element from the CIB cannot be rendered +in the configuration language. In that case, the element will be +shown as raw xml, prefixed by this command. That element can then +be edited like any other. If the shell finds out that after the +change it can digest it, then it is going to be converted into +the normal configuration language. Otherwise, there is no need to +use `xml` for configuration. + +Usage: +............... + xml <xml> +............... + +[[cmdhelp_template,edit and import a configuration from a template]] +=== `template` + +User may be assisted in the cluster configuration by templates +prepared in advance. Templates consist of a typical ready +configuration which may be edited to suit particular user needs. + +This command enters a template level where additional commands +for configuration/template management are available. + +[[cmdhelp_template_new,create a new configuration from templates]] +==== `new` + +Create a new configuration from one or more templates. Note that +configurations and templates are kept in different places, so it +is possible to have a configuration name equal a template name. + +If you already know which parameters are required, you can set +them directly on the command line. + +The parameter name `id` is set by default to the name of the +configuration. + +Usage: +............... + new <config> <template> [<template> ...] [params name=value ...]" +............... +Examples: +............... + new vip virtual-ip + new bigfs ocfs2 params device=/dev/sdx8 directory=/bigfs +............... + +[[cmdhelp_template_load,load a configuration]] +==== `load` + +Load an existing configuration. Further `edit`, `show`, and +`apply` commands will refer to this configuration. + +Usage: +............... + load <config> +............... + +[[cmdhelp_template_edit,edit a configuration]] +==== `edit` + +Edit current or given configuration using your favourite editor. + +Usage: +............... + edit [<config>] +............... + +[[cmdhelp_template_delete,delete a configuration]] +==== `delete` + +Remove a configuration. The loaded (active) configuration may be +removed by force. + +Usage: +............... + delete <config> [force] +............... + +[[cmdhelp_template_list,list configurations/templates]] +==== `list` + +List existing configurations or templates. + +Usage: +............... + list [templates] +............... + +[[cmdhelp_template_apply,process and apply the current configuration to the current CIB]] +==== `apply` + +Copy the current or given configuration to the current CIB. By +default, the CIB is replaced, unless the method is set to +"update". + +Usage: +............... + apply [<method>] [<config>] + + method :: replace | update +............... + +[[cmdhelp_template_show,show the processed configuration]] +==== `show` + +Process the current or given configuration and display the result. + +Usage: +............... + show [<config>] +............... + +[[cmdhelp_cibstatus,CIB status management and editing]] +=== `cibstatus` + +The `status` section of the CIB keeps the current status of nodes +and resources. It is modified _only_ on events, i.e. when some +resource operation is run or node status changes. For obvious +reasons, the CRM has no user interface with which it is possible +to affect the status section. From the user's point of view, the +status section is essentially a read-only part of the CIB. The +current status is never even written to disk, though it is +available in the PE (Policy Engine) input files which represent +the history of cluster motions. The current status may be read +using the `cibadmin -Q` command. + +It may sometimes be of interest to see how status changes would +affect the Policy Engine. The set of `cibstatus` level commands +allow the user to load status sections from various sources and +then insert or modify resource operations or change nodes' state. + +The effect of those changes may then be observed by running the +<<cmdhelp_configure_ptest,`ptest`>> command at the `configure` level +or `simulate` and `run` commands at this level. The `ptest` +runs with the user edited CIB whereas the latter two commands +run with the CIB which was loaded along with the status section. + +The `simulate` and `run` commands as well as all status +modification commands are implemented using `crm_simulate(8)`. + +[[cmdhelp_cibstatus_load,load the CIB status section]] +==== `load` + +Load a status section from a file, a shadow CIB, or the running +cluster. By default, the current (`live`) status section is +modified. Note that if the `live` status section is modified it +is not going to be updated if the cluster status changes, because +that would overwrite the user changes. To make `crm` drop changes +and resume use of the running cluster status, run `load live`. + +All CIB shadow configurations contain the status section which is +a snapshot of the status section taken at the time the shadow was +created. Obviously, this status section doesn't have much to do +with the running cluster status, unless the shadow CIB has just +been created. Therefore, the `ptest` command by default uses the +running cluster status section. + +Usage: +............... + load {<file>|shadow:<cib>|live} +............... +Example: +............... + load bug-12299.xml + load shadow:test1 +............... + +[[cmdhelp_cibstatus_save,save the CIB status section]] +==== `save` + +The current internal status section with whatever modifications +were performed can be saved to a file or shadow CIB. + +If the file exists and contains a complete CIB, only the status +section is going to be replaced and the rest of the CIB will +remain intact. Otherwise, the current user edited configuration +is saved along with the status section. + +Note that all modifications are saved in the source file as soon +as they are run. + +Usage: +............... + save [<file>|shadow:<cib>] +............... +Example: +............... + save bug-12299.xml +............... + +[[cmdhelp_cibstatus_origin,display origin of the CIB status section]] +==== `origin` + +Show the origin of the status section currently in use. This +essentially shows the latest `load` argument. + +Usage: +............... + origin +............... + +[[cmdhelp_cibstatus_show,show CIB status section]] +==== `show` + +Show the current status section in the XML format. Brace yourself +for some unreadable output. Add `changed` option to get a human +readable output of all changes. + +Usage: +............... + show [changed] +............... + +[[cmdhelp_cibstatus_node,change node status]] +==== `node` + +Change the node status. It is possible to throw a node out of +the cluster, make it a member, or set its state to unclean. + +`online`:: Set the `node_state` `crmd` attribute to `online` +and the `expected` and `join` attributes to `member`. The effect +is that the node becomes a cluster member. + +`offline`:: Set the `node_state` `crmd` attribute to `offline` +and the `expected` attribute to empty. This makes the node +cleanly removed from the cluster. + +`unclean`:: Set the `node_state` `crmd` attribute to `offline` +and the `expected` attribute to `member`. In this case the node +has unexpectedly disappeared. + +Usage: +............... + node <node> {online|offline|unclean} +............... +Example: +............... + node xen-b unclean +............... + +[[cmdhelp_cibstatus_op,edit outcome of a resource operation]] +==== `op` + +Edit the outcome of a resource operation. This way you can +tell CRM that it ran an operation and that the resource agent +returned certain exit code. It is also possible to change the +operation's status. In case the operation status is set to +something other than `done`, the exit code is effectively +ignored. + +Usage: +............... + op <operation> <resource> <exit_code> [<op_status>] [<node>] + + operation :: probe | monitor[:<n>] | start | stop | + promote | demote | notify | migrate_to | migrate_from + exit_code :: <rc> | success | generic | args | + unimplemented | perm | installed | configured | not_running | + master | failed_master + op_status :: pending | done | cancelled | timeout | notsupported | error + + n :: the monitor interval in seconds; if omitted, the first + recurring operation is referenced + rc :: numeric exit code in range 0..9 +............... +Example: +............... + op start d1 xen-b generic + op start d1 xen-b 1 + op monitor d1 xen-b not_running + op stop d1 xen-b 0 timeout +............... + +[[cmdhelp_cibstatus_quorum,set the quorum]] +==== `quorum` + +Set the quorum value. + +Usage: +............... + quorum <bool> +............... +Example: +............... + quorum false +............... + +[[cmdhelp_cibstatus_ticket,manage tickets]] +==== `ticket` + +Modify the ticket status. Tickets can be granted and revoked. +Granted tickets could be activated or put in standby. + +Usage: +............... + ticket <ticket> {grant|revoke|activate|standby} +............... +Example: +............... + ticket ticketA grant +............... + +[[cmdhelp_cibstatus_run,run policy engine]] +==== `run` + +Run the policy engine with the edited status section. + +Add a string of `v` characters to increase verbosity. Specify +`scores` to see allocation scores also. `utilization` turns on +information about the remaining capacity of nodes. + +If you have graphviz installed and X11 session, `dotty(1)` is run +to display the changes graphically. + +Usage: +............... + run [nograph] [v...] [scores] [utilization] +............... +Example: +............... + run +............... + +[[cmdhelp_cibstatus_simulate,simulate cluster transition]] +==== `simulate` + +Run the policy engine with the edited status section and simulate +the transition. + +Add a string of `v` characters to increase verbosity. Specify +`scores` to see allocation scores also. `utilization` turns on +information about the remaining capacity of nodes. + +If you have graphviz installed and X11 session, `dotty(1)` is run +to display the changes graphically. + +Usage: +............... + simulate [nograph] [v...] [scores] [utilization] +............... +Example: +............... + simulate +............... + +[[cmdhelp_history,cluster history]] +=== `history` + +Examining Pacemaker's history is a particularly involved task. +The number of subsystems to be considered, the complexity of the +configuration, and the set of various information sources, most +of which are not exactly human readable, keep analyzing resource +or node problems accessible to only the most knowledgeable. Or, +depending on the point of view, to the most persistent. The +following set of commands has been devised in hope to make +cluster history more accessible. + +Of course, looking at _all_ history could be time consuming +regardless of how good tools at hand are. Therefore, one should +first say which period he or she wants to analyze. If not +otherwise specified, the last hour is considered. Logs and other +relevant information is collected using `hb_report`. Since this +process takes some time and we always need fresh logs, +information is refreshed in a much faster way using `pssh(1)`. If +`python-pssh` is not found on the system, examining live cluster +is still possible though not as comfortable. + +Apart from examining live cluster, events may be retrieved from a +report generated by `hb_report` (see also the `-H` option). In +that case we assume that the period stretching the whole report +needs to be investigated. Of course, it is still possible to +further reduce the time range. + +If you think you may have found a bug or just need clarification +from developers or your support, the `session pack` command can +help create a report. This is an example: +............... + crm(live)history# limit "Jul 18 12:00" "Jul 18 12:30" + crm(live)history# session save strange_restart + crm(live)history# session pack + Report saved in .../strange_restart.tar.bz2 + crm(live)history# +............... +In order to reduce report size and allow developers to +concentrate on the issue, you should beforehand limit the time +frame. Giving a meaningful session name helps too. + +==== `info` + +The `info` command shows most important information about the +cluster. + +Usage: +............... + info +............... +Example: +............... + info +............... + +[[cmdhelp_history_latest,show latest news from the cluster]] +==== `latest` + +The `latest` command shows a bit of recent history, more +precisely whatever happened since the last cluster change (the +latest transition). If the transition is running, the shell will +first wait until it finishes. + +Usage: +............... + latest +............... +Example: +............... + latest +............... + +[[cmdhelp_history_limit,limit timeframe to be examined]] +==== `limit` (`timeframe`) + +All history commands look at events within certain period. It +defaults to the last hour for the live cluster source. There is +no limit for the `hb_report` source. Use this command to set the +timeframe. + +The time period is parsed by the dateutil python module. It +covers wide range of date formats. For instance: + +- 3:00 (today at 3am) +- 15:00 (today at 3pm) +- 2010/9/1 2pm (September 1st 2010 at 2pm) + +We won't bother to give definition of the time specification in +usage below. Either use common sense or read the +http://labix.org/python-dateutil[dateutil] documentation. + +If dateutil is not available, then the time is parsed using +strptime and only the kind as printed by `date(1)` is allowed: + +- Tue Sep 15 20:46:27 CEST 2010 + +Usage: +............... + limit [<from_time> [<to_time>]] +............... +Examples: +............... + limit 10:15 + limit 15h22m 16h + limit "Sun 5 20:46" "Sun 5 22:00" +............... + +[[cmdhelp_history_source,set source to be examined]] +==== `source` + +Events to be examined can come from the current cluster or from a +`hb_report` report. This command sets the source. `source live` +sets source to the running cluster and system logs. If no source +is specified, the current source information is printed. + +In case a report source is specified as a file reference, the file +is going to be unpacked in place where it resides. This directory +is not removed on exit. + +Usage: +............... + source [<dir>|<file>|live] +............... +Examples: +............... + source live + source /tmp/customer_case_22.tar.bz2 + source /tmp/customer_case_22 + source +............... + +[[cmdhelp_history_refresh,refresh live report]] +==== `refresh` + +This command makes sense only for the `live` source and makes +`crm` collect the latest logs and other relevant information from +the logs. If you want to make a completely new report, specify +`force`. + +Usage: +............... + refresh [force] +............... + +[[cmdhelp_history_detail,set the level of detail shown]] +==== `detail` + +How much detail to show from the logs. + +Usage: +............... + detail <detail_level> + + detail_level :: small integer (defaults to 0) +............... +Example: +............... + detail 1 +............... + +[[cmdhelp_history_setnodes,set the list of cluster nodes]] +==== `setnodes` + +In case the host this program runs on is not part of the cluster, +it is necessary to set the list of nodes. + +Usage: +............... + setnodes node <node> [<node> ...] +............... +Example: +............... + setnodes node_a node_b +............... + +[[cmdhelp_history_resource,resource events]] +==== `resource` + +Show actions and any failures that happened on all specified +resources on all nodes. Normally, one gives resource names as +arguments, but it is also possible to use extended regular +expressions. Note that neither groups nor clones or master/slave +names are ever logged. The resource command is going to expand +all of these appropriately, so that clone instances or resources +which are part of a group are shown. + +Usage: +............... + resource <rsc> [<rsc> ...] +............... +Example: +............... + resource bigdb public_ip + resource my_.*_db2 + resource ping_clone +............... + +[[cmdhelp_history_node,node events]] +==== `node` + +Show important events that happened on a node. Important events +are node lost and join, standby and online, and fence. Use either +node names or extended regular expressions. + +Usage: +............... + node <node> [<node> ...] +............... +Example: +............... + node node1 +............... + +[[cmdhelp_history_log,log content]] +==== `log` + +Show messages logged on one or more nodes. Leaving out a node +name produces combined logs of all nodes. Messages are sorted by +time and, if the terminal emulations supports it, displayed in +different colours depending on the node to allow for easier +reading. + +The sorting key is the timestamp as written by syslog which +normally has the maximum resolution of one second. Obviously, +messages generated by events which share the same timestamp may +not be sorted in the same way as they happened. Such close events +may actually happen fairly often. + +Usage: +............... + log [<node>] +............... +Example: +............... + log node-a +............... + +[[cmdhelp_history_exclude,exclude log messages]] +==== `exclude` + +If a log is infested with irrelevant messages, those messages may +be excluded by specifying a regular expression. The regular +expressions used are Python extended. This command is additive. +To drop all regular expressions, use `exclude clear`. Run +`exclude` only to see the current list of regular expressions. +Excludes are saved along with the history sessions. + +Usage: +............... + exclude [<regex>|clear] +............... +Example: +............... + exclude kernel.*ocfs2 +............... + +[[cmdhelp_history_peinputs,list or get PE input files]] +==== `peinputs` + +Every event in the cluster results in generating one or more +Policy Engine (PE) files. These files describe future motions of +resources. The files are listed as full paths in the current +report directory. Add `v` to also see the creation time stamps. + +Usage: +............... + peinputs [{<range>|<number>} ...] [v] + + range :: <n1>:<n2> +............... +Example: +............... + peinputs + peinputs 440:444 446 + peinputs v +............... + +[[cmdhelp_history_transition,show transition]] +==== `transition` + +This command will print actions planned by the PE and run +graphviz (`dotty`) to display a graphical representation of the +transition. Of course, for the latter an X11 session is required. +This command invokes `ptest(8)` in background. + +The `showdot` subcommand runs graphviz (`dotty`) to display a +graphical representation of the `.dot` file which has been +included in the report. Essentially, it shows the calculation +produced by `pengine` which is installed on the node where the +report was produced. In optimal case this output should not +differ from the one produced by the locally installed `pengine`. + +The `log` subcommand shows the full log for the duration of the +transition. + +A transition can also be saved to a CIB shadow for further +analysis or use with `cib` or `configure` commands (use the +`save` subcommand). The shadow file name defaults to the name of +the PE input file. + +If the PE input file number is not provided, it defaults to the +last one, i.e. the last transition. The last transition can also +be referenced with number 0. If the number is negative, then the +corresponding transition relative to the last one is chosen. + +If there are warning and error PE input files or different nodes +were the DC in the observed timeframe, it may happen that PE +input file numbers collide. In that case provide some unique part +of the path to the file. + +After the `ptest` output, logs about events that happened during +the transition are printed. + +Usage: +............... + transition [<number>|<index>|<file>] [nograph] [v...] [scores] [actions] [utilization] + transition showdot [<number>|<index>|<file>] + transition log [<number>|<index>|<file>] + transition save [<number>|<index>|<file> [name]] +............... +Examples: +............... + transition + transition 444 + transition -1 + transition pe-error-3.bz2 + transition node-a/pengine/pe-input-2.bz2 + transition showdot 444 + transition log + transition save 0 enigma-22 +............... + +[[cmdhelp_history_show,show status or configuration of the PE input file]] +==== `show` + +Every transition is saved as a PE file. Use this command to +render that PE file either as configuration or status. The +configuration output is the same as `crm configure show`. + +Usage: +............... + show <pe> [status] + + pe :: <number>|<index>|<file>|live +............... +Examples: +............... + show 2066 + show pe-input-2080.bz2 status +............... + +[[cmdhelp_history_graph,generate a directed graph from the PE file]] +==== `graph` + +Create a graphviz graphical layout from the PE file (the +transition). Every transition contains the cluster configuration +which was active at the time. See also <<cmdhelp_configure_graph,generate a directed graph +from configuration>>. + +Usage: +............... + graph <pe> [<gtype> [<file> [<img_format>]]] + + gtype :: dot + img_format :: `dot` output format (see the `-T` option) +............... +Example: +............... + graph -1 + graph 322 dot clu1.conf.dot + graph 322 dot clu1.conf.svg svg +............... + +[[cmdhelp_history_diff,cluster states/transitions difference]] +==== `diff` + +A transition represents a change in cluster configuration or +state. Use `diff` to see what has changed between two +transitions. + +If you want to specify the current cluster configuration and +status, use the string `live`. + +Normally, the first transition specified should be the one which +is older, but we are not going to enforce that. + +Note that a single configuration update may result in more than +one transition. + +Usage: +............... + diff <pe> <pe> [status] [html] + + pe :: <number>|<index>|<file>|live +............... +Examples: +............... + diff 2066 2067 + diff pe-input-2080.bz2 live status +............... + +[[cmdhelp_history_session,manage history sessions]] +==== `session` + +Sometimes you may want to get back to examining a particular +history period or bug report. In order to make that easier, the +current settings can be saved and later retrieved. + +If the current history being examined is coming from a live +cluster the logs, PE inputs, and other files are saved too, +because they may disappear from nodes. For the existing reports +coming from `hb_report`, only the directory location is saved +(not to waste space). + +A history session may also be packed into a tarball which can +then be sent to support. + +Leave out subcommand to see the current session. + +Usage: +............... + session [{save|load|delete} <name> | pack [<name>] | update | list] +............... +Examples: +............... + session save bnc966622 + session load rsclost-2 + session list +............... + +=== `end` (`cd`, `up`) + +The `end` command ends the current level and the user moves to +the parent level. This command is available everywhere. + +Usage: +............... + end +............... + +=== `help` + +The `help` command prints help for the current level or for the +specified topic (command). This command is available everywhere. + +Usage: +............... + help [<topic>] +............... + +=== `quit` (`exit`, `bye`) + +Leave the program. + +BUGS +---- +Even though all sensible configurations (and most of those that +are not) are going to be supported by the crm shell, I suspect +that it may still happen that certain XML constructs may confuse +the tool. When that happens, please file a bug report. + +The crm shell will not try to update the objects it does not +understand. Of course, it is always possible to edit such objects +in the XML format. + +AUTHOR +------ +Dejan Muhamedagic, <dejan@suse.de> +and many OTHERS + +SEE ALSO +-------- +crm_resource(8), crm_attribute(8), crm_mon(8), cib_shadow(8), +ptest(8), dotty(1), crm_simulate(8), cibadmin(8) + + +COPYING +------- +Copyright \(C) 2008-2011 Dejan Muhamedagic. Free use of this +software is granted under the terms of the GNU General Public License (GPL). + +////////////////////// + vim:ts=4:sw=4:expandtab: +////////////////////// diff --git a/doc/website-v1/man-2.0.adoc b/doc/website-v1/man-2.0.adoc new file mode 100644 index 0000000..a2127d4 --- /dev/null +++ b/doc/website-v1/man-2.0.adoc @@ -0,0 +1,5048 @@ +:man source: crm +:man version: 2.3.2 +:man manual: crmsh documentation + +crm(8) +====== + +NAME +---- +crm - Pacemaker command line interface for configuration and management + + +SYNOPSIS +-------- +*crm* [OPTIONS] [SUBCOMMAND ARGS...] + + +[[topics_Description,Program description]] +DESCRIPTION +----------- +The `crm` shell is a command-line based cluster configuration and +management tool. Its goal is to assist as much as possible with the +configuration and maintenance of Pacemaker-based High Availability +clusters. + +For more information on Pacemaker itself, see http://clusterlabs.org/. + +`crm` works both as a command-line tool to be called directly from the +system shell, and as an interactive shell with extensive tab +completion and help. + +The primary focus of the `crm` shell is to provide a simplified and +consistent interface to Pacemaker, but it also provides tools for +managing the creation and configuration of High Availability clusters +from scratch. To learn more about this aspect of `crm`, see the +`cluster` section below. + +The `crm` shell can be used to manage every aspect of configuring and +maintaining a cluster. It provides a simplified line-based syntax on +top of the XML configuration format used by Pacemaker, commands for +starting and stopping resources, tools for exploring the history of a +cluster including log scraping and a set of cluster scripts useful for +automating the setup and installation of services on the cluster +nodes. + +The `crm` shell is line oriented: every command must start and finish +on the same line. It is possible to use a continuation character (+\+) +to write one command in two or more lines. The continuation character +is commonly used when displaying configurations. + +[[topics_CommandLine,Command line options]] +OPTIONS +------- +*-f, --file*='FILE':: + Load commands from the given file. If a dash +-+ is used in place + of a file name, `crm` will read commands from the shell standard + input (`stdin`). + +*-c, --cib*='CIB':: + Start the session using the given shadow CIB file. + Equivalent to +cib use <CIB>+. + +*-D, --display=*'OUTPUT_TYPE':: + Choose one of the output options: +plain+, +color-always+, +color+, + or +uppercase+. The default is +color+ if the terminal emulation + supports colors. Otherwise, +plain+ is used. + +*-F, --force*:: + Make `crm` proceed with applying changes where it would normally + ask the user to confirm before proceeding. This option is mainly + useful in scripts, and should be used with care. + +*-w, --wait*:: + Make `crm` wait for the cluster transition to finish (for the + changes to take effect) after each processed line. + +*-H, --history*='DIR|FILE|SESSION':: + A directory or file containing a cluster report to load + into the `history` commands, or the name of a previously + saved history session. + +*-h, --help*:: + Print help page. + +*--version*:: + Print crmsh version and build information (Mercurial Hg changeset + hash). + +*-d, --debug*:: + Print verbose debugging information. + +*-R, --regression-tests*:: + Enables extra verbose trace logging used by the regression + tests. Logs all external calls made by crmsh. + +*--scriptdir*='DIR':: + Extra directory where crm looks for cluster scripts, or a list of + directories separated by semi-colons (e.g. +/dir1;/dir2;etc.+). + +*-o, --opt*='OPTION=VALUE':: + Set crmsh option temporarily. If the options are saved using + +options save+ then the value passed here will also be saved. + Multiple options can be set by using +-o+ multiple times. + +[[topics_Introduction,Introduction]] +== Introduction + +This section of the user guide covers general topics about the user +interface and describes some of the features of `crmsh` in detail. + +[[topics_Introduction_Interface,User interface]] +=== User interface + +The main purpose of `crmsh` is to provide a simple yet powerful +interface to the cluster stack. There are two main modes of operation +with the user interface of `crmsh`: + +* Command line (single-shot) use - Use `crm` as a regular UNIX command + from your usual shell. `crm` has full bash completion built in, so + using it in this manner should be as comfortable and familiar as + using any other command-line tool. + +* Interactive mode - By calling `crm` without arguments, or by calling + it with only a sublevel as argument, `crm` enters the interactive + mode. In this mode, it acts as its own command shell, which + remembers which sublevel you are currently in and allows for rapid + and convenient execution of multiple commands within the same + sublevel. This mode also has full tab completion, as well as + built-in interactive help and syntax highlighting. + +Here are a few examples of using `crm` both as a command-line tool and +as an interactive shell: + +.Command line (one-shot) use: +........ +# crm resource stop www_app +........ + +.Interactive use: +........ +# crm +crm(live)# resource +crm(live)resource# unmanage tetris_1 +crm(live)resource# up +crm(live)# node standby node4 +........ + +.Cluster configuration: +........ +# crm configure<<EOF + # + # resources + # + primitive disk0 iscsi \ + params portal=192.168.2.108:3260 target=iqn.2008-07.com.suse:disk0 + primitive fs0 Filesystem \ + params device=/dev/disk/by-label/disk0 directory=/disk0 fstype=ext3 + primitive internal_ip IPaddr params ip=192.168.1.101 + primitive apache apache \ + params configfile=/disk0/etc/apache2/site0.conf + primitive apcfence stonith:apcsmart \ + params ttydev=/dev/ttyS0 hostlist="node1 node2" \ + op start timeout=60s + primitive pingd pingd \ + params name=pingd dampen=5s multiplier=100 host_list="r1 r2" + # + # monitor apache and the UPS + # + monitor apache 60s:30s + monitor apcfence 120m:60s + # + # cluster layout + # + group internal_www \ + disk0 fs0 internal_ip apache + clone fence apcfence \ + meta globally-unique=false clone-max=2 clone-node-max=1 + clone conn pingd \ + meta globally-unique=false clone-max=2 clone-node-max=1 + location node_pref internal_www \ + rule 50: #uname eq node1 \ + rule pingd: defined pingd + # + # cluster properties + # + property stonith-enabled=true + commit +EOF +........ + +The `crm` interface is hierarchical, with commands organized into +separate levels by functionality. To list the available levels and +commands, either execute +help <level>+, or, if at the top level of +the shell, simply typing `help` will provide an overview of all +available levels and commands. + +The +(live)+ string in the `crm` prompt signifies that the current CIB +in use is the cluster live configuration. It is also possible to +work with so-called <<topics_Features_Shadows,shadow CIBs>>. These are separate, inactive +configurations stored in files, that can be applied and thereby +replace the live configuration at any time. + +[[topics_Introduction_Completion,Tab completion]] +=== Tab completion + +The `crm` makes extensive use of tab completion. The completion +is both static (i.e. for `crm` commands) and dynamic. The latter +takes into account the current status of the cluster or +information from installed resource agents. Sometimes, completion +may also be used to get short help on resource parameters. Here +are a few examples: + +............... +crm(live)resource# <TAB><TAB> +bye failcount move restart unmigrate +cd help param show unmove +cleanup list promote start up +demote manage quit status utilization +end meta refresh stop +exit migrate reprobe unmanage + +crm(live)configure# primitive fence-1 <TAB><TAB> +heartbeat: lsb: ocf: stonith: + +crm(live)configure# primitive fence-1 stonith:<TAB><TAB> +apcmaster external/ippower9258 fence_legacy +apcmastersnmp external/kdumpcheck ibmhmc +apcsmart external/libvirt ipmilan + +crm(live)configure# primitive fence-1 stonith:ipmilan params <TAB><TAB> +auth= hostname= ipaddr= login= password= port= priv= + +crm(live)configure# primitive fence-1 stonith:ipmilan params auth=<TAB><TAB> +auth* (string) + The authorization type of the IPMI session ("none", "straight", "md2", or "md5") +............... + +`crmsh` also comes with bash completion usable directly from the +system shell. This should be installed automatically with the command +itself. + +[[topics_Introduction_Shorthand,Shorthand syntax]] +=== Shorthand syntax + +When using the `crm` shell to manage clusters, you will end up typing +a lot of commands many times over. Clear command names like ++configure+ help in understanding and learning to use the cluster +shell, but is easy to misspell and is tedious to type repeatedly. The +interactive mode and tab completion both help with this, but the `crm` +shell also has the ability to understand a variety of shorthand +aliases for all of the commands. + +For example, instead of typing `crm status`, you can type `crm st` or +`crm stat`. Instead of `crm configure` you can type `crm cfg` or even +`crm cf`. `crm resource` can be shorted as `crm rsc`, and so on. + +The exact list of accepted aliases is too long to print in full, but +experimentation and typos should help in discovering more of them. + +[[topics_Features,Features]] +== Features + +The feature set of crmsh covers a wide range of functionality, and +understanding how and when to use the various features of the shell +can be difficult. This section of the guide describes some of the +features and use cases of `crmsh` in more depth. The intention is to +provide a deeper understanding of these features, but also to serve as +a guide to using them. + +[[topics_Features_Shadows,Shadow CIB usage]] +=== Shadow CIB usage + +A Shadow CIB is a normal cluster configuration stored in a file. +They may be manipulated in much the same way as the _live_ CIB, with +the key difference that changes to a shadow CIB have no effect on the +actual cluster resources. An administrator may choose to apply any of +them to the cluster, thus replacing the running configuration with the +one found in the shadow CIB. + +The `crm` prompt always contains the name of the configuration which +is currently in use, or the string _live_ if using the live cluster +configuration. + +When editing the configuration in the `configure` level, no changes +are actually applied until the `commit` command is executed. It is +possible to start editing a configuration as usual, but instead of +committing the changes to the active CIB, save them to a shadow CIB. + +The following example `configure` session demonstrates how this can be +done: +............... +crm(live)configure# cib new test-2 +INFO: test-2 shadow CIB created +crm(test-2)configure# commit +............... + +[[topics_Features_Checks,Configuration semantic checks]] +=== Configuration semantic checks + +Resource definitions may be checked against the meta-data +provided with the resource agents. These checks are currently +carried out: + +- are required parameters set +- existence of defined parameters +- timeout values for operations + +The parameter checks are obvious and need no further explanation. +Failures in these checks are treated as configuration errors. + +The timeouts for operations should be at least as long as those +recommended in the meta-data. Too short timeout values are a +common mistake in cluster configurations and, even worse, they +often slip through if cluster testing was not thorough. Though +operation timeouts issues are treated as warnings, make sure that +the timeouts are usable in your environment. Note also that the +values given are just _advisory minimum_---your resources may +require longer timeouts. + +User may tune the frequency of checks and the treatment of errors +by the <<cmdhelp_options_check-frequency,`check-frequency`>> and +<<cmdhelp_options_check-mode,`check-mode`>> preferences. + +Note that if the +check-frequency+ is set to +always+ and the ++check-mode+ to +strict+, errors are not tolerated and such +configuration cannot be saved. + +[[topics_Features_Templates,Configuration templates]] +=== Configuration templates + +.Deprecation note +**************************** +Configuration templates have been deprecated in favor of the more +capable `cluster scripts`. To learn how to use cluster scripts, see +the dedicated documentation on the `crmsh` website at +http://crmsh.github.io/, or in the <<cmdhelp_script,Script section>>. +**************************** + +Configuration templates are ready made configurations created by +cluster experts. They are designed in such a way so that users +may generate valid cluster configurations with minimum effort. +If you are new to Pacemaker, templates may be the best way to +start. + +We will show here how to create a simple yet functional Apache +configuration: +............... +# crm configure +crm(live)configure# template +crm(live)configure template# list templates +apache filesystem virtual-ip +crm(live)configure template# new web <TAB><TAB> +apache filesystem virtual-ip +crm(live)configure template# new web apache +INFO: pulling in template apache +INFO: pulling in template virtual-ip +crm(live)configure template# list +web2-d web2 vip2 web3 vip web +............... + +We enter the `template` level from `configure`. Use the `list` +command to show templates available on the system. The `new` +command creates a configuration from the +apache+ template. You +can use tab completion to pick templates. Note that the apache +template depends on a virtual IP address which is automatically +pulled along. The `list` command shows the just created +web+ +configuration, among other configurations (I hope that you, +unlike me, will use more sensible and descriptive names). + +The `show` command, which displays the resulting configuration, +may be used to get an idea about the minimum required changes +which have to be done. All +ERROR+ messages show the line numbers +in which the respective parameters are to be defined: +............... +crm(live)configure template# show +ERROR: 23: required parameter ip not set +ERROR: 61: required parameter id not set +ERROR: 65: required parameter configfile not set +crm(live)configure template# edit +............... + +The `edit` command invokes the preferred text editor with the ++web+ configuration. At the top of the file, the user is advised +how to make changes. A good template should require from the user +to specify only parameters. For example, the +web+ configuration +we created above has the following required and optional +parameters (all parameter lines start with +%%+): +............... +$ grep -n ^%% ~/.crmconf/web +23:%% ip +31:%% netmask +35:%% lvs_support +61:%% id +65:%% configfile +71:%% options +76:%% envfiles +............... + +These lines are the only ones that should be modified. Simply +append the parameter value at the end of the line. For instance, +after editing this template, the result could look like this (we +used tabs instead of spaces to make the values stand out): +............... +$ grep -n ^%% ~/.crmconf/web +23:%% ip 192.168.1.101 +31:%% netmask +35:%% lvs_support +61:%% id websvc +65:%% configfile /etc/apache2/httpd.conf +71:%% options +76:%% envfiles +............... + +As you can see, the parameter line format is very simple: +............... +%% <name> <value> +............... + +After editing the file, use `show` again to display the +configuration: +............... +crm(live)configure template# show +primitive virtual-ip IPaddr \ + params ip=192.168.1.101 +primitive apache apache \ + params configfile="/etc/apache2/httpd.conf" +monitor apache 120s:60s +group websvc \ + apache virtual-ip +............... + +The target resource of the apache template is a group which we +named +websvc+ in this sample session. + +This configuration looks exactly as you could type it at the +`configure` level. The point of templates is to save you some +typing. It is important, however, to understand the configuration +produced. + +Finally, the configuration may be applied to the current +crm configuration (note how the configuration changed slightly, +though it is still equivalent, after being digested at the +`configure` level): +............... +crm(live)configure template# apply +crm(live)configure template# cd .. +crm(live)configure# show +node xen-b +node xen-c +primitive apache apache \ + params configfile="/etc/apache2/httpd.conf" \ + op monitor interval=120s timeout=60s +primitive virtual-ip IPaddr \ + params ip=192.168.1.101 +group websvc apache virtual-ip +............... + +Note that this still does not commit the configuration to the CIB +which is used in the shell, either the running one (+live+) or +some shadow CIB. For that you still need to execute the `commit` +command. + +To complete our example, we should also define the preferred node +to run the service: + +............... +crm(live)configure# location websvc-pref websvc 100: xen-b +............... + +If you are not happy with some resource names which are provided +by default, you can rename them now: + +............... +crm(live)configure# rename virtual-ip intranet-ip +crm(live)configure# show +node xen-b +node xen-c +primitive apache apache \ + params configfile="/etc/apache2/httpd.conf" \ + op monitor interval=120s timeout=60s +primitive intranet-ip IPaddr \ + params ip=192.168.1.101 +group websvc apache intranet-ip +location websvc-pref websvc 100: xen-b +............... + +To summarize, working with templates typically consists of the +following steps: + +- `new`: create a new configuration from templates +- `edit`: define parameters, at least the required ones +- `show`: see if the configuration is valid +- `apply`: apply the configuration to the `configure` level + +[[topics_Features_Testing,Resource testing]] +=== Resource testing + +The amount of detail in a cluster makes all configurations prone +to errors. By far the largest number of issues in a cluster is +due to bad resource configuration. The shell can help quickly +diagnose such problems. And considerably reduce your keyboard +wear. + +Let's say that we entered the following configuration: +............... +node xen-b +node xen-c +node xen-d +primitive fencer stonith:external/libvirt \ + params hypervisor_uri="qemu+tcp://10.2.13.1/system" \ + hostlist="xen-b xen-c xen-d" \ + op monitor interval=2h +primitive svc Xinetd \ + params service=systat \ + op monitor interval=30s +primitive intranet-ip IPaddr2 \ + params ip=10.2.13.100 \ + op monitor interval=30s +primitive apache apache \ + params configfile="/etc/apache2/httpd.conf" \ + op monitor interval=120s timeout=60s +group websvc apache intranet-ip +location websvc-pref websvc 100: xen-b +............... + +Before typing `commit` to submit the configuration to the cib we +can make sure that all resources are usable on all nodes: +............... +crm(live)configure# rsctest websvc svc fencer +............... + +It is important that resources being tested are not running on +any nodes. Otherwise, the `rsctest` command will refuse to do +anything. Of course, if the current configuration resides in a +CIB shadow, then a `commit` is irrelevant. The point being that +resources are not running on any node. + +.Note on stopping all resources +**************************** +Alternatively to not committing a configuration, it is also +possible to tell Pacemaker not to start any resources: + +............... +crm(live)configure# property stop-all-resources=yes +............... +Almost none---resources of class stonith are still started. But +shell is not as strict when it comes to stonith resources. +**************************** + +Order of resources is significant insofar that a resource depends +on all resources to its left. In most configurations, it's +probably practical to test resources in several runs, based on +their dependencies. + +Apart from groups, `crm` does not interpret constraints and +therefore knows nothing about resource dependencies. It also +doesn't know if a resource can run on a node at all in case of an +asymmetric cluster. It is up to the user to specify a list of +eligible nodes if a resource is not meant to run on every node. + +[[topics_Features_Security,Access Control Lists (ACL)]] +=== Access Control Lists (ACL) + +.Note on ACLs in Pacemaker 1.1.12 +**************************** +The support for ACLs has been revised in Pacemaker version 1.1.12 and +up. Depending on which version you are using, the information in this +section may no longer be accurate. Look for the `acl_target` +configuration element for more details on the new syntax. +**************************** + +By default, the users from the +haclient+ group have full access +to the cluster (or, more precisely, to the CIB). Access control +lists allow for finer access control to the cluster. + +Access control lists consist of an ordered set of access rules. +Each rule allows read or write access or denies access +completely. Rules are typically combined to produce a specific +role. Then, users may be assigned a role. + +For instance, this is a role which defines a set of rules +allowing management of a single resource: + +............... +role bigdb_admin \ + write meta:bigdb:target-role \ + write meta:bigdb:is-managed \ + write location:bigdb \ + read ref:bigdb +............... + +The first two rules allow modifying the +target-role+ and ++is-managed+ meta attributes which effectively enables users in +this role to stop/start and manage/unmanage the resource. The +constraints write access rule allows moving the resource around. +Finally, the user is granted read access to the resource +definition. + +For proper operation of all Pacemaker programs, it is advisable +to add the following role to all users: + +............... +role read_all \ + read cib +............... + +For finer grained read access try with the rules listed in the +following role: + +............... +role basic_read \ + read node attribute:uname \ + read node attribute:type \ + read property \ + read status +............... + +It is however possible that some Pacemaker programs (e.g. +`ptest`) may not function correctly if the whole CIB is not +readable. + +Some of the ACL rules in the examples above are expanded by the +shell to XPath specifications. For instance, ++meta:bigdb:target-role+ expands to: + +........ +//primitive[@id='bigdb']/meta_attributes/nvpair[@name='target-role'] +........ + +You can see the expansion by showing XML: + +............... +crm(live) configure# show xml bigdb_admin +... +<acls> + <acl_role id="bigdb_admin"> + <write id="bigdb_admin-write" + xpath="//primitive[@id='bigdb']/meta_attributes/nvpair[@name='target-role']"/> +............... + +Many different XPath expressions can have equal meaning. For +instance, the following two are equal, but only the first one is +going to be recognized as shortcut: + +............... +//primitive[@id='bigdb']/meta_attributes/nvpair[@name='target-role'] +//resources/primitive[@id='bigdb']/meta_attributes/nvpair[@name='target-role'] +............... + +XPath is a powerful language, but you should try to keep your ACL +xpaths simple and the builtin shortcuts should be used whenever +possible. + +[[topics_Features_Resourcesets,Syntax: Resource sets]] +=== Syntax: Resource sets + +Using resource sets can be a bit confusing unless one knows the +details of the implementation in Pacemaker as well as how to interpret +the syntax provided by `crmsh`. + +Three different types of resource sets are provided by `crmsh`, and +each one implies different values for the two resource set attributes, ++sequential+ and +require-all+. + ++sequential+:: + If false, the resources in the set do not depend on each other + internally. Setting +sequential+ to +true+ implies a strict order of + dependency within the set. + ++require-all+:: + If false, only one resource in the set is required to fulfil the + requirements of the set. The set of A, B and C with +require-all+ + set to +false+ is be read as "A OR B OR C" when its dependencies + are resolved. + +The three types of resource sets modify the attributes in the +following way: + +1. Implicit sets (no brackets). +sequential=true+, +require-all=true+ +2. Parenthesis set (+(+ ... +)+). +sequential=false+, +require-all=true+ +3. Bracket set (+[+ ... +]+). +sequential=false+, +require-all=false+ + +To create a set with the properties +sequential=true+ and ++require-all=false+, explicitly set +sequential+ in a bracketed set, ++[ A B C sequential=true ]+. + +To create multiple sets with both +sequential+ and +require-all+ set to +true, explicitly set +sequential+ in a parenthesis set: ++A B ( C D sequential=true )+. + +[[topics_Features_AttributeListReferences,Syntax: Attribute list references]] +=== Syntax: Attribute list references + +Attribute lists are used to set attributes and parameters for +resources, constraints and property definitions. For example, to set +the virtual IP used by an +IPAddr2+ resource the attribute +ip+ can be +set in an attribute list for that resource. + +Attribute lists can have identifiers that name them, and other +resources can reuse the same attribute list by referring to that name +using an +$id-ref+. For example, the following statement defines a +simple dummy resource with an attribute list which sets the parameter ++state+ to the value 1 and sets the identifier for the attribute list +to +on-state+: + +.............. +primitive dummy-1 Dummy params $id=on-state state=1 +.............. + +To refer to this attribute list from a different resource, refer to +the +on-state+ name using an id-ref: + +.............. +primitive dummy-2 Dummy params $id-ref=on-state +.............. + +The resource +dummy-2+ will now also have the parameter +state+ set to the value 1. + +[[topics_Features_AttributeReferences,Syntax: Attribute references]] +=== Syntax: Attribute references + +In some cases, referencing complete attribute lists is too +coarse-grained, for example if two different parameters with different +names should have the same value set. Instead of having to copy the +value in multiple places, it is possible to create references to +individual attributes in attribute lists. + +To name an attribute in order to be able to refer to it later, prefix +the attribute name with a +$+ character (as seen above with the +special names +$id+ and +$id-ref+: + +............ +primitive dummy-1 Dummy params $state=1 +............ + +The identifier +state+ can now be used to refer to this attribute from other +primitives, using the +@<id>+ syntax: + +............ +primitive dummy-2 Dummy params @state +............ + +In some cases, using the attribute name as the identifier doesn't work +due to name clashes. In this case, the syntax +$<id>:<name>=<value>+ +can be used to give the attribute a different identifier: + +............ +primitive dummy-1 params $dummy-state-on:state=1 +primitive dummy-2 params @dummy-state-on +............ + +There is also the possibility that two resources both use the same +attribute value but with different names. For example, a web server +may have a parameter +server_ip+ for setting the IP address where it +listens for incoming requests, and a virtual IP resource may have a +parameter called +ip+ which sets the IP address it creates. To +configure these two resources with an IP without repeating the value, +the reference can be given a name using the syntax +@<id>:<name>+. + +Example: +............ +primitive virtual-ip IPaddr2 params $vip:ip=192.168.1.100 +primitive webserver apache params @vip:server_ip +............ + +[[topics_Syntax_RuleExpressions,Syntax: Rule expressions]] +=== Syntax: Rule expressions + +Many of the configuration commands in `crmsh` now support the use of +_rule expressions_, which can influence what attributes apply to a +resource or under which conditions a constraint is applied, depending +on changing conditions like date, time, the value of attributes and +more. + +Here is an example of a simple rule expression used to apply a +a different resource parameter on the node named `node1`: + +.............. +primitive my_resource Special \ + params 2: rule #uname eq node1 interface=eth1 \ + params 1: interface=eth0 +.............. + +This primitive resource has two lists of parameters with descending +priority. The parameter list with the highest priority is applied +first, but only if the rule expressions for that parameter list all +apply. In this case, the rule `#uname eq node1` limits the parameter +list so that it is only applied on `node1`. + +Note that rule expressions are not terminated and are immediately +followed by the data to which the rule is applied. In this case, the +name-value pair `interface=eth1`. + +Rule expressions can contain multiple expressions connected using the +boolean operator `or` and `and`. The full syntax for rule expressions +is listed below. + +.............. +rules :: + rule [id_spec] [$role=<role>] <score>: <expression> + [rule [id_spec] [$role=<role>] <score>: <expression> ...] + +id_spec :: $id=<id> | $id-ref=<id> +score :: <number> | <attribute> | [-]inf +expression :: <simple_exp> [<bool_op> <simple_exp> ...] +bool_op :: or | and +simple_exp :: <attribute> [type:]<binary_op> <value> + | <unary_op> <attribute> + | date <date_expr> +type :: <string> | <version> | <number> +binary_op :: lt | gt | lte | gte | eq | ne +unary_op :: defined | not_defined + +date_expr :: lt <end> + | gt <start> + | in start=<start> end=<end> + | in start=<start> <duration> + | spec <date_spec> +duration|date_spec :: + hours=<value> + | monthdays=<value> + | weekdays=<value> + | yearsdays=<value> + | months=<value> + | weeks=<value> + | years=<value> + | weekyears=<value> + | moon=<value> +.............. + +[[topics_Reference,Command reference]] +== Command reference + +The commands are structured to be compatible with the shell command +line. Sometimes, the underlying Pacemaker grammar uses characters that +have special meaning in bash, that will need to be quoted. This +includes the hash or pound sign (`#`), single and double quotes, and +any significant whitespace. + +Whitespace is also significant when assigning values, meaning that ++key=value+ is different from +key = value+. + +Commands can be referenced using short-hand as long as the short-hand +is unique. This can be either a prefix of the command name or a prefix +string of characters found in the name. + +For example, +status+ can be abbreviated as +st+ or +su+, and ++configure+ as +conf+ or +cfg+. + +The syntax for the commands is given below in an informal, BNF-like +grammar. + +* `<value>` denotes a string. +* `[value]` means that the construct is optional. +* The ellipsis (`...`) signifies that the previous construct may be + repeated. +* `first|second` means either first or second. +* The rest are literals (strings, `:`, `=`). + +[[cmdhelp_root_status,Cluster status]] +=== `status` + +Show cluster status. The status is displayed by `crm_mon`. Supply +additional arguments for more information or different format. +See `crm_mon(8)` for more details. + +Example: +............... +status +status simple +status full +............... + +Usage: +............... +status [<option> ...] + +option :: full + | bynode + | inactive + | ops + | timing + | failcounts + | verbose + | quiet + | html + | xml + | simple + | tickets + | noheaders + | detail + | brief +............... + +[[cmdhelp_root_verify,Verify cluster status]] +=== `verify` + +Performs basic checks for the cluster configuration and +current status, reporting potential issues. + +See `crm_verify(8)` and `crm_simulate(8)` for more details. + +Example: +............... +verify +verify scores +............... + +Usage: +............... +verify [scores] +............... + + +[[cmdhelp_cluster,Cluster setup and management]] +=== `cluster` - Cluster setup and management + +Whole-cluster configuration management with High Availability +awareness. + +The commands on the cluster level allows configuration and +modification of the underlying cluster infrastructure, and also +supplies tools to do whole-cluster systems management. + +These commands enable easy installation and maintenance of a HA +cluster, by providing support for package installation, configuration +of the cluster messaging layer, file system setup and more. + +[[cmdhelp_cluster_add,Add a new node to the cluster]] +==== `add` + +This command simplifies the process of adding a new node to a running +cluster. The new node will be installed and configured with the +packages and configuration files needed to run the cluster +resources. If a cluster file system is used, the new node will be set +up to host the file system. + +This command should be executed from a node already in the cluster. + +Usage: +............... +add <node> +............... + +[[cmdhelp_cluster_copy,Copy file to other cluster nodes]] +==== `copy` + +Copy file to other cluster nodes. + +Copies the given file to all other nodes unless given a +list of nodes to copy to as argument. + +Usage: +............... +copy <filename> [nodes ...] +............... + +Example: +............... +copy /etc/motd +............... + +[[cmdhelp_cluster_diff,Diff file across cluster]] +==== `diff` + +Displays the difference, if any, between a given file +on different nodes. If the second argument is `--checksum`, +a checksum of the file will be calculated and displayed for +each node. + +Usage: +............... +diff <file> [--checksum] [nodes...] +............... + +Example: +............... +diff /etc/crm/crm.conf node2 +diff /etc/resolv.conf --checksum +............... + +[[cmdhelp_cluster_health,Cluster health check]] +==== `health` + +Runs a larger set of tests and queries on all nodes in the cluster to +verify the general system health and detect potential problems. + +Usage: +............... +health +............... + +[[cmdhelp_cluster_init,Initializes a new HA cluster]] +==== `init` + +Installs and configures a basic HA cluster on a set of nodes. + +Usage: +........ +init node1 node2 node3 +init --dry-run node1 node2 node3 +........ + +[[cmdhelp_cluster_remove,Remove a node from the cluster]] +==== `remove` + +This command simplifies the process of removing a node from the +cluster, moving any resources hosted by that node to other nodes. + +Usage: +............... +remove <node> +............... + +[[cmdhelp_cluster_run,Execute an arbitrary command on all nodes]] +==== `run` + +This command takes a shell statement as argument, executes that +statement on all nodes in the cluster, and reports the result. + +Usage: +............... +run <command> +............... + +Example: +............... +run "cat /proc/uptime" +............... + +[[cmdhelp_cluster_start,Start cluster services]] +==== `start` + +Starts the cluster-related system services on this node. + +Usage: +......... +start +......... + +[[cmdhelp_cluster_status,Cluster status check]] +==== `status` + +Reports the status for the cluster messaging layer on the local +node. + +Usage: +............... +status +............... + +[[cmdhelp_cluster_stop,Stop cluster services]] +==== `stop` + +Stops the cluster-related system services on this node. + +Usage: +......... +stop +......... + +[[cmdhelp_cluster_wait_for_startup,Wait for cluster to start]] +==== `wait_for_startup` + +Mostly useful in scripts or automated workflows, this command will +attempt to connect to the local cluster node repeatedly. The command +will keep trying until the cluster node responds, or the `timeout` +elapses. The timeout can be changed by supplying a value in seconds as +an argument. + +Usage: +........ +wait_for_startup +........ + +[[cmdhelp_script,Cluster script management]] +=== `script` - Cluster script management + +A big part of the configuration and management of a cluster is +collecting information about all cluster nodes and deploying changes +to those nodes. Often, just performing the same procedure on all nodes +will encounter problems, due to subtle differences in the +configuration. + +For example, when configuring a cluster for the first time, the +software needs to be installed and configured on all nodes before the +cluster software can be launched and configured using `crmsh`. This +process is cumbersome and error-prone, and the goal is for scripts to +make this process easier. + +Scripts are implemented using the python `parallax` package which +provides a thin wrapper on top of SSH. This allows the scripts to +function through the usual SSH channels used for system maintenance, +requiring no additional software to be installed or maintained. + +[[cmdhelp_script_json,JSON API for cluster scripts]] +==== `json` + +This command provides a JSON API for the cluster scripts, intended for +use in user interface tools that want to interact with the cluster via +scripts. + +The command takes a single argument, which should be a JSON array with +the first member identifying the command to perform. + +The output is line-based: Commands that return multiple results will +return them line-by-line, ending with a terminator value: "end". + +When providing parameter values to this command, they should be +provided as nested objects, so +virtual-ip:ip=192.168.0.5+ on the +command line becomes the JSON object ++{"virtual-ip":{"ip":"192.168.0.5"}}+. + +API: +........ +["list"] +=> [{name, shortdesc, category}] + +["show", <name>] +=> [{name, shortdesc, longdesc, category, <<steps>>}] + +<<steps>> := [{name, shortdesc], longdesc, required, parameters, steps}] + +<<params>> := [{name, shortdesc, longdesc, required, unique, advanced, + type, value, example}] + +["verify", <name>, <<values>>] +=> [{shortdesc, longdesc, text, nodes}] + +["run", <name>, <<values>>] +=> [{shortdesc, rc, output|error}] +........ + + +[[cmdhelp_script_list,List available scripts]] +==== `list` + +Lists the available scripts, sorted by category. Scripts that have the +special `Script` category are hidden by default, since they are mainly +used by other scripts or commands. To also show these, pass `all` as +argument. + +To get a flat list of script names, not sorted by category, pass +`names` as an extra argument. + +Usage: +............ +list [all] [names] +............ + +Example: +............ +list +list all names +............ + +[[cmdhelp_script_run,Run the script]] +==== `run` + +Given a list of parameter values, this command will execute the +actions specified by the cluster script. The format for the parameter +values is the same as for the `verify` command. + +Can optionally take at least two parameters: +* `nodes=<nodes>`: List of nodes that the script runs over +* `dry_run=yes|no`: If set, the script will not perform any modifications. + +Additional parameters may be available depending on the script. + +Use the `show` command to see what parameters are available. + +Usage: +............. +run <script> [args...] +............. + +Example: +............. +run apache install=true +run sbd id=sbd-1 node=node1 sbd_device=/dev/disk/by-uuid/F00D-CAFE +............. + +[[cmdhelp_script_show,Describe the script]] +==== `show` + +Prints a description and short summary of the script, with +descriptions of the accepted parameters. + +Advanced parameters are hidden by default. To show the complete list +of parameters accepted by the script, pass `all` as argument. + +Usage: +............ +show <script> [all] +............ + +Example: +............ +show virtual-ip +............ + +[[cmdhelp_script_verify,Verify the script]] +==== `verify` + +Checks the given parameter values, and returns a list +of actions that will be executed when running the script +if provided the same list of parameter values. + +Usage: +............ +verify <script> [args...] +............ + +Example: +............ +verify sbd id=sbd-1 node=node1 sbd_device=/dev/disk/by-uuid/F00D-CAFE +............ + +[[cmdhelp_corosync,Corosync management]] +=== `corosync` - Corosync management + +Corosync is the underlying messaging layer for most HA clusters. +This level provides commands for editing and managing the corosync +configuration. + +[[cmdhelp_corosync_add-node,Add a corosync node]] +==== `add-node` + +Adds a node to the corosync configuration. This is used with the `udpu` +type configuration in corosync. + +A nodeid for the added node is generated automatically. + +Note that this command assumes that only a single ring is used, and +sets only the address for ring0. + +Usage: +......... +add-node <addr> [name] +......... + +[[cmdhelp_corosync_del-node,Remove a corosync node]] +==== `del-node` + +Removes a node from the corosync configuration. The argument given is +the `ring0_addr` address set in the configuration file. + +Usage: +......... +del-node <addr> +......... + +[[cmdhelp_corosync_diff,Diffs the corosync configuration]] +==== `diff` + +Diffs the corosync configurations on different nodes. If no nodes are +given as arguments, the corosync configurations on all nodes in the +cluster are compared. + +`diff` takes an option argument `--checksum`, to display a checksum +for each file instead of calculating a diff. + +Usage: +......... +diff [--checksum] [node...] +......... + +[[cmdhelp_corosync_edit,Edit the corosync configuration]] +==== `edit` + +Opens the Corosync configuration file in an editor. + +Usage: +......... +edit +......... + +[[cmdhelp_corosync_get,Get a corosync configuration value]] +==== `get` + +Returns the value configured in `corosync.conf`, which is not +necessarily the value used in the running configuration. See `reload` +for telling corosync about configuration changes. + +The argument is the complete dot-separated path to the value. + +If there are multiple values configured with the same path, the +command returns all values for that path. For example, to get all +configured `ring0_addr` values, use this command: + +Example: +........ +get nodelist.node.ring0_addr +........ + +[[cmdhelp_corosync_log,Show the corosync log file]] +==== `log` + +Opens the log file specified in the corosync configuration file. If no +log file is configured, this command returns an error. + +The pager used can be configured either using the PAGER +environment variable or in `crm.conf`. + +Usage: +......... +log +......... + +[[cmdhelp_corosync_pull,Pulls the corosync configuration]] +==== `pull` + +Gets the corosync configuration from another node and copies +it to this node. + +Usage: +......... +pull <node> +......... + +[[cmdhelp_corosync_push,Push the corosync configuration]] +==== `push` + +Pushes the corosync configuration file on this node to +the list of nodes provided. If no target nodes are given, +the configuration is pushed to all other nodes in the cluster. + +It is recommended to use `csync2` to distribute the cluster +configuration files rather than relying on this command. + +Usage: +......... +push [node] ... +......... + +Example: +......... +push node-2 node-3 +......... + +[[cmdhelp_corosync_reload,Reload the corosync configuration]] +==== `reload` + +Tells all instances of corosync in this cluster to reload +`corosync.conf`. + +After pushing a new configuration to all cluster nodes, call this +command to make corosync use the new configuration. + +Usage: +......... +reload +......... + +[[cmdhelp_corosync_set,Set a corosync configuration value]] +==== `set` + +Sets the value identified by the given path. If the value does not +exist in the configuration file, it will be added. However, if the +section containing the value does not exist, the command will fail. + +Usage: +......... +set quorum.expected_votes 2 +......... + +[[cmdhelp_corosync_show,Display the corosync configuration]] +==== `show` + +Displays the corosync configuration on the current node. + +......... +show +......... + +[[cmdhelp_corosync_status,Display the corosync status]] +==== `status` + +Displays the status of Corosync, including the votequorum state. + +Usage: +......... +status +......... + +[[cmdhelp_cib,CIB shadow management]] +=== `cib` - CIB shadow management + +This level is for management of shadow CIBs. It is available both +at the top level and the `configure` level. + +All the commands are implemented using `cib_shadow(8)` and the +`CIB_shadow` environment variable. The user prompt always +includes the name of the currently active shadow or the live CIB. + +[[cmdhelp_cib_cibstatus,CIB status management and editing]] +==== `cibstatus` + +Enter edit and manage the CIB status section level. See the +<<cmdhelp_cibstatus,CIB status management section>>. + +[[cmdhelp_cib_commit,copy a shadow CIB to the cluster]] +==== `commit` + +Apply a shadow CIB to the cluster. If the shadow name is omitted +then the current shadow CIB is applied. + +Temporary shadow CIBs are removed automatically on commit. + +Usage: +............... +commit [<cib>] +............... + +[[cmdhelp_cib_delete,delete a shadow CIB]] +==== `delete` + +Delete an existing shadow CIB. + +Usage: +............... +delete <cib> +............... + +[[cmdhelp_cib_diff,diff between the shadow CIB and the live CIB]] +==== `diff` + +Print differences between the current cluster configuration and +the active shadow CIB. + +Usage: +............... +diff +............... + +[[cmdhelp_cib_import,import a CIB or PE input file to a shadow]] +==== `import` + +At times it may be useful to create a shadow file from the +existing CIB. The CIB may be specified as file or as a PE input +file number. The shell will look up files in the local directory +first and then in the PE directory (typically `/var/lib/pengine`). +Once the CIB file is found, it is copied to a shadow and this +shadow is immediately available for use at both `configure` and +`cibstatus` levels. + +If the shadow name is omitted then the target shadow is named +after the input CIB file. + +Note that there are often more than one PE input file, so you may +need to specify the full name. + +Usage: +............... +import {<file>|<number>} [<shadow>] +............... +Examples: +............... +import pe-warn-2222 +import 2289 issue2 +............... + +[[cmdhelp_cib_list,list all shadow CIBs]] +==== `list` + +List existing shadow CIBs. + +Usage: +............... +list +............... + +[[cmdhelp_cib_new,create a new shadow CIB]] +==== `new` + +Create a new shadow CIB. The live cluster configuration and +status is copied to the shadow CIB. + +If the name of the shadow is omitted, we create a temporary CIB +shadow. It is useful if multiple level sessions are desired +without affecting the cluster. A temporary CIB shadow is short +lived and will be removed either on `commit` or on program exit. +Note that if the temporary shadow is not committed all changes in +the temporary shadow are lost. + +Specify `withstatus` if you want to edit the status section of +the shadow CIB (see the <<cmdhelp_cibstatus,cibstatus section>>). +Add `force` to force overwriting the existing shadow CIB. + +To start with an empty configuration that is not copied from the live +CIB, specify the `empty` keyword. (This also allows a shadow CIB to be +created in case no cluster is running.) + +Usage: +............... +new [<cib>] [withstatus] [force] [empty] +............... + +[[cmdhelp_cib_reset,copy live cib to a shadow CIB]] +==== `reset` + +Copy the current cluster configuration into the shadow CIB. + +Usage: +............... +reset <cib> +............... + +[[cmdhelp_cib_use,change working CIB]] +==== `use` + +Choose a CIB source. If you want to edit the status from the +shadow CIB specify `withstatus` (see <<cmdhelp_cibstatus,`cibstatus`>>). +Leave out the CIB name to switch to the running CIB. + +Usage: +............... +use [<cib>] [withstatus] +............... + +[[cmdhelp_ra,Resource Agents (RA) lists and documentation]] +=== `ra` - Resource Agents (RA) lists and documentation + +This level contains commands which show various information about +the installed resource agents. It is available both at the top +level and at the `configure` level. + +[[cmdhelp_ra_classes,list classes and providers]] +==== `classes` + +Print all resource agents' classes and, where appropriate, a list +of available providers. + +Usage: +............... +classes +............... + +[[cmdhelp_ra_info,show meta data for a RA]] +==== `info` (`meta`) + +Show the meta-data of a resource agent type. This is where users +can find information on how to use a resource agent. It is also +possible to get information from some programs: `pengine`, +`crmd`, `cib`, and `stonithd`. Just specify the program name +instead of an RA. + +Usage: +............... +info [<class>:[<provider>:]]<type> +info <type> <class> [<provider>] (obsolete) +............... +Example: +............... +info apache +info ocf:pacemaker:Dummy +info stonith:ipmilan +info pengine +............... + +[[cmdhelp_ra_list,list RA for a class (and provider)]] +==== `list` + +List available resource agents for the given class. If the class +is `ocf`, supply a provider to get agents which are available +only from that provider. + +Usage: +............... +list <class> [<provider>] +............... +Example: +............... +list ocf pacemaker +............... + +[[cmdhelp_ra_providers,show providers for a RA and a class]] +==== `providers` + +List providers for a resource agent type. The class parameter +defaults to `ocf`. + +Usage: +............... +providers <type> [<class>] +............... +Example: +............... +providers apache +............... + +[[cmdhelp_ra_validate,validate parameters for RA]] +==== `validate` + +If the resource agent supports the `validate-all` action, this calls +the action with the given parameters, printing any warnings or errors +reported by the agent. + +Usage: +................ +validate <agent> [<key>=<value> ...] +................ + +[[cmdhelp_resource,Resource management]] +=== `resource` - Resource management + +At this level resources may be managed. + +All (or almost all) commands are implemented with the CRM tools +such as `crm_resource(8)`. + +[[cmdhelp_resource_ban,ban a resource from a node]] +==== `ban` + +Ban a resource from running on a certain node. If no node is given +as argument, the resource is banned from the current location. + +See `move` for details on other arguments. + +Usage: +............... +ban <rsc> [<node>] [<lifetime>] [force] +............... + +[[cmdhelp_resource_cleanup,cleanup resource status]] +==== `cleanup` + +Cleanup resource status. Typically done after the resource has +temporarily failed. If a node is omitted, cleanup on all nodes. +If there are many nodes, the command may take a while. + ++(Pacemaker 1.1.14)+ Pass force to cleanup the resource itself, +otherwise the cleanup command will apply to the parent resource (if +any). + +Usage: +............... +cleanup <rsc> [<node>] [force] +............... + +[[cmdhelp_resource_clear,Clear any relocation constraint]] +==== `clear` (`unmove`, `unmigrate`, `unban`) + +Remove any relocation constraint created by +the `move`, `migrate` or `ban` command. + +Usage: +............... +clear <rsc> +unmigrate <rsc> +unban <rsc> +............... + +[[cmdhelp_resource_constraints,Show constraints affecting a resource]] +==== `constraints` + +Display the location and colocation constraints affecting the +resource. + +Usage: +................ +constraints <rsc> +................ + +[[cmdhelp_resource_demote,demote a master-slave resource]] +==== `demote` + +Demote a master-slave resource using the `target-role` +attribute. + +Usage: +............... +demote <rsc> +............... + +[[cmdhelp_resource_failcount,manage failcounts]] +==== `failcount` + +Show/edit/delete the failcount of a resource. + +Usage: +............... +failcount <rsc> set <node> <value> +failcount <rsc> delete <node> +failcount <rsc> show <node> +............... +Example: +............... +failcount fs_0 delete node2 +............... + +[[cmdhelp_resource_locate,show the location of resources]] +==== `locate` + +Show the current location of one or more resources. + +Usage: +............... +locate [<rsc> ...] +............... + +[[cmdhelp_resource_maintenance,Enable/disable per-resource maintenance mode]] +==== `maintenance` + +Enables or disables the per-resource maintenance mode. When this mode +is enabled, no monitor operations will be triggered for the resource. +`maintenance` attribute conflicts with the `is-managed`. When setting +the `maintenance` attribute, the user is proposed to remove the +`is-managed` attribute if it exists. + +Usage: +.................. +maintenance <resource> [on|off|true|false] +.................. + +Example: +.................. +maintenance rsc1 +maintenance rsc2 off +.................. + +[[cmdhelp_resource_manage,put a resource into managed mode]] +==== `manage` + +Manage a resource using the `is-managed` attribute. If there +are multiple meta attributes sets, the attribute is set in all of +them. If the resource is a clone, all `is-managed` attributes are +removed from the children resources. +`is-managed` attribute conflicts with the `maintenance`. When setting +the `is-managed` attribute, the user is proposed to remove the +`maintenance` attribute if it exists. + +For details on group management see <<cmdhelp_options_manage-children,`options manage-children`>>. + +Usage: +............... +manage <rsc> +............... + +[[cmdhelp_resource_meta,manage a meta attribute]] +==== `meta` + +Show/edit/delete a meta attribute of a resource. Currently, all +meta attributes of a resource may be managed with other commands +such as `resource stop`. + +Usage: +............... +meta <rsc> set <attr> <value> +meta <rsc> delete <attr> +meta <rsc> show <attr> +............... +Example: +............... +meta ip_0 set target-role stopped +............... + +[[cmdhelp_resource_move,Move a resource to another node]] +==== `move` (`migrate`) + +Move a resource away from its current location. + +If the destination node is left out, the resource is migrated by +creating a constraint which prevents it from running on the current +node. For this type of constraint to be created, the +force+ argument +is required. + +A lifetime may be given for the constraint. Once it expires, the +location constraint will no longer be active. + +Usage: +............... +move <rsc> [<node>] [<lifetime>] [force] +............... + +[[cmdhelp_resource_operations,Show active resource operations]] +==== `operations` + +Show active operations, optionally filtered by resource and node. + +Usage: +................ +operations [<rsc>] [<node>] +................ + +[[cmdhelp_resource_param,manage a parameter of a resource]] +==== `param` + +Show/edit/delete a parameter of a resource. + +Usage: +............... +param <rsc> set <param> <value> +param <rsc> delete <param> +param <rsc> show <param> +............... +Example: +............... +param ip_0 show ip +............... + +[[cmdhelp_resource_promote,promote a master-slave resource]] +==== `promote` + +Promote a master-slave resource using the `target-role` +attribute. + +Usage: +............... +promote <rsc> +............... + +[[cmdhelp_resource_refresh,refresh CIB from the LRM status]] +==== `refresh` + +Refresh CIB from the LRM status. + +.Note +**************************** +`refresh` has been deprecated and is now +an alias for `cleanup`. +**************************** + +Usage: +............... +refresh [<node>] +............... + +[[cmdhelp_resource_reprobe,probe for resources not started by the CRM]] +==== `reprobe` + +Probe for resources not started by the CRM. + +.Note +**************************** +`reprobe` has been deprecated and is now +an alias for `cleanup`. +**************************** + +Usage: +............... +reprobe [<node>] +............... + +[[cmdhelp_resource_restart,restart resources]] +==== `restart` + +Restart one or more resources. This is essentially a shortcut for +resource stop followed by a start. The shell is first going to wait +for the stop to finish, that is for all resources to really stop, and +only then to order the start action. Due to this command +entailing a whole set of operations, informational messages are +printed to let the user see some progress. + +For details on group management see +<<cmdhelp_options_manage-children,`options manage-children`>>. + +Usage: +............... +restart <rsc> [<rsc> ...] +............... +Example: +............... +# crm resource restart g_webserver +INFO: ordering g_webserver to stop +waiting for stop to finish .... done +INFO: ordering g_webserver to start +# +............... + +[[cmdhelp_resource_scores,Display resource scores]] +==== `scores` + +Display the allocation scores for all resources. + +Usage: +................ +scores +................ + +[[cmdhelp_resource_secret,manage sensitive parameters]] +==== `secret` + +Sensitive parameters can be kept in local files rather than CIB +in order to prevent accidental data exposure. Use the `secret` +command to manage such parameters. `stash` and `unstash` move the +value from the CIB and back to the CIB respectively. The `set` +subcommand sets the parameter to the provided value. `delete` +removes the parameter completely. `show` displays the value of +the parameter from the local file. Use `check` to verify if the +local file content is valid. + +Usage: +............... +secret <rsc> set <param> <value> +secret <rsc> stash <param> +secret <rsc> unstash <param> +secret <rsc> delete <param> +secret <rsc> show <param> +secret <rsc> check <param> +............... +Example: +............... +secret fence_1 show password +secret fence_1 stash password +secret fence_1 set password secret_value +............... + +[[cmdhelp_resource_start,start resources]] +==== `start` + +Start one or more resources by setting the `target-role` attribute. If +there are multiple meta attributes sets, the attribute is set in all +of them. If the resource is a clone, all `target-role` attributes are +removed from the children resources. + +For details on group management see +<<cmdhelp_options_manage-children,`options manage-children`>>. + +Usage: +............... +start <rsc> [<rsc> ...] +............... + +[[cmdhelp_resource_status,show status of resources]] +==== `status` (`show`, `list`) + +Print resource status. More than one resource can be shown at once. If +the resource parameter is left out, the status of all resources is +printed. + +Usage: +............... +status [<rsc> ...] +............... + +[[cmdhelp_resource_stop,stop resources]] +==== `stop` + +Stop one or more resources using the `target-role` attribute. If there +are multiple meta attributes sets, the attribute is set in all of +them. If the resource is a clone, all `target-role` attributes are +removed from the children resources. + +For details on group management see +<<cmdhelp_options_manage-children,`options manage-children`>>. + +Usage: +............... +stop <rsc> [<rsc> ...] +............... + +[[cmdhelp_resource_trace,start RA tracing]] +==== `trace` + +Start tracing RA for the given operation. The trace files are +stored in `$HA_VARLIB/trace_ra`. If the operation to be traced is +monitor, note that the number of trace files can grow very +quickly. + +If no operation name is given, crmsh will attempt to trace all +operations for the RA. This includes any configured operations, start +and stop as well as promote/demote for multistate resources. + +To trace the probe operation which exists for all resources, either +set a trace for `monitor` with interval `0`, or use `probe` as the +operation name. + +Usage: +............... +trace <rsc> [<op> [<interval>] ] +............... +Example: +............... +trace fs start +trace webserver +trace webserver probe +trace fs monitor 0 +............... + +[[cmdhelp_resource_unmanage,put a resource into unmanaged mode]] +==== `unmanage` + +Unmanage a resource using the `is-managed` attribute. If there +are multiple meta attributes sets, the attribute is set in all of +them. If the resource is a clone, all `is-managed` attributes are +removed from the children resources. + +For details on group management see <<cmdhelp_options_manage-children,`options manage-children`>>. + +Usage: +............... +unmanage <rsc> +............... + +[[cmdhelp_resource_untrace,stop RA tracing]] +==== `untrace` + +Stop tracing RA for the given operation. If no operation name is +given, crmsh will attempt to stop tracing all operations in resource. + +Usage: +............... +untrace <rsc> [<op> [<interval>] ] +............... +Example: +............... +untrace fs start +untrace webserver +............... + +[[cmdhelp_resource_utilization,manage a utilization attribute]] +==== `utilization` + +Show/edit/delete a utilization attribute of a resource. These +attributes describe hardware requirements. By setting the +`placement-strategy` cluster property appropriately, it is +possible then to distribute resources based on resource +requirements and node size. See also <<cmdhelp_node_utilization,node utilization attributes>>. + +Usage: +............... +utilization <rsc> set <attr> <value> +utilization <rsc> delete <attr> +utilization <rsc> show <attr> +............... +Example: +............... +utilization xen1 set memory 4096 +............... + +[[cmdhelp_node,Node management]] +=== `node` - Node management + +Node management and status commands. + +[[cmdhelp_node_attribute,manage attributes]] +==== `attribute` + +Edit node attributes. This kind of attribute should refer to +relatively static properties, such as memory size. + +Usage: +............... +attribute <node> set <attr> <value> +attribute <node> delete <attr> +attribute <node> show <attr> +............... +Example: +............... +attribute node_1 set memory_size 4096 +............... + +[[cmdhelp_node_clearstate,Clear node state]] +==== `clearstate` + +Resets and clears the state of the specified node. This node is +afterwards assumed clean and offline. This command can be used to +manually confirm that a node has been fenced (e.g., powered off). + +Be careful! This can cause data corruption if you confirm that a node is +down that is, in fact, not cleanly down - the cluster will proceed as if +the fence had succeeded, possibly starting resources multiple times. + +Usage: +............... +clearstate <node> +............... + +[[cmdhelp_node_delete,delete node]] +==== `delete` + +Delete a node. This command will remove the node from the CIB +and, in case the cluster stack is running, use the appropriate +program (`crm_node` or `hb_delnode`) to remove the node from the +membership. + +If the node is still listed as active and a member of our +partition we refuse to remove it. With the global force option +(`-F`) we will try to delete the node anyway. + +Usage: +............... +delete <node> +............... + +[[cmdhelp_node_fence,fence node]] +==== `fence` + +Make CRM fence a node. This functionality depends on stonith +resources capable of fencing the specified node. No such stonith +resources, no fencing will happen. + +Usage: +............... +fence <node> +............... + +[[cmdhelp_node_maintenance,put node into maintenance mode]] +==== `maintenance` + +Set the node status to maintenance. This is equivalent to the +cluster-wide `maintenance-mode` property but puts just one node +into the maintenance mode. If there are maintenaned resources on +the node, the user will be proposed to remove the maintenance +property from them. + +The node parameter defaults to the node where the command is run. + +Usage: +............... +maintenance [<node>] +............... + +[[cmdhelp_node_online,set node online]] +==== `online` + +Set a node to online status. + +The node parameter defaults to the node where the command is run. + +Usage: +............... +online [<node>] +............... + +[[cmdhelp_node_ready,put node into ready mode]] +==== `ready` + +Set the node's maintenance status to `off`. The node should be +now again fully operational and capable of running resource +operations. + +The node parameter defaults to the node where the command is run. + +Usage: +............... +ready [<node>] +............... + +[[cmdhelp_node_server,show node hostname or server address]] +==== `server` + +Remote nodes may have a configured server address which should +be used when contacting the node. This command prints the +server address if configured, else the node name. + +If no parameter is given, the adresses or names for all nodes +are printed. + +Usage: +............... +server [<node> ...] +............... + +[[cmdhelp_node_show,show node]] +==== `show` + +Show a node definition. If the node parameter is omitted then all +nodes are shown. + +Usage: +............... +show [<node>] +............... + +[[cmdhelp_node_standby,put node into standby]] +==== `standby` + +Set a node to standby status. The node parameter defaults to the +node where the command is run. + +Additionally, you may specify a lifetime for the standby---if set to +`reboot`, the node will be back online once it reboots. `forever` will +keep the node in standby after reboot. The life time defaults to +`forever`. + +Usage: +............... +standby [<node>] [<lifetime>] + +lifetime :: reboot | forever +............... + +Example: +............... +standby bob reboot +............... + + +[[cmdhelp_node_status,show nodes' status as XML]] +==== `status` + +Show nodes' status as XML. If the node parameter is omitted then +all nodes are shown. + +Usage: +............... +status [<node>] +............... + +[[cmdhelp_node_status-attr,manage status attributes]] +==== `status-attr` + +Edit node attributes which are in the CIB status section, i.e. +attributes which hold properties of a more volatile nature. One +typical example is attribute generated by the `pingd` utility. + +Usage: +............... +status-attr <node> set <attr> <value> +status-attr <node> delete <attr> +status-attr <node> show <attr> +............... +Example: +............... +status-attr node_1 show pingd +............... + +[[cmdhelp_node_utilization,manage utilization attributes]] +==== `utilization` + +Edit node utilization attributes. These attributes describe +hardware characteristics as integer numbers such as memory size +or the number of CPUs. By setting the `placement-strategy` +cluster property appropriately, it is possible then to distribute +resources based on resource requirements and node size. See also +<<cmdhelp_resource_utilization,resource utilization attributes>>. + +Usage: +............... +utilization <node> set <attr> <value> +utilization <node> delete <attr> +utilization <node> show <attr> +............... +Examples: +............... +utilization node_1 set memory 16384 +utilization node_1 show cpu +............... + +[[cmdhelp_site,GEO clustering site support]] +=== `site` - GEO clustering site support + +A cluster may consist of two or more subclusters in different and +distant locations. This set of commands supports such setups. + +[[cmdhelp_site_ticket,manage site tickets]] +==== `ticket` + +Tickets are cluster-wide attributes. They can be managed at the +site where this command is executed. + +It is then possible to constrain resources depending on the +ticket availability (see the <<cmdhelp_configure_rsc_ticket,`rsc_ticket`>> command +for more details). + +Usage: +............... +ticket {grant|revoke|standby|activate|show|time|delete} <ticket> +............... +Example: +............... +ticket grant ticket1 +............... + +[[cmdhelp_options,User preferences]] +=== `options` - User preferences + +The user may set various options for the crm shell itself. + +[[cmdhelp_options_add-quotes,add quotes around parameters containing spaces]] +==== `add-quotes` + +The shell (as in `/bin/sh`) parser strips quotes from the command +line. This may sometimes make it really difficult to type values +which contain white space. One typical example is the configure +filter command. The crm shell will supply extra quotes around +arguments which contain white space. The default is `yes`. + +.Note on quotes use +**************************** +Adding quotes around arguments automatically has been introduced +with version 1.2.2 and it is technically a regression. Being a +regression is the only reason the `add-quotes` option exists. If +you have custom shell scripts which would break, just set the +`add-quotes` option to `no`. + +For instance, with adding quotes enabled, it is possible to do +the following: +............... +# crm configure primitive d1 Dummy \ + meta description="some description here" +# crm configure filter 'sed "s/hostlist=./&node-c /"' fencing +............... +**************************** + +[[cmdhelp_options_check-frequency,when to perform semantic check]] +==== `check-frequency` + +Semantic check of the CIB or elements modified or created may be +done on every configuration change (`always`), when verifying +(`on-verify`) or `never`. It is by default set to `always`. +Experts may want to change the setting to `on-verify`. + +The checks require that resource agents are present. If they are +not installed at the configuration time set this preference to +`never`. + +See <<topics_Features_Checks,Configuration semantic checks>> for more details. + +[[cmdhelp_options_check-mode,how to treat semantic errors]] +==== `check-mode` + +Semantic check of the CIB or elements modified or created may be +done in the `strict` mode or in the `relaxed` mode. In the former +certain problems are treated as configuration errors. In the +`relaxed` mode all are treated as warnings. The default is `strict`. + +See <<topics_Features_Checks,Configuration semantic checks>> for more details. + +[[cmdhelp_options_colorscheme,set colors for output]] +==== `colorscheme` + +With `output` set to `color`, a comma separated list of colors +from this option are used to emphasize: + +- keywords +- object ids +- attribute names +- attribute values +- scores +- resource references + +`crm` can show colors only if there is curses support for python +installed (usually provided by the `python-curses` package). The +colors are whatever is available in your terminal. Use `normal` +if you want to keep the default foreground color. + +This user preference defaults to +`yellow,normal,cyan,red,green,magenta` which is good for +terminals with dark background. You may want to change the color +scheme and save it in the preferences file for other color +setups. + +Example: +............... +colorscheme yellow,normal,blue,red,green,magenta +............... + +[[cmdhelp_options_editor,set preferred editor program]] +==== `editor` + +The `edit` command invokes an editor. Use this to specify your +preferred editor program. If not set, it will default to either +the value of the `EDITOR` environment variable or to one of the +standard UNIX editors (`vi`,`emacs`,`nano`). + +Usage: +............... +editor program +............... +Example: +............... +editor vim +............... + +[[cmdhelp_options_manage-children,how to handle children resource attributes]] +==== `manage-children` + +Some resource management commands, such as `resource stop`, when +the target resource is a group, may not always produce desired +result. Each element, group and the primitive members, can have a +meta attribute and those attributes may end up with conflicting +values. Consider the following construct: +............... +crm(live)# configure show svc fs virtual-ip +primitive fs Filesystem \ + params device="/dev/drbd0" directory="/srv/nfs" fstype=ext3 \ + op monitor interval=10s \ + meta target-role=Started +primitive virtual-ip IPaddr2 \ + params ip=10.2.13.110 iflabel=1 \ + op monitor interval=10s \ + op start interval=0 \ + meta target-role=Started +group svc fs virtual-ip \ + meta target-role=Stopped +............... + +Even though the element +svc+ should be stopped, the group is +actually running because all its members have the +target-role+ +set to +Started+: +............... +crm(live)# resource show svc +resource svc is running on: xen-f +............... + +Hence, if the user invokes +resource stop svc+ the intention is +not clear. This preference gives the user an opportunity to +better control what happens if attributes of group members have +values which are in conflict with the same attribute of the group +itself. + +Possible values are +ask+ (the default), +always+, and +never+. +If set to +always+, the crm shell removes all children attributes +which have values different from the parent. If set to +never+, +all children attributes are left intact. Finally, if set to ++ask+, the user will be asked for each member what is to be done. + +[[cmdhelp_options_output,set output type]] +==== `output` + +`crm` can adorn configurations in two ways: in color (similar to +for instance the `ls --color` command) and by showing keywords in +upper case. Possible values are `plain`, `color-always`, `color`, +and 'uppercase'. It is possible to combine `uppercase` with one +of the color values in order to get an upper case xmass tree. Just +set this option to `color,uppercase` or `color-always,uppercase`. +In case you need color codes in pipes, `color-always` forces color +codes even in case the terminal is not a tty (just like `ls +--color=always`). + +[[cmdhelp_options_pager,set preferred pager program]] +==== `pager` + +The `view` command displays text through a pager. Use this to +specify your preferred pager program. If not set, it will default +to either the value of the `PAGER` environment variable or to one +of the standard UNIX system pagers (`less`,`more`,`pg`). + +[[cmdhelp_options_reset,reset user preferences to factory defaults]] +==== `reset` + +This command resets all user options to the defaults. If used as +a single-shot command, the rc file (+$HOME/.config/crm/rc+) is +reset to the defaults too. + +[[cmdhelp_options_save,save the user preferences to the rc file]] +==== `save` + +Save current settings to the rc file (+$HOME/.config/crm/rc+). On +further `crm` runs, the rc file is automatically read and parsed. + +[[cmdhelp_options_set,Set the value of a given option]] +==== `set` + +Sets the value of an option. Takes the fully qualified +name of the option as argument, as displayed by +show all+. + +The modified option value is stored in the user-local +configuration file, usually found in +~/.config/crm/crm.conf+. + +Usage: +........ +set <option> <value> +........ + +Example: +........ +set color.warn "magenta bold" +set editor nano +........ + +[[cmdhelp_options_show,show current user preference]] +==== `show` + +Display all current settings. + +Given an option name as argument, `show` will display only the value +of that argument. + +Given +all+ as argument, `show` displays all available user options. + +Usage: +........ +show [all|<option>] +........ + +Example: +........ +show +show skill-level +show all +........ + +[[cmdhelp_options_skill-level,set skill level]] +==== `skill-level` + +Based on the skill-level setting, the user is allowed to use only +a subset of commands. There are three levels: operator, +administrator, and expert. The operator level allows only +commands at the `resource` and `node` levels, but not editing +or deleting resources. The administrator may do that and may also +configure the cluster at the `configure` level and manage the +shadow CIBs. The expert may do all. + +Usage: +............... +skill-level <level> + +level :: operator | administrator | expert +............... + +.Note on security +**************************** +The `skill-level` option is advisory only. There is nothing +stopping any users change their skill level (see +<<topics_Features_Security,Access Control Lists (ACL)>> on how to enforce +access control). +**************************** + +[[cmdhelp_options_sort-elements,sort CIB elements]] +==== `sort-elements` + +`crm` by default sorts CIB elements. If you want them appear in +the order they were created, set this option to `no`. + +Usage: +............... +sort-elements {yes|no} +............... +Example: +............... +sort-elements no +............... + +[[cmdhelp_options_user,set the cluster user]] +==== `user` + +Sufficient privileges are necessary in order to manage a +cluster: programs such as `crm_verify` or `crm_resource` and, +ultimately, `cibadmin` have to be run either as `root` or as the +CRM owner user (typically `hacluster`). You don't have to worry +about that if you run `crm` as `root`. A more secure way is to +run the program with your usual privileges, set this option to +the appropriate user (such as `hacluster`), and setup the +`sudoers` file. + +Usage: +............... +user system-user +............... +Example: +............... +user hacluster +............... + +[[cmdhelp_options_wait,synchronous operation]] +==== `wait` + +In normal operation, `crm` runs a command and gets back +immediately to process other commands or get input from the user. +With this option set to `yes` it will wait for the started +transition to finish. In interactive mode dots are printed to +indicate progress. + +Usage: +............... +wait {yes|no} +............... +Example: +............... +wait yes +............... + +[[cmdhelp_configure,CIB configuration]] +=== `configure` - CIB configuration + +This level enables all CIB object definition commands. + +The configuration may be logically divided into four parts: +nodes, resources, constraints, and (cluster) properties and +attributes. Each of these commands support one or more basic CIB +objects. + +Nodes and attributes describing nodes are managed using the +`node` command. + +Commands for resources are: + +- `primitive` +- `monitor` +- `group` +- `clone` +- `ms`/`master` (master-slave) + +In order to streamline large configurations, it is possible to +define a template which can later be referenced in primitives: + +- `rsc_template` + +In that case the primitive inherits all attributes defined in the +template. + +There are three types of constraints: + +- `location` +- `colocation` +- `order` + +It is possible to define fencing order (stonith resource +priorities): + +- `fencing_topology` + +Finally, there are the cluster properties, resource meta +attributes defaults, and operations defaults. All are just a set +of attributes. These attributes are managed by the following +commands: + +- `property` +- `rsc_defaults` +- `op_defaults` + +In addition to the cluster configuration, the Access Control +Lists (ACL) can be setup to allow access to parts of the CIB for +users other than +root+ and +hacluster+. The following commands +manage ACL: + +- `user` +- `role` + +In Pacemaker 1.1.12 and up, this command replaces the `user` command +for handling ACLs: + +- `acl_target` + +The changes are applied to the current CIB only on ending the +configuration session or using the `commit` command. + +Comments start with +#+ in the first line. The comments are tied +to the element which follows. If the element moves, its comments +will follow. + +[[cmdhelp_configure_acl_target,Define target access rights]] +==== `acl_target` + +Defines an ACL target. + +Usage: +................ +acl_target <tid> [<role> ...] +................ +Example: +................ +acl_target joe resource_admin constraint_editor +................ + +[[cmdhelp_configure_alert,Event-driven alerts]] +==== `alert` + +.Version note +**************************** +This feature is only available +in Pacemaker 1.1.15+. +**************************** + +Event-driven alerts enables calling scripts whenever interesting +events occur in the cluster (nodes joining or leaving, resources +starting or stopping, etc.). + +The +path+ is an arbitrary file path to an alert script. Existing +external scripts used with ClusterMon resources can be used as alert +scripts, since the interface is compatible. + +Each alert may have a number of receipients configured. These will be +passed to the script as arguments. The first recipient will also be +passed as the +CRM_alert_recipient+ environment variable, for +compatibility with existing scripts that only support one recipient. + +The available meta attributes are +timeout+ (default 30s) and ++timestamp-format+ (default `"%H:%M:%S.%06N"`). + +Some configurations may require each recipient to be delimited by +brackets, to avoid ambiguity. In the example +alert-2+ below, the meta +attribute for `timeout` is defined after the recipient, so the +brackets are used to ensure that the meta attribute is set for the +alert and not just the recipient. This can be avoided by setting any +alert attributes before defining the recipients. + +Usage: +............... +alert <id> <path> \ + [attributes <nvpair> ...] \ + [meta <nvpair> ...] \ + [to [{] <recipient> + [attributes <nvpair> ...] \ + [meta <nvpair> ...] [}] \ + ...] +............... + +Example: +............... +alert alert-1 /srv/pacemaker/pcmk_alert_sample.sh \ + to /var/log/cluster-alerts.log + +alert alert-2 /srv/pacemaker/example_alert.sh \ + meta timeout=60s \ + to { /var/log/cluster-alerts.log } +............... + +[[cmdhelp_configure_cib,CIB shadow management]] +==== `cib` + +This level is for management of shadow CIBs. It is available at +the `configure` level to enable saving intermediate changes to a +shadow CIB instead of to the live cluster. This short excerpt +shows how: +............... +crm(live)configure# cib new test-2 +INFO: test-2 shadow CIB created +crm(test-2)configure# commit +............... +Note how the current CIB in the prompt changed from +live+ to ++test-2+ after issuing the `cib new` command. See also the +<<cmdhelp_cib,CIB shadow management>> for more information. + +[[cmdhelp_configure_cibstatus,CIB status management and editing]] +==== `cibstatus` + +Enter edit and manage the CIB status section level. See the +<<cmdhelp_cibstatus,CIB status management section>>. + +[[cmdhelp_configure_clone,define a clone]] +==== `clone` + +The `clone` command creates a resource clone. It may contain a +single primitive resource or one group of resources. + +Usage: +............... +clone <name> <rsc> + [description=<description>] + [meta <attr_list>] + [params <attr_list>] + +attr_list :: [$id=<id>] <attr>=<val> [<attr>=<val>...] | $id-ref=<id> +............... +Example: +............... +clone cl_fence apc_1 \ + meta clone-node-max=1 globally-unique=false +............... + +[[cmdhelp_configure_colocation,colocate resources]] +==== `colocation` (`collocation`) + +This constraint expresses the placement relation between two +or more resources. If there are more than two resources, then the +constraint is called a resource set. + +The score is used to indicate the priority of the constraint. A +positive score indicates that the resources should run on the same +node. A negative score that they should not run on the same +node. Values of positive or negative +infinity+ indicate a mandatory +constraint. + +In the two resource form, the cluster will place +<with-rsc>+ first, +and then decide where to put the +<rsc>+ resource. + +Collocation resource sets have an extra attribute (+sequential+) +to allow for sets of resources which don't depend on each other +in terms of state. The shell syntax for such sets is to put +resources in parentheses. + +Sets cannot be nested. + +The optional +node-attribute+ can be used to colocate resources on a +set of nodes and not necessarily on the same node. For example, by +setting a node attribute +color+ on all nodes and setting the ++node-attribute+ value to +color+ as well, the colocated resources +will be placed on any node that has the same color. + +For more details on how to configure resource sets, see +<<topics_Features_Resourcesets,`Syntax: Resource sets`>>. + +Usage: +............... +colocation <id> <score>: <rsc>[:<role>] <with-rsc>[:<role>] + [node-attribute=<node_attr>] + +colocation <id> <score>: <resource_sets> + [node-attribute=<node_attr>] + +resource_sets :: <resource_set> [<resource_set> ...] + +resource_set :: ["("|"["] <rsc>[:<role>] [<rsc>[:<role>] ...] \ + [<attributes>] [")"|"]"] + +attributes :: [require-all=(true|false)] [sequential=(true|false)] + +............... +Example: +............... +colocation never_put_apache_with_dummy -inf: apache dummy +colocation c1 inf: A ( B C ) +............... + +[[cmdhelp_configure_commit,commit the changes to the CIB]] +==== `commit` + +Commit the current configuration to the CIB in use. As noted +elsewhere, commands in a configure session don't have immediate +effect on the CIB. All changes are applied at one point in time, +either using `commit` or when the user leaves the configure +level. In case the CIB in use changed in the meantime, presumably +by somebody else, the crm shell will refuse to apply the changes. + +If you know that it's fine to still apply them, add +force+ to the +command line. + +To disable CIB patching and apply the changes by replacing the CIB +completely, add +replace+ to the command line. Note that this can lead +to previous changes being overwritten if some other process +concurrently modifies the CIB. + +Usage: +............... +commit [force] [replace] +............... + +[[cmdhelp_configure_default-timeouts,set timeouts for operations to minimums from the meta-data]] +==== `default-timeouts` + +This command takes the timeouts from the actions section of the +resource agent meta-data and sets them for the operations of the +primitive. + +Usage: +............... +default-timeouts <id> [<id>...] +............... + +.Note on `default-timeouts` +**************************** +The use of this command is discouraged in favor of manually +determining the best timeouts required for the particular +configuration. Relying on the resource agent to supply appropriate +timeouts can cause the resource to fail at the worst possible moment. + +Appropriate timeouts for resource actions are context-sensitive, and +should be carefully considered with the whole configuration in mind. +**************************** + +[[cmdhelp_configure_delete,delete CIB objects]] +==== `delete` + +Delete one or more objects. If an object to be deleted belongs to +a container object, such as a group, and it is the only resource +in that container, then the container is deleted as well. Any +related constraints are removed as well. + +If the object is a started resource, it will not be deleted unless the ++--force+ flag is passed to the command, or the +force+ option is set. + +Usage: +............... +delete [--force] <id> [<id>...] +............... + +[[cmdhelp_configure_edit,edit CIB objects]] +==== `edit` + +This command invokes the editor with the object description. As +with the `show` command, the user may choose to edit all objects +or a set of objects. + +If the user insists, he or she may edit the XML edition of the +object. If you do that, don't modify any id attributes. + +Usage: +............... +edit [xml] [<id> ...] +edit [xml] changed +............... + +.Note on renaming element ids +**************************** +The edit command sometimes cannot properly handle modifying +element ids. In particular for elements which belong to group or +ms resources. Group and ms resources themselves also cannot be +renamed. Please use the `rename` command instead. +**************************** + +[[cmdhelp_configure_erase,erase the CIB]] +==== `erase` + +The `erase` clears all configuration. Apart from nodes. To remove +nodes, you have to specify an additional keyword `nodes`. + +Note that removing nodes from the live cluster may have some +strange/interesting/unwelcome effects. + +Usage: +............... +erase [nodes] +............... + +[[cmdhelp_configure_fencing_topology,node fencing order]] +==== `fencing_topology` + +If multiple fencing (stonith) devices are available capable of +fencing a node, their order may be specified by +fencing_topology+. +The order is specified per node. + +Stonith resources can be separated by +,+ in which case all of +them need to succeed. If they fail, the next stonith resource (or +set of resources) is used. In other words, use comma to separate +resources which all need to succeed and whitespace for serial +order. It is not allowed to use whitespace around comma. + +If the node is left out, the order is used for all nodes. +That should reduce the configuration size in some stonith setups. + +From Pacemaker version 1.1.14, it is possible to use a node attribute +as the +target+ in a fencing topology. The syntax for this usage is +described below. + +From Pacemaker version 1.1.14, it is also possible to use regular +expression patterns as the +target+ in a fencing topology. The configured +fencing sequence then applies to all devices matching the pattern. + +Usage: +............... +fencing_topology <stonith_resources> [<stonith_resources> ...] +fencing_topology <fencing_order> [<fencing_order> ...] + +fencing_order :: <target> <stonith_resources> [<stonith_resources> ...] + +stonith_resources :: <rsc>[,<rsc>...] +target :: <node>: | attr:<node-attribute>=<value> | pattern:<pattern> +............... +Example: +............... +# Only kill the power if poison-pill fails +fencing_topology poison-pill power + +# As above for node-a, but a different strategy for node-b +fencing_topology \ + node-a: poison-pill power \ + node-b: ipmi serial + +# Fencing anything on rack 1 requires fencing via both APC 1 and 2, +# to defeat the redundancy provided by two separate UPS units. +fencing_topology attr:rack=1 apc01,apc02 + +# Fencing for all machines named green.* is done using the pear +# fencing device first, while all machines named red.* are fenced +# using the apple fencing device first. +fencing_topology \ + pattern:green.* pear apple \ + pattern:red.* apple pear +............... + +[[cmdhelp_configure_filter,filter CIB objects]] +==== `filter` + +This command filters the given CIB elements through an external +program. The program should accept input on `stdin` and send +output to `stdout` (the standard UNIX filter conventions). As +with the `show` command, the user may choose to filter all or +just a subset of elements. + +It is possible to filter the XML representation of objects, but +probably not as useful as the configuration language. The +presentation is somewhat different from what would be displayed +by the `show` command---each element is shown on a single line, +i.e. there are no backslashes and no other embelishments. + +Don't forget to put quotes around the filter if it contains +spaces. + +Usage: +............... +filter <prog> [xml] [<id> ...] +filter <prog> [xml] changed +............... +Examples: +............... +filter "sed '/^primitive/s/target-role=[^ ]*//'" +# crm configure filter "sed '/^primitive/s/target-role=[^ ]*//'" +crm configure <<END + filter "sed '/threshold=\"1\"/s/=\"1\"/=\"0\"/g'" +END +............... + +.Note on quotation marks +************************** +Filter commands which feature a blend of quotation marks can be +difficult to get right, especially when used directly from bash, since +bash does its own quotation parsing. In these cases, it can be easier +to supply the filter command as standard input. See the last example +above. +************************** + +[[cmdhelp_configure_get_property,Get property value]] +==== `get-property` + +Show the value of the given property. If the value is not set, the +command will print the default value for the property, if known. + +If no property name is passed to the command, the list of known +cluster properties is printed. + +If the property is set multiple times, for example using multiple +property sets with different rule expressions, the output of this +command is undefined. + +Pass the argument +-t+ or +--true+ to `get-property` to translate +the argument value into +true+ or +false+. If the value is not +set, the command will print +false+. + +Usage: +............... +get-property [-t|--true] [<name>] +............... + +Example: +............... +get-property stonith-enabled +get-property -t maintenance-mode +............... + +[[cmdhelp_configure_graph,generate a directed graph]] +==== `graph` + +Create a graphviz graphical layout from the current cluster +configuration. + +Currently, only `dot` (directed graph) is supported. It is +essentially a visualization of resource ordering. + +The graph may be saved to a file which can be used as source for +various graphviz tools (by default it is displayed in the user's +X11 session). Optionally, by specifying the format, one can also +produce an image instead. + +For more or different graphviz attributes, it is possible to save +the default set of attributes to an ini file. If this file exists +it will always override the builtin settings. The +exportsettings+ +subcommand also prints the location of the ini file. + +Usage: +............... +graph [<gtype> [<file> [<img_format>]]] +graph exportsettings + +gtype :: dot +img_format :: `dot` output format (see the +-T+ option) +............... +Example: +............... +graph dot +graph dot clu1.conf.dot +graph dot clu1.conf.svg svg +............... + +[[cmdhelp_configure_group,define a group]] +==== `group` + +The `group` command creates a group of resources. This can be useful +when resources depend on other resources and require that those +resources start in order on the same node. A common use of resource +groups is to ensure that a server and a virtual IP are located +together, and that the virtual IP is started before the server. + +Grouped resources are started in the order they appear in the group, +and stopped in the reverse order. If a resource in the group cannot +run anywhere, resources following it in the group will not start. + +`group` can be passed the "container" meta attribute, to indicate that +it is to be used to group VM resources monitored using Nagios. The +resource referred to by the container attribute must be of type +`ocf:heartbeat:Xen`, `ocf:heartbeat:VirtualDomain` or `ocf:heartbeat:lxc`. + +Usage: +............... +group <name> <rsc> [<rsc>...] + [description=<description>] + [meta attr_list] + [params attr_list] + +attr_list :: [$id=<id>] <attr>=<val> [<attr>=<val>...] | $id-ref=<id> +............... +Example: +............... +group internal_www disk0 fs0 internal_ip apache \ + meta target_role=stopped + +group vm-and-services vm vm-sshd meta container="vm" +............... + +[[cmdhelp_configure_load,import the CIB from a file]] +==== `load` + +Load a part of configuration (or all of it) from a local file or +a network URL. The +replace+ method replaces the current +configuration with the one from the source. The +update+ method +tries to import the contents into the current configuration. The ++push+ method imports the contents into the current configuration +and removes any lines that are not present in the given +configuration. +The file may be a CLI file or an XML file. + +If the URL is `-`, the configuration is read from standard input. + +Usage: +............... +load [xml] <method> URL + +method :: replace | update | push +............... +Example: +............... +load xml update myfirstcib.xml +load xml replace http://storage.big.com/cibs/bigcib.xml +load xml push smallcib.xml +............... + +[[cmdhelp_configure_location,a location preference]] +==== `location` + +`location` defines the preference of nodes for the given +resource. The location constraints consist of one or more rules +which specify a score to be awarded if the rule matches. + +The resource referenced by the location constraint can be one of the +following: + +* Plain resource reference: +location loc1 webserver 100: node1+ +* Resource set in curly brackets: +location loc1 { virtual-ip webserver } 100: node1+ +* Tag containing resource ids: +location loc1 tag1 100: node1+ +* Resource pattern: +location loc1 /web.*/ 100: node1+ + +The +resource-discovery+ attribute allows probes to be selectively +enabled or disabled per resource and node. + +The syntax for resource sets is described in detail for +<<cmdhelp_configure_colocation,`colocation`>>. + +For more details on how to configure resource sets, see +<<topics_Features_Resourcesets,`Syntax: Resource sets`>>. + +For more information on rule expressions, see +<<topics_Syntax_RuleExpressions,Syntax: Rule expressions>>. + +Usage: +............... +location <id> <rsc> [<attributes>] {<node_pref>|<rules>} + +rsc :: /<rsc-pattern>/ + | { resource_sets } + | <rsc> + +attributes :: role=<role> | resource-discovery=always|never|exclusive + +node_pref :: <score>: <node> + +rules :: + rule [id_spec] [$role=<role>] <score>: <expression> + [rule [id_spec] [$role=<role>] <score>: <expression> ...] + +id_spec :: $id=<id> | $id-ref=<id> +score :: <number> | <attribute> | [-]inf +expression :: <simple_exp> [<bool_op> <simple_exp> ...] +bool_op :: or | and +simple_exp :: <attribute> [type:]<binary_op> <value> + | <unary_op> <attribute> + | date <date_expr> +type :: string | version | number +binary_op :: lt | gt | lte | gte | eq | ne +unary_op :: defined | not_defined + +date_expr :: lt <end> + | gt <start> + | in start=<start> end=<end> + | in start=<start> <duration> + | spec <date_spec> +duration|date_spec :: + hours=<value> + | monthdays=<value> + | weekdays=<value> + | yearsdays=<value> + | months=<value> + | weeks=<value> + | years=<value> + | weekyears=<value> + | moon=<value> +............... +Examples: +............... +location conn_1 internal_www 100: node1 + +location conn_1 internal_www \ + rule 50: #uname eq node1 \ + rule pingd: defined pingd + +location conn_2 dummy_float \ + rule -inf: not_defined pingd or pingd number:lte 0 + +# never probe for rsc1 on node1 +location no-probe rsc1 resource-discovery=never -inf: node1 +............... + +[[cmdhelp_configure_modgroup,modify group]] +==== `modgroup` + +Add or remove primitives in a group. The `add` subcommand appends +the new group member by default. Should it go elsewhere, there +are `after` and `before` clauses. + +Usage: +............... +modgroup <id> add <id> [after <id>|before <id>] +modgroup <id> remove <id> +............... +Examples: +............... +modgroup share1 add storage2 before share1-fs +............... + +[[cmdhelp_configure_monitor,add monitor operation to a primitive]] +==== `monitor` + +Monitor is by far the most common operation. It is possible to +add it without editing the whole resource. Also, long primitive +definitions may be a bit uncluttered. In order to make this +command as concise as possible, less common operation attributes +are not available. If you need them, then use the `op` part of +the `primitive` command. + +Usage: +............... +monitor <rsc>[:<role>] <interval>[:<timeout>] +............... +Example: +............... +monitor apcfence 60m:60s +............... + +Note that after executing the command, the monitor operation may +be shown as part of the primitive definition. + +[[cmdhelp_configure_ms,define a master-slave resource]] +==== `ms` (`master`) + +The `ms` command creates a master/slave resource type. It may contain a +single primitive resource or one group of resources. + +Usage: +............... +ms <name> <rsc> + [description=<description>] + [meta attr_list] + [params attr_list] + +attr_list :: [$id=<id>] <attr>=<val> [<attr>=<val>...] | $id-ref=<id> +............... +Example: +............... +ms disk1 drbd1 \ + meta notify=true globally-unique=false +............... + +.Note on `id-ref` usage +**************************** +Instance or meta attributes (`params` and `meta`) may contain +a reference to another set of attributes. In that case, no other +attributes are allowed. Since attribute sets' ids, though they do +exist, are not shown in the `crm`, it is also possible to +reference an object instead of an attribute set. `crm` will +automatically replace such a reference with the right id: + +............... +crm(live)configure# primitive a2 www-2 meta $id-ref=a1 +crm(live)configure# show a2 +primitive a2 apache \ + meta $id-ref=a1-meta_attributes + [...] +............... +It is advisable to give meaningful names to attribute sets which +are going to be referenced. +**************************** + +[[cmdhelp_configure_node,define a cluster node]] +==== `node` + +The node command describes a cluster node. Nodes in the CIB are +commonly created automatically by the CRM. Hence, you should not +need to deal with nodes unless you also want to define node +attributes. Note that it is also possible to manage node +attributes at the `node` level. + +Usage: +............... +node [$id=<id>] <uname>[:<type>] + [description=<description>] + [attributes [$id=<id>] [<score>:] [rule...] + <param>=<value> [<param>=<value>...]] | $id-ref=<ref> + [utilization [$id=<id>] [<score>:] [rule...] + <param>=<value> [<param>=<value>...]] | $id-ref=<ref> + +type :: normal | member | ping | remote +............... +Example: +............... +node node1 +node big_node attributes memory=64 +............... + +[[cmdhelp_configure_op_defaults,set resource operations defaults]] +==== `op_defaults` + +Set defaults for the operations meta attributes. + +For more information on rule expressions, see +<<topics_Syntax_RuleExpressions,Syntax: Rule expressions>>. + +Usage: +............... +op_defaults [$id=<set_id>] [rule ...] <option>=<value> [<option>=<value> ...] +............... +Example: +............... +op_defaults record-pending=true +............... + +[[cmdhelp_configure_order,order resources]] +==== `order` + +This constraint expresses the order of actions on two resources +or more resources. If there are more than two resources, then the +constraint is called a resource set. + +Ordered resource sets have an extra attribute to allow for sets +of resources whose actions may run in parallel. The shell syntax +for such sets is to put resources in parentheses. + +If the subsequent resource can start or promote after any one of the +resources in a set has done, enclose the set in brackets (+[+ and +]+). + +Sets cannot be nested. + +Three strings are reserved to specify a kind of order constraint: ++Mandatory+, +Optional+, and +Serialize+. It is preferred to use +one of these settings instead of score. Previous versions mapped +scores +0+ and +inf+ to keywords +advisory+ and +mandatory+. +That is still valid but deprecated. + +For more details on how to configure resource sets, see +<<topics_Features_Resourcesets,`Syntax: Resource sets`>>. + +Usage: +............... +order <id> [{kind|<score>}:] first then [symmetrical=<bool>] + +order <id> [{kind|<score>}:] resource_sets [symmetrical=<bool>] + +kind :: Mandatory | Optional | Serialize + +first :: <rsc>[:<action>] + +then :: <rsc>[:<action>] + +resource_sets :: resource_set [resource_set ...] + +resource_set :: ["["|"("] <rsc>[:<action>] [<rsc>[:<action>] ...] \ + [attributes] ["]"|")"] + +attributes :: [require-all=(true|false)] [sequential=(true|false)] + +............... +Example: +............... +order o-1 Mandatory: apache:start ip_1 +order o-2 Serialize: A ( B C ) +order o-3 inf: [ A B ] C +order o-4 first-resource then-resource +............... + +[[cmdhelp_configure_primitive,define a resource]] +==== `primitive` + +The primitive command describes a resource. It may be referenced +only once in group, clone, or master-slave objects. If it's not +referenced, then it is placed as a single resource in the CIB. + +Operations may be specified anonymously, as a group or by reference: + +* "Anonymous", as a list of +op+ specifications. Use this + method if you don't need to reference the set of operations + elsewhere. This is the most common way to define operations. + +* If reusing operation sets is desired, use the +operations+ keyword + along with an id to give the operations set a name. Use the + +operations+ keyword and an id-ref value set to the id of another + operations set, to apply the same set of operations to this + primitive. + +Operation attributes which are not recognized are saved as +instance attributes of that operation. A typical example is ++OCF_CHECK_LEVEL+. + +For multistate resources, roles are specified as +role=<role>+. + +A template may be defined for resources which are of the same +type and which share most of the configuration. See +<<cmdhelp_configure_rsc_template,`rsc_template`>> for more information. + +Attributes containing time values, such as the +interval+ attribute on +operations, are configured either as a plain number, which is +interpreted as a time in seconds, or using one of the following +suffixes: + +* +s+, +sec+ - time in seconds (same as no suffix) +* +ms+, +msec+ - time in milliseconds +* +us+, +usec+ - time in microseconds +* +m+, +min+ - time in minutes +* +h+, +hr+ - time in hours + +Usage: +............... +primitive <rsc> {[<class>:[<provider>:]]<type>|@<template>} + [description=<description>] + [[params] attr_list] + [meta attr_list] + [utilization attr_list] + [operations id_spec] + [op op_type [<attribute>=<value>...] ...] + +attr_list :: [$id=<id>] [<score>:] [rule...] + <attr>=<val> [<attr>=<val>...]] | $id-ref=<id> +id_spec :: $id=<id> | $id-ref=<id> +op_type :: start | stop | monitor +............... +Example: +............... +primitive apcfence stonith:apcsmart \ + params ttydev=/dev/ttyS0 hostlist="node1 node2" \ + op start timeout=60s \ + op monitor interval=30m timeout=60s + +primitive www8 apache \ + configfile=/etc/apache/www8.conf \ + operations $id-ref=apache_ops + +primitive db0 mysql \ + params config=/etc/mysql/db0.conf \ + op monitor interval=60s \ + op monitor interval=300s OCF_CHECK_LEVEL=10 + +primitive r0 ocf:linbit:drbd \ + params drbd_resource=r0 \ + op monitor role=Master interval=60s \ + op monitor role=Slave interval=300s + +primitive xen0 @vm_scheme1 xmfile=/etc/xen/vm/xen0 + +primitive mySpecialRsc Special \ + params 3: rule #uname eq node1 interface=eth1 \ + params 2: rule #uname eq node2 interface=eth2 port=8888 \ + params 1: interface=eth0 port=9999 + +............... + +[[cmdhelp_configure_property,set a cluster property]] +==== `property` + +Set cluster configuration properties. To list the +available cluster configuration properties, use the +<<cmdhelp_ra_info,`ra info`>> command with +pengine+, +crmd+, ++cib+ and +stonithd+ as arguments. +When setting the +maintenance-mode+ property, it will +inform the user if there are nodes or resources that +have the +maintenance+ property. + +For more information on rule expressions, see +<<topics_Syntax_RuleExpressions,Syntax: Rule expressions>>. + +Usage: +............... +property [<set_id>:] [rule ...] <option>=<value> [<option>=<value> ...] +............... +Example: +............... +property stonith-enabled=true +property rule date spec years=2014 stonith-enabled=false +............... + +[[cmdhelp_configure_ptest,show cluster actions if changes were committed]] +==== `ptest` (`simulate`) + +Show PE (Policy Engine) motions using `ptest(8)` or +`crm_simulate(8)`. + +A CIB is constructed using the current user edited configuration +and the status from the running CIB. The resulting CIB is run +through `ptest` (or `crm_simulate`) to show changes which would +happen if the configuration is committed. + +The status section may be loaded from another source and modified +using the <<cmdhelp_cibstatus,`cibstatus`>> level commands. In that case, the +`ptest` command will issue a message informing the user that the +Policy Engine graph is not calculated based on the current status +section and therefore won't show what would happen to the +running but some imaginary cluster. + +If you have graphviz installed and X11 session, `dotty(1)` is run +to display the changes graphically. + +Add a string of +v+ characters to increase verbosity. `ptest` +can also show allocation scores. +utilization+ turns on +information about the remaining capacity of nodes. With the ++actions+ option, `ptest` will print all resource actions. + +The `ptest` program has been replaced by `crm_simulate` in newer +Pacemaker versions. In some installations both could be +installed. Use `simulate` to enfore using `crm_simulate`. + +Usage: +............... +ptest [nograph] [v...] [scores] [actions] [utilization] +............... +Examples: +............... +ptest scores +ptest vvvvv +simulate actions +............... + +[[cmdhelp_configure_refresh,refresh from CIB]] +==== `refresh` + +Refresh the internal structures from the CIB. All changes made +during this session are lost. + +Usage: +............... +refresh +............... + +[[cmdhelp_configure_rename,rename a CIB object]] +==== `rename` + +Rename an object. It is recommended to use this command to rename +a resource, because it will take care of updating all related +constraints and a parent resource. Changing ids with the edit +command won't have the same effect. + +If you want to rename a resource, it must be in the stopped state. + +Usage: +............... +rename <old_id> <new_id> +............... + +[[cmdhelp_configure_role,define role access rights]] +==== `role` + +An ACL role is a set of rules which describe access rights to +CIB. Rules consist of an access right +read+, +write+, or +deny+ +and a specification denoting part of the configuration to which +the access right applies. The specification can be an XPath or a +combination of tag and id references. If an attribute is +appended, then the specification applies only to that attribute +of the matching element. + +There is a number of shortcuts for XPath specifications. The ++meta+, +params+, and +utilization+ shortcuts reference resource +meta attributes, parameters, and utilization respectively. The +`location` may be used to specify location constraints most of +the time to allow resource `move` and `unmove` commands. The +`property` references cluster properties. The `node` allows +reading node attributes. +nodeattr+ and +nodeutil+ reference node +attributes and node capacity (utilization). The `status` shortcut +references the whole status section of the CIB. Read access to +status is necessary for various monitoring tools such as +`crm_mon(8)` (aka `crm status`). + +For more information on rule expressions, see +<<topics_Syntax_RuleExpressions,Syntax: Rule expressions>>. + +Usage: +............... +role <role-id> rule [rule ...] + +rule :: acl-right cib-spec [attribute:<attribute>] + +acl-right :: read | write | deny + +cib-spec :: xpath-spec | tag-ref-spec +xpath-spec :: xpath:<xpath> | shortcut +tag-ref-spec :: tag:<tag> | ref:<id> | tag:<tag> ref:<id> + +shortcut :: meta:<rsc>[:<attr>] + params:<rsc>[:<attr>] + utilization:<rsc> + location:<rsc> + property[:<attr>] + node[:<node>] + nodeattr[:<attr>] + nodeutil[:<node>] + status +............... +Example: +............... +role app1_admin \ + write meta:app1:target-role \ + write meta:app1:is-managed \ + write location:app1 \ + read ref:app1 +............... + +[[cmdhelp_configure_rsc_defaults,set resource defaults]] +==== `rsc_defaults` + +Set defaults for the resource meta attributes. + +For more information on rule expressions, see +<<topics_Syntax_RuleExpressions,Syntax: Rule expressions>>. + +Usage: +............... +rsc_defaults [<set_id>:] [rule ...] <option>=<value> [<option>=<value> ...] +............... +Example: +............... +rsc_defaults failure-timeout=3m +............... + +[[cmdhelp_configure_rsc_template,define a resource template]] +==== `rsc_template` + +The `rsc_template` command creates a resource template. It may be +referenced in primitives. It is used to reduce large +configurations with many similar resources. + +Usage: +............... +rsc_template <name> [<class>:[<provider>:]]<type> + [description=<description>] + [params attr_list] + [meta attr_list] + [utilization attr_list] + [operations id_spec] + [op op_type [<attribute>=<value>...] ...] + +attr_list :: [$id=<id>] <attr>=<val> [<attr>=<val>...] | $id-ref=<id> +id_spec :: $id=<id> | $id-ref=<id> +op_type :: start | stop | monitor +............... +Example: +............... +rsc_template public_vm Xen \ + op start timeout=300s \ + op stop timeout=300s \ + op monitor interval=30s timeout=60s \ + op migrate_from timeout=600s \ + op migrate_to timeout=600s +primitive xen0 @public_vm \ + params xmfile=/etc/xen/xen0 +primitive xen1 @public_vm \ + params xmfile=/etc/xen/xen1 +............... + +[[cmdhelp_configure_rsc_ticket,resources ticket dependency]] +==== `rsc_ticket` + +This constraint expresses dependency of resources on cluster-wide +attributes, also known as tickets. Tickets are mainly used in +geo-clusters, which consist of multiple sites. A ticket may be +granted to a site, thus allowing resources to run there. + +The +loss-policy+ attribute specifies what happens to the +resource (or resources) if the ticket is revoked. The default is +either +stop+ or +demote+ depending on whether a resource is +multi-state. + +See also the <<cmdhelp_site_ticket,`site`>> set of commands. + +Usage: +............... +rsc_ticket <id> <ticket_id>: <rsc>[:<role>] [<rsc>[:<role>] ...] + [loss-policy=<loss_policy_action>] + +loss_policy_action :: stop | demote | fence | freeze +............... +Example: +............... +rsc_ticket ticket-A_public-ip ticket-A: public-ip +rsc_ticket ticket-A_bigdb ticket-A: bigdb loss-policy=fence +rsc_ticket ticket-B_storage ticket-B: drbd-a:Master drbd-b:Master +............... + + +[[cmdhelp_configure_rsctest,test resources as currently configured]] +==== `rsctest` + +Test resources with current resource configuration. If no nodes +are specified, tests are run on all known nodes. + +The order of resources is significant: it is assumed that later +resources depend on earlier ones. + +If a resource is multi-state, it is assumed that the role on +which later resources depend is master. + +Tests are run sequentially to prevent running the same resource +on two or more nodes. Tests are carried out only if none of the +specified nodes currently run any of the specified resources. +However, it won't verify whether resources run on the other +nodes. + +Superuser privileges are obviously required: either run this as +root or setup the `sudoers` file appropriately. + +Note that resource testing may take some time. + +Usage: +............... +rsctest <rsc_id> [<rsc_id> ...] [<node_id> ...] +............... +Examples: +............... +rsctest my_ip websvc +rsctest websvc nodeB +............... + +[[cmdhelp_configure_save,save the CIB to a file]] +==== `save` + +Save the current configuration to a file. Optionally, as XML. Use ++-+ instead of file name to write the output to `stdout`. + +The `save` command accepts the same selection arguments as the `show` +command. See the <<cmdhelp_configure_show,help section>> for `show` +for more details. + +Usage: +............... +save [xml] [<id> | type:<type | tag:<tag> | + related:<obj> | changed ...] <file> +............... +Example: +............... +save myfirstcib.txt +save web-server server-config.txt +............... + +[[cmdhelp_configure_schema,set or display current CIB RNG schema]] +==== `schema` + +CIB's content is validated by a RNG schema. Pacemaker supports +several, depending on version. At least the following schemas are +accepted by `crmsh`: + +* +pacemaker-1.0+ +* +pacemaker-1.1+ +* +pacemaker-1.2+ +* +pacemaker-1.3+ +* +pacemaker-2.0+ + +Use this command to display or switch to another RNG schema. + +Usage: +............... +schema [<schema>] +............... +Example: +............... +schema pacemaker-1.1 +............... + +[[cmdhelp_configure_set,set an attribute value]] +==== `set` + +Set the value of a configured attribute. The attribute must +have a value configured previously, and can be an agent +parameter, meta attribute or utilization value. + +The first argument to the command is a path to an attribute. +This is a dot-separated sequence beginning with the name of +the resource, and ending with the name of the attribute to +set. + +Usage: +............... +set <path> <value> +............... +Examples: +............... +set vip1.ip 192.168.20.5 +set vm-a.force_stop 1 +............... + +[[cmdhelp_configure_show,display CIB objects]] +==== `show` + +The `show` command displays CIB objects. Without any argument, it +displays all objects in the CIB, but the set of objects displayed by +`show` can be limited to only objects with the given IDs or by using +one or more of the special prefixes described below. + +The XML representation for the objects can be displayed by passing ++xml+ as the first argument. + +To show one or more specific objects, pass the object IDs as +arguments. + +To show all objects of a certain type, use the +type:+ prefix. + +To show all objects in a tag, use the +tag:+ prefix. + +To show all constraints related to a primitive, use the +related:+ prefix. + +To show all modified objects, pass the argument +changed+. + +The prefixes can be used together on a single command line. For +example, to show both the tag itself and the objects tagged by it the +following combination can be used: +show tag:my-tag my-tag+. + +To refine a selection of objects using multiple modifiers, the keywords ++and+ and +or+ can be used. For example, to select all primitives tagged ++foo+, the following combination can be used: ++show type:primitive and tag:foo+. + +To hide values when displaying the configuration, use the ++obscure:<glob>+ argument. This can be useful when sending the +configuration over a public channel, to avoid exposing potentially +sensitive information. The +<glob>+ argument is a bash-style pattern +matching attribute keys. + +Usage: +............... +show [xml] [<id> + | changed + | type:<type> + | tag:<id> + | related:<obj> + | obscure:<glob> + ...] + +type :: node | primitive | group | clone | ms | rsc_template + | location | colocation | order + | rsc_ticket + | property | rsc_defaults | op_defaults + | fencing_topology + | role | user | acl_target + | tag +............... + +Example: +............... +show webapp +show type:primitive +show xml tag:db tag:fs +show related:webapp +show type:primitive obscure:passwd +............... + +[[cmdhelp_configure_tag,Define resource tags]] +==== `tag` + +Define a resource tag. A tag is an id referring to one or more +resources, without implying any constraints between the tagged +resources. This can be useful for grouping conceptually related +resources. + +Usage: +............... +tag <tag-name>: <rsc> [<rsc> ...] +tag <tag-name> <rsc> [<rsc> ...] +............... +Example: +............... +tag web: p-webserver p-vip +tag ips server-vip admin-vip +............... + +[[cmdhelp_configure_template,edit and import a configuration from a template]] +==== `template` + +The specified template is loaded into the editor. It's up to the +user to make a good CRM configuration out of it. See also the +<<cmdhelp_template,template section>>. + +Usage: +............... +template [xml] url +............... +Example: +............... +template two-apaches.txt +............... + +[[cmdhelp_configure_upgrade,upgrade the CIB]] +==== `upgrade` + +Attempts to upgrade the CIB to validate with the current +version. Commonly, this is required if the error +`CIB not supported` occurs. It typically means that the +active CIB version is coming from an older release. + +As a safety precaution, the force argument is required if the ++validation-with+ attribute is set to anything other than ++0.6+. Thus in most cases, it is required. + +Usage: +............... +upgrade [force] +............... + +Example: +............... +upgrade force +............... + +[[cmdhelp_configure_user,define user access rights]] +==== `user` + +Users which normally cannot view or manage cluster configuration +can be allowed access to parts of the CIB. The access is defined +by a set of +read+, +write+, and +deny+ rules as in role +definitions or by referencing roles. The latter is considered +best practice. + +For more information on rule expressions, see +<<topics_Syntax_RuleExpressions,Syntax: Rule expressions>>. + +Usage: +............... +user <uid> {roles|rules} + +roles :: role:<role-ref> [role:<role-ref> ...] +rules :: rule [rule ...] +............... +Example: +............... +user joe \ + role:app1_admin \ + role:read_all +............... + +[[cmdhelp_configure_validate_all,call agent validate-all for resource]] +==== `validate-all` + +Call the `validate-all` action for the resource, if possible. + +Limitations: + +* The resource agent must implement the `validate-all` action. +* The current user must be root. +* The primitive resource must not use nvpair references. + +Usage: +............... +validate-all <rsc> +............... + + +[[cmdhelp_configure_verify,verify the CIB with crm_verify]] +==== `verify` + +Verify the contents of the CIB which would be committed. + +Usage: +............... +verify +............... + +[[cmdhelp_configure_xml,raw xml]] +==== `xml` + +Even though we promissed no xml, it may happen, but hopefully +very very seldom, that an element from the CIB cannot be rendered +in the configuration language. In that case, the element will be +shown as raw xml, prefixed by this command. That element can then +be edited like any other. If the shell finds out that after the +change it can digest it, then it is going to be converted into +the normal configuration language. Otherwise, there is no need to +use `xml` for configuration. + +Usage: +............... +xml <xml> +............... + +[[cmdhelp_template,edit and import a configuration from a template]] +=== `template` - Import configuration from templates + +User may be assisted in the cluster configuration by templates +prepared in advance. Templates consist of a typical ready +configuration which may be edited to suit particular user needs. + +This command enters a template level where additional commands +for configuration/template management are available. + +[[cmdhelp_template_apply,process and apply the current configuration to the current CIB]] +==== `apply` + +Copy the current or given configuration to the current CIB. By +default, the CIB is replaced, unless the method is set to +"update". + +Usage: +............... +apply [<method>] [<config>] + +method :: replace | update +............... + +[[cmdhelp_template_delete,delete a configuration]] +==== `delete` + +Remove a configuration. The loaded (active) configuration may be +removed by force. + +Usage: +............... +delete <config> [force] +............... + +[[cmdhelp_template_edit,edit a configuration]] +==== `edit` + +Edit current or given configuration using your favourite editor. + +Usage: +............... +edit [<config>] +............... + +[[cmdhelp_template_list,list configurations/templates]] +==== `list` + +When called with no argument, lists existing templates and +configurations. + +Given the argument +templates+, lists the available templates. + +Given the argument +configs+, lists the available configurations. + +Usage: +............... +list [templates|configs] +............... + +[[cmdhelp_template_load,load a configuration]] +==== `load` + +Load an existing configuration. Further `edit`, `show`, and +`apply` commands will refer to this configuration. + +Usage: +............... +load <config> +............... + +[[cmdhelp_template_new,create a new configuration from templates]] +==== `new` + +Create a new configuration from one or more templates. Note that +configurations and templates are kept in different places, so it +is possible to have a configuration name equal a template name. + +If you already know which parameters are required, you can set +them directly on the command line. + +The parameter name +id+ is set by default to the name of the +configuration. + +If no parameters are being set and you don't want a particular name +for your configuration, you can call this command with a template name +as the only parameter. A unique configuration name based on the +template name will be generated. + +Usage: +............... +new [<config>] <template> [<template> ...] [params name=value ...] +............... + +Example: +............... +new vip virtual-ip +new bigfs ocfs2 params device=/dev/sdx8 directory=/bigfs +new apache +............... + +[[cmdhelp_template_show,show the processed configuration]] +==== `show` + +Process the current or given configuration and display the result. + +Usage: +............... +show [<config>] +............... + +[[cmdhelp_cibstatus,CIB status management and editing]] +=== `cibstatus` - CIB status management and editing + +The `status` section of the CIB keeps the current status of nodes +and resources. It is modified _only_ on events, i.e. when some +resource operation is run or node status changes. For obvious +reasons, the CRM has no user interface with which it is possible +to affect the status section. From the user's point of view, the +status section is essentially a read-only part of the CIB. The +current status is never even written to disk, though it is +available in the PE (Policy Engine) input files which represent +the history of cluster motions. The current status may be read +using the +cibadmin -Q+ command. + +It may sometimes be of interest to see how status changes would +affect the Policy Engine. The set of `cibstatus` level commands +allow the user to load status sections from various sources and +then insert or modify resource operations or change nodes' state. + +The effect of those changes may then be observed by running the +<<cmdhelp_configure_ptest,`ptest`>> command at the `configure` level +or `simulate` and `run` commands at this level. The `ptest` +runs with the user edited CIB whereas the latter two commands +run with the CIB which was loaded along with the status section. + +The `simulate` and `run` commands as well as all status +modification commands are implemented using `crm_simulate(8)`. + +[[cmdhelp_cibstatus_load,load the CIB status section]] +==== `load` + +Load a status section from a file, a shadow CIB, or the running +cluster. By default, the current (+live+) status section is +modified. Note that if the +live+ status section is modified it +is not going to be updated if the cluster status changes, because +that would overwrite the user changes. To make `crm` drop changes +and resume use of the running cluster status, run +load live+. + +All CIB shadow configurations contain the status section which is +a snapshot of the status section taken at the time the shadow was +created. Obviously, this status section doesn't have much to do +with the running cluster status, unless the shadow CIB has just +been created. Therefore, the `ptest` command by default uses the +running cluster status section. + +Usage: +............... +load {<file>|shadow:<cib>|live} +............... +Example: +............... +load bug-12299.xml +load shadow:test1 +............... + +[[cmdhelp_cibstatus_node,change node status]] +==== `node` + +Change the node status. It is possible to throw a node out of +the cluster, make it a member, or set its state to unclean. + ++online+:: Set the +node_state+ `crmd` attribute to +online+ +and the +expected+ and +join+ attributes to +member+. The effect +is that the node becomes a cluster member. + ++offline+:: Set the +node_state+ `crmd` attribute to +offline+ +and the +expected+ attribute to empty. This makes the node +cleanly removed from the cluster. + ++unclean+:: Set the +node_state+ `crmd` attribute to +offline+ +and the +expected+ attribute to +member+. In this case the node +has unexpectedly disappeared. + +Usage: +............... +node <node> {online|offline|unclean} +............... +Example: +............... +node xen-b unclean +............... + +[[cmdhelp_cibstatus_op,edit outcome of a resource operation]] +==== `op` + +Edit the outcome of a resource operation. This way you can +tell CRM that it ran an operation and that the resource agent +returned certain exit code. It is also possible to change the +operation's status. In case the operation status is set to +something other than +done+, the exit code is effectively +ignored. + +Usage: +............... +op <operation> <resource> <exit_code> [<op_status>] [<node>] + +operation :: probe | monitor[:<n>] | start | stop | + promote | demote | notify | migrate_to | migrate_from +exit_code :: <rc> | success | generic | args | + unimplemented | perm | installed | configured | not_running | + master | failed_master +op_status :: pending | done | cancelled | timeout | notsupported | error + +n :: the monitor interval in seconds; if omitted, the first + recurring operation is referenced +rc :: numeric exit code in range 0..9 +............... +Example: +............... +op start d1 xen-b generic +op start d1 xen-b 1 +op monitor d1 xen-b not_running +op stop d1 xen-b 0 timeout +............... + +[[cmdhelp_cibstatus_origin,display origin of the CIB status section]] +==== `origin` + +Show the origin of the status section currently in use. This +essentially shows the latest `load` argument. + +Usage: +............... +origin +............... + +[[cmdhelp_cibstatus_quorum,set the quorum]] +==== `quorum` + +Set the quorum value. + +Usage: +............... +quorum <bool> +............... +Example: +............... +quorum false +............... + +[[cmdhelp_cibstatus_run,run policy engine]] +==== `run` + +Run the policy engine with the edited status section. + +Add a string of +v+ characters to increase verbosity. Specify ++scores+ to see allocation scores also. +utilization+ turns on +information about the remaining capacity of nodes. + +If you have graphviz installed and X11 session, `dotty(1)` is run +to display the changes graphically. + +Usage: +............... +run [nograph] [v...] [scores] [utilization] +............... +Example: +............... +run +............... + +[[cmdhelp_cibstatus_save,save the CIB status section]] +==== `save` + +The current internal status section with whatever modifications +were performed can be saved to a file or shadow CIB. + +If the file exists and contains a complete CIB, only the status +section is going to be replaced and the rest of the CIB will +remain intact. Otherwise, the current user edited configuration +is saved along with the status section. + +Note that all modifications are saved in the source file as soon +as they are run. + +Usage: +............... +save [<file>|shadow:<cib>] +............... +Example: +............... +save bug-12299.xml +............... + +[[cmdhelp_cibstatus_show,show CIB status section]] +==== `show` + +Show the current status section in the XML format. Brace yourself +for some unreadable output. Add +changed+ option to get a human +readable output of all changes. + +Usage: +............... +show [changed] +............... + +[[cmdhelp_cibstatus_simulate,simulate cluster transition]] +==== `simulate` + +Run the policy engine with the edited status section and simulate +the transition. + +Add a string of +v+ characters to increase verbosity. Specify ++scores+ to see allocation scores also. +utilization+ turns on +information about the remaining capacity of nodes. + +If you have graphviz installed and X11 session, `dotty(1)` is run +to display the changes graphically. + +Usage: +............... +simulate [nograph] [v...] [scores] [utilization] +............... +Example: +............... +simulate +............... + +[[cmdhelp_cibstatus_ticket,manage tickets]] +==== `ticket` + +Modify the ticket status. Tickets can be granted and revoked. +Granted tickets could be activated or put in standby. + +Usage: +............... +ticket <ticket> {grant|revoke|activate|standby} +............... +Example: +............... +ticket ticketA grant +............... + +[[cmdhelp_assist,Configuration assistant]] +=== `assist` - Configuration assistant + +The `assist` sublevel is a collection of helper +commands that create or modify resources and +constraints, to simplify the creation of certain +configurations. + +For more information on individual commands, see +the help text for those commands. + +[[cmdhelp_assist_template,Create template for primitives]] +==== `template` + +This command takes a list of primitives as argument, and creates a new +`rsc_template` for these primitives. It can only do this if the +primitives do not already share a template and are of the same type. + +Usage: +........ +template primitive-1 primitive-2 primitive-3 +........ + +[[cmdhelp_assist_weak-bond,Create a weak bond between resources]] +==== `weak-bond` + +A colocation between a group of resources says that the resources +should be located together, but it also means that those resources are +dependent on each other. If one of the resources fails, the others +will be restarted. + +If this is not desired, it is possible to circumvent: By placing the +resources in a non-sequential set and colocating the set with a dummy +resource which is not monitored, the resources will be placed together +but will have no further dependency on each other. + +This command creates both the constraint and the dummy resource needed +for such a colocation. + +Usage: +........ +weak-bond resource-1 resource-2 +........ + +[[cmdhelp_maintenance,Maintenance mode commands]] +=== `maintenance` - Maintenance mode commands + +Maintenance mode commands are commands that manipulate resources +directly without going through the cluster infrastructure. Therefore, +it is essential to ensure that the cluster does not attempt to monitor +or manipulate the resources while these commands are being executed. + +To ensure this, these commands require that maintenance mode is set +either for the particular resource, or for the whole cluster. + +[[cmdhelp_maintenance_action,Invoke a resource action]] +==== `action` + +Invokes the given action for the resource. This is +done directly via the resource agent, so the command must +be issued while the cluster or the resource is in +maintenance mode. + +Unless the action is `start` or `monitor`, the action must be invoked +on the same node as where the resource is running. If the resource is +running on multiple nodes, the command will fail. + +To use SSH for executing resource actions on multiple nodes, append +`ssh` after the action name. This requires SSH access to be configured +between the nodes and the parallax python package to be installed. + +Usage: +............... +action <rsc> <action> +action <rsc> <action> ssh +............... +Example: +............... +action webserver reload +action webserver monitor ssh +............... + +[[cmdhelp_maintenance_off,Disable maintenance mode]] +==== `off` + +Disables maintenances mode, either for the whole cluster +or for the given resource. + +Usage: +............... +off +off <rsc> +............... +Example: +............... +off rsc1 +............... + +[[cmdhelp_maintenance_on,Enable maintenance mode]] +==== `on` + +Enables maintenances mode, either for the whole cluster +or for the given resource. + +Usage: +............... +on +on <rsc> +............... +Example: +............... +on rsc1 +............... + +[[cmdhelp_history,Cluster history]] +=== `history` - Cluster history + +Examining Pacemaker's history is a particularly involved task. The +number of subsystems to be considered, the complexity of the +configuration, and the set of various information sources, most of +which are not exactly human readable, keep analyzing resource or node +problems accessible to only the most knowledgeable. Or, depending on +the point of view, to the most persistent. The following set of +commands has been devised in hope to make cluster history more +accessible. + +Of course, looking at _all_ history could be time consuming regardless +of how good the tools at hand are. Therefore, one should first say +which period he or she wants to analyze. If not otherwise specified, +the last hour is considered. Logs and other relevant information is +collected using `crm report`. Since this process takes some time and +we always need fresh logs, information is refreshed in a much faster +way using the python parallax module. If +python-parallax+ is not +found on the system, examining a live cluster is still possible -- +though not as comfortable. + +Apart from examining a live cluster, events may be retrieved from a +report generated by `crm report` (see also the +-H+ option). In that +case we assume that the period stretching the whole report needs to be +investigated. Of course, it is still possible to further reduce the +time range. + +If you have discovered an issue that you want to show someone else, +you can use the `session pack` command to save the current session as +a tarball, similar to those generated by `crm report`. + +In order to minimize the size of the tarball, and to make it easier +for others to find the interesting events, it is recommended to limit +the time frame which the saved session covers. This can be done using +the `timeframe` command (example below). + +It is also possible to name the saved session using the `session save` +command. + +Example: +............... +crm(live)history# limit "Jul 18 12:00" "Jul 18 12:30" +crm(live)history# session save strange_restart +crm(live)history# session pack +Report saved in .../strange_restart.tar.bz2 +crm(live)history# +............... + +[[cmdhelp_history_detail,set the level of detail shown]] +==== `detail` + +How much detail to show from the logs. Valid detail levels are either +`0` or `1`, where `1` is the highest detail level. The default detail +level is `0`. + +Usage: +............... +detail <detail_level> + +detail_level :: small integer (defaults to 0) +............... +Example: +............... +detail 1 +............... + +[[cmdhelp_history_diff,cluster states/transitions difference]] +==== `diff` + +A transition represents a change in cluster configuration or +state. Use `diff` to see what has changed between two +transitions. + +If you want to specify the current cluster configuration and +status, use the string +live+. + +Normally, the first transition specified should be the one which +is older, but we are not going to enforce that. + +Note that a single configuration update may result in more than +one transition. + +Usage: +............... +diff <pe> <pe> [status] [html] + +pe :: <number>|<index>|<file>|live +............... +Examples: +............... +diff 2066 2067 +diff pe-input-2080.bz2 live status +............... + +[[cmdhelp_history_events,Show events in log]] +==== `events` + +By analysing the log output and looking for particular +patterns, the `events` command helps sifting through +the logs to find when particular events like resources +changing state or node failure may have occurred. + +This can be used to generate a combined list of events +from all nodes. + +Usage: +............... +events +............... + +Example: +............... +events +............... + +[[cmdhelp_history_exclude,exclude log messages]] +==== `exclude` + +If a log is infested with irrelevant messages, those messages may +be excluded by specifying a regular expression. The regular +expressions used are Python extended. This command is additive. +To drop all regular expressions, use +exclude clear+. Run +`exclude` only to see the current list of regular expressions. +Excludes are saved along with the history sessions. + +Usage: +............... +exclude [<regex>|clear] +............... +Example: +............... +exclude kernel.*ocfs2 +............... + +[[cmdhelp_history_graph,generate a directed graph from the PE file]] +==== `graph` + +Create a graphviz graphical layout from the PE file (the +transition). Every transition contains the cluster configuration +which was active at the time. See also <<cmdhelp_configure_graph,generate a directed graph +from configuration>>. + +Usage: +............... +graph <pe> [<gtype> [<file> [<img_format>]]] + +gtype :: dot +img_format :: `dot` output format (see the +-T+ option) +............... +Example: +............... +graph -1 +graph 322 dot clu1.conf.dot +graph 322 dot clu1.conf.svg svg +............... + +[[cmdhelp_history_info,Cluster information summary]] +==== `info` + +The `info` command provides a summary of the information source, which +can be either a live cluster snapshot or a previously generated +report. + +Usage: +............... +info +............... +Example: +............... +info +............... + +[[cmdhelp_history_latest,show latest news from the cluster]] +==== `latest` + +The `latest` command shows a bit of recent history, more +precisely whatever happened since the last cluster change (the +latest transition). If the transition is running, the shell will +first wait until it finishes. + +Usage: +............... +latest +............... +Example: +............... +latest +............... + +[[cmdhelp_history_limit,limit timeframe to be examined]] +==== `limit` (`timeframe`) + +This command can be used to modify the time span to examine. All +history commands look at events within a certain time span. + +For the `live` source, the default time span is the _last hour_. + +There is no time span limit for the `hb_report` source. + +The time period is parsed by the `dateutil` python module. It +covers a wide range of date formats. For instance: + +- 3:00 (today at 3am) +- 15:00 (today at 3pm) +- 2010/9/1 2pm (September 1st 2010 at 2pm) + +For more examples of valid time/date statements, please refer to the +`python-dateutil` documentation: + +- https://dateutil.readthedocs.org/[dateutil.readthedocs.org] + +If the dateutil module is not available, then the time is parsed using +strptime and only the kind as printed by `date(1)` is allowed: + +- Tue Sep 15 20:46:27 CEST 2010 + +Usage: +............... +limit [<from_time>] [<to_time>] +............... +Examples: +............... +limit 10:15 +limit 15h22m 16h +limit "Sun 5 20:46" "Sun 5 22:00" +............... + +[[cmdhelp_history_log,log content]] +==== `log` + +Show messages logged on one or more nodes. Leaving out a node +name produces combined logs of all nodes. Messages are sorted by +time and, if the terminal emulations supports it, displayed in +different colours depending on the node to allow for easier +reading. + +The sorting key is the timestamp as written by syslog which +normally has the maximum resolution of one second. Obviously, +messages generated by events which share the same timestamp may +not be sorted in the same way as they happened. Such close events +may actually happen fairly often. + +Usage: +............... +log [<node> [<node> ...] ] +............... +Example: +............... +log node-a +............... + +[[cmdhelp_history_node,node events]] +==== `node` + +Show important events that happened on a node. Important events +are node lost and join, standby and online, and fence. Use either +node names or extended regular expressions. + +Usage: +............... +node <node> [<node> ...] +............... +Example: +............... +node node1 +............... + +[[cmdhelp_history_peinputs,list or get PE input files]] +==== `peinputs` + +Every event in the cluster results in generating one or more +Policy Engine (PE) files. These files describe future motions of +resources. The files are listed as full paths in the current +report directory. Add +v+ to also see the creation time stamps. + +Usage: +............... +peinputs [{<range>|<number>} ...] [v] + +range :: <n1>:<n2> +............... +Example: +............... +peinputs +peinputs 440:444 446 +peinputs v +............... + +[[cmdhelp_history_refresh,refresh live report]] +==== `refresh` + +This command makes sense only for the +live+ source and makes +`crm` collect the latest logs and other relevant information from +the logs. If you want to make a completely new report, specify ++force+. + +Usage: +............... +refresh [force] +............... + +[[cmdhelp_history_resource,resource events]] +==== `resource` + +Show actions and any failures that happened on all specified +resources on all nodes. Normally, one gives resource names as +arguments, but it is also possible to use extended regular +expressions. Note that neither groups nor clones or master/slave +names are ever logged. The resource command is going to expand +all of these appropriately, so that clone instances or resources +which are part of a group are shown. + +Usage: +............... +resource <rsc> [<rsc> ...] +............... +Example: +............... +resource bigdb public_ip +resource my_.*_db2 +resource ping_clone +............... + +[[cmdhelp_history_session,manage history sessions]] +==== `session` + +Sometimes you may want to get back to examining a particular +history period or bug report. In order to make that easier, the +current settings can be saved and later retrieved. + +If the current history being examined is coming from a live +cluster the logs, PE inputs, and other files are saved too, +because they may disappear from nodes. For the existing reports +coming from `hb_report`, only the directory location is saved +(not to waste space). + +A history session may also be packed into a tarball which can +then be sent to support. + +Leave out subcommand to see the current session. + +Usage: +............... +session [{save|load|delete} <name> | pack [<name>] | update | list] +............... +Examples: +............... +session save bnc966622 +session load rsclost-2 +session list +............... + +[[cmdhelp_history_setnodes,set the list of cluster nodes]] +==== `setnodes` + +In case the host this program runs on is not part of the cluster, +it is necessary to set the list of nodes. + +Usage: +............... +setnodes node <node> [<node> ...] +............... +Example: +............... +setnodes node_a node_b +............... + +[[cmdhelp_history_show,show status or configuration of the PE input file]] +==== `show` + +Every transition is saved as a PE file. Use this command to +render that PE file either as configuration or status. The +configuration output is the same as `crm configure show`. + +Usage: +............... +show <pe> [status] + +pe :: <number>|<index>|<file>|live +............... +Examples: +............... +show 2066 +show pe-input-2080.bz2 status +............... + +[[cmdhelp_history_source,set source to be examined]] +==== `source` + +Events to be examined can come from the current cluster or from a +`hb_report` report. This command sets the source. `source live` +sets source to the running cluster and system logs. If no source +is specified, the current source information is printed. + +In case a report source is specified as a file reference, the file +is going to be unpacked in place where it resides. This directory +is not removed on exit. + +Usage: +............... +source [<dir>|<file>|live] +............... +Examples: +............... +source live +source /tmp/customer_case_22.tar.bz2 +source /tmp/customer_case_22 +source +............... + +[[cmdhelp_history_transition,show transition]] +==== `transition` + +This command will print actions planned by the PE and run +graphviz (`dotty`) to display a graphical representation of the +transition. Of course, for the latter an X11 session is required. +This command invokes `ptest(8)` in background. + +The +showdot+ subcommand runs graphviz (`dotty`) to display a +graphical representation of the +.dot+ file which has been +included in the report. Essentially, it shows the calculation +produced by `pengine` which is installed on the node where the +report was produced. In optimal case this output should not +differ from the one produced by the locally installed `pengine`. + +The `log` subcommand shows the full log for the duration of the +transition. + +A transition can also be saved to a CIB shadow for further +analysis or use with `cib` or `configure` commands (use the +`save` subcommand). The shadow file name defaults to the name of +the PE input file. + +If the PE input file number is not provided, it defaults to the +last one, i.e. the last transition. The last transition can also +be referenced with number 0. If the number is negative, then the +corresponding transition relative to the last one is chosen. + +If there are warning and error PE input files or different nodes +were the DC in the observed timeframe, it may happen that PE +input file numbers collide. In that case provide some unique part +of the path to the file. + +After the `ptest` output, logs about events that happened during +the transition are printed. + +The `tags` subcommand scans the logs for the transition and return a +list of key events during that transition. For example, the tag ++error+ will be returned if there are any errors logged during the +transition. + +Usage: +............... +transition [<number>|<index>|<file>] [nograph] [v...] [scores] [actions] [utilization] +transition showdot [<number>|<index>|<file>] +transition log [<number>|<index>|<file>] +transition save [<number>|<index>|<file> [name]] +transition tags [<number>|<index>|<file>] +............... +Examples: +............... +transition +transition 444 +transition -1 +transition pe-error-3.bz2 +transition node-a/pengine/pe-input-2.bz2 +transition showdot 444 +transition log +transition save 0 enigma-22 +............... + +[[cmdhelp_history_transitions,List transitions]] +==== `transitions` + +A transition represents a change in cluster configuration or +state. This command lists the transitions in the current timeframe. + +Usage: +............... +transitions +............... +Example: +............... +transitions +............... + + +[[cmdhelp_history_wdiff,cluster states/transitions difference]] +==== `wdiff` + +A transition represents a change in cluster configuration or +state. Use `wdiff` to see what has changed between two +transitions as word differences on a line-by-line basis. + +If you want to specify the current cluster configuration and +status, use the string +live+. + +Normally, the first transition specified should be the one which +is older, but we are not going to enforce that. + +Note that a single configuration update may result in more than +one transition. + +Usage: +............... +wdiff <pe> <pe> [status] + +pe :: <number>|<index>|<file>|live +............... +Examples: +............... +wdiff 2066 2067 +wdiff pe-input-2080.bz2 live status +............... + +[[cmdhelp_root_report,Create cluster status report]] +=== `report` + +Interface to a tool for creating a cluster report. A report is an +archive containing log files, configuration files, system information +and other relevant data for a given time period. This is a useful tool +for collecting data to attach to bug reports, or for detecting the +root cause of errors resulting in resource failover, for example. + +See `crmsh_hb_report(8)` for more details on arguments, +or call `crm report -h` + +Usage: +............... +report -f {time|"cts:"testnum} [-t time] [-u user] [-l file] + [-n nodes] [-E files] [-p patt] [-L patt] [-e prog] + [-MSDZAVsvhd] [dest] +............... + +Examples: +............... +report -f 2pm report_1 +report -f "2007/9/5 12:30" -t "2007/9/5 14:00" report_2 +report -f 1:00 -t 3:00 -l /var/log/cluster/ha-debug report_3 +report -f "09sep07 2:00" -u hbadmin report_4 +report -f 18:00 -p "usern.*" -p "admin.*" report_5 +report -f cts:133 ctstest_133 +............... + +=== `end` (`cd`, `up`) + +The `end` command ends the current level and the user moves to +the parent level. This command is available everywhere. + +Usage: +............... +end +............... + +=== `help` + +The `help` command prints help for the current level or for the +specified topic (command). This command is available everywhere. + +Usage: +............... +help [<topic>] +............... + +=== `quit` (`exit`, `bye`) + +Leave the program. + +BUGS +---- +Even though all sensible configurations (and most of those that +are not) are going to be supported by the crm shell, I suspect +that it may still happen that certain XML constructs may confuse +the tool. When that happens, please file a bug report. + +The crm shell will not try to update the objects it does not +understand. Of course, it is always possible to edit such objects +in the XML format. + +AUTHORS +------- +Dejan Muhamedagic, <dejan@suse.de> +Kristoffer Gronlund <kgronlund@suse.com> +and many OTHERS + +SEE ALSO +-------- +crm_resource(8), crm_attribute(8), crm_mon(8), cib_shadow(8), +ptest(8), dotty(1), crm_simulate(8), cibadmin(8) + + +COPYING +------- +Copyright \(C) 2008-2013 Dejan Muhamedagic. +Copyright \(C) 2013 Kristoffer Gronlund. + +Free use of this software is granted under the terms of the GNU General Public License (GPL). + +////////////////////// + vim:ts=4:sw=4:expandtab: +////////////////////// diff --git a/doc/website-v1/man-3.adoc b/doc/website-v1/man-3.adoc new file mode 100644 index 0000000..e4411cc --- /dev/null +++ b/doc/website-v1/man-3.adoc @@ -0,0 +1,5309 @@ +:man source: crm +:man version: 2.3.0 +:man manual: crmsh documentation + +crm(8) +====== + +NAME +---- +crm - Pacemaker command line interface for configuration and management + + +SYNOPSIS +-------- +*crm* [OPTIONS] [SUBCOMMAND ARGS...] + + +[[topics_Description,Program description]] +DESCRIPTION +----------- +The `crm` shell is a command-line based cluster configuration and +management tool. Its goal is to assist as much as possible with the +configuration and maintenance of Pacemaker-based High Availability +clusters. + +For more information on Pacemaker itself, see http://clusterlabs.org/. + +`crm` works both as a command-line tool to be called directly from the +system shell, and as an interactive shell with extensive tab +completion and help. + +The primary focus of the `crm` shell is to provide a simplified and +consistent interface to Pacemaker, but it also provides tools for +managing the creation and configuration of High Availability clusters +from scratch. To learn more about this aspect of `crm`, see the +`cluster` section below. + +The `crm` shell can be used to manage every aspect of configuring and +maintaining a cluster. It provides a simplified line-based syntax on +top of the XML configuration format used by Pacemaker, commands for +starting and stopping resources, tools for exploring the history of a +cluster including log scraping and a set of cluster scripts useful for +automating the setup and installation of services on the cluster +nodes. + +The `crm` shell is line oriented: every command must start and finish +on the same line. It is possible to use a continuation character (+\+) +to write one command in two or more lines. The continuation character +is commonly used when displaying configurations. + +[[topics_CommandLine,Command line options]] +OPTIONS +------- +*-f, --file*='FILE':: + Load commands from the given file. If a dash +-+ is used in place + of a file name, `crm` will read commands from the shell standard + input (`stdin`). + +*-c, --cib*='CIB':: + Start the session using the given shadow CIB file. + Equivalent to +cib use <CIB>+. + +*-D, --display=*'OUTPUT_TYPE':: + Choose one of the output options: +plain+, +color-always+, +color+, + or +uppercase+. The default is +color+ if the terminal emulation + supports colors. Otherwise, +plain+ is used. + +*-F, --force*:: + Make `crm` proceed with applying changes where it would normally + ask the user to confirm before proceeding. This option is mainly + useful in scripts, and should be used with care. + +*-w, --wait*:: + Make `crm` wait for the cluster transition to finish (for the + changes to take effect) after each processed line. + +*-H, --history*='DIR|FILE|SESSION':: + A directory or file containing a cluster report to load + into the `history` commands, or the name of a previously + saved history session. + +*-h, --help*:: + Print help page. + +*--version*:: + Print crmsh version and build information (Mercurial Hg changeset + hash). + +*-d, --debug*:: + Print verbose debugging information. + +*-R, --regression-tests*:: + Enables extra verbose trace logging used by the regression + tests. Logs all external calls made by crmsh. + +*--scriptdir*='DIR':: + Extra directory where crm looks for cluster scripts, or a list of + directories separated by semi-colons (e.g. +/dir1;/dir2;etc.+). + +*-o, --opt*='OPTION=VALUE':: + Set crmsh option temporarily. If the options are saved using + +options save+ then the value passed here will also be saved. + Multiple options can be set by using +-o+ multiple times. + +[[topics_Introduction,Introduction]] +== Introduction + +This section of the user guide covers general topics about the user +interface and describes some of the features of `crmsh` in detail. + +[[topics_Introduction_Interface,User interface]] +=== User interface + +The main purpose of `crmsh` is to provide a simple yet powerful +interface to the cluster stack. There are two main modes of operation +with the user interface of `crmsh`: + +* Command line (single-shot) use - Use `crm` as a regular UNIX command + from your usual shell. `crm` has full bash completion built in, so + using it in this manner should be as comfortable and familiar as + using any other command-line tool. + +* Interactive mode - By calling `crm` without arguments, or by calling + it with only a sublevel as argument, `crm` enters the interactive + mode. In this mode, it acts as its own command shell, which + remembers which sublevel you are currently in and allows for rapid + and convenient execution of multiple commands within the same + sublevel. This mode also has full tab completion, as well as + built-in interactive help and syntax highlighting. + +Here are a few examples of using `crm` both as a command-line tool and +as an interactive shell: + +.Command line (one-shot) use: +........ +# crm resource stop www_app +........ + +.Interactive use: +........ +# crm +crm(live)# resource +crm(live)resource# unmanage tetris_1 +crm(live)resource# up +crm(live)# node standby node4 +........ + +.Cluster configuration: +........ +# crm configure<<EOF + # + # resources + # + primitive disk0 iscsi \ + params portal=192.168.2.108:3260 target=iqn.2008-07.com.suse:disk0 + primitive fs0 Filesystem \ + params device=/dev/disk/by-label/disk0 directory=/disk0 fstype=ext3 + primitive internal_ip IPaddr params ip=192.168.1.101 + primitive apache apache \ + params configfile=/disk0/etc/apache2/site0.conf + primitive apcfence stonith:apcsmart \ + params ttydev=/dev/ttyS0 hostlist="node1 node2" \ + op start timeout=60s + primitive pingd pingd \ + params name=pingd dampen=5s multiplier=100 host_list="r1 r2" + # + # monitor apache and the UPS + # + monitor apache 60s:30s + monitor apcfence 120m:60s + # + # cluster layout + # + group internal_www \ + disk0 fs0 internal_ip apache + clone fence apcfence \ + meta globally-unique=false clone-max=2 clone-node-max=1 + clone conn pingd \ + meta globally-unique=false clone-max=2 clone-node-max=1 + location node_pref internal_www \ + rule 50: #uname eq node1 \ + rule pingd: defined pingd + # + # cluster properties + # + property stonith-enabled=true + commit +EOF +........ + +The `crm` interface is hierarchical, with commands organized into +separate levels by functionality. To list the available levels and +commands, either execute +help <level>+, or, if at the top level of +the shell, simply typing `help` will provide an overview of all +available levels and commands. + +The +(live)+ string in the `crm` prompt signifies that the current CIB +in use is the cluster live configuration. It is also possible to +work with so-called <<topics_Features_Shadows,shadow CIBs>>. These are separate, inactive +configurations stored in files, that can be applied and thereby +replace the live configuration at any time. + +[[topics_Introduction_Completion,Tab completion]] +=== Tab completion + +The `crm` makes extensive use of tab completion. The completion +is both static (i.e. for `crm` commands) and dynamic. The latter +takes into account the current status of the cluster or +information from installed resource agents. Sometimes, completion +may also be used to get short help on resource parameters. Here +are a few examples: + +............... +crm(live)resource# <TAB><TAB> +bye failcount move restart unmigrate +cd help param show unmove +cleanup list promote start up +demote manage quit status utilization +end meta refresh stop +exit migrate reprobe unmanage + +crm(live)configure# primitive fence-1 <TAB><TAB> +heartbeat: lsb: ocf: stonith: + +crm(live)configure# primitive fence-1 stonith:<TAB><TAB> +apcmaster external/ippower9258 fence_legacy +apcmastersnmp external/kdumpcheck ibmhmc +apcsmart external/libvirt ipmilan + +crm(live)configure# primitive fence-1 stonith:ipmilan params <TAB><TAB> +auth= hostname= ipaddr= login= password= port= priv= + +crm(live)configure# primitive fence-1 stonith:ipmilan params auth=<TAB><TAB> +auth* (string) + The authorization type of the IPMI session ("none", "straight", "md2", or "md5") +............... + +`crmsh` also comes with bash completion usable directly from the +system shell. This should be installed automatically with the command +itself. + +[[topics_Introduction_Shorthand,Shorthand syntax]] +=== Shorthand syntax + +When using the `crm` shell to manage clusters, you will end up typing +a lot of commands many times over. Clear command names like ++configure+ help in understanding and learning to use the cluster +shell, but is easy to misspell and is tedious to type repeatedly. The +interactive mode and tab completion both help with this, but the `crm` +shell also has the ability to understand a variety of shorthand +aliases for all of the commands. + +For example, instead of typing `crm status`, you can type `crm st` or +`crm stat`. Instead of `crm configure` you can type `crm cfg` or even +`crm cf`. `crm resource` can be shorted as `crm rsc`, and so on. + +The exact list of accepted aliases is too long to print in full, but +experimentation and typos should help in discovering more of them. + +[[topics_Features,Features]] +== Features + +The feature set of crmsh covers a wide range of functionality, and +understanding how and when to use the various features of the shell +can be difficult. This section of the guide describes some of the +features and use cases of `crmsh` in more depth. The intention is to +provide a deeper understanding of these features, but also to serve as +a guide to using them. + +[[topics_Features_Shadows,Shadow CIB usage]] +=== Shadow CIB usage + +A Shadow CIB is a normal cluster configuration stored in a file. +They may be manipulated in much the same way as the _live_ CIB, with +the key difference that changes to a shadow CIB have no effect on the +actual cluster resources. An administrator may choose to apply any of +them to the cluster, thus replacing the running configuration with the +one found in the shadow CIB. + +The `crm` prompt always contains the name of the configuration which +is currently in use, or the string _live_ if using the live cluster +configuration. + +When editing the configuration in the `configure` level, no changes +are actually applied until the `commit` command is executed. It is +possible to start editing a configuration as usual, but instead of +committing the changes to the active CIB, save them to a shadow CIB. + +The following example `configure` session demonstrates how this can be +done: +............... +crm(live)configure# cib new test-2 +INFO: test-2 shadow CIB created +crm(test-2)configure# commit +............... + +[[topics_Features_Checks,Configuration semantic checks]] +=== Configuration semantic checks + +Resource definitions may be checked against the meta-data +provided with the resource agents. These checks are currently +carried out: + +- are required parameters set +- existence of defined parameters +- timeout values for operations + +The parameter checks are obvious and need no further explanation. +Failures in these checks are treated as configuration errors. + +The timeouts for operations should be at least as long as those +recommended in the meta-data. Too short timeout values are a +common mistake in cluster configurations and, even worse, they +often slip through if cluster testing was not thorough. Though +operation timeouts issues are treated as warnings, make sure that +the timeouts are usable in your environment. Note also that the +values given are just _advisory minimum_---your resources may +require longer timeouts. + +User may tune the frequency of checks and the treatment of errors +by the <<cmdhelp_options_check-frequency,`check-frequency`>> and +<<cmdhelp_options_check-mode,`check-mode`>> preferences. + +Note that if the +check-frequency+ is set to +always+ and the ++check-mode+ to +strict+, errors are not tolerated and such +configuration cannot be saved. + +[[topics_Features_Templates,Configuration templates]] +=== Configuration templates + +.Deprecation note +**************************** +Configuration templates have been deprecated in favor of the more +capable `cluster scripts`. To learn how to use cluster scripts, see +the dedicated documentation on the `crmsh` website at +http://crmsh.github.io/, or in the <<cmdhelp_script,Script section>>. +**************************** + +Configuration templates are ready made configurations created by +cluster experts. They are designed in such a way so that users +may generate valid cluster configurations with minimum effort. +If you are new to Pacemaker, templates may be the best way to +start. + +We will show here how to create a simple yet functional Apache +configuration: +............... +# crm configure +crm(live)configure# template +crm(live)configure template# list templates +apache filesystem virtual-ip +crm(live)configure template# new web <TAB><TAB> +apache filesystem virtual-ip +crm(live)configure template# new web apache +INFO: pulling in template apache +INFO: pulling in template virtual-ip +crm(live)configure template# list +web2-d web2 vip2 web3 vip web +............... + +We enter the `template` level from `configure`. Use the `list` +command to show templates available on the system. The `new` +command creates a configuration from the +apache+ template. You +can use tab completion to pick templates. Note that the apache +template depends on a virtual IP address which is automatically +pulled along. The `list` command shows the just created +web+ +configuration, among other configurations (I hope that you, +unlike me, will use more sensible and descriptive names). + +The `show` command, which displays the resulting configuration, +may be used to get an idea about the minimum required changes +which have to be done. All +ERROR+ messages show the line numbers +in which the respective parameters are to be defined: +............... +crm(live)configure template# show +ERROR: 23: required parameter ip not set +ERROR: 61: required parameter id not set +ERROR: 65: required parameter configfile not set +crm(live)configure template# edit +............... + +The `edit` command invokes the preferred text editor with the ++web+ configuration. At the top of the file, the user is advised +how to make changes. A good template should require from the user +to specify only parameters. For example, the +web+ configuration +we created above has the following required and optional +parameters (all parameter lines start with +%%+): +............... +$ grep -n ^%% ~/.crmconf/web +23:%% ip +31:%% netmask +35:%% lvs_support +61:%% id +65:%% configfile +71:%% options +76:%% envfiles +............... + +These lines are the only ones that should be modified. Simply +append the parameter value at the end of the line. For instance, +after editing this template, the result could look like this (we +used tabs instead of spaces to make the values stand out): +............... +$ grep -n ^%% ~/.crmconf/web +23:%% ip 192.168.1.101 +31:%% netmask +35:%% lvs_support +61:%% id websvc +65:%% configfile /etc/apache2/httpd.conf +71:%% options +76:%% envfiles +............... + +As you can see, the parameter line format is very simple: +............... +%% <name> <value> +............... + +After editing the file, use `show` again to display the +configuration: +............... +crm(live)configure template# show +primitive virtual-ip IPaddr \ + params ip=192.168.1.101 +primitive apache apache \ + params configfile="/etc/apache2/httpd.conf" +monitor apache 120s:60s +group websvc \ + apache virtual-ip +............... + +The target resource of the apache template is a group which we +named +websvc+ in this sample session. + +This configuration looks exactly as you could type it at the +`configure` level. The point of templates is to save you some +typing. It is important, however, to understand the configuration +produced. + +Finally, the configuration may be applied to the current +crm configuration (note how the configuration changed slightly, +though it is still equivalent, after being digested at the +`configure` level): +............... +crm(live)configure template# apply +crm(live)configure template# cd .. +crm(live)configure# show +node xen-b +node xen-c +primitive apache apache \ + params configfile="/etc/apache2/httpd.conf" \ + op monitor interval=120s timeout=60s +primitive virtual-ip IPaddr \ + params ip=192.168.1.101 +group websvc apache virtual-ip +............... + +Note that this still does not commit the configuration to the CIB +which is used in the shell, either the running one (+live+) or +some shadow CIB. For that you still need to execute the `commit` +command. + +To complete our example, we should also define the preferred node +to run the service: + +............... +crm(live)configure# location websvc-pref websvc 100: xen-b +............... + +If you are not happy with some resource names which are provided +by default, you can rename them now: + +............... +crm(live)configure# rename virtual-ip intranet-ip +crm(live)configure# show +node xen-b +node xen-c +primitive apache apache \ + params configfile="/etc/apache2/httpd.conf" \ + op monitor interval=120s timeout=60s +primitive intranet-ip IPaddr \ + params ip=192.168.1.101 +group websvc apache intranet-ip +location websvc-pref websvc 100: xen-b +............... + +To summarize, working with templates typically consists of the +following steps: + +- `new`: create a new configuration from templates +- `edit`: define parameters, at least the required ones +- `show`: see if the configuration is valid +- `apply`: apply the configuration to the `configure` level + +[[topics_Features_Testing,Resource testing]] +=== Resource testing + +The amount of detail in a cluster makes all configurations prone +to errors. By far the largest number of issues in a cluster is +due to bad resource configuration. The shell can help quickly +diagnose such problems. And considerably reduce your keyboard +wear. + +Let's say that we entered the following configuration: +............... +node xen-b +node xen-c +node xen-d +primitive fencer stonith:external/libvirt \ + params hypervisor_uri="qemu+tcp://10.2.13.1/system" \ + hostlist="xen-b xen-c xen-d" \ + op monitor interval=2h +primitive svc Xinetd \ + params service=systat \ + op monitor interval=30s +primitive intranet-ip IPaddr2 \ + params ip=10.2.13.100 \ + op monitor interval=30s +primitive apache apache \ + params configfile="/etc/apache2/httpd.conf" \ + op monitor interval=120s timeout=60s +group websvc apache intranet-ip +location websvc-pref websvc 100: xen-b +............... + +Before typing `commit` to submit the configuration to the cib we +can make sure that all resources are usable on all nodes: +............... +crm(live)configure# rsctest websvc svc fencer +............... + +It is important that resources being tested are not running on +any nodes. Otherwise, the `rsctest` command will refuse to do +anything. Of course, if the current configuration resides in a +CIB shadow, then a `commit` is irrelevant. The point being that +resources are not running on any node. + +.Note on stopping all resources +**************************** +Alternatively to not committing a configuration, it is also +possible to tell Pacemaker not to start any resources: + +............... +crm(live)configure# property stop-all-resources=yes +............... +Almost none---resources of class stonith are still started. But +shell is not as strict when it comes to stonith resources. +**************************** + +Order of resources is significant insofar that a resource depends +on all resources to its left. In most configurations, it's +probably practical to test resources in several runs, based on +their dependencies. + +Apart from groups, `crm` does not interpret constraints and +therefore knows nothing about resource dependencies. It also +doesn't know if a resource can run on a node at all in case of an +asymmetric cluster. It is up to the user to specify a list of +eligible nodes if a resource is not meant to run on every node. + +[[topics_Features_Security,Access Control Lists (ACL)]] +=== Access Control Lists (ACL) + +.Note on ACLs in Pacemaker 1.1.12 +**************************** +The support for ACLs has been revised in Pacemaker version 1.1.12 and +up. Depending on which version you are using, the information in this +section may no longer be accurate. Look for the `acl_target` +configuration element for more details on the new syntax. +**************************** + +By default, the users from the +haclient+ group have full access +to the cluster (or, more precisely, to the CIB). Access control +lists allow for finer access control to the cluster. + +Access control lists consist of an ordered set of access rules. +Each rule allows read or write access or denies access +completely. Rules are typically combined to produce a specific +role. Then, users may be assigned a role. + +For instance, this is a role which defines a set of rules +allowing management of a single resource: + +............... +role bigdb_admin \ + write meta:bigdb:target-role \ + write meta:bigdb:is-managed \ + write location:bigdb \ + read ref:bigdb +............... + +The first two rules allow modifying the +target-role+ and ++is-managed+ meta attributes which effectively enables users in +this role to stop/start and manage/unmanage the resource. The +constraints write access rule allows moving the resource around. +Finally, the user is granted read access to the resource +definition. + +For proper operation of all Pacemaker programs, it is advisable +to add the following role to all users: + +............... +role read_all \ + read cib +............... + +For finer grained read access try with the rules listed in the +following role: + +............... +role basic_read \ + read node attribute:uname \ + read node attribute:type \ + read property \ + read status +............... + +It is however possible that some Pacemaker programs (e.g. +`ptest`) may not function correctly if the whole CIB is not +readable. + +Some of the ACL rules in the examples above are expanded by the +shell to XPath specifications. For instance, ++meta:bigdb:target-role+ expands to: + +........ +//primitive[@id='bigdb']/meta_attributes/nvpair[@name='target-role'] +........ + +You can see the expansion by showing XML: + +............... +crm(live) configure# show xml bigdb_admin +... +<acls> + <acl_role id="bigdb_admin"> + <write id="bigdb_admin-write" + xpath="//primitive[@id='bigdb']/meta_attributes/nvpair[@name='target-role']"/> +............... + +Many different XPath expressions can have equal meaning. For +instance, the following two are equal, but only the first one is +going to be recognized as shortcut: + +............... +//primitive[@id='bigdb']/meta_attributes/nvpair[@name='target-role'] +//resources/primitive[@id='bigdb']/meta_attributes/nvpair[@name='target-role'] +............... + +XPath is a powerful language, but you should try to keep your ACL +xpaths simple and the builtin shortcuts should be used whenever +possible. + +[[topics_Features_Resourcesets,Syntax: Resource sets]] +=== Syntax: Resource sets + +Using resource sets can be a bit confusing unless one knows the +details of the implementation in Pacemaker as well as how to interpret +the syntax provided by `crmsh`. + +Three different types of resource sets are provided by `crmsh`, and +each one implies different values for the two resource set attributes, ++sequential+ and +require-all+. + ++sequential+:: + If false, the resources in the set do not depend on each other + internally. Setting +sequential+ to +true+ implies a strict order of + dependency within the set. + ++require-all+:: + If false, only one resource in the set is required to fulfil the + requirements of the set. The set of A, B and C with +require-all+ + set to +false+ is be read as "A OR B OR C" when its dependencies + are resolved. + +The three types of resource sets modify the attributes in the +following way: + +1. Implicit sets (no brackets). +sequential=true+, +require-all=true+ +2. Parenthesis set (+(+ ... +)+). +sequential=false+, +require-all=true+ +3. Bracket set (+[+ ... +]+). +sequential=false+, +require-all=false+ + +To create a set with the properties +sequential=true+ and ++require-all=false+, explicitly set +sequential+ in a bracketed set, ++[ A B C sequential=true ]+. + +To create multiple sets with both +sequential+ and +require-all+ set to +true, explicitly set +sequential+ in a parenthesis set: ++A B ( C D sequential=true )+. + +[[topics_Features_AttributeListReferences,Syntax: Attribute list references]] +=== Syntax: Attribute list references + +Attribute lists are used to set attributes and parameters for +resources, constraints and property definitions. For example, to set +the virtual IP used by an +IPAddr2+ resource the attribute +ip+ can be +set in an attribute list for that resource. + +Attribute lists can have identifiers that name them, and other +resources can reuse the same attribute list by referring to that name +using an +$id-ref+. For example, the following statement defines a +simple dummy resource with an attribute list which sets the parameter ++state+ to the value 1 and sets the identifier for the attribute list +to +on-state+: + +.............. +primitive dummy-1 Dummy params $id=on-state state=1 +.............. + +To refer to this attribute list from a different resource, refer to +the +on-state+ name using an id-ref: + +.............. +primitive dummy-2 Dummy params $id-ref=on-state +.............. + +The resource +dummy-2+ will now also have the parameter +state+ set to the value 1. + +[[topics_Features_AttributeReferences,Syntax: Attribute references]] +=== Syntax: Attribute references + +In some cases, referencing complete attribute lists is too +coarse-grained, for example if two different parameters with different +names should have the same value set. Instead of having to copy the +value in multiple places, it is possible to create references to +individual attributes in attribute lists. + +To name an attribute in order to be able to refer to it later, prefix +the attribute name with a +$+ character (as seen above with the +special names +$id+ and +$id-ref+: + +............ +primitive dummy-1 Dummy params $state=1 +............ + +The identifier +state+ can now be used to refer to this attribute from other +primitives, using the +@<id>+ syntax: + +............ +primitive dummy-2 Dummy params @state +............ + +In some cases, using the attribute name as the identifier doesn't work +due to name clashes. In this case, the syntax +$<id>:<name>=<value>+ +can be used to give the attribute a different identifier: + +............ +primitive dummy-1 params $dummy-state-on:state=1 +primitive dummy-2 params @dummy-state-on +............ + +There is also the possibility that two resources both use the same +attribute value but with different names. For example, a web server +may have a parameter +server_ip+ for setting the IP address where it +listens for incoming requests, and a virtual IP resource may have a +parameter called +ip+ which sets the IP address it creates. To +configure these two resources with an IP without repeating the value, +the reference can be given a name using the syntax +@<id>:<name>+. + +Example: +............ +primitive virtual-ip IPaddr2 params $vip:ip=192.168.1.100 +primitive webserver apache params @vip:server_ip +............ + +[[topics_Syntax_RuleExpressions,Syntax: Rule expressions]] +=== Syntax: Rule expressions + +Many of the configuration commands in `crmsh` now support the use of +_rule expressions_, which can influence what attributes apply to a +resource or under which conditions a constraint is applied, depending +on changing conditions like date, time, the value of attributes and +more. + +Here is an example of a simple rule expression used to apply a +a different resource parameter on the node named `node1`: + +.............. +primitive my_resource Special \ + params 2: rule #uname eq node1 interface=eth1 \ + params 1: interface=eth0 +.............. + +This primitive resource has two lists of parameters with descending +priority. The parameter list with the highest priority is applied +first, but only if the rule expressions for that parameter list all +apply. In this case, the rule `#uname eq node1` limits the parameter +list so that it is only applied on `node1`. + +Note that rule expressions are not terminated and are immediately +followed by the data to which the rule is applied. In this case, the +name-value pair `interface=eth1`. + +Rule expressions can contain multiple expressions connected using the +boolean operator `or` and `and`. The full syntax for rule expressions +is listed below. + +.............. +rules :: + rule [id_spec] [$role=<role>] <score>: <expression> + [rule [id_spec] [$role=<role>] <score>: <expression> ...] + +id_spec :: $id=<id> | $id-ref=<id> +score :: <number> | <attribute> | [-]inf +expression :: <simple_exp> [<bool_op> <simple_exp> ...] +bool_op :: or | and +simple_exp :: <attribute> [type:]<binary_op> <value> + | <unary_op> <attribute> + | date <date_expr> +type :: <string> | <version> | <number> +binary_op :: lt | gt | lte | gte | eq | ne +unary_op :: defined | not_defined + +date_expr :: lt <end> + | gt <start> + | in start=<start> end=<end> + | in start=<start> <duration> + | spec <date_spec> +duration|date_spec :: + hours=<value> + | monthdays=<value> + | weekdays=<value> + | yearsdays=<value> + | months=<value> + | weeks=<value> + | years=<value> + | weekyears=<value> + | moon=<value> +.............. + +[[topics_Reference,Command reference]] +== Command reference + +The commands are structured to be compatible with the shell command +line. Sometimes, the underlying Pacemaker grammar uses characters that +have special meaning in bash, that will need to be quoted. This +includes the hash or pound sign (`#`), single and double quotes, and +any significant whitespace. + +Whitespace is also significant when assigning values, meaning that ++key=value+ is different from +key = value+. + +Commands can be referenced using short-hand as long as the short-hand +is unique. This can be either a prefix of the command name or a prefix +string of characters found in the name. + +For example, +status+ can be abbreviated as +st+ or +su+, and ++configure+ as +conf+ or +cfg+. + +The syntax for the commands is given below in an informal, BNF-like +grammar. + +* `<value>` denotes a string. +* `[value]` means that the construct is optional. +* The ellipsis (`...`) signifies that the previous construct may be + repeated. +* `first|second` means either first or second. +* The rest are literals (strings, `:`, `=`). + +[[cmdhelp_root_status,Cluster status]] +=== `status` + +Show cluster status. The status is displayed by `crm_mon`. Supply +additional arguments for more information or different format. +See `crm_mon(8)` for more details. + +Example: +............... +status +status simple +status full +............... + +Usage: +............... +status [<option> ...] + +option :: full + | bynode + | inactive + | ops + | timing + | failcounts + | verbose + | quiet + | html + | xml + | simple + | tickets + | noheaders + | detail + | brief +............... + +[[cmdhelp_root_verify,Verify cluster status]] +=== `verify` + +Performs basic checks for the cluster configuration and +current status, reporting potential issues. + +See `crm_verify(8)` and `crm_simulate(8)` for more details. + +Example: +............... +verify +verify scores +............... + +Usage: +............... +verify [scores] +............... + + +[[cmdhelp_cluster,Cluster setup and management]] +=== `cluster` - Cluster setup and management + +Whole-cluster configuration management with High Availability +awareness. + +The commands on the cluster level allows configuration and +modification of the underlying cluster infrastructure, and also +supplies tools to do whole-cluster systems management. + +These commands enable easy installation and maintenance of a HA +cluster, by providing support for package installation, configuration +of the cluster messaging layer, file system setup and more. + +[[cmdhelp_cluster_add,Add a new node to the cluster]] +==== `add` + +Add a new node to the cluster. The new node will be +configured as a cluster member. + +Options: + +*-y, --yes*:: + Answer "yes" to all prompts (use with caution) + +Usage: +............... +add [options] [<node> ...] +............... + +[[cmdhelp_cluster_copy,Copy file to other cluster nodes]] +==== `copy` + +Copy file to other cluster nodes. + +Copies the given file to all other nodes unless given a +list of nodes to copy to as argument. + +Usage: +............... +copy <filename> [nodes ...] +............... + +Example: +............... +copy /etc/motd +............... + +[[cmdhelp_cluster_diff,Diff file across cluster]] +==== `diff` + +Displays the difference, if any, between a given file +on different nodes. If the second argument is `--checksum`, +a checksum of the file will be calculated and displayed for +each node. + +Usage: +............... +diff <file> [--checksum] [nodes...] +............... + +Example: +............... +diff /etc/crm/crm.conf node2 +diff /etc/resolv.conf --checksum +............... + +[[cmdhelp_cluster_geo_init,Configure cluster as geo cluster]] +==== `geo-init` + +Create a new geo cluster with the current cluster as the +first member. Pass the complete geo cluster topology as +arguments to this command, and then use `geo-join` and +`geo-init-arbitrator` to add the remaining members to +the geo cluster. + +Options: + +*-q, --quiet*:: + Be quiet (don't describe what's happening, just do it) + +*-y, --yes*:: + Answer "yes" to all prompts (use with caution) + +*--arbitrator=IP*:: + IP address of geo cluster arbitrator + +*--clusters=DESC*:: + Cluster description (see details below) + +*--tickets=LIST*:: + Tickets to create (space-separated) + + +Cluster Description: + +This is a map of cluster names to IP addresses. +Each IP address will be configured as a virtual IP +representing that cluster in the geo cluster +configuration. + +Example with two clusters named paris and amsterdam: + +............ + --clusters "paris=192.168.10.10 amsterdam=192.168.10.11" +............ + +Name clusters using the +--name+ parameter to `init`. + +Usage: +............... +geo-init [options] +............... + + +[[cmdhelp_cluster_geo_init_arbitrator,Initialize node as geo cluster arbitrator]] +==== `geo-init-arbitrator` + +Configure the current node as a geo arbitrator. The command +requires an existing geo cluster or geo arbitrator from which +to get the geo cluster configuration. + +Options: + +*--clusters=DESC*:: + Cluster description (see +geo-init+ for details) + +*-c IP, --cluster-node=IP*:: + IP address of an already-configured geo cluster + +Usage: +............... +geo-init-arbitrator [options] +............... + + +[[cmdhelp_cluster_geo_join,Join cluster to existing geo cluster]] +==== `geo-join` + +This command should be run from one of the nodes in a cluster +which is currently not a member of a geo cluster. The geo +cluster configuration will be fetched from the provided node, +and the cluster will be added to the geo cluster. + +Note that each cluster in a geo cluster needs to have a unique +name set. The cluster name can be set using the `--name` argument +to `init`, or by configuring corosync with the cluster name in +an existing cluster. + +Options: + +*-c IP, --cluster-node=IP*:: + IP address of an already-configured geo cluster or arbitrator + +Usage: +............... +geo-join [options] +............... + + +[[cmdhelp_cluster_health,Cluster health check]] +==== `health` + +Runs a larger set of tests and queries on all nodes in the cluster to +verify the general system health and detect potential problems. + +Usage: +............... +health +............... + +[[cmdhelp_cluster_init,Initializes a new HA cluster]] +==== `init` + +Initialize a cluster from scratch. This command configures +a complete cluster, and can also add additional cluster +nodes to the initial one-node cluster using the `--nodes` +option. + +Options: + +*-q, --quiet*:: + Be quiet (don't describe what's happening, just do it) + +*-y, --yes*:: + Answer "yes" to all prompts (use with caution, this + is destructive, especially during the "storage" stage) + +*-t TEMPLATE, --template=TEMPLATE**:: + Optionally configure cluster with template "name" + (currently only "ocfs2" is valid here) + +*-n NAME, --name=NAME*:: + Set the name of the configured cluster. + +*-N NODES, --nodes=NODES*:: + Additional nodes to add to the created cluster. May + include the current node, which will always be the + initial cluster node. + +*-w WATCHDOG, --watchdog=WATCHDOG*:: + Use the given watchdog device. + +Network configuration: + +Options for configuring the network and messaging layer. + +*-i IF, --interface=IF*:: + Bind to IP address on interface IF + +*-u, --unicast*:: + Configure corosync to communicate over unicast (UDP), + and not multicast. Default is multicast unless an + environment where multicast cannot be used is + detected. + +*-A IP, --admin-ip=IP*:: + Configure IP address as an administration virtual IP + +Storage configuration: + +Options for configuring shared storage. + +*-p DEVICE, --partition-device=DEVICE*:: + Partition this shared storage device (only used in + "storage" stage) + +*-s DEVICE, --sbd-device=DEVICE*:: + Block device to use for SBD fencing + +*-o DEVICE, --ocfs2-device=DEVICE*:: + Block device to use for OCFS2 (only used in "vgfs" + stage) + + +Stage can be one of: + +*ssh*:: + Create SSH keys for passwordless SSH between cluster nodes + +*csync2*:: + Configure csync2 + +*corosync*:: + Configure corosync + +*storage*:: + Partition shared storage (ocfs2 template only) + +*sbd*:: + Configure SBD (requires -s <dev>) + +*cluster*:: + Bring the cluster online + +*vgfs*:: + Create volume group and filesystem (ocfs2 template only, requires `-o <dev>`) + +*admin*:: + Create administration virtual IP (optional) + +[NOTE] +============ +- If stage is not specified, the script will run through each stage + in sequence, with prompts for required information. +- If using the ocfs2 template, the storage stage will partition a block + device into two pieces, one for SBD, the remainder for OCFS2. This is + good for testing and demonstration, but not ideal for production. + To use storage you have already configured, pass -s and -o to specify + the block devices for SBD and OCFS2, and the automatic partitioning + will be skipped. +============ + +Usage: +............... +init [options] [STAGE] +............... + + +[[cmdhelp_cluster_join,Join existing cluster]] +==== `join` + +Join the current node to an existing cluster. The +current node cannot be a member of a cluster already. +Pass any node in the existing cluster as the argument +to the `-c` option. + +Options: + +*-q, --quiet*:: + Be quiet (don't describe what's happening, just do it) + +*-y, --yes*:: + Answer "yes" to all prompts (use with caution) + +*-w WATCHDOG, --watchdog=WATCHDOG*:: + Use the given watchdog device + +Network configuration: + +Options for configuring the network and messaging layer. + + +*-c HOST, --cluster-node=HOST*:: + IP address or hostname of existing cluster node + +*-i IF, --interface=IF*:: + Bind to IP address on interface IF + + +Stage can be one of: + +*ssh*:: + Obtain SSH keys from existing cluster node (requires -c <host>) + +*csync2*:: + Configure csync2 (requires -c <host>) + +*ssh_merge*:: + Merge root's SSH known_hosts across all nodes (csync2 must + already be configured). + +*cluster*:: + Start the cluster on this node + +If stage is not specified, each stage will be invoked in sequence. + +Usage: +............... +join [options] [STAGE] +............... + + +[[cmdhelp_cluster_remove,Remove node(s) from the cluster]] +==== `remove` + +Remove one or more nodes from the cluster. + +This command can remove the last node in the cluster, +thus effectively removing the whole cluster. To remove +the last node, pass `--force` argument to `crm` or set +the `config.core.force` option. + +Options: + +*-q, --quiet*:: + Be quiet (don't describe what's happening, just do it) + +*-y, --yes*:: + Answer "yes" to all prompts (use with caution) + +*-c HOST, --cluster-node=HOST*:: + IP address or hostname of cluster node which will be + removed from the cluster + +Usage: +............... +remove [options] [<node> ...] +............... + + +[[cmdhelp_cluster_run,Execute an arbitrary command on all nodes]] +==== `run` + +This command takes a shell statement as argument, executes that +statement on all nodes in the cluster, and reports the result. + +Usage: +............... +run <command> +............... + +Example: +............... +run "cat /proc/uptime" +............... + +[[cmdhelp_cluster_start,Start cluster services]] +==== `start` + +Starts the cluster-related system services on this node. + +Usage: +......... +start +......... + +[[cmdhelp_cluster_status,Cluster status check]] +==== `status` + +Reports the status for the cluster messaging layer on the local +node. + +Usage: +............... +status +............... + +[[cmdhelp_cluster_stop,Stop cluster services]] +==== `stop` + +Stops the cluster-related system services on this node. + +Usage: +......... +stop +......... + +[[cmdhelp_cluster_wait_for_startup,Wait for cluster to start]] +==== `wait_for_startup` + +Mostly useful in scripts or automated workflows, this command will +attempt to connect to the local cluster node repeatedly. The command +will keep trying until the cluster node responds, or the `timeout` +elapses. The timeout can be changed by supplying a value in seconds as +an argument. + +Usage: +........ +wait_for_startup +........ + +[[cmdhelp_script,Cluster script management]] +=== `script` - Cluster script management + +A big part of the configuration and management of a cluster is +collecting information about all cluster nodes and deploying changes +to those nodes. Often, just performing the same procedure on all nodes +will encounter problems, due to subtle differences in the +configuration. + +For example, when configuring a cluster for the first time, the +software needs to be installed and configured on all nodes before the +cluster software can be launched and configured using `crmsh`. This +process is cumbersome and error-prone, and the goal is for scripts to +make this process easier. + +Scripts are implemented using the python `parallax` package which +provides a thin wrapper on top of SSH. This allows the scripts to +function through the usual SSH channels used for system maintenance, +requiring no additional software to be installed or maintained. + +[[cmdhelp_script_json,JSON API for cluster scripts]] +==== `json` + +This command provides a JSON API for the cluster scripts, intended for +use in user interface tools that want to interact with the cluster via +scripts. + +The command takes a single argument, which should be a JSON array with +the first member identifying the command to perform. + +The output is line-based: Commands that return multiple results will +return them line-by-line, ending with a terminator value: "end". + +When providing parameter values to this command, they should be +provided as nested objects, so +virtual-ip:ip=192.168.0.5+ on the +command line becomes the JSON object ++{"virtual-ip":{"ip":"192.168.0.5"}}+. + +API: +........ +["list"] +=> [{name, shortdesc, category}] + +["show", <name>] +=> [{name, shortdesc, longdesc, category, <<steps>>}] + +<<steps>> := [{name, shortdesc], longdesc, required, parameters, steps}] + +<<params>> := [{name, shortdesc, longdesc, required, unique, advanced, + type, value, example}] + +["verify", <name>, <<values>>] +=> [{shortdesc, longdesc, text, nodes}] + +["run", <name>, <<values>>] +=> [{shortdesc, rc, output|error}] +........ + + +[[cmdhelp_script_list,List available scripts]] +==== `list` + +Lists the available scripts, sorted by category. Scripts that have the +special `Script` category are hidden by default, since they are mainly +used by other scripts or commands. To also show these, pass `all` as +argument. + +To get a flat list of script names, not sorted by category, pass +`names` as an extra argument. + +Usage: +............ +list [all] [names] +............ + +Example: +............ +list +list all names +............ + +[[cmdhelp_script_run,Run the script]] +==== `run` + +Given a list of parameter values, this command will execute the +actions specified by the cluster script. The format for the parameter +values is the same as for the `verify` command. + +Can optionally take at least two parameters: +* `nodes=<nodes>`: List of nodes that the script runs over +* `dry_run=yes|no`: If set, the script will not perform any modifications. + +Additional parameters may be available depending on the script. + +Use the `show` command to see what parameters are available. + +Usage: +............. +run <script> [args...] +............. + +Example: +............. +run apache install=true +run sbd id=sbd-1 node=node1 sbd_device=/dev/disk/by-uuid/F00D-CAFE +............. + +[[cmdhelp_script_show,Describe the script]] +==== `show` + +Prints a description and short summary of the script, with +descriptions of the accepted parameters. + +Advanced parameters are hidden by default. To show the complete list +of parameters accepted by the script, pass `all` as argument. + +Usage: +............ +show <script> [all] +............ + +Example: +............ +show virtual-ip +............ + +[[cmdhelp_script_verify,Verify the script]] +==== `verify` + +Checks the given parameter values, and returns a list +of actions that will be executed when running the script +if provided the same list of parameter values. + +Usage: +............ +verify <script> [args...] +............ + +Example: +............ +verify sbd id=sbd-1 node=node1 sbd_device=/dev/disk/by-uuid/F00D-CAFE +............ + +[[cmdhelp_corosync,Corosync management]] +=== `corosync` - Corosync management + +Corosync is the underlying messaging layer for most HA clusters. +This level provides commands for editing and managing the corosync +configuration. + +[[cmdhelp_corosync_add-node,Add a corosync node]] +==== `add-node` + +Adds a node to the corosync configuration. This is used with the `udpu` +type configuration in corosync. + +A nodeid for the added node is generated automatically. + +Note that this command assumes that only a single ring is used, and +sets only the address for ring0. + +Usage: +......... +add-node <addr> [name] +......... + +[[cmdhelp_corosync_del-node,Remove a corosync node]] +==== `del-node` + +Removes a node from the corosync configuration. The argument given is +the `ring0_addr` address set in the configuration file. + +Usage: +......... +del-node <addr> +......... + +[[cmdhelp_corosync_diff,Diffs the corosync configuration]] +==== `diff` + +Diffs the corosync configurations on different nodes. If no nodes are +given as arguments, the corosync configurations on all nodes in the +cluster are compared. + +`diff` takes an option argument `--checksum`, to display a checksum +for each file instead of calculating a diff. + +Usage: +......... +diff [--checksum] [node...] +......... + +[[cmdhelp_corosync_edit,Edit the corosync configuration]] +==== `edit` + +Opens the Corosync configuration file in an editor. + +Usage: +......... +edit +......... + +[[cmdhelp_corosync_get,Get a corosync configuration value]] +==== `get` + +Returns the value configured in `corosync.conf`, which is not +necessarily the value used in the running configuration. See `reload` +for telling corosync about configuration changes. + +The argument is the complete dot-separated path to the value. + +If there are multiple values configured with the same path, the +command returns all values for that path. For example, to get all +configured `ring0_addr` values, use this command: + +Example: +........ +get nodelist.node.ring0_addr +........ + +[[cmdhelp_corosync_log,Show the corosync log file]] +==== `log` + +Opens the log file specified in the corosync configuration file. If no +log file is configured, this command returns an error. + +The pager used can be configured either using the PAGER +environment variable or in `crm.conf`. + +Usage: +......... +log +......... + +[[cmdhelp_corosync_pull,Pulls the corosync configuration]] +==== `pull` + +Gets the corosync configuration from another node and copies +it to this node. + +Usage: +......... +pull <node> +......... + +[[cmdhelp_corosync_push,Push the corosync configuration]] +==== `push` + +Pushes the corosync configuration file on this node to +the list of nodes provided. If no target nodes are given, +the configuration is pushed to all other nodes in the cluster. + +It is recommended to use `csync2` to distribute the cluster +configuration files rather than relying on this command. + +Usage: +......... +push [node] ... +......... + +Example: +......... +push node-2 node-3 +......... + +[[cmdhelp_corosync_reload,Reload the corosync configuration]] +==== `reload` + +Tells all instances of corosync in this cluster to reload +`corosync.conf`. + +After pushing a new configuration to all cluster nodes, call this +command to make corosync use the new configuration. + +Usage: +......... +reload +......... + +[[cmdhelp_corosync_set,Set a corosync configuration value]] +==== `set` + +Sets the value identified by the given path. If the value does not +exist in the configuration file, it will be added. However, if the +section containing the value does not exist, the command will fail. + +Usage: +......... +set quorum.expected_votes 2 +......... + +[[cmdhelp_corosync_show,Display the corosync configuration]] +==== `show` + +Displays the corosync configuration on the current node. + +......... +show +......... + +[[cmdhelp_corosync_status,Display the corosync status]] +==== `status` + +Displays the status of Corosync, including the votequorum state. + +Usage: +......... +status +......... + +[[cmdhelp_cib,CIB shadow management]] +=== `cib` - CIB shadow management + +This level is for management of shadow CIBs. It is available both +at the top level and the `configure` level. + +All the commands are implemented using `cib_shadow(8)` and the +`CIB_shadow` environment variable. The user prompt always +includes the name of the currently active shadow or the live CIB. + +[[cmdhelp_cib_cibstatus,CIB status management and editing]] +==== `cibstatus` + +Enter edit and manage the CIB status section level. See the +<<cmdhelp_cibstatus,CIB status management section>>. + +[[cmdhelp_cib_commit,copy a shadow CIB to the cluster]] +==== `commit` + +Apply a shadow CIB to the cluster. If the shadow name is omitted +then the current shadow CIB is applied. + +Temporary shadow CIBs are removed automatically on commit. + +Usage: +............... +commit [<cib>] +............... + +[[cmdhelp_cib_delete,delete a shadow CIB]] +==== `delete` + +Delete an existing shadow CIB. + +Usage: +............... +delete <cib> +............... + +[[cmdhelp_cib_diff,diff between the shadow CIB and the live CIB]] +==== `diff` + +Print differences between the current cluster configuration and +the active shadow CIB. + +Usage: +............... +diff +............... + +[[cmdhelp_cib_import,import a CIB or PE input file to a shadow]] +==== `import` + +At times it may be useful to create a shadow file from the +existing CIB. The CIB may be specified as file or as a PE input +file number. The shell will look up files in the local directory +first and then in the PE directory (typically `/var/lib/pengine`). +Once the CIB file is found, it is copied to a shadow and this +shadow is immediately available for use at both `configure` and +`cibstatus` levels. + +If the shadow name is omitted then the target shadow is named +after the input CIB file. + +Note that there are often more than one PE input file, so you may +need to specify the full name. + +Usage: +............... +import {<file>|<number>} [<shadow>] +............... +Examples: +............... +import pe-warn-2222 +import 2289 issue2 +............... + +[[cmdhelp_cib_list,list all shadow CIBs]] +==== `list` + +List existing shadow CIBs. + +Usage: +............... +list +............... + +[[cmdhelp_cib_new,create a new shadow CIB]] +==== `new` + +Create a new shadow CIB. The live cluster configuration and +status is copied to the shadow CIB. + +If the name of the shadow is omitted, we create a temporary CIB +shadow. It is useful if multiple level sessions are desired +without affecting the cluster. A temporary CIB shadow is short +lived and will be removed either on `commit` or on program exit. +Note that if the temporary shadow is not committed all changes in +the temporary shadow are lost. + +Specify `withstatus` if you want to edit the status section of +the shadow CIB (see the <<cmdhelp_cibstatus,cibstatus section>>). +Add `force` to force overwriting the existing shadow CIB. + +To start with an empty configuration that is not copied from the live +CIB, specify the `empty` keyword. (This also allows a shadow CIB to be +created in case no cluster is running.) + +Usage: +............... +new [<cib>] [withstatus] [force] [empty] +............... + +[[cmdhelp_cib_reset,copy live cib to a shadow CIB]] +==== `reset` + +Copy the current cluster configuration into the shadow CIB. + +Usage: +............... +reset <cib> +............... + +[[cmdhelp_cib_use,change working CIB]] +==== `use` + +Choose a CIB source. If you want to edit the status from the +shadow CIB specify `withstatus` (see <<cmdhelp_cibstatus,`cibstatus`>>). +Leave out the CIB name to switch to the running CIB. + +Usage: +............... +use [<cib>] [withstatus] +............... + +[[cmdhelp_ra,Resource Agents (RA) lists and documentation]] +=== `ra` - Resource Agents (RA) lists and documentation + +This level contains commands which show various information about +the installed resource agents. It is available both at the top +level and at the `configure` level. + +[[cmdhelp_ra_classes,list classes and providers]] +==== `classes` + +Print all resource agents' classes and, where appropriate, a list +of available providers. + +Usage: +............... +classes +............... + +[[cmdhelp_ra_info,show meta data for a RA]] +==== `info` (`meta`) + +Show the meta-data of a resource agent type. This is where users +can find information on how to use a resource agent. It is also +possible to get information from some programs: `pengine`, +`crmd`, `cib`, and `stonithd`. Just specify the program name +instead of an RA. + +Usage: +............... +info [<class>:[<provider>:]]<type> +info <type> <class> [<provider>] (obsolete) +............... +Example: +............... +info apache +info ocf:pacemaker:Dummy +info stonith:ipmilan +info pengine +............... + +[[cmdhelp_ra_list,list RA for a class (and provider)]] +==== `list` + +List available resource agents for the given class. If the class +is `ocf`, supply a provider to get agents which are available +only from that provider. + +Usage: +............... +list <class> [<provider>] +............... +Example: +............... +list ocf pacemaker +............... + +[[cmdhelp_ra_providers,show providers for a RA and a class]] +==== `providers` + +List providers for a resource agent type. The class parameter +defaults to `ocf`. + +Usage: +............... +providers <type> [<class>] +............... +Example: +............... +providers apache +............... + +[[cmdhelp_ra_validate,validate parameters for RA]] +==== `validate` + +If the resource agent supports the `validate-all` action, this calls +the action with the given parameters, printing any warnings or errors +reported by the agent. + +Usage: +................ +validate <agent> [<key>=<value> ...] +................ + +[[cmdhelp_resource,Resource management]] +=== `resource` - Resource management + +At this level resources may be managed. + +All (or almost all) commands are implemented with the CRM tools +such as `crm_resource(8)`. + +[[cmdhelp_resource_ban,ban a resource from a node]] +==== `ban` + +Ban a resource from running on a certain node. If no node is given +as argument, the resource is banned from the current location. + +See `move` for details on other arguments. + +Usage: +............... +ban <rsc> [<node>] [<lifetime>] [force] +............... + +[[cmdhelp_resource_cleanup,cleanup resource status]] +==== `cleanup` + +Cleanup resource status. Typically done after the resource has +temporarily failed. If a node is omitted, cleanup on all nodes. +If there are many nodes, the command may take a while. + ++(Pacemaker 1.1.14)+ Pass force to cleanup the resource itself, +otherwise the cleanup command will apply to the parent resource (if +any). + +Usage: +............... +cleanup <rsc> [<node>] [force] +............... + +[[cmdhelp_resource_clear,Clear any relocation constraint]] +==== `clear` (`unmove`, `unmigrate`, `unban`) + +Remove any relocation constraint created by +the `move`, `migrate` or `ban` command. + +Usage: +............... +clear <rsc> +unmigrate <rsc> +unban <rsc> +............... + +[[cmdhelp_resource_constraints,Show constraints affecting a resource]] +==== `constraints` + +Display the location and colocation constraints affecting the +resource. + +Usage: +................ +constraints <rsc> +................ + +[[cmdhelp_resource_demote,demote a master-slave resource]] +==== `demote` + +Demote a master-slave resource using the `target-role` +attribute. + +Usage: +............... +demote <rsc> +............... + +[[cmdhelp_resource_failcount,manage failcounts]] +==== `failcount` + +Show/edit/delete the failcount of a resource. + +Usage: +............... +failcount <rsc> set <node> <value> +failcount <rsc> delete <node> +failcount <rsc> show <node> +............... +Example: +............... +failcount fs_0 delete node2 +............... + +[[cmdhelp_resource_locate,show the location of resources]] +==== `locate` + +Show the current location of one or more resources. + +Usage: +............... +locate [<rsc> ...] +............... + +[[cmdhelp_resource_maintenance,Enable/disable per-resource maintenance mode]] +==== `maintenance` + +Enables or disables the per-resource maintenance mode. When this mode +is enabled, no monitor operations will be triggered for the resource. +`maintenance` attribute conflicts with the `is-managed`. When setting +the `maintenance` attribute, the user is proposed to remove the +`is-managed` attribute if it exists. + +Usage: +.................. +maintenance <resource> [on|off|true|false] +.................. + +Example: +.................. +maintenance rsc1 +maintenance rsc2 off +.................. + +[[cmdhelp_resource_manage,put a resource into managed mode]] +==== `manage` + +Manage a resource using the `is-managed` attribute. If there +are multiple meta attributes sets, the attribute is set in all of +them. If the resource is a clone, all `is-managed` attributes are +removed from the children resources. +`is-managed` attribute conflicts with the `maintenance`. When setting +the `is-managed` attribute, the user is proposed to remove the +`maintenance` attribute if it exists. + +For details on group management see <<cmdhelp_options_manage-children,`options manage-children`>>. + +Usage: +............... +manage <rsc> +............... + +[[cmdhelp_resource_meta,manage a meta attribute]] +==== `meta` + +Show/edit/delete a meta attribute of a resource. Currently, all +meta attributes of a resource may be managed with other commands +such as `resource stop`. + +Usage: +............... +meta <rsc> set <attr> <value> +meta <rsc> delete <attr> +meta <rsc> show <attr> +............... +Example: +............... +meta ip_0 set target-role stopped +............... + +[[cmdhelp_resource_move,Move a resource to another node]] +==== `move` (`migrate`) + +Move a resource away from its current location. + +If the destination node is left out, the resource is migrated by +creating a constraint which prevents it from running on the current +node. For this type of constraint to be created, the +force+ argument +is required. + +A lifetime may be given for the constraint. Once it expires, the +location constraint will no longer be active. + +Usage: +............... +move <rsc> [<node>] [<lifetime>] [force] +............... + +[[cmdhelp_resource_operations,Show active resource operations]] +==== `operations` + +Show active operations, optionally filtered by resource and node. + +Usage: +................ +operations [<rsc>] [<node>] +................ + +[[cmdhelp_resource_param,manage a parameter of a resource]] +==== `param` + +Show/edit/delete a parameter of a resource. + +Usage: +............... +param <rsc> set <param> <value> +param <rsc> delete <param> +param <rsc> show <param> +............... +Example: +............... +param ip_0 show ip +............... + +[[cmdhelp_resource_promote,promote a master-slave resource]] +==== `promote` + +Promote a master-slave resource using the `target-role` +attribute. + +Usage: +............... +promote <rsc> +............... + +[[cmdhelp_resource_refresh,refresh CIB from the LRM status]] +==== `refresh` + +Refresh CIB from the LRM status. + +.Note +**************************** +`refresh` has been deprecated and is now +an alias for `cleanup`. +**************************** + +Usage: +............... +refresh [<node>] +............... + +[[cmdhelp_resource_reprobe,probe for resources not started by the CRM]] +==== `reprobe` + +Probe for resources not started by the CRM. + +.Note +**************************** +`reprobe` has been deprecated and is now +an alias for `cleanup`. +**************************** + +Usage: +............... +reprobe [<node>] +............... + +[[cmdhelp_resource_restart,restart resources]] +==== `restart` + +Restart one or more resources. This is essentially a shortcut for +resource stop followed by a start. The shell is first going to wait +for the stop to finish, that is for all resources to really stop, and +only then to order the start action. Due to this command +entailing a whole set of operations, informational messages are +printed to let the user see some progress. + +For details on group management see +<<cmdhelp_options_manage-children,`options manage-children`>>. + +Usage: +............... +restart <rsc> [<rsc> ...] +............... +Example: +............... +# crm resource restart g_webserver +INFO: ordering g_webserver to stop +waiting for stop to finish .... done +INFO: ordering g_webserver to start +# +............... + +[[cmdhelp_resource_scores,Display resource scores]] +==== `scores` + +Display the allocation scores for all resources. + +Usage: +................ +scores +................ + +[[cmdhelp_resource_secret,manage sensitive parameters]] +==== `secret` + +Sensitive parameters can be kept in local files rather than CIB +in order to prevent accidental data exposure. Use the `secret` +command to manage such parameters. `stash` and `unstash` move the +value from the CIB and back to the CIB respectively. The `set` +subcommand sets the parameter to the provided value. `delete` +removes the parameter completely. `show` displays the value of +the parameter from the local file. Use `check` to verify if the +local file content is valid. + +Usage: +............... +secret <rsc> set <param> <value> +secret <rsc> stash <param> +secret <rsc> unstash <param> +secret <rsc> delete <param> +secret <rsc> show <param> +secret <rsc> check <param> +............... +Example: +............... +secret fence_1 show password +secret fence_1 stash password +secret fence_1 set password secret_value +............... + +[[cmdhelp_resource_start,start resources]] +==== `start` + +Start one or more resources by setting the `target-role` attribute. If +there are multiple meta attributes sets, the attribute is set in all +of them. If the resource is a clone, all `target-role` attributes are +removed from the children resources. + +For details on group management see +<<cmdhelp_options_manage-children,`options manage-children`>>. + +Usage: +............... +start <rsc> [<rsc> ...] +............... + +[[cmdhelp_resource_status,show status of resources]] +==== `status` (`show`, `list`) + +Print resource status. More than one resource can be shown at once. If +the resource parameter is left out, the status of all resources is +printed. + +Usage: +............... +status [<rsc> ...] +............... + +[[cmdhelp_resource_stop,stop resources]] +==== `stop` + +Stop one or more resources using the `target-role` attribute. If there +are multiple meta attributes sets, the attribute is set in all of +them. If the resource is a clone, all `target-role` attributes are +removed from the children resources. + +For details on group management see +<<cmdhelp_options_manage-children,`options manage-children`>>. + +Usage: +............... +stop <rsc> [<rsc> ...] +............... + +[[cmdhelp_resource_trace,start RA tracing]] +==== `trace` + +Start tracing RA for the given operation. The trace files are +stored in `$HA_VARLIB/trace_ra`. If the operation to be traced is +monitor, note that the number of trace files can grow very +quickly. + +If no operation name is given, crmsh will attempt to trace all +operations for the RA. This includes any configured operations, start +and stop as well as promote/demote for multistate resources. + +To trace the probe operation which exists for all resources, either +set a trace for `monitor` with interval `0`, or use `probe` as the +operation name. + +Usage: +............... +trace <rsc> [<op> [<interval>] ] +............... +Example: +............... +trace fs start +trace webserver +trace webserver probe +trace fs monitor 0 +............... + +[[cmdhelp_resource_unmanage,put a resource into unmanaged mode]] +==== `unmanage` + +Unmanage a resource using the `is-managed` attribute. If there +are multiple meta attributes sets, the attribute is set in all of +them. If the resource is a clone, all `is-managed` attributes are +removed from the children resources. + +For details on group management see <<cmdhelp_options_manage-children,`options manage-children`>>. + +Usage: +............... +unmanage <rsc> +............... + +[[cmdhelp_resource_untrace,stop RA tracing]] +==== `untrace` + +Stop tracing RA for the given operation. If no operation name is +given, crmsh will attempt to stop tracing all operations in resource. + +Usage: +............... +untrace <rsc> [<op> [<interval>] ] +............... +Example: +............... +untrace fs start +untrace webserver +............... + +[[cmdhelp_resource_utilization,manage a utilization attribute]] +==== `utilization` + +Show/edit/delete a utilization attribute of a resource. These +attributes describe hardware requirements. By setting the +`placement-strategy` cluster property appropriately, it is +possible then to distribute resources based on resource +requirements and node size. See also <<cmdhelp_node_utilization,node utilization attributes>>. + +Usage: +............... +utilization <rsc> set <attr> <value> +utilization <rsc> delete <attr> +utilization <rsc> show <attr> +............... +Example: +............... +utilization xen1 set memory 4096 +............... + +[[cmdhelp_node,Node management]] +=== `node` - Node management + +Node management and status commands. + +[[cmdhelp_node_attribute,manage attributes]] +==== `attribute` + +Edit node attributes. This kind of attribute should refer to +relatively static properties, such as memory size. + +Usage: +............... +attribute <node> set <attr> <value> +attribute <node> delete <attr> +attribute <node> show <attr> +............... +Example: +............... +attribute node_1 set memory_size 4096 +............... + +[[cmdhelp_node_clearstate,Clear node state]] +==== `clearstate` + +Resets and clears the state of the specified node. This node is +afterwards assumed clean and offline. This command can be used to +manually confirm that a node has been fenced (e.g., powered off). + +Be careful! This can cause data corruption if you confirm that a node is +down that is, in fact, not cleanly down - the cluster will proceed as if +the fence had succeeded, possibly starting resources multiple times. + +Usage: +............... +clearstate <node> +............... + +[[cmdhelp_node_delete,delete node]] +==== `delete` + +Delete a node. This command will remove the node from the CIB +and, in case the cluster stack is running, use the appropriate +program (`crm_node` or `hb_delnode`) to remove the node from the +membership. + +If the node is still listed as active and a member of our +partition we refuse to remove it. With the global force option +(`-F`) we will try to delete the node anyway. + +Usage: +............... +delete <node> +............... + +[[cmdhelp_node_fence,fence node]] +==== `fence` + +Make CRM fence a node. This functionality depends on stonith +resources capable of fencing the specified node. No such stonith +resources, no fencing will happen. + +Usage: +............... +fence <node> +............... + +[[cmdhelp_node_maintenance,put node into maintenance mode]] +==== `maintenance` + +Set the node status to maintenance. This is equivalent to the +cluster-wide `maintenance-mode` property but puts just one node +into the maintenance mode. If there are maintenaned resources on +the node, the user will be proposed to remove the maintenance +property from them. + +The node parameter defaults to the node where the command is run. + +Usage: +............... +maintenance [<node>] +............... + +[[cmdhelp_node_online,set node online]] +==== `online` + +Set a node to online status. + +The node parameter defaults to the node where the command is run. + +Usage: +............... +online [<node>] +............... + +[[cmdhelp_node_ready,put node into ready mode]] +==== `ready` + +Set the node's maintenance status to `off`. The node should be +now again fully operational and capable of running resource +operations. + +The node parameter defaults to the node where the command is run. + +Usage: +............... +ready [<node>] +............... + +[[cmdhelp_node_server,show node hostname or server address]] +==== `server` + +Remote nodes may have a configured server address which should +be used when contacting the node. This command prints the +server address if configured, else the node name. + +If no parameter is given, the addresses or names for all nodes +are printed. + +Usage: +............... +server [<node> ...] +............... + +[[cmdhelp_node_show,show node]] +==== `show` + +Show a node definition. If the node parameter is omitted then all +nodes are shown. + +Usage: +............... +show [<node>] +............... + +[[cmdhelp_node_standby,put node into standby]] +==== `standby` + +Set a node to standby status. The node parameter defaults to the +node where the command is run. + +Additionally, you may specify a lifetime for the standby---if set to +`reboot`, the node will be back online once it reboots. `forever` will +keep the node in standby after reboot. The life time defaults to +`forever`. + +Usage: +............... +standby [<node>] [<lifetime>] + +lifetime :: reboot | forever +............... + +Example: +............... +standby bob reboot +............... + + +[[cmdhelp_node_status,show nodes' status as XML]] +==== `status` + +Show nodes' status as XML. If the node parameter is omitted then +all nodes are shown. + +Usage: +............... +status [<node>] +............... + +[[cmdhelp_node_status-attr,manage status attributes]] +==== `status-attr` + +Edit node attributes which are in the CIB status section, i.e. +attributes which hold properties of a more volatile nature. One +typical example is attribute generated by the `pingd` utility. + +Usage: +............... +status-attr <node> set <attr> <value> +status-attr <node> delete <attr> +status-attr <node> show <attr> +............... +Example: +............... +status-attr node_1 show pingd +............... + +[[cmdhelp_node_utilization,manage utilization attributes]] +==== `utilization` + +Edit node utilization attributes. These attributes describe +hardware characteristics as integer numbers such as memory size +or the number of CPUs. By setting the `placement-strategy` +cluster property appropriately, it is possible then to distribute +resources based on resource requirements and node size. See also +<<cmdhelp_resource_utilization,resource utilization attributes>>. + +Usage: +............... +utilization <node> set <attr> <value> +utilization <node> delete <attr> +utilization <node> show <attr> +............... +Examples: +............... +utilization node_1 set memory 16384 +utilization node_1 show cpu +............... + +[[cmdhelp_site,GEO clustering site support]] +=== `site` - GEO clustering site support + +A cluster may consist of two or more subclusters in different and +distant locations. This set of commands supports such setups. + +[[cmdhelp_site_ticket,manage site tickets]] +==== `ticket` + +Tickets are cluster-wide attributes. They can be managed at the +site where this command is executed. + +It is then possible to constrain resources depending on the +ticket availability (see the <<cmdhelp_configure_rsc_ticket,`rsc_ticket`>> command +for more details). + +Usage: +............... +ticket {grant|revoke|standby|activate|show|time|delete} <ticket> +............... +Example: +............... +ticket grant ticket1 +............... + +[[cmdhelp_options,User preferences]] +=== `options` - User preferences + +The user may set various options for the crm shell itself. + +[[cmdhelp_options_add-quotes,add quotes around parameters containing spaces]] +==== `add-quotes` + +The shell (as in `/bin/sh`) parser strips quotes from the command +line. This may sometimes make it really difficult to type values +which contain white space. One typical example is the configure +filter command. The crm shell will supply extra quotes around +arguments which contain white space. The default is `yes`. + +.Note on quotes use +**************************** +Adding quotes around arguments automatically has been introduced +with version 1.2.2 and it is technically a regression. Being a +regression is the only reason the `add-quotes` option exists. If +you have custom shell scripts which would break, just set the +`add-quotes` option to `no`. + +For instance, with adding quotes enabled, it is possible to do +the following: +............... +# crm configure primitive d1 Dummy \ + meta description="some description here" +# crm configure filter 'sed "s/hostlist=./&node-c /"' fencing +............... +**************************** + +[[cmdhelp_options_check-frequency,when to perform semantic check]] +==== `check-frequency` + +Semantic check of the CIB or elements modified or created may be +done on every configuration change (`always`), when verifying +(`on-verify`) or `never`. It is by default set to `always`. +Experts may want to change the setting to `on-verify`. + +The checks require that resource agents are present. If they are +not installed at the configuration time set this preference to +`never`. + +See <<topics_Features_Checks,Configuration semantic checks>> for more details. + +[[cmdhelp_options_check-mode,how to treat semantic errors]] +==== `check-mode` + +Semantic check of the CIB or elements modified or created may be +done in the `strict` mode or in the `relaxed` mode. In the former +certain problems are treated as configuration errors. In the +`relaxed` mode all are treated as warnings. The default is `strict`. + +See <<topics_Features_Checks,Configuration semantic checks>> for more details. + +[[cmdhelp_options_colorscheme,set colors for output]] +==== `colorscheme` + +With `output` set to `color`, a comma separated list of colors +from this option are used to emphasize: + +- keywords +- object ids +- attribute names +- attribute values +- scores +- resource references + +`crm` can show colors only if there is curses support for python +installed (usually provided by the `python-curses` package). The +colors are whatever is available in your terminal. Use `normal` +if you want to keep the default foreground color. + +This user preference defaults to +`yellow,normal,cyan,red,green,magenta` which is good for +terminals with dark background. You may want to change the color +scheme and save it in the preferences file for other color +setups. + +Example: +............... +colorscheme yellow,normal,blue,red,green,magenta +............... + +[[cmdhelp_options_editor,set preferred editor program]] +==== `editor` + +The `edit` command invokes an editor. Use this to specify your +preferred editor program. If not set, it will default to either +the value of the `EDITOR` environment variable or to one of the +standard UNIX editors (`vi`,`emacs`,`nano`). + +Usage: +............... +editor program +............... +Example: +............... +editor vim +............... + +[[cmdhelp_options_manage-children,how to handle children resource attributes]] +==== `manage-children` + +Some resource management commands, such as `resource stop`, when +the target resource is a group, may not always produce desired +result. Each element, group and the primitive members, can have a +meta attribute and those attributes may end up with conflicting +values. Consider the following construct: +............... +crm(live)# configure show svc fs virtual-ip +primitive fs Filesystem \ + params device="/dev/drbd0" directory="/srv/nfs" fstype=ext3 \ + op monitor interval=10s \ + meta target-role=Started +primitive virtual-ip IPaddr2 \ + params ip=10.2.13.110 iflabel=1 \ + op monitor interval=10s \ + op start interval=0 \ + meta target-role=Started +group svc fs virtual-ip \ + meta target-role=Stopped +............... + +Even though the element +svc+ should be stopped, the group is +actually running because all its members have the +target-role+ +set to +Started+: +............... +crm(live)# resource show svc +resource svc is running on: xen-f +............... + +Hence, if the user invokes +resource stop svc+ the intention is +not clear. This preference gives the user an opportunity to +better control what happens if attributes of group members have +values which are in conflict with the same attribute of the group +itself. + +Possible values are +ask+ (the default), +always+, and +never+. +If set to +always+, the crm shell removes all children attributes +which have values different from the parent. If set to +never+, +all children attributes are left intact. Finally, if set to ++ask+, the user will be asked for each member what is to be done. + +[[cmdhelp_options_output,set output type]] +==== `output` + +`crm` can adorn configurations in two ways: in color (similar to +for instance the `ls --color` command) and by showing keywords in +upper case. Possible values are `plain`, `color-always`, `color`, +and 'uppercase'. It is possible to combine `uppercase` with one +of the color values in order to get an upper case xmass tree. Just +set this option to `color,uppercase` or `color-always,uppercase`. +In case you need color codes in pipes, `color-always` forces color +codes even in case the terminal is not a tty (just like `ls +--color=always`). + +[[cmdhelp_options_pager,set preferred pager program]] +==== `pager` + +The `view` command displays text through a pager. Use this to +specify your preferred pager program. If not set, it will default +to either the value of the `PAGER` environment variable or to one +of the standard UNIX system pagers (`less`,`more`,`pg`). + +[[cmdhelp_options_reset,reset user preferences to factory defaults]] +==== `reset` + +This command resets all user options to the defaults. If used as +a single-shot command, the rc file (+$HOME/.config/crm/rc+) is +reset to the defaults too. + +[[cmdhelp_options_save,save the user preferences to the rc file]] +==== `save` + +Save current settings to the rc file (+$HOME/.config/crm/rc+). On +further `crm` runs, the rc file is automatically read and parsed. + +[[cmdhelp_options_set,Set the value of a given option]] +==== `set` + +Sets the value of an option. Takes the fully qualified +name of the option as argument, as displayed by +show all+. + +The modified option value is stored in the user-local +configuration file, usually found in +~/.config/crm/crm.conf+. + +Usage: +........ +set <option> <value> +........ + +Example: +........ +set color.warn "magenta bold" +set editor nano +........ + +[[cmdhelp_options_show,show current user preference]] +==== `show` + +Display all current settings. + +Given an option name as argument, `show` will display only the value +of that argument. + +Given +all+ as argument, `show` displays all available user options. + +Usage: +........ +show [all|<option>] +........ + +Example: +........ +show +show skill-level +show all +........ + +[[cmdhelp_options_skill-level,set skill level]] +==== `skill-level` + +Based on the skill-level setting, the user is allowed to use only +a subset of commands. There are three levels: operator, +administrator, and expert. The operator level allows only +commands at the `resource` and `node` levels, but not editing +or deleting resources. The administrator may do that and may also +configure the cluster at the `configure` level and manage the +shadow CIBs. The expert may do all. + +Usage: +............... +skill-level <level> + +level :: operator | administrator | expert +............... + +.Note on security +**************************** +The `skill-level` option is advisory only. There is nothing +stopping any users change their skill level (see +<<topics_Features_Security,Access Control Lists (ACL)>> on how to enforce +access control). +**************************** + +[[cmdhelp_options_sort-elements,sort CIB elements]] +==== `sort-elements` + +`crm` by default sorts CIB elements. If you want them appear in +the order they were created, set this option to `no`. + +Usage: +............... +sort-elements {yes|no} +............... +Example: +............... +sort-elements no +............... + +[[cmdhelp_options_user,set the cluster user]] +==== `user` + +Sufficient privileges are necessary in order to manage a +cluster: programs such as `crm_verify` or `crm_resource` and, +ultimately, `cibadmin` have to be run either as `root` or as the +CRM owner user (typically `hacluster`). You don't have to worry +about that if you run `crm` as `root`. A more secure way is to +run the program with your usual privileges, set this option to +the appropriate user (such as `hacluster`), and setup the +`sudoers` file. + +Usage: +............... +user system-user +............... +Example: +............... +user hacluster +............... + +[[cmdhelp_options_wait,synchronous operation]] +==== `wait` + +In normal operation, `crm` runs a command and gets back +immediately to process other commands or get input from the user. +With this option set to `yes` it will wait for the started +transition to finish. In interactive mode dots are printed to +indicate progress. + +Usage: +............... +wait {yes|no} +............... +Example: +............... +wait yes +............... + +[[cmdhelp_configure,CIB configuration]] +=== `configure` - CIB configuration + +This level enables all CIB object definition commands. + +The configuration may be logically divided into four parts: +nodes, resources, constraints, and (cluster) properties and +attributes. Each of these commands support one or more basic CIB +objects. + +Nodes and attributes describing nodes are managed using the +`node` command. + +Commands for resources are: + +- `primitive` +- `monitor` +- `group` +- `clone` +- `ms`/`master` (master-slave) + +In order to streamline large configurations, it is possible to +define a template which can later be referenced in primitives: + +- `rsc_template` + +In that case the primitive inherits all attributes defined in the +template. + +There are three types of constraints: + +- `location` +- `colocation` +- `order` + +It is possible to define fencing order (stonith resource +priorities): + +- `fencing_topology` + +Finally, there are the cluster properties, resource meta +attributes defaults, and operations defaults. All are just a set +of attributes. These attributes are managed by the following +commands: + +- `property` +- `rsc_defaults` +- `op_defaults` + +In addition to the cluster configuration, the Access Control +Lists (ACL) can be setup to allow access to parts of the CIB for +users other than +root+ and +hacluster+. The following commands +manage ACL: + +- `user` +- `role` + +In Pacemaker 1.1.12 and up, this command replaces the `user` command +for handling ACLs: + +- `acl_target` + +The changes are applied to the current CIB only on ending the +configuration session or using the `commit` command. + +Comments start with +#+ in the first line. The comments are tied +to the element which follows. If the element moves, its comments +will follow. + +[[cmdhelp_configure_acl_target,Define target access rights]] +==== `acl_target` + +Defines an ACL target. + +Usage: +................ +acl_target <tid> [<role> ...] +................ +Example: +................ +acl_target joe resource_admin constraint_editor +................ + +[[cmdhelp_configure_alert,Event-driven alerts]] +==== `alert` + +.Version note +**************************** +This feature is only available +in Pacemaker 1.1.15+. +**************************** + +Event-driven alerts enables calling scripts whenever interesting +events occur in the cluster (nodes joining or leaving, resources +starting or stopping, etc.). + +The +path+ is an arbitrary file path to an alert script. Existing +external scripts used with ClusterMon resources can be used as alert +scripts, since the interface is compatible. + +Each alert may have a number of receipients configured. These will be +passed to the script as arguments. The first recipient will also be +passed as the +CRM_alert_recipient+ environment variable, for +compatibility with existing scripts that only support one recipient. + +The available meta attributes are +timeout+ (default 30s) and ++timestamp-format+ (default `"%H:%M:%S.%06N"`). + +Some configurations may require each recipient to be delimited by +brackets, to avoid ambiguity. In the example +alert-2+ below, the meta +attribute for `timeout` is defined after the recipient, so the +brackets are used to ensure that the meta attribute is set for the +alert and not just the recipient. This can be avoided by setting any +alert attributes before defining the recipients. + +Usage: +............... +alert <id> <path> \ + [attributes <nvpair> ...] \ + [meta <nvpair> ...] \ + [to [{] <recipient> + [attributes <nvpair> ...] \ + [meta <nvpair> ...] [}] \ + ...] +............... + +Example: +............... +alert alert-1 /srv/pacemaker/pcmk_alert_sample.sh \ + to /var/log/cluster-alerts.log + +alert alert-2 /srv/pacemaker/example_alert.sh \ + meta timeout=60s \ + to { /var/log/cluster-alerts.log } +............... + +[[cmdhelp_configure_cib,CIB shadow management]] +==== `cib` + +This level is for management of shadow CIBs. It is available at +the `configure` level to enable saving intermediate changes to a +shadow CIB instead of to the live cluster. This short excerpt +shows how: +............... +crm(live)configure# cib new test-2 +INFO: test-2 shadow CIB created +crm(test-2)configure# commit +............... +Note how the current CIB in the prompt changed from +live+ to ++test-2+ after issuing the `cib new` command. See also the +<<cmdhelp_cib,CIB shadow management>> for more information. + +[[cmdhelp_configure_cibstatus,CIB status management and editing]] +==== `cibstatus` + +Enter edit and manage the CIB status section level. See the +<<cmdhelp_cibstatus,CIB status management section>>. + +[[cmdhelp_configure_clone,define a clone]] +==== `clone` + +The `clone` command creates a resource clone. It may contain a +single primitive resource or one group of resources. + +Usage: +............... +clone <name> <rsc> + [description=<description>] + [meta <attr_list>] + [params <attr_list>] + +attr_list :: [$id=<id>] <attr>=<val> [<attr>=<val>...] | $id-ref=<id> +............... +Example: +............... +clone cl_fence apc_1 \ + meta clone-node-max=1 globally-unique=false +............... + +[[cmdhelp_configure_colocation,colocate resources]] +==== `colocation` (`collocation`) + +This constraint expresses the placement relation between two +or more resources. If there are more than two resources, then the +constraint is called a resource set. + +The score is used to indicate the priority of the constraint. A +positive score indicates that the resources should run on the same +node. A negative score that they should not run on the same +node. Values of positive or negative +infinity+ indicate a mandatory +constraint. + +In the two resource form, the cluster will place +<with-rsc>+ first, +and then decide where to put the +<rsc>+ resource. + +Collocation resource sets have an extra attribute (+sequential+) +to allow for sets of resources which don't depend on each other +in terms of state. The shell syntax for such sets is to put +resources in parentheses. + +Sets cannot be nested. + +The optional +node-attribute+ can be used to colocate resources on a +set of nodes and not necessarily on the same node. For example, by +setting a node attribute +color+ on all nodes and setting the ++node-attribute+ value to +color+ as well, the colocated resources +will be placed on any node that has the same color. + +For more details on how to configure resource sets, see +<<topics_Features_Resourcesets,`Syntax: Resource sets`>>. + +Usage: +............... +colocation <id> <score>: <rsc>[:<role>] <with-rsc>[:<role>] + [node-attribute=<node_attr>] + +colocation <id> <score>: <resource_sets> + [node-attribute=<node_attr>] + +resource_sets :: <resource_set> [<resource_set> ...] + +resource_set :: ["("|"["] <rsc>[:<role>] [<rsc>[:<role>] ...] \ + [<attributes>] [")"|"]"] + +attributes :: [require-all=(true|false)] [sequential=(true|false)] + +............... +Example: +............... +colocation never_put_apache_with_dummy -inf: apache dummy +colocation c1 inf: A ( B C ) +............... + +[[cmdhelp_configure_commit,commit the changes to the CIB]] +==== `commit` + +Commit the current configuration to the CIB in use. As noted +elsewhere, commands in a configure session don't have immediate +effect on the CIB. All changes are applied at one point in time, +either using `commit` or when the user leaves the configure +level. In case the CIB in use changed in the meantime, presumably +by somebody else, the crm shell will refuse to apply the changes. + +If you know that it's fine to still apply them, add +force+ to the +command line. + +To disable CIB patching and apply the changes by replacing the CIB +completely, add +replace+ to the command line. Note that this can lead +to previous changes being overwritten if some other process +concurrently modifies the CIB. + +Usage: +............... +commit [force] [replace] +............... + +[[cmdhelp_configure_default-timeouts,set timeouts for operations to minimums from the meta-data]] +==== `default-timeouts` + +This command takes the timeouts from the actions section of the +resource agent meta-data and sets them for the operations of the +primitive. + +Usage: +............... +default-timeouts <id> [<id>...] +............... + +.Note on `default-timeouts` +**************************** +The use of this command is discouraged in favor of manually +determining the best timeouts required for the particular +configuration. Relying on the resource agent to supply appropriate +timeouts can cause the resource to fail at the worst possible moment. + +Appropriate timeouts for resource actions are context-sensitive, and +should be carefully considered with the whole configuration in mind. +**************************** + +[[cmdhelp_configure_delete,delete CIB objects]] +==== `delete` + +Delete one or more objects. If an object to be deleted belongs to +a container object, such as a group, and it is the only resource +in that container, then the container is deleted as well. Any +related constraints are removed as well. + +If the object is a started resource, it will not be deleted unless the ++--force+ flag is passed to the command, or the +force+ option is set. + +Usage: +............... +delete [--force] <id> [<id>...] +............... + +[[cmdhelp_configure_edit,edit CIB objects]] +==== `edit` + +This command invokes the editor with the object description. As +with the `show` command, the user may choose to edit all objects +or a set of objects. + +If the user insists, he or she may edit the XML edition of the +object. If you do that, don't modify any id attributes. + +Usage: +............... +edit [xml] [<id> ...] +edit [xml] changed +............... + +.Note on renaming element ids +**************************** +The edit command sometimes cannot properly handle modifying +element ids. In particular for elements which belong to group or +ms resources. Group and ms resources themselves also cannot be +renamed. Please use the `rename` command instead. +**************************** + +[[cmdhelp_configure_erase,erase the CIB]] +==== `erase` + +The `erase` clears all configuration. Apart from nodes. To remove +nodes, you have to specify an additional keyword `nodes`. + +Note that removing nodes from the live cluster may have some +strange/interesting/unwelcome effects. + +Usage: +............... +erase [nodes] +............... + +[[cmdhelp_configure_fencing_topology,node fencing order]] +==== `fencing_topology` + +If multiple fencing (stonith) devices are available capable of +fencing a node, their order may be specified by +fencing_topology+. +The order is specified per node. + +Stonith resources can be separated by +,+ in which case all of +them need to succeed. If they fail, the next stonith resource (or +set of resources) is used. In other words, use comma to separate +resources which all need to succeed and whitespace for serial +order. It is not allowed to use whitespace around comma. + +If the node is left out, the order is used for all nodes. +That should reduce the configuration size in some stonith setups. + +From Pacemaker version 1.1.14, it is possible to use a node attribute +as the +target+ in a fencing topology. The syntax for this usage is +described below. + +From Pacemaker version 1.1.14, it is also possible to use regular +expression patterns as the +target+ in a fencing topology. The configured +fencing sequence then applies to all devices matching the pattern. + +Usage: +............... +fencing_topology <stonith_resources> [<stonith_resources> ...] +fencing_topology <fencing_order> [<fencing_order> ...] + +fencing_order :: <target> <stonith_resources> [<stonith_resources> ...] + +stonith_resources :: <rsc>[,<rsc>...] +target :: <node>: | attr:<node-attribute>=<value> | pattern:<pattern> +............... +Example: +............... +# Only kill the power if poison-pill fails +fencing_topology poison-pill power + +# As above for node-a, but a different strategy for node-b +fencing_topology \ + node-a: poison-pill power \ + node-b: ipmi serial + +# Fencing anything on rack 1 requires fencing via both APC 1 and 2, +# to defeat the redundancy provided by two separate UPS units. +fencing_topology attr:rack=1 apc01,apc02 + +# Fencing for all machines named green.* is done using the pear +# fencing device first, while all machines named red.* are fenced +# using the apple fencing device first. +fencing_topology \ + pattern:green.* pear apple \ + pattern:red.* apple pear +............... + +[[cmdhelp_configure_filter,filter CIB objects]] +==== `filter` + +This command filters the given CIB elements through an external +program. The program should accept input on `stdin` and send +output to `stdout` (the standard UNIX filter conventions). As +with the `show` command, the user may choose to filter all or +just a subset of elements. + +It is possible to filter the XML representation of objects, but +probably not as useful as the configuration language. The +presentation is somewhat different from what would be displayed +by the `show` command---each element is shown on a single line, +i.e. there are no backslashes and no other embelishments. + +Don't forget to put quotes around the filter if it contains +spaces. + +Usage: +............... +filter <prog> [xml] [<id> ...] +filter <prog> [xml] changed +............... +Examples: +............... +filter "sed '/^primitive/s/target-role=[^ ]*//'" +# crm configure filter "sed '/^primitive/s/target-role=[^ ]*//'" +crm configure <<END + filter "sed '/threshold=\"1\"/s/=\"1\"/=\"0\"/g'" +END +............... + +.Note on quotation marks +************************** +Filter commands which feature a blend of quotation marks can be +difficult to get right, especially when used directly from bash, since +bash does its own quotation parsing. In these cases, it can be easier +to supply the filter command as standard input. See the last example +above. +************************** + +[[cmdhelp_configure_get_property,Get property value]] +==== `get-property` + +Show the value of the given property. If the value is not set, the +command will print the default value for the property, if known. + +If no property name is passed to the command, the list of known +cluster properties is printed. + +If the property is set multiple times, for example using multiple +property sets with different rule expressions, the output of this +command is undefined. + +Pass the argument +-t+ or +--true+ to `get-property` to translate +the argument value into +true+ or +false+. If the value is not +set, the command will print +false+. + +Usage: +............... +get-property [-t|--true] [<name>] +............... + +Example: +............... +get-property stonith-enabled +get-property -t maintenance-mode +............... + +[[cmdhelp_configure_graph,generate a directed graph]] +==== `graph` + +Create a graphviz graphical layout from the current cluster +configuration. + +Currently, only `dot` (directed graph) is supported. It is +essentially a visualization of resource ordering. + +The graph may be saved to a file which can be used as source for +various graphviz tools (by default it is displayed in the user's +X11 session). Optionally, by specifying the format, one can also +produce an image instead. + +For more or different graphviz attributes, it is possible to save +the default set of attributes to an ini file. If this file exists +it will always override the builtin settings. The +exportsettings+ +subcommand also prints the location of the ini file. + +Usage: +............... +graph [<gtype> [<file> [<img_format>]]] +graph exportsettings + +gtype :: dot +img_format :: `dot` output format (see the +-T+ option) +............... +Example: +............... +graph dot +graph dot clu1.conf.dot +graph dot clu1.conf.svg svg +............... + +[[cmdhelp_configure_group,define a group]] +==== `group` + +The `group` command creates a group of resources. This can be useful +when resources depend on other resources and require that those +resources start in order on the same node. A common use of resource +groups is to ensure that a server and a virtual IP are located +together, and that the virtual IP is started before the server. + +Grouped resources are started in the order they appear in the group, +and stopped in the reverse order. If a resource in the group cannot +run anywhere, resources following it in the group will not start. + +`group` can be passed the "container" meta attribute, to indicate that +it is to be used to group VM resources monitored using Nagios. The +resource referred to by the container attribute must be of type +`ocf:heartbeat:Xen`, `ocf:heartbeat:VirtualDomain` or `ocf:heartbeat:lxc`. + +Usage: +............... +group <name> <rsc> [<rsc>...] + [description=<description>] + [meta attr_list] + [params attr_list] + +attr_list :: [$id=<id>] <attr>=<val> [<attr>=<val>...] | $id-ref=<id> +............... +Example: +............... +group internal_www disk0 fs0 internal_ip apache \ + meta target_role=stopped + +group vm-and-services vm vm-sshd meta container="vm" +............... + +[[cmdhelp_configure_load,import the CIB from a file]] +==== `load` + +Load a part of configuration (or all of it) from a local file or +a network URL. The +replace+ method replaces the current +configuration with the one from the source. The +update+ method +tries to import the contents into the current configuration. The ++push+ method imports the contents into the current configuration +and removes any lines that are not present in the given +configuration. +The file may be a CLI file or an XML file. + +If the URL is `-`, the configuration is read from standard input. + +Usage: +............... +load [xml] <method> URL + +method :: replace | update | push +............... +Example: +............... +load xml update myfirstcib.xml +load xml replace http://storage.big.com/cibs/bigcib.xml +load xml push smallcib.xml +............... + +[[cmdhelp_configure_location,a location preference]] +==== `location` + +`location` defines the preference of nodes for the given +resource. The location constraints consist of one or more rules +which specify a score to be awarded if the rule matches. + +The resource referenced by the location constraint can be one of the +following: + +* Plain resource reference: +location loc1 webserver 100: node1+ +* Resource set in curly brackets: +location loc1 { virtual-ip webserver } 100: node1+ +* Tag containing resource ids: +location loc1 tag1 100: node1+ +* Resource pattern: +location loc1 /web.*/ 100: node1+ + +The +resource-discovery+ attribute allows probes to be selectively +enabled or disabled per resource and node. + +The syntax for resource sets is described in detail for +<<cmdhelp_configure_colocation,`colocation`>>. + +For more details on how to configure resource sets, see +<<topics_Features_Resourcesets,`Syntax: Resource sets`>>. + +For more information on rule expressions, see +<<topics_Syntax_RuleExpressions,Syntax: Rule expressions>>. + +Usage: +............... +location <id> <rsc> [<attributes>] {<node_pref>|<rules>} + +rsc :: /<rsc-pattern>/ + | { resource_sets } + | <rsc> + +attributes :: role=<role> | resource-discovery=always|never|exclusive + +node_pref :: <score>: <node> + +rules :: + rule [id_spec] [$role=<role>] <score>: <expression> + [rule [id_spec] [$role=<role>] <score>: <expression> ...] + +id_spec :: $id=<id> | $id-ref=<id> +score :: <number> | <attribute> | [-]inf +expression :: <simple_exp> [<bool_op> <simple_exp> ...] +bool_op :: or | and +simple_exp :: <attribute> [type:]<binary_op> <value> + | <unary_op> <attribute> + | date <date_expr> +type :: string | version | number +binary_op :: lt | gt | lte | gte | eq | ne +unary_op :: defined | not_defined + +date_expr :: lt <end> + | gt <start> + | in start=<start> end=<end> + | in start=<start> <duration> + | spec <date_spec> +duration|date_spec :: + hours=<value> + | monthdays=<value> + | weekdays=<value> + | yearsdays=<value> + | months=<value> + | weeks=<value> + | years=<value> + | weekyears=<value> + | moon=<value> +............... +Examples: +............... +location conn_1 internal_www 100: node1 + +location conn_1 internal_www \ + rule 50: #uname eq node1 \ + rule pingd: defined pingd + +location conn_2 dummy_float \ + rule -inf: not_defined pingd or pingd number:lte 0 + +# never probe for rsc1 on node1 +location no-probe rsc1 resource-discovery=never -inf: node1 +............... + +[[cmdhelp_configure_modgroup,modify group]] +==== `modgroup` + +Add or remove primitives in a group. The `add` subcommand appends +the new group member by default. Should it go elsewhere, there +are `after` and `before` clauses. + +Usage: +............... +modgroup <id> add <id> [after <id>|before <id>] +modgroup <id> remove <id> +............... +Examples: +............... +modgroup share1 add storage2 before share1-fs +............... + +[[cmdhelp_configure_monitor,add monitor operation to a primitive]] +==== `monitor` + +Monitor is by far the most common operation. It is possible to +add it without editing the whole resource. Also, long primitive +definitions may be a bit uncluttered. In order to make this +command as concise as possible, less common operation attributes +are not available. If you need them, then use the `op` part of +the `primitive` command. + +Usage: +............... +monitor <rsc>[:<role>] <interval>[:<timeout>] +............... +Example: +............... +monitor apcfence 60m:60s +............... + +Note that after executing the command, the monitor operation may +be shown as part of the primitive definition. + +[[cmdhelp_configure_ms,define a master-slave resource]] +==== `ms` (`master`) + +The `ms` command creates a master/slave resource type. It may contain a +single primitive resource or one group of resources. + +Usage: +............... +ms <name> <rsc> + [description=<description>] + [meta attr_list] + [params attr_list] + +attr_list :: [$id=<id>] <attr>=<val> [<attr>=<val>...] | $id-ref=<id> +............... +Example: +............... +ms disk1 drbd1 \ + meta notify=true globally-unique=false +............... + +.Note on `id-ref` usage +**************************** +Instance or meta attributes (`params` and `meta`) may contain +a reference to another set of attributes. In that case, no other +attributes are allowed. Since attribute sets' ids, though they do +exist, are not shown in the `crm`, it is also possible to +reference an object instead of an attribute set. `crm` will +automatically replace such a reference with the right id: + +............... +crm(live)configure# primitive a2 www-2 meta $id-ref=a1 +crm(live)configure# show a2 +primitive a2 apache \ + meta $id-ref=a1-meta_attributes + [...] +............... +It is advisable to give meaningful names to attribute sets which +are going to be referenced. +**************************** + +[[cmdhelp_configure_node,define a cluster node]] +==== `node` + +The node command describes a cluster node. Nodes in the CIB are +commonly created automatically by the CRM. Hence, you should not +need to deal with nodes unless you also want to define node +attributes. Note that it is also possible to manage node +attributes at the `node` level. + +Usage: +............... +node [$id=<id>] <uname>[:<type>] + [description=<description>] + [attributes [$id=<id>] [<score>:] [rule...] + <param>=<value> [<param>=<value>...]] | $id-ref=<ref> + [utilization [$id=<id>] [<score>:] [rule...] + <param>=<value> [<param>=<value>...]] | $id-ref=<ref> + +type :: normal | member | ping | remote +............... +Example: +............... +node node1 +node big_node attributes memory=64 +............... + +[[cmdhelp_configure_op_defaults,set resource operations defaults]] +==== `op_defaults` + +Set defaults for the operations meta attributes. + +For more information on rule expressions, see +<<topics_Syntax_RuleExpressions,Syntax: Rule expressions>>. + +Usage: +............... +op_defaults [$id=<set_id>] [rule ...] <option>=<value> [<option>=<value> ...] +............... +Example: +............... +op_defaults record-pending=true +............... + +[[cmdhelp_configure_order,order resources]] +==== `order` + +This constraint expresses the order of actions on two resources +or more resources. If there are more than two resources, then the +constraint is called a resource set. + +Ordered resource sets have an extra attribute to allow for sets +of resources whose actions may run in parallel. The shell syntax +for such sets is to put resources in parentheses. + +If the subsequent resource can start or promote after any one of the +resources in a set has done, enclose the set in brackets (+[+ and +]+). + +Sets cannot be nested. + +Three strings are reserved to specify a kind of order constraint: ++Mandatory+, +Optional+, and +Serialize+. It is preferred to use +one of these settings instead of score. Previous versions mapped +scores +0+ and +inf+ to keywords +advisory+ and +mandatory+. +That is still valid but deprecated. + +For more details on how to configure resource sets, see +<<topics_Features_Resourcesets,`Syntax: Resource sets`>>. + +Usage: +............... +order <id> [{kind|<score>}:] first then [symmetrical=<bool>] + +order <id> [{kind|<score>}:] resource_sets [symmetrical=<bool>] + +kind :: Mandatory | Optional | Serialize + +first :: <rsc>[:<action>] + +then :: <rsc>[:<action>] + +resource_sets :: resource_set [resource_set ...] + +resource_set :: ["["|"("] <rsc>[:<action>] [<rsc>[:<action>] ...] \ + [attributes] ["]"|")"] + +attributes :: [require-all=(true|false)] [sequential=(true|false)] + +............... +Example: +............... +order o-1 Mandatory: apache:start ip_1 +order o-2 Serialize: A ( B C ) +order o-3 inf: [ A B ] C +order o-4 first-resource then-resource +............... + +[[cmdhelp_configure_primitive,define a resource]] +==== `primitive` + +The primitive command describes a resource. It may be referenced +only once in group, clone, or master-slave objects. If it's not +referenced, then it is placed as a single resource in the CIB. + +Operations may be specified anonymously, as a group or by reference: + +* "Anonymous", as a list of +op+ specifications. Use this + method if you don't need to reference the set of operations + elsewhere. This is the most common way to define operations. + +* If reusing operation sets is desired, use the +operations+ keyword + along with an id to give the operations set a name. Use the + +operations+ keyword and an id-ref value set to the id of another + operations set, to apply the same set of operations to this + primitive. + +Operation attributes which are not recognized are saved as +instance attributes of that operation. A typical example is ++OCF_CHECK_LEVEL+. + +For multistate resources, roles are specified as +role=<role>+. + +A template may be defined for resources which are of the same +type and which share most of the configuration. See +<<cmdhelp_configure_rsc_template,`rsc_template`>> for more information. + +Attributes containing time values, such as the +interval+ attribute on +operations, are configured either as a plain number, which is +interpreted as a time in seconds, or using one of the following +suffixes: + +* +s+, +sec+ - time in seconds (same as no suffix) +* +ms+, +msec+ - time in milliseconds +* +us+, +usec+ - time in microseconds +* +m+, +min+ - time in minutes +* +h+, +hr+ - time in hours + +Usage: +............... +primitive <rsc> {[<class>:[<provider>:]]<type>|@<template>} + [description=<description>] + [[params] attr_list] + [meta attr_list] + [utilization attr_list] + [operations id_spec] + [op op_type [<attribute>=<value>...] ...] + +attr_list :: [$id=<id>] [<score>:] [rule...] + <attr>=<val> [<attr>=<val>...]] | $id-ref=<id> +id_spec :: $id=<id> | $id-ref=<id> +op_type :: start | stop | monitor +............... +Example: +............... +primitive apcfence stonith:apcsmart \ + params ttydev=/dev/ttyS0 hostlist="node1 node2" \ + op start timeout=60s \ + op monitor interval=30m timeout=60s + +primitive www8 apache \ + configfile=/etc/apache/www8.conf \ + operations $id-ref=apache_ops + +primitive db0 mysql \ + params config=/etc/mysql/db0.conf \ + op monitor interval=60s \ + op monitor interval=300s OCF_CHECK_LEVEL=10 + +primitive r0 ocf:linbit:drbd \ + params drbd_resource=r0 \ + op monitor role=Master interval=60s \ + op monitor role=Slave interval=300s + +primitive xen0 @vm_scheme1 xmfile=/etc/xen/vm/xen0 + +primitive mySpecialRsc Special \ + params 3: rule #uname eq node1 interface=eth1 \ + params 2: rule #uname eq node2 interface=eth2 port=8888 \ + params 1: interface=eth0 port=9999 + +............... + +[[cmdhelp_configure_property,set a cluster property]] +==== `property` + +Set cluster configuration properties. To list the +available cluster configuration properties, use the +<<cmdhelp_ra_info,`ra info`>> command with +pengine+, +crmd+, ++cib+ and +stonithd+ as arguments. +When setting the +maintenance-mode+ property, it will +inform the user if there are nodes or resources that +have the +maintenance+ property. + +For more information on rule expressions, see +<<topics_Syntax_RuleExpressions,Syntax: Rule expressions>>. + +Usage: +............... +property [<set_id>:] [rule ...] <option>=<value> [<option>=<value> ...] +............... +Example: +............... +property stonith-enabled=true +property rule date spec years=2014 stonith-enabled=false +............... + +[[cmdhelp_configure_ptest,show cluster actions if changes were committed]] +==== `ptest` (`simulate`) + +Show PE (Policy Engine) motions using `ptest(8)` or +`crm_simulate(8)`. + +A CIB is constructed using the current user edited configuration +and the status from the running CIB. The resulting CIB is run +through `ptest` (or `crm_simulate`) to show changes which would +happen if the configuration is committed. + +The status section may be loaded from another source and modified +using the <<cmdhelp_cibstatus,`cibstatus`>> level commands. In that case, the +`ptest` command will issue a message informing the user that the +Policy Engine graph is not calculated based on the current status +section and therefore won't show what would happen to the +running but some imaginary cluster. + +If you have graphviz installed and X11 session, `dotty(1)` is run +to display the changes graphically. + +Add a string of +v+ characters to increase verbosity. `ptest` +can also show allocation scores. +utilization+ turns on +information about the remaining capacity of nodes. With the ++actions+ option, `ptest` will print all resource actions. + +The `ptest` program has been replaced by `crm_simulate` in newer +Pacemaker versions. In some installations both could be +installed. Use `simulate` to enfore using `crm_simulate`. + +Usage: +............... +ptest [nograph] [v...] [scores] [actions] [utilization] +............... +Examples: +............... +ptest scores +ptest vvvvv +simulate actions +............... + +[[cmdhelp_configure_refresh,refresh from CIB]] +==== `refresh` + +Refresh the internal structures from the CIB. All changes made +during this session are lost. + +Usage: +............... +refresh +............... + +[[cmdhelp_configure_rename,rename a CIB object]] +==== `rename` + +Rename an object. It is recommended to use this command to rename +a resource, because it will take care of updating all related +constraints and a parent resource. Changing ids with the edit +command won't have the same effect. + +If you want to rename a resource, it must be in the stopped state. + +Usage: +............... +rename <old_id> <new_id> +............... + +[[cmdhelp_configure_role,define role access rights]] +==== `role` + +An ACL role is a set of rules which describe access rights to +CIB. Rules consist of an access right +read+, +write+, or +deny+ +and a specification denoting part of the configuration to which +the access right applies. The specification can be an XPath or a +combination of tag and id references. If an attribute is +appended, then the specification applies only to that attribute +of the matching element. + +There is a number of shortcuts for XPath specifications. The ++meta+, +params+, and +utilization+ shortcuts reference resource +meta attributes, parameters, and utilization respectively. The +`location` may be used to specify location constraints most of +the time to allow resource `move` and `unmove` commands. The +`property` references cluster properties. The `node` allows +reading node attributes. +nodeattr+ and +nodeutil+ reference node +attributes and node capacity (utilization). The `status` shortcut +references the whole status section of the CIB. Read access to +status is necessary for various monitoring tools such as +`crm_mon(8)` (aka `crm status`). + +For more information on rule expressions, see +<<topics_Syntax_RuleExpressions,Syntax: Rule expressions>>. + +Usage: +............... +role <role-id> rule [rule ...] + +rule :: acl-right cib-spec [attribute:<attribute>] + +acl-right :: read | write | deny + +cib-spec :: xpath-spec | tag-ref-spec +xpath-spec :: xpath:<xpath> | shortcut +tag-ref-spec :: tag:<tag> | ref:<id> | tag:<tag> ref:<id> + +shortcut :: meta:<rsc>[:<attr>] + params:<rsc>[:<attr>] + utilization:<rsc> + location:<rsc> + property[:<attr>] + node[:<node>] + nodeattr[:<attr>] + nodeutil[:<node>] + status +............... +Example: +............... +role app1_admin \ + write meta:app1:target-role \ + write meta:app1:is-managed \ + write location:app1 \ + read ref:app1 +............... + +[[cmdhelp_configure_rsc_defaults,set resource defaults]] +==== `rsc_defaults` + +Set defaults for the resource meta attributes. + +For more information on rule expressions, see +<<topics_Syntax_RuleExpressions,Syntax: Rule expressions>>. + +Usage: +............... +rsc_defaults [<set_id>:] [rule ...] <option>=<value> [<option>=<value> ...] +............... +Example: +............... +rsc_defaults failure-timeout=3m +............... + +[[cmdhelp_configure_rsc_template,define a resource template]] +==== `rsc_template` + +The `rsc_template` command creates a resource template. It may be +referenced in primitives. It is used to reduce large +configurations with many similar resources. + +Usage: +............... +rsc_template <name> [<class>:[<provider>:]]<type> + [description=<description>] + [params attr_list] + [meta attr_list] + [utilization attr_list] + [operations id_spec] + [op op_type [<attribute>=<value>...] ...] + +attr_list :: [$id=<id>] <attr>=<val> [<attr>=<val>...] | $id-ref=<id> +id_spec :: $id=<id> | $id-ref=<id> +op_type :: start | stop | monitor +............... +Example: +............... +rsc_template public_vm Xen \ + op start timeout=300s \ + op stop timeout=300s \ + op monitor interval=30s timeout=60s \ + op migrate_from timeout=600s \ + op migrate_to timeout=600s +primitive xen0 @public_vm \ + params xmfile=/etc/xen/xen0 +primitive xen1 @public_vm \ + params xmfile=/etc/xen/xen1 +............... + +[[cmdhelp_configure_rsc_ticket,resources ticket dependency]] +==== `rsc_ticket` + +This constraint expresses dependency of resources on cluster-wide +attributes, also known as tickets. Tickets are mainly used in +geo-clusters, which consist of multiple sites. A ticket may be +granted to a site, thus allowing resources to run there. + +The +loss-policy+ attribute specifies what happens to the +resource (or resources) if the ticket is revoked. The default is +either +stop+ or +demote+ depending on whether a resource is +multi-state. + +See also the <<cmdhelp_site_ticket,`site`>> set of commands. + +Usage: +............... +rsc_ticket <id> <ticket_id>: <rsc>[:<role>] [<rsc>[:<role>] ...] + [loss-policy=<loss_policy_action>] + +loss_policy_action :: stop | demote | fence | freeze +............... +Example: +............... +rsc_ticket ticket-A_public-ip ticket-A: public-ip +rsc_ticket ticket-A_bigdb ticket-A: bigdb loss-policy=fence +rsc_ticket ticket-B_storage ticket-B: drbd-a:Master drbd-b:Master +............... + + +[[cmdhelp_configure_rsctest,test resources as currently configured]] +==== `rsctest` + +Test resources with current resource configuration. If no nodes +are specified, tests are run on all known nodes. + +The order of resources is significant: it is assumed that later +resources depend on earlier ones. + +If a resource is multi-state, it is assumed that the role on +which later resources depend is master. + +Tests are run sequentially to prevent running the same resource +on two or more nodes. Tests are carried out only if none of the +specified nodes currently run any of the specified resources. +However, it won't verify whether resources run on the other +nodes. + +Superuser privileges are obviously required: either run this as +root or setup the `sudoers` file appropriately. + +Note that resource testing may take some time. + +Usage: +............... +rsctest <rsc_id> [<rsc_id> ...] [<node_id> ...] +............... +Examples: +............... +rsctest my_ip websvc +rsctest websvc nodeB +............... + +[[cmdhelp_configure_save,save the CIB to a file]] +==== `save` + +Save the current configuration to a file. Optionally, as XML. Use ++-+ instead of file name to write the output to `stdout`. + +The `save` command accepts the same selection arguments as the `show` +command. See the <<cmdhelp_configure_show,help section>> for `show` +for more details. + +Usage: +............... +save [xml] [<id> | type:<type | tag:<tag> | + related:<obj> | changed ...] <file> +............... +Example: +............... +save myfirstcib.txt +save web-server server-config.txt +............... + +[[cmdhelp_configure_schema,set or display current CIB RNG schema]] +==== `schema` + +CIB's content is validated by a RNG schema. Pacemaker supports +several, depending on version. At least the following schemas are +accepted by `crmsh`: + +* +pacemaker-1.0+ +* +pacemaker-1.1+ +* +pacemaker-1.2+ +* +pacemaker-1.3+ +* +pacemaker-2.0+ + +Use this command to display or switch to another RNG schema. + +Usage: +............... +schema [<schema>] +............... +Example: +............... +schema pacemaker-1.1 +............... + +[[cmdhelp_configure_set,set an attribute value]] +==== `set` + +Set the value of a configured attribute. The attribute must +have a value configured previously, and can be an agent +parameter, meta attribute or utilization value. + +The first argument to the command is a path to an attribute. +This is a dot-separated sequence beginning with the name of +the resource, and ending with the name of the attribute to +set. + +Usage: +............... +set <path> <value> +............... +Examples: +............... +set vip1.ip 192.168.20.5 +set vm-a.force_stop 1 +............... + +[[cmdhelp_configure_show,display CIB objects]] +==== `show` + +The `show` command displays CIB objects. Without any argument, it +displays all objects in the CIB, but the set of objects displayed by +`show` can be limited to only objects with the given IDs or by using +one or more of the special prefixes described below. + +The XML representation for the objects can be displayed by passing ++xml+ as the first argument. + +To show one or more specific objects, pass the object IDs as +arguments. + +To show all objects of a certain type, use the +type:+ prefix. + +To show all objects in a tag, use the +tag:+ prefix. + +To show all constraints related to a primitive, use the +related:+ prefix. + +To show all modified objects, pass the argument +changed+. + +The prefixes can be used together on a single command line. For +example, to show both the tag itself and the objects tagged by it the +following combination can be used: +show tag:my-tag my-tag+. + +To refine a selection of objects using multiple modifiers, the keywords ++and+ and +or+ can be used. For example, to select all primitives tagged ++foo+, the following combination can be used: ++show type:primitive and tag:foo+. + +To hide values when displaying the configuration, use the ++obscure:<glob>+ argument. This can be useful when sending the +configuration over a public channel, to avoid exposing potentially +sensitive information. The +<glob>+ argument is a bash-style pattern +matching attribute keys. + +Usage: +............... +show [xml] [<id> + | changed + | type:<type> + | tag:<id> + | related:<obj> + | obscure:<glob> + ...] + +type :: node | primitive | group | clone | ms | rsc_template + | location | colocation | order + | rsc_ticket + | property | rsc_defaults | op_defaults + | fencing_topology + | role | user | acl_target + | tag +............... + +Example: +............... +show webapp +show type:primitive +show xml tag:db tag:fs +show related:webapp +show type:primitive obscure:passwd +............... + +[[cmdhelp_configure_tag,Define resource tags]] +==== `tag` + +Define a resource tag. A tag is an id referring to one or more +resources, without implying any constraints between the tagged +resources. This can be useful for grouping conceptually related +resources. + +Usage: +............... +tag <tag-name>: <rsc> [<rsc> ...] +tag <tag-name> <rsc> [<rsc> ...] +............... +Example: +............... +tag web: p-webserver p-vip +tag ips server-vip admin-vip +............... + +[[cmdhelp_configure_template,edit and import a configuration from a template]] +==== `template` + +The specified template is loaded into the editor. It's up to the +user to make a good CRM configuration out of it. See also the +<<cmdhelp_template,template section>>. + +Usage: +............... +template [xml] url +............... +Example: +............... +template two-apaches.txt +............... + +[[cmdhelp_configure_upgrade,upgrade the CIB]] +==== `upgrade` + +Attempts to upgrade the CIB to validate with the current +version. Commonly, this is required if the error +`CIB not supported` occurs. It typically means that the +active CIB version is coming from an older release. + +As a safety precaution, the force argument is required if the ++validation-with+ attribute is set to anything other than ++0.6+. Thus in most cases, it is required. + +Usage: +............... +upgrade [force] +............... + +Example: +............... +upgrade force +............... + +[[cmdhelp_configure_user,define user access rights]] +==== `user` + +Users which normally cannot view or manage cluster configuration +can be allowed access to parts of the CIB. The access is defined +by a set of +read+, +write+, and +deny+ rules as in role +definitions or by referencing roles. The latter is considered +best practice. + +For more information on rule expressions, see +<<topics_Syntax_RuleExpressions,Syntax: Rule expressions>>. + +Usage: +............... +user <uid> {roles|rules} + +roles :: role:<role-ref> [role:<role-ref> ...] +rules :: rule [rule ...] +............... +Example: +............... +user joe \ + role:app1_admin \ + role:read_all +............... + +[[cmdhelp_configure_validate_all,call agent validate-all for resource]] +==== `validate-all` + +Call the `validate-all` action for the resource, if possible. + +Limitations: + +* The resource agent must implement the `validate-all` action. +* The current user must be root. +* The primitive resource must not use nvpair references. + +Usage: +............... +validate-all <rsc> +............... + + +[[cmdhelp_configure_verify,verify the CIB with crm_verify]] +==== `verify` + +Verify the contents of the CIB which would be committed. + +Usage: +............... +verify +............... + +[[cmdhelp_configure_xml,raw xml]] +==== `xml` + +Even though we promissed no xml, it may happen, but hopefully +very very seldom, that an element from the CIB cannot be rendered +in the configuration language. In that case, the element will be +shown as raw xml, prefixed by this command. That element can then +be edited like any other. If the shell finds out that after the +change it can digest it, then it is going to be converted into +the normal configuration language. Otherwise, there is no need to +use `xml` for configuration. + +Usage: +............... +xml <xml> +............... + +[[cmdhelp_template,edit and import a configuration from a template]] +=== `template` - Import configuration from templates + +User may be assisted in the cluster configuration by templates +prepared in advance. Templates consist of a typical ready +configuration which may be edited to suit particular user needs. + +This command enters a template level where additional commands +for configuration/template management are available. + +[[cmdhelp_template_apply,process and apply the current configuration to the current CIB]] +==== `apply` + +Copy the current or given configuration to the current CIB. By +default, the CIB is replaced, unless the method is set to +"update". + +Usage: +............... +apply [<method>] [<config>] + +method :: replace | update +............... + +[[cmdhelp_template_delete,delete a configuration]] +==== `delete` + +Remove a configuration. The loaded (active) configuration may be +removed by force. + +Usage: +............... +delete <config> [force] +............... + +[[cmdhelp_template_edit,edit a configuration]] +==== `edit` + +Edit current or given configuration using your favourite editor. + +Usage: +............... +edit [<config>] +............... + +[[cmdhelp_template_list,list configurations/templates]] +==== `list` + +When called with no argument, lists existing templates and +configurations. + +Given the argument +templates+, lists the available templates. + +Given the argument +configs+, lists the available configurations. + +Usage: +............... +list [templates|configs] +............... + +[[cmdhelp_template_load,load a configuration]] +==== `load` + +Load an existing configuration. Further `edit`, `show`, and +`apply` commands will refer to this configuration. + +Usage: +............... +load <config> +............... + +[[cmdhelp_template_new,create a new configuration from templates]] +==== `new` + +Create a new configuration from one or more templates. Note that +configurations and templates are kept in different places, so it +is possible to have a configuration name equal a template name. + +If you already know which parameters are required, you can set +them directly on the command line. + +The parameter name +id+ is set by default to the name of the +configuration. + +If no parameters are being set and you don't want a particular name +for your configuration, you can call this command with a template name +as the only parameter. A unique configuration name based on the +template name will be generated. + +Usage: +............... +new [<config>] <template> [<template> ...] [params name=value ...] +............... + +Example: +............... +new vip virtual-ip +new bigfs ocfs2 params device=/dev/sdx8 directory=/bigfs +new apache +............... + +[[cmdhelp_template_show,show the processed configuration]] +==== `show` + +Process the current or given configuration and display the result. + +Usage: +............... +show [<config>] +............... + +[[cmdhelp_cibstatus,CIB status management and editing]] +=== `cibstatus` - CIB status management and editing + +The `status` section of the CIB keeps the current status of nodes +and resources. It is modified _only_ on events, i.e. when some +resource operation is run or node status changes. For obvious +reasons, the CRM has no user interface with which it is possible +to affect the status section. From the user's point of view, the +status section is essentially a read-only part of the CIB. The +current status is never even written to disk, though it is +available in the PE (Policy Engine) input files which represent +the history of cluster motions. The current status may be read +using the +cibadmin -Q+ command. + +It may sometimes be of interest to see how status changes would +affect the Policy Engine. The set of `cibstatus` level commands +allow the user to load status sections from various sources and +then insert or modify resource operations or change nodes' state. + +The effect of those changes may then be observed by running the +<<cmdhelp_configure_ptest,`ptest`>> command at the `configure` level +or `simulate` and `run` commands at this level. The `ptest` +runs with the user edited CIB whereas the latter two commands +run with the CIB which was loaded along with the status section. + +The `simulate` and `run` commands as well as all status +modification commands are implemented using `crm_simulate(8)`. + +[[cmdhelp_cibstatus_load,load the CIB status section]] +==== `load` + +Load a status section from a file, a shadow CIB, or the running +cluster. By default, the current (+live+) status section is +modified. Note that if the +live+ status section is modified it +is not going to be updated if the cluster status changes, because +that would overwrite the user changes. To make `crm` drop changes +and resume use of the running cluster status, run +load live+. + +All CIB shadow configurations contain the status section which is +a snapshot of the status section taken at the time the shadow was +created. Obviously, this status section doesn't have much to do +with the running cluster status, unless the shadow CIB has just +been created. Therefore, the `ptest` command by default uses the +running cluster status section. + +Usage: +............... +load {<file>|shadow:<cib>|live} +............... +Example: +............... +load bug-12299.xml +load shadow:test1 +............... + +[[cmdhelp_cibstatus_node,change node status]] +==== `node` + +Change the node status. It is possible to throw a node out of +the cluster, make it a member, or set its state to unclean. + ++online+:: Set the +node_state+ `crmd` attribute to +online+ +and the +expected+ and +join+ attributes to +member+. The effect +is that the node becomes a cluster member. + ++offline+:: Set the +node_state+ `crmd` attribute to +offline+ +and the +expected+ attribute to empty. This makes the node +cleanly removed from the cluster. + ++unclean+:: Set the +node_state+ `crmd` attribute to +offline+ +and the +expected+ attribute to +member+. In this case the node +has unexpectedly disappeared. + +Usage: +............... +node <node> {online|offline|unclean} +............... +Example: +............... +node xen-b unclean +............... + +[[cmdhelp_cibstatus_op,edit outcome of a resource operation]] +==== `op` + +Edit the outcome of a resource operation. This way you can +tell CRM that it ran an operation and that the resource agent +returned certain exit code. It is also possible to change the +operation's status. In case the operation status is set to +something other than +done+, the exit code is effectively +ignored. + +Usage: +............... +op <operation> <resource> <exit_code> [<op_status>] [<node>] + +operation :: probe | monitor[:<n>] | start | stop | + promote | demote | notify | migrate_to | migrate_from +exit_code :: <rc> | success | generic | args | + unimplemented | perm | installed | configured | not_running | + master | failed_master +op_status :: pending | done | cancelled | timeout | notsupported | error + +n :: the monitor interval in seconds; if omitted, the first + recurring operation is referenced +rc :: numeric exit code in range 0..9 +............... +Example: +............... +op start d1 xen-b generic +op start d1 xen-b 1 +op monitor d1 xen-b not_running +op stop d1 xen-b 0 timeout +............... + +[[cmdhelp_cibstatus_origin,display origin of the CIB status section]] +==== `origin` + +Show the origin of the status section currently in use. This +essentially shows the latest `load` argument. + +Usage: +............... +origin +............... + +[[cmdhelp_cibstatus_quorum,set the quorum]] +==== `quorum` + +Set the quorum value. + +Usage: +............... +quorum <bool> +............... +Example: +............... +quorum false +............... + +[[cmdhelp_cibstatus_run,run policy engine]] +==== `run` + +Run the policy engine with the edited status section. + +Add a string of +v+ characters to increase verbosity. Specify ++scores+ to see allocation scores also. +utilization+ turns on +information about the remaining capacity of nodes. + +If you have graphviz installed and X11 session, `dotty(1)` is run +to display the changes graphically. + +Usage: +............... +run [nograph] [v...] [scores] [utilization] +............... +Example: +............... +run +............... + +[[cmdhelp_cibstatus_save,save the CIB status section]] +==== `save` + +The current internal status section with whatever modifications +were performed can be saved to a file or shadow CIB. + +If the file exists and contains a complete CIB, only the status +section is going to be replaced and the rest of the CIB will +remain intact. Otherwise, the current user edited configuration +is saved along with the status section. + +Note that all modifications are saved in the source file as soon +as they are run. + +Usage: +............... +save [<file>|shadow:<cib>] +............... +Example: +............... +save bug-12299.xml +............... + +[[cmdhelp_cibstatus_show,show CIB status section]] +==== `show` + +Show the current status section in the XML format. Brace yourself +for some unreadable output. Add +changed+ option to get a human +readable output of all changes. + +Usage: +............... +show [changed] +............... + +[[cmdhelp_cibstatus_simulate,simulate cluster transition]] +==== `simulate` + +Run the policy engine with the edited status section and simulate +the transition. + +Add a string of +v+ characters to increase verbosity. Specify ++scores+ to see allocation scores also. +utilization+ turns on +information about the remaining capacity of nodes. + +If you have graphviz installed and X11 session, `dotty(1)` is run +to display the changes graphically. + +Usage: +............... +simulate [nograph] [v...] [scores] [utilization] +............... +Example: +............... +simulate +............... + +[[cmdhelp_cibstatus_ticket,manage tickets]] +==== `ticket` + +Modify the ticket status. Tickets can be granted and revoked. +Granted tickets could be activated or put in standby. + +Usage: +............... +ticket <ticket> {grant|revoke|activate|standby} +............... +Example: +............... +ticket ticketA grant +............... + +[[cmdhelp_assist,Configuration assistant]] +=== `assist` - Configuration assistant + +The `assist` sublevel is a collection of helper +commands that create or modify resources and +constraints, to simplify the creation of certain +configurations. + +For more information on individual commands, see +the help text for those commands. + +[[cmdhelp_assist_template,Create template for primitives]] +==== `template` + +This command takes a list of primitives as argument, and creates a new +`rsc_template` for these primitives. It can only do this if the +primitives do not already share a template and are of the same type. + +Usage: +........ +template primitive-1 primitive-2 primitive-3 +........ + +[[cmdhelp_assist_weak-bond,Create a weak bond between resources]] +==== `weak-bond` + +A colocation between a group of resources says that the resources +should be located together, but it also means that those resources are +dependent on each other. If one of the resources fails, the others +will be restarted. + +If this is not desired, it is possible to circumvent: By placing the +resources in a non-sequential set and colocating the set with a dummy +resource which is not monitored, the resources will be placed together +but will have no further dependency on each other. + +This command creates both the constraint and the dummy resource needed +for such a colocation. + +Usage: +........ +weak-bond resource-1 resource-2 +........ + +[[cmdhelp_maintenance,Maintenance mode commands]] +=== `maintenance` - Maintenance mode commands + +Maintenance mode commands are commands that manipulate resources +directly without going through the cluster infrastructure. Therefore, +it is essential to ensure that the cluster does not attempt to monitor +or manipulate the resources while these commands are being executed. + +To ensure this, these commands require that maintenance mode is set +either for the particular resource, or for the whole cluster. + +[[cmdhelp_maintenance_action,Invoke a resource action]] +==== `action` + +Invokes the given action for the resource. This is +done directly via the resource agent, so the command must +be issued while the cluster or the resource is in +maintenance mode. + +Unless the action is `start` or `monitor`, the action must be invoked +on the same node as where the resource is running. If the resource is +running on multiple nodes, the command will fail. + +To use SSH for executing resource actions on multiple nodes, append +`ssh` after the action name. This requires SSH access to be configured +between the nodes and the parallax python package to be installed. + +Usage: +............... +action <rsc> <action> +action <rsc> <action> ssh +............... +Example: +............... +action webserver reload +action webserver monitor ssh +............... + +[[cmdhelp_maintenance_off,Disable maintenance mode]] +==== `off` + +Disables maintenances mode, either for the whole cluster +or for the given resource. + +Usage: +............... +off +off <rsc> +............... +Example: +............... +off rsc1 +............... + +[[cmdhelp_maintenance_on,Enable maintenance mode]] +==== `on` + +Enables maintenances mode, either for the whole cluster +or for the given resource. + +Usage: +............... +on +on <rsc> +............... +Example: +............... +on rsc1 +............... + +[[cmdhelp_history,Cluster history]] +=== `history` - Cluster history + +Examining Pacemaker's history is a particularly involved task. The +number of subsystems to be considered, the complexity of the +configuration, and the set of various information sources, most of +which are not exactly human readable, keep analyzing resource or node +problems accessible to only the most knowledgeable. Or, depending on +the point of view, to the most persistent. The following set of +commands has been devised in hope to make cluster history more +accessible. + +Of course, looking at _all_ history could be time consuming regardless +of how good the tools at hand are. Therefore, one should first say +which period he or she wants to analyze. If not otherwise specified, +the last hour is considered. Logs and other relevant information is +collected using `crm report`. Since this process takes some time and +we always need fresh logs, information is refreshed in a much faster +way using the python parallax module. If +python-parallax+ is not +found on the system, examining a live cluster is still possible -- +though not as comfortable. + +Apart from examining a live cluster, events may be retrieved from a +report generated by `crm report` (see also the +-H+ option). In that +case we assume that the period stretching the whole report needs to be +investigated. Of course, it is still possible to further reduce the +time range. + +If you have discovered an issue that you want to show someone else, +you can use the `session pack` command to save the current session as +a tarball, similar to those generated by `crm report`. + +In order to minimize the size of the tarball, and to make it easier +for others to find the interesting events, it is recommended to limit +the time frame which the saved session covers. This can be done using +the `timeframe` command (example below). + +It is also possible to name the saved session using the `session save` +command. + +Example: +............... +crm(live)history# limit "Jul 18 12:00" "Jul 18 12:30" +crm(live)history# session save strange_restart +crm(live)history# session pack +Report saved in .../strange_restart.tar.bz2 +crm(live)history# +............... + +[[cmdhelp_history_detail,set the level of detail shown]] +==== `detail` + +How much detail to show from the logs. Valid detail levels are either +`0` or `1`, where `1` is the highest detail level. The default detail +level is `0`. + +Usage: +............... +detail <detail_level> + +detail_level :: small integer (defaults to 0) +............... +Example: +............... +detail 1 +............... + +[[cmdhelp_history_diff,cluster states/transitions difference]] +==== `diff` + +A transition represents a change in cluster configuration or +state. Use `diff` to see what has changed between two +transitions. + +If you want to specify the current cluster configuration and +status, use the string +live+. + +Normally, the first transition specified should be the one which +is older, but we are not going to enforce that. + +Note that a single configuration update may result in more than +one transition. + +Usage: +............... +diff <pe> <pe> [status] [html] + +pe :: <number>|<index>|<file>|live +............... +Examples: +............... +diff 2066 2067 +diff pe-input-2080.bz2 live status +............... + +[[cmdhelp_history_events,Show events in log]] +==== `events` + +By analysing the log output and looking for particular +patterns, the `events` command helps sifting through +the logs to find when particular events like resources +changing state or node failure may have occurred. + +This can be used to generate a combined list of events +from all nodes. + +Usage: +............... +events +............... + +Example: +............... +events +............... + +[[cmdhelp_history_exclude,exclude log messages]] +==== `exclude` + +If a log is infested with irrelevant messages, those messages may +be excluded by specifying a regular expression. The regular +expressions used are Python extended. This command is additive. +To drop all regular expressions, use +exclude clear+. Run +`exclude` only to see the current list of regular expressions. +Excludes are saved along with the history sessions. + +Usage: +............... +exclude [<regex>|clear] +............... +Example: +............... +exclude kernel.*ocfs2 +............... + +[[cmdhelp_history_graph,generate a directed graph from the PE file]] +==== `graph` + +Create a graphviz graphical layout from the PE file (the +transition). Every transition contains the cluster configuration +which was active at the time. See also <<cmdhelp_configure_graph,generate a directed graph +from configuration>>. + +Usage: +............... +graph <pe> [<gtype> [<file> [<img_format>]]] + +gtype :: dot +img_format :: `dot` output format (see the +-T+ option) +............... +Example: +............... +graph -1 +graph 322 dot clu1.conf.dot +graph 322 dot clu1.conf.svg svg +............... + +[[cmdhelp_history_info,Cluster information summary]] +==== `info` + +The `info` command provides a summary of the information source, which +can be either a live cluster snapshot or a previously generated +report. + +Usage: +............... +info +............... +Example: +............... +info +............... + +[[cmdhelp_history_latest,show latest news from the cluster]] +==== `latest` + +The `latest` command shows a bit of recent history, more +precisely whatever happened since the last cluster change (the +latest transition). If the transition is running, the shell will +first wait until it finishes. + +Usage: +............... +latest +............... +Example: +............... +latest +............... + +[[cmdhelp_history_limit,limit timeframe to be examined]] +==== `limit` (`timeframe`) + +This command can be used to modify the time span to examine. All +history commands look at events within a certain time span. + +For the `live` source, the default time span is the _last hour_. + +There is no time span limit for the `hb_report` source. + +The time period is parsed by the `dateutil` python module. It +covers a wide range of date formats. For instance: + +- 3:00 (today at 3am) +- 15:00 (today at 3pm) +- 2010/9/1 2pm (September 1st 2010 at 2pm) + +For more examples of valid time/date statements, please refer to the +`python-dateutil` documentation: + +- https://dateutil.readthedocs.org/[dateutil.readthedocs.org] + +If the dateutil module is not available, then the time is parsed using +strptime and only the kind as printed by `date(1)` is allowed: + +- Tue Sep 15 20:46:27 CEST 2010 + +Usage: +............... +limit [<from_time>] [<to_time>] +............... +Examples: +............... +limit 10:15 +limit 15h22m 16h +limit "Sun 5 20:46" "Sun 5 22:00" +............... + +[[cmdhelp_history_log,log content]] +==== `log` + +Show messages logged on one or more nodes. Leaving out a node +name produces combined logs of all nodes. Messages are sorted by +time and, if the terminal emulations supports it, displayed in +different colours depending on the node to allow for easier +reading. + +The sorting key is the timestamp as written by syslog which +normally has the maximum resolution of one second. Obviously, +messages generated by events which share the same timestamp may +not be sorted in the same way as they happened. Such close events +may actually happen fairly often. + +Usage: +............... +log [<node> [<node> ...] ] +............... +Example: +............... +log node-a +............... + +[[cmdhelp_history_node,node events]] +==== `node` + +Show important events that happened on a node. Important events +are node lost and join, standby and online, and fence. Use either +node names or extended regular expressions. + +Usage: +............... +node <node> [<node> ...] +............... +Example: +............... +node node1 +............... + +[[cmdhelp_history_peinputs,list or get PE input files]] +==== `peinputs` + +Every event in the cluster results in generating one or more +Policy Engine (PE) files. These files describe future motions of +resources. The files are listed as full paths in the current +report directory. Add +v+ to also see the creation time stamps. + +Usage: +............... +peinputs [{<range>|<number>} ...] [v] + +range :: <n1>:<n2> +............... +Example: +............... +peinputs +peinputs 440:444 446 +peinputs v +............... + +[[cmdhelp_history_refresh,refresh live report]] +==== `refresh` + +This command makes sense only for the +live+ source and makes +`crm` collect the latest logs and other relevant information from +the logs. If you want to make a completely new report, specify ++force+. + +Usage: +............... +refresh [force] +............... + +[[cmdhelp_history_resource,resource events]] +==== `resource` + +Show actions and any failures that happened on all specified +resources on all nodes. Normally, one gives resource names as +arguments, but it is also possible to use extended regular +expressions. Note that neither groups nor clones or master/slave +names are ever logged. The resource command is going to expand +all of these appropriately, so that clone instances or resources +which are part of a group are shown. + +Usage: +............... +resource <rsc> [<rsc> ...] +............... +Example: +............... +resource bigdb public_ip +resource my_.*_db2 +resource ping_clone +............... + +[[cmdhelp_history_session,manage history sessions]] +==== `session` + +Sometimes you may want to get back to examining a particular +history period or bug report. In order to make that easier, the +current settings can be saved and later retrieved. + +If the current history being examined is coming from a live +cluster the logs, PE inputs, and other files are saved too, +because they may disappear from nodes. For the existing reports +coming from `hb_report`, only the directory location is saved +(not to waste space). + +A history session may also be packed into a tarball which can +then be sent to support. + +Leave out subcommand to see the current session. + +Usage: +............... +session [{save|load|delete} <name> | pack [<name>] | update | list] +............... +Examples: +............... +session save bnc966622 +session load rsclost-2 +session list +............... + +[[cmdhelp_history_setnodes,set the list of cluster nodes]] +==== `setnodes` + +In case the host this program runs on is not part of the cluster, +it is necessary to set the list of nodes. + +Usage: +............... +setnodes node <node> [<node> ...] +............... +Example: +............... +setnodes node_a node_b +............... + +[[cmdhelp_history_show,show status or configuration of the PE input file]] +==== `show` + +Every transition is saved as a PE file. Use this command to +render that PE file either as configuration or status. The +configuration output is the same as `crm configure show`. + +Usage: +............... +show <pe> [status] + +pe :: <number>|<index>|<file>|live +............... +Examples: +............... +show 2066 +show pe-input-2080.bz2 status +............... + +[[cmdhelp_history_source,set source to be examined]] +==== `source` + +Events to be examined can come from the current cluster or from a +`hb_report` report. This command sets the source. `source live` +sets source to the running cluster and system logs. If no source +is specified, the current source information is printed. + +In case a report source is specified as a file reference, the file +is going to be unpacked in place where it resides. This directory +is not removed on exit. + +Usage: +............... +source [<dir>|<file>|live] +............... +Examples: +............... +source live +source /tmp/customer_case_22.tar.bz2 +source /tmp/customer_case_22 +source +............... + +[[cmdhelp_history_transition,show transition]] +==== `transition` + +This command will print actions planned by the PE and run +graphviz (`dotty`) to display a graphical representation of the +transition. Of course, for the latter an X11 session is required. +This command invokes `ptest(8)` in background. + +The +showdot+ subcommand runs graphviz (`dotty`) to display a +graphical representation of the +.dot+ file which has been +included in the report. Essentially, it shows the calculation +produced by `pengine` which is installed on the node where the +report was produced. In optimal case this output should not +differ from the one produced by the locally installed `pengine`. + +The `log` subcommand shows the full log for the duration of the +transition. + +A transition can also be saved to a CIB shadow for further +analysis or use with `cib` or `configure` commands (use the +`save` subcommand). The shadow file name defaults to the name of +the PE input file. + +If the PE input file number is not provided, it defaults to the +last one, i.e. the last transition. The last transition can also +be referenced with number 0. If the number is negative, then the +corresponding transition relative to the last one is chosen. + +If there are warning and error PE input files or different nodes +were the DC in the observed timeframe, it may happen that PE +input file numbers collide. In that case provide some unique part +of the path to the file. + +After the `ptest` output, logs about events that happened during +the transition are printed. + +The `tags` subcommand scans the logs for the transition and return a +list of key events during that transition. For example, the tag ++error+ will be returned if there are any errors logged during the +transition. + +Usage: +............... +transition [<number>|<index>|<file>] [nograph] [v...] [scores] [actions] [utilization] +transition showdot [<number>|<index>|<file>] +transition log [<number>|<index>|<file>] +transition save [<number>|<index>|<file> [name]] +transition tags [<number>|<index>|<file>] +............... +Examples: +............... +transition +transition 444 +transition -1 +transition pe-error-3.bz2 +transition node-a/pengine/pe-input-2.bz2 +transition showdot 444 +transition log +transition save 0 enigma-22 +............... + +[[cmdhelp_history_transitions,List transitions]] +==== `transitions` + +A transition represents a change in cluster configuration or +state. This command lists the transitions in the current timeframe. + +Usage: +............... +transitions +............... +Example: +............... +transitions +............... + + +[[cmdhelp_history_wdiff,cluster states/transitions difference]] +==== `wdiff` + +A transition represents a change in cluster configuration or +state. Use `wdiff` to see what has changed between two +transitions as word differences on a line-by-line basis. + +If you want to specify the current cluster configuration and +status, use the string +live+. + +Normally, the first transition specified should be the one which +is older, but we are not going to enforce that. + +Note that a single configuration update may result in more than +one transition. + +Usage: +............... +wdiff <pe> <pe> [status] + +pe :: <number>|<index>|<file>|live +............... +Examples: +............... +wdiff 2066 2067 +wdiff pe-input-2080.bz2 live status +............... + +[[cmdhelp_root_report,Create cluster status report]] +=== `report` + +Interface to a tool for creating a cluster report. A report is an +archive containing log files, configuration files, system information +and other relevant data for a given time period. This is a useful tool +for collecting data to attach to bug reports, or for detecting the +root cause of errors resulting in resource failover, for example. + +See `crmsh_hb_report(8)` for more details on arguments, +or call `crm report -h` + +Usage: +............... +report -f {time|"cts:"testnum} [-t time] [-u user] [-l file] + [-n nodes] [-E files] [-p patt] [-L patt] [-e prog] + [-MSDZAVsvhd] [dest] +............... + +Examples: +............... +report -f 2pm report_1 +report -f "2007/9/5 12:30" -t "2007/9/5 14:00" report_2 +report -f 1:00 -t 3:00 -l /var/log/cluster/ha-debug report_3 +report -f "09sep07 2:00" -u hbadmin report_4 +report -f 18:00 -p "usern.*" -p "admin.*" report_5 +report -f cts:133 ctstest_133 +............... + +=== `end` (`cd`, `up`) + +The `end` command ends the current level and the user moves to +the parent level. This command is available everywhere. + +Usage: +............... +end +............... + +=== `help` + +The `help` command prints help for the current level or for the +specified topic (command). This command is available everywhere. + +Usage: +............... +help [<topic>] +............... + +=== `quit` (`exit`, `bye`) + +Leave the program. + +BUGS +---- +Even though all sensible configurations (and most of those that +are not) are going to be supported by the crm shell, I suspect +that it may still happen that certain XML constructs may confuse +the tool. When that happens, please file a bug report. + +The crm shell will not try to update the objects it does not +understand. Of course, it is always possible to edit such objects +in the XML format. + +AUTHORS +------- +Dejan Muhamedagic, <dejan@suse.de> +Kristoffer Gronlund <kgronlund@suse.com> +and many OTHERS + +SEE ALSO +-------- +crm_resource(8), crm_attribute(8), crm_mon(8), cib_shadow(8), +ptest(8), dotty(1), crm_simulate(8), cibadmin(8) + + +COPYING +------- +Copyright \(C) 2008-2013 Dejan Muhamedagic. +Copyright \(C) 2013 Kristoffer Gronlund. + +Free use of this software is granted under the terms of the GNU General Public License (GPL). + +////////////////////// + vim:ts=4:sw=4:expandtab: +////////////////////// diff --git a/doc/website-v1/man-4.3.adoc b/doc/website-v1/man-4.3.adoc new file mode 100644 index 0000000..2b82298 --- /dev/null +++ b/doc/website-v1/man-4.3.adoc @@ -0,0 +1,5160 @@ +:man source: crm +:man version: 4.0.0 +:man manual: crmsh documentation + +crm(8) +====== + +NAME +---- +crm - Pacemaker command line interface for configuration and management + + +SYNOPSIS +-------- +*crm* [OPTIONS] [SUBCOMMAND ARGS...] + + +[[topics_Description,Program description]] +DESCRIPTION +----------- +The `crm` shell is a command-line based cluster configuration and +management tool. Its goal is to assist as much as possible with the +configuration and maintenance of Pacemaker-based High Availability +clusters. + +For more information on Pacemaker itself, see http://clusterlabs.org/. + +`crm` works both as a command-line tool to be called directly from the +system shell, and as an interactive shell with extensive tab +completion and help. + +The primary focus of the `crm` shell is to provide a simplified and +consistent interface to Pacemaker, but it also provides tools for +managing the creation and configuration of High Availability clusters +from scratch. To learn more about this aspect of `crm`, see the +`cluster` section below. + +The `crm` shell can be used to manage every aspect of configuring and +maintaining a cluster. It provides a simplified line-based syntax on +top of the XML configuration format used by Pacemaker, commands for +starting and stopping resources, tools for exploring the history of a +cluster including log scraping and a set of cluster scripts useful for +automating the setup and installation of services on the cluster +nodes. + +The `crm` shell is line oriented: every command must start and finish +on the same line. It is possible to use a continuation character (+\+) +to write one command in two or more lines. The continuation character +is commonly used when displaying configurations. + +[[topics_CommandLine,Command line options]] +OPTIONS +------- +*-f, --file*='FILE':: + Load commands from the given file. If a dash +-+ is used in place + of a file name, `crm` will read commands from the shell standard + input (`stdin`). + +*-c, --cib*='CIB':: + Start the session using the given shadow CIB file. + Equivalent to +cib use <CIB>+. + +*-D, --display=*'OUTPUT_TYPE':: + Choose one of the output options: +plain+, +color-always+, +color+, + or +uppercase+. The default is +color+ if the terminal emulation + supports colors. Otherwise, +plain+ is used. + +*-F, --force*:: + Make `crm` proceed with applying changes where it would normally + ask the user to confirm before proceeding. This option is mainly + useful in scripts, and should be used with care. + +*-w, --wait*:: + Make `crm` wait for the cluster transition to finish (for the + changes to take effect) after each processed line. + +*-H, --history*='DIR|FILE|SESSION':: + A directory or file containing a cluster report to load + into the `history` commands, or the name of a previously + saved history session. + +*-h, --help*:: + Print help page. + +*--version*:: + Print crmsh version and build information (Mercurial Hg changeset + hash). + +*-d, --debug*:: + Print verbose debugging information. + +*-R, --regression-tests*:: + Enables extra verbose trace logging used by the regression + tests. Logs all external calls made by crmsh. + +*--scriptdir*='DIR':: + Extra directory where crm looks for cluster scripts, or a list of + directories separated by semi-colons (e.g. +/dir1;/dir2;etc.+). + +*-o, --opt*='OPTION=VALUE':: + Set crmsh option temporarily. If the options are saved using + +options save+ then the value passed here will also be saved. + Multiple options can be set by using +-o+ multiple times. + +[[topics_Introduction,Introduction]] +== Introduction + +This section of the user guide covers general topics about the user +interface and describes some of the features of `crmsh` in detail. + +[[topics_Introduction_Interface,User interface]] +=== User interface + +The main purpose of `crmsh` is to provide a simple yet powerful +interface to the cluster stack. There are two main modes of operation +with the user interface of `crmsh`: + +* Command line (single-shot) use - Use `crm` as a regular UNIX command + from your usual shell. `crm` has full bash completion built in, so + using it in this manner should be as comfortable and familiar as + using any other command-line tool. + +* Interactive mode - By calling `crm` without arguments, or by calling + it with only a sublevel as argument, `crm` enters the interactive + mode. In this mode, it acts as its own command shell, which + remembers which sublevel you are currently in and allows for rapid + and convenient execution of multiple commands within the same + sublevel. This mode also has full tab completion, as well as + built-in interactive help and syntax highlighting. + +Here are a few examples of using `crm` both as a command-line tool and +as an interactive shell: + +.Command line (one-shot) use: +........ +# crm resource stop www_app +........ + +.Interactive use: +........ +# crm +crm(live)# resource +crm(live)resource# unmanage tetris_1 +crm(live)resource# up +crm(live)# node standby node4 +........ + +.Cluster configuration: +........ +# crm configure<<EOF + # + # resources + # + primitive disk0 iscsi \ + params portal=192.168.2.108:3260 target=iqn.2008-07.com.suse:disk0 + primitive fs0 Filesystem \ + params device=/dev/disk/by-label/disk0 directory=/disk0 fstype=ext3 + primitive internal_ip IPaddr params ip=192.168.1.101 + primitive apache apache \ + params configfile=/disk0/etc/apache2/site0.conf + primitive apcfence stonith:apcsmart \ + params ttydev=/dev/ttyS0 hostlist="node1 node2" \ + op start timeout=60s + primitive pingd pingd \ + params name=pingd dampen=5s multiplier=100 host_list="r1 r2" + # + # monitor apache and the UPS + # + monitor apache 60s:30s + monitor apcfence 120m:60s + # + # cluster layout + # + group internal_www \ + disk0 fs0 internal_ip apache + clone fence apcfence \ + meta globally-unique=false clone-max=2 clone-node-max=1 + clone conn pingd \ + meta globally-unique=false clone-max=2 clone-node-max=1 + location node_pref internal_www \ + rule 50: #uname eq node1 \ + rule pingd: defined pingd + # + # cluster properties + # + property stonith-enabled=true + commit +EOF +........ + +The `crm` interface is hierarchical, with commands organized into +separate levels by functionality. To list the available levels and +commands, either execute +help <level>+, or, if at the top level of +the shell, simply typing `help` will provide an overview of all +available levels and commands. + +The +(live)+ string in the `crm` prompt signifies that the current CIB +in use is the cluster live configuration. It is also possible to +work with so-called <<topics_Features_Shadows,shadow CIBs>>. These are separate, inactive +configurations stored in files, that can be applied and thereby +replace the live configuration at any time. + +[[topics_Introduction_Completion,Tab completion]] +=== Tab completion + +The `crm` makes extensive use of tab completion. The completion +is both static (i.e. for `crm` commands) and dynamic. The latter +takes into account the current status of the cluster or +information from installed resource agents. Sometimes, completion +may also be used to get short help on resource parameters. Here +are a few examples: + +............... +crm(live)resource# <TAB><TAB> +ban demote maintenance param scores trace +cd failcount manage promote secret unmanage +cleanup help meta quit start untrace +clear locate move refresh status up +constraints ls operations restart stop utilization + +crm(live)configure# primitive fence-1 <TAB><TAB> +lsb: ocf: service: stonith: systemd: + +crm(live)configure# primitive fence-1 stonith:<TAB><TAB> +apcmaster external/ippower9258 fence_legacy +apcmastersnmp external/kdumpcheck ibmhmc +apcsmart external/libvirt ipmilan + +crm(live)configure# primitive fence-1 stonith:ipmilan params <TAB><TAB> +auth= hostname= ipaddr= login= password= port= priv= + +crm(live)configure# primitive fence-1 stonith:ipmilan params auth=<TAB><TAB> +auth* (string) + The authorization type of the IPMI session ("none", "straight", "md2", or "md5") +............... + +`crmsh` also comes with bash completion usable directly from the +system shell. This should be installed automatically with the command +itself. + +[[topics_Introduction_Shorthand,Shorthand syntax]] +=== Shorthand syntax + +When using the `crm` shell to manage clusters, you will end up typing +a lot of commands many times over. Clear command names like ++configure+ help in understanding and learning to use the cluster +shell, but is easy to misspell and is tedious to type repeatedly. The +interactive mode and tab completion both help with this, but the `crm` +shell also has the ability to understand a variety of shorthand +aliases for all of the commands. + +For example, instead of typing `crm status`, you can type `crm st` or +`crm stat`. Instead of `crm configure` you can type `crm cfg` or even +`crm cf`. `crm resource` can be shorted as `crm rsc`, and so on. + +The exact list of accepted aliases is too long to print in full, but +experimentation and typos should help in discovering more of them. + +[[topics_Features,Features]] +== Features + +The feature set of crmsh covers a wide range of functionality, and +understanding how and when to use the various features of the shell +can be difficult. This section of the guide describes some of the +features and use cases of `crmsh` in more depth. The intention is to +provide a deeper understanding of these features, but also to serve as +a guide to using them. + +[[topics_Features_Shadows,Shadow CIB usage]] +=== Shadow CIB usage + +A Shadow CIB is a normal cluster configuration stored in a file. +They may be manipulated in much the same way as the _live_ CIB, with +the key difference that changes to a shadow CIB have no effect on the +actual cluster resources. An administrator may choose to apply any of +them to the cluster, thus replacing the running configuration with the +one found in the shadow CIB. + +The `crm` prompt always contains the name of the configuration which +is currently in use, or the string _live_ if using the live cluster +configuration. + +When editing the configuration in the `configure` level, no changes +are actually applied until the `commit` command is executed. It is +possible to start editing a configuration as usual, but instead of +committing the changes to the active CIB, save them to a shadow CIB. + +The following example `configure` session demonstrates how this can be +done: +............... +crm(live)configure# cib new test-2 +INFO: test-2 shadow CIB created +crm(test-2)configure# commit +............... + +[[topics_Features_Checks,Configuration semantic checks]] +=== Configuration semantic checks + +Resource definitions may be checked against the meta-data +provided with the resource agents. These checks are currently +carried out: + +- are required parameters set +- existence of defined parameters +- timeout values for operations + +The parameter checks are obvious and need no further explanation. +Failures in these checks are treated as configuration errors. + +The timeouts for operations should be at least as long as those +recommended in the meta-data. Too short timeout values are a +common mistake in cluster configurations and, even worse, they +often slip through if cluster testing was not thorough. Though +operation timeouts issues are treated as warnings, make sure that +the timeouts are usable in your environment. Note also that the +values given are just _advisory minimum_---your resources may +require longer timeouts. + +User may tune the frequency of checks and the treatment of errors +by the <<cmdhelp_options_check-frequency,`check-frequency`>> and +<<cmdhelp_options_check-mode,`check-mode`>> preferences. + +Note that if the +check-frequency+ is set to +always+ and the ++check-mode+ to +strict+, errors are not tolerated and such +configuration cannot be saved. + +[[topics_Features_Templates,Configuration templates]] +=== Configuration templates + +.Deprecation note +**************************** +Configuration templates have been deprecated in favor of the more +capable `cluster scripts`. To learn how to use cluster scripts, see +the dedicated documentation on the `crmsh` website at +http://crmsh.github.io/, or in the <<cmdhelp_script,Script section>>. +**************************** + +Configuration templates are ready made configurations created by +cluster experts. They are designed in such a way so that users +may generate valid cluster configurations with minimum effort. +If you are new to Pacemaker, templates may be the best way to +start. + +We will show here how to create a simple yet functional Apache +configuration: +............... +# crm configure +crm(live)configure# template +crm(live)configure template# list templates +apache filesystem virtual-ip +crm(live)configure template# new web <TAB><TAB> +apache filesystem virtual-ip +crm(live)configure template# new web apache +INFO: pulling in template apache +INFO: pulling in template virtual-ip +crm(live)configure template# list +web2-d web2 vip2 web3 vip web +............... + +We enter the `template` level from `configure`. Use the `list` +command to show templates available on the system. The `new` +command creates a configuration from the +apache+ template. You +can use tab completion to pick templates. Note that the apache +template depends on a virtual IP address which is automatically +pulled along. The `list` command shows the just created +web+ +configuration, among other configurations (I hope that you, +unlike me, will use more sensible and descriptive names). + +The `show` command, which displays the resulting configuration, +may be used to get an idea about the minimum required changes +which have to be done. All +ERROR+ messages show the line numbers +in which the respective parameters are to be defined: +............... +crm(live)configure template# show +ERROR: 23: required parameter ip not set +ERROR: 61: required parameter id not set +ERROR: 65: required parameter configfile not set +crm(live)configure template# edit +............... + +The `edit` command invokes the preferred text editor with the ++web+ configuration. At the top of the file, the user is advised +how to make changes. A good template should require from the user +to specify only parameters. For example, the +web+ configuration +we created above has the following required and optional +parameters (all parameter lines start with +%%+): +............... +$ grep -n ^%% ~/.crmconf/web +23:%% ip +31:%% netmask +35:%% lvs_support +61:%% id +65:%% configfile +71:%% options +76:%% envfiles +............... + +These lines are the only ones that should be modified. Simply +append the parameter value at the end of the line. For instance, +after editing this template, the result could look like this (we +used tabs instead of spaces to make the values stand out): +............... +$ grep -n ^%% ~/.crmconf/web +23:%% ip 192.168.1.101 +31:%% netmask +35:%% lvs_support +61:%% id websvc +65:%% configfile /etc/apache2/httpd.conf +71:%% options +76:%% envfiles +............... + +As you can see, the parameter line format is very simple: +............... +%% <name> <value> +............... + +After editing the file, use `show` again to display the +configuration: +............... +crm(live)configure template# show +primitive virtual-ip IPaddr \ + params ip=192.168.1.101 +primitive apache apache \ + params configfile="/etc/apache2/httpd.conf" +monitor apache 120s:60s +group websvc \ + apache virtual-ip +............... + +The target resource of the apache template is a group which we +named +websvc+ in this sample session. + +This configuration looks exactly as you could type it at the +`configure` level. The point of templates is to save you some +typing. It is important, however, to understand the configuration +produced. + +Finally, the configuration may be applied to the current +crm configuration (note how the configuration changed slightly, +though it is still equivalent, after being digested at the +`configure` level): +............... +crm(live)configure template# apply +crm(live)configure template# cd .. +crm(live)configure# show +node xen-b +node xen-c +primitive apache apache \ + params configfile="/etc/apache2/httpd.conf" \ + op monitor interval=120s timeout=60s +primitive virtual-ip IPaddr \ + params ip=192.168.1.101 +group websvc apache virtual-ip +............... + +Note that this still does not commit the configuration to the CIB +which is used in the shell, either the running one (+live+) or +some shadow CIB. For that you still need to execute the `commit` +command. + +To complete our example, we should also define the preferred node +to run the service: + +............... +crm(live)configure# location websvc-pref websvc 100: xen-b +............... + +If you are not happy with some resource names which are provided +by default, you can rename them now: + +............... +crm(live)configure# rename virtual-ip intranet-ip +crm(live)configure# show +node xen-b +node xen-c +primitive apache apache \ + params configfile="/etc/apache2/httpd.conf" \ + op monitor interval=120s timeout=60s +primitive intranet-ip IPaddr \ + params ip=192.168.1.101 +group websvc apache intranet-ip +location websvc-pref websvc 100: xen-b +............... + +To summarize, working with templates typically consists of the +following steps: + +- `new`: create a new configuration from templates +- `edit`: define parameters, at least the required ones +- `show`: see if the configuration is valid +- `apply`: apply the configuration to the `configure` level + +[[topics_Features_Testing,Resource testing]] +=== Resource testing + +The amount of detail in a cluster makes all configurations prone +to errors. By far the largest number of issues in a cluster is +due to bad resource configuration. The shell can help quickly +diagnose such problems. And considerably reduce your keyboard +wear. + +Let's say that we entered the following configuration: +............... +node xen-b +node xen-c +node xen-d +primitive fencer stonith:external/libvirt \ + params hypervisor_uri="qemu+tcp://10.2.13.1/system" \ + hostlist="xen-b xen-c xen-d" \ + op monitor interval=2h +primitive svc Xinetd \ + params service=systat \ + op monitor interval=30s +primitive intranet-ip IPaddr2 \ + params ip=10.2.13.100 \ + op monitor interval=30s +primitive apache apache \ + params configfile="/etc/apache2/httpd.conf" \ + op monitor interval=120s timeout=60s +group websvc apache intranet-ip +location websvc-pref websvc 100: xen-b +............... + +Before typing `commit` to submit the configuration to the cib we +can make sure that all resources are usable on all nodes: +............... +crm(live)configure# rsctest websvc svc fencer +............... + +It is important that resources being tested are not running on +any nodes. Otherwise, the `rsctest` command will refuse to do +anything. Of course, if the current configuration resides in a +CIB shadow, then a `commit` is irrelevant. The point being that +resources are not running on any node. + +.Note on stopping all resources +**************************** +Alternatively to not committing a configuration, it is also +possible to tell Pacemaker not to start any resources: + +............... +crm(live)configure# property stop-all-resources=yes +............... +Almost none---resources of class stonith are still started. But +shell is not as strict when it comes to stonith resources. +**************************** + +Order of resources is significant insofar that a resource depends +on all resources to its left. In most configurations, it's +probably practical to test resources in several runs, based on +their dependencies. + +Apart from groups, `crm` does not interpret constraints and +therefore knows nothing about resource dependencies. It also +doesn't know if a resource can run on a node at all in case of an +asymmetric cluster. It is up to the user to specify a list of +eligible nodes if a resource is not meant to run on every node. + +[[topics_Features_Security,Access Control Lists (ACL)]] +=== Access Control Lists (ACL) + +.Note on ACLs in Pacemaker 1.1.12 +**************************** +The support for ACLs has been revised in Pacemaker version 1.1.12 and +up. Depending on which version you are using, the information in this +section may no longer be accurate. Look for the `acl_target` +configuration element for more details on the new syntax. +**************************** + +By default, the users from the +haclient+ group have full access +to the cluster (or, more precisely, to the CIB). Access control +lists allow for finer access control to the cluster. + +Access control lists consist of an ordered set of access rules. +Each rule allows read or write access or denies access +completely. Rules are typically combined to produce a specific +role. Then, users may be assigned a role. + +For instance, this is a role which defines a set of rules +allowing management of a single resource: + +............... +role bigdb_admin \ + write meta:bigdb:target-role \ + write meta:bigdb:is-managed \ + write location:bigdb \ + read ref:bigdb +............... + +The first two rules allow modifying the +target-role+ and ++is-managed+ meta attributes which effectively enables users in +this role to stop/start and manage/unmanage the resource. The +constraints write access rule allows moving the resource around. +Finally, the user is granted read access to the resource +definition. + +For proper operation of all Pacemaker programs, it is advisable +to add the following role to all users: + +............... +role read_all \ + read cib +............... + +For finer grained read access try with the rules listed in the +following role: + +............... +role basic_read \ + read node attribute:uname \ + read node attribute:type \ + read property \ + read status +............... + +It is however possible that some Pacemaker programs (e.g. +`ptest`) may not function correctly if the whole CIB is not +readable. + +Some of the ACL rules in the examples above are expanded by the +shell to XPath specifications. For instance, ++meta:bigdb:target-role+ expands to: + +........ +//primitive[@id='bigdb']/meta_attributes/nvpair[@name='target-role'] +........ + +You can see the expansion by showing XML: + +............... +crm(live) configure# show xml bigdb_admin +... +<acls> + <acl_role id="bigdb_admin"> + <write id="bigdb_admin-write" + xpath="//primitive[@id='bigdb']/meta_attributes/nvpair[@name='target-role']"/> +............... + +Many different XPath expressions can have equal meaning. For +instance, the following two are equal, but only the first one is +going to be recognized as shortcut: + +............... +//primitive[@id='bigdb']/meta_attributes/nvpair[@name='target-role'] +//resources/primitive[@id='bigdb']/meta_attributes/nvpair[@name='target-role'] +............... + +XPath is a powerful language, but you should try to keep your ACL +xpaths simple and the builtin shortcuts should be used whenever +possible. + +[[topics_Features_Resourcesets,Syntax: Resource sets]] +=== Syntax: Resource sets + +Using resource sets can be a bit confusing unless one knows the +details of the implementation in Pacemaker as well as how to interpret +the syntax provided by `crmsh`. + +Three different types of resource sets are provided by `crmsh`, and +each one implies different values for the two resource set attributes, ++sequential+ and +require-all+. + ++sequential+:: + If false, the resources in the set do not depend on each other + internally. Setting +sequential+ to +true+ implies a strict order of + dependency within the set. + ++require-all+:: + If false, only one resource in the set is required to fulfil the + requirements of the set. The set of A, B and C with +require-all+ + set to +false+ is be read as "A OR B OR C" when its dependencies + are resolved. + +The three types of resource sets modify the attributes in the +following way: + +1. Implicit sets (no brackets). +sequential=true+, +require-all=true+ +2. Parenthesis set (+(+ ... +)+). +sequential=false+, +require-all=true+ +3. Bracket set (+[+ ... +]+). +sequential=false+, +require-all=false+ + +To create a set with the properties +sequential=true+ and ++require-all=false+, explicitly set +sequential+ in a bracketed set, ++[ A B C sequential=true ]+. + +To create multiple sets with both +sequential+ and +require-all+ set to +true, explicitly set +sequential+ in a parenthesis set: ++A B ( C D sequential=true )+. + +[[topics_Features_AttributeListReferences,Syntax: Attribute list references]] +=== Syntax: Attribute list references + +Attribute lists are used to set attributes and parameters for +resources, constraints and property definitions. For example, to set +the virtual IP used by an +IPAddr2+ resource the attribute +ip+ can be +set in an attribute list for that resource. + +Attribute lists can have identifiers that name them, and other +resources can reuse the same attribute list by referring to that name +using an +$id-ref+. For example, the following statement defines a +simple dummy resource with an attribute list which sets the parameter ++state+ to the value 1 and sets the identifier for the attribute list +to +on-state+: + +.............. +primitive dummy-1 Dummy params $id=on-state state=1 +.............. + +To refer to this attribute list from a different resource, refer to +the +on-state+ name using an id-ref: + +.............. +primitive dummy-2 Dummy params $id-ref=on-state +.............. + +The resource +dummy-2+ will now also have the parameter +state+ set to the value 1. + +[[topics_Features_AttributeReferences,Syntax: Attribute references]] +=== Syntax: Attribute references + +In some cases, referencing complete attribute lists is too +coarse-grained, for example if two different parameters with different +names should have the same value set. Instead of having to copy the +value in multiple places, it is possible to create references to +individual attributes in attribute lists. + +To name an attribute in order to be able to refer to it later, prefix +the attribute name with a +$+ character (as seen above with the +special names +$id+ and +$id-ref+: + +............ +primitive dummy-1 Dummy params $state=1 +............ + +The identifier +state+ can now be used to refer to this attribute from other +primitives, using the +@<id>+ syntax: + +............ +primitive dummy-2 Dummy params @state +............ + +In some cases, using the attribute name as the identifier doesn't work +due to name clashes. In this case, the syntax +$<id>:<name>=<value>+ +can be used to give the attribute a different identifier: + +............ +primitive dummy-1 params $dummy-state-on:state=1 +primitive dummy-2 params @dummy-state-on +............ + +There is also the possibility that two resources both use the same +attribute value but with different names. For example, a web server +may have a parameter +server_ip+ for setting the IP address where it +listens for incoming requests, and a virtual IP resource may have a +parameter called +ip+ which sets the IP address it creates. To +configure these two resources with an IP without repeating the value, +the reference can be given a name using the syntax +@<id>:<name>+. + +Example: +............ +primitive virtual-ip IPaddr2 params $vip:ip=192.168.1.100 +primitive webserver apache params @vip:server_ip +............ + +[[topics_Syntax_RuleExpressions,Syntax: Rule expressions]] +=== Syntax: Rule expressions + +Many of the configuration commands in `crmsh` now support the use of +_rule expressions_, which can influence what attributes apply to a +resource or under which conditions a constraint is applied, depending +on changing conditions like date, time, the value of attributes and +more. + +Here is an example of a simple rule expression used to apply a +a different resource parameter on the node named `node1`: + +.............. +primitive my_resource Special \ + params 2: rule #uname eq node1 interface=eth1 \ + params 1: interface=eth0 +.............. + +This primitive resource has two lists of parameters with descending +priority. The parameter list with the highest priority is applied +first, but only if the rule expressions for that parameter list all +apply. In this case, the rule `#uname eq node1` limits the parameter +list so that it is only applied on `node1`. + +Note that rule expressions are not terminated and are immediately +followed by the data to which the rule is applied. In this case, the +name-value pair `interface=eth1`. + +Rule expressions can contain multiple expressions connected using the +boolean operator `or` and `and`. The full syntax for rule expressions +is listed below. + +.............. +rules :: + rule [id_spec] [$role=<role>] <score>: <expression> + [rule [id_spec] [$role=<role>] <score>: <expression> ...] + +id_spec :: $id=<id> | $id-ref=<id> +score :: <number> | <attribute> | [-]inf +expression :: <simple_exp> [<bool_op> <simple_exp> ...] +bool_op :: or | and +simple_exp :: <attribute> [type:]<binary_op> <value> + | <unary_op> <attribute> + | date <date_expr> +type :: <string> | <version> | <number> +binary_op :: lt | gt | lte | gte | eq | ne +unary_op :: defined | not_defined + +date_expr :: lt <end> + | gt <start> + | in start=<start> end=<end> + | in start=<start> <duration> + | spec <date_spec> +duration|date_spec :: + hours=<value> + | monthdays=<value> + | weekdays=<value> + | yearsdays=<value> + | months=<value> + | weeks=<value> + | years=<value> + | weekyears=<value> + | moon=<value> +.............. + +[[topics_Lifetime,Lifetime parameter format]] +== Lifetime parameter format + +Lifetimes can be specified in the ISO 8601 time format or the ISO 8601 +duration format. To distinguish between months and minutes, use the PT +prefix before specifying minutes. The duration format is one of ++PnYnMnDTnHnMnS+, +PnW+, +P<date>T<time>+. + +P = duration. Y = year. M = month. W = week. D = day. T = time. H = +hour. M = minute. S = second. + +Examples: +................. +PT5M = 5 minutes later. +3D = 3 days later. +PT1H = 1 hour later. +................. + +The cluster checks lifetimes at an interval defined by the +cluster-recheck-interval property (default 15 minutes). + + +[[topics_Reference,Command reference]] +== Command reference + +The commands are structured to be compatible with the shell command +line. Sometimes, the underlying Pacemaker grammar uses characters that +have special meaning in bash, that will need to be quoted. This +includes the hash or pound sign (`#`), single and double quotes, and +any significant whitespace. + +Whitespace is also significant when assigning values, meaning that ++key=value+ is different from +key = value+. + +Commands can be referenced using short-hand as long as the short-hand +is unique. This can be either a prefix of the command name or a prefix +string of characters found in the name. + +For example, +status+ can be abbreviated as +st+ or +su+, and ++configure+ as +conf+ or +cfg+. + +The syntax for the commands is given below in an informal, BNF-like +grammar. + +* `<value>` denotes a string. +* `[value]` means that the construct is optional. +* The ellipsis (`...`) signifies that the previous construct may be + repeated. +* `first|second` means either first or second. +* The rest are literals (strings, `:`, `=`). + +[[cmdhelp_root_status,Cluster status]] +=== `status` + +Show cluster status. The status is displayed by `crm_mon`. Supply +additional arguments for more information or different format. +See `crm_mon(8)` for more details. + +Example: +............... +status +status simple +status full +............... + +Usage: +............... +status [<option> ...] + +option :: full + | bynode + | inactive + | ops + | timing + | failcounts + | verbose + | quiet + | html + | xml + | simple + | tickets + | noheaders + | detail + | brief +............... + +[[cmdhelp_root_verify,Verify cluster status]] +=== `verify` + +Performs basic checks for the cluster configuration and +current status, reporting potential issues. + +See `crm_verify(8)` and `crm_simulate(8)` for more details. + +Example: +............... +verify +verify scores +............... + +Usage: +............... +verify [scores] +............... + +[[cmdhelp_cluster,Cluster setup and management]] +=== `cluster` - Cluster setup and management + +Whole-cluster configuration management with High Availability +awareness. + +The commands on the cluster level allows configuration and +modification of the underlying cluster infrastructure, and also +supplies tools to do whole-cluster systems management. + +These commands enable easy installation and maintenance of a HA +cluster, by providing support for package installation, configuration +of the cluster messaging layer, file system setup and more. + +[[cmdhelp_cluster_add,Add a new node to the cluster,From Code]] +==== `add` +See "crm cluster help add" or "crm cluster add --help" + +[[cmdhelp_cluster_copy,Copy file to other cluster nodes]] +==== `copy` + +Copy file to other cluster nodes. + +Copies the given file to all other nodes unless given a +list of nodes to copy to as argument. + +Usage: +............... +copy <filename> [nodes ...] +............... + +Example: +............... +copy /etc/motd +............... + +[[cmdhelp_cluster_diff,Diff file across cluster]] +==== `diff` + +Displays the difference, if any, between a given file +on different nodes. If the second argument is `--checksum`, +a checksum of the file will be calculated and displayed for +each node. + +Usage: +............... +diff <file> [--checksum] [nodes...] +............... + +Example: +............... +diff /etc/crm/crm.conf node2 +diff /etc/resolv.conf --checksum +............... + +[[cmdhelp_cluster_disable,Disable cluster services]] +==== `disable` + +Disable the cluster-related system services on this node. + +Usage: +............... +disable +............... + +[[cmdhelp_cluster_enable,Enable cluster services]] +==== `enable` + +Enable the cluster-related system services on this node. + +Usage: +............... +enable +............... + +[[cmdhelp_cluster_geo_init,Configure cluster as geo cluster,From Code]] +==== `geo-init` +See "crm cluster help geo_init" or "crm cluster geo_init --help" + +[[cmdhelp_cluster_geo_init_arbitrator,Initialize node as geo cluster arbitrator,From Code]] +==== `geo-init-arbitrator` +See "crm cluster help geo_init_arbitrator" or "crm cluster geo_init_arbitrator --help" + +[[cmdhelp_cluster_geo_join,Join cluster to existing geo cluster,From Code]] +==== `geo-join` +See "crm cluster help geo_join" or "crm cluster geo_join --help" + +[[cmdhelp_cluster_health,Cluster health check]] +==== `health` + +Runs a larger set of tests and queries on all nodes in the cluster to +verify the general system health and detect potential problems. + +Usage: +............... +health +............... + +[[cmdhelp_cluster_init,Initializes a new HA cluster,From Code]] +==== `init` +See "crm cluster help init" or "crm cluster init --help" + +[[cmdhelp_cluster_join,Join existing cluster,From Code]] +==== `join` +See "crm cluster help join" or "crm cluster join --help" + +[[cmdhelp_cluster_remove,Remove node(s) from the cluster,From Code]] +==== `remove` +See "crm cluster help remove" or "crm cluster remove --help" + +[[cmdhelp_cluster_crash_test,Cluster crash test tool set,From Code]] +==== `crash_test` +See "crm cluster help crash_test" or "crm cluster crash_test --help" + +[[cmdhelp_cluster_restart,Restart cluster services]] +==== `restart` + +Restarts the cluster-related system services on this node. + +Usage: +......... +restart +......... + +[[cmdhelp_cluster_rename,Rename the cluster]] +==== `rename` + +Rename the cluster name + +Usage: +............... +rename <new_cluster_name> +............... + + +[[cmdhelp_cluster_run,Execute an arbitrary command on all nodes/specific node]] +==== `run` + +This command takes a shell statement as argument, executes that +statement on all nodes in the cluster or a specific node, +and reports the result. + +Usage: +............... +run <command> [node ...] +............... + +Example: +............... +run "cat /proc/uptime" +run "ls" node1 node2 +............... + +[[cmdhelp_cluster_start,Start cluster services]] +==== `start` + +Starts the cluster-related system services on this node. + +Usage: +......... +start +......... + +[[cmdhelp_cluster_status,Cluster status check]] +==== `status` + +Reports the status for the cluster messaging layer on the local +node. + +Usage: +............... +status +............... + +[[cmdhelp_cluster_stop,Stop cluster services]] +==== `stop` + +Stops the cluster-related system services on this node. + +Usage: +......... +stop +......... + +[[cmdhelp_cluster_wait_for_startup,Wait for cluster to start]] +==== `wait_for_startup` + +Mostly useful in scripts or automated workflows, this command will +attempt to connect to the local cluster node repeatedly. The command +will keep trying until the cluster node responds, or the `timeout` +elapses. The timeout can be changed by supplying a value in seconds as +an argument. + +Usage: +........ +wait_for_startup +........ + +[[cmdhelp_script,Cluster script management]] +=== `script` - Cluster script management + +A big part of the configuration and management of a cluster is +collecting information about all cluster nodes and deploying changes +to those nodes. Often, just performing the same procedure on all nodes +will encounter problems, due to subtle differences in the +configuration. + +For example, when configuring a cluster for the first time, the +software needs to be installed and configured on all nodes before the +cluster software can be launched and configured using `crmsh`. This +process is cumbersome and error-prone, and the goal is for scripts to +make this process easier. + +Scripts are implemented using the python `parallax` package which +provides a thin wrapper on top of SSH. This allows the scripts to +function through the usual SSH channels used for system maintenance, +requiring no additional software to be installed or maintained. + +[[cmdhelp_script_json,JSON API for cluster scripts]] +==== `json` + +This command provides a JSON API for the cluster scripts, intended for +use in user interface tools that want to interact with the cluster via +scripts. + +The command takes a single argument, which should be a JSON array with +the first member identifying the command to perform. + +The output is line-based: Commands that return multiple results will +return them line-by-line, ending with a terminator value: "end". + +When providing parameter values to this command, they should be +provided as nested objects, so +virtual-ip:ip=192.168.0.5+ on the +command line becomes the JSON object ++{"virtual-ip":{"ip":"192.168.0.5"}}+. + +API: +........ +["list"] +=> [{name, shortdesc, category}] + +["show", <name>] +=> [{name, shortdesc, longdesc, category, <<steps>>}] + +<<steps>> := [{name, shortdesc], longdesc, required, parameters, steps}] + +<<params>> := [{name, shortdesc, longdesc, required, unique, advanced, + type, value, example}] + +["verify", <name>, <<values>>] +=> [{shortdesc, longdesc, text, nodes}] + +["run", <name>, <<values>>] +=> [{shortdesc, rc, output|error}] +........ + + +[[cmdhelp_script_list,List available scripts]] +==== `list` + +Lists the available scripts, sorted by category. Scripts that have the +special `Script` category are hidden by default, since they are mainly +used by other scripts or commands. To also show these, pass `all` as +argument. + +To get a flat list of script names, not sorted by category, pass +`names` as an extra argument. + +Usage: +............ +list [all] [names] +............ + +Example: +............ +list +list all names +............ + +[[cmdhelp_script_run,Run the script]] +==== `run` + +Given a list of parameter values, this command will execute the +actions specified by the cluster script. The format for the parameter +values is the same as for the `verify` command. + +Can optionally take at least two parameters: +* `nodes=<nodes>`: List of nodes that the script runs over +* `dry_run=yes|no`: If set, the script will not perform any modifications. + +Additional parameters may be available depending on the script. + +Use the `show` command to see what parameters are available. + +Usage: +............. +run <script> [args...] +............. + +Example: +............. +run apache install=true +run sbd id=sbd-1 node=node1 sbd_device=/dev/disk/by-uuid/F00D-CAFE +............. + +[[cmdhelp_script_show,Describe the script]] +==== `show` + +Prints a description and short summary of the script, with +descriptions of the accepted parameters. + +Advanced parameters are hidden by default. To show the complete list +of parameters accepted by the script, pass `all` as argument. + +Usage: +............ +show <script> [all] +............ + +Example: +............ +show virtual-ip +............ + +[[cmdhelp_script_verify,Verify the script]] +==== `verify` + +Checks the given parameter values, and returns a list +of actions that will be executed when running the script +if provided the same list of parameter values. + +Usage: +............ +verify <script> [args...] +............ + +Example: +............ +verify sbd id=sbd-1 node=node1 sbd_device=/dev/disk/by-uuid/F00D-CAFE +............ + +[[cmdhelp_corosync,Corosync management]] +=== `corosync` - Corosync management + +Corosync is the underlying messaging layer for most HA clusters. +This level provides commands for editing and managing the corosync +configuration. + +[[cmdhelp_corosync_add-node,Add a corosync node]] +==== `add-node` + +Adds a node to the corosync configuration. This is used with the `udpu` +type configuration in corosync. + +A nodeid for the added node is generated automatically. + +Note that this command assumes that only a single ring is used, and +sets only the address for ring0. + +Usage: +......... +add-node <addr> [name] +......... + +[[cmdhelp_corosync_del-node,Remove a corosync node]] +==== `del-node` + +Removes a node from the corosync configuration. The argument given is +the `ring0_addr` address set in the configuration file. + +Usage: +......... +del-node <addr> +......... + +[[cmdhelp_corosync_diff,Diffs the corosync configuration]] +==== `diff` + +Diffs the corosync configurations on different nodes. If no nodes are +given as arguments, the corosync configurations on all nodes in the +cluster are compared. + +`diff` takes an option argument `--checksum`, to display a checksum +for each file instead of calculating a diff. + +Usage: +......... +diff [--checksum] [node...] +......... + +[[cmdhelp_corosync_edit,Edit the corosync configuration]] +==== `edit` + +Opens the Corosync configuration file in an editor. + +Usage: +......... +edit +......... + +[[cmdhelp_corosync_get,Get a corosync configuration value]] +==== `get` + +Returns the value configured in `corosync.conf`, which is not +necessarily the value used in the running configuration. See `reload` +for telling corosync about configuration changes. + +The argument is the complete dot-separated path to the value. + +If there are multiple values configured with the same path, the +command returns all values for that path. For example, to get all +configured `ring0_addr` values, use this command: + +Example: +........ +get nodelist.node.ring0_addr +........ + +[[cmdhelp_corosync_log,Show the corosync log file]] +==== `log` + +Opens the log file specified in the corosync configuration file. If no +log file is configured, this command returns an error. + +The pager used can be configured either using the PAGER +environment variable or in `crm.conf`. + +Usage: +......... +log +......... + +[[cmdhelp_corosync_pull,Pulls the corosync configuration]] +==== `pull` + +Gets the corosync configuration from another node and copies +it to this node. + +Usage: +......... +pull <node> +......... + +[[cmdhelp_corosync_push,Push the corosync configuration]] +==== `push` + +Pushes the corosync configuration file on this node to +the list of nodes provided. If no target nodes are given, +the configuration is pushed to all other nodes in the cluster. + +It is recommended to use `csync2` to distribute the cluster +configuration files rather than relying on this command. + +Usage: +......... +push [node] ... +......... + +Example: +......... +push node-2 node-3 +......... + +[[cmdhelp_corosync_reload,Reload the corosync configuration]] +==== `reload` + +Tells all instances of corosync in this cluster to reload +`corosync.conf`. + +After pushing a new configuration to all cluster nodes, call this +command to make corosync use the new configuration. + +Usage: +......... +reload +......... + +[[cmdhelp_corosync_set,Set a corosync configuration value]] +==== `set` + +Sets the value identified by the given path. If the value does not +exist in the configuration file, it will be added. However, if the +section containing the value does not exist, the command will fail. + +Usage: +......... +set quorum.expected_votes 2 +......... + +[[cmdhelp_corosync_show,Display the corosync configuration]] +==== `show` + +Displays the corosync configuration on the current node. + +......... +show +......... + +[[cmdhelp_corosync_status,Display the corosync status]] +==== `status` + +Displays the corosync ring status(default), also can display quorum/qdevice/qnetd status. + +Usage: +......... +status [ring|quorum|qdevice|qnetd] +......... + +[[cmdhelp_cib,CIB shadow management]] +=== `cib` - CIB shadow management + +This level is for management of shadow CIBs. It is available both +at the top level and the `configure` level. + +All the commands are implemented using `cib_shadow(8)` and the +`CIB_shadow` environment variable. The user prompt always +includes the name of the currently active shadow or the live CIB. + +[[cmdhelp_cib_cibstatus,CIB status management and editing]] +==== `cibstatus` + +Enter edit and manage the CIB status section level. See the +<<cmdhelp_cibstatus,CIB status management section>>. + +[[cmdhelp_cib_commit,copy a shadow CIB to the cluster]] +==== `commit` + +Apply a shadow CIB to the cluster. If the shadow name is omitted +then the current shadow CIB is applied. + +Temporary shadow CIBs are removed automatically on commit. + +Usage: +............... +commit [<cib>] +............... + +[[cmdhelp_cib_delete,delete a shadow CIB]] +==== `delete` + +Delete an existing shadow CIB. + +Usage: +............... +delete <cib> +............... + +[[cmdhelp_cib_diff,diff between the shadow CIB and the live CIB]] +==== `diff` + +Print differences between the current cluster configuration and +the active shadow CIB. + +Usage: +............... +diff +............... + +[[cmdhelp_cib_import,import a CIB or PE input file to a shadow]] +==== `import` + +At times it may be useful to create a shadow file from the +existing CIB. The CIB may be specified as file or as a PE input +file number. The shell will look up files in the local directory +first and then in the PE directory (typically `/var/lib/pengine`). +Once the CIB file is found, it is copied to a shadow and this +shadow is immediately available for use at both `configure` and +`cibstatus` levels. + +If the shadow name is omitted then the target shadow is named +after the input CIB file. + +Note that there are often more than one PE input file, so you may +need to specify the full name. + +Usage: +............... +import {<file>|<number>} [<shadow>] +............... +Examples: +............... +import pe-warn-2222 +import 2289 issue2 +............... + +[[cmdhelp_cib_list,list all shadow CIBs]] +==== `list` + +List existing shadow CIBs. + +Usage: +............... +list +............... + +[[cmdhelp_cib_new,create a new shadow CIB]] +==== `new` + +Create a new shadow CIB. The live cluster configuration and +status is copied to the shadow CIB. + +If the name of the shadow is omitted, we create a temporary CIB +shadow. It is useful if multiple level sessions are desired +without affecting the cluster. A temporary CIB shadow is short +lived and will be removed either on `commit` or on program exit. +Note that if the temporary shadow is not committed all changes in +the temporary shadow are lost. + +Specify `withstatus` if you want to edit the status section of +the shadow CIB (see the <<cmdhelp_cibstatus,cibstatus section>>). +Add `force` to force overwriting the existing shadow CIB. + +To start with an empty configuration that is not copied from the live +CIB, specify the `empty` keyword. (This also allows a shadow CIB to be +created in case no cluster is running.) + +Usage: +............... +new [<cib>] [withstatus] [force] [empty] +............... + +[[cmdhelp_cib_reset,copy live cib to a shadow CIB]] +==== `reset` + +Copy the current cluster configuration into the shadow CIB. + +Usage: +............... +reset <cib> +............... + +[[cmdhelp_cib_use,change working CIB]] +==== `use` + +Choose a CIB source. If you want to edit the status from the +shadow CIB specify `withstatus` (see <<cmdhelp_cibstatus,`cibstatus`>>). +Leave out the CIB name to switch to the running CIB. + +Usage: +............... +use [<cib>] [withstatus] +............... + +[[cmdhelp_ra,Resource Agents (RA) lists and documentation]] +=== `ra` - Resource Agents (RA) lists and documentation + +This level contains commands which show various information about +the installed resource agents. It is available both at the top +level and at the `configure` level. + +[[cmdhelp_ra_classes,list classes and providers]] +==== `classes` + +Print all resource agents' classes and, where appropriate, a list +of available providers. + +Usage: +............... +classes +............... + +[[cmdhelp_ra_info,show meta data for a RA]] +==== `info` (`meta`) + +Show the meta-data of a resource agent type. This is where users +can find information on how to use a resource agent. It is also +possible to get information from some programs: `pengine`, +`crmd`, `cib`, and `stonithd`. Just specify the program name +instead of an RA. + +Usage: +............... +info [<class>:[<provider>:]]<type> +info <type> <class> [<provider>] (obsolete) +............... +Example: +............... +info apache +info ocf:pacemaker:Dummy +info stonith:ipmilan +info pengine +............... + +[[cmdhelp_ra_list,list RA for a class (and provider)]] +==== `list` + +List available resource agents for the given class. If the class +is `ocf`, supply a provider to get agents which are available +only from that provider. + +Usage: +............... +list <class> [<provider>] +............... +Example: +............... +list ocf pacemaker +............... + +[[cmdhelp_ra_providers,show providers for a RA and a class]] +==== `providers` + +List providers for a resource agent type. The class parameter +defaults to `ocf`. + +Usage: +............... +providers <type> [<class>] +............... +Example: +............... +providers apache +............... + +[[cmdhelp_ra_validate,validate parameters for RA]] +==== `validate` + +If the resource agent supports the `validate-all` action, this calls +the action with the given parameters, printing any warnings or errors +reported by the agent. + +Usage: +................ +validate <agent> [<key>=<value> ...] +................ + +[[cmdhelp_resource,Resource management]] +=== `resource` - Resource management + +At this level resources may be managed. + +All (or almost all) commands are implemented with the CRM tools +such as `crm_resource(8)`. + +[[cmdhelp_resource_ban,ban a resource from a node]] +==== `ban` + +Ban a resource from running on a certain node. If no node is given +as argument, the resource is banned from the current location. + +See `move` for details on other arguments. + +Usage: +............... +ban <rsc> [<node>] [<lifetime>] [force] +............... + +[[cmdhelp_resource_cleanup,cleanup resource status]] +==== `cleanup` + +If resource has any past failures, clear its history and fail +count. Typically done after the resource has temporarily +failed. + +If a node is omitted, cleanup on all nodes. + ++(Pacemaker 1.1.14)+ Pass force to cleanup the resource itself, +otherwise the cleanup command will apply to the parent resource (if +any). + +Usage: +............... +cleanup [<rsc>] [<node>] [force] +............... + +[[cmdhelp_resource_clear,Clear any relocation constraint]] +==== `clear` (`unmove`, `unmigrate`, `unban`) + +Remove any relocation constraint created by +the `move`, `migrate` or `ban` command. + +Usage: +............... +clear <rsc> +unmigrate <rsc> +unban <rsc> +............... + +[[cmdhelp_resource_constraints,Show constraints affecting a resource]] +==== `constraints` + +Display the location and colocation constraints affecting the +resource. + +Usage: +................ +constraints <rsc> +................ + +[[cmdhelp_resource_demote,demote a master-slave resource]] +==== `demote` + +Demote a master-slave resource using the `target-role` +attribute. + +Usage: +............... +demote <rsc> +............... + +[[cmdhelp_resource_failcount,manage failcounts]] +==== `failcount` + +Show/edit/delete the failcount of a resource. +When `set` a non-zero value, `operation` and `interval` should be +provided when multiple operation failcount entries exist. +`interval` is a value in seconds. + +Usage: +............... +failcount <rsc> set <node> <value> [operation] [interval] +failcount <rsc> delete <node> +failcount <rsc> show <node> +............... +Example: +............... +failcount fs_0 delete node2 +............... + +[[cmdhelp_resource_locate,show the location of resources]] +==== `locate` + +Show the current location of one or more resources. + +Usage: +............... +locate [<rsc> ...] +............... + +[[cmdhelp_resource_maintenance,Enable/disable per-resource maintenance mode]] +==== `maintenance` + +Enables or disables the per-resource maintenance mode. When this mode +is enabled, no monitor operations will be triggered for the resource. +`maintenance` attribute conflicts with the `is-managed`. When setting +the `maintenance` attribute, the user is proposed to remove the +`is-managed` attribute if it exists. + +Usage: +.................. +maintenance <resource> [on|off|true|false] +.................. + +Example: +.................. +maintenance rsc1 +maintenance rsc2 off +.................. + +[[cmdhelp_resource_manage,put a resource into managed mode]] +==== `manage` + +Manage a resource using the `is-managed` attribute. If there +are multiple meta attributes sets, the attribute is set in all of +them. If the resource is a clone, all `is-managed` attributes are +removed from the children resources. +`is-managed` attribute conflicts with the `maintenance`. When setting +the `is-managed` attribute, the user is proposed to remove the +`maintenance` attribute if it exists. + +For details on group management see <<cmdhelp_options_manage-children,`options manage-children`>>. + +Usage: +............... +manage <rsc> +............... + +[[cmdhelp_resource_meta,manage a meta attribute]] +==== `meta` + +Show/edit/delete a meta attribute of a resource. Currently, all +meta attributes of a resource may be managed with other commands +such as `resource stop`. + +Usage: +............... +meta <rsc> set <attr> <value> +meta <rsc> delete <attr> +meta <rsc> show <attr> +............... +Example: +............... +meta ip_0 set target-role stopped +............... + +[[cmdhelp_resource_move,Move a resource to another node]] +==== `move` (`migrate`) + +Move a resource away from its current location. + +If the destination node is left out, the resource is migrated by +creating a constraint which prevents it from running on the current +node. For this type of constraint to be created, the +force+ argument +is required. + +A lifetime may be given for the constraint. Once it expires, the +location constraint will no longer be active. + +Usage: +............... +move <rsc> [<node>] [<lifetime>] [force] +............... + +[[cmdhelp_resource_operations,Show active resource operations]] +==== `operations` + +Show active operations, optionally filtered by resource and node. + +Usage: +................ +operations [<rsc>] [<node>] +................ + +[[cmdhelp_resource_param,manage a parameter of a resource]] +==== `param` + +Show/edit/delete a parameter of a resource. + +Usage: +............... +param <rsc> set <param> <value> +param <rsc> delete <param> +param <rsc> show <param> +............... +Example: +............... +param ip_0 show ip +............... + +[[cmdhelp_resource_promote,promote a master-slave resource]] +==== `promote` + +Promote a master-slave resource using the `target-role` +attribute. + +Usage: +............... +promote <rsc> +............... + +[[cmdhelp_resource_refresh,Recheck current resource status and drop failure history]] +==== `refresh` + +Delete resource's history (including failures) so its current state is rechecked. + +Usage: +............... +refresh [<rsc>] [<node>] [force] +............... + +[[cmdhelp_resource_restart,restart resources]] +==== `restart` + +Restart one or more resources. This is essentially a shortcut for +resource stop followed by a start. The shell is first going to wait +for the stop to finish, that is for all resources to really stop, and +only then to order the start action. Due to this command +entailing a whole set of operations, informational messages are +printed to let the user see some progress. + +For details on group management see +<<cmdhelp_options_manage-children,`options manage-children`>>. + +Usage: +............... +restart <rsc> [<rsc> ...] +............... +Example: +............... +# crm resource restart g_webserver +INFO: ordering g_webserver to stop +waiting for stop to finish .... done +INFO: ordering g_webserver to start +# +............... + +[[cmdhelp_resource_scores,Display resource scores]] +==== `scores` + +Display the allocation scores for all resources. + +Usage: +................ +scores +................ + +[[cmdhelp_resource_secret,manage sensitive parameters]] +==== `secret` + +Sensitive parameters can be kept in local files rather than CIB +in order to prevent accidental data exposure. Use the `secret` +command to manage such parameters. `stash` and `unstash` move the +value from the CIB and back to the CIB respectively. The `set` +subcommand sets the parameter to the provided value. `delete` +removes the parameter completely. `show` displays the value of +the parameter from the local file. Use `check` to verify if the +local file content is valid. + +Usage: +............... +secret <rsc> set <param> <value> +secret <rsc> stash <param> +secret <rsc> unstash <param> +secret <rsc> delete <param> +secret <rsc> show <param> +secret <rsc> check <param> +............... +Example: +............... +secret fence_1 show password +secret fence_1 stash password +secret fence_1 set password secret_value +............... + +[[cmdhelp_resource_start,start resources]] +==== `start` + +Start one or more resources by setting the `target-role` attribute. If +there are multiple meta attributes sets, the attribute is set in all +of them. If the resource is a clone, all `target-role` attributes are +removed from the children resources. + +For details on group management see +<<cmdhelp_options_manage-children,`options manage-children`>>. + +Usage: +............... +start <rsc> [<rsc> ...] +............... + +[[cmdhelp_resource_status,show status of resources]] +==== `status` (`show`, `list`) + +Print resource status. More than one resource can be shown at once. If +the resource parameter is left out, the status of all resources is +printed. + +Usage: +............... +status [<rsc> ...] +............... + +[[cmdhelp_resource_stop,stop resources]] +==== `stop` + +Stop one or more resources using the `target-role` attribute. If there +are multiple meta attributes sets, the attribute is set in all of +them. If the resource is a clone, all `target-role` attributes are +removed from the children resources. + +For details on group management see +<<cmdhelp_options_manage-children,`options manage-children`>>. + +Usage: +............... +stop <rsc> [<rsc> ...] +............... + +[[cmdhelp_resource_trace,start RA tracing]] +==== `trace` + +Start tracing RA for the given operation. The trace files are +stored in `$HA_VARLIB/trace_ra`. If the operation to be traced is +monitor, note that the number of trace files can grow very +quickly. + +If no operation name is given, crmsh will attempt to trace all +operations for the RA. This includes any configured operations, start +and stop as well as promote/demote for multistate resources. + +To trace the probe operation which exists for all resources, either +set a trace for `monitor` with interval `0`, or use `probe` as the +operation name. + +Note: RA tracing is only supported by OCF resource agents; +The pacemaker-execd daemon does not log recurring monitor operations +unless an error occurred. + +Usage: +............... +trace <rsc> [<op> [<interval>] ] +............... +Example: +............... +trace fs start +trace webserver +trace webserver probe +trace fs monitor 0 +............... + +[[cmdhelp_resource_unmanage,put a resource into unmanaged mode]] +==== `unmanage` + +Unmanage a resource using the `is-managed` attribute. If there +are multiple meta attributes sets, the attribute is set in all of +them. If the resource is a clone, all `is-managed` attributes are +removed from the children resources. + +For details on group management see <<cmdhelp_options_manage-children,`options manage-children`>>. + +Usage: +............... +unmanage <rsc> +............... + +[[cmdhelp_resource_untrace,stop RA tracing]] +==== `untrace` + +Stop tracing RA for the given operation. If no operation name is +given, crmsh will attempt to stop tracing all operations in resource. + +Usage: +............... +untrace <rsc> [<op> [<interval>] ] +............... +Example: +............... +untrace fs start +untrace webserver +............... + +[[cmdhelp_resource_utilization,manage a utilization attribute]] +==== `utilization` + +Show/edit/delete a utilization attribute of a resource. These +attributes describe hardware requirements. By setting the +`placement-strategy` cluster property appropriately, it is +possible then to distribute resources based on resource +requirements and node size. See also <<cmdhelp_node_utilization,node utilization attributes>>. + +Usage: +............... +utilization <rsc> set <attr> <value> +utilization <rsc> delete <attr> +utilization <rsc> show <attr> +............... +Example: +............... +utilization xen1 set memory 4096 +............... + +[[cmdhelp_node,Node management]] +=== `node` - Node management + +Node management and status commands. + +[[cmdhelp_node_attribute,manage attributes]] +==== `attribute` + +Edit node attributes. This kind of attribute should refer to +relatively static properties, such as memory size. + +Usage: +............... +attribute <node> set <attr> <value> +attribute <node> delete <attr> +attribute <node> show <attr> +............... +Example: +............... +attribute node_1 set memory_size 4096 +............... + +[[cmdhelp_node_clearstate,Clear node state]] +==== `clearstate` + +Resets and clears the state of the specified node. This node is +afterwards assumed clean and offline. This command can be used to +manually confirm that a node has been fenced (e.g., powered off). + +Be careful! This can cause data corruption if you confirm that a node is +down that is, in fact, not cleanly down - the cluster will proceed as if +the fence had succeeded, possibly starting resources multiple times. + +Usage: +............... +clearstate <node> +............... + +[[cmdhelp_node_delete,delete node]] +==== `delete` + +Delete a node. This command will remove the node from the CIB +and, in case the cluster stack is running, use the appropriate +program (`crm_node` or `hb_delnode`) to remove the node from the +membership. + +If the node is still listed as active and a member of our +partition we refuse to remove it. With the global force option +(`-F`) we will try to delete the node anyway. + +Usage: +............... +delete <node> +............... + +[[cmdhelp_node_fence,fence node]] +==== `fence` + +Make CRM fence a node. This functionality depends on stonith +resources capable of fencing the specified node. No such stonith +resources, no fencing will happen. + +Usage: +............... +fence <node> +............... + +[[cmdhelp_node_maintenance,put node into maintenance mode]] +==== `maintenance` + +Set the node status to maintenance. This is equivalent to the +cluster-wide `maintenance-mode` property but puts just one node +into the maintenance mode. If there are maintenaned resources on +the node, the user will be proposed to remove the maintenance +property from them. + +The node parameter defaults to the node where the command is run. + +Usage: +............... +maintenance [<node>] +............... + +[[cmdhelp_node_online,set node online]] +==== `online` + +Set a node to online status. + +The node parameter defaults to the node where the command is run. + +Usage: +............... +online [<node>] +............... + +[[cmdhelp_node_ready,put node into ready mode]] +==== `ready` + +Set the node's maintenance status to `off`. The node should be +now again fully operational and capable of running resource +operations. + +The node parameter defaults to the node where the command is run. + +Usage: +............... +ready [<node>] +............... + +[[cmdhelp_node_server,show node hostname or server address]] +==== `server` + +Remote nodes may have a configured server address which should +be used when contacting the node. This command prints the +server address if configured, else the node name. + +If no parameter is given, the addresses or names for all nodes +are printed. + +Usage: +............... +server [<node> ...] +............... + +[[cmdhelp_node_show,show node]] +==== `show` + +Show a node definition. If the node parameter is omitted then all +nodes are shown. + +Usage: +............... +show [<node>] +............... + +[[cmdhelp_node_standby,put node into standby]] +==== `standby` + +Set a node to standby status. The node parameter defaults to the +node where the command is run. + +Additionally, you may specify a lifetime for the standby---if set to +`reboot`, the node will be back online once it reboots. `forever` will +keep the node in standby after reboot. The life time defaults to +`forever`. + +Usage: +............... +standby [<node>] [<lifetime>] + +lifetime :: reboot | forever +............... + +Example: +............... +standby bob reboot +............... + +[[cmdhelp_node_status-attr,manage status attributes]] +==== `status-attr` + +Edit node attributes which are in the CIB status section, i.e. +attributes which hold properties of a more volatile nature. One +typical example is attribute generated by the `pingd` utility. + +Usage: +............... +status-attr <node> set <attr> <value> +status-attr <node> delete <attr> +status-attr <node> show <attr> +............... +Example: +............... +status-attr node_1 show pingd +............... + +[[cmdhelp_node_utilization,manage utilization attributes]] +==== `utilization` + +Edit node utilization attributes. These attributes describe +hardware characteristics as integer numbers such as memory size +or the number of CPUs. By setting the `placement-strategy` +cluster property appropriately, it is possible then to distribute +resources based on resource requirements and node size. See also +<<cmdhelp_resource_utilization,resource utilization attributes>>. + +Usage: +............... +utilization <node> set <attr> <value> +utilization <node> delete <attr> +utilization <node> show <attr> +............... +Examples: +............... +utilization node_1 set memory 16384 +utilization node_1 show cpu +............... + +[[cmdhelp_site,GEO clustering site support]] +=== `site` - GEO clustering site support + +A cluster may consist of two or more subclusters in different and +distant locations. This set of commands supports such setups. + +[[cmdhelp_site_ticket,manage site tickets]] +==== `ticket` + +Tickets are cluster-wide attributes. They can be managed at the +site where this command is executed. + +It is then possible to constrain resources depending on the +ticket availability (see the <<cmdhelp_configure_rsc_ticket,`rsc_ticket`>> command +for more details). + +Usage: +............... +ticket {grant|revoke|standby|activate|show|time|delete} <ticket> +............... +Example: +............... +ticket grant ticket1 +............... + +[[cmdhelp_options,User preferences]] +=== `options` - User preferences + +The user may set various options for the crm shell itself. + +[[cmdhelp_options_add-quotes,add quotes around parameters containing spaces]] +==== `add-quotes` + +The shell (as in `/bin/sh`) parser strips quotes from the command +line. This may sometimes make it really difficult to type values +which contain white space. One typical example is the configure +filter command. The crm shell will supply extra quotes around +arguments which contain white space. The default is `yes`. + +.Note on quotes use +**************************** +Adding quotes around arguments automatically has been introduced +with version 1.2.2 and it is technically a regression. Being a +regression is the only reason the `add-quotes` option exists. If +you have custom shell scripts which would break, just set the +`add-quotes` option to `no`. + +For instance, with adding quotes enabled, it is possible to do +the following: +............... +# crm configure primitive d1 Dummy \ + meta description="some description here" +# crm configure filter 'sed "s/hostlist=./&node-c /"' fencing +............... +**************************** + +[[cmdhelp_options_check-frequency,when to perform semantic check]] +==== `check-frequency` + +Semantic check of the CIB or elements modified or created may be +done on every configuration change (`always`), when verifying +(`on-verify`) or `never`. It is by default set to `always`. +Experts may want to change the setting to `on-verify`. + +The checks require that resource agents are present. If they are +not installed at the configuration time set this preference to +`never`. + +See <<topics_Features_Checks,Configuration semantic checks>> for more details. + +[[cmdhelp_options_check-mode,how to treat semantic errors]] +==== `check-mode` + +Semantic check of the CIB or elements modified or created may be +done in the `strict` mode or in the `relaxed` mode. In the former +certain problems are treated as configuration errors. In the +`relaxed` mode all are treated as warnings. The default is `strict`. + +See <<topics_Features_Checks,Configuration semantic checks>> for more details. + +[[cmdhelp_options_colorscheme,set colors for output]] +==== `colorscheme` + +With `output` set to `color`, a comma separated list of colors +from this option are used to emphasize: + +- keywords +- object ids +- attribute names +- attribute values +- scores +- resource references + +`crm` can show colors only if there is curses support for python +installed (usually provided by the `python-curses` package). The +colors are whatever is available in your terminal. Use `normal` +if you want to keep the default foreground color. + +This user preference defaults to +`yellow,normal,cyan,red,green,magenta` which is good for +terminals with dark background. You may want to change the color +scheme and save it in the preferences file for other color +setups. + +Example: +............... +colorscheme yellow,normal,blue,red,green,magenta +............... + +[[cmdhelp_options_editor,set preferred editor program]] +==== `editor` + +The `edit` command invokes an editor. Use this to specify your +preferred editor program. If not set, it will default to either +the value of the `EDITOR` environment variable or to one of the +standard UNIX editors (`vi`,`emacs`,`nano`). + +Usage: +............... +editor program +............... +Example: +............... +editor vim +............... + +[[cmdhelp_options_manage-children,how to handle children resource attributes]] +==== `manage-children` + +Some resource management commands, such as `resource stop`, when +the target resource is a group, may not always produce desired +result. Each element, group and the primitive members, can have a +meta attribute and those attributes may end up with conflicting +values. Consider the following construct: +............... +crm(live)# configure show svc fs virtual-ip +primitive fs Filesystem \ + params device="/dev/drbd0" directory="/srv/nfs" fstype=ext3 \ + op monitor interval=10s \ + meta target-role=Started +primitive virtual-ip IPaddr2 \ + params ip=10.2.13.110 iflabel=1 \ + op monitor interval=10s \ + op start interval=0 \ + meta target-role=Started +group svc fs virtual-ip \ + meta target-role=Stopped +............... + +Even though the element +svc+ should be stopped, the group is +actually running because all its members have the +target-role+ +set to +Started+: +............... +crm(live)# resource show svc +resource svc is running on: xen-f +............... + +Hence, if the user invokes +resource stop svc+ the intention is +not clear. This preference gives the user an opportunity to +better control what happens if attributes of group members have +values which are in conflict with the same attribute of the group +itself. + +Possible values are +ask+ (the default), +always+, and +never+. +If set to +always+, the crm shell removes all children attributes +which have values different from the parent. If set to +never+, +all children attributes are left intact. Finally, if set to ++ask+, the user will be asked for each member what is to be done. + +[[cmdhelp_options_output,set output type]] +==== `output` + +`crm` can adorn configurations in two ways: in color (similar to +for instance the `ls --color` command) and by showing keywords in +upper case. Possible values are `plain`, `color-always`, `color`, +and 'uppercase'. It is possible to combine `uppercase` with one +of the color values in order to get an upper case xmass tree. Just +set this option to `color,uppercase` or `color-always,uppercase`. +In case you need color codes in pipes, `color-always` forces color +codes even in case the terminal is not a tty (just like `ls +--color=always`). + +[[cmdhelp_options_pager,set preferred pager program]] +==== `pager` + +The `view` command displays text through a pager. Use this to +specify your preferred pager program. If not set, it will default +to either the value of the `PAGER` environment variable or to one +of the standard UNIX system pagers (`less`,`more`,`pg`). + +[[cmdhelp_options_reset,reset user preferences to factory defaults]] +==== `reset` + +This command resets all user options to the defaults. If used as +a single-shot command, the rc file (+$HOME/.config/crm/rc+) is +reset to the defaults too. + +[[cmdhelp_options_save,save the user preferences to the rc file]] +==== `save` + +Save current settings to the rc file (+$HOME/.config/crm/rc+). On +further `crm` runs, the rc file is automatically read and parsed. + +[[cmdhelp_options_set,Set the value of a given option]] +==== `set` + +Sets the value of an option. Takes the fully qualified +name of the option as argument, as displayed by +show all+. + +The modified option value is stored in the user-local +configuration file, usually found in +~/.config/crm/crm.conf+. + +Usage: +........ +set <option> <value> +........ + +Example: +........ +set color.warn "magenta bold" +set editor nano +........ + +[[cmdhelp_options_show,show current user preference]] +==== `show` + +Display all current settings. + +Given an option name as argument, `show` will display only the value +of that argument. + +Given +all+ as argument, `show` displays all available user options. + +Usage: +........ +show [all|<option>] +........ + +Example: +........ +show +show skill-level +show all +........ + +[[cmdhelp_options_skill-level,set skill level]] +==== `skill-level` + +Based on the skill-level setting, the user is allowed to use only +a subset of commands. There are three levels: operator, +administrator, and expert. The operator level allows only +commands at the `resource` and `node` levels, but not editing +or deleting resources. The administrator may do that and may also +configure the cluster at the `configure` level and manage the +shadow CIBs. The expert may do all. + +Usage: +............... +skill-level <level> + +level :: operator | administrator | expert +............... + +.Note on security +**************************** +The `skill-level` option is advisory only. There is nothing +stopping any users change their skill level (see +<<topics_Features_Security,Access Control Lists (ACL)>> on how to enforce +access control). +**************************** + +[[cmdhelp_options_sort-elements,sort CIB elements]] +==== `sort-elements` + +`crm` by default sorts CIB elements. If you want them appear in +the order they were created, set this option to `no`. + +Usage: +............... +sort-elements {yes|no} +............... +Example: +............... +sort-elements no +............... + +[[cmdhelp_options_user,set the cluster user]] +==== `user` + +Sufficient privileges are necessary in order to manage a +cluster: programs such as `crm_verify` or `crm_resource` and, +ultimately, `cibadmin` have to be run either as `root` or as the +CRM owner user (typically `hacluster`). You don't have to worry +about that if you run `crm` as `root`. A more secure way is to +run the program with your usual privileges, set this option to +the appropriate user (such as `hacluster`), and setup the +`sudoers` file. + +Usage: +............... +user system-user +............... +Example: +............... +user hacluster +............... + +[[cmdhelp_options_wait,synchronous operation]] +==== `wait` + +In normal operation, `crm` runs a command and gets back +immediately to process other commands or get input from the user. +With this option set to `yes` it will wait for the started +transition to finish. In interactive mode dots are printed to +indicate progress. + +Usage: +............... +wait {yes|no} +............... +Example: +............... +wait yes +............... + +[[cmdhelp_configure,CIB configuration]] +=== `configure` - CIB configuration + +This level enables all CIB object definition commands. + +The configuration may be logically divided into four parts: +nodes, resources, constraints, and (cluster) properties and +attributes. Each of these commands support one or more basic CIB +objects. + +Nodes and attributes describing nodes are managed using the +`node` command. + +Commands for resources are: + +- `primitive` +- `monitor` +- `group` +- `clone` +- `ms`/`master` (master-slave) + +In order to streamline large configurations, it is possible to +define a template which can later be referenced in primitives: + +- `rsc_template` + +In that case the primitive inherits all attributes defined in the +template. + +There are three types of constraints: + +- `location` +- `colocation` +- `order` + +It is possible to define fencing order (stonith resource +priorities): + +- `fencing_topology` + +Finally, there are the cluster properties, resource meta +attributes defaults, and operations defaults. All are just a set +of attributes. These attributes are managed by the following +commands: + +- `property` +- `rsc_defaults` +- `op_defaults` + +In addition to the cluster configuration, the Access Control +Lists (ACL) can be setup to allow access to parts of the CIB for +users other than +root+ and +hacluster+. The following commands +manage ACL: + +- `user` +- `role` + +In Pacemaker 1.1.12 and up, this command replaces the `user` command +for handling ACLs: + +- `acl_target` + +The changes are applied to the current CIB only on ending the +configuration session or using the `commit` command. + +Comments start with +#+ in the first line. The comments are tied +to the element which follows. If the element moves, its comments +will follow. + +[[cmdhelp_configure_acl_target,Define target access rights]] +==== `acl_target` + +Defines an ACL target. + +Usage: +................ +acl_target <tid> [<role> ...] +................ +Example: +................ +acl_target joe resource_admin constraint_editor +................ + +[[cmdhelp_configure_alert,Event-driven alerts]] +==== `alert` + +.Version note +**************************** +This feature is only available +in Pacemaker 1.1.15+. +**************************** + +Event-driven alerts enables calling scripts whenever interesting +events occur in the cluster (nodes joining or leaving, resources +starting or stopping, etc.). + +The +path+ is an arbitrary file path to an alert script. Existing +external scripts used with ClusterMon resources can be used as alert +scripts, since the interface is compatible. + +Each alert may have a number of receipients configured. These will be +passed to the script as arguments. The first recipient will also be +passed as the +CRM_alert_recipient+ environment variable, for +compatibility with existing scripts that only support one recipient. + +The available meta attributes are +timeout+ (default 30s) and ++timestamp-format+ (default `"%H:%M:%S.%06N"`). + +Some configurations may require each recipient to be delimited by +brackets, to avoid ambiguity. In the example +alert-2+ below, the meta +attribute for `timeout` is defined after the recipient, so the +brackets are used to ensure that the meta attribute is set for the +alert and not just the recipient. This can be avoided by setting any +alert attributes before defining the recipients. + +Usage: +............... +alert <id> <path> \ + [attributes <nvpair> ...] \ + [meta <nvpair> ...] \ + [select [nodes | fencing | resources | attributes '{' <attribute> ... '}' ] ...] \ + [to [{] <recipient> + [attributes <nvpair> ...] \ + [meta <nvpair> ...] [}] \ + ...] +............... + +Example: +............... +alert alert-1 /srv/pacemaker/pcmk_alert_sample.sh \ + to /var/log/cluster-alerts.log + +alert alert-2 /srv/pacemaker/example_alert.sh \ + meta timeout=60s \ + to { /var/log/cluster-alerts.log } + +alert alert-3 /srv/pacemaker/example_alert.sh \ + select fencing \ + to { /var/log/fencing-alerts.log } + +............... + +[[cmdhelp_configure_bundle,Container bundle]] +==== `bundle` + +A bundle is a single resource specifying the settings, networking +requirements, and storage requirements for any number of containers +generated from the same container image. + +Pacemaker bundles support Docker (since version 1.1.17) and rkt (since +version 1.1.18) container technologies. + +A bundle must contain exactly one +docker+ or +rkt+ element. + +The bundle definition may contain a reference to a primitive +resource which defining the resource running inside the +container. + +Example: +............... + +primitive httpd-apache ocf:heartbeat:apache + +bundle httpd \ + docker image=pcmk:httpd replicas=3 \ + network ip-range-start=10.10.10.123 host-netmask=24 \ + port-mapping port=80 \ + storage \ + storage-mapping target-dir=/var/www/html source-dir=/srv/www options=rw \ + primitive httpd-apache + +............... + +[[cmdhelp_configure_cib,CIB shadow management]] +==== `cib` + +This level is for management of shadow CIBs. It is available at +the `configure` level to enable saving intermediate changes to a +shadow CIB instead of to the live cluster. This short excerpt +shows how: +............... +crm(live)configure# cib new test-2 +INFO: test-2 shadow CIB created +crm(test-2)configure# commit +............... +Note how the current CIB in the prompt changed from +live+ to ++test-2+ after issuing the `cib new` command. See also the +<<cmdhelp_cib,CIB shadow management>> for more information. + +[[cmdhelp_configure_cibstatus,CIB status management and editing]] +==== `cibstatus` + +Enter edit and manage the CIB status section level. See the +<<cmdhelp_cibstatus,CIB status management section>>. + +[[cmdhelp_configure_clone,define a clone]] +==== `clone` + +The `clone` command creates a resource clone. It may contain a +single primitive resource or one group of resources. + +Usage: +............... +clone <name> <rsc> + [description=<description>] + [meta <attr_list>] + [params <attr_list>] + +attr_list :: [$id=<id>] <attr>=<val> [<attr>=<val>...] | $id-ref=<id> +............... +Example: +............... +clone cl_fence apc_1 \ + meta clone-node-max=1 globally-unique=false + +clone disk1 drbd1 \ + meta promotable=true notify=true globally-unique=false +............... + +[[cmdhelp_configure_colocation,colocate resources]] +==== `colocation` (`collocation`) + +This constraint expresses the placement relation between two +or more resources. If there are more than two resources, then the +constraint is called a resource set. + +The score is used to indicate the priority of the constraint. A +positive score indicates that the resources should run on the same +node. A negative score that they should not run on the same +node. Values of positive or negative +infinity+ indicate a mandatory +constraint. + +In the two resource form, the cluster will place +<with-rsc>+ first, +and then decide where to put the +<rsc>+ resource. + +Collocation resource sets have an extra attribute (+sequential+) +to allow for sets of resources which don't depend on each other +in terms of state. The shell syntax for such sets is to put +resources in parentheses. + +Sets cannot be nested. + +The optional +node-attribute+ can be used to colocate resources on a +set of nodes and not necessarily on the same node. For example, by +setting a node attribute +color+ on all nodes and setting the ++node-attribute+ value to +color+ as well, the colocated resources +will be placed on any node that has the same color. + +For more details on how to configure resource sets, see +<<topics_Features_Resourcesets,`Syntax: Resource sets`>>. + +Usage: +............... +colocation <id> <score>: <rsc>[:<role>] <with-rsc>[:<role>] + [node-attribute=<node_attr>] + +colocation <id> <score>: <resource_sets> + [node-attribute=<node_attr>] + +resource_sets :: <resource_set> [<resource_set> ...] + +resource_set :: ["("|"["] <rsc>[:<role>] [<rsc>[:<role>] ...] \ + [<attributes>] [")"|"]"] + +attributes :: [require-all=(true|false)] [sequential=(true|false)] + +............... +Example: +............... +colocation never_put_apache_with_dummy -inf: apache dummy +colocation c1 inf: A ( B C ) +............... + +[[cmdhelp_configure_commit,commit the changes to the CIB]] +==== `commit` + +Commit the current configuration to the CIB in use. As noted +elsewhere, commands in a configure session don't have immediate +effect on the CIB. All changes are applied at one point in time, +either using `commit` or when the user leaves the configure +level. In case the CIB in use changed in the meantime, presumably +by somebody else, the crm shell will refuse to apply the changes. + +If you know that it's fine to still apply them, add +force+ to the +command line. + +To disable CIB patching and apply the changes by replacing the CIB +completely, add +replace+ to the command line. Note that this can lead +to previous changes being overwritten if some other process +concurrently modifies the CIB. + +Usage: +............... +commit [force] [replace] +............... + +[[cmdhelp_configure_default-timeouts,set timeouts for operations to minimums from the meta-data]] +==== `default-timeouts` + +This command takes the timeouts from the actions section of the +resource agent meta-data and sets them for the operations of the +primitive. + +Usage: +............... +default-timeouts <id> [<id>...] +............... + +.Note on `default-timeouts` +**************************** +The use of this command is discouraged in favor of manually +determining the best timeouts required for the particular +configuration. Relying on the resource agent to supply appropriate +timeouts can cause the resource to fail at the worst possible moment. + +Appropriate timeouts for resource actions are context-sensitive, and +should be carefully considered with the whole configuration in mind. +**************************** + +[[cmdhelp_configure_delete,delete CIB objects]] +==== `delete` + +Delete one or more objects. If an object to be deleted belongs to +a container object, such as a group, and it is the only resource +in that container, then the container is deleted as well. Any +related constraints are removed as well. + +If the object is a started resource, it will not be deleted unless the ++--force+ flag is passed to the command, or the +force+ option is set. + +Usage: +............... +delete [--force] <id> [<id>...] +............... + +[[cmdhelp_configure_edit,edit CIB objects]] +==== `edit` + +This command invokes the editor with the object description. As +with the `show` command, the user may choose to edit all objects +or a set of objects. + +If the user insists, he or she may edit the XML edition of the +object. If you do that, don't modify any id attributes. + +Usage: +............... +edit [xml] [<id> ...] +edit [xml] changed +............... + +.Note on renaming element ids +**************************** +The edit command sometimes cannot properly handle modifying +element ids. In particular for elements which belong to group or +ms resources. Group and ms resources themselves also cannot be +renamed. Please use the `rename` command instead. +**************************** + +[[cmdhelp_configure_erase,erase the CIB]] +==== `erase` + +The `erase` clears all configuration. Apart from nodes. To remove +nodes, you have to specify an additional keyword `nodes`. + +Note that removing nodes from the live cluster may have some +strange/interesting/unwelcome effects. + +Usage: +............... +erase [nodes] +............... + +[[cmdhelp_configure_fencing_topology,node fencing order]] +==== `fencing_topology` + +If multiple fencing (stonith) devices are available capable of +fencing a node, their order may be specified by +fencing_topology+. +The order is specified per node. + +Stonith resources can be separated by +,+ in which case all of +them need to succeed. If they fail, the next stonith resource (or +set of resources) is used. In other words, use comma to separate +resources which all need to succeed and whitespace for serial +order. It is not allowed to use whitespace around comma. + +If the node is left out, the order is used for all nodes. +That should reduce the configuration size in some stonith setups. + +From Pacemaker version 1.1.14, it is possible to use a node attribute +as the +target+ in a fencing topology. The syntax for this usage is +described below. + +From Pacemaker version 1.1.14, it is also possible to use regular +expression patterns as the +target+ in a fencing topology. The configured +fencing sequence then applies to all devices matching the pattern. + +Usage: +............... +fencing_topology <stonith_resources> [<stonith_resources> ...] +fencing_topology <fencing_order> [<fencing_order> ...] + +fencing_order :: <target> <stonith_resources> [<stonith_resources> ...] + +stonith_resources :: <rsc>[,<rsc>...] +target :: <node>: | attr:<node-attribute>=<value> | pattern:<pattern> +............... +Example: +............... +# Only kill the power if poison-pill fails +fencing_topology poison-pill power + +# As above for node-a, but a different strategy for node-b +fencing_topology \ + node-a: poison-pill power \ + node-b: ipmi serial + +# Fencing anything on rack 1 requires fencing via both APC 1 and 2, +# to defeat the redundancy provided by two separate UPS units. +fencing_topology attr:rack=1 apc01,apc02 + +# Fencing for all machines named green.* is done using the pear +# fencing device first, while all machines named red.* are fenced +# using the apple fencing device first. +fencing_topology \ + pattern:green.* pear apple \ + pattern:red.* apple pear +............... + +[[cmdhelp_configure_filter,filter CIB objects]] +==== `filter` + +This command filters the given CIB elements through an external +program. The program should accept input on `stdin` and send +output to `stdout` (the standard UNIX filter conventions). As +with the `show` command, the user may choose to filter all or +just a subset of elements. + +It is possible to filter the XML representation of objects, but +probably not as useful as the configuration language. The +presentation is somewhat different from what would be displayed +by the `show` command---each element is shown on a single line, +i.e. there are no backslashes and no other embelishments. + +Don't forget to put quotes around the filter if it contains +spaces. + +Usage: +............... +filter <prog> [xml] [<id> ...] +filter <prog> [xml] changed +............... +Examples: +............... +filter "sed '/^primitive/s/target-role=[^ ]*//'" +# crm configure filter "sed '/^primitive/s/target-role=[^ ]*//'" +crm configure <<END + filter "sed '/threshold=\"1\"/s/=\"1\"/=\"0\"/g'" +END +............... + +.Note on quotation marks +************************** +Filter commands which feature a blend of quotation marks can be +difficult to get right, especially when used directly from bash, since +bash does its own quotation parsing. In these cases, it can be easier +to supply the filter command as standard input. See the last example +above. +************************** + +[[cmdhelp_configure_get_property,Get property value]] +==== `get-property` + +Show the value of the given property. If the value is not set, the +command will print the default value for the property, if known. + +If no property name is passed to the command, the list of known +cluster properties is printed. + +If the property is set multiple times, for example using multiple +property sets with different rule expressions, the output of this +command is undefined. + +Pass the argument +-t+ or +--true+ to `get-property` to translate +the argument value into +true+ or +false+. If the value is not +set, the command will print +false+. + +Usage: +............... +get-property [-t|--true] [<name>] +............... + +Example: +............... +get-property stonith-enabled +get-property -t maintenance-mode +............... + +[[cmdhelp_configure_graph,generate a directed graph]] +==== `graph` + +Create a graphviz graphical layout from the current cluster +configuration. + +Currently, only `dot` (directed graph) is supported. It is +essentially a visualization of resource ordering. + +The graph may be saved to a file which can be used as source for +various graphviz tools (by default it is displayed in the user's +X11 session). Optionally, by specifying the format, one can also +produce an image instead. + +For more or different graphviz attributes, it is possible to save +the default set of attributes to an ini file. If this file exists +it will always override the builtin settings. The +exportsettings+ +subcommand also prints the location of the ini file. + +Usage: +............... +graph [<gtype> [<file> [<img_format>]]] +graph exportsettings + +gtype :: dot +img_format :: `dot` output format (see the +-T+ option) +............... +Example: +............... +graph dot +graph dot clu1.conf.dot +graph dot clu1.conf.svg svg +............... + +[[cmdhelp_configure_group,define a group]] +==== `group` + +The `group` command creates a group of resources. This can be useful +when resources depend on other resources and require that those +resources start in order on the same node. A common use of resource +groups is to ensure that a server and a virtual IP are located +together, and that the virtual IP is started before the server. + +Grouped resources are started in the order they appear in the group, +and stopped in the reverse order. If a resource in the group cannot +run anywhere, resources following it in the group will not start. + +`group` can be passed the "container" meta attribute, to indicate that +it is to be used to group VM resources monitored using Nagios. The +resource referred to by the container attribute must be of type +`ocf:heartbeat:Xen`, `ocf:heartbeat:VirtualDomain` or `ocf:heartbeat:lxc`. + +Usage: +............... +group <name> <rsc> [<rsc>...] + [description=<description>] + [meta attr_list] + [params attr_list] + +attr_list :: [$id=<id>] <attr>=<val> [<attr>=<val>...] | $id-ref=<id> +............... +Example: +............... +group internal_www disk0 fs0 internal_ip apache \ + meta target_role=stopped + +group vm-and-services vm vm-sshd meta container="vm" +............... + +[[cmdhelp_configure_load,import the CIB from a file]] +==== `load` + +Load a part of configuration (or all of it) from a local file or +a network URL. The +replace+ method replaces the current +configuration with the one from the source. The +update+ method +tries to import the contents into the current configuration. The ++push+ method imports the contents into the current configuration +and removes any lines that are not present in the given +configuration. +The file may be a CLI file or an XML file. + +If the URL is `-`, the configuration is read from standard input. + +Usage: +............... +load [xml] <method> URL + +method :: replace | update | push +............... +Example: +............... +load xml update myfirstcib.xml +load xml replace http://storage.big.com/cibs/bigcib.xml +load xml push smallcib.xml +............... + +[[cmdhelp_configure_location,a location preference]] +==== `location` + +`location` defines the preference of nodes for the given +resource. The location constraints consist of one or more rules +which specify a score to be awarded if the rule matches. + +The resource referenced by the location constraint can be one of the +following: + +* Plain resource reference: +location loc1 webserver 100: node1+ +* Resource set in curly brackets: +location loc1 { virtual-ip webserver } 100: node1+ +* Tag containing resource ids: +location loc1 tag1 100: node1+ +* Resource pattern: +location loc1 /web.*/ 100: node1+ + +The +resource-discovery+ attribute allows probes to be selectively +enabled or disabled per resource and node. + +The syntax for resource sets is described in detail for +<<cmdhelp_configure_colocation,`colocation`>>. + +For more details on how to configure resource sets, see +<<topics_Features_Resourcesets,`Syntax: Resource sets`>>. + +For more information on rule expressions, see +<<topics_Syntax_RuleExpressions,Syntax: Rule expressions>>. + +Usage: +............... +location <id> <rsc> [<attributes>] {<node_pref>|<rules>} + +rsc :: /<rsc-pattern>/ + | { resource_sets } + | <rsc> + +attributes :: role=<role> | resource-discovery=always|never|exclusive + +node_pref :: <score>: <node> + +rules :: + rule [id_spec] [$role=<role>] <score>: <expression> + [rule [id_spec] [$role=<role>] <score>: <expression> ...] + +id_spec :: $id=<id> | $id-ref=<id> +score :: <number> | <attribute> | [-]inf +expression :: <simple_exp> [<bool_op> <simple_exp> ...] +bool_op :: or | and +simple_exp :: <attribute> [type:]<binary_op> <value> + | <unary_op> <attribute> + | date <date_expr> +type :: string | version | number +binary_op :: lt | gt | lte | gte | eq | ne +unary_op :: defined | not_defined + +date_expr :: lt <end> + | gt <start> + | in start=<start> end=<end> + | in start=<start> <duration> + | spec <date_spec> +duration|date_spec :: + hours=<value> + | monthdays=<value> + | weekdays=<value> + | yearsdays=<value> + | months=<value> + | weeks=<value> + | years=<value> + | weekyears=<value> + | moon=<value> +............... +Examples: +............... +location conn_1 internal_www 100: node1 + +location conn_1 internal_www \ + rule 50: #uname eq node1 \ + rule pingd: defined pingd + +location conn_2 dummy_float \ + rule -inf: not_defined pingd or pingd number:lte 0 + +# never probe for rsc1 on node1 +location no-probe rsc1 resource-discovery=never -inf: node1 +............... + +[[cmdhelp_configure_modgroup,modify group]] +==== `modgroup` + +Add or remove primitives in a group. The `add` subcommand appends +the new group member by default. Should it go elsewhere, there +are `after` and `before` clauses. + +Usage: +............... +modgroup <id> add <id> [after <id>|before <id>] +modgroup <id> remove <id> +............... +Examples: +............... +modgroup share1 add storage2 before share1-fs +............... + +[[cmdhelp_configure_monitor,add monitor operation to a primitive]] +==== `monitor` + +Monitor is by far the most common operation. It is possible to +add it without editing the whole resource. Also, long primitive +definitions may be a bit uncluttered. In order to make this +command as concise as possible, less common operation attributes +are not available. If you need them, then use the `op` part of +the `primitive` command. + +Usage: +............... +monitor <rsc>[:<role>] <interval>[:<timeout>] +............... +Example: +............... +monitor apcfence 60m:60s +............... + +Note that after executing the command, the monitor operation may +be shown as part of the primitive definition. + +[[cmdhelp_configure_ms,define a master-slave resource]] +==== `ms` (`master`) + +The `ms` command creates a master/slave resource type. It may contain a +single primitive resource or one group of resources. + +Usage: +............... +ms <name> <rsc> + [description=<description>] + [meta attr_list] + [params attr_list] + +attr_list :: [$id=<id>] <attr>=<val> [<attr>=<val>...] | $id-ref=<id> +............... +Example: +............... +ms disk1 drbd1 \ + meta notify=true globally-unique=false +............... + +.Note on `ms` deprecated +**************************** +From Pacemaker-2.0, the resource type referred to as "master/slave", +"stateful", or "multi-state" is no longer a separate resource type, +but a variation of clone now referred to as a "promotable clone". +For backward compatibility, above configurations are also accepted. +............... +clone disk1 drbd1 \ + meta promotable=true notify=true globally-unique=false +............... +**************************** + +.Note on `id-ref` usage +**************************** +Instance or meta attributes (`params` and `meta`) may contain +a reference to another set of attributes. In that case, no other +attributes are allowed. Since attribute sets' ids, though they do +exist, are not shown in the `crm`, it is also possible to +reference an object instead of an attribute set. `crm` will +automatically replace such a reference with the right id: + +............... +crm(live)configure# primitive a2 www-2 meta $id-ref=a1 +crm(live)configure# show a2 +primitive a2 apache \ + meta $id-ref=a1-meta_attributes + [...] +............... +It is advisable to give meaningful names to attribute sets which +are going to be referenced. +**************************** + +[[cmdhelp_configure_node,define a cluster node]] +==== `node` + +The node command describes a cluster node. Nodes in the CIB are +commonly created automatically by the CRM. Hence, you should not +need to deal with nodes unless you also want to define node +attributes. Note that it is also possible to manage node +attributes at the `node` level. + +Usage: +............... +node [$id=<id>] <uname>[:<type>] + [description=<description>] + [attributes [$id=<id>] [<score>:] [rule...] + <param>=<value> [<param>=<value>...]] | $id-ref=<ref> + [utilization [$id=<id>] [<score>:] [rule...] + <param>=<value> [<param>=<value>...]] | $id-ref=<ref> + +type :: normal | member | ping | remote +............... +Example: +............... +node node1 +node big_node attributes memory=64 +............... + +[[cmdhelp_configure_op_defaults,set resource operations defaults]] +==== `op_defaults` + +Set defaults for the operations meta attributes. + +For more information on rule expressions, see +<<topics_Syntax_RuleExpressions,Syntax: Rule expressions>>. + +Usage: +............... +op_defaults [$id=<set_id>] [rule ...] <option>=<value> [<option>=<value> ...] +............... +Example: +............... +op_defaults record-pending=true +............... + +[[cmdhelp_configure_order,order resources]] +==== `order` + +This constraint expresses the order of actions on two resources +or more resources. If there are more than two resources, then the +constraint is called a resource set. + +Ordered resource sets have an extra attribute to allow for sets +of resources whose actions may run in parallel. The shell syntax +for such sets is to put resources in parentheses. + +If the subsequent resource can start or promote after any one of the +resources in a set has done, enclose the set in brackets (+[+ and +]+). + +Sets cannot be nested. + +Three strings are reserved to specify a kind of order constraint: ++Mandatory+, +Optional+, and +Serialize+. It is preferred to use +one of these settings instead of score. Previous versions mapped +scores +0+ and +inf+ to keywords +advisory+ and +mandatory+. +That is still valid but deprecated. + +For more details on how to configure resource sets, see +<<topics_Features_Resourcesets,`Syntax: Resource sets`>>. + +Usage: +............... +order <id> [kind:] first then [symmetrical=<bool>] + +order <id> [kind:] resource_sets [symmetrical=<bool>] + +kind :: Mandatory | Optional | Serialize + +first :: <rsc>[:<action>] + +then :: <rsc>[:<action>] + +resource_sets :: resource_set [resource_set ...] + +resource_set :: ["["|"("] <rsc>[:<action>] [<rsc>[:<action>] ...] \ + [attributes] ["]"|")"] + +attributes :: [require-all=(true|false)] [sequential=(true|false)] + +............... +Example: +............... +order o-1 Mandatory: apache:start ip_1 +order o-2 Serialize: A ( B C ) +order o-4 first-resource then-resource +............... + +[[cmdhelp_configure_primitive,define a resource]] +==== `primitive` + +The primitive command describes a resource. It may be referenced +only once in group, clone, or master-slave objects. If it's not +referenced, then it is placed as a single resource in the CIB. + +Operations may be specified anonymously, as a group or by reference: + +* "Anonymous", as a list of +op+ specifications. Use this + method if you don't need to reference the set of operations + elsewhere. This is the most common way to define operations. + +* If reusing operation sets is desired, use the +operations+ keyword + along with an id to give the operations set a name. Use the + +operations+ keyword and an id-ref value set to the id of another + operations set, to apply the same set of operations to this + primitive. + +Operation attributes which are not recognized are saved as +instance attributes of that operation. A typical example is ++OCF_CHECK_LEVEL+. + +For multistate resources, roles are specified as +role=<role>+. + +A template may be defined for resources which are of the same +type and which share most of the configuration. See +<<cmdhelp_configure_rsc_template,`rsc_template`>> for more information. + +Attributes containing time values, such as the +interval+ attribute on +operations, are configured either as a plain number, which is +interpreted as a time in seconds, or using one of the following +suffixes: + +* +s+, +sec+ - time in seconds (same as no suffix) +* +ms+, +msec+ - time in milliseconds +* +us+, +usec+ - time in microseconds +* +m+, +min+ - time in minutes +* +h+, +hr+ - time in hours + +Usage: +............... +primitive <rsc> {[<class>:[<provider>:]]<type>|@<template>} + [description=<description>] + [[params] attr_list] + [meta attr_list] + [utilization attr_list] + [operations id_spec] + [op op_type [<attribute>=<value>...] + [[op_params] attr_list] + [op_meta attr_list] ...] + +attr_list :: [$id=<id>] [<score>:] [rule...] + <attr>=<val> [<attr>=<val>...]] | $id-ref=<id> +id_spec :: $id=<id> | $id-ref=<id> +op_type :: start | stop | monitor +............... +Example: +............... +primitive apcfence stonith:apcsmart \ + params ttydev=/dev/ttyS0 hostlist="node1 node2" \ + op start timeout=60s \ + op monitor interval=30m timeout=60s + +primitive www8 apache \ + configfile=/etc/apache/www8.conf \ + operations $id-ref=apache_ops + +primitive db0 mysql \ + params config=/etc/mysql/db0.conf \ + op monitor interval=60s \ + op monitor interval=300s OCF_CHECK_LEVEL=10 + +primitive r0 ocf:linbit:drbd \ + params drbd_resource=r0 \ + op monitor role=Master interval=60s \ + op monitor role=Slave interval=300s + +primitive xen0 @vm_scheme1 xmfile=/etc/xen/vm/xen0 + +primitive mySpecialRsc Special \ + params 3: rule #uname eq node1 interface=eth1 \ + params 2: rule #uname eq node2 interface=eth2 port=8888 \ + params 1: interface=eth0 port=9999 + +primitive A ocf:pacemaker:Dummy \ + op start \ + op_meta 2: rule #ra-version version:gt 1.0 timeout=120s \ + op_meta 1: timeout=60s +............... + +[[cmdhelp_configure_property,set a cluster property]] +==== `property` + +Set cluster configuration properties. To list the +available cluster configuration properties, use the +<<cmdhelp_ra_info,`ra info`>> command with +pengine+, +crmd+, ++cib+ and +stonithd+ as arguments. +When setting the +maintenance-mode+ property, it will +inform the user if there are nodes or resources that +have the +maintenance+ property. + +For more information on rule expressions, see +<<topics_Syntax_RuleExpressions,Syntax: Rule expressions>>. + +Usage: +............... +property [<set_id>:] [rule ...] <option>=<value> [<option>=<value> ...] +............... +Example: +............... +property stonith-enabled=true +property rule date spec years=2014 stonith-enabled=false +............... + +[[cmdhelp_configure_ptest,show cluster actions if changes were committed]] +==== `ptest` (`simulate`) + +Show PE (Policy Engine) motions using `ptest(8)` or +`crm_simulate(8)`. + +A CIB is constructed using the current user edited configuration +and the status from the running CIB. The resulting CIB is run +through `ptest` (or `crm_simulate`) to show changes which would +happen if the configuration is committed. + +The status section may be loaded from another source and modified +using the <<cmdhelp_cibstatus,`cibstatus`>> level commands. In that case, the +`ptest` command will issue a message informing the user that the +Policy Engine graph is not calculated based on the current status +section and therefore won't show what would happen to the +running but some imaginary cluster. + +If you have graphviz installed and X11 session, `dotty(1)` is run +to display the changes graphically. + +Add a string of +v+ characters to increase verbosity. `ptest` +can also show allocation scores. +utilization+ turns on +information about the remaining capacity of nodes. With the ++actions+ option, `ptest` will print all resource actions. + +The `ptest` program has been replaced by `crm_simulate` in newer +Pacemaker versions. In some installations both could be +installed. Use `simulate` to enfore using `crm_simulate`. + +Usage: +............... +ptest [nograph] [v...] [scores] [actions] [utilization] +............... +Examples: +............... +ptest scores +ptest vvvvv +simulate actions +............... + +[[cmdhelp_configure_refresh,refresh from CIB]] +==== `refresh` + +Refresh the internal structures from the CIB. All changes made +during this session are lost. + +Usage: +............... +refresh +............... + +[[cmdhelp_configure_rename,rename a CIB object]] +==== `rename` + +Rename an object. It is recommended to use this command to rename +a resource, because it will take care of updating all related +constraints and a parent resource. Changing ids with the edit +command won't have the same effect. + +If you want to rename a resource, it must be in the stopped state. + +Usage: +............... +rename <old_id> <new_id> +............... + +[[cmdhelp_configure_role,define role access rights]] +==== `role` + +An ACL role is a set of rules which describe access rights to +CIB. Rules consist of an access right +read+, +write+, or +deny+ +and a specification denoting part of the configuration to which +the access right applies. The specification can be an XPath or a +combination of tag and id references. If an attribute is +appended, then the specification applies only to that attribute +of the matching element. + +There is a number of shortcuts for XPath specifications. The ++meta+, +params+, and +utilization+ shortcuts reference resource +meta attributes, parameters, and utilization respectively. The +`location` may be used to specify location constraints most of +the time to allow resource `move` and `unmove` commands. The +`property` references cluster properties. The `node` allows +reading node attributes. +nodeattr+ and +nodeutil+ reference node +attributes and node capacity (utilization). The `status` shortcut +references the whole status section of the CIB. Read access to +status is necessary for various monitoring tools such as +`crm_mon(8)` (aka `crm status`). + +For more information on rule expressions, see +<<topics_Syntax_RuleExpressions,Syntax: Rule expressions>>. + +Usage: +............... +role <role-id> rule [rule ...] + +rule :: acl-right cib-spec [attribute:<attribute>] + +acl-right :: read | write | deny + +cib-spec :: xpath-spec | tag-ref-spec +xpath-spec :: xpath:<xpath> | shortcut +tag-ref-spec :: tag:<tag> | ref:<id> | tag:<tag> ref:<id> + +shortcut :: meta:<rsc>[:<attr>] + params:<rsc>[:<attr>] + utilization:<rsc> + location:<rsc> + property[:<attr>] + node[:<node>] + nodeattr[:<attr>] + nodeutil[:<node>] + status +............... +Example: +............... +role app1_admin \ + write meta:app1:target-role \ + write meta:app1:is-managed \ + write location:app1 \ + read ref:app1 +............... + +[[cmdhelp_configure_rsc_defaults,set resource defaults]] +==== `rsc_defaults` + +Set defaults for the resource meta attributes. + +For more information on rule expressions, see +<<topics_Syntax_RuleExpressions,Syntax: Rule expressions>>. + +Usage: +............... +rsc_defaults [<set_id>:] [rule ...] <option>=<value> [<option>=<value> ...] +............... +Example: +............... +rsc_defaults failure-timeout=3m +............... + +[[cmdhelp_configure_rsc_template,define a resource template]] +==== `rsc_template` + +The `rsc_template` command creates a resource template. It may be +referenced in primitives. It is used to reduce large +configurations with many similar resources. + +Usage: +............... +rsc_template <name> [<class>:[<provider>:]]<type> + [description=<description>] + [params attr_list] + [meta attr_list] + [utilization attr_list] + [operations id_spec] + [op op_type [<attribute>=<value>...] ...] + +attr_list :: [$id=<id>] <attr>=<val> [<attr>=<val>...] | $id-ref=<id> +id_spec :: $id=<id> | $id-ref=<id> +op_type :: start | stop | monitor +............... +Example: +............... +rsc_template public_vm Xen \ + op start timeout=300s \ + op stop timeout=300s \ + op monitor interval=30s timeout=60s \ + op migrate_from timeout=600s \ + op migrate_to timeout=600s +primitive xen0 @public_vm \ + params xmfile=/etc/xen/xen0 +primitive xen1 @public_vm \ + params xmfile=/etc/xen/xen1 +............... + +[[cmdhelp_configure_rsc_ticket,resources ticket dependency]] +==== `rsc_ticket` + +This constraint expresses dependency of resources on cluster-wide +attributes, also known as tickets. Tickets are mainly used in +geo-clusters, which consist of multiple sites. A ticket may be +granted to a site, thus allowing resources to run there. + +The +loss-policy+ attribute specifies what happens to the +resource (or resources) if the ticket is revoked. The default is +either +stop+ or +demote+ depending on whether a resource is +multi-state. + +See also the <<cmdhelp_site_ticket,`site`>> set of commands. + +Usage: +............... +rsc_ticket <id> <ticket_id>: <rsc>[:<role>] [<rsc>[:<role>] ...] + [loss-policy=<loss_policy_action>] + +loss_policy_action :: stop | demote | fence | freeze +............... +Example: +............... +rsc_ticket ticket-A_public-ip ticket-A: public-ip +rsc_ticket ticket-A_bigdb ticket-A: bigdb loss-policy=fence +rsc_ticket ticket-B_storage ticket-B: drbd-a:Master drbd-b:Master +............... + + +[[cmdhelp_configure_rsctest,test resources as currently configured]] +==== `rsctest` + +Test resources with current resource configuration. If no nodes +are specified, tests are run on all known nodes. + +The order of resources is significant: it is assumed that later +resources depend on earlier ones. + +If a resource is multi-state, it is assumed that the role on +which later resources depend is master. + +Tests are run sequentially to prevent running the same resource +on two or more nodes. Tests are carried out only if none of the +specified nodes currently run any of the specified resources. +However, it won't verify whether resources run on the other +nodes. + +Superuser privileges are obviously required: either run this as +root or setup the `sudoers` file appropriately. + +Note that resource testing may take some time. + +Usage: +............... +rsctest <rsc_id> [<rsc_id> ...] [<node_id> ...] +............... +Examples: +............... +rsctest my_ip websvc +rsctest websvc nodeB +............... + +[[cmdhelp_configure_save,save the CIB to a file]] +==== `save` + +Save the current configuration to a file. Optionally, as XML. Use ++-+ instead of file name to write the output to `stdout`. + +The `save` command accepts the same selection arguments as the `show` +command. See the <<cmdhelp_configure_show,help section>> for `show` +for more details. + +Usage: +............... +save [xml] [<id> | type:<type | tag:<tag> | + related:<obj> | changed ...] <file> +............... +Example: +............... +save myfirstcib.txt +save web-server server-config.txt +............... + +[[cmdhelp_configure_schema,set or display current CIB RNG schema]] +==== `schema` + +CIB's content is validated by a RNG schema. Pacemaker supports +several, depending on version. At least the following schemas are +accepted by `crmsh`: + +* +pacemaker-1.0+ +* +pacemaker-1.1+ +* +pacemaker-1.2+ +* +pacemaker-1.3+ +* +pacemaker-2.0+ + +Use this command to display or switch to another RNG schema. + +Usage: +............... +schema [<schema>] +............... +Example: +............... +schema pacemaker-1.1 +............... + +[[cmdhelp_configure_set,set an attribute value]] +==== `set` + +Set the value of a configured attribute. The attribute must +configured previously, and can be an agent parameter, meta attribute, +utilization value or operation value. + +The first argument to the command is a path to an attribute. +This is a dot-separated sequence beginning with the name of +the resource or object, and ending with the name of the attribute to +set. To set operation value, `op_type` should be specified; when multi +operations exist like multi monitors, `interval` should be specified. + +Usage: +............... +set <path> <value> + +path:: id.[op_type.][interval.]name +............... +Examples: +............... +set vip1.ip 192.168.20.5 +set vm-a.force_stop 1 +set vip1.monitor.on-fail ignore +set drbd.monitor.10s.interval 20s +............... + +[[cmdhelp_configure_show,display CIB objects]] +==== `show` + +The `show` command displays CIB objects. Without any argument, it +displays all objects in the CIB, but the set of objects displayed by +`show` can be limited to only objects with the given IDs or by using +one or more of the special prefixes described below. + +The XML representation for the objects can be displayed by passing ++xml+ as the first argument. + +To show one or more specific objects, pass the object IDs as +arguments. + +To show all objects of a certain type, use the +type:+ prefix. + +To show all objects in a tag, use the +tag:+ prefix. + +To show all constraints related to a primitive, use the +related:+ prefix. + +To show all modified objects, pass the argument +changed+. + +The prefixes can be used together on a single command line. For +example, to show both the tag itself and the objects tagged by it the +following combination can be used: +show tag:my-tag my-tag+. + +To refine a selection of objects using multiple modifiers, the keywords ++and+ and +or+ can be used. For example, to select all primitives tagged ++foo+, the following combination can be used: ++show type:primitive and tag:foo+. + +To hide values when displaying the configuration, use the ++obscure:<glob>+ argument. This can be useful when sending the +configuration over a public channel, to avoid exposing potentially +sensitive information. The +<glob>+ argument is a bash-style pattern +matching attribute keys. + +In +/etc/crm/crm.conf+, +obscure_pattern+ option is the persisent configuration of CLI. +Example, for the high security concern, +............... +[core] +obscure_pattern = passw* | ip +............... +Which makes +crm configure show+ is equal to +............... +node-1:~ # crm configure show obscure:passw* obscure:ip +node 1084783297: node1 +primitive fence_device stonith:fence_ilo5 \ + params password="******" +primitive ip IPaddr2 \ + params ip="******" +............... +The default suggestion is +passw*+ +If you don't want to obscure, change the value to blank. + +Usage: +............... +show [xml] [<id> + | changed + | type:<type> + | tag:<id> + | related:<obj> + | obscure:<glob> + ...] + +type :: node | primitive | group | clone | ms | rsc_template + | location | colocation | order + | rsc_ticket + | property | rsc_defaults | op_defaults + | fencing_topology + | role | user | acl_target + | tag +............... + +Example: +............... +show webapp +show type:primitive +show xml tag:db tag:fs +show related:webapp +show type:primitive obscure:passwd +............... + +[[cmdhelp_configure_tag,Define resource tags]] +==== `tag` + +Define a resource tag. A tag is an id referring to one or more +resources, without implying any constraints between the tagged +resources. This can be useful for grouping conceptually related +resources. + +Usage: +............... +tag <tag-name>: <rsc> [<rsc> ...] +tag <tag-name> <rsc> [<rsc> ...] +............... +Example: +............... +tag web: p-webserver p-vip +tag ips server-vip admin-vip +............... + +[[cmdhelp_configure_template,edit and import a configuration from a template]] +==== `template` + +The specified template is loaded into the editor. It's up to the +user to make a good CRM configuration out of it. See also the +<<cmdhelp_template,template section>>. + +Usage: +............... +template [xml] url +............... +Example: +............... +template two-apaches.txt +............... + +[[cmdhelp_configure_upgrade,upgrade the CIB]] +==== `upgrade` + +Attempts to upgrade the CIB to validate with the current +version. Commonly, this is required if the error +`CIB not supported` occurs. It typically means that the +active CIB version is coming from an older release. + +As a safety precaution, the force argument is required if the ++validation-with+ attribute is set to anything other than ++0.6+. Thus in most cases, it is required. + +Usage: +............... +upgrade [force] +............... + +Example: +............... +upgrade force +............... + +[[cmdhelp_configure_user,define user access rights]] +==== `user` + +Users which normally cannot view or manage cluster configuration +can be allowed access to parts of the CIB. The access is defined +by a set of +read+, +write+, and +deny+ rules as in role +definitions or by referencing roles. The latter is considered +best practice. + +For more information on rule expressions, see +<<topics_Syntax_RuleExpressions,Syntax: Rule expressions>>. + +Usage: +............... +user <uid> {roles|rules} + +roles :: role:<role-ref> [role:<role-ref> ...] +rules :: rule [rule ...] +............... +Example: +............... +user joe \ + role:app1_admin \ + role:read_all +............... + +[[cmdhelp_configure_validate_all,call agent validate-all for resource]] +==== `validate-all` + +Call the `validate-all` action for the resource, if possible. + +Limitations: + +* The resource agent must implement the `validate-all` action. +* The current user must be root. +* The primitive resource must not use nvpair references. + +Usage: +............... +validate-all <rsc> +............... + + +[[cmdhelp_configure_verify,verify the CIB with crm_verify]] +==== `verify` + +Verify the contents of the CIB which would be committed. + +Usage: +............... +verify +............... + +[[cmdhelp_configure_xml,raw xml]] +==== `xml` + +Even though we promissed no xml, it may happen, but hopefully +very very seldom, that an element from the CIB cannot be rendered +in the configuration language. In that case, the element will be +shown as raw xml, prefixed by this command. That element can then +be edited like any other. If the shell finds out that after the +change it can digest it, then it is going to be converted into +the normal configuration language. Otherwise, there is no need to +use `xml` for configuration. + +Usage: +............... +xml <xml> +............... + +[[cmdhelp_template,edit and import a configuration from a template]] +=== `template` - Import configuration from templates + +User may be assisted in the cluster configuration by templates +prepared in advance. Templates consist of a typical ready +configuration which may be edited to suit particular user needs. + +This command enters a template level where additional commands +for configuration/template management are available. + +[[cmdhelp_template_apply,process and apply the current configuration to the current CIB]] +==== `apply` + +Copy the current or given configuration to the current CIB. By +default, the CIB is replaced, unless the method is set to +"update". + +Usage: +............... +apply [<method>] [<config>] + +method :: replace | update +............... + +[[cmdhelp_template_delete,delete a configuration]] +==== `delete` + +Remove a configuration. The loaded (active) configuration may be +removed by force. + +Usage: +............... +delete <config> [force] +............... + +[[cmdhelp_template_edit,edit a configuration]] +==== `edit` + +Edit current or given configuration using your favourite editor. + +Usage: +............... +edit [<config>] +............... + +[[cmdhelp_template_list,list configurations/templates]] +==== `list` + +When called with no argument, lists existing templates and +configurations. + +Given the argument +templates+, lists the available templates. + +Given the argument +configs+, lists the available configurations. + +Usage: +............... +list [templates|configs] +............... + +[[cmdhelp_template_load,load a configuration]] +==== `load` + +Load an existing configuration. Further `edit`, `show`, and +`apply` commands will refer to this configuration. + +Usage: +............... +load <config> +............... + +[[cmdhelp_template_new,create a new configuration from templates]] +==== `new` + +Create a new configuration from one or more templates. Note that +configurations and templates are kept in different places, so it +is possible to have a configuration name equal a template name. + +If you already know which parameters are required, you can set +them directly on the command line. + +The parameter name +id+ is set by default to the name of the +configuration. + +If no parameters are being set and you don't want a particular name +for your configuration, you can call this command with a template name +as the only parameter. A unique configuration name based on the +template name will be generated. + +Usage: +............... +new [<config>] <template> [<template> ...] [params name=value ...] +............... + +Example: +............... +new vip virtual-ip +new bigfs ocfs2 params device=/dev/sdx8 directory=/bigfs +new apache +............... + +[[cmdhelp_template_show,show the processed configuration]] +==== `show` + +Process the current or given configuration and display the result. + +Usage: +............... +show [<config>] +............... + +[[cmdhelp_cibstatus,CIB status management and editing]] +=== `cibstatus` - CIB status management and editing + +The `status` section of the CIB keeps the current status of nodes +and resources. It is modified _only_ on events, i.e. when some +resource operation is run or node status changes. For obvious +reasons, the CRM has no user interface with which it is possible +to affect the status section. From the user's point of view, the +status section is essentially a read-only part of the CIB. The +current status is never even written to disk, though it is +available in the PE (Policy Engine) input files which represent +the history of cluster motions. The current status may be read +using the +cibadmin -Q+ command. + +It may sometimes be of interest to see how status changes would +affect the Policy Engine. The set of `cibstatus` level commands +allow the user to load status sections from various sources and +then insert or modify resource operations or change nodes' state. + +The effect of those changes may then be observed by running the +<<cmdhelp_configure_ptest,`ptest`>> command at the `configure` level +or `simulate` and `run` commands at this level. The `ptest` +runs with the user edited CIB whereas the latter two commands +run with the CIB which was loaded along with the status section. + +The `simulate` and `run` commands as well as all status +modification commands are implemented using `crm_simulate(8)`. + +[[cmdhelp_cibstatus_load,load the CIB status section]] +==== `load` + +Load a status section from a file, a shadow CIB, or the running +cluster. By default, the current (+live+) status section is +modified. Note that if the +live+ status section is modified it +is not going to be updated if the cluster status changes, because +that would overwrite the user changes. To make `crm` drop changes +and resume use of the running cluster status, run +load live+. + +All CIB shadow configurations contain the status section which is +a snapshot of the status section taken at the time the shadow was +created. Obviously, this status section doesn't have much to do +with the running cluster status, unless the shadow CIB has just +been created. Therefore, the `ptest` command by default uses the +running cluster status section. + +Usage: +............... +load {<file>|shadow:<cib>|live} +............... +Example: +............... +load bug-12299.xml +load shadow:test1 +............... + +[[cmdhelp_cibstatus_node,change node status]] +==== `node` + +Change the node status. It is possible to throw a node out of +the cluster, make it a member, or set its state to unclean. + ++online+:: Set the +node_state+ `crmd` attribute to +online+ +and the +expected+ and +join+ attributes to +member+. The effect +is that the node becomes a cluster member. + ++offline+:: Set the +node_state+ `crmd` attribute to +offline+ +and the +expected+ attribute to empty. This makes the node +cleanly removed from the cluster. + ++unclean+:: Set the +node_state+ `crmd` attribute to +offline+ +and the +expected+ attribute to +member+. In this case the node +has unexpectedly disappeared. + +Usage: +............... +node <node> {online|offline|unclean} +............... +Example: +............... +node xen-b unclean +............... + +[[cmdhelp_cibstatus_op,edit outcome of a resource operation]] +==== `op` + +Edit the outcome of a resource operation. This way you can +tell CRM that it ran an operation and that the resource agent +returned certain exit code. It is also possible to change the +operation's status. In case the operation status is set to +something other than +done+, the exit code is effectively +ignored. + +Usage: +............... +op <operation> <resource> <exit_code> [<op_status>] [<node>] + +operation :: probe | monitor[:<n>] | start | stop | + promote | demote | notify | migrate_to | migrate_from +exit_code :: <rc> | success | generic | args | + unimplemented | perm | installed | configured | not_running | + master | failed_master +op_status :: pending | done | cancelled | timeout | notsupported | error + +n :: the monitor interval in seconds; if omitted, the first + recurring operation is referenced +rc :: numeric exit code in range 0..9 +............... +Example: +............... +op start d1 xen-b generic +op start d1 xen-b 1 +op monitor d1 xen-b not_running +op stop d1 xen-b 0 timeout +............... + +[[cmdhelp_cibstatus_origin,display origin of the CIB status section]] +==== `origin` + +Show the origin of the status section currently in use. This +essentially shows the latest `load` argument. + +Usage: +............... +origin +............... + +[[cmdhelp_cibstatus_quorum,set the quorum]] +==== `quorum` + +Set the quorum value. + +Usage: +............... +quorum <bool> +............... +Example: +............... +quorum false +............... + +[[cmdhelp_cibstatus_run,run policy engine]] +==== `run` + +Run the policy engine with the edited status section. + +Add a string of +v+ characters to increase verbosity. Specify ++scores+ to see allocation scores also. +utilization+ turns on +information about the remaining capacity of nodes. + +If you have graphviz installed and X11 session, `dotty(1)` is run +to display the changes graphically. + +Usage: +............... +run [nograph] [v...] [scores] [utilization] +............... +Example: +............... +run +............... + +[[cmdhelp_cibstatus_save,save the CIB status section]] +==== `save` + +The current internal status section with whatever modifications +were performed can be saved to a file or shadow CIB. + +If the file exists and contains a complete CIB, only the status +section is going to be replaced and the rest of the CIB will +remain intact. Otherwise, the current user edited configuration +is saved along with the status section. + +Note that all modifications are saved in the source file as soon +as they are run. + +Usage: +............... +save [<file>|shadow:<cib>] +............... +Example: +............... +save bug-12299.xml +............... + +[[cmdhelp_cibstatus_show,show CIB status section]] +==== `show` + +Show the current status section in the XML format. Brace yourself +for some unreadable output. Add +changed+ option to get a human +readable output of all changes. + +Usage: +............... +show [changed] +............... + +[[cmdhelp_cibstatus_simulate,simulate cluster transition]] +==== `simulate` + +Run the policy engine with the edited status section and simulate +the transition. + +Add a string of +v+ characters to increase verbosity. Specify ++scores+ to see allocation scores also. +utilization+ turns on +information about the remaining capacity of nodes. + +If you have graphviz installed and X11 session, `dotty(1)` is run +to display the changes graphically. + +Usage: +............... +simulate [nograph] [v...] [scores] [utilization] +............... +Example: +............... +simulate +............... + +[[cmdhelp_cibstatus_ticket,manage tickets]] +==== `ticket` + +Modify the ticket status. Tickets can be granted and revoked. +Granted tickets could be activated or put in standby. + +Usage: +............... +ticket <ticket> {grant|revoke|activate|standby} +............... +Example: +............... +ticket ticketA grant +............... + +[[cmdhelp_assist,Configuration assistant]] +=== `assist` - Configuration assistant + +The `assist` sublevel is a collection of helper +commands that create or modify resources and +constraints, to simplify the creation of certain +configurations. + +For more information on individual commands, see +the help text for those commands. + +[[cmdhelp_assist_template,Create template for primitives]] +==== `template` + +This command takes a list of primitives as argument, and creates a new +`rsc_template` for these primitives. It can only do this if the +primitives do not already share a template and are of the same type. + +Usage: +........ +template primitive-1 primitive-2 primitive-3 +........ + +[[cmdhelp_assist_weak-bond,Create a weak bond between resources]] +==== `weak-bond` + +A colocation between a group of resources says that the resources +should be located together, but it also means that those resources are +dependent on each other. If one of the resources fails, the others +will be restarted. + +If this is not desired, it is possible to circumvent: By placing the +resources in a non-sequential set and colocating the set with a dummy +resource which is not monitored, the resources will be placed together +but will have no further dependency on each other. + +This command creates both the constraint and the dummy resource needed +for such a colocation. + +Usage: +........ +weak-bond resource-1 resource-2 +........ + +[[cmdhelp_maintenance,Maintenance mode commands]] +=== `maintenance` - Maintenance mode commands + +Maintenance mode commands are commands that manipulate resources +directly without going through the cluster infrastructure. Therefore, +it is essential to ensure that the cluster does not attempt to monitor +or manipulate the resources while these commands are being executed. + +To ensure this, these commands require that maintenance mode is set +either for the particular resource, or for the whole cluster. + +[[cmdhelp_maintenance_action,Invoke a resource action]] +==== `action` + +Invokes the given action for the resource. This is +done directly via the resource agent, so the command must +be issued while the cluster or the resource is in +maintenance mode. + +Unless the action is `start` or `monitor`, the action must be invoked +on the same node as where the resource is running. If the resource is +running on multiple nodes, the command will fail. + +To use SSH for executing resource actions on multiple nodes, append +`ssh` after the action name. This requires SSH access to be configured +between the nodes and the parallax python package to be installed. + +Usage: +............... +action <rsc> <action> +action <rsc> <action> ssh +............... +Example: +............... +action webserver reload +action webserver monitor ssh +............... + +[[cmdhelp_maintenance_off,Disable maintenance mode]] +==== `off` + +Disables maintenances mode, either for the whole cluster +or for the given resource. + +Usage: +............... +off +off <rsc> +............... +Example: +............... +off rsc1 +............... + +[[cmdhelp_maintenance_on,Enable maintenance mode]] +==== `on` + +Enables maintenances mode, either for the whole cluster +or for the given resource. + +Usage: +............... +on +on <rsc> +............... +Example: +............... +on rsc1 +............... + +[[cmdhelp_history,Cluster history]] +=== `history` - Cluster history + +Examining Pacemaker's history is a particularly involved task. The +number of subsystems to be considered, the complexity of the +configuration, and the set of various information sources, most of +which are not exactly human readable, keep analyzing resource or node +problems accessible to only the most knowledgeable. Or, depending on +the point of view, to the most persistent. The following set of +commands has been devised in hope to make cluster history more +accessible. + +Of course, looking at _all_ history could be time consuming regardless +of how good the tools at hand are. Therefore, one should first say +which period he or she wants to analyze. If not otherwise specified, +the last hour is considered. Logs and other relevant information is +collected using `crm report`. Since this process takes some time and +we always need fresh logs, information is refreshed in a much faster +way using the python parallax module. If +python-parallax+ is not +found on the system, examining a live cluster is still possible -- +though not as comfortable. + +Apart from examining a live cluster, events may be retrieved from a +report generated by `crm report` (see also the +-H+ option). In that +case we assume that the period stretching the whole report needs to be +investigated. Of course, it is still possible to further reduce the +time range. + +If you have discovered an issue that you want to show someone else, +you can use the `session pack` command to save the current session as +a tarball, similar to those generated by `crm report`. + +In order to minimize the size of the tarball, and to make it easier +for others to find the interesting events, it is recommended to limit +the time frame which the saved session covers. This can be done using +the `timeframe` command (example below). + +It is also possible to name the saved session using the `session save` +command. + +Example: +............... +crm(live)history# limit "Jul 18 12:00" "Jul 18 12:30" +crm(live)history# session save strange_restart +crm(live)history# session pack +Report saved in .../strange_restart.tar.bz2 +crm(live)history# +............... + +[[cmdhelp_history_detail,set the level of detail shown]] +==== `detail` + +How much detail to show from the logs. Valid detail levels are either +`0` or `1`, where `1` is the highest detail level. The default detail +level is `0`. + +Usage: +............... +detail <detail_level> + +detail_level :: small integer (defaults to 0) +............... +Example: +............... +detail 1 +............... + +[[cmdhelp_history_diff,cluster states/transitions difference]] +==== `diff` + +A transition represents a change in cluster configuration or +state. Use `diff` to see what has changed between two +transitions. + +If you want to specify the current cluster configuration and +status, use the string +live+. + +Normally, the first transition specified should be the one which +is older, but we are not going to enforce that. + +Note that a single configuration update may result in more than +one transition. + +Usage: +............... +diff <pe> <pe> [status] [html] + +pe :: <number>|<index>|<file>|live +............... +Examples: +............... +diff 2066 2067 +diff pe-input-2080.bz2 live status +............... + +[[cmdhelp_history_events,Show events in log]] +==== `events` + +By analysing the log output and looking for particular +patterns, the `events` command helps sifting through +the logs to find when particular events like resources +changing state or node failure may have occurred. + +This can be used to generate a combined list of events +from all nodes. + +Usage: +............... +events +............... + +Example: +............... +events +............... + +[[cmdhelp_history_exclude,exclude log messages]] +==== `exclude` + +If a log is infested with irrelevant messages, those messages may +be excluded by specifying a regular expression. The regular +expressions used are Python extended. This command is additive. +To drop all regular expressions, use +exclude clear+. Run +`exclude` only to see the current list of regular expressions. +Excludes are saved along with the history sessions. + +Usage: +............... +exclude [<regex>|clear] +............... +Example: +............... +exclude kernel.*ocfs2 +............... + +[[cmdhelp_history_graph,generate a directed graph from the PE file]] +==== `graph` + +Create a graphviz graphical layout from the PE file (the +transition). Every transition contains the cluster configuration +which was active at the time. See also <<cmdhelp_configure_graph,generate a directed graph +from configuration>>. + +Usage: +............... +graph <pe> [<gtype> [<file> [<img_format>]]] + +gtype :: dot +img_format :: `dot` output format (see the +-T+ option) +............... +Example: +............... +graph -1 +graph 322 dot clu1.conf.dot +graph 322 dot clu1.conf.svg svg +............... + +[[cmdhelp_history_info,Cluster information summary]] +==== `info` + +The `info` command provides a summary of the information source, which +can be either a live cluster snapshot or a previously generated +report. + +Usage: +............... +info +............... +Example: +............... +info +............... + +[[cmdhelp_history_latest,show latest news from the cluster]] +==== `latest` + +The `latest` command shows a bit of recent history, more +precisely whatever happened since the last cluster change (the +latest transition). If the transition is running, the shell will +first wait until it finishes. + +Usage: +............... +latest +............... +Example: +............... +latest +............... + +[[cmdhelp_history_limit,limit timeframe to be examined]] +==== `limit` (`timeframe`) + +This command can be used to modify the time span to examine. All +history commands look at events within a certain time span. + +For the `live` source, the default time span is the _last hour_. + +There is no time span limit for the `hb_report` source. + +The time period is parsed by the `dateutil` python module. It +covers a wide range of date formats. For instance: + +- 3:00 (today at 3am) +- 15:00 (today at 3pm) +- 2010/9/1 2pm (September 1st 2010 at 2pm) + +For more examples of valid time/date statements, please refer to the +`python-dateutil` documentation: + +- https://dateutil.readthedocs.org/[dateutil.readthedocs.org] + +If the dateutil module is not available, then the time is parsed using +strptime and only the kind as printed by `date(1)` is allowed: + +- Tue Sep 15 20:46:27 CEST 2010 + +Usage: +............... +limit [<from_time>] [<to_time>] +............... +Examples: +............... +limit 10:15 +limit 15h22m 16h +limit "Sun 5 20:46" "Sun 5 22:00" +............... + +[[cmdhelp_history_log,log content]] +==== `log` + +Show messages logged on one or more nodes. Leaving out a node +name produces combined logs of all nodes. Messages are sorted by +time and, if the terminal emulations supports it, displayed in +different colours depending on the node to allow for easier +reading. + +The sorting key is the timestamp as written by syslog which +normally has the maximum resolution of one second. Obviously, +messages generated by events which share the same timestamp may +not be sorted in the same way as they happened. Such close events +may actually happen fairly often. + +Usage: +............... +log [<node> [<node> ...] ] +............... +Example: +............... +log node-a +............... + +[[cmdhelp_history_node,node events]] +==== `node` + +Show important events that happened on a node. Important events +are node lost and join, standby and online, and fence. Use either +node names or extended regular expressions. + +Usage: +............... +node <node> [<node> ...] +............... +Example: +............... +node node1 +............... + +[[cmdhelp_history_peinputs,list or get PE input files]] +==== `peinputs` + +Every event in the cluster results in generating one or more +Policy Engine (PE) files. These files describe future motions of +resources. The files are listed as full paths in the current +report directory. Add +v+ to also see the creation time stamps. + +Usage: +............... +peinputs [{<range>|<number>} ...] [v] + +range :: <n1>:<n2> +............... +Example: +............... +peinputs +peinputs 440:444 446 +peinputs v +............... + +[[cmdhelp_history_refresh,refresh live report]] +==== `refresh` + +This command makes sense only for the +live+ source and makes +`crm` collect the latest logs and other relevant information from +the logs. If you want to make a completely new report, specify ++force+. + +Usage: +............... +refresh [force] +............... + +[[cmdhelp_history_resource,resource events]] +==== `resource` + +Show actions and any failures that happened on all specified +resources on all nodes. Normally, one gives resource names as +arguments, but it is also possible to use extended regular +expressions. Note that neither groups nor clones or master/slave +names are ever logged. The resource command is going to expand +all of these appropriately, so that clone instances or resources +which are part of a group are shown. + +Usage: +............... +resource <rsc> [<rsc> ...] +............... +Example: +............... +resource bigdb public_ip +resource my_.*_db2 +resource ping_clone +............... + +[[cmdhelp_history_session,manage history sessions]] +==== `session` + +Sometimes you may want to get back to examining a particular +history period or bug report. In order to make that easier, the +current settings can be saved and later retrieved. + +If the current history being examined is coming from a live +cluster the logs, PE inputs, and other files are saved too, +because they may disappear from nodes. For the existing reports +coming from `hb_report`, only the directory location is saved +(not to waste space). + +A history session may also be packed into a tarball which can +then be sent to support. + +Leave out subcommand to see the current session. + +Usage: +............... +session [{save|load|delete} <name> | pack [<name>] | update | list] +............... +Examples: +............... +session save bnc966622 +session load rsclost-2 +session list +............... + +[[cmdhelp_history_setnodes,set the list of cluster nodes]] +==== `setnodes` + +In case the host this program runs on is not part of the cluster, +it is necessary to set the list of nodes. + +Usage: +............... +setnodes node <node> [<node> ...] +............... +Example: +............... +setnodes node_a node_b +............... + +[[cmdhelp_history_show,show status or configuration of the PE input file]] +==== `show` + +Every transition is saved as a PE file. Use this command to +render that PE file either as configuration or status. The +configuration output is the same as `crm configure show`. + +Usage: +............... +show <pe> [status] + +pe :: <number>|<index>|<file>|live +............... +Examples: +............... +show 2066 +show pe-input-2080.bz2 status +............... + +[[cmdhelp_history_source,set source to be examined]] +==== `source` + +Events to be examined can come from the current cluster or from a +`hb_report` report. This command sets the source. `source live` +sets source to the running cluster and system logs. If no source +is specified, the current source information is printed. + +In case a report source is specified as a file reference, the file +is going to be unpacked in place where it resides. This directory +is not removed on exit. + +Usage: +............... +source [<dir>|<file>|live] +............... +Examples: +............... +source live +source /tmp/customer_case_22.tar.bz2 +source /tmp/customer_case_22 +source +............... + +[[cmdhelp_history_transition,show transition]] +==== `transition` + +This command will print actions planned by the PE and run +graphviz (`dotty`) to display a graphical representation of the +transition. Of course, for the latter an X11 session is required. +This command invokes `ptest(8)` in background. + +The +showdot+ subcommand runs graphviz (`dotty`) to display a +graphical representation of the +.dot+ file which has been +included in the report. Essentially, it shows the calculation +produced by `pengine` which is installed on the node where the +report was produced. In optimal case this output should not +differ from the one produced by the locally installed `pengine`. + +The `log` subcommand shows the full log for the duration of the +transition. + +A transition can also be saved to a CIB shadow for further +analysis or use with `cib` or `configure` commands (use the +`save` subcommand). The shadow file name defaults to the name of +the PE input file. + +If the PE input file number is not provided, it defaults to the +last one, i.e. the last transition. The last transition can also +be referenced with number 0. If the number is negative, then the +corresponding transition relative to the last one is chosen. + +If there are warning and error PE input files or different nodes +were the DC in the observed timeframe, it may happen that PE +input file numbers collide. In that case provide some unique part +of the path to the file. + +After the `ptest` output, logs about events that happened during +the transition are printed. + +The `tags` subcommand scans the logs for the transition and return a +list of key events during that transition. For example, the tag ++error+ will be returned if there are any errors logged during the +transition. + +Usage: +............... +transition [<number>|<index>|<file>] [nograph] [v...] [scores] [actions] [utilization] +transition showdot [<number>|<index>|<file>] +transition log [<number>|<index>|<file>] +transition save [<number>|<index>|<file> [name]] +transition tags [<number>|<index>|<file>] +............... +Examples: +............... +transition +transition 444 +transition -1 +transition pe-error-3.bz2 +transition node-a/pengine/pe-input-2.bz2 +transition showdot 444 +transition log +transition save 0 enigma-22 +............... + +[[cmdhelp_history_transitions,List transitions]] +==== `transitions` + +A transition represents a change in cluster configuration or +state. This command lists the transitions in the current timeframe. + +Usage: +............... +transitions +............... +Example: +............... +transitions +............... + + +[[cmdhelp_history_wdiff,cluster states/transitions difference]] +==== `wdiff` + +A transition represents a change in cluster configuration or +state. Use `wdiff` to see what has changed between two +transitions as word differences on a line-by-line basis. + +If you want to specify the current cluster configuration and +status, use the string +live+. + +Normally, the first transition specified should be the one which +is older, but we are not going to enforce that. + +Note that a single configuration update may result in more than +one transition. + +Usage: +............... +wdiff <pe> <pe> [status] + +pe :: <number>|<index>|<file>|live +............... +Examples: +............... +wdiff 2066 2067 +wdiff pe-input-2080.bz2 live status +............... + +[[cmdhelp_root_report,Create cluster status report]] +=== `report` + +Interface to a tool for creating a cluster report. A report is an +archive containing log files, configuration files, system information +and other relevant data for a given time period. This is a useful tool +for collecting data to attach to bug reports, or for detecting the +root cause of errors resulting in resource failover, for example. + +See `crmsh_hb_report(8)` for more details on arguments, +or call `crm report -h` + +Usage: +............... +report -f {time|"cts:"testnum} [-t time] [-u user] [-l file] + [-n nodes] [-E files] [-p patt] [-L patt] [-e prog] + [-MSDZAVsvhd] [dest] +............... + +Examples: +............... +report -f 2pm report_1 +report -f "2007/9/5 12:30" -t "2007/9/5 14:00" report_2 +report -f 1:00 -t 3:00 -l /var/log/cluster/ha-debug report_3 +report -f "09sep07 2:00" -u hbadmin report_4 +report -f 18:00 -p "usern.*" -p "admin.*" report_5 +report -f cts:133 ctstest_133 +............... + +=== `end` (`cd`, `up`) + +The `end` command ends the current level and the user moves to +the parent level. This command is available everywhere. + +Usage: +............... +end +............... + +=== `help` + +The `help` command prints help for the current level or for the +specified topic (command). This command is available everywhere. + +Usage: +............... +help [<topic>] +............... + +=== `quit` (`exit`, `bye`) + +Leave the program. + +BUGS +---- +Even though all sensible configurations (and most of those that +are not) are going to be supported by the crm shell, I suspect +that it may still happen that certain XML constructs may confuse +the tool. When that happens, please file a bug report. + +The crm shell will not try to update the objects it does not +understand. Of course, it is always possible to edit such objects +in the XML format. + +AUTHORS +------- +Dejan Muhamedagic, <dejan@suse.de> +Kristoffer Gronlund <kgronlund@suse.com> +and many OTHERS + +SEE ALSO +-------- +crm_resource(8), crm_attribute(8), crm_mon(8), cib_shadow(8), +ptest(8), dotty(1), crm_simulate(8), cibadmin(8) + + +COPYING +------- +Copyright \(C) 2008-2013 Dejan Muhamedagic. +Copyright \(C) 2013 Kristoffer Gronlund. + +Free use of this software is granted under the terms of the GNU General Public License (GPL). + +////////////////////// + vim:ts=4:sw=4:expandtab: +////////////////////// diff --git a/doc/website-v1/news.adoc b/doc/website-v1/news.adoc new file mode 100644 index 0000000..9cd0d6d --- /dev/null +++ b/doc/website-v1/news.adoc @@ -0,0 +1,26 @@ += News + +link:/news/2017-01-31-release-3_0_0[2017-01-31 10:00] + +:leveloffset: 1 + +include::news/2017-01-31-release-3_0_0.adoc[] + +:leveloffset: 0 + +'''' +* link:/news/2016-09-05-release-2_2_2[2016-09-05 19:00 Releasing crmsh version 2.2.2] +* link:/news/2016-09-02-release-2_3_1[2016-09-02 10:00 Releasing crmsh version 2.3.1] +* link:/news/2016-09-01-release-2_1_7[2016-09-01 09:00 Announcing crmsh stable release 2.1.7] +* link:/news/2016-08-12-release-2_3_0[2016-08-12 10:30 Releasing crmsh version 2.3.0] +* link:/news/2016-04-28-release-2_2_1[2016-04-28 01:00 crmsh 2.2.1 and 2.1.6 are released] +* link:/news/2016-01-15-release-2_2_0[2016-01-15 15:00 crmsh 2.2.0 is released] +* link:/news/2016-01-12-release-2_1_5[2016-01-12 10:00 Announcing crmsh stable release 2.1.5] +* link:/news/2015-05-25-getting-started-jp[2015-05-25 13:30 Getting Started translated to Japanese] +* link:/news/2015-05-13-release-2_1_4[2015-05-13 15:30 Announcing crmsh stable release 2.1.4] +* link:/news/2015-04-10-release-2_1_3[2015-04-10 12:30 Announcing crmsh stable release 2.1.3] +* link:/news/2015-01-26-release-2_1_2[2015-01-26 11:05 Announcing crmsh release 2.1.2] +* link:/news/2014-10-28-release-2_1_1[2014-10-29 00:20 Announcing crmsh release 2.1.1] +* link:/news/2014-06-30-release-2_1[2014-06-30 09:00 Announcing crmsh release 2.1] + +link:https://savannah.nongnu.org/news/?group_id=10890[Old News Archive] diff --git a/doc/website-v1/news/2014-06-30-release-2_1.adoc b/doc/website-v1/news/2014-06-30-release-2_1.adoc new file mode 100644 index 0000000..4c59a90 --- /dev/null +++ b/doc/website-v1/news/2014-06-30-release-2_1.adoc @@ -0,0 +1,93 @@ +Announcing crmsh release 2.1 +============================ +:Author: Kristoffer Gronlund +:Email: kgronlund@suse.com +:Date: 2014-06-30 09:00 + +Today we are proud to announce the release of `crmsh` version 2.1! +This version primarily fixes all known issues found since the release +of `crmsh` 2.0 in April, but also has some major new features. + +A massive thank you to everyone who has helped out with bug fixes, +comments and contributions for this release! + +For a complete list of changes since the previous version, please +refer to the changelog: + +* https://github.com/crmsh/crmsh/blob/2.1.0/ChangeLog + +Packages for several popular Linux distributions can be downloaded +from the Stable repository at the OBS: + +* http://download.opensuse.org/repositories/network:/ha-clustering:/Stable/ + +Archives of the tagged release: + +* https://github.com/crmsh/crmsh/archive/2.1.0.tar.gz +* https://github.com/crmsh/crmsh/archive/2.1.0.zip + +Here are some of the highlights of this release: + +== Rule expressions in attribute lists + +One of the biggest features in this release is full support for rule +expressions wherever the XML syntax allows them. + +Here is an example of using rule expressions in an attribute list in +order to set the virtual IP of an IPAddr2 resource to a different +value on a specific node. + +---- +primitive vip-on-node1 IPAddr2 \ + rule 10: #uname eq node1 ip=10.0.0.5 \ + rule 1: ip=10.0.0.6 +---- + +== Tags in the CIB + +A new feature added to Pacemaker recently is tags. This is a way +to refer to multiple resources at once without creating any +colocation or ordering relationship between them. For example, you +could add all resources related to the database to a db tag, and +then stop or start them all with a single command. + +---- +tag db drbd:Master fs sql-db +---- + +It is also possible to refer to tags in constraints. + +== Wildcards in show/edit + +The configure show and edit commands can now use glob-style +wildcards to refer to multiple resources: + +---- +configure edit db-* +---- + +== Nvpair references + +Sometimes, different resources name the same parameters with different +names. For example, an IPAddr2 may have an ip parameter that should be +the same as a web servers server_ip parameter. By using nvpair +references, it is possible to configure the ip in a single location. + +Note that this is a new feature in Pacemaker 1.1.12 and up. + +---- +primitive vip IPAddr2 params $my-ip:ip=192.168.0.1 +primitive www apache params @my-ip:server_ip +---- + +== New ACL syntax + +The support for Access Control Lists has been revised in Pacemaker +1.1.12, and this release of crmsh supports the new syntax. Two new +commands have been added: `acl_target` and `acl_group`. For more details, +see the documentation. + +Thank you, + +Kristoffer and Dejan + diff --git a/doc/website-v1/news/2014-10-28-release-2_1_1.adoc b/doc/website-v1/news/2014-10-28-release-2_1_1.adoc new file mode 100644 index 0000000..6b67f4f --- /dev/null +++ b/doc/website-v1/news/2014-10-28-release-2_1_1.adoc @@ -0,0 +1,58 @@ +Announcing crmsh release 2.1.1 +============================== +:Author: Kristoffer Gronlund +:Email: kgronlund@suse.com +:Date: 2014-10-29 00:20 + +Today we are proud to announce the release of `crmsh` version 2.1.1! +This version primarily fixes all known issues found since the release +of `crmsh` 2.1 in June. We recommend that all users of crmsh upgrade +to this version, especially if using Pacemaker 1.1.12 or newer. + +A massive thank you to everyone who has helped out with bug fixes, +comments and contributions for this release! + +For a complete list of changes since the previous version, please +refer to the changelog: + +* https://github.com/crmsh/crmsh/blob/2.1.1/ChangeLog + +Packages for several popular Linux distributions can be downloaded +from the Stable repository at the OBS: + +* http://download.opensuse.org/repositories/network:/ha-clustering:/Stable/ + +Archives of the tagged release: + +* https://github.com/crmsh/crmsh/archive/2.1.1.tar.gz +* https://github.com/crmsh/crmsh/archive/2.1.1.zip + +Changes since the previous release: + + - cibconfig: Clean up output from crm_verify (bnc#893138) + - high: constants: Add acl_target and acl_group to cib_cli_map (bnc#894041) + - high: parse: split shortcuts into valid rules + - medium: Handle broken CIB in find_objects + - high: scripts: Handle corosync.conf without nodelist in add-node (bnc#862577) + - medium: config: Assign default path in all cases + - high: cibconfig: Generate valid CLI syntax for attribute lists (bnc#897462) + - high: cibconfig: Add tag:<tag> to get all resources in tag + - doc: Documentation for show tag:<tag> + - low: report: Sort list of nodes + - high: parse: Allow empty attribute values in nvpairs (bnc#898625) + - high: cibconfig: Delay reinitialization after commit + - low: cibconfig: Improve wording of commit prompt + - low: cibconfig: Fix vim modeline + - high: report: Find nodes for any log type (boo#900654) + - high: hb_report: Collect logs from journald (boo#900654) + - high: cibconfig: Don't crash if given an invalid pattern (bnc#901714) + - high: xmlutil: Filter list of referenced resources (bnc#901714) + - medium: ui_resource: Only act on resources (#64) + - medium: ui_resource: Flatten, then filter (#64) + - high: ui_resource: Use correct name for error function (bnc#901453) + - high: ui_resource: resource trace failed if operation existed (bnc#901453) + - Improved test suite + +Thank you, + +Kristoffer and Dejan diff --git a/doc/website-v1/news/2015-01-26-release-2_1_2.adoc b/doc/website-v1/news/2015-01-26-release-2_1_2.adoc new file mode 100644 index 0000000..081bf1b --- /dev/null +++ b/doc/website-v1/news/2015-01-26-release-2_1_2.adoc @@ -0,0 +1,69 @@ +Announcing crmsh release 2.1.2 +============================== +:Author: Kristoffer Gronlund +:Email: kgronlund@suse.com +:Date: 2015-01-26 11:05 + +Today we are proud to announce the release of `crmsh` version 2.1.2! +This version primarily fixes all known issues found since the release +of `crmsh` 2.1.1 in October. We recommend that all users of crmsh upgrade +to this version, especially if using Pacemaker 1.1.12 or newer. + +A massive thank you to everyone who has helped out with bug fixes, +comments and contributions for this release! + +For a complete list of changes since the previous version, please +refer to the changelog: + +* https://github.com/crmsh/crmsh/blob/2.1.2/ChangeLog + +Packages for several popular Linux distributions can be downloaded +from the Stable repository at the OBS: + +* http://download.opensuse.org/repositories/network:/ha-clustering:/Stable/ + +Archives of the tagged release: + +* https://github.com/crmsh/crmsh/archive/2.1.2.tar.gz +* https://github.com/crmsh/crmsh/archive/2.1.2.zip + +Changes since the previous release: + + - medium: ui_resource: Set probe interval 0 if not set (bnc#905050) + - doc: Document probe op in resource trace (bnc#905050) + - high: config: Fix path to system-wide crm.conf (#67) + - medium: config: Fall back to /etc/crm/crmsh.conf (#67) + - low: cliformat: Colorize id: as identifier (boo#905338) + - medium: cibconfig: Don't bump epoch if stripping version + - medium: ui_context: Lazily import readline + - medium: config: Add core.ignore_missing_metadata (#68) (boo#905910) + - medium: cibconfig: Strip digest from v1 diffs (bnc#914098) + - medium: cibconfig: Detect v1 format and don't patch container changes (bnc#914098) + - high: xmlutil: Treat node type=member as normal (boo#904698) + - medium: xmlutil: Use idmgmt when creating new elements (bnc#901543) + - low: ui_resource: --reprobe and --refresh are deprecated (bnc#905092) + - doc: Document deprecation of refresh and reprobe (bnc#905092) + - medium: parse: Support resource-discovery in location constraints + - medium: Allow removing groups even if is_running (boo#905271) + - medium: cibconfig: Delete containers first in edits (boo#905268) + - medium: ui_history: Fix crash using empty object set + - Low: term: get rid of annying ^O in piped-to-less-R output + - medium: parse: Allow nvpair with no value using name= syntax (#71) + - medium: parse: Enable name[=value] for nvpair (#71) + - medium: utils: Check if path basename is less (#74) + - medium: utils: crm_daemon_dir is added to PATH in envsetup (#67) + - medium: cmd_status: Show pending if available, enable extra options + - high: utils: Locate binaries across sudo boundary (bnc#912483) + - Medium: history: match error/crit messages of pcmk 1.1.12 + - low: ui_options: Add underscore aliases for legacy options + - medium: constants: Fix transition start detection + - medium: constants: Update transition regex (#77) + - medium: orderedset: Add OrderedSet type + - medium: cibconfig: Use orderedset to avoid reordering bugs (#79) + - low: xmlutil: logic bug in sanity_check_nvpairs + - medium: util: Don't fall back to current time + - medium: report: Fall back to end_ts = start_ts + +Thank you, + +Kristoffer and Dejan diff --git a/doc/website-v1/news/2015-04-10-release-2_1_3.adoc b/doc/website-v1/news/2015-04-10-release-2_1_3.adoc new file mode 100644 index 0000000..c186ff0 --- /dev/null +++ b/doc/website-v1/news/2015-04-10-release-2_1_3.adoc @@ -0,0 +1,68 @@ +Announcing crmsh stable release 2.1.3 +===================================== +:Author: Kristoffer Gronlund +:Email: kgronlund@suse.com +:Date: 2015-04-10 12:30 + +Today we are proud to announce the release of `crmsh` version 2.1.3! +This version fixes all known issues found since the release of `crmsh` +2.1.2 in January. We recommend that all users of crmsh upgrade +to this version, especially if using Pacemaker 1.1.12 or newer. + +A massive thank you to everyone who has helped out with bug fixes, +comments and contributions for this release! + +For a complete list of changes since the previous version, please +refer to the changelog: + +* https://github.com/ClusterLabs/crmsh/blob/2.1.3/ChangeLog + +Packages for several popular Linux distributions can be downloaded +from the Stable repository at the OBS: + +* http://download.opensuse.org/repositories/network:/ha-clustering:/Stable/ + +Archives of the tagged release: + +* https://github.com/ClusterLabs/crmsh/archive/2.1.3.tar.gz +* https://github.com/ClusterLabs/crmsh/archive/2.1.3.zip + +Changes since the previous release: + + - medium: parse: nvpair attributes with no value = <nvpair name=".."/> (#71) + - doc: Add link to clusterlabs.org + - medium: report: Convert RE exception to simpler UI output + - medium: report: Include transitions with configuration changes (bnc#917131) + - medium: config: Fix case-sensitivity for booleans + - medium: ra: Handle non-OCF agent meta-data better + - Medium: cibconf: preserve cib user attributes + - low: cibconfig: Improved debug output when schema change fails + - medium: parse: Treat pacemaker-next schema as 2.0+ + - medium: schema: Test if node type is optional via schema + - medium: schema: Remove extra debug output + - low: pacemaker: Remove debug output + - medium: cibconfig: If a change results in no diff, exit silently + - medium: cibconfig: Allow delete of objects that don't exist without returning error code + - medium: cibconfig: Allow removal of non-existing elements if --force is set + - low: allow (0,1) as option booleans + - low: allow pacemaker 1.0 version detection + - Low: hb_report: add -Q to usage + - Low: hb_report: add -X option for extra ssh options + - doc: Move the main crmsh repository to the ClusterLabs organization on github + - high: ui_configure: Remove acl_group command (bnc#921056) + - high: cibconfig: Don't delete valid tickets when removing referenced objects (bnc#922039) + - high: ui_context: Wait for DC after commit, not before (#85) + - medium: templates: Clearer descriptions for editing templates (boo#921028) + - high: cibconfig: Derive id for ops from referenced resource name (boo#921028) + - medium: ui_template: Always generate id unless explicitly defined (boo#921028) + - low: template: Add 'new <template>' shortcut + - medium: ui_template: Make new command more robust (bnc#924641) + - medium: parse: Disallow location rules without resources + - high: parse: Don't allow constraints without applicants + - medium: cliformat: Escape double-quotes in nvpair values + - low: hb_report: Use crmsh config to find pengine/cib dirs (bsc#926377) + - low: main: Catch any ValueErrors that may leak through + +Thank you, + +Kristoffer and Dejan diff --git a/doc/website-v1/news/2015-05-13-release-2_1_4.adoc b/doc/website-v1/news/2015-05-13-release-2_1_4.adoc new file mode 100644 index 0000000..31297cf --- /dev/null +++ b/doc/website-v1/news/2015-05-13-release-2_1_4.adoc @@ -0,0 +1,126 @@ +Announcing crmsh stable release 2.1.4 +===================================== +:Author: Kristoffer Gronlund +:Email: kgronlund@suse.com +:Date: 2015-05-13 15:30 + +Today we are proud to announce the release of `crmsh` version 2.1.4! +2.1.4 is a minor bug fix release with no major issues, so users +already running 2.1.3 are mostly fine. Instead, the main reason +for releasing 2.1.4 is as an excuse to talk about some other things +that are happening with crmsh! + +The details for this release are available below. + +History Guide +~~~~~~~~~~~~~ + +Dejan has written a guide to using the crmsh history +command. For those who are unfamiliar with the history explorer or +want to know more about how to use it, this guide is a great +introduction to what it does and how to use it. + +History is not a new crmsh feature, but, as we failed to +advertise it and nothing works without proper marketing, it +probably hasn't seen a very wide use. That's surely a pity and we +hope that this gentle history guide is going to help. + +So, if you use crmsh and if you need help troubleshooting +clusters (I surely do!), take a look here: + +http://crmsh.github.io/history-guide/ + +FYI, the comprehensive crmsh help also has a short description of +the feature: + +........ +crm history help +........ + +Goes without saying: all commands are described too. + +If you don't use crmsh, you'll still find a lot of useful +information in the guide, so don't skip it. + +Hawk Presentation +~~~~~~~~~~~~~~~~~ + +I presented Hawk [1] and the History Explorer interface which +builds upon the crmsh history feature at openSUSE conf in The Hague +earlier this month. The video of that presentation is online here: + +++++++++++++ +<iframe width="420" height="315" src="https://www.youtube.com/embed/mngfxzXkFLw" frameborder="0" allowfullscreen></iframe> +++++++++++++ + +https://www.youtube.com/watch?v=mngfxzXkFLw + +[1]: https://github.com/ClusterLabs/hawk + + +2.2.0 Development News +~~~~~~~~~~~~~~~~~~~~~~~ + +While 2.1.4 is the latest stable release, I am also working on releasing +2.2.0 which will come with a bunch of new features. I'm still working +on some of these and not everything is in the repository yet, so +2.2.0 is probably at least a month or so away still. I was perhaps +a bit optimistic when I tagged RC1 back in October last year. ;) + +However, right now I'd like to focus on one thing that is already in +2.2.0 and which is available if you use the development packages from +OBS: command shorthands. This makes crmsh a lot more convenient to use +from the command line. Basically, you can use any unambiguous subset +of a command name to refer to that command, and crmsh will figure out +what you mean. This may sound confusing, so an example will help with +explaining what I mean: + +This is one way of showing the current cluster configuration: + +........ +crm configure show +........ + +However, now you can shorten this to the following: + +........ +crm cfg show +........ + +Other examples of shorthand are `crm rsc stop r1` or `crm st` +for status. And of course, tab completion in bash still works for +the shorthand variants. + +The examples used here are not comprehensive. crmsh is pretty clever +at figuring out which command was intended. Download the development +release and try it out! + +2.1.4 Details +~~~~~~~~~~~~~ + +For a complete list of changes since the previous version, please +refer to the changelog: + +* https://github.com/ClusterLabs/crmsh/blob/2.1.4/ChangeLog + +Packages for several popular Linux distributions can be downloaded +from the Stable repository at the OBS: + +* http://download.opensuse.org/repositories/network:/ha-clustering:/Stable/ + +Archives of the tagged release: + +* https://github.com/ClusterLabs/crmsh/archive/2.1.4.tar.gz +* https://github.com/ClusterLabs/crmsh/archive/2.1.4.zip + +Changes since the previous release: + +- Medium: hb_report: use faster zypper interface if available +- medium: ui_configure: Wait for DC when removing running resource +- low: schema: Don't leak PacemakerError exceptions (#93) +- parse: Don't require trailing colon in tag definitions +- medium: utils: Allow 1/0 as boolean values for parameters + +Thank you, + +Kristoffer and Dejan diff --git a/doc/website-v1/news/2015-05-25-getting-started-jp.adoc b/doc/website-v1/news/2015-05-25-getting-started-jp.adoc new file mode 100644 index 0000000..c5c6759 --- /dev/null +++ b/doc/website-v1/news/2015-05-25-getting-started-jp.adoc @@ -0,0 +1,17 @@ +Getting Started translated to Japanese +====================================== +:Author: Kristoffer Gronlund +:Email: kgronlund@suse.com +:Date: 2015-05-25 13:30 + +Many thanks to Motoharu Kubo at 3ware for offering to translate the +`crmsh` documentation to Japanese! + +The first document to be translated is the link:/start-guide/[Getting Started] guide, +now available in Japanese at the following location: + +* https://blog.3ware.co.jp/2015/05/crmsh-getting-started/ + +Thank you, +Kristoffer and Dejan + diff --git a/doc/website-v1/news/2016-01-12-release-2_1_5.adoc b/doc/website-v1/news/2016-01-12-release-2_1_5.adoc new file mode 100644 index 0000000..93a3242 --- /dev/null +++ b/doc/website-v1/news/2016-01-12-release-2_1_5.adoc @@ -0,0 +1,56 @@ +Announcing crmsh stable release 2.1.5 +===================================== +:Author: Kristoffer Gronlund +:Email: kgronlund@suse.com +:Date: 2016-01-12 10:00 + +Today we are proud to announce the release of `crmsh` version 2.1.5! +This release mainly consists of bug fixes, as well as compatibility +with Pacemaker 1.1.14. + +For a complete list of changes since the previous version, please +refer to the changelog: + +* https://github.com/ClusterLabs/crmsh/blob/2.1.5/ChangeLog + +Packages for several popular Linux distributions can be downloaded +from the Stable repository at the OBS: + +* http://download.opensuse.org/repositories/network:/ha-clustering:/Stable/ + +Archives of the tagged release: + +* https://github.com/ClusterLabs/crmsh/archive/2.1.5.tar.gz +* https://github.com/ClusterLabs/crmsh/archive/2.1.5.zip + +Changes since the previous release: + +- medium: report: Try to load source as session if possible (bsc#927407) +- medium: crm_gv: Wrap non-identifier names in quotes (bsc#931837) +- medium: crm_gv: Improved quoting of non-identifier node names (bsc#931837) +- medium: crm_pkg: Fix cluster init bug on RH-based systems +- medium: hb_report: Collect logs from pacemaker.log +- medium: constants: Add 'provides' meta attribute (bsc#936587) +- high: parse: Add attributes to terminator set (bsc#940920) +- Medium: cibconfig: skip sanity check for properties other than cib-bootstrap-options +- medium: config: Add report_tool_options (bsc#917638) +- low: main: Bash completion didn't handle sudo correctly +- high: report: New detection to fix missing transitions (bnc#917131) +- medium: report: Add pacemaker.log to find_node_log list (bsc#941734) +- high: hb_report: Prefer pacemaker.log if it exists (bsc#941681) +- high: report: Output format from pacemaker has changed (bsc#941681) +- high: report: Update transition edge regexes (bsc#942906) +- medium: report: Reintroduce empty transition pruning (bsc#943291) +- medium: log_patterns: Remove reference to function name in log patterns (bsc#942906) +- low: hb_report: Collect libqb version (bsc#943327) +- high: parse: Fix crash when referencing score types by name (bsc#940194) +- low: constants: Add meta attributes for remote nodes +- low: ui_history: Swap from and to times if to < from +- high: cibconfig: Do not fail on unknown pacemaker schemas (bsc#946893) +- high: log_patterns_118: Update the correct set of log patterns (bsc#942906) +- high: xmlutil: Order is significant in resource_set (bsc#955434) +- high: cibconfig: Fix XML import bug for cloned groups (bsc#959895) + +Thank you, + +Kristoffer and Dejan diff --git a/doc/website-v1/news/2016-01-15-release-2_2_0.adoc b/doc/website-v1/news/2016-01-15-release-2_2_0.adoc new file mode 100644 index 0000000..664526e --- /dev/null +++ b/doc/website-v1/news/2016-01-15-release-2_2_0.adoc @@ -0,0 +1,210 @@ +crmsh 2.2.0 is released +======================= +:Author: Kristoffer Gronlund +:Email: kgronlund@suse.com +:Date: 2016-01-15 15:00 + +In June of last year, I released Release Candidate 3 of crmsh 2.2.0, +and I honestly expected to have the final version ready no more than a +few weeks later. Well, it took around 6 months, but now it is finally +here! + +The source code can be downloaded from Github: + +* https://github.com/ClusterLabs/crmsh/releases/tag/2.2.0 + +Packages for several popular Linux distributions can be downloaded +from the Stable repository at the OBS: + +* http://download.opensuse.org/repositories/network:/ha-clustering:/Stable/ + +Archives of the tagged release: + +* https://github.com/ClusterLabs/crmsh/archive/2.2.0.tar.gz +* https://github.com/ClusterLabs/crmsh/archive/2.2.0.zip + +There are many new features and quite a few bug fixes, but I'll try to +describe some of the major changes below. For the full list of changes +since version 2.1, see the ChangeLog. I am also including the list +of changes since RC3 in the release notes below. + +* https://github.com/ClusterLabs/crmsh/blob/2.2.0/ChangeLog + +New features introduced in this version: + +* Support for many of the new features introduced in Pacemaker + 1.1.14, including improved support for Pacemaker Remote and + pattern-based fencing topology configurations. + +* The cluster scripts have been greatly improved and are now used as + the wizards backend for hawk. Together with the improvements in the + cluster scripts, the crmsh template sublevel has been deprecated, + and all of the templates are now available as cluster scripts. The + template sublevel will be removed in a future version. + +* The history explorer now replaces the combined log and event files + from hb_report using the commands crm history log and crm history + events. Additionally, the info summary now highlights transitions + that contain error logs or other potentially interesting events. + +* Improved verification before committing new changes, for example + crmsh now warns if colocation constraints are applied to group + children. + +* resource start / stop / restart now take multiple resource arguments. + +* Added a maintenance sublevel with commands intended to simplify the + use of maintenance mode. + +* Switched to using the python parallax library instead of pssh for + remote node communication. Parallax is a fork of pssh with + additional features added specifically for using it as a python + library rather than a command line tool. Packages for + python-parallax can be downloaded from the OBS, or it can be + installed from PyPI. + +Full list of changes since 2.2.0-rc3: + +- medium: history: Fix live report refresh (bsc#950422) (bsc#927414) +- medium: history: Ignore central log +- medium: cibconfig: Detect false container children +- low: clidisplay: Avoid crash when colorizing None +- medium: scripts: Load single file yml scripts +- medium: scripts: Reformat scripts to simplified form +- medium: ui_history: Add events command (bsc#952449) +- low: hb_report: Drop function from event patterns +- high: cibconfig: Preserve failure through edit (bsc#959965) +- high: cibconfig: fail if new object already exists (bsc#959965) +- medium: ui_cib: Call crm_shadow in batch mode to avoid spawning subshell (bsc#961392) +- high: cibconfig: Fix XML import bug for cloned groups (bsc#959895) +- high: ui_configure: Move validate-all validation to a separate command (bsc#956442) +- high: scripts: Don't require scripts to be an array of one element +- medium: scripts: Enable setting category in legacy wizards (bnc#957926) +- high: scripts: Don't delete steps from upgraded wizards (bnc#957925) +- high: ra: Only run validate-all if current user is root +- high: cibconfig: Call validate-all action on agent in verify (bsc#956442) +- high: script: Fix issues found in cluster scripts +- high: ui_ra: Add ra validate command (bsc#956442) +- low: resource: Fix unban alias for unmigrate +- high: ui_resource: Add constraints and operations commands +- high: ui_resource: Enable start/stop/status for multiple resources at once (bsc#952775) +- high: scripts: Conservatively verify scripts that modify the CIB (bsc#951954) +- high: xmlutil: Order is significant in resource_set (bsc#955434) +- medium: scripts: Lower copy target to string +- doc: configure load can read from stdin +- medium: script: (filesystem) create stopped (bsc#952670) +- medium: scripts: Check required parameters for optional sub-steps +- high: scripts: Eval CIB text in correct scope (bsc#952600) +- medium: utils: Fix python 2.6 compatibility +- medium: ui_script: Tag legacy wizards as legacy in show (bsc#952226) +- medium: scripts: No optional steps in legacy wizards (bsc#952226) +- high: utils: Revised time zone handling (bsc#951759) +- high: report: Fix syslog parser regexps (bsc#951759) +- low: constants: Tweaked graph colours +- high: scripts: Fix DRBD script resource reference (bsc#951028) +- low: constants: Tweaked graph colors +- medium: report: Make transitions without end stretch to 2525 +- high: utils: Handle time zones in parse_time (bsc#949511) +- medium: hb_report: Remove reference to function name in event patterns (bsc#942906) +- medium: ui_script: Optionally print common params +- medium: cibconfig: Fix sanity check for attribute-based fencing topology (#110) +- high: cibconfig: Fix bug with node/resource collision +- high: scripts: Determine output format of script correctly (bsc#949980) +- doc: add explanatory comments to fencing_topology +- doc: add missing backslash in fencing_topology example +- doc: add missing <> to fencing_topology syntax +- low: don't use deprecated crm_attribute -U option +- doc: resource-discovery for location constraints +- high: utils: Fix cluster_copy_file error when nodes provided +- low: xmlutil: More informative message when updating resource references after rename +- doc: fix some command syntax grammar in the man page +- high: cibconfig: Delete constraints before resources +- high: cibconfig: Fix bug in is_edit_valid (bsc#948547) +- medium: hb_report: Don't cat binary logs +- high: cibconfig: Allow node/rsc id collision in _set_update (bsc#948547) +- low: report: Silence tar warning on early stream close +- high: cibconfig: Allow nodes and resources with the same ID (bsc#948547) +- high: log_patterns_118: Update the correct set of log patterns (bsc#942906) +- low: ui_resource: Silence spurious migration non-warning from pacemaker +- medium: config: Always fall back to /usr/bin:/usr/sbin:/bin:/sbin for programs (bsc#947818) +- medium: report: Enable opening .xz-compressed report tarballs +- medium: cibconfig: Only warn for grouped children in colocations (bsc#927423) +- medium: cibconfig: Allow order constraints on group children (bsc#927423) +- medium: cibconfig: Warn if configuring constraint on child resource (bsc#927423) (#101) +- high: ui_node: Show remote nodes in crm node list (bsc#877962) +- high: config: Remove config.core.supported_schemas (bsc#946893) +- medium: report: Mark transitions with errors with a star in info output (bsc#943470) +- low: report: Remove first transition tag regex +- medium: report: Add transition tags command (bsc#943470) +- low: ui_history: Better error handling and documentation for the detail command +- low: ui_history: Swap from and to times if to < from +- medium: cibconfig: XML parser support for node-attr fencing topology +- medium: parse: Updated syntax for fencing-topology target attribute +- medium: parse: Add support for node attribute as fencing topology target +- high: scripts: Add enum type to script values +- low: scripts: [MailTo] install mailx package +- low: scripts: Fix typo in email type verifier +- high: script: Fix subscript agent reference bug +- low: constants: Add meta attributes for remote nodes +- medium: scripts: Fix typo in lvm script +- high: scripts: Generate actions for includes if none are defined +- low: scripts: [virtual-ip] make lvs_support an advanced parameter +- medium: crm_pssh: Timeout is an int (bsc#943820) +- medium: scripts: Add MailTo script +- low: scripts: Improved script parameter validation +- high: parse: Fix crash when referencing score types by name (bsc#940194) +- doc: Clarify documentation for colocations using node-attribute +- high: ui_script: Print cached errors in json run +- medium: scripts: Use --no option over --force unless force: true is set in the script +- medium: options: Add --no option +- high: scripts: Default to passing --force to crm after all +- high: scripts: Add force parameter to cib and crm actions, and don't pass --force by default +- low: scripts: Make virtual IP optional [nfsserver] +- medium: scripts: Ensure that the Filesystem resource exists [nfsserver] (bsc#898658) +- medium: report: Reintroduce empty transition pruning (bsc#943291) +- low: hb_report: Collect libqb version (bsc#943327) +- medium: log_patterns: Remove reference to function name in log patterns (bsc#942906) +- low: hb_report: Increase time to wait for the logmark +- high: hb_report: Always prefer syslog if available (bsc#942906) +- high: report: Update transition edge regexes (bsc#942906) +- medium: scripts: Switch install default to false +- low: scripts: Catch attempt to pass dict as parameter value +- high: report: Output format from pacemaker has changed (bsc#941681) +- high: hb_report: Prefer pacemaker.log if it exists (bsc#941681) +- medium: report: Add pacemaker.log to find_node_log list (bsc#941734) +- high: hb_report: Correct path to hb_report after move to subdirectory (bsc#936026) +- low: main: Bash completion didn't handle sudo correctly +- medium: config: Add report_tool_options (bsc#917638) +- high: parse: Add attributes to terminator set (bsc#940920) +- Medium: cibconfig: skip sanity check for properties other than cib-bootstrap-options +- medium: ui_script: Fix bug in verify json encoding +- low: ui_script: Check JSON command syntax +- medium: ui_script: Add name to action output (fate#318211) +- low: scripts: Preserve formatting of longdescs +- low: scripts: Clearer shortdesc for filesystem +- low: scripts: Fix formatting for SAP scripts +- low: scripts: add missing type annotations to libvirt script +- low: scripts: make overridden parameters non-advanced by default +- low: scripts: Tweak description for libvirt +- low: scripts: Strip shortdesc for scripts and params +- low: scripts: Title and category for exportfs +- high: ui_script: drop end sentinel from API output (fate#318211) +- low: scripts: Fix possible reference error in agent include +- low: scripts: Clearer error message +- low: Remove build revision from version +- low: Add HAProxy script to data manifest +- medium: constants: Add 'provides' meta attribute (bsc#936587) +- medium: scripts: Add HAProxy script +- high: hb_report: find utility scripts after move (bsc#936026) +- high: ui_report: Move hb_report to subdirectory (bsc#936026) +- high: Makefile: Don't unstall hb_report using data-manifest (bsc#936026) +- medium: report: Fall back to cluster-glue hb_report if necessary (bsc#936026) +- medium: scripts: stop inserting comments as values +- high: scripts: subscript values not required if subscript has no parameters / all defaults (fate#318211) +- medium: scripts: Fix name override for subscripts (fate#318211) +- low: scripts: Clean up generated CIB (fate#318211) + +As usual, a huge thank you to all contributors and users of crmsh! + +Cheers, +Kristoffer diff --git a/doc/website-v1/news/2016-04-28-release-2_2_1.adoc b/doc/website-v1/news/2016-04-28-release-2_2_1.adoc new file mode 100644 index 0000000..e162b2f --- /dev/null +++ b/doc/website-v1/news/2016-04-28-release-2_2_1.adoc @@ -0,0 +1,73 @@ +crmsh 2.2.1 and 2.1.6 are released +================================== +:Author: Kristoffer Gronlund +:Email: kgronlund@suse.com +:Date: 2016-04-28 01:00 + +Today I'm announcing two new releases of crmsh: 2.2.1 and 2.1.6. + +Both are stable releases, and I would recommend all users to +upgrade to crmsh 2.2.1 if they can. + +== crmsh 2.2.1 + +Here is a brief list of the most significant changes in this release: + +* Rewritten history explorer internally to improve performance significantly +* Allow configuring reload operation +* Fix fencing for remote nodes +* Recycle corosync node IDs when possible +* Several bug fixes in crm report +* Warn if generated report is empty +* Improved SBD cluster script +* Add push method for configure load +* Fixed cluster init + +The source code can be downloaded from Github: + +* https://github.com/ClusterLabs/crmsh/releases/tag/2.2.1 + +Packages for several popular Linux distributions can be downloaded +from the Stable repository at the OBS: + +* http://download.opensuse.org/repositories/network:/ha-clustering:/Stable/ + +Archives of the tagged release: + +* https://github.com/ClusterLabs/crmsh/archive/2.2.1.tar.gz +* https://github.com/ClusterLabs/crmsh/archive/2.2.1.zip + +For the full list of changes since version 2.2.0, see the ChangeLog, +available at: + +* https://github.com/ClusterLabs/crmsh/blob/2.2.1/ChangeLog + +== crmsh 2.1.6 + +This is a bug fix release. +Most fixes are minor or related to time handling in reports. + +For a complete list of changes since the previous version, please +refer to the changelog: + +* https://github.com/ClusterLabs/crmsh/blob/2.1.6/ChangeLog + +Currently I'm not building binary releases for 2.1.6 as the stable +series (at OBS) is at 2.2. This release is intended for users of +the 2.1 series who have yet to migrate to 2.2. + +Archives of the tagged release: + +https://github.com/ClusterLabs/crmsh/archive/2.1.6.tar.gz +https://github.com/ClusterLabs/crmsh/archive/2.1.6.zip + +For the full list of changes since version 2.1.5, see the ChangeLog, +available at: + +* https://github.com/ClusterLabs/crmsh/blob/2.1.5/ChangeLog + + +As usual, a huge thank you to all contributors and users of crmsh! + +Cheers, +Kristoffer diff --git a/doc/website-v1/news/2016-08-12-release-2_3_0.adoc b/doc/website-v1/news/2016-08-12-release-2_3_0.adoc new file mode 100644 index 0000000..bb3efa9 --- /dev/null +++ b/doc/website-v1/news/2016-08-12-release-2_3_0.adoc @@ -0,0 +1,76 @@ +Releasing crmsh version 2.3.0 +============================= +:Author: Kristoffer Gronlund +:Email: kgronlund@suse.com +:Date: 2016-08-12 10:30 + +Hello everyone! + +I am proud to present crmsh version 2.3.0, the latest stable +release. I would recommend all users to upgrade to 2.3.0 if they +can. + +For this release, I would like to begin by highlighting the new +contributors to crmsh since 2.2.0 was released in January: + +* Marc A. Smith added the new subcommand "configure load push", which + removes any configuration lines that aren't included in the cib + provided when pushing. + +* Andrei Maruha added an optional name parameter to the "corosync + add-node" command, and made the add-node command recycle old node + IDs if possible. + +* Kai Kang fixed a build system bug when removing generated docs, + causing issues with parallel make. + +* Daniel Hoffend contributed various fixes improving support for + building crmsh for Debian and Ubuntu. + +* Pedro Salgado fixed a bug in the graph rendering code in crmsh, + added a tox configuration file to make testing with multiple + versions of Python easy, and updated the Travis CI configuration to + use tox. + +* Nate Clark fixed a bug in the parser for fencing hierarchies. + +I would also like to thank all the other contributors, testers and +users who have helped in making this release as stable and reliable as +possible. + +Some of the other major features in 2.3.0 include: + +* Support for the new event-based alerts feature in Pacemaker 1.1.15 + +* Greatly improved timezone handling in crm report and the history + explorer + +* Improvements to the cluster scripts / wizards, as well as new + wizards for LVM on DRBD, and NFS on LVM and DRBD and VMware/vCenter + +* Better support for fencing remote nodes + +The source code can be downloaded from Github: + +* https://github.com/ClusterLabs/crmsh/releases/tag/2.3.0 + +Packages for several popular Linux distributions can be downloaded +from the Stable repository at the OBS: + +* http://download.opensuse.org/repositories/network:/ha-clustering:/Stable/ + +Archives of the tagged release: + +* https://github.com/ClusterLabs/crmsh/archive/2.3.0.tar.gz +* https://github.com/ClusterLabs/crmsh/archive/2.3.0.zip + +For the full list of changes since version 2.3.0, see the ChangeLog, +available at: + +* https://github.com/ClusterLabs/crmsh/blob/2.3.0/ChangeLog + + +As usual, a huge thank you to all contributors and users of crmsh! + +Cheers, +Kristoffer diff --git a/doc/website-v1/news/2016-09-01-release-2_1_7.adoc b/doc/website-v1/news/2016-09-01-release-2_1_7.adoc new file mode 100644 index 0000000..d1e477e --- /dev/null +++ b/doc/website-v1/news/2016-09-01-release-2_1_7.adoc @@ -0,0 +1,46 @@ +Announcing crmsh stable release 2.1.7 +===================================== +:Author: Kristoffer Gronlund +:Email: kgronlund@suse.com +:Date: 2016-09-01 09:00 + +Today I are proud to announce the release of `crmsh` version 2.1.7! +The major new thing in this release is a backports of the events-based +alerts support from the 2.3 branch. + +Big thanks to Hideo Yamauchi for his patience and testing of the +alerts backport. + +This time, the list of changes is small enough that I can add it right +here: + +- high: parse: Backport of event-driven alerts parser (#150) +- high: hb_report: Don't collect logs from journalctl if -M is set (bsc#990025) +- high: hb_report: Skip lines without timestamps in log correctly (bsc#989810) +- high: constants: Add maintenance to set of known attributes (bsc#981659) +- high: utils: Avoid deadlock if DC changes during idle wait (bsc#978480) +- medium: scripts: no-quorum-policy=ignore is deprecated (bsc#981056) +- low: cibconfig: Don't mix up CLI name with XML tag + +You can also get the list of changes from the changelog: + +* https://github.com/ClusterLabs/crmsh/blob/2.1.7/ChangeLog + +Right now, I don't have a set of pre-built rpm packages for Linux +distributions ready, but I am going to make this available soon. This +is in particular for centOS 6.x which still relies on Python 2.6 +support which makes running the later releases there more +difficult. These packages will most likely appear as a subrepository +here (more details coming soon): + +* http://download.opensuse.org/repositories/network:/ha-clustering:/Stable/ + +Archives of the tagged release: + +* https://github.com/ClusterLabs/crmsh/archive/2.1.7.tar.gz +* https://github.com/ClusterLabs/crmsh/archive/2.1.7.zip + + +Thank you, + +Kristoffer diff --git a/doc/website-v1/news/2016-09-02-release-2_3_1.adoc b/doc/website-v1/news/2016-09-02-release-2_3_1.adoc new file mode 100644 index 0000000..2a90bef --- /dev/null +++ b/doc/website-v1/news/2016-09-02-release-2_3_1.adoc @@ -0,0 +1,33 @@ +Releasing crmsh version 2.3.1 +============================= +:Author: Kristoffer Gronlund +:Email: kgronlund@suse.com +:Date: 2016-09-02 10:00 + +Hello everyone! + +Today I am releasing crmsh version 2.3.1. The only change this time is +to lower the Python version requirement from 2.7 to 2.6. This is so +that crmsh remains compatible with centOS 6, where there is no +standardized Python 2.7 version available. For users of other +distributions where Python 2.7 is available, there are no other +changes in this release and no need to upgrade. + +The source code can be downloaded from Github: + +* https://github.com/ClusterLabs/crmsh/releases/tag/2.3.1 + +Packages for several popular Linux distributions can be downloaded +from the Stable repository at the OBS: + +* http://download.opensuse.org/repositories/network:/ha-clustering:/Stable/ + +Archives of the tagged release: + +* https://github.com/ClusterLabs/crmsh/archive/2.3.1.tar.gz +* https://github.com/ClusterLabs/crmsh/archive/2.3.1.zip + +As usual, a huge thank you to all contributors and users of crmsh! + +Cheers, +Kristoffer diff --git a/doc/website-v1/news/2016-09-05-release-2_2_2.adoc b/doc/website-v1/news/2016-09-05-release-2_2_2.adoc new file mode 100644 index 0000000..9816b3c --- /dev/null +++ b/doc/website-v1/news/2016-09-05-release-2_2_2.adoc @@ -0,0 +1,36 @@ +Releasing crmsh version 2.2.2 +============================= +:Author: Kristoffer Gronlund +:Email: kgronlund@suse.com +:Date: 2016-09-05 19:00 + +Hello everyone! + +Today I am releasing crmsh version 2.2.2. The biggest change in this +release is the backport of the support for event-based alerts from the +2.3 branch. The full list of changes follows below: + +- high: parse: Backport of event-driven alerts parser (#150) +- high: hb_report: Don't collect logs from journalctl if -M is set (bsc#990025) +- high: hb_report: Skip lines without timestamps in log correctly (bsc#989810) +- high: constants: Add maintenance to set of known attributes (bsc#981659) +- high: utils: Avoid deadlock if DC changes during idle wait (bsc#978480) +- medium: scripts: no-quorum-policy=ignore is deprecated (bsc#981056) +- medium: tmpfiles: Create temporary directory if non-existing (bsc#981583) +- medium: xmlutil: reduce unknown attribute to warning (bsc#981659) +- medium: ui_resource: Add force argument to resource cleanup (bsc#979420) +- parse: Use original _TARGET_RE + +The source code can be downloaded from Github: + +* https://github.com/ClusterLabs/crmsh/releases/tag/2.2.2 + +Archives of the tagged release: + +* https://github.com/ClusterLabs/crmsh/archive/2.2.2.tar.gz +* https://github.com/ClusterLabs/crmsh/archive/2.2.2.zip + +As usual, a huge thank you to all contributors and users of crmsh! + +Cheers, +Kristoffer diff --git a/doc/website-v1/news/2017-01-31-release-3_0_0.adoc b/doc/website-v1/news/2017-01-31-release-3_0_0.adoc new file mode 100644 index 0000000..31eacea --- /dev/null +++ b/doc/website-v1/news/2017-01-31-release-3_0_0.adoc @@ -0,0 +1,48 @@ +Releasing crmsh version 3.0.0 +============================= +:Author: Kristoffer Gronlund +:Email: kgronlund@suse.com +:Date: 2017-01-31 10:00 + +Hello everyone! + +I'm happy to announce the release of crmsh version 3.0.0 today. The +main reason for the major version bump is because I have merged the +sleha-bootstrap project with crmsh, replacing the cluster +init/add/remove commands with the corresponding commands from +sleha-bootstrap. + +At the moment, these commands are highly specific to SLE and openSUSE, +unfortunately. I am working on making them as distribution agnostic as +possible, but would appreciate help from users of other distributions +in making them work as well on those platforms as they do on +SLE/openSUSE. + +Briefly, the "cluster init" command configures a complete cluster from +scratch, including optional configuration of fencing via SBD, shared +storage using OCFS2, setting up the Hawk web interface etc. + +There are some other changes in this release as well, see the +ChangeLog for the complete list of changes: + +* https://github.com/ClusterLabs/crmsh/blob/3.0.0/ChangeLog + +The source code can be downloaded from Github: + +* https://github.com/ClusterLabs/crmsh/releases/tag/3.0.0 + +This version of crmsh will be available in openSUSE Tumbleweed as soon +as possible, and packages for several popular Linux distributions are +available from the Stable repository at the OBS: + +* http://download.opensuse.org/repositories/network:/ha-clustering:/Stable/ + +Archives of the tagged release: + +* https://github.com/ClusterLabs/crmsh/archive/3.0.0.tar.gz +* https://github.com/ClusterLabs/crmsh/archive/3.0.0.zip + +As usual, a huge thank you to all contributors and users of crmsh! + +Cheers, +Kristoffer diff --git a/doc/website-v1/news/2021-06-17-release-4_3_1.adoc b/doc/website-v1/news/2021-06-17-release-4_3_1.adoc new file mode 100644 index 0000000..1b6a4a6 --- /dev/null +++ b/doc/website-v1/news/2021-06-17-release-4_3_1.adoc @@ -0,0 +1,55 @@ +Releasing crmsh version 4.3.1 +============================= +:Author: Xin Liang +:Email: XLiang@suse.com +:Date: 2021-06-17 11:00 + +Hello everyone! + +I'm happy to announce the release of crmsh version 4.3.1 + +Major changes since 4.3.0 + +Features: + +* Add "crm cluster crash_test" for cluster failure simulation (#825) + +* Add ocfs2.OCFS2Manager to manage ocfs2 stage process with cluster lvm2 (#798) + +* Support setup SBD via bootstrap "sbd" stage on an existing cluster (#744) + +* Enable configuring qdevice on interactive mode (#765) + +Fixes: + +* Adjust sbd watchdog timeout when using diskless SBD with qdevice (#818) + +* Not allow property setting with an empty value (#817) + +* Keep consistent for "help <sub-command>" and "<sub-command> -h" for those using argparse (#644) + +* Sync corosync.conf before finished joining (#775) + +* Adjust qdevice configure/remove process to avoid race condition due to quorum lost (#741) + +* Walk through hb_report process under hacluster (#742) + +There are some other changes in this release as well, see the +ChangeLog for the complete list of changes: + +* https://github.com/ClusterLabs/crmsh/blob/master/ChangeLog + +The source code can be downloaded from Github: + +* https://github.com/ClusterLabs/crmsh/releases/tag/4.3.1 + +Development packages for openSUSE Tumbleweed +are available from the Open Build System, here: + +* https://build.opensuse.org/package/show/network:ha-clustering:Factory/crmsh + +As usual, a huge thank you to all contributors and users of crmsh! + + +Regards, +xin diff --git a/doc/website-v1/postprocess.py b/doc/website-v1/postprocess.py new file mode 100644 index 0000000..859abaa --- /dev/null +++ b/doc/website-v1/postprocess.py @@ -0,0 +1,141 @@ +#!/usr/bin/env python +# create a table of contents for pages that need it + +import sys +import re +import argparse + +TOC_PAGES = ['man/index.html', + 'man-4.3/index.html', + 'man-3/index.html', + 'man-2.0/index.html', + 'man-1.2/index.html'] +V2_PAGES = ['index.html'] +INSERT_AFTER = '<!--TOC-->' + +def read_toc_data(infile, debug): + topics_data = [] + commands_data = [] + f = open(infile) + for line in f: + if line.startswith('[['): + line = line[2:-3] # strip [[ and ]]\n + info, short_help = line.split(',', 1) + short_help = short_help.strip() + info_split = info.split('_') + if info_split[0] == 'topics': + if len(info_split) == 2: + topics_data.append((1, short_help, info)) + elif len(info_split) >= 3: + topics_data.append((2, short_help, info)) + elif info_split[0] == 'cmdhelp': + if len(info_split) == 2: + commands_data.append((2, info_split[1], info)) + elif len(info_split) >= 3: + commands_data.append((3, '_'.join(info_split[2:]), info)) + toc = '' + if len(topics_data) > 0 or len(commands_data) > 0: + toc = '<div id="toc">\n' + for depth, text, link in topics_data: + toc += '<div class="toclevel%s"><a href="#%s">%s</a></div>\n' % ( + depth, link, text) + for depth, text, link in commands_data: + toc += '<div class="toclevel%s"><a href="#%s">%s</a></div>\n' % ( + depth, link, text) + toc += '</div>\n' + return toc + +def generate_toc(infile, outfile, debug): + + if debug: + print "Infile:", infile + toc = read_toc_data(infile, debug) + ''' + toc_data = [] + section = re.compile(r"<h(?P<depth>[0-9])( id=\"(?P<id>[^\"]+)\")?>(?P<text>.*)</h[0-9]>") + for line in f: + m = section.match(line) + if m: + if debug: + print "toc_data: %s" % str(((m.group('depth'), m.group('text'), m.group('id')))) + toc_data.append((m.group('depth'), m.group('text'), m.group('id'))) + + toc = '' + if len(toc_data) > 0: + toc = '<div id="toc">\n' + for depth, text, link in toc_data: + if depth >= 2 and link is not None: + toc += '<div class="toclevel%s"><a href="#%s">%s</a></div>\n' % ( + int(depth) - 1, link, text) + toc += '</div>\n' +''' + + # Write TOC to outfile + if outfile: + if debug: + print "Writing TOC:" + print "----" + print toc + print "----" + print "Outfile:", outfile + fil = open(outfile) + f = fil.readlines() + fil.close() + f2 = open(outfile, 'w') + for line in f: + f2.write(line) + if toc and line.startswith(INSERT_AFTER): + f2.write(toc) + f2.close() + +def generate_v2(page, debug): + f = open(page).readlines() + toc_data = [] + section = re.compile(r"<h(?P<depth>[0-9])( id=\"(?P<id>[^\"]+)\")?>(?P<text>.*)</h[0-9]>") + for line in f: + m = section.match(line) + if m: + if debug: + print "toc_data: %s" % str(((m.group('depth'), m.group('text'), m.group('id')))) + toc_data.append((m.group('depth'), m.group('text'), m.group('id'))) + + toc = '' + if len(toc_data) > 0: + toc = '<div id="toc">\n' + for depth, text, link in toc_data: + if depth >= 2 and link is not None: + toc += '<div class="toclevel%s"><a href="#%s">%s</a></div>\n' % ( + int(depth) - 1, link, text) + toc += '</div>\n' + f2 = open(page, 'w') + for line in f: + f2.write(line) + if toc and line.startswith(INSERT_AFTER): + f2.write(toc) + f2.close() + +def main(): + parser = argparse.ArgumentParser(description="Generate table of contents") + parser.add_argument('-d', '--debug', dest='debug', action='store_true', + help="Enable debug output") + parser.add_argument('-o', '--output', metavar='output', type=str, + help="File to inject TOC into") + parser.add_argument('input', metavar='input', type=str, + help="File to read TOC metadata from") + args = parser.parse_args() + debug = args.debug + outfile = args.output + infile = args.input + print "+ %s -> %s" % (infile, outfile) + gen = False + for tocpage in TOC_PAGES: + if not gen and outfile.endswith(tocpage): + generate_toc(infile, outfile, debug) + gen = True + for tocpage in V2_PAGES: + if not gen and outfile.endswith(tocpage): + generate_v2(outfile, debug) + gen = True + +if __name__ == "__main__": + main() diff --git a/doc/website-v1/rsctest-guide.adoc b/doc/website-v1/rsctest-guide.adoc new file mode 100644 index 0000000..2dcd865 --- /dev/null +++ b/doc/website-v1/rsctest-guide.adoc @@ -0,0 +1,238 @@ += Resource testing = + +Never created a pacemaker cluster configuration before? Please +read on. + +Ever created a pacemaker configuration without errors? All +resources worked from the get go on all your nodes? Really? We +want a photo of you! + +Seriously, it is so error prone to get a cluster resource +definition right that I think I ever only managed to do it with +`Dummy` resources. There are many intricate details that have to be +just right, and all of them are stuffed in a single place as simple +name-value attributes. Then there are multiple nodes, each node +containing a complex system environment inevitably always in flux and +changing (entropy anybody?). + +Now, once you defined your set of resources and are about to +_commit_ the configuration (at that point it usually takes a +deep breath to do so), be ready to meet an avalanche of error +messages, not all of which are easy to understand or follow. Not +to mention that you need to read the logs too. Even though we do +have a link:history-tutorial.html[tool] to help with digging through +the logs, it is going to be an interesting experience and not quite +recommended if you're just starting with pacemaker clusters. Even the +experts can save a lot of time and headaches by following the advice +below. + +== Basic usage == + +Enter resource testing. It is a special feature designed to help +users find problems in resource configurations. + +The usage is very simple: + +---- +crm(live)configure# rsctest web-server +Probing resources .. +testing on xen-f: apache web-ip +testing on xen-g: apache web-ip +crm(live)configure# +---- + +What actually happened above and what is it good for? From the +output we can infer that the `web-server` resource is actually a +group comprising one apache web server and one IP address. +Indeed: + +---- +crm(live)configure# show web-server +group web-server apache web-ip \ + meta target-role="Stopped" +crm(live)configure# +---- + +The `rsctest` command first established that the resources are +stopped on all nodes in the cluster. Then it tests the resources +in the order defined by the resource group on all nodes. It does +this by manually starting the resources, one by one, then running +a "monitor" for each resource to make sure that the resources are +healthy, and finally stopping the resources in reverse order. + +Since there is no additional output, the test passed. It looks +like we have a properly defined web server group. + +== Reporting problems == + +Now, the above run was not very interesting so let's spoil the +idyll: + +---- +xen-f:~ # mv /etc/apache2/httpd.conf /tmp +---- + +We moved the apache configuration file away on node `xen-f`. The +`apache` resource should fail now: + +---- +crm(live)configure# rsctest web-server +Probing resources .. +testing on xen-f: apache +host xen-f (exit code 5) +xen-f stderr: +2013/10/17_16:51:26 ERROR: Configuration file /etc/apache2/httpd.conf not found! +2013/10/17_16:51:26 ERROR: environment is invalid, resource considered stopped + +testing on xen-g: apache web-ip +crm(live)configure# +---- + +As expected, `apache` failed to start on node `xen-f`. When the +cluster resource manager runs an operation on a resource, all +messages are logged (there is no terminal attached to the +cluster, anyway). All one can see in the resource status is the type +of the exit code. In this case, it is an installation problem. + +For instance, the output could look like this: + +---- +xen-f:~ # crm status +Last updated: Thu Oct 17 19:21:44 2013 +Last change: Thu Oct 17 19:21:28 2013 by root via crm_resource on xen-f +... +Failed actions: + apache_start_0 on xen-f 'not installed' (5): call=2074, status=complete, +last-rc-change='Thu Oct 17 19:21:31 2013', queued=164ms, exec=0ms +---- + +That does not look very informative. With `rsctest` we can +immediately see what the problem is. It saves us prowling the +logs looking for messages of the `apache` resource agent. + +Note that the IP address is not tested, because the resource it +depends on could not be started. + +== What is tested? == + +The start, monitor, and stop operations, in exactly that order, +are tested for every resource specified. Note that normally the +two latter operations should never fail if the resource agent is +well implemented. The RA should under normal circumstances be +able to stop or monitor a started resource. However, this is +_not_ a replacement for resource agent testing. If that is what +you are looking for, see +http://www.linux-ha.org/doc/dev-guides/_testing_resource_agents.html[the +RA testing chapter] of the RA development guide. + +== Protecting resources == + +The `rsctest` command goes to great lengths to prevent starting a +resource on more than one node at the same time. For some stuff +that would actually mean data corruption and we certainly don't +want that to happen. + +---- +xen-f:~ # (echo start web-server; echo show web-server) | crm -w resource +resource web-server is running on: xen-g +xen-f:~ # crm configure rsctest web-server +Probing resources .WARNING: apache:probe: resource running at xen-g +.WARNING: web-ip:probe: resource running at xen-g + +Stop all resources before testing! +xen-f:~ # crm configure rsctest web-server xen-f +Probing resources .WARNING: apache:probe: resource running at xen-g +.WARNING: web-ip:probe: resource running at xen-g + +Stop all resources before testing! +xen-f:~ # +---- + +As you can see, if `rsctest` finds any of the resources running +on any node it refuses to run any tests. + +== Multi-state and clone resources == + +Apart from groups, the `rsctest` can also handle the other two +special kinds of resources. Let's take a look at one `drbd`-based +configuration: + +---- +crm(live)configure# show ms_drbd_nfs drbd0-vg +primitive drbd0-vg ocf:heartbeat:LVM \ + params volgrpname="drbd0-vg" +primitive p_drbd_nfs ocf:linbit:drbd \ + meta target-role="Stopped" \ + params drbd_resource="nfs" \ + op monitor interval="15" role="Master" \ + op monitor interval="30" role="Slave" \ + op start interval="0" timeout="300" \ + op stop interval="0" timeout="120" +ms ms_drbd_nfs p_drbd_nfs \ + meta notify="true" clone-max="2" +crm(live)configure# +---- + +The `nfs` drbd resource contains a volume group `drbd0-vg`. + +---- +crm(live)configure# rsctest ms_drbd_nfs drbd0-vg +Probing resources .. +testing on xen-f: p_drbd_nfs drbd0-vg +testing on xen-g: p_drbd_nfs drbd0-vg +crm(live)configure# +---- + +For the multi-state (master-slave) resources, the involved +resource motions are somewhat more complex: the resource is first +started on both nodes and then promoted on the node where the +next resource is to be tested (in this case the volume group). +Then it gets demoted to slave and promoted on the other +node to master so that the depending resources can be tested on +that node too. + +Note that even though we asked for `ms_drbd_nfs` to be tested, +there is `p_drbd_nfs` in the output which is the primitive +encapsulated in the master-slave resource. You can specify either +one. + +== Stonith resources == + +The stonith resources are also special and need special +treatment. What is tested is just the device status. Actually +fencing nodes was deemed too drastic. Please use `node fence` to +test the fencing device effectiveness. It also does not matter +whether the stonith resource is "running" on any node: being +started is just something that happens virtually in the +`stonithd` process. + +== Summary == + +- use `rsctest` to make sure that the resources can be started + correctly on all nodes + +- `rsctest` protects resources by making sure beforehand that + none of them is currently running on any of the cluster nodes + +- `rsctest` understands groups, master-slave (multi-state), and + clone resources, but nothing else of the configuration + (constraints or any other placement/order cluster configuration + elements) + +- it is up to the user to test resources only on nodes which are + really supposed to run them and in a proper order (if that + order is expressed via constraints) + +- `rsctest` cannot protect resources if they are running on + nodes which are not present in the cluster or from bad RA + implementations (but neither would a cluster resource manager) + +- `rsctest` was designed as a debugging and configuration aid, and is + not intended to provide full Resource Agent test coverage. + +== `crmsh` help and online resources (_sic!_) == + +- link:crm.8.html#topics_Testing[`crm help Testing`] + +- link:crm.8.html#cmdhelp_configure_rsctest[`crm configure help +rsctest`] diff --git a/doc/website-v1/scripts.adoc b/doc/website-v1/scripts.adoc new file mode 100644 index 0000000..7742729 --- /dev/null +++ b/doc/website-v1/scripts.adoc @@ -0,0 +1,660 @@ += Cluster Scripts = +:source-highlighter: pygments + +.Version information +NOTE: This section applies to `crmsh 2.2+` only. + +== Introduction == + +A big part of the configuration and management of a cluster is +collecting information about all cluster nodes and deploying changes +to those nodes. Often, just performing the same procedure on all nodes +will encounter problems, due to subtle differences in the +configuration. + +For example, when configuring a cluster for the first time, the +software needs to be installed and configured on all nodes before the +cluster software can be launched and configured using `crmsh`. This +process is cumbersome and error-prone, and the goal is for scripts to +make this process easier. + +Another important function of scripts is collecting information and +reporting potential issues with the cluster. For example, software +versions may differ between nodes, causing byzantine errors or random +failure. `crmsh` comes packaged with a `health` script which will +detect and warn about many of these types of problems. + +There are many tools for managing a collection of nodes, and scripts +are not intended to replace these tools. Rather, they provide an +integrated way to perform tasks across the cluster that would +otherwise be tedious, repetitive and error-prone. The scripts +functionality in the crm shell is mainly inspired by Ansible, a +light-weight and efficient configuration management tool. + +Scripts are implemented using the python `parallax` package which +provides a thin wrapper on top of SSH. This allows the scripts to +function through the usual SSH channels used for system maintenance, +requiring no additional software to be installed or maintained. + +For many scripts that only configure cluster resources or only perform +changes on the local machine, the use of SSH is not necessary. These +scripts can be used even if there is no way for `crmsh` to reach the +other nodes other than through the cluster configuration. + +NOTE: The scripts functionality in `crmsh` has been greatly expanded +and improved in `crmsh` 2.2. Many new scripts have been added, and in +addition the scripts are now used as the backend for the wizards +functionality in HAWK, the HA web interface. For more information, see +https://github.com/ClusterLabs/hawk. + +== Usage == + +Scripts are available through the `cluster` sub-level in the crm +shell. Some scripts have custom commands linked to them for +convenience, such as the `init`, `add` and `remove` commands available +in the `cluster` sublevel, for creating new clusters, introducing new +nodes into the cluster and for removing nodes from a running cluster. + +Other scripts can be accessed through the `script` sub-level. + +=== Common Parameters === + +Which parameters a script accepts varies from script to +script. However, there is a set of parameters that are common to all +scripts. These parameters can be passed to any script. + +`nodes`:: + List of nodes to execute the script for +`dry_run`:: + If set, simulate execution only + (default: no) +`action`:: + If set, only execute a single action (index, as returned by verify) +`statefile`:: + When single-stepping, the state is saved in the given file +`user`:: + Run script as the given user +`sudo`:: + If set, crm will prompt for a sudo password and use sudo when appropriate + (default: no) +`port`:: + Port to connect on +`timeout`:: + Execution timeout in seconds + (default: 600) + +=== List available scripts === + +To list the available scripts, use the following command: + +......... +# crm script +list +......... + +The available scripts are listed along with a short +description. Optionally, the arguments +all+ or +names+ can be +used. Without the +all+ flag, some scripts that are used by `crmsh` to +implement certain commands are hidden from view. With the +names+ +flag, only a plain list of script names is printed. + +=== Script description === + +To get more details about a script, run the `show` command. For +example, to get more information about what the `virtual-ip` script does +and what parameters it accepts, use the following command: + +......... +# crm script +show virtual-ip +......... + +`show` will print a longer description of the script, along with a +list of parameters divided into _steps_. Each script is divided into a +series of steps which are performed in order. Some steps may not +accept any parameters, but for those that do, the available parameters +are listed here. + +By default, only a basic subset of the available parameters is printed +in order to make the scripts easier to use. By passing `all` to the +`show` command, the advanced parameters are also shown. In addition, +there is a list of common parameters + +`show` will print a longer explanation for the script, along with +a list of parameters, each parameter having a description, a note +saying if it is an optional or required parameter, and if optional, +what the default value is. + +=== Verifying parameters === + +Since a script potentially performs a series of actions and may fail +for various reasons at any point, it is advisable to review the +actions that a script will perform before actually running it. To do +this, the `verify` command can be used. + +Pass the parameters that you would pass to `run`, and `verify` will +check that the parameter values are OK, as well as print the sequence +of steps that will be performed given the particular parameter values +given. + +The following is an example showing how to verify the creation of a +Virtual IP resource, using the `virtual-ip` script: + +.......... +# crm script +verify virtual-ip id=my-virtual-ip ip=192.168.0.10 +.......... + +`crmsh` will print something similar to the following output: + +........... +1. Configure cluster resources + + primitive my-virtual-ip ocf:heartbeat:IPaddr2 + ip="192.168.0.10" + op start timeout="20" op stop timeout="20" + op monitor interval="10" timeout="20" +........... + +In this particular case, there is only a single step, and that step +configures a primitive resource. Other scripts may configure multiple +resources and constraints, or may perform multiple steps in sequence. + +=== Running a script === + +To run a script, all required parameters and any optional parameters +that should have values other than the default should be provided as +`key=value` pairs on the command line. + +The following example shows how to create a Virtual IP resource using +the `virtual-ip` script: + +........ +# crm script +run virtual-ip id=my-virtual-ip ip=192.168.0.10 +........ + +==== Single-stepping a script ==== + +It is possible to run a script action-by-action, with manual intervention +between actions. First of all, list the actions to perform given a +certain set of parameter values: + +........ +crm script verify health +........ + +To execute a single action, two things need to be provided: + +1. The index of the action to execute (printed by `verify`) +2. a file in which `crmsh` stores the state of execution. + +Note that it is entirely possible to run actions out-of-order, however +this is unlikely to work in practice since actions often rely on the +outcome of previous actions. + +The following command will execute the first action of the `health` +script and store the output in a temporary file named `health.json`: + +........ +crm script run health action=1 statefile='health.json' +........ + +The statefile contains the script parameters and the output of +previous steps, encoded as `json` data. + +To continue executing the next action in sequence, enter the next +action index: + +........ +crm script run health action=2 statefile='health.json' +........ + +Note that the `dry_run` flag that can be used to do partial execution +of scripts is not taken into consideration when single-stepping +through a script. + +== Creating a script == + +This section will describe how to create a new script, where to put +the script to allow `crmsh` to find it, and how to test that the +script works as intended. + +=== How scripts work, in detail === + +NOTE: The implementation of cluster scripts was revised between +`crmsh` 2.0 and `crmsh` 2.2. This section describes the revised +cluster script format. The old format is still accepted by `crmsh`. + +A cluster script consists of four main sections: + +. The name and description of the script. +. Any other scripts or agents included by this script, and any parameter value overrides to those provided by the included script. +. A set of parameters accepted by the script itself, in addition to those accepted by any scripts or agents included in the script. +. A sequence of actions which the script will perform. + +When the script runs, the actions defined in `main.yml` as described +below are executed one at a time. Each action prescribes a +modification that is applied to the cluster. Some actions work by +calling out to scripts on each of the cluster nodes, and others apply +only on the local node from which the script was executed. + +=== Actions === + +Scripts perform actions that are classified into a few basic +types. Each action is performed by calling out to a shell script, +but the arguments and location of that script varies depending on the +type. + +Here are the types of script actions that can be performed: + +cib:: + * Applies a new CIB configuration to the cluster + +install:: + * Ensures that the given list of packages is installed on all + cluster nodes using the system package manager. + +service:: + * Manages system services using the system init tools. The argument + should be a space-separated list of <service>:<state> pairs. + +call:: + * Run a shell command as specified in the action, either on the + local node on or all nodes. + +copy:: + * Installs a file on the cluster nodes. + * Using a configuration template, install a file on the cluster + nodes. + +crm:: + * Runs the given command using the `crm` shell. This can be used to + start and stop resources, for example. + +collect:: + * Runs on all cluster nodes + * Gathers information about the nodes, both general information and + information specific to the script. + +validate:: + * Runs on the local node + * Validate parameter values and node state based on collected + information. Can modify default values and report issues that + would prevent the script from applying successfully. + +apply:: + * Runs on all or any cluster nodes + * Applies changes, returning information about the applied changes + to the local node. + +apply_local:: + * Runs on the local node + * Applies changes to the cluster, where an action taken on a single + node affect the entire cluster. This includes updating the CIB in + Pacemaker, and also reloading the configuration for Corosync. + +report:: + * Runs on the local node + * This is similar to the _apply_local_ action, with the difference + that the output of a Report action is not interpreted as JSON data + to be passed to the next action. Instead, the output is printed to + the screen. + +==== When expressions ==== + +Actions can be made conditional on the value of script parameters using +the +when:+ expression. This expression has two basic forms. + +The first form is in the form of the name of a script parameter. For +example, given a boolean script parameter named +install+, an action +can be made conditional on that parameter being true using the syntax ++when: install+. + +The second form is a more complex expression. All parameters are +interpreted as either a string value or None if no value was provided. +These can be compared to string literals using python-style +comparators. For example, an action can be conditional on the string +parameter +mode+ having the value +"advanced"+ using the following +syntax: +when: mode == "advanced"+. + +=== Basic structure === + +The crm shell looks for scripts in two primary locations: Included +scripts are installed in the system-wide shared folder, usually +`/usr/share/crmsh/scripts/`. Local and custom scripts are loaded from +the user-local XDG_CONFIG folder, usually found at +`~/.local/crm/scripts/`. These locations may differ depending on how +the crm shell was installed and which system is used, but these are +the locations used on most distributions. + +To create a new script, make a new folder in the user-local scripts +folder and give it a unique name. In this example, we will call our +new script `check-uptime`. + +........ +mkdir -p ~/.local/crm/scripts/check-uptime +........ + +In this directory, create a file called `main.yml`. This is a YAML +document which describes the script, which parameters it requires, and +what actions it will perform. + +YAML is a human-readable markup language which is designed to be easy +to read and modify, while at the same time be compatible with JSON. To +learn more, see http:://yaml.org/[yaml.org]. + +Here is an example `main.yml` file which wraps the resource agent +`ocf:heartbeat:IPaddr2`. + +[source,yaml] +---- +# The version must be exactly 2.2, and must always be +# specified in the script. If the version is missing or +# is less than 2.2, the script is assumed to be a legacy +# script (specified in the format used before crmsh 2.2). +version: 2.2 +shortdesc: Virtual IP +category: Basic +include: + - agent: ocf:heartbeat:IPaddr2 + name: virtual-ip + parameters: + - name: id + type: resource + required: true + - name: ip + type: ip_address + required: true + - name: cidr_netmask + type: integer + required: false + - name: broadcast + type: ip_address + required: false + ops: | + op start timeout="20" op stop timeout="20" + op monitor interval="10" timeout="20" +actions: + - include: virtual-ip +---- + +For a bigger example, here is the `apache` agent which includes +multiple optional steps, the optional installation of packages, +defines multiple cluster resources and potentially calls bash commands +on each of the cluster nodes. + +[source,yaml] +---- +# Copyright (C) 2009 Dejan Muhamedagic +# Copyright (C) 2015 Kristoffer Gronlund +# +# License: GNU General Public License (GPL) +version: 2.2 +category: Server +shortdesc: Apache Webserver +longdesc: | + Configure a resource group containing a virtual IP address and + an instance of the Apache web server. + + You can optionally configure a Filesystem resource which will be + mounted before the web server is started. + + You can also optionally configure a database resource which will + be started before the web server but after mounting the optional + filesystem. +include: + - agent: ocf:heartbeat:apache + name: apache + longdesc: | + The Apache configuration file specified here must be available via the + same path on all cluster nodes, and Apache must be configured with + mod_status enabled. If in doubt, try running Apache manually via + its init script first, and ensure http://localhost:80/server-status is + accessible. + ops: | + op start timeout="40" + op stop timeout="60" + op monitor interval="10" timeout="20" + - script: virtual-ip + shortdesc: The IP address configured here will start before the Apache instance. + parameters: + - name: id + value: "{{id}}-vip" + - script: filesystem + shortdesc: Optional filesystem mounted before the web server is started. + required: false + - script: database + shortdesc: Optional database started before the web server is started. + required: false +parameters: + - name: install + type: boolean + shortdesc: Install and configure apache + value: false +actions: + - install: + - apache2 + shortdesc: Install the apache package + when: install + - service: + - apache: disable + shortdesc: Let cluster manage apache + when: install + - call: a2enmod status; true + shortdesc: Enable status module + when: install + - include: filesystem + - include: database + - include: virtual-ip + - include: apache + - cib: | + group g-{{id}} + {{filesystem:id}} + {{database:id}} + {{virtual-ip:id}} + {{id}} +---- + +The language for referring to parameter values in `cib` actions is +described below. + +=== Command arguments === + +The actions that accept a command as argument must not refer to +commands written in python. They can be plain bash scripts or any +other executable script as long as the nodes have the necessary +dependencies installed. However, see below why implementing scripts in +Python is easier. + +Actions report their progress either by returning JSON on standard +output, or by returning a non-zero return value and printing an error +message to standard error. + +Any JSON returned by an action will be available to the following +steps in the script. When the script executes, it does so in a +temporary folder created for that purpose. In that folder is a file +named `script.input`, containing a JSON array with the output produced +by previous steps. + +The first element in the array (the zeroth element, to be precise) is +a dict containing the parameter values. + +The following elements are dicts with the hostname of each node as key +and the output of the action generated by that node as value. + +In most cases, only local actions (`validate` and `apply_local`) will +use the information in previous steps, but scripts are not limited in +what they can do. + +With this knowledge, we can implement `fetch.py` and `report.py`. + +`fetch.py`: + +[source,python] +---- +#!/usr/bin/python3 +import crm_script as crm +try: + uptime = open('/proc/uptime').read().split()[0] + crm.exit_ok(uptime) +except Exception as e: + crm.exit_fail("Couldn't open /proc/uptime: %s" % (e)) +---- + +`report.py`: + +[source,python] +---- +#!/usr/bin/python3 +import crm_script as crm +show_all = crm.is_true(crm.param('show_all')) +uptimes = list(crm.output(1).items()) +max_uptime = '', 0 +for host, uptime in uptimes: + if float(uptime) > max_uptime[1]: + max_uptime = host, float(uptime) +if show_all: + print("Uptimes: %s" % (', '.join("%s: %s" % v for v in uptimes))) +print("Longest uptime is %s seconds on host %s" % (max_uptime[1], max_uptime[0])) +---- + +See below for more details on the helper library `crm_script`. + +Save the scripts as executable files in the same directory as the +`main.yml` file. + +Before running the script, it is possible to verify that the files are +in a valid format and in the right location. Run the following +command: + +........ +crm script verify check-uptime +........ + +If the verification is successful, try executing the script with the +following command: + +........ +crm script run check-uptime +........ + +Example output: + +[source,bash] +---- +# crm script run check-uptime +INFO: Check uptime of nodes +INFO: Nodes: ha-three, ha-one +OK: Fetch uptimes +OK: Report uptime +Longest uptime is 161054.04 seconds on host ha-one +---- + +To see if the `show_all` parameter works as intended, run the +following: + +........ +crm script run check-uptime show_all=yes +........ + +Example output: + +[source,bash] +---- +# crm script run check-uptime show_all=yes +INFO: Check uptime of nodes +INFO: Nodes: ha-three, ha-one +OK: Fetch uptimes +OK: Report uptime +Uptimes: ha-one: 161069.83, ha-three: 159950.38 +Longest uptime is 161069.83 seconds on host ha-one +---- + +=== Remote permissions === + +Some scripts may require super-user access to remote or local +nodes. It is recommended that this is handled through SSH certificates +and agents, to facilitate password-less access to nodes. + +=== Running scripts without a cluster === + +All cluster scripts can optionally take a `nodes` argument, which +determines the nodes that the script will run on. This node list is +not limited to nodes already in the cluster. It is even possible to +execute cluster scripts before a cluster is set up, such as the +`health` and `init` scripts used by the `cluster` sub-level. + +........ +crm script run health nodes=example1,example2 +........ + +The list of nodes can be comma- or space-separated, but if the list +contains spaces, the whole argument will have to be quoted: + +........ +crm script run health nodes="example1 example2" +........ + +=== Running in validate mode === + +It may be desirable to do a dry-run of a script, to see if any +problems are present that would make the script fail before trying to +apply it. To do this, add the argument `dry_run=yes` to the invocation: + +......... +crm script run health dry_run=yes +......... + +The script execution will stop at the first `apply` action. Note that +non-modifying steps that happen after the first `apply` action will +not be performed in a dry run. + +=== Helper library === + +When the script data is copied to each node, a small helper library is +also passed along with the script. This library can be found in +`utils/crm_script.py` in the source repository. This library helps +with producing output in the correct format, parsing the +`script.input` data provided to scripts, and more. + +.`crm_script` API +`host()`:: + Returns hostname of current node +`get_input()`:: + Returns the input data list. The first element in the list + is a dict of the script parameters. The rest are the output + from previous steps. +`parameters()`:: + Returns the script parameters as a dict. +`param(name)`:: + Returns the value of the named script parameter. +`output(step_idx)`:: + Returns the output of the given step, with the first step being step 1. +`exit_ok(data)`:: + Exits the step returning `data` as output. +`exit_fail(msg)`:: + Exits the step returning `msg` as error message. +`is_true(value)`:: + Converts a truth value from string to boolean. +`call(cmd, shell=False)`:: + Perform a system call. Returns `(rc, stdout, stderr)`. + +=== The handles language === + +CIB configurations and commands can refer to the value of parameters +in the text of the action. This is done using a custom language, +similar to handlebars. + +The language accepts the following constructions: + +............ +{{name}} = Inserts the value of the parameter <name> +{{script:name}} = Inserts the value of the parameter <name> from the + included script named <script>. +{{#name}} ... {{/name}} = Inserts the text between the mustasches when + name is truthy. +{{^name}} ... {{/name}} = Inserts the text between the mustasches when + name is falsy. +............ + diff --git a/doc/website-v1/start-guide.adoc b/doc/website-v1/start-guide.adoc new file mode 100644 index 0000000..7ad6a82 --- /dev/null +++ b/doc/website-v1/start-guide.adoc @@ -0,0 +1,208 @@ += Getting Started + +So, you've successfully installed `crmsh` on one or more machines, and +now you want to configure a basic cluster. This guide is intended to +provide step-by-step instructions for configuring Pacemaker +with a single resource capable of failing over between a pair of +nodes, and then builds on that base to cover some more advanced topics +of cluster management. + +**** +Haven't installed yet? Please follow the +link:/installation[installation instructions] +before continuing this guide. Only `crmsh` and +its dependencies need to be installed before +following this guide. +**** + +Before continuing, make sure that this command executes successfully +on all nodes, and returns a version number that is `3.0` or higher: + +........ +crm --version +........ + +**** +In crmsh 3, the cluster init commands were replaced by the SLE HA +bootstrap scripts. These rely on `csync2` for configuration file +management, so make sure that you have the `csync2` command +installed before proceeding. This requirement may be removed in +the future. +**** + +.Example cluster +************************** + +These are the machines used as an example in this guide. Please +replace the references to these names and IP addresses to the values +appropriate for your cluster: + + +[options="header,footer"] +|======================= +|Name |IP +|alice |10.0.0.2 +|bob |10.0.0.3 +|======================= +************************** + + +== The cluster stack + +The composition of the GNU/Linux cluster stack has changed somewhat +over the years. The stack described here is the currently most common +variant, but there are other ways of configuring these tools. + +Simply put, a High Availability cluster is a set of machines (commonly +referred to as *nodes*) with redundant capacity, such that if one or +more of these machines experience failure of any kind, the other nodes +in the cluster can take over the responsibilities previously handled +by the failed node. + +The cluster stack is a set of programs running on all of these nodes, +communicating with each other over the network to monitor each other +and deciding where, when and how resources are stopped, started or +reconfigured. + +The main component of the stack is *Pacemaker*, the software +responsible for managing cluster resources, allocating them to cluster +nodes according to the rules specified in the *CIB*. + +The CIB is an XML document maintained by Pacemaker, which describes +all cluster resources, their configuration and the constraints that +decide where and how they are managed. This document is not edited +directly, and with the help of `crmsh` it is possible to avoid +exposure to the underlying XML at all. + +Beneath Pacemaker in the stack sits *Corosync*, a cluster +communication system. Corosync provides the communication capabilities +and cluster membership functionality used by Pacemaker. Corosync is +configured through the file `/etc/corosync/corosync.conf`. `crmsh` +provides tools for configuring corosync similar to Pacemaker. + +Aside from these two components, the stack also consists of a +collection of *Resource Agents*. These are basically scripts that wrap +software that the cluster needs to manage, providing a unified +interface to configuration, supervision and management of the +software. For example, there are agents that handle virtual IP +resources, web servers, databases and filesystems. + +`crmsh` is a command line tool which interfaces against all of these +components, providing a unified interface for configuration and +management of the whole cluster stack. + +== SSH + +`crmsh` runs as a command line tool on any one of the cluster +nodes. In order for to to control all cluster nodes, it needs to be +able to execute commands remotely. `crmsh` does this by invoking +`ssh`. + +Configure `/etc/hosts` on each of the nodes so that the names of the +other nodes map to the IP addresses of those nodes. For example in a +cluster consisting of `alice` and `bob`, executing `ping bob` when +logged in as root on `alice` should successfully locate `bob` on the +network. Given the IP addresses of `alice` and `bob` above, the +following should be entered into `/etc/hosts` on both nodes: + +........ +10.0.0.2 alice +10.0.0.3 bob +........ + +== Install and configure + +To configure the basic cluster, we use the `cluster init` command +provided by `crmsh`. This command has quite a few options for +setting up the cluster, but we will use a fairly basic configuration. + +........ +crm cluster init --name demo-cluster --nodes "alice bob" +........ + +The initialization tool will now ask a series of questions about the +configuration, and then proceed to configure and start the cluster +on both nodes. + +== Check cluster status + +To see if Pacemaker is running, what nodes are part of the cluster and +what resources are active, use the `status` command: + +......... +crm status +......... + +If this command fails or times out, there is some problem with +Pacemaker or Corosync on the local machine. Perhaps some dependency is +missing, a firewall is blocking cluster communication or some other +unrelated problem has occurred. If this is the case, the `cluster +health` command may be of use. + +== Cluster health check + +To check the health status of the machines in the cluster, use the +following command: + +........ +crm cluster health +........ + +This command will perform multiple diagnostics on all nodes in the +cluster, and return information about low disk space, communication +issues or problems with mismatching software versions between nodes, +for example. + +If no cluster has been configured or there is some fundamental problem +with cluster communications, `crmsh` may be unable to figure out what +nodes are part of the cluster. If this is the case, the list of nodes +can be provided to the health command directly: + +........ +crm cluster health nodes=alice,bob +........ + +== Adding a resource + +To test the cluster and make sure it is working properly, we can +configure a Dummy resource. The Dummy resource agent is a simple +resource that doesn't actually manage any software. It exposes a +single numerical parameter called `state` which can be used to test +the basic functionality of the cluster before introducing the +complexities of actual resources. + +To configure a Dummy resource, run the following command: + +........ +crm configure primitive p0 Dummy +........ + +This creates a new resource, gives it the name `p0` and sets the +agent for the resource to be the `Dummy` agent. + +`crm status` should now show the `p0` resource as started on one +of the cluster nodes: + +........ +# crm status +Last updated: Wed Jul 2 21:49:26 2014 +Last change: Wed Jul 2 21:49:19 2014 +Stack: corosync +Current DC: alice (2) - partition with quorum +Version: 1.1.11-c3f1a7f +2 Nodes configured +1 Resources configured + + +Online: [ alice bob ] + + p0 (ocf::heartbeat:Dummy): Started alice +........ + +The resource can be stopped or started using the `resource start` and +`resource stop` commands: + +........ +crm resource stop p0 +crm resource start p0 +........ |
