:man source: crm :man version: 4.6.0 :man manual: crmsh documentation crm(8) ====== NAME ---- crm - Pacemaker command line interface for configuration and management SYNOPSIS -------- *crm* [OPTIONS] [SUBCOMMAND ARGS...] [[topics_Description,Program description]] DESCRIPTION ----------- The `crm` shell is a command-line based cluster configuration and management tool. Its goal is to assist as much as possible with the configuration and maintenance of Pacemaker-based High Availability clusters. For more information on Pacemaker itself, see http://clusterlabs.org/. `crm` works both as a command-line tool to be called directly from the system shell, and as an interactive shell with extensive tab completion and help. The primary focus of the `crm` shell is to provide a simplified and consistent interface to Pacemaker, but it also provides tools for managing the creation and configuration of High Availability clusters from scratch. To learn more about this aspect of `crm`, see the `cluster` section below. The `crm` shell can be used to manage every aspect of configuring and maintaining a cluster. It provides a simplified line-based syntax on top of the XML configuration format used by Pacemaker, commands for starting and stopping resources, tools for exploring the history of a cluster including log scraping and a set of cluster scripts useful for automating the setup and installation of services on the cluster nodes. The `crm` shell is line oriented: every command must start and finish on the same line. It is possible to use a continuation character (+\+) to write one command in two or more lines. The continuation character is commonly used when displaying configurations. [[topics_CommandLine,Command line options]] OPTIONS ------- *-f, --file*='FILE':: Load commands from the given file. If a dash +-+ is used in place of a file name, `crm` will read commands from the shell standard input (`stdin`). *-c, --cib*='CIB':: Start the session using the given shadow CIB file. Equivalent to +cib use +. *-D, --display=*'OUTPUT_TYPE':: Choose one of the output options: +plain+, +color-always+, +color+, or +uppercase+. The default is +color+ if the terminal emulation supports colors. Otherwise, +plain+ is used. *-F, --force*:: Make `crm` proceed with applying changes where it would normally ask the user to confirm before proceeding. This option is mainly useful in scripts, and should be used with care. *-w, --wait*:: Make `crm` wait for the cluster transition to finish (for the changes to take effect) after each processed line. *-H, --history*='DIR|FILE|SESSION':: A directory or file containing a cluster report to load into the `history` commands, or the name of a previously saved history session. *-h, --help*:: Print help page. *--version*:: Print crmsh version and build information (Mercurial Hg changeset hash). *-d, --debug*:: Print verbose debugging information. *-R, --regression-tests*:: Enables extra verbose trace logging used by the regression tests. Logs all external calls made by crmsh. *--scriptdir*='DIR':: Extra directory where crm looks for cluster scripts, or a list of directories separated by semi-colons (e.g. +/dir1;/dir2;etc.+). *-o, --opt*='OPTION=VALUE':: Set crmsh option temporarily. If the options are saved using +options save+ then the value passed here will also be saved. Multiple options can be set by using +-o+ multiple times. [[topics_Introduction,Introduction]] == Introduction This section of the user guide covers general topics about the user interface and describes some of the features of `crmsh` in detail. [[topics_Introduction_Interface,User interface]] === User interface The main purpose of `crmsh` is to provide a simple yet powerful interface to the cluster stack. There are two main modes of operation with the user interface of `crmsh`: * Command line (single-shot) use - Use `crm` as a regular UNIX command from your usual shell. `crm` has full bash completion built in, so using it in this manner should be as comfortable and familiar as using any other command-line tool. * Interactive mode - By calling `crm` without arguments, or by calling it with only a sublevel as argument, `crm` enters the interactive mode. In this mode, it acts as its own command shell, which remembers which sublevel you are currently in and allows for rapid and convenient execution of multiple commands within the same sublevel. This mode also has full tab completion, as well as built-in interactive help and syntax highlighting. Here are a few examples of using `crm` both as a command-line tool and as an interactive shell: .Command line (one-shot) use: ........ # crm resource stop www_app ........ .Interactive use: ........ # crm crm(live)# resource crm(live)resource# unmanage tetris_1 crm(live)resource# up crm(live)# node standby node4 ........ .Cluster configuration: ........ # crm configure<+, or, if at the top level of the shell, simply typing `help` will provide an overview of all available levels and commands. The +(live)+ string in the `crm` prompt signifies that the current CIB in use is the cluster live configuration. It is also possible to work with so-called <>. These are separate, inactive configurations stored in files, that can be applied and thereby replace the live configuration at any time. [[topics_Introduction_Completion,Tab completion]] === Tab completion The `crm` makes extensive use of tab completion. The completion is both static (i.e. for `crm` commands) and dynamic. The latter takes into account the current status of the cluster or information from installed resource agents. Sometimes, completion may also be used to get short help on resource parameters. Here are a few examples: ............... crm(live)resource# ban demote maintenance param scores trace cd failcount manage promote secret unmanage cleanup help meta quit start untrace clear locate move refresh status up constraints ls operations restart stop utilization crm(live)configure# primitive fence-1 lsb: ocf: service: stonith: systemd: crm(live)configure# primitive fence-1 stonith: apcmaster external/ippower9258 fence_legacy apcmastersnmp external/kdumpcheck ibmhmc apcsmart external/libvirt ipmilan crm(live)configure# primitive fence-1 stonith:ipmilan params auth= hostname= ipaddr= login= password= port= priv= crm(live)configure# primitive fence-1 stonith:ipmilan params auth= auth* (string) The authorization type of the IPMI session ("none", "straight", "md2", or "md5") ............... `crmsh` also comes with bash completion usable directly from the system shell. This should be installed automatically with the command itself. [[topics_Introduction_Shorthand,Shorthand syntax]] === Shorthand syntax When using the `crm` shell to manage clusters, you will end up typing a lot of commands many times over. Clear command names like +configure+ help in understanding and learning to use the cluster shell, but is easy to misspell and is tedious to type repeatedly. The interactive mode and tab completion both help with this, but the `crm` shell also has the ability to understand a variety of shorthand aliases for all of the commands. For example, instead of typing `crm status`, you can type `crm st` or `crm stat`. Instead of `crm configure` you can type `crm cfg` or even `crm cf`. `crm resource` can be shorted as `crm rsc`, and so on. The exact list of accepted aliases is too long to print in full, but experimentation and typos should help in discovering more of them. [[topics_Features,Features]] == Features The feature set of crmsh covers a wide range of functionality, and understanding how and when to use the various features of the shell can be difficult. This section of the guide describes some of the features and use cases of `crmsh` in more depth. The intention is to provide a deeper understanding of these features, but also to serve as a guide to using them. [[topics_Features_Shadows,Shadow CIB usage]] === Shadow CIB usage A Shadow CIB is a normal cluster configuration stored in a file. They may be manipulated in much the same way as the _live_ CIB, with the key difference that changes to a shadow CIB have no effect on the actual cluster resources. An administrator may choose to apply any of them to the cluster, thus replacing the running configuration with the one found in the shadow CIB. The `crm` prompt always contains the name of the configuration which is currently in use, or the string _live_ if using the live cluster configuration. When editing the configuration in the `configure` level, no changes are actually applied until the `commit` command is executed. It is possible to start editing a configuration as usual, but instead of committing the changes to the active CIB, save them to a shadow CIB. The following example `configure` session demonstrates how this can be done: ............... crm(live)configure# cib new test-2 INFO: test-2 shadow CIB created crm(test-2)configure# commit ............... [[topics_Features_Checks,Configuration semantic checks]] === Configuration semantic checks Resource definitions may be checked against the meta-data provided with the resource agents. These checks are currently carried out: - are required parameters set - existence of defined parameters - timeout values for operations The parameter checks are obvious and need no further explanation. Failures in these checks are treated as configuration errors. The timeouts for operations should be at least as long as those recommended in the meta-data. Too short timeout values are a common mistake in cluster configurations and, even worse, they often slip through if cluster testing was not thorough. Though operation timeouts issues are treated as warnings, make sure that the timeouts are usable in your environment. Note also that the values given are just _advisory minimum_---your resources may require longer timeouts. User may tune the frequency of checks and the treatment of errors by the <> and <> preferences. Note that if the +check-frequency+ is set to +always+ and the +check-mode+ to +strict+, errors are not tolerated and such configuration cannot be saved. [[topics_Features_Templates,Configuration templates]] === Configuration templates .Deprecation note **************************** Configuration templates have been deprecated in favor of the more capable `cluster scripts`. To learn how to use cluster scripts, see the dedicated documentation on the `crmsh` website at http://crmsh.github.io/, or in the <>. **************************** Configuration templates are ready made configurations created by cluster experts. They are designed in such a way so that users may generate valid cluster configurations with minimum effort. If you are new to Pacemaker, templates may be the best way to start. We will show here how to create a simple yet functional Apache configuration: ............... # crm configure crm(live)configure# template crm(live)configure template# list templates apache filesystem virtual-ip crm(live)configure template# new web apache filesystem virtual-ip crm(live)configure template# new web apache INFO: pulling in template apache INFO: pulling in template virtual-ip crm(live)configure template# list web2-d web2 vip2 web3 vip web ............... We enter the `template` level from `configure`. Use the `list` command to show templates available on the system. The `new` command creates a configuration from the +apache+ template. You can use tab completion to pick templates. Note that the apache template depends on a virtual IP address which is automatically pulled along. The `list` command shows the just created +web+ configuration, among other configurations (I hope that you, unlike me, will use more sensible and descriptive names). The `show` command, which displays the resulting configuration, may be used to get an idea about the minimum required changes which have to be done. All +ERROR+ messages show the line numbers in which the respective parameters are to be defined: ............... crm(live)configure template# show ERROR: 23: required parameter ip not set ERROR: 61: required parameter id not set ERROR: 65: required parameter configfile not set crm(live)configure template# edit ............... The `edit` command invokes the preferred text editor with the +web+ configuration. At the top of the file, the user is advised how to make changes. A good template should require from the user to specify only parameters. For example, the +web+ configuration we created above has the following required and optional parameters (all parameter lines start with +%%+): ............... $ grep -n ^%% ~/.crmconf/web 23:%% ip 31:%% netmask 35:%% lvs_support 61:%% id 65:%% configfile 71:%% options 76:%% envfiles ............... These lines are the only ones that should be modified. Simply append the parameter value at the end of the line. For instance, after editing this template, the result could look like this (we used tabs instead of spaces to make the values stand out): ............... $ grep -n ^%% ~/.crmconf/web 23:%% ip 192.168.1.101 31:%% netmask 35:%% lvs_support 61:%% id websvc 65:%% configfile /etc/apache2/httpd.conf 71:%% options 76:%% envfiles ............... As you can see, the parameter line format is very simple: ............... %% ............... After editing the file, use `show` again to display the configuration: ............... crm(live)configure template# show primitive virtual-ip IPaddr \ params ip=192.168.1.101 primitive apache apache \ params configfile="/etc/apache2/httpd.conf" monitor apache 120s:60s group websvc \ apache virtual-ip ............... The target resource of the apache template is a group which we named +websvc+ in this sample session. This configuration looks exactly as you could type it at the `configure` level. The point of templates is to save you some typing. It is important, however, to understand the configuration produced. Finally, the configuration may be applied to the current crm configuration (note how the configuration changed slightly, though it is still equivalent, after being digested at the `configure` level): ............... crm(live)configure template# apply crm(live)configure template# cd .. crm(live)configure# show node xen-b node xen-c primitive apache apache \ params configfile="/etc/apache2/httpd.conf" \ op monitor interval=120s timeout=60s primitive virtual-ip IPaddr \ params ip=192.168.1.101 group websvc apache virtual-ip ............... Note that this still does not commit the configuration to the CIB which is used in the shell, either the running one (+live+) or some shadow CIB. For that you still need to execute the `commit` command. To complete our example, we should also define the preferred node to run the service: ............... crm(live)configure# location websvc-pref websvc 100: xen-b ............... If you are not happy with some resource names which are provided by default, you can rename them now: ............... crm(live)configure# rename virtual-ip intranet-ip crm(live)configure# show node xen-b node xen-c primitive apache apache \ params configfile="/etc/apache2/httpd.conf" \ op monitor interval=120s timeout=60s primitive intranet-ip IPaddr \ params ip=192.168.1.101 group websvc apache intranet-ip location websvc-pref websvc 100: xen-b ............... To summarize, working with templates typically consists of the following steps: - `new`: create a new configuration from templates - `edit`: define parameters, at least the required ones - `show`: see if the configuration is valid - `apply`: apply the configuration to the `configure` level [[topics_Features_Testing,Resource testing]] === Resource testing The amount of detail in a cluster makes all configurations prone to errors. By far the largest number of issues in a cluster is due to bad resource configuration. The shell can help quickly diagnose such problems. And considerably reduce your keyboard wear. Let's say that we entered the following configuration: ............... node xen-b node xen-c node xen-d primitive fencer stonith:external/libvirt \ params hypervisor_uri="qemu+tcp://10.2.13.1/system" \ hostlist="xen-b xen-c xen-d" \ op monitor interval=2h primitive svc Xinetd \ params service=systat \ op monitor interval=30s primitive intranet-ip IPaddr2 \ params ip=10.2.13.100 \ op monitor interval=30s primitive apache apache \ params configfile="/etc/apache2/httpd.conf" \ op monitor interval=120s timeout=60s group websvc apache intranet-ip location websvc-pref websvc 100: xen-b ............... Before typing `commit` to submit the configuration to the cib we can make sure that all resources are usable on all nodes: ............... crm(live)configure# rsctest websvc svc fencer ............... It is important that resources being tested are not running on any nodes. Otherwise, the `rsctest` command will refuse to do anything. Of course, if the current configuration resides in a CIB shadow, then a `commit` is irrelevant. The point being that resources are not running on any node. .Note on stopping all resources **************************** Alternatively to not committing a configuration, it is also possible to tell Pacemaker not to start any resources: ............... crm(live)configure# property stop-all-resources=yes ............... Almost none---resources of class stonith are still started. But shell is not as strict when it comes to stonith resources. **************************** Order of resources is significant insofar that a resource depends on all resources to its left. In most configurations, it's probably practical to test resources in several runs, based on their dependencies. Apart from groups, `crm` does not interpret constraints and therefore knows nothing about resource dependencies. It also doesn't know if a resource can run on a node at all in case of an asymmetric cluster. It is up to the user to specify a list of eligible nodes if a resource is not meant to run on every node. [[topics_Features_Security,Access Control Lists (ACL)]] === Access Control Lists (ACL) .Note on ACLs in Pacemaker 1.1.12 **************************** The support for ACLs has been revised in Pacemaker version 1.1.12 and up. Depending on which version you are using, the information in this section may no longer be accurate. Look for the `acl_target` configuration element for more details on the new syntax. **************************** By default, the users from the +haclient+ group have full access to the cluster (or, more precisely, to the CIB). Access control lists allow for finer access control to the cluster. Access control lists consist of an ordered set of access rules. Each rule allows read or write access or denies access completely. Rules are typically combined to produce a specific role. Then, users may be assigned a role. For instance, this is a role which defines a set of rules allowing management of a single resource: ............... role bigdb_admin \ write meta:bigdb:target-role \ write meta:bigdb:is-managed \ write location:bigdb \ read ref:bigdb ............... The first two rules allow modifying the +target-role+ and +is-managed+ meta attributes which effectively enables users in this role to stop/start and manage/unmanage the resource. The constraints write access rule allows moving the resource around. Finally, the user is granted read access to the resource definition. For proper operation of all Pacemaker programs, it is advisable to add the following role to all users: ............... role read_all \ read cib ............... For finer grained read access try with the rules listed in the following role: ............... role basic_read \ read node attribute:uname \ read node attribute:type \ read property \ read status ............... It is however possible that some Pacemaker programs (e.g. `ptest`) may not function correctly if the whole CIB is not readable. Some of the ACL rules in the examples above are expanded by the shell to XPath specifications. For instance, +meta:bigdb:target-role+ expands to: ........ //primitive[@id='bigdb']/meta_attributes/nvpair[@name='target-role'] ........ You can see the expansion by showing XML: ............... crm(live) configure# show xml bigdb_admin ... ............... Many different XPath expressions can have equal meaning. For instance, the following two are equal, but only the first one is going to be recognized as shortcut: ............... //primitive[@id='bigdb']/meta_attributes/nvpair[@name='target-role'] //resources/primitive[@id='bigdb']/meta_attributes/nvpair[@name='target-role'] ............... XPath is a powerful language, but you should try to keep your ACL xpaths simple and the builtin shortcuts should be used whenever possible. [[topics_Features_Resourcesets,Syntax: Resource sets]] === Syntax: Resource sets Using resource sets can be a bit confusing unless one knows the details of the implementation in Pacemaker as well as how to interpret the syntax provided by `crmsh`. Three different types of resource sets are provided by `crmsh`, and each one implies different values for the two resource set attributes, +sequential+ and +require-all+. +sequential+:: If false, the resources in the set do not depend on each other internally. Setting +sequential+ to +true+ implies a strict order of dependency within the set. +require-all+:: If false, only one resource in the set is required to fulfil the requirements of the set. The set of A, B and C with +require-all+ set to +false+ is be read as "A OR B OR C" when its dependencies are resolved. The three types of resource sets modify the attributes in the following way: 1. Implicit sets (no brackets). +sequential=true+, +require-all=true+ 2. Parenthesis set (+(+ ... +)+). +sequential=false+, +require-all=true+ 3. Bracket set (+[+ ... +]+). +sequential=false+, +require-all=false+ To create a set with the properties +sequential=true+ and +require-all=false+, explicitly set +sequential+ in a bracketed set, +[ A B C sequential=true ]+. To create multiple sets with both +sequential+ and +require-all+ set to true, explicitly set +sequential+ in a parenthesis set: +A B ( C D sequential=true )+. [[topics_Features_AttributeListReferences,Syntax: Attribute list references]] === Syntax: Attribute list references Attribute lists are used to set attributes and parameters for resources, constraints and property definitions. For example, to set the virtual IP used by an +IPAddr2+ resource the attribute +ip+ can be set in an attribute list for that resource. Attribute lists can have identifiers that name them, and other resources can reuse the same attribute list by referring to that name using an +$id-ref+. For example, the following statement defines a simple dummy resource with an attribute list which sets the parameter +state+ to the value 1 and sets the identifier for the attribute list to +on-state+: .............. primitive dummy-1 Dummy params $id=on-state state=1 .............. To refer to this attribute list from a different resource, refer to the +on-state+ name using an id-ref: .............. primitive dummy-2 Dummy params $id-ref=on-state .............. The resource +dummy-2+ will now also have the parameter +state+ set to the value 1. [[topics_Features_AttributeReferences,Syntax: Attribute references]] === Syntax: Attribute references In some cases, referencing complete attribute lists is too coarse-grained, for example if two different parameters with different names should have the same value set. Instead of having to copy the value in multiple places, it is possible to create references to individual attributes in attribute lists. To name an attribute in order to be able to refer to it later, prefix the attribute name with a +$+ character (as seen above with the special names +$id+ and +$id-ref+: ............ primitive dummy-1 Dummy params $state=1 ............ The identifier +state+ can now be used to refer to this attribute from other primitives, using the +@+ syntax: ............ primitive dummy-2 Dummy params @state ............ In some cases, using the attribute name as the identifier doesn't work due to name clashes. In this case, the syntax +$:=+ can be used to give the attribute a different identifier: ............ primitive dummy-1 params $dummy-state-on:state=1 primitive dummy-2 params @dummy-state-on ............ There is also the possibility that two resources both use the same attribute value but with different names. For example, a web server may have a parameter +server_ip+ for setting the IP address where it listens for incoming requests, and a virtual IP resource may have a parameter called +ip+ which sets the IP address it creates. To configure these two resources with an IP without repeating the value, the reference can be given a name using the syntax +@:+. Example: ............ primitive virtual-ip IPaddr2 params $vip:ip=192.168.1.100 primitive webserver apache params @vip:server_ip ............ [[topics_Syntax_RuleExpressions,Syntax: Rule expressions]] === Syntax: Rule expressions Many of the configuration commands in `crmsh` now support the use of _rule expressions_, which can influence what attributes apply to a resource or under which conditions a constraint is applied, depending on changing conditions like date, time, the value of attributes and more. Here is an example of a simple rule expression used to apply a a different resource parameter on the node named `node1`: .............. primitive my_resource Special \ params 2: rule #uname eq node1 interface=eth1 \ params 1: interface=eth0 .............. This primitive resource has two lists of parameters with descending priority. The parameter list with the highest priority is applied first, but only if the rule expressions for that parameter list all apply. In this case, the rule `#uname eq node1` limits the parameter list so that it is only applied on `node1`. Note that rule expressions are not terminated and are immediately followed by the data to which the rule is applied. In this case, the name-value pair `interface=eth1`. Rule expressions can contain multiple expressions connected using the boolean operator `or` and `and`. The full syntax for rule expressions is listed below. .............. rules :: rule [id_spec] [$role=] : [rule [id_spec] [$role=] : ...] id_spec :: $id= | $id-ref= score :: | | [-]inf expression :: [ ...] bool_op :: or | and simple_exp :: [type:] | | date type :: | | binary_op :: lt | gt | lte | gte | eq | ne unary_op :: defined | not_defined date_expr :: lt | gt | in start= end= | in start= | spec duration|date_spec :: hours= | monthdays= | weekdays= | yearsdays= | months= | weeks= | years= | weekyears= | moon= .............. [[topics_Lifetime,Lifetime parameter format]] == Lifetime parameter format Lifetimes can be specified in the ISO 8601 time format or the ISO 8601 duration format. To distinguish between months and minutes, use the PT prefix before specifying minutes. The duration format is one of +PnYnMnDTnHnMnS+, +PnW+, +PT