diff options
| author | Daniel Baumann <daniel.baumann@progress-linux.org> | 2024-05-05 17:47:29 +0000 |
|---|---|---|
| committer | Daniel Baumann <daniel.baumann@progress-linux.org> | 2024-05-05 17:47:29 +0000 |
| commit | 4f5791ebd03eaec1c7da0865a383175b05102712 (patch) | |
| tree | 8ce7b00f7a76baa386372422adebbe64510812d4 /ctdb/doc | |
| parent | Initial commit. (diff) | |
| download | samba-upstream.tar.xz samba-upstream.zip | |
Adding upstream version 2:4.17.12+dfsg.upstream/2%4.17.12+dfsgupstream
Signed-off-by: Daniel Baumann <daniel.baumann@progress-linux.org>
Diffstat (limited to '')
25 files changed, 9715 insertions, 0 deletions
diff --git a/ctdb/doc/cluster_mutex_helper.txt b/ctdb/doc/cluster_mutex_helper.txt new file mode 100644 index 0000000..4ee018f --- /dev/null +++ b/ctdb/doc/cluster_mutex_helper.txt @@ -0,0 +1,80 @@ +Writing CTDB cluster mutex helpers +================================== + +CTDB uses cluster-wide mutexes to protect against a "split brain", +which could occur if the cluster becomes partitioned due to network +failure or similar. + +CTDB uses a cluster-wide mutex for its "cluster lock", which is used +to ensure that only one database recovery can happen at a time. For +an overview of cluster lock configuration see the CLUSTER LOCK +section in ctdb(7). CTDB tries to ensure correct operation of the +cluster lock by attempting to take the cluster lock when CTDB knows +that it should already be held. + +By default, CTDB uses a supplied mutex helper that uses a fcntl(2) +lock on a specified file in the cluster filesystem. + +However, a user supplied mutex helper can be used as an alternative. +The rest of this document describes the API for mutex helpers. + +A mutex helper is an external executable +---------------------------------------- + +A mutex helper is an external executable that can be run by CTDB. +There are no CTDB-specific compilation dependencies. This means that +a helper could easily be scripted around existing commands. Mutex +helpers are run relatively rarely and are not time critical. +Therefore, reliability is preferred over high performance. + +Taking a mutex with a helper +---------------------------- + +1. Helper is executed with helper-specific arguments + +2. Helper attempts to take mutex + +3. On success, the helper writes ASCII 0 to standard output + +4. Helper stays running, holding mutex, awaiting termination by CTDB + +5. When a helper receives SIGTERM it must release any mutex it is + holding and then exit. + +Status codes +------------ + +CTDB ignores the exit code of a helper. Instead, CTDB reacts to a +single ASCII character that is sent to it via a helper's standard +output. + +Valid status codes are: + +0 - The helper took the mutex and is holding it, awaiting termination. + +1 - The helper was unable to take the mutex due to contention. + +2 - The helper took too long to take the mutex. + + Helpers do not need to implement this status code. CTDB + already implements any required timeout handling. + +3 - An unexpected error occurred. + +If a 0 status code is sent then it the helper should periodically +check if the (original) parent processes still exists while awaiting +termination. If the parent process disappears then the helper should +release the mutex and exit. This avoids stale mutexes. Note that a +helper should never wait for parent process ID 1! + +If a non-0 status code is sent then the helper can exit immediately. +However, if the helper does not exit then it must terminate if it +receives SIGTERM. + +Logging +------- + +Anything written to standard error by a helper is incorporated into +CTDB's logs. A helper should generally only output to stderr for +unexpected errors and avoid output to stderr on success or on mutex +contention. diff --git a/ctdb/doc/ctdb-etcd.7.xml b/ctdb/doc/ctdb-etcd.7.xml new file mode 100644 index 0000000..f84989f --- /dev/null +++ b/ctdb/doc/ctdb-etcd.7.xml @@ -0,0 +1,119 @@ +<?xml version="1.0" encoding="iso-8859-1"?> +<!DOCTYPE refentry + PUBLIC "-//OASIS//DTD DocBook XML V4.5//EN" + "http://www.oasis-open.org/docbook/xml/4.5/docbookx.dtd"> +<refentry id="ctdb-etcd.7"> + + <refentryinfo> + <author> + <contrib> + This documentation was written by + Jose A. Rivera + </contrib> + </author> + + <copyright> + <year>2016</year> + <holder>Jose A. Rivera</holder> + </copyright> + <legalnotice> + <para> + This program is free software; you can redistribute it and/or + modify it under the terms of the GNU General Public License as + published by the Free Software Foundation; either version 3 of + the License, or (at your option) any later version. + </para> + <para> + This program is distributed in the hope that it will be + useful, but WITHOUT ANY WARRANTY; without even the implied + warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR + PURPOSE. See the GNU General Public License for more details. + </para> + <para> + You should have received a copy of the GNU General Public + License along with this program; if not, see + <ulink url="http://www.gnu.org/licenses"/>. + </para> + </legalnotice> + </refentryinfo> + + <refmeta> + <refentrytitle>ctdb-etcd</refentrytitle> + <manvolnum>7</manvolnum> + <refmiscinfo class="source">ctdb</refmiscinfo> + <refmiscinfo class="manual">CTDB - clustered TDB database</refmiscinfo> + </refmeta> + + <refnamediv> + <refname>ctdb-etcd</refname> + <refpurpose>CTDB etcd integration</refpurpose> + </refnamediv> + + <refsynopsisdiv> + <cmdsynopsis> + <command>ctdb_etcd_lock</command> + </cmdsynopsis> + </refsynopsisdiv> + + <refsect1> + <title>DESCRIPTION</title> + <para> + ctdb_etcd_lock is intended to be run as a mutex helper for CTDB. It + will try to connect to an existing etcd cluster and grab a lock in that + cluster to function as CTDB's cluster lock. Please see + <emphasis>ctdb/doc/cluster_mutex_helper.txt</emphasis> for details on + the mutex helper API. To use this, include the following line in + the <literal>[cluster]</literal> section of + <citerefentry><refentrytitle>ctdb.conf</refentrytitle> + <manvolnum>5</manvolnum></citerefentry>: + </para> + <screen format="linespecific"> +cluster lock = !/usr/local/usr/libexec/ctdb/ctdb_etcd_lock + </screen> + <para> + You can also pass "-v", "-vv", or "-vvv" to include verbose output in + the CTDB log. Additional "v"s indicate increases in verbosity. + </para> + <para> + This mutex helper expects the system Python interpreter to have access + to the etcd Python module. It also expects an etcd cluster to be + configured and running. To integrate with this, there is an optional + config file of the following format: + </para> + <screen format="linespecific"> +key = value + </screen> + <para> + The following configuration parameters (and their defaults) are defined + for use by ctdb_etcd_lock: + </para> + <screen format="linespecific"> +port = 2379 # connecting port for the etcd cluster +lock_ttl = 9 # seconds for TTL +refresh = 2 # seconds between attempts to maintain lock +locks_dir = _ctdb # where to store CTDB locks in etcd + # The final etcd directory for any given lock looks like: + # /_locks/{locks_dir}/{netbios name}/ + </screen> + <para> + In addition, any keyword parameter that can be used to configure an + etcd client may be specified and modified here. For more documentation + on these parameters, see here: https://github.com/jplana/python-etcd/ + </para> + </refsect1> + + <refsect1> + <title>SEE ALSO</title> + <para> + <citerefentry><refentrytitle>ctdb</refentrytitle> + <manvolnum>7</manvolnum></citerefentry>, + + <citerefentry><refentrytitle>ctdbd</refentrytitle> + <manvolnum>1</manvolnum></citerefentry>, + + <ulink url="http://ctdb.samba.org/"/> + </para> + </refsect1> + + +</refentry> diff --git a/ctdb/doc/ctdb-script.options.5.xml b/ctdb/doc/ctdb-script.options.5.xml new file mode 100644 index 0000000..700d1f7 --- /dev/null +++ b/ctdb/doc/ctdb-script.options.5.xml @@ -0,0 +1,1076 @@ +<?xml version="1.0" encoding="iso-8859-1"?> +<!DOCTYPE refentry + PUBLIC "-//OASIS//DTD DocBook XML V4.5//EN" + "http://www.oasis-open.org/docbook/xml/4.5/docbookx.dtd"> + +<refentry id="ctdb-script.options.5"> + + <refmeta> + <refentrytitle>ctdb-script.options</refentrytitle> + <manvolnum>5</manvolnum> + <refmiscinfo class="source">ctdb</refmiscinfo> + <refmiscinfo class="manual">CTDB - clustered TDB database</refmiscinfo> + </refmeta> + + <refnamediv> + <refname>ctdb-script.options</refname> + <refpurpose>CTDB scripts configuration files</refpurpose> + </refnamediv> + + <refsect1> + <title>DESCRIPTION</title> + + <para> + Each CTDB script has 2 possible locations for its configuration options: + </para> + + <variablelist> + + <varlistentry> + <term> + <filename>/usr/local/etc/ctdb/script.options</filename> + </term> + <listitem> + <para> + This is a catch-all global file for general purpose + scripts and for options that are used in multiple event + scripts. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term> + <parameter>SCRIPT</parameter>.options + </term> + <listitem> + <para> + That is, options for + <filename><parameter>SCRIPT</parameter></filename> are + placed in a file alongside the script, with a ".script" + suffix added. This style is usually recommended for event + scripts. + </para> + + <para> + Options in this script-specific file override those in + the global file. + </para> + </listitem> + </varlistentry> + + </variablelist> + + <para> + These files should include simple shell-style variable + assignments and shell-style comments. + </para> + + </refsect1> + + <refsect1> + <title>NETWORK CONFIGURATION</title> + + <refsect2> + <title>10.interface</title> + + <para> + This event script handles monitoring of interfaces using by + public IP addresses. + </para> + + <variablelist> + + <varlistentry> + <term> + CTDB_PARTIALLY_ONLINE_INTERFACES=yes|no + </term> + <listitem> + <para> + Whether one or more offline interfaces should cause a + monitor event to fail if there are other interfaces that + are up. If this is "yes" and a node has some interfaces + that are down then <command>ctdb status</command> will + display the node as "PARTIALLYONLINE". + </para> + + <para> + Note that CTDB_PARTIALLY_ONLINE_INTERFACES=yes is not + generally compatible with NAT gateway or LVS. NAT + gateway relies on the interface configured by + CTDB_NATGW_PUBLIC_IFACE to be up and LVS replies on + CTDB_LVS_PUBLIC_IFACE to be up. CTDB does not check if + these options are set in an incompatible way so care is + needed to understand the interaction. + </para> + + <para> + Default is "no". + </para> + </listitem> + </varlistentry> + + </variablelist> + </refsect2> + + <refsect2> + <title>11.natgw</title> + + <para> + Provides CTDB's NAT gateway functionality. + </para> + + <para> + NAT gateway is used to configure fallback routing for nodes + when they do not host any public IP addresses. For example, + it allows unhealthy nodes to reliably communicate with + external infrastructure. One node in a NAT gateway group will + be designated as the NAT gateway leader node and other (follower) + nodes will be configured with fallback routes via the NAT + gateway leader node. For more information, see the + <citetitle>NAT GATEWAY</citetitle> section in + <citerefentry><refentrytitle>ctdb</refentrytitle> + <manvolnum>7</manvolnum></citerefentry>. + </para> + + <variablelist> + + <varlistentry> + <term>CTDB_NATGW_DEFAULT_GATEWAY=<parameter>IPADDR</parameter></term> + <listitem> + <para> + IPADDR is an alternate network gateway to use on the NAT + gateway leader node. If set, a fallback default route + is added via this network gateway. + </para> + <para> + No default. Setting this variable is optional - if not + set that no route is created on the NAT gateway leader + node. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term>CTDB_NATGW_NODES=<parameter>FILENAME</parameter></term> + <listitem> + <para> + FILENAME contains the list of nodes that belong to the + same NAT gateway group. + </para> + <para> + File format: + <screen> +<parameter>IPADDR</parameter> <optional>follower-only</optional> + </screen> + </para> + <para> + IPADDR is the private IP address of each node in the NAT + gateway group. + </para> + <para> + If "follower-only" is specified then the corresponding node + can not be the NAT gateway leader node. In this case + <varname>CTDB_NATGW_PUBLIC_IFACE</varname> and + <varname>CTDB_NATGW_PUBLIC_IP</varname> are optional and + unused. + </para> + <para> + No default, usually + <filename>/usr/local/etc/ctdb/natgw_nodes</filename> when enabled. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term>CTDB_NATGW_PRIVATE_NETWORK=<parameter>IPADDR/MASK</parameter></term> + <listitem> + <para> + IPADDR/MASK is the private sub-network that is + internally routed via the NAT gateway leader node. This + is usually the private network that is used for node + addresses. + </para> + <para> + No default. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term>CTDB_NATGW_PUBLIC_IFACE=<parameter>IFACE</parameter></term> + <listitem> + <para> + IFACE is the network interface on which the + CTDB_NATGW_PUBLIC_IP will be configured. + </para> + <para> + No default. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term>CTDB_NATGW_PUBLIC_IP=<parameter>IPADDR/MASK</parameter></term> + <listitem> + <para> + IPADDR/MASK indicates the IP address that is used for + outgoing traffic (originating from + CTDB_NATGW_PRIVATE_NETWORK) on the NAT gateway leader + node. This <emphasis>must not</emphasis> be a + configured public IP address. + </para> + <para> + No default. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term>CTDB_NATGW_STATIC_ROUTES=<parameter>IPADDR/MASK[@GATEWAY]</parameter> ...</term> + <listitem> + <para> + Each IPADDR/MASK identifies a network or host to which + NATGW should create a fallback route, instead of + creating a single default route. This can be used when + there is already a default route, via an interface that + can not reach required infrastructure, that overrides + the NAT gateway default route. + </para> + <para> + If GATEWAY is specified then the corresponding route on + the NATGW leader node will be via GATEWAY. Such routes + are created even if + <varname>CTDB_NATGW_DEFAULT_GATEWAY</varname> is not + specified. If GATEWAY is not specified for some + networks then routes are only created on the NATGW + leader node for those networks if + <varname>CTDB_NATGW_DEFAULT_GATEWAY</varname> is + specified. + </para> + <para> + This should be used with care to avoid causing traffic + to unnecessarily double-hop through the NAT gateway + leader, even when a node is hosting public IP addresses. + Each specified network or host should probably have a + corresponding automatically created link route or static + route to avoid this. + </para> + <para> + No default. + </para> + </listitem> + </varlistentry> + + </variablelist> + + <refsect3> + <title>Example</title> + <screen> +CTDB_NATGW_NODES=/usr/local/etc/ctdb/natgw_nodes +CTDB_NATGW_PRIVATE_NETWORK=192.168.1.0/24 +CTDB_NATGW_DEFAULT_GATEWAY=10.0.0.1 +CTDB_NATGW_PUBLIC_IP=10.0.0.227/24 +CTDB_NATGW_PUBLIC_IFACE=eth0 + </screen> + + <para> + A variation that ensures that infrastructure (ADS, DNS, ...) + directly attached to the public network (10.0.0.0/24) is + always reachable would look like this: + </para> + <screen> +CTDB_NATGW_NODES=/usr/local/etc/ctdb/natgw_nodes +CTDB_NATGW_PRIVATE_NETWORK=192.168.1.0/24 +CTDB_NATGW_PUBLIC_IP=10.0.0.227/24 +CTDB_NATGW_PUBLIC_IFACE=eth0 +CTDB_NATGW_STATIC_ROUTES=10.0.0.0/24 + </screen> + <para> + Note that <varname>CTDB_NATGW_DEFAULT_GATEWAY</varname> is + not specified. + </para> + </refsect3> + + </refsect2> + + <refsect2> + <title>13.per_ip_routing</title> + + <para> + Provides CTDB's policy routing functionality. + </para> + + <para> + A node running CTDB may be a component of a complex network + topology. In particular, public addresses may be spread + across several different networks (or VLANs) and it may not be + possible to route packets from these public addresses via the + system's default route. Therefore, CTDB has support for + policy routing via the <filename>13.per_ip_routing</filename> + eventscript. This allows routing to be specified for packets + sourced from each public address. The routes are added and + removed as CTDB moves public addresses between nodes. + </para> + + <para> + For more information, see the <citetitle>POLICY + ROUTING</citetitle> section in + <citerefentry><refentrytitle>ctdb</refentrytitle> + <manvolnum>7</manvolnum></citerefentry>. + </para> + + <variablelist> + <varlistentry> + <term>CTDB_PER_IP_ROUTING_CONF=<parameter>FILENAME</parameter></term> + <listitem> + <para> + FILENAME contains elements for constructing the desired + routes for each source address. + </para> + + <para> + The special FILENAME value + <constant>__auto_link_local__</constant> indicates that no + configuration file is provided and that CTDB should + generate reasonable link-local routes for each public IP + address. + </para> + + <para> + File format: + <screen> + <parameter>IPADDR</parameter> <parameter>DEST-IPADDR/MASK</parameter> <optional><parameter>GATEWAY-IPADDR</parameter></optional> + </screen> + </para> + + <para> + No default, usually + <filename>/usr/local/etc/ctdb/policy_routing</filename> + when enabled. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term> + CTDB_PER_IP_ROUTING_RULE_PREF=<parameter>NUM</parameter> + </term> + <listitem> + <para> + NUM sets the priority (or preference) for the routing + rules that are added by CTDB. + </para> + + <para> + This should be (strictly) greater than 0 and (strictly) + less than 32766. A priority of 100 is recommended, unless + this conflicts with a priority already in use on the + system. See + <citerefentry><refentrytitle>ip</refentrytitle> + <manvolnum>8</manvolnum></citerefentry>, for more details. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term> + CTDB_PER_IP_ROUTING_TABLE_ID_LOW=<parameter>LOW-NUM</parameter>, + CTDB_PER_IP_ROUTING_TABLE_ID_HIGH=<parameter>HIGH-NUM</parameter> + </term> + <listitem> + <para> + CTDB determines a unique routing table number to use for + the routing related to each public address. LOW-NUM and + HIGH-NUM indicate the minimum and maximum routing table + numbers that are used. + </para> + + <para> + <citerefentry><refentrytitle>ip</refentrytitle> + <manvolnum>8</manvolnum></citerefentry> uses some + reserved routing table numbers below 255. Therefore, + CTDB_PER_IP_ROUTING_TABLE_ID_LOW should be (strictly) + greater than 255. + </para> + + <para> + CTDB uses the standard file + <filename>/etc/iproute2/rt_tables</filename> to maintain + a mapping between the routing table numbers and labels. + The label for a public address + <replaceable>ADDR</replaceable> will look like + ctdb.<replaceable>addr</replaceable>. This means that + the associated rules and routes are easy to read (and + manipulate). + </para> + + <para> + No default, usually 1000 and 9000. + </para> + </listitem> + </varlistentry> + </variablelist> + + <refsect3> + <title>Example</title> + <screen> +CTDB_PER_IP_ROUTING_CONF=/usr/local/etc/ctdb/policy_routing +CTDB_PER_IP_ROUTING_RULE_PREF=100 +CTDB_PER_IP_ROUTING_TABLE_ID_LOW=1000 +CTDB_PER_IP_ROUTING_TABLE_ID_HIGH=9000 + </screen> + </refsect3> + + </refsect2> + + <refsect2> + <title>91.lvs</title> + + <para> + Provides CTDB's LVS functionality. + </para> + + <para> + For a general description see the <citetitle>LVS</citetitle> + section in <citerefentry><refentrytitle>ctdb</refentrytitle> + <manvolnum>7</manvolnum></citerefentry>. + </para> + + <variablelist> + + <varlistentry> + <term> + CTDB_LVS_NODES=<parameter>FILENAME</parameter> + </term> + <listitem> + <para> + FILENAME contains the list of nodes that belong to the + same LVS group. + </para> + <para> + File format: + <screen> +<parameter>IPADDR</parameter> <optional>follower-only</optional> + </screen> + </para> + <para> + IPADDR is the private IP address of each node in the LVS + group. + </para> + <para> + If "follower-only" is specified then the corresponding node + can not be the LVS leader node. In this case + <varname>CTDB_LVS_PUBLIC_IFACE</varname> and + <varname>CTDB_LVS_PUBLIC_IP</varname> are optional and + unused. + </para> + <para> + No default, usually + <filename>/usr/local/etc/ctdb/lvs_nodes</filename> when enabled. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term> + CTDB_LVS_PUBLIC_IFACE=<parameter>INTERFACE</parameter> + </term> + <listitem> + <para> + INTERFACE is the network interface that clients will use + to connection to <varname>CTDB_LVS_PUBLIC_IP</varname>. + This is optional for follower-only nodes. + No default. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term> + CTDB_LVS_PUBLIC_IP=<parameter>IPADDR</parameter> + </term> + <listitem> + <para> + CTDB_LVS_PUBLIC_IP is the LVS public address. No + default. + </para> + </listitem> + </varlistentry> + + </variablelist> + </refsect2> + + </refsect1> + + <refsect1> + <title>SERVICE CONFIGURATION</title> + + <para> + CTDB can be configured to manage and/or monitor various NAS (and + other) services via its eventscripts. + </para> + + <para> + In the simplest case CTDB will manage a service. This means the + service will be started and stopped along with CTDB, CTDB will + monitor the service and CTDB will do any required + reconfiguration of the service when public IP addresses are + failed over. + </para> + + <refsect2> + <title>20.multipathd</title> + + <para> + Provides CTDB's Linux multipathd service management. + </para> + + <para> + It can monitor multipath devices to ensure that active paths + are available. + </para> + + <variablelist> + <varlistentry> + <term> + CTDB_MONITOR_MPDEVICES=<parameter>MP-DEVICE-LIST</parameter> + </term> + <listitem> + <para> + MP-DEVICE-LIST is a list of multipath devices for CTDB to monitor? + </para> + <para> + No default. + </para> + </listitem> + </varlistentry> + </variablelist> + </refsect2> + + <refsect2> + <title>31.clamd</title> + + <para> + This event script provide CTDB's ClamAV anti-virus service + management. + </para> + + <para> + This eventscript is not enabled by default. Use <command>ctdb + enablescript</command> to enable it. + </para> + + <variablelist> + + <varlistentry> + <term> + CTDB_CLAMD_SOCKET=<parameter>FILENAME</parameter> + </term> + <listitem> + <para> + FILENAME is the socket to monitor ClamAV. + </para> + <para> + No default. + </para> + </listitem> + </varlistentry> + + </variablelist> + + </refsect2> + + <refsect2> + <title>48.netbios</title> + + <para> + Provides CTDB's NetBIOS service management. + </para> + + <variablelist> + <varlistentry> + <term> + CTDB_SERVICE_NMB=<parameter>SERVICE</parameter> + </term> + <listitem> + <para> + Distribution specific SERVICE for managing nmbd. + </para> + <para> + Default is distribution-dependant. + </para> + </listitem> + </varlistentry> + + </variablelist> + + </refsect2> + + <refsect2> + <title>49.winbind</title> + + <para> + Provides CTDB's Samba winbind service management. + </para> + + <variablelist> + + <varlistentry> + <term> + CTDB_SERVICE_WINBIND=<parameter>SERVICE</parameter> + </term> + <listitem> + <para> + Distribution specific SERVICE for managing winbindd. + </para> + <para> + Default is "winbind". + </para> + </listitem> + </varlistentry> + + </variablelist> + + </refsect2> + + <refsect2> + <title>50.samba</title> + + <para> + Provides the core of CTDB's Samba file service management. + </para> + + <variablelist> + + <varlistentry> + <term> + CTDB_SAMBA_CHECK_PORTS=<parameter>PORT-LIST</parameter> + </term> + <listitem> + <para> + When monitoring Samba, check TCP ports in + space-separated PORT-LIST. + </para> + <para> + Default is to monitor ports that Samba is configured to listen on. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term> + CTDB_SAMBA_SKIP_SHARE_CHECK=yes|no + </term> + <listitem> + <para> + As part of monitoring, should CTDB skip the check for + the existence of each directory configured as share in + Samba. This may be desirable if there is a large number + of shares. + </para> + <para> + Default is no. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term> + CTDB_SERVICE_SMB=<parameter>SERVICE</parameter> + </term> + <listitem> + <para> + Distribution specific SERVICE for managing smbd. + </para> + <para> + Default is distribution-dependant. + </para> + </listitem> + </varlistentry> + + </variablelist> + + </refsect2> + + <refsect2> + <title>60.nfs</title> + + <para> + This event script (along with 06.nfs) provides CTDB's NFS + service management. + </para> + + <para> + This includes parameters for the kernel NFS server. + Alternative NFS subsystems (such as <ulink + url="https://github.com/nfs-ganesha/nfs-ganesha/wiki">NFS-Ganesha</ulink>) + can be integrated using <varname>CTDB_NFS_CALLOUT</varname>. + </para> + + <variablelist> + + <varlistentry> + <term> + CTDB_NFS_CALLOUT=<parameter>COMMAND</parameter> + </term> + <listitem> + <para> + COMMAND specifies the path to a callout to handle + interactions with the configured NFS system, including + startup, shutdown, monitoring. + </para> + <para> + Default is the included + <command>nfs-linux-kernel-callout</command>. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term> + CTDB_NFS_CHECKS_DIR=<parameter>DIRECTORY</parameter> + </term> + <listitem> + <para> + Specifies the path to a DIRECTORY containing files that + describe how to monitor the responsiveness of NFS RPC + services. See the README file for this directory for an + explanation of the contents of these "check" files. + </para> + <para> + CTDB_NFS_CHECKS_DIR can be used to point to different + sets of checks for different NFS servers. + </para> + <para> + One way of using this is to have it point to, say, + <filename>/usr/local/etc/ctdb/nfs-checks-enabled.d</filename> + and populate it with symbolic links to the desired check + files. This avoids duplication and is upgrade-safe. + </para> + <para> + Default is + <filename>/usr/local/etc/ctdb/nfs-checks.d</filename>, + which contains NFS RPC checks suitable for Linux kernel + NFS. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term> + CTDB_NFS_SKIP_SHARE_CHECK=yes|no + </term> + <listitem> + <para> + As part of monitoring, should CTDB skip the check for + the existence of each directory exported via NFS. This + may be desirable if there is a large number of exports. + </para> + <para> + Default is no. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term> + CTDB_RPCINFO_LOCALHOST=<parameter>IPADDR</parameter>|<parameter>HOSTNAME</parameter> + </term> + <listitem> + <para> + IPADDR or HOSTNAME indicates the address that + <command>rpcinfo</command> should connect to when doing + <command>rpcinfo</command> check on IPv4 RPC service during + monitoring. Optimally this would be "localhost". + However, this can add some performance overheads. + </para> + <para> + Default is "127.0.0.1". + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term> + CTDB_RPCINFO_LOCALHOST6=<parameter>IPADDR</parameter>|<parameter>HOSTNAME</parameter> + </term> + <listitem> + <para> + IPADDR or HOSTNAME indicates the address that + <command>rpcinfo</command> should connect to when doing + <command>rpcinfo</command> check on IPv6 RPC service + during monitoring. Optimally this would be "localhost6" + (or similar). However, this can add some performance + overheads. + </para> + <para> + Default is "::1". + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term> + CTDB_NFS_STATE_FS_TYPE=<parameter>TYPE</parameter> + </term> + <listitem> + <para> + The type of filesystem used for a clustered NFS' shared + state. No default. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term> + CTDB_NFS_STATE_MNT=<parameter>DIR</parameter> + </term> + <listitem> + <para> + The directory where a clustered NFS' shared state will be + located. No default. + </para> + </listitem> + </varlistentry> + + </variablelist> + + </refsect2> + + <refsect2> + <title>70.iscsi</title> + + <para> + Provides CTDB's Linux iSCSI tgtd service management. + </para> + + <variablelist> + + <varlistentry> + <term> + CTDB_START_ISCSI_SCRIPTS=<parameter>DIRECTORY</parameter> + </term> + <listitem> + <para> + DIRECTORY on shared storage containing scripts to start + tgtd for each public IP address. + </para> + <para> + No default. + </para> + </listitem> + </varlistentry> + </variablelist> + </refsect2> + + </refsect1> + + <refsect1> + <title> + DATABASE SETUP + </title> + + <para> + CTDB checks the consistency of databases during startup. + </para> + + <refsect2> + <title>00.ctdb</title> + + <variablelist> + + <varlistentry> + <term>CTDB_MAX_CORRUPT_DB_BACKUPS=<parameter>NUM</parameter></term> + <listitem> + <para> + NUM is the maximum number of volatile TDB database + backups to be kept (for each database) when a corrupt + database is found during startup. Volatile TDBs are + zeroed during startup so backups are needed to debug + any corruption that occurs before a restart. + </para> + <para> + Default is 10. + </para> + </listitem> + </varlistentry> + + </variablelist> + </refsect2> + + </refsect1> + + <refsect1> + <title>SYSTEM RESOURCE MONITORING</title> + + <refsect2> + <title> + 05.system + </title> + + <para> + Provides CTDB's filesystem and memory usage monitoring. + </para> + + <para> + CTDB can experience seemingly random (performance and other) + issues if system resources become too constrained. Options in + this section can be enabled to allow certain system resources + to be checked. They allows warnings to be logged and nodes to + be marked unhealthy when system resource usage reaches the + configured thresholds. + </para> + + <para> + Some checks are enabled by default. It is recommended that + these checks remain enabled or are augmented by extra checks. + There is no supported way of completely disabling the checks. + </para> + + <variablelist> + + <varlistentry> + <term> + CTDB_MONITOR_FILESYSTEM_USAGE=<parameter>FS-LIMIT-LIST</parameter> + </term> + <listitem> + <para> + FS-LIMIT-LIST is a space-separated list of + <parameter>FILESYSTEM</parameter>:<parameter>WARN_LIMIT</parameter><optional>:<parameter>UNHEALTHY_LIMIT</parameter></optional> + triples indicating that warnings should be logged if the + space used on FILESYSTEM reaches WARN_LIMIT%. If usage + reaches UNHEALTHY_LIMIT then the node should be flagged + unhealthy. Either WARN_LIMIT or UNHEALTHY_LIMIT may be + left blank, meaning that check will be omitted. + </para> + + <para> + Default is to warn for each filesystem containing a + database directory + (<literal>volatile database directory</literal>, + <literal>persistent database directory</literal>, + <literal>state database directory</literal>) + with a threshold of 90%. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term> + CTDB_MONITOR_MEMORY_USAGE=<parameter>MEM-LIMITS</parameter> + </term> + <listitem> + <para> + MEM-LIMITS takes the form + <parameter>WARN_LIMIT</parameter><optional>:<parameter>UNHEALTHY_LIMIT</parameter></optional> + indicating that warnings should be logged if memory + usage reaches WARN_LIMIT%. If usage reaches + UNHEALTHY_LIMIT then the node should be flagged + unhealthy. Either WARN_LIMIT or UNHEALTHY_LIMIT may be + left blank, meaning that check will be omitted. + </para> + <para> + Default is 80, so warnings will be logged when memory + usage reaches 80%. + </para> + </listitem> + </varlistentry> + + </variablelist> + </refsect2> + + </refsect1> + + + <refsect1> + <title>EVENT SCRIPT DEBUGGING</title> + + <refsect2> + <title> + debug-hung-script.sh + </title> + + <variablelist> + + <varlistentry> + <term>CTDB_DEBUG_HUNG_SCRIPT_STACKPAT=<parameter>REGEXP</parameter></term> + <listitem> + <para> + REGEXP specifies interesting processes for which stack + traces should be logged when debugging hung eventscripts + and those processes are matched in pstree output. + REGEXP is an extended regexp so choices are separated by + pipes ('|'). However, REGEXP should not contain + parentheses. See also the <citerefentry><refentrytitle>ctdb.conf</refentrytitle> + <manvolnum>5</manvolnum></citerefentry> + [event] "debug script" option. + </para> + <para> + Default is "exportfs|rpcinfo". + </para> + </listitem> + </varlistentry> + + </variablelist> + </refsect2> + + </refsect1> + + <refsect1> + <title>FILES</title> + + <simplelist> + <member><filename>/usr/local/etc/ctdb/script.options</filename></member> + </simplelist> + </refsect1> + + <refsect1> + <title>SEE ALSO</title> + <para> + <citerefentry><refentrytitle>ctdbd</refentrytitle> + <manvolnum>1</manvolnum></citerefentry>, + + <citerefentry><refentrytitle>ctdb</refentrytitle> + <manvolnum>7</manvolnum></citerefentry>, + + <ulink url="http://ctdb.samba.org/"/> + </para> + </refsect1> + + <refentryinfo> + <author> + <contrib> + This documentation was written by + Amitay Isaacs, + Martin Schwenke + </contrib> + </author> + + <copyright> + <year>2007</year> + <holder>Andrew Tridgell</holder> + <holder>Ronnie Sahlberg</holder> + </copyright> + <legalnotice> + <para> + This program is free software; you can redistribute it and/or + modify it under the terms of the GNU General Public License as + published by the Free Software Foundation; either version 3 of + the License, or (at your option) any later version. + </para> + <para> + This program is distributed in the hope that it will be + useful, but WITHOUT ANY WARRANTY; without even the implied + warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR + PURPOSE. See the GNU General Public License for more details. + </para> + <para> + You should have received a copy of the GNU General Public + License along with this program; if not, see + <ulink url="http://www.gnu.org/licenses"/>. + </para> + </legalnotice> + </refentryinfo> + +</refentry> diff --git a/ctdb/doc/ctdb-statistics.7.xml b/ctdb/doc/ctdb-statistics.7.xml new file mode 100644 index 0000000..387852d --- /dev/null +++ b/ctdb/doc/ctdb-statistics.7.xml @@ -0,0 +1,689 @@ +<?xml version="1.0" encoding="iso-8859-1"?> +<!DOCTYPE refentry + PUBLIC "-//OASIS//DTD DocBook XML V4.5//EN" + "http://www.oasis-open.org/docbook/xml/4.5/docbookx.dtd"> + +<refentry id="ctdb-statistics.7"> + + <refmeta> + <refentrytitle>ctdb-statistics</refentrytitle> + <manvolnum>7</manvolnum> + <refmiscinfo class="source">ctdb</refmiscinfo> + <refmiscinfo class="manual">CTDB - clustered TDB database</refmiscinfo> + </refmeta> + + <refnamediv> + <refname>ctdb-statistics</refname> + <refpurpose>CTDB statistics output</refpurpose> + </refnamediv> + + <refsect1> + <title>OVERALL STATISTICS</title> + + <para> + CTDB maintains information about various messages communicated + and some of the important operations per node. See the + <citerefentry><refentrytitle>ctdb</refentrytitle> + <manvolnum>1</manvolnum></citerefentry> commands + <command>statistics</command> and <command>statisticsreset</command> + for displaying statistics. + </para> + + <refsect2> + <title>Example: ctdb statistics</title> + <screen> +CTDB version 1 +Current time of statistics : Fri Sep 12 13:32:32 2014 +Statistics collected since : (000 01:49:20) Fri Sep 12 11:43:12 2014 + num_clients 6 + frozen 0 + recovering 0 + num_recoveries 2 + client_packets_sent 281293 + client_packets_recv 296317 + node_packets_sent 452387 + node_packets_recv 182394 + keepalive_packets_sent 3927 + keepalive_packets_recv 3928 + node + req_call 48605 + reply_call 1 + req_dmaster 23404 + reply_dmaster 24917 + reply_error 0 + req_message 958 + req_control 197513 + reply_control 153705 + client + req_call 130866 + req_message 770 + req_control 168921 + timeouts + call 0 + control 0 + traverse 0 + locks + num_calls 220 + num_current 0 + num_pending 0 + num_failed 0 + total_calls 130866 + pending_calls 0 + childwrite_calls 1 + pending_childwrite_calls 0 + memory_used 334490 + max_hop_count 18 + total_ro_delegations 2 + total_ro_revokes 2 + hop_count_buckets: 42816 5464 26 1 0 0 0 0 0 0 0 0 0 0 0 0 + lock_buckets: 9 165 14 15 7 2 2 0 0 0 0 0 0 0 0 0 + locks_latency MIN/AVG/MAX 0.000685/0.160302/6.369342 sec out of 214 + reclock_ctdbd MIN/AVG/MAX 0.004940/0.004969/0.004998 sec out of 2 + reclock_recd MIN/AVG/MAX 0.000000/0.000000/0.000000 sec out of 0 + call_latency MIN/AVG/MAX 0.000006/0.000719/4.562991 sec out of 126626 + childwrite_latency MIN/AVG/MAX 0.014527/0.014527/0.014527 sec out of 1 + </screen> + </refsect2> + + <refsect2> + <title>CTDB version</title> + <para> + Version of the ctdb protocol used by the node. + </para> + </refsect2> + + <refsect2> + <title>Current time of statistics</title> + <para> + Time when the statistics are generated. + </para> + <para> + This is useful when collecting statistics output periodically + for post-processing. + </para> + </refsect2> + + <refsect2> + <title>Statistics collected since</title> + <para> + Time when ctdb was started or the last time statistics was reset. + The output shows the duration and the timestamp. + </para> + </refsect2> + + <refsect2> + <title>num_clients</title> + <para> + Number of processes currently connected to CTDB's unix socket. + This includes recovery daemon, ctdb tool and samba processes + (smbd, winbindd). + </para> + </refsect2> + + <refsect2> + <title>frozen</title> + <para> + 1 if the databases are currently frozen, 0 otherwise. + </para> + </refsect2> + + <refsect2> + <title>recovering</title> + <para> + 1 if recovery is active, 0 otherwise. + </para> + </refsect2> + + <refsect2> + <title>num_recoveries</title> + <para> + Number of recoveries since the start of ctdb or since the last + statistics reset. + </para> + </refsect2> + + <refsect2> + <title>client_packets_sent</title> + <para> + Number of packets sent to client processes via unix domain socket. + </para> + </refsect2> + + <refsect2> + <title>client_packets_recv</title> + <para> + Number of packets received from client processes via unix domain socket. + </para> + </refsect2> + + <refsect2> + <title>node_packets_sent</title> + <para> + Number of packets sent to the other nodes in the cluster via TCP. + </para> + </refsect2> + + <refsect2> + <title>node_packets_recv</title> + <para> + Number of packets received from the other nodes in the cluster via TCP. + </para> + </refsect2> + + <refsect2> + <title>keepalive_packets_sent</title> + <para> + Number of keepalive messages sent to other nodes. + </para> + <para> + CTDB periodically sends keepalive messages to other nodes. + See <citetitle>KeepaliveInterval</citetitle> tunable in + <citerefentry><refentrytitle>ctdb-tunables</refentrytitle> + <manvolnum>7</manvolnum></citerefentry> for more details. + </para> + </refsect2> + + <refsect2> + <title>keepalive_packets_recv</title> + <para> + Number of keepalive messages received from other nodes. + </para> + </refsect2> + + <refsect2> + <title>node</title> + <para> + This section lists various types of messages processed which + originated from other nodes via TCP. + </para> + + <refsect3> + <title>req_call</title> + <para> + Number of REQ_CALL messages from the other nodes. + </para> + </refsect3> + + <refsect3> + <title>reply_call</title> + <para> + Number of REPLY_CALL messages from the other nodes. + </para> + </refsect3> + + <refsect3> + <title>req_dmaster</title> + <para> + Number of REQ_DMASTER messages from the other nodes. + </para> + </refsect3> + + <refsect3> + <title>reply_dmaster</title> + <para> + Number of REPLY_DMASTER messages from the other nodes. + </para> + </refsect3> + + <refsect3> + <title>reply_error</title> + <para> + Number of REPLY_ERROR messages from the other nodes. + </para> + </refsect3> + + <refsect3> + <title>req_message</title> + <para> + Number of REQ_MESSAGE messages from the other nodes. + </para> + </refsect3> + + <refsect3> + <title>req_control</title> + <para> + Number of REQ_CONTROL messages from the other nodes. + </para> + </refsect3> + + <refsect3> + <title>reply_control</title> + <para> + Number of REPLY_CONTROL messages from the other nodes. + </para> + </refsect3> + + <refsect3> + <title>req_tunnel</title> + <para> + Number of REQ_TUNNEL messages from the other nodes. + </para> + </refsect3> + + </refsect2> + + <refsect2> + <title>client</title> + <para> + This section lists various types of messages processed which + originated from clients via unix domain socket. + </para> + + <refsect3> + <title>req_call</title> + <para> + Number of REQ_CALL messages from the clients. + </para> + </refsect3> + + <refsect3> + <title>req_message</title> + <para> + Number of REQ_MESSAGE messages from the clients. + </para> + </refsect3> + + <refsect3> + <title>req_control</title> + <para> + Number of REQ_CONTROL messages from the clients. + </para> + </refsect3> + + <refsect3> + <title>req_tunnel</title> + <para> + Number of REQ_TUNNEL messages from the clients. + </para> + </refsect3> + + </refsect2> + + <refsect2> + <title>timeouts</title> + <para> + This section lists timeouts occurred when sending various messages. + </para> + + <refsect3> + <title>call</title> + <para> + Number of timeouts for REQ_CALL messages. + </para> + </refsect3> + + <refsect3> + <title>control</title> + <para> + Number of timeouts for REQ_CONTROL messages. + </para> + </refsect3> + + <refsect3> + <title>traverse</title> + <para> + Number of timeouts for database traverse operations. + </para> + </refsect3> + </refsect2> + + <refsect2> + <title>locks</title> + <para> + This section lists locking statistics. + </para> + + <refsect3> + <title>num_calls</title> + <para> + Number of completed lock calls. This includes database locks + and record locks. + </para> + </refsect3> + + <refsect3> + <title>num_current</title> + <para> + Number of scheduled lock calls. This includes database locks + and record locks. + </para> + </refsect3> + + <refsect3> + <title>num_pending</title> + <para> + Number of queued lock calls. This includes database locks and + record locks. + </para> + </refsect3> + + <refsect3> + <title>num_failed</title> + <para> + Number of failed lock calls. This includes database locks and + record locks. + </para> + </refsect3> + + </refsect2> + + <refsect2> + <title>total_calls</title> + <para> + Number of req_call messages processed from clients. This number + should be same as client --> req_call. + </para> + </refsect2> + + <refsect2> + <title>pending_calls</title> + <para> + Number of req_call messages which are currently being processed. + This number indicates the number of record migrations in flight. + </para> + </refsect2> + + <refsect2> + <title>childwrite_calls</title> + <para> + Number of record update calls. Record update calls are used to + update a record under a transaction. + </para> + </refsect2> + + <refsect2> + <title>pending_childwrite_calls</title> + <para> + Number of record update calls currently active. + </para> + </refsect2> + + <refsect2> + <title>memory_used</title> + <para> + The amount of memory in bytes currently used by CTDB using + talloc. This includes all the memory used for CTDB's internal + data structures. This does not include the memory mapped TDB + databases. + </para> + </refsect2> + + <refsect2> + <title>max_hop_count</title> + <para> + The maximum number of hops required for a record migration request + to obtain the record. High numbers indicate record contention. + </para> + </refsect2> + + <refsect2> + <title>total_ro_delegations</title> + <para> + Number of readonly delegations created. + </para> + </refsect2> + + <refsect2> + <title>total_ro_revokes</title> + <para> + Number of readonly delegations that were revoked. The difference + between total_ro_revokes and total_ro_delegations gives the + number of currently active readonly delegations. + </para> + </refsect2> + + <refsect2> + <title>hop_count_buckets</title> + <para> + Distribution of migration requests based on hop counts values. + Buckets are 0, < 2, < 4, < 8, + < 16, < 32, < 64, < 128, + < 256, < 512, < 1024, < 2048, + < 4096, < 8192, < 16384, ≥ 16384. + </para> + </refsect2> + + <refsect2> + <title>lock_buckets</title> + <para> + Distribution of record lock requests based on time required to + obtain locks. Buckets are < 1ms, < 10ms, + < 100ms, < 1s, < 2s, < 4s, + < 8s, < 16s, < 32s, < 64s, + ≥ 64s. + </para> + </refsect2> + + <refsect2> + <title>locks_latency</title> + <para> + The minimum, the average and the maximum time (in seconds) + required to obtain record locks. + </para> + </refsect2> + + <refsect2> + <title>reclock_ctdbd</title> + <para> + The minimum, the average and the maximum time (in seconds) + required to check if recovery lock is still held by recovery + daemon when recovery mode is changed. This check is done in ctdb daemon. + </para> + </refsect2> + + <refsect2> + <title>reclock_recd</title> + <para> + The minimum, the average and the maximum time (in seconds) + required to check if recovery lock is still held by recovery + daemon during recovery. This check is done in recovery daemon. + </para> + </refsect2> + + <refsect2> + <title>call_latency</title> + <para> + The minimum, the average and the maximum time (in seconds) required + to process a REQ_CALL message from client. This includes the time + required to migrate a record from remote node, if the record is + not available on the local node. + </para> + </refsect2> + + <refsect2> + <title>childwrite_latency</title> + <para>Default: 0</para> + <para> + The minimum, the average and the maximum time (in seconds) + required to update records under a transaction. + </para> + </refsect2> + </refsect1> + + <refsect1> + <title>DATABASE STATISTICS</title> + + <para> + CTDB maintains per database statistics about important operations. + See the <citerefentry><refentrytitle>ctdb</refentrytitle> + <manvolnum>1</manvolnum></citerefentry> command + <command>dbstatistics</command> for displaying database statistics. + </para> + + <refsect2> + <title>Example: ctdb dbstatistics notify_index.tdb</title> + <screen> +DB Statistics: notify_index.tdb + ro_delegations 0 + ro_revokes 0 + locks + total 131 + failed 0 + current 0 + pending 0 + hop_count_buckets: 9890 5454 26 1 0 0 0 0 0 0 0 0 0 0 0 0 + lock_buckets: 4 117 10 0 0 0 0 0 0 0 0 0 0 0 0 0 + locks_latency MIN/AVG/MAX 0.000683/0.004198/0.014730 sec out of 131 + Num Hot Keys: 3 + Count:7 Key:2f636c75737465726673 + Count:18 Key:2f636c757374657266732f64617461 + Count:7 Key:2f636c757374657266732f646174612f636c69656e7473 + </screen> + </refsect2> + + <refsect2> + <title>DB Statistics</title> + <para> + Name of the database. + </para> + </refsect2> + + <refsect2> + <title>ro_delegations</title> + <para> + Number of readonly delegations created in the database. + </para> + </refsect2> + + <refsect2> + <title>ro_revokes</title> + <para> + Number of readonly delegations revoked. The difference in + ro_delegations and ro_revokes indicates the currently active + readonly delegations. + </para> + </refsect2> + + <refsect2> + <title>locks</title> + <para> + This section lists locking statistics. + </para> + + <refsect3> + <title>total</title> + <para> + Number of completed lock calls. This includes database locks + and record locks. + </para> + </refsect3> + + <refsect3> + <title>failed</title> + <para> + Number of failed lock calls. This includes database locks and + record locks. + </para> + </refsect3> + + <refsect3> + <title>current</title> + <para> + Number of scheduled lock calls. This includes database locks + and record locks. + </para> + </refsect3> + + <refsect3> + <title>pending</title> + <para> + Number of queued lock calls. This includes database locks and + record locks. + </para> + </refsect3> + + </refsect2> + + <refsect2> + <title>hop_count_buckets</title> + <para> + Distribution of migration requests based on hop counts values. + Buckets are 0, < 2, < 4, < 8, + < 16, < 32, < 64, < 128, + < 256, < 512, < 1024, < 2048, + < 4096, < 8192, < 16384, ≥ 16384. + </para> + </refsect2> + + <refsect2> + <title>lock_buckets</title> + <para> + Distribution of record lock requests based on time required to + obtain locks. Buckets are < 1ms, < 10ms, + < 100ms, < 1s, < 2s, < 4s, + < 8s, < 16s, < 32s, < 64s, + ≥ 64s. + </para> + </refsect2> + + <refsect2> + <title>locks_latency</title> + <para> + The minimum, the average and the maximum time (in seconds) + required to obtain record locks. + </para> + </refsect2> + + <refsect2> + <title>Num Hot Keys</title> + <para> + Number of contended records determined by hop count. CTDB keeps + track of top 10 hot records and the output shows hex encoded + keys for the hot records. + </para> + </refsect2> + </refsect1> + + <refsect1> + <title>SEE ALSO</title> + <para> + <citerefentry><refentrytitle>ctdb</refentrytitle> + <manvolnum>1</manvolnum></citerefentry>, + + <citerefentry><refentrytitle>ctdbd</refentrytitle> + <manvolnum>1</manvolnum></citerefentry>, + + <citerefentry><refentrytitle>ctdb-tunables</refentrytitle> + <manvolnum>7</manvolnum></citerefentry>, + + <ulink url="http://ctdb.samba.org/"/> + </para> + </refsect1> + + <refentryinfo> + <author> + <contrib> + This documentation was written by + Amitay Isaacs, + Martin Schwenke + </contrib> + </author> + + <copyright> + <year>2007</year> + <holder>Andrew Tridgell</holder> + <holder>Ronnie Sahlberg</holder> + </copyright> + <legalnotice> + <para> + This program is free software; you can redistribute it and/or + modify it under the terms of the GNU General Public License as + published by the Free Software Foundation; either version 3 of + the License, or (at your option) any later version. + </para> + <para> + This program is distributed in the hope that it will be + useful, but WITHOUT ANY WARRANTY; without even the implied + warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR + PURPOSE. See the GNU General Public License for more details. + </para> + <para> + You should have received a copy of the GNU General Public + License along with this program; if not, see + <ulink url="http://www.gnu.org/licenses"/>. + </para> + </legalnotice> + </refentryinfo> + +</refentry> + diff --git a/ctdb/doc/ctdb-tunables.7.xml b/ctdb/doc/ctdb-tunables.7.xml new file mode 100644 index 0000000..700709e --- /dev/null +++ b/ctdb/doc/ctdb-tunables.7.xml @@ -0,0 +1,772 @@ +<?xml version="1.0" encoding="iso-8859-1"?> +<!DOCTYPE refentry + PUBLIC "-//OASIS//DTD DocBook XML V4.5//EN" + "http://www.oasis-open.org/docbook/xml/4.5/docbookx.dtd"> + +<refentry id="ctdb-tunables.7"> + + <refmeta> + <refentrytitle>ctdb-tunables</refentrytitle> + <manvolnum>7</manvolnum> + <refmiscinfo class="source">ctdb</refmiscinfo> + <refmiscinfo class="manual">CTDB - clustered TDB database</refmiscinfo> + </refmeta> + + <refnamediv> + <refname>ctdb-tunables</refname> + <refpurpose>CTDB tunable configuration variables</refpurpose> + </refnamediv> + + <refsect1> + <title>DESCRIPTION</title> + + <para> + CTDB's behaviour can be configured by setting run-time tunable + variables. This lists and describes all tunables. See the + <citerefentry><refentrytitle>ctdb</refentrytitle> + <manvolnum>1</manvolnum></citerefentry> + <command>listvars</command>, <command>setvar</command> and + <command>getvar</command> commands for more details. + </para> + + <para> + Unless otherwise stated, tunables should be set to the same + value on all nodes. Setting tunables to different values across + nodes may produce unexpected results. Future releases may set + (some or most) tunables globally across the cluster but doing so + is currently a manual process. + </para> + + <para> + Tunables can be set at startup from the + <filename>/usr/local/etc/ctdb/ctdb.tunables</filename> + configuration file. + + <literallayout> +<replaceable>TUNABLE</replaceable>=<replaceable>VALUE</replaceable> + </literallayout> + + Comment lines beginning with '#' are permitted. Whitespace may + be used for formatting/alignment. VALUE must be a non-negative + integer and must be the last thing on a line (i.e. no trailing + garbage, trailing comments are not permitted). + </para> + + <para> + For example: + + <screen format="linespecific"> +MonitorInterval=20 + </screen> + </para> + + <para> + The available tunable variables are listed alphabetically below. + </para> + + <refsect2> + <title>AllowClientDBAttach</title> + <para>Default: 1</para> + <para> + When set to 0, clients are not allowed to attach to any databases. + This can be used to temporarily block any new processes from + attaching to and accessing the databases. This is mainly used + for detaching a volatile database using 'ctdb detach'. + </para> + </refsect2> + + <refsect2> + <title>AllowMixedVersions</title> + <para>Default: 0</para> + <para> + CTDB will not allow incompatible versions to co-exist in + a cluster. If a version mismatch is found, then losing CTDB + will shutdown. To disable the incompatible version check, + set this tunable to 1. + </para> + <para> + For version checking, CTDB uses major and minor version. + For example, CTDB 4.6.1 and CTDB 4.6.2 are matching versions; + CTDB 4.5.x and CTDB 4.6.y do not match. + </para> + <para> + CTDB with version check support will lose to CTDB without + version check support. Between two different CTDB versions with + version check support, one running for less time will lose. + If the running time for both CTDB versions with version check + support is equal (to seconds), then the older version will lose. + The losing CTDB daemon will shutdown. + </para> + </refsect2> + + <refsect2> + <title>AllowUnhealthyDBRead</title> + <para>Default: 0</para> + <para> + When set to 1, ctdb allows database traverses to read unhealthy + databases. By default, ctdb does not allow reading records from + unhealthy databases. + </para> + </refsect2> + + <refsect2> + <title>ControlTimeout</title> + <para>Default: 60</para> + <para> + This is the default setting for timeout for when sending a + control message to either the local or a remote ctdb daemon. + </para> + </refsect2> + + <refsect2> + <title>DatabaseHashSize</title> + <para>Default: 100001</para> + <para> + Number of the hash chains for the local store of the tdbs that + ctdb manages. + </para> + </refsect2> + + <refsect2> + <title>DatabaseMaxDead</title> + <para>Default: 5</para> + <para> + Maximum number of dead records per hash chain for the tdb databses + managed by ctdb. + </para> + </refsect2> + + <refsect2> + <title>DBRecordCountWarn</title> + <para>Default: 100000</para> + <para> + When set to non-zero, ctdb will log a warning during recovery if + a database has more than this many records. This will produce a + warning if a database grows uncontrollably with orphaned records. + </para> + </refsect2> + + <refsect2> + <title>DBRecordSizeWarn</title> + <para>Default: 10000000</para> + <para> + When set to non-zero, ctdb will log a warning during recovery + if a single record is bigger than this size. This will produce + a warning if a database record grows uncontrollably. + </para> + </refsect2> + + <refsect2> + <title>DBSizeWarn</title> + <para>Default: 1000000000</para> + <para> + When set to non-zero, ctdb will log a warning during recovery if + a database size is bigger than this. This will produce a warning + if a database grows uncontrollably. + </para> + </refsect2> + + <refsect2> + <title>DeferredAttachTO</title> + <para>Default: 120</para> + <para> + When databases are frozen we do not allow clients to attach to + the databases. Instead of returning an error immediately to the + client, the attach request from the client is deferred until + the database becomes available again at which stage we respond + to the client. + </para> + <para> + This timeout controls how long we will defer the request from the + client before timing it out and returning an error to the client. + </para> + </refsect2> + + <refsect2> + <title>ElectionTimeout</title> + <para>Default: 3</para> + <para> + The number of seconds to wait for the election of recovery + master to complete. If the election is not completed during this + interval, then that round of election fails and ctdb starts a + new election. + </para> + </refsect2> + + <refsect2> + <title>EnableBans</title> + <para>Default: 1</para> + <para> + This parameter allows ctdb to ban a node if the node is misbehaving. + </para> + <para> + When set to 0, this disables banning completely in the cluster + and thus nodes can not get banned, even it they break. Don't + set to 0 unless you know what you are doing. + </para> + </refsect2> + + <refsect2> + <title>EventScriptTimeout</title> + <para>Default: 30</para> + <para> + Maximum time in seconds to allow an event to run before timing + out. This is the total time for all enabled scripts that are + run for an event, not just a single event script. + </para> + <para> + Note that timeouts are ignored for some events ("takeip", + "releaseip", "startrecovery", "recovered") and converted to + success. The logic here is that the callers of these events + implement their own additional timeout. + </para> + </refsect2> + + <refsect2> + <title>FetchCollapse</title> + <para>Default: 1</para> + <para> + This parameter is used to avoid multiple migration requests for + the same record from a single node. All the record requests for + the same record are queued up and processed when the record is + migrated to the current node. + </para> + <para> + When many clients across many nodes try to access the same record + at the same time this can lead to a fetch storm where the record + becomes very active and bounces between nodes very fast. This + leads to high CPU utilization of the ctdbd daemon, trying to + bounce that record around very fast, and poor performance. + This can improve performance and reduce CPU utilization for + certain workloads. + </para> + </refsect2> + + <refsect2> + <title>HopcountMakeSticky</title> + <para>Default: 50</para> + <para> + For database(s) marked STICKY (using 'ctdb setdbsticky'), + any record that is migrating so fast that hopcount + exceeds this limit is marked as STICKY record for + <varname>StickyDuration</varname> seconds. This means that + after each migration the sticky record will be kept on the node + <varname>StickyPindown</varname>milliseconds and prevented from + being migrated off the node. + </para> + <para> + This will improve performance for certain workloads, such as + locking.tdb if many clients are opening/closing the same file + concurrently. + </para> + </refsect2> + + <refsect2> + <title>IPAllocAlgorithm</title> + <para>Default: 2</para> + <para> + Selects the algorithm that CTDB should use when doing public + IP address allocation. Meaningful values are: + </para> + <variablelist> + <varlistentry> + <term>0</term> + <listitem> + <para> + Deterministic IP address allocation. + </para> + <para> + This is a simple and fast option. However, it can cause + unnecessary address movement during fail-over because + each address has a "home" node. Works badly when some + nodes do not have any addresses defined. Should be used + with care when addresses are defined across multiple + networks. + </para> + </listitem> + </varlistentry> + <varlistentry> + <term>1</term> + <listitem> + <para> + Non-deterministic IP address allocation. + </para> + <para> + This is a relatively fast option that attempts to do a + minimise unnecessary address movements. Addresses do + not have a "home" node. Rebalancing is limited but it + usually adequate. Works badly when addresses are + defined across multiple networks. + </para> + </listitem> + </varlistentry> + <varlistentry> + <term>2</term> + <listitem> + <para> + LCP2 IP address allocation. + </para> + <para> + Uses a heuristic to assign addresses defined across + multiple networks, usually balancing addresses on each + network evenly across nodes. Addresses do not have a + "home" node. Minimises unnecessary address movements. + The algorithm is complex, so is slower than other + choices for a large number of addresses. However, it + can calculate an optimal assignment of 900 addresses in + under 10 seconds on modern hardware. + </para> + </listitem> + </varlistentry> + </variablelist> + <para> + If the specified value is not one of these then the default + will be used. + </para> + </refsect2> + + <refsect2> + <title>KeepaliveInterval</title> + <para>Default: 5</para> + <para> + How often in seconds should the nodes send keep-alive packets to + each other. + </para> + </refsect2> + + <refsect2> + <title>KeepaliveLimit</title> + <para>Default: 5</para> + <para> + After how many keepalive intervals without any traffic should + a node wait until marking the peer as DISCONNECTED. + </para> + <para> + If a node has hung, it can take + <varname>KeepaliveInterval</varname> * + (<varname>KeepaliveLimit</varname> + 1) seconds before + ctdb determines that the node is DISCONNECTED and performs + a recovery. This limit should not be set too high to enable + early detection and avoid any application timeouts (e.g. SMB1) + to kick in before the fail over is completed. + </para> + </refsect2> + + <refsect2> + <title>LockProcessesPerDB</title> + <para>Default: 200</para> + <para> + This is the maximum number of lock helper processes ctdb will + create for obtaining record locks. When ctdb cannot get a record + lock without blocking, it creates a helper process that waits + for the lock to be obtained. + </para> + </refsect2> + + <refsect2> + <title>LogLatencyMs</title> + <para>Default: 0</para> + <para> + When set to non-zero, ctdb will log if certains operations + take longer than this value, in milliseconds, to complete. + These operations include "process a record request from client", + "take a record or database lock", "update a persistent database + record" and "vacuum a database". + </para> + </refsect2> + + <refsect2> + <title>MaxQueueDropMsg</title> + <para>Default: 1000000</para> + <para> + This is the maximum number of messages to be queued up for + a client before ctdb will treat the client as hung and will + terminate the client connection. + </para> + </refsect2> + + <refsect2> + <title>MonitorInterval</title> + <para>Default: 15</para> + <para> + How often should ctdb run the 'monitor' event in seconds to check + for a node's health. + </para> + </refsect2> + + <refsect2> + <title>MonitorTimeoutCount</title> + <para>Default: 20</para> + <para> + How many 'monitor' events in a row need to timeout before a node + is flagged as UNHEALTHY. This setting is useful if scripts can + not be written so that they do not hang for benign reasons. + </para> + </refsect2> + + <refsect2> + <title>NoIPFailback</title> + <para>Default: 0</para> + <para> + When set to 1, ctdb will not perform failback of IP addresses + when a node becomes healthy. When a node becomes UNHEALTHY, + ctdb WILL perform failover of public IP addresses, but when the + node becomes HEALTHY again, ctdb will not fail the addresses back. + </para> + <para> + Use with caution! Normally when a node becomes available to the + cluster ctdb will try to reassign public IP addresses onto the + new node as a way to distribute the workload evenly across the + clusternode. Ctdb tries to make sure that all running nodes have + approximately the same number of public addresses it hosts. + </para> + <para> + When you enable this tunable, ctdb will no longer attempt to + rebalance the cluster by failing IP addresses back to the new + nodes. An unbalanced cluster will therefore remain unbalanced + until there is manual intervention from the administrator. When + this parameter is set, you can manually fail public IP addresses + over to the new node(s) using the 'ctdb moveip' command. + </para> + </refsect2> + + <refsect2> + <title>NoIPTakeover</title> + <para>Default: 0</para> + <para> + When set to 1, ctdb will not allow IP addresses to be failed + over to other nodes. Any IP addresses already hosted on + healthy nodes will remain. Any IP addresses hosted on + unhealthy nodes will be released by unhealthy nodes and will + become un-hosted. + </para> + </refsect2> + + <refsect2> + <title>PullDBPreallocation</title> + <para>Default: 10*1024*1024</para> + <para> + This is the size of a record buffer to pre-allocate for sending + reply to PULLDB control. Usually record buffer starts with size + of the first record and gets reallocated every time a new record + is added to the record buffer. For a large number of records, + this can be very inefficient to grow the record buffer one record + at a time. + </para> + </refsect2> + + <refsect2> + <title>QueueBufferSize</title> + <para>Default: 1024</para> + <para> + This is the maximum amount of data (in bytes) ctdb will read + from a socket at a time. + </para> + <para> + For a busy setup, if ctdb is not able to process the TCP sockets + fast enough (large amount of data in Recv-Q for tcp sockets), + then this tunable value should be increased. However, large + values can keep ctdb busy processing packets and prevent ctdb + from handling other events. + </para> + </refsect2> + + <refsect2> + <title>RecBufferSizeLimit</title> + <para>Default: 1000000</para> + <para> + This is the limit on the size of the record buffer to be sent + in various controls. This limit is used by new controls used + for recovery and controls used in vacuuming. + </para> + </refsect2> + + <refsect2> + <title>RecdFailCount</title> + <para>Default: 10</para> + <para> + If the recovery daemon has failed to ping the main daemon for + this many consecutive intervals, the main daemon will consider + the recovery daemon as hung and will try to restart it to recover. + </para> + </refsect2> + + <refsect2> + <title>RecdPingTimeout</title> + <para>Default: 60</para> + <para> + If the main daemon has not heard a "ping" from the recovery daemon + for this many seconds, the main daemon will log a message that + the recovery daemon is potentially hung. This also increments a + counter which is checked against <varname>RecdFailCount</varname> + for detection of hung recovery daemon. + </para> + </refsect2> + + <refsect2> + <title>RecLockLatencyMs</title> + <para>Default: 1000</para> + <para> + When using a reclock file for split brain prevention, if set + to non-zero this tunable will make the recovery daemon log a + message if the fcntl() call to lock/testlock the recovery file + takes longer than this number of milliseconds. + </para> + </refsect2> + + <refsect2> + <title>RecoverInterval</title> + <para>Default: 1</para> + <para> + How frequently in seconds should the recovery daemon perform the + consistency checks to determine if it should perform a recovery. + </para> + </refsect2> + + <refsect2> + <title>RecoverTimeout</title> + <para>Default: 120</para> + <para> + This is the default setting for timeouts for controls when sent + from the recovery daemon. We allow longer control timeouts from + the recovery daemon than from normal use since the recovery + daemon often use controls that can take a lot longer than normal + controls. + </para> + </refsect2> + + <refsect2> + <title>RecoveryBanPeriod</title> + <para>Default: 300</para> + <para> + The duration in seconds for which a node is banned if the node + fails during recovery. After this time has elapsed the node will + automatically get unbanned and will attempt to rejoin the cluster. + </para> + <para> + A node usually gets banned due to real problems with the node. + Don't set this value too small. Otherwise, a problematic node + will try to re-join cluster too soon causing unnecessary recoveries. + </para> + </refsect2> + + <refsect2> + <title>RecoveryDropAllIPs</title> + <para>Default: 120</para> + <para> + If a node is stuck in recovery, or stopped, or banned, for this + many seconds, then ctdb will release all public addresses on + that node. + </para> + </refsect2> + + <refsect2> + <title>RecoveryGracePeriod</title> + <para>Default: 120</para> + <para> + During recoveries, if a node has not caused recovery failures + during the last grace period in seconds, any records of + transgressions that the node has caused recovery failures will be + forgiven. This resets the ban-counter back to zero for that node. + </para> + </refsect2> + + <refsect2> + <title>RepackLimit</title> + <para>Default: 10000</para> + <para> + During vacuuming, if the number of freelist records are more than + <varname>RepackLimit</varname>, then the database is repacked + to get rid of the freelist records to avoid fragmentation. + </para> + </refsect2> + + <refsect2> + <title>RerecoveryTimeout</title> + <para>Default: 10</para> + <para> + Once a recovery has completed, no additional recoveries are + permitted until this timeout in seconds has expired. + </para> + </refsect2> + + <refsect2> + <title>SeqnumInterval</title> + <para>Default: 1000</para> + <para> + Some databases have seqnum tracking enabled, so that samba will + be able to detect asynchronously when there has been updates + to the database. Every time a database is updated its sequence + number is increased. + </para> + <para> + This tunable is used to specify in milliseconds how frequently + ctdb will send out updates to remote nodes to inform them that + the sequence number is increased. + </para> + </refsect2> + + <refsect2> + <title>StatHistoryInterval</title> + <para>Default: 1</para> + <para> + Granularity of the statistics collected in the statistics + history. This is reported by 'ctdb stats' command. + </para> + </refsect2> + + <refsect2> + <title>StickyDuration</title> + <para>Default: 600</para> + <para> + Once a record has been marked STICKY, this is the duration in + seconds, the record will be flagged as a STICKY record. + </para> + </refsect2> + + <refsect2> + <title>StickyPindown</title> + <para>Default: 200</para> + <para> + Once a STICKY record has been migrated onto a node, it will be + pinned down on that node for this number of milliseconds. Any + request from other nodes to migrate the record off the node will + be deferred. + </para> + </refsect2> + + <refsect2> + <title>TakeoverTimeout</title> + <para>Default: 9</para> + <para> + This is the duration in seconds in which ctdb tries to complete IP + failover. + </para> + </refsect2> + + <refsect2> + <title>TickleUpdateInterval</title> + <para>Default: 20</para> + <para> + Every <varname>TickleUpdateInterval</varname> seconds, ctdb + synchronizes the client connection information across nodes. + </para> + </refsect2> + + <refsect2> + <title>TraverseTimeout</title> + <para>Default: 20</para> + <para> + This is the duration in seconds for which a database traverse + is allowed to run. If the traverse does not complete during + this interval, ctdb will abort the traverse. + </para> + </refsect2> + + <refsect2> + <title>VacuumFastPathCount</title> + <para>Default: 60</para> + <para> + During a vacuuming run, ctdb usually processes only the records + marked for deletion also called the fast path vacuuming. After + finishing <varname>VacuumFastPathCount</varname> number of fast + path vacuuming runs, ctdb will trigger a scan of complete database + for any empty records that need to be deleted. + </para> + </refsect2> + + <refsect2> + <title>VacuumInterval</title> + <para>Default: 10</para> + <para> + Periodic interval in seconds when vacuuming is triggered for + volatile databases. + </para> + </refsect2> + + <refsect2> + <title>VacuumMaxRunTime</title> + <para>Default: 120</para> + <para> + The maximum time in seconds for which the vacuuming process is + allowed to run. If vacuuming process takes longer than this + value, then the vacuuming process is terminated. + </para> + </refsect2> + + <refsect2> + <title>VerboseMemoryNames</title> + <para>Default: 0</para> + <para> + When set to non-zero, ctdb assigns verbose names for some of + the talloc allocated memory objects. These names are visible + in the talloc memory report generated by 'ctdb dumpmemory'. + </para> + </refsect2> + + </refsect1> + + <refsect1> + <title>FILES></title> + + <simplelist> + <member><filename>/usr/local/etc/ctdb/ctdb.tunables</filename></member> + </simplelist> + </refsect1> + + <refsect1> + <title>SEE ALSO</title> + <para> + <citerefentry><refentrytitle>ctdb</refentrytitle> + <manvolnum>1</manvolnum></citerefentry>, + + <citerefentry><refentrytitle>ctdbd</refentrytitle> + <manvolnum>1</manvolnum></citerefentry>, + + <citerefentry><refentrytitle>ctdb.conf</refentrytitle> + <manvolnum>5</manvolnum></citerefentry>, + + <citerefentry><refentrytitle>ctdb</refentrytitle> + <manvolnum>7</manvolnum></citerefentry>, + + <ulink url="http://ctdb.samba.org/"/> + </para> + </refsect1> + + <refentryinfo> + <author> + <contrib> + This documentation was written by + Ronnie Sahlberg, + Amitay Isaacs, + Martin Schwenke + </contrib> + </author> + + <copyright> + <year>2007</year> + <holder>Andrew Tridgell</holder> + <holder>Ronnie Sahlberg</holder> + </copyright> + <legalnotice> + <para> + This program is free software; you can redistribute it and/or + modify it under the terms of the GNU General Public License as + published by the Free Software Foundation; either version 3 of + the License, or (at your option) any later version. + </para> + <para> + This program is distributed in the hope that it will be + useful, but WITHOUT ANY WARRANTY; without even the implied + warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR + PURPOSE. See the GNU General Public License for more details. + </para> + <para> + You should have received a copy of the GNU General Public + License along with this program; if not, see + <ulink url="http://www.gnu.org/licenses"/>. + </para> + </legalnotice> + </refentryinfo> + +</refentry> diff --git a/ctdb/doc/ctdb.1.xml b/ctdb/doc/ctdb.1.xml new file mode 100644 index 0000000..6f9a176 --- /dev/null +++ b/ctdb/doc/ctdb.1.xml @@ -0,0 +1,1863 @@ +<?xml version="1.0" encoding="iso-8859-1"?> +<!DOCTYPE refentry + PUBLIC "-//OASIS//DTD DocBook XML V4.5//EN" + "http://www.oasis-open.org/docbook/xml/4.5/docbookx.dtd"> +<refentry id="ctdb.1"> + + <refentryinfo> + <author> + <contrib> + This documentation was written by + Ronnie Sahlberg, + Amitay Isaacs, + Martin Schwenke + </contrib> + </author> + + <copyright> + <year>2007</year> + <holder>Andrew Tridgell</holder> + <holder>Ronnie Sahlberg</holder> + </copyright> + <legalnotice> + <para> + This program is free software; you can redistribute it and/or + modify it under the terms of the GNU General Public License as + published by the Free Software Foundation; either version 3 of + the License, or (at your option) any later version. + </para> + <para> + This program is distributed in the hope that it will be + useful, but WITHOUT ANY WARRANTY; without even the implied + warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR + PURPOSE. See the GNU General Public License for more details. + </para> + <para> + You should have received a copy of the GNU General Public + License along with this program; if not, see + <ulink url="http://www.gnu.org/licenses"/>. + </para> + </legalnotice> + </refentryinfo> + + <refmeta> + <refentrytitle>ctdb</refentrytitle> + <manvolnum>1</manvolnum> + <refmiscinfo class="source">ctdb</refmiscinfo> + <refmiscinfo class="manual">CTDB - clustered TDB database</refmiscinfo> + </refmeta> + + <refnamediv> + <refname>ctdb</refname> + <refpurpose>CTDB management utility</refpurpose> + </refnamediv> + + <refsynopsisdiv> + <cmdsynopsis> + <command>ctdb</command> + <arg rep="repeat"><replaceable>OPTION</replaceable></arg> + <arg choice="req"><replaceable>COMMAND</replaceable></arg> + <arg choice="opt"><replaceable>COMMAND-ARGS</replaceable></arg> + </cmdsynopsis> + </refsynopsisdiv> + + <refsect1> + <title>DESCRIPTION</title> + <para> + ctdb is a utility to view and manage a CTDB cluster. + </para> + + <para> + The following terms are used when referring to nodes in a + cluster: + <variablelist> + <varlistentry> + <term>PNN</term> + <listitem> + <para> + Physical Node Number. The physical node number is an + integer that describes the node in the cluster. The + first node has physical node number 0. in a cluster. + </para> + </listitem> + </varlistentry> + <varlistentry> + <term>PNN-LIST</term> + <listitem> + <para> + This is either a single PNN, a comma-separate list of PNNs + or "all". + </para> + </listitem> + </varlistentry> + </variablelist> + </para> + + <para> + Commands that reference a database use the following terms: + <variablelist> + <varlistentry> + <term>DB</term> + <listitem> + <para> + This is either a database name, such as + <filename>locking.tdb</filename> or a database ID such + as "0x42fe72c5". + </para> + </listitem> + </varlistentry> + <varlistentry> + <term>DB-LIST</term> + <listitem> + <para> + A space separated list of at least one + <parameter>DB</parameter>. + </para> + </listitem> + </varlistentry> + </variablelist> + </para> + </refsect1> + + <refsect1> + <title>OPTIONS</title> + + <variablelist> + <varlistentry><term>-n <parameter>PNN</parameter></term> + <listitem> + <para> + The node specified by PNN should be queried for the + requested information. Default is to query the daemon + running on the local host. + </para> + </listitem> + </varlistentry> + + <varlistentry><term>-Y</term> + <listitem> + <para> + Produce output in machine readable form for easier parsing + by scripts. This uses a field delimiter of ':'. Not all + commands support this option. + </para> + </listitem> + </varlistentry> + + <varlistentry><term>-x <parameter>SEPARATOR</parameter></term> + <listitem> + <para> + Use SEPARATOR to delimit fields in machine readable output. + This implies -Y. + </para> + </listitem> + </varlistentry> + + <varlistentry><term>-X</term> + <listitem> + <para> + Produce output in machine readable form for easier parsing + by scripts. This uses a field delimiter of '|'. Not all + commands support this option. + </para> + <para> + This is equivalent to "-x|" and avoids some shell quoting + issues. + </para> + </listitem> + </varlistentry> + + <varlistentry><term>-t <parameter>TIMEOUT</parameter></term> + <listitem> + <para> + Indicates that ctdb should wait up to TIMEOUT seconds for + a response to most commands sent to the CTDB daemon. The + default is 10 seconds. + </para> + </listitem> + </varlistentry> + + <varlistentry><term>-T <parameter>TIMELIMIT</parameter></term> + <listitem> + <para> + Indicates that TIMELIMIT is the maximum run time (in + seconds) for the ctdb command. When TIMELIMIT is exceeded + the ctdb command will terminate with an error. The default + is 120 seconds. + </para> + </listitem> + </varlistentry> + + <varlistentry><term>-? --help</term> + <listitem> + <para> + Print some help text to the screen. + </para> + </listitem> + </varlistentry> + + <varlistentry><term>--usage</term> + <listitem> + <para> + Print usage information to the screen. + </para> + </listitem> + </varlistentry> + + <varlistentry><term>-d --debug=<parameter>DEBUGLEVEL</parameter></term> + <listitem> + <para> + Change the debug level for the command. Default is NOTICE. + </para> + </listitem> + </varlistentry> + </variablelist> + </refsect1> + + + <refsect1> + <title>ADMINISTRATIVE COMMANDS</title> + <para> + These are commands used to monitor and administer a CTDB cluster. + </para> + + <refsect2> + <title>pnn</title> + <para> + This command displays the PNN of the current node. + </para> + </refsect2> + + <refsect2> + <title>status</title> + <para> + This command shows the current status of all CTDB nodes based + on information from the queried node. + </para> + + <para> + Note: If the queried node is INACTIVE then the status + might not be current. + </para> + + <refsect3> + <title>Node status</title> + <para> + This includes the number of physical nodes and the status of + each node. See <citerefentry><refentrytitle>ctdb</refentrytitle> + <manvolnum>7</manvolnum></citerefentry> for information + about node states. + </para> + </refsect3> + + <refsect3> + <title>Generation</title> + <para> + The generation id is a number that indicates the current generation + of a cluster instance. Each time a cluster goes through a + reconfiguration or a recovery its generation id will be changed. + </para> + <para> + This number does not have any particular meaning other than + to keep track of when a cluster has gone through a + recovery. It is a random number that represents the current + instance of a ctdb cluster and its databases. The CTDB + daemon uses this number internally to be able to tell when + commands to operate on the cluster and the databases was + issued in a different generation of the cluster, to ensure + that commands that operate on the databases will not survive + across a cluster database recovery. After a recovery, all + old outstanding commands will automatically become invalid. + </para> + <para> + Sometimes this number will be shown as "INVALID". This only means that + the ctdbd daemon has started but it has not yet merged with the cluster through a recovery. + All nodes start with generation "INVALID" and are not assigned a real + generation id until they have successfully been merged with a cluster + through a recovery. + </para> + </refsect3> + + <refsect3> + <title>Virtual Node Number (VNN) map</title> + <para> + Consists of the number of virtual nodes and mapping from + virtual node numbers to physical node numbers. Only nodes + that are participating in the VNN map can become lmaster for + database records. + </para> + </refsect3> + + <refsect3> + <title>Recovery mode</title> + <para> + This is the current recovery mode of the cluster. There are two possible modes: + </para> + <para> + NORMAL - The cluster is fully operational. + </para> + <para> + RECOVERY - The cluster databases have all been frozen, pausing all services while the cluster awaits a recovery process to complete. A recovery process should finish within seconds. If a cluster is stuck in the RECOVERY state this would indicate a cluster malfunction which needs to be investigated. + </para> + <para> + Once the leader detects an inconsistency, for example a node + becomes disconnected/connected, the recovery daemon will trigger a + cluster recovery process, where all databases are remerged across the + cluster. When this process starts, the leader will first + "freeze" all databases to prevent applications such as samba from + accessing the databases and it will also mark the recovery mode as + RECOVERY. + </para> + <para> + When the CTDB daemon starts up, it will start in RECOVERY + mode. Once the node has been merged into a cluster and all + databases have been recovered, the node mode will change into + NORMAL mode and the databases will be "thawed", allowing samba + to access the databases again. + </para> + </refsect3> + <refsect3> + <title>Leader</title> + <para> + This is the cluster node that is currently designated as the + leader. This node is responsible of monitoring the + consistency of the cluster and to perform the actual + recovery process when reqired. + </para> + <para> + Only one node at a time can be the designated leader. Which + node is designated the leader is decided by an election + process in the recovery daemons running on each node. + </para> + </refsect3> + + <refsect3> + <title>Example</title> + <screen> +# ctdb status +Number of nodes:4 +pnn:0 192.168.2.200 OK (THIS NODE) +pnn:1 192.168.2.201 OK +pnn:2 192.168.2.202 OK +pnn:3 192.168.2.203 OK +Generation:1362079228 +Size:4 +hash:0 lmaster:0 +hash:1 lmaster:1 +hash:2 lmaster:2 +hash:3 lmaster:3 +Recovery mode:NORMAL (0) +Leader:0 + </screen> + </refsect3> + </refsect2> + + <refsect2> + <title>nodestatus <optional><parameter>PNN-LIST</parameter></optional></title> + <para> + This command is similar to the <command>status</command> + command. It displays the "node status" subset of output. The + main differences are: + </para> + + <itemizedlist> + <listitem> + <para> + The exit code is the bitwise-OR of the flags for each + specified node, while <command>ctdb status</command> exits + with 0 if it was able to retrieve status for all nodes. + </para> + </listitem> + + <listitem> + <para> + <command>ctdb status</command> provides status information + for all nodes. <command>ctdb nodestatus</command> + defaults to providing status for only the current node. + If PNN-LIST is provided then status is given for + the indicated node(s). + </para> + </listitem> + </itemizedlist> + + <para> + A common invocation in scripts is <command>ctdb nodestatus + all</command> to check whether all nodes in a cluster are + healthy. + </para> + + <refsect3> + <title>Example</title> + <screen> +# ctdb nodestatus +pnn:0 10.0.0.30 OK (THIS NODE) + +# ctdb nodestatus all +Number of nodes:2 +pnn:0 10.0.0.30 OK (THIS NODE) +pnn:1 10.0.0.31 OK + </screen> + </refsect3> + </refsect2> + + <refsect2> + <title>leader</title> + <para> + This command shows the pnn of the node which is currently the leader. + </para> + + <para> + Note: If the queried node is INACTIVE then the status + might not be current. + </para> + </refsect2> + + <refsect2> + <title>uptime</title> + <para> + This command shows the uptime for the ctdb daemon. When the last recovery or ip-failover completed and how long it took. If the "duration" is shown as a negative number, this indicates that there is a recovery/failover in progress and it started that many seconds ago. + </para> + + <refsect3> + <title>Example</title> + <screen> +# ctdb uptime +Current time of node : Thu Oct 29 10:38:54 2009 +Ctdbd start time : (000 16:54:28) Wed Oct 28 17:44:26 2009 +Time of last recovery/failover: (000 16:53:31) Wed Oct 28 17:45:23 2009 +Duration of last recovery/failover: 2.248552 seconds + </screen> + </refsect3> + </refsect2> + + <refsect2> + <title>listnodes</title> + <para> + This command shows lists the ip addresses of all the nodes in the cluster. + </para> + + <refsect3> + <title>Example</title> + <screen> +# ctdb listnodes +192.168.2.200 +192.168.2.201 +192.168.2.202 +192.168.2.203 + </screen> + </refsect3> + </refsect2> + + <refsect2> + <title>natgw {leader|list|status}</title> + <para> + This command shows different aspects of NAT gateway status. + For an overview of CTDB's NAT gateway functionality please see + the <citetitle>NAT GATEWAY</citetitle> section in + <citerefentry><refentrytitle>ctdb</refentrytitle> + <manvolnum>7</manvolnum></citerefentry>. + </para> + + <variablelist> + <varlistentry> + <term>leader</term> + <listitem> + <para> + Show the PNN and private IP address of the current NAT + gateway leader node. + </para> + <para> + Example output: + </para> + <screen> +1 192.168.2.201 + </screen> + </listitem> + </varlistentry> + <varlistentry> + <term>list</term> + <listitem> + <para> + List the private IP addresses of nodes in the current + NAT gateway group, annotating the leader node. + </para> + <para> + Example output: + </para> + <screen> +192.168.2.200 +192.168.2.201 LEADER +192.168.2.202 +192.168.2.203 + </screen> + </listitem> + </varlistentry> + <varlistentry> + <term>status</term> + <listitem> + <para> + List the nodes in the current NAT gateway group and + their status. + </para> + <para> + Example output: + </para> + <screen> +pnn:0 192.168.2.200 UNHEALTHY (THIS NODE) +pnn:1 192.168.2.201 OK +pnn:2 192.168.2.202 OK +pnn:3 192.168.2.203 OK + </screen> + </listitem> + </varlistentry> + </variablelist> + </refsect2> + + <refsect2> + <title>ping</title> + <para> + This command will "ping" specified CTDB nodes in the cluster + to verify that they are running. + </para> + <refsect3> + <title>Example</title> + <screen> +# ctdb ping +response from 0 time=0.000054 sec (3 clients) + </screen> + </refsect3> + </refsect2> + + <refsect2> + <title>ifaces</title> + <para> + This command will display the list of network interfaces, which could + host public addresses, along with their status. + </para> + <refsect3> + <title>Example</title> + <screen> +# ctdb ifaces +Interfaces on node 0 +name:eth5 link:up references:2 +name:eth4 link:down references:0 +name:eth3 link:up references:1 +name:eth2 link:up references:1 + +# ctdb -X ifaces +|Name|LinkStatus|References| +|eth5|1|2| +|eth4|0|0| +|eth3|1|1| +|eth2|1|1| + </screen> + </refsect3> + </refsect2> + + <refsect2> + <title>ip</title> + <para> + This command will display the list of public addresses that are provided by the cluster and which physical node is currently serving this ip. By default this command will ONLY show those public addresses that are known to the node itself. To see the full list of all public ips across the cluster you must use "ctdb ip all". + </para> + <refsect3> + <title>Example</title> + <screen> +# ctdb ip -v +Public IPs on node 0 +172.31.91.82 node[1] active[] available[eth2,eth3] configured[eth2,eth3] +172.31.91.83 node[0] active[eth3] available[eth2,eth3] configured[eth2,eth3] +172.31.91.84 node[1] active[] available[eth2,eth3] configured[eth2,eth3] +172.31.91.85 node[0] active[eth2] available[eth2,eth3] configured[eth2,eth3] +172.31.92.82 node[1] active[] available[eth5] configured[eth4,eth5] +172.31.92.83 node[0] active[eth5] available[eth5] configured[eth4,eth5] +172.31.92.84 node[1] active[] available[eth5] configured[eth4,eth5] +172.31.92.85 node[0] active[eth5] available[eth5] configured[eth4,eth5] + +# ctdb -X ip -v +|Public IP|Node|ActiveInterface|AvailableInterfaces|ConfiguredInterfaces| +|172.31.91.82|1||eth2,eth3|eth2,eth3| +|172.31.91.83|0|eth3|eth2,eth3|eth2,eth3| +|172.31.91.84|1||eth2,eth3|eth2,eth3| +|172.31.91.85|0|eth2|eth2,eth3|eth2,eth3| +|172.31.92.82|1||eth5|eth4,eth5| +|172.31.92.83|0|eth5|eth5|eth4,eth5| +|172.31.92.84|1||eth5|eth4,eth5| +|172.31.92.85|0|eth5|eth5|eth4,eth5| + </screen> + </refsect3> + </refsect2> + + <refsect2> + <title>ipinfo <parameter>IP</parameter></title> + <para> + This command will display details about the specified public addresses. + </para> + <refsect3> + <title>Example</title> + <screen> +# ctdb ipinfo 172.31.92.85 +Public IP[172.31.92.85] info on node 0 +IP:172.31.92.85 +CurrentNode:0 +NumInterfaces:2 +Interface[1]: Name:eth4 Link:down References:0 +Interface[2]: Name:eth5 Link:up References:2 (active) + </screen> + </refsect3> + </refsect2> + + <refsect2> + <title>event run|status|script list|script enable|script disable</title> + <para> + This command is used to control event daemon and to inspect + status of various events. + </para> + + <para> + The commands below require a component to be specified. In + the current version the only valid component is + <literal>legacy</literal>. + </para> + + <variablelist> + <varlistentry> + <term>run <parameter>TIMEOUT</parameter> <parameter>COMPONENT</parameter> <parameter>EVENT</parameter> <optional><parameter>ARGUMENTS</parameter></optional> </term> + <listitem> + <para> + This command can be used to manually run specified EVENT + in COMPONENT with optional ARGUMENTS. The event will be + allowed to run a maximum of TIMEOUT seconds. If TIMEOUT + is 0, then there is no time limit for running the event. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term>status <parameter>COMPONENT</parameter> <parameter>EVENT</parameter></term> + <listitem> + <para> + This command displays the last execution status of the + specified EVENT in COMPONENT. + </para> + <para> + The command will terminate with the exit status + corresponding to the overall status of event that is + displayed. + </para> + <para> + The output is the list of event scripts executed. + Each line shows the name, status, duration and start time + for each script. + </para> + <para> + Example + </para> + <screen> +# ctdb event status legacy monitor +00.ctdb OK 0.014 Sat Dec 17 19:39:11 2016 +01.reclock OK 0.013 Sat Dec 17 19:39:11 2016 +05.system OK 0.029 Sat Dec 17 19:39:11 2016 +06.nfs OK 0.014 Sat Dec 17 19:39:11 2016 +10.interface OK 0.037 Sat Dec 17 19:39:11 2016 +11.natgw OK 0.011 Sat Dec 17 19:39:11 2016 +11.routing OK 0.007 Sat Dec 17 19:39:11 2016 +13.per_ip_routing OK 0.007 Sat Dec 17 19:39:11 2016 +20.multipathd OK 0.007 Sat Dec 17 19:39:11 2016 +31.clamd OK 0.007 Sat Dec 17 19:39:11 2016 +40.vsftpd OK 0.013 Sat Dec 17 19:39:11 2016 +41.httpd OK 0.018 Sat Dec 17 19:39:11 2016 +49.winbind OK 0.023 Sat Dec 17 19:39:11 2016 +50.samba OK 0.100 Sat Dec 17 19:39:12 2016 +60.nfs OK 0.376 Sat Dec 17 19:39:12 2016 +70.iscsi OK 0.009 Sat Dec 17 19:39:12 2016 +91.lvs OK 0.007 Sat Dec 17 19:39:12 2016 + </screen> + </listitem> + </varlistentry> + + <varlistentry> + <term>script list <parameter>COMPONENT</parameter></term> + <listitem> + <para> + List the available event scripts in COMPONENT. Enabled + scripts are flagged with a '*'. + </para> + <para> + Generally, event scripts are provided by CTDB. However, + local or 3rd party event scripts may also be available. + These are shown in a separate section after those + provided by CTDB. + </para> + <para> + Example + </para> + <screen> +# ctdb event script list legacy +* 00.ctdb +* 01.reclock +* 05.system +* 06.nfs +* 10.interface + 11.natgw + 11.routing + 13.per_ip_routing + 20.multipathd + 31.clamd + 40.vsftpd + 41.httpd +* 49.winbind +* 50.samba +* 60.nfs + 70.iscsi + 91.lvs + +* 02.local + </screen> + </listitem> + </varlistentry> + + <varlistentry> + <term>script enable <parameter>COMPONENT</parameter> <parameter>SCRIPT</parameter></term> + <listitem> + <para> + Enable the specified event SCRIPT in COMPONENT. Only + enabled scripts will be executed when running any event. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term>script disable <parameter>COMPONENT</parameter> <parameter>SCRIPT</parameter></term> + <listitem> + <para> + Disable the specified event SCRIPT in COMPONENT. This + will prevent the script from executing when running any + event. + </para> + </listitem> + </varlistentry> + </variablelist> + </refsect2> + + <refsect2> + <title>scriptstatus</title> + <para> + This command displays which event scripts where run in the previous + monitoring cycle and the result of each script. If a script + failed with an error, causing the node to become unhealthy, + the output from that script is also shown. + </para> + <para> + This command is deprecated. It's provided for backward + compatibility. In place of <command>ctdb scriptstatus</command>, + use <command>ctdb event status</command>. + </para> + <refsect3> + <title>Example</title> + <screen> +# ctdb scriptstatus +00.ctdb OK 0.011 Sat Dec 17 19:40:46 2016 +01.reclock OK 0.010 Sat Dec 17 19:40:46 2016 +05.system OK 0.030 Sat Dec 17 19:40:46 2016 +06.nfs OK 0.014 Sat Dec 17 19:40:46 2016 +10.interface OK 0.041 Sat Dec 17 19:40:46 2016 +11.natgw OK 0.008 Sat Dec 17 19:40:46 2016 +11.routing OK 0.007 Sat Dec 17 19:40:46 2016 +13.per_ip_routing OK 0.007 Sat Dec 17 19:40:46 2016 +20.multipathd OK 0.007 Sat Dec 17 19:40:46 2016 +31.clamd OK 0.007 Sat Dec 17 19:40:46 2016 +40.vsftpd OK 0.013 Sat Dec 17 19:40:46 2016 +41.httpd OK 0.015 Sat Dec 17 19:40:46 2016 +49.winbind OK 0.022 Sat Dec 17 19:40:46 2016 +50.samba ERROR 0.077 Sat Dec 17 19:40:46 2016 + OUTPUT: ERROR: samba tcp port 445 is not responding + </screen> + </refsect3> + </refsect2> + + <refsect2> + <title>listvars</title> + <para> + List all tuneable variables, except the values of the obsolete tunables + like VacuumMinInterval. The obsolete tunables can be retrieved only + explicitly with the "ctdb getvar" command. + </para> + <refsect3> + <title>Example</title> + <screen> +# ctdb listvars +SeqnumInterval = 1000 +ControlTimeout = 60 +TraverseTimeout = 20 +KeepaliveInterval = 5 +KeepaliveLimit = 5 +RecoverTimeout = 120 +RecoverInterval = 1 +ElectionTimeout = 3 +TakeoverTimeout = 9 +MonitorInterval = 15 +TickleUpdateInterval = 20 +EventScriptTimeout = 30 +MonitorTimeoutCount = 20 +RecoveryGracePeriod = 120 +RecoveryBanPeriod = 300 +DatabaseHashSize = 100001 +DatabaseMaxDead = 5 +RerecoveryTimeout = 10 +EnableBans = 1 +NoIPFailback = 0 +VerboseMemoryNames = 0 +RecdPingTimeout = 60 +RecdFailCount = 10 +LogLatencyMs = 0 +RecLockLatencyMs = 1000 +RecoveryDropAllIPs = 120 +VacuumInterval = 10 +VacuumMaxRunTime = 120 +RepackLimit = 10000 +VacuumFastPathCount = 60 +MaxQueueDropMsg = 1000000 +AllowUnhealthyDBRead = 0 +StatHistoryInterval = 1 +DeferredAttachTO = 120 +AllowClientDBAttach = 1 +RecoverPDBBySeqNum = 1 +DeferredRebalanceOnNodeAdd = 300 +FetchCollapse = 1 +HopcountMakeSticky = 50 +StickyDuration = 600 +StickyPindown = 200 +NoIPTakeover = 0 +DBRecordCountWarn = 100000 +DBRecordSizeWarn = 10000000 +DBSizeWarn = 100000000 +PullDBPreallocation = 10485760 +LockProcessesPerDB = 200 +RecBufferSizeLimit = 1000000 +QueueBufferSize = 1024 +IPAllocAlgorithm = 2 + </screen> + </refsect3> + </refsect2> + + <refsect2> + <title>getvar <parameter>NAME</parameter></title> + <para> + Get the runtime value of a tuneable variable. + </para> + <refsect3> + <title>Example</title> + <screen> +# ctdb getvar MonitorInterval +MonitorInterval = 15 + </screen> + </refsect3> + </refsect2> + + <refsect2> + <title>setvar <parameter>NAME</parameter> <parameter>VALUE</parameter></title> + <para> + Set the runtime value of a tuneable variable. + </para> + <refsect3> + <title>Example</title> + <screen> +# ctdb setvar MonitorInterval 20 + </screen> + </refsect3> + </refsect2> + + <refsect2> + <title>lvs {leader|list|status}</title> + <para> + This command shows different aspects of LVS status. For an + overview of CTDB's LVS functionality please see the + <citetitle>LVS</citetitle> section in + <citerefentry><refentrytitle>ctdb</refentrytitle> + <manvolnum>7</manvolnum></citerefentry>. + </para> + + <variablelist> + <varlistentry> + <term>leader</term> + <listitem> + <para> + Shows the PNN of the current LVS leader node. + </para> + <para> + Example output: + </para> + <screen> +2 + </screen> + </listitem> + </varlistentry> + <varlistentry> + <term>list</term> + <listitem> + <para> + Lists the currently usable LVS nodes. + </para> + <para> + Example output: + </para> + <screen> +2 10.0.0.13 +3 10.0.0.14 + </screen> + </listitem> + </varlistentry> + <varlistentry> + <term>status</term> + <listitem> + <para> + List the nodes in the current LVS group and their status. + </para> + <para> + Example output: + </para> + <screen> +pnn:0 10.0.0.11 UNHEALTHY (THIS NODE) +pnn:1 10.0.0.12 UNHEALTHY +pnn:2 10.0.0.13 OK +pnn:3 10.0.0.14 OK + </screen> + </listitem> + </varlistentry> + </variablelist> + + </refsect2> + + + <refsect2> + <title>getcapabilities</title> + + <para> + This command shows the capabilities of the current node. See + the <citetitle>CAPABILITIES</citetitle> section in + <citerefentry><refentrytitle>ctdb</refentrytitle> + <manvolnum>7</manvolnum></citerefentry> for more details. + </para> + + <para> + Example output: + </para> + <screen> +LEADER: YES +LMASTER: YES + </screen> + + </refsect2> + + <refsect2> + <title>statistics</title> + <para> + Collect statistics from the CTDB daemon about + how many calls it has served. Information about + various fields in statistics can be found in + <citerefentry><refentrytitle>ctdb-statistics</refentrytitle> + <manvolnum>7</manvolnum></citerefentry>. + </para> + <refsect3> + <title>Example</title> + <screen> +# ctdb statistics +CTDB version 1 +Current time of statistics : Tue Mar 8 15:18:51 2016 +Statistics collected since : (003 21:31:32) Fri Mar 4 17:47:19 2016 + num_clients 9 + frozen 0 + recovering 0 + num_recoveries 2 + client_packets_sent 8170534 + client_packets_recv 7166132 + node_packets_sent 16549998 + node_packets_recv 5244418 + keepalive_packets_sent 201969 + keepalive_packets_recv 201969 + node + req_call 26 + reply_call 0 + req_dmaster 9 + reply_dmaster 12 + reply_error 0 + req_message 1339231 + req_control 8177506 + reply_control 6831284 + client + req_call 15 + req_message 334809 + req_control 6831308 + timeouts + call 0 + control 0 + traverse 0 + locks + num_calls 8 + num_current 0 + num_pending 0 + num_failed 0 + total_calls 15 + pending_calls 0 + childwrite_calls 0 + pending_childwrite_calls 0 + memory_used 394879 + max_hop_count 1 + total_ro_delegations 0 + total_ro_revokes 0 + hop_count_buckets: 8 5 0 0 0 0 0 0 0 0 0 0 0 0 0 0 + lock_buckets: 0 0 8 0 0 0 0 0 0 0 0 0 0 0 0 0 + locks_latency MIN/AVG/MAX 0.010005/0.010418/0.011010 sec out of 8 + reclock_ctdbd MIN/AVG/MAX 0.002538/0.002538/0.002538 sec out of 1 + reclock_recd MIN/AVG/MAX 0.000000/0.000000/0.000000 sec out of 0 + call_latency MIN/AVG/MAX 0.000044/0.002142/0.011702 sec out of 15 + childwrite_latency MIN/AVG/MAX 0.000000/0.000000/0.000000 sec out of 0 + </screen> + </refsect3> + </refsect2> + + <refsect2> + <title>statisticsreset</title> + <para> + This command is used to clear all statistics counters in a node. + </para> + <para> + Example: ctdb statisticsreset + </para> + </refsect2> + + <refsect2> + <title>dbstatistics <parameter>DB</parameter></title> + <para> + Display statistics about the database DB. Information + about various fields in dbstatistics can be found in + <citerefentry><refentrytitle>ctdb-statistics</refentrytitle> + <manvolnum>7</manvolnum></citerefentry>. + </para> + <refsect3> + <title>Example</title> + <screen> +# ctdb dbstatistics locking.tdb +DB Statistics: locking.tdb + ro_delegations 0 + ro_revokes 0 + locks + total 14356 + failed 0 + current 0 + pending 0 + hop_count_buckets: 28087 2 1 0 0 0 0 0 0 0 0 0 0 0 0 0 + lock_buckets: 0 14188 38 76 32 19 3 0 0 0 0 0 0 0 0 0 + locks_latency MIN/AVG/MAX 0.001066/0.012686/4.202292 sec out of 14356 + vacuum_latency MIN/AVG/MAX 0.000472/0.002207/15.243570 sec out of 224530 + Num Hot Keys: 1 + Count:8 Key:ff5bd7cb3ee3822edc1f0000000000000000000000000000 + </screen> + </refsect3> + </refsect2> + + <refsect2> + <title>getreclock</title> + <para> + Show details of the recovery lock, if any. + </para> + + <para> + Example output: + </para> + <screen> + /clusterfs/.ctdb/recovery.lock + </screen> + + </refsect2> + + <refsect2> + <title>getdebug</title> + <para> + Get the current debug level for the node. the debug level controls what information is written to the log file. + </para> + <para> + The debug levels are mapped to the corresponding syslog levels. + When a debug level is set, only those messages at that level and higher + levels will be printed. + </para> + <para> + The list of debug levels from highest to lowest are : + </para> + <para> + ERROR WARNING NOTICE INFO DEBUG + </para> + </refsect2> + + <refsect2> + <title>setdebug <parameter>DEBUGLEVEL</parameter></title> + <para> + Set the debug level of a node. This controls what information will be logged. + </para> + <para> + The debuglevel is one of ERROR WARNING NOTICE INFO DEBUG + </para> + </refsect2> + + <refsect2> + <title>getpid</title> + <para> + This command will return the process id of the ctdb daemon. + </para> + </refsect2> + + <refsect2> + <title>disable</title> + <para> + This command is used to administratively disable a node in the cluster. + A disabled node will still participate in the cluster and host + clustered TDB records but its public ip address has been taken over by + a different node and it no longer hosts any services. + </para> + </refsect2> + + <refsect2> + <title>enable</title> + <para> + Re-enable a node that has been administratively disabled. + </para> + </refsect2> + + <refsect2> + <title>stop</title> + <para> + This command is used to administratively STOP a node in the cluster. + A STOPPED node is connected to the cluster but will not host any + public ip addresse, nor does it participate in the VNNMAP. + The difference between a DISABLED node and a STOPPED node is that + a STOPPED node does not host any parts of the database which means + that a recovery is required to stop/continue nodes. + </para> + </refsect2> + + <refsect2> + <title>continue</title> + <para> + Re-start a node that has been administratively stopped. + </para> + </refsect2> + + <refsect2> + <title>addip <parameter>IPADDR</parameter>/<parameter>mask</parameter> <parameter>IFACE</parameter></title> + <para> + This command is used to add a new public ip to a node + during runtime. It should be followed by a <command>ctdb + ipreallocate</command>. This allows public addresses to be + added to a cluster without having to restart the ctdb daemons. + </para> + <para> + Note that this only updates the runtime instance of ctdb. Any + changes will be lost next time ctdb is restarted and the public + addresses file is re-read. If you want this change to be + permanent you must also update the public addresses file manually. + </para> + </refsect2> + + <refsect2> + <title>delip <parameter>IPADDR</parameter></title> + <para> + This command flags IPADDR for deletion from a node at runtime. + It should be followed by a <command>ctdb + ipreallocate</command>. If IPADDR is currently hosted by the + node it is being removed from, this ensures that the IP will + first be failed over to another node, if possible, and that it + is then actually removed. + </para> + <para> + Note that this only updates the runtime instance of CTDB. Any + changes will be lost next time CTDB is restarted and the + public addresses file is re-read. If you want this change to + be permanent you must also update the public addresses file + manually. + </para> + </refsect2> + + <refsect2> + <title>moveip <parameter>IPADDR</parameter> <parameter>PNN</parameter></title> + <para> + This command can be used to manually fail a public ip address to a + specific node. + </para> + <para> + In order to manually override the "automatic" distribution of public + ip addresses that ctdb normally provides, this command only works + when you have changed the tunables for the daemon to: + </para> + <para> + IPAllocAlgorithm != 0 + </para> + <para> + NoIPFailback = 1 + </para> + </refsect2> + + <refsect2> + <title>shutdown</title> + <para> + This command will shutdown a specific CTDB daemon. + </para> + </refsect2> + + <refsect2> + <title>setlmasterrole on|off</title> + <para> + This command is used to enable/disable the LMASTER capability for a node at runtime. This capability determines whether or not a node can be used as an LMASTER for records in the database. A node that does not have the LMASTER capability will not show up in the vnnmap. + </para> + + <para> + Nodes will by default have this capability, but it can be stripped off nodes by the setting in the sysconfig file or by using this command. + </para> + <para> + Once this setting has been enabled/disabled, you need to perform a recovery for it to take effect. + </para> + <para> + See also "ctdb getcapabilities" + </para> + </refsect2> + + <refsect2> + <title>setleaderrole on|off</title> + <para> + This command is used to enable/disable the LEADER capability + for a node at runtime. This capability determines whether or + not a node can be elected leader of the cluster. A node that + does not have the LEADER capability can not be elected + leader. If the current leader has this capability removed then + an election will occur. + </para> + + <para> + Nodes have this capability enabled by default, but it can be + removed via the <command>cluster:leader capability</command> + configuration setting or by using this command. + </para> + <para> + See also "ctdb getcapabilities" + </para> + </refsect2> + + <refsect2> + <title>reloadnodes</title> + <para> + This command is used when adding new nodes, or removing + existing nodes from an existing cluster. + </para> + <para> + Procedure to add nodes: + </para> + <orderedlist> + <listitem> + <para> + To expand an existing cluster, first ensure with + <command>ctdb status</command> that all nodes are up and + running and that they are all healthy. Do not try to + expand a cluster unless it is completely healthy! + </para> + </listitem> + <listitem> + <para> + On all nodes, edit <filename>/usr/local/etc/ctdb/nodes</filename> + and <emphasis>add the new nodes at the end of this + file</emphasis>. + </para> + </listitem> + <listitem> + <para> + Verify that all the nodes have identical + <filename>/usr/local/etc/ctdb/nodes</filename> files after adding + the new nodes. + </para> + </listitem> + <listitem> + <para> + Run <command>ctdb reloadnodes</command> to force all nodes + to reload the nodes file. + </para> + </listitem> + <listitem> + <para> + Use <command>ctdb status</command> on all nodes and verify + that they now show the additional nodes. + </para> + </listitem> + <listitem> + <para> + Install and configure the new node and bring it online. + </para> + </listitem> + </orderedlist> + <para> + Procedure to remove nodes: + </para> + <orderedlist> + <listitem> + <para> + To remove nodes from an existing cluster, first ensure + with <command>ctdb status</command> that all nodes, except + the node to be deleted, are up and running and that they + are all healthy. Do not try to remove nodes from a + cluster unless the cluster is completely healthy! + </para> + </listitem> + <listitem> + <para> + Shutdown and power off the node to be removed. + </para> + </listitem> + <listitem> + <para> + On all other nodes, edit the + <filename>/usr/local/etc/ctdb/nodes</filename> file and + <emphasis>comment out</emphasis> the nodes to be removed. + <emphasis>Do not delete the lines for the deleted + nodes</emphasis>, just comment them out by adding a '#' at + the beginning of the lines. + </para> + </listitem> + <listitem> + <para> + Run <command>ctdb reloadnodes</command> to force all nodes + to reload the nodes file. + </para> + </listitem> + <listitem> + <para> + Use <command>ctdb status</command> on all nodes and verify + that the deleted nodes are no longer listed. + </para> + </listitem> + </orderedlist> + + </refsect2> + + <refsect2> + <title> + reloadips + <optional><parameter>PNN-LIST</parameter></optional> + </title> + <para> + This command reloads the public addresses configuration file + on the specified nodes. When it completes addresses will be + reconfigured and reassigned across the cluster as necessary. + </para> + + <para> + This command is currently unable to make changes to the + netmask or interfaces associated with existing addresses. + Such changes must be made in 2 steps by deleting addresses in + question and re-adding then. Unfortunately this will disrupt + connections to the changed addresses. + </para> + </refsect2> + + <refsect2> + <title>getdbmap</title> + <para> + This command lists all clustered TDB databases that the CTDB + daemon has attached to. Some databases are flagged as PERSISTENT, + this means that the database stores data persistently and the + data will remain across reboots. One example of such a database + is secrets.tdb where information about how the cluster was joined + to the domain is stored. Some database are flagged as REPLICATED, + this means that the data in that database is replicated across all + the nodes. But the data will not remain across reboots. This + type of database is used by CTDB to store it's internal state. + </para> + <para> + If a PERSISTENT database is not in a healthy state the database + is flagged as UNHEALTHY. If there's at least one completely + healthy node running in the cluster, it's possible that the + content is restored by a recovery run automatically. Otherwise an + administrator needs to analyze the problem. + </para> + <para> + See also "ctdb getdbstatus", "ctdb backupdb", "ctdb restoredb", + "ctdb dumpbackup", "ctdb wipedb", "ctdb setvar AllowUnhealthyDBRead 1" + and (if samba or tdb-utils are installed) "tdbtool check". + </para> + <para> + Most databases are not persistent and only store the state + information that the currently running samba daemons need. These + databases are always wiped when ctdb/samba starts and when a + node is rebooted. + </para> + + <refsect3> + <title>Example</title> + <screen> +# ctdb getdbmap +Number of databases:10 +dbid:0x435d3410 name:notify.tdb path:/usr/local/var/lib/ctdb/notify.tdb.0 +dbid:0x42fe72c5 name:locking.tdb path:/usr/local/var/lib/ctdb/locking.tdb.0 +dbid:0x1421fb78 name:brlock.tdb path:/usr/local/var/lib/ctdb/brlock.tdb.0 +dbid:0x17055d90 name:connections.tdb path:/usr/local/var/lib/ctdb/connections.tdb.0 +dbid:0xc0bdde6a name:sessionid.tdb path:/usr/local/var/lib/ctdb/sessionid.tdb.0 +dbid:0x122224da name:test.tdb path:/usr/local/var/lib/ctdb/test.tdb.0 +dbid:0x2672a57f name:idmap2.tdb path:/usr/local/var/lib/ctdb/persistent/idmap2.tdb.0 PERSISTENT +dbid:0xb775fff6 name:secrets.tdb path:/usr/local/var/lib/ctdb/persistent/secrets.tdb.0 PERSISTENT +dbid:0xe98e08b6 name:group_mapping.tdb path:/usr/local/var/lib/ctdb/persistent/group_mapping.tdb.0 PERSISTENT +dbid:0x7bbbd26c name:passdb.tdb path:/usr/local/var/lib/ctdb/persistent/passdb.tdb.0 PERSISTENT + +# ctdb getdbmap # example for unhealthy database +Number of databases:1 +dbid:0xb775fff6 name:secrets.tdb path:/usr/local/var/lib/ctdb/persistent/secrets.tdb.0 PERSISTENT UNHEALTHY + +# ctdb -X getdbmap +|ID|Name|Path|Persistent|Unhealthy| +|0x7bbbd26c|passdb.tdb|/usr/local/var/lib/ctdb/persistent/passdb.tdb.0|1|0| + </screen> + </refsect3> + </refsect2> + + <refsect2> + <title> + backupdb + <parameter>DB</parameter> + <parameter>FILE</parameter> + </title> + <para> + Copy the contents of database DB to FILE. FILE can later be + read back using <command>restoredb</command>. This is mainly + useful for backing up persistent databases such as + <filename>secrets.tdb</filename> and similar. + </para> + </refsect2> + + <refsect2> + <title> + restoredb + <parameter>FILE</parameter> + <optional><parameter>DB</parameter></optional> + </title> + <para> + This command restores a persistent database that was + previously backed up using backupdb. By default the data will + be restored back into the same database as it was created + from. By specifying dbname you can restore the data into a + different database. + </para> + </refsect2> + + <refsect2> + <title>setdbreadonly <parameter>DB</parameter></title> + <para> + This command will enable the read-only record support for a + database. This is an experimental feature to improve + performance for contended records primarily in locking.tdb and + brlock.tdb. When enabling this feature you must set it on all + nodes in the cluster. + </para> + </refsect2> + + <refsect2> + <title>setdbsticky <parameter>DB</parameter></title> + <para> + This command will enable the sticky record support for the + specified database. This is an experimental feature to + improve performance for contended records primarily in + locking.tdb and brlock.tdb. When enabling this feature you + must set it on all nodes in the cluster. + </para> + </refsect2> + + </refsect1> + + <refsect1> + <title>INTERNAL COMMANDS</title> + + <para> + Internal commands are used by CTDB's scripts and are not + required for managing a CTDB cluster. Their parameters and + behaviour are subject to change. + </para> + + <refsect2> + <title>gettickles <parameter>IPADDR</parameter></title> + <para> + Show TCP connections that are registered with CTDB to be + "tickled" if there is a failover. + </para> + </refsect2> + + <refsect2> + <title>gratarp <parameter>IPADDR</parameter> <parameter>INTERFACE</parameter></title> + <para> + Send out a gratuitous ARP for the specified interface through + the specified interface. This command is mainly used by the + ctdb eventscripts. + </para> + </refsect2> + + <refsect2> + <title> + pdelete <parameter>DB</parameter> <parameter>KEY</parameter> + </title> + <para> + Delete KEY from DB. + </para> + </refsect2> + + <refsect2> + <title> + pfetch <parameter>DB</parameter> <parameter>KEY</parameter> + </title> + <para> + Print the value associated with KEY in DB. + </para> + </refsect2> + + <refsect2> + <title> + pstore + <parameter>DB</parameter> + <parameter>KEY</parameter> + <parameter>FILE</parameter> + </title> + <para> + Store KEY in DB with contents of FILE as the associated value. + </para> + </refsect2> + + <refsect2> + <title> + ptrans + <parameter>DB</parameter> + <optional><parameter>FILE</parameter></optional> + </title> + <para> + Read a list of key-value pairs, one per line from FILE, and + store them in DB using a single transaction. An empty value + is equivalent to deleting the given key. + </para> + <para> + The key and value should be separated by spaces or tabs. Each + key/value should be a printable string enclosed in + double-quotes. + </para> + </refsect2> + + <refsect2> + <title>runstate [setup|first_recovery|startup|running]</title> + <para> + Print the runstate of the specified node. Runstates are used + to serialise important state transitions in CTDB, particularly + during startup. + </para> + <para> + If one or more optional runstate arguments are specified then + the node must be in one of these runstates for the command to + succeed. + </para> + <refsect3> + <title>Example</title> + <screen> +# ctdb runstate +RUNNING + </screen> + </refsect3> + </refsect2> + + <refsect2> + <title>setifacelink <parameter>IFACE</parameter> up|down</title> + <para> + Set the internal state of network interface IFACE. This is + typically used in the <filename>10.interface</filename> script + in the "monitor" event. + </para> + <para> + Example: ctdb setifacelink eth0 up + </para> + </refsect2> + + <refsect2> + <title>tickle</title> + <para> + Read a list of TCP connections, one per line, from standard + input and send a TCP tickle to the source host for each + connection. A connection is specified as: + </para> + <synopsis> + <parameter>SRC-IPADDR</parameter>:<parameter>SRC-PORT</parameter> <parameter>DST-IPADDR</parameter>:<parameter>DST-PORT</parameter> + </synopsis> + <para> + A single connection can be specified on the command-line + rather than on standard input. + </para> + <para> + A TCP tickle is a TCP ACK packet with an invalid sequence and + acknowledge number and will when received by the source host + result in it sending an immediate correct ACK back to the + other end. + </para> + <para> + TCP tickles are useful to "tickle" clients after a IP failover has + occurred since this will make the client immediately recognize the + TCP connection has been disrupted and that the client will need + to reestablish. This greatly speeds up the time it takes for a client + to detect and reestablish after an IP failover in the ctdb cluster. + </para> + </refsect2> + + <refsect2> + <title>version</title> + <para> + Display the CTDB version. + </para> + </refsect2> + + </refsect1> + + <refsect1> + <title>DEBUGGING COMMANDS</title> + <para> + These commands are primarily used for CTDB development and testing and + should not be used for normal administration. + </para> + + + <refsect2> + <title>OPTIONS</title> + + <variablelist> + <varlistentry><term>--print-emptyrecords</term> + <listitem> + <para> + This enables printing of empty records when dumping databases + with the catdb, cattbd and dumpdbbackup commands. Records with + empty data segment are considered deleted by ctdb and cleaned + by the vacuuming mechanism, so this switch can come in handy for + debugging the vacuuming behaviour. + </para> + </listitem> + </varlistentry> + + <varlistentry><term>--print-datasize</term> + <listitem> + <para> + This lets database dumps (catdb, cattdb, dumpdbbackup) print the + size of the record data instead of dumping the data contents. + </para> + </listitem> + </varlistentry> + + <varlistentry><term>--print-lmaster</term> + <listitem> + <para> + This lets catdb print the lmaster for each record. + </para> + </listitem> + </varlistentry> + + <varlistentry><term>--print-hash</term> + <listitem> + <para> + This lets database dumps (catdb, cattdb, dumpdbbackup) print the + hash for each record. + </para> + </listitem> + </varlistentry> + + <varlistentry><term>--print-recordflags</term> + <listitem> + <para> + This lets catdb and dumpdbbackup print the + record flags for each record. Note that cattdb always + prints the flags. + </para> + </listitem> + </varlistentry> + + </variablelist> + </refsect2> + + <refsect2> + <title>process-exists <parameter>PID</parameter> <parameter>[SRVID]</parameter></title> + <para> + This command checks if a specific process exists on the CTDB + host. This is mainly used by Samba to check if remote instances + of samba are still running or not. When the optional SRVID + argument is specified, the command check if a specific process + exists on the CTDB host and has registered for specified SRVID. + </para> + </refsect2> + + <refsect2> + <title>getdbstatus <parameter>DB</parameter></title> + <para> + This command displays more details about a database. + </para> + <refsect3> + <title>Example</title> + <screen> +# ctdb getdbstatus test.tdb.0 +dbid: 0x122224da +name: test.tdb +path: /usr/local/var/lib/ctdb/test.tdb.0 +PERSISTENT: no +HEALTH: OK + +# ctdb getdbstatus registry.tdb # with a corrupted TDB +dbid: 0xf2a58948 +name: registry.tdb +path: /usr/local/var/lib/ctdb/persistent/registry.tdb.0 +PERSISTENT: yes +HEALTH: NO-HEALTHY-NODES - ERROR - Backup of corrupted TDB in '/usr/local/var/lib/ctdb/persistent/registry.tdb.0.corrupted.20091208091949.0Z' + </screen> + </refsect3> + </refsect2> + + <refsect2> + <title>catdb <parameter>DB</parameter></title> + <para> + Print a dump of the clustered TDB database DB. + </para> + </refsect2> + + <refsect2> + <title>cattdb <parameter>DB</parameter></title> + <para> + Print a dump of the contents of the local TDB database DB. + </para> + </refsect2> + + <refsect2> + <title>dumpdbbackup <parameter>FILE</parameter></title> + <para> + Print a dump of the contents from database backup FILE, + similar to <command>catdb</command>. + </para> + </refsect2> + + <refsect2> + <title>wipedb <parameter>DB</parameter></title> + <para> + Remove all contents of database DB. + </para> + </refsect2> + + <refsect2> + <title>recover</title> + <para> + This command will trigger the recovery daemon to do a cluster + recovery. + </para> + </refsect2> + + <refsect2> + <title>ipreallocate, sync</title> + <para> + This command will force the leader to perform a full ip + reallocation process and redistribute all ip addresses. This + is useful to "reset" the allocations back to its default state + if they have been changed using the "moveip" command. While a + "recover" will also perform this reallocation, a recovery is + much more hevyweight since it will also rebuild all the + databases. + </para> + </refsect2> + + <refsect2> + <title>attach <parameter>DBNAME</parameter> [persistent|replicated]</title> + <para> + Create a new CTDB database called DBNAME and attach to it on + all nodes. + </para> + </refsect2> + + <refsect2> + <title>detach <parameter>DB-LIST</parameter></title> + <para> + Detach specified non-persistent database(s) from the cluster. This + command will disconnect specified database(s) on all nodes in + the cluster. This command should only be used when none of the + specified database(s) are in use. + </para> + <para> + All nodes should be active and tunable AllowClientDBAccess should + be disabled on all nodes before detaching databases. + </para> + </refsect2> + + <refsect2> + <title>dumpmemory</title> + <para> + This is a debugging command. This command will make the ctdb + daemon to write a fill memory allocation map to standard output. + </para> + </refsect2> + + <refsect2> + <title>rddumpmemory</title> + <para> + This is a debugging command. This command will dump the talloc memory + allocation tree for the recovery daemon to standard output. + </para> + </refsect2> + + <refsect2> + <title>ban <parameter>BANTIME</parameter></title> + <para> + Administratively ban a node for BANTIME seconds. The node + will be unbanned after BANTIME seconds have elapsed. + </para> + <para> + A banned node does not participate in the cluster. It does + not host any records for the clustered TDB and does not host + any public IP addresses. + </para> + <para> + Nodes are automatically banned if they misbehave. For + example, a node may be banned if it causes too many cluster + recoveries. + </para> + <para> + To administratively exclude a node from a cluster use the + <command>stop</command> command. + </para> + </refsect2> + + <refsect2> + <title>unban</title> + <para> + This command is used to unban a node that has either been + administratively banned using the ban command or has been + automatically banned. + </para> + </refsect2> + + </refsect1> + + <!-- UNDOCUMENTED: stats + addtickle deltickle + tfetch tstore readkey writekey + checktcpport getdbseqnum + --> + + <refsect1> + <title>SEE ALSO</title> + <para> + <citerefentry><refentrytitle>ctdbd</refentrytitle> + <manvolnum>1</manvolnum></citerefentry>, + + <citerefentry><refentrytitle>onnode</refentrytitle> + <manvolnum>1</manvolnum></citerefentry>, + + <citerefentry><refentrytitle>ctdb</refentrytitle> + <manvolnum>7</manvolnum></citerefentry>, + + <citerefentry><refentrytitle>ctdb-statistics</refentrytitle> + <manvolnum>7</manvolnum></citerefentry>, + + <citerefentry><refentrytitle>ctdb-tunables</refentrytitle> + <manvolnum>7</manvolnum></citerefentry>, + + <ulink url="http://ctdb.samba.org/"/> + </para> + </refsect1> + + +</refentry> diff --git a/ctdb/doc/ctdb.7.xml b/ctdb/doc/ctdb.7.xml new file mode 100644 index 0000000..351f3e8 --- /dev/null +++ b/ctdb/doc/ctdb.7.xml @@ -0,0 +1,1182 @@ +<?xml version="1.0" encoding="iso-8859-1"?> +<!DOCTYPE refentry + PUBLIC "-//OASIS//DTD DocBook XML V4.5//EN" + "http://www.oasis-open.org/docbook/xml/4.5/docbookx.dtd"> +<refentry id="ctdb.7"> + +<refmeta> + <refentrytitle>ctdb</refentrytitle> + <manvolnum>7</manvolnum> + <refmiscinfo class="source">ctdb</refmiscinfo> + <refmiscinfo class="manual">CTDB - clustered TDB database</refmiscinfo> +</refmeta> + + +<refnamediv> + <refname>ctdb</refname> + <refpurpose>Clustered TDB</refpurpose> +</refnamediv> + +<refsect1> + <title>DESCRIPTION</title> + + <para> + CTDB is a clustered database component in clustered Samba that + provides a high-availability load-sharing CIFS server cluster. + </para> + + <para> + The main functions of CTDB are: + </para> + + <itemizedlist> + <listitem> + <para> + Provide a clustered version of the TDB database with automatic + rebuild/recovery of the databases upon node failures. + </para> + </listitem> + + <listitem> + <para> + Monitor nodes in the cluster and services running on each node. + </para> + </listitem> + + <listitem> + <para> + Manage a pool of public IP addresses that are used to provide + services to clients. Alternatively, CTDB can be used with + LVS. + </para> + </listitem> + </itemizedlist> + + <para> + Combined with a cluster filesystem CTDB provides a full + high-availablity (HA) environment for services such as clustered + Samba, NFS and other services. + </para> + + <para> + In addition to the CTDB manual pages there is much more + information available at + <ulink url="https://wiki.samba.org/index.php/CTDB_and_Clustered_Samba"/>. + </para> +</refsect1> + +<refsect1> + <title>ANATOMY OF A CTDB CLUSTER</title> + + <para> + A CTDB cluster is a collection of nodes with 2 or more network + interfaces. All nodes provide network (usually file/NAS) services + to clients. Data served by file services is stored on shared + storage (usually a cluster filesystem) that is accessible by all + nodes. + </para> + <para> + CTDB provides an "all active" cluster, where services are load + balanced across all nodes. + </para> +</refsect1> + + <refsect1> + <title>Cluster leader</title> + + <para> + CTDB uses a <emphasis>cluster leader and follower</emphasis> + model of cluster management. All nodes in a cluster elect one + node to be the leader. The leader node coordinates privileged + operations such as database recovery and IP address failover. + </para> + + <para> + CTDB previously referred to the leader as the <emphasis>recovery + master</emphasis> or <emphasis>recmaster</emphasis>. References + to these terms may still be found in documentation and code. + </para> + </refsect1> + + <refsect1> + <title>Cluster Lock</title> + + <para> + CTDB uses a cluster lock to assert its privileged role in the + cluster. This node takes the cluster lock when it becomes + leader and holds the lock until it is no longer leader. The + <emphasis>cluster lock</emphasis> helps CTDB to avoid a + <emphasis>split brain</emphasis>, where a cluster becomes + partitioned and each partition attempts to operate + independently. Issues that can result from a split brain + include file data corruption, because file locking metadata may + not be tracked correctly. + </para> + + <para> + CTDB previously referred to the cluster lock as the + <emphasis>recovery lock</emphasis>. The abbreviation + <emphasis>reclock</emphasis> is still used - just "clock" would + be confusing. + </para> + + <para> + <emphasis>CTDB is unable configure a default cluster + lock</emphasis>, because this would depend on factors such as + cluster filesystem mountpoints. However, <emphasis>running CTDB + without a cluster lock is not recommended</emphasis> as there + will be no split brain protection. + </para> + + <para> + When a cluster lock is configured it is used as the election + mechanism. Nodes race to take the cluster lock and the winner + is the cluster leader. This avoids problems when a node wins an + election but is unable to take the lock - this can occur if a + cluster becomes partitioned (for example, due to a communication + failure) and a different leader is elected by the nodes in each + partition, or if the cluster filesystem has a high failover + latency. + </para> + + <para> + By default, the cluster lock is implemented using a file + (specified by <parameter>cluster lock</parameter> in the + <literal>[cluster]</literal> section of + <citerefentry><refentrytitle>ctdb.conf</refentrytitle> + <manvolnum>5</manvolnum></citerefentry>) residing in shared + storage (usually) on a cluster filesystem. To support a + cluster lock the cluster filesystem must support lock + coherence. See + <citerefentry><refentrytitle>ping_pong</refentrytitle> + <manvolnum>1</manvolnum></citerefentry> for more details. + </para> + + <para> + The cluster lock can also be implemented using an arbitrary + cluster mutex helper (or call-out). This is indicated by using + an exclamation point ('!') as the first character of the + <parameter>cluster lock</parameter> parameter. For example, a + value of <command>!/usr/local/bin/myhelper cluster</command> + would run the given helper with the specified arguments. The + helper will continue to run as long as it holds its mutex. See + <filename>ctdb/doc/cluster_mutex_helper.txt</filename> in the + source tree, and related code, for clues about writing helpers. + </para> + + <para> + When a file is specified for the <parameter>cluster + lock</parameter> parameter (i.e. no leading '!') the file lock + is implemented by a default helper + (<command>/usr/local/libexec/ctdb/ctdb_mutex_fcntl_helper</command>). + This helper has arguments as follows: + + <!-- cmdsynopsis would not require long line but does not work :-( --> + <synopsis> +<command>ctdb_mutex_fcntl_helper</command> <parameter>FILE</parameter> <optional><parameter>RECHECK-INTERVAL</parameter></optional> + </synopsis> + + <command>ctdb_mutex_fcntl_helper</command> will take a lock on + FILE and then check every RECHECK-INTERVAL seconds to ensure + that FILE still exists and that its inode number is unchanged + from when the lock was taken. The default value for + RECHECK-INTERVAL is 5. + </para> + + <para> + CTDB does sanity checks to ensure that the cluster lock is held + as expected. + </para> + </refsect1> + + <refsect1> + <title>Private vs Public addresses</title> + + <para> + Each node in a CTDB cluster has multiple IP addresses assigned + to it: + + <itemizedlist> + <listitem> + <para> + A single private IP address that is used for communication + between nodes. + </para> + </listitem> + <listitem> + <para> + One or more public IP addresses that are used to provide + NAS or other services. + </para> + </listitem> + </itemizedlist> + </para> + + <refsect2> + <title>Private address</title> + + <para> + Each node is configured with a unique, permanently assigned + private address. This address is configured by the operating + system. This address uniquely identifies a physical node in + the cluster and is the address that CTDB daemons will use to + communicate with the CTDB daemons on other nodes. + </para> + + <para> + Private addresses are listed in the file + <filename>/usr/local/etc/ctdb/nodes</filename>). This file + contains the list of private addresses for all nodes in the + cluster, one per line. This file must be the same on all nodes + in the cluster. + </para> + + <para> + Some users like to put this configuration file in their + cluster filesystem. A symbolic link should be used in this + case. + </para> + + <para> + Private addresses should not be used by clients to connect to + services provided by the cluster. + </para> + <para> + It is strongly recommended that the private addresses are + configured on a private network that is separate from client + networks. This is because the CTDB protocol is both + unauthenticated and unencrypted. If clients share the private + network then steps need to be taken to stop injection of + packets to relevant ports on the private addresses. It is + also likely that CTDB protocol traffic between nodes could + leak sensitive information if it can be intercepted. + </para> + + <para> + Example <filename>/usr/local/etc/ctdb/nodes</filename> for a four node + cluster: + </para> + <screen format="linespecific"> +192.168.1.1 +192.168.1.2 +192.168.1.3 +192.168.1.4 + </screen> + </refsect2> + + <refsect2> + <title>Public addresses</title> + + <para> + Public addresses are used to provide services to clients. + Public addresses are not configured at the operating system + level and are not permanently associated with a particular + node. Instead, they are managed by CTDB and are assigned to + interfaces on physical nodes at runtime. + </para> + <para> + The CTDB cluster will assign/reassign these public addresses + across the available healthy nodes in the cluster. When one + node fails, its public addresses will be taken over by one or + more other nodes in the cluster. This ensures that services + provided by all public addresses are always available to + clients, as long as there are nodes available capable of + hosting this address. + </para> + + <para> + The public address configuration is stored in + <filename>/usr/local/etc/ctdb/public_addresses</filename> on + each node. This file contains a list of the public addresses + that the node is capable of hosting, one per line. Each entry + also contains the netmask and the interface to which the + address should be assigned. If this file is missing then no + public addresses are configured. + </para> + + <para> + Some users who have the same public addresses on all nodes + like to put this configuration file in their cluster + filesystem. A symbolic link should be used in this case. + </para> + + <para> + Example <filename>/usr/local/etc/ctdb/public_addresses</filename> for a + node that can host 4 public addresses, on 2 different + interfaces: + </para> + <screen format="linespecific"> +10.1.1.1/24 eth1 +10.1.1.2/24 eth1 +10.1.2.1/24 eth2 +10.1.2.2/24 eth2 + </screen> + + <para> + In many cases the public addresses file will be the same on + all nodes. However, it is possible to use different public + address configurations on different nodes. + </para> + + <para> + Example: 4 nodes partitioned into two subgroups: + </para> + <screen format="linespecific"> +Node 0:/usr/local/etc/ctdb/public_addresses + 10.1.1.1/24 eth1 + 10.1.1.2/24 eth1 + +Node 1:/usr/local/etc/ctdb/public_addresses + 10.1.1.1/24 eth1 + 10.1.1.2/24 eth1 + +Node 2:/usr/local/etc/ctdb/public_addresses + 10.1.2.1/24 eth2 + 10.1.2.2/24 eth2 + +Node 3:/usr/local/etc/ctdb/public_addresses + 10.1.2.1/24 eth2 + 10.1.2.2/24 eth2 + </screen> + <para> + In this example nodes 0 and 1 host two public addresses on the + 10.1.1.x network while nodes 2 and 3 host two public addresses + for the 10.1.2.x network. + </para> + <para> + Public address 10.1.1.1 can be hosted by either of nodes 0 or + 1 and will be available to clients as long as at least one of + these two nodes are available. + </para> + <para> + If both nodes 0 and 1 become unavailable then public address + 10.1.1.1 also becomes unavailable. 10.1.1.1 can not be failed + over to nodes 2 or 3 since these nodes do not have this public + address configured. + </para> + <para> + The <command>ctdb ip</command> command can be used to view the + current assignment of public addresses to physical nodes. + </para> + </refsect2> + </refsect1> + + + <refsect1> + <title>Node status</title> + + <para> + The current status of each node in the cluster can be viewed by the + <command>ctdb status</command> command. + </para> + + <para> + A node can be in one of the following states: + </para> + + <variablelist> + <varlistentry> + <term>OK</term> + <listitem> + <para> + This node is healthy and fully functional. It hosts public + addresses to provide services. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term>DISCONNECTED</term> + <listitem> + <para> + This node is not reachable by other nodes via the private + network. It is not currently participating in the cluster. + It <emphasis>does not</emphasis> host public addresses to + provide services. It might be shut down. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term>DISABLED</term> + <listitem> + <para> + This node has been administratively disabled. This node is + partially functional and participates in the cluster. + However, it <emphasis>does not</emphasis> host public + addresses to provide services. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term>UNHEALTHY</term> + <listitem> + <para> + A service provided by this node has failed a health check + and should be investigated. This node is partially + functional and participates in the cluster. However, it + <emphasis>does not</emphasis> host public addresses to + provide services. Unhealthy nodes should be investigated + and may require an administrative action to rectify. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term>BANNED</term> + <listitem> + <para> + CTDB is not behaving as designed on this node. For example, + it may have failed too many recovery attempts. Such nodes + are banned from participating in the cluster for a + configurable time period before they attempt to rejoin the + cluster. A banned node <emphasis>does not</emphasis> host + public addresses to provide services. All banned nodes + should be investigated and may require an administrative + action to rectify. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term>STOPPED</term> + <listitem> + <para> + This node has been administratively exclude from the + cluster. A stopped node does no participate in the cluster + and <emphasis>does not</emphasis> host public addresses to + provide services. This state can be used while performing + maintenance on a node. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term>PARTIALLYONLINE</term> + <listitem> + <para> + A node that is partially online participates in a cluster + like a healthy (OK) node. Some interfaces to serve public + addresses are down, but at least one interface is up. See + also <command>ctdb ifaces</command>. + </para> + </listitem> + </varlistentry> + + </variablelist> + </refsect1> + + <refsect1> + <title>CAPABILITIES</title> + + <para> + Cluster nodes can have several different capabilities enabled. + These are listed below. + </para> + + <variablelist> + + <varlistentry> + <term>LEADER</term> + <listitem> + <para> + Indicates that a node can become the CTDB cluster leader. + The current leader is decided via an + election held by all active nodes with this capability. + </para> + <para> + Default is YES. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term>LMASTER</term> + <listitem> + <para> + Indicates that a node can be the location master (LMASTER) + for database records. The LMASTER always knows which node + has the latest copy of a record in a volatile database. + </para> + <para> + Default is YES. + </para> + </listitem> + </varlistentry> + + </variablelist> + + <para> + The LEADER and LMASTER capabilities can be disabled when CTDB + is used to create a cluster spanning across WAN links. In this + case CTDB acts as a WAN accelerator. + </para> + + </refsect1> + + <refsect1> + <title>LVS</title> + + <para> + LVS is a mode where CTDB presents one single IP address for the + entire cluster. This is an alternative to using public IP + addresses and round-robin DNS to loadbalance clients across the + cluster. + </para> + + <para> + This is similar to using a layer-4 loadbalancing switch but with + some restrictions. + </para> + + <para> + One extra LVS public address is assigned on the public network + to each LVS group. Each LVS group is a set of nodes in the + cluster that presents the same LVS address public address to the + outside world. Normally there would only be one LVS group + spanning an entire cluster, but in situations where one CTDB + cluster spans multiple physical sites it might be useful to have + one LVS group for each site. There can be multiple LVS groups + in a cluster but each node can only be member of one LVS group. + </para> + + <para> + Client access to the cluster is load-balanced across the HEALTHY + nodes in an LVS group. If no HEALTHY nodes exists then all + nodes in the group are used, regardless of health status. CTDB + will, however never load-balance LVS traffic to nodes that are + BANNED, STOPPED, DISABLED or DISCONNECTED. The <command>ctdb + lvs</command> command is used to show which nodes are currently + load-balanced across. + </para> + + <para> + In each LVS group, one of the nodes is selected by CTDB to be + the LVS leader. This node receives all traffic from clients + coming in to the LVS public address and multiplexes it across + the internal network to one of the nodes that LVS is using. + When responding to the client, that node will send the data back + directly to the client, bypassing the LVS leader node. The + command <command>ctdb lvs leader</command> will show which node + is the current LVS leader. + </para> + + <para> + The path used for a client I/O is: + <orderedlist> + <listitem> + <para> + Client sends request packet to LVS leader. + </para> + </listitem> + <listitem> + <para> + LVS leader passes the request on to one node across the + internal network. + </para> + </listitem> + <listitem> + <para> + Selected node processes the request. + </para> + </listitem> + <listitem> + <para> + Node responds back to client. + </para> + </listitem> + </orderedlist> + </para> + + <para> + This means that all incoming traffic to the cluster will pass + through one physical node, which limits scalability. You can + send more data to the LVS address that one physical node can + multiplex. This means that you should not use LVS if your I/O + pattern is write-intensive since you will be limited in the + available network bandwidth that node can handle. LVS does work + very well for read-intensive workloads where only smallish READ + requests are going through the LVS leader bottleneck and the + majority of the traffic volume (the data in the read replies) + goes straight from the processing node back to the clients. For + read-intensive i/o patterns you can achieve very high throughput + rates in this mode. + </para> + + <para> + Note: you can use LVS and public addresses at the same time. + </para> + + <para> + If you use LVS, you must have a permanent address configured for + the public interface on each node. This address must be routable + and the cluster nodes must be configured so that all traffic + back to client hosts are routed through this interface. This is + also required in order to allow samba/winbind on the node to + talk to the domain controller. This LVS IP address can not be + used to initiate outgoing traffic. + </para> + <para> + Make sure that the domain controller and the clients are + reachable from a node <emphasis>before</emphasis> you enable + LVS. Also ensure that outgoing traffic to these hosts is routed + out through the configured public interface. + </para> + + <refsect2> + <title>Configuration</title> + + <para> + To activate LVS on a CTDB node you must specify the + <varname>CTDB_LVS_PUBLIC_IFACE</varname>, + <varname>CTDB_LVS_PUBLIC_IP</varname> and + <varname>CTDB_LVS_NODES</varname> configuration variables. + <varname>CTDB_LVS_NODES</varname> specifies a file containing + the private address of all nodes in the current node's LVS + group. + </para> + + <para> + Example: + <screen format="linespecific"> +CTDB_LVS_PUBLIC_IFACE=eth1 +CTDB_LVS_PUBLIC_IP=10.1.1.237 +CTDB_LVS_NODES=/usr/local/etc/ctdb/lvs_nodes + </screen> + </para> + + <para> + Example <filename>/usr/local/etc/ctdb/lvs_nodes</filename>: + </para> + <screen format="linespecific"> +192.168.1.2 +192.168.1.3 +192.168.1.4 + </screen> + + <para> + Normally any node in an LVS group can act as the LVS leader. + Nodes that are highly loaded due to other demands maybe + flagged with the "follower-only" option in the + <varname>CTDB_LVS_NODES</varname> file to limit the LVS + functionality of those nodes. + </para> + + <para> + LVS nodes file that excludes 192.168.1.4 from being + the LVS leader node: + </para> + <screen format="linespecific"> +192.168.1.2 +192.168.1.3 +192.168.1.4 follower-only + </screen> + + </refsect2> + </refsect1> + + <refsect1> + <title>TRACKING AND RESETTING TCP CONNECTIONS</title> + + <para> + CTDB tracks TCP connections from clients to public IP addresses, + on known ports. When an IP address moves from one node to + another, all existing TCP connections to that IP address are + reset. The node taking over this IP address will also send + gratuitous ARPs (for IPv4, or neighbour advertisement, for + IPv6). This allows clients to reconnect quickly, rather than + waiting for TCP timeouts, which can be very long. + </para> + + <para> + It is important that established TCP connections do not survive + a release and take of a public IP address on the same node. + Such connections can get out of sync with sequence and ACK + numbers, potentially causing a disruptive ACK storm. + </para> + + </refsect1> + + <refsect1> + <title>NAT GATEWAY</title> + + <para> + NAT gateway (NATGW) is an optional feature that is used to + configure fallback routing for nodes. This allows cluster nodes + to connect to external services (e.g. DNS, AD, NIS and LDAP) + when they do not host any public addresses (e.g. when they are + unhealthy). + </para> + <para> + This also applies to node startup because CTDB marks nodes as + UNHEALTHY until they have passed a "monitor" event. In this + context, NAT gateway helps to avoid a "chicken and egg" + situation where a node needs to access an external service to + become healthy. + </para> + <para> + Another way of solving this type of problem is to assign an + extra static IP address to a public interface on every node. + This is simpler but it uses an extra IP address per node, while + NAT gateway generally uses only one extra IP address. + </para> + + <refsect2> + <title>Operation</title> + + <para> + One extra NATGW public address is assigned on the public + network to each NATGW group. Each NATGW group is a set of + nodes in the cluster that shares the same NATGW address to + talk to the outside world. Normally there would only be one + NATGW group spanning an entire cluster, but in situations + where one CTDB cluster spans multiple physical sites it might + be useful to have one NATGW group for each site. + </para> + <para> + There can be multiple NATGW groups in a cluster but each node + can only be member of one NATGW group. + </para> + <para> + In each NATGW group, one of the nodes is selected by CTDB to + be the NATGW leader and the other nodes are consider to be + NATGW followers. NATGW followers establish a fallback default route + to the NATGW leader via the private network. When a NATGW + follower hosts no public IP addresses then it will use this route + for outbound connections. The NATGW leader hosts the NATGW + public IP address and routes outgoing connections from + follower nodes via this IP address. It also establishes a + fallback default route. + </para> + </refsect2> + + <refsect2> + <title>Configuration</title> + + <para> + NATGW is usually configured similar to the following example configuration: + </para> + <screen format="linespecific"> +CTDB_NATGW_NODES=/usr/local/etc/ctdb/natgw_nodes +CTDB_NATGW_PRIVATE_NETWORK=192.168.1.0/24 +CTDB_NATGW_PUBLIC_IP=10.0.0.227/24 +CTDB_NATGW_PUBLIC_IFACE=eth0 +CTDB_NATGW_DEFAULT_GATEWAY=10.0.0.1 + </screen> + + <para> + Normally any node in a NATGW group can act as the NATGW + leader. Some configurations may have special nodes that lack + connectivity to a public network. In such cases, those nodes + can be flagged with the "follower-only" option in the + <varname>CTDB_NATGW_NODES</varname> file to limit the NATGW + functionality of those nodes. + </para> + + <para> + See the <citetitle>NAT GATEWAY</citetitle> section in + <citerefentry><refentrytitle>ctdb-script.options</refentrytitle> + <manvolnum>5</manvolnum></citerefentry> for more details of + NATGW configuration. + </para> + </refsect2> + + + <refsect2> + <title>Implementation details</title> + + <para> + When the NATGW functionality is used, one of the nodes is + selected to act as a NAT gateway for all the other nodes in + the group when they need to communicate with the external + services. The NATGW leader is selected to be a node that is + most likely to have usable networks. + </para> + + <para> + The NATGW leader hosts the NATGW public IP address + <varname>CTDB_NATGW_PUBLIC_IP</varname> on the configured public + interfaces <varname>CTDB_NATGW_PUBLIC_IFACE</varname> and acts as + a router, masquerading outgoing connections from follower nodes + via this IP address. If + <varname>CTDB_NATGW_DEFAULT_GATEWAY</varname> is set then it + also establishes a fallback default route to the configured + this gateway with a metric of 10. A metric 10 route is used + so it can co-exist with other default routes that may be + available. + </para> + + <para> + A NATGW follower establishes its fallback default route to the + NATGW leader via the private network + <varname>CTDB_NATGW_PRIVATE_NETWORK</varname>with a metric of 10. + This route is used for outbound connections when no other + default route is available because the node hosts no public + addresses. A metric 10 routes is used so that it can co-exist + with other default routes that may be available when the node + is hosting public addresses. + </para> + + <para> + <varname>CTDB_NATGW_STATIC_ROUTES</varname> can be used to + have NATGW create more specific routes instead of just default + routes. + </para> + + <para> + This is implemented in the <filename>11.natgw</filename> + eventscript. Please see the eventscript file and the + <citetitle>NAT GATEWAY</citetitle> section in + <citerefentry><refentrytitle>ctdb-script.options</refentrytitle> + <manvolnum>5</manvolnum></citerefentry> for more details. + </para> + + </refsect2> + </refsect1> + + <refsect1> + <title>POLICY ROUTING</title> + + <para> + Policy routing is an optional CTDB feature to support complex + network topologies. Public addresses may be spread across + several different networks (or VLANs) and it may not be possible + to route packets from these public addresses via the system's + default route. Therefore, CTDB has support for policy routing + via the <filename>13.per_ip_routing</filename> eventscript. + This allows routing to be specified for packets sourced from + each public address. The routes are added and removed as CTDB + moves public addresses between nodes. + </para> + + <refsect2> + <title>Configuration variables</title> + + <para> + There are 4 configuration variables related to policy routing: + <varname>CTDB_PER_IP_ROUTING_CONF</varname>, + <varname>CTDB_PER_IP_ROUTING_RULE_PREF</varname>, + <varname>CTDB_PER_IP_ROUTING_TABLE_ID_LOW</varname>, + <varname>CTDB_PER_IP_ROUTING_TABLE_ID_HIGH</varname>. See the + <citetitle>POLICY ROUTING</citetitle> section in + <citerefentry><refentrytitle>ctdb-script.options</refentrytitle> + <manvolnum>5</manvolnum></citerefentry> for more details. + </para> + </refsect2> + + <refsect2> + <title>Configuration</title> + + <para> + The format of each line of + <varname>CTDB_PER_IP_ROUTING_CONF</varname> is: + </para> + + <screen> +<public_address> <network> [ <gateway> ] + </screen> + + <para> + Leading whitespace is ignored and arbitrary whitespace may be + used as a separator. Lines that have a "public address" item + that doesn't match an actual public address are ignored. This + means that comment lines can be added using a leading + character such as '#', since this will never match an IP + address. + </para> + + <para> + A line without a gateway indicates a link local route. + </para> + + <para> + For example, consider the configuration line: + </para> + + <screen> + 192.168.1.99 192.168.1.0/24 + </screen> + + <para> + If the corresponding public_addresses line is: + </para> + + <screen> + 192.168.1.99/24 eth2,eth3 + </screen> + + <para> + <varname>CTDB_PER_IP_ROUTING_RULE_PREF</varname> is 100, and + CTDB adds the address to eth2 then the following routing + information is added: + </para> + + <screen> + ip rule add from 192.168.1.99 pref 100 table ctdb.192.168.1.99 + ip route add 192.168.1.0/24 dev eth2 table ctdb.192.168.1.99 + </screen> + + <para> + This causes traffic from 192.168.1.99 to 192.168.1.0/24 go via + eth2. + </para> + + <para> + The <command>ip rule</command> command will show (something + like - depending on other public addresses and other routes on + the system): + </para> + + <screen> + 0: from all lookup local + 100: from 192.168.1.99 lookup ctdb.192.168.1.99 + 32766: from all lookup main + 32767: from all lookup default + </screen> + + <para> + <command>ip route show table ctdb.192.168.1.99</command> will show: + </para> + + <screen> + 192.168.1.0/24 dev eth2 scope link + </screen> + + <para> + The usual use for a line containing a gateway is to add a + default route corresponding to a particular source address. + Consider this line of configuration: + </para> + + <screen> + 192.168.1.99 0.0.0.0/0 192.168.1.1 + </screen> + + <para> + In the situation described above this will cause an extra + routing command to be executed: + </para> + + <screen> + ip route add 0.0.0.0/0 via 192.168.1.1 dev eth2 table ctdb.192.168.1.99 + </screen> + + <para> + With both configuration lines, <command>ip route show table + ctdb.192.168.1.99</command> will show: + </para> + + <screen> + 192.168.1.0/24 dev eth2 scope link + default via 192.168.1.1 dev eth2 + </screen> + </refsect2> + + <refsect2> + <title>Sample configuration</title> + + <para> + Here is a more complete example configuration. + </para> + + <screen> +/usr/local/etc/ctdb/public_addresses: + + 192.168.1.98 eth2,eth3 + 192.168.1.99 eth2,eth3 + +/usr/local/etc/ctdb/policy_routing: + + 192.168.1.98 192.168.1.0/24 + 192.168.1.98 192.168.200.0/24 192.168.1.254 + 192.168.1.98 0.0.0.0/0 192.168.1.1 + 192.168.1.99 192.168.1.0/24 + 192.168.1.99 192.168.200.0/24 192.168.1.254 + 192.168.1.99 0.0.0.0/0 192.168.1.1 + </screen> + + <para> + The routes local packets as expected, the default route is as + previously discussed, but packets to 192.168.200.0/24 are + routed via the alternate gateway 192.168.1.254. + </para> + + </refsect2> + </refsect1> + + <refsect1> + <title>NOTIFICATIONS</title> + + <para> + When certain state changes occur in CTDB, it can be configured + to perform arbitrary actions via notifications. For example, + sending SNMP traps or emails when a node becomes unhealthy or + similar. + </para> + + <para> + The notification mechanism runs all executable files ending in + ".script" in + <filename>/usr/local/etc/ctdb/events/notification/</filename>, + ignoring any failures and continuing to run all files. + </para> + + <para> + CTDB currently generates notifications after CTDB changes to + these states: + </para> + + <simplelist> + <member>init</member> + <member>setup</member> + <member>startup</member> + <member>healthy</member> + <member>unhealthy</member> + </simplelist> + + </refsect1> + + <refsect1> + <title>LOG LEVELS</title> + + <para> + Valid log levels, in increasing order of verbosity, are: + </para> + + <simplelist> + <member>ERROR</member> + <member>WARNING</member> + <member>NOTICE</member> + <member>INFO</member> + <member>DEBUG</member> + </simplelist> + </refsect1> + + + <refsect1> + <title>REMOTE CLUSTER NODES</title> + <para> +It is possible to have a CTDB cluster that spans across a WAN link. +For example where you have a CTDB cluster in your datacentre but you also +want to have one additional CTDB node located at a remote branch site. +This is similar to how a WAN accelerator works but with the difference +that while a WAN-accelerator often acts as a Proxy or a MitM, in +the ctdb remote cluster node configuration the Samba instance at the remote site +IS the genuine server, not a proxy and not a MitM, and thus provides 100% +correct CIFS semantics to clients. + </para> + + <para> + See the cluster as one single multihomed samba server where one of + the NICs (the remote node) is very far away. + </para> + + <para> + NOTE: This does require that the cluster filesystem you use can cope + with WAN-link latencies. Not all cluster filesystems can handle + WAN-link latencies! Whether this will provide very good WAN-accelerator + performance or it will perform very poorly depends entirely + on how optimized your cluster filesystem is in handling high latency + for data and metadata operations. + </para> + + <para> + To activate a node as being a remote cluster node you need to + set the following two parameters in + /usr/local/etc/ctdb/ctdb.conf for the remote node: + <screen format="linespecific"> +[legacy] + lmaster capability = false + leader capability = false + </screen> + </para> + + <para> + Verify with the command "ctdb getcapabilities" that that node no longer + has the leader or the lmaster capabilities. + </para> + + </refsect1> + + + <refsect1> + <title>SEE ALSO</title> + + <para> + <citerefentry><refentrytitle>ctdb</refentrytitle> + <manvolnum>1</manvolnum></citerefentry>, + + <citerefentry><refentrytitle>ctdbd</refentrytitle> + <manvolnum>1</manvolnum></citerefentry>, + + <citerefentry><refentrytitle>ctdb_diagnostics</refentrytitle> + <manvolnum>1</manvolnum></citerefentry>, + + <citerefentry><refentrytitle>ltdbtool</refentrytitle> + <manvolnum>1</manvolnum></citerefentry>, + + <citerefentry><refentrytitle>onnode</refentrytitle> + <manvolnum>1</manvolnum></citerefentry>, + + <citerefentry><refentrytitle>ping_pong</refentrytitle> + <manvolnum>1</manvolnum></citerefentry>, + + <citerefentry><refentrytitle>ctdb.conf</refentrytitle> + <manvolnum>5</manvolnum></citerefentry>, + + <citerefentry><refentrytitle>ctdb-script.options</refentrytitle> + <manvolnum>5</manvolnum></citerefentry>, + + <citerefentry><refentrytitle>ctdb.sysconfig</refentrytitle> + <manvolnum>5</manvolnum></citerefentry>, + + <citerefentry><refentrytitle>ctdb-statistics</refentrytitle> + <manvolnum>7</manvolnum></citerefentry>, + + <citerefentry><refentrytitle>ctdb-tunables</refentrytitle> + <manvolnum>7</manvolnum></citerefentry>, + + <ulink url="https://wiki.samba.org/index.php/CTDB_and_Clustered_Samba"/>, + + <ulink url="http://ctdb.samba.org/"/> + </para> + </refsect1> + + <refentryinfo> + <author> + <contrib> + This documentation was written by + Ronnie Sahlberg, + Amitay Isaacs, + Martin Schwenke + </contrib> + </author> + + <copyright> + <year>2007</year> + <holder>Andrew Tridgell</holder> + <holder>Ronnie Sahlberg</holder> + </copyright> + <legalnotice> + <para> + This program is free software; you can redistribute it and/or + modify it under the terms of the GNU General Public License as + published by the Free Software Foundation; either version 3 of + the License, or (at your option) any later version. + </para> + <para> + This program is distributed in the hope that it will be + useful, but WITHOUT ANY WARRANTY; without even the implied + warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR + PURPOSE. See the GNU General Public License for more details. + </para> + <para> + You should have received a copy of the GNU General Public + License along with this program; if not, see + <ulink url="http://www.gnu.org/licenses"/>. + </para> + </legalnotice> + </refentryinfo> + +</refentry> diff --git a/ctdb/doc/ctdb.conf.5.xml b/ctdb/doc/ctdb.conf.5.xml new file mode 100644 index 0000000..87a7ea5 --- /dev/null +++ b/ctdb/doc/ctdb.conf.5.xml @@ -0,0 +1,652 @@ +<?xml version="1.0" encoding="iso-8859-1"?> +<!DOCTYPE refentry + PUBLIC "-//OASIS//DTD DocBook XML V4.5//EN" + "http://www.oasis-open.org/docbook/xml/4.5/docbookx.dtd"> + +<refentry id="ctdb.conf.5"> + + <refmeta> + <refentrytitle>ctdb.conf</refentrytitle> + <manvolnum>5</manvolnum> + <refmiscinfo class="source">ctdb</refmiscinfo> + <refmiscinfo class="manual">CTDB - clustered TDB database</refmiscinfo> + </refmeta> + + <refnamediv> + <refname>ctdb.conf</refname> + <refpurpose>CTDB configuration file</refpurpose> + </refnamediv> + + <refsect1> + <title>DESCRIPTION</title> + + <para> + This file contains CTDB configuration options that affect the + operation of CTDB daemons and command-line tools. The default + location of this file is + <filename>/usr/local/etc/ctdb/ctdb.conf</filename>. + </para> + + <para> + Note that this is a Samba-style configuration file, so it has a + very different syntax to previous CTDB configuration files. + </para> + + <para> + For event script options please see + <citerefentry><refentrytitle>ctdb-script.options</refentrytitle> + <manvolnum>5</manvolnum></citerefentry>. + </para> + + <para> + Configuration options are grouped into several sections below. + There are only a few options in each section, allowing them to + be ordered (approximately) in decreasing order of importance. + </para> + + </refsect1> + + <refsect1> + <title> + LOGGING CONFIGURATION + </title> + + <para> + Options in this section control CTDB's logging. They are valid + within the <emphasis>logging</emphasis> section of file, + indicated by <literal>[logging]</literal>. + </para> + + <variablelist> + + <varlistentry> + <term>log level = <parameter>LOGLEVEL</parameter></term> + <listitem> + <para> + LOGLEVEL is a string that controls the verbosity of + ctdbd's logging. See the <citetitle>LOG + LEVELS</citetitle> section in + <citerefentry><refentrytitle>ctdb</refentrytitle> + <manvolnum>7</manvolnum></citerefentry> for more details. + </para> + <para> + Default: <literal>NOTICE</literal> + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term>location = <parameter>STRING</parameter></term> + <listitem> + <para> + STRING specifies where ctdbd will write its log. + </para> + <para> + Valid values are: + </para> + <variablelist> + <varlistentry> + <term>file:<parameter>FILENAME</parameter></term> + <listitem> + <para> + FILENAME where ctdbd will write its log. This is usually + <filename>/usr/local/var/log/log.ctdb</filename>. + </para> + </listitem> + </varlistentry> + <varlistentry> + <term>syslog<optional>:<parameter>METHOD</parameter></optional></term> + <listitem> + <para> + CTDB will log to syslog. By default this will use + the syslog(3) API. + </para> + <para> + If METHOD is specified then it specifies an + extension that causes logging to be done in a + non-blocking fashion. This can be useful under + heavy loads that might cause the syslog daemon to + dequeue messages too slowly, which would otherwise + cause CTDB to block when logging. METHOD must be + one of: + </para> + <variablelist> + <varlistentry> + <term>nonblocking</term> + <listitem> + <para> + CTDB will log to syslog via + <filename>/dev/log</filename> in non-blocking + mode. + </para> + </listitem> + </varlistentry> + <varlistentry> + <term>udp</term> + <listitem> + <para> + CTDB will log to syslog via UDP to + localhost:514. The syslog daemon must be + configured to listen on (at least) + localhost:514. Most implementations will log + the messages against hostname "localhost" - + this is a limit of the implementation for + compatibility with more syslog daemon + implementations. + </para> + </listitem> + </varlistentry> + <varlistentry> + <term>udp-rfc5424</term> + <listitem> + <para> + As with "udp" but messages are sent in RFC5424 + format. This method will log the correct + hostname but is not as widely implemented in + syslog daemons. + </para> + </listitem> + </varlistentry> + </variablelist> + </listitem> + </varlistentry> + </variablelist> + <para> + Default: + file:<filename>/usr/local/var/log/log.ctdb</filename> + </para> + </listitem> + </varlistentry> + + </variablelist> + </refsect1> + + <refsect1> + <title> + CLUSTER CONFIGURATION + </title> + + <para> + Options in this section affect the CTDB cluster setup. They + are valid within the <emphasis>cluster</emphasis> section of + file, indicated by <literal>[cluster]</literal>. + </para> + + <variablelist> + + <varlistentry> + <term>cluster lock = <parameter>LOCK</parameter></term> + <listitem> + <para> + LOCK specifies the cluster-wide mutex used to detect and + prevent a partitioned cluster (or "split brain"). + </para> + <para> + For information about the cluster lock please see the + <citetitle>CLUSTER LOCK</citetitle> section in + <citerefentry><refentrytitle>ctdb</refentrytitle> + <manvolnum>7</manvolnum></citerefentry>. + </para> + <para> + Default: NONE. However, uses of a cluster lock is + <emphasis>strongly recommended</emphasis>. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term>leader capability = true|false</term> + <listitem> + <para> + Indicates whether a node can become the leader + for the cluster. If this is set to + <literal>false</literal> then the node will not be able to + become the leader for the cluster. This feature + is primarily used for making a cluster span across a WAN + link and use CTDB as a WAN-accelerator. + </para> + <para> + Please see the <citetitle>REMOTE CLUSTER NODES</citetitle> + section in + <citerefentry><refentrytitle>ctdb</refentrytitle> + <manvolnum>7</manvolnum></citerefentry> for more + information. + </para> + <para> + Default: <literal>true</literal> + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term>leader timeout = <parameter>SECONDS</parameter></term> + <listitem> + <para> + Number of SECONDS without a leader broadcast before a node + triggers an election. + </para> + <para> + Default: <literal>5</literal> + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term>node address = <parameter>IPADDR</parameter></term> + <listitem> + <para> + IPADDR is the private IP address that ctdbd will bind to. + </para> + <para> + This option is only required when automatic address + detection can not be used. This can be the case when + running multiple ctdbd daemons/nodes on the same physical + host (usually for testing), using InfiniBand for the + private network or on Linux when sysctl + net.ipv4.ip_nonlocal_bind=1. + </para> + <para> + Default: CTDB selects the first address from the nodes + list that it can bind to. See also the <citetitle>PRIVATE + ADDRESS</citetitle> section in + <citerefentry><refentrytitle>ctdb</refentrytitle> + <manvolnum>7</manvolnum></citerefentry>. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term>transport = tcp|ib</term> + <listitem> + <para> + This option specifies which transport to use for ctdbd + internode communications on the private network. + </para> + <para> + <literal>ib</literal> means InfiniBand. The InfiniBand + support is not regularly tested. If it is known to be + broken then it may be disabled so that a value of + <literal>ib</literal> is considered invalid. + </para> + <para> + Default: <literal>tcp</literal> + </para> + </listitem> + </varlistentry> + + </variablelist> + </refsect1> + + <refsect1> + <title> + DATABASE CONFIGURATION + </title> + + <para> + Options in this section affect the CTDB database setup. They + are valid within the <emphasis>database</emphasis> section of + file, indicated by <literal>[database]</literal>. + </para> + + <variablelist> + + <varlistentry> + <term>volatile database directory = <parameter>DIRECTORY</parameter></term> + <listitem> + <para> + DIRECTORY on local storage where CTDB keeps a local copy + of volatile TDB databases. This directory is local for + each node and should not be stored on the shared cluster + filesystem. + </para> + <para> + Mounting a tmpfs (or similar memory filesystem) on this + directory can provide a significant performance + improvement when there is I/O contention on the local + disk. + </para> + <para> + Default: <filename>/usr/local/var/lib/ctdb/volatile</filename> + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term>persistent database directory=<parameter>DIRECTORY</parameter></term> + <listitem> + <para> + DIRECTORY on local storage where CTDB keeps a local copy + of persistent TDB databases. This directory is local for + each node and should not be stored on the shared cluster + filesystem. + </para> + <para> + Default: <filename>/usr/local/var/lib/ctdb/persistent</filename> + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term>state database directory = <parameter>DIRECTORY</parameter></term> + <listitem> + <para> + DIRECTORY on local storage where CTDB keeps a local copy + of internal state TDB databases. This directory is local + for each node and should not be stored on the shared + cluster filesystem. + </para> + <para> + Default: <filename>/usr/local/var/lib/ctdb/state</filename> + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term>tdb mutexes = true|false</term> + <listitem> + <para> + This parameter enables TDB_MUTEX_LOCKING feature on + volatile databases if the robust mutexes are + supported. This optimizes the record locking using robust + mutexes and is much more efficient that using posix locks. + </para> + <para> + If robust mutexes are unreliable on the platform being + used then they can be disabled by setting this to + <literal>false</literal>. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term>lock debug script = <parameter>FILENAME</parameter></term> + <listitem> + <para> + FILENAME is a script used by CTDB's database locking code + to attempt to provide debugging information when CTDB is + unable to lock an entire database or a record. + </para> + <para> + This script should be a bare filename relative to the CTDB + configuration directory + (<filename>/usr/local/etc/ctdb/</filename>). Any + directory prefix is ignored and the path is calculated + relative to this directory. + </para> + <para> + CTDB provides a lock debugging script and installs it as + <filename>/usr/local/etc/ctdb/debug_locks.sh</filename>. + </para> + <para> + Default: NONE + </para> + </listitem> + </varlistentry> + + </variablelist> + </refsect1> + + <refsect1> + <title> + EVENT HANDLING CONFIGURATION + </title> + + <para> + Options in this section affect CTDB event handling. They are + valid within the <emphasis>event</emphasis> section of file, + indicated by <literal>[event]</literal>. + </para> + + <variablelist> + + <varlistentry> + <term>debug script = <parameter>FILENAME</parameter></term> + <listitem> + <para> + FILENAME is a script used by CTDB's event handling code to + attempt to provide debugging information when an event + times out. + </para> + <para> + This script should be a bare filename relative to the CTDB + configuration directory + (<filename>/usr/local/etc/ctdb/</filename>). Any + directory prefix is ignored and the path is calculated + relative to this directory. + </para> + <para> + CTDB provides a script for debugging timed out event + scripts and installs it as + <filename>/usr/local/etc/ctdb/debug-hung-script.sh</filename>. + </para> + <para> + Default: NONE + </para> + </listitem> + </varlistentry> + + </variablelist> + </refsect1> + + <refsect1> + <title> + FAILOVER CONFIGURATION + </title> + + <para> + Options in this section affect CTDB failover. They are + valid within the <emphasis>failover</emphasis> section of file, + indicated by <literal>[failover]</literal>. + </para> + + <variablelist> + + <varlistentry> + <term>disabled = true|false</term> + <listitem> + <para> + If set to <literal>true</literal> then public IP failover + is disabled. + </para> + <para> + Default: <literal>false</literal> + </para> + </listitem> + </varlistentry> + + </variablelist> + </refsect1> + + <refsect1> + <title> + LEGACY CONFIGURATION + </title> + + <para> + Options in this section affect legacy CTDB setup. They are valid + within the <emphasis>legacy</emphasis> section of file, + indicated by <literal>[legacy]</literal>. + </para> + + <variablelist> + + <varlistentry> + <term>ctdb start as stopped = true|false</term> + <listitem> + <para> + If set to <literal>true</literal> CTDB starts in the + STOPPED state. + </para> + <para> + To allow the node to take part in the cluster it must be + manually continued with the <command>ctdb + continue</command> command. + </para> + <para> + Please see the <citetitle>NODE STATES</citetitle> section + in <citerefentry><refentrytitle>ctdb</refentrytitle> + <manvolnum>7</manvolnum></citerefentry> for more + information about the STOPPED state. + </para> + <para> + Default: <literal>false</literal> + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term>start as disabled = true|false</term> + <listitem> + <para> + If set to <literal>true</literal> CTDB starts in the + DISABLED state. + </para> + <para> + To allow the node to host public IP addresses and + services, it must be manually enabled using the + <command>ctdb enable</command> command. + </para> + <para> + Please see the <citetitle>NODE STATES</citetitle> section + in <citerefentry><refentrytitle>ctdb</refentrytitle> + <manvolnum>7</manvolnum></citerefentry> for more + information about the DISABLED state. + </para> + <para> + Default: <literal>false</literal> + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term>realtime scheduling = true|false</term> + <listitem> + <para> + Usually CTDB runs with real-time priority. This helps it + to perform effectively on a busy system, such as when + there are thousands of Samba clients. If you are running + CTDB on a platform that does not support real-time + priority, you can set this to <literal>false</literal>. + </para> + <para> + Default: <literal>true</literal> + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term>lmaster capability = true|false</term> + <listitem> + <para> + Indicates whether a node can become a location master for + records in a database. If this is set to + <literal>false</literal> then the node will not be part of + the vnnmap. This feature is primarily used for making a + cluster span across a WAN link and use CTDB as a + WAN-accelerator. + </para> + <para> + Please see the <citetitle>REMOTE CLUSTER NODES</citetitle> + section in + <citerefentry><refentrytitle>ctdb</refentrytitle> + <manvolnum>7</manvolnum></citerefentry> for more + information. + </para> + <para> + Default: <literal>true</literal> + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term>script log level = <parameter>LOGLEVEL</parameter></term> + <listitem> + <para> + This option sets the debug level of event script output to + LOGLEVEL. + </para> + <para> + See the <citetitle>DEBUG LEVELS</citetitle> section in + <citerefentry><refentrytitle>ctdb</refentrytitle> + <manvolnum>7</manvolnum></citerefentry> for more + information. + </para> + <para> + Default: <literal>ERROR</literal> + </para> + </listitem> + </varlistentry> + + </variablelist> + + </refsect1> + + <refsect1> + <title>FILES</title> + + <simplelist> + <member><filename>/usr/local/etc/ctdb/ctdb.conf</filename></member> + </simplelist> + </refsect1> + + <refsect1> + <title>SEE ALSO</title> + <para> + <citerefentry><refentrytitle>ctdbd</refentrytitle> + <manvolnum>1</manvolnum></citerefentry>, + + <citerefentry><refentrytitle>onnode</refentrytitle> + <manvolnum>1</manvolnum></citerefentry>, + + <citerefentry><refentrytitle>ctdb.sysconfig</refentrytitle> + <manvolnum>5</manvolnum></citerefentry>, + + <citerefentry><refentrytitle>ctdb-script.options</refentrytitle> + <manvolnum>5</manvolnum></citerefentry>, + + <citerefentry><refentrytitle>ctdb</refentrytitle> + <manvolnum>7</manvolnum></citerefentry>, + + <citerefentry><refentrytitle>ctdb-tunables</refentrytitle> + <manvolnum>7</manvolnum></citerefentry>, + + <ulink url="http://ctdb.samba.org/"/> + </para> + </refsect1> + + <info> + <author> + <contrib> + This documentation was written by + Amitay Isaacs, + Martin Schwenke + </contrib> + </author> + + <copyright> + <year>2007</year> + <holder>Andrew Tridgell</holder> + <holder>Ronnie Sahlberg</holder> + </copyright> + <legalnotice> + <para> + This program is free software; you can redistribute it and/or + modify it under the terms of the GNU General Public License as + published by the Free Software Foundation; either version 3 of + the License, or (at your option) any later version. + </para> + <para> + This program is distributed in the hope that it will be + useful, but WITHOUT ANY WARRANTY; without even the implied + warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR + PURPOSE. See the GNU General Public License for more details. + </para> + <para> + You should have received a copy of the GNU General Public + License along with this program; if not, see + <ulink url="http://www.gnu.org/licenses"/>. + </para> + </legalnotice> + </info> + +</refentry> diff --git a/ctdb/doc/ctdb.sysconfig.5.xml b/ctdb/doc/ctdb.sysconfig.5.xml new file mode 100644 index 0000000..2618934 --- /dev/null +++ b/ctdb/doc/ctdb.sysconfig.5.xml @@ -0,0 +1,240 @@ +<?xml version="1.0" encoding="iso-8859-1"?> +<!DOCTYPE refentry + PUBLIC "-//OASIS//DTD DocBook XML V4.5//EN" + "http://www.oasis-open.org/docbook/xml/4.5/docbookx.dtd"> + +<refentry id="ctdb.sysconfig.5"> + + <refmeta> + <refentrytitle>ctdb.sysconfig</refentrytitle> + <manvolnum>5</manvolnum> + <refmiscinfo class="source">ctdb</refmiscinfo> + <refmiscinfo class="manual">CTDB - clustered TDB database</refmiscinfo> + </refmeta> + + <refnamediv> + <refname>ctdb.sysconfig</refname> + <refpurpose>CTDB daemon configuration file</refpurpose> + </refnamediv> + + <refsect1> + <title>DESCRIPTION</title> + + <para> + This file contains configuration that affects the operation of + CTDB. This is a distribution-specific service configuration + file such as <filename>/etc/sysconfig/ctdb</filename> (Red Hat) + or <filename>/etc/default/ctdb</filename> (Debian) and is a + shell script (see + <citerefentry><refentrytitle>sh</refentrytitle> + <manvolnum>1</manvolnum></citerefentry>). + </para> + + </refsect1> + + <refsect1> + <title> + GLOBAL CONFIGURATION + </title> + + <variablelist> + + <varlistentry> + <term>CTDB_INIT_STYLE=debian|redhat|suse</term> + <listitem> + <para> + This is the init style used by the Linux distribution (or + other operating system) being used. This is usually + determined dynamically by checking the system. This + variable is used by the initscript to determine which init + system primitives to use. It is also used by some + eventscripts to choose the name of initscripts for certain + services, since these can vary between distributions. + </para> + <para> + If using CTDB's event scripts are unable to determine an + appropriate default then this option can also be placed in + a relevant + <citerefentry><refentrytitle>ctdb-script.options</refentrytitle> + <manvolnum>5</manvolnum></citerefentry> file. + </para> + <para> + Default: NONE. Guessed, based on features of + distribution. + </para> + </listitem> + </varlistentry> + + </variablelist> + </refsect1> + + <refsect1> + <title> + RESOURCE LIMITS + </title> + + <refsect2> + <title> + Maximum number of open files + </title> + + <para> + CTDB can use a lot of file descriptors, especially when used + with Samba. If there are thousands of smbd processes + connected to CTDB when this can mean that thousands of file + descriptors are used. For CTDB, it is often necessary to + increase limit on the maximum number of open files. + </para> + + <para> + The maximum number of open files should be configured using an + operating system mechanism. + </para> + + <variablelist> + + <varlistentry> + <term> + systemd + </term> + <listitem> + <para> + The + <literal>LimitNOFILE=<option>LIMIT</option></literal> + option can be used in a unit/service file increase the + maximum number of open files. See + <citerefentry><refentrytitle>systemd.exec</refentrytitle> + <manvolnum>5</manvolnum></citerefentry> for details. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term> + SYSV init + </term> + <listitem> + <para> + Use a command like <command>ulimit -n + <option>LIMIT</option></command> to increase the maximum + number of open files. This command can be put in the + relevant distribution-specific service configuration file. + </para> + </listitem> + </varlistentry> + + </variablelist> + + </refsect2> + + <refsect2> + <title> + Allowing core dumps + </title> + + <para> + Many distributions do not allow core dump files to be + generated by default. To assist with debugging, core files + can be enabled. This should be configured using an operating + system mechanism. + </para> + + <variablelist> + + <varlistentry> + <term> + systemd + </term> + <listitem> + <para> + The <literal>LimitCORE=0|unlimited</literal> option can + be used in a unit/service file. <literal>0</literal> + disallows core files, <literal>unlimited</literal> + allows them. maximum number of open files. See + <citerefentry><refentrytitle>systemd.exec</refentrytitle> + <manvolnum>5</manvolnum></citerefentry> for details. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term> + SYSV init + </term> + <listitem> + <para> + Use a command like <command>ulimit -c 0|unlimited</command> + to disable or enable core files as required. This + command can be put in the relevant distribution-specific + service configuration file. + </para> + </listitem> + </varlistentry> + + </variablelist> + + </refsect2> + + </refsect1> + + <refsect1> + <title>FILES</title> + + <simplelist> + <member><filename>/etc/sysconfig/ctdb</filename></member> + <member><filename>/etc/default/ctdb</filename></member> + <member><filename>/usr/local/etc/ctdb/script.options</filename></member> + </simplelist> + </refsect1> + + <refsect1> + <title>SEE ALSO</title> + <para> + <citerefentry><refentrytitle>ctdbd</refentrytitle> + <manvolnum>1</manvolnum></citerefentry>, + + <citerefentry><refentrytitle>ctdb-script.options</refentrytitle> + <manvolnum>5</manvolnum></citerefentry>, + + <citerefentry><refentrytitle>ctdb</refentrytitle> + <manvolnum>7</manvolnum></citerefentry>, + + <ulink url="http://ctdb.samba.org/"/> + </para> + </refsect1> + + <refentryinfo> + <author> + <contrib> + This documentation was written by + Martin Schwenke + </contrib> + </author> + + <copyright> + <year>2007</year> + <holder>Andrew Tridgell</holder> + <holder>Ronnie Sahlberg</holder> + </copyright> + <legalnotice> + <para> + This program is free software; you can redistribute it and/or + modify it under the terms of the GNU General Public License as + published by the Free Software Foundation; either version 3 of + the License, or (at your option) any later version. + </para> + <para> + This program is distributed in the hope that it will be + useful, but WITHOUT ANY WARRANTY; without even the implied + warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR + PURPOSE. See the GNU General Public License for more details. + </para> + <para> + You should have received a copy of the GNU General Public + License along with this program; if not, see + <ulink url="http://www.gnu.org/licenses"/>. + </para> + </legalnotice> + </refentryinfo> + +</refentry> diff --git a/ctdb/doc/ctdb_diagnostics.1.xml b/ctdb/doc/ctdb_diagnostics.1.xml new file mode 100644 index 0000000..274c70d --- /dev/null +++ b/ctdb/doc/ctdb_diagnostics.1.xml @@ -0,0 +1,128 @@ +<?xml version="1.0" encoding="iso-8859-1"?> +<!DOCTYPE refentry + PUBLIC "-//OASIS//DTD DocBook XML V4.5//EN" + "http://www.oasis-open.org/docbook/xml/4.5/docbookx.dtd"> +<refentry id="ctdb_diagnostics.1"> + + <refmeta> + <refentrytitle>ctdb_diagnostics</refentrytitle> + <manvolnum>1</manvolnum> + <refmiscinfo class="source">ctdb</refmiscinfo> + <refmiscinfo class="manual">CTDB - clustered TDB database</refmiscinfo> + </refmeta> + + <refnamediv> + <refname>ctdb_diagnostics</refname> + <refpurpose>dump diagnostic information about CTDB/Samba installation</refpurpose> + </refnamediv> + + <refsynopsisdiv> + <cmdsynopsis> + <command>ctdb_diagnostics</command> + <arg>OPTIONS</arg> + <arg choice="plain">...</arg> + </cmdsynopsis> + </refsynopsisdiv> + + <refsect1> + <title>DESCRIPTION</title> + <para> + ctdb_diagnostics is used to dump diagnostic information about a + clustered Samba installation. This includes configuration + files, output of relevant commands and logs. This information + can be used to check the correctness of the configuration and to + diagnose problems. + </para> + </refsect1> + + <refsect1> + <title>OPTIONS</title> + + <variablelist> + + <varlistentry> + <term>-n <nodes></term> + <listitem> + <para> + Comma separated list of nodes to operate on + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term>-c</term> + <listitem> + <para> + Ignore comment lines (starting with '#') in file comparisons + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term>-w</term> + <listitem> + <para> + Ignore whitespace in file comparisons + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term>--no-ads</term> + <listitem> + <para> + Do not use commands that assume an Active Directory Server + </para> + </listitem> + </varlistentry> + + </variablelist> + + </refsect1> + + <refsect1> + <title>SEE ALSO</title> + <para> + <citerefentry><refentrytitle>ctdb</refentrytitle> + <manvolnum>1</manvolnum></citerefentry>, + <citerefentry><refentrytitle>ctdb</refentrytitle> + <manvolnum>7</manvolnum></citerefentry>, + <ulink url="https://ctdb.samba.org/"/> + </para> + </refsect1> + + <refentryinfo> + <author> + <contrib> + This documentation was written by Martijn van Brummelen + </contrib> + </author> + + <copyright> + <year>2015</year> + <holder>Martijn van Brummelen</holder> + </copyright> + + <legalnotice> + <para> + This program is free software; you can redistribute it and/or + modify it under the terms of the GNU General Public License as + published by the Free Software Foundation; either version 3 of + the License, or (at your option) any later version. + </para> + <para> + This program is distributed in the hope that it will be + useful, but WITHOUT ANY WARRANTY; without even the implied + warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR + PURPOSE. See the GNU General Public License for more details. + </para> + <para> + You should have received a copy of the GNU General Public + License along with this program; if not, see + <ulink url="http://www.gnu.org/licenses"/>. + </para> + </legalnotice> + + </refentryinfo> + +</refentry> diff --git a/ctdb/doc/ctdb_mutex_ceph_rados_helper.7.xml b/ctdb/doc/ctdb_mutex_ceph_rados_helper.7.xml new file mode 100644 index 0000000..dd3dbab --- /dev/null +++ b/ctdb/doc/ctdb_mutex_ceph_rados_helper.7.xml @@ -0,0 +1,96 @@ +<?xml version="1.0" encoding="iso-8859-1"?> +<!DOCTYPE refentry + PUBLIC "-//OASIS//DTD DocBook XML V4.5//EN" + "http://www.oasis-open.org/docbook/xml/4.5/docbookx.dtd"> +<refentry id="ctdb_mutex_ceph_rados_helper.7"> + + <refmeta> + <refentrytitle>Ceph RADOS Mutex</refentrytitle> + <manvolnum>7</manvolnum> + <refmiscinfo class="source">ctdb</refmiscinfo> + <refmiscinfo class="manual">CTDB - clustered TDB database</refmiscinfo> + </refmeta> + + <refnamediv> + <refname>ctdb_mutex_ceph_rados_helper</refname> + <refpurpose>Ceph RADOS cluster mutex helper</refpurpose> + </refnamediv> + + <refsect1> + <title>DESCRIPTION</title> + <para> + ctdb_mutex_ceph_rados_helper can be used as a cluster lock provider + for CTDB. When configured, split brain avoidance during CTDB recovery + will be handled using locks against an object located in a Ceph RADOS + pool. + To enable this functionality, include the following line in the + <literal>[cluster]</literal> section of + <citerefentry><refentrytitle>ctdb.conf</refentrytitle> + <manvolnum>5</manvolnum></citerefentry>: + </para> + <screen format="linespecific"> +cluster lock = !ctdb_mutex_ceph_rados_helper [Cluster] [User] [Pool] [Object] + +Cluster: Ceph cluster name (e.g. ceph) +User: Ceph cluster user name (e.g. client.admin) +Pool: Ceph RADOS pool name +Object: Ceph RADOS object name + </screen> + <para> + The Ceph cluster <parameter>Cluster</parameter> must be up and running, + with a configuration, and keyring file for <parameter>User</parameter> + located in a librados default search path (e.g. /etc/ceph/). + <parameter>Pool</parameter> must already exist. + </para> + <para> + For informational purposes, ctdb_mutex_ceph_rados_helper will also + register the cluster lock holder in Ceph Manager's service map. + </para> + </refsect1> + + <refsect1> + <title>SEE ALSO</title> + <para> + <citerefentry><refentrytitle>ctdb</refentrytitle> + <manvolnum>7</manvolnum></citerefentry>, + + <citerefentry><refentrytitle>ctdbd</refentrytitle> + <manvolnum>1</manvolnum></citerefentry>, + + <ulink url="http://ctdb.samba.org/"/> + </para> + </refsect1> + + <refentryinfo> + <author> + <contrib> + This documentation was written by David Disseldorp + </contrib> + </author> + + <copyright> + <year>2016</year> + <holder>David Disseldorp</holder> + </copyright> + <legalnotice> + <para> + This program is free software; you can redistribute it and/or + modify it under the terms of the GNU General Public License as + published by the Free Software Foundation; either version 3 of + the License, or (at your option) any later version. + </para> + <para> + This program is distributed in the hope that it will be + useful, but WITHOUT ANY WARRANTY; without even the implied + warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR + PURPOSE. See the GNU General Public License for more details. + </para> + <para> + You should have received a copy of the GNU General Public + License along with this program; if not, see + <ulink url="http://www.gnu.org/licenses"/>. + </para> + </legalnotice> + </refentryinfo> + +</refentry> diff --git a/ctdb/doc/ctdbd.1.xml b/ctdb/doc/ctdbd.1.xml new file mode 100644 index 0000000..314964e --- /dev/null +++ b/ctdb/doc/ctdbd.1.xml @@ -0,0 +1,129 @@ +<?xml version="1.0" encoding="iso-8859-1"?> +<!DOCTYPE refentry + PUBLIC "-//OASIS//DTD DocBook XML V4.5//EN" + "http://www.oasis-open.org/docbook/xml/4.5/docbookx.dtd"> + +<refentry id="ctdbd.1"> + + <refmeta> + <refentrytitle>ctdbd</refentrytitle> + <manvolnum>1</manvolnum> + <refmiscinfo class="source">ctdb</refmiscinfo> + <refmiscinfo class="manual">CTDB - clustered TDB database</refmiscinfo> + </refmeta> + + <refnamediv> + <refname>ctdbd</refname> + <refpurpose>The CTDB cluster daemon</refpurpose> + </refnamediv> + + <refsynopsisdiv> + <cmdsynopsis> + <command>ctdbd</command> + <arg rep="repeat"><replaceable>OPTION</replaceable></arg> + </cmdsynopsis> + </refsynopsisdiv> + + <refsect1> + <title>DESCRIPTION</title> + <para> + ctdbd is the main CTDB daemon. + </para> + + <para> + See <citerefentry><refentrytitle>ctdb</refentrytitle> + <manvolnum>7</manvolnum></citerefentry> for an overview of CTDB. + </para> + </refsect1> + + <refsect1> + <title>GENERAL OPTIONS</title> + + <variablelist> + <varlistentry> + <term>-i, --interactive</term> + <listitem> + <para> + Enable interactive mode. This will make ctdbd run in the + foreground and not detach from the terminal. In this mode + ctdbd will log to stderr. + </para> + <para> + By default ctdbd will detach itself and run in the + background as a daemon, logging to the configured + destination. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term>-?, --help</term> + <listitem> + <para> + Display a summary of options. + </para> + </listitem> + </varlistentry> + + </variablelist> + </refsect1> + + <refsect1> + <title>SEE ALSO</title> + <para> + <citerefentry><refentrytitle>ctdb</refentrytitle> + <manvolnum>1</manvolnum></citerefentry>, + + <citerefentry><refentrytitle>onnode</refentrytitle> + <manvolnum>1</manvolnum></citerefentry>, + + <citerefentry><refentrytitle>ctdb.conf</refentrytitle> + <manvolnum>5</manvolnum></citerefentry>, + + <citerefentry><refentrytitle>ctdb</refentrytitle> + <manvolnum>7</manvolnum></citerefentry>, + + <citerefentry><refentrytitle>ctdb-tunables</refentrytitle> + <manvolnum>7</manvolnum></citerefentry>, + + <ulink url="http://ctdb.samba.org/"/> + </para> + </refsect1> + + <refentryinfo> + <author> + <contrib> + This documentation was written by + Ronnie Sahlberg, + Amitay Isaacs, + Martin Schwenke + </contrib> + </author> + + <copyright> + <year>2007</year> + <holder>Andrew Tridgell</holder> + <holder>Ronnie Sahlberg</holder> + </copyright> + <legalnotice> + <para> + This program is free software; you can redistribute it and/or + modify it under the terms of the GNU General Public License as + published by the Free Software Foundation; either version 3 of + the License, or (at your option) any later version. + </para> + <para> + This program is distributed in the hope that it will be + useful, but WITHOUT ANY WARRANTY; without even the implied + warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR + PURPOSE. See the GNU General Public License for more details. + </para> + <para> + You should have received a copy of the GNU General Public + License along with this program; if not, see + <ulink url="http://www.gnu.org/licenses"/>. + </para> + </legalnotice> + </refentryinfo> + +</refentry> diff --git a/ctdb/doc/examples/11.natgw.options b/ctdb/doc/examples/11.natgw.options new file mode 100644 index 0000000..e2460cb --- /dev/null +++ b/ctdb/doc/examples/11.natgw.options @@ -0,0 +1,25 @@ +# NAT gateway configuration +# +# See ctdb.conf for main cluster configuration details +# +# Cluster provides file services on following IP addresses +# +# 10.1.1.101 - 10.1.1.106 +# +# When a node is not hosting any IPs, it cannot connect to network +# infrastructure (e.g. DNS, Active Directory, ...). +# +# Using NAT gateway feature of CTDB allows a node not hosting IPs to connect +# to network infrastructure using the additional CTDB_NATGW_PUBLIC_IP. + +# ---------- /etc/ctdb/natgw_nodes ---------- +# 192.168.1.1 +# 192.168.1.2 +# 192.168.1.3 +# ---------- /etc/ctdb/natgw_nodes ---------- +# +CTDB_NATGW_PUBLIC_IP=10.1.1.121/24 +CTDB_NATGW_PUBLIC_IFACE=eth1 +CTDB_NATGW_DEFAULT_GATEWAY=10.1.1.254 +CTDB_NATGW_PRIVATE_NETWORK=192.168.1.0/24 +CTDB_NATGW_NODES=/etc/ctdb/natgw_nodes diff --git a/ctdb/doc/examples/20.nfs_ganesha.check b/ctdb/doc/examples/20.nfs_ganesha.check new file mode 100644 index 0000000..3288f16 --- /dev/null +++ b/ctdb/doc/examples/20.nfs_ganesha.check @@ -0,0 +1,8 @@ +# nfs_ganesha +restart_every=2 +unhealthy_after=6 +service_stop_cmd="$CTDB_NFS_CALLOUT stop nfs" +service_start_cmd="$CTDB_NFS_CALLOUT start nfs" +service_check_cmd="$CTDB_NFS_CALLOUT check nfs" +# Ganesha initscript restarts rpc.statd and stack trace is desirable! +service_debug_cmd="program_stack_traces status 5" diff --git a/ctdb/doc/examples/91.lvs.options b/ctdb/doc/examples/91.lvs.options new file mode 100644 index 0000000..adb3660 --- /dev/null +++ b/ctdb/doc/examples/91.lvs.options @@ -0,0 +1,12 @@ +# LVS configuration +# +# ---------- /etc/ctdb/lvs_nodes ---------- +# 192.168.1.1 +# 192.168.1.2 +# 192.168.1.3 +# ---------- /etc/ctdb/lvs_nodes ---------- +# +CTDB_LVS_NODES=/etc/ctdb/lvs_nodes + +CTDB_LVS_PUBLIC_IP=10.1.1.101 +CTDB_LVS_PUBLIC_IFACE=eth1 diff --git a/ctdb/doc/examples/README b/ctdb/doc/examples/README new file mode 100644 index 0000000..a4ea222 --- /dev/null +++ b/ctdb/doc/examples/README @@ -0,0 +1,15 @@ +This directory includes... + +Sample CTDB configuration files: + + o ctdb.conf - Main configuration file + o 11.natgw.options - Options for the 11.natgw event script + o 91.lvs.options - Options for the 91.lvs event script + +Sample 60.nfs configuration for NFS ganesha - callout script and +.check file + + o nfs-ganesha-callout + o 20.nfs_ganesha.check + +See the comment at the top of nfs-ganesha-callout for instructions. diff --git a/ctdb/doc/examples/config_migrate.sh b/ctdb/doc/examples/config_migrate.sh new file mode 100755 index 0000000..874e96c --- /dev/null +++ b/ctdb/doc/examples/config_migrate.sh @@ -0,0 +1,741 @@ +#!/bin/sh + +# config_migrate.sh - migrate old ctdbd.conf file to new configuration files +# +# Input files are old-style CTDB configuration files, including: +# +# /etc/ctdb/ctdbd.conf +# /usr/local/etc/ctdb/ctdbd.conf +# /etc/sysconfig/ctdb +# /etc/defaults/ctdb +# +# These files are sourced by this script. They used to be sourced by +# ctdbd_wrapper, so this should not be too surprising. +# +# By default, the output directory is the given configuration +# directory. An alternate output directory can be specified if this +# isn't desired. +# +# The output directory will contain the following if non-empty: +# +# * ctdb.conf (may be empty) +# * script.options +# * ctdb.tunables +# * ctdb.sysconfig - consider installing as /etc/sysconfig/ctdb, +# /etc/default/ctdb, or similar +# * commands.sh - consider running commands in this files +# * README.warn - warnings about removed/invalid configuration options + +usage () +{ + cat <<EOF +usage: config_migrate.sh [-f] [-d <ctdb-config-dir>] [-o <out-dir>] <file> ... +EOF + exit 1 +} + +config_dir="" +out_dir="" +force=false + +while getopts "d:fho:?" opt ; do + case "$opt" in + d) config_dir="$OPTARG" ;; + f) force=true ;; + o) out_dir="$OPTARG" ;; + \?|h) usage ;; + esac +done +shift $((OPTIND - 1)) + +if [ $# -lt 1 ] ; then + usage +fi + +if [ -z "$config_dir" ] ; then + echo "Assuming \"/etc/ctdb\" as ctdb configuration directory" + echo "If that's not correct, please specify config dir with -d" + echo + config_dir="/etc/ctdb" +else + echo "Using \"$config_dir\" as ctdb configuration directory" + echo +fi + +if [ -z "$out_dir" ] ; then + echo "No output directory specified, using \"$config_dir\"" + echo + out_dir="$config_dir" +fi + +############################################################ + +# +# Output file handling +# + +out_file_check_and_create () +{ + _out_file="$1" + + if [ -f "$_out_file" ] ; then + if ! $force ; then + echo "Not overwriting existing file: ${_out_file}" >&2 + return 1 + fi + mv -v "$_out_file" "${_out_file}.convertsave" + fi + + touch "$_out_file" + + return 0 +} + +out_file_remove_if_empty () +{ + _out_file="$1" + + if [ ! -s "$_out_file" ] ; then + rm "$_out_file" + fi +} + +############################################################ + +# +# Option/tunable/service conversion and validity checking +# +# This is basically the data that drives most of the rest of the +# script +# + +# Convert a ctdbd.conf opt+val into a ctdb.conf section+opt+val +# +# If opt is matched and val is empty then output is printed, allowing +# this function to be reused to check if opt is valid. +# +# Note that for boolean options, the expected value and the new value +# form part of the data. +get_ctdb_conf_option () +{ + _opt="$1" + _val="$2" + + awk -v opt="${_opt}" -v val="${_val}" \ + '$3 == opt { + if (!$4 || !val || val == $4) { + if ($5) { + print $1, $2, $5 + } else { + print $1, $2, val + } + } + }' <<EOF +cluster node-address CTDB_NODE_ADDRESS +cluster cluster-lock CTDB_RECOVERY_LOCK +cluster transport CTDB_TRANSPORT +cluster leader-capability CTDB_CAPABILITY_RECMASTER no false +database lock-debug-script CTDB_DEBUG_LOCKS +database persistent-database-directory CTDB_DBDIR_PERSISTENT +database state-database-directory CTDB_DBDIR_STATE +database volatile-database-directory CTDB_DBDIR +event debug-script CTDB_DEBUG_HUNG_SCRIPT +legacy lmaster-capability CTDB_CAPABILITY_LMASTER no false +legacy realtime-scheduling CTDB_NOSETSCHED yes false +legacy script-log-level CTDB_SCRIPT_LOG_LEVEL +legacy start-as-disabled CTDB_START_AS_DISABLED yes true +legacy start-as-stopped CTDB_START_AS_STOPPED yes true +logging location CTDB_LOGGING +logging log-level CTDB_DEBUGLEVEL +EOF + +} + +# Check if an option will convert to a ctdb.conf option +check_ctdb_conf_option () +{ + _opt="$1" + + _out=$(get_ctdb_conf_option "$_opt" "") + [ -n "$_out" ] +} + +# Convert a ctdbd.conf tunable option into a ctdb.conf section+opt +# +# The difference between this and get_ctdb_conf_option() is that only +# the tunable part of the option is passed as opt and it is matched +# case-insensitively. +get_ctdb_conf_tunable_option () +{ + _opt="$1" + _val="$2" + + awk -v opt="${_opt}" -v val="${_val}" \ + 'tolower($3) == tolower(opt) { + if (!$4 || !val || (val == 0 ? 0 : 1) == $4) { + if ($5) { + print $1, $2, $5 + } else { + print $1, $2, val + } + } + }' <<EOF +database tdb-mutexes TDBMutexEnabled 0 false +failover disabled DisableIPFailover 1 true +EOF + +} + +# Check if a tunable will convert to a ctdb.conf option +check_ctdb_conf_tunable_option () +{ + _opt="$1" + + _out=$(get_ctdb_conf_tunable_option "$_opt" "") + [ -n "$_out" ] +} + +# Check if an option has been removed +check_removed_option () +{ + _option="$1" + + grep -Fqx "$_option" <<EOF +CTDB_BASE +CTDB_PIDFILE +CTDB_SOCKET +CTDB_EVENT_SCRIPT_DIR +CTDB_NOTIFY_SCRIPT +CTDB_PUBLIC_INTERFACE +CTDB_MAX_PERSISTENT_CHECK_ERRORS +CTDB_SHUTDOWN_TIMEOUT +CTDB_MONITOR_SWAP_USAGE +EOF +} + +# Check if an option is a valid script option +check_valid_script_option () +{ + _option="$1" + + grep -Fqx "$_option" <<EOF +# 10.interface +CTDB_PARTIALLY_ONLINE_INTERFACES +# 11.natgw +CTDB_NATGW_DEFAULT_GATEWAY +CTDB_NATGW_NODES +CTDB_NATGW_PRIVATE_NETWORK +CTDB_NATGW_PUBLIC_IFACE +CTDB_NATGW_PUBLIC_IP +CTDB_NATGW_STATIC_ROUTES +# 13.per_ip_routing +CTDB_PER_IP_ROUTING_CONF +CTDB_PER_IP_ROUTING_RULE_PREF +CTDB_PER_IP_ROUTING_TABLE_ID_LOW +CTDB_PER_IP_ROUTING_TABLE_ID_HIGH +# 90.lvs +CTDB_LVS_NODES +CTDB_LVS_PUBLIC_IFACE +CTDB_LVS_PUBLIC_IP +# 20.multipathd +CTDB_MONITOR_MPDEVICES +# 31.clamd +CTDB_CLAMD_SOCKET +# 48.netbios +CTDB_SERVICE_NMB +# 49.winbind +CTDB_SERVICE_WINBIND +# 50.samba +CTDB_SAMBA_CHECK_PORTS +CTDB_SAMBA_SKIP_SHARE_CHECK +CTDB_SERVICE_SMB +# 60.nfs +CTDB_NFS_CALLOUT +CTDB_NFS_CHECKS_DIR +CTDB_NFS_SKIP_SHARE_CHECK +CTDB_RPCINFO_LOCALHOST +CTDB_RPCINFO_LOCALHOST6 +CTDB_NFS_STATE_FS_TYPE +CTDB_NFS_STATE_MNT +# 70.iscsi +CTDB_START_ISCSI_SCRIPTS +# 00.ctdb +CTDB_MAX_CORRUPT_DB_BACKUPS +# 05.system +CTDB_MONITOR_FILESYSTEM_USAGE +CTDB_MONITOR_MEMORY_USAGE +# debug_hung_scripts.sh +CTDB_DEBUG_HUNG_SCRIPT_STACKPAT +EOF +} + +# Check if a tunable is valid +check_valid_tunable () +{ + _tunable="$1" + + grep -Fiqx "$_tunable" <<EOF +AllowClientDBAttach +AllowMixedVersions +AllowUnhealthyDBRead +ControlTimeout +DBRecordCountWarn +DBRecordSizeWarn +DBSizeWarn +DatabaseHashSize +DatabaseMaxDead +DeferredAttachTO +DisableIPFailover +ElectionTimeout +EnableBans +EventScriptTimeout +FetchCollapse +HopcountMakeSticky +IPAllocAlgorithm +KeepaliveInterval +KeepaliveLimit +LockProcessesPerDB +LogLatencyMs +MaxQueueDropMsg +MonitorInterval +MonitorTimeoutCount +NoIPFailback +NoIPTakeover +PullDBPreallocation +QueueBufferSize +RecBufferSizeLimit +RecLockLatencyMs +RecdFailCount +RecdPingTimeout +RecoverInterval +RecoverTimeout +RecoveryBanPeriod +RecoveryDropAllIPs +RecoveryGracePeriod +RepackLimit +RerecoveryTimeout +SeqnumInterval +StatHistoryInterval +StickyDuration +StickyPindown +TDBMutexEnabled +TakeoverTimeout +TickleUpdateInterval +TraverseTimeout +VacuumFastPathCount +VacuumInterval +VacuumMaxRunTime +VerboseMemoryNames +EOF +} + +# Check if a tunable has been removed +check_removed_tunable () +{ + _tunable="$1" + + grep -Fiqx "$_tunable" <<EOF +NoIPHostOnAllDisabled +VacuumLimit +EOF +} + +# Print a command to enable an event script for the given service +print_event_script_enable_command () +{ + _service="$1" + + _component="" + _script="" + case "$_service" in + samba) _component="legacy" ; _script="50.samba" ;; + winbind) _component="legacy" ; _script="49.winbind" ;; + apache2|httpd) _component="legacy" ; _script="41.httpd" ;; + clamd) _component="legacy" ; _script="31.clamd" ;; + iscsi) _component="legacy" ; _script="70.iscsi" ;; + nfs) _component="legacy" ; _script="60.nfs" ;; + vsftpd) _component="legacy" ; _script="40.vsftpd" ;; + esac + + if [ -z "$_script" ] ; then + return 1 + fi + + cat <<EOF +# Enable the ${_service} service +ctdb event script enable ${_component} ${_script} + +EOF +} + +# Check if the given service is valid +check_valid_service () +{ + _service="$1" + + print_event_script_enable_command "$_service" >/dev/null +} + +############################################################ + +# +# Utilities +# + +# List all options starting with "CTDB_" set in given configuration files +list_options () +{ + set | + sed -n 's|^\(CTDB_[^=]*\)=\(.*\)|\1 \2|p' | + while read -r _var _val ; do + # Strip quotes from value + _val=$(echo "$_val" | sed -e "s|^'||" -e "s|'\$||") + + echo "${_var} ${_val}" + done +} + +# List all tunables set in the given configuration files +list_tunables () +{ + list_options | + while read -r _opt _val ; do + case "$_opt" in + CTDB_SET_*) echo "${_opt#CTDB_SET_} ${_val}" ;; + esac + done +} + +# List all managed services according to the given configuration files +list_managed_services () +{ + # + # CTDB_MANAGES_<service>="yes" + # + list_options | + while read -r _opt _val ; do + case "$_opt" in + CTDB_MANAGES_*) : ;; + *) continue ;; + esac + + if [ "$_val" != "yes" ] ; then + continue + fi + + # Trim and downcase + echo "${_opt#CTDB_MANAGES_}" | tr '[:upper:]' '[:lower:]' + done + + # + # CTDB_MANAGED_SERVICES + # + for _service in $CTDB_MANAGED_SERVICES ; do + echo "$_service" + done +} + +############################################################ + +# +# Print warnings for removed and unknown options +# + + +# Print a warning as a bullet list item +# +# Arguments after the 1st are printed as a subsequent paragraph. +warn () +{ + bullet="$1" ; shift + + printf '* %s\n\n' "$bullet" + + if [ $# -gt 0 ] ; then + printf ' %s\n\n' "$*" + fi +} + +warn_about_CTDB_DBDIR_tmpfs_yes () +{ + if $ctdb_dbdir_tmpfs_magic ; then + warn "Option \"CTDB_DBDIR=tmpfs\" is no longer available:" \ + "Permanently mount a tmpfs filesystem on the volatile" \ + "database directory" + fi +} + +warn_about_unknown_managed_services () +{ + list_managed_services | + while read -r _s ; do + if check_valid_service "$_s" ; then + continue + fi + warn "Unknown service \"${_s}\" marked as managed" \ + "If this is a 3rd party service, please enable it manually" + done +} + +warn_about_removed_and_unknown_options () +{ + list_options | + while read -r _opt _val ; do + if check_ctdb_conf_option "$_opt" ; then + continue + fi + + if check_valid_script_option "$_opt" ; then + continue + fi + + case "$_opt" in + CTDB_MANAGED_SERVICES|\ + CTDB_MANAGES_*|\ + CTDB_SET_*|\ + CTDB_NODES|\ + CTDB_PUBLIC_ADDRESSES|\ + CTDB_MAX_OPEN_FILES|\ + CTDB_SUPPRESS_COREFILE) + # Handled elsewhere + continue + ;; + esac + + if check_removed_option "$_opt" ; then + warn "Option \"${_opt}\" is no longer available" \ + "Please see the WHATSNEW.txt" + continue + fi + + warn "Option \"${_opt}\" is unknown" + done +} + +warn_about_removed_and_unknown_tunables () +{ + list_tunables | + while read -r _var _val ; do + if check_valid_tunable "$_var" ; then + continue + fi + + if check_removed_tunable "$_var" ; then + warn "Tunable \"${_var}\" is no longer available" \ + "Please see the WHATSNEW.txt" + continue + fi + + warn "Tunable \"${_var}\" is unknown" + done +} + +############################################################ + +# +# Top-level file builders +# + +build_ctdb_conf () +{ + _out_file="$1" + + out_file_check_and_create "$_out_file" || return + + list_options | + while read -r _opt _val ; do + case "$_opt" in + CTDB_SET_*) + _opt="${_opt#CTDB_SET_}" + _out=$(get_ctdb_conf_tunable_option "$_opt" "$_val") + ;; + *) + _out=$(get_ctdb_conf_option "$_opt" "$_val") + esac + if [ -z "$_out" ] ; then + continue + fi + + # $_out is section and key, replace dashes with spaces + # Intentional word splitting + # shellcheck disable=SC2086 + set -- $_out + _section=$(echo "$1" | sed -e 's|-| |g') + _key=$(echo "$2" | sed -e 's|-| |g') + _newval="$3" + + if ! grep -Fqx "[${_section}]" "$_out_file" ; then + # Add blank line if file is not empty + if [ -s "$_out_file" ] ; then + echo >>"$_out_file" + fi + # Create section at end of file + echo "[${_section}]" >>"$_out_file" + fi + + # Must escape leading TAB or sed eats it + sed -i -e "/\\[${_section}\\]/a\ +\\ ${_key} = ${_newval} +" "$_out_file" + + done + +} + +build_script_options () +{ + _out_file="$1" + + out_file_check_and_create "$_out_file" || return + + list_options | + while read -r _var _val ; do + if check_valid_script_option "$_var" ; then + echo "${_var}=${_val}" + fi + done >>"$_out_file" + + out_file_remove_if_empty "$_out_file" +} + +build_ctdb_tunables () +{ + _out_file="$1" + + out_file_check_and_create "$_out_file" || return + + list_tunables | + while read -r _var _val ; do + if check_ctdb_conf_tunable_option "$_var" ; then + continue + fi + if ! check_valid_tunable "$_var" ; then + continue + fi + echo "${_var}=${_val}" + done >>"$_out_file" + + out_file_remove_if_empty "$_out_file" +} + +build_ctdb_sysconfig () +{ + _out_file="$1" + + out_file_check_and_create "$_out_file" || return + + if [ -n "$CTDB_SUPPRESS_COREFILE" ] ; then + if [ "$CTDB_SUPPRESS_COREFILE" = "yes" ] ; then + echo "ulimit -c 0" + else + echo "ulimit -c unlimited" + fi >>"$_out_file" + fi + + if [ -n "$CTDB_MAX_OPEN_FILES" ] ; then + echo "ulimit -n ${CTDB_MAX_OPEN_FILES}" >>"$_out_file" + fi + + out_file_remove_if_empty "$_out_file" +} + +build_commands_sh () +{ + _out_file="$1" + + out_file_check_and_create "$_out_file" || return + + # + # Enable script for managed services + # + list_managed_services | + while read -r _service ; do + print_event_script_enable_command "$_service" + done >>"$_out_file" + + # + # CTDB_NODES no longer available + # + if [ -n "$CTDB_NODES" ] ; then + if [ "$CTDB_NODES" = "${config_dir}/nodes" ] ; then + cat <<EOF +# CTDB_NODES=${CTDB_NODES} +# Looks like the standard location. Nothing to do. + +EOF + else + cat <<EOF +# CTDB_NODES=${CTDB_NODES} +# Looks like a non-standard location. Use the default location +# in the configuration directory or create a symlink. +ln -s "$CTDB_NODES" "${config_dir}/nodes" + +EOF + fi >>"$_out_file" + fi + + # + # CTDB_PUBLIC_ADDRESSES no longer available + # + if [ -n "$CTDB_PUBLIC_ADDRESSES" ] ; then + _pa="public_addresses" + if [ "$CTDB_PUBLIC_ADDRESSES" = "${config_dir}/${_pa}" ] ; then + cat <<EOF +# CTDB_PUBLIC_ADDRESSES=${CTDB_PUBLIC_ADDRESSES} +# Looks like the standard location. Nothing to do. + +EOF + else + cat <<EOF +# CTDB_PUBLIC_ADDRESSES=${CTDB_PUBLIC_ADDRESSES} +# Looks like a non-standard location. Use the default location +# in the configuration directory or create a symlink. +ln -s "$CTDB_PUBLIC_ADDRESSES" "${config_dir}/${_pa}" + +EOF + fi >>"$_out_file" + fi + + out_file_remove_if_empty "$_out_file" +} + +build_README_warn () +{ + _out_file="$1" + + out_file_check_and_create "$_out_file" || return + + { + warn_about_CTDB_DBDIR_tmpfs_yes + warn_about_unknown_managed_services + warn_about_removed_and_unknown_options + warn_about_removed_and_unknown_tunables + } >>"$_out_file" + + out_file_remove_if_empty "$_out_file" +} + +############################################################ + +mkdir -p "$out_dir" || exit 1 + +# Source the input files +for i ; do + # Unknown non-constant source + # shellcheck disable=SC1090 + . "$i" +done + +# Special case +ctdb_dbdir_tmpfs_magic=false +if [ "$CTDB_DBDIR" = "tmpfs" ] ; then + ctdb_dbdir_tmpfs_magic=true + unset CTDB_DBDIR +fi + +build_ctdb_conf "${out_dir}/ctdb.conf" +build_script_options "${out_dir}/script.options" +build_ctdb_tunables "${out_dir}/ctdb.tunables" +build_ctdb_sysconfig "${out_dir}/ctdb.sysconfig" +build_commands_sh "${out_dir}/commands.sh" +build_README_warn "${out_dir}/README.warn" diff --git a/ctdb/doc/examples/config_migrate.test_input b/ctdb/doc/examples/config_migrate.test_input new file mode 100644 index 0000000..e823fd4 --- /dev/null +++ b/ctdb/doc/examples/config_migrate.test_input @@ -0,0 +1,50 @@ +CTDB_BASE="/etc/ctdb" + +CTDB_RECOVERY_LOCK="/some/place/on/shared/storage" + +CTDB_NODES="/etc/ctdb/nodes" + +CTDB_PUBLIC_ADDRESSES="/clusterfs/.ctdb/public_addresses" + +CTDB_SOCKET="/var/run/ctdb.sock" + +CTDB_MANAGES_SAMBA="yes" +CTDB_MANAGES_WINBIND="yes" +CTDB_MANAGES_NFS="yes" +CTDB_MANAGED_SERVICES="vsftpd" +CTDB_MANAGES_FOOBAR="yes" + +CTDB_MAX_OPEN_FILES=10000 + +CTDB_LOGGING="syslog" +CTDB_DEBUGLEVEL="ERROR" + +CTDB_SAMBA_CHECK_PORTS="445" +CTDB_SAMBA_SKIP_SHARE_CHECK="yes" + +CTDB_START_AS_STOPPED="yes" +CTDB_CAPABILITY_RECMASTER="no" +CTDB_CAPABILITY_LMASTER="yes" + +CTDB_NOSETSCHED="yes" + +CTDB_FOO="bar" + +CTDB_NATGW_PUBLIC_IP=10.1.1.121/24 +CTDB_NATGW_PUBLIC_IFACE=eth1 +CTDB_NATGW_DEFAULT_GATEWAY=10.1.1.254 +CTDB_NATGW_PRIVATE_NETWORK=192.168.1.0/24 +CTDB_NATGW_NODES=/etc/ctdb/natgw_nodes + +CTDB_SET_TDBMutexEnabled=0 +CTDB_SET_IPALLOCALGORITHM=0 +CTDB_SET_noiphostonalldisabled=0 +CTDB_SET_DisableIpFailover=1 +CTDB_SET_foobar=0 + +CTDB_SUPPRESS_COREFILE="yes" +CTDB_MAX_OPEN_FILES="1048576" + +CTDB_NOTIFY_SCRIPT=/etc/ctdb/notify-custom.sh + +CTDB_DBDIR=tmpfs diff --git a/ctdb/doc/examples/ctdb.conf b/ctdb/doc/examples/ctdb.conf new file mode 100644 index 0000000..3a8ccc6 --- /dev/null +++ b/ctdb/doc/examples/ctdb.conf @@ -0,0 +1,73 @@ +# +# CTDB configuration for simple cluster +# +# This is the sample configuration for a 3-node CTDB cluster providing file +# services via Samba and NFS. +# +# Cluster details: +# +# internal network (192.168.1.0/24) +# -------+----------------------+-----------------------+---------- +# | | | +# | | | +# eth0 | 192.168.1.1 eth0 | 192.168.1.2 eth0 | 192.168.1.3 +# +-----+-----+ +-----+-----+ +-----+-----+ +# | | | | | | +# | Node 1 | | Node 2 | | Node 3 | +# | | | | | | +# +-----+-----+ +-----+-----+ +-----+-----+ +# eth1 | 10.1.1.1 eth1 | 10.1.1.2 eth1 | 10.1.1.3 +# | | | +# | | | +# -------+----------------------+-----------------------+---------- +# public network (10.1.1.0/24) +# +# +# Storage details: +# +# Each node has a shared storage - /shared +# +# +# Service details: +# +# Cluster provides file services on following IP addresses +# +# 10.1.1.101 - 10.1.1.106 +# +# Each node also has a fixed IP address on public network. This is used to +# communicate to network infrastructure (e.g. DNS, Active Directory, ...). +# Make sure that file services are not available on these fixed IP addresses +# (e.g. network filtering, using cluster hostname instead of IPs) + +[logging] + # Enable logging to syslog + location = syslog + + # Default log level + log level = NOTICE + +[cluster] + cluster lock = /shared/cluster.lock + +# +# Nodes configuration +# +# ---------- /etc/ctdb/nodes ---------- +# 192.168.1.1 +# 192.168.1.2 +# 192.168.1.3 +# ---------- /etc/ctdb/nodes ---------- +# +# +# Public addresses configuration +# +# ---------- /etc/ctdb/public_addresses ---------- +# 10.1.1.101/24 eth1 +# 10.1.1.102/24 eth1 +# 10.1.1.103/24 eth1 +# 10.1.1.104/24 eth1 +# 10.1.1.105/24 eth1 +# 10.1.1.106/24 eth1 +# ---------- /etc/ctdb/public_addresses ---------- +# +# Event script configuration - see *.options examples diff --git a/ctdb/doc/examples/ctdb.spec.in b/ctdb/doc/examples/ctdb.spec.in new file mode 100644 index 0000000..addb7e1 --- /dev/null +++ b/ctdb/doc/examples/ctdb.spec.in @@ -0,0 +1,291 @@ +%define with_systemd %{?_with_systemd: 1} %{?!_with_systemd: 0} +%define initdir %{_sysconfdir}/init.d +Name: ctdb +Summary: Clustered TDB +Vendor: Samba Team +Packager: Samba Team <samba@samba.org> +Version: @VERSION@ +Release: @RELEASE@ +Epoch: 0 +License: GNU GPL version 3 +Group: System Environment/Daemons +URL: http://ctdb.samba.org/ + +Source: ctdb-%{version}.tar.gz + +# Packages +Requires: coreutils, sed, gawk, iptables, iproute, procps, ethtool, sudo +# Commands - package name might vary +Requires: /usr/bin/killall, /bin/kill, /bin/ss + +Provides: ctdb = %{version} + +Prefix: /usr +BuildRoot: %{_tmppath}/%{name}-%{version}-root + +# Allow build with system libraries +# To enable, run rpmbuild with, +# "--with system_talloc" +# "--with system_tdb" +# "--with system_tevent" +#%define with_included_talloc %{?_with_system_talloc: 0} %{?!_with_system_talloc: 1} +#%define with_included_tdb %{?_with_system_tdb: 0} %{?!_with_system_tdb: 1} +#%define with_included_tevent %{?_with_system_tevent: 0} %{?!_with_system_tevent: 1} + +%define with_included_talloc 0 +%define with_included_tevent 0 +%define with_included_tdb 0 + +# Required minimum library versions when building with system libraries +%define libtalloc_version 2.0.8 +%define libtdb_version 1.3.11 +%define libtevent_version 0.9.16 + +%if ! %with_included_talloc +BuildRequires: libtalloc-devel >= %{libtalloc_version} +Requires: libtalloc >= %{libtalloc_version} +%endif +%if ! %with_included_tdb +BuildRequires: libtdb-devel >= %{libtdb_version} +Requires: libtdb >= %{libtdb_version} +%endif +%if ! %with_included_tevent +BuildRequires: libtevent-devel >= %{libtevent_version} +Requires: libtevent >= %{libtevent_version} +%endif + +# To build the ctdb-pcp-pmda package, run rpmbuild with "--with pmda" +%define with_pcp_pmda %{?_with_pmda: 1} %{?!_with_pmda: 0} +%if %with_pcp_pmda +BuildRequires: pcp-libs-devel +%endif + +%if %{with_systemd} +BuildRequires: systemd-units +%endif + +%description +ctdb is the clustered database used by samba + +####################################################################### + + + +%prep +%setup -q +# setup the init script and sysconfig file +%setup -T -D -n ctdb-%{version} -q + +%build + +## check for ccache +if ccache -h >/dev/null 2>&1 ; then + CC="ccache gcc" +else + CC="gcc" +fi + +export CC + +CFLAGS="$RPM_OPT_FLAGS $EXTRA -D_GNU_SOURCE" \ +./configure \ + --builtin-libraries=replace,popt \ + --bundled-libraries=!talloc,!tevent,!tdb \ + --minimum-library-version=talloc:%libtalloc_version,tdb:%libtdb_version,tevent:%libtevent_version \ +%if %with_pcp_pmda + --enable-pmda \ +%endif + --prefix=%{_prefix} \ + --includedir=%{_includedir}/ctdb \ + --libdir=%{_libdir} \ + --libexecdir=%{_libexecdir} \ + --sysconfdir=%{_sysconfdir} \ + --mandir=%{_mandir} \ + --localstatedir=%{_localstatedir} + +make -j + +%install +# Clean up in case there is trash left from a previous build +rm -rf $RPM_BUILD_ROOT + +# Create the target build directory hierarchy +mkdir -p $RPM_BUILD_ROOT%{_sysconfdir}/sudoers.d + +DESTDIR=$RPM_BUILD_ROOT make -j install + +install -m644 config/ctdb.conf $RPM_BUILD_ROOT%{_sysconfdir}/ctdb +install -m644 config/ctdb.tunables $RPM_BUILD_ROOT%{_sysconfdir}/ctdb +install -m644 config/script.options $RPM_BUILD_ROOT%{_sysconfdir}/ctdb + +mkdir -p $RPM_BUILD_ROOT%{_sysconfdir}/sysconfig +install -m644 config/ctdb.sysconfig $RPM_BUILD_ROOT%{_sysconfdir}/sysconfig/ctdb + +%if %{with_systemd} +mkdir -p $RPM_BUILD_ROOT%{_unitdir} +install -m 755 config/ctdb.service $RPM_BUILD_ROOT%{_unitdir} +%else +mkdir -p $RPM_BUILD_ROOT%{initdir} +install -m755 config/ctdb.init $RPM_BUILD_ROOT%{initdir}/ctdb +%endif + +# This is a hack. All documents should be installed in /usr/share/doc. +cp config/events/README README.eventscripts +rm -f $RPM_BUILD_ROOT%{_sysconfdir}/ctdb/nfs-checks.d/README +cp config/nfs-checks.d/README README.nfs-checks.d +cp config/notification.README README.notification + +# Remove "*.old" files +find $RPM_BUILD_ROOT -name "*.old" -exec rm -f {} \; + +%clean +rm -rf $RPM_BUILD_ROOT + +%post + +# If mandatory 00.ctdb event script is not enabled then enable it and +# some other scripts. The assumption here is that this is a +# first-time install or an upgrade to a version that requires event +# scripts to be enabled via symlinks. +required_script="00.ctdb" +required_path="%{_sysconfdir}/ctdb/events/legacy/${required_script}.script" +if [ ! -L "$required_path" ] && [ ! -e "$required_path" ] ; then + default_scripts="${required_script} + 01.reclock + 05.system + 10.interface + " + for t in $default_scripts ; do + tgt="%{_datadir}/ctdb/events/legacy/${t}.script" + name="%{_sysconfdir}/ctdb/events/legacy/${t}.script" + # Directory is created via install and files + ln -s "$tgt" "$name" + done +fi + +%preun + +# Uninstall, not upgrade. Clean up by removing any remaining links. +if [ "$1" = "0" ] ; then + for i in "%{_sysconfdir}/ctdb/events/legacy/"*.script ; do + if [ -L "$i" ] ; then + rm -f "$i" + fi + done +fi + +%files +%defattr(-,root,root) + +%config(noreplace) %{_sysconfdir}/ctdb/ctdb.conf +%config(noreplace) %{_sysconfdir}/ctdb/ctdb.tunables +%config(noreplace) %{_sysconfdir}/ctdb/script.options +%{_sysconfdir}/ctdb/notify.sh +%config(noreplace) %{_sysconfdir}/ctdb/debug-hung-script.sh +%config(noreplace) %{_sysconfdir}/ctdb/ctdb-crash-cleanup.sh +%config(noreplace) %{_sysconfdir}/ctdb/debug_locks.sh + +%config(noreplace, missingok) %{_sysconfdir}/sysconfig/ctdb + +%if %{with_systemd} +%{_unitdir}/ctdb.service +%else +%attr(755,root,root) %{initdir}/ctdb +%endif + +%doc README COPYING +%doc README.eventscripts README.notification +%doc doc/recovery-process.txt +%doc doc/cluster_mutex_helper.txt +%doc doc/*.html +%doc doc/examples +%{_sysconfdir}/sudoers.d/ctdb +%dir %{_sysconfdir}/ctdb +%{_sysconfdir}/ctdb/functions +%dir %{_sysconfdir}/ctdb/events +%{_sysconfdir}/ctdb/events/* +%dir %{_sysconfdir}/ctdb/nfs-checks.d +%config(noreplace) %{_sysconfdir}/ctdb/nfs-checks.d/00.portmapper.check +%config(noreplace) %{_sysconfdir}/ctdb/nfs-checks.d/10.status.check +%config(noreplace) %{_sysconfdir}/ctdb/nfs-checks.d/20.nfs.check +%config(noreplace) %{_sysconfdir}/ctdb/nfs-checks.d/30.nlockmgr.check +%config(noreplace) %{_sysconfdir}/ctdb/nfs-checks.d/40.mountd.check +%config(noreplace) %{_sysconfdir}/ctdb/nfs-checks.d/50.rquotad.check +%{_sysconfdir}/ctdb/statd-callout +%{_sysconfdir}/ctdb/nfs-linux-kernel-callout +%{_sbindir}/ctdbd +%{_bindir}/ctdb +%{_bindir}/ping_pong +%{_bindir}/ltdbtool +%{_bindir}/ctdb_diagnostics +%{_bindir}/onnode +%dir %{_libexecdir}/ctdb +%{_libexecdir}/ctdb/* +%dir %{_libdir}/ctdb +%{_libexecdir}/tdb_mutex_check +%{_libdir}/ctdb/lib* +%dir %{_datadir}/ctdb/events +%{_datadir}/ctdb/events/* +%{_mandir}/man1/ctdb.1.gz +%{_mandir}/man1/ctdb_diagnostics.1.gz +%{_mandir}/man1/ctdbd.1.gz +%{_mandir}/man1/onnode.1.gz +%{_mandir}/man1/ltdbtool.1.gz +%{_mandir}/man1/ping_pong.1.gz +%{_mandir}/man5/ctdb.conf.5.gz +%{_mandir}/man5/ctdb-script.options.5.gz +%{_mandir}/man5/ctdb.sysconfig.5.gz +%{_mandir}/man7/ctdb.7.gz +%{_mandir}/man7/ctdb-statistics.7.gz +%{_mandir}/man7/ctdb-tunables.7.gz +%attr(0700,root,root) %dir %{_localstatedir}/lib/ctdb +%attr(0700,root,root) %dir %{_localstatedir}/lib/ctdb/* +%attr(0700,root,root) %dir %{_localstatedir}/run/ctdb + + +%package devel +Summary: CTDB development libraries +Group: Development/Libraries + +%description devel +development libraries for ctdb + +%files devel +%defattr(-,root,root) + +%package tests +Summary: CTDB test suite +Group: Development/Tools +Requires: ctdb = %{version} +Requires: nc, tcpdump + +%description tests +test suite for ctdb + +%files tests +%defattr(-,root,root) +%dir %{_datadir}/%{name}/tests +%{_datadir}/%{name}/tests/* +%dir %{_libexecdir}/%{name}/tests +%{_libexecdir}/%{name}/tests/* +%{_bindir}/ctdb_run_tests +%{_bindir}/ctdb_run_cluster_tests +%{_bindir}/ctdb_local_daemons +%doc tests/README + +%if %with_pcp_pmda + +%package pcp-pmda +Summary: CTDB PCP pmda support +Group: Development/Tools +Requires: ctdb = %{version} +Requires: pcp-libs + +%description pcp-pmda +Performance Co-Pilot (PCP) support for CTDB + +%files pcp-pmda +%dir %{_localstatedir}/lib/pcp/pmdas/ctdb +%{_localstatedir}/lib/pcp/pmdas/ctdb/* + +%endif diff --git a/ctdb/doc/examples/nfs-ganesha-callout b/ctdb/doc/examples/nfs-ganesha-callout new file mode 100755 index 0000000..7c7b074 --- /dev/null +++ b/ctdb/doc/examples/nfs-ganesha-callout @@ -0,0 +1,352 @@ +#!/bin/sh + +# This is an example CTDB NFS callout script for Ganesha. It is based +# on the last version of 60.ganesha shipped with CTDB. As such, it +# does not try to monitor RPC services that were not monitored by +# 60.ganesha - this might be a useful improvement. It has also not +# been properly tested. + +# You should check your version of NFS Ganesha to see if it ships with +# a newer callout. + +# To use this: +# +# * Set CTDB_NFS_CALLOUT in your CTDB configuration to point to (a +# copy of) this script, making sure it is executable. +# +# * Create a new directory alongside the nfs-checks.d directory, for +# example nfs-checks-ganesha.d. Install 20.nfs-ganesha.check in +# this directory. Symlink to any other check files from +# nfs-checks.d that should still be used, such as +# 00.portmapper.check. Set CTDB_NFS_CHECKS_DIR to point to this new +# directory of check files. +# +# * It is recommended, but not required, to install the grace_period +# script (usually shipped in a utils package for NFS-Ganesha) to +# /usr/bin/grace_period + +# I (Martin Schwenke) hereby relicense all of my contributions to this +# callout (and, previously, to 60.ganesha) to a license compatible +# with NFS Ganesha (right now this is LGPLv3, but I'm flexible). +# There may be other contributions to be considered for relicensing, +# particularly those in commit 28cbe527d47822f870e8252495ab2a1c8fddd12f. + +###################################################################### + +# Exit on 1st error +set -e + +# Filesystem type and mount point for the (typically clustered) +# volume that will contain the NFS-Ganesha state. +state_fs="${CTDB_NFS_STATE_FS_TYPE:-gpfs}" +state_dir="${CTDB_NFS_STATE_MNT}" # No sane default. + +# To change the following, edit the default values below. Do not set +# these - they aren't configuration variables, just hooks for testing. +nfs_exports_file="${CTDB_NFS_EXPORTS_FILE:-/etc/ganesha/ganesha.conf}" +nfs_service="${CTDB_NFS_SERVICE:-nfs-ganesha}" +ganesha_rec_subdir=${CTDB_GANESHA_REC_SUBDIR:-.ganesha} +procfs=${PROCFS_PATH:-/proc} + +case "$state_fs" in +gpfs) + GANRECDIR="/var/lib/nfs/ganesha" + ;; +glusterfs) + if [ -z "${state_dir}" ]; then + echo "CTDB_NFS_STATE_MNT not defined for GlusterFS" + exit 1 + fi + host=$(hostname) + NODESTATEDIR="$state_dir/nfs-ganesha/$host" + GANSTATEDIR="$state_dir/nfs-ganesha/.noderefs" + NODESTATELN="$GANSTATEDIR/$host" + ;; +esac + + +################################################## + +usage () +{ + _c=$(basename "$0") + cat <<EOF +usage: $_c { shutdown | startup } + $_c { stop | start | check } nfs + $_c { releaseip | takeip } + $_c { monitor-list-shares } +EOF + exit 1 +} + + +################################################## +# Basic service stop and start + +basic_stop () +{ + case "$1" in + nfs) + service "$nfs_service" stop + ;; + *) + usage + esac +} + +basic_start () +{ + case "$1" in + nfs) + service "$nfs_service" start + ;; + *) + usage + esac +} + +################################################## +# "stop" and "start" options for restarting + +service_stop () +{ + case "$1" in + nfs) + basic_stop "nfs" + ;; + nlockmgr) + # Do nothing - used by statd-callout + : + ;; + *) + usage + esac +} + +service_start () +{ + case "$1" in + nfs) + basic_start "nfs" + ;; + nlockmgr) + # Do nothing - used by statd-callout + : + ;; + *) + usage + esac +} + +################################################## +# Nitty gritty - monitoring and IP handling + +# Check that a symlink exists, create it otherwise. +# Usage: check_ln <TARGET> <LINK> +check_ln () +{ + if [ ! -L "${2}" ] ; then + rm -vrf "${2}" + else + _t=$(readlink "${2}") + if [ "$_t" != "${1}" ] ; then + rm -v "${2}" + fi + fi + # This is not an "else". It also re-creates the link if it was + # removed above! + if [ ! -e "${2}" ]; then + ln -sfv "${1}" "${2}" + fi +} + +# Return 'active' if the shared filesystem is accessible. +get_cluster_fs_state () +{ + case $state_fs in + gpfs) + /usr/lpp/mmfs/bin/mmgetstate | awk 'NR == 4 { print $3 }' + ;; + glusterfs) + # Since we're past create_ganesha_recdirs(), we're active. + echo "active" + ;; + *) + echo "File system $state_fs not supported" + exit 1 + ;; + esac +} + +create_ganesha_recdirs () +{ + if ! _mounts=$(mount | grep "$state_fs"); then + echo "Failed to find mounts of type $state_fs" + exit 1 + fi + if [ -z "$_mounts" ]; then + echo "startup $state_fs not ready" + exit 0 + fi + + case $state_fs in + gpfs) + _mntpt=$(echo "$_mounts" | sort | awk 'NR == 1 {print $3}') + _link_dst="${_mntpt}/${ganesha_rec_subdir}" + mkdir -vp "$_link_dst" + check_ln "$_link_dst" "$GANRECDIR" + ;; + glusterfs) + [ -d /var/lib/nfs.backup ] || \ + mv /var/lib/nfs /var/lib/nfs.backup + check_ln "$NODESTATEDIR" /var/lib/nfs + + mkdir -p "${NODESTATEDIR}/ganesha/v4recov" + mkdir -p "${NODESTATEDIR}/ganesha/v4old" + mkdir -p "${NODESTATEDIR}/statd/sm" + mkdir -p "${NODESTATEDIR}/statd/sm.bak" + touch "${NODESTATEDIR}/state" + touch "${NODESTATEDIR}/statd/state" + + mkdir -p "$GANSTATEDIR" + check_ln "$NODESTATEDIR" "$NODESTATELN" + for _dir in "${GANSTATEDIR}/"* ; do + # Handle no directories case + if [ ! -d "$_dir" ] ; then + break + fi + + _node="${_dir##*/}" # basename + if [ "${_node}" != "${host}" ]; then + check_ln "${GANSTATEDIR}/${_node}/ganesha" \ + "${NODESTATEDIR}/ganesha/${_node}" + check_ln "${GANSTATEDIR}/${_node}/statd" \ + "${NODESTATEDIR}/statd/${_node}" + fi + done + ;; + esac +} + +service_check () +{ + create_ganesha_recdirs + + # Always succeed if cluster filesystem is not active + _cluster_fs_state=$(get_cluster_fs_state) + if [ "$_cluster_fs_state" != "active" ] ; then + return 0 + fi + + # Check that NFS Ganesha is running, according to PID file + _pidfile="/var/run/ganesha.pid" + _ganesha="/usr/bin/ganesha.nfsd" + if ! { read -r _pid < "$_pidfile" && \ + grep "$_ganesha" "${procfs}/${_pid}/cmdline" ; } >/dev/null 2>&1 + then + + echo "ERROR: NFS Ganesha not running according to PID file" + return 1 + fi + + return 0 +} + +#------------------------------------------------- + +nfs_releaseip () +{ + if [ -x "/usr/bin/grace_period" ]; then + /usr/bin/grace_period "2:${2}" + else + dbus-send --print-reply --system --dest=org.ganesha.nfsd \ + /org/ganesha/nfsd/admin org.ganesha.nfsd.admin.grace \ + string:"2:${2}" + fi +} + +nfs_takeip () +{ + case $state_fs in + glusterfs) + check_ln "$NODESTATEDIR" "${GANSTATEDIR}/${2}" + ;; + esac + if [ -x "/usr/bin/grace_period" ]; then + /usr/bin/grace_period "5:${2}" + else + dbus-send --print-reply --system --dest=org.ganesha.nfsd \ + /org/ganesha/nfsd/admin org.ganesha.nfsd.admin.grace \ + string:"5:${2}" + fi +} + +################################################## +# service init startup and final shutdown + +nfs_shutdown () +{ + basic_stop "nfs" +} + +nfs_startup () +{ + basic_stop "nfs" || true + + create_ganesha_recdirs + + basic_start "nfs" + _f="${procfs}/sys/net/ipv4/tcp_tw_recycle" + if [ -f "$_f" ] ; then + echo 1 >"$_f" + fi +} + +################################################## +# list share directories + +nfs_monitor_list_shares () +{ + grep Path "$nfs_exports_file" | + cut -f2 -d\" | + sort -u +} + +################################################## + +nfs_register () +{ + cat <<EOF +shutdown +startup +stop +start +check +releaseip +takeip +monitor-list-shares +EOF +} + +################################################## + +action="$1" +shift + +case "$action" in +shutdown) nfs_shutdown ;; +startup) nfs_startup ;; +stop) service_stop "$1" ;; +start) service_start "$1" ;; +check) service_check "$1" ;; +releaseip) nfs_releaseip "$@" ;; +takeip) nfs_takeip "$@" ;; +monitor-list-shares) nfs_monitor_list_shares ;; +register) nfs_register ;; +monitor-pre|monitor-post|releaseip-pre|takeip-pre) + # Not required/implemented + : + ;; +*) + usage +esac diff --git a/ctdb/doc/ltdbtool.1.xml b/ctdb/doc/ltdbtool.1.xml new file mode 100644 index 0000000..790db0c --- /dev/null +++ b/ctdb/doc/ltdbtool.1.xml @@ -0,0 +1,300 @@ +<?xml version="1.0" encoding="iso-8859-1"?> +<!DOCTYPE refentry + PUBLIC "-//OASIS//DTD DocBook XML V4.5//EN" + "http://www.oasis-open.org/docbook/xml/4.5/docbookx.dtd"> +<refentry id="ltdbtool.1"> + + <refmeta> + <refentrytitle>ltdbtool</refentrytitle> + <manvolnum>1</manvolnum> + <refmiscinfo class="source">ctdb</refmiscinfo> + <refmiscinfo class="manual">CTDB - clustered TDB database</refmiscinfo> + </refmeta> + + <refnamediv> + <refname>ltdbtool</refname> + <refpurpose>manipulate CTDB's local TDB files</refpurpose> + </refnamediv> + + <refsynopsisdiv> + <cmdsynopsis> + <command>ltdbtool</command> + <arg rep="repeat"><replaceable>OPTION</replaceable></arg> + <arg choice="req"><replaceable>COMMAND</replaceable></arg> + <arg><replaceable>COMMAND-ARGS</replaceable></arg> + </cmdsynopsis> +</refsynopsisdiv> + + <refsect1> + <title>DESCRIPTION</title> + + <para> + ltdbtool is a utility to manipulate CTDB's local TDB databases + (LTDBs) without connecting to a CTDB daemon. + </para> + + <para> + It can be used to: + </para> + + <itemizedlist> + <listitem> + <para> + dump the contents of a LTDB, optionally printing the CTDB + record header information, + </para> + </listitem> + <listitem> + <para> + convert between an LTDB and a non-clustered tdb + by adding or removing CTDB headers and + </para> + </listitem> + <listitem> + <para>convert between 64 and 32 bit LTDBs where the CTDB record + headers differ by 4 bytes of padding. + </para> + </listitem> + </itemizedlist> + </refsect1> + + <refsect1> + <title>OPTIONS</title> + + <variablelist> + <varlistentry><term>-e</term> + <listitem> + <para> + Dump empty records. These are normally excluded. + </para> + </listitem> + </varlistentry> + + <varlistentry><term>-p</term> + <listitem> + <para> + Dump with header information, similar to "ctdb catdb". + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term> + -s + <group choice="req"> + <arg choice="plain">0</arg> + <arg choice="plain">32</arg> + <arg choice="plain">64</arg> + </group> + </term> + <listitem> + <para> + Specify how to determine the CTDB record header size + for the input database: + <variablelist spacing="normal"> + <varlistentry><term>0</term> + <listitem> + <para>no CTDB header</para> + </listitem> + </varlistentry> + <varlistentry><term>32</term> + <listitem> + <para>CTDB header size of a 32 bit system (20 bytes)</para> + </listitem> + </varlistentry> + <varlistentry><term>64</term> + <listitem> + <para>CTDB header size of a 64 bit system (24 bytes)</para> + </listitem> + </varlistentry> + </variablelist> + The default is 32 or 64 depending on the system architecture. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term> + -o + <group choice="req"> + <arg choice="plain">0</arg> + <arg choice="plain">32</arg> + <arg choice="plain">64</arg> + </group> + </term> + <listitem> + <para> + Specify how to determine the CTDB record header size + for the output database, see -s. + </para> + </listitem> + </varlistentry> + + <varlistentry><term>-S <parameter>SIZE</parameter></term> + <listitem> + <para> + Explicitly specify the CTDB record header SIZE of the + input database in bytes. + </para> + </listitem> + </varlistentry> + + <varlistentry><term>-O <parameter>SIZE</parameter></term> + <listitem> + <para> + Explicitly specify the CTDB record header SIZE for the + output database in bytes. + </para> + </listitem> + </varlistentry> + + <varlistentry><term>-h</term> + <listitem> + <para> + Print help text. + </para> + </listitem> + </varlistentry> + </variablelist> + </refsect1> + + <refsect1> + <title>COMMANDS</title> + + <variablelist> + <varlistentry><term>help</term> + <listitem> + <para> + Print help text. + </para> + </listitem> + </varlistentry> + + <varlistentry><term>dump <parameter>IDB</parameter></term> + <listitem> + <para> + Dump the contents of an LTDB input file IDB to standard + output in a human-readable format. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term> + convert <parameter>IDB</parameter> <parameter>ODB</parameter> + </term> + <listitem> + <para> + Copy an LTDB input file IDB to output file ODB, optionally + adding or removing CTDB headers. + </para> + </listitem> + </varlistentry> + </variablelist> + </refsect1> + + <refsect1> + <title>EXAMPLES</title> + + <para> + Print a local tdb in "tdbdump" style: + </para> + <screen format="linespecific"> + ltdbtool dump idmap2.tdb.0 + </screen> + + <para> + Print a local tdb with header information similar to "ctdb catdb": + </para> + <screen format="linespecific"> + ltdbtool dump -p idmap2.tdb.0 + </screen> + + <para> + Strip the CTDB headers from records: + </para> + <screen format="linespecific"> + ltdbtool convert -o0 idmap2.tdb.0 idmap.tdb + </screen> + + <para> + Strip 64 bit CTDB headers from records, running on i386: + </para> + <screen format="linespecific"> + ltdbtool convert -s64 -o0 idmap2.tdb.0 idmap.tdb + </screen> + + <para> + Strip the CTDB headers from records by piping through tdbrestore: + </para> + <screen format="linespecific"> + ltdbtool dump idmap2.tdb.0 | tdbrestore idmap.tdb + </screen> + + <para> + Convert a local tdb from a 64 bit system for usage on a 32 bit system: + </para> + <screen format="linespecific"> + ltdbtool convert -s64 -o32 idmap2.tdb.0 idmap2.tdb.1 + </screen> + + <para> + Add a default header: + </para> + <screen format="linespecific"> + ltdbtool convert -s0 idmap.tdb idmap2.tdb.0 + </screen> + </refsect1> + + <refsect1><title>SEE ALSO</title> + <para> + <citerefentry><refentrytitle>ctdb</refentrytitle> + <manvolnum>1</manvolnum></citerefentry>, + + <citerefentry><refentrytitle>tdbdump</refentrytitle> + <manvolnum>1</manvolnum></citerefentry>, + + <citerefentry><refentrytitle>tdbrestore</refentrytitle> + <manvolnum>1</manvolnum></citerefentry>, + + <citerefentry><refentrytitle>ctdb</refentrytitle> + <manvolnum>7</manvolnum></citerefentry>, + + <ulink url="http://ctdb.samba.org/"/> + </para> + </refsect1> + + <refentryinfo> + <author> + <contrib> + This documentation was written by Gregor Beck + </contrib> + </author> + + <copyright> + <year>2011</year> + <holder>Gregor Beck</holder> + <holder>Michael Adam</holder> + </copyright> + <legalnotice> + <para> + This program is free software; you can redistribute it and/or + modify it under the terms of the GNU General Public License as + published by the Free Software Foundation; either version 3 of + the License, or (at your option) any later version. + </para> + <para> + This program is distributed in the hope that it will be + useful, but WITHOUT ANY WARRANTY; without even the implied + warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR + PURPOSE. See the GNU General Public License for more details. + </para> + <para> + You should have received a copy of the GNU General Public + License along with this program; if not, see + <ulink url="http://www.gnu.org/licenses"/>. + </para> + </legalnotice> + </refentryinfo> + +</refentry> diff --git a/ctdb/doc/onnode.1.xml b/ctdb/doc/onnode.1.xml new file mode 100644 index 0000000..f7da04a --- /dev/null +++ b/ctdb/doc/onnode.1.xml @@ -0,0 +1,315 @@ +<?xml version="1.0" encoding="iso-8859-1"?> +<!DOCTYPE refentry + PUBLIC "-//OASIS//DTD DocBook XML V4.5//EN" + "http://www.oasis-open.org/docbook/xml/4.5/docbookx.dtd"> +<refentry id="onnode.1"> + + <refmeta> + <refentrytitle>onnode</refentrytitle> + <manvolnum>1</manvolnum> + <refmiscinfo class="source">ctdb</refmiscinfo> + <refmiscinfo class="manual">CTDB - clustered TDB database</refmiscinfo> + </refmeta> + + <refnamediv> + <refname>onnode</refname> + <refpurpose>run commands on CTDB cluster nodes</refpurpose> + </refnamediv> + + <refsynopsisdiv> + <cmdsynopsis> + <command>onnode</command> + <arg rep="repeat"><replaceable>OPTION</replaceable></arg> + <arg choice="req"><replaceable>NODES</replaceable></arg> + <arg choice="req"><replaceable>COMMAND</replaceable></arg> + </cmdsynopsis> + </refsynopsisdiv> + + <refsect1> + <title>DESCRIPTION</title> + <para> + onnode is a utility to run commands on a specific node of a CTDB + cluster, or on all nodes. + </para> + <para> + <replaceable>NODES</replaceable> specifies which node(s) to run + a command on. See section <citetitle>NODES + SPECIFICATION</citetitle> for details. + </para> + <para> + <replaceable>COMMAND</replaceable> can be any shell command. The + onnode utility uses ssh or rsh to connect to the remote nodes + and run the command. + </para> + </refsect1> + + <refsect1> + <title>OPTIONS</title> + + <variablelist> + <varlistentry><term>-c</term> + <listitem> + <para> + Execute COMMAND in the current working directory on the + specified nodes. + </para> + </listitem> + </varlistentry> + + <varlistentry><term>-f <parameter>FILENAME</parameter></term> + <listitem> + <para> + Specify an alternative nodes FILENAME to use instead of + the default. See the discussion of + <filename>/usr/local/etc/ctdb/nodes</filename> in the + FILES section for more details. + </para> + </listitem> + </varlistentry> + + <varlistentry><term>-i</term> + <listitem> + <para> + Keep standard input open, allowing data to be piped to + onnode. Normally onnode closes stdin to avoid surprises + when scripting. Note that this option is ignored when + using <option>-p</option> or if <envar>ONNODE_SSH</envar> + is set to anything other than "ssh". + </para> + </listitem> + </varlistentry> + + <varlistentry><term>-n</term> + <listitem> + <para> + Allow nodes to be specified by name rather than node + numbers. These nodes don't need to be listed in the nodes + file. You can avoid the nodes file entirely by combining + this with <code>-f /dev/null</code>. + </para> + </listitem> + </varlistentry> + + <varlistentry><term>-p</term> + <listitem> + <para> + Run COMMAND in parallel on the specified nodes. The + default is to run COMMAND sequentially on each node. + </para> + </listitem> + </varlistentry> + + <varlistentry><term>-P</term> + <listitem> + <para> + Push files to nodes. Names of files to push are specified + rather than the usual command. Quoting is fragile/broken + - filenames with whitespace in them are not supported. + </para> + </listitem> + </varlistentry> + + <varlistentry><term>-q</term> + <listitem> + <para> + Do not print node addresses. Normally, onnode prints + informational node addresses if more than one node is + specified. This overrides -v. + </para> + </listitem> + </varlistentry> + + <varlistentry><term>-v</term> + <listitem> + <para> + Print node addresses even if only one node is specified. + Normally, onnode prints informational node addresses when + more than one node is specified. + </para> + </listitem> + </varlistentry> + + <varlistentry><term>-h, --help</term> + <listitem> + <para> + Show a short usage guide. + </para> + </listitem> + </varlistentry> + </variablelist> + </refsect1> + + <refsect1> + <title>NODES SPECIFICATION</title> + + <para> + Nodes can be specified via numeric node numbers (from 0 to N-1) + or mnemonics. Multiple nodes are specified using lists of + nodes, separated by commas, and ranges of numeric node numbers, + separated by dashes. If nodes are specified multiple times then + the command will be executed multiple times on those nodes. The + order of nodes is significant. + </para> + + <para> + The following mnemonics are available: + </para> + + <variablelist> + <varlistentry><term>all</term> + <listitem> + <para> + All nodes. + </para> + </listitem> + </varlistentry> + <varlistentry><term>any</term> + <listitem> + <para> + A node where ctdbd is running. This semi-random but + there is a bias towards choosing a low numbered node. + </para> + </listitem> + </varlistentry> + <varlistentry><term>ok | healthy</term> + <listitem> + <para> + All nodes that are not disconnected, banned, disabled or + unhealthy. + </para> + </listitem> + </varlistentry> + <varlistentry><term>con | connected</term> + <listitem> + <para> + All nodes that are not disconnected. + </para> + </listitem> + </varlistentry> + </variablelist> + </refsect1> + + <refsect1> + <title>EXAMPLES</title> + + <para> + The following command would show the process ID of ctdbd on all nodes + </para> + <screen format="linespecific"> + onnode all ctdb getpid + </screen> + + <para> + The following command would show the last 5 lines of log on each + node, preceded by the node's hostname + </para> + <screen format="linespecific"> + onnode all "hostname; tail -5 /usr/local/var/log/log.ctdb" + </screen> + + <para> + The following command would restart the ctdb service on all + nodes, in parallel. + </para> + <screen format="linespecific"> + onnode -p all service ctdb restart + </screen> + + <para> + The following command would run ./foo in the current working + directory, in parallel, on nodes 0, 2, 3 and 4. + </para> + <screen format="linespecific"> + onnode -c -p 0,2-4 ./foo + </screen> + </refsect1> + + <refsect1> + <title>FILES</title> + + <variablelist> + <varlistentry><term><filename>/usr/local/etc/ctdb/nodes</filename></term> + <listitem> + <para> + Default file containing a list of each node's IP address + or hostname. + </para> + <para> + As above, a file specified via the <option>-f</option> + is given precedence. If a + relative path is specified and no corresponding file + exists relative to the current directory then the file is + also searched for in the CTDB configuration directory. + </para> + <para> + Otherwise the default is + <filename>/usr/local/etc/ctdb/nodes</filename>. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term><filename>/usr/local/etc/ctdb/onnode.conf</filename></term> + <listitem> + <para> + If this file exists it is sourced by onnode. The main + purpose is to allow the administrator to set + <envar>ONNODE_SSH</envar> to something other than "ssh". + In this case the -t option is ignored. + </para> + </listitem> + </varlistentry> + </variablelist> + </refsect1> + + <refsect1> + <title>SEE ALSO</title> + + <para> + <citerefentry><refentrytitle>ctdb</refentrytitle> + <manvolnum>7</manvolnum></citerefentry>, + + <ulink url="http://ctdb.samba.org/"/> + </para> + </refsect1> + + <refentryinfo> + <author> + <contrib> + This documentation was written by + Andrew Tridgell, + Martin Schwenke + </contrib> + </author> + + <copyright> + <year>2007</year> + <holder>Andrew Tridgell</holder> + <holder>Ronnie Sahlberg</holder> + </copyright> + <copyright> + <year>2008</year> + <holder>Martin Schwenke</holder> + </copyright> + <legalnotice> + <para> + This program is free software; you can redistribute it and/or + modify it under the terms of the GNU General Public License as + published by the Free Software Foundation; either version 3 of + the License, or (at your option) any later version. + </para> + <para> + This program is distributed in the hope that it will be + useful, but WITHOUT ANY WARRANTY; without even the implied + warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR + PURPOSE. See the GNU General Public License for more details. + </para> + <para> + You should have received a copy of the GNU General Public + License along with this program; if not, see + <ulink url="http://www.gnu.org/licenses"/>. + </para> + </legalnotice> + </refentryinfo> + +</refentry> diff --git a/ctdb/doc/ping_pong.1.xml b/ctdb/doc/ping_pong.1.xml new file mode 100644 index 0000000..47e90e8 --- /dev/null +++ b/ctdb/doc/ping_pong.1.xml @@ -0,0 +1,164 @@ +<?xml version="1.0" encoding="iso-8859-1"?> +<!DOCTYPE refentry + PUBLIC "-//OASIS//DTD DocBook XML V4.5//EN" + "http://www.oasis-open.org/docbook/xml/4.5/docbookx.dtd"> +<refentry id="ping_pong.1"> + + <refmeta> + <refentrytitle>ping_pong</refentrytitle> + <manvolnum>1</manvolnum> + <refmiscinfo class="source">ctdb</refmiscinfo> + <refmiscinfo class="manual">CTDB - clustered TDB database</refmiscinfo> + </refmeta> + + <refnamediv> + <refname>ping_pong</refname> + <refpurpose>measures the ping-pong byte range lock latency</refpurpose> + </refnamediv> + + <refsynopsisdiv> + <cmdsynopsis> + <command>ping_pong</command> + <group choice="req"> + <arg choice="plain">-r</arg> + <arg choice="plain">-w</arg> + <arg choice="plain">-rw</arg> + </group> + <arg>-m</arg> + <arg>-c</arg> + <arg choice="req"><replaceable>FILENAME</replaceable></arg> + <arg choice="req"><replaceable>NUM-LOCKS</replaceable></arg> + </cmdsynopsis> + </refsynopsisdiv> + + <refsect1> + <title>DESCRIPTION</title> + <para> + ping_pong measures the byte range lock latency. It is especially + useful on a cluster of nodes sharing a common lock manager as it + will give some indication of the lock manager's performance + under stress. + </para> + + <para> + FILENAME is a file on shared storage to use for byte range + locking tests. + </para> + + <para> + NUM-LOCKS is the number of byte range locks, so needs to be + (strictly) greater than the number of nodes in the cluster. + </para> + </refsect1> + + <refsect1> + <title>OPTIONS</title> + + <variablelist> + <varlistentry> + <term>-r</term> + <listitem> + <para> + test read performance + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term>-w</term> + <listitem> + <para> + test write performance + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term>-m</term> + <listitem> + <para> + use mmap + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term>-c</term> + <listitem> + <para> + validate the locks + </para> + </listitem> + </varlistentry> + + </variablelist> + </refsect1> + + <refsect1> + <title>EXAMPLES</title> + <para> + Testing lock coherence + </para> + <screen format="linespecific"> + ping_pong test.dat N + </screen> + + <para> + Testing lock coherence with lock validation + </para> + <screen format="linespecific"> + ping_pong -c test.dat N + </screen> + + <para> + Testing IO coherence + </para> + <screen format="linespecific"> + ping_pong -rw test.dat N + </screen> + </refsect1> + + <refsect1> + <title>SEE ALSO</title> + <para> + <citerefentry><refentrytitle>ctdb</refentrytitle> + <manvolnum>7</manvolnum></citerefentry>, + + <ulink url="https://wiki.samba.org/index.php/Ping_pong"/> + + </para> + </refsect1> + + <refentryinfo> + <author> + <contrib> + This documentation was written by Mathieu Parent + </contrib> + </author> + + <copyright> + <year>2002</year> + <holder>Andrew Tridgell</holder> + </copyright> + <legalnotice> + <para> + This program is free software; you can redistribute it and/or + modify it under the terms of the GNU General Public License as + published by the Free Software Foundation; either version 3 of + the License, or (at your option) any later version. + </para> + <para> + This program is distributed in the hope that it will be + useful, but WITHOUT ANY WARRANTY; without even the implied + warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR + PURPOSE. See the GNU General Public License for more details. + </para> + <para> + You should have received a copy of the GNU General Public + License along with this program; if not, see + <ulink url="http://www.gnu.org/licenses"/>. + </para> + </legalnotice> + </refentryinfo> + +</refentry> diff --git a/ctdb/doc/readonlyrecords.txt b/ctdb/doc/readonlyrecords.txt new file mode 100644 index 0000000..cc6d7ee --- /dev/null +++ b/ctdb/doc/readonlyrecords.txt @@ -0,0 +1,343 @@ +Read-Only locks in CTDB +======================= + +Problem +======= +CTDB currently only supports exclusive Read-Write locks for clients(samba) accessing the +TDB databases. +This mostly works well but when very many clients are accessing the same file, +at the same time, this causes the exclusive lock as well as the record itself to +rapidly bounce between nodes and acts as a scalability limitation. + +This primarily affects locking.tdb and brlock.tdb, two databases where record access is +read-mostly and where writes are semi-rare. + +For the common case, if CTDB provided shared non-exclusive Read-Only lock semantics +this would greatly improve scaling for these workloads. + + +Desired properties +================== +We can not make backward incompatible changes the ctdb_ltdb header for the records. + +A Read-Only lock enabled ctdb demon must be able to interoperate with a non-Read-Only +lock enbled daemon. + +Getting a Read-Only lock should not be slower than getting a Read-Write lock. + +When revoking Read-Only locks for a record, this should involve only those nodes that +currently hold a Read-Only lock and should avoid broadcasting opportunistic revocations. +(must track which nodes are delegated to) + +When a Read-Write lock is requested, if there are Read-Only locks delegated to other +nodes, the DMASTER will defer the record migration until all read-only locks are first +revoked (synchronous revoke). + +Due to the cost of revoking Read-Only locks has on getting a Read-Write lock, the +implementation should try to avoid creating Read-Only locks unless it has indication +that there is contention. This may mean that even if client requests a Read-Only lock +we might still provide a full Read-Write lock in order to avoid the cost of revoking +the locks in some cases. + +Read-Only locks require additional state to be stored in a separate database, containing +information about which nodes have have been delegated Read-Only locks. +This database should be kept at minimal size. + +Read-Only locks should not significantly complicate the normal record +create/migration/deletion cycle for normal records. + +Read-Only locks should not complicate the recovery process. + +Read-Only locks should not complicate the vacuuming process. + +We should avoid forking new child processes as far as possible from the main daemon. + +Client-side implementation, samba, libctdb, others, should have minimal impact when +Read-Only locks are implemented. +Client-side implementation must be possible with only minor conditionals added to the +existing lock-check-fetch-unlock loop that clients use today for Read-Write locks. So +that clients only need one single loop that can handle both Read-Write locking as well +as Read-Only locking. Clients should not need two nearly identical loops. + + +Implementation +============== + +Four new flags are allocated in the ctdb_ltdb record header. +HAVE_DELEGATIONS, HAVE_READONLY_LOCK, REVOKING_READONLY and REVOKE_COMPLETE + +HAVE_DELEGATIONS is a flag that can only be set on the node that is currently the +DMASTER for the record. When set, this flag indicates that there are Read-Only locks +delegated to other nodes in the cluster for this record. + +HAVE_READONLY is a flag that is only set on nodes that are NOT the DMASTER for the +record. If set this flag indicates that this record contains an up-to-date Read-Only +version of this record. A client that only needs to read, but not to write, the record +can safely use the content of this record as is regardless of the value of the DMASTER +field of the record. + +REVOKING_READONLY is a flag that is used while a set of read only delegations are being +revoked. +This flag is only set when HAVE_DELEGATIONS is also set, and is cleared at the same time +as HAVE_DELEGATIONS is cleared. +Normal operations is that first the HAVE_DELEGATIONS flag is set when the first +delegation is generated. When the delegations are about to be revoked, the +REVOKING_READONLY flag is set too. +Once all delegations are revoked, both flags are cleared at the same time. +While REVOKING_READONLY is set, any requests for the record, either normal request or +request for readonly will be deferred. +Deferred requests are linked on a list for deferred requests until the time that the +revokation is completed. +This flags is set by the main ctdb daemon when it starts revoking this record. + +REVOKE_COMPLETE +The actual revoke of records is done by a child process, spawned from the main ctdb +daemon when it starts the process to revoke the records. +Once the child process has finished revoking all delegations it will set the flag +REVOKE_COMPLETE for this record to signal to the main daemon that the record has been +successfully revoked. +At this stage the child process will also trigger an event in the main daemon that +revoke is complete and that the main daemon should start re-processing all deferred +requests. + + + +Once the revoke process is completed there will be at least one deferred request to +access this record. That is the initical call to for an exclusive fetch_lock() that +triggered the revoke process to be started. +In addition to this deferred request there may also be additional requests that have +also become deferred while the revoke was in process. These can be either exclusive +fetch_locks() or they can be readonly lock requests. +Once the revoke is completed the main daemon will reprocess all exclusive fetch_lock() +requests immediately and respond to these clients. +Any requests for readadonly lock requests will be deferred for an additional period of +time before they are re-processed. +This is to allow the client that needs a fetch_lock() to update the record to get some +time to access and work on the record without having to compete with the possibly +very many readonly requests. + + + + + +The ctdb_db structure is expanded so that it contains one extra TDB database for each +normal, non-persistent datbase. +This new database is used for tracking delegations for the records. +A record in the normal database that has "HAVE_DELEGATION" set will always have a +corresponding record at the same key. This record contains the set of all nodes that +the record is delegated to. +This tracking database is lockless, using TDB_NOLOCK, and is only ever accessed by +the main ctdbd daemon. +The lockless nature and the fact that no other process ever access this TDB means we +are guaranteed non-blocking access to records in the tracking database. + +The ctdb_call PDU is allocated with a new flag WANT_READONLY and possibly also a new +callid: CTDB_FETCH_WITH_HEADER_FUNC. +This new function returns not only the record, as CTDB_FETCH_FUNC does, but also +returns the full ctdb_ltdb record HEADER prepended to the record. +This function is optional, clients that do not care what the header is can continue +using just CTDB_FETCH_FUNC + + +This flag is used to requesting a read-only record from the DMASTER/LMASTER. +If the record does not yet exist, this is a returned as an error to the client and the +client will retry the request loop. + +A new control is added to make remote nodes remove the HAVE_READONLY_LOCK from a record +and to invalidate any deferred readonly copies from the databases. + + + +Client implementation +===================== +Clients today use a loop for record fetch lock that looks like this + try_again: + lock record in tdb + + if record does not exist in tdb, + unlock record + ask ctdb to migrate record onto the node + goto try_again + + if record dmaster != this node pnn + unlock record + ask ctdb to migrate record onto the node + goto try_again + + finished: + +where we basically spin, until the record is migrated onto the node and we have managed +to pin it down. + +This will change to instead to something like + + try_again: + lock record in tdb + + if record does not exist in tdb, + unlock record + ask ctdb to migrate record onto the node + goto try_again + + if record dmaster == current node pnn + goto finished + + if read-only lock + if HAVE_READONLY_LOCK or HAVE_DELEGATIONS is set + goto finished + else + unlock record + ask ctdb for read-only copy (WANT_READONLY[|WITH_HEADER]) + if failed to get read-only copy (*A) + ask ctdb to migrate the record onto the node + goto try_again + lock record in tdb + goto finished + + unlock record + ask ctdb to migrate record onto the node + goto try_again + + finished: + +If the record does not yet exist in the local TDB, we always perform a full fetch for a +Read-Write lock even if only a Read-Only lock was requested. +This means that for first access we always grab a Read-Write lock and thus upgrade any +requests for Read-Only locks into a Read-Write request. +This creates the record, migrates it onto the node and makes the local node become +the DMASTER for the record. + +Future reference to this same record by the local samba daemons will still access/lock +the record locally without triggereing a Read-Only delegation to be created since the +record is already hosted on the local node as DMASTER. + +Only if the record is contended, i.e. it has been created an migrated onto the node but +we are no longer the DMASTER for this record, only for this case will we create a +Read-Only delegation. +This heuristics provide a mechanism where we will not create Read-Only delegations until +we have some indication that the record may be contended. + +This avoids creating and revoking Read-Only delegations when only a single client is +repeatedly accessing the same set of records. +This also aims to limit the size of the tracking tdb. + + +Server implementation +===================== +When receiving a ctdb_call with the WANT_READONLY flag: + +If this is the LMASTER for the record and the record does not yet exist, LMASTER will +return an error back to the client (*A above) and the client will try to recover. +In particular, LMASTER will not create a new record for this case. + +If this is the LMASTER for the record and the record exists, the PDU will be forwarded to +the DMASTER for the record. + +If this node is not the DMASTER for this record, we forward the PDU back to the +LMASTER. Just as we always do today. + +If this is the DMASTER for the record, we need to create a Read-Only delegation. +This is done by + lock record + increase the RSN by one for this record + set the HAVE_DELEGATIONS flag for the record + write the updated record to the TDB + create/update the tracking TDB nd add this new node to the set of delegations + send a modified copy of the record back to the requesting client. + modifications are that RSN is decremented by one, so delegated records are "older" than on the DMASTER, + it has HAVE_DELEGATIONS flag stripped off, and has HAVE_READONLY_LOCK added. + unlock record + +Important to note is that this does not trigger a record migration. + + +When receiving a ctdb_call without the WANT_READONLY flag: + +If this is the DMASTER for the this might trigger a migration. If there exists +delegations we must first revoke these before allowing the Read-Write request from +proceeding. So, +IF the record has HAVE_DELEGATIONS set, we create a child process and defer processing +of this PDU until the child process has completed. + +From the child process we will call out to all nodes that have delegations for this +record and tell them to invalidate this record by clearing the HAVE_READONLY_LOCK from +the record. +Once all delegated nodes respond back, the child process signals back to the main daemon +the revoke has completed. (child process may not access the tracking tdb since it is +lockless) + +Main process is triggered to re-process the PDU once the child process has finished. +Main daemon deletes the corresponding record in the tracking database, clears the +HAVE_DELEGATIONS flag for the record and then proceeds to perform the migration as usual. + +When receiving a ctdb_call without the flag we want all delegations to be revoked, +so we must take care that the delegations are revoked unconditionally before we even +check if we are already the DMASTER (in which case the ctdb_call would normally just +be no-op (*B below)) + + + +Recovery process changes +======================== +A recovery implicitly clears/revokes any read only records and delegations from all +databases. + +During delegations of Read-Only locks, this is done in such way that delegated records +will have a RSN smaller than the DMASTER. This guarantees that read-only copies always +have a RSN that is smaller than the DMASTER. + +During recoveries we do not need to take any special action other than always picking +the copy of the record that has the highest RSN, which is what we already do today. + +During the recovery process, we strip all flags off all records while writing the new +content of the database during the PUSH_DB control. + +During processing of the PUSH_DB control and once the new database has been written we +then also wipe the tracking database. + +This makes changes to the recovery process minimal and nonintrusive. + + + +Vacuuming process +================= +Vacuuming needs only minimal changes. + + +When vacuuming runs, it will do a fetch_lock to migrate any remote records back onto the +LMASTER before the record can be purged. This will automatically force all delegations +for that record to be revoked before the migration is copied back onto the LMASTER. +This handles the case where LMASTER is not the DMASTER for the record that will be +purged. +The migration in this case does force any delegations to be revoked before the +vacuuming takes place. + +Missing is the case when delegations exist and the LMASTER is also the DMASTER. +For this case we need to change the vacuuming to unconditionally always try to do a +fetch_lock when HAVE_DELEGATIONS is set, even if the record is already stored locally. +(*B) +This fetch lock will not cause any migrations by the ctdb daemon, but since it does +not have the WANT_READONLY this will still force the delegations to be revoked but no +migration will trigger. + + +Traversal process +================= +Traversal process is changed to ignore any records with the HAVE_READONLY_LOCK + + +Forward/Backward Compatibility +============================== +Non-readonly locking daemons must be able to interoperate with readonly locking enabled daemons. + +Non-readonly enabled daemons fetching records from Readonly enabled daemons: +Non-readonly enabled daemons do not know, and never set the WANT_READONLY flag so these daemons will always request a full migration for a full fetch-lock for all records. Thus a request from a non-readonly enabled daemon will always cause any existing delegations to be immediately revoked. Access will work but performance may be harmed since there will be a lot of revoking of delegations. + +Readonly enabled daemons fetching records with WANT_READONLY from non-readonly enabled daemons: +Non-readonly enabled daemons ingore the WANT_READONLY flag and never return delegations. They always return a full record migration. +Full record migration is allowed by the protocol, even if the originator only requests the 'hint' WANT_READONLY, +so this access also interoperates between daemons with different capabilities. + + + + |