diff options
Diffstat (limited to '')
| -rw-r--r-- | heartbeat/README.galera | 148 |
1 files changed, 148 insertions, 0 deletions
diff --git a/heartbeat/README.galera b/heartbeat/README.galera new file mode 100644 index 0000000..dd45618 --- /dev/null +++ b/heartbeat/README.galera @@ -0,0 +1,148 @@ +Notes regarding the Galera resource agent +--- + +In the resource agent, the action of bootstrapping a Galera cluster is +implemented into a series of small steps, by using: + + * Two CIB attributes `last-committed` and `bootstrap` to elect a + bootstrap node that will restart the cluster. + + * One CIB attribute `sync-needed` that will identify that joining + nodes are in the process of synchronizing their local database + via SST. + + * A Master/Slave pacemaker resource which helps splitting the boot + into steps, up to a point where a galera node is available. + + * the recurring monitor action to coordinate switch from one + state to another. + +How boot works +==== + +There are two things to know to understand how the resource agent +restart a Galera cluster. + +### Bootstrap the cluster with the right node + +When synced, the nodes of a galera cluster have in common a last seqno, +which identifies the last transaction considered successful by a +majority of nodes in the cluster (think quorum). + +To restart a cluster, the resource agent must ensure that it will +bootstrap the cluster from an node which is up-to-date, i.e which has +the highest seqno of all nodes. + +As a result, if the resource agent cannot retrieve the seqno on all +nodes, it won't be able to safely identify a bootstrap node, and +will simply refuse to start the galera cluster. + +### synchronizing nodes can be a long operation + +Starting a bootstrap node is relatively fast, so it's performed +during the "promote" operation, which is a one-off, time-bounded +operation. + +Subsequent nodes will need to synchronize via SST, which consists +in "pushing" an entire Galera DB from one node to another. + +There is no perfect time-out, as time spent during synchronization +depends on the size of the DB. Thus, joiner nodes are started during +the "monitor" operation, which is a recurring operation that can +better track the progress of the SST. + + +State flow +==== + +General idea for starting Galera: + + * Before starting the Galera cluster each node needs to go in Slave + state so that the agent records its last seqno into the CIB. + __ This uses attribute last-committed __ + + * When all node went in Slave, the agent can safely determine the + last seqno and elect a bootstrap node (`detect_first_master()`). + __ This uses attribute bootstrap __ + + * The agent then sets the score of the elected bootstrap node to + Master so that pacemaker promote it and start the first Galera + server. + + * Once the first Master is running, the agent can start joiner + nodes during the "monitor" operation, and starts monitoring + their SST sync. + __ This uses attribute sync-needed __ + + * Only when SST is over on joiner nodes, the agent promotes them + to Master. At this point, the entire Galera cluster is up. + + +Attribute usage and liveness +==== + +Here is how attributes are created on a per-node basis. If you +modify the resource agent make sure those properties still hold. + +### last-committed + +It is just a temporary hint for the resource agent to help +elect a bootstrap node. Once the bootstrap attribute is set on one +of the nodes, we can get rid of last-committed. + + - Used : during Slave state to compare seqno + - Created: before entering Slave state: + . at startup in `galera_start()` + . or when a Galera node is stopped in `galera_demote()` + - Deleted: just before node starts in `galera_start_local_node()`; + cleaned-up during `galera_demote()` and `galera_stop()` + +We delete last-committed before starting Galera, to avoid race +conditions that could arise due to discrepancies between the CIB and +Galera. + +### bootstrap + +Attribute set on the node that is elected to bootstrap Galera. + +- Used : during promotion in `galera_start_local_node()` +- Created: at startup once all nodes have `last-committed`; + or during monitor if all nodes have failed +- Deleted: in `galera_start_local_node()`, just after the bootstrap + node started and is ready; + cleaned-up during `galera_demote()` and `galera_stop()` + +There cannot be more than one bootstrap node at any time, otherwise +the Galera cluster would stop replicating properly. + +### sync-needed + +While this attribute is set on a node, the Galera node is in JOIN +state, i.e. SST is in progress and the node cannot serve queries. + +The resource agent relies on the underlying SST method to monitor +the progress of the SST. For instance, with `wsrep_sst_rsync`, +timeout would be reported by rsync, the Galera node would go in +Non-primary state, which would make `galera_monitor()` fail. + +- Used : during recurring slave monitor in `check_sync_status()` +- Created: in `galera_start_local_node()`, just after the joiner + node started and entered the Galera cluster +- Deleted: during recurring slave monitor in `check_sync_status()` + as soon as the Galera code reports to be SYNC-ed. + +### no-grastate + +If a galera node was unexpectedly killed in a middle of a replication, +InnoDB can retain the equivalent of a XA transaction in prepared state +in its redo log. If so, mysqld cannot recover state (nor last seqno) +automatically, and special recovery heuristic has to be used to +unblock the node. + +This transient attribute is used to keep track of forced recoveries to +prevent bootstrapping a cluster from a recovered node when possible. + +- Used : during `detect_first_master()` to elect the bootstrap node +- Created: in `detect_last_commit()` if the node has a pending XA + transaction to recover in the redo log +- Deleted: when a node is promoted to Master. |