diff options
Diffstat (limited to 'src/cephadm/containers/keepalived/README.md')
| -rw-r--r-- | src/cephadm/containers/keepalived/README.md | 233 |
1 files changed, 233 insertions, 0 deletions
diff --git a/src/cephadm/containers/keepalived/README.md b/src/cephadm/containers/keepalived/README.md new file mode 100644 index 000000000..bd7b605ac --- /dev/null +++ b/src/cephadm/containers/keepalived/README.md @@ -0,0 +1,233 @@ +# quay.io/ceph/keepalived + +A small [ubi8-minimal](https://catalog.redhat.com/software/containers/registry/registry.access.redhat.com/repository/ubi8/ubi-minimal) based Docker container that provides a method of IP high availability via [keepalived](http://www.keepalived.org/) (VRRP failover), and optional Kubernetes API Server monitoring. If allowed to auto configure (default behaviour) it will automatically generate a unicast based failover configuration with a minimal amount of user supplied information. + +For specific information on Keepalived, please see the man page on [keepalived.conf](http://linux.die.net/man/5/keepalived.conf) or the [Keepalived User Guide](http://www.keepalived.org/pdf/UserGuide.pdf). + + +## Index +- [quay.io/ceph/keepalived](#cephkeepalived) + - [Index](#index) + - [Prerequisites](#prerequisites) + - [Configuration](#configuration) + - [Execution Control](#execution-control) + - [Autoconfiguration Options](#autoconfiguration-options) + - [Kubernetes Options](#kubernetes-options) + - [Suggested Kubernetes Settings](#suggested-kubernetes-settings) + - [Example Keepalived Configs](#example-keepalived-configs) + - [Example Autogenerated Keepalived Master Config](#example-autogenerated-keepalived-master-config) + - [Example Autogenerated Keepalived Backup Config](#example-autogenerated-keepalived-backup-config) + - [Example Run Commands](#example-run-commands) + - [Example Master Run Command](#example-master-run-command) + - [Example Backup Run Command](#example-backup-run-command) + + +## Prerequisites + +Before attempting to deploy the keepalived container, the host must allow non local binding of ipv4 addresses. To do this, configure the sysctl tunable `net.ipv4.ip_nonlocal_bind=1`. + +In addition to enabling the nonlocal binds, the container must be run with both host networking (`--net=host`) and security setting CAP_NET_ADMIN (`--cap-add NET_ADMIN`) capability. These allow the container to manage the host's networking configuration, and this is essential to the function of keepalived. + + +## Configuration +### Execution Control + +| Variable | Default | +|:---------------------:|:------------------------------------------------:| +| `KEEPALIVED_AUTOCONF` | `true` | +| `KEEPALIVED_CONF` | `/etc/keepalived/keepalived.conf` | +| `KEEPALIVED_CMD` | `/usr/sbin/keepalived -n -l -f $KEEPALIVED_CONF` | +| `KEEPALIVED_DEBUG` | `false` | + +* `KEEPALIVED_AUTOCONF` - Enables or disables the auto-configuration of keepalived. + +* `KEEPALIVED_CONF` - The path to the keepalived configuration file. + +* `KEEPALIVED_CMD` - The command called to execute keepalived. + +* `KEEPALIVED_DEBUG` - Enables or disables debug level logging for keepalived (adds `-D` to `KEEPALIVED_CMD`. + + +### Autoconfiguration Options + +| Variable | Default | +|:-------------------------------------------:|:----------------------------------:| +| `KEEPALIVED_ADVERT_INT` | `1` | +| `KEEPALIVED_AUTH_PASS` | `pwd$KEEPALIVED_VIRTUAL_ROUTER_ID` | +| `KEEPALIVED_INTERFACE` | `eth0` | +| `KEEPALIVED_PRIORITY` | `200` | +| `KEEPALIVED_STATE` | `MASTER` | +| `KEEPALIVED_TRACK_INTERFACE_###` | | +| `KEEPALIVED_UNICAST_SRC_IP` | | +| `KEEPALIVED_UNICAST_PEER_###` | | +| `KEEPALIVED_VIRTUAL_IPADDRESS_###` | | +| `KEEPALIVED_VIRTUAL_IPADDRESS_EXCLUDED_###` | | +| `KEEPALIVED_VIRTUAL_ROUTER_ID` | `1` | +| `KEEPALIVED_KUBE_APISERVER_CHECK` | `false` | + +* `KEEPALIVED_ADVERT_INT` - The VRRP advertisement interval (in seconds). + +* `KEEPALIVED_AUTH_PASS` - A shared password used to authenticate each node in a VRRP group (**Note:** If password is longer than 8 characters, only the first 8 characters are used). + +* `KEEPALIVED_INTERFACE` - The host interface that keepalived will monitor and use for VRRP traffic. + +* `KEEPALIVED_PRIORITY` - Election value, the server configured with the highest priority will become the Master. + +* `KEEPALIVED_STATE` - Defines the server role as Master or Backup. (**Options:** `MASTER` or `BACKUP`). + +* `KEEPALIVED_TRACK_INTERFACE_###` - An interface that's state should be monitored (e.g. eth0). More than one can be supplied as long as the variable name ends in a number from 0-999. + +* `KEEPALIVED_UNICAST_SRC_IP` - The IP on the host that the keepalived daemon should bind to. **Note:** If not specified, it will be the first IP bound to the interface specified in `KEEPALIVED_INTERFACE`. + +* `KEEPALIVED_UNICAST_PEER_###` - An IP of a peer participating in the VRRP group. More tha one can be supplied as long as the variable name ends in a number from 0-999. + +* `KEEPALIVED_VIRTUAL_IPADDRESS_###` - An instance of an address that will be monitored and failed over from one host to another. These should be a quoted string in the form of: `<IPADDRESS>/<MASK> brd <BROADCAST_IP> dev <DEVICE> scope <SCOPE> label <LABEL>` At a minimum the ip address, mask and device should be specified e.g. `KEEPALIVED_VIRTUAL_IPADDRESS_1="10.10.0.2/24 dev eth0"`. More than one can be supplied as long as the variable name ends in a number from 0-999. **Note:** Keepalived has a hard limit of **20** addresses that can be monitored. More can be failed over with the monitored addresses via `KEEPALIVED_VIRTUAL_IPADDRESS_EXCLUDED_###`. + + +* `KEEPALIVED_VIRTUAL_IPADDRESS_EXCLUDED_###` - An instance of an address that will be failed over with the monitored addresses supplied via `KEEPALIVED_VIRTUAL_IPADDRESS_###`. These should be a quoted string in the form of: `<IPADDRESS>/<MASK> brd <BROADCAST_IP> dev <DEVICE> scope <SCOPE> label <LABEL>` At a minimum the ip address, mask and device should be specified e.g. `KEEPALIVED_VIRTUAL_IPADDRESS_EXCLUDED_1="172.16.1.20/24 dev eth1"`. More than one can be supplied as long as the variable name ends in a number from 0-999. + +* `KEEPALIVED_VIRTUAL_ROUTER_ID` - A unique number from 0 to 255 that should identify the VRRP group. Master and Backup should have the same value. Multiple instances of keepalived can be run on the same host, but each pair **MUST** have a unique virtual router id. + +* `KEEPALIVED_KUBE_APISERVER_CHECK` - If enabled it configures a simple check script for the Kubernetes API-Server. For more information on this feature, please see the [Kubernetes Options](#kubernetes-options) section. + + +### Kubernetes Options + + +| **Variable** | **Default** | +|:-----------------------------:|:----------------------------------------------:| +| `KUBE_APISERVER_ADDRESS` | parsed from `KEEPALIVED_VIRTUAL_IPADDRESS_###` | +| `KUBE_APISERVER_PORT` | `6443` | +| `KUBE_APISERVER_CHK_INTERVAL` | `3` | +| `KUBE_APISERVER_CHK_FALL` | `10` | +| `KUBE_APISERVER_CHK_RISE` | `2` | +| `KUBE_APISERVER_CHK_WEIGHT` | `-50` | + + + +* `KUBE_APISERVER_ADDRESS` - The Virtual IP being used for the Kube API Server. If none is supplied, it is assumed to be the lowest numbered entry in the `KEEPALIVED_VIRTUAL_IPADDRESS_###` variables. + +* `KUBE_APISERVER_PORT` - The port to use in conjunction with the `KUBE_APISERVER_ADDRESS`. + +* `KUBE_APISERVER_CHK_INTERVAL` - The interval in seconds between calling the script. + +* `KUBE_APISERVER_CHK_FALL` - The number of consecutive non-zero script exits before setting the state to `FAULT`. + +* `KUBE_APISERVER_CHK_RISE` - The number of consecutive zero script exits before exiting the `FAULT` state. + +* `KUBE_APISERVER_CHK_WEIGHT` - The weight to apply to the priority when the service enters the `FAULT` state. + + + +--- + +### Suggested Kubernetes Settings + +Assuming there are three nodes running the kube-apiserver, you cannot rely on setting just the`KEEPALIVED_STATE` parameter to manage failover across the nodes. + +To manage kube-apiserver failover, enable the healthcheck option with `KEEPALIVED_KUBE_APISERVER_CHECK`, and set the `KEEPALIVED_PRIORITY` manually for the three instances. + +| **Node** | **Priority** | +|:--------:|:------------:| +| node-01 | 200 | +| node-02 | 190 | +| node-03 | 180 | + +With the default weight of `-50`, if `node-01` has an issue, it's priority will drop to `150` and allow `node-02` to take over, the same is repeated if `node-02` has a failure dropping it's weight to `140` and `node-03` takes over. + +Recovery occurs in the same order with the system with the highest priority being promoted to master. + +### Example Keepalived Configs + +##### Example Autogenerated Keepalived Master Config +``` +vrrp_instance MAIN { + state MASTER + interface eth0 + virtual_router_id 2 + priority 200 + advert_int 1 + unicast_src_ip 10.10.0.21 + unicast_peer { + 10.10.0.22 + } + authentication { + auth_type PASS + auth_pass pwd1 + } + virtual_ipaddress { + 10.10.0.2/24 dev eth0 + } + virtual_ipaddress_excluded { + 172.16.1.20/24 dev eth1 + } + track_interface { + eth0 + eth1 + } +} +``` + +##### Example Autogenerated Keepalived Backup Config +``` +vrrp_instance MAIN { + state BACKUP + interface eth0 + virtual_router_id 2 + priority 100 + advert_int 1 + unicast_src_ip 10.10.0.22 + unicast_peer { + 10.10.0.21 + } + authentication { + auth_type PASS + auth_pass pwd1 + } + virtual_ipaddress { + 10.10.0.2/24 dev eth0 + } + virtual_ipaddress_excluded { + 172.16.1.20/24 dev eth1 + } + track_interface { + eth0 + eth1 + } +} + +``` + + +## Example Run Commands +##### Example Master Run Command +```bash +docker run -d --net=host --cap-add NET_ADMIN \ +-e KEEPALIVED_AUTOCONF=true \ +-e KEEPALIVED_STATE=MASTER \ +-e KEEPALIVED_INTERFACE=eth0 \ +-e KEEPALIVED_VIRTUAL_ROUTER_ID=2 \ +-e KEEPALIVED_UNICAST_SRC_IP=10.10.0.21 \ +-e KEEPALIVED_UNICAST_PEER_0=10.10.0.22 \ +-e KEEPALIVED_TRACK_INTERFACE_1=eth0 \ +-e KEEPALIVED_TRACK_INTERFACE_2=eth1 \ +-e KEEPALIVED_VIRTUAL_IPADDRESS_1="10.10.0.3/24 dev eth0" \ +-e KEEPALIVED_VIRTUAL_IPADDRESS_EXCLUDED_1="172.16.1.20/24 dev eth1" \ +quay.io/ceph/keepalived +``` + +##### Example Backup Run Command +```bash +docker run -d --net=host --cap-add NET_ADMIN \ +-e KEEPALIVED_AUTOCONF=true \ +-e KEEPALIVED_STATE=BACKUP \ +-e KEEPALIVED_INTERFACE=eth0 \ +-e KEEPALIVED_VIRTUAL_ROUTER_ID=2 \ +-e KEEPALIVED_UNICAST_SRC_IP=10.10.0.22 \ +-e KEEPALIVED_UNICAST_PEER_0=10.10.0.21 \ +-e KEEPALIVED_TRACK_INTERFACE_1=eth0 \ +-e KEEPALIVED_TRACK_INTERFACE_2=eth1 \ +-e KEEPALIVED_VIRTUAL_IPADDRESS_1="10.10.0.3/24 dev eth0" \ +-e KEEPALIVED_VIRTUAL_IPADDRESS_EXCLUDED_1="172.16.1.20/24 dev eth1" \ +quay.io/ceph/keepalived +``` |