diff options
Diffstat (limited to 'src/cephadm/containers')
| -rw-r--r-- | src/cephadm/containers/keepalived/Dockerfile | 24 | ||||
| -rw-r--r-- | src/cephadm/containers/keepalived/LICENSE | 21 | ||||
| -rw-r--r-- | src/cephadm/containers/keepalived/README.md | 233 | ||||
| -rwxr-xr-x | src/cephadm/containers/keepalived/skel/init.sh | 22 |
4 files changed, 300 insertions, 0 deletions
diff --git a/src/cephadm/containers/keepalived/Dockerfile b/src/cephadm/containers/keepalived/Dockerfile new file mode 100644 index 000000000..ac305f72b --- /dev/null +++ b/src/cephadm/containers/keepalived/Dockerfile @@ -0,0 +1,24 @@ +FROM registry.access.redhat.com/ubi8/ubi-minimal:latest + +RUN microdnf install --nodocs \ + bash \ + curl \ + iproute \ + keepalived-2.1.5 \ + && rm /etc/keepalived/keepalived.conf && microdnf clean all + +COPY /skel / + +RUN chmod +x init.sh + +CMD ["./init.sh"] + +# Build specific labels +LABEL maintainer="Guillaume Abrioux <gabrioux@redhat.com>" +LABEL com.redhat.component="keepalived-container" +LABEL version=2.1.5 +LABEL name="keepalived" +LABEL description="keepalived for Ceph" +LABEL summary="Provides keepalived on RHEL 8 for Ceph." +LABEL io.k8s.display-name="Keepalived on RHEL 8" +LABEL io.openshift.tags="Ceph keepalived" diff --git a/src/cephadm/containers/keepalived/LICENSE b/src/cephadm/containers/keepalived/LICENSE new file mode 100644 index 000000000..74b10b143 --- /dev/null +++ b/src/cephadm/containers/keepalived/LICENSE @@ -0,0 +1,21 @@ +MIT License + +Copyright (c) 2017 University of Michigan + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. diff --git a/src/cephadm/containers/keepalived/README.md b/src/cephadm/containers/keepalived/README.md new file mode 100644 index 000000000..bd7b605ac --- /dev/null +++ b/src/cephadm/containers/keepalived/README.md @@ -0,0 +1,233 @@ +# quay.io/ceph/keepalived + +A small [ubi8-minimal](https://catalog.redhat.com/software/containers/registry/registry.access.redhat.com/repository/ubi8/ubi-minimal) based Docker container that provides a method of IP high availability via [keepalived](http://www.keepalived.org/) (VRRP failover), and optional Kubernetes API Server monitoring. If allowed to auto configure (default behaviour) it will automatically generate a unicast based failover configuration with a minimal amount of user supplied information. + +For specific information on Keepalived, please see the man page on [keepalived.conf](http://linux.die.net/man/5/keepalived.conf) or the [Keepalived User Guide](http://www.keepalived.org/pdf/UserGuide.pdf). + + +## Index +- [quay.io/ceph/keepalived](#cephkeepalived) + - [Index](#index) + - [Prerequisites](#prerequisites) + - [Configuration](#configuration) + - [Execution Control](#execution-control) + - [Autoconfiguration Options](#autoconfiguration-options) + - [Kubernetes Options](#kubernetes-options) + - [Suggested Kubernetes Settings](#suggested-kubernetes-settings) + - [Example Keepalived Configs](#example-keepalived-configs) + - [Example Autogenerated Keepalived Master Config](#example-autogenerated-keepalived-master-config) + - [Example Autogenerated Keepalived Backup Config](#example-autogenerated-keepalived-backup-config) + - [Example Run Commands](#example-run-commands) + - [Example Master Run Command](#example-master-run-command) + - [Example Backup Run Command](#example-backup-run-command) + + +## Prerequisites + +Before attempting to deploy the keepalived container, the host must allow non local binding of ipv4 addresses. To do this, configure the sysctl tunable `net.ipv4.ip_nonlocal_bind=1`. + +In addition to enabling the nonlocal binds, the container must be run with both host networking (`--net=host`) and security setting CAP_NET_ADMIN (`--cap-add NET_ADMIN`) capability. These allow the container to manage the host's networking configuration, and this is essential to the function of keepalived. + + +## Configuration +### Execution Control + +| Variable | Default | +|:---------------------:|:------------------------------------------------:| +| `KEEPALIVED_AUTOCONF` | `true` | +| `KEEPALIVED_CONF` | `/etc/keepalived/keepalived.conf` | +| `KEEPALIVED_CMD` | `/usr/sbin/keepalived -n -l -f $KEEPALIVED_CONF` | +| `KEEPALIVED_DEBUG` | `false` | + +* `KEEPALIVED_AUTOCONF` - Enables or disables the auto-configuration of keepalived. + +* `KEEPALIVED_CONF` - The path to the keepalived configuration file. + +* `KEEPALIVED_CMD` - The command called to execute keepalived. + +* `KEEPALIVED_DEBUG` - Enables or disables debug level logging for keepalived (adds `-D` to `KEEPALIVED_CMD`. + + +### Autoconfiguration Options + +| Variable | Default | +|:-------------------------------------------:|:----------------------------------:| +| `KEEPALIVED_ADVERT_INT` | `1` | +| `KEEPALIVED_AUTH_PASS` | `pwd$KEEPALIVED_VIRTUAL_ROUTER_ID` | +| `KEEPALIVED_INTERFACE` | `eth0` | +| `KEEPALIVED_PRIORITY` | `200` | +| `KEEPALIVED_STATE` | `MASTER` | +| `KEEPALIVED_TRACK_INTERFACE_###` | | +| `KEEPALIVED_UNICAST_SRC_IP` | | +| `KEEPALIVED_UNICAST_PEER_###` | | +| `KEEPALIVED_VIRTUAL_IPADDRESS_###` | | +| `KEEPALIVED_VIRTUAL_IPADDRESS_EXCLUDED_###` | | +| `KEEPALIVED_VIRTUAL_ROUTER_ID` | `1` | +| `KEEPALIVED_KUBE_APISERVER_CHECK` | `false` | + +* `KEEPALIVED_ADVERT_INT` - The VRRP advertisement interval (in seconds). + +* `KEEPALIVED_AUTH_PASS` - A shared password used to authenticate each node in a VRRP group (**Note:** If password is longer than 8 characters, only the first 8 characters are used). + +* `KEEPALIVED_INTERFACE` - The host interface that keepalived will monitor and use for VRRP traffic. + +* `KEEPALIVED_PRIORITY` - Election value, the server configured with the highest priority will become the Master. + +* `KEEPALIVED_STATE` - Defines the server role as Master or Backup. (**Options:** `MASTER` or `BACKUP`). + +* `KEEPALIVED_TRACK_INTERFACE_###` - An interface that's state should be monitored (e.g. eth0). More than one can be supplied as long as the variable name ends in a number from 0-999. + +* `KEEPALIVED_UNICAST_SRC_IP` - The IP on the host that the keepalived daemon should bind to. **Note:** If not specified, it will be the first IP bound to the interface specified in `KEEPALIVED_INTERFACE`. + +* `KEEPALIVED_UNICAST_PEER_###` - An IP of a peer participating in the VRRP group. More tha one can be supplied as long as the variable name ends in a number from 0-999. + +* `KEEPALIVED_VIRTUAL_IPADDRESS_###` - An instance of an address that will be monitored and failed over from one host to another. These should be a quoted string in the form of: `<IPADDRESS>/<MASK> brd <BROADCAST_IP> dev <DEVICE> scope <SCOPE> label <LABEL>` At a minimum the ip address, mask and device should be specified e.g. `KEEPALIVED_VIRTUAL_IPADDRESS_1="10.10.0.2/24 dev eth0"`. More than one can be supplied as long as the variable name ends in a number from 0-999. **Note:** Keepalived has a hard limit of **20** addresses that can be monitored. More can be failed over with the monitored addresses via `KEEPALIVED_VIRTUAL_IPADDRESS_EXCLUDED_###`. + + +* `KEEPALIVED_VIRTUAL_IPADDRESS_EXCLUDED_###` - An instance of an address that will be failed over with the monitored addresses supplied via `KEEPALIVED_VIRTUAL_IPADDRESS_###`. These should be a quoted string in the form of: `<IPADDRESS>/<MASK> brd <BROADCAST_IP> dev <DEVICE> scope <SCOPE> label <LABEL>` At a minimum the ip address, mask and device should be specified e.g. `KEEPALIVED_VIRTUAL_IPADDRESS_EXCLUDED_1="172.16.1.20/24 dev eth1"`. More than one can be supplied as long as the variable name ends in a number from 0-999. + +* `KEEPALIVED_VIRTUAL_ROUTER_ID` - A unique number from 0 to 255 that should identify the VRRP group. Master and Backup should have the same value. Multiple instances of keepalived can be run on the same host, but each pair **MUST** have a unique virtual router id. + +* `KEEPALIVED_KUBE_APISERVER_CHECK` - If enabled it configures a simple check script for the Kubernetes API-Server. For more information on this feature, please see the [Kubernetes Options](#kubernetes-options) section. + + +### Kubernetes Options + + +| **Variable** | **Default** | +|:-----------------------------:|:----------------------------------------------:| +| `KUBE_APISERVER_ADDRESS` | parsed from `KEEPALIVED_VIRTUAL_IPADDRESS_###` | +| `KUBE_APISERVER_PORT` | `6443` | +| `KUBE_APISERVER_CHK_INTERVAL` | `3` | +| `KUBE_APISERVER_CHK_FALL` | `10` | +| `KUBE_APISERVER_CHK_RISE` | `2` | +| `KUBE_APISERVER_CHK_WEIGHT` | `-50` | + + + +* `KUBE_APISERVER_ADDRESS` - The Virtual IP being used for the Kube API Server. If none is supplied, it is assumed to be the lowest numbered entry in the `KEEPALIVED_VIRTUAL_IPADDRESS_###` variables. + +* `KUBE_APISERVER_PORT` - The port to use in conjunction with the `KUBE_APISERVER_ADDRESS`. + +* `KUBE_APISERVER_CHK_INTERVAL` - The interval in seconds between calling the script. + +* `KUBE_APISERVER_CHK_FALL` - The number of consecutive non-zero script exits before setting the state to `FAULT`. + +* `KUBE_APISERVER_CHK_RISE` - The number of consecutive zero script exits before exiting the `FAULT` state. + +* `KUBE_APISERVER_CHK_WEIGHT` - The weight to apply to the priority when the service enters the `FAULT` state. + + + +--- + +### Suggested Kubernetes Settings + +Assuming there are three nodes running the kube-apiserver, you cannot rely on setting just the`KEEPALIVED_STATE` parameter to manage failover across the nodes. + +To manage kube-apiserver failover, enable the healthcheck option with `KEEPALIVED_KUBE_APISERVER_CHECK`, and set the `KEEPALIVED_PRIORITY` manually for the three instances. + +| **Node** | **Priority** | +|:--------:|:------------:| +| node-01 | 200 | +| node-02 | 190 | +| node-03 | 180 | + +With the default weight of `-50`, if `node-01` has an issue, it's priority will drop to `150` and allow `node-02` to take over, the same is repeated if `node-02` has a failure dropping it's weight to `140` and `node-03` takes over. + +Recovery occurs in the same order with the system with the highest priority being promoted to master. + +### Example Keepalived Configs + +##### Example Autogenerated Keepalived Master Config +``` +vrrp_instance MAIN { + state MASTER + interface eth0 + virtual_router_id 2 + priority 200 + advert_int 1 + unicast_src_ip 10.10.0.21 + unicast_peer { + 10.10.0.22 + } + authentication { + auth_type PASS + auth_pass pwd1 + } + virtual_ipaddress { + 10.10.0.2/24 dev eth0 + } + virtual_ipaddress_excluded { + 172.16.1.20/24 dev eth1 + } + track_interface { + eth0 + eth1 + } +} +``` + +##### Example Autogenerated Keepalived Backup Config +``` +vrrp_instance MAIN { + state BACKUP + interface eth0 + virtual_router_id 2 + priority 100 + advert_int 1 + unicast_src_ip 10.10.0.22 + unicast_peer { + 10.10.0.21 + } + authentication { + auth_type PASS + auth_pass pwd1 + } + virtual_ipaddress { + 10.10.0.2/24 dev eth0 + } + virtual_ipaddress_excluded { + 172.16.1.20/24 dev eth1 + } + track_interface { + eth0 + eth1 + } +} + +``` + + +## Example Run Commands +##### Example Master Run Command +```bash +docker run -d --net=host --cap-add NET_ADMIN \ +-e KEEPALIVED_AUTOCONF=true \ +-e KEEPALIVED_STATE=MASTER \ +-e KEEPALIVED_INTERFACE=eth0 \ +-e KEEPALIVED_VIRTUAL_ROUTER_ID=2 \ +-e KEEPALIVED_UNICAST_SRC_IP=10.10.0.21 \ +-e KEEPALIVED_UNICAST_PEER_0=10.10.0.22 \ +-e KEEPALIVED_TRACK_INTERFACE_1=eth0 \ +-e KEEPALIVED_TRACK_INTERFACE_2=eth1 \ +-e KEEPALIVED_VIRTUAL_IPADDRESS_1="10.10.0.3/24 dev eth0" \ +-e KEEPALIVED_VIRTUAL_IPADDRESS_EXCLUDED_1="172.16.1.20/24 dev eth1" \ +quay.io/ceph/keepalived +``` + +##### Example Backup Run Command +```bash +docker run -d --net=host --cap-add NET_ADMIN \ +-e KEEPALIVED_AUTOCONF=true \ +-e KEEPALIVED_STATE=BACKUP \ +-e KEEPALIVED_INTERFACE=eth0 \ +-e KEEPALIVED_VIRTUAL_ROUTER_ID=2 \ +-e KEEPALIVED_UNICAST_SRC_IP=10.10.0.22 \ +-e KEEPALIVED_UNICAST_PEER_0=10.10.0.21 \ +-e KEEPALIVED_TRACK_INTERFACE_1=eth0 \ +-e KEEPALIVED_TRACK_INTERFACE_2=eth1 \ +-e KEEPALIVED_VIRTUAL_IPADDRESS_1="10.10.0.3/24 dev eth0" \ +-e KEEPALIVED_VIRTUAL_IPADDRESS_EXCLUDED_1="172.16.1.20/24 dev eth1" \ +quay.io/ceph/keepalived +``` diff --git a/src/cephadm/containers/keepalived/skel/init.sh b/src/cephadm/containers/keepalived/skel/init.sh new file mode 100755 index 000000000..9c86cfad2 --- /dev/null +++ b/src/cephadm/containers/keepalived/skel/init.sh @@ -0,0 +1,22 @@ +#!/bin/bash + +set -e +set -o pipefail + + +KEEPALIVED_DEBUG=${KEEPALIVED_DEBUG:-false} +KEEPALIVED_KUBE_APISERVER_CHECK=${KEEPALIVED_KUBE_APISERVER_CHECK:-false} +KEEPALIVED_CONF=${KEEPALIVED_CONF:-/etc/keepalived/keepalived.conf} +KEEPALIVED_VAR_RUN=${KEEPALIVED_VAR_RUN:-/var/run/keepalived} + +if [[ ${KEEPALIVED_DEBUG,,} == 'true' ]]; then + kd_cmd="/usr/sbin/keepalived -n -l -D -f $KEEPALIVED_CONF" +else + kd_cmd="/usr/sbin/keepalived -n -l -f $KEEPALIVED_CONF" +fi + +KEEPALIVED_CMD=${KEEPALIVED_CMD:-"$kd_cmd"} + +rm -fr "$KEEPALIVED_VAR_RUN" + +exec $KEEPALIVED_CMD
\ No newline at end of file |