## Alert Rule Standards The alert rules should adhere to the following principles - each alert must have a unique name - each alert should define a common structure - labels : must contain severity and type - annotations : must provide description - expr : must define the promql expression - alert : defines the alert name - alerts that have a corresponding section within docs.ceph.com must include a documentation field in the annotations section - critical alerts should declare an oid in the labels section - critical alerts should have a corresponding entry in the Ceph MIB   ## Testing Prometheus Rules Once you have updated the `ceph_default_alerts.yml` file, you should use the `validate_rules.py` script directly, or via `tox` to ensure the format of any update or change aligns to our rule structure guidelines. The validate_rules.py script will process the rules and look for any configuration anomalies and output a report if problems are detected. Here's an example run, to illustrate the format and the kinds of issues detected. ``` [paul@myhost tests]$ ./validate_rules.py Checking rule groups cluster health : .. mon : E.W.. osd : E...W......W.E.. mds : WW mgr : WW pgs : ..WWWW.. nodes : .EEEE pools : EEEW. healthchecks : . cephadm : WW. prometheus : W rados : W Summary Rule file : ../alerts/ceph_default_alerts.yml Unit Test file : test_alerts.yml Rule groups processed : 12 Rules processed : 51 Rule errors : 10 Rule warnings : 16 Rule name duplicates : 0 Unit tests missing : 4 Problem Report Group Severity Alert Name Problem Description ----- -------- ---------- ------------------- cephadm Warning Cluster upgrade has failed critical level alert is missing an SNMP oid entry cephadm Warning A daemon managed by cephadm is down critical level alert is missing an SNMP oid entry mds Warning Ceph Filesystem damage detected critical level alert is missing an SNMP oid entry mds Warning Ceph Filesystem switched to READ ONLY critical level alert is missing an SNMP oid entry mgr Warning mgr module failure critical level alert is missing an SNMP oid entry mgr Warning mgr prometheus module is not active critical level alert is missing an SNMP oid entry mon Error Monitor down, quorum is at risk documentation link error: #mon-downwah not found on the page mon Warning Ceph mon disk space critically low critical level alert is missing an SNMP oid entry nodes Error network packets dropped invalid alert structure. Missing field: for nodes Error network packet errors invalid alert structure. Missing field: for nodes Error storage filling up invalid alert structure. Missing field: for nodes Error MTU Mismatch invalid alert structure. Missing field: for osd Error 10% OSDs down invalid alert structure. Missing field: for osd Error Flapping OSD invalid alert structure. Missing field: for osd Warning OSD Full critical level alert is missing an SNMP oid entry osd Warning Too many devices predicted to fail critical level alert is missing an SNMP oid entry pgs Warning Placement Group (PG) damaged critical level alert is missing an SNMP oid entry pgs Warning Recovery at risk, cluster too full critical level alert is missing an SNMP oid entry pgs Warning I/O blocked to some data critical level alert is missing an SNMP oid entry pgs Warning Cluster too full, automatic data recovery impaired critical level alert is missing an SNMP oid entry pools Error pool full invalid alert structure. Missing field: for pools Error pool filling up (growth forecast) invalid alert structure. Missing field: for pools Error Ceph pool is too full for recovery/rebalance invalid alert structure. Missing field: for pools Warning Ceph pool is full - writes blocked critical level alert is missing an SNMP oid entry prometheus Warning Scrape job is missing critical level alert is missing an SNMP oid entry rados Warning Data not found/missing critical level alert is missing an SNMP oid entry Unit tests are incomplete. Tests missing for the following alerts; - Placement Group (PG) damaged - OSD Full - storage filling up - pool filling up (growth forecast) ```