Alert rules
This page contains a markdown version of the alert rules described in the postgresql-k8s-operator repository. The following file(s) are the source of truth:
src/prometheus_alert_rules/postgresql_rules.yamlsrc/prometheus_alert_rules/pgbouncer_rules.yamlsrc/prometheus_alert_rules/patroni_rules.yaml
This documentation describes the latest alert rule expressions. See the YAML file(s) on listed above if you require an older version.
PostgresqlExporterK8s
| Alert | Severity | Notes |
|---|---|---|
| PostgresqlDown |  |
PostgresSQL instance is down. If you are not upgrading or configuring cross-region async replication clusters, check for errors in the Loki logs. |
| PostgresqlRestarted |  |
PostgresSQL instance has restarted. If you are not enabling/disabling TLS or upgrading or configuring cross-region async replication clusters, check for errors in the Loki logs. |
| PostgresqlExporterError | |
PostgresSQL instance is showing an exporter error. There may be a buggy query in query.yaml |
| PostgresqlTableNotAutoVacuumed |  |
A PostgresSQL table in instance is not auto vacuumed. A table has not been auto vacuumed for 7 days.Double-check your VACUUM settings. |
| PostgresqlTableNotAutoAnalyzed | |
A PostgresSQL table in instance is not auto analyzed. A table has not been auto analyzed for 7 days.Double-check your AUTOVACUUM ANALYZE settings. |
| PostgresqlTooManyConnections | |
PostgresSQL instance is using > 80% of the maximum connections. Consider checking how many connections the client application is opening, or using PgBouncer in front of the database. |
| PostgresqlNotEnoughConnections | |
PostgresSQL instance does not have enough connections. PostgreSQL instance should have more connections (> 5). Consider double-checking how many connections the client application is opening and/or using PgBouncer in front of the database. |
| PostgresqlDeadLocks | |
PostgresSQL instance has dead locks. See more details with the pg_locks view. |
| PostgresqlHighRollbackRate | |
PostgresSQL instance has a high rollback rate instance. The ratio of transactions being aborted compared to committed is > 2 %. This is probably happening due to unoptimized configurations related to commit delay, connections, memory, and WAL files. |
| PostgresqlCommitRateLow | |
PostgresSQL instance has a low commit rate. PostgresSQL seems to be processing very few transactions. Check for long-running queries and configuration issues, like insufficient cache size. |
| PostgresqlLowXidConsumption | |
PostgresSQL instance shows low XID consumption. PostgresSQL seems to be consuming transaction IDs very slowly. Run ANALYZE to update the optimizer statistics, ensure that query plans are correct, and double-check your VACUUM settings. |
| PostgresqlHighRateStatementTimeout | |
PostgresSQL instance shows a high rate of statement timeout. Either tune statement_timeout when sending queries or use EXPLAIN ANALYZE to understand how the queries can be improved. |
| PostgresqlHighRateDeadlock | |
PostgresSQL instance shows a high deadlock rate. More details can be obtained through the pg_locks view. |
| PostgresqlUnusedReplicationSlot | |
PostgresSQL instance has unused replication slots. Check if a replica is not using any of them before deleting it. |
| PostgresqlTooManyDeadTuples | |
PostgresSQL instance has too many dead tuples. Double-check your VACUUM settings. |
| PostgresqlConfigurationChanged | |
PostgresSQL instance configuration has changed. PostgresSQL database configuration has changed. |
| PostgresqlSslCompressionActive | |
PostgresSQL instance SSL compression is active. Database connections with SSL compression are enabled. This may add significant jitter in replication delay.Replicas should turn off SSL compression via sslcompression=0 in recovery.conf. |
| PostgresqlTooManyLocksAcquired | |
PostgreSQL instance has acquired too many locks. If this alert happens frequently, you may need to increase the PostgresSQL setting max_locks_per_transaction. |
| PostgresqlBloatIndexHigh(>80%) | |
PostgreSQL instance has a high bloat index (> 80%). An index is bloated.Consider running REINDEX INDEX CONCURRENTLY <index name>; |
| PostgresqlBloatTableHigh(>80%) | |
PostgreSQL instance has a high bloat table (> 80%). A table is bloated.Consider running VACUUM {{ $labels.relname }}; |
| PostgresqlInvalidIndex | |
PostgresSQL instance )= has an invalid index. A table has an invalid index. Consider running DROP INDEX <index name>; |
PgbouncerExporterK8s
| Alert | Severity | Notes |
|---|---|---|
| PgbouncerActiveConnections | |
PgBouncer instance has > 200 active connections Consider checking the client application responsible for generating those additional connections. |
| PgbouncerErrors | |
PgBouncer instance is logging errors. This may be due to a a server restart or an admin typing commands at the PgBouncer console. |
| PgbouncerMaxConnections | |
PgBouncer instance has reached max_client_conn.Consider checking how many connections the client application is opening. |
PatroniExporterK8s
| Alert | Severity | Notes |
|---|---|---|
| PatroniPostgresqlDown | |
Patroni PostgreSQL instance is down. Check for errors in the Loki logs. |
| PatroniHasNoLeader | |
Patroni instance has no leader node. A leader node (neither primary nor standby) cannot be found inside a cluster. Check for errors in the Loki logs. |