ClickHouse monitoring and alerting

ClickHouse itself and its backup sidecar each expose Prometheus metrics, off by default. The backup sidecar deserves the closer attention below: its watch loop retries a failed backup on the next scheduled tick instead of stopping, so a persistent failure never crashes the container or shows up in pod status (see Backup freshness).

Requires Prometheus Operator's CRDs

Both ClickHouse's own ServiceMonitor and the backup sidecar's render a ServiceMonitor resource. If Prometheus Operator's CRDs aren't installed in the cluster, enabling either one makes the release fail to install: leave both false until they are.

ClickHouse server metrics

ClickHouse's own metrics (query performance, resource usage) are exposed on the standard http-metrics port. Two settings gate this, both false by default:

clickhouse-server:
  metrics:
    enabled: true
    serviceMonitor:
      enabled: true

metrics.enabled turns on the metrics endpoint itself; metrics.serviceMonitor.enabled additionally ships a matching ServiceMonitor for it: both are required to get a scraped target.

Key metrics

ClickHouse exposes multiple metrics; the ones below are the ones that map directly to a guardrail this chart configures by default (see Scheduling and reliability and Other guardrails), so a breach here means a specific, known threshold was hit, not a symptom to interpret from scratch.

Metric	Type	Meaning
ClickHouseErrorMetric_TOO_MANY_SIMULTANEOUS_QUERIES	Counter	A query was rejected outright for exceeding MAX_CONCURRENT_QUERIES (32 by default). ClickHouseMetrics_Query (gauge) tracks the current concurrent count if you want to watch the trend before it gets there.
ClickHouseErrorMetric_TIMEOUT_EXCEEDED	Counter	A query was killed for running past MAX_EXECUTION_TIME (55 seconds by default).
ClickHouseMetrics_MemoryTracking / ClickHouseErrorMetric_MEMORY_LIMIT_EXCEEDED	Gauge (bytes) / Counter	Server-wide memory currently tracked, and how many times a query was killed for exceeding MAX_SERVER_MEMORY_USAGE_TO_RAM_RATIO (0.9 of the pod's memory limit by default).
ClickHouseAsyncMetrics_MaxPartCountForPartition	Gauge	The active-part count of the single worst-case partition across every table. PARTS_TO_DELAY_INSERT/PARTS_TO_THROW_INSERT (1000/3000 by default) check it before accepting an insert into that partition. ClickHouse's own guidance: values above 300 already indicate overload.
ClickHouseProfileEvents_DelayedInserts / ClickHouseProfileEvents_RejectedInserts	Counter	Early warning (an insert was artificially slowed) vs. hard failure (an insert was rejected with Too many parts) for the same parts-per-partition guardrails above.
ClickHouseProfileEvents_ExternalAggregationCompressedBytes / ExternalSortCompressedBytes	Counter (bytes)	Cumulative data spilled to disk because a query's GROUP BY/ORDER BY crossed MAX_BYTES_RATIO_BEFORE_EXTERNAL_GROUP_BY/_SORT (0.4 by default). A rising rate means queries are increasingly falling back to the slower disk-spill path.
ClickHouseAsyncMetrics_DiskAvailable_default	Gauge (bytes)	Free space on the metadata volume. Unlike the filesystem cache, this volume doesn't degrade gracefully if it fills (see What happens if this volume fills up).

Filesystem cache hit ratio has its own metrics

Covered separately in Filesystem cache: ClickHouseProfileEvents_CachedReadBufferReadFromCacheHits/...Misses (the same counters behind that page's SQL query) and ClickHouseMetrics_FilesystemCacheSize/...SizeLimit for how full the cache currently is.

A rising rate on either of these is a hardware-sizing signal, not just an alert

Sustained ExternalAggregationCompressedBytes/...SortCompressedBytes growth or MEMORY_LIMIT_EXCEEDED increases both point to RAM, not CPU, being the actual bottleneck for your aggregation workload. See Hardware recommendations for when to move from ClickHouse's general-purpose RAM/core ratio to their data-warehouse one.

Key thresholds

These assume the chart's default guardrail values above; adjust the literal numbers if you've customized clickhouse.serverConfig. Critical means a query or insert has already failed because of it; warning means you're trending toward one.

Condition	Based on	Suggested threshold	Severity
A query was rejected for too many concurrent queries	ClickHouseErrorMetric_TOO_MANY_SIMULTANEOUS_QUERIES	Any increase	Critical
A query was killed for running too long	ClickHouseErrorMetric_TIMEOUT_EXCEEDED	Any increase	Critical
A query was killed for exceeding its memory limit	ClickHouseErrorMetric_MEMORY_LIMIT_EXCEEDED	Any increase	Critical
An insert was rejected outright (Too many parts)	ClickHouseProfileEvents_RejectedInserts	Any increase	Critical
An insert is being throttled, but not yet rejected	ClickHouseProfileEvents_DelayedInserts	Any increase	Warning
A partition is approaching the insert-delay threshold	ClickHouseAsyncMetrics_MaxPartCountForPartition	Above 80% of PARTS_TO_DELAY_INSERT (800 by default) for 15 minutes	Warning
Queries are increasingly spilling to disk	Rate of ExternalAggregationCompressedBytes + ExternalSortCompressedBytes	Sustained non-zero rate for 30 minutes	Warning
The metadata volume is projected to fill up	ClickHouseAsyncMetrics_DiskAvailable_default	predict_linear(...[1d]) < 0 (projected to hit zero within 24h)	Critical

Recommended alerts

An example implementing the thresholds above as a Prometheus Operator PrometheusRule:

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: clickhouse-server
  namespace: <namespace>
  labels:
    release: <your-prometheus-operator-release-label>
spec:
  groups:
    - name: clickhouse-server
      rules:
        - alert: ClickHouseTooManyConcurrentQueries
          expr: increase(ClickHouseErrorMetric_TOO_MANY_SIMULTANEOUS_QUERIES[5m]) > 0
          for: 1m
          labels:
            severity: critical
          annotations:
            summary: 'A query was rejected for exceeding the concurrent query limit'
            description: 'MAX_CONCURRENT_QUERIES (32 by default) was hit. ClickHouseMetrics_Query shows the current concurrent count if you want to confirm the trend.'

        - alert: ClickHouseQueryTimeout
          expr: increase(ClickHouseErrorMetric_TIMEOUT_EXCEEDED[5m]) > 0
          for: 1m
          labels:
            severity: critical
          annotations:
            summary: 'A query was killed for exceeding MAX_EXECUTION_TIME'
            description: 'MAX_EXECUTION_TIME is 55 seconds by default. Check slow query logs for what ran long.'

        - alert: ClickHouseQueryMemoryLimitExceeded
          expr: increase(ClickHouseErrorMetric_MEMORY_LIMIT_EXCEEDED[5m]) > 0
          for: 1m
          labels:
            severity: critical
          annotations:
            summary: 'A query was killed for exceeding the server memory limit'
            description: 'MAX_SERVER_MEMORY_USAGE_TO_RAM_RATIO is 0.9 by default. ClickHouseMetrics_MemoryTracking shows the server-wide trend leading up to this.'

        - alert: ClickHouseInsertRejected
          expr: increase(ClickHouseProfileEvents_RejectedInserts[5m]) > 0
          for: 1m
          labels:
            severity: critical
          annotations:
            summary: "An insert was rejected with 'Too many parts'"
            description: 'PARTS_TO_THROW_INSERT (3000 by default) was hit for at least one partition. ClickHouseAsyncMetrics_MaxPartCountForPartition shows the worst-case partition if you want to confirm which one.'

        - alert: ClickHouseInsertDelayed
          expr: increase(ClickHouseProfileEvents_DelayedInserts[5m]) > 0
          for: 5m
          labels:
            severity: warning
          annotations:
            summary: 'Inserts are being throttled due to a high part count'
            description: 'PARTS_TO_DELAY_INSERT (1000 by default) was hit for at least one partition. Not yet a rejection, but merges are falling behind inserts. Left unaddressed, this trends toward ClickHouseInsertRejected.'

        - alert: ClickHouseMaxPartCountApproachingLimit
          expr: ClickHouseAsyncMetrics_MaxPartCountForPartition > 800
          for: 15m
          labels:
            severity: warning
          annotations:
            summary: 'A partition is approaching the insert-delay threshold'
            description: 'Above 80% of PARTS_TO_DELAY_INSERT (1000 by default). Sustained high insert rate outpacing background merges. Consider batching inserts at the source.'

        - alert: ClickHouseExternalSpillIncreasing
          expr: rate(ClickHouseProfileEvents_ExternalAggregationCompressedBytes[30m]) > 0 or rate(ClickHouseProfileEvents_ExternalSortCompressedBytes[30m]) > 0
          for: 30m
          labels:
            severity: warning
          annotations:
            summary: 'Queries are spilling GROUP BY/ORDER BY to disk'
            description: 'MAX_BYTES_RATIO_BEFORE_EXTERNAL_GROUP_BY/_SORT (0.4 by default) is being crossed repeatedly, a sign these queries are memory-constrained relative to their data volume.'

        - alert: ClickHouseMetadataDiskWillFillUp
          expr: predict_linear(ClickHouseAsyncMetrics_DiskAvailable_default[1d], 86400) < 0
          for: 10m
          labels:
            severity: critical
          annotations:
            summary: 'The metadata volume is projected to fill up within 24 hours'
            description: 'Unlike the filesystem cache, this volume does not degrade gracefully when full: inserts and merges fail outright. See Metadata volume in the sizing guide.'

ClickHouseInsertRejected and its threshold choices mirror Altinity's own clickhouse-operator reference alerting rules (ClickHouseRejectedInsert/ClickHouseDelayedInsertThrottling/ClickHouseMaxPartCountForPartition), and ClickHouseMetadataDiskWillFillUp's predict_linear pattern mirrors their ClickHouseDiskUsage rule. Both are scoped down here to the settings this chart actually configures. Most of Altinity's other rules (replica lag, ZooKeeper/Keeper sessions, Kafka, Distributed tables) don't apply: this chart runs ClickHouse single-node, without replication, Kafka, or sharding.

Backup sidecar metrics

The backup sidecar fails silently: do not rely on the pod's Kubernetes status

A failed backup does not crash the container, does not restart the pod, and does not turn any Kubernetes-level health indicator red. The clickhouse pod can show Running/Ready with 0 restarts while backups have been failing for days. The only reliable way to know is to build alerting on the Prometheus metrics below.

Metrics endpoint

The backup sidecar exposes Prometheus-format metrics on its own REST API, on the backup-rest port (7171), at /metrics. Like every other route on that API, it requires the same HTTP Basic Auth credentials as the rest of the backup API (API_USERNAME/API_PASSWORD, see Prerequisites): an unauthenticated scrape gets a 401.

To check it manually before wiring up Prometheus: API_USERNAME/API_PASSWORD are already injected into the clickhouse-backup container itself (it needs them to enforce this same auth on incoming requests), so there's no need to pull them from the secret separately:

kubectl exec -n <namespace> <clickhouse-pod> -c clickhouse-backup -- \
  sh -c 'curl -s -u "$API_USERNAME:$API_PASSWORD" http://localhost:7171/metrics'

Scraping with Prometheus Operator

Set clickhouse.backup.metrics.enabled: true to have the chart ship a matching ServiceMonitor for the backup-rest port automatically: it sources both the username and password from the same clickhouse-credentials secret already used elsewhere (see Prerequisites):

clickhouse:
  backup:
    metrics:
      enabled: true

Key metrics

Every scheduled tick (full or incremental) runs as a create_remote operation, whether or not that day's full backup also triggers a rebase (see Understanding the backup schedule):

Metric	Type	Meaning
clickhouse_backup_last_create_remote_status	Gauge	0=failed, 1=success, 2=unknown. The outcome of the most recent scheduled backup attempt, full or incremental.
clickhouse_backup_last_create_remote_finish	Gauge (unix timestamp)	Timestamp of the most recent attempt, success or failure. Going stale means the watch loop itself has stopped ticking, not just that a backup failed.
clickhouse_backup_successful_create_remotes / clickhouse_backup_failed_create_remotes	Counter	Running counters, useful for a failure-rate alert over a rolling window.
clickhouse_backup_last_rebase_status / clickhouse_backup_last_rebase_finish	Gauge	Same shape, for the weekly rebase step that runs after a full backup once a previous chain already exists. Stays unset for the first ~2 weeks after a fresh install, since full_type: rebase only applies from the second full cycle onward.
clickhouse_backup_in_progress_commands	Gauge	How many backup operations are currently running. Stuck above 0 for far longer than a backup normally takes usually means a hung operation.
clickhouse_backup_number_backups_remote / clickhouse_backup_last_backup_size_remote	Gauge	Chain count and size currently on remote storage, a basic sanity check that the chain count/size hasn't unexpectedly collapsed.

Key thresholds

These thresholds are metric-agnostic: they apply however the metrics above are collected, whether that's Prometheus, Datadog, Grafana Cloud, or any other monitoring solution. They assume the chart's default schedule (daily create_remote, weekly rebase); adjust them if you've customized WATCH_SCHEDULES.

Condition	Based on	Suggested threshold	Severity
No backup attempt (full or incremental) recently	Time since clickhouse_backup_last_create_remote_finish	More than 36 hours (1.5x the default daily cadence)	Critical
The last backup attempt failed	clickhouse_backup_last_create_remote_status	Equal to 0 (failed) for 15 minutes straight	Critical
No weekly full-backup rebase recently	Time since clickhouse_backup_last_rebase_finish	More than 10 days (only meaningful once a second full cycle has run, expect this to be stale for the first two weeks after a fresh install)	Warning
A backup operation has been running too long	clickhouse_backup_in_progress_commands	Greater than 0 for 2 hours straight	Warning
The most recent remote backup reports a size of zero	clickhouse_backup_last_backup_size_remote and clickhouse_backup_number_backups_remote	Size equal to 0 while at least one backup already exists remotely	Critical

Recommended alerts

An example implementing the thresholds above as a Prometheus Operator PrometheusRule:

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: clickhouse-backup
  namespace: <namespace>
  labels:
    release: <your-prometheus-operator-release-label>
spec:
  groups:
    - name: clickhouse-backup
      rules:
        - alert: ClickHouseBackupStalled
          expr: time() - clickhouse_backup_last_create_remote_finish > 36 * 3600
          for: 10m
          labels:
            severity: critical
          annotations:
            summary: 'No backup attempt (full or incremental) in over 36 hours'
            description: 'The clickhouse-backup sidecar has not attempted a backup in more than 36 hours (1.5x the default daily cadence). The watch loop may have stopped, or its config may be invalid. Pod status will not show this: check the sidecar logs directly.'

        - alert: ClickHouseBackupFailing
          expr: clickhouse_backup_last_create_remote_status == 0
          for: 15m
          labels:
            severity: critical
          annotations:
            summary: 'The last backup attempt failed'
            description: 'clickhouse_backup_last_create_remote_status is 0 (failed). Check the sidecar logs for the error; clickhouse-backup retries on the next scheduled tick, it does not stop or crash on its own.'

        - alert: ClickHouseFullBackupStalled
          expr: time() - clickhouse_backup_last_rebase_finish > 10 * 24 * 3600
          for: 10m
          labels:
            severity: warning
          annotations:
            summary: 'No weekly full-backup rebase in over 10 days'
            description: 'Only meaningful once a second full cycle has run. Expect this to be stale for the first two weeks after a fresh install, since full_type: rebase only applies from the second full backup onward.'

        - alert: ClickHouseBackupStuck
          expr: clickhouse_backup_in_progress_commands > 0
          for: 2h
          labels:
            severity: warning
          annotations:
            summary: 'A backup operation has been running for over 2 hours'
            description: 'clickhouse_backup_in_progress_commands has stayed above 0 for 2 hours straight, longer than a scheduled full or incremental backup normally takes. Check the sidecar logs for a hung operation. Raise the threshold if your data volume makes 2 hours a normal backup duration.'

        - alert: ClickHouseBackupSizeZero
          expr: clickhouse_backup_last_backup_size_remote == 0 and clickhouse_backup_number_backups_remote > 0
          for: 10m
          labels:
            severity: critical
          annotations:
            summary: 'The most recent remote backup reports a size of zero'
            description: 'At least one backup exists on remote storage, but the most recent one reports 0 bytes, likely a broken or empty backup rather than a real one. Check the sidecar logs and verify the backup with `clickhouse-backup list remote`.'

ClickHouseBackupStalled's 36-hour window is deliberately generous (1.5x the 24-hour default cadence) to avoid flapping on a single delayed tick, matching the convention clickhouse-backup's own maintainers use in their reference alerting rules for the ClickHouse Operator. ClickHouseBackupSizeZero mirrors the same reference rule set's ClickHouseRemoteBackupSizeZero.