ClickHouse sizing and hardening

Hardware recommendations

Use the table below as a starting point for compute sizing, based on the number of active endpoints (machines) reporting data:

Tier	Endpoints	ClickHouse cores	ClickHouse RAM	Cache volume
Small	~100	4	16 GiB	30Gi
Medium	~1,000	6	24 GiB	50Gi
Large	~10,000	8+	32 GiB+	100Gi

CPU is the main lever for the heavier aggregation queries: scale cores with data volume rather than over-provisioning RAM alone. Start with 4 GiB of RAM per core (ClickHouse's own "general purpose" ratio), and move up to 8 GiB of RAM per core (their "data warehousing" ratio) if you observe external GROUP BY/ORDER BY spill or memory-limit query rejections under normal load: both are concrete signs that RAM, not CPU, is the actual bottleneck for your aggregation workload (see Key metrics for the exact counters to watch).

• Node type: a general-purpose or memory-optimized instance type with 4 to 8 GiB of RAM per core. Avoid compute-optimized instance types: they tend to be under-provisioned on RAM for ClickHouse's aggregation workloads.
• Disk: use local NVMe or network-attached SSD (high IOPS) for the metadata and cache volumes. Never use HDD-class storage or network filesystems (NFS/EFS/Azure Files); their latency characteristics are incompatible with ClickHouse's read/write patterns.
• Dedicated, stable node: schedule ClickHouse on a dedicated, on-demand node (not spot/preemptible) to avoid disruption from node reclamation. Avoid co-locating noisy-neighbor workloads.
• Object storage: elastic by design, not a sizing driver the way local disk is, since it scales independently of your cluster.

Filesystem cache

The local filesystem cache sits in front of object storage and only needs to hold your working set (the data actually touched by recent queries), not your entire historical dataset. Because analytics features typically read only the current state per machine or account, superseded historical data doesn't inflate the working set even as your total object storage footprint grows: the cache tracks your number of active endpoints, not your total accumulated history.

That said, the cache remains one of the most effective levers for query performance, and it's always preferable to oversize it than to undersize it: growing it later means resizing a StatefulSet's persistent volume claim template, an operation Kubernetes doesn't support in place (see the caution below).

Guidance for sizing the cache:

• The chart defaults clickhouse.serverConfig.CACHE_MAX_SIZE to 48Gi, backed by a 50Gi cache persistent volume (the Medium tier above); the same setting applies regardless of which object storage provider you use. Use the Cache volume column in the table above as a starting point for your own tier, pairing CACHE_MAX_SIZE about 2Gi below the volume size (see the margin guidance below). Adjust it upward based on observed read latency: consistently slow reads on previously-queried data are a sign the cache is undersized relative to your working set.
• Revisit this sizing as your endpoint count grows.

Resizing the cache volume later is a manual operation

Kubernetes doesn't allow resizing a StatefulSet's persistent volume claim template in place, so growing the cache volume after installation (while fully possible) takes a manual procedure, not a simple values change. Size upfront using the table above based on your expected endpoint count, rather than starting small and expecting to resize easily later. If your environment is resource-constrained, we still recommend not going below 30Gi (the Small tier) for the cache volume.

Measure the actual cache hit ratio

system.events exposes the raw hit/miss counters behind the cache, a more precise signal than latency alone:

SELECT round(sumIf(value, event = 'CachedReadBufferReadFromCacheHits') / (sumIf(value, event = 'CachedReadBufferReadFromCacheHits') + sumIf(value, event = 'CachedReadBufferReadFromCacheMisses')), 4) AS cache_hit_ratio FROM system.events;

A consistently low ratio is a concrete signal to grow CACHE_MAX_SIZE and the cache persistent volume above.

clickhouse:
  serverConfig:
    CACHE_MAX_SIZE: '98Gi' # pairs with a 100Gi cache volume from the Large tier above

Size the cache persistent volume larger than CACHE_MAX_SIZE

CACHE_MAX_SIZE is a soft limit that ClickHouse enforces itself through LRU eviction, not a hard guarantee against the underlying volume filling up. Always give the persistent volume headroom above it, never size it equal to or below: for example, CACHE_MAX_SIZE: '28Gi' on a 30Gi volume is fine.

If the volume does fill up regardless, ClickHouse doesn't fail queries or inserts; it logs a warning and skips caching that entry, falling back to reading directly from object storage. Left unaddressed, though, this permanently degrades reads on that data to the slower path instead of the cached one.

Metadata volume

Unlike the cache, this volume isn't sized against your working set: its own footprint barely grows with data volume. The S3 disk's local metadata (a mapping file per object stored, tens of bytes each) and table definitions stay in the low megabytes even at large scale, regardless of endpoint count or retention.

Start with a minimum of 20Gi anyway. The footprint itself doesn't need that much, but it's cheap headroom against the backup sidecar's FREEZE hardlinks, which briefly coexist on this same volume during a backup cycle (see Backup and restore), and any local system tables you re-enable beyond the chart's defaults.

What happens if this volume fills up

Unlike the cache disk, ClickHouse doesn't degrade gracefully here: this is a load-bearing volume, not a best-effort cache. If it fills, inserts and merges fail with real errors instead of falling back elsewhere. Reserve free space on this disk yourself, via the <default> disk's keep_free_space_bytes in your own 00-object-storage.xml (see ClickHouse storage), so ClickHouse refuses new writes predictably once space runs low instead of running the disk down to 0 bytes free.

Scheduling and reliability

The chart applies a set of guardrails by default, so ClickHouse fails predictably (clean rejections) instead of destabilizing the node under load. The ones below need input from you; see Other guardrails for the rest.

Reserved resources

Set requests equal to limits on both CPU and memory for the main ClickHouse container, so the scheduler reserves the full amount upfront instead of over-committing the node. Cap CPU below the node's allocatable capacity to leave headroom for the kubelet. Adjust the values below to match your sizing tier:

clickhouse-server:
  resources:
    requests:
      cpu: 8
      memory: 32Gi
    limits:
      cpu: 8
      memory: 32Gi

The clickhouse-backup sidecar in the same pod runs on its own, separate (and smaller) budget, so the pod as a whole doesn't reach Kubernetes' Guaranteed QoS class. This only pins the main container's resources.

Node stability

You need to prevent your Kubernetes autoscaler, if any, from consolidating or evicting the node while ClickHouse is running on it, which would otherwise detach and reattach its volume mid-operation:

clickhouse-server:
  podAnnotations:
    karpenter.sh/do-not-disrupt: 'true' # adjust the annotation for your own cluster autoscaler if not using Karpenter

Where possible, pin ClickHouse to dedicated, on-demand node(s) reserved for it: this keeps it off spot/preemptible capacity (a reclaimed spot node means an abrupt eviction and a volume re-attach) and away from noisy-neighbor workloads competing for the same CPU/memory:

clickhouse-server:
  nodeSelector:
    workload: clickhouse
  tolerations:
    - key: workload
      operator: Equal
      value: clickhouse
      effect: NoSchedule

This assumes a dedicated node pool that you have provisioned as on-demand (not spot/preemptible), labeled workload: clickhouse and tainted workload=clickhouse:NoSchedule; adjust the label and taint to match your own cluster's convention. The nodeSelector only pins ClickHouse to that pool, it doesn't select the pool's capacity type, so keeping it off spot depends on the pool itself being on-demand.

On a dedicated, non-shared node you can also give the ClickHouse pod a high priorityClassName. There it's largely belt-and-suspenders (nothing of consequence competes for the node), but it's a zero-cost extra hardening guardrail: even a dedicated node still runs DaemonSets (CNI, log/metrics agents), and priority decides who the kubelet evicts first if the node comes under resource pressure.

infrastructure-critical is not a built-in Kubernetes class (the only native ones, system-node-critical and system-cluster-critical, are reserved for control-plane/system pods), so you must create it in your cluster beforehand (the chart doesn't):

apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: infrastructure-critical
value: 1000000 # above your default workloads, well below the system-* classes (~2e9)
globalDefault: false
description: 'High scheduling priority for infrastructure-critical workloads'

Then reference it on the ClickHouse pod:

clickhouse-server:
  priorityClassName: infrastructure-critical

Make sure your reserved node pool has capacity in the PV's zone

The metadata and cache volumes (network-attached SSD) are zone-locked once provisioned. If ClickHouse's node is lost and your reserved node pool can't provision replacement capacity in that same zone, the pod stays Pending indefinitely: it cannot reschedule to a node in a different zone without a new volume. Make sure the node pool you dedicate to ClickHouse spans (or can scale in) every zone your cluster provisions volumes in.

Other guardrails

Beyond node-level scheduling, the chart also configures a set of server-side guardrails by default (memory and concurrency limits, disk headroom, query spill and timeout thresholds, and insert deduplication) so ClickHouse fails predictably instead of destabilizing under load. These ship with sensible defaults and don't need any configuration from you.

If you observe symptoms like disk spill, memory pressure, or query rejections under high dashboard concurrency, please reach out to our support team to evaluate a configuration better suited to your workload.