Skip to main content

ClickHouse storage

ClickHouse's storage splits into three parts:

  • Object storage: the bucket you provide (S3-compatible, Azure Blob Storage, or GCS), holding the actual table data. This is where the bulk of the storage footprint lives.
  • Filesystem cache: a local persistent volume acting as an LRU cache in front of object storage, for fast reads of recently accessed data (the same pattern used by ClickHouse Cloud).
  • Metadata: a separate local persistent volume holding table structure and part manifests for the data held in object storage.

This page covers the two local volumes first, then the object storage backends and their configuration.

Local persistent volumes​

The metadata and cache volumes are both provisioned locally, independently of your object storage backend, and kept apart from each other: as shown in the diagram above, ClickHouse writes to both, but only the cache ever talks to object storage.

Metadata volume​

The chart already provisions the metadata volume via persistence (enabled, 20Gi, ReadWriteOnce), mounted at ClickHouse's data directory, where it holds table structure and part manifests for the data held in object storage. In your values you only set the storage class (the chart leaves it unset, so your cluster's default StorageClass applies, which may not be SSD-backed) and, if needed, adjust the size:

clickhouse-server:
  persistence:
    storageClass: gp3 # set an SSD/NVMe-backed class, see Storage class below
    size: 20Gi # optional; defaults to 20Gi, see Metadata volume sizing below

The size is illustrative, see Metadata volume for sizing guidance; unlike the cache, its footprint barely grows with data volume.

Filesystem cache​

Provisioned as a second, separate volume via extraVolumeClaimTemplates + extraVolumeMounts, mounted at /cache, kept apart from the metadata volume so a full cache doesn't starve the metadata disk (and vice versa). Point your object storage backend's cache disk <path> at this mount (shown in each backend's configuration below):

clickhouse-server:
  extraVolumeClaimTemplates:
    - metadata:
        name: cache
      spec:
        accessModes:
          - ReadWriteOnce
        storageClassName: gp3 # adapt to your provider, use an SSD/NVMe-backed class, see Storage class below
        resources:
          requests:
            storage: 50Gi # must stay above CACHE_MAX_SIZE (see Filesystem cache below); 50Gi pairs with the chart's default 48Gi
  extraVolumeMounts:
    - name: cache
      mountPath: /cache

The storage above is illustrative, see Filesystem cache for sizing guidance: it only needs to hold your working set, not your full dataset, and should be sized above CACHE_MAX_SIZE.

Storage class​

Use the same class of storage for both volumes: high IOPS is required for metadata as well as the cache:

ClassExamplesNotes
Network-attached SSD (recommended)AWS EBS gp3 / io2, Azure Premium SSD (Premium_LRS), GCP pd-ssd / pd-balancedSurvives node loss or reschedule: the volume follows the pod to a new node.
Local NVMeAWS instance store (i3 / i4i instance types), an on-prem local-volume StorageClassFastest option, but pinned to the node it was provisioned on, see the caution below.
Not supportedHDD-class (st1 / sc1, pd-standard, Standard HDD), network filesystems (NFS, EFS, Azure Files, Filestore)Latency is incompatible with ClickHouse's read/write patterns.
Local NVMe pins the pod to its node

A local-path StorageClass ties the volume to the specific node it was provisioned on: unlike network-attached storage, it cannot simply follow the pod to a different node. If your cluster autoscaler consolidates or drains that node, the pod cannot be rescheduled without losing the volume. If you use local NVMe, prevent your autoscaler from disrupting the node (see Node stability), and plan to restore from backup rather than expect a simple reschedule if the node is lost anyway.

Object storage backends​

ClickHouse stores its table data on an object storage bucket that you provide. Each provider below has a single supported authentication path: credential-less on AWS S3 and Azure (no static keys to create, store, or rotate), and static access-key/secret-key credentials on GCS and MinIO: GCS because credential-less access isn't possible for ClickHouse's own disk (see the GCS tab for why), MinIO because it has no cloud identity provider to federate with in the first place.

Use a dedicated bucket, and a separate one for backups

Use a bucket dedicated to ClickHouse: don't share it with unrelated data. The backup sidecar writes to a bucket of its own too: use a genuinely separate bucket for it, not just a different prefix inside ClickHouse's own data bucket. A shared bucket means a single failure (a bad lifecycle policy, an over-permissioned credential, an accidental deletion) can take out your live data and your backups together, defeating the point of having a backup at all.

Outbound network access

If your cluster restricts egress (NetworkPolicies, an egress firewall or proxy), ClickHouse and the backup sidecar both need outbound access to your object storage endpoint. With a credential-less setup, they also need to reach your cloud provider's identity token-exchange endpoint to trade the projected token for short-lived credentials: AWS STS (sts.amazonaws.com, or your region's STS endpoint) for IRSA, Microsoft Entra ID (login.microsoftonline.com) for Workload Identity.

Bucket lifecycle policies​

ClickHouse's own data bucket and the backup sidecar's bucket need different, in places opposite lifecycle rules and protections. Both are covered here. Read the subsection for each bucket you're configuring.

ClickHouse data bucket​

Don't apply a lifecycle rule to the bucket (or prefix) backing a ClickHouse disk that transitions or expires objects: ClickHouse expects every object it wrote to remain immediately readable indefinitely. A transition to an archive storage tier (S3 Glacier/Deep Archive and equivalents, including S3 Intelligent-Tiering's optional Archive Access tiers) makes the object unreadable until restored, which can make ClickHouse fail to start or throw errors reading existing parts; an expiration rule deletes data ClickHouse still expects to find, risking data loss. Only ClickHouse itself (via TTL / DROP PARTITION) or an informed administrator should ever remove these objects.

The one lifecycle rule that is safe: aborting incomplete multipart uploads after a few days. This only cleans up orphaned upload fragments left behind by interrupted uploads (never live data), and prevents them from accumulating unnoticed in your bucket.

Don't add versioning or Object Lock to this bucket either

ClickHouse deletes and rewrites objects continuously as part of normal operation (merges, TTL, DROP PARTITION). Neither adds a real disaster-recovery benefit here (that's what the backup bucket is for), and both work against normal operation instead: versioning piles up delete markers with no purge plan of their own, and Object Lock blocks ClickHouse's own legitimate deletes outright, not just accidental ones.

Backup bucket​

The same core rule as the data bucket applies here too, for the same reason: don't let a lifecycle rule transition or expire objects that clickhouse-backup still manages. It tracks its own retention (BACKUPS_TO_KEEP_REMOTE) and chain dependencies (REBASE_BEFORE_REMOVE_OLD_REMOTE, see Tunable settings) directly, and expects every backup inside that window to stay immediately readable and to be deleted on its own terms, not S3's. An external expiration rule deletes backups it still expects to find; a transition to an archive tier can break a rebase's server-side copy from a still-referenced older chain. The same multipart-abort rule above is safe here too.

Beyond that baseline, this bucket exists specifically to survive a disaster that takes out ClickHouse's own bucket, so hardening it further is worth the extra cost and complexity that isn't justified on the data bucket:

  • Versioning: recommended. Defense in depth against a compromised credential or a bug deleting backups outright, on top of clickhouse-backup's own retention logic.
  • Object Lock (Governance or Compliance mode): worth it if you have a ransomware or compliance requirement to defend against, but set the retention period shorter than your effective BACKUPS_TO_KEEP_REMOTE window. Otherwise it blocks clickhouse-backup's own routine pruning too, not just malicious deletes. We haven't verified how it handles a blocked delete, so the safe design goal is that the two windows never actually collide.
  • Cross-region or cross-account replication: recommended for a real disaster-recovery posture. A separate bucket already protects against a shared misconfiguration or credential; replicating it further protects against losing the account or region it lives in.
  • Archive-tier transition: only safe for backups you've explicitly exported outside of clickhouse-backup's own tracked path (a manual copy to a bucket or prefix it doesn't manage). Never transition objects inside the bucket or prefix it actively manages: the same read-availability assumption that rules this out on the data bucket applies here.

Configuration​

On AWS EKS, ClickHouse authenticates to S3 using IAM Roles for Service Accounts (IRSA): there is no access key to create, store, or rotate. This requires:

  1. An IAM OIDC identity provider registered for your EKS cluster (most EKS clusters already have one for other workloads); see AWS's IAM roles for service accounts documentation if you need to set one up.
  2. A dedicated Kubernetes ServiceAccount for ClickHouse, created by the chart, annotated with an IAM role ARN.
  3. An IAM role that trusts your cluster's OIDC provider, scoped to that ServiceAccount.
  4. A permission policy granting only the S3 access ClickHouse and its backup sidecar need.

Chart configuration: create a dedicated ServiceAccount for ClickHouse and annotate it with the role ARN:

clickhouse-server:
  serviceAccount:
    annotations:
      eks.amazonaws.com/role-arn: 'arn:aws:iam::<account-id>:role/<role-name>'

Trust policy: scoped to the ServiceAccount above (replace <account-id>, <oidc-provider-url>, and <namespace>):

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Federated": "arn:aws:iam::<account-id>:oidc-provider/<oidc-provider-url>"
      },
      "Action": "sts:AssumeRoleWithWebIdentity",
      "Condition": {
        "StringEquals": {
          "<oidc-provider-url>:sub": "system:serviceaccount:<namespace>:clickhouse",
          "<oidc-provider-url>:aud": "sts.amazonaws.com"
        }
      }
    }
  ]
}

Permission policy: scoped to the bucket dedicated to ClickHouse:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": ["s3:ListBucket", "s3:GetBucketLocation", "s3:ListBucketMultipartUploads"],
      "Resource": [
        "arn:aws:s3:::<bucket-name>",
        "arn:aws:s3:::<backup-bucket-name>"
      ]
    },
    {
      "Effect": "Allow",
      "Action": [
        "s3:GetObject",
        "s3:PutObject",
        "s3:DeleteObject",
        "s3:AbortMultipartUpload",
        "s3:ListMultipartUploadParts"
      ],
      "Resource": [
        "arn:aws:s3:::<bucket-name>/*",
        "arn:aws:s3:::<backup-bucket-name>/*"
      ]
    }
  ]
}
tip

Scope the policy to ClickHouse's own bucket and, since IRSA gives a single role to the whole pod, the separate backup bucket too (see the tip above): the policy grants read/write/delete on both, so neither should be shared with unrelated data. <bucket-name> must match the one you set in the disk <endpoint> below; <backup-bucket-name> is the one you'll set in the backup sidecar's own destination config (see Backup and restore).

With the ServiceAccount in place, the disk configuration uses the environment-provided credentials: no extraEnvVars needed, since the ServiceAccount's IRSA token is automatically mounted into every container in the pod, including the backup sidecar. You only need to declare the raw disk itself, named object_disk; the filesystem cache in front of it, and the storage policy that puts new tables on it, are already configured by the chart:

clickhouse-server:
  configdFiles:
    00-object-storage.xml: |
      <clickhouse>
        <storage_configuration>
          <disks>
            <default>
              <!-- 1 GiB, refuse writes below this threshold so the local metadata disk never fills to 100% -->
              <keep_free_space_bytes>1073741824</keep_free_space_bytes>
            </default>
            <object_disk>
              <type>s3</type>
              <endpoint>https://<bucket-name>.s3.<region>.amazonaws.com/<namespace>/</endpoint>
              <use_environment_credentials>true</use_environment_credentials>
            </object_disk>
          </disks>
        </storage_configuration>
      </clickhouse>
Name the file so it sorts before object-cache.xml

ClickHouse loads config.d files in alphabetical order, and the chart's own cache configuration (object-cache.xml) expects the object_disk disk above to already exist by the time it's parsed. Name your file 00-object-storage.xml, as shown above; anything that sorts before object-cache.xml works, but this is the convention used throughout this page.

Target bucket: set by the <bucket-name> portion of the <endpoint> URL above: this is ClickHouse's own data bucket, independent from the bucket the backup sidecar writes to.

Credentials​

AWS S3 and Azure authenticate credential-less by default (IRSA and Microsoft Entra Workload ID respectively, see the tabs above), so there's no long-lived secret to store in your cluster for either. GCS and MinIO are the exceptions: ClickHouse's own disk always needs a static access-key/secret-key pair for both, regardless of cluster identity setup: GCS because ClickHouse's S3 client can't consume a federated token (see the GCS tab), MinIO because it has no cloud identity provider to federate with in the first place. That static credential is passed the same way as other sensitive GitGuardian configuration, see Helm sensitive information management.

Last updated on

Something we didn’t cover?

See our Roadmap
Subscribe on GitHub
Submit a request
API status