Aller au contenu principal

ClickHouse backup and restore

How ClickHouse backup works​

A backup sidecar runs alongside ClickHouse, continuously watching for changes and taking scheduled full and incremental backups automatically: there is no separate cron job to configure.

Backups are:

  • Application-consistent: taken via ClickHouse's native FREEZE mechanism (zero-copy hardlinks), not a raw volume snapshot, so a backup never captures a table mid-write.
  • Written to your object storage: both the metadata/manifests and the underlying data blobs are copied to your configured bucket, under the path prefix you configure below.
  • Deduplicated and chain-aware: incremental backups only store what changed; retention policies remove whole backup chains rather than leaving orphaned increments.
Never change permissions on the local backup path

FREEZE works by creating hard links to the live table data, not copies. Permissions, ownership, and attributes on a hard link are a property of the underlying data itself: they're shared across every hard link pointing to it. Changing permissions on a file under the backup path changes them on the live data ClickHouse is actively reading and writing too, which can lead to data corruption. Never manually touch permissions there; let ClickHouse and the backup sidecar manage that path exclusively.

Backup sidecar configuration​

A backup sidecar is configured by default on the ClickHouse pod, via clickhouse-server.sidecars: the chart's mechanism for adding an extra container alongside the volumes ClickHouse itself uses. It runs clickhouse-backup in server --watch mode, which self-schedules full and incremental backups in-process; there is no separate cron job, and nothing else to deploy.

Object storage destination​

The backup destination has no default: you must configure it before backups can run. This splits into two parts, both provider-specific: non-sensitive settings (bucket/container, region, path) go under clickhouse.backupConfig (see Tunable settings below for the rest of that map), while credentials (where a static key is required at all) are passed via a separately-created Kubernetes secret, never inline in your values file (the same pattern described in Prerequisites). The target bucket is set independently from ClickHouse's own data bucket (see Object storage backends): use a genuinely separate bucket, not just a different prefix in the same one (see the tip in that section for why). Pick your provider below.

clickhouse:
  backupConfig:
    REMOTE_STORAGE: 's3'
    S3_BUCKET: '<bucket-name>'
    S3_REGION: '<region>'
    S3_PATH: 'backups/metadata'
    S3_OBJECT_DISK_PATH: 'backups/objects'

Target bucket: S3_BUCKET. No credential entry needed: with IRSA (see Object storage backends), the ServiceAccount's token is automatically mounted into every container in the pod, including this sidecar.

Bucket lifecycle policies​

Lifecycle rules and bucket protections for both buckets are documented together in Bucket lifecycle policies: see the Backup bucket subsection for this one (versioning, Object Lock, replication, archive-tier caveats) and the ClickHouse data bucket subsection for ClickHouse's own data bucket. The two need different, in places opposite, rules.

Tunable settings​

Beyond the object storage destination above, a handful of other sidecar settings are exposed under the same clickhouse.backupConfig map, with sensible defaults out of the box: override only the ones you need to change, without redeclaring the whole sidecar spec:

clickhouse:
  backupConfig:
    BACKUPS_TO_KEEP_REMOTE: '30' # only overriding retention here
SettingPurposeDefault
WATCH_SCHEDULESCron-driven full/incremental backup schedule, see Understanding the backup schedule below.name=ch-bkp,full=0 2 * * 0,increment=0 2 * * *,full_type=rebase,delete_previous_cycle=false
BACKUPS_TO_KEEP_REMOTENumber of individual backups to keep in object storage: every full and every increment counts separately, see Understanding the backup schedule below.7
REBASE_BEFORE_REMOVE_OLD_REMOTEWithout this, BACKUPS_TO_KEEP_REMOTE isn't a hard cap: with incremental chains, clickhouse-backup won't delete a chain that a newer increment still depends on, so the retained count can grow past the limit. This rebases the oldest increment still inside the retention window onto a standalone full backup, making it independent of the older chain before it, which can then be deleted, keeping retention strict.true
UPLOAD_CONCURRENCYTables uploaded in parallel during a backup, and (same value) files per table uploaded in parallel.2
DOWNLOAD_CONCURRENCYSame as UPLOAD_CONCURRENCY, for restore.2
S3_CONCURRENCYParallel chunks for a single file's S3 multipart transfer, a finer-grained level than UPLOAD_CONCURRENCY/DOWNLOAD_CONCURRENCY.2
REBASE_CONCURRENCYTables processed in parallel during a rebase operation (see full_type=rebase below).2
LOG_LEVELSidecar log verbosity.info
ALLOW_EMPTY_BACKUPSWithout this, a fresh install with no tables yet is treated as a fatal error, crash-looping the sidecar.true

The concurrency settings default to a conservative, fixed value rather than clickhouse-backup's own auto-detection, which sizes off the node's total CPU count rather than this container's actual CPU limit: raise them if you've given the sidecar more CPU and want faster backups and restores.

A few other settings are already configured by the chart with sensible defaults and aren't exposed for override; see clickhouse-backup's own configuration reference for the exhaustive list of available settings.

Understanding the backup schedule​

WATCH_SCHEDULES drives the backup cadence with cron expressions. One schedule has the form:

name=<name>,full=<cron>[,increment=<cron>][,full_type=create|rebase][,delete_previous_cycle=true|false]
KeyRequiredMeaning
nameYesPrefix for this chain's backup names. Lets you run more than one independent schedule side by side, separated with ;.
fullYesCron expression for the full backup.
incrementNoCron expression for the incremental backup. Omit for full-only backups.
full_typeNo (create)create re-uploads all data for every full backup. rebase performs a server-side copy of the previous chain instead (no re-upload, much faster), but only once a previous backup already exists; the very first cycle always falls back to create.
delete_previous_cycleNo (false)true deletes every older backup of this chain as soon as a new full succeeds, keeping only the current cycle. false leaves older cycles in place, pruned only by BACKUPS_TO_KEEP_REMOTE above.

Cron expressions accept the standard 5-field syntax (minute, hour, day, month, weekday), an optional leading seconds field, and @every/@daily-style descriptors.

The chart's default:

clickhouse:
  backupConfig:
    WATCH_SCHEDULES: 'name=ch-bkp,full=0 2 * * 0,increment=0 2 * * *,full_type=rebase,delete_previous_cycle=false'

This runs a full backup every Sunday at 2am (via rebase once a previous one exists, so no re-upload) and an incremental backup every day at 2am. On Sunday, the full and increment cron expressions fire on the same tick: the full backup always takes priority, so no separate increment runs that day.

Retention window: with delete_previous_cycle: false, BACKUPS_TO_KEEP_REMOTE keeps the N most recent backups, counting each full and each increment as its own entry. At 7 entries a week (1 full + 6 increments), BACKUPS_TO_KEEP_REMOTE: '7' keeps roughly 1 week of history.

Backup freshness​

Backup freshness is surfaced inside the product using ClickHouse's own system tables. Treat a backup as unhealthy if no valid full backup exists within your recovery point objective, or if no new backup has been taken recently.

Container health does not reflect backup health

The sidecar's watch loop retries a failed backup on the next scheduled tick instead of stopping: a persistent failure (bad credentials, an unreachable bucket, a full disk) logs an error on every attempt but never crashes the container. Pod status and restart count stay green throughout. See ClickHouse monitoring and alerting to set up an external signal that does catch this.

Restore​

Restoring ClickHouse data is a manual, deliberate operation: it is not triggered automatically. clickhouse-backup restores in place, on the same ClickHouse instance the sidecar runs alongside: it needs direct filesystem access to attach the backup's data parts, the same requirement that keeps it deployed as a sidecar in the first place (see How ClickHouse backup works above). There's no restore-into-a-separate-empty-instance workflow without standing up a whole second ClickHouse deployment.

High level procedure:

  1. Snapshot the current state first: clickhouse-backup create_remote on the affected table(s), even if that data is already degraded. Gives you a fallback distinct from your regular backup chain if the restore itself goes wrong.
  2. Identify the backup to restore from: list remote backups, check it isn't flagged as broken, and that its backup chain is intact.
  3. Check version compatibility before committing to a full restore: clickhouse-backup download --schema <backup_name> pulls just the schema and metadata (lightweight, no table data), so you can inspect clickhouse_version in the resulting metadata.json against the currently running ClickHouse version.
  4. Stop writes to the affected table(s): pause or scale down the producers writing to them.
  5. Restore into a different name first (--restore-database-mapping/--restore-table-mapping), never straight into the live table.
  6. Verify the restored copy (row counts, spot-check known values) before touching production.
  7. Cut over: once the mapped copy checks out, restore again directly onto the live table name with --rm (restore/restore_remote --rm <backup_name>): clickhouse-backup drops the existing schema and reattaches from the backup.

To just test that a backup works (not to respond to a real incident), restore into a throwaway table or database name instead, so a drill never touches production data.

Known pitfall: repeated restore attempts

If you retry a restore using the same backup name after a failed or interrupted attempt, the restore tool may resume from a stale local state and only restore the schema (DDL), silently skipping the actual data, with no error. If a restore looks empty or incomplete, clear the tool's local state before retrying, or use a distinct target each time.

Disaster recovery scope​

A ClickHouse-only restore is not a complete disaster recovery for every feature

Some features that use ClickHouse also join that data with information stored in PostgreSQL (for example, an inventory of which machines belong to your organization). Restoring ClickHouse alone, into an environment whose PostgreSQL differs, produces inconsistent results in the product: the ClickHouse data and the PostgreSQL registry disagree with each other.

A complete disaster recovery for these features requires restoring both ClickHouse and PostgreSQL to a consistent point in time. Plan your PostgreSQL backup strategy (see Backup & Restore) and your ClickHouse backup strategy together, and test a combined restore as part of your disaster-recovery drills.

Recovery objectives​

We do not currently publish a fixed Recovery Point Objective (RPO) or Recovery Time Objective (RTO) for ClickHouse: these depend on your backup cadence and data volume. We recommend running a restore drill against your own data volume to measure your actual RTO, and setting your backup cadence (frequency of full and incremental backups) to match the data-loss window you're willing to accept.

Il manque quelque chose ?

Voir notre feuille de route
S'abonner sur GitHub
Soumettre une demande
Statut de l’API