A Kubernetes cluster stores two distinct categories of critical data:
- Cluster state ā everything defined in your manifests: Deployments, Services, Secrets, Ingress rules, ConfigMaps, PVCs. Stored in SQLite or etcd.
- Application data ā your PostgreSQL database files, uploaded media, logs. Stored in PersistentVolumes on the node disk.
If you lose the cluster state, you lose the configuration of your entire system ā you'd need to manually recreate every resource. If you lose the application data, you lose your actual business data ā customer records, orders, uploads.
Both must be backed up separately and regularly.
100%
Rendering interactive visual diagram...
Understanding K3s Data Storage
SQLite (Default ā Single Node)
Project Structure
# The SQLite database file
/var/lib/rancher/k3s/server/db/state.db
# Full K3s data directory structure
/var/lib/rancher/k3s/
āāā server/
ā āāā db/
ā ā āāā state.db
ā SQLite: all cluster stateā āāā tls/
ā CA + server certificatesā ā āāā server-ca.crt
ā ā āāā server-ca.key
ā CRITICAL: losing this = losing your clusterā ā āāā ...
ā āāā node-token
ā Join token for new agentsā āāā manifests/
ā HelmCharts auto-applied at startupāāā storage/
ā local-path PVC data āāā pvc-xxx-yyy/
ā Each PVC gets its own directory āāā pgdata/
ā Your PostgreSQL database filesEmbedded etcd (Multi-Node / HA)
When you install with --cluster-init, K3s uses embedded etcd instead of SQLite:
bash
/var/lib/rancher/k3s/server/db/etcd/ ā etcd WAL and snapshot files
Backup Strategy 1: SQLite (Default Setup)
Manual Backup
bash
# IMPORTANT: Use sqlite3's .backup command for a consistent snapshot
# (simple cp may capture a file mid-write and produce a corrupt backup)
# Install sqlite3
sudo apt-get install -y sqlite3
# Take a live, consistent backup (no downtime required)
sudo sqlite3 /var/lib/rancher/k3s/server/db/state.db \
".backup /backup/k3s-state-$(date +%Y%m%d-%H%M%S).db"
# Verify the backup is valid
sqlite3 /backup/k3s-state-*.db "PRAGMA integrity_check;"
# ok
Automated Daily Backup Script
bash
# Create the backup script
sudo tee /usr/local/bin/k3s-backup.sh << 'SCRIPT'
#!/bin/bash
set -euo pipefail
BACKUP_DIR="/backup/k3s"
DATE=$(date +%Y%m%d-%H%M%S)
LOG_FILE="/var/log/k3s-backup.log"
log() { echo "[$(date -Iseconds)] $*" | tee -a "$LOG_FILE"; }
log "Starting K3s backup..."
mkdir -p "$BACKUP_DIR/state" "$BACKUP_DIR/pvdata" "$BACKUP_DIR/tls"
# 1. SQLite cluster state (live, consistent snapshot)
sqlite3 /var/lib/rancher/k3s/server/db/state.db \
".backup $BACKUP_DIR/state/state-$DATE.db"
log "SQLite backup complete: state-$DATE.db"
# 2. TLS certificates and CA keys
tar czf "$BACKUP_DIR/tls/tls-$DATE.tar.gz" \
-C /var/lib/rancher/k3s/server tls/
log "TLS backup complete: tls-$DATE.tar.gz"
# 3. PVC data (local-path storage)
# Stop writes first if possible (or use pg_dump for PostgreSQL)
tar czf "$BACKUP_DIR/pvdata/pvdata-$DATE.tar.gz" \
-C /var/lib/rancher/k3s storage/ \
--warning=no-file-changed # Ignore files changing during backup
log "PVC data backup complete: pvdata-$DATE.tar.gz"
# 4. Export all Kubernetes resources as YAML (useful for human inspection)
k3s kubectl get all,ingress,configmap,secret,pvc,clusterissuer \
-A -o yaml > "$BACKUP_DIR/state/resources-$DATE.yaml" 2>/dev/null || true
log "Resource YAML export complete"
# 5. Delete backups older than 7 days (keep disk usage bounded)
find "$BACKUP_DIR" -name "*.db" -mtime +7 -delete
find "$BACKUP_DIR" -name "*.tar.gz" -mtime +7 -delete
find "$BACKUP_DIR" -name "*.yaml" -mtime +7 -delete
log "Old backups pruned"
# 6. Sync to off-site storage (S3, R2, B2, etc.)
if command -v rclone &>/dev/null; then
rclone sync "$BACKUP_DIR" remote:your-bucket/k3s-backups/ \
--transfers=4 \
--retries=3 \
>> "$LOG_FILE" 2>&1
log "Rclone sync to remote complete"
fi
log "Backup completed successfully"
SCRIPT
sudo chmod +x /usr/local/bin/k3s-backup.sh
# Test it manually first
sudo /usr/local/bin/k3s-backup.sh
ls -lh /backup/k3s/state/
Schedule with Cron
bash
# Daily at 2:00 AM
(sudo crontab -l 2>/dev/null; echo "0 2 * * * /usr/local/bin/k3s-backup.sh >> /var/log/k3s-backup.log 2>&1") | sudo crontab -
# Verify cron is scheduled
sudo crontab -l
Backup Strategy 2: Embedded etcd (HA Setup)
K3s has built-in etcd snapshot commands:
bash
# Take an on-demand snapshot
sudo k3s etcd-snapshot save \
--name "pre-upgrade-$(date +%Y%m%d)" \
--snapshot-compress # Compress the snapshot
# List all snapshots
sudo k3s etcd-snapshot list
# NAME NODE CREATED SIZE
# pre-upgrade-20240115 server-1 2024-01-15 10:30:00 +0000 UTC 2.4 MiB
# on-demand-default-20240114 server-1 2024-01-14 02:00:00 +0000 UTC 2.3 MiB
# Snapshots are stored at:
ls /var/lib/rancher/k3s/server/db/snapshots/
Automated etcd Snapshots (Config File)
yaml
# /etc/rancher/k3s/config.yaml (add to existing config)
# Automatic snapshots every 6 hours
etcd-snapshot-schedule-cron: "0 */6 * * *"
etcd-snapshot-retention: 10 # Keep 10 snapshots
etcd-snapshot-dir: /backup/etcd-snapshots
etcd-snapshot-compress: true
bash
sudo systemctl restart k3s
Sync etcd Snapshots Off-Site
bash
# /usr/local/bin/sync-etcd-snapshots.sh
#!/bin/bash
rclone sync /backup/etcd-snapshots remote:your-bucket/etcd-snapshots/
PostgreSQL Application Data: Database Dumps
For PostgreSQL running in K3s, the safest backup method is logical dumps via pg_dump ā they are consistent regardless of whether writes are in progress:
bash
# Option A: Run pg_dump inside the pod
kubectl -n myapp exec -it postgres-0 -- \
pg_dump -U appuser -d myapp --format=custom \
> /backup/db/myapp-$(date +%Y%m%d-%H%M%S).dump
# Verify the dump
kubectl -n myapp exec -it postgres-0 -- \
pg_restore -U appuser -d myapp --list /dev/stdin < /backup/db/myapp-*.dump | head -20
# Option B: Kubernetes CronJob for automated dumps
yaml
# postgres-backup-cronjob.yaml
apiVersion: batch/v1
kind: CronJob
metadata:
name: postgres-backup
namespace: myapp
spec:
schedule: "0 1 * * *" # Daily at 1 AM
concurrencyPolicy: Forbid # Don't run concurrent backups
successfulJobsHistoryLimit: 3
failedJobsHistoryLimit: 1
jobTemplate:
spec:
template:
spec:
restartPolicy: OnFailure
containers:
- name: backup
image: postgres:16-alpine
command:
- /bin/sh
- -c
- |
set -e
TIMESTAMP=$(date +%Y%m%d-%H%M%S)
BACKUP_FILE="/backups/myapp-${TIMESTAMP}.dump"
echo "Starting backup at ${TIMESTAMP}"
pg_dump \
-h postgres.myapp.svc.cluster.local \
-U $POSTGRES_USER \
-d $POSTGRES_DB \
--format=custom \
-f "${BACKUP_FILE}"
echo "Backup size: $(du -sh ${BACKUP_FILE})"
# Delete dumps older than 7 days
find /backups -name "*.dump" -mtime +7 -delete
echo "Backup complete"
env:
- name: POSTGRES_USER
valueFrom:
secretKeyRef:
name: postgres-secret
key: POSTGRES_USER
- name: PGPASSWORD
valueFrom:
secretKeyRef:
name: postgres-secret
key: POSTGRES_PASSWORD
- name: POSTGRES_DB
valueFrom:
secretKeyRef:
name: postgres-secret
key: POSTGRES_DB
volumeMounts:
- name: backup-storage
mountPath: /backups
volumes:
- name: backup-storage
persistentVolumeClaim:
claimName: backup-pvc
Off-Site Storage with rclone
rclone supports 40+ storage providers (S3, Cloudflare R2, Backblaze B2, Google Cloud Storage, etc.):
bash
# Install rclone
curl https://rclone.org/install.sh | sudo bash
# Configure a remote (interactive ā follow the prompts)
rclone config
# Choose: n (new remote)
# Name: remote
# Type: s3 (or r2, b2, gcs, etc.)
# Follow provider-specific prompts
# Test the connection
rclone ls remote:your-bucket
# Sync backups (one-directional: local ā remote)
rclone sync /backup/k3s remote:your-bucket/k3s-backups/ \
--transfers=4 \
--retries=3 \
--verbose
# Set up hourly sync via cron
(sudo crontab -l 2>/dev/null; echo "0 * * * * rclone sync /backup/k3s remote:your-bucket/k3s-backups/ >> /var/log/rclone-sync.log 2>&1") | sudo crontab -
Cloudflare R2 (Free Tier: 10 GB storage, 1M writes/month)
bash
# Cloudflare R2 is fully S3-compatible ā great free option for backups
rclone config
# Type: s3
# Provider: Cloudflare
# access_key_id: YOUR_R2_ACCESS_KEY
# secret_access_key: YOUR_R2_SECRET_KEY
# endpoint: https://ACCOUNT_ID.r2.cloudflarestorage.com
Disaster Recovery: Restoring from Backup
Restore SQLite Cluster State
bash
# On a FRESH server with K3s installed (but not running any apps yet):
# 1. Stop K3s
sudo systemctl stop k3s
# 2. Replace the SQLite database with the backup
sudo cp /backup/k3s/state/state-20240115-020000.db \
/var/lib/rancher/k3s/server/db/state.db
# 3. Restore TLS certificates (CRITICAL ā without these, K3s won't start)
sudo tar xzf /backup/k3s/tls/tls-20240115-020000.tar.gz \
-C /var/lib/rancher/k3s/server/
# 4. Restore PVC data
sudo tar xzf /backup/k3s/pvdata/pvdata-20240115-020000.tar.gz \
-C /var/lib/rancher/k3s/
# 5. Start K3s
sudo systemctl start k3s
# 6. Verify
kubectl get nodes
kubectl get pods -A
Restore etcd Snapshot
bash
# 1. Stop K3s on the server
sudo systemctl stop k3s
# 2. Restore etcd snapshot (this replaces the entire cluster state)
sudo k3s server \
--cluster-reset \
--cluster-reset-restore-path=/backup/etcd-snapshots/pre-upgrade-20240115
# After this command, k3s exits. Start it normally:
sudo systemctl start k3s
# 3. Verify
kubectl get nodes
kubectl get pods -A
Restore PostgreSQL from pg_dump
bash
# Restore a database dump into the running PostgreSQL pod
cat /backup/db/myapp-20240115-020000.dump | \
kubectl -n myapp exec -i postgres-0 -- \
pg_restore -U appuser -d myapp --clean --if-exists
# Or drop and recreate the database
kubectl -n myapp exec -i postgres-0 -- \
psql -U appuser -c "DROP DATABASE myapp; CREATE DATABASE myapp;"
cat /backup/db/myapp-20240115-020000.dump | \
kubectl -n myapp exec -i postgres-0 -- \
pg_restore -U appuser -d myapp
Backup Monitoring and Alerting
Track backup health with a simple Prometheus metric:
bash
# Add to your backup script: write a timestamp file after each successful backup
echo "$(date +%s)" > /var/lib/node_exporter/textfile/k3s_backup_last_success_timestamp.prom
cat > /var/lib/node_exporter/textfile/k3s_backup_last_success_timestamp.prom << EOF
# HELP k3s_backup_last_success_timestamp_seconds Unix timestamp of last successful K3s backup
# TYPE k3s_backup_last_success_timestamp_seconds gauge
k3s_backup_last_success_timestamp_seconds $(date +%s)
EOF
Prometheus alert ā fire if no backup in 26 hours:
yaml
- alert: K3sBackupMissing
expr: time() - k3s_backup_last_success_timestamp_seconds > 93600 # 26 hours
for: 1h
annotations:
summary: "K3s backup has not run in over 26 hours"
description: "Check /var/log/k3s-backup.log on the server for errors"
Backup Checklist
Before you consider your backup strategy production-ready, verify:
- SQLite or etcd snapshot runs daily at minimum, hourly preferred
- TLS certificates (
/var/lib/rancher/k3s/server/tls/) are included in backup - PostgreSQL is backed up via
pg_dump(NOT just filesystem copy of data dir) - Off-site copy exists (S3, R2, Backblaze) ā on-site backup is not sufficient
- Backup logs are monitored ā a silent backup failure is catastrophic
- Restoration was tested ā run a full restore on a test server quarterly
- RTO/RPO defined ā how long can you be down? How much data can you lose?
Test Your Backups ā Untested Backups Are Not Backups
Set up a test VPS (cheapest plan), restore your latest backup, and verify your application starts and data is correct. Do this at least quarterly. Many teams discover their backup is corrupt or incomplete only when they actually need it ā which is the worst possible time.
Summary
A production K3s backup strategy covers two layers:
- Cluster state:
sqlite3 .backup(ork3s etcd-snapshot save) runs automatically via cron, captures the full Kubernetes resource configuration - Application data:
pg_dumpinside a Kubernetes CronJob captures consistent database snapshots - Off-site sync: rclone copies all backups to S3/R2/B2 ā protecting against disk failure and VPS loss
- Monitoring: Prometheus alerts fire if a backup hasn't run in 26+ hours
- Restoration: Documented and tested ā not theoretical
In the final lesson, you will set up Prometheus and Grafana to monitor your K3s cluster with real-time dashboards and alerting.