sns-network-solutions/infra/sns-support/backup-strategy.md

Backup Strategy — SNS Support

Entity: SNS Support · Status: Buildable now

What it is

3-2-1 backup strategy using FOSS tools. Three copies, two media types, one offsite. Every VM and critical data set has a defined backup path and tested restore procedure.

Stack

Component	Software	FOSS	Role
VM/container backup	Proxmox Backup Server (PBS)	Yes	Incremental, deduplicated VM backups
File-level backup	restic	Yes	Encrypted, deduplicated file backups
Offsite target	AWS S3 (Glacier)	No	Encrypted offsite copies
Local target	PBS datastore (ZFS)	Yes	Fast restores, primary backup
Scheduling	PBS built-in + cron	Yes	Nightly automated runs
Verification	PBS verify + restic check	Yes	Catch corruption early

3-2-1 Implementation

Copy 1: Live data (Proxmox VM disks, app databases)
Copy 2: PBS on-prem (separate ZFS pool or separate physical host)
Copy 3: S3 Glacier (encrypted, offsite)

Data type	Copy 1	Copy 2 (local)	Copy 3 (offsite)
VMs + LXC	Proxmox host	PBS (nightly snapshot)	PBS → S3 sync (weekly)
Databases (MariaDB, SQLite)	Running instance	mysqldump → restic → local	restic → S3
Git repos (Gitea)	Gitea VM	PBS snapshot + Gitea dump	restic → S3
Config/secrets	Vaultwarden	PBS snapshot	restic → S3 (encrypted at rest)
Static sites	Caddy VM	PBS snapshot	Git repo is the backup

Retention Policy

Location	Daily	Weekly	Monthly	Max age
PBS (local)	7	4	3	~4 months
S3 Glacier	—	4	6	~7 months
restic (local)	7	4	6	~7 months

Build Steps

PBS (primary — already referenced in sns-systems)

1. Dedicated VM or physical host with its own ZFS pool.
2. Add as storage in Proxmox: Datacenter → Storage → Proxmox Backup Server.
3. Create backup jobs: nightly at 02:00, all VMs/CTs in the pool.
4. Enable verification: weekly verify job to catch bit rot.
5. Encryption: PBS supports client-side encryption — enable for offsite copies.

restic (file-level + offsite)

# Initialize S3 repo (one-time)
export AWS_ACCESS_KEY_ID=<backup-writer-key>
export AWS_SECRET_ACCESS_KEY=<secret>
restic -r s3:s3.amazonaws.com/sns-backups-<account>/restic init

# Nightly backup (cron)
restic -r s3:... backup /srv/critical-data \
  --exclude-caches \
  --tag nightly

# Prune old snapshots
restic -r s3:... forget \
  --keep-daily 7 --keep-weekly 4 --keep-monthly 6 \
  --prune

Database dumps (pre-backup hook)

#!/bin/bash
# /etc/cron.d/db-backup (runs before restic)
mysqldump --all-databases | gzip > /srv/backups/mysql/all-$(date +%F).sql.gz
# Restic picks up /srv/backups/ in its nightly run

Restore Testing

Monthly: Restore one random VM from PBS to a temporary ID. Boot it, verify services start. Destroy the test restore.

Quarterly: Restore from S3/restic to a clean VM. Verify data integrity end-to-end.

Document restore test results in projects/ or a simple log file.

Exclusions

When running cluster-wide backups to PBS, skip guests with raw disk passthrough (physical drives mounted directly into the VM). These are storage/archive disks that are too large for PBS snapshots and are not meaningful to back up as VM state.

VMID	Name	Passthrough disks	Reason to skip
104	portainer	/dev/sdb (976G) + /dev/sdc (488G)	USB external drives on mtr; back up their contents separately if needed, not via vzdump

Rule: If vzdump would include a raw /dev/sdX passthrough >100G, skip that guest from automated PBS runs. Back up only the OS disk (if needed) by temporarily detaching the passthrough, or use restic/rsync on the mounted filesystem instead.

Security Posture

• Encryption at rest: PBS supports encryption. restic encrypts by default (AES-256). S3 bucket has SSE-S3 enabled as a second layer.
• Backup credentials isolated: backup-writer IAM role can only PutObject to the backup bucket. Cannot list, read, or delete — prevents ransomware from wiping backups.
• S3 Object Lock (future): Enable when client data is stored — makes backups immutable for the retention period.
• PBS access: Only Proxmox host can reach PBS. No other host has write access.
• restic repo password: Stored in Vaultwarden, not in cron scripts (use a password file with 600 permissions owned by root).

Upgrade Path

• S3 Object Lock + Compliance mode: When you need immutable backups for client contracts or compliance.
• PBS replication: Second PBS at a different site for faster disaster recovery (instead of restoring from S3).
• Velero: If/when Kubernetes enters the picture, for persistent volume backups.