# Backup Strategy — SNS Support

**Entity:** SNS Support · **Status:** Buildable now

## What it is

3-2-1 backup strategy using FOSS tools. Three copies, two media types, one offsite.
Every VM and critical data set has a defined backup path and tested restore procedure.

## Stack

| Component | Software | FOSS | Role |
|-----------|----------|------|------|
| VM/container backup | Proxmox Backup Server (PBS) | Yes | Incremental, deduplicated VM backups |
| File-level backup | restic | Yes | Encrypted, deduplicated file backups |
| Offsite target | AWS S3 (Glacier) | No | Encrypted offsite copies |
| Local target | PBS datastore (ZFS) | Yes | Fast restores, primary backup |
| Scheduling | PBS built-in + cron | Yes | Nightly automated runs |
| Verification | PBS verify + restic check | Yes | Catch corruption early |

## 3-2-1 Implementation

```
Copy 1: Live data (Proxmox VM disks, app databases)
Copy 2: PBS on-prem (separate ZFS pool or separate physical host)
Copy 3: S3 Glacier (encrypted, offsite)
```

| Data type | Copy 1 | Copy 2 (local) | Copy 3 (offsite) |
|-----------|--------|----------------|-------------------|
| VMs + LXC | Proxmox host | PBS (nightly snapshot) | PBS → S3 sync (weekly) |
| Databases (MariaDB, SQLite) | Running instance | mysqldump → restic → local | restic → S3 |
| Git repos (Gitea) | Gitea VM | PBS snapshot + Gitea dump | restic → S3 |
| Config/secrets | Vaultwarden | PBS snapshot | restic → S3 (encrypted at rest) |
| Static sites | Caddy VM | PBS snapshot | Git repo is the backup |

## Retention Policy

| Location | Daily | Weekly | Monthly | Max age |
|----------|-------|--------|---------|---------|
| PBS (local) | 7 | 4 | 3 | ~4 months |
| S3 Glacier | — | 4 | 6 | ~7 months |
| restic (local) | 7 | 4 | 6 | ~7 months |

## Build Steps

### PBS (primary — already referenced in sns-systems)

1. Dedicated VM or physical host with its own ZFS pool.
2. Add as storage in Proxmox: Datacenter → Storage → Proxmox Backup Server.
3. Create backup jobs: nightly at 02:00, all VMs/CTs in the pool.
4. Enable verification: weekly verify job to catch bit rot.
5. Encryption: PBS supports client-side encryption — enable for offsite copies.

### restic (file-level + offsite)

```bash
# Initialize S3 repo (one-time)
export AWS_ACCESS_KEY_ID=<backup-writer-key>
export AWS_SECRET_ACCESS_KEY=<secret>
restic -r s3:s3.amazonaws.com/sns-backups-<account>/restic init

# Nightly backup (cron)
restic -r s3:... backup /srv/critical-data \
  --exclude-caches \
  --tag nightly

# Prune old snapshots
restic -r s3:... forget \
  --keep-daily 7 --keep-weekly 4 --keep-monthly 6 \
  --prune
```

### Database dumps (pre-backup hook)

```bash
#!/bin/bash
# /etc/cron.d/db-backup (runs before restic)
mysqldump --all-databases | gzip > /srv/backups/mysql/all-$(date +%F).sql.gz
# Restic picks up /srv/backups/ in its nightly run
```

## Restore Testing

**Monthly:** Restore one random VM from PBS to a temporary ID. Boot it, verify
services start. Destroy the test restore.

**Quarterly:** Restore from S3/restic to a clean VM. Verify data integrity end-to-end.

Document restore test results in `projects/` or a simple log file.

## Exclusions

When running cluster-wide backups to PBS, **skip guests with raw disk passthrough
(physical drives mounted directly into the VM)**. These are storage/archive disks
that are too large for PBS snapshots and are not meaningful to back up as VM state.

| VMID | Name | Passthrough disks | Reason to skip |
|------|------|-------------------|----------------|
| 104 | portainer | `/dev/sdb` (976G) + `/dev/sdc` (488G) | USB external drives on mtr; back up their *contents* separately if needed, not via vzdump |

**Rule:** If `vzdump` would include a raw `/dev/sdX` passthrough >100G, skip that
guest from automated PBS runs. Back up only the OS disk (if needed) by temporarily
detaching the passthrough, or use restic/rsync on the mounted filesystem instead.

## Security Posture

- **Encryption at rest:** PBS supports encryption. restic encrypts by default (AES-256).
  S3 bucket has SSE-S3 enabled as a second layer.
- **Backup credentials isolated:** `backup-writer` IAM role can only PutObject to the
  backup bucket. Cannot list, read, or delete — prevents ransomware from wiping backups.
- **S3 Object Lock (future):** Enable when client data is stored — makes backups
  immutable for the retention period.
- **PBS access:** Only Proxmox host can reach PBS. No other host has write access.
- **restic repo password:** Stored in Vaultwarden, not in cron scripts (use a
  password file with 600 permissions owned by root).

## Upgrade Path

- **S3 Object Lock + Compliance mode:** When you need immutable backups for client
  contracts or compliance.
- **PBS replication:** Second PBS at a different site for faster disaster recovery
  (instead of restoring from S3).
- **Velero:** If/when Kubernetes enters the picture, for persistent volume backups.

<!-- ponytail: No immutable backups yet. Ceiling: a compromised root on the Proxmox
     host could delete PBS datastore. Upgrade: S3 Object Lock + separate PBS with
     different credentials. -->