238c40d5e0
- Flagstar: program completed, all 35 awards already distributed - USDA Business Builder: TERMINATED July 15, 2025 (RFBC killed by current admin) - Updated funding-research.html with strikethrough + red badges + updated capital stack - Updated meeting agenda funding table (closures + added EASSI) - Added plane-projects/ scaffolds (Compose YAML, used API instead due to CE compat) - Added supply-checklist.csv, trailer-registry.md, README.html, kiowa-meeting-agenda.html
129 lines
5.0 KiB
Markdown
129 lines
5.0 KiB
Markdown
# Backup Strategy — SNS Support
|
|
|
|
**Entity:** SNS Support · **Status:** Buildable now
|
|
|
|
## What it is
|
|
|
|
3-2-1 backup strategy using FOSS tools. Three copies, two media types, one offsite.
|
|
Every VM and critical data set has a defined backup path and tested restore procedure.
|
|
|
|
## Stack
|
|
|
|
| Component | Software | FOSS | Role |
|
|
|-----------|----------|------|------|
|
|
| VM/container backup | Proxmox Backup Server (PBS) | Yes | Incremental, deduplicated VM backups |
|
|
| File-level backup | restic | Yes | Encrypted, deduplicated file backups |
|
|
| Offsite target | AWS S3 (Glacier) | No | Encrypted offsite copies |
|
|
| Local target | PBS datastore (ZFS) | Yes | Fast restores, primary backup |
|
|
| Scheduling | PBS built-in + cron | Yes | Nightly automated runs |
|
|
| Verification | PBS verify + restic check | Yes | Catch corruption early |
|
|
|
|
## 3-2-1 Implementation
|
|
|
|
```
|
|
Copy 1: Live data (Proxmox VM disks, app databases)
|
|
Copy 2: PBS on-prem (separate ZFS pool or separate physical host)
|
|
Copy 3: S3 Glacier (encrypted, offsite)
|
|
```
|
|
|
|
| Data type | Copy 1 | Copy 2 (local) | Copy 3 (offsite) |
|
|
|-----------|--------|----------------|-------------------|
|
|
| VMs + LXC | Proxmox host | PBS (nightly snapshot) | PBS → S3 sync (weekly) |
|
|
| Databases (MariaDB, SQLite) | Running instance | mysqldump → restic → local | restic → S3 |
|
|
| Git repos (Gitea) | Gitea VM | PBS snapshot + Gitea dump | restic → S3 |
|
|
| Config/secrets | Vaultwarden | PBS snapshot | restic → S3 (encrypted at rest) |
|
|
| Static sites | Caddy VM | PBS snapshot | Git repo is the backup |
|
|
|
|
## Retention Policy
|
|
|
|
| Location | Daily | Weekly | Monthly | Max age |
|
|
|----------|-------|--------|---------|---------|
|
|
| PBS (local) | 7 | 4 | 3 | ~4 months |
|
|
| S3 Glacier | — | 4 | 6 | ~7 months |
|
|
| restic (local) | 7 | 4 | 6 | ~7 months |
|
|
|
|
## Build Steps
|
|
|
|
### PBS (primary — already referenced in sns-systems)
|
|
|
|
1. Dedicated VM or physical host with its own ZFS pool.
|
|
2. Add as storage in Proxmox: Datacenter → Storage → Proxmox Backup Server.
|
|
3. Create backup jobs: nightly at 02:00, all VMs/CTs in the pool.
|
|
4. Enable verification: weekly verify job to catch bit rot.
|
|
5. Encryption: PBS supports client-side encryption — enable for offsite copies.
|
|
|
|
### restic (file-level + offsite)
|
|
|
|
```bash
|
|
# Initialize S3 repo (one-time)
|
|
export AWS_ACCESS_KEY_ID=<backup-writer-key>
|
|
export AWS_SECRET_ACCESS_KEY=<secret>
|
|
restic -r s3:s3.amazonaws.com/sns-backups-<account>/restic init
|
|
|
|
# Nightly backup (cron)
|
|
restic -r s3:... backup /srv/critical-data \
|
|
--exclude-caches \
|
|
--tag nightly
|
|
|
|
# Prune old snapshots
|
|
restic -r s3:... forget \
|
|
--keep-daily 7 --keep-weekly 4 --keep-monthly 6 \
|
|
--prune
|
|
```
|
|
|
|
### Database dumps (pre-backup hook)
|
|
|
|
```bash
|
|
#!/bin/bash
|
|
# /etc/cron.d/db-backup (runs before restic)
|
|
mysqldump --all-databases | gzip > /srv/backups/mysql/all-$(date +%F).sql.gz
|
|
# Restic picks up /srv/backups/ in its nightly run
|
|
```
|
|
|
|
## Restore Testing
|
|
|
|
**Monthly:** Restore one random VM from PBS to a temporary ID. Boot it, verify
|
|
services start. Destroy the test restore.
|
|
|
|
**Quarterly:** Restore from S3/restic to a clean VM. Verify data integrity end-to-end.
|
|
|
|
Document restore test results in `projects/` or a simple log file.
|
|
|
|
## Exclusions
|
|
|
|
When running cluster-wide backups to PBS, **skip guests with raw disk passthrough
|
|
(physical drives mounted directly into the VM)**. These are storage/archive disks
|
|
that are too large for PBS snapshots and are not meaningful to back up as VM state.
|
|
|
|
| VMID | Name | Passthrough disks | Reason to skip |
|
|
|------|------|-------------------|----------------|
|
|
| 104 | portainer | `/dev/sdb` (976G) + `/dev/sdc` (488G) | USB external drives on mtr; back up their *contents* separately if needed, not via vzdump |
|
|
|
|
**Rule:** If `vzdump` would include a raw `/dev/sdX` passthrough >100G, skip that
|
|
guest from automated PBS runs. Back up only the OS disk (if needed) by temporarily
|
|
detaching the passthrough, or use restic/rsync on the mounted filesystem instead.
|
|
|
|
## Security Posture
|
|
|
|
- **Encryption at rest:** PBS supports encryption. restic encrypts by default (AES-256).
|
|
S3 bucket has SSE-S3 enabled as a second layer.
|
|
- **Backup credentials isolated:** `backup-writer` IAM role can only PutObject to the
|
|
backup bucket. Cannot list, read, or delete — prevents ransomware from wiping backups.
|
|
- **S3 Object Lock (future):** Enable when client data is stored — makes backups
|
|
immutable for the retention period.
|
|
- **PBS access:** Only Proxmox host can reach PBS. No other host has write access.
|
|
- **restic repo password:** Stored in Vaultwarden, not in cron scripts (use a
|
|
password file with 600 permissions owned by root).
|
|
|
|
## Upgrade Path
|
|
|
|
- **S3 Object Lock + Compliance mode:** When you need immutable backups for client
|
|
contracts or compliance.
|
|
- **PBS replication:** Second PBS at a different site for faster disaster recovery
|
|
(instead of restoring from S3).
|
|
- **Velero:** If/when Kubernetes enters the picture, for persistent volume backups.
|
|
|
|
<!-- ponytail: No immutable backups yet. Ceiling: a compromised root on the Proxmox
|
|
host could delete PBS datastore. Upgrade: S3 Object Lock + separate PBS with
|
|
different credentials. -->
|