sns-network-solutions/infra/sns-networking/ids-and-logging.md

IDS & Centralized Logging — SNS Networking

Entity: SNS Networking · Status: Buildable now

What it is

Intrusion detection and centralized log aggregation so you know when something goes wrong before a client tells you. Lightweight — no SIEM until the scale demands it.

Stack

Component	Software	FOSS	Role
Network IDS	Suricata	Yes	Packet inspection on Proxmox bridge
Host IDS	AIDE	Yes	File integrity monitoring (detect unauthorized changes)
Log aggregation	Loki (Grafana)	Yes	Centralized logs, queryable
Log shipper	Promtail / Alloy	Yes	Ships logs from all hosts to Loki
Alerting	Grafana Alerting	Yes	Fires on log patterns + IDS events
Rootkit scan	rkhunter	Yes	Weekly cron, email on detection

Architecture

┌────────────┐  ┌────────────┐  ┌────────────┐
│  Host A    │  │  Host B    │  │  Host C    │
│  promtail  │  │  promtail  │  │  promtail  │
└─────┬──────┘  └─────┬──────┘  └─────┬──────┘
      │                │                │
      └────────────────┼────────────────┘
                       ▼
              ┌─────────────────┐
              │  Loki (VM)      │
              │  + Grafana      │
              └─────────────────┘
                       │
                       ▼
              Grafana Alerting → Email / Ntfy / Slack

Suricata runs on the Proxmox host, tapping vmbr1 (service bridge) in AF_PACKET mode. Alerts feed into Loki via promtail watching Suricata's eve.json.

What Gets Logged

Source	What	Retention
SSH	All auth attempts (success + fail)	90 days
UFW	All denied packets	30 days
Fail2ban	All bans/unbans	90 days
Caddy	Access logs (all requests)	30 days
Suricata	Alerts (eve.json)	90 days
auditd	Sudo, sensitive file access	90 days
Systemd journal	Service starts/stops/crashes	14 days

Build Steps

1. Loki + Grafana: Deploy as Docker Compose on a monitoring VM.

   services:
     loki:
       image: grafana/loki:latest
       volumes: [./loki-data:/loki]
       command: -config.file=/etc/loki/local-config.yaml
     grafana:
       image: grafana/grafana:latest
       volumes: [./grafana-data:/var/lib/grafana]
   

2. Promtail on each host: Install via apt or binary. Config:

   clients:
     - url: http://loki.internal.sns:3100/loki/api/v1/push
   scrape_configs:
     - job_name: syslog
       static_configs:
         - targets: [localhost]
           labels:
             job: syslog
             host: ${HOSTNAME}
       journal:
         labels:
           job: journal
   

3. Suricata on Proxmox host:

   apt install suricata
   suricata-update  # download rulesets
   # /etc/suricata/suricata.yaml: af-packet interface = vmbr1
   systemctl enable --now suricata
   

4. AIDE on critical hosts:

   apt install aide
   aide --init
   mv /var/lib/aide/aide.db.new /var/lib/aide/aide.db
   # Cron: daily `aide --check` → email diff
   

5. Grafana alerts: Create alert rules for:

   • Fail2ban ban count > 10/hour
   • Suricata alert severity >= 2
   • SSH root login attempt (should never happen)
   • AIDE file change on critical paths

Security Posture

• Detection, not prevention (at this layer). Prevention is UFW + Fail2ban. Suricata alerts tell you something got past the first layer.
• File integrity (AIDE) catches rootkits, unauthorized config changes, backdoors.
• Log immutability: Loki is write-once from the host perspective. Compromising a host doesn't let you erase your tracks from Loki (different VM, different creds).
• Retention is cost-bounded. 90 days is enough for incident response without filling disks.

Upgrade Path

• Wazuh: Full HIDS + SIEM + compliance dashboards when client contracts require formal security monitoring (SOC2, etc.).
• CrowdSec: Collaborative IDS — share blocklists with the community. Drop-in alongside Suricata.
• Elasticsearch/OpenSearch: If Loki's label-based querying isn't sufficient for complex correlation. YAGNI until you have a dedicated security analyst.