Quiet VPS Watchdog

Status: In Production (private) · Source available on request

Problem

A personal server that runs n8n, an AI agent gateway, a static site, a feed reader and a handful of Coolify-managed containers needs watching. The usual failure mode of agent-driven monitoring is not missing incidents — it is noise: an LLM asked “is everything fine?” every few minutes will eventually say something, and an alert that repeats every cycle stops being read.

The goal was the opposite: a watchdog that produces nothing when the server is healthy, exactly one message when a problem appears, silence while that same problem persists, and exactly one message when it clears.

What it does

A single dependency-free Python script runs every five minutes from the agent’s scheduler. It checks:

• Disk and inode usage on the root and Docker filesystems, against warning and critical thresholds
• Memory pressure, measured from MemAvailable rather than free memory
• Required systemd units (system and user scope) and any unit that has entered the failed state
• Expected Docker containers by name pattern — present, running, and not unhealthy or stuck in health: starting — plus any other container that is dead or restarting
• HTTP readiness of the public and Tailscale-internal endpoints, with per-target allowed status codes
• TLS certificate expiry for every public hostname
• Age of the newest n8n database backup

Every finding becomes an Issue(severity, key, detail). The list is sorted, serialised and hashed; that hash is the run’s signature.

The quiet part

The script’s own output is the whole notification policy:

issues and signature changed   → print the issue list      (one alert)
issues and signature unchanged → print nothing             (no repeat)
no issues, previous run had    → print one recovery line   (one all-clear)
no issues, previous run clean  → print nothing             (silence)

The scheduler’s instruction to the agent is a single sentence: deliver stdout verbatim; empty stdout means healthy and must remain silent. The agent never decides whether something is wrong — it is a delivery channel for a deterministic result.

State is one small JSON file (signature, had_issues, issue_count, checked_at) written atomically through a temp file and os.replace, so a crash mid-write cannot corrupt the memory of the previous run. A --mode report flag prints a full human-readable readiness summary on demand, for when someone actually wants to see the numbers.

Design decisions

• Zero LLM calls in the check path. Every judgment is a threshold, a status string or a certificate date. The cost of a healthy cycle is a few subprocess calls, and the result is the same every time for the same inputs.
• Deduplicate on content, not on time. A new disk warning next to an existing container failure changes the signature, so it is reported; the same two problems ten minutes later do not.
• No autonomous remediation. The watchdog restarts nothing and touches no service. A container that is down stays down until a person decides what to do — the operator, not the script, owns the blast radius.
• Fail loudly on its own errors. If the config is missing a required section or a check raises unexpectedly, the script exits non-zero with the error on stderr instead of reporting “healthy” by omission.
• Config is data. Targets, thresholds, container patterns and backup globs live in a JSON file next to the script; adding a service is a config change, not a code change.

Part of a wider self-hosted agent setup

The watchdog is one piece of running an AI agent (Hermes) against real infrastructure with narrow permissions:

• Read-only n8n access for the agent. The agent reaches the self-hosted n8n instance (2.36.7) through a local stdio MCP bridge — a third-party project, hermes-n8n-mcp, not my code. The bridge exposes eleven tools; the agent’s configuration allows eight, all read-only (health, list_workflows, get_workflow, find_workflows, list_executions, get_execution, recent_failures, export_workflow). The mutation tools — activate_workflow, deactivate_workflow — and raw container logs are excluded at the client. The n8n API key lives in a separate environment file, never in the agent’s configuration.
• No public ports for the bridge. Stdio only; n8n is reached on the loopback interface.

The same principle runs through both: the agent gets a deterministic, bounded view of production, and anything that changes production stays with a human.

Known limitations

• Single host. The script has no notion of a fleet; it was written for one VPS and reads local /proc, systemctl and docker.
• Thresholds are static. There is no anomaly detection, trend, or rate-of-change alerting.
• The alert channel is whatever the agent’s scheduler delivers to; the script itself has no notifier.
• Source is private because the configuration names internal hosts and containers. A sanitised copy can be shared on request.