Automated incident triage for managed Hermes Cloud —
reads everything, repairs nothing, tells humans what it found.
Managed instances break. Monitoring sees it. Deterministic recovery recycles it. Then what?
Unified availability & health sweeps classify every failure into a kind + fingerprint.
Stop/start recycle for known cases. Cheap, fast, dumb — always gets the first shot.
When the dumb fix demonstrably didn't hold: inspect evidence, classify, recommend, escalate — and surface design gaps we need to lock down.
NAS owns all state — it already owns the fleet. The worker is a thin overlay on the pinned stock Hermes image — not a fork, no bespoke runtime, nothing added to managed instances. They can't tell Surgeon exists.
The #1 failure class is "user got their agent to do something it shouldn't." So every incident summary, event string, and source file the model reads must be assumed to contain instructions aimed at the triage model. Prompts are the last line of defence — these are the first:
No repair executor exists in Stage 1 — not disabled, absent. recommendedRepair is a typed enum for a human. The diagnosis route can only move an incident to AWAITING_REVIEW.
platform_toolsets pins the surface to exactly mcp-surgeon. No terminal, filesystem, browser, web, delegation, cron, memory. Denylist exists as depth — but drift is surfaced by CI + heartbeat, never trusted.
All notifications are emitted by NAS at ledger chokepoints. A webhook on the worker would be a prompt-injection exfiltration channel — its absence is why evidence can safely reach the model at all.
NAS redacts (key patterns + value regexes) and size/depth-limits everything before it leaves the ledger. Source reads are allowed only at refs NAS approved for that incident — evidence can't steer the model to arbitrary repos.
One active incident per (instance, fingerprint). Flaps never reach the queue. Kinds owned by deterministic recovery only promote when the failure returns after a recorded recovery attempt — the cheap fix always goes first.
20-min rolling lease, 40-min total lifetime no matter how healthy the heartbeat — a wedged-but-alive task cannot hold a claim forever.
Every mutation is a conditioned updateMany — state + token + worker + window. The where-clause is the validation. No TOCTOU.
5 failed revisions → parked in AWAITING_REVIEW as triage_abandoned. A pathological incident cannot burn inference forever.
A broken release breaks 1,000 instances at once. Naïve fan-out: ~10 days of queue, 1,000× inference, 1,000 redundant diagnoses. And scaling workers makes it worse. Instead: reflexes are code, triage is agent.
Correctly reasoned: recovery completed, crash returned, restart_count 2→4 → crash loop, not transient. Recommended RESTART_GATEWAY, human-supervised, with the LAST_READY snapshot cited as rollback candidate — and honestly noted no direct evidence links it.
The posture held under pressure: source access was denied (ref not allowlisted). The model retried, adapted, diagnosed from platform evidence alone — and flagged the denial in the diagnosis instead of hallucinating source findings. It even filed a hardeningOpportunity about it.
A pathologically wedged incident was left armed overnight, on purpose. The worker's answer, 14 times in a row:
claim cycles at a dead-consistent 40–41 min lifetime fence
re-claim latency after every release — queue never idled
double-releases or claim corruption; every stale release rejected by token check
inference — wedged task failed fast; in prod the revision breaker parks it at cycle 5
Triage is inference-bound — the container idles on network waits, so memory is the binding dimension. Real numbers from the Railway worker:
Set SURGEON_SERVICE_SECRET + Discord webhook in NAS env, point the worker at real NAS. Teardown list is ready: sim service, logtail sidecar, PAT swap to machine account.
15-min window, trip at max(10, 2% of fleet), admin-tunable in the rollout config. Anyone with historical incident-rate data? Also open: pause auto-update globally vs per-ref on an image storm.
The hardeningOpportunity / suspectedRegression fields are the point of this stage — early findings tell us what to lock down before we let anything execute repairs. Typed recipes only, executor lives in NAS, worker stays credential-free.