NS-588 · NAS Team · Shadow-UAT complete

🩺 Hermes Surgeon

Automated incident triage for managed Hermes Cloud —
reads everything, repairs nothing, tells humans what it found.

Stage 1 · read-only validated end-to-end 2026-07-17 NAS PR #669 ready for review
The problem

The gap between the dumb fix and a human logging in

Managed instances break. Monitoring sees it. Deterministic recovery recycles it. Then what?

HAVE Monitoring

Unified availability & health sweeps classify every failure into a kind + fingerprint.

HAVE Deterministic recovery

Stop/start recycle for known cases. Cheap, fast, dumb — always gets the first shot.

NEW Triage

When the dumb fix demonstrably didn't hold: inspect evidence, classify, recommend, escalate — and surface design gaps we need to lock down.

Architecture

Two components, one hard trust boundary

NAS owns all state — it already owns the fleet. The worker is a thin overlay on the pinned stock Hermes image — not a fork, no bespoke runtime, nothing added to managed instances. They can't tell Surgeon exists.

NAS — control plane (owns everything stateful) monitor sweep classify + debounce incident ledger Postgres, append-only admin review queue + Discord pushes redaction evidence bounding claim leasing atomic, fenced storm gate pure code, no LLM HMAC-authenticated internal API — /api/internal/surgeon/* ±5min skew · nonce replay-guard · rate limit · 503 until secret configured TRUST BOUNDARY Worker (Railway) pinned stock Hermes image s6 poller loop claim → kanban card → agent restricted profile allowlist: mcp-surgeon ONLY ∅ credentials no comms, no Fly, no DB, no repair Sonnet 5 triage model 2GB · max_spawn 2 target claim / evidence → ← diagnosis
The load-bearing decision

Evidence is adversarial. Every control is structural.

The #1 failure class is "user got their agent to do something it shouldn't." So every incident summary, event string, and source file the model reads must be assumed to contain instructions aimed at the triage model. Prompts are the last line of defence — these are the first:

1 Read-only by construction

No repair executor exists in Stage 1 — not disabled, absent. recommendedRepair is a typed enum for a human. The diagnosis route can only move an incident to AWAITING_REVIEW.

2 Allowlist, not denylist

platform_toolsets pins the surface to exactly mcp-surgeon. No terminal, filesystem, browser, web, delegation, cron, memory. Denylist exists as depth — but drift is surfaced by CI + heartbeat, never trusted.

3 Zero comms credentials on the worker

All notifications are emitted by NAS at ledger chokepoints. A webhook on the worker would be a prompt-injection exfiltration channel — its absence is why evidence can safely reach the model at all.

4 Bounded evidence, gated source

NAS redacts (key patterns + value regexes) and size/depth-limits everything before it leaves the ledger. Source reads are allowed only at refs NAS approved for that incident — evidence can't steer the model to arbitrary repos.

Incident lifecycle

Debounced in, triaged once, reviewed by a human

One active incident per (instance, fingerprint). Flaps never reach the queue. Kinds owned by deterministic recovery only promote when the failure returns after a recorded recovery attempt — the cheap fix always goes first.

OBSERVING READY_FOR_TRIAGE TRIAGING AWAITING_REVIEW RESOLVED

Leases + hard fence

20-min rolling lease, 40-min total lifetime no matter how healthy the heartbeat — a wedged-but-alive task cannot hold a claim forever.

Atomic consumption

Every mutation is a conditioned updateMany — state + token + worker + window. The where-clause is the validation. No TOCTOU.

Circuit breaker

5 failed revisions → parked in AWAITING_REVIEW as triage_abandoned. A pathological incident cannot burn inference forever.

Fleet incidents · implemented + unit-tested

A storm is 1 incident with N victims — not N incidents

A broken release breaks 1,000 instances at once. Naïve fan-out: ~10 days of queue, 1,000× inference, 1,000 redundant diagnoses. And scaling workers makes it worse. Instead: reflexes are code, triage is agent.

gate: pure code in the monitor sweep · trip at max(10, 2% fleet) in 15 min · reflexes pause change, never initiate it
Shadow UAT · 2026-07-17 · real image, real model, simulated NAS

Full cycle: 3 minutes, claim → diagnosis

23:54:19 claimed railway-surgeon-1
23:54:42 evidence_served HMAC + claim token ok
23:57:37 diagnosis_accepted → AWAITING_REVIEW
0.55
CONFIDENCE — HONEST, NOT PERFORMATIVE

GATEWAY_CRASH · HIGH

Correctly reasoned: recovery completed, crash returned, restart_count 2→4 → crash loop, not transient. Recommended RESTART_GATEWAY, human-supervised, with the LAST_READY snapshot cited as rollback candidate — and honestly noted no direct evidence links it.

The posture held under pressure: source access was denied (ref not allowlisted). The model retried, adapted, diagnosed from platform evidence alone — and flagged the denial in the diagnosis instead of hallucinating source findings. It even filed a hardeningOpportunity about it.

Shadow UAT · resilience datapoint

15 hours unattended on a wedged incident

A pathologically wedged incident was left armed overnight, on purpose. The worker's answer, 14 times in a row:

14

claim cycles at a dead-consistent 40–41 min lifetime fence

≤60s

re-claim latency after every release — queue never idled

0

double-releases or claim corruption; every stale release rejected by token check

~$0

inference — wedged task failed fast; in prod the revision breaker parks it at cycle 5

Shadow UAT · sizing telemetry (cgroup samples via heartbeat)

Provisioning from measurement, not vibes

Triage is inference-bound — the container idles on network waits, so memory is the binding dimension. Real numbers from the Railway worker:

Decisions wanted today

Three asks

1

Review & merge #669, then cutover

Set SURGEON_SERVICE_SECRET + Discord webhook in NAS env, point the worker at real NAS. Teardown list is ready: sim service, logtail sidecar, PAT swap to machine account.

2

Sanity-check storm thresholds

15-min window, trip at max(10, 2% of fleet), admin-tunable in the rollout config. Anyone with historical incident-rate data? Also open: pause auto-update globally vs per-ref on an image storm.

3

Stage 2 stays gated on Stage 1 findings

The hardeningOpportunity / suspectedRegression fields are the point of this stage — early findings tell us what to lock down before we let anything execute repairs. Typed recipes only, executor lives in NAS, worker stays credential-free.

← → step · F fullscreen