Ops
Day-2 operations, what happens when a runner crashes, how to stop something gone wrong, and where secrets actually live. None of this needs SSH access to a box; it's all plane behavior or an Admin/API action.
What happens when a runner crashes mid-job
Every dispatched job has a live heartbeat, the runner pings the plane roughly every 2 seconds while it works. If those heartbeats stop (the process crashed, the pod got OOM-killed, the node disappeared), the plane notices within about 6 seconds and reclaims the job automatically: it goes back on the queue for a different runner to pick up, and a "generation" counter on that job increments.
The generation counter is what makes this safe instead of dangerous. If the original runner wasn't actually dead, just slow to report, or its heartbeat got lost on a flaky network, and it finishes the job late and tries to report a result, the plane sees that its generation number is now stale and rejects the write. This is called fencing, and it's what stops a "zombie" runner from overwriting the result a second, legitimate runner already produced. You don't do anything to enable this, it's always on.
One thing to watch: a job that keeps crash-looping (bad graph code, a dependency
that's always down) would otherwise reclaim forever. There's a ceiling. reclaim.max_retries in langgraph.json (default 3, or set
RUNKITE_RECLAIM_MAX_RETRIES), after which the plane stops retrying
and marks the run as failed instead of endlessly recycling it. Set it to
0 for unlimited retries if you'd rather it keep trying forever (not
recommended for anything but a very short-lived debugging session).
Stopping a run that's gone wrong
For one specific run: cancel it from Admin → Runs or via the Agent Protocol API. The runner sees its assignment marked non-current and should stop producing side effects on its next check, well-behaved agent code checks the cancellation signal between steps, so a very long single LLM call in flight may still finish that one call before noticing.
For something bigger, an entire agent or tenant misbehaving, use Admin → Kill switches instead of cancelling runs one at a time: it stops new runs from starting and can cancel every in-flight run in scope in one action. See Kill & break-glass for the full walkthrough, including the opposite tool, a time-boxed policy bypass for when you need something to work again right now while you fix the actual root cause.
Where secrets actually live
Provider credentials (a Salesforce OAuth secret, a GitHub token) live in your connector config or a secrets manager, never copied into every agent's process environment. When an agent needs to call out, the plane mints a short-lived, run-bound session at the moment it's needed and hands back only that. See Secrets for the three ways to reference a credential (env var, mounted file, or HashiCorp Vault) and Connectors & HITL for the full setup.
Reclaim and hooks that run on a schedule (so you know what to expect in logs)
| Loop | Cadence | What it does |
|---|---|---|
| Reclaim sweep | every 2s | Finds jobs whose heartbeat went stale (~6s) and re-queues them |
| Cron scheduler | configurable (default check every ~15s) | Fires due scheduled agents, exactly once even with multiple replicas |
| Policy overlay poll | every 15s | Picks up Admin-created grants / mandatory-HITL rules on replicas that didn't create them |
| Queue depth sample | every 5s | Refreshes the human-facing queue-depth gauge (metrics only, not used for decisions) |
None of these need manual triggering, they're background loops inside the control plane process. They're listed here so a "why did this take up to 15 seconds to show up" question has an obvious answer instead of feeling like a bug.
Reference: docs/ops-runbook.md · Fencing note · Poison-pill note · Admin UI guide