Cronomicon

Day-2 operations for an Cronomicon runner that is already installed and registered. For the initial install + configure walk-through, see Runner Install Guide; for the credential-model posture and blast-radius record, see runner-security.html.

This guide covers, in order: checking status & health, managing a runner day-to-day (drain / kill / token hygiene), updating the agent binary, re-syncing a runner's config and identity with the server, and removing / retiring a runner.

One concept to hold first — declared vs local config. A runner declares name, os, capabilities, maxConcurrent, inventory, version, and its protocolVersion to the server only at registration (POST /api/v1/runners/register). The poll loop sends nothing but the bearer token — it is a heartbeat and a kill/drain channel, not a config update. So the server's view of a runner is frozen at its last registration. Anything that changes those declared fields — including upgrading the binary (which changes version/protocolVersion) — only reaches the server when the runner re-declares. On a protocol v4+ agent (0.47.5+) this is automatic: the agent sends a digest of its declared config with every poll, so a change propagates on the next poll after a restart — the server spots the drift and requests the re-declare itself. The Runners → Resync button remains as a force-lever (see "Resync"). Local fields (keys, known_hosts, CA, poll cadence, fan-out, local inventory) need only a restart, or take effect on the next run.

1 · Check status & health

On the host (systemd):

systemctl status cronomicon-runner          # running / failed, recent log lines
systemctl is-active cronomicon-runner        # active | inactive | failed
journalctl -u cronomicon-runner -f           # tail the agent log
journalctl -u cronomicon-runner --since '1 hour ago'
/usr/local/bin/cronomicon-runner version     # build version + commit

On the host (container):

docker ps --filter name=cronomicon-runner    # Up / Exited
docker logs -f cronomicon-runner-01
docker exec cronomicon-runner-01 cronomicon-runner version

Healthy logs show registered with server (first start) or resumed runner identity (restart), then steady poll activity on the cadence. A target dial that fails host-key verification surfaces per-host; a server reap shows server returned 404 on poll — re-registering.

On the server (the Runners view): each runner shows a status, its load (active runs / maxConcurrent), declared capabilities, version, last-seen time, and an inventory chip. Status meanings:

A runner that stays offline for CRONOMICON_RUNNER_DEREGISTER_AFTER (server default 14d) is deleted by a secondary sweep. An offline runner that comes back resumes its ID (D4) and flips to online on its next successful poll.

Click any runner row to expand its detail panel. The compact row carries only Test and Deregister; the remaining per-runner actions — ⚙ Settings, Resync, Scan keys, Drain and Copy upgrade command — live in the expanded detail. The gating there follows one rule: Resync, Scan keys and Drain are offered while the runner is reachable — online or degraded, because a degraded runner still heartbeats and still holds work, and that is the state where you most need them. offline still hides them: there is nobody there to ask. There is no protocol-version gate on these buttons: the server's protocol floor tracks the current wire protocol, so every agent that can talk to this server speaks it. One below the floor is refused at registration, at redeclare and on every poll with 426 protocol_too_old, never silently degraded. Copy upgrade command is always offered whatever the version badge says — it is also the reinstall command, and “up to date” compares version strings, so a locally-built agent can read current while speaking an older protocol. The detail panel also surfaces health-at-a-glance (status, live heartbeat age, load, sandbox posture), the full capability list, registration provenance (with a link to the token that created the runner), a read-only managed-settings summary, an agent-version upgrade hint, the runner's currently-running jobs, and its recent job history.

The panel is laid out for readability. It reads top-down as what this runner is, then what you can change: Registration and Managed settings — the two read-only blocks — sit above the four editable controls, so the ⚙ Settings entry point (which lives on the Managed settings header, beside the pending-change dot) is near the top instead of below the fold. Registration facts such as Protocol and Registered are stated once, under Registration. Fields the runner has no value for are collected into a single muted “Not recorded:” line rather than each rendering a labelled —, and background explanation sits behind an ⓘ disclosure on the section it belongs to.

A second, independent reaper acts on workflow-launched runs. The workflow engine's orphan reaper (a startup sweep plus a 10-minute periodic pass) fails the queued/running child runs of an orphaned or 24h-stale workflow with reason orchestrator_lost — including runs assigned to a perfectly healthy runner. Unlike runner_lost, this shows a plain Failed badge (no amber Lost badge): it means the orchestrator lost track of the workflow (e.g. a server restart mid-workflow), not that the runner vanished.

2 · Manage day-to-day

Drain (graceful — do this before maintenance)

Draining lets active runs finish while the runner stops claiming new work — the safe way to take a runner out of rotation.

• From the UI: Runners → Drain (the button shows while the runner is reachable — online or degraded). On its next poll the agent stops claiming, lets active runs finish, and exits once idle. Because the generated unit is Restart=on-failure, a clean drain-exit (code 0) leaves the service stopped — bring it back later with systemctl start cronomicon-runner. Starting it back up is the come-back signal: the restarted agent's first poll re-admits it — same id, same key, online and claiming work in one round-trip. There is no separate "activate" step; if you want a runner to stay out of rotation, leave the service stopped (or Deregister to revoke its key entirely).
• From the host: systemctl stop cronomicon-runner sends SIGTERM, which triggers the same graceful drain. TimeoutStopSec=300 in the unit gives long ansible/terraform plays up to 5 minutes to wind down before a hard kill.

Always drain before maintenance so you don't kill mid-flight runs.

Kill a single run

To stop one run without touching the rest, kill it from the server: the next poll delivers a kill for that specific traceId and the agent cancels just that run's execution/session. Because kill and drain arrive only on a poll, the agent keeps polling while executing — a busy runner is still controllable.

Registration-token hygiene is not a kill-switch

Registration tokens are single-use per install (minted from the Runners view; each dies on its first successful registration and records which runner consumed it). Revoking an unused token only stops the install that would have used it. Runners that already registered keep their long-lived crn_run_* key and keep working until you deregister them (see "Remove"). To revoke a specific runner, use its Deregister button — token hygiene never touches registered runners.

No hot reload

The agent has no SIGHUP/config-reload path — only SIGINT/SIGTERM (drain). Every config change needs a restart (systemctl restart cronomicon-runner); the restart is also all a declared-config change needs — drift detection re-declares it on the next poll (see "Resync").

3 · Update (upgrade the agent)

Keep agents and the server in lock-step across upgrades. The wire protocol is versioned with a server-enforced minimum (runner-security.html §4); a drift becomes a loud, diagnosable failure rather than silent mis-driving.

systemd host:

# 1. Drain first so no run is interrupted.
#    (UI → Drain, or `systemctl stop` which drains on SIGTERM.)
sudo systemctl stop cronomicon-runner

# 2. Build/copy the new binary and install it over the old one.
cd backend && CGO_ENABLED=0 go build -o cronomicon-runner ./cmd/cronomicon-runner
sudo install -m 0755 cronomicon-runner /usr/local/bin/cronomicon-runner

# 3. Start again — it resumes its identity from the identity file.
sudo systemctl start cronomicon-runner
cronomicon-runner version    # confirm the new build

Container: pull the image tagged with the server's new version (or rebuild your own), then recreate the container against the same named volume so identity persists:

docker pull ghcr.io/resetsmith/cronomicon-runner:2.0.2   # -fat for ansible/terraform
docker stop cronomicon-runner-01 && docker rm cronomicon-runner-01
docker run -d --name cronomicon-runner-01 --restart unless-stopped \
  -v cronomicon-runner-data:/var/lib/cronomicon-runner \
  ...same -e flags as install... \
  ghcr.io/resetsmith/cronomicon-runner:2.0.2

The server keeps showing the OLD version/protocol until you re-register. A restart resumes the stored identity and never re-declares — so after an upgrade the runners row still reflects the version, capabilities, and protocolVersion from the last registration. This matters after an upgrade that changes the protocol version. The server's protocol floor tracks the current wire protocol, so every agent that can talk to this server speaks it: one below the floor is refused at registration, at redeclare and on every poll with 426 protocol_too_old, never silently degraded. So an agent whose binary you upgraded across a protocol bump keeps working, while one you did not upgrade stops receiving work at its next poll and says so in its own log — rather than running on with fields it silently drops. The server picks up a new build automatically on the first poll after the restart (drift detection); the Resync button forces it.

4 · Resync (reconcile a runner with the server)

Resync = re-aligning the server's view of a runner with the runner's actual identity and config. It's needed because declared config crosses the wire only at (re-)registration (see the note at the top). Since protocol v4 the agent can re-declare its current local config in place — same id, same API key, same agency membership, no deregistration, active runs unaffected — and since 0.47.5 the server asks for that automatically: the agent sends a config digest with every poll, and a mismatch with the stored declared set triggers the re-register op on that same poll (flap-guarded to at most one op per 5 minutes per runner). Runners → Resync forces the same op on demand. The button authenticates nothing new: the agent re-reads its own config, so a resync can never change what a runner is from the server side. Pick the case that matches:

Case A — the server reaped/deregistered the runner (automatic)

If the server has deleted the runner (manual deregister, or 14-day offline sweep), the agent's next poll returns 404; it logs server returned 404 on poll — re-registering, discards the stale identity, and re-registers automatically with its current config. Nothing to do — provided the registration token in its env is still valid. Its install token is long dead (single-use, consumed at first registration) — mint a fresh crn_reg_* (Runners → Register New Runner), put it in the runner's env, and restart so the re-register can succeed. This is the one flow that still needs a registration token: the reaped row's crn_run_* key was revoked with it. (The Resync button can't help here — the row is gone and the runner's key revoked; this path always needs a registration token.)

Case B — you changed declared config (capabilities / name / os / maxConcurrent / inventory / toolchains)

Checkout-mode capabilities are declared too. The tokens a runner advertises — ansible, checkout (from -allow-checkout), collection:<fqcn> (installed collections), vault (from -vault-password-file), sandboxed (systemd-run available) — plus the detected toolchain versions on the Runners page are all captured at registration. So enabling checkout, provisioning a new collection, adding the vault file, or gaining a systemd sandbox does not reach the server on a plain restart: it needs a resync (or re-registration), same as any declared change below.

A plain restart will not propagate these — the agent resumes its identity and skips registration. To propagate:

• Just restart (preferred): restart the service so the agent is reading the new values (systemctl restart cronomicon-runner). At startup it recomputes its config digest (re-probing toolchains, collections, and the sandbox), the first poll carries it, and the server requests the re-declare automatically — no click needed. The Resync button forces the same thing without waiting a poll (and a newly installed collection is re-detected at re-declare time even without a restart; flag-derived tokens like checkout/vault and the startup sandboxed probe still need the restart first). One caveat: the startup probe is what gets declared — if the agent starts before its toolchain is ready (e.g. ansible on a slow-booting venv), the reduced token set propagates; restart or Resync once the toolchain is healthy (the agent logs the detected set at startup). No registration token involved — the agent authenticates with its existing crn_run_* key.
• Alternative — deregister, let it re-register: edit the config (e.g. /etc/cronomicon-runner/runner.env), then Runners → Deregister the runner. On its next poll the agent 404s and re-registers with the new config. The old row is gone; the runner keeps the same name but gets a new id. (Restart the service after editing config so it's reading the new values when it re-registers.)
• Last resort — delete the identity file: drain, stop the service, delete the identity file (e.g. /var/lib/cronomicon-runner/identity.json), then start. The agent registers fresh with the new config. Note this mints a new id; the old runners row lingers as offline until the reaper deregisters it (5m → 14d). Prefer the deregister path so there's no orphan row.

The deregister and identity-file paths need a valid registration token at re-register time; the Resync button does not.

Case C — you changed keys, known_hosts, or local inventory

These are runner-local and not declared to the server, so no re-registration is involved:

• Key files and known_hosts are read at dial time — editing their contents in place takes effect on the next run, no restart needed. Only changing their paths (the -key-dir / -key-map / -known-hosts config) needs a restart.
• One key file serves bash AND ansible. A key in -key-dir/-key-map named after the host's authKeyEnvVar is bridged to ansible automatically, so you do not provision the same name a second time as an env var. Each run's log shows what it used: cronomicon: auth: key "NAME" → <path>, cronomicon: auth: --private-key NAME → <path>, and cronomicon: trust: known_hosts → <path>. Prove a name resolves with cronomicon-runner doctor --auth NAME [user@host] before a run; the expanded Runner detail also lists a runner's resolvable key names as chips. (Opt out with -no-auth-bridge.) See Runner Security Guide §2, "How a runner authenticates."
• Local inventory (-local-inventory, local mode) is loaded at agent start — edit it, then systemctl restart cronomicon-runner.

Case D — protocol/version drift after an upgrade

This is the "make the server see the new build" follow-up to an Update (§3): the post-upgrade restart is enough — the new binary's first poll carries a changed digest (its version differs) and the server records the new version and protocolVersion automatically; Resync forces it without waiting a poll. Confirm on the Runners view that the version (shown under the runner's name) updated.

4.5 · Managed settings (tune a live runner from its row)

Managed settings are the exception to "declared config crosses the wire only at registration." A third category joins declared and local: a small set of server-managed operational overrides the operator edits on the runner's row — the ⚙ Settings button opens a drawer — that ride to the agent over the poll channel and apply in-memory on the next poll. No SSH, no restart, no env edit.

The managed set is: max jobs (maxConcurrent), the three sandbox caps (MemoryMax/CPUQuota/TasksMax), the checkout policy (allowCheckout + the repo checkoutRepos allowlist), and a subtract-only capability mask (remove run-types from what the runner will claim). Each field is tri-state: set it to override the runner's local value, or leave it blank/inherit to keep the local value. The secrets these depend on — the checkout deploy token and the vault password — are never managed here; they stay in runner-local custody (installed at §3 / runner-install.html), so the server never holds secret bytes.

Each write bumps a version; the agent echoes the version it has applied as its poll ack, so the row shows a pending dot (● on the Settings button) until the change round-trips — typically one poll cycle. The capability mask is enforced server-side at claim (a masked run-type is simply never assigned to the runner) and honored agent-side too. Managed overrides do not touch the config digest, so changing one never triggers a drift re-registration (§4).

One row-level flag is deliberately not part of the managed set: Secret injection. It is a per-runner trust decision, not an operational tune, and it lives in its own control on the runner's detail panel behind configureApp. When it is on, this runner may claim runs whose job or script declares Runner Security Guide, and it receives the resolved secret values in its dispatch manifest; when it is off, such a run is never dispatched to it at all. Grant it only to runners you trust with the secrets in the scopes they serve. Every registered agent speaks the current wire protocol, so an agent that could ignore the manifest's secrets block cannot exist on this server (§4, and runner-security.html §4).

Every registered agent receives managed settings. Delivery is keyed on the settingsVersion param the agent's poll carries, and since the protocol floor tracks the current version there is no agent old enough to be missing it.

4.6 · Trust a target's host key (scan & approve)

You do not need to hand-assemble a known_hosts before a runner's first SSH run. Host-key verification is strict — an unknown or changed target key is refused — but you can trust a key at the moment it's actually known, with an audit trail. This also covers ansible runs: local-toolchain runs verify against the same known_hosts the agent seeds here, so an approved key serves both Go-SSH and ansible (no separate ~/.ssh/known_hosts for ansible).

1. A runner's SSH run to an unverified host fails with a host_key_unverified reason naming the host (the run's status reason). The run is a soft failure — nothing was executed on an unverified host.
2. On the Runners view, press Scan keys on that runner's row and enter the target(s). The next poll delivers a keyscan op; the agent captures each host's presented key from its own vantage and uploads it.
3. The key appears under Pending host-key approvals with its full SHA256 fingerprint. Compare it out-of-band against the host itself (ssh-keygen -lf /etc/ssh/ssh_host_*_key.pub on the host, or the fingerprint its admin publishes), then Approve.
4. The next poll delivers a trust-hosts op; the agent appends the exact line to its known_hosts (idempotently — a re-delivery never duplicates). Retry the run; it now verifies.

Reject a key you can't verify — it is never trusted. Approving without the out-of-band comparison is still trust-on-first-use (the scan only shows what the host presented, not that it wasn't already compromised) — see Runner Security Guide §8. A pre-seeded known_hosts (--known-hosts at install) still works and skips the scan for hosts you already trust.

4.7 · Tags — label a runner, then send work to it

A runner's tags are free-text labels you set on its row (expand the runner, edit the tag chips). They help you find runners in a long list, and they also decide where work runs: a job — or a single ad-hoc run — can be pinned to a tag, and only runners carrying that tag may claim it.

This is the answer to a question agencies could not express. An agency says which department a runner belongs to; every runner inside it is interchangeable. A tag says where the runner sits — which network segment it can reach, which datacentre it lives in, which VLAN its route table knows about. When a job must reach hosts only some of your runners can see, tag those runners and pin the job.

Tag the property, not the machine. Pins name a tag, never a specific runner, and that is deliberate: a runner's identity is minted at registration, so re-enrolling a host gives it a new one and any pin naming the old identity would quietly stop matching. A tag survives re-enrollment, and tagging three runners vlan-dmz gives the pin three ways to succeed instead of one. Name tags after the capability or location an operator would ask for — vlan-dmz, dc-east, pci-zone — not after the host.

Case does not matter. vlan-dmz, VLAN-DMZ and Vlan-Dmz are the same tag when a pin is matched, and a single runner cannot carry two spellings of one tag. Tags still display exactly as you typed them.

Editing tags takes effect on the next poll. Adding a tag lets that runner start claiming pinned work immediately; removing one stops it. Runs already in flight are unaffected — a claim is a decision made once, not re-checked mid-run. Deregistering a runner takes its tags with it.

Tags are not capabilities, and neither replaces the other. A capability (vault, checkout, a run-type) describes what a runner can do, is declared by the agent itself from its own environment, and is matched by a job's requires. A tag describes where a runner is, is set by an operator in Cronomicon, and is matched by a pin. A run must satisfy both, plus its agency. If you find yourself wanting to grant a “capability” from the UI, you want a tag.

Pinning never widens reach. A pin can only narrow which runners may claim a run; it is applied on top of department isolation, never instead of it. Pinning a run to a tag that only another department's runner carries does not send it there — the run waits, and says so (§5.5). There is no combination of pin and tag that reaches a runner the department rules would deny.

Who sets the pin, and where, is covered in the manuals: the per-run and per-job controls in the User Manual, and the Git-declared form in the Administrator Manual.

5 · Remove / retire a runner

Retire one runner (keep the host)

# 1. Drain (UI → Drain, or stop, which drains on SIGTERM).
sudo systemctl stop cronomicon-runner
# 2. Deregister on the server: Runners → Deregister.
# 3. Delete the identity so a stray restart can't resume the (now-gone) row.
sudo rm -f /var/lib/cronomicon-runner/identity.json
# 4. (optional) keep it from coming back on boot:
sudo systemctl disable cronomicon-runner

Order matters: drain → deregister → delete identity. Deleting the identity without deregistering leaves an offline row to be reaped; deregistering without deleting the identity lets a restart re-register the runner right back.

Fully uninstall from the host (reverse of runner-install.sh)

sudo systemctl disable --now cronomicon-runner
sudo rm -f /etc/systemd/system/cronomicon-runner.service
sudo systemctl daemon-reload
sudo rm -f /usr/local/bin/cronomicon-runner
sudo rm -rf /etc/cronomicon-runner          # config (runner.env)
sudo rm -rf /var/lib/cronomicon-runner      # state: identity + keys + known_hosts
sudo userdel cronomicon-runner              # remove the service account
sudo groupdel cronomicon-runner             # if it has no other members

Then Deregister the runner on the Runners view if you haven't already.

Container

# Drain first (UI → Drain), then:
docker stop cronomicon-runner-01            # SIGTERM → graceful drain (up to the stop timeout)
docker rm cronomicon-runner-01
docker volume rm cronomicon-runner-data     # destroys identity + keys — only when fully retiring

Retire a compromised runner (segment-local rotation)

Because custody is distributed (model b), retiring a compromised runner is a segment-local rotation, not an estate-wide one (runner-security.html §7):

1. Deregister it on the Runners view (revokes its server access).
2. Delete its identity file (the crn_run_* bearer) so the host can no longer act as that runner.
3. Rotate the SSH keys it held — only the keys for that runner's segment are exposed; other runners' keys and the control plane's secrets are not.

5.5 · Online, but nothing runs on it

Registration and eligibility are separate things. A runner can be online, healthy and idle while a department's runs queue beside it — because Cronomicon ships names, not values, and claiming is gated on properties an operator has to set. Treat an agency as a provisioning domain: a department's runners carry that department's key-dir, secrets and sidecar templates, and claim-gating guarantees a run only lands where its names resolve.

You do not have to guess which gate. A run that no online runner can claim records the reason, computed when it is enqueued and stored on the run, so it still answers the question days later — “why didn't it run last night?” cannot be settled by inspecting today's fleet, because the runner that was missing may be online now.

After changing a runner's capabilities or provisioning, Resync (§4) or restart the agent so it re-declares; queued runs are re-evaluated on the next claim sweep and need no re-trigger.

6 · Troubleshooting — installed & running, but not in the UI

A runner appears in the Runners view the instant POST /api/v1/runners/register succeeds — there is no approval gate. So a service that is "active (running)" but never shows up means registration never completed. The agent logs every startup phase before it runs it, so the last line in the journal names exactly where it stopped:

journalctl -u cronomicon-runner -n 40 --no-pager

The healthy startup sequence is:

runner agent starting
probing tier-2 sandbox availability (systemd-run)     # or: sandbox probe skipped (NoSandbox set)
detecting host capabilities (toolchain probes)
registering with server
registered with server id=…
runner agent online

Whichever … line is last with nothing after it is the phase that stalled or failed. Match it:

Reachability check from the host

curl -sS -o /dev/null -w 'healthz %{http_code} %{time_total}s\n' https://SERVER/healthz
curl -sS -m 20 -X POST https://SERVER/api/v1/runners/register \
  -H 'Authorization: Bearer crn_reg_dummy' -H 'Content-Type: application/json' -d '{}' -i | head
# Expect a FAST JSON 401 (unauthorized). An HTML/login page ⇒ SSO proxy intercept;
# a hang/timeout ⇒ the network or proxy is black-holing the register endpoint.

Silent startup hang (no logs after starting, SIGKILL after the stop timeout)

If the agent logs the runner agent starting banner (and maybe a probing…/detecting… line) and then nothing, and systemd eventually reports stop-sigterm timed out … Killing with status=9/KILL, it is blocked inside a startup probe. The overwhelming cause is exec.LookPath — which stats every $PATH directory — hitting a directory on a hung mount (stale NFS / autofs to an unreachable server), where the stat() blocks uninterruptibly. Confirm the exact frame (Go dumps every goroutine's stack on SIGQUIT):

systemctl kill --kill-who=main --signal=SIGQUIT cronomicon-runner
journalctl -u cronomicon-runner --no-pager | grep -A20 'goroutine 1 \['
# A frame in probeSandbox / detectRunTypes / exec.LookPath confirms it.

Find the stuck directory:

PID=$(systemctl show -p MainPID --value cronomicon-runner)
tr '\0' '\n' < /proc/$PID/environ | grep '^PATH=' | cut -d= -f2 | tr ':' '\n' \
  | while read d; do printf '%-24s ' "$d"; timeout 3 stat "$d/." >/dev/null 2>&1; echo "rc=$? (124=HUNG)"; done
timeout 5 df -h    # if df itself hangs, a stale network mount is the culprit

Mitigations, in order:

1. Repair or unmount the stale mount — this is the real fix. Playbook runs resolve their toolchains through the same PATH, so a hung directory breaks runs too, not just registration. A runner on such a host can register but cannot reliably execute.
2. Pin a clean PATH in runner.env that omits the hung directory (keep wherever which ansible systemd-run actually resolve), then restart. runner-install.sh (recent releases) does this automatically at install time and prints which directories it excluded.
3. CRONOMICON_RUNNER_NO_SANDBOX=true skips the systemd-run probe specifically — useful when systemd-run itself stalls (e.g. a non-login system user with no reachable manager) rather than a broad PATH problem.

The agent bounds every startup probe with a deadline, so a hung mount degrades a probe (logged: PATH lookup timed out — treating tool as absent) instead of wedging startup — the runner registers and runs unsandboxed. The stale mount still needs repair for playbook runs to resolve their tools.

See also: Runner Install Guide (install + configure), runner-security.html (credential-model posture & blast radius), env-matrix.md (server-side vars such as CRONOMICON_RUNNER_OFFLINE_AFTER / CRONOMICON_RUNNER_DEREGISTER_AFTER), cronomicon-runner.service (the hardened unit).