Cronomicon

The credential-model record for the runner execution path, analogous to the SSH-executor key-custody record in execution-update.md EX.9. It states, in concrete terms, what the runner holds, what Cronomicon holds, what a compromise reaches, and how that trades against the in-app SSH executor.

One-line posture (credential model b, runners-update.md D1 — locked). The runner holds its own keys and inventory; Cronomicon never decrypts or transmits private-key bytes. The control plane holds nothing that reaches a target — it only tells a runner what to run and references where.

1 · What flows over the wire (and what never does)

A runner claims a run, then fetches a manifest (GET /api/v1/runs/{traceId}/manifest). The manifest carries (internal/runnerproto, ManifestResponse):

• traceId, jobName, runType, executor
• the resolved command: interp[] + body (interpreter argv + script/command)
• env — the run's env snapshot (plain config values, runs.env_json), not secrets
• targets: name, address, port, user, via (bastion), and authKeyEnvVar — the NAME of the key, never the key
• inventoryMode, scope, timeoutSeconds, sensitiveLogging
• inventory (ansible only) — the byte-exact Ansible inventory file the agent writes to a temp file for ansible-playbook -i, and limit (an --limit expression of host/group NAMES). The inventory is secret-free by invariant: secret-bearing vars (ansible_ssh_pass, ansible_become_pass, …) are rejected at INGEST (internal/inventory.ValidateSecrets, fail-closed) so a decrypted secret value can never reach this field. It is attached only for cronomicon-mode ansible runs; a local-mode runner is never shipped an inventory (the no-leak property — its host list never crosses the wire).
• sshUser / sshKeyRef (ansible runs carrying a connect-as override) — the run's identity override as names only: a username and the derived CRONOMICON_KEY_<label> reference. The agent emits them as ansible connection extra-vars; the key's material travels only in the keys block, and the agent fails the run closed if it did not arrive.
• ansibleOptions (ansible runs that set them) — the operator's per-run --check/--diff/--tags/--skip-tags/verbosity/--become/-e selection. Flags, NAMES and operator-typed literals only. Note the argv exposure: -e name=value pairs become process arguments on the runner host, readable by any local user via ps — a weaker position than the env channel, so extra-vars are documented as a non-channel for secret material.
• checkout (checkout runs only) — the repo clone URL, a full commit SHA (never a ref), and the entry playbook / project root for a runner-side pinned project checkout. It carries no credential: the deploy token is runner-local (-checkout-token-file), and the SHA is verified against the runner's own -checkout-repos allowlist before anything is fetched. See §8 for the full checkout security model.

What never flows server → agent: private-key bytes, KEK-decrypted secret material, or any vault secret value (including inside the inventory field — see above). The manifest's authKeyEnvVar is a reference; the agent resolves it locally (internal/agent/ssh.go loadSigner) to a key it already holds. The (a) model — the server minting/shipping short-lived key material via an Cronomicon CA — is shelved; nothing in this path implements it. This is the load-bearing property: a server-side compromise yields no key that reaches a target, because the server never had one.

First-class SSH key credentials are an in-app-executor concept, not a runner one. Cronomicon' typed, envelope-encrypted SSH keys (Env Vars → SSH Keys; a host's authCredentialId) are held and used by the in-app SSH executor, which dials from the control plane. On the runner path the manifest carries the authKeyEnvVar NAME and the agent resolves it to a key it holds locally — credential material never crosses the wire. So a host executed by a runner must reference its key by name (the picker's key-by-name option), not by credential.

In inventory=local mode the manifest is even thinner — it carries only the scope name, not target topology, so the control plane never learns the host list of an isolated segment (runner-install.html §7).

2 · Key custody is DISTRIBUTED, per-runner

Each runner holds only the keys for its own segment, resolved by name in this order (loadSigner):

1. an explicit -key-map entry (NAME=/path/to/key),
2. an env var of that NAME holding a PEM, or
3. a file named after the NAME in -key-dir.

Blast radius of a runner compromise is bounded to that runner's segment. Compromising runner-isolated-seg-a exposes only the keys that runner holds and the targets it can reach — not every target in the estate, and not the keys held by other runners. This is the central contrast with the in-app SSH executor.

Contrast with the in-app SSH executor (EX.9)

EX.9 consciously accepted the centralized custody + broad outbound reach of the in-app executor as the right trade for the v1 reachable-target workload. The runner path makes the opposite trade — it distributes custody and keeps the control plane key-free — which is exactly why it's the chosen path for ansible/ terraform (T-a, owns its own vault/inventory) and network-isolated targets (T-b, the agent lives where the app can't reach). The cost of distribution is operational: keys and known_hosts must be provisioned and rotated on each runner rather than in one place.

How a runner authenticates, end to end (the three paths)

Cronomicon has three execution paths; it helps to see how each proves who it is to a target and which host it trusts. Only the two runner rows are model (b):

One key file serves both runner paths. A key provisioned once — a -key-map entry, or a file in -key-dir named after the authKeyEnvVar — serves bash/Go-SSH runs AND ansible: the agent bridges the NAME to that file's path and hands ansible --private-key <path> when a run's targets share one key, and/or fills the env var an inventory's ansible_ssh_private_key_file: "{{ lookup('env', NAME) }}" reads. You do not provision the same credential twice (a key file for bash + an env-var-holding-a-path for ansible). Each run's log names what authenticated it — cronomicon: auth: key "NAME" → <path> and cronomicon: auth: --private-key NAME → <path>. A run carrying a connect-as key override suppresses this auto-wire entirely (the log says so), because the two wire the same concept through different precedence tiers and only the override's tier reliably wins.

An env-held PEM is Go-SSH-only. A key held only as a PEM in the agent's environment (resolution step 2) can serve Go-SSH runs but cannot be handed to ansible — ansible needs a file path, and a PEM is bytes, so the bridge never materializes it. For a NAME ansible must also use, provision it as a -key-map entry or a -key-dir file (step 1 or 3); the installer's --generate-key NAME writes exactly such a file.

One trust store for both paths. ansible's OpenSSH verifies target host keys against the agent's own known_hosts — the agent injects ANSIBLE_SSH_COMMON_ARGS=-o UserKnownHostsFile=<store> -o StrictHostKeyChecking=yes — so Scan & approve (§5) covers ansible runs too: the same approved host keys serve bash and ansible, with no ~/.ssh/known_hosts symlink workaround needed. A checkout project shipping its own ansible.cfg can still override this (presumed intentional). -no-auth-bridge opts out of both the key bridge and the known_hosts injection (estates that manage their own ansible SSH config); -ansible-ssh-common-args replaces the injected value for ProxyJump / cipher-restricted estates — it must stay fail-closed (never add StrictHostKeyChecking=no).

Prove the chain before a run. cronomicon-runner doctor --auth NAME [user@host] resolves NAME exactly as a run would and, given a target, dials it with the agent's own stack — reporting auth OK / host-key-unknown (points at Scan & approve) / permission-denied distinctly. The full doctor also lists resolvable key names and warns on a passphrase-protected or unreadable key; the expanded Runner detail shows those names as chips, so a missing credential is visible before the first failed run.

Non-root caveats. The runner unit runs with NoNewPrivileges (§7): that blocks privilege escalation on the runner host itself — an ansible become / local sudo executed on the runner is denied — but it does not affect become on a remote target (that is the target's own sudo, reached over SSH, unaffected by the runner's sandbox). Key auth also assumes the target accepts the runner's public key: a GSSAPI / Kerberos (AD) target still needs the runner's key in the login user's authorized_keys, or SSH falls through to a Kerberos method the runner isn't configured for.

Per-run connect-as credentials ride the delivery channel, under the same gates

An operator (or a job definition) may attach a stored SSH credential to a run as its connect-as identity — the target's key selection for that one run. On the runner path this does not create a new material flow: the credential becomes an implicit key binding on the run and rides the existing key-delivery channel — agency membership checked at resolve, delivery audited to change_log, material written to a 0600 file off the run tree and wiped at run end, bytes redacted from ingested logs in lockstep. Each manifest target's authKeyEnvVar is rewritten to the delivered key's derived CRONOMICON_KEY_<label> reference, which the agent resolves to the delivered file ahead of its own key-map/key-dir.

Ansible runs. The same identity applies to ansible run types, over a different carriage but the same material path. Rewriting targets is not enough there: ansible-playbook opens its own connections and the agent never reads targets[].user. So the manifest carries two explicit fields — sshUser and sshKeyRef (the derived CRONOMICON_KEY_<label> reference, names only) — which the agent turns into connection extra-vars (-e ansible_user=…, -e ansible_ssh_private_key_file=<delivered path>). Extra-vars are ansible's highest-precedence source, so the override beats an inventory-authored identity on every host; the lower-precedence -u/--private-key flags would have lost to the inventory on exactly the hosts an admin had wired. Two fail-closed properties: the agent refuses the run if sshKeyRef names material that was not delivered (rather than silently connecting with the inventory's key), and it suppresses the --private-key auto-wire while an override is in force, so one run can never carry two keys through two precedence tiers. Material custody is the same as for the ssh family: the same delivery channel, the same agency check, audit, 0600 off-tree file, wipe-on-cleanup and lockstep redaction. As with the ssh-family case, the override decides who chooses the pairing, not what may be delivered. Terraform carries no identity at any layer.

What changed is who chooses the pairing — the operator at trigger time (gated by Manage Env Vars, the same permission as binding a key reference) or the job author — not what may be delivered: the same material was already reachable via a declared CRONOMICON_KEY_ binding under the same gates. Two enforcement points keep the posture honest: a credential-carrying run is claimable only by an allow_secret_injection runner (the claim fence covers the per-run credential exactly like a declared binding), and a local-inventory runner refuses the run (409) — its users and keys come from its own inventory, so it cannot honor the override, and refusing beats running as the wrong identity.

3 · Run-ownership authorization (no cross-run access)

A valid runner token authenticates a runner, but the manifest and log endpoints additionally authorize by run ownership: a runner may only fetch the manifest for, and post logs to, runs it actually claimed (run.runner_id == caller). Otherwise the response is a deliberate 404 (not 403, so a non-owner can't even confirm the run exists). This matters even under model (b), because a manifest still exposes another run's env snapshot + target topology, and the log endpoint would otherwise allow cross-run log spoofing (any runner posting logs onto any run). Both GET …/manifest and POST …/log are gated; a regression test covers a non-owning runner being refused.

4 · Protocol-version handshake

The agent and server share a wire-protocol version (runnerproto.ProtocolVersion, currently 13; the comment above that constant documents the wire shape). The server enforces a floor, runnerproto.MinProtocolVersion, that tracks the current version: an agent that declares anything older — or declares nothing — is refused at POST /runners/register and at redeclare with 426 protocol_too_old, a loud and actionable error rather than a silent mis-drive. Server and agent ship from one repository and the Copy upgrade command on the Runners page moves a fleet with its server, so keep them in lock-step across upgrades; the handshake turns drift into an explicit, diagnosable failure at the one place it can be checked.

No per-feature back-compat gates. The server carries no family of per-feature agent_too_old refusals behind the floor. Every manifest field and control op (inventory, checkout, secrets, connect-as identity, ansible options, the become-password file, resync, host-key scans, file-watch distribution) would be silently dropped by an agent that did not understand it, and that agent would run the wrong thing; with the floor at the current version that situation cannot arise, because a runner below the floor cannot register, so its row cannot exist. Everything past registration assumes the current wire shape. The security posture of the individual ops is unchanged: the re-register op carries no configuration (the agent re-reads its own local config and re-declares it with its existing crn_run_* key, so a compromised server or operator session cannot use resync to grant a runner capabilities, and no registration token is involved); the drift digest is a hash of what the agent already declares, never a channel for new config, and its delivery is flap-guarded.

5 · Host-key verification expectations (on the agent)

Host-key verification for the target hop is the agent's responsibility and is strict by design (internal/agent/ssh.go hostKeyCallback):

• Point -known-hosts / CRONOMICON_RUNNER_KNOWN_HOSTS at an OpenSSH known_hosts file. The presented target key must be in it. Unset ⇒ the agent uses a default path (beside its identity file) and creates it empty, so a fresh runner starts with an empty trust store — every unknown host is refused until approved (below).
• There is no fall-open. An unknown or changed target key is a hard connect failure, surfaced per-host — there is no InsecureIgnoreHostKey path on the target hop.
• This one store serves ansible too. Local-toolchain (ansible) runs verify against the same known_hosts — the agent points ansible's OpenSSH at it — so Scan & approve below applies to ansible as well as Go-SSH runs (see §2, "How a runner authenticates"). Disable with -no-auth-bridge.

Scan & approve — human-approved TOFU. Instead of assembling a known_hosts by hand, you can trust a target's key at the moment it's actually known: an SSH run that fails host verification surfaces a host_key_unverified reason naming the host; in the Runners view, Scan keys queues a scan (delivered as a keyscan control op) and the agent captures the presented key from its own vantage and uploads it to Pending host-key approvals. You compare the full SHA256 fingerprint out-of-band against the host itself, then Approve — the next poll delivers a trust-hosts op and the agent appends the exact line to its known_hosts. Retry the run; it now verifies. Every scan, approval, and rejection is audited.

Approving without out-of-band verification is still TOFU. The scan tells you what key the host presented to the runner right now — it cannot tell you the host wasn't already compromised or MITM'd. The approval dialog shows the full SHA256 so you can compare it against the fingerprint the host's admin publishes (or ssh-keygen -lf on the host itself). This is strictly better than pasting blind ssh-keyscan output — it's auditable and asks at the right moment — but the out-of-band comparison is what makes it trustworthy. Pre-seeding a curated known_hosts (--known-hosts) remains available and skips the scan for hosts you already trust.

6 · Bastion / ProxyJump trust note

When a target's via is set, the agent makes a ProxyJump-style hop: it dials the bastion, then tunnels an SSH connection to the target through it (internal/agent/ssh.go dial). Trust implications:

• The agent must be able to reach the bastion, and the bastion the target — the app is not in this path at all for runner runs.
• The agent authenticates both hops with its own key (model b) and applies the same strict host-key verification to the bastion as to the target — both must appear in known_hosts. A compromised or spoofed bastion is therefore caught by host-key verification, not silently trusted.
• Because the bastion sits between the agent and the target, it remains a privileged pivot inside the segment: treat the bastion host with the same custody discipline as the runner itself.

7 · Operational hardening (defense in depth)

The credential model is the primary control; these reduce the impact of a host compromise:

• Run as non-root, sandboxed. The cronomicon-runner.service unit runs a dedicated user with NoNewPrivileges, ProtectSystem=strict, a single writable StateDirectory, and a restricted syscall/address-family set. The container images run as a non-root user with no shell (slim) / a single-purpose userland (fat).
• Protect the identity file and keys. The identity file (crn_run_* bearer) is written 0600; keep it and the key directory on storage only the runner user can read. Possession of the identity key lets a host act as that runner (claim its runs, post its logs) — but still cannot retrieve another runner's keys or the control plane's secrets.
• Rotate per-runner. To retire a compromised runner: Deregister it on the Runners view, delete its identity file, and rotate the SSH keys it held. Because custody is distributed, this is a segment-local rotation, not an estate-wide one.
• Least-capability runners. Declare only the capabilities a runner needs. Capability routing then guarantees it never claims (and never sees the manifest for) run-types outside its scope.

8 · Ansible checkout: security model, dependencies & Vault

Runners started with -allow-checkout execute playbook projects checked out at a server-pinned commit SHA — a materially larger power than the body-only path, bounded by four controls plus the sandbox. The threat model is single-tenant-trusted: playbook authors are internal operators gated by GitLab MR review, so the realistic threats are honest mistakes and poisoned third-party dependencies, not a hostile author. (If runners ever execute mutually-untrusted teams' code, that model must be revisited before enabling checkout for them.)

The four controls that bound checkout:

• Opt-in + repo allowlist. Checkout requires BOTH a job-side declaration (a project wrapper) AND the runner's -allow-checkout flag; the runner also refuses any repo outside its -checkout-repos allowlist. A compromised server cannot point a runner at an arbitrary repo, and an un-flagged runner is unaffected by the feature entirely.
• SHA verification. The manifest carries a full commit SHA, never a ref. The agent verifies the commit exists before archiving and runs exactly that tree, so a repo compromise cannot swap the tree under a pinned run. The SHA is pinned at enqueue (a mid-run re-sync can't retarget an in-flight run) and logged as provenance.
• Read-only, per-runner deploy credential. The runner fetches with its own read_repository-scoped GitLab deploy token (-checkout-token-file) — never shipped by the server, never written into the persisted mirror config. Worst-case leak is read access to one repo from one runner, not GitLab write or account access; revocation is per-runner. runner-install.sh places this file for you — --checkout-token-file <path> installs a staged file, or --checkout-token - reads it from a hidden stdin prompt — always at 0640 root:cronomicon-runner. The installer refuses a token passed as a flag value (it would leak via ps / shell history).
• Tree secret-scan. The INI-only inventory guard never sees playbook trees, so a vars/main.yml or .j2 could smuggle a plaintext secret the redactor will never know. cronomicon validate (error in CI) and git sync (advisory warning) scan a project's YAML/INI/template files for inline secret-bearing vars outside vault-encrypted files — matching both key=value and key: value forms. Vault-encrypted ($ANSIBLE_VAULT) files and lookup('env',…) indirection are exempt.

Tier 2 sandbox (host-native). Each local-toolchain run is wrapped in a per-run systemd-run --scope cgroup for resource caps; the filesystem/namespace/syscall hardening (ProtectSystem=strict, PrivateTmp, NoNewPrivileges, the syscall filter) is inherited by the child from this runner's own systemd unit. Sandbox availability is detected, never assumed: where no usable systemd manager exists, runs execute unsandboxed and are reported as such (a loud startup warning and a per-run provenance line for checkout runs). A sandbox-capable runner advertises a sandboxed capability token, so a job may requires: [sandboxed] to never land on an unsandboxed host. Egress control (an IPAddressAllow allowlist on the runner unit) ships commented opt-in in cronomicon-runner.service.

Supply-chain: two controls and one accepted risk.

• Pinning is the supply-chain posture. The galaxy source defaults to the public galaxy.ansible.com (override with -galaxy-server for a private mirror/hub). Because the source can be open, reproducibility rests on pinning — every requirements.yml collection needs an exact version: and every git role a full commit SHA. cronomicon validate fails CI on an unpinned entry; sync warns. A collection already present at the exact pinned version in the runner's provisioned base is not re-downloaded (so a pre-provisioned runner runs offline); an unsatisfied pin whose download fails fails the run loudly — there is deliberately no fallback to a differently-versioned local copy.
• Provenance is logged. Each checkout run emits the verified SHA, the ansible-core version, and every dependency install line — buying back the auditability that run-time installs cost.
• Vault: decrypted values bypass redaction — use no_log. A job opts into Ansible Vault with requires: [vault]; the agent then passes --vault-password-file from its OWN runner-local config (the password never travels — configure it with -vault-password-file (installed by runner-install.sh via --vault-pass-file <path> or a hidden --vault-pass - prompt, at 0640 root:cronomicon-runner; never as a flag value), and register that same password as an Cronomicon stored secret so the log redactor masks it). The value-based redactor can only mask values Cronomicon knows; a value your playbook decrypts from vault at run time is not one of them. Any task that could print a vault-decrypted secret MUST set no_log: true — that is the only masking for decrypted vault values. Prefer env-var indirection (lookup('env','NAME')) over vault where you can: it is the only secret path the redactor fully covers.

9 · Referencing Cronomicon values in scripts & inventories

Cronomicon resolves a small, closed set of reserved CRONOMICON_* prefixes into values at run time. Any CRONOMICON_* name that starts with one of these four prefixes is a reference the app injects; every other CRONOMICON_* name is server/runner configuration and is never exposed to a run. An operator reading a script or inventory can tell the two apart at a glance.

Derived-reference rule. Rows keep their bare names (e.g. a Secrets row NWD_BECOME_PASS, an SSH Keys row ansible_rh8_key); you form the reference by prefixing the section — reference = CRONOMICON_<SECTION>_<row name, verbatim>. Resolution strips the known prefix verbatim (case-preserved) to find the row or file. So CRONOMICON_SECRET_NWD_BECOME_PASS → the revealed value of Secrets row NWD_BECOME_PASS, and CRONOMICON_KEY_ansible_rh8_key → the path to key ansible_rh8_key. Nothing in the store or on the host is renamed.

Value vs. path. CRONOMICON_KEY_* resolves to a filesystem path — pass it where a tool expects a key file (ssh -i "$CRONOMICON_KEY_ansible_rh8_key"). The other three resolve to values — never write a value into a file the run leaves behind.

Secret hygiene in a run. An injected CRONOMICON_SECRET_* / CRONOMICON_VAR_* value is only as safe as the script handling it:

• Quote every expansion — "$CRONOMICON_SECRET_X", never bare $CRONOMICON_SECRET_X — so a value with spaces or globs is not resplit.
• Never echo a secret and never pass it on the command line (--pass=$CRONOMICON_SECRET_X) — argv is world-visible via ps. Prefer stdin or an env var the child reads directly.
• Turn off tracing around secrets — set -x prints every expanded value; disable it (set +x) before touching one, or never enable it in a script that handles secrets.
• Fail fast if a reference was not injected — guard with the POSIX idiom
  : "${CRONOMICON_SECRET_X:?not injected}"
  so the run aborts loudly instead of proceeding with an empty credential.

The value-based log redactor masks values Cronomicon knows, but that is a backstop — the controls above keep a secret from leaking through argv, a trace, or a file in the first place.

10 · Residual risks / notes

• Distributed custody is more surface, less depth. Model (b) trades one high-value custodian (the app, EX.9) for many lower-value ones (the runners). The aggregate key material is larger and lives in more places; the win is that no single compromise is estate-wide. Provisioning/rotation discipline per runner is the operational cost you accept for the bounded blast radius.
• A runner is trusted within its segment. Within its own segment a runner has the reach its keys grant. Scope each runner's keys to the minimum (runner-install.html §8) so "that runner's segment" stays small.
• The control plane is not a key oracle. This is the property to preserve in any future change: nothing should ever make the manifest or poll path carry key bytes or decrypted secrets. If a central-key-custody mandate ever appears, that is the shelved (a) model and a separate, conscious decision — not a quiet edit to this path.