Cronomicon

How to get the most out of Ansible inside Cronomicon: where playbooks run, the two ways to bring them in, how inventories become scopes, how SSH keys and become passwords resolve on a runner, how secrets reach a play safely, and the failure modes you will actually meet. It assumes you know Ansible itself; what it teaches is the Cronomicon-specific shape of an Ansible estate.

Companion reading: the User Manual for the per-screen reference (Jobs §6, Scopes & Env Vars §11), the Administrator Manual for GitOps and the runner fleet, the Bash Guide for shell scripts, and the Runner Install Guide / Runner Config Guide / Runner Security Guide guides for the agent itself.

The one-sentence model. A playbook is a Script (run type ansible), a Scope carries the inventory it targets, a Job binds the two to schedules and env, and the run always executes on a runner that advertises the ansible capability — never inside the Cronomicon server, and never over the in-app SSH executor.

1 · The execution model

Ansible (like Terraform) is a runner-only run type. The in-app SSH executor exists for shell-family scripts; for ansible the executor resolution always lands on runner — a global ssh default can never apply to it, and explicitly choosing SSH in the Run dialog is rejected with 422 invalid_executor (the dialog flips you back to Runner). See Administrator Manual §4.1.

That means you need at least one runner whose host carries the Ansible toolchain:

• Capabilities auto-detect. An agent probes for ansible-playbook at startup and claims the ansible run type if present — install Ansible later and a service restart propagates it via the config-drift resync. In containers, build the runner image with Ansible installed (the “fat” variant in the runner install guide); a minimal image has no Ansible to detect.
• Capability routing is safe by default. A run for a type no online runner advertises stays queued (“waiting for an ansible-capable runner”) rather than failing. The same claim-gating covers requires: tokens (checkout, vault, collection:<fqcn>) and agency isolation — an unsatisfiable run waits and names what it is waiting for; it is never silently dropped.
• What a runner advertises is visible. The Runners view shows each runner's capability chips — including its ansible-core version and detected collections — so “which runner will take this play” is a glance, not a guess.

Sizing note. A runner claims up to maxConcurrent runs; each Ansible run is a full ansible-playbook process with its own forks. Tune fan-out with an ANSIBLE_FORKS variable on the scope (see §4) rather than raising maxConcurrent first — ten concurrent plays of 50 forks each is 500 SSH sessions from one host.

2 · Two ways in: self-contained playbooks vs checkout projects

Cronomicon has two distinct paths for executing a playbook, and choosing the right one up front saves a migration later.

Path A — the default: keep playbooks self-contained

Any raw *.yml under scripts/ in the definitions repo is auto-discovered on sync as a Script with run type ansible (a sub-folder like scripts/ansible/patch.yml just becomes part of the catalog's folder tree). A job references it by name:

# jobs/patch-tuesday.yaml
apiVersion: cronomicon.io/v1
kind: Job
metadata:
  name: patch-tuesday
spec:
  script_ref: patch.yml
  scope: Production
  scheduleRefs: [monthly-patch-window]
  timeout_seconds: 3600

Self-contained plays run everywhere with zero runner configuration, and they are what the in-app Composer and the Scripts catalog handle most naturally. Prefer this path until you genuinely need roles.

Path B — checkout projects for real role trees

A composite project — roles, group vars, templates, a vaulted secrets file — is declared as one Script whose members are not synthesized into standalone catalog entries:

# scripts/deploy-app.yaml
apiVersion: cronomicon.io/v1
kind: Script
metadata:
  name: deploy-app
spec:
  run_type: ansible
  project_root: ansible/deploy-app     # repo-relative directory
  entry: site.yml                      # playbook run from the materialized tree

At enqueue the server pins the exact commit SHA; the runner fetches the repo (allowlist-checked), verifies the SHA, materializes the tree, installs requirements.yml per run (already-present pinned collections are not re-downloaded), and runs the entry playbook from it. cronomicon validate fails CI on an unpinned collection or a non-SHA git role, and a tree secret-scan catches inline plaintext secrets a vars/*.yml or template could smuggle past the inventory guard. Full operator detail: Administrator Manual §7.8 and the Runner Install Guide guide.

⚠ Submodules do not compose with Path B. The sync engine updates git submodules recursively, so a playbooks repo mounted as a submodule under scripts/ is discovered — but only its self-contained playbooks work: checkout materialization uses git archive, which omits submodule content, and a checkout always targets the synced definitions repo. Any playbook that needs roles, includes or vault files must have its project files committed directly in the definitions repo. When migrating an existing Ansible repo, converge it into the definitions repo (or repoint the sync at it) rather than mounting it as a submodule.

3 · Inventory: how scopes carry your hosts

An Ansible run must target a scope that declares the ansible run type and carries an inventory — the server refuses to build a runner manifest without one (409 no_inventory). Two ways to get there:

• Git-synced: files in inventory/*.ini (read flat, not recursively — flatten and namespace nested department trees). A # cronomicon:v1 types=bash,ansible pragma at the top of the file (or a .cronomicon.yaml sidecar) declares the scope's run types.
• In-app: Scopes → + Add Scope and paste the full INI into the Ansible INI Inventory input — validated inline, projected atomically on save. See User Manual §11.1.

A synced inventory does triple duty: the raw file ships byte-exact to the runner for ansible-playbook -i; it is parsed into an advisory group/host-vars projection you can read in the Scopes view (never authoritative for execution); and its hosts are imported into the SSH host registry so the rest of the app knows them. Group structure ([group:children], [group:vars]) is preserved — and is what per-run group targeting works against (§4).

Inline secret values are rejected at ingest — by design. An inventory carrying ansible_ssh_pass, any *_become_pass, or inline-vault markers is rejected fail-closed with a line-numbered inventory_secret_rejected before anything is persisted or shipped. The supported pattern is env-var-NAME indirection — {{ lookup('env','NAME') }} — resolved on the runner (§5–§6). A literal CRONOMICON_*= assignment is likewise rejected: that namespace belongs to the injector; your content may reference those names, never define them.

For network-isolated segments, a runner can hold the inventory itself (--inventory local + --local-inventory): the manifest then carries only the scope name and the host list never touches the control plane. Pair with agencies to pin each scope's runs to the runners inside its segment (§7.7).

4 · Running plays: targeting, inputs and env

Group targeting & --limit

When the effective scope has a parsed inventory, the Run dialog's “Where it runs” fold offers a Groups control — pick inventory groups and the runner receives them as ansible --limit (an SSH shell run over the same scope would expand them to member hosts; both hit the identical set). For selections the picker can't express, the Advanced Ansible --limit field takes a raw pattern like webservers:&staged:!quarantine and overrides the group/host selection. An unknown group is rejected 422 group_membership; the raw field is ansible/runner-only. Details: User Manual §6.5.

Prefer per-run targeting over inventory edits. A “run it just on the web tier this once” should be a Groups selection on the Run dialog, not a temporary inventory fork.

Run inputs replace vars_prompt

Interactive prompts have no terminal to ask on — a play that uses vars_prompt will hang or take defaults. Declare the questions on the job (or seed them on the Script) as run inputs instead; the Run dialog asks the operator, and each answer arrives in the run's environment under its declared name:

# on the Job (spec.prompts) — or on the Script, which seeds the Composer
prompts:
  - name: TARGET_RELEASE
    label: "Release tag to deploy"
    required: true
  - name: DRAIN_FIRST
    label: "Drain nodes first?"
    default: "yes"
    options: ["yes", "no"]

# in the playbook — read the answer from the env
vars:
  release: "{{ lookup('env','TARGET_RELEASE') }}"

By default an unfilled required input warns and records rather than blocking (prompt_enforcement: warn); set prompt_enforcement: block on jobs where running without the answer must be impossible. Note a job with required run inputs is effectively manual — a scheduled fire has nobody to ask.

Environment layering

Ansible reads its own configuration from the environment, which makes Cronomicon's env layering (job env ← schedule env ← per-run override, override wins per key) the natural tuning surface — ANSIBLE_FORKS, ANSIBLE_TIMEOUT, ANSIBLE_STDOUT_CALLBACK and friends all work as plain Variables or job env rows. Bound Variables arrive as CRONOMICON_VAR_<key> (see §6); plain job/schedule/override env arrives under its own name. A job can also declare env_passthrough: — env-var names the runner forwards from its own local environment into the play (names only; the values live on the runner, e.g. in secrets.env).

Give every play a realistic timeout_seconds (the whole run is killed and marked failed on overrun) and use concurrency_policy: Forbid on plays that must not overlap themselves — a slow patch window firing again on the next cron is the classic Ansible pile-up.

5 · SSH auth on the runner: names, not bytes

Cronomicon ships key names, never key material, to a runner by default. The agent resolves each name locally — against its key-map or key-dir (default /var/lib/cronomicon-runner/keys/<NAME>) — so a compromised control plane never yields target keys, and a compromised runner exposes only its own segment. Provision keys at install time (--generate-key <NAME> mints one and prints the public half) or drop files into the key-dir later.

Where the name comes from, in precedence order on an Ansible run:

• An explicit ansible_ssh_private_key_file in the inventory always wins. The supported secret-free form is the lookup pattern:

  web-01 ansible_host=10.1.2.11 ansible_ssh_private_key_file="{{ lookup('env','prod_web_key') }}"

  The runner scans the inventory for lookup('env',NAME) names, and bridges each one to the matching key-dir file in the child environment. The name must be a bare literal — a nested template inside the lookup fails with “env passthrough var(s) not set”.
• Otherwise the auth bridge: when every target of the run resolves to exactly one distinct key name (per-host cronomicon_auth_key_env_var, the scope sidecar's authKeyEnvVar, or the host record in Settings → SSH Targets), the agent auto-appends --private-key <path> and verifies target host keys against the same known_hosts that Scan & approve seeds. One provisioned key serves bash and ansible runs alike; no double provisioning.

Become passwords

Passwordless sudo remains the preferred arrangement. A NOPASSWD rule scoped to what the play actually needs is auditable on the target and keeps the credential out of the run entirely. What follows is the exception path. And never rely on -K: there is no terminal to prompt on.

There are two supported ways in, and they answer different questions.

1 · A job-level become password (preferred). The job names a Secrets row and Cronomicon does the rest:

spec:
  become_password_secret: BECOME_PASSWORD   # a NAME, never a value

The field holds a bare key. The password stays in the Secrets catalogue with every control that implies — KEK or Vault at rest, department resolution, reveal auditing, log redaction — and is resolved at dispatch. The agent writes it to a 0600 file on tmpfs off the run tree, passes the path to ansible-playbook --become-password-file, and wipes it when the run ends. The value never enters the environment and never appears in a template. An environment variable is readable from /proc by anything that can see the process; a wiped file is a materially smaller window.

Two consequences worth knowing. The flag is global to the run, so it cannot vary per host the way an inventory-authored ansible_become_password can — a fleet with heterogeneous become passwords wants pattern 2. And the job automatically requires the become-file capability, so its runs wait for a runner with ansible-core ≥ 2.12 rather than being assigned to one that would silently ignore the flag.

2 · Inventory lookup (per-host variation). The older pattern, still correct when different hosts need different passwords: put ansible_become_password="{{ lookup('env','PROD_BECOME_PASS') }}" in the inventory (an inline value is rejected at ingest) and supply the value either as a reference binding — which is what lets one playbook serve several departments, see §6 — or runner-locally via /etc/cronomicon-runner/secrets.env or the Vault Agent sidecar (§6).

Setting a become password is a grant, and it is gated. Naming a Secret here has the runner write that value to a file, so the Composer requires Manage Env Vars and membership of the department that owns the row — you cannot bind another department's password. Git sync is deliberately advisory instead: a synced job naming a missing Secret produces a warning rather than failing the whole sync, and the run fails closed at dispatch.

⚠ Gotcha — Stored SSH-key credentials don't reach the runner path. A host whose key is attached as a first-class Stored credential (Env Vars → SSH Keys, picked by ID in SSH Targets) authenticates fine on the in-app SSH executor, but the runner manifest carries only the key name field — a credential-by-ID host presents no key name to a runner-executed Ansible run. For inventories that runners execute, reference keys by name (key-dir/key-map), or deliver the bound key via reference-binding injection on a flagged runner (§6).

A target the runner has never seen fails soft with host_key_unverified: run Scan keys from the Runners view, compare the fingerprint out-of-band, and Approve it into the runner's known_hosts — the flow is audited, strict thereafter. See Runner Security Guide.

6 · Secrets in plays: three doors, one rule

The rule: a secret value lives either on the runner, in Cronomicon's encrypted store / Vault, or in an ansible-vault file — never in the repo, the inventory, a schedule env, or a per-run override. The three doors:

• Runner-local env (names-only). The play references {{ lookup('env','NAME') }}; the value sits in the runner's secrets.env — provisioned by you, or continuously rendered by the opt-in Vault Agent sidecar (the runner never holds a Vault token; the value never crosses the control-plane channel). Zero server involvement, works on any agent.
• Reference bindings with per-runner injection. Declare the Secrets / Variables / SSH keys a job consumes as reference bindings (§11.6); they arrive as CRONOMICON_SECRET_<name> / CRONOMICON_VAR_<name> env vars, and a bound key's material as a private /dev/shm file, only on a runner an operator has flagged with the Secret injection toggle. Everything is fail-closed — an unresolvable binding or an un-flagged runner leaves the run queued, never partially supplied — injected values join the run's redaction dictionary, and every dispatch writes a value-free audit row. This is the door that lets Vault-backed secrets reach a play with no runner-side file at all.
• ansible-vault (checkout projects). The job declares requires: [vault]; a runner provisioned with --vault-pass-file supplies --vault-password-file itself (the password never travels, and the token claim-gates the run to vault-capable runners). Register that vault password as an Cronomicon stored secret too, so any accidental echo of it is masked.

One playbook, every department's credential

The common shape in a departmental estate is one shared playbook that each department runs with its own credential. Two mechanisms make that work without a naming convention in the play.

Per-department rows under one name. Two departments may each hold a Secret named BECOME_PASSWORD in the same scope. A run resolves its own department's row — the play just asks for the name it always asked for:

- hosts: all
  become: true
  vars:
    ansible_become_password: "{{ lookup('env','CRONOMICON_SECRET_BECOME_PASSWORD') }}"

This is legal despite the reserved namespace: the CRONOMICON_*= guard matches an assignment, so referencing an injector-owned name inside a lookup is fine. Which row a run gets is decided by the run's scope, not by who triggered it — which is why a central operations team can run a department's job with that department's credential without ever being able to see it.

Aliasing, for rows that cannot share a name. When the catalogue already has TEAMA_SUDO and TEAMB_SUDO and renaming is not on the table, a binding can declare the name its value is injected under. Both alias to BECOME_PASSWORD, and the same one playbook works. The alias is a destination only: it never changes which row resolves, and an out-of-department row aliased to a friendly name is still refused.

⚠ When two departments own the same name in one scope. If a run's scope belongs to both departments, the reference is ambiguous and the run is refused rather than being given somebody's credential at random — the audit trail would otherwise imply an intent that never existed. The refusal says ambiguous, not missing, so nobody goes hunting for a row that is there twice. Authoring surfaces name the owners; the run does not. If your scopes map one-to-one onto departments this can never fire.

Redaction has a deliberate gap you must close with no_log. Cronomicon masks stored-secret values and injected reference values in run logs — but a value a playbook decrypts at run time (ansible-vault) or fetches itself was never registered with the redactor. Put no_log: true on tasks that handle decrypted or fetched secrets, and keep -vvv debugging away from them.

7 · Best-practice checklist

• Stay on Path A until roles force Path B. A self-contained play needs zero runner setup and composes best in-app.
• Pin everything in Path B. Exact collection versions in requirements.yml, SHA-pinned git roles — the validators enforce it, CI (cronomicon validate .) catches it at merge time.
• Names, never values. lookup('env','NAME') for keys; spec.become_password_secret for a become password; reference bindings or the sidecar for delivery; nothing secret in inventory, vars files, schedule env, or run overrides.
• Passwordless sudo first. A NOPASSWD rule scoped to what the play needs beats every password-delivery mechanism here. Use §5's become password when a target's policy forbids that — not as the default.
• Let the scope pick the department's credential. One playbook asking for one name, with each department owning its own row, beats a naming convention per team (§6). Keep scopes mapped one-to-one onto departments and the ambiguity refusal can never fire.
• Replace vars_prompt with run inputs (spec.prompts), and mark truly unskippable ones prompt_enforcement: block.
• Target per run, don't fork inventories. Groups / --limit in the Run dialog; agencies for network segmentation; local-inventory runners for isolated segments.
• Bound every play: timeout_seconds sized to reality, concurrency_policy: Forbid on non-reentrant plays, ANSIBLE_FORKS tuned at the scope.
• no_log: true wherever a task handles a value the redactor has never seen.
• Keep plays idempotent — Cronomicon retries (retries:) and schedules re-fire; a play that cannot safely run twice will eventually run twice.

8 · Troubleshooting

For anything runner-side — sandbox posture, checkout flags, key provisioning, upgrades — continue in the Runner Install Guide and Runner Config Guide guides.