Cronomicon

2 First-run bootstrap

Access grants give access, and only an admin can create the grants. Thus you seed the first admin with the CRONOMICON_BOOTSTRAP_ADMIN_GROUP environment variable. This is the sequence (§8 gives the full procedure):

1. Deploy with CRONOMICON_BOOTSTRAP_ADMIN_GROUP set to one of your AD groups.
2. Log in. The server gives you admin access. The grants table has no effect on this.
3. Go to Settings → Users & Access. Create the real access grants (AD group → role → agency, or All scopes).
4. Remove the environment variable and deploy again. The server writes a clear warning to the log for the full time that the bypass is active.

3 How the UI gates actions

The SPA (the single-page application, which is the Cronomicon console in your browser) calls GET /api/v1/capabilities. The response contains two deployment feature flags: vault and apprise. Vault is true only when a Vault client is really wired, not only configured. The response also contains the caller's permission booleans, which agree with the server gates: compose, manageRoles, configureApp, manageEnvVars, publishSchedule, and, triggerJobs and killJobs. The response also contains unrestricted. That flag is not a permission. It is scope reach: it tells you if the caller has an All scopes grant. A sibling flag, composeUnbound, says whether the caller may author an All-scoped (global) definition — only an unrestricted compose grant qualifies, and the Job Composer reads it to withhold the “All agencies (global)” scope option from everyone else.

Two properties of these flags are easy to get wrong:

• The flags have union semantics. triggerJobs: true means “can trigger somewhere”, never “can trigger here”. Thus the Jobs catalog asks the server about each row and does not trust the union.
• unrestricted tells you how far the caller can go, not what the caller can do.

The flags show or hide the controls:

• compose shows + Create and the per-row Edit on Jobs, and the Workflow Editor. It is a grantable permission, checked per object against the definition's scope. The Schedule Builder, schedule definitions, calendars, reactions, revision restores and the recycle bin are a stricter tier: they belong to the unrestricted administrator only, whatever roles a delegate holds.
• publishSchedule shows + Publish to GitLab.
• manageEnvVars enables the edit controls for Variables and Secrets. It also enables the Reference Bindings editor on the Job and Script detail views. Without it, that editor is read-only.
• configureApp enables the Scopes page and most Settings panels. This includes a runner's per-row ⚙ Settings drawer (managed overrides) and its warning-colored Secret injection control.
• manageRoles gates the Users & Access panel.

Two write surfaces need no role permission at all: operator tags and annotations (the Notes / contact / critical sidecar on jobs and workflows) are writable by any signed-in session (CSRF-protected), including on Git-sourced definitions — they are operator property, not part of the definition. Two row affordances are worth knowing when you audit the UI: cronomicon-source rows carry a History button (revision snapshots, §5), and where your visible catalog holds two definitions of one name, lists qualify the name with its agency (deploy · FIN) — the badge appears only when a collision is already inside your own visibility, so it can never reveal another department's catalog.

The SPA copies the real requirePerm gates. Thus a non-admin sees a polite notice, not a form that returns 403 on save. Git-authored items are read-only in the in-app editors, for all roles. You edit those items in Git. The SPA also handles an expired or missing session correctly. An API 401 sends the browser to the login screen and does not leave dead controls. The startup /me probe is the one exception.

Roles say what you can do. Grants say where. Access grants decide authorization. One grant row says: this AD group has this role here. “Here” is one agency, or All scopes (unrestricted). A permission and a scope must come from the same grant. The server never resolves the two independently and then makes a union, because such a union would let a user's strongest verb apply to their widest reach.

Visibility stays a union, and this is intentional. A restricted operator sees the GLOBAL (scope-less) rows plus its granted scopes, and nothing more. listJobs, listSchedules, listUpcomingSchedules, listWorkflowRuns, and the Env Vars and Secrets lists are scope-filtered. An out-of-scope GET /jobs/{id} or secret reveal returns 404, not the record. Pause and resume return the same detail, so they also return 404. There is no existence oracle, and no script body or plaintext env can leak (SU-2). Every job sub-route that must behave this way — including the file-sightings ledger — goes through one shared read gate, so a binned or out-of-scope job answers 404 uniformly; the documented exceptions are pause/resume (a permission check), reference bindings (a 404/403 split), and tags (a binned definition may still be tagged). Writes go through the applicable ScopeWritable predicate. A restricted actor can write only into its own scopes, never into GLOBAL. The signal comes from data, not from the role, so a scope-restricted admin stays restricted.

Composing follows the same rule, per object. An Cronomicon-composed job must name a scope — the scope is what decides which agencies can see and run it. “All agencies (global)” is an explicit choice (the API takes ""), not a default, and an omitted scope is refused with 422 validation_failed. A Git job with no scope still syncs, into All, with a sync warning. Authoring authority is decided per definition, not by the flat flag: an edit is checked against both the scope the job is leaving and the scope it is entering, and saving a workflow requires authority over every job in its graph. Where two sources hold a job of the same name, the compose-time check resolves each step through the same source-precedence rule the engine fires by, so the job that is checked is the job that runs.

The server enforces the execution verbs in the same way. To trigger, kill, pause, or resume, the caller must have triggerJobs or killJobs on the run's scope. An unscoped job has no authority of its own. A restricted operator must attach a scope that they hold at trigger time. The Run dialog opens on the scope picker when a selection is necessary. Only an unrestricted operator can run the job unbound on the general pool.

You configure this in the Settings → Users & Access → Access Grants card, which manageRoles gates. Each grant is one row: an AD group, a role, and where it applies. “Where” is a single agency, or All scopes (unrestricted). A grant is the only thing that gives access. An identity whose groups match no grant gets no authority at all. “Operator, but only for Tax” is exactly one row.

Access administration can itself be delegated. A manageRoles holder whose grant names an agency administers access for that agency only, under three server rules: they may only administer access for the agencies they administer; only an unrestricted administrator may grant access to every agency (the All scopes option is withheld from a delegate's picker); and they may not grant a permission they do not themselves hold in that agency — the anti-amplification rule. The rules apply to the incoming grant and to the existing row on an edit or delete, so a delegate cannot free-ride on a wider row by modifying it.

The card is a list, not a role × scope grid, and this is intentional. A grid grows one column for each scope and becomes too wide at two dozen departments. The list has search, filters, and sort functions. One filter entry finds the grants that go to everything. Do a check of those rows first.

A grant that names an agency expands through that agency's scopes at login. If you add a scope to the agency next month, the grant includes it with no edit. Each grant edit signs out all other operators. The session epoch (the counter that makes open sessions invalid) increases, so the change gets to signed-in sessions immediately, not at cookie expiry. The Roles card is next to the grants. Roles are data (see below). Access Grants is the only access surface on the page.

Roles are data, not code. In the Roles card, an unrestricted administrator can create, edit, and delete roles. Role definitions are shared by every agency, so a delegated (agency-bound) administrator can grant and revoke the existing roles for their agencies but cannot change the templates; reads stay on manageRoles. The four built-in roles (admin, approver, operator, viewer) are included by default. You can edit them, but you can not delete them. A role is a pure permission template over seven permissions: trigger jobs, kill jobs, manage env vars, publish schedule, configure app, manage roles, and Create jobs & workflows — the authoring permission, which lets a role author jobs and workflows within the agencies it is granted while global (All) definitions, reusable schedules, calendars and the recycle bin stay admin-only. There is no “view” permission, because no enforcement point for one is possible. Thus viewer has no permissions at all. This is correct, not an error: a viewer's access is visibility, which comes from its granted scopes, and visibility is not a verb.

Two floors keep the recovery path safe: admin can never lose manageRoles, and no edit can cause zero roles to have it. The server refuses to delete a role while a grant refers to it. Each role write signs out all other sessions.

4 How Execution Works

This section gives you a model of what occurs when a job runs: how the server selects the executor, the life of an SSH run, how the server resolves hosts and bastions, host-key verification, and how the cron scheduler decides to fire.

4.1 Executor selection

For a manual run, the precedence is: the per-trigger override, then the job's executor, then the global default, then the run-type default (shell → ssh; ansible and terraform → runner). Scheduled and workflow runs do not use the first step. Safety rule: a global ssh default can never apply to ansible or terraform runs. Those always go to runner. Thus the system never puts work in the queue that no executor can claim.

Executor selection decides what kind of thing runs the job. A separate and later decision — runner pinning — decides which runner may claim it, when the executor resolves to runner. It has its own precedence chain and its own two layers of ownership; see §7.10. The two interact in one place: pinning a run whose executor resolves to ssh is rejected, because the in-app executor has no runner for a tag to select.

4.2 The life of an SSH run

1. Claim. The executor takes the oldest queued ssh shell run, sets it to running, and writes a run-start event.
2. Open the log. The executor loads a redactor for the run's scope.
3. Resolve the command. This is an inline command, an inline script, or a scriptPath from the Git clone. The path gets a sandbox check: no .. and no absolute paths.
4. Resolve the targets. An explicit single host wins over the scope. Without one, the run goes to all member hosts of the scope (§4.3).
5. Apply the timeout. The job's timeout_seconds limits the full run. On overrun, the server kills the run and sets it to failed with cronomicon: job timed out.
6. Fan out. The run occurs on each host, with a maximum of 4 hosts in parallel. Each host's output streams line by line. When there is more than one host, each line has the prefix [hostname].
7. Aggregate. All hosts OK → success. All hosts failed → failure. A mix → warning (partial).

Every terminal run-status write is status-guarded. The stop route, the runner's normal finalize, the drain-timeout path and the orphan reaper each write only while the run is still running, and each checks that it won the write before emitting a run-end. Two consequences to expect. Stopping a run that finished a moment earlier returns a 409 instead of silently overwriting a real outcome with an asserted one. And a late log arriving after an operator stopped a run does not overwrite the disposition — it declines and logs. Runner load accounting is deliberately not gated on winning the write: the runner released its slot either way, and gating it would leak load until restart.

4.3 Targets, hosts & bastions

How each path carries the pin:

• Cron fires. The scheduler makes its enqueue params from the job's name, scope, env, executor, agencies and target host, read per fire from the job row that also supplies job-level env. An edit of a pin thus applies on the next fire, with no scheduler reload, the same as env.
• Workflow steps. The workflow engine makes its own run insert and does not go through the scheduler's params. Its column list carries the pin on the first try, on each retry, and on skipped-step rows. Thus History shows the target that the step would have used.
• Ansible runs. For an ansible run, the blast radius is the inventory plus --limit. The in-app SSH executor and the runner manifest both compute the limit through one shared helper, so the SSH executor dialing exactly the pinned host and a runner receiving the full inventory with no limit cannot disagree.

If you want a job to fan out on its full scope, leave its target host empty: the fan-out is then intentional. A pin only ever decreases the blast radius.

A pin that cannot go into a limit pattern fails with a clear message. It does not widen the run. Ansible limit patterns have metacharacters, and the limit builder refuses each name that has one. The trigger boundary rejects those first for per-run overrides, but a job's target host has no such boundary on its own: the Composer accepts free text, and Git YAML is arbitrary. If the builder put a pin such as web[01:50] into --limit without a guard, the pin would drop, no limit would stay, and a full-inventory run would occur. That is exactly the silent widening the guards exist to prevent. Thus each boundary that can make one has a guard: the Compose endpoint returns 422 for such a job at authoring time (create and update); a manual trigger returns 422 for such a target before a run enters the queue; and the manifest fails the run with 409 for Git-synced and pre-existing rows that get through neither. Git sync only warns and still writes the row. One bad pin in one file must not stop a full repo's sync, and the 409 makes that advisory posture safe. All three guards stand down when a raw ansibleLimit passthrough is given: the passthrough wins, the pin is never folded in, and there is nothing to widen.

The pin wins fully over a per-run host subset. It does not make a union with it, on either executor. A union would make --limit wider than the set that SSH dials, because a comma in an ansible pattern is a union, not an intersection. In practice the two are never both set, because the trigger boundary clears the pin when a subset is given, but the limit does not depend on that. A pin that names a host that left the scope shows as a per-host resolve error: a clear failure, not a quiet widening. Definition-level multi-host and group pinning stays intentionally out of scope.

An SSH host record (Settings → SSH Targets) contains: hostname, address (the dial host or IP; the hostname if blank), port (22), OS, user (root if blank), via (a bastion name; blank = direct), authKeyEnvVar (the name of the env var or secret that contains the private key), and a stored host key. A scope member with no matching host record becomes a reported per-host failure, never a silent skip.

• Direct: Cronomicon dials address:port and authenticates as user with the named key.
• Via bastion (ProxyJump): Cronomicon dials the bastion first, makes a tunnel to the target through it, then does the handshake to the target over that tunnel. The bastion hop authenticates with the bastion's own authKeyEnvVar key when one is configured. It uses the target's key only if the bastion has none. Thus a bastion with a different key operates correctly, and the target's private key never goes to the bastion. The dial timeout is 15 s for each hop.

Auth keys: the key that authKeyEnvVar names must be an unencrypted private key. The server refuses keys with a passphrase, .pub files, certs, and .ppk files. Use Test connection on a host or bastion to probe it. The results are: verified, cred_error (an auth or key problem), conn_error (not reachable, or a host-key mismatch).

4.4 The cron scheduler

An in-process engine fires job and workflow schedules. It accepts 5-field cron (min hour dom month dow), 6-field seconds-first cron, and descriptors such as @daily. All times are in the effective application time zone: the Settings → General Timezone if set and loadable, or the host TZ. The UI shows each timestamp in the same zone. Thus a “2 a.m.” schedule means the same thing in the editor, in Upcoming, in History, and in reality (an edit of the zone re-times each schedule). On each tick, a fire must pass these gates:

1. Operator pause. Paused jobs and workflows do not fire. The suppression is recorded as a skipped run.
2. Global concurrency cap. At or above maxConcurrent (default 5), the fire does not occur — also recorded as a skipped run.
3. Concurrency policy. If another run holds the same concurrency key, a Forbid job records the fire one time as a skipped run; a Queue job parks it instead (see the policy note below).
4. Working calendars. A bound skip calendar covering the day, a bound run-day calendar that does not, or any global calendar covering the day, all suppress the fire. Evaluated after the pause check and before the concurrency cap, so a policy suppression is recorded as one rather than being swallowed by a cap that happened to be full at the same moment. See the calendar note below.

Suppression rows de-duplicate per episode, keyed on the definition, the schedule entry and the reason — a job held for an hour writes one row per cause, not one per tick, and a change of cause (the cap clears but a calendar then covers the day) writes a fresh row rather than hiding behind the old one.

The scheduler is only one of several run producers, and not every producer applies every gate. A run can start from a cron fire, a click, a service-account token, a reaction, a pending-run promotion, or a file arrival — and each origin applies a deliberate subset, from one shared gate matrix (unknown origins get every gate):

The asymmetries are deliberate. A person clicking Run past a pause sees a confirmation dialog that makes the override a decision; a service account cannot see that dialog, so the token path honours the pause. Entry-level calendars are a property of a schedule entry's clock, so only the scheduler applies them — a fleet-wide global calendar freeze stops every producer.

A fire reads the job again. Thus job-level env, and the job's pinned target host (§4.3), apply on the next fire without a scheduler reload.

The scheduler reloads when a sync completes, and on a 5-minute backstop. In-app Schedule Builder edits apply immediately. Empty and Manual schedules never fire.

The concurrency policy is Allow, Forbid, or Queue. Allow (the default) lets runs overlap. Forbid records a colliding fire as skipped. Queue parks it: the fire becomes a pending_runs row due immediately, so the ordinary 15-second promoter re-judges the gate on every pass and releases the run when the key clears — no new table and no new loop. Because it rides pending_runs, the 24-hour catch-up grace applies verbatim: a run queued behind a permanently wedged job expires to missed rather than waiting forever. The queue depth is capped at 3 per key — a compiled safety backstop, not a setting — and a fire beyond the cap falls back to Forbid behaviour and says so in its reason. A manual trigger answers 409 on a Forbid collision but 202 on a Queue park (with the queue depth in the body) — an integration that retries on 409 must not retry a 202. Queued rows appear on Schedules → Upcoming with a queued chip and “waiting for the gate” in place of a timestamp, each cancellable. Workflows have no concurrency policy and cannot queue. There is no Replace policy: a fire that must displace a running instance is an operator's stop followed by a run, never an implicit kill.

The concurrency key resolves in three steps: an operator-set custom key if the job declares one, else the job's uid, else the source/name pair as a last resort. Finished runs keep the key they were enqueued with, so an audit over old History rows still groups as it did.

Activation windows. A schedule entry can have startAt and endAt: RFC3339 instants that limit when its cron can fire. The server stores them on the first-class schedules catalog and on the runtime definition_schedules expansion (migration 770). A ref expansion copies the window down with cron and env, so the runtime row stays self-contained. Enforcement is a cron.Schedule decorator around the parsed expression at registration. Thus the timing wheel, the reload path, and the fire path did not change. An entry before its window is registered but gives no fire time until its start. An entry past its endAt gives the zero time (the robfig “never again” contract) and stays registered but silent. Both bounds are absolute instants. An app-timezone change has no effect on them; only the cron fields are zone-relative.

Know these three semantics: a startAt in the past is inert (no catch-up, no backfill, and no thundering herd of fires at the moment of a reload); an elapsed window never deletes the entry, because git-source rows must round-trip; and the window gates scheduled fires only. Manual runs, run-now, and webhook triggers do not use the scheduler, so the window has no effect on them. The window goes into the schedule's content_hash only when set, so an unbounded row's digest does not depend on it. You can also author windows in Git YAML (startAt/endAt on a schedule entry or a schedules/*.yaml spec). A malformed window there is an advisory validation error, and the entry syncs unbounded. It does not stop the repo's sync.

Schedule modes. A schedule entry fires by one of three rules. cronutil.ParseSpec resolves the mode centrally, and registration goes through the same cron.Schedule mechanism, so the engine does not know that there is more than cron. cron is a cron expression, with an optional window. interval (the interval column) fires every N from startAt. N is a Go duration (36h, 90m) or a day count (7d), with a minimum of one minute. Interval needs startAt, because a repeating interval needs a phase anchor and cron has none. once is a startAt with no cron and no interval: one fire, then the zero time forever. Cron and interval are mutually exclusive. The API (422), Git YAML (validation error), and the composer all enforce this in the same way.

Two storage notes. cron is NOT NULL on both schedules and definition_schedules. Thus an interval or one-shot entry stores cron = ''. Empty cron is this system's “no cron” convention, and each read path resolves the mode through ParseSpec, not from one column alone. Interval arithmetic is on absolute instants: pure duration addition, with no calendar walk. Thus DST (daylight saving time) has no effect on an interval, but an interval does not keep a wall-clock time of day across a DST change. That is the correct meaning of “every 36 hours”. An operator who means “every day at 5 p.m.” wants cron, and the UI text points them there. The interval segment joins content_hash only when set, so a plain-cron row's digest does not depend on it.

Deferred ad-hoc runs. The Run dialog's When to run section puts a fully-validated manual trigger in pending_runs. It does not put the run in the queue. The frozen EnqueueParams (the env snapshot, the override envelope, the targeting, and the connect-as identity) go on the row. A promotion loop in the scheduler (15s cadence) replays them through the ordinary enqueue path at run_at. This is intentionally not a one-shot schedule entry: that changes the job definition, needs Compose or Admin, and cannot carry the run envelope. It is also not a runs row with a not-before: that would add a new predicate to both hot claim queries, and would count against the concurrency cap from insert. The run-trigger endpoints create the row, so it has the run-trigger gate: session, CSRF, and scope, and the triggerJobs permission. It does not have the Compose gate. The gates apply at promotion, when the run really enters the system: the cap, the concurrency policy, operator pause, a global calendar freeze, and that the definition still exists, is enabled, and is not in the recycle bin. A blocked row tries again on each tick, and carries a waitingOn field naming what holds it (concurrency, recycleBin, or the pause/freeze hold) on Schedules → Upcoming, where it stays listed and cancellable.

Catch-up is different from cron. An operator who selected a specific instant almost always wants the run to fire after a restart. Thus a due row found 24 h late or less fires. A row found later is set to missed: it writes a durable History marker timestamped at the intended instant, raises the missed-run alert, and stays visible on Schedules → Upcoming until you dismiss it (terminal parked rows age out on the runs retention window). An in-app delete of the job or workflow is a soft delete (§8.7), so its parked runs are held (waitingOn: recycleBin) rather than destroyed — restoring the definition releases them, purging it removes them. Cancel is DELETE /pending-runs/{id} (SU-2 scope-gated). In the promote/cancel race, the first actor wins.

Pending runs fire with frozen authorization (RB-Q12, pinned by test). The server authorized the trigger when the row was created, and promotion does not check again. Thus, if you revoke a person's access, the runs that they already scheduled stay. A check at fire time was examined and rejected: a parked run whose authorization silently goes away is a run that never occurs and never says why. The correct response is operational, and it belongs in your offboarding runbook: revoke the grants, then remove that user's pending runs. Schedules → Upcoming lists them with their creator, and you can cancel each one.

Working calendars. A calendar is a named set of wall-clock dates (migrations 860/870: calendars and calendar_days, the second with an ON DELETE CASCADE foreign key into the first). A schedule entry binds calendars in two roles, stored as JSON arrays in the skip_calendars and only_calendars columns on both the first-class schedules catalog and the runtime definition_schedules expansion. A ref expansion copies them down as strings, exactly as it does env and the activation window, so the runtime row stays self-contained. The gate is a fourth item in the fire-time list above, and it runs in fire() and fireWorkflow().

The day is a wall-clock day in the application time zone, never UTC. This is the one part that must be right. If days matched in UTC, each deployment east or west of Greenwich would suppress the wrong side of midnight: a 23:30 local fire on a holiday would not be suppressed, and a fire early the next morning would be. Tests assert both directions.

Precedence is skip first, then only. The effective skip set is the entry's own skip_calendars plus the days of every calendar marked global. If the instant's day is in that set, the fire is suppressed and the reason names the responsible calendar. Only after that does the only_calendars test run: a non-empty only-set suppresses a fire that lands outside it. Thus an entry that says “only on fiscal-close days, but never on a holiday” behaves correctly, and the audit row names the holiday rather than the only-set.

The global flag is skip-polarity only, and the API refuses both directions of the unsafe combination: a calendar used in any entry's only-role cannot be made global, and a global calendar cannot be bound into an only-role. A global only-calendar would mean that nothing in the system fires except on its listed days. That is a fleet outage one checkbox away, so it is unrepresentable rather than discouraged.

Suppressed fires are recorded, not silent. A skip suppression always writes a terminal skipped run through the same recordSkippedFire path the Forbid policy uses, with the responsible calendar in a structured suppressed_by_calendar column beside the human reason text. History filters on that column (?calendar=, or ?calendar=* for any), so a rewording of the message cannot break an audit query. An only-mode suppression records only when its calendar sets record_suppressed, default off: a business-days-only entry would otherwise write an audit row every weekend and bury the signal. Skips are the compliance question, so skips always record.

Calendars are never Git-authored, deliberately and permanently — there is no calendars/*.yaml, no discovery and no prune. Git-authored jobs and workflows that bind a calendar are a different thing and are fully supported: skipCalendars and onlyCalendars on a schedule entry or a schedules/*.yaml spec. An unknown calendar name there is a sync validation error, matching the API's 422, because a typo in either role fails silently at run time: a dangling skip binding fires, and a dangling only binding never fires again. Writes need the Compose capability (Admin), on the same argument as schedule definitions — whoever can edit a calendar can lift a change freeze or, in only-mode, halt production automation. Reads are session-only, because the Inventory and Upcoming views need them.

Storage and hash notes. The two calendar segments join a schedule's content_hash only when non-empty and are polarity-tagged, so an unbound row's digest does not depend on them, while flipping an entry from “never on holidays” to “only on holidays” does change the digest. There is no referential integrity between the JSON columns and calendars; that is priced in and guarded at authoring time on every write path, including reference expansion. Deleting a bound calendar returns 409 unless forced.

Noticing, as distinct from killing. A job can carry two soft deadlines beside its hard timeout_seconds: Warn after (seconds) (warn_after_seconds) raises an alert when a run has been going that long, and Must finish by (HH:MM) (must_finish_by) is a wall-clock deadline in the application time zone — a deadline earlier in the day than the run's start resolves to tomorrow's instant. Neither ends the run. The hard timeout remains the only thing that kills. Enforcement is a server-side scan (one-minute cadence) rather than a per-run timer, because a runner-executed run holds no server goroutine to hang a deadline on — which is also why an SLA survives a restart. The warn fires once per run, claimed through runs.sla_warned_at so two schedulers cannot both page (migration 930).

The missed-run detector answers “the schedule expected a fire and no run appeared”. Every five minutes it walks each enabled schedule's expected fires — resolved through the same parser the engine fires by, activation windows included, in the application time zone — from a persisted cursor, five minutes of grace behind now and bounded at 24 hours of catch-up. Its first pass after a deploy anchors the cursor and scans nothing, so a fresh deploy does not alert on every fire since the epoch. It reports a miss only when nothing explains the gap: no run row near the instant, no recorded suppression for that definition and entry anywhere that day, the definition not paused, disabled, or binned, and no parked Queue row waiting on the gate. The day-wide suppression clause has a known cost — a real miss can go unreported on a day that also had a suppression — bought deliberately to avoid false pages. Workflow schedules are covered; per-workflow SLAs (a deadline on a whole graph) are not in this version.

Both facts can page. Alert rules take two triggers beside the run outcomes: SLA breach and Missed run, matched by name — an existing “any outcome” rule deliberately does not cover them, so enabling the feature converts no old rule into a pager. The same is true of Run suppressed rules. Every such alert also writes an Activity row, so an install with no SMTP or Apprise still sees it; the alert outcome is warning, not failure. Failure, missed-run and SLA-breach notification bodies also carry the definition's annotation when one is set — up to two lines, Critical: yes and Contact: …, never the notes — and the subject line is unchanged, so mail filters keyed on it are unaffected.

File-arrival triggers. A job can declare a watch: block beside its schedule (stored in jobs.watch_json, migration 960): a list of glob paths, each with an optional stableSeconds. When a matching file appears — and holds still for the stability window (default 5 seconds) — a run starts, carrying CRONOMICON_WATCH_PATH (the full path), CRONOMICON_WATCH_FILE (the basename) and CRONOMICON_WATCH_SIZE (bytes) in its environment. The file is never copied anywhere: the job's script already runs where the file is. Declaring a watch is the opt-in — it is deliberately not gated on requestable, which belongs to a different door. The watching is done by runner agents, by polling, inside an operator-set allowlist — the deployment and security model is in §7.

The same file fires once. Arrivals de-duplicate on a unique (job, path, size, mtime) key — a key, not a check-then-insert, because two runners reporting the same arrival in the same second is the ordinary case. A file replaced at the same path fires again (the nightly-drop case); an unchanged file does not re-fire on every scan. Arrivals meet the ordinary gates: pause and the fleet cap are honoured, a Forbid job does not start twice because two files landed, and a Queue job parks the second. Every arrival — including one that started nothing — leaves a row in the job's file-sightings ledger (GET /jobs/{jobId}/file-sightings; a sighting with no run carries its refusedReason), aged out on the runs retention window. A file-arrival run records trigger_kind = 'webhook', the external-event kind (see Appendix D).

4.5 The reaction engine

A reaction runs one definition when another finishes. Beside the schedule modes (cron, interval, one-shot), the click, the service-account token and the file arrival, it is the trigger with no clock and no external event — pure cause and effect inside the system. Jobs and workflows may react to each other in any combination; the four pairings are one mechanism, not four.

The reactor polls; it does not hook the terminal writers. A loop beside the pending-run promoter scans runs and workflow_runs every 15 seconds for rows that reached a terminal state. It reads the tables that are the truth because there is no single terminal seam to hook: the kill route and the workflow-skip path do not call the notification dispatcher, so a reactor built on that seam would miss every stopped run — the exact class of event stopped exists to catch.

The scan window overlaps deliberately. completed_at is written by several uncoordinated writers and is not monotonic across concurrent runs, so a strict watermark would silently drop an event that landed with an earlier stamp. The reactor scans a 15-minute lookback behind its cursor and de-duplicates on a delivery key, which is correct under overlap, restart and clock skew alike.

Outcomes are normalised before matching. Jobs have five terminal states and workflows are binary, so both are projected onto success / failure / stopped. A job warning normalises to success — it ran and it finished. stopped is an operator ending something without classifying it; a stop that carries a disposition (§4.2) follows that disposition instead. A cancelled workflow is stopped keyed off the cancel flag, not the status, because a cancel arriving during the final step is never observed and the run finalises success. A skipped run — including a calendar suppression — is not an event: nothing ran, so nothing may cascade.

A reaction resolves into a pending run, not a dispatch path. It writes a pending_runs row, so the concurrency cap, the concurrency policy and the definition-exists check are judged once, at promotion, by the same code an ad-hoc deferred run uses. The two gates that path does not apply — operator pause and calendars — the reactor applies itself.

Five gates can stop a delivery, judged in this order and each recorded as itself:

1. The reaction, or its owning definition, is disabled or paused.
2. A global working calendar covers today. Global tier only: a bound entry-level binding is a property of a schedule entry's clock, and a reaction has no clock. A fleet-wide freeze must stop a cascade, because a chain that runs through a freeze is worse than a schedule that does — nobody authored it to happen today.
3. The per-reaction minimum interval. A blocked event is dropped, not deferred: unlike a cap deferral the event has already been consumed, and there is no later instant at which it would still be true.
4. The chain-depth ceiling, fixed at 5 and a compile-time constant rather than a setting — it is a safety backstop, and exposing it invites raising it. A downstream run inherits its cause's depth plus one; a workflow's child steps inherit their parent run's depth unchanged, since running a workflow's own steps is not another hop.
5. The catch-up grace. Anything whose upstream finished more than 24 hours ago is recorded as expired and not fired, so a Monday restart after a weekend outage cannot discharge a weekend of cascades at once.

Every decision is recorded, including the ones not to fire. Each (reaction, upstream run) pair writes a reaction_deliveries row — written before the verdict, so the record of having considered an event survives a crash mid-decision. The row carries fired, expired, error or a suppressed_* verdict naming the gate, with the calendar and day, the interval or the depth in its detail. The table is retention-pruned alongside runs; the durable half of the trail is reacted_to_run_id on the run row itself, which is what History's because-of link reads.

The delivery log has no console view and no API today. This is stated rather than left to be discovered, because the product tells operators that a suppressed reaction is recorded, and this is where that record lives. Read it with SQL against the database:

SELECT delivered_at, owner_kind, owner_name, name, src_kind, src_run_id, outcome, result, detail
 FROM reaction_deliveries
 ORDER BY delivered_at DESC
 LIMIT 50;

The one suppression that does not need SQL is the depth ceiling, which also writes an Activity row (below). So a first-line operator has one honest boundary: the reaction itself, its outcome and its missing flag are visible on Schedules → Reactions, and every other “why did it not fire” needs an administrator.

Migrations 880, 890 and 900. 880 adds the reactions edge table and the delivery log, plus reaction_depth / reacted_to_run_id on both run tables. 890 rebuilds runs and workflow_runs to admit trigger_kind='reaction' — note that rebuild stashes and restores run_agencies and runs.workflow_run_id, which SQLite's cascade would otherwise wipe and blank. 900 adds an error verdict for a delivery that decided to fire and could not. Migration 950 later rebuilt workflow_runs again to admit trigger_kind='workflow' for sub-workflow child runs (§4.6), retracting 890's recorded assumption that a workflow is never triggered by a workflow — the same stash-and-restore care applies, because runs.workflow_run_id is a real foreign key.

Deleting a watched definition is refused, not cascaded (RX-24, the CAL-22 shape). DELETE /jobs/{id} and DELETE /workflows/{id} return 409 listing the reactions that watch the definition, unless ?force=true is passed, which deletes it and leaves those reactions dangling. The Git sync prune deliberately does not guard: there is no request to fail and no operator to ask, so a definition removed from the repo is pruned and its watchers go dangling. Dangling is therefore a supported state that the Reactions tab flags as missing, not an impossible one. A definition in the recycle bin reports the same way: a reaction watching a binned definition shows missing and cannot fire, a new reaction cannot be authored onto one, and a restore makes the existing watchers whole again.

The refusal carries its own error code, reactions_watching, rather than the generic conflict the git-source refusal uses. Both delete routes can 409 for those two unrelated reasons, and only one of them is clearable — so a client that keys on the status alone has to guess (and would report an in-app job as Git-authored whenever reactions blocked its delete). Every delete surface branches on the code, keeps the confirm open and offers Delete anyway, which is the same request with ?force=true.

4.6 Sub-workflows: a workflow as a step

A workflow step can be another workflow (type: workflow). The child executes as a real workflow run — its own row, its own step walk, its own History entry, its own cancellability — linked to its parent by workflow_runs.parent_workflow_run_id plus the parent step's node id (migration 950; the child records trigger_kind='workflow'). It is composition, not inlined expansion: the child's result enters the parent's name-keyed results map exactly as a job step's does, so {fromStep} references and branch conditions address it identically, and a step-level retries re-triggers the child whole.

What crosses the boundary, and what does not:

Two guards, because dual-source means the graph can change between authoring and fire. Authoring walks the stored graph across both sources with the proposed graph substituted, and a cycle refusal names the whole path. At run time a depth ceiling of 3 (a compiled constant, not a setting; it composes with the reaction depth ceiling rather than sharing it) stops a descent authoring could not see. A refused descent — past the ceiling, or through a dangling reference — leaves a terminal child row stamped where the child would have been (“Refused: sub-workflow nesting would reach depth…”), so a parent failing with no obvious child has an answer in History. Cancelling a parent propagates: the cancel walks parent_workflow_run_id depth-first, each child being its own run with its own walk. A restart still fails in-flight runs, parents and children alike — a nested run is just another running row to the orphan sweep.

In the visual editor the palette offers + Sub-workflow; it renders as one collapsed node. The one place it is not offered is as a bare parallel arm — wrap it in a one-step sequence instead, and the validator says so in those words. Git-authored workflows gain a structural check at sync for the same vocabulary; that failure is an advisory log line, not a sync error (§5.2).

5 GitOps & GitLab Sync

Cronomicon keeps a local clone of your GitLab “job-definitions” repo. On each sync, it makes its runtime tables from the clone again. For Git definitions, the flow goes in one direction: Git wins on read. A separate Publish path writes back.

5.1 Repo layout

Each YAML file needs apiVersion: cronomicon.io/v1 and a kind:. An unknown value is a validation error with a line number. The sync walks each of jobs/, scripts/, schedules/, and workflows/ recursively. Thus definitions can be in sub-folders (for example, jobs/db/backup.yaml or workflows/team/deploy.yaml), and they show as a folder tree in the UI. The nested path is recorded as source_path. Identity stays metadata.name and id, so references and detail routes see no change. Only inventory/ is read flat, not recursively.

Reactions are Git-authorable. A jobs/<name>.yaml or workflows/<name>.yaml may carry a spec.reactions list, each entry naming the watched definition (onKind, onName, and an explicit onSource when it crosses the Git/in-app boundary) and the onOutcome to match. cronomicon validate checks everything checkable without a database, so a bad outcome or a malformed name fails at merge-request time; the upstream-exists check runs at sync, where a bad reference drops the whole definition and skips its prune rather than half-landing it. A definition reacting to a sibling in the same repo validates on the first sync — the repo is authoritative for its own definitions. Reactions on a Git-defined job cannot be authored in-app: the API refuses, because a sync replaces that definition’s reactions wholesale and an in-app write would vanish at the next sync.

A synced inventory/*.ini does three things:

• The raw file goes byte-exact to an cronomicon-mode runner for ansible-playbook -i.
• The server also parses the file into an advisory projection: groups, [group:children] and [group:vars], and inline host-vars. Scopes shows the projection read-only, with an advisory — not authoritative badge. On a construct outside the supported subset, the parser stops with a clear unavailable status and discards the half-parsed tree. The raw inventory still goes out unchanged.
• The server imports the hosts into ssh_hosts as source='git' rows. Thus the in-app SSH executor dials inventory hosts with zero operator action. ansible_host, ansible_port, and ansible_user map to address, port, and user. The auth-key name comes from a per-host cronomicon_auth_key_env_var or from the per-scope sidecar authKeyEnvVar: names only, never key material.

On a hostname clash, an operator-authored cronomicon overlay row wins. Imported rows are read-only in Settings → SSH Targets, with a git badge. To override one, author an Cronomicon host. On sync, the server prunes stale imported rows by owner. It never touches operator rows. The projection is never authoritative for execution.

5.2 How a sync runs (read path)

A sync starts in one of three ways: a poll (5 minutes or less), a GitLab push webhook (the primary path; it decreases the “shows after next sync” window to seconds), or a manual resync. Each sync does these steps, in this sequence:

1. Fetch, then git reset --hard to the branch.
2. Initialize and update the git submodules recursively.
3. Parse scripts/ and schedules/ first, so that refs resolve. Then parse jobs/, workflows/, and inventory/.
4. Do the cross-reference validation.
5. Upsert (insert the row, or update it if it exists) in one transaction.
6. Prune each source='git' row that this pass did not see.
7. Record the HEAD SHA, reload the scheduler, and write a gitsync event.

The rules, in order:

• The server validates the X-Gitlab-Token first, always. A caller without the secret gets a 401 and learns nothing about the configuration of this install.
• Webhook off → 403, and the response names the setting. This is a refusal, not a silent accept, because GitLab records it as a failed delivery in its own hook log. That log is where an operator looks to find out why a push did not sync.
• An event type whose flag is off → 202, and no sync. The delivery is not an error; the server just does not act on it. The server ignores, in the same way, an event that has no flag at all (Issue Hook, Pipeline Hook, or a wildcard hook that points at us). A delivery with no X-Gitlab-Event header (our own tools, or a curl smoke test) counts as a push.

Default policy: a fresh install is enabled, with all three events on. The policy read fails open: no row, no table, or a read error gives “enabled, with all events on”. Thus a short database error cannot quietly stop a repo's sync. The flags only ever let you turn delivery off.

5.3 How to read sync events

In History → Git Sync, the status values are: success (for example, “Synced 4 jobs, 7 scripts, 2 schedules, 1 workflows, 3 scopes”); warning (partial: some files had validation errors); failure (nothing landed, usually a clone or auth failure). Pulled = an inbound sync. Pushed = a publish (also in Schedule Pushes).

5.4 CI validation (recommended)

The binary contains cronomicon validate, which runs the same parser that the runtime uses. Connect the included CI template to the job-definitions repo, so that broken YAML is rejected at merge time:

cronomicon validate .
# jobs/backup.yaml:2: unsupported apiVersion "cronomicon.io/v2"

§10.3 in the User Manual tells you about the Publish (write) path and its 412/422 handling.

5.5 The other source's history: revisions & the recycle bin

Git definitions carry their own history in Git. Cronomicon-source jobs, workflows and schedules get the equivalent in the database: every in-app save appends a snapshot to an append-only definition_revisions log, written inside the same transaction as the write, and a no-op save writes nothing. The snapshot is the compose input, not the database row, and a restore re-submits it through the ordinary write path — re-validated, re-audited, and RBAC-checked, never a raw row overwrite. Rows show a History button; the endpoints (GET /definitions/{kind}/{name}/revisions, POST …/revisions/{no}/restore) are unrestricted-admin only. Not everything is in a snapshot, deliberately: reactions and schedule bindings are runtime state, calendars and scopes are not versioned, and operator tags are excluded from restore (they were never part of the definition).

An in-app delete is a soft delete. It stamps deleted_at / deleted_by and the definition moves to Settings → Recycle Bin (ConfigureApp to see; restore and Delete forever are unrestricted-admin). A binned definition is unreachable — it does not fire, list, trigger, or react (killJob is the studied exception, so a running run of a just-binned job can still be stopped) — but it keeps its annotation, tags, and schedule bindings: binning a schedule captures the bindings its referrers fire on into its tombstone revision, and a restore replays them in one transaction, skipping owners deleted in the meantime. A binned definition also still occupies its name: creating a replacement answers 409 until you restore or purge, and the message says so. Purge is the hard delete — the shared implementation behind Delete forever and the retention reaper (default 30 days, §8.8): cascades finally fire, and the entity code retires so a later same-named definition mints a fresh log folder.

6 Secrets & Security

How Cronomicon keeps credentials safe: envelope encryption at rest, optional Vault, and audited reveals. This section also tells you exactly how, and where, log redaction operates, and its intentional gaps.

6.1 Secret storage

• Stored secrets get AES-256-GCM envelope encryption: a random per-secret DEK (data encryption key) encrypts the value, and the master KEK (32-byte base64, from CRONOMICON_KEK_FILE) wraps the DEK. The variable sits outside the reserved CRONOMICON_SECRET_ reference prefix on purpose. The DB never holds plaintext or a bare DEK. On return from each envelope encrypt or decrypt, the server zeroizes the unwrapped DEK and the loaded KEK, best-effort. This decreases the time that key material stays recoverable in memory. It is best-effort only, because Go has no guaranteed secure erase. Without a KEK, stored-secret operations fail with “set CRONOMICON_KEK_FILE or CRONOMICON_KEK.”
• Vault secrets store only a reference (path#field, in the Vault KV v2 key-value secrets engine). The value stays in Vault, and the server gets it live with the configured Vault auth method (AppRole or a token). You cannot reveal it through Cronomicon.
• Reveal is a POST, not a GET, exactly because each reveal writes a Change Log audit entry. It is fail-closed: the server writes the audit row before it returns the plaintext. If the audit write fails, the endpoint returns 500 “could not record the reveal; value withheld”. It does not let a secret out with no trail. Reveal needs the Manage env vars permission. A Vault-source secret returns 409. A secret deleted between the scope-load and the reveal returns 404, not an empty value. Migrate to Vault is one-way and zeroes the local ciphertext.

An SSH key is a first-class credential, not an opaque secret referenced by name. A credential created directly has no backing secret row, so its decrypted material also goes into the log-redaction dictionary. A key that shows in a run log is masked line by line, exactly the same as a stored PEM secret (a key in the standard PEM text format). The key-by-name path is permanent for runner-executed and inventory-imported hosts. Those resolve keys by name locally and normally get no credential material; see §14. The one exception: a job or script that binds an CRONOMICON_KEY_ reference sends resolved private-key material to a flagged protocol-v6 runner (see §7).

Who can point a run at a stored key. Operators can select a stored credential as a run's connect-as identity: for each run in the Run dialog, or as a job's standing default in the Composer or Git YAML (spec.ssh_credential, by label). Manage Env Vars gates both selections. The rule is the same as for reference bindings: a reference is a grant over stored key material. A username override alone needs no extra permission. The audit envelope and the job spec carry the label only, never material. On the runner path, the credential goes on the existing v6 key-delivery channel as an implicit key binding: agency membership is enforced, delivery is audited, material is redacted in logs, and only an allow_secret_injection runner can claim the run. A local-inventory runner refuses such a run (409) and does not run as the wrong identity. The connect-as selection decides who selects the pair (the operator at trigger time, or the job author), not what the server can deliver: the same material is available through a declared key binding, under the same gates.

Ansible runs. The same connect-as pair applies to ansible run types. The delivery channel is different, because the consumer is different. Cronomicon's own SSH client dials an ssh-family run, so the override rewrites the resolved target list. ansible-playbook dials for itself, so the override travels as two explicit manifest fields, which the agent turns into connection extra-vars (-e ansible_user=…, -e ansible_ssh_private_key_file=…). That tier is intentional: extra-vars are ansible's highest-precedence source, so the override wins over an inventory-authored ansible_user on each host. The -u and --private-key flags are in the lowest tier. They would lose to the inventory exactly on the hosts that an admin had configured, and would apply silently and only sometimes. Key material still moves only over the reviewed v6 delivery channel. If the referenced key did not arrive, the agent fails the run closed and does not fall back to the inventory's key. The server's protocol floor tracks the current wire protocol, so an agent that could not carry an identity cannot register at all. Terraform stays excluded at every layer (422 at both authoring boundaries; dropped at each enqueue fold). It authenticates through its providers, so the fields could do nothing.

Advanced Ansible run options. Operators can set --check, --diff, --tags and --skip-tags, verbosity, --become and --become-user, and -e extra-vars for each run. They are per-run only and live only in the run's override envelope: no job columns, and no scheduler or workflow fold. They are the flags that an operator wants at trigger time, not standing job properties. Two results are operational. (The server's protocol floor guarantees that every registered agent understands these options, so a requested dry-run can never apply for real.) First, -e values are argv-visible: they land in the runner's process table, where each local user on that host can read them. Thus they are documented as a non-channel for secret material (references stay the way to inject a secret), and the un-redacted audit envelope records them exactly. Second, the server refuses ansible_user and ansible_ssh_private_key_file as extra-vars (422). -e is the tier that the connect-as identity uses, and ansible takes the last occurrence of a repeated variable. If the server accepted them, a run could connect as a different identity than the one in its own audit record. The agent also sends the identity pair after the operator extra-vars, so the sequence enforces the same rule independently. The --become toggle has no extra permission gate: a playbook can already declare become, so the control gives nothing that the caller could not already run. A become password is never a free-text field, because it is secret material. See the next paragraph for how a password-bearing run is gated.

Become passwords. A job can name a Secrets row that supplies its Ansible become password: spec.become_password_secret in Git YAML, or the Composer field, by bare key. The field holds a name, never a value. The password lives in the Secrets catalogue, with each control that this gives: KEK or Vault at rest, scope and department resolution, reveal auditing, and log redaction. The server resolves it at dispatch through the same reference path as each other secret. Run inputs stay the wrong channel, and this does not open one: prompt answers land in runs.env_json in plaintext and are log-visible by design, which is exactly where a password must never be.

Delivery is as a file, not an environment variable. The resolved value goes to the runner in the sensitive block. The agent writes it to a 0600 file on tmpfs (a filesystem that exists in memory only), off the run tree. It shows only the path under the reference, gives that path to ansible-playbook --become-password-file, and wipes the file when the run ends. Anything that can see the process can read an env var from /proc. An operator who pipes one into sudo -S puts it in the target's process table, where redaction cannot go. A wiped file is a much smaller window. The mechanism is general: each secret binding can ask for file delivery. The become password is only its first consumer.

The permission rationale above does not apply to a password-bearing run. “A playbook can already declare become” is true of the toggle. It is not true of a run that carries a credential that the caller supplied. The gate is secret-read entitlement (the named row must resolve for the run's scope and department, and it fails closed if it does not), plus an injection-flagged runner. To set the field is itself a grant over stored secret material, so it carries Manage Env Vars, the same rule as spec.ssh_credential.

Two operational notes. A job with a become password automatically declares the become-file requirement token. Only an agent with ansible-core ≥ 2.12 advertises that token. Thus such a run waits for a capable runner; the server does not assign it to an incapable one and then refuse it. If the run looks stuck, make sure that an online runner in its department reports become-file. Also, the flag is global to the run: it cannot change per host, as an inventory-authored ansible_become_password can. A fleet with different become passwords per host must keep the inventory pattern. Passwordless sudo stays the preferred configuration for both run types. This is the exception path, not a recommendation.

Composer env passthrough. spec.env_passthrough is the RX.9 allowlist of env-var NAMES that a runner resolves from its own environment into a local-toolchain child process. Git YAML and the Composer (Advanced → Runner environment passthrough) both author it, with the same serialization. Validation uses the same env namespace rules: the POSIX character set for environment-variable names; CRONOMICON_* refused (those are injected references, not the runner's environment); and local-toolchain run types only.

6.1b Authoring-time reference validation & the oracle rule

Dispatch-time resolution is intentionally vague to the operator. runref.OperatorMessage collapses out-of-scope and missing into one sentence (“reference X is unavailable for this run”). Thus no person can use runs to probe for secret names, one run at a time (M2). The precise cause goes to the server log only. That rule did not change. Do not relax it on the run surface.

There is also a separate authoring surface: POST /api/v1/references/validate. This surface is precise, under a tighter and equally mechanical rule: the verdict can show nothing that the caller's own GET /env-vars and GET /env-secrets would not already show. Both of those lists are scope-filtered per actor (auth.ScopeReadable / secrets.Service.List(grant.CanRead)). The validator gets the same filter and degrades row by row:

• A row in a scope that the caller can read gets a precise name: “exists, but only in scope prod”.
• A row in a scope that the caller cannot read collapses to the generic not_found, byte-identical to a name that exists nowhere. Without this, Phase 1 would give a restricted actor, through the back door, the cross-scope oracle that M2 prevents.
• The caller must have read access to the requested scope itself, or the server returns 403.

The endpoint is session-gated, not Manage env vars-gated. The caller's own visibility limits the verdict, and the chips must show for the read-only operators who look at job detail. It is CSRF-gated, although it is read-only, so a cross-origin page cannot probe reference names against a live session. It is capped at 200 references per request. The endpoint never returns a value, in any form.

The visibility predicate is the same query that dispatch runs (runref.lookupScoped). Thus a ✓ at authoring time and a successful injection cannot disagree. TestValidateAgreesWithDispatch is the guard.

Bindings belong to a job's identity, not its name. Every binding reader — runner dispatch, the in-app SSH executor, log redaction, the unscoped-run pre-flight, claim eligibility, the stuck-run explanation, the become password, the referenced script — identifies the job by the uid frozen onto the run. Since two departments may hold same-named jobs (§6.1h), a name lookup could hand one twin the other's credentials; the uid cannot. Scripts keep name identity permanently — a single catalog with no agency.

The alias — injecting under a different name. A binding can name the destination key its value lands on, separately from the row it resolves. The alias is a destination, never a selector: it changes the key the job body reads and cannot reach a row that the caller could not already bind, so it adds no read authority. Note the consequence for identity: a binding is keyed on kind\0name\0as, matching dispatch's own key, so one row bound twice under two destinations is a legitimate pair and lands two keys in the manifest. Keying on kind and name alone collapsed them, which showed one verdict over two bindings — a green tick over a reference that would not resolve. Two different rows aimed at one destination are still refused, because the result would depend on resolution order. The companion GET /api/v1/reference-usage returns per-reference binding counts (jobs scope-filtered to the caller's grants; scripts unscoped): counts only, never which job.

6.1c Agency membership tables & the pre-flight report

Isolation runs on one axis: agency membership. Five tables carry it, all of the same shape (a composite primary key, with ON DELETE CASCADE on both sides): runner_agencies, scope_agencies, secret_agencies, env_var_agencies, and ssh_credential_agencies. Every run snapshots its agency set at enqueue into runs.agencies_json.

An empty membership set means “no agency restriction”. That is exactly what keeps a global secret global at run time. The rule for a change goes the other way: an unmembered secret is shared infrastructure, which each department's jobs consume. Thus its writes and its reveal are unrestricted-only (RB-Q14). No single department's admin can change or read what all departments depend on, while each department's runs still consume it. SSH keys follow the same rule: key material writes are gated on the agency that owns the key; unmembered keys are unrestricted-only; and a restricted operator who creates a secret, variable, or SSH key must assign it to a minimum of one agency that they hold (the dialogs show the picker). Without that, the row would start shared and lock its creator out immediately after creation.

The pre-flight report

GET /api/v1/agency-preflight (ConfigureApp) lists each reference binding that would not resolve under the agency predicates as membership stands. Run it before you narrow membership, and after. The report has two halves, and you can review each independently:

• referenceFindings: secret and variable bindings, under this rule: a reference resolves only if the row's agency set intersects the run's set, and the scope rule also holds.
• keyFindings: SSH-key bindings, under the same intersection rule. A key assigned to an agency can only be bound by jobs in that agency, so review this half with special care before you assign key membership.

An empty report on a fresh install is normal, and it does not mean that narrowing is safe. It means that the restriction has nothing to catch yet, because no membership narrows anything. Read membershipAssigned next to the findings. The report gets teeth only when you assign membership through PUT /api/v1/{scope,secret,env-var,ssh-credential}-agencies (ConfigureApp + CSRF, replace-per-row; the change log records each change). Script-owned bindings have no static answer, because a script has no scope of its own. The report counts them and does not guess.

The server refuses to delete an agency, with 409, while anything still belongs to it. The guard counts all four membership tables, not only scopes and runners. Without that, an agency whose only members were a secret and a key would delete with no error, and the cascade would drop those access-control facts with no signal.

6.1d The agency predicates

Three predicates read membership:

• Dispatch. claimRun matches the run's agency set (frozen at enqueue in runs.agencies_json, materialized into run_agencies) against runner membership. The general-pool rule is disjoint: only a runner that belongs to no agency can claim an untagged run.
• Reference resolution. A secret or variable resolves when (it has no agency membership, or its agencies intersect the run's) and the scope rule also holds.
• SSH keys sit on the same axis: agency-filtered, and not scope-filtered. ssh_credentials has no scope column; the foreign key to the host or the bastion is the narrowing signal.

Resolution reads the run's frozen snapshot, never live membership. The world at the moment the run was authorized decides the run's injectable set. Thus a scope moved to a new agency mid-flight cannot change what an in-flight run can reach. The out-of-agency message on a run stays intentionally generic, for the same cause as the out-of-scope one: a run must not become an existence oracle. The precise cause is on the authoring surface (§11.6).

6.1e Agency membership: the per-agency editor

Two read-only endpoints supply the operator surface that closes the last gap in this program:

• GET /api/v1/agency-matrix (session): the full grid in one read. It contains each scope, secret, variable, SSH key, and runner, with its membership. Rows with no membership are included intentionally. An empty row is not missing data. It is the statement “unrestricted, reachable from everywhere”, which is exactly what an admin must see before a change that narrows it. It is one endpoint, not nine client-side reads, because the entity-id → name join that a client would do is the bookkeeping that this view removes.
• GET /api/v1/agencies/{agencyId} (session): one agency's contents, plus onlineRunners and queuedRuns.

There is no single matrix screen. A rows × agency-columns grid over every isolated entity does not scale past about a dozen departments, so each question is answered where it is asked. “What is in Tax?” is the per-agency view (Scopes → Agencies, expand a row), which is an editor. “Which agencies hold this secret?” is an Agencies column on the entity's own catalogue row (Env Vars → Variables / Secrets / SSH Keys, Scopes, Runners).

The per-agency editor writes through PUT /api/v1/agencies/{agencyId}/members (ConfigureApp + CSRF, audited), which replaces one agency's member list and touches only that agency's rows. That isolation is the reason it exists rather than reusing the per-entity setters: an agency-shaped editor built on PUT /{kind}-agencies would have to read an entity's whole agency list, modify it and write it back, so two admins editing two different departments would race and the later save would silently undo the earlier one. The per-entity setters (PUT /api/v1/{scope,secret,env-var,ssh-credential}-agencies, replace-per-row) serve that axis, as does PUT /runner-agencies, which takes runnerId, not id. GET /agency-matrix exists because the editor's add-picker reads it.

6.1f Membership is not paperwork

An empty membership set means “no agency restriction”, so a row created by an unrestricted admin defaults to global. That default becomes load-bearing when departments create their own credentials, because of the case below.

If an actor holds the permission on more than one department, the server refuses the creation with a 422 that names the candidates. It does not guess: “their department” has no single answer, and a guess would invent an ownership claim that nobody made.

6.1g Per-agency ownership, and the ambiguity refusal

Two departments can hold a Secret, Variable, or SSH credential under the same key in the same scope. Uniqueness is (key, scope, owner_agency). owner_agency stores the agency id, so a renamed department cannot orphan its rows. Resolution gets exactly one preference tier:

1. Rows whose owner is one of the run's departments: “your department's row”.
2. Rows with no owner: shared and global rows.

In a tier, the pre-existing rule holds (scope-exact wins over global). This lets one shared playbook ask for BECOME_PASSWORD, and each department's run gets its own department's value, with no naming convention and no per-team job. The run's scope decides which row a run gets, never the person who triggered it. This is exactly why a central operations team can run a department's job with that department's credential, and can never read it.

Two smaller results. The cross-kind conflict (a Secret and a Variable cannot share a key) is keyed on the same triple. It applies to one owner: different departments do not collide, and the message leaks less. And ownership is fixed at creation: there is no transfer action. Thus you cannot delete an agency that owns rows until they are deleted (the refusal names them).

6.1h Per-agency names, and permanent identity

The same widening applies to definitions: two departments may hold same-named jobs and workflows. Each definition carries a uid — assigned once, never reused, unchanged across sync, edit and recycle-bin restore — and the uid is the primary key of jobs, workflows, and schedules. Name uniqueness comes in three deliberate shapes:

• Cronomicon-source jobs and workflows: unique within each agency their scope maps to; the All pool overlaps everything. A set-overlap rule no SQLite UNIQUE can express, so it is enforced as a checked invariant on every write path (compose, sync, restore). A workflow's name pool derives from its steps' job scopes — a workflow with no job steps sits in the All pool and collides with every same-named workflow.
• Git definitions: keep full (source, name) uniqueness — one repository, one namespace, and sync keeps its atomic upsert.
• Schedules: keep full (source, name) uniqueness — a schedule has no scope, and the name-addressed /schedules/{name}?source= route stays valid.

A name conflict is refused with one deliberately generic sentence — “name already in use” — that never names the owning agency (the §6.1b oracle rule again: the refusal must not become a cross-department catalog probe). Where a bare name is genuinely ambiguous, Cronomicon refuses rather than guesses: a workflow step matching two jobs answers “qualify the step by identity”, a recycle-bin restore or purge answers 409, and a bare-name definition lookup answers 409 ambiguous_name until you pass ?source=. On screen, a collision inside your own visibility renders the name agency-qualified (deploy · FIN).

6.2 Log redaction & its gaps

The server redacts run output at ingest, server-side, before it touches the disk. This is identical for both executors. The dictionary contains: each env-var value in the run's scope, plus the global */“All” scope; the decrypted plaintext of stored secrets; and the run's ad-hoc per-run env override values. Thus a secret pasted into a run override in error is also masked. Matches become [REDACTED], longest value first, so a shorter value cannot show part of a longer one. A multi-line secret, such as a PEM key, is split, and each line goes into the dictionary. Redaction runs on each single log line, and a multi-line blob would never match.

Through the reference-injection subsystem, the redactor also gets, for each run, the values that the run really injects through its declared bindings. This includes Vault-source values, which the per-scope stored-secret dictionary cannot hold. Runner ingest of a run that injects a secret is fail-closed (503 redactor_unavailable; nothing persists, and the agent tries again when the fault clears). The run's authoritative injects_secret flag drives this decision. Thus a deleted binding, or the CRONOMICON_SECRETS_INJECTION_ENABLED kill-switch turned mid-run, cannot move a secret-bearing run to the lenient path.

Separately, A12 captured inter-job outputs are stored raw, because the engine injects them into later steps. The run-detail response masks them against the same dictionary at display time, fail-closed: each value is masked if the redactor cannot be built. An output that captures an injected secret value is refused at the capture site: the run fails with reason output_secret_leak, before outputs_json is written. Thus it never goes to a later step. The in-app SSH executor and the runner ingest path behave identically here.

What is intentionally not redacted:

• Short values (less than 5 characters) and common literals (true, prod, localhost, …). A mask on them would damage ordinary log text.
• Vault values echoed without a binding. A Vault value that a run injects through a declared reference binding is masked (it enters the per-run dictionary). But a Vault value that your script gets and shows itself is not held locally, so it is not in the dictionary. Be careful when a script shows an un-injected Vault value.
• Schedule env values. They are plaintext in Git, and never redacted.
• The server's own process log (cronomicon.log). Redaction applies to run logs at ingest. The process log has no redactor. Thus a credential that gets into a server log line (usually inside an error string) is written as-is. The same is true of stdout, but the file is a durable artifact on the data volume. It is 0640 in a 0750 directory, and size-bounded. Set CRONOMICON_LOG_FILE_ENABLED=false to keep server logs on stdout only. See §8.4 for its location on disk, and Appendix C for its controls.

What is redacted, beyond run logs: the compliance audit stream (audit.log, §8.8). Its free-text fields (target, summary, details, and reason on auth events) pass through a process-wide dictionary: every stored secret, stored SSH credential, encrypted integration setting and multi-line variable the server can decrypt, in every scope. The row is masked before it is stored, so the database row, the audit.log line and the CSV export all carry the same text. The dictionary rebuilds itself after any write to those stores, so a rotation's own audit row is masked with the new value. Vault values are the residual here too: they exist only at dispatch time, per run, so a global dictionary cannot hold them.

6.3 How to keep the KEK safe, and the trust boundary

Credentialed outbound egress is hardened. The Vault (X-Vault-Token) and GitLab (Private-Token) clients refuse HTTP redirects, because Go's stdlib does not remove custom headers across a cross-host redirect. Each outbound client that the server starts (Vault KV/status, GitLab REST, go-git clone/fetch/push, S3/MinIO backup, OIDC, Apprise) goes through a shared SSRF egress guard. The guard blocks cloud-metadata (169.254.169.254, fd00:ec2::254), loopback, and link-local targets, and it closes DNS-rebind: it dials the checked IP. CRONOMICON_OUTBOUND_ALLOW_PRIVATE (default true) keeps an internal Vault or GitLab reachable. CRONOMICON_OUTBOUND_ALLOW_LOOPBACK (default false) lets in a 127.0.0.1 Vault-agent sidecar. The credential provider for the EC2 and ECS instance metadata service (IMDS) is exempt, so S3 IAM (identity and access management) roles still operate.

6.4 HashiCorp Vault authentication

A Vault secret stores only a path#field reference. The server gets the value live at dispatch. Settings → Integrations → Vault decides which credential Cronomicon presents to do that: address, Auth method, and one write-only credential field. Two auth methods are available.

Token auth supplies no Role ID, and the save accepts that. The status badge and the client wiring use one shared completeness check, so the badge cannot show ok while the client is still a stub: Not configured means exactly that the client has not been wired.

A switch of the auth method clears the stored credential, intentionally. An AppRole Secret ID is not a Vault token, and a credential carried across a switch made a client that failed each read. The switch needs the new credential. The write-only field has the label Token or Secret ID, to agree with the selected method, and its placeholder shows if the credential is set or unset. Response-wrapped credentials stay AppRole-only. A pair of response wrapping and token auth is refused at construction. The alternative sends a single-use wrapping token as X-Vault-Token and gets 403 on each read, with nothing to show why.

The environment path did not change and still wins over the stored config: CRONOMICON_VAULT_* is always AppRole. Only the UI/API-configured path can select token auth.

6.5 Service accounts: something other than a person can start a run

A service account is a machine principal: a name, a role, an agency (or All scopes), and a bearer token. Manage them in Settings → Service Accounts — gated by manageRoles, deliberately, because minting one is granting access, which is the same act as writing an access grant (migration 910). The token has the crnsvc_ prefix, is shown once at mint, and authenticates as Authorization: Bearer crnsvc_… — no cookie, no CSRF. An expiry is optional; last_used_at is stamped best-effort on each authentication so you can find dead tokens. Deleting an account is a revoke, not a delete: the row is retained as an audit actor, so History never shows a run with an actor that does not exist.

The trigger surface is two endpoints, addressed by name: POST /trigger/jobs/{name} and POST /trigger/workflows/{name}, taking the same body as the Run dialog's endpoint. A bare name that exists in both sources answers 409 ambiguous_name until ?source= disambiguates. Runs started this way record triggerKind: webhook with actor svc:<name>. Two gates are specific to this door: a job must opt in with its requestable flag (§3; refusals answer not_requestable), while a workflow has no such flag — for workflows the token's role-and-agency grant is the whole gate, so scope workflow-triggering accounts narrowly. The token path honours an operator pause (§4.4's gate matrix): a service account cannot see the confirmation dialog that makes a pause override deliberate, so it does not get one.

6.6 Recommended security posture

§6.3 explains the trust boundary. This section is the operator checklist that follows from it — the settings worth being deliberate about, and the two credentials whose handling decides whether a restore is a recovery or a second incident.

The KEK belongs in a file, not the environment

Cronomicon reads the key encryption key from CRONOMICON_KEK_FILE first and CRONOMICON_KEK only as a fallback; if both are set, the file wins. Prefer the file and remove the environment variable rather than leaving both in place — an environment variable is readable from /proc/<pid>/environ, from a container inspect, and from anything that dumps process configuration, and a leftover copy buys nothing once the file is authoritative. Generate the key with real entropy; it must decode to exactly 32 bytes:

openssl rand -base64 32

The file's permissions are entirely yours to set. Cronomicon reads the file and validates only that it decodes to 32 bytes — it does not check the mode or the owner, so a world-readable KEK loads without a word of complaint. Mount it read-only, owned by the account the server runs as, mode 0400. The only names the server reads are the CRONOMICON_KEK* forms; without one, the first stored-secret operation fails with "no KEK configured".

Rotation works, but you have to finish it by hand

KEK rotation is zero-downtime and needs no outage. Every encrypted row records the version it was sealed under, and historical keys stay available, so old and new coexist:

CRONOMICON_KEK_VERSION=2
CRONOMICON_KEK_FILE=/run/secrets/kek-v2     # the new key
CRONOMICON_KEK_1_FILE=/run/secrets/kek-v1   # the old key, still able to decrypt

Rotation is lazy: a row moves to the new version only when it is rewritten. There is cronomicon rewrap-secrets to finish it. Without that command an operator would have to touch every encrypted item by hand — and there are three stores, not one, so missing any of them leaves the old key still required:

Run the re-wrap from inside the deployment, so the container's own mounted KEK is used and no person handles key material. It works against the live database — no maintenance window — and it is idempotent, so re-running is the recovery for an interrupted pass:

docker exec <cronomicon> cronomicon rewrap-secrets --dry-run   # what is outstanding, per store
docker exec <cronomicon> cronomicon rewrap-secrets             # move it all to the active version

Only once --dry-run reports rotation complete should you drop CRONOMICON_KEK_1. That report is the whole point of the command: it counts what is sealed under which version without decrypting anything, so it still answers the question when a superseded key is no longer configured. If any item cannot be re-wrapped the command says so and exits non-zero — almost always because the key that sealed it is missing from the environment. Supply it as CRONOMICON_KEK_<N> and re-run; do not read a partial pass as a finished one.

Runner enrollment: pick a posture deliberately

A runner proves itself once, at registration, with a registration token; from then on it uses the per-runner API key it was issued. There are two postures, and the difference only becomes visible during a recovery.

The reason this matters is recovery, not convenience. After restoring the database from a backup, any runner enrolled after that backup was taken will find its row missing, discard its identity, and try to register again. With a bootstrap token it comes back by itself. Without one it sits offline until someone attends to it — on every affected host.

What contains a bootstrap token is worth knowing before you decide: it permits enrolling a runner and nothing else. A newly enrolled runner arrives with no agency membership and no tags. That is a real limit, but it is narrower than it first sounds: dispatch is a disjoint two-branch rule, so a runner in no agency is not idle — it is the general pool, eligible for every untagged run and for no agency-tagged one. A newly enrolled runner therefore cannot reach a department's work until an operator places it, which is the containment that matters; but it is not inert, and a runner appearing unbound has to be treated as a question rather than a chore.

// /etc/cronomicon-runner/config.json   (0640 root:cronomicon-runner)
{ "registrationToken": "crn_reg_..." }

// in the unit:
ExecStart=/usr/local/bin/cronomicon-runner -config /etc/cronomicon-runner/config.json

Do not pass a token on the installer command line. runner-install.sh accepts the registration token as an inline argument, which places it in shell history and in the process list. That is a reasonable trade for a single-use token that expires within a day; it is not one for a bootstrap token. Write the config file by hand instead. The installer's checkout token already demonstrates the safe pattern (--checkout-token-file, or --checkout-token - to be prompted with the input hidden) — prefer that shape wherever a secret has to reach a runner host.

Settings to confirm before going to production

After any restore, verify runner placement

A runner that re-registers returns with a new identity, and therefore with no agency membership and no tags. It re-detects its own capabilities, but placement is an operator act and is not restored with it. Such a runner reports itself online, lists its capabilities, shows zero load, and claims nothing — agency dispatch is hard and disjoint, and a job carrying a runner-tag pin will not match it either. Green status is not evidence that work can be dispatched. Before declaring a restore complete, check the per-agency coverage in the Runners view and re-bind anything that came back unplaced.

7 Runner Fleet Management

A runner is the cronomicon-runner binary on a separate host. It registers, long-polls for runs tagged executor='runner', does the runs, and streams the logs back while they execute — a flush of pending whole lines every two seconds, sealed by a trailing envelope at the end. It is firewall-friendly: only outbound HTTP to Cronomicon. The server never dials runners.

7.1 When you need a runner, and which image

Capability routing keeps mixed fleets safe: the server assigns a run only to a runner whose capabilities include that run type. A run for a type that no online runner advertises stays queued (“waiting for a {type}-capable runner”). It does not fail.

Capabilities auto-detect by default. With no -c / CRONOMICON_RUNNER_CAPABILITIES, the agent probes the host's toolchains at startup (bash, perl, pwsh, python3, ansible-playbook, terraform) and claims the applicable run-types. If you install a toolchain later, a service restart sends it through the config-drift resync. Set the flag explicitly only to narrow what a runner claims. The table above still applies to containers, where the image decides which toolchains exist to detect.

7.2 Installation & registration

One-click: the + Add Runner button in the Runners view mints a token and gives you one command with no flags: curl -fsSL https://cronomicon.example.com/install/<token> | sudo bash. The /install/<token> endpoint puts the server URL, the token, and the binary download into the script (capabilities still auto-detect), so nothing goes on the command line. The manual flag form below stays for pre-authored and config-managed installs.

sudo ./runner-install.sh \
 -s https://cronomicon.example.com \
 -t crn_reg_xxxxxxxx \
 -n runner-dc1-01 \
 --known-hosts ./known_hosts \
 --key-dir ./keys

The script creates the cronomicon-runner user, installs the binary to /usr/local/bin, writes /etc/cronomicon-runner/runner.env, installs a hardened systemd unit, and starts the service. The key directory (/var/lib/cronomicon-runner/keys, 0700) and CRONOMICON_RUNNER_KEY_DIR are always provisioned. Thus a key added later operates with no runner.env edit. --generate-key <NAME> makes a passphrase-less ed25519 key that the runner owns, and it prints the public half for the targets' authorized_keys. --known-hosts is optional: a fresh runner starts with an empty known_hosts and trusts target hosts through the operator-approved Scan & approve flow (§7.5). It does not refuse SSH runs. --ca-cert installs a private CA (certificate authority) bundle. --inventory local plus --local-inventory provision a network-isolated (T-b) runner.

Checkout and Vault runners take one-pass secret placement: --allow-checkout, --checkout-repos <csv>, --checkout-token-file <path>, and --vault-pass-file <path>. The two secret files are copied into 0640 root:cronomicon-runner custody, and the bytes never cross the server. A secret passed as a flag value is refused: use the file form, or the hidden-prompt --checkout-token -. The full flag table is in the Install Guide §3. Or run a container:

docker volume create cronomicon-runner-data
docker run -d --name cronomicon-runner-01 --restart unless-stopped \
 -e CRONOMICON_RUNNER_SERVER=https://cronomicon.example.com \
 -e CRONOMICON_RUNNER_REGISTRATION_TOKEN=crn_reg_xxxx \
 -e CRONOMICON_RUNNER_NAME=runner-dc1-01 \
 -e CRONOMICON_RUNNER_KEY_DIR=/var/lib/cronomicon-runner/keys \
 -e CRONOMICON_RUNNER_KNOWN_HOSTS=/var/lib/cronomicon-runner/known_hosts \
 -v cronomicon-runner-data:/var/lib/cronomicon-runner \
 ghcr.io/resetsmith/cronomicon-runner:2.0.2 # -fat for ansible/terraform

7.3 The two-token model & identity

• Registration token (crn_reg_*): single-use, one per install. You mint it from the Runners view, with an optional label. It expires in 24h, dies on its first successful registration, and records which runner consumed it. The server stores only a SHA-256 hash; the plaintext shows one time at mint. You can revoke unused tokens from the list.
• Per-runner API key (crn_run_*): minted at registration, valid 365 days, kept in the agent's identity file (0600 JSON, default /var/lib/cronomicon-runner/identity.json). The agent uses it for all later polls, manifests, and logs. You can blank the registration token after this.

Keep --name stable and the identity file on durable storage. Then a restart continues the same registry row and does not orphan it.

7.4 Inventory modes

You select the mode for each runner: cronomicon (the default: the server resolves scope hosts and sends fully-resolved targets, with key names only), or local (the manifest carries only the scope name, and the agent resolves it against its own -local-inventory file). Use local for network-isolated segments, where the host list must stay off the control plane.

7.5 Security model

7.6 Lifecycle: drain, offline/reaping, deregister

• Drain: the runner stops claiming new work, completes the active runs, then goes offline (immediately, if it had none). The server-side drain deadline default is 1 hour. After that, a force-kill occurs, and the runs are marked danger / drain_timeout. Under systemd, systemctl stop sends SIGTERM. The unit's TimeoutStopSec=300 (5 min) is the separate stop-grace window for that signal. Drain before maintenance.
• Resync: declares the runner's current local config again, in place (same id, same API key; agencies kept; active runs not affected). This occurs automatically: the agent polls with a config digest, and the server requests the re-declare on a mismatch. Thus a declared-config change or a new agent binary propagates on the next poll after the restart (with a guard against rapid repeats). The button stays as the force-lever.
• Offline / reaping: a reaper (about every 60s) marks a runner offline after CRONOMICON_RUNNER_OFFLINE_AFTER (default 5 min) and sets its in-flight runs to failure with reason runner_lost (an amber Lost badge). A runner offline longer than CRONOMICON_RUNNER_DEREGISTER_AFTER (default 14 days) is deregistered automatically. Offline is not permanent: an authenticated poll from an offline runner re-admits it to online — see the callout below.
• Degraded: a freshness reading, derived at read time and never stored. An online runner whose last heartbeat is more than 2 minutes old reports degraded. So does one that registered but never polled, or whose heartbeat does not parse. Agents poll about one time each minute, so this is two missed polls: late enough to not flag ordinary jitter.
• Deregister: permanent. It revokes the API key immediately. To fully retire a runner: Drain → stop the service → Deregister → delete the identity file. If the runner was compromised, also rotate the keys that it held.

The OpenAPI enum lists degraded, and the UI draws a Degraded tile, but each stored write is online, offline, or draining, and the table's CHECK constraint permits no more.

Derived, not stored, and this is intentional. The stored status is the runner's lifecycle; this is a freshness reading on top of it. To store it would need a wider CHECK constraint, and the reaper's WHERE status IN ('online','draining') sweep would need the new value. Without that, a degraded runner would never be offlined or deregistered. As built, it is one function in the shared response mapper. Only online degrades: draining is already on its way out, and offline is already reaped. The derived value is never written back, so it has no effect on the reaper, drain deadlines, or deregistration.

What it means for an operator: the Degraded tile counts this window, and the UI treats a degraded runner as reachable. Resync, Scan keys, and Drain stay available, because a degraded runner still heartbeats and still holds work, and that is the state where you most need them. Dispatch and claiming are not affected: they use the runner's stored status, not this reading.

Offline is not a one-way trap. Registration only INSERTs and redeclare leaves status alone by design, so the poll handler is the path that brings an offline runner back. An authenticated poll is treated as proof of life. The bearer token is bound to the runner id, and after a drain completes no agent process remains — so a poll can only mean a deliberate restart. The row flips to online (guarded, so a concurrent deregister loses harmlessly), the stale drain deadline is cleared, an activity row records the re-admission, and the same response falls through to deliver pending resync, managed settings, and work. Recovery is one round-trip. The mid-long-poll status re-check behaves the same way: a runner inside a long-poll is alive by definition, so an offline flip landing there re-admits instead of shutting down a healthy agent. That case is not hypothetical — it is what happens when Cronomicon itself is down past the 5-minute threshold and comes back.

7.7 Agency-based isolation

Capability routing (§7.1) is not the only constraint: a run must also go to a runner that can reach its network. Agencies are operator-defined network-isolation zones. The catalog and the per-scope binding live with Scopes (§11; a scope is bound to the one agency that its hosts are in). Each runner is assigned to zero or more agencies, edited inline per runner in its expanded row: removable chips, plus an add-picker. Each change commits through PUT /api/v1/runner-agencies for that one runner. There is no whole-fleet membership grid (§13.1).

At enqueue, the effective scope's agency name is snapshotted onto the run: frozen and immutable, the same as scope. At claim, it is matched against runner membership under a disjoint rule:

• Only a runner that is a member of that agency can claim a run tagged with the agency.
• Only a runner with no agencies at all (the general pool) can claim an untagged run (a run whose scope has no agency).

With no agencies defined, each run is untagged, and dispatch does not change. Isolation becomes active only when an operator binds a scope and assigns runners. Membership is operator-assigned only. A runner never self-declares its agencies; the registration path does not accept membership.

Runner management is departmental. To change a runner (drain, resync, managed settings, secret-injection, host-key approval, deregistration, tags, and the connection test), the caller must hold the configure app permission on an agency that the runner belongs to. A runner in no agency is the general pool, which each department depends on. Thus a change to it takes an unrestricted operator: the same rule as shared secrets. To mint and revoke registration tokens is also unrestricted-only, for the same cause: a new runner joins the general pool and serves each department, so enrolment is a fleet-wide act. To re-home anything between agencies, the caller needs authority on both sides of the move. To claim something shared, the caller needs unrestricted access.

7.8 Ansible checkout projects (opt-in)

By default, an ansible runner gets one self-contained playbook body per run. Checkout mode lets a runner do a full playbook project: roles, vars_files, include_vars, and .j2 templates. The runner gets the playbooks repo at a server-pinned commit SHA and runs its entry playbook from a materialized tree. It is opt-in per job (a kind: Script wrapper that declares project_root/entry in Git) and per runner (-allow-checkout). Body-only runs do not change. The full operator walk-through is in the runner guides: the Install / Config / Security Guide buttons on the Runners view. They open standalone pages at /runner-install.html (§12) and /runner-security.html (§8), single-sourced from documentation/runner-*.html. These are the essentials for the fleet operator:

• Four controls limit the added power: the opt-in, plus a per-runner repo allowlist (-checkout-repos); a full commit SHA, pinned at enqueue and verified by the agent (never a ref); a read-only, per-runner deploy credential (the worst leak is read access to one repo); and a tree secret-scan in cronomicon validate and sync, which catches inline plaintext secrets that a vars/*.yml or .j2 could move past the inventory guard.
• Dependencies. A project's requirements.yml is installed per run. Collections already present at the exact pinned version are not downloaded again (this operates offline). Pins are enforced: cronomicon validate fails CI on an unpinned collection or a non-SHA git role; sync warns.
• Vault (opt-in). A job declares requires: [vault]. The agent supplies --vault-password-file from its own config; the password never travels. Register that password as an Cronomicon stored secret, so that logs mask it. Note: values that a playbook decrypts at run time get around redaction, so they need no_log: true. Or make HashiCorp Vault the source of truth for that secret, and for each {{ lookup('env','NAME') }} runner-local secret, with the opt-in Vault Agent sidecar on the runner host. The AppRole-authed sidecar renders /etc/cronomicon-runner/secrets.env (0640 root:cronomicon-runner), and the unit sources it optionally. The runner never holds a Vault token, and the value never crosses the control-plane→runner channel.
• Capability tokens & claim-gating. Runners advertise checkout, collection:<fqcn> (a fully qualified collection name), vault, and sandboxed tokens. They are detected at registration and shown on the Runners page. A job's requires set claim-gates it to a runner that satisfies each token: the same disjoint machinery as agencies. A run that no runner satisfies WAITS with “no online runner satisfies this run's requirements […]”. It does not fail.
• Tier 2 sandbox. Each local-toolchain run is wrapped in a per-run systemd-run --scope for resource caps. It gets the runner unit's filesystem and syscall hardening. Availability is probed and reported. Where the sandbox is absent, runs operate unsandboxed and say so.
• One key, both engines. One provisioned SSH key serves bash/Go-SSH runs and ansible runs. For a single-key run, the agent adds --private-key <path> to ansible-playbook automatically and injects the authKeyEnvVar name into the ansible child env (the auth bridge). Ansible also verifies target host keys against the same known_hosts that Scan & approve seeds. Thus there is no double-provisioning and no ~/.ssh/known_hosts symlink. An explicit env var, or an inventory ansible_ssh_private_key_file, still wins. -no-auth-bridge (CRONOMICON_RUNNER_NO_AUTH_BRIDGE) opts an estate out, and -ansible-ssh-common-args lets it own the full SSH arg string.

7.9 How to manage a runner from the UI

Runner registry rows are click-to-expand. The compact row keeps Test + Deregister and shows the runner's tags as chips. The expanded detail panel holds: the health data (heartbeat, load/max, Sandboxed or Unsandboxed posture); a copyable Runner ID, which is a UUID (universally unique identifier) string and not an integer; the full capability list; its resolvable Key names chips (a host's authKeyEnvVar must match one); an agent-vs-bundled Upgrade available badge; recent runs (GET /runs?runnerId=<uuid>); and the action strip: ⚙ Settings, Resync, Scan keys, Drain.

Server-managed settings let an operator override a runner's operational controls with no SSH, no restart, and no env edit. Use the ⚙ Settings drawer → PATCH /api/v1/runners/{id}/settings (ConfigureApp + CSRF, activity-audited; migration 560). Managed keys are tri-state overrides (present = the server wins; absent = the runner's local value holds): maxConcurrent; the three sandbox caps (sandboxMemoryMax/sandboxCpuQuota/sandboxTasksMax); allowCheckout; checkoutRepos; and a subtract-only capabilityMask, which narrows a runner's claimed run-types and never widens them. Delivery is additive, over the existing poll channel (protocol v4). The row shows a pending-ack dot until the agent echoes the applied version back. A Resync only re-declares the runner's own local config and can never change it from the server side. A managed override is different: it is a genuine server→runner authority direction, and it changes runner behavior without a resync. Injected secrets and the runner's own key material still stay runner-local.

Tags are operator-authored, SQLite-only labels, edited inline from the expanded row (PUT /api/v1/runner-tags/{runnerId}, ConfigureApp; trimmed and deduplicated; a maximum of 30 tags, each a maximum of 64 characters). They have no sync-preservation concern, because runners self-register and do not come from a Git catalog. They are not merely descriptive: a job or a single run can be pinned to a tag, and only runners carrying it may claim that work — see §7.10. Editing a runner's tags therefore changes what it is eligible to run, from its next poll. For install-time and startup triage, cronomicon-runner doctor runs bounded preflight checks: the server-URL scheme; /healthz reachability (it can tell an SSO login page from a real response); each $PATH directory responsive; the identity file writable; and registration-token / resumed-identity readiness. It prints a labeled PASS/WARN/FAIL report. It is wired as a non-fatal systemd ExecStartPre (doctor --quick) and runs one time at install. doctor --auth NAME [user@host] resolves a credential exactly as a run would (key-map → env-PEM → key-dir) and dials the target with the agent's own SSH stack. It classifies the result: auth OK, host-key-unknown, permission-denied, or connect-failed. It never logs key material.

7.10 Runner pinning (targeting work by tag)

Agencies answer whose runner may take a run. Pinning answers which. Inside one department every runner is interchangeable as far as agency isolation is concerned, which is the right model until the runners stop being interchangeable in fact — one sits in a DMZ, one reaches only the east datacentre, one is inside a PCI segment. A pin names a runner tag, and only runners carrying that tag may claim the run.

Tags rather than runner identities, deliberately: a runner's id is minted at registration, so re-enrolling a host produces a new one and any pin naming the old id would silently stop matching. A tag survives re-enrollment and lets several runners satisfy the same pin. Matching is case-insensitive; stored casing is preserved for display.

Where a pin is authored

A job's pin has one storage location, and one place it is written from:

• The declared pin — spec.runner_tag in the job's YAML, persisted to jobs.runner_tag. Git owns it and every sync overwrites it, exactly like description. It sits in review beside its siblings executor, target_host and ssh_credential, which is where a durable “this job must reach the DMZ” belongs. Jobs authored in Cronomicon set the same field from the composer.

Precedence

Resolved once, at trigger time, and frozen onto the run — a later change never retroactively moves a run that is already queued. Highest first:

1. the per-run override from the Run dialog or the trigger API;
2. the job's declared pin;
3. unpinned — any otherwise-eligible runner.

The per-run rung is three-state, and the middle state is the point: absent falls through to the job's pin, while an explicitly empty value means force-unpinned — a deliberate decision to ignore the pin the job declares, for this run. Sending no field and sending "" are different acts with different outcomes, and the API keeps them distinct (absent versus ""). That distinction is how a single run escapes a pinned job without the job being edited — and it is the only way to do so from inside Cronomicon.

Rules worth knowing before you rely on it

• A pin narrows and never widens. It is applied on top of agency isolation, not instead of it. Pinning a run to a tag only another department's runner carries does not send it there — the run waits. There is no combination of pin and tag that reaches a runner the department rules would deny, and that is an invariant, not a current behaviour.
• A pin is not a reservation and not a priority. It restricts who may take the work; it does not hold capacity, jump the queue, or wake an offline runner. (A real per-run priority field does exist on the trigger API — higher claims first, ties oldest-first, per-trigger only and deliberately without a job-level default — but it is a different lever: ordering among eligible runs, never eligibility.)
• Pinning a run that resolves to the in-app SSH executor is rejected (422 invalid_runner_tag). That executor is the control plane itself, so there is no runner for a tag to select. The check tests the resolved executor, so a job with no explicit executor can reach it through the default chain in §4.1.
• An unmatched tag is legal. The fleet is never consulted at enqueue: a runner may be enrolled a minute later, and a scheduled job must not start failing nightly over a transient fleet condition. The run queues and records why.
• Scheduled runs, deferred runs, workflow steps and reactions all inherit a job's pin automatically, resolved through the same rungs as the Run button.

A pinned run that nothing can claim records a reason distinguishing the two cases an operator would otherwise conflate: no runner in the fleet carries that tag (a typo, or a runner nobody has tagged) versus runners carry it but none is online and otherwise eligible (offline, draining, wrong department, wrong run-type). They send you to different screens, and they read: “this run is pinned to runners tagged <tag>, and no runner in the fleet carries that tag” versus “this run is pinned to runners tagged <tag>, and none of them is online and otherwise eligible” — the fleet-wide count deliberately ignores runner status. The runner Config Guide's stuck-run table lists both.

7.11 The file-watch duty (file-arrival triggers)

When jobs declare watch: globs (§4.4), the watching is done by runner agents — the server never watches a filesystem. An agent polls its watched directories every 10 seconds. Polling rather than inotify is a decision, not a shortcut: the stability window already requires repeated stats, inotify does not fire for changes made by another host on NFS/CIFS/SMB — the most common drop-directory shape — and polling adds no agent dependency and no watch-descriptor limit. The cost is latency bounded by the scan interval. A file must hold still for its stability window (default 5 seconds; per-path stableSeconds for large files over slow links) before it counts as arrived.

The security posture is inverted, and that is the point: the server chooses the globs, the agent enforces. A runner watches only what falls inside its local -watch-paths allowlist (env: CRONOMICON_RUNNER_WATCH_PATHS), re-checked on every scan, behind the -allow-watch opt-in (CRONOMICON_RUNNER_ALLOW_WATCH). An empty allowlist permits nothing — the same fail-closed rule as scope grants — and -allow-watch with an empty list logs a loud warning rather than watching silently. The reverse direction holds too: the server refuses a sighting for a path it did not distribute to that runner, and watch distribution respects agency isolation on the same disjoint rule claimRun uses. Watch specs ride every poll (standing configuration: an edited watch must reach the agent with no operator action, and a restarted agent must re-acquire the full set); a server whose watch list for an agent has been retracted to empty answers a bare 204, and the agent applies the empty set — so deleting a watched job stops the scanning without a restart. Sightings are reported over POST /runners/{id}/file-sightings (runner token). The protocol floor guarantees that every registered agent understands the watch field, so no agent can drop it silently and leave a job that simply never runs. The agent echoes the watched job's jobUid, and the server refuses a sighting whose uid does not match its own copy of the watch instruction.

8 Deployment & Day-2

This section is for the administrator who deploys and operates the service. Cronomicon is a single Go binary that serves an embedded UI. Its data is SQLite on one volume, behind your reverse proxy and identity provider. It is a control plane: it schedules the work and monitors the work, but the runners and the SSH executor do the work.

8.1 Topology

8.2 The environment variables that are important

Config is env-only; there is no config file. The full matrix is in Appendix C. These variables carry the most load:

8.3 First production deploy

Cronomicon ships as one container image, published for linux/amd64 from every release tag as ghcr.io/resetsmith/cronomicon:<version> (also tagged <major>.<minor> and latest; pin the full version). You can also build it yourself from backend/Dockerfile, as described below. How you run it (Compose, systemd, Kubernetes) is your choice. The steps below describe a Compose layout that matches the topology above: a reverse proxy that terminates TLS and does forward-auth against an identity provider, with the app on an internal network and its port never published.

1. Give the proxy its certificates, or configure ACME (the protocol that gets certificates automatically). Set the real hostnames in the Compose file and in the identity-provider config.
2. Point your identity provider at your LDAP or Active Directory server. Make sure that group membership shows in Remote-Groups.
3. cp cronomicon.env.example cronomicon.env. Set CRONOMICON_TRUSTED_PROXIES to the proxy's static internal IP or network (a /32 is the tightest setting).
4. Write the base64 KEK to a file mounted read-only into the container (CRONOMICON_KEK_FILE), and back it up separately.
5. Set CRONOMICON_BOOTSTRAP_ADMIN_GROUP for this first deploy. Then run docker compose up -d.
6. Log in as a member of that group (you get admin). Create the real access grants in Settings → Users & Access.
7. Remove CRONOMICON_BOOTSTRAP_ADMIN_GROUP and run docker compose up -d again. Make sure that the warning stops.

If you build the image yourself: stage 1 of backend/Dockerfile builds the SPA from source, and the Go binary embeds the result. Thus the image can only serve a bundle built from source HEAD. That stage runs vite build, not npm run build (which is tsc -b && vite build). Type checks are a source gate: they belong where a type error must fail, locally and in CI, not in an image build. Leaving tsc out also saves about 700 MB of peak RSS (the memory that the build holds) and 14 s in the image build.

Know this failure mode. A stale file left behind on a deploy host's checkout (one that git has since deleted, and which no clean checkout contains) is invisible to vite build, which bundles only what the entry reaches. But tsc -b type-checks each file under src/, imported or not, so an unrelated type change can make such an orphan fail the build with nothing but exit code: 1, while the same build passes from a fresh clone.

backend/web/dist is excluded from the build context, so the image never falls back to a committed bundle: what it serves is always what stage 1 built.

8.4 Health, persistence & verification

All durable state is on the cronomicon-data volume at /var/lib/cronomicon: SQLite in WAL (write-ahead logging) mode, the git-cache, the logs, and the backups. The container is disposable; the volume is not. Health endpoints (all unauthenticated): /healthz (liveness + version), /readyz (503 until the DB is reachable and the migrations are clean), /version, and the internal-only /metrics. Probe a live stack from outside the trust boundary:

# 1. Reachability through the proxy
curl -fsS https://cronomicon.example.com/healthz

# 2. Spoofed identity straight at the app port must NOT yield a session (expect 401/403, never 200)
curl -s -o /dev/null -w '%{http_code}\n' \
  -H 'Remote-User: attacker' -H 'Remote-Groups: admins' \
  http://cronomicon-host:8080/api/v1/me

# 3. /metrics must not be reachable through the public hostname (expect 404)
curl -s -o /dev/null -w '%{http_code}\n' https://cronomicon.example.com/metrics

The critical check calls the app port directly with spoofed Remote-User/Remote-Groups headers. The app must remove them and return a code that is not 200. A 200 there means that the trust boundary is broken.

The run-log tree has one folder per entity. A run's log goes to {log dir}/{code}/{traceId}.log. {code} is an opaque 8-hex-character code that the server assigns to the owning job or workflow. It is intentionally not the entity's name: for a git-sourced definition, the name is whatever the repo says. Runs with no definition (today, only the SSH Test connection probe) go in _system. Each folder has a _meta.json sidecar that names the kind, source, and name that own it, plus createdAt and, when the entity is deleted, deletedAt. The sidecar is the only way to tell, from the log tree alone, what a folder is, or if its entity still exists. This is what makes the tree readable after it is archived, or mounted where the database is not.

logs/
 0000001f/ <- a job's folder
 _meta.json {"code":"0000001f","kind":"job","source":"git","name":"nightly-backup",...}
 3f9c1a20-....log
 7d20b8e4-....log
 000000a3/ <- a deleted job: _meta.json carries "deletedAt", folder stays
 _meta.json
 _system/ <- runs owned by no definition (SSH "Test connection")
 b81e5f77-....log
 cronomicon.log <- the process log (knobs in Appendix C)
 cronomicon.log.1 <- rotated by size; does not end in .log, so the reaper skips it
 audit.log <- the compliance audit stream (§8.8)
 audit.log.20260726 <- rotated by UTC date; likewise invisible to the reaper

All four kinds of file share this one directory. Only the *.log run logs are subject to the run-log retention window. The live cronomicon.log and audit.log are excluded by name, and their rotated generations do not match *.log at all. The _meta.json sidecars are also never reaped.

Know this before you look on disk. A delete of a job or workflow keeps its folder in place, so the audit trail outlives the definition. The sidecar gets a deletedAt stamp; the folder does not go away. A definition created again under the same name gets a new code and a new folder, so it can never receive the old one's logs. Also, a GitLab sync prune is not a delete: a git job that leaves one sync (a repo reorganization, a rename, a short YAML error, a branch switch) and comes back keeps its code and its folder. The run-log retention window (§8.6) still reaps the files in dead folders, so the tree stays bounded.

8.5 The trusted-proxy control

8.6 Backups & restore

A retention and backup worker runs a daily sweep at a fixed wall-clock UTC time (CRONOMICON_BACKUP_AT, default 02:00). It prunes aged rows and on-disk run logs, per the eleven retention windows under Settings → Audit & Compliance. It reads the windows again at the top of each sweep; the CRONOMICON_RETENTION_* env vars only seed those windows on first boot. It then takes a consistent VACUUM INTO snapshot and uploads it to S3, if a bucket is configured. Backups are configured only through CRONOMICON_BACKUP_S3_* env. There is no DB-backed backup setting. On boot, the worker runs an immediate catch-up sweep only if the last successful backup is overdue (24h or more). Thus frequent restarts cannot push the only durable backup past 24h. Success and failure export as Prometheus metrics (cronomicon_backup_last_success_timestamp_seconds, cronomicon_backup_failures_total). Restore is a file swap, and the included cronomicon restore subcommand scripts it: download the latest or a named snapshot, swap, and run PRAGMA integrity_check. Stop the server first:

1. Stop the container.
2. Run cronomicon restore --list, then cronomicon restore (latest) or cronomicon restore --from <snapshot>. Or do it by hand: get the snapshot (aws s3 cp … / mc cp …), remove the live DB and the stale sidecars (cronomicon.db, -wal, -shm), and copy the snapshot into place.
3. Start the container. It applies pending migrations. /readyz reports database: ok when clean.
4. Supply the KEK again (and the OIDC/session keys). They do not live in the DB. Without the original KEK, stored secrets are unrecoverable.
5. If the S3 log archive is on, open Settings → Log Storage and run Sync now with reconcile (POST /settings/log-storage/sync?reconcile=1) once the server is up. A snapshot older than the last sync does not know which logs are already in the bucket; reconcile lists the prefix, restores the archive markers, and resets the archived count to what the bucket holds. History then reads those logs from S3 (§8.6.1).

8.6.1 The S3 log archive

Run logs are files, not rows: a database-only restore keeps every run in History and loses every log those runs point at. The Local + S3 archive backend under Settings → Execution → Log Storage closes that gap. It is an archive tier, not a replacement: local disk stays the only write target during a run (live tailing depends on an append-only file with stable offsets), and a scheduled sync copies each sealed log to the bucket afterwards — every terminal run whose log has not been archived, oldest first, sixty seconds after it completed. The marker on the run row is set only after the upload's reported size matches the local file. Once it is set, History serves the log from the bucket through the server whenever the local file has been reaped: same endpoint, same offset header, never a redirect, so the bucket is never reachable from a browser.

The timetable is yours. Every 5 or 15 minutes, hourly, every 6 hours, or daily at a UTC wall-clock time. No interval is unsafe: syncs never overlap (a tick that fires during a tick is skipped, and the interval is measured from the end of the last one), each tick is bounded (200 files or 10 minutes, the rest carries), and a missed tick is simply caught up. What the interval decides is exposure — logs completed since the last sync exist only on this host's disk — against request volume, which is one PUT per run whatever the interval. The first sync after enabling archives the whole backlog on disk, budgeted. Sync now runs one tick on demand; the status line beneath the fields shows the last sync, the pending count (growing tick over tick means the interval is too long for the volume or the link is too slow), and the last error.

Retention has two knobs. Run Log Files governs local disk, with one rule: while the archive backend is on, a log the sync has not yet copied is never reaped locally, however old — the tier that exists for recovery must not lose the very logs it was meant to keep. The eleventh window, Archived Run Logs (S3), governs the bucket and defaults to 0 = keep forever: Cronomicon deletes from the bucket only when you set it. When set, the sync deletes expired objects itself and the bucket policy must grant s3:DeleteObject — the opposite of the backup bucket's guidance, so keep the two buckets (or prefixes and policies) apart. At 0, withhold delete and use an S3 lifecycle rule if you want expiry at all. A log the window has expired reads as no log in History, like any reaped one.

Private nodes. Any S3-compatible endpoint works (MinIO, Ceph, NetApp): set Endpoint to host[:port], leave it empty for AWS in the given region, switch Use SSL off for plain HTTP on a LAN, or paste the internal CA's PEM into CA Bundle for HTTPS the process trust store does not know. Static keys are both-or-neither; both empty uses the host's AWS credential chain (env, profile, instance role). The save probes the bucket with the credentials it would store and refuses with the S3 error rather than persisting a connection that does not work. The egress guard applies (private ranges allowed by default, loopback and cloud metadata blocked); the secret key is envelope-encrypted and covered by cronomicon rewrap-secrets.

Monitoring. cronomicon_log_archive_last_success_timestamp_seconds, cronomicon_log_archive_failures_total and cronomicon_log_archive_pending join the backup metrics; a third consecutive failed tick logs at ERROR. Each tick that did something writes one Activity row (log-archive); idle ticks write nothing. Two gauges cover the redaction dictionary: cronomicon_redaction_dictionary_size and cronomicon_redaction_dictionary_rebuilds_total{outcome} (complete | partial | failed) for the process-wide redaction dictionary (§8.8).

8.7 Day-2 runbook essentials

8.8 The compliance audit stream (audit.log)

The server writes a third file, next to the per-run logs and the process log: audit.log, the compliance audit stream. It is not a diagnostic log. It is a schema: one JSON Lines record per line, with fixed keys and a version stamp ("v": 1). Thus a SIEM (a security information and event management system), a log shipper, or an auditor's script can parse it without a record of releases. It is intentionally not routed through the process logger. The operator can change that logger's level, format, and destination, and that would silently reshape the audit record.

Where it is. audit.log is in the run-log directory (Settings → Log Storage), and it follows that directory live when the setting changes. Or pin an absolute path with CRONOMICON_AUDIT_LOG. Mode 0640 in a 0750 directory: the same custody as the logs next to it. CRONOMICON_AUDIT_LOG_ENABLED=false turns the file off fully.

What it carries. The full change_log record (each configuration change, with the actor); a named subset of activity (run-end, workflow-end, config, gitsync, push); and each auth event (§8.9). Pure run telemetry is excluded: run-start and workflow-start are the announcement half of a pair, and the other half already carries the outcome. To include them would about double the stream and say nothing new.

{"v":1,"at":"2026-07-27T09:14:02Z","source":"auth","kind":"login",
 "outcome":"success","actor":"jdoe","remoteAddr":"172.28.0.2","clientIp":"10.4.1.77",
 "userAgent":"Mozilla/5.0 …"}
{"v":1,"at":"2026-07-27T09:15:40Z","source":"change_log","actor":"jdoe",
 "category":"Settings","action":"updated","target":"Audit & Compliance"}
{"v":1,"at":"2026-07-27T09:22:11Z","source":"activity","kind":"run-end",
 "outcome":"success","actor":"jdoe","jobName":"nightly-backup",
 "traceId":"3f9c1a20-…","scope":"prod","durationMs":41822}

Rotation is daily, by UTC date, not by size. An auditor asks “what happened on the 14th”, and a date-named file answers that with no read. Generations are audit.log.YYYYMMDD. They intentionally do not end in .log, so the run-log reaper never sees them, and the live audit.log is excluded from that reaper by name. A server that was down over midnight rotates yesterday's records out at startup (the day comes from the file's mtime, not from the clock). Thus a file's name never lies about what is in it.

Retention is its own control. auditLogFiles shows as Audit Stream Files under Settings → Audit & Compliance → Data Retention — one of eleven retention windows (runs, activity, workflow runs, change log, schedule pushes, run log files, audit stream files, plus Recycle Bin, default 30 days, Definition History, default 365, Runner Placement History, default 30, and Archived Run Logs (S3), default 0 = keep forever — §8.6.1). Recycle Bin is the one window whose expiry destroys a definition rather than a record: an expired binned definition is purged as if Delete forever had been pressed (§5.5). auditLogFiles' default is 730 days, intentionally longer than changeLog's 365 and activity's 90. That gap is the point: the file is an export of those tables, and it must outlive the rows that it exported. A value below changeLog throws the exported tail away while the source rows are still there. It has no environment variable. New deployments start at the default.

8.9 The authentication & authorization trail

Auth events go into their own table (auth_events). The Auth Events filter under Settings → Audit & Compliance exports them, and audit.log carries them in full. They are pruned on the changeLog retention window. Eight kinds are recorded:

Each event carries: the actor, where one is known; a stable machine-readable reason token (exchange_failed, bad_nonce, insufficient_scope, …), on which a detection rule can key, not a human sentence; the target; the user agent; free-text details; and two client addresses.

Why two address columns. remote_addr is the immediate peer, verbatim. Behind a reverse proxy, that is the proxy, on each request. That is exactly why one column would be uninformative, not wrong. client_ip is the derived originating address. With both recorded, the question becomes empirical: if client_ip follows real operator addresses while remote_addr stays constant, forwarded-header attribution operates. If the two always match, your proxy does not set X-Forwarded-For, and the correction is a proxy-side change, not an Cronomicon change.

9 Troubleshooting & FAQ

The most frequent problems, in groups by area. Click a question to expand the cause and the correction.

Authentication & access

Git sync & publishing

Secrets & runners

General

10 Glossary

Definitions of the terms that Cronomicon and this manual use.

Runner pin

A restriction that only runners carrying a given tag may claim a job's runs. Declared on the job (spec.runner_tag → jobs.runner_tag, sync-overwritten for Git jobs, Composer-set for Cronomicon ones), with a per-run override above it. Applied on top of agency isolation, never instead of it, so it can only narrow eligibility (§7.10).

Force-unpinned

An explicitly empty per-run pin — a deliberate decision to ignore the pin the job declares, as distinct from having no opinion and falling through to it. It is why the trigger API's runnerTag is three-state (absent / "" / value) rather than two.

Reaction

A stored edge saying “when this definition finishes with that outcome, run me”. Edge-triggered and stateless; it resolves into a pending_runs row rather than dispatching directly (§4.5).

Reaction delivery

One row per (reaction, upstream run) in reaction_deliveries, written before the verdict and carrying it afterwards: fired, expired, error or a suppressed_* value naming the gate. Retention-pruned with runs; the durable link is reacted_to_run_id on the run row.

Disposition

The outcome an operator assigns when stopping a run. It becomes the run’s status; killed_by independently records that a human ended it, so the two are readable separately.

A12 inter-job data passing

The mechanism with which a workflow step consumes an upstream step's output. The producer prints ::cronomicon-output name=KEY::VALUE on stdout. The consumer declares inputs: { VAR: { fromStep, fromOutput } } (authored in Git YAML today). KEY must have env-var style. A missing output resolves to empty, deterministically.

A13 unification rule

Variables and secrets share one key namespace per scope. A KEY can exist only one time across both, in each sequence. A violation returns 409 with an “(A13)” message.

Access grant

One row that says this AD group has this role here. “Here” is an agency, or All scopes (unrestricted). A grant is the only thing that gives access. The permission and the scope come from the same grant, so the strongest verb never applies to the widest reach. You edit grants in Settings → Users & Access → Access Grants. Each edit signs out all other operators. An identity that matches no grant has no authority.

Activity feed

The event stream (/activity), newest first, of run starts and ends, config changes, pushes, Git syncs, and SSH tests. Color shows the outcome.

Agency

An operator-managed network-isolation zone, and also the department axis for access: grants are authored against agencies, and secrets, variables, SSH keys, and runners are departmentally owned through agency membership. A scope binds to the one agency that its hosts are in. Each runner is assigned to one or more agencies (operator-assigned; the agent never self-declares). Dispatch is hard-isolated: a job dispatches only to a runner in its agency, and an untagged job only to an untagged runner (the general pool). Agencies are operator-owned and never synced: a binding continues through a Git re-sync. The catalog and the per-scope binding are on the Scopes page; runner membership is on Runners.

Cronomicon-source

A definition authored in-app (no Git round-trip) and stored in the database. Admins can edit and delete it in the UI. Git sync never touches it.

Annotation

The operator-owned Notes / contact / critical sidecar on a job or workflow. Stored only in Cronomicon, keyed on the definition's uid, sync-preserved, writable by any signed-in session (no role permission), and excluded from revisions and restore. critical is display-only: it colours a chip and adds a line to failure notifications, and never changes how the job runs (§5).

Ansible inventory / projection

An inventory/*.ini (git-synced or authored in-app) attached to a scope. The raw inventory goes unchanged to ansible-playbook -i and is authoritative for execution. A parsed projection (groups, [group:children], [group:vars], host_vars) shows read-only as advisory — not authoritative. On the first construct outside the supported subset, it degrades to “preview unavailable” with a clear message. Inline secret values are rejected fail-closed at ingest; env-var-NAME indirection is the correction. Inventory hosts are imported into ssh_hosts on sync. See §11.

apiVersion / kind

Necessary top-level fields on each Cronomicon YAML. apiVersion must be cronomicon.io/v1. kind is one of Job, Script, Schedule, Playbook, Terraform, Workflow, InventorySidecar. An unknown value is an error with a line number.

Application timezone / effective zone

The one zone that governs both cron schedules and UI display: the stored Settings → General → Timezone, if set and loadable, or the host TZ. The scheduler evaluates cron in it, and the SPA shows each timestamp in it. Thus “2 a.m.” means the same thing in the schedule editor, Upcoming, History, and reality. It shows read-only as appTimezone on GET /settings/general. An invalid zone is rejected with 422. A stored value that does not load degrades to the host zone, with a warning. (Without it, cron would fire in the container OS zone, and each viewer would see their own browser zone.)

Apprise

A notification gateway that takes a URL or a DSN (data source name). It delivers one event to many services (email, Slack, Discord). Configured in Settings → Notifications, or with CRONOMICON_APPRISE_URL.

authKeyEnvVar

The name of the env var or stored secret that holds the unencrypted SSH private key for a host or bastion. The server sends only this name to runners. On this name path, key bytes never cross the wire; the runner resolves them locally. Exception: a runner flagged for secret injection instead gets resolved key material for bound CRONOMICON_KEY_* references, in the manifest Keys block. See Manifest. It is coequal with authCredentialId (the first-class SSH key credential). The name path is permanent for runner-executed and inventory-imported hosts, which resolve keys by name locally and never get credential material.

Bastion (jump host / ProxyJump)

An intermediate SSH host named in a target's “via” field. Cronomicon dials the bastion, makes a tunnel to the target through it, and makes sure of the target's host key over the tunnel.

base_sha / If-Match

The Git commit on which a publish edit was based, sent as an If-Match precondition. If the file changed after it, the server returns 412 with a diff, so you can rebase and try again.

bash floor

A scope's supported run types always include bash, and you cannot remove it. This prevents an empty type set.

Body-lint

A sync-time scan of each Script body (computed on the same read as the content hash). It shows advisory findings (crlf, non_utf8, no_shebang, shebang_mismatch, oversized) as warning chips in the Scripts catalog. It is advisory only: it never blocks a sync, never drops a script, and never enters the content hash. See §8.3.

Bootstrap admin

A first-run env var (CRONOMICON_BOOTSTRAP_ADMIN_GROUP) that gives admin to each member of a named AD group, independent of the grants. With it, the first admin can seed the real access grants. The server writes a clear warning; remove the variable after this. Trusted-header mode only: the OIDC login path never applies it. The lockout-recovery lever on all deployments is the offline cronomicon grant-admin subcommand.

Calendar (working calendar)

A named set of wall-clock dates (migrations 860/870), bound to a schedule entry in two roles: skip_calendars vetoes a fire that lands on one of its days, only_calendars suppresses a fire that lands outside its days. Skip is evaluated first and wins. A calendar marked global unions its days into every entry's skip set, and is refused in the only-role in both directions. Days match in the application time zone, never UTC. Operator-authored only — never Git-synced, though Git-authored jobs and workflows may bind one by name. Suppressed fires are recorded as skipped runs carrying suppressed_by_calendar. See §4.4.

Change Log

The audited record of in-app configuration changes (Timestamp, User, Category, Action, Target, Details). View it in History → Change Log. It is CSV-exportable. Retention is the Change Log window under Settings → Audit & Compliance (default 1 year).

Compose capability

The Create jobs & workflows permission (GET /capabilities → compose) — a grantable role permission since the seven-permission model, checked per object against the definition's scope. It gates authoring jobs and workflows within the granted agencies; schedule definitions, calendars, reactions, revisions and the recycle bin stay with the unrestricted administrator (composeUnbound reports whether All-scoped authoring is available). It is one of the role gates that the server enforces with requirePerm, with Manage env vars, Configure app, Manage roles, and Publish. The run-lifecycle actions (trigger, kill, pause, resume; workflow trigger, pause, cancel) are also role-gated, with triggerJobs/killJobs on the run's scope.

Concurrency cap (maxConcurrent)

The global limit on simultaneous queued+running runs (Settings → General; the scheduler uses 5 when unset). A cron fire over the cap does not occur (recorded as skipped). A manual run colliding with a Forbid policy is rejected with 409; on a Queue policy it is accepted 202 and parked (§4.4).

concurrency_key / concurrency_policy

A grouping key (an operator-set custom key, else the job's uid, else source/name) and a policy (Allow, the default; Forbid; or Queue) that gate concurrent runs. A fire that Forbid suppresses is recorded one time per episode, as a skipped run; Queue parks the fire (cap 3 per key) until the gate clears (§4.4).

Control plane

Cronomicon's function: it schedules and monitors jobs but does not do them itself. With no runner registered, and the SSH executor off, runs queue but never occur. This is expected, not an outage.

CSRF double-submit

State-changing operator requests must send the cronomicon_csrf cookie again, in an X-CSRF-Token header. A mismatch is rejected with 403. Runner bearer requests are exempt.

DEK (Data Encryption Key)

A random per-secret key that encrypts the secret value (AES-256-GCM). It is stored only in KEK-wrapped form. It is the center of the envelope-encryption scheme.

Deregister

Permanent removal of a runner from the registry. It revokes the runner's API key immediately. It occurs manually, or the reaper does it after the deregister window (default 14 days offline).

Developer login bypass

A non-production button (CRONOMICON_DEV_AUTH=true) that mints a synthetic admin session with the explicit * (all-scopes) grant: unrestricted under the A5 scope model. Use it for a local UI preview. It returns 404 in production.

Dirty schema

A half-applied (failed) migration. /readyz returns 503, and the container stays unhealthy. Recovery is: restore from a snapshot, roll back the image, and fix forward. Never use migrate down.

Drain

The controlled retirement of a runner. It stops claiming new work, completes the active runs, then goes offline (immediately, if it had none). The server-side deadline default is 1 hour. Systemd's TimeoutStopSec=300 is the separate SIGTERM grace window.

Definition uid (permanent identity)

The identity a job, workflow, or schedule keeps for life — assigned once, never reused, unchanged across sync, edit and restore, and the primary key. Names may repeat across agencies (§6.1h); the uid is what cascades, bindings, concurrency keys and history filters (?jobUid=) key on.

Dual-source model

Each Job, Workflow, Schedule, and Scope can live in Git (version-controlled, read-only over the API, prunable), or an operator can author it in-app (then it is its own source of truth, and it is never pruned). Both go through the same engine, keyed by (source, name).

Envelope encryption

The scheme in which a per-secret DEK encrypts the value, and the KEK encrypts the DEK. Thus the database never holds plaintext or an unwrapped key.

Event kind

The category tag on an Activity card: run-start, run-end, config, push, gitsync, ssh-test.

Executor

The attribute that decides where a run occurs: ssh (the in-app SSH executor) or runner (a distributed agent). “Auto” means no fixed per-job executor. Ansible and terraform are runner-only.

File-arrival trigger (watch)

A job's watch: globs, observed by runner agents polling inside their -watch-paths allowlist. A stable file starts a run carrying CRONOMICON_WATCH_* env; every arrival lands in the job's file-sightings ledger (§4.4, §7.11).

Folder tree

Sub-folders in the GitOps repo (under jobs/, scripts/, schedules/, workflows/) show as a navigable breadcrumb and drill-in tree in each catalog. A search flattens to full-path results. A definition's folder location (its file path) is separate from its identity (metadata.name/id). Thus a move never changes references or routes. See §8.1.

Forward-only migrations

Cronomicon applies up-migrations automatically on each boot. Down-migrations are for dev and test only. Production recovery from a bad migration is: restore from a backup.

GitOps pruning

During a sync, each source='git' row that this pass did not see in the repo is deleted. This is how a removal of a YAML file in Git removes the definition. Cronomicon-source rows are never pruned.

Honest View / Recent Logins

The Settings → Users & Access list of users who really signed in at least one time. Cronomicon never enumerates the directory.

Identity file

A 0600 JSON file ({id, apiKey}) that a runner writes at its first registration and reads at a restart. With it, the runner continues its registry row and does not orphan it. Default /var/lib/cronomicon-runner/identity.json.

Inventory mode (cronomicon vs local)

A per-runner selection. cronomicon (default): the server resolves scope hosts and sends fully-resolved targets. local: the manifest carries only the scope name, and the agent resolves it against its own inventory (for isolated segments).

Job-definitions repo

The GitLab repository that is the source of truth for all job, script, schedule, workflow, and inventory definitions. Cronomicon keeps a local clone and parses it again on each sync.

Job-level env

A plaintext env map declared on a Job (Composer → Environment (job-level)) and injected into each run, independent of the firing schedule. It is the base layer of the effective run env: job-level env, then the firing schedule's env, then the per-run override (the later value wins). It is plaintext and not redacted. Keep credentials in the secrets system.

KEK (Key Encryption Key)

A 32-byte base64 master key that envelope-encrypts stored secrets and the SMTP password. Supplied with CRONOMICON_KEK_FILE or CRONOMICON_KEK. Back it up separately from the DB backup. If you lose it, stored secrets are unrecoverable.

Manifest

The execution payload that a runner gets after it claims a run: the trace id, the job, the run type, the interpreter and body, an env snapshot (with the dispatcher-owned CRONOMICON_RUN_* context), and target references with authKeyEnvVar names. Before, it never carried secret bytes. Since protocol v6, a binding-bearing run to a runner flagged for secret injection also carries resolved CRONOMICON_SECRET_*/CRONOMICON_VAR_* values in a dedicated Secrets block, and resolved SSH-key material in a Keys block. The Secrets block is keyed by each binding's destination, not by the row name alone, so one aliased row can appear under two keys (§6.1b). It fails closed on each binding that cannot resolve, is refused to an unflagged runner, and hard-fails if the assigned agent speaks a protocol below 6.

Orphan

A script or schedule that no job or workflow references (Used-by 0). A schedule with no reference never fires.

Per-run override (override envelope)

Ad-hoc scope/executor/targetHosts/env given in the Run dialog for a single manual run. The effective values land in the run's own columns. What was overridden is also recorded as an audit envelope (runs.override_json) and shows in History as “Ad-hoc overrides (this run)”. Override env is plaintext, but it goes to the per-run log redactor. The envelope also carries audit-only keys that never change what the run does: the prompt metadata, and reviewedSections — which Run dialog sections the operator had open, plus confirmation when a deviating run was confirmed. when-to-run is also on that allowlist; unknown values are dropped, and absence means the caller was not the dialog. See §6.5.

Per-instance Disable

The one workflow field that the UI can change on each workflow, Git included. It pauses scheduled fires and blocks manual triggers (409 workflow_disabled). It is the documented exception to Git-canonicality.

Degraded (runner)

A runner whose last heartbeat is more than 2 minutes old, or which registered and never polled. Derived at read time, never stored. The stored status is the runner's lifecycle, and this is a freshness reading over it. Thus only online degrades, and the reaper's sweep does not change. It names the window in which a dead runner read Online before, because offline is declared only after CRONOMICON_RUNNER_OFFLINE_AFTER. The UI counts it as reachable: recovery actions stay available. See §7.6.

Service account

A machine principal: a name, a role, an agency (or All scopes), and an crnsvc_ bearer token. Managed in Settings → Service Accounts (manageRoles). Triggers jobs and workflows by name over /trigger/…; a job must opt in with requestable. Deleting one revokes the token but keeps the row as an audit actor (§6.5).

Reaper

A server sweep (about 60s) that marks runners offline after CRONOMICON_RUNNER_OFFLINE_AFTER (default 5m) and deregisters runners offline longer than CRONOMICON_RUNNER_DEREGISTER_AFTER (default 14d). Reaping is reversible: an authenticated poll from an offline runner re-admits it to online.

Redacted log / [REDACTED]

Run output in which scope env values, decrypted stored-secret values, and stored SSH-credential key material are masked as [REDACTED] at server-side ingest (multi-line PEM keys are masked line by line). No UI control reveals the raw log. Values under 5 characters, common literals, Vault values, and schedule env are not masked.

Registration token (crn_reg_*)

A single-use bearer token (24h expiry), minted per install from the Runners view. It dies on its first successful registration and records which runner consumed it. The server stores only a SHA-256 hash; the plaintext shows one time at mint. It is for first contact only: resync and all later calls use the runner's long-lived key, and a revoke of an unused token never affects registered runners. To mint and revoke tokens is unrestricted-only: a new runner joins the general pool and serves each department.

Reserved reference prefixes

The closed set of four CRONOMICON_* names that Cronomicon treats as run-injected references, not configuration: CRONOMICON_VAR_, CRONOMICON_SECRET_, CRONOMICON_KEY_, CRONOMICON_RUN_. A name under one is resolved and injected into runs. Each other CRONOMICON_* name is server or runner config. The match is exact-prefix (CRONOMICON_SECRET_ never matches CRONOMICON_SECRETS_), and operator-authored env can never define an CRONOMICON_* key. The stdlib-only internal/envref package is the single source of truth, shared by the server and the runner agent.

CRONOMICON_RUN_* (run context)

The dispatcher-owned, non-bindable run-context prefix. A fixed set of log-safe run-metadata vars: CRONOMICON_RUN_ID, _JOB, _JOB_SOURCE (git|cronomicon), _SCOPE (""=global), _TYPE, _TRIGGERED_BY, _EXECUTOR (ssh|runner). On the SSH path, they merge into the remote command env. On the runner path, they merge into the manifest Env block. They never resolve from a store row, and operator-authored env cannot define them.

Recycle bin

Where an in-app delete puts a definition: a soft delete (deleted_at/deleted_by), reversible from Settings → Recycle Bin until restored, purged, or expired by its retention window (default 30 days). A binned definition is unreachable but keeps its name, annotation, tags, and schedule bindings (§5.5).

Revision history

The append-only per-definition snapshot log for cronomicon-source jobs, workflows and schedules (definition_revisions). Every in-app save appends the compose input; a restore re-submits it through the ordinary, re-validated write path. Git definitions carry their history in Git instead (§5.5).

Reveal

The explicit, audited unmask of a stored secret. It is a POST, because each reveal writes a Change Log entry. Vault secrets cannot be revealed.

Resync

An alignment of the server's view of a runner with the runner's current local config, in place: same id, same API key, no deregistration. It is automatic (the agent polls with a config digest, and the server sends the re-register control op on a mismatch, with a flap guard). The Runners-view Resync button forces it. The agent reads its own config again, so a resync can never change what a runner is from the server side.

Role

A named permission template over the seven permissions, stored as data. The four built-in roles (Admin, Approver, Operator, Viewer) are included by default, and you cannot delete them. Admins can create custom roles in the Roles card. A role gets to a user only through an access grant that names it, and a user can hold more than one. Display precedence: admin > approver > operator > viewer.

Run / run lifecycle

A single execution of a job, with a status, a trace ID, timestamps, a duration, and an exit code. Both executors share one lifecycle (trigger → queued → running → terminal). A run row's executor column keeps them from a cross-claim.

Run prompt / user-defined run variable

A variable declared on a Job (name, label, required, default, options) that shows as a fillable field (text, or a <select> on options) in the ad-hoc Run dialog. Answers go on the per-run env-override path. It is warn-only: a blank required prompt never blocks the run, and the unfilled name is recorded under promptWarnings in the run's override envelope, for History. It is not the same as the inferred Script variables heuristic, which is only a Composer seed and a drift lint, never part of the run path. See §6.

Runner / runner agent

The external cronomicon-runner binary on a separate host. It registers, long-polls for executor='runner' runs, does them, and streams logs back while they execute (a 2-second flush, sealed by a trailing envelope). Ansible, terraform, and isolated targets need it. Outbound HTTP only.

Runner API key (crn_run_*)

The long-lived (365-day) per-runner key, minted at registration and kept in the identity file. The runner uses it for all later polls, manifests, and logs.

orchestrator_lost

The reason set on a workflow run, and on its queued and running child runs on all executors, that the workflow orphan reaper reclaimed after a mid-workflow restart or the 24h stale limit. It shows a plain Failed badge, not the amber “Lost” badge.

runner_lost

The reason set on a run whose runner was reaped offline mid-run. The run becomes a failure, with an amber “Lost” badge.

Schedule (definition)

A first-class named cron for use by more than one job (+ optional plaintext env), referenced by jobs and workflows through scheduleRefs. Source: git or cronomicon. An edit of an cronomicon schedule sends the new cron, and its activation window, to each referencer.

Activation window

Optional startAt/endAt bounds on a schedule entry. They delay its first fire, or end it, without a change to the cron expression. They are absolute instants and gate scheduled fires only. A past start is inert (no backfill). An elapsed end leaves the entry listed as expired, never deleted.

Schedule mode

The rule by which a schedule entry fires: cron (a cron expression), interval (every N from startAt, which is necessary as the phase anchor), or once (a startAt with neither: one fire, then never). Cron and interval are mutually exclusive. An interval-mode row stores cron = ''. Schedule reads report it as the derived mode field.

Schedule Push

A commit published to GitLab from the publish builder. It is audited in History → Schedule Pushes, with the file, the schedule change, the SHA, and the status.

scheduleRefs / script_ref

scheduleRefs lists the first-class Schedules that a job or workflow binds to. script_ref points a Job at a Script by name (with the extension, and the full scripts/-relative path for a sub-folder raw script). A ref that points to nothing is a hard sync error with a line number, and it drops that definition.

Script variables

The environment variables that a Script's body references, extracted at sync time per run type (bash/sh, perl, powershell, python; ansible and terraform are not yet analyzed). They show read-only as status chips (✓ provided, ● required and missing, ○ optional with a default) in Scripts, the Run dialog, and the Composer. They are advisory; the scope's Env Vars are not auto-injected. See §8.4.

Scope

A named target environment: a host list plus its supported run types, with an optional Ansible inventory and a bound agency. Managed on its own top-level Scopes page (User Manual §11). Capability is advisory. A scope rename does not cascade. A job's scope is validated only at run time.

Scope grant (allowed scopes)

The set of scopes an identity can reach, tri-state and fail-closed: * = unrestricted; a named set = exactly those; empty = zero scope access. Reach comes from the agency (or All scopes) named on each access grant, expanded through the agency's scopes at login. No grant means no access, and a scope-restricted admin stays restricted.

Source-of-truth rule

Git is truth for definitions. The database is truth for state and operator config. Where they overlap, Git wins on read. The one exception is the per-instance disabled-workflow control.

SSH executor (in-app)

The opt-in in-process worker pool that runs bash, perl, powershell, and python over SSH, direct or through a bastion. Off by default. CRONOMICON_SSH_EXECUTOR_ENABLED turns it on. Default concurrency: 4.

SSH key credential

A first-class, typed credential (the ssh_credentials entity) for the private keys that Cronomicon uses for automated SSH. It gets the same envelope encryption as stored secrets, and validation on save (the key type, the SHA256 fingerprint, and a copyable authorized_keys public line are derived; keys with a passphrase are rejected). Managed in Env Vars → SSH Keys and attached to a host or bastion with a picker that writes authCredentialId. It is coequal with the authKeyEnvVar name path, not a replacement for it. The API never returns private material, and the material goes into the log-redaction dictionary.

Stored / Vault secret

Stored: the value is encrypted at rest with envelope encryption, and you can reveal it (audited). Vault: only a path#field reference is stored. The value lives in HashiCorp Vault, and the server gets it live with the configured Vault auth method (AppRole or a token; see §6.4). You cannot reveal it in Cronomicon.

Target host (job pin)

The optional single host that a job definition pins itself to (jobs.target_host). Empty means: fan out on each host in the effective scope. All four run paths carry the pin: manual trigger, cron fire, workflow step (with retries and skip records), and the ansible manifest. Before, only the manual trigger applied it, and the other three enqueued NULL, which target resolution reads as fan-out. The pin wins fully over a per-run host subset, and it folds into an ansible --limit unless a raw ansibleLimit passthrough is given. A pin with an ansible pattern metacharacter is refused, not dropped (422 at Compose and at trigger; 409 at the manifest; Git sync warns and still writes). See §4.3.

Target group / --limit

Inventory targeting in the Run dialog. targetGroups[] selects parsed inventory groups. One resolver expands the members for the SSH executor and passes the group NAMES to a runner as ansible --limit, so both executors get the identical set (no drift). ansibleLimit is a raw --limit passthrough for patterns that the projection cannot model (ansible/runner only; 422 on the SSH executor). Group targeting is rejected when the scope's projection is unavailable. See §11.

Trace ID

A backend-minted UUIDv7 (a unique identifier that sorts by time) that identifies one run or workflow run. It shows in middle-ellipsis form (3c4d-74f7…d5474); the full value shows on hover and in the drill-down.

Trusted Header SSO

The default auth mode. A reverse proxy runs forward-auth against your identity provider and injects the Remote-User/-Email/-Name/-Groups headers, which Cronomicon trusts per request. There is no app session cookie.

CRONOMICON_TRUSTED_PROXIES

The CIDR/IP allowlist of the reverse-proxy peer whose Remote-* headers Cronomicon will trust: the full trusted-header model. Fail-closed: empty means the server does not boot. It must be the IP that Cronomicon sees as the connecting peer.

Used by / blast radius

The reverse index on Scripts and Schedules. It shows how many, and which, jobs and workflows reference them, so you can judge the effect of a change or a delete.

VACUUM INTO

The SQLite command used for nightly backups: a consistent snapshot that opens fully, taken without a lock on live traffic (WAL mode), and optionally uploaded to S3.

11 Appendices

Appendix A — Cron reference

5-field sequence: minute(0-59) hour(0-23) day-of-month(1-31) month(1-12) day-of-week(0-6, 0=Sun). The backend also accepts 6-field seconds-first cron (sec min hour dom month dow). Available tokens: *, */n, a-b, single values, and comma lists. All cron evaluates in the effective application timezone: the stored Timezone setting, if set and loadable, or the host TZ. The UI shows each timestamp in the same zone (GET /settings/general → appTimezone).

Appendix B — API & endpoint map (for reference)

Unauthenticated surface: /healthz, /readyz, /version, /metrics (internal-only), /auth/providers. In oidc mode, also /login & /callback. /webhooks/gitlab is token-gated.

Appendix C — Key environment variables

Precedence: an CRONOMICON_* env var, when set, overrides the matching DB-backed Setting. Most settings resolve one time, at startup. The exceptions are the retention windows (read again at the top of each nightly sweep) and the run-log directory, which points each log writer to the new location immediately on save (no restart). Build-time args (VERSION, COMMIT, BUILD_DATE) show at /version.

Appendix D — Status & color vocabulary

One function produces these labels, and a build-time guard fails if a status word is written by hand anywhere else, so the printed text cannot drift between screens. Wire values are the API enum; only the printed text is mapped.

Status reasons that can show next to a failed run: killed (an operator stop, left unclassified), runner_lost (a runner reaped mid-run; the amber “Lost” badge), executor_lost (an in-app SSH run that a crash or restart left running, reconciled by the startup sweep or the age reaper; a plain Failed badge with the reason, not the amber Lost badge), orchestrator_lost (a workflow run, and its queued and running child runs, that the workflow orphan reaper reclaimed after a mid-workflow restart or the 24h stale limit; a plain Failed badge), drain_timeout (force-killed at the drain deadline), vault_unreachable, log_stream_lost (a runner's final, non-partial log upload arrived without a parseable trailing envelope; mid-run X-Log-Partial: 1 chunks are exempt by design, §7.5), cronomicon: job timed out.

Trigger kinds (runs.trigger_kind / workflow_runs.trigger_kind): manual (a click), scheduled (a cron/interval/one-shot fire), workflow (a workflow step, including a sub-workflow child), reaction, and webhook — which is deliberately overloaded three ways: a GitLab webhook, a service-account token trigger (actor svc:<name>), and a file arrival. Filtering History on webhook returns all three; the actor and the run's environment (CRONOMICON_WATCH_*) tell them apart.