Per-reconciler status snapshots.
curl --request GET \ --url https://stratum.example.com/v1/observability/admin/reconcilers \ --header 'Authorization: Bearer <token>'Cedar action observability:ReadSubstrate. Returns the
in-memory tick state of every registered reconciler — the
surface backing the operator UI’s “Reconcilers” panel.
Each item reports: last tick timestamp + outcome + duration,
per-action counts, next tick ETA, interval, dry-run flag, and
lock status. Items are sorted by reconciler name.
ships the 4 load-bearing reconcilers: substrate-truth hypervisor / storage / network reconcilers plus the Platform Health pack reconciler. extends this to the 3 observability sampler/runners + adds a per-reconciler recent audit-event tail.
503 means the registry isn’t wired yet (early boot before all reconcilers register).
Authorizations
Section titled “Authorizations”Responses
Section titled “Responses”Reconciler snapshot list.
Response envelope for GET /v1/observability/admin/reconcilers.
object
Per-reconciler snapshot returned by GET /v1/observability/admin/reconcilers. See.
object
Stable reconciler handle. Examples - hypervisor, storage, network, platform-health.
Substrate-reconciler — full safety stack (advisory lock + confirmation + circuit breaker + dry-run). pack-reconciler — Platform Health pack. sampler — stateless polling worker (no lock, no circuit breaker).
Wall-clock timestamp of the last completed tick. Absent until the first tick fires.
Wall-clock timestamp of the last tick that actually EXECUTED the reconciler, as opposed to lastTickAt, which advances on every tick including ones that did no work - skipped on a peer’s leader lock, or gated on a readiness dependency. Absent means this node has never run it, in which case health is unknown. When lastRanAt lags far behind lastTickAt the health is real but STALE - it describes that older pass, not the state of the world now. A follower’s leader-scoped reconciler sits in that state indefinitely and legitimately so.
Categorical outcome of the last tick. Drives the UI status-pill color. waiting_on_dependency — held during a restore/bootstrap bring-up until a readiness-DAG dependency converges. disabled — the reconciler’s service is switched off or not on this install profile: the pass did nothing, and no alarm is raised. interrupted — the pass was cut short because the process is shutting down. Deliberately not an error: the reconciler was asked to stop, so the previous health is carried forward and lastRanAt does not advance.
Wall-clock duration of the last tick in milliseconds. Zero when lastTickAt is absent.
Per-action counts for the last tick (e.g. orphans_observed, would_reap, reaped). Always present; empty when no actions counted.
object
Action name (e.g. orphans_observed, would_reap, reaped, installed, replaced, noop).
How many times this action fired in the last tick.
Error message from the last tick if it errored. Empty when lastTickOutcome != error.
Projected timestamp of the next tick (lastTickAt + intervalSeconds). Absent for on-demand reconcilers.
Periodic cadence. Zero for on-demand reconcilers.
True when the reconciler is in dry-run mode.
Advisory-lock state at snapshot time. Samplers report unused.
Framework-derived health rollup, the at-a-glance operator signal. converged - last pass reached desired state with no drift; degraded - progressing but with drift, a tripped breaker, or items in backoff; failed - last pass errored (incl. a recovered panic); restoring - in restore/bootstrap mode. never_ran - no pass has ever EXECUTED this reconciler on this node, so there is nothing to report. The common case is a leader-scoped reconciler on a follower - it ticks on schedule but every pass skips on the peer’s advisory lock, so it has verified nothing. Such a reconciler previously reported converged, a claim about a pass that never ran. It is NOT an error state: nothing is known to be broken and nothing is known to be right, so render it neutrally. Pairs with lastRanAt, which is null in exactly this state. Note the deliberate contrast with lastTickOutcome: a follower reads lastTickOutcome=skipped_lock_held (it DID tick) alongside health=never_ran (it has never run), and that distinction is the point. ALWAYS present - a snapshotter that computes no rollup is normalized to never_ran rather than omitted, so a consumer never has to tell a missing field apart from a bug.
Count of items observed diverged from desired on the last pass — the signal a DR drill asserts on. Zero means fully converged. Zero/absent for snapshotters without a diff notion.
Count of items waiting on their owner rather than the platform (drift reason blocked_on_owner,), e.g. a tenant’s bucket delete parked on a non-empty bucket. Not included in drift and never degrades health. Zero for snapshotters without the notion.
Why a disabled reconciler is not running, in the service registry’s words - “not available on the single-node install profile” or “switched off by an operator”. Present only while health is disabled.
Example
{ "items": [ { "kind": "substrate-reconciler", "lastTickOutcome": "ok", "lockStatus": "held", "health": "converged" } ]}Upstream temporarily unavailable (driver / scheduler / dependency).
object
The findings of a rejected IAM policy document (custom and resource policies).
One finding of a rejected IAM policy document. A rejection lists every finding, so the author fixes the document in one round trip.
object
The finding, e.g. INVALID_DOCUMENT, UNKNOWN_ACTION, ACTION_NOT_APPLICABLE.
Index of the statement at fault; -1 for the document as a whole.
The offending action, where one is at fault (ACTION_NOT_APPLICABLE).
Field errors, for codes that carry them: each problem at its field
(see FieldError). Documented on the responses that set it, e.g.
approveNodeJoin’s INVALID_INPUT and JOIN_APPROVAL_REFUSED.
Absent otherwise.
One field’s problem, the stage-2 installer API’s shape.
field is the field’s path, message what is wrong with it.
object
The field’s path, e.g. links[0].interface, disks[1].confirm,
mgmt.address or role; empty for the whole request.
Structured facts about the refusal, for codes that carry them. The
shape is fixed per code and documented on the response that answers
it, e.g. IMAGE_POOL_MISMATCH (see ImagePoolMismatchDetails) and
IMAGE_IN_USE. Absent for codes that carry none.
object
Example
{ "issues": [ { "code": "ACTION_NOT_APPLICABLE", "action": "compute:CreateVpc" } ]}