Trigger an on-demand reconcile pass (Reconcile now).
curl --request POST \ --url https://stratum.example.com/v1/observability/admin/reconcilers/example/reconcile \ --header 'Authorization: Bearer <token>'Cedar action observability:ReconcileSubstrate (a distinct WRITE
grant — platform-admin only; NOT covered by the p-system project-admin
ReadSubstrate carve-out, so read-only substrate operators cannot
trigger passes). Runs one immediate pass for the named reconciler via
the framework Manager and returns the resulting detail so the operator
sees the outcome. Useful during a DR bring-up or after a manual fix,
rather than waiting for the poll backstop. 404 for a name the Manager
does not manage; 503 before the Manager is wired..
Authorizations
Section titled “Authorizations”Parameters
Section titled “Parameters”Path Parameters
Section titled “Path Parameters”The reconciler’s stable name (e.g. network, dns, lb).
Responses
Section titled “Responses”The reconciler’s detail after the on-demand pass.
Richer per-reconciler view returned by GET /v1/observability/admin/reconcilers/{name}: the list snapshot plus recent tick history + circuit-breaker state.
object
Per-reconciler snapshot returned by GET /v1/observability/admin/reconcilers. See.
object
Stable reconciler handle. Examples - hypervisor, storage, network, platform-health.
Substrate-reconciler — full safety stack (advisory lock + confirmation + circuit breaker + dry-run). pack-reconciler — Platform Health pack. sampler — stateless polling worker (no lock, no circuit breaker).
Wall-clock timestamp of the last completed tick. Absent until the first tick fires.
Wall-clock timestamp of the last tick that actually EXECUTED the reconciler, as opposed to lastTickAt, which advances on every tick including ones that did no work - skipped on a peer’s leader lock, or gated on a readiness dependency. Absent means this node has never run it, in which case health is unknown. When lastRanAt lags far behind lastTickAt the health is real but STALE - it describes that older pass, not the state of the world now. A follower’s leader-scoped reconciler sits in that state indefinitely and legitimately so.
Categorical outcome of the last tick. Drives the UI status-pill color. waiting_on_dependency — held during a restore/bootstrap bring-up until a readiness-DAG dependency converges. disabled — the reconciler’s service is switched off or not on this install profile: the pass did nothing, and no alarm is raised. interrupted — the pass was cut short because the process is shutting down. Deliberately not an error: the reconciler was asked to stop, so the previous health is carried forward and lastRanAt does not advance.
Wall-clock duration of the last tick in milliseconds. Zero when lastTickAt is absent.
Per-action counts for the last tick (e.g. orphans_observed, would_reap, reaped). Always present; empty when no actions counted.
object
Action name (e.g. orphans_observed, would_reap, reaped, installed, replaced, noop).
How many times this action fired in the last tick.
Error message from the last tick if it errored. Empty when lastTickOutcome != error.
Projected timestamp of the next tick (lastTickAt + intervalSeconds). Absent for on-demand reconcilers.
Periodic cadence. Zero for on-demand reconcilers.
True when the reconciler is in dry-run mode.
Advisory-lock state at snapshot time. Samplers report unused.
Framework-derived health rollup, the at-a-glance operator signal. converged - last pass reached desired state with no drift; degraded - progressing but with drift, a tripped breaker, or items in backoff; failed - last pass errored (incl. a recovered panic); restoring - in restore/bootstrap mode. never_ran - no pass has ever EXECUTED this reconciler on this node, so there is nothing to report. The common case is a leader-scoped reconciler on a follower - it ticks on schedule but every pass skips on the peer’s advisory lock, so it has verified nothing. Such a reconciler previously reported converged, a claim about a pass that never ran. It is NOT an error state: nothing is known to be broken and nothing is known to be right, so render it neutrally. Pairs with lastRanAt, which is null in exactly this state. Note the deliberate contrast with lastTickOutcome: a follower reads lastTickOutcome=skipped_lock_held (it DID tick) alongside health=never_ran (it has never run), and that distinction is the point. ALWAYS present - a snapshotter that computes no rollup is normalized to never_ran rather than omitted, so a consumer never has to tell a missing field apart from a bug.
Count of items observed diverged from desired on the last pass — the signal a DR drill asserts on. Zero means fully converged. Zero/absent for snapshotters without a diff notion.
Count of items waiting on their owner rather than the platform (drift reason blocked_on_owner,), e.g. a tenant’s bucket delete parked on a non-empty bucket. Not included in drift and never degrades health. Zero for snapshotters without the notion.
Why a disabled reconciler is not running, in the service registry’s words - “not available on the single-node install profile” or “switched off by an operator”. Present only while health is disabled.
Recent ticks, most-recent-first (bounded ring). Empty for snapshotters that don’t record history.
One past tick in a reconciler’s history.
object
When the tick completed.
Categorical outcome of that tick.
Wall-clock duration of that tick in milliseconds.
Items observed diverged from desired on that tick.
Error message if that tick errored.
Circuit-breaker state. The framework breaker is a per-pass guard (refuse to act when a direction’s divergence count exceeds the threshold), not an open/half-open FSM — so this reports whether the LAST pass tripped it and why. Reason fields are present only when tripped.
object
True when the reconciler’s last pass tripped the breaker.
When the last trip occurred.
Divergence count that tripped the breaker.
The threshold the divergence count exceeded.
Resource kind whose divergence tripped the breaker.
Convergence direction (restore | reap | align) that tripped it.
Short human-readable explanation of the trip, e.g. “res/restore divergence count 5 exceeds circuit-breaker threshold 3”. Present only when tripped.
Example
{ "status": { "kind": "substrate-reconciler", "lastTickOutcome": "ok", "lockStatus": "held", "health": "converged" }, "history": [ { "outcome": "ok" } ]}Resource not found.
object
The findings of a rejected IAM policy document (custom and resource policies).
One finding of a rejected IAM policy document. A rejection lists every finding, so the author fixes the document in one round trip.
object
The finding, e.g. INVALID_DOCUMENT, UNKNOWN_ACTION, ACTION_NOT_APPLICABLE.
Index of the statement at fault; -1 for the document as a whole.
The offending action, where one is at fault (ACTION_NOT_APPLICABLE).
Field errors, for codes that carry them: each problem at its field
(see FieldError). Documented on the responses that set it, e.g.
approveNodeJoin’s INVALID_INPUT and JOIN_APPROVAL_REFUSED.
Absent otherwise.
One field’s problem, the stage-2 installer API’s shape.
field is the field’s path, message what is wrong with it.
object
The field’s path, e.g. links[0].interface, disks[1].confirm,
mgmt.address or role; empty for the whole request.
Structured facts about the refusal, for codes that carry them. The
shape is fixed per code and documented on the response that answers
it, e.g. IMAGE_POOL_MISMATCH (see ImagePoolMismatchDetails) and
IMAGE_IN_USE. Absent for codes that carry none.
object
Example
{ "issues": [ { "code": "ACTION_NOT_APPLICABLE", "action": "compute:CreateVpc" } ]}Upstream temporarily unavailable (driver / scheduler / dependency).
object
The findings of a rejected IAM policy document (custom and resource policies).
One finding of a rejected IAM policy document. A rejection lists every finding, so the author fixes the document in one round trip.
object
The finding, e.g. INVALID_DOCUMENT, UNKNOWN_ACTION, ACTION_NOT_APPLICABLE.
Index of the statement at fault; -1 for the document as a whole.
The offending action, where one is at fault (ACTION_NOT_APPLICABLE).
Field errors, for codes that carry them: each problem at its field
(see FieldError). Documented on the responses that set it, e.g.
approveNodeJoin’s INVALID_INPUT and JOIN_APPROVAL_REFUSED.
Absent otherwise.
One field’s problem, the stage-2 installer API’s shape.
field is the field’s path, message what is wrong with it.
object
The field’s path, e.g. links[0].interface, disks[1].confirm,
mgmt.address or role; empty for the whole request.
Structured facts about the refusal, for codes that carry them. The
shape is fixed per code and documented on the response that answers
it, e.g. IMAGE_POOL_MISMATCH (see ImagePoolMismatchDetails) and
IMAGE_IN_USE. Absent for codes that carry none.
object
Example
{ "issues": [ { "code": "ACTION_NOT_APPLICABLE", "action": "compute:CreateVpc" } ]}