Reconciler status across the whole cluster.

GET

/v1/observability/admin/reconcilers/fleet

Shell / cURL

curl --request GET \
  --url https://stratum.example.com/v1/observability/admin/reconcilers/fleet \
  --header 'Authorization: Bearer <token>'

Cedar action observability:ReadSubstrate. One entry per reconciler, merged from every node’s self-reported row.

Prefer this over GET /observability/admin/reconcilers for any “is this reconciler working?” question. That endpoint reports the IN-MEMORY state of whichever control plane served the request and does not say which node that was, so on a multi-node cluster it gives a different answer depending on where it lands.

Concretely, the per-node endpoint reports never_ran for a leader-gated reconciler on every node except the one holding its lock — correct, and unreadable without knowing which node is leading. This endpoint answers that directly with leaderNodeId.

Rollup rules, which differ by shape because the right answer differs:

leaderGated: true — health is the LEADER’s health. A follower skipped the pass, so its never_ran says nothing about the work. When no node claims to be leading, health is unknown, never converged: the work is happening nowhere.
leaderGated: false (per-node) — health is the WORST across nodes, because one broken node is a broken reconciler on that node, and all rows are returned since each is independently meaningful.
Either shape — when every fresh row says disabled, health is disabled: the reconciler’s service is switched off or not on this install profile. It does no work, takes no lock (so no node leads it), and raises no alarm. It is not a fault.

Staleness is a verdict, not a gap. A row that has not refreshed within its freshness window (three tick intervals, floored at two minutes) is marked stale, excluded from the rollup, and sets stale on the entry. If NO node has reported recently, health is unknown — absence of evidence must not read as agreement, which is how “one node has been down an hour” used to hide behind “converged everywhere”.

503 means the cluster-status store isn’t wired (a binary with no Postgres). It deliberately does NOT fall back to the serving node’s memory, because silently answering for one node is the failure this endpoint exists to fix.

Authorizations

bearerAuth

Responses

200

Cluster-wide reconciler status.

application/json

Envelope for the cluster-wide reconciler view.

object

items

required

Array<object>

One reconciler as the CLUSTER sees it, merged across every node’s self-reported row.

object

name

required

string

kind

required

string

Allowed values: substrate-reconciler pack-reconciler sampler

leaderGated

required

Whether this reconciler elects a leader. It changes what the rollup means: a leader-gated reconciler has ONE authoritative view (the node running it), while a per-node one legitimately has one per node.

boolean

leaderNodeId

The node that ran the most recent pass, for a leader-gated reconciler. The leader lock is held per pass (peers take it in turn,), so several nodes may run within one interval; this is the freshest. Empty when nobody is currently leading — worth surfacing, because it means the work is not happening anywhere. This replaces the per-node LOCK indicator, which operators read as “a lock is stuck” when it actually meant “this node is the one doing the work”.

string

health

required

The rollup. For leader-gated, the worst of every pass within one interval of the leader’s (and of any node whose tick errored before it could run), so a failing node is not hidden by a good pass elsewhere. For per-node, the worst across nodes. unknown when no node has reported inside its freshness window, or when a leader-gated reconciler has no leader. disabled when every fresh node reports the reconciler’s service switched off or not on this install profile: it does no work and raises no alarm.

string

Allowed values: converged degraded failed restoring never_ran disabled unknown

stale

required

True when any node’s row is beyond its freshness window, so the rollup cannot claim full agreement.

boolean

reason

Explains a health the numbers alone do not. Two cases today. A disabled reconciler says why its service is not running here: “not available on the single-node install profile” or “switched off by an operator”. The reason is the service registry’s decision, which every control plane reads from the same rows, so the serving node states it. An unknown for a reconciler this control plane has registered but which no node has ever published status for - which means its manager is missing the status sink, not that anything is converging badly. Such a reconciler is reported with an empty nodes list rather than omitted from the response. A view whose promise is “this is the cluster answer” must not quietly answer a smaller question: it happened with substrate-bridge, which runs on a second Pool-less manager and simply was not in the list. Note the difference from stale: stale means the rows are old, this means they have never existed, and the two call for different responses.

string

nodes

required

Every reporting node’s row, leader first. Empty means no node has ever reported - see reason.

Array<object>

One node’s contribution to a reconciler’s cluster rollup.

object

nodeId

required

string

health

required

This node’s own rollup. never_ran on a follower of a leader-gated reconciler is CORRECT and uninformative — it skipped the pass.

string

lastRanAt

Last pass that actually executed on this node. Absent if none has.

string format: date-time

isLeader

required

Whether this node ran the last pass.

boolean

drift

required

integer

blocked

Items waiting on their owner rather than the platform (drift reason blocked_on_owner,). Not part of drift and never degrades health; shown so a node reading converged with drift 0 does not hide them.

integer

reportedAt

required

When this node last refreshed its row.

string format: date-time

stale

required

The row has not refreshed within the freshness window. Its health is kept for context but is excluded from the rollup — a node that stopped reporting is evidence of nothing.

boolean

Example

{
  "items": [
    {
      "kind": "substrate-reconciler",
      "health": "converged"
    }
  ]
}

503

Upstream temporarily unavailable (driver / scheduler / dependency).

application/problem+json

object

type

required

string format: uri

title

required

string

status

required

integer

>= 400 <= 599

detail

string

instance

string format: uri-reference

requestId

required

string

code

required

string

issues

The findings of a rejected IAM policy document (custom and resource policies).

Array<object>

One finding of a rejected IAM policy document. A rejection lists every finding, so the author fixes the document in one round trip.

object

code

required

The finding, e.g. INVALID_DOCUMENT, UNKNOWN_ACTION, ACTION_NOT_APPLICABLE.

string

statement

required

Index of the statement at fault; -1 for the document as a whole.

integer

>= -1

detail

required

string

action

The offending action, where one is at fault (ACTION_NOT_APPLICABLE).

string

errors

Field errors, for codes that carry them: each problem at its field (see FieldError). Documented on the responses that set it, e.g. approveNodeJoin’s INVALID_INPUT and JOIN_APPROVAL_REFUSED. Absent otherwise.

Array<object>

One field’s problem, the stage-2 installer API’s shape. field is the field’s path, message what is wrong with it.

object

field

required

The field’s path, e.g. links[0].interface, disks[1].confirm, mgmt.address or role; empty for the whole request.

string

message

required

string

details

Structured facts about the refusal, for codes that carry them. The shape is fixed per code and documented on the response that answers it, e.g. IMAGE_POOL_MISMATCH (see ImagePoolMismatchDetails) and IMAGE_IN_USE. Absent for codes that carry none.

object

key

additional properties

any

Example

{
  "issues": [
    {
      "code": "ACTION_NOT_APPLICABLE",
      "action": "compute:CreateVpc"
    }
  ]
}