Skip to content

stratumctl admin reconcilers

Inspect the reconciler fleet (bootstrap-admin only)

Terminal window
stratumctl admin reconcilers [flags]

Aliases: reconciler

Show which items diverged and why each is not converging

Terminal window
stratumctl admin reconcilers drift <name> [flags]

Dump one reconciler’s per-item drift.

“drift: 3” says three things diverged. This says which three, and why each one is not converging: awaiting_confirmation (with the streak, so a blip reads differently from something wedged for an hour), backoff (with the apply error that caused it), circuit_breaker, dry_run, a suppressed reap, apply_failed, blocked_on_owner (waiting on the tenant, e.g. a bucket delete parked because the bucket still holds objects; counted as blocked, not drift, and never degrades health), or acting.

COVERAGE is the other half, and the part a drift diff cannot supply. It reports the desired-state status histogram and whether the reconciler’s work query actually selects each status. A non-terminal status the query does not select, holding rows that keep aging, means those rows are reconciled by nobody — the shape, where delete_blocked buckets were stranded forever while drift read 0.

Terminal window
stratumctl admin reconcilers drift db
stratumctl admin reconcilers drift db --output json
Flag Type Default Description
-o, --output string text output format: text | json

Cluster-wide reconciler status (which node runs each one)

Terminal window
stratumctl admin reconcilers fleet [flags]

Show every reconciler as the CLUSTER sees it.

Prefer this over list for “is this reconciler working?”.

list reports the in-memory state of whichever control plane answered and does not say which node that was, so on a multi-node cluster it gives a different answer depending on where the request lands. In particular a leader-gated reconciler reads never_ran on every node except the one holding its lock — correct, and unreadable without knowing which node is leading. This view names it.

Health rolls up differently by shape, because the right answer differs. A leader-gated reconciler takes the LEADER’s health: a follower skipped the pass, so its never_ran says nothing about the work. A per-node reconciler takes the WORST across nodes, since one broken node is genuinely broken.

STALE is a verdict, not a gap. A node that stops reporting is excluded from the rollup rather than contributing its last-known-good health, and if nobody has reported recently the answer is unknown — never converged. That is what stops “one node has been down an hour” hiding behind “converged everywhere”.

Terminal window
stratumctl admin reconcilers fleet
stratumctl admin reconcilers fleet --output json
Flag Type Default Description
-o, --output string text output format: text | json

List every reconciler with health, drift, lock state and observed cadence

Terminal window
stratumctl admin reconcilers list [flags]

List the reconciler fleet as this control plane sees it.

Reconciler status is PER NODE. A reconciler shown as skipped here may simply be leader-gated — another node holds its advisory lock and is doing the work. Query all nodes before concluding anything; a symptom on one node is not a fault.

The CADENCE column is the one that catches a trigger storm: it compares the observed gap between the last two ticks against the configured interval. A reconciler firing far above its configured rate is as broken as one not firing at all, and nothing surfaced it before.

STALE marks a health that describes an older pass — the reconciler has ticked since, but those ticks did no work. never_ran means no pass has ever executed on this node, so there is nothing to report; it is not a claim that anything is wrong, and it is not a claim that anything is right.

Terminal window
stratumctl admin reconcilers list
stratumctl admin reconcilers list --output json
Flag Type Default Description
-o, --output string text output format: text | json