Per-item drift diff — which items diverged and why they are not converging.
curl --request GET \ --url https://stratum.example.com/v1/observability/admin/reconcilers/example/drift \ --header 'Authorization: Bearer <token>'Cedar action observability:ReadSubstrate. The drift-diff dump.
The list surface reports drift: 3 — three items diverged. It does not say
which three, what desired-vs-actual was, or why they are not converging.
This endpoint answers all three, from state the framework already had and
never surfaced: the set-diff, the confirmation-window streaks, the per-item
backoff and its last error, and the circuit breaker.
items describes the last executed pass (see lastRanAt) — a skipped
pass computes no set-diff, so it does not clobber the last real answer.
coverage is the part a drift diff cannot supply. stranded
delete_blocked buckets forever because that status was missing from the
reconciler’s work query, so those rows never entered the set-diff at all and
drift stayed 0. Coverage reports the full desired-state status histogram
and marks which statuses the work query actually selects, so a non-terminal
unselected status with a non-zero count and an aging oldest row is visible
as exactly what it is. Only reconcilers that report coverage populate it.
503 before the registry is wired; 404 for an unknown reconciler name.
Authorizations
Section titled “Authorizations”Parameters
Section titled “Parameters”Path Parameters
Section titled “Path Parameters”The reconciler’s stable name (e.g. network, dns, lb).
Responses
Section titled “Responses”Per-item drift report.
Per-reconciler drift-diff dump.
object
The tick these items were observed on.
Last tick that actually EXECUTED. When it differs from asOf the items below describe that older pass.
The reconciler’s health rollup, repeated so this response stands alone.
Whether this reconciler can report per-item drift at all. Distinguishes the two things an empty items otherwise conflates:
true with empty items - the last executed pass found nothing diverged.
false with empty items - this reconciler CANNOT report items, so the empty list carries no information whatsoever.
False is the common case today. Per-item capture lives in the framework’s Converge[T] primitive and only 2 of the 21 registered reconcilers use it; the rest are bespoke converge-workers that compute their own diff, and opt in by declaring themselves reporters.
A consumer MUST read this before concluding anything from an empty items list. Reading empty as converged is the exact failure this surface exists to remove.
Every item diverged on the last executed pass. Always present. Read itemsReported before concluding anything from an empty list.
One diverged item and, crucially, WHY it is not converging.
object
Resource kind the reconciler diffed (subnet, bucket, domain).
Which half of the set-diff produced this item. restore - desired, absent from the substrate. reap - in the substrate, no desired row. align - present in both, values differ.
The item’s stable identity.
Consecutive passes this item has been diverged. The number that tells a transient blip from something stuck for an hour.
The reconciler’s ConfirmationTicks - the framework waits for streak to reach this before acting, so a transient read never triggers a mutation.
Why this item did not converge on this pass.
awaiting_confirmation - streak has not reached confirmationsNeeded yet.
backoff - a previous apply failed and it is waiting out an exponential window; lastErr says what broke. This is the “converging forever and never getting there” state.
circuit_breaker - the direction’s candidate count exceeded the threshold, so the framework refused to act on any item in it.
dry_run - confirmed, would have been acted on, nothing committed.
suppressed_restore_mode / suppressed_no_reap - a reap deliberately not performed. Recorded so a fleet left in safe-mode does not look identical to one with nothing to reap.
suppressed_unconfigured - the reconciler cannot act because a prerequisite is not configured (no backup target, no credential, no endpoint). Deliberately distinct from suppressed_no_reap: that one means an operator CHOSE safe-mode, and telling someone their fleet is in safe-mode when the truth is “setup is incomplete” sends them hunting for a switch nobody set.
apply_failed - acted this pass and the apply errored; enters backoff next.
blocked_on_owner - the platform has done everything it can and the item waits on its OWNER, e.g. a tenant’s bucket delete parked because the bucket still holds objects. Counted in the reconciler’s blocked, not its drift, and never degrades health: a page for something only the tenant can clear gets muted, and then it protects nothing.
acting - confirmed and applied successfully on this pass.
Reconciler-specific reasons. A reconciler may report why an item waits in its own terms; every such reason is listed here, and a test fails the build when one is emitted that is not. cooldown - an auto scaling group’s policy-driven change waits out the group’s cooldown. process_suspended - the process that would act (Launch, Terminate, InstanceRefresh, AZRebalance) is suspended on the group. refresh_stalled - an instance refresh cannot advance: its healthy floor cannot be met at the group’s maxSize. rebalance_stalled - an AZ rebalance cannot move a member without exceeding maxSize. lb_health_suspect - every member of a load_balancer-health group reads unhealthy from the load balancer’s single vantage; replacement is held and judged on instance health. replace_budget - more members are unhealthy than may be replaced at once; they wait for earlier replacements to finish. unexpected_state / unexpected_kind / unexpected_value - the reconciler read a state, kind or value it does not know, and reports it instead of acting on it. image_mode - the node runs image mode (bootc): its /usr is the read-only stratum-os image, so the item changes only when a new stratum-os image is booted. The reconciler reports the difference and never writes. A host binary that differs from the control plane’s means the OS image and the control plane were not released together.
Last apply error for this specific item. Populated for reason=backoff and reason=apply_failed - the string that says what is actually wrong.
Rendering of the desired side, when the reconciler supplies a describer. Absent otherwise - the framework set-diffs an opaque generic type and cannot stringify it unaided.
Rendering of the actual side. Only direction=align has two sides to show.
Consecutive failed applies for this item. Populated for reason=backoff and reason=apply_failed, absent elsewhere. reason alone cannot separate “failed once, retries shortly” from “failed twelve times and is parked at the backoff cap”, and those call for opposite operator responses - wait, versus go and look at it.
How many more reconciler passes this item will be skipped for. 0 or absent means it is eligible on the next pass. Passes rather than seconds because that is the unit the framework actually measures backoff in - deterministic under test, and unaffected by a tick that runs late.
When retryAfterPasses is expected to elapse, derived from the reconciler’s CONFIGURED tick interval and anchored on the last tick. An estimate, deliberately. The framework schedules in passes, and a pass can be delayed by lock contention or a slow predecessor, so the real retry can land later than this - never earlier. Absent when the item is not in backoff or the interval is unknown.
Items dropped past driftCap, reported explicitly so a capped report is never mistaken for a complete one.
The per-pass cap on retained item detail.
Work-query coverage histogram, when the reconciler reports one.
How many desired-state rows sit in one status, and whether the reconciler’s work query SELECTS that status.
This is the shape and the one thing a per-item drift diff cannot catch: delete_blocked buckets were never re-examined because the status was missing from the work query, so those rows never entered the set-diff and drift stayed 0 while they were stranded. Only comparing the table against what the query selects surfaces it.
object
Resource kind (bucket, volume).
The desired-state status value as stored.
How many rows currently sit in that status.
Whether this reconciler’s work query picks the status up.
Whether the status is legitimately final, so “correctly not selected” is distinguishable from “silently stranded”.
The component that drives rows in this status when it is NOT this reconciler - a lifecycle worker, a different reconciler, or a synchronous service path. Absent means nothing owns it.
selected=false alone does not mean stranded, and reading it that way gives a confident wrong answer in both directions. network/vpc rows in creating are not selected by the network reconciler and are perfectly healthy - LifecycleWorker drives them. network/nat_gateway rows in creating look identical from the query’s side and are stranded forever, because the worker was never extended to NAT gateways.
So the stranded-rows signature is the full conjunction: count > 0, selected=false, terminal=false, ownedBy absent, and oldestAgeSeconds climbing. Naming the owner is what makes the other buckets quiet enough for that signal to be worth alerting on.
Age of the oldest row in this status. A non-terminal, unselected status whose oldest row keeps aging is the signal needed and lacked.
Whether this reconciler reports work-query coverage at all.
Read it before reading coverage. An empty coverage means “reports it, and every covered table is empty” when this is true, and “cannot report it” when false - and those are different answers to the only question the section exists for, “is anything stranded here?”. Without the flag an operator gets the same blank either way, which is the same absent-means-fine conflation as an empty drift item list before.
Best-effort coverage query failure. Surfaced here rather than failing the whole response.
Example
{ "items": [ { "direction": "restore", "reason": "awaiting_confirmation" } ]}Resource not found.
object
The findings of a rejected IAM policy document (custom and resource policies).
One finding of a rejected IAM policy document. A rejection lists every finding, so the author fixes the document in one round trip.
object
The finding, e.g. INVALID_DOCUMENT, UNKNOWN_ACTION, ACTION_NOT_APPLICABLE.
Index of the statement at fault; -1 for the document as a whole.
The offending action, where one is at fault (ACTION_NOT_APPLICABLE).
Field errors, for codes that carry them: each problem at its field
(see FieldError). Documented on the responses that set it, e.g.
approveNodeJoin’s INVALID_INPUT and JOIN_APPROVAL_REFUSED.
Absent otherwise.
One field’s problem, the stage-2 installer API’s shape.
field is the field’s path, message what is wrong with it.
object
The field’s path, e.g. links[0].interface, disks[1].confirm,
mgmt.address or role; empty for the whole request.
Structured facts about the refusal, for codes that carry them. The
shape is fixed per code and documented on the response that answers
it, e.g. IMAGE_POOL_MISMATCH (see ImagePoolMismatchDetails) and
IMAGE_IN_USE. Absent for codes that carry none.
object
Example
{ "issues": [ { "code": "ACTION_NOT_APPLICABLE", "action": "compute:CreateVpc" } ]}Upstream temporarily unavailable (driver / scheduler / dependency).
object
The findings of a rejected IAM policy document (custom and resource policies).
One finding of a rejected IAM policy document. A rejection lists every finding, so the author fixes the document in one round trip.
object
The finding, e.g. INVALID_DOCUMENT, UNKNOWN_ACTION, ACTION_NOT_APPLICABLE.
Index of the statement at fault; -1 for the document as a whole.
The offending action, where one is at fault (ACTION_NOT_APPLICABLE).
Field errors, for codes that carry them: each problem at its field
(see FieldError). Documented on the responses that set it, e.g.
approveNodeJoin’s INVALID_INPUT and JOIN_APPROVAL_REFUSED.
Absent otherwise.
One field’s problem, the stage-2 installer API’s shape.
field is the field’s path, message what is wrong with it.
object
The field’s path, e.g. links[0].interface, disks[1].confirm,
mgmt.address or role; empty for the whole request.
Structured facts about the refusal, for codes that carry them. The
shape is fixed per code and documented on the response that answers
it, e.g. IMAGE_POOL_MISMATCH (see ImagePoolMismatchDetails) and
IMAGE_IN_USE. Absent for codes that carry none.
object
Example
{ "issues": [ { "code": "ACTION_NOT_APPLICABLE", "action": "compute:CreateVpc" } ]}