Substrate readiness — capability gaps that stop the cluster serving.
curl --request GET \ --url https://stratum.example.com/v1/admin/substrate/readiness \ --header 'Authorization: Bearer <token>'Bootstrap-admin endpoint reporting what this cluster cannot currently do, as distinct from what is broken.
A cluster can be entirely healthy — every service green, every node up and still be unable to create an instance, because a required Day-2 step has not been completed. The most common case is storage: on a Ceph-only install the installer deliberately provisions no local LVM and seeds no storage classes, because pools and classes are an operator action after the Ceph cluster is formed. That is correct behaviour and it means a brand-new cluster has no usable storage until someone finishes the job.
Without this endpoint that state is invisible until the first instance create fails, with an error at the point of use rather than a hint on the substrate page.
Findings are derived from CURRENT state on every request — never accumulated — so one clears as soon as its condition is resolved.
ready is false only when a blocking finding is present. Warnings do
not make a working cluster report itself unusable.
Faults belong elsewhere: a service that is down appears in
/v1/admin/substrate/services, and a metric out of range raises a health
alarm..
Authorizations
Section titled “Authorizations”Responses
Section titled “Responses”Readiness report for the cluster.
object
False when any CORE finding (one with no service) is blocking, i.e. a
core operation cannot be performed at all. Warnings do not clear this
flag, and neither does a service-scoped finding: “ready for instances,
not for S3” is reported in services.
Capability gaps, most severe first and stable within a severity so a polling UI does not reshuffle the list.
object
Stable identifier for the kind of gap. Clients key off this for icons and deep links, and operators may script against it.
blocking — a core operation cannot be performed at all.
warning — it works, but a foreseeable case will fail.
info — nothing is wrong: a deliberate state worth knowing about, such
as a node installed offline keeping time on its own clock
(TimeSourceLocalClock). Clients should not render it as a problem.
One-line statement of what is missing.
What this prevents, phrased in terms of what the operator was trying to do rather than the internal object that is absent. May span several lines (the path-MTU findings list each failing path on its own line).
The next action to take. Every finding carries one — a finding without a remediation is just a complaint.
The specific object involved (e.g. a storage class name), when the finding is about one. Absent for cluster-wide gaps.
The service the finding is about (s3, db, k8s, a service
endpoint’s key),. Absent for a core finding, which affects
the cluster’s ability to run anything. A service-scoped blocking
finding makes that service not ready (see services) without making
the cluster not ready.
The code of the core finding that has to be resolved before this one
can be acted on at all, e.g. StorageAwaitsCeph on an
operator recipe that cannot be built before the cluster has a storage
class. The finding still counts (its service is not ready); the
service-not-ready alarm leaves it out. Absent when it can be acted on
now.
The single API call that resolves the finding, when one exists.
object
The facts of a StoragePathMTU or OverlayPathMTU finding in machine
form, for a console to draw its table and its fix button from.
object
The MTU the network is set to, as the failing interfaces run it.
The least any failing path carries. Absent when a path has not been measured (the host-helper measures only a path that failed, and not more often than every few minutes).
The failing paths, one per sender and peer.
object
The sending node’s id.
Its name, when it has one.
The peer’s node id.
The peer’s address on the network.
The sending interface.
The sending interface’s MTU.
What the path carries. Absent when not measured.
The way out the platform can do itself: lower the network’s MTU to what
every failing path carries, with PUT /v1/admin/cluster/networks/{network}/mtu
and {"mtu": mtu}. Absent when it is not offered (see noFixReason):
the path has to be raised.
object
What to set, the least any failing path carries.
The lowest MTU offered for the network (overlay 1558, storage 1500).
Why no fix is offered. Absent when fix is present.
Per-service rollup of the service-scoped findings: a
service is not ready when any of its findings is blocking. Services
with no findings are absent, except a service that is switched off or
not on this install profile: it is listed with
disabled: true, no findings, and ready: true (nothing blocks it, it
is off on purpose). A console shows it as off, not as a gap.
object
object
The codes of the service’s findings, most severe first.
The service is switched off or not on this install profile.
Why a disabled service is off.
What keeps the service not ready: one entry per blocking finding
code, with the subjects it was raised for (e.g. the recipe names).
The CLI’s service line and the installer report’s
serviceBlocking show exactly this. Absent when ready.
object
Checks whose input could not be read on this evaluation. Their findings are absent because they are unknown, not because the condition is fine; a short list with entries here is not a clean bill of health.
The cluster’s first convergence after install. From the control
plane’s first start until every substrate service and every reconciler
expected on this install has reported a first healthy pass, or until
deadline, the cluster is settling: warning findings move to held and
firing alerts are held in the admin alert list. A console shows “Starting”
with ready of total instead of errors while the phase is settling.
An upgrade starts it again from its last restart or reboot (reason: upgrade).
Derived from current state on every request; nothing is stored. After the
deadline the phase is ready or degraded from the same inputs.
object
settling: not everything has converged and the deadline has not
passed. ready: every item has converged. degraded: the deadline
passed and something has not converged; pending names it.
Items that have converged.
Items expected on this install.
The items not converged yet, services first, then reconcilers.
One substrate service on one node, or one reconciler, that has not converged.
object
The node, for a service.
The item’s current health in its own vocabulary: a service’s
healthy|unhealthy|unknown, a reconciler’s fleet health
(never_ran, failed, unknown,…).
The control plane’s first start on this cluster, or the last restart or reboot an upgrade caused when that is later.
When settling ends whether or not everything converged.
Why the cluster settles from since.
Inputs that could not be read. The phase is never ready while one is missing.
Warning findings raised while the cluster settles (settling.phase is
settling). They are not in findings, and do not count in
services. Findings are recomputed on every request, so one still
present when settling ends moves back to findings. Blocking findings
are never held. A console shows these, if at all, as part of the
startup progress rather than as problems.
object
Stable identifier for the kind of gap. Clients key off this for icons and deep links, and operators may script against it.
blocking — a core operation cannot be performed at all.
warning — it works, but a foreseeable case will fail.
info — nothing is wrong: a deliberate state worth knowing about, such
as a node installed offline keeping time on its own clock
(TimeSourceLocalClock). Clients should not render it as a problem.
One-line statement of what is missing.
What this prevents, phrased in terms of what the operator was trying to do rather than the internal object that is absent. May span several lines (the path-MTU findings list each failing path on its own line).
The next action to take. Every finding carries one — a finding without a remediation is just a complaint.
The specific object involved (e.g. a storage class name), when the finding is about one. Absent for cluster-wide gaps.
The service the finding is about (s3, db, k8s, a service
endpoint’s key),. Absent for a core finding, which affects
the cluster’s ability to run anything. A service-scoped blocking
finding makes that service not ready (see services) without making
the cluster not ready.
The code of the core finding that has to be resolved before this one
can be acted on at all, e.g. StorageAwaitsCeph on an
operator recipe that cannot be built before the cluster has a storage
class. The finding still counts (its service is not ready); the
service-not-ready alarm leaves it out. Absent when it can be acted on
now.
The single API call that resolves the finding, when one exists.
object
The facts of a StoragePathMTU or OverlayPathMTU finding in machine
form, for a console to draw its table and its fix button from.
object
The MTU the network is set to, as the failing interfaces run it.
The least any failing path carries. Absent when a path has not been measured (the host-helper measures only a path that failed, and not more often than every few minutes).
The failing paths, one per sender and peer.
object
The sending node’s id.
Its name, when it has one.
The peer’s node id.
The peer’s address on the network.
The sending interface.
The sending interface’s MTU.
What the path carries. Absent when not measured.
The way out the platform can do itself: lower the network’s MTU to what
every failing path carries, with PUT /v1/admin/cluster/networks/{network}/mtu
and {"mtu": mtu}. Absent when it is not offered (see noFixReason):
the path has to be raised.
object
What to set, the least any failing path carries.
The lowest MTU offered for the network (overlay 1558, storage 1500).
Why no fix is offered. Absent when fix is present.
Example
{ "findings": [ { "code": "NoStorageClass", "severity": "blocking", "title": "No storage class is defined", "subject": "replicated", "service": "s3", "waitsFor": "StorageAwaitsCeph", "fix": { "method": "POST", "path": "/v1/admin/storage/ceph/rgw" }, "pathMtu": { "network": "storage", "mtu": 9000, "carriedMtu": 1500, "paths": [ { "address": "192.0.2.112", "interface": "vlan90" } ], "fix": { "action": "set-network-mtu", "network": "storage", "mtu": 1500 } } } ], "settling": { "phase": "settling", "pending": [ { "kind": "service" } ], "reason": "install" }, "held": [ { "code": "NoStorageClass", "severity": "blocking", "title": "No storage class is defined", "subject": "replicated", "service": "s3", "waitsFor": "StorageAwaitsCeph", "fix": { "method": "POST", "path": "/v1/admin/storage/ceph/rgw" }, "pathMtu": { "network": "storage", "mtu": 9000, "carriedMtu": 1500, "paths": [ { "address": "192.0.2.112", "interface": "vlan90" } ], "fix": { "action": "set-network-mtu", "network": "storage", "mtu": 1500 } } } ]}Authenticated but not authorized.
object
The findings of a rejected IAM policy document (custom and resource policies).
One finding of a rejected IAM policy document. A rejection lists every finding, so the author fixes the document in one round trip.
object
The finding, e.g. INVALID_DOCUMENT, UNKNOWN_ACTION, ACTION_NOT_APPLICABLE.
Index of the statement at fault; -1 for the document as a whole.
The offending action, where one is at fault (ACTION_NOT_APPLICABLE).
Field errors, for codes that carry them: each problem at its field
(see FieldError). Documented on the responses that set it, e.g.
approveNodeJoin’s INVALID_INPUT and JOIN_APPROVAL_REFUSED.
Absent otherwise.
One field’s problem, the stage-2 installer API’s shape.
field is the field’s path, message what is wrong with it.
object
The field’s path, e.g. links[0].interface, disks[1].confirm,
mgmt.address or role; empty for the whole request.
Structured facts about the refusal, for codes that carry them. The
shape is fixed per code and documented on the response that answers
it, e.g. IMAGE_POOL_MISMATCH (see ImagePoolMismatchDetails) and
IMAGE_IN_USE. Absent for codes that carry none.
object
Example
{ "issues": [ { "code": "ACTION_NOT_APPLICABLE", "action": "compute:CreateVpc" } ]}