List substrate quadlet services + live systemd/podman state.

GET

/v1/admin/substrate/services

Shell / cURL

curl --request GET \
  --url https://stratum.example.com/v1/admin/substrate/services \
  --header 'Authorization: Bearer <token>'

Bootstrap-admin endpoint returning, per substrate quadlet (postgres, openbao, ovn-central, stratum-controlplane, victoriametrics, vmagent, victorialogs, otel-collector, tempo, vmalert, alertmanager, node-exporter, stratum-ui,…), the runtime state systemd + podman report: ActiveState, SubState, UnitFileState, NRestarts, last error snippet, the container’s image ref + restart count.

Lets operators answer “is anything crash-looping on this node?” from the product without SSH’ing for systemctl status.

name is the service’s log-pipeline identity, not its systemd unit name: a quadlet drops .container, a native unit drops .service (ovn-controller, not ovn-controller.service), and any other unit type keeps its suffix (podman.socket). It can be passed straight to the per-service logs endpoint below — before it could not, and doing so returned an empty 200 for every native unit.

The controlplane shells out to systemctl show + podman inspect against the host’s sockets, which are bind-mounted into the stratum-controlplane container. Single-node at v1; multi-node aggregation is post-v1 work..

Authorizations

bearerAuth

Responses

200

Per-quadlet runtime status.

application/json

object

items

required

Array<object>

object

name

required

Service base name. For quadlets, the .container template stripped (e.g. postgres); for native systemd units the unit name; for OVS bridges the bridge name (e.g. br-int); for cephadm daemons a normalized group name (ceph-mon, ceph-mgr, ceph-crash, ceph-osd.<id>).

One synthetic name exists: ceph-foreign-fsid, raised on a node that carries Ceph daemon units belonging to a cluster other than the one it is a member of. It is always unhealthy, at most one per node, and its lastError names the offending fsids. Daemons of a foreign fsid are NOT reported under their own group names, so the live daemon keeps ceph-mon / ceph-mgr rather than being displaced by a ghost.

string

node

Stratum node id the row was sampled on.

string

state

required

Mirrors systemd’s ActiveState.

string

Allowed values: active activating reloading deactivating inactive failed unknown

subState

Mirrors systemd’s SubState; sub-classifies state (e.g. running, exited, failed).

string

health

required

Operator-facing roll-up. active -> healthy; failed / inactive / deactivating -> unhealthy; reloading / parse-failure -> unknown.

activating is unknown — a unit on its way up — EXCEPT when subState is auto-restart, which is unhealthy. systemd only enters auto-restart after a start has failed, so it is a crash loop rather than a cold start; reporting it as unknown is how a ghost daemon looped 931 times over 13 hours without failing any check.

string

Allowed values: healthy unhealthy unknown

image

Container image ref the host’s podman runtime is using, read through the podman API socket. Empty when there is no container right now (a stopped unit, or one between restarts), or when podman could not be asked, in which case lastError says so.

string

restartCount

Max of systemd’s NRestarts and the podman runtime restart count. Captures both unit-level restarts (systemd) and in-unit container recycles (podman) — both indicate instability even though systemd considers the unit active in the second case.

integer

activeSince

Systemd’s ActiveEnterTimestamp. Empty when the unit never entered active.

string format: date-time

lastError

Short failure summary surfaced from systemd’s Result field (e.g. systemd result: exit-code). Otherwise, for a quadlet whose container podman could not be asked about, podman inspect <container>: <reason>: its image and podman restart count are then unknown, not absent, and state and health still come from systemd. Empty when there is nothing to report.

string

unitFileState

Systemd’s unit-file enablement state (enabled, disabled, static).

string

socketActivated

True when the row’s health is rolled up from a .socket sibling instead of its own .service ActiveState. UIs should render the pill as “idle / active” for these rows: inactive is the steady state for a socket-activated daemon (the.socket is listening for the first client connect).

boolean

backingDaemons

Modular-libvirt drill-down: per-daemon state behind a socket-activated front door (e.g. libvirtd’s virtproxyd / virtqemud / virtnetworkd / virtstoraged). UIs hide these in steady state; expanding the row surfaces them during incident debugging. Does not affect the parent row’s health by design — a failing virtnetworkd doesn’t render libvirtd as unhealthy because the operator workflow that depends on it might be unrelated.

Array<object>

object

name

required

Daemon unit prefix (the corresponding units are <name>.socket and <name>.service).

string

socketState

required

ActiveState of the daemon’s .socket. active is healthy; anything else is broken.

string

Allowed values: active activating reloading deactivating inactive failed unknown

serviceState

required

ActiveState of the daemon’s .service. Informational only — inactive is the steady-state for a socket- activated daemon (it’s waiting for the first connect).

string

Allowed values: active activating reloading deactivating inactive failed unknown

sampledAt

When this row was last sampled on its node. The endpoint aggregates per-node samples across the fleet, so a node that stopped answering keeps its last sample with an ageing sampledAt; UIs should flag rows older than a couple of poll intervals as stale (the node may be down). Absent only on the live-local fallback path (no sampler store wired).

string format: date-time

Example

{
  "items": [
    {
      "name": "postgres",
      "state": "active",
      "health": "healthy",
      "backingDaemons": [
        {
          "name": "virtqemud",
          "socketState": "active",
          "serviceState": "active"
        }
      ]
    }
  ]
}

403

Authenticated but not authorized.

application/problem+json

object

type

required

string format: uri

title

required

string

status

required

integer

>= 400 <= 599

detail

string

instance

string format: uri-reference

requestId

required

string

code

required

string

issues

The findings of a rejected IAM policy document (custom and resource policies).

Array<object>

One finding of a rejected IAM policy document. A rejection lists every finding, so the author fixes the document in one round trip.

object

code

required

The finding, e.g. INVALID_DOCUMENT, UNKNOWN_ACTION, ACTION_NOT_APPLICABLE.

string

statement

required

Index of the statement at fault; -1 for the document as a whole.

integer

>= -1

detail

required

string

action

The offending action, where one is at fault (ACTION_NOT_APPLICABLE).

string

errors

Field errors, for codes that carry them: each problem at its field (see FieldError). Documented on the responses that set it, e.g. approveNodeJoin’s INVALID_INPUT and JOIN_APPROVAL_REFUSED. Absent otherwise.

Array<object>

One field’s problem, the stage-2 installer API’s shape. field is the field’s path, message what is wrong with it.

object

field

required

The field’s path, e.g. links[0].interface, disks[1].confirm, mgmt.address or role; empty for the whole request.

string

message

required

string

details

Structured facts about the refusal, for codes that carry them. The shape is fixed per code and documented on the response that answers it, e.g. IMAGE_POOL_MISMATCH (see ImagePoolMismatchDetails) and IMAGE_IN_USE. Absent for codes that carry none.

object

key

additional properties

any

Example

{
  "issues": [
    {
      "code": "ACTION_NOT_APPLICABLE",
      "action": "compute:CreateVpc"
    }
  ]
}