Substrate services

Status

Console: Admin › Substrate › Substrate Services. One row per service, across every node, refreshed every 15 seconds: state, health, how long it has been active, how often it restarted, and its last error. Select a service for its per-node health and its recent logs.

Substrate services with restart counts and per-node health.

Some services are healthy while inactive: socket-activated ones start on the first connection, and timer-driven ones run, exit and wait for the next tick.

The same rollup, with the platform’s other health checks, from the CLI:

stratumctl admin selfcheck        # one pass/fail report; exits non-zero on a failure
stratumctl admin doctor           # component versions against what this release was tested with

Restarting a service

Restarts are done on the node, with systemd:

sudo systemctl status stratum-controlplane
sudo systemctl restart stratum-controlplane
sudo journalctl -u stratum-controlplane -b

A control-plane restart is safe: the API is briefly unavailable and instances keep running. Restart the database or the secrets store only when a finding or a runbook says so; the control plane cannot work without them.

Resource limits

Each platform service runs with a memory and CPU cap. Console: Admin › Substrate › Service resource limits lists them, tightest headroom above the observed peak first. Edit limits sets a new cap for the whole cloud or one node; each node applies it on its next pass. A cap below the service’s observed peak is refused. Some services apply a new cap only at their next restart; the page shows them as Restart pending.

Service resource limits.

Reconcilers

Reconcilers are the control loops that converge every resource toward its desired state. Console: Admin › Substrate › Reconcilers.

stratumctl admin reconcilers fleet          # every reconciler as the cloud sees it
stratumctl admin reconcilers list           # as the control plane that answered sees it
stratumctl admin reconcilers drift dns-recursor   # which items diverge, and why

Health	Means
converged	The last pass found nothing to change.
drift	Items diverge; drift names each one and why it is not converging (waiting for confirmation, backing off after an error, blocked on the tenant).
never_ran	No pass has run on this node. On a cluster, normal for a reconciler another node leads.
STALE	The health describes an older pass.

The alarms reconciler-not-converging, reconciler-items-in-backoff and reconciler-tick-storm watch them.