Skip to content
StratumStratum Docs

Read a substrate service's resource limits, and what each node applies.

GET
/v1/admin/substrate/services/{serviceName}/resources
Code sample: Shell / cURL
curl --request GET \
  --url https://stratum.example.com/v1/admin/substrate/services/example/resources \
  --header 'Authorization: Bearer <token>'

Bootstrap-admin endpoint returning one service’s memory and CPU caps: the compiled default and where its number came from, every explicit override an operator has set, and what each node resolved and last reported it actually has.

Before the substrate quadlets carried caps only where a template literal put one, and a template literal never reaches a node that is already installed — on cl-ucousbfs only 3 of 21 rendered quadlets carried a cap while 12 had one in the templates. The per-node view here is what makes that difference visible instead of assumed.

floorBytes is the smallest memoryMax a PUT will accept: the larger of the compiled per-service measurement and the highest cgroup memory.peak the fleet has reported. A cap below a working set the service has actually reached does not fail closed, it OOM-kills the service, and for DNS or the control plane that is fleet-wide.

Cedar action: substrate:GetServiceResources.

Authorizations

Section titled “Authorizations”

Parameters

Section titled “Parameters”

Path Parameters

Section titled “Path Parameters”
serviceName
required
string

The service to describe or cap, as the substrate services list names it (otel-collector, postgres-ha, stratum-controlplane), or one of the two tenant data planes (dataplane-dns-recursor, dataplane-lb).

Responses

Section titled “Responses”

200

Section titled “200”

The service’s limits, defaults and per-node state.

Media typeapplication/json

Everything an operator needs to set a resource cap for one substrate service or tenant data plane: the compiled default, what is explicitly set at each scope, what each node resolved and reports, and the floor a request has to clear.

object
serviceName
required

The service this describes, as the substrate services list names it.

string
default
required

One service’s resolved caps, with the scope that supplied each value so an operator can see where to change it.

object
serviceName
required
string
nodeId

The node this was resolved for. Absent for an install-wide answer.

string
memoryMax
required

Podman --memory value.

string
cpuQuota
required

Podman --cpus value.

string
memorySource
required

Which scope supplied memoryMax. default is the compiled catalog.

string
Allowed values: node install default
cpuSource
required

Which scope supplied cpuQuota.

string
Allowed values: node install default
restart
required

How a change to this service is applied on a node. immediate — the converger writes the quadlet, reloads systemd and restarts the unit itself. deferred — it writes and reloads, then reports pendingRestart and stops, because restarting this service is itself a cluster-wide event (the control plane’s restart kills IMDS fleet-wide; Postgres’s is a Patroni failover). The new cap applies at the next restart, which an operator sequences.

string
Allowed values: immediate deferred
observedPeakBytes

The compiled measurement the default was derived from.

integer format: int64
why

What the default number is doing and where it came from.

string
records
required

The explicit rows an operator has set, at every scope. Empty means nothing is overridden and the compiled default applies everywhere.

Array<object>

One explicitly-set row. A field absent from limits inherits from the next scope out.

object
scope
required
string
Allowed values: install node
scopeId
required

Empty for install scope; the node id for node scope.

string
serviceName
required
string
limits
required

A memory and/or CPU cap. An omitted or empty field means “no opinion at this scope” — it inherits — which is what lets a node row override memory while leaving CPU to the install.

object
memoryMax

Podman --memory value: a byte count with an optional k/m/g suffix. Stored as written and handed to podman verbatim.

string
cpuQuota

Podman --cpus value.

string
updatedAt
required
string format: date-time
updatedBy
string
nodes
required

Per-node effective value and last reported state.

Array<object>

One node’s line in a service description.

object
nodeId
required
string
effective
required

One service’s resolved caps, with the scope that supplied each value so an operator can see where to change it.

object
serviceName
required
string
nodeId

The node this was resolved for. Absent for an install-wide answer.

string
memoryMax
required

Podman --memory value.

string
cpuQuota
required

Podman --cpus value.

string
memorySource
required

Which scope supplied memoryMax. default is the compiled catalog.

string
Allowed values: node install default
cpuSource
required

Which scope supplied cpuQuota.

string
Allowed values: node install default
restart
required

How a change to this service is applied on a node. immediate — the converger writes the quadlet, reloads systemd and restarts the unit itself. deferred — it writes and reloads, then reports pendingRestart and stops, because restarting this service is itself a cluster-wide event (the control plane’s restart kills IMDS fleet-wide; Postgres’s is a Patroni failover). The new cap applies at the next restart, which an operator sequences.

string
Allowed values: immediate deferred
observedPeakBytes

The compiled measurement the default was derived from.

integer format: int64
why

What the default number is doing and where it came from.

string
status

What one node last reported about one service — written by that node about itself, never by a peer.

object
nodeId
required
string
serviceName
required
string
desiredMemory

What the node resolved it should apply.

string
desiredCpu
string
appliedMemory

What the rendered quadlet on that node actually carries. Empty means the converger looked and found no cap at all.

string
appliedCpu
string
pendingRestart
required

The converger wrote the new cap and reloaded systemd but did not restart the unit, because this service’s restart is a cluster-wide event. The cap is on disk and applies at the next restart.

boolean
observedPeakBytes

This node’s cgroup memory.peak for the service. 0 means not observed.

integer format: int64
observedCurrentBytes

This node’s cgroup memory.current for the service — what it is using now, as opposed to the most it has ever used. Read from the same cgroup directory in the same pass as the peak, because there is no metric that covers every service: cAdvisor and a systemd-cgroup exporter are not scraped, and the per-process series that exist are keyed by executable rather than by service.

0 means not observed. Reported only — never used as a bound, because a sample taken while a service happens to be idle would lower the refusal floor that protects it.

integer format: int64
lastError

Why this service could not be converged on this node. Empty when it could.

string
reportedAt
required

When this node last reported. Compare against now: the converger runs far more often than 30 minutes, so an older report means the node stopped converging rather than that it has nothing to say.

string format: date-time
drifted
required

The node reported an applied value that disagrees with what it resolved. Always false when reporting is false — read the two together, because a node that has never run the converger is not a converged node.

boolean
reporting
required

This node has reported at least once. False means the converger has never run there, which is an absence rather than a clean bill of health.

boolean
observedPeakBytes

The highest cgroup memory.peak any node has reported for this service — the true high-water mark, read on the node rather than queried from the metrics backend. Absent when no node has reported one.

integer format: int64
observedCurrentBytes

The highest cgroup memory.current any node has reported for this service — what it is using now rather than the most it has ever used, aggregated as the fleet maximum exactly as the peak is. Read from the same cgroup directory in the same pass; there is no metric that covers every service. Absent when no node has reported one.

Never contributes to floorBytes: a sample taken while a service happens to be idle would lower the very bound that protects it.

integer format: int64
floorBytes
required

The smallest memoryMax this service will currently accept. The larger of the compiled per-service measurement and the highest peak the fleet has reported. A request below this is refused, because an undersized cap does not fail closed — it OOM-kills the service.

integer format: int64
dataPlane
required

True for the two tenant data planes (dataplane-dns-recursor, dataplane-lb), which are podman pods launched per VPC or per load balancer rather than rendered quadlets. They have no file on any node, so they never report drift, and their caps take effect when the sweeper next recreates the pod. They do report observed memory, sampled from the pod’s own cgroup on the nodes hosting one.

boolean
applicability
required

Whether this service is expected to report node status on this install, and if not, why not.

Without it, four rows on the admin resource-limits page rendered identically — no peak, no node count, nodesReporting: 0 — for three different reasons, only one of which is a problem. The one state an operator needs to notice was indistinguishable from two that are entirely normal, so no alarm could be written against it.

  • expected — this install renders the service and every node should be reporting it. Zero reporters here is a fault: the host-helper is down, the converger never ran, the intent was never published, or the node is unreachable.
  • data-plane — a per-VPC or per-load-balancer pod, created imperatively by the sweeper rather than rendered as a quadlet. It reports an observation (observedPeakBytes / observedCurrentBytes, sampled from the pod’s own cgroup) on whichever nodes currently host one, and never reports drift there is no rendered file to compare and its cap reaches the pod through the sweeper. nodesReporting: 0 here means no pod is placed on any node, which is placement doing its job rather than a fault.
  • not-rendered — the renderer gates this service out on this install’s shape. The non-HA postgres and victoriametrics on an HA cluster install; on a single-node install those are the live ones and postgres-ha / vminsert / vmselect / vmstorage are gated out instead. Nothing can report it because nothing renders it.
string
Allowed values: expected data-plane not-rendered

Example 

{
  "serviceName": "otel-collector",
  "default": {
    "memoryMax": "4g",
    "cpuQuota": "2.0",
    "memorySource": "node",
    "cpuSource": "node",
    "restart": "immediate"
  },
  "records": [
    {
      "scope": "install",
      "limits": {
        "memoryMax": "2g",
        "cpuQuota": "2.0"
      }
    }
  ],
  "nodes": [
    {
      "effective": {
        "memoryMax": "4g",
        "cpuQuota": "2.0",
        "memorySource": "node",
        "cpuSource": "node",
        "restart": "immediate"
      }
    }
  ],
  "applicability": "expected"
}

403

Section titled “403”

Authenticated but not authorized.

Media typeapplication/problem+json
object
type
required
string format: uri
title
required
string
status
required
integer
>= 400 <= 599
detail
string
instance
string format: uri-reference
requestId
required
string
code
required
string
issues

The findings of a rejected IAM policy document (custom and resource policies).

Array<object>

One finding of a rejected IAM policy document. A rejection lists every finding, so the author fixes the document in one round trip.

object
code
required

The finding, e.g. INVALID_DOCUMENT, UNKNOWN_ACTION, ACTION_NOT_APPLICABLE.

string
statement
required

Index of the statement at fault; -1 for the document as a whole.

integer
>= -1
detail
required
string
action

The offending action, where one is at fault (ACTION_NOT_APPLICABLE).

string
errors

Field errors, for codes that carry them: each problem at its field (see FieldError). Documented on the responses that set it, e.g. approveNodeJoin’s INVALID_INPUT and JOIN_APPROVAL_REFUSED. Absent otherwise.

Array<object>

One field’s problem, the stage-2 installer API’s shape. field is the field’s path, message what is wrong with it.

object
field
required

The field’s path, e.g. links[0].interface, disks[1].confirm, mgmt.address or role; empty for the whole request.

string
message
required
string
details

Structured facts about the refusal, for codes that carry them. The shape is fixed per code and documented on the response that answers it, e.g. IMAGE_POOL_MISMATCH (see ImagePoolMismatchDetails) and IMAGE_IN_USE. Absent for codes that carry none.

object
key
additional properties
any

Example 

{
  "issues": [
    {
      "code": "ACTION_NOT_APPLICABLE",
      "action": "compute:CreateVpc"
    }
  ]
}

404

Section titled “404”

Resource not found.

Media typeapplication/problem+json
object
type
required
string format: uri
title
required
string
status
required
integer
>= 400 <= 599
detail
string
instance
string format: uri-reference
requestId
required
string
code
required
string
issues

The findings of a rejected IAM policy document (custom and resource policies).

Array<object>

One finding of a rejected IAM policy document. A rejection lists every finding, so the author fixes the document in one round trip.

object
code
required

The finding, e.g. INVALID_DOCUMENT, UNKNOWN_ACTION, ACTION_NOT_APPLICABLE.

string
statement
required

Index of the statement at fault; -1 for the document as a whole.

integer
>= -1
detail
required
string
action

The offending action, where one is at fault (ACTION_NOT_APPLICABLE).

string
errors

Field errors, for codes that carry them: each problem at its field (see FieldError). Documented on the responses that set it, e.g. approveNodeJoin’s INVALID_INPUT and JOIN_APPROVAL_REFUSED. Absent otherwise.

Array<object>

One field’s problem, the stage-2 installer API’s shape. field is the field’s path, message what is wrong with it.

object
field
required

The field’s path, e.g. links[0].interface, disks[1].confirm, mgmt.address or role; empty for the whole request.

string
message
required
string
details

Structured facts about the refusal, for codes that carry them. The shape is fixed per code and documented on the response that answers it, e.g. IMAGE_POOL_MISMATCH (see ImagePoolMismatchDetails) and IMAGE_IN_USE. Absent for codes that carry none.

object
key
additional properties
any

Example 

{
  "issues": [
    {
      "code": "ACTION_NOT_APPLICABLE",
      "action": "compute:CreateVpc"
    }
  ]
}