Set a substrate service's resource limits at one scope.
curl --request PUT \ --url https://stratum.example.com/v1/admin/substrate/services/example/resources \ --header 'Authorization: Bearer <token>' \ --header 'Content-Type: application/json' \ --data '{ "scope": "install", "scopeId": "example", "memoryMax": "8g", "cpuQuota": "4.0" }'Records the DESIRED limit and returns 202. Nothing is converged inline:
each node’s converger picks the change up on its next pass, rewrites the
rendered quadlet, reloads systemd and — for a service whose restart is
immediate — restarts the unit. Progress is visible in the nodes
array of the GET.
A service whose restart is deferred (the control plane, Postgres,
OpenBao) has the new cap written to disk and reported as
pendingRestart, because restarting it is itself a cluster-wide event.
It takes effect at the next restart, which an operator sequences.
Scope precedence is node, then install, then the compiled default, taken per field — so a node row can override memory while leaving CPU to the install. Sparse body:
- field omitted or empty → clear the override at this scope and inherit from the scope above
- field present with a value → set the override here
Clearing both fields removes the row entirely rather than storing one that expresses nothing.
A memoryMax below floorBytes is refused with 422 and
SERVICE_LIMIT_BELOW_OBSERVED_PEAK, quoting the requested value, the
floor and where the floor came from.
Cedar action: substrate:UpdateServiceResources.
Authorizations
Section titled “Authorizations”Parameters
Section titled “Parameters”Path Parameters
Section titled “Path Parameters”The service to describe or cap, as the substrate services list names it (otel-collector, postgres-ha, stratum-controlplane), or one of the two tenant data planes (dataplane-dns-recursor, dataplane-lb).
Request Bodyrequired
Section titled “Request Bodyrequired”Sparse upsert of one service’s caps at one scope. Omitting or emptying a field clears the override at this scope; clearing both removes the row.
object
Which scope to set. install is the cluster-wide default; node
overrides it for one node, and needs scopeId.
The node id, required when scope is node and ignored otherwise.
Podman --memory value: a byte count with an optional k/m/g
suffix. Must be at least the service’s floorBytes.
Example
8gPodman --cpus value.
Example
4.0Responses
Section titled “Responses”The desired limit is recorded. Convergence happens on each node’s
next reconciler pass; poll the GET to watch it land.
Everything an operator needs to set a resource cap for one substrate service or tenant data plane: the compiled default, what is explicitly set at each scope, what each node resolved and reports, and the floor a request has to clear.
object
The service this describes, as the substrate services list names it.
One service’s resolved caps, with the scope that supplied each value so an operator can see where to change it.
object
The node this was resolved for. Absent for an install-wide answer.
Podman --memory value.
Podman --cpus value.
Which scope supplied memoryMax. default is the compiled catalog.
Which scope supplied cpuQuota.
How a change to this service is applied on a node. immediate — the
converger writes the quadlet, reloads systemd and restarts the unit
itself. deferred — it writes and reloads, then reports
pendingRestart and stops, because restarting this service is itself
a cluster-wide event (the control plane’s restart kills IMDS
fleet-wide; Postgres’s is a Patroni failover). The new cap applies at
the next restart, which an operator sequences.
The compiled measurement the default was derived from.
What the default number is doing and where it came from.
The explicit rows an operator has set, at every scope. Empty means nothing is overridden and the compiled default applies everywhere.
One explicitly-set row. A field absent from limits inherits from the next scope out.
object
Empty for install scope; the node id for node scope.
A memory and/or CPU cap. An omitted or empty field means “no opinion at this scope” — it inherits — which is what lets a node row override memory while leaving CPU to the install.
object
Podman --memory value: a byte count with an optional k/m/g
suffix. Stored as written and handed to podman verbatim.
Podman --cpus value.
Per-node effective value and last reported state.
One node’s line in a service description.
object
One service’s resolved caps, with the scope that supplied each value so an operator can see where to change it.
object
The node this was resolved for. Absent for an install-wide answer.
Podman --memory value.
Podman --cpus value.
Which scope supplied memoryMax. default is the compiled catalog.
Which scope supplied cpuQuota.
How a change to this service is applied on a node. immediate — the
converger writes the quadlet, reloads systemd and restarts the unit
itself. deferred — it writes and reloads, then reports
pendingRestart and stops, because restarting this service is itself
a cluster-wide event (the control plane’s restart kills IMDS
fleet-wide; Postgres’s is a Patroni failover). The new cap applies at
the next restart, which an operator sequences.
The compiled measurement the default was derived from.
What the default number is doing and where it came from.
What one node last reported about one service — written by that node about itself, never by a peer.
object
What the node resolved it should apply.
What the rendered quadlet on that node actually carries. Empty means the converger looked and found no cap at all.
The converger wrote the new cap and reloaded systemd but did not restart the unit, because this service’s restart is a cluster-wide event. The cap is on disk and applies at the next restart.
This node’s cgroup memory.peak for the service. 0 means not observed.
This node’s cgroup memory.current for the service — what it is using
now, as opposed to the most it has ever used. Read from the same cgroup
directory in the same pass as the peak, because there is no metric that
covers every service: cAdvisor and a systemd-cgroup exporter are not
scraped, and the per-process series that exist are keyed by executable
rather than by service.
0 means not observed. Reported only — never used as a bound, because a sample taken while a service happens to be idle would lower the refusal floor that protects it.
Why this service could not be converged on this node. Empty when it could.
When this node last reported. Compare against now: the converger runs far more often than 30 minutes, so an older report means the node stopped converging rather than that it has nothing to say.
The node reported an applied value that disagrees with what it
resolved. Always false when reporting is false — read the two
together, because a node that has never run the converger is not a
converged node.
This node has reported at least once. False means the converger has never run there, which is an absence rather than a clean bill of health.
The highest cgroup memory.peak any node has reported for this
service — the true high-water mark, read on the node rather than
queried from the metrics backend. Absent when no node has reported
one.
The highest cgroup memory.current any node has reported for this
service — what it is using now rather than the most it has ever used,
aggregated as the fleet maximum exactly as the peak is. Read from the
same cgroup directory in the same pass; there is no metric that covers
every service. Absent when no node has reported one.
Never contributes to floorBytes: a sample taken while a service happens
to be idle would lower the very bound that protects it.
The smallest memoryMax this service will currently accept. The
larger of the compiled per-service measurement and the highest peak
the fleet has reported. A request below this is refused, because an
undersized cap does not fail closed — it OOM-kills the service.
True for the two tenant data planes (dataplane-dns-recursor,
dataplane-lb), which are podman pods launched per VPC or per load
balancer rather than rendered quadlets. They have no file on any node, so
they never report drift, and their caps take effect when the sweeper next
recreates the pod. They do report observed memory, sampled from the pod’s
own cgroup on the nodes hosting one.
Whether this service is expected to report node status on this install, and if not, why not.
Without it, four rows on the admin resource-limits page rendered identically
— no peak, no node count, nodesReporting: 0 — for three different reasons,
only one of which is a problem. The one state an operator needs to notice was
indistinguishable from two that are entirely normal, so no alarm could be
written against it.
expected— this install renders the service and every node should be reporting it. Zero reporters here is a fault: the host-helper is down, the converger never ran, the intent was never published, or the node is unreachable.data-plane— a per-VPC or per-load-balancer pod, created imperatively by the sweeper rather than rendered as a quadlet. It reports an observation (observedPeakBytes/observedCurrentBytes, sampled from the pod’s own cgroup) on whichever nodes currently host one, and never reports drift there is no rendered file to compare and its cap reaches the pod through the sweeper.nodesReporting: 0here means no pod is placed on any node, which is placement doing its job rather than a fault.not-rendered— the renderer gates this service out on this install’s shape. The non-HApostgresandvictoriametricson an HA cluster install; on a single-node install those are the live ones andpostgres-ha/vminsert/vmselect/vmstorageare gated out instead. Nothing can report it because nothing renders it.
Example
{ "serviceName": "otel-collector", "default": { "memoryMax": "4g", "cpuQuota": "2.0", "memorySource": "node", "cpuSource": "node", "restart": "immediate" }, "records": [ { "scope": "install", "limits": { "memoryMax": "2g", "cpuQuota": "2.0" } } ], "nodes": [ { "effective": { "memoryMax": "4g", "cpuQuota": "2.0", "memorySource": "node", "cpuSource": "node", "restart": "immediate" } } ], "applicability": "expected"}Bad request.
object
The findings of a rejected IAM policy document (custom and resource policies).
One finding of a rejected IAM policy document. A rejection lists every finding, so the author fixes the document in one round trip.
object
The finding, e.g. INVALID_DOCUMENT, UNKNOWN_ACTION, ACTION_NOT_APPLICABLE.
Index of the statement at fault; -1 for the document as a whole.
The offending action, where one is at fault (ACTION_NOT_APPLICABLE).
Field errors, for codes that carry them: each problem at its field
(see FieldError). Documented on the responses that set it, e.g.
approveNodeJoin’s INVALID_INPUT and JOIN_APPROVAL_REFUSED.
Absent otherwise.
One field’s problem, the stage-2 installer API’s shape.
field is the field’s path, message what is wrong with it.
object
The field’s path, e.g. links[0].interface, disks[1].confirm,
mgmt.address or role; empty for the whole request.
Structured facts about the refusal, for codes that carry them. The
shape is fixed per code and documented on the response that answers
it, e.g. IMAGE_POOL_MISMATCH (see ImagePoolMismatchDetails) and
IMAGE_IN_USE. Absent for codes that carry none.
object
Example
{ "issues": [ { "code": "ACTION_NOT_APPLICABLE", "action": "compute:CreateVpc" } ]}Authenticated but not authorized.
object
The findings of a rejected IAM policy document (custom and resource policies).
One finding of a rejected IAM policy document. A rejection lists every finding, so the author fixes the document in one round trip.
object
The finding, e.g. INVALID_DOCUMENT, UNKNOWN_ACTION, ACTION_NOT_APPLICABLE.
Index of the statement at fault; -1 for the document as a whole.
The offending action, where one is at fault (ACTION_NOT_APPLICABLE).
Field errors, for codes that carry them: each problem at its field
(see FieldError). Documented on the responses that set it, e.g.
approveNodeJoin’s INVALID_INPUT and JOIN_APPROVAL_REFUSED.
Absent otherwise.
One field’s problem, the stage-2 installer API’s shape.
field is the field’s path, message what is wrong with it.
object
The field’s path, e.g. links[0].interface, disks[1].confirm,
mgmt.address or role; empty for the whole request.
Structured facts about the refusal, for codes that carry them. The
shape is fixed per code and documented on the response that answers
it, e.g. IMAGE_POOL_MISMATCH (see ImagePoolMismatchDetails) and
IMAGE_IN_USE. Absent for codes that carry none.
object
Example
{ "issues": [ { "code": "ACTION_NOT_APPLICABLE", "action": "compute:CreateVpc" } ]}Resource not found.
object
The findings of a rejected IAM policy document (custom and resource policies).
One finding of a rejected IAM policy document. A rejection lists every finding, so the author fixes the document in one round trip.
object
The finding, e.g. INVALID_DOCUMENT, UNKNOWN_ACTION, ACTION_NOT_APPLICABLE.
Index of the statement at fault; -1 for the document as a whole.
The offending action, where one is at fault (ACTION_NOT_APPLICABLE).
Field errors, for codes that carry them: each problem at its field
(see FieldError). Documented on the responses that set it, e.g.
approveNodeJoin’s INVALID_INPUT and JOIN_APPROVAL_REFUSED.
Absent otherwise.
One field’s problem, the stage-2 installer API’s shape.
field is the field’s path, message what is wrong with it.
object
The field’s path, e.g. links[0].interface, disks[1].confirm,
mgmt.address or role; empty for the whole request.
Structured facts about the refusal, for codes that carry them. The
shape is fixed per code and documented on the response that answers
it, e.g. IMAGE_POOL_MISMATCH (see ImagePoolMismatchDetails) and
IMAGE_IN_USE. Absent for codes that carry none.
object
Example
{ "issues": [ { "code": "ACTION_NOT_APPLICABLE", "action": "compute:CreateVpc" } ]}The requested limit is below the service’s observed peak usage, or is not a value podman accepts.
object
The findings of a rejected IAM policy document (custom and resource policies).
One finding of a rejected IAM policy document. A rejection lists every finding, so the author fixes the document in one round trip.
object
The finding, e.g. INVALID_DOCUMENT, UNKNOWN_ACTION, ACTION_NOT_APPLICABLE.
Index of the statement at fault; -1 for the document as a whole.
The offending action, where one is at fault (ACTION_NOT_APPLICABLE).
Field errors, for codes that carry them: each problem at its field
(see FieldError). Documented on the responses that set it, e.g.
approveNodeJoin’s INVALID_INPUT and JOIN_APPROVAL_REFUSED.
Absent otherwise.
One field’s problem, the stage-2 installer API’s shape.
field is the field’s path, message what is wrong with it.
object
The field’s path, e.g. links[0].interface, disks[1].confirm,
mgmt.address or role; empty for the whole request.
Structured facts about the refusal, for codes that carry them. The
shape is fixed per code and documented on the response that answers
it, e.g. IMAGE_POOL_MISMATCH (see ImagePoolMismatchDetails) and
IMAGE_IN_USE. Absent for codes that carry none.
object
Example
{ "issues": [ { "code": "ACTION_NOT_APPLICABLE", "action": "compute:CreateVpc" } ]}