Execution Lifecycle

Every workflow execution transitions through a well-defined set of states. Understanding these states helps you monitor executions, debug failures, and design robust workflows.

TL;DR

A workflow execution moves through: NEW → VALID → READY → acquire entities → LOCKED → RUNNING → COMPLETED (or FAILED_SAFE / FAILED_UNSAFE), and then to the matching _ACK state once the engine has finished with it — that is where an execution comes to rest. A FAILED_SAFE means nothing irreversible happened; FAILED_UNSAFE means manual review is needed — and it is the one path that keeps its entity locks instead of releasing them, until human-driven remediation turns it into FAILED_SAFE.

State Machine

stateDiagram-v2
        classDef goodState stroke:green
        classDef warningState stroke:orange
        classDef badState stroke:red

        class NEW goodState
        class READY goodState
        class VALID goodState
        class LOCKING goodState
        class LOCKED goodState
        class BLOCKED_WAITING goodState
        class RESOURCE_DISCOVERY goodState
        class RESOURCES_DISCOVERED goodState
        class SCHEDULED goodState
        class RUNNING goodState
        class COMPLETED goodState
        class COMPLETED_ACK goodState
        class FAILED_SAFE warningState
        class ERROR warningState
        class ROLLBACK warningState
        class FAILED_UNSAFE badState
        class FAILED_UNSAFE_ACK badState
        class FAILED_SAFE_ACK warningState

    [*] --> NEW
    NEW --> VALID: resolved & validated
    NEW --> FAILED_SAFE: validation failed

    VALID --> READY
    VALID --> BLOCKED_WAITING: awaiting conditions

    READY --> RESOURCE_DISCOVERY
    RESOURCE_DISCOVERY --> RESOURCES_DISCOVERED

    RESOURCES_DISCOVERED --> SCHEDULED
    SCHEDULED --> LOCKING
    LOCKING --> LOCKED

    LOCKED --> RUNNING
    RUNNING --> COMPLETED: all steps done
    RUNNING --> ERROR: step failed

    ERROR --> FAILED_SAFE: nothing to roll back
    ERROR --> ROLLBACK: all ledger entries reversible
    ERROR --> FAILED_UNSAFE: irreversible FB executed

    ROLLBACK --> FAILED_SAFE: rollback succeeded
    ROLLBACK --> FAILED_UNSAFE: rollback failed / blocked by a human-only entry

    COMPLETED --> COMPLETED_ACK: DB updates applied, locks released
    FAILED_SAFE --> FAILED_SAFE_ACK: locks released
    FAILED_UNSAFE --> FAILED_UNSAFE_ACK: locks RETAINED
    FAILED_UNSAFE_ACK --> ROLLBACK: remediation rollback all / continueRollback
    FAILED_UNSAFE_ACK --> FAILED_SAFE: remediated (ledger complete)

    COMPLETED_ACK --> [*]
    FAILED_SAFE_ACK --> [*]
    FAILED_UNSAFE_ACK --> [*]

States in Detail

Submission Phase

NEW : The execution has been created. The engine resolves function block references, validates the workflow definition against the schema, and checks that all referenced function blocks are registered. - Success → VALID - Failure (unresolvable FB, invalid schema) → FAILED_SAFE

VALID : The workflow is structurally correct and all dependencies are resolved. - Normally → READY - If external conditions required → BLOCKED_WAITING (reserved for future use)

Acquisition Phase

READY : The execution is ready to acquire resources. This is the entry point for the resource discovery phase. - → RESOURCE_DISCOVERY

RESOURCE_DISCOVERY : The engine traverses the workflow tree and creates ACQUIRE jobs for each step that defines entity requirements. Workers execute the function block’s acquire() methods and return the entities needed. - → RESOURCES_DISCOVERED

RESOURCES_DISCOVERED : All entity requirements have been collected. The engine knows which devices, interfaces, and groups are needed. - → SCHEDULED

Locking Phase

SCHEDULED : The execution is ready for locking. If the required entities are locked by another execution, the workflow waits here. - → LOCKING

LOCKING : The engine has sent a lock request to the CMS for all required entities. - Success → LOCKED - Failure → FAILED_SAFE

LOCKED : All entities are exclusively locked. No other workflow can modify them until this execution completes. - → RUNNING

Execution Phase

RUNNING : Steps are actively executing. The engine creates EXECUTE jobs on the blackboard, processes results, and advances through the step tree. - All steps complete → COMPLETED - Any step fails → ERROR

Terminal States

COMPLETED : All steps finished successfully. DB updates are applied to the CMS, locks are released.

ERROR : A step failed. The engine decides how the execution terminates based on what actually reached a worker — a job never handed to a worker provably never ran and does not count as executed, and neither does a step whose every attempt reported that it changed nothing: - No executed step was non-pure, or every one changed nothing → nothing to revert → FAILED_SAFE - Every remaining non-pure function block is registered reversible → ROLLBACK - Any remaining non-pure function block is irreversible → rollback is all-or-nothing, so none is attempted → FAILED_UNSAFE

ROLLBACK : The engine reverses the executed non-pure steps by pushing ROLLBACK jobs to the blackboard wave by wave in descending execution-wave order — the reverse of the deterministic traversal that created the original jobs. Jobs within a wave roll back in parallel; the next (lower) wave starts only once the whole wave succeeded. - All rollback jobs succeed → FAILED_SAFE - Any rollback job fails → the phase halts → FAILED_UNSAFE - Re-entered by remediation: resolved entries are skipped; an open irreversible entry, or an entry whose rollback failed, that stops the walk → FAILED_UNSAFE, with the blocking entries named in the reason

FAILED_SAFE : The execution failed with no remaining side effects: either nothing non-pure executed, every non-pure step reported that it changed nothing, everything else was successfully rolled back, or a human remediated an unsafe failure. Safe to retry or discard.

FAILED_UNSAFE : The execution failed with potential side effects. Manual review required, and the execution’s entity locks are deliberately kept (see below).

Locks are NOT released on an unsafe failure

COMPLETED and FAILED_SAFE release the execution’s CMS locks on their way to _ACK. FAILED_UNSAFE does not — this is the one deliberate asymmetry in the lifecycle.

An unsafe failure means the state of the affected entities can no longer be derived from the engine’s records: a non-pure step may have reached the network, and nothing proved otherwise. Releasing the locks would let the next workflow plan against that unknown state. So the locks stay held, the execution’s cmsResourcesLocked flag stays true, and the handler logs at error level that it is retaining them:

Workflow failed unsafely — CMS resource locks RETAINED
    workflow_execution_id=<uuid> lockReference=<uuid> lockedDevices=[...] ...

Consequences an operator needs to know:

A later workflow that needs one of those entities cannot lock it. It stays in LOCKING and retries on a capped exponential backoff (5 minutes at most between attempts), indefinitely — it never spins hot and never wedges the engine, and the retry is deliberate: once the lock is released, the workflow resumes on its own with no re-submission. The CMS refusal is recorded on the execution as its retryNote, which stays while the workflow waits and is cleared once the retry that acquires the lock goes through, also when that retry runs after an engine restart.
The CompositeLock in the CMS is created without an expiry, so nothing sweeps it up. It is released only through human-driven remediation (see below), or by deleting the CMS CompositeLock (reference = execution UUID) out-of-band as a break-glass measure.
Finding the run that holds a lock does not need the logs. The run entry (GET /workflow-execution/id/<uuid>) carries cmsResourcesLocked, the resolved context (devices with their hostnames) and heldLocks (the ids of the locked entities), and the paged run list (GET /workflow-execution/page?cmsResourcesLocked=true&target=<device id>) answers which run keeps this device locked.

FAILED_UNSAFE_ACK is still terminal. _ACK means the engine initiates nothing further on its own, not the execution was cleaned up. Keeping it terminal is what stops the engine from reprocessing the execution every epoch and keeps it out of the active-execution and license counts. Only remediation, which a human drives, moves it on.

Remediation: the way the locks come off

The remediation API walks the execution’s compensation ledger — the executed non-pure steps, except those that changed nothing, each of which must end rolled back (a worker ran the function block’s rollback()) or resolved (a human attested the fix with a mandatory note; the engine records the request’s authenticated principal as resolvedBy / resolvedByUsername). The moment every entry is compensated, the execution transitions to FAILED_SAFE:

from ROLLBACK when a rollback phase re-entered by rollback {"all": true} or continueRollback completes the ledger;
from FAILED_UNSAFE_ACK when a resolve, or the success of a single-entry rollback, completes it.

From there the ordinary failed-safe path takes over: the engine releases the retained CMS locks and acknowledges the execution as FAILED_SAFE_ACK; blocked workflows resume on their own. The reason records how the ledger was compensated (see Reason notes). A CMS outage during that release is retried like any other failed-safe unlock, and an engine restart in between resumes it, because FAILED_SAFE is not an _ACK state. A remediated execution releases its locks without applying any CMS updates the run had aggregated; the operator reconciles the CMS by hand where an attestation implies it.

# The ledger: every executed non-pure step that may have changed something, and its compensation state
curl http://localhost:3030/workflow-execution/<id>/remediation

# Roll back entries (each must be reversible and wave-gated), or walk all waves
curl -X POST http://localhost:3030/workflow-execution/<id>/remediation/rollback \
  -H 'Content-Type: application/json' -d '{"jobs": ["<entry-job-uuid>"]}'
curl -X POST http://localhost:3030/workflow-execution/<id>/remediation/rollback \
  -H 'Content-Type: application/json' -d '{"all": true}'

# Attest entries fixed by hand; completing the ledger moves the execution to FAILED_SAFE in-request
curl -X POST http://localhost:3030/workflow-execution/<id>/remediation/resolve \
  -H 'Content-Type: application/json' \
  -d '{"all": true, "note": "network state restored and verified"}'

# Remediate and continue rollback: attest the blocker, and let the engine reverse the waves below it
curl -X POST http://localhost:3030/workflow-execution/<id>/remediation/resolve \
  -H 'Content-Type: application/json' \
  -d '{"jobs": ["<entry-job-uuid>"], "note": "flash restored by hand", "continueRollback": true}'

Two grants, one per act. Requesting a rollback needs workflow-execution-rollback:write; resolving needs workflow-execution-resolve:write. Neither implies the other: a rollback asks the engine to reverse a step on the devices, while a resolve releases the retained locks on a human’s word about entities whose state the engine cannot derive. A resolve with "continueRollback": true also restarts the rollback, so it needs both grants; without the rollback grant it answers 403 naming workflow-execution-rollback:write, and nothing is recorded. Reading the ledger needs workflow-execution:read.
Wave order is enforced for automation: an entry may be rolled back only when every entry in a strictly higher execution wave is already compensated — resolving the irreversible step of a high wave is what licenses automation to reverse the waves beneath it. Resolving itself has no wave gate. rollback {"all": true} puts the execution back into ROLLBACK: the automatic rollback phase walks the remaining waves descending, skipping resolved entries, and ends FAILED_SAFE once the ledger is complete or FAILED_UNSAFE — back to FAILED_UNSAFE_ACK, where remediation continues — when a rollback fails or an entry only a human can compensate blocks the walk (the reason names it). The request itself answers 409 when nothing is left to roll back, when a wave is still in flight, or when the walk is blocked from the start.
Rolling back single entries ({"jobs": [...]}) keeps the execution in FAILED_UNSAFE_ACK.
A failed rollback attempt leaves the entry ROLLBACK_FAILED, open to another attempt: name it in {"jobs": [...]} to run its rollback again, or resolve it by hand instead. A walk never re-runs it on its own; only the entries an operator names are run again. While the current wave holds such an entry uncompensated, rollback {"all": true} answers 409 naming it (name it to re-run, or resolve it), and a resolve with continueRollback records its attestation but does not enter the walk, and its response says why. The function block’s second rollback sees the earlier attempts in previousRollbacks and decides what to do.
Remediate and continue rollback ("continueRollback": true on resolve) attests the entry and, in the same request, puts the execution back into ROLLBACK when the attestation unblocked a wave, so the engine carries on wave by wave without a second call. If the current wave still has a single-entry rollback in flight, the phase waits for it and continues on its success. If another entry only a human can compensate still blocks the walk, the attestation is recorded and the execution stays in FAILED_UNSAFE_ACK; createdRollbackJobs and the ledger’s status say which of these happened.
An execution whose ledger is empty (it failed unsafely before any non-pure step reached a worker, or every one that did changed nothing) is vacuously complete: resolve {"all": true} on it is the operator act that moves it to FAILED_SAFE. Its reason then records remediation: resolved by <user> instead of remediated: ….
The ledger also lists, under leftOut, the steps it leaves out because every attempt reported sideEffects: false: one item per step and entity, with stepLabel, entityId, functionBlockId and its attempts, each as {jobUuid, status, sideEffects}. They are not entries: they need no compensation and do not count toward complete.
The ledger stays readable during and after remediation: GET .../remediation answers for FAILED_UNSAFE_ACK, ROLLBACK, FAILED_SAFE and FAILED_SAFE_ACK executions, so the compensated entries and their attestations remain the audit trail. The mutating routes answer 409 for anything but FAILED_UNSAFE_ACK — during ROLLBACK, because a rollback is in progress.

An unlock_failed run that did only CMS work has an empty ledger

Declared CMS updates (dbUpdates) are not side effects, so a run whose steps were all pure ends with an empty ledger even when the CMS write-back of those updates failed (unsafeCause unlock_failed). Check the CMS anyway before you resolve it: the updates may or may not have been applied.

Reason notes

Remediation and the rollback phase append notes to the execution’s reason, separated by |. An engine note already present is not appended again, so repeated phase passes and restart replays converge on the same reason. An operator act (remediation: …) is appended every time, so an entry re-run by name after a failed wave leaves a second trace. A reason longer than its 10 000-character column loses its middle (…[truncated]…): the original failure cause at the head and the latest notes at the tail are kept.

Note	Appended when
`Rollback failed at step <label>: <message>`	A rollback attempt failed — in the `ROLLBACK` phase (automatic or re-entered by remediation) or as a single-entry remediation rollback in `FAILED_UNSAFE_ACK`
`Rollback at step <label> never reached a worker: <message>`	The pending-job cleanup failed a rollback attempt no worker ever polled. The attempt never ran, so its entry stays `OPEN`; in the `ROLLBACK` phase it still halts the walk
`Rollback blocked pending human resolution of: <entries>`	The `ROLLBACK` phase has no rollbackable wave left while the ledger is incomplete
`remediation: rollback-all requested by <user>`	`rollback {"all": true}` re-enters `ROLLBACK`
`remediation: rollback continued by <user>`	A resolve with `continueRollback` re-enters `ROLLBACK`
`remediation: resolved by <user>`	A resolve completes an empty ledger
`remediated: <n> rolled back, <m> resolved by <users>`	The execution turns `FAILED_SAFE` after a human took part: an entry was resolved, or needed more than one rollback attempt. `by <users>` lists the resolving principals’ usernames and is omitted when there are none

<user> is the acting principal’s username, or an anonymous principal when the credential carries none.

Failure codes

A failed run carries the free-text reason, which describes what went wrong and may be reworded, and a stable reasonCode a client can group, filter and phrase on. The code names the first failure, the cause that started it; it is null for a run that did not fail. Codes are only added, never renamed.

`reasonCode`	The run failed because
`invalid_workflow`	the definition cannot run: duplicate step labels, no steps, an unknown step, no resolvable `runOn`, or a workflow reference step, which the engine does not run yet
`unresolved_reference`	the definition names a function block or workflow that is not registered
`invalid_parameters`	a step’s parameters break its function block’s parameter schema
`interpolation_failed`	a `{{ }}` token could not be evaluated
`lock_refused`	the CMS refused or could not complete the lock on the run’s entities, or the lock was never requested because the run’s starter maps to no CMS user
`acquire_failed`	resource discovery failed
`step_failed`	a worker reported a step’s job as failed
`invalid_result`	a step’s result breaks its function block’s result schema, its dbUpdates write outside the step’s entities or collide with another step’s, or a pure function block reported side effects
`assertion_failed`	a step’s assertion evaluated to false
`job_timed_out`	no worker claimed a job before the deadline, or a claimed job went silent
`worker_lost`	the worker holding a job was deleted or went offline
`unlock_failed`	the CMS write-back or the release of the run’s locks failed
`aborted`	a person aborted the run (reserved: abort does not stop runs yet)
`engine_error`	an unexpected fault in the engine or its infrastructure

A run that ended FAILED_UNSAFE also names why it could not be made safe, in unsafeCause:

`unsafeCause`	Meaning
`rollback_failed`	a rollback job failed
`rollback_blocked`	an irreversible function block had executed, so no rollback was attempted
`unlock_failed`	the CMS write-back or lock release failed
`engine_fault`	the engine failed while executing and cannot tell what the run had done

Every path into FAILED_UNSAFE names one, and a later unsafe end replaces it with its own: a remediated run whose unlock the CMS then denies ends with unlock_failed, while reasonCode still names the first failure. It is null for every run that did not end unsafe. A run remediated to FAILED_SAFE keeps both codes: FAILED_SAFE or FAILED_SAFE_ACK with an unsafeCause is a run a person made safe.

Acknowledged States (the real terminal states)

COMPLETED, FAILED_SAFE and FAILED_UNSAFE are outcome states: reaching one means the engine still has work to do (apply DB updates, release locks — or, for FAILED_UNSAFE, deliberately not). Each is followed by its acknowledged counterpart, and those three are the states an execution actually rests in:

COMPLETED_ACK / FAILED_SAFE_ACK / FAILED_UNSAFE_ACK : The engine is finished and initiates nothing further. COMPLETED_ACK and FAILED_SAFE_ACK are final. FAILED_UNSAFE_ACK still holds its entity locks, and remediation is the one way out of it (to FAILED_SAFE, or back into ROLLBACK).

Poll for the _ACK state, not the outcome state

When you wait for an execution to finish, treat completed_ack, failed_safe_ack and failed_unsafe_ack as the finish line. completed is transient — an execution seen in it is still releasing locks and writing entity updates.

Monitoring Executions

Via API

# Executions not yet acknowledged, plus those that finished in the last 5 minutes
curl http://localhost:3030/workflow-execution/active

# Get a specific execution
curl http://localhost:3030/workflow-execution/id/<execution-id>

# List executions by state
curl http://localhost:3030/workflow-execution/state/RUNNING

Via Monitor App

The Monitor App’s Executions view shows all executions with their current state, progress (jobs completed/total), and drill-down to individual job results.

Stuck Execution Recovery

If an execution appears stuck:

Symptom	Likely cause	Resolution
Stuck in `SCHEDULED`	Entities locked by another execution	Wait for the blocking execution to finish (aborting is not implemented)
Stuck in `RUNNING`	Worker crashed or disconnected	Wait for stuck job timeout (12 min default), then jobs are auto-failed
Stuck in `LOCKING`	CMS unreachable	Check CMS connectivity, restart engine
Stuck in `LOCKING`, CMS healthy	An entity is still locked by an earlier `FAILED_UNSAFE_ACK` execution, which keeps its locks by design (see the `retryNote`)	Remediate that execution (`GET /workflow-execution/<id>/remediation`, then roll back / resolve its entries) — completing its ledger releases the lock and the waiting workflow resumes on its own

Aborting is not implemented

DELETE /workflow-execution/<execution-id> authorizes the caller (the abort grant or the principal who started the run) and then answers 501 Not Implemented. It stops nothing and writes nothing; abortedBy stays empty. A run that cannot lock its entities waits in LOCKING and resumes on its own once they are free.

2026-03-18 09:12:072026-10-29 14:24:39