Skip to content

Execution Lifecycle

Every workflow execution transitions through a well-defined set of states. Understanding these states helps you monitor executions, debug failures, and design robust workflows.

TL;DR

A workflow execution moves through: NEW → VALID → READY → acquire entities → LOCKED → RUNNING → COMPLETED (or FAILED_SAFE / FAILED_UNSAFE), and then to the matching _ACK state once the engine has finished with it — that is where an execution comes to rest. A FAILED_SAFE means nothing irreversible happened; FAILED_UNSAFE means manual review is needed — and it is the one path that keeps its entity locks instead of releasing them.

State Machine

stateDiagram-v2
        classDef goodState stroke:green
        classDef warningState stroke:orange
        classDef badState stroke:red

        class NEW goodState
        class READY goodState
        class VALID goodState
        class LOCKING goodState
        class LOCKED goodState
        class BLOCKED_WAITING goodState
        class RESOURCE_DISCOVERY goodState
        class RESOURCES_DISCOVERED goodState
        class SCHEDULED goodState
        class RUNNING goodState
        class COMPLETED goodState
        class COMPLETED_ACK goodState
        class FAILED_SAFE warningState
        class ERROR warningState
        class ROLLBACK warningState
        class FAILED_UNSAFE badState
        class FAILED_UNSAFE_ACK badState
        class FAILED_SAFE_ACK warningState

    [*] --> NEW
    NEW --> VALID: resolved & validated
    NEW --> FAILED_SAFE: validation failed

    VALID --> READY
    VALID --> BLOCKED_WAITING: awaiting conditions

    READY --> RESOURCE_DISCOVERY
    RESOURCE_DISCOVERY --> RESOURCES_DISCOVERED

    RESOURCES_DISCOVERED --> SCHEDULED
    SCHEDULED --> LOCKING
    LOCKING --> LOCKED

    LOCKED --> RUNNING
    RUNNING --> COMPLETED: all steps done
    RUNNING --> ERROR: step failed

    ERROR --> FAILED_SAFE: nothing non-pure executed
    ERROR --> ROLLBACK: all executed non-pure FBs reversible
    ERROR --> FAILED_UNSAFE: irreversible FB executed

    ROLLBACK --> FAILED_SAFE: rollback succeeded
    ROLLBACK --> FAILED_UNSAFE: rollback failed

    COMPLETED --> COMPLETED_ACK: DB updates applied, locks released
    FAILED_SAFE --> FAILED_SAFE_ACK: locks released
    FAILED_UNSAFE --> FAILED_UNSAFE_ACK: locks RETAINED

    COMPLETED_ACK --> [*]
    FAILED_SAFE_ACK --> [*]
    FAILED_UNSAFE_ACK --> [*]

States in Detail

Submission Phase

NEW : The execution has been created. The engine resolves function block references, validates the workflow definition against the schema, and checks that all referenced function blocks are registered. - Success → VALID - Failure (unresolvable FB, invalid schema) → FAILED_SAFE

VALID : The workflow is structurally correct and all dependencies are resolved. - Normally → READY - If external conditions required → BLOCKED_WAITING (reserved for future use)

Acquisition Phase

READY : The execution is ready to acquire resources. This is the entry point for the resource discovery phase. - → RESOURCE_DISCOVERY

RESOURCE_DISCOVERY : The engine traverses the workflow tree and creates ACQUIRE jobs for each step that defines entity requirements. Workers execute the function block’s acquire() methods and return the entities needed. - → RESOURCES_DISCOVERED

RESOURCES_DISCOVERED : All entity requirements have been collected. The engine knows which devices, interfaces, and groups are needed. - → SCHEDULED

Locking Phase

SCHEDULED : The execution is ready for locking. If the required entities are locked by another execution, the workflow waits here. - → LOCKING

LOCKING : The engine has sent a lock request to the CMS for all required entities. - Success → LOCKED - Failure → FAILED_SAFE

LOCKED : All entities are exclusively locked. No other workflow can modify them until this execution completes. - → RUNNING

Execution Phase

RUNNING : Steps are actively executing. The engine creates EXECUTE jobs on the blackboard, processes results, and advances through the step tree. - All steps complete → COMPLETED - Any step fails → ERROR

Terminal States

COMPLETED : All steps finished successfully. DB updates are applied to the CMS, locks are released.

ERROR : A step failed. The engine decides how the execution terminates based on what actually reached a worker — a job never handed to a worker provably never ran and does not count as executed: - No executed step was non-pure → nothing to revert → FAILED_SAFE - Every executed non-pure function block is registered reversible → ROLLBACK - Any executed non-pure function block is irreversible → rollback is all-or-nothing, so none is attempted → FAILED_UNSAFE

ROLLBACK : The engine reverses the executed non-pure steps by pushing ROLLBACK jobs to the blackboard wave by wave in descending execution-wave order — the reverse of the deterministic traversal that created the original jobs. Jobs within a wave roll back in parallel; the next (lower) wave starts only once the whole wave succeeded. - All rollback jobs succeed → FAILED_SAFE - Any rollback job fails → the phase halts → FAILED_UNSAFE

FAILED_SAFE : The execution failed with no remaining side effects: either nothing non-pure executed, or everything that did was successfully rolled back. Safe to retry or discard.

FAILED_UNSAFE : The execution failed with potential side effects. Manual review required, and the execution’s entity locks are deliberately kept (see below).

Locks are NOT released on an unsafe failure

COMPLETED and FAILED_SAFE release the execution’s CMS locks on their way to _ACK. FAILED_UNSAFE does not — this is the one deliberate asymmetry in the lifecycle.

An unsafe failure means the state of the affected entities can no longer be derived from the engine’s records: a non-pure step may have reached the network, and nothing proved otherwise. Releasing the locks would let the next workflow plan against that unknown state. So the locks stay held, the execution’s cmsResourcesLocked flag stays true, and the handler logs at error level that it is retaining them:

Workflow failed unsafely — CMS resource locks RETAINED
    workflowId=<uuid> lockReference=<uuid> lockedDevices=[...] ...

Consequences an operator needs to know:

  • A later workflow that needs one of those entities cannot lock it. It stays in LOCKING and retries on a capped exponential backoff (5 minutes at most between attempts), indefinitely — it never spins hot and never wedges the engine, and the retry is deliberate: once the lock is released, the workflow resumes on its own with no re-submission. The CMS refusal is recorded on the execution as its retryNote.
  • The CompositeLock in the CMS is created without an expiry, so nothing sweeps it up. It is released only when a human releases it out-of-band.
  • There is no engine-side unlock/acknowledge API yet. Releasing the lock today means deleting the CMS CompositeLock whose reference is the execution UUID.

FAILED_UNSAFE_ACK is still terminal. _ACK means the engine is done with this execution, not the execution was cleaned up. Keeping it terminal is what stops the engine from reprocessing the execution every epoch and keeps it out of the active-execution and license counts.

Acknowledged States (the real terminal states)

COMPLETED, FAILED_SAFE and FAILED_UNSAFE are outcome states: reaching one means the engine still has work to do (apply DB updates, release locks — or, for FAILED_UNSAFE, deliberately not). Each is followed by its acknowledged counterpart, and those three are the states an execution actually rests in:

COMPLETED_ACK / FAILED_SAFE_ACK / FAILED_UNSAFE_ACK : The engine is finished and the execution is final. Nothing transitions out of these. Note that FAILED_UNSAFE_ACK still holds its entity locks.

Poll for the _ACK state, not the outcome state

When you wait for an execution to finish, treat completed_ack, failed_safe_ack and failed_unsafe_ack as the finish line. completed is transient — an execution seen in it is still releasing locks and writing entity updates.

Monitoring Executions

Via API

# List active executions
curl http://localhost:3030/workflow-execution/active

# Get a specific execution
curl http://localhost:3030/workflow-execution/id/<execution-id>

# List executions by state
curl http://localhost:3030/workflow-execution/state/RUNNING

Via Monitor App

The Monitor App’s Executions view shows all executions with their current state, progress (jobs completed/total), and drill-down to individual job results.

Stuck Execution Recovery

If an execution appears stuck:

Symptom Likely cause Resolution
Stuck in SCHEDULED Entities locked by another execution Wait or abort the blocking execution
Stuck in RUNNING Worker crashed or disconnected Wait for stuck job timeout (12 min default), then jobs are auto-failed
Stuck in LOCKING CMS unreachable Check CMS connectivity, restart engine
Stuck in LOCKING, CMS healthy An entity is still locked by an earlier FAILED_UNSAFE_ACK execution, which keeps its locks by design (see the retryNote) Review that execution, correct the network state, then release its CMS lock (reference = its execution UUID) — the waiting workflow resumes on its own

Executions can be aborted:

curl -X DELETE http://localhost:3030/workflow-execution/<execution-id>

Limited implementation

The abort endpoint exists but has limited functionality. It may not cleanly stop in-flight jobs or release locks in all scenarios. Use with caution in production.