Execution Lifecycle
Every workflow execution transitions through a well-defined set of states. Understanding these states helps you monitor executions, debug failures, and design robust workflows.
TL;DR
A workflow execution moves through: NEW → VALID → READY → acquire entities →
LOCKED → RUNNING → COMPLETED (or FAILED_SAFE / FAILED_UNSAFE), and then
to the matching _ACK state once the engine has finished with it — that is
where an execution comes to rest. A FAILED_SAFE means nothing irreversible happened;
FAILED_UNSAFE means manual review is needed — and it is the one path that keeps its
entity locks instead of releasing them, until human-driven remediation turns it into
FAILED_SAFE (unless the CMS no longer holds them: lost locks).
State Machine
stateDiagram-v2
classDef goodState stroke:green
classDef warningState stroke:orange
classDef badState stroke:red
class NEW goodState
class READY goodState
class VALID goodState
class LOCKING goodState
class LOCKED goodState
class BLOCKED_WAITING goodState
class RESOURCE_DISCOVERY goodState
class RESOURCES_DISCOVERED goodState
class SCHEDULED goodState
class RUNNING goodState
class COMPLETED goodState
class COMPLETED_ACK goodState
class FAILED_SAFE warningState
class ERROR warningState
class ROLLBACK warningState
class FAILED_UNSAFE badState
class FAILED_UNSAFE_ACK badState
class FAILED_SAFE_ACK warningState
[*] --> NEW
NEW --> VALID: resolved & validated
NEW --> FAILED_SAFE: validation failed
VALID --> READY
VALID --> BLOCKED_WAITING: awaiting conditions
READY --> RESOURCE_DISCOVERY
RESOURCE_DISCOVERY --> RESOURCES_DISCOVERED
RESOURCES_DISCOVERED --> SCHEDULED
SCHEDULED --> LOCKING
LOCKING --> LOCKED
LOCKED --> RUNNING
RUNNING --> COMPLETED: all steps done
RUNNING --> ERROR: step failed
ERROR --> FAILED_SAFE: nothing to roll back
ERROR --> ROLLBACK: all ledger entries reversible
ERROR --> FAILED_UNSAFE: irreversible FB executed
ROLLBACK --> FAILED_SAFE: rollback succeeded
ROLLBACK --> FAILED_UNSAFE: rollback failed / blocked by a human-only entry
COMPLETED --> COMPLETED_ACK: DB updates applied, locks released
COMPLETED --> FAILED_SAFE: CMS refused the DB updates, no side effects (write_refused)
COMPLETED --> FAILED_UNSAFE: CMS refused the DB updates, side effects (write_refused)
FAILED_SAFE --> FAILED_SAFE_ACK: locks released
FAILED_UNSAFE --> FAILED_UNSAFE_ACK: locks RETAINED
FAILED_UNSAFE_ACK --> ROLLBACK: remediation rollback all / continueRollback
FAILED_UNSAFE_ACK --> FAILED_SAFE: remediated (ledger complete)
FAILED_UNSAFE_ACK --> COMPLETED_ACK: write-back retried, CMS applied the DB updates
COMPLETED_ACK --> [*]
FAILED_SAFE_ACK --> [*]
FAILED_UNSAFE_ACK --> [*]
States in Detail
Submission Phase
NEW
: The execution has been created. The engine resolves function block references, validates the workflow definition against the schema, and checks that all referenced function blocks are registered.
- Success → VALID
- Failure (unresolvable FB, invalid schema) → FAILED_SAFE
VALID
: The workflow is structurally correct and all dependencies are resolved.
- Normally → READY
- If external conditions required → BLOCKED_WAITING (reserved for future use)
Acquisition Phase
READY
: The execution is ready to acquire resources. This is the entry point for the resource discovery phase.
- → RESOURCE_DISCOVERY
RESOURCE_DISCOVERY
: The engine traverses the workflow tree and creates ACQUIRE jobs for each step that defines entity requirements. Workers execute the function block’s acquire() methods and return the entities needed.
- → RESOURCES_DISCOVERED
RESOURCES_DISCOVERED
: All entity requirements have been collected. The engine knows which devices, interfaces, and groups are needed.
- → SCHEDULED
Locking Phase
SCHEDULED
: The execution is ready for locking. If the required entities are locked by another execution, the workflow waits here.
- → LOCKING
LOCKING
: The engine reads the CMS’s network zones, then sends a lock request to the CMS for all required entities. The run
keeps the zones it read as its snapshot; from here until the unlock it makes no CMS call
(Where the engine reads zones).
- Success → LOCKED
- A declared zone the CMS does not hold, or a lock the CMS refuses → FAILED_SAFE
- The zones cannot be read, or the entities are locked elsewhere → retried with backoff
LOCKED
: All entities are exclusively locked. No other workflow can modify them until this execution completes.
- → RUNNING
Execution Phase
RUNNING
: Steps are actively executing. The engine creates EXECUTE jobs on the blackboard, processes results, and advances through the step tree.
- All steps complete → COMPLETED
- Any step fails → ERROR
Terminal States
COMPLETED
: All steps finished successfully. DB updates are applied to the CMS, locks are released.
When the CMS refuses the DB updates it applies none of them, and the run fails with
write_refused without its updates:
- Every executed step was pure or reported that it changed nothing → FAILED_SAFE
- Otherwise the run has side effects, which the CMS now does not reflect → FAILED_UNSAFE.
The run keeps its updates for an operator to
retry the write-back; the engine never sends them again on
its own.
ERROR
: A step failed. The engine decides how the execution terminates based on what
actually reached a worker — a job never handed to a worker provably never ran
and does not count as executed, and neither does a step whose every attempt reported
that it changed nothing:
- No executed step was non-pure, or every one changed nothing, and no pure one
claimed side effects →
nothing to revert → FAILED_SAFE
- Every remaining non-pure function block is registered reversible → ROLLBACK
- Any remaining function block, non-pure or pure with a side-effects claim, is
irreversible → rollback is all-or-nothing,
so none is attempted → FAILED_UNSAFE
ROLLBACK
: The engine reverses the executed non-pure steps by pushing ROLLBACK jobs to the
blackboard wave by wave in descending execution-wave order — the reverse of the
deterministic traversal that created the original jobs. Jobs within a wave roll back
in parallel; the next (lower) wave starts only once the whole wave succeeded.
- All rollback jobs succeed → FAILED_SAFE
- Any rollback job fails → the phase halts → FAILED_UNSAFE
- Re-entered by remediation: resolved entries are skipped; an open irreversible entry,
or an entry whose rollback failed, that stops the walk → FAILED_UNSAFE, with the
blocking entries named in the reason
FAILED_SAFE : The execution failed with no remaining side effects: either nothing non-pure executed, every non-pure step reported that it changed nothing, everything else was successfully rolled back, or a human remediated an unsafe failure. A run whose DB updates the CMS refused ends here only when it has no side effects. Safe to retry or discard.
FAILED_UNSAFE : The execution failed with potential side effects. Manual review required, and the execution’s entity locks are deliberately kept (see below). This includes a run with side effects whose DB updates the CMS refused: the devices may hold what the CMS refused to record.
Locks are NOT released on an unsafe failure
COMPLETED and FAILED_SAFE release the execution’s CMS locks on their way to _ACK; a
FAILED_SAFE whose DB updates the CMS refused releases them without the updates.
FAILED_UNSAFE does not — this is the one deliberate asymmetry in the lifecycle. A run whose
locks the CMS no longer holds has none to keep (When the CMS no longer holds a run’s
locks).
An unsafe failure means the state of the affected entities can no longer be derived from the
engine’s records: a non-pure step may have reached the network, and nothing proved otherwise.
Releasing the locks would let the next workflow plan against that unknown state. So the locks
stay held, the execution’s cmsResourcesLocked flag stays true, and the handler logs at
error level that it is retaining them:
Workflow failed unsafely — CMS resource locks RETAINED
workflow_execution_id=<uuid> lockReference=<uuid> lockedDevices=[...] ...
Consequences an operator needs to know:
- A later workflow that needs one of those entities cannot lock it. It stays in
LOCKINGand retries on a capped exponential backoff (5 minutes at most between attempts), indefinitely — it never spins hot and never wedges the engine, and the retry is deliberate: once the lock is released, the workflow resumes on its own with no re-submission. The CMS refusal is recorded on the execution as itsretryNote, which stays while the workflow waits and is cleared once the retry that acquires the lock goes through, also when that retry runs after an engine restart. - The
CompositeLockin the CMS is created without an expiry, so nothing sweeps it up unless an operator gives it one in the CMS admin (“Refresh composite locks” sets an hour), after which the CMS’s expiry sweep releases it. It is released only through human-driven remediation (see below), or by releasing the CMSCompositeLock(reference= execution UUID) in the CMS admin (“Unlock composite locks”) as a break-glass measure. The engine is not told, and it does not ask: that nobody but the run releases a lock the run holds is an assumption (held-lock-is-released-only-by-its-execution). It learns of such a release only when the run releases its lock, and the run then ends as lost locks end. Until then the run goes on as if it held the lock: its forward jobs keep running, and so does a rollback, automatic or asked for, on devices another run may hold by then. Every such job acted after the CMS last confirmed the lock, so the run endsFAILED_UNSAFEwithout locks and owes the CMS check (below), however its steps ended. Release a run’s lock this way only to resolve the run afterwards: the run then ends unsafe, and a resolve confirming the CMS check ends it safe. - Finding the run that holds a lock does not need the logs. The run entry
(
GET /workflow-execution/id/<uuid>) carriescmsResourcesLocked, the resolvedcontext(devices with their hostnames) andheldLocks(the ids of the locked entities), and the paged run list (GET /workflow-execution/page?cmsResourcesLocked=true&target=<device id>) answers which run keeps this device locked.
Releasing and retiring a run’s lock
A run that holds its CMS lock releases it in two steps, each committed before the next:
- The terminal handler (
COMPLETED,FAILED_SAFE) sends the unlock with the run’s DB updates and records the CMS’s answer on the run:cmsResourcesLockedbecomesfalse,cmsLockReleasesays how the lock ended, andcmsWritesAppliedwhether the CMS holds the run’s DB updates. The status does not change yet. - The next pass retires the lock in the CMS (
retireLock), which from then on refuses any lock or unlock of the run’s reference withRETIRED, and only then acknowledges the run (_ACK).
A crash or a lost reply between any two steps of the terminal handler repeats only the steps not yet
committed. The CMS keeps every lock row, so a repeated unlock is answered from that history: when the
run’s own earlier unlock released the lock, the CMS says so (alreadyReleased) and applies nothing a
second time, and the run completes as if the first reply had arrived. Before it sends each unlock, the
engine commits on its own whether that unlock carries the run’s DB updates and the status its release
ends in (cms_unlock_attempt), so it can tell what an earlier unlock applied even when the transaction
that sent it rolled back. Which unlock the CMS answered is recorded on the run, in the transaction that
records the answer: an answer the run never committed counts as none, and the unlock’s outcome stays
unknown until the same unlock is sent again and answered (see An unlock of unknown outcome blocks
rollback and resolve). Only answers that say what became of the lock count: a
release, no lock held, WRITE_REFUSED (the lock still held), or RETIRED; an UNEXPECTED refusal does
not.
Every run that sent a lock request retires its reference before it is acknowledged, once it holds no
lock by its own record: its release is recorded, or no lock was ever granted. The engine commits that
it sends a lock request before it sends it (cms_lock_request), so a request whose transaction rolled
back still counts. A lock request that timed out on the engine’s side may still reach the CMS later.
Before the retire, core answers it with a new lock of the run’s reference, which the retire then
finds held (LOCK_HELD): the engine releases that lock with no DB updates, keeping its record of its
own release, and retires again. After the retire, core refuses it with RETIRED and takes nothing; a
retire of a reference with no lock row leaves a retired row that never held anything, so this holds
for a run whose lock request was never answered too.
cmsLockRelease |
Released by | cmsWritesApplied |
|---|---|---|
owner_unlock |
the run’s own unlock, this one or an earlier one whose reply was lost | this unlock: true with DB updates, false without. An earlier one: judged over the unlocks sent since the CMS last answered that it held the lock, as none before that released it: true when every one carried the updates, false when none did, null when they differ, as it cannot tell which went through |
admin, expiry_sweep, clear_all |
an operator, the CMS’s expiry sweep or its clear-all, without the run | false: none of the run’s DB updates were applied |
not_found |
unknown: the CMS has no record of the lock, or none of who released it (a row a core older than 2.7.0 left released without a cause reads RELEASED or RETIRED with releasedBy null) |
null when an unlock of the run carried DB updates, false when none did |
A lock row disappears only when the run never locked, or when prune_lock_history deletes a lock
retired at least a day before. A run with a lock answer recorded that has not retired its lock
should always find its row, so the engine logs not_found as an error: a core pod older than 2.7.0
or a restored CMS database would explain it. An unlock answered RETIRED means the run retired its
lock before and lost the record of it (a restored engine database), also logged as an error; the
answer still names how the lock was released, and the run records that as any release.
A run whose DB updates the CMS holds is never rolled back, and none of its steps is resolved without
the CMS correction (run-with-applied-writes-is-not-rolled-back). The CMS applies a run’s updates
only in an unlock of the run, the engine records every unlock before it sends it, and it takes no
rollback and no resolve while one of them has no recorded answer
(below), so the engine
learns that the updates were applied before it would undo anything, and the run completes with them.
One case stays outside the engine’s reach, and it assumes it away
(no-unlock-arrives-after-a-later-answer): an unlock still on its way that reaches the CMS after the
CMS answered a later unlock of the same run.
Should that ever happen, or the engine lose its record of an unlock, a backstop catches it at the
release: a FAILED_SAFE run whose release finds that an earlier unlock applied, or may have applied,
its DB updates ends FAILED_UNSAFE with unlock_failed and no locks to keep, and the engine logs an
error. The CMS then holds updates of a network state the run since rolled back or resolved, so the
CMS and the network are not known to agree. The end is decided in one place (endOnHeldWrites,
src/engine/core-engine/lifecycle/rollback/held-writes-end.ts). As for any run
without locks, rollback is refused. Such a run owes a
CMS correction before it may end safe: check the CMS against the devices and correct it where
needed, then resolve the run confirming it (below).
Both fields are served on the run entry (GET /workflow-execution/id/<uuid>). Runs that ended before
the fields existed serve null for both and were never retired. With no release recorded they never
owe a CMS correction: a run that ended unsafe without locks before the change ends safe on a plain
resolve, whatever the CMS holds.
When the CMS no longer holds a run’s locks
The engine learns that the CMS released a run’s locks without it only when it asks the CMS to release them and the CMS answers that it holds none, naming who released them, or that it has no record of them. The run’s reason names that cause: an operator, the expiry sweep or a clear-all released them, so the CMS applied none of the run’s DB updates; or the CMS has no record of them, and cannot say whether it applied any. From then on another run may hold those devices, so the engine rolls back none of the run’s steps and ends the run at once:
FAILED_UNSAFEwithunlock_failedwhile a step that may have changed a device is neither rolled back nor resolved, while the CMS has no record of the lock and an unlock of the run carried DB updates (cmsWritesAppliednull): the CMS may or may not hold them, or while a job of the run may have acted after the CMS last confirmed the lock (below). ACOMPLETEDrun with such steps ends this way instead ofCOMPLETED_ACK: the devices may hold what the CMS never recorded.FAILED_SAFEotherwise, with its DB updates dropped; its reason names the cause and says every step that may have changed a device is rolled back or resolved. AFAILED_SAFErun whose ledger is already compensated goes on toFAILED_SAFE_ACK, its reason noting the loss.
Either way its pending jobs are canceled, cmsResourcesLocked becomes false, and the run retires
the released lock before it is acknowledged, so nothing tries to release a lock again. A run reaches
FAILED_UNSAFE_ACK without locks either by ending FAILED_UNSAFE here or by learning of the loss on
the operator’s write-back retry (Retrying a refused write-back).
Either way, remediation/rollback answers 409 and continueRollback is refused, since rollback
jobs would change devices another run may lock (the ledger’s rollbackRefusal says why), and only a
resolve ends the run; when the CMS has no record of the lock and may hold the run’s DB updates, or a
job of the run acted after the CMS last confirmed the lock, that resolve must confirm the CMS check too
(below).
FAILED_UNSAFE_ACK is still terminal. _ACK means the engine initiates nothing further on its
own, not the execution was cleaned up. Keeping it terminal is what stops the engine from
reprocessing the execution every epoch and keeps it out of the active-execution and license counts.
Only remediation, which a human drives, moves it on.
When the CMS holds, or may hold, the run’s writes
A run whose lock came off while the CMS holds, or may hold, its DB updates (cmsWritesApplied true
or null once released: the CMS has no record of the lock, or, in the case that
should never happen, an earlier unlock carrying them released the lock of a
run whose steps were rolled back or resolved since) describes in the CMS a network
state its devices no longer have, or one nobody can tell. Resolving its steps does not change that,
so the run stays FAILED_UNSAFE_ACK until an operator attests that the CMS was checked against the
devices and corrected where needed:
curl -X POST http://localhost:3030/workflow-execution/<id>/remediation/resolve \
-H 'Content-Type: application/json' \
-d '{"all": true, "note": "set the hostname back in the CMS", "cmsCorrected": true}'
The ledger serves cmsCorrectionOwed: true while it is owed, with its cause in cmsCorrectionCause
(writes, acted_without_lock as below,
or both) and the steps that acted in actedWithoutLockSteps, and cmsCorrection (who, when, the
note) once attested. A resolve that would otherwise end such a run FAILED_SAFE answers 409
naming what is still owed, and records nothing; cmsCorrected is refused too on a run that owes no
correction, and while a step stays neither rolled back nor resolved after the request, as the
devices the CMS must describe are not settled yet. One function decides what a run still owes before
it may be called safe (stillOwed, src/engine/core-engine/lifecycle/rollback/still-owed.ts): the
ledger entries and the CMS correction; the safe exit and the ledger read it, and the Monitor App reads the ledger.
The Monitor App offers such a run one act, Correct the CMS, then confirm.
When the run’s jobs acted after the CMS last confirmed its lock
The engine records on the run where the CMS last confirmed that it held the run’s lock
(cmsLockConfirmedOrder): the grant, or an unlock the CMS refused with the lock held. It compares by order, not by clock: one
database sequence (lock_order) gives the confirmation its place when the engine asks, before the
call, and every job result its place when it is inserted, so a result with a lower place was written
before the engine asked. When a release then shows that someone other than the run released
the lock (an operator, the expiry sweep, a clear-all), or that the CMS has no record of it, every job
of the run that may act on a device and whose worker’s result has no lower place than that
confirmation may have acted after the release, on
devices another run held by then: a rollback job, an attempt of a non-pure step that is not
left out, or an attempt of a pure step that
claimed side effects. A rollback may have undone another
run’s change that the CMS now records, so the CMS and the network are not known to agree, even with
every step rolled back. The run ends FAILED_UNSAFE without locks, its reason naming those steps:
Another run may have held the devices while <steps> acted, as the CMS last confirmed the run's lock
before they did. Here the CMS may be right and the devices wrong: another run’s changes on these
devices may have been undone, so restore them on the devices or correct the CMS, then resolve with
cmsCorrected: true, the same attestation as above. A run that owes both says both. A run none of whose jobs acted since the
last confirmation, or whose steps were all pure or left out, ends as its ledger and DB updates say.
When the run’s lock was taken again
Core names the lock row every answer is about: the grant (lock.id), the unlock and the
retire (lockId). One row is one continuous hold; a lock released and taken again, by anyone, is a
new row with a new id. The engine records the row the CMS granted the run (cmsLockId on the run
entry, the id the CMS admin’s lock list and core’s lock id <id> log lines show), asks every unlock
of the run’s lock to name it, and compares the row each answer names with it: an
answer about another row, or one that names none, never confirms the run’s lock nor counts as its
release, and is read as the run’s lock gone with no record of how (not_found): the run ends as
lost locks end. When that release carried DB updates, or
an unlock of the run did since the CMS last showed the lock held, it ends FAILED_UNSAFE without
locks, owing the CMS check, since the CMS may have applied them under that other row.
This needs neops-core 2.8.0 or later: /health refuses a core that does not serve
the row argument and answer fields (cms-requirements.ts).
A lock request of the run that outlived its timeout on the engine’s side can reach core after an
operator released the run’s lock, and take the devices again, while they are free, in a new row
under the same reference. Without the row, an unlock
would apply the run’s DB updates under the newer lock, although another run may have locked and
changed the devices in between. Naming the row, core refuses the unlock with LOCK_REPLACED,
applying nothing and keeping the newer lock, with how the run’s row ended. The run records that release (admin, expiry_sweep,
clear_all, or owner_unlock when its own unlock whose reply was lost released it) and ends as that
answer ends it: an owner_unlock answer means an unlock of the run whose reply was lost applied its
DB updates and released the run’s row, exactly as an alreadyReleased answer does, and the run ends
as that release ends: a completed run completes with them; a failed-safe run, which should never get
this answer, ends FAILED_UNSAFE without locks and owes the CMS check (endOnHeldWrites,
above).
The newer lock is never used: the retire finds it held and releases it with no
DB updates first, naming that newer row. A run locked before the engine recorded the row names none,
compares nothing, and behaves as before.
The engine does not compare the jobs with core’s unlockedAt, so a release that came after the
last job returned still ends the run this way; comparing them, with a margin for clock skew, could
let such a run end safe.
Remediation: the way the locks come off
The remediation API walks the execution’s compensation ledger — the executed non-pure steps,
except those that changed nothing, and the
pure ones that claimed side effects, each of
which must end rolled back (a worker ran the function block’s rollback()) or resolved
(a human attested the fix with a mandatory note; the engine records the request’s authenticated
principal as resolvedBy / resolvedByUsername). The moment every entry is compensated, the execution transitions to FAILED_SAFE:
- from
ROLLBACKwhen a rollback phase re-entered byrollback {"all": true}orcontinueRollbackcompletes the ledger; - from
FAILED_UNSAFE_ACKwhen a resolve, or the success of a single-entry rollback, completes it.
From there the ordinary failed-safe path takes over: the engine releases the retained CMS
locks, if any are left (When the CMS no longer holds a run’s
locks), and acknowledges the execution as FAILED_SAFE_ACK; blocked workflows resume on their own.
The reason records how the ledger was compensated (see Reason notes).
A CMS outage during that release is retried like any other failed-safe unlock, and an engine
restart in between resumes it, because FAILED_SAFE is not an _ACK state. A remediated execution
releases its locks without applying any CMS updates the run had aggregated; the operator reconciles
the CMS by hand where an attestation implies it.
# The ledger: every executed step that may have changed something, and its compensation state
curl http://localhost:3030/workflow-execution/<id>/remediation
# Roll back entries (each must be reversible and wave-gated), or walk all waves
curl -X POST http://localhost:3030/workflow-execution/<id>/remediation/rollback \
-H 'Content-Type: application/json' -d '{"jobs": ["<entry-job-uuid>"]}'
curl -X POST http://localhost:3030/workflow-execution/<id>/remediation/rollback \
-H 'Content-Type: application/json' -d '{"all": true}'
# Attest entries fixed by hand; completing the ledger moves the execution to FAILED_SAFE in-request
curl -X POST http://localhost:3030/workflow-execution/<id>/remediation/resolve \
-H 'Content-Type: application/json' \
-d '{"all": true, "note": "network state restored and verified"}'
# Remediate and continue rollback: attest the blocker, and let the engine reverse the waves below it
curl -X POST http://localhost:3030/workflow-execution/<id>/remediation/resolve \
-H 'Content-Type: application/json' \
-d '{"jobs": ["<entry-job-uuid>"], "note": "flash restored by hand", "continueRollback": true}'
- Two grants, one per act. Requesting a rollback needs
workflow-execution-rollback:write; resolving needsworkflow-execution-resolve:write, and so does retrying a refused write-back, which also releases the locks on an operator’s word. Neither implies the other: a rollback asks the engine to reverse a step on the devices, while a resolve releases the retained locks on a human’s word about entities whose state the engine cannot derive. A resolve with"continueRollback": truealso restarts the rollback, so it needs both grants; without the rollback grant it answers403namingworkflow-execution-rollback:write, and nothing is recorded. Reading the ledger needsworkflow-execution:read. - Wave order is enforced for automation: an entry may be rolled back only when every entry in
a strictly higher execution wave is already compensated — resolving the irreversible step of a
high wave is what licenses automation to reverse the waves beneath it. Resolving itself has no
wave gate.
rollback {"all": true}puts the execution back intoROLLBACK: the automatic rollback phase walks the remaining waves descending, skipping resolved entries, and endsFAILED_SAFEonce the ledger is complete orFAILED_UNSAFE— back toFAILED_UNSAFE_ACK, where remediation continues — when a rollback fails or an entry only a human can compensate blocks the walk (the reason names it). The request itself answers409when nothing is left to roll back, when a wave is still in flight, or when the walk is blocked from the start. - Rolling back single entries (
{"jobs": [...]}) keeps the execution inFAILED_UNSAFE_ACK. - A failed rollback attempt leaves the entry
ROLLBACK_FAILED, open to another attempt: name it in{"jobs": [...]}to run its rollback again, or resolve it by hand instead. A walk never re-runs it on its own; only the entries an operator names are run again. While the current wave holds such an entry uncompensated,rollback {"all": true}answers409naming it (name it to re-run, or resolve it), and a resolve withcontinueRollbackrecords its attestation but does not enter the walk, and its response says why. The function block’s second rollback sees the earlier attempts inpreviousRollbacksand decides what to do. - Remediate and continue rollback (
"continueRollback": trueon resolve) attests the entry and, in the same request, puts the execution back intoROLLBACKwhen the attestation unblocked a wave, so the engine carries on wave by wave without a second call. If the current wave still has a single-entry rollback in flight, the phase waits for it and continues on its success. If another entry only a human can compensate still blocks the walk, the attestation is recorded and the execution stays inFAILED_UNSAFE_ACK;createdRollbackJobsand the ledger’sstatussay which of these happened. - An execution whose ledger is empty (it failed unsafely before any non-pure step reached a
worker, or every one that did changed nothing) is vacuously complete:
resolve {"all": true}on it is the operator act that moves it toFAILED_SAFE. Its reason then recordsremediation: resolved by <user>instead ofremediated: …. - The ledger also lists, under
leftOut, the steps it leaves out because every attempt reportedsideEffects: false: one item per step and entity, withstepLabel,entityId,functionBlockIdand itsattempts, each as{jobUuid, status, sideEffects}. They are not entries: they need no compensation and do not count towardcomplete. - The ledger stays readable during and after remediation:
GET .../remediationanswers forFAILED_UNSAFE_ACK,ROLLBACK,FAILED_SAFEandFAILED_SAFE_ACKexecutions, so the compensated entries and their attestations remain the audit trail. The mutating routes answer409for anything butFAILED_UNSAFE_ACK— duringROLLBACK, because a rollback is in progress.
An unlock_failed run that did only CMS work has an empty ledger
Declared CMS updates (dbUpdates) are not side effects, so a run whose steps were all pure
ends with an empty ledger even when the CMS write-back of those updates failed (unsafeCause
unlock_failed). The unlock that failed has no recorded answer, so a resolve answers 409
until you retry the write-back,
which sends that unlock again: the updates may or may not have been applied, and its answer
says which.
Retrying a refused write-back
A run with side effects whose DB updates the CMS refused
(unsafeCause write_refused) keeps its updates. Once the refusal is recorded, the engine’s own unlocks never
send them: not the release after a remediation, not a retry, not after a restart. Only an
operator sends them again, once the cause is fixed in the CMS:
The engine sends the kept updates in one unlock that also releases the locks:
- The CMS applies them: the locks come off and the run moves to
COMPLETED, which retires the lock and acknowledges it asCOMPLETED_ACK, the end it would have reached had the first write-back gone through. Its reason records the retry. An earlier retry whose reply was lost counts: the CMS answers that it already applied the updates and applies nothing again. - The CMS refuses them again: the run stays
FAILED_UNSAFE_ACKwith its locks, and its reason records the new refusal. Retry again after another fix, or roll back or resolve. - The CMS neither applies nor refuses them (unreachable, or it refuses the call itself): the
request answers
503or502and the run is unchanged. Nothing retries in the background; send the request again. A503says the outcome is unknown: the CMS may have applied the updates and lost the reply, or may still be applying them. Until a retry is answered, rollback and resolve answer409(below); the next retry sends the same unlock again, and if the CMS takes the updates then, or took them before, the run completes with them. A503also comes from the engine itself while it changes epochs (The engine is changing epochs; retry the write-back.); the run is unchanged then too. A500means the engine failed around the call: the CMS may or may not have applied the updates, so check it before you send the request again. - The CMS no longer holds the run’s locks: it took nothing this time. The request answers
409, and the run records who released them, so the ledger stops offering a retry, and rollback is refused as for every run without locks. The run passes throughFAILED_UNSAFEto retire the released lock and rests inFAILED_UNSAFE_ACKagain. Check the devices, then resolve the run. When the CMS has no record of the locks it may hold the writes: the answer says to check the CMS against the devices and correct it where needed, then resolve the run confirming the correction (below), and not to run the workflow again to apply the writes: that could apply them twice.
A retry of a refused write-back is taken only while every ledger entry is still open. Once a rollback job exists or an
entry is resolved, the devices no longer hold what the updates describe, so the route answers
409, the updates are dead, and the run ends the usual way: FAILED_SAFE once the ledger is
complete, releasing the locks without the updates. The ledger’s writeBackRetryable says whether
the route takes a retry now. A run that ended FAILED_SAFE with write_refused has no retry:
nothing on the devices needs the updates, so fix the cause and run the workflow again.
An unlock of unknown outcome blocks rollback and resolve
The engine records each unlock before it sends it, with the DB updates it carries and the status its
release ends in, and on the run which unlock the CMS last answered about the run’s lock. An unlock
with no recorded answer has an unknown outcome: the CMS may have released the run’s locks and taken
any updates it carried. On an acknowledged unsafe run that is a write-back retry whose reply was lost or that is
still on its way, an automatic release whose call the CMS refused (unlock_failed, after a completed
or a failed-safe run), or an answer whose transaction never committed. The ledger says so
(unknownUnlockOutstanding).
While it is unknown, remediation/rollback and remediation/resolve answer 409 and change nothing
(unknown-unlock-blocks-remediation); the engine sends no unlock for them. The operator sends that
same unlock again with the write-back route, whatever sent it first
(retry-completes-unknown-unlock):
The engine sends the recorded unlock, with the DB updates it carried, never the run’s current ones. The CMS takes one unlock of a run’s lock at a time, so a copy of the first one that reached the CMS goes through first and the answer is final for it:
- The lock is released, by this unlock or the earlier copy: the run goes back to the status of the
release that sent the unknown unlock and ends as that release ends:
COMPLETED_ACKwith its writes for a write-back retry or a completed run’s release,FAILED_SAFE_ACKfor a failed-safe run’s release. When someone else released it, or the CMS has no record of it, the request answers409and the run ends as lost locks end. - The CMS refuses the updates (
WRITE_REFUSED): the lock is held and nothing was applied. Rollback and resolve are taken again. A run that endedunlock_failedwith DB updates now ends as a refused write-back does (write_refused), so the retry stays open once the cause is fixed: it passes throughFAILED_UNSAFEtoFAILED_UNSAFE_ACKagain, and the response shows the status before that acknowledgement. With no side effects it endsFAILED_SAFEinstead: its updates are dropped and its locks released without them (applied: false,status: failed_safe). - The CMS refuses the engine’s call itself:
502. This call took nothing, and the earlier unlock is still unknown, so rollback and resolve keep answering409. Fix the credential the engine calls the CMS with (NEOPS_CMS_TOKEN) or what it is granted in the CMS, then send the request again. - The CMS does not answer, or says nothing certain about the lock:
503. Nothing changed; send the request again. Nothing in the engine settles an unknown unlock while the CMS stays silent: the run keeps its locks and takes no rollback or resolve until the CMS answers. If it lasts, collect what operations lists for support.
One case stays open: the CMS refuses the updates sent again while the first copy is still queued in
front of core, and that copy passes later because the CMS data changed in between
(no-unlock-arrives-after-a-later-answer assumes it away).
The ROLLBACK phase never meets an unknown unlock: the engine sends none before a run is COMPLETED
or FAILED_SAFE, a request that re-enters the phase passed the check above, and no route sends an
unlock for a run in ROLLBACK.
Reason notes
Remediation and the rollback phase append notes to the execution’s reason, separated by |.
An engine note already present is not appended again, so repeated phase passes and restart
replays converge on the same reason. An operator act (remediation: …) is appended every time,
so an entry re-run by name after a failed wave leaves a second trace. A reason longer than its
10 000-character column loses its middle (…[truncated]…): the original failure cause at the
head and the latest notes at the tail are kept.
| Note | Appended when |
|---|---|
Rollback failed at step <label>: <message> |
A rollback attempt failed — in the ROLLBACK phase (automatic or re-entered by remediation) or as a single-entry remediation rollback in FAILED_UNSAFE_ACK |
Rollback at step <label> never reached a worker: <message> |
The pending-job cleanup failed a rollback attempt no worker ever polled. The attempt never ran, so its entry stays OPEN; in the ROLLBACK phase it still halts the walk |
Rollback blocked pending human resolution of: <entries> |
The ROLLBACK phase has no rollbackable wave left while the ledger is incomplete |
remediation: rollback-all requested by <user> |
rollback {"all": true} re-enters ROLLBACK |
remediation: rollback continued by <user> |
A resolve with continueRollback re-enters ROLLBACK |
remediation: resolved by <user> |
A resolve completes an empty ledger |
remediation: CMS correction confirmed by <user> |
A resolve attests that the CMS was checked against the devices and corrected where needed (cmsCorrected) |
remediation: write-back retried by <user>; the CMS applied it |
A retried write-back completes the run |
remediation: write-back retried by <user>; the CMS refused it again: <message> |
The CMS refuses a retried write-back |
remediation: write-back retried by <user>; the CMS no longer held the run's locks, and this call took nothing: <cause> (the cause the CMS named) |
The CMS no longer holds the run’s locks when a write-back is retried |
An earlier unlock of the run released its locks with its writes, so the CMS holds writes of a network state the run since rolled back or resolved. This should never happen: the unlock reached the CMS after the CMS answered a later one of the run, or the engine lost its record of it. The run ends unsafe: check the CMS against the devices and correct it where needed, then resolve the run confirming the CMS correction |
The backstop: a FAILED_SAFE release finds that an earlier unlock carrying the DB updates already released the locks (cmsWritesApplied true); the run ends FAILED_UNSAFE |
An earlier unlock of the run released its locks, so the CMS may hold writes of a network state the run since rolled back or resolved. This should never happen: the unlock reached the CMS after the CMS answered a later one of the run, or the engine lost its record of it. The run ends unsafe: check the CMS against the devices and correct it where needed, then resolve the run confirming the CMS correction |
The same when the unlocks since the lock was last held carried the updates and none, or none is recorded, so it cannot tell which went through (cmsWritesApplied null) |
remediation: write-back retried by <user>. The unlock of unknown outcome was sent again: the CMS released the workflow execution's locks with its writes (or with none of its writes) |
A write-back retry sent an unlock of unknown outcome again, and the CMS released the locks, by that unlock or the earlier copy |
remediation: write-back retried by <user>; the CMS refused the writes sent again: <message> |
The CMS refuses the updates of an unknown unlock sent again that was not a write-back retry before; the reason of the refused write-back precedes it |
first failure of the release: <reason> |
Stands before the note above: the reason the execution ended with before the retry (Failed to unlock resources: …), kept because the refused write-back’s reason takes its place |
remediation: write-back retried by <user>. The unlock of unknown outcome was sent again. <reason> |
The same when the unlock sent again found the locks released by someone else or unknown to the CMS: the reason of lost locks follows |
remediated: <n> rolled back, <m> resolved by <users> |
The execution turns FAILED_SAFE after a human took part: an entry was resolved, or needed more than one rollback attempt. by <users> lists the resolving principals’ usernames and is omitted when there are none |
<user> is the acting principal’s username, or an anonymous principal when the credential
carries none.
Failure codes
A failed run carries the free-text reason, which describes what went wrong and may be reworded,
and a stable reasonCode a client can group, filter and phrase on. The code names the first
failure, the cause that started it; it is null for a run that did not fail. Codes are only added,
never renamed.
reasonCode |
The run failed because |
|---|---|
invalid_workflow |
the definition cannot run: duplicate step labels, no steps, an unknown step, no resolvable runOn, or a workflow reference step, which the engine does not run yet |
unresolved_reference |
the definition names a function block or workflow that is not registered |
invalid_parameters |
a step’s parameters break its function block’s parameter schema, or a networkZone in the run resolves to no zone (Network zones) |
unknown_network_zone |
a networkZone names a zone the CMS does not hold: when resource discovery starts (from the engine’s zone table, read once more from the CMS when it lacks the zone) or in the lock phase (from a fresh read of the CMS) the run fails before anything is locked; a job on an interface whose device the run does not hold fails its step (Network zones) |
interpolation_failed |
a {{ }} token could not be evaluated |
lock_refused |
the CMS refused or could not complete the lock on the run’s entities, the engine refused to lock against a CMS too old for it (GET /health says why), or the lock was never requested because the run’s starter maps to no CMS user |
acquire_failed |
resource discovery failed |
step_failed |
a worker reported a step’s job as failed |
invalid_result |
a step’s result breaks its function block’s result schema, its dbUpdates write outside the step’s entities, collide with another step’s or would be refused by the CMS for their shape or zone (Network zones), or a pure function block reported side effects |
assertion_failed |
a step’s assertion evaluated to false |
job_timed_out |
no worker claimed a job before the deadline, or a claimed job went silent |
worker_lost |
the worker holding a job was deleted or went offline |
unlock_failed |
the CMS write-back or the release of the run’s locks failed, or the release found the locks gone |
write_refused |
the CMS refused the run’s write-back and applied none of it. The run ends FAILED_SAFE when it has no side effects, else FAILED_UNSAFE, where an operator can retry the write-back |
aborted |
a person aborted the run (reserved: abort does not stop runs yet) |
engine_error |
an unexpected fault in the engine or its infrastructure |
A run that ended FAILED_UNSAFE also names why it could not be made safe, in unsafeCause:
unsafeCause |
Meaning |
|---|---|
rollback_failed |
a rollback job failed |
rollback_blocked |
an irreversible function block had executed, so no rollback was attempted |
unlock_failed |
the CMS write-back or lock release failed, or the release found the locks gone |
write_refused |
the CMS refused the write-back of a run with side effects, so the CMS does not hold what the devices may now hold |
engine_fault |
the engine failed while executing and cannot tell what the run had done, or a safety check refused what the engine was about to do, such as releasing the locks of a run whose ledger is not compensated |
A refused safety check ends the run there and then, in the engine’s own handling of the run: the
handler’s own transaction is rolled back, and the engine commits FAILED_UNSAFE with engine_fault
in a fresh one, appending to reason the property it would have broken. This applies only to a run
that may have touched the network, one that holds CMS locks or has started running; any other run
takes the ordinary error path. The run keeps whatever CMS locks it holds, its pending jobs are canceled,
it is acknowledged like any other unsafe end, and the operator remediates it through the compensation
ledger. On the operator’s write-back retry, a refused safety check answers 409 naming the property
and leaves the run as it was.
Every path into FAILED_UNSAFE names one, and a later unsafe end replaces it with its own: a
remediated run whose unlock the CMS then denies ends with unlock_failed, while reasonCode still
names the first failure. It is null for every run that did not end unsafe. A run remediated to
FAILED_SAFE keeps both codes: FAILED_SAFE or FAILED_SAFE_ACK with an unsafeCause is a run a
person made safe. So does a write_refused run that reached COMPLETED_ACK through a
retried write-back: a COMPLETED_ACK with both codes is a run
whose writes the CMS took only on an operator’s retry.
Acknowledged States (the real terminal states)
COMPLETED, FAILED_SAFE and FAILED_UNSAFE are outcome states: reaching one means the engine
still has work to do (apply DB updates, release locks — or, for FAILED_UNSAFE, deliberately not).
Each is followed by its acknowledged counterpart, and those three are the states an execution
actually rests in:
COMPLETED_ACK / FAILED_SAFE_ACK / FAILED_UNSAFE_ACK
: The engine is finished and initiates nothing further. COMPLETED_ACK and FAILED_SAFE_ACK
are final. FAILED_UNSAFE_ACK still holds its entity locks (unless the CMS no longer holds them:
lost locks), and remediation is the one way
out of it (to FAILED_SAFE, back into ROLLBACK, or, for a refused write-back, to
COMPLETED_ACK through a retry).
Poll for the _ACK state, not the outcome state
When you wait for an execution to finish, treat completed_ack, failed_safe_ack and
failed_unsafe_ack as the finish line. completed is transient — an execution seen in it
is still releasing locks and writing entity updates.
Monitoring Executions
Via API
# Executions not yet acknowledged, plus those that finished in the last 5 minutes
curl http://localhost:3030/workflow-execution/active
# Get a specific execution
curl http://localhost:3030/workflow-execution/id/<execution-id>
# List executions by state
curl http://localhost:3030/workflow-execution/state/RUNNING
Via Monitor App
The Monitor App’s Executions view shows all executions with their current state, progress (jobs completed/total), and drill-down to individual job results.
Stuck Execution Recovery
If an execution appears stuck:
| Symptom | Likely cause | Resolution |
|---|---|---|
Stuck in SCHEDULED |
Entities locked by another execution | Wait for the blocking execution to finish (aborting is not implemented) |
Stuck in RUNNING |
Worker crashed or disconnected | Wait for stuck job timeout (12 min default), then jobs are auto-failed |
Stuck in LOCKING |
CMS unreachable | Check CMS connectivity, restart engine |
Stuck in LOCKING, CMS healthy |
An entity is still locked by an earlier FAILED_UNSAFE_ACK execution, which keeps its locks by design (see the retryNote) |
Remediate that execution (GET /workflow-execution/<id>/remediation, then roll back / resolve its entries) — completing its ledger releases the lock and the waiting workflow resumes on its own |
Aborting is not implemented
DELETE /workflow-execution/<execution-id> authorizes the caller (the abort grant or the
principal who started the run) and then answers 501 Not Implemented. It stops nothing
and writes nothing; abortedBy stays empty. A run that cannot lock its entities waits in
LOCKING and resumes on its own once they are free.

