Skip to content

Workflow as a Transaction

Network operations are inherently risky. A config push might succeed on half your devices and fail on the rest. A firmware upgrade might complete but the verification step might time out. Traditional scripts leave you guessing: what actually happened?

The workflow engine borrows from database transaction theory to bring predictability to this chaos. Every workflow execution is tracked as a transaction with well-defined failure semantics.

The Core Idea

Group a sequence of operations so the engine always knows the state of the world:

  • What ran – which steps executed on which devices
  • What changed – which steps had side effects (configuration writes, entity modifications)
  • What’s safe – whether the execution can be retried, rolled back, or needs manual intervention

The engine achieves this through the pure and idempotent contracts on function blocks, combined with entity locking and atomic state transitions.

ACID-Like Properties

Property How neops implements it
Atomicity Operations within a workflow are tracked as a logical unit. DB updates are applied atomically on completion.
Consistency Workflow definitions are validated before execution. Parameter schemas are checked. Assertions can guard steps.
Isolation Entity locking prevents concurrent workflows from modifying the same devices, interfaces, or groups.
Durability Every state transition is persisted. Results, logs, and DB updates survive restarts.

What this means for you

  • If your workflow only reads data before failing → the engine tells you nothing changed (FAILED_SAFE)
  • If your workflow wrote some config before failing → the engine tells you manual review needed (FAILED_UNSAFE)
  • No more guessing what state your network is in after a failure

Failure Classification

When a workflow fails, the engine classifies the failure based on what actually executed:

stateDiagram-v2
    classDef goodState stroke:green
    classDef warningState stroke:orange
    classDef badState stroke:red

    [*] --> NEW
    NEW --> VALID
    NEW --> FAILED_SAFE
    VALID --> LOCKED: acquire, lock
    LOCKED --> RUNNING
    RUNNING --> COMPLETED
    RUNNING --> FAILED_SAFE
    RUNNING --> FAILED_UNSAFE
    COMPLETED --> [*]
    FAILED_SAFE --> [*]
    FAILED_UNSAFE --> [*]

    class NEW goodState
    class VALID goodState
    class LOCKED goodState
    class RUNNING goodState
    class COMPLETED goodState
    class FAILED_SAFE warningState
    class FAILED_UNSAFE badState

Simplified

This view collapses the acquire/lock phases and omits the acknowledged terminal states. See Execution Lifecycle for the full state machine.

FAILED_SAFE

The workflow failed, but nothing irreversible happened. This occurs when:

  • The workflow failed before any steps executed
  • All executed steps were pure (read-only) – no side effects at all
  • The workflow was successfully rolled back

You can safely discard or re-run this workflow without worrying about leftover state.

FAILED_UNSAFE

The workflow failed and some side effects may have occurred. This happens when:

  • A non-pure step that cannot be rolled back (an irreversible function block) executed before the failure
  • The rollback itself failed
  • An external system was modified in a way that cannot be automatically undone

Manual review is required to assess and correct the state.

An unsafe failure keeps its entity locks

This is the one failure path that does not release the entities it locked. The engine can no longer derive what state those devices are in, so it refuses to let the next workflow plan against them: the locks stay held until a human has reviewed the execution and released them out-of-band. See Execution Lifecycle.

The Decision Tree

graph TD
    Fail["Workflow step failed"] --> AnyExec{"Did any non-pure<br/>step execute?"}
    AnyExec -->|No| Safe1["FAILED_SAFE"]
    AnyExec -->|Yes| AllRev{"Are all executed non-pure<br/>steps reversible?"}
    AllRev -->|Yes| RB["Roll back<br/>(reverse order)"]
    RB -->|Succeeded| Safe2["FAILED_SAFE"]
    RB -->|Failed| Unsafe1["FAILED_UNSAFE"]
    AllRev -->|No| Unsafe2["FAILED_UNSAFE"]

Auto-retry not yet implemented

Automatic retries for idempotent steps are still on the roadmap: the engine tracks isIdempotentExecution but does not yet use it for retry decisions. Today a failed execution goes straight to the rollback decision shown above.

Entity Locking

Before execution, the engine acquires exclusive locks on all entities in the workflow’s scope:

  1. Acquisition – The engine queries the CMS for the required entities
  2. Locking – The CMS grants exclusive write locks
  3. Execution – Steps run against the locked entities
  4. Release – On completion (COMPLETED) or a safe failure (FAILED_SAFE), locks are released and DB updates are applied atomically. FAILED_UNSAFE is the exception: its locks are retained

This prevents two workflows from modifying the same device simultaneously. If a device is already locked by another workflow, the new workflow waits in SCHEDULED state until the lock becomes available.

Locking granularity

Locks are per-entity: a workflow locking device A does not block a different workflow operating on device B. Locks are exclusive – only one workflow can hold a lock on a given entity at a time.

Practical Implications

For workflow authors:

  • Mark read-only function blocks as isPure: true – this gives the engine maximum flexibility for failure classification
  • Implement rollback() and register function blocks as reversible: true wherever possible – rollback is all-or-nothing, so a single irreversible step in the executed set forces FAILED_UNSAFE, while an all-reversible set is rolled back automatically to FAILED_SAFE
  • Mark config-push function blocks as isIdempotent: true if re-running them is safe (declarative config, not append-based) – this will enable automatic retries when that feature is implemented
  • Order steps so pure (read-only) steps execute first – if the workflow fails during the pure phase, it is automatically FAILED_SAFE

For operators:

  • FAILED_SAFE workflows can be safely ignored or re-triggered; their entity locks are released, so subsequent workflows are not blocked
  • FAILED_UNSAFE workflows need investigation – check the execution details in the Monitor App to see which steps ran on which devices
  • An unsafe failure blocks subsequent workflows on the same entities until its locks are released by hand. That is deliberate: the alternative is a workflow planning against a network state nobody has verified. Blocked workflows wait on capped backoff and resume automatically once the lock is released