Skip to content

Retry & Rollback

The engine provides two mechanisms for handling step failures: automatic retries and rollback.

Retry Configuration

Not Yet Implemented

retryConfig is defined in the workflow schema and can be set on steps, but the execution engine does not yet consume it. Failed steps are not retried based on this configuration. This feature is planned for a future release.

Add retryConfig to a step to configure retry behavior on failure (when implemented):

- type: functionBlock
  label: push_config
  functionBlock: "fb.examples.neops.io/configureDevice:1.0.0"
  retryConfig:
    maxRetries: 3
    delay: 10
Field Required Description
maxRetries Yes Number of retry attempts (1-100)
delay No Seconds between attempts (0-600)
condition No JMESPath condition that must be true for retry

Planned Retry Behavior

  1. A step fails
  2. The engine checks if the step has retryConfig
  3. If retryCount < maxRetries, the engine waits delay seconds and creates a new job
  4. The retry job replaces the failed job (tracked via replaces/replacedBy references)
  5. If all retries fail, the step is marked as permanently failed

Repeat Configuration

Not Yet Implemented

repeatConfig is defined in the workflow schema and can be set on steps, but the execution engine does not yet consume it. Steps always execute once. This feature is planned for a future release.

Repeat a step multiple times (independent of failure), when implemented:

- type: functionBlock
  label: poll_status
  functionBlock: "fb.examples.neops.io/checkStatus:1.0.0"
  repeatConfig:
    repeats: 5
    delay: 60
    condition:
      type: jmes
      jmes: "{{ poll_status.result.data.status != 'ready' }}"
Field Required Description
repeats No Fixed number of repetitions (1-100)
delay No Seconds between iterations (0-600)
condition No Continue repeating while condition is true

Provide either repeats (fixed count), condition (repeat while true), or both (whichever limit is reached first). Use repeat for polling patterns (wait until a device reboots, check convergence).

Failure Classification

When a step fails (after retries are exhausted, once implemented), the engine classifies the overall workflow failure:

graph TD
    StepFail["Step failed"] --> COE{"continueOnError?<br/>(planned)"}
    COE -->|Yes| Continue["Continue to next step"]
    COE -->|No| Classify["Classify failure"]
    Classify --> AnyNonPure{"Did any non-pure<br/>step execute?"}
    AnyNonPure -->|No| FS["FAILED_SAFE"]
    AnyNonPure -->|Yes| AllRev{"All executed non-pure<br/>FBs reversible?"}
    AllRev -->|Yes| RB["ROLLBACK<br/>(wave by wave, reverse order)"]
    RB -->|all waves succeed| FS2["FAILED_SAFE"]
    RB -->|any rollback job fails| FU["FAILED_UNSAFE"]
    AllRev -->|No| FU2["FAILED_UNSAFE"]

Implementation status

The classification and rollback paths above are active. continueOnError is not yet enforced, and automatic idempotency-based retries are not yet implemented — the engine tracks isIdempotentExecution but does not yet use it for decisions.

Result Meaning Action
FAILED_SAFE No side effects remain (nothing non-pure executed, or everything that did was rolled back) Safe to discard or re-trigger
FAILED_UNSAFE Side effects may have occurred Investigate in Monitor App. The entity locks are kept until a human releases them (why)
COMPLETED All steps succeeded Nothing to do

Rollback

When a workflow fails, the engine automatically reverses the side effects of the steps that already ran — provided all of them can be reversed. Rollback is all-or-nothing over the executed non-pure steps:

  1. The engine collects every EXECUTE job that reached a worker and whose function block is non-pure. Pure steps have nothing to revert, and a job that was never handed to a worker provably never ran, so neither counts. A step that was retried is rolled back exactly once, based on its latest attempt.
  2. If every collected function block is registered reversible, the execution enters ROLLBACK and the engine pushes ROLLBACK jobs to the blackboard wave by wave in descending execution-wave order — the reverse of the deterministic traversal that created the original jobs. Jobs within one wave roll back in parallel; the next (lower) wave starts only once every rollback job of the current wave succeeded.
  3. Workers run the function block’s rollback() method with the same parameters and context as the original job.
  4. All waves succeed → FAILED_SAFE. Any rollback job fails → the phase halts immediately and the workflow is FAILED_UNSAFE — which also keeps its entity locks.

If any executed non-pure function block is not reversible, no rollback is attempted at all — a partial rollback would leave a state nobody can reason about — and the workflow fails FAILED_UNSAFE directly.

To make a step eligible, implement rollback() in the Worker SDK and register the function block with reversible: true.

Designing for Failure

Use isPure liberally. : Every read-only function block should be marked pure. This gives the engine maximum information for failure classification.

Order steps strategically. : Put pure steps first (data collection, validation) and configuration steps last. If the workflow fails during data collection, it is FAILED_SAFE.

Design idempotent configuration steps. : Prefer declarative configuration (replace entire section) over imperative (append line). Declarative configs are naturally idempotent, which will enable automatic retries when that feature is implemented.