Skip to content

Retry & Rollback

The engine provides two mechanisms for handling step failures: automatic retries and rollback.

Automatic Retries

A failed job is retried automatically, up to three times, at once and without a delay, when its function block is registered isPure or isIdempotent, or when the failed attempt reported that it changed nothing (sideEffects: false, see Steps that changed nothing). The engine creates a new job that replaces the failed one (replaces/replacedBy). Any other failed job fails its step on the first attempt.

A worker can opt a failure out of the retry by pushing it with retryable: false in the result:

{
  "jobExecutionId": "…",
  "status": "failed",
  "result": { "success": false, "message": "named host not discovered", "retryable": false }
}

Use it for a verdict that would come out the same on every attempt, such as a check that found what it was looking for to be absent. A result without the field, or with retryable: true, is retried as above. GET /blackboard/jobs/{jobId}/result serves the field back, null when the worker did not set it.

sideEffects: false alone means up to three retries

A failure that reports sideEffects: false is retried even when its function block is neither pure nor idempotent: the attempt changed nothing, so another one is safe. If the failure would come out the same on every attempt, such as a missing setting or a template that does not compile, also set retryable: false, or the step runs up to three more times before it fails. The Worker SDK sets both for you when a block failed before its run() was entered. A retry that does change something keeps the step in the rollback, because a step is left out only when every attempt reported sideEffects: false.

Retry Configuration

Not Yet Implemented

retryConfig is defined in the workflow schema and can be set on steps, but the execution engine does not yet consume it. Failed steps are not retried based on this configuration. This feature is planned for a future release.

Add retryConfig to a step to configure retry behavior on failure (when implemented):

- type: functionBlock
  label: push_config
  functionBlock: "fb.examples.neops.io/configureDevice:1.0.0"
  retryConfig:
    maxRetries: 3
    delay: 10
Field Required Description
maxRetries Yes Number of retry attempts (1-100)
delay No Seconds between attempts (0-600)
condition No JMESPath condition that must be true for retry

Planned Retry Behavior

  1. A step fails
  2. The engine checks if the step has retryConfig
  3. If retryCount < maxRetries, the engine waits delay seconds and creates a new job
  4. The retry job replaces the failed job (tracked via replaces/replacedBy references)
  5. If all retries fail, the step is marked as permanently failed

Repeat Configuration

Not Yet Implemented

repeatConfig is defined in the workflow schema and can be set on steps, but the execution engine does not yet consume it. Steps always execute once. This feature is planned for a future release.

Repeat a step multiple times (independent of failure), when implemented:

- type: functionBlock
  label: poll_status
  functionBlock: "fb.examples.neops.io/checkStatus:1.0.0"
  repeatConfig:
    repeats: 5
    delay: 60
    condition:
      type: jmes
      jmes: "{{ poll_status.result.data.status != 'ready' }}"
Field Required Description
repeats No Fixed number of repetitions (1-100)
delay No Seconds between iterations (0-600)
condition No Continue repeating while condition is true

Provide either repeats (fixed count), condition (repeat while true), or both (whichever limit is reached first). Use repeat for polling patterns (wait until a device reboots, check convergence).

Failure Classification

When a step fails (after any automatic retries), the engine classifies the overall workflow failure:

graph TD
    StepFail["Step failed"] --> COE{"continueOnError?<br/>(planned)"}
    COE -->|Yes| Continue["Continue to next step"]
    COE -->|No| Classify["Classify failure"]
    Classify --> AnyNonPure{"Did any non-pure step execute<br/>that may have changed something?"}
    AnyNonPure -->|No| FS["FAILED_SAFE"]
    AnyNonPure -->|Yes| AllRev{"All of those<br/>FBs reversible?"}
    AllRev -->|Yes| RB["ROLLBACK<br/>(wave by wave, reverse order)"]
    RB -->|all waves succeed| FS2["FAILED_SAFE"]
    RB -->|any rollback job fails| FU["FAILED_UNSAFE"]
    AllRev -->|No| FU2["FAILED_UNSAFE"]

A non-pure step “may have changed something” unless it was left out because every attempt reported sideEffects: false.

Implementation status

The classification, rollback and automatic-retry paths above are active. continueOnError is not yet enforced. The run-level isPureExecution and isIdempotentExecution flags are tracked and logged but decide nothing: the classification comes from the jobs that reached a worker.

Result Meaning Action
FAILED_SAFE No side effects remain (nothing non-pure executed, every non-pure step reported it changed nothing, or everything else was rolled back) Safe to discard or re-trigger
FAILED_UNSAFE Side effects may have occurred Investigate in Monitor App. The entity locks are kept (why) until remediation completes the compensation ledger; the engine then releases them (how)
COMPLETED All steps succeeded Nothing to do

Steps that changed nothing

A worker can report that an attempt changed nothing, neither on the device nor anywhere else, by setting sideEffects: false in the result of an EXECUTE job (side_effects=False on a Worker SDK result). It may do so on a success or on a failure:

{
  "jobExecutionId": "…",
  "status": "failed",
  "result": { "success": false, "message": "reload delay not set", "sideEffects": false, "retryable": false }
}

The engine trusts the claim, and leaves the step out of the rollback when both of these hold for its (step, entity):

  • Every attempt said it. Each EXECUTE attempt that reached a worker has a result the worker wrote with sideEffects: false. One attempt without the flag, an attempt with no result, or a result the engine wrote itself (a timeout, a lost worker) means side effects are assumed, and the step stays in.
  • No rollback job exists for it. A step that already has a rollback attempt is never left out: once a rollback has run, its outcome counts. A late result with the flag that arrives after a rollback of that step leaves it in the ledger.

An exception in the block always means side effects are assumed; only an explicit return marks a step safe. The claim is the engine’s only evidence, so a block reports it only where it holds, typically on a check that fails before anything is sent to the device.

Situation Outcome
The only non-pure step that ran fails with sideEffects: false FAILED_SAFE, no rollback
A step succeeds with sideEffects: false, then a later step fails The step is left out: it is not rolled back, and an irreversible one does not block the rollback
One attempt lacks the flag, or a worker was lost The step is rolled back (or blocks, if irreversible)
A result with the flag arrives after the step was rolled back The step stays in the ledger

The left-out steps are not ledger entries, but the remediation ledger lists them, with each attempt’s claim, under leftOut.

Pure blocks cannot claim side effects

A pure function block has no side effects by definition, so it has nothing to claim. If a pure block’s EXECUTE result says sideEffects: true, its registration or the block is wrong:

  • The Worker SDK does not send such a result. It pushes a failure instead, with retryable: false, no sideEffects and the message pure block reported side effects: fix the block or its registration, and logs an error.
  • An engine that receives such a result anyway does not store the claim. It stores a failure of its own in its place, with retryable: false, so the run fails at once with reasonCode invalid_result and the violation in its reason.

The job’s row then says resultRejected: pure_side_effects. A result whose data breaks the block’s result schema says resultRejected: result_schema on its row; a run that fails with invalid_result while none of its jobs says either failed on a step’s dbUpdates.

Rollback

When a workflow fails, the engine automatically reverses the side effects of the steps that already ran — provided all of them can be reversed. Rollback is all-or-nothing over the executed non-pure steps:

  1. The engine collects every EXECUTE job that reached a worker and whose function block is non-pure. Pure steps have nothing to revert, and a job that was never handed to a worker provably never ran, so neither counts. Nor does a step that changed nothing. A step that was retried is rolled back exactly once, with the parameters and context of its latest attempt.
  2. If every collected function block is registered reversible, the execution enters ROLLBACK and the engine pushes ROLLBACK jobs to the blackboard wave by wave in descending execution-wave order — the reverse of the deterministic traversal that created the original jobs. Jobs within one wave roll back in parallel; the next (lower) wave starts only once every rollback job of the current wave succeeded.
  3. Workers run the function block’s rollback() method with the same parameters and context as the original job, and with the forward result of the step.
  4. All waves succeed → FAILED_SAFE. Any rollback job fails → the phase halts immediately and the workflow is FAILED_UNSAFE — which also keeps its entity locks.

If any executed non-pure function block is not reversible, no rollback is attempted at all — a partial rollback would leave a state nobody can reason about — and the workflow fails FAILED_UNSAFE directly.

To make a step eligible, implement rollback() in the Worker SDK and register the function block with reversible: true.

Rollback Data

A reversible function block can record what it needs to undo its change, such as the running configuration it is about to replace, before it changes anything. In the Worker SDK the block calls save_rollback_data() with an instance of its rollback-data class. The snapshot rides along with the EXECUTE result: the worker sends it as rollbackData, a sibling of result, and also on a failure or after an exception in the block.

{
  "jobExecutionId": "…",
  "status": "success",
  "result": { "success": true, "data": { "changed": 3 } },
  "rollbackData": { "running": "hostname leaf1\n…" }
}

Checks and storage

The block declares the shape as rollbackDataJsonSchema when it registers. When a result arrives, the engine checks its snapshot:

  1. the job is an EXECUTE job;
  2. the function block is reversible and has a registered rollbackDataJsonSchema;
  3. it fits within ROLLBACK_DATA_MAX_BYTES (5 MiB by default), measured as the UTF-8 byte length of its compact JSON;
  4. it passes exact validation against the schema: no type conversion and no defaults filled in, so the value must already match.

A snapshot that passes is stored unchanged with the job result, in a column of its own. It is never part of <step>.result, the workflow context or any read API; the engine reads it only to hand it to the rollback. The job rows of the read API say only its size, rollbackDataSizeBytes (the compact UTF-8 JSON it was capped by; null when none is held), and rollbackDataRejected.

A rejected snapshot never fails the push. The result, its acquires and its dbUpdates are stored as usual; only the snapshot is dropped, and the job result records why:

rollbackDataRejected The snapshot was dropped because
not_execute it came with a job that is not an EXECUTE job
not_reversible the function block is not registered reversible
no_schema the function block registered no rollbackDataJsonSchema
too_large it exceeds ROLLBACK_DATA_MAX_BYTES
invalid it does not match the schema exactly, or it holds text Postgres jsonb cannot store: a NUL character or a lone surrogate

The engine logs a refusal by the rule it breaks and that rule’s place in the schema, never by the value or a key: a key the schema declares reads * in that place, so a maxLength under a property logs as #/properties/*/maxLength. A push still answers 400 only when the result itself is malformed.

Because the snapshot travels with the result, an attempt whose result never arrives (a lost push, a dead worker) has none.

A push after the engine gave up

The engine fails a job on the stuck timeout or when its worker goes offline, and moves on. A worker that was only paused or cut off then finishes the job, change included, and pushes. That push still answers 409 and the engine’s failure stands: the first committed result wins, so the status, the reason and outcomeKnown: false do not change, and whatever the push says about the outcome, sideEffects: false included, counts for nothing. The step stays in the rollback.

Its snapshot is kept if it can still help. It goes through the same checks, and the engine stores it, or the reason it dropped it, beside its own result, when all of these hold:

  • the existing result is one the engine wrote with reasonCode job_timed_out or worker_lost. A second push against a result the worker wrote changes nothing;
  • the push comes from the lease holder of that attempt. Anyone else gets 403, as on any push;
  • no ROLLBACK job of the step has been polled or has ended. A worker that polled one has its forwardResult already, so the snapshot comes too late and is not stored;
  • no later attempt of the step has reached a worker. The paused worker may have taken its snapshot after that attempt changed the device, and restoring such a snapshot would undo nothing, so a rollback is better off refusing;
  • the run is not in FAILED_SAFE, FAILED_SAFE_ACK or COMPLETED_ACK: nothing there can read it;
  • no earlier late push left a snapshot or a rejection: the first one stays.

The retention sweep clears a late snapshot like any other. Stored does not mean used: the rollback reads the earliest attempt that may have changed something, and that may be an earlier attempt with no snapshot.

Every such push logs a warning that its change is not recorded, with the job, the worker, the status it pushed, its sideEffects and lateSnapshot: stored, none_sent, too_late, already_held or the rollbackDataRejected reason. The log never carries the snapshot. On the job, a held snapshot shows as rollbackDataSizeBytes beside the engine’s failure.

So the lease of a failed job stays good for this one write, until a rollback or a later attempt is taken or the run settles. Deleting a worker does not end it: a deleted worker’s late push is treated the same.

With several workers, another worker can poll the rollback first. The rollback then runs without the snapshot, and it may run while the late worker is still changing the device. The block decides what to do without a snapshot.

What the rollback receives

Every ROLLBACK job carries forwardResult when a worker polls it. The engine resolves it at poll time from the attempts of the step it reverses. It walks the EXECUTE attempts that reached a worker, oldest first, and skips only those whose worker-written result said sideEffects: false; an attempt whose outcome is unknown is never skipped. The first remaining attempt, the earliest that may have changed something, is handed over whole: every field comes from that one attempt, never mixed across attempts. A retry runs on whatever the failed attempt left behind, so the earliest snapshot is the one taken before the step changed anything.

Field Meaning
success / message / data That attempt’s result
rollbackData Its stored snapshot, or null
rollbackDataRejected Why its snapshot was dropped, or null
outcomeKnown false when the engine wrote that attempt’s result (a timeout, a lost worker) or there is no result yet. success is then false and data is null; rollbackData is null unless a late push left its snapshot
previousRollbacks The step’s earlier rollback attempts that reached a worker, oldest first, each as {success, message, outcomeKnown}

A poll response carries at most one rollback job. The Worker SDK fails a rollback job whose forwardResult is missing or does not parse into the block’s result and rollback-data classes, with retryable: false, without calling rollback().

The block decides

What a rollback does without a snapshot (rollbackData is null: none was saved, it was rejected, no result of that attempt arrived in time, or retention removed it), and what it does when it runs a second time (previousRollbacks is not empty), is up to the function block. A block that refuses returns a failed rollback result, and the entry ends ROLLBACK_FAILED, and the run stays unsafe until an operator re-runs or resolves it.

Retention

A sweep removes stored rollback data from runs that no longer need it, every ROLLBACK_DATA_SWEEP_INTERVAL_MS (hourly by default):

  • runs in FAILED_SAFE or FAILED_SAFE_ACK;
  • runs in COMPLETED_ACK;
  • runs that have been in FAILED_UNSAFE_ACK for longer than ROLLBACK_DATA_RETENTION_DAYS (30 by default) without an active rollback job.

A rollback requested after that receives rollbackData: null. The variables are listed in Configuration.

Rollback data is not protected

The engine stores snapshots as plain JSON, not encrypted, and the worker routes that carry them (POST /blackboard/job, POST /blackboard/job/result) take no credential. A snapshot of a device configuration is as sensitive as the configuration. Snapshots are left out of the Worker SDK’s job log and of the engine’s validation error messages.

Designing for Failure

Use isPure liberally. : Every function block without side effects, one that only reads or only declares CMS changes, should be marked pure. This gives the engine maximum information for failure classification.

Order steps strategically. : Put pure steps first (data collection, validation) and configuration steps last. If the workflow fails during data collection, it is FAILED_SAFE.

Design idempotent configuration steps. : Prefer declarative configuration (replace entire section) over imperative (append line). Declarative configs are naturally idempotent, and a failed job of an idempotent function block is retried automatically.

Report what did not happen. : A check that fails before anything is sent to the device can return sideEffects: false. The step then stays out of the rollback, and an irreversible block no longer turns such a failure into FAILED_UNSAFE. Add retryable: false when the check would fail the same way again.

Snapshot before you write. : A reversible function block that saves its rollback data before its first change gives its rollback() the state to restore.