Skip to content

E2E Playbooks

Quick start

Jump to Writing a New Playbook if you want to create one now. Use test/workflow-lifecycle/playbooks/tests/simple-workflow-2.ts as a complete reference example.

Prerequisites

Familiarity with Architecture (engine internals) and Testing (unit + E2E setup) is recommended before writing playbooks.

The Playbook pattern is the primary way to write E2E tests for the workflow engine. A playbook defines a complete execution scenario declaratively: the CMS state, the workflow, the function blocks, the expected jobs, and the expected terminal state. Each playbook also declares which safety and liveness properties it verifies, with reasoning.

Writing a New Playbook

1. Define the playbook

Create a file in the appropriate tests/ directory. Here is a complete, minimal example (this is a real playbook that runs in CI — see test/workflow-lifecycle/playbooks/tests/doc-example-playbook.ts):

import { Playbook } from '@test/workflow-lifecycle/playbooks/playbook'
import { verify } from '@test/support/verification'
import { successJob } from '@test/workflow-lifecycle/playbooks/utils/job-builders'
import { WorkflowExecutionStatus } from '@src/model/execution/workflow-execution-status'
import { EntityType } from '@src/model/workflow/schema/entity-type'
import { FunctionBlockType } from '@src/model/entities/function-block'

export const DocExamplePlaybook: Playbook = {
  description: 'Single-step ping workflow completes on two devices',
  verifiesProperties: [
    verify('lifecycle-reaches-terminal-state', 'Workflow reaches COMPLETED_ACK after ping executes'),
  ],

  // CMS mock state — all three arrays required. Or use: new SimpleDb().create()
  cmsDbBeforeExecution: {
    devices: [
      { id: 1, hostname: '10.0.0.1', facts: {} },
      { id: 2, hostname: '10.0.0.2', facts: {} },
    ],
    deviceGroups: [],
    interfaces: [],
  },

  // Workflow definition — the YAML/JSON equivalent as a TypeScript object
  publishWorkflowDefinitionDto: {
    workflow: {
      type: 'workflow',
      label: 'my_workflow',
      name: 'my_workflow',
      package: 'wf.neops.test', // package/name:version = workflow ID
      majorVersion: 1,
      minorVersion: 0,
      patchVersion: 0,
      seedEntity: EntityType.DEVICE,
      runOn: EntityType.DEVICE,
      parameters: {},
      config: { executionStrategy: {} },
      steps: [
        {
          type: 'functionBlock',
          label: 'ping',
          description: 'Ping the device',
          functionBlock: 'fb.neops.io/ping:1', // package/name:majorVersion
          runOn: EntityType.DEVICE,
          parameters: { host: 'localhost' },
        },
      ],
    },
  },

  // Function blocks to register — must match what the workflow references
  registerFunctionBlockDtos: [
    {
      name: 'ping', // ← combined with package = fb.neops.io/ping
      package: 'fb.neops.io',
      majorVersion: 1,
      minorVersion: 0,
      patchVersion: 0,
      description: 'Ping a host',
      runOn: EntityType.DEVICE,
      fbType: FunctionBlockType.CHECK,
      parameterJsonSchema: {
        type: 'object',
        properties: { host: { type: 'string' } },
        required: ['host'],
      },
      resultDataJsonSchema: {},
    },
  ],

  // Trigger execution on device IDs 1 and 2
  executeWorkflowDto: {
    workflow: 'wf.neops.test/my_workflow:1.0.0', // package/name:version
    executeOnParameters: { ids: [1, 2] },
    parameters: { host: 'localhost' },
  },

  // Expected jobs — successJob() creates { job, jobResult: { status: SUCCESS, result: { success: true } } }
  expectedWorkflowExecutionStateAfterSubmit: WorkflowExecutionStatus.RESOURCE_DISCOVERY,
  expectedAcquireJobs: [successJob({ functionBlockId: 'fb.neops.io/ping:1.0.0' })],
  expectedExecuteJobs: [successJob({ functionBlockId: 'fb.neops.io/ping:1.0.0', context: {} })],
  expectedTerminalState: WorkflowExecutionStatus.COMPLETED_ACK,
}

Key conventions:

  • Function block IDs follow the pattern package/name:major.minor.patch (e.g., fb.neops.io/ping:1.0.0)
  • Workflow references use the same pattern (e.g., wf.neops.test/my_workflow:1.0.0)
  • successJob(job) creates a success response; pass extra result fields as a second arg: successJob(job, { dbUpdates: {...} })
  • For expectedNumberOfPolls: successJob(job, undefined, { expectedNumberOfPolls: 2 }). If omitted, the executor verifies the job was matched at least once
  • For ordering assertions: successJob(job, undefined, { requiresCompletedSteps: ['prior_step'] }) fails the playbook if this job is polled before every selected expectation was fully matched. An entry is either a step label or a partial job matcher in the job shape (e.g. { stepLabel: 'prior_step', context: { device: { id: 1 } } }) matched as a subset against expectations. Entries selecting no expectations are vacuously satisfied (e.g. condition-skipped branches)
  • To assert an exactly empty parameter object: successJob(job, undefined, { exactParameters: true }). The default matcher is a subset match, under which parameters: {} matches anything
  • To pin a failure to its specific cause rather than any route to the terminal status, set expectedTerminalReasonContains on the playbook — the terminal execution’s reason must contain that substring
  • Every failing playbook sets expectedTerminalReasonCode, the kind of failure — the terminal execution’s stable reasonCode (ReasonCode) must equal it, and a playbook that leaves it unset asserts the run carries none. The code names the first failure, so a run that ended unsafe because a rollback was blocked still carries the code of the step that failed
  • To pin parts of the run entry the API serves (GET /workflow-execution/id/:id), set expectedRunEntry — a subset match, e.g. cmsResourcesLocked, heldLocks or the resolved context.devices. For every playbook the executor also checks that the entry’s lock fields agree with the locks the mock CMS still holds for the run
  • All three cmsDbBeforeExecution arrays (devices, deviceGroups, interfaces) are required
  • To test validation failures that reject at submission, use expectedStateOverrides:
// Playbook expecting workflow definition to be rejected
expectedStateOverrides: {
  workflowDefinitionPostResponse: HttpStatus.BAD_REQUEST,
},
expectedTerminalState: WorkflowExecutionStatus.FAILED_SAFE_ACK,
// No jobs will run — the executor aborts after the BAD_REQUEST

2. Wire to a test spec

Use setupPlaybookSuite inside a describe block — each it is written explicitly so IDE test runners (WebStorm, VSCode) can detect them:

import { setupPlaybookSuite } from '@test/workflow-lifecycle/playbooks/utils/run-playbook-suite'
import { MyPlaybook } from './tests/my-playbook'

describe('My Test Suite', () => {
  const suite = setupPlaybookSuite()

  it('single-step ping completes on two devices', () => suite.run(MyPlaybook))
})

setupPlaybookSuite registers beforeEach/afterEach hooks (NestJS app init with fresh database + teardown) and returns a context with a run method that wires PlaybookExecutor automatically. Each test gets an isolated PostgreSQL database; on success it is dropped. On failure in local runs, it is retained for debugging (look for [TESTDB] in test output for the database name). In CI, databases are always dropped.

3. Verify and regenerate

npm run lint:fix
npm run test
npm run test:e2e
npm run playbook:index

To run a single playbook during development:

npm run test:e2e -- --testPathPattern=my-test-suite

The npm script sets PROPERTY_EVIDENCE, so the run records its property evidence; a bare npx jest records none. A filtered run overwrites the full run’s evidence, so point it elsewhere: PROPERTY_EVIDENCE=/tmp/subset.json npm run test:e2e -- --testPathPattern=my-test-suite.

Architecture

  • Playbook – A TypeScript object declaring the complete test scenario, including property verifications with reasoning.
  • setupPlaybookSuite() – Registers app init/teardown hooks and returns a context for running playbooks.
  • PlaybookExecutor – Runs the playbook step by step, simulating the worker side.
  • DeclaredRemediation – Plays the remediation a playbook declares against the execution resting in FAILED_UNSAFE_ACK (see Remediation).
  • RemediationDriver – Helper API over a resting execution: the remediation routes, a worker for the jobs they create, CMS lock state, engine restarts, further executions. Used by DeclaredRemediation and by the remediation mechanics spec.
  • E2eInteractor – Provides HTTP helper methods for API interaction.
  • src/properties/ – The property registry: one module per area, the single source of truth for every system property. Its types and helpers (Enforces, assertSafety, the trace monitors) live apart in src/properties/support/.
  • successJob() – Helper to reduce job expectation boilerplate.

Playbook Interface

interface Playbook {
  description: string                    // What this playbook tests
  verifiesProperties: readonly Claim[]   // verify(id, reasoning) claims on registry properties
  cmsDbBeforeExecution: CmsDB            // Mock CMS state (devices, deviceGroups, interfaces — all required)
  cmsNetworkZones?: string[]             // Zones the CMS holds besides default
  workers?: PlaybookWorker[]             // { name, networkZones? }; unset = one worker serving default
  publishWorkflowDefinitionDto: PublishWorkflowDefinitionDto // the publish endpoint's own request DTO, sent as-is
  registerFunctionBlockDtos: RegisterFunctionBlockPlaybookDto[]
  executeWorkflowDto?: InstanceType<typeof ExecuteWorkflowDto>  // Omit = registration-only test
  expectedWorkflowExecutionStateAfterSubmit?: WorkflowExecutionStatus
  expectedAcquireJobs?: PlaybookJobAndResponse[]
  expectedExecuteJobs?: PlaybookExecuteJobAndResponse[]  // Each may declare how its ledger entry is remediated
  executeJobsAreNeverPolled?: true       // Fail the execute jobs via the never-polled cleanup instead of polling
  expectedRollbackJobs?: PlaybookJobAndResponse[]  // Wave by wave, in descending execution-wave order
  rollbackJobsAreNeverPolled?: true      // The worker died: fail the automatic rollback wave via the cleanup
  expectedTerminalState?: WorkflowExecutionStatusTerminalStates  // COMPLETED_ACK | FAILED_SAFE_ACK | FAILED_UNSAFE_ACK
  expectedTerminalReasonContains?: string // Pin the failure to its cause, not just the status
  expectedTerminalReasonCode?: ReasonCode // Pin the kind of failure (the stable code beside the reason)
  expectedRunEntry?: DeepPartial<WorkflowEntryDto> // Subset of the terminal run entry the API serves
  cmsDbAfterExecution?: CmsDB            // Verify CMS state after execution via isSubset()
  remediation?: PlaybookRemediation      // { via: 'jobByJob' | 'rollbackAll' | 'continueRollback' }
  expectedStateOverrides?: ExpectedStateOverrides  // Override expected HTTP status for workflow registration (e.g., BAD_REQUEST for validation failures)
}

Execution Flow

The PlaybookExecutor runs the playbook in this order:

  1. Register workflow – POST /workflow-definition/publish. If expectedStateOverrides.workflowDefinitionPostResponse is set to BAD_REQUEST, assert and abort early
  2. Initialize CMS – Set up mock CMS data via CmsLockMockService
  3. Register function blocks – POST /function-blocks/register for each FB
  4. Execute workflow – POST /workflow-execution. If no executeWorkflowDto, abort (registration-only test)
  5. Wait for RESOURCE_DISCOVERY – Or a terminal state if validation fails early
  6. Register workers – Register each of workers with its zones, or one worker named playbook-test-worker that lists none
  7. Acquire phase – Poll for ACQUIRE jobs immediately, match via isSubset(), post expected responses
  8. Execute phase – Wait for EXECUTE jobs (polling until available), post responses, loop while RUNNING
  9. Rollback phase – While the workflow is in ROLLBACK, poll one rollback wave per iteration and answer it from expectedRollbackJobs. Waves must arrive in the declared order; jobs within a wave may arrive in any order. Ends in FAILED_SAFE_ACK or FAILED_UNSAFE_ACK
  10. Assert terminal state – Verify the execution ended in the expected state and reason, and its CMS lock retention: FAILED_UNSAFE_ACK keeps every lock it ever acquired, every other terminal state holds none
  11. Assert CMS state – If cmsDbAfterExecution is set, verify via isSubset(). On mismatch, the intersection() helper shows the overlap for debugging
  12. Remediation – If remediation is set, remediate the resting FAILED_UNSAFE_ACK execution as declared (see Remediation)

Steps 2-12 live in runAssumingPublished(app), which execute() delegates to after step 1. That split is deliberate: the Playbook shape has room for exactly one registration, so a test whose point is a publish sequence – “publish v1, a later publish is rejected as 422 BelowFloor, then a loose-pin execution must still resolve v1” – cannot be a plain playbook. Such tests (see publish-resolution.e2e-spec.ts) drive their publishes directly via supertest, then call runAssumingPublished to reuse the same acquire/execute/rollback machinery instead of duplicating it.

Remediation

Playbooks test real-world cases declaratively. A playbook that ends in FAILED_UNSAFE_ACK can go on to declare how a human remediates it: the walk over the compensation ledger (GET/POST /workflow-execution/:id/remediation[/rollback|/resolve]) that ends, once every entry is compensated, with the execution turning FAILED_SAFE. The ordinary failed-safe handler then releases the retained CMS locks and acknowledges it to FAILED_SAFE_ACK.

Playbooks versus mechanics. A playbook models what an operator does and must not script the machinery. Gate refusals, API contract corners, duplicate results and engine restarts are mechanisms. Each is pinned by its own test in test/workflow-lifecycle/remediation-mechanics.e2e-spec.ts, which reaches FAILED_UNSAFE_ACK through the executor (suite.runToRest(playbook) returns a RemediationDriver) and then drives the mechanism directly.

Declaring a remediation

A ledger entry is an executed non-pure job, so its remediation is declared on the job’s expectation in expectedExecuteJobs. No selectors are needed: the driver finds the entry by the expectation’s stepLabel and entityId.

...flashFailsUnsafely([1], {
  flash: { resolve: { note: 'device reflashed manually' } },
  configure_b: { rollback: [rollbackFails('session dropped'), ROLLBACK_SUCCEEDS] },
  configure_a: { rollback: [ROLLBACK_SUCCEEDS] },
}),
remediation: { via: 'jobByJob' },

An entry takes one of these forms:

  • { rollback: [...] } lists the worker’s answer to each rollback attempt the operator requests, in order. Every attempt but the last must fail.
  • { resolve: { note } } is a human attestation that the step was fixed by hand.
  • { rollback: [...], resolve: { note } } attests the step after declared attempts that all failed.
  • No field means the entry keeps the state the terminal left it in. That is OPEN, unless the automatic rollback attempted it.

The playbook’s remediation.via names the operator’s working mode:

via What the operator does
jobByJob Works through the ledger in descending wave order, attestations before rollbacks within a wave. Each entry gets a resolve, or one rollback { jobs: [entry] } per declared attempt
rollbackAll Sends every due attestation, then rollback { all }, which re-enters the ROLLBACK phase with the first wave. Each wave the engine creates is answered in turn. When a failed attempt ends the phase back in FAILED_UNSAFE_ACK, rollback { all } answers 409, so the operator re-runs the failed entries by name (rollback { jobs }) and then continues the walk with rollback { all }, for as long as the highest uncompensated wave is reversible and declared
continueRollback Like rollbackAll, but the last due attestation carries continueRollback and re-enters the ROLLBACK phase itself. rollback { all } is used only when no attestation is left to do it

What is derived, never declared

The end state follows from the declarations, and a playbook must not restate it:

  • Entry states. The driver predicts each entry’s state as it goes: RESOLVED after a resolve, ROLLED_BACK or ROLLBACK_FAILED after an answered attempt. It asserts the touched entries after every request and every answered attempt. A RESOLVED entry must also name the acting principal (resolvedByUsername === TEST_PRINCIPAL.username) and the declared note. ROLLBACK_IN_PROGRESS is never asserted.
  • Starting state. Before remediation starts, an entry whose rollback job the main phase answered is ROLLED_BACK or ROLLBACK_FAILED. Every other entry is OPEN, including one whose rollback job no worker polled.
  • Waves. A request must create exactly the rollback jobs its mode implies. Every job in flight must sit in the highest uncompensated wave. The worker is handed exactly those jobs, and a surplus job fails the playbook.
  • Phase. While rollback attempts are in flight, the execution is in ROLLBACK if a rollback { all } or continueRollback request created them, and in FAILED_UNSAFE_ACK for a job-by-job rollback. When the ROLLBACK phase stops short of a complete ledger, because an attempt failed or only a human can compensate the next wave, the execution must rest in FAILED_UNSAFE_ACK before the operator acts again. Remediation requests sent during ROLLBACK are refused.
  • End state. The ledger is complete exactly when every entry ends compensated. A complete ledger means the execution has been remediated: it reaches FAILED_SAFE_ACK with remediated: in its reason, cmsResourcesLocked false and no CMS lock held. An incomplete ledger leaves it resting in FAILED_UNSAFE_ACK, with cmsResourcesLocked true and every lock it ever acquired. The driver waits for the status with a bounded ledger poll, never with an unbounded awaitIdle.

A declaration the ledger cannot honour fails the playbook with a clear message. Examples are a rollback declared for an irreversible entry, a rollback the wave gate refuses, and a job the engine created for an entry with no declared attempt left.

The dead worker

rollbackJobsAreNeverPolled mirrors executeJobsAreNeverPolled for the automatic rollback phase. The first rollback wave is never polled, and one job-cleanup cycle fails it. The run ends FAILED_UNSAFE_ACK with a reason containing Rollback at step … never reached a worker, and every entry stays OPEN, so the entries can then be remediated as usual.

The shared fixture test/workflow-lifecycle/playbooks/shared/remediation-workflow.ts provides the workflow and main phases the remediation playbooks use. The workflow has three steps: configure_a and configure_b are reversible, flash is irreversible. The playbooks live in tests/remediation-*.ts and are wired in test/workflow-lifecycle/unsafe-failure-remediation.e2e-spec.ts.

State Transitions (Test-Observable)

The executor observes this subset of the full state machine. States collapsed into [engine-internal] are traversed automatically by the engine and not visible to playbook tests:

stateDiagram-v2
    engine_internal: [engine-internal]
    rollback: [rollback]

    [*] --> engine_internal
    engine_internal --> RESOURCE_DISCOVERY
    RESOURCE_DISCOVERY --> RUNNING

    RUNNING --> COMPLETED_ACK
    RUNNING --> FAILED_SAFE_ACK
    RUNNING --> FAILED_UNSAFE_ACK: irreversible FB executed
    RUNNING --> rollback: all executed non-pure FBs reversible

    rollback --> FAILED_UNSAFE_ACK
    rollback --> FAILED_SAFE_ACK

    FAILED_UNSAFE_ACK --> FAILED_UNSAFE_ACK: partial remediation (locks retained)
    FAILED_UNSAFE_ACK --> rollback: rollback all / continueRollback
    FAILED_UNSAFE_ACK --> FAILED_SAFE_ACK: ledger complete (FAILED_SAFE, locks released)

    COMPLETED_ACK --> [*]
    FAILED_SAFE_ACK --> [*]
    FAILED_UNSAFE_ACK --> [*]

The full engine state machine includes intermediate states (NEW, VALID, BLOCKED_WAITING, READY, SCHEDULED, LOCKING, LOCKED, RESOURCES_DISCOVERED, ERROR, ROLLBACK) that the executor skips over via polling. See src/model/execution/workflow-execution-status.ts for the complete enum.

System Properties

Properties live in the registry under src/properties/, one module per area, each named by a kebab-case ID such as no-cross-context-write (safety: bad things don’t happen) or lifecycle-reaches-terminal-state (liveness: good things eventually happen). npm run props -- list lists them; npm run props -- show <id> shows where each is enforced and verified.

Recording claims

A playbook claims a property with verify('<id>', reasoning) from test/support/verification.ts, in its verifiesProperties. The reasoning says why this playbook covers the property, and the compiler rejects an unknown ID. A spec outside the playbook system claims one with itVerifies('<id>', reasoning, fn) from the same module. The verification itself still comes from the test’s assertions (expected jobs, terminal state, CMS state).

The executor records a playbook’s claims from inside the running test, and the npm test scripts write what ran to .property-evidence/ (PROPERTY_EVIDENCE, paths in package.json’s config). npm run check:evidence reads both evidence files and fails a claim that never ran or ran only in failing tests, so it needs a full npm run test and a full npm run test:e2e before it; a filtered run leaves claims unrun. npm run check checks the registry without evidence.

A property’s ID never changes and is never reused: a reworded statement keeps its ID, a new property gets a new one in the area module it belongs to, followed by npm run index. The property-driven-development skill holds the full registry rules.

Run npm run playbook:index to regenerate the playbook index: coverage counts per property and all playbook descriptions.

Directory Organization

flowchart LR
    root["test/workflow-lifecycle/playbooks/"]

    root --> pb["playbook.ts<br/>Playbook interface"]
    root --> pe["playbook-executor.ts<br/>Shared execution engine"]
    root --> shared["shared/<br/>Shared FB definitions and fixtures<br/>(e.g. BasePythonEval, remediation workflow)"]
    root --> base["base/<br/>E2eInteractor base class"]
    root --> cms["cms/<br/>CMS database factories"]
    root --> utils["utils/<br/>Job builders, job matching, assertion helpers, suite runner"]
    root --> remediation["remediation/<br/>DeclaredRemediation, its plan derivation, RemediationDriver"]
    root --> tests["tests/<br/>Core workflow lifecycle scenarios"]
    root --> exec["workflow-execution-*/<br/>Execution feature tests<br/>(JMES, db-updates, embedded workflows)"]
    root --> valid["workflow-validation-*/<br/>Validation and version resolution tests"]
    root --> pubres["publish-resolution/<br/>Publish-sequence tests<br/>(drive publishes directly, then runAssumingPublished)"]
    root --> design["design-*/<br/>End-to-end workflow designs<br/>(customer scenarios, documentation examples)"]

E2eInteractor Helpers

Method Endpoint Description
registerWorker(name) POST /workers/register Register a test worker, returns UUID
registerWorkflow(dto, expectedStatus?) POST /workflow-definition/publish Deploy a workflow definition
registerFunctionBlocks(dtos) POST /function-blocks/register Register function blocks
executeWorkflow(dto) POST /workflow-execution Trigger a workflow execution
pollJobs(dto) POST /blackboard/job Poll for jobs, validates no context overlap
postJobResult(dto) POST /blackboard/job/result Push a job result
pollWorkflowExecution(app, id) GET /workflow-execution/id/{id} Fetch the current execution once
waitForState(execution, states, message) GET /workflow-execution/id/{id} Poll until state matches
waitForJobs(dto, execution, status) POST /blackboard/job Poll until jobs appear or state changes
initCms(cmsDB) (mock injection) Set up CMS mock data

CMS Database Factories

Reuse existing factories or define inline CMS data for specific needs:

All implement CmsDbFactory (cms/cms-db-factory.ts) and expose a create() returning a CmsDB.

Factory Location Contents
SimpleDb cms/simple-db.ts 2 devices, 4 interfaces, 2 device groups
EmptyDb design-discovery/db/empty-db.ts Empty (for discovery tests)
DbUpdatesDb workflow-execution-db-updates/db/db-updates-db.ts 2 devices, no interfaces/groups
TwinZoneDb cms/twin-zone-db.ts 2 devices r1 at 10.0.0.1, one in zone-a and one in zone-b, each with eth0

Job Matching

Jobs are matched using isSubset(subset, superset) — the playbook only specifies fields it cares about. Dynamic fields like jobId can be omitted. The executor validates that polled job contexts don’t overlap (devices, deviceGroups, interfaces arrays checked pairwise via progressive merge).

Every poll asks as each registered worker in turn. A job expectation’s polledBy names the worker that must get the job, which is how a playbook pins a job to its zone; see network-zones/tests/.

Each expected job is tracked via a matchedCount. After a phase completes, the executor verifies every expected job was matched the expected number of times. For failure scenarios (FAILED_SAFE_ACK, FAILED_UNSAFE_ACK), unmatched jobs are tolerated since the workflow may abort before creating all jobs.

Debugging Failed Playbooks

When a playbook assertion fails, the expectCondition helper includes the current WorkflowExecution object (status, reason) in the failure output. Look for:

  • [TESTDB] Setup test: ... in console output to find the database name for the failed test
  • In local runs, the database is retained on failure — connect directly to inspect state
  • The intersection() helper shows the overlap between expected and actual CMS state on cmsDbAfterExecution mismatches
  • Context overlap errors come from contextesOverlap() in playbook.ts (note the typo in the function name — search for contextesOverlap in source)

See Also