E2E Playbooks

Quick start

Jump to Writing a New Playbook if you want to create one now. Use test/workflow-lifecycle/playbooks/tests/simple-workflow-2.ts as a complete reference example.

Prerequisites

Familiarity with Architecture (engine internals) and Testing (unit + E2E setup) is recommended before writing playbooks.

The Playbook pattern is the primary way to write E2E tests for the workflow engine. A playbook defines a complete execution scenario declaratively: the CMS state, the workflow, the function blocks, the expected jobs, and the expected terminal state. Each playbook also declares which safety and liveness properties it verifies, with reasoning.

Writing a New Playbook

1. Define the playbook

Create a file in the appropriate tests/ directory. Here is a complete, minimal example (this is a real playbook that runs in CI — see test/workflow-lifecycle/playbooks/tests/doc-example-playbook.ts):

import { Playbook } from '@test/workflow-lifecycle/playbooks/playbook'
import { P, verify } from '@test/workflow-lifecycle/playbooks/properties'
import { successJob } from '@test/workflow-lifecycle/playbooks/utils/job-builders'
import { WorkflowExecutionStatus } from '@src/model/execution/workflow-execution-status'
import { EntityType } from '@src/model/workflow/schema/entity-type'
import { FunctionBlockType } from '@src/model/entities/function-block'

export const DocExamplePlaybook: Playbook = {
  description: 'Single-step ping workflow completes on two devices',
  verifiesProperties: [
    verify(P.lifecycleReachesTerminalState, 'Workflow reaches COMPLETED_ACK after ping executes'),
    verify(P.singleStepCompletes, 'One-step ping workflow runs to completion on 2 devices'),
  ],

  // CMS mock state — all three arrays required. Or use: new SimpleDb().create()
  cmsDbBeforeExecution: {
    devices: [
      { id: 1, hostname: '10.0.0.1', facts: {} },
      { id: 2, hostname: '10.0.0.2', facts: {} },
    ],
    deviceGroups: [],
    interfaces: [],
  },

  // Workflow definition — the YAML/JSON equivalent as a TypeScript object
  publishWorkflowDefinitionDto: {
    workflow: {
      type: 'workflow',
      label: 'my_workflow',
      name: 'my_workflow',
      package: 'wf.neops.test', // package/name:version = workflow ID
      majorVersion: 1,
      minorVersion: 0,
      patchVersion: 0,
      seedEntity: EntityType.DEVICE,
      runOn: EntityType.DEVICE,
      parameters: {},
      config: { executionStrategy: {} },
      steps: [
        {
          type: 'functionBlock',
          label: 'ping',
          description: 'Ping the device',
          functionBlock: 'fb.neops.io/ping:1', // package/name:majorVersion
          runOn: EntityType.DEVICE,
          parameters: { host: 'localhost' },
        },
      ],
    },
  },

  // Function blocks to register — must match what the workflow references
  registerFunctionBlockDtos: [
    {
      name: 'ping', // ← combined with package = fb.neops.io/ping
      package: 'fb.neops.io',
      majorVersion: 1,
      minorVersion: 0,
      patchVersion: 0,
      description: 'Ping a host',
      runOn: EntityType.DEVICE,
      fbType: FunctionBlockType.CHECK,
      parameterJsonSchema: {
        type: 'object',
        properties: { host: { type: 'string' } },
        required: ['host'],
      },
      resultDataJsonSchema: {},
    },
  ],

  // Trigger execution on device IDs 1 and 2
  executeWorkflowDto: {
    workflow: 'wf.neops.test/my_workflow:1.0.0', // package/name:version
    executeOnParameters: { ids: [1, 2] },
    parameters: { host: 'localhost' },
  },

  // Expected jobs — successJob() creates { job, jobResult: { status: SUCCESS, result: { success: true } } }
  expectedWorkflowExecutionStateAfterSubmit: WorkflowExecutionStatus.RESOURCE_DISCOVERY,
  expectedAcquireJobs: [successJob({ functionBlockId: 'fb.neops.io/ping:1.0.0' })],
  expectedExecuteJobs: [successJob({ functionBlockId: 'fb.neops.io/ping:1.0.0', context: {} })],
  expectedTerminalState: WorkflowExecutionStatus.COMPLETED_ACK,
}

Key conventions:

• Function block IDs follow the pattern package/name:major.minor.patch (e.g., fb.neops.io/ping:1.0.0)
• Workflow references use the same pattern (e.g., wf.neops.test/my_workflow:1.0.0)
• successJob(job) creates a success response; pass extra result fields as a second arg: successJob(job, { dbUpdates: {...} })
• For expectedNumberOfPolls: successJob(job, undefined, { expectedNumberOfPolls: 2 }). If omitted, the executor verifies the job was matched at least once
• For ordering assertions: successJob(job, undefined, { requiresCompletedSteps: ['prior_step'] }) fails the playbook if this job is polled before every selected expectation was fully matched. An entry is either a step label or a partial job matcher in the job shape (e.g. { stepLabel: 'prior_step', context: { device: { id: 1 } } }) matched as a subset against expectations. Entries selecting no expectations are vacuously satisfied (e.g. condition-skipped branches)
• To assert an exactly empty parameter object: successJob(job, undefined, { exactParameters: true }). The default matcher is a subset match, under which parameters: {} matches anything
• To pin a failure to its specific cause rather than any route to the terminal status, set expectedTerminalReasonContains on the playbook — the terminal execution’s reason must contain that substring
• Every failing playbook sets expectedTerminalReasonCode, the kind of failure — the terminal execution’s stable reasonCode (ReasonCode) must equal it, and a playbook that leaves it unset asserts the run carries none. The code names the first failure, so a run that ended unsafe because a rollback was blocked still carries the code of the step that failed
• To pin parts of the run entry the API serves (GET /workflow-execution/id/:id), set expectedRunEntry — a subset match, e.g. cmsResourcesLocked, heldLocks or the resolved context.devices. For every playbook the executor also checks that the entry’s lock fields agree with the locks the mock CMS still holds for the run
• All three cmsDbBeforeExecution arrays (devices, deviceGroups, interfaces) are required
• To test validation failures that reject at submission, use expectedStateOverrides:

// Playbook expecting workflow definition to be rejected
expectedStateOverrides: {
  workflowDefinitionPostResponse: HttpStatus.BAD_REQUEST,
},
expectedTerminalState: WorkflowExecutionStatus.FAILED_SAFE_ACK,
// No jobs will run — the executor aborts after the BAD_REQUEST

2. Wire to a test spec

Use setupPlaybookSuite inside a describe block — each it is written explicitly so IDE test runners (WebStorm, VSCode) can detect them:

import { setupPlaybookSuite } from '@test/workflow-lifecycle/playbooks/utils/run-playbook-suite'
import { MyPlaybook } from './tests/my-playbook'

describe('My Test Suite', () => {
  const suite = setupPlaybookSuite()

  it('single-step ping completes on two devices', () => suite.run(MyPlaybook))
})

setupPlaybookSuite registers beforeEach/afterEach hooks (NestJS app init with fresh database + teardown) and returns a context with a run method that wires PlaybookExecutor automatically. Each test gets an isolated PostgreSQL database; on success it is dropped. On failure in local runs, it is retained for debugging (look for [TESTDB] in test output for the database name). In CI, databases are always dropped.

3. Verify and regenerate

npm run lint:fix
npm run test
npm run test:e2e
npm run playbook:index

To run a single playbook during development:

npx jest --config ./test/jest-e2e.json --testPathPattern=my-test-suite

Architecture

• Playbook – A TypeScript object declaring the complete test scenario, including property verifications with reasoning.
• setupPlaybookSuite() – Registers app init/teardown hooks and returns a context for running playbooks.
• PlaybookExecutor – Runs the playbook step by step, simulating the worker side.
• DeclaredRemediation – Plays the remediation a playbook declares against the execution resting in FAILED_UNSAFE_ACK (see Remediation).
• RemediationDriver – Helper API over a resting execution: the remediation routes, a worker for the jobs they create, CMS lock state, engine restarts, further executions. Used by DeclaredRemediation and by the remediation mechanics spec.
• E2eInteractor – Provides HTTP helper methods for API interaction.
• properties.ts – Single source of truth for all system properties (SafetyProperty | LivenessProperty).
• successJob() – Helper to reduce job expectation boilerplate.

Playbook Interface

interface Playbook {
  description: string                    // What this playbook tests
  verifiesProperties: SystemPropertyVerification[]  // Properties + reasoning
  cmsDbBeforeExecution: CmsDB            // Mock CMS state (devices, deviceGroups, interfaces — all required)
  publishWorkflowDefinitionDto: PublishWorkflowDefinitionDto // the publish endpoint's own request DTO, sent as-is
  registerFunctionBlockDtos: RegisterFunctionBlockPlaybookDto[]
  executeWorkflowDto?: InstanceType<typeof ExecuteWorkflowDto>  // Omit = registration-only test
  expectedWorkflowExecutionStateAfterSubmit?: WorkflowExecutionStatus
  expectedAcquireJobs?: PlaybookJobAndResponse[]
  expectedExecuteJobs?: PlaybookExecuteJobAndResponse[]  // Each may declare how its ledger entry is remediated
  executeJobsAreNeverPolled?: true       // Fail the execute jobs via the never-polled cleanup instead of polling
  expectedRollbackJobs?: PlaybookJobAndResponse[]  // Wave by wave, in descending execution-wave order
  rollbackJobsAreNeverPolled?: true      // The worker died: fail the automatic rollback wave via the cleanup
  expectedTerminalState?: WorkflowExecutionStatusTerminalStates  // COMPLETED_ACK | FAILED_SAFE_ACK | FAILED_UNSAFE_ACK
  expectedTerminalReasonContains?: string // Pin the failure to its cause, not just the status
  expectedTerminalReasonCode?: ReasonCode // Pin the kind of failure (the stable code beside the reason)
  expectedRunEntry?: DeepPartial<WorkflowEntryDto> // Subset of the terminal run entry the API serves
  cmsDbAfterExecution?: CmsDB            // Verify CMS state after execution via isSubset()
  remediation?: PlaybookRemediation      // { via: 'jobByJob' | 'rollbackAll' | 'continueRollback' }
  expectedStateOverrides?: ExpectedStateOverrides  // Override expected HTTP status for workflow registration (e.g., BAD_REQUEST for validation failures)
}

Execution Flow

The PlaybookExecutor runs the playbook in this order:

1. Register workflow – POST /workflow-definition/publish. If expectedStateOverrides.workflowDefinitionPostResponse is set to BAD_REQUEST, assert and abort early
2. Initialize CMS – Set up mock CMS data via CmsLockMockService
3. Register function blocks – POST /function-blocks/register for each FB
4. Execute workflow – POST /workflow-execution. If no executeWorkflowDto, abort (registration-only test)
5. Wait for RESOURCE_DISCOVERY – Or a terminal state if validation fails early
6. Register worker – Create a test worker named playbook-test-worker
7. Acquire phase – Poll for ACQUIRE jobs immediately, match via isSubset(), post expected responses
8. Execute phase – Wait for EXECUTE jobs (polling until available), post responses, loop while RUNNING
9. Rollback phase – While the workflow is in ROLLBACK, poll one rollback wave per iteration and answer it from expectedRollbackJobs. Waves must arrive in the declared order; jobs within a wave may arrive in any order. Ends in FAILED_SAFE_ACK or FAILED_UNSAFE_ACK
10. Assert terminal state – Verify the execution ended in the expected state and reason, and its CMS lock retention: FAILED_UNSAFE_ACK keeps every lock it ever acquired, every other terminal state holds none
11. Assert CMS state – If cmsDbAfterExecution is set, verify via isSubset(). On mismatch, the intersection() helper shows the overlap for debugging
12. Remediation – If remediation is set, remediate the resting FAILED_UNSAFE_ACK execution as declared (see Remediation)

Steps 2-12 live in runAssumingPublished(app), which execute() delegates to after step 1. That split is deliberate: the Playbook shape has room for exactly one registration, so a test whose point is a publish sequence – “publish v1, a later publish is rejected as 422 BelowFloor, then a loose-pin execution must still resolve v1” – cannot be a plain playbook. Such tests (see publish-resolution.e2e-spec.ts) drive their publishes directly via supertest, then call runAssumingPublished to reuse the same acquire/execute/rollback machinery instead of duplicating it.

Remediation

Playbooks test real-world cases declaratively. A playbook that ends in FAILED_UNSAFE_ACK can go on to declare how a human remediates it: the walk over the compensation ledger (GET/POST /workflow-execution/:id/remediation[/rollback|/resolve]) that ends, once every entry is compensated, with the execution turning FAILED_SAFE. The ordinary failed-safe handler then releases the retained CMS locks and acknowledges it to FAILED_SAFE_ACK.

Playbooks versus mechanics. A playbook models what an operator does and must not script the machinery. Gate refusals, API contract corners, duplicate results and engine restarts are mechanisms. Each is pinned by its own test in test/workflow-lifecycle/remediation-mechanics.e2e-spec.ts, which reaches FAILED_UNSAFE_ACK through the executor (suite.runToRest(playbook) returns a RemediationDriver) and then drives the mechanism directly.

Declaring a remediation

A ledger entry is an executed non-pure job, so its remediation is declared on the job’s expectation in expectedExecuteJobs. No selectors are needed: the driver finds the entry by the expectation’s stepLabel and entityId.

...flashFailsUnsafely([1], {
  flash: { resolve: { note: 'device reflashed manually' } },
  configure_b: { rollback: [rollbackFails('session dropped'), ROLLBACK_SUCCEEDS] },
  configure_a: { rollback: [ROLLBACK_SUCCEEDS] },
}),
remediation: { via: 'jobByJob' },

An entry takes one of these forms:

• { rollback: [...] } lists the worker’s answer to each rollback attempt the operator requests, in order. Every attempt but the last must fail.
• { resolve: { note } } is a human attestation that the step was fixed by hand.
• { rollback: [...], resolve: { note } } attests the step after declared attempts that all failed.
• No field means the entry keeps the state the terminal left it in. That is OPEN, unless the automatic rollback attempted it.

The playbook’s remediation.via names the operator’s working mode:

via	What the operator does
jobByJob	Works through the ledger in descending wave order, attestations before rollbacks within a wave. Each entry gets a resolve, or one rollback { jobs: [entry] } per declared attempt
rollbackAll	Sends every due attestation, then rollback { all }, which re-enters the ROLLBACK phase with the first wave. Each wave the engine creates is answered in turn. When a failed attempt ends the phase back in FAILED_UNSAFE_ACK, rollback { all } answers 409, so the operator re-runs the failed entries by name (rollback { jobs }) and then continues the walk with rollback { all }, for as long as the highest uncompensated wave is reversible and declared
continueRollback	Like rollbackAll, but the last due attestation carries continueRollback and re-enters the ROLLBACK phase itself. rollback { all } is used only when no attestation is left to do it

What is derived, never declared

The end state follows from the declarations, and a playbook must not restate it:

• Entry states. The driver predicts each entry’s state as it goes: RESOLVED after a resolve, ROLLED_BACK or ROLLBACK_FAILED after an answered attempt. It asserts the touched entries after every request and every answered attempt. A RESOLVED entry must also name the acting principal (resolvedByUsername === TEST_PRINCIPAL.username) and the declared note. ROLLBACK_IN_PROGRESS is never asserted.
• Starting state. Before remediation starts, an entry whose rollback job the main phase answered is ROLLED_BACK or ROLLBACK_FAILED. Every other entry is OPEN, including one whose rollback job no worker polled.
• Waves. A request must create exactly the rollback jobs its mode implies. Every job in flight must sit in the highest uncompensated wave. The worker is handed exactly those jobs, and a surplus job fails the playbook.
• Phase. While rollback attempts are in flight, the execution is in ROLLBACK if a rollback { all } or continueRollback request created them, and in FAILED_UNSAFE_ACK for a job-by-job rollback. When the ROLLBACK phase stops short of a complete ledger, because an attempt failed or only a human can compensate the next wave, the execution must rest in FAILED_UNSAFE_ACK before the operator acts again. Remediation requests sent during ROLLBACK are refused.
• End state. The ledger is complete exactly when every entry ends compensated. A complete ledger means the execution has been remediated: it reaches FAILED_SAFE_ACK with remediated: in its reason, cmsResourcesLocked false and no CMS lock held. An incomplete ledger leaves it resting in FAILED_UNSAFE_ACK, with cmsResourcesLocked true and every lock it ever acquired. The driver waits for the status with a bounded ledger poll, never with an unbounded awaitIdle.

A declaration the ledger cannot honour fails the playbook with a clear message. Examples are a rollback declared for an irreversible entry, a rollback the wave gate refuses, and a job the engine created for an entry with no declared attempt left.

The dead worker

rollbackJobsAreNeverPolled mirrors executeJobsAreNeverPolled for the automatic rollback phase. The first rollback wave is never polled, and one job-cleanup cycle fails it. The run ends FAILED_UNSAFE_ACK with a reason containing Rollback at step … never reached a worker, and every entry stays OPEN, so the entries can then be remediated as usual.

The shared fixture test/workflow-lifecycle/playbooks/shared/remediation-workflow.ts provides the workflow and main phases the remediation playbooks use. The workflow has three steps: configure_a and configure_b are reversible, flash is irreversible. The playbooks live in tests/remediation-*.ts and are wired in test/workflow-lifecycle/unsafe-failure-remediation.e2e-spec.ts.

State Transitions (Test-Observable)

The executor observes this subset of the full state machine. States collapsed into [engine-internal] are traversed automatically by the engine and not visible to playbook tests:

stateDiagram-v2
    engine_internal: [engine-internal]
    rollback: [rollback]

    [*] --> engine_internal
    engine_internal --> RESOURCE_DISCOVERY
    RESOURCE_DISCOVERY --> RUNNING

    RUNNING --> COMPLETED_ACK
    RUNNING --> FAILED_SAFE_ACK
    RUNNING --> FAILED_UNSAFE_ACK: irreversible FB executed
    RUNNING --> rollback: all executed non-pure FBs reversible

    rollback --> FAILED_UNSAFE_ACK
    rollback --> FAILED_SAFE_ACK

    FAILED_UNSAFE_ACK --> FAILED_UNSAFE_ACK: partial remediation (locks retained)
    FAILED_UNSAFE_ACK --> rollback: rollback all / continueRollback
    FAILED_UNSAFE_ACK --> FAILED_SAFE_ACK: ledger complete (FAILED_SAFE, locks released)

    COMPLETED_ACK --> [*]
    FAILED_SAFE_ACK --> [*]
    FAILED_UNSAFE_ACK --> [*]

The full engine state machine includes intermediate states (NEW, VALID, BLOCKED_WAITING, READY, SCHEDULED, LOCKING, LOCKED, RESOURCES_DISCOVERED, ERROR, ROLLBACK) that the executor skips over via polling. See src/model/execution/workflow-execution-status.ts for the complete enum.

System Properties

Properties are typed objects defined in test/workflow-lifecycle/playbooks/properties.ts, discriminated by type:

• SafetyProperty – Bad things don’t happen (e.g., noCrossContextWrite, rejectEmptySteps)
• LivenessProperty – Good things eventually happen (e.g., lifecycleReachesTerminalState, fbVersionResolvedByMajor)

Properties are self-reported metadata

The verifiesProperties array is documentation, not runtime enforcement. The executor does not read or check it. The actual verification comes from the playbook’s assertions (expected jobs, terminal state, CMS state). The verify() reasoning string exists to force authors to articulate why a property is covered and to enable auditing.

Run npm run playbook:index to generate an index with the full property registry, coverage counts, and all playbook descriptions.

Directory Organization

flowchart LR
    root["test/workflow-lifecycle/playbooks/"]

    root --> pb["playbook.ts<br/>Playbook interface"]
    root --> pe["playbook-executor.ts<br/>Shared execution engine"]
    root --> props["properties.ts<br/>SystemProperty registry (P.*)"]
    root --> idx["PLAYBOOK_INDEX.md<br/>Auto-generated (npm run playbook:index)"]
    root --> shared["shared/<br/>Shared FB definitions and fixtures<br/>(e.g. BasePythonEval, remediation workflow)"]
    root --> base["base/<br/>E2eInteractor base class"]
    root --> cms["cms/<br/>CMS database factories"]
    root --> utils["utils/<br/>Job builders, job matching, assertion helpers, suite runner"]
    root --> remediation["remediation/<br/>DeclaredRemediation, its plan derivation, RemediationDriver"]
    root --> tests["tests/<br/>Core workflow lifecycle scenarios"]
    root --> exec["workflow-execution-*/<br/>Execution feature tests<br/>(JMES, db-updates, embedded workflows)"]
    root --> valid["workflow-validation-*/<br/>Validation and version resolution tests"]
    root --> pubres["publish-resolution/<br/>Publish-sequence tests<br/>(drive publishes directly, then runAssumingPublished)"]
    root --> design["design-*/<br/>End-to-end workflow designs<br/>(customer scenarios, documentation examples)"]

E2eInteractor Helpers

Method	Endpoint	Description
registerWorker(name)	POST /workers/register	Register a test worker, returns UUID
registerWorkflow(dto, expectedStatus?)	POST /workflow-definition/publish	Deploy a workflow definition
registerFunctionBlocks(dtos)	POST /function-blocks/register	Register function blocks
executeWorkflow(dto)	POST /workflow-execution	Trigger a workflow execution
pollJobs(dto)	POST /blackboard/job	Poll for jobs, validates no context overlap
postJobResult(dto)	POST /blackboard/job/result	Push a job result
pollWorkflowExecution(app, id)	GET /workflow-execution/id/{id}	Fetch the current execution once
waitForState(execution, states, message)	GET /workflow-execution/id/{id}	Poll until state matches
waitForJobs(dto, execution, status)	POST /blackboard/job	Poll until jobs appear or state changes
initCms(cmsDB)	(mock injection)	Set up CMS mock data

CMS Database Factories

Reuse existing factories or define inline CMS data for specific needs:

All implement CmsDbFactory (cms/cms-db-factory.ts) and expose a create() returning a CmsDB.

Factory	Location	Contents
SimpleDb	cms/simple-db.ts	2 devices, 4 interfaces, 2 device groups
EmptyDb	design-discovery/db/empty-db.ts	Empty (for discovery tests)
DbUpdatesDb	workflow-execution-db-updates/db/db-updates-db.ts	2 devices, no interfaces/groups

Job Matching

Jobs are matched using isSubset(subset, superset) — the playbook only specifies fields it cares about. Dynamic fields like jobId can be omitted. The executor validates that polled job contexts don’t overlap (devices, deviceGroups, interfaces arrays checked pairwise via progressive merge).

Each expected job is tracked via a matchedCount. After a phase completes, the executor verifies every expected job was matched the expected number of times. For failure scenarios (FAILED_SAFE_ACK, FAILED_UNSAFE_ACK), unmatched jobs are tolerated since the workflow may abort before creating all jobs.

Debugging Failed Playbooks

When a playbook assertion fails, the expectCondition helper includes the current WorkflowExecution object (status, reason) in the failure output. Look for:

• [TESTDB] Setup test: ... in console output to find the database name for the failed test
• In local runs, the database is retained on failure — connect directly to inspect state
• The intersection() helper shows the overlap between expected and actual CMS state on cmsDbAfterExecution mismatches
• Context overlap errors come from contextesOverlap() in playbook.ts (note the typo in the function name — search for contextesOverlap in source)

See Also

• Architecture – Engine internals and handler registry
• Testing – Unit tests, E2E setup, debugging
• Adding Handlers – When your new handler needs E2E coverage

2026-03-18 09:12:072026-10-29 14:24:39