E2E Playbooks

Quick start

Jump to Writing a New Playbook if you want to create one now. Use test/workflow-lifecycle/playbooks/tests/simple-workflow-2.ts as a complete reference example.

Prerequisites

Familiarity with Architecture (engine internals) and Testing (unit + E2E setup) is recommended before writing playbooks.

The Playbook pattern is the primary way to write E2E tests for the workflow engine. A playbook defines a complete execution scenario declaratively: the CMS state, the workflow, the function blocks, the expected jobs, and the expected terminal state. Each playbook also declares which safety and liveness properties it verifies, with reasoning.

Writing a New Playbook

1. Define the playbook

Create a file in the appropriate tests/ directory. Here is a complete, minimal example (this is a real playbook that runs in CI — see test/workflow-lifecycle/playbooks/tests/doc-example-playbook.ts):

import { Playbook } from '@test/workflow-lifecycle/playbooks/playbook'
import { P, verify } from '@test/workflow-lifecycle/playbooks/properties'
import { successJob } from '@test/workflow-lifecycle/playbooks/utils/job-builders'
import { WorkflowExecutionStatus } from '@src/model/execution/workflow-execution-status'
import { EntityType } from '@src/model/workflow/schema/entity-type'
import { FunctionBlockType } from '@src/model/entities/function-block'

export const DocExamplePlaybook: Playbook = {
  description: 'Single-step ping workflow completes on two devices',
  verifiesProperties: [
    verify(P.lifecycleReachesTerminalState, 'Workflow reaches COMPLETED_ACK after ping executes'),
    verify(P.singleStepCompletes, 'One-step ping workflow runs to completion on 2 devices'),
  ],

  // CMS mock state — all three arrays required. Or use: new SimpleDb().create()
  cmsDbBeforeExecution: {
    devices: [
      { id: 1, hostname: '10.0.0.1', facts: {} },
      { id: 2, hostname: '10.0.0.2', facts: {} },
    ],
    deviceGroups: [],
    interfaces: [],
  },

  // Workflow definition — the YAML/JSON equivalent as a TypeScript object
  createWorkflowDefinitionDto: {
    workflow: {
      type: 'workflow',
      label: 'my_workflow',
      name: 'my_workflow',
      package: 'wf.neops.test', // package/name:version = workflow ID
      majorVersion: 1,
      minorVersion: 0,
      patchVersion: 0,
      seedEntity: EntityType.DEVICE,
      runOn: EntityType.DEVICE,
      parameters: {},
      config: { executionStrategy: {} },
      steps: [
        {
          type: 'functionBlock',
          label: 'ping',
          description: 'Ping the device',
          functionBlock: 'fb.neops.io/ping:1', // package/name:majorVersion
          runOn: EntityType.DEVICE,
          parameters: { host: 'localhost' },
        },
      ],
    },
  },

  // Function blocks to register — must match what the workflow references
  registerFunctionBlockDtos: [
    {
      name: 'ping', // ← combined with package = fb.neops.io/ping
      package: 'fb.neops.io',
      majorVersion: 1,
      minorVersion: 0,
      patchVersion: 0,
      description: 'Ping a host',
      runOn: EntityType.DEVICE,
      fbType: FunctionBlockType.CHECK,
      parameterJsonSchema: {
        type: 'object',
        properties: { host: { type: 'string' } },
        required: ['host'],
      },
      resultDataJsonSchema: {},
    },
  ],

  // Trigger execution on device IDs 1 and 2
  executeWorkflowDto: {
    workflow: 'wf.neops.test/my_workflow:1.0.0', // package/name:version
    executeOnParameters: { ids: [1, 2] },
    parameters: { host: 'localhost' },
  },

  // Expected jobs — successJob() creates { job, jobResult: { status: SUCCESS, result: { success: true } } }
  expectedWorkflowExecutionStateAfterSubmit: WorkflowExecutionStatus.RESOURCE_DISCOVERY,
  expectedAcquireJobs: [successJob({ functionBlockId: 'fb.neops.io/ping:1.0.0' })],
  expectedExecuteJobs: [successJob({ functionBlockId: 'fb.neops.io/ping:1.0.0', context: {} })],
  expectedTerminalState: WorkflowExecutionStatus.COMPLETED_ACK,
}

Key conventions:

• Function block IDs follow the pattern package/name:major.minor.patch (e.g., fb.neops.io/ping:1.0.0)
• Workflow references use the same pattern (e.g., wf.neops.test/my_workflow:1.0.0)
• successJob(job) creates a success response; pass extra result fields as a second arg: successJob(job, { dbUpdates: {...} })
• For expectedNumberOfPolls: successJob(job, undefined, { expectedNumberOfPolls: 2 }). If omitted, the executor verifies the job was matched at least once
• All three cmsDbBeforeExecution arrays (devices, deviceGroups, interfaces) are required
• To test validation failures that reject at submission, use expectedStateOverrides:

// Playbook expecting workflow definition to be rejected
expectedStateOverrides: {
  workflowDefinitionPostResponse: HttpStatus.BAD_REQUEST,
},
expectedTerminalState: WorkflowExecutionStatus.FAILED_SAFE_ACK,
// No jobs will run — the executor aborts after the BAD_REQUEST

2. Wire to a test spec

Use setupPlaybookSuite inside a describe block — each it is written explicitly so IDE test runners (WebStorm, VSCode) can detect them:

import { setupPlaybookSuite } from '@test/workflow-lifecycle/playbooks/utils/run-playbook-suite'
import { MyPlaybook } from './tests/my-playbook'

describe('My Test Suite', () => {
  const suite = setupPlaybookSuite()

  it('single-step ping completes on two devices', () => suite.run(MyPlaybook))
})

setupPlaybookSuite registers beforeEach/afterEach hooks (NestJS app init with fresh database + teardown) and returns a context with a run method that wires PlaybookExecutor automatically. Each test gets an isolated PostgreSQL database; on success it is dropped. On failure in local runs, it is retained for debugging (look for [TESTDB] in test output for the database name). In CI, databases are always dropped.

3. Verify and regenerate

npm run lint:fix
npm run test
npm run test:e2e
npm run playbook:index

To run a single playbook during development:

npx jest --config ./test/jest-e2e.json --testPathPattern=my-test-suite

Architecture

• Playbook -- A TypeScript object declaring the complete test scenario, including property verifications with reasoning.
• setupPlaybookSuite() -- Registers app init/teardown hooks and returns a context for running playbooks.
• PlaybookExecutor -- Runs the playbook step by step, simulating the worker side.
• E2eInteractor -- Provides HTTP helper methods for API interaction.
• properties.ts -- Single source of truth for all system properties (SafetyProperty | LivenessProperty).
• successJob() -- Helper to reduce job expectation boilerplate.

Playbook Interface

interface Playbook {
  description: string                    // What this playbook tests
  verifiesProperties: SystemPropertyVerification[]  // Properties + reasoning
  cmsDbBeforeExecution: CmsDB            // Mock CMS state (devices, deviceGroups, interfaces — all required)
  createWorkflowDefinitionDto: InstanceType<typeof CreateWorkflowDefinitionDto>
  registerFunctionBlockDtos: RegisterFunctionBlockPlaybookDto[]
  executeWorkflowDto?: InstanceType<typeof ExecuteWorkflowDto>  // Omit = registration-only test
  expectedWorkflowExecutionStateAfterSubmit?: WorkflowExecutionStatus
  expectedAcquireJobs?: PlaybookJobAndResponse[]
  expectedExecuteJobs?: PlaybookJobAndResponse[]
  expectedRollbackJobs?: PlaybookJobAndResponse[]
  expectedTerminalState?: WorkflowExecutionStatusTerminalStates  // COMPLETED_ACK | FAILED_SAFE_ACK | FAILED_UNSAFE_ACK
  cmsDbAfterExecution?: CmsDB            // Verify CMS state after execution via isSubset()
  expectedStateOverrides?: ExpectedStateOverrides  // Override expected HTTP status for workflow registration (e.g., BAD_REQUEST for validation failures)
}

Execution Flow

The PlaybookExecutor runs the playbook in this order:

1. Initialize CMS -- Set up mock CMS data via CmsLockMockService
2. Register workflow -- POST /workflow-definition. If expectedStateOverrides.workflowDefinitionPostResponse is set to BAD_REQUEST, assert and abort early
3. Register function blocks -- POST /function-blocks/register for each FB
4. Execute workflow -- POST /workflow-execution. If no executeWorkflowDto, abort (registration-only test)
5. Wait for RESOURCE_DISCOVERY -- Or a terminal state if validation fails early
6. Register worker -- Create a test worker named playbook-test-worker
7. Acquire phase -- Poll for ACQUIRE jobs immediately, match via isSubset(), post expected responses
8. Execute phase -- Wait for EXECUTE jobs (polling until available), post responses, loop while RUNNING
9. Rollback phase -- If workflow enters FAILED_UNSAFE, poll and respond to rollback jobs. Can result in either FAILED_UNSAFE_ACK or FAILED_SAFE_ACK
10. Assert terminal state -- Verify the execution ended in the expected state
11. Assert CMS state -- If cmsDbAfterExecution is set, verify via isSubset(). On mismatch, the intersection() helper shows the overlap for debugging

State Transitions (Test-Observable)

The executor observes this subset of the full state machine. States collapsed into [engine-internal] are traversed automatically by the engine and not visible to playbook tests:

stateDiagram-v2
    engine_internal: [engine-internal]
    rollback: [rollback]

    [*] --> engine_internal
    engine_internal --> RESOURCE_DISCOVERY
    RESOURCE_DISCOVERY --> RUNNING

    RUNNING --> COMPLETED_ACK
    RUNNING --> FAILED_SAFE_ACK
    RUNNING --> FAILED_UNSAFE

    FAILED_UNSAFE --> rollback
    rollback --> FAILED_UNSAFE_ACK
    rollback --> FAILED_SAFE_ACK

    COMPLETED_ACK --> [*]
    FAILED_SAFE_ACK --> [*]
    FAILED_UNSAFE_ACK --> [*]

The full engine state machine includes intermediate states (NEW, VALID, BLOCKED_WAITING, READY, SCHEDULED, LOCKING, LOCKED, RESOURCES_DISCOVERED, ERROR, ROLLBACK) that the executor skips over via polling. See src/model/execution/workflow-execution-status.ts for the complete enum.

System Properties

Properties are typed objects defined in test/workflow-lifecycle/playbooks/properties.ts, discriminated by type:

• SafetyProperty -- Bad things don't happen (e.g., noCrossContextWrite, rejectEmptySteps)
• LivenessProperty -- Good things eventually happen (e.g., lifecycleReachesTerminalState, fbVersionResolvedByMajor)

Properties are self-reported metadata

The verifiesProperties array is documentation, not runtime enforcement. The executor does not read or check it. The actual verification comes from the playbook's assertions (expected jobs, terminal state, CMS state). The verify() reasoning string exists to force authors to articulate why a property is covered and to enable auditing.

Run npm run playbook:index to generate an index with the full property registry, coverage counts, and all playbook descriptions.

Directory Organization

test/workflow-lifecycle/playbooks/
├── playbook.ts               # Playbook interface
├── playbook-executor.ts      # Shared execution engine
├── properties.ts             # SystemProperty registry (P.*)
├── PLAYBOOK_INDEX.md         # Auto-generated (npm run playbook:index)
├── shared/                   # Shared FB definitions (e.g., BasePythonEval)
├── base/                     # E2eInteractor base class
├── cms/                      # CMS database factories
├── utils/                    # Job builders, assertion helpers, suite runner
├── tests/                    # Core workflow lifecycle scenarios
├── workflow-execution-*/     # Execution feature tests (JMES, db-updates)
└── workflow-validation-*/    # Validation & version resolution tests

E2eInteractor Helpers

Method	Endpoint	Description
registerWorker(name)	POST /workers/register	Register a test worker, returns UUID
registerWorkflow(dto, expectedStatus?)	POST /workflow-definition	Deploy a workflow definition
registerFunctionBlocks(dtos)	POST /function-blocks/register	Register function blocks
executeWorkflow(dto)	POST /workflow-execution	Trigger a workflow execution
pollJobs(dto)	POST /blackboard/job	Poll for jobs, validates no context overlap
postJobResult(dto)	POST /blackboard/job/result	Push a job result
waitForState(execution, states, message)	GET /workflow-execution/id/{id}	Poll until state matches
waitForJobs(dto, execution, status)	POST /blackboard/job	Poll until jobs appear or state changes
initCms(cmsDB)	(mock injection)	Set up CMS mock data

CMS Database Factories

Reuse existing factories or define inline CMS data for specific needs:

Factory	Contents
SimpleDb	2 devices, 4 interfaces, 2 device groups
EmptyDb	Empty (for discovery tests)
DbUpdatesDb	2 devices, no interfaces/groups

Job Matching

Jobs are matched using isSubset(subset, superset) — the playbook only specifies fields it cares about. Dynamic fields like jobId can be omitted. The executor validates that polled job contexts don't overlap (devices, deviceGroups, interfaces arrays checked pairwise via progressive merge).

Each expected job is tracked via a matchedCount. After a phase completes, the executor verifies every expected job was matched the expected number of times. For failure scenarios (FAILED_SAFE_ACK, FAILED_UNSAFE_ACK), unmatched jobs are tolerated since the workflow may abort before creating all jobs.

Debugging Failed Playbooks

When a playbook assertion fails, the expectCondition helper includes the current WorkflowExecution object (status, reason) in the failure output. Look for:

• [TESTDB] Setup test: ... in console output to find the database name for the failed test
• In local runs, the database is retained on failure — connect directly to inspect state
• The intersection() helper shows the overlap between expected and actual CMS state on cmsDbAfterExecution mismatches
• Context overlap errors come from contextesOverlap() in playbook.ts (note the typo in the function name — search for contextesOverlap in source)

See Also

• Architecture -- Engine internals and handler registry
• Testing -- Unit tests, E2E setup, debugging
• Adding Handlers -- When your new handler needs E2E coverage