Skip to content

Neops Workflow Engine

The Workflow Engine is the orchestration core of the neops platform. It takes declarative YAML workflow definitions, breaks them into discrete jobs, distributes them to workers via a shared job queue called the blackboard, and tracks execution with transaction-like safety guarantees. Entity data (devices, interfaces, groups, credentials) is managed by the CMS (Content Management System -- the neops-core component). If a step fails, the engine knows whether it can safely retry, roll back, or must stop -- based on each function block's purity and idempotency contracts.

Architecture

sequenceDiagram
    participant U as User / API
    participant WE as Workflow Engine
    participant CMS as CMS
    participant BB as Blackboard
    participant W as Worker
    participant ND as Network Devices

    U->>WE: Submit workflow (YAML)
    WE->>WE: Validate + resolve function blocks
    WE->>CMS: Acquire entity data + locks
    CMS-->>WE: Entity context
    WE->>BB: Create jobs
    W->>BB: Poll for jobs
    BB-->>W: Job payload
    W->>ND: Execute function block
    W-->>BB: Write results
    BB-->>WE: Job results
    WE->>CMS: Persist updates + release locks

You define what should happen (a workflow). The engine handles when, where, and how safely it executes. Workers handle the how -- connecting to devices and running the actual logic.

Platform Components

The Worker SDK documentation covers how to write function blocks. CMS and Platform documentation is under development.

Where to Start

If you want to... Start here
Write and run your first workflow Getting Started → then Workflows
Understand the concepts and architecture Concepts → then Transactions
Write function blocks for your workflows Worker SDK → then come back to Function Blocks
Deploy and operate the engine Deployment → then Operations
Evaluate what's implemented vs. planned Implementation Status → then Concepts

Explore the Documentation

  • Getting Started

    Run the engine, deploy your first workflow, and see it execute end to end.

    Quick start

  • Concepts

    Workflows, function blocks, the blackboard, transactions, and the execution model.

    Core concepts

  • Workflows

    Write workflow definitions: steps, parameters, conditions, acquire, retry, and scheduling.

    Workflow authoring

  • Function Blocks

    How the engine registers, resolves, and orchestrates function blocks.

    Function blocks

  • Monitor App

    Use the built-in UI to deploy workflows, trigger executions, and inspect jobs.

    Monitor App

  • Deployment & Operations

    Configuration, Docker deployment, worker management, and health monitoring.

    Deploy

  • Development

    Engine architecture, testing patterns, and how to extend the engine.

    For contributors

  • Appendix

    Glossary, schema reference, FAQ, and troubleshooting.

    Appendix

Why Workflows?

If you already automate with Ansible playbooks, Nornir runbooks, or standalone Python scripts, here is what neops workflows add:

Declarative, not imperative : Define what should happen as YAML, not how. The engine handles execution order, parallelism, and error recovery. Your automation logic lives in function blocks -- small, tested, reusable units.

Transaction safety : The engine tracks whether each step is pure (read-only) or idempotent (safe to re-run). A workflow that only ran pure steps can be safely discarded. One that ran idempotent steps can be retried. The engine makes this decision automatically.

Vendor-agnostic composition : Function blocks connect to devices through the Worker SDK's proxy/plugin system. A workflow that collects interface counters works on Cisco, Juniper, and Arista without branching logic -- the right plugin is resolved at runtime.

Built-in state tracking : When a function block modifies a device, interface, or group, the SDK diffs the changes and the engine persists them atomically to the CMS. No manual API calls, no forgotten updates.

Observable execution : Every workflow execution is tracked as a state machine. You can inspect which step is running, which devices have completed, and what failed -- through the API or the Monitor App.

How It Works

  1. A workflow definition (YAML) describes a sequence of steps -- function blocks, nested workflows, or references to other workflows.
  2. When executed, the engine resolves function block versions, acquires entity context from the CMS, and locks resources.
  3. The engine creates jobs on the blackboard -- a shared queue where workers poll for work.
  4. Workers pick up jobs matching their registered function blocks, execute them against network devices, and push results back.
  5. The engine processes results, continues to the next step or handles failure (retry, rollback, or stop).
  6. On completion, the engine releases locks and applies any data changes to the CMS atomically.