Neops Workflow Engine

YAML-driven workflow orchestration engine for network automation. Composes reusable function blocks into transactional workflows, manages job distribution to workers via a blackboard pattern, and handles execution lifecycle, retries, and rollbacks.

Quick Start

Start development dependencies (PostgreSQL + Monitor App):

docker compose up -d

Run the engine locally with hot reload:

npm run start:dev

Service	URL
Workflow Engine API	http://localhost:3030
Swagger UI	http://localhost:3030/api
Monitor App	http://localhost:5173
PostgreSQL	localhost:5434

See docker-compose.yml for configurable variables (POSTGRES_PASSWORD, POSTGRES_PORT, MONITOR_PORT).

Prerequisites

• Node.js 20+
• npm 10+
• PostgreSQL 15+ (or Docker)

Development Setup

git clone https://github.com/zebbra/neops-workflow-engine.git
cd neops-workflow-engine
npm install

Start a PostgreSQL instance (if not using Docker Compose):

docker run -d --name neops-postgres \
  -e POSTGRES_PASSWORD=unsafe \
  -p 5434:5432 \
  postgres:15

Create a .env file or export environment variables:

POSTGRES_HOST=localhost
POSTGRES_PORT=5434
POSTGRES_USER=postgres
POSTGRES_PASSWORD=unsafe

Run database migrations and start the engine:

npx mikro-orm migration:fresh
npm run start:dev

Verify:

curl http://localhost:3030/health/

Monitor App

The Monitor App is a built-in web UI for managing workflows, triggering executions, and inspecting jobs and workers:

make run-monitor

Open http://localhost:5173. The app connects to the engine at http://localhost:3030 by default (configurable in Settings).

Testing

npm run test          # unit tests
npm run test:e2e      # end-to-end tests (requires running PostgreSQL)
npm run test:cov      # unit tests with coverage
npm run lint          # ESLint
npm run format        # Prettier check

CI in Docker

Run the full CI pipeline (lint + unit tests + e2e tests) in a single Docker build:

export NPM_TOKEN=ghp_xxx
DOCKER_BUILDKIT=1 docker build \
  --secret id=npm_token,env=NPM_TOKEN \
  --target run-ci \
  -t zebbra/neops-workflow-engine:ci .

Individual stages: --target linter, --target tester, --target tester-e2e.

Docker Image

Build the production image:

export NPM_TOKEN=ghp_xxx
DOCKER_BUILDKIT=1 docker build \
  --secret id=npm_token,env=NPM_TOKEN \
  -t quay.io/zebbra/neops-workflow-engine:latest .

The build uses BuildKit secrets so the token is never written to image layers.

Documentation

Build and serve the docs locally:

make doc-serve

The full documentation covers workflow authoring, function blocks, deployment, the Monitor App, and more.

Database Schema

classDiagram
direction BT
class function_block {
   timestamp with time zone created_at
   timestamp with time zone updated_at
   jsonb json_schema
   varchar(255) description
   varchar(255) package
   varchar(255) name
   varchar(255) version
}
class function_block_registration {
   timestamp with time zone created_at
   timestamp with time zone updated_at
   varchar(255) function_block_package
   varchar(255) function_block_name
   varchar(255) function_block_version
}
class job {
   timestamp with time zone created_at
   timestamp with time zone updated_at
   varchar(255) function_block_package
   varchar(255) function_block_name
   varchar(255) function_block_version
   jsonb parameters
   jsonb context
   uuid workflow_execution_uuid
   uuid uuid
}
class job_result {
   timestamp with time zone created_at
   timestamp with time zone updated_at
   text status
   jsonb result
   uuid job_uuid
}
class worker_registration {
   timestamp with time zone created_at
   timestamp with time zone updated_at
   uuid uuid
}
class workflow {
   timestamp with time zone created_at
   timestamp with time zone updated_at
   jsonb workflow_definition
   varchar(255) id
}
class workflow_execution {
   timestamp with time zone created_at
   timestamp with time zone updated_at
   varchar(255) workflow_id
   uuid uuid
}

function_block_registration  -->  function_block : function_block_package, function_block_name, function_block_version-package, name, version
job  -->  function_block : function_block_package, function_block_name, function_block_version-package, name, version
job_result  -->  job : job_uuid-uuid
job  -->  workflow_execution : workflow_execution_uuid-uuid
workflow_execution  -->  workflow : workflow_id-id

Project Structure

src/
├── engine/           # Core execution engine (workflow runner, job scheduler)
├── resources/        # NestJS controllers and services (REST API)
├── model/            # Domain models and DTOs
├── persistence/      # MikroORM entities and migrations
├── validator/        # Workflow JSON Schema validation
├── swagger/          # OpenAPI spec generation
├── config.ts         # Environment configuration
└── main.ts           # Application entrypoint

rest/monitor-app/     # Svelte-based Monitor App (UI)
examples/             # Standalone workflow YAML examples
schema/               # Generated JSON Schema for workflows
docs/                 # MkDocs documentation source
test/                 # E2E test suites