Skip to content

The Blackboard

The blackboard is the shared job queue that decouples the workflow engine from the workers. The engine writes jobs; workers read them, execute them, and write results back. This pattern allows workers to scale independently, fail without taking down the engine, and be deployed across different machines or containers.

How It Works

sequenceDiagram
    participant Engine as Workflow Engine
    participant BB as Blackboard
    participant W1 as Worker 1 (echo, ping)
    participant W2 as Worker 2 (configBackup)

    Engine->>BB: Create jobs
    par Workers poll
        W1->>BB: Poll for matching jobs
        W2->>BB: Poll for matching jobs
    end
    BB-->>W1: Job assignment
    BB-->>W2: Job assignment
    W1->>W1: Execute function block
    W2->>W2: Execute function block
    W1->>BB: Push result
    W2->>BB: Push result
    BB-->>Engine: Result events

The blackboard is not a separate service – it is a set of database tables and REST endpoints within the workflow engine. Workers interact with it over HTTP.

Job Lifecycle

Every job transitions through three states:

stateDiagram-v2
    [*] --> PENDING: Engine creates job
    PENDING --> POLLED: Worker claims job
    POLLED --> PUSHED: Worker submits result
    PUSHED --> [*]

PENDING : The engine created the job but no worker has claimed it. The job sits in the queue until a worker with the matching function block polls for it.

POLLED : A worker claimed the job. The job is now assigned to that worker and will not be returned to other workers. If the worker fails to push a result within the timeout (default: 12 minutes), the job is marked as failed.

PUSHED : The worker submitted a result (success or failure). The engine processes the result and advances the workflow.

Job Types

Type When it’s created What the worker does
ACQUIRE During resource discovery phase Runs the function block’s acquire() method to determine which additional entities are needed
EXECUTE During the running phase Runs the function block’s run() method with the full entity context
ROLLBACK During error recovery Runs the function block’s rollback() method to undo changes, given the step’s forward result (forwardResult)

Polling Mechanics

Workers poll the blackboard periodically (the Worker SDK defaults to every 10 seconds, but this is configured on the worker side). Each poll request includes:

  • workerId – The worker’s unique ID
  • functionBlockIds – List of function block IDs the worker can execute

The blackboard returns pending jobs that match the worker’s function blocks (default: 1 job per poll, configurable via limit; at most one ROLLBACK job per response). The assignment is atomic – once a job is returned to a worker, it is marked as POLLED and will not be returned to other workers.

Each returned job carries a lease, leaseToken: 64 random hex characters, new for every polled job. The worker proves it holds the job by sending its workerId and that lease with every push for the job, its result and its log lines. A push from another worker, or with another lease, is answered 403 LEASE_MISMATCH; a job that holds no lease (polled before the engine handed leases out) is answered 403 LEASE_MISSING and is left to the job cleanup.

Jobs the engine closes without a worker

The engine can close a PENDING job before any worker claims it, in two ways: a workflow failure cancels its still-pending jobs, and the job cleanup fails a job no worker polled within BLACKBOARD_PENDING_JOB_TIMEOUT. Either way the job gets a result while no worker holds it, which sets its canceled flag. Such a job provably never ran and is never claimable again. The claim query skips it, and the partial index backing that query, job_pending_claim_index, drops it through its predicate worker_uuid IS NULL AND NOT canceled.

sequenceDiagram
    participant W as Worker
    participant BB as Blackboard

    loop Every 10 seconds
        W->>BB: POST /blackboard/job<br/>{workerId, functionBlockIds}
        alt Jobs available
            BB->>W: [Job 1, Job 2, ...]
            W->>W: Execute function blocks
            W->>BB: POST /blackboard/job/result<br/>{jobExecutionId, workerId, leaseToken, status, result}
        else No matching jobs
            BB->>W: []
        end
    end

Log Lines

Workers push their log lines to the engine, which adds what it knows and stores them in the log store (NEOPS_LOG_BACKEND), or writes them to its own stdout when none is configured:

  • POST /blackboard/job/logs – lines of a job, with jobId, workerId and the job’s leaseToken. The engine adds the run, workflow, step, job, function block, worker and entity from the job row. Lines that arrive after the job’s result are kept and marked late.
  • POST /workers/:uuid/logs – lines outside any job (startup, heartbeat, poll errors). They carry the worker only.

Both answer 202 with the number of accepted lines and a reason for each refused one (a message, a field value or the entry’s metadata over its size limit, or a job line timed more than a minute before the job existed). A key the body may not carry is refused for the whole request: 400 UNKNOWN_LOG_KEY for a key the route does not declare, 400 RESERVED_LOG_FIELD for a fields key the record or core’s task log already uses, each with the path of the key. The limits are published in schema/LogRecord.schema.json (x-neops-limits); a body is at most 1 MiB.

Reading Log Lines

Four routes read the lines back from the log store. Each returns a page of records in the total order (timestamp, service, process, job_id, seq):

Route Permission Returns
GET /workflow-execution/:id/logs workflow-execution-log:read the run’s lines from the engine, the workers and neops-core; filters stepLabel, jobId, jobType, entityType, entityId, workerId, includeCore, includeEngine
GET /blackboard/jobs/:jobId/logs workflow-execution-log:read one job’s lines; includeAttempts=true adds the earlier attempts it retries
GET /workers/:uuid/logs worker:read a worker’s lines outside any job; with an identifier filter, its job lines across runs that match it, which also needs workflow-execution-log:read
GET /logs workflow-execution-log:read lines across runs; filters stepLabel, jobType, entityType, entityId, workerId, includeCore, includeEngine; the window starts 24 hours ago; needs at least one narrowing filter (below)

Every route takes level (a minimum: warn returns warn and error), limit (1 to 1000, default 200), order (asc or desc), a window start/end, and exact-match identifier filters: workflowId and functionBlockId (pkg/name:x.y.z) and their parts workflowPackage, workflowName, workflowVersion, functionBlockPackage, functionBlockName, functionBlockVersion. Only job lines carry them. The window starts five minutes before the run or job was created (24 hours ago for a worker), ends five minutes from now, and spans at most 30 days. An ascending page carries nextCursor; pass it as after to read on. A read with after returns only records older than settledUntil (now minus NEOPS_LOG_SETTLE_MS), so a polling view never skips a line that a slower producer delivers late; a line that takes longer than that to arrive appears on the next read without after. A descending read is one page of the newest records.

GET /logs answers 400 LOG_FILTER_REQUIRED unless it names a workflow or function block (id or any part), an entity (entityType, entityId), a worker or a job type. The store sorts every line in the window to return a page, so a read narrowed only by level, step or service would sort a whole day of the installation; a narrowing filter keeps the sorted set to the lines that filter matches. Widening the window to 30 days multiplies that set accordingly.

Without a log store the routes answer 503 LOGS_NOT_CONFIGURED; a failing store answers 502 LOG_BACKEND_ERROR. GET /version reports under logs whether logs can be read (queryable) and how delivery to the store is going.

Result Structure

When a worker pushes a result, it includes:

  • status – Success or failure
  • result – The function block’s outcome:
    • success and message
    • data – The function block’s return data (conforms to the resultDataJsonSchema)
    • dbUpdates (for EXECUTE jobs) – Entity create/patch/delete operations to apply to the CMS
    • retryable (optional, on a failure) – false when the failure would recur on every attempt, so the engine does not retry the job; see Automatic Retries
    • sideEffects (optional, for EXECUTE jobs) – false when the attempt changed nothing, on a success or a failure; see Steps that changed nothing
  • acquires (for ACQUIRE jobs) – Additional entity acquisition requests
  • rollbackData (optional, for EXECUTE jobs of reversible function blocks) – The snapshot the function block’s rollback() needs. It is checked against the block’s rollbackDataJsonSchema and stored apart from the result; a snapshot that fails a check is dropped without failing the push. See Rollback Data

The engine processes results asynchronously through an event-driven system. Each result triggers the next step in the workflow execution.

Scaling

The blackboard pattern enables horizontal scaling:

  • Multiple workers can poll simultaneously. Each gets different jobs.
  • Specialized workers can register only specific function blocks (e.g., a worker with Cisco credentials only handles IOS-related FBs).
  • Worker failure is isolated. If a worker crashes, its jobs time out and can be reassigned.
  • Engine restart preserves state. All jobs and execution state are in PostgreSQL.
Why not a message queue?

The blackboard uses PostgreSQL instead of a dedicated message queue (RabbitMQ, Redis) for simplicity and transactional consistency. Job state, workflow state, and function block registrations are all in the same database, enabling atomic operations across them. For most network automation workloads, PostgreSQL’s performance is more than sufficient.

See also: