Worker Management
Workers are processes running the Worker SDK that execute function blocks. The engine tracks worker health, manages job assignment, and automatically handles unresponsive workers.
Worker States
stateDiagram-v2
[*] --> ONLINE: register + heartbeat
ONLINE --> UNREACHABLE: no ping for 2 min
UNREACHABLE --> ONLINE: ping received
UNREACHABLE --> OFFLINE: no ping for 6 min
OFFLINE --> ONLINE: ping received
OFFLINE --> Deleted: auto-cleanup after 24h
Deleted --> [*]
[*] --> REFUSED: registration refused
REFUSED --> REFUSED: refused again
REFUSED --> Deleted: no attempt for 24h, or the worker registers
| State | Condition | Impact on jobs |
|---|---|---|
| ONLINE | Heartbeat within last 2 minutes | Jobs are assigned normally |
| UNREACHABLE | No heartbeat for 2-6 minutes | No new jobs assigned, existing jobs continue |
| OFFLINE | No heartbeat for 6+ minutes | In-flight jobs are auto-failed |
| Deleted | Offline for 24+ hours | Worker registration removed |
| REFUSED | Registration refused; the row carries the refusal |
Never gets a job; not a worker, see Refused workers |
Threshold Cascade
The thresholds are designed to cascade:
0 min Worker stops pinging
2 min UNREACHABLE (WORKER_UNREACHABLE_THRESHOLD_MS)
6 min OFFLINE (3x unreachable, WORKER_OFFLINE_THRESHOLD_MS)
12 min Stuck jobs failed (2x offline, BLACKBOARD_STUCK_JOB_TIMEOUT)
24 h Worker deleted (WORKER_OFFLINE_DELETE_THRESHOLD_MS)
Worker Lifecycle
Registration
Workers register on startup via POST /workers/register and receive a UUID. They then register their function blocks with this UUID, enabling the blackboard to route jobs to them.
A worker also names the network zones it serves (networkZones, from the SDK’s NEOPS_WORKER_NETWORK_ZONES) and only claims jobs of those zones. Once the CMS holds a zone besides default, every worker must name its zones; a central worker sets NEOPS_WORKER_NETWORK_ZONES=default in its own service definition. See Network Zones.
Registration enforces the same rule as POST /version/compatibility, which a worker does not have to call: the engine refuses a generated Python client that the compatibility check rejects, or that is on the kill-switch list. While the engine is 0.x a client must be on the engine’s minor, so a 0.44 client restarting against a 0.45 engine is refused, and so is a 0.45 client against a 0.44 engine. Upgrade the engine first, then its workers, and pin both in the same change.
Client 0.46.0-beta.1 (0.46.0b1 on PyPI), which Worker SDK 0.5.0-beta.1 pins, is withdrawn: it is on the kill-switch list, because that SDK exits when the engine echoes a worker that listed no zones as ["default"]. Move those workers to Worker SDK 0.5.0-beta.2 or later. A 0.5.0-beta.1 worker calls POST /version/compatibility first, which answers compatible: false without a reason, and exits with its own line: Client 0.46.0-beta.1 is not compatible with workflow engine <engine version>; refusing to start. Run the SDK release built for this engine. The 426 at registration, which says the client is withdrawn and names Worker SDK 0.5.0-beta.2 or later, shows only when that check could not run, as for SDK 0.4 in Upgrading to 0.46.
A refused worker gets 426 Upgrade Required naming both versions, the engine logs a warning with the address it connected from and lists the refusal, and no working worker is created, so the SDK exits at startup. At the license’s worker cap the quota check answers its 403 first, so an old worker sees the upgrade message only once a slot frees. The engine reads the client version from the User-Agent header, not from the version the worker registers, since a worker built from a commit reports 0.0.0.
The check does not refuse what it cannot read. A worker that does not use the generated client, or whose header a proxy rewrites, is not checked, nor is a client version the compatibility check cannot parse. An engine that reports develop, which every image not built from a release tag does, applies only the kill-switch list. An engine stamped through NEOPS_VERSION compares against that stamp, so stamp only the real release.
The check applies only at registration, so a worker that registered before an engine upgrade keeps taking jobs until it restarts. After upgrading the engine, look through GET /workers/online for an SDK version older than the release pinned with this engine (the list does not show the client version), then stop that worker or remove it with DELETE /workers/:uuid, which fails the jobs it holds. Its next ping answers 404, the SDK exits, and its restart is refused. A worker built from a commit reports 0.0.0 and cannot be told apart in this list.
Refused workers
A refused registration shows up in GET /workers and in the Monitor App’s Workers list with state refused and the message the worker was told in refusal, so a restart loop of an old worker is visible without reading logs. The Monitor App shows only its link and its reason: a refused row has no jobs, function blocks or logs, and nothing to remove.
Such a row reports an attempt and is no worker. It is not in GET /workers/online, does not count against the license’s worker quota, and its UUID, which the worker never learns, answers 404 on ping, unregister, log push and job poll, on function block registration, and on GET /workers/:uuid/jobs.
- One row per worker. A refusal is recorded under
(version, name, ipAddress), as the worker reported them; an absent field is a value of its own. Workers that report the same three share a row, which is what the replicas of one deployment do. A repeat refresheslastPingAtandrefusaland brings a soft-deleted row back. - It clears itself. A registration with the same
nameandipAddressremoves the rows of that worker, whateverversionthey recorded. A worker that reports no name clears nothing, since it cannot tell itself from any other unnamed worker. Otherwise the row ages out like an offline worker, 24 hours after the last attempt (WORKER_OFFLINE_DELETE_THRESHOLD_MS);POST /workers/cleanup-offlinealso removes one idle for longer thanWORKER_OFFLINE_THRESHOLD_MS. - It is bounded.
POST /workers/registeris public and the name is the caller’s, so at most 100 refused rows are kept, soft-deleted ones included. At the cap a new key takes the place of the oldest soft-deleted row. With none to replace it stays in the log only, while known keys keep updating. Several new keys arriving at once just below the cap can pass it by a few rows; it comes back down as those age out. - Names are not proof.
name,ipAddressandversionare what the caller claims, also in a refused row. The address it really connected from is in the log warning, asremoteAddress. - The log stays the record. A refusal that could not be stored is logged and still answers
426. At the worker cap the quota answers403first, so that attempt is neither refused nor listed.
Heartbeat
Workers send periodic pings (POST /workers/:uuid/ping). The Worker SDK defaults to every 20 seconds. If a ping returns 404 (worker was cleaned up), the worker shuts down.
Graceful Shutdown
On SIGTERM/SIGINT, workers call POST /workers/:uuid/unregister. This immediately marks the worker as offline without deleting its function block registrations. The worker can come back online later by re-registering and sending a ping.
Stuck Job Cleanup
The engine runs a cleanup service every minute (BLACKBOARD_JOB_CHECK_INTERVAL). It finds jobs that have been in POLLED state longer than BLACKBOARD_STUCK_JOB_TIMEOUT (default: 12 minutes) and marks them as failed.
This handles:
- Workers that crash without graceful shutdown
- Network partitions between workers and the engine
- Worker processes that hang during execution
API Reference
| Endpoint | Method | Description |
|---|---|---|
/workers/register |
POST | Register a new worker |
/workers/:uuid/ping |
POST | Send heartbeat |
/workers/:uuid/unregister |
POST | Graceful shutdown |
/workers |
GET | List all workers (add ?findDeleted=true for soft-deleted, ?populateFunctionBlocks=true for each worker’s FB ids) |
/workers/online |
GET | List online workers only |
/workers/:uuid |
GET | Worker details |
/workers/:uuid/function-blocks |
GET | Worker’s registered FBs |
/workers/:uuid/jobs |
GET | Worker’s jobs (filter with ?status=polled) |
/workers/:uuid |
DELETE | Delete a worker |
/workers/cleanup |
POST | Manual cleanup (offline > 24h) |
/workers/cleanup-offline |
POST | Manual cleanup (all currently offline) |
/network-zones |
GET | Every network zone, default first, with its online workers and waiting jobs |
Scaling Workers
- Horizontal scaling – Deploy multiple workers with the same function blocks. The blackboard distributes jobs across them.
- Specialized workers – Deploy workers with different function block sets (e.g., one worker for Cisco FBs, another for Juniper).
- Workers per network zone – Deploy at least one worker in each zone, inside the network it reaches. Two workers per zone keep its jobs moving when one stops; the
developer-previewengine admits at most 2 workers per installation, so that needs the licensed (enterprise) engine once there is more than one zone. - Geographic distribution – Workers can run in different locations as long as they can reach the engine’s REST API and the target devices.
Monitoring worker health
Use GET /workers/online to check how many workers are available. If jobs are
queuing up (many PENDING jobs), add more workers. If workers are frequently going
UNREACHABLE, check network connectivity between workers and the engine.

