Skip to content

Operations

Health Checks

The engine exposes a health endpoint at GET /health/. Use it for:

  • Container health checks (Docker, Kubernetes)
  • Load balancer probes
  • Monitoring systems
curl http://localhost:3030/health/

A healthy response:

{
  "status": "ok"
}

Monitoring Executions

Active Executions

# All active (non-terminal) executions
curl http://localhost:3030/workflow-execution/active

# Filter by state
curl http://localhost:3030/workflow-execution/state/RUNNING
curl http://localhost:3030/workflow-execution/state/LOCKING

Execution Details

# Full execution details including jobs
curl http://localhost:3030/workflow-execution/id/<execution-id>

Active Jobs

# All active jobs (PENDING or POLLED)
curl http://localhost:3030/blackboard/jobs/active

# All jobs across all executions
curl http://localhost:3030/blackboard/jobs

Common Issues

Workflow stuck in SCHEDULED

Cause: The entities are locked by another active execution.

Resolution: Either wait for the blocking execution to complete, or abort it:

curl -X DELETE http://localhost:3030/workflow-execution/<blocking-execution-id>

Workflow stuck in LOCKING

Cause: The CMS is unreachable or not responding to the lock request.

Resolution: Check CMS connectivity (NEOPS_CMS_URL) and CMS logs. Restart the engine if the CMS was temporarily down – the engine will retry on startup.

Jobs not being picked up (many PENDING)

Possible causes:

  1. No workers online – Check GET /workers/online
  2. No worker has the required FB – Check if any online worker has the function block registered
  3. Workers not polling – Check worker logs for connectivity issues

FAILED_UNSAFE investigation

When an execution ends as FAILED_UNSAFE (meaning side effects may have occurred):

  1. Get the execution details: GET /workflow-execution/id/<id>
  2. Find the failed job(s) and which device(s) they ran on
  3. Check which steps executed successfully (those may have side effects)
  4. Verify the device state manually or through a read-only workflow
  5. Release the entity locks the execution is still holding (see below)

The affected entities stay locked

An unsafe failure is the one terminal state that does not release its CMS locks — the engine can no longer derive what state those entities are in, so it refuses to let another workflow plan against them. Find it in the logs by its error line:

Workflow failed unsafely — CMS resource locks RETAINED
    workflowId=<uuid> lockReference=<uuid> lockedDevices=[...] lockedDeviceGroups=[...]

Until someone releases that lock, any later workflow needing one of those entities waits in LOCKING, retrying on a capped exponential backoff (at most every 5 minutes) with the CMS refusal recorded as its retryNote — it never spins hot and never wedges the engine, and it resumes on its own once the lock is gone. The lock is created without an expiry, so nothing sweeps it up by itself.

There is no engine API to release it yet — this is a deliberate first cut, with the operator-facing unlock/acknowledge flow still to come. Today it means deleting the CMS CompositeLock whose reference equals the execution UUID, after the network state has been verified or corrected.

Engine restart recovery

On restart, the engine:

  1. Recovers active executions from PostgreSQL
  2. Re-evaluates execution state
  3. Resumes event processing

In-flight jobs (POLLED by workers) are handled by the stuck job cleanup. If the worker is still running, it may push results after the engine restarts.

Aborting Executions

curl -X DELETE http://localhost:3030/workflow-execution/<execution-id>

Limited implementation

The abort endpoint is currently a placeholder. Full abort behavior (entity lock release, in-flight job cancellation) is planned. For now, stuck executions will eventually time out through the stuck job cleanup mechanism.

Database Maintenance

The engine uses MikroORM with PostgreSQL. Standard PostgreSQL maintenance applies:

  • Vacuuming – PostgreSQL auto-vacuum handles this in most cases
  • Backup – Standard pg_dump for backups
  • Disk space – Completed executions and their jobs are retained indefinitely. Consider periodic cleanup of old execution data if disk space is a concern.

Version Information

# Engine version and API version
curl http://localhost:3030/version/

# Check worker SDK compatibility
curl -X POST http://localhost:3030/version/compatibility/ \
  -H "Content-Type: application/json" \
  -d '{"version": "1.0.0", "library": "neops_worker_sdk"}'