Skip to content

Operations

Health Checks

The engine exposes two health endpoints:

  • GET /health answers 503 only when neops-core is reachable but too old for the engine; an unreachable CMS or key set leaves it 200. Use it for readiness probes, load balancers, compose depends_on health gates and monitoring.
  • GET /health/live answers 200 whenever the process serves HTTP and reads no dependency. Use it for liveness probes, so a too-old core takes the engine out of service without restarting it.
curl http://localhost:3030/health

A response:

{
  "healthy": true,
  "authz": { "mode": "enforce", "keySource": "jwks", "keyReachable": true },
  "cms": { "reachable": true, "compatible": true, "checkedAt": "2026-10-26T09:00:00.000Z" }
}

healthy is true while the process serves requests, with one exception: a CMS that is reachable but too old for this engine (cms.compatible is false). Then GET /health answers 503 with healthy: false, because every run that locks devices would fail against that CMS. Everything else is a diagnostic: an unreachable CMS or key set leaves healthy true and the status 200, so an outage never fails a probe. The engine’s Helm chart (chart/neops-workflow-engine/values.yaml) points its readiness probe at /health and its liveness probe at /health/live, so a CMS outage neither restarts nor unreadies the pod, and an incompatible CMS unreadies it without restarting it: a restart cannot fix the CMS.

GET /health/live is the liveness endpoint. It answers 200 with {"live": true} whenever the process serves HTTP, and it reads no dependency: neither the CMS nor the key set. Point a probe that restarts the engine on failure at it, never at /health. Like /health, it needs no credential.

  • authz tells a wrong or unreachable verification key apart from a CMS outage without reading logs. mode is NEOPS_AUTHZ_MODE. keySource is pem (a pinned NEOPS_JWT_PUBLIC_KEY[_PATH]), jwks (NEOPS_JWKS_URL) or null (mode disabled). keyReachable is, for a JWKS, whether the key set was obtainable the last time the engine fetched it: at boot, or verifying a token. While it is false, every gated route answers 503 KEY_UNAVAILABLE. It is null before the first fetch, and always null for a PEM, which has no runtime dependency. The JWKS URL is never included.
  • cms is whether the CMS GraphQL endpoint (NEOPS_CMS_URL) answered the engine’s probe, sent with the engine’s NEOPS_CMS_TOKEN, as a GraphQL server: a status below 500 with a JSON body carrying data or errors. A GraphQL error answer, such as a rejected token, still counts as reachable. A 5xx, a timeout, a refused connection, or a body that is not a GraphQL answer counts as unreachable; a plain 404 or a proxy’s HTML page means NEOPS_CMS_URL does not point at the CMS GraphQL endpoint. The probe runs at boot, repeats every 30 seconds until it knows whether the CMS is compatible (see cms.compatible), and then runs at most every 30 seconds in the background of a health request, with a 2-second timeout. A health request never waits for it, so cms is absent until the first probe has finished. Because the chart’s readiness probe polls /health every few seconds, each pod keeps probing the CMS about every 30 seconds for as long as it runs.
  • cms.compatible is whether the CMS schema serves every field the engine’s CMS queries select. The probe introspects those types (__type(name: …) { fields { name } }); the list of required fields and the minimum neops-core release live in one place, src/engine/core-engine/cms-client/cms-requirements.ts. It is null when the probe could not tell: the CMS was unreachable, or answered with a GraphQL error, such as a rejected NEOPS_CMS_TOKEN (the CMS introspects types only for an authenticated caller). Until it is known, the boot probe repeats every 30 seconds. When it is false, cms.incompatibleReason names the missing fields and the minimum release, for example the CMS lacks LockErrorType.code; this engine needs neops-core >= 2.2.1-beta.4, and the engine logs the same reason as an error, with the CMS URL, when it detects it. Without that check, an older CMS rejects the lock mutation and every run that locks devices ends FAILED_SAFE_ACK with Cannot query field 'code' on type 'LockErrorType', while runs on global still succeed. The check reads the schema at NEOPS_CMS_URL: a proxy in front of the CMS that hides fields, or a CMS with introspection disabled, makes it report false or null respectively.

When the CMS is too old

  • Symptom. Compose reports the engine service as unhealthy. In Kubernetes the pod is running but not ready, and helm --wait times out. The Monitor App says the engine runs but neops-core is too old for it, with the reason, wherever it reaches the engine; behind a Kubernetes Service the unready pod receives no traffic, so there it reports the engine as not answering.
  • Reason. cms.incompatibleReason in curl http://localhost:3030/health, or the error line starting Upgrade neops-core: in the engine log.
  • Action. Upgrade neops-core to at least the release named there.
  • Recovery. Within about 30 seconds of the upgrade, without restarting the engine: the platform’s health checks keep the probe running and the next one finds the fields.

Monitoring Executions

Active Executions

# Executions not yet acknowledged, plus those that finished in the last 5 minutes
curl http://localhost:3030/workflow-execution/active

# Filter by state
curl http://localhost:3030/workflow-execution/state/RUNNING
curl http://localhost:3030/workflow-execution/state/LOCKING

Execution Details

# Full execution details including jobs
curl http://localhost:3030/workflow-execution/id/<execution-id>

Active Jobs

# All active jobs (PENDING or POLLED)
curl http://localhost:3030/blackboard/jobs/active

# All jobs across all executions
curl http://localhost:3030/blackboard/jobs

Common Issues

Workflow stuck in SCHEDULED

Cause: The entities are locked by another active execution.

Resolution: Either wait for the blocking execution to complete, or abort it:

curl -X DELETE http://localhost:3030/workflow-execution/<blocking-execution-id>

Workflow stuck in LOCKING

Cause: The CMS is unreachable or not responding to the lock request.

Resolution: Check CMS connectivity (NEOPS_CMS_URL) and CMS logs. Restart the engine if the CMS was temporarily down – the engine will retry on startup.

Jobs not being picked up (many PENDING)

Possible causes:

  1. No workers online – Check GET /workers/online
  2. No worker has the required FB – Check if any online worker has the function block registered
  3. Workers not polling – Check worker logs for connectivity issues

FAILED_UNSAFE investigation

When an execution ends as FAILED_UNSAFE (meaning side effects may have occurred):

  1. Get the execution details: GET /workflow-execution/id/<id>
  2. Find the failed job(s) and which device(s) they ran on
  3. Check which steps executed successfully (those may have side effects)
  4. Verify the device state manually or through a read-only workflow
  5. Release the entity locks the execution is still holding (see below)

The affected entities stay locked

An unsafe failure is the one terminal state that does not release its CMS locks — the engine can no longer derive what state those entities are in, so it refuses to let another workflow plan against them. Find it in the logs by its error line:

Workflow failed unsafely — CMS resource locks RETAINED
    workflow_execution_id=<uuid> lockReference=<uuid> lockedDevices=[...] lockedDeviceGroups=[...]

Until someone releases that lock, any later workflow needing one of those entities waits in LOCKING, retrying on a capped exponential backoff (at most every 5 minutes) with the CMS refusal recorded as its retryNote — it never spins hot and never wedges the engine, and it resumes on its own once the lock is gone. The lock is created without an expiry, so nothing sweeps it up by itself.

Release the lock through the remediation API: GET /workflow-execution/<id>/remediation lists every executed non-pure step; roll the reversible ones back (POST .../remediation/rollback, grant workflow-execution-rollback:write) and attest the rest fixed by hand (POST .../remediation/resolve, grant workflow-execution-resolve:write, with a note; the authenticated principal and the note form the audit trail). Add "continueRollback": true to a resolve to have the engine roll back the waves below the attested entry on its own (that request needs both grants); rollback {"all": true} and continueRollback put the execution back into ROLLBACK, which ends FAILED_SAFE, or returns to FAILED_UNSAFE_ACK when a rollback attempt fails or an entry only a human can compensate blocks it. The moment every entry is compensated the execution becomes FAILED_SAFE (its reason records remediated: …, or remediation: resolved by <user> for an empty ledger), and the engine releases the locks and acknowledges it as FAILED_SAFE_ACK. See Execution Lifecycle → Remediation. Deleting the CMS CompositeLock (reference = execution UUID) by hand still works as a break-glass measure, but bypasses the audit trail.

Engine restart recovery

On restart, the engine:

  1. Recovers active executions from PostgreSQL
  2. Re-evaluates execution state
  3. Resumes event processing

In-flight jobs (POLLED by workers) are handled by the stuck job cleanup. If the worker is still running, it may push results after the engine restarts. For one WORKER_OFFLINE_THRESHOLD_MS after start the cleanup judges no worker offline and no job stuck, so workers that outlived an engine outage get to ping before their jobs are failed.

An execution whose CMS unlock failed twelve times in a row is parked until the CMS answers again: the engine probes the CMS every PARKED_EXECUTION_PROBE_INTERVAL_MS (5 minutes by default) and resumes parked executions, which then release their locks. A restart resumes them too, but is not required.

Database unavailable

While the engine cannot reach PostgreSQL, every API route, the worker poll and heartbeat included, answers 503 Service Unavailable with Retry-After: 5. That covers:

  • a connection that cannot be opened: the host does not resolve (ENOTFOUND, EAI_AGAIN), cannot be routed to (EHOSTUNREACH, ENETUNREACH), refuses (ECONNREFUSED, on one address or on every address of a dual-stack host) or times out (ETIMEDOUT)
  • a connection lost mid-request (ECONNRESET, pg’s “Connection terminated unexpectedly” or “Client has encountered a connection error and is not queryable”, SQLSTATE class 08)
  • Postgres starting up or shutting down (57P01, 57P02, 57P03)
  • the connection pool giving up waiting for a free connection (knex’s acquire timeout)

Any other database error, such as a constraint violation or a query timeout, still answers 500. During an outage the engine logs one Database unavailable, answering 503 warning every 30 seconds, with the number of 503s it answered in between.

Aborting Executions

Aborting is not implemented. DELETE /workflow-execution/<execution-id> keeps its authorization contract and then answers 501 Not Implemented, writing nothing. Stuck executions end through the job cleanup timeouts; an unsafe one through remediation.

Database Maintenance

The engine uses MikroORM with PostgreSQL. Standard PostgreSQL maintenance applies:

  • Vacuuming – PostgreSQL auto-vacuum handles this in most cases
  • Backup – Standard pg_dump for backups
  • Disk space – Completed executions and their jobs are retained indefinitely. Consider periodic cleanup of old execution data if disk space is a concern.

Version Information

# Engine version and API version
curl http://localhost:3030/version/

# Check worker SDK compatibility
curl -X POST http://localhost:3030/version/compatibility/ \
  -H "Content-Type: application/json" \
  -d '{"version": "1.0.0", "library": "neops_worker_sdk"}'