Operations
Health Checks
The engine exposes two health endpoints:
GET /healthanswers 503 only when neops-core is reachable but too old for the engine; an unreachable CMS or key set leaves it 200. Use it for readiness probes, load balancers, composedepends_onhealth gates and monitoring.GET /health/liveanswers 200 whenever the process serves HTTP and reads no dependency. Use it for liveness probes, so a too-old core takes the engine out of service without restarting it.
A response:
{
"healthy": true,
"authz": { "mode": "enforce", "keySource": "jwks", "keyReachable": true },
"cms": { "reachable": true, "compatible": true, "checkedAt": "2026-10-26T09:00:00.000Z" }
}
healthy is true while the process serves requests, with one exception: a CMS that is reachable but too old for this engine (cms.compatible is false). Then GET /health answers 503 with healthy: false, because every run that locks devices would fail against that CMS. Everything else is a diagnostic: an unreachable CMS or key set leaves healthy true and the status 200, so an outage never fails a probe. The engine’s Helm chart (chart/neops-workflow-engine/values.yaml) points its readiness probe at /health and its liveness probe at /health/live, so a CMS outage neither restarts nor unreadies the pod, and an incompatible CMS unreadies it without restarting it: a restart cannot fix the CMS.
GET /health/live is the liveness endpoint. It answers 200 with {"live": true} whenever the process serves HTTP, and it reads no dependency: neither the CMS nor the key set. Point a probe that restarts the engine on failure at it, never at /health. Like /health, it needs no credential.
authztells a wrong or unreachable verification key apart from a CMS outage without reading logs.modeisNEOPS_AUTHZ_MODE.keySourceispem(a pinnedNEOPS_JWT_PUBLIC_KEY[_PATH]),jwks(NEOPS_JWKS_URL) ornull(modedisabled).keyReachableis, for a JWKS, whether the key set was obtainable the last time the engine fetched it: at boot, or verifying a token. While it isfalse, every gated route answers503 KEY_UNAVAILABLE. It isnullbefore the first fetch, and alwaysnullfor a PEM, which has no runtime dependency. The JWKS URL is never included.cmsis whether the CMS GraphQL endpoint (NEOPS_CMS_URL) answered the engine’s probe, sent with the engine’sNEOPS_CMS_TOKEN, as a GraphQL server: a status below 500 with a JSON body carryingdataorerrors. A GraphQL error answer, such as a rejected token, still counts as reachable. A 5xx, a timeout, a refused connection, or a body that is not a GraphQL answer counts as unreachable; a plain404or a proxy’s HTML page meansNEOPS_CMS_URLdoes not point at the CMS GraphQL endpoint. The probe runs at boot, repeats every 30 seconds until it knows whether the CMS is compatible (seecms.compatible), and then runs at most every 30 seconds in the background of a health request, with a 2-second timeout. A health request never waits for it, socmsis absent until the first probe has finished. Because the chart’s readiness probe polls/healthevery few seconds, each pod keeps probing the CMS about every 30 seconds for as long as it runs.cms.compatibleis whether the CMS schema serves every field the engine’s CMS queries select, and every argument they must pass it. The probe introspects those types and the arguments of their fields (__type(name: …) { fields { name args { name } } }); the list of required fields and the minimum neops-core release live in one place,src/engine/core-engine/cms-client/cms-requirements.ts. It isnullwhen the probe could not tell: the CMS was unreachable, or answered with a GraphQL error, such as a rejectedNEOPS_CMS_TOKEN(the CMS introspects types only for an authenticated caller). Until it is known, the boot probe repeats every 30 seconds. When it isfalse,cms.incompatibleReasonnames the missing fields, the minimum release and the release the CMS runs (itsappSettings.version), for examplethe CMS lacks UnlockResources.lockId, RetireLock.lockId, Mutation.unlockResources(lockId), UnlockResources.unlockedAt; this engine needs neops-core >= 2.8.0, found 2.8.0-beta.1, and the engine logs the same reason as an error, with the CMS URL, when it detects it. The minimum release is the neops-core release whose GraphQL client the engine is built with; the check reads only the listed fields, so an older core that serves them still reads as compatible. While it isfalse, the engine locks nothing: every run endsFAILED_SAFE_ACKwithreasonCodelock_refusedandFailed to lock resources:followed by that reason, runs onglobalincluded. Before it refuses a run, the engine probes again when the last probe is 30 seconds old or more, so a core upgraded since is not refused. While it isnull, the engine locks as usual, and an older CMS rejects the lock: the run endsFAILED_SAFE_ACKwithlock_refusedandCannot query field 'networkZone' on type 'DeviceType'. The check reads the schema atNEOPS_CMS_URL: a proxy in front of the CMS that hides fields, or a CMS with introspection disabled, makes it reportfalseornullrespectively.
When the CMS is too old
- Symptom. Compose reports the engine service as unhealthy. In Kubernetes the pod is running but not ready, and
helm --waittimes out. The Monitor App says the engine runs but neops-core is too old for it, with the reason, wherever it reaches the engine; behind a Kubernetes Service the unready pod receives no traffic, so there it reports the engine as not answering. - Reason.
cms.incompatibleReasonincurl http://localhost:3030/health, or the error line startingUpgrade neops-core:in the engine log. - Action. Upgrade neops-core to at least the release named there.
- Recovery. Within about 30 seconds of the upgrade, without restarting the engine: the platform’s health checks keep the probe running and the next one finds the fields. Runs started in the meantime ended
FAILED_SAFE_ACKwithlock_refused, having locked nothing; start them again.
Upgrading to 0.46
Upgrade in this order:
- Let every run finish, and start none until the engine runs 0.46 (the engine cannot abort a run yet).
- Upgrade neops-core to 2.4.0 or later.
- Upgrade the engine to 0.46.
- Upgrade the workers to Worker SDK 0.5. SDK 0.4 pins engine client 0.45.0: its worker exits at its own version
check with
Client 0.45.0 is not compatible with this engine. Update the SDK.The426at registration (see Registration) shows only when that check could not reach the engine. The worker side of the upgrade is in the Worker SDK’s worker lifecycle. - Set
NEOPS_WORKER_NETWORK_ZONES=defaulton the central workers: once the CMS holds a zone besidesdefault, a worker that lists no zones is refused. - Create the zones in the CMS, then start each zone’s workers with
NEOPS_WORKER_NETWORK_ZONESset to their zones.
From a 0.46 beta, restart every worker after the upgrade, so the Monitor’s Workers page can flag one that lists no
zones: a worker the beta linked to default reads as listing its zones until it registers again. Do the same after
going back to 0.45 and forward again, where the upgrade marks every worker that kept running as listing none.
Out of order:
- An engine upgraded first refuses to lock against the older core, so new runs end
FAILED_SAFE_ACKwithlock_refused. A run that 0.45 had already locked is unlocked by 0.46, whose unlock selects a field the older core lacks: that run endsFAILED_UNSAFE_ACKwithunlock_failedand keeps its locks. Upgrade core first, then retry the write-back (POST /workflow-execution/<id>/remediation/write-back): a resolve answers409while that unlock has no recorded answer, and the retry sends it again, with the run’s writes, to the upgraded core. - A run that finishes on 0.45 after core is upgraded can have its write-back refused with
WRITE_REFUSED, which 0.45 reads as a failed unlock. It retries the write-back with backoff and parks the run after twelve failures, still holding its locks. Because the CMS answers, 0.45’s parked-run probe resumes the run withinPARKED_EXECUTION_PROBE_INTERVAL_MS(5 minutes by default), and the loop starts again. Upgrade the engine to 0.46: it resumes the parked run when it starts, reads the refusal, and ends the run withwrite_refused(see A write-back the CMS refuses).
Saved log queries: the lines Rollback job failed and Rollback job succeeded name their job by job_id, as every
other job line does, no longer by rollbackJob or completedRollbackJob.
Monitoring Executions
Active Executions
# Executions not yet acknowledged, plus those that finished in the last 5 minutes
curl http://localhost:3030/workflow-execution/active
# Filter by state
curl http://localhost:3030/workflow-execution/state/RUNNING
curl http://localhost:3030/workflow-execution/state/LOCKING
Execution Details
# Full execution details including jobs
curl http://localhost:3030/workflow-execution/id/<execution-id>
Active Jobs
# All active jobs (PENDING or POLLED)
curl http://localhost:3030/blackboard/jobs/active
# All jobs across all executions
curl http://localhost:3030/blackboard/jobs
Common Issues
Workflow stuck in SCHEDULED
Cause: The entities are locked by another active execution.
Resolution: Either wait for the blocking execution to complete, or abort it:
Workflow stuck in LOCKING
Cause: The CMS is unreachable or not responding to the lock request.
Resolution: Check CMS connectivity (NEOPS_CMS_URL) and CMS logs. Restart the engine if the CMS was temporarily down – the engine will retry on startup.
Jobs not being picked up (many PENDING)
Possible causes:
- No workers online – Check
GET /workers/online - No worker has the required FB – Check if any online worker has the function block registered
- Workers not polling – Check worker logs for connectivity issues
A worker’s result push answered 400
When the push’s JSON parsed but failed validation, the response and the worker’s log name the fields. Since 0.46.0
the engine fails the job the push names with invalid_result (resultRejected: result_body) when the pushing worker
holds it, so the run ends at once and goes through its normal failure handling (see
A result push that fails validation).
Before, the job stayed claimed until BLACKBOARD_STUCK_JOB_TIMEOUT, its run kept its locks, and the epoch’s drain held
new runs back meanwhile. The fix is in the worker or its function block: a result that passes validation.
A body that is not JSON the engine can parse is answered before validation runs and fails no job: a 400 with
entity.parse.failed, such as a NaN or Infinity a Python worker wrote into its result data, or a 413 for a body
over the size limit. That job still waits for BLACKBOARD_STUCK_JOB_TIMEOUT.
FAILED_UNSAFE investigation
When an execution ends as FAILED_UNSAFE (meaning side effects may have occurred):
- Get the execution details:
GET /workflow-execution/id/<id> - Find the failed job(s) and which device(s) they ran on
- Check which steps executed successfully (those may have side effects)
- Verify the device state manually or through a read-only workflow
- Release the entity locks the execution is still holding (see below)
The affected entities stay locked
An unsafe failure is the one terminal state that does not release its CMS locks — the engine can no longer derive what state those entities are in, so it refuses to let another workflow plan against them. A run whose locks the CMS no longer holds has none (lost locks). Find it in the logs by its error line:
Workflow failed unsafely — CMS resource locks RETAINED
workflow_execution_id=<uuid> lockReference=<uuid> lockedDevices=[...] lockedDeviceGroups=[...]
Until someone releases that lock, any later workflow needing one of those entities waits in
LOCKING, retrying on a capped exponential backoff (at most every 5 minutes) with the CMS
refusal recorded as its retryNote — it never spins hot and never wedges the engine, and it
resumes on its own once the lock is gone. The lock is created without an expiry, so
nothing sweeps it up by itself.
Release the lock through the remediation API: GET /workflow-execution/<id>/remediation
lists every executed step that may have changed something; roll the reversible ones back
(POST .../remediation/rollback, grant workflow-execution-rollback:write) and attest the rest
fixed by hand (POST .../remediation/resolve, grant workflow-execution-resolve:write, with a
note; the authenticated principal and the note form the audit trail). Add
"continueRollback": true to a resolve to have the engine roll back the waves below the
attested entry on its own (that request needs both grants); rollback {"all": true} and continueRollback put the execution
back into ROLLBACK, which ends FAILED_SAFE, or returns to FAILED_UNSAFE_ACK when a
rollback attempt fails or an entry only a human can compensate blocks it. The moment every
entry is compensated the execution becomes FAILED_SAFE (its reason records remediated: …,
or remediation: resolved by <user> for an empty ledger), and the engine releases the locks
and acknowledges it as FAILED_SAFE_ACK. See
Execution Lifecycle → Remediation.
Releasing the CMS CompositeLock (reference = execution UUID) in the CMS admin (“Unlock
composite locks”) still works as a break-glass measure, but bypasses the engine’s audit trail:
the CMS records the release as admin and applies none of the run’s writes. The engine is not
told and does not ask: it finds the release only when the run releases its lock, and the run then
ends as lost locks
end. A rollback in between, automatic or asked for, acts on devices that are no longer locked and
that another run may hold. Every job of the run that acted after the CMS last confirmed the lock
leaves it FAILED_UNSAFE without locks, owing a resolve that confirms the CMS was checked against
the devices (cmsCorrected); see
When the run’s jobs acted after the CMS last confirmed its lock.
Release a run this way only to resolve it afterwards: it then ends unsafe. The same holds for
“Refresh composite locks”, which gives a lock an expiry the CMS’s sweep then enforces.
A write-back retry refused naming incomplete-remediation-retains-locks
While an unlock of a run has no recorded answer, rollback and resolve answer 409, and the
write-back retry sends that unlock again
(An unlock of unknown outcome blocks rollback and resolve).
When sending it would complete a run whose ledger an operator already touched, the retry answers
409 naming incomplete-remediation-retains-locks and changes nothing. The engine never lets such
an unlock come about (it takes no rollback and no resolve while an unlock is unknown), so the run’s
records disagree with its ledger: engine records were lost or edited.
No procedure settles this, in the engine or by hand: the run keeps its CMS locks and takes no rollback or resolve. Collect, for support:
- the run’s ledger and reason:
GET /workflow-execution/<id>/remediation; - its unlock attempts:
select * from cms_unlock_attempt where workflow_execution_uuid = '<id>' order by id; - the CMS’s record of its lock: core’s
lockStatequery withreference= the run’s id (the engine never sends it; ask the CMS directly); - the engine’s log lines carrying
workflow_execution_id=<id>.
A 409 of the same kind that names jobs-act-only-on-locked-entities is transient: a job of the run
still has no result, and the engine does not send the unlock again until every job has one. Send the
retry again then.
Engine restart recovery
On restart, the engine:
- Recovers active executions from PostgreSQL
- Re-evaluates execution state
- Resumes event processing
In-flight jobs (POLLED by workers) are handled by the stuck job cleanup. If the worker is still running, it may push results after the engine restarts. Both waits below count from when the engine starts serving requests, so slow migrations don’t use them up. For one WORKER_OFFLINE_THRESHOLD_MS the cleanup judges no worker offline, so workers that outlived an engine outage get to ping before their jobs are failed. Stuck jobs are judged from the job’s own age, not from pings, so the stuck check waits only BLACKBOARD_JOB_CHECK_INTERVAL or 30 s (the worker’s result-push retry cap), whichever is longer: enough for a worker that finished during the outage to push its result, without adding an offline threshold to a job that was already stuck.
An execution whose CMS unlock, or the retire of its released lock, failed twelve times in a row is parked until the CMS answers again: the engine probes the CMS every PARKED_EXECUTION_PROBE_INTERVAL_MS (5 minutes by default) and resumes the parked executions that hold their locks, or that hold none but have not yet retired the lock they asked for or released: they then release their locks or retire the lock, and a run parked while still in the lock phase asks for its lock again. A restart resumes them too, and every other parked execution, but is not required.
Database unavailable
While the engine cannot reach PostgreSQL, every API route, the worker poll and heartbeat included, answers 503 Service Unavailable with Retry-After: 5. That covers:
- a connection that cannot be opened: the host does not resolve (
ENOTFOUND,EAI_AGAIN), cannot be routed to (EHOSTUNREACH,ENETUNREACH), refuses (ECONNREFUSED, on one address or on every address of a dual-stack host) or times out (ETIMEDOUT) - a connection lost mid-request (
ECONNRESET, pg’s “Connection terminated unexpectedly” or “Client has encountered a connection error and is not queryable”, SQLSTATE class08) - Postgres starting up or shutting down (
57P01,57P02,57P03) - the connection pool giving up waiting for a free connection (knex’s acquire timeout)
Any other database error, such as a constraint violation or a query timeout, still answers 500. During an outage the engine logs one Database unavailable, answering 503 warning every 30 seconds, with the number of 503s it answered in between.
CMS unavailable while a run executes
A run calls the CMS only in its lock phase and at its unlock (see Where the engine reads zones). A locked run keeps executing while the CMS is down: its results are taken and checked without it, and only its unlock waits for the CMS.
A write-back the CMS refuses
The CMS checks a run’s writes at its unlock, and refuses the whole write-back when the writes can never be applied as
sent: an invalid value, a broken constraint, or a write its rules forbid. It then applies none of them and keeps the
locks. The run fails with reasonCode write_refused and the CMS’s message in its reason. Once the refusal is recorded, the engine never
sends the writes again on its own, not on a retry and not after a restart.
- No side effects (every executed step was pure or reported that it changed nothing): the run ends
FAILED_SAFE_ACK. Its writes are discarded, as after a rollback, and the engine releases the locks with an unlock that carries no writes. Fix the workflow or the data it reads and run it again. -
Side effects: the devices may hold what the CMS refused to record. The run ends
FAILED_UNSAFE_ACKwithunsafeCausewrite_refused, keeps its writes and keeps its locks, like any unsafe run. There are three ways out:- Fix the cause in the CMS and retry the write-back
(
POST /workflow-execution/<id>/remediation/write-back, or Retry write-back… in the Monitor). Applied, the locks come off and the run isCOMPLETED_ACK, once it has retired its lock. Refused again, the run stays as it was with the new reason. A retry answered503may still have been applied, as its answer says: send it again, and the CMS answers that it already applied the writes rather than applying them twice. Until the engine knows, rollback and resolve answer409. See Retrying a refused write-back. - Roll the steps back, resolving those only a person can undo. The engine then releases the locks without the writes, and the CMS keeps what it held before the run.
- Resolve the steps once you have checked the devices, and reconcile the CMS by hand. The locks come off without the writes.
A retry is taken only before a rollback or resolve starts: from then on the writes no longer describe the devices.
- Fix the cause in the CMS and retry the write-back
(
Any other unlock that fails retries with backoff. An unlock that fails twelve times in a row parks the run, still
holding its locks, until the CMS answers again: the engine probes the CMS every PARKED_EXECUTION_PROBE_INTERVAL_MS
(5 minutes by default) and then resumes the run (see Engine restart recovery). Telling a
refused write-back apart needs neops-core 2.4.0 or later, which answers it with the code WRITE_REFUSED.
The engine needs a neops-core that names the lock row on every lock answer and takes it on an
unlock (lockId, neops-core 2.8.0 or later), which retires a
reference with no lock row by leaving a retired one, and applies a run’s writes only under the lock row it granted the
run, never under a newer one a late lock request of the run took. /health refuses a core whose schema lacks any of
these, whatever release it reports (cms-requirements.ts lists them and names the minimum release). The run entry’s
cmsLockId is the id the CMS admin’s lock list shows for the run’s lock. Roll out
core first, then this engine version: a core pod older than 2.8.0 answers every unlockResources
and retireLock of this engine with a GraphQL validation error and executes nothing. The engine never falls back to an
unlock that names no row, and never ends a run on that error: a release retries it like any unlock that fails
(above), the run keeping its status and its locks, backing off for about half an hour, then parked and resumed by the
parked-run probe; a retire retries, parks and is resumed the same way, the run’s locks already released; a
write-back retry answers 503 and changes nothing. Once every core pod
runs 2.8.0 or later, send the write-back retry again. A pod older than 2.8.0-beta.1 also leaves no
retired row for a retire with no lock row, so a late lock request can take a lock nobody releases. /health probes one
pod, so it cannot tell a mixed fleet. Since 2.7.0
core keeps every lock row and answers a repeated unlock from that history:
an unlock whose reply was lost is answered as already released when it is sent again, and a lock released without the
run names who released it. Each run’s entry records the answer (cmsLockRelease, cmsWritesApplied). Once that
record is committed, or once a run whose lock request got no answer ends, the engine retires the run’s lock reference in
the CMS (retireLock) before it acknowledges the run, so the CMS refuses any later lock or unlock of that reference. Retired locks stay in the CMS until pruned there
(./manage.py prune_lock_history --retired-before <days>). A retire the CMS does not take is retried like an unlock.
A CMS that stops answering
Each CMS GraphQL call the engine makes, except the /health probe, gives up when the whole answer has not arrived
within NEOPS_CMS_REQUEST_TIMEOUT_MS (30 seconds by default); a paged read waits that long for each page. The
/health probe keeps its own 2-second timeout. The engine then treats a timed-out call like any other CMS outage:
- A read of the network zones fails. Worker registration waits for that read for at most 10 seconds, inside the
worker’s own 30-second request timeout, then answers as it does while the CMS is down: from the engine’s last read
of the zones, or
503if the engine has never read them; a zone that read lacks is refused as unread, not as missing from the CMS. The Worker SDK takes that400as final and exits, even for a zone created a moment ago: restart the worker once the CMS answers. The read goes on, and the table takes it in if it arrives. -
A run whose declarations name a zone the table lacks waits for that read at most 2 seconds when its resource discovery starts, so the CMS does not hold the engine’s other runs. Without the read the run refuses no zone and sits in
RESOURCE_DISCOVERY: itsACQUIREjob in that zone waits for a worker of the zone, and none can register while the table lacks it. The run page showsZone <name> · No worker yet; the Workers page shows nothing, sinceGET /network-zoneslists the table’s zones only.- A zone the CMS holds: its workers exited on the
400their registration got while the CMS was unread. Restart them onceGET /healthshows the CMS answering; they register, and the job runs. - A zone the CMS lacks (a typo, or not created yet): the run waits the whole
BLACKBOARD_PENDING_JOB_TIMEOUT, 60 minutes by default, then endsFAILED_SAFE_ACKwithreasonCodeacquire_failed. The run’s reason names the step and the zone and says no worker polled the job.
A publish naming such a zone waits for the read up to
NEOPS_CMS_REQUEST_TIMEOUT_MSbefore it answers. A run’s lock attempt retries with backoff, as when the lock itself fails. - A zone the CMS holds: its workers exited on the
-
Locking and unlocking retry with backoff. Neither is done twice when the CMS finished it after the engine gave up: a lock retry finds the run’s lock already taken, and an unlock retry finds it released and counts it as done.
A CMS that accepts connections but never answers, for example behind a stuck proxy, no longer holds locks, unlocks and registrations forever. Raise the value if large unlocks, such as a discovery that writes thousands of devices, legitimately take longer than 30 seconds: too low a value makes them time out and retry, again and again.
Aborting Executions
Aborting is not implemented. DELETE /workflow-execution/<execution-id> keeps its
authorization contract and then answers 501 Not Implemented, writing nothing. Stuck
executions end through the job cleanup timeouts; an unsafe one through remediation.
Database Maintenance
The engine uses MikroORM with PostgreSQL. Standard PostgreSQL maintenance applies:
- Vacuuming – PostgreSQL auto-vacuum handles this in most cases
- Backup – Standard
pg_dumpfor backups - Disk space – Completed executions and their jobs are retained indefinitely. Consider periodic cleanup of old execution data if disk space is a concern.
Going back to 0.45
Go back in this order, the reverse of Upgrading to 0.46:
- Let every run finish.
- Stop every worker.
- Take the engine back to 0.45: the SQL and the migration revert below.
- Take neops-core below 2.4.0, if you go back that far, as core’s own release notes for 2.4.0 describe under going back. Core goes after the engine: a 0.46 engine refuses to lock against a core older than 2.4.0-beta.2, and every unlock it sends there fails.
- Start the workers on Worker SDK 0.4.
Let every run finish first (the engine cannot abort a run yet): a job still waiting loses its zone, and 0.45 hands it to whichever worker polls first.
Stop every worker too, and run only workers on Worker SDK 0.4 while you stay on 0.45. 0.45 has no zones, so it hands a job of any zone to whichever worker polls first, a 0.5 worker that keeps running included.
0.45 cannot retry a write-back. Retry or remediate every write_refused run you can on 0.46 first; an unsafe one left
stays FAILED_UNSAFE_ACK and is remediated on 0.45 like any unsafe run.
0.45 would also let an operator roll back a refused run whose locks the CMS no longer holds, and its rollback jobs could reach devices another run now holds. Resolve those runs on 0.46: this query must return no rows before you go on.
select uuid from workflow_execution
where unsafe_cause = 'write_refused' and status = 'FAILED_UNSAFE_ACK' and not cms_resources_locked;
The down() of the 0.46 migrations restores 0.45’s CHECK constraints, which refuse the values 0.46 writes, so it fails
while any row holds one. Clear those values first. The rows keep their reason, which still says what failed:
update job_result set result_rejected = null where result_rejected in ('network_zone', 'cms_lookup', 'result_body', 'patch_path');
update job_result set reason_code = 'step_failed' where reason_code = 'unknown_network_zone';
update workflow_execution set reason_code = 'step_failed' where reason_code = 'unknown_network_zone';
update job_result set reason_code = 'unlock_failed' where reason_code = 'write_refused';
update workflow_execution set reason_code = 'unlock_failed' where reason_code = 'write_refused';
update workflow_execution set unsafe_cause = 'unlock_failed' where unsafe_cause = 'write_refused';
Then revert the 0.46 migrations, the first of them, Migration20261030184705, included, from the 0.46 image’s working
directory (/app), with the engine’s database settings in the environment, and start 0.45:
node -e "const { MikroORM } = require('@mikro-orm/core');
const config = require('./dist/neops-workflow-engine/src/mikro-orm.config').default;
MikroORM.init(config).then(async orm => {
await orm.getMigrator().down({ to: 'Migration20261030184705' });
await orm.close(true);
});"
The command loads the engine’s configuration, so it reads what the engine reads, from the environment or from a .env
file in the working directory:
POSTGRES_HOST(defaultlocalhost),POSTGRES_PORT(default5434) andPOSTGRES_PASSWORD.POSTGRES_USER(defaultpostgres). The database is alwaysneops-workflow.- Any
MIKRO_ORM_*variable,MIKRO_ORM_USERincluded, overrides the setting it names. - A malformed value of any other engine variable stops the command, as it would stop the engine.
A down that fails changes nothing: the migrations revert in one transaction, so the database stays as 0.46 left it. Run the SQL above and repeat the command.
Reverting drops the engine’s zone table, each job’s zone and each run’s zone snapshot.
The command also reverts 0.45’s own last two migrations, because migrations revert in name order and their names sort
after Migration20261030184705: Migration20261102104651 renames rollback_data_size_bytes back to
rollback_data_bytes, and Migration20261102113953 deletes the rows of refused workers. 0.45 applies both again when it
starts, so its list of refused workers starts empty.
Reverting also drops the zones each worker registered. When you go forward to 0.46 again, restart every worker so that
it registers its zones anew: a worker that kept running counts as serving only default.
Version Information
# Engine version and API version
curl http://localhost:3030/version/
# Check worker SDK compatibility
curl -X POST http://localhost:3030/version/compatibility/ \
-H "Content-Type: application/json" \
-d '{"version": "1.0.0", "library": "neops_worker_sdk"}'

