Configuration
All configuration is via environment variables. No config files are required. The engine reads .env if present (via dotenv).
A numeric variable takes a plain integer: digits only, no unit, no exponent, no separators. A byte
size takes a number with an optional binary unit: 5242880, 100kb, 50mb, 5MiB and 5 MiB
all work, and mb means MiB. Leading and trailing spaces are ignored. Any other value, such as
60000ms, 1.5e6 or 5 242 880, stops the engine at boot with an error that names the variable
and the value, so a typo cannot start the engine with a limit you did not set.
Runtime & Logging
| Variable | Default | Description |
|---|---|---|
NODE_ENV |
development |
development, production, or test. The published image sets production in its runner stage. The development escapes read the declared value, so an unset NODE_ENV does not open them |
DEBUG |
false |
Enable verbose validation errors. Also switches deferred retry delays from seconds to milliseconds – intended for tests, never for production |
NEOPS_LOG_LEVEL |
info |
error, warn, info, debug. Defaults to debug when DEBUG=true. Replaces LOG_LEVEL |
NEOPS_LOG_FORMAT |
pretty |
json (production) or pretty (development); text is read as pretty. Auto-selects json in production. Replaces LOG_FORMAT |
PORT |
3030 |
HTTP server port |
Log store
The engine writes its own log lines to a log store (VictoriaLogs or Loki) next to stdout. Lines go
through an in-memory queue to the store; while the store is unreachable or refuses the engine’s
settings, and when the queue overflows, they wait in the log_outbox table of the engine’s own
database and are replayed once the store accepts a probe. Request and response body dumps and the
job result dump stay on stdout only.
| Variable | Default | Description |
|---|---|---|
NEOPS_LOG_BACKEND |
(unset) | victorialogs or loki. Unset: stdout only, nothing queued or stored, logs are not queryable. db is core’s value; the engine treats it as unset and says so at boot. Any other value aborts startup |
NEOPS_LOG_URL |
(unset) | Base URL of the store, e.g. http://victorialogs:9428 or http://loki:3100. Required with a backend; startup is refused without it |
NEOPS_LOG_USERNAME / NEOPS_LOG_PASSWORD |
(unset) | Basic auth for the store |
NEOPS_LOG_TENANT |
(unset) | Loki: a tenant id without whitespace, sent as X-Scope-OrgID. VictoriaLogs: <uint32>[:<uint32>], sent as AccountID and ProjectID (0 when no project is given). A malformed value aborts startup |
NEOPS_LOG_ENV |
(unset) | Deployment label written as env on every record |
NEOPS_LOG_QUEUE_MAX_RECORDS |
10000 |
In-memory queue; when full, its oldest lines move to the outbox |
NEOPS_LOG_BATCH_MAX_RECORDS |
500 |
Records per push; a full batch is pushed before the next tick |
NEOPS_LOG_BATCH_MAX_BYTES |
2097152 |
JSON-encoded bytes per push |
NEOPS_LOG_FLUSH_INTERVAL_MS |
200 |
Flusher period |
NEOPS_LOG_OUTBOX_MAX_ROWS |
500000 |
Outbox row cap; beyond it the oldest rows are deleted |
NEOPS_LOG_OUTBOX_MAX_BYTES |
1073741824 |
Outbox byte cap (1 GiB); beyond it the oldest rows are deleted |
NEOPS_LOG_OUTBOX_MAX_ATTEMPTS |
10 |
Replay failures after which a row is deleted; only failures in a tick where the store accepted another push count |
NEOPS_LOG_SETTLE_MS |
3000 |
Live-read horizon of the log read routes; values below 1000 are raised to 1000 |
A 401, 403, 404 or 405 from the store (a wrong URL, wrong credentials, or an
auth-enabled Loki without NEOPS_LOG_TENANT) logs one error and keeps every line in the outbox,
probing every 30 s. A network error, a timeout, a 5xx or a 429 keeps the lines in the outbox
and probes every 5 s (or after Retry-After). A store that refuses a line itself (400, 413,
422) loses that line only. Postgres jsonb cannot hold a NUL character or an unpaired UTF-16
surrogate, so a line that waits in the outbox has each replaced by U+FFFD; a line pushed straight
from the queue keeps them.
The outbox is a second copy of every line during an outage
While the store is away the engine’s Postgres holds up to NEOPS_LOG_OUTBOX_MAX_BYTES of
log rows, and a backup taken then carries them. The table is empty in steady state.
HTTP
| Variable | Default | Description |
|---|---|---|
NEOPS_CORS_ORIGINS |
(unset) | Comma-separated exact browser origins allowed to call the API cross-origin, e.g. https://neops.example,https://monitor.example. Unset with NODE_ENV=development reflects any origin; unset otherwise sends no CORS headers at all |
Most routes accept a request body of up to 100 kB. One route takes bulk payloads from workers and is raised to 50 MB:
POST /blackboard/job/result– job results, which can carry bulk entity updates
The two log push routes, POST /blackboard/job/logs and POST /workers/:uuid/logs, accept
1 MiB. That limit is part of their contract (x-neops-limits.bytesPerRequest in
schema/LogRecord.schema.json, which workers chunk by), so MAX_REQUEST_BODY_SIZE does not change it.
| Variable | Default | Description |
|---|---|---|
MAX_REQUEST_BODY_SIZE |
(unset) | Replaces the 50 MB limit of POST /blackboard/job/result, e.g. 100mb, 100MiB or a plain byte count. Leave unset to keep the built-in limit |
Set this when workers push results larger than 50 MB and get 413 Payload Too Large.
It does not change the 100 kB default
MAX_REQUEST_BODY_SIZE only adjusts the result route. Every other route, including workflow
publishing and the log push routes, keeps its limit and is unaffected by this variable.
A malformed value aborts startup
The value is checked when the engine starts, and anything that is not a byte size stops the boot with an error. This is deliberate: an unparseable limit would otherwise be read as unlimited, so a typo would silently remove the cap instead of setting it.
License
The license is compiled into the image and cannot be changed at runtime – there is no
environment variable that grants or upgrades one. To see which license is active and how much of
each quota is in use, call GET /license/; see
API Reference.
The build-time variables behind it are documented under Docker Deployment.
Database
The engine requires PostgreSQL 15+ and is tested against 17 (the version docker-compose.yml runs).
Its database user needs the CREATE privilege on its database (the database owner has it): a migration
installs the pg_trgm extension, which the run list’s text search uses.
| Variable | Default | Description |
|---|---|---|
POSTGRES_HOST |
localhost |
Database hostname |
POSTGRES_PORT |
5434 |
Database port |
POSTGRES_USER |
postgres |
Database user |
POSTGRES_PASSWORD |
unsafe |
Database password (override in production) |
Change the default password
The default password unsafe is for local development only. Always set a strong
password in production deployments.
Database Setup
Run migrations to create the schema:
npx mikro-orm migration:fresh # development (drops and recreates)
npx mikro-orm migration:up # production (applies pending migrations)
CMS Connection
The engine communicates with the neops CMS via GraphQL for entity acquisition and locking.
| Variable | Default | Description |
|---|---|---|
NEOPS_CMS_URL |
http://localhost:8000/graphql |
CMS GraphQL endpoint |
NEOPS_CMS_TOKEN |
unsafe |
Authentication token for CMS API. unsafe is the built-in placeholder; the engine refuses to start carrying it unless NODE_ENV=development |
CMS dependency
The CMS is required for entity acquisition and locking. Without a CMS connection,
workflows that use acquire clauses or seedEntity other than global will fail
during the resource discovery phase.
Authentication & Authorization
The engine verifies Neops-issued RS256 access tokens and authorizes a route from the permissions claim they carry. Permissions are granted in the neops CMS as <element>:<flag> strings.
| Variable | Default | Description |
|---|---|---|
NEOPS_AUTHZ_MODE |
enforce |
enforce, permissive, or disabled – see below |
NEOPS_JWT_PUBLIC_KEY |
(unset) | The CMS’s public key as an SPKI PEM, inline |
NEOPS_JWT_PUBLIC_KEY_PATH |
(unset) | A file holding that PEM, e.g. a mounted ConfigMap |
NEOPS_JWKS_URL |
(unset) | The CMS’s JWKS endpoint. https unless NODE_ENV=development AND NEOPS_ALLOW_INSECURE_JWKS=true |
NEOPS_ALLOW_INSECURE_JWKS |
false |
Accepts a plaintext-HTTP NEOPS_JWKS_URL, alongside a development NODE_ENV |
NEOPS_ALLOW_DISABLED_AUTHZ |
false |
Runs NEOPS_AUTHZ_MODE=disabled where NODE_ENV is not development |
NEOPS_JWT_ISSUER |
neops |
The iss a token must carry; matches the CMS’s own setting |
NEOPS_JWT_AUDIENCE |
neops-services |
The aud a token must carry; matches the CMS’s own setting |
Which routes these apply to
Every route handler carries exactly one authorization marker. The public set – the container
probes, GET /version, POST /version/compatibility and the worker channel – is served
anonymously in every mode. Every other route needs a valid token, and enforce needs the
route’s permission as well. test/auth/route-table.e2e-spec.ts holds the full table.
Providing the verification key
Pin the key. It is a public key, so it belongs in a ConfigMap – mount the same PEM the CMS loads from its own NEOPS_JWT_PUBLIC_KEY_PATH, or pass it inline. NEOPS_JWKS_URL is the fallback: it puts the trust anchor on the network, and the CMS serves exactly one key with no rotation window.
The engine refuses to start when none of the three is set, naming them in the failure.
A plaintext-HTTP JWKS is a total bypass
Whoever can answer for that host – a pod on the network, a compromised sidecar, DNS or ARP
control – serves their own key set, and can then mint tokens with any subject and any
permissions. Accepting one requires both NODE_ENV=development and
NEOPS_ALLOW_INSECURE_JWKS=true.
Modes
| Mode | Authentication | Authorization |
|---|---|---|
enforce (default) |
Valid Neops JWT required | The route’s permission must be in the token |
permissive |
Valid Neops JWT required | A token reporting authz_enforced: false is allowed through, with a warning |
disabled |
None – anonymous | None. Refuses to start unless NODE_ENV=development or NEOPS_ALLOW_DISABLED_AUTHZ=true |
permissive is for a CMS running without its permissions plugin: a CMS whose NEOPS_PLUGINS omits neops_permissions_simple issues tokens whose permissions claim is always empty and whose authz_enforced claim is false, so an enforce engine denies every gated route to every user. An enforce engine logs one WARN naming that pairing when the first such token arrives. Authentication stays required in permissive, so every request still carries an identity.
disabled is for a workstation with no CMS at all. It verifies nothing and serves every endpoint anonymously, including the ones that start executions against real devices.
NEOPS_ALLOW_DISABLED_AUTHZ=true runs it where NODE_ENV is anything else, and the engine logs an ERROR naming the mode at every boot for as long as it is set. That pairing is the recourse for an engine that can verify no token at all – a wrong key, issuer or audience answers 401 to every request in permissive too, since that mode still requires a valid token.
The token’s authz_enforced claim only ever tightens: a token minted by an enforcing CMS is enforced whatever the engine’s mode says. Operator configuration is the one input that loosens anything.
Worker Management
These thresholds control how the engine detects and handles unresponsive workers. The defaults cascade: unreachable (2 min) -> offline (3x = 6 min) -> stuck job (2x offline = 12 min).
| Variable | Default | Description |
|---|---|---|
WORKER_UNREACHABLE_THRESHOLD_MS |
120000 (2 min) |
Time since last heartbeat before worker is marked unreachable |
WORKER_OFFLINE_THRESHOLD_MS |
360000 (6 min) |
Time since last heartbeat before worker is marked offline and its jobs are failed. An offline worker no longer counts against the license worker quota |
WORKER_OFFLINE_DELETE_THRESHOLD_MS |
86400000 (24 h) |
Time before an offline worker is soft-deleted |
WORKER_CLEANUP_ENABLED |
true |
Enable automatic deletion of stale workers |
WORKER_CLEANUP_INTERVAL_MS |
3600000 (1 h) |
How often to check for stale workers |
Workflow Engine
The engine is a single-writer design: one instance drives the epoch machine for a given database. On startup it takes a Postgres session-level advisory lock and waits for any outgoing instance to release it, so a rolling deploy hands over without operator action. If the lock is still held when the timeout expires, the process refuses to start rather than running a second engine against the same database.
| Variable | Default | Description |
|---|---|---|
ENGINE_INSTANCE_LOCK_TIMEOUT_MS |
60000 (60 s) |
How long startup waits for another instance to release the single-instance lock before failing |
EPOCH_MAX_PARALLEL_RUNS |
unset (no cap) | Soft cap on how many workflows an epoch locks before it stops accepting new locks |
EPOCH_RETENTION_DAYS |
30 |
Age after which completed epochs are pruned; 0 or negative disables pruning |
EPOCH_CLEANUP_INTERVAL_MS |
3600000 (1 h) |
How often the epoch retention sweep runs |
Never run two engines against one database
Scale the engine vertically, not horizontally. A second instance is refused by the lock; if you deliberately need two engines, give each its own database.
EPOCH_MAX_PARALLEL_RUNS is a soft cap
It bounds sustained concurrency, not each individual epoch. The coordinator checks the cap between events against a count that may already be stale, while the packing itself happens inside concurrent handler transactions that do not consult it — so a burst of workflows reaching lock acquisition together can overshoot (three workflows packed with the cap set to one has been observed). Leave it unset unless you need to throttle sustained load; do not rely on it as a hard “at most N locked at once” guarantee.
PARKED_EXECUTION_PROBE_INTERVAL_MS (default 300000, 5 minutes; 0 disables) sets how
often the engine probes the CMS while executions are parked after repeated unlock failures;
once the CMS answers, the parked executions resume and release their locks without a
restart.
Completed epochs are pruned in batches on the EPOCH_CLEANUP_INTERVAL_MS schedule.
The latest epoch is never deleted, and neither is any epoch a non-terminal workflow
execution is still parked in — so pruning can never disturb work in flight, however
aggressive the retention setting.
Blackboard (Job Management)
| Variable | Default | Description |
|---|---|---|
BLACKBOARD_JOB_CHECK_INTERVAL |
60000 (1 min) |
How often the engine runs its job cleanup cycle (offline workers, stuck jobs, never-polled jobs). For one WORKER_OFFLINE_THRESHOLD_MS after start, the offline and stuck checks are skipped so workers can reconnect first |
BLACKBOARD_STUCK_JOB_TIMEOUT |
720000 (12 min) |
Jobs in POLLED state longer than this are auto-failed (default: 2× WORKER_OFFLINE_THRESHOLD_MS) |
BLACKBOARD_PENDING_JOB_TIMEOUT |
3600000 (60 min) |
Jobs in PENDING state that no worker ever polled within this deadline are auto-failed, so a workflow — and the epoch holding it — cannot wait forever for a worker that never comes (default: 5× BLACKBOARD_STUCK_JOB_TIMEOUT). Applies to acquire, execute and rollback jobs alike; a job claimed by a worker while the cleanup runs is left to that worker |
Rollback Data
Rollback data is the snapshot a reversible function block sends with its execute result for its own rollback().
| Variable | Default | Description |
|---|---|---|
ROLLBACK_DATA_MAX_BYTES |
5242880 (5 MiB) |
A byte size such as 5242880, 10mb or 10MiB. Largest snapshot the engine stores per attempt, measured as the UTF-8 byte length of its compact JSON. A larger one is dropped as too_large; the result itself is still stored. The Worker SDK applies its own 5 MiB cap, measured the same way |
ROLLBACK_DATA_RETENTION_DAYS |
30 |
How long a run that rests in FAILED_UNSAFE_ACK, with no rollback job active, keeps its snapshots. Runs that ended COMPLETED_ACK, FAILED_SAFE or FAILED_SAFE_ACK lose them at the next sweep |
ROLLBACK_DATA_SWEEP_INTERVAL_MS |
3600000 (1 h) |
How often the sweep that removes snapshots runs |
Unset or 0 keeps the default for these three. A negative value stops the engine at boot, as does
any value that is not an integer, or for ROLLBACK_DATA_MAX_BYTES not a byte size.
Example .env
# Production configuration
NODE_ENV=production
NEOPS_LOG_LEVEL=info
NEOPS_LOG_FORMAT=json
# Database
POSTGRES_HOST=postgres.internal
POSTGRES_PORT=5432
POSTGRES_USER=neops_engine
POSTGRES_PASSWORD=<secure-password>
# CMS
NEOPS_CMS_URL=http://neops-cms.internal:8000/graphql
NEOPS_CMS_TOKEN=<cms-auth-token>
