Production Patterns

Patterns for running neops workers reliably at scale.

Container Deployment

A minimal Dockerfile for a worker:

FROM python:3.12-slim

WORKDIR /app

COPY pyproject.toml uv.lock ./
RUN pip install uv && uv sync --no-dev

COPY . .

CMD ["uv", "run", "neops_worker"]

Key considerations:

Pin the Python version to match your development environment.
The worker imports configuration from the project’s config module — ensure it is available in the container image.
Use .env for configuration or inject environment variables via your orchestrator (Docker Compose, Kubernetes).
Exclude test code via .dockerignore rather than selective COPY.

Docker Compose

services:
  worker:
    build: .
    environment:
      URL_BLACKBOARD: http://workflow-engine:3030
      DIR_FUNCTION_BLOCKS: ./my_function_blocks
      WORKER_NAME: docker-worker-01
    restart: unless-stopped
    depends_on:
      - workflow-engine

Scaling Workers

Horizontal scaling

Run multiple worker instances. Each worker:

Registers independently with the workflow engine.
Polls for jobs it can execute (based on registered function blocks).
Processes one job at a time (max_workers=1).

The workflow engine distributes jobs across available workers. To handle more concurrent work, add more worker instances.

Specialization

Different workers can register different function block packages:

Worker	`DIR_FUNCTION_BLOCKS`	Handles
`config-worker`	`./config_blocks`	Config backup, push, compliance
`inventory-worker`	`./inventory_blocks`	Discovery, inventory collection
`monitoring-worker`	`./monitoring_blocks`	Health checks, SNMP polling

This lets you scale each concern independently and isolate failures.

Health Monitoring

Heartbeat

The worker sends heartbeats every HEARTBEAT_INTERVAL seconds, also while it drains a job after a shutdown request. If the workflow engine stops receiving heartbeats for its offline threshold (6 minutes by default), it marks the worker offline and fails the in-flight job as worker-offline; it never re-queues a claimed job.

Container healthcheck

The worker removes HEALTH_FILE (default /tmp/neops-worker-alive) when it starts, so a restarted container cannot report healthy from the previous process’s file. Once it polls, it touches the file at the start of every poll cycle, after every job, and on every heartbeat tick. neops_worker_health exits 0 when that file is younger than --max-age seconds (default: 3 × HEARTBEAT_INTERVAL) and 1 otherwise; python -m neops_worker_sdk.cli.health is the same check.

The image declares this check as its HEALTHCHECK, so Docker and Docker Compose report the worker’s health without any extra configuration:

HEALTHCHECK --interval=30s --timeout=10s --start-period=300s --retries=3 \
    CMD ["/app/.venv/bin/neops_worker_health"]

To change the timings in Compose, override it (uv run --no-sync neops_worker_health runs the same check):

services:
  worker:
    healthcheck:
      test: ["CMD", "/app/.venv/bin/neops_worker_health"]
      interval: 30s
      timeout: 10s
      start_period: 300s  # >= 2 x STARTUP_RETRY_TIMEOUT: registration can retry for 240 s
      retries: 3

The Helm chart (chart/neops-worker) runs the same command as a startupProbe (30 × 10 s, the same 300 s budget as start_period) and a livenessProbe. Both are on by default and are configured in values.yaml; set livenessProbe.enabled: false (or startupProbe.enabled: false) to drop one. There is no readiness probe, since the worker serves no traffic.

livenessProbe:
  enabled: true
  exec:
    command: ["/app/.venv/bin/neops_worker_health"]
  periodSeconds: 30
  timeoutSeconds: 10
  failureThreshold: 3

The probes need a writable, non-empty HEALTH_FILE. With readOnlyRootFilesystem: true, mount an emptyDir at /tmp (or point HEALTH_FILE at a writable volume). An empty HEALTH_FILE disables the file, and the probes then fail on every check.

Before it polls, the worker registers itself and then its function blocks, each retried for up to STARTUP_RETRY_TIMEOUT seconds (120 by default) while the engine is unreachable. Keep start_period at or above twice that, or a worker waiting out an engine restart is reported unhealthy before it ever polls.

The heartbeat tick refreshes the file whether or not the engine answers, so an engine outage does not make a busy worker look dead: a job that is still running, or a result the worker keeps re-sending for up to RESULT_PUSH_TIMEOUT, is not lost to a restart. It does so only for HEALTH_STALL_TIMEOUT seconds (default 720) after the poll loop last advanced. A job that hangs therefore turns the worker unhealthy once that time has passed plus --max-age. The file shows that the worker process is advancing; it does not check that the engine is reachable.

Who acts on unhealthy

Plain Docker Compose only reports the health status; it never restarts an unhealthy container (restart: reacts to the process exiting). Restarting on unhealthy needs an orchestrator that does it: Kubernetes (as a liveness probe), Docker Swarm, or a helper such as autoheal.

A worker the engine has expired (its heartbeat gets a 404) exits with code 1 and does not re-register on its own, so keep restart: unless-stopped (or on-failure) as well: that restart is what brings it back.

Log-based monitoring

The worker logs structured events at key lifecycle points:

Event	Level	Indicates
`NEOPS Worker starting...`	INFO	Worker initializing
`Found N function block(s)`	INFO	Discovery completed
`Registering worker with backend...`	INFO	Registration in progress
`Processing N job(s)`	INFO	Jobs received and processing
`Received SIGTERM. Shutting down...`	INFO	Graceful shutdown initiated
`Shutdown timeout of Ns elapsed with a job still running`	ERROR	The drain hit `SHUTDOWN_TIMEOUT`; the job’s outcome is unknown to the engine
`Result push for job … failed … retrying`	WARNING	The engine is unreachable; delivery is retried for up to `RESULT_PUSH_TIMEOUT`
`Worker registration failed`	ERROR	Cannot reach the workflow engine, it kept failing until `STARTUP_RETRY_TIMEOUT`, or it answered with a client error such as a 404 from a wrong `URL_BLACKBOARD`
`The engine refused this worker: …`	ERROR	The engine refused the worker’s client version (HTTP 426): the SDK does not match this engine. The line carries the engine’s reason, and the worker exits without retrying
`Worker expired! Backend rejected ping with 404.`	ERROR	Worker invalidated by backend

Forward these logs to your monitoring stack (ELK, Grafana Loki, Datadog) for alerting.

Error Recovery

Failure	Worker behavior	Engine behavior
Network partition	Misses heartbeats, reconnects on recovery; a finished job’s result is re-sent for up to `RESULT_PUSH_TIMEOUT`	Marks worker offline after 6 min and fails its in-flight jobs as worker-offline (compensating reversible blocks); accepts a late result until then
Function block exception	Reports failure result to blackboard, without `side_effects`, with any rollback data already saved	Assumes side effects; retries the step if the block is pure or idempotent. A step that fails for good makes the run FAILED_SAFE if it is pure, is rolled back if it is reversible, and otherwise makes the run FAILED_UNSAFE
Worker crash	Process exits	Fails the in-flight job as worker-offline after 6 min; never re-queues it
Workflow engine restart	Registration and result delivery are retried with backoff	Re-accepts worker registrations
Engine down at worker start	Retries registration for `STARTUP_RETRY_TIMEOUT`, then exits 1	—

Security Considerations

Credentials: Device credentials live in the CMS. Workers access them through WorkflowContext. Never store credentials in function block code or environment variables.
Network access: Workers need network access to both the workflow engine API and the managed devices. Use network segmentation to limit blast radius.
TLS: Configure URL_BLACKBOARD with https:// in production. The underlying HTTP client respects standard TLS settings.
Least privilege: Each worker only needs access to the devices its function blocks manage. Use separate workers for different network zones.

2026-02-26 17:18:082026-11-03 09:53:22