Troubleshooting
Indexed by what you actually see. Most of these are boot-order races that already have a wait guarding them — if you hit one, the usual cause is a hand-run command that skipped the wait.
Start here
make doctor # host preflight: RAM, subnets, mount, images
export COMPOSE_FILE=docker-compose.yml:docker-compose.worker.yml
export SCENARIO=wan-and-fabric # compose refuses to interpolate without it
docker compose ps # what is running, what exited
make local-lab-logs # worker + lab_bootstrap, followed
docker compose logs cms # CMS startup crashes land here
./containerlab inspect -t generated/wan-and-fabric/clab/neops-lab.clab.json
Without COMPOSE_FILE exported, plain docker compose commands do not see the worker or lab_bootstrap services — the base compose file does not declare them. The make targets set it themselves.
Pool overlaps with other one on this address space
Where: docker compose up, creating lab-net or the project default network.
Cause: another docker network overlaps 172.30.0.0/23. On a host with many compose projects docker’s auto-assigned /16 blocks run out and some other project’s default network landed on 172.30.0.0/16 before the lab’s networks existed. (The lab’s own networks all carry fixed subnets, so they never compete with each other.)
Fix: make doctor names the overlapping network. If it is an unused leftover (docker network inspect <name> shows no containers): docker network rm <name> — the owning project’s next up recreates it. If it is genuinely in use, stop that stack first.
Workflow with ID null not found
Where: the engine, when you trigger discovery.
Cause: the workflow definition has not been registered yet. docker compose up -d returns once containers are Started, not Completed, and lab_bootstrap is a one-shot container that POSTs the definitions and exits.
The guard: make local-lab-up runs docker compose wait lab_bootstrap, which blocks until that container exits and propagates its exit code — so a failed registration fails the target instead of silently producing this error later.
Fix: let make local-lab-up finish before running discovery. If the container exited non-zero, read its log:
200 unchanged is the idempotent re-run. A 409 means the version in the YAML already exists with different content — bump patchVersion/minorVersion (the log prints the engine’s suggestedVersion); a 404 on /workflow-definition/publish means an engine older than 0.42.2-beta.3.
Discovery stays running, the worker log says 413 Payload Too Large
Where: make local-lab-discover after the worker discovered every device (Generated 448 db update(s)) and then Failed to push job result … (413).
Cause: an engine image without the raised per-route body limits (older than 0.42.2-beta.3). Discovery returns a few hundred Interface rows in one job result; the engine rejects the push and re-dispatches the job.
Fix: refresh the engine — make local-env-up pulls with --policy always, and the default develop tag is rebuilt on every merge to the engine’s develop branch.
Function block … not found
Where: the execution’s terminal state, as failed_safe_ack — Function block with id fb.base.neops.io/global_discover_network:0.1.0 not found.
Two different causes, in order of likelihood:
-
You raced the worker. The worker registers its function blocks with the engine asynchronously after its container starts.
make local-lab-upandmake local-lab-discoverboth run./wait_ready fb.base.neops.io/global_discover_network:0.1.0first, which polls the engine’s per-function-block worker registry until an online worker exists. Run the make target rather thanrun_workflowdirectly. -
The image does not carry the block.
neops/fbships inside theneops-worker-sdkimage; it is not mounted from this repo. Check what you are running:⚠️ An old prerelease tag or an old local build may not carry it — see the next entry.
See Discovery and The /app/lab mount.
no online worker at wait_ready
Where: make local-lab-up burns its full wait_ready budget and gives up with
Cause: the worker image you pinned does not carry
global_discover_network, e.g. an old prerelease tag or an old local build. The
block ships inside the image rather than being bind-mounted from here, so the
worker starts, registers the blocks it has, and never announces this one;
wait_ready waits for it until the budget runs out.
Fix: check what the image actually holds, and fall back to the published
develop tag, a 0.3.x release, or a fresh build of an
SDK checkout that has it:
docker run --rm --entrypoint sh "$NEOPS_WORKER_SDK_IMAGE" -c 'ls neops/fb/base/global/discover_network.py'
make -C ../neops-worker-sdk-py build-docker # -> neops-worker-sdk:latest
echo 'NEOPS_WORKER_SDK_IMAGE=neops-worker-sdk:latest' >> .env
make local-lab-up
Re-pinning another 0.3.x the same way? Run the discover_network.py
check against it before make local-lab-up.
Worker exits ~2 min after start; discovery hangs after wait_ready
Affects engines older than v0.43.0-beta.6, which lack
engine #402. The default develop image has it.
Where: make local-lab-up succeeds and wait_ready reports an online
worker, but make local-lab-discover sits in running and never finishes. The
worker log shows Registration failed: (500) for one function block (e.g.
network_device_apply_configuration), retries for about two minutes, then the
container exits with code 1. The engine log shows value too long for type
character varying(255).
Cause: an engine older than v0.43.0-beta.6 stores a function block’s
description and help text in varchar(255) columns, and the 0.2.0-beta.* worker ships a block whose help
text is longer. That insert fails, the worker gives up and exits. The discovery
block registers before the failing one, so wait_ready passes — and the engine
keeps listing the dead worker as online.
Fix: run an engine at v0.43.0-beta.6 or later: drop an older
NEOPS_ENGINE_TAG, re-pull the engine and restart the stack. The worker pin
does not need to move.
bootstrap fails with 404 or 409 on the workflow definition
Where: docker compose wait lab_bootstrap returns non-zero and
docker compose logs lab_bootstrap shows a FAILED line.
409— the version already exists with different content. Published workflow definitions are immutable; editing ascenarios/_base/workflows/document in place and re-running is exactly what triggers this. BumpmajorVersion/minorVersion/patchVersionin the YAML, and update the matchingwf.lab.neops.io/simple_lab_discovery:<version>in theMakefile’sDISCOVER_WORKFLOWvariable, which bothlocal-lab-discoverandkind-lab-discoverread.422— the document is publishable but the engine computed a higher version floor than the document declares. Raise the version.- A
404handled silently — the engine predates068753a0and has no/workflow-definition/publish;register.pyfalls back to the legacyPOST /workflow-definition. Nothing to fix, but you are on an old engine image.
401 Unauthorized from the engine
Where: wait_ready, run_workflow or lab_bootstrap stops with 401 or 403 and a line naming NEOPS_ENGINE_TOKEN.
Cause: the engine is running NEOPS_AUTHZ_MODE=enforce or permissive, so every call to a gated route carries a bearer token. The make targets mint one per run; a hand-run script needs one in the environment. A 403 means the token is valid and its account lacks the route’s permission.
Fix:
export NEOPS_ENGINE_TOKEN=$(./lab_token)
./wait_ready fb.base.neops.io/global_discover_network:0.1.0
A token lasts 15 minutes, so a long session needs a fresh one. If the account is short of a permission, widen its role:
See Authorization.
Devices never become reachable
Where: wait_devices times out — timeout: 172.30.0.31:22, … not reachable within 240s.
Cause: devices boot slower than containerlab deploy returns. FRR is up in a second or two; Nokia SR Linux takes tens of seconds to bring up its management stack. Running discovery before then makes those hosts fail with a connection error and drop out of the 15-device count.
The guard: make local-lab-up deploys containerlab early — before the worker-readiness wait — precisely so SR Linux boot time overlaps with something useful, then polls TCP 22 on every mgmt IP.
The poll runs from inside the lab network, on purpose
Not from the host. On macOS and Docker Desktop the host has no route to
the lab-net bridge IPs (172.30.0.0/24), so a host-side poll hangs until
timeout even when every device is perfectly healthy. The worker is on
lab-net, so container-to-container reachability — the same path discovery
actually uses — works on every platform.
This is also why the command carries a lab/ prefix: the repo is mounted
at /app/lab inside the worker.
Fix: give it more time (./wait_devices --timeout 600 from inside the worker), or check the device itself:
Never replace a wait with a sleep
Each of the three waits exists because of a real, reproduced failure. A fixed sleep is both slower on a fast machine and still racy on a slow one.
The CMS crashes at startup
Symptom: the cms container exits or never becomes healthy; the log mentions token_service check_keys.
Cause: missing cms/jwt/{private,public}.pem. Newer neops-cms-free images require an RSA keypair for RS256 JWT issuance, and docker-compose.yml mounts cms/jwt read-only into the container.
Fix:
Idempotent, and a prerequisite of local-env-init / local-env-up — so this only bites when the stack is started by hand.
The web client renders an empty page
Symptom: http://localhost:8080/ returns HTTP 200 and a blank <app-root>. curl cannot see this failure — the HTML is fine; the Angular bootstrap is what fails.
Cause: scenarios/_base/cms/oidc-config.json must contain at least one well-formed OpenIdConfiguration entry with inline authWellknownEndpoints (no network discovery). The web client calls OidcSecurityService.checkAuth() in an APP_INITIALIZER; a null or [] config makes it fail before anything renders. The CMS serves the file through its appSettings.oidcConfig GraphQL resolver, so a missing OIDC_CONFIG_PATH produces the same result.
Fix: check the file is present and non-empty, and that OIDC_CONFIG_PATH=/etc/neops/oidc-config.json is still set on the cms service.
Verify UI behaviour with a browser, not curl
Drive a real browser (the Playwright MCP, or just open the page) when
checking lab or UI behaviour. This failure mode is invisible to curl.
The Global scope lost its columns
Symptom: the web client’s device table is back to default columns after a CMS restart.
Cause: the CMS image seeds a scope named Global on every startup with always_update_on_restart=True, and init_scopes runs on every manage.py invocation — rewriting the seeded scope’s columns, filters and drill-down. dashboard_configuration is not affected; init_scopes never writes it.
Fix:
The scope is Global, capital G
Postgres name uniqueness is case-sensitive, so a lowercase global creates
a silent duplicate scope rather than updating the seeded one.
docker compose pull fails on an image that exists
Symptom: pull access denied for neops-worker-sdk, repository does not exist.
Cause: you set NEOPS_*_IMAGE to a local tag with no registry host, so docker resolves it as docker.io/library/….
Fix: run the pull with --ignore-pull-failures, or skip it — pull_policy: missing means docker compose up -d uses the present local image without contacting a registry.
containerlab deploy fails to start or cannot reach docker
Cause: ./containerlab runs containerlab in a container through the docker socket; it needs a rootful docker daemon and, on Docker Desktop, the repo under a shared path. make doctor checks both.
With CLAB_NATIVE (a host binary) the classic failure is “This containerlab command requires root privileges or root via SUID to run” — usually after a containerlab upgrade: the new binary is installed without the SUID bit while your clab_admins membership survives, so id still looks right and only ls -l "$(command -v containerlab)" shows the missing s. make clab-suid restores it (idempotent).
Fix: make doctor, or make clab-suid for the native path; details in Prerequisites. Then confirm with the two-node probe:
./containerlab deploy -t clab/probe.clab.yml
./containerlab destroy -t clab/probe.clab.yml --cleanup
Duplicate interface rows after re-running discovery
Cause: devices are keyed by IP so they are skipped on a second run, but interfaces are always recorded.
Fix: run discovery against a fresh CMS.
make local-lab-down local-env-prune
make local-env-init && make local-lab-up && make local-lab-discover
Search is empty after the CMS image changed its index layout
Symptom: device or interface search returns nothing, or groups.* searches and acquires fail,
after pulling a newer CMS image onto an existing elasticsearch volume.
Cause: a newer neops-core keeps devices and interfaces in one network index and joins
groups and clients through ES|QL lookup indices (which is why the lab runs Elasticsearch 8.19).
elastic_index --create at CMS start creates the new indices empty and leaves the old ones alone,
so nothing is in them until the rows are written again. The old devices and interfaces indices
stay on the volume, unused, until it is pruned.
Fix: with no workflow executing (its searches and acquires fail while the indices are rebuilt), refill every index from the database:
or start from empty volumes with make local-env-prune (below), which also drops the old indices.
The reverse case: the first start of 8.19 upgrades the volume in place, and 8.9 cannot read
it afterwards. Checking out an older revision of this repo, which pins 8.9.2, makes
elasticsearch exit with status 1 on that volume. The index is derived data: run
make local-env-prune and bring the lab up again.
Nothing works and you want to start over
make local-lab-down local-env-prune
make local-env-init && make local-lab-up && make local-lab-discover
local-lab-down tolerates there being no deployed lab (the ./containerlab destroy line is prefixed with -, so make continues on failure). local-env-prune drops the Elasticsearch and Postgres volumes — leftover volume state across down/up cycles has previously made elastic_index --create fail and stale CMS data confuse re-inits.
