Neops Core 2.3.0
Released 2026-10-29
Core 2.3.0 covers everything merged since 2.2.0. It brings two larger pieces of work: search on Elasticsearch 8.18 with ES|QL, where index writes follow the database commit and repair themselves after an outage (https://github.com/zebbra/neops-core/pull/2184 and https://github.com/zebbra/neops-core/pull/2379), and centralized logging, where task logs go to a log store of your choice and carry the workflow run they belong to (https://github.com/zebbra/neops-core/pull/2409). Around them: locking by expansion for workflow acquires, safer host and rate-limit handling, and a series of fixes.
Take a database backup before you upgrade. The upgrade applies Core migrations up to 0204_merge_20261029_logging_develop, and the search indices are rebuilt.
At a glance
| Area | What changed | Why it matters |
|---|---|---|
| Elasticsearch version | Core needs Elasticsearch 8.18 or newer. | Cross-entity search runs ES|QL LOOKUP JOIN (ES|QL is the query language cross-entity search uses), which older versions cannot parse. |
| Deploy | The search indices are rebuilt automatically on upgrade: by ./neops up, and by Core itself in init.sh, which the Helm migrate hook runs (elastic_index --rebuild-if-stale). |
Search is unavailable or incomplete while the rebuild runs, and the Helm hook needs a longer --timeout. |
| Index sync | Index writes run after the database commits, and every write carries the row’s version. | A rolled-back change never reaches search, and an older write never replaces a newer one. |
| Restricted users | A row whose access change has not reached the index yet is hidden from restricted users. | Search never shows them what the database no longer grants. |
| Repair | Rows whose index write failed are re-indexed on workflow lock requests, by the periodic index sync task where Celery beat runs (Helm: on by default; neops-docker-compose: the cms-tasks profile), or with manage.py resync_unsynced_rows. |
Without lock requests and without beat, search stays behind after an outage until you run the command. |
| Aggregations | Aggregations over a related entity’s field (groups.name on devices, device.hostname on interfaces, interface.name on clients, …) return the buckets they returned before the indices were split. A field nothing has is an error. |
Dashboard tiles that went empty after the split show data again; a misspelled field says so instead of showing nothing. |
| Task log store | NEOPS_LOG_BACKEND selects where task logs go: db, victorialogs (new) or loki, with NEOPS_LOG_URL for a remote store. Unset, Core writes them to stdout. |
Before, unset meant the database. A deployment that sets nothing can no longer read its task logs back: Query.logging answers queryable=false. |
| Log settings renamed | NEOPS_LOGGER_* and EXECUTION_LOG_RETENTION_* are replaced by NEOPS_LOG_* names, and Core refuses to start while an old name is set. NEOPS_LOG_RETENTION takes a unit (30d, 12w). |
A deployment that still sets the old names does not start until they are renamed. |
| Workflow runs on log lines | Lock and unlock lines carry the workflow run (workflow_execution_id), stored on LogEntry and written as a field to VictoriaLogs and Loki. Reading them needs workflow-execution-log:read. |
Core’s lines of a 2.0 workflow run can be found by its id, and execution:read, which every role holds, no longer shows them. |
| Provider logging | Providers that support it have a per-task Log Level, NEOPS_TASK_LOG_LEVEL sets the default, and templates can call neops.log*, neops.fail and neops.skip. |
Bulky response bodies move to DEBUG, and a template can stop or skip an element with a reason instead of a syntax error. |
| Allowed hosts | The image no longer sets DJANGO_ALLOWED_HOSTS=*. ALLOWED_HOSTS comes from NEOPS_PUBLIC_URL, or from DJANGO_ALLOWED_HOSTS when set, and the API refuses to start with neither. |
Host header validation is on by default. |
| Health probes | /healthz (liveness) and /readyz (the database answers) are served before the host check; the chart probes them instead of /admin/login/. |
Probes keep working once real host names are configured. |
| Rate limits | A rate-limited login, refreshToken, logout or exchangeOidcCode answers HTTP 429 with Retry-After. |
Clients and proxies can tell a throttle from a failed login, and know when to retry. |
| Metrics | The neops_metrics plugin serves /metrics only with NEOPS_METRICS_TOKEN or NEOPS_METRICS_ALLOW_ANONYMOUS=true. |
Task names, fact keys and exception classes are no longer readable without a credential on the app port. |
| Locking | Workflow acquires can expand to related entities (EntityAcquireInput.resolveRelated), and lock creation, release and the global lock switch are race-safe. |
Concurrent workflows no longer corrupt or lose locks. |
Before upgrading, read “Before you upgrade”. After it, run the three checks in “After you upgrade, check”. To go back, follow “Going back to an earlier release”.
Upgrade notes
Before you upgrade
- Both: Elasticsearch must be 8.18 or newer; see “Elasticsearch 8.18 or newer”.
- Both: rename the log settings and choose a task log store; see “Task log settings”. Core does not start while an old name is set.
- Both: set
NEOPS_PUBLIC_URLorDJANGO_ALLOWED_HOSTS; see “Allowed hosts are required”. neops-docker-compose setsNEOPS_PUBLIC_URLfromNEOPS_CMS_URL. - neops-docker-compose:
./neops upbacks up only.envand its state file, not the database, so beforegit pull && ./neops up, take a backup with./neops backupand note the neops-docker-compose commit you are on (git rev-parse HEAD; the repository has no release tags yet). - Helm: set
--timeoutandmigrate.activeDeadlineSecondsas “Beforehelm upgrade” under “Where it runs” says. Note the Core image you upgrade to (quay.io/zebbra/neops-core:<the version>, or theimage.tagyou set,neops-core.image.tagunder the umbrella chart; once it runs,kubectl -n <namespace> get deploy <release>-neops-core -o jsonpath='{.spec.template.spec.containers[0].image}'prints it), since a rollback by--atomicneeds it, and the replica counts of the Core worker, beat and workflow engine Deployments (kubectl -n <namespace> get deploy), which going back restores. - Both: list the facts whose values disagree in type with the query in “Facts whose values disagree in type”, and fix them.
Task log settings
Core and the workflow engine read the same NEOPS_LOG_* names. Core refuses to start while one of the old names below is set, with an ImproperlyConfigured error that names its replacement:
| Old name | New name |
|---|---|
NEOPS_LOGGER_CLASS |
NEOPS_LOG_BACKEND (db, victorialogs or loki), or NEOPS_LOG_CLASS for an explicit class |
NEOPS_LOGGER_ENV |
NEOPS_LOG_ENV |
NEOPS_LOGGER_LOKI_BASE_URL, NEOPS_LOGGER_VLOGS_BASE_URL |
NEOPS_LOG_URL |
NEOPS_LOGGER_LOKI_USERNAME / _PASSWORD, NEOPS_LOGGER_VLOGS_USERNAME / _PASSWORD |
NEOPS_LOG_USERNAME / NEOPS_LOG_PASSWORD |
NEOPS_LOGGER_VLOGS_INSERT_BATCH_SIZE |
NEOPS_LOG_BATCH_MAX_RECORDS |
NEOPS_LOGGER_VLOGS_FLUSH_INTERVAL_SECONDS |
NEOPS_LOG_FLUSH_INTERVAL_MS (milliseconds) |
EXECUTION_LOG_RETENTION_ENABLED |
NEOPS_LOG_RETENTION_ENABLED |
EXECUTION_LOG_RETENTION_DAYS |
NEOPS_LOG_RETENTION, with a unit: 90d or 12w |
EXECUTION_LOG_RETENTION_INTERVAL_MINUTES |
NEOPS_LOG_RETENTION_INTERVAL_MINUTES |
EXECUTION_LOG_RETENTION_BATCH_SIZE |
NEOPS_LOG_RETENTION_BATCH_SIZE |
- Where task logs go. 2.2.0 wrote them to the database when nothing was set. 2.3.0 writes them to stdout when
NEOPS_LOG_BACKENDis unset, where nothing can read them back. SetNEOPS_LOG_BACKEND=dbto keep the 2.2.0 behaviour.victorialogsandlokialso needNEOPS_LOG_URL;NEOPS_LOG_TENANTsets the LokiX-Scope-OrgIDor the VictoriaLogs<account>[:<project>]. neops-docker-compose setsdbby default, andvictorialogswith itscompose.logs.yamloverlay. The core chart setsdbin itsvalues.yaml. - Retention.
NEOPS_LOG_RETENTIONdefaults to30d, whereEXECUTION_LOG_RETENTION_DAYSdefaulted to 90 days; the core chart ships90d. A bare number refuses to start, since it means days to Core and months to VictoriaLogs. Retention is still off unlessNEOPS_LOG_RETENTION_ENABLEDis true, and it deletes executions older than the horizon with their results and logs, not only the logs. - Global task log level.
NEOPS_TASK_LOG_LEVEL(DEBUGorINFO, defaultINFO) applies to tasks without their own Log Level.WARNINGrefuses to start, since providers that do not classify their lines log everything at INFO. - Old logs stay where they are. Switching the store does not move existing lines;
manage.py migrate_logscopies them between the database, Loki and VictoriaLogs.
Allowed hosts are required
The image used to set DJANGO_ALLOWED_HOSTS=*, which turned off Django’s host check and let a request’s Host header decide the OIDC login URLs Core hands out. That default is gone. ALLOWED_HOSTS is now DJANGO_ALLOWED_HOSTS when it is set (including *), otherwise the host of NEOPS_PUBLIC_URL. With neither, the API refuses to start (ImproperlyConfigured: No allowed Host is configured); management commands and Celery are not affected. When other services reach the API by another name, such as the in-cluster Service, list every name in DJANGO_ALLOWED_HOSTS, the public host included. /healthz and /readyz answer any host, so point probes there; the core chart does.
Metrics endpoint needs a credential
With the neops_metrics plugin enabled, /metrics is mounted only when NEOPS_METRICS_TOKEN is set (scrape with that bearer token) or NEOPS_METRICS_ALLOW_ANONYMOUS=true. Without either, the route is not mounted and a scraper that read it before gets nothing. The standalone metrics exporter still has no authentication; keep its port private.
Elasticsearch 8.18 or newer
sync_lookup_indices, which elastic_index --rebuild runs, fails on an older Elasticsearch. Upgrade Elasticsearch before Core.
The rebuild runs on upgrade, and search is down while it runs
The index definitions change in this release, so the upgrade rebuilds the indices. The rebuild deletes and recreates them, then fills them from the database. Until it finishes:
- search returns no or partial results;
- a search that finds no index fails with “Search is not available right now: an index it needs is missing or being rebuilt” instead of rebuilding it inside the request;
- workflow locks that resolve their entities through search are refused with
INDEX_SYNC_PENDING, which the engine retries. A rebuild that fails or is killed part way keeps them refused, with a message naming it, until a rebuild succeeds; each new rebuild takes over the one before it (it marks the earlier, stopped one finished and redoes its work).
If Elasticsearch refuses documents, the rebuild finishes anyway, exits with code 3, and lists them: fix them as the next paragraph describes, and list them again with manage.py index_sync_status --list rejected --limit 0. init.sh takes code 3 as a warning and goes on. --rebuild and --populate set refused documents aside the same way. The usual cause is a fact whose values disagree in type between rows: the rebuild gives a field the type of the first value it indexes, and refuses the rows whose value does not fit it. Each refused row is set aside: it is marked rejected, stays out of search, and is indexed again once it is saved with data Elasticsearch accepts. Until then it is also out of reach of the workflow locks that resolve entities through search: such a lock is granted without the row, and says nothing about it. With https://github.com/zebbra/neops-docker-compose/pull/25, ./neops doctor warns while any remain. The rebuild finishes all the same, and searching locks are accepted again.
The first value the rebuild indexes sets the field’s type, so a few odd rows can get many others refused, and saving a refused row cannot get it in while the field keeps the other type. The rebuild indexes in a fixed order, so the same database gets the same documents refused on every rebuild: index by index (clients, executions, groups, network), and in network all devices before any interface, so a key that devices and interfaces both carry takes its type from the devices. Within a model the rows go in its own order, ties by id: devices by hostname, interfaces by name, groups by name descending, clients and executions by id. With --parallel (ELASTICSEARCH_DSL_PARALLEL) or more than one shard (ELASTICSEARCH_DSL_INDEX_SHARDS), Elasticsearch can see the documents in another order. Names sort by the database’s collation, so a restore onto a Postgres with another collation can change which rows are refused. Core already has the means to keep a fact or check to one type: its schema in /admin/core/elasticsyncconfig/ (“Generate schemas”, or edit the key’s schema), which the index respects; values that do not fit it are indexed as null. For a type conflict, give the key one type there, or turn its sync off, or change the rows of the other type; “After upgrading: give every flagged fact and check key one type” below finds the keys. Then run elastic_index --rebuild --force, since --rebuild-if-stale finds the index current. A refusal that reads “Limit of total fields [N] has been exceeded” means the index holds as many fields as ELASTICSEARCH_DSL_INDEX_FIELDS_LIMIT allows. Restrict the synced keys with schemas, or turn sync off for noisy keys, then run elastic_index --rebuild --force; or raise that setting, which changes the schema hash, so the next --rebuild-if-stale rebuilds.
The report: before exiting, the rebuild prints “rebuilt; N documents refused and set aside” with the count per model, then the refusals grouped by the field Elasticsearch names (for example “facts.kind: 10,004 rows, type conflict: …”), each with its remedy, then each refused document with Elasticsearch’s reason, up to 50, and the command that lists them all. That command does not list executions yet; the rebuild’s output does, and so does SELECT id FROM core_execution WHERE elastic_sync_status = 'rejected'. The report goes to stdout, and the exit message, “DocumentsSetAside: rebuilt; N documents refused and set aside, listed above”, to stderr, so in container logs that line can land inside the report rather than after it.
Core decides on its own whether to rebuild. elastic_index --rebuild-if-stale rebuilds each index that this release or this Elasticsearch version did not build, and returns at once when there is none. A rebuild that finishes, --rebuild-if-stale or --rebuild, records in each index it rebuilt, under _meta.neops_build in the mapping (its build record): the schema hash (a fingerprint of the index definitions), the Elasticsearch version, and when the database applied Core’s migration 0203_index_stamp_epoch. An index without that record is stale: one that --create just made, one that a killed or failed rebuild left behind, and every index built before this release. So is every index recorded before a rollback that migrated the database back past 0203_index_stamp_epoch: the older Core wrote rows without keeping the indices up to date, and the upgrade after it applies the migration again, at a new time. The Helm rollback steps below undo the migrations, as compose’s do. A rollback that leaves them applied does not reset the record; where the older Core rebuilt groups or clients, their missing record still gets network rebuilt, see below. A rebuild that set documents aside records it too, so a later --rebuild-if-stale does not retry them.
- A later release that changes one index rebuilds only that index. A rebuilt
network,groupsorclientsindex brings the four lookup indices (four small indices that link devices, groups, interfaces and clients for cross-entity search) with it, and a changed lookup index bringsnetwork,groupsandclients. So does anetwork,groupsorclientsindex that is missing or has no build record, since whatever wrote it may have changed group memberships that the other two still hold; one whose record names only another schema hash is rebuilt without them. - The first run after an Elasticsearch upgrade rebuilds every index, even for a patch version; with Helm that is the next install or upgrade of Core. So does a change to
ELASTICSEARCH_DSL_INDEX_SHARDS,ELASTICSEARCH_DSL_INDEX_REPLICASorELASTICSEARCH_DSL_INDEX_FIELDS_LIMIT. - On Elasticsearch older than 8.18,
--createand--rebuild-if-stalefail before they replace or delete any index. - When two callers run it at once, the second waits for the first, then finds nothing to do.
elastic_index --create,--delete,--populateand--rebuildwait the same way, and so doessync_lookup_indices. A waiting command prints “Waiting for another rebuild to finish”; underinit.shthe log pauses after “Setup elastic indexes (waits while another rebuild runs)”. - On an empty database it takes a few seconds.
Where it runs:
- Helm. The core chart’s migrate Job, a
post-install,post-upgradehook, runsinit.shfrom the Core image it deploys.init.shnow runselastic_index --rebuild-if-staleafter--createand fails when it fails. When the rebuild only set documents aside,init.shlogs “Rebuild stale elastic indexes: Warning: Elasticsearch refused documents, which are set aside (see above)” and the hook succeeds. The new Core pods already serve requests while the hook runs, since their readiness check only asks the database; search is down from the start of the rebuild until the hook finishes. - Before
helm upgrade: Kubernetes stops the Job aftermigrate.activeDeadlineSeconds(default 900 seconds), and Helm waits for the hook only until--timeout(default5m). For this upgrade, and for the first upgrade after an Elasticsearch upgrade, set the deadline above the time a rebuild takes on your inventory and--timeouta few minutes above the deadline, since the deadline also covers the image pull, the waits and the migrations. With the neops umbrella chart, for example:helm upgrade ... --timeout 35m --set neops-core.migrate.activeDeadlineSeconds=1800. As a reference, 5,000 devices, 50,000 interfaces and 5,000 clients without facts rebuilt in 30 to 50 seconds on a laptop; facts and checks make every document larger. - During: in a second terminal, as soon as
kubectl -n <namespace> get job <release>-neops-core-migratelists the Job (neops-neops-core-migratefor a release namedneops), keep its log:kubectl -n <namespace> logs -f job/<release>-neops-core-migrate | tee migrate.log. Helm deletes the Job, and its log, once it succeeds, and that log is the only one that gives Elasticsearch’s reason for each document set aside. - After: list the rows the rebuild set aside:
kubectl -n <namespace> exec deploy/<release>-neops-core -- /docker-entrypoint.sh ./manage.py index_sync_status --list rejected --limit 0. It exits 0 whatever it lists. To gate a deployment, runindex_sync_statuswithout--list: it exits 1 while any row is rejected, and also while a rebuild is unfinished, unlocks are quarantined, rows have been pending or in error too long, or group writes or unlocks have waited too long for their index writes (--max-pending-agesets how long, except for rows in error). A later run that finds nothing to rebuild prints only how many rows are still set aside.elastic_index --populate --no-sync-lookups --models <app.model>prints the reasons again; without--no-sync-lookupsit also recreates all four lookup indices. - After, if
migrate.loghas no “Rebuild stale elastic indexes:” line: the hook ran aninit.shwithout this step. Every core chart onoci://quay.io/zebbra/charts(2.1.2 and later) runsinit.shfrom the image it deploys, so the pods most likely run a Core image from before this release; such an image has no--rebuild-if-stale. Check which image they run:kubectl -n <namespace> get deploy <release>-neops-core -o jsonpath='{.spec.template.spec.containers[0].image}'. By default it is the core chart’s appVersion; under the neops umbrella chart, that is theneops-coreversion neops-helm’scharts/neops/Chart.yamlpins.image.tag(neops-core.image.tagunder the umbrella chart) overrides it. If it is not this release’s image, set that tag to this release’s version and runhelm upgradeagain. If it is, run the rebuild yourself:kubectl -n <namespace> exec deploy/<release>-neops-core -- /docker-entrypoint.sh ./manage.py elastic_index --rebuild-if-stale. Without a migrate.log, check the image the same way, then run it: it returns at once when every index is current. Exit 0 means done. Exit 3 means it finished but set rows aside; list them as above. Any other exit means it failed: fix the cause and run it again. Search is down while it runs, and the command stops if your terminal session drops. - Helm gave up (“timed out waiting for the condition” in Helm 3, “context deadline exceeded” in Helm 4) while
kubectl -n <namespace> get job <release>-neops-core-migratestill shows the Job active: it keeps rebuilding. Do not runhelm upgradeagain yet: it deletes the running Job and the new one starts the rebuild over. Wait for it withkubectl -n <namespace> wait --for=condition=complete job/<release>-neops-core-migrate --timeout=30m(the deadline you set); the nexthelm upgradethen finds nothing to rebuild.--atomic(--rollback-on-failurein Helm 4) rolls Core back to the previous release while the Job goes on rebuilding the indices for the new one, so leave it off for this upgrade or give it a long enough--timeout. kubectl -n <namespace> get job <release>-neops-core-migrateshows the Job failed (Helm 3 says “DeadlineExceeded” or “BackoffLimitExceeded”, Helm 4 “status: Failed”): search stays empty or partial, and lock requests that search stay refused, until a rebuild succeeds. Kubernetes retries a failed rebuild from the start up tomigrate.backoffLimit(10) times; the same error twice in the log will not heal on its own. Fix the cause, then runhelm upgradeagain with a longer deadline and--timeout. From a running Core pod,kubectl -n <namespace> exec deploy/<release>-neops-core -- /docker-entrypoint.sh ./manage.py elastic_index --rebuild-if-staledoes the same, but stops if your terminal session drops.index_sync_status --list rejected --limit 0lists rows: they stay out of search until they are fixed as described above. The query in the section on facts whose values disagree in type finds such rows before an upgrade.- The log stops at “Waiting for another rebuild to finish” or after “Setup elastic indexes (waits while another rebuild runs)”: another rebuild holds the lock. Find it with
SELECT a.pid, a.client_addr, a.backend_start FROM pg_locks l JOIN pg_stat_activity a USING (pid) WHERE l.locktype = 'advisory' AND l.classid = 1852141424 AND l.objid = 1932355960 AND l.granted AND l.database = (SELECT oid FROM pg_database WHERE datname = current_database());in the Core database. A session whose pod is gone is ended withSELECT pg_terminate_backend(<pid>);. The Core image has nopsql; to run a query, put it inquery.sqland pass it to Core’s Django shell:kubectl -n <namespace> exec -i deploy/<release>-neops-core -- /docker-entrypoint.sh ./manage.py shell -c "import sys; from django.db import connection; cursor = connection.cursor(); cursor.execute(sys.stdin.read()); print(cursor.fetchall())" < query.sql. With neops-docker-compose,./neops compose -- exec -T postgres-cms psql -U neops -d neopsopens the Core database. - Going back to an earlier release: see “Going back to an earlier release” below.
- The names in these notes assume a release name without
neops-corein it and nofullnameOverride;kubectl -n <namespace> get job,deploy -l app.kubernetes.io/name=neops-corelists the real ones. - neops-docker-compose. With https://github.com/zebbra/neops-docker-compose/pull/27, every
./neops upand./neops installrunselastic_index --rebuild-if-stalein thecmscontainer once it is healthy; itscms-initservice does not runinit.sh. Exit 3 is a warning:uplogs that documents were set aside, the number of rows markedrejectedand the command that lists them, and records the start (the images it started, which the nextupcompares against). Any other non-zero exit failsup, which then does not record the start. For a Core without--rebuild-if-stale,uprunselastic_index --rebuild --forcewhenever the Core or the Elasticsearch image changed, and fails on any non-zero exit, 3 included. A compose release without that PR still decides fromelastic_index --schema-hash, fails on exit 3, and so rebuilds again on every./neops upuntil no row is refused. - Neither. Run the same steps as
init.shafter each upgrade:
cd backend
uv run python manage.py migrate
uv run python manage.py elastic_index --create
uv run python manage.py elastic_index --rebuild-if-stale
Exit code 3 from the last command means it finished but set documents aside; uv run python manage.py index_sync_status --list rejected --limit 0 lists them. A script under set -e or a configuration tool stops on 3 unless it accepts that code, as init.sh does.
A manual elastic_index --rebuild records the indices it rebuilt, also when it set documents aside, so the next --rebuild-if-stale leaves them alone. --rebuild --models records only the indices of those models, --rebuild --no-sync-lookups leaves the lookup indices unrecorded, and --populate records nothing.
This release keeps devices and interfaces in one index, network. The devices and interfaces indices of 2.2.0 and older stay behind after the upgrade and are no longer written. Delete them with ./neops compose -- exec -T cms python manage.py shell -c "from elasticsearch_dsl.connections import connections; connections.get_connection().indices.delete(index=['devices', 'interfaces'], ignore_unavailable=True)" (Helm: the same manage.py shell command through kubectl -n <namespace> exec deploy/<release>-neops-core -- /docker-entrypoint.sh ./manage.py). A rollback to such a release recreates them with the elastic_index --rebuild --force its steps run.
The rebuild is required, not only recommended. Documents written before this release carry Elasticsearch’s own version numbers, which are higher than the new row versions. A device, interface, client or group document moves its row past such a number on its next write, and so do the lookup rows that cross-entity search joins on; only the rows nobody writes stay stale until the rebuild recreates them.
Facts whose values disagree in type
Devices and interfaces now share one mapping in the network index. Elasticsearch refuses a device or interface whose fact or check does not fit the type an earlier device or interface gave the same field: a number against a word (mtu: 1500 on a device, mtu: "jumbo" on an interface), a date against a word ("2026-10-01T10:00:00Z" against "never", or against "2026-10-01 10:00:00", which Elasticsearch keeps as text), two date forms ("2026/10/01" against "2026-10-01"), or an object against a plain value. Two devices, or two interfaces, that disagree were refused before this release too. The rebuild sets the refused rows aside and finishes, as described above: they stay out of search, and out of every dashboard tile, until they are fixed.
An object against a plain value, true/false against a number or a date, two date forms, or a conflict in both directions fail every time. A word against a number, true/false or a date, and a date against a number, fail only when the word or the date is indexed after the other value. Devices are indexed before interfaces, so a key that holds a number on a device and a word or a date on an interface is refused on the interfaces; with --parallel or more than one shard the order, and so the rows set aside, can change from run to run. Retrying is no fix: fix every field the query below lists first.
Before upgrading (compose: before git pull), list the candidates. The query is read-only. It walks the facts and checks of every device and interface that is not deleted down to their values, inside objects and arrays, and lists each field whose values are of more than one kind: number, boolean, string, date, date yyyy/MM/dd (a date written 2026/10/01) or object. It lists more than Elasticsearch refuses: Elasticsearch accepts "1500" where it expects a number. It misses a few rare cases:
- Neops turns an object whose values are all objects, the first two with a key in common (the facts object itself included), into a list, with each outer key as
id, so those values reach Elasticsearch under other fields; - Elasticsearch splits a key that contains a dot (
"vlan.100") into nested fields; - it reads a few strings differently from Elasticsearch: a time zone given by name (
2026-10-01T10:00:00Europe/Zurich), a negative year, a day that does not exist (2026-02-30,2026/04/31), year 0 written0000/01/01, and a time after a date with no day (2026T10,2026-02T10:20), which Elasticsearch 8 reads as a date (2026T10as 1970-01-01 10:00) and the query, like Elasticsearch 9, counts as a string; - Elasticsearch stores a value named
ip,ip4,ip6,ipv4,ipv6oripaddress(in any case, with an optional leading or trailing_or-) as an address, and refuses one that is not an address (IPv4: "dhcp") on its own, as it did before this release. Neops drops such values first only in facts whose name containsipin lower case (ipv4,ipAddress); it keeps them in other facts (IP,IPv4) and in checks.
An empty result therefore makes a refusal unlikely, not impossible; the rebuild names any row it sets aside.
The query reads one row at a time. Its temporary files on the CMS database’s volume are freed row by row and stay at up to about ten times the largest row’s facts, more when those nest deeply, unless facts hold maps keyed by values rather than fixed names, such as BGP peers keyed by their address or routes keyed by prefix: on 5,000 devices with 60 peers each, keyed by each device’s own peer addresses, and 100,000 interfaces (150 MB of facts and checks) they reached 1.1 GB and it took 70 s; with the same peers on every device it wrote none and took 27 s. Its first line stops it once its temporary files reach 2 GB, since by default that volume also holds Elasticsearch’s data. If it stops with “temporary file size exceeds temp_file_limit”, raise the limit to less than the free space ./neops compose -- exec -T postgres-cms df -h /var/lib/postgresql/data shows, minus 5% of the volume’s size: Elasticsearch blocks writes to every index when its disk is 95% full. Setting the limit needs a database superuser, which compose’s neops user is.
./neops compose -- exec -T postgres-cms psql -v ON_ERROR_STOP=1 -U neops -d neops -f - <<'SQL'
SET temp_file_limit = '2GB';
WITH source(entity, id, field, value) AS (
SELECT 'device', id, 'facts', facts FROM core_device WHERE deleted_at IS NULL
UNION ALL SELECT 'device', id, 'checks', checks FROM core_device WHERE deleted_at IS NULL
UNION ALL SELECT 'interface', id, 'facts', facts FROM core_interface WHERE deleted_at IS NULL
UNION ALL SELECT 'interface', id, 'checks', checks FROM core_interface WHERE deleted_at IS NULL
), kinds AS (
SELECT s.entity, k.path, k.kind, count(*) AS rows, min(s.id) AS example
FROM source s CROSS JOIN LATERAL (
WITH RECURSIVE node(path, value) AS (
SELECT s.field, s.value
UNION ALL
SELECT n.path || coalesce('.' || c.key, ''), c.value
FROM node n CROSS JOIN LATERAL (
SELECT key, value FROM jsonb_each(CASE WHEN jsonb_typeof(n.value) = 'object' THEN n.value ELSE '{}' END)
UNION ALL
SELECT NULL, value FROM jsonb_array_elements(CASE WHEN jsonb_typeof(n.value) = 'array' THEN n.value ELSE '[]' END)
) AS c
)
SELECT DISTINCT path,
CASE
WHEN jsonb_typeof(value) <> 'string' THEN jsonb_typeof(value)
WHEN value #>> '{}' ~ '^[0-9]{4}(-(0[1-9]|1[0-2])(-(0[1-9]|[12][0-9]|3[01])T?)?)?$' THEN 'date'
WHEN value #>> '{}' ~ '^[0-9]{4}-(0[1-9]|1[0-2])-(0[1-9]|[12][0-9]|3[01])T([01][0-9]|2[0-3])(:[0-5][0-9](:[0-5][0-9]([.,][0-9]{1,9})?)?)?(Z|[+-]((0[0-9]|1[0-7])(:[0-5][0-9](:[0-5][0-9])?|[0-5][0-9]([0-5][0-9])?)?|18(:00(:00)?|00(00)?)?))?$' THEN 'date'
WHEN value #>> '{}' ~ '^[0-9]{4}/(0[1-9]|1[0-2])/(0[1-9]|[12][0-9]|3[01])( ([01][0-9]|2[0-3]):[0-5][0-9]:[0-5][0-9])?$' THEN 'date yyyy/MM/dd'
ELSE 'string'
END AS kind
FROM node WHERE jsonb_typeof(value) NOT IN ('array', 'null')
) AS k
GROUP BY s.entity, k.path, k.kind
)
SELECT path AS field, string_agg(format('%s on %ss (rows: %s, e.g. id %s)', kind, entity, rows, example), '; ' ORDER BY kind, entity) AS kinds
FROM kinds GROUP BY path HAVING count(DISTINCT kind) > 1
ORDER BY 1;
SQL
Each row names a field and, for each kind of its values on devices and on interfaces, how many rows hold it and the id of one of them. Make the values agree, or remove the odd ones, before the upgrade; a row set aside after it is indexed again once its values agree, or its key has one type as the next section describes, and the indices are rebuilt. Facts usually come from a workflow, so fix what writes them too, or its next run brings the conflict back. Without neops-docker-compose, run the query with psql -v ON_ERROR_STOP=1 against the CMS database; as a user who is not a superuser, psql stops at the first line with “permission denied to set parameter”, so remove that line and watch the free space instead. The same refusal hits a device or interface written after the upgrade; “Aggregations over related entities” below says what it does then.
After upgrading: give every flagged fact and check key one type
- When: the upgrade’s rebuild exited 3 or logged “rebuilt; N documents refused and set aside” (under init.sh: “Rebuild stale elastic indexes: Warning: Elasticsearch refused documents …”), or
manage.py index_sync_status --list rejected --limit 0lists rows. Running the analysis after every upgrade does no harm. The check exists only from this release on, so it cannot run before the upgrade. - Find the keys: run Generate schemas in
/admin/core/elasticsyncconfig/(a dry run is enough), or the command.--sample-size -1reads every row; the command’s default reads 100 per entity type and can miss the rows that conflict.
# neops-docker-compose
./neops compose -- exec -T cms python manage.py elastic_sync_schema --analyze --sample-size -1
# Helm
kubectl -n <namespace> exec deploy/<release>-neops-core -- /docker-entrypoint.sh ./manage.py elastic_sync_schema --analyze --sample-size -1
It lists under “Type conflicts” every fact and check key whose schemas let values of more than one Elasticsearch type into one field: one schema allowing both an integer and a string, for example, or a device and an interface schema typing the same key differently, since devices and interfaces share one index. It reads the schemas only, so an empty list is not a guarantee: a key no schema covers (none stored, and none generated in the run), a property its schema does not name, and facts whose dict of dicts is indexed as one list (NEOPS_ELASTIC_CONVERT_FACTS, on by default) can still mix types.
- Fix them: in /admin/core/elasticsyncconfig/, find the row by its fact key, entity type and data type. A flag labelled “(generated)” has no row yet, which is the case for every key of an install that synced without schemas: run Generate schemas with Create new, or the command with --create --sample-size -1 in place of --analyze --sample-size -1, then edit the row it created. Create new stores a schema built from today’s values for every key that has none, not only the flagged ones, so from then on a value of another type indexes as null instead of being refused. To cover only the flagged key, add its row by hand in the same admin page. Give the flagged path one type, for example "type": "string" in place of "type": ["integer", "string"], or keep one branch of an anyOf; for a device and interface disagreement, make both rows agree. A row whose value no longer fits then indexes the whole key, or a check’s whole metrics, as null. Or turn the row’s sync off, which stops indexing the key. Then rebuild. Like the upgrade’s rebuild, it deletes and recreates the indices, so search is incomplete until it finishes:
# neops-docker-compose
./neops compose -- exec -T cms python manage.py elastic_index --rebuild --force
# Helm
kubectl -n <namespace> exec deploy/<release>-neops-core -- /docker-entrypoint.sh ./manage.py elastic_index --rebuild --force
The rebuild logs a “Schema validation failed” warning for each key and entity type whose rows were indexed as null, at most one a minute, with the number of rows.
The fix worked when this rebuild exits 0, not 3, and manage.py index_sync_status --list rejected --limit 0 lists nothing. Rows still listed were refused for a reason the analysis cannot see; what it reads is described under “Find the keys”.
- Update saved queries: once a key is typed as a number, date, boolean or IP address, a query for a value of the old type fails instead of matching nothing. For example, facts.mtu:jumbo on an mtu now typed integer answers “Query failed: For input string”, and a dashboard tile saved with it shows that error. Change or remove those queries.
Locations’ facts and checks are now filtered by the device_group schemas when they are saved, as the rebuild already did, so each location re-syncs once.
Stored aggregations: id fields renamed, fields that do not aggregate reported
Migration 0202_scope_aggregation_id_fields stores device.id (interface aggregations), interface.id and location.id (client aggregations) as device_id, interface_id and location_id, the fields that hold the ids now. Both spellings keep working in queries and in dashboard cards.
An aggregation whose field no entity has (a typo, or a path that never aggregated) now fails with an error that names the accepted paths, instead of showing no buckets. In the web client the card stays in its loading state and a short “Error loading” notice appears. After upgrading, list them:
./neops compose -- exec -T cms python manage.py check_aggregation_fields # neops-docker-compose
uv run python manage.py check_aggregation_fields # source checkout, in backend/
It reads the scope aggregations and the aggregation cards of every scope’s dashboards (the overview dashboard and the four entity detail dashboards), and changes nothing. For each one it lists, it says where it is, why its field does not aggregate, which paths the entity accepts, and how to fix it. The web client edits neither, so:
- a dashboard card: send the scope’s corrected JSON with the
scopesUpsertmutation, or edit it in the CMS admin site (<NEOPS_CMS_URL>/admin/, Scopes, the scope, the JSON field and path the check prints), or remove the card. The admin site refuses to save a scope whose overview dashboard (dashboard_configuration) is still empty ({}), so use the mutation for those; - a scope aggregation: edit it in the CMS admin site (Scope aggregations) or with the
scopeAggregationUpsertmutation, or delete it.
It exits 0 when every field aggregates, 2 when it lists some, and 1 when it cannot read the database, or part of it; it still lists what it could read. With --json the last line of its output is the result as JSON: {"unsupported": [...]}, plus "error" when it could not read everything.
manage.py migrate prints the same findings as warning core.W001 before it migrates; under ./neops up they appear in ./neops compose logs cms-init. When upgrading from a release before 2.1, the database has no dashboard columns yet at that point: the scope aggregations are still listed, and warning core.W002 says the cards could not be read, so run the command after the upgrade for them. A new installation, which has no tables yet, prints neither.
After you upgrade, check
Run each with ./neops compose -- exec -T cms python manage.py <command> (Helm: kubectl -n <namespace> exec deploy/<release>-neops-core -- /docker-entrypoint.sh ./manage.py <command>):
index_sync_status --list rejected --limit 0lists nothing; rows it lists were set aside, see “After upgrading: give every flagged fact and check key one type”.check_aggregation_fieldsexits 0; exit 2 lists aggregations to fix, see “Stored aggregations: id fields renamed, fields that do not aggregate reported”.elastic_sync_schema --analyze --sample-size -1lists no “Type conflicts”.
Going back to an earlier release
Going back undoes the logging migration too: migrate core 0188_refreshtoken_auth_provider removes 0192_logentry_workflow_execution_id, so log lines lose the workflow run they were linked to. Restoring the backup you took before the upgrade is the only way to get the database exactly as it was; everything written since is then lost. The earlier release reads the old log setting names: put NEOPS_LOGGER_* and EXECUTION_LOG_RETENTION_* back where you set NEOPS_LOG_* yourself (Helm values, an .env of your own), or it falls back to its defaults (database logs, retention off), and remove them again before the next upgrade. Keep DJANGO_ALLOWED_HOSTS or NEOPS_PUBLIC_URL; the earlier image ignores neither.
First, on both, while this release still runs
Make every device and interface fit the types its facts and checks are indexed with. Otherwise the earlier release’s rebuild (with neops-docker-compose, the one ./neops up --allow-downgrade runs) fails with “BulkIndexError:
# neops-docker-compose
./neops compose -- exec -T cms python manage.py elastic_sync_schema --analyze --sample-size -1 > type-conflicts-before-rollback.txt
# Helm
kubectl -n <namespace> exec deploy/<release>-neops-core -- /docker-entrypoint.sh ./manage.py elastic_sync_schema --analyze --sample-size -1 > type-conflicts-before-rollback.txt
Then give each flagged key one type in /admin/core/elasticsyncconfig/, or turn its sync off; or fix the rows. Then export the stored schemas, with the changes you just made:
# neops-docker-compose
./neops compose -- exec -T cms python manage.py dumpdata core.ElasticSyncConfig --indent 2 --output /tmp/elastic-sync-config.json
./neops compose -- cp cms:/tmp/elastic-sync-config.json ./elastic-sync-config.json
# Helm (not rehearsed)
POD=$(kubectl -n <namespace> get pod -l app.kubernetes.io/name=neops-core,app.kubernetes.io/instance=<release>,app.kubernetes.io/component=app -o name | head -1)
kubectl -n <namespace> exec "$POD" -- /docker-entrypoint.sh ./manage.py dumpdata core.ElasticSyncConfig --indent 2 --output /tmp/elastic-sync-config.json
kubectl -n <namespace> cp "${POD#pod/}":/tmp/elastic-sync-config.json ./elastic-sync-config.json
--output is needed: redirecting stdout puts Core’s start-up output into the file. A restored backup holds the schemas from before the upgrade, and 2.2.0 cannot list the conflicts, so the way back from the backup loads this export (step 3 of “neops-docker-compose, from the backup”). On the ways that keep the data (“neops-docker-compose, keeping the data written since”, and “Planned.” under “Helm”), the rows survive migrate core 0188_refreshtoken_auth_provider, so the export is a safety copy there. The Helm commands take one pod, so that the export and the copy run in the same one. index_sync_status --list rejected --limit 0 lists the rows this release set aside. The earlier release filters facts and checks through the same stored schemas when it indexes them (in 2.2.0, ElasticSyncConfig), so its rebuild then succeeds. On Helm, the way back is the one under “Helm” below; going back by restoring a database backup is not covered here.
neops-docker-compose, keeping the data written since
To keep the data written since, undo this release’s migrations while its image is still in place, with nothing writing, then start the earlier release, which rebuilds its search indices, and delete the ones only this release uses. If the commit you noted pins an older Elasticsearch, empty its data directory first (the block’s find line does it only then), as in step 2 of “neops-docker-compose, from the backup” below; set NEOPS_DATA_DIR in the shell as there.
./neops compose -- stop cms cms-worker cms-beat engine
./neops compose -- run --rm --no-deps cms python manage.py migrate cron 0015_add_execution_log_retention_periodic_task_20260201_0145
./neops compose -- run --rm --no-deps cms python manage.py migrate core 0188_refreshtoken_auth_provider # the last core migration of 2.2.0
./neops compose -- stop elasticsearch
current=$(git show HEAD:compose.yaml | grep -o 'elasticsearch/elasticsearch:[0-9.]*' | cut -d: -f2)
pinned=$(git show <the commit you noted>:compose.yaml | grep -o 'elasticsearch/elasticsearch:[0-9.]*' | cut -d: -f2)
echo "leaving Elasticsearch ${current:-unknown}, going back to ${pinned:-unknown}; an older or unknown pin empties the data"
[ -n "$current" ] && printf '%s\n' "$current" "$pinned" | sort -V -C || sudo find "${NEOPS_DATA_DIR:-./data}/elasticsearch" -mindepth 1 -delete # only when that commit pins an older Elasticsearch
git checkout <the commit you noted>
./neops up --allow-downgrade
./neops compose -- exec -T cms python manage.py shell -c "from elasticsearch_dsl.connections import connections; connections.get_connection().indices.delete(index=['network', 'device_group_edges_by_device_lookup', 'device_group_edges_by_group_lookup', 'client_group_edges_lookup', 'client_interface_edges_lookup'], ignore_unavailable=True)"
The last command deletes the network index and the four lookup indices: the earlier release neither uses nor rebuilds them, so they would keep this release’s documents, which go stale, and the next upgrade must rebuild them. After going back to 2.2.1-beta.2 to 2.2.1-beta.5, which already use them, skip it. After going back to 2.2.1-beta.7, run it: that beta does not use them. Do not run the earlier release’s elastic_index --rebuild on top of the one ./neops up ran: 2.2.0’s stops at the first document Elasticsearch refuses and leaves search empty. The cron migration removes the “Unlock Index Sync” periodic task this release added, which 2.2.0 has no task for; skip it on an image without the cron plugin (it answers “No installed app with label ‘cron’”). For another release, python manage.py showmigrations core cron in its image names the last migrations. migrate core 0188_refreshtoken_auth_provider leaves four migrations 2.2.0 does not have applied, since 0188 does not depend on them: 0175_add_elastic_relation_hash_fields and the merges 0181_merge_20260728_0703, 0183_merge_20260810_2137 and 0185_merge_20260917_esql. The columns 0175 added stay on the device, interface, group, client and execution tables: elastic_relation_hash (not null, empty by default) and elastic_relation_synced_at (nullable). 2.2.0 does not read them and inserts rows without them, and the next upgrade finds the four applied; no migration target undoes only them. Undoing 0202_scope_aggregation_id_fields stores device.id, interface.id and location.id again; a dashboard card saved with device_id, interface_id or location_id after the upgrade keeps that field, which 2.2.0 does not aggregate.
neops-docker-compose, from the backup
Going back with the backup from “Before you upgrade” takes four steps. Their commands read NEOPS_DATA_DIR from the shell, so if .env sets it, export the same value first:
-
Stop the stack, note the Elasticsearch this release pins, and check out the commit you noted, in the shell you use for the next steps:
-
If that commit pins an older Elasticsearch than the one you leave, empty its data directory and keep the directory itself, which Elasticsearch (uid 1000) owns: Elasticsearch cannot open data a newer version wrote, and the rebuild in step 4 refills it. Compose commits before 2026-10-27 pin 8.9.2.
pinned=$(git show HEAD:compose.yaml | grep -o 'elasticsearch/elasticsearch:[0-9.]*' | cut -d: -f2) echo "leaving Elasticsearch ${current:-unknown}, going back to ${pinned:-unknown}; an older or unknown pin empties the data" [ -n "$current" ] && printf '%s\n' "$current" "$pinned" | sort -V -C || sudo find "${NEOPS_DATA_DIR:-./data}/elasticsearch" -mindepth 1 -delete # only when it pins an older Elasticsearch -
Move the database directories aside, so the restore loads into empty databases: it drops only the tables the backup holds, and the tables this release added would make the next upgrade’s migration fail.
sudo mv "${NEOPS_DATA_DIR:-./data}/cms/postgres" "${NEOPS_DATA_DIR:-./data}/cms/postgres.after-upgrade" sudo mv "${NEOPS_DATA_DIR:-./data}/engine/postgres" "${NEOPS_DATA_DIR:-./data}/engine/postgres.after-upgrade" # only if that commit pins another engineThen run steps 1 and 2 of “Restore” in neops-docker-compose’s
docs/30-operations.md(put.envback, start the databases,pg_restore) and stop there: its step 3 deletes the search indices and then runs./neops upwithout--allow-downgrade; here step 4’sup --allow-downgraderebuilds the indices instead. That step 2 restores both dumps,engine.dumptoo, so the workflow engine goes back to the same point as Core: the workflow data written since the upgrade is lost too.Then, while only
postgres-cmsandpostgres-enginerun, load the sync schemas you exported under “First, on both, while this release still runs”:./neops compose -- run --rm --no-deps -T cms python manage.py loaddata --format=json - < elastic-sync-config.jsonIt reads the file on stdin, since a
run --rmcontainer cannot receive acp.loaddatakeeps the dump’s ids, so the restored table must not already hold rows with the same fact key, entity type and data type under other ids (2.2.0’sunique_together); a 2.2.0 install that never used Generate schemas has none. If it has, the load fails withdjango.db.utils.IntegrityError: Problem installing fixture '-': Could not load core.ElasticSyncConfig(pk=<id>): duplicate key value violates unique constraint "core_elasticsyncconfig_fact_key_entity_type_dat_931cc391_uniq"and loads nothing, sinceloaddataruns in one transaction. Then delete every stored schema row, which the export file replaces, and load again, still with only the databases running: -
Run
./neops up --allow-downgrade, which rebuilds the earlier release’s search indices since the Core image changed, then delete the ones only this release uses with the last command of “neops-docker-compose, keeping the data written since” above. Ifupstill fails with “document(s) failed to index”, fix the key in 2.2.0’s/admin/core/elasticsyncconfig/, using thetype-conflicts-before-rollback.txtyou made under “First, on both, while this release still runs”, and run./neops up --allow-downgradeagain. It worked whenupexits 0;./neops doctoralone is no proof, since it passed with an empty groups index after a failedup.
Switch back to your branch before the next git pull.
Helm
Planned. helm rollback does not undo this release’s migrations, and the earlier release cannot run on them: it fails to create devices, interfaces, clients, groups and executions, since their new columns have no database default, and it would read the unlock rows of 0190_composite_lock_index_sync as locks. So undo them first, after “First, on both, while this release still runs” above.
- Stop what writes: note the replica counts first (
kubectl -n <namespace> get deploy), then scale the Core worker and beat Deployments and the workflow engine’s Deployment to 0, for examplekubectl -n <namespace> scale deploy/<release>-neops-core-worker deploy/<release>-neops-core-beat --replicas=0(beat exists only withbeat.enabled). -
Undo the migrations. Until step 3 replaces them, this release’s own pods fail most requests, so run step 3 right after.
kubectl -n <namespace> exec deploy/<release>-neops-core -- /docker-entrypoint.sh ./manage.py migrate cron 0015_add_execution_log_retention_periodic_task_20260201_0145 kubectl -n <namespace> exec deploy/<release>-neops-core -- /docker-entrypoint.sh ./manage.py migrate core 0188_refreshtoken_auth_providerThese are the last migrations of 2.2.0; for another release,
./manage.py showmigrations core cronin its image names them. -
helm -n <namespace> rollback <release> <revision>(helm -n <namespace> history <release>lists the revisions). Then check withkubectl -n <namespace> get deploythat the worker, beat and engine are back to the counts you noted; scale them back if not. -
Rebuild the earlier release’s indices and delete the ones only this release uses, with these three commands. Done when the rebuild exits 0 and a device search returns devices. After a rollback to 2.2.1-beta.2 to 2.2.1-beta.5, which already use those indices, run only the first two. After a rollback to 2.2.1-beta.7, which does not, run all three.
kubectl -n <namespace> delete job <release>-neops-core-migrate --cascade=foreground --ignore-not-found kubectl -n <namespace> exec deploy/<release>-neops-core -- /docker-entrypoint.sh ./manage.py elastic_index --rebuild --force kubectl -n <namespace> exec deploy/<release>-neops-core -- /docker-entrypoint.sh ./manage.py shell -c "from elasticsearch_dsl.connections import connections; connections.get_connection().indices.delete(index=['network', 'device_group_edges_by_device_lookup', 'device_group_edges_by_group_lookup', 'client_group_edges_lookup', 'client_interface_edges_lookup'], ignore_unavailable=True)"
The next upgrade applies the migrations again and finds every index stale, so it rebuilds them all; the index delete frees the space of indices nothing reads any more.
By --atomic. A rollback by --atomic, or any helm rollback without step 2, leaves this release’s migrations applied, and the older image cannot undo migrations it does not know. Until step 2 has run, the older release cannot create entities. First stop the migrate Job with the first command of step 4, since it keeps rebuilding and retrying for the release you left, and stop the writers as in step 1. List the rows this release set aside while their status column still exists, since the older Core has no index_sync_status, with this query in the Core database (the Helm bullet under “Where it runs” says how to run one): SELECT 'device', id FROM core_device WHERE elastic_sync_status = 'rejected' UNION ALL SELECT 'interface', id FROM core_interface WHERE elastic_sync_status = 'rejected' UNION ALL SELECT 'group', id FROM core_devicegroup WHERE elastic_sync_status = 'rejected' UNION ALL SELECT 'client', id FROM core_client WHERE elastic_sync_status = 'rejected' UNION ALL SELECT 'execution', id FROM core_execution WHERE elastic_sync_status = 'rejected'; Then run step 2 from a copy of a Core pod that runs this release’s image, the one you noted (unnoted, it is quay.io/zebbra/neops-core:<the version you upgraded to>, or the image.tag you set, neops-core.image.tag under the umbrella chart). The copy keeps the pod’s environment and its command, which only starts the API, and drops its labels, so the Service sends it no requests:
POD=$(kubectl -n <namespace> get pod -l app.kubernetes.io/name=neops-core,app.kubernetes.io/instance=<release>,app.kubernetes.io/component=app -o name | head -1)
kubectl -n <namespace> debug "$POD" --copy-to=neops-core-migrate-back --set-image='neops-core=<this release's Core image>'
kubectl -n <namespace> wait --for=condition=Ready pod/neops-core-migrate-back --timeout=5m
kubectl -n <namespace> exec neops-core-migrate-back -- /docker-entrypoint.sh ./manage.py migrate cron 0015_add_execution_log_retention_periodic_task_20260201_0145
kubectl -n <namespace> exec neops-core-migrate-back -- /docker-entrypoint.sh ./manage.py migrate core 0188_refreshtoken_auth_provider
kubectl -n <namespace> delete pod neops-core-migrate-back
Then run the rest of step 4, and bring the writers back: kubectl -n <namespace> scale deploy/<release>-neops-core-worker deploy/<release>-neops-core-beat deploy/<engine deployment> --replicas=<the counts before step 1> (one command per count, if they differ). Or run helm -n <namespace> rollback <release> <the revision --atomic restored>, then check the counts as in step 3. A rollback after which neither step 2 nor the index delete ran cannot be detected: the next upgrade trusts the indices the older Core wrote under, and can show a restricted user a device whose access was revoked meanwhile.
Centralized logging
https://github.com/zebbra/neops-core/pull/2409 brings Core’s half of the centralized logging design; the workflow engine, worker SDK and neops-docker-compose follow it with the same settings and record keys.
- Stores.
NEOPS_LOG_BACKENDpicks the writer:db(DbLogger),victorialogs(the newVictoriaLogsLogger, https://github.com/zebbra/neops-core/pull/2355) orloki.NEOPS_LOG_CLASSnames any class explicitly, such asneops.core.log.loggers.optimized_db.OptimizedDbLogger. Loki is pushed through Core’s own emitter with structured metadata;python-logging-lokiis gone. VictoriaLogs splits a message above about 200 KB into[part i/n]lines, since VictoriaLogs silently drops lines above 256 KiB. - Record keys. Lines written to VictoriaLogs and Loki carry the canonical record keys shared with the workflow engine, including the workflow run (
workflow_execution_id) and the keys for function block identifiers that the engine and worker fill in. Core itself writes the workflow run on the lines of its lock mutations: the lock reference is bound asworkflow_execution_id, and the unlock audit lines keep it.LogEntrystores it in a new column (migration0192_logentry_workflow_execution_id). - Log context. Every task log line, and every console line of the API, the Celery worker and beat, carries the ambient log context and a per-process
seq. - Reading.
Query.loggingneedsexecution:reador the newworkflow-execution-log:read, which the operator and admin profiles grant. Lines of a workflow run are returned only withworkflow-execution-log:read. Itsqueryargument is now an escaped, case-insensitive substring filter on every store instead of a raw store query,newestandoldestare no longer swapped, and a store that cannot be queried answersqueryable=false. - Lines survive rollbacks.
OptimizedDbLoggerbuffers writes and flushes them from a background thread in autocommit, so lines logged inside a transaction that rolls back are kept. A line whose linked row is not committed yet is retried for 30 seconds and then written without that link, never dropped.DbLoggernow also keeps lines that carry no execution, such as the attribution line of each change applied under a lock (https://github.com/zebbra/neops-core/pull/2397), and retention purges them. - Unlock audit lines are written after the unlock commits, so they are no longer atomic with it.
- Providers.
generic-rest-factsv2 and v3 andgeneric-netbox-jinja-factshave a per-task Log Level (DEBUG, INFO, WARNING; https://github.com/zebbra/neops-core/pull/2282). INFO logs one line per request, DEBUG adds bodies and facts, capped at 200 KB, and failed requests are logged at WARNING with their body. Saved tasks keep working:disable_logmaps to WARNING andlog_long_returnsto DEBUG. Lines below a task’s level go to the container log instead of disappearing (DEBUG lines are dropped), andNEOPS_TASK_LOG_LEVELsets the default (https://github.com/zebbra/neops-core/pull/2286). - Templates. Provider templates can write to the task log with
neops.log(message, level)andneops.log_debug/info/warning/error(https://github.com/zebbra/neops-core/pull/2283), and stop on purpose withneops.fail(reason)or skip an element withneops.skip(reason)(https://github.com/zebbra/neops-core/pull/2285). - Metrics. The enterprise
neops_metricsplugin (https://github.com/zebbra/neops-core/pull/2213) exports task, log size and retention metrics, and optionally database and table sizes;/metricsneeds a credential, see “Metrics endpoint needs a credential”. - Performance. Execution results and log entries are written in batches, retention raw-deletes the execution cascade, and Nornir hosts load without a platform query per device (https://github.com/zebbra/neops-core/pull/2367).
Other changes
Locking and workflows
- Locking by expansion. A workflow acquire can walk from the step’s entities to related ones (
EntityAcquireInput.resolveRelatedwith an optionaltype;queryContainingJmesExprbecomes optional). The GraphQL change is additive (https://github.com/zebbra/neops-core/pull/2332, https://github.com/zebbra/neops-core/pull/2336). - Race-safe locks. One locked global lock at a time (migration
0189_unique_locked_global_lock), a global lock mode switch serialised against fine-grained acquisitions, concurrent identical lock requests that get the existing lock, a lock whose resources vanished that fails instead of half-locking, interfaces re-readFOR UPDATEduring lock creation, lock ids stored as integers, and an abort that no longer overwrites a concurrently completed execution (https://github.com/zebbra/neops-core/pull/2320 to https://github.com/zebbra/neops-core/pull/2336, and the follow-ups https://github.com/zebbra/neops-core/pull/2369 to https://github.com/zebbra/neops-core/pull/2376). Every lock refusal carries aLockErrorCode, and an invaliddb_updatesvalue is reported as a lock error naming the field (https://github.com/zebbra/neops-core/pull/2391). - Remediation resolve permission. The new
workflow-execution-resolveelement, granted to the admin profile, lets the workflow engine gate remediation resolve separately from rollback (https://github.com/zebbra/neops-core/pull/2398). - Failed input fails the execution. When
executeNeopsTaskrejects the task input, the execution is set tofailedinstead of stayingpending(https://github.com/zebbra/neops-core/pull/2390). - Scopes return every entity they match, not the first 10 (https://github.com/zebbra/neops-core/pull/2378), and topology scans no longer fail past the Elasticsearch result window (https://github.com/zebbra/neops-core/pull/2385).
Authentication and API
- API keys report the new key’s id;
list_api_keysandrevoke_api_keyare new (https://github.com/zebbra/neops-core/pull/2351, https://github.com/zebbra/neops-core/pull/2366). - Access tokens carry a
principal_typeclaim,userfor every token Core issues (https://github.com/zebbra/neops-core/pull/2394). - SSO. The social account is linked at the first email match and keeps its roles (https://github.com/zebbra/neops-core/pull/2365); a failed OIDC discovery redirects with
provider_error(https://github.com/zebbra/neops-core/pull/2352); SSO logout returns to the frontend named by?client(https://github.com/zebbra/neops-core/pull/2350); and the OIDC provider lookup is skipped where allauth is not installed (https://github.com/zebbra/neops-core/pull/2380). - Rate limits answer HTTP 429 with
Retry-Afterandextensions.codeRATE_LIMITED, or 503RATE_LIMIT_UNAVAILABLEwhen the limiter’s cache fails (https://github.com/zebbra/neops-core/pull/2403). Behind a proxy, setRATELIMIT_IP_META_KEY, or every user shares one bucket. - CORS.
CORS_ALLOWED_ORIGINSis honoured, andCorsMiddlewareruns first so short-circuited responses carry CORS headers (https://github.com/zebbra/neops-core/pull/2343, https://github.com/zebbra/neops-core/pull/2347). - Smaller fixes. The bearer token is authenticated once per request (https://github.com/zebbra/neops-core/pull/2349), ordering by password is refused (https://github.com/zebbra/neops-core/pull/2348), unknown paths answer 404 instead of 500 now that the legacy SPA catch-all is gone (https://github.com/zebbra/neops-core/pull/2342, https://github.com/zebbra/neops-core/pull/2357), Core warns at start-up when stored grants have no permissions plugin to read them (https://github.com/zebbra/neops-core/pull/2353), and the BitField permission labels are recorded by migrations (https://github.com/zebbra/neops-core/pull/2299).
Deployment
init.shjudgeselastic_index --createby its own exit status (https://github.com/zebbra/neops-core/pull/2341), andELASTICSEARCH_SSL_VERIFY_DISABLEis parsed as a boolean (https://github.com/zebbra/neops-core/pull/2356).- The
enterprise-bullseyeimage flavor is no longer built (https://github.com/zebbra/neops-core/pull/2279).
Inventory and providers
- The device inventory provider falls back to Django credentials (https://github.com/zebbra/neops-core/pull/2209), the importer restores an archived device found by its external reference (https://github.com/zebbra/neops-core/pull/2363), facts and checks skip interfaces deleted earlier in the run (https://github.com/zebbra/neops-core/pull/2364), the LogicMonitor API supports
DELETE(https://github.com/zebbra/neops-core/pull/2337), and the C9500-16X boots fromflash:(https://github.com/zebbra/neops-core/pull/2393).
Dependencies
- Security updates for aiohttp, certifi, dnspython, msgpack and tornado. The remaining Twisted and autobahn advisories are on the annotated ignore list in
backend/scripts/pip_audit.sh, since the affected code is not reachable (https://github.com/zebbra/neops-core/pull/2354, https://github.com/zebbra/neops-core/pull/2382, https://github.com/zebbra/neops-core/pull/2410, https://github.com/zebbra/neops-core/pull/2418, https://github.com/zebbra/neops-core/pull/2459).
Aggregations over related entities
The search documents link related rows by id instead of embedding them, so an aggregation such as groups.name on devices is now computed from the related indices. Every field develop’s documents mapped aggregates as it did on develop: a row counts once per distinct value, values keep their type, and buckets are ordered by count, then value. A text field such as groups.name still buckets by word (zurich, hq); groups.name.raw buckets by name.
Three results differ from develop, on purpose:
groups.location.<field>on devices counts each device once per location it is in. Develop returned no buckets: its device document never carried a group’s location.- A user with group-restricted roles sees the clients those groups grant in client aggregations. On develop the restriction matched no client at all.
- A related entity’s field that develop’s embedded copy left out (for example
device.serialon interfaces) now aggregates instead of returning no buckets.
Two more follow from where a related entity’s dynamic facts (facts.*, checks.*) are read. They are now read from that entity’s own index and keyed by its mapping; develop read the copy embedded in the searched entity’s document, keyed by the searched index’s mapping, which took each fact’s type from the first linked copy indexed there.
- Facts named like an IP address. The device, interface and group indices map a fact named
ip,ipv4,ipv6,ip4,ip6oripaddress, in any letter case (IPv4), as an IP address; the client index keeps it as a plain string. So the paths that cross between clients and the other entities key it differently from develop: - clients by
interface.facts.ipv4,interface.device.facts.ipv4,interface.device.groups.facts.ipv4,interface.device.groups.location.facts.ipv4orlocation.facts.ipv4: one address written two ways (10.0.0.1and::ffff:10.0.0.1,2001:DB8::Aand2001:db8::a) is now one bucket in its normal form, and buckets with equal counts are ordered by address. Develop kept them apart, as written, and ordered them as text. - interfaces by
clients.facts.ipv4, devices byinterfaces.clients.facts.ipv4, groups bydevices.interfaces.clients.facts.ipv4: the other way round. Develop merged the spellings and ordered by address; now they are kept apart, as written, and ordered as text (10.0.0.10before10.0.0.9).
With a size limit, the top buckets can differ accordingly. Only paths where exactly one of the searched entity and the entity holding the fact is a client differ; interfaces by clients.location.facts.ipv4, for example, is unchanged.
- Facts whose values differ in type between rows (speed: "auto" on one interface, speed: 1000 on another; and devices and interfaces share one mapping, so also speed: "fast" on a device). The related index types such a fact by the first document that carried it, which can be a row the searched entity never reaches (an interface without clients, a group without devices) or, for devices and interfaces, the other entity. Values can then be keyed as strings ("1000", ordered as text) where develop keyed numbers or dates, or the other way round.
Develop also refused the whole searched document when a linked row’s fact did not fit the type the searched index had given that fact: with speed: 1000 on one device’s interface and speed: "auto" on another’s, develop’s devices index refused the second device, which then dropped out of every device search and aggregation, and its clients and groups indices could refuse that interface’s clients and the device’s groups the same way. Such rows now count; only the interface itself can be refused, as below, as develop’s interfaces index refused it.
Separately, and not caused by aggregation: devices and interfaces share one mapping in the network index (https://github.com/zebbra/neops-core/pull/2379), so Elasticsearch refuses, as a whole, a device or interface whose fact or check does not fit the type the other entity gave it, in either direction (an interface with mtu: "jumbo" after a device with mtu: 1500). Develop kept device and interface facts under separate mappings. What follows from a refusal:
- Searches and aggregations leave out a new row, and show an existing row with its old values: a rename or a group change does not reach search. So do the paths that read that row’s document from another entity: interfaces by
device.<field>(notdevice.id); clients byinterface.<field>(notinterface.id) and byinterface.device.<field>; devices byinterfaces.<field>; groups bydevices.<field>and bydevices.interfaces.<field>. Paths that go on through the database (device.groups.*,interface.device.groups.*,interfaces.clients.*,devices.interfaces.clients.*) and interfaces bydevice.idand clients byinterface.idstill count the rows behind it. - A mutation that saves such a row (
deviceUpsert,deviceAddToGroups) succeeds once the database commits: its response carriesextensions.indexing = "pending", and the row stays inerroruntil the periodic index sync task, ormanage.py resync_unsynced_rows, sets it aside as rejected. Only a script that callssave()outside a transaction still fails with “1 document(s) failed to index.”, although the change is saved. - A fact a workflow writes through an unlock’s
db_updates: the unlock succeeds, but every lock that searches the index is then refused withINDEX_SYNC_PENDINGuntil the document has been rejectedNEOPS_UNLOCK_INDEX_SYNC_MAX_ATTEMPTStimes (default 3, at leastNEOPS_UNLOCK_INDEX_SYNC_GRACE_SECONDS, 60 s, apart). From then on every lock that resolves that device or interface, by id or by query, is refused until its document is accepted: fix the fact and runresync_unlock_index --retry-quarantined, or give it up withresync_unlock_index --clear <reference>(the command lists the references). - Once set aside, the row is logged (“Elasticsearch rejected the document of Device
“). elastic_index --rebuildsets it aside too, as the upgrade notes above describe.
Sorting by a related entity’s field works again as on develop: interfaces by device.id, device.hostname and device.ip; clients by interface.id, interface.name, interface.device.id, interface.device.hostname, interface.device.ip and location.id, location.name, location.address, location.city, location.country, location.lat, location.lon.
While the indices are being rebuilt, aggregating or sorting by a related entity’s field fails with “The search indices are being rebuilt” instead of combining half-filled indices, and after a rebuild that failed or was killed part way, until a rebuild succeeds, with a message that names it and the rebuild command; a field of the entity itself still answers. When a search index is missing, an aggregation now fails, as lists already did, with “Search is not available right now: an index it needs is missing or being rebuilt” and the command that rebuilds it, instead of showing 0. Without https://github.com/zebbra/neops-docker-compose/pull/27, ./neops up does not repair a missing network, clients or groups index: it creates it empty, and until the indices are rebuilt, searches miss those rows without saying so, and a filter on one of their facts through a related entity (groups.facts.site:zh on devices) fails with “Unknown column [facts.site]”. With it, up’s --rebuild-if-stale rebuilds such an index. A missing lookup index ./neops up recreates and refills either way. Any other error Elasticsearch answers with reports its reason, or its type, with the index when Elasticsearch names one; a connection error or timeout still shows the client’s own message.
An archived row that a forward link still reaches (an imported client’s archived interface) has no document, so its values are read from the database and keyed as Elasticsearch would have keyed them under the field’s mapping. A value Elasticsearch would have refused ("1G" in a numeric fact) is left out and logged as “Left out archived …” on every request that reads it. A fact that only archived rows carry has no mapping: it is keyed as dynamic mapping would have mapped its first value, in row order; develop took the first document indexed, so a fact with values of different types can key differently.
Develop answered each of these in one request. Now they read the related rows for every request, so they cost more as those tables grow, and faster than linearly at a million rows. Measured on one Elasticsearch node with 5,000 devices, 100,000 interfaces and 50,000 clients; the first run is what a dashboard pays after the index changed, a repeated one is answered partly from Elasticsearch’s request cache:
| Aggregation or sort | First run | Repeated |
|---|---|---|
a field of the entity itself, groups.<field> on devices, device.<field> on interfaces, location.<field> on clients |
under 50 ms | under 50 ms |
interfaces.<field> on devices |
35 ms (name), 0.4 s (id) |
4 ms, 0.2 s |
clients.<field> on interfaces |
1.1 s | 0.4 s |
interfaces.clients.<field> on devices, devices.interfaces.<field> on groups |
0.6 s | 0.2 to 0.5 s |
interface.<field> and interface.device.<field> on clients |
0.4 to 0.8 s | 0.4 s |
clients sorted by interface.name or interface.device.hostname |
0.5 s | 0.5 to 0.7 s |
backend/neops_modules/opencore/neops_core/neops/core/documents/ELASTICSEARCH.md, section 7, has the details.
How index sync behaves now
Writes follow the commit
A change writes to the database inside its transaction and only marks the row pending there. The Elasticsearch write runs after the commit and reads the committed row. A rollback, or a process that dies before the commit, leaves search unchanged.
Restricted users and pending rows
When a change alters who may see a row (a device’s group membership, a client’s groups, an interface moved to another device), the row is flagged until its index write lands. While it is flagged, users with group-restricted roles do not see it through search. Admins and unrestricted users always see it.
Other changes, such as discovery updating facts, do not hide anything.
If more than 10,000 rows of one type wait for such a write (NEOPS_PERMISSION_PENDING_EXCLUDE_CAP), a restricted search fails with “Search is catching up after an indexing failure” instead of returning an incomplete list. The message names what brings the number down: the next workflow lock request, the periodic task where Celery beat runs, or manage.py resync_unsynced_rows. Rows hidden from restricted users that Elasticsearch rejected count too, and wait until their data is corrected and the row saved; manage.py index_sync_status --list rejected --limit 0 names them.
Repair after an Elasticsearch outage
A row whose index write failed stays error (or pending if the process died before writing). A device group whose documents a membership change has not rewritten yet is recorded the same way. Core re-indexes such rows and groups once NEOPS_UNLOCK_INDEX_SYNC_GRACE_SECONDS (default 60) have passed since the row’s status last changed (elastic_status_changed_at), so a change’s own write gets there first and a row that just failed is not retried at once. The longest waiting go first. It runs:
- on every lock request from the workflow engine, a few rows at a time (
NEOPS_INDEX_RESYNC_INLINE_BUDGET, default 20); - from the periodic unlock index sync task, when Celery beat runs (Helm: on by default; neops-docker-compose: the
cms-tasksprofile); - on demand with
manage.py resync_unsynced_rows [--budget N].
A run stops at the first sign that Elasticsearch or the database is unavailable. After that, lock requests wait NEOPS_INDEX_RESYNC_INLINE_BACKOFF_SECONDS (default 60) before trying again.
Runs that overlap share the work. A run leases the rows it takes, 20 at a time, for 30 seconds: other runs skip them meanwhile, and rows leased by a run that died are taken again once the lease ends. The periodic task runs one at a time; a run that finds another still running exits at once, and each run stops after 45 seconds. It holds a PostgreSQL advisory lock in an open transaction while it runs, so a server whose idle_in_transaction_session_timeout is below 45 seconds ends that session mid-run and lets the next run start alongside it; the row leases still keep the two from writing the same rows. A lock request that resolves its entities through search during an index rebuild is refused before it repairs anything.
Repairing a row writes its own documents and lookup rows, then the counts of the rows it names itself (an interface’s device, a client’s interface, a device’s groups), once per run. A group’s members are not rewritten by repairing the group: membership changes, archives and restores write them. Selecting the rows to repair reads a partial index (migration 0201_resync_selection_indexes), so a lock request with nothing to repair no longer scans the interface table.
A document Elasticsearch rejects as such (a mapping or parse error) is marked rejected and skipped until the row changes again. The reason is logged when Elasticsearch rejects the document and not again while the row waits, so if a manual resync_unsynced_rows run set the row aside, the sweep’s line is in that run’s output; the write that failed first logged the same reason as ES sync failed for <row> where the change was made.
A row that fails for a reason of its own, such as data that cannot be serialized, stays error until it changes and is logged with its traceback. The run goes on with the next row, and the row is tried again the same wait after that try (elastic_sync_retry_at). Its elastic_status_changed_at keeps the time it first failed, so a row that keeps failing shows its real age.
Runbook: search disagrees with the database
The commands are shown for neops-docker-compose. With Helm, run kubectl -n <namespace> exec deploy/<release>-neops-core -- /docker-entrypoint.sh ./manage.py <command> instead; from a source checkout, uv run python manage.py <command> in backend/.
- Re-index what Core knows is behind:
It prints how many rows it re-indexed, set aside, left for the next sweep and failed on. It exits 1 if Elasticsearch or the database is unavailable, and 3 if rows failed for a reason of their own (the log names them; they are retried later). Exit 3 here is not the rebuild’s 3: it means rows failed on their own data and will be retried. It takes every unsynced row at once, without the wait, except the rejected ones.
- If search is still wrong, refill the indices from the database without deleting them:
./neops compose -- exec -T cms python manage.py elastic_index --populate --no-sync-lookups
./neops compose -- exec -T cms python manage.py sync_lookup_indices --populate-only
- Only if an index has the wrong mapping (for example after Elasticsearch created it on its own), rebuild it. An index Elasticsearch created has no build record, so
--rebuild-if-stalerebuilds it, with the indices it shares or mirrors, and leaves the others alone. This takes search down while it runs:
